2024-03-28T18:40:58.638 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T18:40:58.638 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625389 --owner mchangir --description fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625389/orig.config.yaml 2024-03-28T18:40:58.688 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T18:40:58.780 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625389 branch: wip-mchangir-testing-main-20240318.032620 description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} email: mchangir@redhat.com first_in_suite: false job_id: '7625389' kernel: flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: fs: - name: a - name: b session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - Replacing daemon mds sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: cephfs_name: a client.1: cephfs_name: b selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpyBMUwGrjTQ7aZWPwN83VIoQIjtIAHFM/eoBRSfAydbDVy717FR+I+Z/Bes6ieeXc3y+6w+7Un8hrvLAE/pKM= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBxvKut37D7jeWMtpGEFyG8+HC+Kayd8PIovHDyqxJsTbKAefpdTobI8Th5G8HA/sVPrJJdz6g8m64MvjuavA8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T18:40:58.781 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T18:40:58.781 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T18:40:58.782 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T18:40:58.783 INFO:teuthology.task.internal:Checking packages... 2024-03-28T18:40:58.809 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T18:40:58.809 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T18:40:58.809 INFO:teuthology.packaging:ref: None 2024-03-28T18:40:58.809 INFO:teuthology.packaging:tag: None 2024-03-28T18:40:58.809 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T18:40:58.809 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:40:58.809 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T18:40:58.954 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T18:40:58.956 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T18:40:58.963 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T18:40:58.963 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T18:40:58.974 INFO:teuthology.task.internal:Saving configuration 2024-03-28T18:40:58.993 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T18:40:58.999 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T18:40:59.025 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625389', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:34:42.942109', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpyBMUwGrjTQ7aZWPwN83VIoQIjtIAHFM/eoBRSfAydbDVy717FR+I+Z/Bes6ieeXc3y+6w+7Un8hrvLAE/pKM='} 2024-03-28T18:40:59.049 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625389', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:34:42.940633', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBxvKut37D7jeWMtpGEFyG8+HC+Kayd8PIovHDyqxJsTbKAefpdTobI8Th5G8HA/sVPrJJdz6g8m64MvjuavA8='} 2024-03-28T18:40:59.049 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T18:40:59.054 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-28T18:40:59.055 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-03-28T18:40:59.055 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T18:40:59.157 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1b8a787ee0>, signals=[15]) 2024-03-28T18:40:59.157 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T18:40:59.164 INFO:teuthology.task.internal:Opening connections... 2024-03-28T18:40:59.164 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-03-28T18:40:59.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:40:59.272 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-03-28T18:40:59.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:40:59.359 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T18:40:59.366 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-03-28T18:40:59.384 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-03-28T18:40:59.384 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-03-28T18:40:59.440 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T18:40:59.441 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T18:40:59.442 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-03-28T18:40:59.467 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-03-28T18:40:59.483 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-03-28T18:40:59.483 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-03-28T18:40:59.538 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T18:40:59.539 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T18:40:59.540 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-03-28T18:40:59.558 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T18:40:59.568 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T18:40:59.573 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T18:40:59.573 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-03-28T18:40:59.576 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-03-28T18:40:59.594 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T18:40:59.599 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T18:40:59.600 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-03-28T18:40:59.633 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-03-28T18:40:59.655 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T18:40:59.711 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T18:40:59.726 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T18:40:59.726 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, timeout 300 2024-03-28T18:40:59.726 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T18:40:59.727 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T18:40:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:40:59.741 DEBUG:teuthology.task.kernel:sha1 for mon.b is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:40:59.742 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T18:40:59.742 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-28T18:40:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:40:59.743 DEBUG:teuthology.task.kernel:sha1 for mon.a is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:40:59.744 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T18:40:59.744 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-03-28T18:40:59.797 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-361.el9.x86_64 2024-03-28T18:40:59.797 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:40:59.797 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T18:40:59.797 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:40:59.801 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-361.el9.x86_64 2024-03-28T18:40:59.802 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:40:59.802 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T18:40:59.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:40:59.942 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum upgrade -y linux-firmware 2024-03-28T18:40:59.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum upgrade -y linux-firmware 2024-03-28T18:41:02.088 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-28T18:41:02.092 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-28T18:41:03.534 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-28T18:41:03.621 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-28T18:41:04.706 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-28T18:41:04.733 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-03-28T18:41:08.355 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-03-28T18:41:12.464 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-28T18:41:14.165 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-28T18:41:15.394 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-28T18:41:20.533 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-28T18:41:22.897 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-28T18:41:22.911 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-28T18:41:22.911 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-28T18:41:22.911 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 1 Package 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 375 M 2024-03-28T18:41:22.912 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-28T18:41:27.056 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 909 kB/s | 19 MB 00:21 2024-03-28T18:41:30.392 INFO:teuthology.orchestra.run.smithi027.stdout:linux-firmware-20240219-143.el9.noarch.rpm 50 MB/s | 375 MB 00:07 2024-03-28T18:41:30.394 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:41:30.394 INFO:teuthology.orchestra.run.smithi027.stdout:Total 50 MB/s | 375 MB 00:07 2024-03-28T18:41:30.394 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-28T18:41:30.483 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-28T18:41:30.483 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-28T18:41:31.798 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.1 MB 00:01 2024-03-28T18:41:32.988 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-28T18:41:32.988 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-28T18:41:33.282 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-28T18:41:34.703 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-28T18:41:39.600 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-28T18:41:39.644 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-28T18:41:39.867 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T18:41:39.961 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:41:41.385 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:41:41.385 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T18:41:41.789 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:41:41.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:41:41.789 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-03-28T18:41:41.789 INFO:teuthology.orchestra.run.smithi027.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T18:41:41.790 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:41:41.790 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-28T18:41:41.831 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 1 Package 2024-03-28T18:41:41.843 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:41:41.844 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 375 M 2024-03-28T18:41:41.844 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-28T18:41:42.078 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.a... 2024-03-28T18:41:42.079 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:41:42.231 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T18:41:42.231 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T18:41:42.234 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi027.front.sepia.ceph.com to complete... 2024-03-28T18:41:49.134 INFO:teuthology.orchestra.run.smithi155.stdout:linux-firmware-20240219-143.el9.noarch.rpm 51 MB/s | 375 MB 00:07 2024-03-28T18:41:49.135 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:41:49.135 INFO:teuthology.orchestra.run.smithi155.stdout:Total 51 MB/s | 375 MB 00:07 2024-03-28T18:41:49.136 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-28T18:41:49.246 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-28T18:41:49.246 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-28T18:41:51.764 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-28T18:41:51.764 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-28T18:41:53.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28 18:41:53 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T18:41:53.874 INFO:teuthology.orchestra.run.smithi027.stderr:FINISHED --2024-03-28 18:41:53-- 2024-03-28T18:41:53.874 INFO:teuthology.orchestra.run.smithi027.stderr:Total wall clock time: 12s 2024-03-28T18:41:53.874 INFO:teuthology.orchestra.run.smithi027.stderr:Downloaded: 1 files, 321M in 11s (28.3 MB/s) 2024-03-28T18:41:53.877 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.a... 2024-03-28T18:41:53.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T18:41:55.912 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying... ######################################## 2024-03-28T18:41:55.976 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing... ######################################## 2024-03-28T18:41:56.061 INFO:teuthology.orchestra.run.smithi027.stdout:Updating / installing... 2024-03-28T18:41:58.158 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-28T18:41:58.358 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T18:41:58.449 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:41:59.934 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:41:59.934 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T18:42:00.333 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:42:00.333 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:42:00.333 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-03-28T18:42:00.333 INFO:teuthology.orchestra.run.smithi155.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T18:42:00.333 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:42:00.333 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-28T18:42:00.640 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.b... 2024-03-28T18:42:00.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:42:00.787 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T18:42:00.787 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T18:42:00.790 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi155.front.sepia.ceph.com to complete... 2024-03-28T18:42:03.905 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T18:42:12.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28 18:42:12 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T18:42:12.187 INFO:teuthology.orchestra.run.smithi155.stderr:FINISHED --2024-03-28 18:42:12-- 2024-03-28T18:42:12.187 INFO:teuthology.orchestra.run.smithi155.stderr:Total wall clock time: 11s 2024-03-28T18:42:12.187 INFO:teuthology.orchestra.run.smithi155.stderr:Downloaded: 1 files, 321M in 11s (28.9 MB/s) 2024-03-28T18:42:12.190 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.b... 2024-03-28T18:42:12.191 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T18:42:14.131 INFO:teuthology.orchestra.run.smithi155.stdout:Verifying... ######################################## 2024-03-28T18:42:14.273 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing... ######################################## 2024-03-28T18:42:14.353 INFO:teuthology.orchestra.run.smithi155.stdout:Updating / installing... 2024-03-28T18:42:19.760 INFO:teuthology.orchestra.run.smithi027.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T18:42:21.905 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T18:42:38.031 INFO:teuthology.orchestra.run.smithi155.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T18:42:43.044 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T18:42:43.044 DEBUG:teuthology.orchestra.run.smithi027:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T18:42:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T18:42:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T18:42:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T18:42:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T18:42:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T18:42:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T18:42:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T18:42:43.124 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T18:42:43.125 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T18:42:43.126 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T18:42:43.127 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T18:42:43.128 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T18:42:43.129 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T18:42:43.130 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T18:42:43.131 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T18:42:43.132 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T18:42:43.133 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T18:42:43.134 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T18:42:43.135 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T18:42:43.136 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T18:42:43.137 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T18:42:43.138 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T18:42:43.139 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T18:42:43.140 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T18:42:43.141 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T18:42:43.142 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T18:42:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T18:42:43.144 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T18:42:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T18:42:43.146 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T18:42:43.147 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T18:42:43.148 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T18:42:43.149 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T18:42:43.150 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T18:42:43.151 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T18:42:43.152 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T18:42:43.153 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T18:42:43.154 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T18:42:43.155 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T18:42:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T18:42:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T18:42:43.158 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T18:42:43.159 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T18:42:43.160 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T18:42:43.161 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T18:42:43.162 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T18:42:43.163 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T18:42:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T18:42:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T18:42:43.166 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T18:42:43.167 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T18:42:43.168 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T18:42:43.169 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T18:42:43.170 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T18:42:43.171 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T18:42:43.172 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T18:42:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T18:42:43.174 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T18:42:43.175 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T18:42:43.176 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T18:42:43.177 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T18:42:43.178 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T18:42:43.179 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T18:42:43.180 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T18:42:43.181 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T18:42:43.182 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T18:42:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T18:42:43.184 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T18:42:43.185 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T18:42:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T18:42:43.187 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T18:42:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T18:42:43.189 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T18:42:43.190 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T18:42:43.191 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T18:42:43.192 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T18:42:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T18:42:43.194 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T18:42:43.195 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T18:42:43.196 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T18:42:43.197 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T18:42:43.198 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T18:42:43.199 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T18:42:43.200 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T18:42:43.201 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T18:42:43.202 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T18:42:43.203 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T18:42:43.204 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T18:42:43.205 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T18:42:43.206 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T18:42:43.207 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T18:42:43.208 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T18:42:43.209 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T18:42:43.210 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T18:42:43.211 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T18:42:43.212 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T18:42:43.213 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T18:42:43.214 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T18:42:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T18:42:43.216 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T18:42:43.217 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T18:42:43.218 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T18:42:43.219 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T18:42:43.220 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T18:42:43.221 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T18:42:43.222 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T18:42:43.223 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T18:42:43.224 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T18:42:43.225 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T18:42:43.226 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T18:42:43.227 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T18:42:43.228 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T18:42:43.229 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T18:42:43.230 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T18:42:43.231 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T18:42:43.232 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T18:42:43.233 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T18:42:43.234 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T18:42:43.235 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T18:42:43.236 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T18:42:43.237 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T18:42:43.238 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T18:42:43.239 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T18:42:43.240 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T18:42:43.241 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T18:42:43.242 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T18:42:43.243 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T18:42:43.244 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T18:42:43.245 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T18:42:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T18:42:43.247 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T18:42:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T18:42:43.249 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T18:42:43.250 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T18:42:43.251 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T18:42:43.252 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T18:42:43.253 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T18:42:43.254 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T18:42:43.255 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T18:42:43.256 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T18:42:43.257 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T18:42:43.258 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T18:42:43.259 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T18:42:43.262 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T18:42:43.263 DEBUG:teuthology.orchestra.run.smithi027:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T18:42:43.286 INFO:teuthology.orchestra.run.smithi027.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T18:42:43.286 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T18:42:43.286 INFO:teuthology.orchestra.run.smithi027.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T18:42:43.286 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:for file in vmlinuz System.map config; do 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout: fi 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:done 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T18:42:43.287 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-03-28T18:42:43.288 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 68.el9 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8264254 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-03-28T18:42:43.378 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:42:43.379 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T18:42:43.380 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T18:42:43.380 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 6.8.0-g1df0c7e8a235 2024-03-28T18:42:43.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T18:42:43.892 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-03-28T18:42:45.574 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T18:42:45.604 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-03-28T18:42:45.606 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-03-28T18:42:45.622 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.dcfnCQzJf3 2024-03-28T18:42:45.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dcfnCQzJf3 2024-03-28T18:42:45.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.dcfnCQzJf3 2024-03-28T18:42:45.843 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.dcfnCQzJf3 is 6KB 2024-03-28T18:42:45.891 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.dcfnCQzJf3 2024-03-28T18:42:45.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T18:42:45.978 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T18:42:45.978 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T18:42:45.978 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T18:42:45.978 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T18:42:45.980 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T18:42:46.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-03-28T18:43:00.578 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T18:43:00.579 DEBUG:teuthology.orchestra.run.smithi155:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T18:43:00.651 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T18:43:00.651 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T18:43:00.651 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T18:43:00.652 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T18:43:00.653 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T18:43:00.654 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T18:43:00.655 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T18:43:00.656 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T18:43:00.657 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T18:43:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T18:43:00.659 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T18:43:00.659 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T18:43:00.659 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T18:43:00.659 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T18:43:00.661 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T18:43:00.661 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T18:43:00.662 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T18:43:00.663 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T18:43:00.664 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T18:43:00.665 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T18:43:00.666 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T18:43:00.667 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T18:43:00.668 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T18:43:00.669 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T18:43:00.670 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T18:43:00.671 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T18:43:00.672 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T18:43:00.673 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T18:43:00.674 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T18:43:00.675 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T18:43:00.676 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T18:43:00.677 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T18:43:00.678 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T18:43:00.679 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T18:43:00.680 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T18:43:00.681 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T18:43:00.682 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T18:43:00.683 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T18:43:00.684 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T18:43:00.685 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T18:43:00.686 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T18:43:00.687 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T18:43:00.688 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T18:43:00.689 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T18:43:00.690 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T18:43:00.691 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T18:43:00.692 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T18:43:00.693 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T18:43:00.694 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T18:43:00.695 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T18:43:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T18:43:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T18:43:00.698 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T18:43:00.699 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T18:43:00.700 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T18:43:00.701 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T18:43:00.702 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T18:43:00.703 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T18:43:00.704 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T18:43:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T18:43:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T18:43:00.707 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T18:43:00.708 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T18:43:00.709 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T18:43:00.710 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T18:43:00.711 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T18:43:00.712 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T18:43:00.713 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T18:43:00.714 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T18:43:00.715 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T18:43:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T18:43:00.717 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T18:43:00.718 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T18:43:00.719 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T18:43:00.720 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T18:43:00.721 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T18:43:00.722 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T18:43:00.723 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T18:43:00.724 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T18:43:00.725 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T18:43:00.726 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T18:43:00.727 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T18:43:00.728 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T18:43:00.729 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T18:43:00.730 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T18:43:00.731 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T18:43:00.732 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T18:43:00.733 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T18:43:00.734 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T18:43:00.735 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T18:43:00.736 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T18:43:00.737 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T18:43:00.738 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T18:43:00.739 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T18:43:00.740 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T18:43:00.741 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T18:43:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T18:43:00.743 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T18:43:00.744 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T18:43:00.745 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T18:43:00.746 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T18:43:00.747 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T18:43:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T18:43:00.749 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T18:43:00.750 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T18:43:00.751 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T18:43:00.752 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T18:43:00.753 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T18:43:00.754 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T18:43:00.755 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T18:43:00.756 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T18:43:00.757 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T18:43:00.758 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T18:43:00.759 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T18:43:00.760 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T18:43:00.761 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T18:43:00.762 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T18:43:00.763 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T18:43:00.764 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T18:43:00.765 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T18:43:00.766 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T18:43:00.767 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T18:43:00.768 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T18:43:00.769 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T18:43:00.770 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T18:43:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T18:43:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T18:43:00.773 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T18:43:00.774 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T18:43:00.775 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T18:43:00.776 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T18:43:00.777 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T18:43:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T18:43:00.779 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T18:43:00.780 INFO:teuthology.orchestra.run.smithi155.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T18:43:00.783 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T18:43:00.783 DEBUG:teuthology.orchestra.run.smithi155:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T18:43:00.805 INFO:teuthology.orchestra.run.smithi155.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T18:43:00.805 INFO:teuthology.orchestra.run.smithi155.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T18:43:00.805 INFO:teuthology.orchestra.run.smithi155.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T18:43:00.805 INFO:teuthology.orchestra.run.smithi155.stdout:fi 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:for file in vmlinuz System.map config; do 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout: fi 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:done 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:fi 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi155.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T18:43:00.807 INFO:teuthology.orchestra.run.smithi155.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T18:43:00.807 INFO:teuthology.orchestra.run.smithi155.stdout:fi 2024-03-28T18:43:00.807 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 68.el9 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8264254 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T18:43:00.883 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:43:00.884 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T18:43:00.885 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T18:43:00.885 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 6.8.0-g1df0c7e8a235 2024-03-28T18:43:00.885 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T18:43:01.467 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-03-28T18:43:02.886 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T18:43:02.925 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-03-28T18:43:02.928 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-28T18:43:02.943 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.1rUzaNB9Op 2024-03-28T18:43:02.943 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1rUzaNB9Op 2024-03-28T18:43:03.016 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.1rUzaNB9Op 2024-03-28T18:43:03.161 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.1rUzaNB9Op is 6KB 2024-03-28T18:43:03.210 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.1rUzaNB9Op 2024-03-28T18:43:03.224 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T18:43:03.289 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T18:43:03.289 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T18:43:03.289 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T18:43:03.289 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T18:43:03.290 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T18:43:03.430 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-03-28T18:43:16.156 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-28T18:43:16.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:43:16.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:43:33.451 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-28T18:43:33.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-28T18:43:33.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:43:34.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-28T18:43:43.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:43:43.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:43:46.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-28T18:43:51.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-28T18:43:58.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:43:58.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:44:00.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-28T18:44:00.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:44:01.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-28T18:44:04.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-28T18:44:16.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-28T18:44:16.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:44:16.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:44:16.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:44:19.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-28T18:44:37.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:44:37.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:44:56.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-03-28T18:45:16.048 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T18:45:17.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:45:17.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:45:17.722 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-03-28T18:45:18.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:45:18.430 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.8.0-g1df0c7e8a235"... 2024-03-28T18:45:18.430 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-03-28T18:45:18.472 INFO:teuthology.orchestra.run.smithi027.stdout:6.8.0-g1df0c7e8a235 2024-03-28T18:45:18.472 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T18:45:18.472 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T18:45:18.472 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T18:45:19.473 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-28T18:45:19.473 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T18:45:19.602 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-03-28T18:45:19.645 DEBUG:teuthology.parallel:result is None 2024-03-28T18:45:31.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-28T18:45:31.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:45:31.515 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-28T18:45:32.201 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-28T18:45:32.201 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.8.0-g1df0c7e8a235"... 2024-03-28T18:45:32.202 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-28T18:45:32.252 INFO:teuthology.orchestra.run.smithi155.stdout:6.8.0-g1df0c7e8a235 2024-03-28T18:45:32.252 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T18:45:32.252 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T18:45:32.252 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T18:45:33.253 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-28T18:45:33.254 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T18:45:33.396 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-03-28T18:45:33.459 DEBUG:teuthology.parallel:result is None 2024-03-28T18:45:33.459 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T18:45:33.470 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T18:45:33.470 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T18:45:33.478 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T18:45:33.533 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T18:45:33.540 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T18:45:33.553 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T18:45:33.553 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T18:45:33.561 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T18:45:33.642 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T18:45:33.649 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T18:45:33.649 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T18:45:33.656 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T18:45:33.764 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:45:33.776 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:45:33.816 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:45:33.822 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:45:33.827 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T18:45:33.834 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T18:45:33.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T18:45:33.841 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T18:45:33.940 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T18:45:33.950 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T18:45:33.950 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T18:45:33.958 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T18:45:34.011 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:45:34.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:45:34.249 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:45:34.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:45:34.444 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:45:34.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T18:45:34.550 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:45:34.637 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:45:34.787 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:45:34.869 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:45:34.968 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:45:34.968 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T18:45:35.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-03-28T18:45:35.068 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-03-28T18:45:35.190 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:45:35.198 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:45:35.726 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T18:45:35.734 INFO:teuthology.task.internal:Starting timer... 2024-03-28T18:45:35.734 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T18:45:35.775 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T18:45:35.784 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-28T18:45:35.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-03-28T18:45:35.941 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-03-28T18:45:35.947 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-03-28T18:45:36.081 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-03-28T18:45:36.087 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T18:45:36.087 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-03-28T18:45:36.151 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-03-28T18:45:36.152 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-03-28T18:45:36.207 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-03-28T18:45:36.207 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-03-28T18:45:36.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:45:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:45:36.296 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:45:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:45:36.368 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T18:45:36.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-03-28T18:45:36.463 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-03-28T18:45:36.554 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T18:45:36.563 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-28T18:45:36.569 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-28T18:45:36.723 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T18:45:36.738 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T18:45:36.740 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T18:52:05.352 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-03-28T18:52:05.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:52:05.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:52:05.559 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-03-28T18:52:05.716 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-03-28T18:52:05.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-28T18:52:05.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:52:05.855 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-28T18:52:05.987 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-28T18:52:05.987 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T18:52:05.998 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T18:52:05.998 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T18:52:05.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:52:06.006 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T18:52:06.006 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:52:06.115 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T18:52:06.116 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T18:52:06.166 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T18:52:06.175 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T18:52:06.256 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-03-28T18:52:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-03-28T18:52:06.298 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-03-28T18:52:06.332 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-03-28T18:52:06.346 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T18:52:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T18:52:06.398 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T18:52:06.450 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T18:52:06.527 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:52:06.535 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:52:06.535 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-28T18:52:06.535 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:52:06.535 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:52:06.535 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:52:06.535 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:52:06.591 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:52:06.600 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:52:06.601 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-03-28T18:52:06.601 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:52:06.601 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:52:06.601 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:52:06.601 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:52:06.602 INFO:teuthology.run_tasks:Running task install... 2024-03-28T18:52:06.611 DEBUG:teuthology.task.install:project ceph 2024-03-28T18:52:06.611 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T18:52:06.611 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T18:52:06.612 INFO:teuthology.task.install:Using flavor: default 2024-03-28T18:52:06.622 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T18:52:06.622 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T18:52:06.622 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T18:52:06.623 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:52:06.625 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T18:52:06.625 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:52:06.845 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T18:52:06.846 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T18:52:06.850 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T18:52:06.850 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T18:52:06.959 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T18:52:06.959 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:52:06.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T18:52:07.013 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T18:52:07.013 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:52:07.013 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T18:52:07.072 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T18:52:07.072 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T18:52:07.107 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T18:52:07.107 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T18:52:07.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T18:52:07.244 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T18:52:07.370 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T18:52:07.372 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T18:52:07.454 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-03-28T18:52:07.458 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-03-28T18:52:07.465 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-03-28T18:52:07.470 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-03-28T18:52:07.824 INFO:teuthology.orchestra.run.smithi155.stdout:74 files removed 2024-03-28T18:52:07.846 INFO:teuthology.orchestra.run.smithi027.stdout:74 files removed 2024-03-28T18:52:07.866 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T18:52:07.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T18:52:08.897 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 158 kB/s | 83 kB 00:00 2024-03-28T18:52:08.929 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 159 kB/s | 83 kB 00:00 2024-03-28T18:52:09.479 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-28T18:52:09.489 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-28T18:52:10.005 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-03-28T18:52:10.015 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-03-28T18:52:10.902 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.7 MB 00:00 2024-03-28T18:52:10.904 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 8.1 MB/s | 6.7 MB 00:00 2024-03-28T18:52:12.576 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-28T18:52:12.636 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-28T18:52:14.055 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-03-28T18:52:14.598 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-03-28T18:52:16.298 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-28T18:52:16.649 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-28T18:52:21.649 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-28T18:52:22.400 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:01 2024-03-28T18:52:23.317 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-03-28T18:52:23.992 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-03-28T18:52:24.936 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-03-28T18:52:25.582 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-03-28T18:52:30.644 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-28T18:52:31.585 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-28T18:52:33.237 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:52:33.239 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:52:33.260 INFO:teuthology.orchestra.run.smithi155.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T18:52:33.262 INFO:teuthology.orchestra.run.smithi155.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T18:52:33.263 INFO:teuthology.orchestra.run.smithi155.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T18:52:33.265 INFO:teuthology.orchestra.run.smithi155.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T18:52:33.267 INFO:teuthology.orchestra.run.smithi155.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T18:52:33.271 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T18:52:33.272 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T18:52:33.360 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-28T18:52:33.389 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-28T18:52:33.389 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-28T18:52:33.389 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-28T18:52:33.389 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-28T18:52:33.389 INFO:teuthology.orchestra.run.smithi155.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T18:52:33.389 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T18:52:33.389 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T18:52:33.390 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T18:52:33.391 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T18:52:33.392 INFO:teuthology.orchestra.run.smithi155.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T18:52:33.392 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T18:52:33.393 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T18:52:33.394 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T18:52:33.395 INFO:teuthology.orchestra.run.smithi155.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T18:52:33.396 INFO:teuthology.orchestra.run.smithi155.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T18:52:33.397 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T18:52:33.398 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T18:52:33.399 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T18:52:33.400 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T18:52:33.401 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T18:52:33.402 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T18:52:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T18:52:33.404 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T18:52:33.405 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T18:52:33.406 INFO:teuthology.orchestra.run.smithi155.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout:Install 150 Packages 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 24 Packages 2024-03-28T18:52:33.407 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:52:33.408 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 225 M 2024-03-28T18:52:33.408 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-28T18:52:34.153 INFO:teuthology.orchestra.run.smithi155.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-03-28T18:52:34.391 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:52:34.392 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:52:34.412 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T18:52:34.413 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T18:52:34.415 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T18:52:34.416 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T18:52:34.418 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T18:52:34.422 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T18:52:34.423 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T18:52:34.520 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T18:52:34.539 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T18:52:34.540 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T18:52:34.541 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T18:52:34.542 INFO:teuthology.orchestra.run.smithi027.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-03-28T18:52:34.543 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T18:52:34.544 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T18:52:34.545 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T18:52:34.546 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T18:52:34.547 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T18:52:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T18:52:34.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T18:52:34.550 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================================ 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout:Install 150 Packages 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 24 Packages 2024-03-28T18:52:34.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:52:34.552 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 225 M 2024-03-28T18:52:34.552 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-03-28T18:52:34.737 INFO:teuthology.orchestra.run.smithi155.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.5 MB/s | 890 kB 00:00 2024-03-28T18:52:34.845 INFO:teuthology.orchestra.run.smithi155.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.3 MB/s | 145 kB 00:00 2024-03-28T18:52:35.388 INFO:teuthology.orchestra.run.smithi155.stdout:(4/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 13 MB/s | 20 MB 00:01 2024-03-28T18:52:35.521 INFO:teuthology.orchestra.run.smithi155.stdout:(5/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 3.2 MB/s | 5.4 MB 00:01 2024-03-28T18:52:35.613 INFO:teuthology.orchestra.run.smithi155.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 3.0 MB/s | 2.3 MB 00:00 2024-03-28T18:52:35.680 INFO:teuthology.orchestra.run.smithi155.stdout:(7/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 5.1 MB/s | 1.5 MB 00:00 2024-03-28T18:52:36.061 INFO:teuthology.orchestra.run.smithi027.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-03-28T18:52:36.389 INFO:teuthology.orchestra.run.smithi155.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 14 MB/s | 9.9 MB 00:00 2024-03-28T18:52:36.511 INFO:teuthology.orchestra.run.smithi027.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.9 MB/s | 890 kB 00:00 2024-03-28T18:52:36.522 INFO:teuthology.orchestra.run.smithi155.stdout:(9/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 4.7 MB/s | 4.7 MB 00:00 2024-03-28T18:52:36.539 INFO:teuthology.orchestra.run.smithi155.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 169 kB/s | 25 kB 00:00 2024-03-28T18:52:36.622 INFO:teuthology.orchestra.run.smithi155.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.6 MB/s | 220 kB 00:00 2024-03-28T18:52:36.627 INFO:teuthology.orchestra.run.smithi027.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.2 MB/s | 145 kB 00:00 2024-03-28T18:52:36.681 INFO:teuthology.orchestra.run.smithi155.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 469 kB/s | 27 kB 00:00 2024-03-28T18:52:36.748 INFO:teuthology.orchestra.run.smithi155.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 474 kB/s | 31 kB 00:00 2024-03-28T18:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 3.9 MB/s | 2.3 MB 00:00 2024-03-28T18:52:37.371 INFO:teuthology.orchestra.run.smithi027.stdout:(5/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 3.0 MB/s | 5.4 MB 00:01 2024-03-28T18:52:37.381 INFO:teuthology.orchestra.run.smithi155.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 1.2 MB/s | 746 kB 00:00 2024-03-28T18:52:37.473 INFO:teuthology.orchestra.run.smithi155.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.8 MB/s | 163 kB 00:00 2024-03-28T18:52:37.621 INFO:teuthology.orchestra.run.smithi027.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 3.7 MB/s | 1.5 MB 00:00 2024-03-28T18:52:37.707 INFO:teuthology.orchestra.run.smithi155.stdout:(16/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 8.0 MB/s | 17 MB 00:02 2024-03-28T18:52:37.740 INFO:teuthology.orchestra.run.smithi155.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 477 kB/s | 127 kB 00:00 2024-03-28T18:52:37.799 INFO:teuthology.orchestra.run.smithi155.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 5.4 MB/s | 501 kB 00:00 2024-03-28T18:52:37.924 INFO:teuthology.orchestra.run.smithi155.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 364 kB/s | 45 kB 00:00 2024-03-28T18:52:38.033 INFO:teuthology.orchestra.run.smithi155.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.2 MB/s | 131 kB 00:00 2024-03-28T18:52:38.158 INFO:teuthology.orchestra.run.smithi155.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.3 MB/s | 162 kB 00:00 2024-03-28T18:52:38.292 INFO:teuthology.orchestra.run.smithi155.stdout:(22/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 9.4 MB/s | 5.1 MB 00:00 2024-03-28T18:52:38.334 INFO:teuthology.orchestra.run.smithi155.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 1.8 MB/s | 321 kB 00:00 2024-03-28T18:52:38.367 INFO:teuthology.orchestra.run.smithi155.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T18:52:38.409 INFO:teuthology.orchestra.run.smithi155.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-28T18:52:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 4.4 MB/s | 4.7 MB 00:01 2024-03-28T18:52:38.434 INFO:teuthology.orchestra.run.smithi155.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-28T18:52:38.527 INFO:teuthology.orchestra.run.smithi155.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-28T18:52:38.627 INFO:teuthology.orchestra.run.smithi155.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 342 kB/s | 34 kB 00:00 2024-03-28T18:52:38.719 INFO:teuthology.orchestra.run.smithi155.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.5 MB/s | 139 kB 00:00 2024-03-28T18:52:38.794 INFO:teuthology.orchestra.run.smithi155.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 8.1 MB/s | 3.1 MB 00:00 2024-03-28T18:52:39.036 INFO:teuthology.orchestra.run.smithi155.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 5.7 MB/s | 1.8 MB 00:00 2024-03-28T18:52:39.136 INFO:teuthology.orchestra.run.smithi155.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.5 MB/s | 248 kB 00:00 2024-03-28T18:52:39.220 INFO:teuthology.orchestra.run.smithi155.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 597 kB/s | 49 kB 00:00 2024-03-28T18:52:39.303 INFO:teuthology.orchestra.run.smithi155.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 199 kB/s | 16 kB 00:00 2024-03-28T18:52:39.395 INFO:teuthology.orchestra.run.smithi155.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.8 MB/s | 264 kB 00:00 2024-03-28T18:52:39.696 INFO:teuthology.orchestra.run.smithi155.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 8.2 MB/s | 7.4 MB 00:00 2024-03-28T18:52:39.779 INFO:teuthology.orchestra.run.smithi155.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 1.9 MB/s | 748 kB 00:00 2024-03-28T18:52:39.805 INFO:teuthology.orchestra.run.smithi155.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 169 kB/s | 18 kB 00:00 2024-03-28T18:52:40.022 INFO:teuthology.orchestra.run.smithi155.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-28T18:52:40.064 INFO:teuthology.orchestra.run.smithi155.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.5 MB/s | 920 kB 00:00 2024-03-28T18:52:40.089 INFO:teuthology.orchestra.run.smithi155.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.7 MB/s | 106 kB 00:00 2024-03-28T18:52:40.122 INFO:teuthology.orchestra.run.smithi155.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-28T18:52:40.138 INFO:teuthology.orchestra.run.smithi027.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 5.8 MB/s | 9.9 MB 00:01 2024-03-28T18:52:40.164 INFO:teuthology.orchestra.run.smithi155.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 440 kB/s | 32 kB 00:00 2024-03-28T18:52:40.188 INFO:teuthology.orchestra.run.smithi027.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 516 kB/s | 25 kB 00:00 2024-03-28T18:52:40.256 INFO:teuthology.orchestra.run.smithi155.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 23 MB/s | 3.0 MB 00:00 2024-03-28T18:52:40.273 INFO:teuthology.orchestra.run.smithi155.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 176 kB/s | 17 kB 00:00 2024-03-28T18:52:40.298 INFO:teuthology.orchestra.run.smithi155.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 422 kB/s | 10 kB 00:00 2024-03-28T18:52:40.324 INFO:teuthology.orchestra.run.smithi155.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-03-28T18:52:40.340 INFO:teuthology.orchestra.run.smithi155.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-28T18:52:40.391 INFO:teuthology.orchestra.run.smithi155.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-28T18:52:40.433 INFO:teuthology.orchestra.run.smithi155.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 6.6 MB/s | 578 kB 00:00 2024-03-28T18:52:40.458 INFO:teuthology.orchestra.run.smithi155.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 751 kB/s | 46 kB 00:00 2024-03-28T18:52:40.492 INFO:teuthology.orchestra.run.smithi155.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-28T18:52:40.525 INFO:teuthology.orchestra.run.smithi155.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-28T18:52:40.559 INFO:teuthology.orchestra.run.smithi155.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-28T18:52:40.584 INFO:teuthology.orchestra.run.smithi155.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 609 kB/s | 34 kB 00:00 2024-03-28T18:52:40.726 INFO:teuthology.orchestra.run.smithi155.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-03-28T18:52:40.768 INFO:teuthology.orchestra.run.smithi155.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-03-28T18:52:40.810 INFO:teuthology.orchestra.run.smithi155.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.9 MB/s | 78 kB 00:00 2024-03-28T18:52:40.880 INFO:teuthology.orchestra.run.smithi027.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 5.2 MB/s | 17 MB 00:03 2024-03-28T18:52:40.952 INFO:teuthology.orchestra.run.smithi155.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-03-28T18:52:40.977 INFO:teuthology.orchestra.run.smithi155.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 161 kB/s | 24 kB 00:00 2024-03-28T18:52:40.989 INFO:teuthology.orchestra.run.smithi027.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.0 MB/s | 220 kB 00:00 2024-03-28T18:52:41.019 INFO:teuthology.orchestra.run.smithi155.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 448 kB/s | 18 kB 00:00 2024-03-28T18:52:41.044 INFO:teuthology.orchestra.run.smithi155.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-28T18:52:41.064 INFO:teuthology.orchestra.run.smithi027.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 365 kB/s | 27 kB 00:00 2024-03-28T18:52:41.070 INFO:teuthology.orchestra.run.smithi155.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 654 kB/s | 32 kB 00:00 2024-03-28T18:52:41.131 INFO:teuthology.orchestra.run.smithi027.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 476 kB/s | 31 kB 00:00 2024-03-28T18:52:41.211 INFO:teuthology.orchestra.run.smithi155.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-28T18:52:41.236 INFO:teuthology.orchestra.run.smithi155.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-28T18:52:41.278 INFO:teuthology.orchestra.run.smithi155.stdout:(66/174): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-28T18:52:41.298 INFO:teuthology.orchestra.run.smithi027.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.4 MB/s | 746 kB 00:00 2024-03-28T18:52:41.304 INFO:teuthology.orchestra.run.smithi155.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-03-28T18:52:41.329 INFO:teuthology.orchestra.run.smithi155.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 854 kB/s | 41 kB 00:00 2024-03-28T18:52:41.363 INFO:teuthology.orchestra.run.smithi155.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.1 MB/s | 238 kB 00:00 2024-03-28T18:52:41.373 INFO:teuthology.orchestra.run.smithi027.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.2 MB/s | 163 kB 00:00 2024-03-28T18:52:41.388 INFO:teuthology.orchestra.run.smithi155.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-28T18:52:41.422 INFO:teuthology.orchestra.run.smithi155.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-28T18:52:41.447 INFO:teuthology.orchestra.run.smithi155.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 844 kB/s | 47 kB 00:00 2024-03-28T18:52:41.448 INFO:teuthology.orchestra.run.smithi027.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 127 kB 00:00 2024-03-28T18:52:41.472 INFO:teuthology.orchestra.run.smithi155.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 873 kB/s | 43 kB 00:00 2024-03-28T18:52:41.574 INFO:teuthology.orchestra.run.smithi027.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.9 MB/s | 501 kB 00:00 2024-03-28T18:52:41.781 INFO:teuthology.orchestra.run.smithi155.stdout:(74/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-28T18:52:41.806 INFO:teuthology.orchestra.run.smithi155.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 121 kB/s | 39 kB 00:00 2024-03-28T18:52:41.832 INFO:teuthology.orchestra.run.smithi155.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 834 kB/s | 41 kB 00:00 2024-03-28T18:52:41.865 INFO:teuthology.orchestra.run.smithi155.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-28T18:52:41.890 INFO:teuthology.orchestra.run.smithi155.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-28T18:52:41.916 INFO:teuthology.orchestra.run.smithi155.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-28T18:52:41.941 INFO:teuthology.orchestra.run.smithi155.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 901 kB/s | 44 kB 00:00 2024-03-28T18:52:41.966 INFO:teuthology.orchestra.run.smithi155.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-28T18:52:42.000 INFO:teuthology.orchestra.run.smithi155.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-28T18:52:42.080 INFO:teuthology.orchestra.run.smithi155.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.1 MB 00:00 2024-03-28T18:52:42.109 INFO:teuthology.orchestra.run.smithi155.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-28T18:52:42.133 INFO:teuthology.orchestra.run.smithi027.stdout:(18/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 3.1 MB/s | 20 MB 00:06 2024-03-28T18:52:42.134 INFO:teuthology.orchestra.run.smithi155.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 124 kB 00:00 2024-03-28T18:52:42.168 INFO:teuthology.orchestra.run.smithi155.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-28T18:52:42.199 INFO:teuthology.orchestra.run.smithi027.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 690 kB/s | 45 kB 00:00 2024-03-28T18:52:42.291 INFO:teuthology.orchestra.run.smithi027.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.4 MB/s | 131 kB 00:00 2024-03-28T18:52:42.358 INFO:teuthology.orchestra.run.smithi027.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-28T18:52:42.360 INFO:teuthology.orchestra.run.smithi155.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 840 kB/s | 187 kB 00:00 2024-03-28T18:52:42.426 INFO:teuthology.orchestra.run.smithi155.stdout:(88/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-03-28T18:52:42.460 INFO:teuthology.orchestra.run.smithi155.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-03-28T18:52:42.494 INFO:teuthology.orchestra.run.smithi155.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-03-28T18:52:42.508 INFO:teuthology.orchestra.run.smithi027.stdout:(22/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 5.5 MB/s | 5.1 MB 00:00 2024-03-28T18:52:42.542 INFO:teuthology.orchestra.run.smithi027.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 321 kB 00:00 2024-03-28T18:52:42.584 INFO:teuthology.orchestra.run.smithi027.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T18:52:42.617 INFO:teuthology.orchestra.run.smithi027.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-28T18:52:42.651 INFO:teuthology.orchestra.run.smithi027.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-28T18:52:42.735 INFO:teuthology.orchestra.run.smithi027.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-28T18:52:42.802 INFO:teuthology.orchestra.run.smithi027.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 516 kB/s | 34 kB 00:00 2024-03-28T18:52:42.885 INFO:teuthology.orchestra.run.smithi027.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.6 MB/s | 139 kB 00:00 2024-03-28T18:52:43.077 INFO:teuthology.orchestra.run.smithi155.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 35 kB/s | 21 kB 00:00 2024-03-28T18:52:43.187 INFO:teuthology.orchestra.run.smithi155.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 37 kB/s | 25 kB 00:00 2024-03-28T18:52:43.286 INFO:teuthology.orchestra.run.smithi027.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 4.5 MB/s | 1.8 MB 00:00 2024-03-28T18:52:43.361 INFO:teuthology.orchestra.run.smithi027.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.2 MB/s | 3.1 MB 00:00 2024-03-28T18:52:43.445 INFO:teuthology.orchestra.run.smithi027.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.9 MB/s | 248 kB 00:00 2024-03-28T18:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 861 kB/s | 49 kB 00:00 2024-03-28T18:52:43.587 INFO:teuthology.orchestra.run.smithi027.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 199 kB/s | 16 kB 00:00 2024-03-28T18:52:43.679 INFO:teuthology.orchestra.run.smithi027.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.8 MB/s | 264 kB 00:00 2024-03-28T18:52:43.871 INFO:teuthology.orchestra.run.smithi027.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.8 MB/s | 748 kB 00:00 2024-03-28T18:52:43.930 INFO:teuthology.orchestra.run.smithi027.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 319 kB/s | 18 kB 00:00 2024-03-28T18:52:44.121 INFO:teuthology.orchestra.run.smithi027.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-28T18:52:44.180 INFO:teuthology.orchestra.run.smithi027.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 920 kB 00:00 2024-03-28T18:52:44.222 INFO:teuthology.orchestra.run.smithi027.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-28T18:52:44.264 INFO:teuthology.orchestra.run.smithi027.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.5 MB/s | 307 kB 00:00 2024-03-28T18:52:44.298 INFO:teuthology.orchestra.run.smithi027.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 979 kB/s | 32 kB 00:00 2024-03-28T18:52:44.439 INFO:teuthology.orchestra.run.smithi027.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-03-28T18:52:44.482 INFO:teuthology.orchestra.run.smithi027.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 410 kB/s | 17 kB 00:00 2024-03-28T18:52:44.528 INFO:teuthology.orchestra.run.smithi155.stdout:(93/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 5.9 MB/s | 48 MB 00:08 2024-03-28T18:52:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:(45/174): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-03-28T18:52:44.632 INFO:teuthology.orchestra.run.smithi027.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 67 kB/s | 10 kB 00:00 2024-03-28T18:52:44.657 INFO:teuthology.orchestra.run.smithi027.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-03-28T18:52:44.691 INFO:teuthology.orchestra.run.smithi027.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 786 kB/s | 44 kB 00:00 2024-03-28T18:52:44.733 INFO:teuthology.orchestra.run.smithi027.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 661 kB 00:00 2024-03-28T18:52:44.775 INFO:teuthology.orchestra.run.smithi027.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.4 MB/s | 578 kB 00:00 2024-03-28T18:52:44.800 INFO:teuthology.orchestra.run.smithi027.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 763 kB/s | 46 kB 00:00 2024-03-28T18:52:44.834 INFO:teuthology.orchestra.run.smithi027.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-28T18:52:45.017 INFO:teuthology.orchestra.run.smithi027.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 875 kB/s | 187 kB 00:00 2024-03-28T18:52:45.042 INFO:teuthology.orchestra.run.smithi027.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-28T18:52:45.068 INFO:teuthology.orchestra.run.smithi027.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 716 kB/s | 34 kB 00:00 2024-03-28T18:52:45.210 INFO:teuthology.orchestra.run.smithi027.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-03-28T18:52:45.252 INFO:teuthology.orchestra.run.smithi027.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-28T18:52:45.327 INFO:teuthology.orchestra.run.smithi027.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 78 kB 00:00 2024-03-28T18:52:45.461 INFO:teuthology.orchestra.run.smithi027.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-28T18:52:45.478 INFO:teuthology.orchestra.run.smithi027.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 182 kB/s | 24 kB 00:00 2024-03-28T18:52:45.512 INFO:teuthology.orchestra.run.smithi027.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 559 kB/s | 18 kB 00:00 2024-03-28T18:52:45.537 INFO:teuthology.orchestra.run.smithi027.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-03-28T18:52:45.562 INFO:teuthology.orchestra.run.smithi027.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 658 kB/s | 32 kB 00:00 2024-03-28T18:52:45.713 INFO:teuthology.orchestra.run.smithi027.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-28T18:52:45.746 INFO:teuthology.orchestra.run.smithi027.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-28T18:52:45.788 INFO:teuthology.orchestra.run.smithi027.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-28T18:52:45.805 INFO:teuthology.orchestra.run.smithi027.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-28T18:52:45.846 INFO:teuthology.orchestra.run.smithi027.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 41 kB 00:00 2024-03-28T18:52:45.872 INFO:teuthology.orchestra.run.smithi027.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 3.6 MB/s | 238 kB 00:00 2024-03-28T18:52:45.914 INFO:teuthology.orchestra.run.smithi027.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-03-28T18:52:45.947 INFO:teuthology.orchestra.run.smithi027.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-03-28T18:52:45.972 INFO:teuthology.orchestra.run.smithi027.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 849 kB/s | 47 kB 00:00 2024-03-28T18:52:45.998 INFO:teuthology.orchestra.run.smithi027.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 874 kB/s | 43 kB 00:00 2024-03-28T18:52:46.153 INFO:teuthology.orchestra.run.smithi155.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 548 kB/s | 1.6 MB 00:02 2024-03-28T18:52:46.362 INFO:teuthology.orchestra.run.smithi155.stdout:(95/174): postgresql-contrib-13.14-1.el9.x86_64 497 kB/s | 911 kB 00:01 2024-03-28T18:52:46.457 INFO:teuthology.orchestra.run.smithi027.stdout:(74/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-28T18:52:46.481 INFO:teuthology.orchestra.run.smithi027.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 84 kB/s | 39 kB 00:00 2024-03-28T18:52:46.523 INFO:teuthology.orchestra.run.smithi027.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 41 kB 00:00 2024-03-28T18:52:46.549 INFO:teuthology.orchestra.run.smithi027.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 300 kB 00:00 2024-03-28T18:52:46.607 INFO:teuthology.orchestra.run.smithi027.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 986 kB/s | 56 kB 00:00 2024-03-28T18:52:46.633 INFO:teuthology.orchestra.run.smithi027.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 763 kB/s | 63 kB 00:00 2024-03-28T18:52:46.674 INFO:teuthology.orchestra.run.smithi027.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-28T18:52:46.691 INFO:teuthology.orchestra.run.smithi027.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-28T18:52:46.750 INFO:teuthology.orchestra.run.smithi027.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-28T18:52:46.792 INFO:teuthology.orchestra.run.smithi027.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-28T18:52:46.834 INFO:teuthology.orchestra.run.smithi027.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 103 kB 00:00 2024-03-28T18:52:46.859 INFO:teuthology.orchestra.run.smithi027.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 1.9 MB/s | 124 kB 00:00 2024-03-28T18:52:47.062 INFO:teuthology.orchestra.run.smithi155.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 147 kB/s | 133 kB 00:00 2024-03-28T18:52:47.392 INFO:teuthology.orchestra.run.smithi027.stdout:(86/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 6.6 MB/s | 48 MB 00:07 2024-03-28T18:52:47.443 INFO:teuthology.orchestra.run.smithi027.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 321 kB/s | 187 kB 00:00 2024-03-28T18:52:48.054 INFO:teuthology.orchestra.run.smithi155.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 248 kB/s | 245 kB 00:00 2024-03-28T18:52:48.399 INFO:teuthology.orchestra.run.smithi027.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 122 kB/s | 187 kB 00:01 2024-03-28T18:52:48.444 INFO:teuthology.orchestra.run.smithi027.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 195 kB/s | 195 kB 00:00 2024-03-28T18:52:48.652 INFO:teuthology.orchestra.run.smithi027.stdout:(90/174): elfutils-libelf-devel-0.190-2.el9.x86 104 kB/s | 21 kB 00:00 2024-03-28T18:52:48.786 INFO:teuthology.orchestra.run.smithi027.stdout:(91/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 407 kB/s | 25 kB 00:00 2024-03-28T18:52:49.036 INFO:teuthology.orchestra.run.smithi027.stdout:(92/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 17 MB/s | 4.1 MB 00:00 2024-03-28T18:52:49.120 INFO:teuthology.orchestra.run.smithi027.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 466 kB/s | 803 kB 00:01 2024-03-28T18:52:49.129 INFO:teuthology.orchestra.run.smithi155.stdout:(98/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 699 kB/s | 4.1 MB 00:06 2024-03-28T18:52:49.154 INFO:teuthology.orchestra.run.smithi155.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 226 kB/s | 248 kB 00:01 2024-03-28T18:52:49.253 INFO:teuthology.orchestra.run.smithi027.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-28T18:52:49.287 INFO:teuthology.orchestra.run.smithi027.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 141 kB/s | 125 kB 00:00 2024-03-28T18:52:49.337 INFO:teuthology.orchestra.run.smithi027.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-03-28T18:52:49.496 INFO:teuthology.orchestra.run.smithi027.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.7 MB/s | 245 kB 00:00 2024-03-28T18:52:49.546 INFO:teuthology.orchestra.run.smithi027.stdout:(98/174): postgresql-contrib-13.14-1.el9.x86_64 2.1 MB/s | 911 kB 00:00 2024-03-28T18:52:49.636 INFO:teuthology.orchestra.run.smithi027.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-03-28T18:52:49.680 INFO:teuthology.orchestra.run.smithi027.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-03-28T18:52:49.722 INFO:teuthology.orchestra.run.smithi027.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 422 kB/s | 36 kB 00:00 2024-03-28T18:52:49.764 INFO:teuthology.orchestra.run.smithi027.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 574 kB/s | 47 kB 00:00 2024-03-28T18:52:49.813 INFO:teuthology.orchestra.run.smithi155.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 55 kB/s | 36 kB 00:00 2024-03-28T18:52:49.814 INFO:teuthology.orchestra.run.smithi027.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.0 MB/s | 94 kB 00:00 2024-03-28T18:52:49.848 INFO:teuthology.orchestra.run.smithi027.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 270 kB/s | 22 kB 00:00 2024-03-28T18:52:50.064 INFO:teuthology.orchestra.run.smithi027.stdout:(105/174): postgresql-server-13.14-1.el9.x86_64 7.5 MB/s | 5.8 MB 00:00 2024-03-28T18:52:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:(101/174): qatlib-24.02.0-1.el9.x86_64.rpm 221 kB/s | 221 kB 00:00 2024-03-28T18:52:50.207 INFO:teuthology.orchestra.run.smithi027.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-03-28T18:52:50.465 INFO:teuthology.orchestra.run.smithi027.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 76 kB/s | 19 kB 00:00 2024-03-28T18:52:50.515 INFO:teuthology.orchestra.run.smithi027.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 688 kB/s | 309 kB 00:00 2024-03-28T18:52:50.521 INFO:teuthology.orchestra.run.smithi155.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 67 kB/s | 47 kB 00:00 2024-03-28T18:52:50.558 INFO:teuthology.orchestra.run.smithi027.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 774 kB/s | 548 kB 00:00 2024-03-28T18:52:50.683 INFO:teuthology.orchestra.run.smithi027.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 162 kB/s | 26 kB 00:00 2024-03-28T18:52:50.733 INFO:teuthology.orchestra.run.smithi027.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 296 kB/s | 49 kB 00:00 2024-03-28T18:52:50.738 INFO:teuthology.orchestra.run.smithi155.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 105 kB/s | 22 kB 00:00 2024-03-28T18:52:50.833 INFO:teuthology.orchestra.run.smithi027.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-28T18:52:50.862 INFO:teuthology.orchestra.run.smithi027.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 548 kB/s | 67 kB 00:00 2024-03-28T18:52:50.890 INFO:teuthology.orchestra.run.smithi027.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-28T18:52:50.947 INFO:teuthology.orchestra.run.smithi155.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.1 MB/s | 863 kB 00:00 2024-03-28T18:52:50.962 INFO:teuthology.orchestra.run.smithi027.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-28T18:52:50.980 INFO:teuthology.orchestra.run.smithi155.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 114 kB/s | 94 kB 00:00 2024-03-28T18:52:51.013 INFO:teuthology.orchestra.run.smithi027.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-28T18:52:51.055 INFO:teuthology.orchestra.run.smithi027.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 276 kB/s | 43 kB 00:00 2024-03-28T18:52:51.097 INFO:teuthology.orchestra.run.smithi027.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 252 kB/s | 32 kB 00:00 2024-03-28T18:52:51.139 INFO:teuthology.orchestra.run.smithi027.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 182 kB/s | 15 kB 00:00 2024-03-28T18:52:51.172 INFO:teuthology.orchestra.run.smithi027.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-28T18:52:51.197 INFO:teuthology.orchestra.run.smithi027.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-28T18:52:51.231 INFO:teuthology.orchestra.run.smithi027.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-28T18:52:51.290 INFO:teuthology.orchestra.run.smithi027.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 401 kB/s | 35 kB 00:00 2024-03-28T18:52:51.332 INFO:teuthology.orchestra.run.smithi027.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.5 MB/s | 203 kB 00:00 2024-03-28T18:52:51.416 INFO:teuthology.orchestra.run.smithi027.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-03-28T18:52:51.441 INFO:teuthology.orchestra.run.smithi027.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-28T18:52:51.491 INFO:teuthology.orchestra.run.smithi027.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 72 kB/s | 11 kB 00:00 2024-03-28T18:52:51.541 INFO:teuthology.orchestra.run.smithi027.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 143 kB/s | 18 kB 00:00 2024-03-28T18:52:51.592 INFO:teuthology.orchestra.run.smithi027.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 234 kB/s | 23 kB 00:00 2024-03-28T18:52:51.642 INFO:teuthology.orchestra.run.smithi027.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 130 kB/s | 19 kB 00:00 2024-03-28T18:52:51.692 INFO:teuthology.orchestra.run.smithi027.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 132 kB/s | 20 kB 00:00 2024-03-28T18:52:51.742 INFO:teuthology.orchestra.run.smithi027.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-03-28T18:52:51.816 INFO:teuthology.orchestra.run.smithi027.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 268 kB/s | 46 kB 00:00 2024-03-28T18:52:51.851 INFO:teuthology.orchestra.run.smithi027.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 537 kB/s | 79 kB 00:00 2024-03-28T18:52:51.864 INFO:teuthology.orchestra.run.smithi155.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 350 kB/s | 309 kB 00:00 2024-03-28T18:52:51.910 INFO:teuthology.orchestra.run.smithi027.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 630 kB/s | 58 kB 00:00 2024-03-28T18:52:51.943 INFO:teuthology.orchestra.run.smithi027.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-28T18:52:51.964 INFO:teuthology.orchestra.run.smithi155.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 196 kB/s | 19 kB 00:00 2024-03-28T18:52:51.993 INFO:teuthology.orchestra.run.smithi027.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 116 kB/s | 16 kB 00:00 2024-03-28T18:52:52.019 INFO:teuthology.orchestra.run.smithi027.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 856 kB/s | 90 kB 00:00 2024-03-28T18:52:52.086 INFO:teuthology.orchestra.run.smithi027.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 314 kB/s | 28 kB 00:00 2024-03-28T18:52:52.106 INFO:teuthology.orchestra.run.smithi155.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 473 kB/s | 548 kB 00:01 2024-03-28T18:52:52.136 INFO:teuthology.orchestra.run.smithi027.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 220 kB/s | 31 kB 00:00 2024-03-28T18:52:52.169 INFO:teuthology.orchestra.run.smithi027.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-03-28T18:52:52.206 INFO:teuthology.orchestra.run.smithi155.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 261 kB/s | 26 kB 00:00 2024-03-28T18:52:52.260 INFO:teuthology.orchestra.run.smithi027.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 487 kB/s | 59 kB 00:00 2024-03-28T18:52:52.312 INFO:teuthology.orchestra.run.smithi027.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 207 kB/s | 36 kB 00:00 2024-03-28T18:52:52.315 INFO:teuthology.orchestra.run.smithi155.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 451 kB/s | 49 kB 00:00 2024-03-28T18:52:52.362 INFO:teuthology.orchestra.run.smithi027.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 266 kB/s | 51 kB 00:00 2024-03-28T18:52:52.395 INFO:teuthology.orchestra.run.smithi027.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-28T18:52:52.421 INFO:teuthology.orchestra.run.smithi027.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 839 kB/s | 90 kB 00:00 2024-03-28T18:52:52.423 INFO:teuthology.orchestra.run.smithi155.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 626 kB/s | 67 kB 00:00 2024-03-28T18:52:52.462 INFO:teuthology.orchestra.run.smithi027.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.3 MB/s | 427 kB 00:00 2024-03-28T18:52:52.521 INFO:teuthology.orchestra.run.smithi027.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 210 kB/s | 20 kB 00:00 2024-03-28T18:52:52.546 INFO:teuthology.orchestra.run.smithi027.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-28T18:52:52.621 INFO:teuthology.orchestra.run.smithi027.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-28T18:52:52.699 INFO:teuthology.orchestra.run.smithi155.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 551 kB/s | 151 kB 00:00 2024-03-28T18:52:52.731 INFO:teuthology.orchestra.run.smithi027.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.9 MB/s | 321 kB 00:00 2024-03-28T18:52:52.939 INFO:teuthology.orchestra.run.smithi027.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 8.3 MB/s | 3.1 MB 00:00 2024-03-28T18:52:53.248 INFO:teuthology.orchestra.run.smithi027.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 4.9 MB/s | 3.4 MB 00:00 2024-03-28T18:52:53.766 INFO:teuthology.orchestra.run.smithi155.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 787 kB/s | 838 kB 00:01 2024-03-28T18:52:54.032 INFO:teuthology.orchestra.run.smithi027.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 59 kB/s | 42 kB 00:00 2024-03-28T18:52:54.508 INFO:teuthology.orchestra.run.smithi155.stdout:(114/174): postgresql-server-13.14-1.el9.x86_64 727 kB/s | 5.8 MB 00:08 2024-03-28T18:52:54.583 INFO:teuthology.orchestra.run.smithi155.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 672 kB/s | 548 kB 00:00 2024-03-28T18:52:54.691 INFO:teuthology.orchestra.run.smithi155.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 298 kB/s | 32 kB 00:00 2024-03-28T18:52:54.800 INFO:teuthology.orchestra.run.smithi155.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 139 kB/s | 15 kB 00:00 2024-03-28T18:52:55.032 INFO:teuthology.orchestra.run.smithi027.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 249 kB/s | 248 kB 00:00 2024-03-28T18:52:55.083 INFO:teuthology.orchestra.run.smithi155.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 607 kB/s | 172 kB 00:00 2024-03-28T18:52:55.150 INFO:teuthology.orchestra.run.smithi155.stdout:(119/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 68 kB/s | 43 kB 00:00 2024-03-28T18:52:55.617 INFO:teuthology.orchestra.run.smithi155.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 650 kB/s | 303 kB 00:00 2024-03-28T18:52:55.651 INFO:teuthology.orchestra.run.smithi155.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 633 kB/s | 358 kB 00:00 2024-03-28T18:52:55.735 INFO:teuthology.orchestra.run.smithi155.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 304 kB/s | 35 kB 00:00 2024-03-28T18:52:55.774 INFO:teuthology.orchestra.run.smithi027.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 97 kB/s | 71 kB 00:00 2024-03-28T18:52:55.943 INFO:teuthology.orchestra.run.smithi155.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 696 kB/s | 203 kB 00:00 2024-03-28T18:52:56.216 INFO:teuthology.orchestra.run.smithi027.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 573 kB/s | 1.8 MB 00:03 2024-03-28T18:52:56.218 INFO:teuthology.orchestra.run.smithi155.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 532 kB/s | 144 kB 00:00 2024-03-28T18:52:56.335 INFO:teuthology.orchestra.run.smithi155.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-03-28T18:52:56.382 INFO:teuthology.orchestra.run.smithi027.stdout:(158/174): NetworkManager-1.47.2-1.el9.x86_64.r 647 kB/s | 2.3 MB 00:03 2024-03-28T18:52:56.458 INFO:teuthology.orchestra.run.smithi027.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 53 kB/s | 36 kB 00:00 2024-03-28T18:52:56.468 INFO:teuthology.orchestra.run.smithi155.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 134 kB/s | 18 kB 00:00 2024-03-28T18:52:56.578 INFO:teuthology.orchestra.run.smithi155.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-03-28T18:52:56.678 INFO:teuthology.orchestra.run.smithi155.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 195 kB/s | 19 kB 00:00 2024-03-28T18:52:56.787 INFO:teuthology.orchestra.run.smithi155.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-03-28T18:52:57.183 INFO:teuthology.orchestra.run.smithi027.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 199 kB/s | 192 kB 00:00 2024-03-28T18:52:57.242 INFO:teuthology.orchestra.run.smithi027.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 32 kB/s | 23 kB 00:00 2024-03-28T18:52:57.559 INFO:teuthology.orchestra.run.smithi027.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 218 kB/s | 256 kB 00:01 2024-03-28T18:52:57.687 INFO:teuthology.orchestra.run.smithi155.stdout:(130/174): libarrow-9.0.0-10.el9.x86_64.rpm 791 kB/s | 4.4 MB 00:05 2024-03-28T18:52:57.796 INFO:teuthology.orchestra.run.smithi155.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 431 kB/s | 46 kB 00:00 2024-03-28T18:52:57.921 INFO:teuthology.orchestra.run.smithi155.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 635 kB/s | 79 kB 00:00 2024-03-28T18:52:58.000 INFO:teuthology.orchestra.run.smithi027.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 133 kB/s | 108 kB 00:00 2024-03-28T18:52:58.029 INFO:teuthology.orchestra.run.smithi155.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 535 kB/s | 58 kB 00:00 2024-03-28T18:52:58.146 INFO:teuthology.orchestra.run.smithi155.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 770 kB/s | 1.0 MB 00:01 2024-03-28T18:52:58.167 INFO:teuthology.orchestra.run.smithi027.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 168 kB/s | 155 kB 00:00 2024-03-28T18:52:58.246 INFO:teuthology.orchestra.run.smithi155.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 165 kB/s | 16 kB 00:00 2024-03-28T18:52:58.355 INFO:teuthology.orchestra.run.smithi155.stdout:(136/174): python3-grpcio-1.46.7-10.el9.x86_64. 796 kB/s | 2.0 MB 00:02 2024-03-28T18:52:58.426 INFO:teuthology.orchestra.run.smithi027.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 157 kB/s | 136 kB 00:00 2024-03-28T18:52:58.455 INFO:teuthology.orchestra.run.smithi155.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 640 kB/s | 272 kB 00:00 2024-03-28T18:52:58.480 INFO:teuthology.orchestra.run.smithi155.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 386 kB/s | 90 kB 00:00 2024-03-28T18:52:58.506 INFO:teuthology.orchestra.run.smithi155.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 188 kB/s | 28 kB 00:00 2024-03-28T18:52:58.573 INFO:teuthology.orchestra.run.smithi155.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 264 kB/s | 31 kB 00:00 2024-03-28T18:52:58.623 INFO:teuthology.orchestra.run.smithi155.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 507 kB/s | 59 kB 00:00 2024-03-28T18:52:58.690 INFO:teuthology.orchestra.run.smithi155.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 307 kB/s | 36 kB 00:00 2024-03-28T18:52:58.732 INFO:teuthology.orchestra.run.smithi155.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 472 kB/s | 51 kB 00:00 2024-03-28T18:52:58.749 INFO:teuthology.orchestra.run.smithi155.stdout:(144/174): python3-routes-2.5.1-5.el9.noarch.rp 705 kB/s | 188 kB 00:00 2024-03-28T18:52:58.794 INFO:teuthology.orchestra.run.smithi027.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 81 kB/s | 64 kB 00:00 2024-03-28T18:52:58.916 INFO:teuthology.orchestra.run.smithi155.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 490 kB/s | 90 kB 00:00 2024-03-28T18:52:58.969 INFO:teuthology.orchestra.run.smithi027.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 36 kB/s | 29 kB 00:00 2024-03-28T18:52:59.041 INFO:teuthology.orchestra.run.smithi155.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2024-03-28T18:52:59.066 INFO:teuthology.orchestra.run.smithi155.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 611 kB/s | 230 kB 00:00 2024-03-28T18:52:59.308 INFO:teuthology.orchestra.run.smithi155.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 720 kB/s | 192 kB 00:00 2024-03-28T18:52:59.367 INFO:teuthology.orchestra.run.smithi155.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 692 kB/s | 427 kB 00:00 2024-03-28T18:52:59.786 INFO:teuthology.orchestra.run.smithi155.stdout:(150/174): librados2-19.0.0-2145.g3be9a5c9.el9. 7.1 MB/s | 3.4 MB 00:00 2024-03-28T18:52:59.828 INFO:teuthology.orchestra.run.smithi155.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-28T18:52:59.945 INFO:teuthology.orchestra.run.smithi155.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 5.4 MB/s | 3.1 MB 00:00 2024-03-28T18:53:00.220 INFO:teuthology.orchestra.run.smithi155.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 5.9 MB/s | 2.3 MB 00:00 2024-03-28T18:53:00.279 INFO:teuthology.orchestra.run.smithi155.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 728 kB/s | 42 kB 00:00 2024-03-28T18:53:00.363 INFO:teuthology.orchestra.run.smithi155.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-28T18:53:00.413 INFO:teuthology.orchestra.run.smithi155.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-28T18:53:00.463 INFO:teuthology.orchestra.run.smithi155.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 733 kB/s | 36 kB 00:00 2024-03-28T18:53:00.513 INFO:teuthology.orchestra.run.smithi155.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2024-03-28T18:53:00.677 INFO:teuthology.orchestra.run.smithi027.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 551 kB/s | 1.2 MB 00:02 2024-03-28T18:53:00.705 INFO:teuthology.orchestra.run.smithi155.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.3 MB/s | 256 kB 00:00 2024-03-28T18:53:00.748 INFO:teuthology.orchestra.run.smithi155.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 560 kB/s | 23 kB 00:00 2024-03-28T18:53:00.839 INFO:teuthology.orchestra.run.smithi155.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 108 kB 00:00 2024-03-28T18:53:00.898 INFO:teuthology.orchestra.run.smithi155.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-03-28T18:53:00.957 INFO:teuthology.orchestra.run.smithi155.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-28T18:53:01.299 INFO:teuthology.orchestra.run.smithi155.stdout:(164/174): thrift-0.15.0-2.el9.x86_64.rpm 726 kB/s | 1.6 MB 00:02 2024-03-28T18:53:01.349 INFO:teuthology.orchestra.run.smithi155.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 163 kB/s | 64 kB 00:00 2024-03-28T18:53:01.416 INFO:teuthology.orchestra.run.smithi155.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 247 kB/s | 29 kB 00:00 2024-03-28T18:53:01.650 INFO:teuthology.orchestra.run.smithi155.stdout:(167/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.1 MB/s | 1.8 MB 00:01 2024-03-28T18:53:01.700 INFO:teuthology.orchestra.run.smithi155.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 3.5 MB/s | 1.2 MB 00:00 2024-03-28T18:53:01.758 INFO:teuthology.orchestra.run.smithi155.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 2.2 MB 00:00 2024-03-28T18:53:01.809 INFO:teuthology.orchestra.run.smithi155.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 632 kB/s | 31 kB 00:00 2024-03-28T18:53:01.842 INFO:teuthology.orchestra.run.smithi155.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-03-28T18:53:01.936 INFO:teuthology.orchestra.run.smithi027.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 370 kB/s | 465 kB 00:01 2024-03-28T18:53:01.942 INFO:teuthology.orchestra.run.smithi155.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.8 MB/s | 2.3 MB 00:00 2024-03-28T18:53:01.984 INFO:teuthology.orchestra.run.smithi155.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.2 MB/s | 1.1 MB 00:00 2024-03-28T18:53:02.286 INFO:teuthology.orchestra.run.smithi027.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 632 kB/s | 2.2 MB 00:03 2024-03-28T18:53:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 651 kB/s | 2.3 MB 00:03 2024-03-28T18:53:02.609 INFO:teuthology.orchestra.run.smithi155.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 97 kB/s | 74 kB 00:00 2024-03-28T18:53:02.626 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:53:02.626 INFO:teuthology.orchestra.run.smithi155.stdout:Total 7.7 MB/s | 225 MB 00:29 2024-03-28T18:53:02.661 INFO:teuthology.orchestra.run.smithi027.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 43 kB/s | 31 kB 00:00 2024-03-28T18:53:02.895 INFO:teuthology.orchestra.run.smithi027.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 282 kB/s | 74 kB 00:00 2024-03-28T18:53:04.263 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-28T18:53:04.412 INFO:teuthology.orchestra.run.smithi027.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 517 kB/s | 1.1 MB 00:02 2024-03-28T18:53:04.424 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:53:04.425 INFO:teuthology.orchestra.run.smithi027.stdout:Total 7.5 MB/s | 225 MB 00:29 2024-03-28T18:53:04.642 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-28T18:53:04.642 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-28T18:53:06.113 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-03-28T18:53:06.513 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-03-28T18:53:06.513 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-03-28T18:53:07.040 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-28T18:53:07.041 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-28T18:53:08.990 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-03-28T18:53:08.990 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-03-28T18:53:14.274 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-28T18:53:14.400 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T18:53:14.448 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T18:53:14.495 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:53:14.541 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:53:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T18:53:14.734 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T18:53:14.808 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T18:53:14.843 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T18:53:14.932 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T18:53:15.342 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T18:53:15.391 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:53:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:53:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:53:15.662 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:53:15.709 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:53:15.799 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T18:53:16.098 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T18:53:16.156 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T18:53:16.180 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:53:16.334 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:53:16.373 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:53:16.463 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:53:16.509 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T18:53:16.542 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:53:16.609 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:53:16.642 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-03-28T18:53:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T18:53:16.810 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T18:53:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:53:16.924 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:53:16.934 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:53:16.983 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:53:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T18:53:17.025 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T18:53:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T18:53:17.183 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T18:53:17.213 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T18:53:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T18:53:17.362 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T18:53:17.515 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:53:17.605 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:53:17.729 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T18:53:17.786 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:53:17.795 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:53:17.860 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:53:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:53:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:53:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:53:18.190 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:53:18.222 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:53:18.279 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T18:53:18.373 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T18:53:18.507 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T18:53:18.541 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T18:53:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T18:53:18.578 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T18:53:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T18:53:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:53:18.665 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T18:53:18.775 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T18:53:18.788 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:53:18.812 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:53:18.839 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T18:53:18.882 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T18:53:18.944 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:53:18.971 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T18:53:18.992 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T18:53:19.012 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T18:53:19.059 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:53:19.073 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T18:53:19.117 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T18:53:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:53:19.149 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T18:53:19.262 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T18:53:19.304 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T18:53:19.355 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T18:53:19.385 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T18:53:19.421 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:53:19.482 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:53:19.502 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T18:53:19.522 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T18:53:19.563 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T18:53:19.676 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T18:53:19.848 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T18:53:19.852 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T18:53:19.917 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T18:53:20.038 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:53:20.127 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:53:20.348 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:53:20.415 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:53:20.565 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T18:53:20.664 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T18:53:20.752 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:53:20.783 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T18:53:20.842 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:53:20.869 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:53:20.923 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T18:53:20.966 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T18:53:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T18:53:21.041 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T18:53:21.082 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T18:53:21.101 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T18:53:21.163 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T18:53:21.200 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T18:53:21.263 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T18:53:21.389 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T18:53:21.457 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T18:53:21.482 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T18:53:21.506 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T18:53:21.621 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T18:53:21.670 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T18:53:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T18:53:21.734 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T18:53:21.784 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T18:53:21.813 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T18:53:21.823 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T18:53:21.872 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T18:53:21.945 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T18:53:21.985 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T18:53:22.078 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T18:53:22.079 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T18:53:22.109 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T18:53:22.143 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T18:53:22.213 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T18:53:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T18:53:22.269 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T18:53:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T18:53:22.359 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T18:53:22.374 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T18:53:22.399 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T18:53:22.468 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T18:53:22.516 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T18:53:22.552 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T18:53:22.579 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T18:53:22.623 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T18:53:22.646 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T18:53:22.834 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T18:53:22.889 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T18:53:22.927 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T18:53:22.984 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T18:53:23.113 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T18:53:23.150 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T18:53:23.195 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T18:53:23.225 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T18:53:23.257 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T18:53:23.333 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T18:53:23.403 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T18:53:23.439 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T18:53:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T18:53:23.580 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T18:53:23.612 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:53:23.641 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:53:23.684 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T18:53:23.714 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T18:53:23.775 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T18:53:23.777 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T18:53:23.838 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T18:53:23.912 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T18:53:23.913 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T18:53:23.955 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T18:53:24.154 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T18:53:24.214 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T18:53:24.379 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T18:53:24.383 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T18:53:24.437 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T18:53:24.477 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T18:53:24.515 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T18:53:24.556 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T18:53:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T18:53:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T18:53:24.677 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T18:53:24.700 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T18:53:24.730 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T18:53:24.764 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T18:53:24.777 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T18:53:24.904 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T18:53:24.944 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T18:53:24.977 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T18:53:24.978 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T18:53:25.031 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T18:53:25.043 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T18:53:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T18:53:25.131 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T18:53:25.217 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T18:53:25.232 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T18:53:25.271 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T18:53:25.310 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T18:53:25.330 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T18:53:25.382 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T18:53:25.387 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T18:53:25.423 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T18:53:25.427 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T18:53:25.479 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T18:53:25.520 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T18:53:25.589 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T18:53:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T18:53:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T18:53:25.793 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T18:53:25.851 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T18:53:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T18:53:25.987 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T18:53:26.141 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T18:53:26.185 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T18:53:26.232 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T18:53:26.277 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T18:53:26.335 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T18:53:26.511 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T18:53:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T18:53:26.691 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T18:53:26.697 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T18:53:26.730 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T18:53:26.788 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T18:53:26.855 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T18:53:26.898 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T18:53:26.921 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T18:53:26.948 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T18:53:26.986 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T18:53:27.060 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T18:53:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T18:53:27.211 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T18:53:27.322 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T18:53:27.332 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T18:53:27.384 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T18:53:27.560 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T18:53:27.625 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T18:53:27.664 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T18:53:27.710 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T18:53:27.752 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T18:53:27.798 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T18:53:27.866 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T18:53:27.925 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T18:53:27.972 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T18:53:28.120 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T18:53:28.135 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T18:53:28.182 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T18:53:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T18:53:28.593 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T18:53:28.674 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T18:53:28.720 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T18:53:28.896 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T18:53:29.022 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T18:53:29.077 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T18:53:29.113 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T18:53:29.171 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T18:53:29.315 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T18:53:29.388 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T18:53:29.753 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T18:53:29.842 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T18:53:29.880 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T18:53:29.911 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T18:53:29.951 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T18:53:30.322 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T18:53:30.349 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:53:30.415 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T18:53:30.460 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:53:30.470 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T18:53:30.488 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T18:53:30.503 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T18:53:30.541 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T18:53:30.562 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T18:53:30.706 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T18:53:30.745 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T18:53:31.000 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T18:53:31.042 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T18:53:31.051 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T18:53:31.096 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:53:31.165 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T18:53:31.193 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:53:31.221 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T18:53:32.023 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T18:53:32.176 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T18:53:33.788 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T18:53:33.811 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:53:33.851 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:53:33.872 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T18:53:33.917 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T18:53:33.947 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T18:53:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T18:53:34.423 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T18:53:34.462 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:53:34.474 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:53:34.513 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:53:34.585 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:53:34.619 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T18:53:34.677 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T18:53:34.684 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:53:34.684 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T18:53:34.684 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T18:53:34.684 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:34.722 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:53:35.243 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T18:53:35.273 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:53:35.374 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:53:35.400 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T18:53:38.081 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:53:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:53:38.796 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:53:38.843 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:53:39.023 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:53:39.024 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T18:53:39.024 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T18:53:39.024 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:53:39.078 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:53:47.567 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:53:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-03-28T18:53:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-03-28T18:53:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-03-28T18:53:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-03-28T18:53:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-03-28T18:53:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-03-28T18:53:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-03-28T18:53:47.568 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:47.806 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:53:47.854 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:53:47.854 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:47.854 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T18:53:47.855 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:49.456 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:53:49.533 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:53:49.533 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:49.533 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T18:53:49.533 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:53:49.533 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:53:49.533 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:49.781 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:53:49.849 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:53:49.849 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:49.850 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T18:53:49.850 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:53:49.850 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:53:49.850 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:50.332 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:53:50.405 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:53:50.405 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:50.406 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T18:53:50.406 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:53:50.406 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:53:50.406 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:50.526 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T18:53:51.112 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:53:51.144 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:53:51.196 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T18:53:51.294 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T18:53:51.322 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:53:51.654 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:53:51.698 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-03-28T18:53:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:53:52.819 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:53:52.849 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:53:52.866 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:53:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:53:52.920 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:52.920 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T18:53:52.920 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:53:52.989 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:53:53.170 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T18:53:53.209 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:53:53.277 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:53:53.277 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:53.277 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T18:53:53.277 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:53:53.277 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:53:53.277 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:53.335 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:53:53.374 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:53:53.465 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T18:53:54.423 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T18:53:54.454 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:53:54.531 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:53:54.531 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:54.531 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T18:53:54.531 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:53:54.531 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:53:54.531 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:54.588 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:53:54.602 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:53:54.667 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:53:54.667 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:54.667 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T18:53:54.667 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:54.690 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:53:54.690 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:54.690 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T18:53:54.690 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:53:54.690 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:53:54.690 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:53:54.739 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:53:54.835 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:53:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T18:53:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:53:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:53:54.836 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:54.960 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:53:55.045 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:53:55.045 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:55.045 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T18:53:55.045 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:53:55.045 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:53:55.045 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:53:55.206 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:53:55.281 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:53:55.281 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:55.281 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T18:53:55.281 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:53:55.281 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:53:55.282 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:53:55.578 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:53:55.657 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:53:55.657 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:55.657 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T18:53:55.658 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:53:55.658 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:53:55.658 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:53:55.780 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T18:53:56.440 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:53:56.481 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:53:56.534 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T18:53:56.657 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T18:53:56.692 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:53:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:53:57.077 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:53:58.258 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:53:58.311 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:53:58.470 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:53:58.658 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T18:53:58.718 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:53:58.795 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:53:58.795 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:53:58.795 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T18:53:58.795 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:53:58.795 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:53:58.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:53:58.855 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:53:58.897 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:53:59.015 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T18:54:00.063 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T18:54:00.098 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:54:00.098 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:54:00.161 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T18:54:00.172 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:54:00.172 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:54:00.172 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T18:54:00.172 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:54:00.173 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:54:00.173 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:00.194 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T18:54:00.216 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:54:00.233 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T18:54:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:54:00.290 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:54:00.290 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T18:54:00.290 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:00.367 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:54:00.458 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:54:00.458 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:54:00.459 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T18:54:00.459 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:54:00.459 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:54:00.459 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:00.516 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T18:54:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T18:54:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T18:54:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T18:54:00.803 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T18:54:00.850 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T18:54:00.878 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:54:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T18:54:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T18:54:00.954 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:54:00.954 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:54:00.954 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T18:54:00.954 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:54:00.954 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:54:00.954 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T18:54:01.077 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T18:54:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T18:54:01.213 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T18:54:02.031 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:54:02.077 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:54:02.139 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:54:03.481 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T18:54:03.576 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T18:54:03.623 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T18:54:03.752 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T18:54:03.786 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T18:54:03.820 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T18:54:03.862 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T18:54:03.903 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T18:54:03.903 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:54:04.033 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:54:04.067 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T18:54:04.067 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:54:04.197 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:54:04.266 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T18:54:04.321 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T18:54:04.354 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T18:54:04.386 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T18:54:04.454 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T18:54:04.533 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T18:54:04.575 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T18:54:04.608 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T18:54:04.642 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T18:54:04.675 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:54:04.708 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T18:54:04.745 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:54:04.745 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:54:05.094 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:54:05.094 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:54:05.240 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:54:05.270 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:54:05.324 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T18:54:05.356 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T18:54:05.405 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T18:54:05.524 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:54:05.771 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:54:05.771 INFO:teuthology.orchestra.run.smithi155.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T18:54:05.771 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:06.078 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:54:06.139 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T18:54:06.173 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T18:54:06.211 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T18:54:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T18:54:06.639 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T18:54:06.691 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T18:54:06.849 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T18:54:06.919 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T18:54:06.971 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T18:54:07.043 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T18:54:07.095 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T18:54:07.206 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T18:54:07.350 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T18:54:07.419 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:54:08.606 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:54:08.660 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:54:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T18:54:10.129 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T18:54:10.179 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T18:54:10.326 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T18:54:10.396 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T18:54:10.431 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T18:54:10.466 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T18:54:10.503 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T18:54:10.503 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T18:54:10.503 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T18:54:10.503 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T18:54:10.505 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T18:54:10.506 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T18:54:10.506 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T18:54:10.506 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T18:54:10.514 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T18:54:10.514 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:54:10.528 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T18:54:10.528 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T18:54:10.529 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T18:54:10.530 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T18:54:10.531 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T18:54:10.532 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T18:54:10.533 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T18:54:10.534 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T18:54:10.535 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T18:54:10.536 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T18:54:10.537 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T18:54:10.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T18:54:10.539 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T18:54:10.540 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T18:54:10.541 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T18:54:10.542 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T18:54:10.543 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T18:54:10.544 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T18:54:10.655 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:54:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T18:54:10.703 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:54:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:54:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T18:54:11.088 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T18:54:11.122 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T18:54:11.155 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T18:54:11.236 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T18:54:11.330 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T18:54:11.376 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T18:54:11.410 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T18:54:11.444 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T18:54:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:54:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T18:54:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:54:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:54:11.895 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:54:11.895 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi027.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:13.092 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T18:54:13.092 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:13.092 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-03-28T18:54:13.092 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T18:54:13.092 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T18:54:13.092 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T18:54:13.092 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T18:54:13.093 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.094 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.095 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T18:54:13.096 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.6-16.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.097 INFO:teuthology.orchestra.run.smithi155.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T18:54:13.098 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T18:54:13.099 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T18:54:13.100 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T18:54:13.101 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T18:54:13.102 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T18:54:13.103 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T18:54:13.104 INFO:teuthology.orchestra.run.smithi155.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T18:54:13.104 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T18:54:13.104 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T18:54:13.104 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T18:54:13.104 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:13.104 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-28T18:54:13.825 DEBUG:teuthology.parallel:result is None 2024-03-28T18:54:17.665 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T18:54:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T18:54:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T18:54:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T18:54:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T18:54:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T18:54:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T18:54:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T18:54:17.666 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T18:54:17.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T18:54:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T18:54:17.669 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T18:54:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T18:54:17.671 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T18:54:17.672 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T18:54:17.673 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T18:54:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T18:54:17.675 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T18:54:17.676 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T18:54:17.677 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T18:54:17.678 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T18:54:17.679 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T18:54:17.680 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T18:54:17.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T18:54:17.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T18:54:20.284 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T18:54:20.284 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:20.284 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-03-28T18:54:20.284 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T18:54:20.284 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T18:54:20.284 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T18:54:20.284 INFO:teuthology.orchestra.run.smithi027.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T18:54:20.284 INFO:teuthology.orchestra.run.smithi027.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T18:54:20.285 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.286 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.287 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T18:54:20.288 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-16.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T18:54:20.289 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T18:54:20.290 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T18:54:20.291 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T18:54:20.292 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T18:54:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T18:54:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T18:54:20.295 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T18:54:20.296 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T18:54:20.297 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:20.297 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-03-28T18:54:20.958 DEBUG:teuthology.parallel:result is None 2024-03-28T18:54:20.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:54:21.178 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T18:54:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:54:21.240 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:54:21.240 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T18:54:21.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:54:21.459 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T18:54:21.530 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:54:21.531 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:54:21.531 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T18:54:21.532 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T18:54:21.532 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:21.533 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T18:54:21.622 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:54:21.623 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T18:54:21.728 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T18:54:21.728 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:21.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T18:54:21.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T18:54:21.931 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:54:21.931 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T18:54:22.029 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T18:54:22.164 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T18:54:22.164 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:22.164 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T18:54:22.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T18:54:22.366 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:54:22.366 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T18:54:22.459 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T18:54:22.598 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T18:54:22.598 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:22.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T18:54:22.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T18:54:22.783 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:54:22.783 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T18:54:22.883 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T18:54:22.976 INFO:teuthology.run_tasks:Running task ceph... 2024-03-28T18:54:23.063 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-28T18:54:23.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2024-03-28T18:54:23.070 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2024-03-28T18:54:23.168 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-28T18:54:23.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T18:54:23.172 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T18:54:23.269 INFO:tasks.ceph:Creating extra log directories... 2024-03-28T18:54:23.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T18:54:23.273 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T18:54:23.376 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-28T18:54:23.376 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-28T18:54:23.376 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-28T18:54:23.376 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:23.376 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-28T18:54:23.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-28T18:54:23.650 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-28T18:54:23.728 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:54:23.728 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-28T18:54:23.782 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-28T18:54:24.049 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-28T18:54:24.152 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-28T18:54:24.152 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-28T18:54:24.153 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625389', 'branch': 'wip-mchangir-testing-main-20240318.032620', 'description': 'fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7625389', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-testing-main-20240318.032620'}, 'ceph': {'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'kclient': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mchangir-testing-main-20240318.032620', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'owner': 'mchangir', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 3645, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'sleep_before_teardown': 0, 'subset': '132/256', 'suite': 'fs', 'suite_branch': 'wip-mchangir-testing-main-20240318.032620', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpyBMUwGrjTQ7aZWPwN83VIoQIjtIAHFM/eoBRSfAydbDVy717FR+I+Z/Bes6ieeXc3y+6w+7Un8hrvLAE/pKM=', 'smithi155.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDBxvKut37D7jeWMtpGEFyG8+HC+Kayd8PIovHDyqxJsTbKAefpdTobI8Th5G8HA/sVPrJJdz6g8m64MvjuavA8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-27_03:45:23', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-28T18:54:24.154 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T18:54:24.160 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T18:54:24.214 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T18:54:24.218 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T18:54:24.304 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:24.305 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:54:24.345 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:54:24.346 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-03-28T18:54:24.428 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:54:24.428 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:54:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-28T18:54:24.429 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:54:24.429 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:54:24.429 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-28 18:54:13.176913226 +0000 2024-03-28T18:54:24.429 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-28 18:51:15.969819111 +0000 2024-03-28T18:54:24.429 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-28 18:51:15.969819111 +0000 2024-03-28T18:54:24.429 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-28 18:51:15.969819111 +0000 2024-03-28T18:54:24.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:54:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-28T18:54:24.614 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-28T18:54:24.614 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000336313 s, 1.5 MB/s 2024-03-28T18:54:24.616 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:54:24.670 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-03-28T18:54:24.710 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:54:24.710 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:54:24.711 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-03-28T18:54:24.711 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:54:24.711 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:54:24.711 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-28 18:54:13.177913200 +0000 2024-03-28T18:54:24.711 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-28 18:51:16.568803806 +0000 2024-03-28T18:54:24.711 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-28 18:51:16.568803806 +0000 2024-03-28T18:54:24.711 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-28 18:51:16.568803806 +0000 2024-03-28T18:54:24.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:54:24.828 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-28T18:54:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-28T18:54:24.829 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00036675 s, 1.4 MB/s 2024-03-28T18:54:24.831 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:54:24.883 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-03-28T18:54:24.928 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:54:24.928 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:54:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-28T18:54:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:54:24.928 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:54:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-28 18:54:13.177913200 +0000 2024-03-28T18:54:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-28 18:51:17.185788041 +0000 2024-03-28T18:54:24.929 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-28 18:51:17.185788041 +0000 2024-03-28T18:54:24.929 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-28 18:51:17.185788041 +0000 2024-03-28T18:54:24.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:54:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-28T18:54:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-28T18:54:25.047 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000295276 s, 1.7 MB/s 2024-03-28T18:54:25.050 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:54:25.094 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-03-28T18:54:25.176 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:54:25.176 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:54:25.176 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-03-28T18:54:25.176 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:54:25.176 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:54:25.176 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-03-28 18:54:13.178913175 +0000 2024-03-28T18:54:25.176 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-28 18:51:17.802772278 +0000 2024-03-28T18:54:25.176 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-28 18:51:17.802772278 +0000 2024-03-28T18:54:25.177 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-03-28 18:51:17.802772278 +0000 2024-03-28T18:54:25.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:54:25.304 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-03-28T18:54:25.304 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-03-28T18:54:25.304 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000302749 s, 1.7 MB/s 2024-03-28T18:54:25.307 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:54:25.352 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:25.352 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:54:25.352 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:54:25.410 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:54:25.410 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-03-28T18:54:25.462 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:54:25.462 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:54:25.463 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-28T18:54:25.463 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:54:25.463 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:54:25.463 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-28 18:54:06.252215326 +0000 2024-03-28T18:54:25.463 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-28 18:51:08.827888799 +0000 2024-03-28T18:54:25.463 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-28 18:51:08.827888799 +0000 2024-03-28T18:54:25.463 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-28 18:51:08.827888799 +0000 2024-03-28T18:54:25.464 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:54:25.549 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-28T18:54:25.549 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-28T18:54:25.550 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000468612 s, 1.1 MB/s 2024-03-28T18:54:25.554 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:54:25.608 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-03-28T18:54:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:54:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:54:25.659 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-03-28T18:54:25.659 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:54:25.659 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:54:25.660 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-28 18:54:06.253215306 +0000 2024-03-28T18:54:25.660 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-28 18:51:09.396877194 +0000 2024-03-28T18:54:25.660 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-28 18:51:09.396877194 +0000 2024-03-28T18:54:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-28 18:51:09.396877194 +0000 2024-03-28T18:54:25.660 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:54:25.751 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-28T18:54:25.751 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-28T18:54:25.751 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000310087 s, 1.7 MB/s 2024-03-28T18:54:25.755 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:54:25.811 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-03-28T18:54:25.860 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:54:25.860 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:54:25.860 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-28T18:54:25.861 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:54:25.861 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:54:25.861 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-28 18:54:06.253215306 +0000 2024-03-28T18:54:25.861 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-28 18:51:09.939866120 +0000 2024-03-28T18:54:25.861 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-28 18:51:09.939866120 +0000 2024-03-28T18:54:25.861 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-28 18:51:09.939866120 +0000 2024-03-28T18:54:25.861 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:54:25.950 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-28T18:54:25.950 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-28T18:54:25.950 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000336604 s, 1.5 MB/s 2024-03-28T18:54:25.954 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:54:26.010 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-28 18:54:06.254215285 +0000 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-28 18:51:10.519854289 +0000 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-28 18:51:10.520854269 +0000 2024-03-28T18:54:26.057 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-28 18:51:10.519854289 +0000 2024-03-28T18:54:26.057 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:54:26.137 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-28T18:54:26.137 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-28T18:54:26.137 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000312953 s, 1.6 MB/s 2024-03-28T18:54:26.141 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:54:26.196 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:26.197 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T18:54:26.197 INFO:tasks.ceph:Generating config... 2024-03-28T18:54:26.198 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-28T18:54:26.198 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-03-28T18:54:26.198 INFO:tasks.ceph:[client] debug client = 20 2024-03-28T18:54:26.198 INFO:tasks.ceph:[client] debug ms = 1 2024-03-28T18:54:26.198 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-28T18:54:26.198 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-28T18:54:26.198 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-28T18:54:26.198 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-28T18:54:26.199 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-28T18:54:26.200 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-28T18:54:26.200 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-28T18:54:26.200 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-28T18:54:26.200 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-28T18:54:26.201 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-28T18:54:26.201 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-28T18:54:26.201 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-28T18:54:26.201 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-28T18:54:26.201 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-28T18:54:26.201 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-28T18:54:26.201 INFO:tasks.ceph:Setting up mon.a... 2024-03-28T18:54:26.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-28T18:54:26.398 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2024-03-28T18:54:26.408 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-28T18:54:26.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T18:54:26.660 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27'), ('mon.b', '172.21.15.155'), ('mon.c', '[v2:172.21.15.155:3301,v1:172.21.15.155:6790]')] 2024-03-28T18:54:26.661 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27,172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-28T18:54:26.662 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:26.662 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T18:54:26.718 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --add b 172.21.15.155 --addv c '[v2:172.21.15.155:3301,v1:172.21.15.155:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:26.870 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:26.870 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid c27ecc06-ff19-45f9-bdba-d624e1db1fbd 2024-03-28T18:54:26.870 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-03-28T18:54:26.870 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2024-03-28T18:54:26.870 INFO:teuthology.orchestra.run.smithi027.stdout:fsid c27ecc06-ff19-45f9-bdba-d624e1db1fbd 2024-03-28T18:54:26.871 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2024-03-28T18:54:26.871075+0000 2024-03-28T18:54:26.871 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-03-28T18:54:26.871075+0000 2024-03-28T18:54:26.871 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 17 (quincy) 2024-03-28T18:54:26.871 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2024-03-28T18:54:26.871 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-03-28T18:54:26.871 INFO:teuthology.orchestra.run.smithi027.stdout:1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.b 2024-03-28T18:54:26.871 INFO:teuthology.orchestra.run.smithi027.stdout:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-03-28T18:54:26.871 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-28T18:54:26.873 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T18:54:26.928 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c27ecc06-ff19-45f9-bdba-d624e1db1fbd... 2024-03-28T18:54:26.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T18:54:26.934 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T18:54:27.105 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2024-03-28T18:54:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2024-03-28T18:54:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2024-03-28T18:54:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2024-03-28T18:54:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = "172.21.15.27,172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790]" 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = c27ecc06-ff19-45f9-bdba-d624e1db1fbd 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 600 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock profile = high_recovery_ops 2024-03-28T18:54:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug trim objects = true 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: bdev async discard = True 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: bdev enable discard = True 2024-03-28T18:54:27.111 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore allocator = bitmap 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore block size = 96636764160 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore fsck on mount = True 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluefs = 1/20 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluestore = 1/20 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 20 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: debug rocksdb = 4/10 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd full ratio = 0.9 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd nearfull ratio = 0.8 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: osd blocked scrub grace period = 3600 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: osd failsafe full ratio = 0.95 2024-03-28T18:54:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: osd max markdown count = 1000 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: osd objectstore = bluestore 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: osd op complaint time = 180 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2024-03-28T18:54:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T18:54:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: # 1m isn't quite enough 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: mon_down_mkfs_grace = 2m 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_filestore_osds = false 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: mon op complaint time = 120 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2024-03-28T18:54:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: client mount timeout = 600 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: client_shutdown_timeout = 120 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: debug client = 20 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: rados mon op timeout = 900 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: rados osd op timeout = 900 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.b] 2024-03-28T18:54:27.116 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.c] 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.a] 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.c] 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.b] 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.d] 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout:[mds] 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: debug mds = 20 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: debug mds balancer = 20 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal fragment size max = 10000 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal merge size = 5 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal split bits = 3 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal split size = 100 2024-03-28T18:54:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: mds debug frag = True 2024-03-28T18:54:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: mds debug scatterstat = True 2024-03-28T18:54:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: mds op complaint time = 180 2024-03-28T18:54:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: mds verify scatter = True 2024-03-28T18:54:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: osd op complaint time = 180 2024-03-28T18:54:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: rados mon op timeout = 900 2024-03-28T18:54:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: rados osd op timeout = 900 2024-03-28T18:54:27.118 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-03-28T18:54:27.118 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-03-28T18:54:27.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-03-28T18:54:27.120 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-03-28T18:54:27.121 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790]" 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = c27ecc06-ff19-45f9-bdba-d624e1db1fbd 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-03-28T18:54:27.122 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.123 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-03-28T18:54:27.124 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: osd blocked scrub grace period = 3600 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: osd max markdown count = 1000 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-28T18:54:27.125 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-03-28T18:54:27.126 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.127 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-03-28T18:54:27.128 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout: client_shutdown_timeout = 120 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-28T18:54:27.129 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: mds bal fragment size max = 10000 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: mds bal merge size = 5 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: mds bal split bits = 3 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: mds bal split size = 100 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-03-28T18:54:27.130 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-03-28T18:54:27.131 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-03-28T18:54:27.131 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-03-28T18:54:27.131 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-03-28T18:54:27.131 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-03-28T18:54:27.171 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-28T18:54:27.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-28T18:54:27.329 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-28T18:54:27.329 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:27.329 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-28T18:54:27.380 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:27.380 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-28T18:54:27.430 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-03-28T18:54:27.430 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:27.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T18:54:27.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T18:54:27.579 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:54:27.579 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:27.633 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2024-03-28T18:54:27.633 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:54:27.633 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T18:54:27.633 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T18:54:27.786 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:54:27.786 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:27.881 INFO:tasks.ceph:Setting up mon nodes... 2024-03-28T18:54:27.881 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-28T18:54:27.881 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T18:54:28.092 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T18:54:28.102 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T18:54:28.297 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T18:54:28.309 INFO:tasks.ceph:Setting up mds nodes... 2024-03-28T18:54:28.309 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-28T18:54:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-28T18:54:28.532 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-28T18:54:28.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-28T18:54:28.849 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-28T18:54:28.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-28T18:54:28.969 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-28T18:54:29.152 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-28T18:54:29.161 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-28T18:54:29.258 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-28T18:54:29.439 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-28T18:54:29.448 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-28T18:54:29.541 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-28T18:54:29.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T18:54:29.672 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-28T18:54:29.741 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-28T18:54:29.872 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-28T18:54:29.936 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-28T18:54:29.936 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T18:54:29.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-28T18:54:30.043 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:30.043 INFO:tasks.ceph:role: osd.0 2024-03-28T18:54:30.043 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-28T18:54:30.043 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T18:54:30.153 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T18:54:30.153 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T18:54:30.153 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T18:54:30.153 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T18:54:30.153 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T18:54:30.154 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-28T18:54:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T18:54:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T18:54:30.154 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T18:54:30.154 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T18:54:30.677 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-28T18:54:30.685 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-28T18:54:30.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-28T18:54:30.881 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-28T18:54:31.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-28T18:54:31.098 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:31.098 INFO:tasks.ceph:role: osd.1 2024-03-28T18:54:31.098 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-28T18:54:31.098 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T18:54:31.189 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T18:54:31.189 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T18:54:31.189 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T18:54:31.189 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T18:54:31.189 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T18:54:31.189 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-28T18:54:31.189 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T18:54:31.190 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T18:54:31.190 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T18:54:31.190 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T18:54:31.675 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-28T18:54:31.682 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-28T18:54:31.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-28T18:54:31.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-28T18:54:31.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-28T18:54:32.000 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:32.000 INFO:tasks.ceph:role: osd.2 2024-03-28T18:54:32.000 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-28T18:54:32.001 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T18:54:32.086 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T18:54:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T18:54:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T18:54:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T18:54:32.086 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T18:54:32.086 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-28T18:54:32.086 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T18:54:32.087 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T18:54:32.087 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T18:54:32.087 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T18:54:32.590 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-28T18:54:32.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-28T18:54:32.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-28T18:54:32.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-28T18:54:32.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-28T18:54:33.011 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:33.012 INFO:tasks.ceph:role: osd.3 2024-03-28T18:54:33.012 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com 2024-03-28T18:54:33.012 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T18:54:33.121 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T18:54:33.121 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T18:54:33.122 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T18:54:33.122 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T18:54:33.122 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T18:54:33.122 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-03-28T18:54:33.122 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T18:54:33.122 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T18:54:33.122 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T18:54:33.122 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T18:54:33.599 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-03-28T18:54:33.605 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-03-28T18:54:33.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-28T18:54:33.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-28T18:54:33.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:34.033+0000 7f184c7f6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-28T18:54:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:34.038+0000 7f184c7f6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-28T18:54:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:34.038+0000 7f184c7f6640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T18:54:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:34.038+0000 7f184c7f6640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T18:54:34.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:34.038+0000 7f184c7f6640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T18:54:34.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:34.048+0000 7f184c7f6640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-28T18:54:37.235 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-28T18:54:37.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:37.506+0000 7fc79d60d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-28T18:54:37.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:37.507+0000 7fc79d60d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-28T18:54:37.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:37.507+0000 7fc79d60d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T18:54:37.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:37.507+0000 7fc79d60d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T18:54:37.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:37.507+0000 7fc79d60d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T18:54:37.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:37.509+0000 7fc79d60d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-28T18:54:40.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-28T18:54:40.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:40.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:40.956+0000 7f366ea0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-28T18:54:40.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:40.956+0000 7f366ea0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-28T18:54:40.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:40.956+0000 7f366ea0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T18:54:40.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:40.956+0000 7f366ea0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T18:54:40.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:40.956+0000 7f366ea0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T18:54:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:40.959+0000 7f366ea0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-28T18:54:44.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-28T18:54:44.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:44.406+0000 7f60b41e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-28T18:54:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:44.406+0000 7f60b41e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-28T18:54:44.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:44.406+0000 7f60b41e1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T18:54:44.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:44.406+0000 7f60b41e1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T18:54:44.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:44.406+0000 7f60b41e1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T18:54:44.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:54:44.409+0000 7f60b41e1640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-28T18:54:47.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-28T18:54:47.443 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-28T18:54:47.541 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:47.541 INFO:tasks.ceph:role: osd.4 2024-03-28T18:54:47.541 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-28T18:54:47.541 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T18:54:47.639 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T18:54:47.639 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T18:54:47.639 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T18:54:47.639 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T18:54:47.639 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T18:54:47.639 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-28T18:54:47.639 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T18:54:47.639 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T18:54:47.640 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T18:54:47.640 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T18:54:48.150 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-28T18:54:48.157 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-28T18:54:48.157 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-28T18:54:48.309 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-28T18:54:48.409 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-28T18:54:48.496 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:48.496 INFO:tasks.ceph:role: osd.5 2024-03-28T18:54:48.496 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-28T18:54:48.497 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T18:54:48.581 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T18:54:48.581 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T18:54:48.581 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T18:54:48.582 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T18:54:48.582 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T18:54:48.582 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-28T18:54:48.582 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T18:54:48.582 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T18:54:48.582 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T18:54:48.582 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T18:54:49.049 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-28T18:54:49.054 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-28T18:54:49.055 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-28T18:54:49.164 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-28T18:54:49.243 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-28T18:54:49.324 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:49.324 INFO:tasks.ceph:role: osd.6 2024-03-28T18:54:49.325 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-28T18:54:49.325 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T18:54:49.406 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T18:54:49.406 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T18:54:49.407 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T18:54:49.407 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T18:54:49.407 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T18:54:49.407 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-28T18:54:49.407 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T18:54:49.407 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T18:54:49.407 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T18:54:49.407 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T18:54:49.932 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-28T18:54:49.937 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-28T18:54:49.937 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-28T18:54:50.117 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-28T18:54:50.223 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-28T18:54:50.304 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T18:54:50.304 INFO:tasks.ceph:role: osd.7 2024-03-28T18:54:50.304 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-28T18:54:50.305 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T18:54:50.400 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T18:54:50.400 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T18:54:50.400 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T18:54:50.401 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T18:54:50.401 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T18:54:50.401 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-28T18:54:50.401 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T18:54:50.401 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T18:54:50.401 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T18:54:50.401 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T18:54:50.857 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-28T18:54:50.863 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-28T18:54:50.863 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-28T18:54:50.968 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-28T18:54:51.050 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:51.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:51.181+0000 7f3df3119640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-28T18:54:51.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:51.181+0000 7f3df3119640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-28T18:54:51.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:51.181+0000 7f3df3119640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T18:54:51.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:51.181+0000 7f3df3119640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T18:54:51.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:51.181+0000 7f3df3119640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T18:54:51.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:51.184+0000 7f3df3119640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-28T18:54:54.110 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-28T18:54:54.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:54:54.166 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:54:54.203 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:54.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:54.378+0000 7f94ec328640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-28T18:54:54.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:54.379+0000 7f94ec328640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-28T18:54:54.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:54.379+0000 7f94ec328640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T18:54:54.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:54.379+0000 7f94ec328640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T18:54:54.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:54.379+0000 7f94ec328640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T18:54:54.380 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:54.381+0000 7f94ec328640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-28T18:54:57.297 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-28T18:54:57.393 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:54:57.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:57.532+0000 7ff207248640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-28T18:54:57.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:57.533+0000 7ff207248640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-28T18:54:57.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:57.533+0000 7ff207248640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T18:54:57.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:57.533+0000 7ff207248640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T18:54:57.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:57.533+0000 7ff207248640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T18:54:57.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:54:57.535+0000 7ff207248640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-28T18:55:00.449 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-28T18:55:00.548 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:55:00.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:00.685+0000 7fba99220640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-28T18:55:00.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:00.685+0000 7fba99220640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-28T18:55:00.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:00.685+0000 7fba99220640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T18:55:00.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:00.685+0000 7fba99220640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T18:55:00.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:00.685+0000 7fba99220640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T18:55:00.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:00.687+0000 7fba99220640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-28T18:55:03.849 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-28T18:55:03.943 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-28T18:55:03.943 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:03.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-28T18:55:04.055 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:04.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-28T18:55:04.162 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:04.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-28T18:55:04.271 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:04.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-28T18:55:04.376 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:04.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-28T18:55:04.483 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:04.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-28T18:55:04.590 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:04.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-28T18:55:04.699 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:04.700 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-28T18:55:04.798 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:04.798 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-28T18:55:04.895 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:04.895 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-28T18:55:04.989 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:04.989 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-28T18:55:05.078 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:05.078 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-28T18:55:05.217 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:05.217 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-28T18:55:05.311 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:05.311 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-28T18:55:05.406 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:05.406 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-28T18:55:05.469 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:05.469 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-28T18:55:05.523 INFO:tasks.ceph:Adding keys to all mons... 2024-03-28T18:55:05.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T18:55:05.547 DEBUG:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T18:55:05.640 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.y] 2024-03-28T18:55:05.640 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDkvAVmPWCLBRAAFJaR5+u+8dy0/z43Bs8Fdw== 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.a] 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDkvAVmCtcyHxAAEA5l6DD3kNC7dlVaE5YDQg== 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.c] 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDkvAVmzgSpMhAAGEC9DTm42TsHGbm2uGSG5w== 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDqvAVmpBUIAhAAzBgyBHgNciyE/jmqI66/dQ== 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDtvAVmb8w+HhAArjxa52zeTuJAhIsbh41hDA== 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDwvAVmfEAKORAAY5HCiYfDZ9jPrZrZEsmO0g== 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.3] 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQD0vAVmCo1CGBAAvIzRn2X+tXe1wuUU4ZBOKg== 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDkvAVmnAbPERAAQziKULSMyTGyq2T6v+WJ3A== 2024-03-28T18:55:05.641 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.b] 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDlvAVmPGAjCRAAwSpkcN3hT9YRgPqXwgGWkg== 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.d] 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDlvAVmKvdBGhAAuD0VoEWUG9kvoJFU8Eajhg== 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQD7vAVm1CPgChAAsOwXqCWizxBcpYTAfTXy2g== 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQD+vAVmd7eTFhAAhoGtHqS4wtMzlSp+oAdXyA== 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQABvQVm0nnJHxAAjD/XwoFlNcIAwcHdRBOZLQ== 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAEvQVmmlDZKBAAkj0HCqxAA8ETpXSolYlkEw== 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2024-03-28T18:55:05.642 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDlvAVmaYguKBAAwvqkBuZqvZXf9+zS1luatQ== 2024-03-28T18:55:05.643 INFO:teuthology.orchestra.run.smithi155.stdout:[client.1] 2024-03-28T18:55:05.643 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQDlvAVmmFcPNBAADEr1zhGHJrAr6eJOgHl8eg== 2024-03-28T18:55:05.644 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.y] 2024-03-28T18:55:05.644 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDkvAVmPWCLBRAAFJaR5+u+8dy0/z43Bs8Fdw== 2024-03-28T18:55:05.644 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-03-28T18:55:05.644 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDkvAVmCtcyHxAAEA5l6DD3kNC7dlVaE5YDQg== 2024-03-28T18:55:05.644 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-03-28T18:55:05.644 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDkvAVmzgSpMhAAGEC9DTm42TsHGbm2uGSG5w== 2024-03-28T18:55:05.644 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDqvAVmpBUIAhAAzBgyBHgNciyE/jmqI66/dQ== 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDtvAVmb8w+HhAArjxa52zeTuJAhIsbh41hDA== 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDwvAVmfEAKORAAY5HCiYfDZ9jPrZrZEsmO0g== 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD0vAVmCo1CGBAAvIzRn2X+tXe1wuUU4ZBOKg== 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDkvAVmnAbPERAAQziKULSMyTGyq2T6v+WJ3A== 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDlvAVmPGAjCRAAwSpkcN3hT9YRgPqXwgGWkg== 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDlvAVmKvdBGhAAuD0VoEWUG9kvoJFU8Eajhg== 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.4] 2024-03-28T18:55:05.645 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD7vAVm1CPgChAAsOwXqCWizxBcpYTAfTXy2g== 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.5] 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD+vAVmd7eTFhAAhoGtHqS4wtMzlSp+oAdXyA== 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.6] 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQABvQVm0nnJHxAAjD/XwoFlNcIAwcHdRBOZLQ== 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.7] 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQAEvQVmmlDZKBAAkj0HCqxAA8ETpXSolYlkEw== 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDlvAVmaYguKBAAwvqkBuZqvZXf9+zS1luatQ== 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2024-03-28T18:55:05.646 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDlvAVmmFcPNBAADEr1zhGHJrAr6eJOgHl8eg== 2024-03-28T18:55:05.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T18:55:05.658 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T18:55:05.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T18:55:05.810 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T18:55:05.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T18:55:05.961 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T18:55:06.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:06.118 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:06.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:06.278 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:06.425 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:06.431 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:06.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:06.590 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:06.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T18:55:06.748 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T18:55:06.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T18:55:06.904 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T18:55:07.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T18:55:07.069 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-28T18:55:07.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:07.219 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:07.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:07.376 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:07.529 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:07.535 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:07.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:07.692 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T18:55:07.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T18:55:07.843 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T18:55:07.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T18:55:07.998 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T18:55:08.150 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-28T18:55:08.150 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-28T18:55:08.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T18:55:08.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-28T18:55:08.537 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-28T18:55:08.638 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T18:55:08.784 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-28T18:55:08.877 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-28T18:55:08.972 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T18:55:09.129 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-28T18:55:09.204 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:55:09.212 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T18:55:09.265 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-28T18:55:09.265 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-28T18:55:09.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-28T18:55:09.272 INFO:tasks.ceph.mon.a:Started 2024-03-28T18:55:09.272 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-28T18:55:09.272 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-28T18:55:09.277 INFO:tasks.ceph.mon.b:Started 2024-03-28T18:55:09.277 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-28T18:55:09.277 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-28T18:55:09.282 INFO:tasks.ceph.mon.c:Started 2024-03-28T18:55:09.282 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-28T18:55:09.282 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-28T18:55:09.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-28T18:55:09.289 INFO:tasks.ceph.mgr.y:Started 2024-03-28T18:55:09.289 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-28T18:55:09.289 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-28T18:55:09.294 INFO:tasks.ceph.mgr.x:Started 2024-03-28T18:55:09.294 INFO:tasks.ceph:Setting crush tunables to default 2024-03-28T18:55:09.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2024-03-28T18:55:09.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.478+0000 7fec6cc00640 1 Processor -- start 2024-03-28T18:55:09.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.479+0000 7fec6cc00640 1 -- start start 2024-03-28T18:55:09.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.480+0000 7fec6cc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fec68152ec0 0x7fec68156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:09.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.480+0000 7fec6cc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6806d500 con 0x7fec680ad040 2024-03-28T18:55:09.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.480+0000 7fec6cc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec68159b50 con 0x7fec680c8ff0 2024-03-28T18:55:09.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.480+0000 7fec6cc00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec68158cd0 con 0x7fec68152ec0 2024-03-28T18:55:09.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.480+0000 7fec65400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fec680c8ff0 0x7fec68159440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:39566/0 (socket says 172.21.15.27:39566) 2024-03-28T18:55:09.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.480+0000 7fec65400640 1 -- 172.21.15.27:0/2001838007 learned_addr learned my addr 172.21.15.27:0/2001838007 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:09.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.480+0000 7fec65e00640 1 --2- 172.21.15.27:0/2001838007 >> v2:172.21.15.155:3300/0 conn(0x7fec68152ec0 0x7fec68156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:09.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.481+0000 7fec65e00640 1 -- 172.21.15.27:0/2001838007 >> v2:172.21.15.155:3300/0 conn(0x7fec68152ec0 msgr2=0x7fec68156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T18:55:09.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.481+0000 7fec65e00640 1 -- 172.21.15.27:0/2001838007 >> v2:172.21.15.155:3300/0 conn(0x7fec68152ec0 msgr2=0x7fec68156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T18:55:09.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.481+0000 7fec65e00640 1 --2- 172.21.15.27:0/2001838007 >> v2:172.21.15.155:3300/0 conn(0x7fec68152ec0 0x7fec68156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:55:09.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.481+0000 7fec65e00640 1 --2- 172.21.15.27:0/2001838007 >> v2:172.21.15.155:3300/0 conn(0x7fec68152ec0 0x7fec68156180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:55:09.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527411718 0 0) 0x7fec6806d500 con 0x7fec680ad040 2024-03-28T18:55:09.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec4c003520 con 0x7fec680ad040 2024-03-28T18:55:09.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3994601731 0 0) 0x7fec4c003520 con 0x7fec680ad040 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 >> v2:172.21.15.155:3300/0 conn(0x7fec68152ec0 msgr2=0x7fec68156180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 --2- 172.21.15.27:0/2001838007 >> v2:172.21.15.155:3300/0 conn(0x7fec68152ec0 0x7fec68156180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 >> v1:172.21.15.155:6789/0 conn(0x7fec680c8ff0 legacy=0x7fec68159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec680c5aa0 con 0x7fec680ad040 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fec600019f0 con 0x7fec680ad040 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.507+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 >> v1:172.21.15.27:6789/0 conn(0x7fec680ad040 legacy=0x7fec68160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec64a00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec4c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec64a00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec4c004800 0x7fec4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec64a00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec4c0053c0 0x7fec4c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68158cd0 con 0x7fec4c003a70 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c00c300 con 0x7fec4c004800 2024-03-28T18:55:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c00c500 con 0x7fec4c0053c0 2024-03-28T18:55:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec65e00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec4c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec65400640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec4c004800 0x7fec4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.508+0000 7fec66800640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec4c0053c0 0x7fec4c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:09.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.509+0000 7fec65e00640 1 -- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec4c0053c0 msgr2=0x7fec4c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.509+0000 7fec65e00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec4c0053c0 0x7fec4c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.509+0000 7fec65e00640 1 -- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec4c004800 msgr2=0x7fec4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.509+0000 7fec65e00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec4c004800 0x7fec4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.509+0000 7fec65e00640 1 -- 172.21.15.27:0/2001838007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c003520 con 0x7fec4c003a70 2024-03-28T18:55:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.509+0000 7fec66800640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec4c0053c0 0x7fec4c00ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.509+0000 7fec66800640 1 -- 172.21.15.27:0/2001838007 reap_dead start 2024-03-28T18:55:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.510+0000 7fec65e00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec4c003e90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fec5800b9e0 tx=0x7fec4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b09f67bca25e11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.511+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58036020 con 0x7fec4c003a70 2024-03-28T18:55:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.511+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec58032620 con 0x7fec4c003a70 2024-03-28T18:55:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.511+0000 7fec64a00640 1 -- 172.21.15.27:0/2001838007 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5800b5e0 con 0x7fec4c003a70 2024-03-28T18:55:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.512+0000 7fec6cc00640 1 -- 172.21.15.27:0/2001838007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 msgr2=0x7fec4c003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.512+0000 7fec6cc00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec4c003e90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fec5800b9e0 tx=0x7fec4c005830 comp rx=0 tx=0).stop 2024-03-28T18:55:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.512+0000 7fec6cc00640 1 -- 172.21.15.27:0/2001838007 shutdown_connections 2024-03-28T18:55:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.512+0000 7fec6cc00640 1 --2- 172.21.15.27:0/2001838007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec4c003e90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.512+0000 7fec6cc00640 1 -- 172.21.15.27:0/2001838007 >> 172.21.15.27:0/2001838007 conn(0x7fec680671f0 msgr2=0x7fec68067600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.512+0000 7fec6cc00640 1 -- 172.21.15.27:0/2001838007 shutdown_connections 2024-03-28T18:55:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.512+0000 7fec6cc00640 1 -- 172.21.15.27:0/2001838007 wait complete. 2024-03-28T18:55:09.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.514+0000 7fec6cc00640 1 Processor -- start 2024-03-28T18:55:09.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.514+0000 7fec6cc00640 1 -- start start 2024-03-28T18:55:09.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.515+0000 7fec6cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec680c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:09.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.515+0000 7fec6cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec680cbbe0 0x7fec681ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:09.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.515+0000 7fec6cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68156380 0x7fec68154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:09.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.515+0000 7fec6cc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec680c6060 con 0x7fec4c003a70 2024-03-28T18:55:09.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.515+0000 7fec6cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68158fb0 con 0x7fec68156380 2024-03-28T18:55:09.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.515+0000 7fec6cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6806d8f0 con 0x7fec680cbbe0 2024-03-28T18:55:09.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.515+0000 7fec65e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec680c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:09.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec680c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49636/0 (socket says 172.21.15.27:49636) 2024-03-28T18:55:09.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65e00640 1 -- 172.21.15.27:0/4283547452 learned_addr learned my addr 172.21.15.27:0/4283547452 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:09.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65400640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec680cbbe0 0x7fec681ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:09.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec66800640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68156380 0x7fec68154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65e00640 1 -- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec680cbbe0 msgr2=0x7fec681ec6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65e00640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec680cbbe0 0x7fec681ec6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65e00640 1 -- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68156380 msgr2=0x7fec68154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65e00640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68156380 0x7fec68154c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65e00640 1 -- 172.21.15.27:0/4283547452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6814ae80 con 0x7fec4c003a70 2024-03-28T18:55:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec66800640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68156380 0x7fec68154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.516+0000 7fec65e00640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec680c1570 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fec5800bb00 tx=0x7fec58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.517+0000 7fec5e000640 1 -- 172.21.15.27:0/4283547452 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58036030 con 0x7fec4c003a70 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.517+0000 7fec5e000640 1 -- 172.21.15.27:0/4283547452 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec58032e10 con 0x7fec4c003a70 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.517+0000 7fec5e000640 1 -- 172.21.15.27:0/4283547452 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5803b640 con 0x7fec4c003a70 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.517+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec680cb760 con 0x7fec4c003a70 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.517+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec68156080 con 0x7fec4c003a70 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.517+0000 7fec5e000640 1 -- 172.21.15.27:0/4283547452 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fec5804b060 con 0x7fec4c003a70 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.517+0000 7fec5e000640 1 -- 172.21.15.27:0/4283547452 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fec5804d8b0 con 0x7fec4c003a70 2024-03-28T18:55:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.518+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6804f470 con 0x7fec4c003a70 2024-03-28T18:55:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.522+0000 7fec5e000640 1 -- 172.21.15.27:0/4283547452 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fec5804d450 con 0x7fec4c003a70 2024-03-28T18:55:09.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.595+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fec680c9910 con 0x7fec4c003a70 2024-03-28T18:55:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.599+0000 7fec5e000640 1 -- 172.21.15.27:0/4283547452 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fec580324e0 con 0x7fec4c003a70 2024-03-28T18:55:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 msgr2=0x7fec680c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec680c1570 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fec5800bb00 tx=0x7fec58004730 comp rx=0 tx=0).stop 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 shutdown_connections 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68156380 0x7fec68154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec680cbbe0 0x7fec681ec6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 --2- 172.21.15.27:0/4283547452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec4c003a70 0x7fec680c1570 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 >> 172.21.15.27:0/4283547452 conn(0x7fec680671f0 msgr2=0x7fec680caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 shutdown_connections 2024-03-28T18:55:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:09.600+0000 7fec6cc00640 1 -- 172.21.15.27:0/4283547452 wait complete. 2024-03-28T18:55:09.618 INFO:tasks.ceph:check_enable_crimson: False 2024-03-28T18:55:09.618 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-28T18:55:09.619 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:09.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-28T18:55:09.710 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:09.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-28T18:55:09.803 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:09.803 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-28T18:55:09.943 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:55:09.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-28T18:55:10.041 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:10.041 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-28T18:55:10.139 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:10.139 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-28T18:55:10.223 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:10.223 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-28T18:55:10.273 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:10.273+0000 7f4325438040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:55:10.281 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:10.281+0000 7effbea13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:55:10.344 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:10.344+0000 7f4325438040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:55:10.351 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:10.351+0000 7effbea13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:55:10.354 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:55:10.354 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-28T18:55:10.478 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new ac377893-aec5-494f-977a-d7c9410c3482 0 2024-03-28T18:55:10.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.709+0000 7f3e77e00640 1 Processor -- start 2024-03-28T18:55:10.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.709+0000 7f3e77e00640 1 -- start start 2024-03-28T18:55:10.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.710+0000 7f3e77e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3e78152ba0 0x7f3e780726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:10.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.710+0000 7f3e77e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780c9050 0x7f3e780cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:10.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.710+0000 7f3e77e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e78072c00 con 0x7f3e780cb960 2024-03-28T18:55:10.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.710+0000 7f3e77e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e78158d90 con 0x7f3e78152ba0 2024-03-28T18:55:10.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.710+0000 7f3e77e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e78159070 con 0x7f3e780c9050 2024-03-28T18:55:10.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.710+0000 7f3e76000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780c9050 0x7f3e780cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:10.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.710+0000 7f3e76a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3e78152ba0 0x7f3e780726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:10.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.711+0000 7f3e76a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3e78152ba0 0x7f3e780726c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46192/0 (socket says 172.21.15.155:46192) 2024-03-28T18:55:10.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.711+0000 7f3e76a00640 1 -- 172.21.15.155:0/2258074112 learned_addr learned my addr 172.21.15.155:0/2258074112 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:10.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.711+0000 7f3e76000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780c9050 0x7f3e780cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:53450/0 (socket says 172.21.15.155:53450) 2024-03-28T18:55:10.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.711+0000 7f3e76000640 1 -- 172.21.15.155:0/2258074112 >> v2:172.21.15.155:3300/0 conn(0x7f3e78152ba0 msgr2=0x7f3e780726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:10.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.711+0000 7f3e76000640 1 --2- 172.21.15.155:0/2258074112 >> v2:172.21.15.155:3300/0 conn(0x7f3e78152ba0 0x7f3e780726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.711+0000 7f3e76000640 1 -- 172.21.15.155:0/2258074112 >> v1:172.21.15.155:6789/0 conn(0x7f3e780cb960 legacy=0x7f3e78155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:10.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.711+0000 7f3e76000640 1 -- 172.21.15.155:0/2258074112 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e78159350 con 0x7f3e780c9050 2024-03-28T18:55:10.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.711+0000 7f3e76a00640 1 --2- 172.21.15.155:0/2258074112 >> v2:172.21.15.155:3300/0 conn(0x7f3e78152ba0 0x7f3e780726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:10.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.712+0000 7f3e76000640 1 --2- 172.21.15.155:0/2258074112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780c9050 0x7f3e780cb420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e640097a0 tx=0x7f3e6402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc74257d28c58166 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:10.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.712+0000 7f3e75600640 1 -- 172.21.15.155:0/2258074112 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64038070 con 0x7f3e780c9050 2024-03-28T18:55:10.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.712+0000 7f3e75600640 1 -- 172.21.15.155:0/2258074112 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e64031700 con 0x7f3e780c9050 2024-03-28T18:55:10.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.712+0000 7f3e75600640 1 -- 172.21.15.155:0/2258074112 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64031ad0 con 0x7f3e780c9050 2024-03-28T18:55:10.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.713+0000 7f3e77e00640 1 -- 172.21.15.155:0/2258074112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780c9050 msgr2=0x7f3e780cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:10.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.713+0000 7f3e77e00640 1 --2- 172.21.15.155:0/2258074112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780c9050 0x7f3e780cb420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e640097a0 tx=0x7f3e6402b7a0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.713+0000 7f3e77e00640 1 -- 172.21.15.155:0/2258074112 shutdown_connections 2024-03-28T18:55:10.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.713+0000 7f3e77e00640 1 --2- 172.21.15.155:0/2258074112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780c9050 0x7f3e780cb420 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.713+0000 7f3e77e00640 1 --2- 172.21.15.155:0/2258074112 >> v2:172.21.15.155:3300/0 conn(0x7f3e78152ba0 0x7f3e780726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.713+0000 7f3e77e00640 1 -- 172.21.15.155:0/2258074112 >> 172.21.15.155:0/2258074112 conn(0x7f3e7806b4e0 msgr2=0x7f3e7806b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:10.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.713+0000 7f3e77e00640 1 -- 172.21.15.155:0/2258074112 shutdown_connections 2024-03-28T18:55:10.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.713+0000 7f3e77e00640 1 -- 172.21.15.155:0/2258074112 wait complete. 2024-03-28T18:55:10.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.715+0000 7f3e77e00640 1 Processor -- start 2024-03-28T18:55:10.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.715+0000 7f3e77e00640 1 -- start start 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.716+0000 7f3e77e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e780c9050 0x7f3e7814a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.716+0000 7f3e77e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780cb960 0x7f3e7814a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.716+0000 7f3e77e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e78152ba0 0x7f3e780afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.716+0000 7f3e77e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e78152f80 con 0x7f3e780c9050 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e77e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e78159910 con 0x7f3e78152ba0 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e77e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e78153100 con 0x7f3e780cb960 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780cb960 0x7f3e7814a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e77400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e78152ba0 0x7f3e780afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780cb960 0x7f3e7814a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:53454/0 (socket says 172.21.15.155:53454) 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 -- 172.21.15.155:0/916573728 learned_addr learned my addr 172.21.15.155:0/916573728 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 -- 172.21.15.155:0/916573728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e78152ba0 msgr2=0x7f3e780afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 --2- 172.21.15.155:0/916573728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e78152ba0 0x7f3e780afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 -- 172.21.15.155:0/916573728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e780c9050 msgr2=0x7f3e7814a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:10.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 --2- 172.21.15.155:0/916573728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e780c9050 0x7f3e7814a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 -- 172.21.15.155:0/916573728 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e78154370 con 0x7f3e780cb960 2024-03-28T18:55:10.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.717+0000 7f3e76000640 1 --2- 172.21.15.155:0/916573728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780cb960 0x7f3e7814a600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e640098c0 tx=0x7f3e640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:10.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.718+0000 7f3e6f400640 1 -- 172.21.15.155:0/916573728 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e64038040 con 0x7f3e780cb960 2024-03-28T18:55:10.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.718+0000 7f3e6f400640 1 -- 172.21.15.155:0/916573728 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6403c610 con 0x7f3e780cb960 2024-03-28T18:55:10.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.718+0000 7f3e6f400640 1 -- 172.21.15.155:0/916573728 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6403c9e0 con 0x7f3e780cb960 2024-03-28T18:55:10.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.718+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e780adcd0 con 0x7f3e780cb960 2024-03-28T18:55:10.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.718+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e780ae290 con 0x7f3e780cb960 2024-03-28T18:55:10.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.718+0000 7f3e6f400640 1 -- 172.21.15.155:0/916573728 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3e64045020 con 0x7f3e780cb960 2024-03-28T18:55:10.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.718+0000 7f3e6f400640 1 -- 172.21.15.155:0/916573728 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3e64047990 con 0x7f3e780cb960 2024-03-28T18:55:10.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.719+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7804f400 con 0x7f3e780cb960 2024-03-28T18:55:10.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.722+0000 7f3e6f400640 1 -- 172.21.15.155:0/916573728 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3e64030070 con 0x7f3e780cb960 2024-03-28T18:55:10.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.800+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ac377893-aec5-494f-977a-d7c9410c3482", "id": 0} v 0) v1 -- 0x7f3e7814ca80 con 0x7f3e780cb960 2024-03-28T18:55:10.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.807+0000 7f3e6f400640 1 -- 172.21.15.155:0/916573728 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac377893-aec5-494f-977a-d7c9410c3482", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3e64047410 con 0x7f3e780cb960 2024-03-28T18:55:10.808 INFO:teuthology.orchestra.run.smithi155.stdout:0 2024-03-28T18:55:10.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780cb960 msgr2=0x7f3e7814a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:10.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 --2- 172.21.15.155:0/916573728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780cb960 0x7f3e7814a600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3e640098c0 tx=0x7f3e640046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 shutdown_connections 2024-03-28T18:55:10.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 --2- 172.21.15.155:0/916573728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e78152ba0 0x7f3e780afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 --2- 172.21.15.155:0/916573728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e780cb960 0x7f3e7814a600 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 --2- 172.21.15.155:0/916573728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e780c9050 0x7f3e7814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:10.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 >> 172.21.15.155:0/916573728 conn(0x7f3e7806b4e0 msgr2=0x7f3e78151e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:10.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 shutdown_connections 2024-03-28T18:55:10.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:10.809+0000 7f3e77e00640 1 -- 172.21.15.155:0/916573728 wait complete. 2024-03-28T18:55:10.827 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new d2f8b913-b0d6-4b30-9553-1428cdf1509c 1 2024-03-28T18:55:10.983 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:10.983+0000 7effbea13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:55:10.985 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:10.985+0000 7f4325438040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:55:11.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.011+0000 7f0c76e00640 1 Processor -- start 2024-03-28T18:55:11.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.011+0000 7f0c76e00640 1 -- start start 2024-03-28T18:55:11.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.012+0000 7f0c76e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c7006c8a0 0x7f0c70158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.012+0000 7f0c76e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c701529e0 0x7f0c70152e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.012+0000 7f0c76e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c70073070 con 0x7f0c700c4f00 2024-03-28T18:55:11.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.012+0000 7f0c76e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c70071e50 con 0x7f0c701529e0 2024-03-28T18:55:11.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.012+0000 7f0c76e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c70071bf0 con 0x7f0c7006c8a0 2024-03-28T18:55:11.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.013+0000 7f0c75000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c7006c8a0 0x7f0c70158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.013+0000 7f0c75000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c7006c8a0 0x7f0c70158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:53466/0 (socket says 172.21.15.155:53466) 2024-03-28T18:55:11.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.013+0000 7f0c75000640 1 -- 172.21.15.155:0/3406279187 learned_addr learned my addr 172.21.15.155:0/3406279187 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:11.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.013+0000 7f0c76400640 1 --2- 172.21.15.155:0/3406279187 >> v2:172.21.15.27:3300/0 conn(0x7f0c701529e0 0x7f0c70152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.013+0000 7f0c75000640 1 -- 172.21.15.155:0/3406279187 >> v2:172.21.15.27:3300/0 conn(0x7f0c701529e0 msgr2=0x7f0c70152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:11.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.013+0000 7f0c75000640 1 --2- 172.21.15.155:0/3406279187 >> v2:172.21.15.27:3300/0 conn(0x7f0c701529e0 0x7f0c70152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.013+0000 7f0c75000640 1 -- 172.21.15.155:0/3406279187 >> v1:172.21.15.27:6789/0 conn(0x7f0c700c4f00 legacy=0x7f0c70072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:11.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.013+0000 7f0c75000640 1 -- 172.21.15.155:0/3406279187 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70071910 con 0x7f0c7006c8a0 2024-03-28T18:55:11.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.014+0000 7f0c75000640 1 --2- 172.21.15.155:0/3406279187 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c7006c8a0 0x7f0c70158dd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c70071fd0 tx=0x7f0c5c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7083f03a0a12da6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:11.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.014+0000 7f0c6fe00640 1 -- 172.21.15.155:0/3406279187 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c0317c0 con 0x7f0c7006c8a0 2024-03-28T18:55:11.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.014+0000 7f0c6fe00640 1 -- 172.21.15.155:0/3406279187 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c037070 con 0x7f0c7006c8a0 2024-03-28T18:55:11.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c6fe00640 1 -- 172.21.15.155:0/3406279187 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c031b80 con 0x7f0c7006c8a0 2024-03-28T18:55:11.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c76e00640 1 -- 172.21.15.155:0/3406279187 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c7006c8a0 msgr2=0x7f0c70158dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c76e00640 1 --2- 172.21.15.155:0/3406279187 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c7006c8a0 0x7f0c70158dd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c70071fd0 tx=0x7f0c5c02b860 comp rx=0 tx=0).stop 2024-03-28T18:55:11.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c76e00640 1 -- 172.21.15.155:0/3406279187 shutdown_connections 2024-03-28T18:55:11.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c76e00640 1 --2- 172.21.15.155:0/3406279187 >> v2:172.21.15.27:3300/0 conn(0x7f0c701529e0 0x7f0c70152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c76e00640 1 --2- 172.21.15.155:0/3406279187 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c7006c8a0 0x7f0c70158dd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c76e00640 1 -- 172.21.15.155:0/3406279187 >> 172.21.15.155:0/3406279187 conn(0x7f0c700c8130 msgr2=0x7f0c700c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:11.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c76e00640 1 -- 172.21.15.155:0/3406279187 shutdown_connections 2024-03-28T18:55:11.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.015+0000 7f0c76e00640 1 -- 172.21.15.155:0/3406279187 wait complete. 2024-03-28T18:55:11.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.017+0000 7f0c76e00640 1 Processor -- start 2024-03-28T18:55:11.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.017+0000 7f0c76e00640 1 -- start start 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c76e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c7006c8a0 0x7f0c701ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c76e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c700c4f00 0x7f0c70154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c76e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c701529e0 0x7f0c70154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c76e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7006c630 con 0x7f0c700c4f00 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c76e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700724d0 con 0x7f0c7006c8a0 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c76e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700bb2d0 con 0x7f0c701529e0 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c75a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c7006c8a0 0x7f0c701ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c76400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c701529e0 0x7f0c70154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c75a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c7006c8a0 0x7f0c701ee7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46206/0 (socket says 172.21.15.155:46206) 2024-03-28T18:55:11.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c75a00640 1 -- 172.21.15.155:0/2722391428 learned_addr learned my addr 172.21.15.155:0/2722391428 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:11.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.018+0000 7f0c75000640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c700c4f00 0x7f0c70154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c76400640 1 -- 172.21.15.155:0/2722391428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c7006c8a0 msgr2=0x7f0c701ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c76400640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c7006c8a0 0x7f0c701ee7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c76400640 1 -- 172.21.15.155:0/2722391428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c700c4f00 msgr2=0x7f0c70154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c76400640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c700c4f00 0x7f0c70154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c76400640 1 -- 172.21.15.155:0/2722391428 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c700c1340 con 0x7f0c701529e0 2024-03-28T18:55:11.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c75a00640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c7006c8a0 0x7f0c701ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:11.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c76400640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c701529e0 0x7f0c70154640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0c68007dc0 tx=0x7f0c6800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:11.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c6d600640 1 -- 172.21.15.155:0/2722391428 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800edd0 con 0x7f0c701529e0 2024-03-28T18:55:11.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c6d600640 1 -- 172.21.15.155:0/2722391428 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c68003990 con 0x7f0c701529e0 2024-03-28T18:55:11.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c6d600640 1 -- 172.21.15.155:0/2722391428 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68003d80 con 0x7f0c701529e0 2024-03-28T18:55:11.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c701ef0f0 con 0x7f0c701529e0 2024-03-28T18:55:11.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.019+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c701ef3d0 con 0x7f0c701529e0 2024-03-28T18:55:11.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.020+0000 7f0c6d600640 1 -- 172.21.15.155:0/2722391428 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0c6801b020 con 0x7f0c701529e0 2024-03-28T18:55:11.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.020+0000 7f0c6d600640 1 -- 172.21.15.155:0/2722391428 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f0c68014b00 con 0x7f0c701529e0 2024-03-28T18:55:11.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.021+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7004f400 con 0x7f0c701529e0 2024-03-28T18:55:11.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.023+0000 7f0c6d600640 1 -- 172.21.15.155:0/2722391428 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0c68014d00 con 0x7f0c701529e0 2024-03-28T18:55:11.051 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.051+0000 7effbea13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:55:11.055 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.055+0000 7f4325438040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:55:11.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.102+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d2f8b913-b0d6-4b30-9553-1428cdf1509c", "id": 1} v 0) v1 -- 0x7f0c70153170 con 0x7f0c701529e0 2024-03-28T18:55:11.104 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.105+0000 7effbea13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:55:11.109 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.109+0000 7f4325438040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:55:11.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.110+0000 7f0c6d600640 1 -- 172.21.15.155:0/2722391428 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d2f8b913-b0d6-4b30-9553-1428cdf1509c", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0c68014420 con 0x7f0c701529e0 2024-03-28T18:55:11.110 INFO:teuthology.orchestra.run.smithi155.stdout:1 2024-03-28T18:55:11.112 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c701529e0 msgr2=0x7f0c70154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c701529e0 0x7f0c70154640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0c68007dc0 tx=0x7f0c6800e4b0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 shutdown_connections 2024-03-28T18:55:11.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c701529e0 0x7f0c70154640 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c700c4f00 0x7f0c70154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 --2- 172.21.15.155:0/2722391428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c7006c8a0 0x7f0c701ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 >> 172.21.15.155:0/2722391428 conn(0x7f0c700c8130 msgr2=0x7f0c70151bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:11.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 shutdown_connections 2024-03-28T18:55:11.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.112+0000 7f0c76e00640 1 -- 172.21.15.155:0/2722391428 wait complete. 2024-03-28T18:55:11.135 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new d285ba44-b5df-4431-b5bd-8d7b3545fe2b 2 2024-03-28T18:55:11.171 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.171+0000 7effbea13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:55:11.184 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.184+0000 7f4325438040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:55:11.223 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.224+0000 7effbea13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:55:11.238 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.238+0000 7f4325438040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:55:11.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.325+0000 7f526ee00640 1 Processor -- start 2024-03-28T18:55:11.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.325+0000 7f526ee00640 1 -- start start 2024-03-28T18:55:11.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f526ee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f5268155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f526ee00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5268158d90 con 0x7f5268152ba0 2024-03-28T18:55:11.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f526ee00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5268159070 con 0x7f52680c9050 2024-03-28T18:55:11.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f526ee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5268159350 con 0x7f5268155790 2024-03-28T18:55:11.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f526d000640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f52680c9050 0x7f52680cb420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:51780/0 (socket says 172.21.15.155:51780) 2024-03-28T18:55:11.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f526d000640 1 -- 172.21.15.155:0/3700359544 learned_addr learned my addr 172.21.15.155:0/3700359544 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-28T18:55:11.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f526e400640 1 --2- 172.21.15.155:0/3700359544 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f5268155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f5267e00640 1 -- 172.21.15.155:0/3700359544 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532128687 0 0) 0x7f5268159070 con 0x7f52680c9050 2024-03-28T18:55:11.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.327+0000 7f526e400640 1 -- 172.21.15.155:0/3700359544 >> v1:172.21.15.155:6789/0 conn(0x7f52680c9050 legacy=0x7f52680cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.328+0000 7f526e400640 1 -- 172.21.15.155:0/3700359544 >> v1:172.21.15.27:6789/0 conn(0x7f5268152ba0 legacy=0x7f52680726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.328+0000 7f526e400640 1 -- 172.21.15.155:0/3700359544 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5268159630 con 0x7f5268155790 2024-03-28T18:55:11.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.329+0000 7f526e400640 1 --2- 172.21.15.155:0/3700359544 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f5268155c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5260009dc0 tx=0x7f526002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7c3d9b45707155d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:11.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.329+0000 7f5267e00640 1 -- 172.21.15.155:0/3700359544 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5260039070 con 0x7f5268155790 2024-03-28T18:55:11.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.329+0000 7f5267e00640 1 -- 172.21.15.155:0/3700359544 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526002cbb0 con 0x7f5268155790 2024-03-28T18:55:11.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.329+0000 7f5267e00640 1 -- 172.21.15.155:0/3700359544 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5260034440 con 0x7f5268155790 2024-03-28T18:55:11.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.330+0000 7f526ee00640 1 -- 172.21.15.155:0/3700359544 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 msgr2=0x7f5268155c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.330+0000 7f526ee00640 1 --2- 172.21.15.155:0/3700359544 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f5268155c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5260009dc0 tx=0x7f526002be00 comp rx=0 tx=0).stop 2024-03-28T18:55:11.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.330+0000 7f526ee00640 1 -- 172.21.15.155:0/3700359544 shutdown_connections 2024-03-28T18:55:11.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.330+0000 7f526ee00640 1 --2- 172.21.15.155:0/3700359544 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f5268155c20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.330+0000 7f526ee00640 1 -- 172.21.15.155:0/3700359544 >> 172.21.15.155:0/3700359544 conn(0x7f526806b4e0 msgr2=0x7f526806b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:11.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.330+0000 7f526ee00640 1 -- 172.21.15.155:0/3700359544 shutdown_connections 2024-03-28T18:55:11.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.330+0000 7f526ee00640 1 -- 172.21.15.155:0/3700359544 wait complete. 2024-03-28T18:55:11.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.332+0000 7f526ee00640 1 Processor -- start 2024-03-28T18:55:11.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.332+0000 7f526ee00640 1 -- start start 2024-03-28T18:55:11.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526ee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52680c9050 0x7f52680abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526ee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268152ba0 0x7f526814b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526ee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f526814b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526ee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52680cbcb0 con 0x7f5268152ba0 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526ee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52680cbb30 con 0x7f52680c9050 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526ee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5268152f80 con 0x7f5268155790 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52680c9050 0x7f52680abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526e400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f526814b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52680c9050 0x7f52680abbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46212/0 (socket says 172.21.15.155:46212) 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526da00640 1 -- 172.21.15.155:0/461215641 learned_addr learned my addr 172.21.15.155:0/461215641 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.333+0000 7f526d000640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268152ba0 0x7f526814b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526da00640 1 -- 172.21.15.155:0/461215641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 msgr2=0x7f526814b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526da00640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f526814b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526da00640 1 -- 172.21.15.155:0/461215641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268152ba0 msgr2=0x7f526814b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526da00640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268152ba0 0x7f526814b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526da00640 1 -- 172.21.15.155:0/461215641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52680b0a20 con 0x7f52680c9050 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526d000640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268152ba0 0x7f526814b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526e400640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f526814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526da00640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52680c9050 0x7f52680abbf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f525c00c420 tx=0x7f525c00c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f5265600640 1 -- 172.21.15.155:0/461215641 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c0052d0 con 0x7f52680c9050 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f5265600640 1 -- 172.21.15.155:0/461215641 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525c0054b0 con 0x7f52680c9050 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f5265600640 1 -- 172.21.15.155:0/461215641 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c006a60 con 0x7f52680c9050 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52680c1580 con 0x7f52680c9050 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52680ac130 con 0x7f52680c9050 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.334+0000 7f5265600640 1 -- 172.21.15.155:0/461215641 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f525c006c00 con 0x7f52680c9050 2024-03-28T18:55:11.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.335+0000 7f5265600640 1 -- 172.21.15.155:0/461215641 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f525c007020 con 0x7f52680c9050 2024-03-28T18:55:11.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.335+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526804f400 con 0x7f52680c9050 2024-03-28T18:55:11.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.338+0000 7f5265600640 1 -- 172.21.15.155:0/461215641 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f525c003640 con 0x7f52680c9050 2024-03-28T18:55:11.361 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.361+0000 7effbea13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:55:11.379 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.379+0000 7f4325438040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:55:11.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.417+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d285ba44-b5df-4431-b5bd-8d7b3545fe2b", "id": 2} v 0) v1 -- 0x7f52680bea60 con 0x7f52680c9050 2024-03-28T18:55:11.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.424+0000 7f5265600640 1 -- 172.21.15.155:0/461215641 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d285ba44-b5df-4431-b5bd-8d7b3545fe2b", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f525c007340 con 0x7f52680c9050 2024-03-28T18:55:11.424 INFO:teuthology.orchestra.run.smithi155.stdout:2 2024-03-28T18:55:11.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.425+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52680c9050 msgr2=0x7f52680abbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.425+0000 7f526ee00640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52680c9050 0x7f52680abbf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f525c00c420 tx=0x7f525c00c8e0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.426+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 shutdown_connections 2024-03-28T18:55:11.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.426+0000 7f526ee00640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5268155790 0x7f526814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.426+0000 7f526ee00640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5268152ba0 0x7f526814b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.426+0000 7f526ee00640 1 --2- 172.21.15.155:0/461215641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52680c9050 0x7f52680abbf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.426+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 >> 172.21.15.155:0/461215641 conn(0x7f526806b4e0 msgr2=0x7f52680c83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:11.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.426+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 shutdown_connections 2024-03-28T18:55:11.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.426+0000 7f526ee00640 1 -- 172.21.15.155:0/461215641 wait complete. 2024-03-28T18:55:11.442 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 91fb0367-acaa-4acc-83a6-6e7240683784 3 2024-03-28T18:55:11.447 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.447+0000 7effbea13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:55:11.466 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.466+0000 7f4325438040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:55:11.503 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.503+0000 7effbea13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:55:11.526 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.526+0000 7f4325438040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:55:11.567 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.567+0000 7effbea13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:55:11.591 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.591+0000 7f4325438040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:55:11.625 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:11.625+0000 7effbea13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:55:11.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.629+0000 7f6407200640 1 Processor -- start 2024-03-28T18:55:11.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.629+0000 7f6407200640 1 -- start start 2024-03-28T18:55:11.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6407200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6400152e00 0x7f6400158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6407200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f640006c8a0 0x7f6400072a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6407200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6400071ef0 con 0x7f64000a7040 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6407200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64000725d0 con 0x7f6400152e00 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6407200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6400071a10 con 0x7f640006c8a0 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f640006c8a0 0x7f6400072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f640006c8a0 0x7f6400072a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46218/0 (socket says 172.21.15.155:46218) 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405400640 1 -- 172.21.15.155:0/3534907369 learned_addr learned my addr 172.21.15.155:0/3534907369 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405e00640 1 --2- 172.21.15.155:0/3534907369 >> v2:172.21.15.27:3300/0 conn(0x7f6400152e00 0x7f6400158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405400640 1 -- 172.21.15.155:0/3534907369 >> v2:172.21.15.27:3300/0 conn(0x7f6400152e00 msgr2=0x7f6400158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405400640 1 --2- 172.21.15.155:0/3534907369 >> v2:172.21.15.27:3300/0 conn(0x7f6400152e00 0x7f6400158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405400640 1 -- 172.21.15.155:0/3534907369 >> v1:172.21.15.155:6789/0 conn(0x7f64000a7040 legacy=0x7f64000c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405400640 1 -- 172.21.15.155:0/3534907369 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64001531e0 con 0x7f640006c8a0 2024-03-28T18:55:11.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.631+0000 7f6405e00640 1 --2- 172.21.15.155:0/3534907369 >> v2:172.21.15.27:3300/0 conn(0x7f6400152e00 0x7f6400158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:11.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.633+0000 7f6405400640 1 --2- 172.21.15.155:0/3534907369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f6400072a60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f64000adc90 tx=0x7f63f002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6df824f3c4a501ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:11.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.633+0000 7f6404a00640 1 -- 172.21.15.155:0/3534907369 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0035020 con 0x7f640006c8a0 2024-03-28T18:55:11.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.633+0000 7f6404a00640 1 -- 172.21.15.155:0/3534907369 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0031870 con 0x7f640006c8a0 2024-03-28T18:55:11.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.633+0000 7f6404a00640 1 -- 172.21.15.155:0/3534907369 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0031c40 con 0x7f640006c8a0 2024-03-28T18:55:11.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.634+0000 7f6407200640 1 -- 172.21.15.155:0/3534907369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 msgr2=0x7f6400072a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.634+0000 7f6407200640 1 --2- 172.21.15.155:0/3534907369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f6400072a60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f64000adc90 tx=0x7f63f002b910 comp rx=0 tx=0).stop 2024-03-28T18:55:11.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.634+0000 7f6407200640 1 -- 172.21.15.155:0/3534907369 shutdown_connections 2024-03-28T18:55:11.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.634+0000 7f6407200640 1 --2- 172.21.15.155:0/3534907369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f6400072a60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.634+0000 7f6407200640 1 --2- 172.21.15.155:0/3534907369 >> v2:172.21.15.27:3300/0 conn(0x7f6400152e00 0x7f6400158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.634+0000 7f6407200640 1 -- 172.21.15.155:0/3534907369 >> 172.21.15.155:0/3534907369 conn(0x7f64000c8130 msgr2=0x7f64000c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:11.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.634+0000 7f6407200640 1 -- 172.21.15.155:0/3534907369 shutdown_connections 2024-03-28T18:55:11.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.634+0000 7f6407200640 1 -- 172.21.15.155:0/3534907369 wait complete. 2024-03-28T18:55:11.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.636+0000 7f6407200640 1 Processor -- start 2024-03-28T18:55:11.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.636+0000 7f6407200640 1 -- start start 2024-03-28T18:55:11.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6407200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f64001ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6407200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64000a7040 0x7f64001593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6407200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6400152e00 0x7f64001598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6407200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64000c53a0 con 0x7f64000a7040 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6407200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64000bb2d0 con 0x7f640006c8a0 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6407200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64000c08e0 con 0x7f6400152e00 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f64001ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f64001ee5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46234/0 (socket says 172.21.15.155:46234) 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405e00640 1 -- 172.21.15.155:0/2563247137 learned_addr learned my addr 172.21.15.155:0/2563247137 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6406800640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6400152e00 0x7f64001598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405400640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64000a7040 0x7f64001593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405e00640 1 -- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6400152e00 msgr2=0x7f64001598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405e00640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6400152e00 0x7f64001598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405e00640 1 -- 172.21.15.155:0/2563247137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64000a7040 msgr2=0x7f64001593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405e00640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64000a7040 0x7f64001593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6405e00640 1 -- 172.21.15.155:0/2563247137 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64001eef90 con 0x7f640006c8a0 2024-03-28T18:55:11.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.637+0000 7f6406800640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6400152e00 0x7f64001598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.638+0000 7f6405e00640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f64001ee5a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f63f400c7f0 tx=0x7f63f400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.638+0000 7f63fa000640 1 -- 172.21.15.155:0/2563247137 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4015070 con 0x7f640006c8a0 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.638+0000 7f63fa000640 1 -- 172.21.15.155:0/2563247137 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4007a30 con 0x7f640006c8a0 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.638+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64000bc270 con 0x7f640006c8a0 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.638+0000 7f63fa000640 1 -- 172.21.15.155:0/2563247137 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4007e00 con 0x7f640006c8a0 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.638+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64000bc830 con 0x7f640006c8a0 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.638+0000 7f63fa000640 1 -- 172.21.15.155:0/2563247137 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f63f4018070 con 0x7f640006c8a0 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.638+0000 7f63fa000640 1 -- 172.21.15.155:0/2563247137 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f63f401a8e0 con 0x7f640006c8a0 2024-03-28T18:55:11.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.640+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640004f400 con 0x7f640006c8a0 2024-03-28T18:55:11.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.642+0000 7f63fa000640 1 -- 172.21.15.155:0/2563247137 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f63f401abc0 con 0x7f640006c8a0 2024-03-28T18:55:11.652 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:11.652+0000 7f4325438040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:55:11.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.721+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd new", "uuid": "91fb0367-acaa-4acc-83a6-6e7240683784", "id": 3} v 0) v1 -- 0x7f640006b6a0 con 0x7f640006c8a0 2024-03-28T18:55:11.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.729+0000 7f63fa000640 1 -- 172.21.15.155:0/2563247137 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "91fb0367-acaa-4acc-83a6-6e7240683784", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f63f401a450 con 0x7f640006c8a0 2024-03-28T18:55:11.730 INFO:teuthology.orchestra.run.smithi155.stdout:3 2024-03-28T18:55:11.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 msgr2=0x7f64001ee5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f64001ee5a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f63f400c7f0 tx=0x7f63f400cbf0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 shutdown_connections 2024-03-28T18:55:11.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6400152e00 0x7f64001598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64000a7040 0x7f64001593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 --2- 172.21.15.155:0/2563247137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640006c8a0 0x7f64001ee5a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.732 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 >> 172.21.15.155:0/2563247137 conn(0x7f64000c8130 msgr2=0x7f64000c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:11.732 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 shutdown_connections 2024-03-28T18:55:11.732 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.731+0000 7f6407200640 1 -- 172.21.15.155:0/2563247137 wait complete. 2024-03-28T18:55:11.748 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 216c6c5e-1540-4217-8bba-88752ec61d5b 4 2024-03-28T18:55:11.981 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.981+0000 7f41e9600640 1 Processor -- start 2024-03-28T18:55:11.982 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.981+0000 7f41e9600640 1 -- start start 2024-03-28T18:55:11.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e9600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f41e4156180 0x7f41e4161070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e9600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e406d500 con 0x7f41e40c8ff0 2024-03-28T18:55:11.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e9600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e4159b50 con 0x7f41e4152ec0 2024-03-28T18:55:11.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e9600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e4158cd0 con 0x7f41e4156180 2024-03-28T18:55:11.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e3400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f41e4156180 0x7f41e4161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e3400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f41e4156180 0x7f41e4161070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46250/0 (socket says 172.21.15.155:46250) 2024-03-28T18:55:11.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e3400640 1 -- 172.21.15.155:0/2816130802 learned_addr learned my addr 172.21.15.155:0/2816130802 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:11.984 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e3400640 1 -- 172.21.15.155:0/2816130802 >> v1:172.21.15.155:6789/0 conn(0x7f41e4152ec0 legacy=0x7f41e4159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.984 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e3400640 1 -- 172.21.15.155:0/2816130802 >> v1:172.21.15.27:6789/0 conn(0x7f41e40c8ff0 legacy=0x7f41e40ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:11.984 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.983+0000 7f41e3400640 1 -- 172.21.15.155:0/2816130802 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e40c5aa0 con 0x7f41e4156180 2024-03-28T18:55:11.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.985+0000 7f41e3400640 1 --2- 172.21.15.155:0/2816130802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4156180 0x7f41e4161070 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009e50 tx=0x7f41dc02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=707dd8abea42615a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:11.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.985+0000 7f41e1600640 1 -- 172.21.15.155:0/2816130802 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc02c9f0 con 0x7f41e4156180 2024-03-28T18:55:11.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.985+0000 7f41e1600640 1 -- 172.21.15.155:0/2816130802 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41dc02cb90 con 0x7f41e4156180 2024-03-28T18:55:11.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.985+0000 7f41e1600640 1 -- 172.21.15.155:0/2816130802 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0343f0 con 0x7f41e4156180 2024-03-28T18:55:11.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.985+0000 7f41e9600640 1 -- 172.21.15.155:0/2816130802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4156180 msgr2=0x7f41e4161070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.985+0000 7f41e9600640 1 --2- 172.21.15.155:0/2816130802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4156180 0x7f41e4161070 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009e50 tx=0x7f41dc02be50 comp rx=0 tx=0).stop 2024-03-28T18:55:11.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.986+0000 7f41e9600640 1 -- 172.21.15.155:0/2816130802 shutdown_connections 2024-03-28T18:55:11.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.986+0000 7f41e9600640 1 --2- 172.21.15.155:0/2816130802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4156180 0x7f41e4161070 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.986+0000 7f41e9600640 1 -- 172.21.15.155:0/2816130802 >> 172.21.15.155:0/2816130802 conn(0x7f41e40671f0 msgr2=0x7f41e4067600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:11.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.986+0000 7f41e9600640 1 -- 172.21.15.155:0/2816130802 shutdown_connections 2024-03-28T18:55:11.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.986+0000 7f41e9600640 1 -- 172.21.15.155:0/2816130802 wait complete. 2024-03-28T18:55:11.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.987+0000 7f41e9600640 1 Processor -- start 2024-03-28T18:55:11.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.987+0000 7f41e9600640 1 -- start start 2024-03-28T18:55:11.989 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e9600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41e40c8ff0 0x7f41e41eb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e9600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4152ec0 0x7f41e40ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e4156180 0x7f41e414c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e40c6060 con 0x7f41e4156180 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e9600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4158fb0 con 0x7f41e4152ec0 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e9600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e41565b0 con 0x7f41e40c8ff0 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41e40c8ff0 0x7f41e41eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4152ec0 0x7f41e40ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41e40c8ff0 0x7f41e41eb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:53506/0 (socket says 172.21.15.155:53506) 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4152ec0 0x7f41e40ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46258/0 (socket says 172.21.15.155:46258) 2024-03-28T18:55:11.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.988+0000 7f41e2a00640 1 -- 172.21.15.155:0/880667621 learned_addr learned my addr 172.21.15.155:0/880667621 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41e3400640 1 --2- 172.21.15.155:0/880667621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e4156180 0x7f41e414c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41e2000640 1 -- 172.21.15.155:0/880667621 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41e40c8ff0 msgr2=0x7f41e41eb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41e2000640 1 --2- 172.21.15.155:0/880667621 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41e40c8ff0 0x7f41e41eb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41e2000640 1 -- 172.21.15.155:0/880667621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e4156180 msgr2=0x7f41e414c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41e2000640 1 --2- 172.21.15.155:0/880667621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e4156180 0x7f41e414c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41e2000640 1 -- 172.21.15.155:0/880667621 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e41ebe20 con 0x7f41e4152ec0 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41e2000640 1 --2- 172.21.15.155:0/880667621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4152ec0 0x7f41e40ade30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41d000b3e0 tx=0x7f41d000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41daa00640 1 -- 172.21.15.155:0/880667621 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0015030 con 0x7f41e4152ec0 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41daa00640 1 -- 172.21.15.155:0/880667621 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d0004020 con 0x7f41e4152ec0 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e41ec100 con 0x7f41e4152ec0 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.989+0000 7f41daa00640 1 -- 172.21.15.155:0/880667621 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0004450 con 0x7f41e4152ec0 2024-03-28T18:55:11.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.990+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e41ec6c0 con 0x7f41e4152ec0 2024-03-28T18:55:11.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.990+0000 7f41daa00640 1 -- 172.21.15.155:0/880667621 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f41d0018070 con 0x7f41e4152ec0 2024-03-28T18:55:11.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.990+0000 7f41daa00640 1 -- 172.21.15.155:0/880667621 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f41d0011630 con 0x7f41e4152ec0 2024-03-28T18:55:11.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.991+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e404f470 con 0x7f41e4152ec0 2024-03-28T18:55:11.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:11.993+0000 7f41daa00640 1 -- 172.21.15.155:0/880667621 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f41d00035f0 con 0x7f41e4152ec0 2024-03-28T18:55:12.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.072+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd new", "uuid": "216c6c5e-1540-4217-8bba-88752ec61d5b", "id": 4} v 0) v1 -- 0x7f41e406d6d0 con 0x7f41e4152ec0 2024-03-28T18:55:12.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.079+0000 7f41daa00640 1 -- 172.21.15.155:0/880667621 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "216c6c5e-1540-4217-8bba-88752ec61d5b", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f41d0003af0 con 0x7f41e4152ec0 2024-03-28T18:55:12.079 INFO:teuthology.orchestra.run.smithi155.stdout:4 2024-03-28T18:55:12.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4152ec0 msgr2=0x7f41e40ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 --2- 172.21.15.155:0/880667621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4152ec0 0x7f41e40ade30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41d000b3e0 tx=0x7f41d000b8a0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 shutdown_connections 2024-03-28T18:55:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 --2- 172.21.15.155:0/880667621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e4156180 0x7f41e414c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 --2- 172.21.15.155:0/880667621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41e4152ec0 0x7f41e40ade30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 --2- 172.21.15.155:0/880667621 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41e40c8ff0 0x7f41e41eb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 >> 172.21.15.155:0/880667621 conn(0x7f41e40671f0 msgr2=0x7f41e40cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 shutdown_connections 2024-03-28T18:55:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.080+0000 7f41e9600640 1 -- 172.21.15.155:0/880667621 wait complete. 2024-03-28T18:55:12.095 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 02feaf2d-6bbd-4042-b802-acdb75512e88 5 2024-03-28T18:55:12.101 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:12.101+0000 7effbea13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:55:12.137 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:12.137+0000 7f4325438040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:55:12.270 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:12.271+0000 7effbea13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:55:12.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.275+0000 7ffb94a00640 1 Processor -- start 2024-03-28T18:55:12.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.275+0000 7ffb94a00640 1 -- start start 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.276+0000 7ffb94a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb900c8db0 0x7ffb90156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.276+0000 7ffb94a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb90152840 0x7ffb90152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.276+0000 7ffb94a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb900735f0 con 0x7ffb900c5aa0 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.276+0000 7ffb94a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb901565d0 con 0x7ffb90152840 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.276+0000 7ffb94a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb900c82d0 con 0x7ffb900c8db0 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.276+0000 7ffb8f200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb90152840 0x7ffb90152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.277+0000 7ffb87e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb900c8db0 0x7ffb90156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.277+0000 7ffb8f200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb90152840 0x7ffb90152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46268/0 (socket says 172.21.15.155:46268) 2024-03-28T18:55:12.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.277+0000 7ffb8f200640 1 -- 172.21.15.155:0/98757600 learned_addr learned my addr 172.21.15.155:0/98757600 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:12.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.277+0000 7ffb8f200640 1 -- 172.21.15.155:0/98757600 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb900c8db0 msgr2=0x7ffb90156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:12.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.277+0000 7ffb8f200640 1 --2- 172.21.15.155:0/98757600 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb900c8db0 0x7ffb90156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.277+0000 7ffb8f200640 1 -- 172.21.15.155:0/98757600 >> v1:172.21.15.27:6789/0 conn(0x7ffb900c5aa0 legacy=0x7ffb90072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:12.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.277+0000 7ffb8f200640 1 -- 172.21.15.155:0/98757600 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb900c7d10 con 0x7ffb90152840 2024-03-28T18:55:12.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.277+0000 7ffb87e00640 1 --2- 172.21.15.155:0/98757600 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb900c8db0 0x7ffb90156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:12.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.278+0000 7ffb8f200640 1 --2- 172.21.15.155:0/98757600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb90152840 0x7ffb90152cd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffb900c76a0 tx=0x7ffb8802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf3ea24ef52cbce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:12.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.278+0000 7ffb8de00640 1 -- 172.21.15.155:0/98757600 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8802c9a0 con 0x7ffb90152840 2024-03-28T18:55:12.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.278+0000 7ffb8de00640 1 -- 172.21.15.155:0/98757600 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8802cb40 con 0x7ffb90152840 2024-03-28T18:55:12.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.278+0000 7ffb8de00640 1 -- 172.21.15.155:0/98757600 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb880343f0 con 0x7ffb90152840 2024-03-28T18:55:12.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.279+0000 7ffb94a00640 1 -- 172.21.15.155:0/98757600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb90152840 msgr2=0x7ffb90152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.279+0000 7ffb94a00640 1 --2- 172.21.15.155:0/98757600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb90152840 0x7ffb90152cd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffb900c76a0 tx=0x7ffb8802be00 comp rx=0 tx=0).stop 2024-03-28T18:55:12.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.279+0000 7ffb94a00640 1 -- 172.21.15.155:0/98757600 shutdown_connections 2024-03-28T18:55:12.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.279+0000 7ffb94a00640 1 --2- 172.21.15.155:0/98757600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb90152840 0x7ffb90152cd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.279+0000 7ffb94a00640 1 --2- 172.21.15.155:0/98757600 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb900c8db0 0x7ffb90156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.279+0000 7ffb94a00640 1 -- 172.21.15.155:0/98757600 >> 172.21.15.155:0/98757600 conn(0x7ffb9006b0c0 msgr2=0x7ffb9006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:12.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.280+0000 7ffb94a00640 1 -- 172.21.15.155:0/98757600 shutdown_connections 2024-03-28T18:55:12.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.280+0000 7ffb94a00640 1 -- 172.21.15.155:0/98757600 wait complete. 2024-03-28T18:55:12.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.281+0000 7ffb94a00640 1 Processor -- start 2024-03-28T18:55:12.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.281+0000 7ffb94a00640 1 -- start start 2024-03-28T18:55:12.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb94a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb900c5aa0 0x7ffb901eeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb94a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb900c8db0 0x7ffb901eb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb94a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb90152840 0x7ffb900afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb94a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb90072100 con 0x7ffb900c8db0 2024-03-28T18:55:12.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb94a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb90071e20 con 0x7ffb900c5aa0 2024-03-28T18:55:12.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb94a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb90072d30 con 0x7ffb90152840 2024-03-28T18:55:12.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb8e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb900c5aa0 0x7ffb901eeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb8f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb90152840 0x7ffb900afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb8f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb90152840 0x7ffb900afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:53520/0 (socket says 172.21.15.155:53520) 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.282+0000 7ffb8f200640 1 -- 172.21.15.155:0/1360594707 learned_addr learned my addr 172.21.15.155:0/1360594707 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb87e00640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb900c8db0 0x7ffb901eb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb8f200640 1 -- 172.21.15.155:0/1360594707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb900c5aa0 msgr2=0x7ffb901eeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb8f200640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb900c5aa0 0x7ffb901eeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb8f200640 1 -- 172.21.15.155:0/1360594707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb900c8db0 msgr2=0x7ffb901eb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb8f200640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb900c8db0 0x7ffb901eb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb8f200640 1 -- 172.21.15.155:0/1360594707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb901ebc20 con 0x7ffb90152840 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb8e800640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb900c5aa0 0x7ffb901eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb8f200640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb90152840 0x7ffb900afe40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffb88009f70 tx=0x7ffb88004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb86a00640 1 -- 172.21.15.155:0/1360594707 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88043020 con 0x7ffb90152840 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb86a00640 1 -- 172.21.15.155:0/1360594707 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8803e450 con 0x7ffb90152840 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb900bebd0 con 0x7ffb90152840 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.283+0000 7ffb86a00640 1 -- 172.21.15.155:0/1360594707 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88009510 con 0x7ffb90152840 2024-03-28T18:55:12.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.284+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb900bf150 con 0x7ffb90152840 2024-03-28T18:55:12.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.284+0000 7ffb86a00640 1 -- 172.21.15.155:0/1360594707 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ffb88046020 con 0x7ffb90152840 2024-03-28T18:55:12.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.284+0000 7ffb86a00640 1 -- 172.21.15.155:0/1360594707 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7ffb88004020 con 0x7ffb90152840 2024-03-28T18:55:12.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.285+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9004f400 con 0x7ffb90152840 2024-03-28T18:55:12.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.287+0000 7ffb86a00640 1 -- 172.21.15.155:0/1360594707 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ffb8804d070 con 0x7ffb90152840 2024-03-28T18:55:12.309 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:12.309+0000 7f4325438040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:55:12.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.366+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "02feaf2d-6bbd-4042-b802-acdb75512e88", "id": 5} v 0) v1 -- 0x7ffb90152fc0 con 0x7ffb90152840 2024-03-28T18:55:12.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.376+0000 7ffb86a00640 1 -- 172.21.15.155:0/1360594707 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "02feaf2d-6bbd-4042-b802-acdb75512e88", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffb901ebc20 con 0x7ffb90152840 2024-03-28T18:55:12.376 INFO:teuthology.orchestra.run.smithi155.stdout:5 2024-03-28T18:55:12.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb90152840 msgr2=0x7ffb900afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb90152840 0x7ffb900afe40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffb88009f70 tx=0x7ffb88004a40 comp rx=0 tx=0).stop 2024-03-28T18:55:12.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 shutdown_connections 2024-03-28T18:55:12.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb90152840 0x7ffb900afe40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb900c8db0 0x7ffb901eb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 --2- 172.21.15.155:0/1360594707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb900c5aa0 0x7ffb901eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 >> 172.21.15.155:0/1360594707 conn(0x7ffb9006b0c0 msgr2=0x7ffb900aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:12.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 shutdown_connections 2024-03-28T18:55:12.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.377+0000 7ffb94a00640 1 -- 172.21.15.155:0/1360594707 wait complete. 2024-03-28T18:55:12.394 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 756afc4a-1b8b-4d12-bd43-65e352c782b5 6 2024-03-28T18:55:12.421 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:12.421+0000 7effbea13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:55:12.457 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:12.456+0000 7f4325438040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:55:12.477 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:12.477+0000 7effbea13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:55:12.516 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:12.515+0000 7f4325438040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:55:12.610 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:12.611+0000 7effbea13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:55:12.627 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.627+0000 7f84a4e00640 1 Processor -- start 2024-03-28T18:55:12.627 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.627+0000 7f84a4e00640 1 -- start start 2024-03-28T18:55:12.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f84a4e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84a00c8db0 0x7f84a0156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f84a4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a0152840 0x7f84a0152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f84a4e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a00735f0 con 0x7f84a00c5aa0 2024-03-28T18:55:12.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f84a4e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a01565d0 con 0x7f84a00c8db0 2024-03-28T18:55:12.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f84a4e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00c82d0 con 0x7f84a0152840 2024-03-28T18:55:12.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f849f400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f84a00c5aa0 0x7f84a0072620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:51808/0 (socket says 172.21.15.155:51808) 2024-03-28T18:55:12.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f849f400640 1 -- 172.21.15.155:0/1236161831 learned_addr learned my addr 172.21.15.155:0/1236161831 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-28T18:55:12.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f849fe00640 1 --2- 172.21.15.155:0/1236161831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a0152840 0x7f84a0152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.629+0000 7f849ea00640 1 --2- 172.21.15.155:0/1236161831 >> v2:172.21.15.155:3300/0 conn(0x7f84a00c8db0 0x7f84a0156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.630+0000 7f849fe00640 1 -- 172.21.15.155:0/1236161831 >> v2:172.21.15.155:3300/0 conn(0x7f84a00c8db0 msgr2=0x7f84a0156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:12.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.630+0000 7f849fe00640 1 --2- 172.21.15.155:0/1236161831 >> v2:172.21.15.155:3300/0 conn(0x7f84a00c8db0 0x7f84a0156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.630+0000 7f849e000640 1 -- 172.21.15.155:0/1236161831 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243375535 0 0) 0x7f84a00735f0 con 0x7f84a00c5aa0 2024-03-28T18:55:12.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.630+0000 7f849fe00640 1 -- 172.21.15.155:0/1236161831 >> v1:172.21.15.155:6789/0 conn(0x7f84a00c5aa0 legacy=0x7f84a0072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.630+0000 7f849fe00640 1 -- 172.21.15.155:0/1236161831 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a00c7d10 con 0x7f84a0152840 2024-03-28T18:55:12.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.631+0000 7f849fe00640 1 --2- 172.21.15.155:0/1236161831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a0152840 0x7f84a0152cd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8498009d90 tx=0x7f849802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb75bc214f6236b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:12.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f849e000640 1 -- 172.21.15.155:0/1236161831 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498039070 con 0x7f84a0152840 2024-03-28T18:55:12.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f849e000640 1 -- 172.21.15.155:0/1236161831 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849802cbc0 con 0x7f84a0152840 2024-03-28T18:55:12.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f849e000640 1 -- 172.21.15.155:0/1236161831 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498034440 con 0x7f84a0152840 2024-03-28T18:55:12.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f84a4e00640 1 -- 172.21.15.155:0/1236161831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a0152840 msgr2=0x7f84a0152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f84a4e00640 1 --2- 172.21.15.155:0/1236161831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a0152840 0x7f84a0152cd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8498009d90 tx=0x7f849802be10 comp rx=0 tx=0).stop 2024-03-28T18:55:12.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f84a4e00640 1 -- 172.21.15.155:0/1236161831 shutdown_connections 2024-03-28T18:55:12.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f84a4e00640 1 --2- 172.21.15.155:0/1236161831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a0152840 0x7f84a0152cd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f84a4e00640 1 --2- 172.21.15.155:0/1236161831 >> v2:172.21.15.155:3300/0 conn(0x7f84a00c8db0 0x7f84a0156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f84a4e00640 1 -- 172.21.15.155:0/1236161831 >> 172.21.15.155:0/1236161831 conn(0x7f84a006b0c0 msgr2=0x7f84a006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:12.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f84a4e00640 1 -- 172.21.15.155:0/1236161831 shutdown_connections 2024-03-28T18:55:12.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.632+0000 7f84a4e00640 1 -- 172.21.15.155:0/1236161831 wait complete. 2024-03-28T18:55:12.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.634+0000 7f84a4e00640 1 Processor -- start 2024-03-28T18:55:12.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.634+0000 7f84a4e00640 1 -- start start 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f84a4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a00c5aa0 0x7f84a00bf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f84a4e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84a00c8db0 0x7f84a00c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f84a4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84a0152840 0x7f84a01ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f84a4e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0072100 con 0x7f84a0152840 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f84a4e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0071e20 con 0x7f84a00c8db0 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f84a4e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0072d30 con 0x7f84a00c5aa0 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f849f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a00c5aa0 0x7f84a00bf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f849ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84a00c8db0 0x7f84a00c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f849ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84a00c8db0 0x7f84a00c49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46316/0 (socket says 172.21.15.155:46316) 2024-03-28T18:55:12.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f849f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a00c5aa0 0x7f84a00bf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:53542/0 (socket says 172.21.15.155:53542) 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.635+0000 7f849ea00640 1 -- 172.21.15.155:0/1072958816 learned_addr learned my addr 172.21.15.155:0/1072958816 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f849f400640 1 -- 172.21.15.155:0/1072958816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84a00c8db0 msgr2=0x7f84a00c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f849f400640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84a00c8db0 0x7f84a00c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f849f400640 1 -- 172.21.15.155:0/1072958816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84a0152840 msgr2=0x7f84a01ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f849fe00640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84a0152840 0x7f84a01ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f849f400640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84a0152840 0x7f84a01ee190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f849f400640 1 -- 172.21.15.155:0/1072958816 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a00c21d0 con 0x7f84a00c5aa0 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f849ea00640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84a00c8db0 0x7f84a00c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f849f400640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a00c5aa0 0x7f84a00bf300 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f849000c450 tx=0x7f849000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:12.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f8497400640 1 -- 172.21.15.155:0/1072958816 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490018070 con 0x7f84a00c5aa0 2024-03-28T18:55:12.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f8497400640 1 -- 172.21.15.155:0/1072958816 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84900074e0 con 0x7f84a00c5aa0 2024-03-28T18:55:12.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f8497400640 1 -- 172.21.15.155:0/1072958816 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490004a20 con 0x7f84a00c5aa0 2024-03-28T18:55:12.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a00c24b0 con 0x7f84a00c5aa0 2024-03-28T18:55:12.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.636+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a00c2a70 con 0x7f84a00c5aa0 2024-03-28T18:55:12.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.637+0000 7f8497400640 1 -- 172.21.15.155:0/1072958816 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f849001b070 con 0x7f84a00c5aa0 2024-03-28T18:55:12.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.637+0000 7f8497400640 1 -- 172.21.15.155:0/1072958816 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f84900129f0 con 0x7f84a00c5aa0 2024-03-28T18:55:12.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.637+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a004f400 con 0x7f84a00c5aa0 2024-03-28T18:55:12.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.640+0000 7f8497400640 1 -- 172.21.15.155:0/1072958816 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f849001d440 con 0x7f84a00c5aa0 2024-03-28T18:55:12.654 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:12.654+0000 7f4325438040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:55:12.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.718+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "756afc4a-1b8b-4d12-bd43-65e352c782b5", "id": 6} v 0) v1 -- 0x7f84a00c2cb0 con 0x7f84a00c5aa0 2024-03-28T18:55:12.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.725+0000 7f8497400640 1 -- 172.21.15.155:0/1072958816 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "756afc4a-1b8b-4d12-bd43-65e352c782b5", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f849001d770 con 0x7f84a00c5aa0 2024-03-28T18:55:12.725 INFO:teuthology.orchestra.run.smithi155.stdout:6 2024-03-28T18:55:12.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a00c5aa0 msgr2=0x7f84a00bf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a00c5aa0 0x7f84a00bf300 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f849000c450 tx=0x7f849000c910 comp rx=0 tx=0).stop 2024-03-28T18:55:12.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 shutdown_connections 2024-03-28T18:55:12.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84a0152840 0x7f84a01ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84a00c8db0 0x7f84a00c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 --2- 172.21.15.155:0/1072958816 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84a00c5aa0 0x7f84a00bf300 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 >> 172.21.15.155:0/1072958816 conn(0x7f84a006b0c0 msgr2=0x7f84a006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:12.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 shutdown_connections 2024-03-28T18:55:12.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.726+0000 7f84a4e00640 1 -- 172.21.15.155:0/1072958816 wait complete. 2024-03-28T18:55:12.728 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:12.728+0000 7effbea13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:55:12.743 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 2b8d85a2-522a-40eb-b801-09e03ed25530 7 2024-03-28T18:55:12.774 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:12.773+0000 7f4325438040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:55:12.913 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:12.913+0000 7effbea13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:55:12.962 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:12.962+0000 7f4325438040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:55:12.965 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:12.965+0000 7effbea13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:55:12.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.966+0000 7fe585800640 1 Processor -- start 2024-03-28T18:55:12.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.966+0000 7fe585800640 1 -- start start 2024-03-28T18:55:12.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.967+0000 7fe585800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe580156b30 0x7fe5801592f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.967+0000 7fe585800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5800ad180 con 0x7fe58014f770 2024-03-28T18:55:12.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.967+0000 7fe585800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5800ad460 con 0x7fe5800be710 2024-03-28T18:55:12.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.967+0000 7fe585800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58014fb50 con 0x7fe580156b30 2024-03-28T18:55:12.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.967+0000 7fe57e000640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe5800be710 0x7fe5800c5970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:51814/0 (socket says 172.21.15.155:51814) 2024-03-28T18:55:12.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.967+0000 7fe57e000640 1 -- 172.21.15.155:0/81305157 learned_addr learned my addr 172.21.15.155:0/81305157 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-28T18:55:12.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57f400640 1 --2- 172.21.15.155:0/81305157 >> v2:172.21.15.27:3300/0 conn(0x7fe580156b30 0x7fe5801592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245147730 0 0) 0x7fe5800ad460 con 0x7fe5800be710 2024-03-28T18:55:12.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe564003520 con 0x7fe5800be710 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933686717 0 0) 0x7fe564003520 con 0x7fe5800be710 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 >> v2:172.21.15.27:3300/0 conn(0x7fe580156b30 msgr2=0x7fe5801592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 --2- 172.21.15.155:0/81305157 >> v2:172.21.15.27:3300/0 conn(0x7fe580156b30 0x7fe5801592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 >> v1:172.21.15.27:6789/0 conn(0x7fe58014f770 legacy=0x7fe5801560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5800ad740 con 0x7fe5800be710 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe56c001670 con 0x7fe5800be710 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 >> v1:172.21.15.155:6789/0 conn(0x7fe5800be710 legacy=0x7fe5800c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe564003a70 0x7fe564003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564004800 0x7fe564004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5640053c0 0x7fe56400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58014fb50 con 0x7fe564004800 2024-03-28T18:55:12.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56400c280 con 0x7fe564003a70 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.968+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56400c480 con 0x7fe5640053c0 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57ea00640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe564003a70 0x7fe564003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57f400640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5640053c0 0x7fe56400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57e000640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564004800 0x7fe564004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57ea00640 1 -- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5640053c0 msgr2=0x7fe56400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57ea00640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5640053c0 0x7fe56400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57ea00640 1 -- 172.21.15.155:0/81305157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564004800 msgr2=0x7fe564004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57ea00640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564004800 0x7fe564004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57ea00640 1 -- 172.21.15.155:0/81305157 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564003520 con 0x7fe564003a70 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.969+0000 7fe57f400640 1 -- 172.21.15.155:0/81305157 reap_dead start 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.970+0000 7fe57ea00640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe564003a70 0x7fe564003eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe570001750 tx=0x7fe564005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c822bce920b93421 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:12.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.970+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570001370 con 0x7fe564003a70 2024-03-28T18:55:12.973 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.970+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5700052d0 con 0x7fe564003a70 2024-03-28T18:55:12.973 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.970+0000 7fe57d600640 1 -- 172.21.15.155:0/81305157 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570003830 con 0x7fe564003a70 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.971+0000 7fe585800640 1 -- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe564003a70 msgr2=0x7fe564003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.971+0000 7fe585800640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe564003a70 0x7fe564003eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe570001750 tx=0x7fe564005830 comp rx=0 tx=0).stop 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.971+0000 7fe585800640 1 -- 172.21.15.155:0/81305157 shutdown_connections 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.971+0000 7fe585800640 1 --2- 172.21.15.155:0/81305157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe564003a70 0x7fe564003eb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.971+0000 7fe585800640 1 -- 172.21.15.155:0/81305157 >> 172.21.15.155:0/81305157 conn(0x7fe5800671f0 msgr2=0x7fe580067600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.971+0000 7fe585800640 1 -- 172.21.15.155:0/81305157 shutdown_connections 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.971+0000 7fe585800640 1 -- 172.21.15.155:0/81305157 wait complete. 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.973+0000 7fe585800640 1 Processor -- start 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.973+0000 7fe585800640 1 -- start start 2024-03-28T18:55:12.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe585800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564003a70 0x7fe5800bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe585800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5800c09f0 0x7fe5800c1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe585800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5800c8dd0 0x7fe580148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe585800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe580156240 con 0x7fe564003a70 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe585800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801560c0 con 0x7fe5800c09f0 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe585800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801563c0 con 0x7fe5800c8dd0 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe57e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5800c09f0 0x7fe5800c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe57e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5800c09f0 0x7fe5800c1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:46330/0 (socket says 172.21.15.155:46330) 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe57e000640 1 -- 172.21.15.155:0/1746407388 learned_addr learned my addr 172.21.15.155:0/1746407388 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe57f400640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5800c8dd0 0x7fe580148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.974+0000 7fe57ea00640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564003a70 0x7fe5800bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe57e000640 1 -- 172.21.15.155:0/1746407388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5800c8dd0 msgr2=0x7fe580148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe57e000640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5800c8dd0 0x7fe580148260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe57e000640 1 -- 172.21.15.155:0/1746407388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564003a70 msgr2=0x7fe5800bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe57e000640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564003a70 0x7fe5800bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe57e000640 1 -- 172.21.15.155:0/1746407388 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe580153c90 con 0x7fe5800c09f0 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe57e000640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5800c09f0 0x7fe5800c1580 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00e470 tx=0x7fe56c00e930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe576a00640 1 -- 172.21.15.155:0/1746407388 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c012590 con 0x7fe5800c09f0 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe576a00640 1 -- 172.21.15.155:0/1746407388 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c018070 con 0x7fe5800c09f0 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5800beb20 con 0x7fe5800c09f0 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe576a00640 1 -- 172.21.15.155:0/1746407388 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c012990 con 0x7fe5800c09f0 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.975+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5800bf0e0 con 0x7fe5800c09f0 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.976+0000 7fe576a00640 1 -- 172.21.15.155:0/1746407388 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe56c012b30 con 0x7fe5800c09f0 2024-03-28T18:55:12.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.976+0000 7fe576a00640 1 -- 172.21.15.155:0/1746407388 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fe56c012e20 con 0x7fe5800c09f0 2024-03-28T18:55:12.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.976+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58004f470 con 0x7fe5800c09f0 2024-03-28T18:55:12.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:12.979+0000 7fe576a00640 1 -- 172.21.15.155:0/1746407388 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe56c01db50 con 0x7fe5800c09f0 2024-03-28T18:55:13.018 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:13.018+0000 7f4325438040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:55:13.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.057+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2b8d85a2-522a-40eb-b801-09e03ed25530", "id": 7} v 0) v1 -- 0x7fe58006d510 con 0x7fe5800c09f0 2024-03-28T18:55:13.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.064+0000 7fe576a00640 1 -- 172.21.15.155:0/1746407388 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b8d85a2-522a-40eb-b801-09e03ed25530", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe56c01d4c0 con 0x7fe5800c09f0 2024-03-28T18:55:13.064 INFO:teuthology.orchestra.run.smithi155.stdout:7 2024-03-28T18:55:13.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5800c09f0 msgr2=0x7fe5800c1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:13.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5800c09f0 0x7fe5800c1580 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00e470 tx=0x7fe56c00e930 comp rx=0 tx=0).stop 2024-03-28T18:55:13.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 shutdown_connections 2024-03-28T18:55:13.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5800c8dd0 0x7fe580148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5800c09f0 0x7fe5800c1580 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 --2- 172.21.15.155:0/1746407388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe564003a70 0x7fe5800bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 >> 172.21.15.155:0/1746407388 conn(0x7fe5800671f0 msgr2=0x7fe58014e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:13.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 shutdown_connections 2024-03-28T18:55:13.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:55:13.066+0000 7fe585800640 1 -- 172.21.15.155:0/1746407388 wait complete. 2024-03-28T18:55:13.082 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-28T18:55:13.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-28T18:55:13.090 INFO:tasks.ceph.osd.0:Started 2024-03-28T18:55:13.090 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-28T18:55:13.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-28T18:55:13.097 INFO:tasks.ceph.osd.1:Started 2024-03-28T18:55:13.097 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-28T18:55:13.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-28T18:55:13.102 INFO:tasks.ceph.osd.2:Started 2024-03-28T18:55:13.102 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-28T18:55:13.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-28T18:55:13.107 INFO:tasks.ceph.osd.3:Started 2024-03-28T18:55:13.107 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-28T18:55:13.108 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-28T18:55:13.111 INFO:tasks.ceph.osd.4:Started 2024-03-28T18:55:13.112 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-28T18:55:13.112 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-28T18:55:13.116 INFO:tasks.ceph.osd.5:Started 2024-03-28T18:55:13.116 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-28T18:55:13.116 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-28T18:55:13.120 INFO:tasks.ceph.osd.6:Started 2024-03-28T18:55:13.121 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-28T18:55:13.121 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-28T18:55:13.126 INFO:tasks.ceph.osd.7:Started 2024-03-28T18:55:13.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:13.170 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:13.168+0000 7effbea13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:55:13.226 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:13.226+0000 7f4325438040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:55:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.365+0000 7f736ea00640 1 Processor -- start 2024-03-28T18:55:13.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.365+0000 7f736ea00640 1 -- start start 2024-03-28T18:55:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.366+0000 7f736ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368152e70 0x7f7368156c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.366+0000 7f736ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73680c2d80 0x7f736806fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f736ea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7368070270 con 0x7f7368156060 2024-03-28T18:55:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f736ea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7368070510 con 0x7f73680c2d80 2024-03-28T18:55:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f736ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73680707f0 con 0x7f7368152e70 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f735fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73680c2d80 0x7f736806fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f736ca00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7368156060 0x7f73680c8d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34372/0 (socket says 172.21.15.27:34372) 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f736ca00640 1 -- 172.21.15.27:0/2961797704 learned_addr learned my addr 172.21.15.27:0/2961797704 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f7367e00640 1 --2- 172.21.15.27:0/2961797704 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368152e70 0x7f7368156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f735fe00640 1 -- 172.21.15.27:0/2961797704 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368152e70 msgr2=0x7f7368156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f735fe00640 1 --2- 172.21.15.27:0/2961797704 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368152e70 0x7f7368156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f735fe00640 1 -- 172.21.15.27:0/2961797704 >> v1:172.21.15.27:6789/0 conn(0x7f7368156060 legacy=0x7f73680c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f7367400640 1 -- 172.21.15.27:0/2961797704 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148415431 0 0) 0x7f7368070270 con 0x7f7368156060 2024-03-28T18:55:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.367+0000 7f735fe00640 1 -- 172.21.15.27:0/2961797704 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7368070ad0 con 0x7f73680c2d80 2024-03-28T18:55:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.369+0000 7f735fe00640 1 --2- 172.21.15.27:0/2961797704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f736806fd30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f73500097a0 tx=0x7f735002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca43393cddf1798 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.369+0000 7f7367400640 1 -- 172.21.15.27:0/2961797704 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73500315a0 con 0x7f73680c2d80 2024-03-28T18:55:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.369+0000 7f7367400640 1 -- 172.21.15.27:0/2961797704 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7350037070 con 0x7f73680c2d80 2024-03-28T18:55:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.369+0000 7f7367400640 1 -- 172.21.15.27:0/2961797704 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350031960 con 0x7f73680c2d80 2024-03-28T18:55:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.370+0000 7f736ea00640 1 -- 172.21.15.27:0/2961797704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 msgr2=0x7f736806fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.370+0000 7f736ea00640 1 --2- 172.21.15.27:0/2961797704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f736806fd30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f73500097a0 tx=0x7f735002b840 comp rx=0 tx=0).stop 2024-03-28T18:55:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.370+0000 7f736ea00640 1 -- 172.21.15.27:0/2961797704 shutdown_connections 2024-03-28T18:55:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.370+0000 7f736ea00640 1 --2- 172.21.15.27:0/2961797704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f736806fd30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.370+0000 7f736ea00640 1 --2- 172.21.15.27:0/2961797704 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368152e70 0x7f7368156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.370+0000 7f736ea00640 1 -- 172.21.15.27:0/2961797704 >> 172.21.15.27:0/2961797704 conn(0x7f7368066dd0 msgr2=0x7f73680671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:13.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.370+0000 7f736ea00640 1 -- 172.21.15.27:0/2961797704 shutdown_connections 2024-03-28T18:55:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.370+0000 7f736ea00640 1 -- 172.21.15.27:0/2961797704 wait complete. 2024-03-28T18:55:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.372+0000 7f736ea00640 1 Processor -- start 2024-03-28T18:55:13.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.372+0000 7f736ea00640 1 -- start start 2024-03-28T18:55:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f736ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f73681579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f736ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7368152e70 0x7f73681433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f736ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368156060 0x7f73680ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f736ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73680c9480 con 0x7f73680c2d80 2024-03-28T18:55:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f736ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7368155b40 con 0x7f7368152e70 2024-03-28T18:55:13.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f736ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73680c9600 con 0x7f7368156060 2024-03-28T18:55:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f7367e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f73681579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f7367e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f73681579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49698/0 (socket says 172.21.15.27:49698) 2024-03-28T18:55:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f7367e00640 1 -- 172.21.15.27:0/1828316968 learned_addr learned my addr 172.21.15.27:0/1828316968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f735fe00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7368152e70 0x7f73681433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.373+0000 7f736ca00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368156060 0x7f73680ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:13.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7367e00640 1 -- 172.21.15.27:0/1828316968 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368156060 msgr2=0x7f73680ace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7367e00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368156060 0x7f73680ace50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7367e00640 1 -- 172.21.15.27:0/1828316968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7368152e70 msgr2=0x7f73681433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7367e00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7368152e70 0x7f73681433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7367e00640 1 -- 172.21.15.27:0/1828316968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7368143930 con 0x7f73680c2d80 2024-03-28T18:55:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7367e00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f73681579c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f735800e870 tx=0x7f735800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7364c00640 1 -- 172.21.15.27:0/1828316968 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358017070 con 0x7f73680c2d80 2024-03-28T18:55:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7364c00640 1 -- 172.21.15.27:0/1828316968 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735800c940 con 0x7f73680c2d80 2024-03-28T18:55:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f7364c00640 1 -- 172.21.15.27:0/1828316968 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735800cd30 con 0x7f73680c2d80 2024-03-28T18:55:13.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.374+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7368156f00 con 0x7f73680c2d80 2024-03-28T18:55:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.375+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73680ac310 con 0x7f73680c2d80 2024-03-28T18:55:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.375+0000 7f7364c00640 1 -- 172.21.15.27:0/1828316968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f735801a070 con 0x7f73680c2d80 2024-03-28T18:55:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.375+0000 7f7364c00640 1 -- 172.21.15.27:0/1828316968 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7358006020 con 0x7f73680c2d80 2024-03-28T18:55:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.376+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736804f400 con 0x7f73680c2d80 2024-03-28T18:55:13.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.378+0000 7f7364c00640 1 -- 172.21.15.27:0/1828316968 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7368143930 con 0x7f73680c2d80 2024-03-28T18:55:13.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.444+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f736806e2a0 con 0x7f73680c2d80 2024-03-28T18:55:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.445+0000 7f7364c00640 1 -- 172.21.15.27:0/1828316968 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f73580064d0 con 0x7f73680c2d80 2024-03-28T18:55:13.444 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:13.444 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:13.060606+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 msgr2=0x7f73681579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f73681579c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f735800e870 tx=0x7f735800ed30 comp rx=0 tx=0).stop 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 shutdown_connections 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7368156060 0x7f73680ace50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7368152e70 0x7f73681433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 --2- 172.21.15.27:0/1828316968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73680c2d80 0x7f73681579c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 >> 172.21.15.27:0/1828316968 conn(0x7f7368066dd0 msgr2=0x7f73680c71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 shutdown_connections 2024-03-28T18:55:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:13.446+0000 7f736ea00640 1 -- 172.21.15.27:0/1828316968 wait complete. 2024-03-28T18:55:13.462 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-28T18:55:13.462 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-28T18:55:13.801 INFO:tasks.ceph.osd.4.smithi155.stderr:2024-03-28T18:55:13.801+0000 7f46aa468640 -1 Falling back to public interface 2024-03-28T18:55:13.809 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-03-28T18:55:13.809+0000 7fed31489640 -1 Falling back to public interface 2024-03-28T18:55:13.812 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-03-28T18:55:13.812+0000 7f7dc6f26640 -1 Falling back to public interface 2024-03-28T18:55:13.824 INFO:tasks.ceph.osd.5.smithi155.stderr:2024-03-28T18:55:13.824+0000 7f88d5b16640 -1 Falling back to public interface 2024-03-28T18:55:13.831 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-03-28T18:55:13.832+0000 7f94f9419640 -1 Falling back to public interface 2024-03-28T18:55:13.838 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-03-28T18:55:13.838+0000 7f302e7e1640 -1 Falling back to public interface 2024-03-28T18:55:13.838 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-03-28T18:55:13.839+0000 7fee14e11640 -1 Falling back to public interface 2024-03-28T18:55:13.846 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-03-28T18:55:13.847+0000 7f8287c0e640 -1 Falling back to public interface 2024-03-28T18:55:14.268 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:14.313 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:14.314+0000 7effbea13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:55:14.383 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:14.383+0000 7f4325438040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:55:14.430 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.431+0000 7fb72b200640 1 Processor -- start 2024-03-28T18:55:14.430 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.431+0000 7fb72b200640 1 -- start start 2024-03-28T18:55:14.432 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.432+0000 7fb72b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0be450 0x7fb72c06be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:14.432 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.432+0000 7fb72b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb72c0c8c80 0x7fb72c06c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:14.432 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.432+0000 7fb72b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c155f00 con 0x7fb72c152ca0 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.432+0000 7fb72b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c074a60 con 0x7fb72c0c8c80 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.432+0000 7fb72b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c1561e0 con 0x7fb72c0be450 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.432+0000 7fb729e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb72c152ca0 0x7fb72c0c5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34382/0 (socket says 172.21.15.27:34382) 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.432+0000 7fb729e00640 1 -- 172.21.15.27:0/3655381934 learned_addr learned my addr 172.21.15.27:0/3655381934 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.432+0000 7fb72a800640 1 --2- 172.21.15.27:0/3655381934 >> v2:172.21.15.27:3300/0 conn(0x7fb72c0c8c80 0x7fb72c06c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.433+0000 7fb729400640 1 --2- 172.21.15.27:0/3655381934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0be450 0x7fb72c06be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.433+0000 7fb728a00640 1 -- 172.21.15.27:0/3655381934 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921280088 0 0) 0x7fb72c155f00 con 0x7fb72c152ca0 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.433+0000 7fb72a800640 1 -- 172.21.15.27:0/3655381934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0be450 msgr2=0x7fb72c06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.433+0000 7fb72a800640 1 --2- 172.21.15.27:0/3655381934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0be450 0x7fb72c06be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.433+0000 7fb72a800640 1 -- 172.21.15.27:0/3655381934 >> v1:172.21.15.27:6789/0 conn(0x7fb72c152ca0 legacy=0x7fb72c0c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:14.433 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.433+0000 7fb72a800640 1 -- 172.21.15.27:0/3655381934 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c074410 con 0x7fb72c0c8c80 2024-03-28T18:55:14.434 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.434+0000 7fb72a800640 1 --2- 172.21.15.27:0/3655381934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c0c8c80 0x7fb72c06c3d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb724009d90 tx=0x7fb72402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a592221423b0ccce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:14.434 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.434+0000 7fb728a00640 1 -- 172.21.15.27:0/3655381934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724039070 con 0x7fb72c0c8c80 2024-03-28T18:55:14.434 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.434+0000 7fb728a00640 1 -- 172.21.15.27:0/3655381934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72402cbc0 con 0x7fb72c0c8c80 2024-03-28T18:55:14.434 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.434+0000 7fb728a00640 1 -- 172.21.15.27:0/3655381934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724034440 con 0x7fb72c0c8c80 2024-03-28T18:55:14.435 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.435+0000 7fb72b200640 1 -- 172.21.15.27:0/3655381934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c0c8c80 msgr2=0x7fb72c06c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:14.435 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.435+0000 7fb72b200640 1 --2- 172.21.15.27:0/3655381934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c0c8c80 0x7fb72c06c3d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb724009d90 tx=0x7fb72402be10 comp rx=0 tx=0).stop 2024-03-28T18:55:14.435 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.435+0000 7fb72b200640 1 -- 172.21.15.27:0/3655381934 shutdown_connections 2024-03-28T18:55:14.435 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.435+0000 7fb72b200640 1 --2- 172.21.15.27:0/3655381934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c0c8c80 0x7fb72c06c3d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:14.435 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.435+0000 7fb72b200640 1 --2- 172.21.15.27:0/3655381934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0be450 0x7fb72c06be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:14.435 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.435+0000 7fb72b200640 1 -- 172.21.15.27:0/3655381934 >> 172.21.15.27:0/3655381934 conn(0x7fb72c0cad80 msgr2=0x7fb72c0cb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:14.435 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.435+0000 7fb72b200640 1 -- 172.21.15.27:0/3655381934 shutdown_connections 2024-03-28T18:55:14.436 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.435+0000 7fb72b200640 1 -- 172.21.15.27:0/3655381934 wait complete. 2024-03-28T18:55:14.436 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.437+0000 7fb72b200640 1 Processor -- start 2024-03-28T18:55:14.436 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.437+0000 7fb72b200640 1 -- start start 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c0be450 0x7fb72c1ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0c8c80 0x7fb72c14be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c152ca0 0x7fb72c14c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c0c5e10 con 0x7fb72c152ca0 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c06d610 con 0x7fb72c0be450 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c0c5f90 con 0x7fb72c0c8c80 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c152ca0 0x7fb72c14c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c152ca0 0x7fb72c14c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49714/0 (socket says 172.21.15.27:49714) 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb72a800640 1 -- 172.21.15.27:0/3691863093 learned_addr learned my addr 172.21.15.27:0/3691863093 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:14.438 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb729400640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0c8c80 0x7fb72c14be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:14.439 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.438+0000 7fb729e00640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c0be450 0x7fb72c1ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb72a800640 1 -- 172.21.15.27:0/3691863093 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0c8c80 msgr2=0x7fb72c14be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb72a800640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0c8c80 0x7fb72c14be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb72a800640 1 -- 172.21.15.27:0/3691863093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c0be450 msgr2=0x7fb72c1ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb72a800640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c0be450 0x7fb72c1ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb72a800640 1 -- 172.21.15.27:0/3691863093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0c4c30 con 0x7fb72c152ca0 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb72a800640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c152ca0 0x7fb72c14c350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb724009eb0 tx=0x7fb724004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb722000640 1 -- 172.21.15.27:0/3691863093 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724039050 con 0x7fb72c152ca0 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb722000640 1 -- 172.21.15.27:0/3691863093 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724045070 con 0x7fb72c152ca0 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb722000640 1 -- 172.21.15.27:0/3691863093 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72403e830 con 0x7fb72c152ca0 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.439+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c1eeed0 con 0x7fb72c152ca0 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.440+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c157380 con 0x7fb72c152ca0 2024-03-28T18:55:14.440 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.440+0000 7fb722000640 1 -- 172.21.15.27:0/3691863093 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb72403e9d0 con 0x7fb72c152ca0 2024-03-28T18:55:14.441 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.440+0000 7fb722000640 1 -- 172.21.15.27:0/3691863093 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb724040b20 con 0x7fb72c152ca0 2024-03-28T18:55:14.441 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.441+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c1ef0d0 con 0x7fb72c152ca0 2024-03-28T18:55:14.442 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.443+0000 7fb722000640 1 -- 172.21.15.27:0/3691863093 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb72c1ef0d0 con 0x7fb72c152ca0 2024-03-28T18:55:14.511 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.511+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb72c075660 con 0x7fb72c152ca0 2024-03-28T18:55:14.512 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.513+0000 7fb722000640 1 -- 172.21.15.27:0/3691863093 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fb72404a460 con 0x7fb72c152ca0 2024-03-28T18:55:14.512 INFO:teuthology.misc.health.smithi027.stdout: 2024-03-28T18:55:14.512 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":10,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:13.060606+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c152ca0 msgr2=0x7fb72c14c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c152ca0 0x7fb72c14c350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb724009eb0 tx=0x7fb724004a40 comp rx=0 tx=0).stop 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 shutdown_connections 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb72c152ca0 0x7fb72c14c350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0c8c80 0x7fb72c14be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 --2- 172.21.15.27:0/3691863093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c0be450 0x7fb72c1ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 >> 172.21.15.27:0/3691863093 conn(0x7fb72c0cad80 msgr2=0x7fb72c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 shutdown_connections 2024-03-28T18:55:14.514 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:14.514+0000 7fb72b200640 1 -- 172.21.15.27:0/3691863093 wait complete. 2024-03-28T18:55:14.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T18:55:14.661 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:14.661+0000 7effbea13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:55:14.724 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:14.724+0000 7effbea13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:55:14.730 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:14.730+0000 7f4325438040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:55:14.790 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:14.790+0000 7f4325438040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:55:14.860 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:14.860+0000 7effbea13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:55:14.926 INFO:tasks.ceph.mgr.x.smithi155.stderr:2024-03-28T18:55:14.926+0000 7f4325438040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:55:15.905 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:15.905+0000 7eff5ee00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T18:55:15.908 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-03-28T18:55:15.908+0000 7eff5ee00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-28T18:55:17.150 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-03-28T18:55:17.150+0000 7f302e7e1640 -1 osd.7 0 log_to_monitors true 2024-03-28T18:55:17.154 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-03-28T18:55:17.155+0000 7fee14e11640 -1 osd.0 0 log_to_monitors true 2024-03-28T18:55:17.339 INFO:tasks.ceph.osd.4.smithi155.stderr:2024-03-28T18:55:17.339+0000 7f46aa468640 -1 osd.4 0 log_to_monitors true 2024-03-28T18:55:17.373 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-03-28T18:55:17.374+0000 7f94f9419640 -1 osd.1 0 log_to_monitors true 2024-03-28T18:55:17.385 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-03-28T18:55:17.385+0000 7fed31489640 -1 osd.6 0 log_to_monitors true 2024-03-28T18:55:17.397 INFO:tasks.ceph.osd.5.smithi155.stderr:2024-03-28T18:55:17.398+0000 7f88d5b16640 -1 osd.5 0 log_to_monitors true 2024-03-28T18:55:17.420 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-03-28T18:55:17.421+0000 7f8287c0e640 -1 osd.2 0 log_to_monitors true 2024-03-28T18:55:17.607 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-03-28T18:55:17.608+0000 7f7dc6f26640 -1 osd.3 0 log_to_monitors true 2024-03-28T18:55:21.130 INFO:tasks.ceph.osd.5.smithi155.stderr:2024-03-28T18:55:21.130+0000 7f88d1400640 -1 osd.5 0 waiting for initial osdmap 2024-03-28T18:55:21.137 INFO:tasks.ceph.osd.5.smithi155.stderr:2024-03-28T18:55:21.136+0000 7f88c6a00640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:55:21.201 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-03-28T18:55:21.200+0000 7fed2c400640 -1 osd.6 0 waiting for initial osdmap 2024-03-28T18:55:21.208 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-03-28T18:55:21.208+0000 7fed22400640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:55:21.224 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-03-28T18:55:21.224+0000 7f302a000640 -1 osd.7 0 waiting for initial osdmap 2024-03-28T18:55:21.236 INFO:tasks.ceph.osd.7.smithi155.stderr:2024-03-28T18:55:21.236+0000 7f301f600640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:55:21.244 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-03-28T18:55:21.245+0000 7f7dc2800640 -1 osd.3 0 waiting for initial osdmap 2024-03-28T18:55:21.245 INFO:tasks.ceph.osd.4.smithi155.stderr:2024-03-28T18:55:21.245+0000 7f46a5e00640 -1 osd.4 0 waiting for initial osdmap 2024-03-28T18:55:21.246 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-03-28T18:55:21.247+0000 7f8282400640 -1 osd.2 0 waiting for initial osdmap 2024-03-28T18:55:21.252 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-03-28T18:55:21.253+0000 7f7db7e00640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:55:21.253 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-03-28T18:55:21.254+0000 7f94f4800640 -1 osd.1 0 waiting for initial osdmap 2024-03-28T18:55:21.254 INFO:tasks.ceph.osd.4.smithi155.stderr:2024-03-28T18:55:21.254+0000 7f469b400640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:55:21.256 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-03-28T18:55:21.257+0000 7f8278400640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:55:21.268 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-03-28T18:55:21.269+0000 7f94e9e00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:55:21.323 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-03-28T18:55:21.324+0000 7fee0f600640 -1 osd.0 0 waiting for initial osdmap 2024-03-28T18:55:21.331 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-03-28T18:55:21.332+0000 7fee05600640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:55:21.333 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:21.488 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.488+0000 7f3d8a000640 1 Processor -- start 2024-03-28T18:55:21.488 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.489+0000 7f3d8a000640 1 -- start start 2024-03-28T18:55:21.490 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.490+0000 7f3d8a000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3d8406dd10 0x7f3d84150690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:21.490 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.490+0000 7f3d8a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d840a9440 0x7f3d84150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:21.490 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.490+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 0x7f3d841537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:21.490 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.490+0000 7f3d8a000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8406e5d0 con 0x7f3d840a9440 2024-03-28T18:55:21.490 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.490+0000 7f3d8a000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d840abcc0 con 0x7f3d8406dd10 2024-03-28T18:55:21.490 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.490+0000 7f3d8a000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d84157400 con 0x7f3d840c3060 2024-03-28T18:55:21.491 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.491+0000 7f3d83e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d840a9440 0x7f3d84150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:21.491 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.491+0000 7f3d89600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 0x7f3d841537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:21.491 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.491+0000 7f3d83e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d840a9440 0x7f3d84150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33770/0 (socket says 172.21.15.27:33770) 2024-03-28T18:55:21.492 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.491+0000 7f3d83e00640 1 -- 172.21.15.27:0/1545207040 learned_addr learned my addr 172.21.15.27:0/1545207040 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:21.492 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.491+0000 7f3d88c00640 1 --2- 172.21.15.27:0/1545207040 >> v2:172.21.15.155:3300/0 conn(0x7f3d8406dd10 0x7f3d84150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:21.492 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.491+0000 7f3d83e00640 1 -- 172.21.15.27:0/1545207040 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 msgr2=0x7f3d841537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:21.492 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.492+0000 7f3d83e00640 1 --2- 172.21.15.27:0/1545207040 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 0x7f3d841537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.492 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.492+0000 7f3d83e00640 1 -- 172.21.15.27:0/1545207040 >> v2:172.21.15.155:3300/0 conn(0x7f3d8406dd10 msgr2=0x7f3d84150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:21.492 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.492+0000 7f3d83e00640 1 --2- 172.21.15.27:0/1545207040 >> v2:172.21.15.155:3300/0 conn(0x7f3d8406dd10 0x7f3d84150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.492 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.492+0000 7f3d83e00640 1 -- 172.21.15.27:0/1545207040 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d841576e0 con 0x7f3d840a9440 2024-03-28T18:55:21.493 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.493+0000 7f3d83e00640 1 --2- 172.21.15.27:0/1545207040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d840a9440 0x7f3d84150bd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d8406bfd0 tx=0x7f3d7002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8f6dccbca8c3e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:21.493 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.494+0000 7f3d83400640 1 -- 172.21.15.27:0/1545207040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70035020 con 0x7f3d840a9440 2024-03-28T18:55:21.493 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.494+0000 7f3d83400640 1 -- 172.21.15.27:0/1545207040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d70031870 con 0x7f3d840a9440 2024-03-28T18:55:21.493 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.494+0000 7f3d83400640 1 -- 172.21.15.27:0/1545207040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70031c40 con 0x7f3d840a9440 2024-03-28T18:55:21.494 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.494+0000 7f3d8a000640 1 -- 172.21.15.27:0/1545207040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d840a9440 msgr2=0x7f3d84150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:21.494 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.494+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1545207040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d840a9440 0x7f3d84150bd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d8406bfd0 tx=0x7f3d7002b910 comp rx=0 tx=0).stop 2024-03-28T18:55:21.494 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.495+0000 7f3d8a000640 1 -- 172.21.15.27:0/1545207040 shutdown_connections 2024-03-28T18:55:21.495 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.495+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1545207040 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 0x7f3d841537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.495 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.495+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1545207040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d840a9440 0x7f3d84150bd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.495 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.495+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1545207040 >> v2:172.21.15.155:3300/0 conn(0x7f3d8406dd10 0x7f3d84150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.495 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.495+0000 7f3d8a000640 1 -- 172.21.15.27:0/1545207040 >> 172.21.15.27:0/1545207040 conn(0x7f3d8406c930 msgr2=0x7f3d8406cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:21.495 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.495+0000 7f3d8a000640 1 -- 172.21.15.27:0/1545207040 shutdown_connections 2024-03-28T18:55:21.495 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.495+0000 7f3d8a000640 1 -- 172.21.15.27:0/1545207040 wait complete. 2024-03-28T18:55:21.496 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.497+0000 7f3d8a000640 1 Processor -- start 2024-03-28T18:55:21.496 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.497+0000 7f3d8a000640 1 -- start start 2024-03-28T18:55:21.499 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8406dd10 0x7f3d841e7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:21.499 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d840a9440 0x7f3d840c0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:21.499 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 0x7f3d841eaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:21.499 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d8a000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8406e0f0 con 0x7f3d8406dd10 2024-03-28T18:55:21.499 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d8a000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8406ef10 con 0x7f3d840a9440 2024-03-28T18:55:21.499 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d8a000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8406e270 con 0x7f3d840c3060 2024-03-28T18:55:21.499 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8406dd10 0x7f3d841e7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8406dd10 0x7f3d841e7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33776/0 (socket says 172.21.15.27:33776) 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.498+0000 7f3d88c00640 1 -- 172.21.15.27:0/1294852949 learned_addr learned my addr 172.21.15.27:0/1294852949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d83e00640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d840a9440 0x7f3d840c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d88c00640 1 -- 172.21.15.27:0/1294852949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 msgr2=0x7f3d841eaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d88c00640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 0x7f3d841eaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d88c00640 1 -- 172.21.15.27:0/1294852949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d840a9440 msgr2=0x7f3d840c0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d88c00640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d840a9440 0x7f3d840c0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d88c00640 1 -- 172.21.15.27:0/1294852949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d84154c80 con 0x7f3d8406dd10 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d88c00640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8406dd10 0x7f3d841e7f90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800e890 tx=0x7f3d7800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d80c00640 1 -- 172.21.15.27:0/1294852949 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78017070 con 0x7f3d8406dd10 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d80c00640 1 -- 172.21.15.27:0/1294852949 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7800c8c0 con 0x7f3d8406dd10 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d80c00640 1 -- 172.21.15.27:0/1294852949 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800cc90 con 0x7f3d8406dd10 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d840728b0 con 0x7f3d8406dd10 2024-03-28T18:55:21.500 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.499+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d841ea9b0 con 0x7f3d8406dd10 2024-03-28T18:55:21.502 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.500+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d84075660 con 0x7f3d8406dd10 2024-03-28T18:55:21.502 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.501+0000 7f3d80c00640 1 -- 172.21.15.27:0/1294852949 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7f3d7801a070 con 0x7f3d8406dd10 2024-03-28T18:55:21.502 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.502+0000 7f3d80c00640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6406bb40 0x7f3d6406e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:21.502 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.502+0000 7f3d80c00640 1 -- 172.21.15.27:0/1294852949 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f3d78089a90 con 0x7f3d8406dd10 2024-03-28T18:55:21.502 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.502+0000 7f3d83e00640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6406bb40 0x7f3d6406e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:21.502 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.503+0000 7f3d83e00640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6406bb40 0x7f3d6406e000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3d70036000 tx=0x7f3d70008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:21.507 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.508+0000 7f3d80c00640 1 -- 172.21.15.27:0/1294852949 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d78059480 con 0x7f3d8406dd10 2024-03-28T18:55:21.705 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.705+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d84154750 con 0x7f3d8406dd10 2024-03-28T18:55:21.705 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.706+0000 7f3d80c00640 1 -- 172.21.15.27:0/1294852949 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f3d78058d80 con 0x7f3d8406dd10 2024-03-28T18:55:21.706 INFO:teuthology.misc.health.smithi027.stdout: 2024-03-28T18:55:21.706 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":12,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:18.910161+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:21.708 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6406bb40 msgr2=0x7f3d6406e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:21.708 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6406bb40 0x7f3d6406e000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3d70036000 tx=0x7f3d70008030 comp rx=0 tx=0).stop 2024-03-28T18:55:21.708 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8406dd10 msgr2=0x7f3d841e7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:21.708 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8406dd10 0x7f3d841e7f90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800e890 tx=0x7f3d7800ed50 comp rx=0 tx=0).stop 2024-03-28T18:55:21.709 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 shutdown_connections 2024-03-28T18:55:21.709 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6406bb40 0x7f3d6406e000 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.709 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d840c3060 0x7f3d841eaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.709 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d840a9440 0x7f3d840c0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.709 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.708+0000 7f3d8a000640 1 --2- 172.21.15.27:0/1294852949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8406dd10 0x7f3d841e7f90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:21.709 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.709+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 >> 172.21.15.27:0/1294852949 conn(0x7f3d8406c930 msgr2=0x7f3d840a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:21.709 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.709+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 shutdown_connections 2024-03-28T18:55:21.709 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:21.709+0000 7f3d8a000640 1 -- 172.21.15.27:0/1294852949 wait complete. 2024-03-28T18:55:21.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T18:55:24.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:55:24.327 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:55:28.529 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:28.725 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.724+0000 7f9a39800640 1 Processor -- start 2024-03-28T18:55:28.725 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.725+0000 7f9a39800640 1 -- start start 2024-03-28T18:55:28.728 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.726+0000 7f9a39800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9a340acf10 0x7f9a340ab500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:28.728 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.726+0000 7f9a39800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a340c7040 0x7f9a340aba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:28.728 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.726+0000 7f9a39800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a340c7510 con 0x7f9a340c6c60 2024-03-28T18:55:28.728 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.726+0000 7f9a39800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a340c34d0 con 0x7f9a340c7040 2024-03-28T18:55:28.729 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.726+0000 7f9a39800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a340c77f0 con 0x7f9a340acf10 2024-03-28T18:55:28.729 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.727+0000 7f9a38e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a340c7040 0x7f9a340aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:28.729 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.727+0000 7f9a38e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a340c7040 0x7f9a340aba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53162/0 (socket says 172.21.15.27:53162) 2024-03-28T18:55:28.729 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.727+0000 7f9a38e00640 1 -- 172.21.15.27:0/2605034320 learned_addr learned my addr 172.21.15.27:0/2605034320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:28.729 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.727+0000 7f9a2be00640 1 --2- 172.21.15.27:0/2605034320 >> v2:172.21.15.155:3300/0 conn(0x7f9a340acf10 0x7f9a340ab500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:28.729 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.727+0000 7f9a38e00640 1 -- 172.21.15.27:0/2605034320 >> v2:172.21.15.155:3300/0 conn(0x7f9a340acf10 msgr2=0x7f9a340ab500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:28.729 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.727+0000 7f9a38e00640 1 --2- 172.21.15.27:0/2605034320 >> v2:172.21.15.155:3300/0 conn(0x7f9a340acf10 0x7f9a340ab500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.729 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.727+0000 7f9a38e00640 1 -- 172.21.15.27:0/2605034320 >> v1:172.21.15.155:6789/0 conn(0x7f9a340c6c60 legacy=0x7f9a340ac800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:28.730 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.727+0000 7f9a38e00640 1 -- 172.21.15.27:0/2605034320 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a340c3160 con 0x7f9a340c7040 2024-03-28T18:55:28.731 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.728+0000 7f9a38e00640 1 --2- 172.21.15.27:0/2605034320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c7040 0x7f9a340aba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009e50 tx=0x7f9a2c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8fc00d2ed6db9e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:28.731 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a33400640 1 -- 172.21.15.27:0/2605034320 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c02c9f0 con 0x7f9a340c7040 2024-03-28T18:55:28.731 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a33400640 1 -- 172.21.15.27:0/2605034320 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2c02cb90 con 0x7f9a340c7040 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a33400640 1 -- 172.21.15.27:0/2605034320 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c0343f0 con 0x7f9a340c7040 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a39800640 1 -- 172.21.15.27:0/2605034320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c7040 msgr2=0x7f9a340aba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a39800640 1 --2- 172.21.15.27:0/2605034320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c7040 0x7f9a340aba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009e50 tx=0x7f9a2c02be50 comp rx=0 tx=0).stop 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a39800640 1 -- 172.21.15.27:0/2605034320 shutdown_connections 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a39800640 1 --2- 172.21.15.27:0/2605034320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c7040 0x7f9a340aba40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a39800640 1 --2- 172.21.15.27:0/2605034320 >> v2:172.21.15.155:3300/0 conn(0x7f9a340acf10 0x7f9a340ab500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a39800640 1 -- 172.21.15.27:0/2605034320 >> 172.21.15.27:0/2605034320 conn(0x7f9a340a9990 msgr2=0x7f9a340a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a39800640 1 -- 172.21.15.27:0/2605034320 shutdown_connections 2024-03-28T18:55:28.732 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.729+0000 7f9a39800640 1 -- 172.21.15.27:0/2605034320 wait complete. 2024-03-28T18:55:28.736 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.731+0000 7f9a39800640 1 Processor -- start 2024-03-28T18:55:28.736 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.731+0000 7f9a39800640 1 -- start start 2024-03-28T18:55:28.736 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.732+0000 7f9a39800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a340acf10 0x7f9a34154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:28.736 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.732+0000 7f9a39800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c6c60 0x7f9a341ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:28.736 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.732+0000 7f9a39800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a340c7040 0x7f9a341ec380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:28.736 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.732+0000 7f9a39800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a340ac100 con 0x7f9a340c6c60 2024-03-28T18:55:28.736 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.732+0000 7f9a39800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a340abf80 con 0x7f9a340c7040 2024-03-28T18:55:28.736 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.732+0000 7f9a39800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a340c5eb0 con 0x7f9a340acf10 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c6c60 0x7f9a341ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c6c60 0x7f9a341ee410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53168/0 (socket says 172.21.15.27:53168) 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 -- 172.21.15.27:0/1439453999 learned_addr learned my addr 172.21.15.27:0/1439453999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a33e00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a340acf10 0x7f9a34154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a38e00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a340c7040 0x7f9a341ec380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 -- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a340acf10 msgr2=0x7f9a34154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a340acf10 0x7f9a34154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 -- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a340c7040 msgr2=0x7f9a341ec380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:28.737 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a340c7040 0x7f9a341ec380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 -- 172.21.15.27:0/1439453999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a341eee90 con 0x7f9a340c6c60 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a38e00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a340c7040 0x7f9a341ec380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.733+0000 7f9a2be00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c6c60 0x7f9a341ee410 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00d800 tx=0x7f9a1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.734+0000 7f9a30c00640 1 -- 172.21.15.27:0/1439453999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c00b6c0 con 0x7f9a340c6c60 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.734+0000 7f9a30c00640 1 -- 172.21.15.27:0/1439453999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1c019070 con 0x7f9a340c6c60 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.734+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a341de230 con 0x7f9a340c6c60 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.734+0000 7f9a30c00640 1 -- 172.21.15.27:0/1439453999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c00ba80 con 0x7f9a340c6c60 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.734+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a341dd4b0 con 0x7f9a340c6c60 2024-03-28T18:55:28.738 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.735+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a34075270 con 0x7f9a340c6c60 2024-03-28T18:55:28.739 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.736+0000 7f9a30c00640 1 -- 172.21.15.27:0/1439453999 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a1c00bc20 con 0x7f9a340c6c60 2024-03-28T18:55:28.739 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.738+0000 7f9a30c00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a1006bbd0 0x7f9a1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:28.740 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.738+0000 7f9a30c00640 1 -- 172.21.15.27:0/1439453999 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9a1c08abb0 con 0x7f9a340c6c60 2024-03-28T18:55:28.740 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.738+0000 7f9a33e00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a1006bbd0 0x7f9a1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:28.740 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.739+0000 7f9a33e00640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a1006bbd0 0x7f9a1006e090 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9a340c2e10 tx=0x7f9a24009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:28.744 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.743+0000 7f9a30c00640 1 -- 172.21.15.27:0/1439453999 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a1c01c360 con 0x7f9a340c6c60 2024-03-28T18:55:28.950 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.949+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a34075660 con 0x7f9a340c6c60 2024-03-28T18:55:28.952 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.951+0000 7f9a30c00640 1 -- 172.21.15.27:0/1439453999 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12848 (secure 0 0 0) 0x7f9a1c0599e0 con 0x7f9a340c6c60 2024-03-28T18:55:28.953 INFO:teuthology.misc.health.smithi027.stdout: 2024-03-28T18:55:28.953 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":16,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:25.182434+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:28.955 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a1006bbd0 msgr2=0x7f9a1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a1006bbd0 0x7f9a1006e090 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9a340c2e10 tx=0x7f9a24009040 comp rx=0 tx=0).stop 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c6c60 msgr2=0x7f9a341ee410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c6c60 0x7f9a341ee410 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00d800 tx=0x7f9a1c00dcc0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 shutdown_connections 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a1006bbd0 0x7f9a1006e090 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a340c7040 0x7f9a341ec380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a340c6c60 0x7f9a341ee410 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 --2- 172.21.15.27:0/1439453999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a340acf10 0x7f9a34154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 >> 172.21.15.27:0/1439453999 conn(0x7f9a340a9990 msgr2=0x7f9a340a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:28.956 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 shutdown_connections 2024-03-28T18:55:28.957 INFO:teuthology.misc.health.smithi027.stderr:2024-03-28T18:55:28.954+0000 7f9a39800640 1 -- 172.21.15.27:0/1439453999 wait complete. 2024-03-28T18:55:28.970 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-28T18:55:28.970 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-28T18:55:28.970 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T18:55:28.971 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T18:55:28.978 INFO:tasks.ceph.mds.a:Started 2024-03-28T18:55:28.978 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T18:55:28.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T18:55:28.986 INFO:tasks.ceph.mds.c:Started 2024-03-28T18:55:28.986 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T18:55:28.986 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T18:55:28.994 INFO:tasks.ceph.mds.b:Started 2024-03-28T18:55:28.994 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T18:55:28.994 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T18:55:29.002 INFO:tasks.ceph.mds.d:Started 2024-03-28T18:55:29.002 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-28T18:55:29.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:29.148 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T18:55:29.148 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T18:55:29.162 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T18:55:29.162 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T18:55:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.206+0000 7f0193600640 1 Processor -- start 2024-03-28T18:55:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.206+0000 7f0193600640 1 -- start start 2024-03-28T18:55:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.207+0000 7f0193600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c0c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.207+0000 7f0193600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018c1530f0 con 0x7f018c0c8e10 2024-03-28T18:55:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.207+0000 7f0193600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018c06d3d0 con 0x7f018c1562f0 2024-03-28T18:55:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.207+0000 7f0193600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0c92f0 con 0x7f018c152d10 2024-03-28T18:55:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.207+0000 7f018be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f018c0c8e10 0x7f018c158be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35582/0 (socket says 172.21.15.27:35582) 2024-03-28T18:55:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.207+0000 7f018be00640 1 -- 172.21.15.27:0/443255269 learned_addr learned my addr 172.21.15.27:0/443255269 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.207+0000 7f0190c00640 1 --2- 172.21.15.27:0/443255269 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c0c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.208+0000 7f018b400640 1 -- 172.21.15.27:0/443255269 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863365480 0 0) 0x7f017c001110 con 0x7f018c0c8e10 2024-03-28T18:55:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.208+0000 7f018b400640 1 -- 172.21.15.27:0/443255269 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0174003520 con 0x7f018c0c8e10 2024-03-28T18:55:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.208+0000 7f018b400640 1 -- 172.21.15.27:0/443255269 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778613124 0 0) 0x7f018c06d3d0 con 0x7f018c1562f0 2024-03-28T18:55:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.208+0000 7f018b400640 1 -- 172.21.15.27:0/443255269 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f017c001110 con 0x7f018c1562f0 2024-03-28T18:55:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.208+0000 7f0190c00640 1 -- 172.21.15.27:0/443255269 >> v1:172.21.15.155:6789/0 conn(0x7f018c1562f0 legacy=0x7f018c15ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.208+0000 7f0190c00640 1 -- 172.21.15.27:0/443255269 >> v1:172.21.15.27:6789/0 conn(0x7f018c0c8e10 legacy=0x7f018c158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.208+0000 7f018b400640 1 -- 172.21.15.27:0/443255269 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1061677230 0 0) 0x7f0174003520 con 0x7f018c0c8e10 2024-03-28T18:55:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.208+0000 7f0190c00640 1 -- 172.21.15.27:0/443255269 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c155810 con 0x7f018c152d10 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.209+0000 7f0190c00640 1 --2- 172.21.15.27:0/443255269 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c0c5890 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f018c06d5a0 tx=0x7f018002b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc97c9ea303c11e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.209+0000 7f018b400640 1 -- 172.21.15.27:0/443255269 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0180037070 con 0x7f018c152d10 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f018b400640 1 -- 172.21.15.27:0/443255269 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0180030510 con 0x7f018c152d10 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f018b400640 1 -- 172.21.15.27:0/443255269 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c001840 con 0x7f018c152d10 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f0193600640 1 -- 172.21.15.27:0/443255269 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 msgr2=0x7f018c0c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f0193600640 1 --2- 172.21.15.27:0/443255269 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c0c5890 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f018c06d5a0 tx=0x7f018002b6d0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f0193600640 1 -- 172.21.15.27:0/443255269 shutdown_connections 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f0193600640 1 --2- 172.21.15.27:0/443255269 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c0c5890 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f0193600640 1 -- 172.21.15.27:0/443255269 >> 172.21.15.27:0/443255269 conn(0x7f018c066fb0 msgr2=0x7f018c0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f0193600640 1 -- 172.21.15.27:0/443255269 shutdown_connections 2024-03-28T18:55:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.210+0000 7f0193600640 1 -- 172.21.15.27:0/443255269 wait complete. 2024-03-28T18:55:29.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.212+0000 7f0193600640 1 Processor -- start 2024-03-28T18:55:29.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.212+0000 7f0193600640 1 -- start start 2024-03-28T18:55:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0193600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f018c0c8e10 0x7f018c0c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0193600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c14da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0193600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c1562f0 0x7f018c14df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0193600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c1592f0 con 0x7f018c1562f0 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0193600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c06d850 con 0x7f018c0c8e10 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0193600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c0c3f20 con 0x7f018c152d10 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0191600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c1562f0 0x7f018c14df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0191600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c1562f0 0x7f018c14df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53212/0 (socket says 172.21.15.27:53212) 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.213+0000 7f0191600640 1 -- 172.21.15.27:0/1222700981 learned_addr learned my addr 172.21.15.27:0/1222700981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0190c00640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f018c0c8e10 0x7f018c0c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f018be00640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c14da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0191600640 1 -- 172.21.15.27:0/1222700981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 msgr2=0x7f018c14da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0191600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c14da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0191600640 1 -- 172.21.15.27:0/1222700981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f018c0c8e10 msgr2=0x7f018c0c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0191600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f018c0c8e10 0x7f018c0c4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0191600640 1 -- 172.21.15.27:0/1222700981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c0c2b20 con 0x7f018c1562f0 2024-03-28T18:55:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0191600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c1562f0 0x7f018c14df80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f018400c6e0 tx=0x7f018400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0188c00640 1 -- 172.21.15.27:0/1222700981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184003b70 con 0x7f018c1562f0 2024-03-28T18:55:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0188c00640 1 -- 172.21.15.27:0/1222700981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0184003d10 con 0x7f018c1562f0 2024-03-28T18:55:29.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.214+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018c0c2e00 con 0x7f018c1562f0 2024-03-28T18:55:29.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.215+0000 7f0188c00640 1 -- 172.21.15.27:0/1222700981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01840052d0 con 0x7f018c1562f0 2024-03-28T18:55:29.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.215+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c0c33c0 con 0x7f018c1562f0 2024-03-28T18:55:29.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.216+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c0c35c0 con 0x7f018c1562f0 2024-03-28T18:55:29.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.217+0000 7f0188c00640 1 -- 172.21.15.27:0/1222700981 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0184005470 con 0x7f018c1562f0 2024-03-28T18:55:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.218+0000 7f0188c00640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f017006bbd0 0x7f017006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.218+0000 7f0188c00640 1 -- 172.21.15.27:0/1222700981 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f018408a4d0 con 0x7f018c1562f0 2024-03-28T18:55:29.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.218+0000 7f0190c00640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f017006bbd0 0x7f017006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.219+0000 7f0190c00640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f017006bbd0 0x7f017006e090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0180034000 tx=0x7f0180008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:29.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.222+0000 7f0188c00640 1 -- 172.21.15.27:0/1222700981 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f018c0c35c0 con 0x7f018c1562f0 2024-03-28T18:55:29.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.426+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f018c04f400 con 0x7f018c1562f0 2024-03-28T18:55:29.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.427+0000 7f0188c00640 1 -- 172.21.15.27:0/1222700981 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12848 (secure 0 0 0) 0x7f0184058cb0 con 0x7f018c1562f0 2024-03-28T18:55:29.428 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:29.428 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:25.182434+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f017006bbd0 msgr2=0x7f017006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f017006bbd0 0x7f017006e090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0180034000 tx=0x7f0180008030 comp rx=0 tx=0).stop 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c1562f0 msgr2=0x7f018c14df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c1562f0 0x7f018c14df80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f018400c6e0 tx=0x7f018400cba0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 shutdown_connections 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f017006bbd0 0x7f017006e090 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c1562f0 0x7f018c14df80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f018c152d10 0x7f018c14da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 --2- 172.21.15.27:0/1222700981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f018c0c8e10 0x7f018c0c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.430+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 >> 172.21.15.27:0/1222700981 conn(0x7f018c066fb0 msgr2=0x7f018c159600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.431+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 shutdown_connections 2024-03-28T18:55:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.431+0000 7f0193600640 1 -- 172.21.15.27:0/1222700981 wait complete. 2024-03-28T18:55:29.451 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:55:29.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:55:29.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.676+0000 7f4209200640 1 Processor -- start 2024-03-28T18:55:29.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.676+0000 7f4209200640 1 -- start start 2024-03-28T18:55:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.677+0000 7f4209200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42040c5aa0 0x7f4204156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.677+0000 7f4209200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42040735f0 con 0x7f4204152840 2024-03-28T18:55:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.677+0000 7f4209200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42040c7d10 con 0x7f42040c8db0 2024-03-28T18:55:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.677+0000 7f4209200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42041565d0 con 0x7f42040c5aa0 2024-03-28T18:55:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.678+0000 7f4203400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42040c5aa0 0x7f4204156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.678+0000 7f4203e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4204152840 0x7f420415ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35594/0 (socket says 172.21.15.27:35594) 2024-03-28T18:55:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.678+0000 7f4203e00640 1 -- 172.21.15.27:0/324628515 learned_addr learned my addr 172.21.15.27:0/324628515 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.678+0000 7f4203400640 1 -- 172.21.15.27:0/324628515 >> v1:172.21.15.155:6789/0 conn(0x7f42040c8db0 legacy=0x7f4204072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.678+0000 7f4203400640 1 -- 172.21.15.27:0/324628515 >> v1:172.21.15.27:6789/0 conn(0x7f4204152840 legacy=0x7f420415ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.678+0000 7f4203400640 1 -- 172.21.15.27:0/324628515 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42040c7ff0 con 0x7f42040c5aa0 2024-03-28T18:55:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.678+0000 7f4202000640 1 -- 172.21.15.27:0/324628515 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569089629 0 0) 0x7f42040735f0 con 0x7f4204152840 2024-03-28T18:55:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.679+0000 7f4203400640 1 --2- 172.21.15.27:0/324628515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c5aa0 0x7f4204156090 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f41f40097a0 tx=0x7f41f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b3e7363f542fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.680+0000 7f4202000640 1 -- 172.21.15.27:0/324628515 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4035020 con 0x7f42040c5aa0 2024-03-28T18:55:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.680+0000 7f4202000640 1 -- 172.21.15.27:0/324628515 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f4031830 con 0x7f42040c5aa0 2024-03-28T18:55:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.680+0000 7f4202000640 1 -- 172.21.15.27:0/324628515 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4031c00 con 0x7f42040c5aa0 2024-03-28T18:55:29.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.680+0000 7f4209200640 1 -- 172.21.15.27:0/324628515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c5aa0 msgr2=0x7f4204156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.680+0000 7f4209200640 1 --2- 172.21.15.27:0/324628515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c5aa0 0x7f4204156090 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f41f40097a0 tx=0x7f41f402b8d0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.680+0000 7f4209200640 1 -- 172.21.15.27:0/324628515 shutdown_connections 2024-03-28T18:55:29.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.680+0000 7f4209200640 1 --2- 172.21.15.27:0/324628515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c5aa0 0x7f4204156090 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.680+0000 7f4209200640 1 -- 172.21.15.27:0/324628515 >> 172.21.15.27:0/324628515 conn(0x7f420406b0c0 msgr2=0x7f420406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.681+0000 7f4209200640 1 -- 172.21.15.27:0/324628515 shutdown_connections 2024-03-28T18:55:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.681+0000 7f4209200640 1 -- 172.21.15.27:0/324628515 wait complete. 2024-03-28T18:55:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.682+0000 7f4209200640 1 Processor -- start 2024-03-28T18:55:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.682+0000 7f4209200640 1 -- start start 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.683+0000 7f4209200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42040c5aa0 0x7f42040c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.683+0000 7f4209200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c8db0 0x7f4204155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.683+0000 7f4209200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4204152840 0x7f420414bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.683+0000 7f4209200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4204072100 con 0x7f42040c8db0 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.683+0000 7f4209200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42040c82d0 con 0x7f4204152840 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.683+0000 7f4209200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4204072d30 con 0x7f42040c5aa0 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.683+0000 7f4202a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c8db0 0x7f4204155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4202a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c8db0 0x7f4204155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53230/0 (socket says 172.21.15.27:53230) 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4202a00640 1 -- 172.21.15.27:0/4063562122 learned_addr learned my addr 172.21.15.27:0/4063562122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:29.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4203400640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42040c5aa0 0x7f42040c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4203e00640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4204152840 0x7f420414bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4202a00640 1 -- 172.21.15.27:0/4063562122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42040c5aa0 msgr2=0x7f42040c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4202a00640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42040c5aa0 0x7f42040c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4202a00640 1 -- 172.21.15.27:0/4063562122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4204152840 msgr2=0x7f420414bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4202a00640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4204152840 0x7f420414bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4202a00640 1 -- 172.21.15.27:0/4063562122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42041eabc0 con 0x7f42040c8db0 2024-03-28T18:55:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.684+0000 7f4202a00640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c8db0 0x7f4204155a80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00c450 tx=0x7f41ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.685+0000 7f41fb400640 1 -- 172.21.15.27:0/4063562122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec0052d0 con 0x7f42040c8db0 2024-03-28T18:55:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.685+0000 7f41fb400640 1 -- 172.21.15.27:0/4063562122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ec005470 con 0x7f42040c8db0 2024-03-28T18:55:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.685+0000 7f41fb400640 1 -- 172.21.15.27:0/4063562122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec012870 con 0x7f42040c8db0 2024-03-28T18:55:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.685+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4204153520 con 0x7f42040c8db0 2024-03-28T18:55:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.685+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42040c1f80 con 0x7f42040c8db0 2024-03-28T18:55:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.686+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420404f400 con 0x7f42040c8db0 2024-03-28T18:55:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.687+0000 7f41fb400640 1 -- 172.21.15.27:0/4063562122 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f41ec012a10 con 0x7f42040c8db0 2024-03-28T18:55:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.688+0000 7f41fb400640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41e006bc00 0x7f41e006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.688+0000 7f41fb400640 1 -- 172.21.15.27:0/4063562122 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f41ec08a2f0 con 0x7f42040c8db0 2024-03-28T18:55:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.688+0000 7f4203400640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41e006bc00 0x7f41e006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.689+0000 7f4203400640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41e006bc00 0x7f41e006e0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41f4036000 tx=0x7f41f4008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.693+0000 7f41fb400640 1 -- 172.21.15.27:0/4063562122 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f41ec059160 con 0x7f42040c8db0 2024-03-28T18:55:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.897+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42040732f0 con 0x7f42040c8db0 2024-03-28T18:55:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.898+0000 7f41fb400640 1 -- 172.21.15.27:0/4063562122 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f41ec058a60 con 0x7f42040c8db0 2024-03-28T18:55:29.899 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:55:29.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.900+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41e006bc00 msgr2=0x7f41e006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.900+0000 7f4209200640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41e006bc00 0x7f41e006e0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41f4036000 tx=0x7f41f4008030 comp rx=0 tx=0).stop 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.900+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c8db0 msgr2=0x7f4204155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.900+0000 7f4209200640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c8db0 0x7f4204155a80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00c450 tx=0x7f41ec00c910 comp rx=0 tx=0).stop 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.901+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 shutdown_connections 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.901+0000 7f4209200640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41e006bc00 0x7f41e006e0c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.901+0000 7f4209200640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4204152840 0x7f420414bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.901+0000 7f4209200640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42040c8db0 0x7f4204155a80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.901+0000 7f4209200640 1 --2- 172.21.15.27:0/4063562122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42040c5aa0 0x7f42040c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.901+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 >> 172.21.15.27:0/4063562122 conn(0x7f420406b0c0 msgr2=0x7f420406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.901+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 shutdown_connections 2024-03-28T18:55:29.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:29.901+0000 7f4209200640 1 -- 172.21.15.27:0/4063562122 wait complete. 2024-03-28T18:55:29.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:55:30.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.148+0000 7f46a3200640 1 Processor -- start 2024-03-28T18:55:30.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.149+0000 7f46a3200640 1 -- start start 2024-03-28T18:55:30.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.150+0000 7f46a3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a40a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.150+0000 7f46a3200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f46a40a61a0 0x7f46a40c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.150+0000 7f46a3200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a4150ec0 con 0x7f46a40c6ee0 2024-03-28T18:55:30.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.150+0000 7f46a3200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a406dc70 con 0x7f46a40a61a0 2024-03-28T18:55:30.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.150+0000 7f46a3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a4153550 con 0x7f46a4150ae0 2024-03-28T18:55:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.151+0000 7f46a1e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a40a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.151+0000 7f46a2800640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f46a40c6ee0 0x7f46a4153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:36388/0 (socket says 172.21.15.27:36388) 2024-03-28T18:55:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.151+0000 7f46a2800640 1 -- 172.21.15.27:0/3087791947 learned_addr learned my addr 172.21.15.27:0/3087791947 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:30.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.151+0000 7f4699400640 1 --2- 172.21.15.27:0/3087791947 >> v2:172.21.15.155:3300/0 conn(0x7f46a40a61a0 0x7f46a40c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.152+0000 7f46a1e00640 1 -- 172.21.15.27:0/3087791947 >> v2:172.21.15.155:3300/0 conn(0x7f46a40a61a0 msgr2=0x7f46a40c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.152+0000 7f46a1e00640 1 --2- 172.21.15.27:0/3087791947 >> v2:172.21.15.155:3300/0 conn(0x7f46a40a61a0 0x7f46a40c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.152+0000 7f46a1e00640 1 -- 172.21.15.27:0/3087791947 >> v1:172.21.15.155:6789/0 conn(0x7f46a40c6ee0 legacy=0x7f46a4153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.152+0000 7f46a1400640 1 -- 172.21.15.27:0/3087791947 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957581795 0 0) 0x7f469c001490 con 0x7f46a40c6ee0 2024-03-28T18:55:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.152+0000 7f46a1e00640 1 -- 172.21.15.27:0/3087791947 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a406df10 con 0x7f46a4150ae0 2024-03-28T18:55:30.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.152+0000 7f4699400640 1 --2- 172.21.15.27:0/3087791947 >> v2:172.21.15.155:3300/0 conn(0x7f46a40a61a0 0x7f46a40c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:30.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.154+0000 7f46a1e00640 1 --2- 172.21.15.27:0/3087791947 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a40a5c60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f46940097a0 tx=0x7f469402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91f8501b42c79bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:30.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.154+0000 7f46a1400640 1 -- 172.21.15.27:0/3087791947 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694031780 con 0x7f46a4150ae0 2024-03-28T18:55:30.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.154+0000 7f46a1400640 1 -- 172.21.15.27:0/3087791947 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4694037070 con 0x7f46a4150ae0 2024-03-28T18:55:30.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.154+0000 7f46a1400640 1 -- 172.21.15.27:0/3087791947 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694031b40 con 0x7f46a4150ae0 2024-03-28T18:55:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.155+0000 7f46a3200640 1 -- 172.21.15.27:0/3087791947 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 msgr2=0x7f46a40a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.155+0000 7f46a3200640 1 --2- 172.21.15.27:0/3087791947 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a40a5c60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f46940097a0 tx=0x7f469402b820 comp rx=0 tx=0).stop 2024-03-28T18:55:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.155+0000 7f46a3200640 1 -- 172.21.15.27:0/3087791947 shutdown_connections 2024-03-28T18:55:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.155+0000 7f46a3200640 1 --2- 172.21.15.27:0/3087791947 >> v2:172.21.15.155:3300/0 conn(0x7f46a40a61a0 0x7f46a40c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.155+0000 7f46a3200640 1 --2- 172.21.15.27:0/3087791947 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a40a5c60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.155+0000 7f46a3200640 1 -- 172.21.15.27:0/3087791947 >> 172.21.15.27:0/3087791947 conn(0x7f46a406b0c0 msgr2=0x7f46a406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.155+0000 7f46a3200640 1 -- 172.21.15.27:0/3087791947 shutdown_connections 2024-03-28T18:55:30.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.155+0000 7f46a3200640 1 -- 172.21.15.27:0/3087791947 wait complete. 2024-03-28T18:55:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.157+0000 7f46a3200640 1 Processor -- start 2024-03-28T18:55:30.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.158+0000 7f46a3200640 1 -- start start 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f46a3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a40a61a0 0x7f46a4152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f46a3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a40c6ee0 0x7f46a4070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f46a3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a4070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f46a3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a406e7b0 con 0x7f46a40c6ee0 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f46a3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a406e4d0 con 0x7f46a40a61a0 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f46a3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a41569d0 con 0x7f46a4150ae0 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f4699400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a40c6ee0 0x7f46a4070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f4699400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a40c6ee0 0x7f46a4070770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53232/0 (socket says 172.21.15.27:53232) 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.159+0000 7f4699400640 1 -- 172.21.15.27:0/3968532430 learned_addr learned my addr 172.21.15.27:0/3968532430 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f46a1e00640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a40a61a0 0x7f46a4152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f46a2800640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a4070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f4699400640 1 -- 172.21.15.27:0/3968532430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 msgr2=0x7f46a4070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f4699400640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a4070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f4699400640 1 -- 172.21.15.27:0/3968532430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a40a61a0 msgr2=0x7f46a4152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f4699400640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a40a61a0 0x7f46a4152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f4699400640 1 -- 172.21.15.27:0/3968532430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a41e05f0 con 0x7f46a40c6ee0 2024-03-28T18:55:30.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f4699400640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a40c6ee0 0x7f46a4070770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f468c00e900 tx=0x7f468c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:30.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.160+0000 7f469aa00640 1 -- 172.21.15.27:0/3968532430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c017070 con 0x7f46a40c6ee0 2024-03-28T18:55:30.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.161+0000 7f469aa00640 1 -- 172.21.15.27:0/3968532430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c00ca00 con 0x7f46a40c6ee0 2024-03-28T18:55:30.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.161+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a414db70 con 0x7f46a40c6ee0 2024-03-28T18:55:30.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.161+0000 7f469aa00640 1 -- 172.21.15.27:0/3968532430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c00cdf0 con 0x7f46a40c6ee0 2024-03-28T18:55:30.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.161+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a4152ab0 con 0x7f46a40c6ee0 2024-03-28T18:55:30.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.162+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a4067dd0 con 0x7f46a40c6ee0 2024-03-28T18:55:30.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.165+0000 7f469aa00640 1 -- 172.21.15.27:0/3968532430 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f468c01a070 con 0x7f46a40c6ee0 2024-03-28T18:55:30.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.166+0000 7f469aa00640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f468006bbd0 0x7f468006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.166+0000 7f469aa00640 1 -- 172.21.15.27:0/3968532430 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f468c08ad90 con 0x7f46a40c6ee0 2024-03-28T18:55:30.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.166+0000 7f46a1e00640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f468006bbd0 0x7f468006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.167+0000 7f46a1e00640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f468006bbd0 0x7f468006e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4694036000 tx=0x7f4694008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:30.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.170+0000 7f469aa00640 1 -- 172.21.15.27:0/3968532430 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f468c059d40 con 0x7f46a40c6ee0 2024-03-28T18:55:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.414+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46a406da50 con 0x7f46a40c6ee0 2024-03-28T18:55:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.415+0000 7f469aa00640 1 -- 172.21.15.27:0/3968532430 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7f468c059640 con 0x7f46a40c6ee0 2024-03-28T18:55:30.416 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:30.417 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-28T18:55:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-03-28T18:55:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f468006bbd0 msgr2=0x7f468006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f468006bbd0 0x7f468006e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4694036000 tx=0x7f4694008030 comp rx=0 tx=0).stop 2024-03-28T18:55:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a40c6ee0 msgr2=0x7f46a4070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a40c6ee0 0x7f46a4070770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f468c00e900 tx=0x7f468c00edc0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 shutdown_connections 2024-03-28T18:55:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f468006bbd0 0x7f468006e090 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46a4150ae0 0x7f46a4070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46a40c6ee0 0x7f46a4070770 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 --2- 172.21.15.27:0/3968532430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a40a61a0 0x7f46a4152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.419+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 >> 172.21.15.27:0/3968532430 conn(0x7f46a406b0c0 msgr2=0x7f46a40c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.420+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 shutdown_connections 2024-03-28T18:55:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.420+0000 7f46a3200640 1 -- 172.21.15.27:0/3968532430 wait complete. 2024-03-28T18:55:30.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:30.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.660+0000 7fe9ff200640 1 Processor -- start 2024-03-28T18:55:30.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.660+0000 7fe9ff200640 1 -- start start 2024-03-28T18:55:30.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9ff200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000c58c0 0x7fea000c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9ff200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea000c8ed0 0x7fea001555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9ff200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea000be5d0 0x7fea00155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9ff200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea00158bd0 con 0x7fea000c8ed0 2024-03-28T18:55:30.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9ff200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea00158eb0 con 0x7fea000be5d0 2024-03-28T18:55:30.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9ff200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea00159190 con 0x7fea000c58c0 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9fd400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea000c8ed0 0x7fea001555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9fd400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea000c8ed0 0x7fea001555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53234/0 (socket says 172.21.15.27:53234) 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9fd400640 1 -- 172.21.15.27:0/2525799089 learned_addr learned my addr 172.21.15.27:0/2525799089 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9fde00640 1 --2- 172.21.15.27:0/2525799089 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000c58c0 0x7fea000c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.662+0000 7fe9fe800640 1 --2- 172.21.15.27:0/2525799089 >> v2:172.21.15.155:3300/0 conn(0x7fea000be5d0 0x7fea00155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.663+0000 7fe9fd400640 1 -- 172.21.15.27:0/2525799089 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000c58c0 msgr2=0x7fea000c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.663+0000 7fe9fd400640 1 --2- 172.21.15.27:0/2525799089 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000c58c0 0x7fea000c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.663+0000 7fe9fd400640 1 -- 172.21.15.27:0/2525799089 >> v2:172.21.15.155:3300/0 conn(0x7fea000be5d0 msgr2=0x7fea00155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.663+0000 7fe9fd400640 1 --2- 172.21.15.27:0/2525799089 >> v2:172.21.15.155:3300/0 conn(0x7fea000be5d0 0x7fea00155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.663+0000 7fe9fd400640 1 -- 172.21.15.27:0/2525799089 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00159470 con 0x7fea000c8ed0 2024-03-28T18:55:30.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.664+0000 7fe9fd400640 1 --2- 172.21.15.27:0/2525799089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c8ed0 0x7fea001555e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fea000c7400 tx=0x7fe9ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9028715c5db579a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:30.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9fca00640 1 -- 172.21.15.27:0/2525799089 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec02e070 con 0x7fea000c8ed0 2024-03-28T18:55:30.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9fca00640 1 -- 172.21.15.27:0/2525799089 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ec037070 con 0x7fea000c8ed0 2024-03-28T18:55:30.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9fca00640 1 -- 172.21.15.27:0/2525799089 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec031a00 con 0x7fea000c8ed0 2024-03-28T18:55:30.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 -- 172.21.15.27:0/2525799089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c8ed0 msgr2=0x7fea001555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 --2- 172.21.15.27:0/2525799089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c8ed0 0x7fea001555e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fea000c7400 tx=0x7fe9ec02b880 comp rx=0 tx=0).stop 2024-03-28T18:55:30.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 -- 172.21.15.27:0/2525799089 shutdown_connections 2024-03-28T18:55:30.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 --2- 172.21.15.27:0/2525799089 >> v2:172.21.15.155:3300/0 conn(0x7fea000be5d0 0x7fea00155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 --2- 172.21.15.27:0/2525799089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c8ed0 0x7fea001555e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 --2- 172.21.15.27:0/2525799089 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000c58c0 0x7fea000c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 -- 172.21.15.27:0/2525799089 >> 172.21.15.27:0/2525799089 conn(0x7fea0006b6f0 msgr2=0x7fea000acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:30.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 -- 172.21.15.27:0/2525799089 shutdown_connections 2024-03-28T18:55:30.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.665+0000 7fe9ff200640 1 -- 172.21.15.27:0/2525799089 wait complete. 2024-03-28T18:55:30.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.667+0000 7fe9ff200640 1 Processor -- start 2024-03-28T18:55:30.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.667+0000 7fe9ff200640 1 -- start start 2024-03-28T18:55:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9ff200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000be5d0 0x7fea0014ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9ff200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c58c0 0x7fea000c0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9ff200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea000c8ed0 0x7fea001eb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9ff200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea000b09a0 con 0x7fea000c58c0 2024-03-28T18:55:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9ff200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea000b0820 con 0x7fea000c8ed0 2024-03-28T18:55:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9ff200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea000ad7d0 con 0x7fea000be5d0 2024-03-28T18:55:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9fd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c58c0 0x7fea000c0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9fd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c58c0 0x7fea000c0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53238/0 (socket says 172.21.15.27:53238) 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.668+0000 7fe9fd400640 1 -- 172.21.15.27:0/379998175 learned_addr learned my addr 172.21.15.27:0/379998175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fde00640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000be5d0 0x7fea0014ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fe800640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea000c8ed0 0x7fea001eb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fd400640 1 -- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000be5d0 msgr2=0x7fea0014ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fd400640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000be5d0 0x7fea0014ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fd400640 1 -- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea000c8ed0 msgr2=0x7fea001eb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fd400640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea000c8ed0 0x7fea001eb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fd400640 1 -- 172.21.15.27:0/379998175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00155aa0 con 0x7fea000c58c0 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fe800640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea000c8ed0 0x7fea001eb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fde00640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000be5d0 0x7fea0014ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9fd400640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c58c0 0x7fea000c0680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec02b8c0 tx=0x7fe9ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9f6000640 1 -- 172.21.15.27:0/379998175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec032880 con 0x7fea000c58c0 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9f6000640 1 -- 172.21.15.27:0/379998175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ec032a60 con 0x7fea000c58c0 2024-03-28T18:55:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9f6000640 1 -- 172.21.15.27:0/379998175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec032e50 con 0x7fea000c58c0 2024-03-28T18:55:30.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.669+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea00155d80 con 0x7fea000c58c0 2024-03-28T18:55:30.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.670+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea00156340 con 0x7fea000c58c0 2024-03-28T18:55:30.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.671+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea00069640 con 0x7fea000c58c0 2024-03-28T18:55:30.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.672+0000 7fe9f6000640 1 -- 172.21.15.27:0/379998175 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe9ec0453f0 con 0x7fea000c58c0 2024-03-28T18:55:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.673+0000 7fe9f6000640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9e006bd70 0x7fe9e006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.673+0000 7fe9f6000640 1 -- 172.21.15.27:0/379998175 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe9ec0ac8d0 con 0x7fea000c58c0 2024-03-28T18:55:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.673+0000 7fe9fde00640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9e006bd70 0x7fe9e006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.674+0000 7fe9fde00640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9e006bd70 0x7fe9e006e230 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0006fd0 tx=0x7fe9f0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.678+0000 7fe9f6000640 1 -- 172.21.15.27:0/379998175 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe9ec040070 con 0x7fea000c58c0 2024-03-28T18:55:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.877+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea00156580 con 0x7fea000c58c0 2024-03-28T18:55:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.880+0000 7fe9f6000640 1 -- 172.21.15.27:0/379998175 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12848 (secure 0 0 0) 0x7fe9ec07b670 con 0x7fea000c58c0 2024-03-28T18:55:30.880 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:30.880 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:25.182434+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:30.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9e006bd70 msgr2=0x7fe9e006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9e006bd70 0x7fe9e006e230 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0006fd0 tx=0x7fe9f0008040 comp rx=0 tx=0).stop 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c58c0 msgr2=0x7fea000c0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c58c0 0x7fea000c0680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec02b8c0 tx=0x7fe9ec0046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 shutdown_connections 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9e006bd70 0x7fe9e006e230 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea000c8ed0 0x7fea001eb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea000c58c0 0x7fea000c0680 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 --2- 172.21.15.27:0/379998175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea000be5d0 0x7fea0014ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.882+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 >> 172.21.15.27:0/379998175 conn(0x7fea0006b6f0 msgr2=0x7fea000ae560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.883+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 shutdown_connections 2024-03-28T18:55:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:30.883+0000 7fe9ff200640 1 -- 172.21.15.27:0/379998175 wait complete. 2024-03-28T18:55:30.902 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:55:30.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:55:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.134+0000 7f8e61800640 1 Processor -- start 2024-03-28T18:55:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.134+0000 7f8e61800640 1 -- start start 2024-03-28T18:55:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.135+0000 7f8e61800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c150990 0x7f8e5c1573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.135+0000 7f8e61800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8e5c0ad4e0 0x7f8e5c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.135+0000 7f8e61800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c0adaa0 con 0x7f8e5c0c51d0 2024-03-28T18:55:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.135+0000 7f8e61800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c071870 con 0x7f8e5c0ad4e0 2024-03-28T18:55:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.135+0000 7f8e61800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c071b50 con 0x7f8e5c150990 2024-03-28T18:55:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.135+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c150990 0x7f8e5c1573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.135+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c150990 0x7f8e5c1573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:47672/0 (socket says 172.21.15.27:47672) 2024-03-28T18:55:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.135+0000 7f8e5be00640 1 -- 172.21.15.27:0/2961244100 learned_addr learned my addr 172.21.15.27:0/2961244100 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.136+0000 7f8e5b400640 1 --2- 172.21.15.27:0/2961244100 >> v2:172.21.15.155:3300/0 conn(0x7f8e5c0ad4e0 0x7f8e5c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.136+0000 7f8e5be00640 1 -- 172.21.15.27:0/2961244100 >> v2:172.21.15.155:3300/0 conn(0x7f8e5c0ad4e0 msgr2=0x7f8e5c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.136+0000 7f8e5be00640 1 --2- 172.21.15.27:0/2961244100 >> v2:172.21.15.155:3300/0 conn(0x7f8e5c0ad4e0 0x7f8e5c0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.136+0000 7f8e5be00640 1 -- 172.21.15.27:0/2961244100 >> v1:172.21.15.155:6789/0 conn(0x7f8e5c0c51d0 legacy=0x7f8e5c153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.136+0000 7f8e5be00640 1 -- 172.21.15.27:0/2961244100 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c071e30 con 0x7f8e5c150990 2024-03-28T18:55:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.137+0000 7f8e5be00640 1 --2- 172.21.15.27:0/2961244100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c150990 0x7f8e5c1573f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009e50 tx=0x7f8e4c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ea7ac6ae9b35b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.138+0000 7f8e5aa00640 1 -- 172.21.15.27:0/2961244100 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c02ca70 con 0x7f8e5c150990 2024-03-28T18:55:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.138+0000 7f8e5aa00640 1 -- 172.21.15.27:0/2961244100 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c02cc10 con 0x7f8e5c150990 2024-03-28T18:55:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.138+0000 7f8e5aa00640 1 -- 172.21.15.27:0/2961244100 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0343f0 con 0x7f8e5c150990 2024-03-28T18:55:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.138+0000 7f8e61800640 1 -- 172.21.15.27:0/2961244100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c150990 msgr2=0x7f8e5c1573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.138+0000 7f8e61800640 1 --2- 172.21.15.27:0/2961244100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c150990 0x7f8e5c1573f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009e50 tx=0x7f8e4c02bed0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.139+0000 7f8e61800640 1 -- 172.21.15.27:0/2961244100 shutdown_connections 2024-03-28T18:55:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.139+0000 7f8e61800640 1 --2- 172.21.15.27:0/2961244100 >> v2:172.21.15.155:3300/0 conn(0x7f8e5c0ad4e0 0x7f8e5c0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.139+0000 7f8e61800640 1 --2- 172.21.15.27:0/2961244100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c150990 0x7f8e5c1573f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.139+0000 7f8e61800640 1 -- 172.21.15.27:0/2961244100 >> 172.21.15.27:0/2961244100 conn(0x7f8e5c0c8130 msgr2=0x7f8e5c0c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.139+0000 7f8e61800640 1 -- 172.21.15.27:0/2961244100 shutdown_connections 2024-03-28T18:55:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.139+0000 7f8e61800640 1 -- 172.21.15.27:0/2961244100 wait complete. 2024-03-28T18:55:31.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.140+0000 7f8e61800640 1 Processor -- start 2024-03-28T18:55:31.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.140+0000 7f8e61800640 1 -- start start 2024-03-28T18:55:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.141+0000 7f8e61800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e5c0ad4e0 0x7f8e5c0bf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.141+0000 7f8e61800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c0c51d0 0x7f8e5c0c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.141+0000 7f8e61800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c150990 0x7f8e5c14f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.141+0000 7f8e61800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c150d70 con 0x7f8e5c150990 2024-03-28T18:55:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.141+0000 7f8e61800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c072110 con 0x7f8e5c0ad4e0 2024-03-28T18:55:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.141+0000 7f8e61800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c150ef0 con 0x7f8e5c0c51d0 2024-03-28T18:55:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c150990 0x7f8e5c14f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c150990 0x7f8e5c14f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53240/0 (socket says 172.21.15.27:53240) 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e60e00640 1 -- 172.21.15.27:0/2028430077 learned_addr learned my addr 172.21.15.27:0/2028430077 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e5be00640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e5c0ad4e0 0x7f8e5c0bf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e5b400640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c0c51d0 0x7f8e5c0c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e60e00640 1 -- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c0c51d0 msgr2=0x7f8e5c0c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e60e00640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c0c51d0 0x7f8e5c0c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e60e00640 1 -- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e5c0ad4e0 msgr2=0x7f8e5c0bf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e60e00640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e5c0ad4e0 0x7f8e5c0bf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e60e00640 1 -- 172.21.15.27:0/2028430077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c0c1860 con 0x7f8e5c150990 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.142+0000 7f8e5b400640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c0c51d0 0x7f8e5c0c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.143+0000 7f8e60e00640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c150990 0x7f8e5c14f810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400c3d0 tx=0x7f8e5400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:31.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.143+0000 7f8e53e00640 1 -- 172.21.15.27:0/2028430077 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54017070 con 0x7f8e5c150990 2024-03-28T18:55:31.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.143+0000 7f8e53e00640 1 -- 172.21.15.27:0/2028430077 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e540052d0 con 0x7f8e5c150990 2024-03-28T18:55:31.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.143+0000 7f8e53e00640 1 -- 172.21.15.27:0/2028430077 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54003830 con 0x7f8e5c150990 2024-03-28T18:55:31.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.143+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c0bf210 con 0x7f8e5c150990 2024-03-28T18:55:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.143+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c0c4120 con 0x7f8e5c150990 2024-03-28T18:55:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.144+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c0ac6e0 con 0x7f8e5c150990 2024-03-28T18:55:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.145+0000 7f8e53e00640 1 -- 172.21.15.27:0/2028430077 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8e5401a070 con 0x7f8e5c150990 2024-03-28T18:55:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.146+0000 7f8e53e00640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e3806bbd0 0x7f8e3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.146+0000 7f8e53e00640 1 -- 172.21.15.27:0/2028430077 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e54089a20 con 0x7f8e5c150990 2024-03-28T18:55:31.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.146+0000 7f8e5be00640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e3806bbd0 0x7f8e3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.147+0000 7f8e5be00640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e3806bbd0 0x7f8e3806e090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c035000 tx=0x7f8e4c030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.150+0000 7f8e53e00640 1 -- 172.21.15.27:0/2028430077 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8e540582c0 con 0x7f8e5c150990 2024-03-28T18:55:31.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.349+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e5c04f400 con 0x7f8e5c150990 2024-03-28T18:55:31.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.350+0000 7f8e53e00640 1 -- 172.21.15.27:0/2028430077 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e5405c360 con 0x7f8e5c150990 2024-03-28T18:55:31.351 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.352+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e3806bbd0 msgr2=0x7f8e3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.352+0000 7f8e61800640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e3806bbd0 0x7f8e3806e090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c035000 tx=0x7f8e4c030000 comp rx=0 tx=0).stop 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.352+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c150990 msgr2=0x7f8e5c14f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.352+0000 7f8e61800640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c150990 0x7f8e5c14f810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400c3d0 tx=0x7f8e5400c890 comp rx=0 tx=0).stop 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.353+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 shutdown_connections 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.353+0000 7f8e61800640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e3806bbd0 0x7f8e3806e090 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.353+0000 7f8e61800640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5c150990 0x7f8e5c14f810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.353+0000 7f8e61800640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e5c0c51d0 0x7f8e5c0c3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.353+0000 7f8e61800640 1 --2- 172.21.15.27:0/2028430077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e5c0ad4e0 0x7f8e5c0bf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.353+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 >> 172.21.15.27:0/2028430077 conn(0x7f8e5c0c8130 msgr2=0x7f8e5c06b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.353+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 shutdown_connections 2024-03-28T18:55:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.353+0000 7f8e61800640 1 -- 172.21.15.27:0/2028430077 wait complete. 2024-03-28T18:55:31.372 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T18:55:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.592+0000 7f0983000640 1 Processor -- start 2024-03-28T18:55:31.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.592+0000 7f0983000640 1 -- start start 2024-03-28T18:55:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0983000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c5e80 0x7f097c0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0983000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f097c0c1f50 0x7f097c0ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0983000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f097c0abd90 0x7f097c0aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0983000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097c0ac220 con 0x7f097c0abd90 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0983000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097c06d650 con 0x7f097c0c1f50 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0983000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c06e1d0 con 0x7f097c0c5e80 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0981000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f097c0abd90 0x7f097c0aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0981000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f097c0abd90 0x7f097c0aa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53248/0 (socket says 172.21.15.27:53248) 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0981000640 1 -- 172.21.15.27:0/1702735030 learned_addr learned my addr 172.21.15.27:0/1702735030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f0973400640 1 --2- 172.21.15.27:0/1702735030 >> v2:172.21.15.155:3300/0 conn(0x7f097c0c1f50 0x7f097c0ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.594+0000 7f097be00640 1 --2- 172.21.15.27:0/1702735030 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c5e80 0x7f097c0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.595+0000 7f0981000640 1 -- 172.21.15.27:0/1702735030 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c5e80 msgr2=0x7f097c0c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.595+0000 7f0981000640 1 --2- 172.21.15.27:0/1702735030 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c5e80 0x7f097c0c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.595+0000 7f0981000640 1 -- 172.21.15.27:0/1702735030 >> v2:172.21.15.155:3300/0 conn(0x7f097c0c1f50 msgr2=0x7f097c0ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:31.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.595+0000 7f0981000640 1 --2- 172.21.15.27:0/1702735030 >> v2:172.21.15.155:3300/0 conn(0x7f097c0c1f50 0x7f097c0ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.595+0000 7f0981000640 1 -- 172.21.15.27:0/1702735030 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c06d8a0 con 0x7f097c0abd90 2024-03-28T18:55:31.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.595+0000 7f097be00640 1 --2- 172.21.15.27:0/1702735030 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c5e80 0x7f097c0c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.596+0000 7f0981000640 1 --2- 172.21.15.27:0/1702735030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0aa570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f097c0c6b60 tx=0x7f097402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeaf9d6d5c554a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.596+0000 7f097b400640 1 -- 172.21.15.27:0/1702735030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097402c950 con 0x7f097c0abd90 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.596+0000 7f097b400640 1 -- 172.21.15.27:0/1702735030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097402caf0 con 0x7f097c0abd90 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.596+0000 7f097b400640 1 -- 172.21.15.27:0/1702735030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09740343f0 con 0x7f097c0abd90 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 -- 172.21.15.27:0/1702735030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 msgr2=0x7f097c0aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 --2- 172.21.15.27:0/1702735030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0aa570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f097c0c6b60 tx=0x7f097402bdb0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 -- 172.21.15.27:0/1702735030 shutdown_connections 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 --2- 172.21.15.27:0/1702735030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0aa570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 --2- 172.21.15.27:0/1702735030 >> v2:172.21.15.155:3300/0 conn(0x7f097c0c1f50 0x7f097c0ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 --2- 172.21.15.27:0/1702735030 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c5e80 0x7f097c0c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 -- 172.21.15.27:0/1702735030 >> 172.21.15.27:0/1702735030 conn(0x7f097c06b9d0 msgr2=0x7f097c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 -- 172.21.15.27:0/1702735030 shutdown_connections 2024-03-28T18:55:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.597+0000 7f0983000640 1 -- 172.21.15.27:0/1702735030 wait complete. 2024-03-28T18:55:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.599+0000 7f0983000640 1 Processor -- start 2024-03-28T18:55:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.599+0000 7f0983000640 1 -- start start 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f0983000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0bf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f0983000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c1f50 0x7f097c1ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f0983000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f097c0c5e80 0x7f097c1dfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f0983000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0c31f0 con 0x7f097c0abd90 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f0983000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c06de60 con 0x7f097c0c5e80 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f0983000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0c3370 con 0x7f097c0c1f50 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f097be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0bf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f097be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0bf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53264/0 (socket says 172.21.15.27:53264) 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f097be00640 1 -- 172.21.15.27:0/3627322373 learned_addr learned my addr 172.21.15.27:0/3627322373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f0981000640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f097c0c5e80 0x7f097c1dfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f0973400640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c1f50 0x7f097c1ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f097be00640 1 -- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c1f50 msgr2=0x7f097c1ee340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.600+0000 7f097be00640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c1f50 0x7f097c1ee340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f097be00640 1 -- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f097c0c5e80 msgr2=0x7f097c1dfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f097be00640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f097c0c5e80 0x7f097c1dfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f097be00640 1 -- 172.21.15.27:0/3627322373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c1eea40 con 0x7f097c0abd90 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f0981000640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f097c0c5e80 0x7f097c1dfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f097be00640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0bf680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f096c00d7d0 tx=0x7f096c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f0978c00640 1 -- 172.21.15.27:0/3627322373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c004030 con 0x7f097c0abd90 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f0978c00640 1 -- 172.21.15.27:0/3627322373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c0041d0 con 0x7f097c0abd90 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f0978c00640 1 -- 172.21.15.27:0/3627322373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c00b6c0 con 0x7f097c0abd90 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c1eed20 con 0x7f097c0abd90 2024-03-28T18:55:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.601+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c1e0410 con 0x7f097c0abd90 2024-03-28T18:55:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.603+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097c1ef240 con 0x7f097c0abd90 2024-03-28T18:55:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.603+0000 7f0978c00640 1 -- 172.21.15.27:0/3627322373 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f096c00b860 con 0x7f097c0abd90 2024-03-28T18:55:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.604+0000 7f0978c00640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f095806bbd0 0x7f095806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.605+0000 7f0978c00640 1 -- 172.21.15.27:0/3627322373 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f096c08a9d0 con 0x7f097c0abd90 2024-03-28T18:55:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.605+0000 7f0973400640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f095806bbd0 0x7f095806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.605+0000 7f0973400640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f095806bbd0 0x7f095806e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f097c0bfbc0 tx=0x7f0964008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.609+0000 7f0978c00640 1 -- 172.21.15.27:0/3627322373 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f096c059810 con 0x7f097c0abd90 2024-03-28T18:55:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.803+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f097c1eadd0 con 0x7f097c0abd90 2024-03-28T18:55:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.804+0000 7f0978c00640 1 -- 172.21.15.27:0/3627322373 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f096c059110 con 0x7f097c0abd90 2024-03-28T18:55:31.805 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:31.805 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-28T18:55:31.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-28T18:55:31.805 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-28T18:55:31.805 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-28T18:55:31.805 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-28T18:55:31.805 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-28T18:55:31.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f095806bbd0 msgr2=0x7f095806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f095806bbd0 0x7f095806e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f097c0bfbc0 tx=0x7f0964008040 comp rx=0 tx=0).stop 2024-03-28T18:55:31.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 msgr2=0x7f097c0bf680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:31.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0bf680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f096c00d7d0 tx=0x7f096c00dc90 comp rx=0 tx=0).stop 2024-03-28T18:55:31.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 shutdown_connections 2024-03-28T18:55:31.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f095806bbd0 0x7f095806e090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f097c0c5e80 0x7f097c1dfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f097c0c1f50 0x7f097c1ee340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 --2- 172.21.15.27:0/3627322373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f097c0abd90 0x7f097c0bf680 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:31.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 >> 172.21.15.27:0/3627322373 conn(0x7f097c06b9d0 msgr2=0x7f097c0c6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:31.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 shutdown_connections 2024-03-28T18:55:31.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:31.806+0000 7f0983000640 1 -- 172.21.15.27:0/3627322373 wait complete. 2024-03-28T18:55:31.823 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'a' 2024-03-28T18:55:31.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_metadata --pg_num_min 64 2024-03-28T18:55:32.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.040+0000 7f256a600640 1 Processor -- start 2024-03-28T18:55:32.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.040+0000 7f256a600640 1 -- start start 2024-03-28T18:55:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.042+0000 7f256a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 0x7f2564153d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.042+0000 7f256a600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f25640727d0 0x7f25640abe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.042+0000 7f256a600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2564150d30 con 0x7f2564157220 2024-03-28T18:55:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.042+0000 7f256a600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25640ac340 con 0x7f25640727d0 2024-03-28T18:55:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.042+0000 7f256a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256406f140 con 0x7f2564150950 2024-03-28T18:55:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.042+0000 7f2563e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2564157220 0x7f25640c27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35602/0 (socket says 172.21.15.27:35602) 2024-03-28T18:55:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.042+0000 7f2563e00640 1 -- 172.21.15.27:0/2484437032 learned_addr learned my addr 172.21.15.27:0/2484437032 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2562a00640 1 --2- 172.21.15.27:0/2484437032 >> v2:172.21.15.155:3300/0 conn(0x7f25640727d0 0x7f25640abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2563400640 1 --2- 172.21.15.27:0/2484437032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 0x7f2564153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2562000640 1 -- 172.21.15.27:0/2484437032 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359839603 0 0) 0x7f2564150d30 con 0x7f2564157220 2024-03-28T18:55:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2562000640 1 -- 172.21.15.27:0/2484437032 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2548003520 con 0x7f2564157220 2024-03-28T18:55:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2562000640 1 -- 172.21.15.27:0/2484437032 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 762763211 0 0) 0x7f2548003520 con 0x7f2564157220 2024-03-28T18:55:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2562a00640 1 -- 172.21.15.27:0/2484437032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 msgr2=0x7f2564153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2562a00640 1 --2- 172.21.15.27:0/2484437032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 0x7f2564153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2562a00640 1 -- 172.21.15.27:0/2484437032 >> v1:172.21.15.27:6789/0 conn(0x7f2564157220 legacy=0x7f25640c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.043+0000 7f2562a00640 1 -- 172.21.15.27:0/2484437032 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256406eba0 con 0x7f25640727d0 2024-03-28T18:55:32.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.045+0000 7f2562a00640 1 --2- 172.21.15.27:0/2484437032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25640727d0 0x7f25640abe00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f25500097a0 tx=0x7f255002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=361d73a9b6b427f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:32.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.045+0000 7f2562000640 1 -- 172.21.15.27:0/2484437032 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550037070 con 0x7f25640727d0 2024-03-28T18:55:32.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.045+0000 7f2562000640 1 -- 172.21.15.27:0/2484437032 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2550030510 con 0x7f25640727d0 2024-03-28T18:55:32.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.045+0000 7f2562000640 1 -- 172.21.15.27:0/2484437032 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c0019f0 con 0x7f25640727d0 2024-03-28T18:55:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.046+0000 7f256a600640 1 -- 172.21.15.27:0/2484437032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25640727d0 msgr2=0x7f25640abe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.046+0000 7f256a600640 1 --2- 172.21.15.27:0/2484437032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25640727d0 0x7f25640abe00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f25500097a0 tx=0x7f255002b710 comp rx=0 tx=0).stop 2024-03-28T18:55:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.046+0000 7f256a600640 1 -- 172.21.15.27:0/2484437032 shutdown_connections 2024-03-28T18:55:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.046+0000 7f256a600640 1 --2- 172.21.15.27:0/2484437032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25640727d0 0x7f25640abe00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.046+0000 7f256a600640 1 --2- 172.21.15.27:0/2484437032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 0x7f2564153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.046+0000 7f256a600640 1 -- 172.21.15.27:0/2484437032 >> 172.21.15.27:0/2484437032 conn(0x7f25640c38a0 msgr2=0x7f25640c3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.046+0000 7f256a600640 1 -- 172.21.15.27:0/2484437032 shutdown_connections 2024-03-28T18:55:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.046+0000 7f256a600640 1 -- 172.21.15.27:0/2484437032 wait complete. 2024-03-28T18:55:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.048+0000 7f256a600640 1 Processor -- start 2024-03-28T18:55:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.048+0000 7f256a600640 1 -- start start 2024-03-28T18:55:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f256a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25640727d0 0x7f25641ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f256a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 0x7f2564155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f256a600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2564157220 0x7f25640c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f256a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25640c2ec0 con 0x7f25640727d0 2024-03-28T18:55:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f256a600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2564156a20 con 0x7f2564157220 2024-03-28T18:55:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f256a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25640c3040 con 0x7f2564150950 2024-03-28T18:55:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f2563400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25640727d0 0x7f25641ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f2563400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25640727d0 0x7f25641ee8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53274/0 (socket says 172.21.15.27:53274) 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f2563400640 1 -- 172.21.15.27:0/2301287498 learned_addr learned my addr 172.21.15.27:0/2301287498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f2563e00640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2564157220 0x7f25640c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.049+0000 7f2562a00640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 0x7f2564155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f2563400640 1 -- 172.21.15.27:0/2301287498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 msgr2=0x7f2564155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f2563400640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 0x7f2564155b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f2563400640 1 -- 172.21.15.27:0/2301287498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2564157220 msgr2=0x7f25640c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f2563400640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2564157220 0x7f25640c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f2563400640 1 -- 172.21.15.27:0/2301287498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25641eee00 con 0x7f25640727d0 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f2563400640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25640727d0 0x7f25641ee8c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f255400e9b0 tx=0x7f255400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f255b400640 1 -- 172.21.15.27:0/2301287498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554017070 con 0x7f25640727d0 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f255b400640 1 -- 172.21.15.27:0/2301287498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255400c9a0 con 0x7f25640727d0 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f255b400640 1 -- 172.21.15.27:0/2301287498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255400cd90 con 0x7f25640727d0 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256414ab80 con 0x7f25640727d0 2024-03-28T18:55:32.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.050+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256414b140 con 0x7f25640727d0 2024-03-28T18:55:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.052+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2564068180 con 0x7f25640727d0 2024-03-28T18:55:32.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.053+0000 7f255b400640 1 -- 172.21.15.27:0/2301287498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f255401a070 con 0x7f25640727d0 2024-03-28T18:55:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.053+0000 7f255b400640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f254406bc00 0x7f254406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.053+0000 7f255b400640 1 -- 172.21.15.27:0/2301287498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f255408a6a0 con 0x7f25640727d0 2024-03-28T18:55:32.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.054+0000 7f2562a00640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f254406bc00 0x7f254406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.055+0000 7f2562a00640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f254406bc00 0x7f254406e0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2550034000 tx=0x7f2550008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.058+0000 7f255b400640 1 -- 172.21.15.27:0/2301287498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2554058e80 con 0x7f25640727d0 2024-03-28T18:55:32.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.263+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2564150100 con 0x7f25640727d0 2024-03-28T18:55:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.443+0000 7f255b400640 1 -- 172.21.15.27:0/2301287498 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64}]=0 pool 'a_metadata' created v17) v1 ==== 128+0+0 (secure 0 0 0) 0x7f255405cf20 con 0x7f25640727d0 2024-03-28T18:55:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'a_metadata' created 2024-03-28T18:55:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f254406bc00 msgr2=0x7f254406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f254406bc00 0x7f254406e0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2550034000 tx=0x7f2550008030 comp rx=0 tx=0).stop 2024-03-28T18:55:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25640727d0 msgr2=0x7f25641ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25640727d0 0x7f25641ee8c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f255400e9b0 tx=0x7f255400ee70 comp rx=0 tx=0).stop 2024-03-28T18:55:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 shutdown_connections 2024-03-28T18:55:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f254406bc00 0x7f254406e0c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2564157220 0x7f25640c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2564150950 0x7f2564155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 --2- 172.21.15.27:0/2301287498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25640727d0 0x7f25641ee8c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.447+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 >> 172.21.15.27:0/2301287498 conn(0x7f25640c38a0 msgr2=0x7f25640c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.448+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 shutdown_connections 2024-03-28T18:55:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.448+0000 7f256a600640 1 -- 172.21.15.27:0/2301287498 wait complete. 2024-03-28T18:55:32.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T18:55:32.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.697+0000 7f961b200640 1 Processor -- start 2024-03-28T18:55:32.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.697+0000 7f961b200640 1 -- start start 2024-03-28T18:55:32.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.698+0000 7f961b200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f961c0c6c90 0x7f961c0c1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.698+0000 7f961b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f961c0aa5e0 0x7f961c0aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.698+0000 7f961b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961c0c6670 con 0x7f961c0c4ed0 2024-03-28T18:55:32.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.698+0000 7f961b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961c0aa170 con 0x7f961c0aa5e0 2024-03-28T18:55:32.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.698+0000 7f961b200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f961c0c1da0 con 0x7f961c0c6c90 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f9619400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f961c0c4ed0 0x7f961c0ab7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35618/0 (socket says 172.21.15.27:35618) 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f9619400640 1 -- 172.21.15.27:0/3173699594 learned_addr learned my addr 172.21.15.27:0/3173699594 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f961a800640 1 --2- 172.21.15.27:0/3173699594 >> v2:172.21.15.27:3300/0 conn(0x7f961c0aa5e0 0x7f961c0aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f9619e00640 1 --2- 172.21.15.27:0/3173699594 >> v2:172.21.15.155:3300/0 conn(0x7f961c0c6c90 0x7f961c0c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f9618a00640 1 -- 172.21.15.27:0/3173699594 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802915407 0 0) 0x7f961c0c6670 con 0x7f961c0c4ed0 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f9618a00640 1 -- 172.21.15.27:0/3173699594 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc003520 con 0x7f961c0c4ed0 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f961a800640 1 -- 172.21.15.27:0/3173699594 >> v2:172.21.15.155:3300/0 conn(0x7f961c0c6c90 msgr2=0x7f961c0c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f961a800640 1 --2- 172.21.15.27:0/3173699594 >> v2:172.21.15.155:3300/0 conn(0x7f961c0c6c90 0x7f961c0c1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f961a800640 1 -- 172.21.15.27:0/3173699594 >> v1:172.21.15.27:6789/0 conn(0x7f961c0c4ed0 legacy=0x7f961c0ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.699+0000 7f961a800640 1 -- 172.21.15.27:0/3173699594 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961c0c1ff0 con 0x7f961c0aa5e0 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.701+0000 7f961a800640 1 --2- 172.21.15.27:0/3173699594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0aa5e0 0x7f961c0aaa10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f961c0c1f20 tx=0x7f961402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30eeffc2722bbd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.701+0000 7f9618a00640 1 -- 172.21.15.27:0/3173699594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96140315a0 con 0x7f961c0aa5e0 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.701+0000 7f9618a00640 1 -- 172.21.15.27:0/3173699594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9614037070 con 0x7f961c0aa5e0 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.701+0000 7f9618a00640 1 -- 172.21.15.27:0/3173699594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614031960 con 0x7f961c0aa5e0 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.701+0000 7f961b200640 1 -- 172.21.15.27:0/3173699594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0aa5e0 msgr2=0x7f961c0aaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.701+0000 7f961b200640 1 --2- 172.21.15.27:0/3173699594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0aa5e0 0x7f961c0aaa10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f961c0c1f20 tx=0x7f961402b840 comp rx=0 tx=0).stop 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.702+0000 7f961b200640 1 -- 172.21.15.27:0/3173699594 shutdown_connections 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.702+0000 7f961b200640 1 --2- 172.21.15.27:0/3173699594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0aa5e0 0x7f961c0aaa10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.702+0000 7f961b200640 1 --2- 172.21.15.27:0/3173699594 >> v2:172.21.15.155:3300/0 conn(0x7f961c0c6c90 0x7f961c0c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.702+0000 7f961b200640 1 -- 172.21.15.27:0/3173699594 >> 172.21.15.27:0/3173699594 conn(0x7f961c06d4d0 msgr2=0x7f961c06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.702+0000 7f961b200640 1 -- 172.21.15.27:0/3173699594 shutdown_connections 2024-03-28T18:55:32.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.702+0000 7f961b200640 1 -- 172.21.15.27:0/3173699594 wait complete. 2024-03-28T18:55:32.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.703+0000 7f961b200640 1 Processor -- start 2024-03-28T18:55:32.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.704+0000 7f961b200640 1 -- start start 2024-03-28T18:55:32.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.704+0000 7f961b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f961c0aa5e0 0x7f961c1eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.704+0000 7f961b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0c4ed0 0x7f961c0a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f961b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f961c0c6c90 0x7f961c1eb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f961b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0c6430 con 0x7f961c0c4ed0 2024-03-28T18:55:32.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f961b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0c62b0 con 0x7f961c0c6c90 2024-03-28T18:55:32.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f961b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0abef0 con 0x7f961c0aa5e0 2024-03-28T18:55:32.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0c4ed0 0x7f961c0a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0c4ed0 0x7f961c0a77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53294/0 (socket says 172.21.15.27:53294) 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619400640 1 -- 172.21.15.27:0/493434985 learned_addr learned my addr 172.21.15.27:0/493434985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619e00640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f961c0aa5e0 0x7f961c1eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f961a800640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f961c0c6c90 0x7f961c1eb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619400640 1 -- 172.21.15.27:0/493434985 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f961c0aa5e0 msgr2=0x7f961c1eeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619400640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f961c0aa5e0 0x7f961c1eeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619400640 1 -- 172.21.15.27:0/493434985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f961c0c6c90 msgr2=0x7f961c1eb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619400640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f961c0c6c90 0x7f961c1eb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.705+0000 7f9619400640 1 -- 172.21.15.27:0/493434985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961c153770 con 0x7f961c0c4ed0 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.706+0000 7f9619400640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0c4ed0 0x7f961c0a77c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f960400cb70 tx=0x7f960400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.706+0000 7f9612000640 1 -- 172.21.15.27:0/493434985 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604017070 con 0x7f961c0c4ed0 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.706+0000 7f9612000640 1 -- 172.21.15.27:0/493434985 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960400ad40 con 0x7f961c0c4ed0 2024-03-28T18:55:32.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.706+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c153a50 con 0x7f961c0c4ed0 2024-03-28T18:55:32.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.706+0000 7f9612000640 1 -- 172.21.15.27:0/493434985 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604012620 con 0x7f961c0c4ed0 2024-03-28T18:55:32.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.706+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c154010 con 0x7f961c0c4ed0 2024-03-28T18:55:32.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.707+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c04f400 con 0x7f961c0c4ed0 2024-03-28T18:55:32.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.709+0000 7f9612000640 1 -- 172.21.15.27:0/493434985 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f960401a070 con 0x7f961c0c4ed0 2024-03-28T18:55:32.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.710+0000 7f9612000640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f95f806bbd0 0x7f95f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:32.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.710+0000 7f9612000640 1 -- 172.21.15.27:0/493434985 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5222+0+0 (secure 0 0 0) 0x7f960408ab80 con 0x7f961c0c4ed0 2024-03-28T18:55:32.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.710+0000 7f9619e00640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f95f806bbd0 0x7f95f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:32.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.710+0000 7f9619e00640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f95f806bbd0 0x7f95f806e090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f960c009a10 tx=0x7f960c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:32.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.713+0000 7f9612000640 1 -- 172.21.15.27:0/493434985 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9604021020 con 0x7f961c0c4ed0 2024-03-28T18:55:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:32.916+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f961c0c4bd0 con 0x7f961c0c4ed0 2024-03-28T18:55:33.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.460+0000 7f9612000640 1 -- 172.21.15.27:0/493434985 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'a_data' created v18) v1 ==== 160+0+0 (secure 0 0 0) 0x7f9604059830 con 0x7f961c0c4ed0 2024-03-28T18:55:33.461 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'a_data' created 2024-03-28T18:55:33.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.463+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f95f806bbd0 msgr2=0x7f95f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:33.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.463+0000 7f961b200640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f95f806bbd0 0x7f95f806e090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f960c009a10 tx=0x7f960c008040 comp rx=0 tx=0).stop 2024-03-28T18:55:33.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.463+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0c4ed0 msgr2=0x7f961c0a77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:33.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.463+0000 7f961b200640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0c4ed0 0x7f961c0a77c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f960400cb70 tx=0x7f960400a430 comp rx=0 tx=0).stop 2024-03-28T18:55:33.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.464+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 shutdown_connections 2024-03-28T18:55:33.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.464+0000 7f961b200640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f95f806bbd0 0x7f95f806e090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.464+0000 7f961b200640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f961c0c6c90 0x7f961c1eb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.464+0000 7f961b200640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961c0c4ed0 0x7f961c0a77c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.464+0000 7f961b200640 1 --2- 172.21.15.27:0/493434985 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f961c0aa5e0 0x7f961c1eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.464+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 >> 172.21.15.27:0/493434985 conn(0x7f961c06d4d0 msgr2=0x7f961c06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:33.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.464+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 shutdown_connections 2024-03-28T18:55:33.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.464+0000 7f961b200640 1 -- 172.21.15.27:0/493434985 wait complete. 2024-03-28T18:55:33.499 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new a a_metadata a_data 2024-03-28T18:55:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.694+0000 7fd507e00640 1 Processor -- start 2024-03-28T18:55:33.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.694+0000 7fd507e00640 1 -- start start 2024-03-28T18:55:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.695+0000 7fd507e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5080c2cb0 0x7fd508071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.695+0000 7fd507e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd5080a81c0 0x7fd5080c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd507e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080c4c50 con 0x7fd5081522b0 2024-03-28T18:55:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd507e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5080c4a00 con 0x7fd5080c2cb0 2024-03-28T18:55:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd507e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5080c5860 con 0x7fd5080a81c0 2024-03-28T18:55:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd506a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5080c2cb0 0x7fd508071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd507400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd5081522b0 0x7fd5080a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35634/0 (socket says 172.21.15.27:35634) 2024-03-28T18:55:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd507400640 1 -- 172.21.15.27:0/2729905294 learned_addr learned my addr 172.21.15.27:0/2729905294 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd506000640 1 --2- 172.21.15.27:0/2729905294 >> v2:172.21.15.155:3300/0 conn(0x7fd5080a81c0 0x7fd5080c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd506a00640 1 -- 172.21.15.27:0/2729905294 >> v2:172.21.15.155:3300/0 conn(0x7fd5080a81c0 msgr2=0x7fd5080c5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd506a00640 1 --2- 172.21.15.27:0/2729905294 >> v2:172.21.15.155:3300/0 conn(0x7fd5080a81c0 0x7fd5080c5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd506a00640 1 -- 172.21.15.27:0/2729905294 >> v1:172.21.15.27:6789/0 conn(0x7fd5081522b0 legacy=0x7fd5080a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd506a00640 1 -- 172.21.15.27:0/2729905294 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080c4f30 con 0x7fd5080c2cb0 2024-03-28T18:55:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.696+0000 7fd505600640 1 -- 172.21.15.27:0/2729905294 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697346955 0 0) 0x7fd5080c4c50 con 0x7fd5081522b0 2024-03-28T18:55:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.698+0000 7fd506a00640 1 --2- 172.21.15.27:0/2729905294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080c2cb0 0x7fd508071b30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80097a0 tx=0x7fd4f802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a0fec7aeed77cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.698+0000 7fd505600640 1 -- 172.21.15.27:0/2729905294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80315a0 con 0x7fd5080c2cb0 2024-03-28T18:55:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.698+0000 7fd505600640 1 -- 172.21.15.27:0/2729905294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8037070 con 0x7fd5080c2cb0 2024-03-28T18:55:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.698+0000 7fd505600640 1 -- 172.21.15.27:0/2729905294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8031960 con 0x7fd5080c2cb0 2024-03-28T18:55:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.698+0000 7fd507e00640 1 -- 172.21.15.27:0/2729905294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080c2cb0 msgr2=0x7fd508071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.698+0000 7fd507e00640 1 --2- 172.21.15.27:0/2729905294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080c2cb0 0x7fd508071b30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80097a0 tx=0x7fd4f802b840 comp rx=0 tx=0).stop 2024-03-28T18:55:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.699+0000 7fd507e00640 1 -- 172.21.15.27:0/2729905294 shutdown_connections 2024-03-28T18:55:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.699+0000 7fd507e00640 1 --2- 172.21.15.27:0/2729905294 >> v2:172.21.15.155:3300/0 conn(0x7fd5080a81c0 0x7fd5080c5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.699+0000 7fd507e00640 1 --2- 172.21.15.27:0/2729905294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080c2cb0 0x7fd508071b30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.699+0000 7fd507e00640 1 -- 172.21.15.27:0/2729905294 >> 172.21.15.27:0/2729905294 conn(0x7fd5080a88b0 msgr2=0x7fd5080a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.699+0000 7fd507e00640 1 -- 172.21.15.27:0/2729905294 shutdown_connections 2024-03-28T18:55:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.699+0000 7fd507e00640 1 -- 172.21.15.27:0/2729905294 wait complete. 2024-03-28T18:55:33.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.700+0000 7fd507e00640 1 Processor -- start 2024-03-28T18:55:33.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.701+0000 7fd507e00640 1 -- start start 2024-03-28T18:55:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd507e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080a81c0 0x7fd50814b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd507e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5080c2cb0 0x7fd5080abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd507e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5081522b0 0x7fd5080ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd507e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080c54f0 con 0x7fd5080a81c0 2024-03-28T18:55:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd507e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508072070 con 0x7fd5081522b0 2024-03-28T18:55:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd507e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080a6e20 con 0x7fd5080c2cb0 2024-03-28T18:55:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd506a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080a81c0 0x7fd50814b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd506a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080a81c0 0x7fd50814b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53306/0 (socket says 172.21.15.27:53306) 2024-03-28T18:55:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd506a00640 1 -- 172.21.15.27:0/1934275198 learned_addr learned my addr 172.21.15.27:0/1934275198 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd506a00640 1 -- 172.21.15.27:0/1934275198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5080c2cb0 msgr2=0x7fd5080abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd506a00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5080c2cb0 0x7fd5080abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd506a00640 1 -- 172.21.15.27:0/1934275198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5081522b0 msgr2=0x7fd5080ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd506a00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5081522b0 0x7fd5080ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.702+0000 7fd506a00640 1 -- 172.21.15.27:0/1934275198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd508149d70 con 0x7fd5080a81c0 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.703+0000 7fd506a00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080a81c0 0x7fd50814b570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80098c0 tx=0x7fd4f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.703+0000 7fd4fea00640 1 -- 172.21.15.27:0/1934275198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8032840 con 0x7fd5080a81c0 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.703+0000 7fd4fea00640 1 -- 172.21.15.27:0/1934275198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f80329e0 con 0x7fd5080a81c0 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.703+0000 7fd4fea00640 1 -- 172.21.15.27:0/1934275198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8032dd0 con 0x7fd5080a81c0 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.703+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5080aca40 con 0x7fd5080a81c0 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.703+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50814a010 con 0x7fd5080a81c0 2024-03-28T18:55:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.704+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50804f400 con 0x7fd5080a81c0 2024-03-28T18:55:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.705+0000 7fd4fea00640 1 -- 172.21.15.27:0/1934275198 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd4f804c020 con 0x7fd5080a81c0 2024-03-28T18:55:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.706+0000 7fd4fea00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4e806bbd0 0x7fd4e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.706+0000 7fd4fea00640 1 -- 172.21.15.27:0/1934275198 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5599+0+0 (secure 0 0 0) 0x7fd4f802f020 con 0x7fd5080a81c0 2024-03-28T18:55:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.706+0000 7fd506000640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4e806bbd0 0x7fd4e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.707+0000 7fd506000640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4e806bbd0 0x7fd4e806e090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5081e0380 tx=0x7fd4f4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.711+0000 7fd4fea00640 1 -- 172.21.15.27:0/1934275198 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd4f8040070 con 0x7fd5080a81c0 2024-03-28T18:55:33.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:33.952+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"} v 0) v1 -- 0x7fd5080c6620 con 0x7fd5080a81c0 2024-03-28T18:55:34.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.477+0000 7fd4fea00640 1 -- 172.21.15.27:0/1934275198 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 157+0+0 (secure 0 0 0) 0x7fd4f807ad40 con 0x7fd5080a81c0 2024-03-28T18:55:34.481 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T18:55:34.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.485+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4e806bbd0 msgr2=0x7fd4e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:34.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.485+0000 7fd507e00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4e806bbd0 0x7fd4e806e090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd5081e0380 tx=0x7fd4f4008040 comp rx=0 tx=0).stop 2024-03-28T18:55:34.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.485+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080a81c0 msgr2=0x7fd50814b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:34.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.485+0000 7fd507e00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080a81c0 0x7fd50814b570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80098c0 tx=0x7fd4f80046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.488+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 shutdown_connections 2024-03-28T18:55:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.488+0000 7fd507e00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4e806bbd0 0x7fd4e806e090 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.488+0000 7fd507e00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5081522b0 0x7fd5080ac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.488+0000 7fd507e00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5080c2cb0 0x7fd5080abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.488+0000 7fd507e00640 1 --2- 172.21.15.27:0/1934275198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5080a81c0 0x7fd50814b570 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.488+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 >> 172.21.15.27:0/1934275198 conn(0x7fd5080a88b0 msgr2=0x7fd5081515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:34.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.490+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 shutdown_connections 2024-03-28T18:55:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.490+0000 7fd507e00640 1 -- 172.21.15.27:0/1934275198 wait complete. 2024-03-28T18:55:34.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:34.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.702+0000 7feb24a00640 1 Processor -- start 2024-03-28T18:55:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.702+0000 7feb24a00640 1 -- start start 2024-03-28T18:55:34.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.703+0000 7feb24a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb200c5320 0x7feb2006e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:34.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.703+0000 7feb24a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7feb20156500 0x7feb200c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:34.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.703+0000 7feb24a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb20159c50 con 0x7feb20156120 2024-03-28T18:55:34.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.703+0000 7feb24a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200c4d70 con 0x7feb200c5320 2024-03-28T18:55:34.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.703+0000 7feb24a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200c1030 con 0x7feb20156500 2024-03-28T18:55:34.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feb20156120 0x7feb200c0920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35648/0 (socket says 172.21.15.27:35648) 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1f400640 1 -- 172.21.15.27:0/2659009157 learned_addr learned my addr 172.21.15.27:0/2659009157 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1ea00640 1 --2- 172.21.15.27:0/2659009157 >> v2:172.21.15.27:3300/0 conn(0x7feb200c5320 0x7feb2006e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1fe00640 1 --2- 172.21.15.27:0/2659009157 >> v2:172.21.15.155:3300/0 conn(0x7feb20156500 0x7feb200c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1e000640 1 -- 172.21.15.27:0/2659009157 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212071393 0 0) 0x7feb20159c50 con 0x7feb20156120 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1e000640 1 -- 172.21.15.27:0/2659009157 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb04003520 con 0x7feb20156120 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1ea00640 1 -- 172.21.15.27:0/2659009157 >> v2:172.21.15.155:3300/0 conn(0x7feb20156500 msgr2=0x7feb200c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1ea00640 1 --2- 172.21.15.27:0/2659009157 >> v2:172.21.15.155:3300/0 conn(0x7feb20156500 0x7feb200c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1ea00640 1 -- 172.21.15.27:0/2659009157 >> v1:172.21.15.27:6789/0 conn(0x7feb20156120 legacy=0x7feb200c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1ea00640 1 -- 172.21.15.27:0/2659009157 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200c4a40 con 0x7feb200c5320 2024-03-28T18:55:34.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.704+0000 7feb1fe00640 1 --2- 172.21.15.27:0/2659009157 >> v2:172.21.15.155:3300/0 conn(0x7feb20156500 0x7feb200c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:34.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.705+0000 7feb1ea00640 1 --2- 172.21.15.27:0/2659009157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb2006e050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7feb200738f0 tx=0x7feb0802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7b678a55aa2b3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:34.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb1e000640 1 -- 172.21.15.27:0/2659009157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08039070 con 0x7feb200c5320 2024-03-28T18:55:34.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb1e000640 1 -- 172.21.15.27:0/2659009157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0802cb60 con 0x7feb200c5320 2024-03-28T18:55:34.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb1e000640 1 -- 172.21.15.27:0/2659009157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08034440 con 0x7feb200c5320 2024-03-28T18:55:34.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb24a00640 1 -- 172.21.15.27:0/2659009157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 msgr2=0x7feb2006e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:34.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb24a00640 1 --2- 172.21.15.27:0/2659009157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb2006e050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7feb200738f0 tx=0x7feb0802bdb0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb24a00640 1 -- 172.21.15.27:0/2659009157 shutdown_connections 2024-03-28T18:55:34.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb24a00640 1 --2- 172.21.15.27:0/2659009157 >> v2:172.21.15.155:3300/0 conn(0x7feb20156500 0x7feb200c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb24a00640 1 --2- 172.21.15.27:0/2659009157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb2006e050 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb24a00640 1 -- 172.21.15.27:0/2659009157 >> 172.21.15.27:0/2659009157 conn(0x7feb2006b0c0 msgr2=0x7feb2006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:34.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb24a00640 1 -- 172.21.15.27:0/2659009157 shutdown_connections 2024-03-28T18:55:34.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.706+0000 7feb24a00640 1 -- 172.21.15.27:0/2659009157 wait complete. 2024-03-28T18:55:34.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.708+0000 7feb24a00640 1 Processor -- start 2024-03-28T18:55:34.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.708+0000 7feb24a00640 1 -- start start 2024-03-28T18:55:34.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb24a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb200a7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb24a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb20156120 0x7feb20149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb24a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb20156500 0x7feb20071f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb24a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200c73d0 con 0x7feb200c5320 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb24a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200c7250 con 0x7feb20156120 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb24a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb200c29a0 con 0x7feb20156500 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb1f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb200a7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb1f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb200a7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53328/0 (socket says 172.21.15.27:53328) 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.709+0000 7feb1f400640 1 -- 172.21.15.27:0/1036955754 learned_addr learned my addr 172.21.15.27:0/1036955754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1ea00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb20156120 0x7feb20149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1fe00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb20156500 0x7feb20071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1f400640 1 -- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb20156500 msgr2=0x7feb20071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:34.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1f400640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb20156500 0x7feb20071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1f400640 1 -- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb20156120 msgr2=0x7feb20149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1f400640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb20156120 0x7feb20149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1f400640 1 -- 172.21.15.27:0/1036955754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200a8fe0 con 0x7feb200c5320 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1ea00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb20156120 0x7feb20149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1fe00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb20156500 0x7feb20071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb1f400640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb200a7090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb1000c430 tx=0x7feb1000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb17400640 1 -- 172.21.15.27:0/1036955754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10017070 con 0x7feb200c5320 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb17400640 1 -- 172.21.15.27:0/1036955754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb10005ce0 con 0x7feb200c5320 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb20073200 con 0x7feb200c5320 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb17400640 1 -- 172.21.15.27:0/1036955754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb100072f0 con 0x7feb200c5320 2024-03-28T18:55:34.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.710+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb20071a10 con 0x7feb200c5320 2024-03-28T18:55:34.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.712+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb20067dd0 con 0x7feb200c5320 2024-03-28T18:55:34.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.712+0000 7feb17400640 1 -- 172.21.15.27:0/1036955754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feb1001a070 con 0x7feb200c5320 2024-03-28T18:55:34.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.714+0000 7feb17400640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feafc06bbd0 0x7feafc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:34.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.714+0000 7feb17400640 1 -- 172.21.15.27:0/1036955754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5689+0+0 (secure 0 0 0) 0x7feb10089f30 con 0x7feb200c5320 2024-03-28T18:55:34.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.714+0000 7feb1ea00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feafc06bbd0 0x7feafc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:34.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.715+0000 7feb1ea00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feafc06bbd0 0x7feafc06e090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feb08036000 tx=0x7feb08030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:34.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.719+0000 7feb17400640 1 -- 172.21.15.27:0/1036955754 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feb10058a20 con 0x7feb200c5320 2024-03-28T18:55:34.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.921+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb2006de30 con 0x7feb200c5320 2024-03-28T18:55:34.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.924+0000 7feb17400640 1 -- 172.21.15.27:0/1036955754 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16562 (secure 0 0 0) 0x7feb10058320 con 0x7feb200c5320 2024-03-28T18:55:34.926 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:34.926 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:33.954956+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feafc06bbd0 msgr2=0x7feafc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feafc06bbd0 0x7feafc06e090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feb08036000 tx=0x7feb08030000 comp rx=0 tx=0).stop 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 msgr2=0x7feb200a7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb200a7090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb1000c430 tx=0x7feb1000c8f0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 shutdown_connections 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feafc06bbd0 0x7feafc06e090 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb20156500 0x7feb20071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb20156120 0x7feb20149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 --2- 172.21.15.27:0/1036955754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200c5320 0x7feb200a7090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.927+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 >> 172.21.15.27:0/1036955754 conn(0x7feb2006b0c0 msgr2=0x7feb200c6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.928+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 shutdown_connections 2024-03-28T18:55:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:34.928+0000 7feb24a00640 1 -- 172.21.15.27:0/1036955754 wait complete. 2024-03-28T18:55:34.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.172+0000 7f5d83800640 1 Processor -- start 2024-03-28T18:55:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.172+0000 7f5d83800640 1 -- start start 2024-03-28T18:55:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.173+0000 7f5d83800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d7c0c5e80 0x7f5d7c0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.173+0000 7f5d83800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5d7c0abf60 0x7f5d7c0aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.173+0000 7f5d83800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c06d8a0 con 0x7f5d7c0c1f50 2024-03-28T18:55:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.173+0000 7f5d83800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c06d650 con 0x7f5d7c0c5e80 2024-03-28T18:55:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.173+0000 7f5d83800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c06e1d0 con 0x7f5d7c0abf60 2024-03-28T18:55:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.174+0000 7f5d80e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d7c0c5e80 0x7f5d7c0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.174+0000 7f5d80e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d7c0c5e80 0x7f5d7c0c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53340/0 (socket says 172.21.15.27:53340) 2024-03-28T18:55:35.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.174+0000 7f5d80e00640 1 -- 172.21.15.27:0/3418184618 learned_addr learned my addr 172.21.15.27:0/3418184618 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.174+0000 7f5d80e00640 1 -- 172.21.15.27:0/3418184618 >> v2:172.21.15.155:3300/0 conn(0x7f5d7c0abf60 msgr2=0x7f5d7c0aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.174+0000 7f5d81800640 1 --2- 172.21.15.27:0/3418184618 >> v2:172.21.15.155:3300/0 conn(0x7f5d7c0abf60 0x7f5d7c0aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.174+0000 7f5d80e00640 1 --2- 172.21.15.27:0/3418184618 >> v2:172.21.15.155:3300/0 conn(0x7f5d7c0abf60 0x7f5d7c0aa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.174+0000 7f5d80e00640 1 -- 172.21.15.27:0/3418184618 >> v1:172.21.15.155:6789/0 conn(0x7f5d7c0c1f50 legacy=0x7f5d7c0ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.174+0000 7f5d80e00640 1 -- 172.21.15.27:0/3418184618 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c06db80 con 0x7f5d7c0c5e80 2024-03-28T18:55:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.175+0000 7f5d80e00640 1 --2- 172.21.15.27:0/3418184618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c5e80 0x7f5d7c0c2cb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06eff0 tx=0x7f5d7002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675afd416fe978f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d7b400640 1 -- 172.21.15.27:0/3418184618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70038070 con 0x7f5d7c0c5e80 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d7b400640 1 -- 172.21.15.27:0/3418184618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7002f050 con 0x7f5d7c0c5e80 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d7b400640 1 -- 172.21.15.27:0/3418184618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70031970 con 0x7f5d7c0c5e80 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d83800640 1 -- 172.21.15.27:0/3418184618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c5e80 msgr2=0x7f5d7c0c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d83800640 1 --2- 172.21.15.27:0/3418184618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c5e80 0x7f5d7c0c2cb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06eff0 tx=0x7f5d7002b7f0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d83800640 1 -- 172.21.15.27:0/3418184618 shutdown_connections 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d83800640 1 --2- 172.21.15.27:0/3418184618 >> v2:172.21.15.155:3300/0 conn(0x7f5d7c0abf60 0x7f5d7c0aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d83800640 1 --2- 172.21.15.27:0/3418184618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c5e80 0x7f5d7c0c2cb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d83800640 1 -- 172.21.15.27:0/3418184618 >> 172.21.15.27:0/3418184618 conn(0x7f5d7c06b9d0 msgr2=0x7f5d7c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d83800640 1 -- 172.21.15.27:0/3418184618 shutdown_connections 2024-03-28T18:55:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.176+0000 7f5d83800640 1 -- 172.21.15.27:0/3418184618 wait complete. 2024-03-28T18:55:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.178+0000 7f5d83800640 1 Processor -- start 2024-03-28T18:55:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.178+0000 7f5d83800640 1 -- start start 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d83800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0abf60 0x7f5d7c14b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d83800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c1f50 0x7f5d7c14bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d83800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c0c5e80 0x7f5d7c1ec930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d83800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0c3370 con 0x7f5d7c0abf60 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d83800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0c31f0 con 0x7f5d7c0c1f50 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d83800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0c6260 con 0x7f5d7c0c5e80 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d80e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0abf60 0x7f5d7c14b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d80e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0abf60 0x7f5d7c14b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53348/0 (socket says 172.21.15.27:53348) 2024-03-28T18:55:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.179+0000 7f5d80e00640 1 -- 172.21.15.27:0/2731420410 learned_addr learned my addr 172.21.15.27:0/2731420410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d7be00640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c1f50 0x7f5d7c14bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d81800640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c0c5e80 0x7f5d7c1ec930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d80e00640 1 -- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c0c5e80 msgr2=0x7f5d7c1ec930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d80e00640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c0c5e80 0x7f5d7c1ec930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d80e00640 1 -- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c1f50 msgr2=0x7f5d7c14bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d80e00640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c1f50 0x7f5d7c14bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d80e00640 1 -- 172.21.15.27:0/2731420410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c0be6c0 con 0x7f5d7c0abf60 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d81800640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c0c5e80 0x7f5d7c1ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d80e00640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0abf60 0x7f5d7c14b9b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d7002b830 tx=0x7f5d700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d78c00640 1 -- 172.21.15.27:0/2731420410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70042070 con 0x7f5d7c0abf60 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d78c00640 1 -- 172.21.15.27:0/2731420410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7003d890 con 0x7f5d7c0abf60 2024-03-28T18:55:35.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c0be9a0 con 0x7f5d7c0abf60 2024-03-28T18:55:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.180+0000 7f5d78c00640 1 -- 172.21.15.27:0/2731420410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70038040 con 0x7f5d7c0abf60 2024-03-28T18:55:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.181+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c14d250 con 0x7f5d7c0abf60 2024-03-28T18:55:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.182+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c04f400 con 0x7f5d7c0abf60 2024-03-28T18:55:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.183+0000 7f5d78c00640 1 -- 172.21.15.27:0/2731420410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5d70045070 con 0x7f5d7c0abf60 2024-03-28T18:55:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.184+0000 7f5d78c00640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d5c06bbd0 0x7f5d5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.184+0000 7f5d78c00640 1 -- 172.21.15.27:0/2731420410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f5d700ac870 con 0x7f5d7c0abf60 2024-03-28T18:55:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.184+0000 7f5d7be00640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d5c06bbd0 0x7f5d5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.184+0000 7f5d7be00640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d5c06bbd0 0x7f5d5c06e090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c0a9380 tx=0x7f5d68009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:35.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.188+0000 7f5d78c00640 1 -- 172.21.15.27:0/2731420410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5d7007b360 con 0x7f5d7c0abf60 2024-03-28T18:55:35.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.390+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d7c0c4b70 con 0x7f5d7c0abf60 2024-03-28T18:55:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.393+0000 7f5d78c00640 1 -- 172.21.15.27:0/2731420410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16562 (secure 0 0 0) 0x7f5d7007ac60 con 0x7f5d7c0abf60 2024-03-28T18:55:35.394 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:35.394 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:33.954956+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d5c06bbd0 msgr2=0x7f5d5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d5c06bbd0 0x7f5d5c06e090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c0a9380 tx=0x7f5d68009040 comp rx=0 tx=0).stop 2024-03-28T18:55:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0abf60 msgr2=0x7f5d7c14b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0abf60 0x7f5d7c14b9b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d7002b830 tx=0x7f5d700046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 shutdown_connections 2024-03-28T18:55:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d5c06bbd0 0x7f5d5c06e090 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c0c5e80 0x7f5d7c1ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c1f50 0x7f5d7c14bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 --2- 172.21.15.27:0/2731420410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0abf60 0x7f5d7c14b9b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 >> 172.21.15.27:0/2731420410 conn(0x7f5d7c06b9d0 msgr2=0x7f5d7c0c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 shutdown_connections 2024-03-28T18:55:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.396+0000 7f5d83800640 1 -- 172.21.15.27:0/2731420410 wait complete. 2024-03-28T18:55:35.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a standby_count_wanted 0 2024-03-28T18:55:35.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.648+0000 7f8ec1400640 1 Processor -- start 2024-03-28T18:55:35.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.648+0000 7f8ec1400640 1 -- start start 2024-03-28T18:55:35.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8ec1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ebc14d4e0 0x7f8ebc1540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8ec1400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8ebc06e960 0x7f8ebc0c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8ec1400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 0x7f8ebc0a9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8ec1400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc0c1ce0 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8ec1400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc06dbe0 con 0x7f8ebc06e960 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8ec1400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc06de80 con 0x7f8ebc0a9400 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8eba800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ebc14d4e0 0x7f8ebc1540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8eba800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ebc14d4e0 0x7f8ebc1540e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53352/0 (socket says 172.21.15.27:53352) 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.650+0000 7f8eba800640 1 -- 172.21.15.27:0/2357763961 learned_addr learned my addr 172.21.15.27:0/2357763961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.651+0000 7f8ebb200640 1 --2- 172.21.15.27:0/2357763961 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 0x7f8ebc0a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.651+0000 7f8eba800640 1 -- 172.21.15.27:0/2357763961 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 msgr2=0x7f8ebc0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.651+0000 7f8eba800640 1 --2- 172.21.15.27:0/2357763961 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 0x7f8ebc0a9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.651+0000 7f8eba800640 1 -- 172.21.15.27:0/2357763961 >> v2:172.21.15.155:3300/0 conn(0x7f8ebc06e960 msgr2=0x7f8ebc0c17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:35.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.651+0000 7f8eb1e00640 1 --2- 172.21.15.27:0/2357763961 >> v2:172.21.15.155:3300/0 conn(0x7f8ebc06e960 0x7f8ebc0c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.651+0000 7f8eba800640 1 --2- 172.21.15.27:0/2357763961 >> v2:172.21.15.155:3300/0 conn(0x7f8ebc06e960 0x7f8ebc0c17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.651+0000 7f8eba800640 1 -- 172.21.15.27:0/2357763961 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc06e160 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.652+0000 7f8eba800640 1 --2- 172.21.15.27:0/2357763961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc1540e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0aa3f0 tx=0x7f8eac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47191f96686e7488 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:35.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.652+0000 7f8eb9e00640 1 -- 172.21.15.27:0/2357763961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac035020 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.652+0000 7f8eb9e00640 1 -- 172.21.15.27:0/2357763961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eac031870 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8eb9e00640 1 -- 172.21.15.27:0/2357763961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac031c40 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 -- 172.21.15.27:0/2357763961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 msgr2=0x7f8ebc1540e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:35.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 --2- 172.21.15.27:0/2357763961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc1540e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0aa3f0 tx=0x7f8eac02b910 comp rx=0 tx=0).stop 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 -- 172.21.15.27:0/2357763961 shutdown_connections 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 --2- 172.21.15.27:0/2357763961 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 0x7f8ebc0a9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 --2- 172.21.15.27:0/2357763961 >> v2:172.21.15.155:3300/0 conn(0x7f8ebc06e960 0x7f8ebc0c17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 --2- 172.21.15.27:0/2357763961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc1540e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 -- 172.21.15.27:0/2357763961 >> 172.21.15.27:0/2357763961 conn(0x7f8ebc06ca00 msgr2=0x7f8ebc06cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 -- 172.21.15.27:0/2357763961 shutdown_connections 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.653+0000 7f8ec1400640 1 -- 172.21.15.27:0/2357763961 wait complete. 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.655+0000 7f8ec1400640 1 Processor -- start 2024-03-28T18:55:35.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.655+0000 7f8ec1400640 1 -- start start 2024-03-28T18:55:35.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.656+0000 7f8ec1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ebc06e960 0x7f8ebc1ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.656+0000 7f8ec1400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 0x7f8ebc1eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.656+0000 7f8ec1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc0c2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.656+0000 7f8ec1400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc14d8c0 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.656+0000 7f8ec1400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0c1f10 con 0x7f8ebc06e960 2024-03-28T18:55:35.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.656+0000 7f8ec1400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc14da40 con 0x7f8ebc0a9400 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc0c2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc0c2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53364/0 (socket says 172.21.15.27:53364) 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 -- 172.21.15.27:0/3983989202 learned_addr learned my addr 172.21.15.27:0/3983989202 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8eba800640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ebc06e960 0x7f8ebc1ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8eb1e00640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 0x7f8ebc1eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 -- 172.21.15.27:0/3983989202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 msgr2=0x7f8ebc1eef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 0x7f8ebc1eef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 -- 172.21.15.27:0/3983989202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ebc06e960 msgr2=0x7f8ebc1ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ebc06e960 0x7f8ebc1ee9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 -- 172.21.15.27:0/3983989202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc14eb30 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.657+0000 7f8ebb200640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc0c2e60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4007e40 tx=0x7f8eb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:35.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.658+0000 7f8eb3400640 1 -- 172.21.15.27:0/3983989202 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4014620 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.658+0000 7f8eb3400640 1 -- 172.21.15.27:0/3983989202 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb4019070 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.658+0000 7f8eb3400640 1 -- 172.21.15.27:0/3983989202 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4014a20 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.658+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc1549d0 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.658+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc154c90 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.659+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc04f470 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.660+0000 7f8eb3400640 1 -- 172.21.15.27:0/3983989202 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8eb4014bc0 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.661+0000 7f8eb3400640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e9806bbd0 0x7f8e9806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:35.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.661+0000 7f8eb3400640 1 -- 172.21.15.27:0/3983989202 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f8eb408b3b0 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.661+0000 7f8eba800640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e9806bbd0 0x7f8e9806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:35.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.662+0000 7f8eba800640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e9806bbd0 0x7f8e9806e090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8eac036000 tx=0x7f8eac03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:35.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.665+0000 7f8eb3400640 1 -- 172.21.15.27:0/3983989202 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8eb4059ea0 con 0x7f8ebc14d4e0 2024-03-28T18:55:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:35.912+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8ebc06ed90 con 0x7f8ebc14d4e0 2024-03-28T18:55:36.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.486+0000 7f8eb3400640 1 -- 172.21.15.27:0/3983989202 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 113+0+0 (secure 0 0 0) 0x7f8eb40597a0 con 0x7f8ebc14d4e0 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e9806bbd0 msgr2=0x7f8e9806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e9806bbd0 0x7f8e9806e090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8eac036000 tx=0x7f8eac03c040 comp rx=0 tx=0).stop 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 msgr2=0x7f8ebc0c2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc0c2e60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4007e40 tx=0x7f8eb400e4b0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 shutdown_connections 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e9806bbd0 0x7f8e9806e090 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ebc14d4e0 0x7f8ebc0c2e60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ebc0a9400 0x7f8ebc1eef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 --2- 172.21.15.27:0/3983989202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ebc06e960 0x7f8ebc1ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 >> 172.21.15.27:0/3983989202 conn(0x7f8ebc06ca00 msgr2=0x7f8ebc14c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:36.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 shutdown_connections 2024-03-28T18:55:36.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.490+0000 7f8ec1400640 1 -- 172.21.15.27:0/3983989202 wait complete. 2024-03-28T18:55:36.513 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-03-28T18:55:36.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false 2024-03-28T18:55:36.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.739+0000 7f0013e00640 1 Processor -- start 2024-03-28T18:55:36.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.739+0000 7f0013e00640 1 -- start start 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.740+0000 7f0013e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0014150990 0x7f00141573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.740+0000 7f0013e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f00140ad4e0 0x7f00140c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.740+0000 7f0013e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f00140c51d0 0x7f0014153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.740+0000 7f0013e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00140adaa0 con 0x7f0014150990 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.740+0000 7f0013e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0014071870 con 0x7f00140ad4e0 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.740+0000 7f0013e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0014071b50 con 0x7f00140c51d0 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0014150990 0x7f00141573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0014150990 0x7f00141573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53118/0 (socket says 172.21.15.27:53118) 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012a00640 1 -- 172.21.15.27:0/2204690073 learned_addr learned my addr 172.21.15.27:0/2204690073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0013400640 1 --2- 172.21.15.27:0/2204690073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f00140c51d0 0x7f0014153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:36.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012000640 1 --2- 172.21.15.27:0/2204690073 >> v2:172.21.15.155:3300/0 conn(0x7f00140ad4e0 0x7f00140c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:36.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012a00640 1 -- 172.21.15.27:0/2204690073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f00140c51d0 msgr2=0x7f0014153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:36.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012a00640 1 --2- 172.21.15.27:0/2204690073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f00140c51d0 0x7f0014153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012a00640 1 -- 172.21.15.27:0/2204690073 >> v2:172.21.15.155:3300/0 conn(0x7f00140ad4e0 msgr2=0x7f00140c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:36.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012a00640 1 --2- 172.21.15.27:0/2204690073 >> v2:172.21.15.155:3300/0 conn(0x7f00140ad4e0 0x7f00140c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.741+0000 7f0012a00640 1 -- 172.21.15.27:0/2204690073 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014071e30 con 0x7f0014150990 2024-03-28T18:55:36.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.742+0000 7f0012a00640 1 --2- 172.21.15.27:0/2204690073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014150990 0x7f00141573f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0014067e70 tx=0x7f000402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb29d2bf00f17562 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:36.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.743+0000 7f0011600640 1 -- 172.21.15.27:0/2204690073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00040317d0 con 0x7f0014150990 2024-03-28T18:55:36.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.743+0000 7f0011600640 1 -- 172.21.15.27:0/2204690073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0004037070 con 0x7f0014150990 2024-03-28T18:55:36.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.743+0000 7f0011600640 1 -- 172.21.15.27:0/2204690073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004031b90 con 0x7f0014150990 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.743+0000 7f0013e00640 1 -- 172.21.15.27:0/2204690073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014150990 msgr2=0x7f00141573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.743+0000 7f0013e00640 1 --2- 172.21.15.27:0/2204690073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014150990 0x7f00141573f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0014067e70 tx=0x7f000402b870 comp rx=0 tx=0).stop 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.744+0000 7f0013e00640 1 -- 172.21.15.27:0/2204690073 shutdown_connections 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.744+0000 7f0013e00640 1 --2- 172.21.15.27:0/2204690073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f00140c51d0 0x7f0014153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.744+0000 7f0013e00640 1 --2- 172.21.15.27:0/2204690073 >> v2:172.21.15.155:3300/0 conn(0x7f00140ad4e0 0x7f00140c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.744+0000 7f0013e00640 1 --2- 172.21.15.27:0/2204690073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0014150990 0x7f00141573f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.744+0000 7f0013e00640 1 -- 172.21.15.27:0/2204690073 >> 172.21.15.27:0/2204690073 conn(0x7f00140c8130 msgr2=0x7f00140c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.744+0000 7f0013e00640 1 -- 172.21.15.27:0/2204690073 shutdown_connections 2024-03-28T18:55:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.744+0000 7f0013e00640 1 -- 172.21.15.27:0/2204690073 wait complete. 2024-03-28T18:55:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.745+0000 7f0013e00640 1 Processor -- start 2024-03-28T18:55:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.746+0000 7f0013e00640 1 -- start start 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.746+0000 7f0013e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00140ad4e0 0x7f00140c3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.746+0000 7f0013e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140c51d0 0x7f00140bf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0013e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014150990 0x7f00140bf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0013e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0014150d70 con 0x7f00140c51d0 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0013e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0014072110 con 0x7f00140ad4e0 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0013e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0014150ef0 con 0x7f0014150990 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140c51d0 0x7f00140bf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140c51d0 0x7f00140bf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53134/0 (socket says 172.21.15.27:53134) 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012000640 1 -- 172.21.15.27:0/2027099967 learned_addr learned my addr 172.21.15.27:0/2027099967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0013400640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014150990 0x7f00140bf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012a00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00140ad4e0 0x7f00140c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012000640 1 -- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014150990 msgr2=0x7f00140bf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012000640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014150990 0x7f00140bf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012000640 1 -- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00140ad4e0 msgr2=0x7f00140c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012000640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00140ad4e0 0x7f00140c3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0012000640 1 -- 172.21.15.27:0/2027099967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014155110 con 0x7f00140c51d0 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.747+0000 7f0013400640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014150990 0x7f00140bf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.748+0000 7f0012000640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140c51d0 0x7f00140bf440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efffc007e60 tx=0x7efffc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.748+0000 7f000b400640 1 -- 172.21.15.27:0/2027099967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc014620 con 0x7f00140c51d0 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.748+0000 7f000b400640 1 -- 172.21.15.27:0/2027099967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc019070 con 0x7f00140c51d0 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.748+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00140bebd0 con 0x7f00140c51d0 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.748+0000 7f000b400640 1 -- 172.21.15.27:0/2027099967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc014a00 con 0x7f00140c51d0 2024-03-28T18:55:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.748+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0014156a80 con 0x7f00140c51d0 2024-03-28T18:55:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.749+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001404f400 con 0x7f00140c51d0 2024-03-28T18:55:36.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.750+0000 7f000b400640 1 -- 172.21.15.27:0/2027099967 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efffc014ba0 con 0x7f00140c51d0 2024-03-28T18:55:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.751+0000 7f000b400640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff006bbd0 0x7efff006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.751+0000 7f000b400640 1 -- 172.21.15.27:0/2027099967 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7efffc08b2f0 con 0x7f00140c51d0 2024-03-28T18:55:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.751+0000 7f0012a00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff006bbd0 0x7efff006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:36.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.752+0000 7f0012a00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff006bbd0 0x7efff006e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0004036000 tx=0x7f0004008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:36.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:36.756+0000 7f000b400640 1 -- 172.21.15.27:0/2027099967 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efffc059de0 con 0x7f00140c51d0 2024-03-28T18:55:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.002+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f00140ad040 con 0x7f00140c51d0 2024-03-28T18:55:37.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.499+0000 7f000b400640 1 -- 172.21.15.27:0/2027099967 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 117+0+0 (secure 0 0 0) 0x7efffc0596e0 con 0x7f00140c51d0 2024-03-28T18:55:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff006bbd0 msgr2=0x7efff006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff006bbd0 0x7efff006e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0004036000 tx=0x7f0004008030 comp rx=0 tx=0).stop 2024-03-28T18:55:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140c51d0 msgr2=0x7f00140bf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140c51d0 0x7f00140bf440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efffc007e60 tx=0x7efffc00e4b0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 shutdown_connections 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff006bbd0 0x7efff006e090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014150990 0x7f00140bf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00140c51d0 0x7f00140bf440 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 --2- 172.21.15.27:0/2027099967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00140ad4e0 0x7f00140c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 >> 172.21.15.27:0/2027099967 conn(0x7f00140c8130 msgr2=0x7f00140c6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.503+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 shutdown_connections 2024-03-28T18:55:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.504+0000 7f0013e00640 1 -- 172.21.15.27:0/2027099967 wait complete. 2024-03-28T18:55:37.526 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a session_timeout 300 2024-03-28T18:55:37.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.759+0000 7f72ebe00640 1 Processor -- start 2024-03-28T18:55:37.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.759+0000 7f72ebe00640 1 -- start start 2024-03-28T18:55:37.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72ebe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f72ec149c20 0x7f72ec153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72ebe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f72ec156840 0x7f72ec0a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72ebe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec0c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72ebe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec154740 con 0x7f72ec149c20 2024-03-28T18:55:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72ebe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec156db0 con 0x7f72ec156840 2024-03-28T18:55:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72ebe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec157090 con 0x7f72ec0a6210 2024-03-28T18:55:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72eaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f72ec149c20 0x7f72ec153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:37.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72e2000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f72ec156840 0x7f72ec0a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72e2000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f72ec156840 0x7f72ec0a5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:42908/0 (socket says 172.21.15.27:42908) 2024-03-28T18:55:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72e2000640 1 -- 172.21.15.27:0/1505322923 learned_addr learned my addr 172.21.15.27:0/1505322923 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:37.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72eb400640 1 --2- 172.21.15.27:0/1505322923 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec0c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72eaa00640 1 -- 172.21.15.27:0/1505322923 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 msgr2=0x7f72ec0c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72eaa00640 1 --2- 172.21.15.27:0/1505322923 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec0c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72eaa00640 1 -- 172.21.15.27:0/1505322923 >> v2:172.21.15.155:3300/0 conn(0x7f72ec156840 msgr2=0x7f72ec0a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:37.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72eaa00640 1 --2- 172.21.15.27:0/1505322923 >> v2:172.21.15.155:3300/0 conn(0x7f72ec156840 0x7f72ec0a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72eaa00640 1 -- 172.21.15.27:0/1505322923 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec157370 con 0x7f72ec149c20 2024-03-28T18:55:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.761+0000 7f72e2000640 1 --2- 172.21.15.27:0/1505322923 >> v2:172.21.15.155:3300/0 conn(0x7f72ec156840 0x7f72ec0a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.763+0000 7f72eaa00640 1 --2- 172.21.15.27:0/1505322923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec149c20 0x7f72ec153e80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72ec1543c0 tx=0x7f72dc02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e6d69bca8d4b31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.763+0000 7f72ea000640 1 -- 172.21.15.27:0/1505322923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc039070 con 0x7f72ec149c20 2024-03-28T18:55:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ea000640 1 -- 172.21.15.27:0/1505322923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc02cb40 con 0x7f72ec149c20 2024-03-28T18:55:37.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ea000640 1 -- 172.21.15.27:0/1505322923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc034440 con 0x7f72ec149c20 2024-03-28T18:55:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ebe00640 1 -- 172.21.15.27:0/1505322923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec149c20 msgr2=0x7f72ec153e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:37.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ebe00640 1 --2- 172.21.15.27:0/1505322923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec149c20 0x7f72ec153e80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f72ec1543c0 tx=0x7f72dc02bd90 comp rx=0 tx=0).stop 2024-03-28T18:55:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ebe00640 1 -- 172.21.15.27:0/1505322923 shutdown_connections 2024-03-28T18:55:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ebe00640 1 --2- 172.21.15.27:0/1505322923 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec0c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ebe00640 1 --2- 172.21.15.27:0/1505322923 >> v2:172.21.15.155:3300/0 conn(0x7f72ec156840 0x7f72ec0a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ebe00640 1 --2- 172.21.15.27:0/1505322923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec149c20 0x7f72ec153e80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.764+0000 7f72ebe00640 1 -- 172.21.15.27:0/1505322923 >> 172.21.15.27:0/1505322923 conn(0x7f72ec0c7220 msgr2=0x7f72ec06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.765+0000 7f72ebe00640 1 -- 172.21.15.27:0/1505322923 shutdown_connections 2024-03-28T18:55:37.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.765+0000 7f72ebe00640 1 -- 172.21.15.27:0/1505322923 wait complete. 2024-03-28T18:55:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.766+0000 7f72ebe00640 1 Processor -- start 2024-03-28T18:55:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.766+0000 7f72ebe00640 1 -- start start 2024-03-28T18:55:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.767+0000 7f72ebe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec06de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.767+0000 7f72ebe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72ec149c20 0x7f72ec1e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.767+0000 7f72ebe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec156840 0x7f72ec1e7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.767+0000 7f72ebe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec14a000 con 0x7f72ec156840 2024-03-28T18:55:37.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.767+0000 7f72ebe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0c5f20 con 0x7f72ec149c20 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.767+0000 7f72ebe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec14a180 con 0x7f72ec0a6210 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec156840 0x7f72ec1e7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eaa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec06de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec156840 0x7f72ec1e7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53144/0 (socket says 172.21.15.27:53144) 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eaa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec06de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:34694/0 (socket says 172.21.15.27:34694) 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eb400640 1 -- 172.21.15.27:0/2521032941 learned_addr learned my addr 172.21.15.27:0/2521032941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72e2000640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72ec149c20 0x7f72ec1e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eb400640 1 -- 172.21.15.27:0/2521032941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 msgr2=0x7f72ec06de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eb400640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec06de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eb400640 1 -- 172.21.15.27:0/2521032941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72ec149c20 msgr2=0x7f72ec1e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:37.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eb400640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72ec149c20 0x7f72ec1e6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.768+0000 7f72eb400640 1 -- 172.21.15.27:0/2521032941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec1ea2e0 con 0x7f72ec156840 2024-03-28T18:55:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.769+0000 7f72eb400640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec156840 0x7f72ec1e7110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e7f0 tx=0x7f72e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.769+0000 7f72e3400640 1 -- 172.21.15.27:0/2521032941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4017070 con 0x7f72ec156840 2024-03-28T18:55:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.769+0000 7f72e3400640 1 -- 172.21.15.27:0/2521032941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e400c730 con 0x7f72ec156840 2024-03-28T18:55:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.769+0000 7f72e3400640 1 -- 172.21.15.27:0/2521032941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e400cb00 con 0x7f72ec156840 2024-03-28T18:55:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.769+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec1ea5c0 con 0x7f72ec156840 2024-03-28T18:55:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.769+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec06e3b0 con 0x7f72ec156840 2024-03-28T18:55:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.770+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec04f400 con 0x7f72ec156840 2024-03-28T18:55:37.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.772+0000 7f72e3400640 1 -- 172.21.15.27:0/2521032941 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f72e401a070 con 0x7f72ec156840 2024-03-28T18:55:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.773+0000 7f72e3400640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72c806bbd0 0x7f72c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.773+0000 7f72e3400640 1 -- 172.21.15.27:0/2521032941 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f72e408ae20 con 0x7f72ec156840 2024-03-28T18:55:37.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.773+0000 7f72eaa00640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72c806bbd0 0x7f72c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:37.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.774+0000 7f72eaa00640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72c806bbd0 0x7f72c806e090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f72dc036000 tx=0x7f72dc030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:37.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:37.778+0000 7f72e3400640 1 -- 172.21.15.27:0/2521032941 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f72e4059360 con 0x7f72ec156840 2024-03-28T18:55:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.020+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f72ec0c5ae0 con 0x7f72ec156840 2024-03-28T18:55:38.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.503+0000 7f72e3400640 1 -- 172.21.15.27:0/2521032941 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"}]=0 v9) v1 ==== 110+0+0 (secure 0 0 0) 0x7f72e405d400 con 0x7f72ec156840 2024-03-28T18:55:38.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.506+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72c806bbd0 msgr2=0x7f72c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:38.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.506+0000 7f72ebe00640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72c806bbd0 0x7f72c806e090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f72dc036000 tx=0x7f72dc030000 comp rx=0 tx=0).stop 2024-03-28T18:55:38.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec156840 msgr2=0x7f72ec1e7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:38.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec156840 0x7f72ec1e7110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e7f0 tx=0x7f72e400ecb0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 shutdown_connections 2024-03-28T18:55:38.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72c806bbd0 0x7f72c806e090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72ec156840 0x7f72ec1e7110 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72ec149c20 0x7f72ec1e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 --2- 172.21.15.27:0/2521032941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72ec0a6210 0x7f72ec06de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 >> 172.21.15.27:0/2521032941 conn(0x7f72ec0c7220 msgr2=0x7f72ec151410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:38.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 shutdown_connections 2024-03-28T18:55:38.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.507+0000 7f72ebe00640 1 -- 172.21.15.27:0/2521032941 wait complete. 2024-03-28T18:55:38.530 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'a' 2024-03-28T18:55:38.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:55:38.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.758+0000 7f4a62c00640 1 Processor -- start 2024-03-28T18:55:38.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.759+0000 7f4a62c00640 1 -- start start 2024-03-28T18:55:38.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a62c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c152d10 0x7f4a5c0c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:38.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a62c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a5c1562f0 0x7f4a5c15ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:38.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a62c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c1530f0 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a62c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5c06d3d0 con 0x7f4a5c1562f0 2024-03-28T18:55:38.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a62c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c0c92f0 con 0x7f4a5c152d10 2024-03-28T18:55:38.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a60c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a5c1562f0 0x7f4a5c15ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:38.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a60c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a5c1562f0 0x7f4a5c15ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53158/0 (socket says 172.21.15.27:53158) 2024-03-28T18:55:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a60c00640 1 -- 172.21.15.27:0/2552868902 learned_addr learned my addr 172.21.15.27:0/2552868902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.760+0000 7f4a5be00640 1 --2- 172.21.15.27:0/2552868902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c152d10 0x7f4a5c0c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.761+0000 7f4a60c00640 1 -- 172.21.15.27:0/2552868902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c152d10 msgr2=0x7f4a5c0c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.761+0000 7f4a60c00640 1 --2- 172.21.15.27:0/2552868902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c152d10 0x7f4a5c0c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.761+0000 7f4a60c00640 1 -- 172.21.15.27:0/2552868902 >> v1:172.21.15.155:6789/0 conn(0x7f4a5c0c8e10 legacy=0x7f4a5c158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:38.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.761+0000 7f4a60c00640 1 -- 172.21.15.27:0/2552868902 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c155810 con 0x7f4a5c1562f0 2024-03-28T18:55:38.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.762+0000 7f4a60c00640 1 --2- 172.21.15.27:0/2552868902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c1562f0 0x7f4a5c15ff50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4a54009e50 tx=0x7f4a5402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d21ac5086fe98c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:38.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.762+0000 7f4a5aa00640 1 -- 172.21.15.27:0/2552868902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5402c9e0 con 0x7f4a5c1562f0 2024-03-28T18:55:38.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.762+0000 7f4a5aa00640 1 -- 172.21.15.27:0/2552868902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5402cb80 con 0x7f4a5c1562f0 2024-03-28T18:55:38.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.762+0000 7f4a5aa00640 1 -- 172.21.15.27:0/2552868902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a540343f0 con 0x7f4a5c1562f0 2024-03-28T18:55:38.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.763+0000 7f4a62c00640 1 -- 172.21.15.27:0/2552868902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c1562f0 msgr2=0x7f4a5c15ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:38.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.763+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2552868902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c1562f0 0x7f4a5c15ff50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4a54009e50 tx=0x7f4a5402be40 comp rx=0 tx=0).stop 2024-03-28T18:55:38.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.763+0000 7f4a62c00640 1 -- 172.21.15.27:0/2552868902 shutdown_connections 2024-03-28T18:55:38.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.763+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2552868902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c1562f0 0x7f4a5c15ff50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.763+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2552868902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c152d10 0x7f4a5c0c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.763+0000 7f4a62c00640 1 -- 172.21.15.27:0/2552868902 >> 172.21.15.27:0/2552868902 conn(0x7f4a5c066fb0 msgr2=0x7f4a5c0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:38.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.763+0000 7f4a62c00640 1 -- 172.21.15.27:0/2552868902 shutdown_connections 2024-03-28T18:55:38.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.763+0000 7f4a62c00640 1 -- 172.21.15.27:0/2552868902 wait complete. 2024-03-28T18:55:38.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.765+0000 7f4a62c00640 1 Processor -- start 2024-03-28T18:55:38.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.765+0000 7f4a62c00640 1 -- start start 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a62c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c0c8e10 0x7f4a5c0c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a62c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a5c152d10 0x7f4a5c1eb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a62c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c1562f0 0x7f4a5c157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a62c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c1592f0 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a62c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c06d850 con 0x7f4a5c152d10 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a62c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c14d9c0 con 0x7f4a5c1562f0 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a5be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c0c8e10 0x7f4a5c0c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a5be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c0c8e10 0x7f4a5c0c1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53174/0 (socket says 172.21.15.27:53174) 2024-03-28T18:55:38.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a5be00640 1 -- 172.21.15.27:0/2149139176 learned_addr learned my addr 172.21.15.27:0/2149139176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:38.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a5b400640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a5c152d10 0x7f4a5c1eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:38.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a60c00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c1562f0 0x7f4a5c157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:38.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a5be00640 1 -- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c1562f0 msgr2=0x7f4a5c157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:38.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a5be00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c1562f0 0x7f4a5c157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a5be00640 1 -- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a5c152d10 msgr2=0x7f4a5c1eb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:38.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.766+0000 7f4a5be00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a5c152d10 0x7f4a5c1eb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:38.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a5be00640 1 -- 172.21.15.27:0/2149139176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c1eafe0 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a60c00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c1562f0 0x7f4a5c157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:38.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a5b400640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a5c152d10 0x7f4a5c1eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:38.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a5be00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c0c8e10 0x7f4a5c0c1fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00d830 tx=0x7f4a4c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:38.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a53e00640 1 -- 172.21.15.27:0/2149139176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c00b6c0 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a53e00640 1 -- 172.21.15.27:0/2149139176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4c019070 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a53e00640 1 -- 172.21.15.27:0/2149139176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c00ba80 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c14dc10 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.767+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c1586e0 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.768+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c14e130 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.770+0000 7f4a53e00640 1 -- 172.21.15.27:0/2149139176 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a4c01e460 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.770+0000 7f4a53e00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3806bbd0 0x7f4a3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:38.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.770+0000 7f4a53e00640 1 -- 172.21.15.27:0/2149139176 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f4a4c08b150 con 0x7f4a5c0c8e10 2024-03-28T18:55:38.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.771+0000 7f4a5b400640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3806bbd0 0x7f4a3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:38.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.771+0000 7f4a5b400640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3806bbd0 0x7f4a3806e090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c068a90 tx=0x7f4a44009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:38.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:38.775+0000 7f4a53e00640 1 -- 172.21.15.27:0/2149139176 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a5c1eafe0 con 0x7f4a5c0c8e10 2024-03-28T18:55:39.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.021+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a5c04f400 con 0x7f4a5c0c8e10 2024-03-28T18:55:39.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.022+0000 7f4a53e00640 1 -- 172.21.15.27:0/2149139176 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4973 (secure 0 0 0) 0x7f4a4c059c40 con 0x7f4a5c0c8e10 2024-03-28T18:55:39.023 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:39.023 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1}]} 2024-03-28T18:55:39.023 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 9 2024-03-28T18:55:39.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3806bbd0 msgr2=0x7f4a3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3806bbd0 0x7f4a3806e090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c068a90 tx=0x7f4a44009040 comp rx=0 tx=0).stop 2024-03-28T18:55:39.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c0c8e10 msgr2=0x7f4a5c0c1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c0c8e10 0x7f4a5c0c1fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00d830 tx=0x7f4a4c00dcf0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 shutdown_connections 2024-03-28T18:55:39.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3806bbd0 0x7f4a3806e090 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5c1562f0 0x7f4a5c157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a5c152d10 0x7f4a5c1eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 --2- 172.21.15.27:0/2149139176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a5c0c8e10 0x7f4a5c0c1fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.025+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 >> 172.21.15.27:0/2149139176 conn(0x7f4a5c066fb0 msgr2=0x7f4a5c1516d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:39.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.026+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 shutdown_connections 2024-03-28T18:55:39.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.026+0000 7f4a62c00640 1 -- 172.21.15.27:0/2149139176 wait complete. 2024-03-28T18:55:39.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:39.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.275+0000 7f9b0b200640 1 Processor -- start 2024-03-28T18:55:39.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.275+0000 7f9b0b200640 1 -- start start 2024-03-28T18:55:39.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.276+0000 7f9b0b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b0c0c58c0 0x7f9b0c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.276+0000 7f9b0b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c158bd0 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.276+0000 7f9b0b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c158eb0 con 0x7f9b0c0be5d0 2024-03-28T18:55:39.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.276+0000 7f9b0b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c159190 con 0x7f9b0c0c58c0 2024-03-28T18:55:39.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.277+0000 7f9b09400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b0c0c8ed0 0x7f9b0c1555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34008/0 (socket says 172.21.15.27:34008) 2024-03-28T18:55:39.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.277+0000 7f9b09400640 1 -- 172.21.15.27:0/1258561091 learned_addr learned my addr 172.21.15.27:0/1258561091 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:39.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.277+0000 7f9b09e00640 1 --2- 172.21.15.27:0/1258561091 >> v2:172.21.15.27:3300/0 conn(0x7f9b0c0c58c0 0x7f9b0c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.277+0000 7f9b08a00640 1 -- 172.21.15.27:0/1258561091 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089035738 0 0) 0x7f9b0c158bd0 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.277+0000 7f9b08a00640 1 -- 172.21.15.27:0/1258561091 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af0003520 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.278+0000 7f9b09e00640 1 -- 172.21.15.27:0/1258561091 >> v1:172.21.15.155:6789/0 conn(0x7f9b0c0be5d0 legacy=0x7f9b0c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.278+0000 7f9b09e00640 1 -- 172.21.15.27:0/1258561091 >> v1:172.21.15.27:6789/0 conn(0x7f9b0c0c8ed0 legacy=0x7f9b0c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.278+0000 7f9b09e00640 1 -- 172.21.15.27:0/1258561091 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c159470 con 0x7f9b0c0c58c0 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.279+0000 7f9b09e00640 1 --2- 172.21.15.27:0/1258561091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c58c0 0x7f9b0c0c5ca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0097a0 tx=0x7f9afc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cda1236ee99122 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.279+0000 7f9b08a00640 1 -- 172.21.15.27:0/1258561091 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc0315a0 con 0x7f9b0c0c58c0 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.279+0000 7f9b08a00640 1 -- 172.21.15.27:0/1258561091 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc037070 con 0x7f9b0c0c58c0 2024-03-28T18:55:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.279+0000 7f9b08a00640 1 -- 172.21.15.27:0/1258561091 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc031960 con 0x7f9b0c0c58c0 2024-03-28T18:55:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.280+0000 7f9b0b200640 1 -- 172.21.15.27:0/1258561091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c58c0 msgr2=0x7f9b0c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.280+0000 7f9b0b200640 1 --2- 172.21.15.27:0/1258561091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c58c0 0x7f9b0c0c5ca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0097a0 tx=0x7f9afc02b840 comp rx=0 tx=0).stop 2024-03-28T18:55:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.280+0000 7f9b0b200640 1 -- 172.21.15.27:0/1258561091 shutdown_connections 2024-03-28T18:55:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.280+0000 7f9b0b200640 1 --2- 172.21.15.27:0/1258561091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c58c0 0x7f9b0c0c5ca0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.280+0000 7f9b0b200640 1 -- 172.21.15.27:0/1258561091 >> 172.21.15.27:0/1258561091 conn(0x7f9b0c06b6f0 msgr2=0x7f9b0c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.280+0000 7f9b0b200640 1 -- 172.21.15.27:0/1258561091 shutdown_connections 2024-03-28T18:55:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.280+0000 7f9b0b200640 1 -- 172.21.15.27:0/1258561091 wait complete. 2024-03-28T18:55:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.281+0000 7f9b0b200640 1 Processor -- start 2024-03-28T18:55:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.282+0000 7f9b0b200640 1 -- start start 2024-03-28T18:55:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.282+0000 7f9b0b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b0c0be5d0 0x7f9b0c1ec0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.282+0000 7f9b0b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b0c0c58c0 0x7f9b0c0abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c8ed0 0x7f9b0c1e09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0b09a0 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0b0820 con 0x7f9b0c0be5d0 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0ad7d0 con 0x7f9b0c0c58c0 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c8ed0 0x7f9b0c1e09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c8ed0 0x7f9b0c1e09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53190/0 (socket says 172.21.15.27:53190) 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0a800640 1 -- 172.21.15.27:0/3382302104 learned_addr learned my addr 172.21.15.27:0/3382302104 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b09e00640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b0c0be5d0 0x7f9b0c1ec0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b09400640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b0c0c58c0 0x7f9b0c0abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0a800640 1 -- 172.21.15.27:0/3382302104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b0c0c58c0 msgr2=0x7f9b0c0abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0a800640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b0c0c58c0 0x7f9b0c0abe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0a800640 1 -- 172.21.15.27:0/3382302104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b0c0be5d0 msgr2=0x7f9b0c1ec0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0a800640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b0c0be5d0 0x7f9b0c1ec0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.283+0000 7f9b0a800640 1 -- 172.21.15.27:0/3382302104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c0c29e0 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.284+0000 7f9b0a800640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c8ed0 0x7f9b0c1e09b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400c7a0 tx=0x7f9b0400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.284+0000 7f9b02000640 1 -- 172.21.15.27:0/3382302104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b04003b70 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.284+0000 7f9b02000640 1 -- 172.21.15.27:0/3382302104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b04003d30 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.284+0000 7f9b02000640 1 -- 172.21.15.27:0/3382302104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0400aa90 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.284+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c0c2c80 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.284+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c1ebbb0 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.285+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0c069640 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.286+0000 7f9b02000640 1 -- 172.21.15.27:0/3382302104 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9b0400ac30 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.287+0000 7f9b02000640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9aec06bbd0 0x7f9aec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.287+0000 7f9b02000640 1 -- 172.21.15.27:0/3382302104 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f9b0408a930 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.287+0000 7f9b09e00640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9aec06bbd0 0x7f9aec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.288+0000 7f9b09e00640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9aec06bbd0 0x7f9aec06e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9afc036000 tx=0x7f9afc03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.292+0000 7f9b02000640 1 -- 172.21.15.27:0/3382302104 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9b040594a0 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.494+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b0c156580 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.496+0000 7f9b02000640 1 -- 172.21.15.27:0/3382302104 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16549 (secure 0 0 0) 0x7f9b04058da0 con 0x7f9b0c0c8ed0 2024-03-28T18:55:39.497 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:39.497 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:35.475849+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.499+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9aec06bbd0 msgr2=0x7f9aec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.499+0000 7f9b0b200640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9aec06bbd0 0x7f9aec06e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9afc036000 tx=0x7f9afc03d040 comp rx=0 tx=0).stop 2024-03-28T18:55:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.499+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c8ed0 msgr2=0x7f9b0c1e09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.499+0000 7f9b0b200640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c8ed0 0x7f9b0c1e09b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400c7a0 tx=0x7f9b0400cc60 comp rx=0 tx=0).stop 2024-03-28T18:55:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.500+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 shutdown_connections 2024-03-28T18:55:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.500+0000 7f9b0b200640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9aec06bbd0 0x7f9aec06e090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.500+0000 7f9b0b200640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0c8ed0 0x7f9b0c1e09b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.500+0000 7f9b0b200640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b0c0c58c0 0x7f9b0c0abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.500+0000 7f9b0b200640 1 --2- 172.21.15.27:0/3382302104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b0c0be5d0 0x7f9b0c1ec0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.500+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 >> 172.21.15.27:0/3382302104 conn(0x7f9b0c06b6f0 msgr2=0x7f9b0c1569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.500+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 shutdown_connections 2024-03-28T18:55:39.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.500+0000 7f9b0b200640 1 -- 172.21.15.27:0/3382302104 wait complete. 2024-03-28T18:55:39.523 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T18:55:39.523 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:55:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.751+0000 7fb407200640 1 Processor -- start 2024-03-28T18:55:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.751+0000 7fb407200640 1 -- start start 2024-03-28T18:55:39.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.752+0000 7fb407200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb408152a70 0x7fb408159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.752+0000 7fb407200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4080c6cc0 0x7fb4080cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.752+0000 7fb407200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4080c5710 con 0x7fb4080cb620 2024-03-28T18:55:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.752+0000 7fb407200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4081556b0 con 0x7fb4080c6cc0 2024-03-28T18:55:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.752+0000 7fb407200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4080c59f0 con 0x7fb408152a70 2024-03-28T18:55:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb405400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4080c6cc0 0x7fb4080cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb406800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb4080cb620 0x7fb408155b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34020/0 (socket says 172.21.15.27:34020) 2024-03-28T18:55:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb406800640 1 -- 172.21.15.27:0/2214172742 learned_addr learned my addr 172.21.15.27:0/2214172742 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb405e00640 1 --2- 172.21.15.27:0/2214172742 >> v2:172.21.15.155:3300/0 conn(0x7fb408152a70 0x7fb408159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb404a00640 1 -- 172.21.15.27:0/2214172742 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962034628 0 0) 0x7fb4080c5710 con 0x7fb4080cb620 2024-03-28T18:55:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb404a00640 1 -- 172.21.15.27:0/2214172742 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec003520 con 0x7fb4080cb620 2024-03-28T18:55:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb405400640 1 -- 172.21.15.27:0/2214172742 >> v2:172.21.15.155:3300/0 conn(0x7fb408152a70 msgr2=0x7fb408159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb405400640 1 --2- 172.21.15.27:0/2214172742 >> v2:172.21.15.155:3300/0 conn(0x7fb408152a70 0x7fb408159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb405400640 1 -- 172.21.15.27:0/2214172742 >> v1:172.21.15.27:6789/0 conn(0x7fb4080cb620 legacy=0x7fb408155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.753+0000 7fb405400640 1 -- 172.21.15.27:0/2214172742 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080c5cd0 con 0x7fb4080c6cc0 2024-03-28T18:55:39.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.755+0000 7fb405400640 1 --2- 172.21.15.27:0/2214172742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb4080cb0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb4080c8870 tx=0x7fb3f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85417f8a893511c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:39.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.755+0000 7fb404a00640 1 -- 172.21.15.27:0/2214172742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40315a0 con 0x7fb4080c6cc0 2024-03-28T18:55:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.755+0000 7fb404a00640 1 -- 172.21.15.27:0/2214172742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4037070 con 0x7fb4080c6cc0 2024-03-28T18:55:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.755+0000 7fb404a00640 1 -- 172.21.15.27:0/2214172742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4031940 con 0x7fb4080c6cc0 2024-03-28T18:55:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.755+0000 7fb407200640 1 -- 172.21.15.27:0/2214172742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 msgr2=0x7fb4080cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.755+0000 7fb407200640 1 --2- 172.21.15.27:0/2214172742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb4080cb0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb4080c8870 tx=0x7fb3f402b840 comp rx=0 tx=0).stop 2024-03-28T18:55:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.756+0000 7fb407200640 1 -- 172.21.15.27:0/2214172742 shutdown_connections 2024-03-28T18:55:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.756+0000 7fb407200640 1 --2- 172.21.15.27:0/2214172742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb4080cb0e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.756+0000 7fb407200640 1 --2- 172.21.15.27:0/2214172742 >> v2:172.21.15.155:3300/0 conn(0x7fb408152a70 0x7fb408159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.756+0000 7fb407200640 1 -- 172.21.15.27:0/2214172742 >> 172.21.15.27:0/2214172742 conn(0x7fb40806b2a0 msgr2=0x7fb40806b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.756+0000 7fb407200640 1 -- 172.21.15.27:0/2214172742 shutdown_connections 2024-03-28T18:55:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.756+0000 7fb407200640 1 -- 172.21.15.27:0/2214172742 wait complete. 2024-03-28T18:55:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.757+0000 7fb407200640 1 Processor -- start 2024-03-28T18:55:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.758+0000 7fb407200640 1 -- start start 2024-03-28T18:55:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.758+0000 7fb407200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb408154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.758+0000 7fb407200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4080cb620 0x7fb4080c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.758+0000 7fb407200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb408152a70 0x7fb4080aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.758+0000 7fb407200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4080c8f50 con 0x7fb4080c6cc0 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.758+0000 7fb407200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4080c8dd0 con 0x7fb408152a70 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.758+0000 7fb407200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4080c5fb0 con 0x7fb4080cb620 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb408154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb408154f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53218/0 (socket says 172.21.15.27:53218) 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405e00640 1 -- 172.21.15.27:0/2813989560 learned_addr learned my addr 172.21.15.27:0/2813989560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405400640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4080cb620 0x7fb4080c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb406800640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb408152a70 0x7fb4080aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405e00640 1 -- 172.21.15.27:0/2813989560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4080cb620 msgr2=0x7fb4080c23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405e00640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4080cb620 0x7fb4080c23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405e00640 1 -- 172.21.15.27:0/2813989560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb408152a70 msgr2=0x7fb4080aa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405e00640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb408152a70 0x7fb4080aa7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.759+0000 7fb405e00640 1 -- 172.21.15.27:0/2813989560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb408153940 con 0x7fb4080c6cc0 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.760+0000 7fb405e00640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb408154f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009db0 tx=0x7fb3f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.760+0000 7fb3fe000640 1 -- 172.21.15.27:0/2813989560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8018020 con 0x7fb4080c6cc0 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.760+0000 7fb3fe000640 1 -- 172.21.15.27:0/2813989560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f8014480 con 0x7fb4080c6cc0 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.760+0000 7fb3fe000640 1 -- 172.21.15.27:0/2813989560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8014850 con 0x7fb4080c6cc0 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.760+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb408153c20 con 0x7fb4080c6cc0 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.760+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4080c2920 con 0x7fb4080c6cc0 2024-03-28T18:55:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.761+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb408067dd0 con 0x7fb4080c6cc0 2024-03-28T18:55:39.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.762+0000 7fb3fe000640 1 -- 172.21.15.27:0/2813989560 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb3f8005d10 con 0x7fb4080c6cc0 2024-03-28T18:55:39.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.763+0000 7fb3fe000640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb3e806bbd0 0x7fb3e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:39.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.763+0000 7fb3fe000640 1 -- 172.21.15.27:0/2813989560 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7fb3f808b720 con 0x7fb4080c6cc0 2024-03-28T18:55:39.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.763+0000 7fb405400640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb3e806bbd0 0x7fb3e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:39.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.764+0000 7fb405400640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb3e806bbd0 0x7fb3e806e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4036000 tx=0x7fb3f4008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:39.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.768+0000 7fb3fe000640 1 -- 172.21.15.27:0/2813989560 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb3f8022070 con 0x7fb4080c6cc0 2024-03-28T18:55:39.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.966+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb408071bd0 con 0x7fb3e806bbd0 2024-03-28T18:55:39.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.989+0000 7fb3fe000640 1 -- 172.21.15.27:0/2813989560 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251946 (secure 0 0 0) 0x7fb408071bd0 con 0x7fb3e806bbd0 2024-03-28T18:55:39.994 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:39.996 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:55:39.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb3e806bbd0 msgr2=0x7fb3e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb3e806bbd0 0x7fb3e806e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4036000 tx=0x7fb3f4008030 comp rx=0 tx=0).stop 2024-03-28T18:55:39.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 msgr2=0x7fb408154f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:39.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb408154f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009db0 tx=0x7fb3f800c490 comp rx=0 tx=0).stop 2024-03-28T18:55:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 shutdown_connections 2024-03-28T18:55:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb3e806bbd0 0x7fb3e806e090 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb408152a70 0x7fb4080aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4080cb620 0x7fb4080c23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 --2- 172.21.15.27:0/2813989560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4080c6cc0 0x7fb408154f80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 >> 172.21.15.27:0/2813989560 conn(0x7fb40806b2a0 msgr2=0x7fb40814d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 shutdown_connections 2024-03-28T18:55:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:39.995+0000 7fb407200640 1 -- 172.21.15.27:0/2813989560 wait complete. 2024-03-28T18:55:40.014 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-28T18:55:38.893302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219004,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755716,"statfs":{"total":773094113280,"available":772869853184,"internally_reserved":0,"allocated":2891776,"data_stored":1752388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":47,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002120"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.965537+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:55:35.965537+0000","last_peered":"2024-03-28T18:55:35.965537+0000","last_clean":"2024-03-28T18:55:35.965537+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:55:35.965537+0000","last_undegraded":"2024-03-28T18:55:35.965537+0000","last_fullsized":"2024-03-28T18:55:35.965537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490376+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:55:35.490376+0000","last_peered":"2024-03-28T18:55:35.490376+0000","last_clean":"2024-03-28T18:55:35.490376+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:55:35.490376+0000","last_undegraded":"2024-03-28T18:55:35.490376+0000","last_fullsized":"2024-03-28T18:55:35.490376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565861+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:55:36.565861+0000","last_peered":"2024-03-28T18:55:36.565861+0000","last_clean":"2024-03-28T18:55:36.565861+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:55:36.565861+0000","last_undegraded":"2024-03-28T18:55:36.565861+0000","last_fullsized":"2024-03-28T18:55:36.565861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790884+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:55:36.790884+0000","last_peered":"2024-03-28T18:55:36.790884+0000","last_clean":"2024-03-28T18:55:36.790884+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:55:36.790884+0000","last_undegraded":"2024-03-28T18:55:36.790884+0000","last_fullsized":"2024-03-28T18:55:36.790884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964095+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:55:35.964095+0000","last_peered":"2024-03-28T18:55:35.964095+0000","last_clean":"2024-03-28T18:55:35.964095+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:55:35.964095+0000","last_undegraded":"2024-03-28T18:55:35.964095+0000","last_fullsized":"2024-03-28T18:55:35.964095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790463+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:55:36.790463+0000","last_peered":"2024-03-28T18:55:36.790463+0000","last_clean":"2024-03-28T18:55:36.790463+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:55:36.790463+0000","last_undegraded":"2024-03-28T18:55:36.790463+0000","last_fullsized":"2024-03-28T18:55:36.790463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963646+0000","last_change":"2024-03-28T18:55:34.491855+0000","last_active":"2024-03-28T18:55:35.963646+0000","last_peered":"2024-03-28T18:55:35.963646+0000","last_clean":"2024-03-28T18:55:35.963646+0000","last_became_active":"2024-03-28T18:55:34.491070+0000","last_became_peered":"2024-03-28T18:55:34.491070+0000","last_unstale":"2024-03-28T18:55:35.963646+0000","last_undegraded":"2024-03-28T18:55:35.963646+0000","last_fullsized":"2024-03-28T18:55:35.963646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789604+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:55:36.789604+0000","last_peered":"2024-03-28T18:55:36.789604+0000","last_clean":"2024-03-28T18:55:36.789604+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:55:36.789604+0000","last_undegraded":"2024-03-28T18:55:36.789604+0000","last_fullsized":"2024-03-28T18:55:36.789604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.965568+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:55:35.965568+0000","last_peered":"2024-03-28T18:55:35.965568+0000","last_clean":"2024-03-28T18:55:35.965568+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:55:35.965568+0000","last_undegraded":"2024-03-28T18:55:35.965568+0000","last_fullsized":"2024-03-28T18:55:35.965568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.494886+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:55:35.494886+0000","last_peered":"2024-03-28T18:55:35.494886+0000","last_clean":"2024-03-28T18:55:35.494886+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:55:35.494886+0000","last_undegraded":"2024-03-28T18:55:35.494886+0000","last_fullsized":"2024-03-28T18:55:35.494886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493849+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:55:35.493849+0000","last_peered":"2024-03-28T18:55:35.493849+0000","last_clean":"2024-03-28T18:55:35.493849+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:55:35.493849+0000","last_undegraded":"2024-03-28T18:55:35.493849+0000","last_fullsized":"2024-03-28T18:55:35.493849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493539+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:55:35.493539+0000","last_peered":"2024-03-28T18:55:35.493539+0000","last_clean":"2024-03-28T18:55:35.493539+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:55:35.493539+0000","last_undegraded":"2024-03-28T18:55:35.493539+0000","last_fullsized":"2024-03-28T18:55:35.493539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790005+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:55:36.790005+0000","last_peered":"2024-03-28T18:55:36.790005+0000","last_clean":"2024-03-28T18:55:36.790005+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:55:36.790005+0000","last_undegraded":"2024-03-28T18:55:36.790005+0000","last_fullsized":"2024-03-28T18:55:36.790005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565452+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:55:36.565452+0000","last_peered":"2024-03-28T18:55:36.565452+0000","last_clean":"2024-03-28T18:55:36.565452+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:55:36.565452+0000","last_undegraded":"2024-03-28T18:55:36.565452+0000","last_fullsized":"2024-03-28T18:55:36.565452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490355+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:55:35.490355+0000","last_peered":"2024-03-28T18:55:35.490355+0000","last_clean":"2024-03-28T18:55:35.490355+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:55:35.490355+0000","last_undegraded":"2024-03-28T18:55:35.490355+0000","last_fullsized":"2024-03-28T18:55:35.490355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790166+0000","last_change":"2024-03-28T18:55:34.509529+0000","last_active":"2024-03-28T18:55:36.790166+0000","last_peered":"2024-03-28T18:55:36.790166+0000","last_clean":"2024-03-28T18:55:36.790166+0000","last_became_active":"2024-03-28T18:55:34.501100+0000","last_became_peered":"2024-03-28T18:55:34.501100+0000","last_unstale":"2024-03-28T18:55:36.790166+0000","last_undegraded":"2024-03-28T18:55:36.790166+0000","last_fullsized":"2024-03-28T18:55:36.790166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791856+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:55:36.791856+0000","last_peered":"2024-03-28T18:55:36.791856+0000","last_clean":"2024-03-28T18:55:36.791856+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:55:36.791856+0000","last_undegraded":"2024-03-28T18:55:36.791856+0000","last_fullsized":"2024-03-28T18:55:36.791856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.788223+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:55:36.788223+0000","last_peered":"2024-03-28T18:55:36.788223+0000","last_clean":"2024-03-28T18:55:36.788223+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:55:36.788223+0000","last_undegraded":"2024-03-28T18:55:36.788223+0000","last_fullsized":"2024-03-28T18:55:36.788223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964728+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:55:35.964728+0000","last_peered":"2024-03-28T18:55:35.964728+0000","last_clean":"2024-03-28T18:55:35.964728+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:55:35.964728+0000","last_undegraded":"2024-03-28T18:55:35.964728+0000","last_fullsized":"2024-03-28T18:55:35.964728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.492925+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:55:35.492925+0000","last_peered":"2024-03-28T18:55:35.492925+0000","last_clean":"2024-03-28T18:55:35.492925+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:55:35.492925+0000","last_undegraded":"2024-03-28T18:55:35.492925+0000","last_fullsized":"2024-03-28T18:55:35.492925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565763+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:55:36.565763+0000","last_peered":"2024-03-28T18:55:36.565763+0000","last_clean":"2024-03-28T18:55:36.565763+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:55:36.565763+0000","last_undegraded":"2024-03-28T18:55:36.565763+0000","last_fullsized":"2024-03-28T18:55:36.565763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790580+0000","last_change":"2024-03-28T18:55:34.505741+0000","last_active":"2024-03-28T18:55:36.790580+0000","last_peered":"2024-03-28T18:55:36.790580+0000","last_clean":"2024-03-28T18:55:36.790580+0000","last_became_active":"2024-03-28T18:55:34.505440+0000","last_became_peered":"2024-03-28T18:55:34.505440+0000","last_unstale":"2024-03-28T18:55:36.790580+0000","last_undegraded":"2024-03-28T18:55:36.790580+0000","last_fullsized":"2024-03-28T18:55:36.790580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789593+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:55:36.789593+0000","last_peered":"2024-03-28T18:55:36.789593+0000","last_clean":"2024-03-28T18:55:36.789593+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:55:36.789593+0000","last_undegraded":"2024-03-28T18:55:36.789593+0000","last_fullsized":"2024-03-28T18:55:36.789593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790371+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:55:36.790371+0000","last_peered":"2024-03-28T18:55:36.790371+0000","last_clean":"2024-03-28T18:55:36.790371+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:55:36.790371+0000","last_undegraded":"2024-03-28T18:55:36.790371+0000","last_fullsized":"2024-03-28T18:55:36.790371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789834+0000","last_change":"2024-03-28T18:55:34.491857+0000","last_active":"2024-03-28T18:55:36.789834+0000","last_peered":"2024-03-28T18:55:36.789834+0000","last_clean":"2024-03-28T18:55:36.789834+0000","last_became_active":"2024-03-28T18:55:34.487194+0000","last_became_peered":"2024-03-28T18:55:34.487194+0000","last_unstale":"2024-03-28T18:55:36.789834+0000","last_undegraded":"2024-03-28T18:55:36.789834+0000","last_fullsized":"2024-03-28T18:55:36.789834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.495025+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:55:35.495025+0000","last_peered":"2024-03-28T18:55:35.495025+0000","last_clean":"2024-03-28T18:55:35.495025+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:55:35.495025+0000","last_undegraded":"2024-03-28T18:55:35.495025+0000","last_fullsized":"2024-03-28T18:55:35.495025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.489933+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:55:35.489933+0000","last_peered":"2024-03-28T18:55:35.489933+0000","last_clean":"2024-03-28T18:55:35.489933+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:55:35.489933+0000","last_undegraded":"2024-03-28T18:55:35.489933+0000","last_fullsized":"2024-03-28T18:55:35.489933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790541+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:55:36.790541+0000","last_peered":"2024-03-28T18:55:36.790541+0000","last_clean":"2024-03-28T18:55:36.790541+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:55:36.790541+0000","last_undegraded":"2024-03-28T18:55:36.790541+0000","last_fullsized":"2024-03-28T18:55:36.790541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565773+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:55:36.565773+0000","last_peered":"2024-03-28T18:55:36.565773+0000","last_clean":"2024-03-28T18:55:36.565773+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:55:36.565773+0000","last_undegraded":"2024-03-28T18:55:36.565773+0000","last_fullsized":"2024-03-28T18:55:36.565773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790851+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:55:36.790851+0000","last_peered":"2024-03-28T18:55:36.790851+0000","last_clean":"2024-03-28T18:55:36.790851+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:55:36.790851+0000","last_undegraded":"2024-03-28T18:55:36.790851+0000","last_fullsized":"2024-03-28T18:55:36.790851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.788123+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:55:36.788123+0000","last_peered":"2024-03-28T18:55:36.788123+0000","last_clean":"2024-03-28T18:55:36.788123+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:55:36.788123+0000","last_undegraded":"2024-03-28T18:55:36.788123+0000","last_fullsized":"2024-03-28T18:55:36.788123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490573+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:55:35.490573+0000","last_peered":"2024-03-28T18:55:35.490573+0000","last_clean":"2024-03-28T18:55:35.490573+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:55:35.490573+0000","last_undegraded":"2024-03-28T18:55:35.490573+0000","last_fullsized":"2024-03-28T18:55:35.490573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.494892+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:55:35.494892+0000","last_peered":"2024-03-28T18:55:35.494892+0000","last_clean":"2024-03-28T18:55:35.494892+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:55:35.494892+0000","last_undegraded":"2024-03-28T18:55:35.494892+0000","last_fullsized":"2024-03-28T18:55:35.494892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790199+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:55:36.790199+0000","last_peered":"2024-03-28T18:55:36.790199+0000","last_clean":"2024-03-28T18:55:36.790199+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:55:36.790199+0000","last_undegraded":"2024-03-28T18:55:36.790199+0000","last_fullsized":"2024-03-28T18:55:36.790199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565993+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:55:36.565993+0000","last_peered":"2024-03-28T18:55:36.565993+0000","last_clean":"2024-03-28T18:55:36.565993+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:55:36.565993+0000","last_undegraded":"2024-03-28T18:55:36.565993+0000","last_fullsized":"2024-03-28T18:55:36.565993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.491581+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:55:35.491581+0000","last_peered":"2024-03-28T18:55:35.491581+0000","last_clean":"2024-03-28T18:55:35.491581+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:55:35.491581+0000","last_undegraded":"2024-03-28T18:55:35.491581+0000","last_fullsized":"2024-03-28T18:55:35.491581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789611+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:55:36.789611+0000","last_peered":"2024-03-28T18:55:36.789611+0000","last_clean":"2024-03-28T18:55:36.789611+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:55:36.789611+0000","last_undegraded":"2024-03-28T18:55:36.789611+0000","last_fullsized":"2024-03-28T18:55:36.789611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789441+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:55:36.789441+0000","last_peered":"2024-03-28T18:55:36.789441+0000","last_clean":"2024-03-28T18:55:36.789441+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:55:36.789441+0000","last_undegraded":"2024-03-28T18:55:36.789441+0000","last_fullsized":"2024-03-28T18:55:36.789441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790355+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:55:36.790355+0000","last_peered":"2024-03-28T18:55:36.790355+0000","last_clean":"2024-03-28T18:55:36.790355+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:55:36.790355+0000","last_undegraded":"2024-03-28T18:55:36.790355+0000","last_fullsized":"2024-03-28T18:55:36.790355+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490186+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:55:35.490186+0000","last_peered":"2024-03-28T18:55:35.490186+0000","last_clean":"2024-03-28T18:55:35.490186+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:55:35.490186+0000","last_undegraded":"2024-03-28T18:55:35.490186+0000","last_fullsized":"2024-03-28T18:55:35.490186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493816+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:55:35.493816+0000","last_peered":"2024-03-28T18:55:35.493816+0000","last_clean":"2024-03-28T18:55:35.493816+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:55:35.493816+0000","last_undegraded":"2024-03-28T18:55:35.493816+0000","last_fullsized":"2024-03-28T18:55:35.493816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791935+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:55:36.791935+0000","last_peered":"2024-03-28T18:55:36.791935+0000","last_clean":"2024-03-28T18:55:36.791935+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:55:36.791935+0000","last_undegraded":"2024-03-28T18:55:36.791935+0000","last_fullsized":"2024-03-28T18:55:36.791935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791900+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:55:36.791900+0000","last_peered":"2024-03-28T18:55:36.791900+0000","last_clean":"2024-03-28T18:55:36.791900+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:55:36.791900+0000","last_undegraded":"2024-03-28T18:55:36.791900+0000","last_fullsized":"2024-03-28T18:55:36.791900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.566044+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:55:36.566044+0000","last_peered":"2024-03-28T18:55:36.566044+0000","last_clean":"2024-03-28T18:55:36.566044+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:55:36.566044+0000","last_undegraded":"2024-03-28T18:55:36.566044+0000","last_fullsized":"2024-03-28T18:55:36.566044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490108+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:55:35.490108+0000","last_peered":"2024-03-28T18:55:35.490108+0000","last_clean":"2024-03-28T18:55:35.490108+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:55:35.490108+0000","last_undegraded":"2024-03-28T18:55:35.490108+0000","last_fullsized":"2024-03-28T18:55:35.490108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493717+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:55:35.493717+0000","last_peered":"2024-03-28T18:55:35.493717+0000","last_clean":"2024-03-28T18:55:35.493717+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:55:35.493717+0000","last_undegraded":"2024-03-28T18:55:35.493717+0000","last_fullsized":"2024-03-28T18:55:35.493717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493058+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:55:35.493058+0000","last_peered":"2024-03-28T18:55:35.493058+0000","last_clean":"2024-03-28T18:55:35.493058+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:55:35.493058+0000","last_undegraded":"2024-03-28T18:55:35.493058+0000","last_fullsized":"2024-03-28T18:55:35.493058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964132+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:55:35.964132+0000","last_peered":"2024-03-28T18:55:35.964132+0000","last_clean":"2024-03-28T18:55:35.964132+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:55:35.964132+0000","last_undegraded":"2024-03-28T18:55:35.964132+0000","last_fullsized":"2024-03-28T18:55:35.964132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565887+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:55:36.565887+0000","last_peered":"2024-03-28T18:55:36.565887+0000","last_clean":"2024-03-28T18:55:36.565887+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:55:36.565887+0000","last_undegraded":"2024-03-28T18:55:36.565887+0000","last_fullsized":"2024-03-28T18:55:36.565887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.566239+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:55:36.566239+0000","last_peered":"2024-03-28T18:55:36.566239+0000","last_clean":"2024-03-28T18:55:36.566239+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:55:36.566239+0000","last_undegraded":"2024-03-28T18:55:36.566239+0000","last_fullsized":"2024-03-28T18:55:36.566239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790840+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:55:36.790840+0000","last_peered":"2024-03-28T18:55:36.790840+0000","last_clean":"2024-03-28T18:55:36.790840+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:55:36.790840+0000","last_undegraded":"2024-03-28T18:55:36.790840+0000","last_fullsized":"2024-03-28T18:55:36.790840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964374+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:55:35.964374+0000","last_peered":"2024-03-28T18:55:35.964374+0000","last_clean":"2024-03-28T18:55:35.964374+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:55:35.964374+0000","last_undegraded":"2024-03-28T18:55:35.964374+0000","last_fullsized":"2024-03-28T18:55:35.964374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490824+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:55:35.490824+0000","last_peered":"2024-03-28T18:55:35.490824+0000","last_clean":"2024-03-28T18:55:35.490824+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:55:35.490824+0000","last_undegraded":"2024-03-28T18:55:35.490824+0000","last_fullsized":"2024-03-28T18:55:35.490824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.491901+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:55:35.491901+0000","last_peered":"2024-03-28T18:55:35.491901+0000","last_clean":"2024-03-28T18:55:35.491901+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:55:35.491901+0000","last_undegraded":"2024-03-28T18:55:35.491901+0000","last_fullsized":"2024-03-28T18:55:35.491901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.787801+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:55:36.787801+0000","last_peered":"2024-03-28T18:55:36.787801+0000","last_clean":"2024-03-28T18:55:36.787801+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:55:36.787801+0000","last_undegraded":"2024-03-28T18:55:36.787801+0000","last_fullsized":"2024-03-28T18:55:36.787801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964128+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:55:35.964128+0000","last_peered":"2024-03-28T18:55:35.964128+0000","last_clean":"2024-03-28T18:55:35.964128+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:55:35.964128+0000","last_undegraded":"2024-03-28T18:55:35.964128+0000","last_fullsized":"2024-03-28T18:55:35.964128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.492912+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:55:35.492912+0000","last_peered":"2024-03-28T18:55:35.492912+0000","last_clean":"2024-03-28T18:55:35.492912+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:55:35.492912+0000","last_undegraded":"2024-03-28T18:55:35.492912+0000","last_fullsized":"2024-03-28T18:55:35.492912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964965+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:55:35.964965+0000","last_peered":"2024-03-28T18:55:35.964965+0000","last_clean":"2024-03-28T18:55:35.964965+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:55:35.964965+0000","last_undegraded":"2024-03-28T18:55:35.964965+0000","last_fullsized":"2024-03-28T18:55:35.964965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.788220+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:55:36.788220+0000","last_peered":"2024-03-28T18:55:36.788220+0000","last_clean":"2024-03-28T18:55:36.788220+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:55:36.788220+0000","last_undegraded":"2024-03-28T18:55:36.788220+0000","last_fullsized":"2024-03-28T18:55:36.788220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.965903+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:55:35.965903+0000","last_peered":"2024-03-28T18:55:35.965903+0000","last_clean":"2024-03-28T18:55:35.965903+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:55:35.965903+0000","last_undegraded":"2024-03-28T18:55:35.965903+0000","last_fullsized":"2024-03-28T18:55:35.965903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490242+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:55:35.490242+0000","last_peered":"2024-03-28T18:55:35.490242+0000","last_clean":"2024-03-28T18:55:35.490242+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:55:35.490242+0000","last_undegraded":"2024-03-28T18:55:35.490242+0000","last_fullsized":"2024-03-28T18:55:35.490242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791310+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:55:36.791310+0000","last_peered":"2024-03-28T18:55:36.791310+0000","last_clean":"2024-03-28T18:55:36.791310+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:55:36.791310+0000","last_undegraded":"2024-03-28T18:55:36.791310+0000","last_fullsized":"2024-03-28T18:55:36.791310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.788428+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:55:36.788428+0000","last_peered":"2024-03-28T18:55:36.788428+0000","last_clean":"2024-03-28T18:55:36.788428+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:55:36.788428+0000","last_undegraded":"2024-03-28T18:55:36.788428+0000","last_fullsized":"2024-03-28T18:55:36.788428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964192+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:55:35.964192+0000","last_peered":"2024-03-28T18:55:35.964192+0000","last_clean":"2024-03-28T18:55:35.964192+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:55:35.964192+0000","last_undegraded":"2024-03-28T18:55:35.964192+0000","last_fullsized":"2024-03-28T18:55:35.964192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963614+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:55:35.963614+0000","last_peered":"2024-03-28T18:55:35.963614+0000","last_clean":"2024-03-28T18:55:35.963614+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:55:35.963614+0000","last_undegraded":"2024-03-28T18:55:35.963614+0000","last_fullsized":"2024-03-28T18:55:35.963614+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790232+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:55:36.790232+0000","last_peered":"2024-03-28T18:55:36.790232+0000","last_clean":"2024-03-28T18:55:36.790232+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:55:36.790232+0000","last_undegraded":"2024-03-28T18:55:36.790232+0000","last_fullsized":"2024-03-28T18:55:36.790232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790187+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:55:36.790187+0000","last_peered":"2024-03-28T18:55:36.790187+0000","last_clean":"2024-03-28T18:55:36.790187+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:55:36.790187+0000","last_undegraded":"2024-03-28T18:55:36.790187+0000","last_fullsized":"2024-03-28T18:55:36.790187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963306+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:55:35.963306+0000","last_peered":"2024-03-28T18:55:35.963306+0000","last_clean":"2024-03-28T18:55:35.963306+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:55:35.963306+0000","last_undegraded":"2024-03-28T18:55:35.963306+0000","last_fullsized":"2024-03-28T18:55:35.963306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565377+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:55:36.565377+0000","last_peered":"2024-03-28T18:55:36.565377+0000","last_clean":"2024-03-28T18:55:36.565377+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:55:36.565377+0000","last_undegraded":"2024-03-28T18:55:36.565377+0000","last_fullsized":"2024-03-28T18:55:36.565377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.787941+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:55:36.787941+0000","last_peered":"2024-03-28T18:55:36.787941+0000","last_clean":"2024-03-28T18:55:36.787941+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:55:36.787941+0000","last_undegraded":"2024-03-28T18:55:36.787941+0000","last_fullsized":"2024-03-28T18:55:36.787941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.489991+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:55:35.489991+0000","last_peered":"2024-03-28T18:55:35.489991+0000","last_clean":"2024-03-28T18:55:35.489991+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:55:35.489991+0000","last_undegraded":"2024-03-28T18:55:35.489991+0000","last_fullsized":"2024-03-28T18:55:35.489991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790972+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:55:36.790972+0000","last_peered":"2024-03-28T18:55:36.790972+0000","last_clean":"2024-03-28T18:55:36.790972+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:55:36.790972+0000","last_undegraded":"2024-03-28T18:55:36.790972+0000","last_fullsized":"2024-03-28T18:55:36.790972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.494475+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:55:35.494475+0000","last_peered":"2024-03-28T18:55:35.494475+0000","last_clean":"2024-03-28T18:55:35.494475+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:55:35.494475+0000","last_undegraded":"2024-03-28T18:55:35.494475+0000","last_fullsized":"2024-03-28T18:55:35.494475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490060+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:55:35.490060+0000","last_peered":"2024-03-28T18:55:35.490060+0000","last_clean":"2024-03-28T18:55:35.490060+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:55:35.490060+0000","last_undegraded":"2024-03-28T18:55:35.490060+0000","last_fullsized":"2024-03-28T18:55:35.490060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791542+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:55:36.791542+0000","last_peered":"2024-03-28T18:55:36.791542+0000","last_clean":"2024-03-28T18:55:36.791542+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:55:36.791542+0000","last_undegraded":"2024-03-28T18:55:36.791542+0000","last_fullsized":"2024-03-28T18:55:36.791542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964668+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:55:35.964668+0000","last_peered":"2024-03-28T18:55:35.964668+0000","last_clean":"2024-03-28T18:55:35.964668+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:55:35.964668+0000","last_undegraded":"2024-03-28T18:55:35.964668+0000","last_fullsized":"2024-03-28T18:55:35.964668+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490004+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:55:35.490004+0000","last_peered":"2024-03-28T18:55:35.490004+0000","last_clean":"2024-03-28T18:55:35.490004+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:55:35.490004+0000","last_undegraded":"2024-03-28T18:55:35.490004+0000","last_fullsized":"2024-03-28T18:55:35.490004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490476+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:55:35.490476+0000","last_peered":"2024-03-28T18:55:35.490476+0000","last_clean":"2024-03-28T18:55:35.490476+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:55:35.490476+0000","last_undegraded":"2024-03-28T18:55:35.490476+0000","last_fullsized":"2024-03-28T18:55:35.490476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790042+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:55:36.790042+0000","last_peered":"2024-03-28T18:55:36.790042+0000","last_clean":"2024-03-28T18:55:36.790042+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:55:36.790042+0000","last_undegraded":"2024-03-28T18:55:36.790042+0000","last_fullsized":"2024-03-28T18:55:36.790042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.494052+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:55:35.494052+0000","last_peered":"2024-03-28T18:55:35.494052+0000","last_clean":"2024-03-28T18:55:35.494052+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:55:35.494052+0000","last_undegraded":"2024-03-28T18:55:35.494052+0000","last_fullsized":"2024-03-28T18:55:35.494052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565921+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:55:36.565921+0000","last_peered":"2024-03-28T18:55:36.565921+0000","last_clean":"2024-03-28T18:55:36.565921+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:55:36.565921+0000","last_undegraded":"2024-03-28T18:55:36.565921+0000","last_fullsized":"2024-03-28T18:55:36.565921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963305+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:55:35.963305+0000","last_peered":"2024-03-28T18:55:35.963305+0000","last_clean":"2024-03-28T18:55:35.963305+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:55:35.963305+0000","last_undegraded":"2024-03-28T18:55:35.963305+0000","last_fullsized":"2024-03-28T18:55:35.963305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493039+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:55:35.493039+0000","last_peered":"2024-03-28T18:55:35.493039+0000","last_clean":"2024-03-28T18:55:35.493039+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:55:35.493039+0000","last_undegraded":"2024-03-28T18:55:35.493039+0000","last_fullsized":"2024-03-28T18:55:35.493039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490351+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:55:35.490351+0000","last_peered":"2024-03-28T18:55:35.490351+0000","last_clean":"2024-03-28T18:55:35.490351+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:55:35.490351+0000","last_undegraded":"2024-03-28T18:55:35.490351+0000","last_fullsized":"2024-03-28T18:55:35.490351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789628+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:55:36.789628+0000","last_peered":"2024-03-28T18:55:36.789628+0000","last_clean":"2024-03-28T18:55:36.789628+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:55:36.789628+0000","last_undegraded":"2024-03-28T18:55:36.789628+0000","last_fullsized":"2024-03-28T18:55:36.789628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.489767+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:55:35.489767+0000","last_peered":"2024-03-28T18:55:35.489767+0000","last_clean":"2024-03-28T18:55:35.489767+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:55:35.489767+0000","last_undegraded":"2024-03-28T18:55:35.489767+0000","last_fullsized":"2024-03-28T18:55:35.489767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790207+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:55:36.790207+0000","last_peered":"2024-03-28T18:55:36.790207+0000","last_clean":"2024-03-28T18:55:36.790207+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:55:36.790207+0000","last_undegraded":"2024-03-28T18:55:36.790207+0000","last_fullsized":"2024-03-28T18:55:36.790207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964242+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:55:35.964242+0000","last_peered":"2024-03-28T18:55:35.964242+0000","last_clean":"2024-03-28T18:55:35.964242+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:55:35.964242+0000","last_undegraded":"2024-03-28T18:55:35.964242+0000","last_fullsized":"2024-03-28T18:55:35.964242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.491406+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:55:35.491406+0000","last_peered":"2024-03-28T18:55:35.491406+0000","last_clean":"2024-03-28T18:55:35.491406+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:55:35.491406+0000","last_undegraded":"2024-03-28T18:55:35.491406+0000","last_fullsized":"2024-03-28T18:55:35.491406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791124+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:55:36.791124+0000","last_peered":"2024-03-28T18:55:36.791124+0000","last_clean":"2024-03-28T18:55:36.791124+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:55:36.791124+0000","last_undegraded":"2024-03-28T18:55:36.791124+0000","last_fullsized":"2024-03-28T18:55:36.791124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963924+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:55:35.963924+0000","last_peered":"2024-03-28T18:55:35.963924+0000","last_clean":"2024-03-28T18:55:35.963924+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:55:35.963924+0000","last_undegraded":"2024-03-28T18:55:35.963924+0000","last_fullsized":"2024-03-28T18:55:35.963924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791306+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:55:36.791306+0000","last_peered":"2024-03-28T18:55:36.791306+0000","last_clean":"2024-03-28T18:55:36.791306+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:55:36.791306+0000","last_undegraded":"2024-03-28T18:55:36.791306+0000","last_fullsized":"2024-03-28T18:55:36.791306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964371+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:55:35.964371+0000","last_peered":"2024-03-28T18:55:35.964371+0000","last_clean":"2024-03-28T18:55:35.964371+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:55:35.964371+0000","last_undegraded":"2024-03-28T18:55:35.964371+0000","last_fullsized":"2024-03-28T18:55:35.964371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.566006+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:55:36.566006+0000","last_peered":"2024-03-28T18:55:36.566006+0000","last_clean":"2024-03-28T18:55:36.566006+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:55:36.566006+0000","last_undegraded":"2024-03-28T18:55:36.566006+0000","last_fullsized":"2024-03-28T18:55:36.566006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790187+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:55:36.790187+0000","last_peered":"2024-03-28T18:55:36.790187+0000","last_clean":"2024-03-28T18:55:36.790187+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:55:36.790187+0000","last_undegraded":"2024-03-28T18:55:36.790187+0000","last_fullsized":"2024-03-28T18:55:36.790187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789987+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:55:36.789987+0000","last_peered":"2024-03-28T18:55:36.789987+0000","last_clean":"2024-03-28T18:55:36.789987+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:55:36.789987+0000","last_undegraded":"2024-03-28T18:55:36.789987+0000","last_fullsized":"2024-03-28T18:55:36.789987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.566127+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:55:36.566127+0000","last_peered":"2024-03-28T18:55:36.566127+0000","last_clean":"2024-03-28T18:55:36.566127+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:55:36.566127+0000","last_undegraded":"2024-03-28T18:55:36.566127+0000","last_fullsized":"2024-03-28T18:55:36.566127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":70944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:55:40.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:55:40.179 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T18:55:40.178+0000 7f7991e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:55:40.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.282+0000 7fe9d7200640 1 Processor -- start 2024-03-28T18:55:40.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.282+0000 7fe9d7200640 1 -- start start 2024-03-28T18:55:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9d8156120 0x7fe9d806e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d7200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe9d8156500 0x7fe9d80c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d7200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d8159c50 con 0x7fe9d80c5320 2024-03-28T18:55:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d7200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d80c4d70 con 0x7fe9d8156500 2024-03-28T18:55:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c1030 con 0x7fe9d8156120 2024-03-28T18:55:40.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d5400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe9d80c5320 0x7fe9d80c0920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34034/0 (socket says 172.21.15.27:34034) 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d5400640 1 -- 172.21.15.27:0/3371346485 learned_addr learned my addr 172.21.15.27:0/3371346485 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d6800640 1 --2- 172.21.15.27:0/3371346485 >> v2:172.21.15.155:3300/0 conn(0x7fe9d8156500 0x7fe9d80c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d5e00640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9d8156120 0x7fe9d806e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.284+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921733179 0 0) 0x7fe9d8159c50 con 0x7fe9d80c5320 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc003520 con 0x7fe9d80c5320 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2203364033 0 0) 0x7fe9bc003520 con 0x7fe9d80c5320 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9d8156120 msgr2=0x7fe9d806e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9d8156120 0x7fe9d806e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 >> v2:172.21.15.155:3300/0 conn(0x7fe9d8156500 msgr2=0x7fe9d80c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 --2- 172.21.15.27:0/3371346485 >> v2:172.21.15.155:3300/0 conn(0x7fe9d8156500 0x7fe9d80c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d80c4a40 con 0x7fe9d80c5320 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe9c4001670 con 0x7fe9d80c5320 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 >> v1:172.21.15.27:6789/0 conn(0x7fe9d80c5320 legacy=0x7fe9d80c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9bc0039b0 0x7fe9bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc004800 0x7fe9bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9d8156120 0x7fe9bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c1030 con 0x7fe9bc004800 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc00ba70 con 0x7fe9bc0039b0 2024-03-28T18:55:40.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.285+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc00bc20 con 0x7fe9d8156120 2024-03-28T18:55:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.286+0000 7fe9d5400640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc004800 0x7fe9bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.286+0000 7fe9d5e00640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9bc0039b0 0x7fe9bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.286+0000 7fe9d6800640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9d8156120 0x7fe9bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.286+0000 7fe9d5400640 1 -- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9d8156120 msgr2=0x7fe9bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.286+0000 7fe9d5400640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9d8156120 0x7fe9bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.286+0000 7fe9d5400640 1 -- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9bc0039b0 msgr2=0x7fe9bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.286+0000 7fe9d5400640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9bc0039b0 0x7fe9bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.286+0000 7fe9d5400640 1 -- 172.21.15.27:0/3371346485 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc003520 con 0x7fe9bc004800 2024-03-28T18:55:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.288+0000 7fe9d5400640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc004800 0x7fe9bc004c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400c160 tx=0x7fe9bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a1dfc46c4ca387 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.288+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4007c60 con 0x7fe9bc004800 2024-03-28T18:55:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.288+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c4030070 con 0x7fe9bc004800 2024-03-28T18:55:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.288+0000 7fe9d4a00640 1 -- 172.21.15.27:0/3371346485 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4005d50 con 0x7fe9bc004800 2024-03-28T18:55:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.288+0000 7fe9d7200640 1 -- 172.21.15.27:0/3371346485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc004800 msgr2=0x7fe9bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.288+0000 7fe9d7200640 1 --2- 172.21.15.27:0/3371346485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc004800 0x7fe9bc004c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400c160 tx=0x7fe9bc005830 comp rx=0 tx=0).stop 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.289+0000 7fe9d6800640 1 -- 172.21.15.27:0/3371346485 reap_dead start 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.289+0000 7fe9d7200640 1 -- 172.21.15.27:0/3371346485 shutdown_connections 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.289+0000 7fe9d7200640 1 -- 172.21.15.27:0/3371346485 >> 172.21.15.27:0/3371346485 conn(0x7fe9d806b0c0 msgr2=0x7fe9d806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.289+0000 7fe9d7200640 1 -- 172.21.15.27:0/3371346485 shutdown_connections 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.289+0000 7fe9d7200640 1 -- 172.21.15.27:0/3371346485 wait complete. 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.290+0000 7fe9d7200640 1 Processor -- start 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.291+0000 7fe9d7200640 1 -- start start 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.291+0000 7fe9d7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9bc004800 0x7fe9d81eed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.291+0000 7fe9d7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9d8072d70 0x7fe9d81ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.291+0000 7fe9d7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d8148bd0 0x7fe9d8149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.291+0000 7fe9d7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c73d0 con 0x7fe9d8148bd0 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.291+0000 7fe9d7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c7250 con 0x7fe9d8072d70 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.291+0000 7fe9d7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c29a0 con 0x7fe9bc004800 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d8148bd0 0x7fe9d8149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d8148bd0 0x7fe9d8149050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53242/0 (socket says 172.21.15.27:53242) 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 -- 172.21.15.27:0/3689074571 learned_addr learned my addr 172.21.15.27:0/3689074571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d5400640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9d8072d70 0x7fe9d81ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d5e00640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9bc004800 0x7fe9d81eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 -- 172.21.15.27:0/3689074571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9bc004800 msgr2=0x7fe9d81eed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9bc004800 0x7fe9d81eed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 -- 172.21.15.27:0/3689074571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9d8072d70 msgr2=0x7fe9d81ef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9d8072d70 0x7fe9d81ef290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 -- 172.21.15.27:0/3689074571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d81ef7d0 con 0x7fe9d8148bd0 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.292+0000 7fe9d6800640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d8148bd0 0x7fe9d8149050 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8156910 tx=0x7fe9d00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.293+0000 7fe9ce000640 1 -- 172.21.15.27:0/3689074571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0011e40 con 0x7fe9d8148bd0 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.293+0000 7fe9ce000640 1 -- 172.21.15.27:0/3689074571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d001b070 con 0x7fe9d8148bd0 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.293+0000 7fe9ce000640 1 -- 172.21.15.27:0/3689074571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0003990 con 0x7fe9d8148bd0 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.293+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d8073310 con 0x7fe9d8148bd0 2024-03-28T18:55:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.293+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d809b370 con 0x7fe9d8148bd0 2024-03-28T18:55:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.294+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d8067dd0 con 0x7fe9d8148bd0 2024-03-28T18:55:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.295+0000 7fe9ce000640 1 -- 172.21.15.27:0/3689074571 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe9d0005d10 con 0x7fe9d8148bd0 2024-03-28T18:55:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.296+0000 7fe9ce000640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9b806bbd0 0x7fe9b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.296+0000 7fe9ce000640 1 -- 172.21.15.27:0/3689074571 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7fe9d008afd0 con 0x7fe9d8148bd0 2024-03-28T18:55:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.297+0000 7fe9d5e00640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9b806bbd0 0x7fe9b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.297+0000 7fe9d5e00640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9b806bbd0 0x7fe9b806e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80099c0 tx=0x7fe9c8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:40.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.301+0000 7fe9ce000640 1 -- 172.21.15.27:0/3689074571 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe9d0014350 con 0x7fe9d8148bd0 2024-03-28T18:55:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.491+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe9d809ae80 con 0x7fe9b806bbd0 2024-03-28T18:55:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.499+0000 7fe9ce000640 1 -- 172.21.15.27:0/3689074571 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251946 (secure 0 0 0) 0x7fe9d809ae80 con 0x7fe9b806bbd0 2024-03-28T18:55:40.503 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:40.505 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:55:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9b806bbd0 msgr2=0x7fe9b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9b806bbd0 0x7fe9b806e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80099c0 tx=0x7fe9c8008040 comp rx=0 tx=0).stop 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d8148bd0 msgr2=0x7fe9d8149050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d8148bd0 0x7fe9d8149050 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8156910 tx=0x7fe9d00115a0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 shutdown_connections 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9b806bbd0 0x7fe9b806e090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9d8148bd0 0x7fe9d8149050 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9d8072d70 0x7fe9d81ef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 --2- 172.21.15.27:0/3689074571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9bc004800 0x7fe9d81eed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 >> 172.21.15.27:0/3689074571 conn(0x7fe9d806b0c0 msgr2=0x7fe9d80a6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:40.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 shutdown_connections 2024-03-28T18:55:40.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.504+0000 7fe9d7200640 1 -- 172.21.15.27:0/3689074571 wait complete. 2024-03-28T18:55:40.523 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-28T18:55:38.893302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219004,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755716,"statfs":{"total":773094113280,"available":772869853184,"internally_reserved":0,"allocated":2891776,"data_stored":1752388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":47,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002120"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.965537+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:55:35.965537+0000","last_peered":"2024-03-28T18:55:35.965537+0000","last_clean":"2024-03-28T18:55:35.965537+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:55:35.965537+0000","last_undegraded":"2024-03-28T18:55:35.965537+0000","last_fullsized":"2024-03-28T18:55:35.965537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490376+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:55:35.490376+0000","last_peered":"2024-03-28T18:55:35.490376+0000","last_clean":"2024-03-28T18:55:35.490376+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:55:35.490376+0000","last_undegraded":"2024-03-28T18:55:35.490376+0000","last_fullsized":"2024-03-28T18:55:35.490376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565861+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:55:36.565861+0000","last_peered":"2024-03-28T18:55:36.565861+0000","last_clean":"2024-03-28T18:55:36.565861+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:55:36.565861+0000","last_undegraded":"2024-03-28T18:55:36.565861+0000","last_fullsized":"2024-03-28T18:55:36.565861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790884+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:55:36.790884+0000","last_peered":"2024-03-28T18:55:36.790884+0000","last_clean":"2024-03-28T18:55:36.790884+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:55:36.790884+0000","last_undegraded":"2024-03-28T18:55:36.790884+0000","last_fullsized":"2024-03-28T18:55:36.790884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964095+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:55:35.964095+0000","last_peered":"2024-03-28T18:55:35.964095+0000","last_clean":"2024-03-28T18:55:35.964095+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:55:35.964095+0000","last_undegraded":"2024-03-28T18:55:35.964095+0000","last_fullsized":"2024-03-28T18:55:35.964095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790463+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:55:36.790463+0000","last_peered":"2024-03-28T18:55:36.790463+0000","last_clean":"2024-03-28T18:55:36.790463+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:55:36.790463+0000","last_undegraded":"2024-03-28T18:55:36.790463+0000","last_fullsized":"2024-03-28T18:55:36.790463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963646+0000","last_change":"2024-03-28T18:55:34.491855+0000","last_active":"2024-03-28T18:55:35.963646+0000","last_peered":"2024-03-28T18:55:35.963646+0000","last_clean":"2024-03-28T18:55:35.963646+0000","last_became_active":"2024-03-28T18:55:34.491070+0000","last_became_peered":"2024-03-28T18:55:34.491070+0000","last_unstale":"2024-03-28T18:55:35.963646+0000","last_undegraded":"2024-03-28T18:55:35.963646+0000","last_fullsized":"2024-03-28T18:55:35.963646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789604+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:55:36.789604+0000","last_peered":"2024-03-28T18:55:36.789604+0000","last_clean":"2024-03-28T18:55:36.789604+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:55:36.789604+0000","last_undegraded":"2024-03-28T18:55:36.789604+0000","last_fullsized":"2024-03-28T18:55:36.789604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.965568+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:55:35.965568+0000","last_peered":"2024-03-28T18:55:35.965568+0000","last_clean":"2024-03-28T18:55:35.965568+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:55:35.965568+0000","last_undegraded":"2024-03-28T18:55:35.965568+0000","last_fullsized":"2024-03-28T18:55:35.965568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.494886+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:55:35.494886+0000","last_peered":"2024-03-28T18:55:35.494886+0000","last_clean":"2024-03-28T18:55:35.494886+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:55:35.494886+0000","last_undegraded":"2024-03-28T18:55:35.494886+0000","last_fullsized":"2024-03-28T18:55:35.494886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493849+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:55:35.493849+0000","last_peered":"2024-03-28T18:55:35.493849+0000","last_clean":"2024-03-28T18:55:35.493849+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:55:35.493849+0000","last_undegraded":"2024-03-28T18:55:35.493849+0000","last_fullsized":"2024-03-28T18:55:35.493849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493539+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:55:35.493539+0000","last_peered":"2024-03-28T18:55:35.493539+0000","last_clean":"2024-03-28T18:55:35.493539+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:55:35.493539+0000","last_undegraded":"2024-03-28T18:55:35.493539+0000","last_fullsized":"2024-03-28T18:55:35.493539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790005+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:55:36.790005+0000","last_peered":"2024-03-28T18:55:36.790005+0000","last_clean":"2024-03-28T18:55:36.790005+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:55:36.790005+0000","last_undegraded":"2024-03-28T18:55:36.790005+0000","last_fullsized":"2024-03-28T18:55:36.790005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565452+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:55:36.565452+0000","last_peered":"2024-03-28T18:55:36.565452+0000","last_clean":"2024-03-28T18:55:36.565452+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:55:36.565452+0000","last_undegraded":"2024-03-28T18:55:36.565452+0000","last_fullsized":"2024-03-28T18:55:36.565452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490355+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:55:35.490355+0000","last_peered":"2024-03-28T18:55:35.490355+0000","last_clean":"2024-03-28T18:55:35.490355+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:55:35.490355+0000","last_undegraded":"2024-03-28T18:55:35.490355+0000","last_fullsized":"2024-03-28T18:55:35.490355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790166+0000","last_change":"2024-03-28T18:55:34.509529+0000","last_active":"2024-03-28T18:55:36.790166+0000","last_peered":"2024-03-28T18:55:36.790166+0000","last_clean":"2024-03-28T18:55:36.790166+0000","last_became_active":"2024-03-28T18:55:34.501100+0000","last_became_peered":"2024-03-28T18:55:34.501100+0000","last_unstale":"2024-03-28T18:55:36.790166+0000","last_undegraded":"2024-03-28T18:55:36.790166+0000","last_fullsized":"2024-03-28T18:55:36.790166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791856+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:55:36.791856+0000","last_peered":"2024-03-28T18:55:36.791856+0000","last_clean":"2024-03-28T18:55:36.791856+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:55:36.791856+0000","last_undegraded":"2024-03-28T18:55:36.791856+0000","last_fullsized":"2024-03-28T18:55:36.791856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.788223+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:55:36.788223+0000","last_peered":"2024-03-28T18:55:36.788223+0000","last_clean":"2024-03-28T18:55:36.788223+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:55:36.788223+0000","last_undegraded":"2024-03-28T18:55:36.788223+0000","last_fullsized":"2024-03-28T18:55:36.788223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964728+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:55:35.964728+0000","last_peered":"2024-03-28T18:55:35.964728+0000","last_clean":"2024-03-28T18:55:35.964728+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:55:35.964728+0000","last_undegraded":"2024-03-28T18:55:35.964728+0000","last_fullsized":"2024-03-28T18:55:35.964728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.492925+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:55:35.492925+0000","last_peered":"2024-03-28T18:55:35.492925+0000","last_clean":"2024-03-28T18:55:35.492925+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:55:35.492925+0000","last_undegraded":"2024-03-28T18:55:35.492925+0000","last_fullsized":"2024-03-28T18:55:35.492925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565763+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:55:36.565763+0000","last_peered":"2024-03-28T18:55:36.565763+0000","last_clean":"2024-03-28T18:55:36.565763+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:55:36.565763+0000","last_undegraded":"2024-03-28T18:55:36.565763+0000","last_fullsized":"2024-03-28T18:55:36.565763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790580+0000","last_change":"2024-03-28T18:55:34.505741+0000","last_active":"2024-03-28T18:55:36.790580+0000","last_peered":"2024-03-28T18:55:36.790580+0000","last_clean":"2024-03-28T18:55:36.790580+0000","last_became_active":"2024-03-28T18:55:34.505440+0000","last_became_peered":"2024-03-28T18:55:34.505440+0000","last_unstale":"2024-03-28T18:55:36.790580+0000","last_undegraded":"2024-03-28T18:55:36.790580+0000","last_fullsized":"2024-03-28T18:55:36.790580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789593+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:55:36.789593+0000","last_peered":"2024-03-28T18:55:36.789593+0000","last_clean":"2024-03-28T18:55:36.789593+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:55:36.789593+0000","last_undegraded":"2024-03-28T18:55:36.789593+0000","last_fullsized":"2024-03-28T18:55:36.789593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790371+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:55:36.790371+0000","last_peered":"2024-03-28T18:55:36.790371+0000","last_clean":"2024-03-28T18:55:36.790371+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:55:36.790371+0000","last_undegraded":"2024-03-28T18:55:36.790371+0000","last_fullsized":"2024-03-28T18:55:36.790371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789834+0000","last_change":"2024-03-28T18:55:34.491857+0000","last_active":"2024-03-28T18:55:36.789834+0000","last_peered":"2024-03-28T18:55:36.789834+0000","last_clean":"2024-03-28T18:55:36.789834+0000","last_became_active":"2024-03-28T18:55:34.487194+0000","last_became_peered":"2024-03-28T18:55:34.487194+0000","last_unstale":"2024-03-28T18:55:36.789834+0000","last_undegraded":"2024-03-28T18:55:36.789834+0000","last_fullsized":"2024-03-28T18:55:36.789834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.495025+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:55:35.495025+0000","last_peered":"2024-03-28T18:55:35.495025+0000","last_clean":"2024-03-28T18:55:35.495025+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:55:35.495025+0000","last_undegraded":"2024-03-28T18:55:35.495025+0000","last_fullsized":"2024-03-28T18:55:35.495025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.489933+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:55:35.489933+0000","last_peered":"2024-03-28T18:55:35.489933+0000","last_clean":"2024-03-28T18:55:35.489933+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:55:35.489933+0000","last_undegraded":"2024-03-28T18:55:35.489933+0000","last_fullsized":"2024-03-28T18:55:35.489933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790541+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:55:36.790541+0000","last_peered":"2024-03-28T18:55:36.790541+0000","last_clean":"2024-03-28T18:55:36.790541+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:55:36.790541+0000","last_undegraded":"2024-03-28T18:55:36.790541+0000","last_fullsized":"2024-03-28T18:55:36.790541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565773+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:55:36.565773+0000","last_peered":"2024-03-28T18:55:36.565773+0000","last_clean":"2024-03-28T18:55:36.565773+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:55:36.565773+0000","last_undegraded":"2024-03-28T18:55:36.565773+0000","last_fullsized":"2024-03-28T18:55:36.565773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790851+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:55:36.790851+0000","last_peered":"2024-03-28T18:55:36.790851+0000","last_clean":"2024-03-28T18:55:36.790851+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:55:36.790851+0000","last_undegraded":"2024-03-28T18:55:36.790851+0000","last_fullsized":"2024-03-28T18:55:36.790851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.788123+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:55:36.788123+0000","last_peered":"2024-03-28T18:55:36.788123+0000","last_clean":"2024-03-28T18:55:36.788123+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:55:36.788123+0000","last_undegraded":"2024-03-28T18:55:36.788123+0000","last_fullsized":"2024-03-28T18:55:36.788123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490573+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:55:35.490573+0000","last_peered":"2024-03-28T18:55:35.490573+0000","last_clean":"2024-03-28T18:55:35.490573+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:55:35.490573+0000","last_undegraded":"2024-03-28T18:55:35.490573+0000","last_fullsized":"2024-03-28T18:55:35.490573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.494892+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:55:35.494892+0000","last_peered":"2024-03-28T18:55:35.494892+0000","last_clean":"2024-03-28T18:55:35.494892+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:55:35.494892+0000","last_undegraded":"2024-03-28T18:55:35.494892+0000","last_fullsized":"2024-03-28T18:55:35.494892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790199+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:55:36.790199+0000","last_peered":"2024-03-28T18:55:36.790199+0000","last_clean":"2024-03-28T18:55:36.790199+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:55:36.790199+0000","last_undegraded":"2024-03-28T18:55:36.790199+0000","last_fullsized":"2024-03-28T18:55:36.790199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565993+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:55:36.565993+0000","last_peered":"2024-03-28T18:55:36.565993+0000","last_clean":"2024-03-28T18:55:36.565993+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:55:36.565993+0000","last_undegraded":"2024-03-28T18:55:36.565993+0000","last_fullsized":"2024-03-28T18:55:36.565993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.491581+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:55:35.491581+0000","last_peered":"2024-03-28T18:55:35.491581+0000","last_clean":"2024-03-28T18:55:35.491581+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:55:35.491581+0000","last_undegraded":"2024-03-28T18:55:35.491581+0000","last_fullsized":"2024-03-28T18:55:35.491581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789611+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:55:36.789611+0000","last_peered":"2024-03-28T18:55:36.789611+0000","last_clean":"2024-03-28T18:55:36.789611+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:55:36.789611+0000","last_undegraded":"2024-03-28T18:55:36.789611+0000","last_fullsized":"2024-03-28T18:55:36.789611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789441+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:55:36.789441+0000","last_peered":"2024-03-28T18:55:36.789441+0000","last_clean":"2024-03-28T18:55:36.789441+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:55:36.789441+0000","last_undegraded":"2024-03-28T18:55:36.789441+0000","last_fullsized":"2024-03-28T18:55:36.789441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790355+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:55:36.790355+0000","last_peered":"2024-03-28T18:55:36.790355+0000","last_clean":"2024-03-28T18:55:36.790355+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:55:36.790355+0000","last_undegraded":"2024-03-28T18:55:36.790355+0000","last_fullsized":"2024-03-28T18:55:36.790355+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490186+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:55:35.490186+0000","last_peered":"2024-03-28T18:55:35.490186+0000","last_clean":"2024-03-28T18:55:35.490186+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:55:35.490186+0000","last_undegraded":"2024-03-28T18:55:35.490186+0000","last_fullsized":"2024-03-28T18:55:35.490186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493816+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:55:35.493816+0000","last_peered":"2024-03-28T18:55:35.493816+0000","last_clean":"2024-03-28T18:55:35.493816+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:55:35.493816+0000","last_undegraded":"2024-03-28T18:55:35.493816+0000","last_fullsized":"2024-03-28T18:55:35.493816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791935+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:55:36.791935+0000","last_peered":"2024-03-28T18:55:36.791935+0000","last_clean":"2024-03-28T18:55:36.791935+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:55:36.791935+0000","last_undegraded":"2024-03-28T18:55:36.791935+0000","last_fullsized":"2024-03-28T18:55:36.791935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791900+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:55:36.791900+0000","last_peered":"2024-03-28T18:55:36.791900+0000","last_clean":"2024-03-28T18:55:36.791900+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:55:36.791900+0000","last_undegraded":"2024-03-28T18:55:36.791900+0000","last_fullsized":"2024-03-28T18:55:36.791900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.566044+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:55:36.566044+0000","last_peered":"2024-03-28T18:55:36.566044+0000","last_clean":"2024-03-28T18:55:36.566044+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:55:36.566044+0000","last_undegraded":"2024-03-28T18:55:36.566044+0000","last_fullsized":"2024-03-28T18:55:36.566044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490108+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:55:35.490108+0000","last_peered":"2024-03-28T18:55:35.490108+0000","last_clean":"2024-03-28T18:55:35.490108+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:55:35.490108+0000","last_undegraded":"2024-03-28T18:55:35.490108+0000","last_fullsized":"2024-03-28T18:55:35.490108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493717+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:55:35.493717+0000","last_peered":"2024-03-28T18:55:35.493717+0000","last_clean":"2024-03-28T18:55:35.493717+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:55:35.493717+0000","last_undegraded":"2024-03-28T18:55:35.493717+0000","last_fullsized":"2024-03-28T18:55:35.493717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493058+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:55:35.493058+0000","last_peered":"2024-03-28T18:55:35.493058+0000","last_clean":"2024-03-28T18:55:35.493058+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:55:35.493058+0000","last_undegraded":"2024-03-28T18:55:35.493058+0000","last_fullsized":"2024-03-28T18:55:35.493058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964132+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:55:35.964132+0000","last_peered":"2024-03-28T18:55:35.964132+0000","last_clean":"2024-03-28T18:55:35.964132+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:55:35.964132+0000","last_undegraded":"2024-03-28T18:55:35.964132+0000","last_fullsized":"2024-03-28T18:55:35.964132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565887+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:55:36.565887+0000","last_peered":"2024-03-28T18:55:36.565887+0000","last_clean":"2024-03-28T18:55:36.565887+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:55:36.565887+0000","last_undegraded":"2024-03-28T18:55:36.565887+0000","last_fullsized":"2024-03-28T18:55:36.565887+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.566239+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:55:36.566239+0000","last_peered":"2024-03-28T18:55:36.566239+0000","last_clean":"2024-03-28T18:55:36.566239+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:55:36.566239+0000","last_undegraded":"2024-03-28T18:55:36.566239+0000","last_fullsized":"2024-03-28T18:55:36.566239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790840+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:55:36.790840+0000","last_peered":"2024-03-28T18:55:36.790840+0000","last_clean":"2024-03-28T18:55:36.790840+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:55:36.790840+0000","last_undegraded":"2024-03-28T18:55:36.790840+0000","last_fullsized":"2024-03-28T18:55:36.790840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964374+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:55:35.964374+0000","last_peered":"2024-03-28T18:55:35.964374+0000","last_clean":"2024-03-28T18:55:35.964374+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:55:35.964374+0000","last_undegraded":"2024-03-28T18:55:35.964374+0000","last_fullsized":"2024-03-28T18:55:35.964374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490824+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:55:35.490824+0000","last_peered":"2024-03-28T18:55:35.490824+0000","last_clean":"2024-03-28T18:55:35.490824+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:55:35.490824+0000","last_undegraded":"2024-03-28T18:55:35.490824+0000","last_fullsized":"2024-03-28T18:55:35.490824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.491901+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:55:35.491901+0000","last_peered":"2024-03-28T18:55:35.491901+0000","last_clean":"2024-03-28T18:55:35.491901+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:55:35.491901+0000","last_undegraded":"2024-03-28T18:55:35.491901+0000","last_fullsized":"2024-03-28T18:55:35.491901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.787801+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:55:36.787801+0000","last_peered":"2024-03-28T18:55:36.787801+0000","last_clean":"2024-03-28T18:55:36.787801+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:55:36.787801+0000","last_undegraded":"2024-03-28T18:55:36.787801+0000","last_fullsized":"2024-03-28T18:55:36.787801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964128+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:55:35.964128+0000","last_peered":"2024-03-28T18:55:35.964128+0000","last_clean":"2024-03-28T18:55:35.964128+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:55:35.964128+0000","last_undegraded":"2024-03-28T18:55:35.964128+0000","last_fullsized":"2024-03-28T18:55:35.964128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.492912+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:55:35.492912+0000","last_peered":"2024-03-28T18:55:35.492912+0000","last_clean":"2024-03-28T18:55:35.492912+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:55:35.492912+0000","last_undegraded":"2024-03-28T18:55:35.492912+0000","last_fullsized":"2024-03-28T18:55:35.492912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964965+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:55:35.964965+0000","last_peered":"2024-03-28T18:55:35.964965+0000","last_clean":"2024-03-28T18:55:35.964965+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:55:35.964965+0000","last_undegraded":"2024-03-28T18:55:35.964965+0000","last_fullsized":"2024-03-28T18:55:35.964965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.788220+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:55:36.788220+0000","last_peered":"2024-03-28T18:55:36.788220+0000","last_clean":"2024-03-28T18:55:36.788220+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:55:36.788220+0000","last_undegraded":"2024-03-28T18:55:36.788220+0000","last_fullsized":"2024-03-28T18:55:36.788220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.965903+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:55:35.965903+0000","last_peered":"2024-03-28T18:55:35.965903+0000","last_clean":"2024-03-28T18:55:35.965903+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:55:35.965903+0000","last_undegraded":"2024-03-28T18:55:35.965903+0000","last_fullsized":"2024-03-28T18:55:35.965903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490242+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:55:35.490242+0000","last_peered":"2024-03-28T18:55:35.490242+0000","last_clean":"2024-03-28T18:55:35.490242+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:55:35.490242+0000","last_undegraded":"2024-03-28T18:55:35.490242+0000","last_fullsized":"2024-03-28T18:55:35.490242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791310+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:55:36.791310+0000","last_peered":"2024-03-28T18:55:36.791310+0000","last_clean":"2024-03-28T18:55:36.791310+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:55:36.791310+0000","last_undegraded":"2024-03-28T18:55:36.791310+0000","last_fullsized":"2024-03-28T18:55:36.791310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.788428+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:55:36.788428+0000","last_peered":"2024-03-28T18:55:36.788428+0000","last_clean":"2024-03-28T18:55:36.788428+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:55:36.788428+0000","last_undegraded":"2024-03-28T18:55:36.788428+0000","last_fullsized":"2024-03-28T18:55:36.788428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964192+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:55:35.964192+0000","last_peered":"2024-03-28T18:55:35.964192+0000","last_clean":"2024-03-28T18:55:35.964192+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:55:35.964192+0000","last_undegraded":"2024-03-28T18:55:35.964192+0000","last_fullsized":"2024-03-28T18:55:35.964192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963614+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:55:35.963614+0000","last_peered":"2024-03-28T18:55:35.963614+0000","last_clean":"2024-03-28T18:55:35.963614+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:55:35.963614+0000","last_undegraded":"2024-03-28T18:55:35.963614+0000","last_fullsized":"2024-03-28T18:55:35.963614+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790232+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:55:36.790232+0000","last_peered":"2024-03-28T18:55:36.790232+0000","last_clean":"2024-03-28T18:55:36.790232+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:55:36.790232+0000","last_undegraded":"2024-03-28T18:55:36.790232+0000","last_fullsized":"2024-03-28T18:55:36.790232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790187+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:55:36.790187+0000","last_peered":"2024-03-28T18:55:36.790187+0000","last_clean":"2024-03-28T18:55:36.790187+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:55:36.790187+0000","last_undegraded":"2024-03-28T18:55:36.790187+0000","last_fullsized":"2024-03-28T18:55:36.790187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963306+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:55:35.963306+0000","last_peered":"2024-03-28T18:55:35.963306+0000","last_clean":"2024-03-28T18:55:35.963306+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:55:35.963306+0000","last_undegraded":"2024-03-28T18:55:35.963306+0000","last_fullsized":"2024-03-28T18:55:35.963306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565377+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:55:36.565377+0000","last_peered":"2024-03-28T18:55:36.565377+0000","last_clean":"2024-03-28T18:55:36.565377+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:55:36.565377+0000","last_undegraded":"2024-03-28T18:55:36.565377+0000","last_fullsized":"2024-03-28T18:55:36.565377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.787941+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:55:36.787941+0000","last_peered":"2024-03-28T18:55:36.787941+0000","last_clean":"2024-03-28T18:55:36.787941+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:55:36.787941+0000","last_undegraded":"2024-03-28T18:55:36.787941+0000","last_fullsized":"2024-03-28T18:55:36.787941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.489991+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:55:35.489991+0000","last_peered":"2024-03-28T18:55:35.489991+0000","last_clean":"2024-03-28T18:55:35.489991+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:55:35.489991+0000","last_undegraded":"2024-03-28T18:55:35.489991+0000","last_fullsized":"2024-03-28T18:55:35.489991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790972+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:55:36.790972+0000","last_peered":"2024-03-28T18:55:36.790972+0000","last_clean":"2024-03-28T18:55:36.790972+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:55:36.790972+0000","last_undegraded":"2024-03-28T18:55:36.790972+0000","last_fullsized":"2024-03-28T18:55:36.790972+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.494475+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:55:35.494475+0000","last_peered":"2024-03-28T18:55:35.494475+0000","last_clean":"2024-03-28T18:55:35.494475+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:55:35.494475+0000","last_undegraded":"2024-03-28T18:55:35.494475+0000","last_fullsized":"2024-03-28T18:55:35.494475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490060+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:55:35.490060+0000","last_peered":"2024-03-28T18:55:35.490060+0000","last_clean":"2024-03-28T18:55:35.490060+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:55:35.490060+0000","last_undegraded":"2024-03-28T18:55:35.490060+0000","last_fullsized":"2024-03-28T18:55:35.490060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791542+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:55:36.791542+0000","last_peered":"2024-03-28T18:55:36.791542+0000","last_clean":"2024-03-28T18:55:36.791542+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:55:36.791542+0000","last_undegraded":"2024-03-28T18:55:36.791542+0000","last_fullsized":"2024-03-28T18:55:36.791542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964668+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:55:35.964668+0000","last_peered":"2024-03-28T18:55:35.964668+0000","last_clean":"2024-03-28T18:55:35.964668+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:55:35.964668+0000","last_undegraded":"2024-03-28T18:55:35.964668+0000","last_fullsized":"2024-03-28T18:55:35.964668+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490004+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:55:35.490004+0000","last_peered":"2024-03-28T18:55:35.490004+0000","last_clean":"2024-03-28T18:55:35.490004+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:55:35.490004+0000","last_undegraded":"2024-03-28T18:55:35.490004+0000","last_fullsized":"2024-03-28T18:55:35.490004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490476+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:55:35.490476+0000","last_peered":"2024-03-28T18:55:35.490476+0000","last_clean":"2024-03-28T18:55:35.490476+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:55:35.490476+0000","last_undegraded":"2024-03-28T18:55:35.490476+0000","last_fullsized":"2024-03-28T18:55:35.490476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790042+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:55:36.790042+0000","last_peered":"2024-03-28T18:55:36.790042+0000","last_clean":"2024-03-28T18:55:36.790042+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:55:36.790042+0000","last_undegraded":"2024-03-28T18:55:36.790042+0000","last_fullsized":"2024-03-28T18:55:36.790042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.494052+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:55:35.494052+0000","last_peered":"2024-03-28T18:55:35.494052+0000","last_clean":"2024-03-28T18:55:35.494052+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:55:35.494052+0000","last_undegraded":"2024-03-28T18:55:35.494052+0000","last_fullsized":"2024-03-28T18:55:35.494052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.565921+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:55:36.565921+0000","last_peered":"2024-03-28T18:55:36.565921+0000","last_clean":"2024-03-28T18:55:36.565921+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:55:36.565921+0000","last_undegraded":"2024-03-28T18:55:36.565921+0000","last_fullsized":"2024-03-28T18:55:36.565921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963305+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:55:35.963305+0000","last_peered":"2024-03-28T18:55:35.963305+0000","last_clean":"2024-03-28T18:55:35.963305+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:55:35.963305+0000","last_undegraded":"2024-03-28T18:55:35.963305+0000","last_fullsized":"2024-03-28T18:55:35.963305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.493039+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:55:35.493039+0000","last_peered":"2024-03-28T18:55:35.493039+0000","last_clean":"2024-03-28T18:55:35.493039+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:55:35.493039+0000","last_undegraded":"2024-03-28T18:55:35.493039+0000","last_fullsized":"2024-03-28T18:55:35.493039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.490351+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:55:35.490351+0000","last_peered":"2024-03-28T18:55:35.490351+0000","last_clean":"2024-03-28T18:55:35.490351+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:55:35.490351+0000","last_undegraded":"2024-03-28T18:55:35.490351+0000","last_fullsized":"2024-03-28T18:55:35.490351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789628+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:55:36.789628+0000","last_peered":"2024-03-28T18:55:36.789628+0000","last_clean":"2024-03-28T18:55:36.789628+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:55:36.789628+0000","last_undegraded":"2024-03-28T18:55:36.789628+0000","last_fullsized":"2024-03-28T18:55:36.789628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.489767+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:55:35.489767+0000","last_peered":"2024-03-28T18:55:35.489767+0000","last_clean":"2024-03-28T18:55:35.489767+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:55:35.489767+0000","last_undegraded":"2024-03-28T18:55:35.489767+0000","last_fullsized":"2024-03-28T18:55:35.489767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790207+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:55:36.790207+0000","last_peered":"2024-03-28T18:55:36.790207+0000","last_clean":"2024-03-28T18:55:36.790207+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:55:36.790207+0000","last_undegraded":"2024-03-28T18:55:36.790207+0000","last_fullsized":"2024-03-28T18:55:36.790207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964242+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:55:35.964242+0000","last_peered":"2024-03-28T18:55:35.964242+0000","last_clean":"2024-03-28T18:55:35.964242+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:55:35.964242+0000","last_undegraded":"2024-03-28T18:55:35.964242+0000","last_fullsized":"2024-03-28T18:55:35.964242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.491406+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:55:35.491406+0000","last_peered":"2024-03-28T18:55:35.491406+0000","last_clean":"2024-03-28T18:55:35.491406+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:55:35.491406+0000","last_undegraded":"2024-03-28T18:55:35.491406+0000","last_fullsized":"2024-03-28T18:55:35.491406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791124+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:55:36.791124+0000","last_peered":"2024-03-28T18:55:36.791124+0000","last_clean":"2024-03-28T18:55:36.791124+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:55:36.791124+0000","last_undegraded":"2024-03-28T18:55:36.791124+0000","last_fullsized":"2024-03-28T18:55:36.791124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.963924+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:55:35.963924+0000","last_peered":"2024-03-28T18:55:35.963924+0000","last_clean":"2024-03-28T18:55:35.963924+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:55:35.963924+0000","last_undegraded":"2024-03-28T18:55:35.963924+0000","last_fullsized":"2024-03-28T18:55:35.963924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.791306+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:55:36.791306+0000","last_peered":"2024-03-28T18:55:36.791306+0000","last_clean":"2024-03-28T18:55:36.791306+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:55:36.791306+0000","last_undegraded":"2024-03-28T18:55:36.791306+0000","last_fullsized":"2024-03-28T18:55:36.791306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:35.964371+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:55:35.964371+0000","last_peered":"2024-03-28T18:55:35.964371+0000","last_clean":"2024-03-28T18:55:35.964371+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:55:35.964371+0000","last_undegraded":"2024-03-28T18:55:35.964371+0000","last_fullsized":"2024-03-28T18:55:35.964371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.566006+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:55:36.566006+0000","last_peered":"2024-03-28T18:55:36.566006+0000","last_clean":"2024-03-28T18:55:36.566006+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:55:36.566006+0000","last_undegraded":"2024-03-28T18:55:36.566006+0000","last_fullsized":"2024-03-28T18:55:36.566006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.790187+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:55:36.790187+0000","last_peered":"2024-03-28T18:55:36.790187+0000","last_clean":"2024-03-28T18:55:36.790187+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:55:36.790187+0000","last_undegraded":"2024-03-28T18:55:36.790187+0000","last_fullsized":"2024-03-28T18:55:36.790187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.789987+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:55:36.789987+0000","last_peered":"2024-03-28T18:55:36.789987+0000","last_clean":"2024-03-28T18:55:36.789987+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:55:36.789987+0000","last_undegraded":"2024-03-28T18:55:36.789987+0000","last_fullsized":"2024-03-28T18:55:36.789987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-28T18:55:36.566127+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:55:36.566127+0000","last_peered":"2024-03-28T18:55:36.566127+0000","last_clean":"2024-03-28T18:55:36.566127+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:55:36.566127+0000","last_undegraded":"2024-03-28T18:55:36.566127+0000","last_fullsized":"2024-03-28T18:55:36.566127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":70944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:55:40.527 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T18:55:40.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:40.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.793+0000 7fae5e800640 1 Processor -- start 2024-03-28T18:55:40.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.794+0000 7fae5e800640 1 -- start start 2024-03-28T18:55:40.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.795+0000 7fae5e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 0x7fae580a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.795+0000 7fae5e800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae580c3af0 con 0x7fae581534c0 2024-03-28T18:55:40.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.795+0000 7fae5e800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae580c3d90 con 0x7fae580c3690 2024-03-28T18:55:40.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.795+0000 7fae5e800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae580ab4e0 con 0x7fae58150c70 2024-03-28T18:55:40.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae57e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fae581534c0 0x7fae58153950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34048/0 (socket says 172.21.15.27:34048) 2024-03-28T18:55:40.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae57e00640 1 -- 172.21.15.27:0/1586702805 learned_addr learned my addr 172.21.15.27:0/1586702805 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:40.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae57400640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 0x7fae580a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730242445 0 0) 0x7fae580c3af0 con 0x7fae581534c0 2024-03-28T18:55:40.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c003520 con 0x7fae581534c0 2024-03-28T18:55:40.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 759492752 0 0) 0x7fae3c003520 con 0x7fae581534c0 2024-03-28T18:55:40.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 msgr2=0x7fae580a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:40.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae56000640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 0x7fae580a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 >> v1:172.21.15.155:6789/0 conn(0x7fae580c3690 legacy=0x7fae580c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580ab710 con 0x7fae581534c0 2024-03-28T18:55:40.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.796+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fae500019f0 con 0x7fae581534c0 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 >> v1:172.21.15.27:6789/0 conn(0x7fae581534c0 legacy=0x7fae58153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56000640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56000640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 0x7fae3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56000640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae3c004d40 0x7fae3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c00bb70 con 0x7fae3c0039e0 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c00bd20 con 0x7fae3c004d40 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c00bf20 con 0x7fae58150c70 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae57400640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56a00640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 0x7fae3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae57400640 1 -- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 msgr2=0x7fae3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae57e00640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae3c004d40 0x7fae3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae57400640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 0x7fae3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae57400640 1 -- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae3c004d40 msgr2=0x7fae3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae57400640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae3c004d40 0x7fae3c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae57400640 1 -- 172.21.15.27:0/1586702805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c003520 con 0x7fae3c0039e0 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.797+0000 7fae56a00640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae58150c70 0x7fae3c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae57400640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae3c003e00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae4800ba30 tx=0x7fae3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd9c244055cbbf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48036020 con 0x7fae3c0039e0 2024-03-28T18:55:40.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae48032630 con 0x7fae3c0039e0 2024-03-28T18:55:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae56000640 1 -- 172.21.15.27:0/1586702805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4800b600 con 0x7fae3c0039e0 2024-03-28T18:55:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae5e800640 1 -- 172.21.15.27:0/1586702805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 msgr2=0x7fae3c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae5e800640 1 --2- 172.21.15.27:0/1586702805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae3c003e00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae4800ba30 tx=0x7fae3c005830 comp rx=0 tx=0).stop 2024-03-28T18:55:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae57e00640 1 -- 172.21.15.27:0/1586702805 reap_dead start 2024-03-28T18:55:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae5e800640 1 -- 172.21.15.27:0/1586702805 shutdown_connections 2024-03-28T18:55:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.799+0000 7fae5e800640 1 -- 172.21.15.27:0/1586702805 >> 172.21.15.27:0/1586702805 conn(0x7fae5806b9d0 msgr2=0x7fae580c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.800+0000 7fae5e800640 1 -- 172.21.15.27:0/1586702805 shutdown_connections 2024-03-28T18:55:40.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.800+0000 7fae5e800640 1 -- 172.21.15.27:0/1586702805 wait complete. 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.801+0000 7fae5e800640 1 Processor -- start 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.801+0000 7fae5e800640 1 -- start start 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.802+0000 7fae5e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae581ec4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.802+0000 7fae5e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae581eca30 0x7fae5813dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.802+0000 7fae5e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae5814c800 0x7fae5813e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.802+0000 7fae5e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae58154060 con 0x7fae3c0039e0 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.802+0000 7fae5e800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae580ac1f0 con 0x7fae581eca30 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.802+0000 7fae5e800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae580abcd0 con 0x7fae5814c800 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae581ec4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae581ec4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53256/0 (socket says 172.21.15.27:53256) 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57400640 1 -- 172.21.15.27:0/809426941 learned_addr learned my addr 172.21.15.27:0/809426941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae56a00640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae581eca30 0x7fae5813dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57e00640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae5814c800 0x7fae5813e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57400640 1 -- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae5814c800 msgr2=0x7fae5813e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57400640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae5814c800 0x7fae5813e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57400640 1 -- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae581eca30 msgr2=0x7fae5813dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57400640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae581eca30 0x7fae5813dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae57400640 1 -- 172.21.15.27:0/809426941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5814c450 con 0x7fae3c0039e0 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.803+0000 7fae56a00640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae581eca30 0x7fae5813dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.804+0000 7fae57400640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae581ec4f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae4800bb50 tx=0x7fae48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:40.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.804+0000 7fae4f400640 1 -- 172.21.15.27:0/809426941 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48041070 con 0x7fae3c0039e0 2024-03-28T18:55:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.804+0000 7fae4f400640 1 -- 172.21.15.27:0/809426941 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae48004020 con 0x7fae3c0039e0 2024-03-28T18:55:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.804+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae581ecee0 con 0x7fae3c0039e0 2024-03-28T18:55:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.804+0000 7fae4f400640 1 -- 172.21.15.27:0/809426941 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48036040 con 0x7fae3c0039e0 2024-03-28T18:55:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.804+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae581dd090 con 0x7fae3c0039e0 2024-03-28T18:55:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.805+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5804f400 con 0x7fae3c0039e0 2024-03-28T18:55:40.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.806+0000 7fae4f400640 1 -- 172.21.15.27:0/809426941 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fae48039020 con 0x7fae3c0039e0 2024-03-28T18:55:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.807+0000 7fae4f400640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae3806bbd0 0x7fae3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.808+0000 7fae4f400640 1 -- 172.21.15.27:0/809426941 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7fae480acc30 con 0x7fae3c0039e0 2024-03-28T18:55:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.808+0000 7fae56a00640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae3806bbd0 0x7fae3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:40.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.808+0000 7fae56a00640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae3806bbd0 0x7fae3806e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fae581ece10 tx=0x7fae44009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:40.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:40.812+0000 7fae4f400640 1 -- 172.21.15.27:0/809426941 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fae4807b720 con 0x7fae3c0039e0 2024-03-28T18:55:41.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.018+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae580c5f00 con 0x7fae3c0039e0 2024-03-28T18:55:41.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.021+0000 7fae4f400640 1 -- 172.21.15.27:0/809426941 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16549 (secure 0 0 0) 0x7fae4807b020 con 0x7fae3c0039e0 2024-03-28T18:55:41.021 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:41.022 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:35.475849+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:41.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae3806bbd0 msgr2=0x7fae3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae3806bbd0 0x7fae3806e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fae581ece10 tx=0x7fae44009040 comp rx=0 tx=0).stop 2024-03-28T18:55:41.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 msgr2=0x7fae581ec4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae581ec4f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fae4800bb50 tx=0x7fae48004730 comp rx=0 tx=0).stop 2024-03-28T18:55:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 shutdown_connections 2024-03-28T18:55:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae3806bbd0 0x7fae3806e090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae5814c800 0x7fae5813e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae581eca30 0x7fae5813dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 --2- 172.21.15.27:0/809426941 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae3c0039e0 0x7fae581ec4f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 >> 172.21.15.27:0/809426941 conn(0x7fae5806b9d0 msgr2=0x7fae580c2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 shutdown_connections 2024-03-28T18:55:41.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.024+0000 7fae5e800640 1 -- 172.21.15.27:0/809426941 wait complete. 2024-03-28T18:55:41.044 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-03-28T18:55:32.332283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-03-28T18:55:32.983411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:55:41.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:55:41.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.278+0000 7f2f07000640 1 Processor -- start 2024-03-28T18:55:41.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.278+0000 7f2f07000640 1 -- start start 2024-03-28T18:55:41.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.279+0000 7f2f07000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f0006e960 0x7f2f001540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.279+0000 7f2f07000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f000c17a0 0x7f2f000ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.279+0000 7f2f07000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0006dbe0 con 0x7f2f0014d4e0 2024-03-28T18:55:41.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.279+0000 7f2f07000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0006de80 con 0x7f2f000c17a0 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.279+0000 7f2f07000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0006e160 con 0x7f2f0006e960 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.280+0000 7f2effe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2f0014d4e0 0x7f2f000a9400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34054/0 (socket says 172.21.15.27:34054) 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.280+0000 7f2effe00640 1 -- 172.21.15.27:0/698949575 learned_addr learned my addr 172.21.15.27:0/698949575 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.280+0000 7f2f05000640 1 --2- 172.21.15.27:0/698949575 >> v2:172.21.15.27:3300/0 conn(0x7f2f000c17a0 0x7f2f000ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.280+0000 7f2eff400640 1 --2- 172.21.15.27:0/698949575 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f0006e960 0x7f2f001540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.280+0000 7f2efea00640 1 -- 172.21.15.27:0/698949575 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904737470 0 0) 0x7f2f0006dbe0 con 0x7f2f0014d4e0 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.280+0000 7f2efea00640 1 -- 172.21.15.27:0/698949575 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee4003520 con 0x7f2f0014d4e0 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.280+0000 7f2f05000640 1 -- 172.21.15.27:0/698949575 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f0006e960 msgr2=0x7f2f001540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.280+0000 7f2f05000640 1 --2- 172.21.15.27:0/698949575 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f0006e960 0x7f2f001540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.281+0000 7f2f05000640 1 -- 172.21.15.27:0/698949575 >> v1:172.21.15.27:6789/0 conn(0x7f2f0014d4e0 legacy=0x7f2f000a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.281+0000 7f2f05000640 1 -- 172.21.15.27:0/698949575 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0006e440 con 0x7f2f000c17a0 2024-03-28T18:55:41.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.282+0000 7f2f05000640 1 --2- 172.21.15.27:0/698949575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f000c17a0 0x7f2f000ab270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80097a0 tx=0x7f2ef802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ef02e3a793921c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:41.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.282+0000 7f2efea00640 1 -- 172.21.15.27:0/698949575 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8035020 con 0x7f2f000c17a0 2024-03-28T18:55:41.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.282+0000 7f2efea00640 1 -- 172.21.15.27:0/698949575 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef8031810 con 0x7f2f000c17a0 2024-03-28T18:55:41.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.282+0000 7f2efea00640 1 -- 172.21.15.27:0/698949575 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8031bc0 con 0x7f2f000c17a0 2024-03-28T18:55:41.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.283+0000 7f2f07000640 1 -- 172.21.15.27:0/698949575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f000c17a0 msgr2=0x7f2f000ab270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.283+0000 7f2f07000640 1 --2- 172.21.15.27:0/698949575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f000c17a0 0x7f2f000ab270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80097a0 tx=0x7f2ef802b8b0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.283+0000 7f2f07000640 1 -- 172.21.15.27:0/698949575 shutdown_connections 2024-03-28T18:55:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.283+0000 7f2f07000640 1 --2- 172.21.15.27:0/698949575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f000c17a0 0x7f2f000ab270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.283+0000 7f2f07000640 1 --2- 172.21.15.27:0/698949575 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f0006e960 0x7f2f001540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.283+0000 7f2f07000640 1 -- 172.21.15.27:0/698949575 >> 172.21.15.27:0/698949575 conn(0x7f2f0006ca00 msgr2=0x7f2f0006cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.283+0000 7f2f07000640 1 -- 172.21.15.27:0/698949575 shutdown_connections 2024-03-28T18:55:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.283+0000 7f2f07000640 1 -- 172.21.15.27:0/698949575 wait complete. 2024-03-28T18:55:41.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.285+0000 7f2f07000640 1 Processor -- start 2024-03-28T18:55:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.285+0000 7f2f07000640 1 -- start start 2024-03-28T18:55:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f07000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0006e960 0x7f2f001ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f07000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f000c17a0 0x7f2f001eb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f07000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f0014d4e0 0x7f2f00148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f07000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000a9b10 con 0x7f2f0014d4e0 2024-03-28T18:55:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f07000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000c1f10 con 0x7f2f0006e960 2024-03-28T18:55:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f07000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000a9c90 con 0x7f2f000c17a0 2024-03-28T18:55:41.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f05000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f0014d4e0 0x7f2f00148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f05000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f0014d4e0 0x7f2f00148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53276/0 (socket says 172.21.15.27:53276) 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f05000640 1 -- 172.21.15.27:0/2657472439 learned_addr learned my addr 172.21.15.27:0/2657472439 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2eff400640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f000c17a0 0x7f2f001eb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2effe00640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0006e960 0x7f2f001ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f05000640 1 -- 172.21.15.27:0/2657472439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f000c17a0 msgr2=0x7f2f001eb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f05000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f000c17a0 0x7f2f001eb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f05000640 1 -- 172.21.15.27:0/2657472439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0006e960 msgr2=0x7f2f001ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.286+0000 7f2f05000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0006e960 0x7f2f001ee9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.287+0000 7f2f05000640 1 -- 172.21.15.27:0/2657472439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00154db0 con 0x7f2f0014d4e0 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.287+0000 7f2f05000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f0014d4e0 0x7f2f00148210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2ef802b8f0 tx=0x7f2ef80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.287+0000 7f2ef7e00640 1 -- 172.21.15.27:0/2657472439 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8041070 con 0x7f2f0014d4e0 2024-03-28T18:55:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.287+0000 7f2ef7e00640 1 -- 172.21.15.27:0/2657472439 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef8031e40 con 0x7f2f0014d4e0 2024-03-28T18:55:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.287+0000 7f2ef7e00640 1 -- 172.21.15.27:0/2657472439 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8035040 con 0x7f2f0014d4e0 2024-03-28T18:55:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.287+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f001edf00 con 0x7f2f0014d4e0 2024-03-28T18:55:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.287+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f00148750 con 0x7f2f0014d4e0 2024-03-28T18:55:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.288+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0004f470 con 0x7f2f0014d4e0 2024-03-28T18:55:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.289+0000 7f2ef7e00640 1 -- 172.21.15.27:0/2657472439 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2ef8038020 con 0x7f2f0014d4e0 2024-03-28T18:55:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.290+0000 7f2ef7e00640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2edc06bbd0 0x7f2edc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.290+0000 7f2ef7e00640 1 -- 172.21.15.27:0/2657472439 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f2ef80ac5d0 con 0x7f2f0014d4e0 2024-03-28T18:55:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.291+0000 7f2effe00640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2edc06bbd0 0x7f2edc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.291+0000 7f2effe00640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2edc06bbd0 0x7f2edc06e090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2f000c69f0 tx=0x7f2ef0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.295+0000 7f2ef7e00640 1 -- 172.21.15.27:0/2657472439 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2ef807b030 con 0x7f2f0014d4e0 2024-03-28T18:55:41.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.496+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f0006ed90 con 0x7f2f0014d4e0 2024-03-28T18:55:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.496+0000 7f2ef7e00640 1 -- 172.21.15.27:0/2657472439 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ef807a930 con 0x7f2f0014d4e0 2024-03-28T18:55:41.496 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:55:41.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.499+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2edc06bbd0 msgr2=0x7f2edc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.499+0000 7f2f07000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2edc06bbd0 0x7f2edc06e090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2f000c69f0 tx=0x7f2ef0009040 comp rx=0 tx=0).stop 2024-03-28T18:55:41.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f0014d4e0 msgr2=0x7f2f00148210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f0014d4e0 0x7f2f00148210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2ef802b8f0 tx=0x7f2ef80046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 shutdown_connections 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2edc06bbd0 0x7f2edc06e090 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f0014d4e0 0x7f2f00148210 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f000c17a0 0x7f2f001eb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 --2- 172.21.15.27:0/2657472439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0006e960 0x7f2f001ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 >> 172.21.15.27:0/2657472439 conn(0x7f2f0006ca00 msgr2=0x7f2f0014c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 shutdown_connections 2024-03-28T18:55:41.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.500+0000 7f2f07000640 1 -- 172.21.15.27:0/2657472439 wait complete. 2024-03-28T18:55:41.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-03-28T18:55:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.744+0000 7f4049000640 1 Processor -- start 2024-03-28T18:55:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.744+0000 7f4049000640 1 -- start start 2024-03-28T18:55:41.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.745+0000 7f4049000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f404414d4e0 0x7f40441540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.745+0000 7f4049000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f404406e960 0x7f40440c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.745+0000 7f4049000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f40440a9400 0x7f40440a9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.745+0000 7f4049000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40440c1ce0 con 0x7f404406e960 2024-03-28T18:55:41.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.745+0000 7f4049000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404406dbe0 con 0x7f404414d4e0 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.745+0000 7f4049000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404406de80 con 0x7f40440a9400 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f404406e960 0x7f40440c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f404406e960 0x7f40440c17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53290/0 (socket says 172.21.15.27:53290) 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041400640 1 -- 172.21.15.27:0/1120702589 learned_addr learned my addr 172.21.15.27:0/1120702589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041e00640 1 --2- 172.21.15.27:0/1120702589 >> v2:172.21.15.155:3300/0 conn(0x7f404414d4e0 0x7f40441540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4042800640 1 --2- 172.21.15.27:0/1120702589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f40440a9400 0x7f40440a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041400640 1 -- 172.21.15.27:0/1120702589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f40440a9400 msgr2=0x7f40440a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041400640 1 --2- 172.21.15.27:0/1120702589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f40440a9400 0x7f40440a9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041400640 1 -- 172.21.15.27:0/1120702589 >> v2:172.21.15.155:3300/0 conn(0x7f404414d4e0 msgr2=0x7f40441540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041400640 1 --2- 172.21.15.27:0/1120702589 >> v2:172.21.15.155:3300/0 conn(0x7f404414d4e0 0x7f40441540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4041400640 1 -- 172.21.15.27:0/1120702589 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404406e160 con 0x7f404406e960 2024-03-28T18:55:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.746+0000 7f4042800640 1 --2- 172.21.15.27:0/1120702589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f40440a9400 0x7f40440a9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.748+0000 7f4041400640 1 --2- 172.21.15.27:0/1120702589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404406e960 0x7f40440c17a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4044072d10 tx=0x7f403002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ff26b38091f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.748+0000 7f4040a00640 1 -- 172.21.15.27:0/1120702589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030035020 con 0x7f404406e960 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.748+0000 7f4040a00640 1 -- 172.21.15.27:0/1120702589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030031870 con 0x7f404406e960 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.748+0000 7f4040a00640 1 -- 172.21.15.27:0/1120702589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030031c40 con 0x7f404406e960 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 -- 172.21.15.27:0/1120702589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404406e960 msgr2=0x7f40440c17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 --2- 172.21.15.27:0/1120702589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404406e960 0x7f40440c17a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4044072d10 tx=0x7f403002b910 comp rx=0 tx=0).stop 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 -- 172.21.15.27:0/1120702589 shutdown_connections 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 --2- 172.21.15.27:0/1120702589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f40440a9400 0x7f40440a9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 --2- 172.21.15.27:0/1120702589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f404406e960 0x7f40440c17a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 --2- 172.21.15.27:0/1120702589 >> v2:172.21.15.155:3300/0 conn(0x7f404414d4e0 0x7f40441540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 -- 172.21.15.27:0/1120702589 >> 172.21.15.27:0/1120702589 conn(0x7f404406ca00 msgr2=0x7f404406cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 -- 172.21.15.27:0/1120702589 shutdown_connections 2024-03-28T18:55:41.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.749+0000 7f4049000640 1 -- 172.21.15.27:0/1120702589 wait complete. 2024-03-28T18:55:41.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.751+0000 7f4049000640 1 Processor -- start 2024-03-28T18:55:41.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.751+0000 7f4049000640 1 -- start start 2024-03-28T18:55:41.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4049000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f404406e960 0x7f40440c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4049000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40440a9400 0x7f4044153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4049000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f404414d4e0 0x7f4044148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4049000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404414d8c0 con 0x7f40440a9400 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4049000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40440c1f10 con 0x7f404414d4e0 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4049000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404414da40 con 0x7f404406e960 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40440a9400 0x7f4044153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40440a9400 0x7f4044153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53292/0 (socket says 172.21.15.27:53292) 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041400640 1 -- 172.21.15.27:0/1195967252 learned_addr learned my addr 172.21.15.27:0/1195967252 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041e00640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f404406e960 0x7f40440c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4042800640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f404414d4e0 0x7f4044148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041400640 1 -- 172.21.15.27:0/1195967252 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f404406e960 msgr2=0x7f40440c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041400640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f404406e960 0x7f40440c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041400640 1 -- 172.21.15.27:0/1195967252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f404414d4e0 msgr2=0x7f4044148b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041400640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f404414d4e0 0x7f4044148b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.752+0000 7f4041400640 1 -- 172.21.15.27:0/1195967252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40441ee890 con 0x7f40440a9400 2024-03-28T18:55:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.753+0000 7f4041400640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40440a9400 0x7f4044153100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f403002b950 tx=0x7f40300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:41.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.753+0000 7f403a000640 1 -- 172.21.15.27:0/1195967252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030040070 con 0x7f40440a9400 2024-03-28T18:55:41.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.753+0000 7f403a000640 1 -- 172.21.15.27:0/1195967252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030030d00 con 0x7f40440a9400 2024-03-28T18:55:41.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.753+0000 7f403a000640 1 -- 172.21.15.27:0/1195967252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030035040 con 0x7f40440a9400 2024-03-28T18:55:41.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.753+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40441eeb70 con 0x7f40440a9400 2024-03-28T18:55:41.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.753+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40441ef130 con 0x7f40440a9400 2024-03-28T18:55:41.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.754+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404414d020 con 0x7f40440a9400 2024-03-28T18:55:41.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.755+0000 7f403a000640 1 -- 172.21.15.27:0/1195967252 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f403004b060 con 0x7f40440a9400 2024-03-28T18:55:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.756+0000 7f403a000640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f402406bc00 0x7f402406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.756+0000 7f403a000640 1 -- 172.21.15.27:0/1195967252 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f40300ac810 con 0x7f40440a9400 2024-03-28T18:55:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.757+0000 7f4041e00640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f402406bc00 0x7f402406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:41.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.757+0000 7f4041e00640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f402406bc00 0x7f402406e0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4034006fd0 tx=0x7f4034008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:41.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.761+0000 7f403a000640 1 -- 172.21.15.27:0/1195967252 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f403007b300 con 0x7f40440a9400 2024-03-28T18:55:41.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.961+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f404404f470 con 0x7f40440a9400 2024-03-28T18:55:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.961+0000 7f403a000640 1 -- 172.21.15.27:0/1195967252 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v20) v1 ==== 99+0+11 (secure 0 0 0) 0x7f403007ac00 con 0x7f40440a9400 2024-03-28T18:55:41.961 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:55:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f402406bc00 msgr2=0x7f402406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f402406bc00 0x7f402406e0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4034006fd0 tx=0x7f4034008040 comp rx=0 tx=0).stop 2024-03-28T18:55:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40440a9400 msgr2=0x7f4044153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40440a9400 0x7f4044153100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f403002b950 tx=0x7f40300046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 shutdown_connections 2024-03-28T18:55:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f402406bc00 0x7f402406e0c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f404414d4e0 0x7f4044148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40440a9400 0x7f4044153100 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 --2- 172.21.15.27:0/1195967252 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f404406e960 0x7f40440c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 >> 172.21.15.27:0/1195967252 conn(0x7f404406ca00 msgr2=0x7f4044145e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.964+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 shutdown_connections 2024-03-28T18:55:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:41.965+0000 7f4049000640 1 -- 172.21.15.27:0/1195967252 wait complete. 2024-03-28T18:55:41.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-03-28T18:55:42.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.214+0000 7fb756400640 1 Processor -- start 2024-03-28T18:55:42.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.214+0000 7fb756400640 1 -- start start 2024-03-28T18:55:42.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.215+0000 7fb756400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb7500a5c60 0x7fb7500a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.215+0000 7fb756400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb750154250 con 0x7fb7501508c0 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.215+0000 7fb756400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb750073500 con 0x7fb7500a66e0 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.215+0000 7fb756400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb750153dc0 con 0x7fb7500a5c60 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.216+0000 7fb755000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb7501508c0 0x7fb750072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34064/0 (socket says 172.21.15.27:34064) 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.216+0000 7fb755000640 1 -- 172.21.15.27:0/2006959407 learned_addr learned my addr 172.21.15.27:0/2006959407 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.216+0000 7fb74fe00640 1 --2- 172.21.15.27:0/2006959407 >> v2:172.21.15.155:3300/0 conn(0x7fb7500a5c60 0x7fb7500a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.216+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454427270 0 0) 0x7fb750154250 con 0x7fb7501508c0 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.216+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb738003520 con 0x7fb7501508c0 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400929468 0 0) 0x7fb738003520 con 0x7fb7501508c0 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 >> v2:172.21.15.155:3300/0 conn(0x7fb7500a5c60 msgr2=0x7fb7500a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 --2- 172.21.15.27:0/2006959407 >> v2:172.21.15.155:3300/0 conn(0x7fb7500a5c60 0x7fb7500a60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 >> v1:172.21.15.155:6789/0 conn(0x7fb7500a66e0 legacy=0x7fb7500a6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7501534d0 con 0x7fb7501508c0 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fb744001670 con 0x7fb7501508c0 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74fe00640 1 --2- 172.21.15.27:0/2006959407 >> v2:172.21.15.155:3300/0 conn(0x7fb7500a5c60 0x7fb7500a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 >> v1:172.21.15.27:6789/0 conn(0x7fb7501508c0 legacy=0x7fb750072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380039b0 0x7fb738003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb738004800 0x7fb738004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380053c0 0x7fb73800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb750153dc0 con 0x7fb7380039b0 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73800c250 con 0x7fb7380053c0 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73800c450 con 0x7fb738004800 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb755000640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380039b0 0x7fb738003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb755a00640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380053c0 0x7fb73800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb755000640 1 -- 172.21.15.27:0/2006959407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb738004800 msgr2=0x7fb738004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb755000640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb738004800 0x7fb738004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb755000640 1 -- 172.21.15.27:0/2006959407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380053c0 msgr2=0x7fb73800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.217+0000 7fb755000640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380053c0 0x7fb73800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.218+0000 7fb755000640 1 -- 172.21.15.27:0/2006959407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb738003520 con 0x7fb7380039b0 2024-03-28T18:55:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.218+0000 7fb755a00640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380053c0 0x7fb73800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:42.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.218+0000 7fb755a00640 1 -- 172.21.15.27:0/2006959407 reap_dead start 2024-03-28T18:55:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.219+0000 7fb755000640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380039b0 0x7fb738003dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb74400c240 tx=0x7fb738005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1246c9f501cf1438 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.219+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744007d10 con 0x7fb7380039b0 2024-03-28T18:55:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.219+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb744030070 con 0x7fb7380039b0 2024-03-28T18:55:42.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.219+0000 7fb74f400640 1 -- 172.21.15.27:0/2006959407 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74402cda0 con 0x7fb7380039b0 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.220+0000 7fb756400640 1 -- 172.21.15.27:0/2006959407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380039b0 msgr2=0x7fb738003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.220+0000 7fb756400640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380039b0 0x7fb738003dd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb74400c240 tx=0x7fb738005830 comp rx=0 tx=0).stop 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.220+0000 7fb756400640 1 -- 172.21.15.27:0/2006959407 shutdown_connections 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.220+0000 7fb756400640 1 --2- 172.21.15.27:0/2006959407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380039b0 0x7fb738003dd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.220+0000 7fb756400640 1 -- 172.21.15.27:0/2006959407 >> 172.21.15.27:0/2006959407 conn(0x7fb7500a88b0 msgr2=0x7fb7500a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.220+0000 7fb756400640 1 -- 172.21.15.27:0/2006959407 shutdown_connections 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.220+0000 7fb756400640 1 -- 172.21.15.27:0/2006959407 wait complete. 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.222+0000 7fb756400640 1 Processor -- start 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.222+0000 7fb756400640 1 -- start start 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb756400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380039b0 0x7fb7500c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb756400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb750149c50 0x7fb75014a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb756400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7501ee390 0x7fb7501ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb756400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7500a8550 con 0x7fb7501ee390 2024-03-28T18:55:42.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb756400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb750153a50 con 0x7fb7380039b0 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb756400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb750150ca0 con 0x7fb750149c50 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb755a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7501ee390 0x7fb7501ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb755a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7501ee390 0x7fb7501ee810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53302/0 (socket says 172.21.15.27:53302) 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.223+0000 7fb755a00640 1 -- 172.21.15.27:0/52630511 learned_addr learned my addr 172.21.15.27:0/52630511 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb755000640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380039b0 0x7fb7500c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb74fe00640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb750149c50 0x7fb75014a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb755a00640 1 -- 172.21.15.27:0/52630511 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb750149c50 msgr2=0x7fb75014a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb755a00640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb750149c50 0x7fb75014a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb755a00640 1 -- 172.21.15.27:0/52630511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380039b0 msgr2=0x7fb7500c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb755a00640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380039b0 0x7fb7500c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb755a00640 1 -- 172.21.15.27:0/52630511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7501eed50 con 0x7fb7501ee390 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb755a00640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7501ee390 0x7fb7501ee810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb75006c070 tx=0x7fb74800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:42.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb74cc00640 1 -- 172.21.15.27:0/52630511 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb748018070 con 0x7fb7501ee390 2024-03-28T18:55:42.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb74cc00640 1 -- 172.21.15.27:0/52630511 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb748003da0 con 0x7fb7501ee390 2024-03-28T18:55:42.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7501574d0 con 0x7fb7501ee390 2024-03-28T18:55:42.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.224+0000 7fb74cc00640 1 -- 172.21.15.27:0/52630511 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7480052b0 con 0x7fb7501ee390 2024-03-28T18:55:42.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.225+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7501551f0 con 0x7fb7501ee390 2024-03-28T18:55:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.226+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7501577b0 con 0x7fb7501ee390 2024-03-28T18:55:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.227+0000 7fb74cc00640 1 -- 172.21.15.27:0/52630511 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb74801b070 con 0x7fb7501ee390 2024-03-28T18:55:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.228+0000 7fb74cc00640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb73006bbd0 0x7fb73006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.228+0000 7fb74cc00640 1 -- 172.21.15.27:0/52630511 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7fb74808a180 con 0x7fb7501ee390 2024-03-28T18:55:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.228+0000 7fb755000640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb73006bbd0 0x7fb73006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.228+0000 7fb755000640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb73006bbd0 0x7fb73006e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb744034000 tx=0x7fb744009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:42.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.232+0000 7fb74cc00640 1 -- 172.21.15.27:0/52630511 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb7501577b0 con 0x7fb7501ee390 2024-03-28T18:55:42.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.431+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7fb75004f400 con 0x7fb7501ee390 2024-03-28T18:55:42.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.432+0000 7fb74cc00640 1 -- 172.21.15.27:0/52630511 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v20) v1 ==== 95+0+11 (secure 0 0 0) 0x7fb7501eed50 con 0x7fb7501ee390 2024-03-28T18:55:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:55:42.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb73006bbd0 msgr2=0x7fb73006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb73006bbd0 0x7fb73006e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb744034000 tx=0x7fb744009030 comp rx=0 tx=0).stop 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7501ee390 msgr2=0x7fb7501ee810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7501ee390 0x7fb7501ee810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb75006c070 tx=0x7fb74800d9d0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 shutdown_connections 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb73006bbd0 0x7fb73006e090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7501ee390 0x7fb7501ee810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb750149c50 0x7fb75014a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 --2- 172.21.15.27:0/52630511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7380039b0 0x7fb7500c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 >> 172.21.15.27:0/52630511 conn(0x7fb7500a88b0 msgr2=0x7fb75014d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.435+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 shutdown_connections 2024-03-28T18:55:42.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.436+0000 7fb756400640 1 -- 172.21.15.27:0/52630511 wait complete. 2024-03-28T18:55:42.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T18:55:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.670+0000 7f71e3e00640 1 Processor -- start 2024-03-28T18:55:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.670+0000 7f71e3e00640 1 -- start start 2024-03-28T18:55:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.671+0000 7f71e3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc152f80 0x7f71dc0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71dc0c7970 0x7f71dc0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e3e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f71dc155730 0x7f71dc155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e3e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71dc0c7310 con 0x7f71dc0c7970 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e3e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71dc158d40 con 0x7f71dc155730 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc159020 con 0x7f71dc152f80 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e0a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71dc0c7970 0x7f71dc0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e0a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71dc0c7970 0x7f71dc0c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53318/0 (socket says 172.21.15.27:53318) 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e0a00640 1 -- 172.21.15.27:0/1751403862 learned_addr learned my addr 172.21.15.27:0/1751403862 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e1e00640 1 --2- 172.21.15.27:0/1751403862 >> v2:172.21.15.155:3300/0 conn(0x7f71dc155730 0x7f71dc155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e1400640 1 --2- 172.21.15.27:0/1751403862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc152f80 0x7f71dc0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e0a00640 1 -- 172.21.15.27:0/1751403862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc152f80 msgr2=0x7f71dc0c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e0a00640 1 --2- 172.21.15.27:0/1751403862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc152f80 0x7f71dc0c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e0a00640 1 -- 172.21.15.27:0/1751403862 >> v2:172.21.15.155:3300/0 conn(0x7f71dc155730 msgr2=0x7f71dc155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e0a00640 1 --2- 172.21.15.27:0/1751403862 >> v2:172.21.15.155:3300/0 conn(0x7f71dc155730 0x7f71dc155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e0a00640 1 -- 172.21.15.27:0/1751403862 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc159300 con 0x7f71dc0c7970 2024-03-28T18:55:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.672+0000 7f71e1400640 1 --2- 172.21.15.27:0/1751403862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc152f80 0x7f71dc0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.674+0000 7f71e0a00640 1 --2- 172.21.15.27:0/1751403862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0c7df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f71dc08d110 tx=0x7f71c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c9d7ce5fca7a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.674+0000 7f71dbe00640 1 -- 172.21.15.27:0/1751403862 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c802e070 con 0x7f71dc0c7970 2024-03-28T18:55:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.674+0000 7f71dbe00640 1 -- 172.21.15.27:0/1751403862 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c8037070 con 0x7f71dc0c7970 2024-03-28T18:55:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.674+0000 7f71dbe00640 1 -- 172.21.15.27:0/1751403862 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8031a00 con 0x7f71dc0c7970 2024-03-28T18:55:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 -- 172.21.15.27:0/1751403862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 msgr2=0x7f71dc0c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 --2- 172.21.15.27:0/1751403862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0c7df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f71dc08d110 tx=0x7f71c802b880 comp rx=0 tx=0).stop 2024-03-28T18:55:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 -- 172.21.15.27:0/1751403862 shutdown_connections 2024-03-28T18:55:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 --2- 172.21.15.27:0/1751403862 >> v2:172.21.15.155:3300/0 conn(0x7f71dc155730 0x7f71dc155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 --2- 172.21.15.27:0/1751403862 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0c7df0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 --2- 172.21.15.27:0/1751403862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc152f80 0x7f71dc0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 -- 172.21.15.27:0/1751403862 >> 172.21.15.27:0/1751403862 conn(0x7f71dc06b6f0 msgr2=0x7f71dc0caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 -- 172.21.15.27:0/1751403862 shutdown_connections 2024-03-28T18:55:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.675+0000 7f71e3e00640 1 -- 172.21.15.27:0/1751403862 wait complete. 2024-03-28T18:55:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.677+0000 7f71e3e00640 1 Processor -- start 2024-03-28T18:55:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.677+0000 7f71e3e00640 1 -- start start 2024-03-28T18:55:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.678+0000 7f71e3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0bed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.678+0000 7f71e3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71dc152f80 0x7f71dc0b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.678+0000 7f71e3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc155730 0x7f71dc0c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e3e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc159b00 con 0x7f71dc0c7970 2024-03-28T18:55:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e3e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc0c5e20 con 0x7f71dc152f80 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc0cbca0 con 0x7f71dc155730 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0bed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0bed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53332/0 (socket says 172.21.15.27:53332) 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1400640 1 -- 172.21.15.27:0/4046035909 learned_addr learned my addr 172.21.15.27:0/4046035909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e0a00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71dc152f80 0x7f71dc0b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1e00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc155730 0x7f71dc0c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1400640 1 -- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc155730 msgr2=0x7f71dc0c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1400640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc155730 0x7f71dc0c3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1400640 1 -- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71dc152f80 msgr2=0x7f71dc0b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1400640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71dc152f80 0x7f71dc0b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e1400640 1 -- 172.21.15.27:0/4046035909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc0b0770 con 0x7f71dc0c7970 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.679+0000 7f71e0a00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71dc152f80 0x7f71dc0b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:42.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.680+0000 7f71e1400640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0bed70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f71d000c820 tx=0x7f71d000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.680+0000 7f71d9600640 1 -- 172.21.15.27:0/4046035909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0015070 con 0x7f71dc0c7970 2024-03-28T18:55:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.680+0000 7f71d9600640 1 -- 172.21.15.27:0/4046035909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d00079d0 con 0x7f71dc0c7970 2024-03-28T18:55:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.680+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71dc0c21d0 con 0x7f71dc0c7970 2024-03-28T18:55:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.680+0000 7f71d9600640 1 -- 172.21.15.27:0/4046035909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0007dc0 con 0x7f71dc0c7970 2024-03-28T18:55:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.680+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71dc0c2790 con 0x7f71dc0c7970 2024-03-28T18:55:42.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.681+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc04f400 con 0x7f71dc0c7970 2024-03-28T18:55:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.682+0000 7f71d9600640 1 -- 172.21.15.27:0/4046035909 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f71d0018070 con 0x7f71dc0c7970 2024-03-28T18:55:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.683+0000 7f71d9600640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71bc06bbd0 0x7f71bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:42.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.683+0000 7f71d9600640 1 -- 172.21.15.27:0/4046035909 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f71d008a910 con 0x7f71dc0c7970 2024-03-28T18:55:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.684+0000 7f71e0a00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71bc06bbd0 0x7f71bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:42.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.685+0000 7f71e0a00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71bc06bbd0 0x7f71bc06e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f71c8036000 tx=0x7f71c8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:42.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.690+0000 7f71d9600640 1 -- 172.21.15.27:0/4046035909 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f71d0059380 con 0x7f71dc0c7970 2024-03-28T18:55:42.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.890+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f71dc0cba80 con 0x7f71dc0c7970 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.891+0000 7f71d9600640 1 -- 172.21.15.27:0/4046035909 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v20) v1 ==== 84+0+197 (secure 0 0 0) 0x7f71d0058c80 con 0x7f71dc0c7970 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-28T18:55:42.891 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 2, 2024-03-28T18:55:42.892 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": "a_metadata" 2024-03-28T18:55:42.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-28T18:55:42.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-03-28T18:55:42.892 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 3, 2024-03-28T18:55:42.892 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": "a_data" 2024-03-28T18:55:42.892 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-28T18:55:42.892 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-03-28T18:55:42.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71bc06bbd0 msgr2=0x7f71bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71bc06bbd0 0x7f71bc06e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f71c8036000 tx=0x7f71c8008030 comp rx=0 tx=0).stop 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 msgr2=0x7f71dc0bed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0bed70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f71d000c820 tx=0x7f71d000cce0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 shutdown_connections 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71bc06bbd0 0x7f71bc06e090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71dc155730 0x7f71dc0c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71dc152f80 0x7f71dc0b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 --2- 172.21.15.27:0/4046035909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71dc0c7970 0x7f71dc0bed70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 >> 172.21.15.27:0/4046035909 conn(0x7f71dc06b6f0 msgr2=0x7f71dc14fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.893+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 shutdown_connections 2024-03-28T18:55:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:42.894+0000 7f71e3e00640 1 -- 172.21.15.27:0/4046035909 wait complete. 2024-03-28T18:55:42.911 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'b' 2024-03-28T18:55:42.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_metadata --pg_num_min 64 2024-03-28T18:55:43.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.123+0000 7f8e4b200640 1 Processor -- start 2024-03-28T18:55:43.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.124+0000 7f8e4b200640 1 -- start start 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.125+0000 7f8e4b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c06d540 0x7f8e4c0c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.125+0000 7f8e4b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0c7150 con 0x7f8e4c1577c0 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.125+0000 7f8e4b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0c7430 con 0x7f8e4c0c6410 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.125+0000 7f8e4b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c0c6f00 con 0x7f8e4c06d540 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.126+0000 7f8e4a800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8e4c1577c0 0x7f8e4c0c3d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34074/0 (socket says 172.21.15.27:34074) 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.126+0000 7f8e4a800640 1 -- 172.21.15.27:0/1733562557 learned_addr learned my addr 172.21.15.27:0/1733562557 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.126+0000 7f8e49e00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c06d540 0x7f8e4c0c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.126+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899826389 0 0) 0x7f8e4c0c7150 con 0x7f8e4c1577c0 2024-03-28T18:55:43.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.126+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e30003520 con 0x7f8e4c1577c0 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840039308 0 0) 0x7f8e4c0c7430 con 0x7f8e4c0c6410 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c0c7150 con 0x7f8e4c0c6410 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3326881409 0 0) 0x7f8e30003520 con 0x7f8e4c1577c0 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c06d540 msgr2=0x7f8e4c0c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c06d540 0x7f8e4c0c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 >> v1:172.21.15.155:6789/0 conn(0x7f8e4c0c6410 legacy=0x7f8e4c1570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0c7710 con 0x7f8e4c1577c0 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f8e440019f0 con 0x7f8e4c1577c0 2024-03-28T18:55:43.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 >> v1:172.21.15.27:6789/0 conn(0x7f8e4c1577c0 legacy=0x7f8e4c0c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e30003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30004800 0x7f8e30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c06d540 0x7f8e300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c0c6f00 con 0x7f8e30003b70 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3000ba70 con 0x7f8e30004800 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3000bc70 con 0x7f8e4c06d540 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e49e00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e30003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e49400640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30004800 0x7f8e30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e49e00640 1 -- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c06d540 msgr2=0x7f8e300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:43.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e49e00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c06d540 0x7f8e300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e49e00640 1 -- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30004800 msgr2=0x7f8e30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:43.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e49e00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30004800 0x7f8e30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e49e00640 1 -- 172.21.15.27:0/1733562557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0c7710 con 0x7f8e30003b70 2024-03-28T18:55:43.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.127+0000 7f8e49400640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30004800 0x7f8e30004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:43.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.129+0000 7f8e49e00640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e30003f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009a90 tx=0x7f8e30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c40e1e77e5c7ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:43.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.129+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c039070 con 0x7f8e30003b70 2024-03-28T18:55:43.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.129+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c02c6b0 con 0x7f8e30003b70 2024-03-28T18:55:43.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.129+0000 7f8e48a00640 1 -- 172.21.15.27:0/1733562557 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c009690 con 0x7f8e30003b70 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.129+0000 7f8e4b200640 1 -- 172.21.15.27:0/1733562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 msgr2=0x7f8e30003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.129+0000 7f8e4b200640 1 --2- 172.21.15.27:0/1733562557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e30003f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009a90 tx=0x7f8e30005830 comp rx=0 tx=0).stop 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.130+0000 7f8e4a800640 1 -- 172.21.15.27:0/1733562557 reap_dead start 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.130+0000 7f8e4b200640 1 -- 172.21.15.27:0/1733562557 shutdown_connections 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.130+0000 7f8e4b200640 1 -- 172.21.15.27:0/1733562557 >> 172.21.15.27:0/1733562557 conn(0x7f8e4c06ea50 msgr2=0x7f8e4c06ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.130+0000 7f8e4b200640 1 -- 172.21.15.27:0/1733562557 shutdown_connections 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.130+0000 7f8e4b200640 1 -- 172.21.15.27:0/1733562557 wait complete. 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.131+0000 7f8e4b200640 1 Processor -- start 2024-03-28T18:55:43.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.132+0000 7f8e4b200640 1 -- start start 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.132+0000 7f8e4b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e4c0c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.132+0000 7f8e4b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e4c1e9820 0x7f8e4c0c0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.132+0000 7f8e4b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c0c06a0 0x7f8e4c0ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.132+0000 7f8e4b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c070ad0 con 0x7f8e30003b70 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.132+0000 7f8e4b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c070950 con 0x7f8e4c1e9820 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.132+0000 7f8e4b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c06d920 con 0x7f8e4c0c06a0 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e4c0c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e4c0c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53336/0 (socket says 172.21.15.27:53336) 2024-03-28T18:55:43.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 -- 172.21.15.27:0/200039867 learned_addr learned my addr 172.21.15.27:0/200039867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e4a800640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c0c06a0 0x7f8e4c0ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 -- 172.21.15.27:0/200039867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c0c06a0 msgr2=0x7f8e4c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c0c06a0 0x7f8e4c0ac200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 -- 172.21.15.27:0/200039867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e4c1e9820 msgr2=0x7f8e4c0c0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e4c1e9820 0x7f8e4c0c0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 -- 172.21.15.27:0/200039867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0ad8d0 con 0x7f8e30003b70 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.133+0000 7f8e49e00640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e4c0c24c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c037040 tx=0x7f8e3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.134+0000 7f8e42000640 1 -- 172.21.15.27:0/200039867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c042020 con 0x7f8e30003b70 2024-03-28T18:55:43.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.134+0000 7f8e42000640 1 -- 172.21.15.27:0/200039867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c03a040 con 0x7f8e30003b70 2024-03-28T18:55:43.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.134+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c0be090 con 0x7f8e30003b70 2024-03-28T18:55:43.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.134+0000 7f8e42000640 1 -- 172.21.15.27:0/200039867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c039070 con 0x7f8e30003b70 2024-03-28T18:55:43.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.134+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c0be650 con 0x7f8e30003b70 2024-03-28T18:55:43.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.135+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c04f400 con 0x7f8e30003b70 2024-03-28T18:55:43.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.136+0000 7f8e42000640 1 -- 172.21.15.27:0/200039867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8e3c045020 con 0x7f8e30003b70 2024-03-28T18:55:43.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.137+0000 7f8e42000640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e2c06bc00 0x7f8e2c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.137+0000 7f8e42000640 1 -- 172.21.15.27:0/200039867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5689+0+0 (secure 0 0 0) 0x7f8e3c0abc10 con 0x7f8e30003b70 2024-03-28T18:55:43.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.137+0000 7f8e49400640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e2c06bc00 0x7f8e2c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.138+0000 7f8e49400640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e2c06bc00 0x7f8e2c06e0c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8e38007fd0 tx=0x7f8e38009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:43.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.141+0000 7f8e42000640 1 -- 172.21.15.27:0/200039867 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8e3c04c020 con 0x7f8e30003b70 2024-03-28T18:55:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.350+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8e4c0c5cb0 con 0x7f8e30003b70 2024-03-28T18:55:43.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.552+0000 7f8e42000640 1 -- 172.21.15.27:0/200039867 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64}]=0 pool 'b_metadata' created v21) v1 ==== 128+0+0 (secure 0 0 0) 0x7f8e3c07a700 con 0x7f8e30003b70 2024-03-28T18:55:43.552 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'b_metadata' created 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e2c06bc00 msgr2=0x7f8e2c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e2c06bc00 0x7f8e2c06e0c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8e38007fd0 tx=0x7f8e38009040 comp rx=0 tx=0).stop 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 msgr2=0x7f8e4c0c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e4c0c24c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c037040 tx=0x7f8e3c004730 comp rx=0 tx=0).stop 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 shutdown_connections 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e2c06bc00 0x7f8e2c06e0c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e4c0c06a0 0x7f8e4c0ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e4c1e9820 0x7f8e4c0c0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 --2- 172.21.15.27:0/200039867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e30003b70 0x7f8e4c0c24c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.556+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 >> 172.21.15.27:0/200039867 conn(0x7f8e4c06ea50 msgr2=0x7f8e4c156e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:43.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.559+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 shutdown_connections 2024-03-28T18:55:43.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.559+0000 7f8e4b200640 1 -- 172.21.15.27:0/200039867 wait complete. 2024-03-28T18:55:43.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T18:55:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.800+0000 7f51e1400640 1 Processor -- start 2024-03-28T18:55:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.800+0000 7f51e1400640 1 -- start start 2024-03-28T18:55:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51e1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51dc150890 0x7f51dc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51e1400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f51dc153eb0 0x7f51dc0c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51e1400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51dc0c2770 con 0x7f51dc0c6a20 2024-03-28T18:55:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51e1400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51dc0c2540 con 0x7f51dc150890 2024-03-28T18:55:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51e1400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51dc0c4390 con 0x7f51dc153eb0 2024-03-28T18:55:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51dbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51dc150890 0x7f51dc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51db400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f51dc0c6a20 0x7f51dc1537a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34080/0 (socket says 172.21.15.27:34080) 2024-03-28T18:55:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51dbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51dc150890 0x7f51dc06e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53338/0 (socket says 172.21.15.27:53338) 2024-03-28T18:55:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51db400640 1 -- 172.21.15.27:0/3968293794 learned_addr learned my addr 172.21.15.27:0/3968293794 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51e0a00640 1 --2- 172.21.15.27:0/3968293794 >> v2:172.21.15.155:3300/0 conn(0x7f51dc153eb0 0x7f51dc0c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51dbe00640 1 -- 172.21.15.27:0/3968293794 >> v2:172.21.15.155:3300/0 conn(0x7f51dc153eb0 msgr2=0x7f51dc0c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51dbe00640 1 --2- 172.21.15.27:0/3968293794 >> v2:172.21.15.155:3300/0 conn(0x7f51dc153eb0 0x7f51dc0c6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.802+0000 7f51dbe00640 1 -- 172.21.15.27:0/3968293794 >> v1:172.21.15.27:6789/0 conn(0x7f51dc0c6a20 legacy=0x7f51dc1537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.803+0000 7f51dbe00640 1 -- 172.21.15.27:0/3968293794 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc0c2a50 con 0x7f51dc150890 2024-03-28T18:55:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.803+0000 7f51e0a00640 1 --2- 172.21.15.27:0/3968293794 >> v2:172.21.15.155:3300/0 conn(0x7f51dc153eb0 0x7f51dc0c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:43.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.804+0000 7f51dbe00640 1 --2- 172.21.15.27:0/3968293794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc06e080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0707f0 tx=0x7f51cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5071cfba7e33d486 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:43.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.804+0000 7f51daa00640 1 -- 172.21.15.27:0/3968293794 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc035020 con 0x7f51dc150890 2024-03-28T18:55:43.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.804+0000 7f51daa00640 1 -- 172.21.15.27:0/3968293794 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc031830 con 0x7f51dc150890 2024-03-28T18:55:43.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.804+0000 7f51daa00640 1 -- 172.21.15.27:0/3968293794 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc031c00 con 0x7f51dc150890 2024-03-28T18:55:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.804+0000 7f51e1400640 1 -- 172.21.15.27:0/3968293794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 msgr2=0x7f51dc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.804+0000 7f51e1400640 1 --2- 172.21.15.27:0/3968293794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc06e080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0707f0 tx=0x7f51cc02b8d0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.805+0000 7f51e1400640 1 -- 172.21.15.27:0/3968293794 shutdown_connections 2024-03-28T18:55:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.805+0000 7f51e1400640 1 --2- 172.21.15.27:0/3968293794 >> v2:172.21.15.155:3300/0 conn(0x7f51dc153eb0 0x7f51dc0c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.805+0000 7f51e1400640 1 --2- 172.21.15.27:0/3968293794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc06e080 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.805+0000 7f51e1400640 1 -- 172.21.15.27:0/3968293794 >> 172.21.15.27:0/3968293794 conn(0x7f51dc06d4d0 msgr2=0x7f51dc06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:43.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.805+0000 7f51e1400640 1 -- 172.21.15.27:0/3968293794 shutdown_connections 2024-03-28T18:55:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.805+0000 7f51e1400640 1 -- 172.21.15.27:0/3968293794 wait complete. 2024-03-28T18:55:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.806+0000 7f51e1400640 1 Processor -- start 2024-03-28T18:55:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.807+0000 7f51e1400640 1 -- start start 2024-03-28T18:55:43.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.807+0000 7f51e1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51dc0c6a20 0x7f51dc1ec590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.807+0000 7f51e1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc1de270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.807+0000 7f51e1400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51dc153eb0 0x7f51dc1ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51e1400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc06e5c0 con 0x7f51dc150890 2024-03-28T18:55:43.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51e1400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc0c3010 con 0x7f51dc0c6a20 2024-03-28T18:55:43.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51e1400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc06e740 con 0x7f51dc153eb0 2024-03-28T18:55:43.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc1de270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc1de270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53342/0 (socket says 172.21.15.27:53342) 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51db400640 1 -- 172.21.15.27:0/1578568504 learned_addr learned my addr 172.21.15.27:0/1578568504 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51dbe00640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51dc0c6a20 0x7f51dc1ec590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51db400640 1 -- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51dc153eb0 msgr2=0x7f51dc1ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51e0a00640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51dc153eb0 0x7f51dc1ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51db400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51dc153eb0 0x7f51dc1ebba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51db400640 1 -- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51dc0c6a20 msgr2=0x7f51dc1ec590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51db400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51dc0c6a20 0x7f51dc1ec590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:43.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51db400640 1 -- 172.21.15.27:0/1578568504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc1dca00 con 0x7f51dc150890 2024-03-28T18:55:43.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.808+0000 7f51dbe00640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51dc0c6a20 0x7f51dc1ec590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:43.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.809+0000 7f51db400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc1de270 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c3c0 tx=0x7f51c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:43.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.809+0000 7f51d3e00640 1 -- 172.21.15.27:0/1578568504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8017070 con 0x7f51dc150890 2024-03-28T18:55:43.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.809+0000 7f51d3e00640 1 -- 172.21.15.27:0/1578568504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8005ce0 con 0x7f51dc150890 2024-03-28T18:55:43.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.809+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51dc1de8c0 con 0x7f51dc150890 2024-03-28T18:55:43.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.809+0000 7f51d3e00640 1 -- 172.21.15.27:0/1578568504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c80072f0 con 0x7f51dc150890 2024-03-28T18:55:43.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.809+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc1dcf80 con 0x7f51dc150890 2024-03-28T18:55:43.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.810+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51dc04f400 con 0x7f51dc150890 2024-03-28T18:55:43.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.812+0000 7f51d3e00640 1 -- 172.21.15.27:0/1578568504 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f51c801a070 con 0x7f51dc150890 2024-03-28T18:55:43.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.813+0000 7f51d3e00640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f51bc06bbd0 0x7f51bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:43.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.813+0000 7f51d3e00640 1 -- 172.21.15.27:0/1578568504 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6054+0+0 (secure 0 0 0) 0x7f51c808a350 con 0x7f51dc150890 2024-03-28T18:55:43.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.813+0000 7f51dbe00640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f51bc06bbd0 0x7f51bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:43.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.814+0000 7f51dbe00640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f51bc06bbd0 0x7f51bc06e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f51cc036000 tx=0x7f51cc008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:43.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:43.817+0000 7f51d3e00640 1 -- 172.21.15.27:0/1578568504 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f51c8021020 con 0x7f51dc150890 2024-03-28T18:55:44.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.019+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f51dc157720 con 0x7f51dc150890 2024-03-28T18:55:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.568+0000 7f51d3e00640 1 -- 172.21.15.27:0/1578568504 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'b_data' created v22) v1 ==== 160+0+0 (secure 0 0 0) 0x7f51c8058d50 con 0x7f51dc150890 2024-03-28T18:55:44.568 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'b_data' created 2024-03-28T18:55:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.572+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f51bc06bbd0 msgr2=0x7f51bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.572+0000 7f51e1400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f51bc06bbd0 0x7f51bc06e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f51cc036000 tx=0x7f51cc008030 comp rx=0 tx=0).stop 2024-03-28T18:55:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.572+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 msgr2=0x7f51dc1de270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:44.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.572+0000 7f51e1400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc1de270 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c3c0 tx=0x7f51c800c880 comp rx=0 tx=0).stop 2024-03-28T18:55:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.573+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 shutdown_connections 2024-03-28T18:55:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.573+0000 7f51e1400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f51bc06bbd0 0x7f51bc06e090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.573+0000 7f51e1400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51dc153eb0 0x7f51dc1ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.573+0000 7f51e1400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51dc150890 0x7f51dc1de270 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.573+0000 7f51e1400640 1 --2- 172.21.15.27:0/1578568504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51dc0c6a20 0x7f51dc1ec590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.573+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 >> 172.21.15.27:0/1578568504 conn(0x7f51dc06d4d0 msgr2=0x7f51dc14f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.573+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 shutdown_connections 2024-03-28T18:55:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.573+0000 7f51e1400640 1 -- 172.21.15.27:0/1578568504 wait complete. 2024-03-28T18:55:44.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new b b_metadata b_data 2024-03-28T18:55:44.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.800+0000 7ff3ebe00640 1 Processor -- start 2024-03-28T18:55:44.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.800+0000 7ff3ebe00640 1 -- start start 2024-03-28T18:55:44.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.802+0000 7ff3ebe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a5c60 0x7ff3ec0a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:44.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.802+0000 7ff3ebe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff3ec0a66e0 0x7ff3ec0a6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:44.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.802+0000 7ff3ebe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec154250 con 0x7ff3ec1508c0 2024-03-28T18:55:44.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.802+0000 7ff3ebe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ec153dc0 con 0x7ff3ec0a66e0 2024-03-28T18:55:44.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.802+0000 7ff3ebe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec073500 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.802+0000 7ff3eaa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff3ec1508c0 0x7ff3ec072c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34090/0 (socket says 172.21.15.27:34090) 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.802+0000 7ff3eaa00640 1 -- 172.21.15.27:0/3526293188 learned_addr learned my addr 172.21.15.27:0/3526293188 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.802+0000 7ff3e2000640 1 --2- 172.21.15.27:0/3526293188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a5c60 0x7ff3ec0a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.803+0000 7ff3eb400640 1 --2- 172.21.15.27:0/3526293188 >> v2:172.21.15.155:3300/0 conn(0x7ff3ec0a66e0 0x7ff3ec0a6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.803+0000 7ff3ea000640 1 -- 172.21.15.27:0/3526293188 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109606285 0 0) 0x7ff3ec154250 con 0x7ff3ec1508c0 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.803+0000 7ff3ea000640 1 -- 172.21.15.27:0/3526293188 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d0003520 con 0x7ff3ec1508c0 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.803+0000 7ff3e2000640 1 -- 172.21.15.27:0/3526293188 >> v2:172.21.15.155:3300/0 conn(0x7ff3ec0a66e0 msgr2=0x7ff3ec0a6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.803+0000 7ff3e2000640 1 --2- 172.21.15.27:0/3526293188 >> v2:172.21.15.155:3300/0 conn(0x7ff3ec0a66e0 0x7ff3ec0a6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.803+0000 7ff3e2000640 1 -- 172.21.15.27:0/3526293188 >> v1:172.21.15.27:6789/0 conn(0x7ff3ec1508c0 legacy=0x7ff3ec072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:44.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.803+0000 7ff3e2000640 1 -- 172.21.15.27:0/3526293188 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec1534d0 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.804+0000 7ff3e2000640 1 --2- 172.21.15.27:0/3526293188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a5c60 0x7ff3ec0a60e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40097a0 tx=0x7ff3d402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8535601bd19e2ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.804+0000 7ff3ea000640 1 -- 172.21.15.27:0/3526293188 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4039070 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.804+0000 7ff3ea000640 1 -- 172.21.15.27:0/3526293188 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d40316f0 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.804+0000 7ff3ea000640 1 -- 172.21.15.27:0/3526293188 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4031aa0 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.805+0000 7ff3ebe00640 1 -- 172.21.15.27:0/3526293188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a5c60 msgr2=0x7ff3ec0a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.805+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/3526293188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a5c60 0x7ff3ec0a60e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40097a0 tx=0x7ff3d402b790 comp rx=0 tx=0).stop 2024-03-28T18:55:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.805+0000 7ff3ebe00640 1 -- 172.21.15.27:0/3526293188 shutdown_connections 2024-03-28T18:55:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.805+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/3526293188 >> v2:172.21.15.155:3300/0 conn(0x7ff3ec0a66e0 0x7ff3ec0a6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.805+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/3526293188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a5c60 0x7ff3ec0a60e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.805+0000 7ff3ebe00640 1 -- 172.21.15.27:0/3526293188 >> 172.21.15.27:0/3526293188 conn(0x7ff3ec0a88b0 msgr2=0x7ff3ec0a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.805+0000 7ff3ebe00640 1 -- 172.21.15.27:0/3526293188 shutdown_connections 2024-03-28T18:55:44.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.805+0000 7ff3ebe00640 1 -- 172.21.15.27:0/3526293188 wait complete. 2024-03-28T18:55:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.807+0000 7ff3ebe00640 1 Processor -- start 2024-03-28T18:55:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.807+0000 7ff3ebe00640 1 -- start start 2024-03-28T18:55:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3ebe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3ec0a5c60 0x7ff3ec152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3ebe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a66e0 0x7ff3ec0c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3ebe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ec1508c0 0x7ff3ec149870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3ebe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0a70b0 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3ebe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec153a50 con 0x7ff3ec1508c0 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3ebe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0a8550 con 0x7ff3ec0a66e0 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3ec0a5c60 0x7ff3ec152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3ec0a5c60 0x7ff3ec152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53358/0 (socket says 172.21.15.27:53358) 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.808+0000 7ff3eaa00640 1 -- 172.21.15.27:0/298094456 learned_addr learned my addr 172.21.15.27:0/298094456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3e2000640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a66e0 0x7ff3ec0c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3eb400640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ec1508c0 0x7ff3ec149870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3eaa00640 1 -- 172.21.15.27:0/298094456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a66e0 msgr2=0x7ff3ec0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3eaa00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a66e0 0x7ff3ec0c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3eaa00640 1 -- 172.21.15.27:0/298094456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ec1508c0 msgr2=0x7ff3ec149870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3eaa00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ec1508c0 0x7ff3ec149870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3eaa00640 1 -- 172.21.15.27:0/298094456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0c5ed0 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3eaa00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3ec0a5c60 0x7ff3ec152460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00c430 tx=0x7ff3dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3e3e00640 1 -- 172.21.15.27:0/298094456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc017070 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3e3e00640 1 -- 172.21.15.27:0/298094456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc005ce0 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3e3e00640 1 -- 172.21.15.27:0/298094456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc0072f0 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.809+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec0c61b0 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.810+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec0c6770 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.811+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec04f400 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.812+0000 7ff3e3e00640 1 -- 172.21.15.27:0/298094456 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff3dc01a070 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.813+0000 7ff3e3e00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff3c806bbd0 0x7ff3c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.813+0000 7ff3e3e00640 1 -- 172.21.15.27:0/298094456 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff3dc08a570 con 0x7ff3ec0a5c60 2024-03-28T18:55:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.813+0000 7ff3e2000640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff3c806bbd0 0x7ff3c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.814+0000 7ff3e2000640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff3c806bbd0 0x7ff3c806e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4036000 tx=0x7ff3d4008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:44.818+0000 7ff3e3e00640 1 -- 172.21.15.27:0/298094456 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff3dc021020 con 0x7ff3ec0a5c60 2024-03-28T18:55:45.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.062+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"} v 0) v1 -- 0x7ff3ec0a7a30 con 0x7ff3ec0a5c60 2024-03-28T18:55:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.586+0000 7ff3e3e00640 1 -- 172.21.15.27:0/298094456 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"}]=0 new fs with metadata pool 4 and data pool 5 v10) v1 ==== 157+0+0 (secure 0 0 0) 0x7ff3dc0587b0 con 0x7ff3ec0a5c60 2024-03-28T18:55:45.590 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 4 and data pool 5 2024-03-28T18:55:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.600+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff3c806bbd0 msgr2=0x7ff3c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.600+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff3c806bbd0 0x7ff3c806e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4036000 tx=0x7ff3d4008030 comp rx=0 tx=0).stop 2024-03-28T18:55:45.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.600+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3ec0a5c60 msgr2=0x7ff3ec152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.600+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3ec0a5c60 0x7ff3ec152460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00c430 tx=0x7ff3dc00c8f0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.605+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 shutdown_connections 2024-03-28T18:55:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.605+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff3c806bbd0 0x7ff3c806e090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.605+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ec1508c0 0x7ff3ec149870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.605+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ec0a66e0 0x7ff3ec0c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.605+0000 7ff3ebe00640 1 --2- 172.21.15.27:0/298094456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3ec0a5c60 0x7ff3ec152460 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.605+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 >> 172.21.15.27:0/298094456 conn(0x7ff3ec0a88b0 msgr2=0x7ff3ec0c06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:45.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.607+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 shutdown_connections 2024-03-28T18:55:45.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.607+0000 7ff3ebe00640 1 -- 172.21.15.27:0/298094456 wait complete. 2024-03-28T18:55:45.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:45.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.822+0000 7f9ee0c00640 1 Processor -- start 2024-03-28T18:55:45.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.822+0000 7f9ee0c00640 1 -- start start 2024-03-28T18:55:45.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.823+0000 7f9ee0c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9edc0c5e80 0x7f9edc0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:45.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.823+0000 7f9ee0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc0ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.823+0000 7f9ee0c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc06d8a0 con 0x7f9edc0abd90 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.823+0000 7f9ee0c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc0ac220 con 0x7f9edc0c5e80 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.823+0000 7f9ee0c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc06d650 con 0x7f9edc0c1f50 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.824+0000 7f9ed9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9edc0c5e80 0x7f9edc0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.824+0000 7f9ed9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9edc0c5e80 0x7f9edc0c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53366/0 (socket says 172.21.15.27:53366) 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.824+0000 7f9ed9e00640 1 -- 172.21.15.27:0/1697260416 learned_addr learned my addr 172.21.15.27:0/1697260416 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.824+0000 7f9ed9400640 1 --2- 172.21.15.27:0/1697260416 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc0ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.824+0000 7f9ed9e00640 1 -- 172.21.15.27:0/1697260416 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 msgr2=0x7f9edc0ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:45.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.824+0000 7f9ed9e00640 1 --2- 172.21.15.27:0/1697260416 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc0ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.824+0000 7f9ed9e00640 1 -- 172.21.15.27:0/1697260416 >> v1:172.21.15.155:6789/0 conn(0x7f9edc0abd90 legacy=0x7f9edc0aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:45.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.824+0000 7f9ed9e00640 1 -- 172.21.15.27:0/1697260416 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc06db80 con 0x7f9edc0c5e80 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.825+0000 7f9ed9e00640 1 --2- 172.21.15.27:0/1697260416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0c5e80 0x7f9edc0c2cb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00bb10 tx=0x7f9ecc02e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4995b80f6278750f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ed8a00640 1 -- 172.21.15.27:0/1697260416 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc03b070 con 0x7f9edc0c5e80 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ed8a00640 1 -- 172.21.15.27:0/1697260416 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc02ede0 con 0x7f9edc0c5e80 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ed8a00640 1 -- 172.21.15.27:0/1697260416 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc036660 con 0x7f9edc0c5e80 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1697260416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0c5e80 msgr2=0x7f9edc0c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1697260416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0c5e80 0x7f9edc0c2cb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00bb10 tx=0x7f9ecc02e030 comp rx=0 tx=0).stop 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1697260416 shutdown_connections 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1697260416 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc0ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1697260416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0c5e80 0x7f9edc0c2cb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1697260416 >> 172.21.15.27:0/1697260416 conn(0x7f9edc06b9d0 msgr2=0x7f9edc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:45.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.826+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1697260416 shutdown_connections 2024-03-28T18:55:45.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.827+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1697260416 wait complete. 2024-03-28T18:55:45.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.828+0000 7f9ee0c00640 1 Processor -- start 2024-03-28T18:55:45.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.828+0000 7f9ee0c00640 1 -- start start 2024-03-28T18:55:45.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ee0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0abd90 0x7f9edc0c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:45.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ee0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc1552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:45.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ee0c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9edc0c5e80 0x7f9edc155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:45.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ee0c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0c31f0 con 0x7f9edc0abd90 2024-03-28T18:55:45.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ee0c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc06e1d0 con 0x7f9edc0c5e80 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ee0c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0c3370 con 0x7f9edc0c1f50 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ed9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0abd90 0x7f9edc0c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ed9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0abd90 0x7f9edc0c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53370/0 (socket says 172.21.15.27:53370) 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.829+0000 7f9ed9e00640 1 -- 172.21.15.27:0/1937205532 learned_addr learned my addr 172.21.15.27:0/1937205532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed9400640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc1552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9eda800640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9edc0c5e80 0x7f9edc155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed9e00640 1 -- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 msgr2=0x7f9edc1552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed9e00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc1552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed9e00640 1 -- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9edc0c5e80 msgr2=0x7f9edc155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:45.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed9e00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9edc0c5e80 0x7f9edc155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed9e00640 1 -- 172.21.15.27:0/1937205532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc1ebad0 con 0x7f9edc0abd90 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed9400640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc1552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9eda800640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9edc0c5e80 0x7f9edc155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed9e00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0abd90 0x7f9edc0c02a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc02e070 tx=0x7f9ecc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed2000640 1 -- 172.21.15.27:0/1937205532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc03b070 con 0x7f9edc0abd90 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed2000640 1 -- 172.21.15.27:0/1937205532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc03d050 con 0x7f9edc0abd90 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ed2000640 1 -- 172.21.15.27:0/1937205532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc034990 con 0x7f9edc0abd90 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.830+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc1ebd70 con 0x7f9edc0abd90 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.831+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc0c07e0 con 0x7f9edc0abd90 2024-03-28T18:55:45.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.832+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc04f400 con 0x7f9edc0abd90 2024-03-28T18:55:45.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.833+0000 7f9ed2000640 1 -- 172.21.15.27:0/1937205532 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ecc048060 con 0x7f9edc0abd90 2024-03-28T18:55:45.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.834+0000 7f9ed2000640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:45.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.834+0000 7f9ed2000640 1 -- 172.21.15.27:0/1937205532 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f9ecc0ac6d0 con 0x7f9edc0abd90 2024-03-28T18:55:45.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.834+0000 7f9ed9400640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:45.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.835+0000 7f9ed9400640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9edc156860 tx=0x7f9ec8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:45.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:45.838+0000 7f9ed2000640 1 -- 172.21.15.27:0/1937205532 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9ecc07ae00 con 0x7f9edc0abd90 2024-03-28T18:55:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.040+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9edc061cf0 con 0x7f9edc0abd90 2024-03-28T18:55:46.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.044+0000 7f9ed2000640 1 -- 172.21.15.27:0/1937205532 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+20320 (secure 0 0 0) 0x7f9ecc07a700 con 0x7f9edc0abd90 2024-03-28T18:55:46.045 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:46.045 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:45.063592+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ebc06bbd0 msgr2=0x7f9ebc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9edc156860 tx=0x7f9ec8008040 comp rx=0 tx=0).stop 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0abd90 msgr2=0x7f9edc0c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0abd90 0x7f9edc0c02a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc02e070 tx=0x7f9ecc004730 comp rx=0 tx=0).stop 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 shutdown_connections 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ebc06bbd0 0x7f9ebc06e090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9edc0c5e80 0x7f9edc155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9edc0c1f50 0x7f9edc1552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 --2- 172.21.15.27:0/1937205532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0abd90 0x7f9edc0c02a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 >> 172.21.15.27:0/1937205532 conn(0x7f9edc06b9d0 msgr2=0x7f9edc070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 shutdown_connections 2024-03-28T18:55:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.047+0000 7f9ee0c00640 1 -- 172.21.15.27:0/1937205532 wait complete. 2024-03-28T18:55:46.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.292+0000 7f6faa800640 1 Processor -- start 2024-03-28T18:55:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.292+0000 7f6faa800640 1 -- start start 2024-03-28T18:55:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.293+0000 7f6faa800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40c8480 0x7f6fa40c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.293+0000 7f6faa800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6fa40a8240 0x7f6fa415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.293+0000 7f6faa800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa40bdbd0 con 0x7f6fa40c52f0 2024-03-28T18:55:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.293+0000 7f6faa800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa40bf910 con 0x7f6fa40a8240 2024-03-28T18:55:46.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.293+0000 7f6faa800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40bf5a0 con 0x7f6fa40c8480 2024-03-28T18:55:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.294+0000 7f6fa8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40c8480 0x7f6fa40c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.294+0000 7f6fa8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40c8480 0x7f6fa40c56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:44890/0 (socket says 172.21.15.27:44890) 2024-03-28T18:55:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.294+0000 7f6fa8a00640 1 -- 172.21.15.27:0/482991969 learned_addr learned my addr 172.21.15.27:0/482991969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.294+0000 7f6fa9e00640 1 --2- 172.21.15.27:0/482991969 >> v2:172.21.15.155:3300/0 conn(0x7f6fa40a8240 0x7f6fa415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.294+0000 7f6fa9400640 1 --1- 172.21.15.27:0/482991969 >> v1:172.21.15.155:6789/0 conn(0x7f6fa40c52f0 0x7f6fa4153360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55484/0 (socket says 172.21.15.27:55484) 2024-03-28T18:55:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.294+0000 7f6fa8a00640 1 -- 172.21.15.27:0/482991969 >> v2:172.21.15.155:3300/0 conn(0x7f6fa40a8240 msgr2=0x7f6fa415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:46.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.294+0000 7f6fa8a00640 1 --2- 172.21.15.27:0/482991969 >> v2:172.21.15.155:3300/0 conn(0x7f6fa40a8240 0x7f6fa415dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.294+0000 7f6fa8a00640 1 -- 172.21.15.27:0/482991969 >> v1:172.21.15.155:6789/0 conn(0x7f6fa40c52f0 legacy=0x7f6fa4153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.295+0000 7f6fa8a00640 1 -- 172.21.15.27:0/482991969 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa4153a70 con 0x7f6fa40c8480 2024-03-28T18:55:46.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.296+0000 7f6fa8a00640 1 --2- 172.21.15.27:0/482991969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40c8480 0x7f6fa40c56d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009e50 tx=0x7f6f9002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc9fc2a8e6effbe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:46.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.296+0000 7f6fa3e00640 1 -- 172.21.15.27:0/482991969 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9002cb00 con 0x7f6fa40c8480 2024-03-28T18:55:46.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.296+0000 7f6fa3e00640 1 -- 172.21.15.27:0/482991969 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9002cca0 con 0x7f6fa40c8480 2024-03-28T18:55:46.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.296+0000 7f6fa3e00640 1 -- 172.21.15.27:0/482991969 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f90034610 con 0x7f6fa40c8480 2024-03-28T18:55:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.297+0000 7f6faa800640 1 -- 172.21.15.27:0/482991969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40c8480 msgr2=0x7f6fa40c56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.297+0000 7f6faa800640 1 --2- 172.21.15.27:0/482991969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40c8480 0x7f6fa40c56d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009e50 tx=0x7f6f9002bf60 comp rx=0 tx=0).stop 2024-03-28T18:55:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.297+0000 7f6faa800640 1 -- 172.21.15.27:0/482991969 shutdown_connections 2024-03-28T18:55:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.297+0000 7f6faa800640 1 --2- 172.21.15.27:0/482991969 >> v2:172.21.15.155:3300/0 conn(0x7f6fa40a8240 0x7f6fa415dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.297+0000 7f6faa800640 1 --2- 172.21.15.27:0/482991969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40c8480 0x7f6fa40c56d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.297+0000 7f6faa800640 1 -- 172.21.15.27:0/482991969 >> 172.21.15.27:0/482991969 conn(0x7f6fa40aa7c0 msgr2=0x7f6fa40aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.297+0000 7f6faa800640 1 -- 172.21.15.27:0/482991969 shutdown_connections 2024-03-28T18:55:46.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.297+0000 7f6faa800640 1 -- 172.21.15.27:0/482991969 wait complete. 2024-03-28T18:55:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.299+0000 7f6faa800640 1 Processor -- start 2024-03-28T18:55:46.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.299+0000 7f6faa800640 1 -- start start 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6faa800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40a8240 0x7f6fa41ef590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6faa800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa40c52f0 0x7f6fa40a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6faa800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa40c8480 0x7f6fa40c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6faa800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40a7e00 con 0x7f6fa40c8480 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6faa800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40a7c80 con 0x7f6fa40c52f0 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6faa800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa4157530 con 0x7f6fa40a8240 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6fa9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa40c8480 0x7f6fa40c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6fa9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa40c8480 0x7f6fa40c1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59236/0 (socket says 172.21.15.27:59236) 2024-03-28T18:55:46.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6fa9e00640 1 -- 172.21.15.27:0/3335923877 learned_addr learned my addr 172.21.15.27:0/3335923877 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6fa9400640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40a8240 0x7f6fa41ef590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.300+0000 7f6fa8a00640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa40c52f0 0x7f6fa40a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa9e00640 1 -- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40a8240 msgr2=0x7f6fa41ef590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa9e00640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40a8240 0x7f6fa41ef590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa9e00640 1 -- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa40c52f0 msgr2=0x7f6fa40a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa9e00640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa40c52f0 0x7f6fa40a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa9e00640 1 -- 172.21.15.27:0/3335923877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40c2c70 con 0x7f6fa40c8480 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa8a00640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa40c52f0 0x7f6fa40a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa9e00640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa40c8480 0x7f6fa40c1b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00e960 tx=0x7f6f9c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa1600640 1 -- 172.21.15.27:0/3335923877 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c017070 con 0x7f6fa40c8480 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa1600640 1 -- 172.21.15.27:0/3335923877 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9c00c980 con 0x7f6fa40c8480 2024-03-28T18:55:46.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6fa1600640 1 -- 172.21.15.27:0/3335923877 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00cd70 con 0x7f6fa40c8480 2024-03-28T18:55:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.301+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa40c2f50 con 0x7f6fa40c8480 2024-03-28T18:55:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.302+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa40a99f0 con 0x7f6fa40c8480 2024-03-28T18:55:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.303+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa4067dd0 con 0x7f6fa40c8480 2024-03-28T18:55:46.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.303+0000 7f6fa1600640 1 -- 172.21.15.27:0/3335923877 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6f9c01a070 con 0x7f6fa40c8480 2024-03-28T18:55:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.304+0000 7f6fa1600640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8406bbd0 0x7f6f8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.304+0000 7f6fa1600640 1 -- 172.21.15.27:0/3335923877 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f6f9c08b1c0 con 0x7f6fa40c8480 2024-03-28T18:55:46.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.304+0000 7f6fa9400640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8406bbd0 0x7f6f8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.305+0000 7f6fa9400640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8406bbd0 0x7f6f8406e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6f94007fd0 tx=0x7f6f94007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:46.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.309+0000 7f6fa1600640 1 -- 172.21.15.27:0/3335923877 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6f9c059200 con 0x7f6fa40c8480 2024-03-28T18:55:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.520+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fa4069640 con 0x7f6fa40c8480 2024-03-28T18:55:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.524+0000 7f6fa1600640 1 -- 172.21.15.27:0/3335923877 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+20320 (secure 0 0 0) 0x7f6f9c05d2a0 con 0x7f6fa40c8480 2024-03-28T18:55:46.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:46.525 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:45.063592+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8406bbd0 msgr2=0x7f6f8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8406bbd0 0x7f6f8406e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6f94007fd0 tx=0x7f6f94007e80 comp rx=0 tx=0).stop 2024-03-28T18:55:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa40c8480 msgr2=0x7f6fa40c1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa40c8480 0x7f6fa40c1b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00e960 tx=0x7f6f9c00ee20 comp rx=0 tx=0).stop 2024-03-28T18:55:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 shutdown_connections 2024-03-28T18:55:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8406bbd0 0x7f6f8406e090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa40c8480 0x7f6fa40c1b90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa40c52f0 0x7f6fa40a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 --2- 172.21.15.27:0/3335923877 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fa40a8240 0x7f6fa41ef590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.527+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 >> 172.21.15.27:0/3335923877 conn(0x7f6fa40aa7c0 msgr2=0x7f6fa40a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.528+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 shutdown_connections 2024-03-28T18:55:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.528+0000 7f6faa800640 1 -- 172.21.15.27:0/3335923877 wait complete. 2024-03-28T18:55:46.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b standby_count_wanted 0 2024-03-28T18:55:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.781+0000 7f5cfe600640 1 Processor -- start 2024-03-28T18:55:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.781+0000 7f5cfe600640 1 -- start start 2024-03-28T18:55:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.783+0000 7f5cfe600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5cf80c8db0 0x7f5cf8156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.783+0000 7f5cfe600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf8152840 0x7f5cf8152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.783+0000 7f5cfe600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf80735f0 con 0x7f5cf80c5aa0 2024-03-28T18:55:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.783+0000 7f5cfe600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf81565d0 con 0x7f5cf80c8db0 2024-03-28T18:55:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.783+0000 7f5cfe600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf80c82d0 con 0x7f5cf8152840 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.783+0000 7f5cfdc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf8152840 0x7f5cf8152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.783+0000 7f5cfd200640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5cf80c5aa0 0x7f5cf8072620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55496/0 (socket says 172.21.15.27:55496) 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.784+0000 7f5cfd200640 1 -- 172.21.15.27:0/2818268515 learned_addr learned my addr 172.21.15.27:0/2818268515 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.784+0000 7f5cf7e00640 1 --2- 172.21.15.27:0/2818268515 >> v2:172.21.15.155:3300/0 conn(0x7f5cf80c8db0 0x7f5cf8156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.784+0000 7f5cf7400640 1 -- 172.21.15.27:0/2818268515 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158673456 0 0) 0x7f5cf80735f0 con 0x7f5cf80c5aa0 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.784+0000 7f5cfdc00640 1 -- 172.21.15.27:0/2818268515 >> v2:172.21.15.155:3300/0 conn(0x7f5cf80c8db0 msgr2=0x7f5cf8156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.784+0000 7f5cfdc00640 1 --2- 172.21.15.27:0/2818268515 >> v2:172.21.15.155:3300/0 conn(0x7f5cf80c8db0 0x7f5cf8156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.784+0000 7f5cfdc00640 1 -- 172.21.15.27:0/2818268515 >> v1:172.21.15.155:6789/0 conn(0x7f5cf80c5aa0 legacy=0x7f5cf8072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.784+0000 7f5cfdc00640 1 -- 172.21.15.27:0/2818268515 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf80c7d10 con 0x7f5cf8152840 2024-03-28T18:55:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.786+0000 7f5cfdc00640 1 --2- 172.21.15.27:0/2818268515 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf8152840 0x7f5cf8152cd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009d90 tx=0x7f5cf002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e86cd52e60902b7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.786+0000 7f5cf7400640 1 -- 172.21.15.27:0/2818268515 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0039070 con 0x7f5cf8152840 2024-03-28T18:55:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.786+0000 7f5cf7400640 1 -- 172.21.15.27:0/2818268515 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf002cbc0 con 0x7f5cf8152840 2024-03-28T18:55:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.786+0000 7f5cf7400640 1 -- 172.21.15.27:0/2818268515 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0034440 con 0x7f5cf8152840 2024-03-28T18:55:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.787+0000 7f5cfe600640 1 -- 172.21.15.27:0/2818268515 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf8152840 msgr2=0x7f5cf8152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.787+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2818268515 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf8152840 0x7f5cf8152cd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009d90 tx=0x7f5cf002be10 comp rx=0 tx=0).stop 2024-03-28T18:55:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.787+0000 7f5cfe600640 1 -- 172.21.15.27:0/2818268515 shutdown_connections 2024-03-28T18:55:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.787+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2818268515 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf8152840 0x7f5cf8152cd0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.787+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2818268515 >> v2:172.21.15.155:3300/0 conn(0x7f5cf80c8db0 0x7f5cf8156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.787+0000 7f5cfe600640 1 -- 172.21.15.27:0/2818268515 >> 172.21.15.27:0/2818268515 conn(0x7f5cf806b0c0 msgr2=0x7f5cf806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.787+0000 7f5cfe600640 1 -- 172.21.15.27:0/2818268515 shutdown_connections 2024-03-28T18:55:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.787+0000 7f5cfe600640 1 -- 172.21.15.27:0/2818268515 wait complete. 2024-03-28T18:55:46.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.788+0000 7f5cfe600640 1 Processor -- start 2024-03-28T18:55:46.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.789+0000 7f5cfe600640 1 -- start start 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.789+0000 7f5cfe600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cf80c5aa0 0x7f5cf80c21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.789+0000 7f5cfe600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf80c8db0 0x7f5cf8158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfe600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cf8152840 0x7f5cf80c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfe600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8072100 con 0x7f5cf8152840 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfe600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8071e20 con 0x7f5cf80c5aa0 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfe600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8072d30 con 0x7f5cf80c8db0 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cf8152840 0x7f5cf80c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cf8152840 0x7f5cf80c3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59244/0 (socket says 172.21.15.27:59244) 2024-03-28T18:55:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfdc00640 1 -- 172.21.15.27:0/2141114838 learned_addr learned my addr 172.21.15.27:0/2141114838 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cf7e00640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf80c8db0 0x7f5cf8158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfd200640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cf80c5aa0 0x7f5cf80c21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfdc00640 1 -- 172.21.15.27:0/2141114838 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf80c8db0 msgr2=0x7f5cf8158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfdc00640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf80c8db0 0x7f5cf8158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfdc00640 1 -- 172.21.15.27:0/2141114838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cf80c5aa0 msgr2=0x7f5cf80c21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfdc00640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cf80c5aa0 0x7f5cf80c21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.790+0000 7f5cfdc00640 1 -- 172.21.15.27:0/2141114838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf80c0c00 con 0x7f5cf8152840 2024-03-28T18:55:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.791+0000 7f5cfdc00640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cf8152840 0x7f5cf80c3fe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009eb0 tx=0x7f5cf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.791+0000 7f5cf4c00640 1 -- 172.21.15.27:0/2141114838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0039050 con 0x7f5cf8152840 2024-03-28T18:55:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.791+0000 7f5cf4c00640 1 -- 172.21.15.27:0/2141114838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf0045070 con 0x7f5cf8152840 2024-03-28T18:55:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.791+0000 7f5cf4c00640 1 -- 172.21.15.27:0/2141114838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0009450 con 0x7f5cf8152840 2024-03-28T18:55:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.791+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf81eb390 con 0x7f5cf8152840 2024-03-28T18:55:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.791+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf80c0e70 con 0x7f5cf8152840 2024-03-28T18:55:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.793+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf804f400 con 0x7f5cf8152840 2024-03-28T18:55:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.793+0000 7f5cf4c00640 1 -- 172.21.15.27:0/2141114838 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5cf003e850 con 0x7f5cf8152840 2024-03-28T18:55:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.794+0000 7f5cf4c00640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cd806bbd0 0x7f5cd806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.794+0000 7f5cf4c00640 1 -- 172.21.15.27:0/2141114838 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f5cf00adbb0 con 0x7f5cf8152840 2024-03-28T18:55:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.795+0000 7f5cfd200640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cd806bbd0 0x7f5cd806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.795+0000 7f5cfd200640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cd806bbd0 0x7f5cd806e090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5cec000f60 tx=0x7f5cec007bb0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:46.799+0000 7f5cf4c00640 1 -- 172.21.15.27:0/2141114838 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5cf0032e20 con 0x7f5cf8152840 2024-03-28T18:55:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.066+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5cf80732f0 con 0x7f5cf8152840 2024-03-28T18:55:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.606+0000 7f5cf4c00640 1 -- 172.21.15.27:0/2141114838 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"}]=0 v13) v1 ==== 113+0+0 (secure 0 0 0) 0x7f5cf007c3f0 con 0x7f5cf8152840 2024-03-28T18:55:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cd806bbd0 msgr2=0x7f5cd806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cd806bbd0 0x7f5cd806e090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5cec000f60 tx=0x7f5cec007bb0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cf8152840 msgr2=0x7f5cf80c3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cf8152840 0x7f5cf80c3fe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009eb0 tx=0x7f5cf0004a40 comp rx=0 tx=0).stop 2024-03-28T18:55:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 shutdown_connections 2024-03-28T18:55:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cd806bbd0 0x7f5cd806e090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cf8152840 0x7f5cf80c3fe0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cf80c8db0 0x7f5cf8158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 --2- 172.21.15.27:0/2141114838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cf80c5aa0 0x7f5cf80c21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 >> 172.21.15.27:0/2141114838 conn(0x7f5cf806b0c0 msgr2=0x7f5cf80ab450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.610+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 shutdown_connections 2024-03-28T18:55:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.611+0000 7f5cfe600640 1 -- 172.21.15.27:0/2141114838 wait complete. 2024-03-28T18:55:47.631 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-03-28T18:55:47.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b allow_standby_replay false 2024-03-28T18:55:47.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.852+0000 7f5ba5400640 1 Processor -- start 2024-03-28T18:55:47.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.852+0000 7f5ba5400640 1 -- start start 2024-03-28T18:55:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.854+0000 7f5ba5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ba0150900 0x7f5ba0157420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.854+0000 7f5ba5400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba00c8490 0x7f5ba01533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.854+0000 7f5ba5400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba00c5e20 con 0x7f5ba0153910 2024-03-28T18:55:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.854+0000 7f5ba5400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba00c75c0 con 0x7f5ba0150900 2024-03-28T18:55:47.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.854+0000 7f5ba5400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba00c78a0 con 0x7f5ba00c8490 2024-03-28T18:55:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.854+0000 7f5b9fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ba0150900 0x7f5ba0157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.854+0000 7f5b9fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ba0150900 0x7f5ba0157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59250/0 (socket says 172.21.15.27:59250) 2024-03-28T18:55:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.854+0000 7f5b9fe00640 1 -- 172.21.15.27:0/3923487483 learned_addr learned my addr 172.21.15.27:0/3923487483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.855+0000 7f5b97e00640 1 --2- 172.21.15.27:0/3923487483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba00c8490 0x7f5ba01533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.855+0000 7f5b9fe00640 1 -- 172.21.15.27:0/3923487483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba00c8490 msgr2=0x7f5ba01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.855+0000 7f5b9fe00640 1 --2- 172.21.15.27:0/3923487483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba00c8490 0x7f5ba01533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.855+0000 7f5b9fe00640 1 -- 172.21.15.27:0/3923487483 >> v1:172.21.15.155:6789/0 conn(0x7f5ba0153910 legacy=0x7f5ba00c52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:47.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.855+0000 7f5b9fe00640 1 -- 172.21.15.27:0/3923487483 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba0156940 con 0x7f5ba0150900 2024-03-28T18:55:47.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.856+0000 7f5b9fe00640 1 --2- 172.21.15.27:0/3923487483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba0150900 0x7f5ba0157420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009870 tx=0x7f5b9002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bd64898b33cd48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.856+0000 7f5b9f400640 1 -- 172.21.15.27:0/3923487483 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b900317c0 con 0x7f5ba0150900 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.856+0000 7f5b9f400640 1 -- 172.21.15.27:0/3923487483 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b90037070 con 0x7f5ba0150900 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.856+0000 7f5b9f400640 1 -- 172.21.15.27:0/3923487483 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b90031b80 con 0x7f5ba0150900 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.857+0000 7f5ba5400640 1 -- 172.21.15.27:0/3923487483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba0150900 msgr2=0x7f5ba0157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.857+0000 7f5ba5400640 1 --2- 172.21.15.27:0/3923487483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba0150900 0x7f5ba0157420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009870 tx=0x7f5b9002b860 comp rx=0 tx=0).stop 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.857+0000 7f5ba5400640 1 -- 172.21.15.27:0/3923487483 shutdown_connections 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.857+0000 7f5ba5400640 1 --2- 172.21.15.27:0/3923487483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba00c8490 0x7f5ba01533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.857+0000 7f5ba5400640 1 --2- 172.21.15.27:0/3923487483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba0150900 0x7f5ba0157420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.857+0000 7f5ba5400640 1 -- 172.21.15.27:0/3923487483 >> 172.21.15.27:0/3923487483 conn(0x7f5ba00aa7c0 msgr2=0x7f5ba00aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:47.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.857+0000 7f5ba5400640 1 -- 172.21.15.27:0/3923487483 shutdown_connections 2024-03-28T18:55:47.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.857+0000 7f5ba5400640 1 -- 172.21.15.27:0/3923487483 wait complete. 2024-03-28T18:55:47.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.859+0000 7f5ba5400640 1 Processor -- start 2024-03-28T18:55:47.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.859+0000 7f5ba5400640 1 -- start start 2024-03-28T18:55:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5ba5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba00c8490 0x7f5ba0070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5ba5400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ba0150900 0x7f5ba014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5ba5400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba0153910 0x7f5ba014fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:47.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5ba5400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba0150ce0 con 0x7f5ba00c8490 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5ba5400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba0156f00 con 0x7f5ba0150900 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5ba5400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba0150e60 con 0x7f5ba0153910 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5b9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba00c8490 0x7f5ba0070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5b9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba00c8490 0x7f5ba0070ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59252/0 (socket says 172.21.15.27:59252) 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.860+0000 7f5b9fe00640 1 -- 172.21.15.27:0/877754764 learned_addr learned my addr 172.21.15.27:0/877754764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.861+0000 7f5ba4a00640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba0153910 0x7f5ba014fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.861+0000 7f5b97e00640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ba0150900 0x7f5ba014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.861+0000 7f5b9fe00640 1 -- 172.21.15.27:0/877754764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba0153910 msgr2=0x7f5ba014fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.861+0000 7f5b9fe00640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba0153910 0x7f5ba014fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.861+0000 7f5b9fe00640 1 -- 172.21.15.27:0/877754764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ba0150900 msgr2=0x7f5ba014f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.861+0000 7f5b9fe00640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ba0150900 0x7f5ba014f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.861+0000 7f5b9fe00640 1 -- 172.21.15.27:0/877754764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba0150070 con 0x7f5ba00c8490 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.861+0000 7f5b9fe00640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba00c8490 0x7f5ba0070ce0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b9002b8a0 tx=0x7f5b900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.862+0000 7f5b9cc00640 1 -- 172.21.15.27:0/877754764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b90032860 con 0x7f5ba00c8490 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.862+0000 7f5b9cc00640 1 -- 172.21.15.27:0/877754764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b90032a20 con 0x7f5ba00c8490 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.862+0000 7f5b9cc00640 1 -- 172.21.15.27:0/877754764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b90032e10 con 0x7f5ba00c8490 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.862+0000 7f5ba5400640 1 -- 172.21.15.27:0/877754764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba01dff60 con 0x7f5ba00c8490 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.862+0000 7f5ba5400640 1 -- 172.21.15.27:0/877754764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba006c170 con 0x7f5ba00c8490 2024-03-28T18:55:47.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.863+0000 7f5b95600640 1 -- 172.21.15.27:0/877754764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba004f400 con 0x7f5ba00c8490 2024-03-28T18:55:47.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.864+0000 7f5b9cc00640 1 -- 172.21.15.27:0/877754764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b9004c060 con 0x7f5ba00c8490 2024-03-28T18:55:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.865+0000 7f5b9cc00640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b7c06bbd0 0x7f5b7c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.865+0000 7f5b9cc00640 1 -- 172.21.15.27:0/877754764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f5b900ac550 con 0x7f5ba00c8490 2024-03-28T18:55:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.865+0000 7f5b97e00640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b7c06bbd0 0x7f5b7c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:47.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.866+0000 7f5b97e00640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b7c06bbd0 0x7f5b7c06e090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00c73c0 tx=0x7f5b88008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:47.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:47.870+0000 7f5b9cc00640 1 -- 172.21.15.27:0/877754764 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b90040070 con 0x7f5ba00c8490 2024-03-28T18:55:48.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.120+0000 7f5b95600640 1 -- 172.21.15.27:0/877754764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5ba00c65a0 con 0x7f5ba00c8490 2024-03-28T18:55:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.608+0000 7f5b9cc00640 1 -- 172.21.15.27:0/877754764 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"}]=0 v14) v1 ==== 117+0+0 (secure 0 0 0) 0x7f5b90031e90 con 0x7f5ba00c8490 2024-03-28T18:55:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.611+0000 7f5b95600640 1 -- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b7c06bbd0 msgr2=0x7f5b7c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.611+0000 7f5b95600640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b7c06bbd0 0x7f5b7c06e090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00c73c0 tx=0x7f5b88008040 comp rx=0 tx=0).stop 2024-03-28T18:55:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.611+0000 7f5b95600640 1 -- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba00c8490 msgr2=0x7f5ba0070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.611+0000 7f5b95600640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba00c8490 0x7f5ba0070ce0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b9002b8a0 tx=0x7f5b900046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.612+0000 7f5b95600640 1 -- 172.21.15.27:0/877754764 shutdown_connections 2024-03-28T18:55:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.612+0000 7f5b95600640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b7c06bbd0 0x7f5b7c06e090 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.612+0000 7f5b95600640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ba0153910 0x7f5ba014fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.612+0000 7f5b95600640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ba0150900 0x7f5ba014f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.612+0000 7f5b95600640 1 --2- 172.21.15.27:0/877754764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba00c8490 0x7f5ba0070ce0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.612+0000 7f5b95600640 1 -- 172.21.15.27:0/877754764 >> 172.21.15.27:0/877754764 conn(0x7f5ba00aa7c0 msgr2=0x7f5ba0154640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.612+0000 7f5b95600640 1 -- 172.21.15.27:0/877754764 shutdown_connections 2024-03-28T18:55:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.612+0000 7f5b95600640 1 -- 172.21.15.27:0/877754764 wait complete. 2024-03-28T18:55:48.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b session_timeout 300 2024-03-28T18:55:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.864+0000 7f930b200640 1 Processor -- start 2024-03-28T18:55:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.865+0000 7f930b200640 1 -- start start 2024-03-28T18:55:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.866+0000 7f930b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c156840 0x7f930c153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.866+0000 7f930b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c154740 con 0x7f930c0c4e10 2024-03-28T18:55:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.866+0000 7f930b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c156db0 con 0x7f930c149c20 2024-03-28T18:55:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.866+0000 7f930b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c157090 con 0x7f930c156840 2024-03-28T18:55:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.866+0000 7f930a800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f930c0c4e10 0x7f930c0c5280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59666/0 (socket says 172.21.15.27:59666) 2024-03-28T18:55:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.866+0000 7f930a800640 1 -- 172.21.15.27:0/4049222271 learned_addr learned my addr 172.21.15.27:0/4049222271 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9309400640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c156840 0x7f930c153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932546581 0 0) 0x7f9304001490 con 0x7f930c0c4e10 2024-03-28T18:55:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0003520 con 0x7f930c0c4e10 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097523617 0 0) 0x7f930c156db0 con 0x7f930c149c20 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9304001490 con 0x7f930c149c20 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698543652 0 0) 0x7f92f0003520 con 0x7f930c0c4e10 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c156840 msgr2=0x7f930c153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c156840 0x7f930c153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 >> v1:172.21.15.155:6789/0 conn(0x7f930c149c20 legacy=0x7f930c0a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c157370 con 0x7f930c0c4e10 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.867+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9304001bc0 con 0x7f930c0c4e10 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 >> v1:172.21.15.27:6789/0 conn(0x7f930c0c4e10 legacy=0x7f930c0c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9308a00640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0003b70 0x7f92f0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9308a00640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f0004800 0x7f92f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9308a00640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c156840 0x7f92f00052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c157090 con 0x7f92f0004800 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f000bbf0 con 0x7f92f0003b70 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f000bdf0 con 0x7f930c156840 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9309400640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f0004800 0x7f92f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9309e00640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0003b70 0x7f92f0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f930a800640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c156840 0x7f92f00052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9309400640 1 -- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c156840 msgr2=0x7f92f00052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9309400640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c156840 0x7f92f00052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9309400640 1 -- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0003b70 msgr2=0x7f92f0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9309400640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0003b70 0x7f92f0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.868+0000 7f9309400640 1 -- 172.21.15.27:0/4049222271 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f0003520 con 0x7f92f0004800 2024-03-28T18:55:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.870+0000 7f9309400640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f0004800 0x7f92f0004c80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009a90 tx=0x7f92f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ceb2e7134f5793 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.870+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8039070 con 0x7f92f0004800 2024-03-28T18:55:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.870+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f802c680 con 0x7f92f0004800 2024-03-28T18:55:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.870+0000 7f9308a00640 1 -- 172.21.15.27:0/4049222271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8009690 con 0x7f92f0004800 2024-03-28T18:55:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.871+0000 7f930b200640 1 -- 172.21.15.27:0/4049222271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f0004800 msgr2=0x7f92f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.871+0000 7f930b200640 1 --2- 172.21.15.27:0/4049222271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f0004800 0x7f92f0004c80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009a90 tx=0x7f92f0005830 comp rx=0 tx=0).stop 2024-03-28T18:55:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.871+0000 7f930a800640 1 -- 172.21.15.27:0/4049222271 reap_dead start 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.871+0000 7f930b200640 1 -- 172.21.15.27:0/4049222271 shutdown_connections 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.871+0000 7f930b200640 1 -- 172.21.15.27:0/4049222271 >> 172.21.15.27:0/4049222271 conn(0x7f930c0c7220 msgr2=0x7f930c06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.871+0000 7f930b200640 1 -- 172.21.15.27:0/4049222271 shutdown_connections 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.871+0000 7f930b200640 1 -- 172.21.15.27:0/4049222271 wait complete. 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.872+0000 7f930b200640 1 Processor -- start 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.873+0000 7f930b200640 1 -- start start 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.873+0000 7f930b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0004800 0x7f930c1444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.873+0000 7f930b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c072c20 0x7f930c13e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930c13ee40 0x7f930c1eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0a63e0 con 0x7f930c13ee40 2024-03-28T18:55:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0c5f20 con 0x7f92f0004800 2024-03-28T18:55:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0a6560 con 0x7f930c072c20 2024-03-28T18:55:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930c13ee40 0x7f930c1eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930c13ee40 0x7f930c1eaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59264/0 (socket says 172.21.15.27:59264) 2024-03-28T18:55:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930a800640 1 -- 172.21.15.27:0/1446151082 learned_addr learned my addr 172.21.15.27:0/1446151082 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f9309e00640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0004800 0x7f930c1444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f9309400640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c072c20 0x7f930c13e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930a800640 1 -- 172.21.15.27:0/1446151082 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c072c20 msgr2=0x7f930c13e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930a800640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c072c20 0x7f930c13e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930a800640 1 -- 172.21.15.27:0/1446151082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0004800 msgr2=0x7f930c1444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930a800640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0004800 0x7f930c1444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.874+0000 7f930a800640 1 -- 172.21.15.27:0/1446151082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c13f5c0 con 0x7f930c13ee40 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.875+0000 7f930a800640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930c13ee40 0x7f930c1eaeb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f930c0c69c0 tx=0x7f930400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.875+0000 7f9302000640 1 -- 172.21.15.27:0/1446151082 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304007ad0 con 0x7f930c13ee40 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.875+0000 7f9302000640 1 -- 172.21.15.27:0/1446151082 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9304018070 con 0x7f930c13ee40 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.875+0000 7f9302000640 1 -- 172.21.15.27:0/1446151082 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304013420 con 0x7f930c13ee40 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.875+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c0a9490 con 0x7f930c13ee40 2024-03-28T18:55:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.875+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c0708f0 con 0x7f930c13ee40 2024-03-28T18:55:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.876+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c04f400 con 0x7f930c13ee40 2024-03-28T18:55:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.877+0000 7f9302000640 1 -- 172.21.15.27:0/1446151082 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f93040135c0 con 0x7f930c13ee40 2024-03-28T18:55:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.878+0000 7f9302000640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f92ec06bbd0 0x7f92ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.879+0000 7f9302000640 1 -- 172.21.15.27:0/1446151082 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f930408b1b0 con 0x7f930c13ee40 2024-03-28T18:55:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.879+0000 7f9309e00640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f92ec06bbd0 0x7f92ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.879+0000 7f9309e00640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f92ec06bbd0 0x7f92ec06e090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f92fc007fd0 tx=0x7f92fc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:48.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:48.883+0000 7f9302000640 1 -- 172.21.15.27:0/1446151082 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f930c154740 con 0x7f930c13ee40 2024-03-28T18:55:49.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.128+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f930c0c5ae0 con 0x7f930c13ee40 2024-03-28T18:55:49.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.621+0000 7f9302000640 1 -- 172.21.15.27:0/1446151082 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"}]=0 v15) v1 ==== 110+0+0 (secure 0 0 0) 0x7f9304059a00 con 0x7f930c13ee40 2024-03-28T18:55:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f92ec06bbd0 msgr2=0x7f92ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f92ec06bbd0 0x7f92ec06e090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f92fc007fd0 tx=0x7f92fc009040 comp rx=0 tx=0).stop 2024-03-28T18:55:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930c13ee40 msgr2=0x7f930c1eaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930c13ee40 0x7f930c1eaeb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f930c0c69c0 tx=0x7f930400e530 comp rx=0 tx=0).stop 2024-03-28T18:55:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 shutdown_connections 2024-03-28T18:55:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f92ec06bbd0 0x7f92ec06e090 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f930c13ee40 0x7f930c1eaeb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f930c072c20 0x7f930c13e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 --2- 172.21.15.27:0/1446151082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0004800 0x7f930c1444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 >> 172.21.15.27:0/1446151082 conn(0x7f930c0c7220 msgr2=0x7f930c06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 shutdown_connections 2024-03-28T18:55:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.624+0000 7f930b200640 1 -- 172.21.15.27:0/1446151082 wait complete. 2024-03-28T18:55:49.642 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'b' 2024-03-28T18:55:49.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:55:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.848+0000 7f4e96200640 1 Processor -- start 2024-03-28T18:55:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.848+0000 7f4e96200640 1 -- start start 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.850+0000 7f4e96200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e90152a70 0x7f4e90159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.850+0000 7f4e96200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e90155b40 0x7f4e90155f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.850+0000 7f4e96200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e900c5710 con 0x7f4e900c6cc0 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.850+0000 7f4e96200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e901556b0 con 0x7f4e90152a70 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.850+0000 7f4e96200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900c59f0 con 0x7f4e90155b40 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.851+0000 7f4e94e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e90152a70 0x7f4e90159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.851+0000 7f4e94e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e90152a70 0x7f4e90159580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59276/0 (socket says 172.21.15.27:59276) 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.851+0000 7f4e94e00640 1 -- 172.21.15.27:0/208508897 learned_addr learned my addr 172.21.15.27:0/208508897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.851+0000 7f4e95800640 1 --2- 172.21.15.27:0/208508897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e90155b40 0x7f4e90155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.851+0000 7f4e94e00640 1 -- 172.21.15.27:0/208508897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e90155b40 msgr2=0x7f4e90155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.851+0000 7f4e94e00640 1 --2- 172.21.15.27:0/208508897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e90155b40 0x7f4e90155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.851+0000 7f4e94e00640 1 -- 172.21.15.27:0/208508897 >> v1:172.21.15.155:6789/0 conn(0x7f4e900c6cc0 legacy=0x7f4e900cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.851+0000 7f4e94e00640 1 -- 172.21.15.27:0/208508897 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e900c5cd0 con 0x7f4e90152a70 2024-03-28T18:55:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.852+0000 7f4e94e00640 1 --2- 172.21.15.27:0/208508897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90152a70 0x7f4e90159580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009870 tx=0x7f4e8402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf2af9c2e075c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e8f400640 1 -- 172.21.15.27:0/208508897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84035020 con 0x7f4e90152a70 2024-03-28T18:55:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e8f400640 1 -- 172.21.15.27:0/208508897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e84031640 con 0x7f4e90152a70 2024-03-28T18:55:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e8f400640 1 -- 172.21.15.27:0/208508897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8402d040 con 0x7f4e90152a70 2024-03-28T18:55:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e96200640 1 -- 172.21.15.27:0/208508897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90152a70 msgr2=0x7f4e90159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e96200640 1 --2- 172.21.15.27:0/208508897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90152a70 0x7f4e90159580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009870 tx=0x7f4e8402b8f0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e96200640 1 -- 172.21.15.27:0/208508897 shutdown_connections 2024-03-28T18:55:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e96200640 1 --2- 172.21.15.27:0/208508897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e90155b40 0x7f4e90155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e96200640 1 --2- 172.21.15.27:0/208508897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90152a70 0x7f4e90159580 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e96200640 1 -- 172.21.15.27:0/208508897 >> 172.21.15.27:0/208508897 conn(0x7f4e9006b2a0 msgr2=0x7f4e9006b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e96200640 1 -- 172.21.15.27:0/208508897 shutdown_connections 2024-03-28T18:55:49.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.853+0000 7f4e96200640 1 -- 172.21.15.27:0/208508897 wait complete. 2024-03-28T18:55:49.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.855+0000 7f4e96200640 1 Processor -- start 2024-03-28T18:55:49.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.855+0000 7f4e96200640 1 -- start start 2024-03-28T18:55:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e96200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e900c6cc0 0x7f4e901eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e96200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e90152a70 0x7f4e9014a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e96200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90155b40 0x7f4e900aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e96200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900cb7f0 con 0x7f4e90155b40 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e96200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e90072a70 con 0x7f4e90152a70 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e96200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e900cb970 con 0x7f4e900c6cc0 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e95800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90155b40 0x7f4e900aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e95800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90155b40 0x7f4e900aaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59280/0 (socket says 172.21.15.27:59280) 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.856+0000 7f4e95800640 1 -- 172.21.15.27:0/3693116727 learned_addr learned my addr 172.21.15.27:0/3693116727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e94e00640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e900c6cc0 0x7f4e901eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e8fe00640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e90152a70 0x7f4e9014a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e95800640 1 -- 172.21.15.27:0/3693116727 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e900c6cc0 msgr2=0x7f4e901eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e95800640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e900c6cc0 0x7f4e901eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e95800640 1 -- 172.21.15.27:0/3693116727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e90152a70 msgr2=0x7f4e9014a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e95800640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e90152a70 0x7f4e9014a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e95800640 1 -- 172.21.15.27:0/3693116727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e90153800 con 0x7f4e90155b40 2024-03-28T18:55:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e95800640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90155b40 0x7f4e900aaf10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e88007e60 tx=0x7f4e8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.857+0000 7f4e8cc00640 1 -- 172.21.15.27:0/3693116727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88014420 con 0x7f4e90155b40 2024-03-28T18:55:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.858+0000 7f4e8cc00640 1 -- 172.21.15.27:0/3693116727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e88019070 con 0x7f4e90155b40 2024-03-28T18:55:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.858+0000 7f4e8cc00640 1 -- 172.21.15.27:0/3693116727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88014820 con 0x7f4e90155b40 2024-03-28T18:55:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.858+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e90153ae0 con 0x7f4e90155b40 2024-03-28T18:55:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.858+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9014ad00 con 0x7f4e90155b40 2024-03-28T18:55:49.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.859+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e90067dd0 con 0x7f4e90155b40 2024-03-28T18:55:49.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.860+0000 7f4e8cc00640 1 -- 172.21.15.27:0/3693116727 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4e880149c0 con 0x7f4e90155b40 2024-03-28T18:55:49.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.861+0000 7f4e8cc00640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e7006bbd0 0x7f4e7006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:49.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.861+0000 7f4e8cc00640 1 -- 172.21.15.27:0/3693116727 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f4e8808b610 con 0x7f4e90155b40 2024-03-28T18:55:49.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.861+0000 7f4e94e00640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e7006bbd0 0x7f4e7006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:49.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.862+0000 7f4e94e00640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e7006bbd0 0x7f4e7006e090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e84036000 tx=0x7f4e8403c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:49.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:49.866+0000 7f4e8cc00640 1 -- 172.21.15.27:0/3693116727 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4e88059dc0 con 0x7f4e90155b40 2024-03-28T18:55:50.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.228+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e90069640 con 0x7f4e90155b40 2024-03-28T18:55:50.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.229+0000 7f4e8cc00640 1 -- 172.21.15.27:0/3693116727 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f4e880596c0 con 0x7f4e90155b40 2024-03-28T18:55:50.229 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:50.229 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:55:50.230 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:55:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e7006bbd0 msgr2=0x7f4e7006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e7006bbd0 0x7f4e7006e090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e84036000 tx=0x7f4e8403c040 comp rx=0 tx=0).stop 2024-03-28T18:55:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90155b40 msgr2=0x7f4e900aaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90155b40 0x7f4e900aaf10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e88007e60 tx=0x7f4e8800e4b0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 shutdown_connections 2024-03-28T18:55:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e7006bbd0 0x7f4e7006e090 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e90155b40 0x7f4e900aaf10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e90152a70 0x7f4e9014a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 --2- 172.21.15.27:0/3693116727 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e900c6cc0 0x7f4e901eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 >> 172.21.15.27:0/3693116727 conn(0x7f4e9006b2a0 msgr2=0x7f4e9006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 shutdown_connections 2024-03-28T18:55:50.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.232+0000 7f4e96200640 1 -- 172.21.15.27:0/3693116727 wait complete. 2024-03-28T18:55:50.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.458+0000 7f2791600640 1 Processor -- start 2024-03-28T18:55:50.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.459+0000 7f2791600640 1 -- start start 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.460+0000 7f2791600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c06fc00 0x7f278c0c3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.460+0000 7f2791600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f278c066dd0 0x7f278c150530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.460+0000 7f2791600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c0c4080 con 0x7f278c153fb0 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.460+0000 7f2791600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278c153b20 con 0x7f278c066dd0 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.460+0000 7f2791600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278c1534d0 con 0x7f278c06fc00 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.460+0000 7f2790c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f278c066dd0 0x7f278c150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.461+0000 7f278b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f278c153fb0 0x7f278c156830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59678/0 (socket says 172.21.15.27:59678) 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.461+0000 7f278b400640 1 -- 172.21.15.27:0/2072202944 learned_addr learned my addr 172.21.15.27:0/2072202944 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.461+0000 7f278be00640 1 --2- 172.21.15.27:0/2072202944 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c06fc00 0x7f278c0c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.461+0000 7f2790c00640 1 -- 172.21.15.27:0/2072202944 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c06fc00 msgr2=0x7f278c0c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.461+0000 7f2790c00640 1 --2- 172.21.15.27:0/2072202944 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c06fc00 0x7f278c0c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.461+0000 7f2790c00640 1 -- 172.21.15.27:0/2072202944 >> v1:172.21.15.27:6789/0 conn(0x7f278c153fb0 legacy=0x7f278c156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.461+0000 7f2790c00640 1 -- 172.21.15.27:0/2072202944 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c1537b0 con 0x7f278c066dd0 2024-03-28T18:55:50.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.461+0000 7f278aa00640 1 -- 172.21.15.27:0/2072202944 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667768399 0 0) 0x7f278c0c4080 con 0x7f278c153fb0 2024-03-28T18:55:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.462+0000 7f2790c00640 1 --2- 172.21.15.27:0/2072202944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c066dd0 0x7f278c150530 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f27840097a0 tx=0x7f278402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23171f5179b6695a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f278aa00640 1 -- 172.21.15.27:0/2072202944 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27840315a0 con 0x7f278c066dd0 2024-03-28T18:55:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f278aa00640 1 -- 172.21.15.27:0/2072202944 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2784037070 con 0x7f278c066dd0 2024-03-28T18:55:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f278aa00640 1 -- 172.21.15.27:0/2072202944 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784031960 con 0x7f278c066dd0 2024-03-28T18:55:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f2791600640 1 -- 172.21.15.27:0/2072202944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c066dd0 msgr2=0x7f278c150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f2791600640 1 --2- 172.21.15.27:0/2072202944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c066dd0 0x7f278c150530 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f27840097a0 tx=0x7f278402b840 comp rx=0 tx=0).stop 2024-03-28T18:55:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f2791600640 1 -- 172.21.15.27:0/2072202944 shutdown_connections 2024-03-28T18:55:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f2791600640 1 --2- 172.21.15.27:0/2072202944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c066dd0 0x7f278c150530 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f2791600640 1 --2- 172.21.15.27:0/2072202944 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c06fc00 0x7f278c0c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f2791600640 1 -- 172.21.15.27:0/2072202944 >> 172.21.15.27:0/2072202944 conn(0x7f278c06b0c0 msgr2=0x7f278c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f2791600640 1 -- 172.21.15.27:0/2072202944 shutdown_connections 2024-03-28T18:55:50.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.463+0000 7f2791600640 1 -- 172.21.15.27:0/2072202944 wait complete. 2024-03-28T18:55:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.465+0000 7f2791600640 1 Processor -- start 2024-03-28T18:55:50.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.466+0000 7f2791600640 1 -- start start 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.466+0000 7f2791600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f278c066dd0 0x7f278c155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.466+0000 7f2791600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c06fc00 0x7f278c149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f2791600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c153fb0 0x7f278c14a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f2791600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c1570c0 con 0x7f278c06fc00 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f2791600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c156f40 con 0x7f278c066dd0 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f2791600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278c06ffe0 con 0x7f278c153fb0 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c06fc00 0x7f278c149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c06fc00 0x7f278c149d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59300/0 (socket says 172.21.15.27:59300) 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278b400640 1 -- 172.21.15.27:0/3481037635 learned_addr learned my addr 172.21.15.27:0/3481037635 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278be00640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f278c066dd0 0x7f278c155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f2790c00640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c153fb0 0x7f278c14a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278b400640 1 -- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c153fb0 msgr2=0x7f278c14a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278b400640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c153fb0 0x7f278c14a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278b400640 1 -- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f278c066dd0 msgr2=0x7f278c155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278b400640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f278c066dd0 0x7f278c155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f278b400640 1 -- 172.21.15.27:0/3481037635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c06b510 con 0x7f278c06fc00 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.467+0000 7f2790c00640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c153fb0 0x7f278c14a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.468+0000 7f278b400640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c06fc00 0x7f278c149d70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f277400c820 tx=0x7f277400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.468+0000 7f2783e00640 1 -- 172.21.15.27:0/3481037635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774018070 con 0x7f278c06fc00 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.468+0000 7f2783e00640 1 -- 172.21.15.27:0/3481037635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277400a640 con 0x7f278c06fc00 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.468+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c1eaff0 con 0x7f278c06fc00 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.468+0000 7f2783e00640 1 -- 172.21.15.27:0/3481037635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277400aa30 con 0x7f278c06fc00 2024-03-28T18:55:50.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.468+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c0c1c10 con 0x7f278c06fc00 2024-03-28T18:55:50.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.469+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c069640 con 0x7f278c06fc00 2024-03-28T18:55:50.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.470+0000 7f2783e00640 1 -- 172.21.15.27:0/3481037635 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f277401b070 con 0x7f278c06fc00 2024-03-28T18:55:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.471+0000 7f2783e00640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f276806bbd0 0x7f276806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.471+0000 7f2783e00640 1 -- 172.21.15.27:0/3481037635 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f277408a500 con 0x7f278c06fc00 2024-03-28T18:55:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.471+0000 7f278be00640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f276806bbd0 0x7f276806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.472+0000 7f278be00640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f276806bbd0 0x7f276806e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f277c006fd0 tx=0x7f277c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:50.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.476+0000 7f2783e00640 1 -- 172.21.15.27:0/3481037635 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2774058c20 con 0x7f278c06fc00 2024-03-28T18:55:50.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.687+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f278c0c19c0 con 0x7f278c06fc00 2024-03-28T18:55:50.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.689+0000 7f2783e00640 1 -- 172.21.15.27:0/3481037635 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+20307 (secure 0 0 0) 0x7f2774058520 con 0x7f278c06fc00 2024-03-28T18:55:50.689 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:50.690 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:46.587251+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:50.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f276806bbd0 msgr2=0x7f276806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f276806bbd0 0x7f276806e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f277c006fd0 tx=0x7f277c008040 comp rx=0 tx=0).stop 2024-03-28T18:55:50.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c06fc00 msgr2=0x7f278c149d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c06fc00 0x7f278c149d70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f277400c820 tx=0x7f277400cce0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 shutdown_connections 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f276806bbd0 0x7f276806e090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f278c153fb0 0x7f278c14a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f278c06fc00 0x7f278c149d70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 --2- 172.21.15.27:0/3481037635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f278c066dd0 0x7f278c155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 >> 172.21.15.27:0/3481037635 conn(0x7f278c06b0c0 msgr2=0x7f278c0a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 shutdown_connections 2024-03-28T18:55:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.692+0000 7f2791600640 1 -- 172.21.15.27:0/3481037635 wait complete. 2024-03-28T18:55:50.711 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T18:55:50.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:55:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.921+0000 7fdf8f600640 1 Processor -- start 2024-03-28T18:55:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.922+0000 7fdf8f600640 1 -- start start 2024-03-28T18:55:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.923+0000 7fdf8f600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88159650 0x7fdf880c1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.923+0000 7fdf8f600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdf88155d60 0x7fdf881561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.923+0000 7fdf8f600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf880c7a10 con 0x7fdf88158cb0 2024-03-28T18:55:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.923+0000 7fdf8f600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf880c7170 con 0x7fdf88155d60 2024-03-28T18:55:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.923+0000 7fdf8f600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf880c7450 con 0x7fdf88159650 2024-03-28T18:55:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.924+0000 7fdf8cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88159650 0x7fdf880c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.924+0000 7fdf87e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fdf88158cb0 0x7fdf8806c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55540/0 (socket says 172.21.15.27:55540) 2024-03-28T18:55:50.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.924+0000 7fdf87e00640 1 -- 172.21.15.27:0/1630157832 learned_addr learned my addr 172.21.15.27:0/1630157832 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:50.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.924+0000 7fdf8d600640 1 --2- 172.21.15.27:0/1630157832 >> v2:172.21.15.155:3300/0 conn(0x7fdf88155d60 0x7fdf881561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.925+0000 7fdf8d600640 1 -- 172.21.15.27:0/1630157832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88159650 msgr2=0x7fdf880c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.925+0000 7fdf8d600640 1 --2- 172.21.15.27:0/1630157832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88159650 0x7fdf880c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.925+0000 7fdf8d600640 1 -- 172.21.15.27:0/1630157832 >> v1:172.21.15.155:6789/0 conn(0x7fdf88158cb0 legacy=0x7fdf8806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.925+0000 7fdf8d600640 1 -- 172.21.15.27:0/1630157832 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf880c7730 con 0x7fdf88155d60 2024-03-28T18:55:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.926+0000 7fdf8d600640 1 --2- 172.21.15.27:0/1630157832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88155d60 0x7fdf881561f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009d90 tx=0x7fdf8002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b4da01d9d6106ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.926+0000 7fdf87400640 1 -- 172.21.15.27:0/1630157832 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80039070 con 0x7fdf88155d60 2024-03-28T18:55:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.926+0000 7fdf87400640 1 -- 172.21.15.27:0/1630157832 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8002cc60 con 0x7fdf88155d60 2024-03-28T18:55:50.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.926+0000 7fdf87400640 1 -- 172.21.15.27:0/1630157832 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80034660 con 0x7fdf88155d60 2024-03-28T18:55:50.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.927+0000 7fdf8f600640 1 -- 172.21.15.27:0/1630157832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88155d60 msgr2=0x7fdf881561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.927+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1630157832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88155d60 0x7fdf881561f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009d90 tx=0x7fdf8002beb0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.927+0000 7fdf8f600640 1 -- 172.21.15.27:0/1630157832 shutdown_connections 2024-03-28T18:55:50.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.927+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1630157832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88155d60 0x7fdf881561f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.927+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1630157832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88159650 0x7fdf880c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.927+0000 7fdf8f600640 1 -- 172.21.15.27:0/1630157832 >> 172.21.15.27:0/1630157832 conn(0x7fdf880c4750 msgr2=0x7fdf880c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:50.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.927+0000 7fdf8f600640 1 -- 172.21.15.27:0/1630157832 shutdown_connections 2024-03-28T18:55:50.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.927+0000 7fdf8f600640 1 -- 172.21.15.27:0/1630157832 wait complete. 2024-03-28T18:55:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.929+0000 7fdf8f600640 1 Processor -- start 2024-03-28T18:55:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.929+0000 7fdf8f600640 1 -- start start 2024-03-28T18:55:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.930+0000 7fdf8f600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88155d60 0x7fdf88152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.930+0000 7fdf8f600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88158cb0 0x7fdf880a8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.930+0000 7fdf8f600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88159650 0x7fdf881ded20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.930+0000 7fdf8f600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf880c1c30 con 0x7fdf88159650 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.930+0000 7fdf8f600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf880c1ab0 con 0x7fdf88158cb0 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.930+0000 7fdf8f600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8806c7e0 con 0x7fdf88155d60 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88159650 0x7fdf881ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88159650 0x7fdf881ded20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59308/0 (socket says 172.21.15.27:59308) 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8d600640 1 -- 172.21.15.27:0/1214312468 learned_addr learned my addr 172.21.15.27:0/1214312468 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf87e00640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88158cb0 0x7fdf880a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8cc00640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88155d60 0x7fdf88152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8d600640 1 -- 172.21.15.27:0/1214312468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88155d60 msgr2=0x7fdf88152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8d600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88155d60 0x7fdf88152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8d600640 1 -- 172.21.15.27:0/1214312468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88158cb0 msgr2=0x7fdf880a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8d600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88158cb0 0x7fdf880a8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.931+0000 7fdf8d600640 1 -- 172.21.15.27:0/1214312468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf881ef560 con 0x7fdf88159650 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.932+0000 7fdf8d600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88159650 0x7fdf881ded20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009eb0 tx=0x7fdf80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.932+0000 7fdf84c00640 1 -- 172.21.15.27:0/1214312468 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80039050 con 0x7fdf88159650 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.932+0000 7fdf84c00640 1 -- 172.21.15.27:0/1214312468 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80045070 con 0x7fdf88159650 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.932+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf880a7370 con 0x7fdf88159650 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.932+0000 7fdf84c00640 1 -- 172.21.15.27:0/1214312468 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80009450 con 0x7fdf88159650 2024-03-28T18:55:50.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.932+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf881ef280 con 0x7fdf88159650 2024-03-28T18:55:50.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.933+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8804f400 con 0x7fdf88159650 2024-03-28T18:55:50.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.934+0000 7fdf84c00640 1 -- 172.21.15.27:0/1214312468 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf8003e850 con 0x7fdf88159650 2024-03-28T18:55:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.935+0000 7fdf84c00640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.935+0000 7fdf84c00640 1 -- 172.21.15.27:0/1214312468 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fdf800adc60 con 0x7fdf88159650 2024-03-28T18:55:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.936+0000 7fdf8cc00640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:50.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.936+0000 7fdf8cc00640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009a10 tx=0x7fdf7c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:50.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:50.944+0000 7fdf84c00640 1 -- 172.21.15.27:0/1214312468 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf8007c530 con 0x7fdf88159650 2024-03-28T18:55:51.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.144+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf880a7540 con 0x7fdf6c06bbd0 2024-03-28T18:55:51.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.161+0000 7fdf84c00640 1 -- 172.21.15.27:0/1214312468 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490417 (secure 0 0 0) 0x7fdf880a7540 con 0x7fdf6c06bbd0 2024-03-28T18:55:51.169 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:55:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.170+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf6c06bbd0 msgr2=0x7fdf6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.170+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009a10 tx=0x7fdf7c008040 comp rx=0 tx=0).stop 2024-03-28T18:55:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.170+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88159650 msgr2=0x7fdf881ded20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.170+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88159650 0x7fdf881ded20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009eb0 tx=0x7fdf80004a40 comp rx=0 tx=0).stop 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.171+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 shutdown_connections 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.171+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf6c06bbd0 0x7fdf6c06e090 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.171+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf88159650 0x7fdf881ded20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.171+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf88158cb0 0x7fdf880a8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.171+0000 7fdf8f600640 1 --2- 172.21.15.27:0/1214312468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf88155d60 0x7fdf88152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.171+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 >> 172.21.15.27:0/1214312468 conn(0x7fdf880c4750 msgr2=0x7fdf88071a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.171+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 shutdown_connections 2024-03-28T18:55:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.171+0000 7fdf8f600640 1 -- 172.21.15.27:0/1214312468 wait complete. 2024-03-28T18:55:51.179 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T18:55:51.171+0000 7f7303400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:55:51.190 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-28T18:55:50.896791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":316,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219448,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755272,"statfs":{"total":773094113280,"available":772869398528,"internally_reserved":0,"allocated":3350528,"data_stored":1984464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.326228"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603978+0000","last_change":"2024-03-28T18:55:45.603782+0000","last_active":"2024-03-28T18:55:46.603978+0000","last_peered":"2024-03-28T18:55:46.603978+0000","last_clean":"2024-03-28T18:55:46.603978+0000","last_became_active":"2024-03-28T18:55:45.603420+0000","last_became_peered":"2024-03-28T18:55:45.603420+0000","last_unstale":"2024-03-28T18:55:46.603978+0000","last_undegraded":"2024-03-28T18:55:46.603978+0000","last_fullsized":"2024-03-28T18:55:46.603978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609332+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:55:46.609332+0000","last_peered":"2024-03-28T18:55:46.609332+0000","last_clean":"2024-03-28T18:55:46.609332+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:55:46.609332+0000","last_undegraded":"2024-03-28T18:55:46.609332+0000","last_fullsized":"2024-03-28T18:55:46.609332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828788+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:55:46.828788+0000","last_peered":"2024-03-28T18:55:46.828788+0000","last_clean":"2024-03-28T18:55:46.828788+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:55:46.828788+0000","last_undegraded":"2024-03-28T18:55:46.828788+0000","last_fullsized":"2024-03-28T18:55:46.828788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825968+0000","last_change":"2024-03-28T18:55:44.602536+0000","last_active":"2024-03-28T18:55:46.825968+0000","last_peered":"2024-03-28T18:55:46.825968+0000","last_clean":"2024-03-28T18:55:46.825968+0000","last_became_active":"2024-03-28T18:55:44.602256+0000","last_became_peered":"2024-03-28T18:55:44.602256+0000","last_unstale":"2024-03-28T18:55:46.825968+0000","last_undegraded":"2024-03-28T18:55:46.825968+0000","last_fullsized":"2024-03-28T18:55:46.825968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612952+0000","last_change":"2024-03-28T18:55:45.625338+0000","last_active":"2024-03-28T18:55:46.612952+0000","last_peered":"2024-03-28T18:55:46.612952+0000","last_clean":"2024-03-28T18:55:46.612952+0000","last_became_active":"2024-03-28T18:55:45.625067+0000","last_became_peered":"2024-03-28T18:55:45.625067+0000","last_unstale":"2024-03-28T18:55:46.612952+0000","last_undegraded":"2024-03-28T18:55:46.612952+0000","last_fullsized":"2024-03-28T18:55:46.612952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829113+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:55:46.829113+0000","last_peered":"2024-03-28T18:55:46.829113+0000","last_clean":"2024-03-28T18:55:46.829113+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:55:46.829113+0000","last_undegraded":"2024-03-28T18:55:46.829113+0000","last_fullsized":"2024-03-28T18:55:46.829113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608925+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:55:46.608925+0000","last_peered":"2024-03-28T18:55:46.608925+0000","last_clean":"2024-03-28T18:55:46.608925+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:55:46.608925+0000","last_undegraded":"2024-03-28T18:55:46.608925+0000","last_fullsized":"2024-03-28T18:55:46.608925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604171+0000","last_change":"2024-03-28T18:55:44.598423+0000","last_active":"2024-03-28T18:55:46.604171+0000","last_peered":"2024-03-28T18:55:46.604171+0000","last_clean":"2024-03-28T18:55:46.604171+0000","last_became_active":"2024-03-28T18:55:44.598092+0000","last_became_peered":"2024-03-28T18:55:44.598092+0000","last_unstale":"2024-03-28T18:55:46.604171+0000","last_undegraded":"2024-03-28T18:55:46.604171+0000","last_fullsized":"2024-03-28T18:55:46.604171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.646619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163641+0000","last_change":"2024-03-28T18:55:45.608201+0000","last_active":"2024-03-28T18:55:47.163641+0000","last_peered":"2024-03-28T18:55:47.163641+0000","last_clean":"2024-03-28T18:55:47.163641+0000","last_became_active":"2024-03-28T18:55:45.607912+0000","last_became_peered":"2024-03-28T18:55:45.607912+0000","last_unstale":"2024-03-28T18:55:47.163641+0000","last_undegraded":"2024-03-28T18:55:47.163641+0000","last_fullsized":"2024-03-28T18:55:47.163641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164845+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:55:47.164845+0000","last_peered":"2024-03-28T18:55:47.164845+0000","last_clean":"2024-03-28T18:55:47.164845+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:55:47.164845+0000","last_undegraded":"2024-03-28T18:55:47.164845+0000","last_fullsized":"2024-03-28T18:55:47.164845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117090+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:55:47.117090+0000","last_peered":"2024-03-28T18:55:47.117090+0000","last_clean":"2024-03-28T18:55:47.117090+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:55:47.117090+0000","last_undegraded":"2024-03-28T18:55:47.117090+0000","last_fullsized":"2024-03-28T18:55:47.117090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604780+0000","last_change":"2024-03-28T18:55:44.604673+0000","last_active":"2024-03-28T18:55:46.604780+0000","last_peered":"2024-03-28T18:55:46.604780+0000","last_clean":"2024-03-28T18:55:46.604780+0000","last_became_active":"2024-03-28T18:55:44.603926+0000","last_became_peered":"2024-03-28T18:55:44.603926+0000","last_unstale":"2024-03-28T18:55:46.604780+0000","last_undegraded":"2024-03-28T18:55:46.604780+0000","last_fullsized":"2024-03-28T18:55:46.604780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116402+0000","last_change":"2024-03-28T18:55:45.610033+0000","last_active":"2024-03-28T18:55:47.116402+0000","last_peered":"2024-03-28T18:55:47.116402+0000","last_clean":"2024-03-28T18:55:47.116402+0000","last_became_active":"2024-03-28T18:55:45.609706+0000","last_became_peered":"2024-03-28T18:55:45.609706+0000","last_unstale":"2024-03-28T18:55:47.116402+0000","last_undegraded":"2024-03-28T18:55:47.116402+0000","last_fullsized":"2024-03-28T18:55:47.116402+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164190+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:55:47.164190+0000","last_peered":"2024-03-28T18:55:47.164190+0000","last_clean":"2024-03-28T18:55:47.164190+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:55:47.164190+0000","last_undegraded":"2024-03-28T18:55:47.164190+0000","last_fullsized":"2024-03-28T18:55:47.164190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117556+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:55:47.117556+0000","last_peered":"2024-03-28T18:55:47.117556+0000","last_clean":"2024-03-28T18:55:47.117556+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:55:47.117556+0000","last_undegraded":"2024-03-28T18:55:47.117556+0000","last_fullsized":"2024-03-28T18:55:47.117556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604505+0000","last_change":"2024-03-28T18:55:44.606715+0000","last_active":"2024-03-28T18:55:46.604505+0000","last_peered":"2024-03-28T18:55:46.604505+0000","last_clean":"2024-03-28T18:55:46.604505+0000","last_became_active":"2024-03-28T18:55:44.606343+0000","last_became_peered":"2024-03-28T18:55:44.606343+0000","last_unstale":"2024-03-28T18:55:46.604505+0000","last_undegraded":"2024-03-28T18:55:46.604505+0000","last_fullsized":"2024-03-28T18:55:46.604505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:51.471450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613822+0000","last_change":"2024-03-28T18:55:45.623483+0000","last_active":"2024-03-28T18:55:46.613822+0000","last_peered":"2024-03-28T18:55:46.613822+0000","last_clean":"2024-03-28T18:55:46.613822+0000","last_became_active":"2024-03-28T18:55:45.622894+0000","last_became_peered":"2024-03-28T18:55:45.622894+0000","last_unstale":"2024-03-28T18:55:46.613822+0000","last_undegraded":"2024-03-28T18:55:46.613822+0000","last_fullsized":"2024-03-28T18:55:46.613822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603964+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:55:46.603964+0000","last_peered":"2024-03-28T18:55:46.603964+0000","last_clean":"2024-03-28T18:55:46.603964+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:55:46.603964+0000","last_undegraded":"2024-03-28T18:55:46.603964+0000","last_fullsized":"2024-03-28T18:55:46.603964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164866+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:55:47.164866+0000","last_peered":"2024-03-28T18:55:47.164866+0000","last_clean":"2024-03-28T18:55:47.164866+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:55:47.164866+0000","last_undegraded":"2024-03-28T18:55:47.164866+0000","last_fullsized":"2024-03-28T18:55:47.164866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605818+0000","last_change":"2024-03-28T18:55:44.601415+0000","last_active":"2024-03-28T18:55:46.605818+0000","last_peered":"2024-03-28T18:55:46.605818+0000","last_clean":"2024-03-28T18:55:46.605818+0000","last_became_active":"2024-03-28T18:55:44.601124+0000","last_became_peered":"2024-03-28T18:55:44.601124+0000","last_unstale":"2024-03-28T18:55:46.605818+0000","last_undegraded":"2024-03-28T18:55:46.605818+0000","last_fullsized":"2024-03-28T18:55:46.605818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:30.767034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603094+0000","last_change":"2024-03-28T18:55:45.604795+0000","last_active":"2024-03-28T18:55:46.603094+0000","last_peered":"2024-03-28T18:55:46.603094+0000","last_clean":"2024-03-28T18:55:46.603094+0000","last_became_active":"2024-03-28T18:55:45.604432+0000","last_became_peered":"2024-03-28T18:55:45.604432+0000","last_unstale":"2024-03-28T18:55:46.603094+0000","last_undegraded":"2024-03-28T18:55:46.603094+0000","last_fullsized":"2024-03-28T18:55:46.603094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:37.638804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826963+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:55:46.826963+0000","last_peered":"2024-03-28T18:55:46.826963+0000","last_clean":"2024-03-28T18:55:46.826963+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:55:46.826963+0000","last_undegraded":"2024-03-28T18:55:46.826963+0000","last_fullsized":"2024-03-28T18:55:46.826963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613559+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:55:46.613559+0000","last_peered":"2024-03-28T18:55:46.613559+0000","last_clean":"2024-03-28T18:55:46.613559+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:55:46.613559+0000","last_undegraded":"2024-03-28T18:55:46.613559+0000","last_fullsized":"2024-03-28T18:55:46.613559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605718+0000","last_change":"2024-03-28T18:55:44.603859+0000","last_active":"2024-03-28T18:55:46.605718+0000","last_peered":"2024-03-28T18:55:46.605718+0000","last_clean":"2024-03-28T18:55:46.605718+0000","last_became_active":"2024-03-28T18:55:44.603540+0000","last_became_peered":"2024-03-28T18:55:44.603540+0000","last_unstale":"2024-03-28T18:55:46.605718+0000","last_undegraded":"2024-03-28T18:55:46.605718+0000","last_fullsized":"2024-03-28T18:55:46.605718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.606069+0000","last_change":"2024-03-28T18:55:45.608142+0000","last_active":"2024-03-28T18:55:46.606069+0000","last_peered":"2024-03-28T18:55:46.606069+0000","last_clean":"2024-03-28T18:55:46.606069+0000","last_became_active":"2024-03-28T18:55:45.606117+0000","last_became_peered":"2024-03-28T18:55:45.606117+0000","last_unstale":"2024-03-28T18:55:46.606069+0000","last_undegraded":"2024-03-28T18:55:46.606069+0000","last_fullsized":"2024-03-28T18:55:46.606069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826653+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:55:46.826653+0000","last_peered":"2024-03-28T18:55:46.826653+0000","last_clean":"2024-03-28T18:55:46.826653+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:55:46.826653+0000","last_undegraded":"2024-03-28T18:55:46.826653+0000","last_fullsized":"2024-03-28T18:55:46.826653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613045+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:55:46.613045+0000","last_peered":"2024-03-28T18:55:46.613045+0000","last_clean":"2024-03-28T18:55:46.613045+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:55:46.613045+0000","last_undegraded":"2024-03-28T18:55:46.613045+0000","last_fullsized":"2024-03-28T18:55:46.613045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614066+0000","last_change":"2024-03-28T18:55:44.606690+0000","last_active":"2024-03-28T18:55:46.614066+0000","last_peered":"2024-03-28T18:55:46.614066+0000","last_clean":"2024-03-28T18:55:46.614066+0000","last_became_active":"2024-03-28T18:55:44.606421+0000","last_became_peered":"2024-03-28T18:55:44.606421+0000","last_unstale":"2024-03-28T18:55:46.614066+0000","last_undegraded":"2024-03-28T18:55:46.614066+0000","last_fullsized":"2024-03-28T18:55:46.614066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117505+0000","last_change":"2024-03-28T18:55:45.610580+0000","last_active":"2024-03-28T18:55:47.117505+0000","last_peered":"2024-03-28T18:55:47.117505+0000","last_clean":"2024-03-28T18:55:47.117505+0000","last_became_active":"2024-03-28T18:55:45.610316+0000","last_became_peered":"2024-03-28T18:55:45.610316+0000","last_unstale":"2024-03-28T18:55:47.117505+0000","last_undegraded":"2024-03-28T18:55:47.117505+0000","last_fullsized":"2024-03-28T18:55:47.117505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604997+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:55:46.604997+0000","last_peered":"2024-03-28T18:55:46.604997+0000","last_clean":"2024-03-28T18:55:46.604997+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:55:46.604997+0000","last_undegraded":"2024-03-28T18:55:46.604997+0000","last_fullsized":"2024-03-28T18:55:46.604997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163833+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:55:47.163833+0000","last_peered":"2024-03-28T18:55:47.163833+0000","last_clean":"2024-03-28T18:55:47.163833+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:55:47.163833+0000","last_undegraded":"2024-03-28T18:55:47.163833+0000","last_fullsized":"2024-03-28T18:55:47.163833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.606207+0000","last_change":"2024-03-28T18:55:44.600523+0000","last_active":"2024-03-28T18:55:46.606207+0000","last_peered":"2024-03-28T18:55:46.606207+0000","last_clean":"2024-03-28T18:55:46.606207+0000","last_became_active":"2024-03-28T18:55:44.597550+0000","last_became_peered":"2024-03-28T18:55:44.597550+0000","last_unstale":"2024-03-28T18:55:46.606207+0000","last_undegraded":"2024-03-28T18:55:46.606207+0000","last_fullsized":"2024-03-28T18:55:46.606207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825927+0000","last_change":"2024-03-28T18:55:45.619798+0000","last_active":"2024-03-28T18:55:46.825927+0000","last_peered":"2024-03-28T18:55:46.825927+0000","last_clean":"2024-03-28T18:55:46.825927+0000","last_became_active":"2024-03-28T18:55:45.616517+0000","last_became_peered":"2024-03-28T18:55:45.616517+0000","last_unstale":"2024-03-28T18:55:46.825927+0000","last_undegraded":"2024-03-28T18:55:46.825927+0000","last_fullsized":"2024-03-28T18:55:46.825927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609158+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:55:46.609158+0000","last_peered":"2024-03-28T18:55:46.609158+0000","last_clean":"2024-03-28T18:55:46.609158+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:55:46.609158+0000","last_undegraded":"2024-03-28T18:55:46.609158+0000","last_fullsized":"2024-03-28T18:55:46.609158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605113+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:55:46.605113+0000","last_peered":"2024-03-28T18:55:46.605113+0000","last_clean":"2024-03-28T18:55:46.605113+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:55:46.605113+0000","last_undegraded":"2024-03-28T18:55:46.605113+0000","last_fullsized":"2024-03-28T18:55:46.605113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604527+0000","last_change":"2024-03-28T18:55:44.606498+0000","last_active":"2024-03-28T18:55:46.604527+0000","last_peered":"2024-03-28T18:55:46.604527+0000","last_clean":"2024-03-28T18:55:46.604527+0000","last_became_active":"2024-03-28T18:55:44.606098+0000","last_became_peered":"2024-03-28T18:55:44.606098+0000","last_unstale":"2024-03-28T18:55:46.604527+0000","last_undegraded":"2024-03-28T18:55:46.604527+0000","last_fullsized":"2024-03-28T18:55:46.604527+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605931+0000","last_change":"2024-03-28T18:55:45.617483+0000","last_active":"2024-03-28T18:55:46.605931+0000","last_peered":"2024-03-28T18:55:46.605931+0000","last_clean":"2024-03-28T18:55:46.605931+0000","last_became_active":"2024-03-28T18:55:45.617169+0000","last_became_peered":"2024-03-28T18:55:45.617169+0000","last_unstale":"2024-03-28T18:55:46.605931+0000","last_undegraded":"2024-03-28T18:55:46.605931+0000","last_fullsized":"2024-03-28T18:55:46.605931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829227+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:55:46.829227+0000","last_peered":"2024-03-28T18:55:46.829227+0000","last_clean":"2024-03-28T18:55:46.829227+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:55:46.829227+0000","last_undegraded":"2024-03-28T18:55:46.829227+0000","last_fullsized":"2024-03-28T18:55:46.829227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613667+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:55:46.613667+0000","last_peered":"2024-03-28T18:55:46.613667+0000","last_clean":"2024-03-28T18:55:46.613667+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:55:46.613667+0000","last_undegraded":"2024-03-28T18:55:46.613667+0000","last_fullsized":"2024-03-28T18:55:46.613667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825795+0000","last_change":"2024-03-28T18:55:44.601836+0000","last_active":"2024-03-28T18:55:46.825795+0000","last_peered":"2024-03-28T18:55:46.825795+0000","last_clean":"2024-03-28T18:55:46.825795+0000","last_became_active":"2024-03-28T18:55:44.601552+0000","last_became_peered":"2024-03-28T18:55:44.601552+0000","last_unstale":"2024-03-28T18:55:46.825795+0000","last_undegraded":"2024-03-28T18:55:46.825795+0000","last_fullsized":"2024-03-28T18:55:46.825795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603619+0000","last_change":"2024-03-28T18:55:45.601801+0000","last_active":"2024-03-28T18:55:46.603619+0000","last_peered":"2024-03-28T18:55:46.603619+0000","last_clean":"2024-03-28T18:55:46.603619+0000","last_became_active":"2024-03-28T18:55:45.601459+0000","last_became_peered":"2024-03-28T18:55:45.601459+0000","last_unstale":"2024-03-28T18:55:46.603619+0000","last_undegraded":"2024-03-28T18:55:46.603619+0000","last_fullsized":"2024-03-28T18:55:46.603619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:43.033170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613982+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:55:46.613982+0000","last_peered":"2024-03-28T18:55:46.613982+0000","last_clean":"2024-03-28T18:55:46.613982+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:55:46.613982+0000","last_undegraded":"2024-03-28T18:55:46.613982+0000","last_fullsized":"2024-03-28T18:55:46.613982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165028+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:55:47.165028+0000","last_peered":"2024-03-28T18:55:47.165028+0000","last_clean":"2024-03-28T18:55:47.165028+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:55:47.165028+0000","last_undegraded":"2024-03-28T18:55:47.165028+0000","last_fullsized":"2024-03-28T18:55:47.165028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608494+0000","last_change":"2024-03-28T18:55:44.611148+0000","last_active":"2024-03-28T18:55:46.608494+0000","last_peered":"2024-03-28T18:55:46.608494+0000","last_clean":"2024-03-28T18:55:46.608494+0000","last_became_active":"2024-03-28T18:55:44.610835+0000","last_became_peered":"2024-03-28T18:55:44.610835+0000","last_unstale":"2024-03-28T18:55:46.608494+0000","last_undegraded":"2024-03-28T18:55:46.608494+0000","last_fullsized":"2024-03-28T18:55:46.608494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608851+0000","last_change":"2024-03-28T18:55:45.623672+0000","last_active":"2024-03-28T18:55:46.608851+0000","last_peered":"2024-03-28T18:55:46.608851+0000","last_clean":"2024-03-28T18:55:46.608851+0000","last_became_active":"2024-03-28T18:55:45.622568+0000","last_became_peered":"2024-03-28T18:55:45.622568+0000","last_unstale":"2024-03-28T18:55:46.608851+0000","last_undegraded":"2024-03-28T18:55:46.608851+0000","last_fullsized":"2024-03-28T18:55:46.608851+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117731+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:55:47.117731+0000","last_peered":"2024-03-28T18:55:47.117731+0000","last_clean":"2024-03-28T18:55:47.117731+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:55:47.117731+0000","last_undegraded":"2024-03-28T18:55:47.117731+0000","last_fullsized":"2024-03-28T18:55:47.117731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614041+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:55:46.614041+0000","last_peered":"2024-03-28T18:55:46.614041+0000","last_clean":"2024-03-28T18:55:46.614041+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:55:46.614041+0000","last_undegraded":"2024-03-28T18:55:46.614041+0000","last_fullsized":"2024-03-28T18:55:46.614041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613204+0000","last_change":"2024-03-28T18:55:44.604077+0000","last_active":"2024-03-28T18:55:46.613204+0000","last_peered":"2024-03-28T18:55:46.613204+0000","last_clean":"2024-03-28T18:55:46.613204+0000","last_became_active":"2024-03-28T18:55:44.603773+0000","last_became_peered":"2024-03-28T18:55:44.603773+0000","last_unstale":"2024-03-28T18:55:46.613204+0000","last_undegraded":"2024-03-28T18:55:46.613204+0000","last_fullsized":"2024-03-28T18:55:46.613204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608665+0000","last_change":"2024-03-28T18:55:45.615444+0000","last_active":"2024-03-28T18:55:46.608665+0000","last_peered":"2024-03-28T18:55:46.608665+0000","last_clean":"2024-03-28T18:55:46.608665+0000","last_became_active":"2024-03-28T18:55:45.615002+0000","last_became_peered":"2024-03-28T18:55:45.615002+0000","last_unstale":"2024-03-28T18:55:46.608665+0000","last_undegraded":"2024-03-28T18:55:46.608665+0000","last_fullsized":"2024-03-28T18:55:46.608665+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605186+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:55:46.605186+0000","last_peered":"2024-03-28T18:55:46.605186+0000","last_clean":"2024-03-28T18:55:46.605186+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:55:46.605186+0000","last_undegraded":"2024-03-28T18:55:46.605186+0000","last_fullsized":"2024-03-28T18:55:46.605186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116841+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:55:47.116841+0000","last_peered":"2024-03-28T18:55:47.116841+0000","last_clean":"2024-03-28T18:55:47.116841+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:55:47.116841+0000","last_undegraded":"2024-03-28T18:55:47.116841+0000","last_fullsized":"2024-03-28T18:55:47.116841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613947+0000","last_change":"2024-03-28T18:55:44.590703+0000","last_active":"2024-03-28T18:55:46.613947+0000","last_peered":"2024-03-28T18:55:46.613947+0000","last_clean":"2024-03-28T18:55:46.613947+0000","last_became_active":"2024-03-28T18:55:44.590408+0000","last_became_peered":"2024-03-28T18:55:44.590408+0000","last_unstale":"2024-03-28T18:55:46.613947+0000","last_undegraded":"2024-03-28T18:55:46.613947+0000","last_fullsized":"2024-03-28T18:55:46.613947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603326+0000","last_change":"2024-03-28T18:55:45.609795+0000","last_active":"2024-03-28T18:55:46.603326+0000","last_peered":"2024-03-28T18:55:46.603326+0000","last_clean":"2024-03-28T18:55:46.603326+0000","last_became_active":"2024-03-28T18:55:45.603695+0000","last_became_peered":"2024-03-28T18:55:45.603695+0000","last_unstale":"2024-03-28T18:55:46.603326+0000","last_undegraded":"2024-03-28T18:55:46.603326+0000","last_fullsized":"2024-03-28T18:55:46.603326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:49.173555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612873+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:55:46.612873+0000","last_peered":"2024-03-28T18:55:46.612873+0000","last_clean":"2024-03-28T18:55:46.612873+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:55:46.612873+0000","last_undegraded":"2024-03-28T18:55:46.612873+0000","last_fullsized":"2024-03-28T18:55:46.612873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602878+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:55:46.602878+0000","last_peered":"2024-03-28T18:55:46.602878+0000","last_clean":"2024-03-28T18:55:46.602878+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:55:46.602878+0000","last_undegraded":"2024-03-28T18:55:46.602878+0000","last_fullsized":"2024-03-28T18:55:46.602878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614195+0000","last_change":"2024-03-28T18:55:44.612172+0000","last_active":"2024-03-28T18:55:46.614195+0000","last_peered":"2024-03-28T18:55:46.614195+0000","last_clean":"2024-03-28T18:55:46.614195+0000","last_became_active":"2024-03-28T18:55:44.611859+0000","last_became_peered":"2024-03-28T18:55:44.611859+0000","last_unstale":"2024-03-28T18:55:46.614195+0000","last_undegraded":"2024-03-28T18:55:46.614195+0000","last_fullsized":"2024-03-28T18:55:46.614195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117969+0000","last_change":"2024-03-28T18:55:45.619723+0000","last_active":"2024-03-28T18:55:47.117969+0000","last_peered":"2024-03-28T18:55:47.117969+0000","last_clean":"2024-03-28T18:55:47.117969+0000","last_became_active":"2024-03-28T18:55:45.619437+0000","last_became_peered":"2024-03-28T18:55:45.619437+0000","last_unstale":"2024-03-28T18:55:47.117969+0000","last_undegraded":"2024-03-28T18:55:47.117969+0000","last_fullsized":"2024-03-28T18:55:47.117969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608071+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:55:46.608071+0000","last_peered":"2024-03-28T18:55:46.608071+0000","last_clean":"2024-03-28T18:55:46.608071+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:55:46.608071+0000","last_undegraded":"2024-03-28T18:55:46.608071+0000","last_fullsized":"2024-03-28T18:55:46.608071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164105+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:55:47.164105+0000","last_peered":"2024-03-28T18:55:47.164105+0000","last_clean":"2024-03-28T18:55:47.164105+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:55:47.164105+0000","last_undegraded":"2024-03-28T18:55:47.164105+0000","last_fullsized":"2024-03-28T18:55:47.164105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.611391+0000","last_change":"2024-03-28T18:55:44.592064+0000","last_active":"2024-03-28T18:55:46.611391+0000","last_peered":"2024-03-28T18:55:46.611391+0000","last_clean":"2024-03-28T18:55:46.611391+0000","last_became_active":"2024-03-28T18:55:44.591263+0000","last_became_peered":"2024-03-28T18:55:44.591263+0000","last_unstale":"2024-03-28T18:55:46.611391+0000","last_undegraded":"2024-03-28T18:55:46.611391+0000","last_fullsized":"2024-03-28T18:55:46.611391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163958+0000","last_change":"2024-03-28T18:55:45.614820+0000","last_active":"2024-03-28T18:55:47.163958+0000","last_peered":"2024-03-28T18:55:47.163958+0000","last_clean":"2024-03-28T18:55:47.163958+0000","last_became_active":"2024-03-28T18:55:45.614517+0000","last_became_peered":"2024-03-28T18:55:45.614517+0000","last_unstale":"2024-03-28T18:55:47.163958+0000","last_undegraded":"2024-03-28T18:55:47.163958+0000","last_fullsized":"2024-03-28T18:55:47.163958+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826245+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:55:46.826245+0000","last_peered":"2024-03-28T18:55:46.826245+0000","last_clean":"2024-03-28T18:55:46.826245+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:55:46.826245+0000","last_undegraded":"2024-03-28T18:55:46.826245+0000","last_fullsized":"2024-03-28T18:55:46.826245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116294+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:55:47.116294+0000","last_peered":"2024-03-28T18:55:47.116294+0000","last_clean":"2024-03-28T18:55:47.116294+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:55:47.116294+0000","last_undegraded":"2024-03-28T18:55:47.116294+0000","last_fullsized":"2024-03-28T18:55:47.116294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164472+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:55:47.164472+0000","last_peered":"2024-03-28T18:55:47.164472+0000","last_clean":"2024-03-28T18:55:47.164472+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:55:47.164472+0000","last_undegraded":"2024-03-28T18:55:47.164472+0000","last_fullsized":"2024-03-28T18:55:47.164472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602907+0000","last_change":"2024-03-28T18:55:44.602097+0000","last_active":"2024-03-28T18:55:46.602907+0000","last_peered":"2024-03-28T18:55:46.602907+0000","last_clean":"2024-03-28T18:55:46.602907+0000","last_became_active":"2024-03-28T18:55:44.601769+0000","last_became_peered":"2024-03-28T18:55:44.601769+0000","last_unstale":"2024-03-28T18:55:46.602907+0000","last_undegraded":"2024-03-28T18:55:46.602907+0000","last_fullsized":"2024-03-28T18:55:46.602907+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603447+0000","last_change":"2024-03-28T18:55:45.616485+0000","last_active":"2024-03-28T18:55:46.603447+0000","last_peered":"2024-03-28T18:55:46.603447+0000","last_clean":"2024-03-28T18:55:46.603447+0000","last_became_active":"2024-03-28T18:55:45.615813+0000","last_became_peered":"2024-03-28T18:55:45.615813+0000","last_unstale":"2024-03-28T18:55:46.603447+0000","last_undegraded":"2024-03-28T18:55:46.603447+0000","last_fullsized":"2024-03-28T18:55:46.603447+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:21.227281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164685+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:55:47.164685+0000","last_peered":"2024-03-28T18:55:47.164685+0000","last_clean":"2024-03-28T18:55:47.164685+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:55:47.164685+0000","last_undegraded":"2024-03-28T18:55:47.164685+0000","last_fullsized":"2024-03-28T18:55:47.164685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603451+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:55:46.603451+0000","last_peered":"2024-03-28T18:55:46.603451+0000","last_clean":"2024-03-28T18:55:46.603451+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:55:46.603451+0000","last_undegraded":"2024-03-28T18:55:46.603451+0000","last_fullsized":"2024-03-28T18:55:46.603451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603782+0000","last_change":"2024-03-28T18:55:44.607744+0000","last_active":"2024-03-28T18:55:46.603782+0000","last_peered":"2024-03-28T18:55:46.603782+0000","last_clean":"2024-03-28T18:55:46.603782+0000","last_became_active":"2024-03-28T18:55:44.606993+0000","last_became_peered":"2024-03-28T18:55:44.606993+0000","last_unstale":"2024-03-28T18:55:46.603782+0000","last_undegraded":"2024-03-28T18:55:46.603782+0000","last_fullsized":"2024-03-28T18:55:46.603782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602563+0000","last_change":"2024-03-28T18:55:45.606307+0000","last_active":"2024-03-28T18:55:46.602563+0000","last_peered":"2024-03-28T18:55:46.602563+0000","last_clean":"2024-03-28T18:55:46.602563+0000","last_became_active":"2024-03-28T18:55:45.605980+0000","last_became_peered":"2024-03-28T18:55:45.605980+0000","last_unstale":"2024-03-28T18:55:46.602563+0000","last_undegraded":"2024-03-28T18:55:46.602563+0000","last_fullsized":"2024-03-28T18:55:46.602563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:52.485966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117600+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:55:47.117600+0000","last_peered":"2024-03-28T18:55:47.117600+0000","last_clean":"2024-03-28T18:55:47.117600+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:55:47.117600+0000","last_undegraded":"2024-03-28T18:55:47.117600+0000","last_fullsized":"2024-03-28T18:55:47.117600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614275+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:55:46.614275+0000","last_peered":"2024-03-28T18:55:46.614275+0000","last_clean":"2024-03-28T18:55:46.614275+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:55:46.614275+0000","last_undegraded":"2024-03-28T18:55:46.614275+0000","last_fullsized":"2024-03-28T18:55:46.614275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608346+0000","last_change":"2024-03-28T18:55:44.607323+0000","last_active":"2024-03-28T18:55:46.608346+0000","last_peered":"2024-03-28T18:55:46.608346+0000","last_clean":"2024-03-28T18:55:46.608346+0000","last_became_active":"2024-03-28T18:55:44.606579+0000","last_became_peered":"2024-03-28T18:55:44.606579+0000","last_unstale":"2024-03-28T18:55:46.608346+0000","last_undegraded":"2024-03-28T18:55:46.608346+0000","last_fullsized":"2024-03-28T18:55:46.608346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163490+0000","last_change":"2024-03-28T18:55:45.622760+0000","last_active":"2024-03-28T18:55:47.163490+0000","last_peered":"2024-03-28T18:55:47.163490+0000","last_clean":"2024-03-28T18:55:47.163490+0000","last_became_active":"2024-03-28T18:55:45.622353+0000","last_became_peered":"2024-03-28T18:55:45.622353+0000","last_unstale":"2024-03-28T18:55:47.163490+0000","last_undegraded":"2024-03-28T18:55:47.163490+0000","last_fullsized":"2024-03-28T18:55:47.163490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:04.349689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166444+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:55:47.166444+0000","last_peered":"2024-03-28T18:55:47.166444+0000","last_clean":"2024-03-28T18:55:47.166444+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:55:47.166444+0000","last_undegraded":"2024-03-28T18:55:47.166444+0000","last_fullsized":"2024-03-28T18:55:47.166444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603346+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:55:46.603346+0000","last_peered":"2024-03-28T18:55:46.603346+0000","last_clean":"2024-03-28T18:55:46.603346+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:55:46.603346+0000","last_undegraded":"2024-03-28T18:55:46.603346+0000","last_fullsized":"2024-03-28T18:55:46.603346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116165+0000","last_change":"2024-03-28T18:55:44.607237+0000","last_active":"2024-03-28T18:55:47.116165+0000","last_peered":"2024-03-28T18:55:47.116165+0000","last_clean":"2024-03-28T18:55:47.116165+0000","last_became_active":"2024-03-28T18:55:44.606960+0000","last_became_peered":"2024-03-28T18:55:44.606960+0000","last_unstale":"2024-03-28T18:55:47.116165+0000","last_undegraded":"2024-03-28T18:55:47.116165+0000","last_fullsized":"2024-03-28T18:55:47.116165+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603242+0000","last_change":"2024-03-28T18:55:45.612992+0000","last_active":"2024-03-28T18:55:46.603242+0000","last_peered":"2024-03-28T18:55:46.603242+0000","last_clean":"2024-03-28T18:55:46.603242+0000","last_became_active":"2024-03-28T18:55:45.612125+0000","last_became_peered":"2024-03-28T18:55:45.612125+0000","last_unstale":"2024-03-28T18:55:46.603242+0000","last_undegraded":"2024-03-28T18:55:46.603242+0000","last_fullsized":"2024-03-28T18:55:46.603242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:35.898911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165571+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:55:47.165571+0000","last_peered":"2024-03-28T18:55:47.165571+0000","last_clean":"2024-03-28T18:55:47.165571+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:55:47.165571+0000","last_undegraded":"2024-03-28T18:55:47.165571+0000","last_fullsized":"2024-03-28T18:55:47.165571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605603+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:55:46.605603+0000","last_peered":"2024-03-28T18:55:46.605603+0000","last_clean":"2024-03-28T18:55:46.605603+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:55:46.605603+0000","last_undegraded":"2024-03-28T18:55:46.605603+0000","last_fullsized":"2024-03-28T18:55:46.605603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604200+0000","last_change":"2024-03-28T18:55:44.607218+0000","last_active":"2024-03-28T18:55:46.604200+0000","last_peered":"2024-03-28T18:55:46.604200+0000","last_clean":"2024-03-28T18:55:46.604200+0000","last_became_active":"2024-03-28T18:55:44.606852+0000","last_became_peered":"2024-03-28T18:55:44.606852+0000","last_unstale":"2024-03-28T18:55:46.604200+0000","last_undegraded":"2024-03-28T18:55:46.604200+0000","last_fullsized":"2024-03-28T18:55:46.604200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:36.624289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829038+0000","last_change":"2024-03-28T18:55:45.618919+0000","last_active":"2024-03-28T18:55:46.829038+0000","last_peered":"2024-03-28T18:55:46.829038+0000","last_clean":"2024-03-28T18:55:46.829038+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:55:46.829038+0000","last_undegraded":"2024-03-28T18:55:46.829038+0000","last_fullsized":"2024-03-28T18:55:46.829038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:29.407508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164081+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:55:47.164081+0000","last_peered":"2024-03-28T18:55:47.164081+0000","last_clean":"2024-03-28T18:55:47.164081+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:55:47.164081+0000","last_undegraded":"2024-03-28T18:55:47.164081+0000","last_fullsized":"2024-03-28T18:55:47.164081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602411+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:55:46.602411+0000","last_peered":"2024-03-28T18:55:46.602411+0000","last_clean":"2024-03-28T18:55:46.602411+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:55:46.602411+0000","last_undegraded":"2024-03-28T18:55:46.602411+0000","last_fullsized":"2024-03-28T18:55:46.602411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609206+0000","last_change":"2024-03-28T18:55:44.598881+0000","last_active":"2024-03-28T18:55:46.609206+0000","last_peered":"2024-03-28T18:55:46.609206+0000","last_clean":"2024-03-28T18:55:46.609206+0000","last_became_active":"2024-03-28T18:55:44.598532+0000","last_became_peered":"2024-03-28T18:55:44.598532+0000","last_unstale":"2024-03-28T18:55:46.609206+0000","last_undegraded":"2024-03-28T18:55:46.609206+0000","last_fullsized":"2024-03-28T18:55:46.609206+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:44.004611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117993+0000","last_change":"2024-03-28T18:55:45.612572+0000","last_active":"2024-03-28T18:55:47.117993+0000","last_peered":"2024-03-28T18:55:47.117993+0000","last_clean":"2024-03-28T18:55:47.117993+0000","last_became_active":"2024-03-28T18:55:45.611844+0000","last_became_peered":"2024-03-28T18:55:45.611844+0000","last_unstale":"2024-03-28T18:55:47.117993+0000","last_undegraded":"2024-03-28T18:55:47.117993+0000","last_fullsized":"2024-03-28T18:55:47.117993+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602718+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:55:46.602718+0000","last_peered":"2024-03-28T18:55:46.602718+0000","last_clean":"2024-03-28T18:55:46.602718+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:55:46.602718+0000","last_undegraded":"2024-03-28T18:55:46.602718+0000","last_fullsized":"2024-03-28T18:55:46.602718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603429+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:55:46.603429+0000","last_peered":"2024-03-28T18:55:46.603429+0000","last_clean":"2024-03-28T18:55:46.603429+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:55:46.603429+0000","last_undegraded":"2024-03-28T18:55:46.603429+0000","last_fullsized":"2024-03-28T18:55:46.603429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829075+0000","last_change":"2024-03-28T18:55:44.609644+0000","last_active":"2024-03-28T18:55:46.829075+0000","last_peered":"2024-03-28T18:55:46.829075+0000","last_clean":"2024-03-28T18:55:46.829075+0000","last_became_active":"2024-03-28T18:55:44.609317+0000","last_became_peered":"2024-03-28T18:55:44.609317+0000","last_unstale":"2024-03-28T18:55:46.829075+0000","last_undegraded":"2024-03-28T18:55:46.829075+0000","last_fullsized":"2024-03-28T18:55:46.829075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:37.662504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116808+0000","last_change":"2024-03-28T18:55:45.621708+0000","last_active":"2024-03-28T18:55:47.116808+0000","last_peered":"2024-03-28T18:55:47.116808+0000","last_clean":"2024-03-28T18:55:47.116808+0000","last_became_active":"2024-03-28T18:55:45.621366+0000","last_became_peered":"2024-03-28T18:55:45.621366+0000","last_unstale":"2024-03-28T18:55:47.116808+0000","last_undegraded":"2024-03-28T18:55:47.116808+0000","last_fullsized":"2024-03-28T18:55:47.116808+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164622+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:55:47.164622+0000","last_peered":"2024-03-28T18:55:47.164622+0000","last_clean":"2024-03-28T18:55:47.164622+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:55:47.164622+0000","last_undegraded":"2024-03-28T18:55:47.164622+0000","last_fullsized":"2024-03-28T18:55:47.164622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116814+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:55:47.116814+0000","last_peered":"2024-03-28T18:55:47.116814+0000","last_clean":"2024-03-28T18:55:47.116814+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:55:47.116814+0000","last_undegraded":"2024-03-28T18:55:47.116814+0000","last_fullsized":"2024-03-28T18:55:47.116814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"23'6","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827637+0000","last_change":"2024-03-28T18:55:44.608949+0000","last_active":"2024-03-28T18:55:46.827637+0000","last_peered":"2024-03-28T18:55:46.827637+0000","last_clean":"2024-03-28T18:55:46.827637+0000","last_became_active":"2024-03-28T18:55:44.608657+0000","last_became_peered":"2024-03-28T18:55:44.608657+0000","last_unstale":"2024-03-28T18:55:46.827637+0000","last_undegraded":"2024-03-28T18:55:46.827637+0000","last_fullsized":"2024-03-28T18:55:46.827637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:04.799314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827662+0000","last_change":"2024-03-28T18:55:45.610686+0000","last_active":"2024-03-28T18:55:46.827662+0000","last_peered":"2024-03-28T18:55:46.827662+0000","last_clean":"2024-03-28T18:55:46.827662+0000","last_became_active":"2024-03-28T18:55:45.610212+0000","last_became_peered":"2024-03-28T18:55:45.610212+0000","last_unstale":"2024-03-28T18:55:46.827662+0000","last_undegraded":"2024-03-28T18:55:46.827662+0000","last_fullsized":"2024-03-28T18:55:46.827662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608274+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:55:46.608274+0000","last_peered":"2024-03-28T18:55:46.608274+0000","last_clean":"2024-03-28T18:55:46.608274+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:55:46.608274+0000","last_undegraded":"2024-03-28T18:55:46.608274+0000","last_fullsized":"2024-03-28T18:55:46.608274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609087+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:55:46.609087+0000","last_peered":"2024-03-28T18:55:46.609087+0000","last_clean":"2024-03-28T18:55:46.609087+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:55:46.609087+0000","last_undegraded":"2024-03-28T18:55:46.609087+0000","last_fullsized":"2024-03-28T18:55:46.609087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"23'3","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117447+0000","last_change":"2024-03-28T18:55:44.602762+0000","last_active":"2024-03-28T18:55:47.117447+0000","last_peered":"2024-03-28T18:55:47.117447+0000","last_clean":"2024-03-28T18:55:47.117447+0000","last_became_active":"2024-03-28T18:55:44.602424+0000","last_became_peered":"2024-03-28T18:55:44.602424+0000","last_unstale":"2024-03-28T18:55:47.117447+0000","last_undegraded":"2024-03-28T18:55:47.117447+0000","last_fullsized":"2024-03-28T18:55:47.117447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826897+0000","last_change":"2024-03-28T18:55:45.601277+0000","last_active":"2024-03-28T18:55:46.826897+0000","last_peered":"2024-03-28T18:55:46.826897+0000","last_clean":"2024-03-28T18:55:46.826897+0000","last_became_active":"2024-03-28T18:55:45.600945+0000","last_became_peered":"2024-03-28T18:55:45.600945+0000","last_unstale":"2024-03-28T18:55:46.826897+0000","last_undegraded":"2024-03-28T18:55:46.826897+0000","last_fullsized":"2024-03-28T18:55:46.826897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164660+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:55:47.164660+0000","last_peered":"2024-03-28T18:55:47.164660+0000","last_clean":"2024-03-28T18:55:47.164660+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:55:47.164660+0000","last_undegraded":"2024-03-28T18:55:47.164660+0000","last_fullsized":"2024-03-28T18:55:47.164660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602857+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:55:46.602857+0000","last_peered":"2024-03-28T18:55:46.602857+0000","last_clean":"2024-03-28T18:55:46.602857+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:55:46.602857+0000","last_undegraded":"2024-03-28T18:55:46.602857+0000","last_fullsized":"2024-03-28T18:55:46.602857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612960+0000","last_change":"2024-03-28T18:55:44.599432+0000","last_active":"2024-03-28T18:55:46.612960+0000","last_peered":"2024-03-28T18:55:46.612960+0000","last_clean":"2024-03-28T18:55:46.612960+0000","last_became_active":"2024-03-28T18:55:44.599123+0000","last_became_peered":"2024-03-28T18:55:44.599123+0000","last_unstale":"2024-03-28T18:55:46.612960+0000","last_undegraded":"2024-03-28T18:55:46.612960+0000","last_fullsized":"2024-03-28T18:55:46.612960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613285+0000","last_change":"2024-03-28T18:55:45.624311+0000","last_active":"2024-03-28T18:55:46.613285+0000","last_peered":"2024-03-28T18:55:46.613285+0000","last_clean":"2024-03-28T18:55:46.613285+0000","last_became_active":"2024-03-28T18:55:45.624031+0000","last_became_peered":"2024-03-28T18:55:45.624031+0000","last_unstale":"2024-03-28T18:55:46.613285+0000","last_undegraded":"2024-03-28T18:55:46.613285+0000","last_fullsized":"2024-03-28T18:55:46.613285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605962+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:55:46.605962+0000","last_peered":"2024-03-28T18:55:46.605962+0000","last_clean":"2024-03-28T18:55:46.605962+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:55:46.605962+0000","last_undegraded":"2024-03-28T18:55:46.605962+0000","last_fullsized":"2024-03-28T18:55:46.605962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614210+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:55:46.614210+0000","last_peered":"2024-03-28T18:55:46.614210+0000","last_clean":"2024-03-28T18:55:46.614210+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:55:46.614210+0000","last_undegraded":"2024-03-28T18:55:46.614210+0000","last_fullsized":"2024-03-28T18:55:46.614210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604974+0000","last_change":"2024-03-28T18:55:44.593567+0000","last_active":"2024-03-28T18:55:46.604974+0000","last_peered":"2024-03-28T18:55:46.604974+0000","last_clean":"2024-03-28T18:55:46.604974+0000","last_became_active":"2024-03-28T18:55:44.593267+0000","last_became_peered":"2024-03-28T18:55:44.593267+0000","last_unstale":"2024-03-28T18:55:46.604974+0000","last_undegraded":"2024-03-28T18:55:46.604974+0000","last_fullsized":"2024-03-28T18:55:46.604974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603331+0000","last_change":"2024-03-28T18:55:45.598713+0000","last_active":"2024-03-28T18:55:46.603331+0000","last_peered":"2024-03-28T18:55:46.603331+0000","last_clean":"2024-03-28T18:55:46.603331+0000","last_became_active":"2024-03-28T18:55:45.597726+0000","last_became_peered":"2024-03-28T18:55:45.597726+0000","last_unstale":"2024-03-28T18:55:46.603331+0000","last_undegraded":"2024-03-28T18:55:46.603331+0000","last_fullsized":"2024-03-28T18:55:46.603331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:48.255745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609101+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:55:46.609101+0000","last_peered":"2024-03-28T18:55:46.609101+0000","last_clean":"2024-03-28T18:55:46.609101+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:55:46.609101+0000","last_undegraded":"2024-03-28T18:55:46.609101+0000","last_fullsized":"2024-03-28T18:55:46.609101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118062+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:55:47.118062+0000","last_peered":"2024-03-28T18:55:47.118062+0000","last_clean":"2024-03-28T18:55:47.118062+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:55:47.118062+0000","last_undegraded":"2024-03-28T18:55:47.118062+0000","last_fullsized":"2024-03-28T18:55:47.118062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"23'8","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826291+0000","last_change":"2024-03-28T18:55:44.610673+0000","last_active":"2024-03-28T18:55:46.826291+0000","last_peered":"2024-03-28T18:55:46.826291+0000","last_clean":"2024-03-28T18:55:46.826291+0000","last_became_active":"2024-03-28T18:55:44.610357+0000","last_became_peered":"2024-03-28T18:55:44.610357+0000","last_unstale":"2024-03-28T18:55:46.826291+0000","last_undegraded":"2024-03-28T18:55:46.826291+0000","last_fullsized":"2024-03-28T18:55:46.826291+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:26.280548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163667+0000","last_change":"2024-03-28T18:55:45.609349+0000","last_active":"2024-03-28T18:55:47.163667+0000","last_peered":"2024-03-28T18:55:47.163667+0000","last_clean":"2024-03-28T18:55:47.163667+0000","last_became_active":"2024-03-28T18:55:45.608743+0000","last_became_peered":"2024-03-28T18:55:45.608743+0000","last_unstale":"2024-03-28T18:55:47.163667+0000","last_undegraded":"2024-03-28T18:55:47.163667+0000","last_fullsized":"2024-03-28T18:55:47.163667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118036+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:55:47.118036+0000","last_peered":"2024-03-28T18:55:47.118036+0000","last_clean":"2024-03-28T18:55:47.118036+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:55:47.118036+0000","last_undegraded":"2024-03-28T18:55:47.118036+0000","last_fullsized":"2024-03-28T18:55:47.118036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604933+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:55:46.604933+0000","last_peered":"2024-03-28T18:55:46.604933+0000","last_clean":"2024-03-28T18:55:46.604933+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:55:46.604933+0000","last_undegraded":"2024-03-28T18:55:46.604933+0000","last_fullsized":"2024-03-28T18:55:46.604933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825952+0000","last_change":"2024-03-28T18:55:44.601103+0000","last_active":"2024-03-28T18:55:46.825952+0000","last_peered":"2024-03-28T18:55:46.825952+0000","last_clean":"2024-03-28T18:55:46.825952+0000","last_became_active":"2024-03-28T18:55:44.599969+0000","last_became_peered":"2024-03-28T18:55:44.599969+0000","last_unstale":"2024-03-28T18:55:46.825952+0000","last_undegraded":"2024-03-28T18:55:46.825952+0000","last_fullsized":"2024-03-28T18:55:46.825952+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:42.018654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604371+0000","last_change":"2024-03-28T18:55:45.594944+0000","last_active":"2024-03-28T18:55:46.604371+0000","last_peered":"2024-03-28T18:55:46.604371+0000","last_clean":"2024-03-28T18:55:46.604371+0000","last_became_active":"2024-03-28T18:55:45.594603+0000","last_became_peered":"2024-03-28T18:55:45.594603+0000","last_unstale":"2024-03-28T18:55:46.604371+0000","last_undegraded":"2024-03-28T18:55:46.604371+0000","last_fullsized":"2024-03-28T18:55:46.604371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608967+0000","last_change":"2024-03-28T18:55:44.602366+0000","last_active":"2024-03-28T18:55:46.608967+0000","last_peered":"2024-03-28T18:55:46.608967+0000","last_clean":"2024-03-28T18:55:46.608967+0000","last_became_active":"2024-03-28T18:55:44.602033+0000","last_became_peered":"2024-03-28T18:55:44.602033+0000","last_unstale":"2024-03-28T18:55:46.608967+0000","last_undegraded":"2024-03-28T18:55:46.608967+0000","last_fullsized":"2024-03-28T18:55:46.608967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612862+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:55:46.612862+0000","last_peered":"2024-03-28T18:55:46.612862+0000","last_clean":"2024-03-28T18:55:46.612862+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:55:46.612862+0000","last_undegraded":"2024-03-28T18:55:46.612862+0000","last_fullsized":"2024-03-28T18:55:46.612862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117107+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:55:47.117107+0000","last_peered":"2024-03-28T18:55:47.117107+0000","last_clean":"2024-03-28T18:55:47.117107+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:55:47.117107+0000","last_undegraded":"2024-03-28T18:55:47.117107+0000","last_fullsized":"2024-03-28T18:55:47.117107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608811+0000","last_change":"2024-03-28T18:55:45.620559+0000","last_active":"2024-03-28T18:55:46.608811+0000","last_peered":"2024-03-28T18:55:46.608811+0000","last_clean":"2024-03-28T18:55:46.608811+0000","last_became_active":"2024-03-28T18:55:45.620249+0000","last_became_peered":"2024-03-28T18:55:45.620249+0000","last_unstale":"2024-03-28T18:55:46.608811+0000","last_undegraded":"2024-03-28T18:55:46.608811+0000","last_fullsized":"2024-03-28T18:55:46.608811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"23'4","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604650+0000","last_change":"2024-03-28T18:55:44.612226+0000","last_active":"2024-03-28T18:55:46.604650+0000","last_peered":"2024-03-28T18:55:46.604650+0000","last_clean":"2024-03-28T18:55:46.604650+0000","last_became_active":"2024-03-28T18:55:44.611921+0000","last_became_peered":"2024-03-28T18:55:44.611921+0000","last_unstale":"2024-03-28T18:55:46.604650+0000","last_undegraded":"2024-03-28T18:55:46.604650+0000","last_fullsized":"2024-03-28T18:55:46.604650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826474+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:55:46.826474+0000","last_peered":"2024-03-28T18:55:46.826474+0000","last_clean":"2024-03-28T18:55:46.826474+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:55:46.826474+0000","last_undegraded":"2024-03-28T18:55:46.826474+0000","last_fullsized":"2024-03-28T18:55:46.826474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826438+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:55:46.826438+0000","last_peered":"2024-03-28T18:55:46.826438+0000","last_clean":"2024-03-28T18:55:46.826438+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:55:46.826438+0000","last_undegraded":"2024-03-28T18:55:46.826438+0000","last_fullsized":"2024-03-28T18:55:46.826438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614060+0000","last_change":"2024-03-28T18:55:45.612727+0000","last_active":"2024-03-28T18:55:46.614060+0000","last_peered":"2024-03-28T18:55:46.614060+0000","last_clean":"2024-03-28T18:55:46.614060+0000","last_became_active":"2024-03-28T18:55:45.610154+0000","last_became_peered":"2024-03-28T18:55:45.610154+0000","last_unstale":"2024-03-28T18:55:46.614060+0000","last_undegraded":"2024-03-28T18:55:46.614060+0000","last_fullsized":"2024-03-28T18:55:46.614060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604120+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:55:46.604120+0000","last_peered":"2024-03-28T18:55:46.604120+0000","last_clean":"2024-03-28T18:55:46.604120+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:55:46.604120+0000","last_undegraded":"2024-03-28T18:55:46.604120+0000","last_fullsized":"2024-03-28T18:55:46.604120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608921+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:55:46.608921+0000","last_peered":"2024-03-28T18:55:46.608921+0000","last_clean":"2024-03-28T18:55:46.608921+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:55:46.608921+0000","last_undegraded":"2024-03-28T18:55:46.608921+0000","last_fullsized":"2024-03-28T18:55:46.608921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603331+0000","last_change":"2024-03-28T18:55:44.592835+0000","last_active":"2024-03-28T18:55:46.603331+0000","last_peered":"2024-03-28T18:55:46.603331+0000","last_clean":"2024-03-28T18:55:46.603331+0000","last_became_active":"2024-03-28T18:55:44.592505+0000","last_became_peered":"2024-03-28T18:55:44.592505+0000","last_unstale":"2024-03-28T18:55:46.603331+0000","last_undegraded":"2024-03-28T18:55:46.603331+0000","last_fullsized":"2024-03-28T18:55:46.603331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:58.286421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605269+0000","last_change":"2024-03-28T18:55:45.611322+0000","last_active":"2024-03-28T18:55:46.605269+0000","last_peered":"2024-03-28T18:55:46.605269+0000","last_clean":"2024-03-28T18:55:46.605269+0000","last_became_active":"2024-03-28T18:55:45.610958+0000","last_became_peered":"2024-03-28T18:55:45.610958+0000","last_unstale":"2024-03-28T18:55:46.605269+0000","last_undegraded":"2024-03-28T18:55:46.605269+0000","last_fullsized":"2024-03-28T18:55:46.605269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827108+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:55:46.827108+0000","last_peered":"2024-03-28T18:55:46.827108+0000","last_clean":"2024-03-28T18:55:46.827108+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:55:46.827108+0000","last_undegraded":"2024-03-28T18:55:46.827108+0000","last_fullsized":"2024-03-28T18:55:46.827108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604724+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:55:46.604724+0000","last_peered":"2024-03-28T18:55:46.604724+0000","last_clean":"2024-03-28T18:55:46.604724+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:55:46.604724+0000","last_undegraded":"2024-03-28T18:55:46.604724+0000","last_fullsized":"2024-03-28T18:55:46.604724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827611+0000","last_change":"2024-03-28T18:55:44.608283+0000","last_active":"2024-03-28T18:55:46.827611+0000","last_peered":"2024-03-28T18:55:46.827611+0000","last_clean":"2024-03-28T18:55:46.827611+0000","last_became_active":"2024-03-28T18:55:44.607999+0000","last_became_peered":"2024-03-28T18:55:44.607999+0000","last_unstale":"2024-03-28T18:55:46.827611+0000","last_undegraded":"2024-03-28T18:55:46.827611+0000","last_fullsized":"2024-03-28T18:55:46.827611+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:22.669778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828611+0000","last_change":"2024-03-28T18:55:45.618390+0000","last_active":"2024-03-28T18:55:46.828611+0000","last_peered":"2024-03-28T18:55:46.828611+0000","last_clean":"2024-03-28T18:55:46.828611+0000","last_became_active":"2024-03-28T18:55:45.617897+0000","last_became_peered":"2024-03-28T18:55:45.617897+0000","last_unstale":"2024-03-28T18:55:46.828611+0000","last_undegraded":"2024-03-28T18:55:46.828611+0000","last_fullsized":"2024-03-28T18:55:46.828611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:16.520575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602446+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:55:46.602446+0000","last_peered":"2024-03-28T18:55:46.602446+0000","last_clean":"2024-03-28T18:55:46.602446+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:55:46.602446+0000","last_undegraded":"2024-03-28T18:55:46.602446+0000","last_fullsized":"2024-03-28T18:55:46.602446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613831+0000","last_change":"2024-03-28T18:55:45.624938+0000","last_active":"2024-03-28T18:55:46.613831+0000","last_peered":"2024-03-28T18:55:46.613831+0000","last_clean":"2024-03-28T18:55:46.613831+0000","last_became_active":"2024-03-28T18:55:45.624362+0000","last_became_peered":"2024-03-28T18:55:45.624362+0000","last_unstale":"2024-03-28T18:55:46.613831+0000","last_undegraded":"2024-03-28T18:55:46.613831+0000","last_fullsized":"2024-03-28T18:55:46.613831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602638+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:55:46.602638+0000","last_peered":"2024-03-28T18:55:46.602638+0000","last_clean":"2024-03-28T18:55:46.602638+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:55:46.602638+0000","last_undegraded":"2024-03-28T18:55:46.602638+0000","last_fullsized":"2024-03-28T18:55:46.602638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116612+0000","last_change":"2024-03-28T18:55:45.610168+0000","last_active":"2024-03-28T18:55:47.116612+0000","last_peered":"2024-03-28T18:55:47.116612+0000","last_clean":"2024-03-28T18:55:47.116612+0000","last_became_active":"2024-03-28T18:55:45.609653+0000","last_became_peered":"2024-03-28T18:55:45.609653+0000","last_unstale":"2024-03-28T18:55:47.116612+0000","last_undegraded":"2024-03-28T18:55:47.116612+0000","last_fullsized":"2024-03-28T18:55:47.116612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116681+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:55:47.116681+0000","last_peered":"2024-03-28T18:55:47.116681+0000","last_clean":"2024-03-28T18:55:47.116681+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:55:47.116681+0000","last_undegraded":"2024-03-28T18:55:47.116681+0000","last_fullsized":"2024-03-28T18:55:47.116681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613952+0000","last_change":"2024-03-28T18:55:45.613701+0000","last_active":"2024-03-28T18:55:46.613952+0000","last_peered":"2024-03-28T18:55:46.613952+0000","last_clean":"2024-03-28T18:55:46.613952+0000","last_became_active":"2024-03-28T18:55:45.613310+0000","last_became_peered":"2024-03-28T18:55:45.613310+0000","last_unstale":"2024-03-28T18:55:46.613952+0000","last_undegraded":"2024-03-28T18:55:46.613952+0000","last_fullsized":"2024-03-28T18:55:46.613952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608809+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:55:46.608809+0000","last_peered":"2024-03-28T18:55:46.608809+0000","last_clean":"2024-03-28T18:55:46.608809+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:55:46.608809+0000","last_undegraded":"2024-03-28T18:55:46.608809+0000","last_fullsized":"2024-03-28T18:55:46.608809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163732+0000","last_change":"2024-03-28T18:55:45.615119+0000","last_active":"2024-03-28T18:55:47.163732+0000","last_peered":"2024-03-28T18:55:47.163732+0000","last_clean":"2024-03-28T18:55:47.163732+0000","last_became_active":"2024-03-28T18:55:45.613841+0000","last_became_peered":"2024-03-28T18:55:45.613841+0000","last_unstale":"2024-03-28T18:55:47.163732+0000","last_undegraded":"2024-03-28T18:55:47.163732+0000","last_fullsized":"2024-03-28T18:55:47.163732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827969+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:55:46.827969+0000","last_peered":"2024-03-28T18:55:46.827969+0000","last_clean":"2024-03-28T18:55:46.827969+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:55:46.827969+0000","last_undegraded":"2024-03-28T18:55:46.827969+0000","last_fullsized":"2024-03-28T18:55:46.827969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604609+0000","last_change":"2024-03-28T18:55:45.619605+0000","last_active":"2024-03-28T18:55:46.604609+0000","last_peered":"2024-03-28T18:55:46.604609+0000","last_clean":"2024-03-28T18:55:46.604609+0000","last_became_active":"2024-03-28T18:55:45.618473+0000","last_became_peered":"2024-03-28T18:55:45.618473+0000","last_unstale":"2024-03-28T18:55:46.604609+0000","last_undegraded":"2024-03-28T18:55:46.604609+0000","last_fullsized":"2024-03-28T18:55:46.604609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614151+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:55:46.614151+0000","last_peered":"2024-03-28T18:55:46.614151+0000","last_clean":"2024-03-28T18:55:46.614151+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:55:46.614151+0000","last_undegraded":"2024-03-28T18:55:46.614151+0000","last_fullsized":"2024-03-28T18:55:46.614151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163424+0000","last_change":"2024-03-28T18:55:45.612284+0000","last_active":"2024-03-28T18:55:47.163424+0000","last_peered":"2024-03-28T18:55:47.163424+0000","last_clean":"2024-03-28T18:55:47.163424+0000","last_became_active":"2024-03-28T18:55:45.607218+0000","last_became_peered":"2024-03-28T18:55:45.607218+0000","last_unstale":"2024-03-28T18:55:47.163424+0000","last_undegraded":"2024-03-28T18:55:47.163424+0000","last_fullsized":"2024-03-28T18:55:47.163424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604943+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:55:46.604943+0000","last_peered":"2024-03-28T18:55:46.604943+0000","last_clean":"2024-03-28T18:55:46.604943+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:55:46.604943+0000","last_undegraded":"2024-03-28T18:55:46.604943+0000","last_fullsized":"2024-03-28T18:55:46.604943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603065+0000","last_change":"2024-03-28T18:55:45.607609+0000","last_active":"2024-03-28T18:55:46.603065+0000","last_peered":"2024-03-28T18:55:46.603065+0000","last_clean":"2024-03-28T18:55:46.603065+0000","last_became_active":"2024-03-28T18:55:45.602582+0000","last_became_peered":"2024-03-28T18:55:45.602582+0000","last_unstale":"2024-03-28T18:55:46.603065+0000","last_undegraded":"2024-03-28T18:55:46.603065+0000","last_fullsized":"2024-03-28T18:55:46.603065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:06:05.335715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829082+0000","last_change":"2024-03-28T18:55:34.491857+0000","last_active":"2024-03-28T18:55:46.829082+0000","last_peered":"2024-03-28T18:55:46.829082+0000","last_clean":"2024-03-28T18:55:46.829082+0000","last_became_active":"2024-03-28T18:55:34.487194+0000","last_became_peered":"2024-03-28T18:55:34.487194+0000","last_unstale":"2024-03-28T18:55:46.829082+0000","last_undegraded":"2024-03-28T18:55:46.829082+0000","last_fullsized":"2024-03-28T18:55:46.829082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828429+0000","last_change":"2024-03-28T18:55:45.607886+0000","last_active":"2024-03-28T18:55:46.828429+0000","last_peered":"2024-03-28T18:55:46.828429+0000","last_clean":"2024-03-28T18:55:46.828429+0000","last_became_active":"2024-03-28T18:55:45.605964+0000","last_became_peered":"2024-03-28T18:55:45.605964+0000","last_unstale":"2024-03-28T18:55:46.828429+0000","last_undegraded":"2024-03-28T18:55:46.828429+0000","last_fullsized":"2024-03-28T18:55:46.828429+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:45.986654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117089+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:55:47.117089+0000","last_peered":"2024-03-28T18:55:47.117089+0000","last_clean":"2024-03-28T18:55:47.117089+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:55:47.117089+0000","last_undegraded":"2024-03-28T18:55:47.117089+0000","last_fullsized":"2024-03-28T18:55:47.117089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165971+0000","last_change":"2024-03-28T18:55:45.618031+0000","last_active":"2024-03-28T18:55:47.165971+0000","last_peered":"2024-03-28T18:55:47.165971+0000","last_clean":"2024-03-28T18:55:47.165971+0000","last_became_active":"2024-03-28T18:55:45.617719+0000","last_became_peered":"2024-03-28T18:55:45.617719+0000","last_unstale":"2024-03-28T18:55:47.165971+0000","last_undegraded":"2024-03-28T18:55:47.165971+0000","last_fullsized":"2024-03-28T18:55:47.165971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826204+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:55:46.826204+0000","last_peered":"2024-03-28T18:55:46.826204+0000","last_clean":"2024-03-28T18:55:46.826204+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:55:46.826204+0000","last_undegraded":"2024-03-28T18:55:46.826204+0000","last_fullsized":"2024-03-28T18:55:46.826204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603156+0000","last_change":"2024-03-28T18:55:45.603783+0000","last_active":"2024-03-28T18:55:46.603156+0000","last_peered":"2024-03-28T18:55:46.603156+0000","last_clean":"2024-03-28T18:55:46.603156+0000","last_became_active":"2024-03-28T18:55:45.603474+0000","last_became_peered":"2024-03-28T18:55:45.603474+0000","last_unstale":"2024-03-28T18:55:46.603156+0000","last_undegraded":"2024-03-28T18:55:46.603156+0000","last_fullsized":"2024-03-28T18:55:46.603156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:42.328182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116551+0000","last_change":"2024-03-28T18:55:34.505741+0000","last_active":"2024-03-28T18:55:47.116551+0000","last_peered":"2024-03-28T18:55:47.116551+0000","last_clean":"2024-03-28T18:55:47.116551+0000","last_became_active":"2024-03-28T18:55:34.505440+0000","last_became_peered":"2024-03-28T18:55:34.505440+0000","last_unstale":"2024-03-28T18:55:47.116551+0000","last_undegraded":"2024-03-28T18:55:47.116551+0000","last_fullsized":"2024-03-28T18:55:47.116551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827452+0000","last_change":"2024-03-28T18:55:45.618912+0000","last_active":"2024-03-28T18:55:46.827452+0000","last_peered":"2024-03-28T18:55:46.827452+0000","last_clean":"2024-03-28T18:55:46.827452+0000","last_became_active":"2024-03-28T18:55:45.618554+0000","last_became_peered":"2024-03-28T18:55:45.618554+0000","last_unstale":"2024-03-28T18:55:46.827452+0000","last_undegraded":"2024-03-28T18:55:46.827452+0000","last_fullsized":"2024-03-28T18:55:46.827452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608624+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:55:46.608624+0000","last_peered":"2024-03-28T18:55:46.608624+0000","last_clean":"2024-03-28T18:55:46.608624+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:55:46.608624+0000","last_undegraded":"2024-03-28T18:55:46.608624+0000","last_fullsized":"2024-03-28T18:55:46.608624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604333+0000","last_change":"2024-03-28T18:55:45.616420+0000","last_active":"2024-03-28T18:55:46.604333+0000","last_peered":"2024-03-28T18:55:46.604333+0000","last_clean":"2024-03-28T18:55:46.604333+0000","last_became_active":"2024-03-28T18:55:45.614806+0000","last_became_peered":"2024-03-28T18:55:45.614806+0000","last_unstale":"2024-03-28T18:55:46.604333+0000","last_undegraded":"2024-03-28T18:55:46.604333+0000","last_fullsized":"2024-03-28T18:55:46.604333+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602688+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:55:46.602688+0000","last_peered":"2024-03-28T18:55:46.602688+0000","last_clean":"2024-03-28T18:55:46.602688+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:55:46.602688+0000","last_undegraded":"2024-03-28T18:55:46.602688+0000","last_fullsized":"2024-03-28T18:55:46.602688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604338+0000","last_change":"2024-03-28T18:55:45.606507+0000","last_active":"2024-03-28T18:55:46.604338+0000","last_peered":"2024-03-28T18:55:46.604338+0000","last_clean":"2024-03-28T18:55:46.604338+0000","last_became_active":"2024-03-28T18:55:45.606205+0000","last_became_peered":"2024-03-28T18:55:45.606205+0000","last_unstale":"2024-03-28T18:55:46.604338+0000","last_undegraded":"2024-03-28T18:55:46.604338+0000","last_fullsized":"2024-03-28T18:55:46.604338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164913+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:55:47.164913+0000","last_peered":"2024-03-28T18:55:47.164913+0000","last_clean":"2024-03-28T18:55:47.164913+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:55:47.164913+0000","last_undegraded":"2024-03-28T18:55:47.164913+0000","last_fullsized":"2024-03-28T18:55:47.164913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116033+0000","last_change":"2024-03-28T18:55:45.607890+0000","last_active":"2024-03-28T18:55:47.116033+0000","last_peered":"2024-03-28T18:55:47.116033+0000","last_clean":"2024-03-28T18:55:47.116033+0000","last_became_active":"2024-03-28T18:55:45.607590+0000","last_became_peered":"2024-03-28T18:55:45.607590+0000","last_unstale":"2024-03-28T18:55:47.116033+0000","last_undegraded":"2024-03-28T18:55:47.116033+0000","last_fullsized":"2024-03-28T18:55:47.116033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603209+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:55:46.603209+0000","last_peered":"2024-03-28T18:55:46.603209+0000","last_clean":"2024-03-28T18:55:46.603209+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:55:46.603209+0000","last_undegraded":"2024-03-28T18:55:46.603209+0000","last_fullsized":"2024-03-28T18:55:46.603209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608156+0000","last_change":"2024-03-28T18:55:45.608851+0000","last_active":"2024-03-28T18:55:46.608156+0000","last_peered":"2024-03-28T18:55:46.608156+0000","last_clean":"2024-03-28T18:55:46.608156+0000","last_became_active":"2024-03-28T18:55:45.608496+0000","last_became_peered":"2024-03-28T18:55:45.608496+0000","last_unstale":"2024-03-28T18:55:46.608156+0000","last_undegraded":"2024-03-28T18:55:46.608156+0000","last_fullsized":"2024-03-28T18:55:46.608156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118072+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:55:47.118072+0000","last_peered":"2024-03-28T18:55:47.118072+0000","last_clean":"2024-03-28T18:55:47.118072+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:55:47.118072+0000","last_undegraded":"2024-03-28T18:55:47.118072+0000","last_fullsized":"2024-03-28T18:55:47.118072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613556+0000","last_change":"2024-03-28T18:55:45.619501+0000","last_active":"2024-03-28T18:55:46.613556+0000","last_peered":"2024-03-28T18:55:46.613556+0000","last_clean":"2024-03-28T18:55:46.613556+0000","last_became_active":"2024-03-28T18:55:45.619212+0000","last_became_peered":"2024-03-28T18:55:45.619212+0000","last_unstale":"2024-03-28T18:55:46.613556+0000","last_undegraded":"2024-03-28T18:55:46.613556+0000","last_fullsized":"2024-03-28T18:55:46.613556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116821+0000","last_change":"2024-03-28T18:55:34.509529+0000","last_active":"2024-03-28T18:55:47.116821+0000","last_peered":"2024-03-28T18:55:47.116821+0000","last_clean":"2024-03-28T18:55:47.116821+0000","last_became_active":"2024-03-28T18:55:34.501100+0000","last_became_peered":"2024-03-28T18:55:34.501100+0000","last_unstale":"2024-03-28T18:55:47.116821+0000","last_undegraded":"2024-03-28T18:55:47.116821+0000","last_fullsized":"2024-03-28T18:55:47.116821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605306+0000","last_change":"2024-03-28T18:55:45.615225+0000","last_active":"2024-03-28T18:55:46.605306+0000","last_peered":"2024-03-28T18:55:46.605306+0000","last_clean":"2024-03-28T18:55:46.605306+0000","last_became_active":"2024-03-28T18:55:45.613602+0000","last_became_peered":"2024-03-28T18:55:45.613602+0000","last_unstale":"2024-03-28T18:55:46.605306+0000","last_undegraded":"2024-03-28T18:55:46.605306+0000","last_fullsized":"2024-03-28T18:55:46.605306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612997+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:55:46.612997+0000","last_peered":"2024-03-28T18:55:46.612997+0000","last_clean":"2024-03-28T18:55:46.612997+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:55:46.612997+0000","last_undegraded":"2024-03-28T18:55:46.612997+0000","last_fullsized":"2024-03-28T18:55:46.612997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116582+0000","last_change":"2024-03-28T18:55:45.609471+0000","last_active":"2024-03-28T18:55:47.116582+0000","last_peered":"2024-03-28T18:55:47.116582+0000","last_clean":"2024-03-28T18:55:47.116582+0000","last_became_active":"2024-03-28T18:55:45.609132+0000","last_became_peered":"2024-03-28T18:55:45.609132+0000","last_unstale":"2024-03-28T18:55:47.116582+0000","last_undegraded":"2024-03-28T18:55:47.116582+0000","last_fullsized":"2024-03-28T18:55:47.116582+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608272+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:55:46.608272+0000","last_peered":"2024-03-28T18:55:46.608272+0000","last_clean":"2024-03-28T18:55:46.608272+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:55:46.608272+0000","last_undegraded":"2024-03-28T18:55:46.608272+0000","last_fullsized":"2024-03-28T18:55:46.608272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164529+0000","last_change":"2024-03-28T18:55:45.617501+0000","last_active":"2024-03-28T18:55:47.164529+0000","last_peered":"2024-03-28T18:55:47.164529+0000","last_clean":"2024-03-28T18:55:47.164529+0000","last_became_active":"2024-03-28T18:55:45.617180+0000","last_became_peered":"2024-03-28T18:55:45.617180+0000","last_unstale":"2024-03-28T18:55:47.164529+0000","last_undegraded":"2024-03-28T18:55:47.164529+0000","last_fullsized":"2024-03-28T18:55:47.164529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826964+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:55:46.826964+0000","last_peered":"2024-03-28T18:55:46.826964+0000","last_clean":"2024-03-28T18:55:46.826964+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:55:46.826964+0000","last_undegraded":"2024-03-28T18:55:46.826964+0000","last_fullsized":"2024-03-28T18:55:46.826964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826286+0000","last_change":"2024-03-28T18:55:45.609169+0000","last_active":"2024-03-28T18:55:46.826286+0000","last_peered":"2024-03-28T18:55:46.826286+0000","last_clean":"2024-03-28T18:55:46.826286+0000","last_became_active":"2024-03-28T18:55:45.601948+0000","last_became_peered":"2024-03-28T18:55:45.601948+0000","last_unstale":"2024-03-28T18:55:46.826286+0000","last_undegraded":"2024-03-28T18:55:46.826286+0000","last_fullsized":"2024-03-28T18:55:46.826286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605787+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:55:46.605787+0000","last_peered":"2024-03-28T18:55:46.605787+0000","last_clean":"2024-03-28T18:55:46.605787+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:55:46.605787+0000","last_undegraded":"2024-03-28T18:55:46.605787+0000","last_fullsized":"2024-03-28T18:55:46.605787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603014+0000","last_change":"2024-03-28T18:55:45.616238+0000","last_active":"2024-03-28T18:55:46.603014+0000","last_peered":"2024-03-28T18:55:46.603014+0000","last_clean":"2024-03-28T18:55:46.603014+0000","last_became_active":"2024-03-28T18:55:45.615371+0000","last_became_peered":"2024-03-28T18:55:45.615371+0000","last_unstale":"2024-03-28T18:55:46.603014+0000","last_undegraded":"2024-03-28T18:55:46.603014+0000","last_fullsized":"2024-03-28T18:55:46.603014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:14.661134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603091+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:55:46.603091+0000","last_peered":"2024-03-28T18:55:46.603091+0000","last_clean":"2024-03-28T18:55:46.603091+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:55:46.603091+0000","last_undegraded":"2024-03-28T18:55:46.603091+0000","last_fullsized":"2024-03-28T18:55:46.603091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164732+0000","last_change":"2024-03-28T18:55:45.613289+0000","last_active":"2024-03-28T18:55:47.164732+0000","last_peered":"2024-03-28T18:55:47.164732+0000","last_clean":"2024-03-28T18:55:47.164732+0000","last_became_active":"2024-03-28T18:55:45.610917+0000","last_became_peered":"2024-03-28T18:55:45.610917+0000","last_unstale":"2024-03-28T18:55:47.164732+0000","last_undegraded":"2024-03-28T18:55:47.164732+0000","last_fullsized":"2024-03-28T18:55:47.164732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604796+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:55:46.604796+0000","last_peered":"2024-03-28T18:55:46.604796+0000","last_clean":"2024-03-28T18:55:46.604796+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:55:46.604796+0000","last_undegraded":"2024-03-28T18:55:46.604796+0000","last_fullsized":"2024-03-28T18:55:46.604796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116103+0000","last_change":"2024-03-28T18:55:45.621789+0000","last_active":"2024-03-28T18:55:47.116103+0000","last_peered":"2024-03-28T18:55:47.116103+0000","last_clean":"2024-03-28T18:55:47.116103+0000","last_became_active":"2024-03-28T18:55:45.621163+0000","last_became_peered":"2024-03-28T18:55:45.621163+0000","last_unstale":"2024-03-28T18:55:47.116103+0000","last_undegraded":"2024-03-28T18:55:47.116103+0000","last_fullsized":"2024-03-28T18:55:47.116103+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166307+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:55:47.166307+0000","last_peered":"2024-03-28T18:55:47.166307+0000","last_clean":"2024-03-28T18:55:47.166307+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:55:47.166307+0000","last_undegraded":"2024-03-28T18:55:47.166307+0000","last_fullsized":"2024-03-28T18:55:47.166307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828245+0000","last_change":"2024-03-28T18:55:45.603774+0000","last_active":"2024-03-28T18:55:46.828245+0000","last_peered":"2024-03-28T18:55:46.828245+0000","last_clean":"2024-03-28T18:55:46.828245+0000","last_became_active":"2024-03-28T18:55:45.603451+0000","last_became_peered":"2024-03-28T18:55:45.603451+0000","last_unstale":"2024-03-28T18:55:46.828245+0000","last_undegraded":"2024-03-28T18:55:46.828245+0000","last_fullsized":"2024-03-28T18:55:46.828245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826131+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:55:46.826131+0000","last_peered":"2024-03-28T18:55:46.826131+0000","last_clean":"2024-03-28T18:55:46.826131+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:55:46.826131+0000","last_undegraded":"2024-03-28T18:55:46.826131+0000","last_fullsized":"2024-03-28T18:55:46.826131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603950+0000","last_change":"2024-03-28T18:55:45.616755+0000","last_active":"2024-03-28T18:55:46.603950+0000","last_peered":"2024-03-28T18:55:46.603950+0000","last_clean":"2024-03-28T18:55:46.603950+0000","last_became_active":"2024-03-28T18:55:45.614611+0000","last_became_peered":"2024-03-28T18:55:45.614611+0000","last_unstale":"2024-03-28T18:55:46.603950+0000","last_undegraded":"2024-03-28T18:55:46.603950+0000","last_fullsized":"2024-03-28T18:55:46.603950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:06.327795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163860+0000","last_change":"2024-03-28T18:55:34.491855+0000","last_active":"2024-03-28T18:55:47.163860+0000","last_peered":"2024-03-28T18:55:47.163860+0000","last_clean":"2024-03-28T18:55:47.163860+0000","last_became_active":"2024-03-28T18:55:34.491070+0000","last_became_peered":"2024-03-28T18:55:34.491070+0000","last_unstale":"2024-03-28T18:55:47.163860+0000","last_undegraded":"2024-03-28T18:55:47.163860+0000","last_fullsized":"2024-03-28T18:55:47.163860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613597+0000","last_change":"2024-03-28T18:55:45.623195+0000","last_active":"2024-03-28T18:55:46.613597+0000","last_peered":"2024-03-28T18:55:46.613597+0000","last_clean":"2024-03-28T18:55:46.613597+0000","last_became_active":"2024-03-28T18:55:45.622508+0000","last_became_peered":"2024-03-28T18:55:45.622508+0000","last_unstale":"2024-03-28T18:55:46.613597+0000","last_undegraded":"2024-03-28T18:55:46.613597+0000","last_fullsized":"2024-03-28T18:55:46.613597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116530+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:55:47.116530+0000","last_peered":"2024-03-28T18:55:47.116530+0000","last_clean":"2024-03-28T18:55:47.116530+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:55:47.116530+0000","last_undegraded":"2024-03-28T18:55:47.116530+0000","last_fullsized":"2024-03-28T18:55:47.116530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117584+0000","last_change":"2024-03-28T18:55:45.618841+0000","last_active":"2024-03-28T18:55:47.117584+0000","last_peered":"2024-03-28T18:55:47.117584+0000","last_clean":"2024-03-28T18:55:47.117584+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:55:47.117584+0000","last_undegraded":"2024-03-28T18:55:47.117584+0000","last_fullsized":"2024-03-28T18:55:47.117584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164332+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:55:47.164332+0000","last_peered":"2024-03-28T18:55:47.164332+0000","last_clean":"2024-03-28T18:55:47.164332+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:55:47.164332+0000","last_undegraded":"2024-03-28T18:55:47.164332+0000","last_fullsized":"2024-03-28T18:55:47.164332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825990+0000","last_change":"2024-03-28T18:55:45.602886+0000","last_active":"2024-03-28T18:55:46.825990+0000","last_peered":"2024-03-28T18:55:46.825990+0000","last_clean":"2024-03-28T18:55:46.825990+0000","last_became_active":"2024-03-28T18:55:45.602610+0000","last_became_peered":"2024-03-28T18:55:45.602610+0000","last_unstale":"2024-03-28T18:55:46.825990+0000","last_undegraded":"2024-03-28T18:55:46.825990+0000","last_fullsized":"2024-03-28T18:55:46.825990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116836+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:55:47.116836+0000","last_peered":"2024-03-28T18:55:47.116836+0000","last_clean":"2024-03-28T18:55:47.116836+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:55:47.116836+0000","last_undegraded":"2024-03-28T18:55:47.116836+0000","last_fullsized":"2024-03-28T18:55:47.116836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602814+0000","last_change":"2024-03-28T18:55:45.608673+0000","last_active":"2024-03-28T18:55:46.602814+0000","last_peered":"2024-03-28T18:55:46.602814+0000","last_clean":"2024-03-28T18:55:46.602814+0000","last_became_active":"2024-03-28T18:55:45.608376+0000","last_became_peered":"2024-03-28T18:55:45.608376+0000","last_unstale":"2024-03-28T18:55:46.602814+0000","last_undegraded":"2024-03-28T18:55:46.602814+0000","last_fullsized":"2024-03-28T18:55:46.602814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:31.781549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608764+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:55:46.608764+0000","last_peered":"2024-03-28T18:55:46.608764+0000","last_clean":"2024-03-28T18:55:46.608764+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:55:46.608764+0000","last_undegraded":"2024-03-28T18:55:46.608764+0000","last_fullsized":"2024-03-28T18:55:46.608764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163497+0000","last_change":"2024-03-28T18:55:45.609565+0000","last_active":"2024-03-28T18:55:47.163497+0000","last_peered":"2024-03-28T18:55:47.163497+0000","last_clean":"2024-03-28T18:55:47.163497+0000","last_became_active":"2024-03-28T18:55:45.609082+0000","last_became_peered":"2024-03-28T18:55:45.609082+0000","last_unstale":"2024-03-28T18:55:47.163497+0000","last_undegraded":"2024-03-28T18:55:47.163497+0000","last_fullsized":"2024-03-28T18:55:47.163497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613540+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:55:46.613540+0000","last_peered":"2024-03-28T18:55:46.613540+0000","last_clean":"2024-03-28T18:55:46.613540+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:55:46.613540+0000","last_undegraded":"2024-03-28T18:55:46.613540+0000","last_fullsized":"2024-03-28T18:55:46.613540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163662+0000","last_change":"2024-03-28T18:55:45.618585+0000","last_active":"2024-03-28T18:55:47.163662+0000","last_peered":"2024-03-28T18:55:47.163662+0000","last_clean":"2024-03-28T18:55:47.163662+0000","last_became_active":"2024-03-28T18:55:45.617895+0000","last_became_peered":"2024-03-28T18:55:45.617895+0000","last_unstale":"2024-03-28T18:55:47.163662+0000","last_undegraded":"2024-03-28T18:55:47.163662+0000","last_fullsized":"2024-03-28T18:55:47.163662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:54.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166181+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:55:47.166181+0000","last_peered":"2024-03-28T18:55:47.166181+0000","last_clean":"2024-03-28T18:55:47.166181+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:55:47.166181+0000","last_undegraded":"2024-03-28T18:55:47.166181+0000","last_fullsized":"2024-03-28T18:55:47.166181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":856064,"data_stored":690301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":101489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":99491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":102507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":101055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:55:51.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:55:51.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.453+0000 7f001a000640 1 Processor -- start 2024-03-28T18:55:51.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.453+0000 7f001a000640 1 -- start start 2024-03-28T18:55:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.455+0000 7f001a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 0x7f001415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.455+0000 7f001a000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00140c2090 con 0x7f0014071bd0 2024-03-28T18:55:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.455+0000 7f001a000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001406c0d0 con 0x7f001415a500 2024-03-28T18:55:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.455+0000 7f001a000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001415dd20 con 0x7f0014072610 2024-03-28T18:55:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.455+0000 7f0013e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0014071bd0 0x7f001415d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59684/0 (socket says 172.21.15.27:59684) 2024-03-28T18:55:51.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.455+0000 7f0013e00640 1 -- 172.21.15.27:0/1593894616 learned_addr learned my addr 172.21.15.27:0/1593894616 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0019600640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 0x7f001415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085158299 0 0) 0x7f00140c2090 con 0x7f0014071bd0 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efffc003520 con 0x7f0014071bd0 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580448618 0 0) 0x7efffc003520 con 0x7f0014071bd0 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 msgr2=0x7f001415d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 0x7f001415d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 >> v1:172.21.15.155:6789/0 conn(0x7f001415a500 legacy=0x7f00140c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001406c3b0 con 0x7f0014071bd0 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f0000001670 con 0x7f0014071bd0 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0019600640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 0x7f001415d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 >> v1:172.21.15.27:6789/0 conn(0x7f0014071bd0 legacy=0x7f001415d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7efffc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efffc004800 0x7efffc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 0x7efffc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001415dd20 con 0x7efffc0039e0 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc00ba70 con 0x7efffc004800 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.456+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efffc00bc70 con 0x7f0014072610 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.457+0000 7f0018c00640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7efffc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.457+0000 7f0013e00640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efffc004800 0x7efffc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.457+0000 7f0019600640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 0x7efffc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.457+0000 7f0018c00640 1 -- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 msgr2=0x7efffc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:51.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.457+0000 7f0018c00640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0014072610 0x7efffc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.457+0000 7f0018c00640 1 -- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efffc004800 msgr2=0x7efffc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:51.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.457+0000 7f0018c00640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efffc004800 0x7efffc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.457+0000 7f0018c00640 1 -- 172.21.15.27:0/1593894616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc003520 con 0x7efffc0039e0 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.458+0000 7f0018c00640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7efffc003dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f00080018e0 tx=0x7efffc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10210cc2931589eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.459+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00080014e0 con 0x7efffc0039e0 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.459+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0008037070 con 0x7efffc0039e0 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.459+0000 7f0013400640 1 -- 172.21.15.27:0/1593894616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008032820 con 0x7efffc0039e0 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.459+0000 7f001a000640 1 -- 172.21.15.27:0/1593894616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 msgr2=0x7efffc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.459+0000 7f001a000640 1 --2- 172.21.15.27:0/1593894616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7efffc003dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f00080018e0 tx=0x7efffc005830 comp rx=0 tx=0).stop 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.459+0000 7f0019600640 1 -- 172.21.15.27:0/1593894616 reap_dead start 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.460+0000 7f001a000640 1 -- 172.21.15.27:0/1593894616 shutdown_connections 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.460+0000 7f001a000640 1 -- 172.21.15.27:0/1593894616 >> 172.21.15.27:0/1593894616 conn(0x7f00140c7aa0 msgr2=0x7f00140c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.460+0000 7f001a000640 1 -- 172.21.15.27:0/1593894616 shutdown_connections 2024-03-28T18:55:51.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.460+0000 7f001a000640 1 -- 172.21.15.27:0/1593894616 wait complete. 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.461+0000 7f001a000640 1 Processor -- start 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.462+0000 7f001a000640 1 -- start start 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.462+0000 7f001a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7f001406e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.462+0000 7f001a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001415ca20 0x7f00140abb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.462+0000 7f001a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00141efa50 0x7f0014159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.462+0000 7f001a000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00140c1d20 con 0x7efffc0039e0 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.462+0000 7f001a000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001406c970 con 0x7f00141efa50 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.462+0000 7f001a000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00140c6b30 con 0x7f001415ca20 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7f001406e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7f001406e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59326/0 (socket says 172.21.15.27:59326) 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 -- 172.21.15.27:0/1290192706 learned_addr learned my addr 172.21.15.27:0/1290192706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0013e00640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001415ca20 0x7f00140abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0019600640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00141efa50 0x7f0014159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 -- 172.21.15.27:0/1290192706 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001415ca20 msgr2=0x7f00140abb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001415ca20 0x7f00140abb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 -- 172.21.15.27:0/1290192706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00141efa50 msgr2=0x7f0014159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00141efa50 0x7f0014159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 -- 172.21.15.27:0/1290192706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001406e640 con 0x7efffc0039e0 2024-03-28T18:55:51.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.463+0000 7f0018c00640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7f001406e100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f000802be20 tx=0x7f00080012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.464+0000 7f0010c00640 1 -- 172.21.15.27:0/1290192706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008007a70 con 0x7efffc0039e0 2024-03-28T18:55:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.464+0000 7f0010c00640 1 -- 172.21.15.27:0/1290192706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0008040070 con 0x7efffc0039e0 2024-03-28T18:55:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.464+0000 7f0010c00640 1 -- 172.21.15.27:0/1290192706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00080014e0 con 0x7efffc0039e0 2024-03-28T18:55:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.464+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001406fff0 con 0x7efffc0039e0 2024-03-28T18:55:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.464+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001406f8d0 con 0x7efffc0039e0 2024-03-28T18:55:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.465+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0014069640 con 0x7efffc0039e0 2024-03-28T18:55:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.466+0000 7f0010c00640 1 -- 172.21.15.27:0/1290192706 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0008031070 con 0x7efffc0039e0 2024-03-28T18:55:51.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.468+0000 7f0010c00640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.468+0000 7f0010c00640 1 -- 172.21.15.27:0/1290192706 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f00080ac640 con 0x7efffc0039e0 2024-03-28T18:55:51.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.468+0000 7f0013e00640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.469+0000 7f0013e00640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff406bbd0 0x7efff406e090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0000001c80 tx=0x7f0000001d50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:51.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.472+0000 7f0010c00640 1 -- 172.21.15.27:0/1290192706 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f001406fff0 con 0x7efffc0039e0 2024-03-28T18:55:51.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.674+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0014072070 con 0x7efff406bbd0 2024-03-28T18:55:51.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.687+0000 7f0010c00640 1 -- 172.21.15.27:0/1290192706 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490417 (secure 0 0 0) 0x7f0014072070 con 0x7efff406bbd0 2024-03-28T18:55:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:51.700 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:55:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff406bbd0 msgr2=0x7efff406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff406bbd0 0x7efff406e090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0000001c80 tx=0x7f0000001d50 comp rx=0 tx=0).stop 2024-03-28T18:55:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 msgr2=0x7f001406e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7f001406e100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f000802be20 tx=0x7f00080012b0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 shutdown_connections 2024-03-28T18:55:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00141efa50 0x7f0014159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001415ca20 0x7f00140abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.696+0000 7f001a000640 1 --2- 172.21.15.27:0/1290192706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efffc0039e0 0x7f001406e100 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.697+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 >> 172.21.15.27:0/1290192706 conn(0x7f00140c7aa0 msgr2=0x7f00140c3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.697+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 shutdown_connections 2024-03-28T18:55:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.697+0000 7f001a000640 1 -- 172.21.15.27:0/1290192706 wait complete. 2024-03-28T18:55:51.715 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-28T18:55:50.896791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":316,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219448,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755272,"statfs":{"total":773094113280,"available":772869398528,"internally_reserved":0,"allocated":3350528,"data_stored":1984464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.326228"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603978+0000","last_change":"2024-03-28T18:55:45.603782+0000","last_active":"2024-03-28T18:55:46.603978+0000","last_peered":"2024-03-28T18:55:46.603978+0000","last_clean":"2024-03-28T18:55:46.603978+0000","last_became_active":"2024-03-28T18:55:45.603420+0000","last_became_peered":"2024-03-28T18:55:45.603420+0000","last_unstale":"2024-03-28T18:55:46.603978+0000","last_undegraded":"2024-03-28T18:55:46.603978+0000","last_fullsized":"2024-03-28T18:55:46.603978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609332+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:55:46.609332+0000","last_peered":"2024-03-28T18:55:46.609332+0000","last_clean":"2024-03-28T18:55:46.609332+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:55:46.609332+0000","last_undegraded":"2024-03-28T18:55:46.609332+0000","last_fullsized":"2024-03-28T18:55:46.609332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828788+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:55:46.828788+0000","last_peered":"2024-03-28T18:55:46.828788+0000","last_clean":"2024-03-28T18:55:46.828788+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:55:46.828788+0000","last_undegraded":"2024-03-28T18:55:46.828788+0000","last_fullsized":"2024-03-28T18:55:46.828788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825968+0000","last_change":"2024-03-28T18:55:44.602536+0000","last_active":"2024-03-28T18:55:46.825968+0000","last_peered":"2024-03-28T18:55:46.825968+0000","last_clean":"2024-03-28T18:55:46.825968+0000","last_became_active":"2024-03-28T18:55:44.602256+0000","last_became_peered":"2024-03-28T18:55:44.602256+0000","last_unstale":"2024-03-28T18:55:46.825968+0000","last_undegraded":"2024-03-28T18:55:46.825968+0000","last_fullsized":"2024-03-28T18:55:46.825968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612952+0000","last_change":"2024-03-28T18:55:45.625338+0000","last_active":"2024-03-28T18:55:46.612952+0000","last_peered":"2024-03-28T18:55:46.612952+0000","last_clean":"2024-03-28T18:55:46.612952+0000","last_became_active":"2024-03-28T18:55:45.625067+0000","last_became_peered":"2024-03-28T18:55:45.625067+0000","last_unstale":"2024-03-28T18:55:46.612952+0000","last_undegraded":"2024-03-28T18:55:46.612952+0000","last_fullsized":"2024-03-28T18:55:46.612952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829113+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:55:46.829113+0000","last_peered":"2024-03-28T18:55:46.829113+0000","last_clean":"2024-03-28T18:55:46.829113+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:55:46.829113+0000","last_undegraded":"2024-03-28T18:55:46.829113+0000","last_fullsized":"2024-03-28T18:55:46.829113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608925+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:55:46.608925+0000","last_peered":"2024-03-28T18:55:46.608925+0000","last_clean":"2024-03-28T18:55:46.608925+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:55:46.608925+0000","last_undegraded":"2024-03-28T18:55:46.608925+0000","last_fullsized":"2024-03-28T18:55:46.608925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604171+0000","last_change":"2024-03-28T18:55:44.598423+0000","last_active":"2024-03-28T18:55:46.604171+0000","last_peered":"2024-03-28T18:55:46.604171+0000","last_clean":"2024-03-28T18:55:46.604171+0000","last_became_active":"2024-03-28T18:55:44.598092+0000","last_became_peered":"2024-03-28T18:55:44.598092+0000","last_unstale":"2024-03-28T18:55:46.604171+0000","last_undegraded":"2024-03-28T18:55:46.604171+0000","last_fullsized":"2024-03-28T18:55:46.604171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.646619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163641+0000","last_change":"2024-03-28T18:55:45.608201+0000","last_active":"2024-03-28T18:55:47.163641+0000","last_peered":"2024-03-28T18:55:47.163641+0000","last_clean":"2024-03-28T18:55:47.163641+0000","last_became_active":"2024-03-28T18:55:45.607912+0000","last_became_peered":"2024-03-28T18:55:45.607912+0000","last_unstale":"2024-03-28T18:55:47.163641+0000","last_undegraded":"2024-03-28T18:55:47.163641+0000","last_fullsized":"2024-03-28T18:55:47.163641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164845+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:55:47.164845+0000","last_peered":"2024-03-28T18:55:47.164845+0000","last_clean":"2024-03-28T18:55:47.164845+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:55:47.164845+0000","last_undegraded":"2024-03-28T18:55:47.164845+0000","last_fullsized":"2024-03-28T18:55:47.164845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117090+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:55:47.117090+0000","last_peered":"2024-03-28T18:55:47.117090+0000","last_clean":"2024-03-28T18:55:47.117090+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:55:47.117090+0000","last_undegraded":"2024-03-28T18:55:47.117090+0000","last_fullsized":"2024-03-28T18:55:47.117090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604780+0000","last_change":"2024-03-28T18:55:44.604673+0000","last_active":"2024-03-28T18:55:46.604780+0000","last_peered":"2024-03-28T18:55:46.604780+0000","last_clean":"2024-03-28T18:55:46.604780+0000","last_became_active":"2024-03-28T18:55:44.603926+0000","last_became_peered":"2024-03-28T18:55:44.603926+0000","last_unstale":"2024-03-28T18:55:46.604780+0000","last_undegraded":"2024-03-28T18:55:46.604780+0000","last_fullsized":"2024-03-28T18:55:46.604780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116402+0000","last_change":"2024-03-28T18:55:45.610033+0000","last_active":"2024-03-28T18:55:47.116402+0000","last_peered":"2024-03-28T18:55:47.116402+0000","last_clean":"2024-03-28T18:55:47.116402+0000","last_became_active":"2024-03-28T18:55:45.609706+0000","last_became_peered":"2024-03-28T18:55:45.609706+0000","last_unstale":"2024-03-28T18:55:47.116402+0000","last_undegraded":"2024-03-28T18:55:47.116402+0000","last_fullsized":"2024-03-28T18:55:47.116402+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164190+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:55:47.164190+0000","last_peered":"2024-03-28T18:55:47.164190+0000","last_clean":"2024-03-28T18:55:47.164190+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:55:47.164190+0000","last_undegraded":"2024-03-28T18:55:47.164190+0000","last_fullsized":"2024-03-28T18:55:47.164190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117556+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:55:47.117556+0000","last_peered":"2024-03-28T18:55:47.117556+0000","last_clean":"2024-03-28T18:55:47.117556+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:55:47.117556+0000","last_undegraded":"2024-03-28T18:55:47.117556+0000","last_fullsized":"2024-03-28T18:55:47.117556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604505+0000","last_change":"2024-03-28T18:55:44.606715+0000","last_active":"2024-03-28T18:55:46.604505+0000","last_peered":"2024-03-28T18:55:46.604505+0000","last_clean":"2024-03-28T18:55:46.604505+0000","last_became_active":"2024-03-28T18:55:44.606343+0000","last_became_peered":"2024-03-28T18:55:44.606343+0000","last_unstale":"2024-03-28T18:55:46.604505+0000","last_undegraded":"2024-03-28T18:55:46.604505+0000","last_fullsized":"2024-03-28T18:55:46.604505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:51.471450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613822+0000","last_change":"2024-03-28T18:55:45.623483+0000","last_active":"2024-03-28T18:55:46.613822+0000","last_peered":"2024-03-28T18:55:46.613822+0000","last_clean":"2024-03-28T18:55:46.613822+0000","last_became_active":"2024-03-28T18:55:45.622894+0000","last_became_peered":"2024-03-28T18:55:45.622894+0000","last_unstale":"2024-03-28T18:55:46.613822+0000","last_undegraded":"2024-03-28T18:55:46.613822+0000","last_fullsized":"2024-03-28T18:55:46.613822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603964+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:55:46.603964+0000","last_peered":"2024-03-28T18:55:46.603964+0000","last_clean":"2024-03-28T18:55:46.603964+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:55:46.603964+0000","last_undegraded":"2024-03-28T18:55:46.603964+0000","last_fullsized":"2024-03-28T18:55:46.603964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164866+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:55:47.164866+0000","last_peered":"2024-03-28T18:55:47.164866+0000","last_clean":"2024-03-28T18:55:47.164866+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:55:47.164866+0000","last_undegraded":"2024-03-28T18:55:47.164866+0000","last_fullsized":"2024-03-28T18:55:47.164866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605818+0000","last_change":"2024-03-28T18:55:44.601415+0000","last_active":"2024-03-28T18:55:46.605818+0000","last_peered":"2024-03-28T18:55:46.605818+0000","last_clean":"2024-03-28T18:55:46.605818+0000","last_became_active":"2024-03-28T18:55:44.601124+0000","last_became_peered":"2024-03-28T18:55:44.601124+0000","last_unstale":"2024-03-28T18:55:46.605818+0000","last_undegraded":"2024-03-28T18:55:46.605818+0000","last_fullsized":"2024-03-28T18:55:46.605818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:30.767034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603094+0000","last_change":"2024-03-28T18:55:45.604795+0000","last_active":"2024-03-28T18:55:46.603094+0000","last_peered":"2024-03-28T18:55:46.603094+0000","last_clean":"2024-03-28T18:55:46.603094+0000","last_became_active":"2024-03-28T18:55:45.604432+0000","last_became_peered":"2024-03-28T18:55:45.604432+0000","last_unstale":"2024-03-28T18:55:46.603094+0000","last_undegraded":"2024-03-28T18:55:46.603094+0000","last_fullsized":"2024-03-28T18:55:46.603094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:37.638804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826963+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:55:46.826963+0000","last_peered":"2024-03-28T18:55:46.826963+0000","last_clean":"2024-03-28T18:55:46.826963+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:55:46.826963+0000","last_undegraded":"2024-03-28T18:55:46.826963+0000","last_fullsized":"2024-03-28T18:55:46.826963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613559+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:55:46.613559+0000","last_peered":"2024-03-28T18:55:46.613559+0000","last_clean":"2024-03-28T18:55:46.613559+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:55:46.613559+0000","last_undegraded":"2024-03-28T18:55:46.613559+0000","last_fullsized":"2024-03-28T18:55:46.613559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605718+0000","last_change":"2024-03-28T18:55:44.603859+0000","last_active":"2024-03-28T18:55:46.605718+0000","last_peered":"2024-03-28T18:55:46.605718+0000","last_clean":"2024-03-28T18:55:46.605718+0000","last_became_active":"2024-03-28T18:55:44.603540+0000","last_became_peered":"2024-03-28T18:55:44.603540+0000","last_unstale":"2024-03-28T18:55:46.605718+0000","last_undegraded":"2024-03-28T18:55:46.605718+0000","last_fullsized":"2024-03-28T18:55:46.605718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.606069+0000","last_change":"2024-03-28T18:55:45.608142+0000","last_active":"2024-03-28T18:55:46.606069+0000","last_peered":"2024-03-28T18:55:46.606069+0000","last_clean":"2024-03-28T18:55:46.606069+0000","last_became_active":"2024-03-28T18:55:45.606117+0000","last_became_peered":"2024-03-28T18:55:45.606117+0000","last_unstale":"2024-03-28T18:55:46.606069+0000","last_undegraded":"2024-03-28T18:55:46.606069+0000","last_fullsized":"2024-03-28T18:55:46.606069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826653+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:55:46.826653+0000","last_peered":"2024-03-28T18:55:46.826653+0000","last_clean":"2024-03-28T18:55:46.826653+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:55:46.826653+0000","last_undegraded":"2024-03-28T18:55:46.826653+0000","last_fullsized":"2024-03-28T18:55:46.826653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613045+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:55:46.613045+0000","last_peered":"2024-03-28T18:55:46.613045+0000","last_clean":"2024-03-28T18:55:46.613045+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:55:46.613045+0000","last_undegraded":"2024-03-28T18:55:46.613045+0000","last_fullsized":"2024-03-28T18:55:46.613045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614066+0000","last_change":"2024-03-28T18:55:44.606690+0000","last_active":"2024-03-28T18:55:46.614066+0000","last_peered":"2024-03-28T18:55:46.614066+0000","last_clean":"2024-03-28T18:55:46.614066+0000","last_became_active":"2024-03-28T18:55:44.606421+0000","last_became_peered":"2024-03-28T18:55:44.606421+0000","last_unstale":"2024-03-28T18:55:46.614066+0000","last_undegraded":"2024-03-28T18:55:46.614066+0000","last_fullsized":"2024-03-28T18:55:46.614066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117505+0000","last_change":"2024-03-28T18:55:45.610580+0000","last_active":"2024-03-28T18:55:47.117505+0000","last_peered":"2024-03-28T18:55:47.117505+0000","last_clean":"2024-03-28T18:55:47.117505+0000","last_became_active":"2024-03-28T18:55:45.610316+0000","last_became_peered":"2024-03-28T18:55:45.610316+0000","last_unstale":"2024-03-28T18:55:47.117505+0000","last_undegraded":"2024-03-28T18:55:47.117505+0000","last_fullsized":"2024-03-28T18:55:47.117505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604997+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:55:46.604997+0000","last_peered":"2024-03-28T18:55:46.604997+0000","last_clean":"2024-03-28T18:55:46.604997+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:55:46.604997+0000","last_undegraded":"2024-03-28T18:55:46.604997+0000","last_fullsized":"2024-03-28T18:55:46.604997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163833+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:55:47.163833+0000","last_peered":"2024-03-28T18:55:47.163833+0000","last_clean":"2024-03-28T18:55:47.163833+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:55:47.163833+0000","last_undegraded":"2024-03-28T18:55:47.163833+0000","last_fullsized":"2024-03-28T18:55:47.163833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.606207+0000","last_change":"2024-03-28T18:55:44.600523+0000","last_active":"2024-03-28T18:55:46.606207+0000","last_peered":"2024-03-28T18:55:46.606207+0000","last_clean":"2024-03-28T18:55:46.606207+0000","last_became_active":"2024-03-28T18:55:44.597550+0000","last_became_peered":"2024-03-28T18:55:44.597550+0000","last_unstale":"2024-03-28T18:55:46.606207+0000","last_undegraded":"2024-03-28T18:55:46.606207+0000","last_fullsized":"2024-03-28T18:55:46.606207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825927+0000","last_change":"2024-03-28T18:55:45.619798+0000","last_active":"2024-03-28T18:55:46.825927+0000","last_peered":"2024-03-28T18:55:46.825927+0000","last_clean":"2024-03-28T18:55:46.825927+0000","last_became_active":"2024-03-28T18:55:45.616517+0000","last_became_peered":"2024-03-28T18:55:45.616517+0000","last_unstale":"2024-03-28T18:55:46.825927+0000","last_undegraded":"2024-03-28T18:55:46.825927+0000","last_fullsized":"2024-03-28T18:55:46.825927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609158+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:55:46.609158+0000","last_peered":"2024-03-28T18:55:46.609158+0000","last_clean":"2024-03-28T18:55:46.609158+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:55:46.609158+0000","last_undegraded":"2024-03-28T18:55:46.609158+0000","last_fullsized":"2024-03-28T18:55:46.609158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605113+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:55:46.605113+0000","last_peered":"2024-03-28T18:55:46.605113+0000","last_clean":"2024-03-28T18:55:46.605113+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:55:46.605113+0000","last_undegraded":"2024-03-28T18:55:46.605113+0000","last_fullsized":"2024-03-28T18:55:46.605113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604527+0000","last_change":"2024-03-28T18:55:44.606498+0000","last_active":"2024-03-28T18:55:46.604527+0000","last_peered":"2024-03-28T18:55:46.604527+0000","last_clean":"2024-03-28T18:55:46.604527+0000","last_became_active":"2024-03-28T18:55:44.606098+0000","last_became_peered":"2024-03-28T18:55:44.606098+0000","last_unstale":"2024-03-28T18:55:46.604527+0000","last_undegraded":"2024-03-28T18:55:46.604527+0000","last_fullsized":"2024-03-28T18:55:46.604527+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605931+0000","last_change":"2024-03-28T18:55:45.617483+0000","last_active":"2024-03-28T18:55:46.605931+0000","last_peered":"2024-03-28T18:55:46.605931+0000","last_clean":"2024-03-28T18:55:46.605931+0000","last_became_active":"2024-03-28T18:55:45.617169+0000","last_became_peered":"2024-03-28T18:55:45.617169+0000","last_unstale":"2024-03-28T18:55:46.605931+0000","last_undegraded":"2024-03-28T18:55:46.605931+0000","last_fullsized":"2024-03-28T18:55:46.605931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829227+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:55:46.829227+0000","last_peered":"2024-03-28T18:55:46.829227+0000","last_clean":"2024-03-28T18:55:46.829227+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:55:46.829227+0000","last_undegraded":"2024-03-28T18:55:46.829227+0000","last_fullsized":"2024-03-28T18:55:46.829227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613667+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:55:46.613667+0000","last_peered":"2024-03-28T18:55:46.613667+0000","last_clean":"2024-03-28T18:55:46.613667+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:55:46.613667+0000","last_undegraded":"2024-03-28T18:55:46.613667+0000","last_fullsized":"2024-03-28T18:55:46.613667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825795+0000","last_change":"2024-03-28T18:55:44.601836+0000","last_active":"2024-03-28T18:55:46.825795+0000","last_peered":"2024-03-28T18:55:46.825795+0000","last_clean":"2024-03-28T18:55:46.825795+0000","last_became_active":"2024-03-28T18:55:44.601552+0000","last_became_peered":"2024-03-28T18:55:44.601552+0000","last_unstale":"2024-03-28T18:55:46.825795+0000","last_undegraded":"2024-03-28T18:55:46.825795+0000","last_fullsized":"2024-03-28T18:55:46.825795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603619+0000","last_change":"2024-03-28T18:55:45.601801+0000","last_active":"2024-03-28T18:55:46.603619+0000","last_peered":"2024-03-28T18:55:46.603619+0000","last_clean":"2024-03-28T18:55:46.603619+0000","last_became_active":"2024-03-28T18:55:45.601459+0000","last_became_peered":"2024-03-28T18:55:45.601459+0000","last_unstale":"2024-03-28T18:55:46.603619+0000","last_undegraded":"2024-03-28T18:55:46.603619+0000","last_fullsized":"2024-03-28T18:55:46.603619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:43.033170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613982+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:55:46.613982+0000","last_peered":"2024-03-28T18:55:46.613982+0000","last_clean":"2024-03-28T18:55:46.613982+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:55:46.613982+0000","last_undegraded":"2024-03-28T18:55:46.613982+0000","last_fullsized":"2024-03-28T18:55:46.613982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165028+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:55:47.165028+0000","last_peered":"2024-03-28T18:55:47.165028+0000","last_clean":"2024-03-28T18:55:47.165028+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:55:47.165028+0000","last_undegraded":"2024-03-28T18:55:47.165028+0000","last_fullsized":"2024-03-28T18:55:47.165028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608494+0000","last_change":"2024-03-28T18:55:44.611148+0000","last_active":"2024-03-28T18:55:46.608494+0000","last_peered":"2024-03-28T18:55:46.608494+0000","last_clean":"2024-03-28T18:55:46.608494+0000","last_became_active":"2024-03-28T18:55:44.610835+0000","last_became_peered":"2024-03-28T18:55:44.610835+0000","last_unstale":"2024-03-28T18:55:46.608494+0000","last_undegraded":"2024-03-28T18:55:46.608494+0000","last_fullsized":"2024-03-28T18:55:46.608494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608851+0000","last_change":"2024-03-28T18:55:45.623672+0000","last_active":"2024-03-28T18:55:46.608851+0000","last_peered":"2024-03-28T18:55:46.608851+0000","last_clean":"2024-03-28T18:55:46.608851+0000","last_became_active":"2024-03-28T18:55:45.622568+0000","last_became_peered":"2024-03-28T18:55:45.622568+0000","last_unstale":"2024-03-28T18:55:46.608851+0000","last_undegraded":"2024-03-28T18:55:46.608851+0000","last_fullsized":"2024-03-28T18:55:46.608851+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117731+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:55:47.117731+0000","last_peered":"2024-03-28T18:55:47.117731+0000","last_clean":"2024-03-28T18:55:47.117731+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:55:47.117731+0000","last_undegraded":"2024-03-28T18:55:47.117731+0000","last_fullsized":"2024-03-28T18:55:47.117731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614041+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:55:46.614041+0000","last_peered":"2024-03-28T18:55:46.614041+0000","last_clean":"2024-03-28T18:55:46.614041+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:55:46.614041+0000","last_undegraded":"2024-03-28T18:55:46.614041+0000","last_fullsized":"2024-03-28T18:55:46.614041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613204+0000","last_change":"2024-03-28T18:55:44.604077+0000","last_active":"2024-03-28T18:55:46.613204+0000","last_peered":"2024-03-28T18:55:46.613204+0000","last_clean":"2024-03-28T18:55:46.613204+0000","last_became_active":"2024-03-28T18:55:44.603773+0000","last_became_peered":"2024-03-28T18:55:44.603773+0000","last_unstale":"2024-03-28T18:55:46.613204+0000","last_undegraded":"2024-03-28T18:55:46.613204+0000","last_fullsized":"2024-03-28T18:55:46.613204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608665+0000","last_change":"2024-03-28T18:55:45.615444+0000","last_active":"2024-03-28T18:55:46.608665+0000","last_peered":"2024-03-28T18:55:46.608665+0000","last_clean":"2024-03-28T18:55:46.608665+0000","last_became_active":"2024-03-28T18:55:45.615002+0000","last_became_peered":"2024-03-28T18:55:45.615002+0000","last_unstale":"2024-03-28T18:55:46.608665+0000","last_undegraded":"2024-03-28T18:55:46.608665+0000","last_fullsized":"2024-03-28T18:55:46.608665+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605186+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:55:46.605186+0000","last_peered":"2024-03-28T18:55:46.605186+0000","last_clean":"2024-03-28T18:55:46.605186+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:55:46.605186+0000","last_undegraded":"2024-03-28T18:55:46.605186+0000","last_fullsized":"2024-03-28T18:55:46.605186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116841+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:55:47.116841+0000","last_peered":"2024-03-28T18:55:47.116841+0000","last_clean":"2024-03-28T18:55:47.116841+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:55:47.116841+0000","last_undegraded":"2024-03-28T18:55:47.116841+0000","last_fullsized":"2024-03-28T18:55:47.116841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613947+0000","last_change":"2024-03-28T18:55:44.590703+0000","last_active":"2024-03-28T18:55:46.613947+0000","last_peered":"2024-03-28T18:55:46.613947+0000","last_clean":"2024-03-28T18:55:46.613947+0000","last_became_active":"2024-03-28T18:55:44.590408+0000","last_became_peered":"2024-03-28T18:55:44.590408+0000","last_unstale":"2024-03-28T18:55:46.613947+0000","last_undegraded":"2024-03-28T18:55:46.613947+0000","last_fullsized":"2024-03-28T18:55:46.613947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603326+0000","last_change":"2024-03-28T18:55:45.609795+0000","last_active":"2024-03-28T18:55:46.603326+0000","last_peered":"2024-03-28T18:55:46.603326+0000","last_clean":"2024-03-28T18:55:46.603326+0000","last_became_active":"2024-03-28T18:55:45.603695+0000","last_became_peered":"2024-03-28T18:55:45.603695+0000","last_unstale":"2024-03-28T18:55:46.603326+0000","last_undegraded":"2024-03-28T18:55:46.603326+0000","last_fullsized":"2024-03-28T18:55:46.603326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:49.173555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612873+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:55:46.612873+0000","last_peered":"2024-03-28T18:55:46.612873+0000","last_clean":"2024-03-28T18:55:46.612873+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:55:46.612873+0000","last_undegraded":"2024-03-28T18:55:46.612873+0000","last_fullsized":"2024-03-28T18:55:46.612873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602878+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:55:46.602878+0000","last_peered":"2024-03-28T18:55:46.602878+0000","last_clean":"2024-03-28T18:55:46.602878+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:55:46.602878+0000","last_undegraded":"2024-03-28T18:55:46.602878+0000","last_fullsized":"2024-03-28T18:55:46.602878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614195+0000","last_change":"2024-03-28T18:55:44.612172+0000","last_active":"2024-03-28T18:55:46.614195+0000","last_peered":"2024-03-28T18:55:46.614195+0000","last_clean":"2024-03-28T18:55:46.614195+0000","last_became_active":"2024-03-28T18:55:44.611859+0000","last_became_peered":"2024-03-28T18:55:44.611859+0000","last_unstale":"2024-03-28T18:55:46.614195+0000","last_undegraded":"2024-03-28T18:55:46.614195+0000","last_fullsized":"2024-03-28T18:55:46.614195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117969+0000","last_change":"2024-03-28T18:55:45.619723+0000","last_active":"2024-03-28T18:55:47.117969+0000","last_peered":"2024-03-28T18:55:47.117969+0000","last_clean":"2024-03-28T18:55:47.117969+0000","last_became_active":"2024-03-28T18:55:45.619437+0000","last_became_peered":"2024-03-28T18:55:45.619437+0000","last_unstale":"2024-03-28T18:55:47.117969+0000","last_undegraded":"2024-03-28T18:55:47.117969+0000","last_fullsized":"2024-03-28T18:55:47.117969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608071+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:55:46.608071+0000","last_peered":"2024-03-28T18:55:46.608071+0000","last_clean":"2024-03-28T18:55:46.608071+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:55:46.608071+0000","last_undegraded":"2024-03-28T18:55:46.608071+0000","last_fullsized":"2024-03-28T18:55:46.608071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164105+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:55:47.164105+0000","last_peered":"2024-03-28T18:55:47.164105+0000","last_clean":"2024-03-28T18:55:47.164105+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:55:47.164105+0000","last_undegraded":"2024-03-28T18:55:47.164105+0000","last_fullsized":"2024-03-28T18:55:47.164105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.611391+0000","last_change":"2024-03-28T18:55:44.592064+0000","last_active":"2024-03-28T18:55:46.611391+0000","last_peered":"2024-03-28T18:55:46.611391+0000","last_clean":"2024-03-28T18:55:46.611391+0000","last_became_active":"2024-03-28T18:55:44.591263+0000","last_became_peered":"2024-03-28T18:55:44.591263+0000","last_unstale":"2024-03-28T18:55:46.611391+0000","last_undegraded":"2024-03-28T18:55:46.611391+0000","last_fullsized":"2024-03-28T18:55:46.611391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163958+0000","last_change":"2024-03-28T18:55:45.614820+0000","last_active":"2024-03-28T18:55:47.163958+0000","last_peered":"2024-03-28T18:55:47.163958+0000","last_clean":"2024-03-28T18:55:47.163958+0000","last_became_active":"2024-03-28T18:55:45.614517+0000","last_became_peered":"2024-03-28T18:55:45.614517+0000","last_unstale":"2024-03-28T18:55:47.163958+0000","last_undegraded":"2024-03-28T18:55:47.163958+0000","last_fullsized":"2024-03-28T18:55:47.163958+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826245+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:55:46.826245+0000","last_peered":"2024-03-28T18:55:46.826245+0000","last_clean":"2024-03-28T18:55:46.826245+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:55:46.826245+0000","last_undegraded":"2024-03-28T18:55:46.826245+0000","last_fullsized":"2024-03-28T18:55:46.826245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116294+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:55:47.116294+0000","last_peered":"2024-03-28T18:55:47.116294+0000","last_clean":"2024-03-28T18:55:47.116294+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:55:47.116294+0000","last_undegraded":"2024-03-28T18:55:47.116294+0000","last_fullsized":"2024-03-28T18:55:47.116294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164472+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:55:47.164472+0000","last_peered":"2024-03-28T18:55:47.164472+0000","last_clean":"2024-03-28T18:55:47.164472+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:55:47.164472+0000","last_undegraded":"2024-03-28T18:55:47.164472+0000","last_fullsized":"2024-03-28T18:55:47.164472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602907+0000","last_change":"2024-03-28T18:55:44.602097+0000","last_active":"2024-03-28T18:55:46.602907+0000","last_peered":"2024-03-28T18:55:46.602907+0000","last_clean":"2024-03-28T18:55:46.602907+0000","last_became_active":"2024-03-28T18:55:44.601769+0000","last_became_peered":"2024-03-28T18:55:44.601769+0000","last_unstale":"2024-03-28T18:55:46.602907+0000","last_undegraded":"2024-03-28T18:55:46.602907+0000","last_fullsized":"2024-03-28T18:55:46.602907+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603447+0000","last_change":"2024-03-28T18:55:45.616485+0000","last_active":"2024-03-28T18:55:46.603447+0000","last_peered":"2024-03-28T18:55:46.603447+0000","last_clean":"2024-03-28T18:55:46.603447+0000","last_became_active":"2024-03-28T18:55:45.615813+0000","last_became_peered":"2024-03-28T18:55:45.615813+0000","last_unstale":"2024-03-28T18:55:46.603447+0000","last_undegraded":"2024-03-28T18:55:46.603447+0000","last_fullsized":"2024-03-28T18:55:46.603447+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:21.227281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164685+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:55:47.164685+0000","last_peered":"2024-03-28T18:55:47.164685+0000","last_clean":"2024-03-28T18:55:47.164685+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:55:47.164685+0000","last_undegraded":"2024-03-28T18:55:47.164685+0000","last_fullsized":"2024-03-28T18:55:47.164685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603451+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:55:46.603451+0000","last_peered":"2024-03-28T18:55:46.603451+0000","last_clean":"2024-03-28T18:55:46.603451+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:55:46.603451+0000","last_undegraded":"2024-03-28T18:55:46.603451+0000","last_fullsized":"2024-03-28T18:55:46.603451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603782+0000","last_change":"2024-03-28T18:55:44.607744+0000","last_active":"2024-03-28T18:55:46.603782+0000","last_peered":"2024-03-28T18:55:46.603782+0000","last_clean":"2024-03-28T18:55:46.603782+0000","last_became_active":"2024-03-28T18:55:44.606993+0000","last_became_peered":"2024-03-28T18:55:44.606993+0000","last_unstale":"2024-03-28T18:55:46.603782+0000","last_undegraded":"2024-03-28T18:55:46.603782+0000","last_fullsized":"2024-03-28T18:55:46.603782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602563+0000","last_change":"2024-03-28T18:55:45.606307+0000","last_active":"2024-03-28T18:55:46.602563+0000","last_peered":"2024-03-28T18:55:46.602563+0000","last_clean":"2024-03-28T18:55:46.602563+0000","last_became_active":"2024-03-28T18:55:45.605980+0000","last_became_peered":"2024-03-28T18:55:45.605980+0000","last_unstale":"2024-03-28T18:55:46.602563+0000","last_undegraded":"2024-03-28T18:55:46.602563+0000","last_fullsized":"2024-03-28T18:55:46.602563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:52.485966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117600+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:55:47.117600+0000","last_peered":"2024-03-28T18:55:47.117600+0000","last_clean":"2024-03-28T18:55:47.117600+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:55:47.117600+0000","last_undegraded":"2024-03-28T18:55:47.117600+0000","last_fullsized":"2024-03-28T18:55:47.117600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614275+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:55:46.614275+0000","last_peered":"2024-03-28T18:55:46.614275+0000","last_clean":"2024-03-28T18:55:46.614275+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:55:46.614275+0000","last_undegraded":"2024-03-28T18:55:46.614275+0000","last_fullsized":"2024-03-28T18:55:46.614275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608346+0000","last_change":"2024-03-28T18:55:44.607323+0000","last_active":"2024-03-28T18:55:46.608346+0000","last_peered":"2024-03-28T18:55:46.608346+0000","last_clean":"2024-03-28T18:55:46.608346+0000","last_became_active":"2024-03-28T18:55:44.606579+0000","last_became_peered":"2024-03-28T18:55:44.606579+0000","last_unstale":"2024-03-28T18:55:46.608346+0000","last_undegraded":"2024-03-28T18:55:46.608346+0000","last_fullsized":"2024-03-28T18:55:46.608346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163490+0000","last_change":"2024-03-28T18:55:45.622760+0000","last_active":"2024-03-28T18:55:47.163490+0000","last_peered":"2024-03-28T18:55:47.163490+0000","last_clean":"2024-03-28T18:55:47.163490+0000","last_became_active":"2024-03-28T18:55:45.622353+0000","last_became_peered":"2024-03-28T18:55:45.622353+0000","last_unstale":"2024-03-28T18:55:47.163490+0000","last_undegraded":"2024-03-28T18:55:47.163490+0000","last_fullsized":"2024-03-28T18:55:47.163490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:04.349689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166444+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:55:47.166444+0000","last_peered":"2024-03-28T18:55:47.166444+0000","last_clean":"2024-03-28T18:55:47.166444+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:55:47.166444+0000","last_undegraded":"2024-03-28T18:55:47.166444+0000","last_fullsized":"2024-03-28T18:55:47.166444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603346+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:55:46.603346+0000","last_peered":"2024-03-28T18:55:46.603346+0000","last_clean":"2024-03-28T18:55:46.603346+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:55:46.603346+0000","last_undegraded":"2024-03-28T18:55:46.603346+0000","last_fullsized":"2024-03-28T18:55:46.603346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116165+0000","last_change":"2024-03-28T18:55:44.607237+0000","last_active":"2024-03-28T18:55:47.116165+0000","last_peered":"2024-03-28T18:55:47.116165+0000","last_clean":"2024-03-28T18:55:47.116165+0000","last_became_active":"2024-03-28T18:55:44.606960+0000","last_became_peered":"2024-03-28T18:55:44.606960+0000","last_unstale":"2024-03-28T18:55:47.116165+0000","last_undegraded":"2024-03-28T18:55:47.116165+0000","last_fullsized":"2024-03-28T18:55:47.116165+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603242+0000","last_change":"2024-03-28T18:55:45.612992+0000","last_active":"2024-03-28T18:55:46.603242+0000","last_peered":"2024-03-28T18:55:46.603242+0000","last_clean":"2024-03-28T18:55:46.603242+0000","last_became_active":"2024-03-28T18:55:45.612125+0000","last_became_peered":"2024-03-28T18:55:45.612125+0000","last_unstale":"2024-03-28T18:55:46.603242+0000","last_undegraded":"2024-03-28T18:55:46.603242+0000","last_fullsized":"2024-03-28T18:55:46.603242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:35.898911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165571+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:55:47.165571+0000","last_peered":"2024-03-28T18:55:47.165571+0000","last_clean":"2024-03-28T18:55:47.165571+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:55:47.165571+0000","last_undegraded":"2024-03-28T18:55:47.165571+0000","last_fullsized":"2024-03-28T18:55:47.165571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605603+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:55:46.605603+0000","last_peered":"2024-03-28T18:55:46.605603+0000","last_clean":"2024-03-28T18:55:46.605603+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:55:46.605603+0000","last_undegraded":"2024-03-28T18:55:46.605603+0000","last_fullsized":"2024-03-28T18:55:46.605603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604200+0000","last_change":"2024-03-28T18:55:44.607218+0000","last_active":"2024-03-28T18:55:46.604200+0000","last_peered":"2024-03-28T18:55:46.604200+0000","last_clean":"2024-03-28T18:55:46.604200+0000","last_became_active":"2024-03-28T18:55:44.606852+0000","last_became_peered":"2024-03-28T18:55:44.606852+0000","last_unstale":"2024-03-28T18:55:46.604200+0000","last_undegraded":"2024-03-28T18:55:46.604200+0000","last_fullsized":"2024-03-28T18:55:46.604200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:36.624289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829038+0000","last_change":"2024-03-28T18:55:45.618919+0000","last_active":"2024-03-28T18:55:46.829038+0000","last_peered":"2024-03-28T18:55:46.829038+0000","last_clean":"2024-03-28T18:55:46.829038+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:55:46.829038+0000","last_undegraded":"2024-03-28T18:55:46.829038+0000","last_fullsized":"2024-03-28T18:55:46.829038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:29.407508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164081+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:55:47.164081+0000","last_peered":"2024-03-28T18:55:47.164081+0000","last_clean":"2024-03-28T18:55:47.164081+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:55:47.164081+0000","last_undegraded":"2024-03-28T18:55:47.164081+0000","last_fullsized":"2024-03-28T18:55:47.164081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602411+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:55:46.602411+0000","last_peered":"2024-03-28T18:55:46.602411+0000","last_clean":"2024-03-28T18:55:46.602411+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:55:46.602411+0000","last_undegraded":"2024-03-28T18:55:46.602411+0000","last_fullsized":"2024-03-28T18:55:46.602411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609206+0000","last_change":"2024-03-28T18:55:44.598881+0000","last_active":"2024-03-28T18:55:46.609206+0000","last_peered":"2024-03-28T18:55:46.609206+0000","last_clean":"2024-03-28T18:55:46.609206+0000","last_became_active":"2024-03-28T18:55:44.598532+0000","last_became_peered":"2024-03-28T18:55:44.598532+0000","last_unstale":"2024-03-28T18:55:46.609206+0000","last_undegraded":"2024-03-28T18:55:46.609206+0000","last_fullsized":"2024-03-28T18:55:46.609206+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:44.004611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117993+0000","last_change":"2024-03-28T18:55:45.612572+0000","last_active":"2024-03-28T18:55:47.117993+0000","last_peered":"2024-03-28T18:55:47.117993+0000","last_clean":"2024-03-28T18:55:47.117993+0000","last_became_active":"2024-03-28T18:55:45.611844+0000","last_became_peered":"2024-03-28T18:55:45.611844+0000","last_unstale":"2024-03-28T18:55:47.117993+0000","last_undegraded":"2024-03-28T18:55:47.117993+0000","last_fullsized":"2024-03-28T18:55:47.117993+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602718+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:55:46.602718+0000","last_peered":"2024-03-28T18:55:46.602718+0000","last_clean":"2024-03-28T18:55:46.602718+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:55:46.602718+0000","last_undegraded":"2024-03-28T18:55:46.602718+0000","last_fullsized":"2024-03-28T18:55:46.602718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603429+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:55:46.603429+0000","last_peered":"2024-03-28T18:55:46.603429+0000","last_clean":"2024-03-28T18:55:46.603429+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:55:46.603429+0000","last_undegraded":"2024-03-28T18:55:46.603429+0000","last_fullsized":"2024-03-28T18:55:46.603429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829075+0000","last_change":"2024-03-28T18:55:44.609644+0000","last_active":"2024-03-28T18:55:46.829075+0000","last_peered":"2024-03-28T18:55:46.829075+0000","last_clean":"2024-03-28T18:55:46.829075+0000","last_became_active":"2024-03-28T18:55:44.609317+0000","last_became_peered":"2024-03-28T18:55:44.609317+0000","last_unstale":"2024-03-28T18:55:46.829075+0000","last_undegraded":"2024-03-28T18:55:46.829075+0000","last_fullsized":"2024-03-28T18:55:46.829075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:37.662504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116808+0000","last_change":"2024-03-28T18:55:45.621708+0000","last_active":"2024-03-28T18:55:47.116808+0000","last_peered":"2024-03-28T18:55:47.116808+0000","last_clean":"2024-03-28T18:55:47.116808+0000","last_became_active":"2024-03-28T18:55:45.621366+0000","last_became_peered":"2024-03-28T18:55:45.621366+0000","last_unstale":"2024-03-28T18:55:47.116808+0000","last_undegraded":"2024-03-28T18:55:47.116808+0000","last_fullsized":"2024-03-28T18:55:47.116808+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164622+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:55:47.164622+0000","last_peered":"2024-03-28T18:55:47.164622+0000","last_clean":"2024-03-28T18:55:47.164622+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:55:47.164622+0000","last_undegraded":"2024-03-28T18:55:47.164622+0000","last_fullsized":"2024-03-28T18:55:47.164622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116814+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:55:47.116814+0000","last_peered":"2024-03-28T18:55:47.116814+0000","last_clean":"2024-03-28T18:55:47.116814+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:55:47.116814+0000","last_undegraded":"2024-03-28T18:55:47.116814+0000","last_fullsized":"2024-03-28T18:55:47.116814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"23'6","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827637+0000","last_change":"2024-03-28T18:55:44.608949+0000","last_active":"2024-03-28T18:55:46.827637+0000","last_peered":"2024-03-28T18:55:46.827637+0000","last_clean":"2024-03-28T18:55:46.827637+0000","last_became_active":"2024-03-28T18:55:44.608657+0000","last_became_peered":"2024-03-28T18:55:44.608657+0000","last_unstale":"2024-03-28T18:55:46.827637+0000","last_undegraded":"2024-03-28T18:55:46.827637+0000","last_fullsized":"2024-03-28T18:55:46.827637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:04.799314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827662+0000","last_change":"2024-03-28T18:55:45.610686+0000","last_active":"2024-03-28T18:55:46.827662+0000","last_peered":"2024-03-28T18:55:46.827662+0000","last_clean":"2024-03-28T18:55:46.827662+0000","last_became_active":"2024-03-28T18:55:45.610212+0000","last_became_peered":"2024-03-28T18:55:45.610212+0000","last_unstale":"2024-03-28T18:55:46.827662+0000","last_undegraded":"2024-03-28T18:55:46.827662+0000","last_fullsized":"2024-03-28T18:55:46.827662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608274+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:55:46.608274+0000","last_peered":"2024-03-28T18:55:46.608274+0000","last_clean":"2024-03-28T18:55:46.608274+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:55:46.608274+0000","last_undegraded":"2024-03-28T18:55:46.608274+0000","last_fullsized":"2024-03-28T18:55:46.608274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609087+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:55:46.609087+0000","last_peered":"2024-03-28T18:55:46.609087+0000","last_clean":"2024-03-28T18:55:46.609087+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:55:46.609087+0000","last_undegraded":"2024-03-28T18:55:46.609087+0000","last_fullsized":"2024-03-28T18:55:46.609087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"23'3","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117447+0000","last_change":"2024-03-28T18:55:44.602762+0000","last_active":"2024-03-28T18:55:47.117447+0000","last_peered":"2024-03-28T18:55:47.117447+0000","last_clean":"2024-03-28T18:55:47.117447+0000","last_became_active":"2024-03-28T18:55:44.602424+0000","last_became_peered":"2024-03-28T18:55:44.602424+0000","last_unstale":"2024-03-28T18:55:47.117447+0000","last_undegraded":"2024-03-28T18:55:47.117447+0000","last_fullsized":"2024-03-28T18:55:47.117447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826897+0000","last_change":"2024-03-28T18:55:45.601277+0000","last_active":"2024-03-28T18:55:46.826897+0000","last_peered":"2024-03-28T18:55:46.826897+0000","last_clean":"2024-03-28T18:55:46.826897+0000","last_became_active":"2024-03-28T18:55:45.600945+0000","last_became_peered":"2024-03-28T18:55:45.600945+0000","last_unstale":"2024-03-28T18:55:46.826897+0000","last_undegraded":"2024-03-28T18:55:46.826897+0000","last_fullsized":"2024-03-28T18:55:46.826897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164660+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:55:47.164660+0000","last_peered":"2024-03-28T18:55:47.164660+0000","last_clean":"2024-03-28T18:55:47.164660+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:55:47.164660+0000","last_undegraded":"2024-03-28T18:55:47.164660+0000","last_fullsized":"2024-03-28T18:55:47.164660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602857+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:55:46.602857+0000","last_peered":"2024-03-28T18:55:46.602857+0000","last_clean":"2024-03-28T18:55:46.602857+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:55:46.602857+0000","last_undegraded":"2024-03-28T18:55:46.602857+0000","last_fullsized":"2024-03-28T18:55:46.602857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612960+0000","last_change":"2024-03-28T18:55:44.599432+0000","last_active":"2024-03-28T18:55:46.612960+0000","last_peered":"2024-03-28T18:55:46.612960+0000","last_clean":"2024-03-28T18:55:46.612960+0000","last_became_active":"2024-03-28T18:55:44.599123+0000","last_became_peered":"2024-03-28T18:55:44.599123+0000","last_unstale":"2024-03-28T18:55:46.612960+0000","last_undegraded":"2024-03-28T18:55:46.612960+0000","last_fullsized":"2024-03-28T18:55:46.612960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613285+0000","last_change":"2024-03-28T18:55:45.624311+0000","last_active":"2024-03-28T18:55:46.613285+0000","last_peered":"2024-03-28T18:55:46.613285+0000","last_clean":"2024-03-28T18:55:46.613285+0000","last_became_active":"2024-03-28T18:55:45.624031+0000","last_became_peered":"2024-03-28T18:55:45.624031+0000","last_unstale":"2024-03-28T18:55:46.613285+0000","last_undegraded":"2024-03-28T18:55:46.613285+0000","last_fullsized":"2024-03-28T18:55:46.613285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605962+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:55:46.605962+0000","last_peered":"2024-03-28T18:55:46.605962+0000","last_clean":"2024-03-28T18:55:46.605962+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:55:46.605962+0000","last_undegraded":"2024-03-28T18:55:46.605962+0000","last_fullsized":"2024-03-28T18:55:46.605962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614210+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:55:46.614210+0000","last_peered":"2024-03-28T18:55:46.614210+0000","last_clean":"2024-03-28T18:55:46.614210+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:55:46.614210+0000","last_undegraded":"2024-03-28T18:55:46.614210+0000","last_fullsized":"2024-03-28T18:55:46.614210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604974+0000","last_change":"2024-03-28T18:55:44.593567+0000","last_active":"2024-03-28T18:55:46.604974+0000","last_peered":"2024-03-28T18:55:46.604974+0000","last_clean":"2024-03-28T18:55:46.604974+0000","last_became_active":"2024-03-28T18:55:44.593267+0000","last_became_peered":"2024-03-28T18:55:44.593267+0000","last_unstale":"2024-03-28T18:55:46.604974+0000","last_undegraded":"2024-03-28T18:55:46.604974+0000","last_fullsized":"2024-03-28T18:55:46.604974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603331+0000","last_change":"2024-03-28T18:55:45.598713+0000","last_active":"2024-03-28T18:55:46.603331+0000","last_peered":"2024-03-28T18:55:46.603331+0000","last_clean":"2024-03-28T18:55:46.603331+0000","last_became_active":"2024-03-28T18:55:45.597726+0000","last_became_peered":"2024-03-28T18:55:45.597726+0000","last_unstale":"2024-03-28T18:55:46.603331+0000","last_undegraded":"2024-03-28T18:55:46.603331+0000","last_fullsized":"2024-03-28T18:55:46.603331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:48.255745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609101+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:55:46.609101+0000","last_peered":"2024-03-28T18:55:46.609101+0000","last_clean":"2024-03-28T18:55:46.609101+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:55:46.609101+0000","last_undegraded":"2024-03-28T18:55:46.609101+0000","last_fullsized":"2024-03-28T18:55:46.609101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118062+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:55:47.118062+0000","last_peered":"2024-03-28T18:55:47.118062+0000","last_clean":"2024-03-28T18:55:47.118062+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:55:47.118062+0000","last_undegraded":"2024-03-28T18:55:47.118062+0000","last_fullsized":"2024-03-28T18:55:47.118062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"23'8","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826291+0000","last_change":"2024-03-28T18:55:44.610673+0000","last_active":"2024-03-28T18:55:46.826291+0000","last_peered":"2024-03-28T18:55:46.826291+0000","last_clean":"2024-03-28T18:55:46.826291+0000","last_became_active":"2024-03-28T18:55:44.610357+0000","last_became_peered":"2024-03-28T18:55:44.610357+0000","last_unstale":"2024-03-28T18:55:46.826291+0000","last_undegraded":"2024-03-28T18:55:46.826291+0000","last_fullsized":"2024-03-28T18:55:46.826291+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:26.280548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163667+0000","last_change":"2024-03-28T18:55:45.609349+0000","last_active":"2024-03-28T18:55:47.163667+0000","last_peered":"2024-03-28T18:55:47.163667+0000","last_clean":"2024-03-28T18:55:47.163667+0000","last_became_active":"2024-03-28T18:55:45.608743+0000","last_became_peered":"2024-03-28T18:55:45.608743+0000","last_unstale":"2024-03-28T18:55:47.163667+0000","last_undegraded":"2024-03-28T18:55:47.163667+0000","last_fullsized":"2024-03-28T18:55:47.163667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118036+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:55:47.118036+0000","last_peered":"2024-03-28T18:55:47.118036+0000","last_clean":"2024-03-28T18:55:47.118036+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:55:47.118036+0000","last_undegraded":"2024-03-28T18:55:47.118036+0000","last_fullsized":"2024-03-28T18:55:47.118036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604933+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:55:46.604933+0000","last_peered":"2024-03-28T18:55:46.604933+0000","last_clean":"2024-03-28T18:55:46.604933+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:55:46.604933+0000","last_undegraded":"2024-03-28T18:55:46.604933+0000","last_fullsized":"2024-03-28T18:55:46.604933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825952+0000","last_change":"2024-03-28T18:55:44.601103+0000","last_active":"2024-03-28T18:55:46.825952+0000","last_peered":"2024-03-28T18:55:46.825952+0000","last_clean":"2024-03-28T18:55:46.825952+0000","last_became_active":"2024-03-28T18:55:44.599969+0000","last_became_peered":"2024-03-28T18:55:44.599969+0000","last_unstale":"2024-03-28T18:55:46.825952+0000","last_undegraded":"2024-03-28T18:55:46.825952+0000","last_fullsized":"2024-03-28T18:55:46.825952+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:42.018654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604371+0000","last_change":"2024-03-28T18:55:45.594944+0000","last_active":"2024-03-28T18:55:46.604371+0000","last_peered":"2024-03-28T18:55:46.604371+0000","last_clean":"2024-03-28T18:55:46.604371+0000","last_became_active":"2024-03-28T18:55:45.594603+0000","last_became_peered":"2024-03-28T18:55:45.594603+0000","last_unstale":"2024-03-28T18:55:46.604371+0000","last_undegraded":"2024-03-28T18:55:46.604371+0000","last_fullsized":"2024-03-28T18:55:46.604371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608967+0000","last_change":"2024-03-28T18:55:44.602366+0000","last_active":"2024-03-28T18:55:46.608967+0000","last_peered":"2024-03-28T18:55:46.608967+0000","last_clean":"2024-03-28T18:55:46.608967+0000","last_became_active":"2024-03-28T18:55:44.602033+0000","last_became_peered":"2024-03-28T18:55:44.602033+0000","last_unstale":"2024-03-28T18:55:46.608967+0000","last_undegraded":"2024-03-28T18:55:46.608967+0000","last_fullsized":"2024-03-28T18:55:46.608967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612862+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:55:46.612862+0000","last_peered":"2024-03-28T18:55:46.612862+0000","last_clean":"2024-03-28T18:55:46.612862+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:55:46.612862+0000","last_undegraded":"2024-03-28T18:55:46.612862+0000","last_fullsized":"2024-03-28T18:55:46.612862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117107+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:55:47.117107+0000","last_peered":"2024-03-28T18:55:47.117107+0000","last_clean":"2024-03-28T18:55:47.117107+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:55:47.117107+0000","last_undegraded":"2024-03-28T18:55:47.117107+0000","last_fullsized":"2024-03-28T18:55:47.117107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608811+0000","last_change":"2024-03-28T18:55:45.620559+0000","last_active":"2024-03-28T18:55:46.608811+0000","last_peered":"2024-03-28T18:55:46.608811+0000","last_clean":"2024-03-28T18:55:46.608811+0000","last_became_active":"2024-03-28T18:55:45.620249+0000","last_became_peered":"2024-03-28T18:55:45.620249+0000","last_unstale":"2024-03-28T18:55:46.608811+0000","last_undegraded":"2024-03-28T18:55:46.608811+0000","last_fullsized":"2024-03-28T18:55:46.608811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"23'4","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604650+0000","last_change":"2024-03-28T18:55:44.612226+0000","last_active":"2024-03-28T18:55:46.604650+0000","last_peered":"2024-03-28T18:55:46.604650+0000","last_clean":"2024-03-28T18:55:46.604650+0000","last_became_active":"2024-03-28T18:55:44.611921+0000","last_became_peered":"2024-03-28T18:55:44.611921+0000","last_unstale":"2024-03-28T18:55:46.604650+0000","last_undegraded":"2024-03-28T18:55:46.604650+0000","last_fullsized":"2024-03-28T18:55:46.604650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826474+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:55:46.826474+0000","last_peered":"2024-03-28T18:55:46.826474+0000","last_clean":"2024-03-28T18:55:46.826474+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:55:46.826474+0000","last_undegraded":"2024-03-28T18:55:46.826474+0000","last_fullsized":"2024-03-28T18:55:46.826474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826438+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:55:46.826438+0000","last_peered":"2024-03-28T18:55:46.826438+0000","last_clean":"2024-03-28T18:55:46.826438+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:55:46.826438+0000","last_undegraded":"2024-03-28T18:55:46.826438+0000","last_fullsized":"2024-03-28T18:55:46.826438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614060+0000","last_change":"2024-03-28T18:55:45.612727+0000","last_active":"2024-03-28T18:55:46.614060+0000","last_peered":"2024-03-28T18:55:46.614060+0000","last_clean":"2024-03-28T18:55:46.614060+0000","last_became_active":"2024-03-28T18:55:45.610154+0000","last_became_peered":"2024-03-28T18:55:45.610154+0000","last_unstale":"2024-03-28T18:55:46.614060+0000","last_undegraded":"2024-03-28T18:55:46.614060+0000","last_fullsized":"2024-03-28T18:55:46.614060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604120+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:55:46.604120+0000","last_peered":"2024-03-28T18:55:46.604120+0000","last_clean":"2024-03-28T18:55:46.604120+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:55:46.604120+0000","last_undegraded":"2024-03-28T18:55:46.604120+0000","last_fullsized":"2024-03-28T18:55:46.604120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608921+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:55:46.608921+0000","last_peered":"2024-03-28T18:55:46.608921+0000","last_clean":"2024-03-28T18:55:46.608921+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:55:46.608921+0000","last_undegraded":"2024-03-28T18:55:46.608921+0000","last_fullsized":"2024-03-28T18:55:46.608921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603331+0000","last_change":"2024-03-28T18:55:44.592835+0000","last_active":"2024-03-28T18:55:46.603331+0000","last_peered":"2024-03-28T18:55:46.603331+0000","last_clean":"2024-03-28T18:55:46.603331+0000","last_became_active":"2024-03-28T18:55:44.592505+0000","last_became_peered":"2024-03-28T18:55:44.592505+0000","last_unstale":"2024-03-28T18:55:46.603331+0000","last_undegraded":"2024-03-28T18:55:46.603331+0000","last_fullsized":"2024-03-28T18:55:46.603331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:58.286421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605269+0000","last_change":"2024-03-28T18:55:45.611322+0000","last_active":"2024-03-28T18:55:46.605269+0000","last_peered":"2024-03-28T18:55:46.605269+0000","last_clean":"2024-03-28T18:55:46.605269+0000","last_became_active":"2024-03-28T18:55:45.610958+0000","last_became_peered":"2024-03-28T18:55:45.610958+0000","last_unstale":"2024-03-28T18:55:46.605269+0000","last_undegraded":"2024-03-28T18:55:46.605269+0000","last_fullsized":"2024-03-28T18:55:46.605269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827108+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:55:46.827108+0000","last_peered":"2024-03-28T18:55:46.827108+0000","last_clean":"2024-03-28T18:55:46.827108+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:55:46.827108+0000","last_undegraded":"2024-03-28T18:55:46.827108+0000","last_fullsized":"2024-03-28T18:55:46.827108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604724+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:55:46.604724+0000","last_peered":"2024-03-28T18:55:46.604724+0000","last_clean":"2024-03-28T18:55:46.604724+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:55:46.604724+0000","last_undegraded":"2024-03-28T18:55:46.604724+0000","last_fullsized":"2024-03-28T18:55:46.604724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827611+0000","last_change":"2024-03-28T18:55:44.608283+0000","last_active":"2024-03-28T18:55:46.827611+0000","last_peered":"2024-03-28T18:55:46.827611+0000","last_clean":"2024-03-28T18:55:46.827611+0000","last_became_active":"2024-03-28T18:55:44.607999+0000","last_became_peered":"2024-03-28T18:55:44.607999+0000","last_unstale":"2024-03-28T18:55:46.827611+0000","last_undegraded":"2024-03-28T18:55:46.827611+0000","last_fullsized":"2024-03-28T18:55:46.827611+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:22.669778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828611+0000","last_change":"2024-03-28T18:55:45.618390+0000","last_active":"2024-03-28T18:55:46.828611+0000","last_peered":"2024-03-28T18:55:46.828611+0000","last_clean":"2024-03-28T18:55:46.828611+0000","last_became_active":"2024-03-28T18:55:45.617897+0000","last_became_peered":"2024-03-28T18:55:45.617897+0000","last_unstale":"2024-03-28T18:55:46.828611+0000","last_undegraded":"2024-03-28T18:55:46.828611+0000","last_fullsized":"2024-03-28T18:55:46.828611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:16.520575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602446+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:55:46.602446+0000","last_peered":"2024-03-28T18:55:46.602446+0000","last_clean":"2024-03-28T18:55:46.602446+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:55:46.602446+0000","last_undegraded":"2024-03-28T18:55:46.602446+0000","last_fullsized":"2024-03-28T18:55:46.602446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613831+0000","last_change":"2024-03-28T18:55:45.624938+0000","last_active":"2024-03-28T18:55:46.613831+0000","last_peered":"2024-03-28T18:55:46.613831+0000","last_clean":"2024-03-28T18:55:46.613831+0000","last_became_active":"2024-03-28T18:55:45.624362+0000","last_became_peered":"2024-03-28T18:55:45.624362+0000","last_unstale":"2024-03-28T18:55:46.613831+0000","last_undegraded":"2024-03-28T18:55:46.613831+0000","last_fullsized":"2024-03-28T18:55:46.613831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602638+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:55:46.602638+0000","last_peered":"2024-03-28T18:55:46.602638+0000","last_clean":"2024-03-28T18:55:46.602638+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:55:46.602638+0000","last_undegraded":"2024-03-28T18:55:46.602638+0000","last_fullsized":"2024-03-28T18:55:46.602638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116612+0000","last_change":"2024-03-28T18:55:45.610168+0000","last_active":"2024-03-28T18:55:47.116612+0000","last_peered":"2024-03-28T18:55:47.116612+0000","last_clean":"2024-03-28T18:55:47.116612+0000","last_became_active":"2024-03-28T18:55:45.609653+0000","last_became_peered":"2024-03-28T18:55:45.609653+0000","last_unstale":"2024-03-28T18:55:47.116612+0000","last_undegraded":"2024-03-28T18:55:47.116612+0000","last_fullsized":"2024-03-28T18:55:47.116612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116681+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:55:47.116681+0000","last_peered":"2024-03-28T18:55:47.116681+0000","last_clean":"2024-03-28T18:55:47.116681+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:55:47.116681+0000","last_undegraded":"2024-03-28T18:55:47.116681+0000","last_fullsized":"2024-03-28T18:55:47.116681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613952+0000","last_change":"2024-03-28T18:55:45.613701+0000","last_active":"2024-03-28T18:55:46.613952+0000","last_peered":"2024-03-28T18:55:46.613952+0000","last_clean":"2024-03-28T18:55:46.613952+0000","last_became_active":"2024-03-28T18:55:45.613310+0000","last_became_peered":"2024-03-28T18:55:45.613310+0000","last_unstale":"2024-03-28T18:55:46.613952+0000","last_undegraded":"2024-03-28T18:55:46.613952+0000","last_fullsized":"2024-03-28T18:55:46.613952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608809+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:55:46.608809+0000","last_peered":"2024-03-28T18:55:46.608809+0000","last_clean":"2024-03-28T18:55:46.608809+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:55:46.608809+0000","last_undegraded":"2024-03-28T18:55:46.608809+0000","last_fullsized":"2024-03-28T18:55:46.608809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163732+0000","last_change":"2024-03-28T18:55:45.615119+0000","last_active":"2024-03-28T18:55:47.163732+0000","last_peered":"2024-03-28T18:55:47.163732+0000","last_clean":"2024-03-28T18:55:47.163732+0000","last_became_active":"2024-03-28T18:55:45.613841+0000","last_became_peered":"2024-03-28T18:55:45.613841+0000","last_unstale":"2024-03-28T18:55:47.163732+0000","last_undegraded":"2024-03-28T18:55:47.163732+0000","last_fullsized":"2024-03-28T18:55:47.163732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827969+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:55:46.827969+0000","last_peered":"2024-03-28T18:55:46.827969+0000","last_clean":"2024-03-28T18:55:46.827969+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:55:46.827969+0000","last_undegraded":"2024-03-28T18:55:46.827969+0000","last_fullsized":"2024-03-28T18:55:46.827969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604609+0000","last_change":"2024-03-28T18:55:45.619605+0000","last_active":"2024-03-28T18:55:46.604609+0000","last_peered":"2024-03-28T18:55:46.604609+0000","last_clean":"2024-03-28T18:55:46.604609+0000","last_became_active":"2024-03-28T18:55:45.618473+0000","last_became_peered":"2024-03-28T18:55:45.618473+0000","last_unstale":"2024-03-28T18:55:46.604609+0000","last_undegraded":"2024-03-28T18:55:46.604609+0000","last_fullsized":"2024-03-28T18:55:46.604609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614151+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:55:46.614151+0000","last_peered":"2024-03-28T18:55:46.614151+0000","last_clean":"2024-03-28T18:55:46.614151+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:55:46.614151+0000","last_undegraded":"2024-03-28T18:55:46.614151+0000","last_fullsized":"2024-03-28T18:55:46.614151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163424+0000","last_change":"2024-03-28T18:55:45.612284+0000","last_active":"2024-03-28T18:55:47.163424+0000","last_peered":"2024-03-28T18:55:47.163424+0000","last_clean":"2024-03-28T18:55:47.163424+0000","last_became_active":"2024-03-28T18:55:45.607218+0000","last_became_peered":"2024-03-28T18:55:45.607218+0000","last_unstale":"2024-03-28T18:55:47.163424+0000","last_undegraded":"2024-03-28T18:55:47.163424+0000","last_fullsized":"2024-03-28T18:55:47.163424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604943+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:55:46.604943+0000","last_peered":"2024-03-28T18:55:46.604943+0000","last_clean":"2024-03-28T18:55:46.604943+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:55:46.604943+0000","last_undegraded":"2024-03-28T18:55:46.604943+0000","last_fullsized":"2024-03-28T18:55:46.604943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603065+0000","last_change":"2024-03-28T18:55:45.607609+0000","last_active":"2024-03-28T18:55:46.603065+0000","last_peered":"2024-03-28T18:55:46.603065+0000","last_clean":"2024-03-28T18:55:46.603065+0000","last_became_active":"2024-03-28T18:55:45.602582+0000","last_became_peered":"2024-03-28T18:55:45.602582+0000","last_unstale":"2024-03-28T18:55:46.603065+0000","last_undegraded":"2024-03-28T18:55:46.603065+0000","last_fullsized":"2024-03-28T18:55:46.603065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:06:05.335715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829082+0000","last_change":"2024-03-28T18:55:34.491857+0000","last_active":"2024-03-28T18:55:46.829082+0000","last_peered":"2024-03-28T18:55:46.829082+0000","last_clean":"2024-03-28T18:55:46.829082+0000","last_became_active":"2024-03-28T18:55:34.487194+0000","last_became_peered":"2024-03-28T18:55:34.487194+0000","last_unstale":"2024-03-28T18:55:46.829082+0000","last_undegraded":"2024-03-28T18:55:46.829082+0000","last_fullsized":"2024-03-28T18:55:46.829082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828429+0000","last_change":"2024-03-28T18:55:45.607886+0000","last_active":"2024-03-28T18:55:46.828429+0000","last_peered":"2024-03-28T18:55:46.828429+0000","last_clean":"2024-03-28T18:55:46.828429+0000","last_became_active":"2024-03-28T18:55:45.605964+0000","last_became_peered":"2024-03-28T18:55:45.605964+0000","last_unstale":"2024-03-28T18:55:46.828429+0000","last_undegraded":"2024-03-28T18:55:46.828429+0000","last_fullsized":"2024-03-28T18:55:46.828429+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:45.986654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117089+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:55:47.117089+0000","last_peered":"2024-03-28T18:55:47.117089+0000","last_clean":"2024-03-28T18:55:47.117089+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:55:47.117089+0000","last_undegraded":"2024-03-28T18:55:47.117089+0000","last_fullsized":"2024-03-28T18:55:47.117089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165971+0000","last_change":"2024-03-28T18:55:45.618031+0000","last_active":"2024-03-28T18:55:47.165971+0000","last_peered":"2024-03-28T18:55:47.165971+0000","last_clean":"2024-03-28T18:55:47.165971+0000","last_became_active":"2024-03-28T18:55:45.617719+0000","last_became_peered":"2024-03-28T18:55:45.617719+0000","last_unstale":"2024-03-28T18:55:47.165971+0000","last_undegraded":"2024-03-28T18:55:47.165971+0000","last_fullsized":"2024-03-28T18:55:47.165971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826204+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:55:46.826204+0000","last_peered":"2024-03-28T18:55:46.826204+0000","last_clean":"2024-03-28T18:55:46.826204+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:55:46.826204+0000","last_undegraded":"2024-03-28T18:55:46.826204+0000","last_fullsized":"2024-03-28T18:55:46.826204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603156+0000","last_change":"2024-03-28T18:55:45.603783+0000","last_active":"2024-03-28T18:55:46.603156+0000","last_peered":"2024-03-28T18:55:46.603156+0000","last_clean":"2024-03-28T18:55:46.603156+0000","last_became_active":"2024-03-28T18:55:45.603474+0000","last_became_peered":"2024-03-28T18:55:45.603474+0000","last_unstale":"2024-03-28T18:55:46.603156+0000","last_undegraded":"2024-03-28T18:55:46.603156+0000","last_fullsized":"2024-03-28T18:55:46.603156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:42.328182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116551+0000","last_change":"2024-03-28T18:55:34.505741+0000","last_active":"2024-03-28T18:55:47.116551+0000","last_peered":"2024-03-28T18:55:47.116551+0000","last_clean":"2024-03-28T18:55:47.116551+0000","last_became_active":"2024-03-28T18:55:34.505440+0000","last_became_peered":"2024-03-28T18:55:34.505440+0000","last_unstale":"2024-03-28T18:55:47.116551+0000","last_undegraded":"2024-03-28T18:55:47.116551+0000","last_fullsized":"2024-03-28T18:55:47.116551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827452+0000","last_change":"2024-03-28T18:55:45.618912+0000","last_active":"2024-03-28T18:55:46.827452+0000","last_peered":"2024-03-28T18:55:46.827452+0000","last_clean":"2024-03-28T18:55:46.827452+0000","last_became_active":"2024-03-28T18:55:45.618554+0000","last_became_peered":"2024-03-28T18:55:45.618554+0000","last_unstale":"2024-03-28T18:55:46.827452+0000","last_undegraded":"2024-03-28T18:55:46.827452+0000","last_fullsized":"2024-03-28T18:55:46.827452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608624+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:55:46.608624+0000","last_peered":"2024-03-28T18:55:46.608624+0000","last_clean":"2024-03-28T18:55:46.608624+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:55:46.608624+0000","last_undegraded":"2024-03-28T18:55:46.608624+0000","last_fullsized":"2024-03-28T18:55:46.608624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604333+0000","last_change":"2024-03-28T18:55:45.616420+0000","last_active":"2024-03-28T18:55:46.604333+0000","last_peered":"2024-03-28T18:55:46.604333+0000","last_clean":"2024-03-28T18:55:46.604333+0000","last_became_active":"2024-03-28T18:55:45.614806+0000","last_became_peered":"2024-03-28T18:55:45.614806+0000","last_unstale":"2024-03-28T18:55:46.604333+0000","last_undegraded":"2024-03-28T18:55:46.604333+0000","last_fullsized":"2024-03-28T18:55:46.604333+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602688+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:55:46.602688+0000","last_peered":"2024-03-28T18:55:46.602688+0000","last_clean":"2024-03-28T18:55:46.602688+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:55:46.602688+0000","last_undegraded":"2024-03-28T18:55:46.602688+0000","last_fullsized":"2024-03-28T18:55:46.602688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604338+0000","last_change":"2024-03-28T18:55:45.606507+0000","last_active":"2024-03-28T18:55:46.604338+0000","last_peered":"2024-03-28T18:55:46.604338+0000","last_clean":"2024-03-28T18:55:46.604338+0000","last_became_active":"2024-03-28T18:55:45.606205+0000","last_became_peered":"2024-03-28T18:55:45.606205+0000","last_unstale":"2024-03-28T18:55:46.604338+0000","last_undegraded":"2024-03-28T18:55:46.604338+0000","last_fullsized":"2024-03-28T18:55:46.604338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164913+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:55:47.164913+0000","last_peered":"2024-03-28T18:55:47.164913+0000","last_clean":"2024-03-28T18:55:47.164913+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:55:47.164913+0000","last_undegraded":"2024-03-28T18:55:47.164913+0000","last_fullsized":"2024-03-28T18:55:47.164913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116033+0000","last_change":"2024-03-28T18:55:45.607890+0000","last_active":"2024-03-28T18:55:47.116033+0000","last_peered":"2024-03-28T18:55:47.116033+0000","last_clean":"2024-03-28T18:55:47.116033+0000","last_became_active":"2024-03-28T18:55:45.607590+0000","last_became_peered":"2024-03-28T18:55:45.607590+0000","last_unstale":"2024-03-28T18:55:47.116033+0000","last_undegraded":"2024-03-28T18:55:47.116033+0000","last_fullsized":"2024-03-28T18:55:47.116033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603209+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:55:46.603209+0000","last_peered":"2024-03-28T18:55:46.603209+0000","last_clean":"2024-03-28T18:55:46.603209+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:55:46.603209+0000","last_undegraded":"2024-03-28T18:55:46.603209+0000","last_fullsized":"2024-03-28T18:55:46.603209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608156+0000","last_change":"2024-03-28T18:55:45.608851+0000","last_active":"2024-03-28T18:55:46.608156+0000","last_peered":"2024-03-28T18:55:46.608156+0000","last_clean":"2024-03-28T18:55:46.608156+0000","last_became_active":"2024-03-28T18:55:45.608496+0000","last_became_peered":"2024-03-28T18:55:45.608496+0000","last_unstale":"2024-03-28T18:55:46.608156+0000","last_undegraded":"2024-03-28T18:55:46.608156+0000","last_fullsized":"2024-03-28T18:55:46.608156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118072+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:55:47.118072+0000","last_peered":"2024-03-28T18:55:47.118072+0000","last_clean":"2024-03-28T18:55:47.118072+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:55:47.118072+0000","last_undegraded":"2024-03-28T18:55:47.118072+0000","last_fullsized":"2024-03-28T18:55:47.118072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613556+0000","last_change":"2024-03-28T18:55:45.619501+0000","last_active":"2024-03-28T18:55:46.613556+0000","last_peered":"2024-03-28T18:55:46.613556+0000","last_clean":"2024-03-28T18:55:46.613556+0000","last_became_active":"2024-03-28T18:55:45.619212+0000","last_became_peered":"2024-03-28T18:55:45.619212+0000","last_unstale":"2024-03-28T18:55:46.613556+0000","last_undegraded":"2024-03-28T18:55:46.613556+0000","last_fullsized":"2024-03-28T18:55:46.613556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116821+0000","last_change":"2024-03-28T18:55:34.509529+0000","last_active":"2024-03-28T18:55:47.116821+0000","last_peered":"2024-03-28T18:55:47.116821+0000","last_clean":"2024-03-28T18:55:47.116821+0000","last_became_active":"2024-03-28T18:55:34.501100+0000","last_became_peered":"2024-03-28T18:55:34.501100+0000","last_unstale":"2024-03-28T18:55:47.116821+0000","last_undegraded":"2024-03-28T18:55:47.116821+0000","last_fullsized":"2024-03-28T18:55:47.116821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605306+0000","last_change":"2024-03-28T18:55:45.615225+0000","last_active":"2024-03-28T18:55:46.605306+0000","last_peered":"2024-03-28T18:55:46.605306+0000","last_clean":"2024-03-28T18:55:46.605306+0000","last_became_active":"2024-03-28T18:55:45.613602+0000","last_became_peered":"2024-03-28T18:55:45.613602+0000","last_unstale":"2024-03-28T18:55:46.605306+0000","last_undegraded":"2024-03-28T18:55:46.605306+0000","last_fullsized":"2024-03-28T18:55:46.605306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612997+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:55:46.612997+0000","last_peered":"2024-03-28T18:55:46.612997+0000","last_clean":"2024-03-28T18:55:46.612997+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:55:46.612997+0000","last_undegraded":"2024-03-28T18:55:46.612997+0000","last_fullsized":"2024-03-28T18:55:46.612997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116582+0000","last_change":"2024-03-28T18:55:45.609471+0000","last_active":"2024-03-28T18:55:47.116582+0000","last_peered":"2024-03-28T18:55:47.116582+0000","last_clean":"2024-03-28T18:55:47.116582+0000","last_became_active":"2024-03-28T18:55:45.609132+0000","last_became_peered":"2024-03-28T18:55:45.609132+0000","last_unstale":"2024-03-28T18:55:47.116582+0000","last_undegraded":"2024-03-28T18:55:47.116582+0000","last_fullsized":"2024-03-28T18:55:47.116582+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608272+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:55:46.608272+0000","last_peered":"2024-03-28T18:55:46.608272+0000","last_clean":"2024-03-28T18:55:46.608272+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:55:46.608272+0000","last_undegraded":"2024-03-28T18:55:46.608272+0000","last_fullsized":"2024-03-28T18:55:46.608272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164529+0000","last_change":"2024-03-28T18:55:45.617501+0000","last_active":"2024-03-28T18:55:47.164529+0000","last_peered":"2024-03-28T18:55:47.164529+0000","last_clean":"2024-03-28T18:55:47.164529+0000","last_became_active":"2024-03-28T18:55:45.617180+0000","last_became_peered":"2024-03-28T18:55:45.617180+0000","last_unstale":"2024-03-28T18:55:47.164529+0000","last_undegraded":"2024-03-28T18:55:47.164529+0000","last_fullsized":"2024-03-28T18:55:47.164529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826964+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:55:46.826964+0000","last_peered":"2024-03-28T18:55:46.826964+0000","last_clean":"2024-03-28T18:55:46.826964+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:55:46.826964+0000","last_undegraded":"2024-03-28T18:55:46.826964+0000","last_fullsized":"2024-03-28T18:55:46.826964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826286+0000","last_change":"2024-03-28T18:55:45.609169+0000","last_active":"2024-03-28T18:55:46.826286+0000","last_peered":"2024-03-28T18:55:46.826286+0000","last_clean":"2024-03-28T18:55:46.826286+0000","last_became_active":"2024-03-28T18:55:45.601948+0000","last_became_peered":"2024-03-28T18:55:45.601948+0000","last_unstale":"2024-03-28T18:55:46.826286+0000","last_undegraded":"2024-03-28T18:55:46.826286+0000","last_fullsized":"2024-03-28T18:55:46.826286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605787+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:55:46.605787+0000","last_peered":"2024-03-28T18:55:46.605787+0000","last_clean":"2024-03-28T18:55:46.605787+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:55:46.605787+0000","last_undegraded":"2024-03-28T18:55:46.605787+0000","last_fullsized":"2024-03-28T18:55:46.605787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603014+0000","last_change":"2024-03-28T18:55:45.616238+0000","last_active":"2024-03-28T18:55:46.603014+0000","last_peered":"2024-03-28T18:55:46.603014+0000","last_clean":"2024-03-28T18:55:46.603014+0000","last_became_active":"2024-03-28T18:55:45.615371+0000","last_became_peered":"2024-03-28T18:55:45.615371+0000","last_unstale":"2024-03-28T18:55:46.603014+0000","last_undegraded":"2024-03-28T18:55:46.603014+0000","last_fullsized":"2024-03-28T18:55:46.603014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:14.661134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603091+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:55:46.603091+0000","last_peered":"2024-03-28T18:55:46.603091+0000","last_clean":"2024-03-28T18:55:46.603091+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:55:46.603091+0000","last_undegraded":"2024-03-28T18:55:46.603091+0000","last_fullsized":"2024-03-28T18:55:46.603091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164732+0000","last_change":"2024-03-28T18:55:45.613289+0000","last_active":"2024-03-28T18:55:47.164732+0000","last_peered":"2024-03-28T18:55:47.164732+0000","last_clean":"2024-03-28T18:55:47.164732+0000","last_became_active":"2024-03-28T18:55:45.610917+0000","last_became_peered":"2024-03-28T18:55:45.610917+0000","last_unstale":"2024-03-28T18:55:47.164732+0000","last_undegraded":"2024-03-28T18:55:47.164732+0000","last_fullsized":"2024-03-28T18:55:47.164732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604796+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:55:46.604796+0000","last_peered":"2024-03-28T18:55:46.604796+0000","last_clean":"2024-03-28T18:55:46.604796+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:55:46.604796+0000","last_undegraded":"2024-03-28T18:55:46.604796+0000","last_fullsized":"2024-03-28T18:55:46.604796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116103+0000","last_change":"2024-03-28T18:55:45.621789+0000","last_active":"2024-03-28T18:55:47.116103+0000","last_peered":"2024-03-28T18:55:47.116103+0000","last_clean":"2024-03-28T18:55:47.116103+0000","last_became_active":"2024-03-28T18:55:45.621163+0000","last_became_peered":"2024-03-28T18:55:45.621163+0000","last_unstale":"2024-03-28T18:55:47.116103+0000","last_undegraded":"2024-03-28T18:55:47.116103+0000","last_fullsized":"2024-03-28T18:55:47.116103+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166307+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:55:47.166307+0000","last_peered":"2024-03-28T18:55:47.166307+0000","last_clean":"2024-03-28T18:55:47.166307+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:55:47.166307+0000","last_undegraded":"2024-03-28T18:55:47.166307+0000","last_fullsized":"2024-03-28T18:55:47.166307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828245+0000","last_change":"2024-03-28T18:55:45.603774+0000","last_active":"2024-03-28T18:55:46.828245+0000","last_peered":"2024-03-28T18:55:46.828245+0000","last_clean":"2024-03-28T18:55:46.828245+0000","last_became_active":"2024-03-28T18:55:45.603451+0000","last_became_peered":"2024-03-28T18:55:45.603451+0000","last_unstale":"2024-03-28T18:55:46.828245+0000","last_undegraded":"2024-03-28T18:55:46.828245+0000","last_fullsized":"2024-03-28T18:55:46.828245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826131+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:55:46.826131+0000","last_peered":"2024-03-28T18:55:46.826131+0000","last_clean":"2024-03-28T18:55:46.826131+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:55:46.826131+0000","last_undegraded":"2024-03-28T18:55:46.826131+0000","last_fullsized":"2024-03-28T18:55:46.826131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603950+0000","last_change":"2024-03-28T18:55:45.616755+0000","last_active":"2024-03-28T18:55:46.603950+0000","last_peered":"2024-03-28T18:55:46.603950+0000","last_clean":"2024-03-28T18:55:46.603950+0000","last_became_active":"2024-03-28T18:55:45.614611+0000","last_became_peered":"2024-03-28T18:55:45.614611+0000","last_unstale":"2024-03-28T18:55:46.603950+0000","last_undegraded":"2024-03-28T18:55:46.603950+0000","last_fullsized":"2024-03-28T18:55:46.603950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:06.327795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163860+0000","last_change":"2024-03-28T18:55:34.491855+0000","last_active":"2024-03-28T18:55:47.163860+0000","last_peered":"2024-03-28T18:55:47.163860+0000","last_clean":"2024-03-28T18:55:47.163860+0000","last_became_active":"2024-03-28T18:55:34.491070+0000","last_became_peered":"2024-03-28T18:55:34.491070+0000","last_unstale":"2024-03-28T18:55:47.163860+0000","last_undegraded":"2024-03-28T18:55:47.163860+0000","last_fullsized":"2024-03-28T18:55:47.163860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613597+0000","last_change":"2024-03-28T18:55:45.623195+0000","last_active":"2024-03-28T18:55:46.613597+0000","last_peered":"2024-03-28T18:55:46.613597+0000","last_clean":"2024-03-28T18:55:46.613597+0000","last_became_active":"2024-03-28T18:55:45.622508+0000","last_became_peered":"2024-03-28T18:55:45.622508+0000","last_unstale":"2024-03-28T18:55:46.613597+0000","last_undegraded":"2024-03-28T18:55:46.613597+0000","last_fullsized":"2024-03-28T18:55:46.613597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116530+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:55:47.116530+0000","last_peered":"2024-03-28T18:55:47.116530+0000","last_clean":"2024-03-28T18:55:47.116530+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:55:47.116530+0000","last_undegraded":"2024-03-28T18:55:47.116530+0000","last_fullsized":"2024-03-28T18:55:47.116530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117584+0000","last_change":"2024-03-28T18:55:45.618841+0000","last_active":"2024-03-28T18:55:47.117584+0000","last_peered":"2024-03-28T18:55:47.117584+0000","last_clean":"2024-03-28T18:55:47.117584+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:55:47.117584+0000","last_undegraded":"2024-03-28T18:55:47.117584+0000","last_fullsized":"2024-03-28T18:55:47.117584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164332+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:55:47.164332+0000","last_peered":"2024-03-28T18:55:47.164332+0000","last_clean":"2024-03-28T18:55:47.164332+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:55:47.164332+0000","last_undegraded":"2024-03-28T18:55:47.164332+0000","last_fullsized":"2024-03-28T18:55:47.164332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825990+0000","last_change":"2024-03-28T18:55:45.602886+0000","last_active":"2024-03-28T18:55:46.825990+0000","last_peered":"2024-03-28T18:55:46.825990+0000","last_clean":"2024-03-28T18:55:46.825990+0000","last_became_active":"2024-03-28T18:55:45.602610+0000","last_became_peered":"2024-03-28T18:55:45.602610+0000","last_unstale":"2024-03-28T18:55:46.825990+0000","last_undegraded":"2024-03-28T18:55:46.825990+0000","last_fullsized":"2024-03-28T18:55:46.825990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116836+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:55:47.116836+0000","last_peered":"2024-03-28T18:55:47.116836+0000","last_clean":"2024-03-28T18:55:47.116836+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:55:47.116836+0000","last_undegraded":"2024-03-28T18:55:47.116836+0000","last_fullsized":"2024-03-28T18:55:47.116836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602814+0000","last_change":"2024-03-28T18:55:45.608673+0000","last_active":"2024-03-28T18:55:46.602814+0000","last_peered":"2024-03-28T18:55:46.602814+0000","last_clean":"2024-03-28T18:55:46.602814+0000","last_became_active":"2024-03-28T18:55:45.608376+0000","last_became_peered":"2024-03-28T18:55:45.608376+0000","last_unstale":"2024-03-28T18:55:46.602814+0000","last_undegraded":"2024-03-28T18:55:46.602814+0000","last_fullsized":"2024-03-28T18:55:46.602814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:31.781549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608764+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:55:46.608764+0000","last_peered":"2024-03-28T18:55:46.608764+0000","last_clean":"2024-03-28T18:55:46.608764+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:55:46.608764+0000","last_undegraded":"2024-03-28T18:55:46.608764+0000","last_fullsized":"2024-03-28T18:55:46.608764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163497+0000","last_change":"2024-03-28T18:55:45.609565+0000","last_active":"2024-03-28T18:55:47.163497+0000","last_peered":"2024-03-28T18:55:47.163497+0000","last_clean":"2024-03-28T18:55:47.163497+0000","last_became_active":"2024-03-28T18:55:45.609082+0000","last_became_peered":"2024-03-28T18:55:45.609082+0000","last_unstale":"2024-03-28T18:55:47.163497+0000","last_undegraded":"2024-03-28T18:55:47.163497+0000","last_fullsized":"2024-03-28T18:55:47.163497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613540+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:55:46.613540+0000","last_peered":"2024-03-28T18:55:46.613540+0000","last_clean":"2024-03-28T18:55:46.613540+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:55:46.613540+0000","last_undegraded":"2024-03-28T18:55:46.613540+0000","last_fullsized":"2024-03-28T18:55:46.613540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163662+0000","last_change":"2024-03-28T18:55:45.618585+0000","last_active":"2024-03-28T18:55:47.163662+0000","last_peered":"2024-03-28T18:55:47.163662+0000","last_clean":"2024-03-28T18:55:47.163662+0000","last_became_active":"2024-03-28T18:55:45.617895+0000","last_became_peered":"2024-03-28T18:55:45.617895+0000","last_unstale":"2024-03-28T18:55:47.163662+0000","last_undegraded":"2024-03-28T18:55:47.163662+0000","last_fullsized":"2024-03-28T18:55:47.163662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:54.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166181+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:55:47.166181+0000","last_peered":"2024-03-28T18:55:47.166181+0000","last_clean":"2024-03-28T18:55:47.166181+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:55:47.166181+0000","last_undegraded":"2024-03-28T18:55:47.166181+0000","last_fullsized":"2024-03-28T18:55:47.166181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":856064,"data_stored":690301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":101489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":99491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":102507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":101055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:55:51.724 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T18:55:51.725 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T18:55:51.725 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-28T18:55:51.725 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-28T18:55:51.726 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-28T18:55:51.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.981+0000 7f0a9ea00640 1 Processor -- start 2024-03-28T18:55:51.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.981+0000 7f0a9ea00640 1 -- start start 2024-03-28T18:55:51.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.983+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c58c0 0x7f0a980c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.983+0000 7f0a9ea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a98158bd0 con 0x7f0a980c8ed0 2024-03-28T18:55:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.983+0000 7f0a9ea00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a98158eb0 con 0x7f0a980be5d0 2024-03-28T18:55:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.983+0000 7f0a9ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98159190 con 0x7f0a980c58c0 2024-03-28T18:55:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.983+0000 7f0a9cc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0a980c8ed0 0x7f0a981555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59686/0 (socket says 172.21.15.27:59686) 2024-03-28T18:55:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.983+0000 7f0a9cc00640 1 -- 172.21.15.27:0/1024174280 learned_addr learned my addr 172.21.15.27:0/1024174280 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:51.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.984+0000 7f0a9d600640 1 --2- 172.21.15.27:0/1024174280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c58c0 0x7f0a980c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.984+0000 7f0a97e00640 1 -- 172.21.15.27:0/1024174280 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669943724 0 0) 0x7f0a98158bd0 con 0x7f0a980c8ed0 2024-03-28T18:55:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.984+0000 7f0a97e00640 1 -- 172.21.15.27:0/1024174280 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a80003520 con 0x7f0a980c8ed0 2024-03-28T18:55:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.984+0000 7f0a9d600640 1 -- 172.21.15.27:0/1024174280 >> v1:172.21.15.155:6789/0 conn(0x7f0a980be5d0 legacy=0x7f0a98155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.984+0000 7f0a9d600640 1 -- 172.21.15.27:0/1024174280 >> v1:172.21.15.27:6789/0 conn(0x7f0a980c8ed0 legacy=0x7f0a981555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.984+0000 7f0a9d600640 1 -- 172.21.15.27:0/1024174280 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98159470 con 0x7f0a980c58c0 2024-03-28T18:55:51.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.985+0000 7f0a9d600640 1 --2- 172.21.15.27:0/1024174280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c58c0 0x7f0a980c5ca0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0097d0 tx=0x7f0a8c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb9ca45963641c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:51.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.986+0000 7f0a97e00640 1 -- 172.21.15.27:0/1024174280 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c035020 con 0x7f0a980c58c0 2024-03-28T18:55:51.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.986+0000 7f0a97e00640 1 -- 172.21.15.27:0/1024174280 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c031800 con 0x7f0a980c58c0 2024-03-28T18:55:51.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.986+0000 7f0a97e00640 1 -- 172.21.15.27:0/1024174280 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c031bd0 con 0x7f0a980c58c0 2024-03-28T18:55:51.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.986+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1024174280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c58c0 msgr2=0x7f0a980c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.986+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/1024174280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c58c0 0x7f0a980c5ca0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0097d0 tx=0x7f0a8c02b8a0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.987+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1024174280 shutdown_connections 2024-03-28T18:55:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.987+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/1024174280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c58c0 0x7f0a980c5ca0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.987+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1024174280 >> 172.21.15.27:0/1024174280 conn(0x7f0a9806b6f0 msgr2=0x7f0a980acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.987+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1024174280 shutdown_connections 2024-03-28T18:55:51.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.987+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1024174280 wait complete. 2024-03-28T18:55:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.988+0000 7f0a9ea00640 1 Processor -- start 2024-03-28T18:55:51.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.988+0000 7f0a9ea00640 1 -- start start 2024-03-28T18:55:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.989+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980be5d0 0x7f0a981523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.989+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c58c0 0x7f0a980c2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.989+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c8ed0 0x7f0a980abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.989+0000 7f0a9ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980b09a0 con 0x7f0a980c58c0 2024-03-28T18:55:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.989+0000 7f0a9ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980b0820 con 0x7f0a980be5d0 2024-03-28T18:55:51.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.989+0000 7f0a9ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980ad7d0 con 0x7f0a980c8ed0 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c58c0 0x7f0a980c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c58c0 0x7f0a980c2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59336/0 (socket says 172.21.15.27:59336) 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 -- 172.21.15.27:0/2292784237 learned_addr learned my addr 172.21.15.27:0/2292784237 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9d600640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980be5d0 0x7f0a981523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9e000640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c8ed0 0x7f0a980abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 -- 172.21.15.27:0/2292784237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c8ed0 msgr2=0x7f0a980abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c8ed0 0x7f0a980abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 -- 172.21.15.27:0/2292784237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980be5d0 msgr2=0x7f0a981523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980be5d0 0x7f0a981523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 -- 172.21.15.27:0/2292784237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a981520c0 con 0x7f0a980c58c0 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.990+0000 7f0a9cc00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c58c0 0x7f0a980c2010 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400c440 tx=0x7f0a8400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.991+0000 7f0a95600640 1 -- 172.21.15.27:0/2292784237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84017070 con 0x7f0a980c58c0 2024-03-28T18:55:51.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.991+0000 7f0a95600640 1 -- 172.21.15.27:0/2292784237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a84005ce0 con 0x7f0a980c58c0 2024-03-28T18:55:51.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.991+0000 7f0a95600640 1 -- 172.21.15.27:0/2292784237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a840072b0 con 0x7f0a980c58c0 2024-03-28T18:55:51.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.991+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a980c4cc0 con 0x7f0a980c58c0 2024-03-28T18:55:51.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.991+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980ac3e0 con 0x7f0a980c58c0 2024-03-28T18:55:51.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.992+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a980c4ab0 con 0x7f0a980c58c0 2024-03-28T18:55:51.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.993+0000 7f0a95600640 1 -- 172.21.15.27:0/2292784237 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0a8401a070 con 0x7f0a980c58c0 2024-03-28T18:55:51.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.995+0000 7f0a95600640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bbd0 0x7f0a7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:51.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.995+0000 7f0a95600640 1 -- 172.21.15.27:0/2292784237 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f0a8408a090 con 0x7f0a980c58c0 2024-03-28T18:55:51.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.995+0000 7f0a9d600640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bbd0 0x7f0a7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:51.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:51.996+0000 7f0a9d600640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bbd0 0x7f0a7806e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c036000 tx=0x7f0a8c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.001+0000 7f0a95600640 1 -- 172.21.15.27:0/2292784237 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0a84058910 con 0x7f0a980c58c0 2024-03-28T18:55:52.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.260+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0a98069640 con 0x7f0a980c58c0 2024-03-28T18:55:52.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.264+0000 7f0a95600640 1 -- 172.21.15.27:0/2292784237 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+172577 (secure 0 0 0) 0x7f0a84058210 con 0x7f0a980c58c0 2024-03-28T18:55:52.267 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bbd0 msgr2=0x7f0a7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bbd0 0x7f0a7806e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c036000 tx=0x7f0a8c008030 comp rx=0 tx=0).stop 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c58c0 msgr2=0x7f0a980c2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c58c0 0x7f0a980c2010 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400c440 tx=0x7f0a8400c900 comp rx=0 tx=0).stop 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 shutdown_connections 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bbd0 0x7f0a7806e090 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980c8ed0 0x7f0a980abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c58c0 0x7f0a980c2010 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2292784237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980be5d0 0x7f0a981523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.269+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 >> 172.21.15.27:0/2292784237 conn(0x7f0a9806b6f0 msgr2=0x7f0a98072570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.270+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 shutdown_connections 2024-03-28T18:55:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.270+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2292784237 wait complete. 2024-03-28T18:55:52.289 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"flags":0,"active_gid":4116,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1637319240},{"type":"v1","addr":"172.21.15.27:6833","nonce":1637319240}]},"active_addr":"172.21.15.27:6833/1637319240","active_change":"2024-03-28T18:55:14.864606+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2098241346}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3793776866}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":603746998}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":986714012}]}]} 2024-03-28T18:55:52.293 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-28T18:55:52.293 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-28T18:55:52.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:52.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.510+0000 7f63cb200640 1 Processor -- start 2024-03-28T18:55:52.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.510+0000 7f63cb200640 1 -- start start 2024-03-28T18:55:52.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.512+0000 7f63cb200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63cc150ae0 0x7f63cc0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.512+0000 7f63cb200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 0x7f63cc0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.512+0000 7f63cb200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc150ec0 con 0x7f63cc0c6ee0 2024-03-28T18:55:52.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.512+0000 7f63cb200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc06dc70 con 0x7f63cc150ae0 2024-03-28T18:55:52.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.512+0000 7f63cb200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc153550 con 0x7f63cc0a61a0 2024-03-28T18:55:52.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.512+0000 7f63c9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63cc150ae0 0x7f63cc0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.512+0000 7f63c9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63cc150ae0 0x7f63cc0a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59352/0 (socket says 172.21.15.27:59352) 2024-03-28T18:55:52.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.512+0000 7f63c9e00640 1 -- 172.21.15.27:0/3051097792 learned_addr learned my addr 172.21.15.27:0/3051097792 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:52.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.513+0000 7f63c3e00640 1 --2- 172.21.15.27:0/3051097792 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 0x7f63cc0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.513+0000 7f63c9e00640 1 -- 172.21.15.27:0/3051097792 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 msgr2=0x7f63cc0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:52.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.513+0000 7f63c9e00640 1 --2- 172.21.15.27:0/3051097792 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 0x7f63cc0c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.513+0000 7f63c9e00640 1 -- 172.21.15.27:0/3051097792 >> v1:172.21.15.155:6789/0 conn(0x7f63cc0c6ee0 legacy=0x7f63cc153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:52.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.513+0000 7f63c9e00640 1 -- 172.21.15.27:0/3051097792 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc06df10 con 0x7f63cc150ae0 2024-03-28T18:55:52.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.514+0000 7f63c9e00640 1 --2- 172.21.15.27:0/3051097792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc0a5c60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009870 tx=0x7f63bc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731cc37afe5a30e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:52.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.514+0000 7f63c9400640 1 -- 172.21.15.27:0/3051097792 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0317c0 con 0x7f63cc150ae0 2024-03-28T18:55:52.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.514+0000 7f63c9400640 1 -- 172.21.15.27:0/3051097792 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc037070 con 0x7f63cc150ae0 2024-03-28T18:55:52.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.514+0000 7f63c9400640 1 -- 172.21.15.27:0/3051097792 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc031b80 con 0x7f63cc150ae0 2024-03-28T18:55:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.515+0000 7f63cb200640 1 -- 172.21.15.27:0/3051097792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 msgr2=0x7f63cc0a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.515+0000 7f63cb200640 1 --2- 172.21.15.27:0/3051097792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc0a5c60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009870 tx=0x7f63bc02b860 comp rx=0 tx=0).stop 2024-03-28T18:55:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.515+0000 7f63cb200640 1 -- 172.21.15.27:0/3051097792 shutdown_connections 2024-03-28T18:55:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.515+0000 7f63cb200640 1 --2- 172.21.15.27:0/3051097792 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 0x7f63cc0c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.515+0000 7f63cb200640 1 --2- 172.21.15.27:0/3051097792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc0a5c60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.515+0000 7f63cb200640 1 -- 172.21.15.27:0/3051097792 >> 172.21.15.27:0/3051097792 conn(0x7f63cc06b0c0 msgr2=0x7f63cc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.515+0000 7f63cb200640 1 -- 172.21.15.27:0/3051097792 shutdown_connections 2024-03-28T18:55:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.515+0000 7f63cb200640 1 -- 172.21.15.27:0/3051097792 wait complete. 2024-03-28T18:55:52.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.517+0000 7f63cb200640 1 Processor -- start 2024-03-28T18:55:52.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.517+0000 7f63cb200640 1 -- start start 2024-03-28T18:55:52.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63cb200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 0x7f63cc0bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63cb200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63cc0c6ee0 0x7f63cc14d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63cb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc14db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63cb200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc06e7b0 con 0x7f63cc150ae0 2024-03-28T18:55:52.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63cb200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc06e4d0 con 0x7f63cc0c6ee0 2024-03-28T18:55:52.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63cb200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1569d0 con 0x7f63cc0a61a0 2024-03-28T18:55:52.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63ca800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc14db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63ca800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc14db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59360/0 (socket says 172.21.15.27:59360) 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.518+0000 7f63ca800640 1 -- 172.21.15.27:0/1560577965 learned_addr learned my addr 172.21.15.27:0/1560577965 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63c9e00640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 0x7f63cc0bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63c3e00640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63cc0c6ee0 0x7f63cc14d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63ca800640 1 -- 172.21.15.27:0/1560577965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 msgr2=0x7f63cc0bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63ca800640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 0x7f63cc0bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63ca800640 1 -- 172.21.15.27:0/1560577965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63cc0c6ee0 msgr2=0x7f63cc14d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63ca800640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63cc0c6ee0 0x7f63cc14d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63ca800640 1 -- 172.21.15.27:0/1560577965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc14e420 con 0x7f63cc150ae0 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63ca800640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc14db60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f63c400c780 tx=0x7f63c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63c2000640 1 -- 172.21.15.27:0/1560577965 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4004b60 con 0x7f63cc150ae0 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63c2000640 1 -- 172.21.15.27:0/1560577965 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4004d00 con 0x7f63cc150ae0 2024-03-28T18:55:52.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.519+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc1eba60 con 0x7f63cc150ae0 2024-03-28T18:55:52.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.520+0000 7f63c2000640 1 -- 172.21.15.27:0/1560577965 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c400aa90 con 0x7f63cc150ae0 2024-03-28T18:55:52.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.520+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc0c04d0 con 0x7f63cc150ae0 2024-03-28T18:55:52.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.521+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc067dd0 con 0x7f63cc150ae0 2024-03-28T18:55:52.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.522+0000 7f63c2000640 1 -- 172.21.15.27:0/1560577965 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f63c400ac30 con 0x7f63cc150ae0 2024-03-28T18:55:52.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.523+0000 7f63c2000640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63a806bbd0 0x7f63a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.523+0000 7f63c2000640 1 -- 172.21.15.27:0/1560577965 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f63c408aff0 con 0x7f63cc150ae0 2024-03-28T18:55:52.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.523+0000 7f63c9e00640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63a806bbd0 0x7f63a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.524+0000 7f63c9e00640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63a806bbd0 0x7f63a806e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f63bc036000 tx=0x7f63bc03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:52.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.527+0000 7f63c2000640 1 -- 172.21.15.27:0/1560577965 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f63c4020070 con 0x7f63cc150ae0 2024-03-28T18:55:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.738+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63cc06da50 con 0x7f63cc150ae0 2024-03-28T18:55:52.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.742+0000 7f63c2000640 1 -- 172.21.15.27:0/1560577965 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+20307 (secure 0 0 0) 0x7f63c40597c0 con 0x7f63cc150ae0 2024-03-28T18:55:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:52.742 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:46.587251+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:52.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.744+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63a806bbd0 msgr2=0x7f63a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.744+0000 7f63cb200640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63a806bbd0 0x7f63a806e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f63bc036000 tx=0x7f63bc03d040 comp rx=0 tx=0).stop 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.744+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 msgr2=0x7f63cc14db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.744+0000 7f63cb200640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc14db60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f63c400c780 tx=0x7f63c400cc40 comp rx=0 tx=0).stop 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.745+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 shutdown_connections 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.745+0000 7f63cb200640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63a806bbd0 0x7f63a806e090 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.745+0000 7f63cb200640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc150ae0 0x7f63cc14db60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.745+0000 7f63cb200640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63cc0c6ee0 0x7f63cc14d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.745+0000 7f63cb200640 1 --2- 172.21.15.27:0/1560577965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63cc0a61a0 0x7f63cc0bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.745+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 >> 172.21.15.27:0/1560577965 conn(0x7f63cc06b0c0 msgr2=0x7f63cc0c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.745+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 shutdown_connections 2024-03-28T18:55:52.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.745+0000 7f63cb200640 1 -- 172.21.15.27:0/1560577965 wait complete. 2024-03-28T18:55:52.761 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-28T18:55:52.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.965+0000 7fd83ee00640 1 Processor -- start 2024-03-28T18:55:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.966+0000 7fd83ee00640 1 -- start start 2024-03-28T18:55:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd83ee00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd838150c70 0x7fd8380a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd83ee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8380c3690 0x7fd8380c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd83ee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c6660 0x7fd8381534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd83ee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8380ab4e0 con 0x7fd8380c3690 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd83ee00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8380ac1f0 con 0x7fd838150c70 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd83ee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380c3af0 con 0x7fd8380c6660 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd837400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8380c3690 0x7fd8380c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd837400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8380c3690 0x7fd8380c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59370/0 (socket says 172.21.15.27:59370) 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.967+0000 7fd837400640 1 -- 172.21.15.27:0/3085487659 learned_addr learned my addr 172.21.15.27:0/3085487659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.968+0000 7fd837e00640 1 --2- 172.21.15.27:0/3085487659 >> v2:172.21.15.155:3300/0 conn(0x7fd838150c70 0x7fd8380a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.968+0000 7fd83ce00640 1 --2- 172.21.15.27:0/3085487659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c6660 0x7fd8381534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.968+0000 7fd837400640 1 -- 172.21.15.27:0/3085487659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c6660 msgr2=0x7fd8381534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.968+0000 7fd837400640 1 --2- 172.21.15.27:0/3085487659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c6660 0x7fd8381534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.968+0000 7fd837400640 1 -- 172.21.15.27:0/3085487659 >> v2:172.21.15.155:3300/0 conn(0x7fd838150c70 msgr2=0x7fd8380a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:52.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.968+0000 7fd837400640 1 --2- 172.21.15.27:0/3085487659 >> v2:172.21.15.155:3300/0 conn(0x7fd838150c70 0x7fd8380a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.968+0000 7fd837400640 1 -- 172.21.15.27:0/3085487659 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8380c3d90 con 0x7fd8380c3690 2024-03-28T18:55:52.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.968+0000 7fd83ce00640 1 --2- 172.21.15.27:0/3085487659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c6660 0x7fd8381534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.969+0000 7fd837400640 1 --2- 172.21.15.27:0/3085487659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8380c3690 0x7fd8380c6120 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd8380c7950 tx=0x7fd82402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6037e22068e23a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.969+0000 7fd836a00640 1 -- 172.21.15.27:0/3085487659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240317c0 con 0x7fd8380c3690 2024-03-28T18:55:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.969+0000 7fd836a00640 1 -- 172.21.15.27:0/3085487659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd824037070 con 0x7fd8380c3690 2024-03-28T18:55:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.969+0000 7fd836a00640 1 -- 172.21.15.27:0/3085487659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824031b80 con 0x7fd8380c3690 2024-03-28T18:55:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 -- 172.21.15.27:0/3085487659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8380c3690 msgr2=0x7fd8380c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 --2- 172.21.15.27:0/3085487659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8380c3690 0x7fd8380c6120 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd8380c7950 tx=0x7fd82402b860 comp rx=0 tx=0).stop 2024-03-28T18:55:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 -- 172.21.15.27:0/3085487659 shutdown_connections 2024-03-28T18:55:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 --2- 172.21.15.27:0/3085487659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c6660 0x7fd8381534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 --2- 172.21.15.27:0/3085487659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8380c3690 0x7fd8380c6120 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 --2- 172.21.15.27:0/3085487659 >> v2:172.21.15.155:3300/0 conn(0x7fd838150c70 0x7fd8380a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 -- 172.21.15.27:0/3085487659 >> 172.21.15.27:0/3085487659 conn(0x7fd83806b9d0 msgr2=0x7fd8380c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 -- 172.21.15.27:0/3085487659 shutdown_connections 2024-03-28T18:55:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.970+0000 7fd83ee00640 1 -- 172.21.15.27:0/3085487659 wait complete. 2024-03-28T18:55:52.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.972+0000 7fd83ee00640 1 Processor -- start 2024-03-28T18:55:52.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.972+0000 7fd83ee00640 1 -- start start 2024-03-28T18:55:52.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c3690 0x7fd838151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8380c6660 0x7fd8381562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838150c70 0x7fd838071d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380abcd0 con 0x7fd838150c70 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380ab9f0 con 0x7fd8380c6660 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380c1810 con 0x7fd8380c3690 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838150c70 0x7fd838071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838150c70 0x7fd838071d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59386/0 (socket says 172.21.15.27:59386) 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd83ce00640 1 -- 172.21.15.27:0/4064399131 learned_addr learned my addr 172.21.15.27:0/4064399131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd837e00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c3690 0x7fd838151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.973+0000 7fd837400640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8380c6660 0x7fd8381562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd83ce00640 1 -- 172.21.15.27:0/4064399131 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c3690 msgr2=0x7fd838151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd83ce00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c3690 0x7fd838151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd83ce00640 1 -- 172.21.15.27:0/4064399131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8380c6660 msgr2=0x7fd8381562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:52.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd83ce00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8380c6660 0x7fd8381562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:52.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd83ce00640 1 -- 172.21.15.27:0/4064399131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838152020 con 0x7fd838150c70 2024-03-28T18:55:52.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd83ce00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838150c70 0x7fd838071d90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd83000cef0 tx=0x7fd83000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:52.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd82fe00640 1 -- 172.21.15.27:0/4064399131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83001a070 con 0x7fd838150c70 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd82fe00640 1 -- 172.21.15.27:0/4064399131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83000d860 con 0x7fd838150c70 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.974+0000 7fd82fe00640 1 -- 172.21.15.27:0/4064399131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83000dc50 con 0x7fd838150c70 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.975+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8381516b0 con 0x7fd838150c70 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.975+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8381557f0 con 0x7fd838150c70 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.976+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83804f400 con 0x7fd838150c70 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.977+0000 7fd82fe00640 1 -- 172.21.15.27:0/4064399131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd83001d070 con 0x7fd838150c70 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.978+0000 7fd82fe00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd81806bbd0 0x7fd81806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.978+0000 7fd82fe00640 1 -- 172.21.15.27:0/4064399131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fd83008b570 con 0x7fd838150c70 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.978+0000 7fd837e00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd81806bbd0 0x7fd81806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:52.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.979+0000 7fd837e00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd81806bbd0 0x7fd81806e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd83814fc20 tx=0x7fd828008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:52.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:52.987+0000 7fd82fe00640 1 -- 172.21.15.27:0/4064399131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd830059790 con 0x7fd838150c70 2024-03-28T18:55:53.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.188+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8380c5f00 con 0x7fd838150c70 2024-03-28T18:55:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.191+0000 7fd82fe00640 1 -- 172.21.15.27:0/4064399131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+20307 (secure 0 0 0) 0x7fd83005d830 con 0x7fd838150c70 2024-03-28T18:55:53.192 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:53.192 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:46.587251+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd81806bbd0 msgr2=0x7fd81806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd81806bbd0 0x7fd81806e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd83814fc20 tx=0x7fd828008040 comp rx=0 tx=0).stop 2024-03-28T18:55:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838150c70 msgr2=0x7fd838071d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838150c70 0x7fd838071d90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd83000cef0 tx=0x7fd83000a710 comp rx=0 tx=0).stop 2024-03-28T18:55:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 shutdown_connections 2024-03-28T18:55:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd81806bbd0 0x7fd81806e090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd838150c70 0x7fd838071d90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8380c6660 0x7fd8381562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 --2- 172.21.15.27:0/4064399131 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8380c3690 0x7fd838151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 >> 172.21.15.27:0/4064399131 conn(0x7fd83806b9d0 msgr2=0x7fd8380a8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 shutdown_connections 2024-03-28T18:55:53.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.194+0000 7fd83ee00640 1 -- 172.21.15.27:0/4064399131 wait complete. 2024-03-28T18:55:53.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-28T18:55:53.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-28T18:55:53.214 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-28T18:55:53.214 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-28T18:55:53.214 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-28T18:55:53.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-28T18:55:53.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-28T18:55:53.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-28T18:55:53.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.541+0000 7f07ac600640 1 Processor -- start 2024-03-28T18:55:53.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.541+0000 7f07ac600640 1 -- start start 2024-03-28T18:55:53.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.543+0000 7f07ac600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40c6760 0x7f07a4166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.543+0000 7f07ac600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f07a40aa300 0x7f07a40c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.543+0000 7f07ac600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40c12d0 con 0x7f07a4166570 2024-03-28T18:55:53.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.543+0000 7f07ac600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a40733f0 con 0x7f07a40aa300 2024-03-28T18:55:53.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.544+0000 7f07ac600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4073640 con 0x7f07a40c6760 2024-03-28T18:55:53.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.544+0000 7f07a9c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40c6760 0x7f07a4166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.544+0000 7f07a9c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40c6760 0x7f07a4166030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45070/0 (socket says 172.21.15.27:45070) 2024-03-28T18:55:53.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.544+0000 7f07a9c00640 1 -- 172.21.15.27:0/4131123325 learned_addr learned my addr 172.21.15.27:0/4131123325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.545+0000 7f07a9c00640 1 -- 172.21.15.27:0/4131123325 >> v2:172.21.15.155:3300/0 conn(0x7f07a40aa300 msgr2=0x7f07a40c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.545+0000 7f07a9c00640 1 --2- 172.21.15.27:0/4131123325 >> v2:172.21.15.155:3300/0 conn(0x7f07a40aa300 0x7f07a40c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.545+0000 7f07a9c00640 1 -- 172.21.15.27:0/4131123325 >> v1:172.21.15.155:6789/0 conn(0x7f07a4166570 legacy=0x7f07a40a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.545+0000 7f07a9c00640 1 -- 172.21.15.27:0/4131123325 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40abe70 con 0x7f07a40c6760 2024-03-28T18:55:53.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.547+0000 7f07a9c00640 1 --2- 172.21.15.27:0/4131123325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40c6760 0x7f07a4166030 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0790009870 tx=0x7f079002b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc1ed5e5e861f8c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.548+0000 7f07a3e00640 1 -- 172.21.15.27:0/4131123325 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790034070 con 0x7f07a40c6760 2024-03-28T18:55:53.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.548+0000 7f07a3e00640 1 -- 172.21.15.27:0/4131123325 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0790030860 con 0x7f07a40c6760 2024-03-28T18:55:53.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.548+0000 7f07a3e00640 1 -- 172.21.15.27:0/4131123325 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790030c10 con 0x7f07a40c6760 2024-03-28T18:55:53.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.549+0000 7f07ac600640 1 -- 172.21.15.27:0/4131123325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40c6760 msgr2=0x7f07a4166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.549+0000 7f07ac600640 1 --2- 172.21.15.27:0/4131123325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40c6760 0x7f07a4166030 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0790009870 tx=0x7f079002b9a0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.550+0000 7f07ac600640 1 -- 172.21.15.27:0/4131123325 shutdown_connections 2024-03-28T18:55:53.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.550+0000 7f07ac600640 1 --2- 172.21.15.27:0/4131123325 >> v2:172.21.15.155:3300/0 conn(0x7f07a40aa300 0x7f07a40c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.550+0000 7f07ac600640 1 --2- 172.21.15.27:0/4131123325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40c6760 0x7f07a4166030 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.550+0000 7f07ac600640 1 -- 172.21.15.27:0/4131123325 >> 172.21.15.27:0/4131123325 conn(0x7f07a40c6c00 msgr2=0x7f07a40c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.550+0000 7f07ac600640 1 -- 172.21.15.27:0/4131123325 shutdown_connections 2024-03-28T18:55:53.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.550+0000 7f07ac600640 1 -- 172.21.15.27:0/4131123325 wait complete. 2024-03-28T18:55:53.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.553+0000 7f07ac600640 1 Processor -- start 2024-03-28T18:55:53.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.553+0000 7f07ac600640 1 -- start start 2024-03-28T18:55:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.555+0000 7f07ac600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40aa300 0x7f07a4164bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.556+0000 7f07ac600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a40c6760 0x7f07a4160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.556+0000 7f07ac600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a4166570 0x7f07a415fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.556+0000 7f07a9c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40aa300 0x7f07a4164bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.557+0000 7f07a9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a40c6760 0x7f07a4160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.557+0000 7f07a9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a40c6760 0x7f07a4160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43472/0 (socket says 172.21.15.27:43472) 2024-03-28T18:55:53.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.555+0000 7f750be00640 1 Processor -- start 2024-03-28T18:55:53.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.556+0000 7f750be00640 1 -- start start 2024-03-28T18:55:53.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.557+0000 7f07a9c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40aa300 0x7f07a4164bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45072/0 (socket says 172.21.15.27:45072) 2024-03-28T18:55:53.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.557+0000 7f07aa600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a4166570 0x7f07a415fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f07aa600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a4166570 0x7f07a415fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59392/0 (socket says 172.21.15.27:59392) 2024-03-28T18:55:53.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f07ac600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40a6370 con 0x7f07a4166570 2024-03-28T18:55:53.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f07ac600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40abb00 con 0x7f07a40c6760 2024-03-28T18:55:53.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f07ac600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40a64f0 con 0x7f07a40aa300 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f750be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75040d7920 0x7f750400edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f750be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75040069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f750be00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75040d7d00 con 0x7f7504050720 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f750be00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75040db250 con 0x7f75040d7920 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f750be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750404ba00 con 0x7f750400fcf0 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f750b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75040069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f750a000640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7504050720 0x7f75040062c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55574/0 (socket says 172.21.15.27:55574) 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f750a000640 1 -- 172.21.15.27:0/233905532 learned_addr learned my addr 172.21.15.27:0/233905532 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:53.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f750aa00640 1 --2- 172.21.15.27:0/233905532 >> v2:172.21.15.27:3300/0 conn(0x7f75040d7920 0x7f750400edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.558+0000 7f07a9200640 1 -- 172.21.15.27:0/4132524790 learned_addr learned my addr 172.21.15.27:0/4132524790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f07aa600640 1 -- 172.21.15.27:0/4132524790 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40aa300 msgr2=0x7f07a4164bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f07aa600640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40aa300 0x7f07a4164bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f07aa600640 1 -- 172.21.15.27:0/4132524790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a40c6760 msgr2=0x7f07a4160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f07aa600640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a40c6760 0x7f07a4160b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.559+0000 7f07aa600640 1 -- 172.21.15.27:0/4132524790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a4086590 con 0x7f07a4166570 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.560+0000 7f07aa600640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a4166570 0x7f07a415fc30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f079c00be50 tx=0x7f079c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.561+0000 7f07a9c00640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f07a40aa300 0x7f07a4164bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.562+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c019070 con 0x7f07a4166570 2024-03-28T18:55:53.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.562+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a4070dc0 con 0x7f07a4166570 2024-03-28T18:55:53.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.562+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a4086f80 con 0x7f07a4166570 2024-03-28T18:55:53.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.562+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c014440 con 0x7f07a4166570 2024-03-28T18:55:53.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.562+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c019070 con 0x7f07a4166570 2024-03-28T18:55:53.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.561+0000 7f7509600640 1 -- 172.21.15.27:0/233905532 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324123237 0 0) 0x7f75040d7d00 con 0x7f7504050720 2024-03-28T18:55:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.561+0000 7f7509600640 1 -- 172.21.15.27:0/233905532 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f0003520 con 0x7f7504050720 2024-03-28T18:55:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.561+0000 7f750b400640 1 -- 172.21.15.27:0/233905532 >> v2:172.21.15.27:3300/0 conn(0x7f75040d7920 msgr2=0x7f750400edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.561+0000 7f750b400640 1 --2- 172.21.15.27:0/233905532 >> v2:172.21.15.27:3300/0 conn(0x7f75040d7920 0x7f750400edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.561+0000 7f750b400640 1 -- 172.21.15.27:0/233905532 >> v1:172.21.15.155:6789/0 conn(0x7f7504050720 legacy=0x7f75040062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.561+0000 7f750b400640 1 -- 172.21.15.27:0/233905532 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750404bce0 con 0x7f750400fcf0 2024-03-28T18:55:53.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.562+0000 7f750aa00640 1 --2- 172.21.15.27:0/233905532 >> v2:172.21.15.27:3300/0 conn(0x7f75040d7920 0x7f750400edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:53.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.565+0000 7f0020800640 1 Processor -- start 2024-03-28T18:55:53.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.565+0000 7f0020800640 1 -- start start 2024-03-28T18:55:53.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.565+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0780001030 con 0x7f07a4166570 2024-03-28T18:55:53.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.565+0000 7f750b400640 1 --2- 172.21.15.27:0/233905532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75040069d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f750c02d600 tx=0x7f750c0a70f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9c34556def10002 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.566+0000 7fc3b7800640 1 Processor -- start 2024-03-28T18:55:53.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7f0020800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f001806cc40 0x7f00180c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7f0020800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f00180c4000 0x7f00181534a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7f0020800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001806e6b0 con 0x7f00181508a0 2024-03-28T18:55:53.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7f0020800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001806e990 con 0x7f001806cc40 2024-03-28T18:55:53.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7f0020800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001806ec70 con 0x7f00180c4000 2024-03-28T18:55:53.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7fc3b7800640 1 -- start start 2024-03-28T18:55:53.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7f7509600640 1 -- 172.21.15.27:0/233905532 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c0b4070 con 0x7f750400fcf0 2024-03-28T18:55:53.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7f7509600640 1 -- 172.21.15.27:0/233905532 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750c0a7d50 con 0x7f750400fcf0 2024-03-28T18:55:53.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.567+0000 7f7509600640 1 -- 172.21.15.27:0/233905532 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c0af640 con 0x7f750400fcf0 2024-03-28T18:55:53.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.568+0000 7f001d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f001806cc40 0x7f00180c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.568+0000 7f001d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f001806cc40 0x7f00180c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59412/0 (socket says 172.21.15.27:59412) 2024-03-28T18:55:53.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.568+0000 7f001d400640 1 -- 172.21.15.27:0/1666317656 learned_addr learned my addr 172.21.15.27:0/1666317656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.569+0000 7f001d400640 1 -- 172.21.15.27:0/1666317656 >> v2:172.21.15.155:3300/0 conn(0x7f00180c4000 msgr2=0x7f00181534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.569+0000 7f001d400640 1 --2- 172.21.15.27:0/1666317656 >> v2:172.21.15.155:3300/0 conn(0x7f00180c4000 0x7f00181534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.569+0000 7f001d400640 1 -- 172.21.15.27:0/1666317656 >> v1:172.21.15.27:6789/0 conn(0x7f00181508a0 legacy=0x7f00180c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.568+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f079c01c070 con 0x7f07a4166570 2024-03-28T18:55:53.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.570+0000 7f001d400640 1 -- 172.21.15.27:0/1666317656 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001806ef50 con 0x7f001806cc40 2024-03-28T18:55:53.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.570+0000 7fc3b7800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3b0152f80 0x7fc3b00c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.570+0000 7fc3b7800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.571+0000 7fc3b7800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b00c7310 con 0x7fc3b0155730 2024-03-28T18:55:53.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.571+0000 7fc3b7800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b0158d40 con 0x7fc3b0152f80 2024-03-28T18:55:53.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.571+0000 7fc3b4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3b0152f80 0x7fc3b00c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.571+0000 7fc3b4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3b0152f80 0x7fc3b00c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59420/0 (socket says 172.21.15.27:59420) 2024-03-28T18:55:53.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7f001d400640 1 --2- 172.21.15.27:0/1666317656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001806cc40 0x7f00180c3ac0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f00180c4700 tx=0x7f000802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92455420c219961f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.571+0000 7f07a1600640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f078406bdd0 0x7f078406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.571+0000 7fc3b4e00640 1 -- 172.21.15.27:0/2247855791 learned_addr learned my addr 172.21.15.27:0/2247855791 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7fc3b7800640 1 -- 172.21.15.27:0/2247855791 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0159020 con 0x7fc3b00c7970 2024-03-28T18:55:53.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7fc3afe00640 1 --2- 172.21.15.27:0/2247855791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7f001ca00640 1 -- 172.21.15.27:0/1666317656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008035020 con 0x7f001806cc40 2024-03-28T18:55:53.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.574+0000 7fc3af400640 1 -- 172.21.15.27:0/2247855791 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650908737 0 0) 0x7fc3b00c7310 con 0x7fc3b0155730 2024-03-28T18:55:53.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.573+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f079c08a850 con 0x7f07a4166570 2024-03-28T18:55:53.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.574+0000 7fc3b4e00640 1 -- 172.21.15.27:0/2247855791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 msgr2=0x7fc3b00c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.574+0000 7fc3b4e00640 1 --2- 172.21.15.27:0/2247855791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.574+0000 7fc3b4e00640 1 -- 172.21.15.27:0/2247855791 >> v1:172.21.15.27:6789/0 conn(0x7fc3b0155730 legacy=0x7fc3b0155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.574+0000 7f001ca00640 1 -- 172.21.15.27:0/1666317656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0008031870 con 0x7f001806cc40 2024-03-28T18:55:53.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.574+0000 7f001ca00640 1 -- 172.21.15.27:0/1666317656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008031c40 con 0x7f001806cc40 2024-03-28T18:55:53.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.574+0000 7f07a9c00640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f078406bdd0 0x7f078406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.574+0000 7fc3b4e00640 1 -- 172.21.15.27:0/2247855791 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0159300 con 0x7fc3b0152f80 2024-03-28T18:55:53.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.575+0000 7f07a1600640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6815/1506376930,v1:172.21.15.27:6817/1506376930] conn(0x7f0784076660 0x7f0784078a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.571+0000 7f750be00640 1 -- 172.21.15.27:0/233905532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 msgr2=0x7f75040069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.571+0000 7f750be00640 1 --2- 172.21.15.27:0/233905532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75040069d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f750c02d600 tx=0x7f750c0a70f0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7f750be00640 1 -- 172.21.15.27:0/233905532 shutdown_connections 2024-03-28T18:55:53.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7f750be00640 1 --2- 172.21.15.27:0/233905532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75040069d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7f750be00640 1 --2- 172.21.15.27:0/233905532 >> v2:172.21.15.27:3300/0 conn(0x7f75040d7920 0x7f750400edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7f750be00640 1 -- 172.21.15.27:0/233905532 >> 172.21.15.27:0/233905532 conn(0x7f750404fc00 msgr2=0x7f7504050010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7f750be00640 1 -- 172.21.15.27:0/233905532 shutdown_connections 2024-03-28T18:55:53.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.572+0000 7f750be00640 1 -- 172.21.15.27:0/233905532 wait complete. 2024-03-28T18:55:53.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.575+0000 7f750be00640 1 Processor -- start 2024-03-28T18:55:53.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.576+0000 7f750be00640 1 -- start start 2024-03-28T18:55:53.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.576+0000 7f0020800640 1 -- 172.21.15.27:0/1666317656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001806cc40 msgr2=0x7f00180c3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.576+0000 7f0020800640 1 --2- 172.21.15.27:0/1666317656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001806cc40 0x7f00180c3ac0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f00180c4700 tx=0x7f000802b910 comp rx=0 tx=0).stop 2024-03-28T18:55:53.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7fc3b4e00640 1 --2- 172.21.15.27:0/2247855791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 0x7fc3b00c58e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009870 tx=0x7fc3a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd1c569f6aabc11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.573+0000 7f1911e00640 1 Processor -- start 2024-03-28T18:55:53.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.573+0000 7f1911e00640 1 -- start start 2024-03-28T18:55:53.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.575+0000 7f1911e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f190c0c6760 0x7f190c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.575+0000 7f1911e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f190c0aa300 0x7f190c0c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.576+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 --> [v2:172.21.15.27:6815/1506376930,v1:172.21.15.27:6817/1506376930] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0784079130 con 0x7f0784076660 2024-03-28T18:55:53.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f079c08f2a0 con 0x7f07a4166570 2024-03-28T18:55:53.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f07a9200640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6815/1506376930,v1:172.21.15.27:6817/1506376930] conn(0x7f0784076660 0x7f0784078a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7fc3af400640 1 -- 172.21.15.27:0/2247855791 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4035020 con 0x7fc3b0152f80 2024-03-28T18:55:53.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f07a9c00640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f078406bdd0 0x7f078406e290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0790036000 tx=0x7f0790008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7fc3af400640 1 -- 172.21.15.27:0/2247855791 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a4031640 con 0x7fc3b0152f80 2024-03-28T18:55:53.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f07a9200640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6815/1506376930,v1:172.21.15.27:6817/1506376930] conn(0x7f0784076660 0x7f0784078a80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7fc3af400640 1 -- 172.21.15.27:0/2247855791 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a402d040 con 0x7fc3b0152f80 2024-03-28T18:55:53.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f0020800640 1 -- 172.21.15.27:0/1666317656 shutdown_connections 2024-03-28T18:55:53.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f0020800640 1 --2- 172.21.15.27:0/1666317656 >> v2:172.21.15.155:3300/0 conn(0x7f00180c4000 0x7f00181534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f0020800640 1 --2- 172.21.15.27:0/1666317656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001806cc40 0x7f00180c3ac0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f0020800640 1 -- 172.21.15.27:0/1666317656 >> 172.21.15.27:0/1666317656 conn(0x7f00180c6c00 msgr2=0x7f00180c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7fc3b7800640 1 -- 172.21.15.27:0/2247855791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 msgr2=0x7fc3b00c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7fc3b7800640 1 --2- 172.21.15.27:0/2247855791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 0x7fc3b00c58e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009870 tx=0x7fc3a402b8f0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f0020800640 1 -- 172.21.15.27:0/1666317656 shutdown_connections 2024-03-28T18:55:53.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f0020800640 1 -- 172.21.15.27:0/1666317656 wait complete. 2024-03-28T18:55:53.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f1911e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190c0ac150 con 0x7f190c1662a0 2024-03-28T18:55:53.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f1911e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c0733f0 con 0x7f190c0aa300 2024-03-28T18:55:53.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f1911e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c0c12d0 con 0x7f190c0c6760 2024-03-28T18:55:53.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f190a000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f190c0c6760 0x7f190c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f190b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f190c0aa300 0x7f190c0c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f190b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f190c0aa300 0x7f190c0c3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59422/0 (socket says 172.21.15.27:59422) 2024-03-28T18:55:53.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f190b400640 1 -- 172.21.15.27:0/3000324867 learned_addr learned my addr 172.21.15.27:0/3000324867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f190b400640 1 -- 172.21.15.27:0/3000324867 >> v2:172.21.15.155:3300/0 conn(0x7f190c0c6760 msgr2=0x7f190c0c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f190b400640 1 --2- 172.21.15.27:0/3000324867 >> v2:172.21.15.155:3300/0 conn(0x7f190c0c6760 0x7f190c0c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f750be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75041801c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f750be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7504050720 0x7f75040dc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.577+0000 7f750be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75040d7920 0x7f75040dc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f750be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7504006f10 con 0x7f7504050720 2024-03-28T18:55:53.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f750aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75041801c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f750be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750404c2a0 con 0x7f75040d7920 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f750be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75040dad20 con 0x7f750400fcf0 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f750aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75041801c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45096/0 (socket says 172.21.15.27:45096) 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f750aa00640 1 -- 172.21.15.27:0/2519190861 learned_addr learned my addr 172.21.15.27:0/2519190861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.578+0000 7f750b400640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75040d7920 0x7f75040dc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f750a000640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7504050720 0x7f75040dc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f750a000640 1 -- 172.21.15.27:0/2519190861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 msgr2=0x7f75041801c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f750a000640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f750400fcf0 0x7f75041801c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f750a000640 1 -- 172.21.15.27:0/2519190861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75040d7920 msgr2=0x7f75040dc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.579+0000 7f750a000640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75040d7920 0x7f75040dc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.580+0000 7f750a000640 1 -- 172.21.15.27:0/2519190861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750400d800 con 0x7f7504050720 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.580+0000 7f750a000640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7504050720 0x7f75040dc0d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f75040096e0 tx=0x7f74fc00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.581+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0074e0 con 0x7f7504050720 2024-03-28T18:55:53.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.581+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7504180a90 con 0x7f7504050720 2024-03-28T18:55:53.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.581+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75040d9c80 con 0x7f7504050720 2024-03-28T18:55:53.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.581+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 <== osd.0 v2:172.21.15.27:6815/1506376930 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0784079130 con 0x7f0784076660 2024-03-28T18:55:53.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.581+0000 7fc3b7800640 1 -- 172.21.15.27:0/2247855791 shutdown_connections 2024-03-28T18:55:53.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.581+0000 7fc3b7800640 1 --2- 172.21.15.27:0/2247855791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.581+0000 7f0020800640 1 Processor -- start 2024-03-28T18:55:53.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.582+0000 7f0020800640 1 -- start start 2024-03-28T18:55:53.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.580+0000 7f190b400640 1 -- 172.21.15.27:0/3000324867 >> v1:172.21.15.27:6789/0 conn(0x7f190c1662a0 legacy=0x7f190c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.580+0000 7f190b400640 1 -- 172.21.15.27:0/3000324867 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c073640 con 0x7f190c0aa300 2024-03-28T18:55:53.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.582+0000 7fc3b7800640 1 --2- 172.21.15.27:0/2247855791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 0x7fc3b00c58e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.582+0000 7fc3b7800640 1 -- 172.21.15.27:0/2247855791 >> 172.21.15.27:0/2247855791 conn(0x7fc3b006b6f0 msgr2=0x7fc3b00caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.581+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc0047e0 con 0x7f7504050720 2024-03-28T18:55:53.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.582+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0074e0 con 0x7f7504050720 2024-03-28T18:55:53.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.583+0000 7f0020800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001806cc40 0x7f00180c2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.583+0000 7f0020800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00180c4000 0x7f00181fb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.583+0000 7f0020800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00181508a0 0x7f001814f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.584+0000 7f001d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00180c4000 0x7f00181fb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.584+0000 7f0020800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0018150e00 con 0x7f00180c4000 2024-03-28T18:55:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.584+0000 7f0020800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0018150c80 con 0x7f00181508a0 2024-03-28T18:55:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.584+0000 7f0020800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00181539e0 con 0x7f001806cc40 2024-03-28T18:55:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.584+0000 7f001d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00180c4000 0x7f00181fb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59442/0 (socket says 172.21.15.27:59442) 2024-03-28T18:55:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.584+0000 7f001d400640 1 -- 172.21.15.27:0/3881620964 learned_addr learned my addr 172.21.15.27:0/3881620964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.584+0000 7f001de00640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001806cc40 0x7f00180c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.584+0000 7fc3b7800640 1 -- 172.21.15.27:0/2247855791 shutdown_connections 2024-03-28T18:55:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.585+0000 7f001d400640 1 -- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001806cc40 msgr2=0x7f00180c2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.585+0000 7f001d400640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f001806cc40 0x7f00180c2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.585+0000 7f001d400640 1 -- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00181508a0 msgr2=0x7f001814f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.585+0000 7f001d400640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00181508a0 0x7f001814f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.585+0000 7f001d400640 1 -- 172.21.15.27:0/3881620964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180a8030 con 0x7f00180c4000 2024-03-28T18:55:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.585+0000 7f190b400640 1 --2- 172.21.15.27:0/3000324867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 0x7f190c0c3bd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f190c0c42c0 tx=0x7f18fc02bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69dbe8ca21b25d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.586+0000 7fc3b7800640 1 -- 172.21.15.27:0/2247855791 wait complete. 2024-03-28T18:55:53.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.586+0000 7f1909600640 1 -- 172.21.15.27:0/3000324867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc02c870 con 0x7f190c0aa300 2024-03-28T18:55:53.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.587+0000 7f1909600640 1 -- 172.21.15.27:0/3000324867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18fc02ca10 con 0x7f190c0aa300 2024-03-28T18:55:53.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.587+0000 7f1909600640 1 -- 172.21.15.27:0/3000324867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc02c870 con 0x7f190c0aa300 2024-03-28T18:55:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.588+0000 7f1911e00640 1 -- 172.21.15.27:0/3000324867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 msgr2=0x7f190c0c3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.588+0000 7f1911e00640 1 --2- 172.21.15.27:0/3000324867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 0x7f190c0c3bd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f190c0c42c0 tx=0x7f18fc02bd50 comp rx=0 tx=0).stop 2024-03-28T18:55:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.588+0000 7f1911e00640 1 -- 172.21.15.27:0/3000324867 shutdown_connections 2024-03-28T18:55:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.586+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f74fc011070 con 0x7f7504050720 2024-03-28T18:55:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.588+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74e4001030 con 0x7f7504050720 2024-03-28T18:55:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.588+0000 7f7502a00640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74ec06bfd0 0x7f74ec06e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.589+0000 7f1911e00640 1 --2- 172.21.15.27:0/3000324867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 0x7f190c0c3bd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.589+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f74fc0072e0 con 0x7f7504050720 2024-03-28T18:55:53.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.589+0000 7f1911e00640 1 --2- 172.21.15.27:0/3000324867 >> v2:172.21.15.155:3300/0 conn(0x7f190c0c6760 0x7f190c0c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.589+0000 7f1911e00640 1 -- 172.21.15.27:0/3000324867 >> 172.21.15.27:0/3000324867 conn(0x7f190c0c6c00 msgr2=0x7f190c0c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.589+0000 7f7502a00640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6808/2953172651,v1:172.21.15.27:6809/2953172651] conn(0x7f74ec0766e0 0x7f74ec078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.587+0000 7f001d400640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00180c4000 0x7f00181fb500 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0008000c00 tx=0x7f00080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.589+0000 7f1911e00640 1 -- 172.21.15.27:0/3000324867 shutdown_connections 2024-03-28T18:55:53.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.589+0000 7f1911e00640 1 -- 172.21.15.27:0/3000324867 wait complete. 2024-03-28T18:55:53.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.589+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008040070 con 0x7f00180c4000 2024-03-28T18:55:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.590+0000 7f750aa00640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74ec06bfd0 0x7f74ec06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.590+0000 7fc3b7800640 1 Processor -- start 2024-03-28T18:55:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.590+0000 7fc3b7800640 1 -- start start 2024-03-28T18:55:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.591+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 --> [v2:172.21.15.27:6808/2953172651,v1:172.21.15.27:6809/2953172651] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74fc01fe50 con 0x7f74ec0766e0 2024-03-28T18:55:53.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.591+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74fc091050 con 0x7f7504050720 2024-03-28T18:55:53.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.590+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000803a420 con 0x7f00180c4000 2024-03-28T18:55:53.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.591+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0008035040 con 0x7f00180c4000 2024-03-28T18:55:53.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.591+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00180a7d50 con 0x7f00180c4000 2024-03-28T18:55:53.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.591+0000 7f750b400640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6808/2953172651,v1:172.21.15.27:6809/2953172651] conn(0x7f74ec0766e0 0x7f74ec078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.591+0000 7f750aa00640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74ec06bfd0 0x7f74ec06e490 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009a60 tx=0x7f74f8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.591+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00180a6b80 con 0x7f00180c4000 2024-03-28T18:55:53.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.592+0000 7fc3b7800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.593+0000 7f8128c00640 1 Processor -- start 2024-03-28T18:55:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.593+0000 7fc3b7800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 0x7fc3b014db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.592+0000 7f750b400640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6808/2953172651,v1:172.21.15.27:6809/2953172651] conn(0x7f74ec0766e0 0x7f74ec078b00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.593+0000 7f8128c00640 1 -- start start 2024-03-28T18:55:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.593+0000 7fc3b7800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3b0155730 0x7fc3b01f8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.593+0000 7fc3b4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.593+0000 7fc3b4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45108/0 (socket says 172.21.15.27:45108) 2024-03-28T18:55:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.593+0000 7fc3b4e00640 1 -- 172.21.15.27:0/1493178945 learned_addr learned my addr 172.21.15.27:0/1493178945 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.593+0000 7fc3afe00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 0x7fc3b014db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.594+0000 7fc3b7800640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0159b00 con 0x7fc3b0152f80 2024-03-28T18:55:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.594+0000 7fc3b7800640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00c5e20 con 0x7fc3b0155730 2024-03-28T18:55:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.594+0000 7fc3b7800640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00cbca0 con 0x7fc3b00c7970 2024-03-28T18:55:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.594+0000 7f1911e00640 1 Processor -- start 2024-03-28T18:55:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.594+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 <== osd.1 v2:172.21.15.27:6808/2953172651 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f74fc01fe50 con 0x7f74ec0766e0 2024-03-28T18:55:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.594+0000 7fc3b5800640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3b0155730 0x7fc3b01f8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.594+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f001804f400 con 0x7f00180c4000 2024-03-28T18:55:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f8128c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8124152f80 0x7f81240c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f8128c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81240c7df0 0x7f8124155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f8128c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81240c7310 con 0x7f81240c7970 2024-03-28T18:55:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f8128c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8124158d40 con 0x7f81240c7df0 2024-03-28T18:55:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f8128c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8124159020 con 0x7f8124152f80 2024-03-28T18:55:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7fc3b4e00640 1 -- 172.21.15.27:0/1493178945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3b0155730 msgr2=0x7fc3b01f8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7fc3b4e00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3b0155730 0x7fc3b01f8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7fc3b4e00640 1 -- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 msgr2=0x7fc3b014db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f8121400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f81240c7970 0x7f8124155730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55582/0 (socket says 172.21.15.27:55582) 2024-03-28T18:55:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f8121400640 1 -- 172.21.15.27:0/1763383721 learned_addr learned my addr 172.21.15.27:0/1763383721 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f8122800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81240c7df0 0x7f8124155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.596+0000 7f8121e00640 1 --2- 172.21.15.27:0/1763383721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8124152f80 0x7f81240c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.596+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f000804b060 con 0x7f00180c4000 2024-03-28T18:55:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.595+0000 7f1911e00640 1 -- start start 2024-03-28T18:55:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.598+0000 7f0012000640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efffc06bca0 0x7efffc06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.598+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f00080accb0 con 0x7f00180c4000 2024-03-28T18:55:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.597+0000 7f8121e00640 1 -- 172.21.15.27:0/1763383721 >> v2:172.21.15.27:3300/0 conn(0x7f81240c7df0 msgr2=0x7f8124155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.597+0000 7f8121e00640 1 --2- 172.21.15.27:0/1763383721 >> v2:172.21.15.27:3300/0 conn(0x7f81240c7df0 0x7f8124155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.597+0000 7f8121e00640 1 -- 172.21.15.27:0/1763383721 >> v1:172.21.15.155:6789/0 conn(0x7f81240c7970 legacy=0x7f8124155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.597+0000 7f8121e00640 1 -- 172.21.15.27:0/1763383721 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8124159300 con 0x7f8124152f80 2024-03-28T18:55:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.598+0000 7f1911e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 0x7f190c0728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.598+0000 7f1911e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f190c0c6760 0x7f190c1f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.598+0000 7f1911e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f190c1662a0 0x7f190c1f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f1911e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0abb00 con 0x7f190c0aa300 2024-03-28T18:55:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f1911e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0a9de0 con 0x7f190c1662a0 2024-03-28T18:55:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f0012000640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:6800/998687689,v1:172.21.15.155:6801/998687689] conn(0x7efffc076530 0x7efffc078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7fc3b4e00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 0x7fc3b014db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7fc3b4e00640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b00c1d80 con 0x7fc3b00c7970 2024-03-28T18:55:53.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7fc3afe00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3b0152f80 0x7fc3b014db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 --> [v2:172.21.15.155:6800/998687689,v1:172.21.15.155:6801/998687689] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0008032e50 con 0x7efffc076530 2024-03-28T18:55:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00080af310 con 0x7f00180c4000 2024-03-28T18:55:53.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f1911e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0a6370 con 0x7f190c0c6760 2024-03-28T18:55:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f190a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f190c0c6760 0x7f190c1f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f190b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f190c1662a0 0x7f190c1f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f190a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f190c0c6760 0x7f190c1f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45130/0 (socket says 172.21.15.27:45130) 2024-03-28T18:55:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f190a000640 1 -- 172.21.15.27:0/1186461629 learned_addr learned my addr 172.21.15.27:0/1186461629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f190aa00640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 0x7f190c0728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f8121e00640 1 --2- 172.21.15.27:0/1763383721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8124152f80 0x7f81240c58e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f811c009d90 tx=0x7f811c02be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d896be59bf1ab80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.600+0000 7fc3b4e00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c17a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc3a402b930 tx=0x7fc3a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.600+0000 7f190aa00640 1 -- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f190c0c6760 msgr2=0x7f190c1f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.600+0000 7f190aa00640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f190c0c6760 0x7f190c1f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.600+0000 7f190aa00640 1 -- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f190c1662a0 msgr2=0x7f190c1f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.600+0000 7f190aa00640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f190c1662a0 0x7f190c1f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.600+0000 7f190aa00640 1 -- 172.21.15.27:0/1186461629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c1f7d80 con 0x7f190c0aa300 2024-03-28T18:55:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.600+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a403aa60 con 0x7fc3b00c7970 2024-03-28T18:55:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a403ac00 con 0x7fc3b00c7970 2024-03-28T18:55:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7fc3b7800640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b00c20d0 con 0x7fc3b00c7970 2024-03-28T18:55:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7fc3b7800640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b00c2690 con 0x7fc3b00c7970 2024-03-28T18:55:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a40443f0 con 0x7fc3b00c7970 2024-03-28T18:55:53.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.602+0000 7f190aa00640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 0x7f190c0728f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f190400c780 tx=0x7f190400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f8120a00640 1 -- 172.21.15.27:0/1763383721 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c039070 con 0x7f8124152f80 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f8120a00640 1 -- 172.21.15.27:0/1763383721 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c02cbe0 con 0x7f8124152f80 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.599+0000 7f8120a00640 1 -- 172.21.15.27:0/1763383721 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c034440 con 0x7f8124152f80 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7f8128c00640 1 -- 172.21.15.27:0/1763383721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8124152f80 msgr2=0x7f81240c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7f8128c00640 1 --2- 172.21.15.27:0/1763383721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8124152f80 0x7f81240c58e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f811c009d90 tx=0x7f811c02be30 comp rx=0 tx=0).stop 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7f8128c00640 1 -- 172.21.15.27:0/1763383721 shutdown_connections 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7f8128c00640 1 --2- 172.21.15.27:0/1763383721 >> v2:172.21.15.27:3300/0 conn(0x7f81240c7df0 0x7f8124155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7f8128c00640 1 --2- 172.21.15.27:0/1763383721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8124152f80 0x7f81240c58e0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f811c009d90 tx=0x7f811c02be30 comp rx=0 tx=0).stop 2024-03-28T18:55:53.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7f8128c00640 1 -- 172.21.15.27:0/1763383721 >> 172.21.15.27:0/1763383721 conn(0x7f812406b6f0 msgr2=0x7f81240caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.601+0000 7f001de00640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efffc06bca0 0x7efffc06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.603+0000 7f001e800640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:6800/998687689,v1:172.21.15.155:6801/998687689] conn(0x7efffc076530 0x7efffc078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.603+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1904017070 con 0x7f190c0aa300 2024-03-28T18:55:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.603+0000 7f8128c00640 1 -- 172.21.15.27:0/1763383721 shutdown_connections 2024-03-28T18:55:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.603+0000 7f8128c00640 1 -- 172.21.15.27:0/1763383721 wait complete. 2024-03-28T18:55:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.604+0000 7fc3b7800640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc38c001030 con 0x7fc3b00c7970 2024-03-28T18:55:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.603+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190400ce30 con 0x7f190c0aa300 2024-03-28T18:55:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.603+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190400a860 con 0x7f190c0aa300 2024-03-28T18:55:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.603+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c1f8060 con 0x7f190c0aa300 2024-03-28T18:55:53.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.603+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c1f8620 con 0x7f190c0aa300 2024-03-28T18:55:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.605+0000 7f001e800640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:6800/998687689,v1:172.21.15.155:6801/998687689] conn(0x7efffc076530 0x7efffc078950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.605+0000 7f001de00640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efffc06bca0 0x7efffc06e160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f000c000f60 tx=0x7f000c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.605+0000 7f8128c00640 1 Processor -- start 2024-03-28T18:55:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.606+0000 7f8128c00640 1 -- start start 2024-03-28T18:55:53.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.607+0000 7f8128c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81240c7970 0x7f8124145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.607+0000 7f8128c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81240c7df0 0x7f8124145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.607+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc3a404b060 con 0x7fc3b00c7970 2024-03-28T18:55:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.607+0000 7f8128c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8124146310 0x7f8124147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.607+0000 7f8128c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8124159b00 con 0x7f81240c7970 2024-03-28T18:55:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.607+0000 7f8128c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240c5e20 con 0x7f8124146310 2024-03-28T18:55:53.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.607+0000 7f8128c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81240cbca0 con 0x7f81240c7df0 2024-03-28T18:55:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.608+0000 7f609be00640 1 Processor -- start 2024-03-28T18:55:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.608+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f190c1f8990 con 0x7f190c0aa300 2024-03-28T18:55:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.609+0000 7f609be00640 1 -- start start 2024-03-28T18:55:53.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.608+0000 7f8121400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81240c7df0 0x7f8124145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.608+0000 7f8121e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81240c7970 0x7f8124145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.609+0000 7fc3acc00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc39006bca0 0x7fc39006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.609+0000 7f8121400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81240c7df0 0x7f8124145dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45138/0 (socket says 172.21.15.27:45138) 2024-03-28T18:55:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.609+0000 7f8121400640 1 -- 172.21.15.27:0/2943006779 learned_addr learned my addr 172.21.15.27:0/2943006779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.609+0000 7f8122800640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8124146310 0x7f8124147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 <== osd.4 v2:172.21.15.155:6800/998687689 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0008032e50 con 0x7efffc076530 2024-03-28T18:55:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7f8121e00640 1 -- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81240c7df0 msgr2=0x7f8124145dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.609+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fc3a40acf20 con 0x7fc3b00c7970 2024-03-28T18:55:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7fc3acc00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6824/150528886,v1:172.21.15.27:6825/150528886] conn(0x7fc390076530 0x7fc390078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.27:6824/150528886,v1:172.21.15.27:6825/150528886] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3a4033ec0 con 0x7fc390076530 2024-03-28T18:55:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3a40ad310 con 0x7fc3b00c7970 2024-03-28T18:55:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7f8121e00640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81240c7df0 0x7f8124145dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7f8121e00640 1 -- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8124146310 msgr2=0x7f8124147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7f8121e00640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8124146310 0x7f8124147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.610+0000 7f8121e00640 1 -- 172.21.15.27:0/2943006779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81240c3d50 con 0x7f81240c7970 2024-03-28T18:55:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.611+0000 7fc3afe00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc39006bca0 0x7fc39006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.611+0000 7fc3b5800640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6824/150528886,v1:172.21.15.27:6825/150528886] conn(0x7fc390076530 0x7fc390078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.611+0000 7f8121e00640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81240c7970 0x7f8124145890 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f811c033000 tx=0x7f811c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.612+0000 7f609be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60940d3c40 0x7f609404ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.612+0000 7f609be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 0x7f60940d6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.613+0000 7f609be00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60940468d0 con 0x7f60940d6fa0 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.613+0000 7f609aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60940d3c40 0x7f609404ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.613+0000 7f609b400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f60940d6fa0 0x7f60940d99a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55590/0 (socket says 172.21.15.27:55590) 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.613+0000 7f609b400640 1 -- 172.21.15.27:0/2500397648 learned_addr learned my addr 172.21.15.27:0/2500397648 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.613+0000 7f609be00640 1 -- 172.21.15.27:0/2500397648 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6094047660 con 0x7f60940d3c40 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.613+0000 7f609a000640 1 --2- 172.21.15.27:0/2500397648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 0x7f60940d6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.613+0000 7f609be00640 1 -- 172.21.15.27:0/2500397648 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60940d65d0 con 0x7f609404b1b0 2024-03-28T18:55:53.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.614+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c044020 con 0x7f81240c7970 2024-03-28T18:55:53.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.614+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c034e50 con 0x7f81240c7970 2024-03-28T18:55:53.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.614+0000 7f609aa00640 1 -- 172.21.15.27:0/2500397648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 msgr2=0x7f60940d6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.614+0000 7f609aa00640 1 --2- 172.21.15.27:0/2500397648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 0x7f60940d6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.615+0000 7f609aa00640 1 -- 172.21.15.27:0/2500397648 >> v1:172.21.15.155:6789/0 conn(0x7f60940d6fa0 legacy=0x7f60940d99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.614+0000 7fc3afe00640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc39006bca0 0x7fc39006e160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00cb8d0 tx=0x7fc39c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.615+0000 7f609a000640 1 --2- 172.21.15.27:0/2500397648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 0x7f60940d6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:53.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.615+0000 7f609aa00640 1 -- 172.21.15.27:0/2500397648 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094046bb0 con 0x7f60940d3c40 2024-03-28T18:55:53.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.615+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c044020 con 0x7f81240c7970 2024-03-28T18:55:53.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.615+0000 7f8128c00640 1 -- 172.21.15.27:0/2943006779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81240c4350 con 0x7f81240c7970 2024-03-28T18:55:53.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.616+0000 7f8128c00640 1 -- 172.21.15.27:0/2943006779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81240c4910 con 0x7f81240c7970 2024-03-28T18:55:53.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.617+0000 7f609aa00640 1 --2- 172.21.15.27:0/2500397648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d3c40 0x7f609404ac70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f608c0097a0 tx=0x7f608c02b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7c036f4f9dde56 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.618+0000 7fc3b5800640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6824/150528886,v1:172.21.15.27:6825/150528886] conn(0x7fc390076530 0x7fc390078950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.618+0000 7f6099600640 1 -- 172.21.15.27:0/2500397648 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c037070 con 0x7f60940d3c40 2024-03-28T18:55:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.618+0000 7f6099600640 1 -- 172.21.15.27:0/2500397648 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c02be50 con 0x7f60940d3c40 2024-03-28T18:55:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.618+0000 7f6099600640 1 -- 172.21.15.27:0/2500397648 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c030930 con 0x7f60940d3c40 2024-03-28T18:55:53.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.619+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f190401a070 con 0x7f190c0aa300 2024-03-28T18:55:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.618+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80f8001030 con 0x7f81240c7970 2024-03-28T18:55:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.620+0000 7f609be00640 1 -- 172.21.15.27:0/2500397648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d3c40 msgr2=0x7f609404ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.620+0000 7f609be00640 1 --2- 172.21.15.27:0/2500397648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d3c40 0x7f609404ac70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f608c0097a0 tx=0x7f608c02b650 comp rx=0 tx=0).stop 2024-03-28T18:55:53.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.620+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 <== osd.2 v2:172.21.15.27:6824/150528886 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc3a4033ec0 con 0x7fc390076530 2024-03-28T18:55:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.619+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 --> [v2:172.21.15.27:6815/1506376930,v1:172.21.15.27:6817/1506376930] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0780002dd0 con 0x7f0784076660 2024-03-28T18:55:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.623+0000 7f07a1600640 1 -- 172.21.15.27:0/4132524790 <== osd.0 v2:172.21.15.27:6815/1506376930 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0780002dd0 con 0x7f0784076660 2024-03-28T18:55:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.620+0000 7f609be00640 1 -- 172.21.15.27:0/2500397648 shutdown_connections 2024-03-28T18:55:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.620+0000 7f609be00640 1 --2- 172.21.15.27:0/2500397648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 0x7f60940d6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.620+0000 7f609be00640 1 --2- 172.21.15.27:0/2500397648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d3c40 0x7f609404ac70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.620+0000 7f609be00640 1 -- 172.21.15.27:0/2500397648 >> 172.21.15.27:0/2500397648 conn(0x7f6094005e90 msgr2=0x7f60940062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.621+0000 7f609be00640 1 -- 172.21.15.27:0/2500397648 shutdown_connections 2024-03-28T18:55:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.622+0000 7f609be00640 1 -- 172.21.15.27:0/2500397648 wait complete. 2024-03-28T18:55:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.621+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f811c047020 con 0x7f81240c7970 2024-03-28T18:55:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.624+0000 7f811a000640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f810406c000 0x7f810406e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.624+0000 7f1903400640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18ec06bca0 0x7f18ec06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.624+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f811c02c9e0 con 0x7f81240c7970 2024-03-28T18:55:53.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.624+0000 7f609be00640 1 Processor -- start 2024-03-28T18:55:53.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f609be00640 1 -- start start 2024-03-28T18:55:53.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f190408a2e0 con 0x7f190c0aa300 2024-03-28T18:55:53.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f811a000640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:6808/330629214,v1:172.21.15.155:6809/330629214] conn(0x7f81040767a0 0x7f8104078bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 --> [v2:172.21.15.155:6808/330629214,v1:172.21.15.155:6809/330629214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f811c03de50 con 0x7f81040767a0 2024-03-28T18:55:53.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f811c0af4c0 con 0x7f81240c7970 2024-03-28T18:55:53.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f1903400640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:6816/2908350031,v1:172.21.15.155:6817/2908350031] conn(0x7f18ec076530 0x7f18ec078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6815/1506376930,v1:172.21.15.27:6817/1506376930] conn(0x7f0784076660 msgr2=0x7f0784078a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f07ac600640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6815/1506376930,v1:172.21.15.27:6817/1506376930] conn(0x7f0784076660 0x7f0784078a80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 --> [v2:172.21.15.155:6816/2908350031,v1:172.21.15.155:6817/2908350031] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18ec079000 con 0x7f18ec076530 2024-03-28T18:55:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1904091050 con 0x7f190c0aa300 2024-03-28T18:55:53.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f078406bdd0 msgr2=0x7f078406e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f07ac600640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f078406bdd0 0x7f078406e290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0790036000 tx=0x7f0790008030 comp rx=0 tx=0).stop 2024-03-28T18:55:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a4166570 msgr2=0x7f07a415fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f07ac600640 1 --2- 172.21.15.27:0/4132524790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07a4166570 0x7f07a415fc30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f079c00be50 tx=0x7f079c00bf40 comp rx=0 tx=0).stop 2024-03-28T18:55:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f8122800640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:6808/330629214,v1:172.21.15.155:6809/330629214] conn(0x7f81040767a0 0x7f8104078bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.625+0000 7f07aa600640 1 -- 172.21.15.27:0/4132524790 reap_dead start 2024-03-28T18:55:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 shutdown_connections 2024-03-28T18:55:53.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 >> 172.21.15.27:0/4132524790 conn(0x7f07a40c6c00 msgr2=0x7f07a4170460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f190b400640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:6816/2908350031,v1:172.21.15.155:6817/2908350031] conn(0x7f18ec076530 0x7f18ec078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 shutdown_connections 2024-03-28T18:55:53.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f07ac600640 1 -- 172.21.15.27:0/4132524790 wait complete. 2024-03-28T18:55:53.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f8121400640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f810406c000 0x7f810406e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f190a000640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18ec06bca0 0x7f18ec06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f609be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 0x7f60940d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f609be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60940d3c40 0x7f6094044ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.626+0000 7f609be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d6fa0 0x7f609415f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.627+0000 7f609be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6094046360 con 0x7f60940d6fa0 2024-03-28T18:55:53.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.627+0000 7f609be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60940461e0 con 0x7f60940d3c40 2024-03-28T18:55:53.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.627+0000 7f609be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6094046e90 con 0x7f609404b1b0 2024-03-28T18:55:53.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.627+0000 7f190b400640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:6816/2908350031,v1:172.21.15.155:6817/2908350031] conn(0x7f18ec076530 0x7f18ec078950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.627+0000 7f609a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60940d3c40 0x7f6094044ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f190a000640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18ec06bca0 0x7f18ec06e160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18f8013fd0 tx=0x7f18f8015040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f609b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d6fa0 0x7f609415f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f609a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60940d3c40 0x7f6094044ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43542/0 (socket says 172.21.15.27:43542) 2024-03-28T18:55:53.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f609a000640 1 -- 172.21.15.27:0/1373690621 learned_addr learned my addr 172.21.15.27:0/1373690621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f8122800640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:6808/330629214,v1:172.21.15.155:6809/330629214] conn(0x7f81040767a0 0x7f8104078bc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f8121400640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f810406c000 0x7f810406e4c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8124151280 tx=0x7f8114009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f609b400640 1 -- 172.21.15.27:0/1373690621 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 msgr2=0x7f60940d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f609b400640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f609404b1b0 0x7f60940d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f609b400640 1 -- 172.21.15.27:0/1373690621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60940d3c40 msgr2=0x7f6094044ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f609b400640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60940d3c40 0x7f6094044ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.628+0000 7f609b400640 1 -- 172.21.15.27:0/1373690621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60940448b0 con 0x7f60940d6fa0 2024-03-28T18:55:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.629+0000 7f609b400640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d6fa0 0x7f609415f9b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f609c064b90 tx=0x7f609c068c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.629+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c0734c0 con 0x7f60940d6fa0 2024-03-28T18:55:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.629+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 <== osd.5 v2:172.21.15.155:6816/2908350031 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f18ec079000 con 0x7f18ec076530 2024-03-28T18:55:53.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.629+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c0736c0 con 0x7f60940d6fa0 2024-03-28T18:55:53.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.629+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c0719d0 con 0x7f60940d6fa0 2024-03-28T18:55:53.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.630+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60940d0be0 con 0x7f60940d6fa0 2024-03-28T18:55:53.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.630+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609416d9c0 con 0x7f60940d6fa0 2024-03-28T18:55:53.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.633+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 <== osd.6 v2:172.21.15.155:6808/330629214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f811c03de50 con 0x7f81040767a0 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.631+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6074001030 con 0x7f60940d6fa0 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.632+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f609c071b70 con 0x7f60940d6fa0 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.634+0000 7f6093400640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f607c06bbd0 0x7f607c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.634+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f609c1061c0 con 0x7f60940d6fa0 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.634+0000 7f6093400640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6800/974955781,v1:172.21.15.27:6801/974955781] conn(0x7f607c076460 0x7f607c078880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.634+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 --> [v2:172.21.15.27:6800/974955781,v1:172.21.15.27:6801/974955781] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f609c097e50 con 0x7f607c076460 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.634+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f609c10d050 con 0x7f60940d6fa0 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.637+0000 7f609aa00640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f607c06bbd0 0x7f607c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.639+0000 7f609a000640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6800/974955781,v1:172.21.15.27:6801/974955781] conn(0x7f607c076460 0x7f607c078880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.639+0000 7f609a000640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6800/974955781,v1:172.21.15.27:6801/974955781] conn(0x7f607c076460 0x7f607c078880 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.642+0000 7f609aa00640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f607c06bbd0 0x7f607c06e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f608c034000 tx=0x7f608c02f000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.642+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 <== osd.3 v2:172.21.15.27:6800/974955781 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f609c097e50 con 0x7f607c076460 2024-03-28T18:55:53.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.643+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 --> [v2:172.21.15.27:6808/2953172651,v1:172.21.15.27:6809/2953172651] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f74e4002dd0 con 0x7f74ec0766e0 2024-03-28T18:55:53.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.646+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 --> [v2:172.21.15.155:6800/998687689,v1:172.21.15.155:6801/998687689] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f00180c5f20 con 0x7efffc076530 2024-03-28T18:55:53.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.647+0000 7f7502a00640 1 -- 172.21.15.27:0/2519190861 <== osd.1 v2:172.21.15.27:6808/2953172651 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f74e4002dd0 con 0x7f74ec0766e0 2024-03-28T18:55:53.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.647+0000 7f0012000640 1 -- 172.21.15.27:0/3881620964 <== osd.4 v2:172.21.15.155:6800/998687689 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f00180c5f20 con 0x7efffc076530 2024-03-28T18:55:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.648+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6808/2953172651,v1:172.21.15.27:6809/2953172651] conn(0x7f74ec0766e0 msgr2=0x7f74ec078b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.648+0000 7f750be00640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6808/2953172651,v1:172.21.15.27:6809/2953172651] conn(0x7f74ec0766e0 0x7f74ec078b00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.648+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74ec06bfd0 msgr2=0x7f74ec06e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.648+0000 7f750be00640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74ec06bfd0 0x7f74ec06e490 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009a60 tx=0x7f74f8008040 comp rx=0 tx=0).stop 2024-03-28T18:55:53.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.649+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7504050720 msgr2=0x7f75040dc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.649+0000 7f750be00640 1 --2- 172.21.15.27:0/2519190861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7504050720 0x7f75040dc0d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f75040096e0 tx=0x7f74fc00c870 comp rx=0 tx=0).stop 2024-03-28T18:55:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.649+0000 7f750b400640 1 -- 172.21.15.27:0/2519190861 reap_dead start 2024-03-28T18:55:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.649+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 shutdown_connections 2024-03-28T18:55:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.649+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 >> 172.21.15.27:0/2519190861 conn(0x7f750404fc00 msgr2=0x7f75040d5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.650+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 shutdown_connections 2024-03-28T18:55:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.650+0000 7f750be00640 1 -- 172.21.15.27:0/2519190861 wait complete. 2024-03-28T18:55:53.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.651+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:6800/998687689,v1:172.21.15.155:6801/998687689] conn(0x7efffc076530 msgr2=0x7efffc078950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.651+0000 7f0020800640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.155:6800/998687689,v1:172.21.15.155:6801/998687689] conn(0x7efffc076530 0x7efffc078950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.652+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efffc06bca0 msgr2=0x7efffc06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.652+0000 7f0020800640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efffc06bca0 0x7efffc06e160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f000c000f60 tx=0x7f000c009040 comp rx=0 tx=0).stop 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.652+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00180c4000 msgr2=0x7f00181fb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.652+0000 7f0020800640 1 --2- 172.21.15.27:0/3881620964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00180c4000 0x7f00181fb500 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0008000c00 tx=0x7f00080046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.652+0000 7f001e800640 1 -- 172.21.15.27:0/3881620964 reap_dead start 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.652+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 shutdown_connections 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.652+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 >> 172.21.15.27:0/3881620964 conn(0x7f00180c6c00 msgr2=0x7f00180c63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.652+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 shutdown_connections 2024-03-28T18:55:53.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.654+0000 7f0020800640 1 -- 172.21.15.27:0/3881620964 wait complete. 2024-03-28T18:55:53.655 INFO:teuthology.orchestra.run.smithi027.stdout:55834574859 2024-03-28T18:55:53.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T18:55:53.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.662+0000 7fc3b7800640 1 -- 172.21.15.27:0/1493178945 --> [v2:172.21.15.27:6824/150528886,v1:172.21.15.27:6825/150528886] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc38c002dd0 con 0x7fc390076530 2024-03-28T18:55:53.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.664+0000 7fc3acc00640 1 -- 172.21.15.27:0/1493178945 <== osd.2 v2:172.21.15.27:6824/150528886 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc38c002dd0 con 0x7fc390076530 2024-03-28T18:55:53.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.665+0000 7fc3a2000640 1 -- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6824/150528886,v1:172.21.15.27:6825/150528886] conn(0x7fc390076530 msgr2=0x7fc390078950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.665+0000 7fc3a2000640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6824/150528886,v1:172.21.15.27:6825/150528886] conn(0x7fc390076530 0x7fc390078950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7fc3a2000640 1 -- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc39006bca0 msgr2=0x7fc39006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7fc3a2000640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc39006bca0 0x7fc39006e160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00cb8d0 tx=0x7fc39c009190 comp rx=0 tx=0).stop 2024-03-28T18:55:53.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7fc3a2000640 1 -- 172.21.15.27:0/1493178945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 msgr2=0x7fc3b00c17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7fc3a2000640 1 --2- 172.21.15.27:0/1493178945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3b00c7970 0x7fc3b00c17a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc3a402b930 tx=0x7fc3a40046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7fc3b5800640 1 -- 172.21.15.27:0/1493178945 reap_dead start 2024-03-28T18:55:53.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.664+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 --> [v2:172.21.15.155:6816/2908350031,v1:172.21.15.155:6817/2908350031] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f190c04f400 con 0x7f18ec076530 2024-03-28T18:55:53.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7f1903400640 1 -- 172.21.15.27:0/1186461629 <== osd.5 v2:172.21.15.155:6816/2908350031 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f190c04f400 con 0x7f18ec076530 2024-03-28T18:55:53.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.662+0000 7f84e9e00640 1 Processor -- start 2024-03-28T18:55:53.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.662+0000 7f84e9e00640 1 -- start start 2024-03-28T18:55:53.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.664+0000 7f84e9e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c8db0 0x7f84e4156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.664+0000 7f84e9e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e40735f0 con 0x7f84e4152840 2024-03-28T18:55:53.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.664+0000 7f84e9e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e40c7d10 con 0x7f84e40c5aa0 2024-03-28T18:55:53.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.664+0000 7f84e9e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e41565d0 con 0x7f84e40c8db0 2024-03-28T18:55:53.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.664+0000 7f84e3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c8db0 0x7f84e4156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.665+0000 7f84e8a00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f84e40c5aa0 0x7f84e4072620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55604/0 (socket says 172.21.15.27:55604) 2024-03-28T18:55:53.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.665+0000 7f84e8a00640 1 -- 172.21.15.27:0/2036188141 learned_addr learned my addr 172.21.15.27:0/2036188141 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:53.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7f84e3400640 1 -- 172.21.15.27:0/2036188141 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464374333 0 0) 0x7f84e40735f0 con 0x7f84e4152840 2024-03-28T18:55:53.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7f84e3e00640 1 -- 172.21.15.27:0/2036188141 >> v1:172.21.15.155:6789/0 conn(0x7f84e40c5aa0 legacy=0x7f84e4072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7f84e3e00640 1 -- 172.21.15.27:0/2036188141 >> v1:172.21.15.27:6789/0 conn(0x7f84e4152840 legacy=0x7f84e415ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.667+0000 7fc3a2000640 1 -- 172.21.15.27:0/1493178945 shutdown_connections 2024-03-28T18:55:53.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.667+0000 7fc3a2000640 1 -- 172.21.15.27:0/1493178945 >> 172.21.15.27:0/1493178945 conn(0x7fc3b006b6f0 msgr2=0x7fc3b0155b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.668+0000 7fc3a2000640 1 -- 172.21.15.27:0/1493178945 shutdown_connections 2024-03-28T18:55:53.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.666+0000 7f84e3e00640 1 -- 172.21.15.27:0/2036188141 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40c7ff0 con 0x7f84e40c8db0 2024-03-28T18:55:53.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.668+0000 7fc3a2000640 1 -- 172.21.15.27:0/1493178945 wait complete. 2024-03-28T18:55:53.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.669+0000 7f84e3e00640 1 --2- 172.21.15.27:0/2036188141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c8db0 0x7f84e4156090 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f84d00097d0 tx=0x7f84d002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d9318ec7247f281 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.669+0000 7f84e3400640 1 -- 172.21.15.27:0/2036188141 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0031780 con 0x7f84e40c8db0 2024-03-28T18:55:53.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.669+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:6816/2908350031,v1:172.21.15.155:6817/2908350031] conn(0x7f18ec076530 msgr2=0x7f18ec078950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.669+0000 7f1911e00640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.155:6816/2908350031,v1:172.21.15.155:6817/2908350031] conn(0x7f18ec076530 0x7f18ec078950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.669+0000 7f84e3400640 1 -- 172.21.15.27:0/2036188141 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d0037070 con 0x7f84e40c8db0 2024-03-28T18:55:53.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.669+0000 7f84e3400640 1 -- 172.21.15.27:0/2036188141 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0031780 con 0x7f84e40c8db0 2024-03-28T18:55:53.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.670+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18ec06bca0 msgr2=0x7f18ec06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.670+0000 7f1911e00640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18ec06bca0 0x7f18ec06e160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f18f8013fd0 tx=0x7f18f8015040 comp rx=0 tx=0).stop 2024-03-28T18:55:53.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.670+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 msgr2=0x7f190c0728f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.670+0000 7f1911e00640 1 --2- 172.21.15.27:0/1186461629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f190c0aa300 0x7f190c0728f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f190400c780 tx=0x7f190400cc40 comp rx=0 tx=0).stop 2024-03-28T18:55:53.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.671+0000 7f190b400640 1 -- 172.21.15.27:0/1186461629 reap_dead start 2024-03-28T18:55:53.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.671+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 shutdown_connections 2024-03-28T18:55:53.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.671+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 >> 172.21.15.27:0/1186461629 conn(0x7f190c0c6c00 msgr2=0x7f190c1644e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.671+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 shutdown_connections 2024-03-28T18:55:53.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.673+0000 7f1911e00640 1 -- 172.21.15.27:0/1186461629 wait complete. 2024-03-28T18:55:53.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.672+0000 7f84e9e00640 1 -- 172.21.15.27:0/2036188141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c8db0 msgr2=0x7f84e4156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.672+0000 7f84e9e00640 1 --2- 172.21.15.27:0/2036188141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c8db0 0x7f84e4156090 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f84d00097d0 tx=0x7f84d002b820 comp rx=0 tx=0).stop 2024-03-28T18:55:53.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.677+0000 7f84e9e00640 1 -- 172.21.15.27:0/2036188141 shutdown_connections 2024-03-28T18:55:53.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.677+0000 7f84e9e00640 1 --2- 172.21.15.27:0/2036188141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c8db0 0x7f84e4156090 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.677+0000 7f84e9e00640 1 -- 172.21.15.27:0/2036188141 >> 172.21.15.27:0/2036188141 conn(0x7f84e406b0c0 msgr2=0x7f84e406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.678+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 --> [v2:172.21.15.155:6808/330629214,v1:172.21.15.155:6809/330629214] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f80f8002dd0 con 0x7f81040767a0 2024-03-28T18:55:53.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.679+0000 7f811a000640 1 -- 172.21.15.27:0/2943006779 <== osd.6 v2:172.21.15.155:6808/330629214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f80f8002dd0 con 0x7f81040767a0 2024-03-28T18:55:53.679 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:53.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.680+0000 7f84e9e00640 1 -- 172.21.15.27:0/2036188141 shutdown_connections 2024-03-28T18:55:53.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.680+0000 7f84e9e00640 1 -- 172.21.15.27:0/2036188141 wait complete. 2024-03-28T18:55:53.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T18:55:53.681 INFO:teuthology.orchestra.run.smithi027.stdout:55834574859 2024-03-28T18:55:53.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.681+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:6808/330629214,v1:172.21.15.155:6809/330629214] conn(0x7f81040767a0 msgr2=0x7f8104078bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.681+0000 7f810f400640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.155:6808/330629214,v1:172.21.15.155:6809/330629214] conn(0x7f81040767a0 0x7f8104078bc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.681+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f810406c000 msgr2=0x7f810406e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.681+0000 7f810f400640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f810406c000 0x7f810406e4c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8124151280 tx=0x7f8114009040 comp rx=0 tx=0).stop 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.682+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81240c7970 msgr2=0x7f8124145890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.682+0000 7f810f400640 1 --2- 172.21.15.27:0/2943006779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81240c7970 0x7f8124145890 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f811c033000 tx=0x7f811c004a40 comp rx=0 tx=0).stop 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.682+0000 7f8122800640 1 -- 172.21.15.27:0/2943006779 reap_dead start 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.682+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 shutdown_connections 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.682+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 >> 172.21.15.27:0/2943006779 conn(0x7f812406b6f0 msgr2=0x7f812414e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.683+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 shutdown_connections 2024-03-28T18:55:53.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.683+0000 7f810f400640 1 -- 172.21.15.27:0/2943006779 wait complete. 2024-03-28T18:55:53.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.683+0000 7f84e9e00640 1 Processor -- start 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.686+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 --> [v2:172.21.15.27:6800/974955781,v1:172.21.15.27:6801/974955781] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6074002dd0 con 0x7f607c076460 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.686+0000 7f6093400640 1 -- 172.21.15.27:0/1373690621 <== osd.3 v2:172.21.15.27:6800/974955781 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6074002dd0 con 0x7f607c076460 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6800/974955781,v1:172.21.15.27:6801/974955781] conn(0x7f607c076460 msgr2=0x7f607c078880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f609be00640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6800/974955781,v1:172.21.15.27:6801/974955781] conn(0x7f607c076460 0x7f607c078880 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f607c06bbd0 msgr2=0x7f607c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f609be00640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f607c06bbd0 0x7f607c06e090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f608c034000 tx=0x7f608c02f000 comp rx=0 tx=0).stop 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d6fa0 msgr2=0x7f609415f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f609be00640 1 --2- 172.21.15.27:0/1373690621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60940d6fa0 0x7f609415f9b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f609c064b90 tx=0x7f609c068c60 comp rx=0 tx=0).stop 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f609b400640 1 -- 172.21.15.27:0/1373690621 reap_dead start 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.688+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 shutdown_connections 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.688+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 >> 172.21.15.27:0/1373690621 conn(0x7f6094005e90 msgr2=0x7f60940069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.688+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 shutdown_connections 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.688+0000 7f609be00640 1 -- 172.21.15.27:0/1373690621 wait complete. 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.684+0000 7f84e9e00640 1 -- start start 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.685+0000 7f84e9e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c5aa0 0x7f84e40abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.685+0000 7f84e9e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e40c8db0 0x7f84e40c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.685+0000 7f84e9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e4152840 0x7f84e40c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.685+0000 7f84e9e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4072100 con 0x7f84e4152840 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.685+0000 7f84e9e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40c82d0 con 0x7f84e40c8db0 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.685+0000 7f84e9e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4072d30 con 0x7f84e40c5aa0 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.686+0000 7f84e8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c5aa0 0x7f84e40abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.686+0000 7f84e8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c5aa0 0x7f84e40abb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45162/0 (socket says 172.21.15.27:45162) 2024-03-28T18:55:53.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.686+0000 7f84e8a00640 1 -- 172.21.15.27:0/3809544776 learned_addr learned my addr 172.21.15.27:0/3809544776 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.686+0000 7f84e3e00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e40c8db0 0x7f84e40c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.686+0000 7f84e9400640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e4152840 0x7f84e40c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f84e9400640 1 -- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c5aa0 msgr2=0x7f84e40abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f84e9400640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e40c5aa0 0x7f84e40abb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f84e9400640 1 -- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e40c8db0 msgr2=0x7f84e40c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f84e9400640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e40c8db0 0x7f84e40c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f84e9400640 1 -- 172.21.15.27:0/3809544776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40ac750 con 0x7f84e4152840 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.687+0000 7f84e3e00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e40c8db0 0x7f84e40c3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:53.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.688+0000 7f84e9400640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e4152840 0x7f84e40c3fa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00c780 tx=0x7f84dc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.689+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc003b90 con 0x7f84e4152840 2024-03-28T18:55:53.698 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:53.698 INFO:teuthology.orchestra.run.smithi027.stdout:55834574858 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.689+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc003d50 con 0x7f84e4152840 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.689+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc005310 con 0x7f84e4152840 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.690+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e41eeeb0 con 0x7f84e4152840 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.691+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e40b0830 con 0x7f84e4152840 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.693+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f84e404f400 con 0x7f84e4152840 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.695+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f84dc0054b0 con 0x7f84e4152840 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.696+0000 7f84e0c00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c406bc00 0x7f84c406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.696+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f84dc08abc0 con 0x7f84e4152840 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.697+0000 7f84e0c00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:6824/4094998171,v1:172.21.15.155:6825/4094998171] conn(0x7f84c4076420 0x7f84c4078840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.697+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 --> [v2:172.21.15.155:6824/4094998171,v1:172.21.15.155:6825/4094998171] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84c4078ef0 con 0x7f84c4076420 2024-03-28T18:55:53.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.697+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84dc091020 con 0x7f84e4152840 2024-03-28T18:55:53.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T18:55:53.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T18:55:53.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.698+0000 7f84e3e00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:6824/4094998171,v1:172.21.15.155:6825/4094998171] conn(0x7f84c4076420 0x7f84c4078840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.698+0000 7f84e8a00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c406bc00 0x7f84c406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.699+0000 7f84e8a00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c406bc00 0x7f84c406e0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f84d8007d50 tx=0x7f84d8007ca0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.704+0000 7f84e3e00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:6824/4094998171,v1:172.21.15.155:6825/4094998171] conn(0x7f84c4076420 0x7f84c4078840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.708+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 <== osd.7 v2:172.21.15.155:6824/4094998171 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f84c4078ef0 con 0x7f84c4076420 2024-03-28T18:55:53.714 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:53.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T18:55:53.715 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:53.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T18:55:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.737+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 --> [v2:172.21.15.155:6824/4094998171,v1:172.21.15.155:6825/4094998171] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f84e4072ae0 con 0x7f84c4076420 2024-03-28T18:55:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.738+0000 7f84e0c00640 1 -- 172.21.15.27:0/3809544776 <== osd.7 v2:172.21.15.155:6824/4094998171 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f84e4072ae0 con 0x7f84c4076420 2024-03-28T18:55:53.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:6824/4094998171,v1:172.21.15.155:6825/4094998171] conn(0x7f84c4076420 msgr2=0x7f84c4078840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9e00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.155:6824/4094998171,v1:172.21.15.155:6825/4094998171] conn(0x7f84c4076420 0x7f84c4078840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c406bc00 msgr2=0x7f84c406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9e00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c406bc00 0x7f84c406e0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f84d8007d50 tx=0x7f84d8007ca0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e4152840 msgr2=0x7f84e40c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9e00640 1 --2- 172.21.15.27:0/3809544776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e4152840 0x7f84e40c3fa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00c780 tx=0x7f84dc00cc40 comp rx=0 tx=0).stop 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9400640 1 -- 172.21.15.27:0/3809544776 reap_dead start 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 shutdown_connections 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.739+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 >> 172.21.15.27:0/3809544776 conn(0x7f84e406b0c0 msgr2=0x7f84e4061b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.740+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 shutdown_connections 2024-03-28T18:55:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.740+0000 7f84e9e00640 1 -- 172.21.15.27:0/3809544776 wait complete. 2024-03-28T18:55:53.765 INFO:teuthology.orchestra.run.smithi027.stdout:55834574858 2024-03-28T18:55:53.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T18:55:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.984+0000 7fc63f200640 1 Processor -- start 2024-03-28T18:55:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.984+0000 7fc63f200640 1 -- start start 2024-03-28T18:55:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.986+0000 7fc63f200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc638050720 0x7fc6380062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.986+0000 7fc63f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc63800fcf0 0x7fc638006800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.987+0000 7fc63e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc63800fcf0 0x7fc638006800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.987+0000 7fc63e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc63800fcf0 0x7fc638006800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59504/0 (socket says 172.21.15.27:59504) 2024-03-28T18:55:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.987+0000 7fc63d400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc638050720 0x7fc6380062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.988+0000 7fc63de00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc6380d7920 0x7fc63800edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55618/0 (socket says 172.21.15.27:55618) 2024-03-28T18:55:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.988+0000 7fc63d400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc638050720 0x7fc6380062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43550/0 (socket says 172.21.15.27:43550) 2024-03-28T18:55:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.988+0000 7fc63e800640 1 -- 172.21.15.27:0/1095129850 learned_addr learned my addr 172.21.15.27:0/1095129850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.988+0000 7fc63f200640 1 -- 172.21.15.27:0/1095129850 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380d7d00 con 0x7fc6380d7920 2024-03-28T18:55:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.989+0000 7fc63f200640 1 -- 172.21.15.27:0/1095129850 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6380db250 con 0x7fc63800fcf0 2024-03-28T18:55:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.989+0000 7fc63f200640 1 -- 172.21.15.27:0/1095129850 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63804ba00 con 0x7fc638050720 2024-03-28T18:55:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.989+0000 7fc63ca00640 1 -- 172.21.15.27:0/1095129850 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923017842 0 0) 0x7fc6380d7d00 con 0x7fc6380d7920 2024-03-28T18:55:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.989+0000 7fc63ca00640 1 -- 172.21.15.27:0/1095129850 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc624003520 con 0x7fc6380d7920 2024-03-28T18:55:53.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.993+0000 7fc63e800640 1 -- 172.21.15.27:0/1095129850 >> v2:172.21.15.155:3300/0 conn(0x7fc638050720 msgr2=0x7fc6380062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:53.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.993+0000 7fc63e800640 1 --2- 172.21.15.27:0/1095129850 >> v2:172.21.15.155:3300/0 conn(0x7fc638050720 0x7fc6380062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.993+0000 7fc63e800640 1 -- 172.21.15.27:0/1095129850 >> v1:172.21.15.155:6789/0 conn(0x7fc6380d7920 legacy=0x7fc63800edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.993+0000 7fc63e800640 1 -- 172.21.15.27:0/1095129850 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63804bce0 con 0x7fc63800fcf0 2024-03-28T18:55:53.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.995+0000 7fc63e800640 1 --2- 172.21.15.27:0/1095129850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63800fcf0 0x7fc638006800 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc63804aa50 tx=0x7fc62c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba7099db8356229 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:53.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.996+0000 7fc63ca00640 1 -- 172.21.15.27:0/1095129850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c035020 con 0x7fc63800fcf0 2024-03-28T18:55:53.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.996+0000 7fc63ca00640 1 -- 172.21.15.27:0/1095129850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c031880 con 0x7fc63800fcf0 2024-03-28T18:55:53.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.997+0000 7fc63ca00640 1 -- 172.21.15.27:0/1095129850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c031c50 con 0x7fc63800fcf0 2024-03-28T18:55:53.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.998+0000 7fc63f200640 1 -- 172.21.15.27:0/1095129850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63800fcf0 msgr2=0x7fc638006800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:53.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.998+0000 7fc63f200640 1 --2- 172.21.15.27:0/1095129850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63800fcf0 0x7fc638006800 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc63804aa50 tx=0x7fc62c02b920 comp rx=0 tx=0).stop 2024-03-28T18:55:53.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.999+0000 7fc63f200640 1 -- 172.21.15.27:0/1095129850 shutdown_connections 2024-03-28T18:55:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.999+0000 7fc63f200640 1 --2- 172.21.15.27:0/1095129850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63800fcf0 0x7fc638006800 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.999+0000 7fc63f200640 1 --2- 172.21.15.27:0/1095129850 >> v2:172.21.15.155:3300/0 conn(0x7fc638050720 0x7fc6380062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:53.999+0000 7fc63f200640 1 -- 172.21.15.27:0/1095129850 >> 172.21.15.27:0/1095129850 conn(0x7fc63804fc00 msgr2=0x7fc638050010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.000+0000 7fc63f200640 1 -- 172.21.15.27:0/1095129850 shutdown_connections 2024-03-28T18:55:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.000+0000 7fc63f200640 1 -- 172.21.15.27:0/1095129850 wait complete. 2024-03-28T18:55:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.003+0000 7fc63f200640 1 Processor -- start 2024-03-28T18:55:54.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.004+0000 7fc63f200640 1 -- start start 2024-03-28T18:55:54.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.006+0000 7fc63f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc63800fcf0 0x7fc63804e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.006+0000 7fc63f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc638050720 0x7fc638173c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.006+0000 7fc63f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6380d7920 0x7fc6380d2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.007+0000 7fc63e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6380d7920 0x7fc6380d2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.007+0000 7fc63e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6380d7920 0x7fc6380d2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59512/0 (socket says 172.21.15.27:59512) 2024-03-28T18:55:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.007+0000 7fc63d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc638050720 0x7fc638173c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.007+0000 7fc63d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc638050720 0x7fc638173c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43566/0 (socket says 172.21.15.27:43566) 2024-03-28T18:55:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.008+0000 7fc63e800640 1 -- 172.21.15.27:0/1278826176 learned_addr learned my addr 172.21.15.27:0/1278826176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.008+0000 7fc63de00640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc63800fcf0 0x7fc63804e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.008+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63800f4c0 con 0x7fc6380d7920 2024-03-28T18:55:54.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.008+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63804c2a0 con 0x7fc638050720 2024-03-28T18:55:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.008+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63800f640 con 0x7fc63800fcf0 2024-03-28T18:55:54.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.009+0000 7fc63e800640 1 -- 172.21.15.27:0/1278826176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc63800fcf0 msgr2=0x7fc63804e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.010+0000 7fc63e800640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc63800fcf0 0x7fc63804e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.010+0000 7fc63e800640 1 -- 172.21.15.27:0/1278826176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc638050720 msgr2=0x7fc638173c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.010+0000 7fc63e800640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc638050720 0x7fc638173c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.010+0000 7fc63e800640 1 -- 172.21.15.27:0/1278826176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc638173480 con 0x7fc6380d7920 2024-03-28T18:55:54.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.010+0000 7fc63de00640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc63800fcf0 0x7fc63804e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.011+0000 7fc63e800640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6380d7920 0x7fc6380d2e40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009960 tx=0x7fc62c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.001+0000 7fec8b000640 1 Processor -- start 2024-03-28T18:55:54.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fc636000640 1 -- 172.21.15.27:0/1278826176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c035040 con 0x7fc6380d7920 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.001+0000 7fec8b000640 1 -- start start 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.003+0000 7fec8b000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec84162bc0 0x7fec840ab420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.004+0000 7fec83400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fec8408b470 0x7fec841624b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55634/0 (socket says 172.21.15.27:55634) 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.004+0000 7fec83e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fec84070d10 0x7fec840710f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59742/0 (socket says 172.21.15.27:59742) 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.004+0000 7fec83400640 1 -- 172.21.15.27:0/3346203335 learned_addr learned my addr 172.21.15.27:0/3346203335 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.004+0000 7fec89000640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec84162bc0 0x7fec840ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.005+0000 7fec8b000640 1 -- 172.21.15.27:0/3346203335 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8408bd40 con 0x7fec84070d10 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.005+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893536319 0 0) 0x7fec8408bd40 con 0x7fec84070d10 2024-03-28T18:55:54.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.005+0000 7fec8b000640 1 -- 172.21.15.27:0/3346203335 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec840c25a0 con 0x7fec8408b470 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.005+0000 7fec8b000640 1 -- 172.21.15.27:0/3346203335 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec840c3070 con 0x7fec84162bc0 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec68005730 con 0x7fec84070d10 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562694423 0 0) 0x7fec840c25a0 con 0x7fec8408b470 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8408bd40 con 0x7fec8408b470 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591401805 0 0) 0x7fec68005730 con 0x7fec84070d10 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec84162bc0 msgr2=0x7fec840ab420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec84162bc0 0x7fec840ab420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 >> v1:172.21.15.155:6789/0 conn(0x7fec8408b470 legacy=0x7fec841624b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8408ba60 con 0x7fec84070d10 2024-03-28T18:55:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fc636000640 1 -- 172.21.15.27:0/1278826176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c031680 con 0x7fc6380d7920 2024-03-28T18:55:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fc636000640 1 -- 172.21.15.27:0/1278826176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c045480 con 0x7fc6380d7920 2024-03-28T18:55:54.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.013+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6380dbb20 con 0x7fc6380d7920 2024-03-28T18:55:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.012+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fec74001670 con 0x7fec84070d10 2024-03-28T18:55:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.013+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 >> v1:172.21.15.27:6789/0 conn(0x7fec84070d10 legacy=0x7fec840710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.013+0000 7fec82a00640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec68006e50 0x7fec68007290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.013+0000 7fec82a00640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec680059b0 0x7fec68005e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.013+0000 7fec82a00640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68006370 0x7fec6800ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.013+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840c3070 con 0x7fec680059b0 2024-03-28T18:55:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.013+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68007ba0 con 0x7fec68006370 2024-03-28T18:55:54.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.013+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec68007da0 con 0x7fec68006e50 2024-03-28T18:55:54.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.014+0000 7fec89000640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68006370 0x7fec6800ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.014+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc638173720 con 0x7fc6380d7920 2024-03-28T18:55:54.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.014+0000 7fec83e00640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec68006e50 0x7fec68007290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.015+0000 7fec89000640 1 -- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec68006e50 msgr2=0x7fec68007290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.015+0000 7fec89000640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec68006e50 0x7fec68007290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.015+0000 7fec89000640 1 -- 172.21.15.27:0/3346203335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec680059b0 msgr2=0x7fec68005e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.015+0000 7fec89000640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec680059b0 0x7fec68005e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.015+0000 7fec89000640 1 -- 172.21.15.27:0/3346203335 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8408ba60 con 0x7fec68006370 2024-03-28T18:55:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.016+0000 7efd37a00640 1 Processor -- start 2024-03-28T18:55:54.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.019+0000 7fc636000640 1 -- 172.21.15.27:0/1278826176 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc62c038020 con 0x7fc6380d7920 2024-03-28T18:55:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.019+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc618005290 con 0x7fc6380d7920 2024-03-28T18:55:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.016+0000 7efd37a00640 1 -- start start 2024-03-28T18:55:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.018+0000 7efd37a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efd300c1db0 0x7efd300c2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.019+0000 7efd37a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd30152780 con 0x7efd300abf10 2024-03-28T18:55:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.019+0000 7efd37a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd301519c0 con 0x7efd3006db40 2024-03-28T18:55:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.019+0000 7efd37a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd301522f0 con 0x7efd300c1db0 2024-03-28T18:55:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.020+0000 7efd35000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efd300abf10 0x7efd30071510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59748/0 (socket says 172.21.15.27:59748) 2024-03-28T18:55:54.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.021+0000 7efd35000640 1 -- 172.21.15.27:0/3901903840 learned_addr learned my addr 172.21.15.27:0/3901903840 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.020+0000 7fc636000640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62006bc20 0x7fc62006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.023+0000 7fec89000640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68006370 0x7fec6800ffd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fec7c008dc0 tx=0x7fec68007a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78036354ecd0f8eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.023+0000 7fc63de00640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62006bc20 0x7fc62006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.024+0000 7fc636000640 1 -- 172.21.15.27:0/1278826176 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fc62c031830 con 0x7fc6380d7920 2024-03-28T18:55:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.024+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c032070 con 0x7fec68006370 2024-03-28T18:55:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.021+0000 7efd2fe00640 1 --2- 172.21.15.27:0/3901903840 >> v2:172.21.15.155:3300/0 conn(0x7efd300c1db0 0x7efd300c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.024+0000 7efd2f400640 1 -- 172.21.15.27:0/3901903840 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135284454 0 0) 0x7efd301519c0 con 0x7efd3006db40 2024-03-28T18:55:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.024+0000 7fec82a00640 1 -- 172.21.15.27:0/3346203335 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c02c990 con 0x7fec68006370 2024-03-28T18:55:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.024+0000 7efd2f400640 1 -- 172.21.15.27:0/3901903840 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd14003520 con 0x7efd3006db40 2024-03-28T18:55:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.025+0000 7fec8b000640 1 -- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68006370 msgr2=0x7fec6800ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.025+0000 7fec8b000640 1 --2- 172.21.15.27:0/3346203335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec68006370 0x7fec6800ffd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fec7c008dc0 tx=0x7fec68007a40 comp rx=0 tx=0).stop 2024-03-28T18:55:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.025+0000 7fec89000640 1 -- 172.21.15.27:0/3346203335 reap_dead start 2024-03-28T18:55:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.026+0000 7fec8b000640 1 -- 172.21.15.27:0/3346203335 shutdown_connections 2024-03-28T18:55:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.026+0000 7fec8b000640 1 -- 172.21.15.27:0/3346203335 >> 172.21.15.27:0/3346203335 conn(0x7fec840c6da0 msgr2=0x7fec840c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.026+0000 7efd2fe00640 1 -- 172.21.15.27:0/3901903840 >> v1:172.21.15.155:6789/0 conn(0x7efd3006db40 legacy=0x7efd3006df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.026+0000 7efd2fe00640 1 -- 172.21.15.27:0/3901903840 >> v1:172.21.15.27:6789/0 conn(0x7efd300abf10 legacy=0x7efd30071510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.026+0000 7efd2fe00640 1 -- 172.21.15.27:0/3901903840 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd30151ca0 con 0x7efd300c1db0 2024-03-28T18:55:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.026+0000 7fc63de00640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62006bc20 0x7fc62006e0e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc63804dfd0 tx=0x7fc630009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.027+0000 7f7774e00640 1 Processor -- start 2024-03-28T18:55:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.028+0000 7f7774e00640 1 -- start start 2024-03-28T18:55:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.028+0000 7efd2fe00640 1 --2- 172.21.15.27:0/3901903840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300c1db0 0x7efd300c2210 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd30152470 tx=0x7efd1802dfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=730db636c6368f29 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.029+0000 7f7774e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7770152ec0 0x7f7770156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.029+0000 7f7774e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77700c8ff0 0x7f7770159440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.029+0000 7f7774e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7770159980 0x7f77700ad040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.029+0000 7f7774e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7770158cd0 con 0x7f7770159980 2024-03-28T18:55:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.029+0000 7f7774e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7770158fb0 con 0x7f7770152ec0 2024-03-28T18:55:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.029+0000 7f7774e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77701566c0 con 0x7f77700c8ff0 2024-03-28T18:55:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.030+0000 7f776de00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7770152ec0 0x7f7770156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.030+0000 7f7765400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77700c8ff0 0x7f7770159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.030+0000 7f776de00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7770152ec0 0x7f7770156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43586/0 (socket says 172.21.15.27:43586) 2024-03-28T18:55:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.030+0000 7f776de00640 1 -- 172.21.15.27:0/3298826109 learned_addr learned my addr 172.21.15.27:0/3298826109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.031+0000 7f7765400640 1 -- 172.21.15.27:0/3298826109 >> v2:172.21.15.155:3300/0 conn(0x7f7770152ec0 msgr2=0x7f7770156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.031+0000 7f7765400640 1 --2- 172.21.15.27:0/3298826109 >> v2:172.21.15.155:3300/0 conn(0x7f7770152ec0 0x7f7770156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.031+0000 7f7765400640 1 -- 172.21.15.27:0/3298826109 >> v2:172.21.15.27:3300/0 conn(0x7f7770159980 msgr2=0x7f77700ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.031+0000 7f7765400640 1 --2- 172.21.15.27:0/3298826109 >> v2:172.21.15.27:3300/0 conn(0x7f7770159980 0x7f77700ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.031+0000 7f7765400640 1 -- 172.21.15.27:0/3298826109 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777006d500 con 0x7f77700c8ff0 2024-03-28T18:55:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.026+0000 7fec8b000640 1 -- 172.21.15.27:0/3346203335 shutdown_connections 2024-03-28T18:55:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.029+0000 7fec8b000640 1 -- 172.21.15.27:0/3346203335 wait complete. 2024-03-28T18:55:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.032+0000 7fec8b000640 1 Processor -- start 2024-03-28T18:55:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.026+0000 7efd2f400640 1 -- 172.21.15.27:0/3901903840 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114375058 0 0) 0x7efd14003520 con 0x7efd3006db40 2024-03-28T18:55:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.031+0000 7efd2f400640 1 -- 172.21.15.27:0/3901903840 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1803b070 con 0x7efd300c1db0 2024-03-28T18:55:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.031+0000 7f776de00640 1 --2- 172.21.15.27:0/3298826109 >> v2:172.21.15.155:3300/0 conn(0x7f7770152ec0 0x7f7770156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.032+0000 7fec8b000640 1 -- start start 2024-03-28T18:55:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7efd2f400640 1 -- 172.21.15.27:0/3901903840 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1802eae0 con 0x7efd300c1db0 2024-03-28T18:55:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7efd2f400640 1 -- 172.21.15.27:0/3901903840 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd200015a0 con 0x7efd300c1db0 2024-03-28T18:55:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7f7765400640 1 --2- 172.21.15.27:0/3298826109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77700c8ff0 0x7f7770159440 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f77700c8fc0 tx=0x7f775802b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75de1b9f3a585bc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7fec8b000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec840c53c0 0x7fec841f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7fec8b000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec8415eff0 0x7fec840be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7fec8b000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec840be9f0 0x7fec8408a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7fec8b000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840c3550 con 0x7fec840be9f0 2024-03-28T18:55:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7fec8b000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec84071800 con 0x7fec840c53c0 2024-03-28T18:55:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.033+0000 7fec8b000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec840c36d0 con 0x7fec8415eff0 2024-03-28T18:55:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.034+0000 7fc636000640 1 -- 172.21.15.27:0/1278826176 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc62c07a660 con 0x7fc6380d7920 2024-03-28T18:55:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.034+0000 7efd37a00640 1 -- 172.21.15.27:0/3901903840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300c1db0 msgr2=0x7efd300c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.034+0000 7efd37a00640 1 --2- 172.21.15.27:0/3901903840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300c1db0 0x7efd300c2210 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd30152470 tx=0x7efd1802dfb0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.034+0000 7f776d400640 1 -- 172.21.15.27:0/3298826109 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758037070 con 0x7f77700c8ff0 2024-03-28T18:55:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.034+0000 7f776d400640 1 -- 172.21.15.27:0/3298826109 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758030710 con 0x7f77700c8ff0 2024-03-28T18:55:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.034+0000 7f776d400640 1 -- 172.21.15.27:0/3298826109 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758030ae0 con 0x7f77700c8ff0 2024-03-28T18:55:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7efd37a00640 1 -- 172.21.15.27:0/3901903840 shutdown_connections 2024-03-28T18:55:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7efd37a00640 1 --2- 172.21.15.27:0/3901903840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300c1db0 0x7efd300c2210 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7efd37a00640 1 -- 172.21.15.27:0/3901903840 >> 172.21.15.27:0/3901903840 conn(0x7efd300c6da0 msgr2=0x7efd300c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7efd37a00640 1 -- 172.21.15.27:0/3901903840 shutdown_connections 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7efd37a00640 1 -- 172.21.15.27:0/3901903840 wait complete. 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7f7774e00640 1 -- 172.21.15.27:0/3298826109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77700c8ff0 msgr2=0x7f7770159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7f7774e00640 1 --2- 172.21.15.27:0/3298826109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77700c8ff0 0x7f7770159440 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f77700c8fc0 tx=0x7f775802b730 comp rx=0 tx=0).stop 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7f7774e00640 1 -- 172.21.15.27:0/3298826109 shutdown_connections 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7f7774e00640 1 --2- 172.21.15.27:0/3298826109 >> v2:172.21.15.27:3300/0 conn(0x7f7770159980 0x7f77700ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7f7774e00640 1 --2- 172.21.15.27:0/3298826109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77700c8ff0 0x7f7770159440 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.035+0000 7f7774e00640 1 --2- 172.21.15.27:0/3298826109 >> v2:172.21.15.155:3300/0 conn(0x7f7770152ec0 0x7f7770156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7f7774e00640 1 -- 172.21.15.27:0/3298826109 >> 172.21.15.27:0/3298826109 conn(0x7f77700671f0 msgr2=0x7f7770067600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7fec83400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec8415eff0 0x7fec840be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7fec83e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec840c53c0 0x7fec841f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7fec83400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec8415eff0 0x7fec840be4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45202/0 (socket says 172.21.15.27:45202) 2024-03-28T18:55:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7fec83400640 1 -- 172.21.15.27:0/1634473781 learned_addr learned my addr 172.21.15.27:0/1634473781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7fec89000640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec840be9f0 0x7fec8408a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7f7774e00640 1 -- 172.21.15.27:0/3298826109 shutdown_connections 2024-03-28T18:55:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7f7774e00640 1 -- 172.21.15.27:0/3298826109 wait complete. 2024-03-28T18:55:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7f96b6a00640 1 Processor -- start 2024-03-28T18:55:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7f96b6a00640 1 -- start start 2024-03-28T18:55:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.038+0000 7f96b6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96b014f7b0 0x7f96b01558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.038+0000 7f96b6a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f96b006fb40 0x7f96b00c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.038+0000 7f96b6a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b01563a0 con 0x7f96b0155e30 2024-03-28T18:55:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.038+0000 7f96b6a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b0156640 con 0x7f96b014f7b0 2024-03-28T18:55:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.038+0000 7f96b6a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b006f620 con 0x7f96b006fb40 2024-03-28T18:55:54.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.038+0000 7efd37a00640 1 Processor -- start 2024-03-28T18:55:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.036+0000 7fec83e00640 1 -- 172.21.15.27:0/1634473781 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec8415eff0 msgr2=0x7fec840be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.037+0000 7fec83e00640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec8415eff0 0x7fec840be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.037+0000 7fec83e00640 1 -- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec840be9f0 msgr2=0x7fec8408a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.037+0000 7fec83e00640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec840be9f0 0x7fec8408a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.037+0000 7fec83e00640 1 -- 172.21.15.27:0/1634473781 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec840bee80 con 0x7fec840c53c0 2024-03-28T18:55:54.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.039+0000 7efd37a00640 1 -- start start 2024-03-28T18:55:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.039+0000 7f96af400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f96b006fb40 0x7f96b00c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.039+0000 7f96af400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f96b006fb40 0x7f96b00c27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43616/0 (socket says 172.21.15.27:43616) 2024-03-28T18:55:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.039+0000 7f96af400640 1 -- 172.21.15.27:0/2089091988 learned_addr learned my addr 172.21.15.27:0/2089091988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.039+0000 7f7774e00640 1 Processor -- start 2024-03-28T18:55:54.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7fec83e00640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec840c53c0 0x7fec841f7c40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fec74004810 tx=0x7fec74004cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7f96afe00640 1 --2- 172.21.15.27:0/2089091988 >> v2:172.21.15.27:3300/0 conn(0x7f96b014f7b0 0x7f96b01558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7efd37a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd3006db40 0x7efd30155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7efd37a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300abf10 0x7efd30153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7efd37a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd300c1db0 0x7efd301538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7fec7be00640 1 -- 172.21.15.27:0/1634473781 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7400f9c0 con 0x7fec840c53c0 2024-03-28T18:55:54.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7fec7be00640 1 -- 172.21.15.27:0/1634473781 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec74018070 con 0x7fec840c53c0 2024-03-28T18:55:54.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7fec7be00640 1 -- 172.21.15.27:0/1634473781 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7400fda0 con 0x7fec840c53c0 2024-03-28T18:55:54.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7efd37a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd300ac2f0 con 0x7efd300c1db0 2024-03-28T18:55:54.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.041+0000 7f7774e00640 1 -- start start 2024-03-28T18:55:54.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7efd37a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd30073620 con 0x7efd300abf10 2024-03-28T18:55:54.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7efd37a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd30071160 con 0x7efd3006db40 2024-03-28T18:55:54.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7f96af400640 1 -- 172.21.15.27:0/2089091988 >> v2:172.21.15.27:3300/0 conn(0x7f96b014f7b0 msgr2=0x7f96b01558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7f96af400640 1 --2- 172.21.15.27:0/2089091988 >> v2:172.21.15.27:3300/0 conn(0x7f96b014f7b0 0x7f96b01558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7f96af400640 1 -- 172.21.15.27:0/2089091988 >> v1:172.21.15.155:6789/0 conn(0x7f96b0155e30 legacy=0x7f96b0152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.040+0000 7f96af400640 1 -- 172.21.15.27:0/2089091988 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b0070480 con 0x7f96b006fb40 2024-03-28T18:55:54.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.041+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec841f8220 con 0x7fec840c53c0 2024-03-28T18:55:54.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.041+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec841f87e0 con 0x7fec840c53c0 2024-03-28T18:55:54.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd2fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300abf10 0x7efd30153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd35a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd300c1db0 0x7efd301538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd2fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300abf10 0x7efd30153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43628/0 (socket says 172.21.15.27:43628) 2024-03-28T18:55:54.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd2fe00640 1 -- 172.21.15.27:0/846508520 learned_addr learned my addr 172.21.15.27:0/846508520 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.043+0000 7f96af400640 1 --2- 172.21.15.27:0/2089091988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b006fb40 0x7f96b00c27b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f96b00a9990 tx=0x7f969c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc82c407a82034fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.044+0000 7f96aea00640 1 -- 172.21.15.27:0/2089091988 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c034070 con 0x7f96b006fb40 2024-03-28T18:55:54.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.044+0000 7f96aea00640 1 -- 172.21.15.27:0/2089091988 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c030610 con 0x7f96b006fb40 2024-03-28T18:55:54.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.044+0000 7f96aea00640 1 -- 172.21.15.27:0/2089091988 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c030a20 con 0x7f96b006fb40 2024-03-28T18:55:54.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.044+0000 7f7774e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77700c8ff0 0x7f7770151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd35a00640 1 -- 172.21.15.27:0/846508520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd3006db40 msgr2=0x7efd30155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd35a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd3006db40 0x7efd30155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd35a00640 1 -- 172.21.15.27:0/846508520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300abf10 msgr2=0x7efd30153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd35a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300abf10 0x7efd30153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.042+0000 7efd35a00640 1 -- 172.21.15.27:0/846508520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd30152cd0 con 0x7efd300c1db0 2024-03-28T18:55:54.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.044+0000 7f7774e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7770152ec0 0x7f77700c2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.044+0000 7efd35a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd300c1db0 0x7efd301538d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efd2000c430 tx=0x7efd2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.044+0000 7f7774e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7770145e20 0x7f77701462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7f7774e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77700c6060 con 0x7f77700c8ff0 2024-03-28T18:55:54.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7f7774e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77700c5d80 con 0x7f7770152ec0 2024-03-28T18:55:54.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7efd2cc00640 1 -- 172.21.15.27:0/846508520 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd20017070 con 0x7efd300c1db0 2024-03-28T18:55:54.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7efd2cc00640 1 -- 172.21.15.27:0/846508520 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd200035f0 con 0x7efd300c1db0 2024-03-28T18:55:54.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7efd2cc00640 1 -- 172.21.15.27:0/846508520 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd200039e0 con 0x7efd300c1db0 2024-03-28T18:55:54.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3014e5f0 con 0x7efd300c1db0 2024-03-28T18:55:54.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7f7774e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777006d8f0 con 0x7f7770145e20 2024-03-28T18:55:54.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd301fc840 con 0x7efd300c1db0 2024-03-28T18:55:54.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.046+0000 7f776de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77700c8ff0 0x7f7770151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7f96b6a00640 1 -- 172.21.15.27:0/2089091988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b006fb40 msgr2=0x7f96b00c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.045+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2089091988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b006fb40 0x7f96b00c27b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f96b00a9990 tx=0x7f969c02b960 comp rx=0 tx=0).stop 2024-03-28T18:55:54.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.046+0000 7f776e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7770145e20 0x7f77701462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.046+0000 7f776e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7770145e20 0x7f77701462b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45216/0 (socket says 172.21.15.27:45216) 2024-03-28T18:55:54.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.046+0000 7f776e800640 1 -- 172.21.15.27:0/3580060423 learned_addr learned my addr 172.21.15.27:0/3580060423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.046+0000 7f7765400640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7770152ec0 0x7f77700c2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd08005290 con 0x7efd300c1db0 2024-03-28T18:55:54.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f776e800640 1 -- 172.21.15.27:0/3580060423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7770152ec0 msgr2=0x7f77700c2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f776e800640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7770152ec0 0x7f77700c2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f96b6a00640 1 -- 172.21.15.27:0/2089091988 shutdown_connections 2024-03-28T18:55:54.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2089091988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b006fb40 0x7f96b00c27b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2089091988 >> v2:172.21.15.27:3300/0 conn(0x7f96b014f7b0 0x7f96b01558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f96b6a00640 1 -- 172.21.15.27:0/2089091988 >> 172.21.15.27:0/2089091988 conn(0x7f96b00c48b0 msgr2=0x7f96b00c4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f776e800640 1 -- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77700c8ff0 msgr2=0x7f7770151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8404f470 con 0x7fec840c53c0 2024-03-28T18:55:54.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f776e800640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77700c8ff0 0x7f7770151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.047+0000 7f776e800640 1 -- 172.21.15.27:0/3580060423 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77700bfc80 con 0x7f7770145e20 2024-03-28T18:55:54.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.048+0000 7f96b6a00640 1 -- 172.21.15.27:0/2089091988 shutdown_connections 2024-03-28T18:55:54.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.048+0000 7f96b6a00640 1 -- 172.21.15.27:0/2089091988 wait complete. 2024-03-28T18:55:54.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.048+0000 7fec7be00640 1 -- 172.21.15.27:0/1634473781 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fec7401d460 con 0x7fec840c53c0 2024-03-28T18:55:54.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.048+0000 7f776de00640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77700c8ff0 0x7f7770151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:54.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.049+0000 7f776e800640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7770145e20 0x7f77701462b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f776800b410 tx=0x7f776800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.049+0000 7fec7be00640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec6406bbd0 0x7fec6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.050+0000 7f7766a00640 1 -- 172.21.15.27:0/3580060423 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768015020 con 0x7f7770145e20 2024-03-28T18:55:54.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.050+0000 7f7774e00640 1 -- 172.21.15.27:0/3580060423 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77700bff60 con 0x7f7770145e20 2024-03-28T18:55:54.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.050+0000 7f96b6a00640 1 Processor -- start 2024-03-28T18:55:54.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.050+0000 7f7774e00640 1 -- 172.21.15.27:0/3580060423 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77700c0520 con 0x7f7770145e20 2024-03-28T18:55:54.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.050+0000 7f96b6a00640 1 -- start start 2024-03-28T18:55:54.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b006fb40 0x7f96b0149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f7766a00640 1 -- 172.21.15.27:0/3580060423 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7768011830 con 0x7f7770145e20 2024-03-28T18:55:54.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f775f400640 1 -- 172.21.15.27:0/3580060423 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7748005290 con 0x7f7770145e20 2024-03-28T18:55:54.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.050+0000 7fec7be00640 1 -- 172.21.15.27:0/1634473781 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fec7408ae30 con 0x7fec840c53c0 2024-03-28T18:55:54.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.051+0000 7fec83400640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec6406bbd0 0x7fec6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b014f7b0 0x7f96b01528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96b0155e30 0x7f96b0152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f96b6a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b014fd10 con 0x7f96b006fb40 2024-03-28T18:55:54.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f96b6a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b014fb90 con 0x7f96b014f7b0 2024-03-28T18:55:54.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f96b6a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0072e90 con 0x7f96b0155e30 2024-03-28T18:55:54.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.052+0000 7f7766a00640 1 -- 172.21.15.27:0/3580060423 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768011c00 con 0x7f7770145e20 2024-03-28T18:55:54.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.053+0000 7f96af400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b014f7b0 0x7f96b01528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.053+0000 7fec83400640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec6406bbd0 0x7fec6406e090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fec70005880 tx=0x7fec70009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.053+0000 7f96af400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b014f7b0 0x7f96b01528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43640/0 (socket says 172.21.15.27:43640) 2024-03-28T18:55:54.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.053+0000 7f96af400640 1 -- 172.21.15.27:0/2526711751 learned_addr learned my addr 172.21.15.27:0/2526711751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.053+0000 7f96b4a00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96b0155e30 0x7f96b0152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.054+0000 7f96af400640 1 -- 172.21.15.27:0/2526711751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96b0155e30 msgr2=0x7f96b0152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.054+0000 7f96af400640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96b0155e30 0x7f96b0152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.054+0000 7f96af400640 1 -- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b006fb40 msgr2=0x7f96b0149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.054+0000 7f96af400640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b006fb40 0x7f96b0149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.054+0000 7f96af400640 1 -- 172.21.15.27:0/2526711751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00c6390 con 0x7f96b014f7b0 2024-03-28T18:55:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.055+0000 7f96af400640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b014f7b0 0x7f96b01528c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f969c000c00 tx=0x7f969c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.057+0000 7f7766a00640 1 -- 172.21.15.27:0/3580060423 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7768018020 con 0x7f7770145e20 2024-03-28T18:55:54.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.054+0000 7efd2cc00640 1 -- 172.21.15.27:0/846508520 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efd2001a070 con 0x7efd300c1db0 2024-03-28T18:55:54.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.056+0000 7efd2cc00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd0c06bbd0 0x7efd0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.056+0000 7efd2cc00640 1 -- 172.21.15.27:0/846508520 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7efd2008a5b0 con 0x7efd300c1db0 2024-03-28T18:55:54.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.057+0000 7efd35000640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd0c06bbd0 0x7efd0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.058+0000 7f7766a00640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f774c06bbd0 0x7f774c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.059+0000 7f7766a00640 1 -- 172.21.15.27:0/3580060423 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f776808abe0 con 0x7f7770145e20 2024-03-28T18:55:54.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.059+0000 7f776de00640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f774c06bbd0 0x7f774c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.060+0000 7efd35000640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd0c06bbd0 0x7efd0c06e090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd280015f0 tx=0x7efd28007a40 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.060+0000 7efd2cc00640 1 -- 172.21.15.27:0/846508520 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efd20021020 con 0x7efd300c1db0 2024-03-28T18:55:54.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.060+0000 7f776de00640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f774c06bbd0 0x7f774c06e090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f77600072e0 tx=0x7f7760009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.060+0000 7ff9ac400640 1 Processor -- start 2024-03-28T18:55:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.060+0000 7ff9ac400640 1 -- start start 2024-03-28T18:55:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.062+0000 7ff9ac400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a4157310 0x7ff9a415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.062+0000 7ff9ac400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a415b020 0x7ff9a40ad020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.062+0000 7ff9ac400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a40c5f30 con 0x7ff9a40c5970 2024-03-28T18:55:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.062+0000 7ff9ac400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a415d1c0 con 0x7ff9a4157310 2024-03-28T18:55:54.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.062+0000 7ff9ac400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a415d4a0 con 0x7ff9a415b020 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.062+0000 7ff9a9a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a4157310 0x7ff9a415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.062+0000 7ff9a9a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a4157310 0x7ff9a415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59570/0 (socket says 172.21.15.27:59570) 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.062+0000 7ff9a9a00640 1 -- 172.21.15.27:0/2614728313 learned_addr learned my addr 172.21.15.27:0/2614728313 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.063+0000 7ff9aa400640 1 --2- 172.21.15.27:0/2614728313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a415b020 0x7ff9a40ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.063+0000 7ff9a9a00640 1 -- 172.21.15.27:0/2614728313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a415b020 msgr2=0x7ff9a40ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.063+0000 7ff9a9a00640 1 --2- 172.21.15.27:0/2614728313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a415b020 0x7ff9a40ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.063+0000 7ff9a9a00640 1 -- 172.21.15.27:0/2614728313 >> v1:172.21.15.27:6789/0 conn(0x7ff9a40c5970 legacy=0x7ff9a415a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.063+0000 7ff9a9a00640 1 -- 172.21.15.27:0/2614728313 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a415d780 con 0x7ff9a4157310 2024-03-28T18:55:54.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.063+0000 7ff9aa400640 1 --2- 172.21.15.27:0/2614728313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a415b020 0x7ff9a40ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.065+0000 7f96a7e00640 1 -- 172.21.15.27:0/2526711751 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c048070 con 0x7f96b014f7b0 2024-03-28T18:55:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.065+0000 7f96a7e00640 1 -- 172.21.15.27:0/2526711751 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c030410 con 0x7f96b014f7b0 2024-03-28T18:55:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.065+0000 7f96a7e00640 1 -- 172.21.15.27:0/2526711751 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c034040 con 0x7f96b014f7b0 2024-03-28T18:55:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.065+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b00c6670 con 0x7f96b014f7b0 2024-03-28T18:55:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.066+0000 7ff9a9a00640 1 --2- 172.21.15.27:0/2614728313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a4157310 0x7ff9a415dca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff998009870 tx=0x7ff99802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8848633e6fb802e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.066+0000 7fec7be00640 1 -- 172.21.15.27:0/1634473781 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fec740595e0 con 0x7fec840c53c0 2024-03-28T18:55:54.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.067+0000 7ff9a3e00640 1 -- 172.21.15.27:0/2614728313 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980317c0 con 0x7ff9a4157310 2024-03-28T18:55:54.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.067+0000 7ff9a3e00640 1 -- 172.21.15.27:0/2614728313 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff998037070 con 0x7ff9a4157310 2024-03-28T18:55:54.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.067+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b014a410 con 0x7f96b014f7b0 2024-03-28T18:55:54.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.067+0000 7ff9a3e00640 1 -- 172.21.15.27:0/2614728313 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff998031b80 con 0x7ff9a4157310 2024-03-28T18:55:54.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.068+0000 7ff9ac400640 1 -- 172.21.15.27:0/2614728313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a4157310 msgr2=0x7ff9a415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.068+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2614728313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a4157310 0x7ff9a415dca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff998009870 tx=0x7ff99802b860 comp rx=0 tx=0).stop 2024-03-28T18:55:54.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.070+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b004f470 con 0x7f96b014f7b0 2024-03-28T18:55:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.071+0000 7f96a7e00640 1 -- 172.21.15.27:0/2526711751 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f969c04b070 con 0x7f96b014f7b0 2024-03-28T18:55:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.072+0000 7ff9ac400640 1 -- 172.21.15.27:0/2614728313 shutdown_connections 2024-03-28T18:55:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.072+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2614728313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a415b020 0x7ff9a40ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.072+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2614728313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a4157310 0x7ff9a415dca0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.072+0000 7ff9ac400640 1 -- 172.21.15.27:0/2614728313 >> 172.21.15.27:0/2614728313 conn(0x7ff9a40671f0 msgr2=0x7ff9a4067600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.072+0000 7ff9ac400640 1 -- 172.21.15.27:0/2614728313 shutdown_connections 2024-03-28T18:55:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.073+0000 7f96a7e00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f969006bca0 0x7f969006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.073+0000 7f96a7e00640 1 -- 172.21.15.27:0/2526711751 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f969c0aca80 con 0x7f96b014f7b0 2024-03-28T18:55:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.072+0000 7ff9ac400640 1 -- 172.21.15.27:0/2614728313 wait complete. 2024-03-28T18:55:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.074+0000 7f96afe00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f969006bca0 0x7f969006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.075+0000 7ff9ac400640 1 Processor -- start 2024-03-28T18:55:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.075+0000 7ff9ac400640 1 -- start start 2024-03-28T18:55:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.077+0000 7f96afe00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f969006bca0 0x7f969006e160 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f96b00c3e80 tx=0x7f96a0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.077+0000 7ff9ac400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9a40c5970 0x7ff9a4158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.077+0000 7ff9ac400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a4157310 0x7ff9a40c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.077+0000 7ff9ac400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a415b020 0x7ff9a40c1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.077+0000 7ff9ac400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a415b610 con 0x7ff9a415b020 2024-03-28T18:55:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.077+0000 7ff9ac400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a415b490 con 0x7ff9a40c5970 2024-03-28T18:55:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.077+0000 7ff9ac400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a40ad560 con 0x7ff9a4157310 2024-03-28T18:55:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.078+0000 7ff9aa400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a415b020 0x7ff9a40c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.078+0000 7ff9a9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9a40c5970 0x7ff9a4158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.078+0000 7ff9a9000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a4157310 0x7ff9a40c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.078+0000 7ff9aa400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a415b020 0x7ff9a40c1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59576/0 (socket says 172.21.15.27:59576) 2024-03-28T18:55:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.078+0000 7ff9aa400640 1 -- 172.21.15.27:0/2282261569 learned_addr learned my addr 172.21.15.27:0/2282261569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.079+0000 7ff9aa400640 1 -- 172.21.15.27:0/2282261569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a4157310 msgr2=0x7ff9a40c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.079+0000 7ff9aa400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a4157310 0x7ff9a40c0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.079+0000 7ff9aa400640 1 -- 172.21.15.27:0/2282261569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9a40c5970 msgr2=0x7ff9a4158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.079+0000 7ff9aa400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9a40c5970 0x7ff9a4158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.079+0000 7ff9aa400640 1 -- 172.21.15.27:0/2282261569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a40c1850 con 0x7ff9a415b020 2024-03-28T18:55:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.080+0000 7ff9aa400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a415b020 0x7ff9a40c1310 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00cd80 tx=0x7ff99c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.080+0000 7f7766a00640 1 -- 172.21.15.27:0/3580060423 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f776801f030 con 0x7f7770145e20 2024-03-28T18:55:54.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.081+0000 7ff9a1600640 1 -- 172.21.15.27:0/2282261569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c00f680 con 0x7ff9a415b020 2024-03-28T18:55:54.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.081+0000 7ff9a1600640 1 -- 172.21.15.27:0/2282261569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c00f820 con 0x7ff9a415b020 2024-03-28T18:55:54.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.081+0000 7ff9a1600640 1 -- 172.21.15.27:0/2282261569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c00fc10 con 0x7ff9a415b020 2024-03-28T18:55:54.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.084+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a40beb00 con 0x7ff9a415b020 2024-03-28T18:55:54.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.084+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a40b0980 con 0x7ff9a415b020 2024-03-28T18:55:54.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.086+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a404f470 con 0x7ff9a415b020 2024-03-28T18:55:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.090+0000 7f96a7e00640 1 -- 172.21.15.27:0/2526711751 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f969c07ac60 con 0x7f96b014f7b0 2024-03-28T18:55:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.095+0000 7ff9a1600640 1 -- 172.21.15.27:0/2282261569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff99c018400 con 0x7ff9a415b020 2024-03-28T18:55:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.101+0000 7ff9a1600640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff98406bca0 0x7ff98406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.102+0000 7ff9a1600640 1 -- 172.21.15.27:0/2282261569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7ff99c08c210 con 0x7ff9a415b020 2024-03-28T18:55:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.102+0000 7ff9a9a00640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff98406bca0 0x7ff98406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.102+0000 7ff9a1600640 1 -- 172.21.15.27:0/2282261569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff99c014030 con 0x7ff9a415b020 2024-03-28T18:55:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.107+0000 7ff9a9a00640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff98406bca0 0x7ff98406e160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff998036000 tx=0x7ff99803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.126+0000 7f606da00640 1 Processor -- start 2024-03-28T18:55:54.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.127+0000 7f606da00640 1 -- start start 2024-03-28T18:55:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.129+0000 7f606da00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f606000fcc0 0x7f60600069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.130+0000 7f606da00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60600d7cd0 con 0x7f60600d78f0 2024-03-28T18:55:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.130+0000 7f6067e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f60600d78f0 0x7f606000ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59764/0 (socket says 172.21.15.27:59764) 2024-03-28T18:55:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.130+0000 7f6067e00640 1 -- 172.21.15.27:0/1290054516 learned_addr learned my addr 172.21.15.27:0/1290054516 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:54.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.130+0000 7f606da00640 1 -- 172.21.15.27:0/1290054516 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60600db220 con 0x7f60600506f0 2024-03-28T18:55:54.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.130+0000 7f606d000640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f606000fcc0 0x7f60600069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.131+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121405873 0 0) 0x7f60600db220 con 0x7f60600506f0 2024-03-28T18:55:54.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.132+0000 7f80f4400640 1 Processor -- start 2024-03-28T18:55:54.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.130+0000 7f606da00640 1 -- 172.21.15.27:0/1290054516 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606004b9d0 con 0x7f606000fcc0 2024-03-28T18:55:54.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.132+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6050003520 con 0x7f60600506f0 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.132+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1346257916 0 0) 0x7f6050003520 con 0x7f60600506f0 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.132+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f606000fcc0 msgr2=0x7f60600069a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.132+0000 7f6066a00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f606000fcc0 0x7f60600069a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.132+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 >> v1:172.21.15.27:6789/0 conn(0x7f60600d78f0 legacy=0x7f606000ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.132+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606004bcb0 con 0x7f60600506f0 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f6054001670 con 0x7f60600506f0 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 >> v1:172.21.15.155:6789/0 conn(0x7f60600506f0 legacy=0x7f6060006290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f6066a00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60500039b0 0x7f6050003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f6066a00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6050004800 0x7f6050004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f6066a00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6050005230 0x7f605000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606004b9d0 con 0x7f60500039b0 2024-03-28T18:55:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6050014900 con 0x7f6050004800 2024-03-28T18:55:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6050014b00 con 0x7f6050005230 2024-03-28T18:55:54.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.134+0000 7f606d000640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6050005230 0x7f605000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.133+0000 7f80f4400640 1 -- start start 2024-03-28T18:55:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.135+0000 7f80f4400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f80ec0c8d70 0x7f80ec0c7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.135+0000 7f80f4400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f80ec0c9280 0x7f80ec0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.135+0000 7f80f4400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0c8300 con 0x7f80ec152d90 2024-03-28T18:55:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.135+0000 7f80f4400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ec0c9840 con 0x7f80ec0c8d70 2024-03-28T18:55:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.135+0000 7f80f4400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ec155770 con 0x7f80ec0c9280 2024-03-28T18:55:54.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.136+0000 7f6067e00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60500039b0 0x7f6050003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.137+0000 7f6067e00640 1 -- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6050005230 msgr2=0x7f605000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.137+0000 7f6067e00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6050005230 0x7f605000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.137+0000 7f6067e00640 1 -- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6050004800 msgr2=0x7f6050004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:55:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.137+0000 7f6067e00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6050004800 0x7f6050004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.137+0000 7f6067e00640 1 -- 172.21.15.27:0/1290054516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606004bcb0 con 0x7f60500039b0 2024-03-28T18:55:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.140+0000 7f6067e00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60500039b0 0x7f6050003db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f605c005130 tx=0x7f6050005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b0bea88ea9b731 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.139+0000 7f80f1a00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f80ec152d90 0x7f80ec0cb340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55668/0 (socket says 172.21.15.27:55668) 2024-03-28T18:55:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.139+0000 7f80f1a00640 1 -- 172.21.15.27:0/3299540223 learned_addr learned my addr 172.21.15.27:0/3299540223 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.139+0000 7f80f1000640 1 --2- 172.21.15.27:0/3299540223 >> v2:172.21.15.27:3300/0 conn(0x7f80ec0c8d70 0x7f80ec0c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.140+0000 7f80f1000640 1 -- 172.21.15.27:0/3299540223 >> v2:172.21.15.155:3300/0 conn(0x7f80ec0c9280 msgr2=0x7f80ec0c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.140+0000 7f80f2400640 1 --2- 172.21.15.27:0/3299540223 >> v2:172.21.15.155:3300/0 conn(0x7f80ec0c9280 0x7f80ec0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.140+0000 7f80f1000640 1 --2- 172.21.15.27:0/3299540223 >> v2:172.21.15.155:3300/0 conn(0x7f80ec0c9280 0x7f80ec0c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.140+0000 7f80f1000640 1 -- 172.21.15.27:0/3299540223 >> v1:172.21.15.155:6789/0 conn(0x7f80ec152d90 legacy=0x7f80ec0cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.140+0000 7f80f1000640 1 -- 172.21.15.27:0/3299540223 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec155a50 con 0x7f80ec0c8d70 2024-03-28T18:55:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.140+0000 7f80f2400640 1 --2- 172.21.15.27:0/3299540223 >> v2:172.21.15.155:3300/0 conn(0x7f80ec0c9280 0x7f80ec0c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.144+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c004d00 con 0x7f60500039b0 2024-03-28T18:55:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.144+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c005ad0 con 0x7f60500039b0 2024-03-28T18:55:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.144+0000 7f80f1000640 1 --2- 172.21.15.27:0/3299540223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec0c7880 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f80ec062b20 tx=0x7f80d8034100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee10257b93dcce87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.144+0000 7f6066a00640 1 -- 172.21.15.27:0/1290054516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c007070 con 0x7f60500039b0 2024-03-28T18:55:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.145+0000 7f606da00640 1 -- 172.21.15.27:0/1290054516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60500039b0 msgr2=0x7f6050003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.145+0000 7f606da00640 1 --2- 172.21.15.27:0/1290054516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60500039b0 0x7f6050003db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f605c005130 tx=0x7f6050005830 comp rx=0 tx=0).stop 2024-03-28T18:55:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.145+0000 7f606d000640 1 -- 172.21.15.27:0/1290054516 reap_dead start 2024-03-28T18:55:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.145+0000 7f606da00640 1 -- 172.21.15.27:0/1290054516 shutdown_connections 2024-03-28T18:55:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.145+0000 7f606da00640 1 -- 172.21.15.27:0/1290054516 >> 172.21.15.27:0/1290054516 conn(0x7f606004fbd0 msgr2=0x7f606004ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.145+0000 7f80ebe00640 1 -- 172.21.15.27:0/3299540223 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8041070 con 0x7f80ec0c8d70 2024-03-28T18:55:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.145+0000 7f80ebe00640 1 -- 172.21.15.27:0/3299540223 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d8034c20 con 0x7f80ec0c8d70 2024-03-28T18:55:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.146+0000 7f606da00640 1 -- 172.21.15.27:0/1290054516 shutdown_connections 2024-03-28T18:55:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.145+0000 7f80ebe00640 1 -- 172.21.15.27:0/3299540223 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8041070 con 0x7f80ec0c8d70 2024-03-28T18:55:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.146+0000 7f80f4400640 1 -- 172.21.15.27:0/3299540223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 msgr2=0x7f80ec0c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.146+0000 7f80f4400640 1 --2- 172.21.15.27:0/3299540223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec0c7880 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f80ec062b20 tx=0x7f80d8034100 comp rx=0 tx=0).stop 2024-03-28T18:55:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.147+0000 7f80f4400640 1 -- 172.21.15.27:0/3299540223 shutdown_connections 2024-03-28T18:55:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.146+0000 7f606da00640 1 -- 172.21.15.27:0/1290054516 wait complete. 2024-03-28T18:55:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.147+0000 7f80f4400640 1 --2- 172.21.15.27:0/3299540223 >> v2:172.21.15.155:3300/0 conn(0x7f80ec0c9280 0x7f80ec0c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.147+0000 7f80f4400640 1 --2- 172.21.15.27:0/3299540223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec0c7880 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.147+0000 7f80f4400640 1 -- 172.21.15.27:0/3299540223 >> 172.21.15.27:0/3299540223 conn(0x7f80ec06b4e0 msgr2=0x7f80ec06b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.149+0000 7f80f4400640 1 -- 172.21.15.27:0/3299540223 shutdown_connections 2024-03-28T18:55:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.149+0000 7f80f4400640 1 -- 172.21.15.27:0/3299540223 wait complete. 2024-03-28T18:55:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.149+0000 7f606da00640 1 Processor -- start 2024-03-28T18:55:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.150+0000 7f606da00640 1 -- start start 2024-03-28T18:55:54.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.152+0000 7f606da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60500039b0 0x7f6060030430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.153+0000 7f606da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606016f640 0x7f60600db690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.153+0000 7f606da00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60600dbbd0 0x7f60600cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.153+0000 7f6067e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60500039b0 0x7f6060030430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.153+0000 7f6067e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60500039b0 0x7f6060030430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43680/0 (socket says 172.21.15.27:43680) 2024-03-28T18:55:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.153+0000 7f6067e00640 1 -- 172.21.15.27:0/930528371 learned_addr learned my addr 172.21.15.27:0/930528371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.153+0000 7f6067400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606016f640 0x7f60600db690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.154+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606000f490 con 0x7f606016f640 2024-03-28T18:55:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.154+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606004c270 con 0x7f60500039b0 2024-03-28T18:55:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.154+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606000f610 con 0x7f60600dbbd0 2024-03-28T18:55:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.154+0000 7f606d000640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60600dbbd0 0x7f60600cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.155+0000 7f606d000640 1 -- 172.21.15.27:0/930528371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60500039b0 msgr2=0x7f6060030430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.155+0000 7f606d000640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60500039b0 0x7f6060030430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.155+0000 7f606d000640 1 -- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606016f640 msgr2=0x7f60600db690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.155+0000 7f606d000640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606016f640 0x7f60600db690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.155+0000 7f606d000640 1 -- 172.21.15.27:0/930528371 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60600d16e0 con 0x7f60600dbbd0 2024-03-28T18:55:54.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.156+0000 7f606d000640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60600dbbd0 0x7f60600cc8f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f60680501b0 tx=0x7f6068068c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.156+0000 7f80f4400640 1 Processor -- start 2024-03-28T18:55:54.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.156+0000 7f605be00640 1 -- 172.21.15.27:0/930528371 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606809a070 con 0x7f60600dbbd0 2024-03-28T18:55:54.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.156+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606016fad0 con 0x7f60600dbbd0 2024-03-28T18:55:54.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.156+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60600d4610 con 0x7f60600dbbd0 2024-03-28T18:55:54.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.157+0000 7f605be00640 1 -- 172.21.15.27:0/930528371 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6068095410 con 0x7f60600dbbd0 2024-03-28T18:55:54.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.157+0000 7f605be00640 1 -- 172.21.15.27:0/930528371 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606809a070 con 0x7f60600dbbd0 2024-03-28T18:55:54.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.157+0000 7f80f4400640 1 -- start start 2024-03-28T18:55:54.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.159+0000 7f80f4400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec1594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.159+0000 7f80f4400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80ec0c9280 0x7f80ec0c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.159+0000 7f80f4400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80ec152d90 0x7f80ec1e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.159+0000 7f80f4400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec156530 con 0x7f80ec0c8d70 2024-03-28T18:55:54.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.159+0000 7f80f4400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec156010 con 0x7f80ec152d90 2024-03-28T18:55:54.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.159+0000 7f80f4400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec072b00 con 0x7f80ec0c9280 2024-03-28T18:55:54.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.159+0000 7f80f1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec1594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.160+0000 7f80f1000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80ec0c9280 0x7f80ec0c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.160+0000 7f80f1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec1594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59612/0 (socket says 172.21.15.27:59612) 2024-03-28T18:55:54.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.160+0000 7f80f1a00640 1 -- 172.21.15.27:0/2064390971 learned_addr learned my addr 172.21.15.27:0/2064390971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:54.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.160+0000 7f80f1a00640 1 -- 172.21.15.27:0/2064390971 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80ec0c9280 msgr2=0x7f80ec0c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.160+0000 7f80f1a00640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80ec0c9280 0x7f80ec0c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.160+0000 7f80f1a00640 1 -- 172.21.15.27:0/2064390971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80ec152d90 msgr2=0x7f80ec1e7f60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:55:54.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.160+0000 7f80f1a00640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80ec152d90 0x7f80ec1e7f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.160+0000 7f80f1a00640 1 -- 172.21.15.27:0/2064390971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec1599e0 con 0x7f80ec0c8d70 2024-03-28T18:55:54.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.161+0000 7f80f1a00640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec1594a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f80e000c750 tx=0x7f80e000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.162+0000 7f80e9600640 1 -- 172.21.15.27:0/2064390971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0017070 con 0x7f80ec0c8d70 2024-03-28T18:55:54.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.162+0000 7f80e9600640 1 -- 172.21.15.27:0/2064390971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e0005ae0 con 0x7f80ec0c8d70 2024-03-28T18:55:54.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.162+0000 7f80e9600640 1 -- 172.21.15.27:0/2064390971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0007070 con 0x7f80ec0c8d70 2024-03-28T18:55:54.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.161+0000 7f605be00640 1 -- 172.21.15.27:0/930528371 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f606809d070 con 0x7f60600dbbd0 2024-03-28T18:55:54.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.162+0000 7f605be00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f604806bdd0 0x7f604806e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.162+0000 7f605be00640 1 -- 172.21.15.27:0/930528371 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f6068071040 con 0x7f60600dbbd0 2024-03-28T18:55:54.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.163+0000 7f6067e00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f604806bdd0 0x7f604806e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.163+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ec0c2af0 con 0x7f80ec0c8d70 2024-03-28T18:55:54.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.164+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec0c30b0 con 0x7f80ec0c8d70 2024-03-28T18:55:54.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.164+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6044005290 con 0x7f60600dbbd0 2024-03-28T18:55:54.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.166+0000 7f6067e00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f604806bdd0 0x7f604806e290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f60600d12f0 tx=0x7f605c009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.169+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ec04f400 con 0x7f80ec0c8d70 2024-03-28T18:55:54.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.170+0000 7f80e9600640 1 -- 172.21.15.27:0/2064390971 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f80e001a070 con 0x7f80ec0c8d70 2024-03-28T18:55:54.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.171+0000 7f80e9600640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80cc06bca0 0x7f80cc06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:54.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.171+0000 7f80e9600640 1 -- 172.21.15.27:0/2064390971 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f80e008a190 con 0x7f80ec0c8d70 2024-03-28T18:55:54.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.178+0000 7f80f1000640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80cc06bca0 0x7f80cc06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:54.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.180+0000 7f80e9600640 1 -- 172.21.15.27:0/2064390971 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f80e0058370 con 0x7f80ec0c8d70 2024-03-28T18:55:54.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.180+0000 7f80f1000640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80cc06bca0 0x7f80cc06e160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f80d803e000 tx=0x7f80d8038000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:54.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.181+0000 7f605be00640 1 -- 172.21.15.27:0/930528371 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f60600d16e0 con 0x7f60600dbbd0 2024-03-28T18:55:54.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:55:54.478 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:55:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.501+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc618005850 con 0x7fc6380d7920 2024-03-28T18:55:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.502+0000 7fc636000640 1 -- 172.21.15.27:0/1278826176 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc62c07e700 con 0x7fc6380d7920 2024-03-28T18:55:54.502 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.506+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62006bc20 msgr2=0x7fc62006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.506+0000 7fc63f200640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62006bc20 0x7fc62006e0e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc63804dfd0 tx=0x7fc630009040 comp rx=0 tx=0).stop 2024-03-28T18:55:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.506+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6380d7920 msgr2=0x7fc6380d2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.506+0000 7fc63f200640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6380d7920 0x7fc6380d2e40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009960 tx=0x7fc62c0046c0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.507+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 shutdown_connections 2024-03-28T18:55:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.507+0000 7fc63f200640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62006bc20 0x7fc62006e0e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.507+0000 7fc63f200640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6380d7920 0x7fc6380d2e40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.507+0000 7fc63f200640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc638050720 0x7fc638173c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.507+0000 7fc63f200640 1 --2- 172.21.15.27:0/1278826176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc63800fcf0 0x7fc63804e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.507+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 >> 172.21.15.27:0/1278826176 conn(0x7fc63804fc00 msgr2=0x7fc638006db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.507+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 shutdown_connections 2024-03-28T18:55:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.507+0000 7fc63f200640 1 -- 172.21.15.27:0/1278826176 wait complete. 2024-03-28T18:55:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.515+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fec84067cb0 con 0x7fec840c53c0 2024-03-28T18:55:54.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.516+0000 7fec7be00640 1 -- 172.21.15.27:0/1634473781 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec74058ee0 con 0x7fec840c53c0 2024-03-28T18:55:54.519 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.522+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f96b006e620 con 0x7f96b014f7b0 2024-03-28T18:55:54.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec6406bbd0 msgr2=0x7fec6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec6406bbd0 0x7fec6406e090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fec70005880 tx=0x7fec70009040 comp rx=0 tx=0).stop 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec840c53c0 msgr2=0x7fec841f7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec840c53c0 0x7fec841f7c40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fec74004810 tx=0x7fec74004cd0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 shutdown_connections 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec6406bbd0 0x7fec6406e090 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec840be9f0 0x7fec8408a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec8415eff0 0x7fec840be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 --2- 172.21.15.27:0/1634473781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec840c53c0 0x7fec841f7c40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.523+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 >> 172.21.15.27:0/1634473781 conn(0x7fec840c6da0 msgr2=0x7fec84160320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.528+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 shutdown_connections 2024-03-28T18:55:54.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.528+0000 7fec8b000640 1 -- 172.21.15.27:0/1634473781 wait complete. 2024-03-28T18:55:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.528+0000 7f96a7e00640 1 -- 172.21.15.27:0/2526711751 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f969c07ed00 con 0x7f96b014f7b0 2024-03-28T18:55:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.528+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7efd08005850 con 0x7efd300c1db0 2024-03-28T18:55:54.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.529+0000 7efd2cc00640 1 -- 172.21.15.27:0/846508520 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd20058790 con 0x7efd300c1db0 2024-03-28T18:55:54.529 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2024-03-28T18:55:54.529 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-28T18:55:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.531+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd0c06bbd0 msgr2=0x7efd0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.531+0000 7efd37a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd0c06bbd0 0x7efd0c06e090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd280015f0 tx=0x7efd28007a40 comp rx=0 tx=0).stop 2024-03-28T18:55:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.531+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd300c1db0 msgr2=0x7efd301538d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.531+0000 7efd37a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd300c1db0 0x7efd301538d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efd2000c430 tx=0x7efd2000c8f0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f969006bca0 msgr2=0x7f969006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f969006bca0 0x7f969006e160 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f96b00c3e80 tx=0x7f96a0008040 comp rx=0 tx=0).stop 2024-03-28T18:55:54.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b014f7b0 msgr2=0x7f96b01528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b014f7b0 0x7f96b01528c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f969c000c00 tx=0x7f969c004730 comp rx=0 tx=0).stop 2024-03-28T18:55:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.533+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 shutdown_connections 2024-03-28T18:55:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.533+0000 7efd37a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd0c06bbd0 0x7efd0c06e090 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.533+0000 7efd37a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd300c1db0 0x7efd301538d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.533+0000 7efd37a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd300abf10 0x7efd30153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.533+0000 7efd37a00640 1 --2- 172.21.15.27:0/846508520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd3006db40 0x7efd30155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.533+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 >> 172.21.15.27:0/846508520 conn(0x7efd300c6da0 msgr2=0x7efd30147a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 shutdown_connections 2024-03-28T18:55:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7efd37a00640 1 -- 172.21.15.27:0/846508520 wait complete. 2024-03-28T18:55:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 shutdown_connections 2024-03-28T18:55:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f969006bca0 0x7f969006e160 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f96b00c3e80 tx=0x7f96a0008040 comp rx=0 tx=0).stop 2024-03-28T18:55:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96b0155e30 0x7f96b0152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b014f7b0 0x7f96b01528c0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f969c000c00 tx=0x7f969c004730 comp rx=0 tx=0).stop 2024-03-28T18:55:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 --2- 172.21.15.27:0/2526711751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b006fb40 0x7f96b0149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.534+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 >> 172.21.15.27:0/2526711751 conn(0x7f96b00c48b0 msgr2=0x7f96b015f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.535+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 shutdown_connections 2024-03-28T18:55:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.535+0000 7f96b6a00640 1 -- 172.21.15.27:0/2526711751 wait complete. 2024-03-28T18:55:54.540 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.0 2024-03-28T18:55:54.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.4 2024-03-28T18:55:54.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.565+0000 7f775f400640 1 -- 172.21.15.27:0/3580060423 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7748005850 con 0x7f7770145e20 2024-03-28T18:55:54.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.567+0000 7f7766a00640 1 -- 172.21.15.27:0/3580060423 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7768088e20 con 0x7f7770145e20 2024-03-28T18:55:54.567 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-28T18:55:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.570+0000 7f775f400640 1 -- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f774c06bbd0 msgr2=0x7f774c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.570+0000 7f775f400640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f774c06bbd0 0x7f774c06e090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f77600072e0 tx=0x7f7760009040 comp rx=0 tx=0).stop 2024-03-28T18:55:54.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.570+0000 7f775f400640 1 -- 172.21.15.27:0/3580060423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7770145e20 msgr2=0x7f77701462b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.570+0000 7f775f400640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7770145e20 0x7f77701462b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f776800b410 tx=0x7f776800b8d0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.572 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.2 2024-03-28T18:55:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.572+0000 7f775f400640 1 -- 172.21.15.27:0/3580060423 shutdown_connections 2024-03-28T18:55:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.572+0000 7f775f400640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f774c06bbd0 0x7f774c06e090 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.572+0000 7f775f400640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7770145e20 0x7f77701462b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.572+0000 7f775f400640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7770152ec0 0x7f77700c2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.572+0000 7f775f400640 1 --2- 172.21.15.27:0/3580060423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77700c8ff0 0x7f7770151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.572+0000 7f775f400640 1 -- 172.21.15.27:0/3580060423 >> 172.21.15.27:0/3580060423 conn(0x7f77700671f0 msgr2=0x7f77700ad040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.573+0000 7f775f400640 1 -- 172.21.15.27:0/3580060423 shutdown_connections 2024-03-28T18:55:54.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.573+0000 7f775f400640 1 -- 172.21.15.27:0/3580060423 wait complete. 2024-03-28T18:55:54.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.5 2024-03-28T18:55:54.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.1 2024-03-28T18:55:54.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.592+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6044005580 con 0x7f60600dbbd0 2024-03-28T18:55:54.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.593+0000 7f605be00640 1 -- 172.21.15.27:0/930528371 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6068068680 con 0x7f60600dbbd0 2024-03-28T18:55:54.592 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-28T18:55:54.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f604806bdd0 msgr2=0x7f604806e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f604806bdd0 0x7f604806e290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f60600d12f0 tx=0x7f605c009030 comp rx=0 tx=0).stop 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60600dbbd0 msgr2=0x7f60600cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60600dbbd0 0x7f60600cc8f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f60680501b0 tx=0x7f6068068c60 comp rx=0 tx=0).stop 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 shutdown_connections 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f604806bdd0 0x7f604806e290 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60600dbbd0 0x7f60600cc8f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f606016f640 0x7f60600db690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 --2- 172.21.15.27:0/930528371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60500039b0 0x7f6060030430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 >> 172.21.15.27:0/930528371 conn(0x7f606004fbd0 msgr2=0x7f60600d88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 shutdown_connections 2024-03-28T18:55:54.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.596+0000 7f606da00640 1 -- 172.21.15.27:0/930528371 wait complete. 2024-03-28T18:55:54.613 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.3 2024-03-28T18:55:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.656+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff9a406d510 con 0x7ff9a415b020 2024-03-28T18:55:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.656+0000 7ff9a1600640 1 -- 172.21.15.27:0/2282261569 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff99c05a9b0 con 0x7ff9a415b020 2024-03-28T18:55:54.656 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff98406bca0 msgr2=0x7ff98406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff98406bca0 0x7ff98406e160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff998036000 tx=0x7ff99803d040 comp rx=0 tx=0).stop 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a415b020 msgr2=0x7ff9a40c1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a415b020 0x7ff9a40c1310 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00cd80 tx=0x7ff99c00a650 comp rx=0 tx=0).stop 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 shutdown_connections 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff98406bca0 0x7ff98406e160 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a415b020 0x7ff9a40c1310 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff9a4157310 0x7ff9a40c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 --2- 172.21.15.27:0/2282261569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9a40c5970 0x7ff9a4158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 >> 172.21.15.27:0/2282261569 conn(0x7ff9a40671f0 msgr2=0x7ff9a40ab850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 shutdown_connections 2024-03-28T18:55:54.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.659+0000 7ff9ac400640 1 -- 172.21.15.27:0/2282261569 wait complete. 2024-03-28T18:55:54.680 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.6 2024-03-28T18:55:54.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.709+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f80ec0728e0 con 0x7f80ec0c8d70 2024-03-28T18:55:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.709+0000 7f80e9600640 1 -- 172.21.15.27:0/2064390971 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f80e005c410 con 0x7f80ec0c8d70 2024-03-28T18:55:54.709 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.711+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80cc06bca0 msgr2=0x7f80cc06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.711+0000 7f80f4400640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80cc06bca0 0x7f80cc06e160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f80d803e000 tx=0x7f80d8038000 comp rx=0 tx=0).stop 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.711+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 msgr2=0x7f80ec1594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.711+0000 7f80f4400640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec1594a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f80e000c750 tx=0x7f80e000cc10 comp rx=0 tx=0).stop 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.712+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 shutdown_connections 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.712+0000 7f80f4400640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80cc06bca0 0x7f80cc06e160 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.712+0000 7f80f4400640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80ec152d90 0x7f80ec1e7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.712+0000 7f80f4400640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80ec0c9280 0x7f80ec0c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.712+0000 7f80f4400640 1 --2- 172.21.15.27:0/2064390971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80ec0c8d70 0x7f80ec1594a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.712+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 >> 172.21.15.27:0/2064390971 conn(0x7f80ec06b4e0 msgr2=0x7f80ec14e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.712+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 shutdown_connections 2024-03-28T18:55:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:54.712+0000 7f80f4400640 1 -- 172.21.15.27:0/2064390971 wait complete. 2024-03-28T18:55:54.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.7 2024-03-28T18:55:55.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T18:55:55.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T18:55:55.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T18:55:55.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T18:55:55.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T18:55:55.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T18:55:55.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T18:55:55.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T18:55:55.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.841+0000 7f3bffa00640 1 Processor -- start 2024-03-28T18:55:55.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.842+0000 7f3bffa00640 1 -- start start 2024-03-28T18:55:55.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.844+0000 7f3bffa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf00071b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.844+0000 7f3bffa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bf004dff0 0x7f3bf004e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.844+0000 7f3bffa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf000f9d0 con 0x7f3bf004dae0 2024-03-28T18:55:55.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.844+0000 7f3bffa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf00da4b0 con 0x7f3bf004dff0 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.844+0000 7f3bffa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00da790 con 0x7f3bf00d7ae0 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.845+0000 7f3bf7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bf004dff0 0x7f3bf004e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.845+0000 7f3bf7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bf004dff0 0x7f3bf004e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59620/0 (socket says 172.21.15.27:59620) 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.845+0000 7f3bf7e00640 1 -- 172.21.15.27:0/2259036405 learned_addr learned my addr 172.21.15.27:0/2259036405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.846+0000 7f3bfd000640 1 --2- 172.21.15.27:0/2259036405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf00071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.847+0000 7f3bf7e00640 1 -- 172.21.15.27:0/2259036405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 msgr2=0x7f3bf00071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.847+0000 7f3bf7e00640 1 --2- 172.21.15.27:0/2259036405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf00071b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.847+0000 7f3bf7e00640 1 -- 172.21.15.27:0/2259036405 >> v1:172.21.15.27:6789/0 conn(0x7f3bf004dae0 legacy=0x7f3bf004ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.847+0000 7f3bf7e00640 1 -- 172.21.15.27:0/2259036405 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf00daa70 con 0x7f3bf004dff0 2024-03-28T18:55:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.849+0000 7f3bfd000640 1 --2- 172.21.15.27:0/2259036405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf00071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:55.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.849+0000 7f3bf7e00640 1 --2- 172.21.15.27:0/2259036405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dff0 0x7f3bf004e7d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3be400ffd0 tx=0x7f3be40349e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c9975b96b9e7df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.851+0000 7f3bf7400640 1 -- 172.21.15.27:0/2259036405 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4038800 con 0x7f3bf004dff0 2024-03-28T18:55:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.852+0000 7f3bf7400640 1 -- 172.21.15.27:0/2259036405 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be40389a0 con 0x7f3bf004dff0 2024-03-28T18:55:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.852+0000 7f3bf7400640 1 -- 172.21.15.27:0/2259036405 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4038db0 con 0x7f3bf004dff0 2024-03-28T18:55:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.856+0000 7f3bffa00640 1 -- 172.21.15.27:0/2259036405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dff0 msgr2=0x7f3bf004e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.856+0000 7f3bffa00640 1 --2- 172.21.15.27:0/2259036405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dff0 0x7f3bf004e7d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3be400ffd0 tx=0x7f3be40349e0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.856+0000 7f3bffa00640 1 -- 172.21.15.27:0/2259036405 shutdown_connections 2024-03-28T18:55:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.856+0000 7f3bffa00640 1 --2- 172.21.15.27:0/2259036405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dff0 0x7f3bf004e7d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.856+0000 7f3bffa00640 1 --2- 172.21.15.27:0/2259036405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf00071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.856+0000 7f3bffa00640 1 -- 172.21.15.27:0/2259036405 >> 172.21.15.27:0/2259036405 conn(0x7f3bf004c660 msgr2=0x7f3bf004ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.857+0000 7f3bffa00640 1 -- 172.21.15.27:0/2259036405 shutdown_connections 2024-03-28T18:55:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.857+0000 7f3bffa00640 1 -- 172.21.15.27:0/2259036405 wait complete. 2024-03-28T18:55:55.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.861+0000 7f3bffa00640 1 Processor -- start 2024-03-28T18:55:55.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.862+0000 7f3bffa00640 1 -- start start 2024-03-28T18:55:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.864+0000 7f3bffa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dae0 0x7f3bf00d8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.864+0000 7f3bfd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dae0 0x7f3bf00d8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.865+0000 7f3bfd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dae0 0x7f3bf00d8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59622/0 (socket says 172.21.15.27:59622) 2024-03-28T18:55:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.865+0000 7f3bfd000640 1 -- 172.21.15.27:0/2446250014 learned_addr learned my addr 172.21.15.27:0/2446250014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.865+0000 7f3bffa00640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bf004dff0 0x7f3bf00301f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.865+0000 7f3bffa00640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf0030730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.866+0000 7f3bffa00640 1 -- 172.21.15.27:0/2446250014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf004e4b0 con 0x7f3bf004dae0 2024-03-28T18:55:55.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.866+0000 7f3bf7e00640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bf004dff0 0x7f3bf00301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.866+0000 7f3bffa00640 1 -- 172.21.15.27:0/2446250014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf00db270 con 0x7f3bf004dff0 2024-03-28T18:55:55.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.866+0000 7f3bfda00640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf0030730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.867+0000 7f3bffa00640 1 -- 172.21.15.27:0/2446250014 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf000fdc0 con 0x7f3bf00d7ae0 2024-03-28T18:55:55.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.868+0000 7f3bfd000640 1 -- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 msgr2=0x7f3bf0030730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.868+0000 7f3bfd000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf0030730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.868+0000 7f3bfd000640 1 -- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bf004dff0 msgr2=0x7f3bf00301f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.868+0000 7f3bfd000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bf004dff0 0x7f3bf00301f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.868+0000 7f3bfd000640 1 -- 172.21.15.27:0/2446250014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0046f70 con 0x7f3bf004dae0 2024-03-28T18:55:55.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.870+0000 7f3bfd000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dae0 0x7f3bf00d8f90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3bf804ff90 tx=0x7f3bf8068c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.870+0000 7f3bf4c00640 1 -- 172.21.15.27:0/2446250014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80724d0 con 0x7f3bf004dae0 2024-03-28T18:55:55.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.870+0000 7f3bffa00640 1 -- 172.21.15.27:0/2446250014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf00caf10 con 0x7f3bf004dae0 2024-03-28T18:55:55.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.871+0000 7f3bffa00640 1 -- 172.21.15.27:0/2446250014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf00d86b0 con 0x7f3bf004dae0 2024-03-28T18:55:55.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.872+0000 7f3bf4c00640 1 -- 172.21.15.27:0/2446250014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf808f030 con 0x7f3bf004dae0 2024-03-28T18:55:55.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.872+0000 7f3bf4c00640 1 -- 172.21.15.27:0/2446250014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80724d0 con 0x7f3bf004dae0 2024-03-28T18:55:55.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.876+0000 7f3bf4c00640 1 -- 172.21.15.27:0/2446250014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3bf80726b0 con 0x7f3bf004dae0 2024-03-28T18:55:55.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.876+0000 7f3bea000640 1 -- 172.21.15.27:0/2446250014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf00057e0 con 0x7f3bf004dae0 2024-03-28T18:55:55.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.878+0000 7f3bf4c00640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bd806bdd0 0x7f3bd806e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.878+0000 7f3bf7e00640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bd806bdd0 0x7f3bd806e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.879+0000 7f3bf4c00640 1 -- 172.21.15.27:0/2446250014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f3bf8105e80 con 0x7f3bf004dae0 2024-03-28T18:55:55.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.884+0000 7f3bf7e00640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bd806bdd0 0x7f3bd806e290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3be403f000 tx=0x7f3be403a000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.892+0000 7f3bf4c00640 1 -- 172.21.15.27:0/2446250014 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3bf00caf10 con 0x7f3bf004dae0 2024-03-28T18:55:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.900+0000 7f5bef800640 1 Processor -- start 2024-03-28T18:55:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.900+0000 7f5bef800640 1 -- start start 2024-03-28T18:55:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.902+0000 7f5bef800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5be80c1db0 0x7f5be80c2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.902+0000 7f5bef800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be806db40 0x7f5be806df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.902+0000 7f5bef800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be8152780 con 0x7f5be80abf10 2024-03-28T18:55:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.903+0000 7f5bef800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be81522f0 con 0x7f5be80c1db0 2024-03-28T18:55:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.903+0000 7f5bef800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be81519c0 con 0x7f5be806db40 2024-03-28T18:55:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.903+0000 7f5be7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5be80c1db0 0x7f5be80c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.903+0000 7f5be7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5be80c1db0 0x7f5be80c2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59628/0 (socket says 172.21.15.27:59628) 2024-03-28T18:55:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.903+0000 7f5be7e00640 1 -- 172.21.15.27:0/1672604673 learned_addr learned my addr 172.21.15.27:0/1672604673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.903+0000 7f5bece00640 1 --1- 172.21.15.27:0/1672604673 >> v1:172.21.15.27:6789/0 conn(0x7f5be80abf10 0x7f5be8071510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59790/0 (socket says 172.21.15.27:59790) 2024-03-28T18:55:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.905+0000 7ff174000640 1 Processor -- start 2024-03-28T18:55:55.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.906+0000 7ff174000640 1 -- start start 2024-03-28T18:55:55.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.906+0000 7f5be7400640 1 -- 172.21.15.27:0/1672604673 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192485231 0 0) 0x7f5be8152780 con 0x7f5be80abf10 2024-03-28T18:55:55.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.906+0000 7f5be7400640 1 -- 172.21.15.27:0/1672604673 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bcc005730 con 0x7f5be80abf10 2024-03-28T18:55:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.909+0000 7ff174000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff16c0c1db0 0x7ff16c0c2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.910+0000 7ff174000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff16c152780 con 0x7ff16c06db40 2024-03-28T18:55:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.910+0000 7ff174000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff16c1519c0 con 0x7ff16c0abf10 2024-03-28T18:55:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.910+0000 7ff174000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff16c1522f0 con 0x7ff16c0c1db0 2024-03-28T18:55:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.910+0000 7ff172000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff16c06db40 0x7ff16c06df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59798/0 (socket says 172.21.15.27:59798) 2024-03-28T18:55:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.911+0000 7ff172000640 1 -- 172.21.15.27:0/1046324287 learned_addr learned my addr 172.21.15.27:0/1046324287 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.911+0000 7ff170c00640 1 --2- 172.21.15.27:0/1046324287 >> v2:172.21.15.155:3300/0 conn(0x7ff16c0c1db0 0x7ff16c0c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.912+0000 7ff16be00640 1 -- 172.21.15.27:0/1046324287 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633301467 0 0) 0x7ff16c1519c0 con 0x7ff16c0abf10 2024-03-28T18:55:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.912+0000 7ff16be00640 1 -- 172.21.15.27:0/1046324287 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff154003520 con 0x7ff16c0abf10 2024-03-28T18:55:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.912+0000 7ff16be00640 1 -- 172.21.15.27:0/1046324287 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913640817 0 0) 0x7ff16c152780 con 0x7ff16c06db40 2024-03-28T18:55:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.912+0000 7ff16be00640 1 -- 172.21.15.27:0/1046324287 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff16c1519c0 con 0x7ff16c06db40 2024-03-28T18:55:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.913+0000 7ff170c00640 1 -- 172.21.15.27:0/1046324287 >> v1:172.21.15.155:6789/0 conn(0x7ff16c0abf10 legacy=0x7ff16c071510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.913+0000 7ff170c00640 1 -- 172.21.15.27:0/1046324287 >> v1:172.21.15.27:6789/0 conn(0x7ff16c06db40 legacy=0x7ff16c06df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.913+0000 7ff170c00640 1 -- 172.21.15.27:0/1046324287 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16c151ca0 con 0x7ff16c0c1db0 2024-03-28T18:55:55.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.915+0000 7ff170c00640 1 --2- 172.21.15.27:0/1046324287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff16c0c1db0 0x7ff16c0c2210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff16c152470 tx=0x7ff16402d1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c0a222cea6b987b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.916+0000 7ff16be00640 1 -- 172.21.15.27:0/1046324287 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16403a070 con 0x7ff16c0c1db0 2024-03-28T18:55:55.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.918+0000 7ff16be00640 1 -- 172.21.15.27:0/1046324287 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff164035440 con 0x7ff16c0c1db0 2024-03-28T18:55:55.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.918+0000 7ff16be00640 1 -- 172.21.15.27:0/1046324287 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff164035810 con 0x7ff16c0c1db0 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.909+0000 7f5be7e00640 1 -- 172.21.15.27:0/1672604673 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be806db40 msgr2=0x7f5be806df90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.909+0000 7f5be7e00640 1 --2- 172.21.15.27:0/1672604673 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be806db40 0x7f5be806df90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.909+0000 7f5be7e00640 1 -- 172.21.15.27:0/1672604673 >> v1:172.21.15.27:6789/0 conn(0x7f5be80abf10 legacy=0x7f5be8071510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.909+0000 7f5be7e00640 1 -- 172.21.15.27:0/1672604673 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be8151ca0 con 0x7f5be80c1db0 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.912+0000 7f5be7e00640 1 --2- 172.21.15.27:0/1672604673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80c1db0 0x7f5be80c2210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009bd0 tx=0x7f5be002c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd2102928ff5db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.914+0000 7f5be7400640 1 -- 172.21.15.27:0/1672604673 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be002ce00 con 0x7f5be80c1db0 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.914+0000 7f5be7400640 1 -- 172.21.15.27:0/1672604673 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be00343f0 con 0x7f5be80c1db0 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.914+0000 7f5be7400640 1 -- 172.21.15.27:0/1672604673 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be00347c0 con 0x7f5be80c1db0 2024-03-28T18:55:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.915+0000 7f5bef800640 1 -- 172.21.15.27:0/1672604673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80c1db0 msgr2=0x7f5be80c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.915+0000 7f5bef800640 1 --2- 172.21.15.27:0/1672604673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80c1db0 0x7f5be80c2210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009bd0 tx=0x7f5be002c260 comp rx=0 tx=0).stop 2024-03-28T18:55:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.916+0000 7f5bef800640 1 -- 172.21.15.27:0/1672604673 shutdown_connections 2024-03-28T18:55:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.917+0000 7f791ea00640 1 Processor -- start 2024-03-28T18:55:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.915+0000 7f36eee00640 1 Processor -- start 2024-03-28T18:55:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.916+0000 7f36eee00640 1 -- start start 2024-03-28T18:55:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.918+0000 7f36eee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f36e8157470 0x7f36e80cb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.918+0000 7f36eee00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f36e80c8f30 0x7f36e815d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.918+0000 7f36eee00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e8159e00 con 0x7f36e80c9650 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.918+0000 7f36eee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e815a0e0 con 0x7f36e8157470 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.918+0000 7f36eee00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e815a3c0 con 0x7f36e80c8f30 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.916+0000 7f5bef800640 1 --2- 172.21.15.27:0/1672604673 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be806db40 0x7f5be806df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.916+0000 7f5bef800640 1 --2- 172.21.15.27:0/1672604673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80c1db0 0x7f5be80c2210 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.916+0000 7f5bef800640 1 -- 172.21.15.27:0/1672604673 >> 172.21.15.27:0/1672604673 conn(0x7f5be80c6da0 msgr2=0x7f5be80c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.917+0000 7f5bef800640 1 -- 172.21.15.27:0/1672604673 shutdown_connections 2024-03-28T18:55:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.917+0000 7f5bef800640 1 -- 172.21.15.27:0/1672604673 wait complete. 2024-03-28T18:55:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.919+0000 7f791ea00640 1 -- start start 2024-03-28T18:55:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.919+0000 7f36e7400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f36e80c8f30 0x7f36e815d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.919+0000 7f36e7400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f36e80c8f30 0x7f36e815d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43728/0 (socket says 172.21.15.27:43728) 2024-03-28T18:55:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.919+0000 7f36e7400640 1 -- 172.21.15.27:0/2876868602 learned_addr learned my addr 172.21.15.27:0/2876868602 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.920+0000 7f5bef800640 1 Processor -- start 2024-03-28T18:55:55.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.920+0000 7f5bef800640 1 -- start start 2024-03-28T18:55:55.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.920+0000 7ff174000640 1 -- 172.21.15.27:0/1046324287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff16c0c1db0 msgr2=0x7ff16c0c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.920+0000 7ff174000640 1 --2- 172.21.15.27:0/1046324287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff16c0c1db0 0x7ff16c0c2210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff16c152470 tx=0x7ff16402d1b0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.921+0000 7ff174000640 1 -- 172.21.15.27:0/1046324287 shutdown_connections 2024-03-28T18:55:55.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.921+0000 7ff174000640 1 --2- 172.21.15.27:0/1046324287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff16c0c1db0 0x7ff16c0c2210 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.921+0000 7ff174000640 1 -- 172.21.15.27:0/1046324287 >> 172.21.15.27:0/1046324287 conn(0x7ff16c0c6da0 msgr2=0x7ff16c0c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:55.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.920+0000 7f36e7400640 1 -- 172.21.15.27:0/2876868602 >> v2:172.21.15.27:3300/0 conn(0x7f36e8157470 msgr2=0x7f36e80cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:55.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.920+0000 7f36e7400640 1 --2- 172.21.15.27:0/2876868602 >> v2:172.21.15.27:3300/0 conn(0x7f36e8157470 0x7f36e80cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.920+0000 7f36e7400640 1 -- 172.21.15.27:0/2876868602 >> v1:172.21.15.27:6789/0 conn(0x7f36e80c9650 legacy=0x7f36e815d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:55.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.920+0000 7f36e7400640 1 -- 172.21.15.27:0/2876868602 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e815a6a0 con 0x7f36e80c8f30 2024-03-28T18:55:55.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.921+0000 7f791ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f791814f920 0x7f79180c9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.922+0000 7f5bef800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5be806db40 0x7f5be80ab230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.923+0000 7f36e7400640 1 --2- 172.21.15.27:0/2876868602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36e80c8f30 0x7f36e815d2d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f36e80cbde0 tx=0x7f36d402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86e16f16d904f5e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.923+0000 7f5bef800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80abf10 0x7f5be81444d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.923+0000 7f5bef800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be80c1db0 0x7f5be8142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.923+0000 7f5bef800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80ac2f0 con 0x7f5be80abf10 2024-03-28T18:55:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.923+0000 7f5bef800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be8073620 con 0x7f5be806db40 2024-03-28T18:55:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.923+0000 7f5bef800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be806e4d0 con 0x7f5be80c1db0 2024-03-28T18:55:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.923+0000 7f5bece00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5be806db40 0x7f5be80ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.924+0000 7f5bece00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5be806db40 0x7f5be80ab230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43744/0 (socket says 172.21.15.27:43744) 2024-03-28T18:55:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.924+0000 7f5bece00640 1 -- 172.21.15.27:0/2772465765 learned_addr learned my addr 172.21.15.27:0/2772465765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.924+0000 7f5bed800640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be80c1db0 0x7f5be8142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.924+0000 7f36e6a00640 1 -- 172.21.15.27:0/2876868602 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d402e070 con 0x7f36e80c8f30 2024-03-28T18:55:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.924+0000 7f5be7e00640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80abf10 0x7f5be81444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.924+0000 7f36e6a00640 1 -- 172.21.15.27:0/2876868602 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d4037070 con 0x7f36e80c8f30 2024-03-28T18:55:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.924+0000 7f36e6a00640 1 -- 172.21.15.27:0/2876868602 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d4031a00 con 0x7f36e80c8f30 2024-03-28T18:55:55.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.925+0000 7ff174000640 1 -- 172.21.15.27:0/1046324287 shutdown_connections 2024-03-28T18:55:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.925+0000 7f36eee00640 1 -- 172.21.15.27:0/2876868602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36e80c8f30 msgr2=0x7f36e815d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.925+0000 7f36eee00640 1 --2- 172.21.15.27:0/2876868602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36e80c8f30 0x7f36e815d2d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f36e80cbde0 tx=0x7f36d402b880 comp rx=0 tx=0).stop 2024-03-28T18:55:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.926+0000 7f5bece00640 1 -- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be80c1db0 msgr2=0x7f5be8142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.926+0000 7f5bece00640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be80c1db0 0x7f5be8142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.926+0000 7f5bece00640 1 -- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80abf10 msgr2=0x7f5be81444d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.926+0000 7f5bece00640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80abf10 0x7f5be81444d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.926+0000 7f5bece00640 1 -- 172.21.15.27:0/2772465765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be8155b80 con 0x7f5be806db40 2024-03-28T18:55:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.926+0000 7f5bed800640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be80c1db0 0x7f5be8142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:55.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.927+0000 7f5be7e00640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80abf10 0x7f5be81444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:55.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.927+0000 7f36eee00640 1 -- 172.21.15.27:0/2876868602 shutdown_connections 2024-03-28T18:55:55.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.927+0000 7f36eee00640 1 --2- 172.21.15.27:0/2876868602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36e80c8f30 0x7f36e815d2d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.927+0000 7f36eee00640 1 --2- 172.21.15.27:0/2876868602 >> v2:172.21.15.27:3300/0 conn(0x7f36e8157470 0x7f36e80cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.927+0000 7f36eee00640 1 -- 172.21.15.27:0/2876868602 >> 172.21.15.27:0/2876868602 conn(0x7f36e8066fb0 msgr2=0x7f36e80673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:55.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.925+0000 7ff174000640 1 -- 172.21.15.27:0/1046324287 wait complete. 2024-03-28T18:55:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.927+0000 7ff174000640 1 Processor -- start 2024-03-28T18:55:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.928+0000 7ff174000640 1 -- start start 2024-03-28T18:55:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.928+0000 7f36eee00640 1 -- 172.21.15.27:0/2876868602 shutdown_connections 2024-03-28T18:55:55.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.928+0000 7f36eee00640 1 -- 172.21.15.27:0/2876868602 wait complete. 2024-03-28T18:55:55.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.929+0000 7f5bece00640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5be806db40 0x7f5be80ab230 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800c550 tx=0x7f5bd800ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.929+0000 7f5be4c00640 1 -- 172.21.15.27:0/2772465765 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8018070 con 0x7f5be806db40 2024-03-28T18:55:55.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be80ab770 con 0x7f5be806db40 2024-03-28T18:55:55.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff174000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff16c06db40 0x7ff16c1eff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff174000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff16c0abf10 0x7ff16c0aaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff174000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff16c0c1db0 0x7ff16c153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7f791ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 0x7f79180cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7f791ea00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f79180cb8c0 0x7f7918158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7f791ea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79181568e0 con 0x7f791814f920 2024-03-28T18:55:55.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7f791ea00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7918156450 con 0x7f79180cb8c0 2024-03-28T18:55:55.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7f791ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791806d960 con 0x7f79180c8f30 2024-03-28T18:55:55.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be81efad0 con 0x7f5be806db40 2024-03-28T18:55:55.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff174000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c0ac2f0 con 0x7ff16c06db40 2024-03-28T18:55:55.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff174000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c073620 con 0x7ff16c0abf10 2024-03-28T18:55:55.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff171600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff16c06db40 0x7ff16c1eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff174000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c071160 con 0x7ff16c0c1db0 2024-03-28T18:55:55.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff171600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff16c06db40 0x7ff16c1eff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59658/0 (socket says 172.21.15.27:59658) 2024-03-28T18:55:55.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7f7917e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f791814f920 0x7f79180c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.930+0000 7ff171600640 1 -- 172.21.15.27:0/3623803228 learned_addr learned my addr 172.21.15.27:0/3623803228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7f7917e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f791814f920 0x7f79180c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59672/0 (socket says 172.21.15.27:59672) 2024-03-28T18:55:55.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7f7917e00640 1 -- 172.21.15.27:0/2450944233 learned_addr learned my addr 172.21.15.27:0/2450944233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7f36eee00640 1 Processor -- start 2024-03-28T18:55:55.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7f36eee00640 1 -- start start 2024-03-28T18:55:55.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.932+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be804f470 con 0x7f5be806db40 2024-03-28T18:55:55.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.932+0000 7f5be4c00640 1 -- 172.21.15.27:0/2772465765 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd80074e0 con 0x7f5be806db40 2024-03-28T18:55:55.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.932+0000 7f5be4c00640 1 -- 172.21.15.27:0/2772465765 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8004a20 con 0x7f5be806db40 2024-03-28T18:55:55.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7f36eee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e80c8f30 0x7f36e8153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.932+0000 7f7917e00640 1 -- 172.21.15.27:0/2450944233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 msgr2=0x7f79180cb380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:55:55.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.932+0000 7f7917e00640 1 --2- 172.21.15.27:0/2450944233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 0x7f79180cb380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.932+0000 7f7917e00640 1 -- 172.21.15.27:0/2450944233 >> v2:172.21.15.155:3300/0 conn(0x7f79180cb8c0 msgr2=0x7f7918158f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:55:55.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.932+0000 7f7917e00640 1 --2- 172.21.15.27:0/2450944233 >> v2:172.21.15.155:3300/0 conn(0x7f79180cb8c0 0x7f7918158f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.932+0000 7f7917e00640 1 -- 172.21.15.27:0/2450944233 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918155b20 con 0x7f791814f920 2024-03-28T18:55:55.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7f36eee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36e80c9650 0x7f36e81f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7f36eee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36e8157470 0x7f36e80c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7f36eee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e806d880 con 0x7f36e80c8f30 2024-03-28T18:55:55.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7f36eee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e815df20 con 0x7f36e80c9650 2024-03-28T18:55:55.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7f36eee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e80cb870 con 0x7f36e8157470 2024-03-28T18:55:55.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7ff171600640 1 -- 172.21.15.27:0/3623803228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff16c0c1db0 msgr2=0x7ff16c153440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7ff171600640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff16c0c1db0 0x7ff16c153440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7ff171600640 1 -- 172.21.15.27:0/3623803228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff16c0abf10 msgr2=0x7ff16c0aaad0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7ff171600640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff16c0abf10 0x7ff16c0aaad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.931+0000 7ff171600640 1 -- 172.21.15.27:0/3623803228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16c154900 con 0x7ff16c06db40 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7ff171600640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff16c06db40 0x7ff16c1eff90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff15800c4a0 tx=0x7ff15800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7ff169600640 1 -- 172.21.15.27:0/3623803228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158017070 con 0x7ff16c06db40 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7ff169600640 1 -- 172.21.15.27:0/3623803228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff158005480 con 0x7ff16c06db40 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7ff169600640 1 -- 172.21.15.27:0/3623803228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158003830 con 0x7ff16c06db40 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16c154be0 con 0x7ff16c06db40 2024-03-28T18:55:55.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.934+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff16c152c40 con 0x7ff16c06db40 2024-03-28T18:55:55.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.935+0000 7f7917e00640 1 --2- 172.21.15.27:0/2450944233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 0x7f79180c9440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f79180c8ae0 tx=0x7f790802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6969bc5794273c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.935+0000 7f36ece00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36e8157470 0x7f36e80c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.935+0000 7f36ece00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36e8157470 0x7f36e80c2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45306/0 (socket says 172.21.15.27:45306) 2024-03-28T18:55:55.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.935+0000 7f36ece00640 1 -- 172.21.15.27:0/1417701809 learned_addr learned my addr 172.21.15.27:0/1417701809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.935+0000 7f36e7e00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e80c8f30 0x7f36e8153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.935+0000 7f7916a00640 1 -- 172.21.15.27:0/2450944233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908035020 con 0x7f791814f920 2024-03-28T18:55:55.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.936+0000 7f7916a00640 1 -- 172.21.15.27:0/2450944233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7908031830 con 0x7f791814f920 2024-03-28T18:55:55.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.936+0000 7f7916a00640 1 -- 172.21.15.27:0/2450944233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908031c00 con 0x7f791814f920 2024-03-28T18:55:55.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.936+0000 7f36ece00640 1 -- 172.21.15.27:0/1417701809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36e80c9650 msgr2=0x7f36e81f00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.936+0000 7f36ece00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36e80c9650 0x7f36e81f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.936+0000 7f36ece00640 1 -- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e80c8f30 msgr2=0x7f36e8153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.937+0000 7f36ece00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e80c8f30 0x7f36e8153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.937+0000 7f36ece00640 1 -- 172.21.15.27:0/1417701809 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80c19f0 con 0x7f36e8157470 2024-03-28T18:55:55.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.937+0000 7f791ea00640 1 -- 172.21.15.27:0/2450944233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 msgr2=0x7f79180c9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.937+0000 7f791ea00640 1 --2- 172.21.15.27:0/2450944233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 0x7f79180c9440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f79180c8ae0 tx=0x7f790802b8d0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.937+0000 7f36e7e00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e80c8f30 0x7f36e8153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.938+0000 7f36ece00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36e8157470 0x7f36e80c2100 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c7b0 tx=0x7f36e000cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.938+0000 7f36dfe00640 1 -- 172.21.15.27:0/1417701809 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0017070 con 0x7f36e8157470 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.938+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e80c1430 con 0x7f36e8157470 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.939+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e81efbc0 con 0x7f36e8157470 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.939+0000 7f36dfe00640 1 -- 172.21.15.27:0/1417701809 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e0003d80 con 0x7f36e8157470 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.939+0000 7f36dfe00640 1 -- 172.21.15.27:0/1417701809 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0017070 con 0x7f36e8157470 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.939+0000 7ff169600640 1 -- 172.21.15.27:0/3623803228 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff15801a070 con 0x7ff16c06db40 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.939+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff16c04f470 con 0x7ff16c06db40 2024-03-28T18:55:55.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.940+0000 7ff169600640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff14c06bbd0 0x7ff14c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.940+0000 7ff169600640 1 -- 172.21.15.27:0/3623803228 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7ff15808a080 con 0x7ff16c06db40 2024-03-28T18:55:55.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.941+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e804f470 con 0x7f36e8157470 2024-03-28T18:55:55.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.941+0000 7f791ea00640 1 -- 172.21.15.27:0/2450944233 shutdown_connections 2024-03-28T18:55:55.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.941+0000 7f791ea00640 1 --2- 172.21.15.27:0/2450944233 >> v2:172.21.15.155:3300/0 conn(0x7f79180cb8c0 0x7f7918158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.941+0000 7f791ea00640 1 --2- 172.21.15.27:0/2450944233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 0x7f79180cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.941+0000 7f791ea00640 1 --2- 172.21.15.27:0/2450944233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 0x7f79180c9440 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.941+0000 7f791ea00640 1 -- 172.21.15.27:0/2450944233 >> 172.21.15.27:0/2450944233 conn(0x7f7918066fb0 msgr2=0x7f79180673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:55.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.941+0000 7ff170c00640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff14c06bbd0 0x7ff14c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.943+0000 7ff170c00640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff14c06bbd0 0x7ff14c06e090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff164037000 tx=0x7ff164031000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.943+0000 7f791ea00640 1 -- 172.21.15.27:0/2450944233 shutdown_connections 2024-03-28T18:55:55.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.943+0000 7f36dfe00640 1 -- 172.21.15.27:0/1417701809 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f36e001a070 con 0x7f36e8157470 2024-03-28T18:55:55.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.945+0000 7f36dfe00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f36c806bdd0 0x7f36c806e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.945+0000 7f36dfe00640 1 -- 172.21.15.27:0/1417701809 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f36e008a4a0 con 0x7f36e8157470 2024-03-28T18:55:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.945+0000 7f5be4c00640 1 -- 172.21.15.27:0/2772465765 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5bd801b070 con 0x7f5be806db40 2024-03-28T18:55:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.943+0000 7f791ea00640 1 -- 172.21.15.27:0/2450944233 wait complete. 2024-03-28T18:55:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.946+0000 7f791ea00640 1 Processor -- start 2024-03-28T18:55:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.947+0000 7f5be4c00640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bc806bbd0 0x7f5bc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.947+0000 7f791ea00640 1 -- start start 2024-03-28T18:55:55.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.947+0000 7f5be4c00640 1 -- 172.21.15.27:0/2772465765 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f5bd808a600 con 0x7f5be806db40 2024-03-28T18:55:55.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.948+0000 7f5be7e00640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bc806bbd0 0x7f5bc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.948+0000 7f36e7e00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f36c806bdd0 0x7f36c806e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.949+0000 7ff169600640 1 -- 172.21.15.27:0/3623803228 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff1580588d0 con 0x7ff16c06db40 2024-03-28T18:55:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.949+0000 7f791ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 0x7f79181ebc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.949+0000 7f791ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79180cb8c0 0x7f79180abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.949+0000 7f791ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 0x7f79181fb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.949+0000 7f791ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791814fd00 con 0x7f791814f920 2024-03-28T18:55:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.949+0000 7f791ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79181560e0 con 0x7f79180cb8c0 2024-03-28T18:55:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.949+0000 7f791ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7918159440 con 0x7f79180c8f30 2024-03-28T18:55:55.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.950+0000 7f36e7e00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f36c806bdd0 0x7f36c806e290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009890 tx=0x7f36d8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.950+0000 7f7917400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79180cb8c0 0x7f79180abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.950+0000 7f7917400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79180cb8c0 0x7f79180abf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43772/0 (socket says 172.21.15.27:43772) 2024-03-28T18:55:55.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.950+0000 7f7917400640 1 -- 172.21.15.27:0/648476427 learned_addr learned my addr 172.21.15.27:0/648476427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.950+0000 7f7917e00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 0x7f79181ebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.950+0000 7f791ca00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 0x7f79181fb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.951+0000 7f791ca00640 1 -- 172.21.15.27:0/648476427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 msgr2=0x7f79181ebc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.951+0000 7f791ca00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 0x7f79181ebc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.951+0000 7f791ca00640 1 -- 172.21.15.27:0/648476427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79180cb8c0 msgr2=0x7f79180abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.951+0000 7f791ca00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79180cb8c0 0x7f79180abf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.951+0000 7f791ca00640 1 -- 172.21.15.27:0/648476427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79181f81e0 con 0x7f791814f920 2024-03-28T18:55:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.951+0000 7f7917400640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79180cb8c0 0x7f79180abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.952+0000 7f791ca00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 0x7f79181fb710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f791000be20 tx=0x7f791000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.952+0000 7f5be4c00640 1 -- 172.21.15.27:0/2772465765 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5bd8022020 con 0x7f5be806db40 2024-03-28T18:55:55.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.952+0000 7f790fe00640 1 -- 172.21.15.27:0/648476427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910019070 con 0x7f791814f920 2024-03-28T18:55:55.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.952+0000 7f790fe00640 1 -- 172.21.15.27:0/648476427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7910014440 con 0x7f791814f920 2024-03-28T18:55:55.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.953+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79181f84c0 con 0x7f791814f920 2024-03-28T18:55:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.956+0000 7f5be7e00640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bc806bbd0 0x7f5bc806e090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5be0035000 tx=0x7f5be0030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.953+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79181f8a80 con 0x7f791814f920 2024-03-28T18:55:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.954+0000 7f790fe00640 1 -- 172.21.15.27:0/648476427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910014830 con 0x7f791814f920 2024-03-28T18:55:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.955+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f0005290 con 0x7f791814f920 2024-03-28T18:55:55.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.961+0000 7f36dfe00640 1 -- 172.21.15.27:0/1417701809 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f36e0021020 con 0x7f36e8157470 2024-03-28T18:55:55.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.965+0000 7f790fe00640 1 -- 172.21.15.27:0/648476427 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f791001c070 con 0x7f791814f920 2024-03-28T18:55:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.967+0000 7f790fe00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78f806bbd0 0x7f78f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.967+0000 7f790fe00640 1 -- 172.21.15.27:0/648476427 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f791008b3e0 con 0x7f791814f920 2024-03-28T18:55:55.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.970+0000 7f790fe00640 1 -- 172.21.15.27:0/648476427 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7910023070 con 0x7f791814f920 2024-03-28T18:55:55.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.970+0000 7f7917e00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78f806bbd0 0x7f78f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.972+0000 7fdfa3800640 1 Processor -- start 2024-03-28T18:55:55.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.973+0000 7fdfa3800640 1 -- start start 2024-03-28T18:55:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.975+0000 7fdfa3800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdf940d7ae0 0x7fdf940071b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.975+0000 7fdfa3800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf9404e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.975+0000 7fdfa3800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9400f9d0 con 0x7fdf9404dae0 2024-03-28T18:55:55.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.975+0000 7fdfa3800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf940da4b0 con 0x7fdf940d7ae0 2024-03-28T18:55:55.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.975+0000 7fdfa3800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf940da790 con 0x7fdf9404dff0 2024-03-28T18:55:55.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.976+0000 7fdfa0e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdf940d7ae0 0x7fdf940071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.976+0000 7fdf9be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf9404e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.976+0000 7fdfa1800640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fdf9404dae0 0x7fdf9404ed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55684/0 (socket says 172.21.15.27:55684) 2024-03-28T18:55:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.976+0000 7fdfa1800640 1 -- 172.21.15.27:0/424861650 learned_addr learned my addr 172.21.15.27:0/424861650 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.977+0000 7fdf9be00640 1 -- 172.21.15.27:0/424861650 >> v2:172.21.15.155:3300/0 conn(0x7fdf940d7ae0 msgr2=0x7fdf940071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:55.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.977+0000 7fdf9be00640 1 --2- 172.21.15.27:0/424861650 >> v2:172.21.15.155:3300/0 conn(0x7fdf940d7ae0 0x7fdf940071b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.977+0000 7fdf9be00640 1 -- 172.21.15.27:0/424861650 >> v1:172.21.15.155:6789/0 conn(0x7fdf9404dae0 legacy=0x7fdf9404ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.977+0000 7fdf9be00640 1 -- 172.21.15.27:0/424861650 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf940daa70 con 0x7fdf9404dff0 2024-03-28T18:55:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.979+0000 7fdf9be00640 1 --2- 172.21.15.27:0/424861650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf9404e7d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf880097a0 tx=0x7fdf8802b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7557d3d85cccc6b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.980+0000 7f7917e00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78f806bbd0 0x7f78f806e090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7908036000 tx=0x7f790803c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.983+0000 7fdf9b400640 1 -- 172.21.15.27:0/424861650 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88037070 con 0x7fdf9404dff0 2024-03-28T18:55:55.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.983+0000 7fdf9b400640 1 -- 172.21.15.27:0/424861650 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88030710 con 0x7fdf9404dff0 2024-03-28T18:55:55.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.983+0000 7fdf9b400640 1 -- 172.21.15.27:0/424861650 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8802be60 con 0x7fdf9404dff0 2024-03-28T18:55:55.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.984+0000 7fdfa3800640 1 -- 172.21.15.27:0/424861650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 msgr2=0x7fdf9404e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.984+0000 7fdfa3800640 1 --2- 172.21.15.27:0/424861650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf9404e7d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf880097a0 tx=0x7fdf8802b660 comp rx=0 tx=0).stop 2024-03-28T18:55:55.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.984+0000 7fdfa3800640 1 -- 172.21.15.27:0/424861650 shutdown_connections 2024-03-28T18:55:55.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.984+0000 7fdfa3800640 1 --2- 172.21.15.27:0/424861650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf9404e7d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.984+0000 7fdfa3800640 1 --2- 172.21.15.27:0/424861650 >> v2:172.21.15.155:3300/0 conn(0x7fdf940d7ae0 0x7fdf940071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.984+0000 7fdfa3800640 1 -- 172.21.15.27:0/424861650 >> 172.21.15.27:0/424861650 conn(0x7fdf9404c660 msgr2=0x7fdf9404ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:55.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.986+0000 7fdfa3800640 1 -- 172.21.15.27:0/424861650 shutdown_connections 2024-03-28T18:55:55.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.986+0000 7fdfa3800640 1 -- 172.21.15.27:0/424861650 wait complete. 2024-03-28T18:55:55.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.988+0000 7fdfa3800640 1 Processor -- start 2024-03-28T18:55:55.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.989+0000 7fdfa3800640 1 -- start start 2024-03-28T18:55:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.990+0000 7fdfa3800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9404dae0 0x7fdf94046ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.990+0000 7fdfa3800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf940d82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.990+0000 7fdfa3800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf940d7ae0 0x7fdf9400eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.991+0000 7fdfa3800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9404e4b0 con 0x7fdf9404dae0 2024-03-28T18:55:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.991+0000 7fdfa3800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf940db270 con 0x7fdf940d7ae0 2024-03-28T18:55:55.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.991+0000 7fdfa3800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9400fdc0 con 0x7fdf9404dff0 2024-03-28T18:55:55.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.992+0000 7fdfa1800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf940d7ae0 0x7fdf9400eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.992+0000 7fdfa1800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf940d7ae0 0x7fdf9400eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43796/0 (socket says 172.21.15.27:43796) 2024-03-28T18:55:55.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.992+0000 7fdfa1800640 1 -- 172.21.15.27:0/3012772476 learned_addr learned my addr 172.21.15.27:0/3012772476 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:55.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.992+0000 7fdfa0e00640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9404dae0 0x7fdf94046ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.992+0000 7fdf9be00640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf940d82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:55.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.993+0000 7fdfa1800640 1 -- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 msgr2=0x7fdf940d82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.993+0000 7fdfa1800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf940d82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.993+0000 7fdfa1800640 1 -- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9404dae0 msgr2=0x7fdf94046ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:55.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.993+0000 7fdfa1800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9404dae0 0x7fdf94046ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:55.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.993+0000 7fdfa1800640 1 -- 172.21.15.27:0/3012772476 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf94047430 con 0x7fdf940d7ae0 2024-03-28T18:55:55.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.993+0000 7fdfa0e00640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9404dae0 0x7fdf94046ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:55.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.993+0000 7fdf9be00640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf940d82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:55.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.993+0000 7fdfa1800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf940d7ae0 0x7fdf9400eaf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c075910 tx=0x7fdf9c068c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:55.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.994+0000 7fdf98c00640 1 -- 172.21.15.27:0/3012772476 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0734c0 con 0x7fdf940d7ae0 2024-03-28T18:55:55.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.994+0000 7fdf98c00640 1 -- 172.21.15.27:0/3012772476 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c073680 con 0x7fdf940d7ae0 2024-03-28T18:55:55.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.994+0000 7fdf98c00640 1 -- 172.21.15.27:0/3012772476 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c071970 con 0x7fdf940d7ae0 2024-03-28T18:55:55.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.994+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf940d9f60 con 0x7fdf940d7ae0 2024-03-28T18:55:55.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:55.994+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf940d9870 con 0x7fdf940d7ae0 2024-03-28T18:55:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.001+0000 7fdf98c00640 1 -- 172.21.15.27:0/3012772476 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf9c071b10 con 0x7fdf940d7ae0 2024-03-28T18:55:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.001+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf940057e0 con 0x7fdf940d7ae0 2024-03-28T18:55:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.002+0000 7fdf98c00640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf7c06bca0 0x7fdf7c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.003+0000 7fdfa0e00640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf7c06bca0 0x7fdf7c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.008+0000 7fdf98c00640 1 -- 172.21.15.27:0/3012772476 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fdf9c106210 con 0x7fdf940d7ae0 2024-03-28T18:55:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.009+0000 7fdfa0e00640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf7c06bca0 0x7fdf7c06e160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009af0 tx=0x7fdf90008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.017+0000 7fdf98c00640 1 -- 172.21.15.27:0/3012772476 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf9c08f030 con 0x7fdf940d7ae0 2024-03-28T18:55:56.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.060+0000 7f5527e00640 1 Processor -- start 2024-03-28T18:55:56.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.060+0000 7f5527e00640 1 -- start start 2024-03-28T18:55:56.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.062+0000 7f5527e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5528156090 0x7f5528160f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.062+0000 7f5527e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55280735f0 con 0x7f55280c5aa0 2024-03-28T18:55:56.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.062+0000 7f5527e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55280c7d10 con 0x7f55280c8db0 2024-03-28T18:55:56.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.062+0000 7f5527e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55280c82d0 con 0x7f5528156090 2024-03-28T18:55:56.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.062+0000 7f5527400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5528156090 0x7f5528160f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.063+0000 7f5527400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5528156090 0x7f5528160f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45342/0 (socket says 172.21.15.27:45342) 2024-03-28T18:55:56.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.063+0000 7f5527400640 1 -- 172.21.15.27:0/34134433 learned_addr learned my addr 172.21.15.27:0/34134433 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:56.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.063+0000 7f5527400640 1 -- 172.21.15.27:0/34134433 >> v1:172.21.15.155:6789/0 conn(0x7f55280c8db0 legacy=0x7f5528152840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:55:56.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.063+0000 7f5527400640 1 -- 172.21.15.27:0/34134433 >> v1:172.21.15.27:6789/0 conn(0x7f55280c5aa0 legacy=0x7f5528072620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:55:56.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.064+0000 7f5527400640 1 -- 172.21.15.27:0/34134433 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55280c7ff0 con 0x7f5528156090 2024-03-28T18:55:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.072+0000 7f5527400640 1 --2- 172.21.15.27:0/34134433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5528156090 0x7f5528160f40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5520009e80 tx=0x7f552002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e855831eead4846a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.073+0000 7f5525600640 1 -- 172.21.15.27:0/34134433 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552002c9c0 con 0x7f5528156090 2024-03-28T18:55:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.073+0000 7f5525600640 1 -- 172.21.15.27:0/34134433 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552002cb60 con 0x7f5528156090 2024-03-28T18:55:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.073+0000 7f5525600640 1 -- 172.21.15.27:0/34134433 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55200343f0 con 0x7f5528156090 2024-03-28T18:55:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.074+0000 7f5527e00640 1 -- 172.21.15.27:0/34134433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5528156090 msgr2=0x7f5528160f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.074+0000 7f5527e00640 1 --2- 172.21.15.27:0/34134433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5528156090 0x7f5528160f40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5520009e80 tx=0x7f552002be20 comp rx=0 tx=0).stop 2024-03-28T18:55:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.075+0000 7f5527e00640 1 -- 172.21.15.27:0/34134433 shutdown_connections 2024-03-28T18:55:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.075+0000 7f5527e00640 1 --2- 172.21.15.27:0/34134433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5528156090 0x7f5528160f40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.075+0000 7f5527e00640 1 -- 172.21.15.27:0/34134433 >> 172.21.15.27:0/34134433 conn(0x7f552806b0c0 msgr2=0x7f552806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.079+0000 7f5527e00640 1 -- 172.21.15.27:0/34134433 shutdown_connections 2024-03-28T18:55:56.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.079+0000 7f5527e00640 1 -- 172.21.15.27:0/34134433 wait complete. 2024-03-28T18:55:56.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.082+0000 7f5527e00640 1 Processor -- start 2024-03-28T18:55:56.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.086+0000 7f5527e00640 1 -- start start 2024-03-28T18:55:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.088+0000 7f5527e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55280c5aa0 0x7f55280ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.088+0000 7f5527e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55280c8db0 0x7f55281466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.089+0000 7f5526a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55280c5aa0 0x7f55280ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.089+0000 7f5526a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55280c5aa0 0x7f55280ac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45354/0 (socket says 172.21.15.27:45354) 2024-03-28T18:55:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.089+0000 7f5526000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55280c8db0 0x7f55281466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.089+0000 7f5527e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5528156090 0x7f55281ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.090+0000 7f5527400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5528156090 0x7f55281ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.090+0000 7f5527400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5528156090 0x7f55281ee9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59704/0 (socket says 172.21.15.27:59704) 2024-03-28T18:55:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.091+0000 7f5527e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528156520 con 0x7f5528156090 2024-03-28T18:55:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.091+0000 7f5527e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5528072100 con 0x7f55280c8db0 2024-03-28T18:55:56.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.091+0000 7f5527e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5528072d30 con 0x7f55280c5aa0 2024-03-28T18:55:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.091+0000 7f5526a00640 1 -- 172.21.15.27:0/2949434272 learned_addr learned my addr 172.21.15.27:0/2949434272 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:56.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.092+0000 7f5527400640 1 -- 172.21.15.27:0/2949434272 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55280c5aa0 msgr2=0x7f55280ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.092+0000 7f5527400640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55280c5aa0 0x7f55280ac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.092+0000 7f5527400640 1 -- 172.21.15.27:0/2949434272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55280c8db0 msgr2=0x7f55281466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.092+0000 7f5526a00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55280c5aa0 0x7f55280ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-28T18:55:56.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.096+0000 7f5527400640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55280c8db0 0x7f55281466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.096+0000 7f5527400640 1 -- 172.21.15.27:0/2949434272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55280aa930 con 0x7f5528156090 2024-03-28T18:55:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.096+0000 7f5526000640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55280c8db0 0x7f55281466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.097+0000 7f5527400640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5528156090 0x7f55281ee9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f55280737c0 tx=0x7f5520004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.097+0000 7f551ea00640 1 -- 172.21.15.27:0/2949434272 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520043020 con 0x7f5528156090 2024-03-28T18:55:56.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.098+0000 7f551ea00640 1 -- 172.21.15.27:0/2949434272 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552003e450 con 0x7f5528156090 2024-03-28T18:55:56.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.101+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55280aabd0 con 0x7f5528156090 2024-03-28T18:55:56.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.101+0000 7f551ea00640 1 -- 172.21.15.27:0/2949434272 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552003e840 con 0x7f5528156090 2024-03-28T18:55:56.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.101+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55280ab190 con 0x7f5528156090 2024-03-28T18:55:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.105+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55280c9500 con 0x7f5528156090 2024-03-28T18:55:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.116+0000 7f551ea00640 1 -- 172.21.15.27:0/2949434272 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5520046020 con 0x7f5528156090 2024-03-28T18:55:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.117+0000 7f551ea00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f550806bbd0 0x7f550806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.117+0000 7f551ea00640 1 -- 172.21.15.27:0/2949434272 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f55200add80 con 0x7f5528156090 2024-03-28T18:55:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.120+0000 7f5526a00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f550806bbd0 0x7f550806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.123+0000 7f551ea00640 1 -- 172.21.15.27:0/2949434272 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f55280c9500 con 0x7f5528156090 2024-03-28T18:55:56.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.124+0000 7f5526a00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f550806bbd0 0x7f550806e090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5518003f40 tx=0x7f5518009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.175+0000 7fb088600640 1 Processor -- start 2024-03-28T18:55:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.176+0000 7fb088600640 1 -- start start 2024-03-28T18:55:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.178+0000 7fb088600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb080152d90 0x7fb0800cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.178+0000 7fb088600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb0800cb880 0x7fb0800c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.178+0000 7fb088600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0800c9840 con 0x7fb0800c8d70 2024-03-28T18:55:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.178+0000 7fb088600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb080155770 con 0x7fb080152d90 2024-03-28T18:55:56.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.178+0000 7fb088600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb080155a50 con 0x7fb0800cb880 2024-03-28T18:55:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.186+0000 7fb085c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb080152d90 0x7fb0800cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.186+0000 7fb085200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0800c8d70 0x7fb0800c7880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38118/0 (socket says 172.21.15.27:38118) 2024-03-28T18:55:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.186+0000 7fb085200640 1 -- 172.21.15.27:0/2562427894 learned_addr learned my addr 172.21.15.27:0/2562427894 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:56.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.186+0000 7fb086600640 1 --2- 172.21.15.27:0/2562427894 >> v2:172.21.15.155:3300/0 conn(0x7fb0800cb880 0x7fb0800c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.187+0000 7fb086600640 1 -- 172.21.15.27:0/2562427894 >> v2:172.21.15.27:3300/0 conn(0x7fb080152d90 msgr2=0x7fb0800cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.188+0000 7fb086600640 1 --2- 172.21.15.27:0/2562427894 >> v2:172.21.15.27:3300/0 conn(0x7fb080152d90 0x7fb0800cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.188+0000 7fb086600640 1 -- 172.21.15.27:0/2562427894 >> v1:172.21.15.27:6789/0 conn(0x7fb0800c8d70 legacy=0x7fb0800c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.188+0000 7fb086600640 1 -- 172.21.15.27:0/2562427894 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb080155d30 con 0x7fb0800cb880 2024-03-28T18:55:56.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.190+0000 7fb085c00640 1 --2- 172.21.15.27:0/2562427894 >> v2:172.21.15.27:3300/0 conn(0x7fb080152d90 0x7fb0800cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:55:56.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.191+0000 7fb086600640 1 --2- 172.21.15.27:0/2562427894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0800cb880 0x7fb0800c7f90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb080062b20 tx=0x7fb07402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ba7359e2ec609bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.192+0000 7fb07fe00640 1 -- 172.21.15.27:0/2562427894 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074035020 con 0x7fb0800cb880 2024-03-28T18:55:56.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.192+0000 7fb07fe00640 1 -- 172.21.15.27:0/2562427894 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb074031870 con 0x7fb0800cb880 2024-03-28T18:55:56.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.192+0000 7fb07fe00640 1 -- 172.21.15.27:0/2562427894 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074031c80 con 0x7fb0800cb880 2024-03-28T18:55:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.193+0000 7fb088600640 1 -- 172.21.15.27:0/2562427894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0800cb880 msgr2=0x7fb0800c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.193+0000 7fb088600640 1 --2- 172.21.15.27:0/2562427894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0800cb880 0x7fb0800c7f90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb080062b20 tx=0x7fb07402b910 comp rx=0 tx=0).stop 2024-03-28T18:55:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.194+0000 7fb088600640 1 -- 172.21.15.27:0/2562427894 shutdown_connections 2024-03-28T18:55:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.194+0000 7fb088600640 1 --2- 172.21.15.27:0/2562427894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0800cb880 0x7fb0800c7f90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.194+0000 7fb088600640 1 --2- 172.21.15.27:0/2562427894 >> v2:172.21.15.27:3300/0 conn(0x7fb080152d90 0x7fb0800cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.194+0000 7fb088600640 1 -- 172.21.15.27:0/2562427894 >> 172.21.15.27:0/2562427894 conn(0x7fb08006b4e0 msgr2=0x7fb08006b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.194+0000 7fb088600640 1 -- 172.21.15.27:0/2562427894 shutdown_connections 2024-03-28T18:55:56.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.195+0000 7fb088600640 1 -- 172.21.15.27:0/2562427894 wait complete. 2024-03-28T18:55:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.198+0000 7fb088600640 1 Processor -- start 2024-03-28T18:55:56.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.199+0000 7fb088600640 1 -- start start 2024-03-28T18:55:56.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.201+0000 7fb088600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0800c8d70 0x7fb0800c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.202+0000 7fb088600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0800cb880 0x7fb0800c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.202+0000 7fb085c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0800c8d70 0x7fb0800c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.202+0000 7fb085c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0800c8d70 0x7fb0800c49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46792/0 (socket says 172.21.15.27:46792) 2024-03-28T18:55:56.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.202+0000 7fb085c00640 1 -- 172.21.15.27:0/1493620973 learned_addr learned my addr 172.21.15.27:0/1493620973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:56.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.202+0000 7fb088600640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb080152d90 0x7fb0801ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.203+0000 7fb085200640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0800cb880 0x7fb0800c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.203+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb080072b00 con 0x7fb0800c8d70 2024-03-28T18:55:56.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.203+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb080156530 con 0x7fb080152d90 2024-03-28T18:55:56.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.203+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0800c9280 con 0x7fb0800cb880 2024-03-28T18:55:56.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.204+0000 7fb085200640 1 -- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb080152d90 msgr2=0x7fb0801ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.204+0000 7fb085200640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb080152d90 0x7fb0801ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.204+0000 7fb085200640 1 -- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0800c8d70 msgr2=0x7fb0800c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.204+0000 7fb085200640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0800c8d70 0x7fb0800c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.204+0000 7fb085200640 1 -- 172.21.15.27:0/1493620973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0801596b0 con 0x7fb0800cb880 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.206+0000 7fb085200640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0800cb880 0x7fb0800c4f10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb0800cacf0 tx=0x7fb07800cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.206+0000 7fb07d600640 1 -- 172.21.15.27:0/1493620973 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07800a6d0 con 0x7fb0800cb880 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.206+0000 7fb07d600640 1 -- 172.21.15.27:0/1493620973 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb078019070 con 0x7fb0800cb880 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.207+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0801ec550 con 0x7fb0800cb880 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.207+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb080158880 con 0x7fb0800cb880 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.208+0000 7fb07d600640 1 -- 172.21.15.27:0/1493620973 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07800aab0 con 0x7fb0800cb880 2024-03-28T18:55:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.210+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb080158410 con 0x7fb0800cb880 2024-03-28T18:55:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.239+0000 7fb07d600640 1 -- 172.21.15.27:0/1493620973 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb07800ac50 con 0x7fb0800cb880 2024-03-28T18:55:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.240+0000 7fb07d600640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb06006bc20 0x7fb06006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.240+0000 7fb07d600640 1 -- 172.21.15.27:0/1493620973 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fb07808b790 con 0x7fb0800cb880 2024-03-28T18:55:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.244+0000 7fb085c00640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb06006bc20 0x7fb06006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.245+0000 7fb085c00640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb06006bc20 0x7fb06006e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0095e0 tx=0x7fb06c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.256+0000 7fb07d600640 1 -- 172.21.15.27:0/1493620973 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb078059f30 con 0x7fb0800cb880 2024-03-28T18:55:56.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.329+0000 7f3bea000640 1 -- 172.21.15.27:0/2446250014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3bf000fba0 con 0x7f3bf004dae0 2024-03-28T18:55:56.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.330+0000 7f3bf4c00640 1 -- 172.21.15.27:0/2446250014 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3bf806f030 con 0x7f3bf004dae0 2024-03-28T18:55:56.332 INFO:teuthology.orchestra.run.smithi027.stdout:55834574859 2024-03-28T18:55:56.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 -- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bd806bdd0 msgr2=0x7f3bd806e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bd806bdd0 0x7f3bd806e290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3be403f000 tx=0x7f3be403a000 comp rx=0 tx=0).stop 2024-03-28T18:55:56.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 -- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dae0 msgr2=0x7f3bf00d8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dae0 0x7f3bf00d8f90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3bf804ff90 tx=0x7f3bf8068c60 comp rx=0 tx=0).stop 2024-03-28T18:55:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 -- 172.21.15.27:0/2446250014 shutdown_connections 2024-03-28T18:55:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bd806bdd0 0x7f3bd806e290 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3be403f000 tx=0x7f3be403a000 comp rx=0 tx=0).stop 2024-03-28T18:55:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bf00d7ae0 0x7f3bf0030730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bf004dff0 0x7f3bf00301f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 --2- 172.21.15.27:0/2446250014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bf004dae0 0x7f3bf00d8f90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.333+0000 7f3bea000640 1 -- 172.21.15.27:0/2446250014 >> 172.21.15.27:0/2446250014 conn(0x7f3bf004c660 msgr2=0x7f3bf00d5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.334+0000 7f3bea000640 1 -- 172.21.15.27:0/2446250014 shutdown_connections 2024-03-28T18:55:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.335+0000 7f3bea000640 1 -- 172.21.15.27:0/2446250014 wait complete. 2024-03-28T18:55:56.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.0 2024-03-28T18:55:56.363 DEBUG:teuthology.parallel:result is None 2024-03-28T18:55:56.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.386+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff16c072af0 con 0x7ff16c06db40 2024-03-28T18:55:56.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.387+0000 7ff169600640 1 -- 172.21.15.27:0/3623803228 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff1580581d0 con 0x7ff16c06db40 2024-03-28T18:55:56.389 INFO:teuthology.orchestra.run.smithi027.stdout:55834574859 2024-03-28T18:55:56.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.391+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff14c06bbd0 msgr2=0x7ff14c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.391+0000 7ff174000640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff14c06bbd0 0x7ff14c06e090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff164037000 tx=0x7ff164031000 comp rx=0 tx=0).stop 2024-03-28T18:55:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.391+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff16c06db40 msgr2=0x7ff16c1eff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.391+0000 7ff174000640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff16c06db40 0x7ff16c1eff90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff15800c4a0 tx=0x7ff15800c960 comp rx=0 tx=0).stop 2024-03-28T18:55:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.392+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 shutdown_connections 2024-03-28T18:55:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.392+0000 7ff174000640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff14c06bbd0 0x7ff14c06e090 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.392+0000 7ff174000640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff16c0c1db0 0x7ff16c153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.392+0000 7ff174000640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff16c0abf10 0x7ff16c0aaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.392+0000 7ff174000640 1 --2- 172.21.15.27:0/3623803228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff16c06db40 0x7ff16c1eff90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.392+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 >> 172.21.15.27:0/3623803228 conn(0x7ff16c0c6da0 msgr2=0x7ff16c14e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.394+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 shutdown_connections 2024-03-28T18:55:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.394+0000 7ff174000640 1 -- 172.21.15.27:0/3623803228 wait complete. 2024-03-28T18:55:56.425 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-03-28T18:55:56.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.425+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f36e806d660 con 0x7f36e8157470 2024-03-28T18:55:56.425 DEBUG:teuthology.parallel:result is None 2024-03-28T18:55:56.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.426+0000 7f36dfe00640 1 -- 172.21.15.27:0/1417701809 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f36e0058cd0 con 0x7f36e8157470 2024-03-28T18:55:56.426 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f36c806bdd0 msgr2=0x7f36c806e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f36c806bdd0 0x7f36c806e290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009890 tx=0x7f36d8008040 comp rx=0 tx=0).stop 2024-03-28T18:55:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36e8157470 msgr2=0x7f36e80c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36e8157470 0x7f36e80c2100 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c7b0 tx=0x7f36e000cc70 comp rx=0 tx=0).stop 2024-03-28T18:55:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 shutdown_connections 2024-03-28T18:55:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f36c806bdd0 0x7f36c806e290 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36e8157470 0x7f36e80c2100 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36e80c9650 0x7f36e81f00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 --2- 172.21.15.27:0/1417701809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e80c8f30 0x7f36e8153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.430+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 >> 172.21.15.27:0/1417701809 conn(0x7f36e8066fb0 msgr2=0x7f36e80c58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.431+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 shutdown_connections 2024-03-28T18:55:56.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.431+0000 7f36eee00640 1 -- 172.21.15.27:0/1417701809 wait complete. 2024-03-28T18:55:56.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.432+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5be8072af0 con 0x7f5be806db40 2024-03-28T18:55:56.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.433+0000 7f5be4c00640 1 -- 172.21.15.27:0/2772465765 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bd80587e0 con 0x7f5be806db40 2024-03-28T18:55:56.433 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:56.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bc806bbd0 msgr2=0x7f5bc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bc806bbd0 0x7f5bc806e090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5be0035000 tx=0x7f5be0030000 comp rx=0 tx=0).stop 2024-03-28T18:55:56.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5be806db40 msgr2=0x7f5be80ab230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5be806db40 0x7f5be80ab230 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800c550 tx=0x7f5bd800ca10 comp rx=0 tx=0).stop 2024-03-28T18:55:56.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 shutdown_connections 2024-03-28T18:55:56.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bc806bbd0 0x7f5bc806e090 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5be80c1db0 0x7f5be8142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80abf10 0x7f5be81444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 --2- 172.21.15.27:0/2772465765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5be806db40 0x7f5be80ab230 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 >> 172.21.15.27:0/2772465765 conn(0x7f5be80c6da0 msgr2=0x7f5be80c21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.438+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 shutdown_connections 2024-03-28T18:55:56.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.438+0000 7f5bef800640 1 -- 172.21.15.27:0/2772465765 wait complete. 2024-03-28T18:55:56.439 INFO:teuthology.orchestra.run.smithi027.stdout:55834574858 2024-03-28T18:55:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.437+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f78f0005850 con 0x7f791814f920 2024-03-28T18:55:56.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.438+0000 7f790fe00640 1 -- 172.21.15.27:0/648476427 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f79100595c0 con 0x7f791814f920 2024-03-28T18:55:56.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.441+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78f806bbd0 msgr2=0x7f78f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.441+0000 7f791ea00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78f806bbd0 0x7f78f806e090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7908036000 tx=0x7f790803c040 comp rx=0 tx=0).stop 2024-03-28T18:55:56.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.441+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 msgr2=0x7f79181fb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.441+0000 7f791ea00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 0x7f79181fb710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f791000be20 tx=0x7f791000bf10 comp rx=0 tx=0).stop 2024-03-28T18:55:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.442+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 shutdown_connections 2024-03-28T18:55:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.442+0000 7f791ea00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78f806bbd0 0x7f78f806e090 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.442+0000 7f791ea00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f791814f920 0x7f79181fb710 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.442+0000 7f791ea00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79180cb8c0 0x7f79180abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.442+0000 7f791ea00640 1 --2- 172.21.15.27:0/648476427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79180c8f30 0x7f79181ebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.442+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 >> 172.21.15.27:0/648476427 conn(0x7f7918066fb0 msgr2=0x7f79180c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.442+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 shutdown_connections 2024-03-28T18:55:56.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.443+0000 7f791ea00640 1 -- 172.21.15.27:0/648476427 wait complete. 2024-03-28T18:55:56.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-03-28T18:55:56.458 DEBUG:teuthology.parallel:result is None 2024-03-28T18:55:56.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2024-03-28T18:55:56.459 DEBUG:teuthology.parallel:result is None 2024-03-28T18:55:56.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.460+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdf9400fba0 con 0x7fdf940d7ae0 2024-03-28T18:55:56.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.461+0000 7fdf98c00640 1 -- 172.21.15.27:0/3012772476 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdf9c0d49b0 con 0x7fdf940d7ae0 2024-03-28T18:55:56.460 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf7c06bca0 msgr2=0x7fdf7c06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf7c06bca0 0x7fdf7c06e160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009af0 tx=0x7fdf90008040 comp rx=0 tx=0).stop 2024-03-28T18:55:56.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf940d7ae0 msgr2=0x7fdf9400eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf940d7ae0 0x7fdf9400eaf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c075910 tx=0x7fdf9c068c60 comp rx=0 tx=0).stop 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 shutdown_connections 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf7c06bca0 0x7fdf7c06e160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf940d7ae0 0x7fdf9400eaf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf9404dff0 0x7fdf940d82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 --2- 172.21.15.27:0/3012772476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf9404dae0 0x7fdf94046ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.463+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 >> 172.21.15.27:0/3012772476 conn(0x7fdf9404c660 msgr2=0x7fdf940d59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.464+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 shutdown_connections 2024-03-28T18:55:56.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.464+0000 7fdfa3800640 1 -- 172.21.15.27:0/3012772476 wait complete. 2024-03-28T18:55:56.467 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.5 2024-03-28T18:55:56.467 DEBUG:teuthology.parallel:result is None 2024-03-28T18:55:56.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-03-28T18:55:56.482 DEBUG:teuthology.parallel:result is None 2024-03-28T18:55:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.578+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f55280ab360 con 0x7f5528156090 2024-03-28T18:55:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.579+0000 7f551ea00640 1 -- 172.21.15.27:0/2949434272 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f552004d030 con 0x7f5528156090 2024-03-28T18:55:56.579 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2024-03-28T18:55:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f550806bbd0 msgr2=0x7f550806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f550806bbd0 0x7f550806e090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5518003f40 tx=0x7f5518009040 comp rx=0 tx=0).stop 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5528156090 msgr2=0x7f55281ee9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5528156090 0x7f55281ee9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f55280737c0 tx=0x7f5520004a40 comp rx=0 tx=0).stop 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 shutdown_connections 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f550806bbd0 0x7f550806e090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5528156090 0x7f55281ee9a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55280c8db0 0x7f55281466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 --2- 172.21.15.27:0/2949434272 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55280c5aa0 0x7f55280ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.581+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 >> 172.21.15.27:0/2949434272 conn(0x7f552806b0c0 msgr2=0x7f5528154aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.582+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 shutdown_connections 2024-03-28T18:55:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.582+0000 7f5527e00640 1 -- 172.21.15.27:0/2949434272 wait complete. 2024-03-28T18:55:56.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-03-28T18:55:56.597 DEBUG:teuthology.parallel:result is None 2024-03-28T18:55:56.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.609+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb08004f400 con 0x7fb0800cb880 2024-03-28T18:55:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.610+0000 7fb07d600640 1 -- 172.21.15.27:0/1493620973 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb078059830 con 0x7fb0800cb880 2024-03-28T18:55:56.610 INFO:teuthology.orchestra.run.smithi027.stdout:55834574858 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.612+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb06006bc20 msgr2=0x7fb06006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.612+0000 7fb088600640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb06006bc20 0x7fb06006e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0095e0 tx=0x7fb06c009190 comp rx=0 tx=0).stop 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0800cb880 msgr2=0x7fb0800c4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0800cb880 0x7fb0800c4f10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb0800cacf0 tx=0x7fb07800cf60 comp rx=0 tx=0).stop 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 shutdown_connections 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb06006bc20 0x7fb06006e0e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb080152d90 0x7fb0801ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0800cb880 0x7fb0800c4f10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 --2- 172.21.15.27:0/1493620973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0800c8d70 0x7fb0800c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 >> 172.21.15.27:0/1493620973 conn(0x7fb08006b4e0 msgr2=0x7fb08014e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 shutdown_connections 2024-03-28T18:55:56.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.613+0000 7fb088600640 1 -- 172.21.15.27:0/1493620973 wait complete. 2024-03-28T18:55:56.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-03-28T18:55:56.628 DEBUG:teuthology.parallel:result is None 2024-03-28T18:55:56.629 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T18:55:56.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:55:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.884+0000 7f647be00640 1 Processor -- start 2024-03-28T18:55:56.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.885+0000 7f647be00640 1 -- start start 2024-03-28T18:55:56.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.886+0000 7f647be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647c0c58c0 0x7f647c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.886+0000 7f647be00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f647c0c8ed0 0x7f647c1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.886+0000 7f647be00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c156230 con 0x7f647c0be5d0 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.886+0000 7f647be00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c158bd0 con 0x7f647c0c8ed0 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.886+0000 7f647be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c158eb0 con 0x7f647c0c58c0 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.886+0000 7f647b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f647c0be5d0 0x7f647c155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38122/0 (socket says 172.21.15.27:38122) 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647b400640 1 -- 172.21.15.27:0/3415989406 learned_addr learned my addr 172.21.15.27:0/3415989406 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f6472000640 1 --2- 172.21.15.27:0/3415989406 >> v2:172.21.15.155:3300/0 conn(0x7f647c0c8ed0 0x7f647c1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647aa00640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647c0c58c0 0x7f647c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115682793 0 0) 0x7f647c156230 con 0x7f647c0be5d0 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6460003520 con 0x7f647c0be5d0 2024-03-28T18:55:56.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1638482476 0 0) 0x7f6460003520 con 0x7f647c0be5d0 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647c0c58c0 msgr2=0x7f647c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647c0c58c0 0x7f647c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 >> v2:172.21.15.155:3300/0 conn(0x7f647c0c8ed0 msgr2=0x7f647c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 --2- 172.21.15.27:0/3415989406 >> v2:172.21.15.155:3300/0 conn(0x7f647c0c8ed0 0x7f647c1555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c159190 con 0x7f647c0be5d0 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f6474001670 con 0x7f647c0be5d0 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.887+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 >> v1:172.21.15.27:6789/0 conn(0x7f647c0be5d0 legacy=0x7f647c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f647a000640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647c0c58c0 0x7f64600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f647a000640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6460003f80 0x7f6460004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f647a000640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6460004d40 0x7f64600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c158bd0 con 0x7f6460003f80 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c158eb0 con 0x7f6460004d40 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646000ba70 con 0x7f647c0c58c0 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f6472000640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6460003f80 0x7f6460004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f647b400640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6460004d40 0x7f64600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f647aa00640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647c0c58c0 0x7f64600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f6472000640 1 -- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647c0c58c0 msgr2=0x7f64600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f6472000640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647c0c58c0 0x7f64600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f6472000640 1 -- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6460004d40 msgr2=0x7f64600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.888+0000 7f6472000640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6460004d40 0x7f64600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.889+0000 7f6472000640 1 -- 172.21.15.27:0/3415989406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6460003520 con 0x7f6460003f80 2024-03-28T18:55:56.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.890+0000 7f6472000640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6460003f80 0x7f6460004800 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6464009ac0 tx=0x7f6460005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9a279b72d765c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.890+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464039070 con 0x7f6460003f80 2024-03-28T18:55:56.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646402c710 con 0x7f6460003f80 2024-03-28T18:55:56.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647a000640 1 -- 172.21.15.27:0/3415989406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464009690 con 0x7f6460003f80 2024-03-28T18:55:56.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647be00640 1 -- 172.21.15.27:0/3415989406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6460003f80 msgr2=0x7f6460004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647be00640 1 --2- 172.21.15.27:0/3415989406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6460003f80 0x7f6460004800 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6464009ac0 tx=0x7f6460005830 comp rx=0 tx=0).stop 2024-03-28T18:55:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647b400640 1 -- 172.21.15.27:0/3415989406 reap_dead start 2024-03-28T18:55:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647be00640 1 -- 172.21.15.27:0/3415989406 shutdown_connections 2024-03-28T18:55:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647be00640 1 -- 172.21.15.27:0/3415989406 >> 172.21.15.27:0/3415989406 conn(0x7f647c06b6f0 msgr2=0x7f647c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647be00640 1 -- 172.21.15.27:0/3415989406 shutdown_connections 2024-03-28T18:55:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.891+0000 7f647be00640 1 -- 172.21.15.27:0/3415989406 wait complete. 2024-03-28T18:55:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.893+0000 7f647be00640 1 Processor -- start 2024-03-28T18:55:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.893+0000 7f647be00640 1 -- start start 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f647be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6460003f80 0x7f647c1ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f647be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f647c1eef20 0x7f647c154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f647be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f647c155310 0x7f647c0c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f647be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b0820 con 0x7f647c1eef20 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f647be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c159750 con 0x7f647c155310 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f647be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b09a0 con 0x7f6460003f80 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f6472000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f647c1eef20 0x7f647c154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f6472000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f647c1eef20 0x7f647c154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46810/0 (socket says 172.21.15.27:46810) 2024-03-28T18:55:56.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.894+0000 7f6472000640 1 -- 172.21.15.27:0/1006384298 learned_addr learned my addr 172.21.15.27:0/1006384298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6472000640 1 -- 172.21.15.27:0/1006384298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6460003f80 msgr2=0x7f647c1ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6472000640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6460003f80 0x7f647c1ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6472000640 1 -- 172.21.15.27:0/1006384298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f647c155310 msgr2=0x7f647c0c3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f647b400640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f647c155310 0x7f647c0c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6472000640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f647c155310 0x7f647c0c3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6472000640 1 -- 172.21.15.27:0/1006384298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c0accf0 con 0x7f647c1eef20 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6472000640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f647c1eef20 0x7f647c154dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6464009be0 tx=0x7f6464004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6473400640 1 -- 172.21.15.27:0/1006384298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464042020 con 0x7f647c1eef20 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6473400640 1 -- 172.21.15.27:0/1006384298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646403a040 con 0x7f647c1eef20 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f6473400640 1 -- 172.21.15.27:0/1006384298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464039070 con 0x7f647c1eef20 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c0c2010 con 0x7f647c1eef20 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.895+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c153700 con 0x7f647c1eef20 2024-03-28T18:55:56.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.897+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c069640 con 0x7f647c1eef20 2024-03-28T18:55:56.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.897+0000 7f6473400640 1 -- 172.21.15.27:0/1006384298 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6464045020 con 0x7f647c1eef20 2024-03-28T18:55:56.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.899+0000 7f6473400640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f645806bbd0 0x7f645806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:56.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.899+0000 7f6473400640 1 -- 172.21.15.27:0/1006384298 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f64640abf10 con 0x7f647c1eef20 2024-03-28T18:55:56.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.899+0000 7f647aa00640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f645806bbd0 0x7f645806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:56.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.900+0000 7f647aa00640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f645806bbd0 0x7f645806e090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f646c009f80 tx=0x7f646c009750 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:56.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:56.903+0000 7f6473400640 1 -- 172.21.15.27:0/1006384298 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f646404c020 con 0x7f647c1eef20 2024-03-28T18:55:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.098+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f647c156580 con 0x7f645806bbd0 2024-03-28T18:55:57.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.113+0000 7f6473400640 1 -- 172.21.15.27:0/1006384298 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490306 (secure 0 0 0) 0x7f647c156580 con 0x7f645806bbd0 2024-03-28T18:55:57.120 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:55:57.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f645806bbd0 msgr2=0x7f645806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f645806bbd0 0x7f645806e090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f646c009f80 tx=0x7f646c009750 comp rx=0 tx=0).stop 2024-03-28T18:55:57.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f647c1eef20 msgr2=0x7f647c154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f647c1eef20 0x7f647c154dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6464009be0 tx=0x7f6464004730 comp rx=0 tx=0).stop 2024-03-28T18:55:57.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 shutdown_connections 2024-03-28T18:55:57.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f645806bbd0 0x7f645806e090 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f647c155310 0x7f647c0c3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f647c1eef20 0x7f647c154dd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 --2- 172.21.15.27:0/1006384298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6460003f80 0x7f647c1ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 >> 172.21.15.27:0/1006384298 conn(0x7f647c06b6f0 msgr2=0x7f647c0ca780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:57.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 shutdown_connections 2024-03-28T18:55:57.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.119+0000 7f647be00640 1 -- 172.21.15.27:0/1006384298 wait complete. 2024-03-28T18:55:57.137 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-28T18:55:56.899138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219448,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755272,"statfs":{"total":773094113280,"available":772869398528,"internally_reserved":0,"allocated":3350528,"data_stored":1984464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.300430"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603978+0000","last_change":"2024-03-28T18:55:45.603782+0000","last_active":"2024-03-28T18:55:46.603978+0000","last_peered":"2024-03-28T18:55:46.603978+0000","last_clean":"2024-03-28T18:55:46.603978+0000","last_became_active":"2024-03-28T18:55:45.603420+0000","last_became_peered":"2024-03-28T18:55:45.603420+0000","last_unstale":"2024-03-28T18:55:46.603978+0000","last_undegraded":"2024-03-28T18:55:46.603978+0000","last_fullsized":"2024-03-28T18:55:46.603978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609332+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:55:46.609332+0000","last_peered":"2024-03-28T18:55:46.609332+0000","last_clean":"2024-03-28T18:55:46.609332+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:55:46.609332+0000","last_undegraded":"2024-03-28T18:55:46.609332+0000","last_fullsized":"2024-03-28T18:55:46.609332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828788+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:55:46.828788+0000","last_peered":"2024-03-28T18:55:46.828788+0000","last_clean":"2024-03-28T18:55:46.828788+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:55:46.828788+0000","last_undegraded":"2024-03-28T18:55:46.828788+0000","last_fullsized":"2024-03-28T18:55:46.828788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825968+0000","last_change":"2024-03-28T18:55:44.602536+0000","last_active":"2024-03-28T18:55:46.825968+0000","last_peered":"2024-03-28T18:55:46.825968+0000","last_clean":"2024-03-28T18:55:46.825968+0000","last_became_active":"2024-03-28T18:55:44.602256+0000","last_became_peered":"2024-03-28T18:55:44.602256+0000","last_unstale":"2024-03-28T18:55:46.825968+0000","last_undegraded":"2024-03-28T18:55:46.825968+0000","last_fullsized":"2024-03-28T18:55:46.825968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612952+0000","last_change":"2024-03-28T18:55:45.625338+0000","last_active":"2024-03-28T18:55:46.612952+0000","last_peered":"2024-03-28T18:55:46.612952+0000","last_clean":"2024-03-28T18:55:46.612952+0000","last_became_active":"2024-03-28T18:55:45.625067+0000","last_became_peered":"2024-03-28T18:55:45.625067+0000","last_unstale":"2024-03-28T18:55:46.612952+0000","last_undegraded":"2024-03-28T18:55:46.612952+0000","last_fullsized":"2024-03-28T18:55:46.612952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829113+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:55:46.829113+0000","last_peered":"2024-03-28T18:55:46.829113+0000","last_clean":"2024-03-28T18:55:46.829113+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:55:46.829113+0000","last_undegraded":"2024-03-28T18:55:46.829113+0000","last_fullsized":"2024-03-28T18:55:46.829113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608925+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:55:46.608925+0000","last_peered":"2024-03-28T18:55:46.608925+0000","last_clean":"2024-03-28T18:55:46.608925+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:55:46.608925+0000","last_undegraded":"2024-03-28T18:55:46.608925+0000","last_fullsized":"2024-03-28T18:55:46.608925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604171+0000","last_change":"2024-03-28T18:55:44.598423+0000","last_active":"2024-03-28T18:55:46.604171+0000","last_peered":"2024-03-28T18:55:46.604171+0000","last_clean":"2024-03-28T18:55:46.604171+0000","last_became_active":"2024-03-28T18:55:44.598092+0000","last_became_peered":"2024-03-28T18:55:44.598092+0000","last_unstale":"2024-03-28T18:55:46.604171+0000","last_undegraded":"2024-03-28T18:55:46.604171+0000","last_fullsized":"2024-03-28T18:55:46.604171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.646619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163641+0000","last_change":"2024-03-28T18:55:45.608201+0000","last_active":"2024-03-28T18:55:47.163641+0000","last_peered":"2024-03-28T18:55:47.163641+0000","last_clean":"2024-03-28T18:55:47.163641+0000","last_became_active":"2024-03-28T18:55:45.607912+0000","last_became_peered":"2024-03-28T18:55:45.607912+0000","last_unstale":"2024-03-28T18:55:47.163641+0000","last_undegraded":"2024-03-28T18:55:47.163641+0000","last_fullsized":"2024-03-28T18:55:47.163641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164845+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:55:47.164845+0000","last_peered":"2024-03-28T18:55:47.164845+0000","last_clean":"2024-03-28T18:55:47.164845+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:55:47.164845+0000","last_undegraded":"2024-03-28T18:55:47.164845+0000","last_fullsized":"2024-03-28T18:55:47.164845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117090+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:55:47.117090+0000","last_peered":"2024-03-28T18:55:47.117090+0000","last_clean":"2024-03-28T18:55:47.117090+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:55:47.117090+0000","last_undegraded":"2024-03-28T18:55:47.117090+0000","last_fullsized":"2024-03-28T18:55:47.117090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604780+0000","last_change":"2024-03-28T18:55:44.604673+0000","last_active":"2024-03-28T18:55:46.604780+0000","last_peered":"2024-03-28T18:55:46.604780+0000","last_clean":"2024-03-28T18:55:46.604780+0000","last_became_active":"2024-03-28T18:55:44.603926+0000","last_became_peered":"2024-03-28T18:55:44.603926+0000","last_unstale":"2024-03-28T18:55:46.604780+0000","last_undegraded":"2024-03-28T18:55:46.604780+0000","last_fullsized":"2024-03-28T18:55:46.604780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116402+0000","last_change":"2024-03-28T18:55:45.610033+0000","last_active":"2024-03-28T18:55:47.116402+0000","last_peered":"2024-03-28T18:55:47.116402+0000","last_clean":"2024-03-28T18:55:47.116402+0000","last_became_active":"2024-03-28T18:55:45.609706+0000","last_became_peered":"2024-03-28T18:55:45.609706+0000","last_unstale":"2024-03-28T18:55:47.116402+0000","last_undegraded":"2024-03-28T18:55:47.116402+0000","last_fullsized":"2024-03-28T18:55:47.116402+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164190+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:55:47.164190+0000","last_peered":"2024-03-28T18:55:47.164190+0000","last_clean":"2024-03-28T18:55:47.164190+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:55:47.164190+0000","last_undegraded":"2024-03-28T18:55:47.164190+0000","last_fullsized":"2024-03-28T18:55:47.164190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117556+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:55:47.117556+0000","last_peered":"2024-03-28T18:55:47.117556+0000","last_clean":"2024-03-28T18:55:47.117556+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:55:47.117556+0000","last_undegraded":"2024-03-28T18:55:47.117556+0000","last_fullsized":"2024-03-28T18:55:47.117556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604505+0000","last_change":"2024-03-28T18:55:44.606715+0000","last_active":"2024-03-28T18:55:46.604505+0000","last_peered":"2024-03-28T18:55:46.604505+0000","last_clean":"2024-03-28T18:55:46.604505+0000","last_became_active":"2024-03-28T18:55:44.606343+0000","last_became_peered":"2024-03-28T18:55:44.606343+0000","last_unstale":"2024-03-28T18:55:46.604505+0000","last_undegraded":"2024-03-28T18:55:46.604505+0000","last_fullsized":"2024-03-28T18:55:46.604505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:51.471450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613822+0000","last_change":"2024-03-28T18:55:45.623483+0000","last_active":"2024-03-28T18:55:46.613822+0000","last_peered":"2024-03-28T18:55:46.613822+0000","last_clean":"2024-03-28T18:55:46.613822+0000","last_became_active":"2024-03-28T18:55:45.622894+0000","last_became_peered":"2024-03-28T18:55:45.622894+0000","last_unstale":"2024-03-28T18:55:46.613822+0000","last_undegraded":"2024-03-28T18:55:46.613822+0000","last_fullsized":"2024-03-28T18:55:46.613822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603964+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:55:46.603964+0000","last_peered":"2024-03-28T18:55:46.603964+0000","last_clean":"2024-03-28T18:55:46.603964+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:55:46.603964+0000","last_undegraded":"2024-03-28T18:55:46.603964+0000","last_fullsized":"2024-03-28T18:55:46.603964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164866+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:55:47.164866+0000","last_peered":"2024-03-28T18:55:47.164866+0000","last_clean":"2024-03-28T18:55:47.164866+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:55:47.164866+0000","last_undegraded":"2024-03-28T18:55:47.164866+0000","last_fullsized":"2024-03-28T18:55:47.164866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605818+0000","last_change":"2024-03-28T18:55:44.601415+0000","last_active":"2024-03-28T18:55:46.605818+0000","last_peered":"2024-03-28T18:55:46.605818+0000","last_clean":"2024-03-28T18:55:46.605818+0000","last_became_active":"2024-03-28T18:55:44.601124+0000","last_became_peered":"2024-03-28T18:55:44.601124+0000","last_unstale":"2024-03-28T18:55:46.605818+0000","last_undegraded":"2024-03-28T18:55:46.605818+0000","last_fullsized":"2024-03-28T18:55:46.605818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:30.767034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603094+0000","last_change":"2024-03-28T18:55:45.604795+0000","last_active":"2024-03-28T18:55:46.603094+0000","last_peered":"2024-03-28T18:55:46.603094+0000","last_clean":"2024-03-28T18:55:46.603094+0000","last_became_active":"2024-03-28T18:55:45.604432+0000","last_became_peered":"2024-03-28T18:55:45.604432+0000","last_unstale":"2024-03-28T18:55:46.603094+0000","last_undegraded":"2024-03-28T18:55:46.603094+0000","last_fullsized":"2024-03-28T18:55:46.603094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:37.638804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826963+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:55:46.826963+0000","last_peered":"2024-03-28T18:55:46.826963+0000","last_clean":"2024-03-28T18:55:46.826963+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:55:46.826963+0000","last_undegraded":"2024-03-28T18:55:46.826963+0000","last_fullsized":"2024-03-28T18:55:46.826963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613559+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:55:46.613559+0000","last_peered":"2024-03-28T18:55:46.613559+0000","last_clean":"2024-03-28T18:55:46.613559+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:55:46.613559+0000","last_undegraded":"2024-03-28T18:55:46.613559+0000","last_fullsized":"2024-03-28T18:55:46.613559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605718+0000","last_change":"2024-03-28T18:55:44.603859+0000","last_active":"2024-03-28T18:55:46.605718+0000","last_peered":"2024-03-28T18:55:46.605718+0000","last_clean":"2024-03-28T18:55:46.605718+0000","last_became_active":"2024-03-28T18:55:44.603540+0000","last_became_peered":"2024-03-28T18:55:44.603540+0000","last_unstale":"2024-03-28T18:55:46.605718+0000","last_undegraded":"2024-03-28T18:55:46.605718+0000","last_fullsized":"2024-03-28T18:55:46.605718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.606069+0000","last_change":"2024-03-28T18:55:45.608142+0000","last_active":"2024-03-28T18:55:46.606069+0000","last_peered":"2024-03-28T18:55:46.606069+0000","last_clean":"2024-03-28T18:55:46.606069+0000","last_became_active":"2024-03-28T18:55:45.606117+0000","last_became_peered":"2024-03-28T18:55:45.606117+0000","last_unstale":"2024-03-28T18:55:46.606069+0000","last_undegraded":"2024-03-28T18:55:46.606069+0000","last_fullsized":"2024-03-28T18:55:46.606069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826653+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:55:46.826653+0000","last_peered":"2024-03-28T18:55:46.826653+0000","last_clean":"2024-03-28T18:55:46.826653+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:55:46.826653+0000","last_undegraded":"2024-03-28T18:55:46.826653+0000","last_fullsized":"2024-03-28T18:55:46.826653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613045+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:55:46.613045+0000","last_peered":"2024-03-28T18:55:46.613045+0000","last_clean":"2024-03-28T18:55:46.613045+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:55:46.613045+0000","last_undegraded":"2024-03-28T18:55:46.613045+0000","last_fullsized":"2024-03-28T18:55:46.613045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614066+0000","last_change":"2024-03-28T18:55:44.606690+0000","last_active":"2024-03-28T18:55:46.614066+0000","last_peered":"2024-03-28T18:55:46.614066+0000","last_clean":"2024-03-28T18:55:46.614066+0000","last_became_active":"2024-03-28T18:55:44.606421+0000","last_became_peered":"2024-03-28T18:55:44.606421+0000","last_unstale":"2024-03-28T18:55:46.614066+0000","last_undegraded":"2024-03-28T18:55:46.614066+0000","last_fullsized":"2024-03-28T18:55:46.614066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117505+0000","last_change":"2024-03-28T18:55:45.610580+0000","last_active":"2024-03-28T18:55:47.117505+0000","last_peered":"2024-03-28T18:55:47.117505+0000","last_clean":"2024-03-28T18:55:47.117505+0000","last_became_active":"2024-03-28T18:55:45.610316+0000","last_became_peered":"2024-03-28T18:55:45.610316+0000","last_unstale":"2024-03-28T18:55:47.117505+0000","last_undegraded":"2024-03-28T18:55:47.117505+0000","last_fullsized":"2024-03-28T18:55:47.117505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604997+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:55:46.604997+0000","last_peered":"2024-03-28T18:55:46.604997+0000","last_clean":"2024-03-28T18:55:46.604997+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:55:46.604997+0000","last_undegraded":"2024-03-28T18:55:46.604997+0000","last_fullsized":"2024-03-28T18:55:46.604997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163833+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:55:47.163833+0000","last_peered":"2024-03-28T18:55:47.163833+0000","last_clean":"2024-03-28T18:55:47.163833+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:55:47.163833+0000","last_undegraded":"2024-03-28T18:55:47.163833+0000","last_fullsized":"2024-03-28T18:55:47.163833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.606207+0000","last_change":"2024-03-28T18:55:44.600523+0000","last_active":"2024-03-28T18:55:46.606207+0000","last_peered":"2024-03-28T18:55:46.606207+0000","last_clean":"2024-03-28T18:55:46.606207+0000","last_became_active":"2024-03-28T18:55:44.597550+0000","last_became_peered":"2024-03-28T18:55:44.597550+0000","last_unstale":"2024-03-28T18:55:46.606207+0000","last_undegraded":"2024-03-28T18:55:46.606207+0000","last_fullsized":"2024-03-28T18:55:46.606207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825927+0000","last_change":"2024-03-28T18:55:45.619798+0000","last_active":"2024-03-28T18:55:46.825927+0000","last_peered":"2024-03-28T18:55:46.825927+0000","last_clean":"2024-03-28T18:55:46.825927+0000","last_became_active":"2024-03-28T18:55:45.616517+0000","last_became_peered":"2024-03-28T18:55:45.616517+0000","last_unstale":"2024-03-28T18:55:46.825927+0000","last_undegraded":"2024-03-28T18:55:46.825927+0000","last_fullsized":"2024-03-28T18:55:46.825927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609158+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:55:46.609158+0000","last_peered":"2024-03-28T18:55:46.609158+0000","last_clean":"2024-03-28T18:55:46.609158+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:55:46.609158+0000","last_undegraded":"2024-03-28T18:55:46.609158+0000","last_fullsized":"2024-03-28T18:55:46.609158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605113+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:55:46.605113+0000","last_peered":"2024-03-28T18:55:46.605113+0000","last_clean":"2024-03-28T18:55:46.605113+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:55:46.605113+0000","last_undegraded":"2024-03-28T18:55:46.605113+0000","last_fullsized":"2024-03-28T18:55:46.605113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604527+0000","last_change":"2024-03-28T18:55:44.606498+0000","last_active":"2024-03-28T18:55:46.604527+0000","last_peered":"2024-03-28T18:55:46.604527+0000","last_clean":"2024-03-28T18:55:46.604527+0000","last_became_active":"2024-03-28T18:55:44.606098+0000","last_became_peered":"2024-03-28T18:55:44.606098+0000","last_unstale":"2024-03-28T18:55:46.604527+0000","last_undegraded":"2024-03-28T18:55:46.604527+0000","last_fullsized":"2024-03-28T18:55:46.604527+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605931+0000","last_change":"2024-03-28T18:55:45.617483+0000","last_active":"2024-03-28T18:55:46.605931+0000","last_peered":"2024-03-28T18:55:46.605931+0000","last_clean":"2024-03-28T18:55:46.605931+0000","last_became_active":"2024-03-28T18:55:45.617169+0000","last_became_peered":"2024-03-28T18:55:45.617169+0000","last_unstale":"2024-03-28T18:55:46.605931+0000","last_undegraded":"2024-03-28T18:55:46.605931+0000","last_fullsized":"2024-03-28T18:55:46.605931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829227+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:55:46.829227+0000","last_peered":"2024-03-28T18:55:46.829227+0000","last_clean":"2024-03-28T18:55:46.829227+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:55:46.829227+0000","last_undegraded":"2024-03-28T18:55:46.829227+0000","last_fullsized":"2024-03-28T18:55:46.829227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613667+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:55:46.613667+0000","last_peered":"2024-03-28T18:55:46.613667+0000","last_clean":"2024-03-28T18:55:46.613667+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:55:46.613667+0000","last_undegraded":"2024-03-28T18:55:46.613667+0000","last_fullsized":"2024-03-28T18:55:46.613667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825795+0000","last_change":"2024-03-28T18:55:44.601836+0000","last_active":"2024-03-28T18:55:46.825795+0000","last_peered":"2024-03-28T18:55:46.825795+0000","last_clean":"2024-03-28T18:55:46.825795+0000","last_became_active":"2024-03-28T18:55:44.601552+0000","last_became_peered":"2024-03-28T18:55:44.601552+0000","last_unstale":"2024-03-28T18:55:46.825795+0000","last_undegraded":"2024-03-28T18:55:46.825795+0000","last_fullsized":"2024-03-28T18:55:46.825795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603619+0000","last_change":"2024-03-28T18:55:45.601801+0000","last_active":"2024-03-28T18:55:46.603619+0000","last_peered":"2024-03-28T18:55:46.603619+0000","last_clean":"2024-03-28T18:55:46.603619+0000","last_became_active":"2024-03-28T18:55:45.601459+0000","last_became_peered":"2024-03-28T18:55:45.601459+0000","last_unstale":"2024-03-28T18:55:46.603619+0000","last_undegraded":"2024-03-28T18:55:46.603619+0000","last_fullsized":"2024-03-28T18:55:46.603619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:43.033170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613982+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:55:46.613982+0000","last_peered":"2024-03-28T18:55:46.613982+0000","last_clean":"2024-03-28T18:55:46.613982+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:55:46.613982+0000","last_undegraded":"2024-03-28T18:55:46.613982+0000","last_fullsized":"2024-03-28T18:55:46.613982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165028+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:55:47.165028+0000","last_peered":"2024-03-28T18:55:47.165028+0000","last_clean":"2024-03-28T18:55:47.165028+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:55:47.165028+0000","last_undegraded":"2024-03-28T18:55:47.165028+0000","last_fullsized":"2024-03-28T18:55:47.165028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608494+0000","last_change":"2024-03-28T18:55:44.611148+0000","last_active":"2024-03-28T18:55:46.608494+0000","last_peered":"2024-03-28T18:55:46.608494+0000","last_clean":"2024-03-28T18:55:46.608494+0000","last_became_active":"2024-03-28T18:55:44.610835+0000","last_became_peered":"2024-03-28T18:55:44.610835+0000","last_unstale":"2024-03-28T18:55:46.608494+0000","last_undegraded":"2024-03-28T18:55:46.608494+0000","last_fullsized":"2024-03-28T18:55:46.608494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608851+0000","last_change":"2024-03-28T18:55:45.623672+0000","last_active":"2024-03-28T18:55:46.608851+0000","last_peered":"2024-03-28T18:55:46.608851+0000","last_clean":"2024-03-28T18:55:46.608851+0000","last_became_active":"2024-03-28T18:55:45.622568+0000","last_became_peered":"2024-03-28T18:55:45.622568+0000","last_unstale":"2024-03-28T18:55:46.608851+0000","last_undegraded":"2024-03-28T18:55:46.608851+0000","last_fullsized":"2024-03-28T18:55:46.608851+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117731+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:55:47.117731+0000","last_peered":"2024-03-28T18:55:47.117731+0000","last_clean":"2024-03-28T18:55:47.117731+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:55:47.117731+0000","last_undegraded":"2024-03-28T18:55:47.117731+0000","last_fullsized":"2024-03-28T18:55:47.117731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614041+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:55:46.614041+0000","last_peered":"2024-03-28T18:55:46.614041+0000","last_clean":"2024-03-28T18:55:46.614041+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:55:46.614041+0000","last_undegraded":"2024-03-28T18:55:46.614041+0000","last_fullsized":"2024-03-28T18:55:46.614041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613204+0000","last_change":"2024-03-28T18:55:44.604077+0000","last_active":"2024-03-28T18:55:46.613204+0000","last_peered":"2024-03-28T18:55:46.613204+0000","last_clean":"2024-03-28T18:55:46.613204+0000","last_became_active":"2024-03-28T18:55:44.603773+0000","last_became_peered":"2024-03-28T18:55:44.603773+0000","last_unstale":"2024-03-28T18:55:46.613204+0000","last_undegraded":"2024-03-28T18:55:46.613204+0000","last_fullsized":"2024-03-28T18:55:46.613204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608665+0000","last_change":"2024-03-28T18:55:45.615444+0000","last_active":"2024-03-28T18:55:46.608665+0000","last_peered":"2024-03-28T18:55:46.608665+0000","last_clean":"2024-03-28T18:55:46.608665+0000","last_became_active":"2024-03-28T18:55:45.615002+0000","last_became_peered":"2024-03-28T18:55:45.615002+0000","last_unstale":"2024-03-28T18:55:46.608665+0000","last_undegraded":"2024-03-28T18:55:46.608665+0000","last_fullsized":"2024-03-28T18:55:46.608665+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605186+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:55:46.605186+0000","last_peered":"2024-03-28T18:55:46.605186+0000","last_clean":"2024-03-28T18:55:46.605186+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:55:46.605186+0000","last_undegraded":"2024-03-28T18:55:46.605186+0000","last_fullsized":"2024-03-28T18:55:46.605186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116841+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:55:47.116841+0000","last_peered":"2024-03-28T18:55:47.116841+0000","last_clean":"2024-03-28T18:55:47.116841+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:55:47.116841+0000","last_undegraded":"2024-03-28T18:55:47.116841+0000","last_fullsized":"2024-03-28T18:55:47.116841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613947+0000","last_change":"2024-03-28T18:55:44.590703+0000","last_active":"2024-03-28T18:55:46.613947+0000","last_peered":"2024-03-28T18:55:46.613947+0000","last_clean":"2024-03-28T18:55:46.613947+0000","last_became_active":"2024-03-28T18:55:44.590408+0000","last_became_peered":"2024-03-28T18:55:44.590408+0000","last_unstale":"2024-03-28T18:55:46.613947+0000","last_undegraded":"2024-03-28T18:55:46.613947+0000","last_fullsized":"2024-03-28T18:55:46.613947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603326+0000","last_change":"2024-03-28T18:55:45.609795+0000","last_active":"2024-03-28T18:55:46.603326+0000","last_peered":"2024-03-28T18:55:46.603326+0000","last_clean":"2024-03-28T18:55:46.603326+0000","last_became_active":"2024-03-28T18:55:45.603695+0000","last_became_peered":"2024-03-28T18:55:45.603695+0000","last_unstale":"2024-03-28T18:55:46.603326+0000","last_undegraded":"2024-03-28T18:55:46.603326+0000","last_fullsized":"2024-03-28T18:55:46.603326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:49.173555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612873+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:55:46.612873+0000","last_peered":"2024-03-28T18:55:46.612873+0000","last_clean":"2024-03-28T18:55:46.612873+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:55:46.612873+0000","last_undegraded":"2024-03-28T18:55:46.612873+0000","last_fullsized":"2024-03-28T18:55:46.612873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602878+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:55:46.602878+0000","last_peered":"2024-03-28T18:55:46.602878+0000","last_clean":"2024-03-28T18:55:46.602878+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:55:46.602878+0000","last_undegraded":"2024-03-28T18:55:46.602878+0000","last_fullsized":"2024-03-28T18:55:46.602878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614195+0000","last_change":"2024-03-28T18:55:44.612172+0000","last_active":"2024-03-28T18:55:46.614195+0000","last_peered":"2024-03-28T18:55:46.614195+0000","last_clean":"2024-03-28T18:55:46.614195+0000","last_became_active":"2024-03-28T18:55:44.611859+0000","last_became_peered":"2024-03-28T18:55:44.611859+0000","last_unstale":"2024-03-28T18:55:46.614195+0000","last_undegraded":"2024-03-28T18:55:46.614195+0000","last_fullsized":"2024-03-28T18:55:46.614195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117969+0000","last_change":"2024-03-28T18:55:45.619723+0000","last_active":"2024-03-28T18:55:47.117969+0000","last_peered":"2024-03-28T18:55:47.117969+0000","last_clean":"2024-03-28T18:55:47.117969+0000","last_became_active":"2024-03-28T18:55:45.619437+0000","last_became_peered":"2024-03-28T18:55:45.619437+0000","last_unstale":"2024-03-28T18:55:47.117969+0000","last_undegraded":"2024-03-28T18:55:47.117969+0000","last_fullsized":"2024-03-28T18:55:47.117969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608071+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:55:46.608071+0000","last_peered":"2024-03-28T18:55:46.608071+0000","last_clean":"2024-03-28T18:55:46.608071+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:55:46.608071+0000","last_undegraded":"2024-03-28T18:55:46.608071+0000","last_fullsized":"2024-03-28T18:55:46.608071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164105+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:55:47.164105+0000","last_peered":"2024-03-28T18:55:47.164105+0000","last_clean":"2024-03-28T18:55:47.164105+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:55:47.164105+0000","last_undegraded":"2024-03-28T18:55:47.164105+0000","last_fullsized":"2024-03-28T18:55:47.164105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.611391+0000","last_change":"2024-03-28T18:55:44.592064+0000","last_active":"2024-03-28T18:55:46.611391+0000","last_peered":"2024-03-28T18:55:46.611391+0000","last_clean":"2024-03-28T18:55:46.611391+0000","last_became_active":"2024-03-28T18:55:44.591263+0000","last_became_peered":"2024-03-28T18:55:44.591263+0000","last_unstale":"2024-03-28T18:55:46.611391+0000","last_undegraded":"2024-03-28T18:55:46.611391+0000","last_fullsized":"2024-03-28T18:55:46.611391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163958+0000","last_change":"2024-03-28T18:55:45.614820+0000","last_active":"2024-03-28T18:55:47.163958+0000","last_peered":"2024-03-28T18:55:47.163958+0000","last_clean":"2024-03-28T18:55:47.163958+0000","last_became_active":"2024-03-28T18:55:45.614517+0000","last_became_peered":"2024-03-28T18:55:45.614517+0000","last_unstale":"2024-03-28T18:55:47.163958+0000","last_undegraded":"2024-03-28T18:55:47.163958+0000","last_fullsized":"2024-03-28T18:55:47.163958+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826245+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:55:46.826245+0000","last_peered":"2024-03-28T18:55:46.826245+0000","last_clean":"2024-03-28T18:55:46.826245+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:55:46.826245+0000","last_undegraded":"2024-03-28T18:55:46.826245+0000","last_fullsized":"2024-03-28T18:55:46.826245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116294+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:55:47.116294+0000","last_peered":"2024-03-28T18:55:47.116294+0000","last_clean":"2024-03-28T18:55:47.116294+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:55:47.116294+0000","last_undegraded":"2024-03-28T18:55:47.116294+0000","last_fullsized":"2024-03-28T18:55:47.116294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164472+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:55:47.164472+0000","last_peered":"2024-03-28T18:55:47.164472+0000","last_clean":"2024-03-28T18:55:47.164472+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:55:47.164472+0000","last_undegraded":"2024-03-28T18:55:47.164472+0000","last_fullsized":"2024-03-28T18:55:47.164472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602907+0000","last_change":"2024-03-28T18:55:44.602097+0000","last_active":"2024-03-28T18:55:46.602907+0000","last_peered":"2024-03-28T18:55:46.602907+0000","last_clean":"2024-03-28T18:55:46.602907+0000","last_became_active":"2024-03-28T18:55:44.601769+0000","last_became_peered":"2024-03-28T18:55:44.601769+0000","last_unstale":"2024-03-28T18:55:46.602907+0000","last_undegraded":"2024-03-28T18:55:46.602907+0000","last_fullsized":"2024-03-28T18:55:46.602907+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603447+0000","last_change":"2024-03-28T18:55:45.616485+0000","last_active":"2024-03-28T18:55:46.603447+0000","last_peered":"2024-03-28T18:55:46.603447+0000","last_clean":"2024-03-28T18:55:46.603447+0000","last_became_active":"2024-03-28T18:55:45.615813+0000","last_became_peered":"2024-03-28T18:55:45.615813+0000","last_unstale":"2024-03-28T18:55:46.603447+0000","last_undegraded":"2024-03-28T18:55:46.603447+0000","last_fullsized":"2024-03-28T18:55:46.603447+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:21.227281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164685+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:55:47.164685+0000","last_peered":"2024-03-28T18:55:47.164685+0000","last_clean":"2024-03-28T18:55:47.164685+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:55:47.164685+0000","last_undegraded":"2024-03-28T18:55:47.164685+0000","last_fullsized":"2024-03-28T18:55:47.164685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603451+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:55:46.603451+0000","last_peered":"2024-03-28T18:55:46.603451+0000","last_clean":"2024-03-28T18:55:46.603451+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:55:46.603451+0000","last_undegraded":"2024-03-28T18:55:46.603451+0000","last_fullsized":"2024-03-28T18:55:46.603451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603782+0000","last_change":"2024-03-28T18:55:44.607744+0000","last_active":"2024-03-28T18:55:46.603782+0000","last_peered":"2024-03-28T18:55:46.603782+0000","last_clean":"2024-03-28T18:55:46.603782+0000","last_became_active":"2024-03-28T18:55:44.606993+0000","last_became_peered":"2024-03-28T18:55:44.606993+0000","last_unstale":"2024-03-28T18:55:46.603782+0000","last_undegraded":"2024-03-28T18:55:46.603782+0000","last_fullsized":"2024-03-28T18:55:46.603782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602563+0000","last_change":"2024-03-28T18:55:45.606307+0000","last_active":"2024-03-28T18:55:46.602563+0000","last_peered":"2024-03-28T18:55:46.602563+0000","last_clean":"2024-03-28T18:55:46.602563+0000","last_became_active":"2024-03-28T18:55:45.605980+0000","last_became_peered":"2024-03-28T18:55:45.605980+0000","last_unstale":"2024-03-28T18:55:46.602563+0000","last_undegraded":"2024-03-28T18:55:46.602563+0000","last_fullsized":"2024-03-28T18:55:46.602563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:52.485966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117600+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:55:47.117600+0000","last_peered":"2024-03-28T18:55:47.117600+0000","last_clean":"2024-03-28T18:55:47.117600+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:55:47.117600+0000","last_undegraded":"2024-03-28T18:55:47.117600+0000","last_fullsized":"2024-03-28T18:55:47.117600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614275+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:55:46.614275+0000","last_peered":"2024-03-28T18:55:46.614275+0000","last_clean":"2024-03-28T18:55:46.614275+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:55:46.614275+0000","last_undegraded":"2024-03-28T18:55:46.614275+0000","last_fullsized":"2024-03-28T18:55:46.614275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608346+0000","last_change":"2024-03-28T18:55:44.607323+0000","last_active":"2024-03-28T18:55:46.608346+0000","last_peered":"2024-03-28T18:55:46.608346+0000","last_clean":"2024-03-28T18:55:46.608346+0000","last_became_active":"2024-03-28T18:55:44.606579+0000","last_became_peered":"2024-03-28T18:55:44.606579+0000","last_unstale":"2024-03-28T18:55:46.608346+0000","last_undegraded":"2024-03-28T18:55:46.608346+0000","last_fullsized":"2024-03-28T18:55:46.608346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163490+0000","last_change":"2024-03-28T18:55:45.622760+0000","last_active":"2024-03-28T18:55:47.163490+0000","last_peered":"2024-03-28T18:55:47.163490+0000","last_clean":"2024-03-28T18:55:47.163490+0000","last_became_active":"2024-03-28T18:55:45.622353+0000","last_became_peered":"2024-03-28T18:55:45.622353+0000","last_unstale":"2024-03-28T18:55:47.163490+0000","last_undegraded":"2024-03-28T18:55:47.163490+0000","last_fullsized":"2024-03-28T18:55:47.163490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:04.349689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166444+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:55:47.166444+0000","last_peered":"2024-03-28T18:55:47.166444+0000","last_clean":"2024-03-28T18:55:47.166444+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:55:47.166444+0000","last_undegraded":"2024-03-28T18:55:47.166444+0000","last_fullsized":"2024-03-28T18:55:47.166444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603346+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:55:46.603346+0000","last_peered":"2024-03-28T18:55:46.603346+0000","last_clean":"2024-03-28T18:55:46.603346+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:55:46.603346+0000","last_undegraded":"2024-03-28T18:55:46.603346+0000","last_fullsized":"2024-03-28T18:55:46.603346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116165+0000","last_change":"2024-03-28T18:55:44.607237+0000","last_active":"2024-03-28T18:55:47.116165+0000","last_peered":"2024-03-28T18:55:47.116165+0000","last_clean":"2024-03-28T18:55:47.116165+0000","last_became_active":"2024-03-28T18:55:44.606960+0000","last_became_peered":"2024-03-28T18:55:44.606960+0000","last_unstale":"2024-03-28T18:55:47.116165+0000","last_undegraded":"2024-03-28T18:55:47.116165+0000","last_fullsized":"2024-03-28T18:55:47.116165+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603242+0000","last_change":"2024-03-28T18:55:45.612992+0000","last_active":"2024-03-28T18:55:46.603242+0000","last_peered":"2024-03-28T18:55:46.603242+0000","last_clean":"2024-03-28T18:55:46.603242+0000","last_became_active":"2024-03-28T18:55:45.612125+0000","last_became_peered":"2024-03-28T18:55:45.612125+0000","last_unstale":"2024-03-28T18:55:46.603242+0000","last_undegraded":"2024-03-28T18:55:46.603242+0000","last_fullsized":"2024-03-28T18:55:46.603242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:35.898911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165571+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:55:47.165571+0000","last_peered":"2024-03-28T18:55:47.165571+0000","last_clean":"2024-03-28T18:55:47.165571+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:55:47.165571+0000","last_undegraded":"2024-03-28T18:55:47.165571+0000","last_fullsized":"2024-03-28T18:55:47.165571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605603+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:55:46.605603+0000","last_peered":"2024-03-28T18:55:46.605603+0000","last_clean":"2024-03-28T18:55:46.605603+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:55:46.605603+0000","last_undegraded":"2024-03-28T18:55:46.605603+0000","last_fullsized":"2024-03-28T18:55:46.605603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604200+0000","last_change":"2024-03-28T18:55:44.607218+0000","last_active":"2024-03-28T18:55:46.604200+0000","last_peered":"2024-03-28T18:55:46.604200+0000","last_clean":"2024-03-28T18:55:46.604200+0000","last_became_active":"2024-03-28T18:55:44.606852+0000","last_became_peered":"2024-03-28T18:55:44.606852+0000","last_unstale":"2024-03-28T18:55:46.604200+0000","last_undegraded":"2024-03-28T18:55:46.604200+0000","last_fullsized":"2024-03-28T18:55:46.604200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:36.624289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829038+0000","last_change":"2024-03-28T18:55:45.618919+0000","last_active":"2024-03-28T18:55:46.829038+0000","last_peered":"2024-03-28T18:55:46.829038+0000","last_clean":"2024-03-28T18:55:46.829038+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:55:46.829038+0000","last_undegraded":"2024-03-28T18:55:46.829038+0000","last_fullsized":"2024-03-28T18:55:46.829038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:29.407508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164081+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:55:47.164081+0000","last_peered":"2024-03-28T18:55:47.164081+0000","last_clean":"2024-03-28T18:55:47.164081+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:55:47.164081+0000","last_undegraded":"2024-03-28T18:55:47.164081+0000","last_fullsized":"2024-03-28T18:55:47.164081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602411+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:55:46.602411+0000","last_peered":"2024-03-28T18:55:46.602411+0000","last_clean":"2024-03-28T18:55:46.602411+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:55:46.602411+0000","last_undegraded":"2024-03-28T18:55:46.602411+0000","last_fullsized":"2024-03-28T18:55:46.602411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609206+0000","last_change":"2024-03-28T18:55:44.598881+0000","last_active":"2024-03-28T18:55:46.609206+0000","last_peered":"2024-03-28T18:55:46.609206+0000","last_clean":"2024-03-28T18:55:46.609206+0000","last_became_active":"2024-03-28T18:55:44.598532+0000","last_became_peered":"2024-03-28T18:55:44.598532+0000","last_unstale":"2024-03-28T18:55:46.609206+0000","last_undegraded":"2024-03-28T18:55:46.609206+0000","last_fullsized":"2024-03-28T18:55:46.609206+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:44.004611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117993+0000","last_change":"2024-03-28T18:55:45.612572+0000","last_active":"2024-03-28T18:55:47.117993+0000","last_peered":"2024-03-28T18:55:47.117993+0000","last_clean":"2024-03-28T18:55:47.117993+0000","last_became_active":"2024-03-28T18:55:45.611844+0000","last_became_peered":"2024-03-28T18:55:45.611844+0000","last_unstale":"2024-03-28T18:55:47.117993+0000","last_undegraded":"2024-03-28T18:55:47.117993+0000","last_fullsized":"2024-03-28T18:55:47.117993+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602718+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:55:46.602718+0000","last_peered":"2024-03-28T18:55:46.602718+0000","last_clean":"2024-03-28T18:55:46.602718+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:55:46.602718+0000","last_undegraded":"2024-03-28T18:55:46.602718+0000","last_fullsized":"2024-03-28T18:55:46.602718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603429+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:55:46.603429+0000","last_peered":"2024-03-28T18:55:46.603429+0000","last_clean":"2024-03-28T18:55:46.603429+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:55:46.603429+0000","last_undegraded":"2024-03-28T18:55:46.603429+0000","last_fullsized":"2024-03-28T18:55:46.603429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829075+0000","last_change":"2024-03-28T18:55:44.609644+0000","last_active":"2024-03-28T18:55:46.829075+0000","last_peered":"2024-03-28T18:55:46.829075+0000","last_clean":"2024-03-28T18:55:46.829075+0000","last_became_active":"2024-03-28T18:55:44.609317+0000","last_became_peered":"2024-03-28T18:55:44.609317+0000","last_unstale":"2024-03-28T18:55:46.829075+0000","last_undegraded":"2024-03-28T18:55:46.829075+0000","last_fullsized":"2024-03-28T18:55:46.829075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:37.662504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116808+0000","last_change":"2024-03-28T18:55:45.621708+0000","last_active":"2024-03-28T18:55:47.116808+0000","last_peered":"2024-03-28T18:55:47.116808+0000","last_clean":"2024-03-28T18:55:47.116808+0000","last_became_active":"2024-03-28T18:55:45.621366+0000","last_became_peered":"2024-03-28T18:55:45.621366+0000","last_unstale":"2024-03-28T18:55:47.116808+0000","last_undegraded":"2024-03-28T18:55:47.116808+0000","last_fullsized":"2024-03-28T18:55:47.116808+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164622+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:55:47.164622+0000","last_peered":"2024-03-28T18:55:47.164622+0000","last_clean":"2024-03-28T18:55:47.164622+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:55:47.164622+0000","last_undegraded":"2024-03-28T18:55:47.164622+0000","last_fullsized":"2024-03-28T18:55:47.164622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116814+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:55:47.116814+0000","last_peered":"2024-03-28T18:55:47.116814+0000","last_clean":"2024-03-28T18:55:47.116814+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:55:47.116814+0000","last_undegraded":"2024-03-28T18:55:47.116814+0000","last_fullsized":"2024-03-28T18:55:47.116814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"23'6","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827637+0000","last_change":"2024-03-28T18:55:44.608949+0000","last_active":"2024-03-28T18:55:46.827637+0000","last_peered":"2024-03-28T18:55:46.827637+0000","last_clean":"2024-03-28T18:55:46.827637+0000","last_became_active":"2024-03-28T18:55:44.608657+0000","last_became_peered":"2024-03-28T18:55:44.608657+0000","last_unstale":"2024-03-28T18:55:46.827637+0000","last_undegraded":"2024-03-28T18:55:46.827637+0000","last_fullsized":"2024-03-28T18:55:46.827637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:04.799314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827662+0000","last_change":"2024-03-28T18:55:45.610686+0000","last_active":"2024-03-28T18:55:46.827662+0000","last_peered":"2024-03-28T18:55:46.827662+0000","last_clean":"2024-03-28T18:55:46.827662+0000","last_became_active":"2024-03-28T18:55:45.610212+0000","last_became_peered":"2024-03-28T18:55:45.610212+0000","last_unstale":"2024-03-28T18:55:46.827662+0000","last_undegraded":"2024-03-28T18:55:46.827662+0000","last_fullsized":"2024-03-28T18:55:46.827662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608274+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:55:46.608274+0000","last_peered":"2024-03-28T18:55:46.608274+0000","last_clean":"2024-03-28T18:55:46.608274+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:55:46.608274+0000","last_undegraded":"2024-03-28T18:55:46.608274+0000","last_fullsized":"2024-03-28T18:55:46.608274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609087+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:55:46.609087+0000","last_peered":"2024-03-28T18:55:46.609087+0000","last_clean":"2024-03-28T18:55:46.609087+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:55:46.609087+0000","last_undegraded":"2024-03-28T18:55:46.609087+0000","last_fullsized":"2024-03-28T18:55:46.609087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"23'3","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117447+0000","last_change":"2024-03-28T18:55:44.602762+0000","last_active":"2024-03-28T18:55:47.117447+0000","last_peered":"2024-03-28T18:55:47.117447+0000","last_clean":"2024-03-28T18:55:47.117447+0000","last_became_active":"2024-03-28T18:55:44.602424+0000","last_became_peered":"2024-03-28T18:55:44.602424+0000","last_unstale":"2024-03-28T18:55:47.117447+0000","last_undegraded":"2024-03-28T18:55:47.117447+0000","last_fullsized":"2024-03-28T18:55:47.117447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826897+0000","last_change":"2024-03-28T18:55:45.601277+0000","last_active":"2024-03-28T18:55:46.826897+0000","last_peered":"2024-03-28T18:55:46.826897+0000","last_clean":"2024-03-28T18:55:46.826897+0000","last_became_active":"2024-03-28T18:55:45.600945+0000","last_became_peered":"2024-03-28T18:55:45.600945+0000","last_unstale":"2024-03-28T18:55:46.826897+0000","last_undegraded":"2024-03-28T18:55:46.826897+0000","last_fullsized":"2024-03-28T18:55:46.826897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164660+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:55:47.164660+0000","last_peered":"2024-03-28T18:55:47.164660+0000","last_clean":"2024-03-28T18:55:47.164660+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:55:47.164660+0000","last_undegraded":"2024-03-28T18:55:47.164660+0000","last_fullsized":"2024-03-28T18:55:47.164660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602857+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:55:46.602857+0000","last_peered":"2024-03-28T18:55:46.602857+0000","last_clean":"2024-03-28T18:55:46.602857+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:55:46.602857+0000","last_undegraded":"2024-03-28T18:55:46.602857+0000","last_fullsized":"2024-03-28T18:55:46.602857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612960+0000","last_change":"2024-03-28T18:55:44.599432+0000","last_active":"2024-03-28T18:55:46.612960+0000","last_peered":"2024-03-28T18:55:46.612960+0000","last_clean":"2024-03-28T18:55:46.612960+0000","last_became_active":"2024-03-28T18:55:44.599123+0000","last_became_peered":"2024-03-28T18:55:44.599123+0000","last_unstale":"2024-03-28T18:55:46.612960+0000","last_undegraded":"2024-03-28T18:55:46.612960+0000","last_fullsized":"2024-03-28T18:55:46.612960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613285+0000","last_change":"2024-03-28T18:55:45.624311+0000","last_active":"2024-03-28T18:55:46.613285+0000","last_peered":"2024-03-28T18:55:46.613285+0000","last_clean":"2024-03-28T18:55:46.613285+0000","last_became_active":"2024-03-28T18:55:45.624031+0000","last_became_peered":"2024-03-28T18:55:45.624031+0000","last_unstale":"2024-03-28T18:55:46.613285+0000","last_undegraded":"2024-03-28T18:55:46.613285+0000","last_fullsized":"2024-03-28T18:55:46.613285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605962+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:55:46.605962+0000","last_peered":"2024-03-28T18:55:46.605962+0000","last_clean":"2024-03-28T18:55:46.605962+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:55:46.605962+0000","last_undegraded":"2024-03-28T18:55:46.605962+0000","last_fullsized":"2024-03-28T18:55:46.605962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614210+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:55:46.614210+0000","last_peered":"2024-03-28T18:55:46.614210+0000","last_clean":"2024-03-28T18:55:46.614210+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:55:46.614210+0000","last_undegraded":"2024-03-28T18:55:46.614210+0000","last_fullsized":"2024-03-28T18:55:46.614210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604974+0000","last_change":"2024-03-28T18:55:44.593567+0000","last_active":"2024-03-28T18:55:46.604974+0000","last_peered":"2024-03-28T18:55:46.604974+0000","last_clean":"2024-03-28T18:55:46.604974+0000","last_became_active":"2024-03-28T18:55:44.593267+0000","last_became_peered":"2024-03-28T18:55:44.593267+0000","last_unstale":"2024-03-28T18:55:46.604974+0000","last_undegraded":"2024-03-28T18:55:46.604974+0000","last_fullsized":"2024-03-28T18:55:46.604974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603331+0000","last_change":"2024-03-28T18:55:45.598713+0000","last_active":"2024-03-28T18:55:46.603331+0000","last_peered":"2024-03-28T18:55:46.603331+0000","last_clean":"2024-03-28T18:55:46.603331+0000","last_became_active":"2024-03-28T18:55:45.597726+0000","last_became_peered":"2024-03-28T18:55:45.597726+0000","last_unstale":"2024-03-28T18:55:46.603331+0000","last_undegraded":"2024-03-28T18:55:46.603331+0000","last_fullsized":"2024-03-28T18:55:46.603331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:48.255745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609101+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:55:46.609101+0000","last_peered":"2024-03-28T18:55:46.609101+0000","last_clean":"2024-03-28T18:55:46.609101+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:55:46.609101+0000","last_undegraded":"2024-03-28T18:55:46.609101+0000","last_fullsized":"2024-03-28T18:55:46.609101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118062+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:55:47.118062+0000","last_peered":"2024-03-28T18:55:47.118062+0000","last_clean":"2024-03-28T18:55:47.118062+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:55:47.118062+0000","last_undegraded":"2024-03-28T18:55:47.118062+0000","last_fullsized":"2024-03-28T18:55:47.118062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"23'8","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826291+0000","last_change":"2024-03-28T18:55:44.610673+0000","last_active":"2024-03-28T18:55:46.826291+0000","last_peered":"2024-03-28T18:55:46.826291+0000","last_clean":"2024-03-28T18:55:46.826291+0000","last_became_active":"2024-03-28T18:55:44.610357+0000","last_became_peered":"2024-03-28T18:55:44.610357+0000","last_unstale":"2024-03-28T18:55:46.826291+0000","last_undegraded":"2024-03-28T18:55:46.826291+0000","last_fullsized":"2024-03-28T18:55:46.826291+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:26.280548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163667+0000","last_change":"2024-03-28T18:55:45.609349+0000","last_active":"2024-03-28T18:55:47.163667+0000","last_peered":"2024-03-28T18:55:47.163667+0000","last_clean":"2024-03-28T18:55:47.163667+0000","last_became_active":"2024-03-28T18:55:45.608743+0000","last_became_peered":"2024-03-28T18:55:45.608743+0000","last_unstale":"2024-03-28T18:55:47.163667+0000","last_undegraded":"2024-03-28T18:55:47.163667+0000","last_fullsized":"2024-03-28T18:55:47.163667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118036+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:55:47.118036+0000","last_peered":"2024-03-28T18:55:47.118036+0000","last_clean":"2024-03-28T18:55:47.118036+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:55:47.118036+0000","last_undegraded":"2024-03-28T18:55:47.118036+0000","last_fullsized":"2024-03-28T18:55:47.118036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604933+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:55:46.604933+0000","last_peered":"2024-03-28T18:55:46.604933+0000","last_clean":"2024-03-28T18:55:46.604933+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:55:46.604933+0000","last_undegraded":"2024-03-28T18:55:46.604933+0000","last_fullsized":"2024-03-28T18:55:46.604933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825952+0000","last_change":"2024-03-28T18:55:44.601103+0000","last_active":"2024-03-28T18:55:46.825952+0000","last_peered":"2024-03-28T18:55:46.825952+0000","last_clean":"2024-03-28T18:55:46.825952+0000","last_became_active":"2024-03-28T18:55:44.599969+0000","last_became_peered":"2024-03-28T18:55:44.599969+0000","last_unstale":"2024-03-28T18:55:46.825952+0000","last_undegraded":"2024-03-28T18:55:46.825952+0000","last_fullsized":"2024-03-28T18:55:46.825952+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:42.018654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604371+0000","last_change":"2024-03-28T18:55:45.594944+0000","last_active":"2024-03-28T18:55:46.604371+0000","last_peered":"2024-03-28T18:55:46.604371+0000","last_clean":"2024-03-28T18:55:46.604371+0000","last_became_active":"2024-03-28T18:55:45.594603+0000","last_became_peered":"2024-03-28T18:55:45.594603+0000","last_unstale":"2024-03-28T18:55:46.604371+0000","last_undegraded":"2024-03-28T18:55:46.604371+0000","last_fullsized":"2024-03-28T18:55:46.604371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608967+0000","last_change":"2024-03-28T18:55:44.602366+0000","last_active":"2024-03-28T18:55:46.608967+0000","last_peered":"2024-03-28T18:55:46.608967+0000","last_clean":"2024-03-28T18:55:46.608967+0000","last_became_active":"2024-03-28T18:55:44.602033+0000","last_became_peered":"2024-03-28T18:55:44.602033+0000","last_unstale":"2024-03-28T18:55:46.608967+0000","last_undegraded":"2024-03-28T18:55:46.608967+0000","last_fullsized":"2024-03-28T18:55:46.608967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612862+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:55:46.612862+0000","last_peered":"2024-03-28T18:55:46.612862+0000","last_clean":"2024-03-28T18:55:46.612862+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:55:46.612862+0000","last_undegraded":"2024-03-28T18:55:46.612862+0000","last_fullsized":"2024-03-28T18:55:46.612862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117107+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:55:47.117107+0000","last_peered":"2024-03-28T18:55:47.117107+0000","last_clean":"2024-03-28T18:55:47.117107+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:55:47.117107+0000","last_undegraded":"2024-03-28T18:55:47.117107+0000","last_fullsized":"2024-03-28T18:55:47.117107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608811+0000","last_change":"2024-03-28T18:55:45.620559+0000","last_active":"2024-03-28T18:55:46.608811+0000","last_peered":"2024-03-28T18:55:46.608811+0000","last_clean":"2024-03-28T18:55:46.608811+0000","last_became_active":"2024-03-28T18:55:45.620249+0000","last_became_peered":"2024-03-28T18:55:45.620249+0000","last_unstale":"2024-03-28T18:55:46.608811+0000","last_undegraded":"2024-03-28T18:55:46.608811+0000","last_fullsized":"2024-03-28T18:55:46.608811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"23'4","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604650+0000","last_change":"2024-03-28T18:55:44.612226+0000","last_active":"2024-03-28T18:55:46.604650+0000","last_peered":"2024-03-28T18:55:46.604650+0000","last_clean":"2024-03-28T18:55:46.604650+0000","last_became_active":"2024-03-28T18:55:44.611921+0000","last_became_peered":"2024-03-28T18:55:44.611921+0000","last_unstale":"2024-03-28T18:55:46.604650+0000","last_undegraded":"2024-03-28T18:55:46.604650+0000","last_fullsized":"2024-03-28T18:55:46.604650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826474+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:55:46.826474+0000","last_peered":"2024-03-28T18:55:46.826474+0000","last_clean":"2024-03-28T18:55:46.826474+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:55:46.826474+0000","last_undegraded":"2024-03-28T18:55:46.826474+0000","last_fullsized":"2024-03-28T18:55:46.826474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826438+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:55:46.826438+0000","last_peered":"2024-03-28T18:55:46.826438+0000","last_clean":"2024-03-28T18:55:46.826438+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:55:46.826438+0000","last_undegraded":"2024-03-28T18:55:46.826438+0000","last_fullsized":"2024-03-28T18:55:46.826438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614060+0000","last_change":"2024-03-28T18:55:45.612727+0000","last_active":"2024-03-28T18:55:46.614060+0000","last_peered":"2024-03-28T18:55:46.614060+0000","last_clean":"2024-03-28T18:55:46.614060+0000","last_became_active":"2024-03-28T18:55:45.610154+0000","last_became_peered":"2024-03-28T18:55:45.610154+0000","last_unstale":"2024-03-28T18:55:46.614060+0000","last_undegraded":"2024-03-28T18:55:46.614060+0000","last_fullsized":"2024-03-28T18:55:46.614060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604120+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:55:46.604120+0000","last_peered":"2024-03-28T18:55:46.604120+0000","last_clean":"2024-03-28T18:55:46.604120+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:55:46.604120+0000","last_undegraded":"2024-03-28T18:55:46.604120+0000","last_fullsized":"2024-03-28T18:55:46.604120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608921+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:55:46.608921+0000","last_peered":"2024-03-28T18:55:46.608921+0000","last_clean":"2024-03-28T18:55:46.608921+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:55:46.608921+0000","last_undegraded":"2024-03-28T18:55:46.608921+0000","last_fullsized":"2024-03-28T18:55:46.608921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603331+0000","last_change":"2024-03-28T18:55:44.592835+0000","last_active":"2024-03-28T18:55:46.603331+0000","last_peered":"2024-03-28T18:55:46.603331+0000","last_clean":"2024-03-28T18:55:46.603331+0000","last_became_active":"2024-03-28T18:55:44.592505+0000","last_became_peered":"2024-03-28T18:55:44.592505+0000","last_unstale":"2024-03-28T18:55:46.603331+0000","last_undegraded":"2024-03-28T18:55:46.603331+0000","last_fullsized":"2024-03-28T18:55:46.603331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:58.286421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605269+0000","last_change":"2024-03-28T18:55:45.611322+0000","last_active":"2024-03-28T18:55:46.605269+0000","last_peered":"2024-03-28T18:55:46.605269+0000","last_clean":"2024-03-28T18:55:46.605269+0000","last_became_active":"2024-03-28T18:55:45.610958+0000","last_became_peered":"2024-03-28T18:55:45.610958+0000","last_unstale":"2024-03-28T18:55:46.605269+0000","last_undegraded":"2024-03-28T18:55:46.605269+0000","last_fullsized":"2024-03-28T18:55:46.605269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827108+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:55:46.827108+0000","last_peered":"2024-03-28T18:55:46.827108+0000","last_clean":"2024-03-28T18:55:46.827108+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:55:46.827108+0000","last_undegraded":"2024-03-28T18:55:46.827108+0000","last_fullsized":"2024-03-28T18:55:46.827108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604724+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:55:46.604724+0000","last_peered":"2024-03-28T18:55:46.604724+0000","last_clean":"2024-03-28T18:55:46.604724+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:55:46.604724+0000","last_undegraded":"2024-03-28T18:55:46.604724+0000","last_fullsized":"2024-03-28T18:55:46.604724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827611+0000","last_change":"2024-03-28T18:55:44.608283+0000","last_active":"2024-03-28T18:55:46.827611+0000","last_peered":"2024-03-28T18:55:46.827611+0000","last_clean":"2024-03-28T18:55:46.827611+0000","last_became_active":"2024-03-28T18:55:44.607999+0000","last_became_peered":"2024-03-28T18:55:44.607999+0000","last_unstale":"2024-03-28T18:55:46.827611+0000","last_undegraded":"2024-03-28T18:55:46.827611+0000","last_fullsized":"2024-03-28T18:55:46.827611+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:22.669778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828611+0000","last_change":"2024-03-28T18:55:45.618390+0000","last_active":"2024-03-28T18:55:46.828611+0000","last_peered":"2024-03-28T18:55:46.828611+0000","last_clean":"2024-03-28T18:55:46.828611+0000","last_became_active":"2024-03-28T18:55:45.617897+0000","last_became_peered":"2024-03-28T18:55:45.617897+0000","last_unstale":"2024-03-28T18:55:46.828611+0000","last_undegraded":"2024-03-28T18:55:46.828611+0000","last_fullsized":"2024-03-28T18:55:46.828611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:16.520575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602446+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:55:46.602446+0000","last_peered":"2024-03-28T18:55:46.602446+0000","last_clean":"2024-03-28T18:55:46.602446+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:55:46.602446+0000","last_undegraded":"2024-03-28T18:55:46.602446+0000","last_fullsized":"2024-03-28T18:55:46.602446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613831+0000","last_change":"2024-03-28T18:55:45.624938+0000","last_active":"2024-03-28T18:55:46.613831+0000","last_peered":"2024-03-28T18:55:46.613831+0000","last_clean":"2024-03-28T18:55:46.613831+0000","last_became_active":"2024-03-28T18:55:45.624362+0000","last_became_peered":"2024-03-28T18:55:45.624362+0000","last_unstale":"2024-03-28T18:55:46.613831+0000","last_undegraded":"2024-03-28T18:55:46.613831+0000","last_fullsized":"2024-03-28T18:55:46.613831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602638+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:55:46.602638+0000","last_peered":"2024-03-28T18:55:46.602638+0000","last_clean":"2024-03-28T18:55:46.602638+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:55:46.602638+0000","last_undegraded":"2024-03-28T18:55:46.602638+0000","last_fullsized":"2024-03-28T18:55:46.602638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116612+0000","last_change":"2024-03-28T18:55:45.610168+0000","last_active":"2024-03-28T18:55:47.116612+0000","last_peered":"2024-03-28T18:55:47.116612+0000","last_clean":"2024-03-28T18:55:47.116612+0000","last_became_active":"2024-03-28T18:55:45.609653+0000","last_became_peered":"2024-03-28T18:55:45.609653+0000","last_unstale":"2024-03-28T18:55:47.116612+0000","last_undegraded":"2024-03-28T18:55:47.116612+0000","last_fullsized":"2024-03-28T18:55:47.116612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116681+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:55:47.116681+0000","last_peered":"2024-03-28T18:55:47.116681+0000","last_clean":"2024-03-28T18:55:47.116681+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:55:47.116681+0000","last_undegraded":"2024-03-28T18:55:47.116681+0000","last_fullsized":"2024-03-28T18:55:47.116681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613952+0000","last_change":"2024-03-28T18:55:45.613701+0000","last_active":"2024-03-28T18:55:46.613952+0000","last_peered":"2024-03-28T18:55:46.613952+0000","last_clean":"2024-03-28T18:55:46.613952+0000","last_became_active":"2024-03-28T18:55:45.613310+0000","last_became_peered":"2024-03-28T18:55:45.613310+0000","last_unstale":"2024-03-28T18:55:46.613952+0000","last_undegraded":"2024-03-28T18:55:46.613952+0000","last_fullsized":"2024-03-28T18:55:46.613952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608809+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:55:46.608809+0000","last_peered":"2024-03-28T18:55:46.608809+0000","last_clean":"2024-03-28T18:55:46.608809+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:55:46.608809+0000","last_undegraded":"2024-03-28T18:55:46.608809+0000","last_fullsized":"2024-03-28T18:55:46.608809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163732+0000","last_change":"2024-03-28T18:55:45.615119+0000","last_active":"2024-03-28T18:55:47.163732+0000","last_peered":"2024-03-28T18:55:47.163732+0000","last_clean":"2024-03-28T18:55:47.163732+0000","last_became_active":"2024-03-28T18:55:45.613841+0000","last_became_peered":"2024-03-28T18:55:45.613841+0000","last_unstale":"2024-03-28T18:55:47.163732+0000","last_undegraded":"2024-03-28T18:55:47.163732+0000","last_fullsized":"2024-03-28T18:55:47.163732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827969+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:55:46.827969+0000","last_peered":"2024-03-28T18:55:46.827969+0000","last_clean":"2024-03-28T18:55:46.827969+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:55:46.827969+0000","last_undegraded":"2024-03-28T18:55:46.827969+0000","last_fullsized":"2024-03-28T18:55:46.827969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604609+0000","last_change":"2024-03-28T18:55:45.619605+0000","last_active":"2024-03-28T18:55:46.604609+0000","last_peered":"2024-03-28T18:55:46.604609+0000","last_clean":"2024-03-28T18:55:46.604609+0000","last_became_active":"2024-03-28T18:55:45.618473+0000","last_became_peered":"2024-03-28T18:55:45.618473+0000","last_unstale":"2024-03-28T18:55:46.604609+0000","last_undegraded":"2024-03-28T18:55:46.604609+0000","last_fullsized":"2024-03-28T18:55:46.604609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614151+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:55:46.614151+0000","last_peered":"2024-03-28T18:55:46.614151+0000","last_clean":"2024-03-28T18:55:46.614151+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:55:46.614151+0000","last_undegraded":"2024-03-28T18:55:46.614151+0000","last_fullsized":"2024-03-28T18:55:46.614151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163424+0000","last_change":"2024-03-28T18:55:45.612284+0000","last_active":"2024-03-28T18:55:47.163424+0000","last_peered":"2024-03-28T18:55:47.163424+0000","last_clean":"2024-03-28T18:55:47.163424+0000","last_became_active":"2024-03-28T18:55:45.607218+0000","last_became_peered":"2024-03-28T18:55:45.607218+0000","last_unstale":"2024-03-28T18:55:47.163424+0000","last_undegraded":"2024-03-28T18:55:47.163424+0000","last_fullsized":"2024-03-28T18:55:47.163424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604943+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:55:46.604943+0000","last_peered":"2024-03-28T18:55:46.604943+0000","last_clean":"2024-03-28T18:55:46.604943+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:55:46.604943+0000","last_undegraded":"2024-03-28T18:55:46.604943+0000","last_fullsized":"2024-03-28T18:55:46.604943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603065+0000","last_change":"2024-03-28T18:55:45.607609+0000","last_active":"2024-03-28T18:55:46.603065+0000","last_peered":"2024-03-28T18:55:46.603065+0000","last_clean":"2024-03-28T18:55:46.603065+0000","last_became_active":"2024-03-28T18:55:45.602582+0000","last_became_peered":"2024-03-28T18:55:45.602582+0000","last_unstale":"2024-03-28T18:55:46.603065+0000","last_undegraded":"2024-03-28T18:55:46.603065+0000","last_fullsized":"2024-03-28T18:55:46.603065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:06:05.335715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829082+0000","last_change":"2024-03-28T18:55:34.491857+0000","last_active":"2024-03-28T18:55:46.829082+0000","last_peered":"2024-03-28T18:55:46.829082+0000","last_clean":"2024-03-28T18:55:46.829082+0000","last_became_active":"2024-03-28T18:55:34.487194+0000","last_became_peered":"2024-03-28T18:55:34.487194+0000","last_unstale":"2024-03-28T18:55:46.829082+0000","last_undegraded":"2024-03-28T18:55:46.829082+0000","last_fullsized":"2024-03-28T18:55:46.829082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828429+0000","last_change":"2024-03-28T18:55:45.607886+0000","last_active":"2024-03-28T18:55:46.828429+0000","last_peered":"2024-03-28T18:55:46.828429+0000","last_clean":"2024-03-28T18:55:46.828429+0000","last_became_active":"2024-03-28T18:55:45.605964+0000","last_became_peered":"2024-03-28T18:55:45.605964+0000","last_unstale":"2024-03-28T18:55:46.828429+0000","last_undegraded":"2024-03-28T18:55:46.828429+0000","last_fullsized":"2024-03-28T18:55:46.828429+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:45.986654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117089+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:55:47.117089+0000","last_peered":"2024-03-28T18:55:47.117089+0000","last_clean":"2024-03-28T18:55:47.117089+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:55:47.117089+0000","last_undegraded":"2024-03-28T18:55:47.117089+0000","last_fullsized":"2024-03-28T18:55:47.117089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165971+0000","last_change":"2024-03-28T18:55:45.618031+0000","last_active":"2024-03-28T18:55:47.165971+0000","last_peered":"2024-03-28T18:55:47.165971+0000","last_clean":"2024-03-28T18:55:47.165971+0000","last_became_active":"2024-03-28T18:55:45.617719+0000","last_became_peered":"2024-03-28T18:55:45.617719+0000","last_unstale":"2024-03-28T18:55:47.165971+0000","last_undegraded":"2024-03-28T18:55:47.165971+0000","last_fullsized":"2024-03-28T18:55:47.165971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826204+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:55:46.826204+0000","last_peered":"2024-03-28T18:55:46.826204+0000","last_clean":"2024-03-28T18:55:46.826204+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:55:46.826204+0000","last_undegraded":"2024-03-28T18:55:46.826204+0000","last_fullsized":"2024-03-28T18:55:46.826204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603156+0000","last_change":"2024-03-28T18:55:45.603783+0000","last_active":"2024-03-28T18:55:46.603156+0000","last_peered":"2024-03-28T18:55:46.603156+0000","last_clean":"2024-03-28T18:55:46.603156+0000","last_became_active":"2024-03-28T18:55:45.603474+0000","last_became_peered":"2024-03-28T18:55:45.603474+0000","last_unstale":"2024-03-28T18:55:46.603156+0000","last_undegraded":"2024-03-28T18:55:46.603156+0000","last_fullsized":"2024-03-28T18:55:46.603156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:42.328182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116551+0000","last_change":"2024-03-28T18:55:34.505741+0000","last_active":"2024-03-28T18:55:47.116551+0000","last_peered":"2024-03-28T18:55:47.116551+0000","last_clean":"2024-03-28T18:55:47.116551+0000","last_became_active":"2024-03-28T18:55:34.505440+0000","last_became_peered":"2024-03-28T18:55:34.505440+0000","last_unstale":"2024-03-28T18:55:47.116551+0000","last_undegraded":"2024-03-28T18:55:47.116551+0000","last_fullsized":"2024-03-28T18:55:47.116551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827452+0000","last_change":"2024-03-28T18:55:45.618912+0000","last_active":"2024-03-28T18:55:46.827452+0000","last_peered":"2024-03-28T18:55:46.827452+0000","last_clean":"2024-03-28T18:55:46.827452+0000","last_became_active":"2024-03-28T18:55:45.618554+0000","last_became_peered":"2024-03-28T18:55:45.618554+0000","last_unstale":"2024-03-28T18:55:46.827452+0000","last_undegraded":"2024-03-28T18:55:46.827452+0000","last_fullsized":"2024-03-28T18:55:46.827452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608624+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:55:46.608624+0000","last_peered":"2024-03-28T18:55:46.608624+0000","last_clean":"2024-03-28T18:55:46.608624+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:55:46.608624+0000","last_undegraded":"2024-03-28T18:55:46.608624+0000","last_fullsized":"2024-03-28T18:55:46.608624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604333+0000","last_change":"2024-03-28T18:55:45.616420+0000","last_active":"2024-03-28T18:55:46.604333+0000","last_peered":"2024-03-28T18:55:46.604333+0000","last_clean":"2024-03-28T18:55:46.604333+0000","last_became_active":"2024-03-28T18:55:45.614806+0000","last_became_peered":"2024-03-28T18:55:45.614806+0000","last_unstale":"2024-03-28T18:55:46.604333+0000","last_undegraded":"2024-03-28T18:55:46.604333+0000","last_fullsized":"2024-03-28T18:55:46.604333+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602688+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:55:46.602688+0000","last_peered":"2024-03-28T18:55:46.602688+0000","last_clean":"2024-03-28T18:55:46.602688+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:55:46.602688+0000","last_undegraded":"2024-03-28T18:55:46.602688+0000","last_fullsized":"2024-03-28T18:55:46.602688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604338+0000","last_change":"2024-03-28T18:55:45.606507+0000","last_active":"2024-03-28T18:55:46.604338+0000","last_peered":"2024-03-28T18:55:46.604338+0000","last_clean":"2024-03-28T18:55:46.604338+0000","last_became_active":"2024-03-28T18:55:45.606205+0000","last_became_peered":"2024-03-28T18:55:45.606205+0000","last_unstale":"2024-03-28T18:55:46.604338+0000","last_undegraded":"2024-03-28T18:55:46.604338+0000","last_fullsized":"2024-03-28T18:55:46.604338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164913+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:55:47.164913+0000","last_peered":"2024-03-28T18:55:47.164913+0000","last_clean":"2024-03-28T18:55:47.164913+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:55:47.164913+0000","last_undegraded":"2024-03-28T18:55:47.164913+0000","last_fullsized":"2024-03-28T18:55:47.164913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116033+0000","last_change":"2024-03-28T18:55:45.607890+0000","last_active":"2024-03-28T18:55:47.116033+0000","last_peered":"2024-03-28T18:55:47.116033+0000","last_clean":"2024-03-28T18:55:47.116033+0000","last_became_active":"2024-03-28T18:55:45.607590+0000","last_became_peered":"2024-03-28T18:55:45.607590+0000","last_unstale":"2024-03-28T18:55:47.116033+0000","last_undegraded":"2024-03-28T18:55:47.116033+0000","last_fullsized":"2024-03-28T18:55:47.116033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603209+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:55:46.603209+0000","last_peered":"2024-03-28T18:55:46.603209+0000","last_clean":"2024-03-28T18:55:46.603209+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:55:46.603209+0000","last_undegraded":"2024-03-28T18:55:46.603209+0000","last_fullsized":"2024-03-28T18:55:46.603209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608156+0000","last_change":"2024-03-28T18:55:45.608851+0000","last_active":"2024-03-28T18:55:46.608156+0000","last_peered":"2024-03-28T18:55:46.608156+0000","last_clean":"2024-03-28T18:55:46.608156+0000","last_became_active":"2024-03-28T18:55:45.608496+0000","last_became_peered":"2024-03-28T18:55:45.608496+0000","last_unstale":"2024-03-28T18:55:46.608156+0000","last_undegraded":"2024-03-28T18:55:46.608156+0000","last_fullsized":"2024-03-28T18:55:46.608156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118072+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:55:47.118072+0000","last_peered":"2024-03-28T18:55:47.118072+0000","last_clean":"2024-03-28T18:55:47.118072+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:55:47.118072+0000","last_undegraded":"2024-03-28T18:55:47.118072+0000","last_fullsized":"2024-03-28T18:55:47.118072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613556+0000","last_change":"2024-03-28T18:55:45.619501+0000","last_active":"2024-03-28T18:55:46.613556+0000","last_peered":"2024-03-28T18:55:46.613556+0000","last_clean":"2024-03-28T18:55:46.613556+0000","last_became_active":"2024-03-28T18:55:45.619212+0000","last_became_peered":"2024-03-28T18:55:45.619212+0000","last_unstale":"2024-03-28T18:55:46.613556+0000","last_undegraded":"2024-03-28T18:55:46.613556+0000","last_fullsized":"2024-03-28T18:55:46.613556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116821+0000","last_change":"2024-03-28T18:55:34.509529+0000","last_active":"2024-03-28T18:55:47.116821+0000","last_peered":"2024-03-28T18:55:47.116821+0000","last_clean":"2024-03-28T18:55:47.116821+0000","last_became_active":"2024-03-28T18:55:34.501100+0000","last_became_peered":"2024-03-28T18:55:34.501100+0000","last_unstale":"2024-03-28T18:55:47.116821+0000","last_undegraded":"2024-03-28T18:55:47.116821+0000","last_fullsized":"2024-03-28T18:55:47.116821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605306+0000","last_change":"2024-03-28T18:55:45.615225+0000","last_active":"2024-03-28T18:55:46.605306+0000","last_peered":"2024-03-28T18:55:46.605306+0000","last_clean":"2024-03-28T18:55:46.605306+0000","last_became_active":"2024-03-28T18:55:45.613602+0000","last_became_peered":"2024-03-28T18:55:45.613602+0000","last_unstale":"2024-03-28T18:55:46.605306+0000","last_undegraded":"2024-03-28T18:55:46.605306+0000","last_fullsized":"2024-03-28T18:55:46.605306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612997+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:55:46.612997+0000","last_peered":"2024-03-28T18:55:46.612997+0000","last_clean":"2024-03-28T18:55:46.612997+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:55:46.612997+0000","last_undegraded":"2024-03-28T18:55:46.612997+0000","last_fullsized":"2024-03-28T18:55:46.612997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116582+0000","last_change":"2024-03-28T18:55:45.609471+0000","last_active":"2024-03-28T18:55:47.116582+0000","last_peered":"2024-03-28T18:55:47.116582+0000","last_clean":"2024-03-28T18:55:47.116582+0000","last_became_active":"2024-03-28T18:55:45.609132+0000","last_became_peered":"2024-03-28T18:55:45.609132+0000","last_unstale":"2024-03-28T18:55:47.116582+0000","last_undegraded":"2024-03-28T18:55:47.116582+0000","last_fullsized":"2024-03-28T18:55:47.116582+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608272+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:55:46.608272+0000","last_peered":"2024-03-28T18:55:46.608272+0000","last_clean":"2024-03-28T18:55:46.608272+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:55:46.608272+0000","last_undegraded":"2024-03-28T18:55:46.608272+0000","last_fullsized":"2024-03-28T18:55:46.608272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164529+0000","last_change":"2024-03-28T18:55:45.617501+0000","last_active":"2024-03-28T18:55:47.164529+0000","last_peered":"2024-03-28T18:55:47.164529+0000","last_clean":"2024-03-28T18:55:47.164529+0000","last_became_active":"2024-03-28T18:55:45.617180+0000","last_became_peered":"2024-03-28T18:55:45.617180+0000","last_unstale":"2024-03-28T18:55:47.164529+0000","last_undegraded":"2024-03-28T18:55:47.164529+0000","last_fullsized":"2024-03-28T18:55:47.164529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826964+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:55:46.826964+0000","last_peered":"2024-03-28T18:55:46.826964+0000","last_clean":"2024-03-28T18:55:46.826964+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:55:46.826964+0000","last_undegraded":"2024-03-28T18:55:46.826964+0000","last_fullsized":"2024-03-28T18:55:46.826964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826286+0000","last_change":"2024-03-28T18:55:45.609169+0000","last_active":"2024-03-28T18:55:46.826286+0000","last_peered":"2024-03-28T18:55:46.826286+0000","last_clean":"2024-03-28T18:55:46.826286+0000","last_became_active":"2024-03-28T18:55:45.601948+0000","last_became_peered":"2024-03-28T18:55:45.601948+0000","last_unstale":"2024-03-28T18:55:46.826286+0000","last_undegraded":"2024-03-28T18:55:46.826286+0000","last_fullsized":"2024-03-28T18:55:46.826286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605787+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:55:46.605787+0000","last_peered":"2024-03-28T18:55:46.605787+0000","last_clean":"2024-03-28T18:55:46.605787+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:55:46.605787+0000","last_undegraded":"2024-03-28T18:55:46.605787+0000","last_fullsized":"2024-03-28T18:55:46.605787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603014+0000","last_change":"2024-03-28T18:55:45.616238+0000","last_active":"2024-03-28T18:55:46.603014+0000","last_peered":"2024-03-28T18:55:46.603014+0000","last_clean":"2024-03-28T18:55:46.603014+0000","last_became_active":"2024-03-28T18:55:45.615371+0000","last_became_peered":"2024-03-28T18:55:45.615371+0000","last_unstale":"2024-03-28T18:55:46.603014+0000","last_undegraded":"2024-03-28T18:55:46.603014+0000","last_fullsized":"2024-03-28T18:55:46.603014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:14.661134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603091+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:55:46.603091+0000","last_peered":"2024-03-28T18:55:46.603091+0000","last_clean":"2024-03-28T18:55:46.603091+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:55:46.603091+0000","last_undegraded":"2024-03-28T18:55:46.603091+0000","last_fullsized":"2024-03-28T18:55:46.603091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164732+0000","last_change":"2024-03-28T18:55:45.613289+0000","last_active":"2024-03-28T18:55:47.164732+0000","last_peered":"2024-03-28T18:55:47.164732+0000","last_clean":"2024-03-28T18:55:47.164732+0000","last_became_active":"2024-03-28T18:55:45.610917+0000","last_became_peered":"2024-03-28T18:55:45.610917+0000","last_unstale":"2024-03-28T18:55:47.164732+0000","last_undegraded":"2024-03-28T18:55:47.164732+0000","last_fullsized":"2024-03-28T18:55:47.164732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604796+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:55:46.604796+0000","last_peered":"2024-03-28T18:55:46.604796+0000","last_clean":"2024-03-28T18:55:46.604796+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:55:46.604796+0000","last_undegraded":"2024-03-28T18:55:46.604796+0000","last_fullsized":"2024-03-28T18:55:46.604796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116103+0000","last_change":"2024-03-28T18:55:45.621789+0000","last_active":"2024-03-28T18:55:47.116103+0000","last_peered":"2024-03-28T18:55:47.116103+0000","last_clean":"2024-03-28T18:55:47.116103+0000","last_became_active":"2024-03-28T18:55:45.621163+0000","last_became_peered":"2024-03-28T18:55:45.621163+0000","last_unstale":"2024-03-28T18:55:47.116103+0000","last_undegraded":"2024-03-28T18:55:47.116103+0000","last_fullsized":"2024-03-28T18:55:47.116103+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166307+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:55:47.166307+0000","last_peered":"2024-03-28T18:55:47.166307+0000","last_clean":"2024-03-28T18:55:47.166307+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:55:47.166307+0000","last_undegraded":"2024-03-28T18:55:47.166307+0000","last_fullsized":"2024-03-28T18:55:47.166307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828245+0000","last_change":"2024-03-28T18:55:45.603774+0000","last_active":"2024-03-28T18:55:46.828245+0000","last_peered":"2024-03-28T18:55:46.828245+0000","last_clean":"2024-03-28T18:55:46.828245+0000","last_became_active":"2024-03-28T18:55:45.603451+0000","last_became_peered":"2024-03-28T18:55:45.603451+0000","last_unstale":"2024-03-28T18:55:46.828245+0000","last_undegraded":"2024-03-28T18:55:46.828245+0000","last_fullsized":"2024-03-28T18:55:46.828245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826131+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:55:46.826131+0000","last_peered":"2024-03-28T18:55:46.826131+0000","last_clean":"2024-03-28T18:55:46.826131+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:55:46.826131+0000","last_undegraded":"2024-03-28T18:55:46.826131+0000","last_fullsized":"2024-03-28T18:55:46.826131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603950+0000","last_change":"2024-03-28T18:55:45.616755+0000","last_active":"2024-03-28T18:55:46.603950+0000","last_peered":"2024-03-28T18:55:46.603950+0000","last_clean":"2024-03-28T18:55:46.603950+0000","last_became_active":"2024-03-28T18:55:45.614611+0000","last_became_peered":"2024-03-28T18:55:45.614611+0000","last_unstale":"2024-03-28T18:55:46.603950+0000","last_undegraded":"2024-03-28T18:55:46.603950+0000","last_fullsized":"2024-03-28T18:55:46.603950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:06.327795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163860+0000","last_change":"2024-03-28T18:55:34.491855+0000","last_active":"2024-03-28T18:55:47.163860+0000","last_peered":"2024-03-28T18:55:47.163860+0000","last_clean":"2024-03-28T18:55:47.163860+0000","last_became_active":"2024-03-28T18:55:34.491070+0000","last_became_peered":"2024-03-28T18:55:34.491070+0000","last_unstale":"2024-03-28T18:55:47.163860+0000","last_undegraded":"2024-03-28T18:55:47.163860+0000","last_fullsized":"2024-03-28T18:55:47.163860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613597+0000","last_change":"2024-03-28T18:55:45.623195+0000","last_active":"2024-03-28T18:55:46.613597+0000","last_peered":"2024-03-28T18:55:46.613597+0000","last_clean":"2024-03-28T18:55:46.613597+0000","last_became_active":"2024-03-28T18:55:45.622508+0000","last_became_peered":"2024-03-28T18:55:45.622508+0000","last_unstale":"2024-03-28T18:55:46.613597+0000","last_undegraded":"2024-03-28T18:55:46.613597+0000","last_fullsized":"2024-03-28T18:55:46.613597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116530+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:55:47.116530+0000","last_peered":"2024-03-28T18:55:47.116530+0000","last_clean":"2024-03-28T18:55:47.116530+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:55:47.116530+0000","last_undegraded":"2024-03-28T18:55:47.116530+0000","last_fullsized":"2024-03-28T18:55:47.116530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117584+0000","last_change":"2024-03-28T18:55:45.618841+0000","last_active":"2024-03-28T18:55:47.117584+0000","last_peered":"2024-03-28T18:55:47.117584+0000","last_clean":"2024-03-28T18:55:47.117584+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:55:47.117584+0000","last_undegraded":"2024-03-28T18:55:47.117584+0000","last_fullsized":"2024-03-28T18:55:47.117584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164332+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:55:47.164332+0000","last_peered":"2024-03-28T18:55:47.164332+0000","last_clean":"2024-03-28T18:55:47.164332+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:55:47.164332+0000","last_undegraded":"2024-03-28T18:55:47.164332+0000","last_fullsized":"2024-03-28T18:55:47.164332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825990+0000","last_change":"2024-03-28T18:55:45.602886+0000","last_active":"2024-03-28T18:55:46.825990+0000","last_peered":"2024-03-28T18:55:46.825990+0000","last_clean":"2024-03-28T18:55:46.825990+0000","last_became_active":"2024-03-28T18:55:45.602610+0000","last_became_peered":"2024-03-28T18:55:45.602610+0000","last_unstale":"2024-03-28T18:55:46.825990+0000","last_undegraded":"2024-03-28T18:55:46.825990+0000","last_fullsized":"2024-03-28T18:55:46.825990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116836+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:55:47.116836+0000","last_peered":"2024-03-28T18:55:47.116836+0000","last_clean":"2024-03-28T18:55:47.116836+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:55:47.116836+0000","last_undegraded":"2024-03-28T18:55:47.116836+0000","last_fullsized":"2024-03-28T18:55:47.116836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602814+0000","last_change":"2024-03-28T18:55:45.608673+0000","last_active":"2024-03-28T18:55:46.602814+0000","last_peered":"2024-03-28T18:55:46.602814+0000","last_clean":"2024-03-28T18:55:46.602814+0000","last_became_active":"2024-03-28T18:55:45.608376+0000","last_became_peered":"2024-03-28T18:55:45.608376+0000","last_unstale":"2024-03-28T18:55:46.602814+0000","last_undegraded":"2024-03-28T18:55:46.602814+0000","last_fullsized":"2024-03-28T18:55:46.602814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:31.781549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608764+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:55:46.608764+0000","last_peered":"2024-03-28T18:55:46.608764+0000","last_clean":"2024-03-28T18:55:46.608764+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:55:46.608764+0000","last_undegraded":"2024-03-28T18:55:46.608764+0000","last_fullsized":"2024-03-28T18:55:46.608764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163497+0000","last_change":"2024-03-28T18:55:45.609565+0000","last_active":"2024-03-28T18:55:47.163497+0000","last_peered":"2024-03-28T18:55:47.163497+0000","last_clean":"2024-03-28T18:55:47.163497+0000","last_became_active":"2024-03-28T18:55:45.609082+0000","last_became_peered":"2024-03-28T18:55:45.609082+0000","last_unstale":"2024-03-28T18:55:47.163497+0000","last_undegraded":"2024-03-28T18:55:47.163497+0000","last_fullsized":"2024-03-28T18:55:47.163497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613540+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:55:46.613540+0000","last_peered":"2024-03-28T18:55:46.613540+0000","last_clean":"2024-03-28T18:55:46.613540+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:55:46.613540+0000","last_undegraded":"2024-03-28T18:55:46.613540+0000","last_fullsized":"2024-03-28T18:55:46.613540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163662+0000","last_change":"2024-03-28T18:55:45.618585+0000","last_active":"2024-03-28T18:55:47.163662+0000","last_peered":"2024-03-28T18:55:47.163662+0000","last_clean":"2024-03-28T18:55:47.163662+0000","last_became_active":"2024-03-28T18:55:45.617895+0000","last_became_peered":"2024-03-28T18:55:45.617895+0000","last_unstale":"2024-03-28T18:55:47.163662+0000","last_undegraded":"2024-03-28T18:55:47.163662+0000","last_fullsized":"2024-03-28T18:55:47.163662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:54.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166181+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:55:47.166181+0000","last_peered":"2024-03-28T18:55:47.166181+0000","last_clean":"2024-03-28T18:55:47.166181+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:55:47.166181+0000","last_undegraded":"2024-03-28T18:55:47.166181+0000","last_fullsized":"2024-03-28T18:55:47.166181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":856064,"data_stored":690301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":101489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":99491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":102507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":101055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:55:57.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:55:57.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.390+0000 7f908a800640 1 Processor -- start 2024-03-28T18:55:57.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.390+0000 7f908a800640 1 -- start start 2024-03-28T18:55:57.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f908a800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f90840c58c0 0x7f90840c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f908a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90840be5d0 0x7f9084155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f908a800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084156230 con 0x7f90840c8ed0 2024-03-28T18:55:57.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f908a800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9084158bd0 con 0x7f90840be5d0 2024-03-28T18:55:57.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f908a800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9084158eb0 con 0x7f90840c58c0 2024-03-28T18:55:57.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f9089e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90840be5d0 0x7f9084155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f9089e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90840be5d0 0x7f9084155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46812/0 (socket says 172.21.15.27:46812) 2024-03-28T18:55:57.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f9089e00640 1 -- 172.21.15.27:0/2513036531 learned_addr learned my addr 172.21.15.27:0/2513036531 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f9089400640 1 --2- 172.21.15.27:0/2513036531 >> v2:172.21.15.155:3300/0 conn(0x7f90840c58c0 0x7f90840c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.392+0000 7f9089e00640 1 -- 172.21.15.27:0/2513036531 >> v2:172.21.15.155:3300/0 conn(0x7f90840c58c0 msgr2=0x7f90840c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.393+0000 7f9089e00640 1 --2- 172.21.15.27:0/2513036531 >> v2:172.21.15.155:3300/0 conn(0x7f90840c58c0 0x7f90840c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.393+0000 7f9089e00640 1 -- 172.21.15.27:0/2513036531 >> v1:172.21.15.155:6789/0 conn(0x7f90840c8ed0 legacy=0x7f90841555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.393+0000 7f9089e00640 1 -- 172.21.15.27:0/2513036531 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084159190 con 0x7f90840be5d0 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.393+0000 7f9089400640 1 --2- 172.21.15.27:0/2513036531 >> v2:172.21.15.155:3300/0 conn(0x7f90840c58c0 0x7f90840c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.394+0000 7f9089e00640 1 --2- 172.21.15.27:0/2513036531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840be5d0 0x7f9084155cf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f908406c060 tx=0x7f907c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b96c1ba1ecfeef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.394+0000 7f9083e00640 1 -- 172.21.15.27:0/2513036531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c02c960 con 0x7f90840be5d0 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.394+0000 7f9083e00640 1 -- 172.21.15.27:0/2513036531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c02cb00 con 0x7f90840be5d0 2024-03-28T18:55:57.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.394+0000 7f9083e00640 1 -- 172.21.15.27:0/2513036531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c0343f0 con 0x7f90840be5d0 2024-03-28T18:55:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.394+0000 7f908a800640 1 -- 172.21.15.27:0/2513036531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840be5d0 msgr2=0x7f9084155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.394+0000 7f908a800640 1 --2- 172.21.15.27:0/2513036531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840be5d0 0x7f9084155cf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f908406c060 tx=0x7f907c02bdc0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.395+0000 7f908a800640 1 -- 172.21.15.27:0/2513036531 shutdown_connections 2024-03-28T18:55:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.395+0000 7f908a800640 1 --2- 172.21.15.27:0/2513036531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840be5d0 0x7f9084155cf0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.395+0000 7f908a800640 1 --2- 172.21.15.27:0/2513036531 >> v2:172.21.15.155:3300/0 conn(0x7f90840c58c0 0x7f90840c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.395+0000 7f908a800640 1 -- 172.21.15.27:0/2513036531 >> 172.21.15.27:0/2513036531 conn(0x7f908406b6f0 msgr2=0x7f90840acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.395+0000 7f908a800640 1 -- 172.21.15.27:0/2513036531 shutdown_connections 2024-03-28T18:55:57.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.395+0000 7f908a800640 1 -- 172.21.15.27:0/2513036531 wait complete. 2024-03-28T18:55:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.397+0000 7f908a800640 1 Processor -- start 2024-03-28T18:55:57.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.397+0000 7f908a800640 1 -- start start 2024-03-28T18:55:57.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f908a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90840be5d0 0x7f90840aaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f908a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840c58c0 0x7f90840c2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f908a800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90840c8ed0 0x7f90840c2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f908a800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90840b0820 con 0x7f90840c58c0 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f908a800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9084159750 con 0x7f90840be5d0 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f908a800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90840b09a0 con 0x7f90840c8ed0 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f9088a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840c58c0 0x7f90840c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f9088a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840c58c0 0x7f90840c2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46816/0 (socket says 172.21.15.27:46816) 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.398+0000 7f9088a00640 1 -- 172.21.15.27:0/2131903049 learned_addr learned my addr 172.21.15.27:0/2131903049 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9089e00640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90840c8ed0 0x7f90840c2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9088a00640 1 -- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90840c8ed0 msgr2=0x7f90840c2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9089400640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90840be5d0 0x7f90840aaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9088a00640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90840c8ed0 0x7f90840c2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9088a00640 1 -- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90840be5d0 msgr2=0x7f90840aaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9088a00640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90840be5d0 0x7f90840aaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9088a00640 1 -- 172.21.15.27:0/2131903049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90841ee8f0 con 0x7f90840c58c0 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9089e00640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90840c8ed0 0x7f90840c2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9089400640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90840be5d0 0x7f90840aaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9088a00640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840c58c0 0x7f90840c2010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f907000b3c0 tx=0x7f907000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9081600640 1 -- 172.21.15.27:0/2131903049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070011830 con 0x7f90840c58c0 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9081600640 1 -- 172.21.15.27:0/2131903049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9070016070 con 0x7f90840c58c0 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f9081600640 1 -- 172.21.15.27:0/2131903049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070011c50 con 0x7f90840c58c0 2024-03-28T18:55:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90841eebd0 con 0x7f90840c58c0 2024-03-28T18:55:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.399+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90841ef190 con 0x7f90840c58c0 2024-03-28T18:55:57.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.401+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90841ef390 con 0x7f90840c58c0 2024-03-28T18:55:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.402+0000 7f9081600640 1 -- 172.21.15.27:0/2131903049 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f907001b460 con 0x7f90840c58c0 2024-03-28T18:55:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.402+0000 7f9081600640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f906406bc00 0x7f906406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.403+0000 7f9081600640 1 -- 172.21.15.27:0/2131903049 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f907008a890 con 0x7f90840c58c0 2024-03-28T18:55:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.403+0000 7f9089400640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f906406bc00 0x7f906406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.403+0000 7f9089400640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f906406bc00 0x7f906406e0c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9078006fd0 tx=0x7f9078008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:57.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.407+0000 7f9081600640 1 -- 172.21.15.27:0/2131903049 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f90841ef390 con 0x7f90840c58c0 2024-03-28T18:55:57.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.606+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9084069640 con 0x7f906406bc00 2024-03-28T18:55:57.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.631+0000 7f9081600640 1 -- 172.21.15.27:0/2131903049 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490306 (secure 0 0 0) 0x7f9084069640 con 0x7f906406bc00 2024-03-28T18:55:57.640 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:57.644 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f906406bc00 msgr2=0x7f906406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f906406bc00 0x7f906406e0c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9078006fd0 tx=0x7f9078008040 comp rx=0 tx=0).stop 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840c58c0 msgr2=0x7f90840c2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840c58c0 0x7f90840c2010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f907000b3c0 tx=0x7f907000b880 comp rx=0 tx=0).stop 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 shutdown_connections 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f906406bc00 0x7f906406e0c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90840c8ed0 0x7f90840c2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90840c58c0 0x7f90840c2010 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 --2- 172.21.15.27:0/2131903049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90840be5d0 0x7f90840aaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 >> 172.21.15.27:0/2131903049 conn(0x7f908406b6f0 msgr2=0x7f9084072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:57.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 shutdown_connections 2024-03-28T18:55:57.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.641+0000 7f908a800640 1 -- 172.21.15.27:0/2131903049 wait complete. 2024-03-28T18:55:57.658 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-28T18:55:56.899138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219448,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755272,"statfs":{"total":773094113280,"available":772869398528,"internally_reserved":0,"allocated":3350528,"data_stored":1984464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.300430"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603978+0000","last_change":"2024-03-28T18:55:45.603782+0000","last_active":"2024-03-28T18:55:46.603978+0000","last_peered":"2024-03-28T18:55:46.603978+0000","last_clean":"2024-03-28T18:55:46.603978+0000","last_became_active":"2024-03-28T18:55:45.603420+0000","last_became_peered":"2024-03-28T18:55:45.603420+0000","last_unstale":"2024-03-28T18:55:46.603978+0000","last_undegraded":"2024-03-28T18:55:46.603978+0000","last_fullsized":"2024-03-28T18:55:46.603978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609332+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:55:46.609332+0000","last_peered":"2024-03-28T18:55:46.609332+0000","last_clean":"2024-03-28T18:55:46.609332+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:55:46.609332+0000","last_undegraded":"2024-03-28T18:55:46.609332+0000","last_fullsized":"2024-03-28T18:55:46.609332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828788+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:55:46.828788+0000","last_peered":"2024-03-28T18:55:46.828788+0000","last_clean":"2024-03-28T18:55:46.828788+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:55:46.828788+0000","last_undegraded":"2024-03-28T18:55:46.828788+0000","last_fullsized":"2024-03-28T18:55:46.828788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825968+0000","last_change":"2024-03-28T18:55:44.602536+0000","last_active":"2024-03-28T18:55:46.825968+0000","last_peered":"2024-03-28T18:55:46.825968+0000","last_clean":"2024-03-28T18:55:46.825968+0000","last_became_active":"2024-03-28T18:55:44.602256+0000","last_became_peered":"2024-03-28T18:55:44.602256+0000","last_unstale":"2024-03-28T18:55:46.825968+0000","last_undegraded":"2024-03-28T18:55:46.825968+0000","last_fullsized":"2024-03-28T18:55:46.825968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612952+0000","last_change":"2024-03-28T18:55:45.625338+0000","last_active":"2024-03-28T18:55:46.612952+0000","last_peered":"2024-03-28T18:55:46.612952+0000","last_clean":"2024-03-28T18:55:46.612952+0000","last_became_active":"2024-03-28T18:55:45.625067+0000","last_became_peered":"2024-03-28T18:55:45.625067+0000","last_unstale":"2024-03-28T18:55:46.612952+0000","last_undegraded":"2024-03-28T18:55:46.612952+0000","last_fullsized":"2024-03-28T18:55:46.612952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829113+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:55:46.829113+0000","last_peered":"2024-03-28T18:55:46.829113+0000","last_clean":"2024-03-28T18:55:46.829113+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:55:46.829113+0000","last_undegraded":"2024-03-28T18:55:46.829113+0000","last_fullsized":"2024-03-28T18:55:46.829113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608925+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:55:46.608925+0000","last_peered":"2024-03-28T18:55:46.608925+0000","last_clean":"2024-03-28T18:55:46.608925+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:55:46.608925+0000","last_undegraded":"2024-03-28T18:55:46.608925+0000","last_fullsized":"2024-03-28T18:55:46.608925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604171+0000","last_change":"2024-03-28T18:55:44.598423+0000","last_active":"2024-03-28T18:55:46.604171+0000","last_peered":"2024-03-28T18:55:46.604171+0000","last_clean":"2024-03-28T18:55:46.604171+0000","last_became_active":"2024-03-28T18:55:44.598092+0000","last_became_peered":"2024-03-28T18:55:44.598092+0000","last_unstale":"2024-03-28T18:55:46.604171+0000","last_undegraded":"2024-03-28T18:55:46.604171+0000","last_fullsized":"2024-03-28T18:55:46.604171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.646619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163641+0000","last_change":"2024-03-28T18:55:45.608201+0000","last_active":"2024-03-28T18:55:47.163641+0000","last_peered":"2024-03-28T18:55:47.163641+0000","last_clean":"2024-03-28T18:55:47.163641+0000","last_became_active":"2024-03-28T18:55:45.607912+0000","last_became_peered":"2024-03-28T18:55:45.607912+0000","last_unstale":"2024-03-28T18:55:47.163641+0000","last_undegraded":"2024-03-28T18:55:47.163641+0000","last_fullsized":"2024-03-28T18:55:47.163641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164845+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:55:47.164845+0000","last_peered":"2024-03-28T18:55:47.164845+0000","last_clean":"2024-03-28T18:55:47.164845+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:55:47.164845+0000","last_undegraded":"2024-03-28T18:55:47.164845+0000","last_fullsized":"2024-03-28T18:55:47.164845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117090+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:55:47.117090+0000","last_peered":"2024-03-28T18:55:47.117090+0000","last_clean":"2024-03-28T18:55:47.117090+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:55:47.117090+0000","last_undegraded":"2024-03-28T18:55:47.117090+0000","last_fullsized":"2024-03-28T18:55:47.117090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604780+0000","last_change":"2024-03-28T18:55:44.604673+0000","last_active":"2024-03-28T18:55:46.604780+0000","last_peered":"2024-03-28T18:55:46.604780+0000","last_clean":"2024-03-28T18:55:46.604780+0000","last_became_active":"2024-03-28T18:55:44.603926+0000","last_became_peered":"2024-03-28T18:55:44.603926+0000","last_unstale":"2024-03-28T18:55:46.604780+0000","last_undegraded":"2024-03-28T18:55:46.604780+0000","last_fullsized":"2024-03-28T18:55:46.604780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116402+0000","last_change":"2024-03-28T18:55:45.610033+0000","last_active":"2024-03-28T18:55:47.116402+0000","last_peered":"2024-03-28T18:55:47.116402+0000","last_clean":"2024-03-28T18:55:47.116402+0000","last_became_active":"2024-03-28T18:55:45.609706+0000","last_became_peered":"2024-03-28T18:55:45.609706+0000","last_unstale":"2024-03-28T18:55:47.116402+0000","last_undegraded":"2024-03-28T18:55:47.116402+0000","last_fullsized":"2024-03-28T18:55:47.116402+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164190+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:55:47.164190+0000","last_peered":"2024-03-28T18:55:47.164190+0000","last_clean":"2024-03-28T18:55:47.164190+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:55:47.164190+0000","last_undegraded":"2024-03-28T18:55:47.164190+0000","last_fullsized":"2024-03-28T18:55:47.164190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117556+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:55:47.117556+0000","last_peered":"2024-03-28T18:55:47.117556+0000","last_clean":"2024-03-28T18:55:47.117556+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:55:47.117556+0000","last_undegraded":"2024-03-28T18:55:47.117556+0000","last_fullsized":"2024-03-28T18:55:47.117556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604505+0000","last_change":"2024-03-28T18:55:44.606715+0000","last_active":"2024-03-28T18:55:46.604505+0000","last_peered":"2024-03-28T18:55:46.604505+0000","last_clean":"2024-03-28T18:55:46.604505+0000","last_became_active":"2024-03-28T18:55:44.606343+0000","last_became_peered":"2024-03-28T18:55:44.606343+0000","last_unstale":"2024-03-28T18:55:46.604505+0000","last_undegraded":"2024-03-28T18:55:46.604505+0000","last_fullsized":"2024-03-28T18:55:46.604505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:51.471450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613822+0000","last_change":"2024-03-28T18:55:45.623483+0000","last_active":"2024-03-28T18:55:46.613822+0000","last_peered":"2024-03-28T18:55:46.613822+0000","last_clean":"2024-03-28T18:55:46.613822+0000","last_became_active":"2024-03-28T18:55:45.622894+0000","last_became_peered":"2024-03-28T18:55:45.622894+0000","last_unstale":"2024-03-28T18:55:46.613822+0000","last_undegraded":"2024-03-28T18:55:46.613822+0000","last_fullsized":"2024-03-28T18:55:46.613822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603964+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:55:46.603964+0000","last_peered":"2024-03-28T18:55:46.603964+0000","last_clean":"2024-03-28T18:55:46.603964+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:55:46.603964+0000","last_undegraded":"2024-03-28T18:55:46.603964+0000","last_fullsized":"2024-03-28T18:55:46.603964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164866+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:55:47.164866+0000","last_peered":"2024-03-28T18:55:47.164866+0000","last_clean":"2024-03-28T18:55:47.164866+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:55:47.164866+0000","last_undegraded":"2024-03-28T18:55:47.164866+0000","last_fullsized":"2024-03-28T18:55:47.164866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605818+0000","last_change":"2024-03-28T18:55:44.601415+0000","last_active":"2024-03-28T18:55:46.605818+0000","last_peered":"2024-03-28T18:55:46.605818+0000","last_clean":"2024-03-28T18:55:46.605818+0000","last_became_active":"2024-03-28T18:55:44.601124+0000","last_became_peered":"2024-03-28T18:55:44.601124+0000","last_unstale":"2024-03-28T18:55:46.605818+0000","last_undegraded":"2024-03-28T18:55:46.605818+0000","last_fullsized":"2024-03-28T18:55:46.605818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:30.767034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603094+0000","last_change":"2024-03-28T18:55:45.604795+0000","last_active":"2024-03-28T18:55:46.603094+0000","last_peered":"2024-03-28T18:55:46.603094+0000","last_clean":"2024-03-28T18:55:46.603094+0000","last_became_active":"2024-03-28T18:55:45.604432+0000","last_became_peered":"2024-03-28T18:55:45.604432+0000","last_unstale":"2024-03-28T18:55:46.603094+0000","last_undegraded":"2024-03-28T18:55:46.603094+0000","last_fullsized":"2024-03-28T18:55:46.603094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:37.638804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826963+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:55:46.826963+0000","last_peered":"2024-03-28T18:55:46.826963+0000","last_clean":"2024-03-28T18:55:46.826963+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:55:46.826963+0000","last_undegraded":"2024-03-28T18:55:46.826963+0000","last_fullsized":"2024-03-28T18:55:46.826963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613559+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:55:46.613559+0000","last_peered":"2024-03-28T18:55:46.613559+0000","last_clean":"2024-03-28T18:55:46.613559+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:55:46.613559+0000","last_undegraded":"2024-03-28T18:55:46.613559+0000","last_fullsized":"2024-03-28T18:55:46.613559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605718+0000","last_change":"2024-03-28T18:55:44.603859+0000","last_active":"2024-03-28T18:55:46.605718+0000","last_peered":"2024-03-28T18:55:46.605718+0000","last_clean":"2024-03-28T18:55:46.605718+0000","last_became_active":"2024-03-28T18:55:44.603540+0000","last_became_peered":"2024-03-28T18:55:44.603540+0000","last_unstale":"2024-03-28T18:55:46.605718+0000","last_undegraded":"2024-03-28T18:55:46.605718+0000","last_fullsized":"2024-03-28T18:55:46.605718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.606069+0000","last_change":"2024-03-28T18:55:45.608142+0000","last_active":"2024-03-28T18:55:46.606069+0000","last_peered":"2024-03-28T18:55:46.606069+0000","last_clean":"2024-03-28T18:55:46.606069+0000","last_became_active":"2024-03-28T18:55:45.606117+0000","last_became_peered":"2024-03-28T18:55:45.606117+0000","last_unstale":"2024-03-28T18:55:46.606069+0000","last_undegraded":"2024-03-28T18:55:46.606069+0000","last_fullsized":"2024-03-28T18:55:46.606069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826653+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:55:46.826653+0000","last_peered":"2024-03-28T18:55:46.826653+0000","last_clean":"2024-03-28T18:55:46.826653+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:55:46.826653+0000","last_undegraded":"2024-03-28T18:55:46.826653+0000","last_fullsized":"2024-03-28T18:55:46.826653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613045+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:55:46.613045+0000","last_peered":"2024-03-28T18:55:46.613045+0000","last_clean":"2024-03-28T18:55:46.613045+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:55:46.613045+0000","last_undegraded":"2024-03-28T18:55:46.613045+0000","last_fullsized":"2024-03-28T18:55:46.613045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614066+0000","last_change":"2024-03-28T18:55:44.606690+0000","last_active":"2024-03-28T18:55:46.614066+0000","last_peered":"2024-03-28T18:55:46.614066+0000","last_clean":"2024-03-28T18:55:46.614066+0000","last_became_active":"2024-03-28T18:55:44.606421+0000","last_became_peered":"2024-03-28T18:55:44.606421+0000","last_unstale":"2024-03-28T18:55:46.614066+0000","last_undegraded":"2024-03-28T18:55:46.614066+0000","last_fullsized":"2024-03-28T18:55:46.614066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117505+0000","last_change":"2024-03-28T18:55:45.610580+0000","last_active":"2024-03-28T18:55:47.117505+0000","last_peered":"2024-03-28T18:55:47.117505+0000","last_clean":"2024-03-28T18:55:47.117505+0000","last_became_active":"2024-03-28T18:55:45.610316+0000","last_became_peered":"2024-03-28T18:55:45.610316+0000","last_unstale":"2024-03-28T18:55:47.117505+0000","last_undegraded":"2024-03-28T18:55:47.117505+0000","last_fullsized":"2024-03-28T18:55:47.117505+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604997+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:55:46.604997+0000","last_peered":"2024-03-28T18:55:46.604997+0000","last_clean":"2024-03-28T18:55:46.604997+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:55:46.604997+0000","last_undegraded":"2024-03-28T18:55:46.604997+0000","last_fullsized":"2024-03-28T18:55:46.604997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163833+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:55:47.163833+0000","last_peered":"2024-03-28T18:55:47.163833+0000","last_clean":"2024-03-28T18:55:47.163833+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:55:47.163833+0000","last_undegraded":"2024-03-28T18:55:47.163833+0000","last_fullsized":"2024-03-28T18:55:47.163833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.606207+0000","last_change":"2024-03-28T18:55:44.600523+0000","last_active":"2024-03-28T18:55:46.606207+0000","last_peered":"2024-03-28T18:55:46.606207+0000","last_clean":"2024-03-28T18:55:46.606207+0000","last_became_active":"2024-03-28T18:55:44.597550+0000","last_became_peered":"2024-03-28T18:55:44.597550+0000","last_unstale":"2024-03-28T18:55:46.606207+0000","last_undegraded":"2024-03-28T18:55:46.606207+0000","last_fullsized":"2024-03-28T18:55:46.606207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825927+0000","last_change":"2024-03-28T18:55:45.619798+0000","last_active":"2024-03-28T18:55:46.825927+0000","last_peered":"2024-03-28T18:55:46.825927+0000","last_clean":"2024-03-28T18:55:46.825927+0000","last_became_active":"2024-03-28T18:55:45.616517+0000","last_became_peered":"2024-03-28T18:55:45.616517+0000","last_unstale":"2024-03-28T18:55:46.825927+0000","last_undegraded":"2024-03-28T18:55:46.825927+0000","last_fullsized":"2024-03-28T18:55:46.825927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609158+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:55:46.609158+0000","last_peered":"2024-03-28T18:55:46.609158+0000","last_clean":"2024-03-28T18:55:46.609158+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:55:46.609158+0000","last_undegraded":"2024-03-28T18:55:46.609158+0000","last_fullsized":"2024-03-28T18:55:46.609158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605113+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:55:46.605113+0000","last_peered":"2024-03-28T18:55:46.605113+0000","last_clean":"2024-03-28T18:55:46.605113+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:55:46.605113+0000","last_undegraded":"2024-03-28T18:55:46.605113+0000","last_fullsized":"2024-03-28T18:55:46.605113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604527+0000","last_change":"2024-03-28T18:55:44.606498+0000","last_active":"2024-03-28T18:55:46.604527+0000","last_peered":"2024-03-28T18:55:46.604527+0000","last_clean":"2024-03-28T18:55:46.604527+0000","last_became_active":"2024-03-28T18:55:44.606098+0000","last_became_peered":"2024-03-28T18:55:44.606098+0000","last_unstale":"2024-03-28T18:55:46.604527+0000","last_undegraded":"2024-03-28T18:55:46.604527+0000","last_fullsized":"2024-03-28T18:55:46.604527+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605931+0000","last_change":"2024-03-28T18:55:45.617483+0000","last_active":"2024-03-28T18:55:46.605931+0000","last_peered":"2024-03-28T18:55:46.605931+0000","last_clean":"2024-03-28T18:55:46.605931+0000","last_became_active":"2024-03-28T18:55:45.617169+0000","last_became_peered":"2024-03-28T18:55:45.617169+0000","last_unstale":"2024-03-28T18:55:46.605931+0000","last_undegraded":"2024-03-28T18:55:46.605931+0000","last_fullsized":"2024-03-28T18:55:46.605931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829227+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:55:46.829227+0000","last_peered":"2024-03-28T18:55:46.829227+0000","last_clean":"2024-03-28T18:55:46.829227+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:55:46.829227+0000","last_undegraded":"2024-03-28T18:55:46.829227+0000","last_fullsized":"2024-03-28T18:55:46.829227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613667+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:55:46.613667+0000","last_peered":"2024-03-28T18:55:46.613667+0000","last_clean":"2024-03-28T18:55:46.613667+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:55:46.613667+0000","last_undegraded":"2024-03-28T18:55:46.613667+0000","last_fullsized":"2024-03-28T18:55:46.613667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825795+0000","last_change":"2024-03-28T18:55:44.601836+0000","last_active":"2024-03-28T18:55:46.825795+0000","last_peered":"2024-03-28T18:55:46.825795+0000","last_clean":"2024-03-28T18:55:46.825795+0000","last_became_active":"2024-03-28T18:55:44.601552+0000","last_became_peered":"2024-03-28T18:55:44.601552+0000","last_unstale":"2024-03-28T18:55:46.825795+0000","last_undegraded":"2024-03-28T18:55:46.825795+0000","last_fullsized":"2024-03-28T18:55:46.825795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603619+0000","last_change":"2024-03-28T18:55:45.601801+0000","last_active":"2024-03-28T18:55:46.603619+0000","last_peered":"2024-03-28T18:55:46.603619+0000","last_clean":"2024-03-28T18:55:46.603619+0000","last_became_active":"2024-03-28T18:55:45.601459+0000","last_became_peered":"2024-03-28T18:55:45.601459+0000","last_unstale":"2024-03-28T18:55:46.603619+0000","last_undegraded":"2024-03-28T18:55:46.603619+0000","last_fullsized":"2024-03-28T18:55:46.603619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:43.033170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613982+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:55:46.613982+0000","last_peered":"2024-03-28T18:55:46.613982+0000","last_clean":"2024-03-28T18:55:46.613982+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:55:46.613982+0000","last_undegraded":"2024-03-28T18:55:46.613982+0000","last_fullsized":"2024-03-28T18:55:46.613982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165028+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:55:47.165028+0000","last_peered":"2024-03-28T18:55:47.165028+0000","last_clean":"2024-03-28T18:55:47.165028+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:55:47.165028+0000","last_undegraded":"2024-03-28T18:55:47.165028+0000","last_fullsized":"2024-03-28T18:55:47.165028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608494+0000","last_change":"2024-03-28T18:55:44.611148+0000","last_active":"2024-03-28T18:55:46.608494+0000","last_peered":"2024-03-28T18:55:46.608494+0000","last_clean":"2024-03-28T18:55:46.608494+0000","last_became_active":"2024-03-28T18:55:44.610835+0000","last_became_peered":"2024-03-28T18:55:44.610835+0000","last_unstale":"2024-03-28T18:55:46.608494+0000","last_undegraded":"2024-03-28T18:55:46.608494+0000","last_fullsized":"2024-03-28T18:55:46.608494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608851+0000","last_change":"2024-03-28T18:55:45.623672+0000","last_active":"2024-03-28T18:55:46.608851+0000","last_peered":"2024-03-28T18:55:46.608851+0000","last_clean":"2024-03-28T18:55:46.608851+0000","last_became_active":"2024-03-28T18:55:45.622568+0000","last_became_peered":"2024-03-28T18:55:45.622568+0000","last_unstale":"2024-03-28T18:55:46.608851+0000","last_undegraded":"2024-03-28T18:55:46.608851+0000","last_fullsized":"2024-03-28T18:55:46.608851+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117731+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:55:47.117731+0000","last_peered":"2024-03-28T18:55:47.117731+0000","last_clean":"2024-03-28T18:55:47.117731+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:55:47.117731+0000","last_undegraded":"2024-03-28T18:55:47.117731+0000","last_fullsized":"2024-03-28T18:55:47.117731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614041+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:55:46.614041+0000","last_peered":"2024-03-28T18:55:46.614041+0000","last_clean":"2024-03-28T18:55:46.614041+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:55:46.614041+0000","last_undegraded":"2024-03-28T18:55:46.614041+0000","last_fullsized":"2024-03-28T18:55:46.614041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613204+0000","last_change":"2024-03-28T18:55:44.604077+0000","last_active":"2024-03-28T18:55:46.613204+0000","last_peered":"2024-03-28T18:55:46.613204+0000","last_clean":"2024-03-28T18:55:46.613204+0000","last_became_active":"2024-03-28T18:55:44.603773+0000","last_became_peered":"2024-03-28T18:55:44.603773+0000","last_unstale":"2024-03-28T18:55:46.613204+0000","last_undegraded":"2024-03-28T18:55:46.613204+0000","last_fullsized":"2024-03-28T18:55:46.613204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608665+0000","last_change":"2024-03-28T18:55:45.615444+0000","last_active":"2024-03-28T18:55:46.608665+0000","last_peered":"2024-03-28T18:55:46.608665+0000","last_clean":"2024-03-28T18:55:46.608665+0000","last_became_active":"2024-03-28T18:55:45.615002+0000","last_became_peered":"2024-03-28T18:55:45.615002+0000","last_unstale":"2024-03-28T18:55:46.608665+0000","last_undegraded":"2024-03-28T18:55:46.608665+0000","last_fullsized":"2024-03-28T18:55:46.608665+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605186+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:55:46.605186+0000","last_peered":"2024-03-28T18:55:46.605186+0000","last_clean":"2024-03-28T18:55:46.605186+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:55:46.605186+0000","last_undegraded":"2024-03-28T18:55:46.605186+0000","last_fullsized":"2024-03-28T18:55:46.605186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116841+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:55:47.116841+0000","last_peered":"2024-03-28T18:55:47.116841+0000","last_clean":"2024-03-28T18:55:47.116841+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:55:47.116841+0000","last_undegraded":"2024-03-28T18:55:47.116841+0000","last_fullsized":"2024-03-28T18:55:47.116841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613947+0000","last_change":"2024-03-28T18:55:44.590703+0000","last_active":"2024-03-28T18:55:46.613947+0000","last_peered":"2024-03-28T18:55:46.613947+0000","last_clean":"2024-03-28T18:55:46.613947+0000","last_became_active":"2024-03-28T18:55:44.590408+0000","last_became_peered":"2024-03-28T18:55:44.590408+0000","last_unstale":"2024-03-28T18:55:46.613947+0000","last_undegraded":"2024-03-28T18:55:46.613947+0000","last_fullsized":"2024-03-28T18:55:46.613947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603326+0000","last_change":"2024-03-28T18:55:45.609795+0000","last_active":"2024-03-28T18:55:46.603326+0000","last_peered":"2024-03-28T18:55:46.603326+0000","last_clean":"2024-03-28T18:55:46.603326+0000","last_became_active":"2024-03-28T18:55:45.603695+0000","last_became_peered":"2024-03-28T18:55:45.603695+0000","last_unstale":"2024-03-28T18:55:46.603326+0000","last_undegraded":"2024-03-28T18:55:46.603326+0000","last_fullsized":"2024-03-28T18:55:46.603326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:49.173555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612873+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:55:46.612873+0000","last_peered":"2024-03-28T18:55:46.612873+0000","last_clean":"2024-03-28T18:55:46.612873+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:55:46.612873+0000","last_undegraded":"2024-03-28T18:55:46.612873+0000","last_fullsized":"2024-03-28T18:55:46.612873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602878+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:55:46.602878+0000","last_peered":"2024-03-28T18:55:46.602878+0000","last_clean":"2024-03-28T18:55:46.602878+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:55:46.602878+0000","last_undegraded":"2024-03-28T18:55:46.602878+0000","last_fullsized":"2024-03-28T18:55:46.602878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614195+0000","last_change":"2024-03-28T18:55:44.612172+0000","last_active":"2024-03-28T18:55:46.614195+0000","last_peered":"2024-03-28T18:55:46.614195+0000","last_clean":"2024-03-28T18:55:46.614195+0000","last_became_active":"2024-03-28T18:55:44.611859+0000","last_became_peered":"2024-03-28T18:55:44.611859+0000","last_unstale":"2024-03-28T18:55:46.614195+0000","last_undegraded":"2024-03-28T18:55:46.614195+0000","last_fullsized":"2024-03-28T18:55:46.614195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117969+0000","last_change":"2024-03-28T18:55:45.619723+0000","last_active":"2024-03-28T18:55:47.117969+0000","last_peered":"2024-03-28T18:55:47.117969+0000","last_clean":"2024-03-28T18:55:47.117969+0000","last_became_active":"2024-03-28T18:55:45.619437+0000","last_became_peered":"2024-03-28T18:55:45.619437+0000","last_unstale":"2024-03-28T18:55:47.117969+0000","last_undegraded":"2024-03-28T18:55:47.117969+0000","last_fullsized":"2024-03-28T18:55:47.117969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608071+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:55:46.608071+0000","last_peered":"2024-03-28T18:55:46.608071+0000","last_clean":"2024-03-28T18:55:46.608071+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:55:46.608071+0000","last_undegraded":"2024-03-28T18:55:46.608071+0000","last_fullsized":"2024-03-28T18:55:46.608071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164105+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:55:47.164105+0000","last_peered":"2024-03-28T18:55:47.164105+0000","last_clean":"2024-03-28T18:55:47.164105+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:55:47.164105+0000","last_undegraded":"2024-03-28T18:55:47.164105+0000","last_fullsized":"2024-03-28T18:55:47.164105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.611391+0000","last_change":"2024-03-28T18:55:44.592064+0000","last_active":"2024-03-28T18:55:46.611391+0000","last_peered":"2024-03-28T18:55:46.611391+0000","last_clean":"2024-03-28T18:55:46.611391+0000","last_became_active":"2024-03-28T18:55:44.591263+0000","last_became_peered":"2024-03-28T18:55:44.591263+0000","last_unstale":"2024-03-28T18:55:46.611391+0000","last_undegraded":"2024-03-28T18:55:46.611391+0000","last_fullsized":"2024-03-28T18:55:46.611391+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163958+0000","last_change":"2024-03-28T18:55:45.614820+0000","last_active":"2024-03-28T18:55:47.163958+0000","last_peered":"2024-03-28T18:55:47.163958+0000","last_clean":"2024-03-28T18:55:47.163958+0000","last_became_active":"2024-03-28T18:55:45.614517+0000","last_became_peered":"2024-03-28T18:55:45.614517+0000","last_unstale":"2024-03-28T18:55:47.163958+0000","last_undegraded":"2024-03-28T18:55:47.163958+0000","last_fullsized":"2024-03-28T18:55:47.163958+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826245+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:55:46.826245+0000","last_peered":"2024-03-28T18:55:46.826245+0000","last_clean":"2024-03-28T18:55:46.826245+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:55:46.826245+0000","last_undegraded":"2024-03-28T18:55:46.826245+0000","last_fullsized":"2024-03-28T18:55:46.826245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116294+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:55:47.116294+0000","last_peered":"2024-03-28T18:55:47.116294+0000","last_clean":"2024-03-28T18:55:47.116294+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:55:47.116294+0000","last_undegraded":"2024-03-28T18:55:47.116294+0000","last_fullsized":"2024-03-28T18:55:47.116294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164472+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:55:47.164472+0000","last_peered":"2024-03-28T18:55:47.164472+0000","last_clean":"2024-03-28T18:55:47.164472+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:55:47.164472+0000","last_undegraded":"2024-03-28T18:55:47.164472+0000","last_fullsized":"2024-03-28T18:55:47.164472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602907+0000","last_change":"2024-03-28T18:55:44.602097+0000","last_active":"2024-03-28T18:55:46.602907+0000","last_peered":"2024-03-28T18:55:46.602907+0000","last_clean":"2024-03-28T18:55:46.602907+0000","last_became_active":"2024-03-28T18:55:44.601769+0000","last_became_peered":"2024-03-28T18:55:44.601769+0000","last_unstale":"2024-03-28T18:55:46.602907+0000","last_undegraded":"2024-03-28T18:55:46.602907+0000","last_fullsized":"2024-03-28T18:55:46.602907+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603447+0000","last_change":"2024-03-28T18:55:45.616485+0000","last_active":"2024-03-28T18:55:46.603447+0000","last_peered":"2024-03-28T18:55:46.603447+0000","last_clean":"2024-03-28T18:55:46.603447+0000","last_became_active":"2024-03-28T18:55:45.615813+0000","last_became_peered":"2024-03-28T18:55:45.615813+0000","last_unstale":"2024-03-28T18:55:46.603447+0000","last_undegraded":"2024-03-28T18:55:46.603447+0000","last_fullsized":"2024-03-28T18:55:46.603447+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:21.227281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164685+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:55:47.164685+0000","last_peered":"2024-03-28T18:55:47.164685+0000","last_clean":"2024-03-28T18:55:47.164685+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:55:47.164685+0000","last_undegraded":"2024-03-28T18:55:47.164685+0000","last_fullsized":"2024-03-28T18:55:47.164685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603451+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:55:46.603451+0000","last_peered":"2024-03-28T18:55:46.603451+0000","last_clean":"2024-03-28T18:55:46.603451+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:55:46.603451+0000","last_undegraded":"2024-03-28T18:55:46.603451+0000","last_fullsized":"2024-03-28T18:55:46.603451+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603782+0000","last_change":"2024-03-28T18:55:44.607744+0000","last_active":"2024-03-28T18:55:46.603782+0000","last_peered":"2024-03-28T18:55:46.603782+0000","last_clean":"2024-03-28T18:55:46.603782+0000","last_became_active":"2024-03-28T18:55:44.606993+0000","last_became_peered":"2024-03-28T18:55:44.606993+0000","last_unstale":"2024-03-28T18:55:46.603782+0000","last_undegraded":"2024-03-28T18:55:46.603782+0000","last_fullsized":"2024-03-28T18:55:46.603782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602563+0000","last_change":"2024-03-28T18:55:45.606307+0000","last_active":"2024-03-28T18:55:46.602563+0000","last_peered":"2024-03-28T18:55:46.602563+0000","last_clean":"2024-03-28T18:55:46.602563+0000","last_became_active":"2024-03-28T18:55:45.605980+0000","last_became_peered":"2024-03-28T18:55:45.605980+0000","last_unstale":"2024-03-28T18:55:46.602563+0000","last_undegraded":"2024-03-28T18:55:46.602563+0000","last_fullsized":"2024-03-28T18:55:46.602563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:52.485966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117600+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:55:47.117600+0000","last_peered":"2024-03-28T18:55:47.117600+0000","last_clean":"2024-03-28T18:55:47.117600+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:55:47.117600+0000","last_undegraded":"2024-03-28T18:55:47.117600+0000","last_fullsized":"2024-03-28T18:55:47.117600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614275+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:55:46.614275+0000","last_peered":"2024-03-28T18:55:46.614275+0000","last_clean":"2024-03-28T18:55:46.614275+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:55:46.614275+0000","last_undegraded":"2024-03-28T18:55:46.614275+0000","last_fullsized":"2024-03-28T18:55:46.614275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608346+0000","last_change":"2024-03-28T18:55:44.607323+0000","last_active":"2024-03-28T18:55:46.608346+0000","last_peered":"2024-03-28T18:55:46.608346+0000","last_clean":"2024-03-28T18:55:46.608346+0000","last_became_active":"2024-03-28T18:55:44.606579+0000","last_became_peered":"2024-03-28T18:55:44.606579+0000","last_unstale":"2024-03-28T18:55:46.608346+0000","last_undegraded":"2024-03-28T18:55:46.608346+0000","last_fullsized":"2024-03-28T18:55:46.608346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163490+0000","last_change":"2024-03-28T18:55:45.622760+0000","last_active":"2024-03-28T18:55:47.163490+0000","last_peered":"2024-03-28T18:55:47.163490+0000","last_clean":"2024-03-28T18:55:47.163490+0000","last_became_active":"2024-03-28T18:55:45.622353+0000","last_became_peered":"2024-03-28T18:55:45.622353+0000","last_unstale":"2024-03-28T18:55:47.163490+0000","last_undegraded":"2024-03-28T18:55:47.163490+0000","last_fullsized":"2024-03-28T18:55:47.163490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:04.349689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166444+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:55:47.166444+0000","last_peered":"2024-03-28T18:55:47.166444+0000","last_clean":"2024-03-28T18:55:47.166444+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:55:47.166444+0000","last_undegraded":"2024-03-28T18:55:47.166444+0000","last_fullsized":"2024-03-28T18:55:47.166444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603346+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:55:46.603346+0000","last_peered":"2024-03-28T18:55:46.603346+0000","last_clean":"2024-03-28T18:55:46.603346+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:55:46.603346+0000","last_undegraded":"2024-03-28T18:55:46.603346+0000","last_fullsized":"2024-03-28T18:55:46.603346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116165+0000","last_change":"2024-03-28T18:55:44.607237+0000","last_active":"2024-03-28T18:55:47.116165+0000","last_peered":"2024-03-28T18:55:47.116165+0000","last_clean":"2024-03-28T18:55:47.116165+0000","last_became_active":"2024-03-28T18:55:44.606960+0000","last_became_peered":"2024-03-28T18:55:44.606960+0000","last_unstale":"2024-03-28T18:55:47.116165+0000","last_undegraded":"2024-03-28T18:55:47.116165+0000","last_fullsized":"2024-03-28T18:55:47.116165+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603242+0000","last_change":"2024-03-28T18:55:45.612992+0000","last_active":"2024-03-28T18:55:46.603242+0000","last_peered":"2024-03-28T18:55:46.603242+0000","last_clean":"2024-03-28T18:55:46.603242+0000","last_became_active":"2024-03-28T18:55:45.612125+0000","last_became_peered":"2024-03-28T18:55:45.612125+0000","last_unstale":"2024-03-28T18:55:46.603242+0000","last_undegraded":"2024-03-28T18:55:46.603242+0000","last_fullsized":"2024-03-28T18:55:46.603242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:35.898911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165571+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:55:47.165571+0000","last_peered":"2024-03-28T18:55:47.165571+0000","last_clean":"2024-03-28T18:55:47.165571+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:55:47.165571+0000","last_undegraded":"2024-03-28T18:55:47.165571+0000","last_fullsized":"2024-03-28T18:55:47.165571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605603+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:55:46.605603+0000","last_peered":"2024-03-28T18:55:46.605603+0000","last_clean":"2024-03-28T18:55:46.605603+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:55:46.605603+0000","last_undegraded":"2024-03-28T18:55:46.605603+0000","last_fullsized":"2024-03-28T18:55:46.605603+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604200+0000","last_change":"2024-03-28T18:55:44.607218+0000","last_active":"2024-03-28T18:55:46.604200+0000","last_peered":"2024-03-28T18:55:46.604200+0000","last_clean":"2024-03-28T18:55:46.604200+0000","last_became_active":"2024-03-28T18:55:44.606852+0000","last_became_peered":"2024-03-28T18:55:44.606852+0000","last_unstale":"2024-03-28T18:55:46.604200+0000","last_undegraded":"2024-03-28T18:55:46.604200+0000","last_fullsized":"2024-03-28T18:55:46.604200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:36.624289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829038+0000","last_change":"2024-03-28T18:55:45.618919+0000","last_active":"2024-03-28T18:55:46.829038+0000","last_peered":"2024-03-28T18:55:46.829038+0000","last_clean":"2024-03-28T18:55:46.829038+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:55:46.829038+0000","last_undegraded":"2024-03-28T18:55:46.829038+0000","last_fullsized":"2024-03-28T18:55:46.829038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:29.407508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164081+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:55:47.164081+0000","last_peered":"2024-03-28T18:55:47.164081+0000","last_clean":"2024-03-28T18:55:47.164081+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:55:47.164081+0000","last_undegraded":"2024-03-28T18:55:47.164081+0000","last_fullsized":"2024-03-28T18:55:47.164081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602411+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:55:46.602411+0000","last_peered":"2024-03-28T18:55:46.602411+0000","last_clean":"2024-03-28T18:55:46.602411+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:55:46.602411+0000","last_undegraded":"2024-03-28T18:55:46.602411+0000","last_fullsized":"2024-03-28T18:55:46.602411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609206+0000","last_change":"2024-03-28T18:55:44.598881+0000","last_active":"2024-03-28T18:55:46.609206+0000","last_peered":"2024-03-28T18:55:46.609206+0000","last_clean":"2024-03-28T18:55:46.609206+0000","last_became_active":"2024-03-28T18:55:44.598532+0000","last_became_peered":"2024-03-28T18:55:44.598532+0000","last_unstale":"2024-03-28T18:55:46.609206+0000","last_undegraded":"2024-03-28T18:55:46.609206+0000","last_fullsized":"2024-03-28T18:55:46.609206+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:44.004611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117993+0000","last_change":"2024-03-28T18:55:45.612572+0000","last_active":"2024-03-28T18:55:47.117993+0000","last_peered":"2024-03-28T18:55:47.117993+0000","last_clean":"2024-03-28T18:55:47.117993+0000","last_became_active":"2024-03-28T18:55:45.611844+0000","last_became_peered":"2024-03-28T18:55:45.611844+0000","last_unstale":"2024-03-28T18:55:47.117993+0000","last_undegraded":"2024-03-28T18:55:47.117993+0000","last_fullsized":"2024-03-28T18:55:47.117993+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602718+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:55:46.602718+0000","last_peered":"2024-03-28T18:55:46.602718+0000","last_clean":"2024-03-28T18:55:46.602718+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:55:46.602718+0000","last_undegraded":"2024-03-28T18:55:46.602718+0000","last_fullsized":"2024-03-28T18:55:46.602718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603429+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:55:46.603429+0000","last_peered":"2024-03-28T18:55:46.603429+0000","last_clean":"2024-03-28T18:55:46.603429+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:55:46.603429+0000","last_undegraded":"2024-03-28T18:55:46.603429+0000","last_fullsized":"2024-03-28T18:55:46.603429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829075+0000","last_change":"2024-03-28T18:55:44.609644+0000","last_active":"2024-03-28T18:55:46.829075+0000","last_peered":"2024-03-28T18:55:46.829075+0000","last_clean":"2024-03-28T18:55:46.829075+0000","last_became_active":"2024-03-28T18:55:44.609317+0000","last_became_peered":"2024-03-28T18:55:44.609317+0000","last_unstale":"2024-03-28T18:55:46.829075+0000","last_undegraded":"2024-03-28T18:55:46.829075+0000","last_fullsized":"2024-03-28T18:55:46.829075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:37.662504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116808+0000","last_change":"2024-03-28T18:55:45.621708+0000","last_active":"2024-03-28T18:55:47.116808+0000","last_peered":"2024-03-28T18:55:47.116808+0000","last_clean":"2024-03-28T18:55:47.116808+0000","last_became_active":"2024-03-28T18:55:45.621366+0000","last_became_peered":"2024-03-28T18:55:45.621366+0000","last_unstale":"2024-03-28T18:55:47.116808+0000","last_undegraded":"2024-03-28T18:55:47.116808+0000","last_fullsized":"2024-03-28T18:55:47.116808+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164622+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:55:47.164622+0000","last_peered":"2024-03-28T18:55:47.164622+0000","last_clean":"2024-03-28T18:55:47.164622+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:55:47.164622+0000","last_undegraded":"2024-03-28T18:55:47.164622+0000","last_fullsized":"2024-03-28T18:55:47.164622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116814+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:55:47.116814+0000","last_peered":"2024-03-28T18:55:47.116814+0000","last_clean":"2024-03-28T18:55:47.116814+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:55:47.116814+0000","last_undegraded":"2024-03-28T18:55:47.116814+0000","last_fullsized":"2024-03-28T18:55:47.116814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"23'6","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827637+0000","last_change":"2024-03-28T18:55:44.608949+0000","last_active":"2024-03-28T18:55:46.827637+0000","last_peered":"2024-03-28T18:55:46.827637+0000","last_clean":"2024-03-28T18:55:46.827637+0000","last_became_active":"2024-03-28T18:55:44.608657+0000","last_became_peered":"2024-03-28T18:55:44.608657+0000","last_unstale":"2024-03-28T18:55:46.827637+0000","last_undegraded":"2024-03-28T18:55:46.827637+0000","last_fullsized":"2024-03-28T18:55:46.827637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:04.799314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827662+0000","last_change":"2024-03-28T18:55:45.610686+0000","last_active":"2024-03-28T18:55:46.827662+0000","last_peered":"2024-03-28T18:55:46.827662+0000","last_clean":"2024-03-28T18:55:46.827662+0000","last_became_active":"2024-03-28T18:55:45.610212+0000","last_became_peered":"2024-03-28T18:55:45.610212+0000","last_unstale":"2024-03-28T18:55:46.827662+0000","last_undegraded":"2024-03-28T18:55:46.827662+0000","last_fullsized":"2024-03-28T18:55:46.827662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608274+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:55:46.608274+0000","last_peered":"2024-03-28T18:55:46.608274+0000","last_clean":"2024-03-28T18:55:46.608274+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:55:46.608274+0000","last_undegraded":"2024-03-28T18:55:46.608274+0000","last_fullsized":"2024-03-28T18:55:46.608274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609087+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:55:46.609087+0000","last_peered":"2024-03-28T18:55:46.609087+0000","last_clean":"2024-03-28T18:55:46.609087+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:55:46.609087+0000","last_undegraded":"2024-03-28T18:55:46.609087+0000","last_fullsized":"2024-03-28T18:55:46.609087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"23'3","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117447+0000","last_change":"2024-03-28T18:55:44.602762+0000","last_active":"2024-03-28T18:55:47.117447+0000","last_peered":"2024-03-28T18:55:47.117447+0000","last_clean":"2024-03-28T18:55:47.117447+0000","last_became_active":"2024-03-28T18:55:44.602424+0000","last_became_peered":"2024-03-28T18:55:44.602424+0000","last_unstale":"2024-03-28T18:55:47.117447+0000","last_undegraded":"2024-03-28T18:55:47.117447+0000","last_fullsized":"2024-03-28T18:55:47.117447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826897+0000","last_change":"2024-03-28T18:55:45.601277+0000","last_active":"2024-03-28T18:55:46.826897+0000","last_peered":"2024-03-28T18:55:46.826897+0000","last_clean":"2024-03-28T18:55:46.826897+0000","last_became_active":"2024-03-28T18:55:45.600945+0000","last_became_peered":"2024-03-28T18:55:45.600945+0000","last_unstale":"2024-03-28T18:55:46.826897+0000","last_undegraded":"2024-03-28T18:55:46.826897+0000","last_fullsized":"2024-03-28T18:55:46.826897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164660+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:55:47.164660+0000","last_peered":"2024-03-28T18:55:47.164660+0000","last_clean":"2024-03-28T18:55:47.164660+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:55:47.164660+0000","last_undegraded":"2024-03-28T18:55:47.164660+0000","last_fullsized":"2024-03-28T18:55:47.164660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602857+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:55:46.602857+0000","last_peered":"2024-03-28T18:55:46.602857+0000","last_clean":"2024-03-28T18:55:46.602857+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:55:46.602857+0000","last_undegraded":"2024-03-28T18:55:46.602857+0000","last_fullsized":"2024-03-28T18:55:46.602857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612960+0000","last_change":"2024-03-28T18:55:44.599432+0000","last_active":"2024-03-28T18:55:46.612960+0000","last_peered":"2024-03-28T18:55:46.612960+0000","last_clean":"2024-03-28T18:55:46.612960+0000","last_became_active":"2024-03-28T18:55:44.599123+0000","last_became_peered":"2024-03-28T18:55:44.599123+0000","last_unstale":"2024-03-28T18:55:46.612960+0000","last_undegraded":"2024-03-28T18:55:46.612960+0000","last_fullsized":"2024-03-28T18:55:46.612960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613285+0000","last_change":"2024-03-28T18:55:45.624311+0000","last_active":"2024-03-28T18:55:46.613285+0000","last_peered":"2024-03-28T18:55:46.613285+0000","last_clean":"2024-03-28T18:55:46.613285+0000","last_became_active":"2024-03-28T18:55:45.624031+0000","last_became_peered":"2024-03-28T18:55:45.624031+0000","last_unstale":"2024-03-28T18:55:46.613285+0000","last_undegraded":"2024-03-28T18:55:46.613285+0000","last_fullsized":"2024-03-28T18:55:46.613285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605962+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:55:46.605962+0000","last_peered":"2024-03-28T18:55:46.605962+0000","last_clean":"2024-03-28T18:55:46.605962+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:55:46.605962+0000","last_undegraded":"2024-03-28T18:55:46.605962+0000","last_fullsized":"2024-03-28T18:55:46.605962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614210+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:55:46.614210+0000","last_peered":"2024-03-28T18:55:46.614210+0000","last_clean":"2024-03-28T18:55:46.614210+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:55:46.614210+0000","last_undegraded":"2024-03-28T18:55:46.614210+0000","last_fullsized":"2024-03-28T18:55:46.614210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604974+0000","last_change":"2024-03-28T18:55:44.593567+0000","last_active":"2024-03-28T18:55:46.604974+0000","last_peered":"2024-03-28T18:55:46.604974+0000","last_clean":"2024-03-28T18:55:46.604974+0000","last_became_active":"2024-03-28T18:55:44.593267+0000","last_became_peered":"2024-03-28T18:55:44.593267+0000","last_unstale":"2024-03-28T18:55:46.604974+0000","last_undegraded":"2024-03-28T18:55:46.604974+0000","last_fullsized":"2024-03-28T18:55:46.604974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603331+0000","last_change":"2024-03-28T18:55:45.598713+0000","last_active":"2024-03-28T18:55:46.603331+0000","last_peered":"2024-03-28T18:55:46.603331+0000","last_clean":"2024-03-28T18:55:46.603331+0000","last_became_active":"2024-03-28T18:55:45.597726+0000","last_became_peered":"2024-03-28T18:55:45.597726+0000","last_unstale":"2024-03-28T18:55:46.603331+0000","last_undegraded":"2024-03-28T18:55:46.603331+0000","last_fullsized":"2024-03-28T18:55:46.603331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:48.255745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.609101+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:55:46.609101+0000","last_peered":"2024-03-28T18:55:46.609101+0000","last_clean":"2024-03-28T18:55:46.609101+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:55:46.609101+0000","last_undegraded":"2024-03-28T18:55:46.609101+0000","last_fullsized":"2024-03-28T18:55:46.609101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118062+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:55:47.118062+0000","last_peered":"2024-03-28T18:55:47.118062+0000","last_clean":"2024-03-28T18:55:47.118062+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:55:47.118062+0000","last_undegraded":"2024-03-28T18:55:47.118062+0000","last_fullsized":"2024-03-28T18:55:47.118062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"23'8","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826291+0000","last_change":"2024-03-28T18:55:44.610673+0000","last_active":"2024-03-28T18:55:46.826291+0000","last_peered":"2024-03-28T18:55:46.826291+0000","last_clean":"2024-03-28T18:55:46.826291+0000","last_became_active":"2024-03-28T18:55:44.610357+0000","last_became_peered":"2024-03-28T18:55:44.610357+0000","last_unstale":"2024-03-28T18:55:46.826291+0000","last_undegraded":"2024-03-28T18:55:46.826291+0000","last_fullsized":"2024-03-28T18:55:46.826291+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:26.280548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163667+0000","last_change":"2024-03-28T18:55:45.609349+0000","last_active":"2024-03-28T18:55:47.163667+0000","last_peered":"2024-03-28T18:55:47.163667+0000","last_clean":"2024-03-28T18:55:47.163667+0000","last_became_active":"2024-03-28T18:55:45.608743+0000","last_became_peered":"2024-03-28T18:55:45.608743+0000","last_unstale":"2024-03-28T18:55:47.163667+0000","last_undegraded":"2024-03-28T18:55:47.163667+0000","last_fullsized":"2024-03-28T18:55:47.163667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118036+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:55:47.118036+0000","last_peered":"2024-03-28T18:55:47.118036+0000","last_clean":"2024-03-28T18:55:47.118036+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:55:47.118036+0000","last_undegraded":"2024-03-28T18:55:47.118036+0000","last_fullsized":"2024-03-28T18:55:47.118036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604933+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:55:46.604933+0000","last_peered":"2024-03-28T18:55:46.604933+0000","last_clean":"2024-03-28T18:55:46.604933+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:55:46.604933+0000","last_undegraded":"2024-03-28T18:55:46.604933+0000","last_fullsized":"2024-03-28T18:55:46.604933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825952+0000","last_change":"2024-03-28T18:55:44.601103+0000","last_active":"2024-03-28T18:55:46.825952+0000","last_peered":"2024-03-28T18:55:46.825952+0000","last_clean":"2024-03-28T18:55:46.825952+0000","last_became_active":"2024-03-28T18:55:44.599969+0000","last_became_peered":"2024-03-28T18:55:44.599969+0000","last_unstale":"2024-03-28T18:55:46.825952+0000","last_undegraded":"2024-03-28T18:55:46.825952+0000","last_fullsized":"2024-03-28T18:55:46.825952+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:42.018654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604371+0000","last_change":"2024-03-28T18:55:45.594944+0000","last_active":"2024-03-28T18:55:46.604371+0000","last_peered":"2024-03-28T18:55:46.604371+0000","last_clean":"2024-03-28T18:55:46.604371+0000","last_became_active":"2024-03-28T18:55:45.594603+0000","last_became_peered":"2024-03-28T18:55:45.594603+0000","last_unstale":"2024-03-28T18:55:46.604371+0000","last_undegraded":"2024-03-28T18:55:46.604371+0000","last_fullsized":"2024-03-28T18:55:46.604371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"23'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608967+0000","last_change":"2024-03-28T18:55:44.602366+0000","last_active":"2024-03-28T18:55:46.608967+0000","last_peered":"2024-03-28T18:55:46.608967+0000","last_clean":"2024-03-28T18:55:46.608967+0000","last_became_active":"2024-03-28T18:55:44.602033+0000","last_became_peered":"2024-03-28T18:55:44.602033+0000","last_unstale":"2024-03-28T18:55:46.608967+0000","last_undegraded":"2024-03-28T18:55:46.608967+0000","last_fullsized":"2024-03-28T18:55:46.608967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612862+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:55:46.612862+0000","last_peered":"2024-03-28T18:55:46.612862+0000","last_clean":"2024-03-28T18:55:46.612862+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:55:46.612862+0000","last_undegraded":"2024-03-28T18:55:46.612862+0000","last_fullsized":"2024-03-28T18:55:46.612862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117107+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:55:47.117107+0000","last_peered":"2024-03-28T18:55:47.117107+0000","last_clean":"2024-03-28T18:55:47.117107+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:55:47.117107+0000","last_undegraded":"2024-03-28T18:55:47.117107+0000","last_fullsized":"2024-03-28T18:55:47.117107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608811+0000","last_change":"2024-03-28T18:55:45.620559+0000","last_active":"2024-03-28T18:55:46.608811+0000","last_peered":"2024-03-28T18:55:46.608811+0000","last_clean":"2024-03-28T18:55:46.608811+0000","last_became_active":"2024-03-28T18:55:45.620249+0000","last_became_peered":"2024-03-28T18:55:45.620249+0000","last_unstale":"2024-03-28T18:55:46.608811+0000","last_undegraded":"2024-03-28T18:55:46.608811+0000","last_fullsized":"2024-03-28T18:55:46.608811+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"23'4","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604650+0000","last_change":"2024-03-28T18:55:44.612226+0000","last_active":"2024-03-28T18:55:46.604650+0000","last_peered":"2024-03-28T18:55:46.604650+0000","last_clean":"2024-03-28T18:55:46.604650+0000","last_became_active":"2024-03-28T18:55:44.611921+0000","last_became_peered":"2024-03-28T18:55:44.611921+0000","last_unstale":"2024-03-28T18:55:46.604650+0000","last_undegraded":"2024-03-28T18:55:46.604650+0000","last_fullsized":"2024-03-28T18:55:46.604650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826474+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:55:46.826474+0000","last_peered":"2024-03-28T18:55:46.826474+0000","last_clean":"2024-03-28T18:55:46.826474+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:55:46.826474+0000","last_undegraded":"2024-03-28T18:55:46.826474+0000","last_fullsized":"2024-03-28T18:55:46.826474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826438+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:55:46.826438+0000","last_peered":"2024-03-28T18:55:46.826438+0000","last_clean":"2024-03-28T18:55:46.826438+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:55:46.826438+0000","last_undegraded":"2024-03-28T18:55:46.826438+0000","last_fullsized":"2024-03-28T18:55:46.826438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614060+0000","last_change":"2024-03-28T18:55:45.612727+0000","last_active":"2024-03-28T18:55:46.614060+0000","last_peered":"2024-03-28T18:55:46.614060+0000","last_clean":"2024-03-28T18:55:46.614060+0000","last_became_active":"2024-03-28T18:55:45.610154+0000","last_became_peered":"2024-03-28T18:55:45.610154+0000","last_unstale":"2024-03-28T18:55:46.614060+0000","last_undegraded":"2024-03-28T18:55:46.614060+0000","last_fullsized":"2024-03-28T18:55:46.614060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604120+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:55:46.604120+0000","last_peered":"2024-03-28T18:55:46.604120+0000","last_clean":"2024-03-28T18:55:46.604120+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:55:46.604120+0000","last_undegraded":"2024-03-28T18:55:46.604120+0000","last_fullsized":"2024-03-28T18:55:46.604120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608921+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:55:46.608921+0000","last_peered":"2024-03-28T18:55:46.608921+0000","last_clean":"2024-03-28T18:55:46.608921+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:55:46.608921+0000","last_undegraded":"2024-03-28T18:55:46.608921+0000","last_fullsized":"2024-03-28T18:55:46.608921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603331+0000","last_change":"2024-03-28T18:55:44.592835+0000","last_active":"2024-03-28T18:55:46.603331+0000","last_peered":"2024-03-28T18:55:46.603331+0000","last_clean":"2024-03-28T18:55:46.603331+0000","last_became_active":"2024-03-28T18:55:44.592505+0000","last_became_peered":"2024-03-28T18:55:44.592505+0000","last_unstale":"2024-03-28T18:55:46.603331+0000","last_undegraded":"2024-03-28T18:55:46.603331+0000","last_fullsized":"2024-03-28T18:55:46.603331+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:58.286421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605269+0000","last_change":"2024-03-28T18:55:45.611322+0000","last_active":"2024-03-28T18:55:46.605269+0000","last_peered":"2024-03-28T18:55:46.605269+0000","last_clean":"2024-03-28T18:55:46.605269+0000","last_became_active":"2024-03-28T18:55:45.610958+0000","last_became_peered":"2024-03-28T18:55:45.610958+0000","last_unstale":"2024-03-28T18:55:46.605269+0000","last_undegraded":"2024-03-28T18:55:46.605269+0000","last_fullsized":"2024-03-28T18:55:46.605269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827108+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:55:46.827108+0000","last_peered":"2024-03-28T18:55:46.827108+0000","last_clean":"2024-03-28T18:55:46.827108+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:55:46.827108+0000","last_undegraded":"2024-03-28T18:55:46.827108+0000","last_fullsized":"2024-03-28T18:55:46.827108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604724+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:55:46.604724+0000","last_peered":"2024-03-28T18:55:46.604724+0000","last_clean":"2024-03-28T18:55:46.604724+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:55:46.604724+0000","last_undegraded":"2024-03-28T18:55:46.604724+0000","last_fullsized":"2024-03-28T18:55:46.604724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"23'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827611+0000","last_change":"2024-03-28T18:55:44.608283+0000","last_active":"2024-03-28T18:55:46.827611+0000","last_peered":"2024-03-28T18:55:46.827611+0000","last_clean":"2024-03-28T18:55:46.827611+0000","last_became_active":"2024-03-28T18:55:44.607999+0000","last_became_peered":"2024-03-28T18:55:44.607999+0000","last_unstale":"2024-03-28T18:55:46.827611+0000","last_undegraded":"2024-03-28T18:55:46.827611+0000","last_fullsized":"2024-03-28T18:55:46.827611+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:22.669778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828611+0000","last_change":"2024-03-28T18:55:45.618390+0000","last_active":"2024-03-28T18:55:46.828611+0000","last_peered":"2024-03-28T18:55:46.828611+0000","last_clean":"2024-03-28T18:55:46.828611+0000","last_became_active":"2024-03-28T18:55:45.617897+0000","last_became_peered":"2024-03-28T18:55:45.617897+0000","last_unstale":"2024-03-28T18:55:46.828611+0000","last_undegraded":"2024-03-28T18:55:46.828611+0000","last_fullsized":"2024-03-28T18:55:46.828611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:16.520575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602446+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:55:46.602446+0000","last_peered":"2024-03-28T18:55:46.602446+0000","last_clean":"2024-03-28T18:55:46.602446+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:55:46.602446+0000","last_undegraded":"2024-03-28T18:55:46.602446+0000","last_fullsized":"2024-03-28T18:55:46.602446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613831+0000","last_change":"2024-03-28T18:55:45.624938+0000","last_active":"2024-03-28T18:55:46.613831+0000","last_peered":"2024-03-28T18:55:46.613831+0000","last_clean":"2024-03-28T18:55:46.613831+0000","last_became_active":"2024-03-28T18:55:45.624362+0000","last_became_peered":"2024-03-28T18:55:45.624362+0000","last_unstale":"2024-03-28T18:55:46.613831+0000","last_undegraded":"2024-03-28T18:55:46.613831+0000","last_fullsized":"2024-03-28T18:55:46.613831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602638+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:55:46.602638+0000","last_peered":"2024-03-28T18:55:46.602638+0000","last_clean":"2024-03-28T18:55:46.602638+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:55:46.602638+0000","last_undegraded":"2024-03-28T18:55:46.602638+0000","last_fullsized":"2024-03-28T18:55:46.602638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116612+0000","last_change":"2024-03-28T18:55:45.610168+0000","last_active":"2024-03-28T18:55:47.116612+0000","last_peered":"2024-03-28T18:55:47.116612+0000","last_clean":"2024-03-28T18:55:47.116612+0000","last_became_active":"2024-03-28T18:55:45.609653+0000","last_became_peered":"2024-03-28T18:55:45.609653+0000","last_unstale":"2024-03-28T18:55:47.116612+0000","last_undegraded":"2024-03-28T18:55:47.116612+0000","last_fullsized":"2024-03-28T18:55:47.116612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116681+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:55:47.116681+0000","last_peered":"2024-03-28T18:55:47.116681+0000","last_clean":"2024-03-28T18:55:47.116681+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:55:47.116681+0000","last_undegraded":"2024-03-28T18:55:47.116681+0000","last_fullsized":"2024-03-28T18:55:47.116681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613952+0000","last_change":"2024-03-28T18:55:45.613701+0000","last_active":"2024-03-28T18:55:46.613952+0000","last_peered":"2024-03-28T18:55:46.613952+0000","last_clean":"2024-03-28T18:55:46.613952+0000","last_became_active":"2024-03-28T18:55:45.613310+0000","last_became_peered":"2024-03-28T18:55:45.613310+0000","last_unstale":"2024-03-28T18:55:46.613952+0000","last_undegraded":"2024-03-28T18:55:46.613952+0000","last_fullsized":"2024-03-28T18:55:46.613952+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608809+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:55:46.608809+0000","last_peered":"2024-03-28T18:55:46.608809+0000","last_clean":"2024-03-28T18:55:46.608809+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:55:46.608809+0000","last_undegraded":"2024-03-28T18:55:46.608809+0000","last_fullsized":"2024-03-28T18:55:46.608809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163732+0000","last_change":"2024-03-28T18:55:45.615119+0000","last_active":"2024-03-28T18:55:47.163732+0000","last_peered":"2024-03-28T18:55:47.163732+0000","last_clean":"2024-03-28T18:55:47.163732+0000","last_became_active":"2024-03-28T18:55:45.613841+0000","last_became_peered":"2024-03-28T18:55:45.613841+0000","last_unstale":"2024-03-28T18:55:47.163732+0000","last_undegraded":"2024-03-28T18:55:47.163732+0000","last_fullsized":"2024-03-28T18:55:47.163732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827969+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:55:46.827969+0000","last_peered":"2024-03-28T18:55:46.827969+0000","last_clean":"2024-03-28T18:55:46.827969+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:55:46.827969+0000","last_undegraded":"2024-03-28T18:55:46.827969+0000","last_fullsized":"2024-03-28T18:55:46.827969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604609+0000","last_change":"2024-03-28T18:55:45.619605+0000","last_active":"2024-03-28T18:55:46.604609+0000","last_peered":"2024-03-28T18:55:46.604609+0000","last_clean":"2024-03-28T18:55:46.604609+0000","last_became_active":"2024-03-28T18:55:45.618473+0000","last_became_peered":"2024-03-28T18:55:45.618473+0000","last_unstale":"2024-03-28T18:55:46.604609+0000","last_undegraded":"2024-03-28T18:55:46.604609+0000","last_fullsized":"2024-03-28T18:55:46.604609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.614151+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:55:46.614151+0000","last_peered":"2024-03-28T18:55:46.614151+0000","last_clean":"2024-03-28T18:55:46.614151+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:55:46.614151+0000","last_undegraded":"2024-03-28T18:55:46.614151+0000","last_fullsized":"2024-03-28T18:55:46.614151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163424+0000","last_change":"2024-03-28T18:55:45.612284+0000","last_active":"2024-03-28T18:55:47.163424+0000","last_peered":"2024-03-28T18:55:47.163424+0000","last_clean":"2024-03-28T18:55:47.163424+0000","last_became_active":"2024-03-28T18:55:45.607218+0000","last_became_peered":"2024-03-28T18:55:45.607218+0000","last_unstale":"2024-03-28T18:55:47.163424+0000","last_undegraded":"2024-03-28T18:55:47.163424+0000","last_fullsized":"2024-03-28T18:55:47.163424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604943+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:55:46.604943+0000","last_peered":"2024-03-28T18:55:46.604943+0000","last_clean":"2024-03-28T18:55:46.604943+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:55:46.604943+0000","last_undegraded":"2024-03-28T18:55:46.604943+0000","last_fullsized":"2024-03-28T18:55:46.604943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603065+0000","last_change":"2024-03-28T18:55:45.607609+0000","last_active":"2024-03-28T18:55:46.603065+0000","last_peered":"2024-03-28T18:55:46.603065+0000","last_clean":"2024-03-28T18:55:46.603065+0000","last_became_active":"2024-03-28T18:55:45.602582+0000","last_became_peered":"2024-03-28T18:55:45.602582+0000","last_unstale":"2024-03-28T18:55:46.603065+0000","last_undegraded":"2024-03-28T18:55:46.603065+0000","last_fullsized":"2024-03-28T18:55:46.603065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:06:05.335715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.829082+0000","last_change":"2024-03-28T18:55:34.491857+0000","last_active":"2024-03-28T18:55:46.829082+0000","last_peered":"2024-03-28T18:55:46.829082+0000","last_clean":"2024-03-28T18:55:46.829082+0000","last_became_active":"2024-03-28T18:55:34.487194+0000","last_became_peered":"2024-03-28T18:55:34.487194+0000","last_unstale":"2024-03-28T18:55:46.829082+0000","last_undegraded":"2024-03-28T18:55:46.829082+0000","last_fullsized":"2024-03-28T18:55:46.829082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828429+0000","last_change":"2024-03-28T18:55:45.607886+0000","last_active":"2024-03-28T18:55:46.828429+0000","last_peered":"2024-03-28T18:55:46.828429+0000","last_clean":"2024-03-28T18:55:46.828429+0000","last_became_active":"2024-03-28T18:55:45.605964+0000","last_became_peered":"2024-03-28T18:55:45.605964+0000","last_unstale":"2024-03-28T18:55:46.828429+0000","last_undegraded":"2024-03-28T18:55:46.828429+0000","last_fullsized":"2024-03-28T18:55:46.828429+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:45.986654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117089+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:55:47.117089+0000","last_peered":"2024-03-28T18:55:47.117089+0000","last_clean":"2024-03-28T18:55:47.117089+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:55:47.117089+0000","last_undegraded":"2024-03-28T18:55:47.117089+0000","last_fullsized":"2024-03-28T18:55:47.117089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.165971+0000","last_change":"2024-03-28T18:55:45.618031+0000","last_active":"2024-03-28T18:55:47.165971+0000","last_peered":"2024-03-28T18:55:47.165971+0000","last_clean":"2024-03-28T18:55:47.165971+0000","last_became_active":"2024-03-28T18:55:45.617719+0000","last_became_peered":"2024-03-28T18:55:45.617719+0000","last_unstale":"2024-03-28T18:55:47.165971+0000","last_undegraded":"2024-03-28T18:55:47.165971+0000","last_fullsized":"2024-03-28T18:55:47.165971+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826204+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:55:46.826204+0000","last_peered":"2024-03-28T18:55:46.826204+0000","last_clean":"2024-03-28T18:55:46.826204+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:55:46.826204+0000","last_undegraded":"2024-03-28T18:55:46.826204+0000","last_fullsized":"2024-03-28T18:55:46.826204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603156+0000","last_change":"2024-03-28T18:55:45.603783+0000","last_active":"2024-03-28T18:55:46.603156+0000","last_peered":"2024-03-28T18:55:46.603156+0000","last_clean":"2024-03-28T18:55:46.603156+0000","last_became_active":"2024-03-28T18:55:45.603474+0000","last_became_peered":"2024-03-28T18:55:45.603474+0000","last_unstale":"2024-03-28T18:55:46.603156+0000","last_undegraded":"2024-03-28T18:55:46.603156+0000","last_fullsized":"2024-03-28T18:55:46.603156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:42.328182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116551+0000","last_change":"2024-03-28T18:55:34.505741+0000","last_active":"2024-03-28T18:55:47.116551+0000","last_peered":"2024-03-28T18:55:47.116551+0000","last_clean":"2024-03-28T18:55:47.116551+0000","last_became_active":"2024-03-28T18:55:34.505440+0000","last_became_peered":"2024-03-28T18:55:34.505440+0000","last_unstale":"2024-03-28T18:55:47.116551+0000","last_undegraded":"2024-03-28T18:55:47.116551+0000","last_fullsized":"2024-03-28T18:55:47.116551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.827452+0000","last_change":"2024-03-28T18:55:45.618912+0000","last_active":"2024-03-28T18:55:46.827452+0000","last_peered":"2024-03-28T18:55:46.827452+0000","last_clean":"2024-03-28T18:55:46.827452+0000","last_became_active":"2024-03-28T18:55:45.618554+0000","last_became_peered":"2024-03-28T18:55:45.618554+0000","last_unstale":"2024-03-28T18:55:46.827452+0000","last_undegraded":"2024-03-28T18:55:46.827452+0000","last_fullsized":"2024-03-28T18:55:46.827452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608624+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:55:46.608624+0000","last_peered":"2024-03-28T18:55:46.608624+0000","last_clean":"2024-03-28T18:55:46.608624+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:55:46.608624+0000","last_undegraded":"2024-03-28T18:55:46.608624+0000","last_fullsized":"2024-03-28T18:55:46.608624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604333+0000","last_change":"2024-03-28T18:55:45.616420+0000","last_active":"2024-03-28T18:55:46.604333+0000","last_peered":"2024-03-28T18:55:46.604333+0000","last_clean":"2024-03-28T18:55:46.604333+0000","last_became_active":"2024-03-28T18:55:45.614806+0000","last_became_peered":"2024-03-28T18:55:45.614806+0000","last_unstale":"2024-03-28T18:55:46.604333+0000","last_undegraded":"2024-03-28T18:55:46.604333+0000","last_fullsized":"2024-03-28T18:55:46.604333+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602688+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:55:46.602688+0000","last_peered":"2024-03-28T18:55:46.602688+0000","last_clean":"2024-03-28T18:55:46.602688+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:55:46.602688+0000","last_undegraded":"2024-03-28T18:55:46.602688+0000","last_fullsized":"2024-03-28T18:55:46.602688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604338+0000","last_change":"2024-03-28T18:55:45.606507+0000","last_active":"2024-03-28T18:55:46.604338+0000","last_peered":"2024-03-28T18:55:46.604338+0000","last_clean":"2024-03-28T18:55:46.604338+0000","last_became_active":"2024-03-28T18:55:45.606205+0000","last_became_peered":"2024-03-28T18:55:45.606205+0000","last_unstale":"2024-03-28T18:55:46.604338+0000","last_undegraded":"2024-03-28T18:55:46.604338+0000","last_fullsized":"2024-03-28T18:55:46.604338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164913+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:55:47.164913+0000","last_peered":"2024-03-28T18:55:47.164913+0000","last_clean":"2024-03-28T18:55:47.164913+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:55:47.164913+0000","last_undegraded":"2024-03-28T18:55:47.164913+0000","last_fullsized":"2024-03-28T18:55:47.164913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116033+0000","last_change":"2024-03-28T18:55:45.607890+0000","last_active":"2024-03-28T18:55:47.116033+0000","last_peered":"2024-03-28T18:55:47.116033+0000","last_clean":"2024-03-28T18:55:47.116033+0000","last_became_active":"2024-03-28T18:55:45.607590+0000","last_became_peered":"2024-03-28T18:55:45.607590+0000","last_unstale":"2024-03-28T18:55:47.116033+0000","last_undegraded":"2024-03-28T18:55:47.116033+0000","last_fullsized":"2024-03-28T18:55:47.116033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603209+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:55:46.603209+0000","last_peered":"2024-03-28T18:55:46.603209+0000","last_clean":"2024-03-28T18:55:46.603209+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:55:46.603209+0000","last_undegraded":"2024-03-28T18:55:46.603209+0000","last_fullsized":"2024-03-28T18:55:46.603209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608156+0000","last_change":"2024-03-28T18:55:45.608851+0000","last_active":"2024-03-28T18:55:46.608156+0000","last_peered":"2024-03-28T18:55:46.608156+0000","last_clean":"2024-03-28T18:55:46.608156+0000","last_became_active":"2024-03-28T18:55:45.608496+0000","last_became_peered":"2024-03-28T18:55:45.608496+0000","last_unstale":"2024-03-28T18:55:46.608156+0000","last_undegraded":"2024-03-28T18:55:46.608156+0000","last_fullsized":"2024-03-28T18:55:46.608156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.118072+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:55:47.118072+0000","last_peered":"2024-03-28T18:55:47.118072+0000","last_clean":"2024-03-28T18:55:47.118072+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:55:47.118072+0000","last_undegraded":"2024-03-28T18:55:47.118072+0000","last_fullsized":"2024-03-28T18:55:47.118072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613556+0000","last_change":"2024-03-28T18:55:45.619501+0000","last_active":"2024-03-28T18:55:46.613556+0000","last_peered":"2024-03-28T18:55:46.613556+0000","last_clean":"2024-03-28T18:55:46.613556+0000","last_became_active":"2024-03-28T18:55:45.619212+0000","last_became_peered":"2024-03-28T18:55:45.619212+0000","last_unstale":"2024-03-28T18:55:46.613556+0000","last_undegraded":"2024-03-28T18:55:46.613556+0000","last_fullsized":"2024-03-28T18:55:46.613556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116821+0000","last_change":"2024-03-28T18:55:34.509529+0000","last_active":"2024-03-28T18:55:47.116821+0000","last_peered":"2024-03-28T18:55:47.116821+0000","last_clean":"2024-03-28T18:55:47.116821+0000","last_became_active":"2024-03-28T18:55:34.501100+0000","last_became_peered":"2024-03-28T18:55:34.501100+0000","last_unstale":"2024-03-28T18:55:47.116821+0000","last_undegraded":"2024-03-28T18:55:47.116821+0000","last_fullsized":"2024-03-28T18:55:47.116821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605306+0000","last_change":"2024-03-28T18:55:45.615225+0000","last_active":"2024-03-28T18:55:46.605306+0000","last_peered":"2024-03-28T18:55:46.605306+0000","last_clean":"2024-03-28T18:55:46.605306+0000","last_became_active":"2024-03-28T18:55:45.613602+0000","last_became_peered":"2024-03-28T18:55:45.613602+0000","last_unstale":"2024-03-28T18:55:46.605306+0000","last_undegraded":"2024-03-28T18:55:46.605306+0000","last_fullsized":"2024-03-28T18:55:46.605306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.612997+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:55:46.612997+0000","last_peered":"2024-03-28T18:55:46.612997+0000","last_clean":"2024-03-28T18:55:46.612997+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:55:46.612997+0000","last_undegraded":"2024-03-28T18:55:46.612997+0000","last_fullsized":"2024-03-28T18:55:46.612997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116582+0000","last_change":"2024-03-28T18:55:45.609471+0000","last_active":"2024-03-28T18:55:47.116582+0000","last_peered":"2024-03-28T18:55:47.116582+0000","last_clean":"2024-03-28T18:55:47.116582+0000","last_became_active":"2024-03-28T18:55:45.609132+0000","last_became_peered":"2024-03-28T18:55:45.609132+0000","last_unstale":"2024-03-28T18:55:47.116582+0000","last_undegraded":"2024-03-28T18:55:47.116582+0000","last_fullsized":"2024-03-28T18:55:47.116582+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608272+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:55:46.608272+0000","last_peered":"2024-03-28T18:55:46.608272+0000","last_clean":"2024-03-28T18:55:46.608272+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:55:46.608272+0000","last_undegraded":"2024-03-28T18:55:46.608272+0000","last_fullsized":"2024-03-28T18:55:46.608272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164529+0000","last_change":"2024-03-28T18:55:45.617501+0000","last_active":"2024-03-28T18:55:47.164529+0000","last_peered":"2024-03-28T18:55:47.164529+0000","last_clean":"2024-03-28T18:55:47.164529+0000","last_became_active":"2024-03-28T18:55:45.617180+0000","last_became_peered":"2024-03-28T18:55:45.617180+0000","last_unstale":"2024-03-28T18:55:47.164529+0000","last_undegraded":"2024-03-28T18:55:47.164529+0000","last_fullsized":"2024-03-28T18:55:47.164529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826964+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:55:46.826964+0000","last_peered":"2024-03-28T18:55:46.826964+0000","last_clean":"2024-03-28T18:55:46.826964+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:55:46.826964+0000","last_undegraded":"2024-03-28T18:55:46.826964+0000","last_fullsized":"2024-03-28T18:55:46.826964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826286+0000","last_change":"2024-03-28T18:55:45.609169+0000","last_active":"2024-03-28T18:55:46.826286+0000","last_peered":"2024-03-28T18:55:46.826286+0000","last_clean":"2024-03-28T18:55:46.826286+0000","last_became_active":"2024-03-28T18:55:45.601948+0000","last_became_peered":"2024-03-28T18:55:45.601948+0000","last_unstale":"2024-03-28T18:55:46.826286+0000","last_undegraded":"2024-03-28T18:55:46.826286+0000","last_fullsized":"2024-03-28T18:55:46.826286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.605787+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:55:46.605787+0000","last_peered":"2024-03-28T18:55:46.605787+0000","last_clean":"2024-03-28T18:55:46.605787+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:55:46.605787+0000","last_undegraded":"2024-03-28T18:55:46.605787+0000","last_fullsized":"2024-03-28T18:55:46.605787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603014+0000","last_change":"2024-03-28T18:55:45.616238+0000","last_active":"2024-03-28T18:55:46.603014+0000","last_peered":"2024-03-28T18:55:46.603014+0000","last_clean":"2024-03-28T18:55:46.603014+0000","last_became_active":"2024-03-28T18:55:45.615371+0000","last_became_peered":"2024-03-28T18:55:45.615371+0000","last_unstale":"2024-03-28T18:55:46.603014+0000","last_undegraded":"2024-03-28T18:55:46.603014+0000","last_fullsized":"2024-03-28T18:55:46.603014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:14.661134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603091+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:55:46.603091+0000","last_peered":"2024-03-28T18:55:46.603091+0000","last_clean":"2024-03-28T18:55:46.603091+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:55:46.603091+0000","last_undegraded":"2024-03-28T18:55:46.603091+0000","last_fullsized":"2024-03-28T18:55:46.603091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164732+0000","last_change":"2024-03-28T18:55:45.613289+0000","last_active":"2024-03-28T18:55:47.164732+0000","last_peered":"2024-03-28T18:55:47.164732+0000","last_clean":"2024-03-28T18:55:47.164732+0000","last_became_active":"2024-03-28T18:55:45.610917+0000","last_became_peered":"2024-03-28T18:55:45.610917+0000","last_unstale":"2024-03-28T18:55:47.164732+0000","last_undegraded":"2024-03-28T18:55:47.164732+0000","last_fullsized":"2024-03-28T18:55:47.164732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.604796+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:55:46.604796+0000","last_peered":"2024-03-28T18:55:46.604796+0000","last_clean":"2024-03-28T18:55:46.604796+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:55:46.604796+0000","last_undegraded":"2024-03-28T18:55:46.604796+0000","last_fullsized":"2024-03-28T18:55:46.604796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116103+0000","last_change":"2024-03-28T18:55:45.621789+0000","last_active":"2024-03-28T18:55:47.116103+0000","last_peered":"2024-03-28T18:55:47.116103+0000","last_clean":"2024-03-28T18:55:47.116103+0000","last_became_active":"2024-03-28T18:55:45.621163+0000","last_became_peered":"2024-03-28T18:55:45.621163+0000","last_unstale":"2024-03-28T18:55:47.116103+0000","last_undegraded":"2024-03-28T18:55:47.116103+0000","last_fullsized":"2024-03-28T18:55:47.116103+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166307+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:55:47.166307+0000","last_peered":"2024-03-28T18:55:47.166307+0000","last_clean":"2024-03-28T18:55:47.166307+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:55:47.166307+0000","last_undegraded":"2024-03-28T18:55:47.166307+0000","last_fullsized":"2024-03-28T18:55:47.166307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.828245+0000","last_change":"2024-03-28T18:55:45.603774+0000","last_active":"2024-03-28T18:55:46.828245+0000","last_peered":"2024-03-28T18:55:46.828245+0000","last_clean":"2024-03-28T18:55:46.828245+0000","last_became_active":"2024-03-28T18:55:45.603451+0000","last_became_peered":"2024-03-28T18:55:45.603451+0000","last_unstale":"2024-03-28T18:55:46.828245+0000","last_undegraded":"2024-03-28T18:55:46.828245+0000","last_fullsized":"2024-03-28T18:55:46.828245+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.826131+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:55:46.826131+0000","last_peered":"2024-03-28T18:55:46.826131+0000","last_clean":"2024-03-28T18:55:46.826131+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:55:46.826131+0000","last_undegraded":"2024-03-28T18:55:46.826131+0000","last_fullsized":"2024-03-28T18:55:46.826131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.603950+0000","last_change":"2024-03-28T18:55:45.616755+0000","last_active":"2024-03-28T18:55:46.603950+0000","last_peered":"2024-03-28T18:55:46.603950+0000","last_clean":"2024-03-28T18:55:46.603950+0000","last_became_active":"2024-03-28T18:55:45.614611+0000","last_became_peered":"2024-03-28T18:55:45.614611+0000","last_unstale":"2024-03-28T18:55:46.603950+0000","last_undegraded":"2024-03-28T18:55:46.603950+0000","last_fullsized":"2024-03-28T18:55:46.603950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:06.327795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163860+0000","last_change":"2024-03-28T18:55:34.491855+0000","last_active":"2024-03-28T18:55:47.163860+0000","last_peered":"2024-03-28T18:55:47.163860+0000","last_clean":"2024-03-28T18:55:47.163860+0000","last_became_active":"2024-03-28T18:55:34.491070+0000","last_became_peered":"2024-03-28T18:55:34.491070+0000","last_unstale":"2024-03-28T18:55:47.163860+0000","last_undegraded":"2024-03-28T18:55:47.163860+0000","last_fullsized":"2024-03-28T18:55:47.163860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613597+0000","last_change":"2024-03-28T18:55:45.623195+0000","last_active":"2024-03-28T18:55:46.613597+0000","last_peered":"2024-03-28T18:55:46.613597+0000","last_clean":"2024-03-28T18:55:46.613597+0000","last_became_active":"2024-03-28T18:55:45.622508+0000","last_became_peered":"2024-03-28T18:55:45.622508+0000","last_unstale":"2024-03-28T18:55:46.613597+0000","last_undegraded":"2024-03-28T18:55:46.613597+0000","last_fullsized":"2024-03-28T18:55:46.613597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116530+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:55:47.116530+0000","last_peered":"2024-03-28T18:55:47.116530+0000","last_clean":"2024-03-28T18:55:47.116530+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:55:47.116530+0000","last_undegraded":"2024-03-28T18:55:47.116530+0000","last_fullsized":"2024-03-28T18:55:47.116530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.117584+0000","last_change":"2024-03-28T18:55:45.618841+0000","last_active":"2024-03-28T18:55:47.117584+0000","last_peered":"2024-03-28T18:55:47.117584+0000","last_clean":"2024-03-28T18:55:47.117584+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:55:47.117584+0000","last_undegraded":"2024-03-28T18:55:47.117584+0000","last_fullsized":"2024-03-28T18:55:47.117584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.164332+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:55:47.164332+0000","last_peered":"2024-03-28T18:55:47.164332+0000","last_clean":"2024-03-28T18:55:47.164332+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:55:47.164332+0000","last_undegraded":"2024-03-28T18:55:47.164332+0000","last_fullsized":"2024-03-28T18:55:47.164332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.825990+0000","last_change":"2024-03-28T18:55:45.602886+0000","last_active":"2024-03-28T18:55:46.825990+0000","last_peered":"2024-03-28T18:55:46.825990+0000","last_clean":"2024-03-28T18:55:46.825990+0000","last_became_active":"2024-03-28T18:55:45.602610+0000","last_became_peered":"2024-03-28T18:55:45.602610+0000","last_unstale":"2024-03-28T18:55:46.825990+0000","last_undegraded":"2024-03-28T18:55:46.825990+0000","last_fullsized":"2024-03-28T18:55:46.825990+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.116836+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:55:47.116836+0000","last_peered":"2024-03-28T18:55:47.116836+0000","last_clean":"2024-03-28T18:55:47.116836+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:55:47.116836+0000","last_undegraded":"2024-03-28T18:55:47.116836+0000","last_fullsized":"2024-03-28T18:55:47.116836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.602814+0000","last_change":"2024-03-28T18:55:45.608673+0000","last_active":"2024-03-28T18:55:46.602814+0000","last_peered":"2024-03-28T18:55:46.602814+0000","last_clean":"2024-03-28T18:55:46.602814+0000","last_became_active":"2024-03-28T18:55:45.608376+0000","last_became_peered":"2024-03-28T18:55:45.608376+0000","last_unstale":"2024-03-28T18:55:46.602814+0000","last_undegraded":"2024-03-28T18:55:46.602814+0000","last_fullsized":"2024-03-28T18:55:46.602814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:31.781549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.608764+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:55:46.608764+0000","last_peered":"2024-03-28T18:55:46.608764+0000","last_clean":"2024-03-28T18:55:46.608764+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:55:46.608764+0000","last_undegraded":"2024-03-28T18:55:46.608764+0000","last_fullsized":"2024-03-28T18:55:46.608764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163497+0000","last_change":"2024-03-28T18:55:45.609565+0000","last_active":"2024-03-28T18:55:47.163497+0000","last_peered":"2024-03-28T18:55:47.163497+0000","last_clean":"2024-03-28T18:55:47.163497+0000","last_became_active":"2024-03-28T18:55:45.609082+0000","last_became_peered":"2024-03-28T18:55:45.609082+0000","last_unstale":"2024-03-28T18:55:47.163497+0000","last_undegraded":"2024-03-28T18:55:47.163497+0000","last_fullsized":"2024-03-28T18:55:47.163497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:46.613540+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:55:46.613540+0000","last_peered":"2024-03-28T18:55:46.613540+0000","last_clean":"2024-03-28T18:55:46.613540+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:55:46.613540+0000","last_undegraded":"2024-03-28T18:55:46.613540+0000","last_fullsized":"2024-03-28T18:55:46.613540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.163662+0000","last_change":"2024-03-28T18:55:45.618585+0000","last_active":"2024-03-28T18:55:47.163662+0000","last_peered":"2024-03-28T18:55:47.163662+0000","last_clean":"2024-03-28T18:55:47.163662+0000","last_became_active":"2024-03-28T18:55:45.617895+0000","last_became_peered":"2024-03-28T18:55:45.617895+0000","last_unstale":"2024-03-28T18:55:47.163662+0000","last_undegraded":"2024-03-28T18:55:47.163662+0000","last_fullsized":"2024-03-28T18:55:47.163662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:54.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-28T18:55:47.166181+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:55:47.166181+0000","last_peered":"2024-03-28T18:55:47.166181+0000","last_clean":"2024-03-28T18:55:47.166181+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:55:47.166181+0000","last_undegraded":"2024-03-28T18:55:47.166181+0000","last_fullsized":"2024-03-28T18:55:47.166181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":856064,"data_stored":690301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":101489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":99491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":102507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":99331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":101055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:55:57.666 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T18:55:57.666 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T18:55:57.666 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-28T18:55:57.666 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-28T18:55:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.976+0000 7f25b5a00640 1 Processor -- start 2024-03-28T18:55:57.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.976+0000 7f25b5a00640 1 -- start start 2024-03-28T18:55:57.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.978+0000 7f25b5a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f25b00c58c0 0x7f25b00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.978+0000 7f25b5a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c8ed0 0x7f25b01555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.978+0000 7f25b5a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b0156230 con 0x7f25b00be5d0 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.978+0000 7f25b5a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b0158bd0 con 0x7f25b00c58c0 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.978+0000 7f25b5a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b0158eb0 con 0x7f25b00c8ed0 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.979+0000 7f25af400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c8ed0 0x7f25b01555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.979+0000 7f25af400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c8ed0 0x7f25b01555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:59352/0 (socket says 172.21.15.27:59352) 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.979+0000 7f25af400640 1 -- 172.21.15.27:0/1872568044 learned_addr learned my addr 172.21.15.27:0/1872568044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.979+0000 7f25b5000640 1 --1- 172.21.15.27:0/1872568044 >> v1:172.21.15.155:6789/0 conn(0x7f25b00be5d0 0x7f25b0155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:36114/0 (socket says 172.21.15.27:36114) 2024-03-28T18:55:57.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.979+0000 7f25afe00640 1 --2- 172.21.15.27:0/1872568044 >> v2:172.21.15.155:3300/0 conn(0x7f25b00c58c0 0x7f25b00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.980+0000 7f25af400640 1 -- 172.21.15.27:0/1872568044 >> v2:172.21.15.155:3300/0 conn(0x7f25b00c58c0 msgr2=0x7f25b00c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:57.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.980+0000 7f25af400640 1 --2- 172.21.15.27:0/1872568044 >> v2:172.21.15.155:3300/0 conn(0x7f25b00c58c0 0x7f25b00c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.980+0000 7f25af400640 1 -- 172.21.15.27:0/1872568044 >> v1:172.21.15.155:6789/0 conn(0x7f25b00be5d0 legacy=0x7f25b0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.980+0000 7f25af400640 1 -- 172.21.15.27:0/1872568044 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b0159190 con 0x7f25b00c8ed0 2024-03-28T18:55:57.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.981+0000 7f25af400640 1 --2- 172.21.15.27:0/1872568044 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c8ed0 0x7f25b01555e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f259c009870 tx=0x7f259c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87d170aeef367672 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:57.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.981+0000 7f25aea00640 1 -- 172.21.15.27:0/1872568044 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c035020 con 0x7f25b00c8ed0 2024-03-28T18:55:57.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.982+0000 7f25aea00640 1 -- 172.21.15.27:0/1872568044 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c031640 con 0x7f25b00c8ed0 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.982+0000 7f25aea00640 1 -- 172.21.15.27:0/1872568044 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c02d040 con 0x7f25b00c8ed0 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.982+0000 7f25b5a00640 1 -- 172.21.15.27:0/1872568044 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c8ed0 msgr2=0x7f25b01555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.982+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1872568044 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c8ed0 0x7f25b01555e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f259c009870 tx=0x7f259c02b8f0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.982+0000 7f25b5a00640 1 -- 172.21.15.27:0/1872568044 shutdown_connections 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.982+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1872568044 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c8ed0 0x7f25b01555e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.982+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1872568044 >> v2:172.21.15.155:3300/0 conn(0x7f25b00c58c0 0x7f25b00c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.982+0000 7f25b5a00640 1 -- 172.21.15.27:0/1872568044 >> 172.21.15.27:0/1872568044 conn(0x7f25b006b6f0 msgr2=0x7f25b00acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.983+0000 7f25b5a00640 1 -- 172.21.15.27:0/1872568044 shutdown_connections 2024-03-28T18:55:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.983+0000 7f25b5a00640 1 -- 172.21.15.27:0/1872568044 wait complete. 2024-03-28T18:55:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.984+0000 7f25b5a00640 1 Processor -- start 2024-03-28T18:55:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.984+0000 7f25b5a00640 1 -- start start 2024-03-28T18:55:57.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.985+0000 7f25b5a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00be5d0 0x7f25b0155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.985+0000 7f25b5a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c58c0 0x7f25b00c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.985+0000 7f25b5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25b00c8ed0 0x7f25b00c0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.985+0000 7f25b5a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00b0820 con 0x7f25b00be5d0 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.985+0000 7f25b5a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b0159750 con 0x7f25b00c8ed0 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.985+0000 7f25b5a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00b09a0 con 0x7f25b00c58c0 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00be5d0 0x7f25b0155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00be5d0 0x7f25b0155140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46818/0 (socket says 172.21.15.27:46818) 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25afe00640 1 -- 172.21.15.27:0/1374966461 learned_addr learned my addr 172.21.15.27:0/1374966461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25af400640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c58c0 0x7f25b00c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25b5000640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25b00c8ed0 0x7f25b00c0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25afe00640 1 -- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c58c0 msgr2=0x7f25b00c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25afe00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c58c0 0x7f25b00c4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25afe00640 1 -- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25b00c8ed0 msgr2=0x7f25b00c0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25afe00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25b00c8ed0 0x7f25b00c0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25afe00640 1 -- 172.21.15.27:0/1374966461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b0155d40 con 0x7f25b00be5d0 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25af400640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c58c0 0x7f25b00c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.986+0000 7f25b5000640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25b00c8ed0 0x7f25b00c0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.987+0000 7f25afe00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00be5d0 0x7f25b0155140 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f25a000e8a0 tx=0x7f25a000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.987+0000 7f25a7e00640 1 -- 172.21.15.27:0/1374966461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0017070 con 0x7f25b00be5d0 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.987+0000 7f25a7e00640 1 -- 172.21.15.27:0/1374966461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a000c940 con 0x7f25b00be5d0 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.987+0000 7f25a7e00640 1 -- 172.21.15.27:0/1374966461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a000cd30 con 0x7f25b00be5d0 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.987+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b0155a60 con 0x7f25b00be5d0 2024-03-28T18:55:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.987+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b01ec900 con 0x7f25b00be5d0 2024-03-28T18:55:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.988+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b0072c20 con 0x7f25b00be5d0 2024-03-28T18:55:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.989+0000 7f25a7e00640 1 -- 172.21.15.27:0/1374966461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f25a001a070 con 0x7f25b00be5d0 2024-03-28T18:55:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.990+0000 7f25a7e00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f259006bbd0 0x7f259006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.990+0000 7f25a7e00640 1 -- 172.21.15.27:0/1374966461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f25a008ad10 con 0x7f25b00be5d0 2024-03-28T18:55:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.990+0000 7f25af400640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f259006bbd0 0x7f259006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.991+0000 7f25af400640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f259006bbd0 0x7f259006e090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f259c036000 tx=0x7f259c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:57.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:57.995+0000 7f25a7e00640 1 -- 172.21.15.27:0/1374966461 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f25b0072c20 con 0x7f25b00be5d0 2024-03-28T18:55:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.236+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f25b00ca4a0 con 0x7f25b00be5d0 2024-03-28T18:55:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.237+0000 7f25a7e00640 1 -- 172.21.15.27:0/1374966461 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f25b00ca4a0 con 0x7f25b00be5d0 2024-03-28T18:55:58.237 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:58.237 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T18:55:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f259006bbd0 msgr2=0x7f259006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f259006bbd0 0x7f259006e090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f259c036000 tx=0x7f259c03c040 comp rx=0 tx=0).stop 2024-03-28T18:55:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00be5d0 msgr2=0x7f25b0155140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00be5d0 0x7f25b0155140 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f25a000e8a0 tx=0x7f25a000ed60 comp rx=0 tx=0).stop 2024-03-28T18:55:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 shutdown_connections 2024-03-28T18:55:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f259006bbd0 0x7f259006e090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25b00c8ed0 0x7f25b00c0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25b00c58c0 0x7f25b00c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 --2- 172.21.15.27:0/1374966461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25b00be5d0 0x7f25b0155140 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.239+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 >> 172.21.15.27:0/1374966461 conn(0x7f25b006b6f0 msgr2=0x7f25b00ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.240+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 shutdown_connections 2024-03-28T18:55:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.240+0000 7f25b5a00640 1 -- 172.21.15.27:0/1374966461 wait complete. 2024-03-28T18:55:58.257 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-28T18:55:58.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:55:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.482+0000 7fa353e00640 1 Processor -- start 2024-03-28T18:55:58.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.482+0000 7fa353e00640 1 -- start start 2024-03-28T18:55:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.483+0000 7fa353e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa3540c58c0 0x7fa3540c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.483+0000 7fa353e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa354158bd0 con 0x7fa3540be5d0 2024-03-28T18:55:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.483+0000 7fa353e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa354158eb0 con 0x7fa3540c8ed0 2024-03-28T18:55:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.484+0000 7fa353e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa354159190 con 0x7fa3540c58c0 2024-03-28T18:55:58.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.484+0000 7fa353400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa3540be5d0 0x7fa354155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38138/0 (socket says 172.21.15.27:38138) 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.484+0000 7fa353400640 1 -- 172.21.15.27:0/3201487259 learned_addr learned my addr 172.21.15.27:0/3201487259 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.484+0000 7fa352a00640 1 --2- 172.21.15.27:0/3201487259 >> v2:172.21.15.155:3300/0 conn(0x7fa3540c58c0 0x7fa3540c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.484+0000 7fa351600640 1 -- 172.21.15.27:0/3201487259 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497527478 0 0) 0x7fa354158bd0 con 0x7fa3540be5d0 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.484+0000 7fa351600640 1 -- 172.21.15.27:0/3201487259 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa338003520 con 0x7fa3540be5d0 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.485+0000 7fa352a00640 1 -- 172.21.15.27:0/3201487259 >> v1:172.21.15.155:6789/0 conn(0x7fa3540c8ed0 legacy=0x7fa3541555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.485+0000 7fa352a00640 1 -- 172.21.15.27:0/3201487259 >> v1:172.21.15.27:6789/0 conn(0x7fa3540be5d0 legacy=0x7fa354155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.485+0000 7fa352a00640 1 -- 172.21.15.27:0/3201487259 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa354159470 con 0x7fa3540c58c0 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.486+0000 7fa352a00640 1 --2- 172.21.15.27:0/3201487259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c58c0 0x7fa3540c5ca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa3440097a0 tx=0x7fa34402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e21e23f0c92456b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:58.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.487+0000 7fa351600640 1 -- 172.21.15.27:0/3201487259 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440315a0 con 0x7fa3540c58c0 2024-03-28T18:55:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.487+0000 7fa351600640 1 -- 172.21.15.27:0/3201487259 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa344037070 con 0x7fa3540c58c0 2024-03-28T18:55:58.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.487+0000 7fa351600640 1 -- 172.21.15.27:0/3201487259 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344031940 con 0x7fa3540c58c0 2024-03-28T18:55:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.488+0000 7fa353e00640 1 -- 172.21.15.27:0/3201487259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c58c0 msgr2=0x7fa3540c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.488+0000 7fa353e00640 1 --2- 172.21.15.27:0/3201487259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c58c0 0x7fa3540c5ca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa3440097a0 tx=0x7fa34402b840 comp rx=0 tx=0).stop 2024-03-28T18:55:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.488+0000 7fa353e00640 1 -- 172.21.15.27:0/3201487259 shutdown_connections 2024-03-28T18:55:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.488+0000 7fa353e00640 1 --2- 172.21.15.27:0/3201487259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c58c0 0x7fa3540c5ca0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.488+0000 7fa353e00640 1 -- 172.21.15.27:0/3201487259 >> 172.21.15.27:0/3201487259 conn(0x7fa35406b6f0 msgr2=0x7fa3540acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.488+0000 7fa353e00640 1 -- 172.21.15.27:0/3201487259 shutdown_connections 2024-03-28T18:55:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.488+0000 7fa353e00640 1 -- 172.21.15.27:0/3201487259 wait complete. 2024-03-28T18:55:58.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.491+0000 7fa353e00640 1 Processor -- start 2024-03-28T18:55:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.491+0000 7fa353e00640 1 -- start start 2024-03-28T18:55:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa353e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3540be5d0 0x7fa3540c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa353e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3540c58c0 0x7fa3541f0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa353e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c8ed0 0x7fa3541e03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa353e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540b09a0 con 0x7fa3540c58c0 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa353e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540b0820 con 0x7fa3540c8ed0 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa353e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540ad7d0 con 0x7fa3540be5d0 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa352000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3540c58c0 0x7fa3541f0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa352000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3540c58c0 0x7fa3541f0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46830/0 (socket says 172.21.15.27:46830) 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa352000640 1 -- 172.21.15.27:0/1230879267 learned_addr learned my addr 172.21.15.27:0/1230879267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.492+0000 7fa352a00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3540be5d0 0x7fa3540c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.493+0000 7fa353400640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c8ed0 0x7fa3541e03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.493+0000 7fa352000640 1 -- 172.21.15.27:0/1230879267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3540be5d0 msgr2=0x7fa3540c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.493+0000 7fa352000640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3540be5d0 0x7fa3540c4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.493+0000 7fa352000640 1 -- 172.21.15.27:0/1230879267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c8ed0 msgr2=0x7fa3541e03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.493+0000 7fa352000640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c8ed0 0x7fa3541e03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.493+0000 7fa352000640 1 -- 172.21.15.27:0/1230879267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3541e15f0 con 0x7fa3540c58c0 2024-03-28T18:55:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.493+0000 7fa352000640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3540c58c0 0x7fa3541f0130 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa34000c3c0 tx=0x7fa34000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.493+0000 7fa34aa00640 1 -- 172.21.15.27:0/1230879267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340017070 con 0x7fa3540c58c0 2024-03-28T18:55:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.494+0000 7fa34aa00640 1 -- 172.21.15.27:0/1230879267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3400052f0 con 0x7fa3540c58c0 2024-03-28T18:55:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.494+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3541e18d0 con 0x7fa3540c58c0 2024-03-28T18:55:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.494+0000 7fa34aa00640 1 -- 172.21.15.27:0/1230879267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340003830 con 0x7fa3540c58c0 2024-03-28T18:55:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.494+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3541e08e0 con 0x7fa3540c58c0 2024-03-28T18:55:58.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.495+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa354069640 con 0x7fa3540c58c0 2024-03-28T18:55:58.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.496+0000 7fa34aa00640 1 -- 172.21.15.27:0/1230879267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa34001a070 con 0x7fa3540c58c0 2024-03-28T18:55:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.497+0000 7fa34aa00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa33406bbd0 0x7fa33406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.498+0000 7fa34aa00640 1 -- 172.21.15.27:0/1230879267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fa34008a1c0 con 0x7fa3540c58c0 2024-03-28T18:55:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.498+0000 7fa352a00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa33406bbd0 0x7fa33406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.499+0000 7fa352a00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa33406bbd0 0x7fa33406e090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa344036000 tx=0x7fa344008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:58.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.503+0000 7fa34aa00640 1 -- 172.21.15.27:0/1230879267 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa3400588f0 con 0x7fa3540c58c0 2024-03-28T18:55:58.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.705+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3541e1aa0 con 0x7fa3540c58c0 2024-03-28T18:55:58.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.708+0000 7fa34aa00640 1 -- 172.21.15.27:0/1230879267 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+20307 (secure 0 0 0) 0x7fa3400581f0 con 0x7fa3540c58c0 2024-03-28T18:55:58.708 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:55:58.708 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:46.587251+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:55:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.710+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa33406bbd0 msgr2=0x7fa33406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.710+0000 7fa353e00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa33406bbd0 0x7fa33406e090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa344036000 tx=0x7fa344008030 comp rx=0 tx=0).stop 2024-03-28T18:55:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.710+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3540c58c0 msgr2=0x7fa3541f0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.710+0000 7fa353e00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3540c58c0 0x7fa3541f0130 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa34000c3c0 tx=0x7fa34000c880 comp rx=0 tx=0).stop 2024-03-28T18:55:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.711+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 shutdown_connections 2024-03-28T18:55:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.711+0000 7fa353e00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa33406bbd0 0x7fa33406e090 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.711+0000 7fa353e00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3540c8ed0 0x7fa3541e03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.711+0000 7fa353e00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3540c58c0 0x7fa3541f0130 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.711+0000 7fa353e00640 1 --2- 172.21.15.27:0/1230879267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3540be5d0 0x7fa3540c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.711+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 >> 172.21.15.27:0/1230879267 conn(0x7fa35406b6f0 msgr2=0x7fa3541e1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.711+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 shutdown_connections 2024-03-28T18:55:58.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.711+0000 7fa353e00640 1 -- 172.21.15.27:0/1230879267 wait complete. 2024-03-28T18:55:58.730 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-03-28T18:55:32.332283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-03-28T18:55:32.983411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-03-28T18:55:43.415397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-03-28T18:55:44.084162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:55:58.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:55:58.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.951+0000 7fe922200640 1 Processor -- start 2024-03-28T18:55:58.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.951+0000 7fe922200640 1 -- start start 2024-03-28T18:55:58.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.953+0000 7fe922200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe91c155790 0x7fe91c155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.953+0000 7fe922200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c158d90 con 0x7fe91c0c9050 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.953+0000 7fe922200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c159070 con 0x7fe91c152ba0 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.953+0000 7fe922200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c159350 con 0x7fe91c155790 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.953+0000 7fe91be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe91c0c9050 0x7fe91c0cb420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38152/0 (socket says 172.21.15.27:38152) 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.953+0000 7fe91be00640 1 -- 172.21.15.27:0/712035370 learned_addr learned my addr 172.21.15.27:0/712035370 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe921800640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe91c155790 0x7fe91c155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708661131 0 0) 0x7fe91c158d90 con 0x7fe91c0c9050 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe904003520 con 0x7fe91c0c9050 2024-03-28T18:55:58.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3020303323 0 0) 0x7fe904003520 con 0x7fe91c0c9050 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe91c155790 msgr2=0x7fe91c155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe91c155790 0x7fe91c155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 >> v1:172.21.15.155:6789/0 conn(0x7fe91c152ba0 legacy=0x7fe91c0726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c159630 con 0x7fe91c0c9050 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe908001670 con 0x7fe91c0c9050 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 >> v1:172.21.15.27:6789/0 conn(0x7fe91c0c9050 legacy=0x7fe91c0cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9040039e0 0x7fe904003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe91c155790 0x7fe904004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.954+0000 7fe91b400640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904004d40 0x7fe9040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c159350 con 0x7fe9040039e0 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90400bb50 con 0x7fe904004d40 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90400bd50 con 0x7fe91c155790 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe920e00640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9040039e0 0x7fe904003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe91be00640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe91c155790 0x7fe904004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe921800640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904004d40 0x7fe9040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe920e00640 1 -- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe91c155790 msgr2=0x7fe904004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe920e00640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe91c155790 0x7fe904004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe920e00640 1 -- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904004d40 msgr2=0x7fe9040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:58.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe920e00640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904004d40 0x7fe9040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.955+0000 7fe920e00640 1 -- 172.21.15.27:0/712035370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe904003520 con 0x7fe9040039e0 2024-03-28T18:55:58.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.957+0000 7fe920e00640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9040039e0 0x7fe904003dc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe9100018b0 tx=0x7fe904005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f9895bc16c1fc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:58.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.958+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe910038070 con 0x7fe9040039e0 2024-03-28T18:55:58.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.958+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe910033650 con 0x7fe9040039e0 2024-03-28T18:55:58.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.958+0000 7fe91b400640 1 -- 172.21.15.27:0/712035370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91002f050 con 0x7fe9040039e0 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.958+0000 7fe922200640 1 -- 172.21.15.27:0/712035370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9040039e0 msgr2=0x7fe904003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.958+0000 7fe922200640 1 --2- 172.21.15.27:0/712035370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9040039e0 0x7fe904003dc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe9100018b0 tx=0x7fe904005830 comp rx=0 tx=0).stop 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.958+0000 7fe921800640 1 -- 172.21.15.27:0/712035370 reap_dead start 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.959+0000 7fe922200640 1 -- 172.21.15.27:0/712035370 shutdown_connections 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.959+0000 7fe922200640 1 -- 172.21.15.27:0/712035370 >> 172.21.15.27:0/712035370 conn(0x7fe91c06b4e0 msgr2=0x7fe91c06b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.959+0000 7fe922200640 1 -- 172.21.15.27:0/712035370 shutdown_connections 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.959+0000 7fe922200640 1 -- 172.21.15.27:0/712035370 wait complete. 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.960+0000 7fe922200640 1 Processor -- start 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.960+0000 7fe922200640 1 -- start start 2024-03-28T18:55:58.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.961+0000 7fe922200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9040039e0 0x7fe91c154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.961+0000 7fe922200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91c0adb30 0x7fe91c0b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.961+0000 7fe922200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91c14b1b0 0x7fe91c14b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.961+0000 7fe922200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c0cbcb0 con 0x7fe91c14b1b0 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.961+0000 7fe922200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c0cbb30 con 0x7fe91c0adb30 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.961+0000 7fe922200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c152f80 con 0x7fe9040039e0 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91c14b1b0 0x7fe91c14b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91c14b1b0 0x7fe91c14b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46848/0 (socket says 172.21.15.27:46848) 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 -- 172.21.15.27:0/1255494513 learned_addr learned my addr 172.21.15.27:0/1255494513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:58.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe920e00640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9040039e0 0x7fe91c154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe91be00640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91c0adb30 0x7fe91c0b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 -- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9040039e0 msgr2=0x7fe91c154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9040039e0 0x7fe91c154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 -- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91c0adb30 msgr2=0x7fe91c0b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91c0adb30 0x7fe91c0b05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 -- 172.21.15.27:0/1255494513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c0af0f0 con 0x7fe91c14b1b0 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe920e00640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9040039e0 0x7fe91c154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.962+0000 7fe921800640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91c14b1b0 0x7fe91c14b660 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe91c062b40 tx=0x7fe9140115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:58.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.963+0000 7fe918c00640 1 -- 172.21.15.27:0/1255494513 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914011e10 con 0x7fe91c14b1b0 2024-03-28T18:55:58.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.963+0000 7fe918c00640 1 -- 172.21.15.27:0/1255494513 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91401b070 con 0x7fe91c14b1b0 2024-03-28T18:55:58.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.963+0000 7fe918c00640 1 -- 172.21.15.27:0/1255494513 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914003990 con 0x7fe91c14b1b0 2024-03-28T18:55:58.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.963+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c14bba0 con 0x7fe91c14b1b0 2024-03-28T18:55:58.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.963+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c0ae030 con 0x7fe91c14b1b0 2024-03-28T18:55:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.964+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91c04f400 con 0x7fe91c14b1b0 2024-03-28T18:55:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.966+0000 7fe918c00640 1 -- 172.21.15.27:0/1255494513 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe914005d10 con 0x7fe91c14b1b0 2024-03-28T18:55:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.967+0000 7fe918c00640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8fc06bc00 0x7fe8fc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.967+0000 7fe918c00640 1 -- 172.21.15.27:0/1255494513 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fe91408afc0 con 0x7fe91c14b1b0 2024-03-28T18:55:58.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.967+0000 7fe920e00640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8fc06bc00 0x7fe8fc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:58.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.968+0000 7fe920e00640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8fc06bc00 0x7fe8fc06e0c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe910035000 tx=0x7fe910005040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:58.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:58.971+0000 7fe918c00640 1 -- 172.21.15.27:0/1255494513 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe914014350 con 0x7fe91c14b1b0 2024-03-28T18:55:59.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.239+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe91c0c8e30 con 0x7fe91c14b1b0 2024-03-28T18:55:59.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.240+0000 7fe918c00640 1 -- 172.21.15.27:0/1255494513 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe914059810 con 0x7fe91c14b1b0 2024-03-28T18:55:59.240 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8fc06bc00 msgr2=0x7fe8fc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8fc06bc00 0x7fe8fc06e0c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe910035000 tx=0x7fe910005040 comp rx=0 tx=0).stop 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91c14b1b0 msgr2=0x7fe91c14b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91c14b1b0 0x7fe91c14b660 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe91c062b40 tx=0x7fe9140115a0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 shutdown_connections 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8fc06bc00 0x7fe8fc06e0c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91c14b1b0 0x7fe91c14b660 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe91c0adb30 0x7fe91c0b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 --2- 172.21.15.27:0/1255494513 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9040039e0 0x7fe91c154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 >> 172.21.15.27:0/1255494513 conn(0x7fe91c06b4e0 msgr2=0x7fe91c151320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:59.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 shutdown_connections 2024-03-28T18:55:59.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.243+0000 7fe922200640 1 -- 172.21.15.27:0/1255494513 wait complete. 2024-03-28T18:55:59.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-03-28T18:55:59.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.488+0000 7fbda0c00640 1 Processor -- start 2024-03-28T18:55:59.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.488+0000 7fbda0c00640 1 -- start start 2024-03-28T18:55:59.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.489+0000 7fbda0c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd9c066dd0 0x7fbd9c0671b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.489+0000 7fbda0c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbd9c0a6f20 0x7fbd9c1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.489+0000 7fbda0c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c153720 con 0x7fbd9c072b80 2024-03-28T18:55:59.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.489+0000 7fbda0c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c154720 con 0x7fbd9c066dd0 2024-03-28T18:55:59.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.489+0000 7fbda0c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c151130 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.490+0000 7fbd99e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd9c066dd0 0x7fbd9c0671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.490+0000 7fbd99e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd9c066dd0 0x7fbd9c0671b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46854/0 (socket says 172.21.15.27:46854) 2024-03-28T18:55:59.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.490+0000 7fbd99e00640 1 -- 172.21.15.27:0/137338029 learned_addr learned my addr 172.21.15.27:0/137338029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:59.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.490+0000 7fbd99e00640 1 -- 172.21.15.27:0/137338029 >> v2:172.21.15.155:3300/0 conn(0x7fbd9c0a6f20 msgr2=0x7fbd9c1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:59.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.490+0000 7fbd99e00640 1 --2- 172.21.15.27:0/137338029 >> v2:172.21.15.155:3300/0 conn(0x7fbd9c0a6f20 0x7fbd9c1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.490+0000 7fbd99e00640 1 -- 172.21.15.27:0/137338029 >> v1:172.21.15.155:6789/0 conn(0x7fbd9c072b80 legacy=0x7fbd9c0c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:59.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.490+0000 7fbd99e00640 1 -- 172.21.15.27:0/137338029 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c0a73b0 con 0x7fbd9c066dd0 2024-03-28T18:55:59.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.492+0000 7fbd99e00640 1 --2- 172.21.15.27:0/137338029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c066dd0 0x7fbd9c0671b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0737c0 tx=0x7fbd8c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed38461c5e289f34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:59.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.492+0000 7fbd99400640 1 -- 172.21.15.27:0/137338029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c02e070 con 0x7fbd9c066dd0 2024-03-28T18:55:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.492+0000 7fbd99400640 1 -- 172.21.15.27:0/137338029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c037070 con 0x7fbd9c066dd0 2024-03-28T18:55:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.492+0000 7fbd99400640 1 -- 172.21.15.27:0/137338029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c031a00 con 0x7fbd9c066dd0 2024-03-28T18:55:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.493+0000 7fbda0c00640 1 -- 172.21.15.27:0/137338029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c066dd0 msgr2=0x7fbd9c0671b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.493+0000 7fbda0c00640 1 --2- 172.21.15.27:0/137338029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c066dd0 0x7fbd9c0671b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0737c0 tx=0x7fbd8c02b880 comp rx=0 tx=0).stop 2024-03-28T18:55:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.493+0000 7fbda0c00640 1 -- 172.21.15.27:0/137338029 shutdown_connections 2024-03-28T18:55:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.493+0000 7fbda0c00640 1 --2- 172.21.15.27:0/137338029 >> v2:172.21.15.155:3300/0 conn(0x7fbd9c0a6f20 0x7fbd9c1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.493+0000 7fbda0c00640 1 --2- 172.21.15.27:0/137338029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c066dd0 0x7fbd9c0671b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.493+0000 7fbda0c00640 1 -- 172.21.15.27:0/137338029 >> 172.21.15.27:0/137338029 conn(0x7fbd9c06b0c0 msgr2=0x7fbd9c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.493+0000 7fbda0c00640 1 -- 172.21.15.27:0/137338029 shutdown_connections 2024-03-28T18:55:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.493+0000 7fbda0c00640 1 -- 172.21.15.27:0/137338029 wait complete. 2024-03-28T18:55:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.495+0000 7fbda0c00640 1 Processor -- start 2024-03-28T18:55:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.495+0000 7fbda0c00640 1 -- start start 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.495+0000 7fbda0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd9c066dd0 0x7fbd9c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbda0c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd9c072b80 0x7fbd9c1eb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbda0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c0a6f20 0x7fbd9c1eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbda0c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c1539e0 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbda0c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c157aa0 con 0x7fbd9c072b80 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbda0c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c153cc0 con 0x7fbd9c066dd0 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd9a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c0a6f20 0x7fbd9c1eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd9a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c0a6f20 0x7fbd9c1eeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46868/0 (socket says 172.21.15.27:46868) 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd9a800640 1 -- 172.21.15.27:0/254571057 learned_addr learned my addr 172.21.15.27:0/254571057 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:59.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd99e00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd9c066dd0 0x7fbd9c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd91400640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd9c072b80 0x7fbd9c1eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd9a800640 1 -- 172.21.15.27:0/254571057 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd9c066dd0 msgr2=0x7fbd9c06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd9a800640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd9c066dd0 0x7fbd9c06e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd9a800640 1 -- 172.21.15.27:0/254571057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd9c072b80 msgr2=0x7fbd9c1eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd9a800640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd9c072b80 0x7fbd9c1eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.496+0000 7fbd9a800640 1 -- 172.21.15.27:0/254571057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c1570f0 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.497+0000 7fbd9a800640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c0a6f20 0x7fbd9c1eeb90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400b970 tx=0x7fbd9400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:59.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.497+0000 7fbd92a00640 1 -- 172.21.15.27:0/254571057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94019070 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.497+0000 7fbd92a00640 1 -- 172.21.15.27:0/254571057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9400cb60 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.497+0000 7fbd92a00640 1 -- 172.21.15.27:0/254571057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94014440 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.497+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c1df5a0 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.497+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c06eb90 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.499+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c155af0 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.499+0000 7fbd92a00640 1 -- 172.21.15.27:0/254571057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbd9401c070 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.500+0000 7fbd92a00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7806bbd0 0x7fbd7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.500+0000 7fbd92a00640 1 -- 172.21.15.27:0/254571057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fbd9408ab10 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.501+0000 7fbd99e00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7806bbd0 0x7fbd7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.501+0000 7fbd99e00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7806bbd0 0x7fbd7806e090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c036000 tx=0x7fbd8c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:59.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.506+0000 7fbd92a00640 1 -- 172.21.15.27:0/254571057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbd940592c0 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.708+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbd9c04f470 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.708+0000 7fbd92a00640 1 -- 172.21.15.27:0/254571057 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v24) v1 ==== 99+0+11 (secure 0 0 0) 0x7fbd94058bc0 con 0x7fbd9c0a6f20 2024-03-28T18:55:59.710 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:55:59.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7806bbd0 msgr2=0x7fbd7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7806bbd0 0x7fbd7806e090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c036000 tx=0x7fbd8c03d040 comp rx=0 tx=0).stop 2024-03-28T18:55:59.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c0a6f20 msgr2=0x7fbd9c1eeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c0a6f20 0x7fbd9c1eeb90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400b970 tx=0x7fbd9400be30 comp rx=0 tx=0).stop 2024-03-28T18:55:59.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 shutdown_connections 2024-03-28T18:55:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7806bbd0 0x7fbd7806e090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9c0a6f20 0x7fbd9c1eeb90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd9c072b80 0x7fbd9c1eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 --2- 172.21.15.27:0/254571057 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd9c066dd0 0x7fbd9c06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 >> 172.21.15.27:0/254571057 conn(0x7fbd9c06b0c0 msgr2=0x7fbd9c0c5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 shutdown_connections 2024-03-28T18:55:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.711+0000 7fbda0c00640 1 -- 172.21.15.27:0/254571057 wait complete. 2024-03-28T18:55:59.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-03-28T18:55:59.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.961+0000 7fb332e00640 1 Processor -- start 2024-03-28T18:55:59.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.962+0000 7fb332e00640 1 -- start start 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb332e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb32c14f920 0x7fb32c0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb332e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb32c0c8f30 0x7fb32c0cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb332e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32c06d960 con 0x7fb32c0cb8c0 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb332e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c1568e0 con 0x7fb32c0c8f30 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb332e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c156450 con 0x7fb32c14f920 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb330e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb32c0cb8c0 0x7fb32c158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38160/0 (socket says 172.21.15.27:38160) 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb330e00640 1 -- 172.21.15.27:0/1053096009 learned_addr learned my addr 172.21.15.27:0/1053096009 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb323e00640 1 --2- 172.21.15.27:0/1053096009 >> v2:172.21.15.27:3300/0 conn(0x7fb32c0c8f30 0x7fb32c0cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.963+0000 7fb32be00640 1 --2- 172.21.15.27:0/1053096009 >> v2:172.21.15.155:3300/0 conn(0x7fb32c14f920 0x7fb32c0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.964+0000 7fb32b400640 1 -- 172.21.15.27:0/1053096009 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466843405 0 0) 0x7fb32c06d960 con 0x7fb32c0cb8c0 2024-03-28T18:55:59.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.964+0000 7fb32b400640 1 -- 172.21.15.27:0/1053096009 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb310003520 con 0x7fb32c0cb8c0 2024-03-28T18:55:59.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.964+0000 7fb323e00640 1 -- 172.21.15.27:0/1053096009 >> v2:172.21.15.155:3300/0 conn(0x7fb32c14f920 msgr2=0x7fb32c0c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:55:59.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.964+0000 7fb323e00640 1 --2- 172.21.15.27:0/1053096009 >> v2:172.21.15.155:3300/0 conn(0x7fb32c14f920 0x7fb32c0c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.964+0000 7fb323e00640 1 -- 172.21.15.27:0/1053096009 >> v1:172.21.15.27:6789/0 conn(0x7fb32c0cb8c0 legacy=0x7fb32c158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.964+0000 7fb323e00640 1 -- 172.21.15.27:0/1053096009 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c155b20 con 0x7fb32c0c8f30 2024-03-28T18:55:59.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.965+0000 7fb323e00640 1 --2- 172.21.15.27:0/1053096009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c0c8f30 0x7fb32c0cb380 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0adc20 tx=0x7fb31402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e71a0245204368 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:59.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.966+0000 7fb32b400640 1 -- 172.21.15.27:0/1053096009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3140315a0 con 0x7fb32c0c8f30 2024-03-28T18:55:59.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.966+0000 7fb32b400640 1 -- 172.21.15.27:0/1053096009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb314037070 con 0x7fb32c0c8f30 2024-03-28T18:55:59.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.966+0000 7fb32b400640 1 -- 172.21.15.27:0/1053096009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314031960 con 0x7fb32c0c8f30 2024-03-28T18:55:59.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.967+0000 7fb332e00640 1 -- 172.21.15.27:0/1053096009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c0c8f30 msgr2=0x7fb32c0cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.967+0000 7fb332e00640 1 --2- 172.21.15.27:0/1053096009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c0c8f30 0x7fb32c0cb380 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0adc20 tx=0x7fb31402b840 comp rx=0 tx=0).stop 2024-03-28T18:55:59.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.967+0000 7fb332e00640 1 -- 172.21.15.27:0/1053096009 shutdown_connections 2024-03-28T18:55:59.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.967+0000 7fb332e00640 1 --2- 172.21.15.27:0/1053096009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c0c8f30 0x7fb32c0cb380 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.967+0000 7fb332e00640 1 --2- 172.21.15.27:0/1053096009 >> v2:172.21.15.155:3300/0 conn(0x7fb32c14f920 0x7fb32c0c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.967+0000 7fb332e00640 1 -- 172.21.15.27:0/1053096009 >> 172.21.15.27:0/1053096009 conn(0x7fb32c066fb0 msgr2=0x7fb32c0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:55:59.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.967+0000 7fb332e00640 1 -- 172.21.15.27:0/1053096009 shutdown_connections 2024-03-28T18:55:59.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.967+0000 7fb332e00640 1 -- 172.21.15.27:0/1053096009 wait complete. 2024-03-28T18:55:59.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.969+0000 7fb332e00640 1 Processor -- start 2024-03-28T18:55:59.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.970+0000 7fb332e00640 1 -- start start 2024-03-28T18:55:59.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb332e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb32c0c8f30 0x7fb32c0c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb332e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb32c0cb8c0 0x7fb32c0ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb332e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c14f920 0x7fb32c0ac7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb332e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c14fd00 con 0x7fb32c14f920 2024-03-28T18:55:59.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb332e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c1560e0 con 0x7fb32c0cb8c0 2024-03-28T18:55:59.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb332e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c159610 con 0x7fb32c0c8f30 2024-03-28T18:55:59.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb330e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c14f920 0x7fb32c0ac7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb330e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c14f920 0x7fb32c0ac7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46898/0 (socket says 172.21.15.27:46898) 2024-03-28T18:55:59.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb330e00640 1 -- 172.21.15.27:0/3969985982 learned_addr learned my addr 172.21.15.27:0/3969985982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:55:59.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb323e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb32c0cb8c0 0x7fb32c0ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.971+0000 7fb32be00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb32c0c8f30 0x7fb32c0c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.972+0000 7fb330e00640 1 -- 172.21.15.27:0/3969985982 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb32c0c8f30 msgr2=0x7fb32c0c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.972+0000 7fb330e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb32c0c8f30 0x7fb32c0c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.972+0000 7fb330e00640 1 -- 172.21.15.27:0/3969985982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb32c0cb8c0 msgr2=0x7fb32c0ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.972+0000 7fb330e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb32c0cb8c0 0x7fb32c0ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.972+0000 7fb330e00640 1 -- 172.21.15.27:0/3969985982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0ab1c0 con 0x7fb32c14f920 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.972+0000 7fb330e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c14f920 0x7fb32c0ac7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb32400cb40 tx=0x7fb32400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.973+0000 7fb328c00640 1 -- 172.21.15.27:0/3969985982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324017070 con 0x7fb32c14f920 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.973+0000 7fb328c00640 1 -- 172.21.15.27:0/3969985982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32400aca0 con 0x7fb32c14f920 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.973+0000 7fb328c00640 1 -- 172.21.15.27:0/3969985982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324012620 con 0x7fb32c14f920 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.973+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c0ab4a0 con 0x7fb32c14f920 2024-03-28T18:55:59.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.973+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c0aba60 con 0x7fb32c14f920 2024-03-28T18:55:59.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.974+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c04f470 con 0x7fb32c14f920 2024-03-28T18:55:59.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.975+0000 7fb328c00640 1 -- 172.21.15.27:0/3969985982 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb32401a070 con 0x7fb32c14f920 2024-03-28T18:55:59.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.977+0000 7fb328c00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb30806bc00 0x7fb30806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:55:59.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.977+0000 7fb328c00640 1 -- 172.21.15.27:0/3969985982 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fb32408ab60 con 0x7fb32c14f920 2024-03-28T18:55:59.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.977+0000 7fb32be00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb30806bc00 0x7fb30806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:55:59.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.978+0000 7fb32be00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb30806bc00 0x7fb30806e0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009aa0 tx=0x7fb31c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:55:59.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:55:59.981+0000 7fb328c00640 1 -- 172.21.15.27:0/3969985982 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb324058cc0 con 0x7fb32c14f920 2024-03-28T18:56:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.211+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7fb32c0c1df0 con 0x7fb32c14f920 2024-03-28T18:56:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.212+0000 7fb328c00640 1 -- 172.21.15.27:0/3969985982 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v24) v1 ==== 95+0+11 (secure 0 0 0) 0x7fb32405cd60 con 0x7fb32c14f920 2024-03-28T18:56:00.212 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:00.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.214+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb30806bc00 msgr2=0x7fb30806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.214+0000 7fb332e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb30806bc00 0x7fb30806e0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009aa0 tx=0x7fb31c009190 comp rx=0 tx=0).stop 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.214+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c14f920 msgr2=0x7fb32c0ac7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.214+0000 7fb332e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c14f920 0x7fb32c0ac7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb32400cb40 tx=0x7fb32400a060 comp rx=0 tx=0).stop 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.215+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 shutdown_connections 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.215+0000 7fb332e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb30806bc00 0x7fb30806e0c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.215+0000 7fb332e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb32c14f920 0x7fb32c0ac7a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.215+0000 7fb332e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb32c0cb8c0 0x7fb32c0ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.215+0000 7fb332e00640 1 --2- 172.21.15.27:0/3969985982 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb32c0c8f30 0x7fb32c0c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.215+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 >> 172.21.15.27:0/3969985982 conn(0x7fb32c066fb0 msgr2=0x7fb32c155260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.215+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 shutdown_connections 2024-03-28T18:56:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.215+0000 7fb332e00640 1 -- 172.21.15.27:0/3969985982 wait complete. 2024-03-28T18:56:00.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-03-28T18:56:00.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.459+0000 7fea9e400640 1 Processor -- start 2024-03-28T18:56:00.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.459+0000 7fea9e400640 1 -- start start 2024-03-28T18:56:00.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.460+0000 7fea9e400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 0x7fea980726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.460+0000 7fea9e400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea980c9050 0x7fea980cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.460+0000 7fea9e400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea98072c00 con 0x7fea980cb960 2024-03-28T18:56:00.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.460+0000 7fea9e400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea98158d90 con 0x7fea980c9050 2024-03-28T18:56:00.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.460+0000 7fea9e400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea98159070 con 0x7fea98152ba0 2024-03-28T18:56:00.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.461+0000 7fea9da00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fea980cb960 0x7fea98155790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38164/0 (socket says 172.21.15.27:38164) 2024-03-28T18:56:00.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.461+0000 7fea9da00640 1 -- 172.21.15.27:0/991193901 learned_addr learned my addr 172.21.15.27:0/991193901 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:00.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.461+0000 7fea97e00640 1 --2- 172.21.15.27:0/991193901 >> v2:172.21.15.155:3300/0 conn(0x7fea980c9050 0x7fea980cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.461+0000 7fea9d000640 1 --2- 172.21.15.27:0/991193901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 0x7fea980726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.461+0000 7fea97400640 1 -- 172.21.15.27:0/991193901 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100138496 0 0) 0x7fea98072c00 con 0x7fea980cb960 2024-03-28T18:56:00.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.461+0000 7fea97400640 1 -- 172.21.15.27:0/991193901 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea80003520 con 0x7fea980cb960 2024-03-28T18:56:00.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.462+0000 7fea97e00640 1 -- 172.21.15.27:0/991193901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 msgr2=0x7fea980726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.462+0000 7fea97e00640 1 --2- 172.21.15.27:0/991193901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 0x7fea980726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.462+0000 7fea97e00640 1 -- 172.21.15.27:0/991193901 >> v1:172.21.15.27:6789/0 conn(0x7fea980cb960 legacy=0x7fea98155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.462+0000 7fea97e00640 1 -- 172.21.15.27:0/991193901 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98159350 con 0x7fea980c9050 2024-03-28T18:56:00.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.463+0000 7fea97e00640 1 --2- 172.21.15.27:0/991193901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c9050 0x7fea980cb420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea880097a0 tx=0x7fea8802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c2189904b1920f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.463+0000 7fea97400640 1 -- 172.21.15.27:0/991193901 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88039070 con 0x7fea980c9050 2024-03-28T18:56:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.463+0000 7fea97400640 1 -- 172.21.15.27:0/991193901 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea880316f0 con 0x7fea980c9050 2024-03-28T18:56:00.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.463+0000 7fea97400640 1 -- 172.21.15.27:0/991193901 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88031aa0 con 0x7fea980c9050 2024-03-28T18:56:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.464+0000 7fea9e400640 1 -- 172.21.15.27:0/991193901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c9050 msgr2=0x7fea980cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.464+0000 7fea9e400640 1 --2- 172.21.15.27:0/991193901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c9050 0x7fea980cb420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea880097a0 tx=0x7fea8802b790 comp rx=0 tx=0).stop 2024-03-28T18:56:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.464+0000 7fea9e400640 1 -- 172.21.15.27:0/991193901 shutdown_connections 2024-03-28T18:56:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.464+0000 7fea9e400640 1 --2- 172.21.15.27:0/991193901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c9050 0x7fea980cb420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.464+0000 7fea9e400640 1 --2- 172.21.15.27:0/991193901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 0x7fea980726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.464+0000 7fea9e400640 1 -- 172.21.15.27:0/991193901 >> 172.21.15.27:0/991193901 conn(0x7fea9806b4e0 msgr2=0x7fea9806b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:00.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.464+0000 7fea9e400640 1 -- 172.21.15.27:0/991193901 shutdown_connections 2024-03-28T18:56:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.464+0000 7fea9e400640 1 -- 172.21.15.27:0/991193901 wait complete. 2024-03-28T18:56:00.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.466+0000 7fea9e400640 1 Processor -- start 2024-03-28T18:56:00.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.466+0000 7fea9e400640 1 -- start start 2024-03-28T18:56:00.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980c9050 0x7fea981ec160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9e400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980cb960 0x7fea981eb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9e400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 0x7fea98156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9e400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98152f80 con 0x7fea980c9050 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9e400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98159910 con 0x7fea980cb960 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9e400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea98153100 con 0x7fea98152ba0 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980c9050 0x7fea981ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980c9050 0x7fea981ec160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46906/0 (socket says 172.21.15.27:46906) 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.467+0000 7fea9d000640 1 -- 172.21.15.27:0/2935188480 learned_addr learned my addr 172.21.15.27:0/2935188480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea97e00640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980cb960 0x7fea981eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea9d000640 1 -- 172.21.15.27:0/2935188480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 msgr2=0x7fea98156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea9d000640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 0x7fea98156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea9d000640 1 -- 172.21.15.27:0/2935188480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980cb960 msgr2=0x7fea981eb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea9d000640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980cb960 0x7fea981eb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea9d000640 1 -- 172.21.15.27:0/2935188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea981ebc50 con 0x7fea980c9050 2024-03-28T18:56:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea97e00640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980cb960 0x7fea981eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea9d000640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980c9050 0x7fea981ec160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00e930 tx=0x7fea8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.468+0000 7fea94c00640 1 -- 172.21.15.27:0/2935188480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c017070 con 0x7fea980c9050 2024-03-28T18:56:00.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.469+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea980b08c0 con 0x7fea980c9050 2024-03-28T18:56:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.469+0000 7fea94c00640 1 -- 172.21.15.27:0/2935188480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c00ca00 con 0x7fea980c9050 2024-03-28T18:56:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.469+0000 7fea94c00640 1 -- 172.21.15.27:0/2935188480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c00cdb0 con 0x7fea980c9050 2024-03-28T18:56:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.469+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea980b05e0 con 0x7fea980c9050 2024-03-28T18:56:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.470+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea980c8770 con 0x7fea980c9050 2024-03-28T18:56:00.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.471+0000 7fea94c00640 1 -- 172.21.15.27:0/2935188480 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fea8c01a070 con 0x7fea980c9050 2024-03-28T18:56:00.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.472+0000 7fea94c00640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7c06bbd0 0x7fea7c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.472+0000 7fea94c00640 1 -- 172.21.15.27:0/2935188480 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fea8c08b1f0 con 0x7fea980c9050 2024-03-28T18:56:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.472+0000 7fea97e00640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7c06bbd0 0x7fea7c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.473+0000 7fea97e00640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7c06bbd0 0x7fea7c06e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea88036000 tx=0x7fea88008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.481+0000 7fea94c00640 1 -- 172.21.15.27:0/2935188480 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fea980c8770 con 0x7fea980c9050 2024-03-28T18:56:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.721+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7fea9804f400 con 0x7fea980c9050 2024-03-28T18:56:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.722+0000 7fea94c00640 1 -- 172.21.15.27:0/2935188480 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v24) v1 ==== 99+0+11 (secure 0 0 0) 0x7fea981ebc50 con 0x7fea980c9050 2024-03-28T18:56:00.721 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7c06bbd0 msgr2=0x7fea7c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7c06bbd0 0x7fea7c06e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea88036000 tx=0x7fea88008030 comp rx=0 tx=0).stop 2024-03-28T18:56:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980c9050 msgr2=0x7fea981ec160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980c9050 0x7fea981ec160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00e930 tx=0x7fea8c00edf0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 shutdown_connections 2024-03-28T18:56:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7c06bbd0 0x7fea7c06e090 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea98152ba0 0x7fea98156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980cb960 0x7fea981eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 --2- 172.21.15.27:0/2935188480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980c9050 0x7fea981ec160 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.725+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 >> 172.21.15.27:0/2935188480 conn(0x7fea9806b4e0 msgr2=0x7fea9806b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.726+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 shutdown_connections 2024-03-28T18:56:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.726+0000 7fea9e400640 1 -- 172.21.15.27:0/2935188480 wait complete. 2024-03-28T18:56:00.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-03-28T18:56:00.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.953+0000 7fa653e00640 1 Processor -- start 2024-03-28T18:56:00.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.954+0000 7fa653e00640 1 -- start start 2024-03-28T18:56:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.955+0000 7fa653e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa6540c8db0 0x7fa654156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.955+0000 7fa653e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6540735f0 con 0x7fa654152840 2024-03-28T18:56:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.955+0000 7fa653e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6540c7d10 con 0x7fa6540c5aa0 2024-03-28T18:56:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.955+0000 7fa653e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6541565d0 con 0x7fa6540c8db0 2024-03-28T18:56:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.955+0000 7fa653400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa654152840 0x7fa65415ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38178/0 (socket says 172.21.15.27:38178) 2024-03-28T18:56:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.955+0000 7fa653400640 1 -- 172.21.15.27:0/651862288 learned_addr learned my addr 172.21.15.27:0/651862288 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa652000640 1 --2- 172.21.15.27:0/651862288 >> v2:172.21.15.155:3300/0 conn(0x7fa6540c8db0 0x7fa654156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703047686 0 0) 0x7fa6540735f0 con 0x7fa654152840 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa634003520 con 0x7fa654152840 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 311753216 0 0) 0x7fa634003520 con 0x7fa654152840 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 >> v2:172.21.15.155:3300/0 conn(0x7fa6540c8db0 msgr2=0x7fa654156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 --2- 172.21.15.27:0/651862288 >> v2:172.21.15.155:3300/0 conn(0x7fa6540c8db0 0x7fa654156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 >> v1:172.21.15.155:6789/0 conn(0x7fa6540c5aa0 legacy=0x7fa654072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6540c7ff0 con 0x7fa654152840 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fa64c001670 con 0x7fa654152840 2024-03-28T18:56:00.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.956+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 >> v1:172.21.15.27:6789/0 conn(0x7fa654152840 legacy=0x7fa65415ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa651600640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6340039b0 0x7fa634003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa651600640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6340043b0 0x7fa634004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa651600640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa634005120 0x7fa63400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6541565d0 con 0x7fa6340039b0 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63400c060 con 0x7fa6340043b0 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63400c260 con 0x7fa634005120 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa652a00640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6340039b0 0x7fa634003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa652a00640 1 -- 172.21.15.27:0/651862288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa634005120 msgr2=0x7fa63400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa652a00640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa634005120 0x7fa63400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa652a00640 1 -- 172.21.15.27:0/651862288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6340043b0 msgr2=0x7fa634004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa652a00640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6340043b0 0x7fa634004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa652a00640 1 -- 172.21.15.27:0/651862288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa634003520 con 0x7fa6340039b0 2024-03-28T18:56:00.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.957+0000 7fa653400640 1 -- 172.21.15.27:0/651862288 reap_dead start 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa652a00640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6340039b0 0x7fa634003d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa644001860 tx=0x7fa634005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d141bbc3201d978 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644038070 con 0x7fa6340039b0 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa644033410 con 0x7fa6340039b0 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa651600640 1 -- 172.21.15.27:0/651862288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644001430 con 0x7fa6340039b0 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa653e00640 1 -- 172.21.15.27:0/651862288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6340039b0 msgr2=0x7fa634003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa653e00640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6340039b0 0x7fa634003d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa644001860 tx=0x7fa634005830 comp rx=0 tx=0).stop 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa653e00640 1 -- 172.21.15.27:0/651862288 shutdown_connections 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa653e00640 1 --2- 172.21.15.27:0/651862288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6340039b0 0x7fa634003d90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.959+0000 7fa653e00640 1 -- 172.21.15.27:0/651862288 >> 172.21.15.27:0/651862288 conn(0x7fa65406b0c0 msgr2=0x7fa65406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.960+0000 7fa653e00640 1 -- 172.21.15.27:0/651862288 shutdown_connections 2024-03-28T18:56:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.960+0000 7fa653e00640 1 -- 172.21.15.27:0/651862288 wait complete. 2024-03-28T18:56:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.961+0000 7fa653e00640 1 Processor -- start 2024-03-28T18:56:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.961+0000 7fa653e00640 1 -- start start 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6340039b0 0x7fa6540c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa654150100 0x7fa6541ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6541eee50 0x7fa6541eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa654072100 con 0x7fa6541eee50 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6540c82d0 con 0x7fa654150100 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6540c78c0 con 0x7fa6340039b0 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6541eee50 0x7fa6541eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6541eee50 0x7fa6541eb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46926/0 (socket says 172.21.15.27:46926) 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.962+0000 7fa653400640 1 -- 172.21.15.27:0/331426321 learned_addr learned my addr 172.21.15.27:0/331426321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa652a00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6340039b0 0x7fa6540c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa652000640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa654150100 0x7fa6541ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa653400640 1 -- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6340039b0 msgr2=0x7fa6540c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa653400640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6340039b0 0x7fa6540c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa653400640 1 -- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa654150100 msgr2=0x7fa6541ee910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa653400640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa654150100 0x7fa6541ee910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa653400640 1 -- 172.21.15.27:0/331426321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6540c9f80 con 0x7fa6541eee50 2024-03-28T18:56:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa652000640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa654150100 0x7fa6541ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa653400640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6541eee50 0x7fa6541eb130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000fd0 tx=0x7fa64c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.963+0000 7fa64b400640 1 -- 172.21.15.27:0/331426321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0074e0 con 0x7fa6541eee50 2024-03-28T18:56:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.964+0000 7fa64b400640 1 -- 172.21.15.27:0/331426321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c0047e0 con 0x7fa6541eee50 2024-03-28T18:56:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.964+0000 7fa64b400640 1 -- 172.21.15.27:0/331426321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64c004bd0 con 0x7fa6541eee50 2024-03-28T18:56:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.964+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65414afc0 con 0x7fa6541eee50 2024-03-28T18:56:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.964+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6540c3eb0 con 0x7fa6541eee50 2024-03-28T18:56:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.965+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65404f400 con 0x7fa6541eee50 2024-03-28T18:56:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.966+0000 7fa64b400640 1 -- 172.21.15.27:0/331426321 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa64c012360 con 0x7fa6541eee50 2024-03-28T18:56:00.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.967+0000 7fa64b400640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa63006bc00 0x7fa63006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:00.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.967+0000 7fa64b400640 1 -- 172.21.15.27:0/331426321 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fa64c08a0d0 con 0x7fa6541eee50 2024-03-28T18:56:00.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.967+0000 7fa652a00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa63006bc00 0x7fa63006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:00.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.968+0000 7fa652a00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa63006bc00 0x7fa63006e0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa644035000 tx=0x7fa644009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:00.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:00.971+0000 7fa64b400640 1 -- 172.21.15.27:0/331426321 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa64c01b030 con 0x7fa6541eee50 2024-03-28T18:56:01.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.171+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fa6540c1d10 con 0x7fa6541eee50 2024-03-28T18:56:01.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.172+0000 7fa64b400640 1 -- 172.21.15.27:0/331426321 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v24) v1 ==== 95+0+11 (secure 0 0 0) 0x7fa64c004d70 con 0x7fa6541eee50 2024-03-28T18:56:01.172 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:01.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa63006bc00 msgr2=0x7fa63006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa63006bc00 0x7fa63006e0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa644035000 tx=0x7fa644009030 comp rx=0 tx=0).stop 2024-03-28T18:56:01.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6541eee50 msgr2=0x7fa6541eb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6541eee50 0x7fa6541eb130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000fd0 tx=0x7fa64c00c8e0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 shutdown_connections 2024-03-28T18:56:01.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa63006bc00 0x7fa63006e0c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6541eee50 0x7fa6541eb130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa654150100 0x7fa6541ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 --2- 172.21.15.27:0/331426321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6340039b0 0x7fa6540c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.174+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 >> 172.21.15.27:0/331426321 conn(0x7fa65406b0c0 msgr2=0x7fa654154c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:01.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.175+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 shutdown_connections 2024-03-28T18:56:01.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.175+0000 7fa653e00640 1 -- 172.21.15.27:0/331426321 wait complete. 2024-03-28T18:56:01.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:56:01.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.388+0000 7f0fffe00640 1 Processor -- start 2024-03-28T18:56:01.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.388+0000 7f0fffe00640 1 -- start start 2024-03-28T18:56:01.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.390+0000 7f0fffe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1000154380 0x7f100006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.390+0000 7f0fffe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f100006c610 0x7f100015ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.390+0000 7f0fffe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10000a7490 con 0x7f10000c26e0 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.390+0000 7f0fffe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1000154760 con 0x7f1000154380 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.390+0000 7f0fffe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10000a7240 con 0x7f100006c610 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.390+0000 7f0ffea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1000154380 0x7f100006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.390+0000 7f0ffea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1000154380 0x7f100006c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46938/0 (socket says 172.21.15.27:46938) 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.390+0000 7f0ffea00640 1 -- 172.21.15.27:0/2521072669 learned_addr learned my addr 172.21.15.27:0/2521072669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.391+0000 7f0fff400640 1 --2- 172.21.15.27:0/2521072669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f100006c610 0x7f100015ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.391+0000 7f0ffea00640 1 -- 172.21.15.27:0/2521072669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f100006c610 msgr2=0x7f100015ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.391+0000 7f0ffea00640 1 --2- 172.21.15.27:0/2521072669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f100006c610 0x7f100015ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.391+0000 7f0ffea00640 1 -- 172.21.15.27:0/2521072669 >> v1:172.21.15.155:6789/0 conn(0x7f10000c26e0 legacy=0x7f10000c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.391+0000 7f0ffea00640 1 -- 172.21.15.27:0/2521072669 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10000725a0 con 0x7f1000154380 2024-03-28T18:56:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.391+0000 7f0fff400640 1 --2- 172.21.15.27:0/2521072669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f100006c610 0x7f100015ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.392+0000 7f0ffea00640 1 --2- 172.21.15.27:0/2521072669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1000154380 0x7f100006c0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009870 tx=0x7f0ff002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22eb47ae7db1ab0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:01.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.392+0000 7f0ffe000640 1 -- 172.21.15.27:0/2521072669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0034070 con 0x7f1000154380 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.392+0000 7f0ffe000640 1 -- 172.21.15.27:0/2521072669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff0030840 con 0x7f1000154380 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.392+0000 7f0ffe000640 1 -- 172.21.15.27:0/2521072669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0030c10 con 0x7f1000154380 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.393+0000 7f0fffe00640 1 -- 172.21.15.27:0/2521072669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1000154380 msgr2=0x7f100006c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.393+0000 7f0fffe00640 1 --2- 172.21.15.27:0/2521072669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1000154380 0x7f100006c0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009870 tx=0x7f0ff002b980 comp rx=0 tx=0).stop 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.393+0000 7f0fffe00640 1 -- 172.21.15.27:0/2521072669 shutdown_connections 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.393+0000 7f0fffe00640 1 --2- 172.21.15.27:0/2521072669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f100006c610 0x7f100015ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.393+0000 7f0fffe00640 1 --2- 172.21.15.27:0/2521072669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1000154380 0x7f100006c0d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.393+0000 7f0fffe00640 1 -- 172.21.15.27:0/2521072669 >> 172.21.15.27:0/2521072669 conn(0x7f10000c7aa0 msgr2=0x7f10000c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.393+0000 7f0fffe00640 1 -- 172.21.15.27:0/2521072669 shutdown_connections 2024-03-28T18:56:01.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.393+0000 7f0fffe00640 1 -- 172.21.15.27:0/2521072669 wait complete. 2024-03-28T18:56:01.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.394+0000 7f0fffe00640 1 Processor -- start 2024-03-28T18:56:01.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.395+0000 7f0fffe00640 1 -- start start 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.395+0000 7f0fffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100006c610 0x7f10000c1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0fffe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1000154380 0x7f10001eb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0fffe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10000a8970 0x7f10001eb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0fffe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10000c2e60 con 0x7f100006c610 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0fffe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10000c31d0 con 0x7f1000154380 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0fffe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10000c77b0 con 0x7f10000a8970 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0ffea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100006c610 0x7f10000c1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0ffea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100006c610 0x7f10000c1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46954/0 (socket says 172.21.15.27:46954) 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0ffea00640 1 -- 172.21.15.27:0/3824204819 learned_addr learned my addr 172.21.15.27:0/3824204819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:01.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0ffea00640 1 -- 172.21.15.27:0/3824204819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10000a8970 msgr2=0x7f10001eb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0ffea00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10000a8970 0x7f10001eb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0ffea00640 1 -- 172.21.15.27:0/3824204819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1000154380 msgr2=0x7f10001eb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0ffea00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1000154380 0x7f10001eb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.396+0000 7f0ffea00640 1 -- 172.21.15.27:0/3824204819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1000156de0 con 0x7f100006c610 2024-03-28T18:56:01.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.397+0000 7f0ffea00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100006c610 0x7f10000c1140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0ff002b9c0 tx=0x7f0ff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:01.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.397+0000 7f0ff6a00640 1 -- 172.21.15.27:0/3824204819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0048070 con 0x7f100006c610 2024-03-28T18:56:01.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.397+0000 7f0ff6a00640 1 -- 172.21.15.27:0/3824204819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff0030640 con 0x7f100006c610 2024-03-28T18:56:01.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.397+0000 7f0ff6a00640 1 -- 172.21.15.27:0/3824204819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0034040 con 0x7f100006c610 2024-03-28T18:56:01.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.397+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100006fea0 con 0x7f100006c610 2024-03-28T18:56:01.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.397+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10000c1680 con 0x7f100006c610 2024-03-28T18:56:01.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.398+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10000c6d10 con 0x7f100006c610 2024-03-28T18:56:01.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.399+0000 7f0ff6a00640 1 -- 172.21.15.27:0/3824204819 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0ff004b070 con 0x7f100006c610 2024-03-28T18:56:01.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.400+0000 7f0ff6a00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0fdc06bc50 0x7f0fdc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.400+0000 7f0ff6a00640 1 -- 172.21.15.27:0/3824204819 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f0ff00acef0 con 0x7f100006c610 2024-03-28T18:56:01.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.400+0000 7f0ff7e00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0fdc06bc50 0x7f0fdc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.401+0000 7f0ff7e00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0fdc06bc50 0x7f0fdc06e110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8000f60 tx=0x7f0fe80079d0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:01.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.404+0000 7f0ff6a00640 1 -- 172.21.15.27:0/3824204819 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f10000c6d10 con 0x7f100006c610 2024-03-28T18:56:01.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.697+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1000069640 con 0x7f100006c610 2024-03-28T18:56:01.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.698+0000 7f0ff6a00640 1 -- 172.21.15.27:0/3824204819 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f0ff007b720 con 0x7f100006c610 2024-03-28T18:56:01.697 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:01.697 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:56:01.698 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0fdc06bc50 msgr2=0x7f0fdc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0fdc06bc50 0x7f0fdc06e110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8000f60 tx=0x7f0fe80079d0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100006c610 msgr2=0x7f10000c1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100006c610 0x7f10000c1140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0ff002b9c0 tx=0x7f0ff0004730 comp rx=0 tx=0).stop 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 shutdown_connections 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0fdc06bc50 0x7f0fdc06e110 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10000a8970 0x7f10001eb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1000154380 0x7f10001eb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 --2- 172.21.15.27:0/3824204819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f100006c610 0x7f10000c1140 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 >> 172.21.15.27:0/3824204819 conn(0x7f10000c7aa0 msgr2=0x7f10000c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:01.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 shutdown_connections 2024-03-28T18:56:01.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.700+0000 7f0fffe00640 1 -- 172.21.15.27:0/3824204819 wait complete. 2024-03-28T18:56:01.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.931+0000 7f1aca800640 1 Processor -- start 2024-03-28T18:56:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.931+0000 7f1aca800640 1 -- start start 2024-03-28T18:56:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.932+0000 7f1aca800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1ac40c58c0 0x7f1ac40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.932+0000 7f1aca800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac4158bd0 con 0x7f1ac40be5d0 2024-03-28T18:56:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.932+0000 7f1aca800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac4158eb0 con 0x7f1ac40c8ed0 2024-03-28T18:56:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.932+0000 7f1aca800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac4159190 con 0x7f1ac40c58c0 2024-03-28T18:56:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.933+0000 7f1ac9e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1ac40be5d0 0x7f1ac4155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38184/0 (socket says 172.21.15.27:38184) 2024-03-28T18:56:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.933+0000 7f1ac9e00640 1 -- 172.21.15.27:0/2697598810 learned_addr learned my addr 172.21.15.27:0/2697598810 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.933+0000 7f1ac9400640 1 --2- 172.21.15.27:0/2697598810 >> v2:172.21.15.155:3300/0 conn(0x7f1ac40c58c0 0x7f1ac40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.933+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934125741 0 0) 0x7f1ac4158bd0 con 0x7f1ac40be5d0 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.933+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aac003520 con 0x7f1ac40be5d0 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851528835 0 0) 0x7f1ac4158eb0 con 0x7f1ac40c8ed0 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac4158bd0 con 0x7f1ac40c8ed0 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2177452668 0 0) 0x7f1aac003520 con 0x7f1ac40be5d0 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 >> v2:172.21.15.155:3300/0 conn(0x7f1ac40c58c0 msgr2=0x7f1ac40c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 --2- 172.21.15.27:0/2697598810 >> v2:172.21.15.155:3300/0 conn(0x7f1ac40c58c0 0x7f1ac40c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 >> v1:172.21.15.155:6789/0 conn(0x7f1ac40c8ed0 legacy=0x7f1ac41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac4159470 con 0x7f1ac40be5d0 2024-03-28T18:56:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f1abc0019f0 con 0x7f1ac40be5d0 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 >> v1:172.21.15.27:6789/0 conn(0x7f1ac40be5d0 legacy=0x7f1ac4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1aac003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1aac004800 0x7f1aac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1aac005280 0x7f1aac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac9400640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1aac003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.934+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4159190 con 0x7f1aac003b90 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac00c1c0 con 0x7f1aac005280 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac00c3c0 con 0x7f1aac004800 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac9e00640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1aac005280 0x7f1aac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac9400640 1 -- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1aac004800 msgr2=0x7f1aac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac8a00640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1aac004800 0x7f1aac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac9400640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1aac004800 0x7f1aac004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac9400640 1 -- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1aac005280 msgr2=0x7f1aac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:01.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac9400640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1aac005280 0x7f1aac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac9400640 1 -- 172.21.15.27:0/2697598810 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac003520 con 0x7f1aac003b90 2024-03-28T18:56:01.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac9e00640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1aac005280 0x7f1aac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:01.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.935+0000 7f1ac9e00640 1 -- 172.21.15.27:0/2697598810 reap_dead start 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.936+0000 7f1ac9400640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1aac003f90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009ac0 tx=0x7f1aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e8ae5ce47eb855 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.936+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8039070 con 0x7f1aac003b90 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.936+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab802c6b0 con 0x7f1aac003b90 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.936+0000 7f1ac3e00640 1 -- 172.21.15.27:0/2697598810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8009690 con 0x7f1aac003b90 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.937+0000 7f1aca800640 1 -- 172.21.15.27:0/2697598810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 msgr2=0x7f1aac003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.937+0000 7f1aca800640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1aac003f90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009ac0 tx=0x7f1aac005830 comp rx=0 tx=0).stop 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.937+0000 7f1aca800640 1 -- 172.21.15.27:0/2697598810 shutdown_connections 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.937+0000 7f1aca800640 1 --2- 172.21.15.27:0/2697598810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1aac003f90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.937+0000 7f1aca800640 1 -- 172.21.15.27:0/2697598810 >> 172.21.15.27:0/2697598810 conn(0x7f1ac406b6f0 msgr2=0x7f1ac40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.937+0000 7f1aca800640 1 -- 172.21.15.27:0/2697598810 shutdown_connections 2024-03-28T18:56:01.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.937+0000 7f1aca800640 1 -- 172.21.15.27:0/2697598810 wait complete. 2024-03-28T18:56:01.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.939+0000 7f1aca800640 1 Processor -- start 2024-03-28T18:56:01.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.939+0000 7f1aca800640 1 -- start start 2024-03-28T18:56:01.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1aca800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1ac40c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1aca800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ac41525c0 0x7f1ac41513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1aca800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ac40c15d0 0x7f1ac4151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1aca800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40b09a0 con 0x7f1aac003b90 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1aca800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40b0820 con 0x7f1ac41525c0 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1aca800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40ad7d0 con 0x7f1ac40c15d0 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1ac9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1ac40c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1ac9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1ac40c46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46974/0 (socket says 172.21.15.27:46974) 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1ac9400640 1 -- 172.21.15.27:0/1489818869 learned_addr learned my addr 172.21.15.27:0/1489818869 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1ac9400640 1 -- 172.21.15.27:0/1489818869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ac40c15d0 msgr2=0x7f1ac4151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1ac9400640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ac40c15d0 0x7f1ac4151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1ac9400640 1 -- 172.21.15.27:0/1489818869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ac41525c0 msgr2=0x7f1ac41513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.940+0000 7f1ac8a00640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ac41525c0 0x7f1ac41513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.941+0000 7f1ac9400640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ac41525c0 0x7f1ac41513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.941+0000 7f1ac9400640 1 -- 172.21.15.27:0/1489818869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac4151e60 con 0x7f1aac003b90 2024-03-28T18:56:01.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.941+0000 7f1ac9400640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1ac40c46a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8000c00 tx=0x7f1ab8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:01.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.941+0000 7f1ac1600640 1 -- 172.21.15.27:0/1489818869 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8042020 con 0x7f1aac003b90 2024-03-28T18:56:01.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.941+0000 7f1ac1600640 1 -- 172.21.15.27:0/1489818869 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab803a040 con 0x7f1aac003b90 2024-03-28T18:56:01.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.941+0000 7f1ac1600640 1 -- 172.21.15.27:0/1489818869 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8039070 con 0x7f1aac003b90 2024-03-28T18:56:01.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.941+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac4152370 con 0x7f1aac003b90 2024-03-28T18:56:01.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.941+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac40c36b0 con 0x7f1aac003b90 2024-03-28T18:56:01.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.943+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac404f400 con 0x7f1aac003b90 2024-03-28T18:56:01.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.944+0000 7f1ac1600640 1 -- 172.21.15.27:0/1489818869 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1ab8045020 con 0x7f1aac003b90 2024-03-28T18:56:01.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.945+0000 7f1ac1600640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1aa406bc00 0x7f1aa406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:01.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.945+0000 7f1ac1600640 1 -- 172.21.15.27:0/1489818869 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f1ab80ac210 con 0x7f1aac003b90 2024-03-28T18:56:01.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.945+0000 7f1ac8a00640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1aa406bc00 0x7f1aa406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:01.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.946+0000 7f1ac8a00640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1aa406bc00 0x7f1aa406e0c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40ca2d0 tx=0x7f1ab0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:01.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:01.949+0000 7f1ac1600640 1 -- 172.21.15.27:0/1489818869 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1ac404f400 con 0x7f1aac003b90 2024-03-28T18:56:02.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.150+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ac4069640 con 0x7f1aac003b90 2024-03-28T18:56:02.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.152+0000 7f1ac1600640 1 -- 172.21.15.27:0/1489818869 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+20307 (secure 0 0 0) 0x7f1ab804c070 con 0x7f1aac003b90 2024-03-28T18:56:02.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:02.152 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:46.587251+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:02.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1aa406bc00 msgr2=0x7f1aa406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1aa406bc00 0x7f1aa406e0c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40ca2d0 tx=0x7f1ab0009040 comp rx=0 tx=0).stop 2024-03-28T18:56:02.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 msgr2=0x7f1ac40c46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1ac40c46a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8000c00 tx=0x7f1ab8004730 comp rx=0 tx=0).stop 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 shutdown_connections 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1aa406bc00 0x7f1aa406e0c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ac40c15d0 0x7f1ac4151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ac41525c0 0x7f1ac41513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 --2- 172.21.15.27:0/1489818869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1aac003b90 0x7f1ac40c46a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.154+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 >> 172.21.15.27:0/1489818869 conn(0x7f1ac406b6f0 msgr2=0x7f1ac41583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.155+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 shutdown_connections 2024-03-28T18:56:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.155+0000 7f1aca800640 1 -- 172.21.15.27:0/1489818869 wait complete. 2024-03-28T18:56:02.174 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-03-28T18:55:32.332283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-03-28T18:55:32.983411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-03-28T18:55:43.415397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-03-28T18:55:44.084162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:56:02.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:56:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.395+0000 7efff5e00640 1 Processor -- start 2024-03-28T18:56:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.395+0000 7efff5e00640 1 -- start start 2024-03-28T18:56:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.396+0000 7efff5e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efff0155790 0x7efff0155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.396+0000 7efff5e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff0158d90 con 0x7efff00c9050 2024-03-28T18:56:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.396+0000 7efff5e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff0159070 con 0x7efff0152ba0 2024-03-28T18:56:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.396+0000 7efff5e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff0159350 con 0x7efff0155790 2024-03-28T18:56:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.396+0000 7effefe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efff00c9050 0x7efff00cb420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38190/0 (socket says 172.21.15.27:38190) 2024-03-28T18:56:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.396+0000 7effefe00640 1 -- 172.21.15.27:0/1939847003 learned_addr learned my addr 172.21.15.27:0/1939847003 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7efff5400640 1 --2- 172.21.15.27:0/1939847003 >> v2:172.21.15.155:3300/0 conn(0x7efff0155790 0x7efff0155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196878134 0 0) 0x7efff0158d90 con 0x7efff00c9050 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd4003520 con 0x7efff00c9050 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892658966 0 0) 0x7effd4003520 con 0x7efff00c9050 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 >> v2:172.21.15.155:3300/0 conn(0x7efff0155790 msgr2=0x7efff0155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 --2- 172.21.15.27:0/1939847003 >> v2:172.21.15.155:3300/0 conn(0x7efff0155790 0x7efff0155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 >> v1:172.21.15.155:6789/0 conn(0x7efff0152ba0 legacy=0x7efff00726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff0159630 con 0x7efff00c9050 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7effdc001670 con 0x7efff00c9050 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 >> v1:172.21.15.27:6789/0 conn(0x7efff00c9050 legacy=0x7efff00cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effd40039b0 0x7effd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7effd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.397+0000 7effef400640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effd40051c0 0x7effd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff0159350 con 0x7effd4004800 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd400c110 con 0x7effd40039b0 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd400c310 con 0x7effd40051c0 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effefe00640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7effd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7efff4a00640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effd40039b0 0x7effd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effefe00640 1 -- 172.21.15.27:0/1939847003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effd40051c0 msgr2=0x7effd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effefe00640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effd40051c0 0x7effd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effefe00640 1 -- 172.21.15.27:0/1939847003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effd40039b0 msgr2=0x7effd4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effefe00640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effd40039b0 0x7effd4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7effefe00640 1 -- 172.21.15.27:0/1939847003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4003520 con 0x7effd4004800 2024-03-28T18:56:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.398+0000 7efff5400640 1 -- 172.21.15.27:0/1939847003 reap_dead start 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.399+0000 7effefe00640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7effd4004c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7effdc007fd0 tx=0x7effd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27de8289e31f0616 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc007ca0 con 0x7effd4004800 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effdc030070 con 0x7effd4004800 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7effef400640 1 -- 172.21.15.27:0/1939847003 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc005d10 con 0x7effd4004800 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7efff5e00640 1 -- 172.21.15.27:0/1939847003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 msgr2=0x7effd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7efff5e00640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7effd4004c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7effdc007fd0 tx=0x7effd4005830 comp rx=0 tx=0).stop 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7efff5e00640 1 -- 172.21.15.27:0/1939847003 shutdown_connections 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7efff5e00640 1 --2- 172.21.15.27:0/1939847003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7effd4004c80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7efff5e00640 1 -- 172.21.15.27:0/1939847003 >> 172.21.15.27:0/1939847003 conn(0x7efff006b4e0 msgr2=0x7efff006b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.400+0000 7efff5e00640 1 -- 172.21.15.27:0/1939847003 shutdown_connections 2024-03-28T18:56:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.401+0000 7efff5e00640 1 -- 172.21.15.27:0/1939847003 wait complete. 2024-03-28T18:56:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.402+0000 7efff5e00640 1 Processor -- start 2024-03-28T18:56:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.402+0000 7efff5e00640 1 -- start start 2024-03-28T18:56:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7efff0157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efff00c0430 0x7efff00aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efff00bea60 0x7efff014a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff5e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff0153100 con 0x7effd4004800 2024-03-28T18:56:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff5e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff0152f80 con 0x7efff00c0430 2024-03-28T18:56:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff5e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff00726c0 con 0x7efff00bea60 2024-03-28T18:56:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7efff0157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7efff0157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46994/0 (socket says 172.21.15.27:46994) 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.403+0000 7efff4a00640 1 -- 172.21.15.27:0/905480760 learned_addr learned my addr 172.21.15.27:0/905480760 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7effefe00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efff00c0430 0x7efff00aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7efff5400640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efff00bea60 0x7efff014a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7efff4a00640 1 -- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efff00bea60 msgr2=0x7efff014a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7efff4a00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efff00bea60 0x7efff014a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7efff4a00640 1 -- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efff00c0430 msgr2=0x7efff00aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7efff4a00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efff00c0430 0x7efff00aaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7efff4a00640 1 -- 172.21.15.27:0/905480760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff00ac380 con 0x7effd4004800 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7effefe00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efff00c0430 0x7efff00aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7efff4a00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7efff0157cd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7effe000e580 tx=0x7effe000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7effecc00640 1 -- 172.21.15.27:0/905480760 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe00052e0 con 0x7effd4004800 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.404+0000 7effecc00640 1 -- 172.21.15.27:0/905480760 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe0018070 con 0x7effd4004800 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.405+0000 7effecc00640 1 -- 172.21.15.27:0/905480760 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe0003640 con 0x7effd4004800 2024-03-28T18:56:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.405+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff00ac660 con 0x7effd4004800 2024-03-28T18:56:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.405+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff01eeef0 con 0x7effd4004800 2024-03-28T18:56:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.406+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff00c55b0 con 0x7effd4004800 2024-03-28T18:56:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.407+0000 7effecc00640 1 -- 172.21.15.27:0/905480760 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7effe00037e0 con 0x7effd4004800 2024-03-28T18:56:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.408+0000 7effecc00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7effd006bc00 0x7effd006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.408+0000 7effecc00640 1 -- 172.21.15.27:0/905480760 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7effe008aa80 con 0x7effd4004800 2024-03-28T18:56:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.408+0000 7effefe00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7effd006bc00 0x7effd006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.409+0000 7effefe00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7effd006bc00 0x7effd006e0c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7effdc034000 tx=0x7effdc007000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:02.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.412+0000 7effecc00640 1 -- 172.21.15.27:0/905480760 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efff00c55b0 con 0x7effd4004800 2024-03-28T18:56:02.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.806+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efff00aa800 con 0x7effd4004800 2024-03-28T18:56:02.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.806+0000 7effecc00640 1 -- 172.21.15.27:0/905480760 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7effe008ce20 con 0x7effd4004800 2024-03-28T18:56:02.806 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7effd006bc00 msgr2=0x7effd006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7effd006bc00 0x7effd006e0c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7effdc034000 tx=0x7effdc007000 comp rx=0 tx=0).stop 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 msgr2=0x7efff0157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7efff0157cd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7effe000e580 tx=0x7effe000ea40 comp rx=0 tx=0).stop 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 shutdown_connections 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7effd006bc00 0x7effd006e0c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efff00bea60 0x7efff014a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efff00c0430 0x7efff00aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 --2- 172.21.15.27:0/905480760 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effd4004800 0x7efff0157cd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 >> 172.21.15.27:0/905480760 conn(0x7efff006b4e0 msgr2=0x7efff006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 shutdown_connections 2024-03-28T18:56:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:02.809+0000 7efff5e00640 1 -- 172.21.15.27:0/905480760 wait complete. 2024-03-28T18:56:02.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-03-28T18:56:03.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.049+0000 7fac37200640 1 Processor -- start 2024-03-28T18:56:03.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.049+0000 7fac37200640 1 -- start start 2024-03-28T18:56:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac37200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fac38150900 0x7fac38157420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac37200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac380c8490 0x7fac381533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac37200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380c5e20 con 0x7fac38153910 2024-03-28T18:56:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac37200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac380c75c0 con 0x7fac380c8490 2024-03-28T18:56:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac37200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac380c78a0 con 0x7fac38150900 2024-03-28T18:56:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac2fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac380c8490 0x7fac381533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac2fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac380c8490 0x7fac381533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46996/0 (socket says 172.21.15.27:46996) 2024-03-28T18:56:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac2fe00640 1 -- 172.21.15.27:0/875611428 learned_addr learned my addr 172.21.15.27:0/875611428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac35e00640 1 --2- 172.21.15.27:0/875611428 >> v2:172.21.15.155:3300/0 conn(0x7fac38150900 0x7fac38157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac2fe00640 1 -- 172.21.15.27:0/875611428 >> v2:172.21.15.155:3300/0 conn(0x7fac38150900 msgr2=0x7fac38157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac2fe00640 1 --2- 172.21.15.27:0/875611428 >> v2:172.21.15.155:3300/0 conn(0x7fac38150900 0x7fac38157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac2fe00640 1 -- 172.21.15.27:0/875611428 >> v1:172.21.15.27:6789/0 conn(0x7fac38153910 legacy=0x7fac380c52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.051+0000 7fac2fe00640 1 -- 172.21.15.27:0/875611428 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38156940 con 0x7fac380c8490 2024-03-28T18:56:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.053+0000 7fac2fe00640 1 --2- 172.21.15.27:0/875611428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac380c8490 0x7fac381533d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fac380ab5d0 tx=0x7fac2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474a8aabc08327cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.053+0000 7fac35400640 1 -- 172.21.15.27:0/875611428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac20035020 con 0x7fac380c8490 2024-03-28T18:56:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.053+0000 7fac35400640 1 -- 172.21.15.27:0/875611428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac20031870 con 0x7fac380c8490 2024-03-28T18:56:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.053+0000 7fac35400640 1 -- 172.21.15.27:0/875611428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac20031c40 con 0x7fac380c8490 2024-03-28T18:56:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.053+0000 7fac37200640 1 -- 172.21.15.27:0/875611428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac380c8490 msgr2=0x7fac381533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.053+0000 7fac37200640 1 --2- 172.21.15.27:0/875611428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac380c8490 0x7fac381533d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fac380ab5d0 tx=0x7fac2002b910 comp rx=0 tx=0).stop 2024-03-28T18:56:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.054+0000 7fac37200640 1 -- 172.21.15.27:0/875611428 shutdown_connections 2024-03-28T18:56:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.054+0000 7fac37200640 1 --2- 172.21.15.27:0/875611428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac380c8490 0x7fac381533d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.054+0000 7fac37200640 1 --2- 172.21.15.27:0/875611428 >> v2:172.21.15.155:3300/0 conn(0x7fac38150900 0x7fac38157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.054+0000 7fac37200640 1 -- 172.21.15.27:0/875611428 >> 172.21.15.27:0/875611428 conn(0x7fac380aa7c0 msgr2=0x7fac380aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.054+0000 7fac37200640 1 -- 172.21.15.27:0/875611428 shutdown_connections 2024-03-28T18:56:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.054+0000 7fac37200640 1 -- 172.21.15.27:0/875611428 wait complete. 2024-03-28T18:56:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.055+0000 7fac37200640 1 Processor -- start 2024-03-28T18:56:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.056+0000 7fac37200640 1 -- start start 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.056+0000 7fac37200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac380c8490 0x7fac380c0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.056+0000 7fac37200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac38150900 0x7fac381ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.056+0000 7fac37200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac38153910 0x7fac38152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.056+0000 7fac37200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac38150ce0 con 0x7fac38153910 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.056+0000 7fac37200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac38156f00 con 0x7fac38150900 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.056+0000 7fac37200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac38150e60 con 0x7fac380c8490 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac38153910 0x7fac38152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac38153910 0x7fac38152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47006/0 (socket says 172.21.15.27:47006) 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 -- 172.21.15.27:0/3469345562 learned_addr learned my addr 172.21.15.27:0/3469345562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac2fe00640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac38150900 0x7fac381ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 -- 172.21.15.27:0/3469345562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac380c8490 msgr2=0x7fac380c0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac380c8490 0x7fac380c0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 -- 172.21.15.27:0/3469345562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac38150900 msgr2=0x7fac381ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac38150900 0x7fac381ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 -- 172.21.15.27:0/3469345562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac381ef100 con 0x7fac38153910 2024-03-28T18:56:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.057+0000 7fac36800640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac38153910 0x7fac38152030 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fac3000c730 tx=0x7fac3000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.058+0000 7fac2e000640 1 -- 172.21.15.27:0/3469345562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30003b70 con 0x7fac38153910 2024-03-28T18:56:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.058+0000 7fac2e000640 1 -- 172.21.15.27:0/3469345562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac30003d10 con 0x7fac38153910 2024-03-28T18:56:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.058+0000 7fac2e000640 1 -- 172.21.15.27:0/3469345562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30005310 con 0x7fac38153910 2024-03-28T18:56:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.058+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac381dca80 con 0x7fac38153910 2024-03-28T18:56:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.058+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac380c0c60 con 0x7fac38153910 2024-03-28T18:56:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.060+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac381dcfa0 con 0x7fac38153910 2024-03-28T18:56:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.060+0000 7fac2e000640 1 -- 172.21.15.27:0/3469345562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fac300056c0 con 0x7fac38153910 2024-03-28T18:56:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.061+0000 7fac2e000640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fac1406bbd0 0x7fac1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.061+0000 7fac2e000640 1 -- 172.21.15.27:0/3469345562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fac3008ad90 con 0x7fac38153910 2024-03-28T18:56:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.061+0000 7fac35e00640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fac1406bbd0 0x7fac1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.062+0000 7fac35e00640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fac1406bbd0 0x7fac1406e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac380aa130 tx=0x7fac28008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.065+0000 7fac2e000640 1 -- 172.21.15.27:0/3469345562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fac381dcfa0 con 0x7fac38153910 2024-03-28T18:56:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.269+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac3804f400 con 0x7fac38153910 2024-03-28T18:56:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.270+0000 7fac2e000640 1 -- 172.21.15.27:0/3469345562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v24) v1 ==== 99+0+11 (secure 0 0 0) 0x7fac30059540 con 0x7fac38153910 2024-03-28T18:56:03.270 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fac1406bbd0 msgr2=0x7fac1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fac1406bbd0 0x7fac1406e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac380aa130 tx=0x7fac28008040 comp rx=0 tx=0).stop 2024-03-28T18:56:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac38153910 msgr2=0x7fac38152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac38153910 0x7fac38152030 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fac3000c730 tx=0x7fac3000cbf0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 shutdown_connections 2024-03-28T18:56:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fac1406bbd0 0x7fac1406e090 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac38153910 0x7fac38152030 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac38150900 0x7fac381ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 --2- 172.21.15.27:0/3469345562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac380c8490 0x7fac380c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 >> 172.21.15.27:0/3469345562 conn(0x7fac380aa7c0 msgr2=0x7fac380c4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 shutdown_connections 2024-03-28T18:56:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.272+0000 7fac37200640 1 -- 172.21.15.27:0/3469345562 wait complete. 2024-03-28T18:56:03.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-03-28T18:56:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.509+0000 7f6a77200640 1 Processor -- start 2024-03-28T18:56:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.510+0000 7f6a77200640 1 -- start start 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.511+0000 7f6a77200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a700c4070 0x7f6a7006b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.511+0000 7f6a77200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6a70159850 0x7f6a700c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.511+0000 7f6a77200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a700c7150 con 0x7f6a700a82f0 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.511+0000 7f6a77200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a700c8720 con 0x7f6a70159850 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.511+0000 7f6a77200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700c8300 con 0x7f6a700c4070 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.511+0000 7f6a6fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6a700a82f0 0x7f6a700c7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38204/0 (socket says 172.21.15.27:38204) 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a6fe00640 1 -- 172.21.15.27:0/3279147378 learned_addr learned my addr 172.21.15.27:0/3279147378 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a75200640 1 --2- 172.21.15.27:0/3279147378 >> v2:172.21.15.27:3300/0 conn(0x7f6a70159850 0x7f6a700c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a67e00640 1 --2- 172.21.15.27:0/3279147378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a700c4070 0x7f6a7006b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a6f400640 1 -- 172.21.15.27:0/3279147378 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221730084 0 0) 0x7f6a700c7150 con 0x7f6a700a82f0 2024-03-28T18:56:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a6f400640 1 -- 172.21.15.27:0/3279147378 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a54003520 con 0x7f6a700a82f0 2024-03-28T18:56:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a75200640 1 -- 172.21.15.27:0/3279147378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a700c4070 msgr2=0x7f6a7006b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a75200640 1 --2- 172.21.15.27:0/3279147378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a700c4070 0x7f6a7006b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a75200640 1 -- 172.21.15.27:0/3279147378 >> v1:172.21.15.27:6789/0 conn(0x7f6a700a82f0 legacy=0x7f6a700c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.512+0000 7f6a75200640 1 -- 172.21.15.27:0/3279147378 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a700c89e0 con 0x7f6a70159850 2024-03-28T18:56:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.513+0000 7f6a75200640 1 --2- 172.21.15.27:0/3279147378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a70159850 0x7f6a700c5730 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009d90 tx=0x7f6a6802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e243d0001b26ba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a6f400640 1 -- 172.21.15.27:0/3279147378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68039070 con 0x7f6a70159850 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a6f400640 1 -- 172.21.15.27:0/3279147378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6802cc50 con 0x7f6a70159850 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a6f400640 1 -- 172.21.15.27:0/3279147378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68034440 con 0x7f6a70159850 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a77200640 1 -- 172.21.15.27:0/3279147378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a70159850 msgr2=0x7f6a700c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a77200640 1 --2- 172.21.15.27:0/3279147378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a70159850 0x7f6a700c5730 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009d90 tx=0x7f6a6802bea0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a77200640 1 -- 172.21.15.27:0/3279147378 shutdown_connections 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a77200640 1 --2- 172.21.15.27:0/3279147378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a70159850 0x7f6a700c5730 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a77200640 1 --2- 172.21.15.27:0/3279147378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a700c4070 0x7f6a7006b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a77200640 1 -- 172.21.15.27:0/3279147378 >> 172.21.15.27:0/3279147378 conn(0x7f6a700aa9d0 msgr2=0x7f6a700aadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.514+0000 7f6a77200640 1 -- 172.21.15.27:0/3279147378 shutdown_connections 2024-03-28T18:56:03.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.515+0000 7f6a77200640 1 -- 172.21.15.27:0/3279147378 wait complete. 2024-03-28T18:56:03.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.516+0000 7f6a77200640 1 Processor -- start 2024-03-28T18:56:03.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.516+0000 7f6a77200640 1 -- start start 2024-03-28T18:56:03.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.517+0000 7f6a77200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a700a82f0 0x7f6a701edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.517+0000 7f6a77200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a700c4070 0x7f6a701e86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.517+0000 7f6a77200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a70159850 0x7f6a701e8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.517+0000 7f6a77200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700a86d0 con 0x7f6a700c4070 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.517+0000 7f6a77200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700c76d0 con 0x7f6a700a82f0 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.517+0000 7f6a77200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a70159cc0 con 0x7f6a70159850 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a700c4070 0x7f6a701e86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a700c4070 0x7f6a701e86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47028/0 (socket says 172.21.15.27:47028) 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 -- 172.21.15.27:0/2123868327 learned_addr learned my addr 172.21.15.27:0/2123868327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a6fe00640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a700a82f0 0x7f6a701edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a75200640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a70159850 0x7f6a701e8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 -- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a70159850 msgr2=0x7f6a701e8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a70159850 0x7f6a701e8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 -- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a700a82f0 msgr2=0x7f6a701edfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a700a82f0 0x7f6a701edfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 -- 172.21.15.27:0/2123868327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a701e8bf0 con 0x7f6a700c4070 2024-03-28T18:56:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a6fe00640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a700a82f0 0x7f6a701edfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.518+0000 7f6a67e00640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a700c4070 0x7f6a701e86b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e8a0 tx=0x7f6a5800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.519+0000 7f6a6cc00640 1 -- 172.21.15.27:0/2123868327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58017070 con 0x7f6a700c4070 2024-03-28T18:56:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.519+0000 7f6a6cc00640 1 -- 172.21.15.27:0/2123868327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5800c8e0 con 0x7f6a700c4070 2024-03-28T18:56:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.519+0000 7f6a6cc00640 1 -- 172.21.15.27:0/2123868327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800ccd0 con 0x7f6a700c4070 2024-03-28T18:56:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.519+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a701ecf40 con 0x7f6a700c4070 2024-03-28T18:56:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.519+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a701ed500 con 0x7f6a700c4070 2024-03-28T18:56:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.520+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7004f470 con 0x7f6a700c4070 2024-03-28T18:56:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.521+0000 7f6a6cc00640 1 -- 172.21.15.27:0/2123868327 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6a5801a070 con 0x7f6a700c4070 2024-03-28T18:56:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.522+0000 7f6a6cc00640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.522+0000 7f6a6cc00640 1 -- 172.21.15.27:0/2123868327 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f6a5808ac80 con 0x7f6a700c4070 2024-03-28T18:56:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.522+0000 7f6a6fe00640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.523+0000 7f6a6fe00640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a60007fd0 tx=0x7f6a60009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:03.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.531+0000 7f6a6cc00640 1 -- 172.21.15.27:0/2123868327 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6a58059430 con 0x7f6a700c4070 2024-03-28T18:56:03.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.801+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f6a70070d80 con 0x7f6a700c4070 2024-03-28T18:56:03.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.801+0000 7f6a6cc00640 1 -- 172.21.15.27:0/2123868327 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v24) v1 ==== 95+0+11 (secure 0 0 0) 0x7f6a58058d30 con 0x7f6a700c4070 2024-03-28T18:56:03.801 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a4c06bbd0 msgr2=0x7f6a4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a60007fd0 tx=0x7f6a60009040 comp rx=0 tx=0).stop 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a700c4070 msgr2=0x7f6a701e86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a700c4070 0x7f6a701e86b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e8a0 tx=0x7f6a5800ed60 comp rx=0 tx=0).stop 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 shutdown_connections 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a4c06bbd0 0x7f6a4c06e090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a70159850 0x7f6a701e8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a700c4070 0x7f6a701e86b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 --2- 172.21.15.27:0/2123868327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a700a82f0 0x7f6a701edfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:03.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 >> 172.21.15.27:0/2123868327 conn(0x7f6a700aa9d0 msgr2=0x7f6a700c5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:03.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 shutdown_connections 2024-03-28T18:56:03.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:03.804+0000 7f6a77200640 1 -- 172.21.15.27:0/2123868327 wait complete. 2024-03-28T18:56:03.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-03-28T18:56:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.086+0000 7f1e81a00640 1 Processor -- start 2024-03-28T18:56:04.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.086+0000 7f1e81a00640 1 -- start start 2024-03-28T18:56:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.087+0000 7f1e81a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c0c5aa0 0x7f1e7c156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.087+0000 7f1e81a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e7c152840 0x7f1e7c152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.087+0000 7f1e81a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c0735f0 con 0x7f1e7c0c8db0 2024-03-28T18:56:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.087+0000 7f1e81a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c1565d0 con 0x7f1e7c152840 2024-03-28T18:56:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.087+0000 7f1e81a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0c82d0 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e7b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1e7c0c8db0 0x7f1e7c072620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38218/0 (socket says 172.21.15.27:38218) 2024-03-28T18:56:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e7b400640 1 -- 172.21.15.27:0/2985399846 learned_addr learned my addr 172.21.15.27:0/2985399846 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e81000640 1 --2- 172.21.15.27:0/2985399846 >> v2:172.21.15.27:3300/0 conn(0x7f1e7c152840 0x7f1e7c152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e7be00640 1 --2- 172.21.15.27:0/2985399846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c0c5aa0 0x7f1e7c156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e7aa00640 1 -- 172.21.15.27:0/2985399846 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761613128 0 0) 0x7f1e7c0735f0 con 0x7f1e7c0c8db0 2024-03-28T18:56:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e7aa00640 1 -- 172.21.15.27:0/2985399846 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e60003520 con 0x7f1e7c0c8db0 2024-03-28T18:56:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e81000640 1 -- 172.21.15.27:0/2985399846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c0c5aa0 msgr2=0x7f1e7c156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e81000640 1 --2- 172.21.15.27:0/2985399846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c0c5aa0 0x7f1e7c156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e81000640 1 -- 172.21.15.27:0/2985399846 >> v1:172.21.15.27:6789/0 conn(0x7f1e7c0c8db0 legacy=0x7f1e7c072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.088+0000 7f1e81000640 1 -- 172.21.15.27:0/2985399846 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c0c7d10 con 0x7f1e7c152840 2024-03-28T18:56:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.090+0000 7f1e81000640 1 --2- 172.21.15.27:0/2985399846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c152840 0x7f1e7c152ca0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009d90 tx=0x7f1e7402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148a6931c3d7552d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.090+0000 7f1e7aa00640 1 -- 172.21.15.27:0/2985399846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74039070 con 0x7f1e7c152840 2024-03-28T18:56:04.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.090+0000 7f1e7aa00640 1 -- 172.21.15.27:0/2985399846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7402cb30 con 0x7f1e7c152840 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.090+0000 7f1e7aa00640 1 -- 172.21.15.27:0/2985399846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74034440 con 0x7f1e7c152840 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.090+0000 7f1e81a00640 1 -- 172.21.15.27:0/2985399846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c152840 msgr2=0x7f1e7c152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.090+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2985399846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c152840 0x7f1e7c152ca0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009d90 tx=0x7f1e7402bd80 comp rx=0 tx=0).stop 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.091+0000 7f1e81a00640 1 -- 172.21.15.27:0/2985399846 shutdown_connections 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.091+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2985399846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c152840 0x7f1e7c152ca0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.091+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2985399846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c0c5aa0 0x7f1e7c156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.091+0000 7f1e81a00640 1 -- 172.21.15.27:0/2985399846 >> 172.21.15.27:0/2985399846 conn(0x7f1e7c06b0c0 msgr2=0x7f1e7c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.091+0000 7f1e81a00640 1 -- 172.21.15.27:0/2985399846 shutdown_connections 2024-03-28T18:56:04.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.091+0000 7f1e81a00640 1 -- 172.21.15.27:0/2985399846 wait complete. 2024-03-28T18:56:04.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.092+0000 7f1e81a00640 1 Processor -- start 2024-03-28T18:56:04.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.093+0000 7f1e81a00640 1 -- start start 2024-03-28T18:56:04.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.093+0000 7f1e81a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c0c5aa0 0x7f1e7c0b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.093+0000 7f1e81a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e7c0c8db0 0x7f1e7c159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.093+0000 7f1e81a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c152840 0x7f1e7c0abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.093+0000 7f1e81a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c072100 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.093+0000 7f1e81a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c071e20 con 0x7f1e7c0c8db0 2024-03-28T18:56:04.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.093+0000 7f1e81a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c072d30 con 0x7f1e7c152840 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c0c5aa0 0x7f1e7c0b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c0c5aa0 0x7f1e7c0b0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47048/0 (socket says 172.21.15.27:47048) 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7be00640 1 -- 172.21.15.27:0/2638388376 learned_addr learned my addr 172.21.15.27:0/2638388376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7b400640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e7c0c8db0 0x7f1e7c159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e81000640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c152840 0x7f1e7c0abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7be00640 1 -- 172.21.15.27:0/2638388376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c152840 msgr2=0x7f1e7c0abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7be00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c152840 0x7f1e7c0abb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7be00640 1 -- 172.21.15.27:0/2638388376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e7c0c8db0 msgr2=0x7f1e7c159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7be00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e7c0c8db0 0x7f1e7c159660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.094+0000 7f1e7be00640 1 -- 172.21.15.27:0/2638388376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c1ebb80 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.095+0000 7f1e7be00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c0c5aa0 0x7f1e7c0b0720 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00d830 tx=0x7f1e6c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.095+0000 7f1e73e00640 1 -- 172.21.15.27:0/2638388376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c00b6c0 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.095+0000 7f1e73e00640 1 -- 172.21.15.27:0/2638388376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c019070 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.095+0000 7f1e73e00640 1 -- 172.21.15.27:0/2638388376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c00ba80 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.095+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7c0af7e0 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.095+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c0ac090 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.096+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7c04f400 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.097+0000 7f1e73e00640 1 -- 172.21.15.27:0/2638388376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1e6c01e460 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.098+0000 7f1e73e00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e5806bbd0 0x7f1e5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.098+0000 7f1e73e00640 1 -- 172.21.15.27:0/2638388376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f1e6c08b4a0 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.098+0000 7f1e7b400640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e5806bbd0 0x7f1e5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.099+0000 7f1e7b400640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e5806bbd0 0x7f1e5806e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c061d00 tx=0x7f1e64009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.103+0000 7f1e73e00640 1 -- 172.21.15.27:0/2638388376 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1e6c059c50 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.306+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e7c0732f0 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.307+0000 7f1e73e00640 1 -- 172.21.15.27:0/2638388376 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v24) v1 ==== 99+0+11 (secure 0 0 0) 0x7f1e6c059550 con 0x7f1e7c0c5aa0 2024-03-28T18:56:04.307 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.309+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e5806bbd0 msgr2=0x7f1e5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.309+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e5806bbd0 0x7f1e5806e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c061d00 tx=0x7f1e64009040 comp rx=0 tx=0).stop 2024-03-28T18:56:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.309+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c0c5aa0 msgr2=0x7f1e7c0b0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.309+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c0c5aa0 0x7f1e7c0b0720 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00d830 tx=0x7f1e6c00dcf0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.310+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 shutdown_connections 2024-03-28T18:56:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.310+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e5806bbd0 0x7f1e5806e090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.310+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e7c152840 0x7f1e7c0abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.310+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e7c0c8db0 0x7f1e7c159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.310+0000 7f1e81a00640 1 --2- 172.21.15.27:0/2638388376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e7c0c5aa0 0x7f1e7c0b0720 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.310+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 >> 172.21.15.27:0/2638388376 conn(0x7f1e7c06b0c0 msgr2=0x7f1e7c151110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.310+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 shutdown_connections 2024-03-28T18:56:04.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.310+0000 7f1e81a00640 1 -- 172.21.15.27:0/2638388376 wait complete. 2024-03-28T18:56:04.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-03-28T18:56:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.538+0000 7fd91a400640 1 Processor -- start 2024-03-28T18:56:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.539+0000 7fd91a400640 1 -- start start 2024-03-28T18:56:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.540+0000 7fd91a400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd914152ba0 0x7fd9140726c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.540+0000 7fd91a400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd914072c00 con 0x7fd9140c9050 2024-03-28T18:56:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.540+0000 7fd91a400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd914158d90 con 0x7fd914155790 2024-03-28T18:56:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.540+0000 7fd91a400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd914159070 con 0x7fd914152ba0 2024-03-28T18:56:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.540+0000 7fd913e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd9140c9050 0x7fd9140cb420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38234/0 (socket says 172.21.15.27:38234) 2024-03-28T18:56:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.540+0000 7fd913e00640 1 -- 172.21.15.27:0/1884184086 learned_addr learned my addr 172.21.15.27:0/1884184086 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.541+0000 7fd919000640 1 --2- 172.21.15.27:0/1884184086 >> v2:172.21.15.155:3300/0 conn(0x7fd914152ba0 0x7fd9140726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.541+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638586025 0 0) 0x7fd914072c00 con 0x7fd9140c9050 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.541+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc003520 con 0x7fd9140c9050 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.541+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3631655583 0 0) 0x7fd8fc003520 con 0x7fd9140c9050 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.541+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 >> v2:172.21.15.155:3300/0 conn(0x7fd914152ba0 msgr2=0x7fd9140726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 --2- 172.21.15.27:0/1884184086 >> v2:172.21.15.155:3300/0 conn(0x7fd914152ba0 0x7fd9140726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 >> v1:172.21.15.155:6789/0 conn(0x7fd914155790 legacy=0x7fd914155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd914159350 con 0x7fd9140c9050 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd900001670 con 0x7fd9140c9050 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 >> v1:172.21.15.27:6789/0 conn(0x7fd9140c9050 legacy=0x7fd9140cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8fc0039b0 0x7fd8fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8fc004800 0x7fd8fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0052e0 0x7fd8fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd914159070 con 0x7fd8fc0039b0 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc00c1b0 con 0x7fd8fc004800 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc00c3b0 con 0x7fd8fc0052e0 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd919000640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8fc0039b0 0x7fd8fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd919a00640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0052e0 0x7fd8fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.542+0000 7fd913e00640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8fc004800 0x7fd8fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.543+0000 7fd919000640 1 -- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0052e0 msgr2=0x7fd8fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.543+0000 7fd919000640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0052e0 0x7fd8fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.543+0000 7fd919000640 1 -- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8fc004800 msgr2=0x7fd8fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.543+0000 7fd919000640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8fc004800 0x7fd8fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.543+0000 7fd919000640 1 -- 172.21.15.27:0/1884184086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc003520 con 0x7fd8fc0039b0 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.543+0000 7fd913e00640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8fc004800 0x7fd8fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.543+0000 7fd919a00640 1 -- 172.21.15.27:0/1884184086 reap_dead start 2024-03-28T18:56:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.544+0000 7fd919000640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8fc0039b0 0x7fd8fc003dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd908009a80 tx=0x7fd8fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6670d8815b6a499c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.544+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908009680 con 0x7fd8fc0039b0 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.544+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd908005990 con 0x7fd8fc0039b0 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.544+0000 7fd913400640 1 -- 172.21.15.27:0/1884184086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908005d80 con 0x7fd8fc0039b0 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.545+0000 7fd91a400640 1 -- 172.21.15.27:0/1884184086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8fc0039b0 msgr2=0x7fd8fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.545+0000 7fd91a400640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8fc0039b0 0x7fd8fc003dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd908009a80 tx=0x7fd8fc005830 comp rx=0 tx=0).stop 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.545+0000 7fd91a400640 1 -- 172.21.15.27:0/1884184086 shutdown_connections 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.545+0000 7fd91a400640 1 --2- 172.21.15.27:0/1884184086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8fc0039b0 0x7fd8fc003dd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.545+0000 7fd91a400640 1 -- 172.21.15.27:0/1884184086 >> 172.21.15.27:0/1884184086 conn(0x7fd91406b4e0 msgr2=0x7fd91406b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.545+0000 7fd91a400640 1 -- 172.21.15.27:0/1884184086 shutdown_connections 2024-03-28T18:56:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.545+0000 7fd91a400640 1 -- 172.21.15.27:0/1884184086 wait complete. 2024-03-28T18:56:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.547+0000 7fd91a400640 1 Processor -- start 2024-03-28T18:56:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.547+0000 7fd91a400640 1 -- start start 2024-03-28T18:56:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd91a400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0039b0 0x7fd9140bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd91a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9140c1830 0x7fd914157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd91a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9141ee860 0x7fd91414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd91a400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd914152f80 con 0x7fd9140c1830 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd91a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd914159910 con 0x7fd9141ee860 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd91a400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd914153100 con 0x7fd8fc0039b0 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd913e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9140c1830 0x7fd914157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd913e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9140c1830 0x7fd914157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47068/0 (socket says 172.21.15.27:47068) 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd913e00640 1 -- 172.21.15.27:0/2281010903 learned_addr learned my addr 172.21.15.27:0/2281010903 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd919000640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0039b0 0x7fd9140bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd913e00640 1 -- 172.21.15.27:0/2281010903 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0039b0 msgr2=0x7fd9140bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd913e00640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0039b0 0x7fd9140bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd919a00640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9141ee860 0x7fd91414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd913e00640 1 -- 172.21.15.27:0/2281010903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9141ee860 msgr2=0x7fd91414e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd913e00640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9141ee860 0x7fd91414e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.548+0000 7fd913e00640 1 -- 172.21.15.27:0/2281010903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9140c1410 con 0x7fd9140c1830 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.549+0000 7fd913e00640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9140c1830 0x7fd914157cf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd9000010e0 tx=0x7fd90000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.549+0000 7fd910c00640 1 -- 172.21.15.27:0/2281010903 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900019070 con 0x7fd9140c1830 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.549+0000 7fd910c00640 1 -- 172.21.15.27:0/2281010903 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9000072d0 con 0x7fd9140c1830 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.549+0000 7fd910c00640 1 -- 172.21.15.27:0/2281010903 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9000047e0 con 0x7fd9140c1830 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.549+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9140c0360 con 0x7fd9140c1830 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.549+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9140af310 con 0x7fd9140c1830 2024-03-28T18:56:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.550+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91404f400 con 0x7fd9140c1830 2024-03-28T18:56:04.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.551+0000 7fd910c00640 1 -- 172.21.15.27:0/2281010903 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd900004980 con 0x7fd9140c1830 2024-03-28T18:56:04.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.552+0000 7fd910c00640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd8f406bbd0 0x7fd8f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:04.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.552+0000 7fd910c00640 1 -- 172.21.15.27:0/2281010903 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fd90008a630 con 0x7fd9140c1830 2024-03-28T18:56:04.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.553+0000 7fd919000640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd8f406bbd0 0x7fd8f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:04.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.553+0000 7fd919000640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd8f406bbd0 0x7fd8f406e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd908035000 tx=0x7fd908030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:04.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.557+0000 7fd910c00640 1 -- 172.21.15.27:0/2281010903 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd9140c0360 con 0x7fd9140c1830 2024-03-28T18:56:04.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.791+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fd9140c8e30 con 0x7fd9140c1830 2024-03-28T18:56:04.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.791+0000 7fd910c00640 1 -- 172.21.15.27:0/2281010903 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v24) v1 ==== 95+0+11 (secure 0 0 0) 0x7fd900058df0 con 0x7fd9140c1830 2024-03-28T18:56:04.791 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:04.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd8f406bbd0 msgr2=0x7fd8f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd8f406bbd0 0x7fd8f406e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd908035000 tx=0x7fd908030000 comp rx=0 tx=0).stop 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9140c1830 msgr2=0x7fd914157cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9140c1830 0x7fd914157cf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd9000010e0 tx=0x7fd90000db40 comp rx=0 tx=0).stop 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 shutdown_connections 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd8f406bbd0 0x7fd8f406e090 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9141ee860 0x7fd91414e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9140c1830 0x7fd914157cf0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 --2- 172.21.15.27:0/2281010903 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8fc0039b0 0x7fd9140bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 >> 172.21.15.27:0/2281010903 conn(0x7fd91406b4e0 msgr2=0x7fd914153dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 shutdown_connections 2024-03-28T18:56:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:04.795+0000 7fd91a400640 1 -- 172.21.15.27:0/2281010903 wait complete. 2024-03-28T18:56:04.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:56:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.072+0000 7ff4d6e00640 1 Processor -- start 2024-03-28T18:56:05.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.072+0000 7ff4d6e00640 1 -- start start 2024-03-28T18:56:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.073+0000 7ff4d6e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff4d00c69a0 0x7ff4d00c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d6e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d0150ec0 con 0x7ff4d0153a30 2024-03-28T18:56:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d6e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d006dc70 con 0x7ff4d0150ae0 2024-03-28T18:56:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d6e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d0153550 con 0x7ff4d00c69a0 2024-03-28T18:56:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d5000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff4d00c69a0 0x7ff4d00c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d5000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff4d00c69a0 0x7ff4d00c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47076/0 (socket says 172.21.15.27:47076) 2024-03-28T18:56:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d5000640 1 -- 172.21.15.27:0/2349696830 learned_addr learned my addr 172.21.15.27:0/2349696830 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d5000640 1 -- 172.21.15.27:0/2349696830 >> v1:172.21.15.155:6789/0 conn(0x7ff4d0150ae0 legacy=0x7ff4d00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d5000640 1 -- 172.21.15.27:0/2349696830 >> v1:172.21.15.27:6789/0 conn(0x7ff4d0153a30 legacy=0x7ff4d015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.074+0000 7ff4d5000640 1 -- 172.21.15.27:0/2349696830 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d006df10 con 0x7ff4d00c69a0 2024-03-28T18:56:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.076+0000 7ff4d5000640 1 --2- 172.21.15.27:0/2349696830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d00c69a0 0x7ff4d00c6e30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009e50 tx=0x7ff4bc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16eecf2fd50dcd0d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:05.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.076+0000 7ff4cfe00640 1 -- 172.21.15.27:0/2349696830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc02cb00 con 0x7ff4d00c69a0 2024-03-28T18:56:05.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.076+0000 7ff4cfe00640 1 -- 172.21.15.27:0/2349696830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc02cca0 con 0x7ff4d00c69a0 2024-03-28T18:56:05.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.076+0000 7ff4cfe00640 1 -- 172.21.15.27:0/2349696830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc034610 con 0x7ff4d00c69a0 2024-03-28T18:56:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.076+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2349696830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d00c69a0 msgr2=0x7ff4d00c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.076+0000 7ff4d6e00640 1 --2- 172.21.15.27:0/2349696830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d00c69a0 0x7ff4d00c6e30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009e50 tx=0x7ff4bc02bf60 comp rx=0 tx=0).stop 2024-03-28T18:56:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.077+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2349696830 shutdown_connections 2024-03-28T18:56:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.077+0000 7ff4d6e00640 1 --2- 172.21.15.27:0/2349696830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d00c69a0 0x7ff4d00c6e30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.077+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2349696830 >> 172.21.15.27:0/2349696830 conn(0x7ff4d006b0c0 msgr2=0x7ff4d006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.077+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2349696830 shutdown_connections 2024-03-28T18:56:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.077+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2349696830 wait complete. 2024-03-28T18:56:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.078+0000 7ff4d6e00640 1 Processor -- start 2024-03-28T18:56:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.079+0000 7ff4d6e00640 1 -- start start 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.079+0000 7ff4d6e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4d00c69a0 0x7ff4d01dd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.079+0000 7ff4d6e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4d0150ae0 0x7ff4d014cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.079+0000 7ff4d6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d0153a30 0x7ff4d01ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.079+0000 7ff4d6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d006e7b0 con 0x7ff4d0153a30 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.079+0000 7ff4d6e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d006e4d0 con 0x7ff4d00c69a0 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.079+0000 7ff4d6e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00a6370 con 0x7ff4d0150ae0 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d0153a30 0x7ff4d01ddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d0153a30 0x7ff4d01ddc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47078/0 (socket says 172.21.15.27:47078) 2024-03-28T18:56:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d6400640 1 -- 172.21.15.27:0/2106981403 learned_addr learned my addr 172.21.15.27:0/2106981403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d5a00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4d00c69a0 0x7ff4d01dd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d5000640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4d0150ae0 0x7ff4d014cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d6400640 1 -- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4d0150ae0 msgr2=0x7ff4d014cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d6400640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4d0150ae0 0x7ff4d014cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d6400640 1 -- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4d00c69a0 msgr2=0x7ff4d01dd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d6400640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4d00c69a0 0x7ff4d01dd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d6400640 1 -- 172.21.15.27:0/2106981403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d01dcc90 con 0x7ff4d0153a30 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.080+0000 7ff4d5a00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4d00c69a0 0x7ff4d01dd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.081+0000 7ff4d6400640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d0153a30 0x7ff4d01ddc10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800c3c0 tx=0x7ff4c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.081+0000 7ff4cd600640 1 -- 172.21.15.27:0/2106981403 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8017070 con 0x7ff4d0153a30 2024-03-28T18:56:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.081+0000 7ff4cd600640 1 -- 172.21.15.27:0/2106981403 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c80052d0 con 0x7ff4d0153a30 2024-03-28T18:56:05.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.081+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d01dcf70 con 0x7ff4d0153a30 2024-03-28T18:56:05.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.081+0000 7ff4cd600640 1 -- 172.21.15.27:0/2106981403 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8003810 con 0x7ff4d0153a30 2024-03-28T18:56:05.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.081+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d01eb680 con 0x7ff4d0153a30 2024-03-28T18:56:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.082+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d0067dd0 con 0x7ff4d0153a30 2024-03-28T18:56:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.083+0000 7ff4cd600640 1 -- 172.21.15.27:0/2106981403 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff4c801a070 con 0x7ff4d0153a30 2024-03-28T18:56:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.084+0000 7ff4cd600640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff4b006bbd0 0x7ff4b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.084+0000 7ff4cd600640 1 -- 172.21.15.27:0/2106981403 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7ff4c808a100 con 0x7ff4d0153a30 2024-03-28T18:56:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.084+0000 7ff4d5a00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff4b006bbd0 0x7ff4b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.085+0000 7ff4d5a00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff4b006bbd0 0x7ff4b006e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0001700 tx=0x7ff4c0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.089+0000 7ff4cd600640 1 -- 172.21.15.27:0/2106981403 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff4d0150ec0 con 0x7ff4d0153a30 2024-03-28T18:56:05.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.331+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4d01eb1b0 con 0x7ff4d0153a30 2024-03-28T18:56:05.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.333+0000 7ff4cd600640 1 -- 172.21.15.27:0/2106981403 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7ff4c80588c0 con 0x7ff4d0153a30 2024-03-28T18:56:05.333 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:05.333 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:56:05.333 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:56:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff4b006bbd0 msgr2=0x7ff4b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff4b006bbd0 0x7ff4b006e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0001700 tx=0x7ff4c0009040 comp rx=0 tx=0).stop 2024-03-28T18:56:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d0153a30 msgr2=0x7ff4d01ddc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d0153a30 0x7ff4d01ddc10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800c3c0 tx=0x7ff4c800c880 comp rx=0 tx=0).stop 2024-03-28T18:56:05.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 shutdown_connections 2024-03-28T18:56:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff4b006bbd0 0x7ff4b006e090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4d0153a30 0x7ff4d01ddc10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4d0150ae0 0x7ff4d014cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 --2- 172.21.15.27:0/2106981403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4d00c69a0 0x7ff4d01dd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.336+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 >> 172.21.15.27:0/2106981403 conn(0x7ff4d006b0c0 msgr2=0x7ff4d00c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.337+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 shutdown_connections 2024-03-28T18:56:05.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.337+0000 7ff4d6e00640 1 -- 172.21.15.27:0/2106981403 wait complete. 2024-03-28T18:56:05.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:05.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.581+0000 7f6e10a00640 1 Processor -- start 2024-03-28T18:56:05.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.581+0000 7f6e10a00640 1 -- start start 2024-03-28T18:56:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e10a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 0x7f6e0c0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e10a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6e0c0c5580 0x7f6e0c0c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e10a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e0c0aada0 0x7f6e0c0c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e10a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0c156870 con 0x7f6e0c0aada0 2024-03-28T18:56:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e10a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0c155cf0 con 0x7f6e0c0c5580 2024-03-28T18:56:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e10a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c155fd0 con 0x7f6e0c1551a0 2024-03-28T18:56:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e0c0aada0 0x7f6e0c0c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e0c0aada0 0x7f6e0c0c1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47088/0 (socket says 172.21.15.27:47088) 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0be00640 1 -- 172.21.15.27:0/3279330016 learned_addr learned my addr 172.21.15.27:0/3279330016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0b400640 1 --2- 172.21.15.27:0/3279330016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 0x7f6e0c0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0aa00640 1 --2- 172.21.15.27:0/3279330016 >> v2:172.21.15.155:3300/0 conn(0x7f6e0c0c5580 0x7f6e0c0c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0be00640 1 -- 172.21.15.27:0/3279330016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 msgr2=0x7f6e0c0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0be00640 1 --2- 172.21.15.27:0/3279330016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 0x7f6e0c0c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0be00640 1 -- 172.21.15.27:0/3279330016 >> v2:172.21.15.155:3300/0 conn(0x7f6e0c0c5580 msgr2=0x7f6e0c0c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.583+0000 7f6e0be00640 1 --2- 172.21.15.27:0/3279330016 >> v2:172.21.15.155:3300/0 conn(0x7f6e0c0c5580 0x7f6e0c0c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.584+0000 7f6e0be00640 1 -- 172.21.15.27:0/3279330016 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c1562b0 con 0x7f6e0c0aada0 2024-03-28T18:56:05.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.584+0000 7f6e0aa00640 1 --2- 172.21.15.27:0/3279330016 >> v2:172.21.15.155:3300/0 conn(0x7f6e0c0c5580 0x7f6e0c0c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.585+0000 7f6e0be00640 1 --2- 172.21.15.27:0/3279330016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c0c1fc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c06e900 tx=0x7f6e0402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2de834d91866829 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.585+0000 7f6e0a000640 1 -- 172.21.15.27:0/3279330016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0402c950 con 0x7f6e0c0aada0 2024-03-28T18:56:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.585+0000 7f6e0a000640 1 -- 172.21.15.27:0/3279330016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0402caf0 con 0x7f6e0c0aada0 2024-03-28T18:56:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.585+0000 7f6e0a000640 1 -- 172.21.15.27:0/3279330016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e040343f0 con 0x7f6e0c0aada0 2024-03-28T18:56:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 -- 172.21.15.27:0/3279330016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 msgr2=0x7f6e0c0c1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 --2- 172.21.15.27:0/3279330016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c0c1fc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c06e900 tx=0x7f6e0402bdb0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 -- 172.21.15.27:0/3279330016 shutdown_connections 2024-03-28T18:56:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 --2- 172.21.15.27:0/3279330016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c0c1fc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 --2- 172.21.15.27:0/3279330016 >> v2:172.21.15.155:3300/0 conn(0x7f6e0c0c5580 0x7f6e0c0c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 --2- 172.21.15.27:0/3279330016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 0x7f6e0c0c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 -- 172.21.15.27:0/3279330016 >> 172.21.15.27:0/3279330016 conn(0x7f6e0c06b0c0 msgr2=0x7f6e0c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 -- 172.21.15.27:0/3279330016 shutdown_connections 2024-03-28T18:56:05.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.586+0000 7f6e10a00640 1 -- 172.21.15.27:0/3279330016 wait complete. 2024-03-28T18:56:05.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.588+0000 7f6e10a00640 1 Processor -- start 2024-03-28T18:56:05.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.588+0000 7f6e10a00640 1 -- start start 2024-03-28T18:56:05.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e10a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e10a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e0c0c5580 0x7f6e0c15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e10a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 0x7f6e0c152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e10a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c155580 con 0x7f6e0c0aada0 2024-03-28T18:56:05.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e10a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c1559a0 con 0x7f6e0c0c5580 2024-03-28T18:56:05.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e10a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c155700 con 0x7f6e0c1551a0 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e0b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e0b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c073340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47090/0 (socket says 172.21.15.27:47090) 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.589+0000 7f6e0b400640 1 -- 172.21.15.27:0/646612008 learned_addr learned my addr 172.21.15.27:0/646612008 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e0aa00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e0c0c5580 0x7f6e0c15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e0be00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 0x7f6e0c152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e0b400640 1 -- 172.21.15.27:0/646612008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 msgr2=0x7f6e0c152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e0b400640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 0x7f6e0c152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e0b400640 1 -- 172.21.15.27:0/646612008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e0c0c5580 msgr2=0x7f6e0c15b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e0b400640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e0c0c5580 0x7f6e0c15b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e0b400640 1 -- 172.21.15.27:0/646612008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c0c7050 con 0x7f6e0c0aada0 2024-03-28T18:56:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e0b400640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c073340 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00d800 tx=0x7f6dfc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e03400640 1 -- 172.21.15.27:0/646612008 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00b6c0 con 0x7f6e0c0aada0 2024-03-28T18:56:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e03400640 1 -- 172.21.15.27:0/646612008 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc019070 con 0x7f6e0c0aada0 2024-03-28T18:56:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.590+0000 7f6e03400640 1 -- 172.21.15.27:0/646612008 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00ba80 con 0x7f6e0c0aada0 2024-03-28T18:56:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.591+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c0721f0 con 0x7f6e0c0aada0 2024-03-28T18:56:05.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.591+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c0727b0 con 0x7f6e0c0aada0 2024-03-28T18:56:05.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.592+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c0c4e20 con 0x7f6e0c0aada0 2024-03-28T18:56:05.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.593+0000 7f6e03400640 1 -- 172.21.15.27:0/646612008 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6dfc00bc20 con 0x7f6e0c0aada0 2024-03-28T18:56:05.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.594+0000 7f6e03400640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dec06bc00 0x7f6dec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:05.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.594+0000 7f6e03400640 1 -- 172.21.15.27:0/646612008 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f6dfc08b1b0 con 0x7f6e0c0aada0 2024-03-28T18:56:05.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.594+0000 7f6e0aa00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dec06bc00 0x7f6dec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:05.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.595+0000 7f6e0aa00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dec06bc00 0x7f6dec06e0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c15eeb0 tx=0x7f6df8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:05.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.598+0000 7f6e03400640 1 -- 172.21.15.27:0/646612008 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6dfc059960 con 0x7f6e0c0aada0 2024-03-28T18:56:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.804+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e0c068180 con 0x7f6e0c0aada0 2024-03-28T18:56:05.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.805+0000 7f6e03400640 1 -- 172.21.15.27:0/646612008 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+20307 (secure 0 0 0) 0x7f6dfc059260 con 0x7f6e0c0aada0 2024-03-28T18:56:05.806 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:05.806 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:46.587251+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dec06bc00 msgr2=0x7f6dec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dec06bc00 0x7f6dec06e0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c15eeb0 tx=0x7f6df8008040 comp rx=0 tx=0).stop 2024-03-28T18:56:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 msgr2=0x7f6e0c073340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c073340 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00d800 tx=0x7f6dfc00dcc0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 shutdown_connections 2024-03-28T18:56:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dec06bc00 0x7f6dec06e0c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e0c1551a0 0x7f6e0c152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e0c0c5580 0x7f6e0c15b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 --2- 172.21.15.27:0/646612008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e0c0aada0 0x7f6e0c073340 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.808+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 >> 172.21.15.27:0/646612008 conn(0x7f6e0c06b0c0 msgr2=0x7f6e0c0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.809+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 shutdown_connections 2024-03-28T18:56:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:05.809+0000 7f6e10a00640 1 -- 172.21.15.27:0/646612008 wait complete. 2024-03-28T18:56:05.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:56:06.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.039+0000 7fbc93000640 1 Processor -- start 2024-03-28T18:56:06.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.039+0000 7fbc93000640 1 -- start start 2024-03-28T18:56:06.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.040+0000 7fbc93000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbc8c0c6cc0 0x7fbc8c159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:06.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.040+0000 7fbc93000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c0c5710 con 0x7fbc8c155b40 2024-03-28T18:56:06.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.040+0000 7fbc93000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c0c59f0 con 0x7fbc8c152a70 2024-03-28T18:56:06.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.040+0000 7fbc93000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c1556b0 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.041+0000 7fbc92600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbc8c155b40 0x7fbc8c155fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38252/0 (socket says 172.21.15.27:38252) 2024-03-28T18:56:06.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.041+0000 7fbc92600640 1 -- 172.21.15.27:0/3489719981 learned_addr learned my addr 172.21.15.27:0/3489719981 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:06.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.041+0000 7fbc91200640 1 --2- 172.21.15.27:0/3489719981 >> v2:172.21.15.155:3300/0 conn(0x7fbc8c0c6cc0 0x7fbc8c159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:06.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.041+0000 7fbc8be00640 1 -- 172.21.15.27:0/3489719981 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653921131 0 0) 0x7fbc8c0c5710 con 0x7fbc8c155b40 2024-03-28T18:56:06.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.041+0000 7fbc8be00640 1 -- 172.21.15.27:0/3489719981 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc70003520 con 0x7fbc8c155b40 2024-03-28T18:56:06.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.042+0000 7fbc8be00640 1 -- 172.21.15.27:0/3489719981 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3719923112 0 0) 0x7fbc70003520 con 0x7fbc8c155b40 2024-03-28T18:56:06.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.042+0000 7fbc91200640 1 -- 172.21.15.27:0/3489719981 >> v1:172.21.15.155:6789/0 conn(0x7fbc8c152a70 legacy=0x7fbc8c0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:06.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.042+0000 7fbc91200640 1 -- 172.21.15.27:0/3489719981 >> v1:172.21.15.27:6789/0 conn(0x7fbc8c155b40 legacy=0x7fbc8c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:06.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.042+0000 7fbc91200640 1 -- 172.21.15.27:0/3489719981 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c0c5cd0 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.043+0000 7fbc91200640 1 --2- 172.21.15.27:0/3489719981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c159580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc780097a0 tx=0x7fbc7802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa17a41363cf3ace server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:06.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc8be00640 1 -- 172.21.15.27:0/3489719981 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78037070 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc8be00640 1 -- 172.21.15.27:0/3489719981 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc78030510 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc8be00640 1 -- 172.21.15.27:0/3489719981 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84001670 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc93000640 1 -- 172.21.15.27:0/3489719981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c0c6cc0 msgr2=0x7fbc8c159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc93000640 1 --2- 172.21.15.27:0/3489719981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c159580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc780097a0 tx=0x7fbc7802b790 comp rx=0 tx=0).stop 2024-03-28T18:56:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc93000640 1 -- 172.21.15.27:0/3489719981 shutdown_connections 2024-03-28T18:56:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc93000640 1 --2- 172.21.15.27:0/3489719981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c159580 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc93000640 1 -- 172.21.15.27:0/3489719981 >> 172.21.15.27:0/3489719981 conn(0x7fbc8c06b2a0 msgr2=0x7fbc8c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc93000640 1 -- 172.21.15.27:0/3489719981 shutdown_connections 2024-03-28T18:56:06.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.044+0000 7fbc93000640 1 -- 172.21.15.27:0/3489719981 wait complete. 2024-03-28T18:56:06.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.046+0000 7fbc93000640 1 Processor -- start 2024-03-28T18:56:06.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.046+0000 7fbc93000640 1 -- start start 2024-03-28T18:56:06.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc93000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c0abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:06.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc93000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc8c152a70 0x7fbc8c1473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:06.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc93000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c155b40 0x7fbc8c1478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:06.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc93000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0cb970 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc93000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0cb7f0 con 0x7fbc8c155b40 2024-03-28T18:56:06.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc93000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0c8dd0 con 0x7fbc8c152a70 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc91c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c0abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc91c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c0abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47094/0 (socket says 172.21.15.27:47094) 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.047+0000 7fbc91c00640 1 -- 172.21.15.27:0/2892948815 learned_addr learned my addr 172.21.15.27:0/2892948815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc92600640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c155b40 0x7fbc8c1478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc91200640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc8c152a70 0x7fbc8c1473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc91c00640 1 -- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc8c152a70 msgr2=0x7fbc8c1473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc91c00640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc8c152a70 0x7fbc8c1473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc91c00640 1 -- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c155b40 msgr2=0x7fbc8c1478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc91c00640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c155b40 0x7fbc8c1478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc91c00640 1 -- 172.21.15.27:0/2892948815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c1e0120 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc92600640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c155b40 0x7fbc8c1478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc91200640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc8c152a70 0x7fbc8c1473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc91c00640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c0abb90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00c8a0 tx=0x7fbc7c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:06.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc89600640 1 -- 172.21.15.27:0/2892948815 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c003b70 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc89600640 1 -- 172.21.15.27:0/2892948815 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c003d30 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc89600640 1 -- 172.21.15.27:0/2892948815 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c00aa90 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.048+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c151780 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.049+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c151a40 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.050+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c067dd0 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.051+0000 7fbc89600640 1 -- 172.21.15.27:0/2892948815 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbc7c00ac30 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.051+0000 7fbc89600640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc6c06bc00 0x7fbc6c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:06.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.052+0000 7fbc89600640 1 -- 172.21.15.27:0/2892948815 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fbc7c08ad50 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.052+0000 7fbc91200640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc6c06bc00 0x7fbc6c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:06.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.053+0000 7fbc91200640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc6c06bc00 0x7fbc6c06e0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbc78034000 tx=0x7fbc7802f000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:06.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.056+0000 7fbc89600640 1 -- 172.21.15.27:0/2892948815 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbc7c08e030 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.298+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc8c069640 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.300+0000 7fbc89600640 1 -- 172.21.15.27:0/2892948815 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7fbc7c059510 con 0x7fbc8c0c6cc0 2024-03-28T18:56:06.299 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:06.300 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:56:06.300 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:56:06.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc6c06bc00 msgr2=0x7fbc6c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:06.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc6c06bc00 0x7fbc6c06e0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbc78034000 tx=0x7fbc7802f000 comp rx=0 tx=0).stop 2024-03-28T18:56:06.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c0c6cc0 msgr2=0x7fbc8c0abb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:06.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c0abb90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00c8a0 tx=0x7fbc7c00cd60 comp rx=0 tx=0).stop 2024-03-28T18:56:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 shutdown_connections 2024-03-28T18:56:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc6c06bc00 0x7fbc6c06e0c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc8c155b40 0x7fbc8c1478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc8c152a70 0x7fbc8c1473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.302+0000 7fbc93000640 1 --2- 172.21.15.27:0/2892948815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c0c6cc0 0x7fbc8c0abb90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.303+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 >> 172.21.15.27:0/2892948815 conn(0x7fbc8c06b2a0 msgr2=0x7fbc8c061b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.303+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 shutdown_connections 2024-03-28T18:56:06.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.303+0000 7fbc93000640 1 -- 172.21.15.27:0/2892948815 wait complete. 2024-03-28T18:56:06.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T18:55:33.955573+0000', 'modified': '2024-03-28T18:55:38.022816+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.155:6834/1270773218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6832', 'nonce': 1270773218}, {'type': 'v1', 'addr': '172.21.15.155:6834', 'nonce': 1270773218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'a', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4273, 'qdb_cluster': [4273]} 2024-03-28T18:56:06.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T18:56:06.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-28T18:56:06.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.950+0000 7f0371000640 1 Processor -- start 2024-03-28T18:56:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.951+0000 7f0371000640 1 -- start start 2024-03-28T18:56:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.952+0000 7f0371000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f036c14f920 0x7f036c0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.952+0000 7f0371000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036c06d960 con 0x7f036c158f00 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.952+0000 7f0371000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036c155b20 con 0x7f036c0c8f30 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.952+0000 7f0371000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036c1568e0 con 0x7f036c14f920 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.952+0000 7f036a800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f036c158f00 0x7f036c159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52268/0 (socket says 172.21.15.27:52268) 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.952+0000 7f036a800640 1 -- 172.21.15.27:0/3266789358 learned_addr learned my addr 172.21.15.27:0/3266789358 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.952+0000 7f0369e00640 1 --2- 172.21.15.27:0/3266789358 >> v2:172.21.15.155:3300/0 conn(0x7f036c14f920 0x7f036c0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102694995 0 0) 0x7f036c06d960 con 0x7f036c158f00 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0350003520 con 0x7f036c158f00 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4006420592 0 0) 0x7f0350003520 con 0x7f036c158f00 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 >> v2:172.21.15.155:3300/0 conn(0x7f036c14f920 msgr2=0x7f036c0c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 --2- 172.21.15.27:0/3266789358 >> v2:172.21.15.155:3300/0 conn(0x7f036c14f920 0x7f036c0c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 >> v1:172.21.15.155:6789/0 conn(0x7f036c0c8f30 legacy=0x7f036c0cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c155e00 con 0x7f036c158f00 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f03640019f0 con 0x7f036c158f00 2024-03-28T18:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 >> v1:172.21.15.27:6789/0 conn(0x7f036c158f00 legacy=0x7f036c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350003a20 0x7f0350003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0350004800 0x7f0350004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0350005210 0x7f035000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1568e0 con 0x7f0350003a20 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035000c140 con 0x7f0350005210 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.953+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035000c340 con 0x7f0350004800 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f0369e00640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350003a20 0x7f0350003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f036a800640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0350005210 0x7f035000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f0369e00640 1 -- 172.21.15.27:0/3266789358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0350004800 msgr2=0x7f0350004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f0369e00640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0350004800 0x7f0350004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f0369e00640 1 -- 172.21.15.27:0/3266789358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0350005210 msgr2=0x7f035000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f0369e00640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0350005210 0x7f035000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f0369e00640 1 -- 172.21.15.27:0/3266789358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0350003520 con 0x7f0350003a20 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f036a800640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0350005210 0x7f035000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.954+0000 7f036a800640 1 -- 172.21.15.27:0/3266789358 reap_dead start 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.955+0000 7f0369e00640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350003a20 0x7f0350003e00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f035c009ac0 tx=0x7f0350005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a544f0820d03c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.955+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c039070 con 0x7f0350003a20 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c02c6b0 con 0x7f0350003a20 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0369400640 1 -- 172.21.15.27:0/3266789358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c009690 con 0x7f0350003a20 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0371000640 1 -- 172.21.15.27:0/3266789358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350003a20 msgr2=0x7f0350003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0371000640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350003a20 0x7f0350003e00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f035c009ac0 tx=0x7f0350005830 comp rx=0 tx=0).stop 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0371000640 1 -- 172.21.15.27:0/3266789358 shutdown_connections 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0371000640 1 --2- 172.21.15.27:0/3266789358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0350003a20 0x7f0350003e00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0371000640 1 -- 172.21.15.27:0/3266789358 >> 172.21.15.27:0/3266789358 conn(0x7f036c066fb0 msgr2=0x7f036c0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0371000640 1 -- 172.21.15.27:0/3266789358 shutdown_connections 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.956+0000 7f0371000640 1 -- 172.21.15.27:0/3266789358 wait complete. 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.958+0000 7f0371000640 1 Processor -- start 2024-03-28T18:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.958+0000 7f0371000640 1 -- start start 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.959+0000 7f0371000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0350003a20 0x7f036c0acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.959+0000 7f0371000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f036c0c2ff0 0x7f036c0c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.959+0000 7f0371000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f036c1494c0 0x7f036c149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.959+0000 7f0371000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c14fd00 con 0x7f036c1494c0 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.959+0000 7f0371000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c156450 con 0x7f036c0c2ff0 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.959+0000 7f0371000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c0c9440 con 0x7f0350003a20 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f036a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f036c1494c0 0x7f036c149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f036a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f036c1494c0 0x7f036c149970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47914/0 (socket says 172.21.15.27:47914) 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f036a800640 1 -- 172.21.15.27:0/178750541 learned_addr learned my addr 172.21.15.27:0/178750541 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f0363e00640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f036c0c2ff0 0x7f036c0c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f0369e00640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0350003a20 0x7f036c0acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f036a800640 1 -- 172.21.15.27:0/178750541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0350003a20 msgr2=0x7f036c0acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f036a800640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0350003a20 0x7f036c0acec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f036a800640 1 -- 172.21.15.27:0/178750541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f036c0c2ff0 msgr2=0x7f036c0c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f036a800640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f036c0c2ff0 0x7f036c0c3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.960+0000 7f036a800640 1 -- 172.21.15.27:0/178750541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c0c39b0 con 0x7f036c1494c0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.961+0000 7f036a800640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f036c1494c0 0x7f036c149970 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f036c1592e0 tx=0x7f036400df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.961+0000 7f0362000640 1 -- 172.21.15.27:0/178750541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036400b860 con 0x7f036c1494c0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.961+0000 7f0362000640 1 -- 172.21.15.27:0/178750541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036400ba00 con 0x7f036c1494c0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.961+0000 7f0371000640 1 -- 172.21.15.27:0/178750541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c0c2230 con 0x7f036c1494c0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.961+0000 7f0362000640 1 -- 172.21.15.27:0/178750541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036400bdf0 con 0x7f036c1494c0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.961+0000 7f0371000640 1 -- 172.21.15.27:0/178750541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c0c27f0 con 0x7f036c1494c0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.963+0000 7f0362000640 1 -- 172.21.15.27:0/178750541 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0364007070 con 0x7f036c1494c0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.964+0000 7f0362000640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f034806bbd0 0x7f034806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.964+0000 7f0362000640 1 -- 172.21.15.27:0/178750541 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f036408aad0 con 0x7f036c1494c0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.965+0000 7f0369e00640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f034806bbd0 0x7f034806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.965+0000 7f0373e0d740 1 Processor -- start 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.965+0000 7f0373e0d740 1 -- start start 2024-03-28T18:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.966+0000 7f0369e00640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f034806bbd0 0x7f034806e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f035c036000 tx=0x7f035c030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.966+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1dee670 0x561fa1e28f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.966+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e294a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.966+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e2a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.966+0000 7f0373e0d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa1d52c90 con 0x561fa1def700 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.966+0000 7f0373e0d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa1cfb980 con 0x561fa1deea50 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.966+0000 7f0373e0d740 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fa1d4ed30 con 0x561fa1dee670 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e2a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e2a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47918/0 (socket says 172.21.15.27:47918) 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 -- 172.21.15.27:0/2953312343 learned_addr learned my addr 172.21.15.27:0/2953312343 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f0369e00640 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1dee670 0x561fa1e28f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f0363e00640 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e294a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 -- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1dee670 msgr2=0x561fa1e28f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1dee670 0x561fa1e28f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 -- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 msgr2=0x561fa1e294a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e294a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 -- 172.21.15.27:0/2953312343 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fa1d13510 con 0x561fa1def700 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f0363e00640 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e294a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f036a800640 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e2a4d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0364060d60 tx=0x7f0364058db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff866b07a5dc8ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f035aa00640 1 -- 172.21.15.27:0/2953312343 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0364056f30 con 0x561fa1def700 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.967+0000 7f035aa00640 1 -- 172.21.15.27:0/2953312343 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03640552a0 con 0x561fa1def700 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f035aa00640 1 -- 172.21.15.27:0/2953312343 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03640582c0 con 0x561fa1def700 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 -- 172.21.15.27:0/2953312343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 msgr2=0x561fa1e2a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e2a4d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0364060d60 tx=0x7f0364058db0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 -- 172.21.15.27:0/2953312343 shutdown_connections 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e2a4d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e294a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2953312343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1dee670 0x561fa1e28f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 -- 172.21.15.27:0/2953312343 >> 172.21.15.27:0/2953312343 conn(0x561fa1dead40 msgr2=0x561fa1e0f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 -- 172.21.15.27:0/2953312343 shutdown_connections 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.968+0000 7f0373e0d740 1 -- 172.21.15.27:0/2953312343 wait complete. 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.969+0000 7f0373e0d740 1 -- start start 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.969+0000 7f0373e0d740 1 Processor -- start 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.970+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1deea50 0x561fa1e0e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.970+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1def700 0x561fa1e3e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.970+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1e115d0 0x561fa1e0c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.970+0000 7f0373e0d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa1e2d290 con 0x561fa1deea50 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.970+0000 7f0373e0d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa1d0edd0 con 0x561fa1e115d0 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.970+0000 7f0373e0d740 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fa1e2be70 con 0x561fa1def700 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.970+0000 7f0373e0d740 10 client.0 _ll_register_callbacks cb 0x561fa1a66ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1deea50 0x561fa1e0e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1deea50 0x561fa1e0e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47922/0 (socket says 172.21.15.27:47922) 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 -- 172.21.15.27:0/399101744 learned_addr learned my addr 172.21.15.27:0/399101744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0363e00640 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1def700 0x561fa1e3e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f036a800640 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1e115d0 0x561fa1e0c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 -- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1def700 msgr2=0x561fa1e3e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1def700 0x561fa1e3e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 -- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1e115d0 msgr2=0x561fa1e0c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1e115d0 0x561fa1e0c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 -- 172.21.15.27:0/399101744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fa1a672d0 con 0x561fa1deea50 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f0369e00640 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1deea50 0x561fa1e0e9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f035c02c320 tx=0x7f035c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a903b0bfbefc09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.971+0000 7f034f400640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f034f400640 1 -- 172.21.15.27:0/399101744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c04c020 con 0x561fa1deea50 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f034f400640 1 -- 172.21.15.27:0/399101744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f035c02ce00 con 0x561fa1deea50 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f0373e0d740 1 -- 172.21.15.27:0/399101744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03640552a0 con 0x561fa1deea50 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f034f400640 1 -- 172.21.15.27:0/399101744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f035c056050 con 0x561fa1deea50 2024-03-28T18:56:07.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f0373e0d740 10 client.4788 fetch_fsmap learned FSMap version 15 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f0373e0d740 1 -- 172.21.15.27:0/399101744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x561fa1b316b0 con 0x561fa1deea50 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f034f400640 1 -- 172.21.15.27:0/399101744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03640552a0 con 0x561fa1deea50 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f034f400640 1 -- 172.21.15.27:0/399101744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 15) v1 ==== 3918+0+0 (secure 0 0 0) 0x7f035c059020 con 0x561fa1deea50 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f0373e0d740 10 client.4788 fetch_fsmap finished waiting for FSMap version 15 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f0373e0d740 10 client.4788 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.972+0000 7f0373e0d740 20 client.4788 populate_metadata read hostname 'smithi027' 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.973+0000 7f0373e0d740 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] conn(0x561fa1e3af10 0x561fa1e3b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.973+0000 7f0373e0d740 4 client.4788 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.973+0000 7f0373e0d740 1 -- 172.21.15.27:0/399101744 --> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561fa1cf4520 con 0x561fa1e3af10 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.973+0000 7f0363e00640 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] conn(0x561fa1e3af10 0x561fa1e3b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.974+0000 7f0363e00640 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] conn(0x561fa1e3af10 0x561fa1e3b2f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=340652ad46236751 server_cookie=12a97004194cc1f3 in_seq=0 out_seq=0 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.974+0000 7f034f400640 10 client.4788 ms_handle_connect on v2:172.21.15.155:6832/1270773218 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f034f400640 1 -- 172.21.15.27:0/399101744 <== mds.0 v2:172.21.15.155:6832/1270773218 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561fa1cf4520 con 0x561fa1e3af10 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f034f400640 10 client.4788 handle_command_reply: tid=0 2024-03-28T18:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 client.4788 shutdown 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 -- 172.21.15.27:0/399101744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1deea50 msgr2=0x561fa1e0e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1deea50 0x561fa1e0e9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f035c02c320 tx=0x7f035c0046c0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 -- 172.21.15.27:0/399101744 shutdown_connections 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] conn(0x561fa1e3af10 0x561fa1e3b2f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1e115d0 0x561fa1e0c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1def700 0x561fa1e3e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 --2- 172.21.15.27:0/399101744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1deea50 0x561fa1e0e9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f034f400640 0 client.4788 ms_handle_reset on v2:172.21.15.155:6832/1270773218 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.976+0000 7f0373e0d740 1 -- 172.21.15.27:0/399101744 >> 172.21.15.27:0/399101744 conn(0x561fa1dee670 msgr2=0x561fa1e2bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.977+0000 7f0373e0d740 1 -- 172.21.15.27:0/399101744 shutdown_connections 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.977+0000 7f0373e0d740 1 -- 172.21.15.27:0/399101744 wait complete. 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.977+0000 7f0373e0d740 20 client.4788 trim_cache size 0 max 16384 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.995+0000 7f0373e0d740 1 Processor -- start 2024-03-28T18:56:07.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.995+0000 7f0373e0d740 1 -- start start 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e3b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e11e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1e115d0 0x561fa1e123c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0373e0d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1fb020 con 0x561fa1def700 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0373e0d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1f8cd0 con 0x561fa1deea50 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0373e0d740 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c1f92d0 con 0x561fa1e115d0 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0363e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e11e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0363e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e11e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47936/0 (socket says 172.21.15.27:47936) 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.996+0000 7f0363e00640 1 -- 172.21.15.27:0/2283358752 learned_addr learned my addr 172.21.15.27:0/2283358752 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f036a800640 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1e115d0 0x561fa1e123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f0363e00640 1 -- 172.21.15.27:0/2283358752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1e115d0 msgr2=0x561fa1e123c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f0363e00640 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1e115d0 0x561fa1e123c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f0363e00640 1 -- 172.21.15.27:0/2283358752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 msgr2=0x561fa1e3b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f0369e00640 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e3b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f0363e00640 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e3b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f0363e00640 1 -- 172.21.15.27:0/2283358752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fa1e0de80 con 0x561fa1def700 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f0363e00640 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e11e80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x561fa1deee70 tx=0x7f0354009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f98028b4b407da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f035a000640 1 -- 172.21.15.27:0/2283358752 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354004e70 con 0x561fa1def700 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f035a000640 1 -- 172.21.15.27:0/2283358752 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035401a030 con 0x561fa1def700 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.997+0000 7f035a000640 1 -- 172.21.15.27:0/2283358752 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354001850 con 0x561fa1def700 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 -- 172.21.15.27:0/2283358752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 msgr2=0x561fa1e11e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e11e80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x561fa1deee70 tx=0x7f0354009040 comp rx=0 tx=0).stop 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 -- 172.21.15.27:0/2283358752 shutdown_connections 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1e115d0 0x561fa1e123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1def700 0x561fa1e11e80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2283358752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1deea50 0x561fa1e3b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 -- 172.21.15.27:0/2283358752 >> 172.21.15.27:0/2283358752 conn(0x561fa1dee670 msgr2=0x561fa1e0e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 -- 172.21.15.27:0/2283358752 shutdown_connections 2024-03-28T18:56:07.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.998+0000 7f0373e0d740 1 -- 172.21.15.27:0/2283358752 wait complete. 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.999+0000 7f0373e0d740 1 -- start start 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:06.999+0000 7f0373e0d740 1 Processor -- start 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1deea50 0x561fa1e9fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1def700 0x561fa1ea5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f0373e0d740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1e115d0 0x561fa1e0e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f0373e0d740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa1e0cde0 con 0x561fa1e115d0 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f0373e0d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa1e3ed60 con 0x561fa1def700 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f0373e0d740 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x561fa1e3a960 con 0x561fa1deea50 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f0373e0d740 10 client.0 _ll_register_callbacks cb 0x7f03500036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f036a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1e115d0 0x561fa1e0e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f036a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1e115d0 0x561fa1e0e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47940/0 (socket says 172.21.15.27:47940) 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.001+0000 7f036a800640 1 -- 172.21.15.27:0/2861371337 learned_addr learned my addr 172.21.15.27:0/2861371337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f0363e00640 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1def700 0x561fa1ea5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f0369e00640 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1deea50 0x561fa1e9fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f036a800640 1 -- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1deea50 msgr2=0x561fa1e9fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f036a800640 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1deea50 0x561fa1e9fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f036a800640 1 -- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1def700 msgr2=0x561fa1ea5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f036a800640 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1def700 0x561fa1ea5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f036a800640 1 -- 172.21.15.27:0/2861371337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fa1e0f370 con 0x561fa1e115d0 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f0363e00640 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1def700 0x561fa1ea5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f036a800640 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1e115d0 0x561fa1e0e620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f036406f040 tx=0x7f036405b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6566a708d75b1f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f034f400640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f034f400640 1 -- 172.21.15.27:0/2861371337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0364075060 con 0x561fa1e115d0 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f034f400640 1 -- 172.21.15.27:0/2861371337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f0364058080 con 0x561fa1e115d0 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.002+0000 7f0373e0d740 1 -- 172.21.15.27:0/2861371337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f035401a030 con 0x561fa1e115d0 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f034f400640 1 -- 172.21.15.27:0/2861371337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0364058a00 con 0x561fa1e115d0 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f0373e0d740 10 client.4794 fetch_fsmap learned FSMap version 15 2024-03-28T18:56:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f0373e0d740 1 -- 172.21.15.27:0/2861371337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x561fa1ea6430 con 0x561fa1e115d0 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f034f400640 1 -- 172.21.15.27:0/2861371337 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035401a030 con 0x561fa1e115d0 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f034f400640 1 -- 172.21.15.27:0/2861371337 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 15) v1 ==== 3918+0+0 (secure 0 0 0) 0x7f0364054980 con 0x561fa1e115d0 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f0373e0d740 10 client.4794 fetch_fsmap finished waiting for FSMap version 15 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f0373e0d740 10 client.4794 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f0373e0d740 20 client.4794 populate_metadata read hostname 'smithi027' 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] conn(0x561fa1e0eb60 0x561fa1e2d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f0373e0d740 4 client.4794 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.003+0000 7f0373e0d740 1 -- 172.21.15.27:0/2861371337 --> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561fa1d601d0 con 0x561fa1e0eb60 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.004+0000 7f0369e00640 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] conn(0x561fa1e0eb60 0x561fa1e2d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.004+0000 7f0369e00640 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] conn(0x561fa1e0eb60 0x561fa1e2d9b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aeffbb1872ac5f47 server_cookie=ba34553f9ac330b1 in_seq=0 out_seq=0 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.004+0000 7f034f400640 10 client.4794 ms_handle_connect on v2:172.21.15.155:6832/1270773218 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f034f400640 1 -- 172.21.15.27:0/2861371337 <== mds.0 v2:172.21.15.155:6832/1270773218 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x561fa1d601d0 con 0x561fa1e0eb60 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f034f400640 10 client.4794 handle_command_reply: tid=0 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 client.4794 shutdown 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 -- 172.21.15.27:0/2861371337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1e115d0 msgr2=0x561fa1e0e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1e115d0 0x561fa1e0e620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f036406f040 tx=0x7f036405b070 comp rx=0 tx=0).stop 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 -- 172.21.15.27:0/2861371337 shutdown_connections 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:6832/1270773218,v1:172.21.15.155:6834/1270773218] conn(0x561fa1e0eb60 0x561fa1e2d9b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561fa1e115d0 0x561fa1e0e620 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa1def700 0x561fa1ea5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 --2- 172.21.15.27:0/2861371337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x561fa1deea50 0x561fa1e9fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 -- 172.21.15.27:0/2861371337 >> 172.21.15.27:0/2861371337 conn(0x561fa1dee670 msgr2=0x561fa1e3b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f034f400640 0 client.4794 ms_handle_reset on v2:172.21.15.155:6832/1270773218 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 -- 172.21.15.27:0/2861371337 shutdown_connections 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.005+0000 7f0373e0d740 1 -- 172.21.15.27:0/2861371337 wait complete. 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.006+0000 7f0373e0d740 20 client.4794 trim_cache size 0 max 16384 2024-03-28T18:56:07.060 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "c27ecc06-ff19-45f9-bdba-d624e1db1fbd", 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4273, 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "a", 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 32.521162998999998, 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 9, 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 24, 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 20, 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 37.838600694 2024-03-28T18:56:07.061 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.006+0000 7f0371000640 1 -- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f034806bbd0 msgr2=0x7f034806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.006+0000 7f0371000640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f034806bbd0 0x7f034806e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f035c036000 tx=0x7f035c030000 comp rx=0 tx=0).stop 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.006+0000 7f0371000640 1 -- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f036c1494c0 msgr2=0x7f036c149970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.006+0000 7f0371000640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f036c1494c0 0x7f036c149970 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f036c1592e0 tx=0x7f036400df60 comp rx=0 tx=0).stop 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.006+0000 7f0371000640 1 -- 172.21.15.27:0/178750541 shutdown_connections 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.006+0000 7f0371000640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f034806bbd0 0x7f034806e090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.006+0000 7f0371000640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f036c1494c0 0x7f036c149970 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.007+0000 7f0371000640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f036c0c2ff0 0x7f036c0c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.007+0000 7f0371000640 1 --2- 172.21.15.27:0/178750541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0350003a20 0x7f036c0acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.007+0000 7f0371000640 1 -- 172.21.15.27:0/178750541 >> 172.21.15.27:0/178750541 conn(0x7f036c066fb0 msgr2=0x7f036c14ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.007+0000 7f0371000640 1 -- 172.21.15.27:0/178750541 shutdown_connections 2024-03-28T18:56:07.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.007+0000 7f0371000640 1 -- 172.21.15.27:0/178750541 wait complete. 2024-03-28T18:56:07.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:07.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.286+0000 7fd147e00640 1 Processor -- start 2024-03-28T18:56:07.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.287+0000 7fd147e00640 1 -- start start 2024-03-28T18:56:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.288+0000 7fd147e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd1480c3910 0x7fd1481529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.288+0000 7fd147e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd148152f20 0x7fd148155630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.288+0000 7fd147e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1480c73f0 con 0x7fd148155b70 2024-03-28T18:56:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.288+0000 7fd147e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1480c7110 con 0x7fd1480c3910 2024-03-28T18:56:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.288+0000 7fd147e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd148072a40 con 0x7fd148152f20 2024-03-28T18:56:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.288+0000 7fd147400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd148155b70 0x7fd148158ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52278/0 (socket says 172.21.15.27:52278) 2024-03-28T18:56:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.288+0000 7fd147400640 1 -- 172.21.15.27:0/1627194576 learned_addr learned my addr 172.21.15.27:0/1627194576 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:07.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd146a00640 1 --2- 172.21.15.27:0/1627194576 >> v2:172.21.15.155:3300/0 conn(0x7fd1480c3910 0x7fd1481529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd146000640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd148152f20 0x7fd148155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914641859 0 0) 0x7fd1480c73f0 con 0x7fd148155b70 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd12c003520 con 0x7fd148155b70 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1307383371 0 0) 0x7fd12c003520 con 0x7fd148155b70 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd148152f20 msgr2=0x7fd148155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd148152f20 0x7fd148155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 >> v2:172.21.15.155:3300/0 conn(0x7fd1480c3910 msgr2=0x7fd1481529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 --2- 172.21.15.27:0/1627194576 >> v2:172.21.15.155:3300/0 conn(0x7fd1480c3910 0x7fd1481529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148072d20 con 0x7fd148155b70 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd1400019f0 con 0x7fd148155b70 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 >> v1:172.21.15.27:6789/0 conn(0x7fd148155b70 legacy=0x7fd148158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd12c0039b0 0x7fd12c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd12c0043a0 0x7fd12c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd148152f20 0x7fd12c004ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.289+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480c7110 con 0x7fd12c0043a0 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd148072a40 con 0x7fd12c0039b0 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c00b9c0 con 0x7fd148152f20 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd146000640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd12c0043a0 0x7fd12c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd147400640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd148152f20 0x7fd12c004ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd146a00640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd12c0039b0 0x7fd12c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd146000640 1 -- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd148152f20 msgr2=0x7fd12c004ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd146000640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd148152f20 0x7fd12c004ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd146000640 1 -- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd12c0039b0 msgr2=0x7fd12c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd146000640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd12c0039b0 0x7fd12c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.290+0000 7fd146000640 1 -- 172.21.15.27:0/1627194576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148072d20 con 0x7fd12c0043a0 2024-03-28T18:56:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.291+0000 7fd146000640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd12c0043a0 0x7fd12c004880 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd134009ac0 tx=0x7fd12c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45076bd2c0532fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.291+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134039070 con 0x7fd12c0043a0 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13402c6b0 con 0x7fd12c0043a0 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd145600640 1 -- 172.21.15.27:0/1627194576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134009690 con 0x7fd12c0043a0 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd147e00640 1 -- 172.21.15.27:0/1627194576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd12c0043a0 msgr2=0x7fd12c004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd147e00640 1 --2- 172.21.15.27:0/1627194576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd12c0043a0 0x7fd12c004880 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd134009ac0 tx=0x7fd12c005830 comp rx=0 tx=0).stop 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd147400640 1 -- 172.21.15.27:0/1627194576 reap_dead start 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd147e00640 1 -- 172.21.15.27:0/1627194576 shutdown_connections 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd147e00640 1 -- 172.21.15.27:0/1627194576 >> 172.21.15.27:0/1627194576 conn(0x7fd14806b0c0 msgr2=0x7fd14806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd147e00640 1 -- 172.21.15.27:0/1627194576 shutdown_connections 2024-03-28T18:56:07.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.292+0000 7fd147e00640 1 -- 172.21.15.27:0/1627194576 wait complete. 2024-03-28T18:56:07.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.294+0000 7fd147e00640 1 Processor -- start 2024-03-28T18:56:07.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.294+0000 7fd147e00640 1 -- start start 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd12c0043a0 0x7fd1480c13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1480c0650 0x7fd1480bd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1480bb780 0x7fd1480bbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd148067b90 con 0x7fd1480bb780 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480732e0 con 0x7fd1480c0650 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd148067d10 con 0x7fd12c0043a0 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1480bb780 0x7fd1480bbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1480bb780 0x7fd1480bbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47958/0 (socket says 172.21.15.27:47958) 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.295+0000 7fd147400640 1 -- 172.21.15.27:0/1135685365 learned_addr learned my addr 172.21.15.27:0/1135685365 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.296+0000 7fd146a00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd12c0043a0 0x7fd1480c13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.296+0000 7fd146000640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1480c0650 0x7fd1480bd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.296+0000 7fd147400640 1 -- 172.21.15.27:0/1135685365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd12c0043a0 msgr2=0x7fd1480c13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.296+0000 7fd147400640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd12c0043a0 0x7fd1480c13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.296+0000 7fd147400640 1 -- 172.21.15.27:0/1135685365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1480c0650 msgr2=0x7fd1480bd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.296+0000 7fd147400640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1480c0650 0x7fd1480bd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.296+0000 7fd147400640 1 -- 172.21.15.27:0/1135685365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14814fa20 con 0x7fd1480bb780 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.296+0000 7fd147400640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1480bb780 0x7fd1480bbc00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd140001ff0 tx=0x7fd14000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.297+0000 7fd13ea00640 1 -- 172.21.15.27:0/1135685365 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14000edc0 con 0x7fd1480bb780 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.297+0000 7fd13ea00640 1 -- 172.21.15.27:0/1135685365 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd140018070 con 0x7fd1480bb780 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.297+0000 7fd13ea00640 1 -- 172.21.15.27:0/1135685365 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140013660 con 0x7fd1480bb780 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.297+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14814f580 con 0x7fd1480bb780 2024-03-28T18:56:07.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.297+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1480bdab0 con 0x7fd1480bb780 2024-03-28T18:56:07.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.298+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd148069640 con 0x7fd1480bb780 2024-03-28T18:56:07.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.299+0000 7fd13ea00640 1 -- 172.21.15.27:0/1135685365 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd140013800 con 0x7fd1480bb780 2024-03-28T18:56:07.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.300+0000 7fd13ea00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.300+0000 7fd13ea00640 1 -- 172.21.15.27:0/1135685365 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fd14008b090 con 0x7fd1480bb780 2024-03-28T18:56:07.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.300+0000 7fd146a00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.301+0000 7fd146a00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd138009a90 tx=0x7fd138008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.309+0000 7fd13ea00640 1 -- 172.21.15.27:0/1135685365 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd1400598e0 con 0x7fd1480bb780 2024-03-28T18:56:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.511+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1480c4240 con 0x7fd1480bb780 2024-03-28T18:56:07.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.514+0000 7fd13ea00640 1 -- 172.21.15.27:0/1135685365 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+20307 (secure 0 0 0) 0x7fd1400591e0 con 0x7fd1480bb780 2024-03-28T18:56:07.515 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:07.515 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:46.587251+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.516+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 msgr2=0x7fd12806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.516+0000 7fd147e00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd138009a90 tx=0x7fd138008040 comp rx=0 tx=0).stop 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.516+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1480bb780 msgr2=0x7fd1480bbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.516+0000 7fd147e00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1480bb780 0x7fd1480bbc00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd140001ff0 tx=0x7fd14000e530 comp rx=0 tx=0).stop 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.517+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 shutdown_connections 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.517+0000 7fd147e00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.517+0000 7fd147e00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1480bb780 0x7fd1480bbc00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.517+0000 7fd147e00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1480c0650 0x7fd1480bd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.517+0000 7fd147e00640 1 --2- 172.21.15.27:0/1135685365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd12c0043a0 0x7fd1480c13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.517+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 >> 172.21.15.27:0/1135685365 conn(0x7fd14806b0c0 msgr2=0x7fd148066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.517+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 shutdown_connections 2024-03-28T18:56:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.517+0000 7fd147e00640 1 -- 172.21.15.27:0/1135685365 wait complete. 2024-03-28T18:56:07.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-03-28T18:55:32.332283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-03-28T18:55:32.983411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-03-28T18:55:43.415397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-03-28T18:55:44.084162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:56:07.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:56:07.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.761+0000 7f2d38a00640 1 Processor -- start 2024-03-28T18:56:07.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.762+0000 7f2d38a00640 1 -- start start 2024-03-28T18:56:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.764+0000 7f2d38a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2d341540f0 0x7f2d340a7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.764+0000 7f2d38a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d340c7310 con 0x7f2d34150db0 2024-03-28T18:56:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.764+0000 7f2d31e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2d34150db0 0x7f2d340c7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52292/0 (socket says 172.21.15.27:52292) 2024-03-28T18:56:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.764+0000 7f2d31e00640 1 -- 172.21.15.27:0/4260333545 learned_addr learned my addr 172.21.15.27:0/4260333545 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.764+0000 7f2d38a00640 1 -- 172.21.15.27:0/4260333545 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d34151190 con 0x7f2d340abcd0 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.764+0000 7f2d38a00640 1 -- 172.21.15.27:0/4260333545 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d34153c60 con 0x7f2d341540f0 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d32800640 1 --2- 172.21.15.27:0/4260333545 >> v2:172.21.15.155:3300/0 conn(0x7f2d341540f0 0x7f2d340a7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390751270 0 0) 0x7f2d340c7310 con 0x7f2d34150db0 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d18003520 con 0x7f2d34150db0 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975955490 0 0) 0x7f2d18003520 con 0x7f2d34150db0 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 >> v2:172.21.15.155:3300/0 conn(0x7f2d341540f0 msgr2=0x7f2d340a7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d30a00640 1 --2- 172.21.15.27:0/4260333545 >> v2:172.21.15.155:3300/0 conn(0x7f2d341540f0 0x7f2d340a7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 >> v1:172.21.15.155:6789/0 conn(0x7f2d340abcd0 legacy=0x7f2d340c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d340c8630 con 0x7f2d34150db0 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.765+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f2d24001670 con 0x7f2d34150db0 2024-03-28T18:56:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 >> v1:172.21.15.27:6789/0 conn(0x7f2d34150db0 legacy=0x7f2d340c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d30a00640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d18003a20 0x7f2d18003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d30a00640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d30a00640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d180051c0 0x7f2d1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34153c60 con 0x7f2d18004800 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1800c0f0 con 0x7f2d18003a20 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1800c2f0 con 0x7f2d180051c0 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d31400640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d31e00640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d18003a20 0x7f2d18003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.766+0000 7f2d32800640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d180051c0 0x7f2d1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.767+0000 7f2d31400640 1 -- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d180051c0 msgr2=0x7f2d1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.767+0000 7f2d31400640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d180051c0 0x7f2d1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.767+0000 7f2d31400640 1 -- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d18003a20 msgr2=0x7f2d18003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.767+0000 7f2d31400640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d18003a20 0x7f2d18003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.767+0000 7f2d31400640 1 -- 172.21.15.27:0/4260333545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18003520 con 0x7f2d18004800 2024-03-28T18:56:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.767+0000 7f2d32800640 1 -- 172.21.15.27:0/4260333545 reap_dead start 2024-03-28T18:56:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.767+0000 7f2d31e00640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d18003a20 0x7f2d18003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:07.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.768+0000 7f2d31400640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d18004c80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2d20001750 tx=0x7f2d18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572d83bd7d93af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.769+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20001370 con 0x7f2d18004800 2024-03-28T18:56:07.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.769+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d200052d0 con 0x7f2d18004800 2024-03-28T18:56:07.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.769+0000 7f2d30a00640 1 -- 172.21.15.27:0/4260333545 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20003830 con 0x7f2d18004800 2024-03-28T18:56:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.769+0000 7f2d38a00640 1 -- 172.21.15.27:0/4260333545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 msgr2=0x7f2d18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.769+0000 7f2d38a00640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d18004c80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2d20001750 tx=0x7f2d18005830 comp rx=0 tx=0).stop 2024-03-28T18:56:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.769+0000 7f2d38a00640 1 -- 172.21.15.27:0/4260333545 shutdown_connections 2024-03-28T18:56:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.769+0000 7f2d38a00640 1 --2- 172.21.15.27:0/4260333545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d18004c80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.769+0000 7f2d38a00640 1 -- 172.21.15.27:0/4260333545 >> 172.21.15.27:0/4260333545 conn(0x7f2d340aa9d0 msgr2=0x7f2d340aadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.770+0000 7f2d38a00640 1 -- 172.21.15.27:0/4260333545 shutdown_connections 2024-03-28T18:56:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.770+0000 7f2d38a00640 1 -- 172.21.15.27:0/4260333545 wait complete. 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.771+0000 7f2d38a00640 1 Processor -- start 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.771+0000 7f2d38a00640 1 -- start start 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.772+0000 7f2d38a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d341dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.772+0000 7f2d38a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341ecda0 0x7f2d340c0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.772+0000 7f2d38a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d340bfc50 0x7f2d340c1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.772+0000 7f2d38a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340c8910 con 0x7f2d18004800 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.772+0000 7f2d38a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341538f0 con 0x7f2d341ecda0 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.772+0000 7f2d38a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340c6c00 con 0x7f2d340bfc50 2024-03-28T18:56:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d341dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d341dffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47984/0 (socket says 172.21.15.27:47984) 2024-03-28T18:56:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 -- 172.21.15.27:0/3947822143 learned_addr learned my addr 172.21.15.27:0/3947822143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 -- 172.21.15.27:0/3947822143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d340bfc50 msgr2=0x7f2d340c1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d340bfc50 0x7f2d340c1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 -- 172.21.15.27:0/3947822143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341ecda0 msgr2=0x7f2d340c0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341ecda0 0x7f2d340c0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 -- 172.21.15.27:0/3947822143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d341e98f0 con 0x7f2d18004800 2024-03-28T18:56:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.773+0000 7f2d31e00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d341dffb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2d24000fd0 tx=0x7f2d2400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.774+0000 7f2d2a000640 1 -- 172.21.15.27:0/3947822143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400c610 con 0x7f2d18004800 2024-03-28T18:56:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.774+0000 7f2d2a000640 1 -- 172.21.15.27:0/3947822143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24018070 con 0x7f2d18004800 2024-03-28T18:56:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.774+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d341ec2c0 con 0x7f2d18004800 2024-03-28T18:56:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.774+0000 7f2d2a000640 1 -- 172.21.15.27:0/3947822143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400c9b0 con 0x7f2d18004800 2024-03-28T18:56:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.774+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d341ec880 con 0x7f2d18004800 2024-03-28T18:56:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.775+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d341ec0b0 con 0x7f2d18004800 2024-03-28T18:56:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.777+0000 7f2d2a000640 1 -- 172.21.15.27:0/3947822143 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2d2400cb50 con 0x7f2d18004800 2024-03-28T18:56:07.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.778+0000 7f2d2a000640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2d1406bbd0 0x7f2d1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:07.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.778+0000 7f2d2a000640 1 -- 172.21.15.27:0/3947822143 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f2d2408aa90 con 0x7f2d18004800 2024-03-28T18:56:07.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.778+0000 7f2d31400640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2d1406bbd0 0x7f2d1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:07.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.779+0000 7f2d31400640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2d1406bbd0 0x7f2d1406e090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2d20033000 tx=0x7f2d2002f000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:07.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:07.783+0000 7f2d2a000640 1 -- 172.21.15.27:0/3947822143 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2d24058bf0 con 0x7f2d18004800 2024-03-28T18:56:08.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.012+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d3404f470 con 0x7f2d18004800 2024-03-28T18:56:08.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.013+0000 7f2d2a000640 1 -- 172.21.15.27:0/3947822143 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d2405cc90 con 0x7f2d18004800 2024-03-28T18:56:08.014 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.016+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2d1406bbd0 msgr2=0x7f2d1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.016+0000 7f2d38a00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2d1406bbd0 0x7f2d1406e090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2d20033000 tx=0x7f2d2002f000 comp rx=0 tx=0).stop 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.016+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 msgr2=0x7f2d341dffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.016+0000 7f2d38a00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d341dffb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2d24000fd0 tx=0x7f2d2400ec10 comp rx=0 tx=0).stop 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.017+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 shutdown_connections 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.017+0000 7f2d38a00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2d1406bbd0 0x7f2d1406e090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.017+0000 7f2d38a00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d340bfc50 0x7f2d340c1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.017+0000 7f2d38a00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341ecda0 0x7f2d340c0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.017+0000 7f2d38a00640 1 --2- 172.21.15.27:0/3947822143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d18004800 0x7f2d341dffb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.017+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 >> 172.21.15.27:0/3947822143 conn(0x7f2d340aa9d0 msgr2=0x7f2d340a8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:08.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.017+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 shutdown_connections 2024-03-28T18:56:08.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.017+0000 7f2d38a00640 1 -- 172.21.15.27:0/3947822143 wait complete. 2024-03-28T18:56:08.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-03-28T18:56:08.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.270+0000 7fc71f200640 1 Processor -- start 2024-03-28T18:56:08.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.270+0000 7fc71f200640 1 -- start start 2024-03-28T18:56:08.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc71f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc718156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc71f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7180c8ff0 0x7fc718159440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc71f200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc718159980 0x7fc7180ad040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc71f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc718158cd0 con 0x7fc7180c8ff0 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc71f200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc718158fb0 con 0x7fc718159980 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc71f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7181566c0 con 0x7fc718152ec0 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc717400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7180c8ff0 0x7fc718159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc717400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7180c8ff0 0x7fc718159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47998/0 (socket says 172.21.15.27:47998) 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc717400640 1 -- 172.21.15.27:0/1700406325 learned_addr learned my addr 172.21.15.27:0/1700406325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc717400640 1 -- 172.21.15.27:0/1700406325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 msgr2=0x7fc718156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc717e00640 1 --2- 172.21.15.27:0/1700406325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc718156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc71d200640 1 --2- 172.21.15.27:0/1700406325 >> v2:172.21.15.155:3300/0 conn(0x7fc718159980 0x7fc7180ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc717400640 1 --2- 172.21.15.27:0/1700406325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc718156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.272+0000 7fc717400640 1 -- 172.21.15.27:0/1700406325 >> v2:172.21.15.155:3300/0 conn(0x7fc718159980 msgr2=0x7fc7180ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:08.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.273+0000 7fc717400640 1 --2- 172.21.15.27:0/1700406325 >> v2:172.21.15.155:3300/0 conn(0x7fc718159980 0x7fc7180ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.273+0000 7fc717400640 1 -- 172.21.15.27:0/1700406325 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71806d500 con 0x7fc7180c8ff0 2024-03-28T18:56:08.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.273+0000 7fc71d200640 1 --2- 172.21.15.27:0/1700406325 >> v2:172.21.15.155:3300/0 conn(0x7fc718159980 0x7fc7180ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:08.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.274+0000 7fc717400640 1 --2- 172.21.15.27:0/1700406325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7180c8ff0 0x7fc718159440 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc7180cae60 tx=0x7fc70402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187508697522c5bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:08.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.274+0000 7fc716a00640 1 -- 172.21.15.27:0/1700406325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704038070 con 0x7fc7180c8ff0 2024-03-28T18:56:08.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.274+0000 7fc716a00640 1 -- 172.21.15.27:0/1700406325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704031740 con 0x7fc7180c8ff0 2024-03-28T18:56:08.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.274+0000 7fc716a00640 1 -- 172.21.15.27:0/1700406325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70402f050 con 0x7fc7180c8ff0 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 -- 172.21.15.27:0/1700406325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7180c8ff0 msgr2=0x7fc718159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 --2- 172.21.15.27:0/1700406325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7180c8ff0 0x7fc718159440 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc7180cae60 tx=0x7fc70402b7e0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 -- 172.21.15.27:0/1700406325 shutdown_connections 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 --2- 172.21.15.27:0/1700406325 >> v2:172.21.15.155:3300/0 conn(0x7fc718159980 0x7fc7180ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 --2- 172.21.15.27:0/1700406325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7180c8ff0 0x7fc718159440 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 --2- 172.21.15.27:0/1700406325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc718156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 -- 172.21.15.27:0/1700406325 >> 172.21.15.27:0/1700406325 conn(0x7fc7180671f0 msgr2=0x7fc718067600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 -- 172.21.15.27:0/1700406325 shutdown_connections 2024-03-28T18:56:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.275+0000 7fc71f200640 1 -- 172.21.15.27:0/1700406325 wait complete. 2024-03-28T18:56:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.276+0000 7fc71f200640 1 Processor -- start 2024-03-28T18:56:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.277+0000 7fc71f200640 1 -- start start 2024-03-28T18:56:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.277+0000 7fc71f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7180c8ff0 0x7fc7181484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.277+0000 7fc71f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc7180bef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc718147fc0 0x7fc7181eb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7180c6060 con 0x7fc718147fc0 2024-03-28T18:56:08.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7180c5d80 con 0x7fc7180c8ff0 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71806d8f0 con 0x7fc718152ec0 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc718147fc0 0x7fc7181eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc718147fc0 0x7fc7181eb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48012/0 (socket says 172.21.15.27:48012) 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71d200640 1 -- 172.21.15.27:0/3334798892 learned_addr learned my addr 172.21.15.27:0/3334798892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc717e00640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7180c8ff0 0x7fc7181484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc717400640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc7180bef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71d200640 1 -- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 msgr2=0x7fc7180bef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71d200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc7180bef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71d200640 1 -- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7180c8ff0 msgr2=0x7fc7181484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71d200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7180c8ff0 0x7fc7181484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc71d200640 1 -- 172.21.15.27:0/3334798892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7180c0e80 con 0x7fc718147fc0 2024-03-28T18:56:08.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc717e00640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7180c8ff0 0x7fc7181484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:08.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.278+0000 7fc717400640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc7180bef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:08.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.279+0000 7fc71d200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc718147fc0 0x7fc7181eb260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc71000cdd0 tx=0x7fc71000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:08.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.279+0000 7fc70fe00640 1 -- 172.21.15.27:0/3334798892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710003b90 con 0x7fc718147fc0 2024-03-28T18:56:08.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.279+0000 7fc70fe00640 1 -- 172.21.15.27:0/3334798892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc710003d30 con 0x7fc718147fc0 2024-03-28T18:56:08.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.279+0000 7fc70fe00640 1 -- 172.21.15.27:0/3334798892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710004260 con 0x7fc718147fc0 2024-03-28T18:56:08.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.279+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7180c1160 con 0x7fc718147fc0 2024-03-28T18:56:08.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.279+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7180bf470 con 0x7fc718147fc0 2024-03-28T18:56:08.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.281+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71804f470 con 0x7fc718147fc0 2024-03-28T18:56:08.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.281+0000 7fc70fe00640 1 -- 172.21.15.27:0/3334798892 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc710004400 con 0x7fc718147fc0 2024-03-28T18:56:08.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.283+0000 7fc70fe00640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc6f806bbd0 0x7fc6f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.283+0000 7fc70fe00640 1 -- 172.21.15.27:0/3334798892 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fc71008b710 con 0x7fc718147fc0 2024-03-28T18:56:08.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.283+0000 7fc717e00640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc6f806bbd0 0x7fc6f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.284+0000 7fc717e00640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc6f806bbd0 0x7fc6f806e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc708006fd0 tx=0x7fc708008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:08.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.287+0000 7fc70fe00640 1 -- 172.21.15.27:0/3334798892 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc710059ec0 con 0x7fc718147fc0 2024-03-28T18:56:08.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.487+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc71806d6d0 con 0x7fc718147fc0 2024-03-28T18:56:08.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.488+0000 7fc70fe00640 1 -- 172.21.15.27:0/3334798892 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v24) v1 ==== 99+0+11 (secure 0 0 0) 0x7fc7100597c0 con 0x7fc718147fc0 2024-03-28T18:56:08.489 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.490+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc6f806bbd0 msgr2=0x7fc6f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.490+0000 7fc71f200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc6f806bbd0 0x7fc6f806e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc708006fd0 tx=0x7fc708008040 comp rx=0 tx=0).stop 2024-03-28T18:56:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.490+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc718147fc0 msgr2=0x7fc7181eb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.490+0000 7fc71f200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc718147fc0 0x7fc7181eb260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc71000cdd0 tx=0x7fc71000a710 comp rx=0 tx=0).stop 2024-03-28T18:56:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.491+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 shutdown_connections 2024-03-28T18:56:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.491+0000 7fc71f200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc6f806bbd0 0x7fc6f806e090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.491+0000 7fc71f200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc718147fc0 0x7fc7181eb260 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.491+0000 7fc71f200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc718152ec0 0x7fc7180bef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.491+0000 7fc71f200640 1 --2- 172.21.15.27:0/3334798892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7180c8ff0 0x7fc7181484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.491+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 >> 172.21.15.27:0/3334798892 conn(0x7fc7180671f0 msgr2=0x7fc71814b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:08.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.491+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 shutdown_connections 2024-03-28T18:56:08.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.491+0000 7fc71f200640 1 -- 172.21.15.27:0/3334798892 wait complete. 2024-03-28T18:56:08.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-03-28T18:56:08.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.746+0000 7fb7f5600640 1 Processor -- start 2024-03-28T18:56:08.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.747+0000 7fb7f5600640 1 -- start start 2024-03-28T18:56:08.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.748+0000 7fb7f5600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb7f00c5aa0 0x7fb7f0156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.748+0000 7fb7f5600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f00735f0 con 0x7fb7f0152840 2024-03-28T18:56:08.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.748+0000 7fb7f5600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f00c7d10 con 0x7fb7f00c8db0 2024-03-28T18:56:08.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.748+0000 7fb7f5600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f01565d0 con 0x7fb7f00c5aa0 2024-03-28T18:56:08.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.749+0000 7fb7f4c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb7f0152840 0x7fb7f015ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52300/0 (socket says 172.21.15.27:52300) 2024-03-28T18:56:08.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.749+0000 7fb7f4c00640 1 -- 172.21.15.27:0/3301296217 learned_addr learned my addr 172.21.15.27:0/3301296217 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:08.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.749+0000 7fb7efe00640 1 --2- 172.21.15.27:0/3301296217 >> v2:172.21.15.155:3300/0 conn(0x7fb7f00c5aa0 0x7fb7f0156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.749+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748985738 0 0) 0x7fb7f00735f0 con 0x7fb7f0152840 2024-03-28T18:56:08.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.749+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d4003520 con 0x7fb7f0152840 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.749+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 709561557 0 0) 0x7fb7d4003520 con 0x7fb7f0152840 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.749+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 >> v2:172.21.15.155:3300/0 conn(0x7fb7f00c5aa0 msgr2=0x7fb7f0156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.749+0000 7fb7ef400640 1 --2- 172.21.15.27:0/3301296217 >> v2:172.21.15.155:3300/0 conn(0x7fb7f00c5aa0 0x7fb7f0156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 >> v1:172.21.15.155:6789/0 conn(0x7fb7f00c8db0 legacy=0x7fb7f0072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f00c7ff0 con 0x7fb7f0152840 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fb7e80019f0 con 0x7fb7f0152840 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 >> v1:172.21.15.27:6789/0 conn(0x7fb7f0152840 legacy=0x7fb7f015ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40039b0 0x7fb7d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d4004800 0x7fb7d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f01565d0 con 0x7fb7d40051c0 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d400c160 con 0x7fb7d40039b0 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d400c360 con 0x7fb7d4004800 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7f4c00640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7efe00640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40039b0 0x7fb7d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.750+0000 7fb7e7e00640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d4004800 0x7fb7d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.751+0000 7fb7f4c00640 1 -- 172.21.15.27:0/3301296217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d4004800 msgr2=0x7fb7d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.751+0000 7fb7f4c00640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d4004800 0x7fb7d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.751+0000 7fb7f4c00640 1 -- 172.21.15.27:0/3301296217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40039b0 msgr2=0x7fb7d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:08.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.751+0000 7fb7f4c00640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40039b0 0x7fb7d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.751+0000 7fb7f4c00640 1 -- 172.21.15.27:0/3301296217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4003520 con 0x7fb7d40051c0 2024-03-28T18:56:08.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.752+0000 7fb7f4c00640 1 -- 172.21.15.27:0/3301296217 reap_dead start 2024-03-28T18:56:08.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.753+0000 7fb7f4c00640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7d400b9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800a860 tx=0x7fb7d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e802df4e89415f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:08.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.753+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e800a480 con 0x7fb7d40051c0 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.753+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e8034630 con 0x7fb7d40051c0 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.753+0000 7fb7ef400640 1 -- 172.21.15.27:0/3301296217 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8034a00 con 0x7fb7d40051c0 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.753+0000 7fb7f5600640 1 -- 172.21.15.27:0/3301296217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 msgr2=0x7fb7d400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.753+0000 7fb7f5600640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7d400b9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800a860 tx=0x7fb7d4005830 comp rx=0 tx=0).stop 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.754+0000 7fb7f5600640 1 -- 172.21.15.27:0/3301296217 shutdown_connections 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.754+0000 7fb7f5600640 1 --2- 172.21.15.27:0/3301296217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7d400b9c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.754+0000 7fb7f5600640 1 -- 172.21.15.27:0/3301296217 >> 172.21.15.27:0/3301296217 conn(0x7fb7f006b0c0 msgr2=0x7fb7f006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.754+0000 7fb7f5600640 1 -- 172.21.15.27:0/3301296217 shutdown_connections 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.754+0000 7fb7f5600640 1 -- 172.21.15.27:0/3301296217 wait complete. 2024-03-28T18:56:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.755+0000 7fb7f5600640 1 Processor -- start 2024-03-28T18:56:08.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.756+0000 7fb7f5600640 1 -- start start 2024-03-28T18:56:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.756+0000 7fb7f5600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7f01e7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.756+0000 7fb7f5600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7f014ad70 0x7fb7f01e8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.756+0000 7fb7f5600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7f014c850 0x7fb7f014ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.756+0000 7fb7f5600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0072100 con 0x7fb7d40051c0 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.756+0000 7fb7f5600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00c82d0 con 0x7fb7f014c850 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.756+0000 7fb7f5600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00c78c0 con 0x7fb7f014ad70 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7efe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7f01e7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7efe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7f01e7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48034/0 (socket says 172.21.15.27:48034) 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7efe00640 1 -- 172.21.15.27:0/3093908388 learned_addr learned my addr 172.21.15.27:0/3093908388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7f4c00640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7f014c850 0x7fb7f014ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7e7e00640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7f014ad70 0x7fb7f01e8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7efe00640 1 -- 172.21.15.27:0/3093908388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7f014ad70 msgr2=0x7fb7f01e8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7efe00640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7f014ad70 0x7fb7f01e8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7efe00640 1 -- 172.21.15.27:0/3093908388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7f014c850 msgr2=0x7fb7f014ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7efe00640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7f014c850 0x7fb7f014ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.757+0000 7fb7efe00640 1 -- 172.21.15.27:0/3093908388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f00b0d90 con 0x7fb7d40051c0 2024-03-28T18:56:08.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.758+0000 7fb7efe00640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7f01e7ed0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0011940 tx=0x7fb7e0011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.761+0000 7fb7ecc00640 1 -- 172.21.15.27:0/3093908388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e000faf0 con 0x7fb7d40051c0 2024-03-28T18:56:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.761+0000 7fb7ecc00640 1 -- 172.21.15.27:0/3093908388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e000fc90 con 0x7fb7d40051c0 2024-03-28T18:56:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.761+0000 7fb7ecc00640 1 -- 172.21.15.27:0/3093908388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0014440 con 0x7fb7d40051c0 2024-03-28T18:56:08.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.761+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f01eeb20 con 0x7fb7d40051c0 2024-03-28T18:56:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.761+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f014d840 con 0x7fb7d40051c0 2024-03-28T18:56:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.763+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f004f400 con 0x7fb7d40051c0 2024-03-28T18:56:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.764+0000 7fb7ecc00640 1 -- 172.21.15.27:0/3093908388 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb7e0012040 con 0x7fb7d40051c0 2024-03-28T18:56:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.764+0000 7fb7ecc00640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7cc06bbd0 0x7fb7cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.765+0000 7fb7ecc00640 1 -- 172.21.15.27:0/3093908388 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fb7e008b2d0 con 0x7fb7d40051c0 2024-03-28T18:56:08.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.765+0000 7fb7e7e00640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7cc06bbd0 0x7fb7cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:08.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.765+0000 7fb7e7e00640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7cc06bbd0 0x7fb7cc06e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb7f014e210 tx=0x7fb7d8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.769+0000 7fb7ecc00640 1 -- 172.21.15.27:0/3093908388 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb7e0019030 con 0x7fb7d40051c0 2024-03-28T18:56:08.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.972+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7fb7f0151190 con 0x7fb7d40051c0 2024-03-28T18:56:08.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.973+0000 7fb7ecc00640 1 -- 172.21.15.27:0/3093908388 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v24) v1 ==== 95+0+11 (secure 0 0 0) 0x7fb7e001c360 con 0x7fb7d40051c0 2024-03-28T18:56:08.974 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:08.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7cc06bbd0 msgr2=0x7fb7cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7cc06bbd0 0x7fb7cc06e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb7f014e210 tx=0x7fb7d8009040 comp rx=0 tx=0).stop 2024-03-28T18:56:08.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 msgr2=0x7fb7f01e7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7f01e7ed0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0011940 tx=0x7fb7e0011e00 comp rx=0 tx=0).stop 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 shutdown_connections 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7cc06bbd0 0x7fb7cc06e090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7f014c850 0x7fb7f014ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7f014ad70 0x7fb7f01e8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 --2- 172.21.15.27:0/3093908388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40051c0 0x7fb7f01e7ed0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 >> 172.21.15.27:0/3093908388 conn(0x7fb7f006b0c0 msgr2=0x7fb7f014e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 shutdown_connections 2024-03-28T18:56:08.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:08.976+0000 7fb7f5600640 1 -- 172.21.15.27:0/3093908388 wait complete. 2024-03-28T18:56:08.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-03-28T18:56:09.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.223+0000 7f90a0000640 1 Processor -- start 2024-03-28T18:56:09.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.224+0000 7f90a0000640 1 -- start start 2024-03-28T18:56:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.225+0000 7f90a0000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f909814f920 0x7f90980c9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.225+0000 7f90a0000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90980c8f30 0x7f90980cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.225+0000 7f90a0000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909806d960 con 0x7f90980cb8c0 2024-03-28T18:56:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.225+0000 7f90a0000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90981568e0 con 0x7f90980c8f30 2024-03-28T18:56:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.225+0000 7f90a0000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9098156450 con 0x7f909814f920 2024-03-28T18:56:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.225+0000 7f909e000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f90980cb8c0 0x7f9098158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52310/0 (socket says 172.21.15.27:52310) 2024-03-28T18:56:09.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.225+0000 7f909cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90980c8f30 0x7f90980cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.226+0000 7f909e000640 1 -- 172.21.15.27:0/661485326 learned_addr learned my addr 172.21.15.27:0/661485326 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.226+0000 7f909d600640 1 --2- 172.21.15.27:0/661485326 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f909814f920 0x7f90980c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.226+0000 7f909cc00640 1 -- 172.21.15.27:0/661485326 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f909814f920 msgr2=0x7f90980c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.226+0000 7f909cc00640 1 --2- 172.21.15.27:0/661485326 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f909814f920 0x7f90980c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.226+0000 7f909cc00640 1 -- 172.21.15.27:0/661485326 >> v1:172.21.15.27:6789/0 conn(0x7f90980cb8c0 legacy=0x7f9098158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.226+0000 7f909cc00640 1 -- 172.21.15.27:0/661485326 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098155b20 con 0x7f90980c8f30 2024-03-28T18:56:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.226+0000 7f9097e00640 1 -- 172.21.15.27:0/661485326 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700756933 0 0) 0x7f909806d960 con 0x7f90980cb8c0 2024-03-28T18:56:09.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.227+0000 7f909cc00640 1 --2- 172.21.15.27:0/661485326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980c8f30 0x7f90980cb380 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f90840097a0 tx=0x7f908402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64655aa7e2b19a24 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:09.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.227+0000 7f9097e00640 1 -- 172.21.15.27:0/661485326 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90840315a0 con 0x7f90980c8f30 2024-03-28T18:56:09.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f9097e00640 1 -- 172.21.15.27:0/661485326 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084037070 con 0x7f90980c8f30 2024-03-28T18:56:09.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f9097e00640 1 -- 172.21.15.27:0/661485326 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084031960 con 0x7f90980c8f30 2024-03-28T18:56:09.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f90a0000640 1 -- 172.21.15.27:0/661485326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980c8f30 msgr2=0x7f90980cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f90a0000640 1 --2- 172.21.15.27:0/661485326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980c8f30 0x7f90980cb380 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f90840097a0 tx=0x7f908402b840 comp rx=0 tx=0).stop 2024-03-28T18:56:09.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f90a0000640 1 -- 172.21.15.27:0/661485326 shutdown_connections 2024-03-28T18:56:09.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f90a0000640 1 --2- 172.21.15.27:0/661485326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980c8f30 0x7f90980cb380 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f90a0000640 1 --2- 172.21.15.27:0/661485326 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f909814f920 0x7f90980c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f90a0000640 1 -- 172.21.15.27:0/661485326 >> 172.21.15.27:0/661485326 conn(0x7f9098066fb0 msgr2=0x7f90980673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:09.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f90a0000640 1 -- 172.21.15.27:0/661485326 shutdown_connections 2024-03-28T18:56:09.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.228+0000 7f90a0000640 1 -- 172.21.15.27:0/661485326 wait complete. 2024-03-28T18:56:09.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.231+0000 7f90a0000640 1 Processor -- start 2024-03-28T18:56:09.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.231+0000 7f90a0000640 1 -- start start 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f90a0000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90980c8f30 0x7f9098149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f90a0000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980cb8c0 0x7f90980affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f90a0000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f909814f920 0x7f90980c2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f90a0000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909814fd00 con 0x7f90980cb8c0 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f90a0000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90981560e0 con 0x7f909814f920 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f90a0000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9098159610 con 0x7f90980c8f30 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f909cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980cb8c0 0x7f90980affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f909cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980cb8c0 0x7f90980affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48048/0 (socket says 172.21.15.27:48048) 2024-03-28T18:56:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.233+0000 7f909cc00640 1 -- 172.21.15.27:0/2938414657 learned_addr learned my addr 172.21.15.27:0/2938414657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f909d600640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90980c8f30 0x7f9098149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f909e000640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f909814f920 0x7f90980c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f909cc00640 1 -- 172.21.15.27:0/2938414657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90980c8f30 msgr2=0x7f9098149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f909cc00640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90980c8f30 0x7f9098149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f909cc00640 1 -- 172.21.15.27:0/2938414657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f909814f920 msgr2=0x7f90980c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f909cc00640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f909814f920 0x7f90980c2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f909cc00640 1 -- 172.21.15.27:0/2938414657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098149bb0 con 0x7f90980cb8c0 2024-03-28T18:56:09.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f909cc00640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980cb8c0 0x7f90980affc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f90840097a0 tx=0x7f90840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:09.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.234+0000 7f9095600640 1 -- 172.21.15.27:0/2938414657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084004230 con 0x7f90980cb8c0 2024-03-28T18:56:09.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.235+0000 7f9095600640 1 -- 172.21.15.27:0/2938414657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90840043f0 con 0x7f90980cb8c0 2024-03-28T18:56:09.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.235+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90980b0500 con 0x7f90980cb8c0 2024-03-28T18:56:09.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.235+0000 7f9095600640 1 -- 172.21.15.27:0/2938414657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084003850 con 0x7f90980cb8c0 2024-03-28T18:56:09.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.235+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90981ebe60 con 0x7f90980cb8c0 2024-03-28T18:56:09.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.236+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909804f470 con 0x7f90980cb8c0 2024-03-28T18:56:09.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.237+0000 7f9095600640 1 -- 172.21.15.27:0/2938414657 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f90840039f0 con 0x7f90980cb8c0 2024-03-28T18:56:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.238+0000 7f9095600640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f907806bbd0 0x7f907806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.238+0000 7f9095600640 1 -- 172.21.15.27:0/2938414657 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f908402f030 con 0x7f90980cb8c0 2024-03-28T18:56:09.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.239+0000 7f909d600640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f907806bbd0 0x7f907806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.239+0000 7f909d600640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f907806bbd0 0x7f907806e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f90880099c0 tx=0x7f9088003990 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.243+0000 7f9095600640 1 -- 172.21.15.27:0/2938414657 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9084041070 con 0x7f90980cb8c0 2024-03-28T18:56:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.517+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90980afd20 con 0x7f90980cb8c0 2024-03-28T18:56:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.518+0000 7f9095600640 1 -- 172.21.15.27:0/2938414657 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v24) v1 ==== 99+0+11 (secure 0 0 0) 0x7f908407b4a0 con 0x7f90980cb8c0 2024-03-28T18:56:09.519 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:09.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.521+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f907806bbd0 msgr2=0x7f907806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.521+0000 7f90a0000640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f907806bbd0 0x7f907806e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f90880099c0 tx=0x7f9088003990 comp rx=0 tx=0).stop 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.521+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980cb8c0 msgr2=0x7f90980affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.521+0000 7f90a0000640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980cb8c0 0x7f90980affc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f90840097a0 tx=0x7f90840046c0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.522+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 shutdown_connections 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.522+0000 7f90a0000640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f907806bbd0 0x7f907806e090 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.522+0000 7f90a0000640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f909814f920 0x7f90980c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.522+0000 7f90a0000640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90980cb8c0 0x7f90980affc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.522+0000 7f90a0000640 1 --2- 172.21.15.27:0/2938414657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90980c8f30 0x7f9098149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.522+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 >> 172.21.15.27:0/2938414657 conn(0x7f9098066fb0 msgr2=0x7f9098067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.522+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 shutdown_connections 2024-03-28T18:56:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.522+0000 7f90a0000640 1 -- 172.21.15.27:0/2938414657 wait complete. 2024-03-28T18:56:09.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-03-28T18:56:09.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.777+0000 7f0163000640 1 Processor -- start 2024-03-28T18:56:09.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.777+0000 7f0163000640 1 -- start start 2024-03-28T18:56:09.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.778+0000 7f0163000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c150990 0x7f015c1573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.778+0000 7f0163000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f015c153aa0 0x7f015c153f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.778+0000 7f0163000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c0adaa0 con 0x7f015c0ad4e0 2024-03-28T18:56:09.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.779+0000 7f0163000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c071870 con 0x7f015c153aa0 2024-03-28T18:56:09.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.779+0000 7f0163000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c071b50 con 0x7f015c150990 2024-03-28T18:56:09.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.779+0000 7f0161200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f015c0ad4e0 0x7f015c0c4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52320/0 (socket says 172.21.15.27:52320) 2024-03-28T18:56:09.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.779+0000 7f0161200640 1 -- 172.21.15.27:0/1298913192 learned_addr learned my addr 172.21.15.27:0/1298913192 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:09.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.779+0000 7f0161c00640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c150990 0x7f015c1573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.779+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116232095 0 0) 0x7f015c0adaa0 con 0x7f015c0ad4e0 2024-03-28T18:56:09.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.779+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0144003520 con 0x7f015c0ad4e0 2024-03-28T18:56:09.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.779+0000 7f0162600640 1 --2- 172.21.15.27:0/1298913192 >> v2:172.21.15.155:3300/0 conn(0x7f015c153aa0 0x7f015c153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968283158 0 0) 0x7f0144003520 con 0x7f015c0ad4e0 2024-03-28T18:56:09.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c150990 msgr2=0x7f015c1573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:09.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c150990 0x7f015c1573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 >> v2:172.21.15.155:3300/0 conn(0x7f015c153aa0 msgr2=0x7f015c153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 --2- 172.21.15.27:0/1298913192 >> v2:172.21.15.155:3300/0 conn(0x7f015c153aa0 0x7f015c153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c071e30 con 0x7f015c0ad4e0 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f0148001670 con 0x7f015c0ad4e0 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 >> v1:172.21.15.27:6789/0 conn(0x7f015c0ad4e0 legacy=0x7f015c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01440039b0 0x7f0144003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c150990 0x7f0144004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f01440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c071870 con 0x7f0144004d40 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c071b50 con 0x7f01440039b0 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014400ba70 con 0x7f015c150990 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f0162600640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f01440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.780+0000 7f0161c00640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01440039b0 0x7f0144003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.781+0000 7f0161200640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c150990 0x7f0144004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.781+0000 7f0162600640 1 -- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c150990 msgr2=0x7f0144004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:09.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.781+0000 7f0162600640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c150990 0x7f0144004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.781+0000 7f0162600640 1 -- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01440039b0 msgr2=0x7f0144003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:09.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.781+0000 7f0162600640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01440039b0 0x7f0144003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.781+0000 7f0162600640 1 -- 172.21.15.27:0/1298913192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0144003520 con 0x7f0144004d40 2024-03-28T18:56:09.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.781+0000 7f0161c00640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01440039b0 0x7f0144003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.783+0000 7f0162600640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f01440051c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0154007360 tx=0x7f0144005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b73f65746b08ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.783+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154039070 con 0x7f0144004d40 2024-03-28T18:56:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.783+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015402ccf0 con 0x7f0144004d40 2024-03-28T18:56:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.783+0000 7f015be00640 1 -- 172.21.15.27:0/1298913192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154009be0 con 0x7f0144004d40 2024-03-28T18:56:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.784+0000 7f0163000640 1 -- 172.21.15.27:0/1298913192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 msgr2=0x7f01440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.784+0000 7f0163000640 1 --2- 172.21.15.27:0/1298913192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f01440051c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0154007360 tx=0x7f0144005830 comp rx=0 tx=0).stop 2024-03-28T18:56:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.784+0000 7f0162600640 1 -- 172.21.15.27:0/1298913192 reap_dead start 2024-03-28T18:56:09.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.784+0000 7f0163000640 1 -- 172.21.15.27:0/1298913192 shutdown_connections 2024-03-28T18:56:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.784+0000 7f0163000640 1 -- 172.21.15.27:0/1298913192 >> 172.21.15.27:0/1298913192 conn(0x7f015c0c8130 msgr2=0x7f015c0c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.784+0000 7f0163000640 1 -- 172.21.15.27:0/1298913192 shutdown_connections 2024-03-28T18:56:09.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.784+0000 7f0163000640 1 -- 172.21.15.27:0/1298913192 wait complete. 2024-03-28T18:56:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.787+0000 7f0163000640 1 Processor -- start 2024-03-28T18:56:09.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.787+0000 7f0163000640 1 -- start start 2024-03-28T18:56:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.789+0000 7f0163000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f015c0c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.789+0000 7f0163000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c0c6790 0x7f015c1ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.789+0000 7f0163000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f015c0c17a0 0x7f015c1ec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.789+0000 7f0163000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0c53a0 con 0x7f0144004d40 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.789+0000 7f0163000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c072110 con 0x7f015c0c17a0 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.789+0000 7f0163000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1e9580 con 0x7f015c0c6790 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.789+0000 7f0161c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f015c0c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f015c0c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48066/0 (socket says 172.21.15.27:48066) 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161c00640 1 -- 172.21.15.27:0/786117639 learned_addr learned my addr 172.21.15.27:0/786117639 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0162600640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f015c0c17a0 0x7f015c1ec3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161200640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c0c6790 0x7f015c1ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161c00640 1 -- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c0c6790 msgr2=0x7f015c1ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161c00640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c0c6790 0x7f015c1ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161c00640 1 -- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f015c0c17a0 msgr2=0x7f015c1ec3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161c00640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f015c0c17a0 0x7f015c1ec3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161c00640 1 -- 172.21.15.27:0/786117639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c1e89b0 con 0x7f0144004d40 2024-03-28T18:56:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161200640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c0c6790 0x7f015c1ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.790+0000 7f0161c00640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f015c0c6250 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0150011570 tx=0x7f0150011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.791+0000 7f0159600640 1 -- 172.21.15.27:0/786117639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150015870 con 0x7f0144004d40 2024-03-28T18:56:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.791+0000 7f0159600640 1 -- 172.21.15.27:0/786117639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0150015a10 con 0x7f0144004d40 2024-03-28T18:56:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.791+0000 7f0159600640 1 -- 172.21.15.27:0/786117639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150015dc0 con 0x7f0144004d40 2024-03-28T18:56:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.791+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c1ecb00 con 0x7f0144004d40 2024-03-28T18:56:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.791+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c1e8f30 con 0x7f0144004d40 2024-03-28T18:56:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.793+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c04f400 con 0x7f0144004d40 2024-03-28T18:56:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.793+0000 7f0159600640 1 -- 172.21.15.27:0/786117639 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f015001d400 con 0x7f0144004d40 2024-03-28T18:56:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.795+0000 7f0159600640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f013c06bbd0 0x7f013c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.795+0000 7f0159600640 1 -- 172.21.15.27:0/786117639 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f015008b5f0 con 0x7f0144004d40 2024-03-28T18:56:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.795+0000 7f0161200640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f013c06bbd0 0x7f013c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.796+0000 7f0161200640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f013c06bbd0 0x7f013c06e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0148001d30 tx=0x7f0148001cc0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:09.799+0000 7f0159600640 1 -- 172.21.15.27:0/786117639 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0150019030 con 0x7f0144004d40 2024-03-28T18:56:10.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.108+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f015c0ad2c0 con 0x7f0144004d40 2024-03-28T18:56:10.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.109+0000 7f0159600640 1 -- 172.21.15.27:0/786117639 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v24) v1 ==== 95+0+11 (secure 0 0 0) 0x7f0150059da0 con 0x7f0144004d40 2024-03-28T18:56:10.110 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f013c06bbd0 msgr2=0x7f013c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f013c06bbd0 0x7f013c06e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0148001d30 tx=0x7f0148001cc0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 msgr2=0x7f015c0c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f015c0c6250 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0150011570 tx=0x7f0150011a30 comp rx=0 tx=0).stop 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 shutdown_connections 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f013c06bbd0 0x7f013c06e090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f015c0c17a0 0x7f015c1ec3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f015c0c6790 0x7f015c1ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 --2- 172.21.15.27:0/786117639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0144004d40 0x7f015c0c6250 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 >> 172.21.15.27:0/786117639 conn(0x7f015c0c8130 msgr2=0x7f015c06b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 shutdown_connections 2024-03-28T18:56:10.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.112+0000 7f0163000640 1 -- 172.21.15.27:0/786117639 wait complete. 2024-03-28T18:56:10.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:56:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.362+0000 7fc0d4e00640 1 Processor -- start 2024-03-28T18:56:10.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.362+0000 7fc0d4e00640 1 -- start start 2024-03-28T18:56:10.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.364+0000 7fc0d4e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc0d00c74e0 0x7fc0d00c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.364+0000 7fc0d4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d00c2290 0x7fc0d0152760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.364+0000 7fc0d4e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d0158d00 con 0x7fc0d0152ca0 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.364+0000 7fc0d4e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d01530e0 con 0x7fc0d00c74e0 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.364+0000 7fc0d4e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d0158fe0 con 0x7fc0d00c2290 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.364+0000 7fc0cfe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc0d0152ca0 0x7fc0d01556e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52330/0 (socket says 172.21.15.27:52330) 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.364+0000 7fc0cfe00640 1 -- 172.21.15.27:0/2042020080 learned_addr learned my addr 172.21.15.27:0/2042020080 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cf400640 1 --2- 172.21.15.27:0/2042020080 >> v2:172.21.15.155:3300/0 conn(0x7fc0d00c74e0 0x7fc0d00c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0c7e00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d00c2290 0x7fc0d0152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951723889 0 0) 0x7fc0d0158d00 con 0x7fc0d0152ca0 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b4003520 con 0x7fc0d0152ca0 2024-03-28T18:56:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3738179333 0 0) 0x7fc0b4003520 con 0x7fc0d0152ca0 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d00c2290 msgr2=0x7fc0d0152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d00c2290 0x7fc0d0152760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 >> v2:172.21.15.155:3300/0 conn(0x7fc0d00c74e0 msgr2=0x7fc0d00c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 --2- 172.21.15.27:0/2042020080 >> v2:172.21.15.155:3300/0 conn(0x7fc0d00c74e0 0x7fc0d00c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d01592c0 con 0x7fc0d0152ca0 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fc0c80019f0 con 0x7fc0d0152ca0 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.365+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 >> v1:172.21.15.27:6789/0 conn(0x7fc0d0152ca0 legacy=0x7fc0d01556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0cea00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0b40039b0 0x7fc0b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0cea00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0cea00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b40051c0 0x7fc0b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d01530e0 con 0x7fc0b4004800 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d0158fe0 con 0x7fc0b40039b0 2024-03-28T18:56:10.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b400bf80 con 0x7fc0b40051c0 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0c7e00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d00c2290 0x7fc0d0152760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0c7e00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0cfe00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b40051c0 0x7fc0b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.366+0000 7fc0cf400640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0b40039b0 0x7fc0b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.367+0000 7fc0c7e00640 1 -- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b40051c0 msgr2=0x7fc0b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.367+0000 7fc0c7e00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b40051c0 0x7fc0b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.367+0000 7fc0c7e00640 1 -- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0b40039b0 msgr2=0x7fc0b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.367+0000 7fc0c7e00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0b40039b0 0x7fc0b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.367+0000 7fc0c7e00640 1 -- 172.21.15.27:0/2042020080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b4003520 con 0x7fc0b4004800 2024-03-28T18:56:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.368+0000 7fc0c7e00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8009b00 tx=0x7fc0b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13eab176daa7c93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.368+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b80097d0 con 0x7fc0b4004800 2024-03-28T18:56:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.368+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b8004030 con 0x7fc0b4004800 2024-03-28T18:56:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.368+0000 7fc0cea00640 1 -- 172.21.15.27:0/2042020080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8004400 con 0x7fc0b4004800 2024-03-28T18:56:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.369+0000 7fc0d4e00640 1 -- 172.21.15.27:0/2042020080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0b4004800 msgr2=0x7fc0b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.369+0000 7fc0d4e00640 1 --2- 172.21.15.27:0/2042020080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0b4004800 0x7fc0b4004c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8009b00 tx=0x7fc0b4005830 comp rx=0 tx=0).stop 2024-03-28T18:56:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.369+0000 7fc0cfe00640 1 -- 172.21.15.27:0/2042020080 reap_dead start 2024-03-28T18:56:10.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.369+0000 7fc0d4e00640 1 -- 172.21.15.27:0/2042020080 shutdown_connections 2024-03-28T18:56:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.369+0000 7fc0d4e00640 1 -- 172.21.15.27:0/2042020080 >> 172.21.15.27:0/2042020080 conn(0x7fc0d006d980 msgr2=0x7fc0d00a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.369+0000 7fc0d4e00640 1 -- 172.21.15.27:0/2042020080 shutdown_connections 2024-03-28T18:56:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.369+0000 7fc0d4e00640 1 -- 172.21.15.27:0/2042020080 wait complete. 2024-03-28T18:56:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.371+0000 7fc0d4e00640 1 Processor -- start 2024-03-28T18:56:10.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.371+0000 7fc0d4e00640 1 -- start start 2024-03-28T18:56:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0d4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b4004800 0x7fc0d00c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0d4e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d00c6b90 0x7fc0d0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0d4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d01ede70 0x7fc0d01ee2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0d4e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d01589f0 con 0x7fc0d01ede70 2024-03-28T18:56:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0d4e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d0159880 con 0x7fc0d00c6b90 2024-03-28T18:56:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0d4e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d00c78c0 con 0x7fc0b4004800 2024-03-28T18:56:10.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0cfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d01ede70 0x7fc0d01ee2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0cfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d01ede70 0x7fc0d01ee2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48084/0 (socket says 172.21.15.27:48084) 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.372+0000 7fc0cfe00640 1 -- 172.21.15.27:0/3361184207 learned_addr learned my addr 172.21.15.27:0/3361184207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0c7e00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d00c6b90 0x7fc0d0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0cf400640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b4004800 0x7fc0d00c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0cfe00640 1 -- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b4004800 msgr2=0x7fc0d00c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0cfe00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b4004800 0x7fc0d00c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0cfe00640 1 -- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d00c6b90 msgr2=0x7fc0d0071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0cfe00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d00c6b90 0x7fc0d0071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0cfe00640 1 -- 172.21.15.27:0/3361184207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d0071db0 con 0x7fc0d01ede70 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0c7e00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d00c6b90 0x7fc0d0071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.373+0000 7fc0cfe00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d01ede70 0x7fc0d01ee2f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8001b60 tx=0x7fc0c8007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:10.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.374+0000 7fc0c7400640 1 -- 172.21.15.27:0/3361184207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c800ab90 con 0x7fc0d01ede70 2024-03-28T18:56:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.374+0000 7fc0c7400640 1 -- 172.21.15.27:0/3361184207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c8018070 con 0x7fc0d01ede70 2024-03-28T18:56:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.374+0000 7fc0c7400640 1 -- 172.21.15.27:0/3361184207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80133f0 con 0x7fc0d01ede70 2024-03-28T18:56:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.374+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d01ee8f0 con 0x7fc0d01ede70 2024-03-28T18:56:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.374+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d0071570 con 0x7fc0d01ede70 2024-03-28T18:56:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.375+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d0067dd0 con 0x7fc0d01ede70 2024-03-28T18:56:10.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.376+0000 7fc0c7400640 1 -- 172.21.15.27:0/3361184207 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc0c8013590 con 0x7fc0d01ede70 2024-03-28T18:56:10.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.377+0000 7fc0c7400640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0ac06bbd0 0x7fc0ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.378+0000 7fc0c7400640 1 -- 172.21.15.27:0/3361184207 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fc0c808b3a0 con 0x7fc0d01ede70 2024-03-28T18:56:10.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.378+0000 7fc0cf400640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0ac06bbd0 0x7fc0ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.379+0000 7fc0cf400640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0ac06bbd0 0x7fc0ac06e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009a10 tx=0x7fc0c0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:10.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.381+0000 7fc0c7400640 1 -- 172.21.15.27:0/3361184207 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc0c8059b60 con 0x7fc0d01ede70 2024-03-28T18:56:10.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.627+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0d006f200 con 0x7fc0d01ede70 2024-03-28T18:56:10.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.628+0000 7fc0c7400640 1 -- 172.21.15.27:0/3361184207 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7fc0c8059460 con 0x7fc0d01ede70 2024-03-28T18:56:10.629 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:10.630 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:56:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:56:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0ac06bbd0 msgr2=0x7fc0ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0ac06bbd0 0x7fc0ac06e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009a10 tx=0x7fc0c0008040 comp rx=0 tx=0).stop 2024-03-28T18:56:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d01ede70 msgr2=0x7fc0d01ee2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d01ede70 0x7fc0d01ee2f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8001b60 tx=0x7fc0c8007bd0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 shutdown_connections 2024-03-28T18:56:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0ac06bbd0 0x7fc0ac06e090 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d01ede70 0x7fc0d01ee2f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d00c6b90 0x7fc0d0071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 --2- 172.21.15.27:0/3361184207 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0b4004800 0x7fc0d00c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.631+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 >> 172.21.15.27:0/3361184207 conn(0x7fc0d006d980 msgr2=0x7fc0d014ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:10.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.632+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 shutdown_connections 2024-03-28T18:56:10.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.632+0000 7fc0d4e00640 1 -- 172.21.15.27:0/3361184207 wait complete. 2024-03-28T18:56:10.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.867+0000 7fd44a600640 1 Processor -- start 2024-03-28T18:56:10.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.868+0000 7fd44a600640 1 -- start start 2024-03-28T18:56:10.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd44a600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd444152a70 0x7fd444159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd44a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 0x7fd444155f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd44a600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4440c5710 con 0x7fd4440c6cc0 2024-03-28T18:56:10.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd44a600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4441556b0 con 0x7fd444152a70 2024-03-28T18:56:10.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd44a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4440c59f0 con 0x7fd444155b40 2024-03-28T18:56:10.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd449200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd444152a70 0x7fd444159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd443e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd4440c6cc0 0x7fd4440cb0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:58238/0 (socket says 172.21.15.27:58238) 2024-03-28T18:56:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd443e00640 1 -- 172.21.15.27:0/3674114364 learned_addr learned my addr 172.21.15.27:0/3674114364 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.870+0000 7fd449c00640 1 --2- 172.21.15.27:0/3674114364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 0x7fd444155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.871+0000 7fd449200640 1 -- 172.21.15.27:0/3674114364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 msgr2=0x7fd444155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.871+0000 7fd449200640 1 --2- 172.21.15.27:0/3674114364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 0x7fd444155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.871+0000 7fd449200640 1 -- 172.21.15.27:0/3674114364 >> v1:172.21.15.155:6789/0 conn(0x7fd4440c6cc0 legacy=0x7fd4440cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.871+0000 7fd449200640 1 -- 172.21.15.27:0/3674114364 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4440c5cd0 con 0x7fd444152a70 2024-03-28T18:56:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.873+0000 7fd449200640 1 --2- 172.21.15.27:0/3674114364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152a70 0x7fd444159580 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd4380097a0 tx=0x7fd43802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bd6b116e8af5a73 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:10.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.873+0000 7fd443400640 1 -- 172.21.15.27:0/3674114364 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4380315a0 con 0x7fd444152a70 2024-03-28T18:56:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.873+0000 7fd443400640 1 -- 172.21.15.27:0/3674114364 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd438037070 con 0x7fd444152a70 2024-03-28T18:56:10.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.873+0000 7fd443400640 1 -- 172.21.15.27:0/3674114364 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438031960 con 0x7fd444152a70 2024-03-28T18:56:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.874+0000 7fd44a600640 1 -- 172.21.15.27:0/3674114364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152a70 msgr2=0x7fd444159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.874+0000 7fd44a600640 1 --2- 172.21.15.27:0/3674114364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152a70 0x7fd444159580 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd4380097a0 tx=0x7fd43802b840 comp rx=0 tx=0).stop 2024-03-28T18:56:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.874+0000 7fd44a600640 1 -- 172.21.15.27:0/3674114364 shutdown_connections 2024-03-28T18:56:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.874+0000 7fd44a600640 1 --2- 172.21.15.27:0/3674114364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 0x7fd444155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.874+0000 7fd44a600640 1 --2- 172.21.15.27:0/3674114364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152a70 0x7fd444159580 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.874+0000 7fd44a600640 1 -- 172.21.15.27:0/3674114364 >> 172.21.15.27:0/3674114364 conn(0x7fd44406b2a0 msgr2=0x7fd44406b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.874+0000 7fd44a600640 1 -- 172.21.15.27:0/3674114364 shutdown_connections 2024-03-28T18:56:10.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.874+0000 7fd44a600640 1 -- 172.21.15.27:0/3674114364 wait complete. 2024-03-28T18:56:10.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.876+0000 7fd44a600640 1 Processor -- start 2024-03-28T18:56:10.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.876+0000 7fd44a600640 1 -- start start 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd44a600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4440c6cc0 0x7fd4441eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd44a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd444152a70 0x7fd44414a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd44a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 0x7fd4440aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd44a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4440cb7f0 con 0x7fd444152a70 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd44a600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd444072a70 con 0x7fd4440c6cc0 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd44a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4440cb970 con 0x7fd444155b40 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd443e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd444152a70 0x7fd44414a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd443e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd444152a70 0x7fd44414a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48092/0 (socket says 172.21.15.27:48092) 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd443e00640 1 -- 172.21.15.27:0/3178192154 learned_addr learned my addr 172.21.15.27:0/3178192154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.877+0000 7fd449200640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4440c6cc0 0x7fd4441eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd443e00640 1 -- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 msgr2=0x7fd4440aaf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd449c00640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 0x7fd4440aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd443e00640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 0x7fd4440aaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd443e00640 1 -- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4440c6cc0 msgr2=0x7fd4441eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd443e00640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4440c6cc0 0x7fd4441eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd443e00640 1 -- 172.21.15.27:0/3178192154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd444153800 con 0x7fd444152a70 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd449200640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4440c6cc0 0x7fd4441eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd443e00640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd444152a70 0x7fd44414a7c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd43000c440 tx=0x7fd430001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd440c00640 1 -- 172.21.15.27:0/3178192154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430017070 con 0x7fd444152a70 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd440c00640 1 -- 172.21.15.27:0/3178192154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd430005ce0 con 0x7fd444152a70 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd440c00640 1 -- 172.21.15.27:0/3178192154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4300072f0 con 0x7fd444152a70 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd444153ae0 con 0x7fd444152a70 2024-03-28T18:56:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.878+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44414ad00 con 0x7fd444152a70 2024-03-28T18:56:10.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.880+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd444067dd0 con 0x7fd444152a70 2024-03-28T18:56:10.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.881+0000 7fd440c00640 1 -- 172.21.15.27:0/3178192154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd43001a070 con 0x7fd444152a70 2024-03-28T18:56:10.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.881+0000 7fd440c00640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:10.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.882+0000 7fd440c00640 1 -- 172.21.15.27:0/3178192154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7fd43008a0f0 con 0x7fd444152a70 2024-03-28T18:56:10.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.882+0000 7fd449200640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:10.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.883+0000 7fd449200640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd438036000 tx=0x7fd43803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:10.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:10.886+0000 7fd440c00640 1 -- 172.21.15.27:0/3178192154 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd430058940 con 0x7fd444152a70 2024-03-28T18:56:11.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.101+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd444069640 con 0x7fd444152a70 2024-03-28T18:56:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.103+0000 7fd440c00640 1 -- 172.21.15.27:0/3178192154 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+20307 (secure 0 0 0) 0x7fd430058240 con 0x7fd444152a70 2024-03-28T18:56:11.104 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:11.105 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:55:46.587251+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 msgr2=0x7fd42406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd438036000 tx=0x7fd43803d040 comp rx=0 tx=0).stop 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd444152a70 msgr2=0x7fd44414a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd444152a70 0x7fd44414a7c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd43000c440 tx=0x7fd430001e70 comp rx=0 tx=0).stop 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 shutdown_connections 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444155b40 0x7fd4440aaf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd444152a70 0x7fd44414a7c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 --2- 172.21.15.27:0/3178192154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4440c6cc0 0x7fd4441eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 >> 172.21.15.27:0/3178192154 conn(0x7fd44406b2a0 msgr2=0x7fd44406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 shutdown_connections 2024-03-28T18:56:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.106+0000 7fd44a600640 1 -- 172.21.15.27:0/3178192154 wait complete. 2024-03-28T18:56:11.125 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:56:11.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.342+0000 7f5286400640 1 Processor -- start 2024-03-28T18:56:11.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.342+0000 7f5286400640 1 -- start start 2024-03-28T18:56:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.344+0000 7f5286400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5280150c70 0x7f52800a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.344+0000 7f5286400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52800c3af0 con 0x7f52800c3690 2024-03-28T18:56:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.344+0000 7f5286400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5280154020 con 0x7f52801534c0 2024-03-28T18:56:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.344+0000 7f5286400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52800ab4e0 con 0x7f5280150c70 2024-03-28T18:56:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.344+0000 7f5277e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f52800c3690 0x7f52800c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52342/0 (socket says 172.21.15.27:52342) 2024-03-28T18:56:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.344+0000 7f5277e00640 1 -- 172.21.15.27:0/713960213 learned_addr learned my addr 172.21.15.27:0/713960213 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.344+0000 7f527f400640 1 --2- 172.21.15.27:0/713960213 >> v2:172.21.15.27:3300/0 conn(0x7f5280150c70 0x7f52800a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.345+0000 7f527ea00640 1 -- 172.21.15.27:0/713960213 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241243453 0 0) 0x7f52800c3af0 con 0x7f52800c3690 2024-03-28T18:56:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.345+0000 7f527ea00640 1 -- 172.21.15.27:0/713960213 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5264003520 con 0x7f52800c3690 2024-03-28T18:56:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.345+0000 7f527f400640 1 -- 172.21.15.27:0/713960213 >> v1:172.21.15.155:6789/0 conn(0x7f52801534c0 legacy=0x7f5280153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.345+0000 7f527f400640 1 -- 172.21.15.27:0/713960213 >> v1:172.21.15.27:6789/0 conn(0x7f52800c3690 legacy=0x7f52800c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.345+0000 7f527f400640 1 -- 172.21.15.27:0/713960213 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52800c3d90 con 0x7f5280150c70 2024-03-28T18:56:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.346+0000 7f527f400640 1 --2- 172.21.15.27:0/713960213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5280150c70 0x7f52800a9cb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f52801541f0 tx=0x7f527002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d143422be8767c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.346+0000 7f527ea00640 1 -- 172.21.15.27:0/713960213 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270035020 con 0x7f5280150c70 2024-03-28T18:56:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.346+0000 7f527ea00640 1 -- 172.21.15.27:0/713960213 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5270031810 con 0x7f5280150c70 2024-03-28T18:56:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.346+0000 7f527ea00640 1 -- 172.21.15.27:0/713960213 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270031bc0 con 0x7f5280150c70 2024-03-28T18:56:11.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.347+0000 7f5286400640 1 -- 172.21.15.27:0/713960213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5280150c70 msgr2=0x7f52800a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.347+0000 7f5286400640 1 --2- 172.21.15.27:0/713960213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5280150c70 0x7f52800a9cb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f52801541f0 tx=0x7f527002b8b0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.347+0000 7f5286400640 1 -- 172.21.15.27:0/713960213 shutdown_connections 2024-03-28T18:56:11.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.347+0000 7f5286400640 1 --2- 172.21.15.27:0/713960213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5280150c70 0x7f52800a9cb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.347+0000 7f5286400640 1 -- 172.21.15.27:0/713960213 >> 172.21.15.27:0/713960213 conn(0x7f528006b9d0 msgr2=0x7f52800c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:11.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.347+0000 7f5286400640 1 -- 172.21.15.27:0/713960213 shutdown_connections 2024-03-28T18:56:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.347+0000 7f5286400640 1 -- 172.21.15.27:0/713960213 wait complete. 2024-03-28T18:56:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.349+0000 7f5286400640 1 Processor -- start 2024-03-28T18:56:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.349+0000 7f5286400640 1 -- start start 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f5286400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52800c3690 0x7f52800c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f5286400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5280150c70 0x7f5280156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f5286400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52801534c0 0x7f52801565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f5286400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52800ab9f0 con 0x7f52801534c0 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f5286400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52800ac1f0 con 0x7f52800c3690 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f5286400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52800abcd0 con 0x7f5280150c70 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f527fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52801534c0 0x7f52801565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f527fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52801534c0 0x7f52801565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48112/0 (socket says 172.21.15.27:48112) 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.350+0000 7f527fe00640 1 -- 172.21.15.27:0/525419714 learned_addr learned my addr 172.21.15.27:0/525419714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.351+0000 7f527f400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52800c3690 0x7f52800c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.351+0000 7f5277e00640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5280150c70 0x7f5280156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.351+0000 7f527fe00640 1 -- 172.21.15.27:0/525419714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5280150c70 msgr2=0x7f5280156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.351+0000 7f527fe00640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5280150c70 0x7f5280156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.351+0000 7f527fe00640 1 -- 172.21.15.27:0/525419714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52800c3690 msgr2=0x7f52800c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.351+0000 7f527fe00640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52800c3690 0x7f52800c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.351+0000 7f527fe00640 1 -- 172.21.15.27:0/525419714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5280156ae0 con 0x7f52801534c0 2024-03-28T18:56:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.351+0000 7f527fe00640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52801534c0 0x7f52801565a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f527800c6b0 tx=0x7f527800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.352+0000 7f5277400640 1 -- 172.21.15.27:0/525419714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278003b70 con 0x7f52801534c0 2024-03-28T18:56:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.352+0000 7f5277400640 1 -- 172.21.15.27:0/525419714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278003d30 con 0x7f52801534c0 2024-03-28T18:56:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.352+0000 7f5277400640 1 -- 172.21.15.27:0/525419714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278005310 con 0x7f52801534c0 2024-03-28T18:56:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.352+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52801ee620 con 0x7f52801534c0 2024-03-28T18:56:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.352+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52801eebe0 con 0x7f52801534c0 2024-03-28T18:56:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.353+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528004f400 con 0x7f52801534c0 2024-03-28T18:56:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.354+0000 7f5277400640 1 -- 172.21.15.27:0/525419714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f52780054b0 con 0x7f52801534c0 2024-03-28T18:56:11.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.355+0000 7f5277400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f525c06bbd0 0x7f525c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:11.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.355+0000 7f5277400640 1 -- 172.21.15.27:0/525419714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f527808ab90 con 0x7f52801534c0 2024-03-28T18:56:11.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.355+0000 7f527f400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f525c06bbd0 0x7f525c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:11.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.356+0000 7f527f400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f525c06bbd0 0x7f525c06e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5270036000 tx=0x7f527003d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:11.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.364+0000 7f5277400640 1 -- 172.21.15.27:0/525419714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5278059340 con 0x7f52801534c0 2024-03-28T18:56:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.606+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52800c5f00 con 0x7f52801534c0 2024-03-28T18:56:11.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.608+0000 7f5277400640 1 -- 172.21.15.27:0/525419714 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f5278058c40 con 0x7f52801534c0 2024-03-28T18:56:11.609 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:11.609 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:56:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:56:11.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f525c06bbd0 msgr2=0x7f525c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f525c06bbd0 0x7f525c06e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5270036000 tx=0x7f527003d040 comp rx=0 tx=0).stop 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52801534c0 msgr2=0x7f52801565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52801534c0 0x7f52801565a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f527800c6b0 tx=0x7f527800cb70 comp rx=0 tx=0).stop 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 shutdown_connections 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f525c06bbd0 0x7f525c06e090 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52801534c0 0x7f52801565a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5280150c70 0x7f5280156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 --2- 172.21.15.27:0/525419714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52800c3690 0x7f52800c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 >> 172.21.15.27:0/525419714 conn(0x7f528006b9d0 msgr2=0x7f5280151770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.611+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 shutdown_connections 2024-03-28T18:56:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:11.612+0000 7f5286400640 1 -- 172.21.15.27:0/525419714 wait complete. 2024-03-28T18:56:12.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T18:55:45.064332+0000', 'modified': '2024-03-28T18:55:49.130048+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4263}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4263': {'gid': 4263, 'name': 'c', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.27:6837/2601859893', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6835', 'nonce': 2601859893}, {'type': 'v1', 'addr': '172.21.15.27:6837', 'nonce': 2601859893}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'b', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4263, 'qdb_cluster': [4263]} 2024-03-28T18:56:12.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T18:56:12.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T18:56:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.268+0000 7f9c94e00640 1 Processor -- start 2024-03-28T18:56:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.269+0000 7f9c94e00640 1 -- start start 2024-03-28T18:56:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c94e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9c900c5aa0 0x7f9c90156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c94e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9c900c8db0 0x7f9c90072620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c94e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c900735f0 con 0x7f9c90072b60 2024-03-28T18:56:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c94e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c901565d0 con 0x7f9c900c8db0 2024-03-28T18:56:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c94e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c900c82d0 con 0x7f9c900c5aa0 2024-03-28T18:56:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c86a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9c900c8db0 0x7f9c90072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c86a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9c900c8db0 0x7f9c90072620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48116/0 (socket says 172.21.15.27:48116) 2024-03-28T18:56:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c86a00640 1 -- 172.21.15.27:0/4065547108 learned_addr learned my addr 172.21.15.27:0/4065547108 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c86a00640 1 -- 172.21.15.27:0/4065547108 >> v2:172.21.15.155:3300/0 conn(0x7f9c900c5aa0 msgr2=0x7f9c90156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c86a00640 1 --2- 172.21.15.27:0/4065547108 >> v2:172.21.15.155:3300/0 conn(0x7f9c900c5aa0 0x7f9c90156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c86a00640 1 -- 172.21.15.27:0/4065547108 >> v1:172.21.15.155:6789/0 conn(0x7f9c90072b60 legacy=0x7f9c90152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.270+0000 7f9c86a00640 1 -- 172.21.15.27:0/4065547108 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c900c7d10 con 0x7f9c900c8db0 2024-03-28T18:56:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c86a00640 1 --2- 172.21.15.27:0/4065547108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c900c8db0 0x7f9c90072620 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9c9006bf20 tx=0x7f9c7802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e1d0fe413ab84b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c8ea00640 1 -- 172.21.15.27:0/4065547108 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7802e070 con 0x7f9c900c8db0 2024-03-28T18:56:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c8ea00640 1 -- 172.21.15.27:0/4065547108 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c78037070 con 0x7f9c900c8db0 2024-03-28T18:56:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c8ea00640 1 -- 172.21.15.27:0/4065547108 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78031a00 con 0x7f9c900c8db0 2024-03-28T18:56:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c94e00640 1 -- 172.21.15.27:0/4065547108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c900c8db0 msgr2=0x7f9c90072620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c94e00640 1 --2- 172.21.15.27:0/4065547108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c900c8db0 0x7f9c90072620 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9c9006bf20 tx=0x7f9c7802b880 comp rx=0 tx=0).stop 2024-03-28T18:56:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c94e00640 1 -- 172.21.15.27:0/4065547108 shutdown_connections 2024-03-28T18:56:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c94e00640 1 --2- 172.21.15.27:0/4065547108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c900c8db0 0x7f9c90072620 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c94e00640 1 --2- 172.21.15.27:0/4065547108 >> v2:172.21.15.155:3300/0 conn(0x7f9c900c5aa0 0x7f9c90156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.272+0000 7f9c94e00640 1 -- 172.21.15.27:0/4065547108 >> 172.21.15.27:0/4065547108 conn(0x7f9c9006b0c0 msgr2=0x7f9c9006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.273+0000 7f9c94e00640 1 -- 172.21.15.27:0/4065547108 shutdown_connections 2024-03-28T18:56:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.273+0000 7f9c94e00640 1 -- 172.21.15.27:0/4065547108 wait complete. 2024-03-28T18:56:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.274+0000 7f9c94e00640 1 Processor -- start 2024-03-28T18:56:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.274+0000 7f9c94e00640 1 -- start start 2024-03-28T18:56:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.275+0000 7f9c94e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c90072b60 0x7f9c900c1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.275+0000 7f9c94e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900c5aa0 0x7f9c900bfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.275+0000 7f9c94e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c900c8db0 0x7f9c90151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.275+0000 7f9c94e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c90072100 con 0x7f9c90072b60 2024-03-28T18:56:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.275+0000 7f9c94e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c90071e20 con 0x7f9c900c5aa0 2024-03-28T18:56:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.275+0000 7f9c94e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c90152f50 con 0x7f9c900c8db0 2024-03-28T18:56:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.275+0000 7f9c8f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c90072b60 0x7f9c900c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c90072b60 0x7f9c900c1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48130/0 (socket says 172.21.15.27:48130) 2024-03-28T18:56:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8f400640 1 -- 172.21.15.27:0/2470010566 learned_addr learned my addr 172.21.15.27:0/2470010566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c86a00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900c5aa0 0x7f9c900bfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c900c8db0 0x7f9c90151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8f400640 1 -- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c900c8db0 msgr2=0x7f9c90151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8f400640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c900c8db0 0x7f9c90151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8f400640 1 -- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900c5aa0 msgr2=0x7f9c900bfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8f400640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900c5aa0 0x7f9c900bfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8f400640 1 -- 172.21.15.27:0/2470010566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c900c0150 con 0x7f9c90072b60 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c86a00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900c5aa0 0x7f9c900bfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c900c8db0 0x7f9c90151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c8f400640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c90072b60 0x7f9c900c1720 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000ba80 tx=0x7f9c8000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.276+0000 7f9c87e00640 1 -- 172.21.15.27:0/2470010566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80019070 con 0x7f9c90072b60 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.277+0000 7f9c87e00640 1 -- 172.21.15.27:0/2470010566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8000cbc0 con 0x7f9c90072b60 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.277+0000 7f9c87e00640 1 -- 172.21.15.27:0/2470010566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80014440 con 0x7f9c90072b60 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.277+0000 7f9c94e00640 1 -- 172.21.15.27:0/2470010566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c901595a0 con 0x7f9c90072b60 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.277+0000 7f9c94e00640 1 -- 172.21.15.27:0/2470010566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c900beb60 con 0x7f9c90072b60 2024-03-28T18:56:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.279+0000 7f9c87e00640 1 -- 172.21.15.27:0/2470010566 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9c8001c070 con 0x7f9c90072b60 2024-03-28T18:56:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.280+0000 7f9c87e00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9c6c06bbd0 0x7f9c6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.280+0000 7f9c87e00640 1 -- 172.21.15.27:0/2470010566 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f9c8008ab10 con 0x7f9c90072b60 2024-03-28T18:56:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.280+0000 7f9c86a00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9c6c06bbd0 0x7f9c6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.280+0000 7f9c991e8740 1 Processor -- start 2024-03-28T18:56:12.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.281+0000 7f9c86a00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9c6c06bbd0 0x7f9c6c06e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9c78036000 tx=0x7f9c78008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.281+0000 7f9c991e8740 1 -- start start 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.281+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01850 0x55a54bf3c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.281+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf3c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.281+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 0x55a54bf3d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c991e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54be65c90 con 0x55a54bf01c30 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c991e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54be0e980 con 0x55a54bf01850 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c991e8740 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a54be61d30 con 0x55a54bf028e0 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c86a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf3c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c86a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf3c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48144/0 (socket says 172.21.15.27:48144) 2024-03-28T18:56:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c86a00640 1 -- 172.21.15.27:0/4187923728 learned_addr learned my addr 172.21.15.27:0/4187923728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c86a00640 1 -- 172.21.15.27:0/4187923728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 msgr2=0x55a54bf3d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 0x55a54bf3d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c8f400640 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01850 0x55a54bf3c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c86a00640 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 0x55a54bf3d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c86a00640 1 -- 172.21.15.27:0/4187923728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01850 msgr2=0x55a54bf3c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c86a00640 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01850 0x55a54bf3c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.282+0000 7f9c86a00640 1 -- 172.21.15.27:0/4187923728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a54be26510 con 0x55a54bf01c30 2024-03-28T18:56:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.283+0000 7f9c86a00640 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf3c680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9c7802b8c0 tx=0x7f9c78030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c0214d7295a35b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.283+0000 7f9c7fe00640 1 -- 172.21.15.27:0/4187923728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78032bc0 con 0x55a54bf01c30 2024-03-28T18:56:12.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.283+0000 7f9c7fe00640 1 -- 172.21.15.27:0/4187923728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c78037040 con 0x55a54bf01c30 2024-03-28T18:56:12.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.283+0000 7f9c7fe00640 1 -- 172.21.15.27:0/4187923728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c78051480 con 0x55a54bf01c30 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 -- 172.21.15.27:0/4187923728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 msgr2=0x55a54bf3c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf3c680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9c7802b8c0 tx=0x7f9c78030000 comp rx=0 tx=0).stop 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 -- 172.21.15.27:0/4187923728 shutdown_connections 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 0x55a54bf3d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf3c680 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 --2- 172.21.15.27:0/4187923728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01850 0x55a54bf3c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 -- 172.21.15.27:0/4187923728 >> 172.21.15.27:0/4187923728 conn(0x55a54befdf20 msgr2=0x55a54bf22930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 -- 172.21.15.27:0/4187923728 shutdown_connections 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.284+0000 7f9c991e8740 1 -- 172.21.15.27:0/4187923728 wait complete. 2024-03-28T18:56:12.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.285+0000 7f9c991e8740 1 -- start start 2024-03-28T18:56:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.285+0000 7f9c991e8740 1 Processor -- start 2024-03-28T18:56:12.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf4f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 0x55a54bf51690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf4f8d0 0x55a54bf51bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c991e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54bf40470 con 0x55a54bf01c30 2024-03-28T18:56:12.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c991e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54be21dd0 con 0x55a54bf028e0 2024-03-28T18:56:12.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c991e8740 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a54bf3f050 con 0x55a54bf4f8d0 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c991e8740 10 client.0 _ll_register_callbacks cb 0x55a54bb79ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c8f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf4f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c8f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf4f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48158/0 (socket says 172.21.15.27:48158) 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.287+0000 7f9c8f400640 1 -- 172.21.15.27:0/1147674191 learned_addr learned my addr 172.21.15.27:0/1147674191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c86a00640 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 0x55a54bf51690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c8f400640 1 -- 172.21.15.27:0/1147674191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf4f8d0 msgr2=0x55a54bf51bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c8f400640 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf4f8d0 0x55a54bf51bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c8f400640 1 -- 172.21.15.27:0/1147674191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 msgr2=0x55a54bf51690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c8f400640 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 0x55a54bf51690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c8f400640 1 -- 172.21.15.27:0/1147674191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a54bb7a2d0 con 0x55a54bf01c30 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c8f400640 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf4f390 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9c80060da0 tx=0x7f9c80058bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6b8a4492da1776 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1147674191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80059020 con 0x55a54bf01c30 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1147674191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f9c80060e40 con 0x55a54bf01c30 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c991e8740 1 -- 172.21.15.27:0/1147674191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c78037040 con 0x55a54bf01c30 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1147674191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c80061040 con 0x55a54bf01c30 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c991e8740 10 client.4863 fetch_fsmap learned FSMap version 15 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.288+0000 7f9c991e8740 1 -- 172.21.15.27:0/1147674191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x55a54bc446b0 con 0x55a54bf01c30 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1147674191 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c78037040 con 0x55a54bf01c30 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1147674191 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 15) v1 ==== 3918+0+0 (secure 0 0 0) 0x7f9c80070ce0 con 0x55a54bf01c30 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c991e8740 10 client.4863 fetch_fsmap finished waiting for FSMap version 15 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c991e8740 10 client.4863 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c991e8740 20 client.4863 populate_metadata read hostname 'smithi027' 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] conn(0x55a54bf20b30 0x55a54bf20f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c991e8740 4 client.4863 mds_command: new command op to 4263 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:56:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c991e8740 1 -- 172.21.15.27:0/1147674191 --> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a54be07520 con 0x55a54bf20b30 2024-03-28T18:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.289+0000 7f9c86a00640 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] conn(0x55a54bf20b30 0x55a54bf20f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.290+0000 7f9c86a00640 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] conn(0x55a54bf20b30 0x55a54bf20f10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fd78e7042348966 server_cookie=b63213874a7ac3bf in_seq=0 out_seq=0 2024-03-28T18:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.290+0000 7f9c7cc00640 10 client.4863 ms_handle_connect on v2:172.21.15.27:6835/2601859893 2024-03-28T18:56:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.290+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1147674191 <== mds.0 v2:172.21.15.27:6835/2601859893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a54be07520 con 0x55a54bf20b30 2024-03-28T18:56:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.290+0000 7f9c7cc00640 10 client.4863 handle_command_reply: tid=0 2024-03-28T18:56:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.290+0000 7f9c991e8740 1 client.4863 shutdown 2024-03-28T18:56:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 -- 172.21.15.27:0/1147674191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 msgr2=0x55a54bf4f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf4f390 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9c80060da0 tx=0x7f9c80058bc0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 -- 172.21.15.27:0/1147674191 shutdown_connections 2024-03-28T18:56:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] conn(0x55a54bf20b30 0x55a54bf20f10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf4f8d0 0x55a54bf51bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 0x55a54bf51690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1147674191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf01c30 0x55a54bf4f390 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c7cc00640 0 client.4863 ms_handle_reset on v2:172.21.15.27:6835/2601859893 2024-03-28T18:56:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 -- 172.21.15.27:0/1147674191 >> 172.21.15.27:0/1147674191 conn(0x55a54bf01850 msgr2=0x55a54bf3f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 -- 172.21.15.27:0/1147674191 shutdown_connections 2024-03-28T18:56:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 1 -- 172.21.15.27:0/1147674191 wait complete. 2024-03-28T18:56:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.291+0000 7f9c991e8740 20 client.4863 trim_cache size 0 max 16384 2024-03-28T18:56:12.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.309+0000 7f9c991e8740 1 Processor -- start 2024-03-28T18:56:12.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.309+0000 7f9c991e8740 1 -- start start 2024-03-28T18:56:12.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01c30 0x55a54befc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 0x55a54bf3ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf3d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c991e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c901fa860 con 0x55a54bf4f8d0 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c991e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c901f8490 con 0x55a54bf01c30 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c991e8740 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c901f8a90 con 0x55a54bf028e0 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf3d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf3d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48172/0 (socket says 172.21.15.27:48172) 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.310+0000 7f9c8fe00640 1 -- 172.21.15.27:0/2908269385 learned_addr learned my addr 172.21.15.27:0/2908269385 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c8f400640 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01c30 0x55a54befc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c86a00640 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 0x55a54bf3ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c8fe00640 1 -- 172.21.15.27:0/2908269385 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 msgr2=0x55a54bf3ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 0x55a54bf3ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c8fe00640 1 -- 172.21.15.27:0/2908269385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01c30 msgr2=0x55a54befc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01c30 0x55a54befc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c8fe00640 1 -- 172.21.15.27:0/2908269385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a54bf3dfc0 con 0x55a54bf4f8d0 2024-03-28T18:56:12.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf3d3c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9c88001530 tx=0x7f9c880019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be64152c3fc6734 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c7f400640 1 -- 172.21.15.27:0/2908269385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88019070 con 0x55a54bf4f8d0 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c7f400640 1 -- 172.21.15.27:0/2908269385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c880052d0 con 0x55a54bf4f8d0 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.311+0000 7f9c7f400640 1 -- 172.21.15.27:0/2908269385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88007270 con 0x55a54bf4f8d0 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 -- 172.21.15.27:0/2908269385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 msgr2=0x55a54bf3d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf3d3c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9c88001530 tx=0x7f9c880019f0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 -- 172.21.15.27:0/2908269385 shutdown_connections 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf3d3c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf028e0 0x55a54bf3ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 --2- 172.21.15.27:0/2908269385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf01c30 0x55a54befc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 -- 172.21.15.27:0/2908269385 >> 172.21.15.27:0/2908269385 conn(0x55a54bf01850 msgr2=0x55a54bf39580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:12.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 -- 172.21.15.27:0/2908269385 shutdown_connections 2024-03-28T18:56:12.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.312+0000 7f9c991e8740 1 -- 172.21.15.27:0/2908269385 wait complete. 2024-03-28T18:56:12.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.313+0000 7f9c991e8740 1 -- start start 2024-03-28T18:56:12.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.313+0000 7f9c991e8740 1 Processor -- start 2024-03-28T18:56:12.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf01c30 0x55a54bf3d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 0x55a54bf60fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c991e8740 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf614f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c991e8740 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54bf22f60 con 0x55a54bf4f8d0 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c991e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a54bf3af00 con 0x55a54bf028e0 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c991e8740 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a54bf20650 con 0x55a54bf01c30 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c991e8740 10 client.0 _ll_register_callbacks cb 0x7f9c8005f0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf614f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48174/0 (socket says 172.21.15.27:48174) 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c8fe00640 1 -- 172.21.15.27:0/1273867933 learned_addr learned my addr 172.21.15.27:0/1273867933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c8f400640 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf01c30 0x55a54bf3d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.314+0000 7f9c86a00640 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 0x55a54bf60fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c8fe00640 1 -- 172.21.15.27:0/1273867933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf01c30 msgr2=0x55a54bf3d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf01c30 0x55a54bf3d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c8fe00640 1 -- 172.21.15.27:0/1273867933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 msgr2=0x55a54bf60fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 0x55a54bf60fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c8fe00640 1 -- 172.21.15.27:0/1273867933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a54bf60a70 con 0x55a54bf4f8d0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c8fe00640 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf614f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c88001a30 tx=0x7f9c88009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66924218fe1b97af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1273867933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88023020 con 0x55a54bf4f8d0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1273867933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f9c880069a0 con 0x55a54bf4f8d0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.315+0000 7f9c991e8740 1 -- 172.21.15.27:0/1273867933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c880052d0 con 0x55a54bf4f8d0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1273867933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c8802d050 con 0x55a54bf4f8d0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c991e8740 10 client.4869 fetch_fsmap learned FSMap version 15 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c991e8740 1 -- 172.21.15.27:0/1273867933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x55a54bf1f0f0 con 0x55a54bf4f8d0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1273867933 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c880052d0 con 0x55a54bf4f8d0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1273867933 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 15) v1 ==== 3918+0+0 (secure 0 0 0) 0x7f9c88030020 con 0x55a54bf4f8d0 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c991e8740 10 client.4869 fetch_fsmap finished waiting for FSMap version 15 2024-03-28T18:56:12.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c991e8740 10 client.4869 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c991e8740 20 client.4869 populate_metadata read hostname 'smithi027' 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] conn(0x55a54bf3edb0 0x55a54bf3f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c991e8740 4 client.4869 mds_command: new command op to 4263 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c991e8740 1 -- 172.21.15.27:0/1273867933 --> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a54befe680 con 0x55a54bf3edb0 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.316+0000 7f9c8f400640 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] conn(0x55a54bf3edb0 0x55a54bf3f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c8f400640 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] conn(0x55a54bf3edb0 0x55a54bf3f190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b2debc79aef4669 server_cookie=27f7cb885ac8199b in_seq=0 out_seq=0 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c7cc00640 10 client.4869 ms_handle_connect on v2:172.21.15.27:6835/2601859893 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c7cc00640 1 -- 172.21.15.27:0/1273867933 <== mds.0 v2:172.21.15.27:6835/2601859893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x55a54befe680 con 0x55a54bf3edb0 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c7cc00640 10 client.4869 handle_command_reply: tid=0 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 client.4869 shutdown 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 -- 172.21.15.27:0/1273867933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 msgr2=0x55a54bf614f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf614f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c88001a30 tx=0x7f9c88009040 comp rx=0 tx=0).stop 2024-03-28T18:56:12.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 -- 172.21.15.27:0/1273867933 shutdown_connections 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.27:6835/2601859893,v1:172.21.15.27:6837/2601859893] conn(0x55a54bf3edb0 0x55a54bf3f190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a54bf4f8d0 0x55a54bf614f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a54bf028e0 0x55a54bf60fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 --2- 172.21.15.27:0/1273867933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a54bf01c30 0x55a54bf3d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c7cc00640 0 client.4869 ms_handle_reset on v2:172.21.15.27:6835/2601859893 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.317+0000 7f9c991e8740 1 -- 172.21.15.27:0/1273867933 >> 172.21.15.27:0/1273867933 conn(0x55a54bf01850 msgr2=0x55a54bf3c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.318+0000 7f9c991e8740 1 -- 172.21.15.27:0/1273867933 shutdown_connections 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.318+0000 7f9c991e8740 1 -- 172.21.15.27:0/1273867933 wait complete. 2024-03-28T18:56:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.318+0000 7f9c991e8740 20 client.4869 trim_cache size 0 max 16384 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "c27ecc06-ff19-45f9-bdba-d624e1db1fbd", 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4263, 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "b", 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 26.728827926000001, 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 15, 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 24, 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 24, 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 43.165230078 2024-03-28T18:56:12.323 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-28T18:56:12.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.318+0000 7f9c94e00640 1 -- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9c6c06bbd0 msgr2=0x7f9c6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.318+0000 7f9c94e00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9c6c06bbd0 0x7f9c6c06e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9c78036000 tx=0x7f9c78008030 comp rx=0 tx=0).stop 2024-03-28T18:56:12.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.318+0000 7f9c94e00640 1 -- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c90072b60 msgr2=0x7f9c900c1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.318+0000 7f9c94e00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c90072b60 0x7f9c900c1720 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000ba80 tx=0x7f9c8000bf40 comp rx=0 tx=0).stop 2024-03-28T18:56:12.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.319+0000 7f9c94e00640 1 -- 172.21.15.27:0/2470010566 shutdown_connections 2024-03-28T18:56:12.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.319+0000 7f9c94e00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9c6c06bbd0 0x7f9c6c06e090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.319+0000 7f9c94e00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c900c8db0 0x7f9c90151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.319+0000 7f9c94e00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900c5aa0 0x7f9c900bfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.319+0000 7f9c94e00640 1 --2- 172.21.15.27:0/2470010566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c90072b60 0x7f9c900c1720 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.319+0000 7f9c94e00640 1 -- 172.21.15.27:0/2470010566 >> 172.21.15.27:0/2470010566 conn(0x7f9c9006b0c0 msgr2=0x7f9c901539b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:12.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.319+0000 7f9c94e00640 1 -- 172.21.15.27:0/2470010566 shutdown_connections 2024-03-28T18:56:12.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.319+0000 7f9c94e00640 1 -- 172.21.15.27:0/2470010566 wait complete. 2024-03-28T18:56:12.335 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-28T18:56:12.360 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-28T18:56:12.360 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-03-28T18:56:12.360 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'a'} 2024-03-28T18:56:12.360 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T18:56:12.360 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:12.360 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'b'} 2024-03-28T18:56:12.360 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T18:56:12.360 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:12.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:12.361 DEBUG:teuthology.orchestra.run.smithi155:> ip netns list 2024-03-28T18:56:12.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:12.420 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link delete ceph-brx 2024-03-28T18:56:12.505 INFO:teuthology.orchestra.run.smithi155.stderr:Cannot find device "ceph-brx" 2024-03-28T18:56:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:56:12.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:12.509 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-28T18:56:12.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:12.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-03-28T18:56:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-03-28T18:56:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:56:12.666 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-28T18:56:12.666 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-28T18:56:12.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.887+0000 7f9e9fe00640 1 Processor -- start 2024-03-28T18:56:12.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.887+0000 7f9e9fe00640 1 -- start start 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.889+0000 7f9e9fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9ea00abca0 0x7f9ea0153c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.889+0000 7f9e9fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 0x7f9ea00ac080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.889+0000 7f9e9fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9ea00ac5c0 0x7f9ea006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.889+0000 7f9e9fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea0062370 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.889+0000 7f9e9fe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea0062650 con 0x7f9ea00abca0 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.889+0000 7f9e9fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00acf90 con 0x7f9ea00aa520 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.889+0000 7f9e9f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9ea00ac5c0 0x7f9ea006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9ea00ac5c0 0x7f9ea006dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48176/0 (socket says 172.21.15.27:48176) 2024-03-28T18:56:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9f400640 1 -- 172.21.15.27:0/2480100071 learned_addr learned my addr 172.21.15.27:0/2480100071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9ea00640 1 --2- 172.21.15.27:0/2480100071 >> v2:172.21.15.155:3300/0 conn(0x7f9ea00abca0 0x7f9ea0153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9e000640 1 --2- 172.21.15.27:0/2480100071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 0x7f9ea00ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9f400640 1 -- 172.21.15.27:0/2480100071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 msgr2=0x7f9ea00ac080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9f400640 1 --2- 172.21.15.27:0/2480100071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 0x7f9ea00ac080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9f400640 1 -- 172.21.15.27:0/2480100071 >> v2:172.21.15.155:3300/0 conn(0x7f9ea00abca0 msgr2=0x7f9ea0153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9f400640 1 --2- 172.21.15.27:0/2480100071 >> v2:172.21.15.155:3300/0 conn(0x7f9ea00abca0 0x7f9ea0153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.890+0000 7f9e9f400640 1 -- 172.21.15.27:0/2480100071 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea00ad270 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.892+0000 7f9e9f400640 1 --2- 172.21.15.27:0/2480100071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea006dee0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0070a40 tx=0x7f9e9802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e55db83d465039 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.892+0000 7f9e9d600640 1 -- 172.21.15.27:0/2480100071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9802c9e0 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.892+0000 7f9e9d600640 1 -- 172.21.15.27:0/2480100071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9802cb80 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.892+0000 7f9e9d600640 1 -- 172.21.15.27:0/2480100071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e980343f0 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 -- 172.21.15.27:0/2480100071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 msgr2=0x7f9ea006dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/2480100071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea006dee0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0070a40 tx=0x7f9e9802be40 comp rx=0 tx=0).stop 2024-03-28T18:56:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 -- 172.21.15.27:0/2480100071 shutdown_connections 2024-03-28T18:56:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/2480100071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea006dee0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/2480100071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 0x7f9ea00ac080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/2480100071 >> v2:172.21.15.155:3300/0 conn(0x7f9ea00abca0 0x7f9ea0153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 -- 172.21.15.27:0/2480100071 >> 172.21.15.27:0/2480100071 conn(0x7f9ea006b0c0 msgr2=0x7f9ea006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 -- 172.21.15.27:0/2480100071 shutdown_connections 2024-03-28T18:56:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.893+0000 7f9e9fe00640 1 -- 172.21.15.27:0/2480100071 wait complete. 2024-03-28T18:56:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.895+0000 7f9e9fe00640 1 Processor -- start 2024-03-28T18:56:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.895+0000 7f9e9fe00640 1 -- start start 2024-03-28T18:56:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.896+0000 7f9e9fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 0x7f9ea014c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.896+0000 7f9e9fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea00abca0 0x7f9ea01525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.896+0000 7f9e9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea0138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.896+0000 7f9e9fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea006e420 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.896+0000 7f9e9fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea014a0a0 con 0x7f9ea00abca0 2024-03-28T18:56:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.896+0000 7f9e9fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea006e5a0 con 0x7f9ea00aa520 2024-03-28T18:56:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.896+0000 7f9e9f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea0138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea0138880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48192/0 (socket says 172.21.15.27:48192) 2024-03-28T18:56:12.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9f400640 1 -- 172.21.15.27:0/3723936423 learned_addr learned my addr 172.21.15.27:0/3723936423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:12.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9e000640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea00abca0 0x7f9ea01525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9ea00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 0x7f9ea014c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9f400640 1 -- 172.21.15.27:0/3723936423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 msgr2=0x7f9ea014c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9f400640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 0x7f9ea014c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9f400640 1 -- 172.21.15.27:0/3723936423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea00abca0 msgr2=0x7f9ea01525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9f400640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea00abca0 0x7f9ea01525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9f400640 1 -- 172.21.15.27:0/3723936423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea01514f0 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.897+0000 7f9e9f400640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea0138880 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9e9802c450 tx=0x7f9e98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.898+0000 7f9e97400640 1 -- 172.21.15.27:0/3723936423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98043020 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.898+0000 7f9e97400640 1 -- 172.21.15.27:0/3723936423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9803e450 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.898+0000 7f9e97400640 1 -- 172.21.15.27:0/3723936423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98009510 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.898+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea01517d0 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.898+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea014c0c0 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.900+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea0067dd0 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.900+0000 7f9e97400640 1 -- 172.21.15.27:0/3723936423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9e98046020 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.901+0000 7f9e97400640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8006bbd0 0x7f9e8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:12.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.901+0000 7f9e97400640 1 -- 172.21.15.27:0/3723936423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6521+0+0 (secure 0 0 0) 0x7f9e980ad340 con 0x7f9ea00ac5c0 2024-03-28T18:56:12.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.901+0000 7f9e9ea00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8006bbd0 0x7f9e8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:12.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.902+0000 7f9e9ea00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8006bbd0 0x7f9e8006e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009b40 tx=0x7f9e90009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:12.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:12.910+0000 7f9e97400640 1 -- 172.21.15.27:0/3723936423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9e9804d030 con 0x7f9ea00ac5c0 2024-03-28T18:56:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.201+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9ea0069640 con 0x7f9ea00ac5c0 2024-03-28T18:56:13.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.202+0000 7f9e97400640 1 -- 172.21.15.27:0/3723936423 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+116 (secure 0 0 0) 0x7f9e9807b560 con 0x7f9ea00ac5c0 2024-03-28T18:56:13.203 INFO:teuthology.orchestra.run.smithi027.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-03-28T18:56:13.203 INFO:teuthology.orchestra.run.smithi027.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-03-28T18:56:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8006bbd0 msgr2=0x7f9e8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8006bbd0 0x7f9e8006e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009b40 tx=0x7f9e90009190 comp rx=0 tx=0).stop 2024-03-28T18:56:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 msgr2=0x7f9ea0138880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea0138880 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9e9802c450 tx=0x7f9e98004a40 comp rx=0 tx=0).stop 2024-03-28T18:56:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 shutdown_connections 2024-03-28T18:56:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8006bbd0 0x7f9e8006e090 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea00ac5c0 0x7f9ea0138880 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea00abca0 0x7f9ea01525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:13.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/3723936423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea00aa520 0x7f9ea014c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:13.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.204+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 >> 172.21.15.27:0/3723936423 conn(0x7f9ea006b0c0 msgr2=0x7f9ea00c59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:13.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.205+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 shutdown_connections 2024-03-28T18:56:13.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:13.205+0000 7f9e9fe00640 1 -- 172.21.15.27:0/3723936423 wait complete. 2024-03-28T18:56:13.225 INFO:tasks.cephfs.mount:Mounting Ceph FS a; just confirmed its presence on cluster 2024-03-28T18:56:13.225 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T18:56:13.225 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-03-28T18:56:13.225 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T18:56:13.225 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:13.225 INFO:tasks.cephfs.mount:self.cephfs_name = a 2024-03-28T18:56:13.226 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T18:56:13.226 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T18:56:13.226 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T18:56:13.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:13.226 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-03-28T18:56:13.280 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T18:56:13.280 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T18:56:13.280 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T18:56:13.280 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T18:56:13.281 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-03-28T18:56:13.282 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-03-28T18:56:13.282 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T18:56:13.282 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42533sec preferred_lft 42533sec 2024-03-28T18:56:13.282 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-03-28T18:56:13.282 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:56:13.282 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:56:13.282 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-03-28T18:56:13.282 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-03-28T18:56:13.283 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T18:56:13.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:13.283 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-28T18:56:13.283 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-28T18:56:13.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-03-28T18:56:13.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-03-28T18:56:13.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-03-28T18:56:13.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T18:56:13.283 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-28T18:56:13.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:13.768 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T18:56:13.850 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-03-28T18:56:13.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:13.854 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-03-28T18:56:13.910 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-03-28T18:56:13.910 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T18:56:13.910 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T18:56:13.910 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T18:56:13.911 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T18:56:13.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:13.911 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-28T18:56:13.911 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-28T18:56:13.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T18:56:13.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T18:56:13.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T18:56:13.912 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-28T18:56:14.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:14.643 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-03-28T18:56:14.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:14.694 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-03-28T18:56:14.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:14.740 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-28T18:56:14.740 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-28T18:56:14.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:56:14.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T18:56:14.740 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-28T18:56:14.937 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T18:56:14.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:14.937 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-28T18:56:14.937 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-28T18:56:14.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T18:56:14.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T18:56:14.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T18:56:14.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T18:56:14.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T18:56:14.938 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-28T18:56:15.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:15.226 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-03-28T18:56:15.226 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-03-28T18:56:15.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-03-28T18:56:15.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T18:56:15.226 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-03-28T18:56:15.389 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'a'} 2024-03-28T18:56:15.389 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:56:15.389 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:15.433 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T18:56:15.433 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:56:15.433 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:15.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-28T18:56:15.615 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T18:56:15.665 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-28T18:56:15.666 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-28T18:56:15.667 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T18:56:15.668 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T18:56:15.668 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T18:56:15.668 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T18:56:15.668 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-28T18:56:15.668 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:56:15.668 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:56:15.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:15.669 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-28T18:56:15.714 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T18:56:15.714 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=a) 2024-03-28T18:56:15.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-28T18:56:15.837 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-28T18:56:15.879 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-28T18:56:15.880 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:56:15.881 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:56:15.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:15.882 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-28T18:56:15.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-03-28T18:56:15.953+0000 7f56e0ef4480 -1 init, newargv = 0x55a620bf6570 newargc=15 2024-03-28T18:56:15.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[46716]: starting ceph client 2024-03-28T18:56:15.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[46716]: starting fuse 2024-03-28T18:56:17.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-03-28T18:56:17.177 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T18:56:17.232 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-03-28T18:56:17.232 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-03-28T18:56:17.233 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-03-28T18:56:17.234 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-03-28T18:56:17.235 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-03-28T18:56:17.235 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-03-28T18:56:17.235 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:56:17.235 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:56:17.235 INFO:teuthology.orchestra.run.smithi027.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:17.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:17.236 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-03-28T18:56:17.287 INFO:teuthology.orchestra.run.smithi027.stdout:41 2024-03-28T18:56:17.287 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-03-28T18:56:17.288 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-03-28T18:56:17.289 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-03-28T18:56:17.474 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.46716.asok 2024-03-28T18:56:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T18:56:17.493 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46716.asok 2024-03-28T18:56:17.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:17.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46716.asok status 2024-03-28T18:56:17.748 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-03-28T18:56:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-03-28T18:56:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-28T18:56:17.748 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "46716", 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4881, 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 4881 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-28T18:56:17.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2547948779 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2547948779 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.4881 192.168.0.1:0/2547948779", 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/2547948779", 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 9, 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 25, 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-03-28T18:56:17.750 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-03-28T18:56:17.751 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "a" 2024-03-28T18:56:17.751 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-03-28T18:56:17.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-28T18:56:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.994+0000 7fb367e00640 1 Processor -- start 2024-03-28T18:56:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.994+0000 7fb367e00640 1 -- start start 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb367e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3680c69a0 0x7fb3680c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb367e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb368153a30 0x7fb368153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb367e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb368150ec0 con 0x7fb368150ae0 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb367e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3681543c0 con 0x7fb3680c69a0 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb367e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36806dc70 con 0x7fb368153a30 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb366000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3680c69a0 0x7fb3680c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb366000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb3680c69a0 0x7fb3680c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44054/0 (socket says 172.21.15.27:44054) 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb366000640 1 -- 172.21.15.27:0/3474957880 learned_addr learned my addr 172.21.15.27:0/3474957880 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.996+0000 7fb367400640 1 --2- 172.21.15.27:0/3474957880 >> v2:172.21.15.155:3300/0 conn(0x7fb368153a30 0x7fb368153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.997+0000 7fb366000640 1 -- 172.21.15.27:0/3474957880 >> v2:172.21.15.155:3300/0 conn(0x7fb368153a30 msgr2=0x7fb368153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.997+0000 7fb366000640 1 --2- 172.21.15.27:0/3474957880 >> v2:172.21.15.155:3300/0 conn(0x7fb368153a30 0x7fb368153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.997+0000 7fb366000640 1 -- 172.21.15.27:0/3474957880 >> v1:172.21.15.155:6789/0 conn(0x7fb368150ae0 legacy=0x7fb3680a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.997+0000 7fb366000640 1 -- 172.21.15.27:0/3474957880 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb368153550 con 0x7fb3680c69a0 2024-03-28T18:56:17.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.998+0000 7fb366000640 1 --2- 172.21.15.27:0/3474957880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb3680c6e30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb3680c0af0 tx=0x7fb35402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4dd70c9d8bd633a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.998+0000 7fb365600640 1 -- 172.21.15.27:0/3474957880 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354035020 con 0x7fb3680c69a0 2024-03-28T18:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.998+0000 7fb365600640 1 -- 172.21.15.27:0/3474957880 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb354031870 con 0x7fb3680c69a0 2024-03-28T18:56:17.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.998+0000 7fb365600640 1 -- 172.21.15.27:0/3474957880 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354031c40 con 0x7fb3680c69a0 2024-03-28T18:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.999+0000 7fb367e00640 1 -- 172.21.15.27:0/3474957880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 msgr2=0x7fb3680c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.999+0000 7fb367e00640 1 --2- 172.21.15.27:0/3474957880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb3680c6e30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb3680c0af0 tx=0x7fb35402b910 comp rx=0 tx=0).stop 2024-03-28T18:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.999+0000 7fb367e00640 1 -- 172.21.15.27:0/3474957880 shutdown_connections 2024-03-28T18:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.999+0000 7fb367e00640 1 --2- 172.21.15.27:0/3474957880 >> v2:172.21.15.155:3300/0 conn(0x7fb368153a30 0x7fb368153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.999+0000 7fb367e00640 1 --2- 172.21.15.27:0/3474957880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb3680c6e30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.999+0000 7fb367e00640 1 -- 172.21.15.27:0/3474957880 >> 172.21.15.27:0/3474957880 conn(0x7fb36806b0c0 msgr2=0x7fb36806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.999+0000 7fb367e00640 1 -- 172.21.15.27:0/3474957880 shutdown_connections 2024-03-28T18:56:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:17.999+0000 7fb367e00640 1 -- 172.21.15.27:0/3474957880 wait complete. 2024-03-28T18:56:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.001+0000 7fb367e00640 1 Processor -- start 2024-03-28T18:56:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.001+0000 7fb367e00640 1 -- start start 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb367e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb368151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb367e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb368150ae0 0x7fb36814fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb367e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb368153a30 0x7fb36814f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb367e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36806e4d0 con 0x7fb3680c69a0 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb367e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36806e1f0 con 0x7fb368150ae0 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb367e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36806e7b0 con 0x7fb368153a30 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb366a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb368151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb366a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb368151520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44056/0 (socket says 172.21.15.27:44056) 2024-03-28T18:56:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb366a00640 1 -- 172.21.15.27:0/424106068 learned_addr learned my addr 172.21.15.27:0/424106068 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:18.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.002+0000 7fb367400640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb368153a30 0x7fb36814f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:18.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb366000640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb368150ae0 0x7fb36814fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb366a00640 1 -- 172.21.15.27:0/424106068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb368153a30 msgr2=0x7fb36814f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb366a00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb368153a30 0x7fb36814f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb366a00640 1 -- 172.21.15.27:0/424106068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb368150ae0 msgr2=0x7fb36814fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb366a00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb368150ae0 0x7fb36814fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb366a00640 1 -- 172.21.15.27:0/424106068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb368073140 con 0x7fb3680c69a0 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb366a00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb368151520 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb35800c400 tx=0x7fb35800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb35f400640 1 -- 172.21.15.27:0/424106068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3580052d0 con 0x7fb3680c69a0 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.003+0000 7fb35f400640 1 -- 172.21.15.27:0/424106068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb358005490 con 0x7fb3680c69a0 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.004+0000 7fb35f400640 1 -- 172.21.15.27:0/424106068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358007260 con 0x7fb3680c69a0 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.004+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3681dcc90 con 0x7fb3680c69a0 2024-03-28T18:56:18.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.004+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3681ebef0 con 0x7fb3680c69a0 2024-03-28T18:56:18.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.005+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb368073310 con 0x7fb3680c69a0 2024-03-28T18:56:18.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.006+0000 7fb35f400640 1 -- 172.21.15.27:0/424106068 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb358007400 con 0x7fb3680c69a0 2024-03-28T18:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.007+0000 7fb35f400640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb34806bc00 0x7fb34806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.007+0000 7fb35f400640 1 -- 172.21.15.27:0/424106068 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fb35808a3d0 con 0x7fb3680c69a0 2024-03-28T18:56:18.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.008+0000 7fb366000640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb34806bc00 0x7fb34806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:18.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.008+0000 7fb366000640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb34806bc00 0x7fb34806e0c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb354036000 tx=0x7fb354008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:18.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.012+0000 7fb35f400640 1 -- 172.21.15.27:0/424106068 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb368150ec0 con 0x7fb3680c69a0 2024-03-28T18:56:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.254+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb368067dd0 con 0x7fb3680c69a0 2024-03-28T18:56:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.255+0000 7fb35f400640 1 -- 172.21.15.27:0/424106068 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+116 (secure 0 0 0) 0x7fb358058a80 con 0x7fb3680c69a0 2024-03-28T18:56:18.255 INFO:teuthology.orchestra.run.smithi027.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-03-28T18:56:18.255 INFO:teuthology.orchestra.run.smithi027.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb34806bc00 msgr2=0x7fb34806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb34806bc00 0x7fb34806e0c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb354036000 tx=0x7fb354008030 comp rx=0 tx=0).stop 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 msgr2=0x7fb368151520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb368151520 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb35800c400 tx=0x7fb35800c8c0 comp rx=0 tx=0).stop 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 shutdown_connections 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb34806bc00 0x7fb34806e0c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb368153a30 0x7fb36814f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb368150ae0 0x7fb36814fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 --2- 172.21.15.27:0/424106068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3680c69a0 0x7fb368151520 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.257+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 >> 172.21.15.27:0/424106068 conn(0x7fb36806b0c0 msgr2=0x7fb36806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.258+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 shutdown_connections 2024-03-28T18:56:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:18.258+0000 7fb367e00640 1 -- 172.21.15.27:0/424106068 wait complete. 2024-03-28T18:56:18.277 INFO:tasks.cephfs.mount:Mounting Ceph FS b; just confirmed its presence on cluster 2024-03-28T18:56:18.277 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T18:56:18.277 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi155.front.sepia.ceph.com 2024-03-28T18:56:18.277 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-28T18:56:18.277 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:18.278 INFO:tasks.cephfs.mount:self.cephfs_name = b 2024-03-28T18:56:18.278 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T18:56:18.278 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T18:56:18.278 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-28T18:56:18.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:18.278 DEBUG:teuthology.orchestra.run.smithi155:> ip addr 2024-03-28T18:56:18.330 INFO:teuthology.orchestra.run.smithi155.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: inet6 ::1/128 scope host 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: link/ether 0c:c4:7a:d9:a0:fa brd ff:ff:ff:ff:ff:ff 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: altname enp7s0f0 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: link/ether 0c:c4:7a:d9:a0:fb brd ff:ff:ff:ff:ff:ff 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout: altname enp7s0f1 2024-03-28T18:56:18.331 INFO:teuthology.orchestra.run.smithi155.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:56:18.332 INFO:teuthology.orchestra.run.smithi155.stdout: link/ether 0c:c4:7a:88:6d:ec brd ff:ff:ff:ff:ff:ff 2024-03-28T18:56:18.332 INFO:teuthology.orchestra.run.smithi155.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T18:56:18.332 INFO:teuthology.orchestra.run.smithi155.stdout: link/ether 0c:c4:7a:88:6d:ed brd ff:ff:ff:ff:ff:ff 2024-03-28T18:56:18.332 INFO:teuthology.orchestra.run.smithi155.stdout: inet 172.21.15.155/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T18:56:18.332 INFO:teuthology.orchestra.run.smithi155.stdout: valid_lft 42516sec preferred_lft 42516sec 2024-03-28T18:56:18.332 INFO:teuthology.orchestra.run.smithi155.stdout: inet6 fe80::ec4:7aff:fe88:6ded/64 scope link 2024-03-28T18:56:18.332 INFO:teuthology.orchestra.run.smithi155.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:56:18.332 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T18:56:18.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:18.333 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec bash -c ' 2024-03-28T18:56:18.333 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-28T18:56:18.333 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link add name ceph-brx type bridge 2024-03-28T18:56:18.333 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip addr flush dev ceph-brx 2024-03-28T18:56:18.333 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link set ceph-brx up 2024-03-28T18:56:18.333 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T18:56:18.333 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-28T18:56:18.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:18.780 DEBUG:teuthology.orchestra.run.smithi155:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T18:56:18.854 INFO:teuthology.orchestra.run.smithi155.stdout:1 2024-03-28T18:56:18.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:18.858 DEBUG:teuthology.orchestra.run.smithi155:> route 2024-03-28T18:56:18.927 INFO:teuthology.orchestra.run.smithi155.stdout:Kernel IP routing table 2024-03-28T18:56:18.928 INFO:teuthology.orchestra.run.smithi155.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T18:56:18.928 INFO:teuthology.orchestra.run.smithi155.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T18:56:18.928 INFO:teuthology.orchestra.run.smithi155.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T18:56:18.928 INFO:teuthology.orchestra.run.smithi155.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T18:56:18.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:18.928 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec bash -c ' 2024-03-28T18:56:18.928 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-28T18:56:18.928 DEBUG:teuthology.orchestra.run.smithi155:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T18:56:18.928 DEBUG:teuthology.orchestra.run.smithi155:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T18:56:18.929 DEBUG:teuthology.orchestra.run.smithi155:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T18:56:18.929 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-28T18:56:19.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:19.640 DEBUG:teuthology.orchestra.run.smithi155:> ip netns list 2024-03-28T18:56:19.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:19.694 DEBUG:teuthology.orchestra.run.smithi155:> ip netns list-id 2024-03-28T18:56:19.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:19.745 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec bash -c ' 2024-03-28T18:56:19.745 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-28T18:56:19.745 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T18:56:19.745 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-28T18:56:19.745 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-28T18:56:19.881 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-28T18:56:19.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:19.881 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec bash -c ' 2024-03-28T18:56:19.881 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-28T18:56:19.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-28T18:56:19.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T18:56:19.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-28T18:56:19.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-28T18:56:19.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-28T18:56:19.881 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-28T18:56:20.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:20.139 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec bash -c ' 2024-03-28T18:56:20.139 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-28T18:56:20.139 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link set brx.0 up 2024-03-28T18:56:20.139 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T18:56:20.139 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-28T18:56:20.289 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'b'} 2024-03-28T18:56:20.289 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:56:20.289 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:20.331 INFO:teuthology.orchestra.run.smithi155.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-28T18:56:20.332 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:56:20.332 DEBUG:teuthology.orchestra.run.smithi155:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:20.418 DEBUG:teuthology.orchestra.run.smithi155:> sudo modprobe fuse 2024-03-28T18:56:20.550 DEBUG:teuthology.orchestra.run.smithi155:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T18:56:20.605 INFO:teuthology.orchestra.run.smithi155.stdout:/proc 2024-03-28T18:56:20.605 INFO:teuthology.orchestra.run.smithi155.stdout:/sys 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/dev 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/security 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/shm 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/pts 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/run 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/cgroup 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/pstore 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/bpf 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/ 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/selinux 2024-03-28T18:56:20.606 INFO:teuthology.orchestra.run.smithi155.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/hugepages 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/mqueue 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/debug 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/tracing 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/fuse/connections 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/config 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/run/user/1000 2024-03-28T18:56:20.607 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph 2024-03-28T18:56:20.608 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-4 2024-03-28T18:56:20.608 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-5 2024-03-28T18:56:20.608 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-6 2024-03-28T18:56:20.608 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-7 2024-03-28T18:56:20.608 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns 2024-03-28T18:56:20.608 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T18:56:20.608 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T18:56:20.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:20.609 DEBUG:teuthology.orchestra.run.smithi155:> ls /sys/fs/fuse/connections 2024-03-28T18:56:20.653 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T18:56:20.654 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=b) 2024-03-28T18:56:20.701 DEBUG:teuthology.orchestra.run.smithi155:> sudo modprobe fuse 2024-03-28T18:56:20.774 DEBUG:teuthology.orchestra.run.smithi155:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T18:56:20.817 INFO:teuthology.orchestra.run.smithi155.stdout:/proc 2024-03-28T18:56:20.817 INFO:teuthology.orchestra.run.smithi155.stdout:/sys 2024-03-28T18:56:20.817 INFO:teuthology.orchestra.run.smithi155.stdout:/dev 2024-03-28T18:56:20.817 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/security 2024-03-28T18:56:20.817 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/shm 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/pts 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/run 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/cgroup 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/pstore 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/bpf 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/ 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/selinux 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:56:20.818 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/hugepages 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/mqueue 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/debug 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/tracing 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/fuse/connections 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/config 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T18:56:20.819 INFO:teuthology.orchestra.run.smithi155.stdout:/run/user/1000 2024-03-28T18:56:20.820 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph 2024-03-28T18:56:20.820 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-4 2024-03-28T18:56:20.820 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-5 2024-03-28T18:56:20.820 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-6 2024-03-28T18:56:20.820 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-7 2024-03-28T18:56:20.820 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns 2024-03-28T18:56:20.820 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T18:56:20.820 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T18:56:20.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:20.821 DEBUG:teuthology.orchestra.run.smithi155:> ls /sys/fs/fuse/connections 2024-03-28T18:56:20.894 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi155.stderr:2024-03-28T18:56:20.894+0000 7f9cae045480 -1 init, newargv = 0x5607bf6f71e0 newargc=15 2024-03-28T18:56:20.894 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi155.stderr:ceph-fuse[42781]: starting ceph client 2024-03-28T18:56:20.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi155.stderr:ceph-fuse[42781]: starting fuse 2024-03-28T18:56:22.008 DEBUG:teuthology.orchestra.run.smithi155:> sudo modprobe fuse 2024-03-28T18:56:22.110 DEBUG:teuthology.orchestra.run.smithi155:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/proc 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/sys 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/dev 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/security 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/shm 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/pts 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/run 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/cgroup 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/pstore 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/bpf 2024-03-28T18:56:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:/ 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/selinux 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/hugepages 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/mqueue 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/debug 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/tracing 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/fs/fuse/connections 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/sys/kernel/config 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/run/user/1000 2024-03-28T18:56:22.167 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph 2024-03-28T18:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-4 2024-03-28T18:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-5 2024-03-28T18:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-6 2024-03-28T18:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout:/var/lib/ceph/osd/ceph-7 2024-03-28T18:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns 2024-03-28T18:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T18:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-28T18:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:22.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:22.169 DEBUG:teuthology.orchestra.run.smithi155:> ls /sys/fs/fuse/connections 2024-03-28T18:56:22.216 INFO:teuthology.orchestra.run.smithi155.stdout:42 2024-03-28T18:56:22.216 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-28T18:56:22.216 DEBUG:teuthology.orchestra.run.smithi155:> sudo stdin-killer -- python3 -c ' 2024-03-28T18:56:22.216 DEBUG:teuthology.orchestra.run.smithi155:> import glob 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> import re 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> import os 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> import subprocess 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> def _find_admin_socket(client_name): 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> files = glob.glob(asok_path) 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> # Given a non-glob path, it better be there 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> if "*" not in asok_path: 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> assert(len(files) == 1) 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> return files[0] 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> for f in files: 2024-03-28T18:56:22.217 DEBUG:teuthology.orchestra.run.smithi155:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> contents = proc_f.read() 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> if mountpoint in contents: 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> return f 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> print(_find_admin_socket("client.1")) 2024-03-28T18:56:22.218 DEBUG:teuthology.orchestra.run.smithi155:> ' 2024-03-28T18:56:22.387 INFO:teuthology.orchestra.run.smithi155.stdout:/var/run/ceph/ceph-client.1.42781.asok 2024-03-28T18:56:22.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-28T18:56:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T18:56:22.404 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42781.asok 2024-03-28T18:56:22.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:22.404 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42781.asok status 2024-03-28T18:56:22.639 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "metadata": { 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "entity_id": "1", 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "hostname": "smithi155", 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "pid": "42781", 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "root": "/" 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "dentry_count": 0, 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "dentry_pinned_count": 0, 2024-03-28T18:56:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: "id": 4738, 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "inst": { 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "name": { 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "client", 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "num": 4738 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "addr": { 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "v1", 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "addr": "192.168.0.1:0", 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "nonce": 4127479828 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "addr": { 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "v1", 2024-03-28T18:56:22.641 INFO:teuthology.orchestra.run.smithi155.stdout: "addr": "192.168.0.1:0", 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "nonce": 4127479828 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "inst_str": "client.4738 192.168.0.1:0/4127479828", 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "addr_str": "192.168.0.1:0/4127479828", 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "inode_count": 1, 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_epoch": 15, 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_epoch": 26, 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_epoch_barrier": 0, 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "blocklisted": false, 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout: "fs_name": "b" 2024-03-28T18:56:22.642 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-28T18:56:22.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:22.655 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-03-28T18:56:22.655 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:22.714 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-03-28T18:56:22.714 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:22.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:22.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:22.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:22.821 DEBUG:teuthology.orchestra.run.smithi155:> stat --file-system '--printf=%T 2024-03-28T18:56:22.821 DEBUG:teuthology.orchestra.run.smithi155:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:22.873 INFO:teuthology.orchestra.run.smithi155.stdout:fuseblk 2024-03-28T18:56:22.873 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:22.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:56:22.874 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:22.971 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-03-28T18:56:23.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.235+0000 7f6381400640 1 Processor -- start 2024-03-28T18:56:23.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.235+0000 7f6381400640 1 -- start start 2024-03-28T18:56:23.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f6381400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f637c0c8ed0 0x7f637c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f6381400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f637c0be5d0 0x7f637c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f6381400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c156230 con 0x7f637c0c58c0 2024-03-28T18:56:23.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f6381400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637c158bd0 con 0x7f637c0be5d0 2024-03-28T18:56:23.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f6381400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637c158eb0 con 0x7f637c0c8ed0 2024-03-28T18:56:23.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f6380a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f637c0be5d0 0x7f637c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f6380a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f637c0be5d0 0x7f637c155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44066/0 (socket says 172.21.15.27:44066) 2024-03-28T18:56:23.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f6380a00640 1 -- 172.21.15.27:0/1029330544 learned_addr learned my addr 172.21.15.27:0/1029330544 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:23.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.237+0000 7f637b400640 1 --2- 172.21.15.27:0/1029330544 >> v2:172.21.15.155:3300/0 conn(0x7f637c0c8ed0 0x7f637c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.238+0000 7f6380a00640 1 -- 172.21.15.27:0/1029330544 >> v2:172.21.15.155:3300/0 conn(0x7f637c0c8ed0 msgr2=0x7f637c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:23.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.238+0000 7f6380a00640 1 --2- 172.21.15.27:0/1029330544 >> v2:172.21.15.155:3300/0 conn(0x7f637c0c8ed0 0x7f637c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.238+0000 7f6380a00640 1 -- 172.21.15.27:0/1029330544 >> v1:172.21.15.155:6789/0 conn(0x7f637c0c58c0 legacy=0x7f637c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.238+0000 7f6380a00640 1 -- 172.21.15.27:0/1029330544 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c159190 con 0x7f637c0be5d0 2024-03-28T18:56:23.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.239+0000 7f6380a00640 1 --2- 172.21.15.27:0/1029330544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c155cf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f637c0c83d0 tx=0x7f637402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e161db5239a303 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:23.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.239+0000 7f637aa00640 1 -- 172.21.15.27:0/1029330544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637402c9f0 con 0x7f637c0be5d0 2024-03-28T18:56:23.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.239+0000 7f637aa00640 1 -- 172.21.15.27:0/1029330544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637402cb90 con 0x7f637c0be5d0 2024-03-28T18:56:23.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.239+0000 7f637aa00640 1 -- 172.21.15.27:0/1029330544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63740343f0 con 0x7f637c0be5d0 2024-03-28T18:56:23.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.240+0000 7f6381400640 1 -- 172.21.15.27:0/1029330544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 msgr2=0x7f637c155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.240+0000 7f6381400640 1 --2- 172.21.15.27:0/1029330544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c155cf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f637c0c83d0 tx=0x7f637402be50 comp rx=0 tx=0).stop 2024-03-28T18:56:23.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.240+0000 7f6381400640 1 -- 172.21.15.27:0/1029330544 shutdown_connections 2024-03-28T18:56:23.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.240+0000 7f6381400640 1 --2- 172.21.15.27:0/1029330544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c155cf0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.240+0000 7f6381400640 1 --2- 172.21.15.27:0/1029330544 >> v2:172.21.15.155:3300/0 conn(0x7f637c0c8ed0 0x7f637c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.240+0000 7f6381400640 1 -- 172.21.15.27:0/1029330544 >> 172.21.15.27:0/1029330544 conn(0x7f637c06b6f0 msgr2=0x7f637c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:23.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.240+0000 7f6381400640 1 -- 172.21.15.27:0/1029330544 shutdown_connections 2024-03-28T18:56:23.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.240+0000 7f6381400640 1 -- 172.21.15.27:0/1029330544 wait complete. 2024-03-28T18:56:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.242+0000 7f6381400640 1 Processor -- start 2024-03-28T18:56:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.242+0000 7f6381400640 1 -- start start 2024-03-28T18:56:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f6381400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c1e24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f6381400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f637c0c58c0 0x7f637c1eec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f6381400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c0c8ed0 0x7f637c1ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f6381400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0b0820 con 0x7f637c0be5d0 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f6381400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c159750 con 0x7f637c0c8ed0 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f6381400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0b09a0 con 0x7f637c0c58c0 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f637be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c1e24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f637be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c1e24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44072/0 (socket says 172.21.15.27:44072) 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f637be00640 1 -- 172.21.15.27:0/70115955 learned_addr learned my addr 172.21.15.27:0/70115955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f6380a00640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c0c8ed0 0x7f637c1ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.243+0000 7f637b400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f637c0c58c0 0x7f637c1eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f637be00640 1 -- 172.21.15.27:0/70115955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f637c0c58c0 msgr2=0x7f637c1eec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f637be00640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f637c0c58c0 0x7f637c1eec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f637be00640 1 -- 172.21.15.27:0/70115955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c0c8ed0 msgr2=0x7f637c1ef1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f637be00640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c0c8ed0 0x7f637c1ef1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f637be00640 1 -- 172.21.15.27:0/70115955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c1ef6e0 con 0x7f637c0be5d0 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f637be00640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c1e24c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f636c00c3c0 tx=0x7f636c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f6373e00640 1 -- 172.21.15.27:0/70115955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c0052d0 con 0x7f637c0be5d0 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f6373e00640 1 -- 172.21.15.27:0/70115955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c005490 con 0x7f637c0be5d0 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c0c18b0 con 0x7f637c0be5d0 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.244+0000 7f6373e00640 1 -- 172.21.15.27:0/70115955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c007260 con 0x7f637c0be5d0 2024-03-28T18:56:23.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.245+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c0c0af0 con 0x7f637c0be5d0 2024-03-28T18:56:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.246+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c069640 con 0x7f637c0be5d0 2024-03-28T18:56:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.247+0000 7f6373e00640 1 -- 172.21.15.27:0/70115955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f636c007400 con 0x7f637c0be5d0 2024-03-28T18:56:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.248+0000 7f6373e00640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f635c06bc00 0x7f635c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.248+0000 7f6373e00640 1 -- 172.21.15.27:0/70115955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f636c08a2c0 con 0x7f637c0be5d0 2024-03-28T18:56:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.248+0000 7f637b400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f635c06bc00 0x7f635c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.249+0000 7f637b400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f635c06bc00 0x7f635c06e0c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f637c1efd20 tx=0x7f6368008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.253+0000 7f6373e00640 1 -- 172.21.15.27:0/70115955 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f636c058a00 con 0x7f637c0be5d0 2024-03-28T18:56:23.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.682+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f637c0b0e70 con 0x7f637c0be5d0 2024-03-28T18:56:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.686+0000 7f6373e00640 1 -- 172.21.15.27:0/70115955 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+20501 (secure 0 0 0) 0x7f636c058300 con 0x7f637c0be5d0 2024-03-28T18:56:23.686 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:23.687 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:56:15.976621+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f635c06bc00 msgr2=0x7f635c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f635c06bc00 0x7f635c06e0c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f637c1efd20 tx=0x7f6368008040 comp rx=0 tx=0).stop 2024-03-28T18:56:23.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 msgr2=0x7f637c1e24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c1e24c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f636c00c3c0 tx=0x7f636c00c880 comp rx=0 tx=0).stop 2024-03-28T18:56:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 shutdown_connections 2024-03-28T18:56:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f635c06bc00 0x7f635c06e0c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c0c8ed0 0x7f637c1ef1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f637c0c58c0 0x7f637c1eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 --2- 172.21.15.27:0/70115955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f637c0be5d0 0x7f637c1e24c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 >> 172.21.15.27:0/70115955 conn(0x7f637c06b6f0 msgr2=0x7f637c0ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:23.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 shutdown_connections 2024-03-28T18:56:23.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.689+0000 7f6381400640 1 -- 172.21.15.27:0/70115955 wait complete. 2024-03-28T18:56:23.708 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-03-28T18:55:32.332283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-03-28T18:55:32.983411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-03-28T18:55:43.415397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-03-28T18:55:44.084162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:56:23.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:56:23.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.930+0000 7fe256000640 1 Processor -- start 2024-03-28T18:56:23.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.931+0000 7fe256000640 1 -- start start 2024-03-28T18:56:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.932+0000 7fe256000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2500c8ff0 0x7fe250156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.932+0000 7fe256000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25006d500 con 0x7fe250152ec0 2024-03-28T18:56:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.932+0000 7fe256000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe250159b50 con 0x7fe2500ad040 2024-03-28T18:56:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.932+0000 7fe256000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe250158cd0 con 0x7fe2500c8ff0 2024-03-28T18:56:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.932+0000 7fe24f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe250152ec0 0x7fe250159440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59038/0 (socket says 172.21.15.27:59038) 2024-03-28T18:56:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.932+0000 7fe24f400640 1 -- 172.21.15.27:0/941662270 learned_addr learned my addr 172.21.15.27:0/941662270 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.933+0000 7fe247e00640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2500c8ff0 0x7fe250156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.933+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768109056 0 0) 0x7fe25006d500 con 0x7fe250152ec0 2024-03-28T18:56:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.933+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe234003520 con 0x7fe250152ec0 2024-03-28T18:56:23.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.933+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 971224517 0 0) 0x7fe234003520 con 0x7fe250152ec0 2024-03-28T18:56:23.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.933+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2500c8ff0 msgr2=0x7fe250156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:23.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.933+0000 7fe24ea00640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2500c8ff0 0x7fe250156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 >> v1:172.21.15.155:6789/0 conn(0x7fe2500ad040 legacy=0x7fe250160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2500c5aa0 con 0x7fe250152ec0 2024-03-28T18:56:23.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe240001670 con 0x7fe250152ec0 2024-03-28T18:56:23.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 >> v1:172.21.15.27:6789/0 conn(0x7fe250152ec0 legacy=0x7fe250159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe234003a50 0x7fe234003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2500c8ff0 0x7fe234004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe234004d90 0x7fe234005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250158cd0 con 0x7fe234003a50 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23400bc40 con 0x7fe234004d90 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23400be40 con 0x7fe2500c8ff0 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24f400640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe234003a50 0x7fe234003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe247e00640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2500c8ff0 0x7fe234004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24f400640 1 -- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2500c8ff0 msgr2=0x7fe234004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24f400640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2500c8ff0 0x7fe234004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.934+0000 7fe24fe00640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe234004d90 0x7fe234005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.935+0000 7fe24f400640 1 -- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe234004d90 msgr2=0x7fe234005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.935+0000 7fe24f400640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe234004d90 0x7fe234005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.935+0000 7fe24f400640 1 -- 172.21.15.27:0/941662270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe234003520 con 0x7fe234003a50 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.935+0000 7fe24fe00640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe234004d90 0x7fe234005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.936+0000 7fe24f400640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe234003a50 0x7fe234003e70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe24000c190 tx=0x7fe234005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f70400ba73919c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.936+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240007c60 con 0x7fe234003a50 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.936+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe240030070 con 0x7fe234003a50 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.936+0000 7fe24ea00640 1 -- 172.21.15.27:0/941662270 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240005d50 con 0x7fe234003a50 2024-03-28T18:56:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.937+0000 7fe256000640 1 -- 172.21.15.27:0/941662270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe234003a50 msgr2=0x7fe234003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.937+0000 7fe256000640 1 --2- 172.21.15.27:0/941662270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe234003a50 0x7fe234003e70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe24000c190 tx=0x7fe234005830 comp rx=0 tx=0).stop 2024-03-28T18:56:23.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.937+0000 7fe24fe00640 1 -- 172.21.15.27:0/941662270 reap_dead start 2024-03-28T18:56:23.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.937+0000 7fe256000640 1 -- 172.21.15.27:0/941662270 shutdown_connections 2024-03-28T18:56:23.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.937+0000 7fe256000640 1 -- 172.21.15.27:0/941662270 >> 172.21.15.27:0/941662270 conn(0x7fe2500671f0 msgr2=0x7fe250067600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:23.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.937+0000 7fe256000640 1 -- 172.21.15.27:0/941662270 shutdown_connections 2024-03-28T18:56:23.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.937+0000 7fe256000640 1 -- 172.21.15.27:0/941662270 wait complete. 2024-03-28T18:56:23.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.939+0000 7fe256000640 1 Processor -- start 2024-03-28T18:56:23.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.939+0000 7fe256000640 1 -- start start 2024-03-28T18:56:23.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe256000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe234003a50 0x7fe250143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe256000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2501eb910 0x7fe2500c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe256000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2500c0b90 0x7fe250142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe256000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2500c6060 con 0x7fe2501eb910 2024-03-28T18:56:23.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe256000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250158fb0 con 0x7fe2500c0b90 2024-03-28T18:56:23.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe256000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25006d8f0 con 0x7fe234003a50 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe247e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2501eb910 0x7fe2500c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe247e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2501eb910 0x7fe2500c0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44102/0 (socket says 172.21.15.27:44102) 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe247e00640 1 -- 172.21.15.27:0/2885749579 learned_addr learned my addr 172.21.15.27:0/2885749579 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe24f400640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe234003a50 0x7fe250143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.940+0000 7fe24fe00640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2500c0b90 0x7fe250142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247e00640 1 -- 172.21.15.27:0/2885749579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe234003a50 msgr2=0x7fe250143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247e00640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe234003a50 0x7fe250143a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247e00640 1 -- 172.21.15.27:0/2885749579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2500c0b90 msgr2=0x7fe250142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247e00640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2500c0b90 0x7fe250142600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247e00640 1 -- 172.21.15.27:0/2885749579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2500c27d0 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247e00640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2501eb910 0x7fe2500c0650 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe2380115d0 tx=0x7fe238011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247400640 1 -- 172.21.15.27:0/2885749579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238015870 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247400640 1 -- 172.21.15.27:0/2885749579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe238015a10 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe247400640 1 -- 172.21.15.27:0/2885749579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238015dc0 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.941+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2500c16e0 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.942+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2500c24f0 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.943+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2500622f0 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.944+0000 7fe247400640 1 -- 172.21.15.27:0/2885749579 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe2380263f0 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.945+0000 7fe247400640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe22c06bc00 0x7fe22c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.945+0000 7fe247400640 1 -- 172.21.15.27:0/2885749579 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fe23808b680 con 0x7fe2501eb910 2024-03-28T18:56:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.945+0000 7fe24f400640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe22c06bc00 0x7fe22c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.946+0000 7fe24f400640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe22c06bc00 0x7fe22c06e0c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe24000c160 tx=0x7fe240009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:23.950+0000 7fe247400640 1 -- 172.21.15.27:0/2885749579 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe238019030 con 0x7fe2501eb910 2024-03-28T18:56:24.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.175+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe25004f470 con 0x7fe2501eb910 2024-03-28T18:56:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.176+0000 7fe247400640 1 -- 172.21.15.27:0/2885749579 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe238059dc0 con 0x7fe2501eb910 2024-03-28T18:56:24.175 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:56:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe22c06bc00 msgr2=0x7fe22c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe22c06bc00 0x7fe22c06e0c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe24000c160 tx=0x7fe240009030 comp rx=0 tx=0).stop 2024-03-28T18:56:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2501eb910 msgr2=0x7fe2500c0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2501eb910 0x7fe2500c0650 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe2380115d0 tx=0x7fe238011a90 comp rx=0 tx=0).stop 2024-03-28T18:56:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 shutdown_connections 2024-03-28T18:56:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe22c06bc00 0x7fe22c06e0c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2500c0b90 0x7fe250142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2501eb910 0x7fe2500c0650 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 --2- 172.21.15.27:0/2885749579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe234003a50 0x7fe250143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 >> 172.21.15.27:0/2885749579 conn(0x7fe2500671f0 msgr2=0x7fe25014b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 shutdown_connections 2024-03-28T18:56:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.178+0000 7fe256000640 1 -- 172.21.15.27:0/2885749579 wait complete. 2024-03-28T18:56:24.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-03-28T18:56:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.421+0000 7f782e800640 1 Processor -- start 2024-03-28T18:56:24.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.422+0000 7f782e800640 1 -- start start 2024-03-28T18:56:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.423+0000 7f782e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78280c5aa0 0x7f7828156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.423+0000 7f782e800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78280735f0 con 0x7f7828152840 2024-03-28T18:56:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.423+0000 7f782e800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78280c7d10 con 0x7f78280c8db0 2024-03-28T18:56:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.423+0000 7f782e800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78281565d0 con 0x7f78280c5aa0 2024-03-28T18:56:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.423+0000 7f782d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78280c5aa0 0x7f7828156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.423+0000 7f782d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78280c5aa0 0x7f7828156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44116/0 (socket says 172.21.15.27:44116) 2024-03-28T18:56:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.423+0000 7f782d400640 1 -- 172.21.15.27:0/1297689056 learned_addr learned my addr 172.21.15.27:0/1297689056 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:24.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.424+0000 7f782d400640 1 -- 172.21.15.27:0/1297689056 >> v1:172.21.15.155:6789/0 conn(0x7f78280c8db0 legacy=0x7f7828072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.424+0000 7f782d400640 1 -- 172.21.15.27:0/1297689056 >> v1:172.21.15.27:6789/0 conn(0x7f7828152840 legacy=0x7f782815ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.424+0000 7f782d400640 1 -- 172.21.15.27:0/1297689056 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78280c7ff0 con 0x7f78280c5aa0 2024-03-28T18:56:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.425+0000 7f782d400640 1 --2- 172.21.15.27:0/1297689056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78280c5aa0 0x7f7828156090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f781c009870 tx=0x7f781c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9866a81c15b553f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.425+0000 7f7827e00640 1 -- 172.21.15.27:0/1297689056 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c035020 con 0x7f78280c5aa0 2024-03-28T18:56:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.425+0000 7f7827e00640 1 -- 172.21.15.27:0/1297689056 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781c031870 con 0x7f78280c5aa0 2024-03-28T18:56:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.425+0000 7f7827e00640 1 -- 172.21.15.27:0/1297689056 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c031c40 con 0x7f78280c5aa0 2024-03-28T18:56:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.426+0000 7f782e800640 1 -- 172.21.15.27:0/1297689056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78280c5aa0 msgr2=0x7f7828156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.426+0000 7f782e800640 1 --2- 172.21.15.27:0/1297689056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78280c5aa0 0x7f7828156090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f781c009870 tx=0x7f781c02b910 comp rx=0 tx=0).stop 2024-03-28T18:56:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.426+0000 7f782e800640 1 -- 172.21.15.27:0/1297689056 shutdown_connections 2024-03-28T18:56:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.426+0000 7f782e800640 1 --2- 172.21.15.27:0/1297689056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78280c5aa0 0x7f7828156090 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.426+0000 7f782e800640 1 -- 172.21.15.27:0/1297689056 >> 172.21.15.27:0/1297689056 conn(0x7f782806b0c0 msgr2=0x7f782806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.426+0000 7f782e800640 1 -- 172.21.15.27:0/1297689056 shutdown_connections 2024-03-28T18:56:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.426+0000 7f782e800640 1 -- 172.21.15.27:0/1297689056 wait complete. 2024-03-28T18:56:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.428+0000 7f782e800640 1 Processor -- start 2024-03-28T18:56:24.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.428+0000 7f782e800640 1 -- start start 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.429+0000 7f782e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78280c5aa0 0x7f7828158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.429+0000 7f782e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78280c8db0 0x7f78280c0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.429+0000 7f782e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828152840 0x7f78280c0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.429+0000 7f782e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7828072100 con 0x7f7828152840 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.429+0000 7f782e800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78280c82d0 con 0x7f78280c8db0 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.429+0000 7f782e800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7828072d30 con 0x7f78280c5aa0 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.429+0000 7f782de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828152840 0x7f78280c0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828152840 0x7f78280c0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44128/0 (socket says 172.21.15.27:44128) 2024-03-28T18:56:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782de00640 1 -- 172.21.15.27:0/3627903059 learned_addr learned my addr 172.21.15.27:0/3627903059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782ca00640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78280c8db0 0x7f78280c0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782d400640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78280c5aa0 0x7f7828158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782de00640 1 -- 172.21.15.27:0/3627903059 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78280c5aa0 msgr2=0x7f7828158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782de00640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78280c5aa0 0x7f7828158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782de00640 1 -- 172.21.15.27:0/3627903059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78280c8db0 msgr2=0x7f78280c0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782de00640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78280c8db0 0x7f78280c0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782de00640 1 -- 172.21.15.27:0/3627903059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78281eb620 con 0x7f7828152840 2024-03-28T18:56:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.430+0000 7f782de00640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828152840 0x7f78280c0fb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f782000c6e0 tx=0x7f782000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.431+0000 7f7825600640 1 -- 172.21.15.27:0/3627903059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820003b70 con 0x7f7828152840 2024-03-28T18:56:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.431+0000 7f7825600640 1 -- 172.21.15.27:0/3627903059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7820003d30 con 0x7f7828152840 2024-03-28T18:56:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.431+0000 7f7825600640 1 -- 172.21.15.27:0/3627903059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820005310 con 0x7f7828152840 2024-03-28T18:56:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.431+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78281eb900 con 0x7f7828152840 2024-03-28T18:56:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.431+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7828159320 con 0x7f7828152840 2024-03-28T18:56:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.432+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782804f400 con 0x7f7828152840 2024-03-28T18:56:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.433+0000 7f7825600640 1 -- 172.21.15.27:0/3627903059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f78200054b0 con 0x7f7828152840 2024-03-28T18:56:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.434+0000 7f7825600640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780806bbd0 0x7f780806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.434+0000 7f7825600640 1 -- 172.21.15.27:0/3627903059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f782008aca0 con 0x7f7828152840 2024-03-28T18:56:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.434+0000 7f782d400640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780806bbd0 0x7f780806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.435+0000 7f782d400640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780806bbd0 0x7f780806e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f781c036000 tx=0x7f781c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.439+0000 7f7825600640 1 -- 172.21.15.27:0/3627903059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f78200593e0 con 0x7f7828152840 2024-03-28T18:56:24.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:56:24.651 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:56:24.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.668+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78281513b0 con 0x7f7828152840 2024-03-28T18:56:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.668+0000 7f7825600640 1 -- 172.21.15.27:0/3627903059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v26) v1 ==== 99+0+11 (secure 0 0 0) 0x7f782008a0a0 con 0x7f7828152840 2024-03-28T18:56:24.668 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780806bbd0 msgr2=0x7f780806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780806bbd0 0x7f780806e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f781c036000 tx=0x7f781c03c040 comp rx=0 tx=0).stop 2024-03-28T18:56:24.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828152840 msgr2=0x7f78280c0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828152840 0x7f78280c0fb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f782000c6e0 tx=0x7f782000cba0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 shutdown_connections 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780806bbd0 0x7f780806e090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7828152840 0x7f78280c0fb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78280c8db0 0x7f78280c0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 --2- 172.21.15.27:0/3627903059 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78280c5aa0 0x7f7828158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.672+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 >> 172.21.15.27:0/3627903059 conn(0x7f782806b0c0 msgr2=0x7f7828072620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.673+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 shutdown_connections 2024-03-28T18:56:24.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.673+0000 7f782e800640 1 -- 172.21.15.27:0/3627903059 wait complete. 2024-03-28T18:56:24.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-03-28T18:56:24.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.951+0000 7f8fc2400640 1 Processor -- start 2024-03-28T18:56:24.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.951+0000 7f8fc2400640 1 -- start start 2024-03-28T18:56:24.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc2400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc2400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fbc152840 0x7f8fbc152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc2400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc0735f0 con 0x7f8fbc0c5aa0 2024-03-28T18:56:24.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc2400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc1565d0 con 0x7f8fbc152840 2024-03-28T18:56:24.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc2400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0c82d0 con 0x7f8fbc0c8db0 2024-03-28T18:56:24.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc1a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fbc152840 0x7f8fbc152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc1a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fbc152840 0x7f8fbc152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44134/0 (socket says 172.21.15.27:44134) 2024-03-28T18:56:24.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc1a00640 1 -- 172.21.15.27:0/1749693667 learned_addr learned my addr 172.21.15.27:0/1749693667 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:24.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fc1a00640 1 -- 172.21.15.27:0/1749693667 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 msgr2=0x7f8fbc156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:24.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.953+0000 7f8fbbe00640 1 --2- 172.21.15.27:0/1749693667 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.954+0000 7f8fc1a00640 1 --2- 172.21.15.27:0/1749693667 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.954+0000 7f8fc1a00640 1 -- 172.21.15.27:0/1749693667 >> v1:172.21.15.155:6789/0 conn(0x7f8fbc0c5aa0 legacy=0x7f8fbc072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:24.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.954+0000 7f8fc1a00640 1 -- 172.21.15.27:0/1749693667 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc0c7d10 con 0x7f8fbc152840 2024-03-28T18:56:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.954+0000 7f8fbbe00640 1 --2- 172.21.15.27:0/1749693667 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:24.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.955+0000 7f8fc1a00640 1 --2- 172.21.15.27:0/1749693667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc152cd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0c76a0 tx=0x7f8fb402be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c4b4ca71608f27 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.955+0000 7f8fbb400640 1 -- 172.21.15.27:0/1749693667 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb402c9a0 con 0x7f8fbc152840 2024-03-28T18:56:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.955+0000 7f8fbb400640 1 -- 172.21.15.27:0/1749693667 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb402cb40 con 0x7f8fbc152840 2024-03-28T18:56:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.955+0000 7f8fbb400640 1 -- 172.21.15.27:0/1749693667 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40343f0 con 0x7f8fbc152840 2024-03-28T18:56:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.956+0000 7f8fc2400640 1 -- 172.21.15.27:0/1749693667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 msgr2=0x7f8fbc152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.956+0000 7f8fc2400640 1 --2- 172.21.15.27:0/1749693667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc152cd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0c76a0 tx=0x7f8fb402be00 comp rx=0 tx=0).stop 2024-03-28T18:56:24.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.956+0000 7f8fc2400640 1 -- 172.21.15.27:0/1749693667 shutdown_connections 2024-03-28T18:56:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.956+0000 7f8fc2400640 1 --2- 172.21.15.27:0/1749693667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc152cd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.956+0000 7f8fc2400640 1 --2- 172.21.15.27:0/1749693667 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.956+0000 7f8fc2400640 1 -- 172.21.15.27:0/1749693667 >> 172.21.15.27:0/1749693667 conn(0x7f8fbc06b0c0 msgr2=0x7f8fbc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.956+0000 7f8fc2400640 1 -- 172.21.15.27:0/1749693667 shutdown_connections 2024-03-28T18:56:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.956+0000 7f8fc2400640 1 -- 172.21.15.27:0/1749693667 wait complete. 2024-03-28T18:56:24.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.958+0000 7f8fc2400640 1 Processor -- start 2024-03-28T18:56:24.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.958+0000 7f8fc2400640 1 -- start start 2024-03-28T18:56:24.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc2400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fbc0c5aa0 0x7f8fbc0c1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc2400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc0c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc2400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc2400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc072100 con 0x7f8fbc152840 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc2400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc071e20 con 0x7f8fbc0c5aa0 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc2400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc072d30 con 0x7f8fbc0c8db0 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44136/0 (socket says 172.21.15.27:44136) 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc1a00640 1 -- 172.21.15.27:0/896938786 learned_addr learned my addr 172.21.15.27:0/896938786 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc1000640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fbc0c5aa0 0x7f8fbc0c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fbbe00640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc0c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.959+0000 7f8fc1a00640 1 -- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 msgr2=0x7f8fbc0c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fc1a00640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc0c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fc1a00640 1 -- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fbc0c5aa0 msgr2=0x7f8fbc0c1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:24.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fc1a00640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fbc0c5aa0 0x7f8fbc0c1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:24.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fc1a00640 1 -- 172.21.15.27:0/896938786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc1eac60 con 0x7f8fbc152840 2024-03-28T18:56:24.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fc1000640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fbc0c5aa0 0x7f8fbc0c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:24.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fbbe00640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:24.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fc1a00640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc14e1f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009f70 tx=0x7f8fb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fb8c00640 1 -- 172.21.15.27:0/896938786 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4043020 con 0x7f8fbc152840 2024-03-28T18:56:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fb8c00640 1 -- 172.21.15.27:0/896938786 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb403e450 con 0x7f8fbc152840 2024-03-28T18:56:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fb8c00640 1 -- 172.21.15.27:0/896938786 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4009510 con 0x7f8fbc152840 2024-03-28T18:56:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc1eaf40 con 0x7f8fbc152840 2024-03-28T18:56:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.960+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc1eb500 con 0x7f8fbc152840 2024-03-28T18:56:24.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.962+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fbc04f400 con 0x7f8fbc152840 2024-03-28T18:56:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.963+0000 7f8fb8c00640 1 -- 172.21.15.27:0/896938786 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8fb4046020 con 0x7f8fbc152840 2024-03-28T18:56:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.964+0000 7f8fb8c00640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8f9c06bbd0 0x7f8f9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.964+0000 7f8fb8c00640 1 -- 172.21.15.27:0/896938786 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f8fb40ad720 con 0x7f8fbc152840 2024-03-28T18:56:24.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.964+0000 7f8fc1000640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8f9c06bbd0 0x7f8f9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:24.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.965+0000 7f8fc1000640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8f9c06bbd0 0x7f8f9c06e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0003eb0 tx=0x7f8fb0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:24.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:24.969+0000 7f8fb8c00640 1 -- 172.21.15.27:0/896938786 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8fb404d030 con 0x7f8fbc152840 2024-03-28T18:56:25.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.184+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f8fbc0c51d0 con 0x7f8fbc152840 2024-03-28T18:56:25.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.185+0000 7f8fb8c00640 1 -- 172.21.15.27:0/896938786 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v26) v1 ==== 95+0+11 (secure 0 0 0) 0x7f8fb402c9b0 con 0x7f8fbc152840 2024-03-28T18:56:25.185 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.187+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8f9c06bbd0 msgr2=0x7f8f9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.187+0000 7f8fc2400640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8f9c06bbd0 0x7f8f9c06e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0003eb0 tx=0x7f8fb0009040 comp rx=0 tx=0).stop 2024-03-28T18:56:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.187+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 msgr2=0x7f8fbc14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.187+0000 7f8fc2400640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc14e1f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009f70 tx=0x7f8fb4004a40 comp rx=0 tx=0).stop 2024-03-28T18:56:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.188+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 shutdown_connections 2024-03-28T18:56:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.188+0000 7f8fc2400640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8f9c06bbd0 0x7f8f9c06e090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.188+0000 7f8fc2400640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fbc152840 0x7f8fbc14e1f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.188+0000 7f8fc2400640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fbc0c8db0 0x7f8fbc0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.188+0000 7f8fc2400640 1 --2- 172.21.15.27:0/896938786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fbc0c5aa0 0x7f8fbc0c1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.188+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 >> 172.21.15.27:0/896938786 conn(0x7f8fbc06b0c0 msgr2=0x7f8fbc06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:25.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.188+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 shutdown_connections 2024-03-28T18:56:25.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.188+0000 7f8fc2400640 1 -- 172.21.15.27:0/896938786 wait complete. 2024-03-28T18:56:25.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-03-28T18:56:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.426+0000 7f42bd600640 1 Processor -- start 2024-03-28T18:56:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.426+0000 7f42bd600640 1 -- start start 2024-03-28T18:56:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.428+0000 7f42bd600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b80c1500 0x7f42b80c18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.428+0000 7f42bd600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f42b8072bd0 0x7f42b8073060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.428+0000 7f42bd600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b8157ef0 con 0x7f42b8157a70 2024-03-28T18:56:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.428+0000 7f42bd600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b80c6960 con 0x7f42b8072bd0 2024-03-28T18:56:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.428+0000 7f42bd600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80c4f10 con 0x7f42b80c1500 2024-03-28T18:56:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.429+0000 7f42b7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b80c1500 0x7f42b80c18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.429+0000 7f42bcc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f42b8072bd0 0x7f42b8073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.429+0000 7f42bcc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f42b8072bd0 0x7f42b8073060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:46880/0 (socket says 172.21.15.27:46880) 2024-03-28T18:56:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.429+0000 7f42bcc00640 1 -- 172.21.15.27:0/4267805246 learned_addr learned my addr 172.21.15.27:0/4267805246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:25.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.430+0000 7f42b7e00640 1 -- 172.21.15.27:0/4267805246 >> v2:172.21.15.155:3300/0 conn(0x7f42b8072bd0 msgr2=0x7f42b8073060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:25.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.430+0000 7f42b7e00640 1 --2- 172.21.15.27:0/4267805246 >> v2:172.21.15.155:3300/0 conn(0x7f42b8072bd0 0x7f42b8073060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.430+0000 7f42b7e00640 1 -- 172.21.15.27:0/4267805246 >> v1:172.21.15.155:6789/0 conn(0x7f42b8157a70 legacy=0x7f42b80c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.430+0000 7f42b7e00640 1 -- 172.21.15.27:0/4267805246 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b815a5b0 con 0x7f42b80c1500 2024-03-28T18:56:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.431+0000 7f42b7e00640 1 --2- 172.21.15.27:0/4267805246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b80c1500 0x7f42b80c18e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f42a80097a0 tx=0x7f42a802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47959ee1c18ea1b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.432+0000 7f42b7400640 1 -- 172.21.15.27:0/4267805246 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8031780 con 0x7f42b80c1500 2024-03-28T18:56:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.432+0000 7f42b7400640 1 -- 172.21.15.27:0/4267805246 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a8037070 con 0x7f42b80c1500 2024-03-28T18:56:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.432+0000 7f42b7400640 1 -- 172.21.15.27:0/4267805246 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8031b40 con 0x7f42b80c1500 2024-03-28T18:56:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.433+0000 7f42bd600640 1 -- 172.21.15.27:0/4267805246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b80c1500 msgr2=0x7f42b80c18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.433+0000 7f42bd600640 1 --2- 172.21.15.27:0/4267805246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b80c1500 0x7f42b80c18e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f42a80097a0 tx=0x7f42a802b820 comp rx=0 tx=0).stop 2024-03-28T18:56:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.433+0000 7f42bd600640 1 -- 172.21.15.27:0/4267805246 shutdown_connections 2024-03-28T18:56:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.433+0000 7f42bd600640 1 --2- 172.21.15.27:0/4267805246 >> v2:172.21.15.155:3300/0 conn(0x7f42b8072bd0 0x7f42b8073060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.433+0000 7f42bd600640 1 --2- 172.21.15.27:0/4267805246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b80c1500 0x7f42b80c18e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.433+0000 7f42bd600640 1 -- 172.21.15.27:0/4267805246 >> 172.21.15.27:0/4267805246 conn(0x7f42b80a88b0 msgr2=0x7f42b80a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.433+0000 7f42bd600640 1 -- 172.21.15.27:0/4267805246 shutdown_connections 2024-03-28T18:56:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.433+0000 7f42bd600640 1 -- 172.21.15.27:0/4267805246 wait complete. 2024-03-28T18:56:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.435+0000 7f42bd600640 1 Processor -- start 2024-03-28T18:56:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.435+0000 7f42bd600640 1 -- start start 2024-03-28T18:56:25.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42bd600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b8072bd0 0x7f42b8155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42bd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80c1500 0x7f42b81f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42bd600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b8157a70 0x7f42b814e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42bd600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80c5440 con 0x7f42b80c1500 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42bd600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80c5160 con 0x7f42b8157a70 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42bd600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80c4ae0 con 0x7f42b8072bd0 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80c1500 0x7f42b81f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80c1500 0x7f42b81f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44148/0 (socket says 172.21.15.27:44148) 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42afe00640 1 -- 172.21.15.27:0/3608300361 learned_addr learned my addr 172.21.15.27:0/3608300361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42bcc00640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b8157a70 0x7f42b814e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.436+0000 7f42b7e00640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b8072bd0 0x7f42b8155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42afe00640 1 -- 172.21.15.27:0/3608300361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b8072bd0 msgr2=0x7f42b8155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42afe00640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b8072bd0 0x7f42b8155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42afe00640 1 -- 172.21.15.27:0/3608300361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b8157a70 msgr2=0x7f42b814e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42afe00640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b8157a70 0x7f42b814e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42afe00640 1 -- 172.21.15.27:0/3608300361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b81f3740 con 0x7f42b80c1500 2024-03-28T18:56:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42afe00640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80c1500 0x7f42b81f2c30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f42a000c3c0 tx=0x7f42a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42b4c00640 1 -- 172.21.15.27:0/3608300361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00052d0 con 0x7f42b80c1500 2024-03-28T18:56:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42b4c00640 1 -- 172.21.15.27:0/3608300361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0005490 con 0x7f42b80c1500 2024-03-28T18:56:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42b4c00640 1 -- 172.21.15.27:0/3608300361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0007260 con 0x7f42b80c1500 2024-03-28T18:56:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.437+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b8153d00 con 0x7f42b80c1500 2024-03-28T18:56:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.438+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b8154cb0 con 0x7f42b80c1500 2024-03-28T18:56:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.439+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b804f400 con 0x7f42b80c1500 2024-03-28T18:56:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.440+0000 7f42b4c00640 1 -- 172.21.15.27:0/3608300361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f42a0007400 con 0x7f42b80c1500 2024-03-28T18:56:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.441+0000 7f42b4c00640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f429406bc00 0x7f429406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.441+0000 7f42b4c00640 1 -- 172.21.15.27:0/3608300361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f42a008a1d0 con 0x7f42b80c1500 2024-03-28T18:56:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.441+0000 7f42b7e00640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f429406bc00 0x7f429406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.442+0000 7f42b7e00640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f429406bc00 0x7f429406e0c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f42a8036000 tx=0x7f42a803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.445+0000 7f42b4c00640 1 -- 172.21.15.27:0/3608300361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f42b8157ef0 con 0x7f42b80c1500 2024-03-28T18:56:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.656+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42b80a9620 con 0x7f42b80c1500 2024-03-28T18:56:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.657+0000 7f42b4c00640 1 -- 172.21.15.27:0/3608300361 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v26) v1 ==== 99+0+11 (secure 0 0 0) 0x7f42a00589b0 con 0x7f42b80c1500 2024-03-28T18:56:25.657 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f429406bc00 msgr2=0x7f429406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f429406bc00 0x7f429406e0c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f42a8036000 tx=0x7f42a803d040 comp rx=0 tx=0).stop 2024-03-28T18:56:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80c1500 msgr2=0x7f42b81f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80c1500 0x7f42b81f2c30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f42a000c3c0 tx=0x7f42a000c880 comp rx=0 tx=0).stop 2024-03-28T18:56:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 shutdown_connections 2024-03-28T18:56:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f429406bc00 0x7f429406e0c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42b8157a70 0x7f42b814e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80c1500 0x7f42b81f2c30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 --2- 172.21.15.27:0/3608300361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42b8072bd0 0x7f42b8155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 >> 172.21.15.27:0/3608300361 conn(0x7f42b80a88b0 msgr2=0x7f42b81580d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 shutdown_connections 2024-03-28T18:56:25.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.660+0000 7f42bd600640 1 -- 172.21.15.27:0/3608300361 wait complete. 2024-03-28T18:56:25.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-03-28T18:56:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.905+0000 7f33c3e00640 1 Processor -- start 2024-03-28T18:56:25.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.906+0000 7f33c3e00640 1 -- start start 2024-03-28T18:56:25.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.907+0000 7f33c3e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f33bc06e0b0 0x7f33bc0c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.907+0000 7f33c3e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33bc0c33b0 con 0x7f33bc0c22e0 2024-03-28T18:56:25.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.907+0000 7f33c3e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33bc0c1970 con 0x7f33bc150c50 2024-03-28T18:56:25.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.907+0000 7f33c3e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33bc06ec10 con 0x7f33bc06e0b0 2024-03-28T18:56:25.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.908+0000 7f33c0a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f33bc0c22e0 0x7f33bc0c2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59050/0 (socket says 172.21.15.27:59050) 2024-03-28T18:56:25.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.908+0000 7f33c0a00640 1 -- 172.21.15.27:0/1620080579 learned_addr learned my addr 172.21.15.27:0/1620080579 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:25.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.908+0000 7f33c1e00640 1 --2- 172.21.15.27:0/1620080579 >> v2:172.21.15.155:3300/0 conn(0x7f33bc06e0b0 0x7f33bc0c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.908+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994571923 0 0) 0x7f33bc0c33b0 con 0x7f33bc0c22e0 2024-03-28T18:56:25.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a0003520 con 0x7f33bc0c22e0 2024-03-28T18:56:25.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686398969 0 0) 0x7f33bc0c1970 con 0x7f33bc150c50 2024-03-28T18:56:25.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33bc0c33b0 con 0x7f33bc150c50 2024-03-28T18:56:25.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1828563213 0 0) 0x7f33a0003520 con 0x7f33bc0c22e0 2024-03-28T18:56:25.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 >> v2:172.21.15.155:3300/0 conn(0x7f33bc06e0b0 msgr2=0x7f33bc0c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:25.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 --2- 172.21.15.27:0/1620080579 >> v2:172.21.15.155:3300/0 conn(0x7f33bc06e0b0 0x7f33bc0c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 >> v1:172.21.15.155:6789/0 conn(0x7f33bc150c50 legacy=0x7f33bc156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc06ee60 con 0x7f33bc0c22e0 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f33a8001670 con 0x7f33bc0c22e0 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 >> v1:172.21.15.27:6789/0 conn(0x7f33bc0c22e0 legacy=0x7f33bc0c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33a0003c00 0x7f33a0004000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33a00051c0 0x7f33a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc06ec10 con 0x7f33a0004800 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a000c160 con 0x7f33a00051c0 2024-03-28T18:56:25.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a000c360 con 0x7f33a0003c00 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.909+0000 7f33c0a00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c1400640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33a0003c00 0x7f33a0004000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c1e00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33a00051c0 0x7f33a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c0a00640 1 -- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33a0003c00 msgr2=0x7f33a0004000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c0a00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33a0003c00 0x7f33a0004000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c0a00640 1 -- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33a00051c0 msgr2=0x7f33a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c0a00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33a00051c0 0x7f33a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c0a00640 1 -- 172.21.15.27:0/1620080579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a0003520 con 0x7f33a0004800 2024-03-28T18:56:25.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c1e00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33a00051c0 0x7f33a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.910+0000 7f33c1e00640 1 -- 172.21.15.27:0/1620080579 reap_dead start 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.911+0000 7f33c0a00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33a0004c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f33a8007fb0 tx=0x7f33a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2173fb4a05fb34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.911+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8007c00 con 0x7f33a0004800 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.911+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a8030070 con 0x7f33a0004800 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.911+0000 7f33bbe00640 1 -- 172.21.15.27:0/1620080579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8005d10 con 0x7f33a0004800 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.912+0000 7f33c3e00640 1 -- 172.21.15.27:0/1620080579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 msgr2=0x7f33a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.912+0000 7f33c3e00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33a0004c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f33a8007fb0 tx=0x7f33a0005830 comp rx=0 tx=0).stop 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.912+0000 7f33c3e00640 1 -- 172.21.15.27:0/1620080579 shutdown_connections 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.912+0000 7f33c3e00640 1 --2- 172.21.15.27:0/1620080579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33a0004c80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.912+0000 7f33c3e00640 1 -- 172.21.15.27:0/1620080579 >> 172.21.15.27:0/1620080579 conn(0x7f33bc067400 msgr2=0x7f33bc0c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:25.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.912+0000 7f33c3e00640 1 -- 172.21.15.27:0/1620080579 shutdown_connections 2024-03-28T18:56:25.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.912+0000 7f33c3e00640 1 -- 172.21.15.27:0/1620080579 wait complete. 2024-03-28T18:56:25.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.914+0000 7f33c3e00640 1 Processor -- start 2024-03-28T18:56:25.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.914+0000 7f33c3e00640 1 -- start start 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.915+0000 7f33c3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33bc1ed040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.915+0000 7f33c3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33bc154390 0x7f33bc148fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.915+0000 7f33c3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33bc149500 0x7f33bc153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.915+0000 7f33c3e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc151030 con 0x7f33a0004800 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.915+0000 7f33c3e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0c1e50 con 0x7f33bc154390 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.915+0000 7f33c3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0c1640 con 0x7f33bc149500 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.915+0000 7f33c1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33bc1ed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33bc1ed040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44162/0 (socket says 172.21.15.27:44162) 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1400640 1 -- 172.21.15.27:0/641531697 learned_addr learned my addr 172.21.15.27:0/641531697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c0a00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33bc154390 0x7f33bc148fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1e00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33bc149500 0x7f33bc153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1400640 1 -- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33bc149500 msgr2=0x7f33bc153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1400640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33bc149500 0x7f33bc153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1400640 1 -- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33bc154390 msgr2=0x7f33bc148fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1400640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33bc154390 0x7f33bc148fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1400640 1 -- 172.21.15.27:0/641531697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc146a40 con 0x7f33a0004800 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1e00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33bc149500 0x7f33bc153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.916+0000 7f33c1400640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33bc1ed040 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00eb20 tx=0x7f33ac0020d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.917+0000 7f33b9600640 1 -- 172.21.15.27:0/641531697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac003b80 con 0x7f33a0004800 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.917+0000 7f33b9600640 1 -- 172.21.15.27:0/641531697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac018070 con 0x7f33a0004800 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.917+0000 7f33b9600640 1 -- 172.21.15.27:0/641531697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac00c780 con 0x7f33a0004800 2024-03-28T18:56:25.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.917+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc154be0 con 0x7f33a0004800 2024-03-28T18:56:25.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.917+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33bc1ed580 con 0x7f33a0004800 2024-03-28T18:56:25.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.918+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33bc1549d0 con 0x7f33a0004800 2024-03-28T18:56:25.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.920+0000 7f33b9600640 1 -- 172.21.15.27:0/641531697 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f33ac00c920 con 0x7f33a0004800 2024-03-28T18:56:25.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.922+0000 7f33b9600640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f339c06bc00 0x7f339c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:25.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.922+0000 7f33b9600640 1 -- 172.21.15.27:0/641531697 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f33ac08afd0 con 0x7f33a0004800 2024-03-28T18:56:25.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.922+0000 7f33c0a00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f339c06bc00 0x7f339c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:25.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.923+0000 7f33c0a00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f339c06bc00 0x7f339c06e0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33a8034000 tx=0x7f33a8039040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:25.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:25.927+0000 7f33b9600640 1 -- 172.21.15.27:0/641531697 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f33ac01a310 con 0x7f33a0004800 2024-03-28T18:56:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.150+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f33bc04f470 con 0x7f33a0004800 2024-03-28T18:56:26.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.151+0000 7f33b9600640 1 -- 172.21.15.27:0/641531697 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v26) v1 ==== 95+0+11 (secure 0 0 0) 0x7f33ac059680 con 0x7f33a0004800 2024-03-28T18:56:26.151 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f339c06bc00 msgr2=0x7f339c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f339c06bc00 0x7f339c06e0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33a8034000 tx=0x7f33a8039040 comp rx=0 tx=0).stop 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 msgr2=0x7f33bc1ed040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33bc1ed040 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f33ac00eb20 tx=0x7f33ac0020d0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 shutdown_connections 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f339c06bc00 0x7f339c06e0c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f33bc149500 0x7f33bc153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33bc154390 0x7f33bc148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 --2- 172.21.15.27:0/641531697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33a0004800 0x7f33bc1ed040 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.153+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 >> 172.21.15.27:0/641531697 conn(0x7f33bc067400 msgr2=0x7f33bc0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.154+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 shutdown_connections 2024-03-28T18:56:26.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.154+0000 7f33c3e00640 1 -- 172.21.15.27:0/641531697 wait complete. 2024-03-28T18:56:26.174 INFO:tasks.mds_thrash:mds thrasher using random seed: 1711652186 2024-03-28T18:56:26.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.396+0000 7fdba9200640 1 Processor -- start 2024-03-28T18:56:26.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.396+0000 7fdba9200640 1 -- start start 2024-03-28T18:56:26.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.398+0000 7fdba9200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdba40c8470 0x7fdba40ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.399+0000 7fdba9200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40ad680 con 0x7fdba40c8c70 2024-03-28T18:56:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.399+0000 7fdba9200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40c4de0 con 0x7fdba4150b60 2024-03-28T18:56:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.399+0000 7fdba9200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba4157210 con 0x7fdba40c8470 2024-03-28T18:56:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.399+0000 7fdba2800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdba40c8c70 0x7fdba415de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44746/0 (socket says 172.21.15.27:44746) 2024-03-28T18:56:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.399+0000 7fdba2800640 1 -- 172.21.15.27:0/1745931108 learned_addr learned my addr 172.21.15.27:0/1745931108 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:26.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.399+0000 7fdba1400640 1 --2- 172.21.15.27:0/1745931108 >> v2:172.21.15.155:3300/0 conn(0x7fdba40c8470 0x7fdba40ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.400+0000 7fdba0a00640 1 -- 172.21.15.27:0/1745931108 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411071511 0 0) 0x7fdba40ad680 con 0x7fdba40c8c70 2024-03-28T18:56:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.400+0000 7fdba0a00640 1 -- 172.21.15.27:0/1745931108 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb88003520 con 0x7fdba40c8c70 2024-03-28T18:56:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.400+0000 7fdba1400640 1 -- 172.21.15.27:0/1745931108 >> v1:172.21.15.155:6789/0 conn(0x7fdba4150b60 legacy=0x7fdba4156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.400+0000 7fdba0a00640 1 -- 172.21.15.27:0/1745931108 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107936358 0 0) 0x7fdb88003520 con 0x7fdba40c8c70 2024-03-28T18:56:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.400+0000 7fdba1400640 1 -- 172.21.15.27:0/1745931108 >> v1:172.21.15.27:6789/0 conn(0x7fdba40c8c70 legacy=0x7fdba415de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.400+0000 7fdba1400640 1 -- 172.21.15.27:0/1745931108 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40c50c0 con 0x7fdba40c8470 2024-03-28T18:56:26.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.401+0000 7fdba1400640 1 --2- 172.21.15.27:0/1745931108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8470 0x7fdba40ade40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb900097d0 tx=0x7fdb9002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=468c4a022e6db98a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.402+0000 7fdba0a00640 1 -- 172.21.15.27:0/1745931108 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb90037070 con 0x7fdba40c8470 2024-03-28T18:56:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.402+0000 7fdba0a00640 1 -- 172.21.15.27:0/1745931108 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb90030510 con 0x7fdba40c8470 2024-03-28T18:56:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.402+0000 7fdba0a00640 1 -- 172.21.15.27:0/1745931108 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c0019f0 con 0x7fdba40c8470 2024-03-28T18:56:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.402+0000 7fdba9200640 1 -- 172.21.15.27:0/1745931108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8470 msgr2=0x7fdba40ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.402+0000 7fdba9200640 1 --2- 172.21.15.27:0/1745931108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8470 0x7fdba40ade40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdb900097d0 tx=0x7fdb9002b790 comp rx=0 tx=0).stop 2024-03-28T18:56:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.403+0000 7fdba9200640 1 -- 172.21.15.27:0/1745931108 shutdown_connections 2024-03-28T18:56:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.403+0000 7fdba9200640 1 --2- 172.21.15.27:0/1745931108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8470 0x7fdba40ade40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.403+0000 7fdba9200640 1 -- 172.21.15.27:0/1745931108 >> 172.21.15.27:0/1745931108 conn(0x7fdba40aaab0 msgr2=0x7fdba40aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.403+0000 7fdba9200640 1 -- 172.21.15.27:0/1745931108 shutdown_connections 2024-03-28T18:56:26.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.403+0000 7fdba9200640 1 -- 172.21.15.27:0/1745931108 wait complete. 2024-03-28T18:56:26.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.404+0000 7fdba9200640 1 Processor -- start 2024-03-28T18:56:26.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.404+0000 7fdba9200640 1 -- start start 2024-03-28T18:56:26.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.405+0000 7fdba9200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdba40c8470 0x7fdba40c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.405+0000 7fdba9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8c70 0x7fdba41eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.405+0000 7fdba9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdba4150b60 0x7fdba41eb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.405+0000 7fdba9200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4150f40 con 0x7fdba4150b60 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.405+0000 7fdba9200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40ad9b0 con 0x7fdba40c8c70 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.405+0000 7fdba9200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40a85a0 con 0x7fdba40c8470 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdba4150b60 0x7fdba41eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdba4150b60 0x7fdba41eb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40308/0 (socket says 172.21.15.27:40308) 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba2800640 1 -- 172.21.15.27:0/2317162371 learned_addr learned my addr 172.21.15.27:0/2317162371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba1e00640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdba40c8470 0x7fdba40c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba1400640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8c70 0x7fdba41eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba2800640 1 -- 172.21.15.27:0/2317162371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdba40c8470 msgr2=0x7fdba40c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba2800640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdba40c8470 0x7fdba40c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba2800640 1 -- 172.21.15.27:0/2317162371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8c70 msgr2=0x7fdba41eaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba2800640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8c70 0x7fdba41eaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.406+0000 7fdba2800640 1 -- 172.21.15.27:0/2317162371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4151a60 con 0x7fdba4150b60 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.407+0000 7fdba2800640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdba4150b60 0x7fdba41eb460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c001560 tx=0x7fdb9c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.407+0000 7fdb9a000640 1 -- 172.21.15.27:0/2317162371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c017070 con 0x7fdba4150b60 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.407+0000 7fdb9a000640 1 -- 172.21.15.27:0/2317162371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c00ad60 con 0x7fdba4150b60 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.407+0000 7fdb9a000640 1 -- 172.21.15.27:0/2317162371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c012640 con 0x7fdba4150b60 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.407+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba41ef0f0 con 0x7fdba4150b60 2024-03-28T18:56:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.407+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba414dc30 con 0x7fdba4150b60 2024-03-28T18:56:26.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.409+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba404f400 con 0x7fdba4150b60 2024-03-28T18:56:26.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.409+0000 7fdb9a000640 1 -- 172.21.15.27:0/2317162371 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdb9c01a070 con 0x7fdba4150b60 2024-03-28T18:56:26.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.410+0000 7fdb9a000640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdb8406bbd0 0x7fdb8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.410+0000 7fdb9a000640 1 -- 172.21.15.27:0/2317162371 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fdb9c08af20 con 0x7fdba4150b60 2024-03-28T18:56:26.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.411+0000 7fdba1e00640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdb8406bbd0 0x7fdb8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.411+0000 7fdba1e00640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdb8406bbd0 0x7fdb8406e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdba40ab4f0 tx=0x7fdb94009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:26.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.415+0000 7fdb9a000640 1 -- 172.21.15.27:0/2317162371 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdb9c058f90 con 0x7fdba4150b60 2024-03-28T18:56:26.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.661+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdba40c4a20 con 0x7fdba4150b60 2024-03-28T18:56:26.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.664+0000 7fdb9a000640 1 -- 172.21.15.27:0/2317162371 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+20501 (secure 0 0 0) 0x7fdb9c05d030 con 0x7fdba4150b60 2024-03-28T18:56:26.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:26.664 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:56:15.976621+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:26.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdb8406bbd0 msgr2=0x7fdb8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdb8406bbd0 0x7fdb8406e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdba40ab4f0 tx=0x7fdb94009040 comp rx=0 tx=0).stop 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdba4150b60 msgr2=0x7fdba41eb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdba4150b60 0x7fdba41eb460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c001560 tx=0x7fdb9c00a430 comp rx=0 tx=0).stop 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 shutdown_connections 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdb8406bbd0 0x7fdb8406e090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdba4150b60 0x7fdba41eb460 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdba40c8c70 0x7fdba41eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 --2- 172.21.15.27:0/2317162371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdba40c8470 0x7fdba40c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 >> 172.21.15.27:0/2317162371 conn(0x7fdba40aaab0 msgr2=0x7fdba40a93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 shutdown_connections 2024-03-28T18:56:26.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.667+0000 7fdba9200640 1 -- 172.21.15.27:0/2317162371 wait complete. 2024-03-28T18:56:26.694 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-03-28T18:55:32.332283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-03-28T18:55:32.983411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-03-28T18:55:43.415397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-03-28T18:55:44.084162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:56:26.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:56:26.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.917+0000 7fcc06200640 1 Processor -- start 2024-03-28T18:56:26.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.917+0000 7fcc06200640 1 -- start start 2024-03-28T18:56:26.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.919+0000 7fcc06200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc000c5aa0 0x7fcc00156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.919+0000 7fcc06200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcc00152840 0x7fcc00152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.919+0000 7fcc06200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc000735f0 con 0x7fcc000c8db0 2024-03-28T18:56:26.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.919+0000 7fcc06200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc001565d0 con 0x7fcc00152840 2024-03-28T18:56:26.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.919+0000 7fcc06200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000c82d0 con 0x7fcc000c5aa0 2024-03-28T18:56:26.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.920+0000 7fcc04e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc000c5aa0 0x7fcc00156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.920+0000 7fcc04e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc000c5aa0 0x7fcc00156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45320/0 (socket says 172.21.15.27:45320) 2024-03-28T18:56:26.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.920+0000 7fcc04e00640 1 -- 172.21.15.27:0/2442727192 learned_addr learned my addr 172.21.15.27:0/2442727192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.920+0000 7fcc05800640 1 --2- 172.21.15.27:0/2442727192 >> v2:172.21.15.155:3300/0 conn(0x7fcc00152840 0x7fcc00152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.921+0000 7fcc04e00640 1 -- 172.21.15.27:0/2442727192 >> v2:172.21.15.155:3300/0 conn(0x7fcc00152840 msgr2=0x7fcc00152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.921+0000 7fcc04e00640 1 --2- 172.21.15.27:0/2442727192 >> v2:172.21.15.155:3300/0 conn(0x7fcc00152840 0x7fcc00152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.921+0000 7fcc04e00640 1 -- 172.21.15.27:0/2442727192 >> v1:172.21.15.155:6789/0 conn(0x7fcc000c8db0 legacy=0x7fcc00072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.921+0000 7fcc04e00640 1 -- 172.21.15.27:0/2442727192 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000c7d10 con 0x7fcc000c5aa0 2024-03-28T18:56:26.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.922+0000 7fcc04e00640 1 --2- 172.21.15.27:0/2442727192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc000c5aa0 0x7fcc00156090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009870 tx=0x7fcbf402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb5592f970fca77 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:26.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.922+0000 7fcbff400640 1 -- 172.21.15.27:0/2442727192 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40317c0 con 0x7fcc000c5aa0 2024-03-28T18:56:26.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcbff400640 1 -- 172.21.15.27:0/2442727192 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4037070 con 0x7fcc000c5aa0 2024-03-28T18:56:26.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcbff400640 1 -- 172.21.15.27:0/2442727192 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4031b80 con 0x7fcc000c5aa0 2024-03-28T18:56:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcc06200640 1 -- 172.21.15.27:0/2442727192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc000c5aa0 msgr2=0x7fcc00156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcc06200640 1 --2- 172.21.15.27:0/2442727192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc000c5aa0 0x7fcc00156090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009870 tx=0x7fcbf402b860 comp rx=0 tx=0).stop 2024-03-28T18:56:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcc06200640 1 -- 172.21.15.27:0/2442727192 shutdown_connections 2024-03-28T18:56:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcc06200640 1 --2- 172.21.15.27:0/2442727192 >> v2:172.21.15.155:3300/0 conn(0x7fcc00152840 0x7fcc00152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcc06200640 1 --2- 172.21.15.27:0/2442727192 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc000c5aa0 0x7fcc00156090 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcc06200640 1 -- 172.21.15.27:0/2442727192 >> 172.21.15.27:0/2442727192 conn(0x7fcc0006b0c0 msgr2=0x7fcc0006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcc06200640 1 -- 172.21.15.27:0/2442727192 shutdown_connections 2024-03-28T18:56:26.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.923+0000 7fcc06200640 1 -- 172.21.15.27:0/2442727192 wait complete. 2024-03-28T18:56:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.926+0000 7fcc06200640 1 Processor -- start 2024-03-28T18:56:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.926+0000 7fcc06200640 1 -- start start 2024-03-28T18:56:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcc06200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc000c5aa0 0x7fcc0014a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcc06200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000c8db0 0x7fcc00153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcc06200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc00152840 0x7fcc000c0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcc06200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc00072100 con 0x7fcc000c8db0 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcc06200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc00071e20 con 0x7fcc000c5aa0 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcc06200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc00072d30 con 0x7fcc00152840 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcbffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000c8db0 0x7fcc00153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcbffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000c8db0 0x7fcc00153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40324/0 (socket says 172.21.15.27:40324) 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcbffe00640 1 -- 172.21.15.27:0/3509139150 learned_addr learned my addr 172.21.15.27:0/3509139150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcc04e00640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc000c5aa0 0x7fcc0014a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.927+0000 7fcc05800640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc00152840 0x7fcc000c0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbffe00640 1 -- 172.21.15.27:0/3509139150 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc00152840 msgr2=0x7fcc000c0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbffe00640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc00152840 0x7fcc000c0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbffe00640 1 -- 172.21.15.27:0/3509139150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc000c5aa0 msgr2=0x7fcc0014a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbffe00640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc000c5aa0 0x7fcc0014a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbffe00640 1 -- 172.21.15.27:0/3509139150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000c8710 con 0x7fcc000c8db0 2024-03-28T18:56:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbffe00640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000c8db0 0x7fcc00153c90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00c400 tx=0x7fcbec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbfcc00640 1 -- 172.21.15.27:0/3509139150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0052d0 con 0x7fcc000c8db0 2024-03-28T18:56:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbfcc00640 1 -- 172.21.15.27:0/3509139150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec005490 con 0x7fcc000c8db0 2024-03-28T18:56:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc001eedf0 con 0x7fcc000c8db0 2024-03-28T18:56:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.928+0000 7fcbfcc00640 1 -- 172.21.15.27:0/3509139150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec007260 con 0x7fcc000c8db0 2024-03-28T18:56:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.929+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc000adfd0 con 0x7fcc000c8db0 2024-03-28T18:56:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.930+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0004f400 con 0x7fcc000c8db0 2024-03-28T18:56:26.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.931+0000 7fcbfcc00640 1 -- 172.21.15.27:0/3509139150 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcbec007400 con 0x7fcc000c8db0 2024-03-28T18:56:26.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.932+0000 7fcbfcc00640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbe006bc00 0x7fcbe006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:26.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.932+0000 7fcbfcc00640 1 -- 172.21.15.27:0/3509139150 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fcbec08a1f0 con 0x7fcc000c8db0 2024-03-28T18:56:26.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.933+0000 7fcc04e00640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbe006bc00 0x7fcbe006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:26.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.933+0000 7fcc04e00640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbe006bc00 0x7fcbe006e0c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4036000 tx=0x7fcbf403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:26.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:26.937+0000 7fcbfcc00640 1 -- 172.21.15.27:0/3509139150 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcbec058380 con 0x7fcc000c8db0 2024-03-28T18:56:27.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.142+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc00152da0 con 0x7fcc000c8db0 2024-03-28T18:56:27.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.143+0000 7fcbfcc00640 1 -- 172.21.15.27:0/3509139150 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcbec05c420 con 0x7fcc000c8db0 2024-03-28T18:56:27.143 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbe006bc00 msgr2=0x7fcbe006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbe006bc00 0x7fcbe006e0c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4036000 tx=0x7fcbf403d040 comp rx=0 tx=0).stop 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000c8db0 msgr2=0x7fcc00153c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000c8db0 0x7fcc00153c90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcbec00c400 tx=0x7fcbec00c8c0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 shutdown_connections 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbe006bc00 0x7fcbe006e0c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc00152840 0x7fcc000c0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc000c8db0 0x7fcc00153c90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 --2- 172.21.15.27:0/3509139150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc000c5aa0 0x7fcc0014a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 >> 172.21.15.27:0/3509139150 conn(0x7fcc0006b0c0 msgr2=0x7fcc000cb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 shutdown_connections 2024-03-28T18:56:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.146+0000 7fcc06200640 1 -- 172.21.15.27:0/3509139150 wait complete. 2024-03-28T18:56:27.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-03-28T18:56:27.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.393+0000 7fea7f200640 1 Processor -- start 2024-03-28T18:56:27.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.394+0000 7fea7f200640 1 -- start start 2024-03-28T18:56:27.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.395+0000 7fea7f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800c8490 0x7fea801523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.395+0000 7fea7f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea800c37a0 0x7fea80152930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.395+0000 7fea7f200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8006b0c0 con 0x7fea800a82f0 2024-03-28T18:56:27.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.395+0000 7fea7f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea8006b680 con 0x7fea800c37a0 2024-03-28T18:56:27.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.395+0000 7fea7f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea801561b0 con 0x7fea800c8490 2024-03-28T18:56:27.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.395+0000 7fea7e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea800c37a0 0x7fea80152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.396+0000 7fea7e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea800c37a0 0x7fea80152930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40334/0 (socket says 172.21.15.27:40334) 2024-03-28T18:56:27.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.396+0000 7fea7e800640 1 -- 172.21.15.27:0/584136110 learned_addr learned my addr 172.21.15.27:0/584136110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:27.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.396+0000 7fea7d400640 1 --2- 172.21.15.27:0/584136110 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800c8490 0x7fea801523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.396+0000 7fea7e800640 1 -- 172.21.15.27:0/584136110 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800c8490 msgr2=0x7fea801523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:27.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.396+0000 7fea7e800640 1 --2- 172.21.15.27:0/584136110 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800c8490 0x7fea801523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.396+0000 7fea7e800640 1 -- 172.21.15.27:0/584136110 >> v1:172.21.15.155:6789/0 conn(0x7fea800a82f0 legacy=0x7fea800c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.396+0000 7fea7e800640 1 -- 172.21.15.27:0/584136110 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8006b3a0 con 0x7fea800c37a0 2024-03-28T18:56:27.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.397+0000 7fea7e800640 1 --2- 172.21.15.27:0/584136110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea80152930 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fea78009e50 tx=0x7fea7802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbac0f19cb7a7fce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:27.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.398+0000 7fea7ca00640 1 -- 172.21.15.27:0/584136110 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7802ca70 con 0x7fea800c37a0 2024-03-28T18:56:27.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.398+0000 7fea7ca00640 1 -- 172.21.15.27:0/584136110 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7802cc10 con 0x7fea800c37a0 2024-03-28T18:56:27.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.398+0000 7fea7ca00640 1 -- 172.21.15.27:0/584136110 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea780343f0 con 0x7fea800c37a0 2024-03-28T18:56:27.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.399+0000 7fea7f200640 1 -- 172.21.15.27:0/584136110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 msgr2=0x7fea80152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.399+0000 7fea7f200640 1 --2- 172.21.15.27:0/584136110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea80152930 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fea78009e50 tx=0x7fea7802bed0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.399+0000 7fea7f200640 1 -- 172.21.15.27:0/584136110 shutdown_connections 2024-03-28T18:56:27.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.399+0000 7fea7f200640 1 --2- 172.21.15.27:0/584136110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea80152930 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.399+0000 7fea7f200640 1 --2- 172.21.15.27:0/584136110 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800c8490 0x7fea801523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.399+0000 7fea7f200640 1 -- 172.21.15.27:0/584136110 >> 172.21.15.27:0/584136110 conn(0x7fea800aa7c0 msgr2=0x7fea800aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:27.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.399+0000 7fea7f200640 1 -- 172.21.15.27:0/584136110 shutdown_connections 2024-03-28T18:56:27.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.399+0000 7fea7f200640 1 -- 172.21.15.27:0/584136110 wait complete. 2024-03-28T18:56:27.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.402+0000 7fea7f200640 1 Processor -- start 2024-03-28T18:56:27.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.402+0000 7fea7f200640 1 -- start start 2024-03-28T18:56:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.403+0000 7fea7f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800a82f0 0x7fea801557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.403+0000 7fea7f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea801eb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.403+0000 7fea7f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea800c8490 0x7fea801eb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.403+0000 7fea7f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea80152e70 con 0x7fea800c37a0 2024-03-28T18:56:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.403+0000 7fea7f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8006cde0 con 0x7fea800c8490 2024-03-28T18:56:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.403+0000 7fea7f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea800c5a80 con 0x7fea800a82f0 2024-03-28T18:56:27.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea801eb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea801eb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40342/0 (socket says 172.21.15.27:40342) 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7d400640 1 -- 172.21.15.27:0/3917604520 learned_addr learned my addr 172.21.15.27:0/3917604520 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7e800640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea800c8490 0x7fea801eb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7de00640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800a82f0 0x7fea801557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7d400640 1 -- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800a82f0 msgr2=0x7fea801557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7d400640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800a82f0 0x7fea801557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7d400640 1 -- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea800c8490 msgr2=0x7fea801eb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7d400640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea800c8490 0x7fea801eb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7d400640 1 -- 172.21.15.27:0/3917604520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea800adc70 con 0x7fea800c37a0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.404+0000 7fea7e800640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea800c8490 0x7fea801eb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.405+0000 7fea7d400640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea801eb470 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fea7000d800 tx=0x7fea7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.405+0000 7fea76000640 1 -- 172.21.15.27:0/3917604520 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7000b6c0 con 0x7fea800c37a0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.405+0000 7fea76000640 1 -- 172.21.15.27:0/3917604520 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea70019070 con 0x7fea800c37a0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.405+0000 7fea76000640 1 -- 172.21.15.27:0/3917604520 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7000ba80 con 0x7fea800c37a0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.405+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea801defc0 con 0x7fea800c37a0 2024-03-28T18:56:27.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.405+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea801597c0 con 0x7fea800c37a0 2024-03-28T18:56:27.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.406+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea801dedb0 con 0x7fea800c37a0 2024-03-28T18:56:27.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.408+0000 7fea76000640 1 -- 172.21.15.27:0/3917604520 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fea7000bc20 con 0x7fea800c37a0 2024-03-28T18:56:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.409+0000 7fea76000640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5c06bc20 0x7fea5c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.409+0000 7fea76000640 1 -- 172.21.15.27:0/3917604520 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fea7008b270 con 0x7fea800c37a0 2024-03-28T18:56:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.409+0000 7fea7de00640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5c06bc20 0x7fea5c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.410+0000 7fea7de00640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5c06bc20 0x7fea5c06e0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea68000f60 tx=0x7fea68009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.413+0000 7fea76000640 1 -- 172.21.15.27:0/3917604520 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fea70059a30 con 0x7fea800c37a0 2024-03-28T18:56:27.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.705+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fea8004f400 con 0x7fea800c37a0 2024-03-28T18:56:27.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.706+0000 7fea76000640 1 -- 172.21.15.27:0/3917604520 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v26) v1 ==== 99+0+11 (secure 0 0 0) 0x7fea70059330 con 0x7fea800c37a0 2024-03-28T18:56:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5c06bc20 msgr2=0x7fea5c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5c06bc20 0x7fea5c06e0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea68000f60 tx=0x7fea68009040 comp rx=0 tx=0).stop 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 msgr2=0x7fea801eb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea801eb470 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fea7000d800 tx=0x7fea7000dcc0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 shutdown_connections 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5c06bc20 0x7fea5c06e0e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea800c8490 0x7fea801eb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea800c37a0 0x7fea801eb470 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 --2- 172.21.15.27:0/3917604520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea800a82f0 0x7fea801557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 >> 172.21.15.27:0/3917604520 conn(0x7fea800aa7c0 msgr2=0x7fea800c5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 shutdown_connections 2024-03-28T18:56:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.709+0000 7fea7f200640 1 -- 172.21.15.27:0/3917604520 wait complete. 2024-03-28T18:56:27.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-03-28T18:56:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.932+0000 7f11f5800640 1 Processor -- start 2024-03-28T18:56:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.932+0000 7f11f5800640 1 -- start start 2024-03-28T18:56:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.934+0000 7f11f5800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11f00c5aa0 0x7f11f0156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.934+0000 7f11f5800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c8db0 0x7f11f0072620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.934+0000 7f11f5800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f00735f0 con 0x7f11f0072b60 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.934+0000 7f11f5800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f01565d0 con 0x7f11f00c5aa0 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.934+0000 7f11f5800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f00c82d0 con 0x7f11f00c8db0 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.934+0000 7f11efe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11f00c5aa0 0x7f11f0156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.934+0000 7f11efe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11f00c5aa0 0x7f11f0156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40352/0 (socket says 172.21.15.27:40352) 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.934+0000 7f11efe00640 1 -- 172.21.15.27:0/240801417 learned_addr learned my addr 172.21.15.27:0/240801417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.935+0000 7f11e7e00640 1 --2- 172.21.15.27:0/240801417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c8db0 0x7f11f0072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.935+0000 7f11efe00640 1 -- 172.21.15.27:0/240801417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c8db0 msgr2=0x7f11f0072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.935+0000 7f11efe00640 1 --2- 172.21.15.27:0/240801417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c8db0 0x7f11f0072620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.935+0000 7f11efe00640 1 -- 172.21.15.27:0/240801417 >> v1:172.21.15.155:6789/0 conn(0x7f11f0072b60 legacy=0x7f11f0152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.935+0000 7f11efe00640 1 -- 172.21.15.27:0/240801417 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f00c7d10 con 0x7f11f00c5aa0 2024-03-28T18:56:27.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.937+0000 7f11efe00640 1 --2- 172.21.15.27:0/240801417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c5aa0 0x7f11f0156090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f11f006bf20 tx=0x7f11e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20722d55ce347fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:27.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.937+0000 7f11ef400640 1 -- 172.21.15.27:0/240801417 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e00315a0 con 0x7f11f00c5aa0 2024-03-28T18:56:27.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.937+0000 7f11ef400640 1 -- 172.21.15.27:0/240801417 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e0037070 con 0x7f11f00c5aa0 2024-03-28T18:56:27.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.937+0000 7f11ef400640 1 -- 172.21.15.27:0/240801417 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e0031960 con 0x7f11f00c5aa0 2024-03-28T18:56:27.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.938+0000 7f11f5800640 1 -- 172.21.15.27:0/240801417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c5aa0 msgr2=0x7f11f0156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.938+0000 7f11f5800640 1 --2- 172.21.15.27:0/240801417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c5aa0 0x7f11f0156090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f11f006bf20 tx=0x7f11e002b840 comp rx=0 tx=0).stop 2024-03-28T18:56:27.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.938+0000 7f11f5800640 1 -- 172.21.15.27:0/240801417 shutdown_connections 2024-03-28T18:56:27.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.938+0000 7f11f5800640 1 --2- 172.21.15.27:0/240801417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c8db0 0x7f11f0072620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.938+0000 7f11f5800640 1 --2- 172.21.15.27:0/240801417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c5aa0 0x7f11f0156090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.938+0000 7f11f5800640 1 -- 172.21.15.27:0/240801417 >> 172.21.15.27:0/240801417 conn(0x7f11f006b0c0 msgr2=0x7f11f006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:27.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.938+0000 7f11f5800640 1 -- 172.21.15.27:0/240801417 shutdown_connections 2024-03-28T18:56:27.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.938+0000 7f11f5800640 1 -- 172.21.15.27:0/240801417 wait complete. 2024-03-28T18:56:27.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.940+0000 7f11f5800640 1 Processor -- start 2024-03-28T18:56:27.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.941+0000 7f11f5800640 1 -- start start 2024-03-28T18:56:27.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f0072b60 0x7f11f00c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f5800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c5aa0 0x7f11f0158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c8db0 0x7f11f00afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f5800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f0072100 con 0x7f11f00c8db0 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f5800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f0071e20 con 0x7f11f0072b60 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f5800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f0152f50 con 0x7f11f00c5aa0 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c8db0 0x7f11f00afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c8db0 0x7f11f00afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40354/0 (socket says 172.21.15.27:40354) 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.942+0000 7f11f4e00640 1 -- 172.21.15.27:0/3792659943 learned_addr learned my addr 172.21.15.27:0/3792659943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11efe00640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f0072b60 0x7f11f00c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11e7e00640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c5aa0 0x7f11f0158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11f4e00640 1 -- 172.21.15.27:0/3792659943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c5aa0 msgr2=0x7f11f0158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11f4e00640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c5aa0 0x7f11f0158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11f4e00640 1 -- 172.21.15.27:0/3792659943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f0072b60 msgr2=0x7f11f00c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11f4e00640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f0072b60 0x7f11f00c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:27.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11f4e00640 1 -- 172.21.15.27:0/3792659943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f01ec380 con 0x7f11f00c8db0 2024-03-28T18:56:27.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11f4e00640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c8db0 0x7f11f00afa20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f11e800c780 tx=0x7f11e800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:27.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.943+0000 7f11ecc00640 1 -- 172.21.15.27:0/3792659943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e8004b60 con 0x7f11f00c8db0 2024-03-28T18:56:27.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.944+0000 7f11ecc00640 1 -- 172.21.15.27:0/3792659943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e8004d00 con 0x7f11f00c8db0 2024-03-28T18:56:27.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.944+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f01eb5c0 con 0x7f11f00c8db0 2024-03-28T18:56:27.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.944+0000 7f11ecc00640 1 -- 172.21.15.27:0/3792659943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e800aa30 con 0x7f11f00c8db0 2024-03-28T18:56:27.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.944+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f01ebe60 con 0x7f11f00c8db0 2024-03-28T18:56:27.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.945+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f01ec060 con 0x7f11f00c8db0 2024-03-28T18:56:27.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.946+0000 7f11ecc00640 1 -- 172.21.15.27:0/3792659943 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11e800abd0 con 0x7f11f00c8db0 2024-03-28T18:56:27.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.947+0000 7f11ecc00640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11cc06bbd0 0x7f11cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:27.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.947+0000 7f11ecc00640 1 -- 172.21.15.27:0/3792659943 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f11e808afe0 con 0x7f11f00c8db0 2024-03-28T18:56:27.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.947+0000 7f11efe00640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11cc06bbd0 0x7f11cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.948+0000 7f11efe00640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11cc06bbd0 0x7f11cc06e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11e0036000 tx=0x7f11e003d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:27.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:27.952+0000 7f11ecc00640 1 -- 172.21.15.27:0/3792659943 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11f01ec060 con 0x7f11f00c8db0 2024-03-28T18:56:28.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.155+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f11f004f400 con 0x7f11f00c8db0 2024-03-28T18:56:28.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.156+0000 7f11ecc00640 1 -- 172.21.15.27:0/3792659943 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v26) v1 ==== 95+0+11 (secure 0 0 0) 0x7f11e8020070 con 0x7f11f00c8db0 2024-03-28T18:56:28.156 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11cc06bbd0 msgr2=0x7f11cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11cc06bbd0 0x7f11cc06e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11e0036000 tx=0x7f11e003d040 comp rx=0 tx=0).stop 2024-03-28T18:56:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c8db0 msgr2=0x7f11f00afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c8db0 0x7f11f00afa20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f11e800c780 tx=0x7f11e800cc40 comp rx=0 tx=0).stop 2024-03-28T18:56:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 shutdown_connections 2024-03-28T18:56:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11cc06bbd0 0x7f11cc06e090 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11f00c8db0 0x7f11f00afa20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f00c5aa0 0x7f11f0158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 --2- 172.21.15.27:0/3792659943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f0072b60 0x7f11f00c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.158+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 >> 172.21.15.27:0/3792659943 conn(0x7f11f006b0c0 msgr2=0x7f11f006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.159+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 shutdown_connections 2024-03-28T18:56:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.159+0000 7f11f5800640 1 -- 172.21.15.27:0/3792659943 wait complete. 2024-03-28T18:56:28.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-03-28T18:56:28.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.399+0000 7f1be2c00640 1 Processor -- start 2024-03-28T18:56:28.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.399+0000 7f1be2c00640 1 -- start start 2024-03-28T18:56:28.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be2c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1bdc072df0 0x7f1bdc0c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:28.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be2c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc0a82c0 0x7f1bdc0c2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:28.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be2c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bdc0c0f80 con 0x7f1bdc156240 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be2c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bdc0c1260 con 0x7f1bdc072df0 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be2c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0c51f0 con 0x7f1bdc0a82c0 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be0e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1bdc072df0 0x7f1bdc0c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be1800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1bdc156240 0x7f1bdc0bff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44748/0 (socket says 172.21.15.27:44748) 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be1800640 1 -- 172.21.15.27:0/1859351524 learned_addr learned my addr 172.21.15.27:0/1859351524 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.401+0000 7f1be2200640 1 --2- 172.21.15.27:0/1859351524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc0a82c0 0x7f1bdc0c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.402+0000 7f1bdbe00640 1 -- 172.21.15.27:0/1859351524 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605766425 0 0) 0x7f1bdc0c0f80 con 0x7f1bdc156240 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.402+0000 7f1be0e00640 1 -- 172.21.15.27:0/1859351524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc0a82c0 msgr2=0x7f1bdc0c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:28.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.402+0000 7f1be0e00640 1 --2- 172.21.15.27:0/1859351524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc0a82c0 0x7f1bdc0c2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.402+0000 7f1be0e00640 1 -- 172.21.15.27:0/1859351524 >> v1:172.21.15.27:6789/0 conn(0x7f1bdc156240 legacy=0x7f1bdc0bff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:28.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.402+0000 7f1be0e00640 1 -- 172.21.15.27:0/1859351524 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc0c4ba0 con 0x7f1bdc072df0 2024-03-28T18:56:28.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.402+0000 7f1be2200640 1 --2- 172.21.15.27:0/1859351524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc0a82c0 0x7f1bdc0c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:28.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.403+0000 7f1be0e00640 1 --2- 172.21.15.27:0/1859351524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc072df0 0x7f1bdc0c21d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1bc80097a0 tx=0x7f1bc802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5f86469f58858 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:28.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.403+0000 7f1bdbe00640 1 -- 172.21.15.27:0/1859351524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8039070 con 0x7f1bdc072df0 2024-03-28T18:56:28.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.403+0000 7f1bdbe00640 1 -- 172.21.15.27:0/1859351524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc80316f0 con 0x7f1bdc072df0 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.403+0000 7f1bdbe00640 1 -- 172.21.15.27:0/1859351524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8031ac0 con 0x7f1bdc072df0 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.404+0000 7f1be2c00640 1 -- 172.21.15.27:0/1859351524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc072df0 msgr2=0x7f1bdc0c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.404+0000 7f1be2c00640 1 --2- 172.21.15.27:0/1859351524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc072df0 0x7f1bdc0c21d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1bc80097a0 tx=0x7f1bc802b790 comp rx=0 tx=0).stop 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.404+0000 7f1be2c00640 1 -- 172.21.15.27:0/1859351524 shutdown_connections 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.404+0000 7f1be2c00640 1 --2- 172.21.15.27:0/1859351524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc0a82c0 0x7f1bdc0c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.404+0000 7f1be2c00640 1 --2- 172.21.15.27:0/1859351524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc072df0 0x7f1bdc0c21d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.404+0000 7f1be2c00640 1 -- 172.21.15.27:0/1859351524 >> 172.21.15.27:0/1859351524 conn(0x7f1bdc0a88b0 msgr2=0x7f1bdc0a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.404+0000 7f1be2c00640 1 -- 172.21.15.27:0/1859351524 shutdown_connections 2024-03-28T18:56:28.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.404+0000 7f1be2c00640 1 -- 172.21.15.27:0/1859351524 wait complete. 2024-03-28T18:56:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.406+0000 7f1be2c00640 1 Processor -- start 2024-03-28T18:56:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.406+0000 7f1be2c00640 1 -- start start 2024-03-28T18:56:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc072df0 0x7f1bdc1eb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:28.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bdc0a82c0 0x7f1bdc149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc156240 0x7f1bdc14cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0c0640 con 0x7f1bdc156240 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0c19d0 con 0x7f1bdc0a82c0 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0c07c0 con 0x7f1bdc072df0 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc156240 0x7f1bdc14cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc156240 0x7f1bdc14cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40366/0 (socket says 172.21.15.27:40366) 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2200640 1 -- 172.21.15.27:0/68217455 learned_addr learned my addr 172.21.15.27:0/68217455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be1800640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc072df0 0x7f1bdc1eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2200640 1 -- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc072df0 msgr2=0x7f1bdc1eb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be0e00640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bdc0a82c0 0x7f1bdc149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2200640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc072df0 0x7f1bdc1eb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2200640 1 -- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bdc0a82c0 msgr2=0x7f1bdc149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2200640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bdc0a82c0 0x7f1bdc149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be2200640 1 -- 172.21.15.27:0/68217455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc14dcf0 con 0x7f1bdc156240 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.407+0000 7f1be1800640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc072df0 0x7f1bdc1eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.408+0000 7f1be2200640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc156240 0x7f1bdc14cbc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4007ef0 tx=0x7f1bd400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.408+0000 7f1bd9600640 1 -- 172.21.15.27:0/68217455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd401a070 con 0x7f1bdc156240 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.408+0000 7f1bd9600640 1 -- 172.21.15.27:0/68217455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd4003b90 con 0x7f1bdc156240 2024-03-28T18:56:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.408+0000 7f1bd9600640 1 -- 172.21.15.27:0/68217455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd400de20 con 0x7f1bdc156240 2024-03-28T18:56:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.408+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bdc14dfd0 con 0x7f1bdc156240 2024-03-28T18:56:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.408+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bdc14b060 con 0x7f1bdc156240 2024-03-28T18:56:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.410+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bdc04f400 con 0x7f1bdc156240 2024-03-28T18:56:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.410+0000 7f1bd9600640 1 -- 172.21.15.27:0/68217455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1bd401d070 con 0x7f1bdc156240 2024-03-28T18:56:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.411+0000 7f1bd9600640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1bbc06bbd0 0x7f1bbc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.411+0000 7f1bd9600640 1 -- 172.21.15.27:0/68217455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f1bd408aff0 con 0x7f1bdc156240 2024-03-28T18:56:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.411+0000 7f1be1800640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1bbc06bbd0 0x7f1bbc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.412+0000 7f1be1800640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1bbc06bbd0 0x7f1bbc06e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0000f60 tx=0x7f1bd0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:28.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.416+0000 7f1bd9600640 1 -- 172.21.15.27:0/68217455 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1bd4010070 con 0x7f1bdc156240 2024-03-28T18:56:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.828+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1bdc155770 con 0x7f1bdc156240 2024-03-28T18:56:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.829+0000 7f1bd9600640 1 -- 172.21.15.27:0/68217455 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v26) v1 ==== 99+0+11 (secure 0 0 0) 0x7f1bd4059100 con 0x7f1bdc156240 2024-03-28T18:56:28.829 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.832+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1bbc06bbd0 msgr2=0x7f1bbc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.832+0000 7f1be2c00640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1bbc06bbd0 0x7f1bbc06e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0000f60 tx=0x7f1bd0009040 comp rx=0 tx=0).stop 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.832+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc156240 msgr2=0x7f1bdc14cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.832+0000 7f1be2c00640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc156240 0x7f1bdc14cbc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4007ef0 tx=0x7f1bd400d4a0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.833+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 shutdown_connections 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.833+0000 7f1be2c00640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1bbc06bbd0 0x7f1bbc06e090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.833+0000 7f1be2c00640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bdc156240 0x7f1bdc14cbc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.833+0000 7f1be2c00640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bdc0a82c0 0x7f1bdc149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.833+0000 7f1be2c00640 1 --2- 172.21.15.27:0/68217455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bdc072df0 0x7f1bdc1eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.833+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 >> 172.21.15.27:0/68217455 conn(0x7f1bdc0a88b0 msgr2=0x7f1bdc06e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.833+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 shutdown_connections 2024-03-28T18:56:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:28.833+0000 7f1be2c00640 1 -- 172.21.15.27:0/68217455 wait complete. 2024-03-28T18:56:28.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-03-28T18:56:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.070+0000 7fa5a4a00640 1 Processor -- start 2024-03-28T18:56:29.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.070+0000 7fa5a4a00640 1 -- start start 2024-03-28T18:56:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.072+0000 7fa5a4a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5a014d4e0 0x7fa5a01540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.072+0000 7fa5a4a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 0x7fa5a00ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.072+0000 7fa5a4a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a006dbe0 con 0x7fa5a006e960 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.072+0000 7fa5a4a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a006de80 con 0x7fa5a014d4e0 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.072+0000 7fa5a4a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a006e160 con 0x7fa5a00c17a0 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.072+0000 7fa59de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5a014d4e0 0x7fa5a01540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.072+0000 7fa59de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5a014d4e0 0x7fa5a01540e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40374/0 (socket says 172.21.15.27:40374) 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.072+0000 7fa59de00640 1 -- 172.21.15.27:0/2234300231 learned_addr learned my addr 172.21.15.27:0/2234300231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.073+0000 7fa59e800640 1 --2- 172.21.15.27:0/2234300231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 0x7fa5a00ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.073+0000 7fa59de00640 1 -- 172.21.15.27:0/2234300231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 msgr2=0x7fa5a00ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.073+0000 7fa59de00640 1 --2- 172.21.15.27:0/2234300231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 0x7fa5a00ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.073+0000 7fa59de00640 1 -- 172.21.15.27:0/2234300231 >> v1:172.21.15.27:6789/0 conn(0x7fa5a006e960 legacy=0x7fa5a00a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.073+0000 7fa59de00640 1 -- 172.21.15.27:0/2234300231 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a006e440 con 0x7fa5a014d4e0 2024-03-28T18:56:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.074+0000 7fa59de00640 1 --2- 172.21.15.27:0/2234300231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a01540e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa590009870 tx=0x7fa59002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e6c695a2543ece server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.074+0000 7fa59ca00640 1 -- 172.21.15.27:0/2234300231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa590034070 con 0x7fa5a014d4e0 2024-03-28T18:56:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa59ca00640 1 -- 172.21.15.27:0/2234300231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa590030840 con 0x7fa5a014d4e0 2024-03-28T18:56:29.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa59ca00640 1 -- 172.21.15.27:0/2234300231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa590030c10 con 0x7fa5a014d4e0 2024-03-28T18:56:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa5a4a00640 1 -- 172.21.15.27:0/2234300231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 msgr2=0x7fa5a01540e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/2234300231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a01540e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa590009870 tx=0x7fa59002b980 comp rx=0 tx=0).stop 2024-03-28T18:56:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa5a4a00640 1 -- 172.21.15.27:0/2234300231 shutdown_connections 2024-03-28T18:56:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/2234300231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 0x7fa5a00ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/2234300231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a01540e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa5a4a00640 1 -- 172.21.15.27:0/2234300231 >> 172.21.15.27:0/2234300231 conn(0x7fa5a006ca00 msgr2=0x7fa5a006cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.075+0000 7fa5a4a00640 1 -- 172.21.15.27:0/2234300231 shutdown_connections 2024-03-28T18:56:29.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.076+0000 7fa5a4a00640 1 -- 172.21.15.27:0/2234300231 wait complete. 2024-03-28T18:56:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.078+0000 7fa5a4a00640 1 Processor -- start 2024-03-28T18:56:29.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.078+0000 7fa5a4a00640 1 -- start start 2024-03-28T18:56:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.079+0000 7fa5a4a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5a006e960 0x7fa5a01eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.079+0000 7fa5a4a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 0x7fa5a01ef030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.079+0000 7fa5a4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a0142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.079+0000 7fa5a4a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00a9b10 con 0x7fa5a014d4e0 2024-03-28T18:56:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.079+0000 7fa5a4a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00c1bf0 con 0x7fa5a006e960 2024-03-28T18:56:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.079+0000 7fa5a4a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00a9c90 con 0x7fa5a00c17a0 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a0142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a0142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40382/0 (socket says 172.21.15.27:40382) 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59e800640 1 -- 172.21.15.27:0/3465037075 learned_addr learned my addr 172.21.15.27:0/3465037075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59de00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5a006e960 0x7fa5a01eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59d400640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 0x7fa5a01ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59e800640 1 -- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 msgr2=0x7fa5a01ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59e800640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 0x7fa5a01ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59e800640 1 -- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5a006e960 msgr2=0x7fa5a01eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59e800640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5a006e960 0x7fa5a01eeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59e800640 1 -- 172.21.15.27:0/3465037075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a00c53a0 con 0x7fa5a014d4e0 2024-03-28T18:56:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.080+0000 7fa59de00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5a006e960 0x7fa5a01eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.081+0000 7fa59e800640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a0142690 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa598007e70 tx=0x7fa59800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.081+0000 7fa596000640 1 -- 172.21.15.27:0/3465037075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa598014620 con 0x7fa5a014d4e0 2024-03-28T18:56:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.081+0000 7fa596000640 1 -- 172.21.15.27:0/3465037075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa598019070 con 0x7fa5a014d4e0 2024-03-28T18:56:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.081+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a00c5680 con 0x7fa5a014d4e0 2024-03-28T18:56:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.081+0000 7fa596000640 1 -- 172.21.15.27:0/3465037075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa598014a20 con 0x7fa5a014d4e0 2024-03-28T18:56:29.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.081+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a00c5c40 con 0x7fa5a014d4e0 2024-03-28T18:56:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.082+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a004f470 con 0x7fa5a014d4e0 2024-03-28T18:56:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.083+0000 7fa596000640 1 -- 172.21.15.27:0/3465037075 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa598014bc0 con 0x7fa5a014d4e0 2024-03-28T18:56:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.084+0000 7fa596000640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa58006bbd0 0x7fa58006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.084+0000 7fa596000640 1 -- 172.21.15.27:0/3465037075 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fa59808b6a0 con 0x7fa5a014d4e0 2024-03-28T18:56:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.085+0000 7fa59de00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa58006bbd0 0x7fa58006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.086+0000 7fa59de00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa58006bbd0 0x7fa58006e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa590036000 tx=0x7fa59003b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:29.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.089+0000 7fa596000640 1 -- 172.21.15.27:0/3465037075 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa598059e00 con 0x7fa5a014d4e0 2024-03-28T18:56:29.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.291+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fa5a006ed90 con 0x7fa5a014d4e0 2024-03-28T18:56:29.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.292+0000 7fa596000640 1 -- 172.21.15.27:0/3465037075 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v26) v1 ==== 95+0+11 (secure 0 0 0) 0x7fa598059700 con 0x7fa5a014d4e0 2024-03-28T18:56:29.291 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:29.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa58006bbd0 msgr2=0x7fa58006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa58006bbd0 0x7fa58006e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa590036000 tx=0x7fa59003b040 comp rx=0 tx=0).stop 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 msgr2=0x7fa5a0142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a0142690 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa598007e70 tx=0x7fa59800e4b0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 shutdown_connections 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa58006bbd0 0x7fa58006e090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a014d4e0 0x7fa5a0142690 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5a00c17a0 0x7fa5a01ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 --2- 172.21.15.27:0/3465037075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5a006e960 0x7fa5a01eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 >> 172.21.15.27:0/3465037075 conn(0x7fa5a006ca00 msgr2=0x7fa5a014ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 shutdown_connections 2024-03-28T18:56:29.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.295+0000 7fa5a4a00640 1 -- 172.21.15.27:0/3465037075 wait complete. 2024-03-28T18:56:29.314 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-03-28T18:56:29.314 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:56:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.527+0000 7f9ddf800640 1 Processor -- start 2024-03-28T18:56:29.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.527+0000 7f9ddf800640 1 -- start start 2024-03-28T18:56:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.529+0000 7f9ddf800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9dd80627e0 0x7f9dd80c07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.529+0000 7f9ddf800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd80ab630 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.529+0000 7f9ddf800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd80ab910 con 0x7f9dd8152ea0 2024-03-28T18:56:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.529+0000 7f9ddf800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd80abc90 con 0x7f9dd80627e0 2024-03-28T18:56:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.529+0000 7f9dd7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9dd80627e0 0x7f9dd80c07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.529+0000 7f9ddd800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9dd80ac6e0 0x7f9dd81557e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44766/0 (socket says 172.21.15.27:44766) 2024-03-28T18:56:29.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.529+0000 7f9ddd800640 1 -- 172.21.15.27:0/3021776433 learned_addr learned my addr 172.21.15.27:0/3021776433 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:29.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.530+0000 7f9dd7e00640 1 -- 172.21.15.27:0/3021776433 >> v1:172.21.15.155:6789/0 conn(0x7f9dd8152ea0 legacy=0x7f9dd806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:29.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.530+0000 7f9dd7e00640 1 -- 172.21.15.27:0/3021776433 >> v1:172.21.15.27:6789/0 conn(0x7f9dd80ac6e0 legacy=0x7f9dd81557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.530+0000 7f9dd7e00640 1 -- 172.21.15.27:0/3021776433 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd80abee0 con 0x7f9dd80627e0 2024-03-28T18:56:29.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.530+0000 7f9dd7400640 1 -- 172.21.15.27:0/3021776433 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127819661 0 0) 0x7f9dd80ab630 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.531+0000 7f9dd7e00640 1 --2- 172.21.15.27:0/3021776433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80627e0 0x7f9dd80c07f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40097a0 tx=0x7f9dc402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7010694ecaa39908 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:29.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.531+0000 7f9dd7400640 1 -- 172.21.15.27:0/3021776433 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4034070 con 0x7f9dd80627e0 2024-03-28T18:56:29.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.531+0000 7f9dd7400640 1 -- 172.21.15.27:0/3021776433 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc4030610 con 0x7f9dd80627e0 2024-03-28T18:56:29.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.531+0000 7f9dd7400640 1 -- 172.21.15.27:0/3021776433 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc40309e0 con 0x7f9dd80627e0 2024-03-28T18:56:29.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.532+0000 7f9ddf800640 1 -- 172.21.15.27:0/3021776433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80627e0 msgr2=0x7f9dd80c07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.532+0000 7f9ddf800640 1 --2- 172.21.15.27:0/3021776433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80627e0 0x7f9dd80c07f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40097a0 tx=0x7f9dc402b960 comp rx=0 tx=0).stop 2024-03-28T18:56:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.532+0000 7f9ddf800640 1 -- 172.21.15.27:0/3021776433 shutdown_connections 2024-03-28T18:56:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.532+0000 7f9ddf800640 1 --2- 172.21.15.27:0/3021776433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80627e0 0x7f9dd80c07f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.532+0000 7f9ddf800640 1 -- 172.21.15.27:0/3021776433 >> 172.21.15.27:0/3021776433 conn(0x7f9dd806b0c0 msgr2=0x7f9dd806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.532+0000 7f9ddf800640 1 -- 172.21.15.27:0/3021776433 shutdown_connections 2024-03-28T18:56:29.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.532+0000 7f9ddf800640 1 -- 172.21.15.27:0/3021776433 wait complete. 2024-03-28T18:56:29.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.534+0000 7f9ddf800640 1 Processor -- start 2024-03-28T18:56:29.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.534+0000 7f9ddf800640 1 -- start start 2024-03-28T18:56:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9ddf800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9dd80627e0 0x7f9dd8153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9ddf800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80ac6e0 0x7f9dd81ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9ddf800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9dd8152ea0 0x7f9dd8150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9ddf800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd8155ef0 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9ddf800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd8153280 con 0x7f9dd8152ea0 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9ddf800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd8156070 con 0x7f9dd80627e0 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9dd7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80ac6e0 0x7f9dd81ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9dd7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80ac6e0 0x7f9dd81ebbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40408/0 (socket says 172.21.15.27:40408) 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9dd7e00640 1 -- 172.21.15.27:0/2129245648 learned_addr learned my addr 172.21.15.27:0/2129245648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9ddce00640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9dd80627e0 0x7f9dd8153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.535+0000 7f9ddd800640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9dd8152ea0 0x7f9dd8150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd7e00640 1 -- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9dd80627e0 msgr2=0x7f9dd8153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd7e00640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9dd80627e0 0x7f9dd8153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd7e00640 1 -- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9dd8152ea0 msgr2=0x7f9dd8150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd7e00640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9dd8152ea0 0x7f9dd8150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd7e00640 1 -- 172.21.15.27:0/2129245648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd81554e0 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9ddd800640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9dd8152ea0 0x7f9dd8150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:29.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd7e00640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80ac6e0 0x7f9dd81ebbb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4000c00 tx=0x7f9dc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd4c00640 1 -- 172.21.15.27:0/2129245648 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4034040 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd4c00640 1 -- 172.21.15.27:0/2129245648 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc4004030 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9dd4c00640 1 -- 172.21.15.27:0/2129245648 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4004400 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.536+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd81eb3b0 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.537+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd8159be0 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.538+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd806c050 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.539+0000 7f9dd4c00640 1 -- 172.21.15.27:0/2129245648 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9dc404c060 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.540+0000 7f9dd4c00640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9db806bbd0 0x7f9db806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:29.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.540+0000 7f9dd4c00640 1 -- 172.21.15.27:0/2129245648 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f9dc40aca50 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.540+0000 7f9ddce00640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9db806bbd0 0x7f9db806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:29.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.541+0000 7f9ddce00640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9db806bbd0 0x7f9db806e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc000f60 tx=0x7f9dcc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:29.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.545+0000 7f9dd4c00640 1 -- 172.21.15.27:0/2129245648 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9dc407b190 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.789+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dd804f400 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.791+0000 7f9dd4c00640 1 -- 172.21.15.27:0/2129245648 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f9dc407aa90 con 0x7f9dd80ac6e0 2024-03-28T18:56:29.791 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:29.791 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:56:29.791 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:56:29.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9db806bbd0 msgr2=0x7f9db806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9db806bbd0 0x7f9db806e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc000f60 tx=0x7f9dcc009040 comp rx=0 tx=0).stop 2024-03-28T18:56:29.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80ac6e0 msgr2=0x7f9dd81ebbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:29.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80ac6e0 0x7f9dd81ebbb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4000c00 tx=0x7f9dc40046c0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 shutdown_connections 2024-03-28T18:56:29.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9db806bbd0 0x7f9db806e090 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9dd8152ea0 0x7f9dd8150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dd80ac6e0 0x7f9dd81ebbb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 --2- 172.21.15.27:0/2129245648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9dd80627e0 0x7f9dd8153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:29.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 >> 172.21.15.27:0/2129245648 conn(0x7f9dd806b0c0 msgr2=0x7f9dd81521c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:29.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 shutdown_connections 2024-03-28T18:56:29.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:29.794+0000 7f9ddf800640 1 -- 172.21.15.27:0/2129245648 wait complete. 2024-03-28T18:56:29.814 INFO:tasks.mds_thrash:Ready to start thrashing 2024-03-28T18:56:29.814 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-03-28T18:56:29.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:56:30.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.030+0000 7ff445400640 1 Processor -- start 2024-03-28T18:56:30.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.031+0000 7ff445400640 1 -- start start 2024-03-28T18:56:30.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.032+0000 7ff445400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff44014d6b0 0x7ff4400c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.032+0000 7ff445400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400c9360 con 0x7ff440153520 2024-03-28T18:56:30.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.032+0000 7ff445400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400c9640 con 0x7ff4400704f0 2024-03-28T18:56:30.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.032+0000 7ff445400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4400c4c00 con 0x7ff44014d6b0 2024-03-28T18:56:30.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.032+0000 7ff43f200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff440153520 0x7ff440153930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44782/0 (socket says 172.21.15.27:44782) 2024-03-28T18:56:30.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.032+0000 7ff43f200640 1 -- 172.21.15.27:0/760805729 learned_addr learned my addr 172.21.15.27:0/760805729 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.033+0000 7ff43e800640 1 --2- 172.21.15.27:0/760805729 >> v2:172.21.15.155:3300/0 conn(0x7ff44014d6b0 0x7ff4400c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.033+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289068635 0 0) 0x7ff4400c9360 con 0x7ff440153520 2024-03-28T18:56:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.033+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff424003520 con 0x7ff440153520 2024-03-28T18:56:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.033+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 119004904 0 0) 0x7ff424003520 con 0x7ff440153520 2024-03-28T18:56:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.033+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 >> v2:172.21.15.155:3300/0 conn(0x7ff44014d6b0 msgr2=0x7ff4400c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.033+0000 7ff43de00640 1 --2- 172.21.15.27:0/760805729 >> v2:172.21.15.155:3300/0 conn(0x7ff44014d6b0 0x7ff4400c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.033+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 >> v1:172.21.15.155:6789/0 conn(0x7ff4400704f0 legacy=0x7ff4400c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.033+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400c4ee0 con 0x7ff440153520 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7ff4380019f0 con 0x7ff440153520 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 >> v1:172.21.15.27:6789/0 conn(0x7ff440153520 legacy=0x7ff440153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43de00640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff424003a00 0x7ff424003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43de00640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff424004800 0x7ff424004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43de00640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240052e0 0x7ff42400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400c4c00 con 0x7ff4240052e0 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42400c1e0 con 0x7ff424004800 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42400c3e0 con 0x7ff424003a00 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43f200640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240052e0 0x7ff42400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff43e800640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff424003a00 0x7ff424003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.034+0000 7ff435e00640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff424004800 0x7ff424004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.035+0000 7ff43f200640 1 -- 172.21.15.27:0/760805729 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff424003a00 msgr2=0x7ff424003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.035+0000 7ff43f200640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff424003a00 0x7ff424003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.035+0000 7ff43f200640 1 -- 172.21.15.27:0/760805729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff424004800 msgr2=0x7ff424004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.035+0000 7ff43f200640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff424004800 0x7ff424004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.035+0000 7ff43f200640 1 -- 172.21.15.27:0/760805729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff424003520 con 0x7ff4240052e0 2024-03-28T18:56:30.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.036+0000 7ff43f200640 1 -- 172.21.15.27:0/760805729 reap_dead start 2024-03-28T18:56:30.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.036+0000 7ff43f200640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240052e0 0x7ff42400b9c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff43800a760 tx=0x7ff424005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e1e01e68fcc99a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:30.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.037+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43800a3b0 con 0x7ff4240052e0 2024-03-28T18:56:30.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.037+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff438034650 con 0x7ff4240052e0 2024-03-28T18:56:30.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.037+0000 7ff43de00640 1 -- 172.21.15.27:0/760805729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff438034a40 con 0x7ff4240052e0 2024-03-28T18:56:30.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.037+0000 7ff445400640 1 -- 172.21.15.27:0/760805729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240052e0 msgr2=0x7ff42400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.037+0000 7ff445400640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240052e0 0x7ff42400b9c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff43800a760 tx=0x7ff424005830 comp rx=0 tx=0).stop 2024-03-28T18:56:30.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.037+0000 7ff445400640 1 -- 172.21.15.27:0/760805729 shutdown_connections 2024-03-28T18:56:30.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.037+0000 7ff445400640 1 --2- 172.21.15.27:0/760805729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4240052e0 0x7ff42400b9c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.037+0000 7ff445400640 1 -- 172.21.15.27:0/760805729 >> 172.21.15.27:0/760805729 conn(0x7ff440066dd0 msgr2=0x7ff4400671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.038+0000 7ff445400640 1 -- 172.21.15.27:0/760805729 shutdown_connections 2024-03-28T18:56:30.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.038+0000 7ff445400640 1 -- 172.21.15.27:0/760805729 wait complete. 2024-03-28T18:56:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.039+0000 7ff445400640 1 Processor -- start 2024-03-28T18:56:30.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.039+0000 7ff445400640 1 -- start start 2024-03-28T18:56:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.040+0000 7ff445400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4240052e0 0x7ff44006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.040+0000 7ff445400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4400ae890 0x7ff4400c2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.040+0000 7ff445400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff440147020 0x7ff4401474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.040+0000 7ff445400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400c8ed0 con 0x7ff440147020 2024-03-28T18:56:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.040+0000 7ff445400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400c8d50 con 0x7ff4400ae890 2024-03-28T18:56:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.040+0000 7ff445400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400c9050 con 0x7ff4240052e0 2024-03-28T18:56:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff440147020 0x7ff4401474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff440147020 0x7ff4401474b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40414/0 (socket says 172.21.15.27:40414) 2024-03-28T18:56:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 -- 172.21.15.27:0/537038543 learned_addr learned my addr 172.21.15.27:0/537038543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43e800640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4240052e0 0x7ff44006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff435e00640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4400ae890 0x7ff4400c2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 -- 172.21.15.27:0/537038543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4240052e0 msgr2=0x7ff44006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4240052e0 0x7ff44006dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 -- 172.21.15.27:0/537038543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4400ae890 msgr2=0x7ff4400c2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4400ae890 0x7ff4400c2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 -- 172.21.15.27:0/537038543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44006c360 con 0x7ff440147020 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.041+0000 7ff43f200640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff440147020 0x7ff4401474b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff43802cb70 tx=0x7ff4400c4410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.042+0000 7ff437400640 1 -- 172.21.15.27:0/537038543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff438043070 con 0x7ff440147020 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.042+0000 7ff437400640 1 -- 172.21.15.27:0/537038543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff438004b60 con 0x7ff440147020 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.042+0000 7ff437400640 1 -- 172.21.15.27:0/537038543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff438039040 con 0x7ff440147020 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.042+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff440151090 con 0x7ff440147020 2024-03-28T18:56:30.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.042+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4400acf40 con 0x7ff440147020 2024-03-28T18:56:30.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.043+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4400aca50 con 0x7ff440147020 2024-03-28T18:56:30.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.044+0000 7ff437400640 1 -- 172.21.15.27:0/537038543 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff438046070 con 0x7ff440147020 2024-03-28T18:56:30.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.045+0000 7ff437400640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff41c06bbd0 0x7ff41c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.045+0000 7ff437400640 1 -- 172.21.15.27:0/537038543 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7ff4380ac1e0 con 0x7ff440147020 2024-03-28T18:56:30.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.045+0000 7ff43e800640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff41c06bbd0 0x7ff41c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.046+0000 7ff43e800640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff41c06bbd0 0x7ff41c06e090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff44014ce60 tx=0x7ff430008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:30.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.055+0000 7ff437400640 1 -- 172.21.15.27:0/537038543 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff43807a240 con 0x7ff440147020 2024-03-28T18:56:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.258+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff44004f400 con 0x7ff41c06bbd0 2024-03-28T18:56:30.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.289+0000 7ff437400640 1 -- 172.21.15.27:0/537038543 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490350 (secure 0 0 0) 0x7ff44004f400 con 0x7ff41c06bbd0 2024-03-28T18:56:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:30.299 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:56:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff41c06bbd0 msgr2=0x7ff41c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff41c06bbd0 0x7ff41c06e090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff44014ce60 tx=0x7ff430008040 comp rx=0 tx=0).stop 2024-03-28T18:56:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff440147020 msgr2=0x7ff4401474b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff440147020 0x7ff4401474b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff43802cb70 tx=0x7ff4400c4410 comp rx=0 tx=0).stop 2024-03-28T18:56:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 shutdown_connections 2024-03-28T18:56:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff41c06bbd0 0x7ff41c06e090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff440147020 0x7ff4401474b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4400ae890 0x7ff4400c2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 --2- 172.21.15.27:0/537038543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4240052e0 0x7ff44006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.297+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 >> 172.21.15.27:0/537038543 conn(0x7ff440066dd0 msgr2=0x7ff440152760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:30.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.298+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 shutdown_connections 2024-03-28T18:56:30.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.298+0000 7ff445400640 1 -- 172.21.15.27:0/537038543 wait complete. 2024-03-28T18:56:30.318 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-28T18:56:28.912012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597700,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":255,"num_write_kb":2450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219628,"kb_used_data":3464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755092,"statfs":{"total":773094113280,"available":772869214208,"internally_reserved":0,"allocated":3547136,"data_stored":2099936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005066"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164203+0000","last_change":"2024-03-28T18:55:45.603782+0000","last_active":"2024-03-28T18:56:16.164203+0000","last_peered":"2024-03-28T18:56:16.164203+0000","last_clean":"2024-03-28T18:56:16.164203+0000","last_became_active":"2024-03-28T18:55:45.603420+0000","last_became_peered":"2024-03-28T18:55:45.603420+0000","last_unstale":"2024-03-28T18:56:16.164203+0000","last_undegraded":"2024-03-28T18:56:16.164203+0000","last_fullsized":"2024-03-28T18:56:16.164203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373460+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:56:16.373460+0000","last_peered":"2024-03-28T18:56:16.373460+0000","last_clean":"2024-03-28T18:56:16.373460+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:56:16.373460+0000","last_undegraded":"2024-03-28T18:56:16.373460+0000","last_fullsized":"2024-03-28T18:56:16.373460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992310+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:56:15.992310+0000","last_peered":"2024-03-28T18:56:15.992310+0000","last_clean":"2024-03-28T18:56:15.992310+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:56:15.992310+0000","last_undegraded":"2024-03-28T18:56:15.992310+0000","last_fullsized":"2024-03-28T18:56:15.992310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989768+0000","last_change":"2024-03-28T18:55:44.602536+0000","last_active":"2024-03-28T18:56:15.989768+0000","last_peered":"2024-03-28T18:56:15.989768+0000","last_clean":"2024-03-28T18:56:15.989768+0000","last_became_active":"2024-03-28T18:55:44.602256+0000","last_became_peered":"2024-03-28T18:55:44.602256+0000","last_unstale":"2024-03-28T18:56:15.989768+0000","last_undegraded":"2024-03-28T18:56:15.989768+0000","last_fullsized":"2024-03-28T18:56:15.989768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994213+0000","last_change":"2024-03-28T18:55:45.625338+0000","last_active":"2024-03-28T18:56:15.994213+0000","last_peered":"2024-03-28T18:56:15.994213+0000","last_clean":"2024-03-28T18:56:15.994213+0000","last_became_active":"2024-03-28T18:55:45.625067+0000","last_became_peered":"2024-03-28T18:55:45.625067+0000","last_unstale":"2024-03-28T18:56:15.994213+0000","last_undegraded":"2024-03-28T18:56:15.994213+0000","last_fullsized":"2024-03-28T18:56:15.994213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991986+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:56:15.991986+0000","last_peered":"2024-03-28T18:56:15.991986+0000","last_clean":"2024-03-28T18:56:15.991986+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:56:15.991986+0000","last_undegraded":"2024-03-28T18:56:15.991986+0000","last_fullsized":"2024-03-28T18:56:15.991986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373257+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:56:16.373257+0000","last_peered":"2024-03-28T18:56:16.373257+0000","last_clean":"2024-03-28T18:56:16.373257+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:56:16.373257+0000","last_undegraded":"2024-03-28T18:56:16.373257+0000","last_fullsized":"2024-03-28T18:56:16.373257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164333+0000","last_change":"2024-03-28T18:55:44.598423+0000","last_active":"2024-03-28T18:56:16.164333+0000","last_peered":"2024-03-28T18:56:16.164333+0000","last_clean":"2024-03-28T18:56:16.164333+0000","last_became_active":"2024-03-28T18:55:44.598092+0000","last_became_peered":"2024-03-28T18:55:44.598092+0000","last_unstale":"2024-03-28T18:56:16.164333+0000","last_undegraded":"2024-03-28T18:56:16.164333+0000","last_fullsized":"2024-03-28T18:56:16.164333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.646619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178114+0000","last_change":"2024-03-28T18:55:45.608201+0000","last_active":"2024-03-28T18:56:16.178114+0000","last_peered":"2024-03-28T18:56:16.178114+0000","last_clean":"2024-03-28T18:56:16.178114+0000","last_became_active":"2024-03-28T18:55:45.607912+0000","last_became_peered":"2024-03-28T18:55:45.607912+0000","last_unstale":"2024-03-28T18:56:16.178114+0000","last_undegraded":"2024-03-28T18:56:16.178114+0000","last_fullsized":"2024-03-28T18:56:16.178114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178494+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:56:16.178494+0000","last_peered":"2024-03-28T18:56:16.178494+0000","last_clean":"2024-03-28T18:56:16.178494+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:56:16.178494+0000","last_undegraded":"2024-03-28T18:56:16.178494+0000","last_fullsized":"2024-03-28T18:56:16.178494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996247+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:56:15.996247+0000","last_peered":"2024-03-28T18:56:15.996247+0000","last_clean":"2024-03-28T18:56:15.996247+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:56:15.996247+0000","last_undegraded":"2024-03-28T18:56:15.996247+0000","last_fullsized":"2024-03-28T18:56:15.996247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164809+0000","last_change":"2024-03-28T18:55:44.604673+0000","last_active":"2024-03-28T18:56:16.164809+0000","last_peered":"2024-03-28T18:56:16.164809+0000","last_clean":"2024-03-28T18:56:16.164809+0000","last_became_active":"2024-03-28T18:55:44.603926+0000","last_became_peered":"2024-03-28T18:55:44.603926+0000","last_unstale":"2024-03-28T18:56:16.164809+0000","last_undegraded":"2024-03-28T18:56:16.164809+0000","last_fullsized":"2024-03-28T18:56:16.164809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995761+0000","last_change":"2024-03-28T18:55:45.610033+0000","last_active":"2024-03-28T18:56:15.995761+0000","last_peered":"2024-03-28T18:56:15.995761+0000","last_clean":"2024-03-28T18:56:15.995761+0000","last_became_active":"2024-03-28T18:55:45.609706+0000","last_became_peered":"2024-03-28T18:55:45.609706+0000","last_unstale":"2024-03-28T18:56:15.995761+0000","last_undegraded":"2024-03-28T18:56:15.995761+0000","last_fullsized":"2024-03-28T18:56:15.995761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178300+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:56:16.178300+0000","last_peered":"2024-03-28T18:56:16.178300+0000","last_clean":"2024-03-28T18:56:16.178300+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:56:16.178300+0000","last_undegraded":"2024-03-28T18:56:16.178300+0000","last_fullsized":"2024-03-28T18:56:16.178300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996865+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:56:15.996865+0000","last_peered":"2024-03-28T18:56:15.996865+0000","last_clean":"2024-03-28T18:56:15.996865+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:56:15.996865+0000","last_undegraded":"2024-03-28T18:56:15.996865+0000","last_fullsized":"2024-03-28T18:56:15.996865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164862+0000","last_change":"2024-03-28T18:55:44.606715+0000","last_active":"2024-03-28T18:56:16.164862+0000","last_peered":"2024-03-28T18:56:16.164862+0000","last_clean":"2024-03-28T18:56:16.164862+0000","last_became_active":"2024-03-28T18:55:44.606343+0000","last_became_peered":"2024-03-28T18:55:44.606343+0000","last_unstale":"2024-03-28T18:56:16.164862+0000","last_undegraded":"2024-03-28T18:56:16.164862+0000","last_fullsized":"2024-03-28T18:56:16.164862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:51.471450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995747+0000","last_change":"2024-03-28T18:55:45.623483+0000","last_active":"2024-03-28T18:56:15.995747+0000","last_peered":"2024-03-28T18:56:15.995747+0000","last_clean":"2024-03-28T18:56:15.995747+0000","last_became_active":"2024-03-28T18:55:45.622894+0000","last_became_peered":"2024-03-28T18:55:45.622894+0000","last_unstale":"2024-03-28T18:56:15.995747+0000","last_undegraded":"2024-03-28T18:56:15.995747+0000","last_fullsized":"2024-03-28T18:56:15.995747+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993916+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:56:15.993916+0000","last_peered":"2024-03-28T18:56:15.993916+0000","last_clean":"2024-03-28T18:56:15.993916+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:56:15.993916+0000","last_undegraded":"2024-03-28T18:56:15.993916+0000","last_fullsized":"2024-03-28T18:56:15.993916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177876+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:56:16.177876+0000","last_peered":"2024-03-28T18:56:16.177876+0000","last_clean":"2024-03-28T18:56:16.177876+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:56:16.177876+0000","last_undegraded":"2024-03-28T18:56:16.177876+0000","last_fullsized":"2024-03-28T18:56:16.177876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165877+0000","last_change":"2024-03-28T18:55:44.601415+0000","last_active":"2024-03-28T18:56:16.165877+0000","last_peered":"2024-03-28T18:56:16.165877+0000","last_clean":"2024-03-28T18:56:16.165877+0000","last_became_active":"2024-03-28T18:55:44.601124+0000","last_became_peered":"2024-03-28T18:55:44.601124+0000","last_unstale":"2024-03-28T18:56:16.165877+0000","last_undegraded":"2024-03-28T18:56:16.165877+0000","last_fullsized":"2024-03-28T18:56:16.165877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:30.767034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993126+0000","last_change":"2024-03-28T18:55:45.604795+0000","last_active":"2024-03-28T18:56:15.993126+0000","last_peered":"2024-03-28T18:56:15.993126+0000","last_clean":"2024-03-28T18:56:15.993126+0000","last_became_active":"2024-03-28T18:55:45.604432+0000","last_became_peered":"2024-03-28T18:55:45.604432+0000","last_unstale":"2024-03-28T18:56:15.993126+0000","last_undegraded":"2024-03-28T18:56:15.993126+0000","last_fullsized":"2024-03-28T18:56:15.993126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:37.638804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990541+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:56:15.990541+0000","last_peered":"2024-03-28T18:56:15.990541+0000","last_clean":"2024-03-28T18:56:15.990541+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:56:15.990541+0000","last_undegraded":"2024-03-28T18:56:15.990541+0000","last_fullsized":"2024-03-28T18:56:15.990541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995164+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:56:15.995164+0000","last_peered":"2024-03-28T18:56:15.995164+0000","last_clean":"2024-03-28T18:56:15.995164+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:56:15.995164+0000","last_undegraded":"2024-03-28T18:56:15.995164+0000","last_fullsized":"2024-03-28T18:56:15.995164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165844+0000","last_change":"2024-03-28T18:55:44.603859+0000","last_active":"2024-03-28T18:56:16.165844+0000","last_peered":"2024-03-28T18:56:16.165844+0000","last_clean":"2024-03-28T18:56:16.165844+0000","last_became_active":"2024-03-28T18:55:44.603540+0000","last_became_peered":"2024-03-28T18:55:44.603540+0000","last_unstale":"2024-03-28T18:56:16.165844+0000","last_undegraded":"2024-03-28T18:56:16.165844+0000","last_fullsized":"2024-03-28T18:56:16.165844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165791+0000","last_change":"2024-03-28T18:55:45.608142+0000","last_active":"2024-03-28T18:56:16.165791+0000","last_peered":"2024-03-28T18:56:16.165791+0000","last_clean":"2024-03-28T18:56:16.165791+0000","last_became_active":"2024-03-28T18:55:45.606117+0000","last_became_peered":"2024-03-28T18:55:45.606117+0000","last_unstale":"2024-03-28T18:56:16.165791+0000","last_undegraded":"2024-03-28T18:56:16.165791+0000","last_fullsized":"2024-03-28T18:56:16.165791+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989871+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:56:15.989871+0000","last_peered":"2024-03-28T18:56:15.989871+0000","last_clean":"2024-03-28T18:56:15.989871+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:56:15.989871+0000","last_undegraded":"2024-03-28T18:56:15.989871+0000","last_fullsized":"2024-03-28T18:56:15.989871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994533+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:56:15.994533+0000","last_peered":"2024-03-28T18:56:15.994533+0000","last_clean":"2024-03-28T18:56:15.994533+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:56:15.994533+0000","last_undegraded":"2024-03-28T18:56:15.994533+0000","last_fullsized":"2024-03-28T18:56:15.994533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996249+0000","last_change":"2024-03-28T18:55:44.606690+0000","last_active":"2024-03-28T18:56:15.996249+0000","last_peered":"2024-03-28T18:56:15.996249+0000","last_clean":"2024-03-28T18:56:15.996249+0000","last_became_active":"2024-03-28T18:55:44.606421+0000","last_became_peered":"2024-03-28T18:55:44.606421+0000","last_unstale":"2024-03-28T18:56:15.996249+0000","last_undegraded":"2024-03-28T18:56:15.996249+0000","last_fullsized":"2024-03-28T18:56:15.996249+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997284+0000","last_change":"2024-03-28T18:55:45.610580+0000","last_active":"2024-03-28T18:56:15.997284+0000","last_peered":"2024-03-28T18:56:15.997284+0000","last_clean":"2024-03-28T18:56:15.997284+0000","last_became_active":"2024-03-28T18:55:45.610316+0000","last_became_peered":"2024-03-28T18:55:45.610316+0000","last_unstale":"2024-03-28T18:56:15.997284+0000","last_undegraded":"2024-03-28T18:56:15.997284+0000","last_fullsized":"2024-03-28T18:56:15.997284+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165194+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:56:16.165194+0000","last_peered":"2024-03-28T18:56:16.165194+0000","last_clean":"2024-03-28T18:56:16.165194+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:56:16.165194+0000","last_undegraded":"2024-03-28T18:56:16.165194+0000","last_fullsized":"2024-03-28T18:56:16.165194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177321+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:56:16.177321+0000","last_peered":"2024-03-28T18:56:16.177321+0000","last_clean":"2024-03-28T18:56:16.177321+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:56:16.177321+0000","last_undegraded":"2024-03-28T18:56:16.177321+0000","last_fullsized":"2024-03-28T18:56:16.177321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166077+0000","last_change":"2024-03-28T18:55:44.600523+0000","last_active":"2024-03-28T18:56:16.166077+0000","last_peered":"2024-03-28T18:56:16.166077+0000","last_clean":"2024-03-28T18:56:16.166077+0000","last_became_active":"2024-03-28T18:55:44.597550+0000","last_became_peered":"2024-03-28T18:55:44.597550+0000","last_unstale":"2024-03-28T18:56:16.166077+0000","last_undegraded":"2024-03-28T18:56:16.166077+0000","last_fullsized":"2024-03-28T18:56:16.166077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989484+0000","last_change":"2024-03-28T18:55:45.619798+0000","last_active":"2024-03-28T18:56:15.989484+0000","last_peered":"2024-03-28T18:56:15.989484+0000","last_clean":"2024-03-28T18:56:15.989484+0000","last_became_active":"2024-03-28T18:55:45.616517+0000","last_became_peered":"2024-03-28T18:55:45.616517+0000","last_unstale":"2024-03-28T18:56:15.989484+0000","last_undegraded":"2024-03-28T18:56:15.989484+0000","last_fullsized":"2024-03-28T18:56:15.989484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.374003+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:56:16.374003+0000","last_peered":"2024-03-28T18:56:16.374003+0000","last_clean":"2024-03-28T18:56:16.374003+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:56:16.374003+0000","last_undegraded":"2024-03-28T18:56:16.374003+0000","last_fullsized":"2024-03-28T18:56:16.374003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994929+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:56:15.994929+0000","last_peered":"2024-03-28T18:56:15.994929+0000","last_clean":"2024-03-28T18:56:15.994929+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:56:15.994929+0000","last_undegraded":"2024-03-28T18:56:15.994929+0000","last_fullsized":"2024-03-28T18:56:15.994929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164852+0000","last_change":"2024-03-28T18:55:44.606498+0000","last_active":"2024-03-28T18:56:16.164852+0000","last_peered":"2024-03-28T18:56:16.164852+0000","last_clean":"2024-03-28T18:56:16.164852+0000","last_became_active":"2024-03-28T18:55:44.606098+0000","last_became_peered":"2024-03-28T18:55:44.606098+0000","last_unstale":"2024-03-28T18:56:16.164852+0000","last_undegraded":"2024-03-28T18:56:16.164852+0000","last_fullsized":"2024-03-28T18:56:16.164852+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164035+0000","last_change":"2024-03-28T18:55:45.617483+0000","last_active":"2024-03-28T18:56:16.164035+0000","last_peered":"2024-03-28T18:56:16.164035+0000","last_clean":"2024-03-28T18:56:16.164035+0000","last_became_active":"2024-03-28T18:55:45.617169+0000","last_became_peered":"2024-03-28T18:55:45.617169+0000","last_unstale":"2024-03-28T18:56:16.164035+0000","last_undegraded":"2024-03-28T18:56:16.164035+0000","last_fullsized":"2024-03-28T18:56:16.164035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992182+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:56:15.992182+0000","last_peered":"2024-03-28T18:56:15.992182+0000","last_clean":"2024-03-28T18:56:15.992182+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:56:15.992182+0000","last_undegraded":"2024-03-28T18:56:15.992182+0000","last_fullsized":"2024-03-28T18:56:15.992182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996404+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:56:15.996404+0000","last_peered":"2024-03-28T18:56:15.996404+0000","last_clean":"2024-03-28T18:56:15.996404+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:56:15.996404+0000","last_undegraded":"2024-03-28T18:56:15.996404+0000","last_fullsized":"2024-03-28T18:56:15.996404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989501+0000","last_change":"2024-03-28T18:55:44.601836+0000","last_active":"2024-03-28T18:56:15.989501+0000","last_peered":"2024-03-28T18:56:15.989501+0000","last_clean":"2024-03-28T18:56:15.989501+0000","last_became_active":"2024-03-28T18:55:44.601552+0000","last_became_peered":"2024-03-28T18:55:44.601552+0000","last_unstale":"2024-03-28T18:56:15.989501+0000","last_undegraded":"2024-03-28T18:56:15.989501+0000","last_fullsized":"2024-03-28T18:56:15.989501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993178+0000","last_change":"2024-03-28T18:55:45.601801+0000","last_active":"2024-03-28T18:56:15.993178+0000","last_peered":"2024-03-28T18:56:15.993178+0000","last_clean":"2024-03-28T18:56:15.993178+0000","last_became_active":"2024-03-28T18:55:45.601459+0000","last_became_peered":"2024-03-28T18:55:45.601459+0000","last_unstale":"2024-03-28T18:56:15.993178+0000","last_undegraded":"2024-03-28T18:56:15.993178+0000","last_fullsized":"2024-03-28T18:56:15.993178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:43.033170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995963+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:56:15.995963+0000","last_peered":"2024-03-28T18:56:15.995963+0000","last_clean":"2024-03-28T18:56:15.995963+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:56:15.995963+0000","last_undegraded":"2024-03-28T18:56:15.995963+0000","last_fullsized":"2024-03-28T18:56:15.995963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178676+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:56:16.178676+0000","last_peered":"2024-03-28T18:56:16.178676+0000","last_clean":"2024-03-28T18:56:16.178676+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:56:16.178676+0000","last_undegraded":"2024-03-28T18:56:16.178676+0000","last_fullsized":"2024-03-28T18:56:16.178676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373043+0000","last_change":"2024-03-28T18:55:44.611148+0000","last_active":"2024-03-28T18:56:16.373043+0000","last_peered":"2024-03-28T18:56:16.373043+0000","last_clean":"2024-03-28T18:56:16.373043+0000","last_became_active":"2024-03-28T18:55:44.610835+0000","last_became_peered":"2024-03-28T18:55:44.610835+0000","last_unstale":"2024-03-28T18:56:16.373043+0000","last_undegraded":"2024-03-28T18:56:16.373043+0000","last_fullsized":"2024-03-28T18:56:16.373043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373399+0000","last_change":"2024-03-28T18:55:45.623672+0000","last_active":"2024-03-28T18:56:16.373399+0000","last_peered":"2024-03-28T18:56:16.373399+0000","last_clean":"2024-03-28T18:56:16.373399+0000","last_became_active":"2024-03-28T18:55:45.622568+0000","last_became_peered":"2024-03-28T18:55:45.622568+0000","last_unstale":"2024-03-28T18:56:16.373399+0000","last_undegraded":"2024-03-28T18:56:16.373399+0000","last_fullsized":"2024-03-28T18:56:16.373399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997025+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:56:15.997025+0000","last_peered":"2024-03-28T18:56:15.997025+0000","last_clean":"2024-03-28T18:56:15.997025+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:56:15.997025+0000","last_undegraded":"2024-03-28T18:56:15.997025+0000","last_fullsized":"2024-03-28T18:56:15.997025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996010+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:56:15.996010+0000","last_peered":"2024-03-28T18:56:15.996010+0000","last_clean":"2024-03-28T18:56:15.996010+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:56:15.996010+0000","last_undegraded":"2024-03-28T18:56:15.996010+0000","last_fullsized":"2024-03-28T18:56:15.996010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994806+0000","last_change":"2024-03-28T18:55:44.604077+0000","last_active":"2024-03-28T18:56:15.994806+0000","last_peered":"2024-03-28T18:56:15.994806+0000","last_clean":"2024-03-28T18:56:15.994806+0000","last_became_active":"2024-03-28T18:55:44.603773+0000","last_became_peered":"2024-03-28T18:55:44.603773+0000","last_unstale":"2024-03-28T18:56:15.994806+0000","last_undegraded":"2024-03-28T18:56:15.994806+0000","last_fullsized":"2024-03-28T18:56:15.994806+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372682+0000","last_change":"2024-03-28T18:55:45.615444+0000","last_active":"2024-03-28T18:56:16.372682+0000","last_peered":"2024-03-28T18:56:16.372682+0000","last_clean":"2024-03-28T18:56:16.372682+0000","last_became_active":"2024-03-28T18:55:45.615002+0000","last_became_peered":"2024-03-28T18:55:45.615002+0000","last_unstale":"2024-03-28T18:56:16.372682+0000","last_undegraded":"2024-03-28T18:56:16.372682+0000","last_fullsized":"2024-03-28T18:56:16.372682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166059+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:56:16.166059+0000","last_peered":"2024-03-28T18:56:16.166059+0000","last_clean":"2024-03-28T18:56:16.166059+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:56:16.166059+0000","last_undegraded":"2024-03-28T18:56:16.166059+0000","last_fullsized":"2024-03-28T18:56:16.166059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996064+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:56:15.996064+0000","last_peered":"2024-03-28T18:56:15.996064+0000","last_clean":"2024-03-28T18:56:15.996064+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:56:15.996064+0000","last_undegraded":"2024-03-28T18:56:15.996064+0000","last_fullsized":"2024-03-28T18:56:15.996064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995899+0000","last_change":"2024-03-28T18:55:44.590703+0000","last_active":"2024-03-28T18:56:15.995899+0000","last_peered":"2024-03-28T18:56:15.995899+0000","last_clean":"2024-03-28T18:56:15.995899+0000","last_became_active":"2024-03-28T18:55:44.590408+0000","last_became_peered":"2024-03-28T18:55:44.590408+0000","last_unstale":"2024-03-28T18:56:15.995899+0000","last_undegraded":"2024-03-28T18:56:15.995899+0000","last_fullsized":"2024-03-28T18:56:15.995899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992984+0000","last_change":"2024-03-28T18:55:45.609795+0000","last_active":"2024-03-28T18:56:15.992984+0000","last_peered":"2024-03-28T18:56:15.992984+0000","last_clean":"2024-03-28T18:56:15.992984+0000","last_became_active":"2024-03-28T18:55:45.603695+0000","last_became_peered":"2024-03-28T18:55:45.603695+0000","last_unstale":"2024-03-28T18:56:15.992984+0000","last_undegraded":"2024-03-28T18:56:15.992984+0000","last_fullsized":"2024-03-28T18:56:15.992984+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:49.173555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994287+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:56:15.994287+0000","last_peered":"2024-03-28T18:56:15.994287+0000","last_clean":"2024-03-28T18:56:15.994287+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:56:15.994287+0000","last_undegraded":"2024-03-28T18:56:15.994287+0000","last_fullsized":"2024-03-28T18:56:15.994287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993148+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:56:15.993148+0000","last_peered":"2024-03-28T18:56:15.993148+0000","last_clean":"2024-03-28T18:56:15.993148+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:56:15.993148+0000","last_undegraded":"2024-03-28T18:56:15.993148+0000","last_fullsized":"2024-03-28T18:56:15.993148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996282+0000","last_change":"2024-03-28T18:55:44.612172+0000","last_active":"2024-03-28T18:56:15.996282+0000","last_peered":"2024-03-28T18:56:15.996282+0000","last_clean":"2024-03-28T18:56:15.996282+0000","last_became_active":"2024-03-28T18:55:44.611859+0000","last_became_peered":"2024-03-28T18:55:44.611859+0000","last_unstale":"2024-03-28T18:56:15.996282+0000","last_undegraded":"2024-03-28T18:56:15.996282+0000","last_fullsized":"2024-03-28T18:56:15.996282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997500+0000","last_change":"2024-03-28T18:55:45.619723+0000","last_active":"2024-03-28T18:56:15.997500+0000","last_peered":"2024-03-28T18:56:15.997500+0000","last_clean":"2024-03-28T18:56:15.997500+0000","last_became_active":"2024-03-28T18:55:45.619437+0000","last_became_peered":"2024-03-28T18:55:45.619437+0000","last_unstale":"2024-03-28T18:56:15.997500+0000","last_undegraded":"2024-03-28T18:56:15.997500+0000","last_fullsized":"2024-03-28T18:56:15.997500+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372410+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:56:16.372410+0000","last_peered":"2024-03-28T18:56:16.372410+0000","last_clean":"2024-03-28T18:56:16.372410+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:56:16.372410+0000","last_undegraded":"2024-03-28T18:56:16.372410+0000","last_fullsized":"2024-03-28T18:56:16.372410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177331+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:56:16.177331+0000","last_peered":"2024-03-28T18:56:16.177331+0000","last_clean":"2024-03-28T18:56:16.177331+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:56:16.177331+0000","last_undegraded":"2024-03-28T18:56:16.177331+0000","last_fullsized":"2024-03-28T18:56:16.177331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"24'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373710+0000","last_change":"2024-03-28T18:55:44.592064+0000","last_active":"2024-03-28T18:56:16.373710+0000","last_peered":"2024-03-28T18:56:16.373710+0000","last_clean":"2024-03-28T18:56:16.373710+0000","last_became_active":"2024-03-28T18:55:44.591263+0000","last_became_peered":"2024-03-28T18:55:44.591263+0000","last_unstale":"2024-03-28T18:56:16.373710+0000","last_undegraded":"2024-03-28T18:56:16.373710+0000","last_fullsized":"2024-03-28T18:56:16.373710+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177095+0000","last_change":"2024-03-28T18:55:45.614820+0000","last_active":"2024-03-28T18:56:16.177095+0000","last_peered":"2024-03-28T18:56:16.177095+0000","last_clean":"2024-03-28T18:56:16.177095+0000","last_became_active":"2024-03-28T18:55:45.614517+0000","last_became_peered":"2024-03-28T18:55:45.614517+0000","last_unstale":"2024-03-28T18:56:16.177095+0000","last_undegraded":"2024-03-28T18:56:16.177095+0000","last_fullsized":"2024-03-28T18:56:16.177095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990462+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:56:15.990462+0000","last_peered":"2024-03-28T18:56:15.990462+0000","last_clean":"2024-03-28T18:56:15.990462+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:56:15.990462+0000","last_undegraded":"2024-03-28T18:56:15.990462+0000","last_fullsized":"2024-03-28T18:56:15.990462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995955+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:56:15.995955+0000","last_peered":"2024-03-28T18:56:15.995955+0000","last_clean":"2024-03-28T18:56:15.995955+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:56:15.995955+0000","last_undegraded":"2024-03-28T18:56:15.995955+0000","last_fullsized":"2024-03-28T18:56:15.995955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177684+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:56:16.177684+0000","last_peered":"2024-03-28T18:56:16.177684+0000","last_clean":"2024-03-28T18:56:16.177684+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:56:16.177684+0000","last_undegraded":"2024-03-28T18:56:16.177684+0000","last_fullsized":"2024-03-28T18:56:16.177684+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993499+0000","last_change":"2024-03-28T18:55:44.602097+0000","last_active":"2024-03-28T18:56:15.993499+0000","last_peered":"2024-03-28T18:56:15.993499+0000","last_clean":"2024-03-28T18:56:15.993499+0000","last_became_active":"2024-03-28T18:55:44.601769+0000","last_became_peered":"2024-03-28T18:55:44.601769+0000","last_unstale":"2024-03-28T18:56:15.993499+0000","last_undegraded":"2024-03-28T18:56:15.993499+0000","last_fullsized":"2024-03-28T18:56:15.993499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993296+0000","last_change":"2024-03-28T18:55:45.616485+0000","last_active":"2024-03-28T18:56:15.993296+0000","last_peered":"2024-03-28T18:56:15.993296+0000","last_clean":"2024-03-28T18:56:15.993296+0000","last_became_active":"2024-03-28T18:55:45.615813+0000","last_became_peered":"2024-03-28T18:55:45.615813+0000","last_unstale":"2024-03-28T18:56:15.993296+0000","last_undegraded":"2024-03-28T18:56:15.993296+0000","last_fullsized":"2024-03-28T18:56:15.993296+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:21.227281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178319+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:56:16.178319+0000","last_peered":"2024-03-28T18:56:16.178319+0000","last_clean":"2024-03-28T18:56:16.178319+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:56:16.178319+0000","last_undegraded":"2024-03-28T18:56:16.178319+0000","last_fullsized":"2024-03-28T18:56:16.178319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993997+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:56:15.993997+0000","last_peered":"2024-03-28T18:56:15.993997+0000","last_clean":"2024-03-28T18:56:15.993997+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:56:15.993997+0000","last_undegraded":"2024-03-28T18:56:15.993997+0000","last_fullsized":"2024-03-28T18:56:15.993997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993504+0000","last_change":"2024-03-28T18:55:44.607744+0000","last_active":"2024-03-28T18:56:15.993504+0000","last_peered":"2024-03-28T18:56:15.993504+0000","last_clean":"2024-03-28T18:56:15.993504+0000","last_became_active":"2024-03-28T18:55:44.606993+0000","last_became_peered":"2024-03-28T18:55:44.606993+0000","last_unstale":"2024-03-28T18:56:15.993504+0000","last_undegraded":"2024-03-28T18:56:15.993504+0000","last_fullsized":"2024-03-28T18:56:15.993504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993475+0000","last_change":"2024-03-28T18:55:45.606307+0000","last_active":"2024-03-28T18:56:15.993475+0000","last_peered":"2024-03-28T18:56:15.993475+0000","last_clean":"2024-03-28T18:56:15.993475+0000","last_became_active":"2024-03-28T18:55:45.605980+0000","last_became_peered":"2024-03-28T18:55:45.605980+0000","last_unstale":"2024-03-28T18:56:15.993475+0000","last_undegraded":"2024-03-28T18:56:15.993475+0000","last_fullsized":"2024-03-28T18:56:15.993475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:52.485966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996297+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:56:15.996297+0000","last_peered":"2024-03-28T18:56:15.996297+0000","last_clean":"2024-03-28T18:56:15.996297+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:56:15.996297+0000","last_undegraded":"2024-03-28T18:56:15.996297+0000","last_fullsized":"2024-03-28T18:56:15.996297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996334+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:56:15.996334+0000","last_peered":"2024-03-28T18:56:15.996334+0000","last_clean":"2024-03-28T18:56:15.996334+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:56:15.996334+0000","last_undegraded":"2024-03-28T18:56:15.996334+0000","last_fullsized":"2024-03-28T18:56:15.996334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373024+0000","last_change":"2024-03-28T18:55:44.607323+0000","last_active":"2024-03-28T18:56:16.373024+0000","last_peered":"2024-03-28T18:56:16.373024+0000","last_clean":"2024-03-28T18:56:16.373024+0000","last_became_active":"2024-03-28T18:55:44.606579+0000","last_became_peered":"2024-03-28T18:55:44.606579+0000","last_unstale":"2024-03-28T18:56:16.373024+0000","last_undegraded":"2024-03-28T18:56:16.373024+0000","last_fullsized":"2024-03-28T18:56:16.373024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176822+0000","last_change":"2024-03-28T18:55:45.622760+0000","last_active":"2024-03-28T18:56:16.176822+0000","last_peered":"2024-03-28T18:56:16.176822+0000","last_clean":"2024-03-28T18:56:16.176822+0000","last_became_active":"2024-03-28T18:55:45.622353+0000","last_became_peered":"2024-03-28T18:55:45.622353+0000","last_unstale":"2024-03-28T18:56:16.176822+0000","last_undegraded":"2024-03-28T18:56:16.176822+0000","last_fullsized":"2024-03-28T18:56:16.176822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:04.349689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179332+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:56:16.179332+0000","last_peered":"2024-03-28T18:56:16.179332+0000","last_clean":"2024-03-28T18:56:16.179332+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:56:16.179332+0000","last_undegraded":"2024-03-28T18:56:16.179332+0000","last_fullsized":"2024-03-28T18:56:16.179332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993245+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:56:15.993245+0000","last_peered":"2024-03-28T18:56:15.993245+0000","last_clean":"2024-03-28T18:56:15.993245+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:56:15.993245+0000","last_undegraded":"2024-03-28T18:56:15.993245+0000","last_fullsized":"2024-03-28T18:56:15.993245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994698+0000","last_change":"2024-03-28T18:55:44.607237+0000","last_active":"2024-03-28T18:56:15.994698+0000","last_peered":"2024-03-28T18:56:15.994698+0000","last_clean":"2024-03-28T18:56:15.994698+0000","last_became_active":"2024-03-28T18:55:44.606960+0000","last_became_peered":"2024-03-28T18:55:44.606960+0000","last_unstale":"2024-03-28T18:56:15.994698+0000","last_undegraded":"2024-03-28T18:56:15.994698+0000","last_fullsized":"2024-03-28T18:56:15.994698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993370+0000","last_change":"2024-03-28T18:55:45.612992+0000","last_active":"2024-03-28T18:56:15.993370+0000","last_peered":"2024-03-28T18:56:15.993370+0000","last_clean":"2024-03-28T18:56:15.993370+0000","last_became_active":"2024-03-28T18:55:45.612125+0000","last_became_peered":"2024-03-28T18:55:45.612125+0000","last_unstale":"2024-03-28T18:56:15.993370+0000","last_undegraded":"2024-03-28T18:56:15.993370+0000","last_fullsized":"2024-03-28T18:56:15.993370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:35.898911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179159+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:56:16.179159+0000","last_peered":"2024-03-28T18:56:16.179159+0000","last_clean":"2024-03-28T18:56:16.179159+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:56:16.179159+0000","last_undegraded":"2024-03-28T18:56:16.179159+0000","last_fullsized":"2024-03-28T18:56:16.179159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166759+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:56:16.166759+0000","last_peered":"2024-03-28T18:56:16.166759+0000","last_clean":"2024-03-28T18:56:16.166759+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:56:16.166759+0000","last_undegraded":"2024-03-28T18:56:16.166759+0000","last_fullsized":"2024-03-28T18:56:16.166759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164552+0000","last_change":"2024-03-28T18:55:44.607218+0000","last_active":"2024-03-28T18:56:16.164552+0000","last_peered":"2024-03-28T18:56:16.164552+0000","last_clean":"2024-03-28T18:56:16.164552+0000","last_became_active":"2024-03-28T18:55:44.606852+0000","last_became_peered":"2024-03-28T18:55:44.606852+0000","last_unstale":"2024-03-28T18:56:16.164552+0000","last_undegraded":"2024-03-28T18:56:16.164552+0000","last_fullsized":"2024-03-28T18:56:16.164552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:36.624289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992021+0000","last_change":"2024-03-28T18:55:45.618919+0000","last_active":"2024-03-28T18:56:15.992021+0000","last_peered":"2024-03-28T18:56:15.992021+0000","last_clean":"2024-03-28T18:56:15.992021+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:56:15.992021+0000","last_undegraded":"2024-03-28T18:56:15.992021+0000","last_fullsized":"2024-03-28T18:56:15.992021+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:29.407508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178011+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:56:16.178011+0000","last_peered":"2024-03-28T18:56:16.178011+0000","last_clean":"2024-03-28T18:56:16.178011+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:56:16.178011+0000","last_undegraded":"2024-03-28T18:56:16.178011+0000","last_fullsized":"2024-03-28T18:56:16.178011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993501+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:56:15.993501+0000","last_peered":"2024-03-28T18:56:15.993501+0000","last_clean":"2024-03-28T18:56:15.993501+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:56:15.993501+0000","last_undegraded":"2024-03-28T18:56:15.993501+0000","last_fullsized":"2024-03-28T18:56:15.993501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.374029+0000","last_change":"2024-03-28T18:55:44.598881+0000","last_active":"2024-03-28T18:56:16.374029+0000","last_peered":"2024-03-28T18:56:16.374029+0000","last_clean":"2024-03-28T18:56:16.374029+0000","last_became_active":"2024-03-28T18:55:44.598532+0000","last_became_peered":"2024-03-28T18:55:44.598532+0000","last_unstale":"2024-03-28T18:56:16.374029+0000","last_undegraded":"2024-03-28T18:56:16.374029+0000","last_fullsized":"2024-03-28T18:56:16.374029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:44.004611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996863+0000","last_change":"2024-03-28T18:55:45.612572+0000","last_active":"2024-03-28T18:56:15.996863+0000","last_peered":"2024-03-28T18:56:15.996863+0000","last_clean":"2024-03-28T18:56:15.996863+0000","last_became_active":"2024-03-28T18:55:45.611844+0000","last_became_peered":"2024-03-28T18:55:45.611844+0000","last_unstale":"2024-03-28T18:56:15.996863+0000","last_undegraded":"2024-03-28T18:56:15.996863+0000","last_fullsized":"2024-03-28T18:56:15.996863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992765+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:56:15.992765+0000","last_peered":"2024-03-28T18:56:15.992765+0000","last_clean":"2024-03-28T18:56:15.992765+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:56:15.992765+0000","last_undegraded":"2024-03-28T18:56:15.992765+0000","last_fullsized":"2024-03-28T18:56:15.992765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993659+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:56:15.993659+0000","last_peered":"2024-03-28T18:56:15.993659+0000","last_clean":"2024-03-28T18:56:15.993659+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:56:15.993659+0000","last_undegraded":"2024-03-28T18:56:15.993659+0000","last_fullsized":"2024-03-28T18:56:15.993659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992080+0000","last_change":"2024-03-28T18:55:44.609644+0000","last_active":"2024-03-28T18:56:15.992080+0000","last_peered":"2024-03-28T18:56:15.992080+0000","last_clean":"2024-03-28T18:56:15.992080+0000","last_became_active":"2024-03-28T18:55:44.609317+0000","last_became_peered":"2024-03-28T18:55:44.609317+0000","last_unstale":"2024-03-28T18:56:15.992080+0000","last_undegraded":"2024-03-28T18:56:15.992080+0000","last_fullsized":"2024-03-28T18:56:15.992080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:37.662504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996075+0000","last_change":"2024-03-28T18:55:45.621708+0000","last_active":"2024-03-28T18:56:15.996075+0000","last_peered":"2024-03-28T18:56:15.996075+0000","last_clean":"2024-03-28T18:56:15.996075+0000","last_became_active":"2024-03-28T18:55:45.621366+0000","last_became_peered":"2024-03-28T18:55:45.621366+0000","last_unstale":"2024-03-28T18:56:15.996075+0000","last_undegraded":"2024-03-28T18:56:15.996075+0000","last_fullsized":"2024-03-28T18:56:15.996075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177583+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:56:16.177583+0000","last_peered":"2024-03-28T18:56:16.177583+0000","last_clean":"2024-03-28T18:56:16.177583+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:56:16.177583+0000","last_undegraded":"2024-03-28T18:56:16.177583+0000","last_fullsized":"2024-03-28T18:56:16.177583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996044+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:56:15.996044+0000","last_peered":"2024-03-28T18:56:15.996044+0000","last_clean":"2024-03-28T18:56:15.996044+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:56:15.996044+0000","last_undegraded":"2024-03-28T18:56:15.996044+0000","last_fullsized":"2024-03-28T18:56:15.996044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"26'9","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:20.908069+0000","last_change":"2024-03-28T18:55:44.608949+0000","last_active":"2024-03-28T18:56:20.908069+0000","last_peered":"2024-03-28T18:56:20.908069+0000","last_clean":"2024-03-28T18:56:20.908069+0000","last_became_active":"2024-03-28T18:55:44.608657+0000","last_became_peered":"2024-03-28T18:55:44.608657+0000","last_unstale":"2024-03-28T18:56:20.908069+0000","last_undegraded":"2024-03-28T18:56:20.908069+0000","last_fullsized":"2024-03-28T18:56:20.908069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:04.799314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990894+0000","last_change":"2024-03-28T18:55:45.610686+0000","last_active":"2024-03-28T18:56:15.990894+0000","last_peered":"2024-03-28T18:56:15.990894+0000","last_clean":"2024-03-28T18:56:15.990894+0000","last_became_active":"2024-03-28T18:55:45.610212+0000","last_became_peered":"2024-03-28T18:55:45.610212+0000","last_unstale":"2024-03-28T18:56:15.990894+0000","last_undegraded":"2024-03-28T18:56:15.990894+0000","last_fullsized":"2024-03-28T18:56:15.990894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372769+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:56:16.372769+0000","last_peered":"2024-03-28T18:56:16.372769+0000","last_clean":"2024-03-28T18:56:16.372769+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:56:16.372769+0000","last_undegraded":"2024-03-28T18:56:16.372769+0000","last_fullsized":"2024-03-28T18:56:16.372769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373687+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:56:16.373687+0000","last_peered":"2024-03-28T18:56:16.373687+0000","last_clean":"2024-03-28T18:56:16.373687+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:56:16.373687+0000","last_undegraded":"2024-03-28T18:56:16.373687+0000","last_fullsized":"2024-03-28T18:56:16.373687+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"23'3","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996740+0000","last_change":"2024-03-28T18:55:44.602762+0000","last_active":"2024-03-28T18:56:15.996740+0000","last_peered":"2024-03-28T18:56:15.996740+0000","last_clean":"2024-03-28T18:56:15.996740+0000","last_became_active":"2024-03-28T18:55:44.602424+0000","last_became_peered":"2024-03-28T18:55:44.602424+0000","last_unstale":"2024-03-28T18:56:15.996740+0000","last_undegraded":"2024-03-28T18:56:15.996740+0000","last_fullsized":"2024-03-28T18:56:15.996740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990431+0000","last_change":"2024-03-28T18:55:45.601277+0000","last_active":"2024-03-28T18:56:15.990431+0000","last_peered":"2024-03-28T18:56:15.990431+0000","last_clean":"2024-03-28T18:56:15.990431+0000","last_became_active":"2024-03-28T18:55:45.600945+0000","last_became_peered":"2024-03-28T18:55:45.600945+0000","last_unstale":"2024-03-28T18:56:15.990431+0000","last_undegraded":"2024-03-28T18:56:15.990431+0000","last_fullsized":"2024-03-28T18:56:15.990431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178326+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:56:16.178326+0000","last_peered":"2024-03-28T18:56:16.178326+0000","last_clean":"2024-03-28T18:56:16.178326+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:56:16.178326+0000","last_undegraded":"2024-03-28T18:56:16.178326+0000","last_fullsized":"2024-03-28T18:56:16.178326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993822+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:56:15.993822+0000","last_peered":"2024-03-28T18:56:15.993822+0000","last_clean":"2024-03-28T18:56:15.993822+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:56:15.993822+0000","last_undegraded":"2024-03-28T18:56:15.993822+0000","last_fullsized":"2024-03-28T18:56:15.993822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994697+0000","last_change":"2024-03-28T18:55:44.599432+0000","last_active":"2024-03-28T18:56:15.994697+0000","last_peered":"2024-03-28T18:56:15.994697+0000","last_clean":"2024-03-28T18:56:15.994697+0000","last_became_active":"2024-03-28T18:55:44.599123+0000","last_became_peered":"2024-03-28T18:55:44.599123+0000","last_unstale":"2024-03-28T18:56:15.994697+0000","last_undegraded":"2024-03-28T18:56:15.994697+0000","last_fullsized":"2024-03-28T18:56:15.994697+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994904+0000","last_change":"2024-03-28T18:55:45.624311+0000","last_active":"2024-03-28T18:56:15.994904+0000","last_peered":"2024-03-28T18:56:15.994904+0000","last_clean":"2024-03-28T18:56:15.994904+0000","last_became_active":"2024-03-28T18:55:45.624031+0000","last_became_peered":"2024-03-28T18:55:45.624031+0000","last_unstale":"2024-03-28T18:56:15.994904+0000","last_undegraded":"2024-03-28T18:56:15.994904+0000","last_fullsized":"2024-03-28T18:56:15.994904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166129+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:56:16.166129+0000","last_peered":"2024-03-28T18:56:16.166129+0000","last_clean":"2024-03-28T18:56:16.166129+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:56:16.166129+0000","last_undegraded":"2024-03-28T18:56:16.166129+0000","last_fullsized":"2024-03-28T18:56:16.166129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996267+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:56:15.996267+0000","last_peered":"2024-03-28T18:56:15.996267+0000","last_clean":"2024-03-28T18:56:15.996267+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:56:15.996267+0000","last_undegraded":"2024-03-28T18:56:15.996267+0000","last_fullsized":"2024-03-28T18:56:15.996267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165151+0000","last_change":"2024-03-28T18:55:44.593567+0000","last_active":"2024-03-28T18:56:16.165151+0000","last_peered":"2024-03-28T18:56:16.165151+0000","last_clean":"2024-03-28T18:56:16.165151+0000","last_became_active":"2024-03-28T18:55:44.593267+0000","last_became_peered":"2024-03-28T18:55:44.593267+0000","last_unstale":"2024-03-28T18:56:16.165151+0000","last_undegraded":"2024-03-28T18:56:16.165151+0000","last_fullsized":"2024-03-28T18:56:16.165151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993212+0000","last_change":"2024-03-28T18:55:45.598713+0000","last_active":"2024-03-28T18:56:15.993212+0000","last_peered":"2024-03-28T18:56:15.993212+0000","last_clean":"2024-03-28T18:56:15.993212+0000","last_became_active":"2024-03-28T18:55:45.597726+0000","last_became_peered":"2024-03-28T18:55:45.597726+0000","last_unstale":"2024-03-28T18:56:15.993212+0000","last_undegraded":"2024-03-28T18:56:15.993212+0000","last_fullsized":"2024-03-28T18:56:15.993212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:48.255745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373226+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:56:16.373226+0000","last_peered":"2024-03-28T18:56:16.373226+0000","last_clean":"2024-03-28T18:56:16.373226+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:56:16.373226+0000","last_undegraded":"2024-03-28T18:56:16.373226+0000","last_fullsized":"2024-03-28T18:56:16.373226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997012+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:56:15.997012+0000","last_peered":"2024-03-28T18:56:15.997012+0000","last_clean":"2024-03-28T18:56:15.997012+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:56:15.997012+0000","last_undegraded":"2024-03-28T18:56:15.997012+0000","last_fullsized":"2024-03-28T18:56:15.997012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"23'8","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989655+0000","last_change":"2024-03-28T18:55:44.610673+0000","last_active":"2024-03-28T18:56:15.989655+0000","last_peered":"2024-03-28T18:56:15.989655+0000","last_clean":"2024-03-28T18:56:15.989655+0000","last_became_active":"2024-03-28T18:55:44.610357+0000","last_became_peered":"2024-03-28T18:55:44.610357+0000","last_unstale":"2024-03-28T18:56:15.989655+0000","last_undegraded":"2024-03-28T18:56:15.989655+0000","last_fullsized":"2024-03-28T18:56:15.989655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:26.280548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177029+0000","last_change":"2024-03-28T18:55:45.609349+0000","last_active":"2024-03-28T18:56:16.177029+0000","last_peered":"2024-03-28T18:56:16.177029+0000","last_clean":"2024-03-28T18:56:16.177029+0000","last_became_active":"2024-03-28T18:55:45.608743+0000","last_became_peered":"2024-03-28T18:55:45.608743+0000","last_unstale":"2024-03-28T18:56:16.177029+0000","last_undegraded":"2024-03-28T18:56:16.177029+0000","last_fullsized":"2024-03-28T18:56:16.177029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996992+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:56:15.996992+0000","last_peered":"2024-03-28T18:56:15.996992+0000","last_clean":"2024-03-28T18:56:15.996992+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:56:15.996992+0000","last_undegraded":"2024-03-28T18:56:15.996992+0000","last_fullsized":"2024-03-28T18:56:15.996992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994769+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:56:15.994769+0000","last_peered":"2024-03-28T18:56:15.994769+0000","last_clean":"2024-03-28T18:56:15.994769+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:56:15.994769+0000","last_undegraded":"2024-03-28T18:56:15.994769+0000","last_fullsized":"2024-03-28T18:56:15.994769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989590+0000","last_change":"2024-03-28T18:55:44.601103+0000","last_active":"2024-03-28T18:56:15.989590+0000","last_peered":"2024-03-28T18:56:15.989590+0000","last_clean":"2024-03-28T18:56:15.989590+0000","last_became_active":"2024-03-28T18:55:44.599969+0000","last_became_peered":"2024-03-28T18:55:44.599969+0000","last_unstale":"2024-03-28T18:56:15.989590+0000","last_undegraded":"2024-03-28T18:56:15.989590+0000","last_fullsized":"2024-03-28T18:56:15.989590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:42.018654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994243+0000","last_change":"2024-03-28T18:55:45.594944+0000","last_active":"2024-03-28T18:56:15.994243+0000","last_peered":"2024-03-28T18:56:15.994243+0000","last_clean":"2024-03-28T18:56:15.994243+0000","last_became_active":"2024-03-28T18:55:45.594603+0000","last_became_peered":"2024-03-28T18:55:45.594603+0000","last_unstale":"2024-03-28T18:56:15.994243+0000","last_undegraded":"2024-03-28T18:56:15.994243+0000","last_fullsized":"2024-03-28T18:56:15.994243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373359+0000","last_change":"2024-03-28T18:55:44.602366+0000","last_active":"2024-03-28T18:56:16.373359+0000","last_peered":"2024-03-28T18:56:16.373359+0000","last_clean":"2024-03-28T18:56:16.373359+0000","last_became_active":"2024-03-28T18:55:44.602033+0000","last_became_peered":"2024-03-28T18:55:44.602033+0000","last_unstale":"2024-03-28T18:56:16.373359+0000","last_undegraded":"2024-03-28T18:56:16.373359+0000","last_fullsized":"2024-03-28T18:56:16.373359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994424+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:56:15.994424+0000","last_peered":"2024-03-28T18:56:15.994424+0000","last_clean":"2024-03-28T18:56:15.994424+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:56:15.994424+0000","last_undegraded":"2024-03-28T18:56:15.994424+0000","last_fullsized":"2024-03-28T18:56:15.994424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995004+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:56:15.995004+0000","last_peered":"2024-03-28T18:56:15.995004+0000","last_clean":"2024-03-28T18:56:15.995004+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:56:15.995004+0000","last_undegraded":"2024-03-28T18:56:15.995004+0000","last_fullsized":"2024-03-28T18:56:15.995004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373198+0000","last_change":"2024-03-28T18:55:45.620559+0000","last_active":"2024-03-28T18:56:16.373198+0000","last_peered":"2024-03-28T18:56:16.373198+0000","last_clean":"2024-03-28T18:56:16.373198+0000","last_became_active":"2024-03-28T18:55:45.620249+0000","last_became_peered":"2024-03-28T18:55:45.620249+0000","last_unstale":"2024-03-28T18:56:16.373198+0000","last_undegraded":"2024-03-28T18:56:16.373198+0000","last_fullsized":"2024-03-28T18:56:16.373198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"23'4","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994418+0000","last_change":"2024-03-28T18:55:44.612226+0000","last_active":"2024-03-28T18:56:15.994418+0000","last_peered":"2024-03-28T18:56:15.994418+0000","last_clean":"2024-03-28T18:56:15.994418+0000","last_became_active":"2024-03-28T18:55:44.611921+0000","last_became_peered":"2024-03-28T18:55:44.611921+0000","last_unstale":"2024-03-28T18:56:15.994418+0000","last_undegraded":"2024-03-28T18:56:15.994418+0000","last_fullsized":"2024-03-28T18:56:15.994418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989828+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:56:15.989828+0000","last_peered":"2024-03-28T18:56:15.989828+0000","last_clean":"2024-03-28T18:56:15.989828+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:56:15.989828+0000","last_undegraded":"2024-03-28T18:56:15.989828+0000","last_fullsized":"2024-03-28T18:56:15.989828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990026+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:56:15.990026+0000","last_peered":"2024-03-28T18:56:15.990026+0000","last_clean":"2024-03-28T18:56:15.990026+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:56:15.990026+0000","last_undegraded":"2024-03-28T18:56:15.990026+0000","last_fullsized":"2024-03-28T18:56:15.990026+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996154+0000","last_change":"2024-03-28T18:55:45.612727+0000","last_active":"2024-03-28T18:56:15.996154+0000","last_peered":"2024-03-28T18:56:15.996154+0000","last_clean":"2024-03-28T18:56:15.996154+0000","last_became_active":"2024-03-28T18:55:45.610154+0000","last_became_peered":"2024-03-28T18:55:45.610154+0000","last_unstale":"2024-03-28T18:56:15.996154+0000","last_undegraded":"2024-03-28T18:56:15.996154+0000","last_fullsized":"2024-03-28T18:56:15.996154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994188+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:56:15.994188+0000","last_peered":"2024-03-28T18:56:15.994188+0000","last_clean":"2024-03-28T18:56:15.994188+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:56:15.994188+0000","last_undegraded":"2024-03-28T18:56:15.994188+0000","last_fullsized":"2024-03-28T18:56:15.994188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373560+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:56:16.373560+0000","last_peered":"2024-03-28T18:56:16.373560+0000","last_clean":"2024-03-28T18:56:16.373560+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:56:16.373560+0000","last_undegraded":"2024-03-28T18:56:16.373560+0000","last_fullsized":"2024-03-28T18:56:16.373560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993632+0000","last_change":"2024-03-28T18:55:44.592835+0000","last_active":"2024-03-28T18:56:15.993632+0000","last_peered":"2024-03-28T18:56:15.993632+0000","last_clean":"2024-03-28T18:56:15.993632+0000","last_became_active":"2024-03-28T18:55:44.592505+0000","last_became_peered":"2024-03-28T18:55:44.592505+0000","last_unstale":"2024-03-28T18:56:15.993632+0000","last_undegraded":"2024-03-28T18:56:15.993632+0000","last_fullsized":"2024-03-28T18:56:15.993632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:58.286421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165539+0000","last_change":"2024-03-28T18:55:45.611322+0000","last_active":"2024-03-28T18:56:16.165539+0000","last_peered":"2024-03-28T18:56:16.165539+0000","last_clean":"2024-03-28T18:56:16.165539+0000","last_became_active":"2024-03-28T18:55:45.610958+0000","last_became_peered":"2024-03-28T18:55:45.610958+0000","last_unstale":"2024-03-28T18:56:16.165539+0000","last_undegraded":"2024-03-28T18:56:16.165539+0000","last_fullsized":"2024-03-28T18:56:16.165539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990629+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:56:15.990629+0000","last_peered":"2024-03-28T18:56:15.990629+0000","last_clean":"2024-03-28T18:56:15.990629+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:56:15.990629+0000","last_undegraded":"2024-03-28T18:56:15.990629+0000","last_fullsized":"2024-03-28T18:56:15.990629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165083+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:56:16.165083+0000","last_peered":"2024-03-28T18:56:16.165083+0000","last_clean":"2024-03-28T18:56:16.165083+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:56:16.165083+0000","last_undegraded":"2024-03-28T18:56:16.165083+0000","last_fullsized":"2024-03-28T18:56:16.165083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991514+0000","last_change":"2024-03-28T18:55:44.608283+0000","last_active":"2024-03-28T18:56:15.991514+0000","last_peered":"2024-03-28T18:56:15.991514+0000","last_clean":"2024-03-28T18:56:15.991514+0000","last_became_active":"2024-03-28T18:55:44.607999+0000","last_became_peered":"2024-03-28T18:55:44.607999+0000","last_unstale":"2024-03-28T18:56:15.991514+0000","last_undegraded":"2024-03-28T18:56:15.991514+0000","last_fullsized":"2024-03-28T18:56:15.991514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:22.669778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991790+0000","last_change":"2024-03-28T18:55:45.618390+0000","last_active":"2024-03-28T18:56:15.991790+0000","last_peered":"2024-03-28T18:56:15.991790+0000","last_clean":"2024-03-28T18:56:15.991790+0000","last_became_active":"2024-03-28T18:55:45.617897+0000","last_became_peered":"2024-03-28T18:55:45.617897+0000","last_unstale":"2024-03-28T18:56:15.991790+0000","last_undegraded":"2024-03-28T18:56:15.991790+0000","last_fullsized":"2024-03-28T18:56:15.991790+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:16.520575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992753+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:56:15.992753+0000","last_peered":"2024-03-28T18:56:15.992753+0000","last_clean":"2024-03-28T18:56:15.992753+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:56:15.992753+0000","last_undegraded":"2024-03-28T18:56:15.992753+0000","last_fullsized":"2024-03-28T18:56:15.992753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996140+0000","last_change":"2024-03-28T18:55:45.624938+0000","last_active":"2024-03-28T18:56:15.996140+0000","last_peered":"2024-03-28T18:56:15.996140+0000","last_clean":"2024-03-28T18:56:15.996140+0000","last_became_active":"2024-03-28T18:55:45.624362+0000","last_became_peered":"2024-03-28T18:55:45.624362+0000","last_unstale":"2024-03-28T18:56:15.996140+0000","last_undegraded":"2024-03-28T18:56:15.996140+0000","last_fullsized":"2024-03-28T18:56:15.996140+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992964+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:56:15.992964+0000","last_peered":"2024-03-28T18:56:15.992964+0000","last_clean":"2024-03-28T18:56:15.992964+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:56:15.992964+0000","last_undegraded":"2024-03-28T18:56:15.992964+0000","last_fullsized":"2024-03-28T18:56:15.992964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995980+0000","last_change":"2024-03-28T18:55:45.610168+0000","last_active":"2024-03-28T18:56:15.995980+0000","last_peered":"2024-03-28T18:56:15.995980+0000","last_clean":"2024-03-28T18:56:15.995980+0000","last_became_active":"2024-03-28T18:55:45.609653+0000","last_became_peered":"2024-03-28T18:55:45.609653+0000","last_unstale":"2024-03-28T18:56:15.995980+0000","last_undegraded":"2024-03-28T18:56:15.995980+0000","last_fullsized":"2024-03-28T18:56:15.995980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995799+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:56:15.995799+0000","last_peered":"2024-03-28T18:56:15.995799+0000","last_clean":"2024-03-28T18:56:15.995799+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:56:15.995799+0000","last_undegraded":"2024-03-28T18:56:15.995799+0000","last_fullsized":"2024-03-28T18:56:15.995799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996052+0000","last_change":"2024-03-28T18:55:45.613701+0000","last_active":"2024-03-28T18:56:15.996052+0000","last_peered":"2024-03-28T18:56:15.996052+0000","last_clean":"2024-03-28T18:56:15.996052+0000","last_became_active":"2024-03-28T18:55:45.613310+0000","last_became_peered":"2024-03-28T18:55:45.613310+0000","last_unstale":"2024-03-28T18:56:15.996052+0000","last_undegraded":"2024-03-28T18:56:15.996052+0000","last_fullsized":"2024-03-28T18:56:15.996052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373539+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:56:16.373539+0000","last_peered":"2024-03-28T18:56:16.373539+0000","last_clean":"2024-03-28T18:56:16.373539+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:56:16.373539+0000","last_undegraded":"2024-03-28T18:56:16.373539+0000","last_fullsized":"2024-03-28T18:56:16.373539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177308+0000","last_change":"2024-03-28T18:55:45.615119+0000","last_active":"2024-03-28T18:56:16.177308+0000","last_peered":"2024-03-28T18:56:16.177308+0000","last_clean":"2024-03-28T18:56:16.177308+0000","last_became_active":"2024-03-28T18:55:45.613841+0000","last_became_peered":"2024-03-28T18:55:45.613841+0000","last_unstale":"2024-03-28T18:56:16.177308+0000","last_undegraded":"2024-03-28T18:56:16.177308+0000","last_fullsized":"2024-03-28T18:56:16.177308+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991068+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:56:15.991068+0000","last_peered":"2024-03-28T18:56:15.991068+0000","last_clean":"2024-03-28T18:56:15.991068+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:56:15.991068+0000","last_undegraded":"2024-03-28T18:56:15.991068+0000","last_fullsized":"2024-03-28T18:56:15.991068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164511+0000","last_change":"2024-03-28T18:55:45.619605+0000","last_active":"2024-03-28T18:56:16.164511+0000","last_peered":"2024-03-28T18:56:16.164511+0000","last_clean":"2024-03-28T18:56:16.164511+0000","last_became_active":"2024-03-28T18:55:45.618473+0000","last_became_peered":"2024-03-28T18:55:45.618473+0000","last_unstale":"2024-03-28T18:56:16.164511+0000","last_undegraded":"2024-03-28T18:56:16.164511+0000","last_fullsized":"2024-03-28T18:56:16.164511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996253+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:56:15.996253+0000","last_peered":"2024-03-28T18:56:15.996253+0000","last_clean":"2024-03-28T18:56:15.996253+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:56:15.996253+0000","last_undegraded":"2024-03-28T18:56:15.996253+0000","last_fullsized":"2024-03-28T18:56:15.996253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176839+0000","last_change":"2024-03-28T18:55:45.612284+0000","last_active":"2024-03-28T18:56:16.176839+0000","last_peered":"2024-03-28T18:56:16.176839+0000","last_clean":"2024-03-28T18:56:16.176839+0000","last_became_active":"2024-03-28T18:55:45.607218+0000","last_became_peered":"2024-03-28T18:55:45.607218+0000","last_unstale":"2024-03-28T18:56:16.176839+0000","last_undegraded":"2024-03-28T18:56:16.176839+0000","last_fullsized":"2024-03-28T18:56:16.176839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993957+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:56:15.993957+0000","last_peered":"2024-03-28T18:56:15.993957+0000","last_clean":"2024-03-28T18:56:15.993957+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:56:15.993957+0000","last_undegraded":"2024-03-28T18:56:15.993957+0000","last_fullsized":"2024-03-28T18:56:15.993957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992957+0000","last_change":"2024-03-28T18:55:45.607609+0000","last_active":"2024-03-28T18:56:15.992957+0000","last_peered":"2024-03-28T18:56:15.992957+0000","last_clean":"2024-03-28T18:56:15.992957+0000","last_became_active":"2024-03-28T18:55:45.602582+0000","last_became_peered":"2024-03-28T18:55:45.602582+0000","last_unstale":"2024-03-28T18:56:15.992957+0000","last_undegraded":"2024-03-28T18:56:15.992957+0000","last_fullsized":"2024-03-28T18:56:15.992957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:06:05.335715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.168035+0000","last_change":"2024-03-28T18:56:16.167085+0000","last_active":"2024-03-28T18:56:16.168035+0000","last_peered":"2024-03-28T18:56:16.168035+0000","last_clean":"2024-03-28T18:56:16.168035+0000","last_became_active":"2024-03-28T18:56:16.166729+0000","last_became_peered":"2024-03-28T18:56:16.166729+0000","last_unstale":"2024-03-28T18:56:16.168035+0000","last_undegraded":"2024-03-28T18:56:16.168035+0000","last_fullsized":"2024-03-28T18:56:16.168035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:49.364836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991762+0000","last_change":"2024-03-28T18:55:45.607886+0000","last_active":"2024-03-28T18:56:15.991762+0000","last_peered":"2024-03-28T18:56:15.991762+0000","last_clean":"2024-03-28T18:56:15.991762+0000","last_became_active":"2024-03-28T18:55:45.605964+0000","last_became_peered":"2024-03-28T18:55:45.605964+0000","last_unstale":"2024-03-28T18:56:15.991762+0000","last_undegraded":"2024-03-28T18:56:15.991762+0000","last_fullsized":"2024-03-28T18:56:15.991762+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:45.986654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995008+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:56:15.995008+0000","last_peered":"2024-03-28T18:56:15.995008+0000","last_clean":"2024-03-28T18:56:15.995008+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:56:15.995008+0000","last_undegraded":"2024-03-28T18:56:15.995008+0000","last_fullsized":"2024-03-28T18:56:15.995008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178931+0000","last_change":"2024-03-28T18:55:45.618031+0000","last_active":"2024-03-28T18:56:16.178931+0000","last_peered":"2024-03-28T18:56:16.178931+0000","last_clean":"2024-03-28T18:56:16.178931+0000","last_became_active":"2024-03-28T18:55:45.617719+0000","last_became_peered":"2024-03-28T18:55:45.617719+0000","last_unstale":"2024-03-28T18:56:16.178931+0000","last_undegraded":"2024-03-28T18:56:16.178931+0000","last_fullsized":"2024-03-28T18:56:16.178931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989874+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:56:15.989874+0000","last_peered":"2024-03-28T18:56:15.989874+0000","last_clean":"2024-03-28T18:56:15.989874+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:56:15.989874+0000","last_undegraded":"2024-03-28T18:56:15.989874+0000","last_fullsized":"2024-03-28T18:56:15.989874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993563+0000","last_change":"2024-03-28T18:55:45.603783+0000","last_active":"2024-03-28T18:56:15.993563+0000","last_peered":"2024-03-28T18:56:15.993563+0000","last_clean":"2024-03-28T18:56:15.993563+0000","last_became_active":"2024-03-28T18:55:45.603474+0000","last_became_peered":"2024-03-28T18:55:45.603474+0000","last_unstale":"2024-03-28T18:56:15.993563+0000","last_undegraded":"2024-03-28T18:56:15.993563+0000","last_fullsized":"2024-03-28T18:56:15.993563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:42.328182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.182051+0000","last_change":"2024-03-28T18:56:16.181039+0000","last_active":"2024-03-28T18:56:16.182051+0000","last_peered":"2024-03-28T18:56:16.182051+0000","last_clean":"2024-03-28T18:56:16.182051+0000","last_became_active":"2024-03-28T18:56:16.180227+0000","last_became_peered":"2024-03-28T18:56:16.180227+0000","last_unstale":"2024-03-28T18:56:16.182051+0000","last_undegraded":"2024-03-28T18:56:16.182051+0000","last_fullsized":"2024-03-28T18:56:16.182051+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:54.706529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989997+0000","last_change":"2024-03-28T18:55:45.618912+0000","last_active":"2024-03-28T18:56:15.989997+0000","last_peered":"2024-03-28T18:56:15.989997+0000","last_clean":"2024-03-28T18:56:15.989997+0000","last_became_active":"2024-03-28T18:55:45.618554+0000","last_became_peered":"2024-03-28T18:55:45.618554+0000","last_unstale":"2024-03-28T18:56:15.989997+0000","last_undegraded":"2024-03-28T18:56:15.989997+0000","last_fullsized":"2024-03-28T18:56:15.989997+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373395+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:56:16.373395+0000","last_peered":"2024-03-28T18:56:16.373395+0000","last_clean":"2024-03-28T18:56:16.373395+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:56:16.373395+0000","last_undegraded":"2024-03-28T18:56:16.373395+0000","last_fullsized":"2024-03-28T18:56:16.373395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164597+0000","last_change":"2024-03-28T18:55:45.616420+0000","last_active":"2024-03-28T18:56:16.164597+0000","last_peered":"2024-03-28T18:56:16.164597+0000","last_clean":"2024-03-28T18:56:16.164597+0000","last_became_active":"2024-03-28T18:55:45.614806+0000","last_became_peered":"2024-03-28T18:55:45.614806+0000","last_unstale":"2024-03-28T18:56:16.164597+0000","last_undegraded":"2024-03-28T18:56:16.164597+0000","last_fullsized":"2024-03-28T18:56:16.164597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993436+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:56:15.993436+0000","last_peered":"2024-03-28T18:56:15.993436+0000","last_clean":"2024-03-28T18:56:15.993436+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:56:15.993436+0000","last_undegraded":"2024-03-28T18:56:15.993436+0000","last_fullsized":"2024-03-28T18:56:15.993436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164798+0000","last_change":"2024-03-28T18:55:45.606507+0000","last_active":"2024-03-28T18:56:16.164798+0000","last_peered":"2024-03-28T18:56:16.164798+0000","last_clean":"2024-03-28T18:56:16.164798+0000","last_became_active":"2024-03-28T18:55:45.606205+0000","last_became_peered":"2024-03-28T18:55:45.606205+0000","last_unstale":"2024-03-28T18:56:16.164798+0000","last_undegraded":"2024-03-28T18:56:16.164798+0000","last_fullsized":"2024-03-28T18:56:16.164798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178591+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:56:16.178591+0000","last_peered":"2024-03-28T18:56:16.178591+0000","last_clean":"2024-03-28T18:56:16.178591+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:56:16.178591+0000","last_undegraded":"2024-03-28T18:56:16.178591+0000","last_fullsized":"2024-03-28T18:56:16.178591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994460+0000","last_change":"2024-03-28T18:55:45.607890+0000","last_active":"2024-03-28T18:56:15.994460+0000","last_peered":"2024-03-28T18:56:15.994460+0000","last_clean":"2024-03-28T18:56:15.994460+0000","last_became_active":"2024-03-28T18:55:45.607590+0000","last_became_peered":"2024-03-28T18:55:45.607590+0000","last_unstale":"2024-03-28T18:56:15.994460+0000","last_undegraded":"2024-03-28T18:56:15.994460+0000","last_fullsized":"2024-03-28T18:56:15.994460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993121+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:56:15.993121+0000","last_peered":"2024-03-28T18:56:15.993121+0000","last_clean":"2024-03-28T18:56:15.993121+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:56:15.993121+0000","last_undegraded":"2024-03-28T18:56:15.993121+0000","last_fullsized":"2024-03-28T18:56:15.993121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372260+0000","last_change":"2024-03-28T18:55:45.608851+0000","last_active":"2024-03-28T18:56:16.372260+0000","last_peered":"2024-03-28T18:56:16.372260+0000","last_clean":"2024-03-28T18:56:16.372260+0000","last_became_active":"2024-03-28T18:55:45.608496+0000","last_became_peered":"2024-03-28T18:55:45.608496+0000","last_unstale":"2024-03-28T18:56:16.372260+0000","last_undegraded":"2024-03-28T18:56:16.372260+0000","last_fullsized":"2024-03-28T18:56:16.372260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997752+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:56:15.997752+0000","last_peered":"2024-03-28T18:56:15.997752+0000","last_clean":"2024-03-28T18:56:15.997752+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:56:15.997752+0000","last_undegraded":"2024-03-28T18:56:15.997752+0000","last_fullsized":"2024-03-28T18:56:15.997752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996035+0000","last_change":"2024-03-28T18:55:45.619501+0000","last_active":"2024-03-28T18:56:15.996035+0000","last_peered":"2024-03-28T18:56:15.996035+0000","last_clean":"2024-03-28T18:56:15.996035+0000","last_became_active":"2024-03-28T18:55:45.619212+0000","last_became_peered":"2024-03-28T18:55:45.619212+0000","last_unstale":"2024-03-28T18:56:15.996035+0000","last_undegraded":"2024-03-28T18:56:15.996035+0000","last_fullsized":"2024-03-28T18:56:15.996035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.377790+0000","last_change":"2024-03-28T18:56:16.375984+0000","last_active":"2024-03-28T18:56:16.377790+0000","last_peered":"2024-03-28T18:56:16.377790+0000","last_clean":"2024-03-28T18:56:16.377790+0000","last_became_active":"2024-03-28T18:56:16.375290+0000","last_became_peered":"2024-03-28T18:56:16.375290+0000","last_unstale":"2024-03-28T18:56:16.377790+0000","last_undegraded":"2024-03-28T18:56:16.377790+0000","last_fullsized":"2024-03-28T18:56:16.377790+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:12.576993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166458+0000","last_change":"2024-03-28T18:55:45.615225+0000","last_active":"2024-03-28T18:56:16.166458+0000","last_peered":"2024-03-28T18:56:16.166458+0000","last_clean":"2024-03-28T18:56:16.166458+0000","last_became_active":"2024-03-28T18:55:45.613602+0000","last_became_peered":"2024-03-28T18:55:45.613602+0000","last_unstale":"2024-03-28T18:56:16.166458+0000","last_undegraded":"2024-03-28T18:56:16.166458+0000","last_fullsized":"2024-03-28T18:56:16.166458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994526+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:56:15.994526+0000","last_peered":"2024-03-28T18:56:15.994526+0000","last_clean":"2024-03-28T18:56:15.994526+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:56:15.994526+0000","last_undegraded":"2024-03-28T18:56:15.994526+0000","last_fullsized":"2024-03-28T18:56:15.994526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995808+0000","last_change":"2024-03-28T18:55:45.609471+0000","last_active":"2024-03-28T18:56:15.995808+0000","last_peered":"2024-03-28T18:56:15.995808+0000","last_clean":"2024-03-28T18:56:15.995808+0000","last_became_active":"2024-03-28T18:55:45.609132+0000","last_became_peered":"2024-03-28T18:55:45.609132+0000","last_unstale":"2024-03-28T18:56:15.995808+0000","last_undegraded":"2024-03-28T18:56:15.995808+0000","last_fullsized":"2024-03-28T18:56:15.995808+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372643+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:56:16.372643+0000","last_peered":"2024-03-28T18:56:16.372643+0000","last_clean":"2024-03-28T18:56:16.372643+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:56:16.372643+0000","last_undegraded":"2024-03-28T18:56:16.372643+0000","last_fullsized":"2024-03-28T18:56:16.372643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178130+0000","last_change":"2024-03-28T18:55:45.617501+0000","last_active":"2024-03-28T18:56:16.178130+0000","last_peered":"2024-03-28T18:56:16.178130+0000","last_clean":"2024-03-28T18:56:16.178130+0000","last_became_active":"2024-03-28T18:55:45.617180+0000","last_became_peered":"2024-03-28T18:55:45.617180+0000","last_unstale":"2024-03-28T18:56:16.178130+0000","last_undegraded":"2024-03-28T18:56:16.178130+0000","last_fullsized":"2024-03-28T18:56:16.178130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990476+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:56:15.990476+0000","last_peered":"2024-03-28T18:56:15.990476+0000","last_clean":"2024-03-28T18:56:15.990476+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:56:15.990476+0000","last_undegraded":"2024-03-28T18:56:15.990476+0000","last_fullsized":"2024-03-28T18:56:15.990476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990313+0000","last_change":"2024-03-28T18:55:45.609169+0000","last_active":"2024-03-28T18:56:15.990313+0000","last_peered":"2024-03-28T18:56:15.990313+0000","last_clean":"2024-03-28T18:56:15.990313+0000","last_became_active":"2024-03-28T18:55:45.601948+0000","last_became_peered":"2024-03-28T18:55:45.601948+0000","last_unstale":"2024-03-28T18:56:15.990313+0000","last_undegraded":"2024-03-28T18:56:15.990313+0000","last_fullsized":"2024-03-28T18:56:15.990313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166023+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:56:16.166023+0000","last_peered":"2024-03-28T18:56:16.166023+0000","last_clean":"2024-03-28T18:56:16.166023+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:56:16.166023+0000","last_undegraded":"2024-03-28T18:56:16.166023+0000","last_fullsized":"2024-03-28T18:56:16.166023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993150+0000","last_change":"2024-03-28T18:55:45.616238+0000","last_active":"2024-03-28T18:56:15.993150+0000","last_peered":"2024-03-28T18:56:15.993150+0000","last_clean":"2024-03-28T18:56:15.993150+0000","last_became_active":"2024-03-28T18:55:45.615371+0000","last_became_peered":"2024-03-28T18:55:45.615371+0000","last_unstale":"2024-03-28T18:56:15.993150+0000","last_undegraded":"2024-03-28T18:56:15.993150+0000","last_fullsized":"2024-03-28T18:56:15.993150+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:14.661134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993580+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:56:15.993580+0000","last_peered":"2024-03-28T18:56:15.993580+0000","last_clean":"2024-03-28T18:56:15.993580+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:56:15.993580+0000","last_undegraded":"2024-03-28T18:56:15.993580+0000","last_fullsized":"2024-03-28T18:56:15.993580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177663+0000","last_change":"2024-03-28T18:55:45.613289+0000","last_active":"2024-03-28T18:56:16.177663+0000","last_peered":"2024-03-28T18:56:16.177663+0000","last_clean":"2024-03-28T18:56:16.177663+0000","last_became_active":"2024-03-28T18:55:45.610917+0000","last_became_peered":"2024-03-28T18:55:45.610917+0000","last_unstale":"2024-03-28T18:56:16.177663+0000","last_undegraded":"2024-03-28T18:56:16.177663+0000","last_fullsized":"2024-03-28T18:56:16.177663+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993822+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:56:15.993822+0000","last_peered":"2024-03-28T18:56:15.993822+0000","last_clean":"2024-03-28T18:56:15.993822+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:56:15.993822+0000","last_undegraded":"2024-03-28T18:56:15.993822+0000","last_fullsized":"2024-03-28T18:56:15.993822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994722+0000","last_change":"2024-03-28T18:55:45.621789+0000","last_active":"2024-03-28T18:56:15.994722+0000","last_peered":"2024-03-28T18:56:15.994722+0000","last_clean":"2024-03-28T18:56:15.994722+0000","last_became_active":"2024-03-28T18:55:45.621163+0000","last_became_peered":"2024-03-28T18:55:45.621163+0000","last_unstale":"2024-03-28T18:56:15.994722+0000","last_undegraded":"2024-03-28T18:56:15.994722+0000","last_fullsized":"2024-03-28T18:56:15.994722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179217+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:56:16.179217+0000","last_peered":"2024-03-28T18:56:16.179217+0000","last_clean":"2024-03-28T18:56:16.179217+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:56:16.179217+0000","last_undegraded":"2024-03-28T18:56:16.179217+0000","last_fullsized":"2024-03-28T18:56:16.179217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991623+0000","last_change":"2024-03-28T18:55:45.603774+0000","last_active":"2024-03-28T18:56:15.991623+0000","last_peered":"2024-03-28T18:56:15.991623+0000","last_clean":"2024-03-28T18:56:15.991623+0000","last_became_active":"2024-03-28T18:55:45.603451+0000","last_became_peered":"2024-03-28T18:55:45.603451+0000","last_unstale":"2024-03-28T18:56:15.991623+0000","last_undegraded":"2024-03-28T18:56:15.991623+0000","last_fullsized":"2024-03-28T18:56:15.991623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989714+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:56:15.989714+0000","last_peered":"2024-03-28T18:56:15.989714+0000","last_clean":"2024-03-28T18:56:15.989714+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:56:15.989714+0000","last_undegraded":"2024-03-28T18:56:15.989714+0000","last_fullsized":"2024-03-28T18:56:15.989714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994162+0000","last_change":"2024-03-28T18:55:45.616755+0000","last_active":"2024-03-28T18:56:15.994162+0000","last_peered":"2024-03-28T18:56:15.994162+0000","last_clean":"2024-03-28T18:56:15.994162+0000","last_became_active":"2024-03-28T18:55:45.614611+0000","last_became_peered":"2024-03-28T18:55:45.614611+0000","last_unstale":"2024-03-28T18:56:15.994162+0000","last_undegraded":"2024-03-28T18:56:15.994162+0000","last_fullsized":"2024-03-28T18:56:15.994162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:06.327795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.377155+0000","last_change":"2024-03-28T18:56:16.375743+0000","last_active":"2024-03-28T18:56:16.377155+0000","last_peered":"2024-03-28T18:56:16.377155+0000","last_clean":"2024-03-28T18:56:16.377155+0000","last_became_active":"2024-03-28T18:56:16.375316+0000","last_became_peered":"2024-03-28T18:56:16.375316+0000","last_unstale":"2024-03-28T18:56:16.377155+0000","last_undegraded":"2024-03-28T18:56:16.377155+0000","last_fullsized":"2024-03-28T18:56:16.377155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:27.569718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995729+0000","last_change":"2024-03-28T18:55:45.623195+0000","last_active":"2024-03-28T18:56:15.995729+0000","last_peered":"2024-03-28T18:56:15.995729+0000","last_clean":"2024-03-28T18:56:15.995729+0000","last_became_active":"2024-03-28T18:55:45.622508+0000","last_became_peered":"2024-03-28T18:55:45.622508+0000","last_unstale":"2024-03-28T18:56:15.995729+0000","last_undegraded":"2024-03-28T18:56:15.995729+0000","last_fullsized":"2024-03-28T18:56:15.995729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995605+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:56:15.995605+0000","last_peered":"2024-03-28T18:56:15.995605+0000","last_clean":"2024-03-28T18:56:15.995605+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:56:15.995605+0000","last_undegraded":"2024-03-28T18:56:15.995605+0000","last_fullsized":"2024-03-28T18:56:15.995605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996676+0000","last_change":"2024-03-28T18:55:45.618841+0000","last_active":"2024-03-28T18:56:15.996676+0000","last_peered":"2024-03-28T18:56:15.996676+0000","last_clean":"2024-03-28T18:56:15.996676+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:56:15.996676+0000","last_undegraded":"2024-03-28T18:56:15.996676+0000","last_fullsized":"2024-03-28T18:56:15.996676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177448+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:56:16.177448+0000","last_peered":"2024-03-28T18:56:16.177448+0000","last_clean":"2024-03-28T18:56:16.177448+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:56:16.177448+0000","last_undegraded":"2024-03-28T18:56:16.177448+0000","last_fullsized":"2024-03-28T18:56:16.177448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990122+0000","last_change":"2024-03-28T18:55:45.602886+0000","last_active":"2024-03-28T18:56:15.990122+0000","last_peered":"2024-03-28T18:56:15.990122+0000","last_clean":"2024-03-28T18:56:15.990122+0000","last_became_active":"2024-03-28T18:55:45.602610+0000","last_became_peered":"2024-03-28T18:55:45.602610+0000","last_unstale":"2024-03-28T18:56:15.990122+0000","last_undegraded":"2024-03-28T18:56:15.990122+0000","last_fullsized":"2024-03-28T18:56:15.990122+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996124+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:56:15.996124+0000","last_peered":"2024-03-28T18:56:15.996124+0000","last_clean":"2024-03-28T18:56:15.996124+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:56:15.996124+0000","last_undegraded":"2024-03-28T18:56:15.996124+0000","last_fullsized":"2024-03-28T18:56:15.996124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993035+0000","last_change":"2024-03-28T18:55:45.608673+0000","last_active":"2024-03-28T18:56:15.993035+0000","last_peered":"2024-03-28T18:56:15.993035+0000","last_clean":"2024-03-28T18:56:15.993035+0000","last_became_active":"2024-03-28T18:55:45.608376+0000","last_became_peered":"2024-03-28T18:55:45.608376+0000","last_unstale":"2024-03-28T18:56:15.993035+0000","last_undegraded":"2024-03-28T18:56:15.993035+0000","last_fullsized":"2024-03-28T18:56:15.993035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:31.781549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372875+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:56:16.372875+0000","last_peered":"2024-03-28T18:56:16.372875+0000","last_clean":"2024-03-28T18:56:16.372875+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:56:16.372875+0000","last_undegraded":"2024-03-28T18:56:16.372875+0000","last_fullsized":"2024-03-28T18:56:16.372875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176679+0000","last_change":"2024-03-28T18:55:45.609565+0000","last_active":"2024-03-28T18:56:16.176679+0000","last_peered":"2024-03-28T18:56:16.176679+0000","last_clean":"2024-03-28T18:56:16.176679+0000","last_became_active":"2024-03-28T18:55:45.609082+0000","last_became_peered":"2024-03-28T18:55:45.609082+0000","last_unstale":"2024-03-28T18:56:16.176679+0000","last_undegraded":"2024-03-28T18:56:16.176679+0000","last_fullsized":"2024-03-28T18:56:16.176679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996230+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:56:15.996230+0000","last_peered":"2024-03-28T18:56:15.996230+0000","last_clean":"2024-03-28T18:56:15.996230+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:56:15.996230+0000","last_undegraded":"2024-03-28T18:56:15.996230+0000","last_fullsized":"2024-03-28T18:56:15.996230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178310+0000","last_change":"2024-03-28T18:55:45.618585+0000","last_active":"2024-03-28T18:56:16.178310+0000","last_peered":"2024-03-28T18:56:16.178310+0000","last_clean":"2024-03-28T18:56:16.178310+0000","last_became_active":"2024-03-28T18:55:45.617895+0000","last_became_peered":"2024-03-28T18:55:45.617895+0000","last_unstale":"2024-03-28T18:56:16.178310+0000","last_undegraded":"2024-03-28T18:56:16.178310+0000","last_fullsized":"2024-03-28T18:56:16.178310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:54.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179210+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:56:16.179210+0000","last_peered":"2024-03-28T18:56:16.179210+0000","last_clean":"2024-03-28T18:56:16.179210+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:56:16.179210+0000","last_undegraded":"2024-03-28T18:56:16.179210+0000","last_fullsized":"2024-03-28T18:56:16.179210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":880640,"data_stored":704107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":299008,"data_stored":116551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":113829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":299008,"data_stored":113297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":118825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":113137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":116117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":884736,"data_stored":704073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:56:30.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:56:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.590+0000 7f9aacc00640 1 Processor -- start 2024-03-28T18:56:30.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.590+0000 7f9aacc00640 1 -- start start 2024-03-28T18:56:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.592+0000 7f9aacc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa80c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.592+0000 7f9aacc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9aa80c8e10 0x7f9aa8158be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.592+0000 7f9aacc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa81530f0 con 0x7f9aa81562f0 2024-03-28T18:56:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.592+0000 7f9aacc00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa806d3d0 con 0x7f9aa80c8e10 2024-03-28T18:56:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.592+0000 7f9aacc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80c92f0 con 0x7f9aa8152d10 2024-03-28T18:56:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.593+0000 7f9aa5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa80c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.593+0000 7f9aa5400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9aa80c8e10 0x7f9aa8158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.593+0000 7f9aa5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa80c5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45438/0 (socket says 172.21.15.27:45438) 2024-03-28T18:56:30.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.593+0000 7f9aa5e00640 1 -- 172.21.15.27:0/1962740846 learned_addr learned my addr 172.21.15.27:0/1962740846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:30.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.593+0000 7f9aa5e00640 1 -- 172.21.15.27:0/1962740846 >> v2:172.21.15.155:3300/0 conn(0x7f9aa80c8e10 msgr2=0x7f9aa8158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:30.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.593+0000 7f9aa5e00640 1 --2- 172.21.15.27:0/1962740846 >> v2:172.21.15.155:3300/0 conn(0x7f9aa80c8e10 0x7f9aa8158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.593+0000 7f9aa5e00640 1 -- 172.21.15.27:0/1962740846 >> v1:172.21.15.155:6789/0 conn(0x7f9aa81562f0 legacy=0x7f9aa80c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.593+0000 7f9aa5e00640 1 -- 172.21.15.27:0/1962740846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8155810 con 0x7f9aa8152d10 2024-03-28T18:56:30.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.594+0000 7f9aa5400640 1 --2- 172.21.15.27:0/1962740846 >> v2:172.21.15.155:3300/0 conn(0x7f9aa80c8e10 0x7f9aa8158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:56:30.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.595+0000 7f9aa5e00640 1 --2- 172.21.15.27:0/1962740846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa80c5890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009870 tx=0x7f9a9802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e26cbd991877d39e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:30.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.595+0000 7f9aa4a00640 1 -- 172.21.15.27:0/1962740846 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a980317c0 con 0x7f9aa8152d10 2024-03-28T18:56:30.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aa4a00640 1 -- 172.21.15.27:0/1962740846 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a98037070 con 0x7f9aa8152d10 2024-03-28T18:56:30.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aa4a00640 1 -- 172.21.15.27:0/1962740846 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98031b80 con 0x7f9aa8152d10 2024-03-28T18:56:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aacc00640 1 -- 172.21.15.27:0/1962740846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 msgr2=0x7f9aa80c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1962740846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa80c5890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009870 tx=0x7f9a9802b860 comp rx=0 tx=0).stop 2024-03-28T18:56:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aacc00640 1 -- 172.21.15.27:0/1962740846 shutdown_connections 2024-03-28T18:56:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1962740846 >> v2:172.21.15.155:3300/0 conn(0x7f9aa80c8e10 0x7f9aa8158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1962740846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa80c5890 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aacc00640 1 -- 172.21.15.27:0/1962740846 >> 172.21.15.27:0/1962740846 conn(0x7f9aa8066fb0 msgr2=0x7f9aa80673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aacc00640 1 -- 172.21.15.27:0/1962740846 shutdown_connections 2024-03-28T18:56:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.596+0000 7f9aacc00640 1 -- 172.21.15.27:0/1962740846 wait complete. 2024-03-28T18:56:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.598+0000 7f9aacc00640 1 Processor -- start 2024-03-28T18:56:30.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.598+0000 7f9aacc00640 1 -- start start 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.599+0000 7f9aacc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa80c8e10 0x7f9aa80c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.599+0000 7f9aacc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa8148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.599+0000 7f9aacc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa81562f0 0x7f9aa8148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.599+0000 7f9aacc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80c5dd0 con 0x7f9aa80c8e10 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.599+0000 7f9aacc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa806d850 con 0x7f9aa81562f0 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.599+0000 7f9aacc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80c5f50 con 0x7f9aa8152d10 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.599+0000 7f9aa5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa80c8e10 0x7f9aa80c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.599+0000 7f9aa5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa80c8e10 0x7f9aa80c3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40426/0 (socket says 172.21.15.27:40426) 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa5e00640 1 -- 172.21.15.27:0/1227451771 learned_addr learned my addr 172.21.15.27:0/1227451771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa5400640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa8148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa6800640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa81562f0 0x7f9aa8148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa5e00640 1 -- 172.21.15.27:0/1227451771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 msgr2=0x7f9aa8148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa5e00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa8148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa5e00640 1 -- 172.21.15.27:0/1227451771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa81562f0 msgr2=0x7f9aa8148b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa5e00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa81562f0 0x7f9aa8148b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa5e00640 1 -- 172.21.15.27:0/1227451771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa81ef020 con 0x7f9aa80c8e10 2024-03-28T18:56:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.600+0000 7f9aa5e00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa80c8e10 0x7f9aa80c3860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9a98000c00 tx=0x7f9a980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:30.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.601+0000 7f9a9e000640 1 -- 172.21.15.27:0/1227451771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98032650 con 0x7f9aa80c8e10 2024-03-28T18:56:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.601+0000 7f9a9e000640 1 -- 172.21.15.27:0/1227451771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a980327f0 con 0x7f9aa80c8e10 2024-03-28T18:56:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.601+0000 7f9a9e000640 1 -- 172.21.15.27:0/1227451771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98032be0 con 0x7f9aa80c8e10 2024-03-28T18:56:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.601+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa81ef300 con 0x7f9aa80c8e10 2024-03-28T18:56:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.601+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa80c3da0 con 0x7f9aa80c8e10 2024-03-28T18:56:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.602+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa804f400 con 0x7f9aa80c8e10 2024-03-28T18:56:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.603+0000 7f9a9e000640 1 -- 172.21.15.27:0/1227451771 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a9804c060 con 0x7f9aa80c8e10 2024-03-28T18:56:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.604+0000 7f9a9e000640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a8406bbd0 0x7f9a8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.604+0000 7f9a9e000640 1 -- 172.21.15.27:0/1227451771 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f9a980ac940 con 0x7f9aa80c8e10 2024-03-28T18:56:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.604+0000 7f9aa5400640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a8406bbd0 0x7f9a8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.605+0000 7f9aa5400640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a8406bbd0 0x7f9a8406e090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009560 tx=0x7f9a90008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:30.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.609+0000 7f9a9e000640 1 -- 172.21.15.27:0/1227451771 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a98040070 con 0x7f9aa80c8e10 2024-03-28T18:56:30.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.807+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9aa806d630 con 0x7f9a8406bbd0 2024-03-28T18:56:30.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.836+0000 7f9a9e000640 1 -- 172.21.15.27:0/1227451771 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490350 (secure 0 0 0) 0x7f9aa806d630 con 0x7f9a8406bbd0 2024-03-28T18:56:30.842 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:30.845 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:56:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a8406bbd0 msgr2=0x7f9a8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a8406bbd0 0x7f9a8406e090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009560 tx=0x7f9a90008040 comp rx=0 tx=0).stop 2024-03-28T18:56:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa80c8e10 msgr2=0x7f9aa80c3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa80c8e10 0x7f9aa80c3860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9a98000c00 tx=0x7f9a980046c0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 shutdown_connections 2024-03-28T18:56:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a8406bbd0 0x7f9a8406e090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa81562f0 0x7f9aa8148b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9aa8152d10 0x7f9aa8148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 --2- 172.21.15.27:0/1227451771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa80c8e10 0x7f9aa80c3860 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.843+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 >> 172.21.15.27:0/1227451771 conn(0x7f9aa8066fb0 msgr2=0x7f9aa80c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.844+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 shutdown_connections 2024-03-28T18:56:30.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:30.844+0000 7f9aacc00640 1 -- 172.21.15.27:0/1227451771 wait complete. 2024-03-28T18:56:30.862 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-28T18:56:28.912012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597700,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":255,"num_write_kb":2450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219628,"kb_used_data":3464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755092,"statfs":{"total":773094113280,"available":772869214208,"internally_reserved":0,"allocated":3547136,"data_stored":2099936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005066"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164203+0000","last_change":"2024-03-28T18:55:45.603782+0000","last_active":"2024-03-28T18:56:16.164203+0000","last_peered":"2024-03-28T18:56:16.164203+0000","last_clean":"2024-03-28T18:56:16.164203+0000","last_became_active":"2024-03-28T18:55:45.603420+0000","last_became_peered":"2024-03-28T18:55:45.603420+0000","last_unstale":"2024-03-28T18:56:16.164203+0000","last_undegraded":"2024-03-28T18:56:16.164203+0000","last_fullsized":"2024-03-28T18:56:16.164203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373460+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:56:16.373460+0000","last_peered":"2024-03-28T18:56:16.373460+0000","last_clean":"2024-03-28T18:56:16.373460+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:56:16.373460+0000","last_undegraded":"2024-03-28T18:56:16.373460+0000","last_fullsized":"2024-03-28T18:56:16.373460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992310+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:56:15.992310+0000","last_peered":"2024-03-28T18:56:15.992310+0000","last_clean":"2024-03-28T18:56:15.992310+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:56:15.992310+0000","last_undegraded":"2024-03-28T18:56:15.992310+0000","last_fullsized":"2024-03-28T18:56:15.992310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989768+0000","last_change":"2024-03-28T18:55:44.602536+0000","last_active":"2024-03-28T18:56:15.989768+0000","last_peered":"2024-03-28T18:56:15.989768+0000","last_clean":"2024-03-28T18:56:15.989768+0000","last_became_active":"2024-03-28T18:55:44.602256+0000","last_became_peered":"2024-03-28T18:55:44.602256+0000","last_unstale":"2024-03-28T18:56:15.989768+0000","last_undegraded":"2024-03-28T18:56:15.989768+0000","last_fullsized":"2024-03-28T18:56:15.989768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994213+0000","last_change":"2024-03-28T18:55:45.625338+0000","last_active":"2024-03-28T18:56:15.994213+0000","last_peered":"2024-03-28T18:56:15.994213+0000","last_clean":"2024-03-28T18:56:15.994213+0000","last_became_active":"2024-03-28T18:55:45.625067+0000","last_became_peered":"2024-03-28T18:55:45.625067+0000","last_unstale":"2024-03-28T18:56:15.994213+0000","last_undegraded":"2024-03-28T18:56:15.994213+0000","last_fullsized":"2024-03-28T18:56:15.994213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991986+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:56:15.991986+0000","last_peered":"2024-03-28T18:56:15.991986+0000","last_clean":"2024-03-28T18:56:15.991986+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:56:15.991986+0000","last_undegraded":"2024-03-28T18:56:15.991986+0000","last_fullsized":"2024-03-28T18:56:15.991986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373257+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:56:16.373257+0000","last_peered":"2024-03-28T18:56:16.373257+0000","last_clean":"2024-03-28T18:56:16.373257+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:56:16.373257+0000","last_undegraded":"2024-03-28T18:56:16.373257+0000","last_fullsized":"2024-03-28T18:56:16.373257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164333+0000","last_change":"2024-03-28T18:55:44.598423+0000","last_active":"2024-03-28T18:56:16.164333+0000","last_peered":"2024-03-28T18:56:16.164333+0000","last_clean":"2024-03-28T18:56:16.164333+0000","last_became_active":"2024-03-28T18:55:44.598092+0000","last_became_peered":"2024-03-28T18:55:44.598092+0000","last_unstale":"2024-03-28T18:56:16.164333+0000","last_undegraded":"2024-03-28T18:56:16.164333+0000","last_fullsized":"2024-03-28T18:56:16.164333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.646619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178114+0000","last_change":"2024-03-28T18:55:45.608201+0000","last_active":"2024-03-28T18:56:16.178114+0000","last_peered":"2024-03-28T18:56:16.178114+0000","last_clean":"2024-03-28T18:56:16.178114+0000","last_became_active":"2024-03-28T18:55:45.607912+0000","last_became_peered":"2024-03-28T18:55:45.607912+0000","last_unstale":"2024-03-28T18:56:16.178114+0000","last_undegraded":"2024-03-28T18:56:16.178114+0000","last_fullsized":"2024-03-28T18:56:16.178114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178494+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:56:16.178494+0000","last_peered":"2024-03-28T18:56:16.178494+0000","last_clean":"2024-03-28T18:56:16.178494+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:56:16.178494+0000","last_undegraded":"2024-03-28T18:56:16.178494+0000","last_fullsized":"2024-03-28T18:56:16.178494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996247+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:56:15.996247+0000","last_peered":"2024-03-28T18:56:15.996247+0000","last_clean":"2024-03-28T18:56:15.996247+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:56:15.996247+0000","last_undegraded":"2024-03-28T18:56:15.996247+0000","last_fullsized":"2024-03-28T18:56:15.996247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164809+0000","last_change":"2024-03-28T18:55:44.604673+0000","last_active":"2024-03-28T18:56:16.164809+0000","last_peered":"2024-03-28T18:56:16.164809+0000","last_clean":"2024-03-28T18:56:16.164809+0000","last_became_active":"2024-03-28T18:55:44.603926+0000","last_became_peered":"2024-03-28T18:55:44.603926+0000","last_unstale":"2024-03-28T18:56:16.164809+0000","last_undegraded":"2024-03-28T18:56:16.164809+0000","last_fullsized":"2024-03-28T18:56:16.164809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995761+0000","last_change":"2024-03-28T18:55:45.610033+0000","last_active":"2024-03-28T18:56:15.995761+0000","last_peered":"2024-03-28T18:56:15.995761+0000","last_clean":"2024-03-28T18:56:15.995761+0000","last_became_active":"2024-03-28T18:55:45.609706+0000","last_became_peered":"2024-03-28T18:55:45.609706+0000","last_unstale":"2024-03-28T18:56:15.995761+0000","last_undegraded":"2024-03-28T18:56:15.995761+0000","last_fullsized":"2024-03-28T18:56:15.995761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178300+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:56:16.178300+0000","last_peered":"2024-03-28T18:56:16.178300+0000","last_clean":"2024-03-28T18:56:16.178300+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:56:16.178300+0000","last_undegraded":"2024-03-28T18:56:16.178300+0000","last_fullsized":"2024-03-28T18:56:16.178300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996865+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:56:15.996865+0000","last_peered":"2024-03-28T18:56:15.996865+0000","last_clean":"2024-03-28T18:56:15.996865+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:56:15.996865+0000","last_undegraded":"2024-03-28T18:56:15.996865+0000","last_fullsized":"2024-03-28T18:56:15.996865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164862+0000","last_change":"2024-03-28T18:55:44.606715+0000","last_active":"2024-03-28T18:56:16.164862+0000","last_peered":"2024-03-28T18:56:16.164862+0000","last_clean":"2024-03-28T18:56:16.164862+0000","last_became_active":"2024-03-28T18:55:44.606343+0000","last_became_peered":"2024-03-28T18:55:44.606343+0000","last_unstale":"2024-03-28T18:56:16.164862+0000","last_undegraded":"2024-03-28T18:56:16.164862+0000","last_fullsized":"2024-03-28T18:56:16.164862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:51.471450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995747+0000","last_change":"2024-03-28T18:55:45.623483+0000","last_active":"2024-03-28T18:56:15.995747+0000","last_peered":"2024-03-28T18:56:15.995747+0000","last_clean":"2024-03-28T18:56:15.995747+0000","last_became_active":"2024-03-28T18:55:45.622894+0000","last_became_peered":"2024-03-28T18:55:45.622894+0000","last_unstale":"2024-03-28T18:56:15.995747+0000","last_undegraded":"2024-03-28T18:56:15.995747+0000","last_fullsized":"2024-03-28T18:56:15.995747+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993916+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:56:15.993916+0000","last_peered":"2024-03-28T18:56:15.993916+0000","last_clean":"2024-03-28T18:56:15.993916+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:56:15.993916+0000","last_undegraded":"2024-03-28T18:56:15.993916+0000","last_fullsized":"2024-03-28T18:56:15.993916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177876+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:56:16.177876+0000","last_peered":"2024-03-28T18:56:16.177876+0000","last_clean":"2024-03-28T18:56:16.177876+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:56:16.177876+0000","last_undegraded":"2024-03-28T18:56:16.177876+0000","last_fullsized":"2024-03-28T18:56:16.177876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165877+0000","last_change":"2024-03-28T18:55:44.601415+0000","last_active":"2024-03-28T18:56:16.165877+0000","last_peered":"2024-03-28T18:56:16.165877+0000","last_clean":"2024-03-28T18:56:16.165877+0000","last_became_active":"2024-03-28T18:55:44.601124+0000","last_became_peered":"2024-03-28T18:55:44.601124+0000","last_unstale":"2024-03-28T18:56:16.165877+0000","last_undegraded":"2024-03-28T18:56:16.165877+0000","last_fullsized":"2024-03-28T18:56:16.165877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:30.767034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993126+0000","last_change":"2024-03-28T18:55:45.604795+0000","last_active":"2024-03-28T18:56:15.993126+0000","last_peered":"2024-03-28T18:56:15.993126+0000","last_clean":"2024-03-28T18:56:15.993126+0000","last_became_active":"2024-03-28T18:55:45.604432+0000","last_became_peered":"2024-03-28T18:55:45.604432+0000","last_unstale":"2024-03-28T18:56:15.993126+0000","last_undegraded":"2024-03-28T18:56:15.993126+0000","last_fullsized":"2024-03-28T18:56:15.993126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:37.638804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990541+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:56:15.990541+0000","last_peered":"2024-03-28T18:56:15.990541+0000","last_clean":"2024-03-28T18:56:15.990541+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:56:15.990541+0000","last_undegraded":"2024-03-28T18:56:15.990541+0000","last_fullsized":"2024-03-28T18:56:15.990541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995164+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:56:15.995164+0000","last_peered":"2024-03-28T18:56:15.995164+0000","last_clean":"2024-03-28T18:56:15.995164+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:56:15.995164+0000","last_undegraded":"2024-03-28T18:56:15.995164+0000","last_fullsized":"2024-03-28T18:56:15.995164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165844+0000","last_change":"2024-03-28T18:55:44.603859+0000","last_active":"2024-03-28T18:56:16.165844+0000","last_peered":"2024-03-28T18:56:16.165844+0000","last_clean":"2024-03-28T18:56:16.165844+0000","last_became_active":"2024-03-28T18:55:44.603540+0000","last_became_peered":"2024-03-28T18:55:44.603540+0000","last_unstale":"2024-03-28T18:56:16.165844+0000","last_undegraded":"2024-03-28T18:56:16.165844+0000","last_fullsized":"2024-03-28T18:56:16.165844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165791+0000","last_change":"2024-03-28T18:55:45.608142+0000","last_active":"2024-03-28T18:56:16.165791+0000","last_peered":"2024-03-28T18:56:16.165791+0000","last_clean":"2024-03-28T18:56:16.165791+0000","last_became_active":"2024-03-28T18:55:45.606117+0000","last_became_peered":"2024-03-28T18:55:45.606117+0000","last_unstale":"2024-03-28T18:56:16.165791+0000","last_undegraded":"2024-03-28T18:56:16.165791+0000","last_fullsized":"2024-03-28T18:56:16.165791+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989871+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:56:15.989871+0000","last_peered":"2024-03-28T18:56:15.989871+0000","last_clean":"2024-03-28T18:56:15.989871+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:56:15.989871+0000","last_undegraded":"2024-03-28T18:56:15.989871+0000","last_fullsized":"2024-03-28T18:56:15.989871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994533+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:56:15.994533+0000","last_peered":"2024-03-28T18:56:15.994533+0000","last_clean":"2024-03-28T18:56:15.994533+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:56:15.994533+0000","last_undegraded":"2024-03-28T18:56:15.994533+0000","last_fullsized":"2024-03-28T18:56:15.994533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996249+0000","last_change":"2024-03-28T18:55:44.606690+0000","last_active":"2024-03-28T18:56:15.996249+0000","last_peered":"2024-03-28T18:56:15.996249+0000","last_clean":"2024-03-28T18:56:15.996249+0000","last_became_active":"2024-03-28T18:55:44.606421+0000","last_became_peered":"2024-03-28T18:55:44.606421+0000","last_unstale":"2024-03-28T18:56:15.996249+0000","last_undegraded":"2024-03-28T18:56:15.996249+0000","last_fullsized":"2024-03-28T18:56:15.996249+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997284+0000","last_change":"2024-03-28T18:55:45.610580+0000","last_active":"2024-03-28T18:56:15.997284+0000","last_peered":"2024-03-28T18:56:15.997284+0000","last_clean":"2024-03-28T18:56:15.997284+0000","last_became_active":"2024-03-28T18:55:45.610316+0000","last_became_peered":"2024-03-28T18:55:45.610316+0000","last_unstale":"2024-03-28T18:56:15.997284+0000","last_undegraded":"2024-03-28T18:56:15.997284+0000","last_fullsized":"2024-03-28T18:56:15.997284+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165194+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:56:16.165194+0000","last_peered":"2024-03-28T18:56:16.165194+0000","last_clean":"2024-03-28T18:56:16.165194+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:56:16.165194+0000","last_undegraded":"2024-03-28T18:56:16.165194+0000","last_fullsized":"2024-03-28T18:56:16.165194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177321+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:56:16.177321+0000","last_peered":"2024-03-28T18:56:16.177321+0000","last_clean":"2024-03-28T18:56:16.177321+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:56:16.177321+0000","last_undegraded":"2024-03-28T18:56:16.177321+0000","last_fullsized":"2024-03-28T18:56:16.177321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166077+0000","last_change":"2024-03-28T18:55:44.600523+0000","last_active":"2024-03-28T18:56:16.166077+0000","last_peered":"2024-03-28T18:56:16.166077+0000","last_clean":"2024-03-28T18:56:16.166077+0000","last_became_active":"2024-03-28T18:55:44.597550+0000","last_became_peered":"2024-03-28T18:55:44.597550+0000","last_unstale":"2024-03-28T18:56:16.166077+0000","last_undegraded":"2024-03-28T18:56:16.166077+0000","last_fullsized":"2024-03-28T18:56:16.166077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989484+0000","last_change":"2024-03-28T18:55:45.619798+0000","last_active":"2024-03-28T18:56:15.989484+0000","last_peered":"2024-03-28T18:56:15.989484+0000","last_clean":"2024-03-28T18:56:15.989484+0000","last_became_active":"2024-03-28T18:55:45.616517+0000","last_became_peered":"2024-03-28T18:55:45.616517+0000","last_unstale":"2024-03-28T18:56:15.989484+0000","last_undegraded":"2024-03-28T18:56:15.989484+0000","last_fullsized":"2024-03-28T18:56:15.989484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.374003+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:56:16.374003+0000","last_peered":"2024-03-28T18:56:16.374003+0000","last_clean":"2024-03-28T18:56:16.374003+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:56:16.374003+0000","last_undegraded":"2024-03-28T18:56:16.374003+0000","last_fullsized":"2024-03-28T18:56:16.374003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994929+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:56:15.994929+0000","last_peered":"2024-03-28T18:56:15.994929+0000","last_clean":"2024-03-28T18:56:15.994929+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:56:15.994929+0000","last_undegraded":"2024-03-28T18:56:15.994929+0000","last_fullsized":"2024-03-28T18:56:15.994929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164852+0000","last_change":"2024-03-28T18:55:44.606498+0000","last_active":"2024-03-28T18:56:16.164852+0000","last_peered":"2024-03-28T18:56:16.164852+0000","last_clean":"2024-03-28T18:56:16.164852+0000","last_became_active":"2024-03-28T18:55:44.606098+0000","last_became_peered":"2024-03-28T18:55:44.606098+0000","last_unstale":"2024-03-28T18:56:16.164852+0000","last_undegraded":"2024-03-28T18:56:16.164852+0000","last_fullsized":"2024-03-28T18:56:16.164852+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164035+0000","last_change":"2024-03-28T18:55:45.617483+0000","last_active":"2024-03-28T18:56:16.164035+0000","last_peered":"2024-03-28T18:56:16.164035+0000","last_clean":"2024-03-28T18:56:16.164035+0000","last_became_active":"2024-03-28T18:55:45.617169+0000","last_became_peered":"2024-03-28T18:55:45.617169+0000","last_unstale":"2024-03-28T18:56:16.164035+0000","last_undegraded":"2024-03-28T18:56:16.164035+0000","last_fullsized":"2024-03-28T18:56:16.164035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992182+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:56:15.992182+0000","last_peered":"2024-03-28T18:56:15.992182+0000","last_clean":"2024-03-28T18:56:15.992182+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:56:15.992182+0000","last_undegraded":"2024-03-28T18:56:15.992182+0000","last_fullsized":"2024-03-28T18:56:15.992182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996404+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:56:15.996404+0000","last_peered":"2024-03-28T18:56:15.996404+0000","last_clean":"2024-03-28T18:56:15.996404+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:56:15.996404+0000","last_undegraded":"2024-03-28T18:56:15.996404+0000","last_fullsized":"2024-03-28T18:56:15.996404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989501+0000","last_change":"2024-03-28T18:55:44.601836+0000","last_active":"2024-03-28T18:56:15.989501+0000","last_peered":"2024-03-28T18:56:15.989501+0000","last_clean":"2024-03-28T18:56:15.989501+0000","last_became_active":"2024-03-28T18:55:44.601552+0000","last_became_peered":"2024-03-28T18:55:44.601552+0000","last_unstale":"2024-03-28T18:56:15.989501+0000","last_undegraded":"2024-03-28T18:56:15.989501+0000","last_fullsized":"2024-03-28T18:56:15.989501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993178+0000","last_change":"2024-03-28T18:55:45.601801+0000","last_active":"2024-03-28T18:56:15.993178+0000","last_peered":"2024-03-28T18:56:15.993178+0000","last_clean":"2024-03-28T18:56:15.993178+0000","last_became_active":"2024-03-28T18:55:45.601459+0000","last_became_peered":"2024-03-28T18:55:45.601459+0000","last_unstale":"2024-03-28T18:56:15.993178+0000","last_undegraded":"2024-03-28T18:56:15.993178+0000","last_fullsized":"2024-03-28T18:56:15.993178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:43.033170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995963+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:56:15.995963+0000","last_peered":"2024-03-28T18:56:15.995963+0000","last_clean":"2024-03-28T18:56:15.995963+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:56:15.995963+0000","last_undegraded":"2024-03-28T18:56:15.995963+0000","last_fullsized":"2024-03-28T18:56:15.995963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178676+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:56:16.178676+0000","last_peered":"2024-03-28T18:56:16.178676+0000","last_clean":"2024-03-28T18:56:16.178676+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:56:16.178676+0000","last_undegraded":"2024-03-28T18:56:16.178676+0000","last_fullsized":"2024-03-28T18:56:16.178676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373043+0000","last_change":"2024-03-28T18:55:44.611148+0000","last_active":"2024-03-28T18:56:16.373043+0000","last_peered":"2024-03-28T18:56:16.373043+0000","last_clean":"2024-03-28T18:56:16.373043+0000","last_became_active":"2024-03-28T18:55:44.610835+0000","last_became_peered":"2024-03-28T18:55:44.610835+0000","last_unstale":"2024-03-28T18:56:16.373043+0000","last_undegraded":"2024-03-28T18:56:16.373043+0000","last_fullsized":"2024-03-28T18:56:16.373043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373399+0000","last_change":"2024-03-28T18:55:45.623672+0000","last_active":"2024-03-28T18:56:16.373399+0000","last_peered":"2024-03-28T18:56:16.373399+0000","last_clean":"2024-03-28T18:56:16.373399+0000","last_became_active":"2024-03-28T18:55:45.622568+0000","last_became_peered":"2024-03-28T18:55:45.622568+0000","last_unstale":"2024-03-28T18:56:16.373399+0000","last_undegraded":"2024-03-28T18:56:16.373399+0000","last_fullsized":"2024-03-28T18:56:16.373399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997025+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:56:15.997025+0000","last_peered":"2024-03-28T18:56:15.997025+0000","last_clean":"2024-03-28T18:56:15.997025+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:56:15.997025+0000","last_undegraded":"2024-03-28T18:56:15.997025+0000","last_fullsized":"2024-03-28T18:56:15.997025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996010+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:56:15.996010+0000","last_peered":"2024-03-28T18:56:15.996010+0000","last_clean":"2024-03-28T18:56:15.996010+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:56:15.996010+0000","last_undegraded":"2024-03-28T18:56:15.996010+0000","last_fullsized":"2024-03-28T18:56:15.996010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994806+0000","last_change":"2024-03-28T18:55:44.604077+0000","last_active":"2024-03-28T18:56:15.994806+0000","last_peered":"2024-03-28T18:56:15.994806+0000","last_clean":"2024-03-28T18:56:15.994806+0000","last_became_active":"2024-03-28T18:55:44.603773+0000","last_became_peered":"2024-03-28T18:55:44.603773+0000","last_unstale":"2024-03-28T18:56:15.994806+0000","last_undegraded":"2024-03-28T18:56:15.994806+0000","last_fullsized":"2024-03-28T18:56:15.994806+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372682+0000","last_change":"2024-03-28T18:55:45.615444+0000","last_active":"2024-03-28T18:56:16.372682+0000","last_peered":"2024-03-28T18:56:16.372682+0000","last_clean":"2024-03-28T18:56:16.372682+0000","last_became_active":"2024-03-28T18:55:45.615002+0000","last_became_peered":"2024-03-28T18:55:45.615002+0000","last_unstale":"2024-03-28T18:56:16.372682+0000","last_undegraded":"2024-03-28T18:56:16.372682+0000","last_fullsized":"2024-03-28T18:56:16.372682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166059+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:56:16.166059+0000","last_peered":"2024-03-28T18:56:16.166059+0000","last_clean":"2024-03-28T18:56:16.166059+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:56:16.166059+0000","last_undegraded":"2024-03-28T18:56:16.166059+0000","last_fullsized":"2024-03-28T18:56:16.166059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996064+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:56:15.996064+0000","last_peered":"2024-03-28T18:56:15.996064+0000","last_clean":"2024-03-28T18:56:15.996064+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:56:15.996064+0000","last_undegraded":"2024-03-28T18:56:15.996064+0000","last_fullsized":"2024-03-28T18:56:15.996064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995899+0000","last_change":"2024-03-28T18:55:44.590703+0000","last_active":"2024-03-28T18:56:15.995899+0000","last_peered":"2024-03-28T18:56:15.995899+0000","last_clean":"2024-03-28T18:56:15.995899+0000","last_became_active":"2024-03-28T18:55:44.590408+0000","last_became_peered":"2024-03-28T18:55:44.590408+0000","last_unstale":"2024-03-28T18:56:15.995899+0000","last_undegraded":"2024-03-28T18:56:15.995899+0000","last_fullsized":"2024-03-28T18:56:15.995899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992984+0000","last_change":"2024-03-28T18:55:45.609795+0000","last_active":"2024-03-28T18:56:15.992984+0000","last_peered":"2024-03-28T18:56:15.992984+0000","last_clean":"2024-03-28T18:56:15.992984+0000","last_became_active":"2024-03-28T18:55:45.603695+0000","last_became_peered":"2024-03-28T18:55:45.603695+0000","last_unstale":"2024-03-28T18:56:15.992984+0000","last_undegraded":"2024-03-28T18:56:15.992984+0000","last_fullsized":"2024-03-28T18:56:15.992984+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:49.173555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994287+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:56:15.994287+0000","last_peered":"2024-03-28T18:56:15.994287+0000","last_clean":"2024-03-28T18:56:15.994287+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:56:15.994287+0000","last_undegraded":"2024-03-28T18:56:15.994287+0000","last_fullsized":"2024-03-28T18:56:15.994287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993148+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:56:15.993148+0000","last_peered":"2024-03-28T18:56:15.993148+0000","last_clean":"2024-03-28T18:56:15.993148+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:56:15.993148+0000","last_undegraded":"2024-03-28T18:56:15.993148+0000","last_fullsized":"2024-03-28T18:56:15.993148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996282+0000","last_change":"2024-03-28T18:55:44.612172+0000","last_active":"2024-03-28T18:56:15.996282+0000","last_peered":"2024-03-28T18:56:15.996282+0000","last_clean":"2024-03-28T18:56:15.996282+0000","last_became_active":"2024-03-28T18:55:44.611859+0000","last_became_peered":"2024-03-28T18:55:44.611859+0000","last_unstale":"2024-03-28T18:56:15.996282+0000","last_undegraded":"2024-03-28T18:56:15.996282+0000","last_fullsized":"2024-03-28T18:56:15.996282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997500+0000","last_change":"2024-03-28T18:55:45.619723+0000","last_active":"2024-03-28T18:56:15.997500+0000","last_peered":"2024-03-28T18:56:15.997500+0000","last_clean":"2024-03-28T18:56:15.997500+0000","last_became_active":"2024-03-28T18:55:45.619437+0000","last_became_peered":"2024-03-28T18:55:45.619437+0000","last_unstale":"2024-03-28T18:56:15.997500+0000","last_undegraded":"2024-03-28T18:56:15.997500+0000","last_fullsized":"2024-03-28T18:56:15.997500+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372410+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:56:16.372410+0000","last_peered":"2024-03-28T18:56:16.372410+0000","last_clean":"2024-03-28T18:56:16.372410+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:56:16.372410+0000","last_undegraded":"2024-03-28T18:56:16.372410+0000","last_fullsized":"2024-03-28T18:56:16.372410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177331+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:56:16.177331+0000","last_peered":"2024-03-28T18:56:16.177331+0000","last_clean":"2024-03-28T18:56:16.177331+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:56:16.177331+0000","last_undegraded":"2024-03-28T18:56:16.177331+0000","last_fullsized":"2024-03-28T18:56:16.177331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"24'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373710+0000","last_change":"2024-03-28T18:55:44.592064+0000","last_active":"2024-03-28T18:56:16.373710+0000","last_peered":"2024-03-28T18:56:16.373710+0000","last_clean":"2024-03-28T18:56:16.373710+0000","last_became_active":"2024-03-28T18:55:44.591263+0000","last_became_peered":"2024-03-28T18:55:44.591263+0000","last_unstale":"2024-03-28T18:56:16.373710+0000","last_undegraded":"2024-03-28T18:56:16.373710+0000","last_fullsized":"2024-03-28T18:56:16.373710+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177095+0000","last_change":"2024-03-28T18:55:45.614820+0000","last_active":"2024-03-28T18:56:16.177095+0000","last_peered":"2024-03-28T18:56:16.177095+0000","last_clean":"2024-03-28T18:56:16.177095+0000","last_became_active":"2024-03-28T18:55:45.614517+0000","last_became_peered":"2024-03-28T18:55:45.614517+0000","last_unstale":"2024-03-28T18:56:16.177095+0000","last_undegraded":"2024-03-28T18:56:16.177095+0000","last_fullsized":"2024-03-28T18:56:16.177095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990462+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:56:15.990462+0000","last_peered":"2024-03-28T18:56:15.990462+0000","last_clean":"2024-03-28T18:56:15.990462+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:56:15.990462+0000","last_undegraded":"2024-03-28T18:56:15.990462+0000","last_fullsized":"2024-03-28T18:56:15.990462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995955+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:56:15.995955+0000","last_peered":"2024-03-28T18:56:15.995955+0000","last_clean":"2024-03-28T18:56:15.995955+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:56:15.995955+0000","last_undegraded":"2024-03-28T18:56:15.995955+0000","last_fullsized":"2024-03-28T18:56:15.995955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177684+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:56:16.177684+0000","last_peered":"2024-03-28T18:56:16.177684+0000","last_clean":"2024-03-28T18:56:16.177684+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:56:16.177684+0000","last_undegraded":"2024-03-28T18:56:16.177684+0000","last_fullsized":"2024-03-28T18:56:16.177684+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993499+0000","last_change":"2024-03-28T18:55:44.602097+0000","last_active":"2024-03-28T18:56:15.993499+0000","last_peered":"2024-03-28T18:56:15.993499+0000","last_clean":"2024-03-28T18:56:15.993499+0000","last_became_active":"2024-03-28T18:55:44.601769+0000","last_became_peered":"2024-03-28T18:55:44.601769+0000","last_unstale":"2024-03-28T18:56:15.993499+0000","last_undegraded":"2024-03-28T18:56:15.993499+0000","last_fullsized":"2024-03-28T18:56:15.993499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993296+0000","last_change":"2024-03-28T18:55:45.616485+0000","last_active":"2024-03-28T18:56:15.993296+0000","last_peered":"2024-03-28T18:56:15.993296+0000","last_clean":"2024-03-28T18:56:15.993296+0000","last_became_active":"2024-03-28T18:55:45.615813+0000","last_became_peered":"2024-03-28T18:55:45.615813+0000","last_unstale":"2024-03-28T18:56:15.993296+0000","last_undegraded":"2024-03-28T18:56:15.993296+0000","last_fullsized":"2024-03-28T18:56:15.993296+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:21.227281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178319+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:56:16.178319+0000","last_peered":"2024-03-28T18:56:16.178319+0000","last_clean":"2024-03-28T18:56:16.178319+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:56:16.178319+0000","last_undegraded":"2024-03-28T18:56:16.178319+0000","last_fullsized":"2024-03-28T18:56:16.178319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993997+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:56:15.993997+0000","last_peered":"2024-03-28T18:56:15.993997+0000","last_clean":"2024-03-28T18:56:15.993997+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:56:15.993997+0000","last_undegraded":"2024-03-28T18:56:15.993997+0000","last_fullsized":"2024-03-28T18:56:15.993997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993504+0000","last_change":"2024-03-28T18:55:44.607744+0000","last_active":"2024-03-28T18:56:15.993504+0000","last_peered":"2024-03-28T18:56:15.993504+0000","last_clean":"2024-03-28T18:56:15.993504+0000","last_became_active":"2024-03-28T18:55:44.606993+0000","last_became_peered":"2024-03-28T18:55:44.606993+0000","last_unstale":"2024-03-28T18:56:15.993504+0000","last_undegraded":"2024-03-28T18:56:15.993504+0000","last_fullsized":"2024-03-28T18:56:15.993504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993475+0000","last_change":"2024-03-28T18:55:45.606307+0000","last_active":"2024-03-28T18:56:15.993475+0000","last_peered":"2024-03-28T18:56:15.993475+0000","last_clean":"2024-03-28T18:56:15.993475+0000","last_became_active":"2024-03-28T18:55:45.605980+0000","last_became_peered":"2024-03-28T18:55:45.605980+0000","last_unstale":"2024-03-28T18:56:15.993475+0000","last_undegraded":"2024-03-28T18:56:15.993475+0000","last_fullsized":"2024-03-28T18:56:15.993475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:52.485966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996297+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:56:15.996297+0000","last_peered":"2024-03-28T18:56:15.996297+0000","last_clean":"2024-03-28T18:56:15.996297+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:56:15.996297+0000","last_undegraded":"2024-03-28T18:56:15.996297+0000","last_fullsized":"2024-03-28T18:56:15.996297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996334+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:56:15.996334+0000","last_peered":"2024-03-28T18:56:15.996334+0000","last_clean":"2024-03-28T18:56:15.996334+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:56:15.996334+0000","last_undegraded":"2024-03-28T18:56:15.996334+0000","last_fullsized":"2024-03-28T18:56:15.996334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373024+0000","last_change":"2024-03-28T18:55:44.607323+0000","last_active":"2024-03-28T18:56:16.373024+0000","last_peered":"2024-03-28T18:56:16.373024+0000","last_clean":"2024-03-28T18:56:16.373024+0000","last_became_active":"2024-03-28T18:55:44.606579+0000","last_became_peered":"2024-03-28T18:55:44.606579+0000","last_unstale":"2024-03-28T18:56:16.373024+0000","last_undegraded":"2024-03-28T18:56:16.373024+0000","last_fullsized":"2024-03-28T18:56:16.373024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176822+0000","last_change":"2024-03-28T18:55:45.622760+0000","last_active":"2024-03-28T18:56:16.176822+0000","last_peered":"2024-03-28T18:56:16.176822+0000","last_clean":"2024-03-28T18:56:16.176822+0000","last_became_active":"2024-03-28T18:55:45.622353+0000","last_became_peered":"2024-03-28T18:55:45.622353+0000","last_unstale":"2024-03-28T18:56:16.176822+0000","last_undegraded":"2024-03-28T18:56:16.176822+0000","last_fullsized":"2024-03-28T18:56:16.176822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:04.349689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179332+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:56:16.179332+0000","last_peered":"2024-03-28T18:56:16.179332+0000","last_clean":"2024-03-28T18:56:16.179332+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:56:16.179332+0000","last_undegraded":"2024-03-28T18:56:16.179332+0000","last_fullsized":"2024-03-28T18:56:16.179332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993245+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:56:15.993245+0000","last_peered":"2024-03-28T18:56:15.993245+0000","last_clean":"2024-03-28T18:56:15.993245+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:56:15.993245+0000","last_undegraded":"2024-03-28T18:56:15.993245+0000","last_fullsized":"2024-03-28T18:56:15.993245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994698+0000","last_change":"2024-03-28T18:55:44.607237+0000","last_active":"2024-03-28T18:56:15.994698+0000","last_peered":"2024-03-28T18:56:15.994698+0000","last_clean":"2024-03-28T18:56:15.994698+0000","last_became_active":"2024-03-28T18:55:44.606960+0000","last_became_peered":"2024-03-28T18:55:44.606960+0000","last_unstale":"2024-03-28T18:56:15.994698+0000","last_undegraded":"2024-03-28T18:56:15.994698+0000","last_fullsized":"2024-03-28T18:56:15.994698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993370+0000","last_change":"2024-03-28T18:55:45.612992+0000","last_active":"2024-03-28T18:56:15.993370+0000","last_peered":"2024-03-28T18:56:15.993370+0000","last_clean":"2024-03-28T18:56:15.993370+0000","last_became_active":"2024-03-28T18:55:45.612125+0000","last_became_peered":"2024-03-28T18:55:45.612125+0000","last_unstale":"2024-03-28T18:56:15.993370+0000","last_undegraded":"2024-03-28T18:56:15.993370+0000","last_fullsized":"2024-03-28T18:56:15.993370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:35.898911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179159+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:56:16.179159+0000","last_peered":"2024-03-28T18:56:16.179159+0000","last_clean":"2024-03-28T18:56:16.179159+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:56:16.179159+0000","last_undegraded":"2024-03-28T18:56:16.179159+0000","last_fullsized":"2024-03-28T18:56:16.179159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166759+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:56:16.166759+0000","last_peered":"2024-03-28T18:56:16.166759+0000","last_clean":"2024-03-28T18:56:16.166759+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:56:16.166759+0000","last_undegraded":"2024-03-28T18:56:16.166759+0000","last_fullsized":"2024-03-28T18:56:16.166759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164552+0000","last_change":"2024-03-28T18:55:44.607218+0000","last_active":"2024-03-28T18:56:16.164552+0000","last_peered":"2024-03-28T18:56:16.164552+0000","last_clean":"2024-03-28T18:56:16.164552+0000","last_became_active":"2024-03-28T18:55:44.606852+0000","last_became_peered":"2024-03-28T18:55:44.606852+0000","last_unstale":"2024-03-28T18:56:16.164552+0000","last_undegraded":"2024-03-28T18:56:16.164552+0000","last_fullsized":"2024-03-28T18:56:16.164552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:36.624289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992021+0000","last_change":"2024-03-28T18:55:45.618919+0000","last_active":"2024-03-28T18:56:15.992021+0000","last_peered":"2024-03-28T18:56:15.992021+0000","last_clean":"2024-03-28T18:56:15.992021+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:56:15.992021+0000","last_undegraded":"2024-03-28T18:56:15.992021+0000","last_fullsized":"2024-03-28T18:56:15.992021+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:29.407508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178011+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:56:16.178011+0000","last_peered":"2024-03-28T18:56:16.178011+0000","last_clean":"2024-03-28T18:56:16.178011+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:56:16.178011+0000","last_undegraded":"2024-03-28T18:56:16.178011+0000","last_fullsized":"2024-03-28T18:56:16.178011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993501+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:56:15.993501+0000","last_peered":"2024-03-28T18:56:15.993501+0000","last_clean":"2024-03-28T18:56:15.993501+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:56:15.993501+0000","last_undegraded":"2024-03-28T18:56:15.993501+0000","last_fullsized":"2024-03-28T18:56:15.993501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.374029+0000","last_change":"2024-03-28T18:55:44.598881+0000","last_active":"2024-03-28T18:56:16.374029+0000","last_peered":"2024-03-28T18:56:16.374029+0000","last_clean":"2024-03-28T18:56:16.374029+0000","last_became_active":"2024-03-28T18:55:44.598532+0000","last_became_peered":"2024-03-28T18:55:44.598532+0000","last_unstale":"2024-03-28T18:56:16.374029+0000","last_undegraded":"2024-03-28T18:56:16.374029+0000","last_fullsized":"2024-03-28T18:56:16.374029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:44.004611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996863+0000","last_change":"2024-03-28T18:55:45.612572+0000","last_active":"2024-03-28T18:56:15.996863+0000","last_peered":"2024-03-28T18:56:15.996863+0000","last_clean":"2024-03-28T18:56:15.996863+0000","last_became_active":"2024-03-28T18:55:45.611844+0000","last_became_peered":"2024-03-28T18:55:45.611844+0000","last_unstale":"2024-03-28T18:56:15.996863+0000","last_undegraded":"2024-03-28T18:56:15.996863+0000","last_fullsized":"2024-03-28T18:56:15.996863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992765+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:56:15.992765+0000","last_peered":"2024-03-28T18:56:15.992765+0000","last_clean":"2024-03-28T18:56:15.992765+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:56:15.992765+0000","last_undegraded":"2024-03-28T18:56:15.992765+0000","last_fullsized":"2024-03-28T18:56:15.992765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993659+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:56:15.993659+0000","last_peered":"2024-03-28T18:56:15.993659+0000","last_clean":"2024-03-28T18:56:15.993659+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:56:15.993659+0000","last_undegraded":"2024-03-28T18:56:15.993659+0000","last_fullsized":"2024-03-28T18:56:15.993659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992080+0000","last_change":"2024-03-28T18:55:44.609644+0000","last_active":"2024-03-28T18:56:15.992080+0000","last_peered":"2024-03-28T18:56:15.992080+0000","last_clean":"2024-03-28T18:56:15.992080+0000","last_became_active":"2024-03-28T18:55:44.609317+0000","last_became_peered":"2024-03-28T18:55:44.609317+0000","last_unstale":"2024-03-28T18:56:15.992080+0000","last_undegraded":"2024-03-28T18:56:15.992080+0000","last_fullsized":"2024-03-28T18:56:15.992080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:37.662504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996075+0000","last_change":"2024-03-28T18:55:45.621708+0000","last_active":"2024-03-28T18:56:15.996075+0000","last_peered":"2024-03-28T18:56:15.996075+0000","last_clean":"2024-03-28T18:56:15.996075+0000","last_became_active":"2024-03-28T18:55:45.621366+0000","last_became_peered":"2024-03-28T18:55:45.621366+0000","last_unstale":"2024-03-28T18:56:15.996075+0000","last_undegraded":"2024-03-28T18:56:15.996075+0000","last_fullsized":"2024-03-28T18:56:15.996075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177583+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:56:16.177583+0000","last_peered":"2024-03-28T18:56:16.177583+0000","last_clean":"2024-03-28T18:56:16.177583+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:56:16.177583+0000","last_undegraded":"2024-03-28T18:56:16.177583+0000","last_fullsized":"2024-03-28T18:56:16.177583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996044+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:56:15.996044+0000","last_peered":"2024-03-28T18:56:15.996044+0000","last_clean":"2024-03-28T18:56:15.996044+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:56:15.996044+0000","last_undegraded":"2024-03-28T18:56:15.996044+0000","last_fullsized":"2024-03-28T18:56:15.996044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"26'9","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:20.908069+0000","last_change":"2024-03-28T18:55:44.608949+0000","last_active":"2024-03-28T18:56:20.908069+0000","last_peered":"2024-03-28T18:56:20.908069+0000","last_clean":"2024-03-28T18:56:20.908069+0000","last_became_active":"2024-03-28T18:55:44.608657+0000","last_became_peered":"2024-03-28T18:55:44.608657+0000","last_unstale":"2024-03-28T18:56:20.908069+0000","last_undegraded":"2024-03-28T18:56:20.908069+0000","last_fullsized":"2024-03-28T18:56:20.908069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:04.799314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990894+0000","last_change":"2024-03-28T18:55:45.610686+0000","last_active":"2024-03-28T18:56:15.990894+0000","last_peered":"2024-03-28T18:56:15.990894+0000","last_clean":"2024-03-28T18:56:15.990894+0000","last_became_active":"2024-03-28T18:55:45.610212+0000","last_became_peered":"2024-03-28T18:55:45.610212+0000","last_unstale":"2024-03-28T18:56:15.990894+0000","last_undegraded":"2024-03-28T18:56:15.990894+0000","last_fullsized":"2024-03-28T18:56:15.990894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372769+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:56:16.372769+0000","last_peered":"2024-03-28T18:56:16.372769+0000","last_clean":"2024-03-28T18:56:16.372769+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:56:16.372769+0000","last_undegraded":"2024-03-28T18:56:16.372769+0000","last_fullsized":"2024-03-28T18:56:16.372769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373687+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:56:16.373687+0000","last_peered":"2024-03-28T18:56:16.373687+0000","last_clean":"2024-03-28T18:56:16.373687+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:56:16.373687+0000","last_undegraded":"2024-03-28T18:56:16.373687+0000","last_fullsized":"2024-03-28T18:56:16.373687+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"23'3","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996740+0000","last_change":"2024-03-28T18:55:44.602762+0000","last_active":"2024-03-28T18:56:15.996740+0000","last_peered":"2024-03-28T18:56:15.996740+0000","last_clean":"2024-03-28T18:56:15.996740+0000","last_became_active":"2024-03-28T18:55:44.602424+0000","last_became_peered":"2024-03-28T18:55:44.602424+0000","last_unstale":"2024-03-28T18:56:15.996740+0000","last_undegraded":"2024-03-28T18:56:15.996740+0000","last_fullsized":"2024-03-28T18:56:15.996740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990431+0000","last_change":"2024-03-28T18:55:45.601277+0000","last_active":"2024-03-28T18:56:15.990431+0000","last_peered":"2024-03-28T18:56:15.990431+0000","last_clean":"2024-03-28T18:56:15.990431+0000","last_became_active":"2024-03-28T18:55:45.600945+0000","last_became_peered":"2024-03-28T18:55:45.600945+0000","last_unstale":"2024-03-28T18:56:15.990431+0000","last_undegraded":"2024-03-28T18:56:15.990431+0000","last_fullsized":"2024-03-28T18:56:15.990431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178326+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:56:16.178326+0000","last_peered":"2024-03-28T18:56:16.178326+0000","last_clean":"2024-03-28T18:56:16.178326+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:56:16.178326+0000","last_undegraded":"2024-03-28T18:56:16.178326+0000","last_fullsized":"2024-03-28T18:56:16.178326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993822+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:56:15.993822+0000","last_peered":"2024-03-28T18:56:15.993822+0000","last_clean":"2024-03-28T18:56:15.993822+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:56:15.993822+0000","last_undegraded":"2024-03-28T18:56:15.993822+0000","last_fullsized":"2024-03-28T18:56:15.993822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994697+0000","last_change":"2024-03-28T18:55:44.599432+0000","last_active":"2024-03-28T18:56:15.994697+0000","last_peered":"2024-03-28T18:56:15.994697+0000","last_clean":"2024-03-28T18:56:15.994697+0000","last_became_active":"2024-03-28T18:55:44.599123+0000","last_became_peered":"2024-03-28T18:55:44.599123+0000","last_unstale":"2024-03-28T18:56:15.994697+0000","last_undegraded":"2024-03-28T18:56:15.994697+0000","last_fullsized":"2024-03-28T18:56:15.994697+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994904+0000","last_change":"2024-03-28T18:55:45.624311+0000","last_active":"2024-03-28T18:56:15.994904+0000","last_peered":"2024-03-28T18:56:15.994904+0000","last_clean":"2024-03-28T18:56:15.994904+0000","last_became_active":"2024-03-28T18:55:45.624031+0000","last_became_peered":"2024-03-28T18:55:45.624031+0000","last_unstale":"2024-03-28T18:56:15.994904+0000","last_undegraded":"2024-03-28T18:56:15.994904+0000","last_fullsized":"2024-03-28T18:56:15.994904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166129+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:56:16.166129+0000","last_peered":"2024-03-28T18:56:16.166129+0000","last_clean":"2024-03-28T18:56:16.166129+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:56:16.166129+0000","last_undegraded":"2024-03-28T18:56:16.166129+0000","last_fullsized":"2024-03-28T18:56:16.166129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996267+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:56:15.996267+0000","last_peered":"2024-03-28T18:56:15.996267+0000","last_clean":"2024-03-28T18:56:15.996267+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:56:15.996267+0000","last_undegraded":"2024-03-28T18:56:15.996267+0000","last_fullsized":"2024-03-28T18:56:15.996267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165151+0000","last_change":"2024-03-28T18:55:44.593567+0000","last_active":"2024-03-28T18:56:16.165151+0000","last_peered":"2024-03-28T18:56:16.165151+0000","last_clean":"2024-03-28T18:56:16.165151+0000","last_became_active":"2024-03-28T18:55:44.593267+0000","last_became_peered":"2024-03-28T18:55:44.593267+0000","last_unstale":"2024-03-28T18:56:16.165151+0000","last_undegraded":"2024-03-28T18:56:16.165151+0000","last_fullsized":"2024-03-28T18:56:16.165151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993212+0000","last_change":"2024-03-28T18:55:45.598713+0000","last_active":"2024-03-28T18:56:15.993212+0000","last_peered":"2024-03-28T18:56:15.993212+0000","last_clean":"2024-03-28T18:56:15.993212+0000","last_became_active":"2024-03-28T18:55:45.597726+0000","last_became_peered":"2024-03-28T18:55:45.597726+0000","last_unstale":"2024-03-28T18:56:15.993212+0000","last_undegraded":"2024-03-28T18:56:15.993212+0000","last_fullsized":"2024-03-28T18:56:15.993212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:48.255745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373226+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:56:16.373226+0000","last_peered":"2024-03-28T18:56:16.373226+0000","last_clean":"2024-03-28T18:56:16.373226+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:56:16.373226+0000","last_undegraded":"2024-03-28T18:56:16.373226+0000","last_fullsized":"2024-03-28T18:56:16.373226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997012+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:56:15.997012+0000","last_peered":"2024-03-28T18:56:15.997012+0000","last_clean":"2024-03-28T18:56:15.997012+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:56:15.997012+0000","last_undegraded":"2024-03-28T18:56:15.997012+0000","last_fullsized":"2024-03-28T18:56:15.997012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"23'8","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989655+0000","last_change":"2024-03-28T18:55:44.610673+0000","last_active":"2024-03-28T18:56:15.989655+0000","last_peered":"2024-03-28T18:56:15.989655+0000","last_clean":"2024-03-28T18:56:15.989655+0000","last_became_active":"2024-03-28T18:55:44.610357+0000","last_became_peered":"2024-03-28T18:55:44.610357+0000","last_unstale":"2024-03-28T18:56:15.989655+0000","last_undegraded":"2024-03-28T18:56:15.989655+0000","last_fullsized":"2024-03-28T18:56:15.989655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:26.280548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177029+0000","last_change":"2024-03-28T18:55:45.609349+0000","last_active":"2024-03-28T18:56:16.177029+0000","last_peered":"2024-03-28T18:56:16.177029+0000","last_clean":"2024-03-28T18:56:16.177029+0000","last_became_active":"2024-03-28T18:55:45.608743+0000","last_became_peered":"2024-03-28T18:55:45.608743+0000","last_unstale":"2024-03-28T18:56:16.177029+0000","last_undegraded":"2024-03-28T18:56:16.177029+0000","last_fullsized":"2024-03-28T18:56:16.177029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996992+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:56:15.996992+0000","last_peered":"2024-03-28T18:56:15.996992+0000","last_clean":"2024-03-28T18:56:15.996992+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:56:15.996992+0000","last_undegraded":"2024-03-28T18:56:15.996992+0000","last_fullsized":"2024-03-28T18:56:15.996992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994769+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:56:15.994769+0000","last_peered":"2024-03-28T18:56:15.994769+0000","last_clean":"2024-03-28T18:56:15.994769+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:56:15.994769+0000","last_undegraded":"2024-03-28T18:56:15.994769+0000","last_fullsized":"2024-03-28T18:56:15.994769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989590+0000","last_change":"2024-03-28T18:55:44.601103+0000","last_active":"2024-03-28T18:56:15.989590+0000","last_peered":"2024-03-28T18:56:15.989590+0000","last_clean":"2024-03-28T18:56:15.989590+0000","last_became_active":"2024-03-28T18:55:44.599969+0000","last_became_peered":"2024-03-28T18:55:44.599969+0000","last_unstale":"2024-03-28T18:56:15.989590+0000","last_undegraded":"2024-03-28T18:56:15.989590+0000","last_fullsized":"2024-03-28T18:56:15.989590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:42.018654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994243+0000","last_change":"2024-03-28T18:55:45.594944+0000","last_active":"2024-03-28T18:56:15.994243+0000","last_peered":"2024-03-28T18:56:15.994243+0000","last_clean":"2024-03-28T18:56:15.994243+0000","last_became_active":"2024-03-28T18:55:45.594603+0000","last_became_peered":"2024-03-28T18:55:45.594603+0000","last_unstale":"2024-03-28T18:56:15.994243+0000","last_undegraded":"2024-03-28T18:56:15.994243+0000","last_fullsized":"2024-03-28T18:56:15.994243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373359+0000","last_change":"2024-03-28T18:55:44.602366+0000","last_active":"2024-03-28T18:56:16.373359+0000","last_peered":"2024-03-28T18:56:16.373359+0000","last_clean":"2024-03-28T18:56:16.373359+0000","last_became_active":"2024-03-28T18:55:44.602033+0000","last_became_peered":"2024-03-28T18:55:44.602033+0000","last_unstale":"2024-03-28T18:56:16.373359+0000","last_undegraded":"2024-03-28T18:56:16.373359+0000","last_fullsized":"2024-03-28T18:56:16.373359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994424+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:56:15.994424+0000","last_peered":"2024-03-28T18:56:15.994424+0000","last_clean":"2024-03-28T18:56:15.994424+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:56:15.994424+0000","last_undegraded":"2024-03-28T18:56:15.994424+0000","last_fullsized":"2024-03-28T18:56:15.994424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995004+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:56:15.995004+0000","last_peered":"2024-03-28T18:56:15.995004+0000","last_clean":"2024-03-28T18:56:15.995004+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:56:15.995004+0000","last_undegraded":"2024-03-28T18:56:15.995004+0000","last_fullsized":"2024-03-28T18:56:15.995004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373198+0000","last_change":"2024-03-28T18:55:45.620559+0000","last_active":"2024-03-28T18:56:16.373198+0000","last_peered":"2024-03-28T18:56:16.373198+0000","last_clean":"2024-03-28T18:56:16.373198+0000","last_became_active":"2024-03-28T18:55:45.620249+0000","last_became_peered":"2024-03-28T18:55:45.620249+0000","last_unstale":"2024-03-28T18:56:16.373198+0000","last_undegraded":"2024-03-28T18:56:16.373198+0000","last_fullsized":"2024-03-28T18:56:16.373198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"23'4","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994418+0000","last_change":"2024-03-28T18:55:44.612226+0000","last_active":"2024-03-28T18:56:15.994418+0000","last_peered":"2024-03-28T18:56:15.994418+0000","last_clean":"2024-03-28T18:56:15.994418+0000","last_became_active":"2024-03-28T18:55:44.611921+0000","last_became_peered":"2024-03-28T18:55:44.611921+0000","last_unstale":"2024-03-28T18:56:15.994418+0000","last_undegraded":"2024-03-28T18:56:15.994418+0000","last_fullsized":"2024-03-28T18:56:15.994418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989828+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:56:15.989828+0000","last_peered":"2024-03-28T18:56:15.989828+0000","last_clean":"2024-03-28T18:56:15.989828+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:56:15.989828+0000","last_undegraded":"2024-03-28T18:56:15.989828+0000","last_fullsized":"2024-03-28T18:56:15.989828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990026+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:56:15.990026+0000","last_peered":"2024-03-28T18:56:15.990026+0000","last_clean":"2024-03-28T18:56:15.990026+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:56:15.990026+0000","last_undegraded":"2024-03-28T18:56:15.990026+0000","last_fullsized":"2024-03-28T18:56:15.990026+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996154+0000","last_change":"2024-03-28T18:55:45.612727+0000","last_active":"2024-03-28T18:56:15.996154+0000","last_peered":"2024-03-28T18:56:15.996154+0000","last_clean":"2024-03-28T18:56:15.996154+0000","last_became_active":"2024-03-28T18:55:45.610154+0000","last_became_peered":"2024-03-28T18:55:45.610154+0000","last_unstale":"2024-03-28T18:56:15.996154+0000","last_undegraded":"2024-03-28T18:56:15.996154+0000","last_fullsized":"2024-03-28T18:56:15.996154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994188+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:56:15.994188+0000","last_peered":"2024-03-28T18:56:15.994188+0000","last_clean":"2024-03-28T18:56:15.994188+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:56:15.994188+0000","last_undegraded":"2024-03-28T18:56:15.994188+0000","last_fullsized":"2024-03-28T18:56:15.994188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373560+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:56:16.373560+0000","last_peered":"2024-03-28T18:56:16.373560+0000","last_clean":"2024-03-28T18:56:16.373560+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:56:16.373560+0000","last_undegraded":"2024-03-28T18:56:16.373560+0000","last_fullsized":"2024-03-28T18:56:16.373560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993632+0000","last_change":"2024-03-28T18:55:44.592835+0000","last_active":"2024-03-28T18:56:15.993632+0000","last_peered":"2024-03-28T18:56:15.993632+0000","last_clean":"2024-03-28T18:56:15.993632+0000","last_became_active":"2024-03-28T18:55:44.592505+0000","last_became_peered":"2024-03-28T18:55:44.592505+0000","last_unstale":"2024-03-28T18:56:15.993632+0000","last_undegraded":"2024-03-28T18:56:15.993632+0000","last_fullsized":"2024-03-28T18:56:15.993632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:58.286421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165539+0000","last_change":"2024-03-28T18:55:45.611322+0000","last_active":"2024-03-28T18:56:16.165539+0000","last_peered":"2024-03-28T18:56:16.165539+0000","last_clean":"2024-03-28T18:56:16.165539+0000","last_became_active":"2024-03-28T18:55:45.610958+0000","last_became_peered":"2024-03-28T18:55:45.610958+0000","last_unstale":"2024-03-28T18:56:16.165539+0000","last_undegraded":"2024-03-28T18:56:16.165539+0000","last_fullsized":"2024-03-28T18:56:16.165539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990629+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:56:15.990629+0000","last_peered":"2024-03-28T18:56:15.990629+0000","last_clean":"2024-03-28T18:56:15.990629+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:56:15.990629+0000","last_undegraded":"2024-03-28T18:56:15.990629+0000","last_fullsized":"2024-03-28T18:56:15.990629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165083+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:56:16.165083+0000","last_peered":"2024-03-28T18:56:16.165083+0000","last_clean":"2024-03-28T18:56:16.165083+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:56:16.165083+0000","last_undegraded":"2024-03-28T18:56:16.165083+0000","last_fullsized":"2024-03-28T18:56:16.165083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991514+0000","last_change":"2024-03-28T18:55:44.608283+0000","last_active":"2024-03-28T18:56:15.991514+0000","last_peered":"2024-03-28T18:56:15.991514+0000","last_clean":"2024-03-28T18:56:15.991514+0000","last_became_active":"2024-03-28T18:55:44.607999+0000","last_became_peered":"2024-03-28T18:55:44.607999+0000","last_unstale":"2024-03-28T18:56:15.991514+0000","last_undegraded":"2024-03-28T18:56:15.991514+0000","last_fullsized":"2024-03-28T18:56:15.991514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:22.669778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991790+0000","last_change":"2024-03-28T18:55:45.618390+0000","last_active":"2024-03-28T18:56:15.991790+0000","last_peered":"2024-03-28T18:56:15.991790+0000","last_clean":"2024-03-28T18:56:15.991790+0000","last_became_active":"2024-03-28T18:55:45.617897+0000","last_became_peered":"2024-03-28T18:55:45.617897+0000","last_unstale":"2024-03-28T18:56:15.991790+0000","last_undegraded":"2024-03-28T18:56:15.991790+0000","last_fullsized":"2024-03-28T18:56:15.991790+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:16.520575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992753+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:56:15.992753+0000","last_peered":"2024-03-28T18:56:15.992753+0000","last_clean":"2024-03-28T18:56:15.992753+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:56:15.992753+0000","last_undegraded":"2024-03-28T18:56:15.992753+0000","last_fullsized":"2024-03-28T18:56:15.992753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996140+0000","last_change":"2024-03-28T18:55:45.624938+0000","last_active":"2024-03-28T18:56:15.996140+0000","last_peered":"2024-03-28T18:56:15.996140+0000","last_clean":"2024-03-28T18:56:15.996140+0000","last_became_active":"2024-03-28T18:55:45.624362+0000","last_became_peered":"2024-03-28T18:55:45.624362+0000","last_unstale":"2024-03-28T18:56:15.996140+0000","last_undegraded":"2024-03-28T18:56:15.996140+0000","last_fullsized":"2024-03-28T18:56:15.996140+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992964+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:56:15.992964+0000","last_peered":"2024-03-28T18:56:15.992964+0000","last_clean":"2024-03-28T18:56:15.992964+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:56:15.992964+0000","last_undegraded":"2024-03-28T18:56:15.992964+0000","last_fullsized":"2024-03-28T18:56:15.992964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995980+0000","last_change":"2024-03-28T18:55:45.610168+0000","last_active":"2024-03-28T18:56:15.995980+0000","last_peered":"2024-03-28T18:56:15.995980+0000","last_clean":"2024-03-28T18:56:15.995980+0000","last_became_active":"2024-03-28T18:55:45.609653+0000","last_became_peered":"2024-03-28T18:55:45.609653+0000","last_unstale":"2024-03-28T18:56:15.995980+0000","last_undegraded":"2024-03-28T18:56:15.995980+0000","last_fullsized":"2024-03-28T18:56:15.995980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995799+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:56:15.995799+0000","last_peered":"2024-03-28T18:56:15.995799+0000","last_clean":"2024-03-28T18:56:15.995799+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:56:15.995799+0000","last_undegraded":"2024-03-28T18:56:15.995799+0000","last_fullsized":"2024-03-28T18:56:15.995799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996052+0000","last_change":"2024-03-28T18:55:45.613701+0000","last_active":"2024-03-28T18:56:15.996052+0000","last_peered":"2024-03-28T18:56:15.996052+0000","last_clean":"2024-03-28T18:56:15.996052+0000","last_became_active":"2024-03-28T18:55:45.613310+0000","last_became_peered":"2024-03-28T18:55:45.613310+0000","last_unstale":"2024-03-28T18:56:15.996052+0000","last_undegraded":"2024-03-28T18:56:15.996052+0000","last_fullsized":"2024-03-28T18:56:15.996052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373539+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:56:16.373539+0000","last_peered":"2024-03-28T18:56:16.373539+0000","last_clean":"2024-03-28T18:56:16.373539+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:56:16.373539+0000","last_undegraded":"2024-03-28T18:56:16.373539+0000","last_fullsized":"2024-03-28T18:56:16.373539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177308+0000","last_change":"2024-03-28T18:55:45.615119+0000","last_active":"2024-03-28T18:56:16.177308+0000","last_peered":"2024-03-28T18:56:16.177308+0000","last_clean":"2024-03-28T18:56:16.177308+0000","last_became_active":"2024-03-28T18:55:45.613841+0000","last_became_peered":"2024-03-28T18:55:45.613841+0000","last_unstale":"2024-03-28T18:56:16.177308+0000","last_undegraded":"2024-03-28T18:56:16.177308+0000","last_fullsized":"2024-03-28T18:56:16.177308+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991068+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:56:15.991068+0000","last_peered":"2024-03-28T18:56:15.991068+0000","last_clean":"2024-03-28T18:56:15.991068+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:56:15.991068+0000","last_undegraded":"2024-03-28T18:56:15.991068+0000","last_fullsized":"2024-03-28T18:56:15.991068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164511+0000","last_change":"2024-03-28T18:55:45.619605+0000","last_active":"2024-03-28T18:56:16.164511+0000","last_peered":"2024-03-28T18:56:16.164511+0000","last_clean":"2024-03-28T18:56:16.164511+0000","last_became_active":"2024-03-28T18:55:45.618473+0000","last_became_peered":"2024-03-28T18:55:45.618473+0000","last_unstale":"2024-03-28T18:56:16.164511+0000","last_undegraded":"2024-03-28T18:56:16.164511+0000","last_fullsized":"2024-03-28T18:56:16.164511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996253+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:56:15.996253+0000","last_peered":"2024-03-28T18:56:15.996253+0000","last_clean":"2024-03-28T18:56:15.996253+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:56:15.996253+0000","last_undegraded":"2024-03-28T18:56:15.996253+0000","last_fullsized":"2024-03-28T18:56:15.996253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176839+0000","last_change":"2024-03-28T18:55:45.612284+0000","last_active":"2024-03-28T18:56:16.176839+0000","last_peered":"2024-03-28T18:56:16.176839+0000","last_clean":"2024-03-28T18:56:16.176839+0000","last_became_active":"2024-03-28T18:55:45.607218+0000","last_became_peered":"2024-03-28T18:55:45.607218+0000","last_unstale":"2024-03-28T18:56:16.176839+0000","last_undegraded":"2024-03-28T18:56:16.176839+0000","last_fullsized":"2024-03-28T18:56:16.176839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993957+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:56:15.993957+0000","last_peered":"2024-03-28T18:56:15.993957+0000","last_clean":"2024-03-28T18:56:15.993957+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:56:15.993957+0000","last_undegraded":"2024-03-28T18:56:15.993957+0000","last_fullsized":"2024-03-28T18:56:15.993957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992957+0000","last_change":"2024-03-28T18:55:45.607609+0000","last_active":"2024-03-28T18:56:15.992957+0000","last_peered":"2024-03-28T18:56:15.992957+0000","last_clean":"2024-03-28T18:56:15.992957+0000","last_became_active":"2024-03-28T18:55:45.602582+0000","last_became_peered":"2024-03-28T18:55:45.602582+0000","last_unstale":"2024-03-28T18:56:15.992957+0000","last_undegraded":"2024-03-28T18:56:15.992957+0000","last_fullsized":"2024-03-28T18:56:15.992957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:06:05.335715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.168035+0000","last_change":"2024-03-28T18:56:16.167085+0000","last_active":"2024-03-28T18:56:16.168035+0000","last_peered":"2024-03-28T18:56:16.168035+0000","last_clean":"2024-03-28T18:56:16.168035+0000","last_became_active":"2024-03-28T18:56:16.166729+0000","last_became_peered":"2024-03-28T18:56:16.166729+0000","last_unstale":"2024-03-28T18:56:16.168035+0000","last_undegraded":"2024-03-28T18:56:16.168035+0000","last_fullsized":"2024-03-28T18:56:16.168035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:49.364836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991762+0000","last_change":"2024-03-28T18:55:45.607886+0000","last_active":"2024-03-28T18:56:15.991762+0000","last_peered":"2024-03-28T18:56:15.991762+0000","last_clean":"2024-03-28T18:56:15.991762+0000","last_became_active":"2024-03-28T18:55:45.605964+0000","last_became_peered":"2024-03-28T18:55:45.605964+0000","last_unstale":"2024-03-28T18:56:15.991762+0000","last_undegraded":"2024-03-28T18:56:15.991762+0000","last_fullsized":"2024-03-28T18:56:15.991762+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:45.986654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995008+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:56:15.995008+0000","last_peered":"2024-03-28T18:56:15.995008+0000","last_clean":"2024-03-28T18:56:15.995008+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:56:15.995008+0000","last_undegraded":"2024-03-28T18:56:15.995008+0000","last_fullsized":"2024-03-28T18:56:15.995008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178931+0000","last_change":"2024-03-28T18:55:45.618031+0000","last_active":"2024-03-28T18:56:16.178931+0000","last_peered":"2024-03-28T18:56:16.178931+0000","last_clean":"2024-03-28T18:56:16.178931+0000","last_became_active":"2024-03-28T18:55:45.617719+0000","last_became_peered":"2024-03-28T18:55:45.617719+0000","last_unstale":"2024-03-28T18:56:16.178931+0000","last_undegraded":"2024-03-28T18:56:16.178931+0000","last_fullsized":"2024-03-28T18:56:16.178931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989874+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:56:15.989874+0000","last_peered":"2024-03-28T18:56:15.989874+0000","last_clean":"2024-03-28T18:56:15.989874+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:56:15.989874+0000","last_undegraded":"2024-03-28T18:56:15.989874+0000","last_fullsized":"2024-03-28T18:56:15.989874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993563+0000","last_change":"2024-03-28T18:55:45.603783+0000","last_active":"2024-03-28T18:56:15.993563+0000","last_peered":"2024-03-28T18:56:15.993563+0000","last_clean":"2024-03-28T18:56:15.993563+0000","last_became_active":"2024-03-28T18:55:45.603474+0000","last_became_peered":"2024-03-28T18:55:45.603474+0000","last_unstale":"2024-03-28T18:56:15.993563+0000","last_undegraded":"2024-03-28T18:56:15.993563+0000","last_fullsized":"2024-03-28T18:56:15.993563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:42.328182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.182051+0000","last_change":"2024-03-28T18:56:16.181039+0000","last_active":"2024-03-28T18:56:16.182051+0000","last_peered":"2024-03-28T18:56:16.182051+0000","last_clean":"2024-03-28T18:56:16.182051+0000","last_became_active":"2024-03-28T18:56:16.180227+0000","last_became_peered":"2024-03-28T18:56:16.180227+0000","last_unstale":"2024-03-28T18:56:16.182051+0000","last_undegraded":"2024-03-28T18:56:16.182051+0000","last_fullsized":"2024-03-28T18:56:16.182051+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:54.706529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989997+0000","last_change":"2024-03-28T18:55:45.618912+0000","last_active":"2024-03-28T18:56:15.989997+0000","last_peered":"2024-03-28T18:56:15.989997+0000","last_clean":"2024-03-28T18:56:15.989997+0000","last_became_active":"2024-03-28T18:55:45.618554+0000","last_became_peered":"2024-03-28T18:55:45.618554+0000","last_unstale":"2024-03-28T18:56:15.989997+0000","last_undegraded":"2024-03-28T18:56:15.989997+0000","last_fullsized":"2024-03-28T18:56:15.989997+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373395+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:56:16.373395+0000","last_peered":"2024-03-28T18:56:16.373395+0000","last_clean":"2024-03-28T18:56:16.373395+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:56:16.373395+0000","last_undegraded":"2024-03-28T18:56:16.373395+0000","last_fullsized":"2024-03-28T18:56:16.373395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164597+0000","last_change":"2024-03-28T18:55:45.616420+0000","last_active":"2024-03-28T18:56:16.164597+0000","last_peered":"2024-03-28T18:56:16.164597+0000","last_clean":"2024-03-28T18:56:16.164597+0000","last_became_active":"2024-03-28T18:55:45.614806+0000","last_became_peered":"2024-03-28T18:55:45.614806+0000","last_unstale":"2024-03-28T18:56:16.164597+0000","last_undegraded":"2024-03-28T18:56:16.164597+0000","last_fullsized":"2024-03-28T18:56:16.164597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993436+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:56:15.993436+0000","last_peered":"2024-03-28T18:56:15.993436+0000","last_clean":"2024-03-28T18:56:15.993436+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:56:15.993436+0000","last_undegraded":"2024-03-28T18:56:15.993436+0000","last_fullsized":"2024-03-28T18:56:15.993436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164798+0000","last_change":"2024-03-28T18:55:45.606507+0000","last_active":"2024-03-28T18:56:16.164798+0000","last_peered":"2024-03-28T18:56:16.164798+0000","last_clean":"2024-03-28T18:56:16.164798+0000","last_became_active":"2024-03-28T18:55:45.606205+0000","last_became_peered":"2024-03-28T18:55:45.606205+0000","last_unstale":"2024-03-28T18:56:16.164798+0000","last_undegraded":"2024-03-28T18:56:16.164798+0000","last_fullsized":"2024-03-28T18:56:16.164798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178591+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:56:16.178591+0000","last_peered":"2024-03-28T18:56:16.178591+0000","last_clean":"2024-03-28T18:56:16.178591+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:56:16.178591+0000","last_undegraded":"2024-03-28T18:56:16.178591+0000","last_fullsized":"2024-03-28T18:56:16.178591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994460+0000","last_change":"2024-03-28T18:55:45.607890+0000","last_active":"2024-03-28T18:56:15.994460+0000","last_peered":"2024-03-28T18:56:15.994460+0000","last_clean":"2024-03-28T18:56:15.994460+0000","last_became_active":"2024-03-28T18:55:45.607590+0000","last_became_peered":"2024-03-28T18:55:45.607590+0000","last_unstale":"2024-03-28T18:56:15.994460+0000","last_undegraded":"2024-03-28T18:56:15.994460+0000","last_fullsized":"2024-03-28T18:56:15.994460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993121+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:56:15.993121+0000","last_peered":"2024-03-28T18:56:15.993121+0000","last_clean":"2024-03-28T18:56:15.993121+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:56:15.993121+0000","last_undegraded":"2024-03-28T18:56:15.993121+0000","last_fullsized":"2024-03-28T18:56:15.993121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372260+0000","last_change":"2024-03-28T18:55:45.608851+0000","last_active":"2024-03-28T18:56:16.372260+0000","last_peered":"2024-03-28T18:56:16.372260+0000","last_clean":"2024-03-28T18:56:16.372260+0000","last_became_active":"2024-03-28T18:55:45.608496+0000","last_became_peered":"2024-03-28T18:55:45.608496+0000","last_unstale":"2024-03-28T18:56:16.372260+0000","last_undegraded":"2024-03-28T18:56:16.372260+0000","last_fullsized":"2024-03-28T18:56:16.372260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997752+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:56:15.997752+0000","last_peered":"2024-03-28T18:56:15.997752+0000","last_clean":"2024-03-28T18:56:15.997752+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:56:15.997752+0000","last_undegraded":"2024-03-28T18:56:15.997752+0000","last_fullsized":"2024-03-28T18:56:15.997752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996035+0000","last_change":"2024-03-28T18:55:45.619501+0000","last_active":"2024-03-28T18:56:15.996035+0000","last_peered":"2024-03-28T18:56:15.996035+0000","last_clean":"2024-03-28T18:56:15.996035+0000","last_became_active":"2024-03-28T18:55:45.619212+0000","last_became_peered":"2024-03-28T18:55:45.619212+0000","last_unstale":"2024-03-28T18:56:15.996035+0000","last_undegraded":"2024-03-28T18:56:15.996035+0000","last_fullsized":"2024-03-28T18:56:15.996035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.377790+0000","last_change":"2024-03-28T18:56:16.375984+0000","last_active":"2024-03-28T18:56:16.377790+0000","last_peered":"2024-03-28T18:56:16.377790+0000","last_clean":"2024-03-28T18:56:16.377790+0000","last_became_active":"2024-03-28T18:56:16.375290+0000","last_became_peered":"2024-03-28T18:56:16.375290+0000","last_unstale":"2024-03-28T18:56:16.377790+0000","last_undegraded":"2024-03-28T18:56:16.377790+0000","last_fullsized":"2024-03-28T18:56:16.377790+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:12.576993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166458+0000","last_change":"2024-03-28T18:55:45.615225+0000","last_active":"2024-03-28T18:56:16.166458+0000","last_peered":"2024-03-28T18:56:16.166458+0000","last_clean":"2024-03-28T18:56:16.166458+0000","last_became_active":"2024-03-28T18:55:45.613602+0000","last_became_peered":"2024-03-28T18:55:45.613602+0000","last_unstale":"2024-03-28T18:56:16.166458+0000","last_undegraded":"2024-03-28T18:56:16.166458+0000","last_fullsized":"2024-03-28T18:56:16.166458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994526+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:56:15.994526+0000","last_peered":"2024-03-28T18:56:15.994526+0000","last_clean":"2024-03-28T18:56:15.994526+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:56:15.994526+0000","last_undegraded":"2024-03-28T18:56:15.994526+0000","last_fullsized":"2024-03-28T18:56:15.994526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995808+0000","last_change":"2024-03-28T18:55:45.609471+0000","last_active":"2024-03-28T18:56:15.995808+0000","last_peered":"2024-03-28T18:56:15.995808+0000","last_clean":"2024-03-28T18:56:15.995808+0000","last_became_active":"2024-03-28T18:55:45.609132+0000","last_became_peered":"2024-03-28T18:55:45.609132+0000","last_unstale":"2024-03-28T18:56:15.995808+0000","last_undegraded":"2024-03-28T18:56:15.995808+0000","last_fullsized":"2024-03-28T18:56:15.995808+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372643+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:56:16.372643+0000","last_peered":"2024-03-28T18:56:16.372643+0000","last_clean":"2024-03-28T18:56:16.372643+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:56:16.372643+0000","last_undegraded":"2024-03-28T18:56:16.372643+0000","last_fullsized":"2024-03-28T18:56:16.372643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178130+0000","last_change":"2024-03-28T18:55:45.617501+0000","last_active":"2024-03-28T18:56:16.178130+0000","last_peered":"2024-03-28T18:56:16.178130+0000","last_clean":"2024-03-28T18:56:16.178130+0000","last_became_active":"2024-03-28T18:55:45.617180+0000","last_became_peered":"2024-03-28T18:55:45.617180+0000","last_unstale":"2024-03-28T18:56:16.178130+0000","last_undegraded":"2024-03-28T18:56:16.178130+0000","last_fullsized":"2024-03-28T18:56:16.178130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990476+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:56:15.990476+0000","last_peered":"2024-03-28T18:56:15.990476+0000","last_clean":"2024-03-28T18:56:15.990476+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:56:15.990476+0000","last_undegraded":"2024-03-28T18:56:15.990476+0000","last_fullsized":"2024-03-28T18:56:15.990476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990313+0000","last_change":"2024-03-28T18:55:45.609169+0000","last_active":"2024-03-28T18:56:15.990313+0000","last_peered":"2024-03-28T18:56:15.990313+0000","last_clean":"2024-03-28T18:56:15.990313+0000","last_became_active":"2024-03-28T18:55:45.601948+0000","last_became_peered":"2024-03-28T18:55:45.601948+0000","last_unstale":"2024-03-28T18:56:15.990313+0000","last_undegraded":"2024-03-28T18:56:15.990313+0000","last_fullsized":"2024-03-28T18:56:15.990313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166023+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:56:16.166023+0000","last_peered":"2024-03-28T18:56:16.166023+0000","last_clean":"2024-03-28T18:56:16.166023+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:56:16.166023+0000","last_undegraded":"2024-03-28T18:56:16.166023+0000","last_fullsized":"2024-03-28T18:56:16.166023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993150+0000","last_change":"2024-03-28T18:55:45.616238+0000","last_active":"2024-03-28T18:56:15.993150+0000","last_peered":"2024-03-28T18:56:15.993150+0000","last_clean":"2024-03-28T18:56:15.993150+0000","last_became_active":"2024-03-28T18:55:45.615371+0000","last_became_peered":"2024-03-28T18:55:45.615371+0000","last_unstale":"2024-03-28T18:56:15.993150+0000","last_undegraded":"2024-03-28T18:56:15.993150+0000","last_fullsized":"2024-03-28T18:56:15.993150+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:14.661134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993580+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:56:15.993580+0000","last_peered":"2024-03-28T18:56:15.993580+0000","last_clean":"2024-03-28T18:56:15.993580+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:56:15.993580+0000","last_undegraded":"2024-03-28T18:56:15.993580+0000","last_fullsized":"2024-03-28T18:56:15.993580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177663+0000","last_change":"2024-03-28T18:55:45.613289+0000","last_active":"2024-03-28T18:56:16.177663+0000","last_peered":"2024-03-28T18:56:16.177663+0000","last_clean":"2024-03-28T18:56:16.177663+0000","last_became_active":"2024-03-28T18:55:45.610917+0000","last_became_peered":"2024-03-28T18:55:45.610917+0000","last_unstale":"2024-03-28T18:56:16.177663+0000","last_undegraded":"2024-03-28T18:56:16.177663+0000","last_fullsized":"2024-03-28T18:56:16.177663+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993822+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:56:15.993822+0000","last_peered":"2024-03-28T18:56:15.993822+0000","last_clean":"2024-03-28T18:56:15.993822+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:56:15.993822+0000","last_undegraded":"2024-03-28T18:56:15.993822+0000","last_fullsized":"2024-03-28T18:56:15.993822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994722+0000","last_change":"2024-03-28T18:55:45.621789+0000","last_active":"2024-03-28T18:56:15.994722+0000","last_peered":"2024-03-28T18:56:15.994722+0000","last_clean":"2024-03-28T18:56:15.994722+0000","last_became_active":"2024-03-28T18:55:45.621163+0000","last_became_peered":"2024-03-28T18:55:45.621163+0000","last_unstale":"2024-03-28T18:56:15.994722+0000","last_undegraded":"2024-03-28T18:56:15.994722+0000","last_fullsized":"2024-03-28T18:56:15.994722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179217+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:56:16.179217+0000","last_peered":"2024-03-28T18:56:16.179217+0000","last_clean":"2024-03-28T18:56:16.179217+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:56:16.179217+0000","last_undegraded":"2024-03-28T18:56:16.179217+0000","last_fullsized":"2024-03-28T18:56:16.179217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991623+0000","last_change":"2024-03-28T18:55:45.603774+0000","last_active":"2024-03-28T18:56:15.991623+0000","last_peered":"2024-03-28T18:56:15.991623+0000","last_clean":"2024-03-28T18:56:15.991623+0000","last_became_active":"2024-03-28T18:55:45.603451+0000","last_became_peered":"2024-03-28T18:55:45.603451+0000","last_unstale":"2024-03-28T18:56:15.991623+0000","last_undegraded":"2024-03-28T18:56:15.991623+0000","last_fullsized":"2024-03-28T18:56:15.991623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989714+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:56:15.989714+0000","last_peered":"2024-03-28T18:56:15.989714+0000","last_clean":"2024-03-28T18:56:15.989714+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:56:15.989714+0000","last_undegraded":"2024-03-28T18:56:15.989714+0000","last_fullsized":"2024-03-28T18:56:15.989714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994162+0000","last_change":"2024-03-28T18:55:45.616755+0000","last_active":"2024-03-28T18:56:15.994162+0000","last_peered":"2024-03-28T18:56:15.994162+0000","last_clean":"2024-03-28T18:56:15.994162+0000","last_became_active":"2024-03-28T18:55:45.614611+0000","last_became_peered":"2024-03-28T18:55:45.614611+0000","last_unstale":"2024-03-28T18:56:15.994162+0000","last_undegraded":"2024-03-28T18:56:15.994162+0000","last_fullsized":"2024-03-28T18:56:15.994162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:06.327795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.377155+0000","last_change":"2024-03-28T18:56:16.375743+0000","last_active":"2024-03-28T18:56:16.377155+0000","last_peered":"2024-03-28T18:56:16.377155+0000","last_clean":"2024-03-28T18:56:16.377155+0000","last_became_active":"2024-03-28T18:56:16.375316+0000","last_became_peered":"2024-03-28T18:56:16.375316+0000","last_unstale":"2024-03-28T18:56:16.377155+0000","last_undegraded":"2024-03-28T18:56:16.377155+0000","last_fullsized":"2024-03-28T18:56:16.377155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:27.569718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995729+0000","last_change":"2024-03-28T18:55:45.623195+0000","last_active":"2024-03-28T18:56:15.995729+0000","last_peered":"2024-03-28T18:56:15.995729+0000","last_clean":"2024-03-28T18:56:15.995729+0000","last_became_active":"2024-03-28T18:55:45.622508+0000","last_became_peered":"2024-03-28T18:55:45.622508+0000","last_unstale":"2024-03-28T18:56:15.995729+0000","last_undegraded":"2024-03-28T18:56:15.995729+0000","last_fullsized":"2024-03-28T18:56:15.995729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995605+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:56:15.995605+0000","last_peered":"2024-03-28T18:56:15.995605+0000","last_clean":"2024-03-28T18:56:15.995605+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:56:15.995605+0000","last_undegraded":"2024-03-28T18:56:15.995605+0000","last_fullsized":"2024-03-28T18:56:15.995605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996676+0000","last_change":"2024-03-28T18:55:45.618841+0000","last_active":"2024-03-28T18:56:15.996676+0000","last_peered":"2024-03-28T18:56:15.996676+0000","last_clean":"2024-03-28T18:56:15.996676+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:56:15.996676+0000","last_undegraded":"2024-03-28T18:56:15.996676+0000","last_fullsized":"2024-03-28T18:56:15.996676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177448+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:56:16.177448+0000","last_peered":"2024-03-28T18:56:16.177448+0000","last_clean":"2024-03-28T18:56:16.177448+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:56:16.177448+0000","last_undegraded":"2024-03-28T18:56:16.177448+0000","last_fullsized":"2024-03-28T18:56:16.177448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990122+0000","last_change":"2024-03-28T18:55:45.602886+0000","last_active":"2024-03-28T18:56:15.990122+0000","last_peered":"2024-03-28T18:56:15.990122+0000","last_clean":"2024-03-28T18:56:15.990122+0000","last_became_active":"2024-03-28T18:55:45.602610+0000","last_became_peered":"2024-03-28T18:55:45.602610+0000","last_unstale":"2024-03-28T18:56:15.990122+0000","last_undegraded":"2024-03-28T18:56:15.990122+0000","last_fullsized":"2024-03-28T18:56:15.990122+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996124+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:56:15.996124+0000","last_peered":"2024-03-28T18:56:15.996124+0000","last_clean":"2024-03-28T18:56:15.996124+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:56:15.996124+0000","last_undegraded":"2024-03-28T18:56:15.996124+0000","last_fullsized":"2024-03-28T18:56:15.996124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993035+0000","last_change":"2024-03-28T18:55:45.608673+0000","last_active":"2024-03-28T18:56:15.993035+0000","last_peered":"2024-03-28T18:56:15.993035+0000","last_clean":"2024-03-28T18:56:15.993035+0000","last_became_active":"2024-03-28T18:55:45.608376+0000","last_became_peered":"2024-03-28T18:55:45.608376+0000","last_unstale":"2024-03-28T18:56:15.993035+0000","last_undegraded":"2024-03-28T18:56:15.993035+0000","last_fullsized":"2024-03-28T18:56:15.993035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:31.781549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372875+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:56:16.372875+0000","last_peered":"2024-03-28T18:56:16.372875+0000","last_clean":"2024-03-28T18:56:16.372875+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:56:16.372875+0000","last_undegraded":"2024-03-28T18:56:16.372875+0000","last_fullsized":"2024-03-28T18:56:16.372875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176679+0000","last_change":"2024-03-28T18:55:45.609565+0000","last_active":"2024-03-28T18:56:16.176679+0000","last_peered":"2024-03-28T18:56:16.176679+0000","last_clean":"2024-03-28T18:56:16.176679+0000","last_became_active":"2024-03-28T18:55:45.609082+0000","last_became_peered":"2024-03-28T18:55:45.609082+0000","last_unstale":"2024-03-28T18:56:16.176679+0000","last_undegraded":"2024-03-28T18:56:16.176679+0000","last_fullsized":"2024-03-28T18:56:16.176679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996230+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:56:15.996230+0000","last_peered":"2024-03-28T18:56:15.996230+0000","last_clean":"2024-03-28T18:56:15.996230+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:56:15.996230+0000","last_undegraded":"2024-03-28T18:56:15.996230+0000","last_fullsized":"2024-03-28T18:56:15.996230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178310+0000","last_change":"2024-03-28T18:55:45.618585+0000","last_active":"2024-03-28T18:56:16.178310+0000","last_peered":"2024-03-28T18:56:16.178310+0000","last_clean":"2024-03-28T18:56:16.178310+0000","last_became_active":"2024-03-28T18:55:45.617895+0000","last_became_peered":"2024-03-28T18:55:45.617895+0000","last_unstale":"2024-03-28T18:56:16.178310+0000","last_undegraded":"2024-03-28T18:56:16.178310+0000","last_fullsized":"2024-03-28T18:56:16.178310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:54.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179210+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:56:16.179210+0000","last_peered":"2024-03-28T18:56:16.179210+0000","last_clean":"2024-03-28T18:56:16.179210+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:56:16.179210+0000","last_undegraded":"2024-03-28T18:56:16.179210+0000","last_fullsized":"2024-03-28T18:56:16.179210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":880640,"data_stored":704107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":299008,"data_stored":116551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":113829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":299008,"data_stored":113297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":118825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":113137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":116117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":884736,"data_stored":704073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:56:30.869 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-03-28T18:56:30.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:56:31.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.130+0000 7fa2f8400640 1 Processor -- start 2024-03-28T18:56:31.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.131+0000 7fa2f8400640 1 -- start start 2024-03-28T18:56:31.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.132+0000 7fa2f8400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa2f00be630 0x7fa2f00c5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.132+0000 7fa2f8400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2f00c5e40 0x7fa2f0159150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.132+0000 7fa2f8400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f00c7910 con 0x7fa2f0158580 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.132+0000 7fa2f8400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f00c76e0 con 0x7fa2f00c5e40 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.132+0000 7fa2f8400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f00c6dd0 con 0x7fa2f00be630 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.132+0000 7fa2f6400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2f00c5e40 0x7fa2f0159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.133+0000 7fa2f6400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2f00c5e40 0x7fa2f0159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40432/0 (socket says 172.21.15.27:40432) 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.133+0000 7fa2f6400640 1 -- 172.21.15.27:0/2610972584 learned_addr learned my addr 172.21.15.27:0/2610972584 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.133+0000 7fa2f5000640 1 --2- 172.21.15.27:0/2610972584 >> v2:172.21.15.155:3300/0 conn(0x7fa2f00be630 0x7fa2f00c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.133+0000 7fa2f6400640 1 -- 172.21.15.27:0/2610972584 >> v2:172.21.15.155:3300/0 conn(0x7fa2f00be630 msgr2=0x7fa2f00c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.133+0000 7fa2f6400640 1 --2- 172.21.15.27:0/2610972584 >> v2:172.21.15.155:3300/0 conn(0x7fa2f00be630 0x7fa2f00c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.133+0000 7fa2f6400640 1 -- 172.21.15.27:0/2610972584 >> v1:172.21.15.155:6789/0 conn(0x7fa2f0158580 legacy=0x7fa2f01560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.133+0000 7fa2f6400640 1 -- 172.21.15.27:0/2610972584 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f00c7bf0 con 0x7fa2f00c5e40 2024-03-28T18:56:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.134+0000 7fa2f6400640 1 --2- 172.21.15.27:0/2610972584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0159150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00aa940 tx=0x7fa2e802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48affadc103e68c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2efe00640 1 -- 172.21.15.27:0/2610972584 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e802ca80 con 0x7fa2f00c5e40 2024-03-28T18:56:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2efe00640 1 -- 172.21.15.27:0/2610972584 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e802cc20 con 0x7fa2f00c5e40 2024-03-28T18:56:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2efe00640 1 -- 172.21.15.27:0/2610972584 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80343f0 con 0x7fa2f00c5e40 2024-03-28T18:56:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2f8400640 1 -- 172.21.15.27:0/2610972584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 msgr2=0x7fa2f0159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2f8400640 1 --2- 172.21.15.27:0/2610972584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0159150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00aa940 tx=0x7fa2e802bee0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2f8400640 1 -- 172.21.15.27:0/2610972584 shutdown_connections 2024-03-28T18:56:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2f8400640 1 --2- 172.21.15.27:0/2610972584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0159150 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2f8400640 1 --2- 172.21.15.27:0/2610972584 >> v2:172.21.15.155:3300/0 conn(0x7fa2f00be630 0x7fa2f00c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2f8400640 1 -- 172.21.15.27:0/2610972584 >> 172.21.15.27:0/2610972584 conn(0x7fa2f00caf80 msgr2=0x7fa2f00cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2f8400640 1 -- 172.21.15.27:0/2610972584 shutdown_connections 2024-03-28T18:56:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.135+0000 7fa2f8400640 1 -- 172.21.15.27:0/2610972584 wait complete. 2024-03-28T18:56:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.137+0000 7fa2f8400640 1 Processor -- start 2024-03-28T18:56:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.137+0000 7fa2f8400640 1 -- start start 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f8400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f00be630 0x7fa2f01ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f8400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f8400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f0158580 0x7fa2f0141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f8400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f00c81b0 con 0x7fa2f00c5e40 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f8400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0158cc0 con 0x7fa2f00be630 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f8400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0158960 con 0x7fa2f0158580 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0140fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40444/0 (socket says 172.21.15.27:40444) 2024-03-28T18:56:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.138+0000 7fa2f5000640 1 -- 172.21.15.27:0/90380132 learned_addr learned my addr 172.21.15.27:0/90380132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f5a00640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f00be630 0x7fa2f01ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f6400640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f0158580 0x7fa2f0141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f5000640 1 -- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f0158580 msgr2=0x7fa2f0141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f5000640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f0158580 0x7fa2f0141510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f5000640 1 -- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f00be630 msgr2=0x7fa2f01ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f5000640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f00be630 0x7fa2f01ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f5000640 1 -- 172.21.15.27:0/90380132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f01ef000 con 0x7fa2f00c5e40 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f6400640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f0158580 0x7fa2f0141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2f5000640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0140fd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00d830 tx=0x7fa2dc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2ed600640 1 -- 172.21.15.27:0/90380132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00b6c0 con 0x7fa2f00c5e40 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.139+0000 7fa2ed600640 1 -- 172.21.15.27:0/90380132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc019070 con 0x7fa2f00c5e40 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.140+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f01ef2e0 con 0x7fa2f00c5e40 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.140+0000 7fa2ed600640 1 -- 172.21.15.27:0/90380132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00ba80 con 0x7fa2f00c5e40 2024-03-28T18:56:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.140+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f0155ae0 con 0x7fa2f00c5e40 2024-03-28T18:56:31.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.141+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f004f400 con 0x7fa2f00c5e40 2024-03-28T18:56:31.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.142+0000 7fa2ed600640 1 -- 172.21.15.27:0/90380132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa2dc01e460 con 0x7fa2f00c5e40 2024-03-28T18:56:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.143+0000 7fa2ed600640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2d006bbd0 0x7fa2d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.143+0000 7fa2ed600640 1 -- 172.21.15.27:0/90380132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fa2dc08b460 con 0x7fa2f00c5e40 2024-03-28T18:56:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.144+0000 7fa2f5a00640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2d006bbd0 0x7fa2d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.145+0000 7fa2f5a00640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2d006bbd0 0x7fa2d006e090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4006fd0 tx=0x7fa2e4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:31.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.157+0000 7fa2ed600640 1 -- 172.21.15.27:0/90380132 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa2dc059c40 con 0x7fa2f00c5e40 2024-03-28T18:56:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.356+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 --> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2f00729e0 con 0x7fa2d006bbd0 2024-03-28T18:56:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.391+0000 7fa2ed600640 1 -- 172.21.15.27:0/90380132 <== mgr.4116 v2:172.21.15.27:6832/1637319240 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490350 (secure 0 0 0) 0x7fa2f00729e0 con 0x7fa2d006bbd0 2024-03-28T18:56:31.398 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:31.401 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2d006bbd0 msgr2=0x7fa2d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2d006bbd0 0x7fa2d006e090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4006fd0 tx=0x7fa2e4008040 comp rx=0 tx=0).stop 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 msgr2=0x7fa2f0140fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0140fd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00d830 tx=0x7fa2dc00dcf0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 shutdown_connections 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2d006bbd0 0x7fa2d006e090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f0158580 0x7fa2f0141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f00c5e40 0x7fa2f0140fd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 --2- 172.21.15.27:0/90380132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f00be630 0x7fa2f01ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.399+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 >> 172.21.15.27:0/90380132 conn(0x7fa2f00caf80 msgr2=0x7fa2f00c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.400+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 shutdown_connections 2024-03-28T18:56:31.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.400+0000 7fa2f8400640 1 -- 172.21.15.27:0/90380132 wait complete. 2024-03-28T18:56:31.422 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-03-28T18:56:30.912851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597700,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":255,"num_write_kb":2450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219628,"kb_used_data":3464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755092,"statfs":{"total":773094113280,"available":772869214208,"internally_reserved":0,"allocated":3547136,"data_stored":2099936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2071,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005386"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164203+0000","last_change":"2024-03-28T18:55:45.603782+0000","last_active":"2024-03-28T18:56:16.164203+0000","last_peered":"2024-03-28T18:56:16.164203+0000","last_clean":"2024-03-28T18:56:16.164203+0000","last_became_active":"2024-03-28T18:55:45.603420+0000","last_became_peered":"2024-03-28T18:55:45.603420+0000","last_unstale":"2024-03-28T18:56:16.164203+0000","last_undegraded":"2024-03-28T18:56:16.164203+0000","last_fullsized":"2024-03-28T18:56:16.164203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373460+0000","last_change":"2024-03-28T18:55:33.500390+0000","last_active":"2024-03-28T18:56:16.373460+0000","last_peered":"2024-03-28T18:56:16.373460+0000","last_clean":"2024-03-28T18:56:16.373460+0000","last_became_active":"2024-03-28T18:55:33.500061+0000","last_became_peered":"2024-03-28T18:55:33.500061+0000","last_unstale":"2024-03-28T18:56:16.373460+0000","last_undegraded":"2024-03-28T18:56:16.373460+0000","last_fullsized":"2024-03-28T18:56:16.373460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992310+0000","last_change":"2024-03-28T18:55:34.485358+0000","last_active":"2024-03-28T18:56:15.992310+0000","last_peered":"2024-03-28T18:56:15.992310+0000","last_clean":"2024-03-28T18:56:15.992310+0000","last_became_active":"2024-03-28T18:55:34.484985+0000","last_became_peered":"2024-03-28T18:55:34.484985+0000","last_unstale":"2024-03-28T18:56:15.992310+0000","last_undegraded":"2024-03-28T18:56:15.992310+0000","last_fullsized":"2024-03-28T18:56:15.992310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989768+0000","last_change":"2024-03-28T18:55:44.602536+0000","last_active":"2024-03-28T18:56:15.989768+0000","last_peered":"2024-03-28T18:56:15.989768+0000","last_clean":"2024-03-28T18:56:15.989768+0000","last_became_active":"2024-03-28T18:55:44.602256+0000","last_became_peered":"2024-03-28T18:55:44.602256+0000","last_unstale":"2024-03-28T18:56:15.989768+0000","last_undegraded":"2024-03-28T18:56:15.989768+0000","last_fullsized":"2024-03-28T18:56:15.989768+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994213+0000","last_change":"2024-03-28T18:55:45.625338+0000","last_active":"2024-03-28T18:56:15.994213+0000","last_peered":"2024-03-28T18:56:15.994213+0000","last_clean":"2024-03-28T18:56:15.994213+0000","last_became_active":"2024-03-28T18:55:45.625067+0000","last_became_peered":"2024-03-28T18:55:45.625067+0000","last_unstale":"2024-03-28T18:56:15.994213+0000","last_undegraded":"2024-03-28T18:56:15.994213+0000","last_fullsized":"2024-03-28T18:56:15.994213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991986+0000","last_change":"2024-03-28T18:55:33.505508+0000","last_active":"2024-03-28T18:56:15.991986+0000","last_peered":"2024-03-28T18:56:15.991986+0000","last_clean":"2024-03-28T18:56:15.991986+0000","last_became_active":"2024-03-28T18:55:33.501702+0000","last_became_peered":"2024-03-28T18:55:33.501702+0000","last_unstale":"2024-03-28T18:56:15.991986+0000","last_undegraded":"2024-03-28T18:56:15.991986+0000","last_fullsized":"2024-03-28T18:56:15.991986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373257+0000","last_change":"2024-03-28T18:55:34.495969+0000","last_active":"2024-03-28T18:56:16.373257+0000","last_peered":"2024-03-28T18:56:16.373257+0000","last_clean":"2024-03-28T18:56:16.373257+0000","last_became_active":"2024-03-28T18:55:34.495306+0000","last_became_peered":"2024-03-28T18:55:34.495306+0000","last_unstale":"2024-03-28T18:56:16.373257+0000","last_undegraded":"2024-03-28T18:56:16.373257+0000","last_fullsized":"2024-03-28T18:56:16.373257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164333+0000","last_change":"2024-03-28T18:55:44.598423+0000","last_active":"2024-03-28T18:56:16.164333+0000","last_peered":"2024-03-28T18:56:16.164333+0000","last_clean":"2024-03-28T18:56:16.164333+0000","last_became_active":"2024-03-28T18:55:44.598092+0000","last_became_peered":"2024-03-28T18:55:44.598092+0000","last_unstale":"2024-03-28T18:56:16.164333+0000","last_undegraded":"2024-03-28T18:56:16.164333+0000","last_fullsized":"2024-03-28T18:56:16.164333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:13.646619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178114+0000","last_change":"2024-03-28T18:55:45.608201+0000","last_active":"2024-03-28T18:56:16.178114+0000","last_peered":"2024-03-28T18:56:16.178114+0000","last_clean":"2024-03-28T18:56:16.178114+0000","last_became_active":"2024-03-28T18:55:45.607912+0000","last_became_peered":"2024-03-28T18:55:45.607912+0000","last_unstale":"2024-03-28T18:56:16.178114+0000","last_undegraded":"2024-03-28T18:56:16.178114+0000","last_fullsized":"2024-03-28T18:56:16.178114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178494+0000","last_change":"2024-03-28T18:55:33.504540+0000","last_active":"2024-03-28T18:56:16.178494+0000","last_peered":"2024-03-28T18:56:16.178494+0000","last_clean":"2024-03-28T18:56:16.178494+0000","last_became_active":"2024-03-28T18:55:33.504076+0000","last_became_peered":"2024-03-28T18:55:33.504076+0000","last_unstale":"2024-03-28T18:56:16.178494+0000","last_undegraded":"2024-03-28T18:56:16.178494+0000","last_fullsized":"2024-03-28T18:56:16.178494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:31.917395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996247+0000","last_change":"2024-03-28T18:55:34.506344+0000","last_active":"2024-03-28T18:56:15.996247+0000","last_peered":"2024-03-28T18:56:15.996247+0000","last_clean":"2024-03-28T18:56:15.996247+0000","last_became_active":"2024-03-28T18:55:34.505432+0000","last_became_peered":"2024-03-28T18:55:34.505432+0000","last_unstale":"2024-03-28T18:56:15.996247+0000","last_undegraded":"2024-03-28T18:56:15.996247+0000","last_fullsized":"2024-03-28T18:56:15.996247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164809+0000","last_change":"2024-03-28T18:55:44.604673+0000","last_active":"2024-03-28T18:56:16.164809+0000","last_peered":"2024-03-28T18:56:16.164809+0000","last_clean":"2024-03-28T18:56:16.164809+0000","last_became_active":"2024-03-28T18:55:44.603926+0000","last_became_peered":"2024-03-28T18:55:44.603926+0000","last_unstale":"2024-03-28T18:56:16.164809+0000","last_undegraded":"2024-03-28T18:56:16.164809+0000","last_fullsized":"2024-03-28T18:56:16.164809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995761+0000","last_change":"2024-03-28T18:55:45.610033+0000","last_active":"2024-03-28T18:56:15.995761+0000","last_peered":"2024-03-28T18:56:15.995761+0000","last_clean":"2024-03-28T18:56:15.995761+0000","last_became_active":"2024-03-28T18:55:45.609706+0000","last_became_peered":"2024-03-28T18:55:45.609706+0000","last_unstale":"2024-03-28T18:56:15.995761+0000","last_undegraded":"2024-03-28T18:56:15.995761+0000","last_fullsized":"2024-03-28T18:56:15.995761+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178300+0000","last_change":"2024-03-28T18:55:33.501865+0000","last_active":"2024-03-28T18:56:16.178300+0000","last_peered":"2024-03-28T18:56:16.178300+0000","last_clean":"2024-03-28T18:56:16.178300+0000","last_became_active":"2024-03-28T18:55:33.501503+0000","last_became_peered":"2024-03-28T18:55:33.501503+0000","last_unstale":"2024-03-28T18:56:16.178300+0000","last_undegraded":"2024-03-28T18:56:16.178300+0000","last_fullsized":"2024-03-28T18:56:16.178300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996865+0000","last_change":"2024-03-28T18:55:34.505027+0000","last_active":"2024-03-28T18:56:15.996865+0000","last_peered":"2024-03-28T18:56:15.996865+0000","last_clean":"2024-03-28T18:56:15.996865+0000","last_became_active":"2024-03-28T18:55:34.500409+0000","last_became_peered":"2024-03-28T18:55:34.500409+0000","last_unstale":"2024-03-28T18:56:15.996865+0000","last_undegraded":"2024-03-28T18:56:15.996865+0000","last_fullsized":"2024-03-28T18:56:15.996865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164862+0000","last_change":"2024-03-28T18:55:44.606715+0000","last_active":"2024-03-28T18:56:16.164862+0000","last_peered":"2024-03-28T18:56:16.164862+0000","last_clean":"2024-03-28T18:56:16.164862+0000","last_became_active":"2024-03-28T18:55:44.606343+0000","last_became_peered":"2024-03-28T18:55:44.606343+0000","last_unstale":"2024-03-28T18:56:16.164862+0000","last_undegraded":"2024-03-28T18:56:16.164862+0000","last_fullsized":"2024-03-28T18:56:16.164862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:51.471450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995747+0000","last_change":"2024-03-28T18:55:45.623483+0000","last_active":"2024-03-28T18:56:15.995747+0000","last_peered":"2024-03-28T18:56:15.995747+0000","last_clean":"2024-03-28T18:56:15.995747+0000","last_became_active":"2024-03-28T18:55:45.622894+0000","last_became_peered":"2024-03-28T18:55:45.622894+0000","last_unstale":"2024-03-28T18:56:15.995747+0000","last_undegraded":"2024-03-28T18:56:15.995747+0000","last_fullsized":"2024-03-28T18:56:15.995747+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993916+0000","last_change":"2024-03-28T18:55:34.482996+0000","last_active":"2024-03-28T18:56:15.993916+0000","last_peered":"2024-03-28T18:56:15.993916+0000","last_clean":"2024-03-28T18:56:15.993916+0000","last_became_active":"2024-03-28T18:55:34.482663+0000","last_became_peered":"2024-03-28T18:55:34.482663+0000","last_unstale":"2024-03-28T18:56:15.993916+0000","last_undegraded":"2024-03-28T18:56:15.993916+0000","last_fullsized":"2024-03-28T18:56:15.993916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177876+0000","last_change":"2024-03-28T18:55:33.502308+0000","last_active":"2024-03-28T18:56:16.177876+0000","last_peered":"2024-03-28T18:56:16.177876+0000","last_clean":"2024-03-28T18:56:16.177876+0000","last_became_active":"2024-03-28T18:55:33.501919+0000","last_became_peered":"2024-03-28T18:55:33.501919+0000","last_unstale":"2024-03-28T18:56:16.177876+0000","last_undegraded":"2024-03-28T18:56:16.177876+0000","last_fullsized":"2024-03-28T18:56:16.177876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165877+0000","last_change":"2024-03-28T18:55:44.601415+0000","last_active":"2024-03-28T18:56:16.165877+0000","last_peered":"2024-03-28T18:56:16.165877+0000","last_clean":"2024-03-28T18:56:16.165877+0000","last_became_active":"2024-03-28T18:55:44.601124+0000","last_became_peered":"2024-03-28T18:55:44.601124+0000","last_unstale":"2024-03-28T18:56:16.165877+0000","last_undegraded":"2024-03-28T18:56:16.165877+0000","last_fullsized":"2024-03-28T18:56:16.165877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:30.767034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993126+0000","last_change":"2024-03-28T18:55:45.604795+0000","last_active":"2024-03-28T18:56:15.993126+0000","last_peered":"2024-03-28T18:56:15.993126+0000","last_clean":"2024-03-28T18:56:15.993126+0000","last_became_active":"2024-03-28T18:55:45.604432+0000","last_became_peered":"2024-03-28T18:55:45.604432+0000","last_unstale":"2024-03-28T18:56:15.993126+0000","last_undegraded":"2024-03-28T18:56:15.993126+0000","last_fullsized":"2024-03-28T18:56:15.993126+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:37.638804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990541+0000","last_change":"2024-03-28T18:55:34.494142+0000","last_active":"2024-03-28T18:56:15.990541+0000","last_peered":"2024-03-28T18:56:15.990541+0000","last_clean":"2024-03-28T18:56:15.990541+0000","last_became_active":"2024-03-28T18:55:34.493728+0000","last_became_peered":"2024-03-28T18:55:34.493728+0000","last_unstale":"2024-03-28T18:56:15.990541+0000","last_undegraded":"2024-03-28T18:56:15.990541+0000","last_fullsized":"2024-03-28T18:56:15.990541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995164+0000","last_change":"2024-03-28T18:55:33.506585+0000","last_active":"2024-03-28T18:56:15.995164+0000","last_peered":"2024-03-28T18:56:15.995164+0000","last_clean":"2024-03-28T18:56:15.995164+0000","last_became_active":"2024-03-28T18:55:33.505675+0000","last_became_peered":"2024-03-28T18:55:33.505675+0000","last_unstale":"2024-03-28T18:56:15.995164+0000","last_undegraded":"2024-03-28T18:56:15.995164+0000","last_fullsized":"2024-03-28T18:56:15.995164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:16.693275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165844+0000","last_change":"2024-03-28T18:55:44.603859+0000","last_active":"2024-03-28T18:56:16.165844+0000","last_peered":"2024-03-28T18:56:16.165844+0000","last_clean":"2024-03-28T18:56:16.165844+0000","last_became_active":"2024-03-28T18:55:44.603540+0000","last_became_peered":"2024-03-28T18:55:44.603540+0000","last_unstale":"2024-03-28T18:56:16.165844+0000","last_undegraded":"2024-03-28T18:56:16.165844+0000","last_fullsized":"2024-03-28T18:56:16.165844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165791+0000","last_change":"2024-03-28T18:55:45.608142+0000","last_active":"2024-03-28T18:56:16.165791+0000","last_peered":"2024-03-28T18:56:16.165791+0000","last_clean":"2024-03-28T18:56:16.165791+0000","last_became_active":"2024-03-28T18:55:45.606117+0000","last_became_peered":"2024-03-28T18:55:45.606117+0000","last_unstale":"2024-03-28T18:56:16.165791+0000","last_undegraded":"2024-03-28T18:56:16.165791+0000","last_fullsized":"2024-03-28T18:56:16.165791+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989871+0000","last_change":"2024-03-28T18:55:34.488144+0000","last_active":"2024-03-28T18:56:15.989871+0000","last_peered":"2024-03-28T18:56:15.989871+0000","last_clean":"2024-03-28T18:56:15.989871+0000","last_became_active":"2024-03-28T18:55:34.487760+0000","last_became_peered":"2024-03-28T18:55:34.487760+0000","last_unstale":"2024-03-28T18:56:15.989871+0000","last_undegraded":"2024-03-28T18:56:15.989871+0000","last_fullsized":"2024-03-28T18:56:15.989871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994533+0000","last_change":"2024-03-28T18:55:33.504436+0000","last_active":"2024-03-28T18:56:15.994533+0000","last_peered":"2024-03-28T18:56:15.994533+0000","last_clean":"2024-03-28T18:56:15.994533+0000","last_became_active":"2024-03-28T18:55:33.502638+0000","last_became_peered":"2024-03-28T18:55:33.502638+0000","last_unstale":"2024-03-28T18:56:15.994533+0000","last_undegraded":"2024-03-28T18:56:15.994533+0000","last_fullsized":"2024-03-28T18:56:15.994533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996249+0000","last_change":"2024-03-28T18:55:44.606690+0000","last_active":"2024-03-28T18:56:15.996249+0000","last_peered":"2024-03-28T18:56:15.996249+0000","last_clean":"2024-03-28T18:56:15.996249+0000","last_became_active":"2024-03-28T18:55:44.606421+0000","last_became_peered":"2024-03-28T18:55:44.606421+0000","last_unstale":"2024-03-28T18:56:15.996249+0000","last_undegraded":"2024-03-28T18:56:15.996249+0000","last_fullsized":"2024-03-28T18:56:15.996249+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997284+0000","last_change":"2024-03-28T18:55:45.610580+0000","last_active":"2024-03-28T18:56:15.997284+0000","last_peered":"2024-03-28T18:56:15.997284+0000","last_clean":"2024-03-28T18:56:15.997284+0000","last_became_active":"2024-03-28T18:55:45.610316+0000","last_became_peered":"2024-03-28T18:55:45.610316+0000","last_unstale":"2024-03-28T18:56:15.997284+0000","last_undegraded":"2024-03-28T18:56:15.997284+0000","last_fullsized":"2024-03-28T18:56:15.997284+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165194+0000","last_change":"2024-03-28T18:55:34.489670+0000","last_active":"2024-03-28T18:56:16.165194+0000","last_peered":"2024-03-28T18:56:16.165194+0000","last_clean":"2024-03-28T18:56:16.165194+0000","last_became_active":"2024-03-28T18:55:34.489373+0000","last_became_peered":"2024-03-28T18:55:34.489373+0000","last_unstale":"2024-03-28T18:56:16.165194+0000","last_undegraded":"2024-03-28T18:56:16.165194+0000","last_fullsized":"2024-03-28T18:56:16.165194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:31.460324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177321+0000","last_change":"2024-03-28T18:55:33.503469+0000","last_active":"2024-03-28T18:56:16.177321+0000","last_peered":"2024-03-28T18:56:16.177321+0000","last_clean":"2024-03-28T18:56:16.177321+0000","last_became_active":"2024-03-28T18:55:33.503083+0000","last_became_peered":"2024-03-28T18:55:33.503083+0000","last_unstale":"2024-03-28T18:56:16.177321+0000","last_undegraded":"2024-03-28T18:56:16.177321+0000","last_fullsized":"2024-03-28T18:56:16.177321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166077+0000","last_change":"2024-03-28T18:55:44.600523+0000","last_active":"2024-03-28T18:56:16.166077+0000","last_peered":"2024-03-28T18:56:16.166077+0000","last_clean":"2024-03-28T18:56:16.166077+0000","last_became_active":"2024-03-28T18:55:44.597550+0000","last_became_peered":"2024-03-28T18:55:44.597550+0000","last_unstale":"2024-03-28T18:56:16.166077+0000","last_undegraded":"2024-03-28T18:56:16.166077+0000","last_fullsized":"2024-03-28T18:56:16.166077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989484+0000","last_change":"2024-03-28T18:55:45.619798+0000","last_active":"2024-03-28T18:56:15.989484+0000","last_peered":"2024-03-28T18:56:15.989484+0000","last_clean":"2024-03-28T18:56:15.989484+0000","last_became_active":"2024-03-28T18:55:45.616517+0000","last_became_peered":"2024-03-28T18:55:45.616517+0000","last_unstale":"2024-03-28T18:56:15.989484+0000","last_undegraded":"2024-03-28T18:56:15.989484+0000","last_fullsized":"2024-03-28T18:56:15.989484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.374003+0000","last_change":"2024-03-28T18:55:34.490114+0000","last_active":"2024-03-28T18:56:16.374003+0000","last_peered":"2024-03-28T18:56:16.374003+0000","last_clean":"2024-03-28T18:56:16.374003+0000","last_became_active":"2024-03-28T18:55:34.489175+0000","last_became_peered":"2024-03-28T18:55:34.489175+0000","last_unstale":"2024-03-28T18:56:16.374003+0000","last_undegraded":"2024-03-28T18:56:16.374003+0000","last_fullsized":"2024-03-28T18:56:16.374003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994929+0000","last_change":"2024-03-28T18:55:33.499601+0000","last_active":"2024-03-28T18:56:15.994929+0000","last_peered":"2024-03-28T18:56:15.994929+0000","last_clean":"2024-03-28T18:56:15.994929+0000","last_became_active":"2024-03-28T18:55:33.499322+0000","last_became_peered":"2024-03-28T18:55:33.499322+0000","last_unstale":"2024-03-28T18:56:15.994929+0000","last_undegraded":"2024-03-28T18:56:15.994929+0000","last_fullsized":"2024-03-28T18:56:15.994929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164852+0000","last_change":"2024-03-28T18:55:44.606498+0000","last_active":"2024-03-28T18:56:16.164852+0000","last_peered":"2024-03-28T18:56:16.164852+0000","last_clean":"2024-03-28T18:56:16.164852+0000","last_became_active":"2024-03-28T18:55:44.606098+0000","last_became_peered":"2024-03-28T18:55:44.606098+0000","last_unstale":"2024-03-28T18:56:16.164852+0000","last_undegraded":"2024-03-28T18:56:16.164852+0000","last_fullsized":"2024-03-28T18:56:16.164852+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164035+0000","last_change":"2024-03-28T18:55:45.617483+0000","last_active":"2024-03-28T18:56:16.164035+0000","last_peered":"2024-03-28T18:56:16.164035+0000","last_clean":"2024-03-28T18:56:16.164035+0000","last_became_active":"2024-03-28T18:55:45.617169+0000","last_became_peered":"2024-03-28T18:55:45.617169+0000","last_unstale":"2024-03-28T18:56:16.164035+0000","last_undegraded":"2024-03-28T18:56:16.164035+0000","last_fullsized":"2024-03-28T18:56:16.164035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:41.809108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992182+0000","last_change":"2024-03-28T18:55:34.491200+0000","last_active":"2024-03-28T18:56:15.992182+0000","last_peered":"2024-03-28T18:56:15.992182+0000","last_clean":"2024-03-28T18:56:15.992182+0000","last_became_active":"2024-03-28T18:55:34.490913+0000","last_became_peered":"2024-03-28T18:55:34.490913+0000","last_unstale":"2024-03-28T18:56:15.992182+0000","last_undegraded":"2024-03-28T18:56:15.992182+0000","last_fullsized":"2024-03-28T18:56:15.992182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996404+0000","last_change":"2024-03-28T18:55:33.506092+0000","last_active":"2024-03-28T18:56:15.996404+0000","last_peered":"2024-03-28T18:56:15.996404+0000","last_clean":"2024-03-28T18:56:15.996404+0000","last_became_active":"2024-03-28T18:55:33.505633+0000","last_became_peered":"2024-03-28T18:55:33.505633+0000","last_unstale":"2024-03-28T18:56:15.996404+0000","last_undegraded":"2024-03-28T18:56:15.996404+0000","last_fullsized":"2024-03-28T18:56:15.996404+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989501+0000","last_change":"2024-03-28T18:55:44.601836+0000","last_active":"2024-03-28T18:56:15.989501+0000","last_peered":"2024-03-28T18:56:15.989501+0000","last_clean":"2024-03-28T18:56:15.989501+0000","last_became_active":"2024-03-28T18:55:44.601552+0000","last_became_peered":"2024-03-28T18:55:44.601552+0000","last_unstale":"2024-03-28T18:56:15.989501+0000","last_undegraded":"2024-03-28T18:56:15.989501+0000","last_fullsized":"2024-03-28T18:56:15.989501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993178+0000","last_change":"2024-03-28T18:55:45.601801+0000","last_active":"2024-03-28T18:56:15.993178+0000","last_peered":"2024-03-28T18:56:15.993178+0000","last_clean":"2024-03-28T18:56:15.993178+0000","last_became_active":"2024-03-28T18:55:45.601459+0000","last_became_peered":"2024-03-28T18:55:45.601459+0000","last_unstale":"2024-03-28T18:56:15.993178+0000","last_undegraded":"2024-03-28T18:56:15.993178+0000","last_fullsized":"2024-03-28T18:56:15.993178+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:43.033170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995963+0000","last_change":"2024-03-28T18:55:34.492496+0000","last_active":"2024-03-28T18:56:15.995963+0000","last_peered":"2024-03-28T18:56:15.995963+0000","last_clean":"2024-03-28T18:56:15.995963+0000","last_became_active":"2024-03-28T18:55:34.492143+0000","last_became_peered":"2024-03-28T18:55:34.492143+0000","last_unstale":"2024-03-28T18:56:15.995963+0000","last_undegraded":"2024-03-28T18:56:15.995963+0000","last_fullsized":"2024-03-28T18:56:15.995963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178676+0000","last_change":"2024-03-28T18:55:33.504082+0000","last_active":"2024-03-28T18:56:16.178676+0000","last_peered":"2024-03-28T18:56:16.178676+0000","last_clean":"2024-03-28T18:56:16.178676+0000","last_became_active":"2024-03-28T18:55:33.502752+0000","last_became_peered":"2024-03-28T18:55:33.502752+0000","last_unstale":"2024-03-28T18:56:16.178676+0000","last_undegraded":"2024-03-28T18:56:16.178676+0000","last_fullsized":"2024-03-28T18:56:16.178676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:04:42.657118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373043+0000","last_change":"2024-03-28T18:55:44.611148+0000","last_active":"2024-03-28T18:56:16.373043+0000","last_peered":"2024-03-28T18:56:16.373043+0000","last_clean":"2024-03-28T18:56:16.373043+0000","last_became_active":"2024-03-28T18:55:44.610835+0000","last_became_peered":"2024-03-28T18:55:44.610835+0000","last_unstale":"2024-03-28T18:56:16.373043+0000","last_undegraded":"2024-03-28T18:56:16.373043+0000","last_fullsized":"2024-03-28T18:56:16.373043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:59.587524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373399+0000","last_change":"2024-03-28T18:55:45.623672+0000","last_active":"2024-03-28T18:56:16.373399+0000","last_peered":"2024-03-28T18:56:16.373399+0000","last_clean":"2024-03-28T18:56:16.373399+0000","last_became_active":"2024-03-28T18:55:45.622568+0000","last_became_peered":"2024-03-28T18:55:45.622568+0000","last_unstale":"2024-03-28T18:56:16.373399+0000","last_undegraded":"2024-03-28T18:56:16.373399+0000","last_fullsized":"2024-03-28T18:56:16.373399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997025+0000","last_change":"2024-03-28T18:55:34.497701+0000","last_active":"2024-03-28T18:56:15.997025+0000","last_peered":"2024-03-28T18:56:15.997025+0000","last_clean":"2024-03-28T18:56:15.997025+0000","last_became_active":"2024-03-28T18:55:34.497378+0000","last_became_peered":"2024-03-28T18:55:34.497378+0000","last_unstale":"2024-03-28T18:56:15.997025+0000","last_undegraded":"2024-03-28T18:56:15.997025+0000","last_fullsized":"2024-03-28T18:56:15.997025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996010+0000","last_change":"2024-03-28T18:55:33.507919+0000","last_active":"2024-03-28T18:56:15.996010+0000","last_peered":"2024-03-28T18:56:15.996010+0000","last_clean":"2024-03-28T18:56:15.996010+0000","last_became_active":"2024-03-28T18:55:33.505962+0000","last_became_peered":"2024-03-28T18:55:33.505962+0000","last_unstale":"2024-03-28T18:56:15.996010+0000","last_undegraded":"2024-03-28T18:56:15.996010+0000","last_fullsized":"2024-03-28T18:56:15.996010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994806+0000","last_change":"2024-03-28T18:55:44.604077+0000","last_active":"2024-03-28T18:56:15.994806+0000","last_peered":"2024-03-28T18:56:15.994806+0000","last_clean":"2024-03-28T18:56:15.994806+0000","last_became_active":"2024-03-28T18:55:44.603773+0000","last_became_peered":"2024-03-28T18:55:44.603773+0000","last_unstale":"2024-03-28T18:56:15.994806+0000","last_undegraded":"2024-03-28T18:56:15.994806+0000","last_fullsized":"2024-03-28T18:56:15.994806+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372682+0000","last_change":"2024-03-28T18:55:45.615444+0000","last_active":"2024-03-28T18:56:16.372682+0000","last_peered":"2024-03-28T18:56:16.372682+0000","last_clean":"2024-03-28T18:56:16.372682+0000","last_became_active":"2024-03-28T18:55:45.615002+0000","last_became_peered":"2024-03-28T18:55:45.615002+0000","last_unstale":"2024-03-28T18:56:16.372682+0000","last_undegraded":"2024-03-28T18:56:16.372682+0000","last_fullsized":"2024-03-28T18:56:16.372682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:27.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166059+0000","last_change":"2024-03-28T18:55:34.489873+0000","last_active":"2024-03-28T18:56:16.166059+0000","last_peered":"2024-03-28T18:56:16.166059+0000","last_clean":"2024-03-28T18:56:16.166059+0000","last_became_active":"2024-03-28T18:55:34.489571+0000","last_became_peered":"2024-03-28T18:55:34.489571+0000","last_unstale":"2024-03-28T18:56:16.166059+0000","last_undegraded":"2024-03-28T18:56:16.166059+0000","last_fullsized":"2024-03-28T18:56:16.166059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996064+0000","last_change":"2024-03-28T18:55:33.507132+0000","last_active":"2024-03-28T18:56:15.996064+0000","last_peered":"2024-03-28T18:56:15.996064+0000","last_clean":"2024-03-28T18:56:15.996064+0000","last_became_active":"2024-03-28T18:55:33.505999+0000","last_became_peered":"2024-03-28T18:55:33.505999+0000","last_unstale":"2024-03-28T18:56:15.996064+0000","last_undegraded":"2024-03-28T18:56:15.996064+0000","last_fullsized":"2024-03-28T18:56:15.996064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995899+0000","last_change":"2024-03-28T18:55:44.590703+0000","last_active":"2024-03-28T18:56:15.995899+0000","last_peered":"2024-03-28T18:56:15.995899+0000","last_clean":"2024-03-28T18:56:15.995899+0000","last_became_active":"2024-03-28T18:55:44.590408+0000","last_became_peered":"2024-03-28T18:55:44.590408+0000","last_unstale":"2024-03-28T18:56:15.995899+0000","last_undegraded":"2024-03-28T18:56:15.995899+0000","last_fullsized":"2024-03-28T18:56:15.995899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992984+0000","last_change":"2024-03-28T18:55:45.609795+0000","last_active":"2024-03-28T18:56:15.992984+0000","last_peered":"2024-03-28T18:56:15.992984+0000","last_clean":"2024-03-28T18:56:15.992984+0000","last_became_active":"2024-03-28T18:55:45.603695+0000","last_became_peered":"2024-03-28T18:55:45.603695+0000","last_unstale":"2024-03-28T18:56:15.992984+0000","last_undegraded":"2024-03-28T18:56:15.992984+0000","last_fullsized":"2024-03-28T18:56:15.992984+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:49.173555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994287+0000","last_change":"2024-03-28T18:55:34.491360+0000","last_active":"2024-03-28T18:56:15.994287+0000","last_peered":"2024-03-28T18:56:15.994287+0000","last_clean":"2024-03-28T18:56:15.994287+0000","last_became_active":"2024-03-28T18:55:34.491071+0000","last_became_peered":"2024-03-28T18:55:34.491071+0000","last_unstale":"2024-03-28T18:56:15.994287+0000","last_undegraded":"2024-03-28T18:56:15.994287+0000","last_fullsized":"2024-03-28T18:56:15.994287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993148+0000","last_change":"2024-03-28T18:55:33.498489+0000","last_active":"2024-03-28T18:56:15.993148+0000","last_peered":"2024-03-28T18:56:15.993148+0000","last_clean":"2024-03-28T18:56:15.993148+0000","last_became_active":"2024-03-28T18:55:33.498110+0000","last_became_peered":"2024-03-28T18:55:33.498110+0000","last_unstale":"2024-03-28T18:56:15.993148+0000","last_undegraded":"2024-03-28T18:56:15.993148+0000","last_fullsized":"2024-03-28T18:56:15.993148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996282+0000","last_change":"2024-03-28T18:55:44.612172+0000","last_active":"2024-03-28T18:56:15.996282+0000","last_peered":"2024-03-28T18:56:15.996282+0000","last_clean":"2024-03-28T18:56:15.996282+0000","last_became_active":"2024-03-28T18:55:44.611859+0000","last_became_peered":"2024-03-28T18:55:44.611859+0000","last_unstale":"2024-03-28T18:56:15.996282+0000","last_undegraded":"2024-03-28T18:56:15.996282+0000","last_fullsized":"2024-03-28T18:56:15.996282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997500+0000","last_change":"2024-03-28T18:55:45.619723+0000","last_active":"2024-03-28T18:56:15.997500+0000","last_peered":"2024-03-28T18:56:15.997500+0000","last_clean":"2024-03-28T18:56:15.997500+0000","last_became_active":"2024-03-28T18:55:45.619437+0000","last_became_peered":"2024-03-28T18:55:45.619437+0000","last_unstale":"2024-03-28T18:56:15.997500+0000","last_undegraded":"2024-03-28T18:56:15.997500+0000","last_fullsized":"2024-03-28T18:56:15.997500+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372410+0000","last_change":"2024-03-28T18:55:34.488579+0000","last_active":"2024-03-28T18:56:16.372410+0000","last_peered":"2024-03-28T18:56:16.372410+0000","last_clean":"2024-03-28T18:56:16.372410+0000","last_became_active":"2024-03-28T18:55:34.487759+0000","last_became_peered":"2024-03-28T18:55:34.487759+0000","last_unstale":"2024-03-28T18:56:16.372410+0000","last_undegraded":"2024-03-28T18:56:16.372410+0000","last_fullsized":"2024-03-28T18:56:16.372410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:48.193636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177331+0000","last_change":"2024-03-28T18:55:33.511702+0000","last_active":"2024-03-28T18:56:16.177331+0000","last_peered":"2024-03-28T18:56:16.177331+0000","last_clean":"2024-03-28T18:56:16.177331+0000","last_became_active":"2024-03-28T18:55:33.511192+0000","last_became_peered":"2024-03-28T18:55:33.511192+0000","last_unstale":"2024-03-28T18:56:16.177331+0000","last_undegraded":"2024-03-28T18:56:16.177331+0000","last_fullsized":"2024-03-28T18:56:16.177331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"24'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373710+0000","last_change":"2024-03-28T18:55:44.592064+0000","last_active":"2024-03-28T18:56:16.373710+0000","last_peered":"2024-03-28T18:56:16.373710+0000","last_clean":"2024-03-28T18:56:16.373710+0000","last_became_active":"2024-03-28T18:55:44.591263+0000","last_became_peered":"2024-03-28T18:55:44.591263+0000","last_unstale":"2024-03-28T18:56:16.373710+0000","last_undegraded":"2024-03-28T18:56:16.373710+0000","last_fullsized":"2024-03-28T18:56:16.373710+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:35.294387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177095+0000","last_change":"2024-03-28T18:55:45.614820+0000","last_active":"2024-03-28T18:56:16.177095+0000","last_peered":"2024-03-28T18:56:16.177095+0000","last_clean":"2024-03-28T18:56:16.177095+0000","last_became_active":"2024-03-28T18:55:45.614517+0000","last_became_peered":"2024-03-28T18:55:45.614517+0000","last_unstale":"2024-03-28T18:56:16.177095+0000","last_undegraded":"2024-03-28T18:56:16.177095+0000","last_fullsized":"2024-03-28T18:56:16.177095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990462+0000","last_change":"2024-03-28T18:55:34.499631+0000","last_active":"2024-03-28T18:56:15.990462+0000","last_peered":"2024-03-28T18:56:15.990462+0000","last_clean":"2024-03-28T18:56:15.990462+0000","last_became_active":"2024-03-28T18:55:34.496966+0000","last_became_peered":"2024-03-28T18:55:34.496966+0000","last_unstale":"2024-03-28T18:56:15.990462+0000","last_undegraded":"2024-03-28T18:56:15.990462+0000","last_fullsized":"2024-03-28T18:56:15.990462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:20.674249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995955+0000","last_change":"2024-03-28T18:55:33.501320+0000","last_active":"2024-03-28T18:56:15.995955+0000","last_peered":"2024-03-28T18:56:15.995955+0000","last_clean":"2024-03-28T18:56:15.995955+0000","last_became_active":"2024-03-28T18:55:33.500941+0000","last_became_peered":"2024-03-28T18:55:33.500941+0000","last_unstale":"2024-03-28T18:56:15.995955+0000","last_undegraded":"2024-03-28T18:56:15.995955+0000","last_fullsized":"2024-03-28T18:56:15.995955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177684+0000","last_change":"2024-03-28T18:55:24.294186+0000","last_active":"2024-03-28T18:56:16.177684+0000","last_peered":"2024-03-28T18:56:16.177684+0000","last_clean":"2024-03-28T18:56:16.177684+0000","last_became_active":"2024-03-28T18:55:24.288252+0000","last_became_peered":"2024-03-28T18:55:24.288252+0000","last_unstale":"2024-03-28T18:56:16.177684+0000","last_undegraded":"2024-03-28T18:56:16.177684+0000","last_fullsized":"2024-03-28T18:56:16.177684+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:23.154000+0000","last_clean_scrub_stamp":"2024-03-28T18:55:23.154000+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.688771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993499+0000","last_change":"2024-03-28T18:55:44.602097+0000","last_active":"2024-03-28T18:56:15.993499+0000","last_peered":"2024-03-28T18:56:15.993499+0000","last_clean":"2024-03-28T18:56:15.993499+0000","last_became_active":"2024-03-28T18:55:44.601769+0000","last_became_peered":"2024-03-28T18:55:44.601769+0000","last_unstale":"2024-03-28T18:56:15.993499+0000","last_undegraded":"2024-03-28T18:56:15.993499+0000","last_fullsized":"2024-03-28T18:56:15.993499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:34.884396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993296+0000","last_change":"2024-03-28T18:55:45.616485+0000","last_active":"2024-03-28T18:56:15.993296+0000","last_peered":"2024-03-28T18:56:15.993296+0000","last_clean":"2024-03-28T18:56:15.993296+0000","last_became_active":"2024-03-28T18:55:45.615813+0000","last_became_peered":"2024-03-28T18:55:45.615813+0000","last_unstale":"2024-03-28T18:56:15.993296+0000","last_undegraded":"2024-03-28T18:56:15.993296+0000","last_fullsized":"2024-03-28T18:56:15.993296+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:21.227281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178319+0000","last_change":"2024-03-28T18:55:34.505527+0000","last_active":"2024-03-28T18:56:16.178319+0000","last_peered":"2024-03-28T18:56:16.178319+0000","last_clean":"2024-03-28T18:56:16.178319+0000","last_became_active":"2024-03-28T18:55:34.504389+0000","last_became_peered":"2024-03-28T18:55:34.504389+0000","last_unstale":"2024-03-28T18:56:16.178319+0000","last_undegraded":"2024-03-28T18:56:16.178319+0000","last_fullsized":"2024-03-28T18:56:16.178319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:24.791611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993997+0000","last_change":"2024-03-28T18:55:33.494720+0000","last_active":"2024-03-28T18:56:15.993997+0000","last_peered":"2024-03-28T18:56:15.993997+0000","last_clean":"2024-03-28T18:56:15.993997+0000","last_became_active":"2024-03-28T18:55:33.494247+0000","last_became_peered":"2024-03-28T18:55:33.494247+0000","last_unstale":"2024-03-28T18:56:15.993997+0000","last_undegraded":"2024-03-28T18:56:15.993997+0000","last_fullsized":"2024-03-28T18:56:15.993997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993504+0000","last_change":"2024-03-28T18:55:44.607744+0000","last_active":"2024-03-28T18:56:15.993504+0000","last_peered":"2024-03-28T18:56:15.993504+0000","last_clean":"2024-03-28T18:56:15.993504+0000","last_became_active":"2024-03-28T18:55:44.606993+0000","last_became_peered":"2024-03-28T18:55:44.606993+0000","last_unstale":"2024-03-28T18:56:15.993504+0000","last_undegraded":"2024-03-28T18:56:15.993504+0000","last_fullsized":"2024-03-28T18:56:15.993504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:41.313667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993475+0000","last_change":"2024-03-28T18:55:45.606307+0000","last_active":"2024-03-28T18:56:15.993475+0000","last_peered":"2024-03-28T18:56:15.993475+0000","last_clean":"2024-03-28T18:56:15.993475+0000","last_became_active":"2024-03-28T18:55:45.605980+0000","last_became_peered":"2024-03-28T18:55:45.605980+0000","last_unstale":"2024-03-28T18:56:15.993475+0000","last_undegraded":"2024-03-28T18:56:15.993475+0000","last_fullsized":"2024-03-28T18:56:15.993475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:52.485966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996297+0000","last_change":"2024-03-28T18:55:34.506818+0000","last_active":"2024-03-28T18:56:15.996297+0000","last_peered":"2024-03-28T18:56:15.996297+0000","last_clean":"2024-03-28T18:56:15.996297+0000","last_became_active":"2024-03-28T18:55:34.506067+0000","last_became_peered":"2024-03-28T18:55:34.506067+0000","last_unstale":"2024-03-28T18:56:15.996297+0000","last_undegraded":"2024-03-28T18:56:15.996297+0000","last_fullsized":"2024-03-28T18:56:15.996297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996334+0000","last_change":"2024-03-28T18:55:33.504861+0000","last_active":"2024-03-28T18:56:15.996334+0000","last_peered":"2024-03-28T18:56:15.996334+0000","last_clean":"2024-03-28T18:56:15.996334+0000","last_became_active":"2024-03-28T18:55:33.502416+0000","last_became_peered":"2024-03-28T18:55:33.502416+0000","last_unstale":"2024-03-28T18:56:15.996334+0000","last_undegraded":"2024-03-28T18:56:15.996334+0000","last_fullsized":"2024-03-28T18:56:15.996334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:12.626918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373024+0000","last_change":"2024-03-28T18:55:44.607323+0000","last_active":"2024-03-28T18:56:16.373024+0000","last_peered":"2024-03-28T18:56:16.373024+0000","last_clean":"2024-03-28T18:56:16.373024+0000","last_became_active":"2024-03-28T18:55:44.606579+0000","last_became_peered":"2024-03-28T18:55:44.606579+0000","last_unstale":"2024-03-28T18:56:16.373024+0000","last_undegraded":"2024-03-28T18:56:16.373024+0000","last_fullsized":"2024-03-28T18:56:16.373024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:14.900004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176822+0000","last_change":"2024-03-28T18:55:45.622760+0000","last_active":"2024-03-28T18:56:16.176822+0000","last_peered":"2024-03-28T18:56:16.176822+0000","last_clean":"2024-03-28T18:56:16.176822+0000","last_became_active":"2024-03-28T18:55:45.622353+0000","last_became_peered":"2024-03-28T18:55:45.622353+0000","last_unstale":"2024-03-28T18:56:16.176822+0000","last_undegraded":"2024-03-28T18:56:16.176822+0000","last_fullsized":"2024-03-28T18:56:16.176822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:18:04.349689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179332+0000","last_change":"2024-03-28T18:55:34.504167+0000","last_active":"2024-03-28T18:56:16.179332+0000","last_peered":"2024-03-28T18:56:16.179332+0000","last_clean":"2024-03-28T18:56:16.179332+0000","last_became_active":"2024-03-28T18:55:34.503397+0000","last_became_peered":"2024-03-28T18:55:34.503397+0000","last_unstale":"2024-03-28T18:56:16.179332+0000","last_undegraded":"2024-03-28T18:56:16.179332+0000","last_fullsized":"2024-03-28T18:56:16.179332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:33.911825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993245+0000","last_change":"2024-03-28T18:55:33.499168+0000","last_active":"2024-03-28T18:56:15.993245+0000","last_peered":"2024-03-28T18:56:15.993245+0000","last_clean":"2024-03-28T18:56:15.993245+0000","last_became_active":"2024-03-28T18:55:33.498728+0000","last_became_peered":"2024-03-28T18:55:33.498728+0000","last_unstale":"2024-03-28T18:56:15.993245+0000","last_undegraded":"2024-03-28T18:56:15.993245+0000","last_fullsized":"2024-03-28T18:56:15.993245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994698+0000","last_change":"2024-03-28T18:55:44.607237+0000","last_active":"2024-03-28T18:56:15.994698+0000","last_peered":"2024-03-28T18:56:15.994698+0000","last_clean":"2024-03-28T18:56:15.994698+0000","last_became_active":"2024-03-28T18:55:44.606960+0000","last_became_peered":"2024-03-28T18:55:44.606960+0000","last_unstale":"2024-03-28T18:56:15.994698+0000","last_undegraded":"2024-03-28T18:56:15.994698+0000","last_fullsized":"2024-03-28T18:56:15.994698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:05.313280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993370+0000","last_change":"2024-03-28T18:55:45.612992+0000","last_active":"2024-03-28T18:56:15.993370+0000","last_peered":"2024-03-28T18:56:15.993370+0000","last_clean":"2024-03-28T18:56:15.993370+0000","last_became_active":"2024-03-28T18:55:45.612125+0000","last_became_peered":"2024-03-28T18:55:45.612125+0000","last_unstale":"2024-03-28T18:56:15.993370+0000","last_undegraded":"2024-03-28T18:56:15.993370+0000","last_fullsized":"2024-03-28T18:56:15.993370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:51:35.898911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179159+0000","last_change":"2024-03-28T18:55:34.501366+0000","last_active":"2024-03-28T18:56:16.179159+0000","last_peered":"2024-03-28T18:56:16.179159+0000","last_clean":"2024-03-28T18:56:16.179159+0000","last_became_active":"2024-03-28T18:55:34.501046+0000","last_became_peered":"2024-03-28T18:55:34.501046+0000","last_unstale":"2024-03-28T18:56:16.179159+0000","last_undegraded":"2024-03-28T18:56:16.179159+0000","last_fullsized":"2024-03-28T18:56:16.179159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166759+0000","last_change":"2024-03-28T18:55:33.499012+0000","last_active":"2024-03-28T18:56:16.166759+0000","last_peered":"2024-03-28T18:56:16.166759+0000","last_clean":"2024-03-28T18:56:16.166759+0000","last_became_active":"2024-03-28T18:55:33.496890+0000","last_became_peered":"2024-03-28T18:55:33.496890+0000","last_unstale":"2024-03-28T18:56:16.166759+0000","last_undegraded":"2024-03-28T18:56:16.166759+0000","last_fullsized":"2024-03-28T18:56:16.166759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:08:32.841393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164552+0000","last_change":"2024-03-28T18:55:44.607218+0000","last_active":"2024-03-28T18:56:16.164552+0000","last_peered":"2024-03-28T18:56:16.164552+0000","last_clean":"2024-03-28T18:56:16.164552+0000","last_became_active":"2024-03-28T18:55:44.606852+0000","last_became_peered":"2024-03-28T18:55:44.606852+0000","last_unstale":"2024-03-28T18:56:16.164552+0000","last_undegraded":"2024-03-28T18:56:16.164552+0000","last_fullsized":"2024-03-28T18:56:16.164552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:36.624289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992021+0000","last_change":"2024-03-28T18:55:45.618919+0000","last_active":"2024-03-28T18:56:15.992021+0000","last_peered":"2024-03-28T18:56:15.992021+0000","last_clean":"2024-03-28T18:56:15.992021+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:56:15.992021+0000","last_undegraded":"2024-03-28T18:56:15.992021+0000","last_fullsized":"2024-03-28T18:56:15.992021+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:29.407508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178011+0000","last_change":"2024-03-28T18:55:34.494525+0000","last_active":"2024-03-28T18:56:16.178011+0000","last_peered":"2024-03-28T18:56:16.178011+0000","last_clean":"2024-03-28T18:56:16.178011+0000","last_became_active":"2024-03-28T18:55:34.494199+0000","last_became_peered":"2024-03-28T18:55:34.494199+0000","last_unstale":"2024-03-28T18:56:16.178011+0000","last_undegraded":"2024-03-28T18:56:16.178011+0000","last_fullsized":"2024-03-28T18:56:16.178011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993501+0000","last_change":"2024-03-28T18:55:33.503680+0000","last_active":"2024-03-28T18:56:15.993501+0000","last_peered":"2024-03-28T18:56:15.993501+0000","last_clean":"2024-03-28T18:56:15.993501+0000","last_became_active":"2024-03-28T18:55:33.503446+0000","last_became_peered":"2024-03-28T18:55:33.503446+0000","last_unstale":"2024-03-28T18:56:15.993501+0000","last_undegraded":"2024-03-28T18:56:15.993501+0000","last_fullsized":"2024-03-28T18:56:15.993501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.374029+0000","last_change":"2024-03-28T18:55:44.598881+0000","last_active":"2024-03-28T18:56:16.374029+0000","last_peered":"2024-03-28T18:56:16.374029+0000","last_clean":"2024-03-28T18:56:16.374029+0000","last_became_active":"2024-03-28T18:55:44.598532+0000","last_became_peered":"2024-03-28T18:55:44.598532+0000","last_unstale":"2024-03-28T18:56:16.374029+0000","last_undegraded":"2024-03-28T18:56:16.374029+0000","last_fullsized":"2024-03-28T18:56:16.374029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:43:44.004611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996863+0000","last_change":"2024-03-28T18:55:45.612572+0000","last_active":"2024-03-28T18:56:15.996863+0000","last_peered":"2024-03-28T18:56:15.996863+0000","last_clean":"2024-03-28T18:56:15.996863+0000","last_became_active":"2024-03-28T18:55:45.611844+0000","last_became_peered":"2024-03-28T18:55:45.611844+0000","last_unstale":"2024-03-28T18:56:15.996863+0000","last_undegraded":"2024-03-28T18:56:15.996863+0000","last_fullsized":"2024-03-28T18:56:15.996863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992765+0000","last_change":"2024-03-28T18:55:34.490059+0000","last_active":"2024-03-28T18:56:15.992765+0000","last_peered":"2024-03-28T18:56:15.992765+0000","last_clean":"2024-03-28T18:56:15.992765+0000","last_became_active":"2024-03-28T18:55:34.488566+0000","last_became_peered":"2024-03-28T18:55:34.488566+0000","last_unstale":"2024-03-28T18:56:15.992765+0000","last_undegraded":"2024-03-28T18:56:15.992765+0000","last_fullsized":"2024-03-28T18:56:15.992765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993659+0000","last_change":"2024-03-28T18:55:33.494143+0000","last_active":"2024-03-28T18:56:15.993659+0000","last_peered":"2024-03-28T18:56:15.993659+0000","last_clean":"2024-03-28T18:56:15.993659+0000","last_became_active":"2024-03-28T18:55:33.493764+0000","last_became_peered":"2024-03-28T18:55:33.493764+0000","last_unstale":"2024-03-28T18:56:15.993659+0000","last_undegraded":"2024-03-28T18:56:15.993659+0000","last_fullsized":"2024-03-28T18:56:15.993659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992080+0000","last_change":"2024-03-28T18:55:44.609644+0000","last_active":"2024-03-28T18:56:15.992080+0000","last_peered":"2024-03-28T18:56:15.992080+0000","last_clean":"2024-03-28T18:56:15.992080+0000","last_became_active":"2024-03-28T18:55:44.609317+0000","last_became_peered":"2024-03-28T18:55:44.609317+0000","last_unstale":"2024-03-28T18:56:15.992080+0000","last_undegraded":"2024-03-28T18:56:15.992080+0000","last_fullsized":"2024-03-28T18:56:15.992080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:37.662504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996075+0000","last_change":"2024-03-28T18:55:45.621708+0000","last_active":"2024-03-28T18:56:15.996075+0000","last_peered":"2024-03-28T18:56:15.996075+0000","last_clean":"2024-03-28T18:56:15.996075+0000","last_became_active":"2024-03-28T18:55:45.621366+0000","last_became_peered":"2024-03-28T18:55:45.621366+0000","last_unstale":"2024-03-28T18:56:15.996075+0000","last_undegraded":"2024-03-28T18:56:15.996075+0000","last_fullsized":"2024-03-28T18:56:15.996075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177583+0000","last_change":"2024-03-28T18:55:34.488191+0000","last_active":"2024-03-28T18:56:16.177583+0000","last_peered":"2024-03-28T18:56:16.177583+0000","last_clean":"2024-03-28T18:56:16.177583+0000","last_became_active":"2024-03-28T18:55:34.487917+0000","last_became_peered":"2024-03-28T18:55:34.487917+0000","last_unstale":"2024-03-28T18:56:16.177583+0000","last_undegraded":"2024-03-28T18:56:16.177583+0000","last_fullsized":"2024-03-28T18:56:16.177583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:38.066254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996044+0000","last_change":"2024-03-28T18:55:33.497935+0000","last_active":"2024-03-28T18:56:15.996044+0000","last_peered":"2024-03-28T18:56:15.996044+0000","last_clean":"2024-03-28T18:56:15.996044+0000","last_became_active":"2024-03-28T18:55:33.497599+0000","last_became_peered":"2024-03-28T18:55:33.497599+0000","last_unstale":"2024-03-28T18:56:15.996044+0000","last_undegraded":"2024-03-28T18:56:15.996044+0000","last_fullsized":"2024-03-28T18:56:15.996044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:36:04.772606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"26'9","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:20.908069+0000","last_change":"2024-03-28T18:55:44.608949+0000","last_active":"2024-03-28T18:56:20.908069+0000","last_peered":"2024-03-28T18:56:20.908069+0000","last_clean":"2024-03-28T18:56:20.908069+0000","last_became_active":"2024-03-28T18:55:44.608657+0000","last_became_peered":"2024-03-28T18:55:44.608657+0000","last_unstale":"2024-03-28T18:56:20.908069+0000","last_undegraded":"2024-03-28T18:56:20.908069+0000","last_fullsized":"2024-03-28T18:56:20.908069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:04.799314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990894+0000","last_change":"2024-03-28T18:55:45.610686+0000","last_active":"2024-03-28T18:56:15.990894+0000","last_peered":"2024-03-28T18:56:15.990894+0000","last_clean":"2024-03-28T18:56:15.990894+0000","last_became_active":"2024-03-28T18:55:45.610212+0000","last_became_peered":"2024-03-28T18:55:45.610212+0000","last_unstale":"2024-03-28T18:56:15.990894+0000","last_undegraded":"2024-03-28T18:56:15.990894+0000","last_fullsized":"2024-03-28T18:56:15.990894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:52:00.472137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372769+0000","last_change":"2024-03-28T18:55:34.481953+0000","last_active":"2024-03-28T18:56:16.372769+0000","last_peered":"2024-03-28T18:56:16.372769+0000","last_clean":"2024-03-28T18:56:16.372769+0000","last_became_active":"2024-03-28T18:55:34.481636+0000","last_became_peered":"2024-03-28T18:55:34.481636+0000","last_unstale":"2024-03-28T18:56:16.372769+0000","last_undegraded":"2024-03-28T18:56:16.372769+0000","last_fullsized":"2024-03-28T18:56:16.372769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373687+0000","last_change":"2024-03-28T18:55:33.498632+0000","last_active":"2024-03-28T18:56:16.373687+0000","last_peered":"2024-03-28T18:56:16.373687+0000","last_clean":"2024-03-28T18:56:16.373687+0000","last_became_active":"2024-03-28T18:55:33.498198+0000","last_became_peered":"2024-03-28T18:55:33.498198+0000","last_unstale":"2024-03-28T18:56:16.373687+0000","last_undegraded":"2024-03-28T18:56:16.373687+0000","last_fullsized":"2024-03-28T18:56:16.373687+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:43.859886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"23'3","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996740+0000","last_change":"2024-03-28T18:55:44.602762+0000","last_active":"2024-03-28T18:56:15.996740+0000","last_peered":"2024-03-28T18:56:15.996740+0000","last_clean":"2024-03-28T18:56:15.996740+0000","last_became_active":"2024-03-28T18:55:44.602424+0000","last_became_peered":"2024-03-28T18:55:44.602424+0000","last_unstale":"2024-03-28T18:56:15.996740+0000","last_undegraded":"2024-03-28T18:56:15.996740+0000","last_fullsized":"2024-03-28T18:56:15.996740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:27.436786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990431+0000","last_change":"2024-03-28T18:55:45.601277+0000","last_active":"2024-03-28T18:56:15.990431+0000","last_peered":"2024-03-28T18:56:15.990431+0000","last_clean":"2024-03-28T18:56:15.990431+0000","last_became_active":"2024-03-28T18:55:45.600945+0000","last_became_peered":"2024-03-28T18:55:45.600945+0000","last_unstale":"2024-03-28T18:56:15.990431+0000","last_undegraded":"2024-03-28T18:56:15.990431+0000","last_fullsized":"2024-03-28T18:56:15.990431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178326+0000","last_change":"2024-03-28T18:55:34.496602+0000","last_active":"2024-03-28T18:56:16.178326+0000","last_peered":"2024-03-28T18:56:16.178326+0000","last_clean":"2024-03-28T18:56:16.178326+0000","last_became_active":"2024-03-28T18:55:34.496302+0000","last_became_peered":"2024-03-28T18:55:34.496302+0000","last_unstale":"2024-03-28T18:56:16.178326+0000","last_undegraded":"2024-03-28T18:56:16.178326+0000","last_fullsized":"2024-03-28T18:56:16.178326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:31.220882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993822+0000","last_change":"2024-03-28T18:55:33.496921+0000","last_active":"2024-03-28T18:56:15.993822+0000","last_peered":"2024-03-28T18:56:15.993822+0000","last_clean":"2024-03-28T18:56:15.993822+0000","last_became_active":"2024-03-28T18:55:33.496456+0000","last_became_peered":"2024-03-28T18:55:33.496456+0000","last_unstale":"2024-03-28T18:56:15.993822+0000","last_undegraded":"2024-03-28T18:56:15.993822+0000","last_fullsized":"2024-03-28T18:56:15.993822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994697+0000","last_change":"2024-03-28T18:55:44.599432+0000","last_active":"2024-03-28T18:56:15.994697+0000","last_peered":"2024-03-28T18:56:15.994697+0000","last_clean":"2024-03-28T18:56:15.994697+0000","last_became_active":"2024-03-28T18:55:44.599123+0000","last_became_peered":"2024-03-28T18:55:44.599123+0000","last_unstale":"2024-03-28T18:56:15.994697+0000","last_undegraded":"2024-03-28T18:56:15.994697+0000","last_fullsized":"2024-03-28T18:56:15.994697+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994904+0000","last_change":"2024-03-28T18:55:45.624311+0000","last_active":"2024-03-28T18:56:15.994904+0000","last_peered":"2024-03-28T18:56:15.994904+0000","last_clean":"2024-03-28T18:56:15.994904+0000","last_became_active":"2024-03-28T18:55:45.624031+0000","last_became_peered":"2024-03-28T18:55:45.624031+0000","last_unstale":"2024-03-28T18:56:15.994904+0000","last_undegraded":"2024-03-28T18:56:15.994904+0000","last_fullsized":"2024-03-28T18:56:15.994904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166129+0000","last_change":"2024-03-28T18:55:34.498879+0000","last_active":"2024-03-28T18:56:16.166129+0000","last_peered":"2024-03-28T18:56:16.166129+0000","last_clean":"2024-03-28T18:56:16.166129+0000","last_became_active":"2024-03-28T18:55:34.498515+0000","last_became_peered":"2024-03-28T18:55:34.498515+0000","last_unstale":"2024-03-28T18:56:16.166129+0000","last_undegraded":"2024-03-28T18:56:16.166129+0000","last_fullsized":"2024-03-28T18:56:16.166129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996267+0000","last_change":"2024-03-28T18:55:33.511361+0000","last_active":"2024-03-28T18:56:15.996267+0000","last_peered":"2024-03-28T18:56:15.996267+0000","last_clean":"2024-03-28T18:56:15.996267+0000","last_became_active":"2024-03-28T18:55:33.511062+0000","last_became_peered":"2024-03-28T18:55:33.511062+0000","last_unstale":"2024-03-28T18:56:15.996267+0000","last_undegraded":"2024-03-28T18:56:15.996267+0000","last_fullsized":"2024-03-28T18:56:15.996267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165151+0000","last_change":"2024-03-28T18:55:44.593567+0000","last_active":"2024-03-28T18:56:16.165151+0000","last_peered":"2024-03-28T18:56:16.165151+0000","last_clean":"2024-03-28T18:56:16.165151+0000","last_became_active":"2024-03-28T18:55:44.593267+0000","last_became_peered":"2024-03-28T18:55:44.593267+0000","last_unstale":"2024-03-28T18:56:16.165151+0000","last_undegraded":"2024-03-28T18:56:16.165151+0000","last_fullsized":"2024-03-28T18:56:16.165151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993212+0000","last_change":"2024-03-28T18:55:45.598713+0000","last_active":"2024-03-28T18:56:15.993212+0000","last_peered":"2024-03-28T18:56:15.993212+0000","last_clean":"2024-03-28T18:56:15.993212+0000","last_became_active":"2024-03-28T18:55:45.597726+0000","last_became_peered":"2024-03-28T18:55:45.597726+0000","last_unstale":"2024-03-28T18:56:15.993212+0000","last_undegraded":"2024-03-28T18:56:15.993212+0000","last_fullsized":"2024-03-28T18:56:15.993212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:48.255745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373226+0000","last_change":"2024-03-28T18:55:34.496237+0000","last_active":"2024-03-28T18:56:16.373226+0000","last_peered":"2024-03-28T18:56:16.373226+0000","last_clean":"2024-03-28T18:56:16.373226+0000","last_became_active":"2024-03-28T18:55:34.495681+0000","last_became_peered":"2024-03-28T18:55:34.495681+0000","last_unstale":"2024-03-28T18:56:16.373226+0000","last_undegraded":"2024-03-28T18:56:16.373226+0000","last_fullsized":"2024-03-28T18:56:16.373226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997012+0000","last_change":"2024-03-28T18:55:33.505972+0000","last_active":"2024-03-28T18:56:15.997012+0000","last_peered":"2024-03-28T18:56:15.997012+0000","last_clean":"2024-03-28T18:56:15.997012+0000","last_became_active":"2024-03-28T18:55:33.501871+0000","last_became_peered":"2024-03-28T18:55:33.501871+0000","last_unstale":"2024-03-28T18:56:15.997012+0000","last_undegraded":"2024-03-28T18:56:15.997012+0000","last_fullsized":"2024-03-28T18:56:15.997012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"23'8","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989655+0000","last_change":"2024-03-28T18:55:44.610673+0000","last_active":"2024-03-28T18:56:15.989655+0000","last_peered":"2024-03-28T18:56:15.989655+0000","last_clean":"2024-03-28T18:56:15.989655+0000","last_became_active":"2024-03-28T18:55:44.610357+0000","last_became_peered":"2024-03-28T18:55:44.610357+0000","last_unstale":"2024-03-28T18:56:15.989655+0000","last_undegraded":"2024-03-28T18:56:15.989655+0000","last_fullsized":"2024-03-28T18:56:15.989655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:26.280548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177029+0000","last_change":"2024-03-28T18:55:45.609349+0000","last_active":"2024-03-28T18:56:16.177029+0000","last_peered":"2024-03-28T18:56:16.177029+0000","last_clean":"2024-03-28T18:56:16.177029+0000","last_became_active":"2024-03-28T18:55:45.608743+0000","last_became_peered":"2024-03-28T18:55:45.608743+0000","last_unstale":"2024-03-28T18:56:16.177029+0000","last_undegraded":"2024-03-28T18:56:16.177029+0000","last_fullsized":"2024-03-28T18:56:16.177029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996992+0000","last_change":"2024-03-28T18:55:34.501624+0000","last_active":"2024-03-28T18:56:15.996992+0000","last_peered":"2024-03-28T18:56:15.996992+0000","last_clean":"2024-03-28T18:56:15.996992+0000","last_became_active":"2024-03-28T18:55:34.501344+0000","last_became_peered":"2024-03-28T18:55:34.501344+0000","last_unstale":"2024-03-28T18:56:15.996992+0000","last_undegraded":"2024-03-28T18:56:15.996992+0000","last_fullsized":"2024-03-28T18:56:15.996992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994769+0000","last_change":"2024-03-28T18:55:33.497237+0000","last_active":"2024-03-28T18:56:15.994769+0000","last_peered":"2024-03-28T18:56:15.994769+0000","last_clean":"2024-03-28T18:56:15.994769+0000","last_became_active":"2024-03-28T18:55:33.496905+0000","last_became_peered":"2024-03-28T18:55:33.496905+0000","last_unstale":"2024-03-28T18:56:15.994769+0000","last_undegraded":"2024-03-28T18:56:15.994769+0000","last_fullsized":"2024-03-28T18:56:15.994769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:41.141604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989590+0000","last_change":"2024-03-28T18:55:44.601103+0000","last_active":"2024-03-28T18:56:15.989590+0000","last_peered":"2024-03-28T18:56:15.989590+0000","last_clean":"2024-03-28T18:56:15.989590+0000","last_became_active":"2024-03-28T18:55:44.599969+0000","last_became_peered":"2024-03-28T18:55:44.599969+0000","last_unstale":"2024-03-28T18:56:15.989590+0000","last_undegraded":"2024-03-28T18:56:15.989590+0000","last_fullsized":"2024-03-28T18:56:15.989590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:42.018654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994243+0000","last_change":"2024-03-28T18:55:45.594944+0000","last_active":"2024-03-28T18:56:15.994243+0000","last_peered":"2024-03-28T18:56:15.994243+0000","last_clean":"2024-03-28T18:56:15.994243+0000","last_became_active":"2024-03-28T18:55:45.594603+0000","last_became_peered":"2024-03-28T18:55:45.594603+0000","last_unstale":"2024-03-28T18:56:15.994243+0000","last_undegraded":"2024-03-28T18:56:15.994243+0000","last_fullsized":"2024-03-28T18:56:15.994243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"23'1","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373359+0000","last_change":"2024-03-28T18:55:44.602366+0000","last_active":"2024-03-28T18:56:16.373359+0000","last_peered":"2024-03-28T18:56:16.373359+0000","last_clean":"2024-03-28T18:56:16.373359+0000","last_became_active":"2024-03-28T18:55:44.602033+0000","last_became_peered":"2024-03-28T18:55:44.602033+0000","last_unstale":"2024-03-28T18:56:16.373359+0000","last_undegraded":"2024-03-28T18:56:16.373359+0000","last_fullsized":"2024-03-28T18:56:16.373359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:20.212765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994424+0000","last_change":"2024-03-28T18:55:34.498959+0000","last_active":"2024-03-28T18:56:15.994424+0000","last_peered":"2024-03-28T18:56:15.994424+0000","last_clean":"2024-03-28T18:56:15.994424+0000","last_became_active":"2024-03-28T18:55:34.498612+0000","last_became_peered":"2024-03-28T18:55:34.498612+0000","last_unstale":"2024-03-28T18:56:15.994424+0000","last_undegraded":"2024-03-28T18:56:15.994424+0000","last_fullsized":"2024-03-28T18:56:15.994424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995004+0000","last_change":"2024-03-28T18:55:33.499197+0000","last_active":"2024-03-28T18:56:15.995004+0000","last_peered":"2024-03-28T18:56:15.995004+0000","last_clean":"2024-03-28T18:56:15.995004+0000","last_became_active":"2024-03-28T18:55:33.496742+0000","last_became_peered":"2024-03-28T18:55:33.496742+0000","last_unstale":"2024-03-28T18:56:15.995004+0000","last_undegraded":"2024-03-28T18:56:15.995004+0000","last_fullsized":"2024-03-28T18:56:15.995004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:33.423481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373198+0000","last_change":"2024-03-28T18:55:45.620559+0000","last_active":"2024-03-28T18:56:16.373198+0000","last_peered":"2024-03-28T18:56:16.373198+0000","last_clean":"2024-03-28T18:56:16.373198+0000","last_became_active":"2024-03-28T18:55:45.620249+0000","last_became_peered":"2024-03-28T18:55:45.620249+0000","last_unstale":"2024-03-28T18:56:16.373198+0000","last_undegraded":"2024-03-28T18:56:16.373198+0000","last_fullsized":"2024-03-28T18:56:16.373198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"23'4","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994418+0000","last_change":"2024-03-28T18:55:44.612226+0000","last_active":"2024-03-28T18:56:15.994418+0000","last_peered":"2024-03-28T18:56:15.994418+0000","last_clean":"2024-03-28T18:56:15.994418+0000","last_became_active":"2024-03-28T18:55:44.611921+0000","last_became_peered":"2024-03-28T18:55:44.611921+0000","last_unstale":"2024-03-28T18:56:15.994418+0000","last_undegraded":"2024-03-28T18:56:15.994418+0000","last_fullsized":"2024-03-28T18:56:15.994418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:07:48.159039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989828+0000","last_change":"2024-03-28T18:55:34.496041+0000","last_active":"2024-03-28T18:56:15.989828+0000","last_peered":"2024-03-28T18:56:15.989828+0000","last_clean":"2024-03-28T18:56:15.989828+0000","last_became_active":"2024-03-28T18:55:34.492488+0000","last_became_peered":"2024-03-28T18:55:34.492488+0000","last_unstale":"2024-03-28T18:56:15.989828+0000","last_undegraded":"2024-03-28T18:56:15.989828+0000","last_fullsized":"2024-03-28T18:56:15.989828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990026+0000","last_change":"2024-03-28T18:55:33.502125+0000","last_active":"2024-03-28T18:56:15.990026+0000","last_peered":"2024-03-28T18:56:15.990026+0000","last_clean":"2024-03-28T18:56:15.990026+0000","last_became_active":"2024-03-28T18:55:33.499835+0000","last_became_peered":"2024-03-28T18:55:33.499835+0000","last_unstale":"2024-03-28T18:56:15.990026+0000","last_undegraded":"2024-03-28T18:56:15.990026+0000","last_fullsized":"2024-03-28T18:56:15.990026+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:24:27.118620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996154+0000","last_change":"2024-03-28T18:55:45.612727+0000","last_active":"2024-03-28T18:56:15.996154+0000","last_peered":"2024-03-28T18:56:15.996154+0000","last_clean":"2024-03-28T18:56:15.996154+0000","last_became_active":"2024-03-28T18:55:45.610154+0000","last_became_peered":"2024-03-28T18:55:45.610154+0000","last_unstale":"2024-03-28T18:56:15.996154+0000","last_undegraded":"2024-03-28T18:56:15.996154+0000","last_fullsized":"2024-03-28T18:56:15.996154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994188+0000","last_change":"2024-03-28T18:55:33.490410+0000","last_active":"2024-03-28T18:56:15.994188+0000","last_peered":"2024-03-28T18:56:15.994188+0000","last_clean":"2024-03-28T18:56:15.994188+0000","last_became_active":"2024-03-28T18:55:33.490132+0000","last_became_peered":"2024-03-28T18:55:33.490132+0000","last_unstale":"2024-03-28T18:56:15.994188+0000","last_undegraded":"2024-03-28T18:56:15.994188+0000","last_fullsized":"2024-03-28T18:56:15.994188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:46:30.445425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373560+0000","last_change":"2024-03-28T18:55:34.487716+0000","last_active":"2024-03-28T18:56:16.373560+0000","last_peered":"2024-03-28T18:56:16.373560+0000","last_clean":"2024-03-28T18:56:16.373560+0000","last_became_active":"2024-03-28T18:55:34.487193+0000","last_became_peered":"2024-03-28T18:55:34.487193+0000","last_unstale":"2024-03-28T18:56:16.373560+0000","last_undegraded":"2024-03-28T18:56:16.373560+0000","last_fullsized":"2024-03-28T18:56:16.373560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993632+0000","last_change":"2024-03-28T18:55:44.592835+0000","last_active":"2024-03-28T18:56:15.993632+0000","last_peered":"2024-03-28T18:56:15.993632+0000","last_clean":"2024-03-28T18:56:15.993632+0000","last_became_active":"2024-03-28T18:55:44.592505+0000","last_became_peered":"2024-03-28T18:55:44.592505+0000","last_unstale":"2024-03-28T18:56:15.993632+0000","last_undegraded":"2024-03-28T18:56:15.993632+0000","last_fullsized":"2024-03-28T18:56:15.993632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:58.286421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165539+0000","last_change":"2024-03-28T18:55:45.611322+0000","last_active":"2024-03-28T18:56:16.165539+0000","last_peered":"2024-03-28T18:56:16.165539+0000","last_clean":"2024-03-28T18:56:16.165539+0000","last_became_active":"2024-03-28T18:55:45.610958+0000","last_became_peered":"2024-03-28T18:55:45.610958+0000","last_unstale":"2024-03-28T18:56:16.165539+0000","last_undegraded":"2024-03-28T18:56:16.165539+0000","last_fullsized":"2024-03-28T18:56:16.165539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990629+0000","last_change":"2024-03-28T18:55:33.505595+0000","last_active":"2024-03-28T18:56:15.990629+0000","last_peered":"2024-03-28T18:56:15.990629+0000","last_clean":"2024-03-28T18:56:15.990629+0000","last_became_active":"2024-03-28T18:55:33.505334+0000","last_became_peered":"2024-03-28T18:55:33.505334+0000","last_unstale":"2024-03-28T18:56:15.990629+0000","last_undegraded":"2024-03-28T18:56:15.990629+0000","last_fullsized":"2024-03-28T18:56:15.990629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:32.431381+0000","last_clean_scrub_stamp":"2024-03-28T18:55:32.431381+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:37.165037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.165083+0000","last_change":"2024-03-28T18:55:34.490306+0000","last_active":"2024-03-28T18:56:16.165083+0000","last_peered":"2024-03-28T18:56:16.165083+0000","last_clean":"2024-03-28T18:56:16.165083+0000","last_became_active":"2024-03-28T18:55:34.488811+0000","last_became_peered":"2024-03-28T18:55:34.488811+0000","last_unstale":"2024-03-28T18:56:16.165083+0000","last_undegraded":"2024-03-28T18:56:16.165083+0000","last_fullsized":"2024-03-28T18:56:16.165083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:43:32.932294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"23'2","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991514+0000","last_change":"2024-03-28T18:55:44.608283+0000","last_active":"2024-03-28T18:56:15.991514+0000","last_peered":"2024-03-28T18:56:15.991514+0000","last_clean":"2024-03-28T18:56:15.991514+0000","last_became_active":"2024-03-28T18:55:44.607999+0000","last_became_peered":"2024-03-28T18:55:44.607999+0000","last_unstale":"2024-03-28T18:56:15.991514+0000","last_undegraded":"2024-03-28T18:56:15.991514+0000","last_fullsized":"2024-03-28T18:56:15.991514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:43.539066+0000","last_clean_scrub_stamp":"2024-03-28T18:55:43.539066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:22.669778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991790+0000","last_change":"2024-03-28T18:55:45.618390+0000","last_active":"2024-03-28T18:56:15.991790+0000","last_peered":"2024-03-28T18:56:15.991790+0000","last_clean":"2024-03-28T18:56:15.991790+0000","last_became_active":"2024-03-28T18:55:45.617897+0000","last_became_peered":"2024-03-28T18:55:45.617897+0000","last_unstale":"2024-03-28T18:56:15.991790+0000","last_undegraded":"2024-03-28T18:56:15.991790+0000","last_fullsized":"2024-03-28T18:56:15.991790+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:08:16.520575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992753+0000","last_change":"2024-03-28T18:55:34.487888+0000","last_active":"2024-03-28T18:56:15.992753+0000","last_peered":"2024-03-28T18:56:15.992753+0000","last_clean":"2024-03-28T18:56:15.992753+0000","last_became_active":"2024-03-28T18:55:34.487590+0000","last_became_peered":"2024-03-28T18:55:34.487590+0000","last_unstale":"2024-03-28T18:56:15.992753+0000","last_undegraded":"2024-03-28T18:56:15.992753+0000","last_fullsized":"2024-03-28T18:56:15.992753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996140+0000","last_change":"2024-03-28T18:55:45.624938+0000","last_active":"2024-03-28T18:56:15.996140+0000","last_peered":"2024-03-28T18:56:15.996140+0000","last_clean":"2024-03-28T18:56:15.996140+0000","last_became_active":"2024-03-28T18:55:45.624362+0000","last_became_peered":"2024-03-28T18:55:45.624362+0000","last_unstale":"2024-03-28T18:56:15.996140+0000","last_undegraded":"2024-03-28T18:56:15.996140+0000","last_fullsized":"2024-03-28T18:56:15.996140+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:12:25.470445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992964+0000","last_change":"2024-03-28T18:55:34.503401+0000","last_active":"2024-03-28T18:56:15.992964+0000","last_peered":"2024-03-28T18:56:15.992964+0000","last_clean":"2024-03-28T18:56:15.992964+0000","last_became_active":"2024-03-28T18:55:34.503121+0000","last_became_peered":"2024-03-28T18:55:34.503121+0000","last_unstale":"2024-03-28T18:56:15.992964+0000","last_undegraded":"2024-03-28T18:56:15.992964+0000","last_fullsized":"2024-03-28T18:56:15.992964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:13.641817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995980+0000","last_change":"2024-03-28T18:55:45.610168+0000","last_active":"2024-03-28T18:56:15.995980+0000","last_peered":"2024-03-28T18:56:15.995980+0000","last_clean":"2024-03-28T18:56:15.995980+0000","last_became_active":"2024-03-28T18:55:45.609653+0000","last_became_peered":"2024-03-28T18:55:45.609653+0000","last_unstale":"2024-03-28T18:56:15.995980+0000","last_undegraded":"2024-03-28T18:56:15.995980+0000","last_fullsized":"2024-03-28T18:56:15.995980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995799+0000","last_change":"2024-03-28T18:55:34.501308+0000","last_active":"2024-03-28T18:56:15.995799+0000","last_peered":"2024-03-28T18:56:15.995799+0000","last_clean":"2024-03-28T18:56:15.995799+0000","last_became_active":"2024-03-28T18:55:34.500968+0000","last_became_peered":"2024-03-28T18:55:34.500968+0000","last_unstale":"2024-03-28T18:56:15.995799+0000","last_undegraded":"2024-03-28T18:56:15.995799+0000","last_fullsized":"2024-03-28T18:56:15.995799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996052+0000","last_change":"2024-03-28T18:55:45.613701+0000","last_active":"2024-03-28T18:56:15.996052+0000","last_peered":"2024-03-28T18:56:15.996052+0000","last_clean":"2024-03-28T18:56:15.996052+0000","last_became_active":"2024-03-28T18:55:45.613310+0000","last_became_peered":"2024-03-28T18:55:45.613310+0000","last_unstale":"2024-03-28T18:56:15.996052+0000","last_undegraded":"2024-03-28T18:56:15.996052+0000","last_fullsized":"2024-03-28T18:56:15.996052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373539+0000","last_change":"2024-03-28T18:55:34.496761+0000","last_active":"2024-03-28T18:56:16.373539+0000","last_peered":"2024-03-28T18:56:16.373539+0000","last_clean":"2024-03-28T18:56:16.373539+0000","last_became_active":"2024-03-28T18:55:34.496462+0000","last_became_peered":"2024-03-28T18:55:34.496462+0000","last_unstale":"2024-03-28T18:56:16.373539+0000","last_undegraded":"2024-03-28T18:56:16.373539+0000","last_fullsized":"2024-03-28T18:56:16.373539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177308+0000","last_change":"2024-03-28T18:55:45.615119+0000","last_active":"2024-03-28T18:56:16.177308+0000","last_peered":"2024-03-28T18:56:16.177308+0000","last_clean":"2024-03-28T18:56:16.177308+0000","last_became_active":"2024-03-28T18:55:45.613841+0000","last_became_peered":"2024-03-28T18:55:45.613841+0000","last_unstale":"2024-03-28T18:56:16.177308+0000","last_undegraded":"2024-03-28T18:56:16.177308+0000","last_fullsized":"2024-03-28T18:56:16.177308+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:31:02.766940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991068+0000","last_change":"2024-03-28T18:55:34.490035+0000","last_active":"2024-03-28T18:56:15.991068+0000","last_peered":"2024-03-28T18:56:15.991068+0000","last_clean":"2024-03-28T18:56:15.991068+0000","last_became_active":"2024-03-28T18:55:34.489669+0000","last_became_peered":"2024-03-28T18:55:34.489669+0000","last_unstale":"2024-03-28T18:56:15.991068+0000","last_undegraded":"2024-03-28T18:56:15.991068+0000","last_fullsized":"2024-03-28T18:56:15.991068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164511+0000","last_change":"2024-03-28T18:55:45.619605+0000","last_active":"2024-03-28T18:56:16.164511+0000","last_peered":"2024-03-28T18:56:16.164511+0000","last_clean":"2024-03-28T18:56:16.164511+0000","last_became_active":"2024-03-28T18:55:45.618473+0000","last_became_peered":"2024-03-28T18:55:45.618473+0000","last_unstale":"2024-03-28T18:56:16.164511+0000","last_undegraded":"2024-03-28T18:56:16.164511+0000","last_fullsized":"2024-03-28T18:56:16.164511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:23.684294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996253+0000","last_change":"2024-03-28T18:55:34.498901+0000","last_active":"2024-03-28T18:56:15.996253+0000","last_peered":"2024-03-28T18:56:15.996253+0000","last_clean":"2024-03-28T18:56:15.996253+0000","last_became_active":"2024-03-28T18:55:34.498534+0000","last_became_peered":"2024-03-28T18:55:34.498534+0000","last_unstale":"2024-03-28T18:56:15.996253+0000","last_undegraded":"2024-03-28T18:56:15.996253+0000","last_fullsized":"2024-03-28T18:56:15.996253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:38.355540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176839+0000","last_change":"2024-03-28T18:55:45.612284+0000","last_active":"2024-03-28T18:56:16.176839+0000","last_peered":"2024-03-28T18:56:16.176839+0000","last_clean":"2024-03-28T18:56:16.176839+0000","last_became_active":"2024-03-28T18:55:45.607218+0000","last_became_peered":"2024-03-28T18:55:45.607218+0000","last_unstale":"2024-03-28T18:56:16.176839+0000","last_undegraded":"2024-03-28T18:56:16.176839+0000","last_fullsized":"2024-03-28T18:56:16.176839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:22.822824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993957+0000","last_change":"2024-03-28T18:55:34.492145+0000","last_active":"2024-03-28T18:56:15.993957+0000","last_peered":"2024-03-28T18:56:15.993957+0000","last_clean":"2024-03-28T18:56:15.993957+0000","last_became_active":"2024-03-28T18:55:34.491814+0000","last_became_peered":"2024-03-28T18:55:34.491814+0000","last_unstale":"2024-03-28T18:56:15.993957+0000","last_undegraded":"2024-03-28T18:56:15.993957+0000","last_fullsized":"2024-03-28T18:56:15.993957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.992957+0000","last_change":"2024-03-28T18:55:45.607609+0000","last_active":"2024-03-28T18:56:15.992957+0000","last_peered":"2024-03-28T18:56:15.992957+0000","last_clean":"2024-03-28T18:56:15.992957+0000","last_became_active":"2024-03-28T18:55:45.602582+0000","last_became_peered":"2024-03-28T18:55:45.602582+0000","last_unstale":"2024-03-28T18:56:15.992957+0000","last_undegraded":"2024-03-28T18:56:15.992957+0000","last_fullsized":"2024-03-28T18:56:15.992957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:06:05.335715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.168035+0000","last_change":"2024-03-28T18:56:16.167085+0000","last_active":"2024-03-28T18:56:16.168035+0000","last_peered":"2024-03-28T18:56:16.168035+0000","last_clean":"2024-03-28T18:56:16.168035+0000","last_became_active":"2024-03-28T18:56:16.166729+0000","last_became_peered":"2024-03-28T18:56:16.166729+0000","last_unstale":"2024-03-28T18:56:16.168035+0000","last_undegraded":"2024-03-28T18:56:16.168035+0000","last_fullsized":"2024-03-28T18:56:16.168035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:51:49.364836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991762+0000","last_change":"2024-03-28T18:55:45.607886+0000","last_active":"2024-03-28T18:56:15.991762+0000","last_peered":"2024-03-28T18:56:15.991762+0000","last_clean":"2024-03-28T18:56:15.991762+0000","last_became_active":"2024-03-28T18:55:45.605964+0000","last_became_peered":"2024-03-28T18:55:45.605964+0000","last_unstale":"2024-03-28T18:56:15.991762+0000","last_undegraded":"2024-03-28T18:56:15.991762+0000","last_fullsized":"2024-03-28T18:56:15.991762+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:45.986654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995008+0000","last_change":"2024-03-28T18:55:34.501447+0000","last_active":"2024-03-28T18:56:15.995008+0000","last_peered":"2024-03-28T18:56:15.995008+0000","last_clean":"2024-03-28T18:56:15.995008+0000","last_became_active":"2024-03-28T18:55:34.497811+0000","last_became_peered":"2024-03-28T18:55:34.497811+0000","last_unstale":"2024-03-28T18:56:15.995008+0000","last_undegraded":"2024-03-28T18:56:15.995008+0000","last_fullsized":"2024-03-28T18:56:15.995008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178931+0000","last_change":"2024-03-28T18:55:45.618031+0000","last_active":"2024-03-28T18:56:16.178931+0000","last_peered":"2024-03-28T18:56:16.178931+0000","last_clean":"2024-03-28T18:56:16.178931+0000","last_became_active":"2024-03-28T18:55:45.617719+0000","last_became_peered":"2024-03-28T18:55:45.617719+0000","last_unstale":"2024-03-28T18:56:16.178931+0000","last_undegraded":"2024-03-28T18:56:16.178931+0000","last_fullsized":"2024-03-28T18:56:16.178931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:21.372889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989874+0000","last_change":"2024-03-28T18:55:34.495109+0000","last_active":"2024-03-28T18:56:15.989874+0000","last_peered":"2024-03-28T18:56:15.989874+0000","last_clean":"2024-03-28T18:56:15.989874+0000","last_became_active":"2024-03-28T18:55:34.494804+0000","last_became_peered":"2024-03-28T18:55:34.494804+0000","last_unstale":"2024-03-28T18:56:15.989874+0000","last_undegraded":"2024-03-28T18:56:15.989874+0000","last_fullsized":"2024-03-28T18:56:15.989874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993563+0000","last_change":"2024-03-28T18:55:45.603783+0000","last_active":"2024-03-28T18:56:15.993563+0000","last_peered":"2024-03-28T18:56:15.993563+0000","last_clean":"2024-03-28T18:56:15.993563+0000","last_became_active":"2024-03-28T18:55:45.603474+0000","last_became_peered":"2024-03-28T18:55:45.603474+0000","last_unstale":"2024-03-28T18:56:15.993563+0000","last_undegraded":"2024-03-28T18:56:15.993563+0000","last_fullsized":"2024-03-28T18:56:15.993563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:29:42.328182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.182051+0000","last_change":"2024-03-28T18:56:16.181039+0000","last_active":"2024-03-28T18:56:16.182051+0000","last_peered":"2024-03-28T18:56:16.182051+0000","last_clean":"2024-03-28T18:56:16.182051+0000","last_became_active":"2024-03-28T18:56:16.180227+0000","last_became_peered":"2024-03-28T18:56:16.180227+0000","last_unstale":"2024-03-28T18:56:16.182051+0000","last_undegraded":"2024-03-28T18:56:16.182051+0000","last_fullsized":"2024-03-28T18:56:16.182051+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:11:54.706529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989997+0000","last_change":"2024-03-28T18:55:45.618912+0000","last_active":"2024-03-28T18:56:15.989997+0000","last_peered":"2024-03-28T18:56:15.989997+0000","last_clean":"2024-03-28T18:56:15.989997+0000","last_became_active":"2024-03-28T18:55:45.618554+0000","last_became_peered":"2024-03-28T18:55:45.618554+0000","last_unstale":"2024-03-28T18:56:15.989997+0000","last_undegraded":"2024-03-28T18:56:15.989997+0000","last_fullsized":"2024-03-28T18:56:15.989997+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.373395+0000","last_change":"2024-03-28T18:55:34.479694+0000","last_active":"2024-03-28T18:56:16.373395+0000","last_peered":"2024-03-28T18:56:16.373395+0000","last_clean":"2024-03-28T18:56:16.373395+0000","last_became_active":"2024-03-28T18:55:34.479400+0000","last_became_peered":"2024-03-28T18:55:34.479400+0000","last_unstale":"2024-03-28T18:56:16.373395+0000","last_undegraded":"2024-03-28T18:56:16.373395+0000","last_fullsized":"2024-03-28T18:56:16.373395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:47:51.571095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164597+0000","last_change":"2024-03-28T18:55:45.616420+0000","last_active":"2024-03-28T18:56:16.164597+0000","last_peered":"2024-03-28T18:56:16.164597+0000","last_clean":"2024-03-28T18:56:16.164597+0000","last_became_active":"2024-03-28T18:55:45.614806+0000","last_became_peered":"2024-03-28T18:55:45.614806+0000","last_unstale":"2024-03-28T18:56:16.164597+0000","last_undegraded":"2024-03-28T18:56:16.164597+0000","last_fullsized":"2024-03-28T18:56:16.164597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:12:05.813829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993436+0000","last_change":"2024-03-28T18:55:34.484409+0000","last_active":"2024-03-28T18:56:15.993436+0000","last_peered":"2024-03-28T18:56:15.993436+0000","last_clean":"2024-03-28T18:56:15.993436+0000","last_became_active":"2024-03-28T18:55:34.482535+0000","last_became_peered":"2024-03-28T18:55:34.482535+0000","last_unstale":"2024-03-28T18:56:15.993436+0000","last_undegraded":"2024-03-28T18:56:15.993436+0000","last_fullsized":"2024-03-28T18:56:15.993436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:23.285280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.164798+0000","last_change":"2024-03-28T18:55:45.606507+0000","last_active":"2024-03-28T18:56:16.164798+0000","last_peered":"2024-03-28T18:56:16.164798+0000","last_clean":"2024-03-28T18:56:16.164798+0000","last_became_active":"2024-03-28T18:55:45.606205+0000","last_became_peered":"2024-03-28T18:55:45.606205+0000","last_unstale":"2024-03-28T18:56:16.164798+0000","last_undegraded":"2024-03-28T18:56:16.164798+0000","last_fullsized":"2024-03-28T18:56:16.164798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:38.677019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178591+0000","last_change":"2024-03-28T18:55:34.500243+0000","last_active":"2024-03-28T18:56:16.178591+0000","last_peered":"2024-03-28T18:56:16.178591+0000","last_clean":"2024-03-28T18:56:16.178591+0000","last_became_active":"2024-03-28T18:55:34.499940+0000","last_became_peered":"2024-03-28T18:55:34.499940+0000","last_unstale":"2024-03-28T18:56:16.178591+0000","last_undegraded":"2024-03-28T18:56:16.178591+0000","last_fullsized":"2024-03-28T18:56:16.178591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:10.119980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994460+0000","last_change":"2024-03-28T18:55:45.607890+0000","last_active":"2024-03-28T18:56:15.994460+0000","last_peered":"2024-03-28T18:56:15.994460+0000","last_clean":"2024-03-28T18:56:15.994460+0000","last_became_active":"2024-03-28T18:55:45.607590+0000","last_became_peered":"2024-03-28T18:55:45.607590+0000","last_unstale":"2024-03-28T18:56:15.994460+0000","last_undegraded":"2024-03-28T18:56:15.994460+0000","last_fullsized":"2024-03-28T18:56:15.994460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:36.370320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993121+0000","last_change":"2024-03-28T18:55:34.486870+0000","last_active":"2024-03-28T18:56:15.993121+0000","last_peered":"2024-03-28T18:56:15.993121+0000","last_clean":"2024-03-28T18:56:15.993121+0000","last_became_active":"2024-03-28T18:55:34.486530+0000","last_became_peered":"2024-03-28T18:55:34.486530+0000","last_unstale":"2024-03-28T18:56:15.993121+0000","last_undegraded":"2024-03-28T18:56:15.993121+0000","last_fullsized":"2024-03-28T18:56:15.993121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:30:17.708174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372260+0000","last_change":"2024-03-28T18:55:45.608851+0000","last_active":"2024-03-28T18:56:16.372260+0000","last_peered":"2024-03-28T18:56:16.372260+0000","last_clean":"2024-03-28T18:56:16.372260+0000","last_became_active":"2024-03-28T18:55:45.608496+0000","last_became_peered":"2024-03-28T18:55:45.608496+0000","last_unstale":"2024-03-28T18:56:16.372260+0000","last_undegraded":"2024-03-28T18:56:16.372260+0000","last_fullsized":"2024-03-28T18:56:16.372260+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:53:28.451302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.997752+0000","last_change":"2024-03-28T18:55:34.512063+0000","last_active":"2024-03-28T18:56:15.997752+0000","last_peered":"2024-03-28T18:56:15.997752+0000","last_clean":"2024-03-28T18:56:15.997752+0000","last_became_active":"2024-03-28T18:55:34.509984+0000","last_became_peered":"2024-03-28T18:55:34.509984+0000","last_unstale":"2024-03-28T18:56:15.997752+0000","last_undegraded":"2024-03-28T18:56:15.997752+0000","last_fullsized":"2024-03-28T18:56:15.997752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:04:25.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996035+0000","last_change":"2024-03-28T18:55:45.619501+0000","last_active":"2024-03-28T18:56:15.996035+0000","last_peered":"2024-03-28T18:56:15.996035+0000","last_clean":"2024-03-28T18:56:15.996035+0000","last_became_active":"2024-03-28T18:55:45.619212+0000","last_became_peered":"2024-03-28T18:55:45.619212+0000","last_unstale":"2024-03-28T18:56:15.996035+0000","last_undegraded":"2024-03-28T18:56:15.996035+0000","last_fullsized":"2024-03-28T18:56:15.996035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:20.561185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.377790+0000","last_change":"2024-03-28T18:56:16.375984+0000","last_active":"2024-03-28T18:56:16.377790+0000","last_peered":"2024-03-28T18:56:16.377790+0000","last_clean":"2024-03-28T18:56:16.377790+0000","last_became_active":"2024-03-28T18:56:16.375290+0000","last_became_peered":"2024-03-28T18:56:16.375290+0000","last_unstale":"2024-03-28T18:56:16.377790+0000","last_undegraded":"2024-03-28T18:56:16.377790+0000","last_fullsized":"2024-03-28T18:56:16.377790+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:12.576993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166458+0000","last_change":"2024-03-28T18:55:45.615225+0000","last_active":"2024-03-28T18:56:16.166458+0000","last_peered":"2024-03-28T18:56:16.166458+0000","last_clean":"2024-03-28T18:56:16.166458+0000","last_became_active":"2024-03-28T18:55:45.613602+0000","last_became_peered":"2024-03-28T18:55:45.613602+0000","last_unstale":"2024-03-28T18:56:16.166458+0000","last_undegraded":"2024-03-28T18:56:16.166458+0000","last_fullsized":"2024-03-28T18:56:16.166458+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994526+0000","last_change":"2024-03-28T18:55:34.502060+0000","last_active":"2024-03-28T18:56:15.994526+0000","last_peered":"2024-03-28T18:56:15.994526+0000","last_clean":"2024-03-28T18:56:15.994526+0000","last_became_active":"2024-03-28T18:55:34.501682+0000","last_became_peered":"2024-03-28T18:55:34.501682+0000","last_unstale":"2024-03-28T18:56:15.994526+0000","last_undegraded":"2024-03-28T18:56:15.994526+0000","last_fullsized":"2024-03-28T18:56:15.994526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995808+0000","last_change":"2024-03-28T18:55:45.609471+0000","last_active":"2024-03-28T18:56:15.995808+0000","last_peered":"2024-03-28T18:56:15.995808+0000","last_clean":"2024-03-28T18:56:15.995808+0000","last_became_active":"2024-03-28T18:55:45.609132+0000","last_became_peered":"2024-03-28T18:55:45.609132+0000","last_unstale":"2024-03-28T18:56:15.995808+0000","last_undegraded":"2024-03-28T18:56:15.995808+0000","last_fullsized":"2024-03-28T18:56:15.995808+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:23.598268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372643+0000","last_change":"2024-03-28T18:55:34.490403+0000","last_active":"2024-03-28T18:56:16.372643+0000","last_peered":"2024-03-28T18:56:16.372643+0000","last_clean":"2024-03-28T18:56:16.372643+0000","last_became_active":"2024-03-28T18:55:34.490085+0000","last_became_peered":"2024-03-28T18:55:34.490085+0000","last_unstale":"2024-03-28T18:56:16.372643+0000","last_undegraded":"2024-03-28T18:56:16.372643+0000","last_fullsized":"2024-03-28T18:56:16.372643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:47.639380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178130+0000","last_change":"2024-03-28T18:55:45.617501+0000","last_active":"2024-03-28T18:56:16.178130+0000","last_peered":"2024-03-28T18:56:16.178130+0000","last_clean":"2024-03-28T18:56:16.178130+0000","last_became_active":"2024-03-28T18:55:45.617180+0000","last_became_peered":"2024-03-28T18:55:45.617180+0000","last_unstale":"2024-03-28T18:56:16.178130+0000","last_undegraded":"2024-03-28T18:56:16.178130+0000","last_fullsized":"2024-03-28T18:56:16.178130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990476+0000","last_change":"2024-03-28T18:55:34.488539+0000","last_active":"2024-03-28T18:56:15.990476+0000","last_peered":"2024-03-28T18:56:15.990476+0000","last_clean":"2024-03-28T18:56:15.990476+0000","last_became_active":"2024-03-28T18:55:34.488242+0000","last_became_peered":"2024-03-28T18:55:34.488242+0000","last_unstale":"2024-03-28T18:56:15.990476+0000","last_undegraded":"2024-03-28T18:56:15.990476+0000","last_fullsized":"2024-03-28T18:56:15.990476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:01:01.875060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990313+0000","last_change":"2024-03-28T18:55:45.609169+0000","last_active":"2024-03-28T18:56:15.990313+0000","last_peered":"2024-03-28T18:56:15.990313+0000","last_clean":"2024-03-28T18:56:15.990313+0000","last_became_active":"2024-03-28T18:55:45.601948+0000","last_became_peered":"2024-03-28T18:55:45.601948+0000","last_unstale":"2024-03-28T18:56:15.990313+0000","last_undegraded":"2024-03-28T18:56:15.990313+0000","last_fullsized":"2024-03-28T18:56:15.990313+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.166023+0000","last_change":"2024-03-28T18:55:34.478884+0000","last_active":"2024-03-28T18:56:16.166023+0000","last_peered":"2024-03-28T18:56:16.166023+0000","last_clean":"2024-03-28T18:56:16.166023+0000","last_became_active":"2024-03-28T18:55:34.478546+0000","last_became_peered":"2024-03-28T18:55:34.478546+0000","last_unstale":"2024-03-28T18:56:16.166023+0000","last_undegraded":"2024-03-28T18:56:16.166023+0000","last_fullsized":"2024-03-28T18:56:16.166023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:13:34.438381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993150+0000","last_change":"2024-03-28T18:55:45.616238+0000","last_active":"2024-03-28T18:56:15.993150+0000","last_peered":"2024-03-28T18:56:15.993150+0000","last_clean":"2024-03-28T18:56:15.993150+0000","last_became_active":"2024-03-28T18:55:45.615371+0000","last_became_peered":"2024-03-28T18:55:45.615371+0000","last_unstale":"2024-03-28T18:56:15.993150+0000","last_undegraded":"2024-03-28T18:56:15.993150+0000","last_fullsized":"2024-03-28T18:56:15.993150+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:14.661134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993580+0000","last_change":"2024-03-28T18:55:34.495296+0000","last_active":"2024-03-28T18:56:15.993580+0000","last_peered":"2024-03-28T18:56:15.993580+0000","last_clean":"2024-03-28T18:56:15.993580+0000","last_became_active":"2024-03-28T18:55:34.495004+0000","last_became_peered":"2024-03-28T18:55:34.495004+0000","last_unstale":"2024-03-28T18:56:15.993580+0000","last_undegraded":"2024-03-28T18:56:15.993580+0000","last_fullsized":"2024-03-28T18:56:15.993580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:01:12.325544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177663+0000","last_change":"2024-03-28T18:55:45.613289+0000","last_active":"2024-03-28T18:56:16.177663+0000","last_peered":"2024-03-28T18:56:16.177663+0000","last_clean":"2024-03-28T18:56:16.177663+0000","last_became_active":"2024-03-28T18:55:45.610917+0000","last_became_peered":"2024-03-28T18:55:45.610917+0000","last_unstale":"2024-03-28T18:56:16.177663+0000","last_undegraded":"2024-03-28T18:56:16.177663+0000","last_fullsized":"2024-03-28T18:56:16.177663+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:39:55.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993822+0000","last_change":"2024-03-28T18:55:34.498566+0000","last_active":"2024-03-28T18:56:15.993822+0000","last_peered":"2024-03-28T18:56:15.993822+0000","last_clean":"2024-03-28T18:56:15.993822+0000","last_became_active":"2024-03-28T18:55:34.497698+0000","last_became_peered":"2024-03-28T18:55:34.497698+0000","last_unstale":"2024-03-28T18:56:15.993822+0000","last_undegraded":"2024-03-28T18:56:15.993822+0000","last_fullsized":"2024-03-28T18:56:15.993822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:05.458032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994722+0000","last_change":"2024-03-28T18:55:45.621789+0000","last_active":"2024-03-28T18:56:15.994722+0000","last_peered":"2024-03-28T18:56:15.994722+0000","last_clean":"2024-03-28T18:56:15.994722+0000","last_became_active":"2024-03-28T18:55:45.621163+0000","last_became_peered":"2024-03-28T18:55:45.621163+0000","last_unstale":"2024-03-28T18:56:15.994722+0000","last_undegraded":"2024-03-28T18:56:15.994722+0000","last_fullsized":"2024-03-28T18:56:15.994722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:23:32.778468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179217+0000","last_change":"2024-03-28T18:55:34.504774+0000","last_active":"2024-03-28T18:56:16.179217+0000","last_peered":"2024-03-28T18:56:16.179217+0000","last_clean":"2024-03-28T18:56:16.179217+0000","last_became_active":"2024-03-28T18:55:34.504511+0000","last_became_peered":"2024-03-28T18:55:34.504511+0000","last_unstale":"2024-03-28T18:56:16.179217+0000","last_undegraded":"2024-03-28T18:56:16.179217+0000","last_fullsized":"2024-03-28T18:56:16.179217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:25:04.807219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.991623+0000","last_change":"2024-03-28T18:55:45.603774+0000","last_active":"2024-03-28T18:56:15.991623+0000","last_peered":"2024-03-28T18:56:15.991623+0000","last_clean":"2024-03-28T18:56:15.991623+0000","last_became_active":"2024-03-28T18:55:45.603451+0000","last_became_peered":"2024-03-28T18:55:45.603451+0000","last_unstale":"2024-03-28T18:56:15.991623+0000","last_undegraded":"2024-03-28T18:56:15.991623+0000","last_fullsized":"2024-03-28T18:56:15.991623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:58.046569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.989714+0000","last_change":"2024-03-28T18:55:34.497824+0000","last_active":"2024-03-28T18:56:15.989714+0000","last_peered":"2024-03-28T18:56:15.989714+0000","last_clean":"2024-03-28T18:56:15.989714+0000","last_became_active":"2024-03-28T18:55:34.497492+0000","last_became_peered":"2024-03-28T18:55:34.497492+0000","last_unstale":"2024-03-28T18:56:15.989714+0000","last_undegraded":"2024-03-28T18:56:15.989714+0000","last_fullsized":"2024-03-28T18:56:15.989714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.994162+0000","last_change":"2024-03-28T18:55:45.616755+0000","last_active":"2024-03-28T18:56:15.994162+0000","last_peered":"2024-03-28T18:56:15.994162+0000","last_clean":"2024-03-28T18:56:15.994162+0000","last_became_active":"2024-03-28T18:55:45.614611+0000","last_became_peered":"2024-03-28T18:55:45.614611+0000","last_unstale":"2024-03-28T18:56:15.994162+0000","last_undegraded":"2024-03-28T18:56:15.994162+0000","last_fullsized":"2024-03-28T18:56:15.994162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:24:06.327795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.377155+0000","last_change":"2024-03-28T18:56:16.375743+0000","last_active":"2024-03-28T18:56:16.377155+0000","last_peered":"2024-03-28T18:56:16.377155+0000","last_clean":"2024-03-28T18:56:16.377155+0000","last_became_active":"2024-03-28T18:56:16.375316+0000","last_became_peered":"2024-03-28T18:56:16.375316+0000","last_unstale":"2024-03-28T18:56:16.377155+0000","last_undegraded":"2024-03-28T18:56:16.377155+0000","last_fullsized":"2024-03-28T18:56:16.377155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:27.569718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995729+0000","last_change":"2024-03-28T18:55:45.623195+0000","last_active":"2024-03-28T18:56:15.995729+0000","last_peered":"2024-03-28T18:56:15.995729+0000","last_clean":"2024-03-28T18:56:15.995729+0000","last_became_active":"2024-03-28T18:55:45.622508+0000","last_became_peered":"2024-03-28T18:55:45.622508+0000","last_unstale":"2024-03-28T18:56:15.995729+0000","last_undegraded":"2024-03-28T18:56:15.995729+0000","last_fullsized":"2024-03-28T18:56:15.995729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:57:52.870118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.995605+0000","last_change":"2024-03-28T18:55:34.499809+0000","last_active":"2024-03-28T18:56:15.995605+0000","last_peered":"2024-03-28T18:56:15.995605+0000","last_clean":"2024-03-28T18:56:15.995605+0000","last_became_active":"2024-03-28T18:55:34.499588+0000","last_became_peered":"2024-03-28T18:55:34.499588+0000","last_unstale":"2024-03-28T18:56:15.995605+0000","last_undegraded":"2024-03-28T18:56:15.995605+0000","last_fullsized":"2024-03-28T18:56:15.995605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:05:41.378665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996676+0000","last_change":"2024-03-28T18:55:45.618841+0000","last_active":"2024-03-28T18:56:15.996676+0000","last_peered":"2024-03-28T18:56:15.996676+0000","last_clean":"2024-03-28T18:56:15.996676+0000","last_became_active":"2024-03-28T18:55:45.618579+0000","last_became_peered":"2024-03-28T18:55:45.618579+0000","last_unstale":"2024-03-28T18:56:15.996676+0000","last_undegraded":"2024-03-28T18:56:15.996676+0000","last_fullsized":"2024-03-28T18:56:15.996676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:54:54.231312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.177448+0000","last_change":"2024-03-28T18:55:34.495097+0000","last_active":"2024-03-28T18:56:16.177448+0000","last_peered":"2024-03-28T18:56:16.177448+0000","last_clean":"2024-03-28T18:56:16.177448+0000","last_became_active":"2024-03-28T18:55:34.494758+0000","last_became_peered":"2024-03-28T18:55:34.494758+0000","last_unstale":"2024-03-28T18:56:16.177448+0000","last_undegraded":"2024-03-28T18:56:16.177448+0000","last_fullsized":"2024-03-28T18:56:16.177448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:31:03.553833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.990122+0000","last_change":"2024-03-28T18:55:45.602886+0000","last_active":"2024-03-28T18:56:15.990122+0000","last_peered":"2024-03-28T18:56:15.990122+0000","last_clean":"2024-03-28T18:56:15.990122+0000","last_became_active":"2024-03-28T18:55:45.602610+0000","last_became_peered":"2024-03-28T18:55:45.602610+0000","last_unstale":"2024-03-28T18:56:15.990122+0000","last_undegraded":"2024-03-28T18:56:15.990122+0000","last_fullsized":"2024-03-28T18:56:15.990122+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:59:02.886001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996124+0000","last_change":"2024-03-28T18:55:34.506578+0000","last_active":"2024-03-28T18:56:15.996124+0000","last_peered":"2024-03-28T18:56:15.996124+0000","last_clean":"2024-03-28T18:56:15.996124+0000","last_became_active":"2024-03-28T18:55:34.505868+0000","last_became_peered":"2024-03-28T18:55:34.505868+0000","last_unstale":"2024-03-28T18:56:15.996124+0000","last_undegraded":"2024-03-28T18:56:15.996124+0000","last_fullsized":"2024-03-28T18:56:15.996124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:04:26.531503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.993035+0000","last_change":"2024-03-28T18:55:45.608673+0000","last_active":"2024-03-28T18:56:15.993035+0000","last_peered":"2024-03-28T18:56:15.993035+0000","last_clean":"2024-03-28T18:56:15.993035+0000","last_became_active":"2024-03-28T18:55:45.608376+0000","last_became_peered":"2024-03-28T18:55:45.608376+0000","last_unstale":"2024-03-28T18:56:15.993035+0000","last_undegraded":"2024-03-28T18:56:15.993035+0000","last_fullsized":"2024-03-28T18:56:15.993035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:00:31.781549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.372875+0000","last_change":"2024-03-28T18:55:34.489286+0000","last_active":"2024-03-28T18:56:16.372875+0000","last_peered":"2024-03-28T18:56:16.372875+0000","last_clean":"2024-03-28T18:56:16.372875+0000","last_became_active":"2024-03-28T18:55:34.488967+0000","last_became_peered":"2024-03-28T18:55:34.488967+0000","last_unstale":"2024-03-28T18:56:16.372875+0000","last_undegraded":"2024-03-28T18:56:16.372875+0000","last_fullsized":"2024-03-28T18:56:16.372875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:40:03.398986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.176679+0000","last_change":"2024-03-28T18:55:45.609565+0000","last_active":"2024-03-28T18:56:16.176679+0000","last_peered":"2024-03-28T18:56:16.176679+0000","last_clean":"2024-03-28T18:56:16.176679+0000","last_became_active":"2024-03-28T18:55:45.609082+0000","last_became_peered":"2024-03-28T18:55:45.609082+0000","last_unstale":"2024-03-28T18:56:16.176679+0000","last_undegraded":"2024-03-28T18:56:16.176679+0000","last_fullsized":"2024-03-28T18:56:16.176679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:08:31.648108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:15.996230+0000","last_change":"2024-03-28T18:55:34.491627+0000","last_active":"2024-03-28T18:56:15.996230+0000","last_peered":"2024-03-28T18:56:15.996230+0000","last_clean":"2024-03-28T18:56:15.996230+0000","last_became_active":"2024-03-28T18:55:34.490463+0000","last_became_peered":"2024-03-28T18:55:34.490463+0000","last_unstale":"2024-03-28T18:56:15.996230+0000","last_undegraded":"2024-03-28T18:56:15.996230+0000","last_fullsized":"2024-03-28T18:56:15.996230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:56:38.168512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.178310+0000","last_change":"2024-03-28T18:55:45.618585+0000","last_active":"2024-03-28T18:56:16.178310+0000","last_peered":"2024-03-28T18:56:16.178310+0000","last_clean":"2024-03-28T18:56:16.178310+0000","last_became_active":"2024-03-28T18:55:45.617895+0000","last_became_peered":"2024-03-28T18:55:45.617895+0000","last_unstale":"2024-03-28T18:56:16.178310+0000","last_undegraded":"2024-03-28T18:56:16.178310+0000","last_fullsized":"2024-03-28T18:56:16.178310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:44.553581+0000","last_clean_scrub_stamp":"2024-03-28T18:55:44.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:20:54.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-28T18:56:16.179210+0000","last_change":"2024-03-28T18:55:34.503561+0000","last_active":"2024-03-28T18:56:16.179210+0000","last_peered":"2024-03-28T18:56:16.179210+0000","last_clean":"2024-03-28T18:56:16.179210+0000","last_became_active":"2024-03-28T18:55:34.503127+0000","last_became_peered":"2024-03-28T18:55:34.503127+0000","last_unstale":"2024-03-28T18:56:16.179210+0000","last_undegraded":"2024-03-28T18:56:16.179210+0000","last_fullsized":"2024-03-28T18:56:16.179210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:55:33.446280+0000","last_clean_scrub_stamp":"2024-03-28T18:55:33.446280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:11:49.494738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":880640,"data_stored":704107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":299008,"data_stored":116551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":113829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":299008,"data_stored":113297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":118825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":113137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":116117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":884736,"data_stored":704073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:56:31.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:31.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.683+0000 7f1e51400640 1 Processor -- start 2024-03-28T18:56:31.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.683+0000 7f1e51400640 1 -- start start 2024-03-28T18:56:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.684+0000 7f1e51400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c150ae0 0x7f1e4c0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.684+0000 7f1e51400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e4c0a61a0 0x7f1e4c0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.684+0000 7f1e51400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e4c150ec0 con 0x7f1e4c0c6ee0 2024-03-28T18:56:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.684+0000 7f1e51400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4c06dc70 con 0x7f1e4c0a61a0 2024-03-28T18:56:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.684+0000 7f1e51400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c153550 con 0x7f1e4c150ae0 2024-03-28T18:56:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.685+0000 7f1e4b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e4c0a61a0 0x7f1e4c0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.685+0000 7f1e4b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e4c0a61a0 0x7f1e4c0c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40450/0 (socket says 172.21.15.27:40450) 2024-03-28T18:56:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.685+0000 7f1e4b400640 1 -- 172.21.15.27:0/2799534939 learned_addr learned my addr 172.21.15.27:0/2799534939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.685+0000 7f1e4be00640 1 --2- 172.21.15.27:0/2799534939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c150ae0 0x7f1e4c0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.685+0000 7f1e4b400640 1 -- 172.21.15.27:0/2799534939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c150ae0 msgr2=0x7f1e4c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.685+0000 7f1e4b400640 1 --2- 172.21.15.27:0/2799534939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c150ae0 0x7f1e4c0a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.685+0000 7f1e4b400640 1 -- 172.21.15.27:0/2799534939 >> v1:172.21.15.155:6789/0 conn(0x7f1e4c0c6ee0 legacy=0x7f1e4c153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:31.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.685+0000 7f1e4b400640 1 -- 172.21.15.27:0/2799534939 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c06df10 con 0x7f1e4c0a61a0 2024-03-28T18:56:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.686+0000 7f1e4b400640 1 --2- 172.21.15.27:0/2799534939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c0a61a0 0x7f1e4c0c69a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009870 tx=0x7f1e3802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ee53e9a3d8ead server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e4aa00640 1 -- 172.21.15.27:0/2799534939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e380317c0 con 0x7f1e4c0a61a0 2024-03-28T18:56:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e4aa00640 1 -- 172.21.15.27:0/2799534939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e38037070 con 0x7f1e4c0a61a0 2024-03-28T18:56:31.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e4aa00640 1 -- 172.21.15.27:0/2799534939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38031b80 con 0x7f1e4c0a61a0 2024-03-28T18:56:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e51400640 1 -- 172.21.15.27:0/2799534939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c0a61a0 msgr2=0x7f1e4c0c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e51400640 1 --2- 172.21.15.27:0/2799534939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c0a61a0 0x7f1e4c0c69a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009870 tx=0x7f1e3802b860 comp rx=0 tx=0).stop 2024-03-28T18:56:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e51400640 1 -- 172.21.15.27:0/2799534939 shutdown_connections 2024-03-28T18:56:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e51400640 1 --2- 172.21.15.27:0/2799534939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c0a61a0 0x7f1e4c0c69a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e51400640 1 --2- 172.21.15.27:0/2799534939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c150ae0 0x7f1e4c0a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e51400640 1 -- 172.21.15.27:0/2799534939 >> 172.21.15.27:0/2799534939 conn(0x7f1e4c06b0c0 msgr2=0x7f1e4c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e51400640 1 -- 172.21.15.27:0/2799534939 shutdown_connections 2024-03-28T18:56:31.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.687+0000 7f1e51400640 1 -- 172.21.15.27:0/2799534939 wait complete. 2024-03-28T18:56:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.689+0000 7f1e51400640 1 Processor -- start 2024-03-28T18:56:31.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.689+0000 7f1e51400640 1 -- start start 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.690+0000 7f1e51400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e4c0a61a0 0x7f1e4c1e2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.690+0000 7f1e51400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c0c6ee0 0x7f1e4c1ee440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.690+0000 7f1e51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c150ae0 0x7f1e4c14cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.690+0000 7f1e51400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c06e7b0 con 0x7f1e4c150ae0 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.690+0000 7f1e51400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c06e4d0 con 0x7f1e4c0a61a0 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.690+0000 7f1e51400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c1569d0 con 0x7f1e4c0c6ee0 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c150ae0 0x7f1e4c14cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c150ae0 0x7f1e4c14cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40458/0 (socket says 172.21.15.27:40458) 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 -- 172.21.15.27:0/3217718886 learned_addr learned my addr 172.21.15.27:0/3217718886 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e4be00640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e4c0a61a0 0x7f1e4c1e2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e4b400640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c0c6ee0 0x7f1e4c1ee440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 -- 172.21.15.27:0/3217718886 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c0c6ee0 msgr2=0x7f1e4c1ee440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:31.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c0c6ee0 0x7f1e4c1ee440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 -- 172.21.15.27:0/3217718886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e4c0a61a0 msgr2=0x7f1e4c1e2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e4c0a61a0 0x7f1e4c1e2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 -- 172.21.15.27:0/3217718886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c1dcae0 con 0x7f1e4c150ae0 2024-03-28T18:56:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.691+0000 7f1e50a00640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c150ae0 0x7f1e4c14cd80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400c7b0 tx=0x7f1e4400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.692+0000 7f1e43e00640 1 -- 172.21.15.27:0/3217718886 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e44004b60 con 0x7f1e4c150ae0 2024-03-28T18:56:31.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.692+0000 7f1e43e00640 1 -- 172.21.15.27:0/3217718886 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e44004d00 con 0x7f1e4c150ae0 2024-03-28T18:56:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.692+0000 7f1e43e00640 1 -- 172.21.15.27:0/3217718886 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4400aa90 con 0x7f1e4c150ae0 2024-03-28T18:56:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.692+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4c14db70 con 0x7f1e4c150ae0 2024-03-28T18:56:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.692+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4c1dea00 con 0x7f1e4c150ae0 2024-03-28T18:56:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.693+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4c067dd0 con 0x7f1e4c150ae0 2024-03-28T18:56:31.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.694+0000 7f1e43e00640 1 -- 172.21.15.27:0/3217718886 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1e4400ac30 con 0x7f1e4c150ae0 2024-03-28T18:56:31.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.695+0000 7f1e43e00640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:31.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.696+0000 7f1e43e00640 1 -- 172.21.15.27:0/3217718886 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f1e4408b0a0 con 0x7f1e4c150ae0 2024-03-28T18:56:31.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.696+0000 7f1e4be00640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:31.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.697+0000 7f1e4be00640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c006fd0 tx=0x7f1e3c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:31.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:31.700+0000 7f1e43e00640 1 -- 172.21.15.27:0/3217718886 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1e44020070 con 0x7f1e4c150ae0 2024-03-28T18:56:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.040+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e4c069640 con 0x7f1e4c150ae0 2024-03-28T18:56:32.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.044+0000 7f1e43e00640 1 -- 172.21.15.27:0/3217718886 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+20501 (secure 0 0 0) 0x7f1e440597e0 con 0x7f1e4c150ae0 2024-03-28T18:56:32.044 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:32.044 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:56:15.976621+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 msgr2=0x7f1e2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c006fd0 tx=0x7f1e3c008040 comp rx=0 tx=0).stop 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c150ae0 msgr2=0x7f1e4c14cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c150ae0 0x7f1e4c14cd80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400c7b0 tx=0x7f1e4400cc70 comp rx=0 tx=0).stop 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 shutdown_connections 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e4c150ae0 0x7f1e4c14cd80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e4c0c6ee0 0x7f1e4c1ee440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 --2- 172.21.15.27:0/3217718886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e4c0a61a0 0x7f1e4c1e2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 >> 172.21.15.27:0/3217718886 conn(0x7f1e4c06b0c0 msgr2=0x7f1e4c0c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 shutdown_connections 2024-03-28T18:56:32.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.048+0000 7f1e51400640 1 -- 172.21.15.27:0/3217718886 wait complete. 2024-03-28T18:56:32.068 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-03-28T18:55:32.332283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-03-28T18:55:32.983411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-03-28T18:55:43.415397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-03-28T18:55:44.084162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:56:32.069 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:56:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.318+0000 7f37a4a00640 1 Processor -- start 2024-03-28T18:56:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.318+0000 7f37a4a00640 1 -- start start 2024-03-28T18:56:32.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.319+0000 7f37a4a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f37a00a82f0 0x7f37a00c5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.319+0000 7f37a4a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00c8490 0x7f37a01523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.319+0000 7f37a4a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a006b0c0 con 0x7f37a00c58b0 2024-03-28T18:56:32.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.319+0000 7f37a4a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a006b680 con 0x7f37a00a82f0 2024-03-28T18:56:32.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.319+0000 7f37a4a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a01561b0 con 0x7f37a00c8490 2024-03-28T18:56:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.320+0000 7f3797e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00c8490 0x7f37a01523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.320+0000 7f3797e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00c8490 0x7f37a01523f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45484/0 (socket says 172.21.15.27:45484) 2024-03-28T18:56:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.320+0000 7f3797e00640 1 -- 172.21.15.27:0/3580665986 learned_addr learned my addr 172.21.15.27:0/3580665986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.320+0000 7f379f400640 1 --2- 172.21.15.27:0/3580665986 >> v2:172.21.15.155:3300/0 conn(0x7f37a00a82f0 0x7f37a00c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.321+0000 7f3797e00640 1 -- 172.21.15.27:0/3580665986 >> v2:172.21.15.155:3300/0 conn(0x7f37a00a82f0 msgr2=0x7f37a00c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:32.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.321+0000 7f3797e00640 1 --2- 172.21.15.27:0/3580665986 >> v2:172.21.15.155:3300/0 conn(0x7f37a00a82f0 0x7f37a00c5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.321+0000 7f3797e00640 1 -- 172.21.15.27:0/3580665986 >> v1:172.21.15.155:6789/0 conn(0x7f37a00c58b0 legacy=0x7f37a0152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.321+0000 7f3797e00640 1 -- 172.21.15.27:0/3580665986 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a006b3a0 con 0x7f37a00c8490 2024-03-28T18:56:32.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.322+0000 7f3797e00640 1 --2- 172.21.15.27:0/3580665986 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00c8490 0x7f37a01523f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3788009870 tx=0x7f378802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26ba90cd2a076d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:32.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.323+0000 7f379ea00640 1 -- 172.21.15.27:0/3580665986 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3788035020 con 0x7f37a00c8490 2024-03-28T18:56:32.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.323+0000 7f379ea00640 1 -- 172.21.15.27:0/3580665986 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3788031640 con 0x7f37a00c8490 2024-03-28T18:56:32.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.323+0000 7f379ea00640 1 -- 172.21.15.27:0/3580665986 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378802d040 con 0x7f37a00c8490 2024-03-28T18:56:32.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.323+0000 7f37a4a00640 1 -- 172.21.15.27:0/3580665986 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00c8490 msgr2=0x7f37a01523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.323+0000 7f37a4a00640 1 --2- 172.21.15.27:0/3580665986 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00c8490 0x7f37a01523f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3788009870 tx=0x7f378802b8f0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.324+0000 7f37a4a00640 1 -- 172.21.15.27:0/3580665986 shutdown_connections 2024-03-28T18:56:32.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.324+0000 7f37a4a00640 1 --2- 172.21.15.27:0/3580665986 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00c8490 0x7f37a01523f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.324+0000 7f37a4a00640 1 --2- 172.21.15.27:0/3580665986 >> v2:172.21.15.155:3300/0 conn(0x7f37a00a82f0 0x7f37a00c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.324+0000 7f37a4a00640 1 -- 172.21.15.27:0/3580665986 >> 172.21.15.27:0/3580665986 conn(0x7f37a00aa7c0 msgr2=0x7f37a00aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.324+0000 7f37a4a00640 1 -- 172.21.15.27:0/3580665986 shutdown_connections 2024-03-28T18:56:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.324+0000 7f37a4a00640 1 -- 172.21.15.27:0/3580665986 wait complete. 2024-03-28T18:56:32.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.326+0000 7f37a4a00640 1 Processor -- start 2024-03-28T18:56:32.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.327+0000 7f37a4a00640 1 -- start start 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.328+0000 7f37a4a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00a82f0 0x7f37a0152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.328+0000 7f37a4a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37a00c58b0 0x7f37a00c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.328+0000 7f37a4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8490 0x7f37a01eb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.328+0000 7f37a4a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a00c37a0 con 0x7f37a00c8490 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.328+0000 7f37a4a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a006cde0 con 0x7f37a00c58b0 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.328+0000 7f37a4a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a00c3920 con 0x7f37a00a82f0 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8490 0x7f37a01eb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8490 0x7f37a01eb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40470/0 (socket says 172.21.15.27:40470) 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379fe00640 1 -- 172.21.15.27:0/999508537 learned_addr learned my addr 172.21.15.27:0/999508537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f3797e00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37a00c58b0 0x7f37a00c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379f400640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00a82f0 0x7f37a0152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379fe00640 1 -- 172.21.15.27:0/999508537 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00a82f0 msgr2=0x7f37a0152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379fe00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00a82f0 0x7f37a0152870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379fe00640 1 -- 172.21.15.27:0/999508537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37a00c58b0 msgr2=0x7f37a00c0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379fe00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37a00c58b0 0x7f37a00c0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.329+0000 7f379fe00640 1 -- 172.21.15.27:0/999508537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a00c2f60 con 0x7f37a00c8490 2024-03-28T18:56:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.330+0000 7f379fe00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8490 0x7f37a01eb160 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f379800c710 tx=0x7f379800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.330+0000 7f3797400640 1 -- 172.21.15.27:0/999508537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798016020 con 0x7f37a00c8490 2024-03-28T18:56:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.330+0000 7f3797400640 1 -- 172.21.15.27:0/999508537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3798006d50 con 0x7f37a00c8490 2024-03-28T18:56:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.330+0000 7f3797400640 1 -- 172.21.15.27:0/999508537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798003bb0 con 0x7f37a00c8490 2024-03-28T18:56:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.330+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a01585a0 con 0x7f37a00c8490 2024-03-28T18:56:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.330+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a0152db0 con 0x7f37a00c8490 2024-03-28T18:56:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.331+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a004f400 con 0x7f37a00c8490 2024-03-28T18:56:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.333+0000 7f3797400640 1 -- 172.21.15.27:0/999508537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f379800a7f0 con 0x7f37a00c8490 2024-03-28T18:56:32.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.335+0000 7f3797400640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f377c06bbd0 0x7f377c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.335+0000 7f3797400640 1 -- 172.21.15.27:0/999508537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f379808aab0 con 0x7f37a00c8490 2024-03-28T18:56:32.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.335+0000 7f379f400640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f377c06bbd0 0x7f377c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.336+0000 7f379f400640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f377c06bbd0 0x7f377c06e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f37a01551e0 tx=0x7f3790008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:32.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.341+0000 7f3797400640 1 -- 172.21.15.27:0/999508537 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3798020020 con 0x7f37a00c8490 2024-03-28T18:56:32.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.558+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37a00c8270 con 0x7f37a00c8490 2024-03-28T18:56:32.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.559+0000 7f3797400640 1 -- 172.21.15.27:0/999508537 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37980591f0 con 0x7f37a00c8490 2024-03-28T18:56:32.562 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:56:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f377c06bbd0 msgr2=0x7f377c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f377c06bbd0 0x7f377c06e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f37a01551e0 tx=0x7f3790008040 comp rx=0 tx=0).stop 2024-03-28T18:56:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8490 msgr2=0x7f37a01eb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8490 0x7f37a01eb160 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f379800c710 tx=0x7f379800cbd0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 shutdown_connections 2024-03-28T18:56:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f377c06bbd0 0x7f377c06e090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a00c8490 0x7f37a01eb160 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37a00c58b0 0x7f37a00c0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 --2- 172.21.15.27:0/999508537 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37a00a82f0 0x7f37a0152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 >> 172.21.15.27:0/999508537 conn(0x7f37a00aa7c0 msgr2=0x7f37a00c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:32.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 shutdown_connections 2024-03-28T18:56:32.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.562+0000 7f37a4a00640 1 -- 172.21.15.27:0/999508537 wait complete. 2024-03-28T18:56:32.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-03-28T18:56:32.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.799+0000 7fb63d800640 1 Processor -- start 2024-03-28T18:56:32.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.799+0000 7fb63d800640 1 -- start start 2024-03-28T18:56:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63d800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb6380c5aa0 0x7fb638156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63d800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb638072620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63d800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb638072b60 0x7fb638152840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63d800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6381565d0 con 0x7fb638072b60 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63d800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6380c82d0 con 0x7fb6380c5aa0 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63d800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380735f0 con 0x7fb6380c8db0 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb638072b60 0x7fb638152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb638072b60 0x7fb638152840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40472/0 (socket says 172.21.15.27:40472) 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.801+0000 7fb63ce00640 1 -- 172.21.15.27:0/74759675 learned_addr learned my addr 172.21.15.27:0/74759675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.802+0000 7fb62f400640 1 --2- 172.21.15.27:0/74759675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb638072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.802+0000 7fb637e00640 1 --2- 172.21.15.27:0/74759675 >> v2:172.21.15.155:3300/0 conn(0x7fb6380c5aa0 0x7fb638156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.802+0000 7fb63ce00640 1 -- 172.21.15.27:0/74759675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 msgr2=0x7fb638072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.802+0000 7fb63ce00640 1 --2- 172.21.15.27:0/74759675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb638072620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.802+0000 7fb63ce00640 1 -- 172.21.15.27:0/74759675 >> v2:172.21.15.155:3300/0 conn(0x7fb6380c5aa0 msgr2=0x7fb638156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.802+0000 7fb63ce00640 1 --2- 172.21.15.27:0/74759675 >> v2:172.21.15.155:3300/0 conn(0x7fb6380c5aa0 0x7fb638156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.802+0000 7fb63ce00640 1 -- 172.21.15.27:0/74759675 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6380c7d10 con 0x7fb638072b60 2024-03-28T18:56:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.802+0000 7fb62f400640 1 --2- 172.21.15.27:0/74759675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb638072620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.803+0000 7fb63ce00640 1 --2- 172.21.15.27:0/74759675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb638072b60 0x7fb638152840 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb63806bf20 tx=0x7fb63002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19902f2c2bd34e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.803+0000 7fb637400640 1 -- 172.21.15.27:0/74759675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63002c9f0 con 0x7fb638072b60 2024-03-28T18:56:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb637400640 1 -- 172.21.15.27:0/74759675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63002cb90 con 0x7fb638072b60 2024-03-28T18:56:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb637400640 1 -- 172.21.15.27:0/74759675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6300343f0 con 0x7fb638072b60 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 -- 172.21.15.27:0/74759675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb638072b60 msgr2=0x7fb638152840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 --2- 172.21.15.27:0/74759675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb638072b60 0x7fb638152840 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb63806bf20 tx=0x7fb63002be50 comp rx=0 tx=0).stop 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 -- 172.21.15.27:0/74759675 shutdown_connections 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 --2- 172.21.15.27:0/74759675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb638072b60 0x7fb638152840 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 --2- 172.21.15.27:0/74759675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb638072620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 --2- 172.21.15.27:0/74759675 >> v2:172.21.15.155:3300/0 conn(0x7fb6380c5aa0 0x7fb638156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 -- 172.21.15.27:0/74759675 >> 172.21.15.27:0/74759675 conn(0x7fb63806b0c0 msgr2=0x7fb63806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 -- 172.21.15.27:0/74759675 shutdown_connections 2024-03-28T18:56:32.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.804+0000 7fb63d800640 1 -- 172.21.15.27:0/74759675 wait complete. 2024-03-28T18:56:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.806+0000 7fb63d800640 1 Processor -- start 2024-03-28T18:56:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.806+0000 7fb63d800640 1 -- start start 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb63d800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb638072b60 0x7fb6380c1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb63d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6380c5aa0 0x7fb638158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb63d800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb6381eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb63d800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb638072100 con 0x7fb6380c5aa0 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb63d800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb638071e20 con 0x7fb638072b60 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb63d800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380c78c0 con 0x7fb6380c8db0 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb62f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6380c5aa0 0x7fb638158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb62f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6380c5aa0 0x7fb638158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40476/0 (socket says 172.21.15.27:40476) 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.807+0000 7fb62f400640 1 -- 172.21.15.27:0/1679201311 learned_addr learned my addr 172.21.15.27:0/1679201311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.808+0000 7fb637e00640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb638072b60 0x7fb6380c1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.808+0000 7fb63ce00640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb6381eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.808+0000 7fb62f400640 1 -- 172.21.15.27:0/1679201311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 msgr2=0x7fb6381eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.808+0000 7fb62f400640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb6381eb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.808+0000 7fb62f400640 1 -- 172.21.15.27:0/1679201311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb638072b60 msgr2=0x7fb6380c1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:32.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.808+0000 7fb62f400640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb638072b60 0x7fb6380c1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:32.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.808+0000 7fb62f400640 1 -- 172.21.15.27:0/1679201311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63814eaa0 con 0x7fb6380c5aa0 2024-03-28T18:56:32.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.808+0000 7fb62f400640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6380c5aa0 0x7fb638158f60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb62000d7d0 tx=0x7fb62000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:32.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.809+0000 7fb634c00640 1 -- 172.21.15.27:0/1679201311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb620004030 con 0x7fb6380c5aa0 2024-03-28T18:56:32.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.809+0000 7fb634c00640 1 -- 172.21.15.27:0/1679201311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6200041d0 con 0x7fb6380c5aa0 2024-03-28T18:56:32.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.809+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63814e280 con 0x7fb6380c5aa0 2024-03-28T18:56:32.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.809+0000 7fb634c00640 1 -- 172.21.15.27:0/1679201311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6200096a0 con 0x7fb6380c5aa0 2024-03-28T18:56:32.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.809+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63814eee0 con 0x7fb6380c5aa0 2024-03-28T18:56:32.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.810+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63814f6e0 con 0x7fb6380c5aa0 2024-03-28T18:56:32.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.811+0000 7fb634c00640 1 -- 172.21.15.27:0/1679201311 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb620009840 con 0x7fb6380c5aa0 2024-03-28T18:56:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.812+0000 7fb634c00640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb61406bc00 0x7fb61406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.812+0000 7fb634c00640 1 -- 172.21.15.27:0/1679201311 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fb62008b090 con 0x7fb6380c5aa0 2024-03-28T18:56:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.813+0000 7fb637e00640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb61406bc00 0x7fb61406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:32.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.813+0000 7fb637e00640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb61406bc00 0x7fb61406e0c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb628006fd0 tx=0x7fb628008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:32.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:32.817+0000 7fb634c00640 1 -- 172.21.15.27:0/1679201311 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb620059870 con 0x7fb6380c5aa0 2024-03-28T18:56:33.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.049+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb63814e7a0 con 0x7fb6380c5aa0 2024-03-28T18:56:33.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.049+0000 7fb634c00640 1 -- 172.21.15.27:0/1679201311 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v26) v1 ==== 99+0+11 (secure 0 0 0) 0x7fb63814e7a0 con 0x7fb6380c5aa0 2024-03-28T18:56:33.049 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:33.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb61406bc00 msgr2=0x7fb61406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb61406bc00 0x7fb61406e0c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb628006fd0 tx=0x7fb628008040 comp rx=0 tx=0).stop 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6380c5aa0 msgr2=0x7fb638158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6380c5aa0 0x7fb638158f60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb62000d7d0 tx=0x7fb62000dc90 comp rx=0 tx=0).stop 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 shutdown_connections 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb61406bc00 0x7fb61406e0c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6380c8db0 0x7fb6381eb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6380c5aa0 0x7fb638158f60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 --2- 172.21.15.27:0/1679201311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb638072b60 0x7fb6380c1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 >> 172.21.15.27:0/1679201311 conn(0x7fb63806b0c0 msgr2=0x7fb6380ca480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 shutdown_connections 2024-03-28T18:56:33.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.052+0000 7fb63d800640 1 -- 172.21.15.27:0/1679201311 wait complete. 2024-03-28T18:56:33.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-03-28T18:56:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.282+0000 7f71b1600640 1 Processor -- start 2024-03-28T18:56:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.283+0000 7f71b1600640 1 -- start start 2024-03-28T18:56:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.284+0000 7f71b1600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f71ac0c5aa0 0x7f71ac156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.284+0000 7f71b1600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71ac0c8db0 0x7f71ac072620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.285+0000 7f71b1600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac0735f0 con 0x7f71ac072b60 2024-03-28T18:56:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.285+0000 7f71b1600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac1565d0 con 0x7f71ac0c8db0 2024-03-28T18:56:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.285+0000 7f71b1600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac0c82d0 con 0x7f71ac0c5aa0 2024-03-28T18:56:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.285+0000 7f71ab400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71ac0c8db0 0x7f71ac072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.285+0000 7f71b0c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f71ac072b60 0x7f71ac152840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44794/0 (socket says 172.21.15.27:44794) 2024-03-28T18:56:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.285+0000 7f71b0c00640 1 -- 172.21.15.27:0/4276505403 learned_addr learned my addr 172.21.15.27:0/4276505403 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.285+0000 7f71abe00640 1 --2- 172.21.15.27:0/4276505403 >> v2:172.21.15.155:3300/0 conn(0x7f71ac0c5aa0 0x7f71ac156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.286+0000 7f71ab400640 1 -- 172.21.15.27:0/4276505403 >> v2:172.21.15.155:3300/0 conn(0x7f71ac0c5aa0 msgr2=0x7f71ac156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.286+0000 7f71ab400640 1 --2- 172.21.15.27:0/4276505403 >> v2:172.21.15.155:3300/0 conn(0x7f71ac0c5aa0 0x7f71ac156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.286+0000 7f71ab400640 1 -- 172.21.15.27:0/4276505403 >> v1:172.21.15.27:6789/0 conn(0x7f71ac072b60 legacy=0x7f71ac152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.286+0000 7f71ab400640 1 -- 172.21.15.27:0/4276505403 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac0c7d10 con 0x7f71ac0c8db0 2024-03-28T18:56:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.287+0000 7f71ab400640 1 --2- 172.21.15.27:0/4276505403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0c8db0 0x7f71ac072620 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f71ac06bf20 tx=0x7f719802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58af1bb13dcb3760 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.287+0000 7f71aaa00640 1 -- 172.21.15.27:0/4276505403 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71980315a0 con 0x7f71ac0c8db0 2024-03-28T18:56:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.287+0000 7f71aaa00640 1 -- 172.21.15.27:0/4276505403 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7198037070 con 0x7f71ac0c8db0 2024-03-28T18:56:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.287+0000 7f71aaa00640 1 -- 172.21.15.27:0/4276505403 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198031960 con 0x7f71ac0c8db0 2024-03-28T18:56:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.288+0000 7f71b1600640 1 -- 172.21.15.27:0/4276505403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0c8db0 msgr2=0x7f71ac072620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.288+0000 7f71b1600640 1 --2- 172.21.15.27:0/4276505403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0c8db0 0x7f71ac072620 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f71ac06bf20 tx=0x7f719802b840 comp rx=0 tx=0).stop 2024-03-28T18:56:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.288+0000 7f71b1600640 1 -- 172.21.15.27:0/4276505403 shutdown_connections 2024-03-28T18:56:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.288+0000 7f71b1600640 1 --2- 172.21.15.27:0/4276505403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac0c8db0 0x7f71ac072620 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.288+0000 7f71b1600640 1 --2- 172.21.15.27:0/4276505403 >> v2:172.21.15.155:3300/0 conn(0x7f71ac0c5aa0 0x7f71ac156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.288+0000 7f71b1600640 1 -- 172.21.15.27:0/4276505403 >> 172.21.15.27:0/4276505403 conn(0x7f71ac06b0c0 msgr2=0x7f71ac06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.288+0000 7f71b1600640 1 -- 172.21.15.27:0/4276505403 shutdown_connections 2024-03-28T18:56:33.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.288+0000 7f71b1600640 1 -- 172.21.15.27:0/4276505403 wait complete. 2024-03-28T18:56:33.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.291+0000 7f71b1600640 1 Processor -- start 2024-03-28T18:56:33.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.291+0000 7f71b1600640 1 -- start start 2024-03-28T18:56:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71b1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac072b60 0x7f71ac1517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71b1600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71ac0c5aa0 0x7f71ac159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71b1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0c8db0 0x7f71ac14b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71b1600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac072100 con 0x7f71ac072b60 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71b1600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac071e20 con 0x7f71ac0c8db0 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71b1600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac152f50 con 0x7f71ac0c5aa0 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac072b60 0x7f71ac1517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac072b60 0x7f71ac1517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40492/0 (socket says 172.21.15.27:40492) 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71abe00640 1 -- 172.21.15.27:0/1220920870 learned_addr learned my addr 172.21.15.27:0/1220920870 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71ab400640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71ac0c5aa0 0x7f71ac159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71b0c00640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0c8db0 0x7f71ac14b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71abe00640 1 -- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71ac0c5aa0 msgr2=0x7f71ac159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71abe00640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71ac0c5aa0 0x7f71ac159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71abe00640 1 -- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0c8db0 msgr2=0x7f71ac14b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71abe00640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0c8db0 0x7f71ac14b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.293+0000 7f71abe00640 1 -- 172.21.15.27:0/1220920870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac1582b0 con 0x7f71ac072b60 2024-03-28T18:56:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.294+0000 7f71b0c00640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0c8db0 0x7f71ac14b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.294+0000 7f71abe00640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac072b60 0x7f71ac1517f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f719c00d800 tx=0x7f719c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.294+0000 7f71a3e00640 1 -- 172.21.15.27:0/1220920870 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c00b6c0 con 0x7f71ac072b60 2024-03-28T18:56:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.294+0000 7f71a3e00640 1 -- 172.21.15.27:0/1220920870 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c019070 con 0x7f71ac072b60 2024-03-28T18:56:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.294+0000 7f71a3e00640 1 -- 172.21.15.27:0/1220920870 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c00ba80 con 0x7f71ac072b60 2024-03-28T18:56:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.294+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac0c16c0 con 0x7f71ac072b60 2024-03-28T18:56:33.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.294+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac1ebd50 con 0x7f71ac072b60 2024-03-28T18:56:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.295+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ac04f400 con 0x7f71ac072b60 2024-03-28T18:56:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.297+0000 7f71a3e00640 1 -- 172.21.15.27:0/1220920870 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f719c00bc20 con 0x7f71ac072b60 2024-03-28T18:56:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.297+0000 7f71a3e00640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f718c06bc00 0x7f718c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.298+0000 7f71a3e00640 1 -- 172.21.15.27:0/1220920870 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f719c08b2e0 con 0x7f71ac072b60 2024-03-28T18:56:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.298+0000 7f71ab400640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f718c06bc00 0x7f718c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.299+0000 7f71ab400640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f718c06bc00 0x7f718c06e0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7198036000 tx=0x7f719803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:33.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.302+0000 7f71a3e00640 1 -- 172.21.15.27:0/1220920870 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f719c01c360 con 0x7f71ac072b60 2024-03-28T18:56:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.500+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f71ac0c52e0 con 0x7f71ac072b60 2024-03-28T18:56:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.501+0000 7f71a3e00640 1 -- 172.21.15.27:0/1220920870 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v26) v1 ==== 95+0+11 (secure 0 0 0) 0x7f719c059a20 con 0x7f71ac072b60 2024-03-28T18:56:33.501 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f718c06bc00 msgr2=0x7f718c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f718c06bc00 0x7f718c06e0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7198036000 tx=0x7f719803d040 comp rx=0 tx=0).stop 2024-03-28T18:56:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac072b60 msgr2=0x7f71ac1517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac072b60 0x7f71ac1517f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f719c00d800 tx=0x7f719c00dcc0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 shutdown_connections 2024-03-28T18:56:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f718c06bc00 0x7f718c06e0c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0c8db0 0x7f71ac14b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71ac0c5aa0 0x7f71ac159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 --2- 172.21.15.27:0/1220920870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac072b60 0x7f71ac1517f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 >> 172.21.15.27:0/1220920870 conn(0x7f71ac06b0c0 msgr2=0x7f71ac154400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:33.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 shutdown_connections 2024-03-28T18:56:33.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.504+0000 7f71b1600640 1 -- 172.21.15.27:0/1220920870 wait complete. 2024-03-28T18:56:33.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-03-28T18:56:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.744+0000 7f2c13000640 1 Processor -- start 2024-03-28T18:56:33.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.744+0000 7f2c13000640 1 -- start start 2024-03-28T18:56:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c13000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2c0c0c1400 0x7f2c0c156120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c13000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c0c0c2230 0x7f2c0c06e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c13000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0c0c0cd0 con 0x7f2c0c152eb0 2024-03-28T18:56:33.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c13000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0c0629f0 con 0x7f2c0c0c2230 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c13000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0c0c2680 con 0x7f2c0c0c1400 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c12600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c0c0c2230 0x7f2c0c06e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c12600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c0c0c2230 0x7f2c0c06e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40502/0 (socket says 172.21.15.27:40502) 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c12600640 1 -- 172.21.15.27:0/2652411675 learned_addr learned my addr 172.21.15.27:0/2652411675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.746+0000 7f2c11200640 1 --2- 172.21.15.27:0/2652411675 >> v2:172.21.15.155:3300/0 conn(0x7f2c0c0c1400 0x7f2c0c156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.747+0000 7f2c12600640 1 -- 172.21.15.27:0/2652411675 >> v2:172.21.15.155:3300/0 conn(0x7f2c0c0c1400 msgr2=0x7f2c0c156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.747+0000 7f2c12600640 1 --2- 172.21.15.27:0/2652411675 >> v2:172.21.15.155:3300/0 conn(0x7f2c0c0c1400 0x7f2c0c156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.747+0000 7f2c12600640 1 -- 172.21.15.27:0/2652411675 >> v1:172.21.15.155:6789/0 conn(0x7f2c0c152eb0 legacy=0x7f2c0c159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:33.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.747+0000 7f2c12600640 1 -- 172.21.15.27:0/2652411675 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c0c0710 con 0x7f2c0c0c2230 2024-03-28T18:56:33.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.748+0000 7f2c12600640 1 --2- 172.21.15.27:0/2652411675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c2230 0x7f2c0c06e070 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c06fad0 tx=0x7f2c0402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4166aef0735fc00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.748+0000 7f2c0be00640 1 -- 172.21.15.27:0/2652411675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0402c960 con 0x7f2c0c0c2230 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.748+0000 7f2c0be00640 1 -- 172.21.15.27:0/2652411675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0402cb00 con 0x7f2c0c0c2230 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.748+0000 7f2c0be00640 1 -- 172.21.15.27:0/2652411675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c040343f0 con 0x7f2c0c0c2230 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.749+0000 7f2c13000640 1 -- 172.21.15.27:0/2652411675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c2230 msgr2=0x7f2c0c06e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.749+0000 7f2c13000640 1 --2- 172.21.15.27:0/2652411675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c2230 0x7f2c0c06e070 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c06fad0 tx=0x7f2c0402bdc0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.749+0000 7f2c13000640 1 -- 172.21.15.27:0/2652411675 shutdown_connections 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.749+0000 7f2c13000640 1 --2- 172.21.15.27:0/2652411675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c2230 0x7f2c0c06e070 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.749+0000 7f2c13000640 1 --2- 172.21.15.27:0/2652411675 >> v2:172.21.15.155:3300/0 conn(0x7f2c0c0c1400 0x7f2c0c156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.749+0000 7f2c13000640 1 -- 172.21.15.27:0/2652411675 >> 172.21.15.27:0/2652411675 conn(0x7f2c0c0c7380 msgr2=0x7f2c0c0c7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.749+0000 7f2c13000640 1 -- 172.21.15.27:0/2652411675 shutdown_connections 2024-03-28T18:56:33.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.749+0000 7f2c13000640 1 -- 172.21.15.27:0/2652411675 wait complete. 2024-03-28T18:56:33.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.751+0000 7f2c13000640 1 Processor -- start 2024-03-28T18:56:33.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.751+0000 7f2c13000640 1 -- start start 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c13000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c1400 0x7f2c0c1ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c13000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c0c0c2230 0x7f2c0c1eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c13000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c0c152eb0 0x7f2c0c0c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c13000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c159040 con 0x7f2c0c0c1400 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c13000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c158d60 con 0x7f2c0c0c2230 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c13000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0c06e5b0 con 0x7f2c0c152eb0 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c1400 0x7f2c0c1ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c1400 0x7f2c0c1ebf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40516/0 (socket says 172.21.15.27:40516) 2024-03-28T18:56:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11c00640 1 -- 172.21.15.27:0/461663658 learned_addr learned my addr 172.21.15.27:0/461663658 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11200640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c0c0c2230 0x7f2c0c1eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11c00640 1 -- 172.21.15.27:0/461663658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c0c152eb0 msgr2=0x7f2c0c0c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11c00640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c0c152eb0 0x7f2c0c0c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11c00640 1 -- 172.21.15.27:0/461663658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c0c0c2230 msgr2=0x7f2c0c1eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11c00640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c0c0c2230 0x7f2c0c1eb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.752+0000 7f2c11c00640 1 -- 172.21.15.27:0/461663658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c1eba50 con 0x7f2c0c0c1400 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.753+0000 7f2c11c00640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c1400 0x7f2c0c1ebf80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2c0000b3f0 tx=0x7f2c0000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.753+0000 7f2c09600640 1 -- 172.21.15.27:0/461663658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c00015020 con 0x7f2c0c0c1400 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.753+0000 7f2c09600640 1 -- 172.21.15.27:0/461663658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c00011830 con 0x7f2c0c0c1400 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.753+0000 7f2c09600640 1 -- 172.21.15.27:0/461663658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c00011c20 con 0x7f2c0c0c1400 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.753+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0c151640 con 0x7f2c0c0c1400 2024-03-28T18:56:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.753+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0c151360 con 0x7f2c0c0c1400 2024-03-28T18:56:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.755+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c04f400 con 0x7f2c0c0c1400 2024-03-28T18:56:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.755+0000 7f2c09600640 1 -- 172.21.15.27:0/461663658 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2c00018020 con 0x7f2c0c0c1400 2024-03-28T18:56:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.756+0000 7f2c09600640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2bec06bbd0 0x7f2bec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.757+0000 7f2c09600640 1 -- 172.21.15.27:0/461663658 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f2c0008aa60 con 0x7f2c0c0c1400 2024-03-28T18:56:33.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.757+0000 7f2c11200640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2bec06bbd0 0x7f2bec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.758+0000 7f2c11200640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2bec06bbd0 0x7f2bec06e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c1edfe0 tx=0x7f2bf8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:33.763+0000 7f2c09600640 1 -- 172.21.15.27:0/461663658 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2c0001f030 con 0x7f2c0c0c1400 2024-03-28T18:56:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.059+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c0c0626f0 con 0x7f2c0c0c1400 2024-03-28T18:56:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.060+0000 7f2c09600640 1 -- 172.21.15.27:0/461663658 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v26) v1 ==== 99+0+11 (secure 0 0 0) 0x7f2c0c1eba50 con 0x7f2c0c0c1400 2024-03-28T18:56:34.060 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2bec06bbd0 msgr2=0x7f2bec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2bec06bbd0 0x7f2bec06e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c1edfe0 tx=0x7f2bf8008040 comp rx=0 tx=0).stop 2024-03-28T18:56:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c1400 msgr2=0x7f2c0c1ebf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c1400 0x7f2c0c1ebf80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2c0000b3f0 tx=0x7f2c0000b8b0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 shutdown_connections 2024-03-28T18:56:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2bec06bbd0 0x7f2bec06e090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c0c152eb0 0x7f2c0c0c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c0c0c2230 0x7f2c0c1eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 --2- 172.21.15.27:0/461663658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c0c0c1400 0x7f2c0c1ebf80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 >> 172.21.15.27:0/461663658 conn(0x7f2c0c0c7380 msgr2=0x7f2c0c1521b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 shutdown_connections 2024-03-28T18:56:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.063+0000 7f2c13000640 1 -- 172.21.15.27:0/461663658 wait complete. 2024-03-28T18:56:34.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-03-28T18:56:34.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.288+0000 7fa2c7e00640 1 Processor -- start 2024-03-28T18:56:34.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.289+0000 7fa2c7e00640 1 -- start start 2024-03-28T18:56:34.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.290+0000 7fa2c7e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa2c80c4f00 0x7fa2c806c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.290+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c806b8c0 0x7fa2c815d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.290+0000 7fa2c7e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c815e250 con 0x7fa2c815ddc0 2024-03-28T18:56:34.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.290+0000 7fa2c7e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c8071870 con 0x7fa2c80c4f00 2024-03-28T18:56:34.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.290+0000 7fa2c7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c80731d0 con 0x7fa2c806b8c0 2024-03-28T18:56:34.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.290+0000 7fa2c7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa2c815ddc0 0x7fa2c80c6bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44806/0 (socket says 172.21.15.27:44806) 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.290+0000 7fa2c7400640 1 -- 172.21.15.27:0/1402071772 learned_addr learned my addr 172.21.15.27:0/1402071772 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6a00640 1 --2- 172.21.15.27:0/1402071772 >> v2:172.21.15.155:3300/0 conn(0x7fa2c80c4f00 0x7fa2c806c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2be000640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c806b8c0 0x7fa2c815d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730353508 0 0) 0x7fa2c815e250 con 0x7fa2c815ddc0 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ac003520 con 0x7fa2c815ddc0 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107669309 0 0) 0x7fa2ac003520 con 0x7fa2c815ddc0 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c806b8c0 msgr2=0x7fa2c815d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c806b8c0 0x7fa2c815d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 >> v2:172.21.15.155:3300/0 conn(0x7fa2c80c4f00 msgr2=0x7fa2c806c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 --2- 172.21.15.27:0/1402071772 >> v2:172.21.15.155:3300/0 conn(0x7fa2c80c4f00 0x7fa2c806c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8071ac0 con 0x7fa2c815ddc0 2024-03-28T18:56:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fa2c00019f0 con 0x7fa2c815ddc0 2024-03-28T18:56:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.291+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 >> v1:172.21.15.27:6789/0 conn(0x7fa2c815ddc0 legacy=0x7fa2c80c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6000640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6000640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c806b8c0 0x7fa2ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6000640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2ac004d40 0x7fa2ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c8071870 con 0x7fa2ac0039b0 2024-03-28T18:56:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c80731d0 con 0x7fa2ac004d40 2024-03-28T18:56:34.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac00b9c0 con 0x7fa2c806b8c0 2024-03-28T18:56:34.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6a00640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6a00640 1 -- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c806b8c0 msgr2=0x7fa2ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:34.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6a00640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c806b8c0 0x7fa2ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6a00640 1 -- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2ac004d40 msgr2=0x7fa2ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:34.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6a00640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2ac004d40 0x7fa2ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.292+0000 7fa2c6a00640 1 -- 172.21.15.27:0/1402071772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac003520 con 0x7fa2ac0039b0 2024-03-28T18:56:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.293+0000 7fa2c6a00640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2ac003dd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009ac0 tx=0x7fa2ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c50ea4d574a2412 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8039070 con 0x7fa2ac0039b0 2024-03-28T18:56:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b802c710 con 0x7fa2ac0039b0 2024-03-28T18:56:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c6000640 1 -- 172.21.15.27:0/1402071772 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8009690 con 0x7fa2ac0039b0 2024-03-28T18:56:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1402071772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 msgr2=0x7fa2ac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c7e00640 1 --2- 172.21.15.27:0/1402071772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2ac003dd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009ac0 tx=0x7fa2ac005830 comp rx=0 tx=0).stop 2024-03-28T18:56:34.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c7400640 1 -- 172.21.15.27:0/1402071772 reap_dead start 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1402071772 shutdown_connections 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1402071772 >> 172.21.15.27:0/1402071772 conn(0x7fa2c80c8130 msgr2=0x7fa2c80c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.294+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1402071772 shutdown_connections 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.295+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1402071772 wait complete. 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.296+0000 7fa2c7e00640 1 Processor -- start 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.296+0000 7fa2c7e00640 1 -- start start 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.297+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2c80c3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.297+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c81e4240 0x7fa2c81efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.297+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c81e47e0 0x7fa2c81efff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.297+0000 7fa2c7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c8072360 con 0x7fa2ac0039b0 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.297+0000 7fa2c7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c8072080 con 0x7fa2c81e47e0 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.297+0000 7fa2c7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c80c52e0 con 0x7fa2c81e4240 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2c80c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2c80c3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40542/0 (socket says 172.21.15.27:40542) 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 -- 172.21.15.27:0/1574929959 learned_addr learned my addr 172.21.15.27:0/1574929959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2be000640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c81e4240 0x7fa2c81efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c7400640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c81e47e0 0x7fa2c81efff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 -- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c81e4240 msgr2=0x7fa2c81efab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c81e4240 0x7fa2c81efab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 -- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c81e47e0 msgr2=0x7fa2c81efff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c81e47e0 0x7fa2c81efff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 -- 172.21.15.27:0/1574929959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c81e4f10 con 0x7fa2ac0039b0 2024-03-28T18:56:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2be000640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c81e4240 0x7fa2c81efab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:34.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.298+0000 7fa2c6a00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2c80c3770 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009be0 tx=0x7fa2b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:34.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.299+0000 7fa2bf400640 1 -- 172.21.15.27:0/1574929959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8042020 con 0x7fa2ac0039b0 2024-03-28T18:56:34.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.299+0000 7fa2bf400640 1 -- 172.21.15.27:0/1574929959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b803a040 con 0x7fa2ac0039b0 2024-03-28T18:56:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.299+0000 7fa2bf400640 1 -- 172.21.15.27:0/1574929959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8039070 con 0x7fa2ac0039b0 2024-03-28T18:56:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.299+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c81e51d0 con 0x7fa2ac0039b0 2024-03-28T18:56:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.299+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c80c3cb0 con 0x7fa2ac0039b0 2024-03-28T18:56:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.300+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c815c540 con 0x7fa2ac0039b0 2024-03-28T18:56:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.301+0000 7fa2bf400640 1 -- 172.21.15.27:0/1574929959 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa2b8045020 con 0x7fa2ac0039b0 2024-03-28T18:56:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.302+0000 7fa2bf400640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2a406bbd0 0x7fa2a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.302+0000 7fa2bf400640 1 -- 172.21.15.27:0/1574929959 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fa2b80ac360 con 0x7fa2ac0039b0 2024-03-28T18:56:34.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.302+0000 7fa2be000640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2a406bbd0 0x7fa2a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.303+0000 7fa2be000640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2a406bbd0 0x7fa2a406e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80c3f80 tx=0x7fa2b0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:34.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.310+0000 7fa2bf400640 1 -- 172.21.15.27:0/1574929959 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa2c815c540 con 0x7fa2ac0039b0 2024-03-28T18:56:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.512+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fa2c804f400 con 0x7fa2ac0039b0 2024-03-28T18:56:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.513+0000 7fa2bf400640 1 -- 172.21.15.27:0/1574929959 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v26) v1 ==== 95+0+11 (secure 0 0 0) 0x7fa2b804c070 con 0x7fa2ac0039b0 2024-03-28T18:56:34.513 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:34.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.515+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2a406bbd0 msgr2=0x7fa2a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.515+0000 7fa2c7e00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2a406bbd0 0x7fa2a406e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80c3f80 tx=0x7fa2b0008040 comp rx=0 tx=0).stop 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.515+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 msgr2=0x7fa2c80c3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.515+0000 7fa2c7e00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2c80c3770 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009be0 tx=0x7fa2b8004730 comp rx=0 tx=0).stop 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.515+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 shutdown_connections 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.515+0000 7fa2c7e00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2a406bbd0 0x7fa2a406e090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.515+0000 7fa2c7e00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c81e47e0 0x7fa2c81efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.515+0000 7fa2c7e00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c81e4240 0x7fa2c81efab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.516+0000 7fa2c7e00640 1 --2- 172.21.15.27:0/1574929959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2ac0039b0 0x7fa2c80c3770 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.516+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 >> 172.21.15.27:0/1574929959 conn(0x7fa2c80c8130 msgr2=0x7fa2c80725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.516+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 shutdown_connections 2024-03-28T18:56:34.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.516+0000 7fa2c7e00640 1 -- 172.21.15.27:0/1574929959 wait complete. 2024-03-28T18:56:34.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:56:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.745+0000 7f849cc00640 1 Processor -- start 2024-03-28T18:56:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.745+0000 7f849cc00640 1 -- start start 2024-03-28T18:56:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.747+0000 7f849cc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84980c8470 0x7f84980ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.747+0000 7f849cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84980c8c70 0x7f84980c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.747+0000 7f849cc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84980ad680 con 0x7f8498150b60 2024-03-28T18:56:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.747+0000 7f849cc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8498157210 con 0x7f84980c8c70 2024-03-28T18:56:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.747+0000 7f849cc00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84980ad9b0 con 0x7f84980c8470 2024-03-28T18:56:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f8495e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8498150b60 0x7f8498156b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44818/0 (socket says 172.21.15.27:44818) 2024-03-28T18:56:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f8495e00640 1 -- 172.21.15.27:0/3100439546 learned_addr learned my addr 172.21.15.27:0/3100439546 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f8496800640 1 --2- 172.21.15.27:0/3100439546 >> v2:172.21.15.27:3300/0 conn(0x7f84980c8c70 0x7f84980c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f848fe00640 1 --2- 172.21.15.27:0/3100439546 >> v2:172.21.15.155:3300/0 conn(0x7f84980c8470 0x7f84980ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f8496800640 1 -- 172.21.15.27:0/3100439546 >> v2:172.21.15.155:3300/0 conn(0x7f84980c8470 msgr2=0x7f84980ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f8496800640 1 --2- 172.21.15.27:0/3100439546 >> v2:172.21.15.155:3300/0 conn(0x7f84980c8470 0x7f84980ade40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f8496800640 1 -- 172.21.15.27:0/3100439546 >> v1:172.21.15.27:6789/0 conn(0x7f8498150b60 legacy=0x7f8498156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f8496800640 1 -- 172.21.15.27:0/3100439546 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84980c4de0 con 0x7f84980c8c70 2024-03-28T18:56:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.748+0000 7f8495400640 1 -- 172.21.15.27:0/3100439546 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909655222 0 0) 0x7f84980ad680 con 0x7f8498150b60 2024-03-28T18:56:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.749+0000 7f8496800640 1 --2- 172.21.15.27:0/3100439546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f84980c9120 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8490009d90 tx=0x7f849002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476aa18db57f6b41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.750+0000 7f8495400640 1 -- 172.21.15.27:0/3100439546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490039070 con 0x7f84980c8c70 2024-03-28T18:56:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.750+0000 7f8495400640 1 -- 172.21.15.27:0/3100439546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849002cbd0 con 0x7f84980c8c70 2024-03-28T18:56:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.750+0000 7f8495400640 1 -- 172.21.15.27:0/3100439546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490034440 con 0x7f84980c8c70 2024-03-28T18:56:34.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.750+0000 7f849cc00640 1 -- 172.21.15.27:0/3100439546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 msgr2=0x7f84980c9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.750+0000 7f849cc00640 1 --2- 172.21.15.27:0/3100439546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f84980c9120 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8490009d90 tx=0x7f849002be20 comp rx=0 tx=0).stop 2024-03-28T18:56:34.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.750+0000 7f849cc00640 1 -- 172.21.15.27:0/3100439546 shutdown_connections 2024-03-28T18:56:34.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.750+0000 7f849cc00640 1 --2- 172.21.15.27:0/3100439546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f84980c9120 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.750+0000 7f849cc00640 1 --2- 172.21.15.27:0/3100439546 >> v2:172.21.15.155:3300/0 conn(0x7f84980c8470 0x7f84980ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.751+0000 7f849cc00640 1 -- 172.21.15.27:0/3100439546 >> 172.21.15.27:0/3100439546 conn(0x7f84980aaab0 msgr2=0x7f84980aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:34.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.751+0000 7f849cc00640 1 -- 172.21.15.27:0/3100439546 shutdown_connections 2024-03-28T18:56:34.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.751+0000 7f849cc00640 1 -- 172.21.15.27:0/3100439546 wait complete. 2024-03-28T18:56:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.752+0000 7f849cc00640 1 Processor -- start 2024-03-28T18:56:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.752+0000 7f849cc00640 1 -- start start 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.753+0000 7f849cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84980c8470 0x7f8498155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.753+0000 7f849cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f8498151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.753+0000 7f849cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8498150b60 0x7f84980c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.753+0000 7f849cc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8498150f40 con 0x7f84980c8c70 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.753+0000 7f849cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84980c53a0 con 0x7f84980c8470 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.753+0000 7f849cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84980a85a0 con 0x7f8498150b60 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f848fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f8498151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f848fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f8498151400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40562/0 (socket says 172.21.15.27:40562) 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f848fe00640 1 -- 172.21.15.27:0/769202407 learned_addr learned my addr 172.21.15.27:0/769202407 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f8495e00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84980c8470 0x7f8498155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f8496800640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8498150b60 0x7f84980c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f848fe00640 1 -- 172.21.15.27:0/769202407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8498150b60 msgr2=0x7f84980c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f848fe00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8498150b60 0x7f84980c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f848fe00640 1 -- 172.21.15.27:0/769202407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84980c8470 msgr2=0x7f8498155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f848fe00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84980c8470 0x7f8498155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.754+0000 7f848fe00640 1 -- 172.21.15.27:0/769202407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84981529c0 con 0x7f84980c8c70 2024-03-28T18:56:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.755+0000 7f848fe00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f8498151400 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f848000d5d0 tx=0x7f848000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.755+0000 7f848e000640 1 -- 172.21.15.27:0/769202407 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84800125f0 con 0x7f84980c8c70 2024-03-28T18:56:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.755+0000 7f848e000640 1 -- 172.21.15.27:0/769202407 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8480019070 con 0x7f84980c8c70 2024-03-28T18:56:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.755+0000 7f848e000640 1 -- 172.21.15.27:0/769202407 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84800129f0 con 0x7f84980c8c70 2024-03-28T18:56:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.755+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84981e98c0 con 0x7f84980c8c70 2024-03-28T18:56:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.755+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8498154080 con 0x7f84980c8c70 2024-03-28T18:56:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.756+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849804f400 con 0x7f84980c8c70 2024-03-28T18:56:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.758+0000 7f848e000640 1 -- 172.21.15.27:0/769202407 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8480012b90 con 0x7f84980c8c70 2024-03-28T18:56:34.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.759+0000 7f848e000640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847406bc00 0x7f847406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.759+0000 7f848e000640 1 -- 172.21.15.27:0/769202407 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f848008bd00 con 0x7f84980c8c70 2024-03-28T18:56:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.759+0000 7f8495e00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847406bc00 0x7f847406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:34.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.760+0000 7f8495e00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847406bc00 0x7f847406e0c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8488000f60 tx=0x7f8488007b50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:34.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:34.764+0000 7f848e000640 1 -- 172.21.15.27:0/769202407 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f84981529c0 con 0x7f84980c8c70 2024-03-28T18:56:35.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.012+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8498156de0 con 0x7f84980c8c70 2024-03-28T18:56:35.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.013+0000 7f848e000640 1 -- 172.21.15.27:0/769202407 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f848005a4c0 con 0x7f84980c8c70 2024-03-28T18:56:35.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:35.013 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:56:35.013 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:56:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.015+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847406bc00 msgr2=0x7f847406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.015+0000 7f849cc00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847406bc00 0x7f847406e0c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8488000f60 tx=0x7f8488007b50 comp rx=0 tx=0).stop 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.015+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 msgr2=0x7f8498151400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.015+0000 7f849cc00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f8498151400 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f848000d5d0 tx=0x7f848000da90 comp rx=0 tx=0).stop 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.015+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 shutdown_connections 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.016+0000 7f849cc00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847406bc00 0x7f847406e0c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.016+0000 7f849cc00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8498150b60 0x7f84980c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.016+0000 7f849cc00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84980c8c70 0x7f8498151400 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.016+0000 7f849cc00640 1 --2- 172.21.15.27:0/769202407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84980c8470 0x7f8498155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.016+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 >> 172.21.15.27:0/769202407 conn(0x7f84980aaab0 msgr2=0x7f84980a88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.016+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 shutdown_connections 2024-03-28T18:56:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.016+0000 7f849cc00640 1 -- 172.21.15.27:0/769202407 wait complete. 2024-03-28T18:56:35.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:35.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.263+0000 7f9ce9800640 1 Processor -- start 2024-03-28T18:56:35.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.263+0000 7f9ce9800640 1 -- start start 2024-03-28T18:56:35.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.264+0000 7f9ce9800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce40a7060 0x7f9ce40c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.264+0000 7f9ce9800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce4072f90 con 0x7f9ce4155700 2024-03-28T18:56:35.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.264+0000 7f9ce9800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce4073270 con 0x7f9ce4152ad0 2024-03-28T18:56:35.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.264+0000 7f9ce9800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4072d40 con 0x7f9ce40a7060 2024-03-28T18:56:35.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.265+0000 7f9ce3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9ce4155700 0x7f9ce40c3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44820/0 (socket says 172.21.15.27:44820) 2024-03-28T18:56:35.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.265+0000 7f9ce3400640 1 -- 172.21.15.27:0/1696249589 learned_addr learned my addr 172.21.15.27:0/1696249589 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:35.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.265+0000 7f9ce3e00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce40a7060 0x7f9ce40c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.265+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807370850 0 0) 0x7f9ce4072f90 con 0x7f9ce4155700 2024-03-28T18:56:35.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.265+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cc8003520 con 0x7f9ce4155700 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2423513693 0 0) 0x7f9cc8003520 con 0x7f9ce4155700 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce40a7060 msgr2=0x7f9ce40c2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce40a7060 0x7f9ce40c2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 >> v1:172.21.15.155:6789/0 conn(0x7f9ce4152ad0 legacy=0x7f9ce406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce41522d0 con 0x7f9ce4155700 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9cd0001670 con 0x7f9ce4155700 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 >> v1:172.21.15.27:6789/0 conn(0x7f9ce4155700 legacy=0x7f9ce40c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce40a7060 0x7f9cc80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cc8003f60 0x7f9cc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc8004d40 0x7f9cc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4072d40 con 0x7f9cc8004d40 2024-03-28T18:56:35.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc800b9c0 con 0x7f9cc8003f60 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc800bbc0 con 0x7f9ce40a7060 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.266+0000 7f9ce8e00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc8004d40 0x7f9cc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.267+0000 7f9ce3e00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce40a7060 0x7f9cc80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.267+0000 7f9ce3400640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cc8003f60 0x7f9cc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.267+0000 7f9ce8e00640 1 -- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce40a7060 msgr2=0x7f9cc80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.267+0000 7f9ce8e00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce40a7060 0x7f9cc80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.267+0000 7f9ce8e00640 1 -- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cc8003f60 msgr2=0x7f9cc8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.267+0000 7f9ce8e00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9cc8003f60 0x7f9cc8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.267+0000 7f9ce8e00640 1 -- 172.21.15.27:0/1696249589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc8003520 con 0x7f9cc8004d40 2024-03-28T18:56:35.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.268+0000 7f9ce8e00640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc8004d40 0x7f9cc80051c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc001e50 tx=0x7f9cc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580e3f1cbe6b764e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:35.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.268+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc001a70 con 0x7f9cc8004d40 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.268+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc003b70 con 0x7f9cc8004d40 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.268+0000 7f9ce2a00640 1 -- 172.21.15.27:0/1696249589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0050d0 con 0x7f9cc8004d40 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.269+0000 7f9ce9800640 1 -- 172.21.15.27:0/1696249589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc8004d40 msgr2=0x7f9cc80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.269+0000 7f9ce9800640 1 --2- 172.21.15.27:0/1696249589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9cc8004d40 0x7f9cc80051c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc001e50 tx=0x7f9cc8005830 comp rx=0 tx=0).stop 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.269+0000 7f9ce8e00640 1 -- 172.21.15.27:0/1696249589 reap_dead start 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.269+0000 7f9ce9800640 1 -- 172.21.15.27:0/1696249589 shutdown_connections 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.269+0000 7f9ce9800640 1 -- 172.21.15.27:0/1696249589 >> 172.21.15.27:0/1696249589 conn(0x7f9ce40c7aa0 msgr2=0x7f9ce40c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.269+0000 7f9ce9800640 1 -- 172.21.15.27:0/1696249589 shutdown_connections 2024-03-28T18:56:35.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.269+0000 7f9ce9800640 1 -- 172.21.15.27:0/1696249589 wait complete. 2024-03-28T18:56:35.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.271+0000 7f9ce9800640 1 Processor -- start 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.271+0000 7f9ce9800640 1 -- start start 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce9800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce406f4f0 0x7f9ce4149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce9800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce414ffa0 0x7f9ce414a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4150460 0x7f9ce41ea130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce9800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4155b60 con 0x7f9ce4150460 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce9800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4073550 con 0x7f9ce414ffa0 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce9800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4152f80 con 0x7f9ce406f4f0 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4150460 0x7f9ce41ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4150460 0x7f9ce41ea130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40572/0 (socket says 172.21.15.27:40572) 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce8e00640 1 -- 172.21.15.27:0/3524156348 learned_addr learned my addr 172.21.15.27:0/3524156348 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce3e00640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce406f4f0 0x7f9ce4149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce3400640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce414ffa0 0x7f9ce414a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce8e00640 1 -- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce406f4f0 msgr2=0x7f9ce4149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce8e00640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce406f4f0 0x7f9ce4149de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce8e00640 1 -- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce414ffa0 msgr2=0x7f9ce414a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce8e00640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce414ffa0 0x7f9ce414a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce8e00640 1 -- 172.21.15.27:0/3524156348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce4151400 con 0x7f9ce4150460 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.272+0000 7f9ce3400640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce414ffa0 0x7f9ce414a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.273+0000 7f9ce8e00640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4150460 0x7f9ce41ea130 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc001f70 tx=0x7f9cdc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.273+0000 7f9cdbe00640 1 -- 172.21.15.27:0/3524156348 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc007db0 con 0x7f9ce4150460 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.273+0000 7f9cdbe00640 1 -- 172.21.15.27:0/3524156348 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc003970 con 0x7f9ce4150460 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.273+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce414b440 con 0x7f9ce4150460 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.273+0000 7f9cdbe00640 1 -- 172.21.15.27:0/3524156348 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc02f6a0 con 0x7f9ce4150460 2024-03-28T18:56:35.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.273+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4150c00 con 0x7f9ce4150460 2024-03-28T18:56:35.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.274+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce4069640 con 0x7f9ce4150460 2024-03-28T18:56:35.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.275+0000 7f9cdbe00640 1 -- 172.21.15.27:0/3524156348 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9cdc02f840 con 0x7f9ce4150460 2024-03-28T18:56:35.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.276+0000 7f9cdbe00640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc406bbd0 0x7f9cc406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.276+0000 7f9cdbe00640 1 -- 172.21.15.27:0/3524156348 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f9cdc0ac420 con 0x7f9ce4150460 2024-03-28T18:56:35.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.276+0000 7f9ce3e00640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc406bbd0 0x7f9cc406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.277+0000 7f9ce3e00640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc406bbd0 0x7f9cc406e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009950 tx=0x7f9cd4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:35.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.281+0000 7f9cdbe00640 1 -- 172.21.15.27:0/3524156348 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9cdc033020 con 0x7f9ce4150460 2024-03-28T18:56:35.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.475+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ce40c5650 con 0x7f9ce4150460 2024-03-28T18:56:35.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.476+0000 7f9cdbe00640 1 -- 172.21.15.27:0/3524156348 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+20501 (secure 0 0 0) 0x7f9cdc07ac00 con 0x7f9ce4150460 2024-03-28T18:56:35.477 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:35.477 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:56:15.976621+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:35.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc406bbd0 msgr2=0x7f9cc406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc406bbd0 0x7f9cc406e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009950 tx=0x7f9cd4008040 comp rx=0 tx=0).stop 2024-03-28T18:56:35.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4150460 msgr2=0x7f9ce41ea130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4150460 0x7f9ce41ea130 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc001f70 tx=0x7f9cdc0014a0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 shutdown_connections 2024-03-28T18:56:35.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc406bbd0 0x7f9cc406e090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4150460 0x7f9ce41ea130 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce414ffa0 0x7f9ce414a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 --2- 172.21.15.27:0/3524156348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce406f4f0 0x7f9ce4149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 >> 172.21.15.27:0/3524156348 conn(0x7f9ce40c7aa0 msgr2=0x7f9ce4067660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:35.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.479+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 shutdown_connections 2024-03-28T18:56:35.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.480+0000 7f9ce9800640 1 -- 172.21.15.27:0/3524156348 wait complete. 2024-03-28T18:56:35.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:35.498 INFO:tasks.mds_thrash.fs.[a]:starting mds_do_thrash for fs a 2024-03-28T18:56:35.499 INFO:tasks.mds_thrash.fs.[a]:waiting for 78 secs before thrashing 2024-03-28T18:56:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.708+0000 7fb14ac00640 1 Processor -- start 2024-03-28T18:56:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.708+0000 7fb14ac00640 1 -- start start 2024-03-28T18:56:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.709+0000 7fb14ac00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1440c58c0 0x7fb1440c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.709+0000 7fb14ac00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1440c8ed0 0x7fb1441555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.709+0000 7fb14ac00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb144156230 con 0x7fb1440be5d0 2024-03-28T18:56:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.709+0000 7fb14ac00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb144158bd0 con 0x7fb1440c58c0 2024-03-28T18:56:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.709+0000 7fb14ac00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb144158eb0 con 0x7fb1440c8ed0 2024-03-28T18:56:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb149800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1440c58c0 0x7fb1440c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb14a200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb1440be5d0 0x7fb144155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44822/0 (socket says 172.21.15.27:44822) 2024-03-28T18:56:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb14a200640 1 -- 172.21.15.27:0/337715583 learned_addr learned my addr 172.21.15.27:0/337715583 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb148e00640 1 --2- 172.21.15.27:0/337715583 >> v2:172.21.15.155:3300/0 conn(0x7fb1440c8ed0 0x7fb1441555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb149800640 1 -- 172.21.15.27:0/337715583 >> v2:172.21.15.155:3300/0 conn(0x7fb1440c8ed0 msgr2=0x7fb1441555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:35.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb149800640 1 --2- 172.21.15.27:0/337715583 >> v2:172.21.15.155:3300/0 conn(0x7fb1440c8ed0 0x7fb1441555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb149800640 1 -- 172.21.15.27:0/337715583 >> v1:172.21.15.27:6789/0 conn(0x7fb1440be5d0 legacy=0x7fb144155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb149800640 1 -- 172.21.15.27:0/337715583 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144159190 con 0x7fb1440c58c0 2024-03-28T18:56:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.710+0000 7fb143e00640 1 -- 172.21.15.27:0/337715583 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053030903 0 0) 0x7fb144156230 con 0x7fb1440be5d0 2024-03-28T18:56:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.711+0000 7fb149800640 1 --2- 172.21.15.27:0/337715583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440c5ca0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb1440c83d0 tx=0x7fb13802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb8ff122c03b835 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.712+0000 7fb143e00640 1 -- 172.21.15.27:0/337715583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138038070 con 0x7fb1440c58c0 2024-03-28T18:56:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.712+0000 7fb143e00640 1 -- 172.21.15.27:0/337715583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb138031710 con 0x7fb1440c58c0 2024-03-28T18:56:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.712+0000 7fb143e00640 1 -- 172.21.15.27:0/337715583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138031ae0 con 0x7fb1440c58c0 2024-03-28T18:56:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.712+0000 7fb14ac00640 1 -- 172.21.15.27:0/337715583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 msgr2=0x7fb1440c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.712+0000 7fb14ac00640 1 --2- 172.21.15.27:0/337715583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440c5ca0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb1440c83d0 tx=0x7fb13802b7b0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.712+0000 7fb14ac00640 1 -- 172.21.15.27:0/337715583 shutdown_connections 2024-03-28T18:56:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.712+0000 7fb14ac00640 1 --2- 172.21.15.27:0/337715583 >> v2:172.21.15.155:3300/0 conn(0x7fb1440c8ed0 0x7fb1441555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.713+0000 7fb14ac00640 1 --2- 172.21.15.27:0/337715583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440c5ca0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.713+0000 7fb14ac00640 1 -- 172.21.15.27:0/337715583 >> 172.21.15.27:0/337715583 conn(0x7fb14406b6f0 msgr2=0x7fb1440acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.713+0000 7fb14ac00640 1 -- 172.21.15.27:0/337715583 shutdown_connections 2024-03-28T18:56:35.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.713+0000 7fb14ac00640 1 -- 172.21.15.27:0/337715583 wait complete. 2024-03-28T18:56:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.714+0000 7fb14ac00640 1 Processor -- start 2024-03-28T18:56:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.714+0000 7fb14ac00640 1 -- start start 2024-03-28T18:56:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.715+0000 7fb14ac00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1440be5d0 0x7fb1440ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.715+0000 7fb14ac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.715+0000 7fb14ac00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1440c8ed0 0x7fb1440c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.715+0000 7fb14ac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1440b0820 con 0x7fb1440c58c0 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.715+0000 7fb14ac00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144159750 con 0x7fb1440c8ed0 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.715+0000 7fb14ac00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1440b09a0 con 0x7fb1440be5d0 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440abce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40584/0 (socket says 172.21.15.27:40584) 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 -- 172.21.15.27:0/2940692289 learned_addr learned my addr 172.21.15.27:0/2940692289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb149800640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1440be5d0 0x7fb1440ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb14a200640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1440c8ed0 0x7fb1440c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 -- 172.21.15.27:0/2940692289 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1440be5d0 msgr2=0x7fb1440ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1440be5d0 0x7fb1440ab7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 -- 172.21.15.27:0/2940692289 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1440c8ed0 msgr2=0x7fb1440c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1440c8ed0 0x7fb1440c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 -- 172.21.15.27:0/2940692289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1440c2a70 con 0x7fb1440c58c0 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.716+0000 7fb148e00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440abce0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb13000d5f0 tx=0x7fb13000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.717+0000 7fb141600640 1 -- 172.21.15.27:0/2940692289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1300125f0 con 0x7fb1440c58c0 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.717+0000 7fb141600640 1 -- 172.21.15.27:0/2940692289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb130019070 con 0x7fb1440c58c0 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.717+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1440c23c0 con 0x7fb1440c58c0 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.717+0000 7fb141600640 1 -- 172.21.15.27:0/2940692289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1300129d0 con 0x7fb1440c58c0 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.717+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1440aca90 con 0x7fb1440c58c0 2024-03-28T18:56:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.718+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb144069640 con 0x7fb1440c58c0 2024-03-28T18:56:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.719+0000 7fb141600640 1 -- 172.21.15.27:0/2940692289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb130012b70 con 0x7fb1440c58c0 2024-03-28T18:56:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.720+0000 7fb141600640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb12406bbd0 0x7fb12406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.720+0000 7fb141600640 1 -- 172.21.15.27:0/2940692289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fb13008b220 con 0x7fb1440c58c0 2024-03-28T18:56:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.720+0000 7fb149800640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb12406bbd0 0x7fb12406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.721+0000 7fb149800640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb12406bbd0 0x7fb12406e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb138035000 tx=0x7fb138008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:35.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.724+0000 7fb141600640 1 -- 172.21.15.27:0/2940692289 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb1300598d0 con 0x7fb1440c58c0 2024-03-28T18:56:35.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.917+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb144156580 con 0x7fb1440c58c0 2024-03-28T18:56:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.921+0000 7fb141600640 1 -- 172.21.15.27:0/2940692289 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+20501 (secure 0 0 0) 0x7fb1300591d0 con 0x7fb1440c58c0 2024-03-28T18:56:35.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:35.921 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:56:15.976621+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb12406bbd0 msgr2=0x7fb12406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb12406bbd0 0x7fb12406e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb138035000 tx=0x7fb138008030 comp rx=0 tx=0).stop 2024-03-28T18:56:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 msgr2=0x7fb1440abce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440abce0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb13000d5f0 tx=0x7fb13000dab0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 shutdown_connections 2024-03-28T18:56:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb12406bbd0 0x7fb12406e090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1440c8ed0 0x7fb1440c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440c58c0 0x7fb1440abce0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 --2- 172.21.15.27:0/2940692289 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1440be5d0 0x7fb1440ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 >> 172.21.15.27:0/2940692289 conn(0x7fb14406b6f0 msgr2=0x7fb144157260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 shutdown_connections 2024-03-28T18:56:35.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:35.924+0000 7fb14ac00640 1 -- 172.21.15.27:0/2940692289 wait complete. 2024-03-28T18:56:35.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:55:22.980934+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-03-28T18:55:32.332283+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-03-28T18:55:32.983411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-03-28T18:55:43.415397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-03-28T18:55:44.084162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:56:35.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:56:36.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.153+0000 7f9743a00640 1 Processor -- start 2024-03-28T18:56:36.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.153+0000 7f9743a00640 1 -- start start 2024-03-28T18:56:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.155+0000 7f9743a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f973c06e960 0x7f973c1540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.155+0000 7f9743a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c06dbe0 con 0x7f973c0c17a0 2024-03-28T18:56:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.155+0000 7f9743a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c06de80 con 0x7f973c14d4e0 2024-03-28T18:56:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.155+0000 7f9743a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973c06e160 con 0x7f973c06e960 2024-03-28T18:56:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.155+0000 7f9741a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f973c0c17a0 0x7f973c0ab270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35166/0 (socket says 172.21.15.27:35166) 2024-03-28T18:56:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.155+0000 7f9741a00640 1 -- 172.21.15.27:0/1041487710 learned_addr learned my addr 172.21.15.27:0/1041487710 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973be00640 1 --2- 172.21.15.27:0/1041487710 >> v2:172.21.15.155:3300/0 conn(0x7f973c06e960 0x7f973c1540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798342519 0 0) 0x7f973c06dbe0 con 0x7f973c0c17a0 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9724003520 con 0x7f973c0c17a0 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 358344213 0 0) 0x7f9724003520 con 0x7f973c0c17a0 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 >> v2:172.21.15.155:3300/0 conn(0x7f973c06e960 msgr2=0x7f973c1540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973b400640 1 --2- 172.21.15.27:0/1041487710 >> v2:172.21.15.155:3300/0 conn(0x7f973c06e960 0x7f973c1540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 >> v1:172.21.15.155:6789/0 conn(0x7f973c14d4e0 legacy=0x7f973c0a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c06e440 con 0x7f973c0c17a0 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.156+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f97340019f0 con 0x7f973c0c17a0 2024-03-28T18:56:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 >> v1:172.21.15.27:6789/0 conn(0x7f973c0c17a0 legacy=0x7f973c0ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973b400640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f9724003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973b400640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9724004800 0x7f9724004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973b400640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97240051c0 0x7f972400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c06e160 con 0x7f9724003a20 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972400c0f0 con 0x7f97240051c0 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972400c2f0 con 0x7f9724004800 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f9741000640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f9724003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f9741a00640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97240051c0 0x7f972400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973be00640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9724004800 0x7f9724004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f9741000640 1 -- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9724004800 msgr2=0x7f9724004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f9741000640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9724004800 0x7f9724004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f9741000640 1 -- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97240051c0 msgr2=0x7f972400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f9741000640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97240051c0 0x7f972400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f9741000640 1 -- 172.21.15.27:0/1041487710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9724003520 con 0x7f9724003a20 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f9741a00640 1 -- 172.21.15.27:0/1041487710 reap_dead start 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.157+0000 7f973be00640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9724004800 0x7f9724004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.159+0000 7f9741000640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f9724003e40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9730001750 tx=0x7f9724005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454a7aeb246f169a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.159+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730001370 con 0x7f9724003a20 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.159+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97300052d0 con 0x7f9724003a20 2024-03-28T18:56:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.159+0000 7f973b400640 1 -- 172.21.15.27:0/1041487710 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730003830 con 0x7f9724003a20 2024-03-28T18:56:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.160+0000 7f9743a00640 1 -- 172.21.15.27:0/1041487710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 msgr2=0x7f9724003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.160+0000 7f9743a00640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f9724003e40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9730001750 tx=0x7f9724005830 comp rx=0 tx=0).stop 2024-03-28T18:56:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.160+0000 7f9743a00640 1 -- 172.21.15.27:0/1041487710 shutdown_connections 2024-03-28T18:56:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.160+0000 7f9743a00640 1 --2- 172.21.15.27:0/1041487710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f9724003e40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.160+0000 7f9743a00640 1 -- 172.21.15.27:0/1041487710 >> 172.21.15.27:0/1041487710 conn(0x7f973c06ca00 msgr2=0x7f973c06cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.160+0000 7f9743a00640 1 -- 172.21.15.27:0/1041487710 shutdown_connections 2024-03-28T18:56:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.160+0000 7f9743a00640 1 -- 172.21.15.27:0/1041487710 wait complete. 2024-03-28T18:56:36.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.161+0000 7f9743a00640 1 Processor -- start 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.162+0000 7f9743a00640 1 -- start start 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.162+0000 7f9743a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f973c157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9743a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f973c1467d0 0x7f973c1ee940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9743a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f973c157780 0x7f973c1eee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9743a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c14d8c0 con 0x7f9724003a20 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9743a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c0c1ba0 con 0x7f973c1467d0 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9743a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c14da40 con 0x7f973c157780 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f973c157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f973c157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58024/0 (socket says 172.21.15.27:58024) 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741000640 1 -- 172.21.15.27:0/277996843 learned_addr learned my addr 172.21.15.27:0/277996843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f973be00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f973c1467d0 0x7f973c1ee940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741a00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f973c157780 0x7f973c1eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741000640 1 -- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f973c157780 msgr2=0x7f973c1eee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741000640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f973c157780 0x7f973c1eee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741000640 1 -- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f973c1467d0 msgr2=0x7f973c1ee940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741000640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f973c1467d0 0x7f973c1ee940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741000640 1 -- 172.21.15.27:0/277996843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c1ef3c0 con 0x7f9724003a20 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f973be00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f973c1467d0 0x7f973c1ee940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.163+0000 7f9741a00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f973c157780 0x7f973c1eee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.164+0000 7f9741000640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f973c157240 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9730005040 tx=0x7f9730001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:36.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.164+0000 7f9738c00640 1 -- 172.21.15.27:0/277996843 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730041070 con 0x7f9724003a20 2024-03-28T18:56:36.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.164+0000 7f9738c00640 1 -- 172.21.15.27:0/277996843 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9730005480 con 0x7f9724003a20 2024-03-28T18:56:36.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.164+0000 7f9738c00640 1 -- 172.21.15.27:0/277996843 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730007a30 con 0x7f9724003a20 2024-03-28T18:56:36.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.164+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c147900 con 0x7f9724003a20 2024-03-28T18:56:36.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.164+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c147f00 con 0x7f9724003a20 2024-03-28T18:56:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.166+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973c06d050 con 0x7f9724003a20 2024-03-28T18:56:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.166+0000 7f9738c00640 1 -- 172.21.15.27:0/277996843 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9730044070 con 0x7f9724003a20 2024-03-28T18:56:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.167+0000 7f9738c00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f971c06bbd0 0x7f971c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.167+0000 7f9738c00640 1 -- 172.21.15.27:0/277996843 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f97300ac970 con 0x7f9724003a20 2024-03-28T18:56:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.168+0000 7f973be00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f971c06bbd0 0x7f971c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.168+0000 7f973be00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f971c06bbd0 0x7f971c06e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f973c1557f0 tx=0x7f9728008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:36.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.172+0000 7f9738c00640 1 -- 172.21.15.27:0/277996843 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f973c06d050 con 0x7f9724003a20 2024-03-28T18:56:36.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.372+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f973c04f470 con 0x7f9724003a20 2024-03-28T18:56:36.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.373+0000 7f9738c00640 1 -- 172.21.15.27:0/277996843 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f973007b020 con 0x7f9724003a20 2024-03-28T18:56:36.373 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-03-28T18:56:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f971c06bbd0 msgr2=0x7f971c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f971c06bbd0 0x7f971c06e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f973c1557f0 tx=0x7f9728008040 comp rx=0 tx=0).stop 2024-03-28T18:56:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 msgr2=0x7f973c157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f973c157240 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9730005040 tx=0x7f9730001120 comp rx=0 tx=0).stop 2024-03-28T18:56:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 shutdown_connections 2024-03-28T18:56:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f971c06bbd0 0x7f971c06e090 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f973c157780 0x7f973c1eee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f973c1467d0 0x7f973c1ee940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 --2- 172.21.15.27:0/277996843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9724003a20 0x7f973c157240 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.375+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 >> 172.21.15.27:0/277996843 conn(0x7f973c06ca00 msgr2=0x7f973c151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.376+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 shutdown_connections 2024-03-28T18:56:36.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.376+0000 7f9743a00640 1 -- 172.21.15.27:0/277996843 wait complete. 2024-03-28T18:56:36.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-03-28T18:56:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.605+0000 7f4720e00640 1 Processor -- start 2024-03-28T18:56:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.606+0000 7f4720e00640 1 -- start start 2024-03-28T18:56:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.607+0000 7f4720e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f471c066dd0 0x7f471c0671b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.607+0000 7f4720e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c0a6f20 0x7f471c1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.607+0000 7f4720e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c153720 con 0x7f471c072b80 2024-03-28T18:56:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.607+0000 7f4720e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c154720 con 0x7f471c066dd0 2024-03-28T18:56:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.607+0000 7f4720e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c151130 con 0x7f471c0a6f20 2024-03-28T18:56:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f4719e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f471c066dd0 0x7f471c0671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f4719e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f471c066dd0 0x7f471c0671b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58028/0 (socket says 172.21.15.27:58028) 2024-03-28T18:56:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f4719e00640 1 -- 172.21.15.27:0/745233001 learned_addr learned my addr 172.21.15.27:0/745233001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f471a800640 1 --2- 172.21.15.27:0/745233001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c0a6f20 0x7f471c1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f4719e00640 1 -- 172.21.15.27:0/745233001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c0a6f20 msgr2=0x7f471c1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f4719e00640 1 --2- 172.21.15.27:0/745233001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c0a6f20 0x7f471c1541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f4719e00640 1 -- 172.21.15.27:0/745233001 >> v1:172.21.15.27:6789/0 conn(0x7f471c072b80 legacy=0x7f471c0c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f4719e00640 1 -- 172.21.15.27:0/745233001 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c0a73b0 con 0x7f471c066dd0 2024-03-28T18:56:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.608+0000 7f4719400640 1 -- 172.21.15.27:0/745233001 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248652653 0 0) 0x7f471c153720 con 0x7f471c072b80 2024-03-28T18:56:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.610+0000 7f4719e00640 1 --2- 172.21.15.27:0/745233001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c066dd0 0x7f471c0671b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f470c009870 tx=0x7f470c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bffc3afb9370d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.610+0000 7f4719400640 1 -- 172.21.15.27:0/745233001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c035020 con 0x7f471c066dd0 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.610+0000 7f4719400640 1 -- 172.21.15.27:0/745233001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c031640 con 0x7f471c066dd0 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.610+0000 7f4719400640 1 -- 172.21.15.27:0/745233001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c02d040 con 0x7f471c066dd0 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.610+0000 7f4720e00640 1 -- 172.21.15.27:0/745233001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c066dd0 msgr2=0x7f471c0671b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.610+0000 7f4720e00640 1 --2- 172.21.15.27:0/745233001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c066dd0 0x7f471c0671b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f470c009870 tx=0x7f470c02b8f0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.611+0000 7f4720e00640 1 -- 172.21.15.27:0/745233001 shutdown_connections 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.611+0000 7f4720e00640 1 --2- 172.21.15.27:0/745233001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c0a6f20 0x7f471c1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.611+0000 7f4720e00640 1 --2- 172.21.15.27:0/745233001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c066dd0 0x7f471c0671b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.611+0000 7f4720e00640 1 -- 172.21.15.27:0/745233001 >> 172.21.15.27:0/745233001 conn(0x7f471c06b0c0 msgr2=0x7f471c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.611+0000 7f4720e00640 1 -- 172.21.15.27:0/745233001 shutdown_connections 2024-03-28T18:56:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.611+0000 7f4720e00640 1 -- 172.21.15.27:0/745233001 wait complete. 2024-03-28T18:56:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.612+0000 7f4720e00640 1 Processor -- start 2024-03-28T18:56:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.612+0000 7f4720e00640 1 -- start start 2024-03-28T18:56:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.613+0000 7f4720e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f471c066dd0 0x7f471c1ebc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.613+0000 7f4720e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c072b80 0x7f471c0be390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.613+0000 7f4720e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0a6f20 0x7f471c155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.613+0000 7f4720e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c1539e0 con 0x7f471c0a6f20 2024-03-28T18:56:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.613+0000 7f4720e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c157aa0 con 0x7f471c066dd0 2024-03-28T18:56:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.613+0000 7f4720e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c153cc0 con 0x7f471c072b80 2024-03-28T18:56:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f471a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0a6f20 0x7f471c155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f471a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0a6f20 0x7f471c155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58034/0 (socket says 172.21.15.27:58034) 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f471a800640 1 -- 172.21.15.27:0/1529886723 learned_addr learned my addr 172.21.15.27:0/1529886723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f4719e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f471c066dd0 0x7f471c1ebc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f4713e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c072b80 0x7f471c0be390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f471a800640 1 -- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c072b80 msgr2=0x7f471c0be390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f471a800640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c072b80 0x7f471c0be390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f471a800640 1 -- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f471c066dd0 msgr2=0x7f471c1ebc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f471a800640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f471c066dd0 0x7f471c1ebc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f471a800640 1 -- 172.21.15.27:0/1529886723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c1ec190 con 0x7f471c0a6f20 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.614+0000 7f4713e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c072b80 0x7f471c0be390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.615+0000 7f471a800640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0a6f20 0x7f471c155360 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4714007e90 tx=0x7f471400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.615+0000 7f4712000640 1 -- 172.21.15.27:0/1529886723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714014420 con 0x7f471c0a6f20 2024-03-28T18:56:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.615+0000 7f4712000640 1 -- 172.21.15.27:0/1529886723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4714019070 con 0x7f471c0a6f20 2024-03-28T18:56:36.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.615+0000 7f4712000640 1 -- 172.21.15.27:0/1529886723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714014800 con 0x7f471c0a6f20 2024-03-28T18:56:36.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.615+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c156310 con 0x7f471c0a6f20 2024-03-28T18:56:36.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.615+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c1568d0 con 0x7f471c0a6f20 2024-03-28T18:56:36.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.616+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c04f470 con 0x7f471c0a6f20 2024-03-28T18:56:36.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.617+0000 7f4712000640 1 -- 172.21.15.27:0/1529886723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f47140149a0 con 0x7f471c0a6f20 2024-03-28T18:56:36.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.618+0000 7f4712000640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46f806bbd0 0x7f46f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:36.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.618+0000 7f4712000640 1 -- 172.21.15.27:0/1529886723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f471408b650 con 0x7f471c0a6f20 2024-03-28T18:56:36.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.618+0000 7f4719e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46f806bbd0 0x7f46f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.619+0000 7f4719e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46f806bbd0 0x7f46f806e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f470c036000 tx=0x7f470c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:36.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.623+0000 7f4712000640 1 -- 172.21.15.27:0/1529886723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f471c1ec190 con 0x7f471c0a6f20 2024-03-28T18:56:36.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.823+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f471c0c7a70 con 0x7f471c0a6f20 2024-03-28T18:56:36.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.824+0000 7f4712000640 1 -- 172.21.15.27:0/1529886723 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v26) v1 ==== 99+0+11 (secure 0 0 0) 0x7f4714059d90 con 0x7f471c0a6f20 2024-03-28T18:56:36.824 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46f806bbd0 msgr2=0x7f46f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46f806bbd0 0x7f46f806e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f470c036000 tx=0x7f470c03c040 comp rx=0 tx=0).stop 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0a6f20 msgr2=0x7f471c155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0a6f20 0x7f471c155360 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4714007e90 tx=0x7f471400e4b0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 shutdown_connections 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46f806bbd0 0x7f46f806e090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0a6f20 0x7f471c155360 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f471c072b80 0x7f471c0be390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 --2- 172.21.15.27:0/1529886723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f471c066dd0 0x7f471c1ebc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:36.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 >> 172.21.15.27:0/1529886723 conn(0x7f471c06b0c0 msgr2=0x7f471c0a6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:36.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 shutdown_connections 2024-03-28T18:56:36.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:36.827+0000 7f4720e00640 1 -- 172.21.15.27:0/1529886723 wait complete. 2024-03-28T18:56:36.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-03-28T18:56:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.063+0000 7f46d3400640 1 Processor -- start 2024-03-28T18:56:37.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.063+0000 7f46d3400640 1 -- start start 2024-03-28T18:56:37.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.065+0000 7f46d3400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14c640 0x7f46cc153860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.065+0000 7f46d3400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc0c1b60 con 0x7f46cc0c37b0 2024-03-28T18:56:37.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.065+0000 7f46d3400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc0c1e40 con 0x7f46cc152370 2024-03-28T18:56:37.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.065+0000 7f46d3400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0730b0 con 0x7f46cc14c640 2024-03-28T18:56:37.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.065+0000 7f46d0a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f46cc0c37b0 0x7f46cc0c3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35178/0 (socket says 172.21.15.27:35178) 2024-03-28T18:56:37.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.065+0000 7f46d0a00640 1 -- 172.21.15.27:0/1901891559 learned_addr learned my addr 172.21.15.27:0/1901891559 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:37.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.065+0000 7f46d1400640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14c640 0x7f46cc153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.065+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146077008 0 0) 0x7f46cc0c1b60 con 0x7f46cc0c37b0 2024-03-28T18:56:37.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b4003520 con 0x7f46cc0c37b0 2024-03-28T18:56:37.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937104842 0 0) 0x7f46b4003520 con 0x7f46cc0c37b0 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14c640 msgr2=0x7f46cc153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14c640 0x7f46cc153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 >> v1:172.21.15.155:6789/0 conn(0x7f46cc152370 legacy=0x7f46cc14bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc0c2120 con 0x7f46cc0c37b0 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f46b8001690 con 0x7f46cc0c37b0 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 >> v1:172.21.15.27:6789/0 conn(0x7f46cc0c37b0 legacy=0x7f46cc0c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46b4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46b4004800 0x7f46b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b4005180 0x7f46b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0730b0 con 0x7f46b4003a50 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b400bfb0 con 0x7f46b4005180 2024-03-28T18:56:37.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b400c1b0 con 0x7f46b4004800 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.066+0000 7f46d0a00640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46b4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.067+0000 7f46d1400640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b4005180 0x7f46b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.067+0000 7f46d0a00640 1 -- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46b4004800 msgr2=0x7f46b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.067+0000 7f46d0a00640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46b4004800 0x7f46b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.067+0000 7f46d0a00640 1 -- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b4005180 msgr2=0x7f46b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.067+0000 7f46d0a00640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b4005180 0x7f46b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.067+0000 7f46d0a00640 1 -- 172.21.15.27:0/1901891559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b4003520 con 0x7f46b4003a50 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.067+0000 7f46d1400640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b4005180 0x7f46b400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.068+0000 7f46d0a00640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46b4003e70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f46b800c210 tx=0x7f46b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dad59f0c924bff2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.068+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8007d10 con 0x7f46b4003a50 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.068+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b8030070 con 0x7f46b4003a50 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.068+0000 7f46cb400640 1 -- 172.21.15.27:0/1901891559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8005d30 con 0x7f46b4003a50 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.069+0000 7f46d3400640 1 -- 172.21.15.27:0/1901891559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 msgr2=0x7f46b4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.069+0000 7f46d3400640 1 --2- 172.21.15.27:0/1901891559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46b4003e70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f46b800c210 tx=0x7f46b4005830 comp rx=0 tx=0).stop 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.069+0000 7f46d1400640 1 -- 172.21.15.27:0/1901891559 reap_dead start 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.069+0000 7f46d3400640 1 -- 172.21.15.27:0/1901891559 shutdown_connections 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.069+0000 7f46d3400640 1 -- 172.21.15.27:0/1901891559 >> 172.21.15.27:0/1901891559 conn(0x7f46cc06ca00 msgr2=0x7f46cc06cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.069+0000 7f46d3400640 1 -- 172.21.15.27:0/1901891559 shutdown_connections 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.069+0000 7f46d3400640 1 -- 172.21.15.27:0/1901891559 wait complete. 2024-03-28T18:56:37.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.071+0000 7f46d3400640 1 Processor -- start 2024-03-28T18:56:37.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.071+0000 7f46d3400640 1 -- start start 2024-03-28T18:56:37.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46cc1514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d3400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14ace0 0x7f46cc1499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46cc149f00 0x7f46cc141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d3400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0c2c00 con 0x7f46b4003a50 2024-03-28T18:56:37.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d3400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0c26e0 con 0x7f46cc149f00 2024-03-28T18:56:37.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d3400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0c2d80 con 0x7f46cc14ace0 2024-03-28T18:56:37.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46cc1514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46cc1514a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58058/0 (socket says 172.21.15.27:58058) 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.072+0000 7f46d0a00640 1 -- 172.21.15.27:0/1620158492 learned_addr learned my addr 172.21.15.27:0/1620158492 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46cbe00640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14ace0 0x7f46cc1499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46d1400640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46cc149f00 0x7f46cc141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46d0a00640 1 -- 172.21.15.27:0/1620158492 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14ace0 msgr2=0x7f46cc1499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46d0a00640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14ace0 0x7f46cc1499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46d0a00640 1 -- 172.21.15.27:0/1620158492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46cc149f00 msgr2=0x7f46cc141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46d0a00640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46cc149f00 0x7f46cc141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46d0a00640 1 -- 172.21.15.27:0/1620158492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc14b4c0 con 0x7f46b4003a50 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46d0a00640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46cc1514a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f46b800c330 tx=0x7f46cc0c2010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.073+0000 7f46c8c00640 1 -- 172.21.15.27:0/1620158492 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8045070 con 0x7f46b4003a50 2024-03-28T18:56:37.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.074+0000 7f46c8c00640 1 -- 172.21.15.27:0/1620158492 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b8000ed0 con 0x7f46b4003a50 2024-03-28T18:56:37.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.074+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc1ee4a0 con 0x7f46b4003a50 2024-03-28T18:56:37.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.074+0000 7f46c8c00640 1 -- 172.21.15.27:0/1620158492 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8001c50 con 0x7f46b4003a50 2024-03-28T18:56:37.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.074+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc1ee740 con 0x7f46b4003a50 2024-03-28T18:56:37.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.075+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc04f470 con 0x7f46b4003a50 2024-03-28T18:56:37.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.076+0000 7f46c8c00640 1 -- 172.21.15.27:0/1620158492 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f46b8007070 con 0x7f46b4003a50 2024-03-28T18:56:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.077+0000 7f46c8c00640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46ac06bbd0 0x7f46ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.078+0000 7f46c8c00640 1 -- 172.21.15.27:0/1620158492 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f46b80abd50 con 0x7f46b4003a50 2024-03-28T18:56:37.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.078+0000 7f46cbe00640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46ac06bbd0 0x7f46ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.078+0000 7f46cbe00640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46ac06bbd0 0x7f46ac06e090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46cc142bc0 tx=0x7f46c0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:37.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.082+0000 7f46c8c00640 1 -- 172.21.15.27:0/1620158492 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f46b8079fc0 con 0x7f46b4003a50 2024-03-28T18:56:37.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.377+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f46cc071cc0 con 0x7f46b4003a50 2024-03-28T18:56:37.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.378+0000 7f46c8c00640 1 -- 172.21.15.27:0/1620158492 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v26) v1 ==== 95+0+11 (secure 0 0 0) 0x7f46b807e060 con 0x7f46b4003a50 2024-03-28T18:56:37.378 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46ac06bbd0 msgr2=0x7f46ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46ac06bbd0 0x7f46ac06e090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f46cc142bc0 tx=0x7f46c0009040 comp rx=0 tx=0).stop 2024-03-28T18:56:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 msgr2=0x7f46cc1514a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46cc1514a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f46b800c330 tx=0x7f46cc0c2010 comp rx=0 tx=0).stop 2024-03-28T18:56:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 shutdown_connections 2024-03-28T18:56:37.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f46ac06bbd0 0x7f46ac06e090 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46cc149f00 0x7f46cc141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f46cc14ace0 0x7f46cc1499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 --2- 172.21.15.27:0/1620158492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4003a50 0x7f46cc1514a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 >> 172.21.15.27:0/1620158492 conn(0x7f46cc06ca00 msgr2=0x7f46cc0715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 shutdown_connections 2024-03-28T18:56:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.381+0000 7f46d3400640 1 -- 172.21.15.27:0/1620158492 wait complete. 2024-03-28T18:56:37.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-03-28T18:56:37.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.618+0000 7fb6dc000640 1 Processor -- start 2024-03-28T18:56:37.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.618+0000 7fb6dc000640 1 -- start start 2024-03-28T18:56:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.620+0000 7fb6dc000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d4159670 0x7fb6d40c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.620+0000 7fb6dc000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d4067610 con 0x7fb6d40671b0 2024-03-28T18:56:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.620+0000 7fb6dc000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d4155970 con 0x7fb6d4066dd0 2024-03-28T18:56:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.620+0000 7fb6dc000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d40a74b0 con 0x7fb6d4159670 2024-03-28T18:56:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.620+0000 7fb6da000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb6d40671b0 0x7fb6d414ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35184/0 (socket says 172.21.15.27:35184) 2024-03-28T18:56:37.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.620+0000 7fb6da000640 1 -- 172.21.15.27:0/3055795525 learned_addr learned my addr 172.21.15.27:0/3055795525 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:37.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d8c00640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d4159670 0x7fb6d40c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953018436 0 0) 0x7fb6d4067610 con 0x7fb6d40671b0 2024-03-28T18:56:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6bc003520 con 0x7fb6d40671b0 2024-03-28T18:56:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3292187651 0 0) 0x7fb6bc003520 con 0x7fb6d40671b0 2024-03-28T18:56:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d4159670 msgr2=0x7fb6d40c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d4159670 0x7fb6d40c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 >> v1:172.21.15.155:6789/0 conn(0x7fb6d4066dd0 legacy=0x7fb6d4158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d4155c30 con 0x7fb6d40671b0 2024-03-28T18:56:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fb6cc001670 con 0x7fb6d40671b0 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 >> v1:172.21.15.27:6789/0 conn(0x7fb6d40671b0 legacy=0x7fb6d414ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.621+0000 7fb6d3e00640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6bc003a50 0x7fb6bc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d3e00640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc004800 0x7fb6bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d3e00640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d4159670 0x7fb6bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d40a74b0 con 0x7fb6bc003a50 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc00bac0 con 0x7fb6bc004800 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc00bcc0 con 0x7fb6d4159670 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d9600640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6bc003a50 0x7fb6bc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d8c00640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc004800 0x7fb6bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d9600640 1 -- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d4159670 msgr2=0x7fb6bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d9600640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d4159670 0x7fb6bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d9600640 1 -- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc004800 msgr2=0x7fb6bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d9600640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc004800 0x7fb6bc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.622+0000 7fb6d9600640 1 -- 172.21.15.27:0/3055795525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc003520 con 0x7fb6bc003a50 2024-03-28T18:56:37.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.624+0000 7fb6d9600640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6bc003a50 0x7fb6bc003e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8001e50 tx=0x7fb6bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628f2dfec1ee3468 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:37.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.624+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8001a70 con 0x7fb6bc003a50 2024-03-28T18:56:37.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.624+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c8003b70 con 0x7fb6bc003a50 2024-03-28T18:56:37.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.624+0000 7fb6d3e00640 1 -- 172.21.15.27:0/3055795525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80050d0 con 0x7fb6bc003a50 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.624+0000 7fb6dc000640 1 -- 172.21.15.27:0/3055795525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6bc003a50 msgr2=0x7fb6bc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.624+0000 7fb6dc000640 1 --2- 172.21.15.27:0/3055795525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6bc003a50 0x7fb6bc003e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8001e50 tx=0x7fb6bc005830 comp rx=0 tx=0).stop 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.624+0000 7fb6da000640 1 -- 172.21.15.27:0/3055795525 reap_dead start 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.625+0000 7fb6dc000640 1 -- 172.21.15.27:0/3055795525 shutdown_connections 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.625+0000 7fb6dc000640 1 -- 172.21.15.27:0/3055795525 >> 172.21.15.27:0/3055795525 conn(0x7fb6d406b0c0 msgr2=0x7fb6d406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.625+0000 7fb6dc000640 1 -- 172.21.15.27:0/3055795525 shutdown_connections 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.625+0000 7fb6dc000640 1 -- 172.21.15.27:0/3055795525 wait complete. 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.626+0000 7fb6dc000640 1 Processor -- start 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.627+0000 7fb6dc000640 1 -- start start 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.627+0000 7fb6dc000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc003a50 0x7fb6d4156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.627+0000 7fb6dc000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d41efe30 0x7fb6d406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.627+0000 7fb6dc000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6d4157340 0x7fb6d41ebc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.627+0000 7fb6dc000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d40c7500 con 0x7fb6d4157340 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.627+0000 7fb6dc000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d41561f0 con 0x7fb6bc003a50 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.627+0000 7fb6dc000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d40a70d0 con 0x7fb6d41efe30 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6da000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6d4157340 0x7fb6d41ebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6da000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6d4157340 0x7fb6d41ebc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58068/0 (socket says 172.21.15.27:58068) 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6da000640 1 -- 172.21.15.27:0/2904294631 learned_addr learned my addr 172.21.15.27:0/2904294631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6d9600640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc003a50 0x7fb6d4156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6d8c00640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d41efe30 0x7fb6d406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6da000640 1 -- 172.21.15.27:0/2904294631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d41efe30 msgr2=0x7fb6d406ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6da000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d41efe30 0x7fb6d406ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6da000640 1 -- 172.21.15.27:0/2904294631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc003a50 msgr2=0x7fb6d4156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6da000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc003a50 0x7fb6d4156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.628+0000 7fb6da000640 1 -- 172.21.15.27:0/2904294631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d40ad890 con 0x7fb6d4157340 2024-03-28T18:56:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.629+0000 7fb6da000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6d4157340 0x7fb6d41ebc40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb6d40718e0 tx=0x7fb6cc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.629+0000 7fb6d1600640 1 -- 172.21.15.27:0/2904294631 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc007440 con 0x7fb6d4157340 2024-03-28T18:56:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.629+0000 7fb6d1600640 1 -- 172.21.15.27:0/2904294631 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc0075e0 con 0x7fb6d4157340 2024-03-28T18:56:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.629+0000 7fb6d1600640 1 -- 172.21.15.27:0/2904294631 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc004a20 con 0x7fb6d4157340 2024-03-28T18:56:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.629+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d41ef350 con 0x7fb6d4157340 2024-03-28T18:56:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.629+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d41ef910 con 0x7fb6d4157340 2024-03-28T18:56:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.630+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d41efb10 con 0x7fb6d4157340 2024-03-28T18:56:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.633+0000 7fb6d1600640 1 -- 172.21.15.27:0/2904294631 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb6cc004bc0 con 0x7fb6d4157340 2024-03-28T18:56:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.634+0000 7fb6d1600640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb6b406bc00 0x7fb6b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.634+0000 7fb6d1600640 1 -- 172.21.15.27:0/2904294631 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fb6cc08a500 con 0x7fb6d4157340 2024-03-28T18:56:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.634+0000 7fb6d9600640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb6b406bc00 0x7fb6b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.635+0000 7fb6d9600640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb6b406bc00 0x7fb6b406e0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8034000 tx=0x7fb6c8009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.638+0000 7fb6d1600640 1 -- 172.21.15.27:0/2904294631 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb6d41efb10 con 0x7fb6d4157340 2024-03-28T18:56:37.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.839+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb6d404f470 con 0x7fb6d4157340 2024-03-28T18:56:37.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.840+0000 7fb6d1600640 1 -- 172.21.15.27:0/2904294631 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v26) v1 ==== 99+0+11 (secure 0 0 0) 0x7fb6cc01b030 con 0x7fb6d4157340 2024-03-28T18:56:37.840 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-03-28T18:56:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb6b406bc00 msgr2=0x7fb6b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb6b406bc00 0x7fb6b406e0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8034000 tx=0x7fb6c8009030 comp rx=0 tx=0).stop 2024-03-28T18:56:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6d4157340 msgr2=0x7fb6d41ebc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6d4157340 0x7fb6d41ebc40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb6d40718e0 tx=0x7fb6cc00c960 comp rx=0 tx=0).stop 2024-03-28T18:56:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 shutdown_connections 2024-03-28T18:56:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb6b406bc00 0x7fb6b406e0c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6d4157340 0x7fb6d41ebc40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb6d41efe30 0x7fb6d406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 --2- 172.21.15.27:0/2904294631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6bc003a50 0x7fb6d4156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 >> 172.21.15.27:0/2904294631 conn(0x7fb6d406b0c0 msgr2=0x7fb6d4153280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 shutdown_connections 2024-03-28T18:56:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:37.843+0000 7fb6dc000640 1 -- 172.21.15.27:0/2904294631 wait complete. 2024-03-28T18:56:37.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-03-28T18:56:38.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.070+0000 7f663be00640 1 Processor -- start 2024-03-28T18:56:38.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.071+0000 7f663be00640 1 -- start start 2024-03-28T18:56:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0c5aa0 0x7f663c156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663be00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f663c0c8db0 0x7f663c072620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663be00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c0735f0 con 0x7f663c072b60 2024-03-28T18:56:38.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663be00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663c1565d0 con 0x7f663c0c8db0 2024-03-28T18:56:38.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0c82d0 con 0x7f663c0c5aa0 2024-03-28T18:56:38.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f663c072b60 0x7f663c152840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35198/0 (socket says 172.21.15.27:35198) 2024-03-28T18:56:38.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663b400640 1 -- 172.21.15.27:0/233910230 learned_addr learned my addr 172.21.15.27:0/233910230 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663aa00640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0c5aa0 0x7f663c156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.072+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623732448 0 0) 0x7f663c0735f0 con 0x7f663c072b60 2024-03-28T18:56:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6620003520 con 0x7f663c072b60 2024-03-28T18:56:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f6633e00640 1 --2- 172.21.15.27:0/233910230 >> v2:172.21.15.155:3300/0 conn(0x7f663c0c8db0 0x7f663c072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013097915 0 0) 0x7f6620003520 con 0x7f663c072b60 2024-03-28T18:56:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0c5aa0 msgr2=0x7f663c156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0c5aa0 0x7f663c156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 >> v2:172.21.15.155:3300/0 conn(0x7f663c0c8db0 msgr2=0x7f663c072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 --2- 172.21.15.27:0/233910230 >> v2:172.21.15.155:3300/0 conn(0x7f663c0c8db0 0x7f663c072620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0c7d10 con 0x7f663c072b60 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f66340019f0 con 0x7f663c072b60 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 >> v1:172.21.15.27:6789/0 conn(0x7f663c072b60 legacy=0x7f663c152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0c5aa0 0x7f66200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f6620004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6620004d40 0x7f66200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c1565d0 con 0x7f6620003f80 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0c82d0 con 0x7f6620004d40 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.073+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662000baf0 con 0x7f663c0c5aa0 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f6633e00640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f6620004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f663aa00640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0c5aa0 0x7f66200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f663b400640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6620004d40 0x7f66200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f6633e00640 1 -- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0c5aa0 msgr2=0x7f66200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f6633e00640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0c5aa0 0x7f66200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f6633e00640 1 -- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6620004d40 msgr2=0x7f66200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f6633e00640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6620004d40 0x7f66200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f6633e00640 1 -- 172.21.15.27:0/233910230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6620003520 con 0x7f6620003f80 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.074+0000 7f663b400640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6620004d40 0x7f66200051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.075+0000 7f6633e00640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f6620004800 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6624009ac0 tx=0x7f6620005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca25659091cbad1c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.075+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624039070 con 0x7f6620003f80 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.075+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662402c710 con 0x7f6620003f80 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.075+0000 7f663a000640 1 -- 172.21.15.27:0/233910230 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624009690 con 0x7f6620003f80 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.076+0000 7f663be00640 1 -- 172.21.15.27:0/233910230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 msgr2=0x7f6620004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.076+0000 7f663be00640 1 --2- 172.21.15.27:0/233910230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f6620004800 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6624009ac0 tx=0x7f6620005830 comp rx=0 tx=0).stop 2024-03-28T18:56:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.076+0000 7f663b400640 1 -- 172.21.15.27:0/233910230 reap_dead start 2024-03-28T18:56:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.076+0000 7f663be00640 1 -- 172.21.15.27:0/233910230 shutdown_connections 2024-03-28T18:56:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.076+0000 7f663be00640 1 -- 172.21.15.27:0/233910230 >> 172.21.15.27:0/233910230 conn(0x7f663c06b0c0 msgr2=0x7f663c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.076+0000 7f663be00640 1 -- 172.21.15.27:0/233910230 shutdown_connections 2024-03-28T18:56:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.076+0000 7f663be00640 1 -- 172.21.15.27:0/233910230 wait complete. 2024-03-28T18:56:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.078+0000 7f663be00640 1 Processor -- start 2024-03-28T18:56:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.078+0000 7f663be00640 1 -- start start 2024-03-28T18:56:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f663c1ec030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0ac4b0 0x7f663c0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c156980 0x7f663c1eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c072100 con 0x7f6620003f80 2024-03-28T18:56:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c071e20 con 0x7f663c156980 2024-03-28T18:56:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c152f50 con 0x7f663c0ac4b0 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f663c1ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f663c1ec030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58084/0 (socket says 172.21.15.27:58084) 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.079+0000 7f663aa00640 1 -- 172.21.15.27:0/1988221366 learned_addr learned my addr 172.21.15.27:0/1988221366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f6633e00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0ac4b0 0x7f663c0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663aa00640 1 -- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0ac4b0 msgr2=0x7f663c0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663b400640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c156980 0x7f663c1eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663aa00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0ac4b0 0x7f663c0c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663aa00640 1 -- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c156980 msgr2=0x7f663c1eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663aa00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c156980 0x7f663c1eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663aa00640 1 -- 172.21.15.27:0/1988221366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c1eb870 con 0x7f6620003f80 2024-03-28T18:56:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f6633e00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0ac4b0 0x7f663c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663b400640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c156980 0x7f663c1eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663aa00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f663c1ec030 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f662c011600 tx=0x7f662c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f6632a00640 1 -- 172.21.15.27:0/1988221366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c00f920 con 0x7f6620003f80 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f6632a00640 1 -- 172.21.15.27:0/1988221366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c00fac0 con 0x7f6620003f80 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f6632a00640 1 -- 172.21.15.27:0/1988221366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c00fe70 con 0x7f6620003f80 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663c1ee260 con 0x7f6620003f80 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.080+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c1eefa0 con 0x7f6620003f80 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.082+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c04f400 con 0x7f6620003f80 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.082+0000 7f6632a00640 1 -- 172.21.15.27:0/1988221366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f662c015460 con 0x7f6620003f80 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.083+0000 7f6632a00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f661806bc00 0x7f661806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.083+0000 7f6632a00640 1 -- 172.21.15.27:0/1988221366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f662c08b6d0 con 0x7f6620003f80 2024-03-28T18:56:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.083+0000 7f6633e00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f661806bc00 0x7f661806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.084+0000 7f6633e00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f661806bc00 0x7f661806e0c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6624036000 tx=0x7f6624030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:38.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.091+0000 7f6632a00640 1 -- 172.21.15.27:0/1988221366 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f662c019030 con 0x7f6620003f80 2024-03-28T18:56:38.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.292+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f663c0732f0 con 0x7f6620003f80 2024-03-28T18:56:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.293+0000 7f6632a00640 1 -- 172.21.15.27:0/1988221366 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v26) v1 ==== 95+0+11 (secure 0 0 0) 0x7f662c059e10 con 0x7f6620003f80 2024-03-28T18:56:38.292 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-03-28T18:56:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f661806bc00 msgr2=0x7f661806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f661806bc00 0x7f661806e0c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6624036000 tx=0x7f6624030000 comp rx=0 tx=0).stop 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 msgr2=0x7f663c1ec030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f663c1ec030 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f662c011600 tx=0x7f662c011ac0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 shutdown_connections 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f661806bc00 0x7f661806e0c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c156980 0x7f663c1eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0ac4b0 0x7f663c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 --2- 172.21.15.27:0/1988221366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6620003f80 0x7f663c1ec030 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.296+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 >> 172.21.15.27:0/1988221366 conn(0x7f663c06b0c0 msgr2=0x7f663c154df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:38.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.297+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 shutdown_connections 2024-03-28T18:56:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.297+0000 7f663be00640 1 -- 172.21.15.27:0/1988221366 wait complete. 2024-03-28T18:56:38.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:56:38.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.528+0000 7f383d600640 1 Processor -- start 2024-03-28T18:56:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.528+0000 7f383d600640 1 -- start start 2024-03-28T18:56:38.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f383d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3838152ea0 0x7f383806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f383d600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38380627e0 0x7f38380c07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f383d600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38380ab630 con 0x7f38380ac6e0 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f383d600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38380abc90 con 0x7f3838152ea0 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f383d600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3838153280 con 0x7f38380627e0 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f3836a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3838152ea0 0x7f383806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f3836a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3838152ea0 0x7f383806f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58092/0 (socket says 172.21.15.27:58092) 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f3836a00640 1 -- 172.21.15.27:0/1332811146 learned_addr learned my addr 172.21.15.27:0/1332811146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.530+0000 7f3836000640 1 --2- 172.21.15.27:0/1332811146 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38380627e0 0x7f38380c07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.531+0000 7f3836a00640 1 -- 172.21.15.27:0/1332811146 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38380627e0 msgr2=0x7f38380c07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.531+0000 7f3836a00640 1 --2- 172.21.15.27:0/1332811146 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38380627e0 0x7f38380c07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.531+0000 7f3836a00640 1 -- 172.21.15.27:0/1332811146 >> v1:172.21.15.155:6789/0 conn(0x7f38380ac6e0 legacy=0x7f38381557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:38.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.531+0000 7f3836a00640 1 -- 172.21.15.27:0/1332811146 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38380ab910 con 0x7f3838152ea0 2024-03-28T18:56:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.532+0000 7f3836a00640 1 --2- 172.21.15.27:0/1332811146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3838152ea0 0x7f383806f550 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3828009870 tx=0x7f382802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa95440050f5bcd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.532+0000 7f3835600640 1 -- 172.21.15.27:0/1332811146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828035020 con 0x7f3838152ea0 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.532+0000 7f3835600640 1 -- 172.21.15.27:0/1332811146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828031640 con 0x7f3838152ea0 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.532+0000 7f3835600640 1 -- 172.21.15.27:0/1332811146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382802d040 con 0x7f3838152ea0 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.533+0000 7f383d600640 1 -- 172.21.15.27:0/1332811146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3838152ea0 msgr2=0x7f383806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.533+0000 7f383d600640 1 --2- 172.21.15.27:0/1332811146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3838152ea0 0x7f383806f550 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3828009870 tx=0x7f382802b8f0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.533+0000 7f383d600640 1 -- 172.21.15.27:0/1332811146 shutdown_connections 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.533+0000 7f383d600640 1 --2- 172.21.15.27:0/1332811146 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38380627e0 0x7f38380c07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.533+0000 7f383d600640 1 --2- 172.21.15.27:0/1332811146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3838152ea0 0x7f383806f550 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.533+0000 7f383d600640 1 -- 172.21.15.27:0/1332811146 >> 172.21.15.27:0/1332811146 conn(0x7f383806b0c0 msgr2=0x7f383806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.533+0000 7f383d600640 1 -- 172.21.15.27:0/1332811146 shutdown_connections 2024-03-28T18:56:38.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.533+0000 7f383d600640 1 -- 172.21.15.27:0/1332811146 wait complete. 2024-03-28T18:56:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.535+0000 7f383d600640 1 Processor -- start 2024-03-28T18:56:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.535+0000 7f383d600640 1 -- start start 2024-03-28T18:56:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.536+0000 7f383d600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38380627e0 0x7f383814e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.536+0000 7f383d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38380ac6e0 0x7f38381e2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.536+0000 7f383d600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3838152ea0 0x7f38380726e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.536+0000 7f383d600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383806fa90 con 0x7f38380ac6e0 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.536+0000 7f383d600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38380ac1c0 con 0x7f38380627e0 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.536+0000 7f383d600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3838155ef0 con 0x7f3838152ea0 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38380ac6e0 0x7f38381e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38380ac6e0 0x7f38381e2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58104/0 (socket says 172.21.15.27:58104) 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836000640 1 -- 172.21.15.27:0/2333891516 learned_addr learned my addr 172.21.15.27:0/2333891516 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836a00640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38380627e0 0x7f383814e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3837400640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3838152ea0 0x7f38380726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836000640 1 -- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3838152ea0 msgr2=0x7f38380726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836000640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3838152ea0 0x7f38380726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836000640 1 -- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38380627e0 msgr2=0x7f383814e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836000640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38380627e0 0x7f383814e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836000640 1 -- 172.21.15.27:0/2333891516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38381ef3b0 con 0x7f38380ac6e0 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3836a00640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38380627e0 0x7f383814e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.537+0000 7f3837400640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3838152ea0 0x7f38380726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.538+0000 7f3836000640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38380ac6e0 0x7f38381e2400 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f382400d620 tx=0x7f382400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.538+0000 7f382ea00640 1 -- 172.21.15.27:0/2333891516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382400b620 con 0x7f38380ac6e0 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.538+0000 7f382ea00640 1 -- 172.21.15.27:0/2333891516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3824019070 con 0x7f38380ac6e0 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.538+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38381ef690 con 0x7f38380ac6e0 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.538+0000 7f382ea00640 1 -- 172.21.15.27:0/2333891516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382400ba00 con 0x7f38380ac6e0 2024-03-28T18:56:38.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.538+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38381efc50 con 0x7f38380ac6e0 2024-03-28T18:56:38.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.539+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383806c050 con 0x7f38380ac6e0 2024-03-28T18:56:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.540+0000 7f382ea00640 1 -- 172.21.15.27:0/2333891516 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f382400bba0 con 0x7f38380ac6e0 2024-03-28T18:56:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.541+0000 7f382ea00640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f381806bc20 0x7f381806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.541+0000 7f382ea00640 1 -- 172.21.15.27:0/2333891516 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f382408b1d0 con 0x7f38380ac6e0 2024-03-28T18:56:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.541+0000 7f3836a00640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f381806bc20 0x7f381806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:38.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.542+0000 7f3836a00640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f381806bc20 0x7f381806e0e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3828036000 tx=0x7f382803c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:38.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.546+0000 7f382ea00640 1 -- 172.21.15.27:0/2333891516 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3824059990 con 0x7f38380ac6e0 2024-03-28T18:56:38.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.898+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f383804f400 con 0x7f38380ac6e0 2024-03-28T18:56:38.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.900+0000 7f382ea00640 1 -- 172.21.15.27:0/2333891516 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f3824059290 con 0x7f38380ac6e0 2024-03-28T18:56:38.900 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:38.900 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:56:38.901 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:56:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f381806bc20 msgr2=0x7f381806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f381806bc20 0x7f381806e0e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3828036000 tx=0x7f382803c040 comp rx=0 tx=0).stop 2024-03-28T18:56:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38380ac6e0 msgr2=0x7f38381e2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:38.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38380ac6e0 0x7f38381e2400 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f382400d620 tx=0x7f382400dae0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 shutdown_connections 2024-03-28T18:56:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f381806bc20 0x7f381806e0e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3838152ea0 0x7f38380726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38380ac6e0 0x7f38381e2400 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 --2- 172.21.15.27:0/2333891516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38380627e0 0x7f383814e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 >> 172.21.15.27:0/2333891516 conn(0x7f383806b0c0 msgr2=0x7f38380c2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 shutdown_connections 2024-03-28T18:56:38.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:38.903+0000 7f383d600640 1 -- 172.21.15.27:0/2333891516 wait complete. 2024-03-28T18:56:38.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:56:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.142+0000 7f964e600640 1 Processor -- start 2024-03-28T18:56:39.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.142+0000 7f964e600640 1 -- start start 2024-03-28T18:56:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f964e600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c60e0 0x7f964806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:39.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f964e600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f96480c3ea0 0x7f964806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:39.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f964e600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96480c1600 con 0x7f9648072df0 2024-03-28T18:56:39.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f964e600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96480c23b0 con 0x7f96480c3ea0 2024-03-28T18:56:39.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f964e600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480c18e0 con 0x7f96480c60e0 2024-03-28T18:56:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f963ea00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f96480c3ea0 0x7f964806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f9647400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c60e0 0x7f964806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f9647e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f9648072df0 0x7f9648158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:47844/0 (socket says 172.21.15.27:47844) 2024-03-28T18:56:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.144+0000 7f9647e00640 1 -- 172.21.15.27:0/161217519 learned_addr learned my addr 172.21.15.27:0/161217519 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:56:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.145+0000 7f9647400640 1 -- 172.21.15.27:0/161217519 >> v2:172.21.15.155:3300/0 conn(0x7f96480c3ea0 msgr2=0x7f964806ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:56:39.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.145+0000 7f9647400640 1 --2- 172.21.15.27:0/161217519 >> v2:172.21.15.155:3300/0 conn(0x7f96480c3ea0 0x7f964806ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.145+0000 7f9647400640 1 -- 172.21.15.27:0/161217519 >> v1:172.21.15.155:6789/0 conn(0x7f9648072df0 legacy=0x7f9648158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.145+0000 7f9647400640 1 -- 172.21.15.27:0/161217519 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480c1bc0 con 0x7f96480c60e0 2024-03-28T18:56:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.145+0000 7f963ea00640 1 --2- 172.21.15.27:0/161217519 >> v2:172.21.15.155:3300/0 conn(0x7f96480c3ea0 0x7f964806ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:56:39.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.147+0000 7f9647400640 1 --2- 172.21.15.27:0/161217519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c60e0 0x7f964806e6f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96380097a0 tx=0x7f963802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa978074d6ba764 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.147+0000 7f9646a00640 1 -- 172.21.15.27:0/161217519 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638038070 con 0x7f96480c60e0 2024-03-28T18:56:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.147+0000 7f9646a00640 1 -- 172.21.15.27:0/161217519 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9638031700 con 0x7f96480c60e0 2024-03-28T18:56:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.147+0000 7f9646a00640 1 -- 172.21.15.27:0/161217519 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638031ad0 con 0x7f96480c60e0 2024-03-28T18:56:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.148+0000 7f964e600640 1 -- 172.21.15.27:0/161217519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c60e0 msgr2=0x7f964806e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.148+0000 7f964e600640 1 --2- 172.21.15.27:0/161217519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c60e0 0x7f964806e6f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96380097a0 tx=0x7f963802b7a0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.148+0000 7f964e600640 1 -- 172.21.15.27:0/161217519 shutdown_connections 2024-03-28T18:56:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.148+0000 7f964e600640 1 --2- 172.21.15.27:0/161217519 >> v2:172.21.15.155:3300/0 conn(0x7f96480c3ea0 0x7f964806ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.148+0000 7f964e600640 1 --2- 172.21.15.27:0/161217519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c60e0 0x7f964806e6f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.148+0000 7f964e600640 1 -- 172.21.15.27:0/161217519 >> 172.21.15.27:0/161217519 conn(0x7f964806b9d0 msgr2=0x7f96480c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.148+0000 7f964e600640 1 -- 172.21.15.27:0/161217519 shutdown_connections 2024-03-28T18:56:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.148+0000 7f964e600640 1 -- 172.21.15.27:0/161217519 wait complete. 2024-03-28T18:56:39.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.150+0000 7f964e600640 1 Processor -- start 2024-03-28T18:56:39.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.150+0000 7f964e600640 1 -- start start 2024-03-28T18:56:39.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f964e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9648072df0 0x7f9648153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:39.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f964e600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c3ea0 0x7f96480a80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:39.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f964e600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96480c60e0 0x7f96481ee470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:39.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f964e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480c6640 con 0x7f9648072df0 2024-03-28T18:56:39.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f964e600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480c64c0 con 0x7f96480c60e0 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f964e600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480c4f00 con 0x7f96480c3ea0 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f9647400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9648072df0 0x7f9648153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f9647400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9648072df0 0x7f9648153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58118/0 (socket says 172.21.15.27:58118) 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f9647400640 1 -- 172.21.15.27:0/3456272459 learned_addr learned my addr 172.21.15.27:0/3456272459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f963ea00640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c3ea0 0x7f96480a80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.151+0000 7f9647e00640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96480c60e0 0x7f96481ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f9647400640 1 -- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c3ea0 msgr2=0x7f96480a80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f9647400640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c3ea0 0x7f96480a80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f9647400640 1 -- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96480c60e0 msgr2=0x7f96481ee470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f9647400640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96480c60e0 0x7f96481ee470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f9647400640 1 -- 172.21.15.27:0/3456272459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96481e2560 con 0x7f9648072df0 2024-03-28T18:56:39.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f9647e00640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96480c60e0 0x7f96481ee470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:56:39.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f9647400640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9648072df0 0x7f9648153290 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9638000c00 tx=0x7f96380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:39.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f963fe00640 1 -- 172.21.15.27:0/3456272459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638042070 con 0x7f9648072df0 2024-03-28T18:56:39.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f963fe00640 1 -- 172.21.15.27:0/3456272459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963803d870 con 0x7f9648072df0 2024-03-28T18:56:39.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f963fe00640 1 -- 172.21.15.27:0/3456272459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638038040 con 0x7f9648072df0 2024-03-28T18:56:39.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96481e2840 con 0x7f9648072df0 2024-03-28T18:56:39.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.152+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96480aae40 con 0x7f9648072df0 2024-03-28T18:56:39.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.154+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964804f400 con 0x7f9648072df0 2024-03-28T18:56:39.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.154+0000 7f963fe00640 1 -- 172.21.15.27:0/3456272459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9638045070 con 0x7f9648072df0 2024-03-28T18:56:39.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.155+0000 7f963fe00640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f962406bbd0 0x7f962406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:56:39.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.156+0000 7f963fe00640 1 -- 172.21.15.27:0/3456272459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f96380acb00 con 0x7f9648072df0 2024-03-28T18:56:39.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.156+0000 7f963ea00640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f962406bbd0 0x7f962406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:56:39.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.156+0000 7f963ea00640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f962406bbd0 0x7f962406e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9630009560 tx=0x7f9630008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:56:39.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.160+0000 7f963fe00640 1 -- 172.21.15.27:0/3456272459 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9638030070 con 0x7f9648072df0 2024-03-28T18:56:39.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.363+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96480c4ce0 con 0x7f9648072df0 2024-03-28T18:56:39.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.365+0000 7f963fe00640 1 -- 172.21.15.27:0/3456272459 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+20501 (secure 0 0 0) 0x7f963807b400 con 0x7f9648072df0 2024-03-28T18:56:39.366 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:56:39.366 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"fsid":"c27ecc06-ff19-45f9-bdba-d624e1db1fbd","created":"2024-03-28T18:55:09.485521+0000","modified":"2024-03-28T18:56:15.976621+0000","last_up_change":"2024-03-28T18:55:22.128522+0000","last_in_change":"2024-03-28T18:55:13.060606+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:55:22.980934+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-03-28T18:55:32.332283+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-03-28T18:55:32.983411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-03-28T18:55:43.415397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-03-28T18:55:44.084162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac377893-aec5-494f-977a-d7c9410c3482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6817","nonce":1506376930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6819","nonce":1506376930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6823","nonce":1506376930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1506376930},{"type":"v1","addr":"172.21.15.27:6821","nonce":1506376930}]},"public_addr":"172.21.15.27:6817/1506376930","cluster_addr":"172.21.15.27:6819/1506376930","heartbeat_back_addr":"172.21.15.27:6823/1506376930","heartbeat_front_addr":"172.21.15.27:6821/1506376930","state":["exists","up"]},{"osd":1,"uuid":"d2f8b913-b0d6-4b30-9553-1428cdf1509c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6809","nonce":2953172651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6811","nonce":2953172651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6816","nonce":2953172651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2953172651},{"type":"v1","addr":"172.21.15.27:6813","nonce":2953172651}]},"public_addr":"172.21.15.27:6809/2953172651","cluster_addr":"172.21.15.27:6811/2953172651","heartbeat_back_addr":"172.21.15.27:6816/2953172651","heartbeat_front_addr":"172.21.15.27:6813/2953172651","state":["exists","up"]},{"osd":2,"uuid":"d285ba44-b5df-4431-b5bd-8d7b3545fe2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6825","nonce":150528886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6827","nonce":150528886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6831","nonce":150528886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":150528886},{"type":"v1","addr":"172.21.15.27:6829","nonce":150528886}]},"public_addr":"172.21.15.27:6825/150528886","cluster_addr":"172.21.15.27:6827/150528886","heartbeat_back_addr":"172.21.15.27:6831/150528886","heartbeat_front_addr":"172.21.15.27:6829/150528886","state":["exists","up"]},{"osd":3,"uuid":"91fb0367-acaa-4acc-83a6-6e7240683784","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6801","nonce":974955781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6803","nonce":974955781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6807","nonce":974955781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":974955781},{"type":"v1","addr":"172.21.15.27:6805","nonce":974955781}]},"public_addr":"172.21.15.27:6801/974955781","cluster_addr":"172.21.15.27:6803/974955781","heartbeat_back_addr":"172.21.15.27:6807/974955781","heartbeat_front_addr":"172.21.15.27:6805/974955781","state":["exists","up"]},{"osd":4,"uuid":"216c6c5e-1540-4217-8bba-88752ec61d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6801","nonce":998687689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6803","nonce":998687689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6807","nonce":998687689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":998687689},{"type":"v1","addr":"172.21.15.155:6805","nonce":998687689}]},"public_addr":"172.21.15.155:6801/998687689","cluster_addr":"172.21.15.155:6803/998687689","heartbeat_back_addr":"172.21.15.155:6807/998687689","heartbeat_front_addr":"172.21.15.155:6805/998687689","state":["exists","up"]},{"osd":5,"uuid":"02feaf2d-6bbd-4042-b802-acdb75512e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6817","nonce":2908350031}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6819","nonce":2908350031}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6823","nonce":2908350031}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":2908350031},{"type":"v1","addr":"172.21.15.155:6821","nonce":2908350031}]},"public_addr":"172.21.15.155:6817/2908350031","cluster_addr":"172.21.15.155:6819/2908350031","heartbeat_back_addr":"172.21.15.155:6823/2908350031","heartbeat_front_addr":"172.21.15.155:6821/2908350031","state":["exists","up"]},{"osd":6,"uuid":"756afc4a-1b8b-4d12-bd43-65e352c782b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6809","nonce":330629214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6811","nonce":330629214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6815","nonce":330629214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":330629214},{"type":"v1","addr":"172.21.15.155:6813","nonce":330629214}]},"public_addr":"172.21.15.155:6809/330629214","cluster_addr":"172.21.15.155:6811/330629214","heartbeat_back_addr":"172.21.15.155:6815/330629214","heartbeat_front_addr":"172.21.15.155:6813/330629214","state":["exists","up"]},{"osd":7,"uuid":"2b8d85a2-522a-40eb-b801-09e03ed25530","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6825","nonce":4094998171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6827","nonce":4094998171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6831","nonce":4094998171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":4094998171},{"type":"v1","addr":"172.21.15.155:6829","nonce":4094998171}]},"public_addr":"172.21.15.155:6825/4094998171","cluster_addr":"172.21.15.155:6827/4094998171","heartbeat_back_addr":"172.21.15.155:6831/4094998171","heartbeat_front_addr":"172.21.15.155:6829/4094998171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.119549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.370925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.447691+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.612444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.346413+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.419789+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.360719+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:55:18.160459+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.39","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:56:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f962406bbd0 msgr2=0x7f962406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f962406bbd0 0x7f962406e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9630009560 tx=0x7f9630008040 comp rx=0 tx=0).stop 2024-03-28T18:56:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9648072df0 msgr2=0x7f9648153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9648072df0 0x7f9648153290 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9638000c00 tx=0x7f96380046c0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 shutdown_connections 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f962406bbd0 0x7f962406e090 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96480c60e0 0x7f96481ee470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f96480c3ea0 0x7f96480a80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 --2- 172.21.15.27:0/3456272459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9648072df0 0x7f9648153290 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.368+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 >> 172.21.15.27:0/3456272459 conn(0x7f964806b9d0 msgr2=0x7f96480a9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.369+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 shutdown_connections 2024-03-28T18:56:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:56:39.369+0000 7f964e600640 1 -- 172.21.15.27:0/3456272459 wait complete. 2024-03-28T18:56:39.389 DEBUG:tasks.mds_thrash:Yielding 2024-03-28T18:56:39.389 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T18:56:39.403 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:56:39.404 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T18:56:39.404 INFO:tasks.workunit:timeout=6h 2024-03-28T18:56:39.404 INFO:tasks.workunit:cleanup=True 2024-03-28T18:56:39.404 DEBUG:teuthology.orchestra.run.smithi027:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:39.405 INFO:tasks.mds_thrash.fs.[b]:starting mds_do_thrash for fs b 2024-03-28T18:56:39.405 INFO:tasks.mds_thrash.fs.[b]:waiting for 25 secs before thrashing 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 29h/41d Inode: 1 Links: 2 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-03-28 18:55:34.492376474 +0000 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-03-28 18:56:22.815845427 +0000 2024-03-28T18:56:39.465 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2024-03-28T18:56:39.466 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T18:56:39.466 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T18:56:39.583 DEBUG:teuthology.orchestra.run.smithi155:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:39.643 INFO:teuthology.orchestra.run.smithi155.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:39.643 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-28T18:56:39.643 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 2ah/42d Inode: 1 Links: 2 2024-03-28T18:56:39.643 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:56:39.643 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-28T18:56:39.643 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-28T18:56:39.643 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-28 18:55:45.594982992 +0000 2024-03-28T18:56:39.644 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-28 18:56:22.967620266 +0000 2024-03-28T18:56:39.644 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2024-03-28T18:56:39.644 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-28T18:56:39.644 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-28T18:56:39.737 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:56:39.738 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:56:39.793 INFO:tasks.workunit.client.1.smithi155.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-28T18:56:39.795 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T18:56:54.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:56:54.791 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:57:04.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:04.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.647+0000 7f4a53e00640 1 Processor -- start 2024-03-28T18:57:04.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.647+0000 7f4a53e00640 1 -- start start 2024-03-28T18:57:04.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.649+0000 7f4a53e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4a540a6e90 0x7f4a54155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:04.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.649+0000 7f4a53e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a540a7400 con 0x7f4a540c1780 2024-03-28T18:57:04.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.649+0000 7f4a53e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a54158b40 con 0x7f4a5406c0d0 2024-03-28T18:57:04.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.649+0000 7f4a53e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a54159470 con 0x7f4a540a6e90 2024-03-28T18:57:04.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.649+0000 7f4a53400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a540c1780 0x7f4a5415fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43256/0 (socket says 172.21.15.27:43256) 2024-03-28T18:57:04.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.649+0000 7f4a53400640 1 -- 172.21.15.27:0/2228396172 learned_addr learned my addr 172.21.15.27:0/2228396172 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:04.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.649+0000 7f4a52000640 1 --2- 172.21.15.27:0/2228396172 >> v2:172.21.15.155:3300/0 conn(0x7f4a540a6e90 0x7f4a54155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:04.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.650+0000 7f4a51600640 1 -- 172.21.15.27:0/2228396172 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636605393 0 0) 0x7f4a540a7400 con 0x7f4a540c1780 2024-03-28T18:57:04.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.650+0000 7f4a51600640 1 -- 172.21.15.27:0/2228396172 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a38003520 con 0x7f4a540c1780 2024-03-28T18:57:04.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.650+0000 7f4a52000640 1 -- 172.21.15.27:0/2228396172 >> v1:172.21.15.155:6789/0 conn(0x7f4a5406c0d0 legacy=0x7f4a54155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:04.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.650+0000 7f4a52000640 1 -- 172.21.15.27:0/2228396172 >> v1:172.21.15.27:6789/0 conn(0x7f4a540c1780 legacy=0x7f4a5415fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:04.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.650+0000 7f4a51600640 1 -- 172.21.15.27:0/2228396172 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433895306 0 0) 0x7f4a38003520 con 0x7f4a540c1780 2024-03-28T18:57:04.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.650+0000 7f4a52000640 1 -- 172.21.15.27:0/2228396172 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a54158e20 con 0x7f4a540a6e90 2024-03-28T18:57:04.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.652+0000 7f4a52000640 1 --2- 172.21.15.27:0/2228396172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540a6e90 0x7f4a54155c40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a400097a0 tx=0x7f4a4002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bca07d2ca67504af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:04.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.652+0000 7f4a51600640 1 -- 172.21.15.27:0/2228396172 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40037070 con 0x7f4a540a6e90 2024-03-28T18:57:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.652+0000 7f4a51600640 1 -- 172.21.15.27:0/2228396172 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a40030510 con 0x7f4a540a6e90 2024-03-28T18:57:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.652+0000 7f4a51600640 1 -- 172.21.15.27:0/2228396172 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44001670 con 0x7f4a540a6e90 2024-03-28T18:57:04.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.653+0000 7f4a53e00640 1 -- 172.21.15.27:0/2228396172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540a6e90 msgr2=0x7f4a54155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.653+0000 7f4a53e00640 1 --2- 172.21.15.27:0/2228396172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540a6e90 0x7f4a54155c40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a400097a0 tx=0x7f4a4002b790 comp rx=0 tx=0).stop 2024-03-28T18:57:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.653+0000 7f4a53e00640 1 -- 172.21.15.27:0/2228396172 shutdown_connections 2024-03-28T18:57:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.653+0000 7f4a53e00640 1 --2- 172.21.15.27:0/2228396172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540a6e90 0x7f4a54155c40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.653+0000 7f4a53e00640 1 -- 172.21.15.27:0/2228396172 >> 172.21.15.27:0/2228396172 conn(0x7f4a540c7aa0 msgr2=0x7f4a540c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.653+0000 7f4a53e00640 1 -- 172.21.15.27:0/2228396172 shutdown_connections 2024-03-28T18:57:04.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.653+0000 7f4a53e00640 1 -- 172.21.15.27:0/2228396172 wait complete. 2024-03-28T18:57:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.655+0000 7f4a53e00640 1 Processor -- start 2024-03-28T18:57:04.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.655+0000 7f4a53e00640 1 -- start start 2024-03-28T18:57:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a53e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5406c0d0 0x7f4a540aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a53e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540a6e90 0x7f4a5414f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a53e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540c1780 0x7f4a5414f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a53e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54152e20 con 0x7f4a540a6e90 2024-03-28T18:57:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a53e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54159900 con 0x7f4a540c1780 2024-03-28T18:57:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a53e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54152fa0 con 0x7f4a5406c0d0 2024-03-28T18:57:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a52000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540a6e90 0x7f4a5414f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:04.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a52000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540a6e90 0x7f4a5414f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38138/0 (socket says 172.21.15.27:38138) 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a52000640 1 -- 172.21.15.27:0/3904106734 learned_addr learned my addr 172.21.15.27:0/3904106734 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a52a00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5406c0d0 0x7f4a540aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a53400640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540c1780 0x7f4a5414f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a52000640 1 -- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5406c0d0 msgr2=0x7f4a540aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a52000640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5406c0d0 0x7f4a540aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a52000640 1 -- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540c1780 msgr2=0x7f4a5414f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.656+0000 7f4a52000640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540c1780 0x7f4a5414f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.657+0000 7f4a52000640 1 -- 172.21.15.27:0/3904106734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a541ebc20 con 0x7f4a540a6e90 2024-03-28T18:57:04.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.657+0000 7f4a53400640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540c1780 0x7f4a5414f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:57:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.657+0000 7f4a52000640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540a6e90 0x7f4a5414f150 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4a40035040 tx=0x7f4a400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.657+0000 7f4a4aa00640 1 -- 172.21.15.27:0/3904106734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40042020 con 0x7f4a540a6e90 2024-03-28T18:57:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.657+0000 7f4a4aa00640 1 -- 172.21.15.27:0/3904106734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a400044c0 con 0x7f4a540a6e90 2024-03-28T18:57:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.657+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5414fbd0 con 0x7f4a540a6e90 2024-03-28T18:57:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.657+0000 7f4a4aa00640 1 -- 172.21.15.27:0/3904106734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40037040 con 0x7f4a540a6e90 2024-03-28T18:57:04.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.657+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a54150640 con 0x7f4a540a6e90 2024-03-28T18:57:04.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.659+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a54069640 con 0x7f4a540a6e90 2024-03-28T18:57:04.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.660+0000 7f4a4aa00640 1 -- 172.21.15.27:0/3904106734 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a40045020 con 0x7f4a540a6e90 2024-03-28T18:57:04.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.661+0000 7f4a4aa00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3406bc00 0x7f4a3406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:04.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.661+0000 7f4a4aa00640 1 -- 172.21.15.27:0/3904106734 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f4a400aca90 con 0x7f4a540a6e90 2024-03-28T18:57:04.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.661+0000 7f4a52a00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3406bc00 0x7f4a3406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:04.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.662+0000 7f4a52a00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3406bc00 0x7f4a3406e0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0012e0 tx=0x7f4a4c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:04.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.666+0000 7f4a4aa00640 1 -- 172.21.15.27:0/3904106734 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a4004c030 con 0x7f4a540a6e90 2024-03-28T18:57:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.916+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a540c1560 con 0x7f4a540a6e90 2024-03-28T18:57:04.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.917+0000 7f4a4aa00640 1 -- 172.21.15.27:0/3904106734 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f4a4007b270 con 0x7f4a540a6e90 2024-03-28T18:57:04.917 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:04.917 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:57:04.917 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3406bc00 msgr2=0x7f4a3406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3406bc00 0x7f4a3406e0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0012e0 tx=0x7f4a4c009040 comp rx=0 tx=0).stop 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540a6e90 msgr2=0x7f4a5414f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540a6e90 0x7f4a5414f150 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4a40035040 tx=0x7f4a400046c0 comp rx=0 tx=0).stop 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 shutdown_connections 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a3406bc00 0x7f4a3406e0c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a540c1780 0x7f4a5414f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a540a6e90 0x7f4a5414f150 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 --2- 172.21.15.27:0/3904106734 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a5406c0d0 0x7f4a540aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 >> 172.21.15.27:0/3904106734 conn(0x7f4a540c7aa0 msgr2=0x7f4a54073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:04.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 shutdown_connections 2024-03-28T18:57:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:04.920+0000 7f4a53e00640 1 -- 172.21.15.27:0/3904106734 wait complete. 2024-03-28T18:57:04.940 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-03-28T18:57:04.940 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T18:57:04.940 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T18:57:04.959 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T18:57:04.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:05.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.239+0000 7ff249000640 1 Processor -- start 2024-03-28T18:57:05.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.240+0000 7ff249000640 1 -- start start 2024-03-28T18:57:05.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.241+0000 7ff249000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff2440c8ed0 0x7ff2440c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:05.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.241+0000 7ff249000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2440be5d0 0x7ff244155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:05.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.241+0000 7ff249000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff244156230 con 0x7ff2440c58c0 2024-03-28T18:57:05.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.241+0000 7ff249000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff244158bd0 con 0x7ff2440be5d0 2024-03-28T18:57:05.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.241+0000 7ff249000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff244158eb0 con 0x7ff2440c8ed0 2024-03-28T18:57:05.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.241+0000 7ff243400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff2440c58c0 0x7ff2441555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43272/0 (socket says 172.21.15.27:43272) 2024-03-28T18:57:05.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.241+0000 7ff243400640 1 -- 172.21.15.27:0/363656392 learned_addr learned my addr 172.21.15.27:0/363656392 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:05.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.242+0000 7ff243e00640 1 --2- 172.21.15.27:0/363656392 >> v2:172.21.15.27:3300/0 conn(0x7ff2440be5d0 0x7ff244155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:05.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.242+0000 7ff242a00640 1 --2- 172.21.15.27:0/363656392 >> v2:172.21.15.155:3300/0 conn(0x7ff2440c8ed0 0x7ff2440c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:05.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.242+0000 7ff242000640 1 -- 172.21.15.27:0/363656392 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589228706 0 0) 0x7ff244156230 con 0x7ff2440c58c0 2024-03-28T18:57:05.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.242+0000 7ff242000640 1 -- 172.21.15.27:0/363656392 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff228003520 con 0x7ff2440c58c0 2024-03-28T18:57:05.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.242+0000 7ff243e00640 1 -- 172.21.15.27:0/363656392 >> v2:172.21.15.155:3300/0 conn(0x7ff2440c8ed0 msgr2=0x7ff2440c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:05.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.242+0000 7ff243e00640 1 --2- 172.21.15.27:0/363656392 >> v2:172.21.15.155:3300/0 conn(0x7ff2440c8ed0 0x7ff2440c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.242+0000 7ff243e00640 1 -- 172.21.15.27:0/363656392 >> v1:172.21.15.27:6789/0 conn(0x7ff2440c58c0 legacy=0x7ff2441555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:05.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.242+0000 7ff243e00640 1 -- 172.21.15.27:0/363656392 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff244159190 con 0x7ff2440be5d0 2024-03-28T18:57:05.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.243+0000 7ff243e00640 1 --2- 172.21.15.27:0/363656392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440be5d0 0x7ff244155cf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff2440c83d0 tx=0x7ff23c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afdf8d53bbf3563a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:05.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.244+0000 7ff242000640 1 -- 172.21.15.27:0/363656392 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c038070 con 0x7ff2440be5d0 2024-03-28T18:57:05.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.244+0000 7ff242000640 1 -- 172.21.15.27:0/363656392 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c031520 con 0x7ff2440be5d0 2024-03-28T18:57:05.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.244+0000 7ff242000640 1 -- 172.21.15.27:0/363656392 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c0318f0 con 0x7ff2440be5d0 2024-03-28T18:57:05.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.244+0000 7ff249000640 1 -- 172.21.15.27:0/363656392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440be5d0 msgr2=0x7ff244155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:05.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.244+0000 7ff249000640 1 --2- 172.21.15.27:0/363656392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440be5d0 0x7ff244155cf0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff2440c83d0 tx=0x7ff23c02b7c0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.245+0000 7ff249000640 1 -- 172.21.15.27:0/363656392 shutdown_connections 2024-03-28T18:57:05.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.245+0000 7ff249000640 1 --2- 172.21.15.27:0/363656392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440be5d0 0x7ff244155cf0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.245+0000 7ff249000640 1 --2- 172.21.15.27:0/363656392 >> v2:172.21.15.155:3300/0 conn(0x7ff2440c8ed0 0x7ff2440c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.245+0000 7ff249000640 1 -- 172.21.15.27:0/363656392 >> 172.21.15.27:0/363656392 conn(0x7ff24406b6f0 msgr2=0x7ff2440acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:05.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.245+0000 7ff249000640 1 -- 172.21.15.27:0/363656392 shutdown_connections 2024-03-28T18:57:05.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.245+0000 7ff249000640 1 -- 172.21.15.27:0/363656392 wait complete. 2024-03-28T18:57:05.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.246+0000 7ff249000640 1 Processor -- start 2024-03-28T18:57:05.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.247+0000 7ff249000640 1 -- start start 2024-03-28T18:57:05.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.247+0000 7ff249000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2440be5d0 0x7ff2440c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:05.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.247+0000 7ff249000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440c58c0 0x7ff244151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:05.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff249000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2440c8ed0 0x7ff2441523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:05.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff249000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2440b0820 con 0x7ff2440c58c0 2024-03-28T18:57:05.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff249000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff244159750 con 0x7ff2440be5d0 2024-03-28T18:57:05.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff249000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2440b09a0 con 0x7ff2440c8ed0 2024-03-28T18:57:05.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff242a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440c58c0 0x7ff244151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:05.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff242a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440c58c0 0x7ff244151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38162/0 (socket says 172.21.15.27:38162) 2024-03-28T18:57:05.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff242a00640 1 -- 172.21.15.27:0/3676086932 learned_addr learned my addr 172.21.15.27:0/3676086932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff243e00640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2440c8ed0 0x7ff2441523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff242a00640 1 -- 172.21.15.27:0/3676086932 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2440c8ed0 msgr2=0x7ff2441523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff242a00640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2440c8ed0 0x7ff2441523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff242a00640 1 -- 172.21.15.27:0/3676086932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2440be5d0 msgr2=0x7ff2440c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff242a00640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2440be5d0 0x7ff2440c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.248+0000 7ff242a00640 1 -- 172.21.15.27:0/3676086932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2440c4a50 con 0x7ff2440c58c0 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.249+0000 7ff242a00640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440c58c0 0x7ff244151e70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00e870 tx=0x7ff22c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.249+0000 7ff23b400640 1 -- 172.21.15.27:0/3676086932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c017070 con 0x7ff2440c58c0 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.249+0000 7ff23b400640 1 -- 172.21.15.27:0/3676086932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c00c940 con 0x7ff2440c58c0 2024-03-28T18:57:05.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.249+0000 7ff23b400640 1 -- 172.21.15.27:0/3676086932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00cd10 con 0x7ff2440c58c0 2024-03-28T18:57:05.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.249+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2440c4d30 con 0x7ff2440c58c0 2024-03-28T18:57:05.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.249+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2440aad00 con 0x7ff2440c58c0 2024-03-28T18:57:05.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.250+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff244069640 con 0x7ff2440c58c0 2024-03-28T18:57:05.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.251+0000 7ff23b400640 1 -- 172.21.15.27:0/3676086932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff22c01a070 con 0x7ff2440c58c0 2024-03-28T18:57:05.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.252+0000 7ff23b400640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff22006bbd0 0x7ff22006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:05.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.252+0000 7ff23b400640 1 -- 172.21.15.27:0/3676086932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7ff22c08af90 con 0x7ff2440c58c0 2024-03-28T18:57:05.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.253+0000 7ff243400640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff22006bbd0 0x7ff22006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:05.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.253+0000 7ff243400640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff22006bbd0 0x7ff22006e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff2340012e0 tx=0x7ff2340015b0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:05.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.260+0000 7ff23b400640 1 -- 172.21.15.27:0/3676086932 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff22c059770 con 0x7ff2440c58c0 2024-03-28T18:57:05.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.501+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2440c1d30 con 0x7ff2440c58c0 2024-03-28T18:57:05.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.503+0000 7ff23b400640 1 -- 172.21.15.27:0/3676086932 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7ff22c059070 con 0x7ff2440c58c0 2024-03-28T18:57:05.503 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:05.503 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:57:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:57:05.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.505+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff22006bbd0 msgr2=0x7ff22006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.505+0000 7ff249000640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff22006bbd0 0x7ff22006e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff2340012e0 tx=0x7ff2340015b0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.505+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440c58c0 msgr2=0x7ff244151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.505+0000 7ff249000640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440c58c0 0x7ff244151e70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00e870 tx=0x7ff22c00ed30 comp rx=0 tx=0).stop 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.506+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 shutdown_connections 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.506+0000 7ff249000640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff22006bbd0 0x7ff22006e090 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.506+0000 7ff249000640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2440c8ed0 0x7ff2441523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.506+0000 7ff249000640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2440c58c0 0x7ff244151e70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.506+0000 7ff249000640 1 --2- 172.21.15.27:0/3676086932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2440be5d0 0x7ff2440c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.506+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 >> 172.21.15.27:0/3676086932 conn(0x7ff24406b6f0 msgr2=0x7ff2440ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.506+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 shutdown_connections 2024-03-28T18:57:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:05.506+0000 7ff249000640 1 -- 172.21.15.27:0/3676086932 wait complete. 2024-03-28T18:57:05.526 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:57:07.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.764+0000 7f55ebc00640 1 Processor -- start 2024-03-28T18:57:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.764+0000 7f55ebc00640 1 -- start start 2024-03-28T18:57:07.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.765+0000 7f55ebc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e40c7970 0x7f55e40c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:07.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.765+0000 7f55ebc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55e4155730 0x7f55e4155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:07.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.765+0000 7f55ebc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40c7310 con 0x7f55e4152f80 2024-03-28T18:57:07.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.765+0000 7f55ebc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e4158d40 con 0x7f55e4155730 2024-03-28T18:57:07.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.765+0000 7f55ebc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e4159020 con 0x7f55e40c7970 2024-03-28T18:57:07.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.765+0000 7f55e9c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55e4155730 0x7f55e4155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:07.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.765+0000 7f55e9c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55e4155730 0x7f55e4155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58194/0 (socket says 172.21.15.27:58194) 2024-03-28T18:57:07.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.766+0000 7f55e9c00640 1 -- 172.21.15.27:0/609358427 learned_addr learned my addr 172.21.15.27:0/609358427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:07.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.766+0000 7f55e3e00640 1 --2- 172.21.15.27:0/609358427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e40c7970 0x7f55e40c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:07.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.766+0000 7f55e9c00640 1 -- 172.21.15.27:0/609358427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e40c7970 msgr2=0x7f55e40c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:07.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.766+0000 7f55e9c00640 1 --2- 172.21.15.27:0/609358427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e40c7970 0x7f55e40c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:07.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.766+0000 7f55e9c00640 1 -- 172.21.15.27:0/609358427 >> v1:172.21.15.155:6789/0 conn(0x7f55e4152f80 legacy=0x7f55e40c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:07.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.766+0000 7f55e9c00640 1 -- 172.21.15.27:0/609358427 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e4159300 con 0x7f55e4155730 2024-03-28T18:57:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.767+0000 7f55e9c00640 1 --2- 172.21.15.27:0/609358427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e4155730 0x7f55e4155bc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009870 tx=0x7f55dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503ae6d5d52794c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55e3400640 1 -- 172.21.15.27:0/609358427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc035020 con 0x7f55e4155730 2024-03-28T18:57:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55e3400640 1 -- 172.21.15.27:0/609358427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc031640 con 0x7f55e4155730 2024-03-28T18:57:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55e3400640 1 -- 172.21.15.27:0/609358427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc02d040 con 0x7f55e4155730 2024-03-28T18:57:07.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55ebc00640 1 -- 172.21.15.27:0/609358427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e4155730 msgr2=0x7f55e4155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55ebc00640 1 --2- 172.21.15.27:0/609358427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e4155730 0x7f55e4155bc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009870 tx=0x7f55dc02b8f0 comp rx=0 tx=0).stop 2024-03-28T18:57:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55ebc00640 1 -- 172.21.15.27:0/609358427 shutdown_connections 2024-03-28T18:57:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55ebc00640 1 --2- 172.21.15.27:0/609358427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e4155730 0x7f55e4155bc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55ebc00640 1 --2- 172.21.15.27:0/609358427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e40c7970 0x7f55e40c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:07.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55ebc00640 1 -- 172.21.15.27:0/609358427 >> 172.21.15.27:0/609358427 conn(0x7f55e406b6f0 msgr2=0x7f55e40caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55ebc00640 1 -- 172.21.15.27:0/609358427 shutdown_connections 2024-03-28T18:57:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.768+0000 7f55ebc00640 1 -- 172.21.15.27:0/609358427 wait complete. 2024-03-28T18:57:07.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.770+0000 7f55ebc00640 1 Processor -- start 2024-03-28T18:57:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.771+0000 7f55ebc00640 1 -- start start 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55ebc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e40c7970 0x7f55e41ec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55ebc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e4152f80 0x7f55e40ca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55ebc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55e4155730 0x7f55e41460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55ebc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e40cbca0 con 0x7f55e40c7970 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55ebc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e4159b00 con 0x7f55e4155730 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55ebc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e41ec800 con 0x7f55e4152f80 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55e9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e40c7970 0x7f55e41ec2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55e9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e40c7970 0x7f55e41ec2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58200/0 (socket says 172.21.15.27:58200) 2024-03-28T18:57:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55e9200640 1 -- 172.21.15.27:0/1462609264 learned_addr learned my addr 172.21.15.27:0/1462609264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55e9c00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55e4155730 0x7f55e41460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.772+0000 7f55e3e00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e4152f80 0x7f55e40ca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e9200640 1 -- 172.21.15.27:0/1462609264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e4152f80 msgr2=0x7f55e40ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e9200640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e4152f80 0x7f55e40ca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e9200640 1 -- 172.21.15.27:0/1462609264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55e4155730 msgr2=0x7f55e41460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e9200640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55e4155730 0x7f55e41460b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e9200640 1 -- 172.21.15.27:0/1462609264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e414b260 con 0x7f55e40c7970 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e9200640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e40c7970 0x7f55e41ec2c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f55d400c760 tx=0x7f55d400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e0c00640 1 -- 172.21.15.27:0/1462609264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4016020 con 0x7f55e40c7970 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e0c00640 1 -- 172.21.15.27:0/1462609264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d4006d50 con 0x7f55e40c7970 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e40c9ce0 con 0x7f55e40c7970 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55e0c00640 1 -- 172.21.15.27:0/1462609264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4003bb0 con 0x7f55e40c7970 2024-03-28T18:57:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.773+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e40ca2a0 con 0x7f55e40c7970 2024-03-28T18:57:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.775+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e404f400 con 0x7f55e40c7970 2024-03-28T18:57:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.776+0000 7f55e0c00640 1 -- 172.21.15.27:0/1462609264 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f55d400a850 con 0x7f55e40c7970 2024-03-28T18:57:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.776+0000 7f55e0c00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f55c406bbd0 0x7f55c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.777+0000 7f55e0c00640 1 -- 172.21.15.27:0/1462609264 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f55d408aa20 con 0x7f55e40c7970 2024-03-28T18:57:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.777+0000 7f55e3e00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f55c406bbd0 0x7f55c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.777+0000 7f55e3e00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f55c406bbd0 0x7f55c406e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f55d0006fd0 tx=0x7f55d0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:07.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:07.781+0000 7f55e0c00640 1 -- 172.21.15.27:0/1462609264 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f55d4020020 con 0x7f55e40c7970 2024-03-28T18:57:08.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.029+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55e40cba80 con 0x7f55e40c7970 2024-03-28T18:57:08.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.031+0000 7f55e0c00640 1 -- 172.21.15.27:0/1462609264 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f55e414b260 con 0x7f55e40c7970 2024-03-28T18:57:08.030 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:08.031 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:57:08.031 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f55c406bbd0 msgr2=0x7f55c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f55c406bbd0 0x7f55c406e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f55d0006fd0 tx=0x7f55d0008040 comp rx=0 tx=0).stop 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e40c7970 msgr2=0x7f55e41ec2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e40c7970 0x7f55e41ec2c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f55d400c760 tx=0x7f55d400cc20 comp rx=0 tx=0).stop 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 shutdown_connections 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f55c406bbd0 0x7f55c406e090 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55e4155730 0x7f55e41460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f55e4152f80 0x7f55e40ca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 --2- 172.21.15.27:0/1462609264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e40c7970 0x7f55e41ec2c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 >> 172.21.15.27:0/1462609264 conn(0x7f55e406b6f0 msgr2=0x7f55e4150d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 shutdown_connections 2024-03-28T18:57:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:08.034+0000 7f55ebc00640 1 -- 172.21.15.27:0/1462609264 wait complete. 2024-03-28T18:57:08.053 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:57:10.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:10.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.297+0000 7fcab0a00640 1 Processor -- start 2024-03-28T18:57:10.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.298+0000 7fcab0a00640 1 -- start start 2024-03-28T18:57:10.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.299+0000 7fcab0a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcaac154140 0x7fcaac0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:10.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.299+0000 7fcab0a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaac150ec0 con 0x7fcaac0c69a0 2024-03-28T18:57:10.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.299+0000 7fcab0a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaac06dc70 con 0x7fcaac150ae0 2024-03-28T18:57:10.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.299+0000 7fcab0a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaac153550 con 0x7fcaac154140 2024-03-28T18:57:10.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.299+0000 7fcaabe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcaac154140 0x7fcaac0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:10.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.300+0000 7fcaabe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcaac154140 0x7fcaac0c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58212/0 (socket says 172.21.15.27:58212) 2024-03-28T18:57:10.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.300+0000 7fcaabe00640 1 -- 172.21.15.27:0/986430269 learned_addr learned my addr 172.21.15.27:0/986430269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:10.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.300+0000 7fcaa3e00640 1 --1- 172.21.15.27:0/986430269 >> v1:172.21.15.27:6789/0 conn(0x7fcaac0c69a0 0x7fcaac153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46592/0 (socket says 172.21.15.27:46592) 2024-03-28T18:57:10.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.300+0000 7fcaabe00640 1 -- 172.21.15.27:0/986430269 >> v1:172.21.15.155:6789/0 conn(0x7fcaac150ae0 legacy=0x7fcaac0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:10.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.300+0000 7fcaabe00640 1 -- 172.21.15.27:0/986430269 >> v1:172.21.15.27:6789/0 conn(0x7fcaac0c69a0 legacy=0x7fcaac153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:10.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.300+0000 7fcaabe00640 1 -- 172.21.15.27:0/986430269 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac06df10 con 0x7fcaac154140 2024-03-28T18:57:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.301+0000 7fcaabe00640 1 --2- 172.21.15.27:0/986430269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac154140 0x7fcaac0c6e30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009e50 tx=0x7fcaa402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7696a1c0dc5246ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.301+0000 7fcaaaa00640 1 -- 172.21.15.27:0/986430269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa402ca80 con 0x7fcaac154140 2024-03-28T18:57:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.301+0000 7fcaaaa00640 1 -- 172.21.15.27:0/986430269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa402cc20 con 0x7fcaac154140 2024-03-28T18:57:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.301+0000 7fcaaaa00640 1 -- 172.21.15.27:0/986430269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa40343f0 con 0x7fcaac154140 2024-03-28T18:57:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.302+0000 7fcab0a00640 1 -- 172.21.15.27:0/986430269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac154140 msgr2=0x7fcaac0c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.302+0000 7fcab0a00640 1 --2- 172.21.15.27:0/986430269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac154140 0x7fcaac0c6e30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009e50 tx=0x7fcaa402bee0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.302+0000 7fcab0a00640 1 -- 172.21.15.27:0/986430269 shutdown_connections 2024-03-28T18:57:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.302+0000 7fcab0a00640 1 --2- 172.21.15.27:0/986430269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac154140 0x7fcaac0c6e30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.302+0000 7fcab0a00640 1 -- 172.21.15.27:0/986430269 >> 172.21.15.27:0/986430269 conn(0x7fcaac06b0c0 msgr2=0x7fcaac06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.302+0000 7fcab0a00640 1 -- 172.21.15.27:0/986430269 shutdown_connections 2024-03-28T18:57:10.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.302+0000 7fcab0a00640 1 -- 172.21.15.27:0/986430269 wait complete. 2024-03-28T18:57:10.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.304+0000 7fcab0a00640 1 Processor -- start 2024-03-28T18:57:10.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.304+0000 7fcab0a00640 1 -- start start 2024-03-28T18:57:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcab0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac0c69a0 0x7fcaac0bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcab0a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaac150ae0 0x7fcaac14ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcab0a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaac154140 0x7fcaac14cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcab0a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac06e7b0 con 0x7fcaac0c69a0 2024-03-28T18:57:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcab0a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac06e4d0 con 0x7fcaac150ae0 2024-03-28T18:57:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcab0a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac0a6370 con 0x7fcaac154140 2024-03-28T18:57:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaab400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac0c69a0 0x7fcaac0bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaab400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac0c69a0 0x7fcaac0bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58214/0 (socket says 172.21.15.27:58214) 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaab400640 1 -- 172.21.15.27:0/3264106023 learned_addr learned my addr 172.21.15.27:0/3264106023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaa3e00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaac150ae0 0x7fcaac14ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaab400640 1 -- 172.21.15.27:0/3264106023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaac154140 msgr2=0x7fcaac14cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaab400640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaac154140 0x7fcaac14cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaab400640 1 -- 172.21.15.27:0/3264106023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaac150ae0 msgr2=0x7fcaac14ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaab400640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaac150ae0 0x7fcaac14ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaab400640 1 -- 172.21.15.27:0/3264106023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac149e00 con 0x7fcaac0c69a0 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.306+0000 7fcaa3e00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaac150ae0 0x7fcaac14ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:57:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.307+0000 7fcaab400640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac0c69a0 0x7fcaac0bfd20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00c430 tx=0x7fca9c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:10.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.307+0000 7fcaa3400640 1 -- 172.21.15.27:0/3264106023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c011790 con 0x7fcaac0c69a0 2024-03-28T18:57:10.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.307+0000 7fcaa3400640 1 -- 172.21.15.27:0/3264106023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c017070 con 0x7fcaac0c69a0 2024-03-28T18:57:10.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.307+0000 7fcaa3400640 1 -- 172.21.15.27:0/3264106023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c011bb0 con 0x7fcaac0c69a0 2024-03-28T18:57:10.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.307+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaac151ef0 con 0x7fcaac0c69a0 2024-03-28T18:57:10.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.307+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaac153260 con 0x7fcaac0c69a0 2024-03-28T18:57:10.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.308+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaac067dd0 con 0x7fcaac0c69a0 2024-03-28T18:57:10.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.309+0000 7fcaa3400640 1 -- 172.21.15.27:0/3264106023 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fca9c01c460 con 0x7fcaac0c69a0 2024-03-28T18:57:10.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.310+0000 7fcaa3400640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fca8806bc00 0x7fca8806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:10.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.310+0000 7fcaa3400640 1 -- 172.21.15.27:0/3264106023 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7fca9c08ac30 con 0x7fcaac0c69a0 2024-03-28T18:57:10.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.311+0000 7fcaa3e00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fca8806bc00 0x7fca8806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:10.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.311+0000 7fcaa3e00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fca8806bc00 0x7fca8806e0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcaac14a240 tx=0x7fca94009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.315+0000 7fcaa3400640 1 -- 172.21.15.27:0/3264106023 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fca9c059370 con 0x7fcaac0c69a0 2024-03-28T18:57:10.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.559+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaac06b6c0 con 0x7fcaac0c69a0 2024-03-28T18:57:10.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.560+0000 7fcaa3400640 1 -- 172.21.15.27:0/3264106023 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7fca9c058c70 con 0x7fcaac0c69a0 2024-03-28T18:57:10.560 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:10.560 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:57:10.560 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fca8806bc00 msgr2=0x7fca8806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fca8806bc00 0x7fca8806e0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcaac14a240 tx=0x7fca94009040 comp rx=0 tx=0).stop 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac0c69a0 msgr2=0x7fcaac0bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac0c69a0 0x7fcaac0bfd20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00c430 tx=0x7fca9c00c8f0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 shutdown_connections 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fca8806bc00 0x7fca8806e0c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaac154140 0x7fcaac14cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaac150ae0 0x7fcaac14ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 --2- 172.21.15.27:0/3264106023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaac0c69a0 0x7fcaac0bfd20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 >> 172.21.15.27:0/3264106023 conn(0x7fcaac06b0c0 msgr2=0x7fcaac155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 shutdown_connections 2024-03-28T18:57:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:10.563+0000 7fcab0a00640 1 -- 172.21.15.27:0/3264106023 wait complete. 2024-03-28T18:57:10.584 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:57:12.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:12.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.821+0000 7f2a93e00640 1 Processor -- start 2024-03-28T18:57:12.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.821+0000 7f2a93e00640 1 -- start start 2024-03-28T18:57:12.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.823+0000 7f2a93e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2a8c0c3ea0 0x7f2a8c159610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:12.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.823+0000 7f2a93e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c0c1600 con 0x7f2a8c06ee00 2024-03-28T18:57:12.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.823+0000 7f2a93e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c0c18e0 con 0x7f2a8c0c60e0 2024-03-28T18:57:12.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.823+0000 7f2a93e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c23b0 con 0x7f2a8c0c3ea0 2024-03-28T18:57:12.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.823+0000 7f2a90a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2a8c06ee00 0x7f2a8c158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46604/0 (socket says 172.21.15.27:46604) 2024-03-28T18:57:12.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.823+0000 7f2a90a00640 1 -- 172.21.15.27:0/2954769916 learned_addr learned my addr 172.21.15.27:0/2954769916 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.823+0000 7f2a91e00640 1 --2- 172.21.15.27:0/2954769916 >> v2:172.21.15.155:3300/0 conn(0x7f2a8c0c3ea0 0x7f2a8c159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013147761 0 0) 0x7f2a8c0c1600 con 0x7f2a8c06ee00 2024-03-28T18:57:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a74003520 con 0x7f2a8c06ee00 2024-03-28T18:57:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4196764851 0 0) 0x7f2a74003520 con 0x7f2a8c06ee00 2024-03-28T18:57:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 >> v2:172.21.15.155:3300/0 conn(0x7f2a8c0c3ea0 msgr2=0x7f2a8c159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 --2- 172.21.15.27:0/2954769916 >> v2:172.21.15.155:3300/0 conn(0x7f2a8c0c3ea0 0x7f2a8c159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 >> v1:172.21.15.155:6789/0 conn(0x7f2a8c0c60e0 legacy=0x7f2a8c06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c0c1bc0 con 0x7f2a8c06ee00 2024-03-28T18:57:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f2a78001670 con 0x7f2a8c06ee00 2024-03-28T18:57:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 >> v1:172.21.15.27:6789/0 conn(0x7f2a8c06ee00 legacy=0x7f2a8c158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.824+0000 7f2a8be00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a740039b0 0x7f2a74003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a8be00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a74004800 0x7f2a74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a8be00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74005210 0x7f2a7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c23b0 con 0x7f2a74004800 2024-03-28T18:57:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7400c140 con 0x7f2a74005210 2024-03-28T18:57:12.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7400c340 con 0x7f2a740039b0 2024-03-28T18:57:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a90a00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a74004800 0x7f2a74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a91e00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74005210 0x7f2a7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:12.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a91400640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a740039b0 0x7f2a74003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a90a00640 1 -- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a740039b0 msgr2=0x7f2a74003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a90a00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a740039b0 0x7f2a74003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.825+0000 7f2a90a00640 1 -- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74005210 msgr2=0x7f2a7400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.826+0000 7f2a90a00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74005210 0x7f2a7400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.826+0000 7f2a90a00640 1 -- 172.21.15.27:0/2954769916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a74003520 con 0x7f2a74004800 2024-03-28T18:57:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.826+0000 7f2a91e00640 1 -- 172.21.15.27:0/2954769916 reap_dead start 2024-03-28T18:57:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.826+0000 7f2a91400640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a740039b0 0x7f2a74003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:57:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.827+0000 7f2a90a00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a74004800 0x7f2a74004c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800c240 tx=0x7f2a74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7960ed97b196b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.827+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78007cf0 con 0x7f2a74004800 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.827+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a78030070 con 0x7f2a74004800 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.827+0000 7f2a8be00640 1 -- 172.21.15.27:0/2954769916 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78005d70 con 0x7f2a74004800 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.828+0000 7f2a93e00640 1 -- 172.21.15.27:0/2954769916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a74004800 msgr2=0x7f2a74004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.828+0000 7f2a93e00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a74004800 0x7f2a74004c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2a7800c240 tx=0x7f2a74005830 comp rx=0 tx=0).stop 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.828+0000 7f2a93e00640 1 -- 172.21.15.27:0/2954769916 shutdown_connections 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.828+0000 7f2a93e00640 1 --2- 172.21.15.27:0/2954769916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a74004800 0x7f2a74004c80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.828+0000 7f2a93e00640 1 -- 172.21.15.27:0/2954769916 >> 172.21.15.27:0/2954769916 conn(0x7f2a8c06b9d0 msgr2=0x7f2a8c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:12.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.828+0000 7f2a93e00640 1 -- 172.21.15.27:0/2954769916 shutdown_connections 2024-03-28T18:57:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.828+0000 7f2a93e00640 1 -- 172.21.15.27:0/2954769916 wait complete. 2024-03-28T18:57:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.830+0000 7f2a93e00640 1 Processor -- start 2024-03-28T18:57:12.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.830+0000 7f2a93e00640 1 -- start start 2024-03-28T18:57:12.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74004800 0x7f2a8c1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a8c14b560 0x7f2a8c1eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a93e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a8c1e7f10 0x7f2a8c1e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a93e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c6640 con 0x7f2a8c14b560 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a93e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c64c0 con 0x7f2a74004800 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a93e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c072df0 con 0x7f2a8c1e7f10 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a90a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a8c14b560 0x7f2a8c1eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a90a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a8c14b560 0x7f2a8c1eead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58222/0 (socket says 172.21.15.27:58222) 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.831+0000 7f2a90a00640 1 -- 172.21.15.27:0/2318665424 learned_addr learned my addr 172.21.15.27:0/2318665424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.832+0000 7f2a91400640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74004800 0x7f2a8c1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.832+0000 7f2a91e00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a8c1e7f10 0x7f2a8c1e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.832+0000 7f2a90a00640 1 -- 172.21.15.27:0/2318665424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a8c1e7f10 msgr2=0x7f2a8c1e83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.832+0000 7f2a90a00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a8c1e7f10 0x7f2a8c1e83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.832+0000 7f2a90a00640 1 -- 172.21.15.27:0/2318665424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74004800 msgr2=0x7f2a8c1ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.832+0000 7f2a90a00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74004800 0x7f2a8c1ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:12.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.832+0000 7f2a90a00640 1 -- 172.21.15.27:0/2318665424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c1ef010 con 0x7f2a8c14b560 2024-03-28T18:57:12.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.832+0000 7f2a90a00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a8c14b560 0x7f2a8c1eead0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2a78001970 tx=0x7f2a8c0c17d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:12.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.833+0000 7f2a89600640 1 -- 172.21.15.27:0/2318665424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78001c30 con 0x7f2a8c14b560 2024-03-28T18:57:12.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.833+0000 7f2a89600640 1 -- 172.21.15.27:0/2318665424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a78004bf0 con 0x7f2a8c14b560 2024-03-28T18:57:12.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.833+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8c0a8ad0 con 0x7f2a8c14b560 2024-03-28T18:57:12.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.833+0000 7f2a89600640 1 -- 172.21.15.27:0/2318665424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78007070 con 0x7f2a8c14b560 2024-03-28T18:57:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.833+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8c14c030 con 0x7f2a8c14b560 2024-03-28T18:57:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.834+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c04f400 con 0x7f2a8c14b560 2024-03-28T18:57:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.835+0000 7f2a89600640 1 -- 172.21.15.27:0/2318665424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2a78048020 con 0x7f2a8c14b560 2024-03-28T18:57:12.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.836+0000 7f2a89600640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a6c06bbd0 0x7f2a6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:12.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.836+0000 7f2a89600640 1 -- 172.21.15.27:0/2318665424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f2a780ac460 con 0x7f2a8c14b560 2024-03-28T18:57:12.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.836+0000 7f2a91400640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a6c06bbd0 0x7f2a6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:12.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.837+0000 7f2a91400640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a6c06bbd0 0x7f2a6c06e090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2a80000c00 tx=0x7f2a80009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:12.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:12.840+0000 7f2a89600640 1 -- 172.21.15.27:0/2318665424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2a7807ab20 con 0x7f2a8c14b560 2024-03-28T18:57:13.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.086+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a8c073330 con 0x7f2a8c14b560 2024-03-28T18:57:13.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.087+0000 7f2a89600640 1 -- 172.21.15.27:0/2318665424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f2a7807a420 con 0x7f2a8c14b560 2024-03-28T18:57:13.087 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:13.087 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:57:13.087 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:57:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a6c06bbd0 msgr2=0x7f2a6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a6c06bbd0 0x7f2a6c06e090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2a80000c00 tx=0x7f2a80009040 comp rx=0 tx=0).stop 2024-03-28T18:57:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a8c14b560 msgr2=0x7f2a8c1eead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a8c14b560 0x7f2a8c1eead0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2a78001970 tx=0x7f2a8c0c17d0 comp rx=0 tx=0).stop 2024-03-28T18:57:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 shutdown_connections 2024-03-28T18:57:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a6c06bbd0 0x7f2a6c06e090 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a8c1e7f10 0x7f2a8c1e83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a8c14b560 0x7f2a8c1eead0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 --2- 172.21.15.27:0/2318665424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a74004800 0x7f2a8c1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 >> 172.21.15.27:0/2318665424 conn(0x7f2a8c06b9d0 msgr2=0x7f2a8c0a65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 shutdown_connections 2024-03-28T18:57:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:13.090+0000 7f2a93e00640 1 -- 172.21.15.27:0/2318665424 wait complete. 2024-03-28T18:57:13.110 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:57:15.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:15.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.349+0000 7f4bc5a00640 1 Processor -- start 2024-03-28T18:57:15.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.349+0000 7f4bc5a00640 1 -- start start 2024-03-28T18:57:15.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.350+0000 7f4bc5a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0150ae0 0x7f4bc00a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.351+0000 7f4bc5a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4bc00a61a0 0x7f4bc00c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.351+0000 7f4bc5a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc0150ec0 con 0x7f4bc00c6ee0 2024-03-28T18:57:15.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.351+0000 7f4bc5a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc006dc70 con 0x7f4bc00a61a0 2024-03-28T18:57:15.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.351+0000 7f4bc5a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc0153550 con 0x7f4bc0150ae0 2024-03-28T18:57:15.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.351+0000 7f4bc5000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4bc00c6ee0 0x7f4bc0153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46616/0 (socket says 172.21.15.27:46616) 2024-03-28T18:57:15.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.351+0000 7f4bc5000640 1 -- 172.21.15.27:0/2065814421 learned_addr learned my addr 172.21.15.27:0/2065814421 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.351+0000 7f4bbf400640 1 --2- 172.21.15.27:0/2065814421 >> v2:172.21.15.155:3300/0 conn(0x7f4bc00a61a0 0x7f4bc00c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.351+0000 7f4bbfe00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0150ae0 0x7f4bc00a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219033248 0 0) 0x7f4bb8001110 con 0x7f4bc00c6ee0 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba4003520 con 0x7f4bc00c6ee0 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718732502 0 0) 0x7f4ba4003520 con 0x7f4bc00c6ee0 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0150ae0 msgr2=0x7f4bc00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0150ae0 0x7f4bc00a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 >> v2:172.21.15.155:3300/0 conn(0x7f4bc00a61a0 msgr2=0x7f4bc00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 --2- 172.21.15.27:0/2065814421 >> v2:172.21.15.155:3300/0 conn(0x7f4bc00a61a0 0x7f4bc00c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc006df10 con 0x7f4bc00c6ee0 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f4bb8001840 con 0x7f4bc00c6ee0 2024-03-28T18:57:15.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 >> v1:172.21.15.27:6789/0 conn(0x7f4bc00c6ee0 legacy=0x7f4bc0153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ba40039b0 0x7f4ba4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba4004800 0x7f4ba4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0150ae0 0x7f4ba40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc006dc70 con 0x7f4ba40039b0 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc0153550 con 0x7f4ba4004800 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba400b9c0 con 0x7f4bc0150ae0 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.352+0000 7f4bbfe00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ba40039b0 0x7f4ba4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.353+0000 7f4bbf400640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba4004800 0x7f4ba4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.353+0000 7f4bc5000640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0150ae0 0x7f4ba40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.353+0000 7f4bbfe00640 1 -- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0150ae0 msgr2=0x7f4ba40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.353+0000 7f4bbfe00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0150ae0 0x7f4ba40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.353+0000 7f4bbfe00640 1 -- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba4004800 msgr2=0x7f4ba4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.353+0000 7f4bbfe00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba4004800 0x7f4ba4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.353+0000 7f4bbfe00640 1 -- 172.21.15.27:0/2065814421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4003520 con 0x7f4ba40039b0 2024-03-28T18:57:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.353+0000 7f4bbf400640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba4004800 0x7f4ba4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:57:15.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.354+0000 7f4bbfe00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ba40039b0 0x7f4ba4003dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e170 tx=0x7f4ba4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0b2fecaac5a318 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.354+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0039070 con 0x7f4ba40039b0 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.354+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb002cbf0 con 0x7f4ba40039b0 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.354+0000 7f4bbea00640 1 -- 172.21.15.27:0/2065814421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0009c10 con 0x7f4ba40039b0 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.355+0000 7f4bc5a00640 1 -- 172.21.15.27:0/2065814421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ba40039b0 msgr2=0x7f4ba4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.355+0000 7f4bc5a00640 1 --2- 172.21.15.27:0/2065814421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ba40039b0 0x7f4ba4003dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e170 tx=0x7f4ba4005830 comp rx=0 tx=0).stop 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.355+0000 7f4bc5000640 1 -- 172.21.15.27:0/2065814421 reap_dead start 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.355+0000 7f4bc5a00640 1 -- 172.21.15.27:0/2065814421 shutdown_connections 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.355+0000 7f4bc5a00640 1 -- 172.21.15.27:0/2065814421 >> 172.21.15.27:0/2065814421 conn(0x7f4bc006b0c0 msgr2=0x7f4bc006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.355+0000 7f4bc5a00640 1 -- 172.21.15.27:0/2065814421 shutdown_connections 2024-03-28T18:57:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.355+0000 7f4bc5a00640 1 -- 172.21.15.27:0/2065814421 wait complete. 2024-03-28T18:57:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.357+0000 7f4bc5a00640 1 Processor -- start 2024-03-28T18:57:15.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.357+0000 7f4bc5a00640 1 -- start start 2024-03-28T18:57:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba40039b0 0x7f4bc00aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0073060 0x7f4bc00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc01e7600 0x7f4bc01ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc006e7b0 con 0x7f4bc01e7600 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc006e4d0 con 0x7f4ba40039b0 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc01569d0 con 0x7f4bc0073060 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc01e7600 0x7f4bc01ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc01e7600 0x7f4bc01ea3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58244/0 (socket says 172.21.15.27:58244) 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.358+0000 7f4bc5000640 1 -- 172.21.15.27:0/3953026027 learned_addr learned my addr 172.21.15.27:0/3953026027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bbfe00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba40039b0 0x7f4bc00aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bbf400640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0073060 0x7f4bc00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bc5000640 1 -- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0073060 msgr2=0x7f4bc00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bc5000640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0073060 0x7f4bc00709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bc5000640 1 -- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba40039b0 msgr2=0x7f4bc00aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bc5000640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba40039b0 0x7f4bc00aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bc5000640 1 -- 172.21.15.27:0/3953026027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc01e6e60 con 0x7f4bc01e7600 2024-03-28T18:57:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bbf400640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0073060 0x7f4bc00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:57:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bc5000640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc01e7600 0x7f4bc01ea3d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00a9760 tx=0x7f4bb800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bb7e00640 1 -- 172.21.15.27:0/3953026027 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80074e0 con 0x7f4bc01e7600 2024-03-28T18:57:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bb7e00640 1 -- 172.21.15.27:0/3953026027 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb80047e0 con 0x7f4bc01e7600 2024-03-28T18:57:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bb7e00640 1 -- 172.21.15.27:0/3953026027 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8004bd0 con 0x7f4bc01e7600 2024-03-28T18:57:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.359+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc00a9790 con 0x7f4bc01e7600 2024-03-28T18:57:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.360+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc00a9d50 con 0x7f4bc01e7600 2024-03-28T18:57:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.361+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc0067dd0 con 0x7f4bc01e7600 2024-03-28T18:57:15.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.362+0000 7f4bb7e00640 1 -- 172.21.15.27:0/3953026027 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4bb801b020 con 0x7f4bc01e7600 2024-03-28T18:57:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.363+0000 7f4bb7e00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b9c06bb70 0x7f4b9c06e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.363+0000 7f4bb7e00640 1 -- 172.21.15.27:0/3953026027 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f4bb808a950 con 0x7f4bc01e7600 2024-03-28T18:57:15.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.363+0000 7f4bbfe00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b9c06bb70 0x7f4b9c06e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:15.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.364+0000 7f4bbfe00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b9c06bb70 0x7f4b9c06e030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0036000 tx=0x7f4bb002e000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:15.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.369+0000 7f4bb7e00640 1 -- 172.21.15.27:0/3953026027 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4bc01e6e60 con 0x7f4bc01e7600 2024-03-28T18:57:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.612+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bc0062500 con 0x7f4bc01e7600 2024-03-28T18:57:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.613+0000 7f4bb7e00640 1 -- 172.21.15.27:0/3953026027 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f4bc0062500 con 0x7f4bc01e7600 2024-03-28T18:57:15.613 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:15.613 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:57:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b9c06bb70 msgr2=0x7f4b9c06e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b9c06bb70 0x7f4b9c06e030 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0036000 tx=0x7f4bb002e000 comp rx=0 tx=0).stop 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc01e7600 msgr2=0x7f4bc01ea3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc01e7600 0x7f4bc01ea3d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00a9760 tx=0x7f4bb800d9d0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 shutdown_connections 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b9c06bb70 0x7f4b9c06e030 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bc01e7600 0x7f4bc01ea3d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bc0073060 0x7f4bc00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 --2- 172.21.15.27:0/3953026027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ba40039b0 0x7f4bc00aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 >> 172.21.15.27:0/3953026027 conn(0x7f4bc006b0c0 msgr2=0x7f4bc014b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:15.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.616+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 shutdown_connections 2024-03-28T18:57:15.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:15.617+0000 7f4bc5a00640 1 -- 172.21.15.27:0/3953026027 wait complete. 2024-03-28T18:57:15.635 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:57:17.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.870+0000 7f3deb200640 1 Processor -- start 2024-03-28T18:57:17.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.871+0000 7f3deb200640 1 -- start start 2024-03-28T18:57:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.872+0000 7f3deb200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3dec0c58c0 0x7f3dec0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.872+0000 7f3deb200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0be5d0 0x7f3dec155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:17.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.872+0000 7f3deb200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dec156230 con 0x7f3dec0c8ed0 2024-03-28T18:57:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.872+0000 7f3deb200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dec158bd0 con 0x7f3dec0c58c0 2024-03-28T18:57:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.872+0000 7f3deb200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec158eb0 con 0x7f3dec0be5d0 2024-03-28T18:57:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.873+0000 7f3de9e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3dec0c58c0 0x7f3dec0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.873+0000 7f3de9e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3dec0c58c0 0x7f3dec0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:54950/0 (socket says 172.21.15.27:54950) 2024-03-28T18:57:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.873+0000 7f3de9e00640 1 -- 172.21.15.27:0/1494923915 learned_addr learned my addr 172.21.15.27:0/1494923915 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.873+0000 7f3dea800640 1 --2- 172.21.15.27:0/1494923915 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0be5d0 0x7f3dec155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.873+0000 7f3de9e00640 1 -- 172.21.15.27:0/1494923915 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0be5d0 msgr2=0x7f3dec155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.873+0000 7f3de9e00640 1 --2- 172.21.15.27:0/1494923915 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0be5d0 0x7f3dec155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.873+0000 7f3de9e00640 1 -- 172.21.15.27:0/1494923915 >> v1:172.21.15.155:6789/0 conn(0x7f3dec0c8ed0 legacy=0x7f3dec1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:17.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.873+0000 7f3de9e00640 1 -- 172.21.15.27:0/1494923915 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dec159190 con 0x7f3dec0c58c0 2024-03-28T18:57:17.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.875+0000 7f3de9e00640 1 --2- 172.21.15.27:0/1494923915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 0x7f3dec0c5ca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009870 tx=0x7f3ddc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb0fe004d23c6f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.875+0000 7f3de8a00640 1 -- 172.21.15.27:0/1494923915 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc035020 con 0x7f3dec0c58c0 2024-03-28T18:57:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.875+0000 7f3de8a00640 1 -- 172.21.15.27:0/1494923915 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ddc031640 con 0x7f3dec0c58c0 2024-03-28T18:57:17.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.875+0000 7f3de8a00640 1 -- 172.21.15.27:0/1494923915 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc02d040 con 0x7f3dec0c58c0 2024-03-28T18:57:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.876+0000 7f3deb200640 1 -- 172.21.15.27:0/1494923915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 msgr2=0x7f3dec0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.876+0000 7f3deb200640 1 --2- 172.21.15.27:0/1494923915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 0x7f3dec0c5ca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc009870 tx=0x7f3ddc02b8f0 comp rx=0 tx=0).stop 2024-03-28T18:57:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.876+0000 7f3deb200640 1 -- 172.21.15.27:0/1494923915 shutdown_connections 2024-03-28T18:57:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.876+0000 7f3deb200640 1 --2- 172.21.15.27:0/1494923915 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0be5d0 0x7f3dec155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.876+0000 7f3deb200640 1 --2- 172.21.15.27:0/1494923915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 0x7f3dec0c5ca0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.876+0000 7f3deb200640 1 -- 172.21.15.27:0/1494923915 >> 172.21.15.27:0/1494923915 conn(0x7f3dec06b6f0 msgr2=0x7f3dec0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.876+0000 7f3deb200640 1 -- 172.21.15.27:0/1494923915 shutdown_connections 2024-03-28T18:57:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.876+0000 7f3deb200640 1 -- 172.21.15.27:0/1494923915 wait complete. 2024-03-28T18:57:17.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.878+0000 7f3deb200640 1 Processor -- start 2024-03-28T18:57:17.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.878+0000 7f3deb200640 1 -- start start 2024-03-28T18:57:17.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3deb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0be5d0 0x7f3dec1539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:17.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3deb200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 0x7f3dec1544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3deb200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0c8ed0 0x7f3dec0c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3deb200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec0b0820 con 0x7f3dec0be5d0 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3deb200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec159750 con 0x7f3dec0c58c0 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3deb200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec0b09a0 con 0x7f3dec0c8ed0 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3de9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0be5d0 0x7f3dec1539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3de9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0be5d0 0x7f3dec1539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60338/0 (socket says 172.21.15.27:60338) 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.879+0000 7f3de9e00640 1 -- 172.21.15.27:0/2728163238 learned_addr learned my addr 172.21.15.27:0/2728163238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3dea800640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0c8ed0 0x7f3dec0c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de9400640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 0x7f3dec1544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de9e00640 1 -- 172.21.15.27:0/2728163238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0c8ed0 msgr2=0x7f3dec0c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de9e00640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0c8ed0 0x7f3dec0c4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de9e00640 1 -- 172.21.15.27:0/2728163238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 msgr2=0x7f3dec1544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de9e00640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 0x7f3dec1544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de9e00640 1 -- 172.21.15.27:0/2728163238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dec1ec3f0 con 0x7f3dec0be5d0 2024-03-28T18:57:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de9e00640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0be5d0 0x7f3dec1539b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de2000640 1 -- 172.21.15.27:0/2728163238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc040070 con 0x7f3dec0be5d0 2024-03-28T18:57:17.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de2000640 1 -- 172.21.15.27:0/2728163238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ddc0399f0 con 0x7f3dec0be5d0 2024-03-28T18:57:17.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3de2000640 1 -- 172.21.15.27:0/2728163238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc035040 con 0x7f3dec0be5d0 2024-03-28T18:57:17.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.880+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dec0c29c0 con 0x7f3dec0be5d0 2024-03-28T18:57:17.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.881+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dec156f70 con 0x7f3dec0be5d0 2024-03-28T18:57:17.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.882+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dec04f400 con 0x7f3dec0be5d0 2024-03-28T18:57:17.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.883+0000 7f3de2000640 1 -- 172.21.15.27:0/2728163238 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3ddc04b060 con 0x7f3dec0be5d0 2024-03-28T18:57:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.884+0000 7f3de2000640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3dc806bc00 0x7f3dc806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.884+0000 7f3de2000640 1 -- 172.21.15.27:0/2728163238 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6637+0+0 (secure 0 0 0) 0x7f3ddc0ac5b0 con 0x7f3dec0be5d0 2024-03-28T18:57:17.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.884+0000 7f3de9400640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3dc806bc00 0x7f3dc806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:17.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.885+0000 7f3de9400640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3dc806bc00 0x7f3dc806e0c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4007fd0 tx=0x7f3dd4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:17.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:17.888+0000 7f3de2000640 1 -- 172.21.15.27:0/2728163238 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3dec04f400 con 0x7f3dec0be5d0 2024-03-28T18:57:18.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.132+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dec069640 con 0x7f3dec0be5d0 2024-03-28T18:57:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.133+0000 7f3de2000640 1 -- 172.21.15.27:0/2728163238 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6251 (secure 0 0 0) 0x7f3ddc07acf0 con 0x7f3dec0be5d0 2024-03-28T18:57:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:18.133 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:55:49.130048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6837/2601859893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2601859893},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601859893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4263,"qdb_cluster":[4263]},"id":2}]} 2024-03-28T18:57:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3dc806bc00 msgr2=0x7f3dc806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3dc806bc00 0x7f3dc806e0c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4007fd0 tx=0x7f3dd4009040 comp rx=0 tx=0).stop 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0be5d0 msgr2=0x7f3dec1539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0be5d0 0x7f3dec1539b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc000c00 tx=0x7f3ddc0046c0 comp rx=0 tx=0).stop 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 shutdown_connections 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3dc806bc00 0x7f3dc806e0c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3dec0c8ed0 0x7f3dec0c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dec0c58c0 0x7f3dec1544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 --2- 172.21.15.27:0/2728163238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dec0be5d0 0x7f3dec1539b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 >> 172.21.15.27:0/2728163238 conn(0x7f3dec06b6f0 msgr2=0x7f3dec0aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:18.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 shutdown_connections 2024-03-28T18:57:18.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:18.136+0000 7f3deb200640 1 -- 172.21.15.27:0/2728163238 wait complete. 2024-03-28T18:57:18.155 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:57:20.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:20.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.395+0000 7efd73e00640 1 Processor -- start 2024-03-28T18:57:20.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.396+0000 7efd73e00640 1 -- start start 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.397+0000 7efd73e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd740c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.397+0000 7efd73e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efd740be5d0 0x7efd74155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.397+0000 7efd73e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd74156230 con 0x7efd740c8ed0 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.397+0000 7efd73e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd74158bd0 con 0x7efd740be5d0 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.397+0000 7efd73e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd74158eb0 con 0x7efd740c58c0 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.397+0000 7efd6be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efd740c8ed0 0x7efd741555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35318/0 (socket says 172.21.15.27:35318) 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.397+0000 7efd6be00640 1 -- 172.21.15.27:0/953095865 learned_addr learned my addr 172.21.15.27:0/953095865 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.397+0000 7efd73400640 1 --2- 172.21.15.27:0/953095865 >> v2:172.21.15.27:3300/0 conn(0x7efd740be5d0 0x7efd74155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.398+0000 7efd72a00640 1 --2- 172.21.15.27:0/953095865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd740c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.398+0000 7efd72000640 1 -- 172.21.15.27:0/953095865 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693196314 0 0) 0x7efd74156230 con 0x7efd740c8ed0 2024-03-28T18:57:20.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.398+0000 7efd72000640 1 -- 172.21.15.27:0/953095865 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd54003520 con 0x7efd740c8ed0 2024-03-28T18:57:20.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.398+0000 7efd73400640 1 -- 172.21.15.27:0/953095865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 msgr2=0x7efd740c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:20.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.398+0000 7efd73400640 1 --2- 172.21.15.27:0/953095865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd740c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.398+0000 7efd73400640 1 -- 172.21.15.27:0/953095865 >> v1:172.21.15.27:6789/0 conn(0x7efd740c8ed0 legacy=0x7efd741555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.398+0000 7efd73400640 1 -- 172.21.15.27:0/953095865 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74159190 con 0x7efd740be5d0 2024-03-28T18:57:20.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.398+0000 7efd72a00640 1 --2- 172.21.15.27:0/953095865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd740c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.399+0000 7efd73400640 1 --2- 172.21.15.27:0/953095865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740be5d0 0x7efd74155cf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009d90 tx=0x7efd6c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94862da1985f2da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd72000640 1 -- 172.21.15.27:0/953095865 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c039070 con 0x7efd740be5d0 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd72000640 1 -- 172.21.15.27:0/953095865 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c02cbd0 con 0x7efd740be5d0 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd72000640 1 -- 172.21.15.27:0/953095865 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c034440 con 0x7efd740be5d0 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd73e00640 1 -- 172.21.15.27:0/953095865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740be5d0 msgr2=0x7efd74155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd73e00640 1 --2- 172.21.15.27:0/953095865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740be5d0 0x7efd74155cf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009d90 tx=0x7efd6c02be20 comp rx=0 tx=0).stop 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd73e00640 1 -- 172.21.15.27:0/953095865 shutdown_connections 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd73e00640 1 --2- 172.21.15.27:0/953095865 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740be5d0 0x7efd74155cf0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd73e00640 1 --2- 172.21.15.27:0/953095865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd740c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd73e00640 1 -- 172.21.15.27:0/953095865 >> 172.21.15.27:0/953095865 conn(0x7efd7406b6f0 msgr2=0x7efd740acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd73e00640 1 -- 172.21.15.27:0/953095865 shutdown_connections 2024-03-28T18:57:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.400+0000 7efd73e00640 1 -- 172.21.15.27:0/953095865 wait complete. 2024-03-28T18:57:20.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.402+0000 7efd73e00640 1 Processor -- start 2024-03-28T18:57:20.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.402+0000 7efd73e00640 1 -- start start 2024-03-28T18:57:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.403+0000 7efd73e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd740be5d0 0x7efd741eb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.403+0000 7efd73e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd741deeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.403+0000 7efd73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740c8ed0 0x7efd741e1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.403+0000 7efd73e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd740b0820 con 0x7efd740c8ed0 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.403+0000 7efd73e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74159750 con 0x7efd740be5d0 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.403+0000 7efd73e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd740b09a0 con 0x7efd740c58c0 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd73400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740c8ed0 0x7efd741e1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd73400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740c8ed0 0x7efd741e1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60364/0 (socket says 172.21.15.27:60364) 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd73400640 1 -- 172.21.15.27:0/1909677594 learned_addr learned my addr 172.21.15.27:0/1909677594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd6be00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd741deeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd72a00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd740be5d0 0x7efd741eb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd73400640 1 -- 172.21.15.27:0/1909677594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 msgr2=0x7efd741deeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd73400640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd741deeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd73400640 1 -- 172.21.15.27:0/1909677594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd740be5d0 msgr2=0x7efd741eb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd73400640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd740be5d0 0x7efd741eb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.404+0000 7efd73400640 1 -- 172.21.15.27:0/1909677594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd741eb680 con 0x7efd740c8ed0 2024-03-28T18:57:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.405+0000 7efd73400640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740c8ed0 0x7efd741e1270 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009eb0 tx=0x7efd6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.405+0000 7efd6b400640 1 -- 172.21.15.27:0/1909677594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c039050 con 0x7efd740c8ed0 2024-03-28T18:57:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.405+0000 7efd6b400640 1 -- 172.21.15.27:0/1909677594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c045070 con 0x7efd740c8ed0 2024-03-28T18:57:20.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.405+0000 7efd6b400640 1 -- 172.21.15.27:0/1909677594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c03e830 con 0x7efd740c8ed0 2024-03-28T18:57:20.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.405+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd741eb920 con 0x7efd740c8ed0 2024-03-28T18:57:20.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.405+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd741ebee0 con 0x7efd740c8ed0 2024-03-28T18:57:20.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.406+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd74069640 con 0x7efd740c8ed0 2024-03-28T18:57:20.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.407+0000 7efd6b400640 1 -- 172.21.15.27:0/1909677594 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efd6c03e9d0 con 0x7efd740c8ed0 2024-03-28T18:57:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.408+0000 7efd6b400640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd5006bbd0 0x7efd5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.408+0000 7efd6b400640 1 -- 172.21.15.27:0/1909677594 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7efd6c0addf0 con 0x7efd740c8ed0 2024-03-28T18:57:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.408+0000 7efd72a00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd5006bbd0 0x7efd5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.409+0000 7efd72a00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd5006bbd0 0x7efd5006e090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efd640096d0 tx=0x7efd64008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:20.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.413+0000 7efd6b400640 1 -- 172.21.15.27:0/1909677594 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efd6c07c5f0 con 0x7efd740c8ed0 2024-03-28T18:57:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.661+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd74156580 con 0x7efd740c8ed0 2024-03-28T18:57:20.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.662+0000 7efd6b400640 1 -- 172.21.15.27:0/1909677594 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5459 (secure 0 0 0) 0x7efd6c07bef0 con 0x7efd740c8ed0 2024-03-28T18:57:20.662 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:20.662 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:19.556827+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T18:57:20.662 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd5006bbd0 msgr2=0x7efd5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd5006bbd0 0x7efd5006e090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efd640096d0 tx=0x7efd64008040 comp rx=0 tx=0).stop 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740c8ed0 msgr2=0x7efd741e1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740c8ed0 0x7efd741e1270 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009eb0 tx=0x7efd6c004a40 comp rx=0 tx=0).stop 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 shutdown_connections 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd5006bbd0 0x7efd5006e090 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd740c8ed0 0x7efd741e1270 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd740c58c0 0x7efd741deeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 --2- 172.21.15.27:0/1909677594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd740be5d0 0x7efd741eb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 >> 172.21.15.27:0/1909677594 conn(0x7efd7406b6f0 msgr2=0x7efd74157d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 shutdown_connections 2024-03-28T18:57:20.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.665+0000 7efd73e00640 1 -- 172.21.15.27:0/1909677594 wait complete. 2024-03-28T18:57:20.684 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-03-28T18:57:20.684 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T18:57:20.684 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:20.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.906+0000 7fee5f800640 1 Processor -- start 2024-03-28T18:57:20.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.907+0000 7fee5f800640 1 -- start start 2024-03-28T18:57:20.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.908+0000 7fee5f800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580c1f50 0x7fee580ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.908+0000 7fee5f800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fee580abd90 0x7fee580aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.908+0000 7fee5f800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5806d8a0 con 0x7fee580c5e80 2024-03-28T18:57:20.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.908+0000 7fee5f800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee580ac220 con 0x7fee580abd90 2024-03-28T18:57:20.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.908+0000 7fee5f800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5806d650 con 0x7fee580c1f50 2024-03-28T18:57:20.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.909+0000 7fee5d800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fee580abd90 0x7fee580aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.909+0000 7fee5d800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fee580abd90 0x7fee580aa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:54976/0 (socket says 172.21.15.27:54976) 2024-03-28T18:57:20.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.909+0000 7fee4fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580c1f50 0x7fee580ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.909+0000 7fee5d800640 1 -- 172.21.15.27:0/1123132182 learned_addr learned my addr 172.21.15.27:0/1123132182 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:20.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.910+0000 7fee5d800640 1 -- 172.21.15.27:0/1123132182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580c1f50 msgr2=0x7fee580ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:20.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.910+0000 7fee5d800640 1 --2- 172.21.15.27:0/1123132182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580c1f50 0x7fee580ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.910+0000 7fee5d800640 1 -- 172.21.15.27:0/1123132182 >> v1:172.21.15.155:6789/0 conn(0x7fee580c5e80 legacy=0x7fee580c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.910+0000 7fee5d800640 1 -- 172.21.15.27:0/1123132182 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5806db80 con 0x7fee580abd90 2024-03-28T18:57:20.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.910+0000 7fee4fe00640 1 --2- 172.21.15.27:0/1123132182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580c1f50 0x7fee580ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:57:20.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.911+0000 7fee5d800640 1 --2- 172.21.15.27:0/1123132182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580abd90 0x7fee580aa570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fee50009e50 tx=0x7fee5002e050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5590a074bdeb9f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee57e00640 1 -- 172.21.15.27:0/1123132182 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5002ebf0 con 0x7fee580abd90 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee57e00640 1 -- 172.21.15.27:0/1123132182 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5002ed90 con 0x7fee580abd90 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee57e00640 1 -- 172.21.15.27:0/1123132182 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50036610 con 0x7fee580abd90 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee5f800640 1 -- 172.21.15.27:0/1123132182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580abd90 msgr2=0x7fee580aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee5f800640 1 --2- 172.21.15.27:0/1123132182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580abd90 0x7fee580aa570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fee50009e50 tx=0x7fee5002e050 comp rx=0 tx=0).stop 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee5f800640 1 -- 172.21.15.27:0/1123132182 shutdown_connections 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee5f800640 1 --2- 172.21.15.27:0/1123132182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580abd90 0x7fee580aa570 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee5f800640 1 --2- 172.21.15.27:0/1123132182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580c1f50 0x7fee580ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee5f800640 1 -- 172.21.15.27:0/1123132182 >> 172.21.15.27:0/1123132182 conn(0x7fee5806b9d0 msgr2=0x7fee580c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee5f800640 1 -- 172.21.15.27:0/1123132182 shutdown_connections 2024-03-28T18:57:20.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.912+0000 7fee5f800640 1 -- 172.21.15.27:0/1123132182 wait complete. 2024-03-28T18:57:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.914+0000 7fee5f800640 1 Processor -- start 2024-03-28T18:57:20.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.914+0000 7fee5f800640 1 -- start start 2024-03-28T18:57:20.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.915+0000 7fee5f800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580abd90 0x7fee581e1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.915+0000 7fee5f800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee580c1f50 0x7fee581e1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.915+0000 7fee5f800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580c5e80 0x7fee5814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.915+0000 7fee5f800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee580c6260 con 0x7fee580c1f50 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.915+0000 7fee5f800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5806e1d0 con 0x7fee580c5e80 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.915+0000 7fee5f800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee580c63e0 con 0x7fee580abd90 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee4fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee580c1f50 0x7fee581e1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee4fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee580c1f50 0x7fee581e1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60378/0 (socket says 172.21.15.27:60378) 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee4fe00640 1 -- 172.21.15.27:0/1485344607 learned_addr learned my addr 172.21.15.27:0/1485344607 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee5d800640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580c5e80 0x7fee5814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee5ce00640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580abd90 0x7fee581e1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee4fe00640 1 -- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580abd90 msgr2=0x7fee581e1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee4fe00640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580abd90 0x7fee581e1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee4fe00640 1 -- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580c5e80 msgr2=0x7fee5814e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee4fe00640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580c5e80 0x7fee5814e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:20.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee4fe00640 1 -- 172.21.15.27:0/1485344607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee581eafb0 con 0x7fee580c1f50 2024-03-28T18:57:20.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee5d800640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580c5e80 0x7fee5814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:57:20.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.916+0000 7fee5ce00640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580abd90 0x7fee581e1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:57:20.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.917+0000 7fee4fe00640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee580c1f50 0x7fee581e1790 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fee4000e930 tx=0x7fee4000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:20.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.917+0000 7fee55600640 1 -- 172.21.15.27:0/1485344607 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40017070 con 0x7fee580c1f50 2024-03-28T18:57:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.917+0000 7fee55600640 1 -- 172.21.15.27:0/1485344607 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4000ca00 con 0x7fee580c1f50 2024-03-28T18:57:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.917+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee581eb270 con 0x7fee580c1f50 2024-03-28T18:57:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.917+0000 7fee55600640 1 -- 172.21.15.27:0/1485344607 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee4000cdf0 con 0x7fee580c1f50 2024-03-28T18:57:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.917+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee580a82c0 con 0x7fee580c1f50 2024-03-28T18:57:20.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.918+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5804f400 con 0x7fee580c1f50 2024-03-28T18:57:20.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.920+0000 7fee55600640 1 -- 172.21.15.27:0/1485344607 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fee4001a070 con 0x7fee580c1f50 2024-03-28T18:57:20.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.921+0000 7fee55600640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee3406bbd0 0x7fee3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:20.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.921+0000 7fee55600640 1 -- 172.21.15.27:0/1485344607 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fee4008acb0 con 0x7fee580c1f50 2024-03-28T18:57:20.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.922+0000 7fee5ce00640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee3406bbd0 0x7fee3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:20.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.922+0000 7fee5ce00640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee3406bbd0 0x7fee3406e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fee48000f60 tx=0x7fee48009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:20.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:20.926+0000 7fee55600640 1 -- 172.21.15.27:0/1485344607 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fee40059430 con 0x7fee580c1f50 2024-03-28T18:57:21.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.173+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee58068180 con 0x7fee580c1f50 2024-03-28T18:57:21.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.175+0000 7fee55600640 1 -- 172.21.15.27:0/1485344607 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5459 (secure 0 0 0) 0x7fee40058d30 con 0x7fee580c1f50 2024-03-28T18:57:21.174 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:21.175 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:19.556827+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T18:57:21.175 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-03-28T18:57:21.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee3406bbd0 msgr2=0x7fee3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee3406bbd0 0x7fee3406e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fee48000f60 tx=0x7fee48009040 comp rx=0 tx=0).stop 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee580c1f50 msgr2=0x7fee581e1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee580c1f50 0x7fee581e1790 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fee4000e930 tx=0x7fee4000edf0 comp rx=0 tx=0).stop 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 shutdown_connections 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee3406bbd0 0x7fee3406e090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee580c5e80 0x7fee5814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee580c1f50 0x7fee581e1790 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 --2- 172.21.15.27:0/1485344607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee580abd90 0x7fee581e1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 >> 172.21.15.27:0/1485344607 conn(0x7fee5806b9d0 msgr2=0x7fee58071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 shutdown_connections 2024-03-28T18:57:21.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:21.178+0000 7fee5f800640 1 -- 172.21.15.27:0/1485344607 wait complete. 2024-03-28T18:57:21.197 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4273": { "addr": "172.21.15.155:6834/1270773218", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 1270773218, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 1270773218, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4273, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T18:55:38.022816+0000", "qdb_cluster": [ 4273 ], "qdb_leader": 4273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4273 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4260": { "addr": "172.21.15.27:6836/3119083312", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 3119083312, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 3119083312, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4260, "incarnation": 16, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T18:57:19.556827+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4260 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T18:57:23.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:23.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.432+0000 7f3a57200640 1 Processor -- start 2024-03-28T18:57:23.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.433+0000 7f3a57200640 1 -- start start 2024-03-28T18:57:23.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.434+0000 7f3a57200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3a58156250 0x7f3a58159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:23.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.434+0000 7f3a57200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a580c5710 con 0x7f3a580c6cc0 2024-03-28T18:57:23.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.434+0000 7f3a57200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a580c59f0 con 0x7f3a58152a70 2024-03-28T18:57:23.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.434+0000 7f3a57200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a581556b0 con 0x7f3a58156250 2024-03-28T18:57:23.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.434+0000 7f3a55400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3a580c6cc0 0x7f3a58155b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35324/0 (socket says 172.21.15.27:35324) 2024-03-28T18:57:23.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.434+0000 7f3a55400640 1 -- 172.21.15.27:0/1036279467 learned_addr learned my addr 172.21.15.27:0/1036279467 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.434+0000 7f3a56800640 1 --2- 172.21.15.27:0/1036279467 >> v2:172.21.15.155:3300/0 conn(0x7f3a58156250 0x7f3a58159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456576743 0 0) 0x7f3a580c5710 con 0x7f3a580c6cc0 2024-03-28T18:57:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a3c003520 con 0x7f3a580c6cc0 2024-03-28T18:57:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 265614591 0 0) 0x7f3a3c003520 con 0x7f3a580c6cc0 2024-03-28T18:57:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 >> v2:172.21.15.155:3300/0 conn(0x7f3a58156250 msgr2=0x7f3a58159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 --2- 172.21.15.27:0/1036279467 >> v2:172.21.15.155:3300/0 conn(0x7f3a58156250 0x7f3a58159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 >> v1:172.21.15.155:6789/0 conn(0x7f3a58152a70 legacy=0x7f3a580cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a580c5cd0 con 0x7f3a580c6cc0 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f3a44001670 con 0x7f3a580c6cc0 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 >> v1:172.21.15.27:6789/0 conn(0x7f3a580c6cc0 legacy=0x7f3a58155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a3c003a00 0x7f3a3c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.435+0000 7f3a54a00640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c004800 0x7f3a3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.436+0000 7f3a54a00640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c005210 0x7f3a3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.436+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a581556b0 con 0x7f3a3c004800 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.436+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c00c160 con 0x7f3a3c003a00 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.436+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c00c360 con 0x7f3a3c005210 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.436+0000 7f3a55e00640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a3c003a00 0x7f3a3c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.436+0000 7f3a55400640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c004800 0x7f3a3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.436+0000 7f3a56800640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c005210 0x7f3a3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.437+0000 7f3a55400640 1 -- 172.21.15.27:0/1036279467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c005210 msgr2=0x7f3a3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.437+0000 7f3a55400640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c005210 0x7f3a3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.437+0000 7f3a55400640 1 -- 172.21.15.27:0/1036279467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a3c003a00 msgr2=0x7f3a3c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:23.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.437+0000 7f3a55400640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a3c003a00 0x7f3a3c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.437+0000 7f3a55400640 1 -- 172.21.15.27:0/1036279467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c003520 con 0x7f3a3c004800 2024-03-28T18:57:23.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.437+0000 7f3a56800640 1 -- 172.21.15.27:0/1036279467 reap_dead start 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.438+0000 7f3a55400640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c004800 0x7f3a3c004c80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3a580c7130 tx=0x7f3a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ffcfee4d5bef58 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.439+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44007c20 con 0x7f3a3c004800 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.439+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a44030070 con 0x7f3a3c004800 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.439+0000 7f3a54a00640 1 -- 172.21.15.27:0/1036279467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44005d70 con 0x7f3a3c004800 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.439+0000 7f3a57200640 1 -- 172.21.15.27:0/1036279467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c004800 msgr2=0x7f3a3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.439+0000 7f3a57200640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c004800 0x7f3a3c004c80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3a580c7130 tx=0x7f3a3c005830 comp rx=0 tx=0).stop 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.439+0000 7f3a57200640 1 -- 172.21.15.27:0/1036279467 shutdown_connections 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.439+0000 7f3a57200640 1 --2- 172.21.15.27:0/1036279467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a3c004800 0x7f3a3c004c80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.439+0000 7f3a57200640 1 -- 172.21.15.27:0/1036279467 >> 172.21.15.27:0/1036279467 conn(0x7f3a5806b2a0 msgr2=0x7f3a5806b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:23.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.440+0000 7f3a57200640 1 -- 172.21.15.27:0/1036279467 shutdown_connections 2024-03-28T18:57:23.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.440+0000 7f3a57200640 1 -- 172.21.15.27:0/1036279467 wait complete. 2024-03-28T18:57:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.442+0000 7f3a57200640 1 Processor -- start 2024-03-28T18:57:23.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.443+0000 7f3a57200640 1 -- start start 2024-03-28T18:57:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a57200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c004800 0x7f3a580c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a57200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a5814c340 0x7f3a581eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a57200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a580c1900 0x7f3a581eefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a57200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a580c8f50 con 0x7f3a5814c340 2024-03-28T18:57:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a57200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a580c8dd0 con 0x7f3a580c1900 2024-03-28T18:57:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a57200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a580c5fb0 con 0x7f3a3c004800 2024-03-28T18:57:23.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a55400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a5814c340 0x7f3a581eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a55400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a5814c340 0x7f3a581eea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60396/0 (socket says 172.21.15.27:60396) 2024-03-28T18:57:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.444+0000 7f3a55400640 1 -- 172.21.15.27:0/76067442 learned_addr learned my addr 172.21.15.27:0/76067442 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.445+0000 7f3a55e00640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c004800 0x7f3a580c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:23.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.445+0000 7f3a56800640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a580c1900 0x7f3a581eefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.445+0000 7f3a55400640 1 -- 172.21.15.27:0/76067442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c004800 msgr2=0x7f3a580c13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.445+0000 7f3a55400640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c004800 0x7f3a580c13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.445+0000 7f3a55400640 1 -- 172.21.15.27:0/76067442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a580c1900 msgr2=0x7f3a581eefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.445+0000 7f3a55400640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a580c1900 0x7f3a581eefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.445+0000 7f3a55400640 1 -- 172.21.15.27:0/76067442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a581ef6c0 con 0x7f3a5814c340 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.446+0000 7f3a55400640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a5814c340 0x7f3a581eea60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c1a0 tx=0x7f3a58155580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.446+0000 7f3a4e000640 1 -- 172.21.15.27:0/76067442 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44046070 con 0x7f3a5814c340 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.446+0000 7f3a4e000640 1 -- 172.21.15.27:0/76067442 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a44000ed0 con 0x7f3a5814c340 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.446+0000 7f3a4e000640 1 -- 172.21.15.27:0/76067442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44001b60 con 0x7f3a5814c340 2024-03-28T18:57:23.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.446+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a58156e70 con 0x7f3a5814c340 2024-03-28T18:57:23.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.446+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a580c4cb0 con 0x7f3a5814c340 2024-03-28T18:57:23.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.448+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a58067dd0 con 0x7f3a5814c340 2024-03-28T18:57:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.450+0000 7f3a4e000640 1 -- 172.21.15.27:0/76067442 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3a44049070 con 0x7f3a5814c340 2024-03-28T18:57:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.451+0000 7f3a4e000640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3a3806bbd0 0x7f3a3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.451+0000 7f3a4e000640 1 -- 172.21.15.27:0/76067442 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f3a440abf40 con 0x7f3a5814c340 2024-03-28T18:57:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.451+0000 7f3a55e00640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3a3806bbd0 0x7f3a3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:23.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.452+0000 7f3a55e00640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3a3806bbd0 0x7f3a3806e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a48001760 tx=0x7f3a48009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:23.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.459+0000 7f3a4e000640 1 -- 172.21.15.27:0/76067442 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3a4407a190 con 0x7f3a5814c340 2024-03-28T18:57:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.720+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a58069640 con 0x7f3a5814c340 2024-03-28T18:57:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.720+0000 7f3a4e000640 1 -- 172.21.15.27:0/76067442 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5463 (secure 0 0 0) 0x7f3a4407e230 con 0x7f3a5814c340 2024-03-28T18:57:23.720 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:23.720 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:57:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-03-28T18:57:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3a3806bbd0 msgr2=0x7f3a3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3a3806bbd0 0x7f3a3806e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a48001760 tx=0x7f3a48009040 comp rx=0 tx=0).stop 2024-03-28T18:57:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a5814c340 msgr2=0x7f3a581eea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a5814c340 0x7f3a581eea60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c1a0 tx=0x7f3a58155580 comp rx=0 tx=0).stop 2024-03-28T18:57:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 shutdown_connections 2024-03-28T18:57:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3a3806bbd0 0x7f3a3806e090 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a580c1900 0x7f3a581eefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a5814c340 0x7f3a581eea60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 --2- 172.21.15.27:0/76067442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a3c004800 0x7f3a580c13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.723+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 >> 172.21.15.27:0/76067442 conn(0x7f3a5806b2a0 msgr2=0x7f3a58154920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.724+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 shutdown_connections 2024-03-28T18:57:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:23.724+0000 7f3a57200640 1 -- 172.21.15.27:0/76067442 wait complete. 2024-03-28T18:57:23.741 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4263 2024-03-28T18:57:23.742 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.c 2024-03-28T18:57:24.173 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T18:57:24.172+0000 7fabb1400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:57:24.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:57:24.943 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:57:31.743 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-03-28T18:57:31.743 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T18:57:31.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T18:57:31.754 INFO:tasks.ceph.mds.c:Started 2024-03-28T18:57:31.754 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:31.917 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T18:57:31.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.976+0000 7f6621800640 1 Processor -- start 2024-03-28T18:57:31.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.977+0000 7f6621800640 1 -- start start 2024-03-28T18:57:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.978+0000 7f6621800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f661c14c000 0x7f661c06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.978+0000 7f6621800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c06e320 0x7f661c0c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.978+0000 7f6621800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c06df30 con 0x7f661c0c2220 2024-03-28T18:57:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.978+0000 7f6621800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c1554e0 con 0x7f661c14c000 2024-03-28T18:57:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.978+0000 7f6621800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c14c3e0 con 0x7f661c06e320 2024-03-28T18:57:31.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.978+0000 7f661be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f661c14c000 0x7f661c06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.978+0000 7f6620e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f661c0c2220 0x7f661c158ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33872/0 (socket says 172.21.15.27:33872) 2024-03-28T18:57:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.979+0000 7f6620e00640 1 -- 172.21.15.27:0/70517913 learned_addr learned my addr 172.21.15.27:0/70517913 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.979+0000 7f661b400640 1 --2- 172.21.15.27:0/70517913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c06e320 0x7f661c0c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.979+0000 7f661be00640 1 -- 172.21.15.27:0/70517913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c06e320 msgr2=0x7f661c0c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.979+0000 7f661be00640 1 --2- 172.21.15.27:0/70517913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c06e320 0x7f661c0c1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.979+0000 7f661be00640 1 -- 172.21.15.27:0/70517913 >> v1:172.21.15.27:6789/0 conn(0x7f661c0c2220 legacy=0x7f661c158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:31.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.979+0000 7f661be00640 1 -- 172.21.15.27:0/70517913 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c155780 con 0x7f661c14c000 2024-03-28T18:57:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.981+0000 7f661be00640 1 --2- 172.21.15.27:0/70517913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c06ebc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f660c0097a0 tx=0x7f660c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3f3490222c98e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:31.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.981+0000 7f661aa00640 1 -- 172.21.15.27:0/70517913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c031780 con 0x7f661c14c000 2024-03-28T18:57:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.981+0000 7f661aa00640 1 -- 172.21.15.27:0/70517913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c037070 con 0x7f661c14c000 2024-03-28T18:57:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.981+0000 7f661aa00640 1 -- 172.21.15.27:0/70517913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c031b40 con 0x7f661c14c000 2024-03-28T18:57:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.982+0000 7f6621800640 1 -- 172.21.15.27:0/70517913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 msgr2=0x7f661c06ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:31.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.982+0000 7f6621800640 1 --2- 172.21.15.27:0/70517913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c06ebc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f660c0097a0 tx=0x7f660c02b820 comp rx=0 tx=0).stop 2024-03-28T18:57:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.982+0000 7f6621800640 1 -- 172.21.15.27:0/70517913 shutdown_connections 2024-03-28T18:57:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.982+0000 7f6621800640 1 --2- 172.21.15.27:0/70517913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c06e320 0x7f661c0c1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.982+0000 7f6621800640 1 --2- 172.21.15.27:0/70517913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c06ebc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.982+0000 7f6621800640 1 -- 172.21.15.27:0/70517913 >> 172.21.15.27:0/70517913 conn(0x7f661c06ced0 msgr2=0x7f661c06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.982+0000 7f6621800640 1 -- 172.21.15.27:0/70517913 shutdown_connections 2024-03-28T18:57:31.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.982+0000 7f6621800640 1 -- 172.21.15.27:0/70517913 wait complete. 2024-03-28T18:57:31.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.984+0000 7f6621800640 1 Processor -- start 2024-03-28T18:57:31.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.984+0000 7f6621800640 1 -- start start 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.985+0000 7f6621800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f661c06e320 0x7f661c1ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.985+0000 7f6621800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c0c2220 0x7f661c0c0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.985+0000 7f6621800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c0c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.985+0000 7f6621800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c156020 con 0x7f661c14c000 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.985+0000 7f6621800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c155d40 con 0x7f661c06e320 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.985+0000 7f6621800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c1591e0 con 0x7f661c0c2220 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f6620e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c0c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f6620e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c0c50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58312/0 (socket says 172.21.15.27:58312) 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f6620e00640 1 -- 172.21.15.27:0/1705986384 learned_addr learned my addr 172.21.15.27:0/1705986384 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f661b400640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c0c2220 0x7f661c0c0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f661be00640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f661c06e320 0x7f661c1ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f6620e00640 1 -- 172.21.15.27:0/1705986384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c0c2220 msgr2=0x7f661c0c0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:31.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f6620e00640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c0c2220 0x7f661c0c0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:31.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f6620e00640 1 -- 172.21.15.27:0/1705986384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f661c06e320 msgr2=0x7f661c1ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:31.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f6620e00640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f661c06e320 0x7f661c1ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:31.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.986+0000 7f6620e00640 1 -- 172.21.15.27:0/1705986384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0c1730 con 0x7f661c14c000 2024-03-28T18:57:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.987+0000 7f6620e00640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c0c50e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f661400cbf0 tx=0x7f661400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.987+0000 7f6613e00640 1 -- 172.21.15.27:0/1705986384 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614017070 con 0x7f661c14c000 2024-03-28T18:57:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.987+0000 7f6613e00640 1 -- 172.21.15.27:0/1705986384 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661400ad40 con 0x7f661c14c000 2024-03-28T18:57:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.987+0000 7f6613e00640 1 -- 172.21.15.27:0/1705986384 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614012600 con 0x7f661c14c000 2024-03-28T18:57:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.987+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c1eb630 con 0x7f661c14c000 2024-03-28T18:57:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.987+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c1ebbf0 con 0x7f661c14c000 2024-03-28T18:57:31.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.989+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c069640 con 0x7f661c14c000 2024-03-28T18:57:31.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.990+0000 7f6613e00640 1 -- 172.21.15.27:0/1705986384 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f661401a070 con 0x7f661c14c000 2024-03-28T18:57:31.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.992+0000 7f6613e00640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65fc06bbd0 0x7f65fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:31.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.992+0000 7f6613e00640 1 -- 172.21.15.27:0/1705986384 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f661408aaa0 con 0x7f661c14c000 2024-03-28T18:57:31.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.992+0000 7f661be00640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65fc06bbd0 0x7f65fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:31.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.993+0000 7f661be00640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65fc06bbd0 0x7f65fc06e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f660c036000 tx=0x7f660c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:31.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:31.997+0000 7f6613e00640 1 -- 172.21.15.27:0/1705986384 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6614058bd0 con 0x7f661c14c000 2024-03-28T18:57:32.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.263+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f661c0c1ec0 con 0x7f661c14c000 2024-03-28T18:57:32.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.264+0000 7f6613e00640 1 -- 172.21.15.27:0/1705986384 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5463 (secure 0 0 0) 0x7f661405cc70 con 0x7f661c14c000 2024-03-28T18:57:32.264 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:32.265 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:57:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-03-28T18:57:32.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.266+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65fc06bbd0 msgr2=0x7f65fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:32.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.266+0000 7f6621800640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65fc06bbd0 0x7f65fc06e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f660c036000 tx=0x7f660c03d040 comp rx=0 tx=0).stop 2024-03-28T18:57:32.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.266+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 msgr2=0x7f661c0c50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:32.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.266+0000 7f6621800640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c0c50e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f661400cbf0 tx=0x7f661400a430 comp rx=0 tx=0).stop 2024-03-28T18:57:32.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.266+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 shutdown_connections 2024-03-28T18:57:32.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.266+0000 7f6621800640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65fc06bbd0 0x7f65fc06e090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:32.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.266+0000 7f6621800640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661c14c000 0x7f661c0c50e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:32.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.267+0000 7f6621800640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f661c0c2220 0x7f661c0c0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:32.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.267+0000 7f6621800640 1 --2- 172.21.15.27:0/1705986384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f661c06e320 0x7f661c1ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:32.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.267+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 >> 172.21.15.27:0/1705986384 conn(0x7f661c06ced0 msgr2=0x7f661c153d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:32.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.267+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 shutdown_connections 2024-03-28T18:57:32.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:32.267+0000 7f6621800640 1 -- 172.21.15.27:0/1705986384 wait complete. 2024-03-28T18:57:32.284 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T18:57:34.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.489+0000 7f0279a00640 1 Processor -- start 2024-03-28T18:57:34.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.489+0000 7f0279a00640 1 -- start start 2024-03-28T18:57:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.490+0000 7f0279a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274159650 0x7f02740c1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.490+0000 7f0279a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0274155d60 0x7f02741561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.491+0000 7f0279a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02740c7a10 con 0x7f0274158cb0 2024-03-28T18:57:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.491+0000 7f0279a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02740c7170 con 0x7f0274155d60 2024-03-28T18:57:34.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.491+0000 7f0279a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02740c7450 con 0x7f0274159650 2024-03-28T18:57:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.491+0000 7f0272a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274159650 0x7f02740c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.491+0000 7f0272a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274159650 0x7f02740c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:36774/0 (socket says 172.21.15.27:36774) 2024-03-28T18:57:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.491+0000 7f0272a00640 1 -- 172.21.15.27:0/1119364546 learned_addr learned my addr 172.21.15.27:0/1119364546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:34.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.492+0000 7f0273400640 1 --2- 172.21.15.27:0/1119364546 >> v2:172.21.15.155:3300/0 conn(0x7f0274155d60 0x7f02741561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.492+0000 7f0272a00640 1 -- 172.21.15.27:0/1119364546 >> v2:172.21.15.155:3300/0 conn(0x7f0274155d60 msgr2=0x7f02741561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.492+0000 7f0272a00640 1 --2- 172.21.15.27:0/1119364546 >> v2:172.21.15.155:3300/0 conn(0x7f0274155d60 0x7f02741561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.492+0000 7f0272a00640 1 -- 172.21.15.27:0/1119364546 >> v1:172.21.15.155:6789/0 conn(0x7f0274158cb0 legacy=0x7f027406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:34.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.492+0000 7f0272a00640 1 -- 172.21.15.27:0/1119364546 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02740c7730 con 0x7f0274159650 2024-03-28T18:57:34.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.494+0000 7f0272a00640 1 --2- 172.21.15.27:0/1119364546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274159650 0x7f02740c1570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0264009870 tx=0x7f026402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77cf5fe3c03c498 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:34.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.494+0000 7f0272000640 1 -- 172.21.15.27:0/1119364546 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264034070 con 0x7f0274159650 2024-03-28T18:57:34.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.494+0000 7f0272000640 1 -- 172.21.15.27:0/1119364546 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0264030840 con 0x7f0274159650 2024-03-28T18:57:34.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.494+0000 7f0272000640 1 -- 172.21.15.27:0/1119364546 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264030c10 con 0x7f0274159650 2024-03-28T18:57:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.495+0000 7f0279a00640 1 -- 172.21.15.27:0/1119364546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274159650 msgr2=0x7f02740c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.495+0000 7f0279a00640 1 --2- 172.21.15.27:0/1119364546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274159650 0x7f02740c1570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0264009870 tx=0x7f026402b980 comp rx=0 tx=0).stop 2024-03-28T18:57:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.495+0000 7f0279a00640 1 -- 172.21.15.27:0/1119364546 shutdown_connections 2024-03-28T18:57:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.495+0000 7f0279a00640 1 --2- 172.21.15.27:0/1119364546 >> v2:172.21.15.155:3300/0 conn(0x7f0274155d60 0x7f02741561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.495+0000 7f0279a00640 1 --2- 172.21.15.27:0/1119364546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274159650 0x7f02740c1570 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.495+0000 7f0279a00640 1 -- 172.21.15.27:0/1119364546 >> 172.21.15.27:0/1119364546 conn(0x7f02740c4750 msgr2=0x7f02740c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.495+0000 7f0279a00640 1 -- 172.21.15.27:0/1119364546 shutdown_connections 2024-03-28T18:57:34.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.495+0000 7f0279a00640 1 -- 172.21.15.27:0/1119364546 wait complete. 2024-03-28T18:57:34.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.498+0000 7f0279a00640 1 Processor -- start 2024-03-28T18:57:34.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.498+0000 7f0279a00640 1 -- start start 2024-03-28T18:57:34.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f0279a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0274155d60 0x7f027406ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:34.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f0279a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274158cb0 0x7f0274150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:34.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f0279a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274159650 0x7f02741514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:34.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f0279a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02740c1c30 con 0x7f0274159650 2024-03-28T18:57:34.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f0279a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02740c1ab0 con 0x7f0274155d60 2024-03-28T18:57:34.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f0279a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027406c7e0 con 0x7f0274158cb0 2024-03-28T18:57:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f0272a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0274155d60 0x7f027406ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f0273400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274159650 0x7f02741514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.499+0000 7f026a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274158cb0 0x7f0274150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.500+0000 7f0273400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274159650 0x7f02741514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58318/0 (socket says 172.21.15.27:58318) 2024-03-28T18:57:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.500+0000 7f0272a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0274155d60 0x7f027406ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:49858/0 (socket says 172.21.15.27:49858) 2024-03-28T18:57:34.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.500+0000 7f0273400640 1 -- 172.21.15.27:0/1408750319 learned_addr learned my addr 172.21.15.27:0/1408750319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.500+0000 7f026a000640 1 -- 172.21.15.27:0/1408750319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0274155d60 msgr2=0x7f027406ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.500+0000 7f026a000640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0274155d60 0x7f027406ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.500+0000 7f026a000640 1 -- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274159650 msgr2=0x7f02741514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.500+0000 7f026a000640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274159650 0x7f02741514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.500+0000 7f026a000640 1 -- 172.21.15.27:0/1408750319 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02741e21a0 con 0x7f0274158cb0 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.501+0000 7f026a000640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274158cb0 0x7f0274150f70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c3c0 tx=0x7f025c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.501+0000 7f026b400640 1 -- 172.21.15.27:0/1408750319 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c0052d0 con 0x7f0274158cb0 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.501+0000 7f026b400640 1 -- 172.21.15.27:0/1408750319 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c005490 con 0x7f0274158cb0 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.501+0000 7f026b400640 1 -- 172.21.15.27:0/1408750319 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c007260 con 0x7f0274158cb0 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.501+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02740a8ac0 con 0x7f0274158cb0 2024-03-28T18:57:34.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.501+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02741e2760 con 0x7f0274158cb0 2024-03-28T18:57:34.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.502+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02741e2b40 con 0x7f0274158cb0 2024-03-28T18:57:34.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.505+0000 7f026b400640 1 -- 172.21.15.27:0/1408750319 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f025c007400 con 0x7f0274158cb0 2024-03-28T18:57:34.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.506+0000 7f026b400640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025006bbd0 0x7f025006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:34.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.506+0000 7f026b400640 1 -- 172.21.15.27:0/1408750319 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f025c08ac10 con 0x7f0274158cb0 2024-03-28T18:57:34.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.506+0000 7f0272a00640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025006bbd0 0x7f025006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:34.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.507+0000 7f0272a00640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025006bbd0 0x7f025006e090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0264036000 tx=0x7f026403b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.512+0000 7f026b400640 1 -- 172.21.15.27:0/1408750319 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f02741e2b40 con 0x7f0274158cb0 2024-03-28T18:57:34.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.796+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f027404f400 con 0x7f0274158cb0 2024-03-28T18:57:34.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.798+0000 7f026b400640 1 -- 172.21.15.27:0/1408750319 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f025c058da0 con 0x7f0274158cb0 2024-03-28T18:57:34.799 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:34.799 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:57:34.800 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-03-28T18:57:34.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.801+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025006bbd0 msgr2=0x7f025006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:34.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.801+0000 7f0279a00640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025006bbd0 0x7f025006e090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0264036000 tx=0x7f026403b040 comp rx=0 tx=0).stop 2024-03-28T18:57:34.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.801+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274158cb0 msgr2=0x7f0274150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.801+0000 7f0279a00640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274158cb0 0x7f0274150f70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c3c0 tx=0x7f025c00c880 comp rx=0 tx=0).stop 2024-03-28T18:57:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.802+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 shutdown_connections 2024-03-28T18:57:34.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.802+0000 7f0279a00640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025006bbd0 0x7f025006e090 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.802+0000 7f0279a00640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0274159650 0x7f02741514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.802+0000 7f0279a00640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0274158cb0 0x7f0274150f70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.802+0000 7f0279a00640 1 --2- 172.21.15.27:0/1408750319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0274155d60 0x7f027406ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:34.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.802+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 >> 172.21.15.27:0/1408750319 conn(0x7f02740c4750 msgr2=0x7f02740c2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:34.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.802+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 shutdown_connections 2024-03-28T18:57:34.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:34.802+0000 7f0279a00640 1 -- 172.21.15.27:0/1408750319 wait complete. 2024-03-28T18:57:34.820 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-03-28T18:57:34.820 INFO:tasks.mds_thrash.fs.[b]:waiting for 71 secs before thrashing 2024-03-28T18:57:53.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:53.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.764+0000 7efda7e00640 1 Processor -- start 2024-03-28T18:57:53.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.765+0000 7efda7e00640 1 -- start start 2024-03-28T18:57:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.766+0000 7efda7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efda8156120 0x7efda806e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.766+0000 7efda7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156500 0x7efda80c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.767+0000 7efda7e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda8159c50 con 0x7efda80c5320 2024-03-28T18:57:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.767+0000 7efda7e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda80c4d70 con 0x7efda8156120 2024-03-28T18:57:53.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.767+0000 7efda7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda80c1030 con 0x7efda8156500 2024-03-28T18:57:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.767+0000 7efda7400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156500 0x7efda80c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.767+0000 7efda7400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156500 0x7efda80c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:34780/0 (socket says 172.21.15.27:34780) 2024-03-28T18:57:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.767+0000 7efda7400640 1 -- 172.21.15.27:0/2427761593 learned_addr learned my addr 172.21.15.27:0/2427761593 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:53.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.767+0000 7efda6a00640 1 --2- 172.21.15.27:0/2427761593 >> v2:172.21.15.27:3300/0 conn(0x7efda8156120 0x7efda806e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:53.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.768+0000 7efda7400640 1 -- 172.21.15.27:0/2427761593 >> v2:172.21.15.27:3300/0 conn(0x7efda8156120 msgr2=0x7efda806e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:53.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.768+0000 7efda7400640 1 --2- 172.21.15.27:0/2427761593 >> v2:172.21.15.27:3300/0 conn(0x7efda8156120 0x7efda806e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:53.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.768+0000 7efda7400640 1 -- 172.21.15.27:0/2427761593 >> v1:172.21.15.155:6789/0 conn(0x7efda80c5320 legacy=0x7efda80c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:53.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.768+0000 7efda7400640 1 -- 172.21.15.27:0/2427761593 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda80c4a40 con 0x7efda8156500 2024-03-28T18:57:53.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.768+0000 7efda6a00640 1 --2- 172.21.15.27:0/2427761593 >> v2:172.21.15.27:3300/0 conn(0x7efda8156120 0x7efda806e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:57:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.769+0000 7efda7400640 1 --2- 172.21.15.27:0/2427761593 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156500 0x7efda80c6730 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efda0009e50 tx=0x7efda002be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41368958c20d4692 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.769+0000 7efda5600640 1 -- 172.21.15.27:0/2427761593 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda002c9f0 con 0x7efda8156500 2024-03-28T18:57:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.769+0000 7efda5600640 1 -- 172.21.15.27:0/2427761593 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda002cb90 con 0x7efda8156500 2024-03-28T18:57:53.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.769+0000 7efda5600640 1 -- 172.21.15.27:0/2427761593 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda00343f0 con 0x7efda8156500 2024-03-28T18:57:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.770+0000 7efda7e00640 1 -- 172.21.15.27:0/2427761593 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156500 msgr2=0x7efda80c6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.770+0000 7efda7e00640 1 --2- 172.21.15.27:0/2427761593 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156500 0x7efda80c6730 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efda0009e50 tx=0x7efda002be50 comp rx=0 tx=0).stop 2024-03-28T18:57:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.770+0000 7efda7e00640 1 -- 172.21.15.27:0/2427761593 shutdown_connections 2024-03-28T18:57:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.770+0000 7efda7e00640 1 --2- 172.21.15.27:0/2427761593 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156500 0x7efda80c6730 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.770+0000 7efda7e00640 1 --2- 172.21.15.27:0/2427761593 >> v2:172.21.15.27:3300/0 conn(0x7efda8156120 0x7efda806e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.770+0000 7efda7e00640 1 -- 172.21.15.27:0/2427761593 >> 172.21.15.27:0/2427761593 conn(0x7efda806b0c0 msgr2=0x7efda806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:53.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.770+0000 7efda7e00640 1 -- 172.21.15.27:0/2427761593 shutdown_connections 2024-03-28T18:57:53.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.771+0000 7efda7e00640 1 -- 172.21.15.27:0/2427761593 wait complete. 2024-03-28T18:57:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.773+0000 7efda7e00640 1 Processor -- start 2024-03-28T18:57:53.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.773+0000 7efda7e00640 1 -- start start 2024-03-28T18:57:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efda80c5320 0x7efda8071330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156120 0x7efda81eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efda8156500 0x7efda8157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda80c73d0 con 0x7efda80c5320 2024-03-28T18:57:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda80c7250 con 0x7efda8156500 2024-03-28T18:57:53.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda80c29a0 con 0x7efda8156120 2024-03-28T18:57:53.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efda80c5320 0x7efda8071330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efda80c5320 0x7efda8071330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56424/0 (socket says 172.21.15.27:56424) 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda6a00640 1 -- 172.21.15.27:0/3059468763 learned_addr learned my addr 172.21.15.27:0/3059468763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda6000640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156120 0x7efda81eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.774+0000 7efda7400640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efda8156500 0x7efda8157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.775+0000 7efda6000640 1 -- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efda8156500 msgr2=0x7efda8157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.775+0000 7efda6000640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efda8156500 0x7efda8157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.775+0000 7efda6000640 1 -- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efda80c5320 msgr2=0x7efda8071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.775+0000 7efda6000640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efda80c5320 0x7efda8071330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.775+0000 7efda6000640 1 -- 172.21.15.27:0/3059468763 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda81ef0f0 con 0x7efda8156120 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.775+0000 7efda6a00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efda80c5320 0x7efda8071330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:57:53.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.775+0000 7efda6000640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156120 0x7efda81eebb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efd9400c340 tx=0x7efd9400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.775+0000 7efd9ea00640 1 -- 172.21.15.27:0/3059468763 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94015e30 con 0x7efda8156120 2024-03-28T18:57:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.776+0000 7efd9ea00640 1 -- 172.21.15.27:0/3059468763 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd940050d0 con 0x7efda8156120 2024-03-28T18:57:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.776+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda8152160 con 0x7efda8156120 2024-03-28T18:57:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.776+0000 7efd9ea00640 1 -- 172.21.15.27:0/3059468763 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94007020 con 0x7efda8156120 2024-03-28T18:57:53.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.776+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda81dee90 con 0x7efda8156120 2024-03-28T18:57:53.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.778+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda8067dd0 con 0x7efda8156120 2024-03-28T18:57:53.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.779+0000 7efd9ea00640 1 -- 172.21.15.27:0/3059468763 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efd940071c0 con 0x7efda8156120 2024-03-28T18:57:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.780+0000 7efd9ea00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd8806bbd0 0x7efd8806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.780+0000 7efd9ea00640 1 -- 172.21.15.27:0/3059468763 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7efd94092af0 con 0x7efda8156120 2024-03-28T18:57:53.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.780+0000 7efda6a00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd8806bbd0 0x7efd8806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:53.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.781+0000 7efda6a00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd8806bbd0 0x7efd8806e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd98006d40 tx=0x7efd98008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:53.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:53.790+0000 7efd9ea00640 1 -- 172.21.15.27:0/3059468763 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efd94060b80 con 0x7efda8156120 2024-03-28T18:57:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.074+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efda8069640 con 0x7efda8156120 2024-03-28T18:57:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.075+0000 7efd9ea00640 1 -- 172.21.15.27:0/3059468763 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6254 (secure 0 0 0) 0x7efd94064c20 con 0x7efda8156120 2024-03-28T18:57:54.075 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:54.075 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:57:54.075 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-03-28T18:57:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.077+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd8806bbd0 msgr2=0x7efd8806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.077+0000 7efda7e00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd8806bbd0 0x7efd8806e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd98006d40 tx=0x7efd98008040 comp rx=0 tx=0).stop 2024-03-28T18:57:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156120 msgr2=0x7efda81eebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156120 0x7efda81eebb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efd9400c340 tx=0x7efd9400c800 comp rx=0 tx=0).stop 2024-03-28T18:57:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 shutdown_connections 2024-03-28T18:57:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efd8806bbd0 0x7efd8806e090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efda8156500 0x7efda8157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efda8156120 0x7efda81eebb0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efd9400c340 tx=0x7efd9400c800 comp rx=0 tx=0).stop 2024-03-28T18:57:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 --2- 172.21.15.27:0/3059468763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efda80c5320 0x7efda8071330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 >> 172.21.15.27:0/3059468763 conn(0x7efda806b0c0 msgr2=0x7efda80a6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 shutdown_connections 2024-03-28T18:57:54.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.078+0000 7efda7e00640 1 -- 172.21.15.27:0/3059468763 wait complete. 2024-03-28T18:57:54.104 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-03-28T18:57:54.104 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T18:57:54.104 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T18:57:54.130 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T18:57:54.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.324+0000 7f730b200640 1 Processor -- start 2024-03-28T18:57:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.324+0000 7f730b200640 1 -- start start 2024-03-28T18:57:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.326+0000 7f730b200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f730c0c8ed0 0x7f730c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.326+0000 7f730b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0be5d0 0x7f730c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.326+0000 7f730b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c156230 con 0x7f730c0c58c0 2024-03-28T18:57:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.326+0000 7f730b200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c158bd0 con 0x7f730c0c8ed0 2024-03-28T18:57:54.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.326+0000 7f730b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c158eb0 con 0x7f730c0be5d0 2024-03-28T18:57:54.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.326+0000 7f7309e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f730c0c58c0 0x7f730c1555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33666/0 (socket says 172.21.15.27:33666) 2024-03-28T18:57:54.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.326+0000 7f7309e00640 1 -- 172.21.15.27:0/4285030434 learned_addr learned my addr 172.21.15.27:0/4285030434 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:54.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.326+0000 7f7309400640 1 --2- 172.21.15.27:0/4285030434 >> v2:172.21.15.155:3300/0 conn(0x7f730c0c8ed0 0x7f730c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:54.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.327+0000 7f7309400640 1 -- 172.21.15.27:0/4285030434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0be5d0 msgr2=0x7f730c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:54.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.327+0000 7f7309400640 1 --2- 172.21.15.27:0/4285030434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0be5d0 0x7f730c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.327+0000 7f7309400640 1 -- 172.21.15.27:0/4285030434 >> v1:172.21.15.27:6789/0 conn(0x7f730c0c58c0 legacy=0x7f730c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:54.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.327+0000 7f7308a00640 1 -- 172.21.15.27:0/4285030434 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454143730 0 0) 0x7f730c156230 con 0x7f730c0c58c0 2024-03-28T18:57:54.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.327+0000 7f7309400640 1 -- 172.21.15.27:0/4285030434 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c159190 con 0x7f730c0c8ed0 2024-03-28T18:57:54.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.328+0000 7f7309400640 1 --2- 172.21.15.27:0/4285030434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 0x7f730c0c5ca0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f72f80097a0 tx=0x7f72f802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efad59a4fcff20ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:54.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f7308a00640 1 -- 172.21.15.27:0/4285030434 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8031780 con 0x7f730c0c8ed0 2024-03-28T18:57:54.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f7308a00640 1 -- 172.21.15.27:0/4285030434 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f8037070 con 0x7f730c0c8ed0 2024-03-28T18:57:54.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f7308a00640 1 -- 172.21.15.27:0/4285030434 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8031b40 con 0x7f730c0c8ed0 2024-03-28T18:57:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f730b200640 1 -- 172.21.15.27:0/4285030434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 msgr2=0x7f730c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f730b200640 1 --2- 172.21.15.27:0/4285030434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 0x7f730c0c5ca0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f72f80097a0 tx=0x7f72f802b820 comp rx=0 tx=0).stop 2024-03-28T18:57:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f730b200640 1 -- 172.21.15.27:0/4285030434 shutdown_connections 2024-03-28T18:57:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f730b200640 1 --2- 172.21.15.27:0/4285030434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0be5d0 0x7f730c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f730b200640 1 --2- 172.21.15.27:0/4285030434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 0x7f730c0c5ca0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f730b200640 1 -- 172.21.15.27:0/4285030434 >> 172.21.15.27:0/4285030434 conn(0x7f730c06b6f0 msgr2=0x7f730c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f730b200640 1 -- 172.21.15.27:0/4285030434 shutdown_connections 2024-03-28T18:57:54.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.329+0000 7f730b200640 1 -- 172.21.15.27:0/4285030434 wait complete. 2024-03-28T18:57:54.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.331+0000 7f730b200640 1 Processor -- start 2024-03-28T18:57:54.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.331+0000 7f730b200640 1 -- start start 2024-03-28T18:57:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.332+0000 7f730b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0be5d0 0x7f730c1ef180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.332+0000 7f730b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0c58c0 0x7f730c071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:54.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.332+0000 7f730b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 0x7f730c151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.332+0000 7f730b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0b0820 con 0x7f730c0be5d0 2024-03-28T18:57:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.332+0000 7f730b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c159750 con 0x7f730c0c8ed0 2024-03-28T18:57:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.332+0000 7f730b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0b09a0 con 0x7f730c0c58c0 2024-03-28T18:57:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.332+0000 7f7309400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0c58c0 0x7f730c071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f7309400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0c58c0 0x7f730c071ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:34816/0 (socket says 172.21.15.27:34816) 2024-03-28T18:57:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f7309400640 1 -- 172.21.15.27:0/1402844738 learned_addr learned my addr 172.21.15.27:0/1402844738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:54.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f730a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 0x7f730c151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f7309e00640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0be5d0 0x7f730c1ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:54.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f730a800640 1 -- 172.21.15.27:0/1402844738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0c58c0 msgr2=0x7f730c071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f730a800640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0c58c0 0x7f730c071ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f730a800640 1 -- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0be5d0 msgr2=0x7f730c1ef180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f730a800640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0be5d0 0x7f730c1ef180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f730a800640 1 -- 172.21.15.27:0/1402844738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0c20b0 con 0x7f730c0c8ed0 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f7309400640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0c58c0 0x7f730c071ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.333+0000 7f730a800640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 0x7f730c151f10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7304009f00 tx=0x7f730400bdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.334+0000 7f7302000640 1 -- 172.21.15.27:0/1402844738 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730401e020 con 0x7f730c0c8ed0 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.334+0000 7f7302000640 1 -- 172.21.15.27:0/1402844738 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730401a830 con 0x7f730c0c8ed0 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.334+0000 7f7302000640 1 -- 172.21.15.27:0/1402844738 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730401ac20 con 0x7f730c0c8ed0 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.334+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c1df4b0 con 0x7f730c0c8ed0 2024-03-28T18:57:54.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.334+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c0c2320 con 0x7f730c0c8ed0 2024-03-28T18:57:54.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.335+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c069640 con 0x7f730c0c8ed0 2024-03-28T18:57:54.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.338+0000 7f7302000640 1 -- 172.21.15.27:0/1402844738 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7304021020 con 0x7f730c0c8ed0 2024-03-28T18:57:54.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.339+0000 7f7302000640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72ec06bbd0 0x7f72ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:54.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.339+0000 7f7302000640 1 -- 172.21.15.27:0/1402844738 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f7304094600 con 0x7f730c0c8ed0 2024-03-28T18:57:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.339+0000 7f7309e00640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72ec06bbd0 0x7f72ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:54.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.340+0000 7f7309e00640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72ec06bbd0 0x7f72ec06e090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72fc000f60 tx=0x7f72fc007c00 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:54.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.348+0000 7f7302000640 1 -- 172.21.15.27:0/1402844738 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7304028020 con 0x7f730c0c8ed0 2024-03-28T18:57:54.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.583+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f730c156580 con 0x7f730c0c8ed0 2024-03-28T18:57:54.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.584+0000 7f7302000640 1 -- 172.21.15.27:0/1402844738 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f7304062cd0 con 0x7f730c0c8ed0 2024-03-28T18:57:54.583 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:54.584 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:57:54.584 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-03-28T18:57:54.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72ec06bbd0 msgr2=0x7f72ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:54.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72ec06bbd0 0x7f72ec06e090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72fc000f60 tx=0x7f72fc007c00 comp rx=0 tx=0).stop 2024-03-28T18:57:54.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 msgr2=0x7f730c151f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:54.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 0x7f730c151f10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7304009f00 tx=0x7f730400bdf0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 shutdown_connections 2024-03-28T18:57:54.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f72ec06bbd0 0x7f72ec06e090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f730c0c8ed0 0x7f730c151f10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f730c0c58c0 0x7f730c071ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.587+0000 7f730b200640 1 --2- 172.21.15.27:0/1402844738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0be5d0 0x7f730c1ef180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:54.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.588+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 >> 172.21.15.27:0/1402844738 conn(0x7f730c06b6f0 msgr2=0x7f730c0ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:54.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.588+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 shutdown_connections 2024-03-28T18:57:54.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:54.588+0000 7f730b200640 1 -- 172.21.15.27:0/1402844738 wait complete. 2024-03-28T18:57:54.612 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T18:57:55.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:57:55.069 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:57:56.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:56.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.906+0000 7fcfe7e00640 1 Processor -- start 2024-03-28T18:57:56.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.906+0000 7fcfe7e00640 1 -- start start 2024-03-28T18:57:56.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.908+0000 7fcfe7e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcfe80c58c0 0x7fcfe80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:56.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.908+0000 7fcfe7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c8ed0 0x7fcfe81555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:56.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.908+0000 7fcfe7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcfe80be5d0 0x7fcfe8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:56.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.908+0000 7fcfe7e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe8158bd0 con 0x7fcfe80be5d0 2024-03-28T18:57:56.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.908+0000 7fcfe7e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe8158eb0 con 0x7fcfe80c58c0 2024-03-28T18:57:56.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.908+0000 7fcfe7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe8159190 con 0x7fcfe80c8ed0 2024-03-28T18:57:56.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.909+0000 7fcfe6000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c8ed0 0x7fcfe81555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:56.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.909+0000 7fcfe6000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c8ed0 0x7fcfe81555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:37488/0 (socket says 172.21.15.27:37488) 2024-03-28T18:57:56.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.909+0000 7fcfe6000640 1 -- 172.21.15.27:0/492521011 learned_addr learned my addr 172.21.15.27:0/492521011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:56.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.909+0000 7fcfe6a00640 1 --2- 172.21.15.27:0/492521011 >> v2:172.21.15.155:3300/0 conn(0x7fcfe80c58c0 0x7fcfe80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:56.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.909+0000 7fcfe7400640 1 --2- 172.21.15.27:0/492521011 >> v2:172.21.15.27:3300/0 conn(0x7fcfe80be5d0 0x7fcfe8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:56.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.910+0000 7fcfe6000640 1 -- 172.21.15.27:0/492521011 >> v2:172.21.15.155:3300/0 conn(0x7fcfe80c58c0 msgr2=0x7fcfe80c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:56.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.910+0000 7fcfe6000640 1 --2- 172.21.15.27:0/492521011 >> v2:172.21.15.155:3300/0 conn(0x7fcfe80c58c0 0x7fcfe80c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:56.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.910+0000 7fcfe6000640 1 -- 172.21.15.27:0/492521011 >> v2:172.21.15.27:3300/0 conn(0x7fcfe80be5d0 msgr2=0x7fcfe8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:56.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.910+0000 7fcfe6000640 1 --2- 172.21.15.27:0/492521011 >> v2:172.21.15.27:3300/0 conn(0x7fcfe80be5d0 0x7fcfe8155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:56.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.910+0000 7fcfe6000640 1 -- 172.21.15.27:0/492521011 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe8159470 con 0x7fcfe80c8ed0 2024-03-28T18:57:56.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.910+0000 7fcfe6a00640 1 --2- 172.21.15.27:0/492521011 >> v2:172.21.15.155:3300/0 conn(0x7fcfe80c58c0 0x7fcfe80c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:57:56.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.911+0000 7fcfe6000640 1 --2- 172.21.15.27:0/492521011 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c8ed0 0x7fcfe81555e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcfe806c060 tx=0x7fcfd402b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dad66120f5cc202 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:56.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.912+0000 7fcfe5600640 1 -- 172.21.15.27:0/492521011 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd402e070 con 0x7fcfe80c8ed0 2024-03-28T18:57:56.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.912+0000 7fcfe5600640 1 -- 172.21.15.27:0/492521011 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd4037070 con 0x7fcfe80c8ed0 2024-03-28T18:57:56.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.912+0000 7fcfe5600640 1 -- 172.21.15.27:0/492521011 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4031a00 con 0x7fcfe80c8ed0 2024-03-28T18:57:56.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.913+0000 7fcfe7e00640 1 -- 172.21.15.27:0/492521011 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c8ed0 msgr2=0x7fcfe81555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:56.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.913+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/492521011 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c8ed0 0x7fcfe81555e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcfe806c060 tx=0x7fcfd402b880 comp rx=0 tx=0).stop 2024-03-28T18:57:56.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.913+0000 7fcfe7e00640 1 -- 172.21.15.27:0/492521011 shutdown_connections 2024-03-28T18:57:56.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.913+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/492521011 >> v2:172.21.15.27:3300/0 conn(0x7fcfe80be5d0 0x7fcfe8155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:56.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.913+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/492521011 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c8ed0 0x7fcfe81555e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:56.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.913+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/492521011 >> v2:172.21.15.155:3300/0 conn(0x7fcfe80c58c0 0x7fcfe80c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:56.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.913+0000 7fcfe7e00640 1 -- 172.21.15.27:0/492521011 >> 172.21.15.27:0/492521011 conn(0x7fcfe806b6f0 msgr2=0x7fcfe80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:56.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.914+0000 7fcfe7e00640 1 -- 172.21.15.27:0/492521011 shutdown_connections 2024-03-28T18:57:56.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.914+0000 7fcfe7e00640 1 -- 172.21.15.27:0/492521011 wait complete. 2024-03-28T18:57:56.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.916+0000 7fcfe7e00640 1 Processor -- start 2024-03-28T18:57:56.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.916+0000 7fcfe7e00640 1 -- start start 2024-03-28T18:57:56.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.918+0000 7fcfe7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe80be5d0 0x7fcfe80bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:56.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.918+0000 7fcfe7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c58c0 0x7fcfe8155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:56.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.918+0000 7fcfe7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfe80c8ed0 0x7fcfe81eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:56.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.918+0000 7fcfe7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe80b09a0 con 0x7fcfe80be5d0 2024-03-28T18:57:56.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.918+0000 7fcfe7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe80b0820 con 0x7fcfe80c8ed0 2024-03-28T18:57:56.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.918+0000 7fcfe7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe80ad7d0 con 0x7fcfe80c58c0 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.918+0000 7fcfe6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe80be5d0 0x7fcfe80bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe80be5d0 0x7fcfe80bea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55654/0 (socket says 172.21.15.27:55654) 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe6a00640 1 -- 172.21.15.27:0/432321078 learned_addr learned my addr 172.21.15.27:0/432321078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe6000640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c58c0 0x7fcfe8155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe7400640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfe80c8ed0 0x7fcfe81eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe6a00640 1 -- 172.21.15.27:0/432321078 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c58c0 msgr2=0x7fcfe8155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe6a00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c58c0 0x7fcfe8155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe6a00640 1 -- 172.21.15.27:0/432321078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfe80c8ed0 msgr2=0x7fcfe81eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe6a00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfe80c8ed0 0x7fcfe81eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:56.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.919+0000 7fcfe6a00640 1 -- 172.21.15.27:0/432321078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe80c3e70 con 0x7fcfe80be5d0 2024-03-28T18:57:56.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.920+0000 7fcfe6a00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe80be5d0 0x7fcfe80bea70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800ebe0 tx=0x7fcfd800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:56.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.920+0000 7fcfdea00640 1 -- 172.21.15.27:0/432321078 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8017070 con 0x7fcfe80be5d0 2024-03-28T18:57:56.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.920+0000 7fcfdea00640 1 -- 172.21.15.27:0/432321078 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd800cc30 con 0x7fcfe80be5d0 2024-03-28T18:57:56.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.920+0000 7fcfdea00640 1 -- 172.21.15.27:0/432321078 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8013470 con 0x7fcfe80be5d0 2024-03-28T18:57:56.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.921+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe81ef8c0 con 0x7fcfe80be5d0 2024-03-28T18:57:56.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.921+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe81555c0 con 0x7fcfe80be5d0 2024-03-28T18:57:56.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.923+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe80b0340 con 0x7fcfe80be5d0 2024-03-28T18:57:56.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.925+0000 7fcfdea00640 1 -- 172.21.15.27:0/432321078 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcfd801a070 con 0x7fcfe80be5d0 2024-03-28T18:57:56.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.926+0000 7fcfdea00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcfc806bbd0 0x7fcfc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:56.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.926+0000 7fcfdea00640 1 -- 172.21.15.27:0/432321078 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fcfd808acf0 con 0x7fcfe80be5d0 2024-03-28T18:57:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.927+0000 7fcfe6000640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcfc806bbd0 0x7fcfc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:56.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.928+0000 7fcfe6000640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcfc806bbd0 0x7fcfc806e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4036000 tx=0x7fcfd403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:56.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:56.936+0000 7fcfdea00640 1 -- 172.21.15.27:0/432321078 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcfe80b0340 con 0x7fcfe80be5d0 2024-03-28T18:57:57.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.314+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfe8155850 con 0x7fcfe80be5d0 2024-03-28T18:57:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.315+0000 7fcfdea00640 1 -- 172.21.15.27:0/432321078 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6254 (secure 0 0 0) 0x7fcfd805f0d0 con 0x7fcfe80be5d0 2024-03-28T18:57:57.315 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:57:57.315 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:57:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-03-28T18:57:57.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcfc806bbd0 msgr2=0x7fcfc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:57.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcfc806bbd0 0x7fcfc806e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4036000 tx=0x7fcfd403d040 comp rx=0 tx=0).stop 2024-03-28T18:57:57.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe80be5d0 msgr2=0x7fcfe80bea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe80be5d0 0x7fcfe80bea70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800ebe0 tx=0x7fcfd800c520 comp rx=0 tx=0).stop 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 shutdown_connections 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcfc806bbd0 0x7fcfc806e090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfe80c8ed0 0x7fcfe81eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfe80c58c0 0x7fcfe8155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 --2- 172.21.15.27:0/432321078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe80be5d0 0x7fcfe80bea70 secure :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800ebe0 tx=0x7fcfd800c520 comp rx=0 tx=0).stop 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 >> 172.21.15.27:0/432321078 conn(0x7fcfe806b6f0 msgr2=0x7fcfe80ae5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 shutdown_connections 2024-03-28T18:57:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:57.318+0000 7fcfe7e00640 1 -- 172.21.15.27:0/432321078 wait complete. 2024-03-28T18:57:57.335 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T18:57:59.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:57:59.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.559+0000 7eff2f000640 1 Processor -- start 2024-03-28T18:57:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.559+0000 7eff2f000640 1 -- start start 2024-03-28T18:57:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.560+0000 7eff2f000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff280c4dc0 0x7eff280c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.560+0000 7eff2f000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2806e860 con 0x7eff280a5ef0 2024-03-28T18:57:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.560+0000 7eff2f000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff28155a60 con 0x7eff280c0cb0 2024-03-28T18:57:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.560+0000 7eff2f000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2806e050 con 0x7eff280c4dc0 2024-03-28T18:57:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.561+0000 7eff2dc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7eff280a5ef0 0x7eff280c3380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36016/0 (socket says 172.21.15.27:36016) 2024-03-28T18:57:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.561+0000 7eff2dc00640 1 -- 172.21.15.27:0/2793938255 learned_addr learned my addr 172.21.15.27:0/2793938255 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:57:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.561+0000 7eff2d200640 1 --1- 172.21.15.27:0/2793938255 >> v1:172.21.15.155:6789/0 conn(0x7eff280c0cb0 0x7eff28156220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55998/0 (socket says 172.21.15.27:55998) 2024-03-28T18:57:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.561+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532326732 0 0) 0x7eff28155a60 con 0x7eff280c0cb0 2024-03-28T18:57:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.561+0000 7eff2e600640 1 --2- 172.21.15.27:0/2793938255 >> v2:172.21.15.27:3300/0 conn(0x7eff280c4dc0 0x7eff280c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.561+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff10003520 con 0x7eff280c0cb0 2024-03-28T18:57:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.561+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546489532 0 0) 0x7eff2806e860 con 0x7eff280a5ef0 2024-03-28T18:57:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff28155a60 con 0x7eff280a5ef0 2024-03-28T18:57:59.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83181223 0 0) 0x7eff10003520 con 0x7eff280c0cb0 2024-03-28T18:57:59.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 >> v2:172.21.15.27:3300/0 conn(0x7eff280c4dc0 msgr2=0x7eff280c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:59.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 --2- 172.21.15.27:0/2793938255 >> v2:172.21.15.27:3300/0 conn(0x7eff280c4dc0 0x7eff280c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 >> v1:172.21.15.27:6789/0 conn(0x7eff280a5ef0 legacy=0x7eff280c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2806e2a0 con 0x7eff280c0cb0 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7eff140015a0 con 0x7eff280c0cb0 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 >> v1:172.21.15.155:6789/0 conn(0x7eff280c0cb0 legacy=0x7eff28156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10004bf0 0x7eff10005010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff100037a0 0x7eff10003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff10004270 0x7eff1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2806e050 con 0x7eff10004bf0 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1000c1e0 con 0x7eff100037a0 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.562+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1000c3e0 con 0x7eff10004270 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.563+0000 7eff2d200640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff100037a0 0x7eff10003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.563+0000 7eff2e600640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff10004270 0x7eff1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.563+0000 7eff2dc00640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10004bf0 0x7eff10005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.564+0000 7eff2d200640 1 -- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff10004270 msgr2=0x7eff1000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.564+0000 7eff2d200640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff10004270 0x7eff1000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.564+0000 7eff2d200640 1 -- 172.21.15.27:0/2793938255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10004bf0 msgr2=0x7eff10005010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:57:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.564+0000 7eff2d200640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10004bf0 0x7eff10005010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.564+0000 7eff2d200640 1 -- 172.21.15.27:0/2793938255 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2806e2a0 con 0x7eff100037a0 2024-03-28T18:57:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.564+0000 7eff2e600640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff10004270 0x7eff1000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:57:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.564+0000 7eff2dc00640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff10004bf0 0x7eff10005010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:57:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.564+0000 7eff2e600640 1 -- 172.21.15.27:0/2793938255 reap_dead start 2024-03-28T18:57:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.565+0000 7eff2d200640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff100037a0 0x7eff10003c20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff14007fd0 tx=0x7eff10005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=991fdc68110156ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.566+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14007ba0 con 0x7eff100037a0 2024-03-28T18:57:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.566+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff14030070 con 0x7eff100037a0 2024-03-28T18:57:59.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.566+0000 7eff27e00640 1 -- 172.21.15.27:0/2793938255 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140074b0 con 0x7eff100037a0 2024-03-28T18:57:59.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.567+0000 7eff2f000640 1 -- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff100037a0 msgr2=0x7eff10003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:59.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.567+0000 7eff2f000640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff100037a0 0x7eff10003c20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff14007fd0 tx=0x7eff10005830 comp rx=0 tx=0).stop 2024-03-28T18:57:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.567+0000 7eff2f000640 1 -- 172.21.15.27:0/2793938255 shutdown_connections 2024-03-28T18:57:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.567+0000 7eff2f000640 1 --2- 172.21.15.27:0/2793938255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff100037a0 0x7eff10003c20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.567+0000 7eff2f000640 1 -- 172.21.15.27:0/2793938255 >> 172.21.15.27:0/2793938255 conn(0x7eff2806cd50 msgr2=0x7eff2806b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:57:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.567+0000 7eff2f000640 1 -- 172.21.15.27:0/2793938255 shutdown_connections 2024-03-28T18:57:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.567+0000 7eff2f000640 1 -- 172.21.15.27:0/2793938255 wait complete. 2024-03-28T18:57:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.570+0000 7eff2f000640 1 Processor -- start 2024-03-28T18:57:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.570+0000 7eff2f000640 1 -- start start 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2f000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff100037a0 0x7eff280ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff280a85c0 0x7eff28159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2f000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff28159670 0x7eff280a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280c72e0 con 0x7eff280a85c0 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2f000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff28155d90 con 0x7eff28159670 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2f000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280a62d0 con 0x7eff100037a0 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2dc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff100037a0 0x7eff280ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2dc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff100037a0 0x7eff280ab9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:37522/0 (socket says 172.21.15.27:37522) 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2dc00640 1 -- 172.21.15.27:0/2345591144 learned_addr learned my addr 172.21.15.27:0/2345591144 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2d200640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff280a85c0 0x7eff28159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:59.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.572+0000 7eff2e600640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff28159670 0x7eff280a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:59.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.573+0000 7eff2dc00640 1 -- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff28159670 msgr2=0x7eff280a4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.573+0000 7eff2dc00640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff28159670 0x7eff280a4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.573+0000 7eff2dc00640 1 -- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff280a85c0 msgr2=0x7eff28159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.573+0000 7eff2dc00640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff280a85c0 0x7eff28159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.573+0000 7eff2dc00640 1 -- 172.21.15.27:0/2345591144 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff281ef3f0 con 0x7eff100037a0 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.573+0000 7eff2e600640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff28159670 0x7eff280a4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.573+0000 7eff2dc00640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff100037a0 0x7eff280ab9d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0077d0 tx=0x7eff1c001770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.574+0000 7eff25600640 1 -- 172.21.15.27:0/2345591144 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c019070 con 0x7eff100037a0 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.574+0000 7eff25600640 1 -- 172.21.15.27:0/2345591144 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c0049f0 con 0x7eff100037a0 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.574+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff280a7770 con 0x7eff100037a0 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.574+0000 7eff25600640 1 -- 172.21.15.27:0/2345591144 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c004de0 con 0x7eff100037a0 2024-03-28T18:57:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.574+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff28158c10 con 0x7eff100037a0 2024-03-28T18:57:59.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.576+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff28067dd0 con 0x7eff100037a0 2024-03-28T18:57:59.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.578+0000 7eff25600640 1 -- 172.21.15.27:0/2345591144 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7eff1c01c070 con 0x7eff100037a0 2024-03-28T18:57:59.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.579+0000 7eff25600640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bbd0 0x7eff0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:57:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.579+0000 7eff25600640 1 -- 172.21.15.27:0/2345591144 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7eff1c08a730 con 0x7eff100037a0 2024-03-28T18:57:59.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.580+0000 7eff2d200640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bbd0 0x7eff0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:57:59.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.581+0000 7eff2d200640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bbd0 0x7eff0806e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff14034000 tx=0x7eff14009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:57:59.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:57:59.588+0000 7eff25600640 1 -- 172.21.15.27:0/2345591144 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7eff1c058e10 con 0x7eff100037a0 2024-03-28T18:58:00.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.082+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff28069640 con 0x7eff100037a0 2024-03-28T18:58:00.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.083+0000 7eff25600640 1 -- 172.21.15.27:0/2345591144 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6254 (secure 0 0 0) 0x7eff1c058710 con 0x7eff100037a0 2024-03-28T18:58:00.083 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:00.083 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:00.083 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-03-28T18:58:00.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bbd0 msgr2=0x7eff0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:00.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bbd0 0x7eff0806e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff14034000 tx=0x7eff14009030 comp rx=0 tx=0).stop 2024-03-28T18:58:00.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff100037a0 msgr2=0x7eff280ab9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:00.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff100037a0 0x7eff280ab9d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0077d0 tx=0x7eff1c001770 comp rx=0 tx=0).stop 2024-03-28T18:58:00.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 shutdown_connections 2024-03-28T18:58:00.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bbd0 0x7eff0806e090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:00.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff28159670 0x7eff280a4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:00.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff280a85c0 0x7eff28159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:00.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 --2- 172.21.15.27:0/2345591144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff100037a0 0x7eff280ab9d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:00.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.086+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 >> 172.21.15.27:0/2345591144 conn(0x7eff2806cd50 msgr2=0x7eff280c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:00.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.087+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 shutdown_connections 2024-03-28T18:58:00.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:00.087+0000 7eff2f000640 1 -- 172.21.15.27:0/2345591144 wait complete. 2024-03-28T18:58:00.103 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T18:58:02.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.392+0000 7f7fd9a00640 1 Processor -- start 2024-03-28T18:58:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.393+0000 7f7fd9a00640 1 -- start start 2024-03-28T18:58:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.395+0000 7f7fd9a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd40c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.395+0000 7f7fd9a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7fd40ab2b0 0x7f7fd40c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.395+0000 7f7fd9a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd4156870 con 0x7f7fd41551a0 2024-03-28T18:58:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.395+0000 7f7fd9a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd4155cf0 con 0x7f7fd40ab2b0 2024-03-28T18:58:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.395+0000 7f7fd9a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd4155fd0 con 0x7f7fd40aada0 2024-03-28T18:58:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.395+0000 7f7fd3e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7fd41551a0 0x7f7fd40c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:56014/0 (socket says 172.21.15.27:56014) 2024-03-28T18:58:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.395+0000 7f7fd3e00640 1 -- 172.21.15.27:0/3696965010 learned_addr learned my addr 172.21.15.27:0/3696965010 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:58:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.395+0000 7f7fd9000640 1 --2- 172.21.15.27:0/3696965010 >> v2:172.21.15.155:3300/0 conn(0x7f7fd40ab2b0 0x7f7fd40c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.396+0000 7f7fd2a00640 1 -- 172.21.15.27:0/3696965010 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025823925 0 0) 0x7f7fd4156870 con 0x7f7fd41551a0 2024-03-28T18:58:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.396+0000 7f7fd2a00640 1 -- 172.21.15.27:0/3696965010 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb8003520 con 0x7f7fd41551a0 2024-03-28T18:58:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.396+0000 7f7fd9000640 1 -- 172.21.15.27:0/3696965010 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 msgr2=0x7f7fd40c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.396+0000 7f7fd9000640 1 --2- 172.21.15.27:0/3696965010 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd40c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.396+0000 7f7fd9000640 1 -- 172.21.15.27:0/3696965010 >> v1:172.21.15.155:6789/0 conn(0x7f7fd41551a0 legacy=0x7f7fd40c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.396+0000 7f7fd9000640 1 -- 172.21.15.27:0/3696965010 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd41562b0 con 0x7f7fd40ab2b0 2024-03-28T18:58:02.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.398+0000 7f7fd9000640 1 --2- 172.21.15.27:0/3696965010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 0x7f7fd40c1fc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009d90 tx=0x7f7fcc0301a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd4cb8c39abcd616 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:02.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.398+0000 7f7fd2a00640 1 -- 172.21.15.27:0/3696965010 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc03d070 con 0x7f7fd40ab2b0 2024-03-28T18:58:02.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.398+0000 7f7fd2a00640 1 -- 172.21.15.27:0/3696965010 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc038440 con 0x7f7fd40ab2b0 2024-03-28T18:58:02.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.398+0000 7f7fd2a00640 1 -- 172.21.15.27:0/3696965010 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc038810 con 0x7f7fd40ab2b0 2024-03-28T18:58:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.399+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3696965010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 msgr2=0x7f7fd40c1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.399+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3696965010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 0x7f7fd40c1fc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009d90 tx=0x7f7fcc0301a0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.399+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3696965010 shutdown_connections 2024-03-28T18:58:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.399+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3696965010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 0x7f7fd40c1fc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.399+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3696965010 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd40c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.399+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3696965010 >> 172.21.15.27:0/3696965010 conn(0x7f7fd406b0c0 msgr2=0x7f7fd406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.399+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3696965010 shutdown_connections 2024-03-28T18:58:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.399+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3696965010 wait complete. 2024-03-28T18:58:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.402+0000 7f7fd9a00640 1 Processor -- start 2024-03-28T18:58:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.402+0000 7f7fd9a00640 1 -- start start 2024-03-28T18:58:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.403+0000 7f7fd9a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd4072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.403+0000 7f7fd9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 0x7f7fd41e93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.403+0000 7f7fd9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd41551a0 0x7f7fd41e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.403+0000 7f7fd9a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40c5750 con 0x7f7fd41551a0 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.403+0000 7f7fd9a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd41559a0 con 0x7f7fd40ab2b0 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.403+0000 7f7fd9a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40c58d0 con 0x7f7fd40aada0 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd4072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd4072bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:37538/0 (socket says 172.21.15.27:37538) 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd3e00640 1 -- 172.21.15.27:0/3178214939 learned_addr learned my addr 172.21.15.27:0/3178214939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd9000640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd41551a0 0x7f7fd41e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd3400640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 0x7f7fd41e93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd3e00640 1 -- 172.21.15.27:0/3178214939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 msgr2=0x7f7fd41e93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd3e00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 0x7f7fd41e93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd3e00640 1 -- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd41551a0 msgr2=0x7f7fd41e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.404+0000 7f7fd3e00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd41551a0 0x7f7fd41e9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.405+0000 7f7fd3e00640 1 -- 172.21.15.27:0/3178214939 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd41e6b10 con 0x7f7fd40aada0 2024-03-28T18:58:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.405+0000 7f7fd3e00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd4072bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c3c0 tx=0x7f7fc400c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.405+0000 7f7fcbe00640 1 -- 172.21.15.27:0/3178214939 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4017070 con 0x7f7fd40aada0 2024-03-28T18:58:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.405+0000 7f7fcbe00640 1 -- 172.21.15.27:0/3178214939 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc4005da0 con 0x7f7fd40aada0 2024-03-28T18:58:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.405+0000 7f7fcbe00640 1 -- 172.21.15.27:0/3178214939 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4007310 con 0x7f7fd40aada0 2024-03-28T18:58:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.405+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd40bba40 con 0x7f7fd40aada0 2024-03-28T18:58:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.406+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd40730f0 con 0x7f7fd40aada0 2024-03-28T18:58:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.407+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd4067dd0 con 0x7f7fd40aada0 2024-03-28T18:58:02.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.410+0000 7f7fcbe00640 1 -- 172.21.15.27:0/3178214939 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7fc401a070 con 0x7f7fd40aada0 2024-03-28T18:58:02.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.411+0000 7f7fcbe00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7fb406bbd0 0x7f7fb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:02.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.411+0000 7f7fcbe00640 1 -- 172.21.15.27:0/3178214939 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f7fc408a5d0 con 0x7f7fd40aada0 2024-03-28T18:58:02.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.412+0000 7f7fd3400640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7fb406bbd0 0x7f7fb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:02.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.413+0000 7f7fd3400640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7fb406bbd0 0x7f7fb406e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40a5c20 tx=0x7f7fc0009110 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:02.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.417+0000 7f7fcbe00640 1 -- 172.21.15.27:0/3178214939 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7fc4021020 con 0x7f7fd40aada0 2024-03-28T18:58:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.693+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fd4069640 con 0x7f7fd40aada0 2024-03-28T18:58:02.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.694+0000 7f7fcbe00640 1 -- 172.21.15.27:0/3178214939 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f7fc40586e0 con 0x7f7fd40aada0 2024-03-28T18:58:02.694 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:02.694 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-03-28T18:58:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.696+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7fb406bbd0 msgr2=0x7f7fb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.696+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7fb406bbd0 0x7f7fb406e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40a5c20 tx=0x7f7fc0009110 comp rx=0 tx=0).stop 2024-03-28T18:58:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.696+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 msgr2=0x7f7fd4072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.696+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd4072bb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c3c0 tx=0x7f7fc400c7c0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.697+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 shutdown_connections 2024-03-28T18:58:02.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.697+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7fb406bbd0 0x7f7fb406e090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.697+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fd41551a0 0x7f7fd41e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.697+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7fd40ab2b0 0x7f7fd41e93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.697+0000 7f7fd9a00640 1 --2- 172.21.15.27:0/3178214939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7fd40aada0 0x7f7fd4072bb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:02.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.697+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 >> 172.21.15.27:0/3178214939 conn(0x7f7fd406b0c0 msgr2=0x7f7fd40c6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:02.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.697+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 shutdown_connections 2024-03-28T18:58:02.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:02.697+0000 7f7fd9a00640 1 -- 172.21.15.27:0/3178214939 wait complete. 2024-03-28T18:58:02.722 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T18:58:04.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:05.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.008+0000 7f38a2000640 1 Processor -- start 2024-03-28T18:58:05.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.008+0000 7f38a2000640 1 -- start start 2024-03-28T18:58:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a2000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f389c0c58c0 0x7f389c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a2000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c156230 con 0x7f389c0c8ed0 2024-03-28T18:58:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a2000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389c158bd0 con 0x7f389c0c58c0 2024-03-28T18:58:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389c158eb0 con 0x7f389c0be5d0 2024-03-28T18:58:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a0c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f389c0c58c0 0x7f389c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a0c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f389c0c58c0 0x7f389c0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:54758/0 (socket says 172.21.15.27:54758) 2024-03-28T18:58:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a0c00640 1 -- 172.21.15.27:0/1687986766 learned_addr learned my addr 172.21.15.27:0/1687986766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.010+0000 7f38a1600640 1 --2- 172.21.15.27:0/1687986766 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:05.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.011+0000 7f38a0c00640 1 -- 172.21.15.27:0/1687986766 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 msgr2=0x7f389c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:05.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.011+0000 7f38a0c00640 1 --2- 172.21.15.27:0/1687986766 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.011+0000 7f38a0c00640 1 -- 172.21.15.27:0/1687986766 >> v1:172.21.15.27:6789/0 conn(0x7f389c0c8ed0 legacy=0x7f389c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:05.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.011+0000 7f38a0c00640 1 -- 172.21.15.27:0/1687986766 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c159190 con 0x7f389c0c58c0 2024-03-28T18:58:05.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.013+0000 7f38a0c00640 1 --2- 172.21.15.27:0/1687986766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c58c0 0x7f389c0c5ca0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3890009870 tx=0x7f389002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42e07b9a3424d3b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:05.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.013+0000 7f389b400640 1 -- 172.21.15.27:0/1687986766 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38900317c0 con 0x7f389c0c58c0 2024-03-28T18:58:05.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.013+0000 7f389b400640 1 -- 172.21.15.27:0/1687986766 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3890037070 con 0x7f389c0c58c0 2024-03-28T18:58:05.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.013+0000 7f389b400640 1 -- 172.21.15.27:0/1687986766 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890031b80 con 0x7f389c0c58c0 2024-03-28T18:58:05.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.014+0000 7f38a2000640 1 -- 172.21.15.27:0/1687986766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c58c0 msgr2=0x7f389c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:05.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.014+0000 7f38a2000640 1 --2- 172.21.15.27:0/1687986766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c58c0 0x7f389c0c5ca0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3890009870 tx=0x7f389002b860 comp rx=0 tx=0).stop 2024-03-28T18:58:05.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.014+0000 7f38a2000640 1 -- 172.21.15.27:0/1687986766 shutdown_connections 2024-03-28T18:58:05.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.014+0000 7f38a2000640 1 --2- 172.21.15.27:0/1687986766 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.014+0000 7f38a2000640 1 --2- 172.21.15.27:0/1687986766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c58c0 0x7f389c0c5ca0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.014+0000 7f38a2000640 1 -- 172.21.15.27:0/1687986766 >> 172.21.15.27:0/1687986766 conn(0x7f389c06b6f0 msgr2=0x7f389c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:05.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.014+0000 7f38a2000640 1 -- 172.21.15.27:0/1687986766 shutdown_connections 2024-03-28T18:58:05.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.015+0000 7f38a2000640 1 -- 172.21.15.27:0/1687986766 wait complete. 2024-03-28T18:58:05.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.017+0000 7f38a2000640 1 Processor -- start 2024-03-28T18:58:05.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.017+0000 7f38a2000640 1 -- start start 2024-03-28T18:58:05.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c0c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:05.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389c0c58c0 0x7f389c0c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:05.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c8ed0 0x7f389c0c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:05.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c0b0820 con 0x7f389c0c58c0 2024-03-28T18:58:05.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a2000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c159750 con 0x7f389c0c8ed0 2024-03-28T18:58:05.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389c0b09a0 con 0x7f389c0be5d0 2024-03-28T18:58:05.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c0c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:05.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c0c2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:37560/0 (socket says 172.21.15.27:37560) 2024-03-28T18:58:05.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a0c00640 1 -- 172.21.15.27:0/3448660427 learned_addr learned my addr 172.21.15.27:0/3448660427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:05.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f389be00640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389c0c58c0 0x7f389c0c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:05.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.019+0000 7f38a1600640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c8ed0 0x7f389c0c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:05.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.020+0000 7f38a0c00640 1 -- 172.21.15.27:0/3448660427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c8ed0 msgr2=0x7f389c0c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:05.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.020+0000 7f38a0c00640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c8ed0 0x7f389c0c4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.020+0000 7f38a0c00640 1 -- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389c0c58c0 msgr2=0x7f389c0c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:05.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.020+0000 7f38a0c00640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389c0c58c0 0x7f389c0c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.020+0000 7f38a0c00640 1 -- 172.21.15.27:0/3448660427 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c0c3990 con 0x7f389c0be5d0 2024-03-28T18:58:05.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.020+0000 7f389be00640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389c0c58c0 0x7f389c0c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:58:05.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.020+0000 7f38a0c00640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c0c2280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3890000c00 tx=0x7f38900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:05.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.021+0000 7f3898c00640 1 -- 172.21.15.27:0/3448660427 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890032650 con 0x7f389c0be5d0 2024-03-28T18:58:05.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.021+0000 7f3898c00640 1 -- 172.21.15.27:0/3448660427 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3890032810 con 0x7f389c0be5d0 2024-03-28T18:58:05.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.021+0000 7f3898c00640 1 -- 172.21.15.27:0/3448660427 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890032c00 con 0x7f389c0be5d0 2024-03-28T18:58:05.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.021+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c0c3db0 con 0x7f389c0be5d0 2024-03-28T18:58:05.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.021+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c153c60 con 0x7f389c0be5d0 2024-03-28T18:58:05.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.023+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c069640 con 0x7f389c0be5d0 2024-03-28T18:58:05.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.025+0000 7f3898c00640 1 -- 172.21.15.27:0/3448660427 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f389004c060 con 0x7f389c0be5d0 2024-03-28T18:58:05.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.026+0000 7f3898c00640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f387c06bbd0 0x7f387c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:05.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.026+0000 7f3898c00640 1 -- 172.21.15.27:0/3448660427 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f38900ac780 con 0x7f389c0be5d0 2024-03-28T18:58:05.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.027+0000 7f389be00640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f387c06bbd0 0x7f387c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:05.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.028+0000 7f389be00640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f387c06bbd0 0x7f387c06e090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f389c152c10 tx=0x7f3888009060 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:05.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.035+0000 7f3898c00640 1 -- 172.21.15.27:0/3448660427 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3890040070 con 0x7f389c0be5d0 2024-03-28T18:58:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.511+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f389c156580 con 0x7f389c0be5d0 2024-03-28T18:58:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.512+0000 7f3898c00640 1 -- 172.21.15.27:0/3448660427 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f389007a810 con 0x7f389c0be5d0 2024-03-28T18:58:05.512 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:05.512 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-03-28T18:58:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.514+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f387c06bbd0 msgr2=0x7f387c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.514+0000 7f38a2000640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f387c06bbd0 0x7f387c06e090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f389c152c10 tx=0x7f3888009060 comp rx=0 tx=0).stop 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.514+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 msgr2=0x7f389c0c2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.514+0000 7f38a2000640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c0c2280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3890000c00 tx=0x7f38900046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.515+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 shutdown_connections 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.515+0000 7f38a2000640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f387c06bbd0 0x7f387c06e090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.515+0000 7f38a2000640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f389c0c8ed0 0x7f389c0c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.515+0000 7f38a2000640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f389c0c58c0 0x7f389c0c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.515+0000 7f38a2000640 1 --2- 172.21.15.27:0/3448660427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f389c0be5d0 0x7f389c0c2280 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.515+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 >> 172.21.15.27:0/3448660427 conn(0x7f389c06b6f0 msgr2=0x7f389c0c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.515+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 shutdown_connections 2024-03-28T18:58:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:05.515+0000 7f38a2000640 1 -- 172.21.15.27:0/3448660427 wait complete. 2024-03-28T18:58:05.533 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T18:58:07.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:07.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.781+0000 7f12e5c00640 1 Processor -- start 2024-03-28T18:58:07.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.781+0000 7f12e5c00640 1 -- start start 2024-03-28T18:58:07.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.782+0000 7f12e5c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f12e0150ae0 0x7f12e00a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:07.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.782+0000 7f12e5c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0150ec0 con 0x7f12e0153a30 2024-03-28T18:58:07.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.782+0000 7f12e5c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e006dc70 con 0x7f12e00a61a0 2024-03-28T18:58:07.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.782+0000 7f12e5c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e0153550 con 0x7f12e0150ae0 2024-03-28T18:58:07.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.783+0000 7f12e5200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f12e0153a30 0x7f12e015dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42240/0 (socket says 172.21.15.27:42240) 2024-03-28T18:58:07.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.783+0000 7f12e5200640 1 -- 172.21.15.27:0/3537931785 learned_addr learned my addr 172.21.15.27:0/3537931785 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:58:07.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.783+0000 7f12dfe00640 1 --2- 172.21.15.27:0/3537931785 >> v2:172.21.15.155:3300/0 conn(0x7f12e0150ae0 0x7f12e00a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:07.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.783+0000 7f12dea00640 1 -- 172.21.15.27:0/3537931785 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801217321 0 0) 0x7f12d8001490 con 0x7f12e0153a30 2024-03-28T18:58:07.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.783+0000 7f12dea00640 1 -- 172.21.15.27:0/3537931785 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c4003520 con 0x7f12e0153a30 2024-03-28T18:58:07.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.784+0000 7f12dfe00640 1 -- 172.21.15.27:0/3537931785 >> v1:172.21.15.155:6789/0 conn(0x7f12e00a61a0 legacy=0x7f12e00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:07.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.784+0000 7f12dfe00640 1 -- 172.21.15.27:0/3537931785 >> v1:172.21.15.27:6789/0 conn(0x7f12e0153a30 legacy=0x7f12e015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:07.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.784+0000 7f12dfe00640 1 -- 172.21.15.27:0/3537931785 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e006df10 con 0x7f12e0150ae0 2024-03-28T18:58:07.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.785+0000 7f12dfe00640 1 --2- 172.21.15.27:0/3537931785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0150ae0 0x7f12e00a5c60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f12d00097a0 tx=0x7f12d002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=645c852fdae77d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:07.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.786+0000 7f12dea00640 1 -- 172.21.15.27:0/3537931785 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d0035020 con 0x7f12e0150ae0 2024-03-28T18:58:07.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.786+0000 7f12dea00640 1 -- 172.21.15.27:0/3537931785 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d0031640 con 0x7f12e0150ae0 2024-03-28T18:58:07.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.786+0000 7f12dea00640 1 -- 172.21.15.27:0/3537931785 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d002d040 con 0x7f12e0150ae0 2024-03-28T18:58:07.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.787+0000 7f12e5c00640 1 -- 172.21.15.27:0/3537931785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0150ae0 msgr2=0x7f12e00a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:07.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.787+0000 7f12e5c00640 1 --2- 172.21.15.27:0/3537931785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0150ae0 0x7f12e00a5c60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f12d00097a0 tx=0x7f12d002b8f0 comp rx=0 tx=0).stop 2024-03-28T18:58:07.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.787+0000 7f12e5c00640 1 -- 172.21.15.27:0/3537931785 shutdown_connections 2024-03-28T18:58:07.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.787+0000 7f12e5c00640 1 --2- 172.21.15.27:0/3537931785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0150ae0 0x7f12e00a5c60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:07.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.787+0000 7f12e5c00640 1 -- 172.21.15.27:0/3537931785 >> 172.21.15.27:0/3537931785 conn(0x7f12e006b0c0 msgr2=0x7f12e006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:07.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.787+0000 7f12e5c00640 1 -- 172.21.15.27:0/3537931785 shutdown_connections 2024-03-28T18:58:07.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.787+0000 7f12e5c00640 1 -- 172.21.15.27:0/3537931785 wait complete. 2024-03-28T18:58:07.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.789+0000 7f12e5c00640 1 Processor -- start 2024-03-28T18:58:07.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.789+0000 7f12e5c00640 1 -- start start 2024-03-28T18:58:07.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12e5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00a61a0 0x7f12e0152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:07.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12e5c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e0150ae0 0x7f12e0148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:07.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12e5c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0153a30 0x7f12e01490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:07.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12e5c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e006e7b0 con 0x7f12e00a61a0 2024-03-28T18:58:07.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12e5c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e006e4d0 con 0x7f12e0153a30 2024-03-28T18:58:07.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12e5c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e00c70b0 con 0x7f12e0150ae0 2024-03-28T18:58:07.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00a61a0 0x7f12e0152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:07.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00a61a0 0x7f12e0152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36554/0 (socket says 172.21.15.27:36554) 2024-03-28T18:58:07.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12dfe00640 1 -- 172.21.15.27:0/4289202338 learned_addr learned my addr 172.21.15.27:0/4289202338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.790+0000 7f12e5200640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0153a30 0x7f12e01490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.791+0000 7f12df400640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e0150ae0 0x7f12e0148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.791+0000 7f12dfe00640 1 -- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e0150ae0 msgr2=0x7f12e0148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.791+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e0150ae0 0x7f12e0148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.791+0000 7f12dfe00640 1 -- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0153a30 msgr2=0x7f12e01490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.791+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0153a30 0x7f12e01490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.791+0000 7f12dfe00640 1 -- 172.21.15.27:0/4289202338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e0149630 con 0x7f12e00a61a0 2024-03-28T18:58:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.791+0000 7f12df400640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e0150ae0 0x7f12e0148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:58:07.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.791+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00a61a0 0x7f12e0152030 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f12d002bf00 tx=0x7f12d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:07.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.792+0000 7f12d7e00640 1 -- 172.21.15.27:0/4289202338 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d0040070 con 0x7f12e00a61a0 2024-03-28T18:58:07.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.792+0000 7f12d7e00640 1 -- 172.21.15.27:0/4289202338 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d00399f0 con 0x7f12e00a61a0 2024-03-28T18:58:07.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.792+0000 7f12d7e00640 1 -- 172.21.15.27:0/4289202338 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d0035040 con 0x7f12e00a61a0 2024-03-28T18:58:07.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.792+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e00bbe40 con 0x7f12e00a61a0 2024-03-28T18:58:07.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.793+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e00bc100 con 0x7f12e00a61a0 2024-03-28T18:58:07.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.794+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e0067dd0 con 0x7f12e00a61a0 2024-03-28T18:58:07.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.796+0000 7f12d7e00640 1 -- 172.21.15.27:0/4289202338 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f12d0038020 con 0x7f12e00a61a0 2024-03-28T18:58:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.797+0000 7f12d7e00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bc20 0x7f12c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.797+0000 7f12d7e00640 1 -- 172.21.15.27:0/4289202338 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f12d00ac690 con 0x7f12e00a61a0 2024-03-28T18:58:07.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.798+0000 7f12df400640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bc20 0x7f12c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:07.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.799+0000 7f12df400640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bc20 0x7f12c006e0e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f12e00ab660 tx=0x7f12cc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:07.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:07.803+0000 7f12d7e00640 1 -- 172.21.15.27:0/4289202338 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f12d007a820 con 0x7f12e00a61a0 2024-03-28T18:58:08.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.075+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e006da50 con 0x7f12e00a61a0 2024-03-28T18:58:08.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.076+0000 7f12d7e00640 1 -- 172.21.15.27:0/4289202338 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f12d007e8c0 con 0x7f12e00a61a0 2024-03-28T18:58:08.077 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:08.077 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:55:38.022816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.155:6834/1270773218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1270773218},{"type":"v1","addr":"172.21.15.155:6834","nonce":1270773218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4273,"qdb_cluster":[4273]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:08.077 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-03-28T18:58:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bc20 msgr2=0x7f12c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bc20 0x7f12c006e0e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f12e00ab660 tx=0x7f12cc009040 comp rx=0 tx=0).stop 2024-03-28T18:58:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00a61a0 msgr2=0x7f12e0152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00a61a0 0x7f12e0152030 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f12d002bf00 tx=0x7f12d00046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 shutdown_connections 2024-03-28T18:58:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bc20 0x7f12c006e0e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e0153a30 0x7f12e01490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e0150ae0 0x7f12e0148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:08.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.079+0000 7f12e5c00640 1 --2- 172.21.15.27:0/4289202338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00a61a0 0x7f12e0152030 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:08.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.080+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 >> 172.21.15.27:0/4289202338 conn(0x7f12e006b0c0 msgr2=0x7f12e01500d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:08.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.080+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 shutdown_connections 2024-03-28T18:58:08.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:08.080+0000 7f12e5c00640 1 -- 172.21.15.27:0/4289202338 wait complete. 2024-03-28T18:58:08.104 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T18:58:09.795 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 63% (7740/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T18:58:10.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:10.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.328+0000 7f7d46c00640 1 Processor -- start 2024-03-28T18:58:10.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.328+0000 7f7d46c00640 1 -- start start 2024-03-28T18:58:10.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.329+0000 7f7d46c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7d400c52f0 0x7f7d400c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.330+0000 7f7d46c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d400bdbd0 con 0x7f7d400c8480 2024-03-28T18:58:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.330+0000 7f7d46c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d400bf5a0 con 0x7f7d400a8240 2024-03-28T18:58:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.330+0000 7f7d46c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d400bf910 con 0x7f7d400c52f0 2024-03-28T18:58:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.330+0000 7f7d44e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7d400c8480 0x7f7d40153360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42242/0 (socket says 172.21.15.27:42242) 2024-03-28T18:58:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.330+0000 7f7d44e00640 1 -- 172.21.15.27:0/487817608 learned_addr learned my addr 172.21.15.27:0/487817608 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:58:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.330+0000 7f7d45800640 1 --2- 172.21.15.27:0/487817608 >> v2:172.21.15.155:3300/0 conn(0x7f7d400c52f0 0x7f7d400c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.331+0000 7f7d45800640 1 -- 172.21.15.27:0/487817608 >> v1:172.21.15.155:6789/0 conn(0x7f7d400a8240 legacy=0x7f7d4015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.331+0000 7f7d45800640 1 -- 172.21.15.27:0/487817608 >> v1:172.21.15.27:6789/0 conn(0x7f7d400c8480 legacy=0x7f7d40153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.331+0000 7f7d45800640 1 -- 172.21.15.27:0/487817608 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40153a70 con 0x7f7d400c52f0 2024-03-28T18:58:10.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.332+0000 7f7d45800640 1 --2- 172.21.15.27:0/487817608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400c52f0 0x7f7d400c56d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7d400bfa90 tx=0x7f7d3402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13bf92e6bd8d01af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:10.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.332+0000 7f7d3fe00640 1 -- 172.21.15.27:0/487817608 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34034070 con 0x7f7d400c52f0 2024-03-28T18:58:10.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.333+0000 7f7d3fe00640 1 -- 172.21.15.27:0/487817608 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34030800 con 0x7f7d400c52f0 2024-03-28T18:58:10.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.333+0000 7f7d3fe00640 1 -- 172.21.15.27:0/487817608 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34030bd0 con 0x7f7d400c52f0 2024-03-28T18:58:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.333+0000 7f7d46c00640 1 -- 172.21.15.27:0/487817608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400c52f0 msgr2=0x7f7d400c56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.333+0000 7f7d46c00640 1 --2- 172.21.15.27:0/487817608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400c52f0 0x7f7d400c56d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7d400bfa90 tx=0x7f7d3402b940 comp rx=0 tx=0).stop 2024-03-28T18:58:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.333+0000 7f7d46c00640 1 -- 172.21.15.27:0/487817608 shutdown_connections 2024-03-28T18:58:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.333+0000 7f7d46c00640 1 --2- 172.21.15.27:0/487817608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400c52f0 0x7f7d400c56d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.333+0000 7f7d46c00640 1 -- 172.21.15.27:0/487817608 >> 172.21.15.27:0/487817608 conn(0x7f7d400aa7c0 msgr2=0x7f7d400aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.333+0000 7f7d46c00640 1 -- 172.21.15.27:0/487817608 shutdown_connections 2024-03-28T18:58:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.334+0000 7f7d46c00640 1 -- 172.21.15.27:0/487817608 wait complete. 2024-03-28T18:58:10.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.335+0000 7f7d46c00640 1 Processor -- start 2024-03-28T18:58:10.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.335+0000 7f7d46c00640 1 -- start start 2024-03-28T18:58:10.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.336+0000 7f7d46c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a8240 0x7f7d4014f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.336+0000 7f7d46c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d400c52f0 0x7f7d40151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.336+0000 7f7d46c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400c8480 0x7f7d401eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.336+0000 7f7d46c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400a7e00 con 0x7f7d400c52f0 2024-03-28T18:58:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.336+0000 7f7d46c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400a7c80 con 0x7f7d400a8240 2024-03-28T18:58:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.336+0000 7f7d46c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40157530 con 0x7f7d400c8480 2024-03-28T18:58:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.337+0000 7f7d44e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d400c52f0 0x7f7d40151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.337+0000 7f7d45800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a8240 0x7f7d4014f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.337+0000 7f7d45800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a8240 0x7f7d4014f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:49938/0 (socket says 172.21.15.27:49938) 2024-03-28T18:58:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.337+0000 7f7d45800640 1 -- 172.21.15.27:0/2286646003 learned_addr learned my addr 172.21.15.27:0/2286646003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.337+0000 7f7d46200640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400c8480 0x7f7d401eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.338+0000 7f7d45800640 1 -- 172.21.15.27:0/2286646003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400c8480 msgr2=0x7f7d401eeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.338+0000 7f7d45800640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400c8480 0x7f7d401eeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.338+0000 7f7d45800640 1 -- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d400c52f0 msgr2=0x7f7d40151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.338+0000 7f7d45800640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d400c52f0 0x7f7d40151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.338+0000 7f7d45800640 1 -- 172.21.15.27:0/2286646003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d400c72b0 con 0x7f7d400a8240 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.338+0000 7f7d44e00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d400c52f0 0x7f7d40151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.338+0000 7f7d45800640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a8240 0x7f7d4014f9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009890 tx=0x7f7d340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.339+0000 7f7d3d600640 1 -- 172.21.15.27:0/2286646003 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34046020 con 0x7f7d400a8240 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.339+0000 7f7d3d600640 1 -- 172.21.15.27:0/2286646003 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34039d10 con 0x7f7d400a8240 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.339+0000 7f7d3d600640 1 -- 172.21.15.27:0/2286646003 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34034040 con 0x7f7d400a8240 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.339+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d401ef600 con 0x7f7d400a8240 2024-03-28T18:58:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.339+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d400bfeb0 con 0x7f7d400a8240 2024-03-28T18:58:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.341+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d40067dd0 con 0x7f7d400a8240 2024-03-28T18:58:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.342+0000 7f7d3d600640 1 -- 172.21.15.27:0/2286646003 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7d34037070 con 0x7f7d400a8240 2024-03-28T18:58:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.344+0000 7f7d3d600640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.344+0000 7f7d3d600640 1 -- 172.21.15.27:0/2286646003 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7d340adb90 con 0x7f7d400a8240 2024-03-28T18:58:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.345+0000 7f7d44e00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.346+0000 7f7d44e00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2006bbd0 0x7f7d2006e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d400a9270 tx=0x7f7d2c007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:10.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.356+0000 7f7d3d600640 1 -- 172.21.15.27:0/2286646003 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7d34050020 con 0x7f7d400a8240 2024-03-28T18:58:10.543 INFO:tasks.workunit.client.1.smithi155.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T18:58:10.543 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-28T18:58:10.543 INFO:tasks.workunit.client.1.smithi155.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T18:58:10.543 INFO:tasks.workunit.client.1.smithi155.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T18:58:10.543 INFO:tasks.workunit.client.1.smithi155.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr: git switch -c 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr:Or undo this operation with: 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr: git switch - 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-28T18:58:10.544 INFO:tasks.workunit.client.1.smithi155.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T18:58:10.545 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-28T18:58:10.545 INFO:tasks.workunit.client.1.smithi155.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T18:58:10.552 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-28T18:58:10.610 INFO:tasks.workunit.client.1.smithi155.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T18:58:10.616 INFO:tasks.workunit.client.1.smithi155.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-28T18:58:10.616 INFO:tasks.workunit.client.1.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T18:58:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.676+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d40069640 con 0x7f7d400a8240 2024-03-28T18:58:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.678+0000 7f7d3d600640 1 -- 172.21.15.27:0/2286646003 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5460 (secure 0 0 0) 0x7f7d3407bc70 con 0x7f7d400a8240 2024-03-28T18:58:10.679 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:10.679 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:09.579258+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 22 2024-03-28T18:58:10.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2006bbd0 msgr2=0x7f7d2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2006bbd0 0x7f7d2006e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d400a9270 tx=0x7f7d2c007e80 comp rx=0 tx=0).stop 2024-03-28T18:58:10.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a8240 msgr2=0x7f7d4014f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a8240 0x7f7d4014f9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009890 tx=0x7f7d340046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 shutdown_connections 2024-03-28T18:58:10.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2006bbd0 0x7f7d2006e090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400c8480 0x7f7d401eeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d400c52f0 0x7f7d40151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 --2- 172.21.15.27:0/2286646003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a8240 0x7f7d4014f9c0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009890 tx=0x7f7d340046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 >> 172.21.15.27:0/2286646003 conn(0x7f7d400aa7c0 msgr2=0x7f7d400a86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 shutdown_connections 2024-03-28T18:58:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.681+0000 7f7d46c00640 1 -- 172.21.15.27:0/2286646003 wait complete. 2024-03-28T18:58:10.707 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-03-28T18:58:10.707 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T18:58:10.707 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:10.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.953+0000 7fa3e1600640 1 Processor -- start 2024-03-28T18:58:10.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.953+0000 7fa3e1600640 1 -- start start 2024-03-28T18:58:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.955+0000 7fa3e1600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc156c40 0x7fa3dc0c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.955+0000 7fa3e1600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc070510 con 0x7fa3dc152e70 2024-03-28T18:58:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.955+0000 7fa3e1600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc0707f0 con 0x7fa3dc0c2d80 2024-03-28T18:58:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.955+0000 7fa3e1600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc156770 con 0x7fa3dc156c40 2024-03-28T18:58:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.956+0000 7fa3daa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa3dc152e70 0x7fa3dc06fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42256/0 (socket says 172.21.15.27:42256) 2024-03-28T18:58:10.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.956+0000 7fa3daa00640 1 -- 172.21.15.27:0/513613417 learned_addr learned my addr 172.21.15.27:0/513613417 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:58:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.956+0000 7fa3db400640 1 --2- 172.21.15.27:0/513613417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc156c40 0x7fa3dc0c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.956+0000 7fa3d9600640 1 -- 172.21.15.27:0/513613417 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312331385 0 0) 0x7fa3dc070510 con 0x7fa3dc152e70 2024-03-28T18:58:10.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.957+0000 7fa3d9600640 1 -- 172.21.15.27:0/513613417 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c0003520 con 0x7fa3dc152e70 2024-03-28T18:58:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.957+0000 7fa3db400640 1 -- 172.21.15.27:0/513613417 >> v1:172.21.15.155:6789/0 conn(0x7fa3dc0c2d80 legacy=0x7fa3dc156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.957+0000 7fa3db400640 1 -- 172.21.15.27:0/513613417 >> v1:172.21.15.27:6789/0 conn(0x7fa3dc152e70 legacy=0x7fa3dc06fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.957+0000 7fa3db400640 1 -- 172.21.15.27:0/513613417 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc070ad0 con 0x7fa3dc156c40 2024-03-28T18:58:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.959+0000 7fa3db400640 1 --2- 172.21.15.27:0/513613417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc156c40 0x7fa3dc0c8d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009dc0 tx=0x7fa3d402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecf49221a5b50ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.959+0000 7fa3d9600640 1 -- 172.21.15.27:0/513613417 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4039070 con 0x7fa3dc156c40 2024-03-28T18:58:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.959+0000 7fa3d9600640 1 -- 172.21.15.27:0/513613417 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d402cc40 con 0x7fa3dc156c40 2024-03-28T18:58:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.959+0000 7fa3d9600640 1 -- 172.21.15.27:0/513613417 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4034440 con 0x7fa3dc156c40 2024-03-28T18:58:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.960+0000 7fa3e1600640 1 -- 172.21.15.27:0/513613417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc156c40 msgr2=0x7fa3dc0c8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.960+0000 7fa3e1600640 1 --2- 172.21.15.27:0/513613417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc156c40 0x7fa3dc0c8d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009dc0 tx=0x7fa3d402be90 comp rx=0 tx=0).stop 2024-03-28T18:58:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.960+0000 7fa3e1600640 1 -- 172.21.15.27:0/513613417 shutdown_connections 2024-03-28T18:58:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.960+0000 7fa3e1600640 1 --2- 172.21.15.27:0/513613417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc156c40 0x7fa3dc0c8d70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.960+0000 7fa3e1600640 1 -- 172.21.15.27:0/513613417 >> 172.21.15.27:0/513613417 conn(0x7fa3dc066dd0 msgr2=0x7fa3dc0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.961+0000 7fa3e1600640 1 -- 172.21.15.27:0/513613417 shutdown_connections 2024-03-28T18:58:10.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.961+0000 7fa3e1600640 1 -- 172.21.15.27:0/513613417 wait complete. 2024-03-28T18:58:10.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.963+0000 7fa3e1600640 1 Processor -- start 2024-03-28T18:58:10.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.963+0000 7fa3e1600640 1 -- start start 2024-03-28T18:58:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3e1600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc0c2d80 0x7fa3dc0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3e1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3dc152e70 0x7fa3dc06d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3e1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3dc156c40 0x7fa3dc1ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3e1600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc155b40 con 0x7fa3dc152e70 2024-03-28T18:58:10.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3e1600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc153250 con 0x7fa3dc156c40 2024-03-28T18:58:10.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3e1600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc1533d0 con 0x7fa3dc0c2d80 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3da000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3dc152e70 0x7fa3dc06d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3da000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3dc152e70 0x7fa3dc06d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36584/0 (socket says 172.21.15.27:36584) 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.965+0000 7fa3da000640 1 -- 172.21.15.27:0/1889621845 learned_addr learned my addr 172.21.15.27:0/1889621845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.966+0000 7fa3db400640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3dc156c40 0x7fa3dc1ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.966+0000 7fa3daa00640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc0c2d80 0x7fa3dc0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.966+0000 7fa3da000640 1 -- 172.21.15.27:0/1889621845 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc0c2d80 msgr2=0x7fa3dc0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.966+0000 7fa3da000640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc0c2d80 0x7fa3dc0ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.966+0000 7fa3da000640 1 -- 172.21.15.27:0/1889621845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3dc156c40 msgr2=0x7fa3dc1ee8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.966+0000 7fa3da000640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3dc156c40 0x7fa3dc1ee8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.966+0000 7fa3da000640 1 -- 172.21.15.27:0/1889621845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc06ca70 con 0x7fa3dc152e70 2024-03-28T18:58:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.967+0000 7fa3da000640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3dc152e70 0x7fa3dc06d250 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800c3c0 tx=0x7fa3c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.967+0000 7fa3d2a00640 1 -- 172.21.15.27:0/1889621845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80052d0 con 0x7fa3dc152e70 2024-03-28T18:58:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.967+0000 7fa3d2a00640 1 -- 172.21.15.27:0/1889621845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8005490 con 0x7fa3dc152e70 2024-03-28T18:58:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.967+0000 7fa3d2a00640 1 -- 172.21.15.27:0/1889621845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8007260 con 0x7fa3dc152e70 2024-03-28T18:58:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.967+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc0ac030 con 0x7fa3dc152e70 2024-03-28T18:58:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.968+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc06d790 con 0x7fa3dc152e70 2024-03-28T18:58:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.969+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc06bad0 con 0x7fa3dc152e70 2024-03-28T18:58:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.972+0000 7fa3d2a00640 1 -- 172.21.15.27:0/1889621845 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa3c8007400 con 0x7fa3dc152e70 2024-03-28T18:58:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.973+0000 7fa3d2a00640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa3bc06bbd0 0x7fa3bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.973+0000 7fa3d2a00640 1 -- 172.21.15.27:0/1889621845 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa3c808ab50 con 0x7fa3dc152e70 2024-03-28T18:58:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.974+0000 7fa3daa00640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa3bc06bbd0 0x7fa3bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:10.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.975+0000 7fa3daa00640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa3bc06bbd0 0x7fa3bc06e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0011e0 tx=0x7fa3cc007d30 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:10.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:10.982+0000 7fa3d2a00640 1 -- 172.21.15.27:0/1889621845 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa3dc06bad0 con 0x7fa3dc152e70 2024-03-28T18:58:11.452 INFO:tasks.workunit.client.1.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T18:58:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.456+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3dc04f400 con 0x7fa3dc152e70 2024-03-28T18:58:11.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.457+0000 7fa3d2a00640 1 -- 172.21.15.27:0/1889621845 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5460 (secure 0 0 0) 0x7fa3c8058c30 con 0x7fa3dc152e70 2024-03-28T18:58:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:11.458 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:09.579258+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:11.458 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 22 2024-03-28T18:58:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.460+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa3bc06bbd0 msgr2=0x7fa3bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.460+0000 7fa3e1600640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa3bc06bbd0 0x7fa3bc06e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc0011e0 tx=0x7fa3cc007d30 comp rx=0 tx=0).stop 2024-03-28T18:58:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.460+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3dc152e70 msgr2=0x7fa3dc06d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.460+0000 7fa3e1600640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3dc152e70 0x7fa3dc06d250 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800c3c0 tx=0x7fa3c800c880 comp rx=0 tx=0).stop 2024-03-28T18:58:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.461+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 shutdown_connections 2024-03-28T18:58:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.461+0000 7fa3e1600640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa3bc06bbd0 0x7fa3bc06e090 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.461+0000 7fa3e1600640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3dc156c40 0x7fa3dc1ee8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.461+0000 7fa3e1600640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3dc152e70 0x7fa3dc06d250 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.461+0000 7fa3e1600640 1 --2- 172.21.15.27:0/1889621845 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3dc0c2d80 0x7fa3dc0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.461+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 >> 172.21.15.27:0/1889621845 conn(0x7fa3dc066dd0 msgr2=0x7fa3dc154f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.461+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 shutdown_connections 2024-03-28T18:58:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:11.461+0000 7fa3e1600640 1 -- 172.21.15.27:0/1889621845 wait complete. 2024-03-28T18:58:11.486 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5166": { "addr": "172.21.15.27:6837/3329030532", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 3329030532, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 3329030532, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5166, "incarnation": 21, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T18:58:09.579258+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5166 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4260": { "addr": "172.21.15.27:6836/3119083312", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 3119083312, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 3119083312, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4260, "incarnation": 16, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T18:57:22.544784+0000", "qdb_cluster": [ 4260 ], "qdb_leader": 4260, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4260 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T18:58:11.562 INFO:tasks.workunit.client.1.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T18:58:11.644 INFO:tasks.workunit.client.1.smithi155.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-28T18:58:11.648 INFO:tasks.workunit.client.1.smithi155.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-28T18:58:11.649 INFO:tasks.workunit.client.1.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T18:58:11.725 INFO:tasks.workunit.client.1.smithi155.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-28T18:58:11.735 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-28T18:58:11.735 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-28T18:58:11.783 INFO:tasks.workunit:Running workunits matching fs/snaps on client.1... 2024-03-28T18:58:11.784 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-03-28T18:58:11.784 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-03-28T18:58:11.918 INFO:tasks.workunit.client.1.smithi155.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-03-28T18:58:12.602 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T18:58:12.601+0000 7faf8de00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:58:13.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:13.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.696+0000 7fb97c200640 1 Processor -- start 2024-03-28T18:58:13.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.696+0000 7fb97c200640 1 -- start start 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb97c200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb974071cf0 0x7fb974159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb97c200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 0x7fb9740c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb97c200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb974072870 con 0x7fb97414f6d0 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb97c200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb974156640 con 0x7fb974071cf0 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb97c200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb974158db0 con 0x7fb974072c60 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb978e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb974071cf0 0x7fb974159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb978e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb974071cf0 0x7fb974159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36598/0 (socket says 172.21.15.27:36598) 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb978e00640 1 -- 172.21.15.27:0/2065331138 learned_addr learned my addr 172.21.15.27:0/2065331138 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.698+0000 7fb979800640 1 --1- 172.21.15.27:0/2065331138 >> v1:172.21.15.27:6789/0 conn(0x7fb97414f6d0 0x7fb9740c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42258/0 (socket says 172.21.15.27:42258) 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.699+0000 7fb973e00640 1 -- 172.21.15.27:0/2065331138 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715672322 0 0) 0x7fb968001110 con 0x7fb97414f6d0 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.699+0000 7fb973e00640 1 -- 172.21.15.27:0/2065331138 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb95c003520 con 0x7fb97414f6d0 2024-03-28T18:58:13.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.699+0000 7fb97a200640 1 --2- 172.21.15.27:0/2065331138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 0x7fb9740c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:13.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.699+0000 7fb978e00640 1 -- 172.21.15.27:0/2065331138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 msgr2=0x7fb9740c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:13.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.699+0000 7fb978e00640 1 --2- 172.21.15.27:0/2065331138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 0x7fb9740c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.699+0000 7fb973e00640 1 -- 172.21.15.27:0/2065331138 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2851608580 0 0) 0x7fb95c003520 con 0x7fb97414f6d0 2024-03-28T18:58:13.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.699+0000 7fb978e00640 1 -- 172.21.15.27:0/2065331138 >> v1:172.21.15.27:6789/0 conn(0x7fb97414f6d0 legacy=0x7fb9740c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:13.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.699+0000 7fb978e00640 1 -- 172.21.15.27:0/2065331138 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb974156360 con 0x7fb974071cf0 2024-03-28T18:58:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.700+0000 7fb978e00640 1 --2- 172.21.15.27:0/2065331138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb974071cf0 0x7fb974159850 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb960009870 tx=0x7fb96002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9baeaed56434373 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.700+0000 7fb973e00640 1 -- 172.21.15.27:0/2065331138 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb960037070 con 0x7fb974071cf0 2024-03-28T18:58:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.701+0000 7fb973e00640 1 -- 172.21.15.27:0/2065331138 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb960004030 con 0x7fb974071cf0 2024-03-28T18:58:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.701+0000 7fb973e00640 1 -- 172.21.15.27:0/2065331138 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb968001840 con 0x7fb974071cf0 2024-03-28T18:58:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.701+0000 7fb97c200640 1 -- 172.21.15.27:0/2065331138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb974071cf0 msgr2=0x7fb974159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.701+0000 7fb97c200640 1 --2- 172.21.15.27:0/2065331138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb974071cf0 0x7fb974159850 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb960009870 tx=0x7fb96002b7e0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.701+0000 7fb97c200640 1 -- 172.21.15.27:0/2065331138 shutdown_connections 2024-03-28T18:58:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.701+0000 7fb97c200640 1 --2- 172.21.15.27:0/2065331138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 0x7fb9740c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.701+0000 7fb97c200640 1 --2- 172.21.15.27:0/2065331138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb974071cf0 0x7fb974159850 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.701+0000 7fb97c200640 1 -- 172.21.15.27:0/2065331138 >> 172.21.15.27:0/2065331138 conn(0x7fb97406c5f0 msgr2=0x7fb97406ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.702+0000 7fb97c200640 1 -- 172.21.15.27:0/2065331138 shutdown_connections 2024-03-28T18:58:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.702+0000 7fb97c200640 1 -- 172.21.15.27:0/2065331138 wait complete. 2024-03-28T18:58:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.703+0000 7fb97c200640 1 Processor -- start 2024-03-28T18:58:13.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.703+0000 7fb97c200640 1 -- start start 2024-03-28T18:58:13.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.704+0000 7fb97c200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb974071cf0 0x7fb974158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:13.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.704+0000 7fb97c200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 0x7fb9740c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:13.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.704+0000 7fb97c200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb97414f6d0 0x7fb9740c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:13.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.704+0000 7fb97c200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97414fab0 con 0x7fb97414f6d0 2024-03-28T18:58:13.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.704+0000 7fb97c200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb974159330 con 0x7fb974071cf0 2024-03-28T18:58:13.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.704+0000 7fb97c200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9740c3c70 con 0x7fb974072c60 2024-03-28T18:58:13.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb97a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb97414f6d0 0x7fb9740c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:13.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb97a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb97414f6d0 0x7fb9740c72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36604/0 (socket says 172.21.15.27:36604) 2024-03-28T18:58:13.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb97a200640 1 -- 172.21.15.27:0/3556804351 learned_addr learned my addr 172.21.15.27:0/3556804351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:13.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb979800640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb974071cf0 0x7fb974158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb978e00640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 0x7fb9740c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb97a200640 1 -- 172.21.15.27:0/3556804351 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 msgr2=0x7fb9740c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb97a200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 0x7fb9740c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb97a200640 1 -- 172.21.15.27:0/3556804351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb974071cf0 msgr2=0x7fb974158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb97a200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb974071cf0 0x7fb974158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.705+0000 7fb97a200640 1 -- 172.21.15.27:0/3556804351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb974153f10 con 0x7fb97414f6d0 2024-03-28T18:58:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.706+0000 7fb97a200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb97414f6d0 0x7fb9740c72e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb96c007cf0 tx=0x7fb96c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.706+0000 7fb971600640 1 -- 172.21.15.27:0/3556804351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c014420 con 0x7fb97414f6d0 2024-03-28T18:58:13.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.706+0000 7fb971600640 1 -- 172.21.15.27:0/3556804351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96c019070 con 0x7fb97414f6d0 2024-03-28T18:58:13.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.706+0000 7fb971600640 1 -- 172.21.15.27:0/3556804351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c014820 con 0x7fb97414f6d0 2024-03-28T18:58:13.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.706+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9741546a0 con 0x7fb97414f6d0 2024-03-28T18:58:13.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.706+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97414ab70 con 0x7fb97414f6d0 2024-03-28T18:58:13.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.707+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb974154870 con 0x7fb97414f6d0 2024-03-28T18:58:13.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.710+0000 7fb971600640 1 -- 172.21.15.27:0/3556804351 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb96c0149c0 con 0x7fb97414f6d0 2024-03-28T18:58:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.711+0000 7fb971600640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb95406bbd0 0x7fb95406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.711+0000 7fb971600640 1 -- 172.21.15.27:0/3556804351 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb96c08b610 con 0x7fb97414f6d0 2024-03-28T18:58:13.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.711+0000 7fb979800640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb95406bbd0 0x7fb95406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:13.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.712+0000 7fb979800640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb95406bbd0 0x7fb95406e090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb968001370 tx=0x7fb968009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:13.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.716+0000 7fb971600640 1 -- 172.21.15.27:0/3556804351 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb96c059ca0 con 0x7fb97414f6d0 2024-03-28T18:58:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.968+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb97404f400 con 0x7fb97414f6d0 2024-03-28T18:58:13.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.968+0000 7fb971600640 1 -- 172.21.15.27:0/3556804351 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5464 (secure 0 0 0) 0x7fb96c0595a0 con 0x7fb97414f6d0 2024-03-28T18:58:13.969 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:13.969 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:13.969 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-03-28T18:58:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb95406bbd0 msgr2=0x7fb95406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb95406bbd0 0x7fb95406e090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb968001370 tx=0x7fb968009040 comp rx=0 tx=0).stop 2024-03-28T18:58:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb97414f6d0 msgr2=0x7fb9740c72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb97414f6d0 0x7fb9740c72e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb96c007cf0 tx=0x7fb96c00e4b0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 shutdown_connections 2024-03-28T18:58:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb95406bbd0 0x7fb95406e090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb97414f6d0 0x7fb9740c72e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb974072c60 0x7fb9740c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 --2- 172.21.15.27:0/3556804351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb974071cf0 0x7fb974158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.972+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 >> 172.21.15.27:0/3556804351 conn(0x7fb97406c5f0 msgr2=0x7fb97414d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.973+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 shutdown_connections 2024-03-28T18:58:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:13.973+0000 7fb97c200640 1 -- 172.21.15.27:0/3556804351 wait complete. 2024-03-28T18:58:13.990 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4273 2024-03-28T18:58:13.990 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.d 2024-03-28T18:58:17.255 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 58% (7194/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 92% (11309/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr: git switch -c 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-03-28T18:58:17.752 INFO:tasks.workunit.client.0.smithi027.stderr:Or undo this operation with: 2024-03-28T18:58:17.753 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-03-28T18:58:17.753 INFO:tasks.workunit.client.0.smithi027.stderr: git switch - 2024-03-28T18:58:17.753 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-03-28T18:58:17.753 INFO:tasks.workunit.client.0.smithi027.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T18:58:17.753 INFO:tasks.workunit.client.0.smithi027.stderr: 2024-03-28T18:58:17.753 INFO:tasks.workunit.client.0.smithi027.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T18:58:17.761 DEBUG:teuthology.orchestra.run.smithi027:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T18:58:17.907 INFO:tasks.workunit.client.0.smithi027.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T18:58:17.914 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T18:58:17.914 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T18:58:18.768 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T18:58:18.889 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T18:58:18.983 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T18:58:18.990 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T18:58:18.990 INFO:tasks.workunit.client.0.smithi027.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T18:58:19.066 INFO:tasks.workunit.client.0.smithi027.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T18:58:19.074 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-03-28T18:58:19.074 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T18:58:19.115 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-03-28T18:58:19.117 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-03-28T18:58:19.117 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-03-28T18:58:19.235 INFO:tasks.workunit.client.0.smithi027.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-03-28T18:58:19.991 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-03-28T18:58:19.991 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T18:58:19.992 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T18:58:20.000 INFO:tasks.ceph.mds.d:Started 2024-03-28T18:58:20.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:20.182 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T18:58:20.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.231+0000 7f8a7ae00640 1 Processor -- start 2024-03-28T18:58:20.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.231+0000 7f8a7ae00640 1 -- start start 2024-03-28T18:58:20.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.232+0000 7f8a7ae00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8a74150ae0 0x7f8a740a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:20.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.232+0000 7f8a7ae00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a740a61a0 0x7f8a740c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:20.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.232+0000 7f8a7ae00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a74150ec0 con 0x7f8a740c6ee0 2024-03-28T18:58:20.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.232+0000 7f8a7ae00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7406dc70 con 0x7f8a740a61a0 2024-03-28T18:58:20.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.232+0000 7f8a7ae00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a74153550 con 0x7f8a74150ae0 2024-03-28T18:58:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.233+0000 7f8a79a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8a74150ae0 0x7f8a740a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.233+0000 7f8a79a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8a74150ae0 0x7f8a740a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:46876/0 (socket says 172.21.15.27:46876) 2024-03-28T18:58:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.233+0000 7f8a79a00640 1 -- 172.21.15.27:0/4245398198 learned_addr learned my addr 172.21.15.27:0/4245398198 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.233+0000 7f8a79000640 1 --2- 172.21.15.27:0/4245398198 >> v2:172.21.15.27:3300/0 conn(0x7f8a740a61a0 0x7f8a740c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.233+0000 7f8a79a00640 1 -- 172.21.15.27:0/4245398198 >> v2:172.21.15.27:3300/0 conn(0x7f8a740a61a0 msgr2=0x7f8a740c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.233+0000 7f8a79a00640 1 --2- 172.21.15.27:0/4245398198 >> v2:172.21.15.27:3300/0 conn(0x7f8a740a61a0 0x7f8a740c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.233+0000 7f8a79a00640 1 -- 172.21.15.27:0/4245398198 >> v1:172.21.15.155:6789/0 conn(0x7f8a740c6ee0 legacy=0x7f8a74153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:20.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.233+0000 7f8a79a00640 1 -- 172.21.15.27:0/4245398198 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7406df10 con 0x7f8a74150ae0 2024-03-28T18:58:20.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.235+0000 7f8a79a00640 1 --2- 172.21.15.27:0/4245398198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 0x7f8a740a5c60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a9760 tx=0x7f8a6802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90254b33c045b86f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:20.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.235+0000 7f8a73e00640 1 -- 172.21.15.27:0/4245398198 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68035020 con 0x7f8a74150ae0 2024-03-28T18:58:20.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.235+0000 7f8a73e00640 1 -- 172.21.15.27:0/4245398198 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a68031870 con 0x7f8a74150ae0 2024-03-28T18:58:20.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.235+0000 7f8a73e00640 1 -- 172.21.15.27:0/4245398198 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68031c40 con 0x7f8a74150ae0 2024-03-28T18:58:20.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.236+0000 7f8a7ae00640 1 -- 172.21.15.27:0/4245398198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 msgr2=0x7f8a740a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.236+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/4245398198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 0x7f8a740a5c60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a9760 tx=0x7f8a6802b910 comp rx=0 tx=0).stop 2024-03-28T18:58:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.236+0000 7f8a7ae00640 1 -- 172.21.15.27:0/4245398198 shutdown_connections 2024-03-28T18:58:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.236+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/4245398198 >> v2:172.21.15.27:3300/0 conn(0x7f8a740a61a0 0x7f8a740c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.236+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/4245398198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 0x7f8a740a5c60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.236+0000 7f8a7ae00640 1 -- 172.21.15.27:0/4245398198 >> 172.21.15.27:0/4245398198 conn(0x7f8a7406b0c0 msgr2=0x7f8a7406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.236+0000 7f8a7ae00640 1 -- 172.21.15.27:0/4245398198 shutdown_connections 2024-03-28T18:58:20.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.236+0000 7f8a7ae00640 1 -- 172.21.15.27:0/4245398198 wait complete. 2024-03-28T18:58:20.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.239+0000 7f8a7ae00640 1 Processor -- start 2024-03-28T18:58:20.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.239+0000 7f8a7ae00640 1 -- start start 2024-03-28T18:58:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.240+0000 7f8a7ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a740a61a0 0x7f8a741e1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.240+0000 7f8a7ae00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a740c6ee0 0x7f8a741e1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.240+0000 7f8a7ae00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 0x7f8a740719d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.240+0000 7f8a7ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7406e7b0 con 0x7f8a740a61a0 2024-03-28T18:58:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.240+0000 7f8a7ae00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7406e4d0 con 0x7f8a74150ae0 2024-03-28T18:58:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.240+0000 7f8a7ae00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a741569d0 con 0x7f8a740c6ee0 2024-03-28T18:58:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a740a61a0 0x7f8a741e1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:20.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a740a61a0 0x7f8a741e1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53852/0 (socket says 172.21.15.27:53852) 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79a00640 1 -- 172.21.15.27:0/725947370 learned_addr learned my addr 172.21.15.27:0/725947370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79000640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a740c6ee0 0x7f8a741e1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a7a400640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 0x7f8a740719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79a00640 1 -- 172.21.15.27:0/725947370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a740c6ee0 msgr2=0x7f8a741e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79a00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a740c6ee0 0x7f8a741e1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79a00640 1 -- 172.21.15.27:0/725947370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 msgr2=0x7f8a740719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79a00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 0x7f8a740719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.241+0000 7f8a79a00640 1 -- 172.21.15.27:0/725947370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a740a5eb0 con 0x7f8a740a61a0 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.242+0000 7f8a79a00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a740a61a0 0x7f8a741e1a30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8a6802b950 tx=0x7f8a680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.242+0000 7f8a71600640 1 -- 172.21.15.27:0/725947370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68040070 con 0x7f8a740a61a0 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.242+0000 7f8a71600640 1 -- 172.21.15.27:0/725947370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a68030d00 con 0x7f8a740a61a0 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.242+0000 7f8a71600640 1 -- 172.21.15.27:0/725947370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68035040 con 0x7f8a740a61a0 2024-03-28T18:58:20.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.242+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a74073280 con 0x7f8a740a61a0 2024-03-28T18:58:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.242+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a741eba10 con 0x7f8a740a61a0 2024-03-28T18:58:20.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.244+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a74067dd0 con 0x7f8a740a61a0 2024-03-28T18:58:20.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.246+0000 7f8a71600640 1 -- 172.21.15.27:0/725947370 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8a68038020 con 0x7f8a740a61a0 2024-03-28T18:58:20.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.247+0000 7f8a71600640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8a5406bbd0 0x7f8a5406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:20.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.248+0000 7f8a71600640 1 -- 172.21.15.27:0/725947370 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8a680ad140 con 0x7f8a740a61a0 2024-03-28T18:58:20.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.248+0000 7f8a79000640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8a5406bbd0 0x7f8a5406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:20.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.249+0000 7f8a79000640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8a5406bbd0 0x7f8a5406e090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8a740c5860 tx=0x7f8a60008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:20.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.251+0000 7f8a71600640 1 -- 172.21.15.27:0/725947370 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8a6807b200 con 0x7f8a740a61a0 2024-03-28T18:58:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.608+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a74069640 con 0x7f8a740a61a0 2024-03-28T18:58:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.609+0000 7f8a71600640 1 -- 172.21.15.27:0/725947370 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7f8a6807f2a0 con 0x7f8a740a61a0 2024-03-28T18:58:20.609 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:20.609 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:20.609 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:58:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8a5406bbd0 msgr2=0x7f8a5406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8a5406bbd0 0x7f8a5406e090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8a740c5860 tx=0x7f8a60008040 comp rx=0 tx=0).stop 2024-03-28T18:58:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a740a61a0 msgr2=0x7f8a741e1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a740a61a0 0x7f8a741e1a30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8a6802b950 tx=0x7f8a680046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 shutdown_connections 2024-03-28T18:58:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8a5406bbd0 0x7f8a5406e090 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a74150ae0 0x7f8a740719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a740c6ee0 0x7f8a741e1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 --2- 172.21.15.27:0/725947370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a740a61a0 0x7f8a741e1a30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.612+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 >> 172.21.15.27:0/725947370 conn(0x7f8a7406b0c0 msgr2=0x7f8a740c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.613+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 shutdown_connections 2024-03-28T18:58:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:20.613+0000 7f8a7ae00640 1 -- 172.21.15.27:0/725947370 wait complete. 2024-03-28T18:58:20.633 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-03-28T18:58:20.633 INFO:tasks.mds_thrash.fs.[a]:waiting for 72 secs before thrashing 2024-03-28T18:58:22.632 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir foo 2024-03-28T18:58:22.653 INFO:tasks.workunit.client.1.smithi155.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-03-28T18:58:25.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:58:25.212 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:58:25.260 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir foo/.snap/barsnap 2024-03-28T18:58:25.285 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-03-28T18:58:25.317 INFO:tasks.workunit.client.1.smithi155.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-03-28T18:58:26.511 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir foo 2024-03-28T18:58:26.528 INFO:tasks.workunit.client.0.smithi027.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-03-28T18:58:28.317 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir foo/.snap/barsnap 2024-03-28T18:58:28.345 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-03-28T18:58:28.376 INFO:tasks.workunit.client.0.smithi027.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-03-28T18:58:29.594 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 'passed: files are identical' 2024-03-28T18:58:29.595 INFO:tasks.workunit.client.1.smithi155.stdout:passed: files are identical 2024-03-28T18:58:29.595 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir foo/.snap/barsnap 2024-03-28T18:58:29.651 INFO:tasks.workunit.client.1.smithi155.stdout:OK 2024-03-28T18:58:29.651 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo OK 2024-03-28T18:58:29.653 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T18:58:29.653 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-28T18:58:29.817 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-03-28T18:58:29.817 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-1.sh 2024-03-28T18:58:29.921 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 1 2024-03-28T18:58:29.984 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 2 2024-03-28T18:58:29.999 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 3 2024-03-28T18:58:30.011 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' -e file4 ']' 2024-03-28T18:58:30.013 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/snap1 2024-03-28T18:58:30.041 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 4 2024-03-28T18:58:30.051 INFO:tasks.workunit.client.1.smithi155.stderr:++ ls 2024-03-28T18:58:30.061 INFO:tasks.workunit.client.1.smithi155.stderr:+ now='file1 2024-03-28T18:58:30.061 INFO:tasks.workunit.client.1.smithi155.stderr:file2 2024-03-28T18:58:30.061 INFO:tasks.workunit.client.1.smithi155.stderr:file3 2024-03-28T18:58:30.061 INFO:tasks.workunit.client.1.smithi155.stderr:file4' 2024-03-28T18:58:30.063 INFO:tasks.workunit.client.1.smithi155.stderr:++ ls .snap/snap1 2024-03-28T18:58:30.087 INFO:tasks.workunit.client.1.smithi155.stderr:+ then='file1 2024-03-28T18:58:30.087 INFO:tasks.workunit.client.1.smithi155.stderr:file2 2024-03-28T18:58:30.087 INFO:tasks.workunit.client.1.smithi155.stderr:file3' 2024-03-28T18:58:30.087 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir .snap/snap1 2024-03-28T18:58:30.110 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 'file1 2024-03-28T18:58:30.110 INFO:tasks.workunit.client.1.smithi155.stderr:file2 2024-03-28T18:58:30.110 INFO:tasks.workunit.client.1.smithi155.stderr:file3 2024-03-28T18:58:30.110 INFO:tasks.workunit.client.1.smithi155.stderr:file4' = 'file1 2024-03-28T18:58:30.110 INFO:tasks.workunit.client.1.smithi155.stderr:file2 2024-03-28T18:58:30.110 INFO:tasks.workunit.client.1.smithi155.stderr:file3' ']' 2024-03-28T18:58:30.110 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 1 2024-03-28T18:58:30.122 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 2 2024-03-28T18:58:30.136 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 3 2024-03-28T18:58:30.148 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/snap1 2024-03-28T18:58:30.171 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 4 2024-03-28T18:58:30.192 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir .snap/snap1 2024-03-28T18:58:30.221 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm file1 file2 file3 file4 2024-03-28T18:58:30.270 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo OK 2024-03-28T18:58:30.270 INFO:tasks.workunit.client.1.smithi155.stdout:OK 2024-03-28T18:58:30.272 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T18:58:30.272 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-28T18:58:30.390 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-03-28T18:58:30.390 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-2.sh 2024-03-28T18:58:30.490 INFO:tasks.workunit.client.1.smithi155.stdout:Create dir 100 to 199 ... 2024-03-28T18:58:30.503 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 100 2024-03-28T18:58:30.641 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 101 2024-03-28T18:58:30.784 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 102 2024-03-28T18:58:30.928 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 103 2024-03-28T18:58:31.063 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 104 2024-03-28T18:58:31.195 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 105 2024-03-28T18:58:31.340 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 106 2024-03-28T18:58:31.497 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 107 2024-03-28T18:58:31.648 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 108 2024-03-28T18:58:31.778 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 109 2024-03-28T18:58:31.929 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 110 2024-03-28T18:58:32.064 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 111 2024-03-28T18:58:32.205 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 112 2024-03-28T18:58:32.209 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 'passed: files are identical' 2024-03-28T18:58:32.209 INFO:tasks.workunit.client.0.smithi027.stdout:passed: files are identical 2024-03-28T18:58:32.210 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir foo/.snap/barsnap 2024-03-28T18:58:32.256 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-03-28T18:58:32.256 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo OK 2024-03-28T18:58:32.257 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T18:58:32.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T18:58:32.325 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 113 2024-03-28T18:58:32.398 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-03-28T18:58:32.398 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-03-28T18:58:32.445 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 114 2024-03-28T18:58:32.480 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 1 2024-03-28T18:58:32.485 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 2 2024-03-28T18:58:32.495 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 3 2024-03-28T18:58:32.507 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' -e file4 ']' 2024-03-28T18:58:32.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/snap1 2024-03-28T18:58:32.530 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 4 2024-03-28T18:58:32.535 INFO:tasks.workunit.client.0.smithi027.stderr:++ ls 2024-03-28T18:58:32.542 INFO:tasks.workunit.client.0.smithi027.stderr:+ now='file1 2024-03-28T18:58:32.542 INFO:tasks.workunit.client.0.smithi027.stderr:file2 2024-03-28T18:58:32.542 INFO:tasks.workunit.client.0.smithi027.stderr:file3 2024-03-28T18:58:32.542 INFO:tasks.workunit.client.0.smithi027.stderr:file4' 2024-03-28T18:58:32.543 INFO:tasks.workunit.client.0.smithi027.stderr:++ ls .snap/snap1 2024-03-28T18:58:32.560 INFO:tasks.workunit.client.0.smithi027.stderr:+ then='file1 2024-03-28T18:58:32.560 INFO:tasks.workunit.client.0.smithi027.stderr:file2 2024-03-28T18:58:32.560 INFO:tasks.workunit.client.0.smithi027.stderr:file3' 2024-03-28T18:58:32.560 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir .snap/snap1 2024-03-28T18:58:32.571 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 115 2024-03-28T18:58:32.575 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 'file1 2024-03-28T18:58:32.575 INFO:tasks.workunit.client.0.smithi027.stderr:file2 2024-03-28T18:58:32.575 INFO:tasks.workunit.client.0.smithi027.stderr:file3 2024-03-28T18:58:32.575 INFO:tasks.workunit.client.0.smithi027.stderr:file4' = 'file1 2024-03-28T18:58:32.575 INFO:tasks.workunit.client.0.smithi027.stderr:file2 2024-03-28T18:58:32.575 INFO:tasks.workunit.client.0.smithi027.stderr:file3' ']' 2024-03-28T18:58:32.575 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 1 2024-03-28T18:58:32.587 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 2 2024-03-28T18:58:32.602 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 3 2024-03-28T18:58:32.618 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/snap1 2024-03-28T18:58:32.637 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 4 2024-03-28T18:58:32.653 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir .snap/snap1 2024-03-28T18:58:32.671 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm file1 file2 file3 file4 2024-03-28T18:58:32.686 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 116 2024-03-28T18:58:32.704 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-03-28T18:58:32.705 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo OK 2024-03-28T18:58:32.705 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T18:58:32.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T18:58:32.807 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-03-28T18:58:32.807 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-03-28T18:58:32.808 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 117 2024-03-28T18:58:32.882 INFO:tasks.workunit.client.0.smithi027.stdout:Create dir 100 to 199 ... 2024-03-28T18:58:32.894 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 100 2024-03-28T18:58:32.931 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 118 2024-03-28T18:58:32.995 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 101 2024-03-28T18:58:33.062 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 119 2024-03-28T18:58:33.110 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 102 2024-03-28T18:58:33.188 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 120 2024-03-28T18:58:33.209 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 103 2024-03-28T18:58:33.310 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 121 2024-03-28T18:58:33.312 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 104 2024-03-28T18:58:33.408 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 105 2024-03-28T18:58:33.432 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 122 2024-03-28T18:58:33.504 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 106 2024-03-28T18:58:33.565 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 123 2024-03-28T18:58:33.616 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 107 2024-03-28T18:58:33.684 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 124 2024-03-28T18:58:33.708 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 108 2024-03-28T18:58:33.797 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 125 2024-03-28T18:58:33.807 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 109 2024-03-28T18:58:33.905 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 110 2024-03-28T18:58:33.915 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 126 2024-03-28T18:58:33.995 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 111 2024-03-28T18:58:34.035 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 127 2024-03-28T18:58:34.091 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 112 2024-03-28T18:58:34.163 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 128 2024-03-28T18:58:34.184 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 113 2024-03-28T18:58:34.283 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 114 2024-03-28T18:58:34.284 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 129 2024-03-28T18:58:34.383 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 115 2024-03-28T18:58:34.402 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 130 2024-03-28T18:58:34.473 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 116 2024-03-28T18:58:34.519 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 131 2024-03-28T18:58:34.572 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 117 2024-03-28T18:58:34.642 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 132 2024-03-28T18:58:34.714 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 118 2024-03-28T18:58:34.766 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 133 2024-03-28T18:58:34.798 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 119 2024-03-28T18:58:34.861 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 134 2024-03-28T18:58:34.887 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 120 2024-03-28T18:58:34.968 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 135 2024-03-28T18:58:34.974 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 121 2024-03-28T18:58:35.070 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 122 2024-03-28T18:58:35.087 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 136 2024-03-28T18:58:35.154 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 123 2024-03-28T18:58:35.199 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 137 2024-03-28T18:58:35.242 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 124 2024-03-28T18:58:35.320 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 138 2024-03-28T18:58:35.335 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 125 2024-03-28T18:58:35.429 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 126 2024-03-28T18:58:35.440 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 139 2024-03-28T18:58:35.519 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 127 2024-03-28T18:58:35.555 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 140 2024-03-28T18:58:35.608 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 128 2024-03-28T18:58:35.695 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 141 2024-03-28T18:58:35.724 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 129 2024-03-28T18:58:35.788 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 142 2024-03-28T18:58:35.811 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 130 2024-03-28T18:58:35.880 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 143 2024-03-28T18:58:35.896 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 131 2024-03-28T18:58:35.986 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 132 2024-03-28T18:58:36.005 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 144 2024-03-28T18:58:36.080 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 133 2024-03-28T18:58:36.128 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 145 2024-03-28T18:58:36.165 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 134 2024-03-28T18:58:36.248 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 146 2024-03-28T18:58:36.255 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 135 2024-03-28T18:58:36.344 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 136 2024-03-28T18:58:36.359 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 147 2024-03-28T18:58:36.435 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 137 2024-03-28T18:58:36.473 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 148 2024-03-28T18:58:36.520 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 138 2024-03-28T18:58:36.598 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 149 2024-03-28T18:58:36.607 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 139 2024-03-28T18:58:36.700 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 140 2024-03-28T18:58:36.722 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 150 2024-03-28T18:58:36.786 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 141 2024-03-28T18:58:36.843 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 151 2024-03-28T18:58:36.872 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 142 2024-03-28T18:58:36.964 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 152 2024-03-28T18:58:36.972 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 143 2024-03-28T18:58:37.070 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 144 2024-03-28T18:58:37.124 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 153 2024-03-28T18:58:37.158 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 145 2024-03-28T18:58:37.238 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 154 2024-03-28T18:58:37.246 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 146 2024-03-28T18:58:37.342 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 147 2024-03-28T18:58:37.350 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 155 2024-03-28T18:58:37.432 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 148 2024-03-28T18:58:37.468 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 156 2024-03-28T18:58:37.520 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 149 2024-03-28T18:58:37.581 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 157 2024-03-28T18:58:37.606 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 150 2024-03-28T18:58:37.696 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 151 2024-03-28T18:58:37.696 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 158 2024-03-28T18:58:37.786 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 152 2024-03-28T18:58:37.814 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 159 2024-03-28T18:58:37.876 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 153 2024-03-28T18:58:37.933 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 160 2024-03-28T18:58:37.971 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 154 2024-03-28T18:58:38.068 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 155 2024-03-28T18:58:38.116 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 161 2024-03-28T18:58:38.162 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 156 2024-03-28T18:58:38.226 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 162 2024-03-28T18:58:38.249 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 157 2024-03-28T18:58:38.346 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 158 2024-03-28T18:58:38.347 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 163 2024-03-28T18:58:38.434 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 159 2024-03-28T18:58:38.468 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 164 2024-03-28T18:58:38.523 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 160 2024-03-28T18:58:38.576 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 165 2024-03-28T18:58:38.614 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 161 2024-03-28T18:58:38.692 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 166 2024-03-28T18:58:38.706 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 162 2024-03-28T18:58:38.801 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 163 2024-03-28T18:58:38.809 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 167 2024-03-28T18:58:38.894 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 164 2024-03-28T18:58:38.928 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 168 2024-03-28T18:58:38.983 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 165 2024-03-28T18:58:39.104 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 169 2024-03-28T18:58:39.173 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 166 2024-03-28T18:58:39.227 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 170 2024-03-28T18:58:39.261 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 167 2024-03-28T18:58:39.343 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 171 2024-03-28T18:58:39.351 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 168 2024-03-28T18:58:39.438 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 169 2024-03-28T18:58:39.464 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 172 2024-03-28T18:58:39.523 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 170 2024-03-28T18:58:39.581 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 173 2024-03-28T18:58:39.621 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 171 2024-03-28T18:58:39.702 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 174 2024-03-28T18:58:39.711 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 172 2024-03-28T18:58:39.803 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 173 2024-03-28T18:58:39.826 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 175 2024-03-28T18:58:39.888 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 174 2024-03-28T18:58:39.952 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 176 2024-03-28T18:58:39.974 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 175 2024-03-28T18:58:40.133 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 177 2024-03-28T18:58:40.142 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 176 2024-03-28T18:58:40.238 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 177 2024-03-28T18:58:40.255 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 178 2024-03-28T18:58:40.337 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 178 2024-03-28T18:58:40.374 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 179 2024-03-28T18:58:40.423 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 179 2024-03-28T18:58:40.488 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 180 2024-03-28T18:58:40.514 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 180 2024-03-28T18:58:40.604 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 181 2024-03-28T18:58:40.606 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 181 2024-03-28T18:58:40.700 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 182 2024-03-28T18:58:40.703 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 182 2024-03-28T18:58:40.789 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 183 2024-03-28T18:58:40.796 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 183 2024-03-28T18:58:40.877 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 184 2024-03-28T18:58:40.907 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 184 2024-03-28T18:58:40.970 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 185 2024-03-28T18:58:41.030 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 185 2024-03-28T18:58:41.171 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 186 2024-03-28T18:58:41.242 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 186 2024-03-28T18:58:41.265 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 187 2024-03-28T18:58:41.364 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 188 2024-03-28T18:58:41.366 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 187 2024-03-28T18:58:41.460 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 189 2024-03-28T18:58:41.492 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 188 2024-03-28T18:58:41.559 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 190 2024-03-28T18:58:41.607 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 189 2024-03-28T18:58:41.651 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 191 2024-03-28T18:58:41.717 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 190 2024-03-28T18:58:41.741 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 192 2024-03-28T18:58:41.834 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 191 2024-03-28T18:58:41.837 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 193 2024-03-28T18:58:41.940 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 194 2024-03-28T18:58:41.958 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 192 2024-03-28T18:58:42.035 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 195 2024-03-28T18:58:42.150 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 193 2024-03-28T18:58:42.236 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 196 2024-03-28T18:58:42.273 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 194 2024-03-28T18:58:42.325 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 197 2024-03-28T18:58:42.399 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 195 2024-03-28T18:58:42.417 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 198 2024-03-28T18:58:42.513 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 199 2024-03-28T18:58:42.518 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 196 2024-03-28T18:58:42.606 INFO:tasks.workunit.client.0.smithi027.stdout:Take first snapshot .snap/test1 2024-03-28T18:58:42.637 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 197 2024-03-28T18:58:42.698 INFO:tasks.workunit.client.0.smithi027.stdout:Create dir 200 to 299 ... 2024-03-28T18:58:42.701 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 200 2024-03-28T18:58:42.773 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 198 2024-03-28T18:58:42.898 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 199 2024-03-28T18:58:43.021 INFO:tasks.workunit.client.1.smithi155.stdout:Take first snapshot .snap/test1 2024-03-28T18:58:43.146 INFO:tasks.workunit.client.1.smithi155.stdout:Create dir 200 to 299 ... 2024-03-28T18:58:43.151 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 200 2024-03-28T18:58:43.247 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 201 2024-03-28T18:58:43.332 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 202 2024-03-28T18:58:43.414 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 203 2024-03-28T18:58:43.502 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 204 2024-03-28T18:58:43.588 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 205 2024-03-28T18:58:43.613 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 201 2024-03-28T18:58:43.669 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 206 2024-03-28T18:58:43.726 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 202 2024-03-28T18:58:43.749 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 207 2024-03-28T18:58:43.834 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 208 2024-03-28T18:58:43.842 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 203 2024-03-28T18:58:43.915 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 209 2024-03-28T18:58:43.962 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 204 2024-03-28T18:58:43.998 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 210 2024-03-28T18:58:44.077 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 205 2024-03-28T18:58:44.154 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 211 2024-03-28T18:58:44.254 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 212 2024-03-28T18:58:44.277 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 206 2024-03-28T18:58:44.333 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 213 2024-03-28T18:58:44.396 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 207 2024-03-28T18:58:44.411 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 214 2024-03-28T18:58:44.494 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 215 2024-03-28T18:58:44.513 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 208 2024-03-28T18:58:44.589 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 216 2024-03-28T18:58:44.625 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 209 2024-03-28T18:58:44.663 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 217 2024-03-28T18:58:44.719 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 210 2024-03-28T18:58:44.743 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 218 2024-03-28T18:58:44.828 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 219 2024-03-28T18:58:44.830 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 211 2024-03-28T18:58:44.912 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 220 2024-03-28T18:58:44.949 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 212 2024-03-28T18:58:44.993 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 221 2024-03-28T18:58:45.070 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 213 2024-03-28T18:58:45.076 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 222 2024-03-28T18:58:45.241 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 214 2024-03-28T18:58:45.255 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 223 2024-03-28T18:58:45.335 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 224 2024-03-28T18:58:45.359 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 215 2024-03-28T18:58:45.420 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 225 2024-03-28T18:58:45.477 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 216 2024-03-28T18:58:45.502 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 226 2024-03-28T18:58:45.582 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 227 2024-03-28T18:58:45.599 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 217 2024-03-28T18:58:45.661 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 228 2024-03-28T18:58:45.718 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 218 2024-03-28T18:58:45.745 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 229 2024-03-28T18:58:45.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:45.829 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 230 2024-03-28T18:58:45.830 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 219 2024-03-28T18:58:45.918 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 231 2024-03-28T18:58:45.948 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 220 2024-03-28T18:58:46.004 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 232 2024-03-28T18:58:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.063+0000 7f2378800640 1 Processor -- start 2024-03-28T18:58:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.064+0000 7f2378800640 1 -- start start 2024-03-28T18:58:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.065+0000 7f2378800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2370152e70 0x7f2370156c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.065+0000 7f2378800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2370156060 0x7f23700c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.065+0000 7f2378800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2370070510 con 0x7f23700c2d80 2024-03-28T18:58:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.065+0000 7f2378800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23700707f0 con 0x7f2370152e70 2024-03-28T18:58:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.065+0000 7f2378800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2370070ad0 con 0x7f2370156060 2024-03-28T18:58:46.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.065+0000 7f2375e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2370152e70 0x7f2370156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.065+0000 7f2375e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2370152e70 0x7f2370156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51258/0 (socket says 172.21.15.27:51258) 2024-03-28T18:58:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.065+0000 7f2375e00640 1 -- 172.21.15.27:0/1400189434 learned_addr learned my addr 172.21.15.27:0/1400189434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.066+0000 7f2376800640 1 --2- 172.21.15.27:0/1400189434 >> v2:172.21.15.155:3300/0 conn(0x7f2370156060 0x7f23700c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.066+0000 7f2375e00640 1 -- 172.21.15.27:0/1400189434 >> v2:172.21.15.155:3300/0 conn(0x7f2370156060 msgr2=0x7f23700c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.066+0000 7f2375e00640 1 --2- 172.21.15.27:0/1400189434 >> v2:172.21.15.155:3300/0 conn(0x7f2370156060 0x7f23700c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.066+0000 7f2375e00640 1 -- 172.21.15.27:0/1400189434 >> v1:172.21.15.155:6789/0 conn(0x7f23700c2d80 legacy=0x7f237006fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.066+0000 7f2375e00640 1 -- 172.21.15.27:0/1400189434 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2370155860 con 0x7f2370152e70 2024-03-28T18:58:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.068+0000 7f2375e00640 1 --2- 172.21.15.27:0/1400189434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2370152e70 0x7f2370156c40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f237006bf60 tx=0x7f236402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2083bd34cfe87ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.068+0000 7f2374a00640 1 -- 172.21.15.27:0/1400189434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364035020 con 0x7f2370152e70 2024-03-28T18:58:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.068+0000 7f2374a00640 1 -- 172.21.15.27:0/1400189434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364031800 con 0x7f2370152e70 2024-03-28T18:58:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.068+0000 7f2374a00640 1 -- 172.21.15.27:0/1400189434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364031c10 con 0x7f2370152e70 2024-03-28T18:58:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.069+0000 7f2378800640 1 -- 172.21.15.27:0/1400189434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2370152e70 msgr2=0x7f2370156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.069+0000 7f2378800640 1 --2- 172.21.15.27:0/1400189434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2370152e70 0x7f2370156c40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f237006bf60 tx=0x7f236402b8a0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.069+0000 7f2378800640 1 -- 172.21.15.27:0/1400189434 shutdown_connections 2024-03-28T18:58:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.069+0000 7f2378800640 1 --2- 172.21.15.27:0/1400189434 >> v2:172.21.15.155:3300/0 conn(0x7f2370156060 0x7f23700c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.069+0000 7f2378800640 1 --2- 172.21.15.27:0/1400189434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2370152e70 0x7f2370156c40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.069+0000 7f2378800640 1 -- 172.21.15.27:0/1400189434 >> 172.21.15.27:0/1400189434 conn(0x7f2370066dd0 msgr2=0x7f23700671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.069+0000 7f2378800640 1 -- 172.21.15.27:0/1400189434 shutdown_connections 2024-03-28T18:58:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.069+0000 7f2378800640 1 -- 172.21.15.27:0/1400189434 wait complete. 2024-03-28T18:58:46.071 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 221 2024-03-28T18:58:46.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.071+0000 7f2378800640 1 Processor -- start 2024-03-28T18:58:46.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.072+0000 7f2378800640 1 -- start start 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2378800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23700c2d80 0x7f2370150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2378800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2370152e70 0x7f23700abe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2378800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2370156060 0x7f23700ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2378800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23701533d0 con 0x7f23700c2d80 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2378800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2370153250 con 0x7f2370152e70 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2378800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23701564f0 con 0x7f2370156060 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2375e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23700c2d80 0x7f2370150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2375e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23700c2d80 0x7f2370150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51272/0 (socket says 172.21.15.27:51272) 2024-03-28T18:58:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.073+0000 7f2375e00640 1 -- 172.21.15.27:0/1837382698 learned_addr learned my addr 172.21.15.27:0/1837382698 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:46.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.074+0000 7f2375400640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2370152e70 0x7f23700abe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.074+0000 7f2376800640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2370156060 0x7f23700ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.074+0000 7f2375e00640 1 -- 172.21.15.27:0/1837382698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2370156060 msgr2=0x7f23700ac3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.074+0000 7f2375e00640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2370156060 0x7f23700ac3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.074+0000 7f2375e00640 1 -- 172.21.15.27:0/1837382698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2370152e70 msgr2=0x7f23700abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.074+0000 7f2375e00640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2370152e70 0x7f23700abe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.074+0000 7f2375e00640 1 -- 172.21.15.27:0/1837382698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237014d5b0 con 0x7f23700c2d80 2024-03-28T18:58:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.075+0000 7f2375e00640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23700c2d80 0x7f2370150f50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2364000c00 tx=0x7f23640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.076+0000 7f236a000640 1 -- 172.21.15.27:0/1837382698 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364040070 con 0x7f23700c2d80 2024-03-28T18:58:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.076+0000 7f236a000640 1 -- 172.21.15.27:0/1837382698 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364039cf0 con 0x7f23700c2d80 2024-03-28T18:58:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.076+0000 7f236a000640 1 -- 172.21.15.27:0/1837382698 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364035040 con 0x7f23700c2d80 2024-03-28T18:58:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.076+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23701ee580 con 0x7f23700c2d80 2024-03-28T18:58:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.076+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2370151490 con 0x7f23700c2d80 2024-03-28T18:58:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.077+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237004f400 con 0x7f23700c2d80 2024-03-28T18:58:46.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.080+0000 7f236a000640 1 -- 172.21.15.27:0/1837382698 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2364038020 con 0x7f23700c2d80 2024-03-28T18:58:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.082+0000 7f236a000640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f235406bc20 0x7f235406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.083+0000 7f236a000640 1 -- 172.21.15.27:0/1837382698 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6865+0+0 (secure 0 0 0) 0x7f2364079300 con 0x7f23700c2d80 2024-03-28T18:58:46.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.083+0000 7f2375400640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f235406bc20 0x7f235406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.084+0000 7f2375400640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f235406bc20 0x7f235406e0e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f23700c5af0 tx=0x7f2360009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.086+0000 7f236a000640 1 -- 172.21.15.27:0/1837382698 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f236407b960 con 0x7f23700c2d80 2024-03-28T18:58:46.095 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 233 2024-03-28T18:58:46.248 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 222 2024-03-28T18:58:46.275 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 234 2024-03-28T18:58:46.360 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 235 2024-03-28T18:58:46.373 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 223 2024-03-28T18:58:46.448 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 236 2024-03-28T18:58:46.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.453+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2370158450 con 0x7f23700c2d80 2024-03-28T18:58:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.454+0000 7f236a000640 1 -- 172.21.15.27:0/1837382698 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7f236407b960 con 0x7f23700c2d80 2024-03-28T18:58:46.455 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:46.455 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f235406bc20 msgr2=0x7f235406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f235406bc20 0x7f235406e0e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f23700c5af0 tx=0x7f2360009040 comp rx=0 tx=0).stop 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23700c2d80 msgr2=0x7f2370150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23700c2d80 0x7f2370150f50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2364000c00 tx=0x7f23640046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 shutdown_connections 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f235406bc20 0x7f235406e0e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2370156060 0x7f23700ac3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2370152e70 0x7f23700abe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 --2- 172.21.15.27:0/1837382698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23700c2d80 0x7f2370150f50 secure :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2364000c00 tx=0x7f23640046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 >> 172.21.15.27:0/1837382698 conn(0x7f2370066dd0 msgr2=0x7f2370154550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 shutdown_connections 2024-03-28T18:58:46.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.457+0000 7f2378800640 1 -- 172.21.15.27:0/1837382698 wait complete. 2024-03-28T18:58:46.478 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-03-28T18:58:46.478 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T18:58:46.478 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T18:58:46.491 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 224 2024-03-28T18:58:46.531 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 237 2024-03-28T18:58:46.539 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T18:58:46.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:46.609 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 238 2024-03-28T18:58:46.692 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 239 2024-03-28T18:58:46.781 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 240 2024-03-28T18:58:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.801+0000 7fb09e200640 1 Processor -- start 2024-03-28T18:58:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.802+0000 7fb09e200640 1 -- start start 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb09e200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb098152f80 0x7fb0980c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb09e200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0980c7970 0x7fb0980c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb09e200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0980c7310 con 0x7fb098155730 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb09e200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb098158d40 con 0x7fb0980c7970 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb09e200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb098159020 con 0x7fb098152f80 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb096a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0980c7970 0x7fb0980c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb096a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0980c7970 0x7fb0980c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42174/0 (socket says 172.21.15.27:42174) 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb096a00640 1 -- 172.21.15.27:0/3622763000 learned_addr learned my addr 172.21.15.27:0/3622763000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.803+0000 7fb097400640 1 --2- 172.21.15.27:0/3622763000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb098152f80 0x7fb0980c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.804+0000 7fb096a00640 1 -- 172.21.15.27:0/3622763000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb098152f80 msgr2=0x7fb0980c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.804+0000 7fb096a00640 1 --2- 172.21.15.27:0/3622763000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb098152f80 0x7fb0980c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.804+0000 7fb096a00640 1 -- 172.21.15.27:0/3622763000 >> v1:172.21.15.27:6789/0 conn(0x7fb098155730 legacy=0x7fb098155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.804+0000 7fb096a00640 1 -- 172.21.15.27:0/3622763000 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb098159300 con 0x7fb0980c7970 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.805+0000 7fb096a00640 1 --2- 172.21.15.27:0/3622763000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0980c7970 0x7fb0980c7df0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb084009870 tx=0x7fb08402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1853fab601b9b4c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.805+0000 7fb096000640 1 -- 172.21.15.27:0/3622763000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084035020 con 0x7fb0980c7970 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.805+0000 7fb096000640 1 -- 172.21.15.27:0/3622763000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb084031640 con 0x7fb0980c7970 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.805+0000 7fb096000640 1 -- 172.21.15.27:0/3622763000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08402d040 con 0x7fb0980c7970 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.806+0000 7fb09e200640 1 -- 172.21.15.27:0/3622763000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0980c7970 msgr2=0x7fb0980c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.806+0000 7fb09e200640 1 --2- 172.21.15.27:0/3622763000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0980c7970 0x7fb0980c7df0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb084009870 tx=0x7fb08402b8f0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.806+0000 7fb09e200640 1 -- 172.21.15.27:0/3622763000 shutdown_connections 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.806+0000 7fb09e200640 1 --2- 172.21.15.27:0/3622763000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0980c7970 0x7fb0980c7df0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.806+0000 7fb09e200640 1 --2- 172.21.15.27:0/3622763000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb098152f80 0x7fb0980c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.806+0000 7fb09e200640 1 -- 172.21.15.27:0/3622763000 >> 172.21.15.27:0/3622763000 conn(0x7fb09806b6f0 msgr2=0x7fb0980caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.806+0000 7fb09e200640 1 -- 172.21.15.27:0/3622763000 shutdown_connections 2024-03-28T18:58:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.806+0000 7fb09e200640 1 -- 172.21.15.27:0/3622763000 wait complete. 2024-03-28T18:58:46.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.808+0000 7fb09e200640 1 Processor -- start 2024-03-28T18:58:46.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.808+0000 7fb09e200640 1 -- start start 2024-03-28T18:58:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.809+0000 7fb09e200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0980c7970 0x7fb0981ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb09e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb098152f80 0x7fb0981485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb09e200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb098155730 0x7fb098148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb09e200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb098159b00 con 0x7fb098152f80 2024-03-28T18:58:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb09e200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0980c5e20 con 0x7fb098155730 2024-03-28T18:58:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb09e200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0980cbca0 con 0x7fb0980c7970 2024-03-28T18:58:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb096a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb098152f80 0x7fb0981485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb096a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb098152f80 0x7fb0981485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42186/0 (socket says 172.21.15.27:42186) 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb096a00640 1 -- 172.21.15.27:0/1731736364 learned_addr learned my addr 172.21.15.27:0/1731736364 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb097400640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0980c7970 0x7fb0981ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb097e00640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb098155730 0x7fb098148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb096a00640 1 -- 172.21.15.27:0/1731736364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0980c7970 msgr2=0x7fb0981ee5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb096a00640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0980c7970 0x7fb0981ee5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb096a00640 1 -- 172.21.15.27:0/1731736364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb098155730 msgr2=0x7fb098148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb096a00640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb098155730 0x7fb098148b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.810+0000 7fb096a00640 1 -- 172.21.15.27:0/1731736364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0980be970 con 0x7fb098152f80 2024-03-28T18:58:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.811+0000 7fb096a00640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb098152f80 0x7fb0981485e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb08402b930 tx=0x7fb0840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:46.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.811+0000 7fb08f400640 1 -- 172.21.15.27:0/1731736364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08403aa60 con 0x7fb098152f80 2024-03-28T18:58:46.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.811+0000 7fb08f400640 1 -- 172.21.15.27:0/1731736364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08403ac00 con 0x7fb098152f80 2024-03-28T18:58:46.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.811+0000 7fb08f400640 1 -- 172.21.15.27:0/1731736364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0840443f0 con 0x7fb098152f80 2024-03-28T18:58:46.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.811+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0980c44e0 con 0x7fb098152f80 2024-03-28T18:58:46.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.811+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0980c4780 con 0x7fb098152f80 2024-03-28T18:58:46.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.813+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09804f400 con 0x7fb098152f80 2024-03-28T18:58:46.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.814+0000 7fb08f400640 1 -- 172.21.15.27:0/1731736364 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb084038020 con 0x7fb098152f80 2024-03-28T18:58:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.815+0000 7fb08f400640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb07806bbd0 0x7fb07806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.816+0000 7fb08f400640 1 -- 172.21.15.27:0/1731736364 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6865+0+0 (secure 0 0 0) 0x7fb0840acb90 con 0x7fb098152f80 2024-03-28T18:58:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.816+0000 7fb097400640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb07806bbd0 0x7fb07806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.817+0000 7fb097400640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb07806bbd0 0x7fb07806e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb088006fd0 tx=0x7fb088008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:46.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:46.821+0000 7fb08f400640 1 -- 172.21.15.27:0/1731736364 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb08403f070 con 0x7fb098152f80 2024-03-28T18:58:46.857 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 241 2024-03-28T18:58:46.944 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 242 2024-03-28T18:58:47.043 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 243 2024-03-28T18:58:47.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.125+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0980cba80 con 0x7fb098152f80 2024-03-28T18:58:47.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.126+0000 7fb08f400640 1 -- 172.21.15.27:0/1731736364 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7fb08407b1e0 con 0x7fb098152f80 2024-03-28T18:58:47.127 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:47.127 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:47.127 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:58:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.129+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb07806bbd0 msgr2=0x7fb07806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.129+0000 7fb09e200640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb07806bbd0 0x7fb07806e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb088006fd0 tx=0x7fb088008040 comp rx=0 tx=0).stop 2024-03-28T18:58:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.129+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb098152f80 msgr2=0x7fb0981485e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.129+0000 7fb09e200640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb098152f80 0x7fb0981485e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb08402b930 tx=0x7fb0840046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.130+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 shutdown_connections 2024-03-28T18:58:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.130+0000 7fb09e200640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb07806bbd0 0x7fb07806e090 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.130+0000 7fb09e200640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb098155730 0x7fb098148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.130+0000 7fb09e200640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb098152f80 0x7fb0981485e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.130+0000 7fb09e200640 1 --2- 172.21.15.27:0/1731736364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0980c7970 0x7fb0981ee5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.130+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 >> 172.21.15.27:0/1731736364 conn(0x7fb09806b6f0 msgr2=0x7fb0980ca110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.130+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 shutdown_connections 2024-03-28T18:58:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:47.130+0000 7fb09e200640 1 -- 172.21.15.27:0/1731736364 wait complete. 2024-03-28T18:58:47.152 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T18:58:47.221 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 244 2024-03-28T18:58:47.326 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 245 2024-03-28T18:58:47.440 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 246 2024-03-28T18:58:47.555 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 247 2024-03-28T18:58:47.678 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 248 2024-03-28T18:58:47.802 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 249 2024-03-28T18:58:47.931 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 250 2024-03-28T18:58:49.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:49.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.366+0000 7efe19000640 1 Processor -- start 2024-03-28T18:58:49.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.366+0000 7efe19000640 1 -- start start 2024-03-28T18:58:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.367+0000 7efe19000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe14150ae0 0x7efe140a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.367+0000 7efe19000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 0x7efe140c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.367+0000 7efe19000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe14150ec0 con 0x7efe140c6ee0 2024-03-28T18:58:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.367+0000 7efe19000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1406dc70 con 0x7efe14150ae0 2024-03-28T18:58:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.367+0000 7efe19000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe14153550 con 0x7efe140a61a0 2024-03-28T18:58:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.368+0000 7efe13400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe14150ae0 0x7efe140a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.368+0000 7efe13400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe14150ae0 0x7efe140a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42192/0 (socket says 172.21.15.27:42192) 2024-03-28T18:58:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.368+0000 7efe13400640 1 -- 172.21.15.27:0/3887598854 learned_addr learned my addr 172.21.15.27:0/3887598854 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.368+0000 7efe0be00640 1 --2- 172.21.15.27:0/3887598854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 0x7efe140c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.368+0000 7efe13400640 1 -- 172.21.15.27:0/3887598854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 msgr2=0x7efe140c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.368+0000 7efe13400640 1 --2- 172.21.15.27:0/3887598854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 0x7efe140c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.368+0000 7efe13400640 1 -- 172.21.15.27:0/3887598854 >> v1:172.21.15.155:6789/0 conn(0x7efe140c6ee0 legacy=0x7efe14153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:49.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.368+0000 7efe13400640 1 -- 172.21.15.27:0/3887598854 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1406df10 con 0x7efe14150ae0 2024-03-28T18:58:49.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.370+0000 7efe13400640 1 --2- 172.21.15.27:0/3887598854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe14150ae0 0x7efe140a5c60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efe04009870 tx=0x7efe0402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb98321efbb010e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:49.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe12a00640 1 -- 172.21.15.27:0/3887598854 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04035020 con 0x7efe14150ae0 2024-03-28T18:58:49.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe12a00640 1 -- 172.21.15.27:0/3887598854 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe04031640 con 0x7efe14150ae0 2024-03-28T18:58:49.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe12a00640 1 -- 172.21.15.27:0/3887598854 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0402d040 con 0x7efe14150ae0 2024-03-28T18:58:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe19000640 1 -- 172.21.15.27:0/3887598854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe14150ae0 msgr2=0x7efe140a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe19000640 1 --2- 172.21.15.27:0/3887598854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe14150ae0 0x7efe140a5c60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efe04009870 tx=0x7efe0402b8f0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe19000640 1 -- 172.21.15.27:0/3887598854 shutdown_connections 2024-03-28T18:58:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe19000640 1 --2- 172.21.15.27:0/3887598854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 0x7efe140c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe19000640 1 --2- 172.21.15.27:0/3887598854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe14150ae0 0x7efe140a5c60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.371+0000 7efe19000640 1 -- 172.21.15.27:0/3887598854 >> 172.21.15.27:0/3887598854 conn(0x7efe1406b0c0 msgr2=0x7efe1406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.372+0000 7efe19000640 1 -- 172.21.15.27:0/3887598854 shutdown_connections 2024-03-28T18:58:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.372+0000 7efe19000640 1 -- 172.21.15.27:0/3887598854 wait complete. 2024-03-28T18:58:49.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.373+0000 7efe19000640 1 Processor -- start 2024-03-28T18:58:49.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.373+0000 7efe19000640 1 -- start start 2024-03-28T18:58:49.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe19000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 0x7efe140bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe19000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe140c6ee0 0x7efe14143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe19000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe14150ae0 0x7efe141443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe19000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1406e7b0 con 0x7efe140c6ee0 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe19000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1406e4d0 con 0x7efe14150ae0 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe19000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe141569d0 con 0x7efe140a61a0 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe0be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe140c6ee0 0x7efe14143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe0be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe140c6ee0 0x7efe14143e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42198/0 (socket says 172.21.15.27:42198) 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.374+0000 7efe0be00640 1 -- 172.21.15.27:0/2952566623 learned_addr learned my addr 172.21.15.27:0/2952566623 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.375+0000 7efe13400640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 0x7efe140bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.375+0000 7efe13e00640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe14150ae0 0x7efe141443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.375+0000 7efe0be00640 1 -- 172.21.15.27:0/2952566623 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 msgr2=0x7efe140bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.375+0000 7efe0be00640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 0x7efe140bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.375+0000 7efe0be00640 1 -- 172.21.15.27:0/2952566623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe14150ae0 msgr2=0x7efe141443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:49.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.375+0000 7efe0be00640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe14150ae0 0x7efe141443d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.375+0000 7efe0be00640 1 -- 172.21.15.27:0/2952566623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe14149c30 con 0x7efe140c6ee0 2024-03-28T18:58:49.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.375+0000 7efe0be00640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe140c6ee0 0x7efe14143e90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00d800 tx=0x7efdfc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:49.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.376+0000 7efe0b400640 1 -- 172.21.15.27:0/2952566623 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc00b6c0 con 0x7efe140c6ee0 2024-03-28T18:58:49.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.376+0000 7efe0b400640 1 -- 172.21.15.27:0/2952566623 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc019070 con 0x7efe140c6ee0 2024-03-28T18:58:49.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.376+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe140a5f40 con 0x7efe140c6ee0 2024-03-28T18:58:49.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.376+0000 7efe0b400640 1 -- 172.21.15.27:0/2952566623 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc00baa0 con 0x7efe140c6ee0 2024-03-28T18:58:49.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.376+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1414bc50 con 0x7efe140c6ee0 2024-03-28T18:58:49.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.377+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe140c1430 con 0x7efe140c6ee0 2024-03-28T18:58:49.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.378+0000 7efe0b400640 1 -- 172.21.15.27:0/2952566623 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efdfc00bc40 con 0x7efe140c6ee0 2024-03-28T18:58:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.379+0000 7efe0b400640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efdf006bbd0 0x7efdf006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.380+0000 7efe0b400640 1 -- 172.21.15.27:0/2952566623 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6865+0+0 (secure 0 0 0) 0x7efdfc08b490 con 0x7efe140c6ee0 2024-03-28T18:58:49.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.380+0000 7efe13400640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efdf006bbd0 0x7efdf006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:49.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.381+0000 7efe13400640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efdf006bbd0 0x7efdf006e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efe04036000 tx=0x7efe0403c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:49.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.384+0000 7efe0b400640 1 -- 172.21.15.27:0/2952566623 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efe140c1430 con 0x7efe140c6ee0 2024-03-28T18:58:49.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.729+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe1406da50 con 0x7efe140c6ee0 2024-03-28T18:58:49.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.731+0000 7efe0b400640 1 -- 172.21.15.27:0/2952566623 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7efdfc059a50 con 0x7efe140c6ee0 2024-03-28T18:58:49.731 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:49.731 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:49.732 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:58:49.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efdf006bbd0 msgr2=0x7efdf006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:49.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efdf006bbd0 0x7efdf006e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efe04036000 tx=0x7efe0403c040 comp rx=0 tx=0).stop 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe140c6ee0 msgr2=0x7efe14143e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe140c6ee0 0x7efe14143e90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00d800 tx=0x7efdfc00dcc0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 shutdown_connections 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efdf006bbd0 0x7efdf006e090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe14150ae0 0x7efe141443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe140c6ee0 0x7efe14143e90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 --2- 172.21.15.27:0/2952566623 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe140a61a0 0x7efe140bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 >> 172.21.15.27:0/2952566623 conn(0x7efe1406b0c0 msgr2=0x7efe140c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 shutdown_connections 2024-03-28T18:58:49.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:49.734+0000 7efe19000640 1 -- 172.21.15.27:0/2952566623 wait complete. 2024-03-28T18:58:49.754 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T18:58:51.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.008+0000 7f1751800640 1 Processor -- start 2024-03-28T18:58:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.008+0000 7f1751800640 1 -- start start 2024-03-28T18:58:52.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.010+0000 7f1751800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c156250 0x7f174c159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:52.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.010+0000 7f1751800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174c0c5710 con 0x7f174c152a70 2024-03-28T18:58:52.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.010+0000 7f1751800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174c0c59f0 con 0x7f174c0c6cc0 2024-03-28T18:58:52.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.010+0000 7f1751800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1556b0 con 0x7f174c156250 2024-03-28T18:58:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.010+0000 7f174be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f174c152a70 0x7f174c0cb0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48606/0 (socket says 172.21.15.27:48606) 2024-03-28T18:58:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.010+0000 7f174be00640 1 -- 172.21.15.27:0/1029705658 learned_addr learned my addr 172.21.15.27:0/1029705658 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:58:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.011+0000 7f1750e00640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c156250 0x7f174c159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.011+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792059002 0 0) 0x7f174c0c5710 con 0x7f174c152a70 2024-03-28T18:58:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.011+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1730003520 con 0x7f174c152a70 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.011+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559177478 0 0) 0x7f1730003520 con 0x7f174c152a70 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.011+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c156250 msgr2=0x7f174c159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.011+0000 7f174aa00640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c156250 0x7f174c159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.011+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 >> v1:172.21.15.155:6789/0 conn(0x7f174c0c6cc0 legacy=0x7f174c155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c0c5cd0 con 0x7f174c152a70 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f173c001670 con 0x7f174c152a70 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 >> v1:172.21.15.27:6789/0 conn(0x7f174c152a70 legacy=0x7f174c0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17300039e0 0x7f1730003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1730004800 0x7f1730004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c156250 0x7f1730005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1556b0 con 0x7f1730004800 2024-03-28T18:58:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730014300 con 0x7f17300039e0 2024-03-28T18:58:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1730014500 con 0x7f174c156250 2024-03-28T18:58:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f174b400640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1730004800 0x7f1730004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.012+0000 7f1750e00640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c156250 0x7f1730005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.013+0000 7f174be00640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17300039e0 0x7f1730003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.013+0000 7f174b400640 1 -- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c156250 msgr2=0x7f1730005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.013+0000 7f174b400640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c156250 0x7f1730005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.013+0000 7f174b400640 1 -- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17300039e0 msgr2=0x7f1730003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:52.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.013+0000 7f174b400640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17300039e0 0x7f1730003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.013+0000 7f174b400640 1 -- 172.21.15.27:0/1029705658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c0c5cd0 con 0x7f1730004800 2024-03-28T18:58:52.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.014+0000 7f174b400640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1730004800 0x7f1730004c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1738001720 tx=0x7f1730005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3a39e94a79b52d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:52.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.015+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738001370 con 0x7f1730004800 2024-03-28T18:58:52.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.015+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17380052d0 con 0x7f1730004800 2024-03-28T18:58:52.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.015+0000 7f174aa00640 1 -- 172.21.15.27:0/1029705658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738003830 con 0x7f1730004800 2024-03-28T18:58:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.015+0000 7f1751800640 1 -- 172.21.15.27:0/1029705658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1730004800 msgr2=0x7f1730004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.015+0000 7f1751800640 1 --2- 172.21.15.27:0/1029705658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1730004800 0x7f1730004c80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1738001720 tx=0x7f1730005830 comp rx=0 tx=0).stop 2024-03-28T18:58:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.016+0000 7f1750e00640 1 -- 172.21.15.27:0/1029705658 reap_dead start 2024-03-28T18:58:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.016+0000 7f1751800640 1 -- 172.21.15.27:0/1029705658 shutdown_connections 2024-03-28T18:58:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.016+0000 7f1751800640 1 -- 172.21.15.27:0/1029705658 >> 172.21.15.27:0/1029705658 conn(0x7f174c06b2a0 msgr2=0x7f174c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:52.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.016+0000 7f1751800640 1 -- 172.21.15.27:0/1029705658 shutdown_connections 2024-03-28T18:58:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.016+0000 7f1751800640 1 -- 172.21.15.27:0/1029705658 wait complete. 2024-03-28T18:58:52.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.018+0000 7f1751800640 1 Processor -- start 2024-03-28T18:58:52.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.018+0000 7f1751800640 1 -- start start 2024-03-28T18:58:52.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.019+0000 7f1751800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1730004800 0x7f174c147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:52.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.019+0000 7f1751800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c151900 0x7f174c150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:52.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.019+0000 7f1751800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174c153060 0x7f174c1494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:52.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.019+0000 7f1751800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0cb970 con 0x7f174c153060 2024-03-28T18:58:52.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.019+0000 7f1751800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0cb7f0 con 0x7f1730004800 2024-03-28T18:58:52.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.019+0000 7f1751800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c0c8dd0 con 0x7f174c151900 2024-03-28T18:58:52.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.020+0000 7f1750e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174c153060 0x7f174c1494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:52.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.020+0000 7f1750e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174c153060 0x7f174c1494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42228/0 (socket says 172.21.15.27:42228) 2024-03-28T18:58:52.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.020+0000 7f1750e00640 1 -- 172.21.15.27:0/4226093311 learned_addr learned my addr 172.21.15.27:0/4226093311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:52.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.020+0000 7f174b400640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c151900 0x7f174c150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:52.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.021+0000 7f174be00640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1730004800 0x7f174c147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:52.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.021+0000 7f1750e00640 1 -- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c151900 msgr2=0x7f174c150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:52.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.021+0000 7f1750e00640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c151900 0x7f174c150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.021+0000 7f1750e00640 1 -- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1730004800 msgr2=0x7f174c147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:52.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.021+0000 7f1750e00640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1730004800 0x7f174c147180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.021+0000 7f1750e00640 1 -- 172.21.15.27:0/4226093311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c151220 con 0x7f174c153060 2024-03-28T18:58:52.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.021+0000 7f174be00640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1730004800 0x7f174c147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:58:52.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.022+0000 7f1750e00640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174c153060 0x7f174c1494f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f174c06c880 tx=0x7f17440115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:52.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.022+0000 7f1743e00640 1 -- 172.21.15.27:0/4226093311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744003990 con 0x7f174c153060 2024-03-28T18:58:52.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.022+0000 7f1743e00640 1 -- 172.21.15.27:0/4226093311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174401b070 con 0x7f174c153060 2024-03-28T18:58:52.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.022+0000 7f1743e00640 1 -- 172.21.15.27:0/4226093311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744003b70 con 0x7f174c153060 2024-03-28T18:58:52.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.022+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174c151e80 con 0x7f174c153060 2024-03-28T18:58:52.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.022+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c1504e0 con 0x7f174c153060 2024-03-28T18:58:52.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.024+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174c069640 con 0x7f174c153060 2024-03-28T18:58:52.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.025+0000 7f1743e00640 1 -- 172.21.15.27:0/4226093311 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1744005d10 con 0x7f174c153060 2024-03-28T18:58:52.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.027+0000 7f1743e00640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172c06bbd0 0x7f172c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:52.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.027+0000 7f1743e00640 1 -- 172.21.15.27:0/4226093311 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6865+0+0 (secure 0 0 0) 0x7f174408b510 con 0x7f174c153060 2024-03-28T18:58:52.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.027+0000 7f174be00640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172c06bbd0 0x7f172c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:52.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.028+0000 7f174be00640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172c06bbd0 0x7f172c06e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f173c001050 tx=0x7f173c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:52.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.036+0000 7f1743e00640 1 -- 172.21.15.27:0/4226093311 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1744025020 con 0x7f174c153060 2024-03-28T18:58:52.121 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 251 2024-03-28T18:58:52.310 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 252 2024-03-28T18:58:52.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.347+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f174c072850 con 0x7f174c153060 2024-03-28T18:58:52.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.349+0000 7f1743e00640 1 -- 172.21.15.27:0/4226093311 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7f1744059b60 con 0x7f174c153060 2024-03-28T18:58:52.350 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:52.350 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:52.350 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:58:52.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172c06bbd0 msgr2=0x7f172c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172c06bbd0 0x7f172c06e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f173c001050 tx=0x7f173c009040 comp rx=0 tx=0).stop 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174c153060 msgr2=0x7f174c1494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174c153060 0x7f174c1494f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f174c06c880 tx=0x7f17440115a0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 shutdown_connections 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172c06bbd0 0x7f172c06e090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f174c153060 0x7f174c1494f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f174c151900 0x7f174c150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 --2- 172.21.15.27:0/4226093311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1730004800 0x7f174c147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 >> 172.21.15.27:0/4226093311 conn(0x7f174c06b2a0 msgr2=0x7f174c14c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 shutdown_connections 2024-03-28T18:58:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:52.353+0000 7f1751800640 1 -- 172.21.15.27:0/4226093311 wait complete. 2024-03-28T18:58:52.378 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T18:58:52.413 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 253 2024-03-28T18:58:52.546 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 254 2024-03-28T18:58:52.673 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 255 2024-03-28T18:58:52.808 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 256 2024-03-28T18:58:52.938 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 257 2024-03-28T18:58:53.065 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 258 2024-03-28T18:58:53.243 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 259 2024-03-28T18:58:53.369 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 260 2024-03-28T18:58:53.491 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 261 2024-03-28T18:58:53.618 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 262 2024-03-28T18:58:53.756 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 263 2024-03-28T18:58:53.883 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 264 2024-03-28T18:58:54.013 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 265 2024-03-28T18:58:54.151 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 266 2024-03-28T18:58:54.279 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 267 2024-03-28T18:58:54.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:54.400 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 268 2024-03-28T18:58:54.502 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 269 2024-03-28T18:58:54.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.607+0000 7f44c9200640 1 Processor -- start 2024-03-28T18:58:54.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.607+0000 7f44c9200640 1 -- start start 2024-03-28T18:58:54.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.608+0000 7f44c9200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44c40c8ed0 0x7f44c40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:54.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.608+0000 7f44c9200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c4158bd0 con 0x7f44c40c58c0 2024-03-28T18:58:54.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.608+0000 7f44c9200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c4158eb0 con 0x7f44c40be5d0 2024-03-28T18:58:54.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.608+0000 7f44c9200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c4159190 con 0x7f44c40c8ed0 2024-03-28T18:58:54.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.609+0000 7f44c2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44c40c8ed0 0x7f44c40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:54.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.609+0000 7f44c2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44c40c8ed0 0x7f44c40c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42232/0 (socket says 172.21.15.27:42232) 2024-03-28T18:58:54.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.609+0000 7f44c2a00640 1 -- 172.21.15.27:0/3753520810 learned_addr learned my addr 172.21.15.27:0/3753520810 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:54.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.609+0000 7f44c2a00640 1 -- 172.21.15.27:0/3753520810 >> v1:172.21.15.155:6789/0 conn(0x7f44c40be5d0 legacy=0x7f44c4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:54.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.609+0000 7f44c2a00640 1 -- 172.21.15.27:0/3753520810 >> v1:172.21.15.27:6789/0 conn(0x7f44c40c58c0 legacy=0x7f44c41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:54.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.609+0000 7f44c2a00640 1 -- 172.21.15.27:0/3753520810 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c4159470 con 0x7f44c40c8ed0 2024-03-28T18:58:54.610 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 270 2024-03-28T18:58:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.610+0000 7f44c2a00640 1 --2- 172.21.15.27:0/3753520810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40c8ed0 0x7f44c40c5ca0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009870 tx=0x7f44b002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e964b84c0174479 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.611+0000 7f44c2000640 1 -- 172.21.15.27:0/3753520810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0035020 con 0x7f44c40c8ed0 2024-03-28T18:58:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.611+0000 7f44c2000640 1 -- 172.21.15.27:0/3753520810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b002d040 con 0x7f44c40c8ed0 2024-03-28T18:58:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.611+0000 7f44c2000640 1 -- 172.21.15.27:0/3753520810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0031a80 con 0x7f44c40c8ed0 2024-03-28T18:58:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.611+0000 7f44c9200640 1 -- 172.21.15.27:0/3753520810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40c8ed0 msgr2=0x7f44c40c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.611+0000 7f44c9200640 1 --2- 172.21.15.27:0/3753520810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40c8ed0 0x7f44c40c5ca0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009870 tx=0x7f44b002b900 comp rx=0 tx=0).stop 2024-03-28T18:58:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.612+0000 7f44c9200640 1 -- 172.21.15.27:0/3753520810 shutdown_connections 2024-03-28T18:58:54.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.612+0000 7f44c9200640 1 --2- 172.21.15.27:0/3753520810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40c8ed0 0x7f44c40c5ca0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:54.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.612+0000 7f44c9200640 1 -- 172.21.15.27:0/3753520810 >> 172.21.15.27:0/3753520810 conn(0x7f44c406b6f0 msgr2=0x7f44c40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:54.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.612+0000 7f44c9200640 1 -- 172.21.15.27:0/3753520810 shutdown_connections 2024-03-28T18:58:54.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.612+0000 7f44c9200640 1 -- 172.21.15.27:0/3753520810 wait complete. 2024-03-28T18:58:54.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.614+0000 7f44c9200640 1 Processor -- start 2024-03-28T18:58:54.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.614+0000 7f44c9200640 1 -- start start 2024-03-28T18:58:54.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.615+0000 7f44c9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40be5d0 0x7f44c40aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:54.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.615+0000 7f44c9200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44c40c58c0 0x7f44c40c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.615+0000 7f44c9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44c40c8ed0 0x7f44c4156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.615+0000 7f44c9200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c40b09a0 con 0x7f44c40be5d0 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.615+0000 7f44c9200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c40b0820 con 0x7f44c40c8ed0 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.615+0000 7f44c9200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c40ad7d0 con 0x7f44c40c58c0 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.615+0000 7f44c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40be5d0 0x7f44c40aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40be5d0 0x7f44c40aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42244/0 (socket says 172.21.15.27:42244) 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c3400640 1 -- 172.21.15.27:0/3209632185 learned_addr learned my addr 172.21.15.27:0/3209632185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c2a00640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44c40c58c0 0x7f44c40c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c3e00640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44c40c8ed0 0x7f44c4156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c3400640 1 -- 172.21.15.27:0/3209632185 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44c40c58c0 msgr2=0x7f44c40c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c3400640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44c40c58c0 0x7f44c40c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c3400640 1 -- 172.21.15.27:0/3209632185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44c40c8ed0 msgr2=0x7f44c4156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:54.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c3400640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44c40c8ed0 0x7f44c4156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:54.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.616+0000 7f44c3400640 1 -- 172.21.15.27:0/3209632185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c41ef510 con 0x7f44c40be5d0 2024-03-28T18:58:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.617+0000 7f44c3400640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40be5d0 0x7f44c40aae00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f44b400c450 tx=0x7f44b400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.617+0000 7f44bb400640 1 -- 172.21.15.27:0/3209632185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b40052d0 con 0x7f44c40be5d0 2024-03-28T18:58:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.617+0000 7f44bb400640 1 -- 172.21.15.27:0/3209632185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b4005470 con 0x7f44c40be5d0 2024-03-28T18:58:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.617+0000 7f44bb400640 1 -- 172.21.15.27:0/3209632185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4012870 con 0x7f44c40be5d0 2024-03-28T18:58:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.617+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c40c2260 con 0x7f44c40be5d0 2024-03-28T18:58:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.617+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c40ca130 con 0x7f44c40be5d0 2024-03-28T18:58:54.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.618+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c40aaa40 con 0x7f44c40be5d0 2024-03-28T18:58:54.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.620+0000 7f44bb400640 1 -- 172.21.15.27:0/3209632185 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f44b4012a10 con 0x7f44c40be5d0 2024-03-28T18:58:54.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.621+0000 7f44bb400640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f44a406bbd0 0x7f44a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:54.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.621+0000 7f44bb400640 1 -- 172.21.15.27:0/3209632185 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6865+0+0 (secure 0 0 0) 0x7f44b408ab30 con 0x7f44c40be5d0 2024-03-28T18:58:54.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.622+0000 7f44c2a00640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f44a406bbd0 0x7f44a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:54.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.623+0000 7f44c2a00640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f44a406bbd0 0x7f44a406e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f44b0036000 tx=0x7f44b003c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:54.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.627+0000 7f44bb400640 1 -- 172.21.15.27:0/3209632185 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f44b4059180 con 0x7f44c40be5d0 2024-03-28T18:58:54.709 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 271 2024-03-28T18:58:54.831 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 272 2024-03-28T18:58:54.954 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 273 2024-03-28T18:58:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.982+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44c4156580 con 0x7f44c40be5d0 2024-03-28T18:58:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.983+0000 7f44bb400640 1 -- 172.21.15.27:0/3209632185 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7f44b4058a80 con 0x7f44c40be5d0 2024-03-28T18:58:54.983 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:54.983 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:58:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.985+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f44a406bbd0 msgr2=0x7f44a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.985+0000 7f44c9200640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f44a406bbd0 0x7f44a406e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f44b0036000 tx=0x7f44b003c040 comp rx=0 tx=0).stop 2024-03-28T18:58:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.985+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40be5d0 msgr2=0x7f44c40aae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.985+0000 7f44c9200640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40be5d0 0x7f44c40aae00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f44b400c450 tx=0x7f44b400c910 comp rx=0 tx=0).stop 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.985+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 shutdown_connections 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.985+0000 7f44c9200640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f44a406bbd0 0x7f44a406e090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.986+0000 7f44c9200640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44c40c8ed0 0x7f44c4156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.986+0000 7f44c9200640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44c40c58c0 0x7f44c40c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.986+0000 7f44c9200640 1 --2- 172.21.15.27:0/3209632185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44c40be5d0 0x7f44c40aae00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.986+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 >> 172.21.15.27:0/3209632185 conn(0x7f44c406b6f0 msgr2=0x7f44c40cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.986+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 shutdown_connections 2024-03-28T18:58:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:54.986+0000 7f44c9200640 1 -- 172.21.15.27:0/3209632185 wait complete. 2024-03-28T18:58:55.008 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T18:58:55.061 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 274 2024-03-28T18:58:55.199 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 275 2024-03-28T18:58:55.331 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 276 2024-03-28T18:58:55.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:58:55.351 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:58:55.421 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 277 2024-03-28T18:58:55.535 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 278 2024-03-28T18:58:55.672 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 279 2024-03-28T18:58:55.804 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 280 2024-03-28T18:58:55.941 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 281 2024-03-28T18:58:56.071 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 282 2024-03-28T18:58:56.217 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 283 2024-03-28T18:58:56.356 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 284 2024-03-28T18:58:56.494 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 285 2024-03-28T18:58:56.617 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 286 2024-03-28T18:58:56.739 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 287 2024-03-28T18:58:56.877 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 288 2024-03-28T18:58:57.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:57.055 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 289 2024-03-28T18:58:57.162 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 290 2024-03-28T18:58:57.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.244+0000 7fe67ca00640 1 Processor -- start 2024-03-28T18:58:57.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.244+0000 7fe67ca00640 1 -- start start 2024-03-28T18:58:57.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.246+0000 7fe67ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe678150ae0 0x7fe6780a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:57.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.246+0000 7fe67ca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6780a61a0 0x7fe6780c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:57.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.246+0000 7fe67ca00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678150ec0 con 0x7fe6780c6ee0 2024-03-28T18:58:57.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.247+0000 7fe67ca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67806dc70 con 0x7fe6780a61a0 2024-03-28T18:58:57.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.247+0000 7fe67ca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe678153550 con 0x7fe678150ae0 2024-03-28T18:58:57.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.247+0000 7fe676a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6780a61a0 0x7fe6780c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:57.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.247+0000 7fe677400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe678150ae0 0x7fe6780a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:57.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.247+0000 7fe676a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6780a61a0 0x7fe6780c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:47438/0 (socket says 172.21.15.27:47438) 2024-03-28T18:58:57.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.247+0000 7fe676a00640 1 -- 172.21.15.27:0/3552802371 learned_addr learned my addr 172.21.15.27:0/3552802371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:57.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.248+0000 7fe676a00640 1 -- 172.21.15.27:0/3552802371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe678150ae0 msgr2=0x7fe6780a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:57.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.248+0000 7fe676a00640 1 --2- 172.21.15.27:0/3552802371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe678150ae0 0x7fe6780a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.248+0000 7fe676a00640 1 -- 172.21.15.27:0/3552802371 >> v1:172.21.15.155:6789/0 conn(0x7fe6780c6ee0 legacy=0x7fe678153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:57.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.248+0000 7fe676a00640 1 -- 172.21.15.27:0/3552802371 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67806df10 con 0x7fe6780a61a0 2024-03-28T18:58:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.250+0000 7fe676a00640 1 --2- 172.21.15.27:0/3552802371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6780a61a0 0x7fe6780c69a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe664009870 tx=0x7fe66402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75a82c2d11398cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.250+0000 7fe676000640 1 -- 172.21.15.27:0/3552802371 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664035020 con 0x7fe6780a61a0 2024-03-28T18:58:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.250+0000 7fe676000640 1 -- 172.21.15.27:0/3552802371 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe664031640 con 0x7fe6780a61a0 2024-03-28T18:58:57.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.250+0000 7fe676000640 1 -- 172.21.15.27:0/3552802371 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66402d040 con 0x7fe6780a61a0 2024-03-28T18:58:57.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.251+0000 7fe67ca00640 1 -- 172.21.15.27:0/3552802371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6780a61a0 msgr2=0x7fe6780c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:57.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.251+0000 7fe67ca00640 1 --2- 172.21.15.27:0/3552802371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6780a61a0 0x7fe6780c69a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe664009870 tx=0x7fe66402b8f0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.251+0000 7fe67ca00640 1 -- 172.21.15.27:0/3552802371 shutdown_connections 2024-03-28T18:58:57.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.251+0000 7fe67ca00640 1 --2- 172.21.15.27:0/3552802371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6780a61a0 0x7fe6780c69a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.251+0000 7fe67ca00640 1 --2- 172.21.15.27:0/3552802371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe678150ae0 0x7fe6780a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.251+0000 7fe67ca00640 1 -- 172.21.15.27:0/3552802371 >> 172.21.15.27:0/3552802371 conn(0x7fe67806b0c0 msgr2=0x7fe67806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:57.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.251+0000 7fe67ca00640 1 -- 172.21.15.27:0/3552802371 shutdown_connections 2024-03-28T18:58:57.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.251+0000 7fe67ca00640 1 -- 172.21.15.27:0/3552802371 wait complete. 2024-03-28T18:58:57.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.253+0000 7fe67ca00640 1 Processor -- start 2024-03-28T18:58:57.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.253+0000 7fe67ca00640 1 -- start start 2024-03-28T18:58:57.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.254+0000 7fe67ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a61a0 0x7fe67814bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.254+0000 7fe67ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6780c6ee0 0x7fe6781ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.254+0000 7fe67ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe678150ae0 0x7fe67814ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.254+0000 7fe67ca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67806e7b0 con 0x7fe6780c6ee0 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.254+0000 7fe67ca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67806e4d0 con 0x7fe678150ae0 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.254+0000 7fe67ca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781569d0 con 0x7fe6780a61a0 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.254+0000 7fe676a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6780c6ee0 0x7fe6781ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe676a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6780c6ee0 0x7fe6781ec410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35572/0 (socket says 172.21.15.27:35572) 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe676a00640 1 -- 172.21.15.27:0/2078585948 learned_addr learned my addr 172.21.15.27:0/2078585948 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe677400640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a61a0 0x7fe67814bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe677e00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe678150ae0 0x7fe67814ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe676a00640 1 -- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a61a0 msgr2=0x7fe67814bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:57.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe676a00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a61a0 0x7fe67814bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe676a00640 1 -- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe678150ae0 msgr2=0x7fe67814ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:57.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe676a00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe678150ae0 0x7fe67814ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe676a00640 1 -- 172.21.15.27:0/2078585948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6781ed0b0 con 0x7fe6780c6ee0 2024-03-28T18:58:57.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.255+0000 7fe677e00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe678150ae0 0x7fe67814ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:58:57.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.256+0000 7fe676a00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6780c6ee0 0x7fe6781ec410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe664000c00 tx=0x7fe6640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:57.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.256+0000 7fe66f400640 1 -- 172.21.15.27:0/2078585948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664040070 con 0x7fe6780c6ee0 2024-03-28T18:58:57.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.256+0000 7fe66f400640 1 -- 172.21.15.27:0/2078585948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6640399f0 con 0x7fe6780c6ee0 2024-03-28T18:58:57.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.256+0000 7fe66f400640 1 -- 172.21.15.27:0/2078585948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664035040 con 0x7fe6780c6ee0 2024-03-28T18:58:57.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.256+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67814b800 con 0x7fe6780c6ee0 2024-03-28T18:58:57.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.256+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe678073510 con 0x7fe6780c6ee0 2024-03-28T18:58:57.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.258+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe678067dd0 con 0x7fe6780c6ee0 2024-03-28T18:58:57.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.259+0000 7fe66f400640 1 -- 172.21.15.27:0/2078585948 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe66404b060 con 0x7fe6780c6ee0 2024-03-28T18:58:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.260+0000 7fe66f400640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bc00 0x7fe65806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.260+0000 7fe66f400640 1 -- 172.21.15.27:0/2078585948 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6865+0+0 (secure 0 0 0) 0x7fe6640ac7d0 con 0x7fe6780c6ee0 2024-03-28T18:58:57.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.260+0000 7fe677400640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bc00 0x7fe65806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:57.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.261+0000 7fe677400640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bc00 0x7fe65806e0c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe678155880 tx=0x7fe668008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:57.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.266+0000 7fe66f400640 1 -- 172.21.15.27:0/2078585948 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe6781ed0b0 con 0x7fe6780c6ee0 2024-03-28T18:58:57.282 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 291 2024-03-28T18:58:57.389 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 292 2024-03-28T18:58:57.521 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 293 2024-03-28T18:58:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.581+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe678069640 con 0x7fe6780c6ee0 2024-03-28T18:58:57.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.582+0000 7fe66f400640 1 -- 172.21.15.27:0/2078585948 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7fe66407ae20 con 0x7fe6780c6ee0 2024-03-28T18:58:57.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:58:57.582 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:58:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:58:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.585+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bc00 msgr2=0x7fe65806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.585+0000 7fe67ca00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bc00 0x7fe65806e0c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe678155880 tx=0x7fe668008040 comp rx=0 tx=0).stop 2024-03-28T18:58:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6780c6ee0 msgr2=0x7fe6781ec410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6780c6ee0 0x7fe6781ec410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe664000c00 tx=0x7fe6640046c0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 shutdown_connections 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bc00 0x7fe65806e0c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe678150ae0 0x7fe67814ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6780c6ee0 0x7fe6781ec410 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 --2- 172.21.15.27:0/2078585948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a61a0 0x7fe67814bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 >> 172.21.15.27:0/2078585948 conn(0x7fe67806b0c0 msgr2=0x7fe67814ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 shutdown_connections 2024-03-28T18:58:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:57.586+0000 7fe67ca00640 1 -- 172.21.15.27:0/2078585948 wait complete. 2024-03-28T18:58:57.614 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T18:58:57.634 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 294 2024-03-28T18:58:57.761 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 295 2024-03-28T18:58:57.887 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 296 2024-03-28T18:58:58.017 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 297 2024-03-28T18:58:58.151 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 298 2024-03-28T18:58:58.356 INFO:tasks.workunit.client.0.smithi027.stdout: create dir 299 2024-03-28T18:58:58.495 INFO:tasks.workunit.client.0.smithi027.stdout:Create a snapshot in every first level dir ... 2024-03-28T18:58:58.534 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/.snap/snap-subdir-test 2024-03-28T18:58:58.581 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/30 file after the snapshot 2024-03-28T18:58:58.589 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/31 file after the snapshot 2024-03-28T18:58:58.601 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/32 file after the snapshot 2024-03-28T18:58:58.610 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/33 file after the snapshot 2024-03-28T18:58:58.624 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/34 file after the snapshot 2024-03-28T18:58:58.638 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/35 file after the snapshot 2024-03-28T18:58:58.651 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/36 file after the snapshot 2024-03-28T18:58:58.664 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/37 file after the snapshot 2024-03-28T18:58:58.678 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/38 file after the snapshot 2024-03-28T18:58:58.692 INFO:tasks.workunit.client.0.smithi027.stdout: create 100/39 file after the snapshot 2024-03-28T18:58:58.703 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/.snap/snap-subdir-test 2024-03-28T18:58:58.748 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/30 file after the snapshot 2024-03-28T18:58:58.754 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/31 file after the snapshot 2024-03-28T18:58:58.764 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/32 file after the snapshot 2024-03-28T18:58:58.778 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/33 file after the snapshot 2024-03-28T18:58:58.792 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/34 file after the snapshot 2024-03-28T18:58:58.805 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/35 file after the snapshot 2024-03-28T18:58:58.820 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/36 file after the snapshot 2024-03-28T18:58:58.834 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/37 file after the snapshot 2024-03-28T18:58:58.847 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/38 file after the snapshot 2024-03-28T18:58:58.862 INFO:tasks.workunit.client.0.smithi027.stdout: create 101/39 file after the snapshot 2024-03-28T18:58:58.874 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/.snap/snap-subdir-test 2024-03-28T18:58:58.914 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/30 file after the snapshot 2024-03-28T18:58:58.920 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/31 file after the snapshot 2024-03-28T18:58:58.933 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/32 file after the snapshot 2024-03-28T18:58:58.945 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/33 file after the snapshot 2024-03-28T18:58:58.959 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/34 file after the snapshot 2024-03-28T18:58:58.972 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/35 file after the snapshot 2024-03-28T18:58:58.985 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/36 file after the snapshot 2024-03-28T18:58:58.999 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/37 file after the snapshot 2024-03-28T18:58:59.013 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/38 file after the snapshot 2024-03-28T18:58:59.025 INFO:tasks.workunit.client.0.smithi027.stdout: create 102/39 file after the snapshot 2024-03-28T18:58:59.038 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/.snap/snap-subdir-test 2024-03-28T18:58:59.091 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/30 file after the snapshot 2024-03-28T18:58:59.096 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/31 file after the snapshot 2024-03-28T18:58:59.108 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/32 file after the snapshot 2024-03-28T18:58:59.121 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/33 file after the snapshot 2024-03-28T18:58:59.136 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/34 file after the snapshot 2024-03-28T18:58:59.150 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/35 file after the snapshot 2024-03-28T18:58:59.163 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/36 file after the snapshot 2024-03-28T18:58:59.178 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/37 file after the snapshot 2024-03-28T18:58:59.193 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/38 file after the snapshot 2024-03-28T18:58:59.207 INFO:tasks.workunit.client.0.smithi027.stdout: create 103/39 file after the snapshot 2024-03-28T18:58:59.235 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/.snap/snap-subdir-test 2024-03-28T18:58:59.333 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/30 file after the snapshot 2024-03-28T18:58:59.337 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/31 file after the snapshot 2024-03-28T18:58:59.350 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/32 file after the snapshot 2024-03-28T18:58:59.361 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/33 file after the snapshot 2024-03-28T18:58:59.374 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/34 file after the snapshot 2024-03-28T18:58:59.386 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/35 file after the snapshot 2024-03-28T18:58:59.398 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/36 file after the snapshot 2024-03-28T18:58:59.411 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/37 file after the snapshot 2024-03-28T18:58:59.425 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/38 file after the snapshot 2024-03-28T18:58:59.438 INFO:tasks.workunit.client.0.smithi027.stdout: create 104/39 file after the snapshot 2024-03-28T18:58:59.451 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/.snap/snap-subdir-test 2024-03-28T18:58:59.503 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/30 file after the snapshot 2024-03-28T18:58:59.508 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/31 file after the snapshot 2024-03-28T18:58:59.519 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/32 file after the snapshot 2024-03-28T18:58:59.531 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/33 file after the snapshot 2024-03-28T18:58:59.544 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/34 file after the snapshot 2024-03-28T18:58:59.558 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/35 file after the snapshot 2024-03-28T18:58:59.571 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/36 file after the snapshot 2024-03-28T18:58:59.584 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/37 file after the snapshot 2024-03-28T18:58:59.598 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/38 file after the snapshot 2024-03-28T18:58:59.612 INFO:tasks.workunit.client.0.smithi027.stdout: create 105/39 file after the snapshot 2024-03-28T18:58:59.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:58:59.626 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/.snap/snap-subdir-test 2024-03-28T18:58:59.658 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/30 file after the snapshot 2024-03-28T18:58:59.663 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/31 file after the snapshot 2024-03-28T18:58:59.673 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/32 file after the snapshot 2024-03-28T18:58:59.685 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/33 file after the snapshot 2024-03-28T18:58:59.694 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/34 file after the snapshot 2024-03-28T18:58:59.703 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/35 file after the snapshot 2024-03-28T18:58:59.714 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/36 file after the snapshot 2024-03-28T18:58:59.724 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/37 file after the snapshot 2024-03-28T18:58:59.734 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/38 file after the snapshot 2024-03-28T18:58:59.744 INFO:tasks.workunit.client.0.smithi027.stdout: create 106/39 file after the snapshot 2024-03-28T18:58:59.757 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/.snap/snap-subdir-test 2024-03-28T18:58:59.793 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/30 file after the snapshot 2024-03-28T18:58:59.800 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/31 file after the snapshot 2024-03-28T18:58:59.811 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/32 file after the snapshot 2024-03-28T18:58:59.826 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/33 file after the snapshot 2024-03-28T18:58:59.840 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/34 file after the snapshot 2024-03-28T18:58:59.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.852+0000 7f8acfa00640 1 Processor -- start 2024-03-28T18:58:59.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.852+0000 7f8acfa00640 1 -- start start 2024-03-28T18:58:59.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8acfa00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8ac80a9cb0 0x7f8ac80c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:59.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8acfa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ac81534c0 0x7f8ac8153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:59.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8acfa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac80c3d90 con 0x7f8ac8150c70 2024-03-28T18:58:59.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8acfa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac80ab4e0 con 0x7f8ac81534c0 2024-03-28T18:58:59.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8acfa00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac80ac1f0 con 0x7f8ac80a9cb0 2024-03-28T18:58:59.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8acd000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8ac8150c70 0x7f8ac80c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54414/0 (socket says 172.21.15.27:54414) 2024-03-28T18:58:59.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8acd000640 1 -- 172.21.15.27:0/2513797779 learned_addr learned my addr 172.21.15.27:0/2513797779 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:58:59.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8acda00640 1 --2- 172.21.15.27:0/2513797779 >> v2:172.21.15.27:3300/0 conn(0x7f8ac81534c0 0x7f8ac8153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.854+0000 7f8ac7e00640 1 --2- 172.21.15.27:0/2513797779 >> v2:172.21.15.155:3300/0 conn(0x7f8ac80a9cb0 0x7f8ac80c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.855+0000 7f8acda00640 1 -- 172.21.15.27:0/2513797779 >> v2:172.21.15.155:3300/0 conn(0x7f8ac80a9cb0 msgr2=0x7f8ac80c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:58:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.855+0000 7f8acda00640 1 --2- 172.21.15.27:0/2513797779 >> v2:172.21.15.155:3300/0 conn(0x7f8ac80a9cb0 0x7f8ac80c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.855+0000 7f8acda00640 1 -- 172.21.15.27:0/2513797779 >> v1:172.21.15.27:6789/0 conn(0x7f8ac8150c70 legacy=0x7f8ac80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.855+0000 7f8acda00640 1 -- 172.21.15.27:0/2513797779 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac80ab710 con 0x7f8ac81534c0 2024-03-28T18:58:59.855 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/35 file after the snapshot 2024-03-28T18:58:59.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.856+0000 7f8acda00640 1 --2- 172.21.15.27:0/2513797779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac8153910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80c4180 tx=0x7f8ac002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9daa4abd377b008c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.856+0000 7f8ac7400640 1 -- 172.21.15.27:0/2513797779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0039070 con 0x7f8ac81534c0 2024-03-28T18:58:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.857+0000 7f8ac7400640 1 -- 172.21.15.27:0/2513797779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac002cbd0 con 0x7f8ac81534c0 2024-03-28T18:58:59.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.857+0000 7f8ac7400640 1 -- 172.21.15.27:0/2513797779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0034440 con 0x7f8ac81534c0 2024-03-28T18:58:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.858+0000 7f8acfa00640 1 -- 172.21.15.27:0/2513797779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 msgr2=0x7f8ac8153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.858+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2513797779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac8153910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80c4180 tx=0x7f8ac002be20 comp rx=0 tx=0).stop 2024-03-28T18:58:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.858+0000 7f8acfa00640 1 -- 172.21.15.27:0/2513797779 shutdown_connections 2024-03-28T18:58:59.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.858+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2513797779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac8153910 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.858+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2513797779 >> v2:172.21.15.155:3300/0 conn(0x7f8ac80a9cb0 0x7f8ac80c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.858+0000 7f8acfa00640 1 -- 172.21.15.27:0/2513797779 >> 172.21.15.27:0/2513797779 conn(0x7f8ac806b9d0 msgr2=0x7f8ac80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:58:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.858+0000 7f8acfa00640 1 -- 172.21.15.27:0/2513797779 shutdown_connections 2024-03-28T18:58:59.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.858+0000 7f8acfa00640 1 -- 172.21.15.27:0/2513797779 wait complete. 2024-03-28T18:58:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.860+0000 7f8acfa00640 1 Processor -- start 2024-03-28T18:58:59.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.860+0000 7f8acfa00640 1 -- start start 2024-03-28T18:58:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acfa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ac80a9cb0 0x7f8ac814a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acfa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ac8150c70 0x7f8ac81eaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acfa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac81eafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acfa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80c1810 con 0x7f8ac81534c0 2024-03-28T18:58:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acfa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80abcd0 con 0x7f8ac80a9cb0 2024-03-28T18:58:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acfa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80c6830 con 0x7f8ac8150c70 2024-03-28T18:58:59.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac81eafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac81eafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35596/0 (socket says 172.21.15.27:35596) 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acda00640 1 -- 172.21.15.27:0/2446344939 learned_addr learned my addr 172.21.15.27:0/2446344939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acd000640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ac80a9cb0 0x7f8ac814a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8ac7e00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ac8150c70 0x7f8ac81eaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acda00640 1 -- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ac8150c70 msgr2=0x7f8ac81eaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acda00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ac8150c70 0x7f8ac81eaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acda00640 1 -- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ac80a9cb0 msgr2=0x7f8ac814a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acda00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ac80a9cb0 0x7f8ac814a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acda00640 1 -- 172.21.15.27:0/2446344939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac80712f0 con 0x7f8ac81534c0 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.861+0000 7f8acd000640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ac80a9cb0 0x7f8ac814a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:58:59.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.862+0000 7f8acda00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac81eafe0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009eb0 tx=0x7f8ac0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:59.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.862+0000 7f8ac4c00640 1 -- 172.21.15.27:0/2446344939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0039050 con 0x7f8ac81534c0 2024-03-28T18:58:59.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.862+0000 7f8ac4c00640 1 -- 172.21.15.27:0/2446344939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac0045070 con 0x7f8ac81534c0 2024-03-28T18:58:59.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.862+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac8070a50 con 0x7f8ac81534c0 2024-03-28T18:58:59.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.862+0000 7f8ac4c00640 1 -- 172.21.15.27:0/2446344939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac003e830 con 0x7f8ac81534c0 2024-03-28T18:58:59.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.862+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac8071010 con 0x7f8ac81534c0 2024-03-28T18:58:59.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.864+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac804f400 con 0x7f8ac81534c0 2024-03-28T18:58:59.864 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/36 file after the snapshot 2024-03-28T18:58:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.867+0000 7f8ac4c00640 1 -- 172.21.15.27:0/2446344939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8ac003e9d0 con 0x7f8ac81534c0 2024-03-28T18:58:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.867+0000 7f8ac4c00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8aa806bbd0 0x7f8aa806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:58:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.868+0000 7f8ac4c00640 1 -- 172.21.15.27:0/2446344939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6865+0+0 (secure 0 0 0) 0x7f8ac00ae800 con 0x7f8ac81534c0 2024-03-28T18:58:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.868+0000 7f8acd000640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8aa806bbd0 0x7f8aa806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:58:59.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.869+0000 7f8acd000640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8aa806bbd0 0x7f8aa806e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8abc000f60 tx=0x7f8abc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:58:59.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:58:59.870+0000 7f8ac4c00640 1 -- 172.21.15.27:0/2446344939 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8ac007cf60 con 0x7f8ac81534c0 2024-03-28T18:58:59.872 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/37 file after the snapshot 2024-03-28T18:58:59.881 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/38 file after the snapshot 2024-03-28T18:58:59.890 INFO:tasks.workunit.client.0.smithi027.stdout: create 107/39 file after the snapshot 2024-03-28T18:58:59.899 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/.snap/snap-subdir-test 2024-03-28T18:58:59.943 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/30 file after the snapshot 2024-03-28T18:58:59.948 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/31 file after the snapshot 2024-03-28T18:58:59.958 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/32 file after the snapshot 2024-03-28T18:58:59.970 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/33 file after the snapshot 2024-03-28T18:58:59.983 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/34 file after the snapshot 2024-03-28T18:58:59.996 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/35 file after the snapshot 2024-03-28T18:59:00.009 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/36 file after the snapshot 2024-03-28T18:59:00.020 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/37 file after the snapshot 2024-03-28T18:59:00.032 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/38 file after the snapshot 2024-03-28T18:59:00.046 INFO:tasks.workunit.client.0.smithi027.stdout: create 108/39 file after the snapshot 2024-03-28T18:59:00.057 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/.snap/snap-subdir-test 2024-03-28T18:59:00.102 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/30 file after the snapshot 2024-03-28T18:59:00.108 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/31 file after the snapshot 2024-03-28T18:59:00.120 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/32 file after the snapshot 2024-03-28T18:59:00.132 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/33 file after the snapshot 2024-03-28T18:59:00.142 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/34 file after the snapshot 2024-03-28T18:59:00.154 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/35 file after the snapshot 2024-03-28T18:59:00.166 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/36 file after the snapshot 2024-03-28T18:59:00.179 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/37 file after the snapshot 2024-03-28T18:59:00.192 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/38 file after the snapshot 2024-03-28T18:59:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.205+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ac80c5f00 con 0x7f8ac81534c0 2024-03-28T18:59:00.205 INFO:tasks.workunit.client.0.smithi027.stdout: create 109/39 file after the snapshot 2024-03-28T18:59:00.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.207+0000 7f8ac4c00640 1 -- 172.21.15.27:0/2446344939 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7f8ac007c860 con 0x7f8ac81534c0 2024-03-28T18:59:00.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:00.206 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:59:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:59:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8aa806bbd0 msgr2=0x7f8aa806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8aa806bbd0 0x7f8aa806e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8abc000f60 tx=0x7f8abc009040 comp rx=0 tx=0).stop 2024-03-28T18:59:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 msgr2=0x7f8ac81eafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac81eafe0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009eb0 tx=0x7f8ac0004a40 comp rx=0 tx=0).stop 2024-03-28T18:59:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 shutdown_connections 2024-03-28T18:59:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8aa806bbd0 0x7f8aa806e090 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ac81534c0 0x7f8ac81eafe0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ac8150c70 0x7f8ac81eaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 --2- 172.21.15.27:0/2446344939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ac80a9cb0 0x7f8ac814a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 >> 172.21.15.27:0/2446344939 conn(0x7f8ac806b9d0 msgr2=0x7f8ac80c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 shutdown_connections 2024-03-28T18:59:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:00.210+0000 7f8acfa00640 1 -- 172.21.15.27:0/2446344939 wait complete. 2024-03-28T18:59:00.214 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/.snap/snap-subdir-test 2024-03-28T18:59:00.229 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T18:59:00.246 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/30 file after the snapshot 2024-03-28T18:59:00.292 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/31 file after the snapshot 2024-03-28T18:59:00.353 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/32 file after the snapshot 2024-03-28T18:59:00.365 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/33 file after the snapshot 2024-03-28T18:59:00.377 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/34 file after the snapshot 2024-03-28T18:59:00.390 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/35 file after the snapshot 2024-03-28T18:59:00.403 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/36 file after the snapshot 2024-03-28T18:59:00.415 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/37 file after the snapshot 2024-03-28T18:59:00.429 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/38 file after the snapshot 2024-03-28T18:59:00.444 INFO:tasks.workunit.client.0.smithi027.stdout: create 110/39 file after the snapshot 2024-03-28T18:59:00.457 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/.snap/snap-subdir-test 2024-03-28T18:59:00.495 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/30 file after the snapshot 2024-03-28T18:59:00.501 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/31 file after the snapshot 2024-03-28T18:59:00.515 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/32 file after the snapshot 2024-03-28T18:59:00.526 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/33 file after the snapshot 2024-03-28T18:59:00.540 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/34 file after the snapshot 2024-03-28T18:59:00.551 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/35 file after the snapshot 2024-03-28T18:59:00.565 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/36 file after the snapshot 2024-03-28T18:59:00.578 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/37 file after the snapshot 2024-03-28T18:59:00.593 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/38 file after the snapshot 2024-03-28T18:59:00.607 INFO:tasks.workunit.client.0.smithi027.stdout: create 111/39 file after the snapshot 2024-03-28T18:59:00.620 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/.snap/snap-subdir-test 2024-03-28T18:59:00.673 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/30 file after the snapshot 2024-03-28T18:59:00.679 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/31 file after the snapshot 2024-03-28T18:59:00.691 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/32 file after the snapshot 2024-03-28T18:59:00.704 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/33 file after the snapshot 2024-03-28T18:59:00.717 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/34 file after the snapshot 2024-03-28T18:59:00.730 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/35 file after the snapshot 2024-03-28T18:59:00.745 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/36 file after the snapshot 2024-03-28T18:59:00.760 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/37 file after the snapshot 2024-03-28T18:59:00.774 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/38 file after the snapshot 2024-03-28T18:59:00.788 INFO:tasks.workunit.client.0.smithi027.stdout: create 112/39 file after the snapshot 2024-03-28T18:59:00.802 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/.snap/snap-subdir-test 2024-03-28T18:59:00.851 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/30 file after the snapshot 2024-03-28T18:59:00.855 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/31 file after the snapshot 2024-03-28T18:59:00.868 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/32 file after the snapshot 2024-03-28T18:59:00.882 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/33 file after the snapshot 2024-03-28T18:59:00.897 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/34 file after the snapshot 2024-03-28T18:59:00.910 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/35 file after the snapshot 2024-03-28T18:59:00.923 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/36 file after the snapshot 2024-03-28T18:59:00.935 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/37 file after the snapshot 2024-03-28T18:59:00.950 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/38 file after the snapshot 2024-03-28T18:59:00.964 INFO:tasks.workunit.client.0.smithi027.stdout: create 113/39 file after the snapshot 2024-03-28T18:59:00.978 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/.snap/snap-subdir-test 2024-03-28T18:59:01.025 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/30 file after the snapshot 2024-03-28T18:59:01.030 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/31 file after the snapshot 2024-03-28T18:59:01.074 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/32 file after the snapshot 2024-03-28T18:59:01.075 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/33 file after the snapshot 2024-03-28T18:59:01.075 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/34 file after the snapshot 2024-03-28T18:59:01.075 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/35 file after the snapshot 2024-03-28T18:59:01.086 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/36 file after the snapshot 2024-03-28T18:59:01.099 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/37 file after the snapshot 2024-03-28T18:59:01.112 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/38 file after the snapshot 2024-03-28T18:59:01.130 INFO:tasks.workunit.client.0.smithi027.stdout: create 114/39 file after the snapshot 2024-03-28T18:59:01.138 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/.snap/snap-subdir-test 2024-03-28T18:59:01.186 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/30 file after the snapshot 2024-03-28T18:59:01.192 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/31 file after the snapshot 2024-03-28T18:59:01.205 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/32 file after the snapshot 2024-03-28T18:59:01.216 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/33 file after the snapshot 2024-03-28T18:59:01.231 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/34 file after the snapshot 2024-03-28T18:59:01.261 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/35 file after the snapshot 2024-03-28T18:59:01.322 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/36 file after the snapshot 2024-03-28T18:59:01.357 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/37 file after the snapshot 2024-03-28T18:59:01.369 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/38 file after the snapshot 2024-03-28T18:59:01.380 INFO:tasks.workunit.client.0.smithi027.stdout: create 115/39 file after the snapshot 2024-03-28T18:59:01.392 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/.snap/snap-subdir-test 2024-03-28T18:59:01.438 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/30 file after the snapshot 2024-03-28T18:59:01.444 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/31 file after the snapshot 2024-03-28T18:59:01.456 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/32 file after the snapshot 2024-03-28T18:59:01.468 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/33 file after the snapshot 2024-03-28T18:59:01.483 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/34 file after the snapshot 2024-03-28T18:59:01.496 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/35 file after the snapshot 2024-03-28T18:59:01.509 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/36 file after the snapshot 2024-03-28T18:59:01.521 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/37 file after the snapshot 2024-03-28T18:59:01.533 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/38 file after the snapshot 2024-03-28T18:59:01.547 INFO:tasks.workunit.client.0.smithi027.stdout: create 116/39 file after the snapshot 2024-03-28T18:59:01.559 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/.snap/snap-subdir-test 2024-03-28T18:59:01.615 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/30 file after the snapshot 2024-03-28T18:59:01.622 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/31 file after the snapshot 2024-03-28T18:59:01.635 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/32 file after the snapshot 2024-03-28T18:59:01.647 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/33 file after the snapshot 2024-03-28T18:59:01.660 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/34 file after the snapshot 2024-03-28T18:59:01.673 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/35 file after the snapshot 2024-03-28T18:59:01.687 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/36 file after the snapshot 2024-03-28T18:59:01.700 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/37 file after the snapshot 2024-03-28T18:59:01.711 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/38 file after the snapshot 2024-03-28T18:59:01.723 INFO:tasks.workunit.client.0.smithi027.stdout: create 117/39 file after the snapshot 2024-03-28T18:59:01.737 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/.snap/snap-subdir-test 2024-03-28T18:59:01.787 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/30 file after the snapshot 2024-03-28T18:59:01.794 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/31 file after the snapshot 2024-03-28T18:59:01.807 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/32 file after the snapshot 2024-03-28T18:59:01.818 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/33 file after the snapshot 2024-03-28T18:59:01.830 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/34 file after the snapshot 2024-03-28T18:59:01.845 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/35 file after the snapshot 2024-03-28T18:59:01.858 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/36 file after the snapshot 2024-03-28T18:59:01.870 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/37 file after the snapshot 2024-03-28T18:59:01.882 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/38 file after the snapshot 2024-03-28T18:59:01.896 INFO:tasks.workunit.client.0.smithi027.stdout: create 118/39 file after the snapshot 2024-03-28T18:59:01.910 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/.snap/snap-subdir-test 2024-03-28T18:59:01.972 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/30 file after the snapshot 2024-03-28T18:59:01.977 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/31 file after the snapshot 2024-03-28T18:59:01.987 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/32 file after the snapshot 2024-03-28T18:59:01.997 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/33 file after the snapshot 2024-03-28T18:59:02.012 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/34 file after the snapshot 2024-03-28T18:59:02.026 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/35 file after the snapshot 2024-03-28T18:59:02.040 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/36 file after the snapshot 2024-03-28T18:59:02.052 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/37 file after the snapshot 2024-03-28T18:59:02.065 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/38 file after the snapshot 2024-03-28T18:59:02.080 INFO:tasks.workunit.client.0.smithi027.stdout: create 119/39 file after the snapshot 2024-03-28T18:59:02.092 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/.snap/snap-subdir-test 2024-03-28T18:59:02.142 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/30 file after the snapshot 2024-03-28T18:59:02.149 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/31 file after the snapshot 2024-03-28T18:59:02.161 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/32 file after the snapshot 2024-03-28T18:59:02.174 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/33 file after the snapshot 2024-03-28T18:59:02.186 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/34 file after the snapshot 2024-03-28T18:59:02.199 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/35 file after the snapshot 2024-03-28T18:59:02.212 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/36 file after the snapshot 2024-03-28T18:59:02.226 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/37 file after the snapshot 2024-03-28T18:59:02.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:02.239 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/38 file after the snapshot 2024-03-28T18:59:02.252 INFO:tasks.workunit.client.0.smithi027.stdout: create 120/39 file after the snapshot 2024-03-28T18:59:02.264 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/.snap/snap-subdir-test 2024-03-28T18:59:02.411 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/30 file after the snapshot 2024-03-28T18:59:02.415 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/31 file after the snapshot 2024-03-28T18:59:02.425 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/32 file after the snapshot 2024-03-28T18:59:02.436 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/33 file after the snapshot 2024-03-28T18:59:02.447 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/34 file after the snapshot 2024-03-28T18:59:02.458 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/35 file after the snapshot 2024-03-28T18:59:02.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.468+0000 7fccbca00640 1 Processor -- start 2024-03-28T18:59:02.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.468+0000 7fccbca00640 1 -- start start 2024-03-28T18:59:02.468 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/36 file after the snapshot 2024-03-28T18:59:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.470+0000 7fccbca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb80731c0 0x7fccb8156e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.470+0000 7fccbca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fccb8071a00 0x7fccb8157350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.470+0000 7fccbca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb80c5c70 con 0x7fccb806c340 2024-03-28T18:59:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.470+0000 7fccbca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb80c63e0 con 0x7fccb8071a00 2024-03-28T18:59:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.470+0000 7fccbca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80c69a0 con 0x7fccb80731c0 2024-03-28T18:59:02.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.471+0000 7fccb5e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fccb806c340 0x7fccb8156180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54426/0 (socket says 172.21.15.27:54426) 2024-03-28T18:59:02.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.471+0000 7fccb5e00640 1 -- 172.21.15.27:0/2963747911 learned_addr learned my addr 172.21.15.27:0/2963747911 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:02.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.471+0000 7fccb6800640 1 --2- 172.21.15.27:0/2963747911 >> v2:172.21.15.155:3300/0 conn(0x7fccb8071a00 0x7fccb8157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:02.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.471+0000 7fccafe00640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb80731c0 0x7fccb8156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:02.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.471+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220616653 0 0) 0x7fccb80c5c70 con 0x7fccb806c340 2024-03-28T18:59:02.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.471+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc9c003520 con 0x7fccb806c340 2024-03-28T18:59:02.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3949410806 0 0) 0x7fcc9c003520 con 0x7fccb806c340 2024-03-28T18:59:02.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb80731c0 msgr2=0x7fccb8156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb80731c0 0x7fccb8156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 >> v2:172.21.15.155:3300/0 conn(0x7fccb8071a00 msgr2=0x7fccb8157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 --2- 172.21.15.27:0/2963747911 >> v2:172.21.15.155:3300/0 conn(0x7fccb8071a00 0x7fccb8157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb80c5f10 con 0x7fccb806c340 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fcca8001670 con 0x7fccb806c340 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 >> v1:172.21.15.27:6789/0 conn(0x7fccb806c340 legacy=0x7fccb8156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c0039b0 0x7fcc9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc9c004800 0x7fcc9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb80731c0 0x7fcc9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80c63e0 con 0x7fcc9c004800 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80c69a0 con 0x7fcc9c0039b0 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c00ba10 con 0x7fccb80731c0 2024-03-28T18:59:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.472+0000 7fccafe00640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc9c004800 0x7fcc9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.473+0000 7fccb6800640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb80731c0 0x7fcc9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.473+0000 7fccafe00640 1 -- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb80731c0 msgr2=0x7fcc9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.473+0000 7fccb5e00640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c0039b0 0x7fcc9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.473+0000 7fccafe00640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb80731c0 0x7fcc9c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.473+0000 7fccafe00640 1 -- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c0039b0 msgr2=0x7fcc9c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.473+0000 7fccafe00640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c0039b0 0x7fcc9c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.473+0000 7fccafe00640 1 -- 172.21.15.27:0/2963747911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c003520 con 0x7fcc9c004800 2024-03-28T18:59:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.473+0000 7fccb5e00640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c0039b0 0x7fcc9c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.474+0000 7fccafe00640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc9c004800 0x7fcc9c004c80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcca0009af0 tx=0x7fcc9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32bc69a514b8f33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.474+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0039070 con 0x7fcc9c004800 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca002c6b0 con 0x7fcc9c004800 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccb5400640 1 -- 172.21.15.27:0/2963747911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0009690 con 0x7fcc9c004800 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccbca00640 1 -- 172.21.15.27:0/2963747911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc9c004800 msgr2=0x7fcc9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccbca00640 1 --2- 172.21.15.27:0/2963747911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc9c004800 0x7fcc9c004c80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcca0009af0 tx=0x7fcc9c005830 comp rx=0 tx=0).stop 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccb6800640 1 -- 172.21.15.27:0/2963747911 reap_dead start 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccbca00640 1 -- 172.21.15.27:0/2963747911 shutdown_connections 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccbca00640 1 -- 172.21.15.27:0/2963747911 >> 172.21.15.27:0/2963747911 conn(0x7fccb80c4750 msgr2=0x7fccb80c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccbca00640 1 -- 172.21.15.27:0/2963747911 shutdown_connections 2024-03-28T18:59:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.475+0000 7fccbca00640 1 -- 172.21.15.27:0/2963747911 wait complete. 2024-03-28T18:59:02.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.477+0000 7fccbca00640 1 Processor -- start 2024-03-28T18:59:02.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.477+0000 7fccbca00640 1 -- start start 2024-03-28T18:59:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.478+0000 7fccbca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c004800 0x7fccb81e6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.478+0000 7fccbca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccb80acad0 0x7fccb814ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.478+0000 7fccbca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb814f490 0x7fccb8151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccbca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80c5700 con 0x7fccb80acad0 2024-03-28T18:59:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccbca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80c5580 con 0x7fcc9c004800 2024-03-28T18:59:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccbca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb8156890 con 0x7fccb814f490 2024-03-28T18:59:02.479 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/37 file after the snapshot 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccafe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccb80acad0 0x7fccb814ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccafe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccb80acad0 0x7fccb814ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35618/0 (socket says 172.21.15.27:35618) 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccafe00640 1 -- 172.21.15.27:0/3521783935 learned_addr learned my addr 172.21.15.27:0/3521783935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccb6800640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb814f490 0x7fccb8151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccafe00640 1 -- 172.21.15.27:0/3521783935 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb814f490 msgr2=0x7fccb8151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccafe00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb814f490 0x7fccb8151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccafe00640 1 -- 172.21.15.27:0/3521783935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c004800 msgr2=0x7fccb81e6b30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccafe00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c004800 0x7fccb81e6b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.479+0000 7fccafe00640 1 -- 172.21.15.27:0/3521783935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb8150cb0 con 0x7fccb80acad0 2024-03-28T18:59:02.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.480+0000 7fccafe00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccb80acad0 0x7fccb814ef50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcca0009c10 tx=0x7fcca0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.481+0000 7fccae000640 1 -- 172.21.15.27:0/3521783935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0039050 con 0x7fccb80acad0 2024-03-28T18:59:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.481+0000 7fccae000640 1 -- 172.21.15.27:0/3521783935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca003a040 con 0x7fccb80acad0 2024-03-28T18:59:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.481+0000 7fccae000640 1 -- 172.21.15.27:0/3521783935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca00044c0 con 0x7fccb80acad0 2024-03-28T18:59:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.481+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb814f920 con 0x7fccb80acad0 2024-03-28T18:59:02.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.481+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb81e7470 con 0x7fccb80acad0 2024-03-28T18:59:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.483+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb804f400 con 0x7fccb80acad0 2024-03-28T18:59:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.483+0000 7fccae000640 1 -- 172.21.15.27:0/3521783935 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcca0047020 con 0x7fccb80acad0 2024-03-28T18:59:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.484+0000 7fccae000640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcc9406bc50 0x7fcc9406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.485+0000 7fccae000640 1 -- 172.21.15.27:0/3521783935 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6865+0+0 (secure 0 0 0) 0x7fcca00ac560 con 0x7fccb80acad0 2024-03-28T18:59:02.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.485+0000 7fccb5e00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcc9406bc50 0x7fcc9406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:02.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.486+0000 7fccb5e00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcc9406bc50 0x7fcc9406e110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcca8001eb0 tx=0x7fcca8001170 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:02.489 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/38 file after the snapshot 2024-03-28T18:59:02.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.491+0000 7fccae000640 1 -- 172.21.15.27:0/3521783935 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcca004e020 con 0x7fccb80acad0 2024-03-28T18:59:02.499 INFO:tasks.workunit.client.0.smithi027.stdout: create 121/39 file after the snapshot 2024-03-28T18:59:02.509 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/.snap/snap-subdir-test 2024-03-28T18:59:02.538 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/30 file after the snapshot 2024-03-28T18:59:02.542 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/31 file after the snapshot 2024-03-28T18:59:02.551 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/32 file after the snapshot 2024-03-28T18:59:02.561 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/33 file after the snapshot 2024-03-28T18:59:02.571 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/34 file after the snapshot 2024-03-28T18:59:02.582 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/35 file after the snapshot 2024-03-28T18:59:02.593 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/36 file after the snapshot 2024-03-28T18:59:02.605 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/37 file after the snapshot 2024-03-28T18:59:02.616 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/38 file after the snapshot 2024-03-28T18:59:02.629 INFO:tasks.workunit.client.0.smithi027.stdout: create 122/39 file after the snapshot 2024-03-28T18:59:02.640 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/.snap/snap-subdir-test 2024-03-28T18:59:02.686 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/30 file after the snapshot 2024-03-28T18:59:02.691 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/31 file after the snapshot 2024-03-28T18:59:02.703 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/32 file after the snapshot 2024-03-28T18:59:02.714 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/33 file after the snapshot 2024-03-28T18:59:02.728 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/34 file after the snapshot 2024-03-28T18:59:02.740 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/35 file after the snapshot 2024-03-28T18:59:02.752 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/36 file after the snapshot 2024-03-28T18:59:02.765 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/37 file after the snapshot 2024-03-28T18:59:02.778 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/38 file after the snapshot 2024-03-28T18:59:02.789 INFO:tasks.workunit.client.0.smithi027.stdout: create 123/39 file after the snapshot 2024-03-28T18:59:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.791+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccb81535c0 con 0x7fccb80acad0 2024-03-28T18:59:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.792+0000 7fccae000640 1 -- 172.21.15.27:0/3521783935 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6258 (secure 0 0 0) 0x7fccb8150cb0 con 0x7fccb80acad0 2024-03-28T18:59:02.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:02.792 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4775,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:57:22.544784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6836/3119083312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3119083312},{"type":"v1","addr":"172.21.15.27:6836","nonce":3119083312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4260,"qdb_cluster":[4260]},"id":2}]} 2024-03-28T18:59:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-03-28T18:59:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcc9406bc50 msgr2=0x7fcc9406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcc9406bc50 0x7fcc9406e110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcca8001eb0 tx=0x7fcca8001170 comp rx=0 tx=0).stop 2024-03-28T18:59:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccb80acad0 msgr2=0x7fccb814ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccb80acad0 0x7fccb814ef50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcca0009c10 tx=0x7fcca0004730 comp rx=0 tx=0).stop 2024-03-28T18:59:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 shutdown_connections 2024-03-28T18:59:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcc9406bc50 0x7fcc9406e110 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fccb814f490 0x7fccb8151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccb80acad0 0x7fccb814ef50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 --2- 172.21.15.27:0/3521783935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc9c004800 0x7fccb81e6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.796+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 >> 172.21.15.27:0/3521783935 conn(0x7fccb80c4750 msgr2=0x7fccb80a8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.797+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 shutdown_connections 2024-03-28T18:59:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:02.797+0000 7fccbca00640 1 -- 172.21.15.27:0/3521783935 wait complete. 2024-03-28T18:59:02.800 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/.snap/snap-subdir-test 2024-03-28T18:59:02.817 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T18:59:02.833 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/30 file after the snapshot 2024-03-28T18:59:02.838 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/31 file after the snapshot 2024-03-28T18:59:02.848 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/32 file after the snapshot 2024-03-28T18:59:02.861 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/33 file after the snapshot 2024-03-28T18:59:02.872 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/34 file after the snapshot 2024-03-28T18:59:02.886 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/35 file after the snapshot 2024-03-28T18:59:02.900 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/36 file after the snapshot 2024-03-28T18:59:02.912 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/37 file after the snapshot 2024-03-28T18:59:02.924 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/38 file after the snapshot 2024-03-28T18:59:02.937 INFO:tasks.workunit.client.0.smithi027.stdout: create 124/39 file after the snapshot 2024-03-28T18:59:02.950 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/.snap/snap-subdir-test 2024-03-28T18:59:03.001 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/30 file after the snapshot 2024-03-28T18:59:03.010 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/31 file after the snapshot 2024-03-28T18:59:03.022 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/32 file after the snapshot 2024-03-28T18:59:03.035 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/33 file after the snapshot 2024-03-28T18:59:03.049 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/34 file after the snapshot 2024-03-28T18:59:03.063 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/35 file after the snapshot 2024-03-28T18:59:03.079 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/36 file after the snapshot 2024-03-28T18:59:03.090 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/37 file after the snapshot 2024-03-28T18:59:03.104 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/38 file after the snapshot 2024-03-28T18:59:03.117 INFO:tasks.workunit.client.0.smithi027.stdout: create 125/39 file after the snapshot 2024-03-28T18:59:03.130 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/.snap/snap-subdir-test 2024-03-28T18:59:03.179 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/30 file after the snapshot 2024-03-28T18:59:03.184 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/31 file after the snapshot 2024-03-28T18:59:03.197 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/32 file after the snapshot 2024-03-28T18:59:03.209 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/33 file after the snapshot 2024-03-28T18:59:03.222 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/34 file after the snapshot 2024-03-28T18:59:03.236 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/35 file after the snapshot 2024-03-28T18:59:03.250 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/36 file after the snapshot 2024-03-28T18:59:03.269 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/37 file after the snapshot 2024-03-28T18:59:03.337 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/38 file after the snapshot 2024-03-28T18:59:03.363 INFO:tasks.workunit.client.0.smithi027.stdout: create 126/39 file after the snapshot 2024-03-28T18:59:03.374 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/.snap/snap-subdir-test 2024-03-28T18:59:03.422 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/30 file after the snapshot 2024-03-28T18:59:03.428 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/31 file after the snapshot 2024-03-28T18:59:03.439 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/32 file after the snapshot 2024-03-28T18:59:03.452 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/33 file after the snapshot 2024-03-28T18:59:03.465 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/34 file after the snapshot 2024-03-28T18:59:03.479 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/35 file after the snapshot 2024-03-28T18:59:03.490 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/36 file after the snapshot 2024-03-28T18:59:03.504 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/37 file after the snapshot 2024-03-28T18:59:03.520 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/38 file after the snapshot 2024-03-28T18:59:03.534 INFO:tasks.workunit.client.0.smithi027.stdout: create 127/39 file after the snapshot 2024-03-28T18:59:03.548 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/.snap/snap-subdir-test 2024-03-28T18:59:03.594 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/30 file after the snapshot 2024-03-28T18:59:03.601 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/31 file after the snapshot 2024-03-28T18:59:03.613 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/32 file after the snapshot 2024-03-28T18:59:03.627 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/33 file after the snapshot 2024-03-28T18:59:03.641 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/34 file after the snapshot 2024-03-28T18:59:03.656 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/35 file after the snapshot 2024-03-28T18:59:03.670 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/36 file after the snapshot 2024-03-28T18:59:03.683 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/37 file after the snapshot 2024-03-28T18:59:03.696 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/38 file after the snapshot 2024-03-28T18:59:03.709 INFO:tasks.workunit.client.0.smithi027.stdout: create 128/39 file after the snapshot 2024-03-28T18:59:03.722 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/.snap/snap-subdir-test 2024-03-28T18:59:03.773 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/30 file after the snapshot 2024-03-28T18:59:03.779 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/31 file after the snapshot 2024-03-28T18:59:03.792 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/32 file after the snapshot 2024-03-28T18:59:03.806 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/33 file after the snapshot 2024-03-28T18:59:03.819 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/34 file after the snapshot 2024-03-28T18:59:03.831 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/35 file after the snapshot 2024-03-28T18:59:03.844 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/36 file after the snapshot 2024-03-28T18:59:03.857 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/37 file after the snapshot 2024-03-28T18:59:03.871 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/38 file after the snapshot 2024-03-28T18:59:03.884 INFO:tasks.workunit.client.0.smithi027.stdout: create 129/39 file after the snapshot 2024-03-28T18:59:03.898 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/.snap/snap-subdir-test 2024-03-28T18:59:03.946 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/30 file after the snapshot 2024-03-28T18:59:03.953 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/31 file after the snapshot 2024-03-28T18:59:03.967 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/32 file after the snapshot 2024-03-28T18:59:03.981 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/33 file after the snapshot 2024-03-28T18:59:03.996 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/34 file after the snapshot 2024-03-28T18:59:04.011 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/35 file after the snapshot 2024-03-28T18:59:04.024 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/36 file after the snapshot 2024-03-28T18:59:04.037 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/37 file after the snapshot 2024-03-28T18:59:04.050 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/38 file after the snapshot 2024-03-28T18:59:04.064 INFO:tasks.workunit.client.0.smithi027.stdout: create 130/39 file after the snapshot 2024-03-28T18:59:04.077 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/.snap/snap-subdir-test 2024-03-28T18:59:04.114 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/30 file after the snapshot 2024-03-28T18:59:04.119 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/31 file after the snapshot 2024-03-28T18:59:04.129 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/32 file after the snapshot 2024-03-28T18:59:04.140 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/33 file after the snapshot 2024-03-28T18:59:04.154 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/34 file after the snapshot 2024-03-28T18:59:04.168 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/35 file after the snapshot 2024-03-28T18:59:04.178 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/36 file after the snapshot 2024-03-28T18:59:04.190 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/37 file after the snapshot 2024-03-28T18:59:04.202 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/38 file after the snapshot 2024-03-28T18:59:04.216 INFO:tasks.workunit.client.0.smithi027.stdout: create 131/39 file after the snapshot 2024-03-28T18:59:04.231 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/.snap/snap-subdir-test 2024-03-28T18:59:04.281 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/30 file after the snapshot 2024-03-28T18:59:04.342 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/31 file after the snapshot 2024-03-28T18:59:04.368 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/32 file after the snapshot 2024-03-28T18:59:04.383 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/33 file after the snapshot 2024-03-28T18:59:04.395 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/34 file after the snapshot 2024-03-28T18:59:04.408 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/35 file after the snapshot 2024-03-28T18:59:04.419 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/36 file after the snapshot 2024-03-28T18:59:04.430 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/37 file after the snapshot 2024-03-28T18:59:04.442 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/38 file after the snapshot 2024-03-28T18:59:04.453 INFO:tasks.workunit.client.0.smithi027.stdout: create 132/39 file after the snapshot 2024-03-28T18:59:04.466 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/.snap/snap-subdir-test 2024-03-28T18:59:04.499 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/30 file after the snapshot 2024-03-28T18:59:04.504 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/31 file after the snapshot 2024-03-28T18:59:04.514 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/32 file after the snapshot 2024-03-28T18:59:04.526 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/33 file after the snapshot 2024-03-28T18:59:04.538 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/34 file after the snapshot 2024-03-28T18:59:04.553 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/35 file after the snapshot 2024-03-28T18:59:04.566 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/36 file after the snapshot 2024-03-28T18:59:04.581 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/37 file after the snapshot 2024-03-28T18:59:04.605 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/38 file after the snapshot 2024-03-28T18:59:04.612 INFO:tasks.workunit.client.0.smithi027.stdout: create 133/39 file after the snapshot 2024-03-28T18:59:04.619 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/.snap/snap-subdir-test 2024-03-28T18:59:04.648 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/30 file after the snapshot 2024-03-28T18:59:04.651 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/31 file after the snapshot 2024-03-28T18:59:04.660 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/32 file after the snapshot 2024-03-28T18:59:04.668 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/33 file after the snapshot 2024-03-28T18:59:04.675 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/34 file after the snapshot 2024-03-28T18:59:04.683 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/35 file after the snapshot 2024-03-28T18:59:04.693 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/36 file after the snapshot 2024-03-28T18:59:04.703 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/37 file after the snapshot 2024-03-28T18:59:04.712 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/38 file after the snapshot 2024-03-28T18:59:04.722 INFO:tasks.workunit.client.0.smithi027.stdout: create 134/39 file after the snapshot 2024-03-28T18:59:04.730 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/.snap/snap-subdir-test 2024-03-28T18:59:04.773 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/30 file after the snapshot 2024-03-28T18:59:04.777 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/31 file after the snapshot 2024-03-28T18:59:04.785 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/32 file after the snapshot 2024-03-28T18:59:04.795 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/33 file after the snapshot 2024-03-28T18:59:04.806 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/34 file after the snapshot 2024-03-28T18:59:04.816 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/35 file after the snapshot 2024-03-28T18:59:04.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:04.826 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/36 file after the snapshot 2024-03-28T18:59:04.837 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/37 file after the snapshot 2024-03-28T18:59:04.843 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/38 file after the snapshot 2024-03-28T18:59:04.851 INFO:tasks.workunit.client.0.smithi027.stdout: create 135/39 file after the snapshot 2024-03-28T18:59:04.858 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/.snap/snap-subdir-test 2024-03-28T18:59:04.882 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/30 file after the snapshot 2024-03-28T18:59:04.886 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/31 file after the snapshot 2024-03-28T18:59:04.894 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/32 file after the snapshot 2024-03-28T18:59:04.902 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/33 file after the snapshot 2024-03-28T18:59:04.909 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/34 file after the snapshot 2024-03-28T18:59:04.917 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/35 file after the snapshot 2024-03-28T18:59:04.925 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/36 file after the snapshot 2024-03-28T18:59:04.933 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/37 file after the snapshot 2024-03-28T18:59:04.941 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/38 file after the snapshot 2024-03-28T18:59:04.949 INFO:tasks.workunit.client.0.smithi027.stdout: create 136/39 file after the snapshot 2024-03-28T18:59:04.956 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/.snap/snap-subdir-test 2024-03-28T18:59:04.986 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/30 file after the snapshot 2024-03-28T18:59:04.990 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/31 file after the snapshot 2024-03-28T18:59:04.999 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/32 file after the snapshot 2024-03-28T18:59:05.007 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/33 file after the snapshot 2024-03-28T18:59:05.017 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/34 file after the snapshot 2024-03-28T18:59:05.026 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/35 file after the snapshot 2024-03-28T18:59:05.034 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/36 file after the snapshot 2024-03-28T18:59:05.043 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/37 file after the snapshot 2024-03-28T18:59:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.046+0000 7f3cc0c00640 1 Processor -- start 2024-03-28T18:59:05.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.047+0000 7f3cc0c00640 1 -- start start 2024-03-28T18:59:05.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.048+0000 7f3cc0c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3cbc152f80 0x7f3cbc0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.048+0000 7f3cc0c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cbc0c7970 0x7f3cbc0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.048+0000 7f3cc0c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc0c7310 con 0x7f3cbc155730 2024-03-28T18:59:05.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.048+0000 7f3cc0c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cbc158d40 con 0x7f3cbc0c7970 2024-03-28T18:59:05.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.048+0000 7f3cc0c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cbc159020 con 0x7f3cbc152f80 2024-03-28T18:59:05.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cbc0c7970 0x7f3cbc0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3cbc0c7970 0x7f3cbc0c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35632/0 (socket says 172.21.15.27:35632) 2024-03-28T18:59:05.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9400640 1 -- 172.21.15.27:0/393992891 learned_addr learned my addr 172.21.15.27:0/393992891 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9e00640 1 --2- 172.21.15.27:0/393992891 >> v2:172.21.15.155:3300/0 conn(0x7f3cbc152f80 0x7f3cbc0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9400640 1 -- 172.21.15.27:0/393992891 >> v2:172.21.15.155:3300/0 conn(0x7f3cbc152f80 msgr2=0x7f3cbc0c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9400640 1 --2- 172.21.15.27:0/393992891 >> v2:172.21.15.155:3300/0 conn(0x7f3cbc152f80 0x7f3cbc0c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9400640 1 -- 172.21.15.27:0/393992891 >> v1:172.21.15.155:6789/0 conn(0x7f3cbc155730 legacy=0x7f3cbc155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9400640 1 -- 172.21.15.27:0/393992891 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc159300 con 0x7f3cbc0c7970 2024-03-28T18:59:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.049+0000 7f3cb9e00640 1 --2- 172.21.15.27:0/393992891 >> v2:172.21.15.155:3300/0 conn(0x7f3cbc152f80 0x7f3cbc0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:59:05.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.050+0000 7f3cb9400640 1 --2- 172.21.15.27:0/393992891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc0c7970 0x7f3cbc0c7df0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc06c000 tx=0x7f3ca802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f69f12c2b39c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:05.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.051+0000 7f3cb8a00640 1 -- 172.21.15.27:0/393992891 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca802e070 con 0x7f3cbc0c7970 2024-03-28T18:59:05.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.051+0000 7f3cb8a00640 1 -- 172.21.15.27:0/393992891 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca8037070 con 0x7f3cbc0c7970 2024-03-28T18:59:05.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.051+0000 7f3cb8a00640 1 -- 172.21.15.27:0/393992891 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8031a00 con 0x7f3cbc0c7970 2024-03-28T18:59:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.052+0000 7f3cc0c00640 1 -- 172.21.15.27:0/393992891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc0c7970 msgr2=0x7f3cbc0c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.052+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/393992891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc0c7970 0x7f3cbc0c7df0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc06c000 tx=0x7f3ca802b880 comp rx=0 tx=0).stop 2024-03-28T18:59:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.052+0000 7f3cc0c00640 1 -- 172.21.15.27:0/393992891 shutdown_connections 2024-03-28T18:59:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.052+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/393992891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc0c7970 0x7f3cbc0c7df0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.052+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/393992891 >> v2:172.21.15.155:3300/0 conn(0x7f3cbc152f80 0x7f3cbc0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.052+0000 7f3cc0c00640 1 -- 172.21.15.27:0/393992891 >> 172.21.15.27:0/393992891 conn(0x7f3cbc06b6f0 msgr2=0x7f3cbc0caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:05.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.053+0000 7f3cc0c00640 1 -- 172.21.15.27:0/393992891 shutdown_connections 2024-03-28T18:59:05.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.053+0000 7f3cc0c00640 1 -- 172.21.15.27:0/393992891 wait complete. 2024-03-28T18:59:05.053 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/38 file after the snapshot 2024-03-28T18:59:05.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.055+0000 7f3cc0c00640 1 Processor -- start 2024-03-28T18:59:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.056+0000 7f3cc0c00640 1 -- start start 2024-03-28T18:59:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.057+0000 7f3cc0c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cbc0c7970 0x7f3cbc14fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.057+0000 7f3cc0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3cbc152f80 0x7f3cbc0bea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.057+0000 7f3cc0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc155730 0x7f3cbc14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.057+0000 7f3cc0c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc159b00 con 0x7f3cbc155730 2024-03-28T18:59:05.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.057+0000 7f3cc0c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc0c5e20 con 0x7f3cbc0c7970 2024-03-28T18:59:05.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.057+0000 7f3cc0c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cbc0cbca0 con 0x7f3cbc152f80 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cba800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc155730 0x7f3cbc14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cba800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc155730 0x7f3cbc14a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35644/0 (socket says 172.21.15.27:35644) 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cba800640 1 -- 172.21.15.27:0/171351257 learned_addr learned my addr 172.21.15.27:0/171351257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cb9400640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3cbc152f80 0x7f3cbc0bea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cb9e00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cbc0c7970 0x7f3cbc14fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cba800640 1 -- 172.21.15.27:0/171351257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3cbc152f80 msgr2=0x7f3cbc0bea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cba800640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3cbc152f80 0x7f3cbc0bea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cba800640 1 -- 172.21.15.27:0/171351257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cbc0c7970 msgr2=0x7f3cbc14fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cba800640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cbc0c7970 0x7f3cbc14fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.058+0000 7f3cba800640 1 -- 172.21.15.27:0/171351257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc0c49a0 con 0x7f3cbc155730 2024-03-28T18:59:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.059+0000 7f3cba800640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc155730 0x7f3cbc14a780 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400c7b0 tx=0x7f3cb400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.059+0000 7f3cb2000640 1 -- 172.21.15.27:0/171351257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb400ce60 con 0x7f3cbc155730 2024-03-28T18:59:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.059+0000 7f3cb2000640 1 -- 172.21.15.27:0/171351257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb4004b10 con 0x7f3cbc155730 2024-03-28T18:59:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.059+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc0c3860 con 0x7f3cbc155730 2024-03-28T18:59:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.059+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc14f710 con 0x7f3cbc155730 2024-03-28T18:59:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.059+0000 7f3cb2000640 1 -- 172.21.15.27:0/171351257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb400a850 con 0x7f3cbc155730 2024-03-28T18:59:05.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.061+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc04f400 con 0x7f3cbc155730 2024-03-28T18:59:05.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.062+0000 7f3cb2000640 1 -- 172.21.15.27:0/171351257 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3cb400a9f0 con 0x7f3cbc155730 2024-03-28T18:59:05.062 INFO:tasks.workunit.client.0.smithi027.stdout: create 137/39 file after the snapshot 2024-03-28T18:59:05.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.063+0000 7f3cb2000640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c9c06bbd0 0x7f3c9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.063+0000 7f3cb2000640 1 -- 172.21.15.27:0/171351257 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f3cb408b480 con 0x7f3cbc155730 2024-03-28T18:59:05.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.064+0000 7f3cb9e00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c9c06bbd0 0x7f3c9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.064+0000 7f3cb9e00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c9c06bbd0 0x7f3c9c06e090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00a950 tx=0x7f3cac008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.070+0000 7f3cb2000640 1 -- 172.21.15.27:0/171351257 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3cb4059a40 con 0x7f3cbc155730 2024-03-28T18:59:05.070 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/.snap/snap-subdir-test 2024-03-28T18:59:05.109 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/30 file after the snapshot 2024-03-28T18:59:05.114 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/31 file after the snapshot 2024-03-28T18:59:05.122 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/32 file after the snapshot 2024-03-28T18:59:05.131 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/33 file after the snapshot 2024-03-28T18:59:05.139 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/34 file after the snapshot 2024-03-28T18:59:05.147 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/35 file after the snapshot 2024-03-28T18:59:05.156 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/36 file after the snapshot 2024-03-28T18:59:05.165 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/37 file after the snapshot 2024-03-28T18:59:05.173 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/38 file after the snapshot 2024-03-28T18:59:05.181 INFO:tasks.workunit.client.0.smithi027.stdout: create 138/39 file after the snapshot 2024-03-28T18:59:05.191 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/.snap/snap-subdir-test 2024-03-28T18:59:05.242 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/30 file after the snapshot 2024-03-28T18:59:05.245 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/31 file after the snapshot 2024-03-28T18:59:05.254 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/32 file after the snapshot 2024-03-28T18:59:05.264 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/33 file after the snapshot 2024-03-28T18:59:05.274 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/34 file after the snapshot 2024-03-28T18:59:05.323 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/35 file after the snapshot 2024-03-28T18:59:05.341 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/36 file after the snapshot 2024-03-28T18:59:05.349 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/37 file after the snapshot 2024-03-28T18:59:05.357 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/38 file after the snapshot 2024-03-28T18:59:05.366 INFO:tasks.workunit.client.0.smithi027.stdout: create 139/39 file after the snapshot 2024-03-28T18:59:05.376 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/.snap/snap-subdir-test 2024-03-28T18:59:05.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.391+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cbc154ff0 con 0x7f3cbc155730 2024-03-28T18:59:05.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.392+0000 7f3cb2000640 1 -- 172.21.15.27:0/171351257 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5459 (secure 0 0 0) 0x7f3cb4059340 con 0x7f3cbc155730 2024-03-28T18:59:05.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:05.391 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:04.573697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T18:59:05.392 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-03-28T18:59:05.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c9c06bbd0 msgr2=0x7f3c9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c9c06bbd0 0x7f3c9c06e090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3cac00a950 tx=0x7f3cac008040 comp rx=0 tx=0).stop 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc155730 msgr2=0x7f3cbc14a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc155730 0x7f3cbc14a780 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400c7b0 tx=0x7f3cb400cc70 comp rx=0 tx=0).stop 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 shutdown_connections 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c9c06bbd0 0x7f3c9c06e090 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cbc155730 0x7f3cbc14a780 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3cbc152f80 0x7f3cbc0bea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 --2- 172.21.15.27:0/171351257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cbc0c7970 0x7f3cbc14fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.394+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 >> 172.21.15.27:0/171351257 conn(0x7f3cbc06b6f0 msgr2=0x7f3cbc155b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.395+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 shutdown_connections 2024-03-28T18:59:05.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.395+0000 7f3cc0c00640 1 -- 172.21.15.27:0/171351257 wait complete. 2024-03-28T18:59:05.411 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-03-28T18:59:05.411 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T18:59:05.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:05.413 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/30 file after the snapshot 2024-03-28T18:59:05.419 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/31 file after the snapshot 2024-03-28T18:59:05.426 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/32 file after the snapshot 2024-03-28T18:59:05.433 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/33 file after the snapshot 2024-03-28T18:59:05.441 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/34 file after the snapshot 2024-03-28T18:59:05.448 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/35 file after the snapshot 2024-03-28T18:59:05.456 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/36 file after the snapshot 2024-03-28T18:59:05.463 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/37 file after the snapshot 2024-03-28T18:59:05.471 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/38 file after the snapshot 2024-03-28T18:59:05.479 INFO:tasks.workunit.client.0.smithi027.stdout: create 140/39 file after the snapshot 2024-03-28T18:59:05.487 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/.snap/snap-subdir-test 2024-03-28T18:59:05.514 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/30 file after the snapshot 2024-03-28T18:59:05.518 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/31 file after the snapshot 2024-03-28T18:59:05.525 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/32 file after the snapshot 2024-03-28T18:59:05.534 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/33 file after the snapshot 2024-03-28T18:59:05.543 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/34 file after the snapshot 2024-03-28T18:59:05.552 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/35 file after the snapshot 2024-03-28T18:59:05.561 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/36 file after the snapshot 2024-03-28T18:59:05.569 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/37 file after the snapshot 2024-03-28T18:59:05.578 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/38 file after the snapshot 2024-03-28T18:59:05.588 INFO:tasks.workunit.client.0.smithi027.stdout: create 141/39 file after the snapshot 2024-03-28T18:59:05.596 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/.snap/snap-subdir-test 2024-03-28T18:59:05.635 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/30 file after the snapshot 2024-03-28T18:59:05.638 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/31 file after the snapshot 2024-03-28T18:59:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.641+0000 7f56c6400640 1 Processor -- start 2024-03-28T18:59:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.641+0000 7f56c6400640 1 -- start start 2024-03-28T18:59:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.643+0000 7f56c6400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c0153ed0 0x7f56c006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.643+0000 7f56c6400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f56c006eb80 0x7f56c015a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.643+0000 7f56c6400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c006efb0 con 0x7f56c006e6f0 2024-03-28T18:59:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.643+0000 7f56c6400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c00c6990 con 0x7f56c006eb80 2024-03-28T18:59:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.643+0000 7f56c6400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c00c3830 con 0x7f56c0153ed0 2024-03-28T18:59:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.643+0000 7f56bf400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c0153ed0 0x7f56c006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.643+0000 7f56bfe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f56c006eb80 0x7f56c015a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.643+0000 7f56bf400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c0153ed0 0x7f56c006ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:49380/0 (socket says 172.21.15.27:49380) 2024-03-28T18:59:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.644+0000 7f56bf400640 1 -- 172.21.15.27:0/4209373161 learned_addr learned my addr 172.21.15.27:0/4209373161 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.644+0000 7f56bf400640 1 -- 172.21.15.27:0/4209373161 >> v2:172.21.15.155:3300/0 conn(0x7f56c006eb80 msgr2=0x7f56c015a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.644+0000 7f56bf400640 1 --2- 172.21.15.27:0/4209373161 >> v2:172.21.15.155:3300/0 conn(0x7f56c006eb80 0x7f56c015a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.644+0000 7f56bf400640 1 -- 172.21.15.27:0/4209373161 >> v1:172.21.15.155:6789/0 conn(0x7f56c006e6f0 legacy=0x7f56c0159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.644+0000 7f56bf400640 1 -- 172.21.15.27:0/4209373161 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c006d1f0 con 0x7f56c0153ed0 2024-03-28T18:59:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.646+0000 7f56bf400640 1 --2- 172.21.15.27:0/4209373161 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c0153ed0 0x7f56c006ccb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009870 tx=0x7f56b002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7732dc3b7b9f425 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:05.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.646+0000 7f56be000640 1 -- 172.21.15.27:0/4209373161 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0034070 con 0x7f56c0153ed0 2024-03-28T18:59:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.646+0000 7f56be000640 1 -- 172.21.15.27:0/4209373161 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b0030840 con 0x7f56c0153ed0 2024-03-28T18:59:05.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.646+0000 7f56be000640 1 -- 172.21.15.27:0/4209373161 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0030c10 con 0x7f56c0153ed0 2024-03-28T18:59:05.646 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/32 file after the snapshot 2024-03-28T18:59:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.647+0000 7f56c6400640 1 -- 172.21.15.27:0/4209373161 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c0153ed0 msgr2=0x7f56c006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.647+0000 7f56c6400640 1 --2- 172.21.15.27:0/4209373161 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c0153ed0 0x7f56c006ccb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009870 tx=0x7f56b002b980 comp rx=0 tx=0).stop 2024-03-28T18:59:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.647+0000 7f56c6400640 1 -- 172.21.15.27:0/4209373161 shutdown_connections 2024-03-28T18:59:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.647+0000 7f56c6400640 1 --2- 172.21.15.27:0/4209373161 >> v2:172.21.15.155:3300/0 conn(0x7f56c006eb80 0x7f56c015a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.647+0000 7f56c6400640 1 --2- 172.21.15.27:0/4209373161 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c0153ed0 0x7f56c006ccb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.647+0000 7f56c6400640 1 -- 172.21.15.27:0/4209373161 >> 172.21.15.27:0/4209373161 conn(0x7f56c006b9d0 msgr2=0x7f56c00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.648+0000 7f56c6400640 1 -- 172.21.15.27:0/4209373161 shutdown_connections 2024-03-28T18:59:05.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.648+0000 7f56c6400640 1 -- 172.21.15.27:0/4209373161 wait complete. 2024-03-28T18:59:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.650+0000 7f56c6400640 1 Processor -- start 2024-03-28T18:59:05.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.650+0000 7f56c6400640 1 -- start start 2024-03-28T18:59:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56c6400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c006e6f0 0x7f56c01f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56c6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c006eb80 0x7f56c01597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56c6400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c0153ed0 0x7f56c0159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56c6400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c00c3db0 con 0x7f56c006eb80 2024-03-28T18:59:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56c6400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c00c3ad0 con 0x7f56c0153ed0 2024-03-28T18:59:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56c6400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c015a810 con 0x7f56c006e6f0 2024-03-28T18:59:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56bea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c006eb80 0x7f56c01597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56bea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c006eb80 0x7f56c01597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35656/0 (socket says 172.21.15.27:35656) 2024-03-28T18:59:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.651+0000 7f56bea00640 1 -- 172.21.15.27:0/1146990751 learned_addr learned my addr 172.21.15.27:0/1146990751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.652+0000 7f56bfe00640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c0153ed0 0x7f56c0159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.652+0000 7f56bf400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c006e6f0 0x7f56c01f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.652+0000 7f56bea00640 1 -- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c006e6f0 msgr2=0x7f56c01f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.652+0000 7f56bea00640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c006e6f0 0x7f56c01f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.654+0000 7f56bea00640 1 -- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c0153ed0 msgr2=0x7f56c0159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.654+0000 7f56bea00640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c0153ed0 0x7f56c0159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.654+0000 7f56bea00640 1 -- 172.21.15.27:0/1146990751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c0070af0 con 0x7f56c006eb80 2024-03-28T18:59:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.654+0000 7f56bfe00640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c0153ed0 0x7f56c0159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:59:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.654+0000 7f56bea00640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c006eb80 0x7f56c01597e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c3f0 tx=0x7f56ac00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:05.654 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/33 file after the snapshot 2024-03-28T18:59:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.655+0000 7f56b7400640 1 -- 172.21.15.27:0/1146990751 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0052d0 con 0x7f56c006eb80 2024-03-28T18:59:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.655+0000 7f56b7400640 1 -- 172.21.15.27:0/1146990751 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac005490 con 0x7f56c006eb80 2024-03-28T18:59:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.655+0000 7f56b7400640 1 -- 172.21.15.27:0/1146990751 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac007260 con 0x7f56c006eb80 2024-03-28T18:59:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.655+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c00a69c0 con 0x7f56c006eb80 2024-03-28T18:59:05.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.655+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c00727f0 con 0x7f56c006eb80 2024-03-28T18:59:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.657+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c004f400 con 0x7f56c006eb80 2024-03-28T18:59:05.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.658+0000 7f56b7400640 1 -- 172.21.15.27:0/1146990751 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f56ac007400 con 0x7f56c006eb80 2024-03-28T18:59:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.660+0000 7f56b7400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f56a006bc20 0x7f56a006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.660+0000 7f56b7400640 1 -- 172.21.15.27:0/1146990751 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f56ac08a1d0 con 0x7f56c006eb80 2024-03-28T18:59:05.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.660+0000 7f56bf400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f56a006bc20 0x7f56a006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:05.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.661+0000 7f56bf400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f56a006bc20 0x7f56a006e0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f56b0036000 tx=0x7f56b0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:05.662 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/34 file after the snapshot 2024-03-28T18:59:05.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.665+0000 7f56b7400640 1 -- 172.21.15.27:0/1146990751 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f56c006efb0 con 0x7f56c006eb80 2024-03-28T18:59:05.672 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/35 file after the snapshot 2024-03-28T18:59:05.680 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/36 file after the snapshot 2024-03-28T18:59:05.688 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/37 file after the snapshot 2024-03-28T18:59:05.695 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/38 file after the snapshot 2024-03-28T18:59:05.702 INFO:tasks.workunit.client.0.smithi027.stdout: create 142/39 file after the snapshot 2024-03-28T18:59:05.712 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/.snap/snap-subdir-test 2024-03-28T18:59:05.751 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/30 file after the snapshot 2024-03-28T18:59:05.755 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/31 file after the snapshot 2024-03-28T18:59:05.763 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/32 file after the snapshot 2024-03-28T18:59:05.771 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/33 file after the snapshot 2024-03-28T18:59:05.779 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/34 file after the snapshot 2024-03-28T18:59:05.789 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/35 file after the snapshot 2024-03-28T18:59:05.798 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/36 file after the snapshot 2024-03-28T18:59:05.808 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/37 file after the snapshot 2024-03-28T18:59:05.817 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/38 file after the snapshot 2024-03-28T18:59:05.826 INFO:tasks.workunit.client.0.smithi027.stdout: create 143/39 file after the snapshot 2024-03-28T18:59:05.836 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/.snap/snap-subdir-test 2024-03-28T18:59:05.874 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/30 file after the snapshot 2024-03-28T18:59:05.880 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/31 file after the snapshot 2024-03-28T18:59:05.893 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/32 file after the snapshot 2024-03-28T18:59:05.904 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/33 file after the snapshot 2024-03-28T18:59:05.913 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/34 file after the snapshot 2024-03-28T18:59:05.923 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/35 file after the snapshot 2024-03-28T18:59:05.933 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/36 file after the snapshot 2024-03-28T18:59:05.943 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/37 file after the snapshot 2024-03-28T18:59:05.953 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/38 file after the snapshot 2024-03-28T18:59:05.965 INFO:tasks.workunit.client.0.smithi027.stdout: create 144/39 file after the snapshot 2024-03-28T18:59:05.975 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/.snap/snap-subdir-test 2024-03-28T18:59:05.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.979+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56c00c41d0 con 0x7f56c006eb80 2024-03-28T18:59:05.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.979+0000 7f56b7400640 1 -- 172.21.15.27:0/1146990751 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5463 (secure 0 0 0) 0x7f56ac058340 con 0x7f56c006eb80 2024-03-28T18:59:05.979 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:05.979 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:04.833957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T18:59:05.979 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f56a006bc20 msgr2=0x7f56a006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f56a006bc20 0x7f56a006e0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f56b0036000 tx=0x7f56b0008030 comp rx=0 tx=0).stop 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c006eb80 msgr2=0x7f56c01597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c006eb80 0x7f56c01597e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c3f0 tx=0x7f56ac00c8b0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 shutdown_connections 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f56a006bc20 0x7f56a006e0e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c0153ed0 0x7f56c0159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c006eb80 0x7f56c01597e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 --2- 172.21.15.27:0/1146990751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c006e6f0 0x7f56c01f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:05.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.982+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 >> 172.21.15.27:0/1146990751 conn(0x7f56c006b9d0 msgr2=0x7f56c01e6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:05.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.983+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 shutdown_connections 2024-03-28T18:59:05.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:05.983+0000 7f56c6400640 1 -- 172.21.15.27:0/1146990751 wait complete. 2024-03-28T18:59:06.004 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5166": { "addr": "172.21.15.27:6837/3329030532", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 3329030532, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 3329030532, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5166, "incarnation": 21, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T18:58:12.558453+0000", "qdb_cluster": [ 5166 ], "qdb_leader": 5166, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5166 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4775": { "addr": "172.21.15.155:6834/3891854856", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3891854856, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3891854856, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4775, "incarnation": 26, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T18:59:04.833957+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4775 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T18:59:06.004 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/30 file after the snapshot 2024-03-28T18:59:06.007 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/31 file after the snapshot 2024-03-28T18:59:06.016 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/32 file after the snapshot 2024-03-28T18:59:06.025 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/33 file after the snapshot 2024-03-28T18:59:06.033 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/34 file after the snapshot 2024-03-28T18:59:06.045 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/35 file after the snapshot 2024-03-28T18:59:06.054 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/36 file after the snapshot 2024-03-28T18:59:06.068 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/37 file after the snapshot 2024-03-28T18:59:06.082 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/38 file after the snapshot 2024-03-28T18:59:06.092 INFO:tasks.workunit.client.0.smithi027.stdout: create 145/39 file after the snapshot 2024-03-28T18:59:06.104 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/.snap/snap-subdir-test 2024-03-28T18:59:06.149 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/30 file after the snapshot 2024-03-28T18:59:06.155 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/31 file after the snapshot 2024-03-28T18:59:06.164 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/32 file after the snapshot 2024-03-28T18:59:06.175 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/33 file after the snapshot 2024-03-28T18:59:06.187 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/34 file after the snapshot 2024-03-28T18:59:06.200 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/35 file after the snapshot 2024-03-28T18:59:06.212 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/36 file after the snapshot 2024-03-28T18:59:06.224 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/37 file after the snapshot 2024-03-28T18:59:06.236 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/38 file after the snapshot 2024-03-28T18:59:06.246 INFO:tasks.workunit.client.0.smithi027.stdout: create 146/39 file after the snapshot 2024-03-28T18:59:06.258 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/.snap/snap-subdir-test 2024-03-28T18:59:06.306 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/30 file after the snapshot 2024-03-28T18:59:06.356 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/31 file after the snapshot 2024-03-28T18:59:06.401 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/32 file after the snapshot 2024-03-28T18:59:06.414 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/33 file after the snapshot 2024-03-28T18:59:06.426 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/34 file after the snapshot 2024-03-28T18:59:06.438 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/35 file after the snapshot 2024-03-28T18:59:06.451 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/36 file after the snapshot 2024-03-28T18:59:06.463 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/37 file after the snapshot 2024-03-28T18:59:06.475 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/38 file after the snapshot 2024-03-28T18:59:06.488 INFO:tasks.workunit.client.0.smithi027.stdout: create 147/39 file after the snapshot 2024-03-28T18:59:06.500 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/.snap/snap-subdir-test 2024-03-28T18:59:06.538 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/30 file after the snapshot 2024-03-28T18:59:06.543 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/31 file after the snapshot 2024-03-28T18:59:06.554 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/32 file after the snapshot 2024-03-28T18:59:06.567 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/33 file after the snapshot 2024-03-28T18:59:06.579 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/34 file after the snapshot 2024-03-28T18:59:06.589 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/35 file after the snapshot 2024-03-28T18:59:06.602 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/36 file after the snapshot 2024-03-28T18:59:06.613 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/37 file after the snapshot 2024-03-28T18:59:06.626 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/38 file after the snapshot 2024-03-28T18:59:06.637 INFO:tasks.workunit.client.0.smithi027.stdout: create 148/39 file after the snapshot 2024-03-28T18:59:06.650 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/.snap/snap-subdir-test 2024-03-28T18:59:06.695 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/30 file after the snapshot 2024-03-28T18:59:06.700 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/31 file after the snapshot 2024-03-28T18:59:06.708 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/32 file after the snapshot 2024-03-28T18:59:06.719 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/33 file after the snapshot 2024-03-28T18:59:06.732 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/34 file after the snapshot 2024-03-28T18:59:06.744 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/35 file after the snapshot 2024-03-28T18:59:06.756 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/36 file after the snapshot 2024-03-28T18:59:06.768 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/37 file after the snapshot 2024-03-28T18:59:06.780 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/38 file after the snapshot 2024-03-28T18:59:06.792 INFO:tasks.workunit.client.0.smithi027.stdout: create 149/39 file after the snapshot 2024-03-28T18:59:06.803 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/.snap/snap-subdir-test 2024-03-28T18:59:06.844 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/30 file after the snapshot 2024-03-28T18:59:06.851 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/31 file after the snapshot 2024-03-28T18:59:06.863 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/32 file after the snapshot 2024-03-28T18:59:06.874 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/33 file after the snapshot 2024-03-28T18:59:06.887 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/34 file after the snapshot 2024-03-28T18:59:06.901 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/35 file after the snapshot 2024-03-28T18:59:06.914 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/36 file after the snapshot 2024-03-28T18:59:06.922 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/37 file after the snapshot 2024-03-28T18:59:06.935 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/38 file after the snapshot 2024-03-28T18:59:06.947 INFO:tasks.workunit.client.0.smithi027.stdout: create 150/39 file after the snapshot 2024-03-28T18:59:06.959 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/.snap/snap-subdir-test 2024-03-28T18:59:07.004 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/30 file after the snapshot 2024-03-28T18:59:07.008 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/31 file after the snapshot 2024-03-28T18:59:07.015 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/32 file after the snapshot 2024-03-28T18:59:07.025 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/33 file after the snapshot 2024-03-28T18:59:07.038 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/34 file after the snapshot 2024-03-28T18:59:07.052 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/35 file after the snapshot 2024-03-28T18:59:07.063 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/36 file after the snapshot 2024-03-28T18:59:07.074 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/37 file after the snapshot 2024-03-28T18:59:07.087 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/38 file after the snapshot 2024-03-28T18:59:07.096 INFO:tasks.workunit.client.0.smithi027.stdout: create 151/39 file after the snapshot 2024-03-28T18:59:07.108 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/.snap/snap-subdir-test 2024-03-28T18:59:07.153 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/30 file after the snapshot 2024-03-28T18:59:07.159 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/31 file after the snapshot 2024-03-28T18:59:07.170 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/32 file after the snapshot 2024-03-28T18:59:07.183 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/33 file after the snapshot 2024-03-28T18:59:07.196 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/34 file after the snapshot 2024-03-28T18:59:07.208 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/35 file after the snapshot 2024-03-28T18:59:07.220 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/36 file after the snapshot 2024-03-28T18:59:07.233 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/37 file after the snapshot 2024-03-28T18:59:07.245 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/38 file after the snapshot 2024-03-28T18:59:07.259 INFO:tasks.workunit.client.0.smithi027.stdout: create 152/39 file after the snapshot 2024-03-28T18:59:07.272 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/.snap/snap-subdir-test 2024-03-28T18:59:07.309 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/30 file after the snapshot 2024-03-28T18:59:07.355 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/31 file after the snapshot 2024-03-28T18:59:07.394 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/32 file after the snapshot 2024-03-28T18:59:07.406 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/33 file after the snapshot 2024-03-28T18:59:07.417 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/34 file after the snapshot 2024-03-28T18:59:07.428 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/35 file after the snapshot 2024-03-28T18:59:07.440 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/36 file after the snapshot 2024-03-28T18:59:07.453 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/37 file after the snapshot 2024-03-28T18:59:07.466 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/38 file after the snapshot 2024-03-28T18:59:07.479 INFO:tasks.workunit.client.0.smithi027.stdout: create 153/39 file after the snapshot 2024-03-28T18:59:07.491 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/.snap/snap-subdir-test 2024-03-28T18:59:07.520 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/30 file after the snapshot 2024-03-28T18:59:07.527 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/31 file after the snapshot 2024-03-28T18:59:07.537 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/32 file after the snapshot 2024-03-28T18:59:07.547 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/33 file after the snapshot 2024-03-28T18:59:07.558 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/34 file after the snapshot 2024-03-28T18:59:07.567 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/35 file after the snapshot 2024-03-28T18:59:07.576 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/36 file after the snapshot 2024-03-28T18:59:07.588 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/37 file after the snapshot 2024-03-28T18:59:07.601 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/38 file after the snapshot 2024-03-28T18:59:07.616 INFO:tasks.workunit.client.0.smithi027.stdout: create 154/39 file after the snapshot 2024-03-28T18:59:07.631 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/.snap/snap-subdir-test 2024-03-28T18:59:07.679 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/30 file after the snapshot 2024-03-28T18:59:07.684 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/31 file after the snapshot 2024-03-28T18:59:07.698 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/32 file after the snapshot 2024-03-28T18:59:07.705 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/33 file after the snapshot 2024-03-28T18:59:07.712 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/34 file after the snapshot 2024-03-28T18:59:07.719 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/35 file after the snapshot 2024-03-28T18:59:07.728 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/36 file after the snapshot 2024-03-28T18:59:07.735 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/37 file after the snapshot 2024-03-28T18:59:07.744 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/38 file after the snapshot 2024-03-28T18:59:07.751 INFO:tasks.workunit.client.0.smithi027.stdout: create 155/39 file after the snapshot 2024-03-28T18:59:07.758 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/.snap/snap-subdir-test 2024-03-28T18:59:07.791 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/30 file after the snapshot 2024-03-28T18:59:07.795 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/31 file after the snapshot 2024-03-28T18:59:07.804 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/32 file after the snapshot 2024-03-28T18:59:07.811 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/33 file after the snapshot 2024-03-28T18:59:07.819 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/34 file after the snapshot 2024-03-28T18:59:07.831 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/35 file after the snapshot 2024-03-28T18:59:07.838 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/36 file after the snapshot 2024-03-28T18:59:07.845 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/37 file after the snapshot 2024-03-28T18:59:07.852 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/38 file after the snapshot 2024-03-28T18:59:07.858 INFO:tasks.workunit.client.0.smithi027.stdout: create 156/39 file after the snapshot 2024-03-28T18:59:07.866 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/.snap/snap-subdir-test 2024-03-28T18:59:07.895 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/30 file after the snapshot 2024-03-28T18:59:07.899 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/31 file after the snapshot 2024-03-28T18:59:07.905 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/32 file after the snapshot 2024-03-28T18:59:07.912 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/33 file after the snapshot 2024-03-28T18:59:07.923 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/34 file after the snapshot 2024-03-28T18:59:07.931 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/35 file after the snapshot 2024-03-28T18:59:07.939 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/36 file after the snapshot 2024-03-28T18:59:07.946 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/37 file after the snapshot 2024-03-28T18:59:07.954 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/38 file after the snapshot 2024-03-28T18:59:07.965 INFO:tasks.workunit.client.0.smithi027.stdout: create 157/39 file after the snapshot 2024-03-28T18:59:07.972 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/.snap/snap-subdir-test 2024-03-28T18:59:08.003 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/30 file after the snapshot 2024-03-28T18:59:08.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:08.011 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/31 file after the snapshot 2024-03-28T18:59:08.014 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/32 file after the snapshot 2024-03-28T18:59:08.022 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/33 file after the snapshot 2024-03-28T18:59:08.029 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/34 file after the snapshot 2024-03-28T18:59:08.036 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/35 file after the snapshot 2024-03-28T18:59:08.042 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/36 file after the snapshot 2024-03-28T18:59:08.049 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/37 file after the snapshot 2024-03-28T18:59:08.054 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/38 file after the snapshot 2024-03-28T18:59:08.060 INFO:tasks.workunit.client.0.smithi027.stdout: create 158/39 file after the snapshot 2024-03-28T18:59:08.067 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/.snap/snap-subdir-test 2024-03-28T18:59:08.095 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/30 file after the snapshot 2024-03-28T18:59:08.098 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/31 file after the snapshot 2024-03-28T18:59:08.105 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/32 file after the snapshot 2024-03-28T18:59:08.111 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/33 file after the snapshot 2024-03-28T18:59:08.119 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/34 file after the snapshot 2024-03-28T18:59:08.127 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/35 file after the snapshot 2024-03-28T18:59:08.134 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/36 file after the snapshot 2024-03-28T18:59:08.142 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/37 file after the snapshot 2024-03-28T18:59:08.152 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/38 file after the snapshot 2024-03-28T18:59:08.161 INFO:tasks.workunit.client.0.smithi027.stdout: create 159/39 file after the snapshot 2024-03-28T18:59:08.171 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/.snap/snap-subdir-test 2024-03-28T18:59:08.200 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/30 file after the snapshot 2024-03-28T18:59:08.205 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/31 file after the snapshot 2024-03-28T18:59:08.214 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/32 file after the snapshot 2024-03-28T18:59:08.223 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/33 file after the snapshot 2024-03-28T18:59:08.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.225+0000 7f1209800640 1 Processor -- start 2024-03-28T18:59:08.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.225+0000 7f1209800640 1 -- start start 2024-03-28T18:59:08.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.226+0000 7f1209800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1204150c70 0x7f12040a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:08.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.226+0000 7f1209800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f12041534c0 0x7f1204153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:08.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.226+0000 7f1209800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040c3af0 con 0x7f12040c3690 2024-03-28T18:59:08.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.226+0000 7f1209800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12040ab4e0 con 0x7f12041534c0 2024-03-28T18:59:08.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.226+0000 7f1209800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040ac1f0 con 0x7f1204150c70 2024-03-28T18:59:08.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.227+0000 7f1203400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f12041534c0 0x7f1204153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:08.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.227+0000 7f1203400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f12041534c0 0x7f1204153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:40380/0 (socket says 172.21.15.27:40380) 2024-03-28T18:59:08.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.227+0000 7f1203400640 1 -- 172.21.15.27:0/371326852 learned_addr learned my addr 172.21.15.27:0/371326852 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:08.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.227+0000 7f1202a00640 1 --2- 172.21.15.27:0/371326852 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1204150c70 0x7f12040a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:08.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.228+0000 7f1203400640 1 -- 172.21.15.27:0/371326852 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1204150c70 msgr2=0x7f12040a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:08.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.228+0000 7f1203400640 1 --2- 172.21.15.27:0/371326852 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1204150c70 0x7f12040a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.228+0000 7f1203400640 1 -- 172.21.15.27:0/371326852 >> v1:172.21.15.155:6789/0 conn(0x7f12040c3690 legacy=0x7f12040c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:08.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.228+0000 7f1203400640 1 -- 172.21.15.27:0/371326852 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040c3d90 con 0x7f12041534c0 2024-03-28T18:59:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.229+0000 7f1203400640 1 --2- 172.21.15.27:0/371326852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12041534c0 0x7f1204153950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009e50 tx=0x7f11fc02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=203e2d38a31da034 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.230+0000 7f1201600640 1 -- 172.21.15.27:0/371326852 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc02ca70 con 0x7f12041534c0 2024-03-28T18:59:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.230+0000 7f1201600640 1 -- 172.21.15.27:0/371326852 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc02cc10 con 0x7f12041534c0 2024-03-28T18:59:08.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.230+0000 7f1201600640 1 -- 172.21.15.27:0/371326852 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0343f0 con 0x7f12041534c0 2024-03-28T18:59:08.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.230+0000 7f1209800640 1 -- 172.21.15.27:0/371326852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12041534c0 msgr2=0x7f1204153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.231+0000 7f1209800640 1 --2- 172.21.15.27:0/371326852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12041534c0 0x7f1204153950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009e50 tx=0x7f11fc02bed0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.231+0000 7f1209800640 1 -- 172.21.15.27:0/371326852 shutdown_connections 2024-03-28T18:59:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.231+0000 7f1209800640 1 --2- 172.21.15.27:0/371326852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12041534c0 0x7f1204153950 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.231+0000 7f1209800640 1 --2- 172.21.15.27:0/371326852 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1204150c70 0x7f12040a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.231+0000 7f1209800640 1 -- 172.21.15.27:0/371326852 >> 172.21.15.27:0/371326852 conn(0x7f120406b9d0 msgr2=0x7f12040c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.231+0000 7f1209800640 1 -- 172.21.15.27:0/371326852 shutdown_connections 2024-03-28T18:59:08.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.231+0000 7f1209800640 1 -- 172.21.15.27:0/371326852 wait complete. 2024-03-28T18:59:08.231 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/34 file after the snapshot 2024-03-28T18:59:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.233+0000 7f1209800640 1 Processor -- start 2024-03-28T18:59:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.233+0000 7f1209800640 1 -- start start 2024-03-28T18:59:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.234+0000 7f1209800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12040c3690 0x7f12040c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.234+0000 7f1209800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1204150c70 0x7f1204144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.234+0000 7f1209800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12041534c0 0x7f12041452d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.234+0000 7f1209800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040abcd0 con 0x7f12040c3690 2024-03-28T18:59:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.234+0000 7f1209800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040ab9f0 con 0x7f1204150c70 2024-03-28T18:59:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.234+0000 7f1209800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12040c1810 con 0x7f12041534c0 2024-03-28T18:59:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12040c3690 0x7f12040c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12040c3690 0x7f12040c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59556/0 (socket says 172.21.15.27:59556) 2024-03-28T18:59:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202a00640 1 -- 172.21.15.27:0/1910234161 learned_addr learned my addr 172.21.15.27:0/1910234161 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202000640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1204150c70 0x7f1204144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1203400640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12041534c0 0x7f12041452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202a00640 1 -- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12041534c0 msgr2=0x7f12041452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202a00640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12041534c0 0x7f12041452d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202a00640 1 -- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1204150c70 msgr2=0x7f1204144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202a00640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1204150c70 0x7f1204144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202a00640 1 -- 172.21.15.27:0/1910234161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204152f60 con 0x7f12040c3690 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.235+0000 7f1202000640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1204150c70 0x7f1204144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.236+0000 7f1202a00640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12040c3690 0x7f12040c02a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f11f400d830 tx=0x7f11f400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.236+0000 7f11faa00640 1 -- 172.21.15.27:0/1910234161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400b6c0 con 0x7f12040c3690 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.236+0000 7f11faa00640 1 -- 172.21.15.27:0/1910234161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4019070 con 0x7f12040c3690 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.236+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12040c2e90 con 0x7f12040c3690 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.236+0000 7f11faa00640 1 -- 172.21.15.27:0/1910234161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400ba80 con 0x7f12040c3690 2024-03-28T18:59:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.236+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12041499f0 con 0x7f12040c3690 2024-03-28T18:59:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.238+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120404f400 con 0x7f12040c3690 2024-03-28T18:59:08.238 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/35 file after the snapshot 2024-03-28T18:59:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.240+0000 7f11faa00640 1 -- 172.21.15.27:0/1910234161 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11f401e460 con 0x7f12040c3690 2024-03-28T18:59:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.241+0000 7f11faa00640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.241+0000 7f11faa00640 1 -- 172.21.15.27:0/1910234161 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f11f408b680 con 0x7f12040c3690 2024-03-28T18:59:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.241+0000 7f1202000640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.242+0000 7f1202000640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11e406bbd0 0x7f11e406e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f11f0007fd0 tx=0x7f11f0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:08.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.246+0000 7f11faa00640 1 -- 172.21.15.27:0/1910234161 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11f4059c40 con 0x7f12040c3690 2024-03-28T18:59:08.246 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/36 file after the snapshot 2024-03-28T18:59:08.254 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/37 file after the snapshot 2024-03-28T18:59:08.262 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/38 file after the snapshot 2024-03-28T18:59:08.269 INFO:tasks.workunit.client.0.smithi027.stdout: create 160/39 file after the snapshot 2024-03-28T18:59:08.277 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/.snap/snap-subdir-test 2024-03-28T18:59:08.314 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/30 file after the snapshot 2024-03-28T18:59:08.342 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/31 file after the snapshot 2024-03-28T18:59:08.373 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/32 file after the snapshot 2024-03-28T18:59:08.381 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/33 file after the snapshot 2024-03-28T18:59:08.391 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/34 file after the snapshot 2024-03-28T18:59:08.400 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/35 file after the snapshot 2024-03-28T18:59:08.410 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/36 file after the snapshot 2024-03-28T18:59:08.419 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/37 file after the snapshot 2024-03-28T18:59:08.427 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/38 file after the snapshot 2024-03-28T18:59:08.436 INFO:tasks.workunit.client.0.smithi027.stdout: create 161/39 file after the snapshot 2024-03-28T18:59:08.443 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/.snap/snap-subdir-test 2024-03-28T18:59:08.483 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/30 file after the snapshot 2024-03-28T18:59:08.487 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/31 file after the snapshot 2024-03-28T18:59:08.496 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/32 file after the snapshot 2024-03-28T18:59:08.504 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/33 file after the snapshot 2024-03-28T18:59:08.515 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/34 file after the snapshot 2024-03-28T18:59:08.523 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/35 file after the snapshot 2024-03-28T18:59:08.535 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/36 file after the snapshot 2024-03-28T18:59:08.546 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/37 file after the snapshot 2024-03-28T18:59:08.559 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/38 file after the snapshot 2024-03-28T18:59:08.572 INFO:tasks.workunit.client.0.smithi027.stdout: create 162/39 file after the snapshot 2024-03-28T18:59:08.584 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/.snap/snap-subdir-test 2024-03-28T18:59:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.593+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1204152290 con 0x7f12040c3690 2024-03-28T18:59:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.594+0000 7f11faa00640 1 -- 172.21.15.27:0/1910234161 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5466 (secure 0 0 0) 0x7f11f4059540 con 0x7f12040c3690 2024-03-28T18:59:08.594 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:08.594 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:06.640902+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:clientreplay","state_seq":15,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T18:59:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11e406bbd0 msgr2=0x7f11e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11e406bbd0 0x7f11e406e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f11f0007fd0 tx=0x7f11f0009040 comp rx=0 tx=0).stop 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12040c3690 msgr2=0x7f12040c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12040c3690 0x7f12040c02a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f11f400d830 tx=0x7f11f400dcf0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 shutdown_connections 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11e406bbd0 0x7f11e406e090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12041534c0 0x7f12041452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1204150c70 0x7f1204144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 --2- 172.21.15.27:0/1910234161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12040c3690 0x7f12040c02a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:08.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 >> 172.21.15.27:0/1910234161 conn(0x7f120406b9d0 msgr2=0x7f1204151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:08.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.597+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 shutdown_connections 2024-03-28T18:59:08.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:08.598+0000 7f1209800640 1 -- 172.21.15.27:0/1910234161 wait complete. 2024-03-28T18:59:08.621 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T18:59:08.621 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/30 file after the snapshot 2024-03-28T18:59:08.627 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/31 file after the snapshot 2024-03-28T18:59:08.636 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/32 file after the snapshot 2024-03-28T18:59:08.646 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/33 file after the snapshot 2024-03-28T18:59:08.655 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/34 file after the snapshot 2024-03-28T18:59:08.667 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/35 file after the snapshot 2024-03-28T18:59:08.677 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/36 file after the snapshot 2024-03-28T18:59:08.686 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/37 file after the snapshot 2024-03-28T18:59:08.691 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 225 2024-03-28T18:59:08.696 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/38 file after the snapshot 2024-03-28T18:59:08.703 INFO:tasks.workunit.client.0.smithi027.stdout: create 163/39 file after the snapshot 2024-03-28T18:59:08.710 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/.snap/snap-subdir-test 2024-03-28T18:59:08.742 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/30 file after the snapshot 2024-03-28T18:59:08.745 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/31 file after the snapshot 2024-03-28T18:59:08.752 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/32 file after the snapshot 2024-03-28T18:59:08.759 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/33 file after the snapshot 2024-03-28T18:59:08.763 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 226 2024-03-28T18:59:08.766 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/34 file after the snapshot 2024-03-28T18:59:08.774 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/35 file after the snapshot 2024-03-28T18:59:08.781 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/36 file after the snapshot 2024-03-28T18:59:08.790 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/37 file after the snapshot 2024-03-28T18:59:08.796 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/38 file after the snapshot 2024-03-28T18:59:08.803 INFO:tasks.workunit.client.0.smithi027.stdout: create 164/39 file after the snapshot 2024-03-28T18:59:08.812 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/.snap/snap-subdir-test 2024-03-28T18:59:08.832 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 227 2024-03-28T18:59:08.844 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/30 file after the snapshot 2024-03-28T18:59:08.849 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/31 file after the snapshot 2024-03-28T18:59:08.856 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/32 file after the snapshot 2024-03-28T18:59:08.863 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/33 file after the snapshot 2024-03-28T18:59:08.872 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/34 file after the snapshot 2024-03-28T18:59:08.878 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/35 file after the snapshot 2024-03-28T18:59:08.886 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/36 file after the snapshot 2024-03-28T18:59:08.894 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/37 file after the snapshot 2024-03-28T18:59:08.897 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 228 2024-03-28T18:59:08.900 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/38 file after the snapshot 2024-03-28T18:59:08.907 INFO:tasks.workunit.client.0.smithi027.stdout: create 165/39 file after the snapshot 2024-03-28T18:59:08.914 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/.snap/snap-subdir-test 2024-03-28T18:59:08.943 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/30 file after the snapshot 2024-03-28T18:59:08.948 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/31 file after the snapshot 2024-03-28T18:59:08.955 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/32 file after the snapshot 2024-03-28T18:59:08.962 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/33 file after the snapshot 2024-03-28T18:59:08.965 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 229 2024-03-28T18:59:08.969 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/34 file after the snapshot 2024-03-28T18:59:08.977 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/35 file after the snapshot 2024-03-28T18:59:08.984 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/36 file after the snapshot 2024-03-28T18:59:08.989 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/37 file after the snapshot 2024-03-28T18:59:08.995 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/38 file after the snapshot 2024-03-28T18:59:09.002 INFO:tasks.workunit.client.0.smithi027.stdout: create 166/39 file after the snapshot 2024-03-28T18:59:09.009 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/.snap/snap-subdir-test 2024-03-28T18:59:09.029 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 230 2024-03-28T18:59:09.040 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/30 file after the snapshot 2024-03-28T18:59:09.046 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/31 file after the snapshot 2024-03-28T18:59:09.054 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/32 file after the snapshot 2024-03-28T18:59:09.060 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/33 file after the snapshot 2024-03-28T18:59:09.067 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/34 file after the snapshot 2024-03-28T18:59:09.072 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/35 file after the snapshot 2024-03-28T18:59:09.080 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/36 file after the snapshot 2024-03-28T18:59:09.087 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/37 file after the snapshot 2024-03-28T18:59:09.094 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/38 file after the snapshot 2024-03-28T18:59:09.097 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 231 2024-03-28T18:59:09.101 INFO:tasks.workunit.client.0.smithi027.stdout: create 167/39 file after the snapshot 2024-03-28T18:59:09.109 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/.snap/snap-subdir-test 2024-03-28T18:59:09.139 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/30 file after the snapshot 2024-03-28T18:59:09.144 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/31 file after the snapshot 2024-03-28T18:59:09.152 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/32 file after the snapshot 2024-03-28T18:59:09.162 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/33 file after the snapshot 2024-03-28T18:59:09.166 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 232 2024-03-28T18:59:09.169 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/34 file after the snapshot 2024-03-28T18:59:09.177 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/35 file after the snapshot 2024-03-28T18:59:09.185 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/36 file after the snapshot 2024-03-28T18:59:09.193 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/37 file after the snapshot 2024-03-28T18:59:09.200 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/38 file after the snapshot 2024-03-28T18:59:09.208 INFO:tasks.workunit.client.0.smithi027.stdout: create 168/39 file after the snapshot 2024-03-28T18:59:09.214 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/.snap/snap-subdir-test 2024-03-28T18:59:09.236 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 233 2024-03-28T18:59:09.247 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/30 file after the snapshot 2024-03-28T18:59:09.252 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/31 file after the snapshot 2024-03-28T18:59:09.259 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/32 file after the snapshot 2024-03-28T18:59:09.266 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/33 file after the snapshot 2024-03-28T18:59:09.272 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/34 file after the snapshot 2024-03-28T18:59:09.279 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/35 file after the snapshot 2024-03-28T18:59:09.287 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/36 file after the snapshot 2024-03-28T18:59:09.293 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/37 file after the snapshot 2024-03-28T18:59:09.305 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/38 file after the snapshot 2024-03-28T18:59:09.310 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 234 2024-03-28T18:59:09.332 INFO:tasks.workunit.client.0.smithi027.stdout: create 169/39 file after the snapshot 2024-03-28T18:59:09.348 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/.snap/snap-subdir-test 2024-03-28T18:59:09.376 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/30 file after the snapshot 2024-03-28T18:59:09.379 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 235 2024-03-28T18:59:09.380 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/31 file after the snapshot 2024-03-28T18:59:09.389 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/32 file after the snapshot 2024-03-28T18:59:09.397 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/33 file after the snapshot 2024-03-28T18:59:09.403 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/34 file after the snapshot 2024-03-28T18:59:09.409 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/35 file after the snapshot 2024-03-28T18:59:09.416 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/36 file after the snapshot 2024-03-28T18:59:09.422 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/37 file after the snapshot 2024-03-28T18:59:09.429 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/38 file after the snapshot 2024-03-28T18:59:09.435 INFO:tasks.workunit.client.0.smithi027.stdout: create 170/39 file after the snapshot 2024-03-28T18:59:09.441 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/.snap/snap-subdir-test 2024-03-28T18:59:09.446 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 236 2024-03-28T18:59:09.469 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/30 file after the snapshot 2024-03-28T18:59:09.472 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/31 file after the snapshot 2024-03-28T18:59:09.479 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/32 file after the snapshot 2024-03-28T18:59:09.485 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/33 file after the snapshot 2024-03-28T18:59:09.491 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/34 file after the snapshot 2024-03-28T18:59:09.497 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/35 file after the snapshot 2024-03-28T18:59:09.504 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/36 file after the snapshot 2024-03-28T18:59:09.513 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/37 file after the snapshot 2024-03-28T18:59:09.514 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 237 2024-03-28T18:59:09.516 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/38 file after the snapshot 2024-03-28T18:59:09.522 INFO:tasks.workunit.client.0.smithi027.stdout: create 171/39 file after the snapshot 2024-03-28T18:59:09.529 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/.snap/snap-subdir-test 2024-03-28T18:59:09.554 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/30 file after the snapshot 2024-03-28T18:59:09.558 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/31 file after the snapshot 2024-03-28T18:59:09.564 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/32 file after the snapshot 2024-03-28T18:59:09.571 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/33 file after the snapshot 2024-03-28T18:59:09.577 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/34 file after the snapshot 2024-03-28T18:59:09.580 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 238 2024-03-28T18:59:09.584 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/35 file after the snapshot 2024-03-28T18:59:09.590 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/36 file after the snapshot 2024-03-28T18:59:09.597 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/37 file after the snapshot 2024-03-28T18:59:09.603 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/38 file after the snapshot 2024-03-28T18:59:09.609 INFO:tasks.workunit.client.0.smithi027.stdout: create 172/39 file after the snapshot 2024-03-28T18:59:09.615 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/.snap/snap-subdir-test 2024-03-28T18:59:09.643 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/30 file after the snapshot 2024-03-28T18:59:09.645 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 239 2024-03-28T18:59:09.647 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/31 file after the snapshot 2024-03-28T18:59:09.655 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/32 file after the snapshot 2024-03-28T18:59:09.662 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/33 file after the snapshot 2024-03-28T18:59:09.670 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/34 file after the snapshot 2024-03-28T18:59:09.677 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/35 file after the snapshot 2024-03-28T18:59:09.684 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/36 file after the snapshot 2024-03-28T18:59:09.691 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/37 file after the snapshot 2024-03-28T18:59:09.697 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/38 file after the snapshot 2024-03-28T18:59:09.703 INFO:tasks.workunit.client.0.smithi027.stdout: create 173/39 file after the snapshot 2024-03-28T18:59:09.709 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/.snap/snap-subdir-test 2024-03-28T18:59:09.709 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 240 2024-03-28T18:59:09.734 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/30 file after the snapshot 2024-03-28T18:59:09.739 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/31 file after the snapshot 2024-03-28T18:59:09.746 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/32 file after the snapshot 2024-03-28T18:59:09.752 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/33 file after the snapshot 2024-03-28T18:59:09.758 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/34 file after the snapshot 2024-03-28T18:59:09.766 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/35 file after the snapshot 2024-03-28T18:59:09.773 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/36 file after the snapshot 2024-03-28T18:59:09.776 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 241 2024-03-28T18:59:09.779 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/37 file after the snapshot 2024-03-28T18:59:09.785 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/38 file after the snapshot 2024-03-28T18:59:09.793 INFO:tasks.workunit.client.0.smithi027.stdout: create 174/39 file after the snapshot 2024-03-28T18:59:09.800 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/.snap/snap-subdir-test 2024-03-28T18:59:09.823 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/30 file after the snapshot 2024-03-28T18:59:09.827 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/31 file after the snapshot 2024-03-28T18:59:09.834 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/32 file after the snapshot 2024-03-28T18:59:09.840 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/33 file after the snapshot 2024-03-28T18:59:09.848 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/34 file after the snapshot 2024-03-28T18:59:09.849 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 242 2024-03-28T18:59:09.855 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/35 file after the snapshot 2024-03-28T18:59:09.864 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/36 file after the snapshot 2024-03-28T18:59:09.871 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/37 file after the snapshot 2024-03-28T18:59:09.878 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/38 file after the snapshot 2024-03-28T18:59:09.885 INFO:tasks.workunit.client.0.smithi027.stdout: create 175/39 file after the snapshot 2024-03-28T18:59:09.891 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/.snap/snap-subdir-test 2024-03-28T18:59:09.915 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 243 2024-03-28T18:59:09.917 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/30 file after the snapshot 2024-03-28T18:59:09.921 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/31 file after the snapshot 2024-03-28T18:59:09.926 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/32 file after the snapshot 2024-03-28T18:59:09.932 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/33 file after the snapshot 2024-03-28T18:59:09.938 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/34 file after the snapshot 2024-03-28T18:59:09.944 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/35 file after the snapshot 2024-03-28T18:59:09.950 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/36 file after the snapshot 2024-03-28T18:59:09.957 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/37 file after the snapshot 2024-03-28T18:59:09.963 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/38 file after the snapshot 2024-03-28T18:59:09.969 INFO:tasks.workunit.client.0.smithi027.stdout: create 176/39 file after the snapshot 2024-03-28T18:59:09.977 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/.snap/snap-subdir-test 2024-03-28T18:59:09.987 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 244 2024-03-28T18:59:10.002 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/30 file after the snapshot 2024-03-28T18:59:10.006 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/31 file after the snapshot 2024-03-28T18:59:10.013 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/32 file after the snapshot 2024-03-28T18:59:10.019 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/33 file after the snapshot 2024-03-28T18:59:10.026 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/34 file after the snapshot 2024-03-28T18:59:10.032 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/35 file after the snapshot 2024-03-28T18:59:10.039 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/36 file after the snapshot 2024-03-28T18:59:10.046 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/37 file after the snapshot 2024-03-28T18:59:10.052 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/38 file after the snapshot 2024-03-28T18:59:10.053 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 245 2024-03-28T18:59:10.060 INFO:tasks.workunit.client.0.smithi027.stdout: create 177/39 file after the snapshot 2024-03-28T18:59:10.065 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/.snap/snap-subdir-test 2024-03-28T18:59:10.089 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/30 file after the snapshot 2024-03-28T18:59:10.094 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/31 file after the snapshot 2024-03-28T18:59:10.101 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/32 file after the snapshot 2024-03-28T18:59:10.108 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/33 file after the snapshot 2024-03-28T18:59:10.114 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/34 file after the snapshot 2024-03-28T18:59:10.121 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/35 file after the snapshot 2024-03-28T18:59:10.122 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 246 2024-03-28T18:59:10.127 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/36 file after the snapshot 2024-03-28T18:59:10.133 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/37 file after the snapshot 2024-03-28T18:59:10.141 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/38 file after the snapshot 2024-03-28T18:59:10.147 INFO:tasks.workunit.client.0.smithi027.stdout: create 178/39 file after the snapshot 2024-03-28T18:59:10.154 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/.snap/snap-subdir-test 2024-03-28T18:59:10.179 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/30 file after the snapshot 2024-03-28T18:59:10.183 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/31 file after the snapshot 2024-03-28T18:59:10.189 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/32 file after the snapshot 2024-03-28T18:59:10.194 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 247 2024-03-28T18:59:10.201 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/33 file after the snapshot 2024-03-28T18:59:10.208 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/34 file after the snapshot 2024-03-28T18:59:10.215 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/35 file after the snapshot 2024-03-28T18:59:10.223 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/36 file after the snapshot 2024-03-28T18:59:10.230 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/37 file after the snapshot 2024-03-28T18:59:10.236 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/38 file after the snapshot 2024-03-28T18:59:10.245 INFO:tasks.workunit.client.0.smithi027.stdout: create 179/39 file after the snapshot 2024-03-28T18:59:10.251 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/.snap/snap-subdir-test 2024-03-28T18:59:10.280 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/30 file after the snapshot 2024-03-28T18:59:10.284 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/31 file after the snapshot 2024-03-28T18:59:10.292 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 248 2024-03-28T18:59:10.293 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/32 file after the snapshot 2024-03-28T18:59:10.300 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/33 file after the snapshot 2024-03-28T18:59:10.367 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/34 file after the snapshot 2024-03-28T18:59:10.383 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 249 2024-03-28T18:59:10.398 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/35 file after the snapshot 2024-03-28T18:59:10.405 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/36 file after the snapshot 2024-03-28T18:59:10.412 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/37 file after the snapshot 2024-03-28T18:59:10.419 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/38 file after the snapshot 2024-03-28T18:59:10.426 INFO:tasks.workunit.client.0.smithi027.stdout: create 180/39 file after the snapshot 2024-03-28T18:59:10.433 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/.snap/snap-subdir-test 2024-03-28T18:59:10.459 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/30 file after the snapshot 2024-03-28T18:59:10.459 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 250 2024-03-28T18:59:10.463 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/31 file after the snapshot 2024-03-28T18:59:10.469 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/32 file after the snapshot 2024-03-28T18:59:10.477 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/33 file after the snapshot 2024-03-28T18:59:10.484 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/34 file after the snapshot 2024-03-28T18:59:10.491 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/35 file after the snapshot 2024-03-28T18:59:10.498 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/36 file after the snapshot 2024-03-28T18:59:10.505 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/37 file after the snapshot 2024-03-28T18:59:10.511 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/38 file after the snapshot 2024-03-28T18:59:10.518 INFO:tasks.workunit.client.0.smithi027.stdout: create 181/39 file after the snapshot 2024-03-28T18:59:10.524 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/.snap/snap-subdir-test 2024-03-28T18:59:10.555 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/30 file after the snapshot 2024-03-28T18:59:10.559 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/31 file after the snapshot 2024-03-28T18:59:10.567 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/32 file after the snapshot 2024-03-28T18:59:10.578 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/33 file after the snapshot 2024-03-28T18:59:10.588 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/34 file after the snapshot 2024-03-28T18:59:10.595 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/35 file after the snapshot 2024-03-28T18:59:10.605 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/36 file after the snapshot 2024-03-28T18:59:10.615 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/37 file after the snapshot 2024-03-28T18:59:10.617 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T18:59:10.617+0000 7fd10f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:59:10.621 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:10.628 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/38 file after the snapshot 2024-03-28T18:59:10.637 INFO:tasks.workunit.client.0.smithi027.stdout: create 182/39 file after the snapshot 2024-03-28T18:59:10.645 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/.snap/snap-subdir-test 2024-03-28T18:59:10.673 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/30 file after the snapshot 2024-03-28T18:59:10.678 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/31 file after the snapshot 2024-03-28T18:59:10.685 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/32 file after the snapshot 2024-03-28T18:59:10.693 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/33 file after the snapshot 2024-03-28T18:59:10.702 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/34 file after the snapshot 2024-03-28T18:59:10.711 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/35 file after the snapshot 2024-03-28T18:59:10.718 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/36 file after the snapshot 2024-03-28T18:59:10.725 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/37 file after the snapshot 2024-03-28T18:59:10.733 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/38 file after the snapshot 2024-03-28T18:59:10.743 INFO:tasks.workunit.client.0.smithi027.stdout: create 183/39 file after the snapshot 2024-03-28T18:59:10.753 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/.snap/snap-subdir-test 2024-03-28T18:59:10.787 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/30 file after the snapshot 2024-03-28T18:59:10.790 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/31 file after the snapshot 2024-03-28T18:59:10.798 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/32 file after the snapshot 2024-03-28T18:59:10.807 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/33 file after the snapshot 2024-03-28T18:59:10.817 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/34 file after the snapshot 2024-03-28T18:59:10.826 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/35 file after the snapshot 2024-03-28T18:59:10.833 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/36 file after the snapshot 2024-03-28T18:59:10.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.840+0000 7f9f14600640 1 Processor -- start 2024-03-28T18:59:10.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.840+0000 7f9f14600640 1 -- start start 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f14600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9f0c152bd0 0x7f9f0c06eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f14600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c06f1e0 0x7f9f0c070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f14600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f0c0c4dc0 0x7f9f0c158c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f14600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0c0c52e0 con 0x7f9f0c0c4dc0 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f14600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0c1557e0 con 0x7f9f0c152bd0 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f14600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c155ac0 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f12600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f0c0c4dc0 0x7f9f0c158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f12600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f0c0c4dc0 0x7f9f0c158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59558/0 (socket says 172.21.15.27:59558) 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f12600640 1 -- 172.21.15.27:0/2056725904 learned_addr learned my addr 172.21.15.27:0/2056725904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:10.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f11c00640 1 --2- 172.21.15.27:0/2056725904 >> v2:172.21.15.155:3300/0 conn(0x7f9f0c152bd0 0x7f9f0c06eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.842+0000 7f9f11200640 1 --2- 172.21.15.27:0/2056725904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c06f1e0 0x7f9f0c070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.843+0000 7f9f12600640 1 -- 172.21.15.27:0/2056725904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c06f1e0 msgr2=0x7f9f0c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.843+0000 7f9f12600640 1 --2- 172.21.15.27:0/2056725904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c06f1e0 0x7f9f0c070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.843+0000 7f9f12600640 1 -- 172.21.15.27:0/2056725904 >> v2:172.21.15.155:3300/0 conn(0x7f9f0c152bd0 msgr2=0x7f9f0c06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.843+0000 7f9f12600640 1 --2- 172.21.15.27:0/2056725904 >> v2:172.21.15.155:3300/0 conn(0x7f9f0c152bd0 0x7f9f0c06eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.843+0000 7f9f12600640 1 -- 172.21.15.27:0/2056725904 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c155da0 con 0x7f9f0c0c4dc0 2024-03-28T18:59:10.844 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/37 file after the snapshot 2024-03-28T18:59:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.844+0000 7f9f12600640 1 --2- 172.21.15.27:0/2056725904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c0c4dc0 0x7f9f0c158c30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c06ec70 tx=0x7f9f0402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4de6e4f5afb2bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.845+0000 7f9f0be00640 1 -- 172.21.15.27:0/2056725904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0402ca80 con 0x7f9f0c0c4dc0 2024-03-28T18:59:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.845+0000 7f9f0be00640 1 -- 172.21.15.27:0/2056725904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0402cc20 con 0x7f9f0c0c4dc0 2024-03-28T18:59:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.845+0000 7f9f0be00640 1 -- 172.21.15.27:0/2056725904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f040343f0 con 0x7f9f0c0c4dc0 2024-03-28T18:59:10.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 -- 172.21.15.27:0/2056725904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c0c4dc0 msgr2=0x7f9f0c158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:10.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 --2- 172.21.15.27:0/2056725904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c0c4dc0 0x7f9f0c158c30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c06ec70 tx=0x7f9f0402bee0 comp rx=0 tx=0).stop 2024-03-28T18:59:10.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 -- 172.21.15.27:0/2056725904 shutdown_connections 2024-03-28T18:59:10.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 --2- 172.21.15.27:0/2056725904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c0c4dc0 0x7f9f0c158c30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:10.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 --2- 172.21.15.27:0/2056725904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c06f1e0 0x7f9f0c070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:10.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 --2- 172.21.15.27:0/2056725904 >> v2:172.21.15.155:3300/0 conn(0x7f9f0c152bd0 0x7f9f0c06eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:10.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 -- 172.21.15.27:0/2056725904 >> 172.21.15.27:0/2056725904 conn(0x7f9f0c066dd0 msgr2=0x7f9f0c0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:10.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 -- 172.21.15.27:0/2056725904 shutdown_connections 2024-03-28T18:59:10.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.846+0000 7f9f14600640 1 -- 172.21.15.27:0/2056725904 wait complete. 2024-03-28T18:59:10.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.848+0000 7f9f14600640 1 Processor -- start 2024-03-28T18:59:10.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.848+0000 7f9f14600640 1 -- start start 2024-03-28T18:59:10.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f14600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c06f1e0 0x7f9f0c1ef2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f14600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c0c4dc0 0x7f9f0c146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f14600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f0c152bd0 0x7f9f0c09a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f14600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c0c6df0 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f14600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c0c6c70 con 0x7f9f0c152bd0 2024-03-28T18:59:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f14600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c070a80 con 0x7f9f0c0c4dc0 2024-03-28T18:59:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f11c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c06f1e0 0x7f9f0c1ef2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f11c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c06f1e0 0x7f9f0c1ef2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59560/0 (socket says 172.21.15.27:59560) 2024-03-28T18:59:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.849+0000 7f9f11c00640 1 -- 172.21.15.27:0/1772015566 learned_addr learned my addr 172.21.15.27:0/1772015566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:10.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.850+0000 7f9f11200640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c0c4dc0 0x7f9f0c146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:10.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.850+0000 7f9f12600640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f0c152bd0 0x7f9f0c09a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:10.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.850+0000 7f9f11c00640 1 -- 172.21.15.27:0/1772015566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c0c4dc0 msgr2=0x7f9f0c146380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:10.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.850+0000 7f9f11c00640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c0c4dc0 0x7f9f0c146380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:10.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.850+0000 7f9f11c00640 1 -- 172.21.15.27:0/1772015566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f0c152bd0 msgr2=0x7f9f0c09a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:10.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.850+0000 7f9f11c00640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f0c152bd0 0x7f9f0c09a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:10.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.850+0000 7f9f11c00640 1 -- 172.21.15.27:0/1772015566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c158e00 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.851+0000 7f9f11c00640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c06f1e0 0x7f9f0c1ef2f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000e9b0 tx=0x7f9f0000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:10.851 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/38 file after the snapshot 2024-03-28T18:59:10.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.851+0000 7f9f09600640 1 -- 172.21.15.27:0/1772015566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00017070 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.851+0000 7f9f09600640 1 -- 172.21.15.27:0/1772015566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0000c9a0 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.851+0000 7f9f09600640 1 -- 172.21.15.27:0/1772015566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000cd90 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.851+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c1590e0 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.851+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0c1eeaf0 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.853+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0c04f400 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.854+0000 7f9f09600640 1 -- 172.21.15.27:0/1772015566 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9f0001a070 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.855+0000 7f9f09600640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9eec06bc00 0x7f9eec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:10.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.855+0000 7f9f09600640 1 -- 172.21.15.27:0/1772015566 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f9f0008ae60 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.855+0000 7f9f11200640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9eec06bc00 0x7f9eec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.856+0000 7f9f11200640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9eec06bc00 0x7f9eec06e0c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0c5ae0 tx=0x7f9ef8009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:10.857 INFO:tasks.workunit.client.0.smithi027.stdout: create 184/39 file after the snapshot 2024-03-28T18:59:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:10.860+0000 7f9f09600640 1 -- 172.21.15.27:0/1772015566 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9f00058e50 con 0x7f9f0c06f1e0 2024-03-28T18:59:10.865 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/.snap/snap-subdir-test 2024-03-28T18:59:10.897 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/30 file after the snapshot 2024-03-28T18:59:10.902 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/31 file after the snapshot 2024-03-28T18:59:10.910 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/32 file after the snapshot 2024-03-28T18:59:10.919 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/33 file after the snapshot 2024-03-28T18:59:10.929 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/34 file after the snapshot 2024-03-28T18:59:10.938 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/35 file after the snapshot 2024-03-28T18:59:10.948 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/36 file after the snapshot 2024-03-28T18:59:10.958 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/37 file after the snapshot 2024-03-28T18:59:10.967 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/38 file after the snapshot 2024-03-28T18:59:10.977 INFO:tasks.workunit.client.0.smithi027.stdout: create 185/39 file after the snapshot 2024-03-28T18:59:10.987 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/.snap/snap-subdir-test 2024-03-28T18:59:11.040 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/30 file after the snapshot 2024-03-28T18:59:11.045 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/31 file after the snapshot 2024-03-28T18:59:11.055 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/32 file after the snapshot 2024-03-28T18:59:11.067 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/33 file after the snapshot 2024-03-28T18:59:11.078 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/34 file after the snapshot 2024-03-28T18:59:11.090 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/35 file after the snapshot 2024-03-28T18:59:11.101 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/36 file after the snapshot 2024-03-28T18:59:11.110 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/37 file after the snapshot 2024-03-28T18:59:11.123 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/38 file after the snapshot 2024-03-28T18:59:11.133 INFO:tasks.workunit.client.0.smithi027.stdout: create 186/39 file after the snapshot 2024-03-28T18:59:11.145 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/.snap/snap-subdir-test 2024-03-28T18:59:11.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.164+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f0c0c3020 con 0x7f9f0c06f1e0 2024-03-28T18:59:11.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.164+0000 7f9f09600640 1 -- 172.21.15.27:0/1772015566 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5467 (secure 0 0 0) 0x7f9f0005cef0 con 0x7f9f0c06f1e0 2024-03-28T18:59:11.164 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:11.164 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:11.164 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-03-28T18:59:11.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.168+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9eec06bc00 msgr2=0x7f9eec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:11.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.168+0000 7f9f14600640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9eec06bc00 0x7f9eec06e0c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0c5ae0 tx=0x7f9ef8009190 comp rx=0 tx=0).stop 2024-03-28T18:59:11.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.168+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c06f1e0 msgr2=0x7f9f0c1ef2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:11.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.168+0000 7f9f14600640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c06f1e0 0x7f9f0c1ef2f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000e9b0 tx=0x7f9f0000ee70 comp rx=0 tx=0).stop 2024-03-28T18:59:11.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.169+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 shutdown_connections 2024-03-28T18:59:11.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.169+0000 7f9f14600640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9eec06bc00 0x7f9eec06e0c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:11.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.169+0000 7f9f14600640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f0c152bd0 0x7f9f0c09a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:11.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.169+0000 7f9f14600640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f0c0c4dc0 0x7f9f0c146380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:11.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.169+0000 7f9f14600640 1 --2- 172.21.15.27:0/1772015566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f0c06f1e0 0x7f9f0c1ef2f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:11.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.169+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 >> 172.21.15.27:0/1772015566 conn(0x7f9f0c066dd0 msgr2=0x7f9f0c154fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:11.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.169+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 shutdown_connections 2024-03-28T18:59:11.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:11.169+0000 7f9f14600640 1 -- 172.21.15.27:0/1772015566 wait complete. 2024-03-28T18:59:11.181 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/30 file after the snapshot 2024-03-28T18:59:11.184 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/31 file after the snapshot 2024-03-28T18:59:11.188 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=4260 2024-03-28T18:59:11.188 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.a 2024-03-28T18:59:11.192 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/32 file after the snapshot 2024-03-28T18:59:11.200 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/33 file after the snapshot 2024-03-28T18:59:11.209 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/34 file after the snapshot 2024-03-28T18:59:11.222 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/35 file after the snapshot 2024-03-28T18:59:11.234 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/36 file after the snapshot 2024-03-28T18:59:11.245 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/37 file after the snapshot 2024-03-28T18:59:11.256 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/38 file after the snapshot 2024-03-28T18:59:11.268 INFO:tasks.workunit.client.0.smithi027.stdout: create 187/39 file after the snapshot 2024-03-28T18:59:11.277 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/.snap/snap-subdir-test 2024-03-28T18:59:11.317 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/30 file after the snapshot 2024-03-28T18:59:11.356 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/31 file after the snapshot 2024-03-28T18:59:11.397 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/32 file after the snapshot 2024-03-28T18:59:11.408 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/33 file after the snapshot 2024-03-28T18:59:11.420 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/34 file after the snapshot 2024-03-28T18:59:11.432 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/35 file after the snapshot 2024-03-28T18:59:11.446 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/36 file after the snapshot 2024-03-28T18:59:11.458 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/37 file after the snapshot 2024-03-28T18:59:11.471 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/38 file after the snapshot 2024-03-28T18:59:11.484 INFO:tasks.workunit.client.0.smithi027.stdout: create 188/39 file after the snapshot 2024-03-28T18:59:11.497 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/.snap/snap-subdir-test 2024-03-28T18:59:11.528 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/30 file after the snapshot 2024-03-28T18:59:11.533 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/31 file after the snapshot 2024-03-28T18:59:11.543 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/32 file after the snapshot 2024-03-28T18:59:11.554 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/33 file after the snapshot 2024-03-28T18:59:11.566 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/34 file after the snapshot 2024-03-28T18:59:11.579 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/35 file after the snapshot 2024-03-28T18:59:11.588 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/36 file after the snapshot 2024-03-28T18:59:11.600 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/37 file after the snapshot 2024-03-28T18:59:11.611 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/38 file after the snapshot 2024-03-28T18:59:11.625 INFO:tasks.workunit.client.0.smithi027.stdout: create 189/39 file after the snapshot 2024-03-28T18:59:11.640 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/.snap/snap-subdir-test 2024-03-28T18:59:11.684 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/30 file after the snapshot 2024-03-28T18:59:11.689 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/31 file after the snapshot 2024-03-28T18:59:11.698 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/32 file after the snapshot 2024-03-28T18:59:11.708 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/33 file after the snapshot 2024-03-28T18:59:11.718 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/34 file after the snapshot 2024-03-28T18:59:11.730 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/35 file after the snapshot 2024-03-28T18:59:11.743 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/36 file after the snapshot 2024-03-28T18:59:11.756 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/37 file after the snapshot 2024-03-28T18:59:11.768 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/38 file after the snapshot 2024-03-28T18:59:11.782 INFO:tasks.workunit.client.0.smithi027.stdout: create 190/39 file after the snapshot 2024-03-28T18:59:11.792 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/.snap/snap-subdir-test 2024-03-28T18:59:11.832 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/30 file after the snapshot 2024-03-28T18:59:11.836 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/31 file after the snapshot 2024-03-28T18:59:11.846 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/32 file after the snapshot 2024-03-28T18:59:11.859 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/33 file after the snapshot 2024-03-28T18:59:11.871 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/34 file after the snapshot 2024-03-28T18:59:11.884 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/35 file after the snapshot 2024-03-28T18:59:11.896 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/36 file after the snapshot 2024-03-28T18:59:11.907 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/37 file after the snapshot 2024-03-28T18:59:11.922 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/38 file after the snapshot 2024-03-28T18:59:11.952 INFO:tasks.workunit.client.0.smithi027.stdout: create 191/39 file after the snapshot 2024-03-28T18:59:11.974 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/.snap/snap-subdir-test 2024-03-28T18:59:12.009 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/30 file after the snapshot 2024-03-28T18:59:12.015 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/31 file after the snapshot 2024-03-28T18:59:12.027 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/32 file after the snapshot 2024-03-28T18:59:12.041 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/33 file after the snapshot 2024-03-28T18:59:12.051 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/34 file after the snapshot 2024-03-28T18:59:12.064 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/35 file after the snapshot 2024-03-28T18:59:12.077 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/36 file after the snapshot 2024-03-28T18:59:12.092 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/37 file after the snapshot 2024-03-28T18:59:12.106 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/38 file after the snapshot 2024-03-28T18:59:12.119 INFO:tasks.workunit.client.0.smithi027.stdout: create 192/39 file after the snapshot 2024-03-28T18:59:12.133 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/.snap/snap-subdir-test 2024-03-28T18:59:12.172 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/30 file after the snapshot 2024-03-28T18:59:12.180 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/31 file after the snapshot 2024-03-28T18:59:12.191 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/32 file after the snapshot 2024-03-28T18:59:12.204 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/33 file after the snapshot 2024-03-28T18:59:12.216 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/34 file after the snapshot 2024-03-28T18:59:12.229 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/35 file after the snapshot 2024-03-28T18:59:12.241 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/36 file after the snapshot 2024-03-28T18:59:12.254 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/37 file after the snapshot 2024-03-28T18:59:12.267 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/38 file after the snapshot 2024-03-28T18:59:12.281 INFO:tasks.workunit.client.0.smithi027.stdout: create 193/39 file after the snapshot 2024-03-28T18:59:12.296 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/.snap/snap-subdir-test 2024-03-28T18:59:12.329 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/30 file after the snapshot 2024-03-28T18:59:12.355 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/31 file after the snapshot 2024-03-28T18:59:12.404 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/32 file after the snapshot 2024-03-28T18:59:12.414 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/33 file after the snapshot 2024-03-28T18:59:12.426 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/34 file after the snapshot 2024-03-28T18:59:12.440 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/35 file after the snapshot 2024-03-28T18:59:12.453 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/36 file after the snapshot 2024-03-28T18:59:12.465 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/37 file after the snapshot 2024-03-28T18:59:12.478 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/38 file after the snapshot 2024-03-28T18:59:12.490 INFO:tasks.workunit.client.0.smithi027.stdout: create 194/39 file after the snapshot 2024-03-28T18:59:12.504 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/.snap/snap-subdir-test 2024-03-28T18:59:12.547 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/30 file after the snapshot 2024-03-28T18:59:12.556 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/31 file after the snapshot 2024-03-28T18:59:12.569 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/32 file after the snapshot 2024-03-28T18:59:12.582 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/33 file after the snapshot 2024-03-28T18:59:12.597 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/34 file after the snapshot 2024-03-28T18:59:12.608 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/35 file after the snapshot 2024-03-28T18:59:12.618 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/36 file after the snapshot 2024-03-28T18:59:12.630 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/37 file after the snapshot 2024-03-28T18:59:12.640 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/38 file after the snapshot 2024-03-28T18:59:12.650 INFO:tasks.workunit.client.0.smithi027.stdout: create 195/39 file after the snapshot 2024-03-28T18:59:12.659 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/.snap/snap-subdir-test 2024-03-28T18:59:12.704 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/30 file after the snapshot 2024-03-28T18:59:12.710 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/31 file after the snapshot 2024-03-28T18:59:12.721 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/32 file after the snapshot 2024-03-28T18:59:12.732 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/33 file after the snapshot 2024-03-28T18:59:12.743 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/34 file after the snapshot 2024-03-28T18:59:12.757 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/35 file after the snapshot 2024-03-28T18:59:12.770 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/36 file after the snapshot 2024-03-28T18:59:12.782 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/37 file after the snapshot 2024-03-28T18:59:12.794 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/38 file after the snapshot 2024-03-28T18:59:12.806 INFO:tasks.workunit.client.0.smithi027.stdout: create 196/39 file after the snapshot 2024-03-28T18:59:12.819 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/.snap/snap-subdir-test 2024-03-28T18:59:12.862 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/30 file after the snapshot 2024-03-28T18:59:12.867 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/31 file after the snapshot 2024-03-28T18:59:12.879 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/32 file after the snapshot 2024-03-28T18:59:12.891 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/33 file after the snapshot 2024-03-28T18:59:12.903 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/34 file after the snapshot 2024-03-28T18:59:12.917 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/35 file after the snapshot 2024-03-28T18:59:12.928 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/36 file after the snapshot 2024-03-28T18:59:12.939 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/37 file after the snapshot 2024-03-28T18:59:12.953 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/38 file after the snapshot 2024-03-28T18:59:12.966 INFO:tasks.workunit.client.0.smithi027.stdout: create 197/39 file after the snapshot 2024-03-28T18:59:12.979 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/.snap/snap-subdir-test 2024-03-28T18:59:13.027 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/30 file after the snapshot 2024-03-28T18:59:13.035 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/31 file after the snapshot 2024-03-28T18:59:13.047 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/32 file after the snapshot 2024-03-28T18:59:13.061 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/33 file after the snapshot 2024-03-28T18:59:13.075 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/34 file after the snapshot 2024-03-28T18:59:13.088 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/35 file after the snapshot 2024-03-28T18:59:13.101 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/36 file after the snapshot 2024-03-28T18:59:13.114 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/37 file after the snapshot 2024-03-28T18:59:13.127 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/38 file after the snapshot 2024-03-28T18:59:13.139 INFO:tasks.workunit.client.0.smithi027.stdout: create 198/39 file after the snapshot 2024-03-28T18:59:13.150 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/.snap/snap-subdir-test 2024-03-28T18:59:13.197 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/30 file after the snapshot 2024-03-28T18:59:13.204 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/31 file after the snapshot 2024-03-28T18:59:13.215 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/32 file after the snapshot 2024-03-28T18:59:13.227 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/33 file after the snapshot 2024-03-28T18:59:13.241 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/34 file after the snapshot 2024-03-28T18:59:13.254 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/35 file after the snapshot 2024-03-28T18:59:13.267 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/36 file after the snapshot 2024-03-28T18:59:13.280 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/37 file after the snapshot 2024-03-28T18:59:13.292 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/38 file after the snapshot 2024-03-28T18:59:13.305 INFO:tasks.workunit.client.0.smithi027.stdout: create 199/39 file after the snapshot 2024-03-28T18:59:13.334 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/.snap/snap-subdir-test 2024-03-28T18:59:13.439 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/30 file after the snapshot 2024-03-28T18:59:13.446 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/31 file after the snapshot 2024-03-28T18:59:13.457 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/32 file after the snapshot 2024-03-28T18:59:13.469 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/33 file after the snapshot 2024-03-28T18:59:13.481 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/34 file after the snapshot 2024-03-28T18:59:13.496 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/35 file after the snapshot 2024-03-28T18:59:13.506 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/36 file after the snapshot 2024-03-28T18:59:13.518 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/37 file after the snapshot 2024-03-28T18:59:13.533 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/38 file after the snapshot 2024-03-28T18:59:13.546 INFO:tasks.workunit.client.0.smithi027.stdout: create 200/39 file after the snapshot 2024-03-28T18:59:13.558 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/.snap/snap-subdir-test 2024-03-28T18:59:13.615 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/30 file after the snapshot 2024-03-28T18:59:13.620 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/31 file after the snapshot 2024-03-28T18:59:13.632 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/32 file after the snapshot 2024-03-28T18:59:13.643 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/33 file after the snapshot 2024-03-28T18:59:13.656 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/34 file after the snapshot 2024-03-28T18:59:13.670 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/35 file after the snapshot 2024-03-28T18:59:13.682 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/36 file after the snapshot 2024-03-28T18:59:13.694 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/37 file after the snapshot 2024-03-28T18:59:13.707 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/38 file after the snapshot 2024-03-28T18:59:13.719 INFO:tasks.workunit.client.0.smithi027.stdout: create 201/39 file after the snapshot 2024-03-28T18:59:13.733 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/.snap/snap-subdir-test 2024-03-28T18:59:13.769 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/30 file after the snapshot 2024-03-28T18:59:13.776 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/31 file after the snapshot 2024-03-28T18:59:13.788 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/32 file after the snapshot 2024-03-28T18:59:13.801 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/33 file after the snapshot 2024-03-28T18:59:13.813 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/34 file after the snapshot 2024-03-28T18:59:13.826 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/35 file after the snapshot 2024-03-28T18:59:13.841 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/36 file after the snapshot 2024-03-28T18:59:13.852 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/37 file after the snapshot 2024-03-28T18:59:13.864 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/38 file after the snapshot 2024-03-28T18:59:13.874 INFO:tasks.workunit.client.0.smithi027.stdout: create 202/39 file after the snapshot 2024-03-28T18:59:13.885 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/.snap/snap-subdir-test 2024-03-28T18:59:13.921 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/30 file after the snapshot 2024-03-28T18:59:13.925 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/31 file after the snapshot 2024-03-28T18:59:13.938 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/32 file after the snapshot 2024-03-28T18:59:13.952 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/33 file after the snapshot 2024-03-28T18:59:13.963 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/34 file after the snapshot 2024-03-28T18:59:13.974 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/35 file after the snapshot 2024-03-28T18:59:13.984 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/36 file after the snapshot 2024-03-28T18:59:13.995 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/37 file after the snapshot 2024-03-28T18:59:14.008 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/38 file after the snapshot 2024-03-28T18:59:14.022 INFO:tasks.workunit.client.0.smithi027.stdout: create 203/39 file after the snapshot 2024-03-28T18:59:14.035 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/.snap/snap-subdir-test 2024-03-28T18:59:14.077 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/30 file after the snapshot 2024-03-28T18:59:14.082 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/31 file after the snapshot 2024-03-28T18:59:14.094 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/32 file after the snapshot 2024-03-28T18:59:14.107 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/33 file after the snapshot 2024-03-28T18:59:14.121 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/34 file after the snapshot 2024-03-28T18:59:14.133 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/35 file after the snapshot 2024-03-28T18:59:14.144 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/36 file after the snapshot 2024-03-28T18:59:14.155 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/37 file after the snapshot 2024-03-28T18:59:14.166 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/38 file after the snapshot 2024-03-28T18:59:14.179 INFO:tasks.workunit.client.0.smithi027.stdout: create 204/39 file after the snapshot 2024-03-28T18:59:14.192 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/.snap/snap-subdir-test 2024-03-28T18:59:14.231 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/30 file after the snapshot 2024-03-28T18:59:14.237 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/31 file after the snapshot 2024-03-28T18:59:14.250 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/32 file after the snapshot 2024-03-28T18:59:14.264 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/33 file after the snapshot 2024-03-28T18:59:14.277 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/34 file after the snapshot 2024-03-28T18:59:14.290 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/35 file after the snapshot 2024-03-28T18:59:14.303 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/36 file after the snapshot 2024-03-28T18:59:14.315 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/37 file after the snapshot 2024-03-28T18:59:14.330 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/38 file after the snapshot 2024-03-28T18:59:14.462 INFO:tasks.workunit.client.0.smithi027.stdout: create 205/39 file after the snapshot 2024-03-28T18:59:14.497 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/.snap/snap-subdir-test 2024-03-28T18:59:14.530 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/30 file after the snapshot 2024-03-28T18:59:14.536 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/31 file after the snapshot 2024-03-28T18:59:14.545 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/32 file after the snapshot 2024-03-28T18:59:14.559 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/33 file after the snapshot 2024-03-28T18:59:14.571 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/34 file after the snapshot 2024-03-28T18:59:14.582 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/35 file after the snapshot 2024-03-28T18:59:14.594 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/36 file after the snapshot 2024-03-28T18:59:14.606 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/37 file after the snapshot 2024-03-28T18:59:14.619 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/38 file after the snapshot 2024-03-28T18:59:14.632 INFO:tasks.workunit.client.0.smithi027.stdout: create 206/39 file after the snapshot 2024-03-28T18:59:14.644 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/.snap/snap-subdir-test 2024-03-28T18:59:14.686 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/30 file after the snapshot 2024-03-28T18:59:14.690 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/31 file after the snapshot 2024-03-28T18:59:14.700 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/32 file after the snapshot 2024-03-28T18:59:14.710 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/33 file after the snapshot 2024-03-28T18:59:14.722 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/34 file after the snapshot 2024-03-28T18:59:14.735 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/35 file after the snapshot 2024-03-28T18:59:14.745 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/36 file after the snapshot 2024-03-28T18:59:14.757 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/37 file after the snapshot 2024-03-28T18:59:14.770 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/38 file after the snapshot 2024-03-28T18:59:14.784 INFO:tasks.workunit.client.0.smithi027.stdout: create 207/39 file after the snapshot 2024-03-28T18:59:14.797 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/.snap/snap-subdir-test 2024-03-28T18:59:14.853 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/30 file after the snapshot 2024-03-28T18:59:14.859 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/31 file after the snapshot 2024-03-28T18:59:14.872 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/32 file after the snapshot 2024-03-28T18:59:14.886 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/33 file after the snapshot 2024-03-28T18:59:14.899 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/34 file after the snapshot 2024-03-28T18:59:14.912 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/35 file after the snapshot 2024-03-28T18:59:14.925 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/36 file after the snapshot 2024-03-28T18:59:14.939 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/37 file after the snapshot 2024-03-28T18:59:14.952 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/38 file after the snapshot 2024-03-28T18:59:14.967 INFO:tasks.workunit.client.0.smithi027.stdout: create 208/39 file after the snapshot 2024-03-28T18:59:14.983 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/.snap/snap-subdir-test 2024-03-28T18:59:15.017 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/30 file after the snapshot 2024-03-28T18:59:15.022 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/31 file after the snapshot 2024-03-28T18:59:15.037 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/32 file after the snapshot 2024-03-28T18:59:15.050 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/33 file after the snapshot 2024-03-28T18:59:15.063 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/34 file after the snapshot 2024-03-28T18:59:15.076 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/35 file after the snapshot 2024-03-28T18:59:15.087 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/36 file after the snapshot 2024-03-28T18:59:15.099 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/37 file after the snapshot 2024-03-28T18:59:15.111 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/38 file after the snapshot 2024-03-28T18:59:15.124 INFO:tasks.workunit.client.0.smithi027.stdout: create 209/39 file after the snapshot 2024-03-28T18:59:15.137 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/.snap/snap-subdir-test 2024-03-28T18:59:15.184 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/30 file after the snapshot 2024-03-28T18:59:15.189 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/31 file after the snapshot 2024-03-28T18:59:15.198 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/32 file after the snapshot 2024-03-28T18:59:15.211 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/33 file after the snapshot 2024-03-28T18:59:15.223 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/34 file after the snapshot 2024-03-28T18:59:15.235 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/35 file after the snapshot 2024-03-28T18:59:15.247 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/36 file after the snapshot 2024-03-28T18:59:15.256 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/37 file after the snapshot 2024-03-28T18:59:15.264 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/38 file after the snapshot 2024-03-28T18:59:15.272 INFO:tasks.workunit.client.0.smithi027.stdout: create 210/39 file after the snapshot 2024-03-28T18:59:15.278 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/.snap/snap-subdir-test 2024-03-28T18:59:15.295 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 251 2024-03-28T18:59:15.303 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/30 file after the snapshot 2024-03-28T18:59:15.307 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/31 file after the snapshot 2024-03-28T18:59:15.314 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/32 file after the snapshot 2024-03-28T18:59:15.321 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/33 file after the snapshot 2024-03-28T18:59:15.329 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/34 file after the snapshot 2024-03-28T18:59:15.338 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/35 file after the snapshot 2024-03-28T18:59:15.345 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/36 file after the snapshot 2024-03-28T18:59:15.355 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/37 file after the snapshot 2024-03-28T18:59:15.422 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/38 file after the snapshot 2024-03-28T18:59:15.446 INFO:tasks.workunit.client.0.smithi027.stdout: create 211/39 file after the snapshot 2024-03-28T18:59:15.455 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/.snap/snap-subdir-test 2024-03-28T18:59:15.481 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/30 file after the snapshot 2024-03-28T18:59:15.485 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 252 2024-03-28T18:59:15.486 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/31 file after the snapshot 2024-03-28T18:59:15.492 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/32 file after the snapshot 2024-03-28T18:59:15.498 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/33 file after the snapshot 2024-03-28T18:59:15.505 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/34 file after the snapshot 2024-03-28T18:59:15.511 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/35 file after the snapshot 2024-03-28T18:59:15.518 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/36 file after the snapshot 2024-03-28T18:59:15.524 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/37 file after the snapshot 2024-03-28T18:59:15.529 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/38 file after the snapshot 2024-03-28T18:59:15.534 INFO:tasks.workunit.client.0.smithi027.stdout: create 212/39 file after the snapshot 2024-03-28T18:59:15.541 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/.snap/snap-subdir-test 2024-03-28T18:59:15.562 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/30 file after the snapshot 2024-03-28T18:59:15.566 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/31 file after the snapshot 2024-03-28T18:59:15.570 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 253 2024-03-28T18:59:15.573 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/32 file after the snapshot 2024-03-28T18:59:15.579 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/33 file after the snapshot 2024-03-28T18:59:15.584 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/34 file after the snapshot 2024-03-28T18:59:15.592 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/35 file after the snapshot 2024-03-28T18:59:15.598 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/36 file after the snapshot 2024-03-28T18:59:15.605 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/37 file after the snapshot 2024-03-28T18:59:15.611 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/38 file after the snapshot 2024-03-28T18:59:15.617 INFO:tasks.workunit.client.0.smithi027.stdout: create 213/39 file after the snapshot 2024-03-28T18:59:15.626 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/.snap/snap-subdir-test 2024-03-28T18:59:15.653 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/30 file after the snapshot 2024-03-28T18:59:15.657 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/31 file after the snapshot 2024-03-28T18:59:15.660 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 254 2024-03-28T18:59:15.663 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/32 file after the snapshot 2024-03-28T18:59:15.669 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/33 file after the snapshot 2024-03-28T18:59:15.675 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/34 file after the snapshot 2024-03-28T18:59:15.682 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/35 file after the snapshot 2024-03-28T18:59:15.688 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/36 file after the snapshot 2024-03-28T18:59:15.694 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/37 file after the snapshot 2024-03-28T18:59:15.700 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/38 file after the snapshot 2024-03-28T18:59:15.707 INFO:tasks.workunit.client.0.smithi027.stdout: create 214/39 file after the snapshot 2024-03-28T18:59:15.713 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/.snap/snap-subdir-test 2024-03-28T18:59:15.735 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/30 file after the snapshot 2024-03-28T18:59:15.738 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/31 file after the snapshot 2024-03-28T18:59:15.744 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 255 2024-03-28T18:59:15.745 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/32 file after the snapshot 2024-03-28T18:59:15.751 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/33 file after the snapshot 2024-03-28T18:59:15.756 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/34 file after the snapshot 2024-03-28T18:59:15.762 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/35 file after the snapshot 2024-03-28T18:59:15.768 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/36 file after the snapshot 2024-03-28T18:59:15.774 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/37 file after the snapshot 2024-03-28T18:59:15.781 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/38 file after the snapshot 2024-03-28T18:59:15.787 INFO:tasks.workunit.client.0.smithi027.stdout: create 215/39 file after the snapshot 2024-03-28T18:59:15.793 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/.snap/snap-subdir-test 2024-03-28T18:59:15.817 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/30 file after the snapshot 2024-03-28T18:59:15.821 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/31 file after the snapshot 2024-03-28T18:59:15.828 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/32 file after the snapshot 2024-03-28T18:59:15.830 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 256 2024-03-28T18:59:15.838 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/33 file after the snapshot 2024-03-28T18:59:15.840 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/34 file after the snapshot 2024-03-28T18:59:15.846 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/35 file after the snapshot 2024-03-28T18:59:15.853 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/36 file after the snapshot 2024-03-28T18:59:15.860 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/37 file after the snapshot 2024-03-28T18:59:15.865 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/38 file after the snapshot 2024-03-28T18:59:15.872 INFO:tasks.workunit.client.0.smithi027.stdout: create 216/39 file after the snapshot 2024-03-28T18:59:15.879 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/.snap/snap-subdir-test 2024-03-28T18:59:15.902 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/30 file after the snapshot 2024-03-28T18:59:15.905 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/31 file after the snapshot 2024-03-28T18:59:15.912 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/32 file after the snapshot 2024-03-28T18:59:15.918 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/33 file after the snapshot 2024-03-28T18:59:15.923 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 257 2024-03-28T18:59:15.924 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/34 file after the snapshot 2024-03-28T18:59:15.929 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/35 file after the snapshot 2024-03-28T18:59:15.935 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/36 file after the snapshot 2024-03-28T18:59:15.942 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/37 file after the snapshot 2024-03-28T18:59:15.947 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/38 file after the snapshot 2024-03-28T18:59:15.954 INFO:tasks.workunit.client.0.smithi027.stdout: create 217/39 file after the snapshot 2024-03-28T18:59:15.961 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/.snap/snap-subdir-test 2024-03-28T18:59:15.985 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/30 file after the snapshot 2024-03-28T18:59:15.989 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/31 file after the snapshot 2024-03-28T18:59:15.997 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/32 file after the snapshot 2024-03-28T18:59:16.003 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/33 file after the snapshot 2024-03-28T18:59:16.009 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/34 file after the snapshot 2024-03-28T18:59:16.016 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/35 file after the snapshot 2024-03-28T18:59:16.019 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 258 2024-03-28T18:59:16.022 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/36 file after the snapshot 2024-03-28T18:59:16.028 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/37 file after the snapshot 2024-03-28T18:59:16.033 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/38 file after the snapshot 2024-03-28T18:59:16.040 INFO:tasks.workunit.client.0.smithi027.stdout: create 218/39 file after the snapshot 2024-03-28T18:59:16.047 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/.snap/snap-subdir-test 2024-03-28T18:59:16.069 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/30 file after the snapshot 2024-03-28T18:59:16.073 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/31 file after the snapshot 2024-03-28T18:59:16.080 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/32 file after the snapshot 2024-03-28T18:59:16.087 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/33 file after the snapshot 2024-03-28T18:59:16.093 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/34 file after the snapshot 2024-03-28T18:59:16.095 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 259 2024-03-28T18:59:16.099 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/35 file after the snapshot 2024-03-28T18:59:16.105 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/36 file after the snapshot 2024-03-28T18:59:16.112 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/37 file after the snapshot 2024-03-28T18:59:16.118 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/38 file after the snapshot 2024-03-28T18:59:16.125 INFO:tasks.workunit.client.0.smithi027.stdout: create 219/39 file after the snapshot 2024-03-28T18:59:16.133 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/.snap/snap-subdir-test 2024-03-28T18:59:16.157 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/30 file after the snapshot 2024-03-28T18:59:16.160 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/31 file after the snapshot 2024-03-28T18:59:16.167 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/32 file after the snapshot 2024-03-28T18:59:16.173 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/33 file after the snapshot 2024-03-28T18:59:16.175 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 260 2024-03-28T18:59:16.179 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/34 file after the snapshot 2024-03-28T18:59:16.185 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/35 file after the snapshot 2024-03-28T18:59:16.191 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/36 file after the snapshot 2024-03-28T18:59:16.197 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/37 file after the snapshot 2024-03-28T18:59:16.204 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/38 file after the snapshot 2024-03-28T18:59:16.210 INFO:tasks.workunit.client.0.smithi027.stdout: create 220/39 file after the snapshot 2024-03-28T18:59:16.217 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/.snap/snap-subdir-test 2024-03-28T18:59:16.241 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/30 file after the snapshot 2024-03-28T18:59:16.244 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/31 file after the snapshot 2024-03-28T18:59:16.251 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/32 file after the snapshot 2024-03-28T18:59:16.258 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/33 file after the snapshot 2024-03-28T18:59:16.262 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 261 2024-03-28T18:59:16.264 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/34 file after the snapshot 2024-03-28T18:59:16.270 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/35 file after the snapshot 2024-03-28T18:59:16.276 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/36 file after the snapshot 2024-03-28T18:59:16.282 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/37 file after the snapshot 2024-03-28T18:59:16.289 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/38 file after the snapshot 2024-03-28T18:59:16.295 INFO:tasks.workunit.client.0.smithi027.stdout: create 221/39 file after the snapshot 2024-03-28T18:59:16.301 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/.snap/snap-subdir-test 2024-03-28T18:59:16.326 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/30 file after the snapshot 2024-03-28T18:59:16.330 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/31 file after the snapshot 2024-03-28T18:59:16.337 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/32 file after the snapshot 2024-03-28T18:59:16.343 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/33 file after the snapshot 2024-03-28T18:59:16.349 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/34 file after the snapshot 2024-03-28T18:59:16.365 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/35 file after the snapshot 2024-03-28T18:59:16.366 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 262 2024-03-28T18:59:16.394 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/36 file after the snapshot 2024-03-28T18:59:16.413 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/37 file after the snapshot 2024-03-28T18:59:16.419 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/38 file after the snapshot 2024-03-28T18:59:16.425 INFO:tasks.workunit.client.0.smithi027.stdout: create 222/39 file after the snapshot 2024-03-28T18:59:16.432 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/.snap/snap-subdir-test 2024-03-28T18:59:16.442 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 263 2024-03-28T18:59:16.452 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/30 file after the snapshot 2024-03-28T18:59:16.456 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/31 file after the snapshot 2024-03-28T18:59:16.462 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/32 file after the snapshot 2024-03-28T18:59:16.469 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/33 file after the snapshot 2024-03-28T18:59:16.475 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/34 file after the snapshot 2024-03-28T18:59:16.482 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/35 file after the snapshot 2024-03-28T18:59:16.488 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/36 file after the snapshot 2024-03-28T18:59:16.494 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/37 file after the snapshot 2024-03-28T18:59:16.500 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/38 file after the snapshot 2024-03-28T18:59:16.506 INFO:tasks.workunit.client.0.smithi027.stdout: create 223/39 file after the snapshot 2024-03-28T18:59:16.512 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/.snap/snap-subdir-test 2024-03-28T18:59:16.528 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 264 2024-03-28T18:59:16.534 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/30 file after the snapshot 2024-03-28T18:59:16.538 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/31 file after the snapshot 2024-03-28T18:59:16.544 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/32 file after the snapshot 2024-03-28T18:59:16.550 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/33 file after the snapshot 2024-03-28T18:59:16.557 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/34 file after the snapshot 2024-03-28T18:59:16.563 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/35 file after the snapshot 2024-03-28T18:59:16.570 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/36 file after the snapshot 2024-03-28T18:59:16.576 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/37 file after the snapshot 2024-03-28T18:59:16.583 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/38 file after the snapshot 2024-03-28T18:59:16.590 INFO:tasks.workunit.client.0.smithi027.stdout: create 224/39 file after the snapshot 2024-03-28T18:59:16.597 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/.snap/snap-subdir-test 2024-03-28T18:59:16.620 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/30 file after the snapshot 2024-03-28T18:59:16.620 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 265 2024-03-28T18:59:16.623 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/31 file after the snapshot 2024-03-28T18:59:16.629 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/32 file after the snapshot 2024-03-28T18:59:16.636 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/33 file after the snapshot 2024-03-28T18:59:16.642 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/34 file after the snapshot 2024-03-28T18:59:16.648 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/35 file after the snapshot 2024-03-28T18:59:16.656 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/36 file after the snapshot 2024-03-28T18:59:16.662 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/37 file after the snapshot 2024-03-28T18:59:16.669 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/38 file after the snapshot 2024-03-28T18:59:16.675 INFO:tasks.workunit.client.0.smithi027.stdout: create 225/39 file after the snapshot 2024-03-28T18:59:16.681 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/.snap/snap-subdir-test 2024-03-28T18:59:16.701 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 266 2024-03-28T18:59:16.702 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/30 file after the snapshot 2024-03-28T18:59:16.705 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/31 file after the snapshot 2024-03-28T18:59:16.711 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/32 file after the snapshot 2024-03-28T18:59:16.717 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/33 file after the snapshot 2024-03-28T18:59:16.724 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/34 file after the snapshot 2024-03-28T18:59:16.729 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/35 file after the snapshot 2024-03-28T18:59:16.736 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/36 file after the snapshot 2024-03-28T18:59:16.742 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/37 file after the snapshot 2024-03-28T18:59:16.749 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/38 file after the snapshot 2024-03-28T18:59:16.755 INFO:tasks.workunit.client.0.smithi027.stdout: create 226/39 file after the snapshot 2024-03-28T18:59:16.761 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/.snap/snap-subdir-test 2024-03-28T18:59:16.780 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 267 2024-03-28T18:59:16.785 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/30 file after the snapshot 2024-03-28T18:59:16.788 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/31 file after the snapshot 2024-03-28T18:59:16.795 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/32 file after the snapshot 2024-03-28T18:59:16.803 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/33 file after the snapshot 2024-03-28T18:59:16.809 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/34 file after the snapshot 2024-03-28T18:59:16.816 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/35 file after the snapshot 2024-03-28T18:59:16.822 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/36 file after the snapshot 2024-03-28T18:59:16.828 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/37 file after the snapshot 2024-03-28T18:59:16.836 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/38 file after the snapshot 2024-03-28T18:59:16.842 INFO:tasks.workunit.client.0.smithi027.stdout: create 227/39 file after the snapshot 2024-03-28T18:59:16.848 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/.snap/snap-subdir-test 2024-03-28T18:59:16.869 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 268 2024-03-28T18:59:16.873 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/30 file after the snapshot 2024-03-28T18:59:16.876 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/31 file after the snapshot 2024-03-28T18:59:16.883 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/32 file after the snapshot 2024-03-28T18:59:16.889 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/33 file after the snapshot 2024-03-28T18:59:16.896 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/34 file after the snapshot 2024-03-28T18:59:16.903 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/35 file after the snapshot 2024-03-28T18:59:16.910 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/36 file after the snapshot 2024-03-28T18:59:16.916 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/37 file after the snapshot 2024-03-28T18:59:16.922 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/38 file after the snapshot 2024-03-28T18:59:16.929 INFO:tasks.workunit.client.0.smithi027.stdout: create 228/39 file after the snapshot 2024-03-28T18:59:16.935 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/.snap/snap-subdir-test 2024-03-28T18:59:16.946 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 269 2024-03-28T18:59:16.962 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/30 file after the snapshot 2024-03-28T18:59:16.966 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/31 file after the snapshot 2024-03-28T18:59:16.973 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/32 file after the snapshot 2024-03-28T18:59:16.980 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/33 file after the snapshot 2024-03-28T18:59:16.986 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/34 file after the snapshot 2024-03-28T18:59:16.992 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/35 file after the snapshot 2024-03-28T18:59:16.998 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/36 file after the snapshot 2024-03-28T18:59:17.004 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/37 file after the snapshot 2024-03-28T18:59:17.010 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/38 file after the snapshot 2024-03-28T18:59:17.017 INFO:tasks.workunit.client.0.smithi027.stdout: create 229/39 file after the snapshot 2024-03-28T18:59:17.022 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/.snap/snap-subdir-test 2024-03-28T18:59:17.049 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/30 file after the snapshot 2024-03-28T18:59:17.049 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 270 2024-03-28T18:59:17.053 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/31 file after the snapshot 2024-03-28T18:59:17.060 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/32 file after the snapshot 2024-03-28T18:59:17.067 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/33 file after the snapshot 2024-03-28T18:59:17.073 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/34 file after the snapshot 2024-03-28T18:59:17.079 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/35 file after the snapshot 2024-03-28T18:59:17.085 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/36 file after the snapshot 2024-03-28T18:59:17.092 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/37 file after the snapshot 2024-03-28T18:59:17.098 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/38 file after the snapshot 2024-03-28T18:59:17.105 INFO:tasks.workunit.client.0.smithi027.stdout: create 230/39 file after the snapshot 2024-03-28T18:59:17.111 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/.snap/snap-subdir-test 2024-03-28T18:59:17.137 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/30 file after the snapshot 2024-03-28T18:59:17.137 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 271 2024-03-28T18:59:17.141 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/31 file after the snapshot 2024-03-28T18:59:17.148 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/32 file after the snapshot 2024-03-28T18:59:17.154 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/33 file after the snapshot 2024-03-28T18:59:17.161 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/34 file after the snapshot 2024-03-28T18:59:17.166 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/35 file after the snapshot 2024-03-28T18:59:17.176 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/36 file after the snapshot 2024-03-28T18:59:17.182 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/37 file after the snapshot 2024-03-28T18:59:17.188 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/38 file after the snapshot 2024-03-28T18:59:17.194 INFO:tasks.workunit.client.0.smithi027.stdout: create 231/39 file after the snapshot 2024-03-28T18:59:17.200 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/.snap/snap-subdir-test 2024-03-28T18:59:17.222 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/30 file after the snapshot 2024-03-28T18:59:17.225 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/31 file after the snapshot 2024-03-28T18:59:17.232 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/32 file after the snapshot 2024-03-28T18:59:17.238 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/33 file after the snapshot 2024-03-28T18:59:17.239 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 272 2024-03-28T18:59:17.244 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/34 file after the snapshot 2024-03-28T18:59:17.250 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/35 file after the snapshot 2024-03-28T18:59:17.257 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/36 file after the snapshot 2024-03-28T18:59:17.263 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/37 file after the snapshot 2024-03-28T18:59:17.270 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/38 file after the snapshot 2024-03-28T18:59:17.276 INFO:tasks.workunit.client.0.smithi027.stdout: create 232/39 file after the snapshot 2024-03-28T18:59:17.283 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/.snap/snap-subdir-test 2024-03-28T18:59:17.305 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/30 file after the snapshot 2024-03-28T18:59:17.309 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/31 file after the snapshot 2024-03-28T18:59:17.315 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/32 file after the snapshot 2024-03-28T18:59:17.321 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/33 file after the snapshot 2024-03-28T18:59:17.328 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/34 file after the snapshot 2024-03-28T18:59:17.334 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/35 file after the snapshot 2024-03-28T18:59:17.335 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 273 2024-03-28T18:59:17.340 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/36 file after the snapshot 2024-03-28T18:59:17.348 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/37 file after the snapshot 2024-03-28T18:59:17.355 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/38 file after the snapshot 2024-03-28T18:59:17.361 INFO:tasks.workunit.client.0.smithi027.stdout: create 233/39 file after the snapshot 2024-03-28T18:59:17.370 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/.snap/snap-subdir-test 2024-03-28T18:59:17.425 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/30 file after the snapshot 2024-03-28T18:59:17.428 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/31 file after the snapshot 2024-03-28T18:59:17.435 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/32 file after the snapshot 2024-03-28T18:59:17.438 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 274 2024-03-28T18:59:17.441 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/33 file after the snapshot 2024-03-28T18:59:17.448 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/34 file after the snapshot 2024-03-28T18:59:17.454 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/35 file after the snapshot 2024-03-28T18:59:17.461 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/36 file after the snapshot 2024-03-28T18:59:17.468 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/37 file after the snapshot 2024-03-28T18:59:17.475 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/38 file after the snapshot 2024-03-28T18:59:17.482 INFO:tasks.workunit.client.0.smithi027.stdout: create 234/39 file after the snapshot 2024-03-28T18:59:17.488 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/.snap/snap-subdir-test 2024-03-28T18:59:17.512 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/30 file after the snapshot 2024-03-28T18:59:17.517 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/31 file after the snapshot 2024-03-28T18:59:17.517 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 275 2024-03-28T18:59:17.523 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/32 file after the snapshot 2024-03-28T18:59:17.528 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/33 file after the snapshot 2024-03-28T18:59:17.535 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/34 file after the snapshot 2024-03-28T18:59:17.543 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/35 file after the snapshot 2024-03-28T18:59:17.548 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/36 file after the snapshot 2024-03-28T18:59:17.554 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/37 file after the snapshot 2024-03-28T18:59:17.561 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/38 file after the snapshot 2024-03-28T18:59:17.568 INFO:tasks.workunit.client.0.smithi027.stdout: create 235/39 file after the snapshot 2024-03-28T18:59:17.574 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/.snap/snap-subdir-test 2024-03-28T18:59:17.596 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/30 file after the snapshot 2024-03-28T18:59:17.600 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/31 file after the snapshot 2024-03-28T18:59:17.601 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 276 2024-03-28T18:59:17.606 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/32 file after the snapshot 2024-03-28T18:59:17.612 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/33 file after the snapshot 2024-03-28T18:59:17.618 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/34 file after the snapshot 2024-03-28T18:59:17.624 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/35 file after the snapshot 2024-03-28T18:59:17.632 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/36 file after the snapshot 2024-03-28T18:59:17.639 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/37 file after the snapshot 2024-03-28T18:59:17.646 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/38 file after the snapshot 2024-03-28T18:59:17.654 INFO:tasks.workunit.client.0.smithi027.stdout: create 236/39 file after the snapshot 2024-03-28T18:59:17.660 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/.snap/snap-subdir-test 2024-03-28T18:59:17.686 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/30 file after the snapshot 2024-03-28T18:59:17.689 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/31 file after the snapshot 2024-03-28T18:59:17.696 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/32 file after the snapshot 2024-03-28T18:59:17.699 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 277 2024-03-28T18:59:17.703 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/33 file after the snapshot 2024-03-28T18:59:17.710 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/34 file after the snapshot 2024-03-28T18:59:17.718 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/35 file after the snapshot 2024-03-28T18:59:17.725 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/36 file after the snapshot 2024-03-28T18:59:17.733 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/37 file after the snapshot 2024-03-28T18:59:17.742 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/38 file after the snapshot 2024-03-28T18:59:17.750 INFO:tasks.workunit.client.0.smithi027.stdout: create 237/39 file after the snapshot 2024-03-28T18:59:17.757 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/.snap/snap-subdir-test 2024-03-28T18:59:17.780 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 278 2024-03-28T18:59:17.782 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/30 file after the snapshot 2024-03-28T18:59:17.786 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/31 file after the snapshot 2024-03-28T18:59:17.793 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/32 file after the snapshot 2024-03-28T18:59:17.799 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/33 file after the snapshot 2024-03-28T18:59:17.805 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/34 file after the snapshot 2024-03-28T18:59:17.813 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/35 file after the snapshot 2024-03-28T18:59:17.819 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/36 file after the snapshot 2024-03-28T18:59:17.825 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/37 file after the snapshot 2024-03-28T18:59:17.831 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/38 file after the snapshot 2024-03-28T18:59:17.838 INFO:tasks.workunit.client.0.smithi027.stdout: create 238/39 file after the snapshot 2024-03-28T18:59:17.845 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/.snap/snap-subdir-test 2024-03-28T18:59:17.861 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 279 2024-03-28T18:59:17.871 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/30 file after the snapshot 2024-03-28T18:59:17.876 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/31 file after the snapshot 2024-03-28T18:59:17.882 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/32 file after the snapshot 2024-03-28T18:59:17.887 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/33 file after the snapshot 2024-03-28T18:59:17.894 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/34 file after the snapshot 2024-03-28T18:59:17.900 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/35 file after the snapshot 2024-03-28T18:59:17.907 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/36 file after the snapshot 2024-03-28T18:59:17.913 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/37 file after the snapshot 2024-03-28T18:59:17.919 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/38 file after the snapshot 2024-03-28T18:59:17.925 INFO:tasks.workunit.client.0.smithi027.stdout: create 239/39 file after the snapshot 2024-03-28T18:59:17.932 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/.snap/snap-subdir-test 2024-03-28T18:59:17.936 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 280 2024-03-28T18:59:17.955 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/30 file after the snapshot 2024-03-28T18:59:17.959 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/31 file after the snapshot 2024-03-28T18:59:17.966 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/32 file after the snapshot 2024-03-28T18:59:17.972 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/33 file after the snapshot 2024-03-28T18:59:17.978 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/34 file after the snapshot 2024-03-28T18:59:17.985 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/35 file after the snapshot 2024-03-28T18:59:17.991 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/36 file after the snapshot 2024-03-28T18:59:17.997 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/37 file after the snapshot 2024-03-28T18:59:18.003 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/38 file after the snapshot 2024-03-28T18:59:18.009 INFO:tasks.workunit.client.0.smithi027.stdout: create 240/39 file after the snapshot 2024-03-28T18:59:18.015 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/.snap/snap-subdir-test 2024-03-28T18:59:18.018 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 281 2024-03-28T18:59:18.041 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/30 file after the snapshot 2024-03-28T18:59:18.044 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/31 file after the snapshot 2024-03-28T18:59:18.051 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/32 file after the snapshot 2024-03-28T18:59:18.058 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/33 file after the snapshot 2024-03-28T18:59:18.066 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/34 file after the snapshot 2024-03-28T18:59:18.072 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/35 file after the snapshot 2024-03-28T18:59:18.079 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/36 file after the snapshot 2024-03-28T18:59:18.085 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/37 file after the snapshot 2024-03-28T18:59:18.091 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/38 file after the snapshot 2024-03-28T18:59:18.097 INFO:tasks.workunit.client.0.smithi027.stdout: create 241/39 file after the snapshot 2024-03-28T18:59:18.103 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/.snap/snap-subdir-test 2024-03-28T18:59:18.116 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 282 2024-03-28T18:59:18.125 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/30 file after the snapshot 2024-03-28T18:59:18.129 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/31 file after the snapshot 2024-03-28T18:59:18.136 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/32 file after the snapshot 2024-03-28T18:59:18.143 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/33 file after the snapshot 2024-03-28T18:59:18.151 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/34 file after the snapshot 2024-03-28T18:59:18.158 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/35 file after the snapshot 2024-03-28T18:59:18.164 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/36 file after the snapshot 2024-03-28T18:59:18.170 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/37 file after the snapshot 2024-03-28T18:59:18.176 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/38 file after the snapshot 2024-03-28T18:59:18.183 INFO:tasks.workunit.client.0.smithi027.stdout: create 242/39 file after the snapshot 2024-03-28T18:59:18.190 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/.snap/snap-subdir-test 2024-03-28T18:59:18.206 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 283 2024-03-28T18:59:18.210 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/30 file after the snapshot 2024-03-28T18:59:18.215 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/31 file after the snapshot 2024-03-28T18:59:18.221 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/32 file after the snapshot 2024-03-28T18:59:18.227 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/33 file after the snapshot 2024-03-28T18:59:18.234 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/34 file after the snapshot 2024-03-28T18:59:18.240 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/35 file after the snapshot 2024-03-28T18:59:18.247 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/36 file after the snapshot 2024-03-28T18:59:18.254 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/37 file after the snapshot 2024-03-28T18:59:18.260 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/38 file after the snapshot 2024-03-28T18:59:18.267 INFO:tasks.workunit.client.0.smithi027.stdout: create 243/39 file after the snapshot 2024-03-28T18:59:18.273 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/.snap/snap-subdir-test 2024-03-28T18:59:18.288 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 284 2024-03-28T18:59:18.298 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/30 file after the snapshot 2024-03-28T18:59:18.302 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/31 file after the snapshot 2024-03-28T18:59:18.309 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/32 file after the snapshot 2024-03-28T18:59:18.316 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/33 file after the snapshot 2024-03-28T18:59:18.322 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/34 file after the snapshot 2024-03-28T18:59:18.329 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/35 file after the snapshot 2024-03-28T18:59:18.335 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/36 file after the snapshot 2024-03-28T18:59:18.341 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/37 file after the snapshot 2024-03-28T18:59:18.348 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/38 file after the snapshot 2024-03-28T18:59:18.355 INFO:tasks.workunit.client.0.smithi027.stdout: create 244/39 file after the snapshot 2024-03-28T18:59:18.360 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/.snap/snap-subdir-test 2024-03-28T18:59:18.382 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 285 2024-03-28T18:59:18.389 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/30 file after the snapshot 2024-03-28T18:59:18.419 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/31 file after the snapshot 2024-03-28T18:59:18.444 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/32 file after the snapshot 2024-03-28T18:59:18.452 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/33 file after the snapshot 2024-03-28T18:59:18.459 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/34 file after the snapshot 2024-03-28T18:59:18.464 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 286 2024-03-28T18:59:18.465 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/35 file after the snapshot 2024-03-28T18:59:18.472 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/36 file after the snapshot 2024-03-28T18:59:18.478 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/37 file after the snapshot 2024-03-28T18:59:18.485 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/38 file after the snapshot 2024-03-28T18:59:18.492 INFO:tasks.workunit.client.0.smithi027.stdout: create 245/39 file after the snapshot 2024-03-28T18:59:18.499 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/.snap/snap-subdir-test 2024-03-28T18:59:18.522 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/30 file after the snapshot 2024-03-28T18:59:18.525 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/31 file after the snapshot 2024-03-28T18:59:18.532 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/32 file after the snapshot 2024-03-28T18:59:18.538 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/33 file after the snapshot 2024-03-28T18:59:18.544 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/34 file after the snapshot 2024-03-28T18:59:18.545 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 287 2024-03-28T18:59:18.550 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/35 file after the snapshot 2024-03-28T18:59:18.555 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/36 file after the snapshot 2024-03-28T18:59:18.561 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/37 file after the snapshot 2024-03-28T18:59:18.566 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/38 file after the snapshot 2024-03-28T18:59:18.573 INFO:tasks.workunit.client.0.smithi027.stdout: create 246/39 file after the snapshot 2024-03-28T18:59:18.580 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/.snap/snap-subdir-test 2024-03-28T18:59:18.602 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/30 file after the snapshot 2024-03-28T18:59:18.605 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/31 file after the snapshot 2024-03-28T18:59:18.613 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/32 file after the snapshot 2024-03-28T18:59:18.620 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/33 file after the snapshot 2024-03-28T18:59:18.622 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 288 2024-03-28T18:59:18.626 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/34 file after the snapshot 2024-03-28T18:59:18.632 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/35 file after the snapshot 2024-03-28T18:59:18.638 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/36 file after the snapshot 2024-03-28T18:59:18.645 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/37 file after the snapshot 2024-03-28T18:59:18.652 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/38 file after the snapshot 2024-03-28T18:59:18.660 INFO:tasks.workunit.client.0.smithi027.stdout: create 247/39 file after the snapshot 2024-03-28T18:59:18.666 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/.snap/snap-subdir-test 2024-03-28T18:59:18.690 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/30 file after the snapshot 2024-03-28T18:59:18.693 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/31 file after the snapshot 2024-03-28T18:59:18.701 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/32 file after the snapshot 2024-03-28T18:59:18.704 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 289 2024-03-28T18:59:18.708 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/33 file after the snapshot 2024-03-28T18:59:18.714 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/34 file after the snapshot 2024-03-28T18:59:18.721 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/35 file after the snapshot 2024-03-28T18:59:18.728 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/36 file after the snapshot 2024-03-28T18:59:18.738 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/37 file after the snapshot 2024-03-28T18:59:18.745 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/38 file after the snapshot 2024-03-28T18:59:18.751 INFO:tasks.workunit.client.0.smithi027.stdout: create 248/39 file after the snapshot 2024-03-28T18:59:18.758 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/.snap/snap-subdir-test 2024-03-28T18:59:18.779 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 290 2024-03-28T18:59:18.782 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/30 file after the snapshot 2024-03-28T18:59:18.785 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/31 file after the snapshot 2024-03-28T18:59:18.793 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/32 file after the snapshot 2024-03-28T18:59:18.800 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/33 file after the snapshot 2024-03-28T18:59:18.806 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/34 file after the snapshot 2024-03-28T18:59:18.812 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/35 file after the snapshot 2024-03-28T18:59:18.819 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/36 file after the snapshot 2024-03-28T18:59:18.826 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/37 file after the snapshot 2024-03-28T18:59:18.833 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/38 file after the snapshot 2024-03-28T18:59:18.840 INFO:tasks.workunit.client.0.smithi027.stdout: create 249/39 file after the snapshot 2024-03-28T18:59:18.847 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/.snap/snap-subdir-test 2024-03-28T18:59:18.850 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 291 2024-03-28T18:59:18.867 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/30 file after the snapshot 2024-03-28T18:59:18.871 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/31 file after the snapshot 2024-03-28T18:59:18.879 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/32 file after the snapshot 2024-03-28T18:59:18.886 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/33 file after the snapshot 2024-03-28T18:59:18.893 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/34 file after the snapshot 2024-03-28T18:59:18.899 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/35 file after the snapshot 2024-03-28T18:59:18.906 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/36 file after the snapshot 2024-03-28T18:59:18.912 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/37 file after the snapshot 2024-03-28T18:59:18.918 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 292 2024-03-28T18:59:18.919 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/38 file after the snapshot 2024-03-28T18:59:18.925 INFO:tasks.workunit.client.0.smithi027.stdout: create 250/39 file after the snapshot 2024-03-28T18:59:18.934 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/.snap/snap-subdir-test 2024-03-28T18:59:18.961 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/30 file after the snapshot 2024-03-28T18:59:18.964 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/31 file after the snapshot 2024-03-28T18:59:18.971 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/32 file after the snapshot 2024-03-28T18:59:18.980 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/33 file after the snapshot 2024-03-28T18:59:18.988 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/34 file after the snapshot 2024-03-28T18:59:18.990 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 293 2024-03-28T18:59:18.994 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/35 file after the snapshot 2024-03-28T18:59:19.000 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/36 file after the snapshot 2024-03-28T18:59:19.007 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/37 file after the snapshot 2024-03-28T18:59:19.013 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/38 file after the snapshot 2024-03-28T18:59:19.021 INFO:tasks.workunit.client.0.smithi027.stdout: create 251/39 file after the snapshot 2024-03-28T18:59:19.028 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/.snap/snap-subdir-test 2024-03-28T18:59:19.051 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/30 file after the snapshot 2024-03-28T18:59:19.055 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/31 file after the snapshot 2024-03-28T18:59:19.063 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/32 file after the snapshot 2024-03-28T18:59:19.067 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 294 2024-03-28T18:59:19.070 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/33 file after the snapshot 2024-03-28T18:59:19.076 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/34 file after the snapshot 2024-03-28T18:59:19.083 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/35 file after the snapshot 2024-03-28T18:59:19.091 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/36 file after the snapshot 2024-03-28T18:59:19.097 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/37 file after the snapshot 2024-03-28T18:59:19.103 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/38 file after the snapshot 2024-03-28T18:59:19.110 INFO:tasks.workunit.client.0.smithi027.stdout: create 252/39 file after the snapshot 2024-03-28T18:59:19.116 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/.snap/snap-subdir-test 2024-03-28T18:59:19.140 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 295 2024-03-28T18:59:19.140 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/30 file after the snapshot 2024-03-28T18:59:19.144 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/31 file after the snapshot 2024-03-28T18:59:19.151 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/32 file after the snapshot 2024-03-28T18:59:19.158 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/33 file after the snapshot 2024-03-28T18:59:19.164 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/34 file after the snapshot 2024-03-28T18:59:19.170 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/35 file after the snapshot 2024-03-28T18:59:19.177 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/36 file after the snapshot 2024-03-28T18:59:19.184 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/37 file after the snapshot 2024-03-28T18:59:19.189 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-03-28T18:59:19.189 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T18:59:19.189 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T18:59:19.195 INFO:tasks.ceph.mds.a:Started 2024-03-28T18:59:19.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:19.196 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/38 file after the snapshot 2024-03-28T18:59:19.201 INFO:tasks.workunit.client.0.smithi027.stdout: create 253/39 file after the snapshot 2024-03-28T18:59:19.208 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/.snap/snap-subdir-test 2024-03-28T18:59:19.211 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 296 2024-03-28T18:59:19.231 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/30 file after the snapshot 2024-03-28T18:59:19.236 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/31 file after the snapshot 2024-03-28T18:59:19.243 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/32 file after the snapshot 2024-03-28T18:59:19.250 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/33 file after the snapshot 2024-03-28T18:59:19.258 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/34 file after the snapshot 2024-03-28T18:59:19.266 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/35 file after the snapshot 2024-03-28T18:59:19.280 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/36 file after the snapshot 2024-03-28T18:59:19.286 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/37 file after the snapshot 2024-03-28T18:59:19.293 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 297 2024-03-28T18:59:19.295 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/38 file after the snapshot 2024-03-28T18:59:19.302 INFO:tasks.workunit.client.0.smithi027.stdout: create 254/39 file after the snapshot 2024-03-28T18:59:19.310 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/.snap/snap-subdir-test 2024-03-28T18:59:19.336 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/30 file after the snapshot 2024-03-28T18:59:19.341 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/31 file after the snapshot 2024-03-28T18:59:19.349 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/32 file after the snapshot 2024-03-28T18:59:19.355 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/33 file after the snapshot 2024-03-28T18:59:19.363 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/34 file after the snapshot 2024-03-28T18:59:19.371 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/35 file after the snapshot 2024-03-28T18:59:19.376 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 298 2024-03-28T18:59:19.384 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/36 file after the snapshot 2024-03-28T18:59:19.413 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/37 file after the snapshot 2024-03-28T18:59:19.418 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T18:59:19.432 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/38 file after the snapshot 2024-03-28T18:59:19.440 INFO:tasks.workunit.client.0.smithi027.stdout: create 255/39 file after the snapshot 2024-03-28T18:59:19.448 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/.snap/snap-subdir-test 2024-03-28T18:59:19.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.449+0000 7f05f6c00640 1 Processor -- start 2024-03-28T18:59:19.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.450+0000 7f05f6c00640 1 -- start start 2024-03-28T18:59:19.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.451+0000 7f05f6c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f0157150 0x7f05f00c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:19.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.451+0000 7f05f6c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f05f00c6ee0 0x7f05f00c3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:19.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.451+0000 7f05f6c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f0157530 con 0x7f05f00c1390 2024-03-28T18:59:19.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.451+0000 7f05f6c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f006e1e0 con 0x7f05f00c6ee0 2024-03-28T18:59:19.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.451+0000 7f05f6c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f006dbd0 con 0x7f05f0157150 2024-03-28T18:59:19.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.451+0000 7f05f4e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f05f00c1390 0x7f05f00c3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37022/0 (socket says 172.21.15.27:37022) 2024-03-28T18:59:19.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.451+0000 7f05f4e00640 1 -- 172.21.15.27:0/3239120842 learned_addr learned my addr 172.21.15.27:0/3239120842 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:19.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.451+0000 7f05f6200640 1 --2- 172.21.15.27:0/3239120842 >> v2:172.21.15.155:3300/0 conn(0x7f05f00c6ee0 0x7f05f00c3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:19.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.452+0000 7f05f5800640 1 --2- 172.21.15.27:0/3239120842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f0157150 0x7f05f00c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:19.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.452+0000 7f05f6200640 1 -- 172.21.15.27:0/3239120842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f0157150 msgr2=0x7f05f00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:19.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.452+0000 7f05f6200640 1 --2- 172.21.15.27:0/3239120842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f0157150 0x7f05f00c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.452+0000 7f05f6200640 1 -- 172.21.15.27:0/3239120842 >> v1:172.21.15.27:6789/0 conn(0x7f05f00c1390 legacy=0x7f05f00c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:19.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.452+0000 7f05f6200640 1 -- 172.21.15.27:0/3239120842 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f015de60 con 0x7f05f00c6ee0 2024-03-28T18:59:19.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.454+0000 7f05f6200640 1 --2- 172.21.15.27:0/3239120842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c6ee0 0x7f05f00c3880 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009d90 tx=0x7f05e802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2161bfb2ab94199 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:19.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.455+0000 7f05efe00640 1 -- 172.21.15.27:0/3239120842 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8039070 con 0x7f05f00c6ee0 2024-03-28T18:59:19.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.455+0000 7f05efe00640 1 -- 172.21.15.27:0/3239120842 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e802cc60 con 0x7f05f00c6ee0 2024-03-28T18:59:19.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.455+0000 7f05efe00640 1 -- 172.21.15.27:0/3239120842 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8034660 con 0x7f05f00c6ee0 2024-03-28T18:59:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.456+0000 7f05f6c00640 1 -- 172.21.15.27:0/3239120842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c6ee0 msgr2=0x7f05f00c3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.456+0000 7f05f6c00640 1 --2- 172.21.15.27:0/3239120842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c6ee0 0x7f05f00c3880 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009d90 tx=0x7f05e802beb0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.456+0000 7f05f6c00640 1 -- 172.21.15.27:0/3239120842 shutdown_connections 2024-03-28T18:59:19.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.456+0000 7f05f6c00640 1 --2- 172.21.15.27:0/3239120842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c6ee0 0x7f05f00c3880 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.456+0000 7f05f6c00640 1 --2- 172.21.15.27:0/3239120842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f0157150 0x7f05f00c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.456+0000 7f05f6c00640 1 -- 172.21.15.27:0/3239120842 >> 172.21.15.27:0/3239120842 conn(0x7f05f006b0c0 msgr2=0x7f05f006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:19.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.456+0000 7f05f6c00640 1 -- 172.21.15.27:0/3239120842 shutdown_connections 2024-03-28T18:59:19.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.456+0000 7f05f6c00640 1 -- 172.21.15.27:0/3239120842 wait complete. 2024-03-28T18:59:19.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.458+0000 7f05f6c00640 1 Processor -- start 2024-03-28T18:59:19.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.458+0000 7f05f6c00640 1 -- start start 2024-03-28T18:59:19.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f6c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c1390 0x7f05f01efd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:19.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f6c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f00c6ee0 0x7f05f0157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:19.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f6c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05f0157150 0x7f05f00a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f00c6ee0 0x7f05f0157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f6c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00c3dc0 con 0x7f05f0157150 2024-03-28T18:59:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f6c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f006e710 con 0x7f05f00c1390 2024-03-28T18:59:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f00c6ee0 0x7f05f0157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:41600/0 (socket says 172.21.15.27:41600) 2024-03-28T18:59:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f4e00640 1 -- 172.21.15.27:0/244764788 learned_addr learned my addr 172.21.15.27:0/244764788 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.460+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00c0fc0 con 0x7f05f00c6ee0 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f6200640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05f0157150 0x7f05f00a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f4e00640 1 -- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c1390 msgr2=0x7f05f01efd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f5800640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c1390 0x7f05f01efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f4e00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c1390 0x7f05f01efd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f4e00640 1 -- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05f0157150 msgr2=0x7f05f00a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f4e00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05f0157150 0x7f05f00a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f4e00640 1 -- 172.21.15.27:0/244764788 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f00a6080 con 0x7f05f00c6ee0 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f6200640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05f0157150 0x7f05f00a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.461+0000 7f05f5800640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c1390 0x7f05f01efd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:59:19.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.462+0000 7f05f4e00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f00c6ee0 0x7f05f0157950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00c470 tx=0x7f05dc001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:19.463 INFO:tasks.workunit.client.1.smithi155.stdout: create dir 299 2024-03-28T18:59:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.462+0000 7f05ed600640 1 -- 172.21.15.27:0/244764788 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc017070 con 0x7f05f00c6ee0 2024-03-28T18:59:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.462+0000 7f05ed600640 1 -- 172.21.15.27:0/244764788 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05dc005ce0 con 0x7f05f00c6ee0 2024-03-28T18:59:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.462+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f01f0260 con 0x7f05f00c6ee0 2024-03-28T18:59:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.462+0000 7f05ed600640 1 -- 172.21.15.27:0/244764788 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0072f0 con 0x7f05f00c6ee0 2024-03-28T18:59:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.463+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f00abba0 con 0x7f05f00c6ee0 2024-03-28T18:59:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.464+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f0067dd0 con 0x7f05f00c6ee0 2024-03-28T18:59:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.465+0000 7f05ed600640 1 -- 172.21.15.27:0/244764788 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f05dc01a070 con 0x7f05f00c6ee0 2024-03-28T18:59:19.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.467+0000 7f05ed600640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f05d006bc20 0x7f05d006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:19.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.467+0000 7f05ed600640 1 -- 172.21.15.27:0/244764788 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f05dc08a2d0 con 0x7f05f00c6ee0 2024-03-28T18:59:19.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.467+0000 7f05f5800640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f05d006bc20 0x7f05d006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:19.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.469+0000 7f05f5800640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f05d006bc20 0x7f05d006e0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009a20 tx=0x7f05e4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:19.472 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/30 file after the snapshot 2024-03-28T18:59:19.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.473+0000 7f05ed600640 1 -- 172.21.15.27:0/244764788 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f05dc058990 con 0x7f05f00c6ee0 2024-03-28T18:59:19.476 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/31 file after the snapshot 2024-03-28T18:59:19.484 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/32 file after the snapshot 2024-03-28T18:59:19.490 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/33 file after the snapshot 2024-03-28T18:59:19.497 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/34 file after the snapshot 2024-03-28T18:59:19.503 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/35 file after the snapshot 2024-03-28T18:59:19.510 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/36 file after the snapshot 2024-03-28T18:59:19.517 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/37 file after the snapshot 2024-03-28T18:59:19.524 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/38 file after the snapshot 2024-03-28T18:59:19.531 INFO:tasks.workunit.client.0.smithi027.stdout: create 256/39 file after the snapshot 2024-03-28T18:59:19.539 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/.snap/snap-subdir-test 2024-03-28T18:59:19.541 INFO:tasks.workunit.client.1.smithi155.stdout:Create a snapshot in every first level dir ... 2024-03-28T18:59:19.558 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/30 file after the snapshot 2024-03-28T18:59:19.562 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/31 file after the snapshot 2024-03-28T18:59:19.568 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/.snap/snap-subdir-test 2024-03-28T18:59:19.568 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/32 file after the snapshot 2024-03-28T18:59:19.575 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/33 file after the snapshot 2024-03-28T18:59:19.583 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/34 file after the snapshot 2024-03-28T18:59:19.588 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/35 file after the snapshot 2024-03-28T18:59:19.593 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/30 file after the snapshot 2024-03-28T18:59:19.595 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/36 file after the snapshot 2024-03-28T18:59:19.596 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/31 file after the snapshot 2024-03-28T18:59:19.602 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/37 file after the snapshot 2024-03-28T18:59:19.604 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/32 file after the snapshot 2024-03-28T18:59:19.609 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/38 file after the snapshot 2024-03-28T18:59:19.611 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/33 file after the snapshot 2024-03-28T18:59:19.616 INFO:tasks.workunit.client.0.smithi027.stdout: create 257/39 file after the snapshot 2024-03-28T18:59:19.620 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/34 file after the snapshot 2024-03-28T18:59:19.625 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/.snap/snap-subdir-test 2024-03-28T18:59:19.626 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/35 file after the snapshot 2024-03-28T18:59:19.632 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/36 file after the snapshot 2024-03-28T18:59:19.638 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/37 file after the snapshot 2024-03-28T18:59:19.645 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/38 file after the snapshot 2024-03-28T18:59:19.650 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/30 file after the snapshot 2024-03-28T18:59:19.651 INFO:tasks.workunit.client.1.smithi155.stdout: create 100/39 file after the snapshot 2024-03-28T18:59:19.656 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/31 file after the snapshot 2024-03-28T18:59:19.661 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/.snap/snap-subdir-test 2024-03-28T18:59:19.663 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/32 file after the snapshot 2024-03-28T18:59:19.670 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/33 file after the snapshot 2024-03-28T18:59:19.677 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/34 file after the snapshot 2024-03-28T18:59:19.683 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/35 file after the snapshot 2024-03-28T18:59:19.687 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/30 file after the snapshot 2024-03-28T18:59:19.689 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/36 file after the snapshot 2024-03-28T18:59:19.690 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/31 file after the snapshot 2024-03-28T18:59:19.696 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/37 file after the snapshot 2024-03-28T18:59:19.698 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/32 file after the snapshot 2024-03-28T18:59:19.703 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/38 file after the snapshot 2024-03-28T18:59:19.705 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/33 file after the snapshot 2024-03-28T18:59:19.710 INFO:tasks.workunit.client.0.smithi027.stdout: create 258/39 file after the snapshot 2024-03-28T18:59:19.712 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/34 file after the snapshot 2024-03-28T18:59:19.718 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/.snap/snap-subdir-test 2024-03-28T18:59:19.720 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/35 file after the snapshot 2024-03-28T18:59:19.726 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/36 file after the snapshot 2024-03-28T18:59:19.733 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/37 file after the snapshot 2024-03-28T18:59:19.740 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/30 file after the snapshot 2024-03-28T18:59:19.741 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/38 file after the snapshot 2024-03-28T18:59:19.744 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/31 file after the snapshot 2024-03-28T18:59:19.748 INFO:tasks.workunit.client.1.smithi155.stdout: create 101/39 file after the snapshot 2024-03-28T18:59:19.751 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/32 file after the snapshot 2024-03-28T18:59:19.755 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/.snap/snap-subdir-test 2024-03-28T18:59:19.757 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/33 file after the snapshot 2024-03-28T18:59:19.764 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/34 file after the snapshot 2024-03-28T18:59:19.770 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/35 file after the snapshot 2024-03-28T18:59:19.777 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/36 file after the snapshot 2024-03-28T18:59:19.779 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/30 file after the snapshot 2024-03-28T18:59:19.782 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/31 file after the snapshot 2024-03-28T18:59:19.784 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/37 file after the snapshot 2024-03-28T18:59:19.790 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/32 file after the snapshot 2024-03-28T18:59:19.794 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/38 file after the snapshot 2024-03-28T18:59:19.797 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/33 file after the snapshot 2024-03-28T18:59:19.801 INFO:tasks.workunit.client.0.smithi027.stdout: create 259/39 file after the snapshot 2024-03-28T18:59:19.803 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/34 file after the snapshot 2024-03-28T18:59:19.808 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/.snap/snap-subdir-test 2024-03-28T18:59:19.810 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/35 file after the snapshot 2024-03-28T18:59:19.817 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/36 file after the snapshot 2024-03-28T18:59:19.823 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/37 file after the snapshot 2024-03-28T18:59:19.829 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/30 file after the snapshot 2024-03-28T18:59:19.830 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/38 file after the snapshot 2024-03-28T18:59:19.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.833+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05f0062500 con 0x7f05f00c6ee0 2024-03-28T18:59:19.833 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/31 file after the snapshot 2024-03-28T18:59:19.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.834+0000 7f05ed600640 1 -- 172.21.15.27:0/244764788 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f05f0062500 con 0x7f05f00c6ee0 2024-03-28T18:59:19.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:19.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:19.834 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.837+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f05d006bc20 msgr2=0x7f05d006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.837+0000 7f05f6c00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f05d006bc20 0x7f05d006e0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f05e4009a20 tx=0x7f05e4008040 comp rx=0 tx=0).stop 2024-03-28T18:59:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.837+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f00c6ee0 msgr2=0x7f05f0157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:19.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.837+0000 7f05f6c00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f00c6ee0 0x7f05f0157950 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00c470 tx=0x7f05dc001e70 comp rx=0 tx=0).stop 2024-03-28T18:59:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.838+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 shutdown_connections 2024-03-28T18:59:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.838+0000 7f05f6c00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f05d006bc20 0x7f05d006e0e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.838+0000 7f05f6c00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05f0157150 0x7f05f00a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.838+0000 7f05f6c00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05f00c6ee0 0x7f05f0157950 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.838+0000 7f05f6c00640 1 --2- 172.21.15.27:0/244764788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f00c1390 0x7f05f01efd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.838+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 >> 172.21.15.27:0/244764788 conn(0x7f05f006b0c0 msgr2=0x7f05f00c2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.838+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 shutdown_connections 2024-03-28T18:59:19.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:19.838+0000 7f05f6c00640 1 -- 172.21.15.27:0/244764788 wait complete. 2024-03-28T18:59:19.839 INFO:tasks.workunit.client.1.smithi155.stdout: create 102/39 file after the snapshot 2024-03-28T18:59:19.840 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/32 file after the snapshot 2024-03-28T18:59:19.845 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/.snap/snap-subdir-test 2024-03-28T18:59:19.848 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/33 file after the snapshot 2024-03-28T18:59:19.856 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/34 file after the snapshot 2024-03-28T18:59:19.859 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-03-28T18:59:19.859 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2024-03-28T18:59:19.863 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/35 file after the snapshot 2024-03-28T18:59:19.867 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/30 file after the snapshot 2024-03-28T18:59:19.869 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/36 file after the snapshot 2024-03-28T18:59:19.870 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/31 file after the snapshot 2024-03-28T18:59:19.876 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/37 file after the snapshot 2024-03-28T18:59:19.877 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/32 file after the snapshot 2024-03-28T18:59:19.883 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/38 file after the snapshot 2024-03-28T18:59:19.885 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/33 file after the snapshot 2024-03-28T18:59:19.889 INFO:tasks.workunit.client.0.smithi027.stdout: create 260/39 file after the snapshot 2024-03-28T18:59:19.891 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/34 file after the snapshot 2024-03-28T18:59:19.896 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/.snap/snap-subdir-test 2024-03-28T18:59:19.898 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/35 file after the snapshot 2024-03-28T18:59:19.905 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/36 file after the snapshot 2024-03-28T18:59:19.912 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/37 file after the snapshot 2024-03-28T18:59:19.919 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/30 file after the snapshot 2024-03-28T18:59:19.919 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/38 file after the snapshot 2024-03-28T18:59:19.923 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/31 file after the snapshot 2024-03-28T18:59:19.923 INFO:tasks.workunit.client.1.smithi155.stdout: create 103/39 file after the snapshot 2024-03-28T18:59:19.929 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/.snap/snap-subdir-test 2024-03-28T18:59:19.930 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/32 file after the snapshot 2024-03-28T18:59:19.937 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/33 file after the snapshot 2024-03-28T18:59:19.944 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/34 file after the snapshot 2024-03-28T18:59:19.949 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/30 file after the snapshot 2024-03-28T18:59:19.951 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/35 file after the snapshot 2024-03-28T18:59:19.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/31 file after the snapshot 2024-03-28T18:59:19.958 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/36 file after the snapshot 2024-03-28T18:59:19.960 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/32 file after the snapshot 2024-03-28T18:59:19.965 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/37 file after the snapshot 2024-03-28T18:59:19.967 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/33 file after the snapshot 2024-03-28T18:59:19.972 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/38 file after the snapshot 2024-03-28T18:59:19.975 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/34 file after the snapshot 2024-03-28T18:59:19.979 INFO:tasks.workunit.client.0.smithi027.stdout: create 261/39 file after the snapshot 2024-03-28T18:59:19.982 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/35 file after the snapshot 2024-03-28T18:59:19.986 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/.snap/snap-subdir-test 2024-03-28T18:59:19.988 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/36 file after the snapshot 2024-03-28T18:59:19.995 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/37 file after the snapshot 2024-03-28T18:59:20.001 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/38 file after the snapshot 2024-03-28T18:59:20.007 INFO:tasks.workunit.client.1.smithi155.stdout: create 104/39 file after the snapshot 2024-03-28T18:59:20.010 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/30 file after the snapshot 2024-03-28T18:59:20.013 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/31 file after the snapshot 2024-03-28T18:59:20.014 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/.snap/snap-subdir-test 2024-03-28T18:59:20.019 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/32 file after the snapshot 2024-03-28T18:59:20.027 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/33 file after the snapshot 2024-03-28T18:59:20.033 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/34 file after the snapshot 2024-03-28T18:59:20.036 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/30 file after the snapshot 2024-03-28T18:59:20.040 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/35 file after the snapshot 2024-03-28T18:59:20.040 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/31 file after the snapshot 2024-03-28T18:59:20.046 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/36 file after the snapshot 2024-03-28T18:59:20.049 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/32 file after the snapshot 2024-03-28T18:59:20.053 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/37 file after the snapshot 2024-03-28T18:59:20.057 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/33 file after the snapshot 2024-03-28T18:59:20.061 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/38 file after the snapshot 2024-03-28T18:59:20.064 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/34 file after the snapshot 2024-03-28T18:59:20.071 INFO:tasks.workunit.client.0.smithi027.stdout: create 262/39 file after the snapshot 2024-03-28T18:59:20.072 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/35 file after the snapshot 2024-03-28T18:59:20.078 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/.snap/snap-subdir-test 2024-03-28T18:59:20.079 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/36 file after the snapshot 2024-03-28T18:59:20.085 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/37 file after the snapshot 2024-03-28T18:59:20.092 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/38 file after the snapshot 2024-03-28T18:59:20.099 INFO:tasks.workunit.client.1.smithi155.stdout: create 105/39 file after the snapshot 2024-03-28T18:59:20.106 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/.snap/snap-subdir-test 2024-03-28T18:59:20.106 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/30 file after the snapshot 2024-03-28T18:59:20.111 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/31 file after the snapshot 2024-03-28T18:59:20.118 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/32 file after the snapshot 2024-03-28T18:59:20.126 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/33 file after the snapshot 2024-03-28T18:59:20.128 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/30 file after the snapshot 2024-03-28T18:59:20.132 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/31 file after the snapshot 2024-03-28T18:59:20.133 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/34 file after the snapshot 2024-03-28T18:59:20.139 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/32 file after the snapshot 2024-03-28T18:59:20.140 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/35 file after the snapshot 2024-03-28T18:59:20.145 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/33 file after the snapshot 2024-03-28T18:59:20.148 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/36 file after the snapshot 2024-03-28T18:59:20.155 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/34 file after the snapshot 2024-03-28T18:59:20.155 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/37 file after the snapshot 2024-03-28T18:59:20.164 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/35 file after the snapshot 2024-03-28T18:59:20.164 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/38 file after the snapshot 2024-03-28T18:59:20.170 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/36 file after the snapshot 2024-03-28T18:59:20.171 INFO:tasks.workunit.client.0.smithi027.stdout: create 263/39 file after the snapshot 2024-03-28T18:59:20.176 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/37 file after the snapshot 2024-03-28T18:59:20.176 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/.snap/snap-subdir-test 2024-03-28T18:59:20.183 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/38 file after the snapshot 2024-03-28T18:59:20.190 INFO:tasks.workunit.client.1.smithi155.stdout: create 106/39 file after the snapshot 2024-03-28T18:59:20.201 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/30 file after the snapshot 2024-03-28T18:59:20.205 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/31 file after the snapshot 2024-03-28T18:59:20.213 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/32 file after the snapshot 2024-03-28T18:59:20.221 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/33 file after the snapshot 2024-03-28T18:59:20.230 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/34 file after the snapshot 2024-03-28T18:59:20.237 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/35 file after the snapshot 2024-03-28T18:59:20.241 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/.snap/snap-subdir-test 2024-03-28T18:59:20.246 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/36 file after the snapshot 2024-03-28T18:59:20.253 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/37 file after the snapshot 2024-03-28T18:59:20.260 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/38 file after the snapshot 2024-03-28T18:59:20.266 INFO:tasks.workunit.client.0.smithi027.stdout: create 264/39 file after the snapshot 2024-03-28T18:59:20.271 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/.snap/snap-subdir-test 2024-03-28T18:59:20.273 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/30 file after the snapshot 2024-03-28T18:59:20.277 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/31 file after the snapshot 2024-03-28T18:59:20.284 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/32 file after the snapshot 2024-03-28T18:59:20.289 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/33 file after the snapshot 2024-03-28T18:59:20.293 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/30 file after the snapshot 2024-03-28T18:59:20.295 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/34 file after the snapshot 2024-03-28T18:59:20.296 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/31 file after the snapshot 2024-03-28T18:59:20.301 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/35 file after the snapshot 2024-03-28T18:59:20.302 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/32 file after the snapshot 2024-03-28T18:59:20.308 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/36 file after the snapshot 2024-03-28T18:59:20.308 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/33 file after the snapshot 2024-03-28T18:59:20.314 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/37 file after the snapshot 2024-03-28T18:59:20.314 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/34 file after the snapshot 2024-03-28T18:59:20.320 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/38 file after the snapshot 2024-03-28T18:59:20.321 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/35 file after the snapshot 2024-03-28T18:59:20.327 INFO:tasks.workunit.client.1.smithi155.stdout: create 107/39 file after the snapshot 2024-03-28T18:59:20.327 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/36 file after the snapshot 2024-03-28T18:59:20.334 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/37 file after the snapshot 2024-03-28T18:59:20.334 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/.snap/snap-subdir-test 2024-03-28T18:59:20.340 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/38 file after the snapshot 2024-03-28T18:59:20.346 INFO:tasks.workunit.client.0.smithi027.stdout: create 265/39 file after the snapshot 2024-03-28T18:59:20.355 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/.snap/snap-subdir-test 2024-03-28T18:59:20.357 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/30 file after the snapshot 2024-03-28T18:59:20.360 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/31 file after the snapshot 2024-03-28T18:59:20.372 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/32 file after the snapshot 2024-03-28T18:59:20.378 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/30 file after the snapshot 2024-03-28T18:59:20.379 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/33 file after the snapshot 2024-03-28T18:59:20.384 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/34 file after the snapshot 2024-03-28T18:59:20.391 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/35 file after the snapshot 2024-03-28T18:59:20.398 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/36 file after the snapshot 2024-03-28T18:59:20.406 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/37 file after the snapshot 2024-03-28T18:59:20.413 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/38 file after the snapshot 2024-03-28T18:59:20.422 INFO:tasks.workunit.client.1.smithi155.stdout: create 108/39 file after the snapshot 2024-03-28T18:59:20.423 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/31 file after the snapshot 2024-03-28T18:59:20.429 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/.snap/snap-subdir-test 2024-03-28T18:59:20.448 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/32 file after the snapshot 2024-03-28T18:59:20.455 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/33 file after the snapshot 2024-03-28T18:59:20.459 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/30 file after the snapshot 2024-03-28T18:59:20.461 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/34 file after the snapshot 2024-03-28T18:59:20.462 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/31 file after the snapshot 2024-03-28T18:59:20.468 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/32 file after the snapshot 2024-03-28T18:59:20.469 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/35 file after the snapshot 2024-03-28T18:59:20.476 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/36 file after the snapshot 2024-03-28T18:59:20.476 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/33 file after the snapshot 2024-03-28T18:59:20.483 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/34 file after the snapshot 2024-03-28T18:59:20.483 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/37 file after the snapshot 2024-03-28T18:59:20.491 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/35 file after the snapshot 2024-03-28T18:59:20.491 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/38 file after the snapshot 2024-03-28T18:59:20.497 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/36 file after the snapshot 2024-03-28T18:59:20.499 INFO:tasks.workunit.client.0.smithi027.stdout: create 266/39 file after the snapshot 2024-03-28T18:59:20.503 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/37 file after the snapshot 2024-03-28T18:59:20.507 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/.snap/snap-subdir-test 2024-03-28T18:59:20.510 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/38 file after the snapshot 2024-03-28T18:59:20.517 INFO:tasks.workunit.client.1.smithi155.stdout: create 109/39 file after the snapshot 2024-03-28T18:59:20.524 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/.snap/snap-subdir-test 2024-03-28T18:59:20.535 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/30 file after the snapshot 2024-03-28T18:59:20.540 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/31 file after the snapshot 2024-03-28T18:59:20.544 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/30 file after the snapshot 2024-03-28T18:59:20.547 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/32 file after the snapshot 2024-03-28T18:59:20.548 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/31 file after the snapshot 2024-03-28T18:59:20.554 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/32 file after the snapshot 2024-03-28T18:59:20.554 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/33 file after the snapshot 2024-03-28T18:59:20.561 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/33 file after the snapshot 2024-03-28T18:59:20.562 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/34 file after the snapshot 2024-03-28T18:59:20.569 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/35 file after the snapshot 2024-03-28T18:59:20.569 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/34 file after the snapshot 2024-03-28T18:59:20.576 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/36 file after the snapshot 2024-03-28T18:59:20.578 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/35 file after the snapshot 2024-03-28T18:59:20.584 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/37 file after the snapshot 2024-03-28T18:59:20.584 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/36 file after the snapshot 2024-03-28T18:59:20.591 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/37 file after the snapshot 2024-03-28T18:59:20.591 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/38 file after the snapshot 2024-03-28T18:59:20.597 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/38 file after the snapshot 2024-03-28T18:59:20.598 INFO:tasks.workunit.client.0.smithi027.stdout: create 267/39 file after the snapshot 2024-03-28T18:59:20.603 INFO:tasks.workunit.client.1.smithi155.stdout: create 110/39 file after the snapshot 2024-03-28T18:59:20.605 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/.snap/snap-subdir-test 2024-03-28T18:59:20.610 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/.snap/snap-subdir-test 2024-03-28T18:59:20.627 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/30 file after the snapshot 2024-03-28T18:59:20.631 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/31 file after the snapshot 2024-03-28T18:59:20.634 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/30 file after the snapshot 2024-03-28T18:59:20.637 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/31 file after the snapshot 2024-03-28T18:59:20.639 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/32 file after the snapshot 2024-03-28T18:59:20.645 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/32 file after the snapshot 2024-03-28T18:59:20.647 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/33 file after the snapshot 2024-03-28T18:59:20.651 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/33 file after the snapshot 2024-03-28T18:59:20.655 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/34 file after the snapshot 2024-03-28T18:59:20.657 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/34 file after the snapshot 2024-03-28T18:59:20.662 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/35 file after the snapshot 2024-03-28T18:59:20.665 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/35 file after the snapshot 2024-03-28T18:59:20.668 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/36 file after the snapshot 2024-03-28T18:59:20.672 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/36 file after the snapshot 2024-03-28T18:59:20.675 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/37 file after the snapshot 2024-03-28T18:59:20.679 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/37 file after the snapshot 2024-03-28T18:59:20.682 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/38 file after the snapshot 2024-03-28T18:59:20.686 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/38 file after the snapshot 2024-03-28T18:59:20.689 INFO:tasks.workunit.client.0.smithi027.stdout: create 268/39 file after the snapshot 2024-03-28T18:59:20.693 INFO:tasks.workunit.client.1.smithi155.stdout: create 111/39 file after the snapshot 2024-03-28T18:59:20.696 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/.snap/snap-subdir-test 2024-03-28T18:59:20.699 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/.snap/snap-subdir-test 2024-03-28T18:59:20.718 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/30 file after the snapshot 2024-03-28T18:59:20.721 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/30 file after the snapshot 2024-03-28T18:59:20.723 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/31 file after the snapshot 2024-03-28T18:59:20.726 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/31 file after the snapshot 2024-03-28T18:59:20.731 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/32 file after the snapshot 2024-03-28T18:59:20.733 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/32 file after the snapshot 2024-03-28T18:59:20.739 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/33 file after the snapshot 2024-03-28T18:59:20.739 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/33 file after the snapshot 2024-03-28T18:59:20.745 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/34 file after the snapshot 2024-03-28T18:59:20.746 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/34 file after the snapshot 2024-03-28T18:59:20.752 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/35 file after the snapshot 2024-03-28T18:59:20.753 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/35 file after the snapshot 2024-03-28T18:59:20.758 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/36 file after the snapshot 2024-03-28T18:59:20.759 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/36 file after the snapshot 2024-03-28T18:59:20.764 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/37 file after the snapshot 2024-03-28T18:59:20.766 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/37 file after the snapshot 2024-03-28T18:59:20.771 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/38 file after the snapshot 2024-03-28T18:59:20.772 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/38 file after the snapshot 2024-03-28T18:59:20.778 INFO:tasks.workunit.client.1.smithi155.stdout: create 112/39 file after the snapshot 2024-03-28T18:59:20.779 INFO:tasks.workunit.client.0.smithi027.stdout: create 269/39 file after the snapshot 2024-03-28T18:59:20.785 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/.snap/snap-subdir-test 2024-03-28T18:59:20.786 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/.snap/snap-subdir-test 2024-03-28T18:59:20.806 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/30 file after the snapshot 2024-03-28T18:59:20.808 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/30 file after the snapshot 2024-03-28T18:59:20.809 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/31 file after the snapshot 2024-03-28T18:59:20.811 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/31 file after the snapshot 2024-03-28T18:59:20.817 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/32 file after the snapshot 2024-03-28T18:59:20.819 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/32 file after the snapshot 2024-03-28T18:59:20.824 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/33 file after the snapshot 2024-03-28T18:59:20.826 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/33 file after the snapshot 2024-03-28T18:59:20.832 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/34 file after the snapshot 2024-03-28T18:59:20.833 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/34 file after the snapshot 2024-03-28T18:59:20.839 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/35 file after the snapshot 2024-03-28T18:59:20.840 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/35 file after the snapshot 2024-03-28T18:59:20.846 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/36 file after the snapshot 2024-03-28T18:59:20.848 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/36 file after the snapshot 2024-03-28T18:59:20.854 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/37 file after the snapshot 2024-03-28T18:59:20.855 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/37 file after the snapshot 2024-03-28T18:59:20.861 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/38 file after the snapshot 2024-03-28T18:59:20.863 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/38 file after the snapshot 2024-03-28T18:59:20.868 INFO:tasks.workunit.client.1.smithi155.stdout: create 113/39 file after the snapshot 2024-03-28T18:59:20.871 INFO:tasks.workunit.client.0.smithi027.stdout: create 270/39 file after the snapshot 2024-03-28T18:59:20.876 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/.snap/snap-subdir-test 2024-03-28T18:59:20.878 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/.snap/snap-subdir-test 2024-03-28T18:59:20.901 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/30 file after the snapshot 2024-03-28T18:59:20.905 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/31 file after the snapshot 2024-03-28T18:59:20.905 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/30 file after the snapshot 2024-03-28T18:59:20.910 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/31 file after the snapshot 2024-03-28T18:59:20.912 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/32 file after the snapshot 2024-03-28T18:59:20.917 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/32 file after the snapshot 2024-03-28T18:59:20.919 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/33 file after the snapshot 2024-03-28T18:59:20.925 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/33 file after the snapshot 2024-03-28T18:59:20.927 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/34 file after the snapshot 2024-03-28T18:59:20.933 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/34 file after the snapshot 2024-03-28T18:59:20.935 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/35 file after the snapshot 2024-03-28T18:59:20.940 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/35 file after the snapshot 2024-03-28T18:59:20.942 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/36 file after the snapshot 2024-03-28T18:59:20.947 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/36 file after the snapshot 2024-03-28T18:59:20.950 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/37 file after the snapshot 2024-03-28T18:59:20.954 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/37 file after the snapshot 2024-03-28T18:59:20.958 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/38 file after the snapshot 2024-03-28T18:59:20.962 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/38 file after the snapshot 2024-03-28T18:59:20.965 INFO:tasks.workunit.client.1.smithi155.stdout: create 114/39 file after the snapshot 2024-03-28T18:59:20.968 INFO:tasks.workunit.client.0.smithi027.stdout: create 271/39 file after the snapshot 2024-03-28T18:59:20.972 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/.snap/snap-subdir-test 2024-03-28T18:59:20.975 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/.snap/snap-subdir-test 2024-03-28T18:59:20.993 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/30 file after the snapshot 2024-03-28T18:59:20.997 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/31 file after the snapshot 2024-03-28T18:59:21.000 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/30 file after the snapshot 2024-03-28T18:59:21.004 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/31 file after the snapshot 2024-03-28T18:59:21.005 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/32 file after the snapshot 2024-03-28T18:59:21.012 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/32 file after the snapshot 2024-03-28T18:59:21.012 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/33 file after the snapshot 2024-03-28T18:59:21.018 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/34 file after the snapshot 2024-03-28T18:59:21.018 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/33 file after the snapshot 2024-03-28T18:59:21.024 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/35 file after the snapshot 2024-03-28T18:59:21.025 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/34 file after the snapshot 2024-03-28T18:59:21.031 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/36 file after the snapshot 2024-03-28T18:59:21.032 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/35 file after the snapshot 2024-03-28T18:59:21.037 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/37 file after the snapshot 2024-03-28T18:59:21.039 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/36 file after the snapshot 2024-03-28T18:59:21.044 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/38 file after the snapshot 2024-03-28T18:59:21.046 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/37 file after the snapshot 2024-03-28T18:59:21.050 INFO:tasks.workunit.client.1.smithi155.stdout: create 115/39 file after the snapshot 2024-03-28T18:59:21.053 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/38 file after the snapshot 2024-03-28T18:59:21.057 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/.snap/snap-subdir-test 2024-03-28T18:59:21.059 INFO:tasks.workunit.client.0.smithi027.stdout: create 272/39 file after the snapshot 2024-03-28T18:59:21.066 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/.snap/snap-subdir-test 2024-03-28T18:59:21.080 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/30 file after the snapshot 2024-03-28T18:59:21.082 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/31 file after the snapshot 2024-03-28T18:59:21.086 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/30 file after the snapshot 2024-03-28T18:59:21.089 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/32 file after the snapshot 2024-03-28T18:59:21.090 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/31 file after the snapshot 2024-03-28T18:59:21.096 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/33 file after the snapshot 2024-03-28T18:59:21.097 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/32 file after the snapshot 2024-03-28T18:59:21.103 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/34 file after the snapshot 2024-03-28T18:59:21.104 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/33 file after the snapshot 2024-03-28T18:59:21.110 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/35 file after the snapshot 2024-03-28T18:59:21.112 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/34 file after the snapshot 2024-03-28T18:59:21.116 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/36 file after the snapshot 2024-03-28T18:59:21.120 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/35 file after the snapshot 2024-03-28T18:59:21.123 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/37 file after the snapshot 2024-03-28T18:59:21.127 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/36 file after the snapshot 2024-03-28T18:59:21.129 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/38 file after the snapshot 2024-03-28T18:59:21.134 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/37 file after the snapshot 2024-03-28T18:59:21.136 INFO:tasks.workunit.client.1.smithi155.stdout: create 116/39 file after the snapshot 2024-03-28T18:59:21.140 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/38 file after the snapshot 2024-03-28T18:59:21.143 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/.snap/snap-subdir-test 2024-03-28T18:59:21.147 INFO:tasks.workunit.client.0.smithi027.stdout: create 273/39 file after the snapshot 2024-03-28T18:59:21.154 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/.snap/snap-subdir-test 2024-03-28T18:59:21.163 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/30 file after the snapshot 2024-03-28T18:59:21.168 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/31 file after the snapshot 2024-03-28T18:59:21.174 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/30 file after the snapshot 2024-03-28T18:59:21.175 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/32 file after the snapshot 2024-03-28T18:59:21.179 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/31 file after the snapshot 2024-03-28T18:59:21.182 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/33 file after the snapshot 2024-03-28T18:59:21.187 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/32 file after the snapshot 2024-03-28T18:59:21.189 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/34 file after the snapshot 2024-03-28T18:59:21.194 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/33 file after the snapshot 2024-03-28T18:59:21.197 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/35 file after the snapshot 2024-03-28T18:59:21.202 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/34 file after the snapshot 2024-03-28T18:59:21.205 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/36 file after the snapshot 2024-03-28T18:59:21.210 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/35 file after the snapshot 2024-03-28T18:59:21.213 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/37 file after the snapshot 2024-03-28T18:59:21.218 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/36 file after the snapshot 2024-03-28T18:59:21.220 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/38 file after the snapshot 2024-03-28T18:59:21.223 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/37 file after the snapshot 2024-03-28T18:59:21.226 INFO:tasks.workunit.client.1.smithi155.stdout: create 117/39 file after the snapshot 2024-03-28T18:59:21.230 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/38 file after the snapshot 2024-03-28T18:59:21.233 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/.snap/snap-subdir-test 2024-03-28T18:59:21.235 INFO:tasks.workunit.client.0.smithi027.stdout: create 274/39 file after the snapshot 2024-03-28T18:59:21.242 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/.snap/snap-subdir-test 2024-03-28T18:59:21.258 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/30 file after the snapshot 2024-03-28T18:59:21.261 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/31 file after the snapshot 2024-03-28T18:59:21.266 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/30 file after the snapshot 2024-03-28T18:59:21.267 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/32 file after the snapshot 2024-03-28T18:59:21.270 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/31 file after the snapshot 2024-03-28T18:59:21.275 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/33 file after the snapshot 2024-03-28T18:59:21.276 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/32 file after the snapshot 2024-03-28T18:59:21.283 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/34 file after the snapshot 2024-03-28T18:59:21.283 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/33 file after the snapshot 2024-03-28T18:59:21.291 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/34 file after the snapshot 2024-03-28T18:59:21.291 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/35 file after the snapshot 2024-03-28T18:59:21.299 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/36 file after the snapshot 2024-03-28T18:59:21.299 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/35 file after the snapshot 2024-03-28T18:59:21.305 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/36 file after the snapshot 2024-03-28T18:59:21.306 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/37 file after the snapshot 2024-03-28T18:59:21.312 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/37 file after the snapshot 2024-03-28T18:59:21.312 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/38 file after the snapshot 2024-03-28T18:59:21.318 INFO:tasks.workunit.client.1.smithi155.stdout: create 118/39 file after the snapshot 2024-03-28T18:59:21.318 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/38 file after the snapshot 2024-03-28T18:59:21.324 INFO:tasks.workunit.client.0.smithi027.stdout: create 275/39 file after the snapshot 2024-03-28T18:59:21.324 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/.snap/snap-subdir-test 2024-03-28T18:59:21.331 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/.snap/snap-subdir-test 2024-03-28T18:59:21.352 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/30 file after the snapshot 2024-03-28T18:59:21.357 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/30 file after the snapshot 2024-03-28T18:59:21.361 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/31 file after the snapshot 2024-03-28T18:59:21.369 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/32 file after the snapshot 2024-03-28T18:59:21.378 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/33 file after the snapshot 2024-03-28T18:59:21.396 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/34 file after the snapshot 2024-03-28T18:59:21.406 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/31 file after the snapshot 2024-03-28T18:59:21.448 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/32 file after the snapshot 2024-03-28T18:59:21.458 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/33 file after the snapshot 2024-03-28T18:59:21.466 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/34 file after the snapshot 2024-03-28T18:59:21.476 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/35 file after the snapshot 2024-03-28T18:59:21.482 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/35 file after the snapshot 2024-03-28T18:59:21.485 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/36 file after the snapshot 2024-03-28T18:59:21.493 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/37 file after the snapshot 2024-03-28T18:59:21.502 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/38 file after the snapshot 2024-03-28T18:59:21.509 INFO:tasks.workunit.client.1.smithi155.stdout: create 119/39 file after the snapshot 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/36 file after the snapshot 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/37 file after the snapshot 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/38 file after the snapshot 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.0.smithi027.stdout: create 276/39 file after the snapshot 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/.snap/snap-subdir-test 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/30 file after the snapshot 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/.snap/snap-subdir-test 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/30 file after the snapshot 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/31 file after the snapshot 2024-03-28T18:59:21.564 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/32 file after the snapshot 2024-03-28T18:59:21.565 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/33 file after the snapshot 2024-03-28T18:59:21.565 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/34 file after the snapshot 2024-03-28T18:59:21.565 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/31 file after the snapshot 2024-03-28T18:59:21.567 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/35 file after the snapshot 2024-03-28T18:59:21.573 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/32 file after the snapshot 2024-03-28T18:59:21.574 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/36 file after the snapshot 2024-03-28T18:59:21.580 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/33 file after the snapshot 2024-03-28T18:59:21.582 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/37 file after the snapshot 2024-03-28T18:59:21.589 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/34 file after the snapshot 2024-03-28T18:59:21.590 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/38 file after the snapshot 2024-03-28T18:59:21.596 INFO:tasks.workunit.client.1.smithi155.stdout: create 120/39 file after the snapshot 2024-03-28T18:59:21.598 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/35 file after the snapshot 2024-03-28T18:59:21.604 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/.snap/snap-subdir-test 2024-03-28T18:59:21.606 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/36 file after the snapshot 2024-03-28T18:59:21.614 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/37 file after the snapshot 2024-03-28T18:59:21.621 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/38 file after the snapshot 2024-03-28T18:59:21.628 INFO:tasks.workunit.client.0.smithi027.stdout: create 277/39 file after the snapshot 2024-03-28T18:59:21.630 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/30 file after the snapshot 2024-03-28T18:59:21.633 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/.snap/snap-subdir-test 2024-03-28T18:59:21.634 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/31 file after the snapshot 2024-03-28T18:59:21.640 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/32 file after the snapshot 2024-03-28T18:59:21.647 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/33 file after the snapshot 2024-03-28T18:59:21.654 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/34 file after the snapshot 2024-03-28T18:59:21.654 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/30 file after the snapshot 2024-03-28T18:59:21.659 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/31 file after the snapshot 2024-03-28T18:59:21.660 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/35 file after the snapshot 2024-03-28T18:59:21.666 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/32 file after the snapshot 2024-03-28T18:59:21.667 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/36 file after the snapshot 2024-03-28T18:59:21.672 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/33 file after the snapshot 2024-03-28T18:59:21.673 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/37 file after the snapshot 2024-03-28T18:59:21.678 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/34 file after the snapshot 2024-03-28T18:59:21.681 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/38 file after the snapshot 2024-03-28T18:59:21.684 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/35 file after the snapshot 2024-03-28T18:59:21.688 INFO:tasks.workunit.client.1.smithi155.stdout: create 121/39 file after the snapshot 2024-03-28T18:59:21.691 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/36 file after the snapshot 2024-03-28T18:59:21.694 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/.snap/snap-subdir-test 2024-03-28T18:59:21.698 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/37 file after the snapshot 2024-03-28T18:59:21.704 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/38 file after the snapshot 2024-03-28T18:59:21.711 INFO:tasks.workunit.client.0.smithi027.stdout: create 278/39 file after the snapshot 2024-03-28T18:59:21.716 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/30 file after the snapshot 2024-03-28T18:59:21.718 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/.snap/snap-subdir-test 2024-03-28T18:59:21.720 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/31 file after the snapshot 2024-03-28T18:59:21.726 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/32 file after the snapshot 2024-03-28T18:59:21.732 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/33 file after the snapshot 2024-03-28T18:59:21.739 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/30 file after the snapshot 2024-03-28T18:59:21.739 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/34 file after the snapshot 2024-03-28T18:59:21.743 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/31 file after the snapshot 2024-03-28T18:59:21.745 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/35 file after the snapshot 2024-03-28T18:59:21.749 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/32 file after the snapshot 2024-03-28T18:59:21.751 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/36 file after the snapshot 2024-03-28T18:59:21.756 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/33 file after the snapshot 2024-03-28T18:59:21.759 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/37 file after the snapshot 2024-03-28T18:59:21.763 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/34 file after the snapshot 2024-03-28T18:59:21.766 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/38 file after the snapshot 2024-03-28T18:59:21.770 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/35 file after the snapshot 2024-03-28T18:59:21.773 INFO:tasks.workunit.client.1.smithi155.stdout: create 122/39 file after the snapshot 2024-03-28T18:59:21.777 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/36 file after the snapshot 2024-03-28T18:59:21.780 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/.snap/snap-subdir-test 2024-03-28T18:59:21.782 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/37 file after the snapshot 2024-03-28T18:59:21.788 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/38 file after the snapshot 2024-03-28T18:59:21.796 INFO:tasks.workunit.client.0.smithi027.stdout: create 279/39 file after the snapshot 2024-03-28T18:59:21.801 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/30 file after the snapshot 2024-03-28T18:59:21.802 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/.snap/snap-subdir-test 2024-03-28T18:59:21.803 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/31 file after the snapshot 2024-03-28T18:59:21.810 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/32 file after the snapshot 2024-03-28T18:59:21.817 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/33 file after the snapshot 2024-03-28T18:59:21.823 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/34 file after the snapshot 2024-03-28T18:59:21.828 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/30 file after the snapshot 2024-03-28T18:59:21.830 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/35 file after the snapshot 2024-03-28T18:59:21.832 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/31 file after the snapshot 2024-03-28T18:59:21.838 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/36 file after the snapshot 2024-03-28T18:59:21.840 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/32 file after the snapshot 2024-03-28T18:59:21.846 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/37 file after the snapshot 2024-03-28T18:59:21.848 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/33 file after the snapshot 2024-03-28T18:59:21.853 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/38 file after the snapshot 2024-03-28T18:59:21.854 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/34 file after the snapshot 2024-03-28T18:59:21.860 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/35 file after the snapshot 2024-03-28T18:59:21.860 INFO:tasks.workunit.client.1.smithi155.stdout: create 123/39 file after the snapshot 2024-03-28T18:59:21.866 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/36 file after the snapshot 2024-03-28T18:59:21.867 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/.snap/snap-subdir-test 2024-03-28T18:59:21.873 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/37 file after the snapshot 2024-03-28T18:59:21.880 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/38 file after the snapshot 2024-03-28T18:59:21.886 INFO:tasks.workunit.client.0.smithi027.stdout: create 280/39 file after the snapshot 2024-03-28T18:59:21.889 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/30 file after the snapshot 2024-03-28T18:59:21.893 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/31 file after the snapshot 2024-03-28T18:59:21.895 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/.snap/snap-subdir-test 2024-03-28T18:59:21.900 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/32 file after the snapshot 2024-03-28T18:59:21.906 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/33 file after the snapshot 2024-03-28T18:59:21.912 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/34 file after the snapshot 2024-03-28T18:59:21.918 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/30 file after the snapshot 2024-03-28T18:59:21.919 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/35 file after the snapshot 2024-03-28T18:59:21.921 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/31 file after the snapshot 2024-03-28T18:59:21.926 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/36 file after the snapshot 2024-03-28T18:59:21.927 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/32 file after the snapshot 2024-03-28T18:59:21.933 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/37 file after the snapshot 2024-03-28T18:59:21.938 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/38 file after the snapshot 2024-03-28T18:59:21.940 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/33 file after the snapshot 2024-03-28T18:59:21.945 INFO:tasks.workunit.client.1.smithi155.stdout: create 124/39 file after the snapshot 2024-03-28T18:59:21.950 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/34 file after the snapshot 2024-03-28T18:59:21.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/.snap/snap-subdir-test 2024-03-28T18:59:21.956 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/35 file after the snapshot 2024-03-28T18:59:21.963 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/36 file after the snapshot 2024-03-28T18:59:21.969 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/37 file after the snapshot 2024-03-28T18:59:21.977 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/30 file after the snapshot 2024-03-28T18:59:21.977 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/38 file after the snapshot 2024-03-28T18:59:21.980 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/31 file after the snapshot 2024-03-28T18:59:21.984 INFO:tasks.workunit.client.0.smithi027.stdout: create 281/39 file after the snapshot 2024-03-28T18:59:21.987 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/32 file after the snapshot 2024-03-28T18:59:21.993 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/.snap/snap-subdir-test 2024-03-28T18:59:21.994 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/33 file after the snapshot 2024-03-28T18:59:22.000 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/34 file after the snapshot 2024-03-28T18:59:22.008 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/35 file after the snapshot 2024-03-28T18:59:22.014 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/36 file after the snapshot 2024-03-28T18:59:22.018 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/30 file after the snapshot 2024-03-28T18:59:22.020 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/37 file after the snapshot 2024-03-28T18:59:22.021 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/31 file after the snapshot 2024-03-28T18:59:22.027 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/32 file after the snapshot 2024-03-28T18:59:22.028 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/38 file after the snapshot 2024-03-28T18:59:22.034 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/33 file after the snapshot 2024-03-28T18:59:22.035 INFO:tasks.workunit.client.1.smithi155.stdout: create 125/39 file after the snapshot 2024-03-28T18:59:22.041 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/34 file after the snapshot 2024-03-28T18:59:22.042 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/.snap/snap-subdir-test 2024-03-28T18:59:22.047 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/35 file after the snapshot 2024-03-28T18:59:22.054 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/36 file after the snapshot 2024-03-28T18:59:22.061 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/37 file after the snapshot 2024-03-28T18:59:22.066 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/30 file after the snapshot 2024-03-28T18:59:22.068 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/38 file after the snapshot 2024-03-28T18:59:22.069 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/31 file after the snapshot 2024-03-28T18:59:22.076 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/32 file after the snapshot 2024-03-28T18:59:22.076 INFO:tasks.workunit.client.0.smithi027.stdout: create 282/39 file after the snapshot 2024-03-28T18:59:22.082 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/33 file after the snapshot 2024-03-28T18:59:22.083 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/.snap/snap-subdir-test 2024-03-28T18:59:22.089 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/34 file after the snapshot 2024-03-28T18:59:22.095 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/35 file after the snapshot 2024-03-28T18:59:22.102 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/36 file after the snapshot 2024-03-28T18:59:22.108 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/37 file after the snapshot 2024-03-28T18:59:22.109 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/30 file after the snapshot 2024-03-28T18:59:22.114 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/31 file after the snapshot 2024-03-28T18:59:22.115 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/38 file after the snapshot 2024-03-28T18:59:22.121 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/32 file after the snapshot 2024-03-28T18:59:22.122 INFO:tasks.workunit.client.1.smithi155.stdout: create 126/39 file after the snapshot 2024-03-28T18:59:22.127 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/33 file after the snapshot 2024-03-28T18:59:22.128 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/.snap/snap-subdir-test 2024-03-28T18:59:22.133 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/34 file after the snapshot 2024-03-28T18:59:22.140 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/35 file after the snapshot 2024-03-28T18:59:22.147 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/36 file after the snapshot 2024-03-28T18:59:22.154 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/30 file after the snapshot 2024-03-28T18:59:22.155 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/37 file after the snapshot 2024-03-28T18:59:22.158 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/31 file after the snapshot 2024-03-28T18:59:22.163 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/38 file after the snapshot 2024-03-28T18:59:22.165 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/32 file after the snapshot 2024-03-28T18:59:22.171 INFO:tasks.workunit.client.0.smithi027.stdout: create 283/39 file after the snapshot 2024-03-28T18:59:22.171 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/33 file after the snapshot 2024-03-28T18:59:22.177 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/.snap/snap-subdir-test 2024-03-28T18:59:22.178 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/34 file after the snapshot 2024-03-28T18:59:22.184 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/35 file after the snapshot 2024-03-28T18:59:22.190 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/36 file after the snapshot 2024-03-28T18:59:22.197 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/37 file after the snapshot 2024-03-28T18:59:22.206 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/38 file after the snapshot 2024-03-28T18:59:22.206 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/30 file after the snapshot 2024-03-28T18:59:22.210 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/31 file after the snapshot 2024-03-28T18:59:22.213 INFO:tasks.workunit.client.1.smithi155.stdout: create 127/39 file after the snapshot 2024-03-28T18:59:22.218 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/32 file after the snapshot 2024-03-28T18:59:22.222 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/.snap/snap-subdir-test 2024-03-28T18:59:22.224 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/33 file after the snapshot 2024-03-28T18:59:22.230 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/34 file after the snapshot 2024-03-28T18:59:22.237 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/35 file after the snapshot 2024-03-28T18:59:22.243 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/30 file after the snapshot 2024-03-28T18:59:22.244 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/36 file after the snapshot 2024-03-28T18:59:22.248 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/31 file after the snapshot 2024-03-28T18:59:22.250 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/37 file after the snapshot 2024-03-28T18:59:22.256 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/32 file after the snapshot 2024-03-28T18:59:22.258 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/38 file after the snapshot 2024-03-28T18:59:22.263 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/33 file after the snapshot 2024-03-28T18:59:22.264 INFO:tasks.workunit.client.0.smithi027.stdout: create 284/39 file after the snapshot 2024-03-28T18:59:22.270 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/34 file after the snapshot 2024-03-28T18:59:22.270 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/.snap/snap-subdir-test 2024-03-28T18:59:22.275 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/35 file after the snapshot 2024-03-28T18:59:22.283 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/36 file after the snapshot 2024-03-28T18:59:22.290 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/37 file after the snapshot 2024-03-28T18:59:22.296 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/30 file after the snapshot 2024-03-28T18:59:22.298 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/38 file after the snapshot 2024-03-28T18:59:22.299 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/31 file after the snapshot 2024-03-28T18:59:22.305 INFO:tasks.workunit.client.1.smithi155.stdout: create 128/39 file after the snapshot 2024-03-28T18:59:22.306 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/32 file after the snapshot 2024-03-28T18:59:22.312 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/.snap/snap-subdir-test 2024-03-28T18:59:22.312 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/33 file after the snapshot 2024-03-28T18:59:22.319 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/34 file after the snapshot 2024-03-28T18:59:22.326 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/35 file after the snapshot 2024-03-28T18:59:22.333 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/30 file after the snapshot 2024-03-28T18:59:22.333 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/36 file after the snapshot 2024-03-28T18:59:22.336 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/31 file after the snapshot 2024-03-28T18:59:22.339 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/37 file after the snapshot 2024-03-28T18:59:22.343 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/32 file after the snapshot 2024-03-28T18:59:22.347 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/38 file after the snapshot 2024-03-28T18:59:22.350 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/33 file after the snapshot 2024-03-28T18:59:22.354 INFO:tasks.workunit.client.0.smithi027.stdout: create 285/39 file after the snapshot 2024-03-28T18:59:22.357 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/34 file after the snapshot 2024-03-28T18:59:22.361 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/.snap/snap-subdir-test 2024-03-28T18:59:22.371 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/35 file after the snapshot 2024-03-28T18:59:22.386 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/30 file after the snapshot 2024-03-28T18:59:22.389 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/31 file after the snapshot 2024-03-28T18:59:22.409 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/32 file after the snapshot 2024-03-28T18:59:22.415 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/36 file after the snapshot 2024-03-28T18:59:22.450 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/37 file after the snapshot 2024-03-28T18:59:22.452 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/33 file after the snapshot 2024-03-28T18:59:22.456 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/38 file after the snapshot 2024-03-28T18:59:22.464 INFO:tasks.workunit.client.1.smithi155.stdout: create 129/39 file after the snapshot 2024-03-28T18:59:22.471 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/.snap/snap-subdir-test 2024-03-28T18:59:22.476 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/34 file after the snapshot 2024-03-28T18:59:22.484 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/35 file after the snapshot 2024-03-28T18:59:22.491 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/36 file after the snapshot 2024-03-28T18:59:22.494 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/30 file after the snapshot 2024-03-28T18:59:22.497 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/31 file after the snapshot 2024-03-28T18:59:22.497 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/37 file after the snapshot 2024-03-28T18:59:22.504 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/38 file after the snapshot 2024-03-28T18:59:22.505 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/32 file after the snapshot 2024-03-28T18:59:22.510 INFO:tasks.workunit.client.0.smithi027.stdout: create 286/39 file after the snapshot 2024-03-28T18:59:22.512 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/33 file after the snapshot 2024-03-28T18:59:22.516 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/.snap/snap-subdir-test 2024-03-28T18:59:22.520 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/34 file after the snapshot 2024-03-28T18:59:22.526 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/35 file after the snapshot 2024-03-28T18:59:22.533 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/36 file after the snapshot 2024-03-28T18:59:22.538 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/37 file after the snapshot 2024-03-28T18:59:22.539 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/30 file after the snapshot 2024-03-28T18:59:22.543 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/31 file after the snapshot 2024-03-28T18:59:22.545 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/38 file after the snapshot 2024-03-28T18:59:22.550 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/32 file after the snapshot 2024-03-28T18:59:22.553 INFO:tasks.workunit.client.1.smithi155.stdout: create 130/39 file after the snapshot 2024-03-28T18:59:22.556 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/33 file after the snapshot 2024-03-28T18:59:22.560 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/.snap/snap-subdir-test 2024-03-28T18:59:22.564 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/34 file after the snapshot 2024-03-28T18:59:22.572 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/35 file after the snapshot 2024-03-28T18:59:22.580 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/36 file after the snapshot 2024-03-28T18:59:22.586 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/30 file after the snapshot 2024-03-28T18:59:22.587 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/37 file after the snapshot 2024-03-28T18:59:22.589 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/31 file after the snapshot 2024-03-28T18:59:22.596 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/38 file after the snapshot 2024-03-28T18:59:22.597 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/32 file after the snapshot 2024-03-28T18:59:22.603 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/33 file after the snapshot 2024-03-28T18:59:22.604 INFO:tasks.workunit.client.0.smithi027.stdout: create 287/39 file after the snapshot 2024-03-28T18:59:22.610 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/34 file after the snapshot 2024-03-28T18:59:22.611 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/.snap/snap-subdir-test 2024-03-28T18:59:22.617 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/35 file after the snapshot 2024-03-28T18:59:22.623 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/36 file after the snapshot 2024-03-28T18:59:22.631 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/37 file after the snapshot 2024-03-28T18:59:22.636 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/30 file after the snapshot 2024-03-28T18:59:22.639 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/38 file after the snapshot 2024-03-28T18:59:22.641 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/31 file after the snapshot 2024-03-28T18:59:22.647 INFO:tasks.workunit.client.1.smithi155.stdout: create 131/39 file after the snapshot 2024-03-28T18:59:22.649 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/32 file after the snapshot 2024-03-28T18:59:22.655 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/.snap/snap-subdir-test 2024-03-28T18:59:22.656 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/33 file after the snapshot 2024-03-28T18:59:22.663 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/34 file after the snapshot 2024-03-28T18:59:22.670 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/35 file after the snapshot 2024-03-28T18:59:22.676 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/36 file after the snapshot 2024-03-28T18:59:22.678 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/30 file after the snapshot 2024-03-28T18:59:22.681 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/31 file after the snapshot 2024-03-28T18:59:22.683 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/37 file after the snapshot 2024-03-28T18:59:22.689 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/32 file after the snapshot 2024-03-28T18:59:22.692 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/38 file after the snapshot 2024-03-28T18:59:22.695 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/33 file after the snapshot 2024-03-28T18:59:22.699 INFO:tasks.workunit.client.0.smithi027.stdout: create 288/39 file after the snapshot 2024-03-28T18:59:22.701 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/34 file after the snapshot 2024-03-28T18:59:22.706 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/.snap/snap-subdir-test 2024-03-28T18:59:22.708 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/35 file after the snapshot 2024-03-28T18:59:22.716 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/36 file after the snapshot 2024-03-28T18:59:22.722 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/37 file after the snapshot 2024-03-28T18:59:22.728 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/30 file after the snapshot 2024-03-28T18:59:22.730 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/38 file after the snapshot 2024-03-28T18:59:22.733 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/31 file after the snapshot 2024-03-28T18:59:22.740 INFO:tasks.workunit.client.1.smithi155.stdout: create 132/39 file after the snapshot 2024-03-28T18:59:22.740 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/32 file after the snapshot 2024-03-28T18:59:22.747 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/33 file after the snapshot 2024-03-28T18:59:22.747 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/.snap/snap-subdir-test 2024-03-28T18:59:22.756 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/34 file after the snapshot 2024-03-28T18:59:22.763 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/35 file after the snapshot 2024-03-28T18:59:22.769 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/30 file after the snapshot 2024-03-28T18:59:22.771 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/36 file after the snapshot 2024-03-28T18:59:22.773 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/31 file after the snapshot 2024-03-28T18:59:22.780 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/37 file after the snapshot 2024-03-28T18:59:22.780 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/32 file after the snapshot 2024-03-28T18:59:22.786 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/33 file after the snapshot 2024-03-28T18:59:22.789 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/38 file after the snapshot 2024-03-28T18:59:22.793 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/34 file after the snapshot 2024-03-28T18:59:22.797 INFO:tasks.workunit.client.0.smithi027.stdout: create 289/39 file after the snapshot 2024-03-28T18:59:22.800 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/35 file after the snapshot 2024-03-28T18:59:22.803 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/.snap/snap-subdir-test 2024-03-28T18:59:22.807 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/36 file after the snapshot 2024-03-28T18:59:22.814 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/37 file after the snapshot 2024-03-28T18:59:22.824 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/38 file after the snapshot 2024-03-28T18:59:22.828 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/30 file after the snapshot 2024-03-28T18:59:22.832 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/31 file after the snapshot 2024-03-28T18:59:22.832 INFO:tasks.workunit.client.1.smithi155.stdout: create 133/39 file after the snapshot 2024-03-28T18:59:22.839 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/.snap/snap-subdir-test 2024-03-28T18:59:22.840 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/32 file after the snapshot 2024-03-28T18:59:22.847 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/33 file after the snapshot 2024-03-28T18:59:22.855 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/34 file after the snapshot 2024-03-28T18:59:22.861 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/35 file after the snapshot 2024-03-28T18:59:22.865 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/30 file after the snapshot 2024-03-28T18:59:22.868 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/36 file after the snapshot 2024-03-28T18:59:22.869 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/31 file after the snapshot 2024-03-28T18:59:22.876 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/32 file after the snapshot 2024-03-28T18:59:22.877 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/37 file after the snapshot 2024-03-28T18:59:22.884 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/33 file after the snapshot 2024-03-28T18:59:22.885 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/38 file after the snapshot 2024-03-28T18:59:22.891 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/34 file after the snapshot 2024-03-28T18:59:22.893 INFO:tasks.workunit.client.0.smithi027.stdout: create 290/39 file after the snapshot 2024-03-28T18:59:22.898 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/35 file after the snapshot 2024-03-28T18:59:22.902 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/.snap/snap-subdir-test 2024-03-28T18:59:22.904 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/36 file after the snapshot 2024-03-28T18:59:22.910 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/37 file after the snapshot 2024-03-28T18:59:22.917 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/38 file after the snapshot 2024-03-28T18:59:22.923 INFO:tasks.workunit.client.1.smithi155.stdout: create 134/39 file after the snapshot 2024-03-28T18:59:22.925 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/30 file after the snapshot 2024-03-28T18:59:22.928 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/31 file after the snapshot 2024-03-28T18:59:22.930 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/.snap/snap-subdir-test 2024-03-28T18:59:22.935 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/32 file after the snapshot 2024-03-28T18:59:22.942 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/33 file after the snapshot 2024-03-28T18:59:22.949 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/34 file after the snapshot 2024-03-28T18:59:22.956 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/35 file after the snapshot 2024-03-28T18:59:22.958 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/30 file after the snapshot 2024-03-28T18:59:22.962 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/31 file after the snapshot 2024-03-28T18:59:22.964 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/36 file after the snapshot 2024-03-28T18:59:22.969 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/32 file after the snapshot 2024-03-28T18:59:22.972 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/37 file after the snapshot 2024-03-28T18:59:22.976 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/33 file after the snapshot 2024-03-28T18:59:22.980 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/38 file after the snapshot 2024-03-28T18:59:22.984 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/34 file after the snapshot 2024-03-28T18:59:22.988 INFO:tasks.workunit.client.0.smithi027.stdout: create 291/39 file after the snapshot 2024-03-28T18:59:22.991 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/35 file after the snapshot 2024-03-28T18:59:22.995 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/.snap/snap-subdir-test 2024-03-28T18:59:22.998 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/36 file after the snapshot 2024-03-28T18:59:23.005 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/37 file after the snapshot 2024-03-28T18:59:23.012 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/38 file after the snapshot 2024-03-28T18:59:23.018 INFO:tasks.workunit.client.1.smithi155.stdout: create 135/39 file after the snapshot 2024-03-28T18:59:23.020 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/30 file after the snapshot 2024-03-28T18:59:23.023 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/.snap/snap-subdir-test 2024-03-28T18:59:23.024 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/31 file after the snapshot 2024-03-28T18:59:23.030 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/32 file after the snapshot 2024-03-28T18:59:23.037 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/33 file after the snapshot 2024-03-28T18:59:23.045 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/34 file after the snapshot 2024-03-28T18:59:23.047 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/30 file after the snapshot 2024-03-28T18:59:23.050 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/31 file after the snapshot 2024-03-28T18:59:23.051 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/35 file after the snapshot 2024-03-28T18:59:23.056 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/32 file after the snapshot 2024-03-28T18:59:23.060 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/36 file after the snapshot 2024-03-28T18:59:23.062 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/33 file after the snapshot 2024-03-28T18:59:23.067 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/34 file after the snapshot 2024-03-28T18:59:23.067 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/37 file after the snapshot 2024-03-28T18:59:23.073 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/35 file after the snapshot 2024-03-28T18:59:23.076 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/38 file after the snapshot 2024-03-28T18:59:23.079 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/36 file after the snapshot 2024-03-28T18:59:23.082 INFO:tasks.workunit.client.0.smithi027.stdout: create 292/39 file after the snapshot 2024-03-28T18:59:23.085 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/37 file after the snapshot 2024-03-28T18:59:23.090 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/.snap/snap-subdir-test 2024-03-28T18:59:23.091 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/38 file after the snapshot 2024-03-28T18:59:23.096 INFO:tasks.workunit.client.1.smithi155.stdout: create 136/39 file after the snapshot 2024-03-28T18:59:23.102 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/.snap/snap-subdir-test 2024-03-28T18:59:23.115 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/30 file after the snapshot 2024-03-28T18:59:23.119 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/31 file after the snapshot 2024-03-28T18:59:23.126 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/30 file after the snapshot 2024-03-28T18:59:23.127 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/32 file after the snapshot 2024-03-28T18:59:23.130 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/31 file after the snapshot 2024-03-28T18:59:23.135 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/32 file after the snapshot 2024-03-28T18:59:23.136 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/33 file after the snapshot 2024-03-28T18:59:23.143 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/33 file after the snapshot 2024-03-28T18:59:23.145 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/34 file after the snapshot 2024-03-28T18:59:23.148 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/34 file after the snapshot 2024-03-28T18:59:23.151 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/35 file after the snapshot 2024-03-28T18:59:23.154 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/35 file after the snapshot 2024-03-28T18:59:23.160 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/36 file after the snapshot 2024-03-28T18:59:23.160 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/36 file after the snapshot 2024-03-28T18:59:23.166 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/37 file after the snapshot 2024-03-28T18:59:23.168 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/37 file after the snapshot 2024-03-28T18:59:23.172 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/38 file after the snapshot 2024-03-28T18:59:23.176 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/38 file after the snapshot 2024-03-28T18:59:23.178 INFO:tasks.workunit.client.1.smithi155.stdout: create 137/39 file after the snapshot 2024-03-28T18:59:23.183 INFO:tasks.workunit.client.0.smithi027.stdout: create 293/39 file after the snapshot 2024-03-28T18:59:23.184 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/.snap/snap-subdir-test 2024-03-28T18:59:23.191 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/.snap/snap-subdir-test 2024-03-28T18:59:23.208 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/30 file after the snapshot 2024-03-28T18:59:23.212 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/31 file after the snapshot 2024-03-28T18:59:23.218 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/30 file after the snapshot 2024-03-28T18:59:23.218 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/32 file after the snapshot 2024-03-28T18:59:23.222 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/31 file after the snapshot 2024-03-28T18:59:23.225 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/33 file after the snapshot 2024-03-28T18:59:23.230 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/32 file after the snapshot 2024-03-28T18:59:23.231 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/34 file after the snapshot 2024-03-28T18:59:23.237 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/35 file after the snapshot 2024-03-28T18:59:23.239 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/33 file after the snapshot 2024-03-28T18:59:23.244 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/36 file after the snapshot 2024-03-28T18:59:23.248 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/34 file after the snapshot 2024-03-28T18:59:23.249 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/37 file after the snapshot 2024-03-28T18:59:23.255 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/38 file after the snapshot 2024-03-28T18:59:23.257 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/35 file after the snapshot 2024-03-28T18:59:23.261 INFO:tasks.workunit.client.1.smithi155.stdout: create 138/39 file after the snapshot 2024-03-28T18:59:23.267 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/36 file after the snapshot 2024-03-28T18:59:23.267 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/.snap/snap-subdir-test 2024-03-28T18:59:23.276 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/37 file after the snapshot 2024-03-28T18:59:23.286 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/38 file after the snapshot 2024-03-28T18:59:23.291 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/30 file after the snapshot 2024-03-28T18:59:23.295 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/31 file after the snapshot 2024-03-28T18:59:23.297 INFO:tasks.workunit.client.0.smithi027.stdout: create 294/39 file after the snapshot 2024-03-28T18:59:23.301 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/32 file after the snapshot 2024-03-28T18:59:23.303 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/.snap/snap-subdir-test 2024-03-28T18:59:23.307 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/33 file after the snapshot 2024-03-28T18:59:23.312 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/34 file after the snapshot 2024-03-28T18:59:23.318 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/35 file after the snapshot 2024-03-28T18:59:23.323 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/36 file after the snapshot 2024-03-28T18:59:23.329 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/37 file after the snapshot 2024-03-28T18:59:23.330 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/30 file after the snapshot 2024-03-28T18:59:23.334 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/38 file after the snapshot 2024-03-28T18:59:23.335 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/31 file after the snapshot 2024-03-28T18:59:23.340 INFO:tasks.workunit.client.1.smithi155.stdout: create 139/39 file after the snapshot 2024-03-28T18:59:23.343 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/32 file after the snapshot 2024-03-28T18:59:23.346 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/.snap/snap-subdir-test 2024-03-28T18:59:23.352 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/33 file after the snapshot 2024-03-28T18:59:23.362 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/34 file after the snapshot 2024-03-28T18:59:23.368 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/30 file after the snapshot 2024-03-28T18:59:23.371 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/35 file after the snapshot 2024-03-28T18:59:23.379 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/31 file after the snapshot 2024-03-28T18:59:23.382 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/36 file after the snapshot 2024-03-28T18:59:23.393 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/37 file after the snapshot 2024-03-28T18:59:23.404 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/38 file after the snapshot 2024-03-28T18:59:23.445 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/32 file after the snapshot 2024-03-28T18:59:23.469 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/33 file after the snapshot 2024-03-28T18:59:23.477 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/34 file after the snapshot 2024-03-28T18:59:23.483 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/35 file after the snapshot 2024-03-28T18:59:23.491 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/36 file after the snapshot 2024-03-28T18:59:23.498 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/37 file after the snapshot 2024-03-28T18:59:23.505 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/38 file after the snapshot 2024-03-28T18:59:23.507 INFO:tasks.workunit.client.0.smithi027.stdout: create 295/39 file after the snapshot 2024-03-28T18:59:23.510 INFO:tasks.workunit.client.1.smithi155.stdout: create 140/39 file after the snapshot 2024-03-28T18:59:23.516 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/.snap/snap-subdir-test 2024-03-28T18:59:23.535 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/30 file after the snapshot 2024-03-28T18:59:23.535 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/.snap/snap-subdir-test 2024-03-28T18:59:23.539 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/31 file after the snapshot 2024-03-28T18:59:23.546 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/32 file after the snapshot 2024-03-28T18:59:23.553 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/33 file after the snapshot 2024-03-28T18:59:23.558 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/34 file after the snapshot 2024-03-28T18:59:23.559 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/30 file after the snapshot 2024-03-28T18:59:23.563 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/31 file after the snapshot 2024-03-28T18:59:23.564 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/35 file after the snapshot 2024-03-28T18:59:23.570 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/36 file after the snapshot 2024-03-28T18:59:23.571 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/32 file after the snapshot 2024-03-28T18:59:23.576 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/37 file after the snapshot 2024-03-28T18:59:23.578 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/33 file after the snapshot 2024-03-28T18:59:23.582 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/38 file after the snapshot 2024-03-28T18:59:23.585 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/34 file after the snapshot 2024-03-28T18:59:23.588 INFO:tasks.workunit.client.1.smithi155.stdout: create 141/39 file after the snapshot 2024-03-28T18:59:23.595 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/35 file after the snapshot 2024-03-28T18:59:23.595 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/.snap/snap-subdir-test 2024-03-28T18:59:23.602 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/36 file after the snapshot 2024-03-28T18:59:23.611 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/37 file after the snapshot 2024-03-28T18:59:23.614 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/30 file after the snapshot 2024-03-28T18:59:23.618 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/31 file after the snapshot 2024-03-28T18:59:23.620 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/38 file after the snapshot 2024-03-28T18:59:23.624 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/32 file after the snapshot 2024-03-28T18:59:23.630 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/33 file after the snapshot 2024-03-28T18:59:23.631 INFO:tasks.workunit.client.0.smithi027.stdout: create 296/39 file after the snapshot 2024-03-28T18:59:23.635 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/34 file after the snapshot 2024-03-28T18:59:23.640 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/.snap/snap-subdir-test 2024-03-28T18:59:23.641 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/35 file after the snapshot 2024-03-28T18:59:23.646 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/36 file after the snapshot 2024-03-28T18:59:23.652 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/37 file after the snapshot 2024-03-28T18:59:23.658 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/38 file after the snapshot 2024-03-28T18:59:23.666 INFO:tasks.workunit.client.1.smithi155.stdout: create 142/39 file after the snapshot 2024-03-28T18:59:23.670 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/30 file after the snapshot 2024-03-28T18:59:23.672 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/.snap/snap-subdir-test 2024-03-28T18:59:23.674 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/31 file after the snapshot 2024-03-28T18:59:23.682 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/32 file after the snapshot 2024-03-28T18:59:23.692 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/33 file after the snapshot 2024-03-28T18:59:23.692 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/30 file after the snapshot 2024-03-28T18:59:23.695 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/31 file after the snapshot 2024-03-28T18:59:23.701 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/32 file after the snapshot 2024-03-28T18:59:23.702 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/34 file after the snapshot 2024-03-28T18:59:23.706 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/33 file after the snapshot 2024-03-28T18:59:23.709 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/35 file after the snapshot 2024-03-28T18:59:23.712 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/34 file after the snapshot 2024-03-28T18:59:23.717 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/35 file after the snapshot 2024-03-28T18:59:23.718 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/36 file after the snapshot 2024-03-28T18:59:23.724 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/36 file after the snapshot 2024-03-28T18:59:23.727 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/37 file after the snapshot 2024-03-28T18:59:23.729 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/37 file after the snapshot 2024-03-28T18:59:23.735 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/38 file after the snapshot 2024-03-28T18:59:23.736 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/38 file after the snapshot 2024-03-28T18:59:23.741 INFO:tasks.workunit.client.1.smithi155.stdout: create 143/39 file after the snapshot 2024-03-28T18:59:23.745 INFO:tasks.workunit.client.0.smithi027.stdout: create 297/39 file after the snapshot 2024-03-28T18:59:23.747 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/.snap/snap-subdir-test 2024-03-28T18:59:23.756 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/.snap/snap-subdir-test 2024-03-28T18:59:23.766 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/30 file after the snapshot 2024-03-28T18:59:23.770 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/31 file after the snapshot 2024-03-28T18:59:23.777 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/32 file after the snapshot 2024-03-28T18:59:23.783 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/33 file after the snapshot 2024-03-28T18:59:23.789 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/34 file after the snapshot 2024-03-28T18:59:23.790 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/30 file after the snapshot 2024-03-28T18:59:23.795 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/35 file after the snapshot 2024-03-28T18:59:23.796 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/31 file after the snapshot 2024-03-28T18:59:23.803 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/36 file after the snapshot 2024-03-28T18:59:23.807 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/32 file after the snapshot 2024-03-28T18:59:23.809 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/37 file after the snapshot 2024-03-28T18:59:23.814 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/38 file after the snapshot 2024-03-28T18:59:23.817 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/33 file after the snapshot 2024-03-28T18:59:23.820 INFO:tasks.workunit.client.1.smithi155.stdout: create 144/39 file after the snapshot 2024-03-28T18:59:23.826 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/.snap/snap-subdir-test 2024-03-28T18:59:23.829 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/34 file after the snapshot 2024-03-28T18:59:23.839 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/35 file after the snapshot 2024-03-28T18:59:23.849 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/30 file after the snapshot 2024-03-28T18:59:23.850 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/36 file after the snapshot 2024-03-28T18:59:23.853 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/31 file after the snapshot 2024-03-28T18:59:23.859 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/32 file after the snapshot 2024-03-28T18:59:23.860 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/37 file after the snapshot 2024-03-28T18:59:23.865 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/33 file after the snapshot 2024-03-28T18:59:23.871 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/34 file after the snapshot 2024-03-28T18:59:23.872 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/38 file after the snapshot 2024-03-28T18:59:23.877 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/35 file after the snapshot 2024-03-28T18:59:23.882 INFO:tasks.workunit.client.0.smithi027.stdout: create 298/39 file after the snapshot 2024-03-28T18:59:23.882 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/36 file after the snapshot 2024-03-28T18:59:23.889 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/37 file after the snapshot 2024-03-28T18:59:23.891 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/.snap/snap-subdir-test 2024-03-28T18:59:23.895 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/38 file after the snapshot 2024-03-28T18:59:23.901 INFO:tasks.workunit.client.1.smithi155.stdout: create 145/39 file after the snapshot 2024-03-28T18:59:23.907 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/.snap/snap-subdir-test 2024-03-28T18:59:23.915 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/30 file after the snapshot 2024-03-28T18:59:23.919 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/31 file after the snapshot 2024-03-28T18:59:23.926 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/32 file after the snapshot 2024-03-28T18:59:23.928 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/30 file after the snapshot 2024-03-28T18:59:23.931 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/31 file after the snapshot 2024-03-28T18:59:23.933 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/33 file after the snapshot 2024-03-28T18:59:23.937 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/32 file after the snapshot 2024-03-28T18:59:23.943 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/34 file after the snapshot 2024-03-28T18:59:23.943 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/33 file after the snapshot 2024-03-28T18:59:23.948 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/34 file after the snapshot 2024-03-28T18:59:23.951 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/35 file after the snapshot 2024-03-28T18:59:23.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/35 file after the snapshot 2024-03-28T18:59:23.959 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/36 file after the snapshot 2024-03-28T18:59:23.960 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/36 file after the snapshot 2024-03-28T18:59:23.965 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/37 file after the snapshot 2024-03-28T18:59:23.970 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/37 file after the snapshot 2024-03-28T18:59:23.971 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/38 file after the snapshot 2024-03-28T18:59:23.977 INFO:tasks.workunit.client.1.smithi155.stdout: create 146/39 file after the snapshot 2024-03-28T18:59:23.980 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/38 file after the snapshot 2024-03-28T18:59:23.983 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/.snap/snap-subdir-test 2024-03-28T18:59:23.992 INFO:tasks.workunit.client.0.smithi027.stdout: create 299/39 file after the snapshot 2024-03-28T18:59:24.002 INFO:tasks.workunit.client.0.smithi027.stdout:Take second snapshot .snap/test2 2024-03-28T18:59:24.005 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/30 file after the snapshot 2024-03-28T18:59:24.009 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/31 file after the snapshot 2024-03-28T18:59:24.015 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/32 file after the snapshot 2024-03-28T18:59:24.022 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/33 file after the snapshot 2024-03-28T18:59:24.027 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/34 file after the snapshot 2024-03-28T18:59:24.033 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/35 file after the snapshot 2024-03-28T18:59:24.039 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/36 file after the snapshot 2024-03-28T18:59:24.045 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/37 file after the snapshot 2024-03-28T18:59:24.050 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/38 file after the snapshot 2024-03-28T18:59:24.055 INFO:tasks.workunit.client.1.smithi155.stdout: create 147/39 file after the snapshot 2024-03-28T18:59:24.060 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/.snap/snap-subdir-test 2024-03-28T18:59:24.079 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/30 file after the snapshot 2024-03-28T18:59:24.083 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/31 file after the snapshot 2024-03-28T18:59:24.089 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/32 file after the snapshot 2024-03-28T18:59:24.095 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/33 file after the snapshot 2024-03-28T18:59:24.101 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/34 file after the snapshot 2024-03-28T18:59:24.107 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/35 file after the snapshot 2024-03-28T18:59:24.113 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/36 file after the snapshot 2024-03-28T18:59:24.118 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/37 file after the snapshot 2024-03-28T18:59:24.123 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/38 file after the snapshot 2024-03-28T18:59:24.129 INFO:tasks.workunit.client.1.smithi155.stdout: create 148/39 file after the snapshot 2024-03-28T18:59:24.136 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/.snap/snap-subdir-test 2024-03-28T18:59:24.141 INFO:tasks.workunit.client.0.smithi027.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-03-28T18:59:24.163 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/30 file after the snapshot 2024-03-28T18:59:24.166 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/31 file after the snapshot 2024-03-28T18:59:24.172 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/32 file after the snapshot 2024-03-28T18:59:24.177 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/33 file after the snapshot 2024-03-28T18:59:24.183 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/34 file after the snapshot 2024-03-28T18:59:24.189 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/35 file after the snapshot 2024-03-28T18:59:24.196 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/36 file after the snapshot 2024-03-28T18:59:24.204 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/37 file after the snapshot 2024-03-28T18:59:24.210 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/38 file after the snapshot 2024-03-28T18:59:24.216 INFO:tasks.workunit.client.1.smithi155.stdout: create 149/39 file after the snapshot 2024-03-28T18:59:24.222 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/.snap/snap-subdir-test 2024-03-28T18:59:24.241 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/30 file after the snapshot 2024-03-28T18:59:24.245 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/31 file after the snapshot 2024-03-28T18:59:24.251 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/32 file after the snapshot 2024-03-28T18:59:24.256 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/33 file after the snapshot 2024-03-28T18:59:24.262 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/34 file after the snapshot 2024-03-28T18:59:24.269 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/35 file after the snapshot 2024-03-28T18:59:24.275 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/36 file after the snapshot 2024-03-28T18:59:24.280 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/37 file after the snapshot 2024-03-28T18:59:24.287 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/38 file after the snapshot 2024-03-28T18:59:24.293 INFO:tasks.workunit.client.1.smithi155.stdout: create 150/39 file after the snapshot 2024-03-28T18:59:24.299 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/.snap/snap-subdir-test 2024-03-28T18:59:24.317 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/30 file after the snapshot 2024-03-28T18:59:24.320 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/31 file after the snapshot 2024-03-28T18:59:24.326 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/32 file after the snapshot 2024-03-28T18:59:24.332 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/33 file after the snapshot 2024-03-28T18:59:24.339 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/34 file after the snapshot 2024-03-28T18:59:24.345 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/35 file after the snapshot 2024-03-28T18:59:24.352 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/36 file after the snapshot 2024-03-28T18:59:24.357 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/37 file after the snapshot 2024-03-28T18:59:24.363 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/38 file after the snapshot 2024-03-28T18:59:24.368 INFO:tasks.workunit.client.1.smithi155.stdout: create 151/39 file after the snapshot 2024-03-28T18:59:24.373 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/.snap/snap-subdir-test 2024-03-28T18:59:24.534 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/30 file after the snapshot 2024-03-28T18:59:24.537 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/31 file after the snapshot 2024-03-28T18:59:24.544 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/32 file after the snapshot 2024-03-28T18:59:24.552 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/33 file after the snapshot 2024-03-28T18:59:24.559 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/34 file after the snapshot 2024-03-28T18:59:24.565 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/35 file after the snapshot 2024-03-28T18:59:24.571 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/36 file after the snapshot 2024-03-28T18:59:24.577 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/37 file after the snapshot 2024-03-28T18:59:24.583 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/38 file after the snapshot 2024-03-28T18:59:24.590 INFO:tasks.workunit.client.1.smithi155.stdout: create 152/39 file after the snapshot 2024-03-28T18:59:24.596 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/.snap/snap-subdir-test 2024-03-28T18:59:24.617 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/30 file after the snapshot 2024-03-28T18:59:24.623 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/31 file after the snapshot 2024-03-28T18:59:24.629 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/32 file after the snapshot 2024-03-28T18:59:24.635 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/33 file after the snapshot 2024-03-28T18:59:24.642 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/34 file after the snapshot 2024-03-28T18:59:24.649 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/35 file after the snapshot 2024-03-28T18:59:24.658 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/36 file after the snapshot 2024-03-28T18:59:24.665 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/37 file after the snapshot 2024-03-28T18:59:24.672 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/38 file after the snapshot 2024-03-28T18:59:24.678 INFO:tasks.workunit.client.1.smithi155.stdout: create 153/39 file after the snapshot 2024-03-28T18:59:24.684 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/.snap/snap-subdir-test 2024-03-28T18:59:24.709 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/30 file after the snapshot 2024-03-28T18:59:24.713 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/31 file after the snapshot 2024-03-28T18:59:24.720 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/32 file after the snapshot 2024-03-28T18:59:24.725 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/33 file after the snapshot 2024-03-28T18:59:24.731 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/34 file after the snapshot 2024-03-28T18:59:24.737 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/35 file after the snapshot 2024-03-28T18:59:24.742 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/36 file after the snapshot 2024-03-28T18:59:24.747 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/37 file after the snapshot 2024-03-28T18:59:24.754 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/38 file after the snapshot 2024-03-28T18:59:24.760 INFO:tasks.workunit.client.1.smithi155.stdout: create 154/39 file after the snapshot 2024-03-28T18:59:24.766 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/.snap/snap-subdir-test 2024-03-28T18:59:24.786 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/30 file after the snapshot 2024-03-28T18:59:24.789 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/31 file after the snapshot 2024-03-28T18:59:24.795 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/32 file after the snapshot 2024-03-28T18:59:24.800 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/33 file after the snapshot 2024-03-28T18:59:24.807 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/34 file after the snapshot 2024-03-28T18:59:24.812 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/35 file after the snapshot 2024-03-28T18:59:24.818 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/36 file after the snapshot 2024-03-28T18:59:24.824 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/37 file after the snapshot 2024-03-28T18:59:24.831 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/38 file after the snapshot 2024-03-28T18:59:24.838 INFO:tasks.workunit.client.1.smithi155.stdout: create 155/39 file after the snapshot 2024-03-28T18:59:24.843 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/.snap/snap-subdir-test 2024-03-28T18:59:24.866 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/30 file after the snapshot 2024-03-28T18:59:24.869 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/31 file after the snapshot 2024-03-28T18:59:24.874 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/32 file after the snapshot 2024-03-28T18:59:24.880 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/33 file after the snapshot 2024-03-28T18:59:24.885 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/34 file after the snapshot 2024-03-28T18:59:24.892 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/35 file after the snapshot 2024-03-28T18:59:24.897 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/36 file after the snapshot 2024-03-28T18:59:24.903 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/37 file after the snapshot 2024-03-28T18:59:24.910 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/38 file after the snapshot 2024-03-28T18:59:24.915 INFO:tasks.workunit.client.1.smithi155.stdout: create 156/39 file after the snapshot 2024-03-28T18:59:24.920 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/.snap/snap-subdir-test 2024-03-28T18:59:24.943 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/30 file after the snapshot 2024-03-28T18:59:24.947 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/31 file after the snapshot 2024-03-28T18:59:24.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/32 file after the snapshot 2024-03-28T18:59:24.959 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/33 file after the snapshot 2024-03-28T18:59:24.966 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/34 file after the snapshot 2024-03-28T18:59:24.972 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/35 file after the snapshot 2024-03-28T18:59:24.978 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/36 file after the snapshot 2024-03-28T18:59:24.984 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/37 file after the snapshot 2024-03-28T18:59:24.991 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/38 file after the snapshot 2024-03-28T18:59:24.997 INFO:tasks.workunit.client.1.smithi155.stdout: create 157/39 file after the snapshot 2024-03-28T18:59:25.003 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/.snap/snap-subdir-test 2024-03-28T18:59:25.028 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/30 file after the snapshot 2024-03-28T18:59:25.032 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/31 file after the snapshot 2024-03-28T18:59:25.039 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/32 file after the snapshot 2024-03-28T18:59:25.045 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/33 file after the snapshot 2024-03-28T18:59:25.051 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/34 file after the snapshot 2024-03-28T18:59:25.057 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/35 file after the snapshot 2024-03-28T18:59:25.063 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/36 file after the snapshot 2024-03-28T18:59:25.070 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/37 file after the snapshot 2024-03-28T18:59:25.076 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/38 file after the snapshot 2024-03-28T18:59:25.083 INFO:tasks.workunit.client.1.smithi155.stdout: create 158/39 file after the snapshot 2024-03-28T18:59:25.090 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/.snap/snap-subdir-test 2024-03-28T18:59:25.114 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/30 file after the snapshot 2024-03-28T18:59:25.119 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/31 file after the snapshot 2024-03-28T18:59:25.125 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/32 file after the snapshot 2024-03-28T18:59:25.130 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/33 file after the snapshot 2024-03-28T18:59:25.137 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/34 file after the snapshot 2024-03-28T18:59:25.143 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/35 file after the snapshot 2024-03-28T18:59:25.150 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/36 file after the snapshot 2024-03-28T18:59:25.156 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/37 file after the snapshot 2024-03-28T18:59:25.164 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/38 file after the snapshot 2024-03-28T18:59:25.170 INFO:tasks.workunit.client.1.smithi155.stdout: create 159/39 file after the snapshot 2024-03-28T18:59:25.176 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/.snap/snap-subdir-test 2024-03-28T18:59:25.195 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/30 file after the snapshot 2024-03-28T18:59:25.198 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/31 file after the snapshot 2024-03-28T18:59:25.204 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/32 file after the snapshot 2024-03-28T18:59:25.210 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/33 file after the snapshot 2024-03-28T18:59:25.216 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/34 file after the snapshot 2024-03-28T18:59:25.222 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/35 file after the snapshot 2024-03-28T18:59:25.229 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/36 file after the snapshot 2024-03-28T18:59:25.235 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/37 file after the snapshot 2024-03-28T18:59:25.241 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/38 file after the snapshot 2024-03-28T18:59:25.248 INFO:tasks.workunit.client.1.smithi155.stdout: create 160/39 file after the snapshot 2024-03-28T18:59:25.255 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/.snap/snap-subdir-test 2024-03-28T18:59:25.279 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/30 file after the snapshot 2024-03-28T18:59:25.284 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/31 file after the snapshot 2024-03-28T18:59:25.291 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/32 file after the snapshot 2024-03-28T18:59:25.296 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/33 file after the snapshot 2024-03-28T18:59:25.302 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/34 file after the snapshot 2024-03-28T18:59:25.309 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/35 file after the snapshot 2024-03-28T18:59:25.315 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/36 file after the snapshot 2024-03-28T18:59:25.323 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/37 file after the snapshot 2024-03-28T18:59:25.328 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/38 file after the snapshot 2024-03-28T18:59:25.334 INFO:tasks.workunit.client.1.smithi155.stdout: create 161/39 file after the snapshot 2024-03-28T18:59:25.339 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/.snap/snap-subdir-test 2024-03-28T18:59:25.364 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/30 file after the snapshot 2024-03-28T18:59:25.368 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/31 file after the snapshot 2024-03-28T18:59:25.375 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/32 file after the snapshot 2024-03-28T18:59:25.380 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/33 file after the snapshot 2024-03-28T18:59:25.394 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/34 file after the snapshot 2024-03-28T18:59:25.454 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/35 file after the snapshot 2024-03-28T18:59:25.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:59:25.488 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/36 file after the snapshot 2024-03-28T18:59:25.489 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:59:25.495 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/37 file after the snapshot 2024-03-28T18:59:25.501 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/38 file after the snapshot 2024-03-28T18:59:25.508 INFO:tasks.workunit.client.1.smithi155.stdout: create 162/39 file after the snapshot 2024-03-28T18:59:25.514 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/.snap/snap-subdir-test 2024-03-28T18:59:25.538 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/30 file after the snapshot 2024-03-28T18:59:25.541 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/31 file after the snapshot 2024-03-28T18:59:25.548 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/32 file after the snapshot 2024-03-28T18:59:25.554 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/33 file after the snapshot 2024-03-28T18:59:25.560 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/34 file after the snapshot 2024-03-28T18:59:25.567 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/35 file after the snapshot 2024-03-28T18:59:25.573 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/36 file after the snapshot 2024-03-28T18:59:25.578 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/37 file after the snapshot 2024-03-28T18:59:25.584 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/38 file after the snapshot 2024-03-28T18:59:25.589 INFO:tasks.workunit.client.1.smithi155.stdout: create 163/39 file after the snapshot 2024-03-28T18:59:25.596 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/.snap/snap-subdir-test 2024-03-28T18:59:25.616 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/30 file after the snapshot 2024-03-28T18:59:25.620 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/31 file after the snapshot 2024-03-28T18:59:25.627 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/32 file after the snapshot 2024-03-28T18:59:25.633 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/33 file after the snapshot 2024-03-28T18:59:25.638 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/34 file after the snapshot 2024-03-28T18:59:25.644 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/35 file after the snapshot 2024-03-28T18:59:25.649 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/36 file after the snapshot 2024-03-28T18:59:25.655 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/37 file after the snapshot 2024-03-28T18:59:25.661 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/38 file after the snapshot 2024-03-28T18:59:25.668 INFO:tasks.workunit.client.1.smithi155.stdout: create 164/39 file after the snapshot 2024-03-28T18:59:25.674 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/.snap/snap-subdir-test 2024-03-28T18:59:25.697 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/30 file after the snapshot 2024-03-28T18:59:25.701 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/31 file after the snapshot 2024-03-28T18:59:25.708 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/32 file after the snapshot 2024-03-28T18:59:25.713 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/33 file after the snapshot 2024-03-28T18:59:25.719 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/34 file after the snapshot 2024-03-28T18:59:25.725 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/35 file after the snapshot 2024-03-28T18:59:25.730 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/36 file after the snapshot 2024-03-28T18:59:25.735 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/37 file after the snapshot 2024-03-28T18:59:25.741 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/38 file after the snapshot 2024-03-28T18:59:25.747 INFO:tasks.workunit.client.1.smithi155.stdout: create 165/39 file after the snapshot 2024-03-28T18:59:25.752 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/.snap/snap-subdir-test 2024-03-28T18:59:25.778 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/30 file after the snapshot 2024-03-28T18:59:25.781 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/31 file after the snapshot 2024-03-28T18:59:25.787 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/32 file after the snapshot 2024-03-28T18:59:25.793 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/33 file after the snapshot 2024-03-28T18:59:25.801 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/34 file after the snapshot 2024-03-28T18:59:25.808 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/35 file after the snapshot 2024-03-28T18:59:25.816 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/36 file after the snapshot 2024-03-28T18:59:25.822 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/37 file after the snapshot 2024-03-28T18:59:25.828 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/38 file after the snapshot 2024-03-28T18:59:25.835 INFO:tasks.workunit.client.1.smithi155.stdout: create 166/39 file after the snapshot 2024-03-28T18:59:25.841 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/.snap/snap-subdir-test 2024-03-28T18:59:25.863 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/30 file after the snapshot 2024-03-28T18:59:25.867 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/31 file after the snapshot 2024-03-28T18:59:25.874 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/32 file after the snapshot 2024-03-28T18:59:25.881 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/33 file after the snapshot 2024-03-28T18:59:25.887 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/34 file after the snapshot 2024-03-28T18:59:25.894 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/35 file after the snapshot 2024-03-28T18:59:25.900 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/36 file after the snapshot 2024-03-28T18:59:25.906 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/37 file after the snapshot 2024-03-28T18:59:25.912 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/38 file after the snapshot 2024-03-28T18:59:25.918 INFO:tasks.workunit.client.1.smithi155.stdout: create 167/39 file after the snapshot 2024-03-28T18:59:25.925 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/.snap/snap-subdir-test 2024-03-28T18:59:25.948 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/30 file after the snapshot 2024-03-28T18:59:25.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/31 file after the snapshot 2024-03-28T18:59:25.959 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/32 file after the snapshot 2024-03-28T18:59:25.965 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/33 file after the snapshot 2024-03-28T18:59:25.970 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/34 file after the snapshot 2024-03-28T18:59:25.976 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/35 file after the snapshot 2024-03-28T18:59:25.983 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/36 file after the snapshot 2024-03-28T18:59:25.990 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/37 file after the snapshot 2024-03-28T18:59:25.995 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/38 file after the snapshot 2024-03-28T18:59:26.001 INFO:tasks.workunit.client.1.smithi155.stdout: create 168/39 file after the snapshot 2024-03-28T18:59:26.008 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/.snap/snap-subdir-test 2024-03-28T18:59:26.029 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/30 file after the snapshot 2024-03-28T18:59:26.032 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/31 file after the snapshot 2024-03-28T18:59:26.039 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/32 file after the snapshot 2024-03-28T18:59:26.046 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/33 file after the snapshot 2024-03-28T18:59:26.052 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/34 file after the snapshot 2024-03-28T18:59:26.058 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/35 file after the snapshot 2024-03-28T18:59:26.066 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/36 file after the snapshot 2024-03-28T18:59:26.072 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/37 file after the snapshot 2024-03-28T18:59:26.080 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/38 file after the snapshot 2024-03-28T18:59:26.086 INFO:tasks.workunit.client.1.smithi155.stdout: create 169/39 file after the snapshot 2024-03-28T18:59:26.093 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/.snap/snap-subdir-test 2024-03-28T18:59:26.117 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/30 file after the snapshot 2024-03-28T18:59:26.122 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/31 file after the snapshot 2024-03-28T18:59:26.128 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/32 file after the snapshot 2024-03-28T18:59:26.134 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/33 file after the snapshot 2024-03-28T18:59:26.141 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/34 file after the snapshot 2024-03-28T18:59:26.147 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/35 file after the snapshot 2024-03-28T18:59:26.154 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/36 file after the snapshot 2024-03-28T18:59:26.159 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/37 file after the snapshot 2024-03-28T18:59:26.165 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/38 file after the snapshot 2024-03-28T18:59:26.171 INFO:tasks.workunit.client.1.smithi155.stdout: create 170/39 file after the snapshot 2024-03-28T18:59:26.177 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/.snap/snap-subdir-test 2024-03-28T18:59:26.203 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/30 file after the snapshot 2024-03-28T18:59:26.206 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/31 file after the snapshot 2024-03-28T18:59:26.212 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/32 file after the snapshot 2024-03-28T18:59:26.218 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/33 file after the snapshot 2024-03-28T18:59:26.224 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/34 file after the snapshot 2024-03-28T18:59:26.231 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/35 file after the snapshot 2024-03-28T18:59:26.237 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/36 file after the snapshot 2024-03-28T18:59:26.244 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/37 file after the snapshot 2024-03-28T18:59:26.250 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/38 file after the snapshot 2024-03-28T18:59:26.257 INFO:tasks.workunit.client.1.smithi155.stdout: create 171/39 file after the snapshot 2024-03-28T18:59:26.263 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/.snap/snap-subdir-test 2024-03-28T18:59:26.285 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/30 file after the snapshot 2024-03-28T18:59:26.289 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/31 file after the snapshot 2024-03-28T18:59:26.294 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/32 file after the snapshot 2024-03-28T18:59:26.300 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/33 file after the snapshot 2024-03-28T18:59:26.307 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/34 file after the snapshot 2024-03-28T18:59:26.313 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/35 file after the snapshot 2024-03-28T18:59:26.318 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/36 file after the snapshot 2024-03-28T18:59:26.325 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/37 file after the snapshot 2024-03-28T18:59:26.333 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/38 file after the snapshot 2024-03-28T18:59:26.340 INFO:tasks.workunit.client.1.smithi155.stdout: create 172/39 file after the snapshot 2024-03-28T18:59:26.345 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/.snap/snap-subdir-test 2024-03-28T18:59:26.365 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/30 file after the snapshot 2024-03-28T18:59:26.369 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/31 file after the snapshot 2024-03-28T18:59:26.374 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/32 file after the snapshot 2024-03-28T18:59:26.380 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/33 file after the snapshot 2024-03-28T18:59:26.385 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/34 file after the snapshot 2024-03-28T18:59:26.391 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/35 file after the snapshot 2024-03-28T18:59:26.449 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/36 file after the snapshot 2024-03-28T18:59:26.479 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/37 file after the snapshot 2024-03-28T18:59:26.485 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/38 file after the snapshot 2024-03-28T18:59:26.492 INFO:tasks.workunit.client.1.smithi155.stdout: create 173/39 file after the snapshot 2024-03-28T18:59:26.498 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/.snap/snap-subdir-test 2024-03-28T18:59:26.523 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/30 file after the snapshot 2024-03-28T18:59:26.526 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/31 file after the snapshot 2024-03-28T18:59:26.532 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/32 file after the snapshot 2024-03-28T18:59:26.538 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/33 file after the snapshot 2024-03-28T18:59:26.543 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/34 file after the snapshot 2024-03-28T18:59:26.546 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-03-28T18:59:26.547 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-03-28T18:59:26.548 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-03-28T18:59:26.549 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-03-28T18:59:26.550 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-03-28T18:59:26.551 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-03-28T18:59:26.552 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-03-28T18:59:26.552 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-03-28T18:59:26.552 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-03-28T18:59:26.552 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-03-28T18:59:26.552 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-03-28T18:59:26.552 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-03-28T18:59:26.552 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/35 file after the snapshot 2024-03-28T18:59:26.554 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/36 file after the snapshot 2024-03-28T18:59:26.560 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/37 file after the snapshot 2024-03-28T18:59:26.565 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/38 file after the snapshot 2024-03-28T18:59:26.571 INFO:tasks.workunit.client.1.smithi155.stdout: create 174/39 file after the snapshot 2024-03-28T18:59:26.577 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/.snap/snap-subdir-test 2024-03-28T18:59:26.595 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/30 file after the snapshot 2024-03-28T18:59:26.598 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/31 file after the snapshot 2024-03-28T18:59:26.605 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/32 file after the snapshot 2024-03-28T18:59:26.610 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/33 file after the snapshot 2024-03-28T18:59:26.616 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/34 file after the snapshot 2024-03-28T18:59:26.622 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/35 file after the snapshot 2024-03-28T18:59:26.628 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/36 file after the snapshot 2024-03-28T18:59:26.633 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/37 file after the snapshot 2024-03-28T18:59:26.640 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/38 file after the snapshot 2024-03-28T18:59:26.646 INFO:tasks.workunit.client.1.smithi155.stdout: create 175/39 file after the snapshot 2024-03-28T18:59:26.652 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/.snap/snap-subdir-test 2024-03-28T18:59:26.677 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/30 file after the snapshot 2024-03-28T18:59:26.680 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/31 file after the snapshot 2024-03-28T18:59:26.685 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/32 file after the snapshot 2024-03-28T18:59:26.691 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/33 file after the snapshot 2024-03-28T18:59:26.696 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/34 file after the snapshot 2024-03-28T18:59:26.703 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/35 file after the snapshot 2024-03-28T18:59:26.708 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/36 file after the snapshot 2024-03-28T18:59:26.714 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/37 file after the snapshot 2024-03-28T18:59:26.719 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/38 file after the snapshot 2024-03-28T18:59:26.725 INFO:tasks.workunit.client.1.smithi155.stdout: create 176/39 file after the snapshot 2024-03-28T18:59:26.730 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/.snap/snap-subdir-test 2024-03-28T18:59:26.750 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/30 file after the snapshot 2024-03-28T18:59:26.753 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/31 file after the snapshot 2024-03-28T18:59:26.759 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/32 file after the snapshot 2024-03-28T18:59:26.766 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/33 file after the snapshot 2024-03-28T18:59:26.771 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/34 file after the snapshot 2024-03-28T18:59:26.777 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/35 file after the snapshot 2024-03-28T18:59:26.782 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/36 file after the snapshot 2024-03-28T18:59:26.788 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/37 file after the snapshot 2024-03-28T18:59:26.794 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/38 file after the snapshot 2024-03-28T18:59:26.800 INFO:tasks.workunit.client.1.smithi155.stdout: create 177/39 file after the snapshot 2024-03-28T18:59:26.806 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/.snap/snap-subdir-test 2024-03-28T18:59:26.829 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/30 file after the snapshot 2024-03-28T18:59:26.833 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/31 file after the snapshot 2024-03-28T18:59:26.840 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/32 file after the snapshot 2024-03-28T18:59:26.845 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/33 file after the snapshot 2024-03-28T18:59:26.849 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/34 file after the snapshot 2024-03-28T18:59:26.855 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/35 file after the snapshot 2024-03-28T18:59:26.862 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/36 file after the snapshot 2024-03-28T18:59:26.869 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/37 file after the snapshot 2024-03-28T18:59:26.876 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/38 file after the snapshot 2024-03-28T18:59:26.882 INFO:tasks.workunit.client.1.smithi155.stdout: create 178/39 file after the snapshot 2024-03-28T18:59:26.887 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/.snap/snap-subdir-test 2024-03-28T18:59:26.906 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/30 file after the snapshot 2024-03-28T18:59:26.909 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/31 file after the snapshot 2024-03-28T18:59:26.915 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/32 file after the snapshot 2024-03-28T18:59:26.920 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/33 file after the snapshot 2024-03-28T18:59:26.926 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/34 file after the snapshot 2024-03-28T18:59:26.931 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/35 file after the snapshot 2024-03-28T18:59:26.937 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/36 file after the snapshot 2024-03-28T18:59:26.944 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/37 file after the snapshot 2024-03-28T18:59:26.950 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/38 file after the snapshot 2024-03-28T18:59:26.957 INFO:tasks.workunit.client.1.smithi155.stdout: create 179/39 file after the snapshot 2024-03-28T18:59:26.962 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/.snap/snap-subdir-test 2024-03-28T18:59:26.982 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/30 file after the snapshot 2024-03-28T18:59:26.986 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/31 file after the snapshot 2024-03-28T18:59:26.992 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/32 file after the snapshot 2024-03-28T18:59:26.998 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/33 file after the snapshot 2024-03-28T18:59:27.003 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/34 file after the snapshot 2024-03-28T18:59:27.009 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/35 file after the snapshot 2024-03-28T18:59:27.014 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/36 file after the snapshot 2024-03-28T18:59:27.021 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/37 file after the snapshot 2024-03-28T18:59:27.027 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/38 file after the snapshot 2024-03-28T18:59:27.033 INFO:tasks.workunit.client.1.smithi155.stdout: create 180/39 file after the snapshot 2024-03-28T18:59:27.038 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/.snap/snap-subdir-test 2024-03-28T18:59:27.058 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/30 file after the snapshot 2024-03-28T18:59:27.061 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/31 file after the snapshot 2024-03-28T18:59:27.070 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/32 file after the snapshot 2024-03-28T18:59:27.076 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/33 file after the snapshot 2024-03-28T18:59:27.082 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/34 file after the snapshot 2024-03-28T18:59:27.087 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/35 file after the snapshot 2024-03-28T18:59:27.093 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/36 file after the snapshot 2024-03-28T18:59:27.099 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/37 file after the snapshot 2024-03-28T18:59:27.105 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/38 file after the snapshot 2024-03-28T18:59:27.111 INFO:tasks.workunit.client.1.smithi155.stdout: create 181/39 file after the snapshot 2024-03-28T18:59:27.118 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/.snap/snap-subdir-test 2024-03-28T18:59:27.142 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/30 file after the snapshot 2024-03-28T18:59:27.144 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/31 file after the snapshot 2024-03-28T18:59:27.151 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/32 file after the snapshot 2024-03-28T18:59:27.157 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/33 file after the snapshot 2024-03-28T18:59:27.163 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/34 file after the snapshot 2024-03-28T18:59:27.169 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/35 file after the snapshot 2024-03-28T18:59:27.176 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/36 file after the snapshot 2024-03-28T18:59:27.182 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/37 file after the snapshot 2024-03-28T18:59:27.188 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/38 file after the snapshot 2024-03-28T18:59:27.194 INFO:tasks.workunit.client.1.smithi155.stdout: create 182/39 file after the snapshot 2024-03-28T18:59:27.199 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/.snap/snap-subdir-test 2024-03-28T18:59:27.218 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/30 file after the snapshot 2024-03-28T18:59:27.221 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/31 file after the snapshot 2024-03-28T18:59:27.229 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/32 file after the snapshot 2024-03-28T18:59:27.237 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/33 file after the snapshot 2024-03-28T18:59:27.246 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/34 file after the snapshot 2024-03-28T18:59:27.252 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/35 file after the snapshot 2024-03-28T18:59:27.258 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/36 file after the snapshot 2024-03-28T18:59:27.264 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/37 file after the snapshot 2024-03-28T18:59:27.273 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/38 file after the snapshot 2024-03-28T18:59:27.280 INFO:tasks.workunit.client.1.smithi155.stdout: create 183/39 file after the snapshot 2024-03-28T18:59:27.288 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/.snap/snap-subdir-test 2024-03-28T18:59:27.320 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/30 file after the snapshot 2024-03-28T18:59:27.325 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/31 file after the snapshot 2024-03-28T18:59:27.334 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/32 file after the snapshot 2024-03-28T18:59:27.344 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/33 file after the snapshot 2024-03-28T18:59:27.354 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/34 file after the snapshot 2024-03-28T18:59:27.364 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/35 file after the snapshot 2024-03-28T18:59:27.373 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/36 file after the snapshot 2024-03-28T18:59:27.382 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/37 file after the snapshot 2024-03-28T18:59:27.391 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/38 file after the snapshot 2024-03-28T18:59:27.398 INFO:tasks.workunit.client.1.smithi155.stdout: create 184/39 file after the snapshot 2024-03-28T18:59:27.413 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/.snap/snap-subdir-test 2024-03-28T18:59:27.497 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/30 file after the snapshot 2024-03-28T18:59:27.501 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/31 file after the snapshot 2024-03-28T18:59:27.507 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/32 file after the snapshot 2024-03-28T18:59:27.513 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/33 file after the snapshot 2024-03-28T18:59:27.520 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/34 file after the snapshot 2024-03-28T18:59:27.528 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/35 file after the snapshot 2024-03-28T18:59:27.536 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/36 file after the snapshot 2024-03-28T18:59:27.544 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/37 file after the snapshot 2024-03-28T18:59:27.550 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/38 file after the snapshot 2024-03-28T18:59:27.557 INFO:tasks.workunit.client.1.smithi155.stdout: create 185/39 file after the snapshot 2024-03-28T18:59:27.562 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/.snap/snap-subdir-test 2024-03-28T18:59:27.584 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/30 file after the snapshot 2024-03-28T18:59:27.588 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/31 file after the snapshot 2024-03-28T18:59:27.596 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/32 file after the snapshot 2024-03-28T18:59:27.602 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/33 file after the snapshot 2024-03-28T18:59:27.608 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/34 file after the snapshot 2024-03-28T18:59:27.615 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/35 file after the snapshot 2024-03-28T18:59:27.623 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/36 file after the snapshot 2024-03-28T18:59:27.629 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/37 file after the snapshot 2024-03-28T18:59:27.637 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/38 file after the snapshot 2024-03-28T18:59:27.643 INFO:tasks.workunit.client.1.smithi155.stdout: create 186/39 file after the snapshot 2024-03-28T18:59:27.649 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/.snap/snap-subdir-test 2024-03-28T18:59:27.676 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/30 file after the snapshot 2024-03-28T18:59:27.679 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/31 file after the snapshot 2024-03-28T18:59:27.686 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/32 file after the snapshot 2024-03-28T18:59:27.691 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/33 file after the snapshot 2024-03-28T18:59:27.698 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/34 file after the snapshot 2024-03-28T18:59:27.704 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/35 file after the snapshot 2024-03-28T18:59:27.710 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/36 file after the snapshot 2024-03-28T18:59:27.716 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/37 file after the snapshot 2024-03-28T18:59:27.720 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/38 file after the snapshot 2024-03-28T18:59:27.726 INFO:tasks.workunit.client.1.smithi155.stdout: create 187/39 file after the snapshot 2024-03-28T18:59:27.733 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/.snap/snap-subdir-test 2024-03-28T18:59:27.761 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/30 file after the snapshot 2024-03-28T18:59:27.764 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/31 file after the snapshot 2024-03-28T18:59:27.770 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/32 file after the snapshot 2024-03-28T18:59:27.776 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/33 file after the snapshot 2024-03-28T18:59:27.782 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/34 file after the snapshot 2024-03-28T18:59:27.789 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/35 file after the snapshot 2024-03-28T18:59:27.796 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/36 file after the snapshot 2024-03-28T18:59:27.803 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/37 file after the snapshot 2024-03-28T18:59:27.809 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/38 file after the snapshot 2024-03-28T18:59:27.816 INFO:tasks.workunit.client.1.smithi155.stdout: create 188/39 file after the snapshot 2024-03-28T18:59:27.823 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/.snap/snap-subdir-test 2024-03-28T18:59:27.848 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/30 file after the snapshot 2024-03-28T18:59:27.853 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/31 file after the snapshot 2024-03-28T18:59:27.862 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/32 file after the snapshot 2024-03-28T18:59:27.869 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/33 file after the snapshot 2024-03-28T18:59:27.877 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/34 file after the snapshot 2024-03-28T18:59:27.884 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/35 file after the snapshot 2024-03-28T18:59:27.891 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/36 file after the snapshot 2024-03-28T18:59:27.900 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/37 file after the snapshot 2024-03-28T18:59:27.909 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/38 file after the snapshot 2024-03-28T18:59:27.915 INFO:tasks.workunit.client.1.smithi155.stdout: create 189/39 file after the snapshot 2024-03-28T18:59:27.922 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/.snap/snap-subdir-test 2024-03-28T18:59:27.942 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/30 file after the snapshot 2024-03-28T18:59:27.946 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/31 file after the snapshot 2024-03-28T18:59:27.954 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/32 file after the snapshot 2024-03-28T18:59:27.961 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/33 file after the snapshot 2024-03-28T18:59:27.969 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/34 file after the snapshot 2024-03-28T18:59:27.974 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/35 file after the snapshot 2024-03-28T18:59:27.981 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/36 file after the snapshot 2024-03-28T18:59:27.988 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/37 file after the snapshot 2024-03-28T18:59:27.995 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/38 file after the snapshot 2024-03-28T18:59:28.003 INFO:tasks.workunit.client.1.smithi155.stdout: create 190/39 file after the snapshot 2024-03-28T18:59:28.012 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/.snap/snap-subdir-test 2024-03-28T18:59:28.034 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/30 file after the snapshot 2024-03-28T18:59:28.039 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/31 file after the snapshot 2024-03-28T18:59:28.047 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/32 file after the snapshot 2024-03-28T18:59:28.055 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/33 file after the snapshot 2024-03-28T18:59:28.061 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/34 file after the snapshot 2024-03-28T18:59:28.070 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/35 file after the snapshot 2024-03-28T18:59:28.077 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/36 file after the snapshot 2024-03-28T18:59:28.083 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/37 file after the snapshot 2024-03-28T18:59:28.089 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/38 file after the snapshot 2024-03-28T18:59:28.097 INFO:tasks.workunit.client.1.smithi155.stdout: create 191/39 file after the snapshot 2024-03-28T18:59:28.106 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/.snap/snap-subdir-test 2024-03-28T18:59:28.141 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/30 file after the snapshot 2024-03-28T18:59:28.146 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/31 file after the snapshot 2024-03-28T18:59:28.153 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/32 file after the snapshot 2024-03-28T18:59:28.161 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/33 file after the snapshot 2024-03-28T18:59:28.169 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/34 file after the snapshot 2024-03-28T18:59:28.174 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-03-28T18:59:28.174 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-03-28T18:59:28.174 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-03-28T18:59:28.174 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-03-28T18:59:28.175 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-03-28T18:59:28.176 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-03-28T18:59:28.177 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-03-28T18:59:28.178 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-03-28T18:59:28.179 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-03-28T18:59:28.180 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-03-28T18:59:28.180 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-03-28T18:59:28.180 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-03-28T18:59:28.180 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-03-28T18:59:28.180 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-03-28T18:59:28.180 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/35 file after the snapshot 2024-03-28T18:59:28.186 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/36 file after the snapshot 2024-03-28T18:59:28.193 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/37 file after the snapshot 2024-03-28T18:59:28.198 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/38 file after the snapshot 2024-03-28T18:59:28.205 INFO:tasks.workunit.client.1.smithi155.stdout: create 192/39 file after the snapshot 2024-03-28T18:59:28.213 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/.snap/snap-subdir-test 2024-03-28T18:59:28.233 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/30 file after the snapshot 2024-03-28T18:59:28.236 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/31 file after the snapshot 2024-03-28T18:59:28.245 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/32 file after the snapshot 2024-03-28T18:59:28.252 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/33 file after the snapshot 2024-03-28T18:59:28.259 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/34 file after the snapshot 2024-03-28T18:59:28.265 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/35 file after the snapshot 2024-03-28T18:59:28.271 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/36 file after the snapshot 2024-03-28T18:59:28.279 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/37 file after the snapshot 2024-03-28T18:59:28.288 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/38 file after the snapshot 2024-03-28T18:59:28.295 INFO:tasks.workunit.client.1.smithi155.stdout: create 193/39 file after the snapshot 2024-03-28T18:59:28.302 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/.snap/snap-subdir-test 2024-03-28T18:59:28.332 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/30 file after the snapshot 2024-03-28T18:59:28.335 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/31 file after the snapshot 2024-03-28T18:59:28.343 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/32 file after the snapshot 2024-03-28T18:59:28.350 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/33 file after the snapshot 2024-03-28T18:59:28.357 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/34 file after the snapshot 2024-03-28T18:59:28.363 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/35 file after the snapshot 2024-03-28T18:59:28.371 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/36 file after the snapshot 2024-03-28T18:59:28.378 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/37 file after the snapshot 2024-03-28T18:59:28.385 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/38 file after the snapshot 2024-03-28T18:59:28.392 INFO:tasks.workunit.client.1.smithi155.stdout: create 194/39 file after the snapshot 2024-03-28T18:59:28.400 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/.snap/snap-subdir-test 2024-03-28T18:59:28.430 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/30 file after the snapshot 2024-03-28T18:59:28.485 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/31 file after the snapshot 2024-03-28T18:59:28.529 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/32 file after the snapshot 2024-03-28T18:59:28.536 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/33 file after the snapshot 2024-03-28T18:59:28.543 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/34 file after the snapshot 2024-03-28T18:59:28.550 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/35 file after the snapshot 2024-03-28T18:59:28.560 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/36 file after the snapshot 2024-03-28T18:59:28.569 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/37 file after the snapshot 2024-03-28T18:59:28.579 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/38 file after the snapshot 2024-03-28T18:59:28.587 INFO:tasks.workunit.client.1.smithi155.stdout: create 195/39 file after the snapshot 2024-03-28T18:59:28.595 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/.snap/snap-subdir-test 2024-03-28T18:59:28.618 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/30 file after the snapshot 2024-03-28T18:59:28.621 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/31 file after the snapshot 2024-03-28T18:59:28.627 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/32 file after the snapshot 2024-03-28T18:59:28.633 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/33 file after the snapshot 2024-03-28T18:59:28.639 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/34 file after the snapshot 2024-03-28T18:59:28.646 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/35 file after the snapshot 2024-03-28T18:59:28.652 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/36 file after the snapshot 2024-03-28T18:59:28.657 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/37 file after the snapshot 2024-03-28T18:59:28.663 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/38 file after the snapshot 2024-03-28T18:59:28.670 INFO:tasks.workunit.client.1.smithi155.stdout: create 196/39 file after the snapshot 2024-03-28T18:59:28.676 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/.snap/snap-subdir-test 2024-03-28T18:59:28.697 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/30 file after the snapshot 2024-03-28T18:59:28.700 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/31 file after the snapshot 2024-03-28T18:59:28.708 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/32 file after the snapshot 2024-03-28T18:59:28.714 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/33 file after the snapshot 2024-03-28T18:59:28.720 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/34 file after the snapshot 2024-03-28T18:59:28.726 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/35 file after the snapshot 2024-03-28T18:59:28.733 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/36 file after the snapshot 2024-03-28T18:59:28.740 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/37 file after the snapshot 2024-03-28T18:59:28.747 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/38 file after the snapshot 2024-03-28T18:59:28.755 INFO:tasks.workunit.client.1.smithi155.stdout: create 197/39 file after the snapshot 2024-03-28T18:59:28.763 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/.snap/snap-subdir-test 2024-03-28T18:59:28.786 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/30 file after the snapshot 2024-03-28T18:59:28.791 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/31 file after the snapshot 2024-03-28T18:59:28.798 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/32 file after the snapshot 2024-03-28T18:59:28.805 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/33 file after the snapshot 2024-03-28T18:59:28.811 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/34 file after the snapshot 2024-03-28T18:59:28.817 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/35 file after the snapshot 2024-03-28T18:59:28.823 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/36 file after the snapshot 2024-03-28T18:59:28.829 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/37 file after the snapshot 2024-03-28T18:59:28.836 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/38 file after the snapshot 2024-03-28T18:59:28.843 INFO:tasks.workunit.client.1.smithi155.stdout: create 198/39 file after the snapshot 2024-03-28T18:59:28.849 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/.snap/snap-subdir-test 2024-03-28T18:59:28.878 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/30 file after the snapshot 2024-03-28T18:59:28.883 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/31 file after the snapshot 2024-03-28T18:59:28.890 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/32 file after the snapshot 2024-03-28T18:59:28.897 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/33 file after the snapshot 2024-03-28T18:59:28.906 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/34 file after the snapshot 2024-03-28T18:59:28.915 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/35 file after the snapshot 2024-03-28T18:59:28.924 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/36 file after the snapshot 2024-03-28T18:59:28.932 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/37 file after the snapshot 2024-03-28T18:59:28.939 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/38 file after the snapshot 2024-03-28T18:59:28.948 INFO:tasks.workunit.client.1.smithi155.stdout: create 199/39 file after the snapshot 2024-03-28T18:59:28.957 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/.snap/snap-subdir-test 2024-03-28T18:59:28.984 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/30 file after the snapshot 2024-03-28T18:59:28.990 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/31 file after the snapshot 2024-03-28T18:59:28.997 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/32 file after the snapshot 2024-03-28T18:59:29.003 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/33 file after the snapshot 2024-03-28T18:59:29.012 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/34 file after the snapshot 2024-03-28T18:59:29.022 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/35 file after the snapshot 2024-03-28T18:59:29.030 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/36 file after the snapshot 2024-03-28T18:59:29.037 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/37 file after the snapshot 2024-03-28T18:59:29.045 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/38 file after the snapshot 2024-03-28T18:59:29.053 INFO:tasks.workunit.client.1.smithi155.stdout: create 200/39 file after the snapshot 2024-03-28T18:59:29.060 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/.snap/snap-subdir-test 2024-03-28T18:59:29.081 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/30 file after the snapshot 2024-03-28T18:59:29.085 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/31 file after the snapshot 2024-03-28T18:59:29.090 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/32 file after the snapshot 2024-03-28T18:59:29.097 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/33 file after the snapshot 2024-03-28T18:59:29.104 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/34 file after the snapshot 2024-03-28T18:59:29.110 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/35 file after the snapshot 2024-03-28T18:59:29.118 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/36 file after the snapshot 2024-03-28T18:59:29.124 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/37 file after the snapshot 2024-03-28T18:59:29.132 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/38 file after the snapshot 2024-03-28T18:59:29.138 INFO:tasks.workunit.client.1.smithi155.stdout: create 201/39 file after the snapshot 2024-03-28T18:59:29.145 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/.snap/snap-subdir-test 2024-03-28T18:59:29.169 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/30 file after the snapshot 2024-03-28T18:59:29.173 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/31 file after the snapshot 2024-03-28T18:59:29.181 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/32 file after the snapshot 2024-03-28T18:59:29.190 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/33 file after the snapshot 2024-03-28T18:59:29.197 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/34 file after the snapshot 2024-03-28T18:59:29.204 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/35 file after the snapshot 2024-03-28T18:59:29.212 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/36 file after the snapshot 2024-03-28T18:59:29.218 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/37 file after the snapshot 2024-03-28T18:59:29.224 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/38 file after the snapshot 2024-03-28T18:59:29.232 INFO:tasks.workunit.client.1.smithi155.stdout: create 202/39 file after the snapshot 2024-03-28T18:59:29.241 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/.snap/snap-subdir-test 2024-03-28T18:59:29.262 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/30 file after the snapshot 2024-03-28T18:59:29.266 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/31 file after the snapshot 2024-03-28T18:59:29.273 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/32 file after the snapshot 2024-03-28T18:59:29.279 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/33 file after the snapshot 2024-03-28T18:59:29.284 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/34 file after the snapshot 2024-03-28T18:59:29.291 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/35 file after the snapshot 2024-03-28T18:59:29.299 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/36 file after the snapshot 2024-03-28T18:59:29.306 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/37 file after the snapshot 2024-03-28T18:59:29.313 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/38 file after the snapshot 2024-03-28T18:59:29.321 INFO:tasks.workunit.client.1.smithi155.stdout: create 203/39 file after the snapshot 2024-03-28T18:59:29.328 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/.snap/snap-subdir-test 2024-03-28T18:59:29.350 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/30 file after the snapshot 2024-03-28T18:59:29.355 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/31 file after the snapshot 2024-03-28T18:59:29.362 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/32 file after the snapshot 2024-03-28T18:59:29.368 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/33 file after the snapshot 2024-03-28T18:59:29.378 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/34 file after the snapshot 2024-03-28T18:59:29.387 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/35 file after the snapshot 2024-03-28T18:59:29.396 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/36 file after the snapshot 2024-03-28T18:59:29.405 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/37 file after the snapshot 2024-03-28T18:59:29.414 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/38 file after the snapshot 2024-03-28T18:59:29.429 INFO:tasks.workunit.client.1.smithi155.stdout: create 204/39 file after the snapshot 2024-03-28T18:59:29.474 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/.snap/snap-subdir-test 2024-03-28T18:59:29.512 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/30 file after the snapshot 2024-03-28T18:59:29.517 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/31 file after the snapshot 2024-03-28T18:59:29.524 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/32 file after the snapshot 2024-03-28T18:59:29.530 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/33 file after the snapshot 2024-03-28T18:59:29.537 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/34 file after the snapshot 2024-03-28T18:59:29.543 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/35 file after the snapshot 2024-03-28T18:59:29.550 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/36 file after the snapshot 2024-03-28T18:59:29.555 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/37 file after the snapshot 2024-03-28T18:59:29.561 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/38 file after the snapshot 2024-03-28T18:59:29.569 INFO:tasks.workunit.client.1.smithi155.stdout: create 205/39 file after the snapshot 2024-03-28T18:59:29.575 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/.snap/snap-subdir-test 2024-03-28T18:59:29.601 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/30 file after the snapshot 2024-03-28T18:59:29.605 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/31 file after the snapshot 2024-03-28T18:59:29.612 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/32 file after the snapshot 2024-03-28T18:59:29.619 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/33 file after the snapshot 2024-03-28T18:59:29.626 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/34 file after the snapshot 2024-03-28T18:59:29.632 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/35 file after the snapshot 2024-03-28T18:59:29.640 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/36 file after the snapshot 2024-03-28T18:59:29.647 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/37 file after the snapshot 2024-03-28T18:59:29.654 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/38 file after the snapshot 2024-03-28T18:59:29.660 INFO:tasks.workunit.client.1.smithi155.stdout: create 206/39 file after the snapshot 2024-03-28T18:59:29.665 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/.snap/snap-subdir-test 2024-03-28T18:59:29.686 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/30 file after the snapshot 2024-03-28T18:59:29.690 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/31 file after the snapshot 2024-03-28T18:59:29.697 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/32 file after the snapshot 2024-03-28T18:59:29.702 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/33 file after the snapshot 2024-03-28T18:59:29.707 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/34 file after the snapshot 2024-03-28T18:59:29.714 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/35 file after the snapshot 2024-03-28T18:59:29.721 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/36 file after the snapshot 2024-03-28T18:59:29.728 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/37 file after the snapshot 2024-03-28T18:59:29.735 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/38 file after the snapshot 2024-03-28T18:59:29.742 INFO:tasks.workunit.client.1.smithi155.stdout: create 207/39 file after the snapshot 2024-03-28T18:59:29.750 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/.snap/snap-subdir-test 2024-03-28T18:59:29.775 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/30 file after the snapshot 2024-03-28T18:59:29.779 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/31 file after the snapshot 2024-03-28T18:59:29.786 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/32 file after the snapshot 2024-03-28T18:59:29.793 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/33 file after the snapshot 2024-03-28T18:59:29.799 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/34 file after the snapshot 2024-03-28T18:59:29.807 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/35 file after the snapshot 2024-03-28T18:59:29.809 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-03-28T18:59:29.809 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-03-28T18:59:29.809 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-03-28T18:59:29.809 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-03-28T18:59:29.810 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-03-28T18:59:29.811 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-03-28T18:59:29.812 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-03-28T18:59:29.813 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-03-28T18:59:29.814 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-03-28T18:59:29.815 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/36 file after the snapshot 2024-03-28T18:59:29.820 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/37 file after the snapshot 2024-03-28T18:59:29.826 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/38 file after the snapshot 2024-03-28T18:59:29.832 INFO:tasks.workunit.client.1.smithi155.stdout: create 208/39 file after the snapshot 2024-03-28T18:59:29.840 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/.snap/snap-subdir-test 2024-03-28T18:59:29.869 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/30 file after the snapshot 2024-03-28T18:59:29.872 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/31 file after the snapshot 2024-03-28T18:59:29.878 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/32 file after the snapshot 2024-03-28T18:59:29.886 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/33 file after the snapshot 2024-03-28T18:59:29.896 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/34 file after the snapshot 2024-03-28T18:59:29.905 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/35 file after the snapshot 2024-03-28T18:59:29.913 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/36 file after the snapshot 2024-03-28T18:59:29.920 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/37 file after the snapshot 2024-03-28T18:59:29.928 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/38 file after the snapshot 2024-03-28T18:59:29.936 INFO:tasks.workunit.client.1.smithi155.stdout: create 209/39 file after the snapshot 2024-03-28T18:59:29.944 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/.snap/snap-subdir-test 2024-03-28T18:59:29.966 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/30 file after the snapshot 2024-03-28T18:59:29.972 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/31 file after the snapshot 2024-03-28T18:59:29.979 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/32 file after the snapshot 2024-03-28T18:59:29.985 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/33 file after the snapshot 2024-03-28T18:59:29.992 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/34 file after the snapshot 2024-03-28T18:59:29.998 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/35 file after the snapshot 2024-03-28T18:59:30.005 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/36 file after the snapshot 2024-03-28T18:59:30.012 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/37 file after the snapshot 2024-03-28T18:59:30.019 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/38 file after the snapshot 2024-03-28T18:59:30.026 INFO:tasks.workunit.client.1.smithi155.stdout: create 210/39 file after the snapshot 2024-03-28T18:59:30.033 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/.snap/snap-subdir-test 2024-03-28T18:59:30.062 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/30 file after the snapshot 2024-03-28T18:59:30.067 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/31 file after the snapshot 2024-03-28T18:59:30.074 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/32 file after the snapshot 2024-03-28T18:59:30.083 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/33 file after the snapshot 2024-03-28T18:59:30.092 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/34 file after the snapshot 2024-03-28T18:59:30.102 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/35 file after the snapshot 2024-03-28T18:59:30.109 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/36 file after the snapshot 2024-03-28T18:59:30.116 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/37 file after the snapshot 2024-03-28T18:59:30.122 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/38 file after the snapshot 2024-03-28T18:59:30.130 INFO:tasks.workunit.client.1.smithi155.stdout: create 211/39 file after the snapshot 2024-03-28T18:59:30.136 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/.snap/snap-subdir-test 2024-03-28T18:59:30.166 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/30 file after the snapshot 2024-03-28T18:59:30.171 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/31 file after the snapshot 2024-03-28T18:59:30.177 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/32 file after the snapshot 2024-03-28T18:59:30.184 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/33 file after the snapshot 2024-03-28T18:59:30.190 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/34 file after the snapshot 2024-03-28T18:59:30.196 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/35 file after the snapshot 2024-03-28T18:59:30.210 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/36 file after the snapshot 2024-03-28T18:59:30.220 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/37 file after the snapshot 2024-03-28T18:59:30.227 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/38 file after the snapshot 2024-03-28T18:59:30.234 INFO:tasks.workunit.client.1.smithi155.stdout: create 212/39 file after the snapshot 2024-03-28T18:59:30.239 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/.snap/snap-subdir-test 2024-03-28T18:59:30.264 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/30 file after the snapshot 2024-03-28T18:59:30.267 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/31 file after the snapshot 2024-03-28T18:59:30.274 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/32 file after the snapshot 2024-03-28T18:59:30.280 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/33 file after the snapshot 2024-03-28T18:59:30.286 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/34 file after the snapshot 2024-03-28T18:59:30.295 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/35 file after the snapshot 2024-03-28T18:59:30.302 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/36 file after the snapshot 2024-03-28T18:59:30.310 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/37 file after the snapshot 2024-03-28T18:59:30.316 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/38 file after the snapshot 2024-03-28T18:59:30.322 INFO:tasks.workunit.client.1.smithi155.stdout: create 213/39 file after the snapshot 2024-03-28T18:59:30.330 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/.snap/snap-subdir-test 2024-03-28T18:59:30.353 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/30 file after the snapshot 2024-03-28T18:59:30.358 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/31 file after the snapshot 2024-03-28T18:59:30.365 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/32 file after the snapshot 2024-03-28T18:59:30.372 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/33 file after the snapshot 2024-03-28T18:59:30.379 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/34 file after the snapshot 2024-03-28T18:59:30.390 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/35 file after the snapshot 2024-03-28T18:59:30.398 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/36 file after the snapshot 2024-03-28T18:59:30.404 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/37 file after the snapshot 2024-03-28T18:59:30.410 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/38 file after the snapshot 2024-03-28T18:59:30.418 INFO:tasks.workunit.client.1.smithi155.stdout: create 214/39 file after the snapshot 2024-03-28T18:59:30.425 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/.snap/snap-subdir-test 2024-03-28T18:59:30.531 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/30 file after the snapshot 2024-03-28T18:59:30.534 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/31 file after the snapshot 2024-03-28T18:59:30.542 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/32 file after the snapshot 2024-03-28T18:59:30.548 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/33 file after the snapshot 2024-03-28T18:59:30.556 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/34 file after the snapshot 2024-03-28T18:59:30.564 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/35 file after the snapshot 2024-03-28T18:59:30.571 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/36 file after the snapshot 2024-03-28T18:59:30.578 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/37 file after the snapshot 2024-03-28T18:59:30.585 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/38 file after the snapshot 2024-03-28T18:59:30.592 INFO:tasks.workunit.client.1.smithi155.stdout: create 215/39 file after the snapshot 2024-03-28T18:59:30.598 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/.snap/snap-subdir-test 2024-03-28T18:59:30.625 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/30 file after the snapshot 2024-03-28T18:59:30.629 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/31 file after the snapshot 2024-03-28T18:59:30.635 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/32 file after the snapshot 2024-03-28T18:59:30.640 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/33 file after the snapshot 2024-03-28T18:59:30.647 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/34 file after the snapshot 2024-03-28T18:59:30.655 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/35 file after the snapshot 2024-03-28T18:59:30.661 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/36 file after the snapshot 2024-03-28T18:59:30.669 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/37 file after the snapshot 2024-03-28T18:59:30.677 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/38 file after the snapshot 2024-03-28T18:59:30.683 INFO:tasks.workunit.client.1.smithi155.stdout: create 216/39 file after the snapshot 2024-03-28T18:59:30.689 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/.snap/snap-subdir-test 2024-03-28T18:59:30.709 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/30 file after the snapshot 2024-03-28T18:59:30.712 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/31 file after the snapshot 2024-03-28T18:59:30.718 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/32 file after the snapshot 2024-03-28T18:59:30.726 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/33 file after the snapshot 2024-03-28T18:59:30.734 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/34 file after the snapshot 2024-03-28T18:59:30.742 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/35 file after the snapshot 2024-03-28T18:59:30.749 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/36 file after the snapshot 2024-03-28T18:59:30.756 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/37 file after the snapshot 2024-03-28T18:59:30.765 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/38 file after the snapshot 2024-03-28T18:59:30.772 INFO:tasks.workunit.client.1.smithi155.stdout: create 217/39 file after the snapshot 2024-03-28T18:59:30.781 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/.snap/snap-subdir-test 2024-03-28T18:59:30.809 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/30 file after the snapshot 2024-03-28T18:59:30.814 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/31 file after the snapshot 2024-03-28T18:59:30.823 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/32 file after the snapshot 2024-03-28T18:59:30.834 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/33 file after the snapshot 2024-03-28T18:59:30.842 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/34 file after the snapshot 2024-03-28T18:59:30.849 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/35 file after the snapshot 2024-03-28T18:59:30.856 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/36 file after the snapshot 2024-03-28T18:59:30.865 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/37 file after the snapshot 2024-03-28T18:59:30.872 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/38 file after the snapshot 2024-03-28T18:59:30.879 INFO:tasks.workunit.client.1.smithi155.stdout: create 218/39 file after the snapshot 2024-03-28T18:59:30.885 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/.snap/snap-subdir-test 2024-03-28T18:59:30.915 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/30 file after the snapshot 2024-03-28T18:59:30.920 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/31 file after the snapshot 2024-03-28T18:59:30.929 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/32 file after the snapshot 2024-03-28T18:59:30.939 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/33 file after the snapshot 2024-03-28T18:59:30.946 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/34 file after the snapshot 2024-03-28T18:59:30.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/35 file after the snapshot 2024-03-28T18:59:30.959 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/36 file after the snapshot 2024-03-28T18:59:30.966 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/37 file after the snapshot 2024-03-28T18:59:30.973 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/38 file after the snapshot 2024-03-28T18:59:30.980 INFO:tasks.workunit.client.1.smithi155.stdout: create 219/39 file after the snapshot 2024-03-28T18:59:30.989 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/.snap/snap-subdir-test 2024-03-28T18:59:31.021 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/30 file after the snapshot 2024-03-28T18:59:31.025 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/31 file after the snapshot 2024-03-28T18:59:31.036 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/32 file after the snapshot 2024-03-28T18:59:31.043 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/33 file after the snapshot 2024-03-28T18:59:31.053 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/34 file after the snapshot 2024-03-28T18:59:31.062 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/35 file after the snapshot 2024-03-28T18:59:31.072 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/36 file after the snapshot 2024-03-28T18:59:31.078 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/37 file after the snapshot 2024-03-28T18:59:31.087 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/38 file after the snapshot 2024-03-28T18:59:31.096 INFO:tasks.workunit.client.1.smithi155.stdout: create 220/39 file after the snapshot 2024-03-28T18:59:31.104 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/.snap/snap-subdir-test 2024-03-28T18:59:31.130 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/30 file after the snapshot 2024-03-28T18:59:31.134 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/31 file after the snapshot 2024-03-28T18:59:31.142 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/32 file after the snapshot 2024-03-28T18:59:31.151 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/33 file after the snapshot 2024-03-28T18:59:31.159 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/34 file after the snapshot 2024-03-28T18:59:31.167 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/35 file after the snapshot 2024-03-28T18:59:31.173 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/36 file after the snapshot 2024-03-28T18:59:31.181 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/37 file after the snapshot 2024-03-28T18:59:31.188 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/38 file after the snapshot 2024-03-28T18:59:31.198 INFO:tasks.workunit.client.1.smithi155.stdout: create 221/39 file after the snapshot 2024-03-28T18:59:31.205 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/.snap/snap-subdir-test 2024-03-28T18:59:31.237 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/30 file after the snapshot 2024-03-28T18:59:31.240 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/31 file after the snapshot 2024-03-28T18:59:31.246 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/32 file after the snapshot 2024-03-28T18:59:31.252 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/33 file after the snapshot 2024-03-28T18:59:31.259 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/34 file after the snapshot 2024-03-28T18:59:31.265 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/35 file after the snapshot 2024-03-28T18:59:31.273 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/36 file after the snapshot 2024-03-28T18:59:31.282 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/37 file after the snapshot 2024-03-28T18:59:31.291 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/38 file after the snapshot 2024-03-28T18:59:31.300 INFO:tasks.workunit.client.1.smithi155.stdout: create 222/39 file after the snapshot 2024-03-28T18:59:31.307 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/.snap/snap-subdir-test 2024-03-28T18:59:31.337 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/30 file after the snapshot 2024-03-28T18:59:31.340 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/31 file after the snapshot 2024-03-28T18:59:31.345 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/32 file after the snapshot 2024-03-28T18:59:31.352 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/33 file after the snapshot 2024-03-28T18:59:31.359 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/34 file after the snapshot 2024-03-28T18:59:31.366 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/35 file after the snapshot 2024-03-28T18:59:31.373 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/36 file after the snapshot 2024-03-28T18:59:31.379 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/37 file after the snapshot 2024-03-28T18:59:31.386 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/38 file after the snapshot 2024-03-28T18:59:31.393 INFO:tasks.workunit.client.1.smithi155.stdout: create 223/39 file after the snapshot 2024-03-28T18:59:31.405 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/.snap/snap-subdir-test 2024-03-28T18:59:31.436 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/30 file after the snapshot 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-03-28T18:59:31.437 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-03-28T18:59:31.438 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-03-28T18:59:31.439 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-03-28T18:59:31.440 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-03-28T18:59:31.441 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-03-28T18:59:31.442 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-03-28T18:59:31.444 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/31 file after the snapshot 2024-03-28T18:59:31.505 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/32 file after the snapshot 2024-03-28T18:59:31.537 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/33 file after the snapshot 2024-03-28T18:59:31.545 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/34 file after the snapshot 2024-03-28T18:59:31.551 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/35 file after the snapshot 2024-03-28T18:59:31.557 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/36 file after the snapshot 2024-03-28T18:59:31.564 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/37 file after the snapshot 2024-03-28T18:59:31.570 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/38 file after the snapshot 2024-03-28T18:59:31.577 INFO:tasks.workunit.client.1.smithi155.stdout: create 224/39 file after the snapshot 2024-03-28T18:59:31.585 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/.snap/snap-subdir-test 2024-03-28T18:59:31.612 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/30 file after the snapshot 2024-03-28T18:59:31.617 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/31 file after the snapshot 2024-03-28T18:59:31.624 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/32 file after the snapshot 2024-03-28T18:59:31.631 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/33 file after the snapshot 2024-03-28T18:59:31.638 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/34 file after the snapshot 2024-03-28T18:59:31.646 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/35 file after the snapshot 2024-03-28T18:59:31.654 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/36 file after the snapshot 2024-03-28T18:59:31.660 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/37 file after the snapshot 2024-03-28T18:59:31.666 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/38 file after the snapshot 2024-03-28T18:59:31.672 INFO:tasks.workunit.client.1.smithi155.stdout: create 225/39 file after the snapshot 2024-03-28T18:59:31.680 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/.snap/snap-subdir-test 2024-03-28T18:59:31.703 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/30 file after the snapshot 2024-03-28T18:59:31.705 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/31 file after the snapshot 2024-03-28T18:59:31.711 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/32 file after the snapshot 2024-03-28T18:59:31.717 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/33 file after the snapshot 2024-03-28T18:59:31.723 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/34 file after the snapshot 2024-03-28T18:59:31.729 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/35 file after the snapshot 2024-03-28T18:59:31.735 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/36 file after the snapshot 2024-03-28T18:59:31.740 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/37 file after the snapshot 2024-03-28T18:59:31.746 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/38 file after the snapshot 2024-03-28T18:59:31.753 INFO:tasks.workunit.client.1.smithi155.stdout: create 226/39 file after the snapshot 2024-03-28T18:59:31.760 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/.snap/snap-subdir-test 2024-03-28T18:59:31.790 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/30 file after the snapshot 2024-03-28T18:59:31.795 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/31 file after the snapshot 2024-03-28T18:59:31.803 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/32 file after the snapshot 2024-03-28T18:59:31.810 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/33 file after the snapshot 2024-03-28T18:59:31.816 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/34 file after the snapshot 2024-03-28T18:59:31.824 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/35 file after the snapshot 2024-03-28T18:59:31.832 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/36 file after the snapshot 2024-03-28T18:59:31.840 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/37 file after the snapshot 2024-03-28T18:59:31.848 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/38 file after the snapshot 2024-03-28T18:59:31.854 INFO:tasks.workunit.client.1.smithi155.stdout: create 227/39 file after the snapshot 2024-03-28T18:59:31.859 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/.snap/snap-subdir-test 2024-03-28T18:59:31.881 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/30 file after the snapshot 2024-03-28T18:59:31.885 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/31 file after the snapshot 2024-03-28T18:59:31.894 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/32 file after the snapshot 2024-03-28T18:59:31.900 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/33 file after the snapshot 2024-03-28T18:59:31.908 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/34 file after the snapshot 2024-03-28T18:59:31.914 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/35 file after the snapshot 2024-03-28T18:59:31.922 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/36 file after the snapshot 2024-03-28T18:59:31.933 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/37 file after the snapshot 2024-03-28T18:59:31.939 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/38 file after the snapshot 2024-03-28T18:59:31.945 INFO:tasks.workunit.client.1.smithi155.stdout: create 228/39 file after the snapshot 2024-03-28T18:59:31.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/.snap/snap-subdir-test 2024-03-28T18:59:31.975 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/30 file after the snapshot 2024-03-28T18:59:31.980 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/31 file after the snapshot 2024-03-28T18:59:31.986 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/32 file after the snapshot 2024-03-28T18:59:31.993 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/33 file after the snapshot 2024-03-28T18:59:32.002 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/34 file after the snapshot 2024-03-28T18:59:32.011 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/35 file after the snapshot 2024-03-28T18:59:32.020 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/36 file after the snapshot 2024-03-28T18:59:32.027 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/37 file after the snapshot 2024-03-28T18:59:32.035 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/38 file after the snapshot 2024-03-28T18:59:32.042 INFO:tasks.workunit.client.1.smithi155.stdout: create 229/39 file after the snapshot 2024-03-28T18:59:32.048 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/.snap/snap-subdir-test 2024-03-28T18:59:32.077 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/30 file after the snapshot 2024-03-28T18:59:32.081 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/31 file after the snapshot 2024-03-28T18:59:32.090 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/32 file after the snapshot 2024-03-28T18:59:32.098 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/33 file after the snapshot 2024-03-28T18:59:32.106 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/34 file after the snapshot 2024-03-28T18:59:32.114 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/35 file after the snapshot 2024-03-28T18:59:32.122 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/36 file after the snapshot 2024-03-28T18:59:32.128 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/37 file after the snapshot 2024-03-28T18:59:32.136 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/38 file after the snapshot 2024-03-28T18:59:32.143 INFO:tasks.workunit.client.1.smithi155.stdout: create 230/39 file after the snapshot 2024-03-28T18:59:32.151 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/.snap/snap-subdir-test 2024-03-28T18:59:32.180 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/30 file after the snapshot 2024-03-28T18:59:32.184 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/31 file after the snapshot 2024-03-28T18:59:32.192 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/32 file after the snapshot 2024-03-28T18:59:32.199 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/33 file after the snapshot 2024-03-28T18:59:32.206 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/34 file after the snapshot 2024-03-28T18:59:32.214 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/35 file after the snapshot 2024-03-28T18:59:32.224 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/36 file after the snapshot 2024-03-28T18:59:32.232 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/37 file after the snapshot 2024-03-28T18:59:32.242 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/38 file after the snapshot 2024-03-28T18:59:32.251 INFO:tasks.workunit.client.1.smithi155.stdout: create 231/39 file after the snapshot 2024-03-28T18:59:32.261 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/.snap/snap-subdir-test 2024-03-28T18:59:32.297 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/30 file after the snapshot 2024-03-28T18:59:32.303 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/31 file after the snapshot 2024-03-28T18:59:32.313 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/32 file after the snapshot 2024-03-28T18:59:32.322 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/33 file after the snapshot 2024-03-28T18:59:32.329 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/34 file after the snapshot 2024-03-28T18:59:32.337 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/35 file after the snapshot 2024-03-28T18:59:32.345 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/36 file after the snapshot 2024-03-28T18:59:32.354 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/37 file after the snapshot 2024-03-28T18:59:32.364 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/38 file after the snapshot 2024-03-28T18:59:32.373 INFO:tasks.workunit.client.1.smithi155.stdout: create 232/39 file after the snapshot 2024-03-28T18:59:32.382 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/.snap/snap-subdir-test 2024-03-28T18:59:32.412 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/30 file after the snapshot 2024-03-28T18:59:32.417 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/31 file after the snapshot 2024-03-28T18:59:32.425 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/32 file after the snapshot 2024-03-28T18:59:32.432 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/33 file after the snapshot 2024-03-28T18:59:32.437 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/34 file after the snapshot 2024-03-28T18:59:32.453 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/35 file after the snapshot 2024-03-28T18:59:32.498 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/36 file after the snapshot 2024-03-28T18:59:32.548 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/37 file after the snapshot 2024-03-28T18:59:32.555 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/38 file after the snapshot 2024-03-28T18:59:32.561 INFO:tasks.workunit.client.1.smithi155.stdout: create 233/39 file after the snapshot 2024-03-28T18:59:32.567 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/.snap/snap-subdir-test 2024-03-28T18:59:32.597 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/30 file after the snapshot 2024-03-28T18:59:32.603 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/31 file after the snapshot 2024-03-28T18:59:32.609 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/32 file after the snapshot 2024-03-28T18:59:32.615 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/33 file after the snapshot 2024-03-28T18:59:32.621 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/34 file after the snapshot 2024-03-28T18:59:32.628 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/35 file after the snapshot 2024-03-28T18:59:32.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:32.635 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/36 file after the snapshot 2024-03-28T18:59:32.641 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/37 file after the snapshot 2024-03-28T18:59:32.648 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/38 file after the snapshot 2024-03-28T18:59:32.654 INFO:tasks.workunit.client.1.smithi155.stdout: create 234/39 file after the snapshot 2024-03-28T18:59:32.661 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/.snap/snap-subdir-test 2024-03-28T18:59:32.687 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/30 file after the snapshot 2024-03-28T18:59:32.692 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/31 file after the snapshot 2024-03-28T18:59:32.700 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/32 file after the snapshot 2024-03-28T18:59:32.706 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/33 file after the snapshot 2024-03-28T18:59:32.714 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/34 file after the snapshot 2024-03-28T18:59:32.721 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/35 file after the snapshot 2024-03-28T18:59:32.731 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/36 file after the snapshot 2024-03-28T18:59:32.738 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/37 file after the snapshot 2024-03-28T18:59:32.746 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/38 file after the snapshot 2024-03-28T18:59:32.751 INFO:tasks.workunit.client.1.smithi155.stdout: create 235/39 file after the snapshot 2024-03-28T18:59:32.758 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/.snap/snap-subdir-test 2024-03-28T18:59:32.780 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/30 file after the snapshot 2024-03-28T18:59:32.783 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/31 file after the snapshot 2024-03-28T18:59:32.790 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/32 file after the snapshot 2024-03-28T18:59:32.796 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/33 file after the snapshot 2024-03-28T18:59:32.803 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/34 file after the snapshot 2024-03-28T18:59:32.810 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/35 file after the snapshot 2024-03-28T18:59:32.818 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/36 file after the snapshot 2024-03-28T18:59:32.826 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/37 file after the snapshot 2024-03-28T18:59:32.833 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/38 file after the snapshot 2024-03-28T18:59:32.840 INFO:tasks.workunit.client.1.smithi155.stdout: create 236/39 file after the snapshot 2024-03-28T18:59:32.846 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/.snap/snap-subdir-test 2024-03-28T18:59:32.866 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/30 file after the snapshot 2024-03-28T18:59:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.869+0000 7f31c6200640 1 Processor -- start 2024-03-28T18:59:32.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.869+0000 7f31c6200640 1 -- start start 2024-03-28T18:59:32.870 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/31 file after the snapshot 2024-03-28T18:59:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.870+0000 7f31c6200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31c0072c60 0x7f31c0159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.870+0000 7f31c6200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c0072870 con 0x7f31c014f6d0 2024-03-28T18:59:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.870+0000 7f31c6200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c0156640 con 0x7f31c0071cf0 2024-03-28T18:59:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.870+0000 7f31c6200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c0158db0 con 0x7f31c0072c60 2024-03-28T18:59:32.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.870+0000 7f31bf400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31c014f6d0 0x7f31c00c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50248/0 (socket says 172.21.15.27:50248) 2024-03-28T18:59:32.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.870+0000 7f31bf400640 1 -- 172.21.15.27:0/302920733 learned_addr learned my addr 172.21.15.27:0/302920733 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:32.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.871+0000 7f31bfe00640 1 --2- 172.21.15.27:0/302920733 >> v2:172.21.15.27:3300/0 conn(0x7f31c0072c60 0x7f31c0159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:32.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.871+0000 7f31be000640 1 -- 172.21.15.27:0/302920733 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093764506 0 0) 0x7f31b0001110 con 0x7f31c014f6d0 2024-03-28T18:59:32.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.871+0000 7f31bfe00640 1 -- 172.21.15.27:0/302920733 >> v1:172.21.15.155:6789/0 conn(0x7f31c0071cf0 legacy=0x7f31c00c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:32.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.871+0000 7f31bfe00640 1 -- 172.21.15.27:0/302920733 >> v1:172.21.15.27:6789/0 conn(0x7f31c014f6d0 legacy=0x7f31c00c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:32.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.871+0000 7f31bfe00640 1 -- 172.21.15.27:0/302920733 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c0156360 con 0x7f31c0072c60 2024-03-28T18:59:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.874+0000 7f31bfe00640 1 --2- 172.21.15.27:0/302920733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0072c60 0x7f31c0159850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009d90 tx=0x7f31b802bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2cda4a6eacef37 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.874+0000 7f31be000640 1 -- 172.21.15.27:0/302920733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8039070 con 0x7f31c0072c60 2024-03-28T18:59:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.874+0000 7f31be000640 1 -- 172.21.15.27:0/302920733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b802ca50 con 0x7f31c0072c60 2024-03-28T18:59:32.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.874+0000 7f31be000640 1 -- 172.21.15.27:0/302920733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b802ce20 con 0x7f31c0072c60 2024-03-28T18:59:32.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.875+0000 7f31c6200640 1 -- 172.21.15.27:0/302920733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0072c60 msgr2=0x7f31c0159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:32.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.875+0000 7f31c6200640 1 --2- 172.21.15.27:0/302920733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0072c60 0x7f31c0159850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009d90 tx=0x7f31b802bd20 comp rx=0 tx=0).stop 2024-03-28T18:59:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.875+0000 7f31c6200640 1 -- 172.21.15.27:0/302920733 shutdown_connections 2024-03-28T18:59:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.875+0000 7f31c6200640 1 --2- 172.21.15.27:0/302920733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0072c60 0x7f31c0159850 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.875+0000 7f31c6200640 1 -- 172.21.15.27:0/302920733 >> 172.21.15.27:0/302920733 conn(0x7f31c006c5f0 msgr2=0x7f31c006ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.875+0000 7f31c6200640 1 -- 172.21.15.27:0/302920733 shutdown_connections 2024-03-28T18:59:32.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.875+0000 7f31c6200640 1 -- 172.21.15.27:0/302920733 wait complete. 2024-03-28T18:59:32.876 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/32 file after the snapshot 2024-03-28T18:59:32.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.877+0000 7f31c6200640 1 Processor -- start 2024-03-28T18:59:32.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.878+0000 7f31c6200640 1 -- start start 2024-03-28T18:59:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31c6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0071cf0 0x7f31c00c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:32.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31c6200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0072c60 0x7f31c0148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31c6200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31c014f6d0 0x7f31c0148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31c6200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c014fab0 con 0x7f31c0071cf0 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31c6200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0159330 con 0x7f31c0072c60 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31c6200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c00c3c70 con 0x7f31c014f6d0 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31bf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0071cf0 0x7f31c00c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31bf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0071cf0 0x7f31c00c7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38464/0 (socket says 172.21.15.27:38464) 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31bf400640 1 -- 172.21.15.27:0/1174265517 learned_addr learned my addr 172.21.15.27:0/1174265517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31bfe00640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31c014f6d0 0x7f31c0148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:32.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.879+0000 7f31bea00640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0072c60 0x7f31c0148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:32.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31bf400640 1 -- 172.21.15.27:0/1174265517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31c014f6d0 msgr2=0x7f31c0148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:32.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31bf400640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31c014f6d0 0x7f31c0148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:32.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31bf400640 1 -- 172.21.15.27:0/1174265517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0072c60 msgr2=0x7f31c0148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:32.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31bf400640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0072c60 0x7f31c0148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:32.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31bf400640 1 -- 172.21.15.27:0/1174265517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c01eeda0 con 0x7f31c0071cf0 2024-03-28T18:59:32.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31bf400640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0071cf0 0x7f31c00c7580 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f31b000c7b0 tx=0x7f31b000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:32.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31b7400640 1 -- 172.21.15.27:0/1174265517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0018070 con 0x7f31c0071cf0 2024-03-28T18:59:32.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31b7400640 1 -- 172.21.15.27:0/1174265517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b00072d0 con 0x7f31c0071cf0 2024-03-28T18:59:32.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31b7400640 1 -- 172.21.15.27:0/1174265517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b00047e0 con 0x7f31c0071cf0 2024-03-28T18:59:32.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c01ef080 con 0x7f31c0071cf0 2024-03-28T18:59:32.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.881+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c01ef640 con 0x7f31c0071cf0 2024-03-28T18:59:32.882 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/33 file after the snapshot 2024-03-28T18:59:32.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.882+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c0155d50 con 0x7f31c0071cf0 2024-03-28T18:59:32.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.884+0000 7f31b7400640 1 -- 172.21.15.27:0/1174265517 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f31b0004980 con 0x7f31c0071cf0 2024-03-28T18:59:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.885+0000 7f31b7400640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f31a006bbd0 0x7f31a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:32.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.885+0000 7f31b7400640 1 -- 172.21.15.27:0/1174265517 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f31b008a1b0 con 0x7f31c0071cf0 2024-03-28T18:59:32.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.887+0000 7f31bea00640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f31a006bbd0 0x7f31a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.887+0000 7f31bea00640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f31a006bbd0 0x7f31a006e090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31c01e9340 tx=0x7f31ac009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:32.887 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/34 file after the snapshot 2024-03-28T18:59:32.892 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/35 file after the snapshot 2024-03-28T18:59:32.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:32.892+0000 7f31b7400640 1 -- 172.21.15.27:0/1174265517 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f31c0072870 con 0x7f31c0071cf0 2024-03-28T18:59:32.898 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/36 file after the snapshot 2024-03-28T18:59:32.904 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/37 file after the snapshot 2024-03-28T18:59:32.910 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/38 file after the snapshot 2024-03-28T18:59:32.917 INFO:tasks.workunit.client.1.smithi155.stdout: create 237/39 file after the snapshot 2024-03-28T18:59:32.923 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/.snap/snap-subdir-test 2024-03-28T18:59:32.944 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/30 file after the snapshot 2024-03-28T18:59:32.947 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/31 file after the snapshot 2024-03-28T18:59:32.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/32 file after the snapshot 2024-03-28T18:59:32.960 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/33 file after the snapshot 2024-03-28T18:59:32.965 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/34 file after the snapshot 2024-03-28T18:59:32.971 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/35 file after the snapshot 2024-03-28T18:59:32.978 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/36 file after the snapshot 2024-03-28T18:59:32.985 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/37 file after the snapshot 2024-03-28T18:59:32.992 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/38 file after the snapshot 2024-03-28T18:59:32.997 INFO:tasks.workunit.client.1.smithi155.stdout: create 238/39 file after the snapshot 2024-03-28T18:59:33.003 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/.snap/snap-subdir-test 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-03-28T18:59:33.009 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-03-28T18:59:33.010 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-03-28T18:59:33.011 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-03-28T18:59:33.012 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-03-28T18:59:33.013 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-03-28T18:59:33.014 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-03-28T18:59:33.014 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-03-28T18:59:33.014 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-03-28T18:59:33.014 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-03-28T18:59:33.014 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-03-28T18:59:33.014 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-03-28T18:59:33.014 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-03-28T18:59:33.029 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/30 file after the snapshot 2024-03-28T18:59:33.033 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/31 file after the snapshot 2024-03-28T18:59:33.039 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/32 file after the snapshot 2024-03-28T18:59:33.045 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/33 file after the snapshot 2024-03-28T18:59:33.052 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/34 file after the snapshot 2024-03-28T18:59:33.058 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/35 file after the snapshot 2024-03-28T18:59:33.065 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/36 file after the snapshot 2024-03-28T18:59:33.071 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/37 file after the snapshot 2024-03-28T18:59:33.078 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/38 file after the snapshot 2024-03-28T18:59:33.085 INFO:tasks.workunit.client.1.smithi155.stdout: create 239/39 file after the snapshot 2024-03-28T18:59:33.092 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/.snap/snap-subdir-test 2024-03-28T18:59:33.120 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/30 file after the snapshot 2024-03-28T18:59:33.124 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/31 file after the snapshot 2024-03-28T18:59:33.131 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/32 file after the snapshot 2024-03-28T18:59:33.137 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/33 file after the snapshot 2024-03-28T18:59:33.145 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/34 file after the snapshot 2024-03-28T18:59:33.153 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/35 file after the snapshot 2024-03-28T18:59:33.160 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/36 file after the snapshot 2024-03-28T18:59:33.168 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/37 file after the snapshot 2024-03-28T18:59:33.174 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/38 file after the snapshot 2024-03-28T18:59:33.182 INFO:tasks.workunit.client.1.smithi155.stdout: create 240/39 file after the snapshot 2024-03-28T18:59:33.188 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/.snap/snap-subdir-test 2024-03-28T18:59:33.217 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/30 file after the snapshot 2024-03-28T18:59:33.220 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/31 file after the snapshot 2024-03-28T18:59:33.226 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/32 file after the snapshot 2024-03-28T18:59:33.232 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/33 file after the snapshot 2024-03-28T18:59:33.238 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/34 file after the snapshot 2024-03-28T18:59:33.244 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/35 file after the snapshot 2024-03-28T18:59:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.246+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31c004f400 con 0x7f31c0071cf0 2024-03-28T18:59:33.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.246+0000 7f31b7400640 1 -- 172.21.15.27:0/1174265517 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f31b0004c70 con 0x7f31c0071cf0 2024-03-28T18:59:33.248 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:33.248 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:33.248 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f31a006bbd0 msgr2=0x7f31a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f31a006bbd0 0x7f31a006e090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31c01e9340 tx=0x7f31ac009040 comp rx=0 tx=0).stop 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0071cf0 msgr2=0x7f31c00c7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0071cf0 0x7f31c00c7580 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f31b000c7b0 tx=0x7f31b000cc70 comp rx=0 tx=0).stop 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 shutdown_connections 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f31a006bbd0 0x7f31a006e090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31c014f6d0 0x7f31c0148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0072c60 0x7f31c0148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 --2- 172.21.15.27:0/1174265517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c0071cf0 0x7f31c00c7580 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 >> 172.21.15.27:0/1174265517 conn(0x7f31c006c5f0 msgr2=0x7f31c00a9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:33.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.250+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 shutdown_connections 2024-03-28T18:59:33.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.251+0000 7f31c6200640 1 -- 172.21.15.27:0/1174265517 wait complete. 2024-03-28T18:59:33.251 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/36 file after the snapshot 2024-03-28T18:59:33.257 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/37 file after the snapshot 2024-03-28T18:59:33.264 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/38 file after the snapshot 2024-03-28T18:59:33.270 INFO:tasks.workunit.client.1.smithi155.stdout: create 241/39 file after the snapshot 2024-03-28T18:59:33.271 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-03-28T18:59:33.271 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T18:59:33.272 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T18:59:33.275 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/.snap/snap-subdir-test 2024-03-28T18:59:33.295 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T18:59:33.295 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:33.309 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/30 file after the snapshot 2024-03-28T18:59:33.312 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/31 file after the snapshot 2024-03-28T18:59:33.322 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/32 file after the snapshot 2024-03-28T18:59:33.331 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/33 file after the snapshot 2024-03-28T18:59:33.341 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/34 file after the snapshot 2024-03-28T18:59:33.352 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/35 file after the snapshot 2024-03-28T18:59:33.360 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/36 file after the snapshot 2024-03-28T18:59:33.373 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/37 file after the snapshot 2024-03-28T18:59:33.384 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/38 file after the snapshot 2024-03-28T18:59:33.394 INFO:tasks.workunit.client.1.smithi155.stdout: create 242/39 file after the snapshot 2024-03-28T18:59:33.404 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/.snap/snap-subdir-test 2024-03-28T18:59:33.438 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/30 file after the snapshot 2024-03-28T18:59:33.443 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/31 file after the snapshot 2024-03-28T18:59:33.452 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/32 file after the snapshot 2024-03-28T18:59:33.521 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/33 file after the snapshot 2024-03-28T18:59:33.558 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/34 file after the snapshot 2024-03-28T18:59:33.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.564+0000 7fdc5c800640 1 Processor -- start 2024-03-28T18:59:33.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.565+0000 7fdc5c800640 1 -- start start 2024-03-28T18:59:33.564 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/35 file after the snapshot 2024-03-28T18:59:33.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.566+0000 7fdc5c800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c8530 0x7fdc540c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:33.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.566+0000 7fdc5c800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc540c54c0 0x7fdc54156c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:33.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.566+0000 7fdc5c800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc540c5c40 con 0x7fdc540c4510 2024-03-28T18:59:33.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.566+0000 7fdc5c800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc540c3480 con 0x7fdc540c54c0 2024-03-28T18:59:33.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.566+0000 7fdc5c800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540c5f20 con 0x7fdc540c8530 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc59e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fdc540c4510 0x7fdc540701c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:60694/0 (socket says 172.21.15.27:60694) 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc59400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c8530 0x7fdc540c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc59e00640 1 -- 172.21.15.27:0/3402455487 learned_addr learned my addr 172.21.15.27:0/3402455487 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc5a800640 1 --2- 172.21.15.27:0/3402455487 >> v2:172.21.15.27:3300/0 conn(0x7fdc540c54c0 0x7fdc54156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc5a800640 1 -- 172.21.15.27:0/3402455487 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c8530 msgr2=0x7fdc540c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc5a800640 1 --2- 172.21.15.27:0/3402455487 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c8530 0x7fdc540c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc5a800640 1 -- 172.21.15.27:0/3402455487 >> v1:172.21.15.155:6789/0 conn(0x7fdc540c4510 legacy=0x7fdc540701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc5a800640 1 -- 172.21.15.27:0/3402455487 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc540c6200 con 0x7fdc540c54c0 2024-03-28T18:59:33.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.567+0000 7fdc58a00640 1 -- 172.21.15.27:0/3402455487 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528091890 0 0) 0x7fdc540c5c40 con 0x7fdc540c4510 2024-03-28T18:59:33.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.569+0000 7fdc5a800640 1 --2- 172.21.15.27:0/3402455487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c54c0 0x7fdc54156c10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0097a0 tx=0x7fdc4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937d676c4e2ecdb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:33.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.569+0000 7fdc58a00640 1 -- 172.21.15.27:0/3402455487 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0315a0 con 0x7fdc540c54c0 2024-03-28T18:59:33.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.569+0000 7fdc58a00640 1 -- 172.21.15.27:0/3402455487 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c037070 con 0x7fdc540c54c0 2024-03-28T18:59:33.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.569+0000 7fdc58a00640 1 -- 172.21.15.27:0/3402455487 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c031960 con 0x7fdc540c54c0 2024-03-28T18:59:33.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.570+0000 7fdc5c800640 1 -- 172.21.15.27:0/3402455487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c54c0 msgr2=0x7fdc54156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:33.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.570+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3402455487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c54c0 0x7fdc54156c10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0097a0 tx=0x7fdc4c02b840 comp rx=0 tx=0).stop 2024-03-28T18:59:33.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.570+0000 7fdc5c800640 1 -- 172.21.15.27:0/3402455487 shutdown_connections 2024-03-28T18:59:33.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.570+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3402455487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c54c0 0x7fdc54156c10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.570+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3402455487 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c8530 0x7fdc540c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.570+0000 7fdc5c800640 1 -- 172.21.15.27:0/3402455487 >> 172.21.15.27:0/3402455487 conn(0x7fdc54066dd0 msgr2=0x7fdc540671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:33.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.570+0000 7fdc5c800640 1 -- 172.21.15.27:0/3402455487 shutdown_connections 2024-03-28T18:59:33.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.570+0000 7fdc5c800640 1 -- 172.21.15.27:0/3402455487 wait complete. 2024-03-28T18:59:33.571 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/36 file after the snapshot 2024-03-28T18:59:33.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.572+0000 7fdc5c800640 1 Processor -- start 2024-03-28T18:59:33.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.572+0000 7fdc5c800640 1 -- start start 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc5c800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c4510 0x7fdc541de720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc5c800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc540c54c0 0x7fdc541d65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc5c800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c8530 0x7fdc541d57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc5c800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540c5910 con 0x7fdc540c8530 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc5c800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5406fca0 con 0x7fdc540c54c0 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc5c800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc540c48f0 con 0x7fdc540c4510 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc5a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c8530 0x7fdc541d57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc59400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc540c54c0 0x7fdc541d65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.573+0000 7fdc59e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c4510 0x7fdc541de720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:33.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc59400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc540c54c0 0x7fdc541d65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:53916/0 (socket says 172.21.15.27:53916) 2024-03-28T18:59:33.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc5a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c8530 0x7fdc541d57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38488/0 (socket says 172.21.15.27:38488) 2024-03-28T18:59:33.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc59400640 1 -- 172.21.15.27:0/3047727118 learned_addr learned my addr 172.21.15.27:0/3047727118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc5a800640 1 -- 172.21.15.27:0/3047727118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c4510 msgr2=0x7fdc541de720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc5a800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c4510 0x7fdc541de720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc5a800640 1 -- 172.21.15.27:0/3047727118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc540c54c0 msgr2=0x7fdc541d65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc5a800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc540c54c0 0x7fdc541d65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc5a800640 1 -- 172.21.15.27:0/3047727118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc541d70d0 con 0x7fdc540c8530 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.574+0000 7fdc5a800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c8530 0x7fdc541d57b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c033000 tx=0x7fdc4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.575+0000 7fdc4a000640 1 -- 172.21.15.27:0/3047727118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c032630 con 0x7fdc540c8530 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.575+0000 7fdc4a000640 1 -- 172.21.15.27:0/3047727118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c0327f0 con 0x7fdc540c8530 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.575+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc541d4220 con 0x7fdc540c8530 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.575+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc541ebbc0 con 0x7fdc540c8530 2024-03-28T18:59:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.575+0000 7fdc4a000640 1 -- 172.21.15.27:0/3047727118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c032be0 con 0x7fdc540c8530 2024-03-28T18:59:33.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.576+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc34005290 con 0x7fdc540c8530 2024-03-28T18:59:33.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.577+0000 7fdc4a000640 1 -- 172.21.15.27:0/3047727118 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdc4c04e430 con 0x7fdc540c8530 2024-03-28T18:59:33.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.578+0000 7fdc4a000640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc3806bc00 0x7fdc3806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:33.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.578+0000 7fdc4a000640 1 -- 172.21.15.27:0/3047727118 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7fdc4c02f030 con 0x7fdc540c8530 2024-03-28T18:59:33.578 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/37 file after the snapshot 2024-03-28T18:59:33.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.579+0000 7fdc59e00640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc3806bc00 0x7fdc3806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:33.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.579+0000 7fdc59e00640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc3806bc00 0x7fdc3806e0c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdc500012e0 tx=0x7fdc50001700 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:33.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.584+0000 7fdc4a000640 1 -- 172.21.15.27:0/3047727118 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdc4c040070 con 0x7fdc540c8530 2024-03-28T18:59:33.584 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/38 file after the snapshot 2024-03-28T18:59:33.590 INFO:tasks.workunit.client.1.smithi155.stdout: create 243/39 file after the snapshot 2024-03-28T18:59:33.597 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/.snap/snap-subdir-test 2024-03-28T18:59:33.626 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/30 file after the snapshot 2024-03-28T18:59:33.632 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/31 file after the snapshot 2024-03-28T18:59:33.640 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/32 file after the snapshot 2024-03-28T18:59:33.649 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/33 file after the snapshot 2024-03-28T18:59:33.659 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/34 file after the snapshot 2024-03-28T18:59:33.669 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/35 file after the snapshot 2024-03-28T18:59:33.679 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/36 file after the snapshot 2024-03-28T18:59:33.690 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/37 file after the snapshot 2024-03-28T18:59:33.698 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/38 file after the snapshot 2024-03-28T18:59:33.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-28T18:59:33.707 INFO:tasks.workunit.client.1.smithi155.stdout: create 244/39 file after the snapshot 2024-03-28T18:59:33.716 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/.snap/snap-subdir-test 2024-03-28T18:59:33.750 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/30 file after the snapshot 2024-03-28T18:59:33.756 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/31 file after the snapshot 2024-03-28T18:59:33.767 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/32 file after the snapshot 2024-03-28T18:59:33.780 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/33 file after the snapshot 2024-03-28T18:59:33.789 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/34 file after the snapshot 2024-03-28T18:59:33.800 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/35 file after the snapshot 2024-03-28T18:59:33.810 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/36 file after the snapshot 2024-03-28T18:59:33.821 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/37 file after the snapshot 2024-03-28T18:59:33.832 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/38 file after the snapshot 2024-03-28T18:59:33.844 INFO:tasks.workunit.client.1.smithi155.stdout: create 245/39 file after the snapshot 2024-03-28T18:59:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.851+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc34005580 con 0x7fdc540c8530 2024-03-28T18:59:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.852+0000 7fdc4a000640 1 -- 172.21.15.27:0/3047727118 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7fdc4c07ab00 con 0x7fdc540c8530 2024-03-28T18:59:33.851 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:33.851 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.854+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc3806bc00 msgr2=0x7fdc3806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.854+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc3806bc00 0x7fdc3806e0c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdc500012e0 tx=0x7fdc50001700 comp rx=0 tx=0).stop 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.854+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c8530 msgr2=0x7fdc541d57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.854+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c8530 0x7fdc541d57b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c033000 tx=0x7fdc4c0046c0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.855+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 shutdown_connections 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.855+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc3806bc00 0x7fdc3806e0c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.855+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc540c8530 0x7fdc541d57b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.855+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc540c54c0 0x7fdc541d65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.855+0000 7fdc5c800640 1 --2- 172.21.15.27:0/3047727118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc540c4510 0x7fdc541de720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.855+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 >> 172.21.15.27:0/3047727118 conn(0x7fdc54066dd0 msgr2=0x7fdc540c2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.855+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 shutdown_connections 2024-03-28T18:59:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:33.855+0000 7fdc5c800640 1 -- 172.21.15.27:0/3047727118 wait complete. 2024-03-28T18:59:33.855 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/.snap/snap-subdir-test 2024-03-28T18:59:33.874 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:59:33.890 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/30 file after the snapshot 2024-03-28T18:59:33.895 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/31 file after the snapshot 2024-03-28T18:59:33.907 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/32 file after the snapshot 2024-03-28T18:59:33.920 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/33 file after the snapshot 2024-03-28T18:59:33.932 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/34 file after the snapshot 2024-03-28T18:59:33.945 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/35 file after the snapshot 2024-03-28T18:59:33.956 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/36 file after the snapshot 2024-03-28T18:59:33.968 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/37 file after the snapshot 2024-03-28T18:59:33.983 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/38 file after the snapshot 2024-03-28T18:59:33.997 INFO:tasks.workunit.client.1.smithi155.stdout: create 246/39 file after the snapshot 2024-03-28T18:59:34.009 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/.snap/snap-subdir-test 2024-03-28T18:59:34.048 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/30 file after the snapshot 2024-03-28T18:59:34.052 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/31 file after the snapshot 2024-03-28T18:59:34.062 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/32 file after the snapshot 2024-03-28T18:59:34.073 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/33 file after the snapshot 2024-03-28T18:59:34.084 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/34 file after the snapshot 2024-03-28T18:59:34.096 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/35 file after the snapshot 2024-03-28T18:59:34.109 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/36 file after the snapshot 2024-03-28T18:59:34.120 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/37 file after the snapshot 2024-03-28T18:59:34.133 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/38 file after the snapshot 2024-03-28T18:59:34.147 INFO:tasks.workunit.client.1.smithi155.stdout: create 247/39 file after the snapshot 2024-03-28T18:59:34.160 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/.snap/snap-subdir-test 2024-03-28T18:59:34.194 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/30 file after the snapshot 2024-03-28T18:59:34.201 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/31 file after the snapshot 2024-03-28T18:59:34.214 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/32 file after the snapshot 2024-03-28T18:59:34.227 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/33 file after the snapshot 2024-03-28T18:59:34.240 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/34 file after the snapshot 2024-03-28T18:59:34.253 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/35 file after the snapshot 2024-03-28T18:59:34.265 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/36 file after the snapshot 2024-03-28T18:59:34.277 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/37 file after the snapshot 2024-03-28T18:59:34.290 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/38 file after the snapshot 2024-03-28T18:59:34.301 INFO:tasks.workunit.client.1.smithi155.stdout: create 248/39 file after the snapshot 2024-03-28T18:59:34.313 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/.snap/snap-subdir-test 2024-03-28T18:59:34.350 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/30 file after the snapshot 2024-03-28T18:59:34.356 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/31 file after the snapshot 2024-03-28T18:59:34.368 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/32 file after the snapshot 2024-03-28T18:59:34.380 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/33 file after the snapshot 2024-03-28T18:59:34.393 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/34 file after the snapshot 2024-03-28T18:59:34.404 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/35 file after the snapshot 2024-03-28T18:59:34.417 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/36 file after the snapshot 2024-03-28T18:59:34.430 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/37 file after the snapshot 2024-03-28T18:59:34.442 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/38 file after the snapshot 2024-03-28T18:59:34.455 INFO:tasks.workunit.client.1.smithi155.stdout: create 249/39 file after the snapshot 2024-03-28T18:59:34.466 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/.snap/snap-subdir-test 2024-03-28T18:59:34.604 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/30 file after the snapshot 2024-03-28T18:59:34.607 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/31 file after the snapshot 2024-03-28T18:59:34.617 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/32 file after the snapshot 2024-03-28T18:59:34.629 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/33 file after the snapshot 2024-03-28T18:59:34.642 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/34 file after the snapshot 2024-03-28T18:59:34.654 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/35 file after the snapshot 2024-03-28T18:59:34.668 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/36 file after the snapshot 2024-03-28T18:59:34.679 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/37 file after the snapshot 2024-03-28T18:59:34.690 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/38 file after the snapshot 2024-03-28T18:59:34.699 INFO:tasks.workunit.client.1.smithi155.stdout: create 250/39 file after the snapshot 2024-03-28T18:59:34.710 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/.snap/snap-subdir-test 2024-03-28T18:59:34.744 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/30 file after the snapshot 2024-03-28T18:59:34.748 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/31 file after the snapshot 2024-03-28T18:59:34.757 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/32 file after the snapshot 2024-03-28T18:59:34.769 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/33 file after the snapshot 2024-03-28T18:59:34.781 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/34 file after the snapshot 2024-03-28T18:59:34.794 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/35 file after the snapshot 2024-03-28T18:59:34.806 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/36 file after the snapshot 2024-03-28T18:59:34.820 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/37 file after the snapshot 2024-03-28T18:59:34.831 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/38 file after the snapshot 2024-03-28T18:59:34.843 INFO:tasks.workunit.client.1.smithi155.stdout: create 251/39 file after the snapshot 2024-03-28T18:59:34.854 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/.snap/snap-subdir-test 2024-03-28T18:59:34.886 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/30 file after the snapshot 2024-03-28T18:59:34.891 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/31 file after the snapshot 2024-03-28T18:59:34.904 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/32 file after the snapshot 2024-03-28T18:59:34.917 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/33 file after the snapshot 2024-03-28T18:59:34.929 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/34 file after the snapshot 2024-03-28T18:59:34.940 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/35 file after the snapshot 2024-03-28T18:59:34.952 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/36 file after the snapshot 2024-03-28T18:59:34.965 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/37 file after the snapshot 2024-03-28T18:59:34.978 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/38 file after the snapshot 2024-03-28T18:59:34.991 INFO:tasks.workunit.client.1.smithi155.stdout: create 252/39 file after the snapshot 2024-03-28T18:59:35.004 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/.snap/snap-subdir-test 2024-03-28T18:59:35.042 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/30 file after the snapshot 2024-03-28T18:59:35.047 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/31 file after the snapshot 2024-03-28T18:59:35.058 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/32 file after the snapshot 2024-03-28T18:59:35.070 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/33 file after the snapshot 2024-03-28T18:59:35.083 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/34 file after the snapshot 2024-03-28T18:59:35.096 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/35 file after the snapshot 2024-03-28T18:59:35.109 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/36 file after the snapshot 2024-03-28T18:59:35.122 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/37 file after the snapshot 2024-03-28T18:59:35.135 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/38 file after the snapshot 2024-03-28T18:59:35.149 INFO:tasks.workunit.client.1.smithi155.stdout: create 253/39 file after the snapshot 2024-03-28T18:59:35.162 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/.snap/snap-subdir-test 2024-03-28T18:59:35.207 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/30 file after the snapshot 2024-03-28T18:59:35.212 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/31 file after the snapshot 2024-03-28T18:59:35.224 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/32 file after the snapshot 2024-03-28T18:59:35.236 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/33 file after the snapshot 2024-03-28T18:59:35.248 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/34 file after the snapshot 2024-03-28T18:59:35.261 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/35 file after the snapshot 2024-03-28T18:59:35.274 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/36 file after the snapshot 2024-03-28T18:59:35.287 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/37 file after the snapshot 2024-03-28T18:59:35.300 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/38 file after the snapshot 2024-03-28T18:59:35.312 INFO:tasks.workunit.client.1.smithi155.stdout: create 254/39 file after the snapshot 2024-03-28T18:59:35.324 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/.snap/snap-subdir-test 2024-03-28T18:59:35.366 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/30 file after the snapshot 2024-03-28T18:59:35.373 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/31 file after the snapshot 2024-03-28T18:59:35.385 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/32 file after the snapshot 2024-03-28T18:59:35.398 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/33 file after the snapshot 2024-03-28T18:59:35.410 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/34 file after the snapshot 2024-03-28T18:59:35.419 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/35 file after the snapshot 2024-03-28T18:59:35.431 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/36 file after the snapshot 2024-03-28T18:59:35.444 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/37 file after the snapshot 2024-03-28T18:59:35.456 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/38 file after the snapshot 2024-03-28T18:59:35.470 INFO:tasks.workunit.client.1.smithi155.stdout: create 255/39 file after the snapshot 2024-03-28T18:59:35.500 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/.snap/snap-subdir-test 2024-03-28T18:59:35.635 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/30 file after the snapshot 2024-03-28T18:59:35.640 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/31 file after the snapshot 2024-03-28T18:59:35.652 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/32 file after the snapshot 2024-03-28T18:59:35.663 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/33 file after the snapshot 2024-03-28T18:59:35.675 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/34 file after the snapshot 2024-03-28T18:59:35.686 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/35 file after the snapshot 2024-03-28T18:59:35.700 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/36 file after the snapshot 2024-03-28T18:59:35.711 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/37 file after the snapshot 2024-03-28T18:59:35.723 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/38 file after the snapshot 2024-03-28T18:59:35.737 INFO:tasks.workunit.client.1.smithi155.stdout: create 256/39 file after the snapshot 2024-03-28T18:59:35.750 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/.snap/snap-subdir-test 2024-03-28T18:59:35.798 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/30 file after the snapshot 2024-03-28T18:59:35.803 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/31 file after the snapshot 2024-03-28T18:59:35.816 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/32 file after the snapshot 2024-03-28T18:59:35.829 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/33 file after the snapshot 2024-03-28T18:59:35.843 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/34 file after the snapshot 2024-03-28T18:59:35.854 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/35 file after the snapshot 2024-03-28T18:59:35.867 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/36 file after the snapshot 2024-03-28T18:59:35.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:35.879 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/37 file after the snapshot 2024-03-28T18:59:35.892 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/38 file after the snapshot 2024-03-28T18:59:35.904 INFO:tasks.workunit.client.1.smithi155.stdout: create 257/39 file after the snapshot 2024-03-28T18:59:35.917 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/.snap/snap-subdir-test 2024-03-28T18:59:35.949 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/30 file after the snapshot 2024-03-28T18:59:35.954 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/31 file after the snapshot 2024-03-28T18:59:35.965 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/32 file after the snapshot 2024-03-28T18:59:35.977 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/33 file after the snapshot 2024-03-28T18:59:35.987 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/34 file after the snapshot 2024-03-28T18:59:35.998 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/35 file after the snapshot 2024-03-28T18:59:36.010 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/36 file after the snapshot 2024-03-28T18:59:36.023 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/37 file after the snapshot 2024-03-28T18:59:36.037 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/38 file after the snapshot 2024-03-28T18:59:36.051 INFO:tasks.workunit.client.1.smithi155.stdout: create 258/39 file after the snapshot 2024-03-28T18:59:36.066 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/.snap/snap-subdir-test 2024-03-28T18:59:36.106 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/30 file after the snapshot 2024-03-28T18:59:36.111 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/31 file after the snapshot 2024-03-28T18:59:36.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.114+0000 7f8549c00640 1 Processor -- start 2024-03-28T18:59:36.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.114+0000 7f8549c00640 1 -- start start 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8549c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85440a9cb0 0x7f85440c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8549c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f85441534c0 0x7f8544153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8549c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85440c3d90 con 0x7f8544150c70 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8549c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85440ab4e0 con 0x7f85440a9cb0 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8549c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85440ac1f0 con 0x7f85441534c0 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8542000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85440a9cb0 0x7f85440c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8542a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8544150c70 0x7f85440c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57364/0 (socket says 172.21.15.27:57364) 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8542a00640 1 -- 172.21.15.27:0/750542262 learned_addr learned my addr 172.21.15.27:0/750542262 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:36.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.116+0000 7f8543400640 1 --2- 172.21.15.27:0/750542262 >> v2:172.21.15.155:3300/0 conn(0x7f85441534c0 0x7f8544153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.117+0000 7f8542000640 1 -- 172.21.15.27:0/750542262 >> v2:172.21.15.155:3300/0 conn(0x7f85441534c0 msgr2=0x7f8544153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.117+0000 7f8542000640 1 --2- 172.21.15.27:0/750542262 >> v2:172.21.15.155:3300/0 conn(0x7f85441534c0 0x7f8544153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.117+0000 7f8542000640 1 -- 172.21.15.27:0/750542262 >> v1:172.21.15.27:6789/0 conn(0x7f8544150c70 legacy=0x7f85440c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:36.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.117+0000 7f8542000640 1 -- 172.21.15.27:0/750542262 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85440ab710 con 0x7f85440a9cb0 2024-03-28T18:59:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.118+0000 7f8542000640 1 --2- 172.21.15.27:0/750542262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f85440c3690 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f85440c4610 tx=0x7f853002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1569afbd46b0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8541600640 1 -- 172.21.15.27:0/750542262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85300315a0 con 0x7f85440a9cb0 2024-03-28T18:59:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8541600640 1 -- 172.21.15.27:0/750542262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8530037070 con 0x7f85440a9cb0 2024-03-28T18:59:36.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8541600640 1 -- 172.21.15.27:0/750542262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530031960 con 0x7f85440a9cb0 2024-03-28T18:59:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8549c00640 1 -- 172.21.15.27:0/750542262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 msgr2=0x7f85440c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8549c00640 1 --2- 172.21.15.27:0/750542262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f85440c3690 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f85440c4610 tx=0x7f853002b840 comp rx=0 tx=0).stop 2024-03-28T18:59:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8549c00640 1 -- 172.21.15.27:0/750542262 shutdown_connections 2024-03-28T18:59:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8549c00640 1 --2- 172.21.15.27:0/750542262 >> v2:172.21.15.155:3300/0 conn(0x7f85441534c0 0x7f8544153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8549c00640 1 --2- 172.21.15.27:0/750542262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f85440c3690 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8549c00640 1 -- 172.21.15.27:0/750542262 >> 172.21.15.27:0/750542262 conn(0x7f854406b9d0 msgr2=0x7f85440c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8549c00640 1 -- 172.21.15.27:0/750542262 shutdown_connections 2024-03-28T18:59:36.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.119+0000 7f8549c00640 1 -- 172.21.15.27:0/750542262 wait complete. 2024-03-28T18:59:36.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.121+0000 7f8549c00640 1 Processor -- start 2024-03-28T18:59:36.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.121+0000 7f8549c00640 1 -- start start 2024-03-28T18:59:36.121 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/32 file after the snapshot 2024-03-28T18:59:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.122+0000 7f8549c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f8544071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.122+0000 7f8549c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8544150c70 0x7f854414b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.122+0000 7f8549c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85441534c0 0x7f854414bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:36.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.122+0000 7f8549c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440c1810 con 0x7f85440a9cb0 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.122+0000 7f8549c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440abcd0 con 0x7f85441534c0 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.122+0000 7f8549c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85440c6830 con 0x7f8544150c70 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.122+0000 7f8542a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f8544071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f8544071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48258/0 (socket says 172.21.15.27:48258) 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542a00640 1 -- 172.21.15.27:0/2521634633 learned_addr learned my addr 172.21.15.27:0/2521634633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542000640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8544150c70 0x7f854414b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542a00640 1 -- 172.21.15.27:0/2521634633 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8544150c70 msgr2=0x7f854414b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542a00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8544150c70 0x7f854414b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542a00640 1 -- 172.21.15.27:0/2521634633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85441534c0 msgr2=0x7f854414bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542a00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85441534c0 0x7f854414bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542a00640 1 -- 172.21.15.27:0/2521634633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854406ec80 con 0x7f85440a9cb0 2024-03-28T18:59:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.123+0000 7f8542a00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f8544071e40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f853400c430 tx=0x7f853400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.124+0000 7f853aa00640 1 -- 172.21.15.27:0/2521634633 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534017070 con 0x7f85440a9cb0 2024-03-28T18:59:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.124+0000 7f853aa00640 1 -- 172.21.15.27:0/2521634633 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8534005ce0 con 0x7f85440a9cb0 2024-03-28T18:59:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.124+0000 7f853aa00640 1 -- 172.21.15.27:0/2521634633 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85340072f0 con 0x7f85440a9cb0 2024-03-28T18:59:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.124+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85441df7f0 con 0x7f85440a9cb0 2024-03-28T18:59:36.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.124+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854406e9a0 con 0x7f85440a9cb0 2024-03-28T18:59:36.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.125+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85441dce20 con 0x7f85440a9cb0 2024-03-28T18:59:36.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.126+0000 7f853aa00640 1 -- 172.21.15.27:0/2521634633 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f853401a070 con 0x7f85440a9cb0 2024-03-28T18:59:36.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.127+0000 7f853aa00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f852406bbd0 0x7f852406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:36.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.128+0000 7f853aa00640 1 -- 172.21.15.27:0/2521634633 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f853408a3c0 con 0x7f85440a9cb0 2024-03-28T18:59:36.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.128+0000 7f8542000640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f852406bbd0 0x7f852406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:36.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.129+0000 7f8542000640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f852406bbd0 0x7f852406e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8530036000 tx=0x7f853003d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:36.129 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/33 file after the snapshot 2024-03-28T18:59:36.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.133+0000 7f853aa00640 1 -- 172.21.15.27:0/2521634633 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85441dce20 con 0x7f85440a9cb0 2024-03-28T18:59:36.138 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/34 file after the snapshot 2024-03-28T18:59:36.149 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/35 file after the snapshot 2024-03-28T18:59:36.159 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/36 file after the snapshot 2024-03-28T18:59:36.170 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/37 file after the snapshot 2024-03-28T18:59:36.182 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/38 file after the snapshot 2024-03-28T18:59:36.196 INFO:tasks.workunit.client.1.smithi155.stdout: create 259/39 file after the snapshot 2024-03-28T18:59:36.207 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/.snap/snap-subdir-test 2024-03-28T18:59:36.241 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/30 file after the snapshot 2024-03-28T18:59:36.250 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/31 file after the snapshot 2024-03-28T18:59:36.260 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/32 file after the snapshot 2024-03-28T18:59:36.270 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/33 file after the snapshot 2024-03-28T18:59:36.282 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/34 file after the snapshot 2024-03-28T18:59:36.293 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/35 file after the snapshot 2024-03-28T18:59:36.304 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/36 file after the snapshot 2024-03-28T18:59:36.316 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/37 file after the snapshot 2024-03-28T18:59:36.327 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/38 file after the snapshot 2024-03-28T18:59:36.339 INFO:tasks.workunit.client.1.smithi155.stdout: create 260/39 file after the snapshot 2024-03-28T18:59:36.351 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/.snap/snap-subdir-test 2024-03-28T18:59:36.389 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/30 file after the snapshot 2024-03-28T18:59:36.392 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/31 file after the snapshot 2024-03-28T18:59:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.400+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8544068180 con 0x7f85440a9cb0 2024-03-28T18:59:36.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.401+0000 7f853aa00640 1 -- 172.21.15.27:0/2521634633 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f8544068180 con 0x7f85440a9cb0 2024-03-28T18:59:36.401 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:36.401 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:36.401 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:36.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f852406bbd0 msgr2=0x7f852406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:36.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f852406bbd0 0x7f852406e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8530036000 tx=0x7f853003d040 comp rx=0 tx=0).stop 2024-03-28T18:59:36.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 msgr2=0x7f8544071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:36.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f8544071e40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f853400c430 tx=0x7f853400c8f0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 shutdown_connections 2024-03-28T18:59:36.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f852406bbd0 0x7f852406e090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85441534c0 0x7f854414bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8544150c70 0x7f854414b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 --2- 172.21.15.27:0/2521634633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85440a9cb0 0x7f8544071e40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:36.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.404+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 >> 172.21.15.27:0/2521634633 conn(0x7f854406b9d0 msgr2=0x7f85440c2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:36.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.405+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 shutdown_connections 2024-03-28T18:59:36.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:36.405+0000 7f8549c00640 1 -- 172.21.15.27:0/2521634633 wait complete. 2024-03-28T18:59:36.405 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/32 file after the snapshot 2024-03-28T18:59:36.416 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/33 file after the snapshot 2024-03-28T18:59:36.425 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:59:36.429 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/34 file after the snapshot 2024-03-28T18:59:36.442 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/35 file after the snapshot 2024-03-28T18:59:36.454 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/36 file after the snapshot 2024-03-28T18:59:36.465 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/37 file after the snapshot 2024-03-28T18:59:36.478 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/38 file after the snapshot 2024-03-28T18:59:36.490 INFO:tasks.workunit.client.1.smithi155.stdout: create 261/39 file after the snapshot 2024-03-28T18:59:36.510 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/.snap/snap-subdir-test 2024-03-28T18:59:36.597 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/30 file after the snapshot 2024-03-28T18:59:36.601 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/31 file after the snapshot 2024-03-28T18:59:36.611 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/32 file after the snapshot 2024-03-28T18:59:36.622 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/33 file after the snapshot 2024-03-28T18:59:36.632 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/34 file after the snapshot 2024-03-28T18:59:36.640 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/35 file after the snapshot 2024-03-28T18:59:36.651 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/36 file after the snapshot 2024-03-28T18:59:36.661 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/37 file after the snapshot 2024-03-28T18:59:36.673 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/38 file after the snapshot 2024-03-28T18:59:36.685 INFO:tasks.workunit.client.1.smithi155.stdout: create 262/39 file after the snapshot 2024-03-28T18:59:36.697 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/.snap/snap-subdir-test 2024-03-28T18:59:36.730 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/30 file after the snapshot 2024-03-28T18:59:36.736 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/31 file after the snapshot 2024-03-28T18:59:36.746 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/32 file after the snapshot 2024-03-28T18:59:36.758 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/33 file after the snapshot 2024-03-28T18:59:36.772 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/34 file after the snapshot 2024-03-28T18:59:36.784 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/35 file after the snapshot 2024-03-28T18:59:36.798 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/36 file after the snapshot 2024-03-28T18:59:36.811 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/37 file after the snapshot 2024-03-28T18:59:36.824 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/38 file after the snapshot 2024-03-28T18:59:36.836 INFO:tasks.workunit.client.1.smithi155.stdout: create 263/39 file after the snapshot 2024-03-28T18:59:36.849 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/.snap/snap-subdir-test 2024-03-28T18:59:36.883 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/30 file after the snapshot 2024-03-28T18:59:36.889 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/31 file after the snapshot 2024-03-28T18:59:36.900 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/32 file after the snapshot 2024-03-28T18:59:36.909 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/33 file after the snapshot 2024-03-28T18:59:36.922 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/34 file after the snapshot 2024-03-28T18:59:36.935 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/35 file after the snapshot 2024-03-28T18:59:36.948 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/36 file after the snapshot 2024-03-28T18:59:36.961 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/37 file after the snapshot 2024-03-28T18:59:36.971 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/38 file after the snapshot 2024-03-28T18:59:36.985 INFO:tasks.workunit.client.1.smithi155.stdout: create 264/39 file after the snapshot 2024-03-28T18:59:36.996 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/.snap/snap-subdir-test 2024-03-28T18:59:37.032 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/30 file after the snapshot 2024-03-28T18:59:37.037 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/31 file after the snapshot 2024-03-28T18:59:37.049 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/32 file after the snapshot 2024-03-28T18:59:37.061 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/33 file after the snapshot 2024-03-28T18:59:37.075 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/34 file after the snapshot 2024-03-28T18:59:37.087 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/35 file after the snapshot 2024-03-28T18:59:37.101 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/36 file after the snapshot 2024-03-28T18:59:37.112 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/37 file after the snapshot 2024-03-28T18:59:37.125 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/38 file after the snapshot 2024-03-28T18:59:37.135 INFO:tasks.workunit.client.1.smithi155.stdout: create 265/39 file after the snapshot 2024-03-28T18:59:37.148 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/.snap/snap-subdir-test 2024-03-28T18:59:37.190 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/30 file after the snapshot 2024-03-28T18:59:37.196 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/31 file after the snapshot 2024-03-28T18:59:37.207 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/32 file after the snapshot 2024-03-28T18:59:37.220 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/33 file after the snapshot 2024-03-28T18:59:37.234 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/34 file after the snapshot 2024-03-28T18:59:37.248 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/35 file after the snapshot 2024-03-28T18:59:37.261 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/36 file after the snapshot 2024-03-28T18:59:37.274 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/37 file after the snapshot 2024-03-28T18:59:37.287 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/38 file after the snapshot 2024-03-28T18:59:37.299 INFO:tasks.workunit.client.1.smithi155.stdout: create 266/39 file after the snapshot 2024-03-28T18:59:37.310 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/.snap/snap-subdir-test 2024-03-28T18:59:37.348 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/30 file after the snapshot 2024-03-28T18:59:37.353 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/31 file after the snapshot 2024-03-28T18:59:37.365 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/32 file after the snapshot 2024-03-28T18:59:37.378 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/33 file after the snapshot 2024-03-28T18:59:37.390 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/34 file after the snapshot 2024-03-28T18:59:37.402 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/35 file after the snapshot 2024-03-28T18:59:37.415 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/36 file after the snapshot 2024-03-28T18:59:37.427 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/37 file after the snapshot 2024-03-28T18:59:37.440 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/38 file after the snapshot 2024-03-28T18:59:37.454 INFO:tasks.workunit.client.1.smithi155.stdout: create 267/39 file after the snapshot 2024-03-28T18:59:37.466 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/.snap/snap-subdir-test 2024-03-28T18:59:37.504 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/30 file after the snapshot 2024-03-28T18:59:37.517 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/31 file after the snapshot 2024-03-28T18:59:37.613 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/32 file after the snapshot 2024-03-28T18:59:37.643 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/33 file after the snapshot 2024-03-28T18:59:37.653 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/34 file after the snapshot 2024-03-28T18:59:37.664 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/35 file after the snapshot 2024-03-28T18:59:37.672 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/36 file after the snapshot 2024-03-28T18:59:37.683 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/37 file after the snapshot 2024-03-28T18:59:37.693 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/38 file after the snapshot 2024-03-28T18:59:37.706 INFO:tasks.workunit.client.1.smithi155.stdout: create 268/39 file after the snapshot 2024-03-28T18:59:37.717 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/.snap/snap-subdir-test 2024-03-28T18:59:37.753 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/30 file after the snapshot 2024-03-28T18:59:37.757 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/31 file after the snapshot 2024-03-28T18:59:37.767 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/32 file after the snapshot 2024-03-28T18:59:37.779 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/33 file after the snapshot 2024-03-28T18:59:37.793 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/34 file after the snapshot 2024-03-28T18:59:37.806 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/35 file after the snapshot 2024-03-28T18:59:37.817 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/36 file after the snapshot 2024-03-28T18:59:37.829 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/37 file after the snapshot 2024-03-28T18:59:37.842 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/38 file after the snapshot 2024-03-28T18:59:37.854 INFO:tasks.workunit.client.1.smithi155.stdout: create 269/39 file after the snapshot 2024-03-28T18:59:37.867 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/.snap/snap-subdir-test 2024-03-28T18:59:37.902 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/30 file after the snapshot 2024-03-28T18:59:37.908 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/31 file after the snapshot 2024-03-28T18:59:37.918 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/32 file after the snapshot 2024-03-28T18:59:37.931 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/33 file after the snapshot 2024-03-28T18:59:37.945 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/34 file after the snapshot 2024-03-28T18:59:37.956 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/35 file after the snapshot 2024-03-28T18:59:37.967 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/36 file after the snapshot 2024-03-28T18:59:37.978 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/37 file after the snapshot 2024-03-28T18:59:37.991 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/38 file after the snapshot 2024-03-28T18:59:38.004 INFO:tasks.workunit.client.1.smithi155.stdout: create 270/39 file after the snapshot 2024-03-28T18:59:38.016 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/.snap/snap-subdir-test 2024-03-28T18:59:38.050 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/30 file after the snapshot 2024-03-28T18:59:38.056 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/31 file after the snapshot 2024-03-28T18:59:38.070 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/32 file after the snapshot 2024-03-28T18:59:38.082 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/33 file after the snapshot 2024-03-28T18:59:38.095 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/34 file after the snapshot 2024-03-28T18:59:38.109 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/35 file after the snapshot 2024-03-28T18:59:38.122 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/36 file after the snapshot 2024-03-28T18:59:38.135 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/37 file after the snapshot 2024-03-28T18:59:38.148 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/38 file after the snapshot 2024-03-28T18:59:38.161 INFO:tasks.workunit.client.1.smithi155.stdout: create 271/39 file after the snapshot 2024-03-28T18:59:38.174 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/.snap/snap-subdir-test 2024-03-28T18:59:38.214 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/30 file after the snapshot 2024-03-28T18:59:38.220 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/31 file after the snapshot 2024-03-28T18:59:38.230 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/32 file after the snapshot 2024-03-28T18:59:38.242 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/33 file after the snapshot 2024-03-28T18:59:38.254 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/34 file after the snapshot 2024-03-28T18:59:38.266 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/35 file after the snapshot 2024-03-28T18:59:38.277 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/36 file after the snapshot 2024-03-28T18:59:38.289 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/37 file after the snapshot 2024-03-28T18:59:38.300 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/38 file after the snapshot 2024-03-28T18:59:38.311 INFO:tasks.workunit.client.1.smithi155.stdout: create 272/39 file after the snapshot 2024-03-28T18:59:38.322 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/.snap/snap-subdir-test 2024-03-28T18:59:38.362 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/30 file after the snapshot 2024-03-28T18:59:38.368 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/31 file after the snapshot 2024-03-28T18:59:38.380 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/32 file after the snapshot 2024-03-28T18:59:38.392 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/33 file after the snapshot 2024-03-28T18:59:38.402 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/34 file after the snapshot 2024-03-28T18:59:38.415 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/35 file after the snapshot 2024-03-28T18:59:38.425 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:38.428 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/36 file after the snapshot 2024-03-28T18:59:38.443 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/37 file after the snapshot 2024-03-28T18:59:38.452 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/38 file after the snapshot 2024-03-28T18:59:38.465 INFO:tasks.workunit.client.1.smithi155.stdout: create 273/39 file after the snapshot 2024-03-28T18:59:38.477 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/.snap/snap-subdir-test 2024-03-28T18:59:38.509 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/30 file after the snapshot 2024-03-28T18:59:38.514 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/31 file after the snapshot 2024-03-28T18:59:38.611 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/32 file after the snapshot 2024-03-28T18:59:38.638 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/33 file after the snapshot 2024-03-28T18:59:38.646 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/34 file after the snapshot 2024-03-28T18:59:38.655 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/35 file after the snapshot 2024-03-28T18:59:38.665 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/36 file after the snapshot 2024-03-28T18:59:38.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.666+0000 7fc99e800640 1 Processor -- start 2024-03-28T18:59:38.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.667+0000 7fc99e800640 1 -- start start 2024-03-28T18:59:38.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980c59d0 0x7fc9980c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:38.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99e800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc9980727e0 0x7fc998159520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:38.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99e800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc998155fe0 con 0x7fc998155600 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99e800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9981565a0 con 0x7fc9980727e0 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99e800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc998159090 con 0x7fc9980c59d0 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc998155600 0x7fc99814f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57378/0 (socket says 172.21.15.27:57378) 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99de00640 1 -- 172.21.15.27:0/2345974485 learned_addr learned my addr 172.21.15.27:0/2345974485 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99ca00640 1 --2- 172.21.15.27:0/2345974485 >> v2:172.21.15.155:3300/0 conn(0x7fc9980727e0 0x7fc998159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.668+0000 7fc99d400640 1 --2- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980c59d0 0x7fc9980c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.669+0000 7fc997e00640 1 -- 172.21.15.27:0/2345974485 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201200118 0 0) 0x7fc998155fe0 con 0x7fc998155600 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.669+0000 7fc997e00640 1 -- 172.21.15.27:0/2345974485 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc980003520 con 0x7fc998155600 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.669+0000 7fc99ca00640 1 -- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980c59d0 msgr2=0x7fc9980c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.669+0000 7fc99ca00640 1 --2- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980c59d0 0x7fc9980c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.669+0000 7fc99ca00640 1 -- 172.21.15.27:0/2345974485 >> v1:172.21.15.27:6789/0 conn(0x7fc998155600 legacy=0x7fc99814f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.669+0000 7fc99ca00640 1 -- 172.21.15.27:0/2345974485 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9981562c0 con 0x7fc9980727e0 2024-03-28T18:59:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.669+0000 7fc99d400640 1 --2- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980c59d0 0x7fc9980c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:59:38.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.670+0000 7fc99ca00640 1 --2- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980727e0 0x7fc998159520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc9840097a0 tx=0x7fc98402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ed4b5c6e2866ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:38.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.671+0000 7fc997e00640 1 -- 172.21.15.27:0/2345974485 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc984039070 con 0x7fc9980727e0 2024-03-28T18:59:38.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.671+0000 7fc997e00640 1 -- 172.21.15.27:0/2345974485 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9840316f0 con 0x7fc9980727e0 2024-03-28T18:59:38.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.671+0000 7fc997e00640 1 -- 172.21.15.27:0/2345974485 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc984031ac0 con 0x7fc9980727e0 2024-03-28T18:59:38.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.672+0000 7fc99e800640 1 -- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980727e0 msgr2=0x7fc998159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:38.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.672+0000 7fc99e800640 1 --2- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980727e0 0x7fc998159520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc9840097a0 tx=0x7fc98402b790 comp rx=0 tx=0).stop 2024-03-28T18:59:38.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.672+0000 7fc99e800640 1 -- 172.21.15.27:0/2345974485 shutdown_connections 2024-03-28T18:59:38.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.672+0000 7fc99e800640 1 --2- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980727e0 0x7fc998159520 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.672+0000 7fc99e800640 1 --2- 172.21.15.27:0/2345974485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980c59d0 0x7fc9980c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.672+0000 7fc99e800640 1 -- 172.21.15.27:0/2345974485 >> 172.21.15.27:0/2345974485 conn(0x7fc9980ac910 msgr2=0x7fc9980accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:38.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.672+0000 7fc99e800640 1 -- 172.21.15.27:0/2345974485 shutdown_connections 2024-03-28T18:59:38.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.672+0000 7fc99e800640 1 -- 172.21.15.27:0/2345974485 wait complete. 2024-03-28T18:59:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.673+0000 7fc99e800640 1 Processor -- start 2024-03-28T18:59:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.674+0000 7fc99e800640 1 -- start start 2024-03-28T18:59:38.673 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/37 file after the snapshot 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.674+0000 7fc99e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980727e0 0x7fc9980c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.674+0000 7fc99e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980c59d0 0x7fc9980bf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc998155600 0x7fc9981eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99814f970 con 0x7fc998155600 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99e800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc998159a60 con 0x7fc9980c59d0 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99e800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99814faf0 con 0x7fc9980727e0 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc998155600 0x7fc9981eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc998155600 0x7fc9981eb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48262/0 (socket says 172.21.15.27:48262) 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99de00640 1 -- 172.21.15.27:0/904902067 learned_addr learned my addr 172.21.15.27:0/904902067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99d400640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980727e0 0x7fc9980c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99ca00640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980c59d0 0x7fc9980bf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99de00640 1 -- 172.21.15.27:0/904902067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980727e0 msgr2=0x7fc9980c48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99de00640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980727e0 0x7fc9980c48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99de00640 1 -- 172.21.15.27:0/904902067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980c59d0 msgr2=0x7fc9980bf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99de00640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980c59d0 0x7fc9980bf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.675+0000 7fc99de00640 1 -- 172.21.15.27:0/904902067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9981ead90 con 0x7fc998155600 2024-03-28T18:59:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.676+0000 7fc99de00640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc998155600 0x7fc9981eb370 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc99000caa0 tx=0x7fc99000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.676+0000 7fc995600640 1 -- 172.21.15.27:0/904902067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990017070 con 0x7fc998155600 2024-03-28T18:59:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.676+0000 7fc995600640 1 -- 172.21.15.27:0/904902067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99000aba0 con 0x7fc998155600 2024-03-28T18:59:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.676+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9980c10e0 con 0x7fc998155600 2024-03-28T18:59:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.676+0000 7fc995600640 1 -- 172.21.15.27:0/904902067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990012400 con 0x7fc998155600 2024-03-28T18:59:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.676+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9980c16a0 con 0x7fc998155600 2024-03-28T18:59:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.678+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc998067dd0 con 0x7fc998155600 2024-03-28T18:59:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.679+0000 7fc995600640 1 -- 172.21.15.27:0/904902067 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc99001a070 con 0x7fc998155600 2024-03-28T18:59:38.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.679+0000 7fc995600640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97806bbd0 0x7fc97806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:38.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.680+0000 7fc995600640 1 -- 172.21.15.27:0/904902067 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7fc99008aba0 con 0x7fc998155600 2024-03-28T18:59:38.679 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/38 file after the snapshot 2024-03-28T18:59:38.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.680+0000 7fc99d400640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97806bbd0 0x7fc97806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:38.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.681+0000 7fc99d400640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97806bbd0 0x7fc97806e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc98c009560 tx=0x7fc98c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:38.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.684+0000 7fc995600640 1 -- 172.21.15.27:0/904902067 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc990058c30 con 0x7fc998155600 2024-03-28T18:59:38.687 INFO:tasks.workunit.client.1.smithi155.stdout: create 274/39 file after the snapshot 2024-03-28T18:59:38.695 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/.snap/snap-subdir-test 2024-03-28T18:59:38.720 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/30 file after the snapshot 2024-03-28T18:59:38.723 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/31 file after the snapshot 2024-03-28T18:59:38.731 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/32 file after the snapshot 2024-03-28T18:59:38.741 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/33 file after the snapshot 2024-03-28T18:59:38.749 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/34 file after the snapshot 2024-03-28T18:59:38.758 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/35 file after the snapshot 2024-03-28T18:59:38.771 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/36 file after the snapshot 2024-03-28T18:59:38.781 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/37 file after the snapshot 2024-03-28T18:59:38.793 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/38 file after the snapshot 2024-03-28T18:59:38.803 INFO:tasks.workunit.client.1.smithi155.stdout: create 275/39 file after the snapshot 2024-03-28T18:59:38.816 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/.snap/snap-subdir-test 2024-03-28T18:59:38.850 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/30 file after the snapshot 2024-03-28T18:59:38.854 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/31 file after the snapshot 2024-03-28T18:59:38.864 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/32 file after the snapshot 2024-03-28T18:59:38.876 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/33 file after the snapshot 2024-03-28T18:59:38.888 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/34 file after the snapshot 2024-03-28T18:59:38.899 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/35 file after the snapshot 2024-03-28T18:59:38.911 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/36 file after the snapshot 2024-03-28T18:59:38.922 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/37 file after the snapshot 2024-03-28T18:59:38.932 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/38 file after the snapshot 2024-03-28T18:59:38.942 INFO:tasks.workunit.client.1.smithi155.stdout: create 276/39 file after the snapshot 2024-03-28T18:59:38.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.952+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc998069640 con 0x7fc998155600 2024-03-28T18:59:38.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.954+0000 7fc995600640 1 -- 172.21.15.27:0/904902067 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7fc99005ccd0 con 0x7fc998155600 2024-03-28T18:59:38.953 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:38.953 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:38.954 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:38.954 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/.snap/snap-subdir-test 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.956+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97806bbd0 msgr2=0x7fc97806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.956+0000 7fc99e800640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97806bbd0 0x7fc97806e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc98c009560 tx=0x7fc98c009190 comp rx=0 tx=0).stop 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.956+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc998155600 msgr2=0x7fc9981eb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.956+0000 7fc99e800640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc998155600 0x7fc9981eb370 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc99000caa0 tx=0x7fc99000cf60 comp rx=0 tx=0).stop 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.957+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 shutdown_connections 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.957+0000 7fc99e800640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97806bbd0 0x7fc97806e090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.957+0000 7fc99e800640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc998155600 0x7fc9981eb370 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.957+0000 7fc99e800640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9980c59d0 0x7fc9980bf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.957+0000 7fc99e800640 1 --2- 172.21.15.27:0/904902067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9980727e0 0x7fc9980c48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:38.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.957+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 >> 172.21.15.27:0/904902067 conn(0x7fc9980ac910 msgr2=0x7fc9980c9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:38.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.957+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 shutdown_connections 2024-03-28T18:59:38.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:38.957+0000 7fc99e800640 1 -- 172.21.15.27:0/904902067 wait complete. 2024-03-28T18:59:38.975 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:59:38.985 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/30 file after the snapshot 2024-03-28T18:59:38.990 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/31 file after the snapshot 2024-03-28T18:59:38.999 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/32 file after the snapshot 2024-03-28T18:59:39.010 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/33 file after the snapshot 2024-03-28T18:59:39.023 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/34 file after the snapshot 2024-03-28T18:59:39.036 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/35 file after the snapshot 2024-03-28T18:59:39.049 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/36 file after the snapshot 2024-03-28T18:59:39.062 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/37 file after the snapshot 2024-03-28T18:59:39.072 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/38 file after the snapshot 2024-03-28T18:59:39.083 INFO:tasks.workunit.client.1.smithi155.stdout: create 277/39 file after the snapshot 2024-03-28T18:59:39.097 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/.snap/snap-subdir-test 2024-03-28T18:59:39.134 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/30 file after the snapshot 2024-03-28T18:59:39.139 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/31 file after the snapshot 2024-03-28T18:59:39.152 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/32 file after the snapshot 2024-03-28T18:59:39.164 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/33 file after the snapshot 2024-03-28T18:59:39.177 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/34 file after the snapshot 2024-03-28T18:59:39.189 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/35 file after the snapshot 2024-03-28T18:59:39.202 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/36 file after the snapshot 2024-03-28T18:59:39.215 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/37 file after the snapshot 2024-03-28T18:59:39.227 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/38 file after the snapshot 2024-03-28T18:59:39.239 INFO:tasks.workunit.client.1.smithi155.stdout: create 278/39 file after the snapshot 2024-03-28T18:59:39.252 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/.snap/snap-subdir-test 2024-03-28T18:59:39.291 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/30 file after the snapshot 2024-03-28T18:59:39.296 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/31 file after the snapshot 2024-03-28T18:59:39.307 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/32 file after the snapshot 2024-03-28T18:59:39.320 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/33 file after the snapshot 2024-03-28T18:59:39.332 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/34 file after the snapshot 2024-03-28T18:59:39.346 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/35 file after the snapshot 2024-03-28T18:59:39.358 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/36 file after the snapshot 2024-03-28T18:59:39.371 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/37 file after the snapshot 2024-03-28T18:59:39.385 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/38 file after the snapshot 2024-03-28T18:59:39.398 INFO:tasks.workunit.client.1.smithi155.stdout: create 279/39 file after the snapshot 2024-03-28T18:59:39.411 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/.snap/snap-subdir-test 2024-03-28T18:59:39.450 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/30 file after the snapshot 2024-03-28T18:59:39.457 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/31 file after the snapshot 2024-03-28T18:59:39.470 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/32 file after the snapshot 2024-03-28T18:59:39.481 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/33 file after the snapshot 2024-03-28T18:59:39.494 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/34 file after the snapshot 2024-03-28T18:59:39.507 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/35 file after the snapshot 2024-03-28T18:59:39.519 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/36 file after the snapshot 2024-03-28T18:59:39.540 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/37 file after the snapshot 2024-03-28T18:59:39.598 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/38 file after the snapshot 2024-03-28T18:59:39.632 INFO:tasks.workunit.client.1.smithi155.stdout: create 280/39 file after the snapshot 2024-03-28T18:59:39.646 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/.snap/snap-subdir-test 2024-03-28T18:59:39.679 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/30 file after the snapshot 2024-03-28T18:59:39.685 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/31 file after the snapshot 2024-03-28T18:59:39.696 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/32 file after the snapshot 2024-03-28T18:59:39.710 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/33 file after the snapshot 2024-03-28T18:59:39.720 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/34 file after the snapshot 2024-03-28T18:59:39.733 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/35 file after the snapshot 2024-03-28T18:59:39.746 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/36 file after the snapshot 2024-03-28T18:59:39.759 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/37 file after the snapshot 2024-03-28T18:59:39.771 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/38 file after the snapshot 2024-03-28T18:59:39.784 INFO:tasks.workunit.client.1.smithi155.stdout: create 281/39 file after the snapshot 2024-03-28T18:59:39.797 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/.snap/snap-subdir-test 2024-03-28T18:59:39.835 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/30 file after the snapshot 2024-03-28T18:59:39.839 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/31 file after the snapshot 2024-03-28T18:59:39.852 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/32 file after the snapshot 2024-03-28T18:59:39.865 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/33 file after the snapshot 2024-03-28T18:59:39.878 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/34 file after the snapshot 2024-03-28T18:59:39.892 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/35 file after the snapshot 2024-03-28T18:59:39.905 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/36 file after the snapshot 2024-03-28T18:59:39.917 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/37 file after the snapshot 2024-03-28T18:59:39.930 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/38 file after the snapshot 2024-03-28T18:59:39.940 INFO:tasks.workunit.client.1.smithi155.stdout: create 282/39 file after the snapshot 2024-03-28T18:59:39.953 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/.snap/snap-subdir-test 2024-03-28T18:59:39.985 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/30 file after the snapshot 2024-03-28T18:59:39.992 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/31 file after the snapshot 2024-03-28T18:59:40.005 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/32 file after the snapshot 2024-03-28T18:59:40.018 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/33 file after the snapshot 2024-03-28T18:59:40.030 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/34 file after the snapshot 2024-03-28T18:59:40.041 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/35 file after the snapshot 2024-03-28T18:59:40.052 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/36 file after the snapshot 2024-03-28T18:59:40.063 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/37 file after the snapshot 2024-03-28T18:59:40.074 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/38 file after the snapshot 2024-03-28T18:59:40.085 INFO:tasks.workunit.client.1.smithi155.stdout: create 283/39 file after the snapshot 2024-03-28T18:59:40.097 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/.snap/snap-subdir-test 2024-03-28T18:59:40.135 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/30 file after the snapshot 2024-03-28T18:59:40.143 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/31 file after the snapshot 2024-03-28T18:59:40.154 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/32 file after the snapshot 2024-03-28T18:59:40.165 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/33 file after the snapshot 2024-03-28T18:59:40.177 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/34 file after the snapshot 2024-03-28T18:59:40.189 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/35 file after the snapshot 2024-03-28T18:59:40.201 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/36 file after the snapshot 2024-03-28T18:59:40.214 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/37 file after the snapshot 2024-03-28T18:59:40.248 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/38 file after the snapshot 2024-03-28T18:59:40.270 INFO:tasks.workunit.client.1.smithi155.stdout: create 284/39 file after the snapshot 2024-03-28T18:59:40.280 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/.snap/snap-subdir-test 2024-03-28T18:59:40.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-28T18:59:40.316 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/30 file after the snapshot 2024-03-28T18:59:40.320 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/31 file after the snapshot 2024-03-28T18:59:40.331 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/32 file after the snapshot 2024-03-28T18:59:40.344 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/33 file after the snapshot 2024-03-28T18:59:40.357 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/34 file after the snapshot 2024-03-28T18:59:40.370 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/35 file after the snapshot 2024-03-28T18:59:40.382 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/36 file after the snapshot 2024-03-28T18:59:40.393 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/37 file after the snapshot 2024-03-28T18:59:40.407 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/38 file after the snapshot 2024-03-28T18:59:40.421 INFO:tasks.workunit.client.1.smithi155.stdout: create 285/39 file after the snapshot 2024-03-28T18:59:40.430 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/.snap/snap-subdir-test 2024-03-28T18:59:40.463 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/30 file after the snapshot 2024-03-28T18:59:40.467 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/31 file after the snapshot 2024-03-28T18:59:40.476 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/32 file after the snapshot 2024-03-28T18:59:40.489 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/33 file after the snapshot 2024-03-28T18:59:40.502 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/34 file after the snapshot 2024-03-28T18:59:40.513 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/35 file after the snapshot 2024-03-28T18:59:40.526 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/36 file after the snapshot 2024-03-28T18:59:40.536 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/37 file after the snapshot 2024-03-28T18:59:40.848 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/38 file after the snapshot 2024-03-28T18:59:40.886 INFO:tasks.workunit.client.1.smithi155.stdout: create 286/39 file after the snapshot 2024-03-28T18:59:40.897 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/.snap/snap-subdir-test 2024-03-28T18:59:40.931 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/30 file after the snapshot 2024-03-28T18:59:40.936 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/31 file after the snapshot 2024-03-28T18:59:40.947 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/32 file after the snapshot 2024-03-28T18:59:40.958 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/33 file after the snapshot 2024-03-28T18:59:40.969 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/34 file after the snapshot 2024-03-28T18:59:40.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:40.981 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/35 file after the snapshot 2024-03-28T18:59:40.993 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/36 file after the snapshot 2024-03-28T18:59:41.004 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/37 file after the snapshot 2024-03-28T18:59:41.014 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/38 file after the snapshot 2024-03-28T18:59:41.024 INFO:tasks.workunit.client.1.smithi155.stdout: create 287/39 file after the snapshot 2024-03-28T18:59:41.033 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/.snap/snap-subdir-test 2024-03-28T18:59:41.063 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/30 file after the snapshot 2024-03-28T18:59:41.068 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/31 file after the snapshot 2024-03-28T18:59:41.079 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/32 file after the snapshot 2024-03-28T18:59:41.091 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/33 file after the snapshot 2024-03-28T18:59:41.103 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/34 file after the snapshot 2024-03-28T18:59:41.112 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/35 file after the snapshot 2024-03-28T18:59:41.123 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/36 file after the snapshot 2024-03-28T18:59:41.134 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/37 file after the snapshot 2024-03-28T18:59:41.147 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/38 file after the snapshot 2024-03-28T18:59:41.159 INFO:tasks.workunit.client.1.smithi155.stdout: create 288/39 file after the snapshot 2024-03-28T18:59:41.173 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/.snap/snap-subdir-test 2024-03-28T18:59:41.202 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/30 file after the snapshot 2024-03-28T18:59:41.208 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/31 file after the snapshot 2024-03-28T18:59:41.217 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/32 file after the snapshot 2024-03-28T18:59:41.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.226+0000 7f8e8b200640 1 Processor -- start 2024-03-28T18:59:41.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.226+0000 7f8e8b200640 1 -- start start 2024-03-28T18:59:41.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.227+0000 7f8e8b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e8c0c58c0 0x7f8e8c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:41.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.227+0000 7f8e8b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c158bd0 con 0x7f8e8c0be5d0 2024-03-28T18:59:41.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.227+0000 7f8e8b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c158eb0 con 0x7f8e8c0c8ed0 2024-03-28T18:59:41.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.227+0000 7f8e8b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c159190 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.228+0000 7f8e89e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e8c0c58c0 0x7f8e8c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:41.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.228+0000 7f8e89e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e8c0c58c0 0x7f8e8c0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48274/0 (socket says 172.21.15.27:48274) 2024-03-28T18:59:41.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.228+0000 7f8e89e00640 1 -- 172.21.15.27:0/2788189101 learned_addr learned my addr 172.21.15.27:0/2788189101 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:41.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.228+0000 7f8e8a800640 1 --1- 172.21.15.27:0/2788189101 >> v1:172.21.15.27:6789/0 conn(0x7f8e8c0be5d0 0x7f8e8c155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57390/0 (socket says 172.21.15.27:57390) 2024-03-28T18:59:41.228 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/33 file after the snapshot 2024-03-28T18:59:41.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.228+0000 7f8e89e00640 1 -- 172.21.15.27:0/2788189101 >> v1:172.21.15.155:6789/0 conn(0x7f8e8c0c8ed0 legacy=0x7f8e8c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:41.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.228+0000 7f8e89e00640 1 -- 172.21.15.27:0/2788189101 >> v1:172.21.15.27:6789/0 conn(0x7f8e8c0be5d0 legacy=0x7f8e8c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:41.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.228+0000 7f8e89e00640 1 -- 172.21.15.27:0/2788189101 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c159470 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.228+0000 7f8e89400640 1 -- 172.21.15.27:0/2788189101 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516461093 0 0) 0x7f8e8c158bd0 con 0x7f8e8c0be5d0 2024-03-28T18:59:41.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.230+0000 7f8e89e00640 1 --2- 172.21.15.27:0/2788189101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 0x7f8e8c0c5ca0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009e50 tx=0x7f8e7c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2d5a769a69d99d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:41.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.230+0000 7f8e89400640 1 -- 172.21.15.27:0/2788189101 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c02ca80 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.230+0000 7f8e89400640 1 -- 172.21.15.27:0/2788189101 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c02cc20 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.230+0000 7f8e89400640 1 -- 172.21.15.27:0/2788189101 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0343f0 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.230+0000 7f8e8b200640 1 -- 172.21.15.27:0/2788189101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 msgr2=0x7f8e8c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.230+0000 7f8e8b200640 1 --2- 172.21.15.27:0/2788189101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 0x7f8e8c0c5ca0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009e50 tx=0x7f8e7c02bee0 comp rx=0 tx=0).stop 2024-03-28T18:59:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.231+0000 7f8e8b200640 1 -- 172.21.15.27:0/2788189101 shutdown_connections 2024-03-28T18:59:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.231+0000 7f8e8b200640 1 --2- 172.21.15.27:0/2788189101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 0x7f8e8c0c5ca0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.231+0000 7f8e8b200640 1 -- 172.21.15.27:0/2788189101 >> 172.21.15.27:0/2788189101 conn(0x7f8e8c06b6f0 msgr2=0x7f8e8c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.231+0000 7f8e8b200640 1 -- 172.21.15.27:0/2788189101 shutdown_connections 2024-03-28T18:59:41.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.231+0000 7f8e8b200640 1 -- 172.21.15.27:0/2788189101 wait complete. 2024-03-28T18:59:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.232+0000 7f8e8b200640 1 Processor -- start 2024-03-28T18:59:41.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.232+0000 7f8e8b200640 1 -- start start 2024-03-28T18:59:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.233+0000 7f8e8b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e8c0be5d0 0x7f8e8c156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.233+0000 7f8e8b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 0x7f8e8c0bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.233+0000 7f8e8b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e8c0c8ed0 0x7f8e8c0befb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.233+0000 7f8e8b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c0b09a0 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.233+0000 7f8e8b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c0b0820 con 0x7f8e8c0c8ed0 2024-03-28T18:59:41.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.233+0000 7f8e8b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c0ad7d0 con 0x7f8e8c0be5d0 2024-03-28T18:59:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e8a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e8c0c8ed0 0x7f8e8c0befb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e89e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e8c0be5d0 0x7f8e8c156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e81400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 0x7f8e8c0bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e89e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e8c0be5d0 0x7f8e8c156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:50070/0 (socket says 172.21.15.27:50070) 2024-03-28T18:59:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e8a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e8c0c8ed0 0x7f8e8c0befb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:53710/0 (socket says 172.21.15.27:53710) 2024-03-28T18:59:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e89e00640 1 -- 172.21.15.27:0/3629519024 learned_addr learned my addr 172.21.15.27:0/3629519024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e81400640 1 -- 172.21.15.27:0/3629519024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e8c0be5d0 msgr2=0x7f8e8c156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:41.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e81400640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e8c0be5d0 0x7f8e8c156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e81400640 1 -- 172.21.15.27:0/3629519024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e8c0c8ed0 msgr2=0x7f8e8c0befb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e81400640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e8c0c8ed0 0x7f8e8c0befb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.234+0000 7f8e81400640 1 -- 172.21.15.27:0/3629519024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c0c1d40 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.235+0000 7f8e81400640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 0x7f8e8c0bea70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400c390 tx=0x7f8e7400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:41.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.235+0000 7f8e82a00640 1 -- 172.21.15.27:0/3629519024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74017070 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.235+0000 7f8e82a00640 1 -- 172.21.15.27:0/3629519024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e740052d0 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.235+0000 7f8e82a00640 1 -- 172.21.15.27:0/3629519024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74003810 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.235+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c0c4590 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.235+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c157180 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.237+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c069640 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.237+0000 7f8e82a00640 1 -- 172.21.15.27:0/3629519024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8e7401a070 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.238+0000 7f8e82a00640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e6806bc00 0x7f8e6806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.238+0000 7f8e82a00640 1 -- 172.21.15.27:0/3629519024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f8e7408a0f0 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.239+0000 7f8e89e00640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e6806bc00 0x7f8e6806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:41.238 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/34 file after the snapshot 2024-03-28T18:59:41.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.239+0000 7f8e89e00640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e6806bc00 0x7f8e6806e0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c02bf20 tx=0x7f8e7c030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.244+0000 7f8e82a00640 1 -- 172.21.15.27:0/3629519024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8e740580e0 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.250 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/35 file after the snapshot 2024-03-28T18:59:41.258 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/36 file after the snapshot 2024-03-28T18:59:41.268 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/37 file after the snapshot 2024-03-28T18:59:41.277 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/38 file after the snapshot 2024-03-28T18:59:41.286 INFO:tasks.workunit.client.1.smithi155.stdout: create 289/39 file after the snapshot 2024-03-28T18:59:41.296 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/.snap/snap-subdir-test 2024-03-28T18:59:41.325 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/30 file after the snapshot 2024-03-28T18:59:41.331 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/31 file after the snapshot 2024-03-28T18:59:41.343 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/32 file after the snapshot 2024-03-28T18:59:41.355 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/33 file after the snapshot 2024-03-28T18:59:41.368 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/34 file after the snapshot 2024-03-28T18:59:41.377 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/35 file after the snapshot 2024-03-28T18:59:41.390 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/36 file after the snapshot 2024-03-28T18:59:41.400 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/37 file after the snapshot 2024-03-28T18:59:41.410 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/38 file after the snapshot 2024-03-28T18:59:41.422 INFO:tasks.workunit.client.1.smithi155.stdout: create 290/39 file after the snapshot 2024-03-28T18:59:41.436 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/.snap/snap-subdir-test 2024-03-28T18:59:41.466 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/30 file after the snapshot 2024-03-28T18:59:41.471 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/31 file after the snapshot 2024-03-28T18:59:41.480 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/32 file after the snapshot 2024-03-28T18:59:41.491 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/33 file after the snapshot 2024-03-28T18:59:41.501 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/34 file after the snapshot 2024-03-28T18:59:41.511 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/35 file after the snapshot 2024-03-28T18:59:41.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.518+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e8c0bf9d0 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.519+0000 7f8e82a00640 1 -- 172.21.15.27:0/3629519024 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f8e7405c180 con 0x7f8e8c0c58c0 2024-03-28T18:59:41.519 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:41.519 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:41.519 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:41.522 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/36 file after the snapshot 2024-03-28T18:59:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e6806bc00 msgr2=0x7f8e6806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e6806bc00 0x7f8e6806e0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c02bf20 tx=0x7f8e7c030000 comp rx=0 tx=0).stop 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 msgr2=0x7f8e8c0bea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 0x7f8e8c0bea70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400c390 tx=0x7f8e7400c850 comp rx=0 tx=0).stop 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 shutdown_connections 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e6806bc00 0x7f8e6806e0c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e8c0c8ed0 0x7f8e8c0befb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0c58c0 0x7f8e8c0bea70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.522+0000 7f8e8b200640 1 --2- 172.21.15.27:0/3629519024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8e8c0be5d0 0x7f8e8c156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.523+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 >> 172.21.15.27:0/3629519024 conn(0x7f8e8c06b6f0 msgr2=0x7f8e8c1586b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.523+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 shutdown_connections 2024-03-28T18:59:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:41.523+0000 7f8e8b200640 1 -- 172.21.15.27:0/3629519024 wait complete. 2024-03-28T18:59:41.530 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/37 file after the snapshot 2024-03-28T18:59:41.541 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:59:41.553 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/38 file after the snapshot 2024-03-28T18:59:41.582 INFO:tasks.workunit.client.1.smithi155.stdout: create 291/39 file after the snapshot 2024-03-28T18:59:41.599 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/.snap/snap-subdir-test 2024-03-28T18:59:41.630 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/30 file after the snapshot 2024-03-28T18:59:41.635 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/31 file after the snapshot 2024-03-28T18:59:41.644 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/32 file after the snapshot 2024-03-28T18:59:41.653 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/33 file after the snapshot 2024-03-28T18:59:41.661 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/34 file after the snapshot 2024-03-28T18:59:41.669 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/35 file after the snapshot 2024-03-28T18:59:41.679 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/36 file after the snapshot 2024-03-28T18:59:41.687 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/37 file after the snapshot 2024-03-28T18:59:41.696 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/38 file after the snapshot 2024-03-28T18:59:41.705 INFO:tasks.workunit.client.1.smithi155.stdout: create 292/39 file after the snapshot 2024-03-28T18:59:41.717 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/.snap/snap-subdir-test 2024-03-28T18:59:41.751 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/30 file after the snapshot 2024-03-28T18:59:41.756 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/31 file after the snapshot 2024-03-28T18:59:41.764 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/32 file after the snapshot 2024-03-28T18:59:41.772 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/33 file after the snapshot 2024-03-28T18:59:41.781 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/34 file after the snapshot 2024-03-28T18:59:41.793 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/35 file after the snapshot 2024-03-28T18:59:41.804 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/36 file after the snapshot 2024-03-28T18:59:41.816 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/37 file after the snapshot 2024-03-28T18:59:41.826 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/38 file after the snapshot 2024-03-28T18:59:41.836 INFO:tasks.workunit.client.1.smithi155.stdout: create 293/39 file after the snapshot 2024-03-28T18:59:41.846 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/.snap/snap-subdir-test 2024-03-28T18:59:41.871 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/30 file after the snapshot 2024-03-28T18:59:41.877 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/31 file after the snapshot 2024-03-28T18:59:41.887 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/32 file after the snapshot 2024-03-28T18:59:41.897 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/33 file after the snapshot 2024-03-28T18:59:41.907 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/34 file after the snapshot 2024-03-28T18:59:41.917 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/35 file after the snapshot 2024-03-28T18:59:41.929 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/36 file after the snapshot 2024-03-28T18:59:41.939 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/37 file after the snapshot 2024-03-28T18:59:41.948 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/38 file after the snapshot 2024-03-28T18:59:41.957 INFO:tasks.workunit.client.1.smithi155.stdout: create 294/39 file after the snapshot 2024-03-28T18:59:41.967 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/.snap/snap-subdir-test 2024-03-28T18:59:42.000 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/30 file after the snapshot 2024-03-28T18:59:42.006 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/31 file after the snapshot 2024-03-28T18:59:42.015 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/32 file after the snapshot 2024-03-28T18:59:42.025 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/33 file after the snapshot 2024-03-28T18:59:42.037 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/34 file after the snapshot 2024-03-28T18:59:42.050 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/35 file after the snapshot 2024-03-28T18:59:42.063 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/36 file after the snapshot 2024-03-28T18:59:42.074 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/37 file after the snapshot 2024-03-28T18:59:42.087 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/38 file after the snapshot 2024-03-28T18:59:42.099 INFO:tasks.workunit.client.1.smithi155.stdout: create 295/39 file after the snapshot 2024-03-28T18:59:42.112 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/.snap/snap-subdir-test 2024-03-28T18:59:42.149 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/30 file after the snapshot 2024-03-28T18:59:42.155 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/31 file after the snapshot 2024-03-28T18:59:42.164 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/32 file after the snapshot 2024-03-28T18:59:42.176 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/33 file after the snapshot 2024-03-28T18:59:42.187 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/34 file after the snapshot 2024-03-28T18:59:42.201 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/35 file after the snapshot 2024-03-28T18:59:42.212 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/36 file after the snapshot 2024-03-28T18:59:42.224 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/37 file after the snapshot 2024-03-28T18:59:42.236 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/38 file after the snapshot 2024-03-28T18:59:42.247 INFO:tasks.workunit.client.1.smithi155.stdout: create 296/39 file after the snapshot 2024-03-28T18:59:42.260 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/.snap/snap-subdir-test 2024-03-28T18:59:42.296 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/30 file after the snapshot 2024-03-28T18:59:42.302 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/31 file after the snapshot 2024-03-28T18:59:42.314 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/32 file after the snapshot 2024-03-28T18:59:42.326 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/33 file after the snapshot 2024-03-28T18:59:42.338 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/34 file after the snapshot 2024-03-28T18:59:42.350 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/35 file after the snapshot 2024-03-28T18:59:42.363 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/36 file after the snapshot 2024-03-28T18:59:42.376 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/37 file after the snapshot 2024-03-28T18:59:42.386 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/38 file after the snapshot 2024-03-28T18:59:42.397 INFO:tasks.workunit.client.1.smithi155.stdout: create 297/39 file after the snapshot 2024-03-28T18:59:42.411 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/.snap/snap-subdir-test 2024-03-28T18:59:42.449 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/30 file after the snapshot 2024-03-28T18:59:42.457 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/31 file after the snapshot 2024-03-28T18:59:42.469 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/32 file after the snapshot 2024-03-28T18:59:42.483 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/33 file after the snapshot 2024-03-28T18:59:42.496 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/34 file after the snapshot 2024-03-28T18:59:42.507 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/35 file after the snapshot 2024-03-28T18:59:42.514 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/36 file after the snapshot 2024-03-28T18:59:42.522 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/37 file after the snapshot 2024-03-28T18:59:42.533 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/38 file after the snapshot 2024-03-28T18:59:42.544 INFO:tasks.workunit.client.1.smithi155.stdout: create 298/39 file after the snapshot 2024-03-28T18:59:42.568 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/.snap/snap-subdir-test 2024-03-28T18:59:42.646 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/30 file after the snapshot 2024-03-28T18:59:42.652 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/31 file after the snapshot 2024-03-28T18:59:42.660 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/32 file after the snapshot 2024-03-28T18:59:42.670 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/33 file after the snapshot 2024-03-28T18:59:42.680 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/34 file after the snapshot 2024-03-28T18:59:42.691 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/35 file after the snapshot 2024-03-28T18:59:42.700 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/36 file after the snapshot 2024-03-28T18:59:42.709 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/37 file after the snapshot 2024-03-28T18:59:42.721 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/38 file after the snapshot 2024-03-28T18:59:42.732 INFO:tasks.workunit.client.1.smithi155.stdout: create 299/39 file after the snapshot 2024-03-28T18:59:42.744 INFO:tasks.workunit.client.1.smithi155.stdout:Take second snapshot .snap/test2 2024-03-28T18:59:42.873 INFO:tasks.workunit.client.1.smithi155.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-03-28T18:59:43.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.785+0000 7fb009000640 1 Processor -- start 2024-03-28T18:59:43.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.785+0000 7fb009000640 1 -- start start 2024-03-28T18:59:43.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.787+0000 7fb009000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0040c69a0 0x7fb0040c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:43.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.787+0000 7fb009000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb004153a30 0x7fb004153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:43.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.787+0000 7fb009000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb004150ec0 con 0x7fb004150ae0 2024-03-28T18:59:43.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.787+0000 7fb009000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00406dc70 con 0x7fb004153a30 2024-03-28T18:59:43.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.787+0000 7fb009000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb004153550 con 0x7fb0040c69a0 2024-03-28T18:59:43.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.787+0000 7fb003e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb004153a30 0x7fb004153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:43.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.787+0000 7fb002a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0040c69a0 0x7fb0040c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:43.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.788+0000 7fb003e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb004153a30 0x7fb004153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:53714/0 (socket says 172.21.15.27:53714) 2024-03-28T18:59:43.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.788+0000 7fb003e00640 1 -- 172.21.15.27:0/1791840949 learned_addr learned my addr 172.21.15.27:0/1791840949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:43.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.788+0000 7fb002a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0040c69a0 0x7fb0040c6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:50086/0 (socket says 172.21.15.27:50086) 2024-03-28T18:59:43.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.788+0000 7fb002000640 1 -- 172.21.15.27:0/1791840949 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013444319 0 0) 0x7faff4001110 con 0x7fb004150ae0 2024-03-28T18:59:43.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.788+0000 7fb002000640 1 -- 172.21.15.27:0/1791840949 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe8003520 con 0x7fb004150ae0 2024-03-28T18:59:43.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.788+0000 7fb002a00640 1 -- 172.21.15.27:0/1791840949 >> v2:172.21.15.155:3300/0 conn(0x7fb004153a30 msgr2=0x7fb004153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:43.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.788+0000 7fb002a00640 1 --2- 172.21.15.27:0/1791840949 >> v2:172.21.15.155:3300/0 conn(0x7fb004153a30 0x7fb004153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:43.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.788+0000 7fb002a00640 1 -- 172.21.15.27:0/1791840949 >> v1:172.21.15.155:6789/0 conn(0x7fb004150ae0 legacy=0x7fb0040a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:43.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.789+0000 7fb002000640 1 -- 172.21.15.27:0/1791840949 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1870416829 0 0) 0x7fafe8003520 con 0x7fb004150ae0 2024-03-28T18:59:43.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.789+0000 7fb002a00640 1 -- 172.21.15.27:0/1791840949 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00406df10 con 0x7fb0040c69a0 2024-03-28T18:59:43.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.789+0000 7fb003e00640 1 --2- 172.21.15.27:0/1791840949 >> v2:172.21.15.155:3300/0 conn(0x7fb004153a30 0x7fb004153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:59:43.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.790+0000 7fb002a00640 1 --2- 172.21.15.27:0/1791840949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0040c69a0 0x7fb0040c6e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faff00097a0 tx=0x7faff002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=263831084d55eed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:43.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.790+0000 7fb002000640 1 -- 172.21.15.27:0/1791840949 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0037070 con 0x7fb0040c69a0 2024-03-28T18:59:43.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb002000640 1 -- 172.21.15.27:0/1791840949 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff0030510 con 0x7fb0040c69a0 2024-03-28T18:59:43.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb002000640 1 -- 172.21.15.27:0/1791840949 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4001840 con 0x7fb0040c69a0 2024-03-28T18:59:43.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb009000640 1 -- 172.21.15.27:0/1791840949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0040c69a0 msgr2=0x7fb0040c6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:43.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb009000640 1 --2- 172.21.15.27:0/1791840949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0040c69a0 0x7fb0040c6e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faff00097a0 tx=0x7faff002b710 comp rx=0 tx=0).stop 2024-03-28T18:59:43.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb009000640 1 -- 172.21.15.27:0/1791840949 shutdown_connections 2024-03-28T18:59:43.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb009000640 1 --2- 172.21.15.27:0/1791840949 >> v2:172.21.15.155:3300/0 conn(0x7fb004153a30 0x7fb004153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:43.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb009000640 1 --2- 172.21.15.27:0/1791840949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0040c69a0 0x7fb0040c6e00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:43.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb009000640 1 -- 172.21.15.27:0/1791840949 >> 172.21.15.27:0/1791840949 conn(0x7fb00406b0c0 msgr2=0x7fb00406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:43.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb009000640 1 -- 172.21.15.27:0/1791840949 shutdown_connections 2024-03-28T18:59:43.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.791+0000 7fb009000640 1 -- 172.21.15.27:0/1791840949 wait complete. 2024-03-28T18:59:43.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.793+0000 7fb009000640 1 Processor -- start 2024-03-28T18:59:43.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.793+0000 7fb009000640 1 -- start start 2024-03-28T18:59:43.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.794+0000 7fb009000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0040c69a0 0x7fb0041e8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:43.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.794+0000 7fb009000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004150ae0 0x7fb0040bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:43.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.794+0000 7fb009000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb004153a30 0x7fb0041eac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:43.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.794+0000 7fb009000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00406e7b0 con 0x7fb004150ae0 2024-03-28T18:59:43.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.794+0000 7fb009000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00406e4d0 con 0x7fb0040c69a0 2024-03-28T18:59:43.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.794+0000 7fb009000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0040a6370 con 0x7fb004153a30 2024-03-28T18:59:43.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb002a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004150ae0 0x7fb0040bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:43.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb002a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004150ae0 0x7fb0040bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48282/0 (socket says 172.21.15.27:48282) 2024-03-28T18:59:43.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb002a00640 1 -- 172.21.15.27:0/600738281 learned_addr learned my addr 172.21.15.27:0/600738281 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:43.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb003400640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0040c69a0 0x7fb0041e8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:43.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb003e00640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb004153a30 0x7fb0041eac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:43.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb002a00640 1 -- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb004153a30 msgr2=0x7fb0041eac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:43.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb002a00640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb004153a30 0x7fb0041eac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:43.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb002a00640 1 -- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0040c69a0 msgr2=0x7fb0041e8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:43.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb002a00640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0040c69a0 0x7fb0041e8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:43.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb002a00640 1 -- 172.21.15.27:0/600738281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0040a6980 con 0x7fb004150ae0 2024-03-28T18:59:43.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.795+0000 7fb003400640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0040c69a0 0x7fb0041e8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:59:43.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.796+0000 7fb002a00640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004150ae0 0x7fb0040bfd20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7faff002b750 tx=0x7faff00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:43.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.796+0000 7faffb400640 1 -- 172.21.15.27:0/600738281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0040020 con 0x7fb004150ae0 2024-03-28T18:59:43.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.796+0000 7faffb400640 1 -- 172.21.15.27:0/600738281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff0038040 con 0x7fb004150ae0 2024-03-28T18:59:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.796+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0041eeee0 con 0x7fb004150ae0 2024-03-28T18:59:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.796+0000 7faffb400640 1 -- 172.21.15.27:0/600738281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0037070 con 0x7fb004150ae0 2024-03-28T18:59:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.797+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb004073540 con 0x7fb004150ae0 2024-03-28T18:59:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.798+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb004067dd0 con 0x7fb004150ae0 2024-03-28T18:59:43.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.799+0000 7faffb400640 1 -- 172.21.15.27:0/600738281 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faff0043020 con 0x7fb004150ae0 2024-03-28T18:59:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.800+0000 7faffb400640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe406bbd0 0x7fafe406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:43.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.801+0000 7faffb400640 1 -- 172.21.15.27:0/600738281 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7faff00ac290 con 0x7fb004150ae0 2024-03-28T18:59:43.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.801+0000 7fb003400640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe406bbd0 0x7fafe406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:43.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.802+0000 7fb003400640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe406bbd0 0x7fafe406e090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faff4001e80 tx=0x7faff4001dd0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:43.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:43.806+0000 7faffb400640 1 -- 172.21.15.27:0/600738281 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faff004a030 con 0x7fb004150ae0 2024-03-28T18:59:44.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.062+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb004069640 con 0x7fb004150ae0 2024-03-28T18:59:44.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.063+0000 7faffb400640 1 -- 172.21.15.27:0/600738281 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7faff007a280 con 0x7fb004150ae0 2024-03-28T18:59:44.062 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:44.063 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:44.063 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe406bbd0 msgr2=0x7fafe406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe406bbd0 0x7fafe406e090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faff4001e80 tx=0x7faff4001dd0 comp rx=0 tx=0).stop 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004150ae0 msgr2=0x7fb0040bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004150ae0 0x7fb0040bfd20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7faff002b750 tx=0x7faff00046c0 comp rx=0 tx=0).stop 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 shutdown_connections 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe406bbd0 0x7fafe406e090 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb004153a30 0x7fb0041eac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb004150ae0 0x7fb0040bfd20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 --2- 172.21.15.27:0/600738281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0040c69a0 0x7fb0041e8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 >> 172.21.15.27:0/600738281 conn(0x7fb00406b0c0 msgr2=0x7fb0041563e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:44.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 shutdown_connections 2024-03-28T18:59:44.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:44.066+0000 7fb009000640 1 -- 172.21.15.27:0/600738281 wait complete. 2024-03-28T18:59:44.085 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-03-28T18:59:45.190 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-03-28T18:59:45.191 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-03-28T18:59:45.192 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-03-28T18:59:45.193 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-03-28T18:59:45.194 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-03-28T18:59:45.195 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-03-28T18:59:46.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:46.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.325+0000 7f0be8e00640 1 Processor -- start 2024-03-28T18:59:46.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.325+0000 7f0be8e00640 1 -- start start 2024-03-28T18:59:46.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.327+0000 7f0be8e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be40ae3d0 0x7f0be40c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.327+0000 7f0be8e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0be40aebe0 0x7f0be4153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.327+0000 7f0be8e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be40c5460 con 0x7f0be40ae7b0 2024-03-28T18:59:46.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.327+0000 7f0be8e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be4153630 con 0x7f0be40aebe0 2024-03-28T18:59:46.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.327+0000 7f0be8e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be40c4bc0 con 0x7f0be40ae3d0 2024-03-28T18:59:46.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.327+0000 7f0be3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0be40ae7b0 0x7f0be4156990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32870/0 (socket says 172.21.15.27:32870) 2024-03-28T18:59:46.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be3e00640 1 -- 172.21.15.27:0/84073225 learned_addr learned my addr 172.21.15.27:0/84073225 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:46.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0bdbe00640 1 --2- 172.21.15.27:0/84073225 >> v2:172.21.15.155:3300/0 conn(0x7f0be40aebe0 0x7f0be4153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be3400640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be40ae3d0 0x7f0be40c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707241319 0 0) 0x7f0be40c5460 con 0x7f0be40ae7b0 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc8003520 con 0x7f0be40ae7b0 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3644850128 0 0) 0x7f0bc8003520 con 0x7f0be40ae7b0 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be40ae3d0 msgr2=0x7f0be40c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be40ae3d0 0x7f0be40c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 >> v2:172.21.15.155:3300/0 conn(0x7f0be40aebe0 msgr2=0x7f0be4153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 --2- 172.21.15.27:0/84073225 >> v2:172.21.15.155:3300/0 conn(0x7f0be40aebe0 0x7f0be4153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be40c4ea0 con 0x7f0be40ae7b0 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.328+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f0bdc001a10 con 0x7f0be40ae7b0 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 >> v1:172.21.15.27:6789/0 conn(0x7f0be40ae7b0 legacy=0x7f0be4156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be2a00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be40ae3d0 0x7f0bc80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be2a00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8003f30 0x7f0bc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be2a00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bc8004d40 0x7f0bc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4153630 con 0x7f0bc8004d40 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be40c4bc0 con 0x7f0bc8003f30 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc800b9c0 con 0x7f0be40ae3d0 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be3e00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bc8004d40 0x7f0bc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be3400640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be40ae3d0 0x7f0bc80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0bdbe00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8003f30 0x7f0bc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be3e00640 1 -- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be40ae3d0 msgr2=0x7f0bc80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be3e00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be40ae3d0 0x7f0bc80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be3e00640 1 -- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8003f30 msgr2=0x7f0bc8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.329+0000 7f0be3e00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8003f30 0x7f0bc8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.330+0000 7f0be3e00640 1 -- 172.21.15.27:0/84073225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8003520 con 0x7f0bc8004d40 2024-03-28T18:59:46.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.331+0000 7f0be3e00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bc8004d40 0x7f0bc80051c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00a870 tx=0x7f0bc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c592f0594d86af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.331+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc00a4c0 con 0x7f0bc8004d40 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.331+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc034630 con 0x7f0bc8004d40 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.331+0000 7f0be2a00640 1 -- 172.21.15.27:0/84073225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc034a20 con 0x7f0bc8004d40 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.332+0000 7f0be8e00640 1 -- 172.21.15.27:0/84073225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bc8004d40 msgr2=0x7f0bc80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.332+0000 7f0be8e00640 1 --2- 172.21.15.27:0/84073225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bc8004d40 0x7f0bc80051c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc00a870 tx=0x7f0bc8005830 comp rx=0 tx=0).stop 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.332+0000 7f0be3e00640 1 -- 172.21.15.27:0/84073225 reap_dead start 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.332+0000 7f0be8e00640 1 -- 172.21.15.27:0/84073225 shutdown_connections 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.332+0000 7f0be8e00640 1 -- 172.21.15.27:0/84073225 >> 172.21.15.27:0/84073225 conn(0x7f0be4072d10 msgr2=0x7f0be406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.332+0000 7f0be8e00640 1 -- 172.21.15.27:0/84073225 shutdown_connections 2024-03-28T18:59:46.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.332+0000 7f0be8e00640 1 -- 172.21.15.27:0/84073225 wait complete. 2024-03-28T18:59:46.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.334+0000 7f0be8e00640 1 Processor -- start 2024-03-28T18:59:46.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.334+0000 7f0be8e00640 1 -- start start 2024-03-28T18:59:46.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8004d40 0x7f0be40bdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be8e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be414c5c0 0x7f0be4151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0be4141ba0 0x7f0be4142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be8e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4154890 con 0x7f0be4141ba0 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be8e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4070890 con 0x7f0bc8004d40 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be8e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4154a10 con 0x7f0be414c5c0 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0be4141ba0 0x7f0be4142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0be4141ba0 0x7f0be4142030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53308/0 (socket says 172.21.15.27:53308) 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be3e00640 1 -- 172.21.15.27:0/3668003658 learned_addr learned my addr 172.21.15.27:0/3668003658 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0bdbe00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be414c5c0 0x7f0be4151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.335+0000 7f0be3400640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8004d40 0x7f0be40bdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0be3e00640 1 -- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be414c5c0 msgr2=0x7f0be4151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0be3e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be414c5c0 0x7f0be4151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0be3e00640 1 -- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8004d40 msgr2=0x7f0be40bdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0be3e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8004d40 0x7f0be40bdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0be3e00640 1 -- 172.21.15.27:0/3668003658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be41ed390 con 0x7f0be4141ba0 2024-03-28T18:59:46.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0bdbe00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be414c5c0 0x7f0be4151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:59:46.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0be3e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0be4141ba0 0x7f0be4142030 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc001d10 tx=0x7f0be41535a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0bdb400640 1 -- 172.21.15.27:0/3668003658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc00a4c0 con 0x7f0be4141ba0 2024-03-28T18:59:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0bdb400640 1 -- 172.21.15.27:0/3668003658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc044070 con 0x7f0be4141ba0 2024-03-28T18:59:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0bdb400640 1 -- 172.21.15.27:0/3668003658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc002010 con 0x7f0be4141ba0 2024-03-28T18:59:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be4151f80 con 0x7f0be4141ba0 2024-03-28T18:59:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.336+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be41ed0b0 con 0x7f0be4141ba0 2024-03-28T18:59:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.338+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be414c3d0 con 0x7f0be4141ba0 2024-03-28T18:59:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.339+0000 7f0bdb400640 1 -- 172.21.15.27:0/3668003658 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0bdc0349f0 con 0x7f0be4141ba0 2024-03-28T18:59:46.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.339+0000 7f0bdb400640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bc006bc00 0x7f0bc006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:46.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.340+0000 7f0bdb400640 1 -- 172.21.15.27:0/3668003658 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f0bdc0ad120 con 0x7f0be4141ba0 2024-03-28T18:59:46.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.340+0000 7f0be3400640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bc006bc00 0x7f0bc006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:46.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.341+0000 7f0be3400640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bc006bc00 0x7f0bc006e0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009b00 tx=0x7f0bd4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:46.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.345+0000 7f0bdb400640 1 -- 172.21.15.27:0/3668003658 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0bdc07b760 con 0x7f0be4141ba0 2024-03-28T18:59:46.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.595+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0be4067dd0 con 0x7f0be4141ba0 2024-03-28T18:59:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.596+0000 7f0bdb400640 1 -- 172.21.15.27:0/3668003658 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f0bdc032e30 con 0x7f0be4141ba0 2024-03-28T18:59:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:46.596 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bc006bc00 msgr2=0x7f0bc006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bc006bc00 0x7f0bc006e0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009b00 tx=0x7f0bd4008040 comp rx=0 tx=0).stop 2024-03-28T18:59:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0be4141ba0 msgr2=0x7f0be4142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0be4141ba0 0x7f0be4142030 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc001d10 tx=0x7f0be41535a0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 shutdown_connections 2024-03-28T18:59:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bc006bc00 0x7f0bc006e0c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0be4141ba0 0x7f0be4142030 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0be414c5c0 0x7f0be4151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 --2- 172.21.15.27:0/3668003658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bc8004d40 0x7f0be40bdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:46.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 >> 172.21.15.27:0/3668003658 conn(0x7f0be4072d10 msgr2=0x7f0be41540e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:46.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 shutdown_connections 2024-03-28T18:59:46.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:46.599+0000 7f0be8e00640 1 -- 172.21.15.27:0/3668003658 wait complete. 2024-03-28T18:59:46.619 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:59:47.356 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-03-28T18:59:47.356 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-03-28T18:59:47.356 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-03-28T18:59:47.356 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-03-28T18:59:47.357 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-03-28T18:59:47.358 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-03-28T18:59:47.359 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-03-28T18:59:47.360 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-03-28T18:59:47.361 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-03-28T18:59:47.362 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-03-28T18:59:47.363 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-03-28T18:59:47.364 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-03-28T18:59:48.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:48.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.867+0000 7f2bc2a00640 1 Processor -- start 2024-03-28T18:59:48.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.867+0000 7f2bc2a00640 1 -- start start 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc14f450 0x7f2bbc0c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2bbc06e390 0x7f2bbc158de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bbc0c2220 0x7f2bbc159320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc155860 con 0x7f2bbc0c2220 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc155b40 con 0x7f2bbc06e390 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc155e20 con 0x7f2bbc14f450 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bbc0c2220 0x7f2bbc159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bbc0c2220 0x7f2bbc159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53312/0 (socket says 172.21.15.27:53312) 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc2000640 1 -- 172.21.15.27:0/1883367554 learned_addr learned my addr 172.21.15.27:0/1883367554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc1600640 1 --2- 172.21.15.27:0/1883367554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc14f450 0x7f2bbc0c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.869+0000 7f2bc0c00640 1 --2- 172.21.15.27:0/1883367554 >> v2:172.21.15.155:3300/0 conn(0x7f2bbc06e390 0x7f2bbc158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.870+0000 7f2bc2000640 1 -- 172.21.15.27:0/1883367554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc14f450 msgr2=0x7f2bbc0c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.870+0000 7f2bc2000640 1 --2- 172.21.15.27:0/1883367554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc14f450 0x7f2bbc0c1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.870+0000 7f2bc2000640 1 -- 172.21.15.27:0/1883367554 >> v2:172.21.15.155:3300/0 conn(0x7f2bbc06e390 msgr2=0x7f2bbc158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.870+0000 7f2bc2000640 1 --2- 172.21.15.27:0/1883367554 >> v2:172.21.15.155:3300/0 conn(0x7f2bbc06e390 0x7f2bbc158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.870+0000 7f2bc2000640 1 -- 172.21.15.27:0/1883367554 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc156100 con 0x7f2bbc0c2220 2024-03-28T18:59:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.871+0000 7f2bc2000640 1 --2- 172.21.15.27:0/1883367554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc0c2220 0x7f2bbc159320 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0aa0a0 tx=0x7f2bb402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3417758c0fef5733 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.871+0000 7f2bbbe00640 1 -- 172.21.15.27:0/1883367554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb402c9f0 con 0x7f2bbc0c2220 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.871+0000 7f2bbbe00640 1 -- 172.21.15.27:0/1883367554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb402cb90 con 0x7f2bbc0c2220 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.871+0000 7f2bbbe00640 1 -- 172.21.15.27:0/1883367554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40343f0 con 0x7f2bbc0c2220 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 -- 172.21.15.27:0/1883367554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc0c2220 msgr2=0x7f2bbc159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/1883367554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc0c2220 0x7f2bbc159320 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0aa0a0 tx=0x7f2bb402be50 comp rx=0 tx=0).stop 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 -- 172.21.15.27:0/1883367554 shutdown_connections 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/1883367554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc0c2220 0x7f2bbc159320 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/1883367554 >> v2:172.21.15.155:3300/0 conn(0x7f2bbc06e390 0x7f2bbc158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/1883367554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc14f450 0x7f2bbc0c1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 -- 172.21.15.27:0/1883367554 >> 172.21.15.27:0/1883367554 conn(0x7f2bbc06d980 msgr2=0x7f2bbc0a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 -- 172.21.15.27:0/1883367554 shutdown_connections 2024-03-28T18:59:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.872+0000 7f2bc2a00640 1 -- 172.21.15.27:0/1883367554 wait complete. 2024-03-28T18:59:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.874+0000 7f2bc2a00640 1 Processor -- start 2024-03-28T18:59:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.874+0000 7f2bc2a00640 1 -- start start 2024-03-28T18:59:48.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bbc06e390 0x7f2bbc0c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:48.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc0c2220 0x7f2bbc1eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:48.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc14f450 0x7f2bbc1eba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc14f830 con 0x7f2bbc14f450 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc1563e0 con 0x7f2bbc06e390 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc14f9b0 con 0x7f2bbc0c2220 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc14f450 0x7f2bbc1eba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc14f450 0x7f2bbc1eba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53324/0 (socket says 172.21.15.27:53324) 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.875+0000 7f2bc2000640 1 -- 172.21.15.27:0/252891562 learned_addr learned my addr 172.21.15.27:0/252891562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bc0c00640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc0c2220 0x7f2bbc1eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bc1600640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bbc06e390 0x7f2bbc0c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bc2000640 1 -- 172.21.15.27:0/252891562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc0c2220 msgr2=0x7f2bbc1eb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bc2000640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc0c2220 0x7f2bbc1eb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bc2000640 1 -- 172.21.15.27:0/252891562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bbc06e390 msgr2=0x7f2bbc0c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bc2000640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bbc06e390 0x7f2bbc0c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bc2000640 1 -- 172.21.15.27:0/252891562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc14b190 con 0x7f2bbc14f450 2024-03-28T18:59:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bc2000640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc14f450 0x7f2bbc1eba60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4036040 tx=0x7f2bb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.876+0000 7f2bb9600640 1 -- 172.21.15.27:0/252891562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4043020 con 0x7f2bbc14f450 2024-03-28T18:59:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.877+0000 7f2bb9600640 1 -- 172.21.15.27:0/252891562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb403e450 con 0x7f2bbc14f450 2024-03-28T18:59:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.877+0000 7f2bb9600640 1 -- 172.21.15.27:0/252891562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4009510 con 0x7f2bbc14f450 2024-03-28T18:59:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.877+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc1ef640 con 0x7f2bbc14f450 2024-03-28T18:59:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.877+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc1ef920 con 0x7f2bbc14f450 2024-03-28T18:59:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.878+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bbc067dd0 con 0x7f2bbc14f450 2024-03-28T18:59:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.879+0000 7f2bb9600640 1 -- 172.21.15.27:0/252891562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2bb4046020 con 0x7f2bbc14f450 2024-03-28T18:59:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.880+0000 7f2bb9600640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ba006bbd0 0x7f2ba006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.880+0000 7f2bb9600640 1 -- 172.21.15.27:0/252891562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6895+0+0 (secure 0 0 0) 0x7f2bb40ad760 con 0x7f2bbc14f450 2024-03-28T18:59:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.880+0000 7f2bc1600640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ba006bbd0 0x7f2ba006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.881+0000 7f2bc1600640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ba006bbd0 0x7f2ba006e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc14ed60 tx=0x7f2bac008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:48.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:48.885+0000 7f2bb9600640 1 -- 172.21.15.27:0/252891562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2bb404d030 con 0x7f2bbc14f450 2024-03-28T18:59:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.131+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bbc069640 con 0x7f2bbc14f450 2024-03-28T18:59:49.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.132+0000 7f2bb9600640 1 -- 172.21.15.27:0/252891562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f2bb407bd20 con 0x7f2bbc14f450 2024-03-28T18:59:49.131 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:49.131 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5325,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:58:12.558453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/3329030532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3329030532},{"type":"v1","addr":"172.21.15.27:6837","nonce":3329030532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-03-28T18:59:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.134+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ba006bbd0 msgr2=0x7f2ba006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.134+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ba006bbd0 0x7f2ba006e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc14ed60 tx=0x7f2bac008040 comp rx=0 tx=0).stop 2024-03-28T18:59:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.134+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc14f450 msgr2=0x7f2bbc1eba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.134+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc14f450 0x7f2bbc1eba60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4036040 tx=0x7f2bb4004a40 comp rx=0 tx=0).stop 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.135+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 shutdown_connections 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.135+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ba006bbd0 0x7f2ba006e090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.135+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bbc14f450 0x7f2bbc1eba60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.135+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2bbc0c2220 0x7f2bbc1eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.135+0000 7f2bc2a00640 1 --2- 172.21.15.27:0/252891562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bbc06e390 0x7f2bbc0c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.135+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 >> 172.21.15.27:0/252891562 conn(0x7f2bbc06d980 msgr2=0x7f2bbc061ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.135+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 shutdown_connections 2024-03-28T18:59:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:49.135+0000 7f2bc2a00640 1 -- 172.21.15.27:0/252891562 wait complete. 2024-03-28T18:59:49.155 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T18:59:49.458 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-03-28T18:59:49.458 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-03-28T18:59:49.458 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-03-28T18:59:49.459 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-03-28T18:59:49.460 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-03-28T18:59:49.461 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-03-28T18:59:49.462 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-03-28T18:59:49.463 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-03-28T18:59:49.464 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-03-28T18:59:49.464 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-03-28T18:59:49.464 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-03-28T18:59:51.093 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-03-28T18:59:51.094 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-03-28T18:59:51.095 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-03-28T18:59:51.096 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-03-28T18:59:51.097 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-03-28T18:59:51.098 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-03-28T18:59:51.099 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-03-28T18:59:51.100 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-03-28T18:59:51.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:51.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.384+0000 7f6b8cc00640 1 Processor -- start 2024-03-28T18:59:51.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.384+0000 7f6b8cc00640 1 -- start start 2024-03-28T18:59:51.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.385+0000 7f6b8cc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6b880c8ed0 0x7f6b880c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.385+0000 7f6b8cc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b88158bd0 con 0x7f6b880c58c0 2024-03-28T18:59:51.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.386+0000 7f6b8cc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b88158eb0 con 0x7f6b880be5d0 2024-03-28T18:59:51.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.386+0000 7f6b8cc00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b88159190 con 0x7f6b880c8ed0 2024-03-28T18:59:51.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.386+0000 7f6b87400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6b880c58c0 0x7f6b881555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32886/0 (socket says 172.21.15.27:32886) 2024-03-28T18:59:51.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.386+0000 7f6b87400640 1 -- 172.21.15.27:0/227455595 learned_addr learned my addr 172.21.15.27:0/227455595 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:51.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.386+0000 7f6b86a00640 1 --2- 172.21.15.27:0/227455595 >> v2:172.21.15.155:3300/0 conn(0x7f6b880c8ed0 0x7f6b880c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.386+0000 7f6b86000640 1 -- 172.21.15.27:0/227455595 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568479984 0 0) 0x7f6b88158bd0 con 0x7f6b880c58c0 2024-03-28T18:59:51.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.387+0000 7f6b86000640 1 -- 172.21.15.27:0/227455595 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b6c003520 con 0x7f6b880c58c0 2024-03-28T18:59:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.387+0000 7f6b86a00640 1 -- 172.21.15.27:0/227455595 >> v1:172.21.15.155:6789/0 conn(0x7f6b880be5d0 legacy=0x7f6b88155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.387+0000 7f6b86a00640 1 -- 172.21.15.27:0/227455595 >> v1:172.21.15.27:6789/0 conn(0x7f6b880c58c0 legacy=0x7f6b881555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.387+0000 7f6b86a00640 1 -- 172.21.15.27:0/227455595 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b88159470 con 0x7f6b880c8ed0 2024-03-28T18:59:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.388+0000 7f6b86a00640 1 --2- 172.21.15.27:0/227455595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c8ed0 0x7f6b880c5ca0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6b700097a0 tx=0x7f6b7002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e66ee14b595a3f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.389+0000 7f6b86000640 1 -- 172.21.15.27:0/227455595 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b700315a0 con 0x7f6b880c8ed0 2024-03-28T18:59:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.389+0000 7f6b86000640 1 -- 172.21.15.27:0/227455595 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b70037070 con 0x7f6b880c8ed0 2024-03-28T18:59:51.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.389+0000 7f6b86000640 1 -- 172.21.15.27:0/227455595 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70031960 con 0x7f6b880c8ed0 2024-03-28T18:59:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.389+0000 7f6b8cc00640 1 -- 172.21.15.27:0/227455595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c8ed0 msgr2=0x7f6b880c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.390+0000 7f6b8cc00640 1 --2- 172.21.15.27:0/227455595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c8ed0 0x7f6b880c5ca0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6b700097a0 tx=0x7f6b7002b840 comp rx=0 tx=0).stop 2024-03-28T18:59:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.390+0000 7f6b8cc00640 1 -- 172.21.15.27:0/227455595 shutdown_connections 2024-03-28T18:59:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.390+0000 7f6b8cc00640 1 --2- 172.21.15.27:0/227455595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c8ed0 0x7f6b880c5ca0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.390+0000 7f6b8cc00640 1 -- 172.21.15.27:0/227455595 >> 172.21.15.27:0/227455595 conn(0x7f6b8806b6f0 msgr2=0x7f6b880acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:51.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.390+0000 7f6b8cc00640 1 -- 172.21.15.27:0/227455595 shutdown_connections 2024-03-28T18:59:51.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.390+0000 7f6b8cc00640 1 -- 172.21.15.27:0/227455595 wait complete. 2024-03-28T18:59:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.391+0000 7f6b8cc00640 1 Processor -- start 2024-03-28T18:59:51.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.392+0000 7f6b8cc00640 1 -- start start 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.392+0000 7f6b8cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b880be5d0 0x7f6b880c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b8cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c58c0 0x7f6b880ab060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b8cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b880c8ed0 0x7f6b880ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b8cc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880b09a0 con 0x7f6b880be5d0 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b8cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880b0820 con 0x7f6b880c58c0 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b8cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880ad7d0 con 0x7f6b880c8ed0 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b880be5d0 0x7f6b880c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b880be5d0 0x7f6b880c2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53330/0 (socket says 172.21.15.27:53330) 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87400640 1 -- 172.21.15.27:0/3937152380 learned_addr learned my addr 172.21.15.27:0/3937152380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b86a00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c58c0 0x7f6b880ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87e00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b880c8ed0 0x7f6b880ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87400640 1 -- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b880c8ed0 msgr2=0x7f6b880ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87400640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b880c8ed0 0x7f6b880ab5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87400640 1 -- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c58c0 msgr2=0x7f6b880ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87400640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c58c0 0x7f6b880ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87400640 1 -- 172.21.15.27:0/3937152380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b880ac880 con 0x7f6b880be5d0 2024-03-28T18:59:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.393+0000 7f6b87e00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b880c8ed0 0x7f6b880ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:59:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.394+0000 7f6b87400640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b880be5d0 0x7f6b880c2280 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800c530 tx=0x7f6b7800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:51.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.394+0000 7f6b7f400640 1 -- 172.21.15.27:0/3937152380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78017070 con 0x7f6b880be5d0 2024-03-28T18:59:51.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.394+0000 7f6b7f400640 1 -- 172.21.15.27:0/3937152380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b78005ce0 con 0x7f6b880be5d0 2024-03-28T18:59:51.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.394+0000 7f6b7f400640 1 -- 172.21.15.27:0/3937152380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780072b0 con 0x7f6b880be5d0 2024-03-28T18:59:51.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.394+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b88155400 con 0x7f6b880be5d0 2024-03-28T18:59:51.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.394+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b880ace00 con 0x7f6b880be5d0 2024-03-28T18:59:51.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.395+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b88069640 con 0x7f6b880be5d0 2024-03-28T18:59:51.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.396+0000 7f6b7f400640 1 -- 172.21.15.27:0/3937152380 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6b7801a070 con 0x7f6b880be5d0 2024-03-28T18:59:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.397+0000 7f6b7f400640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6b6406bc00 0x7f6b6406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.397+0000 7f6b7f400640 1 -- 172.21.15.27:0/3937152380 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7f6b7808a330 con 0x7f6b880be5d0 2024-03-28T18:59:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.398+0000 7f6b86a00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6b6406bc00 0x7f6b6406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.398+0000 7f6b86a00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6b6406bc00 0x7f6b6406e0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6b70036000 tx=0x7f6b70008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:51.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.402+0000 7f6b7f400640 1 -- 172.21.15.27:0/3937152380 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6b780589d0 con 0x7f6b880be5d0 2024-03-28T18:59:51.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.653+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b88156580 con 0x7f6b880be5d0 2024-03-28T18:59:51.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.654+0000 7f6b7f400640 1 -- 172.21.15.27:0/3937152380 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5459 (secure 0 0 0) 0x7f6b780582d0 con 0x7f6b880be5d0 2024-03-28T18:59:51.654 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:51.654 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:50.537001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:51.654 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-03-28T18:59:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6b6406bc00 msgr2=0x7f6b6406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6b6406bc00 0x7f6b6406e0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6b70036000 tx=0x7f6b70008030 comp rx=0 tx=0).stop 2024-03-28T18:59:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b880be5d0 msgr2=0x7f6b880c2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b880be5d0 0x7f6b880c2280 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800c530 tx=0x7f6b7800c9f0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 shutdown_connections 2024-03-28T18:59:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6b6406bc00 0x7f6b6406e0c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b880c8ed0 0x7f6b880ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b880c58c0 0x7f6b880ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 --2- 172.21.15.27:0/3937152380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b880be5d0 0x7f6b880c2280 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 >> 172.21.15.27:0/3937152380 conn(0x7f6b8806b6f0 msgr2=0x7f6b880c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.657+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 shutdown_connections 2024-03-28T18:59:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.658+0000 7f6b8cc00640 1 -- 172.21.15.27:0/3937152380 wait complete. 2024-03-28T18:59:51.676 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-03-28T18:59:51.676 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T18:59:51.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.894+0000 7fe393000640 1 Processor -- start 2024-03-28T18:59:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.895+0000 7fe393000640 1 -- start start 2024-03-28T18:59:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.896+0000 7fe393000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe38c0c4510 0x7fe38c0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.896+0000 7fe393000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 0x7fe38c0c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.896+0000 7fe393000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c0c5c40 con 0x7fe38c0701c0 2024-03-28T18:59:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.896+0000 7fe393000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe38c0c3480 con 0x7fe38c0c4510 2024-03-28T18:59:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.896+0000 7fe393000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0c5a00 con 0x7fe38c0c8530 2024-03-28T18:59:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.896+0000 7fe38be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe38c0c4510 0x7fe38c0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.897+0000 7fe38be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe38c0c4510 0x7fe38c0c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53334/0 (socket says 172.21.15.27:53334) 2024-03-28T18:59:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.897+0000 7fe38be00640 1 -- 172.21.15.27:0/1560470069 learned_addr learned my addr 172.21.15.27:0/1560470069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.897+0000 7fe38b400640 1 --2- 172.21.15.27:0/1560470069 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 0x7fe38c0c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.897+0000 7fe38be00640 1 -- 172.21.15.27:0/1560470069 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 msgr2=0x7fe38c0c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.897+0000 7fe38be00640 1 --2- 172.21.15.27:0/1560470069 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 0x7fe38c0c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.897+0000 7fe38be00640 1 -- 172.21.15.27:0/1560470069 >> v1:172.21.15.27:6789/0 conn(0x7fe38c0701c0 legacy=0x7fe38c0c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.897+0000 7fe38be00640 1 -- 172.21.15.27:0/1560470069 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0c5f20 con 0x7fe38c0c4510 2024-03-28T18:59:51.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.898+0000 7fe38be00640 1 --2- 172.21.15.27:0/1560470069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0c4510 0x7fe38c0c70d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009e50 tx=0x7fe37c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8513bcf59543de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe38aa00640 1 -- 172.21.15.27:0/1560470069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c02ca80 con 0x7fe38c0c4510 2024-03-28T18:59:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe38aa00640 1 -- 172.21.15.27:0/1560470069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c02cc20 con 0x7fe38c0c4510 2024-03-28T18:59:51.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe38aa00640 1 -- 172.21.15.27:0/1560470069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c0343f0 con 0x7fe38c0c4510 2024-03-28T18:59:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe393000640 1 -- 172.21.15.27:0/1560470069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0c4510 msgr2=0x7fe38c0c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe393000640 1 --2- 172.21.15.27:0/1560470069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0c4510 0x7fe38c0c70d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009e50 tx=0x7fe37c02bee0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe393000640 1 -- 172.21.15.27:0/1560470069 shutdown_connections 2024-03-28T18:59:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe393000640 1 --2- 172.21.15.27:0/1560470069 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 0x7fe38c0c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe393000640 1 --2- 172.21.15.27:0/1560470069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0c4510 0x7fe38c0c70d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe393000640 1 -- 172.21.15.27:0/1560470069 >> 172.21.15.27:0/1560470069 conn(0x7fe38c066dd0 msgr2=0x7fe38c0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.899+0000 7fe393000640 1 -- 172.21.15.27:0/1560470069 shutdown_connections 2024-03-28T18:59:51.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.900+0000 7fe393000640 1 -- 172.21.15.27:0/1560470069 wait complete. 2024-03-28T18:59:51.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.901+0000 7fe393000640 1 Processor -- start 2024-03-28T18:59:51.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.901+0000 7fe393000640 1 -- start start 2024-03-28T18:59:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.902+0000 7fe393000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0701c0 0x7fe38c0ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.902+0000 7fe393000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe38c0c4510 0x7fe38c1e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.902+0000 7fe393000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 0x7fe38c0bf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.902+0000 7fe393000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c06fca0 con 0x7fe38c0701c0 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.902+0000 7fe393000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c06f9c0 con 0x7fe38c0c4510 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.902+0000 7fe393000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0c48f0 con 0x7fe38c0c8530 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0701c0 0x7fe38c0ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0701c0 0x7fe38c0ad2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53348/0 (socket says 172.21.15.27:53348) 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 -- 172.21.15.27:0/3664460757 learned_addr learned my addr 172.21.15.27:0/3664460757 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38b400640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe38c0c4510 0x7fe38c1e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe391000640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 0x7fe38c0bf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 -- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 msgr2=0x7fe38c0bf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 0x7fe38c0bf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 -- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe38c0c4510 msgr2=0x7fe38c1e72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe38c0c4510 0x7fe38c1e72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 -- 172.21.15.27:0/3664460757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0ad9c0 con 0x7fe38c0701c0 2024-03-28T18:59:51.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38b400640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe38c0c4510 0x7fe38c1e72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:59:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.903+0000 7fe38be00640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0701c0 0x7fe38c0ad2b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe37c000f80 tx=0x7fe37c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.904+0000 7fe383e00640 1 -- 172.21.15.27:0/3664460757 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c043020 con 0x7fe38c0701c0 2024-03-28T18:59:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.904+0000 7fe383e00640 1 -- 172.21.15.27:0/3664460757 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c03e450 con 0x7fe38c0701c0 2024-03-28T18:59:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.904+0000 7fe383e00640 1 -- 172.21.15.27:0/3664460757 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c009510 con 0x7fe38c0701c0 2024-03-28T18:59:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.904+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c1e9f20 con 0x7fe38c0701c0 2024-03-28T18:59:51.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.904+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c0bf5f0 con 0x7fe38c0701c0 2024-03-28T18:59:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.905+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c04f400 con 0x7fe38c0701c0 2024-03-28T18:59:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.906+0000 7fe383e00640 1 -- 172.21.15.27:0/3664460757 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe37c046020 con 0x7fe38c0701c0 2024-03-28T18:59:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.908+0000 7fe383e00640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe36806bc00 0x7fe36806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.908+0000 7fe383e00640 1 -- 172.21.15.27:0/3664460757 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7fe37c0ad7b0 con 0x7fe38c0701c0 2024-03-28T18:59:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.908+0000 7fe38b400640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe36806bc00 0x7fe36806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:51.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.909+0000 7fe38b400640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe36806bc00 0x7fe36806e0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe374006fd0 tx=0x7fe374008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:51.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:51.912+0000 7fe383e00640 1 -- 172.21.15.27:0/3664460757 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe37c04d030 con 0x7fe38c0701c0 2024-03-28T18:59:52.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.172+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe38c0aa410 con 0x7fe38c0701c0 2024-03-28T18:59:52.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.173+0000 7fe383e00640 1 -- 172.21.15.27:0/3664460757 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5457 (secure 0 0 0) 0x7fe37c07beb0 con 0x7fe38c0701c0 2024-03-28T18:59:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:52.173 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:51.117057+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:52.173 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-03-28T18:59:52.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe36806bc00 msgr2=0x7fe36806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:52.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe36806bc00 0x7fe36806e0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe374006fd0 tx=0x7fe374008040 comp rx=0 tx=0).stop 2024-03-28T18:59:52.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0701c0 msgr2=0x7fe38c0ad2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0701c0 0x7fe38c0ad2b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe37c000f80 tx=0x7fe37c004a40 comp rx=0 tx=0).stop 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 shutdown_connections 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe36806bc00 0x7fe36806e0c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe38c0c8530 0x7fe38c0bf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe38c0c4510 0x7fe38c1e72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 --2- 172.21.15.27:0/3664460757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe38c0701c0 0x7fe38c0ad2b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 >> 172.21.15.27:0/3664460757 conn(0x7fe38c066dd0 msgr2=0x7fe38c1d5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 shutdown_connections 2024-03-28T18:59:52.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:52.176+0000 7fe393000640 1 -- 172.21.15.27:0/3664460757 wait complete. 2024-03-28T18:59:52.198 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5325": { "addr": "172.21.15.27:6836/181571966", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 181571966, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 181571966, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5325, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T18:59:51.117057+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5325 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4775": { "addr": "172.21.15.155:6834/3891854856", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3891854856, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3891854856, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4775, "incarnation": 26, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T18:59:08.605442+0000", "qdb_cluster": [ 4775 ], "qdb_leader": 4775, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4775 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T18:59:53.054 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-03-28T18:59:53.054 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-03-28T18:59:53.054 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-03-28T18:59:53.054 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-03-28T18:59:53.054 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-03-28T18:59:53.054 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-03-28T18:59:53.055 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-03-28T18:59:53.056 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-03-28T18:59:53.057 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-03-28T18:59:53.058 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-03-28T18:59:53.059 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-03-28T18:59:54.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.445+0000 7fb44ee00640 1 Processor -- start 2024-03-28T18:59:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.445+0000 7fb44ee00640 1 -- start start 2024-03-28T18:59:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44ee00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb4480c58c0 0x7fb4480c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44ee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4480c8ed0 0x7fb4481555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44ee00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb448156230 con 0x7fb4480be5d0 2024-03-28T18:59:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44ee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb448158bd0 con 0x7fb4480c8ed0 2024-03-28T18:59:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44ee00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb448158eb0 con 0x7fb4480c58c0 2024-03-28T18:59:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4480c8ed0 0x7fb4481555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4480c8ed0 0x7fb4481555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53356/0 (socket says 172.21.15.27:53356) 2024-03-28T18:59:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44d000640 1 -- 172.21.15.27:0/3440314718 learned_addr learned my addr 172.21.15.27:0/3440314718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.447+0000 7fb44da00640 1 --2- 172.21.15.27:0/3440314718 >> v2:172.21.15.155:3300/0 conn(0x7fb4480c58c0 0x7fb4480c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.448+0000 7fb44d000640 1 -- 172.21.15.27:0/3440314718 >> v2:172.21.15.155:3300/0 conn(0x7fb4480c58c0 msgr2=0x7fb4480c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.448+0000 7fb44d000640 1 --2- 172.21.15.27:0/3440314718 >> v2:172.21.15.155:3300/0 conn(0x7fb4480c58c0 0x7fb4480c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.448+0000 7fb44d000640 1 -- 172.21.15.27:0/3440314718 >> v1:172.21.15.155:6789/0 conn(0x7fb4480be5d0 legacy=0x7fb448155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.448+0000 7fb44d000640 1 -- 172.21.15.27:0/3440314718 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb448159190 con 0x7fb4480c8ed0 2024-03-28T18:59:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.449+0000 7fb44d000640 1 --2- 172.21.15.27:0/3440314718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c8ed0 0x7fb4481555e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb44806c060 tx=0x7fb43402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d085585e02c70b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.449+0000 7fb447e00640 1 -- 172.21.15.27:0/3440314718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43402e070 con 0x7fb4480c8ed0 2024-03-28T18:59:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.450+0000 7fb447e00640 1 -- 172.21.15.27:0/3440314718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb434037070 con 0x7fb4480c8ed0 2024-03-28T18:59:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.450+0000 7fb447e00640 1 -- 172.21.15.27:0/3440314718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb434031a00 con 0x7fb4480c8ed0 2024-03-28T18:59:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.450+0000 7fb44ee00640 1 -- 172.21.15.27:0/3440314718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c8ed0 msgr2=0x7fb4481555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.450+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3440314718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c8ed0 0x7fb4481555e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb44806c060 tx=0x7fb43402b880 comp rx=0 tx=0).stop 2024-03-28T18:59:54.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.450+0000 7fb44ee00640 1 -- 172.21.15.27:0/3440314718 shutdown_connections 2024-03-28T18:59:54.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.450+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3440314718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c8ed0 0x7fb4481555e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.450+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3440314718 >> v2:172.21.15.155:3300/0 conn(0x7fb4480c58c0 0x7fb4480c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.450+0000 7fb44ee00640 1 -- 172.21.15.27:0/3440314718 >> 172.21.15.27:0/3440314718 conn(0x7fb44806b6f0 msgr2=0x7fb4480acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:54.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.451+0000 7fb44ee00640 1 -- 172.21.15.27:0/3440314718 shutdown_connections 2024-03-28T18:59:54.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.451+0000 7fb44ee00640 1 -- 172.21.15.27:0/3440314718 wait complete. 2024-03-28T18:59:54.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.453+0000 7fb44ee00640 1 Processor -- start 2024-03-28T18:59:54.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.454+0000 7fb44ee00640 1 -- start start 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44ee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480be5d0 0x7fb4480c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44ee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c58c0 0x7fb4480c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44ee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480c8ed0 0x7fb4480ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44ee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b0820 con 0x7fb4480c58c0 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44ee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb448159750 con 0x7fb4480c8ed0 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44ee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480b09a0 con 0x7fb4480be5d0 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c58c0 0x7fb4480c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c58c0 0x7fb4480c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53372/0 (socket says 172.21.15.27:53372) 2024-03-28T18:59:54.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.455+0000 7fb44d000640 1 -- 172.21.15.27:0/3607533969 learned_addr learned my addr 172.21.15.27:0/3607533969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:54.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.456+0000 7fb44da00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480be5d0 0x7fb4480c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:54.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.456+0000 7fb44e400640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480c8ed0 0x7fb4480ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.456+0000 7fb44d000640 1 -- 172.21.15.27:0/3607533969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480be5d0 msgr2=0x7fb4480c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.456+0000 7fb44d000640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480be5d0 0x7fb4480c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.456+0000 7fb44d000640 1 -- 172.21.15.27:0/3607533969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480c8ed0 msgr2=0x7fb4480ac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.456+0000 7fb44d000640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480c8ed0 0x7fb4480ac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.456+0000 7fb44d000640 1 -- 172.21.15.27:0/3607533969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480c3860 con 0x7fb4480c58c0 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.457+0000 7fb44d000640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c58c0 0x7fb4480c44c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb434000c00 tx=0x7fb4340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.457+0000 7fb445600640 1 -- 172.21.15.27:0/3607533969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb434032670 con 0x7fb4480c58c0 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.457+0000 7fb445600640 1 -- 172.21.15.27:0/3607533969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb434032850 con 0x7fb4480c58c0 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.457+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4480c2820 con 0x7fb4480c58c0 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.457+0000 7fb445600640 1 -- 172.21.15.27:0/3607533969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb434032c40 con 0x7fb4480c58c0 2024-03-28T18:59:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.457+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4480c2540 con 0x7fb4480c58c0 2024-03-28T18:59:54.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.459+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4480c1ff0 con 0x7fb4480c58c0 2024-03-28T18:59:54.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.460+0000 7fb445600640 1 -- 172.21.15.27:0/3607533969 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb43404c060 con 0x7fb4480c58c0 2024-03-28T18:59:54.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.461+0000 7fb445600640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb42806bc00 0x7fb42806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:54.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.461+0000 7fb445600640 1 -- 172.21.15.27:0/3607533969 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7fb4340ac790 con 0x7fb4480c58c0 2024-03-28T18:59:54.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.462+0000 7fb44da00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb42806bc00 0x7fb42806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:54.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.463+0000 7fb44da00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb42806bc00 0x7fb42806e0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb448152ba0 tx=0x7fb43c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:54.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.466+0000 7fb445600640 1 -- 172.21.15.27:0/3607533969 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb4480c2820 con 0x7fb4480c58c0 2024-03-28T18:59:54.503 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-03-28T18:59:54.504 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-03-28T18:59:54.505 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-03-28T18:59:54.506 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-03-28T18:59:54.507 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-03-28T18:59:54.508 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-03-28T18:59:54.509 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-03-28T18:59:54.663 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T18:59:54.664+0000 7f047ba00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:59:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.793+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb448069640 con 0x7fb4480c58c0 2024-03-28T18:59:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.794+0000 7fb445600640 1 -- 172.21.15.27:0/3607533969 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5464 (secure 0 0 0) 0x7fb434040070 con 0x7fb4480c58c0 2024-03-28T18:59:54.793 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:54.793 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-03-28T18:59:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.796+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb42806bc00 msgr2=0x7fb42806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.796+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb42806bc00 0x7fb42806e0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb448152ba0 tx=0x7fb43c009190 comp rx=0 tx=0).stop 2024-03-28T18:59:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.796+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c58c0 msgr2=0x7fb4480c44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.796+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c58c0 0x7fb4480c44c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb434000c00 tx=0x7fb4340046c0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.797+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 shutdown_connections 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.797+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb42806bc00 0x7fb42806e0c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.797+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480c8ed0 0x7fb4480ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.797+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4480c58c0 0x7fb4480c44c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.797+0000 7fb44ee00640 1 --2- 172.21.15.27:0/3607533969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480be5d0 0x7fb4480c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.797+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 >> 172.21.15.27:0/3607533969 conn(0x7fb44806b6f0 msgr2=0x7fb4480ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.797+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 shutdown_connections 2024-03-28T18:59:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:54.797+0000 7fb44ee00640 1 -- 172.21.15.27:0/3607533969 wait complete. 2024-03-28T18:59:54.816 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5166 2024-03-28T18:59:54.816 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.c 2024-03-28T18:59:54.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:55.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.096+0000 7f1985400640 1 Processor -- start 2024-03-28T18:59:55.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.097+0000 7f1985400640 1 -- start start 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.098+0000 7f1985400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1980156390 0x7f19800c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.098+0000 7f1985400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19800cb1a0 0x7f19800cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.098+0000 7f1985400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800c6250 con 0x7f1980152ba0 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.098+0000 7f1985400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1980155f00 con 0x7f19800cb1a0 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.098+0000 7f1985400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19800c5f70 con 0x7f1980156390 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.098+0000 7f197e800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1980152ba0 0x7f1980158c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32900/0 (socket says 172.21.15.27:32900) 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.098+0000 7f197e800640 1 -- 172.21.15.27:0/1240734318 learned_addr learned my addr 172.21.15.27:0/1240734318 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.098+0000 7f197de00640 1 --2- 172.21.15.27:0/1240734318 >> v2:172.21.15.155:3300/0 conn(0x7f1980156390 0x7f19800c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.099+0000 7f197f200640 1 --2- 172.21.15.27:0/1240734318 >> v2:172.21.15.27:3300/0 conn(0x7f19800cb1a0 0x7f19800cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.099+0000 7f197de00640 1 -- 172.21.15.27:0/1240734318 >> v2:172.21.15.27:3300/0 conn(0x7f19800cb1a0 msgr2=0x7f19800cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.099+0000 7f197de00640 1 --2- 172.21.15.27:0/1240734318 >> v2:172.21.15.27:3300/0 conn(0x7f19800cb1a0 0x7f19800cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.099+0000 7f197de00640 1 -- 172.21.15.27:0/1240734318 >> v1:172.21.15.27:6789/0 conn(0x7f1980152ba0 legacy=0x7f1980158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.099+0000 7f197de00640 1 -- 172.21.15.27:0/1240734318 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19801558b0 con 0x7f1980156390 2024-03-28T18:59:55.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.099+0000 7f197f200640 1 --2- 172.21.15.27:0/1240734318 >> v2:172.21.15.27:3300/0 conn(0x7f19800cb1a0 0x7f19800cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:59:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.101+0000 7f197de00640 1 --2- 172.21.15.27:0/1240734318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 0x7f19800c5840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f19800b1030 tx=0x7f196c02b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=492131f1a02855eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.101+0000 7f197d400640 1 -- 172.21.15.27:0/1240734318 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c038070 con 0x7f1980156390 2024-03-28T18:59:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.101+0000 7f197d400640 1 -- 172.21.15.27:0/1240734318 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196c031710 con 0x7f1980156390 2024-03-28T18:59:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.101+0000 7f197d400640 1 -- 172.21.15.27:0/1240734318 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c031ae0 con 0x7f1980156390 2024-03-28T18:59:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.102+0000 7f1985400640 1 -- 172.21.15.27:0/1240734318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 msgr2=0x7f19800c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.102+0000 7f1985400640 1 --2- 172.21.15.27:0/1240734318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 0x7f19800c5840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f19800b1030 tx=0x7f196c02b7b0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.102+0000 7f1985400640 1 -- 172.21.15.27:0/1240734318 shutdown_connections 2024-03-28T18:59:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.102+0000 7f1985400640 1 --2- 172.21.15.27:0/1240734318 >> v2:172.21.15.27:3300/0 conn(0x7f19800cb1a0 0x7f19800cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.102+0000 7f1985400640 1 --2- 172.21.15.27:0/1240734318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 0x7f19800c5840 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.102+0000 7f1985400640 1 -- 172.21.15.27:0/1240734318 >> 172.21.15.27:0/1240734318 conn(0x7f1980066fb0 msgr2=0x7f19800673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.102+0000 7f1985400640 1 -- 172.21.15.27:0/1240734318 shutdown_connections 2024-03-28T18:59:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.102+0000 7f1985400640 1 -- 172.21.15.27:0/1240734318 wait complete. 2024-03-28T18:59:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.104+0000 7f1985400640 1 Processor -- start 2024-03-28T18:59:55.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.105+0000 7f1985400640 1 -- start start 2024-03-28T18:59:55.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f1985400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19800cb1a0 0x7f19801547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f1985400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1980152ba0 0x7f1980149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f1985400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 0x7f19801573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f1985400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19801593a0 con 0x7f19800cb1a0 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f1985400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198006d700 con 0x7f1980156390 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f1985400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19800c9780 con 0x7f1980152ba0 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f197e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19800cb1a0 0x7f19801547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f197e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19800cb1a0 0x7f19801547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53392/0 (socket says 172.21.15.27:53392) 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f197e800640 1 -- 172.21.15.27:0/1143213573 learned_addr learned my addr 172.21.15.27:0/1143213573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f197f200640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 0x7f19801573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.106+0000 7f197de00640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1980152ba0 0x7f1980149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.107+0000 7f197e800640 1 -- 172.21.15.27:0/1143213573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1980152ba0 msgr2=0x7f1980149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.107+0000 7f197e800640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1980152ba0 0x7f1980149500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.107+0000 7f197e800640 1 -- 172.21.15.27:0/1143213573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 msgr2=0x7f19801573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.107+0000 7f197e800640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 0x7f19801573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.107+0000 7f197e800640 1 -- 172.21.15.27:0/1143213573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1980152010 con 0x7f19800cb1a0 2024-03-28T18:59:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.107+0000 7f197e800640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19800cb1a0 0x7f19801547d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f197000c400 tx=0x7f197000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.108+0000 7f1976a00640 1 -- 172.21.15.27:0/1143213573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970017070 con 0x7f19800cb1a0 2024-03-28T18:59:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.108+0000 7f1976a00640 1 -- 172.21.15.27:0/1143213573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1970005ce0 con 0x7f19800cb1a0 2024-03-28T18:59:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.108+0000 7f1976a00640 1 -- 172.21.15.27:0/1143213573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19700072f0 con 0x7f19800cb1a0 2024-03-28T18:59:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.108+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1980150650 con 0x7f19800cb1a0 2024-03-28T18:59:55.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.108+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1980150370 con 0x7f19800cb1a0 2024-03-28T18:59:55.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.110+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198004f400 con 0x7f19800cb1a0 2024-03-28T18:59:55.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.111+0000 7f1976a00640 1 -- 172.21.15.27:0/1143213573 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f197001a070 con 0x7f19800cb1a0 2024-03-28T18:59:55.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.112+0000 7f1976a00640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f196006bbd0 0x7f196006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.112+0000 7f1976a00640 1 -- 172.21.15.27:0/1143213573 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7f197008a210 con 0x7f19800cb1a0 2024-03-28T18:59:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.112+0000 7f197de00640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f196006bbd0 0x7f196006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.113+0000 7f197de00640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f196006bbd0 0x7f196006e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f196c035000 tx=0x7f196c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.119+0000 7f1976a00640 1 -- 172.21.15.27:0/1143213573 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1980152010 con 0x7f19800cb1a0 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-03-28T18:59:55.233 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-03-28T18:59:55.234 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-03-28T18:59:55.235 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-03-28T18:59:55.236 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-03-28T18:59:55.237 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-03-28T18:59:55.238 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-03-28T18:59:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.480+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f198006d4e0 con 0x7f19800cb1a0 2024-03-28T18:59:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.481+0000 7f1976a00640 1 -- 172.21.15.27:0/1143213573 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5464 (secure 0 0 0) 0x7f19800c6250 con 0x7f19800cb1a0 2024-03-28T18:59:55.480 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:55.480 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f196006bbd0 msgr2=0x7f196006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f196006bbd0 0x7f196006e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f196c035000 tx=0x7f196c008030 comp rx=0 tx=0).stop 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19800cb1a0 msgr2=0x7f19801547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19800cb1a0 0x7f19801547d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f197000c400 tx=0x7f197000c800 comp rx=0 tx=0).stop 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 shutdown_connections 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f196006bbd0 0x7f196006e090 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1980156390 0x7f19801573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1980152ba0 0x7f1980149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 --2- 172.21.15.27:0/1143213573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19800cb1a0 0x7f19801547d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 >> 172.21.15.27:0/1143213573 conn(0x7f1980066fb0 msgr2=0x7f19800c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 shutdown_connections 2024-03-28T18:59:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.484+0000 7f1985400640 1 -- 172.21.15.27:0/1143213573 wait complete. 2024-03-28T18:59:55.506 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-03-28T18:59:55.507 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T18:59:55.507 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T18:59:55.531 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T18:59:55.532 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:55.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:59:55.643 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T18:59:55.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.762+0000 7fe227e00640 1 Processor -- start 2024-03-28T18:59:55.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.762+0000 7fe227e00640 1 -- start start 2024-03-28T18:59:55.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.763+0000 7fe227e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c8ed0 0x7fe2280c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.763+0000 7fe227e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe228158bd0 con 0x7fe2280be5d0 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.763+0000 7fe227e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe228158eb0 con 0x7fe2280c58c0 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.763+0000 7fe227e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe228159190 con 0x7fe2280c8ed0 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.764+0000 7fe227400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe2280be5d0 0x7fe228155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:32904/0 (socket says 172.21.15.27:32904) 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.764+0000 7fe227400640 1 -- 172.21.15.27:0/694068481 learned_addr learned my addr 172.21.15.27:0/694068481 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.764+0000 7fe226000640 1 --2- 172.21.15.27:0/694068481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c8ed0 0x7fe2280c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.764+0000 7fe225600640 1 -- 172.21.15.27:0/694068481 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181320789 0 0) 0x7fe228158bd0 con 0x7fe2280be5d0 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.765+0000 7fe225600640 1 -- 172.21.15.27:0/694068481 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c003520 con 0x7fe2280be5d0 2024-03-28T18:59:55.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.765+0000 7fe226000640 1 -- 172.21.15.27:0/694068481 >> v1:172.21.15.155:6789/0 conn(0x7fe2280c58c0 legacy=0x7fe2281555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.765+0000 7fe226000640 1 -- 172.21.15.27:0/694068481 >> v1:172.21.15.27:6789/0 conn(0x7fe2280be5d0 legacy=0x7fe228155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.765+0000 7fe226000640 1 -- 172.21.15.27:0/694068481 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe228159470 con 0x7fe2280c8ed0 2024-03-28T18:59:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.767+0000 7fe226000640 1 --2- 172.21.15.27:0/694068481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c8ed0 0x7fe2280c5ca0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe214009750 tx=0x7fe21402b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=667671bbba71840b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.767+0000 7fe225600640 1 -- 172.21.15.27:0/694068481 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214037070 con 0x7fe2280c8ed0 2024-03-28T18:59:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.767+0000 7fe225600640 1 -- 172.21.15.27:0/694068481 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214030710 con 0x7fe2280c8ed0 2024-03-28T18:59:55.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.767+0000 7fe225600640 1 -- 172.21.15.27:0/694068481 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214030ae0 con 0x7fe2280c8ed0 2024-03-28T18:59:55.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.768+0000 7fe227e00640 1 -- 172.21.15.27:0/694068481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c8ed0 msgr2=0x7fe2280c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.768+0000 7fe227e00640 1 --2- 172.21.15.27:0/694068481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c8ed0 0x7fe2280c5ca0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe214009750 tx=0x7fe21402b680 comp rx=0 tx=0).stop 2024-03-28T18:59:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.768+0000 7fe227e00640 1 -- 172.21.15.27:0/694068481 shutdown_connections 2024-03-28T18:59:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.768+0000 7fe227e00640 1 --2- 172.21.15.27:0/694068481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c8ed0 0x7fe2280c5ca0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.768+0000 7fe227e00640 1 -- 172.21.15.27:0/694068481 >> 172.21.15.27:0/694068481 conn(0x7fe22806b6f0 msgr2=0x7fe2280acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.768+0000 7fe227e00640 1 -- 172.21.15.27:0/694068481 shutdown_connections 2024-03-28T18:59:55.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.769+0000 7fe227e00640 1 -- 172.21.15.27:0/694068481 wait complete. 2024-03-28T18:59:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.770+0000 7fe227e00640 1 Processor -- start 2024-03-28T18:59:55.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.771+0000 7fe227e00640 1 -- start start 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe227e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2280be5d0 0x7fe228152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe227e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c58c0 0x7fe2281556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe227e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2280c8ed0 0x7fe2281ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe227e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2280b09a0 con 0x7fe2280c8ed0 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe227e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2280b0820 con 0x7fe2280be5d0 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe227e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2280ad7d0 con 0x7fe2280c58c0 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe226a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2280be5d0 0x7fe228152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe227400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2280c8ed0 0x7fe2281ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe226a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2280be5d0 0x7fe228152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:60498/0 (socket says 172.21.15.27:60498) 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe226a00640 1 -- 172.21.15.27:0/834411462 learned_addr learned my addr 172.21.15.27:0/834411462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:55.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.772+0000 7fe226000640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c58c0 0x7fe2281556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.773+0000 7fe227400640 1 -- 172.21.15.27:0/834411462 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c58c0 msgr2=0x7fe2281556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.773+0000 7fe227400640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c58c0 0x7fe2281556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.773+0000 7fe227400640 1 -- 172.21.15.27:0/834411462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2280be5d0 msgr2=0x7fe228152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.773+0000 7fe227400640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2280be5d0 0x7fe228152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.773+0000 7fe227400640 1 -- 172.21.15.27:0/834411462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2280c27e0 con 0x7fe2280c8ed0 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.773+0000 7fe227400640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2280c8ed0 0x7fe2281ebc60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe22000caa0 tx=0x7fe22000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.774+0000 7fe21f400640 1 -- 172.21.15.27:0/834411462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220017070 con 0x7fe2280c8ed0 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.774+0000 7fe21f400640 1 -- 172.21.15.27:0/834411462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22000ab40 con 0x7fe2280c8ed0 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.774+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2280c4a70 con 0x7fe2280c8ed0 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.774+0000 7fe21f400640 1 -- 172.21.15.27:0/834411462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220012400 con 0x7fe2280c8ed0 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.774+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2280c2220 con 0x7fe2280c8ed0 2024-03-28T18:59:55.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.775+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe228069640 con 0x7fe2280c8ed0 2024-03-28T18:59:55.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.777+0000 7fe21f400640 1 -- 172.21.15.27:0/834411462 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe22001a070 con 0x7fe2280c8ed0 2024-03-28T18:59:55.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.778+0000 7fe21f400640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe20806bc00 0x7fe20806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:55.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.779+0000 7fe21f400640 1 -- 172.21.15.27:0/834411462 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7fe22008ab00 con 0x7fe2280c8ed0 2024-03-28T18:59:55.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.779+0000 7fe226a00640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe20806bc00 0x7fe20806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:55.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.780+0000 7fe226a00640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe20806bc00 0x7fe20806e0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe218000f60 tx=0x7fe218009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:55.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:55.784+0000 7fe21f400640 1 -- 172.21.15.27:0/834411462 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe220058a90 con 0x7fe2280c8ed0 2024-03-28T18:59:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.113+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2280ad5b0 con 0x7fe2280c8ed0 2024-03-28T18:59:56.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.114+0000 7fe21f400640 1 -- 172.21.15.27:0/834411462 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5464 (secure 0 0 0) 0x7fe22005cb30 con 0x7fe2280c8ed0 2024-03-28T18:59:56.114 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:56.114 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:56.114 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-03-28T18:59:56.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.117+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe20806bc00 msgr2=0x7fe20806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:56.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.117+0000 7fe227e00640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe20806bc00 0x7fe20806e0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe218000f60 tx=0x7fe218009040 comp rx=0 tx=0).stop 2024-03-28T18:59:56.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.117+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2280c8ed0 msgr2=0x7fe2281ebc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:56.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.117+0000 7fe227e00640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2280c8ed0 0x7fe2281ebc60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe22000caa0 tx=0x7fe22000cea0 comp rx=0 tx=0).stop 2024-03-28T18:59:56.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.118+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 shutdown_connections 2024-03-28T18:59:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.118+0000 7fe227e00640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe20806bc00 0x7fe20806e0c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.118+0000 7fe227e00640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2280c8ed0 0x7fe2281ebc60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.118+0000 7fe227e00640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2280c58c0 0x7fe2281556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.118+0000 7fe227e00640 1 --2- 172.21.15.27:0/834411462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2280be5d0 0x7fe228152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.118+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 >> 172.21.15.27:0/834411462 conn(0x7fe22806b6f0 msgr2=0x7fe2281e2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.118+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 shutdown_connections 2024-03-28T18:59:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:56.118+0000 7fe227e00640 1 -- 172.21.15.27:0/834411462 wait complete. 2024-03-28T18:59:56.146 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T18:59:56.848 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-03-28T18:59:56.848 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-03-28T18:59:56.848 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-03-28T18:59:56.848 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-03-28T18:59:56.848 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-03-28T18:59:56.849 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-03-28T18:59:56.850 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-03-28T18:59:56.851 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-03-28T18:59:56.852 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-03-28T18:59:56.853 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-03-28T18:59:56.854 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-03-28T18:59:58.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:59:58.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.407+0000 7f138a200640 1 Processor -- start 2024-03-28T18:59:58.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.408+0000 7f138a200640 1 -- start start 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f138a200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13840c6a90 0x7f138406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f138a200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6e70 0x7f13840c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f138a200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138406d080 con 0x7f1384156ba0 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f138a200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13840c7830 con 0x7f13840c6a90 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f138a200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1384159980 con 0x7f13840c6e70 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f1388e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13840c6a90 0x7f138406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f1388e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13840c6a90 0x7f138406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54846/0 (socket says 172.21.15.27:54846) 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f1388e00640 1 -- 172.21.15.27:0/479265965 learned_addr learned my addr 172.21.15.27:0/479265965 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.409+0000 7f1389800640 1 --1- 172.21.15.27:0/479265965 >> v1:172.21.15.27:6789/0 conn(0x7f1384156ba0 0x7f1384157000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52940/0 (socket says 172.21.15.27:52940) 2024-03-28T18:59:58.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.410+0000 7f1388e00640 1 -- 172.21.15.27:0/479265965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6e70 msgr2=0x7f13840c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:59:58.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.410+0000 7f1383e00640 1 --2- 172.21.15.27:0/479265965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6e70 0x7f13840c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:58.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.410+0000 7f1388e00640 1 --2- 172.21.15.27:0/479265965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6e70 0x7f13840c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.410+0000 7f1388e00640 1 -- 172.21.15.27:0/479265965 >> v1:172.21.15.27:6789/0 conn(0x7f1384156ba0 legacy=0x7f1384157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:58.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.410+0000 7f1388e00640 1 -- 172.21.15.27:0/479265965 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138406cb00 con 0x7f13840c6a90 2024-03-28T18:59:58.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.410+0000 7f1383e00640 1 --2- 172.21.15.27:0/479265965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6e70 0x7f13840c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:59:58.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.412+0000 7f1388e00640 1 --2- 172.21.15.27:0/479265965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13840c6a90 0x7f138406b0c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1378009870 tx=0x7f137802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b058fde285c673 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.412+0000 7f1383400640 1 -- 172.21.15.27:0/479265965 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378034070 con 0x7f13840c6a90 2024-03-28T18:59:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.412+0000 7f1383400640 1 -- 172.21.15.27:0/479265965 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1378030840 con 0x7f13840c6a90 2024-03-28T18:59:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.412+0000 7f1383400640 1 -- 172.21.15.27:0/479265965 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378030c10 con 0x7f13840c6a90 2024-03-28T18:59:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.413+0000 7f138a200640 1 -- 172.21.15.27:0/479265965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13840c6a90 msgr2=0x7f138406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.413+0000 7f138a200640 1 --2- 172.21.15.27:0/479265965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13840c6a90 0x7f138406b0c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1378009870 tx=0x7f137802b980 comp rx=0 tx=0).stop 2024-03-28T18:59:58.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.413+0000 7f138a200640 1 -- 172.21.15.27:0/479265965 shutdown_connections 2024-03-28T18:59:58.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.413+0000 7f138a200640 1 --2- 172.21.15.27:0/479265965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6e70 0x7f13840c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.413+0000 7f138a200640 1 --2- 172.21.15.27:0/479265965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13840c6a90 0x7f138406b0c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.413+0000 7f138a200640 1 -- 172.21.15.27:0/479265965 >> 172.21.15.27:0/479265965 conn(0x7f13840aa7c0 msgr2=0x7f13840aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:58.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.413+0000 7f138a200640 1 -- 172.21.15.27:0/479265965 shutdown_connections 2024-03-28T18:59:58.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.413+0000 7f138a200640 1 -- 172.21.15.27:0/479265965 wait complete. 2024-03-28T18:59:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.415+0000 7f138a200640 1 Processor -- start 2024-03-28T18:59:58.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.415+0000 7f138a200640 1 -- start start 2024-03-28T18:59:58.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.416+0000 7f138a200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6a90 0x7f13841eef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:58.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f138a200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13840c6e70 0x7f13840c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:58.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f138a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384156ba0 0x7f138414b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f138a200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13840c8630 con 0x7f1384156ba0 2024-03-28T18:59:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f138a200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1384159610 con 0x7f13840c6e70 2024-03-28T18:59:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f138a200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1384152e90 con 0x7f13840c6a90 2024-03-28T18:59:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f1389800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384156ba0 0x7f138414b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f1389800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384156ba0 0x7f138414b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54852/0 (socket says 172.21.15.27:54852) 2024-03-28T18:59:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f1389800640 1 -- 172.21.15.27:0/2888119194 learned_addr learned my addr 172.21.15.27:0/2888119194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T18:59:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f1388e00640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6a90 0x7f13841eef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:58.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.417+0000 7f1383e00640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13840c6e70 0x7f13840c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:58.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.418+0000 7f1389800640 1 -- 172.21.15.27:0/2888119194 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6a90 msgr2=0x7f13841eef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.418+0000 7f1389800640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6a90 0x7f13841eef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.418+0000 7f1389800640 1 -- 172.21.15.27:0/2888119194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13840c6e70 msgr2=0x7f13840c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.418+0000 7f1389800640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13840c6e70 0x7f13840c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.418+0000 7f1389800640 1 -- 172.21.15.27:0/2888119194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384158f00 con 0x7f1384156ba0 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.418+0000 7f1389800640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384156ba0 0x7f138414b540 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f137c00c760 tx=0x7f137c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.418+0000 7f1380c00640 1 -- 172.21.15.27:0/2888119194 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c017070 con 0x7f1384156ba0 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.419+0000 7f1380c00640 1 -- 172.21.15.27:0/2888119194 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c00a640 con 0x7f1384156ba0 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.419+0000 7f1380c00640 1 -- 172.21.15.27:0/2888119194 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c00a9f0 con 0x7f1384156ba0 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.419+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13841591e0 con 0x7f1384156ba0 2024-03-28T18:59:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.419+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13841ec6d0 con 0x7f1384156ba0 2024-03-28T18:59:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.420+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13841ec8d0 con 0x7f1384156ba0 2024-03-28T18:59:58.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.423+0000 7f1380c00640 1 -- 172.21.15.27:0/2888119194 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f137c01a070 con 0x7f1384156ba0 2024-03-28T18:59:58.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.424+0000 7f1380c00640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bbd0 0x7f136406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:59:58.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.424+0000 7f1380c00640 1 -- 172.21.15.27:0/2888119194 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7f137c08ad00 con 0x7f1384156ba0 2024-03-28T18:59:58.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.424+0000 7f1388e00640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bbd0 0x7f136406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:59:58.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.425+0000 7f1388e00640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bbd0 0x7f136406e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1378036000 tx=0x7f1378008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:59:58.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.428+0000 7f1380c00640 1 -- 172.21.15.27:0/2888119194 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f13841ec8d0 con 0x7f1384156ba0 2024-03-28T18:59:58.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.753+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13840c6520 con 0x7f1384156ba0 2024-03-28T18:59:58.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.754+0000 7f1380c00640 1 -- 172.21.15.27:0/2888119194 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5464 (secure 0 0 0) 0x7f137c088e20 con 0x7f1384156ba0 2024-03-28T18:59:58.754 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T18:59:58.754 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T18:59:58.754 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bbd0 msgr2=0x7f136406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bbd0 0x7f136406e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1378036000 tx=0x7f1378008030 comp rx=0 tx=0).stop 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384156ba0 msgr2=0x7f138414b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384156ba0 0x7f138414b540 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f137c00c760 tx=0x7f137c00cc20 comp rx=0 tx=0).stop 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 shutdown_connections 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bbd0 0x7f136406e090 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1384156ba0 0x7f138414b540 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13840c6e70 0x7f13840c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 --2- 172.21.15.27:0/2888119194 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13840c6a90 0x7f13841eef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:59:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 >> 172.21.15.27:0/2888119194 conn(0x7f13840aa7c0 msgr2=0x7f138406ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:59:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.757+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 shutdown_connections 2024-03-28T18:59:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T18:59:58.758+0000 7f138a200640 1 -- 172.21.15.27:0/2888119194 wait complete. 2024-03-28T18:59:58.777 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-03-28T18:59:58.778 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-03-28T18:59:58.779 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-03-28T18:59:58.780 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-03-28T18:59:58.781 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-03-28T18:59:58.782 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-03-28T18:59:58.783 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-03-28T18:59:58.783 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-03-28T18:59:58.783 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-03-28T18:59:58.783 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-03-28T18:59:58.783 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-03-28T18:59:58.783 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-03-28T18:59:58.783 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-03-28T18:59:58.783 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-03-28T18:59:58.784 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T18:59:59.817 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-03-28T18:59:59.818 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T18:59:59.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T18:59:59.827 INFO:tasks.ceph.mds.c:Started 2024-03-28T18:59:59.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:00.031 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:00:00.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.071+0000 7f4b46000640 1 Processor -- start 2024-03-28T19:00:00.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.071+0000 7f4b46000640 1 -- start start 2024-03-28T19:00:00.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.073+0000 7f4b46000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b40150ae0 0x7f4b400a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:00.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.073+0000 7f4b46000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4b400a61a0 0x7f4b400c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:00.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.073+0000 7f4b46000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b40150ec0 con 0x7f4b400c6ee0 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.073+0000 7f4b46000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b4006dc70 con 0x7f4b40150ae0 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.073+0000 7f4b46000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b40153550 con 0x7f4b400a61a0 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.074+0000 7f4b44c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b40150ae0 0x7f4b400a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.074+0000 7f4b44c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b40150ae0 0x7f4b400a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54866/0 (socket says 172.21.15.27:54866) 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.074+0000 7f4b44c00640 1 -- 172.21.15.27:0/1531204013 learned_addr learned my addr 172.21.15.27:0/1531204013 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.074+0000 7f4b3fe00640 1 --2- 172.21.15.27:0/1531204013 >> v2:172.21.15.155:3300/0 conn(0x7f4b400a61a0 0x7f4b400c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.074+0000 7f4b44c00640 1 -- 172.21.15.27:0/1531204013 >> v2:172.21.15.155:3300/0 conn(0x7f4b400a61a0 msgr2=0x7f4b400c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.074+0000 7f4b44c00640 1 --2- 172.21.15.27:0/1531204013 >> v2:172.21.15.155:3300/0 conn(0x7f4b400a61a0 0x7f4b400c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.074+0000 7f4b44c00640 1 -- 172.21.15.27:0/1531204013 >> v1:172.21.15.155:6789/0 conn(0x7f4b400c6ee0 legacy=0x7f4b40153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:00.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.074+0000 7f4b44c00640 1 -- 172.21.15.27:0/1531204013 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4006df10 con 0x7f4b40150ae0 2024-03-28T19:00:00.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.076+0000 7f4b44c00640 1 --2- 172.21.15.27:0/1531204013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b400a5c60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4b400c44a0 tx=0x7f4b3402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df753dc01680d0d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:00.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.076+0000 7f4b3f400640 1 -- 172.21.15.27:0/1531204013 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3402e070 con 0x7f4b40150ae0 2024-03-28T19:00:00.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.076+0000 7f4b3f400640 1 -- 172.21.15.27:0/1531204013 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b34037070 con 0x7f4b40150ae0 2024-03-28T19:00:00.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.076+0000 7f4b3f400640 1 -- 172.21.15.27:0/1531204013 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34031a00 con 0x7f4b40150ae0 2024-03-28T19:00:00.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.077+0000 7f4b46000640 1 -- 172.21.15.27:0/1531204013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 msgr2=0x7f4b400a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:00.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.077+0000 7f4b46000640 1 --2- 172.21.15.27:0/1531204013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b400a5c60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4b400c44a0 tx=0x7f4b3402b880 comp rx=0 tx=0).stop 2024-03-28T19:00:00.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.078+0000 7f4b46000640 1 -- 172.21.15.27:0/1531204013 shutdown_connections 2024-03-28T19:00:00.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.078+0000 7f4b46000640 1 --2- 172.21.15.27:0/1531204013 >> v2:172.21.15.155:3300/0 conn(0x7f4b400a61a0 0x7f4b400c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.078+0000 7f4b46000640 1 --2- 172.21.15.27:0/1531204013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b400a5c60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.078+0000 7f4b46000640 1 -- 172.21.15.27:0/1531204013 >> 172.21.15.27:0/1531204013 conn(0x7f4b4006b0c0 msgr2=0x7f4b4006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.078+0000 7f4b46000640 1 -- 172.21.15.27:0/1531204013 shutdown_connections 2024-03-28T19:00:00.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.078+0000 7f4b46000640 1 -- 172.21.15.27:0/1531204013 wait complete. 2024-03-28T19:00:00.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.081+0000 7f4b46000640 1 Processor -- start 2024-03-28T19:00:00.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.081+0000 7f4b46000640 1 -- start start 2024-03-28T19:00:00.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.082+0000 7f4b46000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b400a61a0 0x7f4b40072d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:00.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.082+0000 7f4b46000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b400c6ee0 0x7f4b4014acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:00.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.082+0000 7f4b46000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b4014b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:00.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.082+0000 7f4b46000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4006e7b0 con 0x7f4b40150ae0 2024-03-28T19:00:00.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.082+0000 7f4b46000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4006e4d0 con 0x7f4b400c6ee0 2024-03-28T19:00:00.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.082+0000 7f4b46000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b401569d0 con 0x7f4b400a61a0 2024-03-28T19:00:00.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.083+0000 7f4b45600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b4014b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:00.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.083+0000 7f4b45600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b4014b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54882/0 (socket says 172.21.15.27:54882) 2024-03-28T19:00:00.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.083+0000 7f4b45600640 1 -- 172.21.15.27:0/23017456 learned_addr learned my addr 172.21.15.27:0/23017456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:00.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.083+0000 7f4b3fe00640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b400c6ee0 0x7f4b4014acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:00.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.083+0000 7f4b44c00640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b400a61a0 0x7f4b40072d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:00.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b45600640 1 -- 172.21.15.27:0/23017456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b400a61a0 msgr2=0x7f4b40072d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:00.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b45600640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b400a61a0 0x7f4b40072d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b45600640 1 -- 172.21.15.27:0/23017456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b400c6ee0 msgr2=0x7f4b4014acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:00.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b45600640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b400c6ee0 0x7f4b4014acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b45600640 1 -- 172.21.15.27:0/23017456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b400717c0 con 0x7f4b40150ae0 2024-03-28T19:00:00.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b45600640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b4014b230 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800c780 tx=0x7f4b3800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:00.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b3cc00640 1 -- 172.21.15.27:0/23017456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38004b60 con 0x7f4b40150ae0 2024-03-28T19:00:00.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b3cc00640 1 -- 172.21.15.27:0/23017456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b38004d00 con 0x7f4b40150ae0 2024-03-28T19:00:00.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b3cc00640 1 -- 172.21.15.27:0/23017456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3800aa90 con 0x7f4b40150ae0 2024-03-28T19:00:00.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.084+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b40071aa0 con 0x7f4b40150ae0 2024-03-28T19:00:00.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.085+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b401ea410 con 0x7f4b40150ae0 2024-03-28T19:00:00.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.086+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b401ea610 con 0x7f4b40150ae0 2024-03-28T19:00:00.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.089+0000 7f4b3cc00640 1 -- 172.21.15.27:0/23017456 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4b3800ac30 con 0x7f4b40150ae0 2024-03-28T19:00:00.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.089+0000 7f4b3cc00640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b20070010 0x7f4b200724d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:00.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.090+0000 7f4b3cc00640 1 -- 172.21.15.27:0/23017456 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7f4b3808b5a0 con 0x7f4b40150ae0 2024-03-28T19:00:00.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.090+0000 7f4b44c00640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b20070010 0x7f4b200724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:00.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.091+0000 7f4b44c00640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b20070010 0x7f4b200724d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b34036000 tx=0x7f4b34008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:00.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.096+0000 7f4b3cc00640 1 -- 172.21.15.27:0/23017456 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4b401ea610 con 0x7f4b40150ae0 2024-03-28T19:00:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.489+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b4006da50 con 0x7f4b40150ae0 2024-03-28T19:00:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.490+0000 7f4b3cc00640 1 -- 172.21.15.27:0/23017456 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f4b38059af0 con 0x7f4b40150ae0 2024-03-28T19:00:00.490 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:00.490 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T19:00:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 37 2024-03-28T19:00:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b20070010 msgr2=0x7f4b200724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b20070010 0x7f4b200724d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b34036000 tx=0x7f4b34008030 comp rx=0 tx=0).stop 2024-03-28T19:00:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 msgr2=0x7f4b4014b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b4014b230 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800c780 tx=0x7f4b3800cc40 comp rx=0 tx=0).stop 2024-03-28T19:00:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 shutdown_connections 2024-03-28T19:00:00.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b20070010 0x7f4b200724d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b40150ae0 0x7f4b4014b230 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b400c6ee0 0x7f4b4014acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.493+0000 7f4b46000640 1 --2- 172.21.15.27:0/23017456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b400a61a0 0x7f4b40072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.494+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 >> 172.21.15.27:0/23017456 conn(0x7f4b4006b0c0 msgr2=0x7f4b40154700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.494+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 shutdown_connections 2024-03-28T19:00:00.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:00.494+0000 7f4b46000640 1 -- 172.21.15.27:0/23017456 wait complete. 2024-03-28T19:00:00.513 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-03-28T19:00:00.513 INFO:tasks.mds_thrash.fs.[a]:waiting for 85 secs before thrashing 2024-03-28T19:00:00.708 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-03-28T19:00:00.708 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-03-28T19:00:00.708 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-03-28T19:00:00.708 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-03-28T19:00:00.708 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-03-28T19:00:00.708 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-03-28T19:00:00.709 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-03-28T19:00:00.710 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-03-28T19:00:00.711 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-03-28T19:00:00.712 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-03-28T19:00:00.713 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-03-28T19:00:00.714 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-03-28T19:00:00.714 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-03-28T19:00:00.714 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-03-28T19:00:00.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.026+0000 7ff1a5800640 1 Processor -- start 2024-03-28T19:00:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.026+0000 7ff1a5800640 1 -- start start 2024-03-28T19:00:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.027+0000 7ff1a5800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 0x7ff1a00ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.027+0000 7ff1a5800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1a00abd90 0x7ff1a00aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.027+0000 7ff1a5800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a006d8a0 con 0x7ff1a00c5e80 2024-03-28T19:00:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.027+0000 7ff1a5800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a00ac220 con 0x7ff1a00abd90 2024-03-28T19:00:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.027+0000 7ff1a5800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a006d650 con 0x7ff1a00c1f50 2024-03-28T19:00:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.028+0000 7ff19f400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1a00abd90 0x7ff1a00aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.028+0000 7ff19f400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1a00abd90 0x7ff1a00aa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:40058/0 (socket says 172.21.15.27:40058) 2024-03-28T19:00:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.028+0000 7ff19f400640 1 -- 172.21.15.27:0/3941836278 learned_addr learned my addr 172.21.15.27:0/3941836278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.028+0000 7ff19ea00640 1 --1- 172.21.15.27:0/3941836278 >> v1:172.21.15.155:6789/0 conn(0x7ff1a00c5e80 0x7ff1a00c2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:51722/0 (socket says 172.21.15.27:51722) 2024-03-28T19:00:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.028+0000 7ff19e000640 1 --2- 172.21.15.27:0/3941836278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 0x7ff1a00ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.029+0000 7ff19f400640 1 -- 172.21.15.27:0/3941836278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 msgr2=0x7ff1a00ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.029+0000 7ff19f400640 1 --2- 172.21.15.27:0/3941836278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 0x7ff1a00ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.029+0000 7ff19f400640 1 -- 172.21.15.27:0/3941836278 >> v1:172.21.15.155:6789/0 conn(0x7ff1a00c5e80 legacy=0x7ff1a00c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.029+0000 7ff19f400640 1 -- 172.21.15.27:0/3941836278 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a006db80 con 0x7ff1a00abd90 2024-03-28T19:00:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.030+0000 7ff19f400640 1 --2- 172.21.15.27:0/3941836278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00abd90 0x7ff1a00aa570 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff198009e50 tx=0x7ff19802e050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19d7fec17fcd0498 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.031+0000 7ff19d600640 1 -- 172.21.15.27:0/3941836278 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19802ebf0 con 0x7ff1a00abd90 2024-03-28T19:00:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.031+0000 7ff19d600640 1 -- 172.21.15.27:0/3941836278 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19802ed90 con 0x7ff1a00abd90 2024-03-28T19:00:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.031+0000 7ff19d600640 1 -- 172.21.15.27:0/3941836278 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198036610 con 0x7ff1a00abd90 2024-03-28T19:00:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.032+0000 7ff1a5800640 1 -- 172.21.15.27:0/3941836278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00abd90 msgr2=0x7ff1a00aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.032+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3941836278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00abd90 0x7ff1a00aa570 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff198009e50 tx=0x7ff19802e050 comp rx=0 tx=0).stop 2024-03-28T19:00:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.032+0000 7ff1a5800640 1 -- 172.21.15.27:0/3941836278 shutdown_connections 2024-03-28T19:00:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.032+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3941836278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00abd90 0x7ff1a00aa570 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.032+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3941836278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 0x7ff1a00ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.032+0000 7ff1a5800640 1 -- 172.21.15.27:0/3941836278 >> 172.21.15.27:0/3941836278 conn(0x7ff1a006b9d0 msgr2=0x7ff1a00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.032+0000 7ff1a5800640 1 -- 172.21.15.27:0/3941836278 shutdown_connections 2024-03-28T19:00:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.032+0000 7ff1a5800640 1 -- 172.21.15.27:0/3941836278 wait complete. 2024-03-28T19:00:01.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.034+0000 7ff1a5800640 1 Processor -- start 2024-03-28T19:00:01.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.034+0000 7ff1a5800640 1 -- start start 2024-03-28T19:00:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.035+0000 7ff1a5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a00abd90 0x7ff1a01ded70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.035+0000 7ff1a5800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 0x7ff1a00a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.035+0000 7ff1a5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c5e80 0x7ff1a01eb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.035+0000 7ff1a5800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00c6260 con 0x7ff1a00abd90 2024-03-28T19:00:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.035+0000 7ff1a5800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a006e1d0 con 0x7ff1a00c5e80 2024-03-28T19:00:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.035+0000 7ff1a5800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00c63e0 con 0x7ff1a00c1f50 2024-03-28T19:00:01.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a00abd90 0x7ff1a01ded70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:01.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a00abd90 0x7ff1a01ded70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54894/0 (socket says 172.21.15.27:54894) 2024-03-28T19:00:01.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19ea00640 1 -- 172.21.15.27:0/3356142779 learned_addr learned my addr 172.21.15.27:0/3356142779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19e000640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 0x7ff1a00a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19ea00640 1 -- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 msgr2=0x7ff1a00a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19ea00640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 0x7ff1a00a7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19ea00640 1 -- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c5e80 msgr2=0x7ff1a01eb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19f400640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c5e80 0x7ff1a01eb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19ea00640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c5e80 0x7ff1a01eb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.036+0000 7ff19ea00640 1 -- 172.21.15.27:0/3356142779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a01df4e0 con 0x7ff1a00abd90 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.037+0000 7ff19f400640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c5e80 0x7ff1a01eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.037+0000 7ff19ea00640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a00abd90 0x7ff1a01ded70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff19000c3c0 tx=0x7ff19000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.037+0000 7ff196a00640 1 -- 172.21.15.27:0/3356142779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1900074e0 con 0x7ff1a00abd90 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.037+0000 7ff196a00640 1 -- 172.21.15.27:0/3356142779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff190007680 con 0x7ff1a00abd90 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.037+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a01eead0 con 0x7ff1a00abd90 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.037+0000 7ff196a00640 1 -- 172.21.15.27:0/3356142779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190004a00 con 0x7ff1a00abd90 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.038+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a00a83e0 con 0x7ff1a00abd90 2024-03-28T19:00:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.039+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a0071180 con 0x7ff1a00abd90 2024-03-28T19:00:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.041+0000 7ff196a00640 1 -- 172.21.15.27:0/3356142779 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff19001a020 con 0x7ff1a00abd90 2024-03-28T19:00:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.042+0000 7ff196a00640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bbd0 0x7ff18006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.042+0000 7ff196a00640 1 -- 172.21.15.27:0/3356142779 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7ff19008a670 con 0x7ff1a00abd90 2024-03-28T19:00:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.043+0000 7ff19e000640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bbd0 0x7ff18006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:01.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.044+0000 7ff19e000640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bbd0 0x7ff18006e090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0071740 tx=0x7ff18c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.048+0000 7ff196a00640 1 -- 172.21.15.27:0/3356142779 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff1a0071180 con 0x7ff1a00abd90 2024-03-28T19:00:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.380+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1a01ee8c0 con 0x7ff1a00abd90 2024-03-28T19:00:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.382+0000 7ff196a00640 1 -- 172.21.15.27:0/3356142779 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6252 (secure 0 0 0) 0x7ff190058cf0 con 0x7ff1a00abd90 2024-03-28T19:00:01.382 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:01.382 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T19:00:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 37 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bbd0 msgr2=0x7ff18006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bbd0 0x7ff18006e090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0071740 tx=0x7ff18c008040 comp rx=0 tx=0).stop 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a00abd90 msgr2=0x7ff1a01ded70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a00abd90 0x7ff1a01ded70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff19000c3c0 tx=0x7ff19000c880 comp rx=0 tx=0).stop 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 shutdown_connections 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bbd0 0x7ff18006e090 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c5e80 0x7ff1a01eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00c1f50 0x7ff1a00a7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 --2- 172.21.15.27:0/3356142779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a00abd90 0x7ff1a01ded70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 >> 172.21.15.27:0/3356142779 conn(0x7ff1a006b9d0 msgr2=0x7ff1a0156370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 shutdown_connections 2024-03-28T19:00:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:01.385+0000 7ff1a5800640 1 -- 172.21.15.27:0/3356142779 wait complete. 2024-03-28T19:00:01.412 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-03-28T19:00:02.532 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-03-28T19:00:02.533 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-03-28T19:00:02.534 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-03-28T19:00:02.535 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-03-28T19:00:02.536 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-03-28T19:00:02.537 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-03-28T19:00:02.537 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-03-28T19:00:02.537 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-03-28T19:00:02.537 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-03-28T19:00:02.537 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-03-28T19:00:02.537 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-03-28T19:00:02.537 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-03-28T19:00:02.537 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-03-28T19:00:03.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.671+0000 7fae35600640 1 Processor -- start 2024-03-28T19:00:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.671+0000 7fae35600640 1 -- start start 2024-03-28T19:00:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae35600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae300c58c0 0x7fae300c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae35600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fae300be5d0 0x7fae30155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae35600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae30156230 con 0x7fae300c8ed0 2024-03-28T19:00:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae35600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae30158bd0 con 0x7fae300c58c0 2024-03-28T19:00:03.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae35600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae30158eb0 con 0x7fae300be5d0 2024-03-28T19:00:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae2fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae300c58c0 0x7fae300c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae2fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae300c58c0 0x7fae300c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54896/0 (socket says 172.21.15.27:54896) 2024-03-28T19:00:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae2fe00640 1 -- 172.21.15.27:0/4173824670 learned_addr learned my addr 172.21.15.27:0/4173824670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.673+0000 7fae34c00640 1 --2- 172.21.15.27:0/4173824670 >> v2:172.21.15.155:3300/0 conn(0x7fae300be5d0 0x7fae30155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:03.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.674+0000 7fae2fe00640 1 -- 172.21.15.27:0/4173824670 >> v2:172.21.15.155:3300/0 conn(0x7fae300be5d0 msgr2=0x7fae30155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.674+0000 7fae2fe00640 1 --2- 172.21.15.27:0/4173824670 >> v2:172.21.15.155:3300/0 conn(0x7fae300be5d0 0x7fae30155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.674+0000 7fae2fe00640 1 -- 172.21.15.27:0/4173824670 >> v1:172.21.15.155:6789/0 conn(0x7fae300c8ed0 legacy=0x7fae301555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:03.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.674+0000 7fae2fe00640 1 -- 172.21.15.27:0/4173824670 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30159190 con 0x7fae300c58c0 2024-03-28T19:00:03.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.676+0000 7fae2fe00640 1 --2- 172.21.15.27:0/4173824670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c58c0 0x7fae300c5ca0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fae3008d050 tx=0x7fae2002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85985d6415d77f7c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:03.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.676+0000 7fae2ea00640 1 -- 172.21.15.27:0/4173824670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20038070 con 0x7fae300c58c0 2024-03-28T19:00:03.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.676+0000 7fae2ea00640 1 -- 172.21.15.27:0/4173824670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2002f050 con 0x7fae300c58c0 2024-03-28T19:00:03.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.676+0000 7fae2ea00640 1 -- 172.21.15.27:0/4173824670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20031970 con 0x7fae300c58c0 2024-03-28T19:00:03.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.677+0000 7fae35600640 1 -- 172.21.15.27:0/4173824670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c58c0 msgr2=0x7fae300c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:03.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.677+0000 7fae35600640 1 --2- 172.21.15.27:0/4173824670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c58c0 0x7fae300c5ca0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fae3008d050 tx=0x7fae2002b7f0 comp rx=0 tx=0).stop 2024-03-28T19:00:03.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.677+0000 7fae35600640 1 -- 172.21.15.27:0/4173824670 shutdown_connections 2024-03-28T19:00:03.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.677+0000 7fae35600640 1 --2- 172.21.15.27:0/4173824670 >> v2:172.21.15.155:3300/0 conn(0x7fae300be5d0 0x7fae30155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:03.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.677+0000 7fae35600640 1 --2- 172.21.15.27:0/4173824670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c58c0 0x7fae300c5ca0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.677+0000 7fae35600640 1 -- 172.21.15.27:0/4173824670 >> 172.21.15.27:0/4173824670 conn(0x7fae3006b6f0 msgr2=0x7fae300acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.677+0000 7fae35600640 1 -- 172.21.15.27:0/4173824670 shutdown_connections 2024-03-28T19:00:03.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.677+0000 7fae35600640 1 -- 172.21.15.27:0/4173824670 wait complete. 2024-03-28T19:00:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.679+0000 7fae35600640 1 Processor -- start 2024-03-28T19:00:03.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.679+0000 7fae35600640 1 -- start start 2024-03-28T19:00:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae35600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae300be5d0 0x7fae30153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae35600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae300c58c0 0x7fae301ebdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae35600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c8ed0 0x7fae300c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae35600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae300b0820 con 0x7fae300c8ed0 2024-03-28T19:00:03.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae35600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae30159750 con 0x7fae300c58c0 2024-03-28T19:00:03.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae35600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae300b09a0 con 0x7fae300be5d0 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae34c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c8ed0 0x7fae300c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae34c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c8ed0 0x7fae300c22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54904/0 (socket says 172.21.15.27:54904) 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.681+0000 7fae34c00640 1 -- 172.21.15.27:0/320593954 learned_addr learned my addr 172.21.15.27:0/320593954 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.682+0000 7fae2fe00640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae300be5d0 0x7fae30153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.682+0000 7fae2f400640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae300c58c0 0x7fae301ebdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.682+0000 7fae34c00640 1 -- 172.21.15.27:0/320593954 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae300be5d0 msgr2=0x7fae30153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.682+0000 7fae34c00640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae300be5d0 0x7fae30153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.682+0000 7fae34c00640 1 -- 172.21.15.27:0/320593954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae300c58c0 msgr2=0x7fae301ebdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:03.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.682+0000 7fae34c00640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae300c58c0 0x7fae301ebdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:03.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.682+0000 7fae34c00640 1 -- 172.21.15.27:0/320593954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30154dd0 con 0x7fae300c8ed0 2024-03-28T19:00:03.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.682+0000 7fae34c00640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c8ed0 0x7fae300c22b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fae28007e60 tx=0x7fae2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:03.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.683+0000 7fae27e00640 1 -- 172.21.15.27:0/320593954 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28014420 con 0x7fae300c8ed0 2024-03-28T19:00:03.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.683+0000 7fae27e00640 1 -- 172.21.15.27:0/320593954 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae28019070 con 0x7fae300c8ed0 2024-03-28T19:00:03.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.683+0000 7fae27e00640 1 -- 172.21.15.27:0/320593954 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28014820 con 0x7fae300c8ed0 2024-03-28T19:00:03.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.683+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae301550b0 con 0x7fae300c8ed0 2024-03-28T19:00:03.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.683+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae301eb8d0 con 0x7fae300c8ed0 2024-03-28T19:00:03.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.684+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae301ebad0 con 0x7fae300c8ed0 2024-03-28T19:00:03.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.685+0000 7fae27e00640 1 -- 172.21.15.27:0/320593954 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fae280149c0 con 0x7fae300c8ed0 2024-03-28T19:00:03.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.686+0000 7fae27e00640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae1006bbd0 0x7fae1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:03.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.686+0000 7fae27e00640 1 -- 172.21.15.27:0/320593954 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7fae2808b7e0 con 0x7fae300c8ed0 2024-03-28T19:00:03.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.687+0000 7fae2fe00640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae1006bbd0 0x7fae1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:03.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.688+0000 7fae2fe00640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae1006bbd0 0x7fae1006e090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fae20035000 tx=0x7fae20008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:03.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:03.696+0000 7fae27e00640 1 -- 172.21.15.27:0/320593954 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fae28059dc0 con 0x7fae300c8ed0 2024-03-28T19:00:04.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.008+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae30069640 con 0x7fae300c8ed0 2024-03-28T19:00:04.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.009+0000 7fae27e00640 1 -- 172.21.15.27:0/320593954 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6252 (secure 0 0 0) 0x7fae280596c0 con 0x7fae300c8ed0 2024-03-28T19:00:04.010 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:04.010 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T19:00:04.010 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 37 2024-03-28T19:00:04.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae1006bbd0 msgr2=0x7fae1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:04.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae1006bbd0 0x7fae1006e090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fae20035000 tx=0x7fae20008030 comp rx=0 tx=0).stop 2024-03-28T19:00:04.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c8ed0 msgr2=0x7fae300c22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c8ed0 0x7fae300c22b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fae28007e60 tx=0x7fae2800e4b0 comp rx=0 tx=0).stop 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 shutdown_connections 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae1006bbd0 0x7fae1006e090 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae300c8ed0 0x7fae300c22b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae300c58c0 0x7fae301ebdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 --2- 172.21.15.27:0/320593954 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae300be5d0 0x7fae30153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 >> 172.21.15.27:0/320593954 conn(0x7fae3006b6f0 msgr2=0x7fae300aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 shutdown_connections 2024-03-28T19:00:04.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:04.013+0000 7fae35600640 1 -- 172.21.15.27:0/320593954 wait complete. 2024-03-28T19:00:04.041 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:00:04.595 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-03-28T19:00:04.595 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-03-28T19:00:04.595 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-03-28T19:00:04.595 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-03-28T19:00:04.595 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-03-28T19:00:04.595 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-03-28T19:00:04.596 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-03-28T19:00:04.597 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-03-28T19:00:04.598 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-03-28T19:00:04.599 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-03-28T19:00:04.600 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-03-28T19:00:06.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.274+0000 7fd855800640 1 Processor -- start 2024-03-28T19:00:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.274+0000 7fd855800640 1 -- start start 2024-03-28T19:00:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.275+0000 7fd855800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8500c7970 0x7fd8500c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.275+0000 7fd855800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd850155730 0x7fd850155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.275+0000 7fd855800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8500c7310 con 0x7fd850152f80 2024-03-28T19:00:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.275+0000 7fd855800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd850158d40 con 0x7fd8500c7970 2024-03-28T19:00:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.275+0000 7fd855800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd850159020 con 0x7fd850155730 2024-03-28T19:00:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.276+0000 7fd84e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8500c7970 0x7fd8500c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.276+0000 7fd84ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd850152f80 0x7fd8500c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36762/0 (socket says 172.21.15.27:36762) 2024-03-28T19:00:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.276+0000 7fd84ea00640 1 -- 172.21.15.27:0/4058881272 learned_addr learned my addr 172.21.15.27:0/4058881272 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.276+0000 7fd84f400640 1 --2- 172.21.15.27:0/4058881272 >> v2:172.21.15.155:3300/0 conn(0x7fd850155730 0x7fd850155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.276+0000 7fd84d600640 1 -- 172.21.15.27:0/4058881272 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839164821 0 0) 0x7fd8500c7310 con 0x7fd850152f80 2024-03-28T19:00:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.277+0000 7fd84e000640 1 -- 172.21.15.27:0/4058881272 >> v2:172.21.15.155:3300/0 conn(0x7fd850155730 msgr2=0x7fd850155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.277+0000 7fd84e000640 1 --2- 172.21.15.27:0/4058881272 >> v2:172.21.15.155:3300/0 conn(0x7fd850155730 0x7fd850155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.277+0000 7fd84e000640 1 -- 172.21.15.27:0/4058881272 >> v1:172.21.15.27:6789/0 conn(0x7fd850152f80 legacy=0x7fd8500c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.277+0000 7fd84e000640 1 -- 172.21.15.27:0/4058881272 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd850159300 con 0x7fd8500c7970 2024-03-28T19:00:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.278+0000 7fd84e000640 1 --2- 172.21.15.27:0/4058881272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8500c7970 0x7fd8500c7df0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd8500c7850 tx=0x7fd83c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23512bd43ee4cece server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.278+0000 7fd84d600640 1 -- 172.21.15.27:0/4058881272 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0315a0 con 0x7fd8500c7970 2024-03-28T19:00:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.278+0000 7fd84d600640 1 -- 172.21.15.27:0/4058881272 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c037070 con 0x7fd8500c7970 2024-03-28T19:00:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.278+0000 7fd84d600640 1 -- 172.21.15.27:0/4058881272 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c031960 con 0x7fd8500c7970 2024-03-28T19:00:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.279+0000 7fd855800640 1 -- 172.21.15.27:0/4058881272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8500c7970 msgr2=0x7fd8500c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.279+0000 7fd855800640 1 --2- 172.21.15.27:0/4058881272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8500c7970 0x7fd8500c7df0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd8500c7850 tx=0x7fd83c02b840 comp rx=0 tx=0).stop 2024-03-28T19:00:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.279+0000 7fd855800640 1 -- 172.21.15.27:0/4058881272 shutdown_connections 2024-03-28T19:00:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.279+0000 7fd855800640 1 --2- 172.21.15.27:0/4058881272 >> v2:172.21.15.155:3300/0 conn(0x7fd850155730 0x7fd850155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.279+0000 7fd855800640 1 --2- 172.21.15.27:0/4058881272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8500c7970 0x7fd8500c7df0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.279+0000 7fd855800640 1 -- 172.21.15.27:0/4058881272 >> 172.21.15.27:0/4058881272 conn(0x7fd85006b6f0 msgr2=0x7fd8500caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.279+0000 7fd855800640 1 -- 172.21.15.27:0/4058881272 shutdown_connections 2024-03-28T19:00:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.279+0000 7fd855800640 1 -- 172.21.15.27:0/4058881272 wait complete. 2024-03-28T19:00:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.281+0000 7fd855800640 1 Processor -- start 2024-03-28T19:00:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.281+0000 7fd855800640 1 -- start start 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd855800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8500c7970 0x7fd850149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd855800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd850152f80 0x7fd8501ebaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd855800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd850155730 0x7fd8501ec030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd855800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8500cbca0 con 0x7fd850155730 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd855800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd850159b00 con 0x7fd8500c7970 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd855800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85014bc90 con 0x7fd850152f80 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd84f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd850155730 0x7fd8501ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd84f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd850155730 0x7fd8501ec030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43306/0 (socket says 172.21.15.27:43306) 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.282+0000 7fd84f400640 1 -- 172.21.15.27:0/555178234 learned_addr learned my addr 172.21.15.27:0/555178234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.283+0000 7fd84ea00640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8500c7970 0x7fd850149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.283+0000 7fd84e000640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd850152f80 0x7fd8501ebaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.283+0000 7fd84f400640 1 -- 172.21.15.27:0/555178234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd850152f80 msgr2=0x7fd8501ebaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.283+0000 7fd84f400640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd850152f80 0x7fd8501ebaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.283+0000 7fd84f400640 1 -- 172.21.15.27:0/555178234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8500c7970 msgr2=0x7fd850149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.283+0000 7fd84f400640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8500c7970 0x7fd850149fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.283+0000 7fd84f400640 1 -- 172.21.15.27:0/555178234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85014b340 con 0x7fd850155730 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.283+0000 7fd84f400640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd850155730 0x7fd8501ec030 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd848007d90 tx=0x7fd84800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.284+0000 7fd846a00640 1 -- 172.21.15.27:0/555178234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848014420 con 0x7fd850155730 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.284+0000 7fd846a00640 1 -- 172.21.15.27:0/555178234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd848019070 con 0x7fd850155730 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.284+0000 7fd846a00640 1 -- 172.21.15.27:0/555178234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848014820 con 0x7fd850155730 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.284+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85014b920 con 0x7fd850155730 2024-03-28T19:00:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.284+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85014a4f0 con 0x7fd850155730 2024-03-28T19:00:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.285+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8500c6440 con 0x7fd850155730 2024-03-28T19:00:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.287+0000 7fd846a00640 1 -- 172.21.15.27:0/555178234 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd8480149c0 con 0x7fd850155730 2024-03-28T19:00:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.288+0000 7fd846a00640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd83006bc20 0x7fd83006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.288+0000 7fd846a00640 1 -- 172.21.15.27:0/555178234 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7fd84808b660 con 0x7fd850155730 2024-03-28T19:00:06.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.289+0000 7fd84ea00640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd83006bc20 0x7fd83006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:06.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.290+0000 7fd84ea00640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd83006bc20 0x7fd83006e0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd840000f60 tx=0x7fd840007c00 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:06.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.294+0000 7fd846a00640 1 -- 172.21.15.27:0/555178234 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd8500c6440 con 0x7fd850155730 2024-03-28T19:00:06.608 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-03-28T19:00:06.608 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-03-28T19:00:06.608 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-03-28T19:00:06.608 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-03-28T19:00:06.608 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-03-28T19:00:06.609 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-03-28T19:00:06.610 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-03-28T19:00:06.611 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-03-28T19:00:06.612 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-03-28T19:00:06.613 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-03-28T19:00:06.614 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-03-28T19:00:06.614 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-03-28T19:00:06.614 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-03-28T19:00:06.614 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-03-28T19:00:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.650+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd85004f400 con 0x7fd850155730 2024-03-28T19:00:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.651+0000 7fd846a00640 1 -- 172.21.15.27:0/555178234 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6252 (secure 0 0 0) 0x7fd848059dc0 con 0x7fd850155730 2024-03-28T19:00:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:06.652 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T19:00:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 37 2024-03-28T19:00:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd83006bc20 msgr2=0x7fd83006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd83006bc20 0x7fd83006e0e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd840000f60 tx=0x7fd840007c00 comp rx=0 tx=0).stop 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd850155730 msgr2=0x7fd8501ec030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd850155730 0x7fd8501ec030 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd848007d90 tx=0x7fd84800e4b0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 shutdown_connections 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd83006bc20 0x7fd83006e0e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd850155730 0x7fd8501ec030 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd850152f80 0x7fd8501ebaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 --2- 172.21.15.27:0/555178234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8500c7970 0x7fd850149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 >> 172.21.15.27:0/555178234 conn(0x7fd85006b6f0 msgr2=0x7fd850155ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 shutdown_connections 2024-03-28T19:00:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:06.655+0000 7fd855800640 1 -- 172.21.15.27:0/555178234 wait complete. 2024-03-28T19:00:06.677 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:00:08.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:08.762 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-03-28T19:00:08.762 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-03-28T19:00:08.762 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-03-28T19:00:08.762 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-03-28T19:00:08.762 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-03-28T19:00:08.763 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-03-28T19:00:08.764 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-03-28T19:00:08.765 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-03-28T19:00:08.766 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-03-28T19:00:08.767 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-03-28T19:00:08.768 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-03-28T19:00:08.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.917+0000 7f8078200640 1 Processor -- start 2024-03-28T19:00:08.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.917+0000 7f8078200640 1 -- start start 2024-03-28T19:00:08.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.919+0000 7f8078200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f807006ee00 0x7f8070158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.919+0000 7f8078200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80700c1600 con 0x7f80700c3ea0 2024-03-28T19:00:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.919+0000 7f8078200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80700c18e0 con 0x7f80700c60e0 2024-03-28T19:00:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.919+0000 7f8078200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80700c23b0 con 0x7f807006ee00 2024-03-28T19:00:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.919+0000 7f8074e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f807006ee00 0x7f8070158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.919+0000 7f8074e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f807006ee00 0x7f8070158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43310/0 (socket says 172.21.15.27:43310) 2024-03-28T19:00:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.919+0000 7f8074e00640 1 -- 172.21.15.27:0/1114969048 learned_addr learned my addr 172.21.15.27:0/1114969048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.920+0000 7f8074e00640 1 -- 172.21.15.27:0/1114969048 >> v1:172.21.15.155:6789/0 conn(0x7f80700c60e0 legacy=0x7f807006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.920+0000 7f8074e00640 1 -- 172.21.15.27:0/1114969048 >> v1:172.21.15.27:6789/0 conn(0x7f80700c3ea0 legacy=0x7f8070159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.920+0000 7f8074e00640 1 -- 172.21.15.27:0/1114969048 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80700c1bc0 con 0x7f807006ee00 2024-03-28T19:00:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.921+0000 7f8074e00640 1 --2- 172.21.15.27:0/1114969048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f8070158f00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f80700c2530 tx=0x7f805c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4876715b8e778da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.922+0000 7f806fe00640 1 -- 172.21.15.27:0/1114969048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c034070 con 0x7f807006ee00 2024-03-28T19:00:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.922+0000 7f806fe00640 1 -- 172.21.15.27:0/1114969048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c030840 con 0x7f807006ee00 2024-03-28T19:00:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.922+0000 7f806fe00640 1 -- 172.21.15.27:0/1114969048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c030c10 con 0x7f807006ee00 2024-03-28T19:00:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.922+0000 7f8078200640 1 -- 172.21.15.27:0/1114969048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 msgr2=0x7f8070158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.922+0000 7f8078200640 1 --2- 172.21.15.27:0/1114969048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f8070158f00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f80700c2530 tx=0x7f805c02b980 comp rx=0 tx=0).stop 2024-03-28T19:00:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.923+0000 7f8078200640 1 -- 172.21.15.27:0/1114969048 shutdown_connections 2024-03-28T19:00:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.923+0000 7f8078200640 1 --2- 172.21.15.27:0/1114969048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f8070158f00 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.923+0000 7f8078200640 1 -- 172.21.15.27:0/1114969048 >> 172.21.15.27:0/1114969048 conn(0x7f807006b9d0 msgr2=0x7f80700c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:08.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.923+0000 7f8078200640 1 -- 172.21.15.27:0/1114969048 shutdown_connections 2024-03-28T19:00:08.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.923+0000 7f8078200640 1 -- 172.21.15.27:0/1114969048 wait complete. 2024-03-28T19:00:08.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.925+0000 7f8078200640 1 Processor -- start 2024-03-28T19:00:08.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.925+0000 7f8078200640 1 -- start start 2024-03-28T19:00:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.926+0000 7f8078200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f80700a7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.926+0000 7f8078200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80700c3ea0 0x7f80701e1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.926+0000 7f8078200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80700c60e0 0x7f80701e1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.926+0000 7f8078200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80700c64c0 con 0x7f807006ee00 2024-03-28T19:00:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.926+0000 7f8078200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8070159b50 con 0x7f80700c60e0 2024-03-28T19:00:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.926+0000 7f8078200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80700c6640 con 0x7f80700c3ea0 2024-03-28T19:00:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8075800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f80700a7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8075800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f80700a7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43318/0 (socket says 172.21.15.27:43318) 2024-03-28T19:00:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8075800640 1 -- 172.21.15.27:0/992764971 learned_addr learned my addr 172.21.15.27:0/992764971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:08.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8076200640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80700c60e0 0x7f80701e1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8074e00640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80700c3ea0 0x7f80701e1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8075800640 1 -- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80700c3ea0 msgr2=0x7f80701e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8075800640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80700c3ea0 0x7f80701e1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8075800640 1 -- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80700c60e0 msgr2=0x7f80701e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8075800640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80700c60e0 0x7f80701e1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8075800640 1 -- 172.21.15.27:0/992764971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80701e0a20 con 0x7f807006ee00 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.927+0000 7f8076200640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80700c60e0 0x7f80701e1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.928+0000 7f8075800640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f80700a7100 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f806400c430 tx=0x7f806400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.928+0000 7f806d600640 1 -- 172.21.15.27:0/992764971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064017070 con 0x7f807006ee00 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.928+0000 7f806d600640 1 -- 172.21.15.27:0/992764971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80640052d0 con 0x7f807006ee00 2024-03-28T19:00:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.928+0000 7f806d600640 1 -- 172.21.15.27:0/992764971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064003830 con 0x7f807006ee00 2024-03-28T19:00:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.928+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80701ee780 con 0x7f807006ee00 2024-03-28T19:00:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.928+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80700a7640 con 0x7f807006ee00 2024-03-28T19:00:08.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.930+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80701eed90 con 0x7f807006ee00 2024-03-28T19:00:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.931+0000 7f806d600640 1 -- 172.21.15.27:0/992764971 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f806401a070 con 0x7f807006ee00 2024-03-28T19:00:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.932+0000 7f806d600640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f805006bbd0 0x7f805006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.932+0000 7f806d600640 1 -- 172.21.15.27:0/992764971 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6981+0+0 (secure 0 0 0) 0x7f806408a2f0 con 0x7f807006ee00 2024-03-28T19:00:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.933+0000 7f8074e00640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f805006bbd0 0x7f805006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.934+0000 7f8074e00640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f805006bbd0 0x7f805006e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f805c036000 tx=0x7f805c03b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:08.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:08.937+0000 7f806d600640 1 -- 172.21.15.27:0/992764971 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f80640588d0 con 0x7f807006ee00 2024-03-28T19:00:09.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.240+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f807004f400 con 0x7f807006ee00 2024-03-28T19:00:09.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.241+0000 7f806d600640 1 -- 172.21.15.27:0/992764971 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f80640581d0 con 0x7f807006ee00 2024-03-28T19:00:09.242 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:09.242 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T18:59:08.605442+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4775":{"gid":4775,"name":"d","rank":0,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6834/3891854856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3891854856},{"type":"v1","addr":"172.21.15.155:6834","nonce":3891854856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4775,"qdb_cluster":[4775]},"id":2}]} 2024-03-28T19:00:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 37 2024-03-28T19:00:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f805006bbd0 msgr2=0x7f805006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f805006bbd0 0x7f805006e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f805c036000 tx=0x7f805c03b040 comp rx=0 tx=0).stop 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 msgr2=0x7f80700a7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f80700a7100 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f806400c430 tx=0x7f806400c8f0 comp rx=0 tx=0).stop 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 shutdown_connections 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f805006bbd0 0x7f805006e090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80700c60e0 0x7f80701e1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80700c3ea0 0x7f80701e1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 --2- 172.21.15.27:0/992764971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f807006ee00 0x7f80700a7100 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 >> 172.21.15.27:0/992764971 conn(0x7f807006b9d0 msgr2=0x7f80700c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 shutdown_connections 2024-03-28T19:00:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:09.244+0000 7f8078200640 1 -- 172.21.15.27:0/992764971 wait complete. 2024-03-28T19:00:09.266 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:00:10.358 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-03-28T19:00:10.358 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-03-28T19:00:10.358 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-03-28T19:00:10.358 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-03-28T19:00:10.358 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-03-28T19:00:10.358 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-03-28T19:00:10.358 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-03-28T19:00:10.358 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-03-28T19:00:10.359 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-03-28T19:00:10.360 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-03-28T19:00:10.361 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-03-28T19:00:10.362 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-03-28T19:00:10.363 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-03-28T19:00:10.364 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-03-28T19:00:10.364 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-03-28T19:00:11.267 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:11.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.508+0000 7f9746000640 1 Processor -- start 2024-03-28T19:00:11.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.508+0000 7f9746000640 1 -- start start 2024-03-28T19:00:11.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9746000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97401551a0 0x7f97400c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:11.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9746000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400c5580 0x7f97400c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:11.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9746000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97400c26d0 con 0x7f97400aada0 2024-03-28T19:00:11.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9746000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9740156870 con 0x7f97401551a0 2024-03-28T19:00:11.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9746000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9740155cf0 con 0x7f97400c5580 2024-03-28T19:00:11.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9744c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97401551a0 0x7f97400c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9744c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97401551a0 0x7f97400c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43320/0 (socket says 172.21.15.27:43320) 2024-03-28T19:00:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9744c00640 1 -- 172.21.15.27:0/3929907406 learned_addr learned my addr 172.21.15.27:0/3929907406 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.510+0000 7f9745600640 1 --1- 172.21.15.27:0/3929907406 >> v1:172.21.15.27:6789/0 conn(0x7f97400aada0 0x7f97400c1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36782/0 (socket says 172.21.15.27:36782) 2024-03-28T19:00:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.511+0000 7f973fe00640 1 --2- 172.21.15.27:0/3929907406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400c5580 0x7f97400c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.511+0000 7f9744c00640 1 -- 172.21.15.27:0/3929907406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400c5580 msgr2=0x7f97400c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.511+0000 7f9744c00640 1 --2- 172.21.15.27:0/3929907406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400c5580 0x7f97400c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.511+0000 7f9744c00640 1 -- 172.21.15.27:0/3929907406 >> v1:172.21.15.27:6789/0 conn(0x7f97400aada0 legacy=0x7f97400c1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.511+0000 7f9744c00640 1 -- 172.21.15.27:0/3929907406 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9740155fd0 con 0x7f97401551a0 2024-03-28T19:00:11.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.512+0000 7f9744c00640 1 --2- 172.21.15.27:0/3929907406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f97400c5040 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9734009870 tx=0x7f973402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34200e9278b9e69d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:11.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.513+0000 7f973f400640 1 -- 172.21.15.27:0/3929907406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340317c0 con 0x7f97401551a0 2024-03-28T19:00:11.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.513+0000 7f973f400640 1 -- 172.21.15.27:0/3929907406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9734037070 con 0x7f97401551a0 2024-03-28T19:00:11.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.513+0000 7f973f400640 1 -- 172.21.15.27:0/3929907406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734031b80 con 0x7f97401551a0 2024-03-28T19:00:11.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.513+0000 7f9746000640 1 -- 172.21.15.27:0/3929907406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 msgr2=0x7f97400c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:11.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.513+0000 7f9746000640 1 --2- 172.21.15.27:0/3929907406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f97400c5040 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9734009870 tx=0x7f973402b860 comp rx=0 tx=0).stop 2024-03-28T19:00:11.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.513+0000 7f9746000640 1 -- 172.21.15.27:0/3929907406 shutdown_connections 2024-03-28T19:00:11.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.513+0000 7f9746000640 1 --2- 172.21.15.27:0/3929907406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400c5580 0x7f97400c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.514+0000 7f9746000640 1 --2- 172.21.15.27:0/3929907406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f97400c5040 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.514+0000 7f9746000640 1 -- 172.21.15.27:0/3929907406 >> 172.21.15.27:0/3929907406 conn(0x7f974006b0c0 msgr2=0x7f974006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:11.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.514+0000 7f9746000640 1 -- 172.21.15.27:0/3929907406 shutdown_connections 2024-03-28T19:00:11.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.514+0000 7f9746000640 1 -- 172.21.15.27:0/3929907406 wait complete. 2024-03-28T19:00:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.517+0000 7f9746000640 1 Processor -- start 2024-03-28T19:00:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.517+0000 7f9746000640 1 -- start start 2024-03-28T19:00:11.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9746000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400aada0 0x7f97400bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:11.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9746000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97400c5580 0x7f974015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:11.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9746000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f974015e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:11.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9746000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97401559a0 con 0x7f97401551a0 2024-03-28T19:00:11.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9746000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9740156590 con 0x7f97400c5580 2024-03-28T19:00:11.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9746000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9740155580 con 0x7f97400aada0 2024-03-28T19:00:11.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9745600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f974015e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:11.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9745600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f974015e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43336/0 (socket says 172.21.15.27:43336) 2024-03-28T19:00:11.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9744c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400aada0 0x7f97400bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:11.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.518+0000 7f9745600640 1 -- 172.21.15.27:0/3521901943 learned_addr learned my addr 172.21.15.27:0/3521901943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:11.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.519+0000 7f973fe00640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97400c5580 0x7f974015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:11.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.519+0000 7f9745600640 1 -- 172.21.15.27:0/3521901943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400aada0 msgr2=0x7f97400bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:11.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.519+0000 7f9745600640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400aada0 0x7f97400bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.519+0000 7f9745600640 1 -- 172.21.15.27:0/3521901943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97400c5580 msgr2=0x7f974015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:11.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.519+0000 7f9745600640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97400c5580 0x7f974015dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.519+0000 7f9745600640 1 -- 172.21.15.27:0/3521901943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974015e760 con 0x7f97401551a0 2024-03-28T19:00:11.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.520+0000 7f9745600640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f974015e220 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f973800c730 tx=0x7f973800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:11.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.520+0000 7f973cc00640 1 -- 172.21.15.27:0/3521901943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738017070 con 0x7f97401551a0 2024-03-28T19:00:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.520+0000 7f973cc00640 1 -- 172.21.15.27:0/3521901943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973800ce40 con 0x7f97401551a0 2024-03-28T19:00:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.520+0000 7f973cc00640 1 -- 172.21.15.27:0/3521901943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973800a840 con 0x7f97401551a0 2024-03-28T19:00:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.520+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97401ebfb0 con 0x7f97401551a0 2024-03-28T19:00:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.520+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97401eb4d0 con 0x7f97401551a0 2024-03-28T19:00:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.522+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971c005290 con 0x7f97401551a0 2024-03-28T19:00:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.525+0000 7f973cc00640 1 -- 172.21.15.27:0/3521901943 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f973801a070 con 0x7f97401551a0 2024-03-28T19:00:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.525+0000 7f973cc00640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f972006bca0 0x7f972006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.526+0000 7f973cc00640 1 -- 172.21.15.27:0/3521901943 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7067+0+0 (secure 0 0 0) 0x7f973808ac40 con 0x7f97401551a0 2024-03-28T19:00:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.526+0000 7f9744c00640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f972006bca0 0x7f972006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:11.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.527+0000 7f9744c00640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f972006bca0 0x7f972006e160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9734036000 tx=0x7f973403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.528+0000 7f973cc00640 1 -- 172.21.15.27:0/3521901943 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f974015e760 con 0x7f97401551a0 2024-03-28T19:00:11.616 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-03-28T19:00:11.616 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-03-28T19:00:11.616 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-03-28T19:00:11.616 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-03-28T19:00:11.617 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-03-28T19:00:11.618 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-03-28T19:00:11.619 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-03-28T19:00:11.620 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-03-28T19:00:11.621 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-03-28T19:00:11.621 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-03-28T19:00:11.621 INFO:tasks.workunit.client.0.smithi027.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-03-28T19:00:11.621 INFO:tasks.workunit.client.0.smithi027.stdout:Take third snapshot .snap/test3 2024-03-28T19:00:11.672 INFO:tasks.workunit.client.0.smithi027.stdout:Delete the snapshots... 2024-03-28T19:00:11.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.854+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f971c005580 con 0x7f97401551a0 2024-03-28T19:00:11.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.855+0000 7f973cc00640 1 -- 172.21.15.27:0/3521901943 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5453 (secure 0 0 0) 0x7f9738058c00 con 0x7f97401551a0 2024-03-28T19:00:11.858 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:11.858 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:10.826312+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:00:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 39 2024-03-28T19:00:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f972006bca0 msgr2=0x7f972006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f972006bca0 0x7f972006e160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9734036000 tx=0x7f973403d040 comp rx=0 tx=0).stop 2024-03-28T19:00:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 msgr2=0x7f974015e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f974015e220 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f973800c730 tx=0x7f973800cbf0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 shutdown_connections 2024-03-28T19:00:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f972006bca0 0x7f972006e160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97401551a0 0x7f974015e220 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97400c5580 0x7f974015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 --2- 172.21.15.27:0/3521901943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97400aada0 0x7f97400bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:11.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.858+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 >> 172.21.15.27:0/3521901943 conn(0x7f974006b0c0 msgr2=0x7f974006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:11.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.859+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 shutdown_connections 2024-03-28T19:00:11.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:11.859+0000 7f9746000640 1 -- 172.21.15.27:0/3521901943 wait complete. 2024-03-28T19:00:11.879 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-03-28T19:00:11.879 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:00:11.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:12.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.122+0000 7f1199c00640 1 Processor -- start 2024-03-28T19:00:12.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.122+0000 7f1199c00640 1 -- start start 2024-03-28T19:00:12.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.123+0000 7f1199c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1194150c70 0x7f11940a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:12.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.123+0000 7f1199c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11940c3af0 con 0x7f11940c3690 2024-03-28T19:00:12.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.123+0000 7f1199c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11940c3d90 con 0x7f11941534c0 2024-03-28T19:00:12.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.123+0000 7f1199c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11940ab4e0 con 0x7f1194150c70 2024-03-28T19:00:12.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.124+0000 7f118be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f11940c3690 0x7f11940c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36790/0 (socket says 172.21.15.27:36790) 2024-03-28T19:00:12.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.124+0000 7f118be00640 1 -- 172.21.15.27:0/1242942730 learned_addr learned my addr 172.21.15.27:0/1242942730 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.124+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027034446 0 0) 0x7f11940c3af0 con 0x7f11940c3690 2024-03-28T19:00:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.124+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1178003520 con 0x7f11940c3690 2024-03-28T19:00:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.124+0000 7f1192a00640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1194150c70 0x7f11940a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529390790 0 0) 0x7f11940c3d90 con 0x7f11941534c0 2024-03-28T19:00:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11940c3af0 con 0x7f11941534c0 2024-03-28T19:00:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291298364 0 0) 0x7f1178003520 con 0x7f11940c3690 2024-03-28T19:00:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1194150c70 msgr2=0x7f11940a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1194150c70 0x7f11940a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 >> v1:172.21.15.155:6789/0 conn(0x7f11941534c0 legacy=0x7f1194153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940ab710 con 0x7f11940c3690 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f117c001670 con 0x7f11940c3690 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 >> v1:172.21.15.27:6789/0 conn(0x7f11940c3690 legacy=0x7f11940c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1178003b70 0x7f1178003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1178004800 0x7f1178004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178005230 0x7f117800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117800c0c0 con 0x7f1178004800 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117800c270 con 0x7f1178003b70 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117800c470 con 0x7f1178005230 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.125+0000 7f1192a00640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1194150c70 0x7f11940a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.126+0000 7f118be00640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1178004800 0x7f1178004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.126+0000 7f1192a00640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1178003b70 0x7f1178003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.126+0000 7f1193400640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178005230 0x7f117800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.126+0000 7f118be00640 1 -- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178005230 msgr2=0x7f117800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.126+0000 7f118be00640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178005230 0x7f117800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.126+0000 7f118be00640 1 -- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1178003b70 msgr2=0x7f1178003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.126+0000 7f118be00640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1178003b70 0x7f1178003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.126+0000 7f118be00640 1 -- 172.21.15.27:0/1242942730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940ab710 con 0x7f1178004800 2024-03-28T19:00:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.127+0000 7f118be00640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1178004800 0x7f1178004c80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f117c00c240 tx=0x7f1178005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36f5fb55c2f56ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.128+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c007d30 con 0x7f1178004800 2024-03-28T19:00:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.128+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c030070 con 0x7f1178004800 2024-03-28T19:00:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.128+0000 7f1192000640 1 -- 172.21.15.27:0/1242942730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c005d30 con 0x7f1178004800 2024-03-28T19:00:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.128+0000 7f1199c00640 1 -- 172.21.15.27:0/1242942730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1178004800 msgr2=0x7f1178004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.128+0000 7f1199c00640 1 --2- 172.21.15.27:0/1242942730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1178004800 0x7f1178004c80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f117c00c240 tx=0x7f1178005830 comp rx=0 tx=0).stop 2024-03-28T19:00:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.128+0000 7f1193400640 1 -- 172.21.15.27:0/1242942730 reap_dead start 2024-03-28T19:00:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.129+0000 7f1199c00640 1 -- 172.21.15.27:0/1242942730 shutdown_connections 2024-03-28T19:00:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.129+0000 7f1199c00640 1 -- 172.21.15.27:0/1242942730 >> 172.21.15.27:0/1242942730 conn(0x7f119406b9d0 msgr2=0x7f11940c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.129+0000 7f1199c00640 1 -- 172.21.15.27:0/1242942730 shutdown_connections 2024-03-28T19:00:12.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.129+0000 7f1199c00640 1 -- 172.21.15.27:0/1242942730 wait complete. 2024-03-28T19:00:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.131+0000 7f1199c00640 1 Processor -- start 2024-03-28T19:00:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.131+0000 7f1199c00640 1 -- start start 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1199c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178004800 0x7f11941eb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1199c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f119409a9a0 0x7f11941ef420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1199c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11941554f0 0x7f11940c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1199c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194154060 con 0x7f11941554f0 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1199c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11940ac1f0 con 0x7f119409a9a0 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1199c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11940abcd0 con 0x7f1178004800 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1193400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11941554f0 0x7f11940c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1193400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11941554f0 0x7f11940c0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43342/0 (socket says 172.21.15.27:43342) 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f1193400640 1 -- 172.21.15.27:0/2515081938 learned_addr learned my addr 172.21.15.27:0/2515081938 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.133+0000 7f118be00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f119409a9a0 0x7f11941ef420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.134+0000 7f1192a00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178004800 0x7f11941eb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.134+0000 7f1193400640 1 -- 172.21.15.27:0/2515081938 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178004800 msgr2=0x7f11941eb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.134+0000 7f1193400640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178004800 0x7f11941eb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.134+0000 7f1193400640 1 -- 172.21.15.27:0/2515081938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f119409a9a0 msgr2=0x7f11941ef420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.134+0000 7f1193400640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f119409a9a0 0x7f11941ef420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.134+0000 7f1193400640 1 -- 172.21.15.27:0/2515081938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119406e1b0 con 0x7f11941554f0 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.134+0000 7f1193400640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11941554f0 0x7f11940c0030 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f11941538a0 tx=0x7f118c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.135+0000 7f118aa00640 1 -- 172.21.15.27:0/2515081938 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c00ee50 con 0x7f11941554f0 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.135+0000 7f118aa00640 1 -- 172.21.15.27:0/2515081938 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c018070 con 0x7f11941554f0 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.135+0000 7f118aa00640 1 -- 172.21.15.27:0/2515081938 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c003dc0 con 0x7f11941554f0 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.135+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11941eec40 con 0x7f11941554f0 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.135+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11940a8a10 con 0x7f11941554f0 2024-03-28T19:00:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.136+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119404f400 con 0x7f11941554f0 2024-03-28T19:00:12.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.138+0000 7f118aa00640 1 -- 172.21.15.27:0/2515081938 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f118c0050d0 con 0x7f11941554f0 2024-03-28T19:00:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.139+0000 7f118aa00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f117006bbd0 0x7f117006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:12.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.139+0000 7f118aa00640 1 -- 172.21.15.27:0/2515081938 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7067+0+0 (secure 0 0 0) 0x7f118c08acd0 con 0x7f11941554f0 2024-03-28T19:00:12.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.140+0000 7f1192a00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f117006bbd0 0x7f117006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.141+0000 7f1192a00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f117006bbd0 0x7f117006e090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1184006e80 tx=0x7f1184008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.144+0000 7f118aa00640 1 -- 172.21.15.27:0/2515081938 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f118c059260 con 0x7f11941554f0 2024-03-28T19:00:12.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.491+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11941541e0 con 0x7f11941554f0 2024-03-28T19:00:12.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.492+0000 7f118aa00640 1 -- 172.21.15.27:0/2515081938 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5450 (secure 0 0 0) 0x7f118c058b60 con 0x7f11941554f0 2024-03-28T19:00:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:12.492 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:11.293662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":5,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:00:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 40 2024-03-28T19:00:12.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.495+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f117006bbd0 msgr2=0x7f117006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:12.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.495+0000 7f1199c00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f117006bbd0 0x7f117006e090 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1184006e80 tx=0x7f1184008040 comp rx=0 tx=0).stop 2024-03-28T19:00:12.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.495+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11941554f0 msgr2=0x7f11940c0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:12.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.495+0000 7f1199c00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11941554f0 0x7f11940c0030 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f11941538a0 tx=0x7f118c00ec00 comp rx=0 tx=0).stop 2024-03-28T19:00:12.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.496+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 shutdown_connections 2024-03-28T19:00:12.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.496+0000 7f1199c00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f117006bbd0 0x7f117006e090 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.496+0000 7f1199c00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11941554f0 0x7f11940c0030 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.496+0000 7f1199c00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f119409a9a0 0x7f11941ef420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.496+0000 7f1199c00640 1 --2- 172.21.15.27:0/2515081938 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1178004800 0x7f11941eb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:12.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.496+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 >> 172.21.15.27:0/2515081938 conn(0x7f119406b9d0 msgr2=0x7f1194152920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:12.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.496+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 shutdown_connections 2024-03-28T19:00:12.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:12.496+0000 7f1199c00640 1 -- 172.21.15.27:0/2515081938 wait complete. 2024-03-28T19:00:12.519 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 40, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5325": { "addr": "172.21.15.27:6836/181571966", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 181571966, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 181571966, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5325, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T18:59:54.159296+0000", "qdb_cluster": [ 5325 ], "qdb_leader": 5325, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5325 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5433": { "addr": "172.21.15.27:6837/700464224", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 700464224, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 700464224, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5433, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:00:11.293662+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5433 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:00:14.438 INFO:tasks.workunit.client.0.smithi027.stdout:.//.snap/test1 2024-03-28T19:00:14.438 INFO:tasks.workunit.client.0.smithi027.stdout:.//.snap/test2 2024-03-28T19:00:14.438 INFO:tasks.workunit.client.0.smithi027.stdout:.//.snap/test3 2024-03-28T19:00:14.455 INFO:tasks.workunit.client.0.smithi027.stdout:./274/.snap/snap-subdir-test 2024-03-28T19:00:14.479 INFO:tasks.workunit.client.0.smithi027.stdout:./166/.snap/snap-subdir-test 2024-03-28T19:00:14.508 INFO:tasks.workunit.client.0.smithi027.stdout:./256/.snap/snap-subdir-test 2024-03-28T19:00:14.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:14.541 INFO:tasks.workunit.client.0.smithi027.stdout:./247/.snap/snap-subdir-test 2024-03-28T19:00:14.571 INFO:tasks.workunit.client.0.smithi027.stdout:./248/.snap/snap-subdir-test 2024-03-28T19:00:14.593 INFO:tasks.workunit.client.0.smithi027.stdout:./147/.snap/snap-subdir-test 2024-03-28T19:00:14.612 INFO:tasks.workunit.client.0.smithi027.stdout:./151/.snap/snap-subdir-test 2024-03-28T19:00:14.638 INFO:tasks.workunit.client.0.smithi027.stdout:./170/.snap/snap-subdir-test 2024-03-28T19:00:14.658 INFO:tasks.workunit.client.0.smithi027.stdout:./167/.snap/snap-subdir-test 2024-03-28T19:00:14.677 INFO:tasks.workunit.client.0.smithi027.stdout:./140/.snap/snap-subdir-test 2024-03-28T19:00:14.698 INFO:tasks.workunit.client.0.smithi027.stdout:./142/.snap/snap-subdir-test 2024-03-28T19:00:14.716 INFO:tasks.workunit.client.0.smithi027.stdout:./150/.snap/snap-subdir-test 2024-03-28T19:00:14.734 INFO:tasks.workunit.client.0.smithi027.stdout:./273/.snap/snap-subdir-test 2024-03-28T19:00:14.752 INFO:tasks.workunit.client.0.smithi027.stdout:./269/.snap/snap-subdir-test 2024-03-28T19:00:14.772 INFO:tasks.workunit.client.0.smithi027.stdout:./105/.snap/snap-subdir-test 2024-03-28T19:00:14.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.789+0000 7f0bf2800640 1 Processor -- start 2024-03-28T19:00:14.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.789+0000 7f0bf2800640 1 -- start start 2024-03-28T19:00:14.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0bf2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bec152d10 0x7f0bec0c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:14.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0bf2800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0bec1562f0 0x7f0bec15ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:14.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0bf2800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec1530f0 con 0x7f0bec0c8e10 2024-03-28T19:00:14.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0bf2800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec06d3d0 con 0x7f0bec152d10 2024-03-28T19:00:14.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0bf2800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec0c92f0 con 0x7f0bec1562f0 2024-03-28T19:00:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0beb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bec152d10 0x7f0bec0c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0beb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bec152d10 0x7f0bec0c5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43356/0 (socket says 172.21.15.27:43356) 2024-03-28T19:00:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0beb400640 1 -- 172.21.15.27:0/3202991585 learned_addr learned my addr 172.21.15.27:0/3202991585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:14.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.792+0000 7f0bebe00640 1 --2- 172.21.15.27:0/3202991585 >> v2:172.21.15.155:3300/0 conn(0x7f0bec1562f0 0x7f0bec15ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.793+0000 7f0beb400640 1 -- 172.21.15.27:0/3202991585 >> v2:172.21.15.155:3300/0 conn(0x7f0bec1562f0 msgr2=0x7f0bec15ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.793+0000 7f0beb400640 1 --2- 172.21.15.27:0/3202991585 >> v2:172.21.15.155:3300/0 conn(0x7f0bec1562f0 0x7f0bec15ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.793+0000 7f0beb400640 1 -- 172.21.15.27:0/3202991585 >> v1:172.21.15.155:6789/0 conn(0x7f0bec0c8e10 legacy=0x7f0bec158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:14.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.793+0000 7f0beb400640 1 -- 172.21.15.27:0/3202991585 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec155810 con 0x7f0bec152d10 2024-03-28T19:00:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.795+0000 7f0beb400640 1 --2- 172.21.15.27:0/3202991585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec152d10 0x7f0bec0c5890 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0bec0c7840 tx=0x7f0bdc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc4ba2cba23eec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.796+0000 7f0bea000640 1 -- 172.21.15.27:0/3202991585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc02e070 con 0x7f0bec152d10 2024-03-28T19:00:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.796+0000 7f0bea000640 1 -- 172.21.15.27:0/3202991585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc037070 con 0x7f0bec152d10 2024-03-28T19:00:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.796+0000 7f0bea000640 1 -- 172.21.15.27:0/3202991585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc031a00 con 0x7f0bec152d10 2024-03-28T19:00:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.796+0000 7f0bf2800640 1 -- 172.21.15.27:0/3202991585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec152d10 msgr2=0x7f0bec0c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.796+0000 7f0bf2800640 1 --2- 172.21.15.27:0/3202991585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec152d10 0x7f0bec0c5890 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0bec0c7840 tx=0x7f0bdc02b880 comp rx=0 tx=0).stop 2024-03-28T19:00:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.797+0000 7f0bf2800640 1 -- 172.21.15.27:0/3202991585 shutdown_connections 2024-03-28T19:00:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.797+0000 7f0bf2800640 1 --2- 172.21.15.27:0/3202991585 >> v2:172.21.15.155:3300/0 conn(0x7f0bec1562f0 0x7f0bec15ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:14.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.797+0000 7f0bf2800640 1 --2- 172.21.15.27:0/3202991585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec152d10 0x7f0bec0c5890 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:14.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.797+0000 7f0bf2800640 1 -- 172.21.15.27:0/3202991585 >> 172.21.15.27:0/3202991585 conn(0x7f0bec066fb0 msgr2=0x7f0bec0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:14.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.797+0000 7f0bf2800640 1 -- 172.21.15.27:0/3202991585 shutdown_connections 2024-03-28T19:00:14.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.797+0000 7f0bf2800640 1 -- 172.21.15.27:0/3202991585 wait complete. 2024-03-28T19:00:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.799+0000 7f0bf2800640 1 Processor -- start 2024-03-28T19:00:14.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.799+0000 7f0bf2800640 1 -- start start 2024-03-28T19:00:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.800+0000 7f0bf2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8e10 0x7f0bec0c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.800+0000 7f0bf2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec152d10 0x7f0bec148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.800+0000 7f0bf2800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec1562f0 0x7f0bec149420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.800+0000 7f0bf2800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec1592f0 con 0x7f0bec0c8e10 2024-03-28T19:00:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.800+0000 7f0bf2800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec06d850 con 0x7f0bec152d10 2024-03-28T19:00:14.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.800+0000 7f0bf2800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0c2bc0 con 0x7f0bec1562f0 2024-03-28T19:00:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.801+0000 7f0beb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8e10 0x7f0bec0c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.801+0000 7f0beb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8e10 0x7f0bec0c4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43362/0 (socket says 172.21.15.27:43362) 2024-03-28T19:00:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.801+0000 7f0beb400640 1 -- 172.21.15.27:0/558875356 learned_addr learned my addr 172.21.15.27:0/558875356 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.801+0000 7f0beaa00640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec152d10 0x7f0bec148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.801+0000 7f0bebe00640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec1562f0 0x7f0bec149420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.802+0000 7f0beb400640 1 -- 172.21.15.27:0/558875356 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec1562f0 msgr2=0x7f0bec149420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.802+0000 7f0beb400640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec1562f0 0x7f0bec149420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.802+0000 7f0beb400640 1 -- 172.21.15.27:0/558875356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec152d10 msgr2=0x7f0bec148ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.802+0000 7f0beb400640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec152d10 0x7f0bec148ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.802+0000 7f0beb400640 1 -- 172.21.15.27:0/558875356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec1ec690 con 0x7f0bec0c8e10 2024-03-28T19:00:14.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.802+0000 7f0beb400640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8e10 0x7f0bec0c4630 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc02b8c0 tx=0x7f0bdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:14.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.802+0000 7f0be3400640 1 -- 172.21.15.27:0/558875356 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc032670 con 0x7f0bec0c8e10 2024-03-28T19:00:14.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.803+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec0c2080 con 0x7f0bec0c8e10 2024-03-28T19:00:14.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.803+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec14f120 con 0x7f0bec0c8e10 2024-03-28T19:00:14.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.803+0000 7f0be3400640 1 -- 172.21.15.27:0/558875356 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc032810 con 0x7f0bec0c8e10 2024-03-28T19:00:14.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.803+0000 7f0be3400640 1 -- 172.21.15.27:0/558875356 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc032c00 con 0x7f0bec0c8e10 2024-03-28T19:00:14.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.805+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc4005290 con 0x7f0bec0c8e10 2024-03-28T19:00:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.806+0000 7f0be3400640 1 -- 172.21.15.27:0/558875356 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0bdc04d3f0 con 0x7f0bec0c8e10 2024-03-28T19:00:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.807+0000 7f0be3400640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.808+0000 7f0be3400640 1 -- 172.21.15.27:0/558875356 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7067+0+0 (secure 0 0 0) 0x7f0bdc0aac10 con 0x7f0bec0c8e10 2024-03-28T19:00:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.809+0000 7f0beaa00640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:14.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.811+0000 7f0beaa00640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8007fd0 tx=0x7f0bd8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:14.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:14.815+0000 7f0be3400640 1 -- 172.21.15.27:0/558875356 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0bdc040070 con 0x7f0bec0c8e10 2024-03-28T19:00:14.815 INFO:tasks.workunit.client.0.smithi027.stdout:./210/.snap/snap-subdir-test 2024-03-28T19:00:14.833 INFO:tasks.workunit.client.0.smithi027.stdout:./265/.snap/snap-subdir-test 2024-03-28T19:00:14.854 INFO:tasks.workunit.client.0.smithi027.stdout:./218/.snap/snap-subdir-test 2024-03-28T19:00:14.862 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-03-28T19:00:14.862 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-03-28T19:00:14.862 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-03-28T19:00:14.862 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-03-28T19:00:14.862 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-03-28T19:00:14.863 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-03-28T19:00:14.864 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-03-28T19:00:14.865 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-03-28T19:00:14.866 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-03-28T19:00:14.867 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-03-28T19:00:14.868 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-03-28T19:00:14.868 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-03-28T19:00:14.868 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-03-28T19:00:14.872 INFO:tasks.workunit.client.0.smithi027.stdout:./290/.snap/snap-subdir-test 2024-03-28T19:00:14.891 INFO:tasks.workunit.client.0.smithi027.stdout:./149/.snap/snap-subdir-test 2024-03-28T19:00:14.911 INFO:tasks.workunit.client.0.smithi027.stdout:./109/.snap/snap-subdir-test 2024-03-28T19:00:14.930 INFO:tasks.workunit.client.0.smithi027.stdout:./292/.snap/snap-subdir-test 2024-03-28T19:00:14.947 INFO:tasks.workunit.client.0.smithi027.stdout:./242/.snap/snap-subdir-test 2024-03-28T19:00:14.967 INFO:tasks.workunit.client.0.smithi027.stdout:./158/.snap/snap-subdir-test 2024-03-28T19:00:14.988 INFO:tasks.workunit.client.0.smithi027.stdout:./243/.snap/snap-subdir-test 2024-03-28T19:00:15.011 INFO:tasks.workunit.client.0.smithi027.stdout:./106/.snap/snap-subdir-test 2024-03-28T19:00:15.030 INFO:tasks.workunit.client.0.smithi027.stdout:./250/.snap/snap-subdir-test 2024-03-28T19:00:15.051 INFO:tasks.workunit.client.0.smithi027.stdout:./221/.snap/snap-subdir-test 2024-03-28T19:00:15.078 INFO:tasks.workunit.client.0.smithi027.stdout:./244/.snap/snap-subdir-test 2024-03-28T19:00:15.096 INFO:tasks.workunit.client.0.smithi027.stdout:./258/.snap/snap-subdir-test 2024-03-28T19:00:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:./268/.snap/snap-subdir-test 2024-03-28T19:00:15.135 INFO:tasks.workunit.client.0.smithi027.stdout:./189/.snap/snap-subdir-test 2024-03-28T19:00:15.153 INFO:tasks.workunit.client.0.smithi027.stdout:./285/.snap/snap-subdir-test 2024-03-28T19:00:15.170 INFO:tasks.workunit.client.0.smithi027.stdout:./208/.snap/snap-subdir-test 2024-03-28T19:00:15.187 INFO:tasks.workunit.client.0.smithi027.stdout:./288/.snap/snap-subdir-test 2024-03-28T19:00:15.206 INFO:tasks.workunit.client.0.smithi027.stdout:./209/.snap/snap-subdir-test 2024-03-28T19:00:15.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.210+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bc4005580 con 0x7f0bec0c8e10 2024-03-28T19:00:15.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.211+0000 7f0be3400640 1 -- 172.21.15.27:0/558875356 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f0bdc07b0c0 con 0x7f0bec0c8e10 2024-03-28T19:00:15.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:15.211 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 42 2024-03-28T19:00:15.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 msgr2=0x7f0bcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:15.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8007fd0 tx=0x7f0bd8009040 comp rx=0 tx=0).stop 2024-03-28T19:00:15.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8e10 msgr2=0x7f0bec0c4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:15.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8e10 0x7f0bec0c4630 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc02b8c0 tx=0x7f0bdc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:00:15.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 shutdown_connections 2024-03-28T19:00:15.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:15.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec1562f0 0x7f0bec149420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:15.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec152d10 0x7f0bec148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:15.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 --2- 172.21.15.27:0/558875356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8e10 0x7f0bec0c4630 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:15.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.214+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 >> 172.21.15.27:0/558875356 conn(0x7f0bec066fb0 msgr2=0x7f0bec067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:15.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.215+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 shutdown_connections 2024-03-28T19:00:15.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:15.215+0000 7f0bf2800640 1 -- 172.21.15.27:0/558875356 wait complete. 2024-03-28T19:00:15.217 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:00:15.217+0000 7f256be00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:00:15.225 INFO:tasks.workunit.client.0.smithi027.stdout:./253/.snap/snap-subdir-test 2024-03-28T19:00:15.236 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=4775 2024-03-28T19:00:15.237 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.d 2024-03-28T19:00:15.244 INFO:tasks.workunit.client.0.smithi027.stdout:./249/.snap/snap-subdir-test 2024-03-28T19:00:15.263 INFO:tasks.workunit.client.0.smithi027.stdout:./133/.snap/snap-subdir-test 2024-03-28T19:00:15.280 INFO:tasks.workunit.client.0.smithi027.stdout:./291/.snap/snap-subdir-test 2024-03-28T19:00:15.297 INFO:tasks.workunit.client.0.smithi027.stdout:./226/.snap/snap-subdir-test 2024-03-28T19:00:15.317 INFO:tasks.workunit.client.0.smithi027.stdout:./173/.snap/snap-subdir-test 2024-03-28T19:00:15.338 INFO:tasks.workunit.client.0.smithi027.stdout:./174/.snap/snap-subdir-test 2024-03-28T19:00:15.353 INFO:tasks.workunit.client.0.smithi027.stdout:./299/.snap/snap-subdir-test 2024-03-28T19:00:15.370 INFO:tasks.workunit.client.0.smithi027.stdout:./262/.snap/snap-subdir-test 2024-03-28T19:00:15.386 INFO:tasks.workunit.client.0.smithi027.stdout:./231/.snap/snap-subdir-test 2024-03-28T19:00:15.404 INFO:tasks.workunit.client.0.smithi027.stdout:./214/.snap/snap-subdir-test 2024-03-28T19:00:15.421 INFO:tasks.workunit.client.0.smithi027.stdout:./281/.snap/snap-subdir-test 2024-03-28T19:00:15.438 INFO:tasks.workunit.client.0.smithi027.stdout:./289/.snap/snap-subdir-test 2024-03-28T19:00:15.456 INFO:tasks.workunit.client.0.smithi027.stdout:./283/.snap/snap-subdir-test 2024-03-28T19:00:15.474 INFO:tasks.workunit.client.0.smithi027.stdout:./232/.snap/snap-subdir-test 2024-03-28T19:00:15.492 INFO:tasks.workunit.client.0.smithi027.stdout:./298/.snap/snap-subdir-test 2024-03-28T19:00:15.509 INFO:tasks.workunit.client.0.smithi027.stdout:./233/.snap/snap-subdir-test 2024-03-28T19:00:15.528 INFO:tasks.workunit.client.0.smithi027.stdout:./155/.snap/snap-subdir-test 2024-03-28T19:00:15.547 INFO:tasks.workunit.client.0.smithi027.stdout:./114/.snap/snap-subdir-test 2024-03-28T19:00:15.566 INFO:tasks.workunit.client.0.smithi027.stdout:./159/.snap/snap-subdir-test 2024-03-28T19:00:15.585 INFO:tasks.workunit.client.0.smithi027.stdout:./187/.snap/snap-subdir-test 2024-03-28T19:00:15.602 INFO:tasks.workunit.client.0.smithi027.stdout:./185/.snap/snap-subdir-test 2024-03-28T19:00:15.619 INFO:tasks.workunit.client.0.smithi027.stdout:./293/.snap/snap-subdir-test 2024-03-28T19:00:15.638 INFO:tasks.workunit.client.0.smithi027.stdout:./228/.snap/snap-subdir-test 2024-03-28T19:00:15.658 INFO:tasks.workunit.client.0.smithi027.stdout:./120/.snap/snap-subdir-test 2024-03-28T19:00:15.676 INFO:tasks.workunit.client.0.smithi027.stdout:./110/.snap/snap-subdir-test 2024-03-28T19:00:15.694 INFO:tasks.workunit.client.0.smithi027.stdout:./267/.snap/snap-subdir-test 2024-03-28T19:00:15.710 INFO:tasks.workunit.client.0.smithi027.stdout:./202/.snap/snap-subdir-test 2024-03-28T19:00:15.730 INFO:tasks.workunit.client.0.smithi027.stdout:./137/.snap/snap-subdir-test 2024-03-28T19:00:15.747 INFO:tasks.workunit.client.0.smithi027.stdout:./246/.snap/snap-subdir-test 2024-03-28T19:00:15.766 INFO:tasks.workunit.client.0.smithi027.stdout:./219/.snap/snap-subdir-test 2024-03-28T19:00:15.785 INFO:tasks.workunit.client.0.smithi027.stdout:./198/.snap/snap-subdir-test 2024-03-28T19:00:15.840 INFO:tasks.workunit.client.0.smithi027.stdout:./136/.snap/snap-subdir-test 2024-03-28T19:00:15.857 INFO:tasks.workunit.client.0.smithi027.stdout:./270/.snap/snap-subdir-test 2024-03-28T19:00:15.876 INFO:tasks.workunit.client.0.smithi027.stdout:./125/.snap/snap-subdir-test 2024-03-28T19:00:15.896 INFO:tasks.workunit.client.0.smithi027.stdout:./194/.snap/snap-subdir-test 2024-03-28T19:00:15.914 INFO:tasks.workunit.client.0.smithi027.stdout:./119/.snap/snap-subdir-test 2024-03-28T19:00:15.930 INFO:tasks.workunit.client.0.smithi027.stdout:./297/.snap/snap-subdir-test 2024-03-28T19:00:15.950 INFO:tasks.workunit.client.0.smithi027.stdout:./230/.snap/snap-subdir-test 2024-03-28T19:00:15.964 INFO:tasks.workunit.client.0.smithi027.stdout:./260/.snap/snap-subdir-test 2024-03-28T19:00:15.983 INFO:tasks.workunit.client.0.smithi027.stdout:./157/.snap/snap-subdir-test 2024-03-28T19:00:16.003 INFO:tasks.workunit.client.0.smithi027.stdout:./172/.snap/snap-subdir-test 2024-03-28T19:00:16.020 INFO:tasks.workunit.client.0.smithi027.stdout:./277/.snap/snap-subdir-test 2024-03-28T19:00:16.039 INFO:tasks.workunit.client.0.smithi027.stdout:./186/.snap/snap-subdir-test 2024-03-28T19:00:16.056 INFO:tasks.workunit.client.0.smithi027.stdout:./237/.snap/snap-subdir-test 2024-03-28T19:00:16.074 INFO:tasks.workunit.client.0.smithi027.stdout:./190/.snap/snap-subdir-test 2024-03-28T19:00:16.091 INFO:tasks.workunit.client.0.smithi027.stdout:./213/.snap/snap-subdir-test 2024-03-28T19:00:16.111 INFO:tasks.workunit.client.0.smithi027.stdout:./104/.snap/snap-subdir-test 2024-03-28T19:00:16.131 INFO:tasks.workunit.client.0.smithi027.stdout:./164/.snap/snap-subdir-test 2024-03-28T19:00:16.149 INFO:tasks.workunit.client.0.smithi027.stdout:./112/.snap/snap-subdir-test 2024-03-28T19:00:16.166 INFO:tasks.workunit.client.0.smithi027.stdout:./295/.snap/snap-subdir-test 2024-03-28T19:00:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:./272/.snap/snap-subdir-test 2024-03-28T19:00:16.205 INFO:tasks.workunit.client.0.smithi027.stdout:./100/.snap/snap-subdir-test 2024-03-28T19:00:16.224 INFO:tasks.workunit.client.0.smithi027.stdout:./134/.snap/snap-subdir-test 2024-03-28T19:00:16.242 INFO:tasks.workunit.client.0.smithi027.stdout:./200/.snap/snap-subdir-test 2024-03-28T19:00:16.260 INFO:tasks.workunit.client.0.smithi027.stdout:./115/.snap/snap-subdir-test 2024-03-28T19:00:16.278 INFO:tasks.workunit.client.0.smithi027.stdout:./284/.snap/snap-subdir-test 2024-03-28T19:00:16.294 INFO:tasks.workunit.client.0.smithi027.stdout:./280/.snap/snap-subdir-test 2024-03-28T19:00:16.314 INFO:tasks.workunit.client.0.smithi027.stdout:./116/.snap/snap-subdir-test 2024-03-28T19:00:16.331 INFO:tasks.workunit.client.0.smithi027.stdout:./245/.snap/snap-subdir-test 2024-03-28T19:00:16.349 INFO:tasks.workunit.client.0.smithi027.stdout:./195/.snap/snap-subdir-test 2024-03-28T19:00:16.368 INFO:tasks.workunit.client.0.smithi027.stdout:./138/.snap/snap-subdir-test 2024-03-28T19:00:16.387 INFO:tasks.workunit.client.0.smithi027.stdout:./135/.snap/snap-subdir-test 2024-03-28T19:00:16.407 INFO:tasks.workunit.client.0.smithi027.stdout:./117/.snap/snap-subdir-test 2024-03-28T19:00:16.423 INFO:tasks.workunit.client.0.smithi027.stdout:./241/.snap/snap-subdir-test 2024-03-28T19:00:16.442 INFO:tasks.workunit.client.0.smithi027.stdout:./143/.snap/snap-subdir-test 2024-03-28T19:00:16.462 INFO:tasks.workunit.client.0.smithi027.stdout:./180/.snap/snap-subdir-test 2024-03-28T19:00:16.481 INFO:tasks.workunit.client.0.smithi027.stdout:./126/.snap/snap-subdir-test 2024-03-28T19:00:16.501 INFO:tasks.workunit.client.0.smithi027.stdout:./161/.snap/snap-subdir-test 2024-03-28T19:00:16.520 INFO:tasks.workunit.client.0.smithi027.stdout:./204/.snap/snap-subdir-test 2024-03-28T19:00:16.538 INFO:tasks.workunit.client.0.smithi027.stdout:./176/.snap/snap-subdir-test 2024-03-28T19:00:16.555 INFO:tasks.workunit.client.0.smithi027.stdout:./255/.snap/snap-subdir-test 2024-03-28T19:00:16.572 INFO:tasks.workunit.client.0.smithi027.stdout:./215/.snap/snap-subdir-test 2024-03-28T19:00:16.591 INFO:tasks.workunit.client.0.smithi027.stdout:./111/.snap/snap-subdir-test 2024-03-28T19:00:16.610 INFO:tasks.workunit.client.0.smithi027.stdout:./238/.snap/snap-subdir-test 2024-03-28T19:00:16.631 INFO:tasks.workunit.client.0.smithi027.stdout:./191/.snap/snap-subdir-test 2024-03-28T19:00:16.651 INFO:tasks.workunit.client.0.smithi027.stdout:./184/.snap/snap-subdir-test 2024-03-28T19:00:16.669 INFO:tasks.workunit.client.0.smithi027.stdout:./192/.snap/snap-subdir-test 2024-03-28T19:00:16.689 INFO:tasks.workunit.client.0.smithi027.stdout:./103/.snap/snap-subdir-test 2024-03-28T19:00:16.707 INFO:tasks.workunit.client.0.smithi027.stdout:./128/.snap/snap-subdir-test 2024-03-28T19:00:16.723 INFO:tasks.workunit.client.0.smithi027.stdout:./229/.snap/snap-subdir-test 2024-03-28T19:00:16.742 INFO:tasks.workunit.client.0.smithi027.stdout:./144/.snap/snap-subdir-test 2024-03-28T19:00:16.761 INFO:tasks.workunit.client.0.smithi027.stdout:./146/.snap/snap-subdir-test 2024-03-28T19:00:16.778 INFO:tasks.workunit.client.0.smithi027.stdout:./227/.snap/snap-subdir-test 2024-03-28T19:00:16.799 INFO:tasks.workunit.client.0.smithi027.stdout:./278/.snap/snap-subdir-test 2024-03-28T19:00:16.839 INFO:tasks.workunit.client.0.smithi027.stdout:./141/.snap/snap-subdir-test 2024-03-28T19:00:16.857 INFO:tasks.workunit.client.0.smithi027.stdout:./234/.snap/snap-subdir-test 2024-03-28T19:00:16.876 INFO:tasks.workunit.client.0.smithi027.stdout:./225/.snap/snap-subdir-test 2024-03-28T19:00:16.893 INFO:tasks.workunit.client.0.smithi027.stdout:./182/.snap/snap-subdir-test 2024-03-28T19:00:16.911 INFO:tasks.workunit.client.0.smithi027.stdout:./107/.snap/snap-subdir-test 2024-03-28T19:00:16.930 INFO:tasks.workunit.client.0.smithi027.stdout:./178/.snap/snap-subdir-test 2024-03-28T19:00:16.946 INFO:tasks.workunit.client.0.smithi027.stdout:./266/.snap/snap-subdir-test 2024-03-28T19:00:16.963 INFO:tasks.workunit.client.0.smithi027.stdout:./240/.snap/snap-subdir-test 2024-03-28T19:00:16.981 INFO:tasks.workunit.client.0.smithi027.stdout:./156/.snap/snap-subdir-test 2024-03-28T19:00:16.995 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-03-28T19:00:16.995 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-03-28T19:00:16.996 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-03-28T19:00:16.997 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-03-28T19:00:16.998 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-03-28T19:00:16.999 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-03-28T19:00:17.000 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-03-28T19:00:17.001 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-03-28T19:00:17.001 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-03-28T19:00:17.001 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-03-28T19:00:17.001 INFO:tasks.workunit.client.0.smithi027.stdout:./131/.snap/snap-subdir-test 2024-03-28T19:00:17.019 INFO:tasks.workunit.client.0.smithi027.stdout:./123/.snap/snap-subdir-test 2024-03-28T19:00:17.037 INFO:tasks.workunit.client.0.smithi027.stdout:./160/.snap/snap-subdir-test 2024-03-28T19:00:17.056 INFO:tasks.workunit.client.0.smithi027.stdout:./165/.snap/snap-subdir-test 2024-03-28T19:00:17.075 INFO:tasks.workunit.client.0.smithi027.stdout:./101/.snap/snap-subdir-test 2024-03-28T19:00:17.093 INFO:tasks.workunit.client.0.smithi027.stdout:./148/.snap/snap-subdir-test 2024-03-28T19:00:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:./168/.snap/snap-subdir-test 2024-03-28T19:00:17.131 INFO:tasks.workunit.client.0.smithi027.stdout:./179/.snap/snap-subdir-test 2024-03-28T19:00:17.148 INFO:tasks.workunit.client.0.smithi027.stdout:./257/.snap/snap-subdir-test 2024-03-28T19:00:17.166 INFO:tasks.workunit.client.0.smithi027.stdout:./252/.snap/snap-subdir-test 2024-03-28T19:00:17.184 INFO:tasks.workunit.client.0.smithi027.stdout:./132/.snap/snap-subdir-test 2024-03-28T19:00:17.202 INFO:tasks.workunit.client.0.smithi027.stdout:./236/.snap/snap-subdir-test 2024-03-28T19:00:17.219 INFO:tasks.workunit.client.0.smithi027.stdout:./211/.snap/snap-subdir-test 2024-03-28T19:00:17.239 INFO:tasks.workunit.client.0.smithi027.stdout:./199/.snap/snap-subdir-test 2024-03-28T19:00:17.255 INFO:tasks.workunit.client.0.smithi027.stdout:./205/.snap/snap-subdir-test 2024-03-28T19:00:17.275 INFO:tasks.workunit.client.0.smithi027.stdout:./163/.snap/snap-subdir-test 2024-03-28T19:00:17.291 INFO:tasks.workunit.client.0.smithi027.stdout:./261/.snap/snap-subdir-test 2024-03-28T19:00:17.309 INFO:tasks.workunit.client.0.smithi027.stdout:./201/.snap/snap-subdir-test 2024-03-28T19:00:17.326 INFO:tasks.workunit.client.0.smithi027.stdout:./282/.snap/snap-subdir-test 2024-03-28T19:00:17.341 INFO:tasks.workunit.client.0.smithi027.stdout:./296/.snap/snap-subdir-test 2024-03-28T19:00:17.359 INFO:tasks.workunit.client.0.smithi027.stdout:./217/.snap/snap-subdir-test 2024-03-28T19:00:17.379 INFO:tasks.workunit.client.0.smithi027.stdout:./153/.snap/snap-subdir-test 2024-03-28T19:00:17.397 INFO:tasks.workunit.client.0.smithi027.stdout:./206/.snap/snap-subdir-test 2024-03-28T19:00:17.416 INFO:tasks.workunit.client.0.smithi027.stdout:./122/.snap/snap-subdir-test 2024-03-28T19:00:17.434 INFO:tasks.workunit.client.0.smithi027.stdout:./162/.snap/snap-subdir-test 2024-03-28T19:00:17.453 INFO:tasks.workunit.client.0.smithi027.stdout:./197/.snap/snap-subdir-test 2024-03-28T19:00:17.470 INFO:tasks.workunit.client.0.smithi027.stdout:./216/.snap/snap-subdir-test 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-03-28T19:00:19.070 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-03-28T19:00:19.071 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-03-28T19:00:19.072 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-03-28T19:00:19.073 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-03-28T19:00:19.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-03-28T19:00:19.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-03-28T19:00:19.076 INFO:tasks.workunit.client.0.smithi027.stdout:./254/.snap/snap-subdir-test 2024-03-28T19:00:19.092 INFO:tasks.workunit.client.0.smithi027.stdout:./113/.snap/snap-subdir-test 2024-03-28T19:00:19.112 INFO:tasks.workunit.client.0.smithi027.stdout:./152/.snap/snap-subdir-test 2024-03-28T19:00:19.130 INFO:tasks.workunit.client.0.smithi027.stdout:./121/.snap/snap-subdir-test 2024-03-28T19:00:19.148 INFO:tasks.workunit.client.0.smithi027.stdout:./118/.snap/snap-subdir-test 2024-03-28T19:00:19.169 INFO:tasks.workunit.client.0.smithi027.stdout:./196/.snap/snap-subdir-test 2024-03-28T19:00:19.188 INFO:tasks.workunit.client.0.smithi027.stdout:./124/.snap/snap-subdir-test 2024-03-28T19:00:19.205 INFO:tasks.workunit.client.0.smithi027.stdout:./239/.snap/snap-subdir-test 2024-03-28T19:00:19.223 INFO:tasks.workunit.client.0.smithi027.stdout:./188/.snap/snap-subdir-test 2024-03-28T19:00:19.237 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-03-28T19:00:19.237 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T19:00:19.238 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T19:00:19.240 INFO:tasks.workunit.client.0.smithi027.stdout:./287/.snap/snap-subdir-test 2024-03-28T19:00:19.248 INFO:tasks.ceph.mds.d:Started 2024-03-28T19:00:19.248 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:19.257 INFO:tasks.workunit.client.0.smithi027.stdout:./275/.snap/snap-subdir-test 2024-03-28T19:00:19.277 INFO:tasks.workunit.client.0.smithi027.stdout:./259/.snap/snap-subdir-test 2024-03-28T19:00:19.296 INFO:tasks.workunit.client.0.smithi027.stdout:./224/.snap/snap-subdir-test 2024-03-28T19:00:19.314 INFO:tasks.workunit.client.0.smithi027.stdout:./264/.snap/snap-subdir-test 2024-03-28T19:00:19.335 INFO:tasks.workunit.client.0.smithi027.stdout:./145/.snap/snap-subdir-test 2024-03-28T19:00:19.357 INFO:tasks.workunit.client.0.smithi027.stdout:./279/.snap/snap-subdir-test 2024-03-28T19:00:19.372 INFO:tasks.workunit.client.0.smithi027.stdout:./263/.snap/snap-subdir-test 2024-03-28T19:00:19.393 INFO:tasks.workunit.client.0.smithi027.stdout:./130/.snap/snap-subdir-test 2024-03-28T19:00:19.415 INFO:tasks.workunit.client.0.smithi027.stdout:./193/.snap/snap-subdir-test 2024-03-28T19:00:19.436 INFO:tasks.workunit.client.0.smithi027.stdout:./223/.snap/snap-subdir-test 2024-03-28T19:00:19.436 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T19:00:19.459 INFO:tasks.workunit.client.0.smithi027.stdout:./207/.snap/snap-subdir-test 2024-03-28T19:00:19.499 INFO:tasks.workunit.client.0.smithi027.stdout:./276/.snap/snap-subdir-test 2024-03-28T19:00:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.519+0000 7f3119400640 1 Processor -- start 2024-03-28T19:00:19.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.520+0000 7f3119400640 1 -- start start 2024-03-28T19:00:19.519 INFO:tasks.workunit.client.0.smithi027.stdout:./271/.snap/snap-subdir-test 2024-03-28T19:00:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.521+0000 7f3119400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3114152d40 0x7f31140c9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.521+0000 7f3119400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3114153120 con 0x7f311406d1f0 2024-03-28T19:00:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.521+0000 7f3119400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31140c9640 con 0x7f31141556e0 2024-03-28T19:00:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.521+0000 7f3119400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31141596a0 con 0x7f3114152d40 2024-03-28T19:00:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.521+0000 7f3111e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f311406d1f0 0x7f31140c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44680/0 (socket says 172.21.15.27:44680) 2024-03-28T19:00:19.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.521+0000 7f3111e00640 1 -- 172.21.15.27:0/2826693369 learned_addr learned my addr 172.21.15.27:0/2826693369 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.522+0000 7f3112800640 1 --2- 172.21.15.27:0/2826693369 >> v2:172.21.15.155:3300/0 conn(0x7f3114152d40 0x7f31140c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.522+0000 7f3111400640 1 -- 172.21.15.27:0/2826693369 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192259918 0 0) 0x7f3114153120 con 0x7f311406d1f0 2024-03-28T19:00:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.522+0000 7f3111400640 1 -- 172.21.15.27:0/2826693369 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f8003520 con 0x7f311406d1f0 2024-03-28T19:00:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.522+0000 7f3112800640 1 -- 172.21.15.27:0/2826693369 >> v1:172.21.15.155:6789/0 conn(0x7f31141556e0 legacy=0x7f3114155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.522+0000 7f3112800640 1 -- 172.21.15.27:0/2826693369 >> v1:172.21.15.27:6789/0 conn(0x7f311406d1f0 legacy=0x7f31140c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:19.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.522+0000 7f3112800640 1 -- 172.21.15.27:0/2826693369 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3114158d70 con 0x7f3114152d40 2024-03-28T19:00:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.524+0000 7f3112800640 1 --2- 172.21.15.27:0/2826693369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3114152d40 0x7f31140c9060 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f31040097a0 tx=0x7f310402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f52fa45240f6161 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.524+0000 7f3111400640 1 -- 172.21.15.27:0/2826693369 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104034070 con 0x7f3114152d40 2024-03-28T19:00:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.524+0000 7f3111400640 1 -- 172.21.15.27:0/2826693369 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104030610 con 0x7f3114152d40 2024-03-28T19:00:19.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.524+0000 7f3111400640 1 -- 172.21.15.27:0/2826693369 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040309c0 con 0x7f3114152d40 2024-03-28T19:00:19.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.525+0000 7f3119400640 1 -- 172.21.15.27:0/2826693369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3114152d40 msgr2=0x7f31140c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.525+0000 7f3119400640 1 --2- 172.21.15.27:0/2826693369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3114152d40 0x7f31140c9060 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f31040097a0 tx=0x7f310402b960 comp rx=0 tx=0).stop 2024-03-28T19:00:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.525+0000 7f3119400640 1 -- 172.21.15.27:0/2826693369 shutdown_connections 2024-03-28T19:00:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.525+0000 7f3119400640 1 --2- 172.21.15.27:0/2826693369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3114152d40 0x7f31140c9060 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.525+0000 7f3119400640 1 -- 172.21.15.27:0/2826693369 >> 172.21.15.27:0/2826693369 conn(0x7f3114066fb0 msgr2=0x7f31140673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.525+0000 7f3119400640 1 -- 172.21.15.27:0/2826693369 shutdown_connections 2024-03-28T19:00:19.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.525+0000 7f3119400640 1 -- 172.21.15.27:0/2826693369 wait complete. 2024-03-28T19:00:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.528+0000 7f3119400640 1 Processor -- start 2024-03-28T19:00:19.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.528+0000 7f3119400640 1 -- start start 2024-03-28T19:00:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3119400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311406d1f0 0x7f31141517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3119400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3114152d40 0x7f31140c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3119400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31141556e0 0x7f31141eb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3119400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3114159b30 con 0x7f311406d1f0 2024-03-28T19:00:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3119400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3114159330 con 0x7f31141556e0 2024-03-28T19:00:19.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3119400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31140c5fd0 con 0x7f3114152d40 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3112800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311406d1f0 0x7f31141517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3112800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311406d1f0 0x7f31141517e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47054/0 (socket says 172.21.15.27:47054) 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.530+0000 7f3112800640 1 -- 172.21.15.27:0/587885959 learned_addr learned my addr 172.21.15.27:0/587885959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.531+0000 7f3111e00640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3114152d40 0x7f31140c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.531+0000 7f3113200640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31141556e0 0x7f31141eb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.531+0000 7f3112800640 1 -- 172.21.15.27:0/587885959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3114152d40 msgr2=0x7f31140c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.531+0000 7f3112800640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3114152d40 0x7f31140c4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.531+0000 7f3112800640 1 -- 172.21.15.27:0/587885959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31141556e0 msgr2=0x7f31141eb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.531+0000 7f3112800640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31141556e0 0x7f31141eb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.531+0000 7f3112800640 1 -- 172.21.15.27:0/587885959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31141eebf0 con 0x7f311406d1f0 2024-03-28T19:00:19.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.532+0000 7f3112800640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311406d1f0 0x7f31141517e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f31040097a0 tx=0x7f31040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:19.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.532+0000 7f310aa00640 1 -- 172.21.15.27:0/587885959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104034040 con 0x7f311406d1f0 2024-03-28T19:00:19.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.533+0000 7f310aa00640 1 -- 172.21.15.27:0/587885959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104030410 con 0x7f311406d1f0 2024-03-28T19:00:19.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.533+0000 7f310aa00640 1 -- 172.21.15.27:0/587885959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104004230 con 0x7f311406d1f0 2024-03-28T19:00:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.534+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31141eeed0 con 0x7f311406d1f0 2024-03-28T19:00:19.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.534+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31141ef490 con 0x7f311406d1f0 2024-03-28T19:00:19.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.536+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311404f400 con 0x7f311406d1f0 2024-03-28T19:00:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.536+0000 7f310aa00640 1 -- 172.21.15.27:0/587885959 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f310404c060 con 0x7f311406d1f0 2024-03-28T19:00:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.537+0000 7f310aa00640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30f406bca0 0x7f30f406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.538+0000 7f310aa00640 1 -- 172.21.15.27:0/587885959 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7067+0+0 (secure 0 0 0) 0x7f31040acbf0 con 0x7f311406d1f0 2024-03-28T19:00:19.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.538+0000 7f3111e00640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30f406bca0 0x7f30f406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:19.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.539+0000 7f3111e00640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30f406bca0 0x7f30f406e160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f31140c3de0 tx=0x7f3100009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:19.541 INFO:tasks.workunit.client.0.smithi027.stdout:./251/.snap/snap-subdir-test 2024-03-28T19:00:19.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.545+0000 7f310aa00640 1 -- 172.21.15.27:0/587885959 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f310407b320 con 0x7f311406d1f0 2024-03-28T19:00:19.561 INFO:tasks.workunit.client.0.smithi027.stdout:./175/.snap/snap-subdir-test 2024-03-28T19:00:19.578 INFO:tasks.workunit.client.0.smithi027.stdout:./222/.snap/snap-subdir-test 2024-03-28T19:00:19.595 INFO:tasks.workunit.client.0.smithi027.stdout:./220/.snap/snap-subdir-test 2024-03-28T19:00:19.615 INFO:tasks.workunit.client.0.smithi027.stdout:./203/.snap/snap-subdir-test 2024-03-28T19:00:19.636 INFO:tasks.workunit.client.0.smithi027.stdout:./181/.snap/snap-subdir-test 2024-03-28T19:00:19.656 INFO:tasks.workunit.client.0.smithi027.stdout:./286/.snap/snap-subdir-test 2024-03-28T19:00:19.674 INFO:tasks.workunit.client.0.smithi027.stdout:./127/.snap/snap-subdir-test 2024-03-28T19:00:19.692 INFO:tasks.workunit.client.0.smithi027.stdout:./108/.snap/snap-subdir-test 2024-03-28T19:00:19.710 INFO:tasks.workunit.client.0.smithi027.stdout:./183/.snap/snap-subdir-test 2024-03-28T19:00:19.726 INFO:tasks.workunit.client.0.smithi027.stdout:./294/.snap/snap-subdir-test 2024-03-28T19:00:19.743 INFO:tasks.workunit.client.0.smithi027.stdout:./235/.snap/snap-subdir-test 2024-03-28T19:00:19.763 INFO:tasks.workunit.client.0.smithi027.stdout:./177/.snap/snap-subdir-test 2024-03-28T19:00:19.782 INFO:tasks.workunit.client.0.smithi027.stdout:./169/.snap/snap-subdir-test 2024-03-28T19:00:19.801 INFO:tasks.workunit.client.0.smithi027.stdout:./212/.snap/snap-subdir-test 2024-03-28T19:00:19.820 INFO:tasks.workunit.client.0.smithi027.stdout:./154/.snap/snap-subdir-test 2024-03-28T19:00:19.839 INFO:tasks.workunit.client.0.smithi027.stdout:./171/.snap/snap-subdir-test 2024-03-28T19:00:19.858 INFO:tasks.workunit.client.0.smithi027.stdout:./139/.snap/snap-subdir-test 2024-03-28T19:00:19.877 INFO:tasks.workunit.client.0.smithi027.stdout:./102/.snap/snap-subdir-test 2024-03-28T19:00:19.895 INFO:tasks.workunit.client.0.smithi027.stdout:./129/.snap/snap-subdir-test 2024-03-28T19:00:19.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.915+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31140c8e40 con 0x7f311406d1f0 2024-03-28T19:00:19.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.916+0000 7f310aa00640 1 -- 172.21.15.27:0/587885959 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f310407ac20 con 0x7f311406d1f0 2024-03-28T19:00:19.915 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:19.916 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:19.916 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 42 2024-03-28T19:00:19.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.919+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30f406bca0 msgr2=0x7f30f406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:19.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.919+0000 7f3119400640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30f406bca0 0x7f30f406e160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f31140c3de0 tx=0x7f3100009040 comp rx=0 tx=0).stop 2024-03-28T19:00:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.919+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311406d1f0 msgr2=0x7f31141517e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.919+0000 7f3119400640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311406d1f0 0x7f31141517e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f31040097a0 tx=0x7f31040046c0 comp rx=0 tx=0).stop 2024-03-28T19:00:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.920+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 shutdown_connections 2024-03-28T19:00:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.920+0000 7f3119400640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30f406bca0 0x7f30f406e160 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.920+0000 7f3119400640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31141556e0 0x7f31141eb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.920+0000 7f3119400640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3114152d40 0x7f31140c4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.920+0000 7f3119400640 1 --2- 172.21.15.27:0/587885959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f311406d1f0 0x7f31141517e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:19.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.920+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 >> 172.21.15.27:0/587885959 conn(0x7f3114066fb0 msgr2=0x7f3114154fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:19.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.921+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 shutdown_connections 2024-03-28T19:00:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:19.921+0000 7f3119400640 1 -- 172.21.15.27:0/587885959 wait complete. 2024-03-28T19:00:19.941 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-03-28T19:00:21.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-03-28T19:00:21.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-03-28T19:00:21.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-03-28T19:00:21.074 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-03-28T19:00:21.075 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-03-28T19:00:21.076 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-03-28T19:00:21.077 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-03-28T19:00:21.078 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-03-28T19:00:21.079 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-03-28T19:00:21.080 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-03-28T19:00:21.080 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-03-28T19:00:21.080 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-03-28T19:00:21.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.200+0000 7f6e1d600640 1 Processor -- start 2024-03-28T19:00:22.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.200+0000 7f6e1d600640 1 -- start start 2024-03-28T19:00:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.201+0000 7f6e1d600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6e18153ed0 0x7f6e1806ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.201+0000 7f6e1d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e1806eb80 0x7f6e1815a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.201+0000 7f6e1d600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e180c3830 con 0x7f6e1806e6f0 2024-03-28T19:00:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.201+0000 7f6e1d600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e180c6990 con 0x7f6e1806eb80 2024-03-28T19:00:22.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.201+0000 7f6e1d600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1806d1f0 con 0x7f6e18153ed0 2024-03-28T19:00:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.202+0000 7f6e17400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e1806eb80 0x7f6e1815a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.202+0000 7f6e16000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6e1806e6f0 0x7f6e18159bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44686/0 (socket says 172.21.15.27:44686) 2024-03-28T19:00:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.202+0000 7f6e16000640 1 -- 172.21.15.27:0/2860940732 learned_addr learned my addr 172.21.15.27:0/2860940732 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.202+0000 7f6e16a00640 1 --2- 172.21.15.27:0/2860940732 >> v2:172.21.15.155:3300/0 conn(0x7f6e18153ed0 0x7f6e1806ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.203+0000 7f6e17400640 1 -- 172.21.15.27:0/2860940732 >> v2:172.21.15.155:3300/0 conn(0x7f6e18153ed0 msgr2=0x7f6e1806ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.203+0000 7f6e17400640 1 --2- 172.21.15.27:0/2860940732 >> v2:172.21.15.155:3300/0 conn(0x7f6e18153ed0 0x7f6e1806ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.203+0000 7f6e17400640 1 -- 172.21.15.27:0/2860940732 >> v1:172.21.15.27:6789/0 conn(0x7f6e1806e6f0 legacy=0x7f6e18159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:22.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.203+0000 7f6e17400640 1 -- 172.21.15.27:0/2860940732 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1806ef60 con 0x7f6e1806eb80 2024-03-28T19:00:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.204+0000 7f6e17400640 1 --2- 172.21.15.27:0/2860940732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806eb80 0x7f6e1815a2d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6e180c5ca0 tx=0x7f6e1002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94dc376f380878f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.205+0000 7f6e15600640 1 -- 172.21.15.27:0/2860940732 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e10039070 con 0x7f6e1806eb80 2024-03-28T19:00:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.205+0000 7f6e15600640 1 -- 172.21.15.27:0/2860940732 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1002cc90 con 0x7f6e1806eb80 2024-03-28T19:00:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.205+0000 7f6e15600640 1 -- 172.21.15.27:0/2860940732 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e10034660 con 0x7f6e1806eb80 2024-03-28T19:00:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.205+0000 7f6e1d600640 1 -- 172.21.15.27:0/2860940732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806eb80 msgr2=0x7f6e1815a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.205+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2860940732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806eb80 0x7f6e1815a2d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6e180c5ca0 tx=0x7f6e1002bee0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.206+0000 7f6e1d600640 1 -- 172.21.15.27:0/2860940732 shutdown_connections 2024-03-28T19:00:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.206+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2860940732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806eb80 0x7f6e1815a2d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.206+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2860940732 >> v2:172.21.15.155:3300/0 conn(0x7f6e18153ed0 0x7f6e1806ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.206+0000 7f6e1d600640 1 -- 172.21.15.27:0/2860940732 >> 172.21.15.27:0/2860940732 conn(0x7f6e1806b9d0 msgr2=0x7f6e180c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.206+0000 7f6e1d600640 1 -- 172.21.15.27:0/2860940732 shutdown_connections 2024-03-28T19:00:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.206+0000 7f6e1d600640 1 -- 172.21.15.27:0/2860940732 wait complete. 2024-03-28T19:00:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.208+0000 7f6e1d600640 1 Processor -- start 2024-03-28T19:00:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.208+0000 7f6e1d600640 1 -- start start 2024-03-28T19:00:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.209+0000 7f6e1d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806e6f0 0x7f6e181584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.209+0000 7f6e1d600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e1806eb80 0x7f6e181ef700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.209+0000 7f6e1d600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e18153ed0 0x7f6e181efc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.209+0000 7f6e1d600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180c3db0 con 0x7f6e1806e6f0 2024-03-28T19:00:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.209+0000 7f6e1d600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e180c3ad0 con 0x7f6e1806eb80 2024-03-28T19:00:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.209+0000 7f6e1d600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1815a810 con 0x7f6e18153ed0 2024-03-28T19:00:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.210+0000 7f6e16a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806e6f0 0x7f6e181584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.210+0000 7f6e16a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806e6f0 0x7f6e181584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47074/0 (socket says 172.21.15.27:47074) 2024-03-28T19:00:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.210+0000 7f6e16a00640 1 -- 172.21.15.27:0/2255120422 learned_addr learned my addr 172.21.15.27:0/2255120422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.210+0000 7f6e16000640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e1806eb80 0x7f6e181ef700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.210+0000 7f6e17400640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e18153ed0 0x7f6e181efc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.211+0000 7f6e17400640 1 -- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e1806eb80 msgr2=0x7f6e181ef700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.211+0000 7f6e17400640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e1806eb80 0x7f6e181ef700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.211+0000 7f6e17400640 1 -- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806e6f0 msgr2=0x7f6e181584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.211+0000 7f6e17400640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806e6f0 0x7f6e181584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.211+0000 7f6e17400640 1 -- 172.21.15.27:0/2255120422 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e181e44b0 con 0x7f6e18153ed0 2024-03-28T19:00:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.212+0000 7f6e17400640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e18153ed0 0x7f6e181efc40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e10033000 tx=0x7f6e10004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.212+0000 7f6e0ea00640 1 -- 172.21.15.27:0/2255120422 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e10039050 con 0x7f6e18153ed0 2024-03-28T19:00:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.213+0000 7f6e0ea00640 1 -- 172.21.15.27:0/2255120422 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e10045070 con 0x7f6e18153ed0 2024-03-28T19:00:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.213+0000 7f6e0ea00640 1 -- 172.21.15.27:0/2255120422 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1003e850 con 0x7f6e18153ed0 2024-03-28T19:00:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.213+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e181570b0 con 0x7f6e18153ed0 2024-03-28T19:00:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.213+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e181f30f0 con 0x7f6e18153ed0 2024-03-28T19:00:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.214+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1804f400 con 0x7f6e18153ed0 2024-03-28T19:00:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.216+0000 7f6e0ea00640 1 -- 172.21.15.27:0/2255120422 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e1003e9f0 con 0x7f6e18153ed0 2024-03-28T19:00:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.217+0000 7f6e0ea00640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6df806bbd0 0x7f6df806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.217+0000 7f6e0ea00640 1 -- 172.21.15.27:0/2255120422 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7067+0+0 (secure 0 0 0) 0x7f6e100ae5d0 con 0x7f6e18153ed0 2024-03-28T19:00:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.219+0000 7f6e16a00640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6df806bbd0 0x7f6df806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:22.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.223+0000 7f6e16a00640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6df806bbd0 0x7f6df806e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6e1815e040 tx=0x7f6e04008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.233+0000 7f6e0ea00640 1 -- 172.21.15.27:0/2255120422 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e1007c5b0 con 0x7f6e18153ed0 2024-03-28T19:00:22.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.629+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e180ac190 con 0x7f6e18153ed0 2024-03-28T19:00:22.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.631+0000 7f6e0ea00640 1 -- 172.21.15.27:0/2255120422 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6248 (secure 0 0 0) 0x7f6e10080650 con 0x7f6e18153ed0 2024-03-28T19:00:22.630 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:22.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:22.631 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 43 2024-03-28T19:00:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.634+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6df806bbd0 msgr2=0x7f6df806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.634+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6df806bbd0 0x7f6df806e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6e1815e040 tx=0x7f6e04008040 comp rx=0 tx=0).stop 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.634+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e18153ed0 msgr2=0x7f6e181efc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.634+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e18153ed0 0x7f6e181efc40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e10033000 tx=0x7f6e10004a40 comp rx=0 tx=0).stop 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.635+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 shutdown_connections 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.635+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6df806bbd0 0x7f6df806e090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.635+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e18153ed0 0x7f6e181efc40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.635+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e1806eb80 0x7f6e181ef700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.635+0000 7f6e1d600640 1 --2- 172.21.15.27:0/2255120422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e1806e6f0 0x7f6e181584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.635+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 >> 172.21.15.27:0/2255120422 conn(0x7f6e1806b9d0 msgr2=0x7f6e18070ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.635+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 shutdown_connections 2024-03-28T19:00:22.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:22.635+0000 7f6e1d600640 1 -- 172.21.15.27:0/2255120422 wait complete. 2024-03-28T19:00:22.657 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-03-28T19:00:22.657 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before thrashing 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-03-28T19:00:23.252 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-03-28T19:00:23.253 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-03-28T19:00:23.254 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-03-28T19:00:23.255 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-03-28T19:00:23.256 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-03-28T19:00:23.257 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-03-28T19:00:23.257 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-03-28T19:00:23.257 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-03-28T19:00:23.257 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-03-28T19:00:23.257 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-03-28T19:00:23.257 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-03-28T19:00:23.257 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-03-28T19:00:25.153 INFO:tasks.workunit.client.0.smithi027.stdout:Delete all the files and directories ... 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-03-28T19:00:25.426 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-03-28T19:00:25.427 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-03-28T19:00:25.428 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-03-28T19:00:25.429 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-03-28T19:00:25.430 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-03-28T19:00:25.431 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-03-28T19:00:25.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:00:25.780 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:00:28.342 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-03-28T19:00:28.342 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-03-28T19:00:28.342 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-03-28T19:00:28.342 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-03-28T19:00:28.342 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-03-28T19:00:28.342 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-03-28T19:00:28.342 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-03-28T19:00:28.343 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-03-28T19:00:28.344 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-03-28T19:00:28.345 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-03-28T19:00:28.346 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-03-28T19:00:28.347 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-03-28T19:00:28.348 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-03-28T19:00:28.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:28.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.924+0000 7f14ed000640 1 Processor -- start 2024-03-28T19:00:28.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.924+0000 7f14ed000640 1 -- start start 2024-03-28T19:00:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14ed000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e80c5320 0x7f14e806e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14ed000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14e8156500 0x7f14e80c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14ed000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e8159c50 con 0x7f14e8156120 2024-03-28T19:00:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14ed000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e80c4d70 con 0x7f14e8156500 2024-03-28T19:00:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14ed000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80c1030 con 0x7f14e80c5320 2024-03-28T19:00:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14e7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14e8156500 0x7f14e80c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14e7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f14e8156120 0x7f14e80c0920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41912/0 (socket says 172.21.15.27:41912) 2024-03-28T19:00:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14e7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14e8156500 0x7f14e80c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56026/0 (socket says 172.21.15.27:56026) 2024-03-28T19:00:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.926+0000 7f14e7400640 1 -- 172.21.15.27:0/1420986141 learned_addr learned my addr 172.21.15.27:0/1420986141 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:28.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.927+0000 7f14e6a00640 1 --2- 172.21.15.27:0/1420986141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e80c5320 0x7f14e806e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:28.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.927+0000 7f14e7e00640 1 -- 172.21.15.27:0/1420986141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e80c5320 msgr2=0x7f14e806e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:28.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.927+0000 7f14e7e00640 1 --2- 172.21.15.27:0/1420986141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e80c5320 0x7f14e806e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:28.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.927+0000 7f14e7e00640 1 -- 172.21.15.27:0/1420986141 >> v1:172.21.15.27:6789/0 conn(0x7f14e8156120 legacy=0x7f14e80c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:28.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.927+0000 7f14e7e00640 1 -- 172.21.15.27:0/1420986141 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80c4a40 con 0x7f14e8156500 2024-03-28T19:00:28.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.927+0000 7f14e6000640 1 -- 172.21.15.27:0/1420986141 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994428689 0 0) 0x7f14e8159c50 con 0x7f14e8156120 2024-03-28T19:00:28.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.929+0000 7f14e7e00640 1 --2- 172.21.15.27:0/1420986141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e8156500 0x7f14e80c6730 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009dc0 tx=0x7f14e002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba91a71676d1ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:28.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.929+0000 7f14e6000640 1 -- 172.21.15.27:0/1420986141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0039070 con 0x7f14e8156500 2024-03-28T19:00:28.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.929+0000 7f14e6000640 1 -- 172.21.15.27:0/1420986141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e002cb30 con 0x7f14e8156500 2024-03-28T19:00:28.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.929+0000 7f14e6000640 1 -- 172.21.15.27:0/1420986141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0034440 con 0x7f14e8156500 2024-03-28T19:00:28.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.930+0000 7f14ed000640 1 -- 172.21.15.27:0/1420986141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e8156500 msgr2=0x7f14e80c6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:28.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.930+0000 7f14ed000640 1 --2- 172.21.15.27:0/1420986141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e8156500 0x7f14e80c6730 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009dc0 tx=0x7f14e002bd80 comp rx=0 tx=0).stop 2024-03-28T19:00:28.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.930+0000 7f14ed000640 1 -- 172.21.15.27:0/1420986141 shutdown_connections 2024-03-28T19:00:28.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.930+0000 7f14ed000640 1 --2- 172.21.15.27:0/1420986141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e8156500 0x7f14e80c6730 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:28.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.930+0000 7f14ed000640 1 --2- 172.21.15.27:0/1420986141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e80c5320 0x7f14e806e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:28.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.930+0000 7f14ed000640 1 -- 172.21.15.27:0/1420986141 >> 172.21.15.27:0/1420986141 conn(0x7f14e806b0c0 msgr2=0x7f14e806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:28.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.931+0000 7f14ed000640 1 -- 172.21.15.27:0/1420986141 shutdown_connections 2024-03-28T19:00:28.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.931+0000 7f14ed000640 1 -- 172.21.15.27:0/1420986141 wait complete. 2024-03-28T19:00:28.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.932+0000 7f14ed000640 1 Processor -- start 2024-03-28T19:00:28.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.933+0000 7f14ed000640 1 -- start start 2024-03-28T19:00:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.934+0000 7f14ed000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e80c5320 0x7f14e80a9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.934+0000 7f14ed000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e8156120 0x7f14e80c1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:28.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.934+0000 7f14ed000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e8156500 0x7f14e8149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:28.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.934+0000 7f14ed000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80c73d0 con 0x7f14e80c5320 2024-03-28T19:00:28.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.934+0000 7f14ed000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80c7250 con 0x7f14e8156500 2024-03-28T19:00:28.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.934+0000 7f14ed000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80c29a0 con 0x7f14e8156120 2024-03-28T19:00:28.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.934+0000 7f14e7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e80c5320 0x7f14e80a9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:28.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.935+0000 7f14e7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e80c5320 0x7f14e80a9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56036/0 (socket says 172.21.15.27:56036) 2024-03-28T19:00:28.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.935+0000 7f14e7400640 1 -- 172.21.15.27:0/749318675 learned_addr learned my addr 172.21.15.27:0/749318675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:28.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.935+0000 7f14e6a00640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e8156120 0x7f14e80c1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.935+0000 7f14e7400640 1 -- 172.21.15.27:0/749318675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e8156120 msgr2=0x7f14e80c1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.935+0000 7f14e7400640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e8156120 0x7f14e80c1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.935+0000 7f14e7400640 1 -- 172.21.15.27:0/749318675 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e8156500 msgr2=0x7f14e8149ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T19:00:28.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.935+0000 7f14e7400640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e8156500 0x7f14e8149ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.935+0000 7f14e7400640 1 -- 172.21.15.27:0/749318675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80a9a80 con 0x7f14e80c5320 2024-03-28T19:00:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.936+0000 7f14e7400640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e80c5320 0x7f14e80a9030 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f14d800c4d0 tx=0x7f14d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.936+0000 7f14df400640 1 -- 172.21.15.27:0/749318675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8018070 con 0x7f14e80c5320 2024-03-28T19:00:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.937+0000 7f14df400640 1 -- 172.21.15.27:0/749318675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d80075a0 con 0x7f14e80c5320 2024-03-28T19:00:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.937+0000 7f14df400640 1 -- 172.21.15.27:0/749318675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8004a20 con 0x7f14e80c5320 2024-03-28T19:00:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.937+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e814a8a0 con 0x7f14e80c5320 2024-03-28T19:00:28.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.937+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e80a9cf0 con 0x7f14e80c5320 2024-03-28T19:00:28.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.940+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e8067dd0 con 0x7f14e80c5320 2024-03-28T19:00:28.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.940+0000 7f14df400640 1 -- 172.21.15.27:0/749318675 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f14d801b070 con 0x7f14e80c5320 2024-03-28T19:00:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.942+0000 7f14df400640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14c806bc20 0x7f14c806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.942+0000 7f14df400640 1 -- 172.21.15.27:0/749318675 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7731+0+0 (secure 0 0 0) 0x7f14d808a540 con 0x7f14e80c5320 2024-03-28T19:00:28.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.943+0000 7f14e6a00640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14c806bc20 0x7f14c806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:28.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.944+0000 7f14e6a00640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14c806bc20 0x7f14c806e0e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f14e814dd30 tx=0x7f14d4009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:28.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:28.947+0000 7f14df400640 1 -- 172.21.15.27:0/749318675 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f14d8058260 con 0x7f14e80c5320 2024-03-28T19:00:29.030 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/19' 2024-03-28T19:00:29.030 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/15' 2024-03-28T19:00:29.030 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/31' 2024-03-28T19:00:29.030 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/36' 2024-03-28T19:00:29.030 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/20' 2024-03-28T19:00:29.030 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/12' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/39' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/10' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/34' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/38' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/11' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/16' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/32' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/30' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/14' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/18' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/13' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/33' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/17' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/35' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed './100/37' 2024-03-28T19:00:29.031 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './100' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/19' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/15' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/31' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/36' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/20' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/12' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/39' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/10' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/34' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/38' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/11' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/16' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/32' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/30' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/14' 2024-03-28T19:00:29.032 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/18' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/13' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/33' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/17' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/35' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './101/37' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './101' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/19' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/15' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/31' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/36' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/20' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/12' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/39' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/10' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/34' 2024-03-28T19:00:29.033 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/38' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/11' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/16' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/32' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/30' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/14' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/18' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/13' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/33' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/17' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/35' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './102/37' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './102' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/19' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/15' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/31' 2024-03-28T19:00:29.034 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/36' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/20' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/12' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/39' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/10' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/34' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/38' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/11' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/16' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/32' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/30' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/14' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/18' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/13' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/33' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/17' 2024-03-28T19:00:29.035 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/35' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './103/37' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './103' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/19' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/15' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/31' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/36' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/20' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/12' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/39' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/10' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/34' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/38' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/11' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/16' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/32' 2024-03-28T19:00:29.036 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/30' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/14' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/18' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/13' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/33' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/17' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/35' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './104/37' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './104' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/19' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/15' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/31' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/36' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/20' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/12' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/39' 2024-03-28T19:00:29.037 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/10' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/34' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/38' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/11' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/16' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/32' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/30' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/14' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/18' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/13' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/33' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/17' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/35' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './105/37' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './105' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/19' 2024-03-28T19:00:29.038 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/15' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/31' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/36' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/20' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/12' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/39' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/10' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/34' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/38' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/11' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/16' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/32' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/30' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/14' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/18' 2024-03-28T19:00:29.039 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/13' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/33' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/17' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/35' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './106/37' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './106' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/19' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/15' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/31' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/36' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/20' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/12' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/39' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/10' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/34' 2024-03-28T19:00:29.040 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/38' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/11' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/16' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/32' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/30' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/14' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/18' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/13' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/33' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/17' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/35' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './107/37' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './107' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/19' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/15' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/31' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/36' 2024-03-28T19:00:29.041 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/20' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/12' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/39' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/10' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/34' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/38' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/11' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/16' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/32' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/30' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/14' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/18' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/13' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/33' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/17' 2024-03-28T19:00:29.042 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/35' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './108/37' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './108' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/19' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/15' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/31' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/36' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/20' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/12' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/39' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/10' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/34' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/38' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/11' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/16' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/32' 2024-03-28T19:00:29.043 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/30' 2024-03-28T19:00:29.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.428+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14e8069640 con 0x7f14e80c5320 2024-03-28T19:00:29.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.429+0000 7f14df400640 1 -- 172.21.15.27:0/749318675 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6248 (secure 0 0 0) 0x7f14d805c300 con 0x7f14e80c5320 2024-03-28T19:00:29.429 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:29.429 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:29.429 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 43 2024-03-28T19:00:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14c806bc20 msgr2=0x7f14c806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14c806bc20 0x7f14c806e0e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f14e814dd30 tx=0x7f14d4009190 comp rx=0 tx=0).stop 2024-03-28T19:00:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e80c5320 msgr2=0x7f14e80a9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e80c5320 0x7f14e80a9030 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f14d800c4d0 tx=0x7f14d800c990 comp rx=0 tx=0).stop 2024-03-28T19:00:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 shutdown_connections 2024-03-28T19:00:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14c806bc20 0x7f14c806e0e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14e8156500 0x7f14e8149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14e8156120 0x7f14e80c1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 --2- 172.21.15.27:0/749318675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14e80c5320 0x7f14e80a9030 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.433+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 >> 172.21.15.27:0/749318675 conn(0x7f14e806b0c0 msgr2=0x7f14e81553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.434+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 shutdown_connections 2024-03-28T19:00:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.434+0000 7f14ed000640 1 -- 172.21.15.27:0/749318675 wait complete. 2024-03-28T19:00:29.454 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-03-28T19:00:29.454 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:00:29.454 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:00:29.475 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:00:29.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:29.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.741+0000 7f0d12e00640 1 Processor -- start 2024-03-28T19:00:29.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.741+0000 7f0d12e00640 1 -- start start 2024-03-28T19:00:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.743+0000 7f0d12e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d0c071cf0 0x7f0d0c159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.743+0000 7f0d12e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c0c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.743+0000 7f0d12e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c072870 con 0x7f0d0c14f6d0 2024-03-28T19:00:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.743+0000 7f0d12e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c156640 con 0x7f0d0c071cf0 2024-03-28T19:00:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.743+0000 7f0d12e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c158db0 con 0x7f0d0c072c60 2024-03-28T19:00:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.743+0000 7f0d03e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d0c071cf0 0x7f0d0c159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.743+0000 7f0d10e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c0c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.743+0000 7f0d0be00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f0d0c14f6d0 0x7f0d0c0c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:36216/0 (socket says 172.21.15.27:36216) 2024-03-28T19:00:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.744+0000 7f0d0be00640 1 -- 172.21.15.27:0/1172675700 learned_addr learned my addr 172.21.15.27:0/1172675700 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.744+0000 7f0d03e00640 1 -- 172.21.15.27:0/1172675700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 msgr2=0x7f0d0c0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.744+0000 7f0d03e00640 1 --2- 172.21.15.27:0/1172675700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c0c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.744+0000 7f0d03e00640 1 -- 172.21.15.27:0/1172675700 >> v1:172.21.15.155:6789/0 conn(0x7f0d0c14f6d0 legacy=0x7f0d0c0c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.744+0000 7f0d03e00640 1 -- 172.21.15.27:0/1172675700 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c156360 con 0x7f0d0c071cf0 2024-03-28T19:00:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.744+0000 7f0d10e00640 1 --2- 172.21.15.27:0/1172675700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c0c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:00:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.745+0000 7f0d03e00640 1 --2- 172.21.15.27:0/1172675700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c071cf0 0x7f0d0c159850 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40097a0 tx=0x7f0cf402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f5185f105501f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d0b400640 1 -- 172.21.15.27:0/1172675700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4031780 con 0x7f0d0c071cf0 2024-03-28T19:00:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d0b400640 1 -- 172.21.15.27:0/1172675700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf4037070 con 0x7f0d0c071cf0 2024-03-28T19:00:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d0b400640 1 -- 172.21.15.27:0/1172675700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4031b40 con 0x7f0d0c071cf0 2024-03-28T19:00:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d12e00640 1 -- 172.21.15.27:0/1172675700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c071cf0 msgr2=0x7f0d0c159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1172675700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c071cf0 0x7f0d0c159850 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40097a0 tx=0x7f0cf402b820 comp rx=0 tx=0).stop 2024-03-28T19:00:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d12e00640 1 -- 172.21.15.27:0/1172675700 shutdown_connections 2024-03-28T19:00:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1172675700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c0c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1172675700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c071cf0 0x7f0d0c159850 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.746+0000 7f0d12e00640 1 -- 172.21.15.27:0/1172675700 >> 172.21.15.27:0/1172675700 conn(0x7f0d0c06c5f0 msgr2=0x7f0d0c06ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.747+0000 7f0d12e00640 1 -- 172.21.15.27:0/1172675700 shutdown_connections 2024-03-28T19:00:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.747+0000 7f0d12e00640 1 -- 172.21.15.27:0/1172675700 wait complete. 2024-03-28T19:00:29.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.749+0000 7f0d12e00640 1 Processor -- start 2024-03-28T19:00:29.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.749+0000 7f0d12e00640 1 -- start start 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d12e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c1ef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d12e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d0c14f6d0 0x7f0d0c14db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d12e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c1e1fa0 0x7f0d0c1eb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d12e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c14fab0 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d12e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c159330 con 0x7f0d0c14f6d0 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d12e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c3c70 con 0x7f0d0c072c60 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d10e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c1e1fa0 0x7f0d0c1eb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d10e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c1e1fa0 0x7f0d0c1eb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56058/0 (socket says 172.21.15.27:56058) 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d10e00640 1 -- 172.21.15.27:0/1229934551 learned_addr learned my addr 172.21.15.27:0/1229934551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d03e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d0c14f6d0 0x7f0d0c14db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.750+0000 7f0d0be00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c1ef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d10e00640 1 -- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 msgr2=0x7f0d0c1ef2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d10e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c1ef2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d10e00640 1 -- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d0c14f6d0 msgr2=0x7f0d0c14db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d10e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d0c14f6d0 0x7f0d0c14db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d10e00640 1 -- 172.21.15.27:0/1229934551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c06e200 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d0be00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c1ef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d10e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c1e1fa0 0x7f0d0c1eb2d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009db0 tx=0x7f0d0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d08c00640 1 -- 172.21.15.27:0/1229934551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04018030 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d08c00640 1 -- 172.21.15.27:0/1229934551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0400cd30 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c1ee840 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.751+0000 7f0d08c00640 1 -- 172.21.15.27:0/1229934551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04003bb0 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.752+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c0c7a60 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.753+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c04f400 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.754+0000 7f0d08c00640 1 -- 172.21.15.27:0/1229934551 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0d04005d10 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.755+0000 7f0d08c00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ce806bc00 0x7f0ce806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:29.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.755+0000 7f0d08c00640 1 -- 172.21.15.27:0/1229934551 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f0d0408b7e0 con 0x7f0d0c1e1fa0 2024-03-28T19:00:29.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.755+0000 7f0d0be00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ce806bc00 0x7f0ce806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:29.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.756+0000 7f0d0be00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ce806bc00 0x7f0ce806e0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c06e980 tx=0x7f0cfc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:29.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:29.760+0000 7f0d08c00640 1 -- 172.21.15.27:0/1229934551 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0d04022070 con 0x7f0d0c1e1fa0 2024-03-28T19:00:30.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.053+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d0c0c2330 con 0x7f0d0c1e1fa0 2024-03-28T19:00:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.053+0000 7f0d08c00640 1 -- 172.21.15.27:0/1229934551 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6248 (secure 0 0 0) 0x7f0d0c06e200 con 0x7f0d0c1e1fa0 2024-03-28T19:00:30.053 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:30.053 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:30.053 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 43 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ce806bc00 msgr2=0x7f0ce806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ce806bc00 0x7f0ce806e0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c06e980 tx=0x7f0cfc009040 comp rx=0 tx=0).stop 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c1e1fa0 msgr2=0x7f0d0c1eb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c1e1fa0 0x7f0d0c1eb2d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009db0 tx=0x7f0d0400c490 comp rx=0 tx=0).stop 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 shutdown_connections 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ce806bc00 0x7f0ce806e0c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d0c1e1fa0 0x7f0d0c1eb2d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d0c14f6d0 0x7f0d0c14db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 --2- 172.21.15.27:0/1229934551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d0c072c60 0x7f0d0c1ef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 >> 172.21.15.27:0/1229934551 conn(0x7f0d0c06c5f0 msgr2=0x7f0d0c06f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 shutdown_connections 2024-03-28T19:00:30.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:30.056+0000 7f0d12e00640 1 -- 172.21.15.27:0/1229934551 wait complete. 2024-03-28T19:00:30.073 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:00:30.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/14' 2024-03-28T19:00:30.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/18' 2024-03-28T19:00:30.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/13' 2024-03-28T19:00:30.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/33' 2024-03-28T19:00:30.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/17' 2024-03-28T19:00:30.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/35' 2024-03-28T19:00:30.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './109/37' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './109' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/19' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/15' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/31' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/36' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/20' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/12' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/39' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/10' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/34' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/38' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/11' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/16' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/32' 2024-03-28T19:00:30.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/30' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/14' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/18' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/13' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/33' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/17' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/35' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './110/37' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './110' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/19' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/15' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/31' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/36' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/20' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/12' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/39' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/10' 2024-03-28T19:00:30.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/34' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/38' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/11' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/16' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/32' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/30' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/14' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/18' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/13' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/33' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/17' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/35' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './111/37' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './111' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/19' 2024-03-28T19:00:30.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/15' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/31' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/36' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/20' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/12' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/39' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/10' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/34' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/38' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/11' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/16' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/32' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/30' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/14' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/18' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/13' 2024-03-28T19:00:30.927 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/33' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/17' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/35' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './112/37' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './112' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/19' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/15' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/31' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/36' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/20' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/12' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/39' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/10' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/34' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/38' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/11' 2024-03-28T19:00:30.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/16' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/32' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/30' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/14' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/18' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/13' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/33' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/17' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/35' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './113/37' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './113' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/19' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/15' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/31' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/36' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/20' 2024-03-28T19:00:30.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/12' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/39' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/10' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/34' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/38' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/11' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/16' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/32' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/30' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/14' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/18' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/13' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/33' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/17' 2024-03-28T19:00:30.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/35' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './114/37' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './114' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/19' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/15' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/31' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/36' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/20' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/12' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/39' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/10' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/34' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/38' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/11' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/16' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/32' 2024-03-28T19:00:30.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/30' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/14' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/18' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/13' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/33' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/17' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/35' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './115/37' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './115' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/19' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/15' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/31' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/36' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/20' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/12' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/39' 2024-03-28T19:00:30.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/10' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/34' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/38' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/11' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/16' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/32' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/30' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/14' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/18' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/13' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/33' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/17' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/35' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './116/37' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './116' 2024-03-28T19:00:30.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/19' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/15' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/31' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/36' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/20' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/12' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/39' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/10' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/34' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/38' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/11' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/16' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/32' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/30' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/14' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/18' 2024-03-28T19:00:30.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/13' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/33' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/17' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/35' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './117/37' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './117' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/19' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/15' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/31' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/36' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/20' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/12' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/39' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/10' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/34' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/38' 2024-03-28T19:00:30.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/11' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/16' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/32' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/30' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/14' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/18' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/13' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/33' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/17' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/35' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './118/37' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './118' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/19' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/15' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/31' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/36' 2024-03-28T19:00:30.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/20' 2024-03-28T19:00:30.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/12' 2024-03-28T19:00:32.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.314+0000 7f15f3200640 1 Processor -- start 2024-03-28T19:00:32.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.314+0000 7f15f3200640 1 -- start start 2024-03-28T19:00:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.316+0000 7f15f3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15ec0c58c0 0x7f15ec0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:32.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.316+0000 7f15f3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0be5d0 0x7f15ec155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.316+0000 7f15f3200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec156230 con 0x7f15ec0c8ed0 2024-03-28T19:00:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.316+0000 7f15f3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ec158bd0 con 0x7f15ec0c58c0 2024-03-28T19:00:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.316+0000 7f15f3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec158eb0 con 0x7f15ec0be5d0 2024-03-28T19:00:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.317+0000 7f15f1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15ec0c58c0 0x7f15ec0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:32.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.317+0000 7f15f1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f15ec0c58c0 0x7f15ec0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56074/0 (socket says 172.21.15.27:56074) 2024-03-28T19:00:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.317+0000 7f15f1e00640 1 -- 172.21.15.27:0/486563007 learned_addr learned my addr 172.21.15.27:0/486563007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.317+0000 7f15f2800640 1 --2- 172.21.15.27:0/486563007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0be5d0 0x7f15ec155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.317+0000 7f15f1e00640 1 -- 172.21.15.27:0/486563007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0be5d0 msgr2=0x7f15ec155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.317+0000 7f15f1e00640 1 --2- 172.21.15.27:0/486563007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0be5d0 0x7f15ec155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.317+0000 7f15f1e00640 1 -- 172.21.15.27:0/486563007 >> v1:172.21.15.155:6789/0 conn(0x7f15ec0c8ed0 legacy=0x7f15ec1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:32.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.317+0000 7f15f1e00640 1 -- 172.21.15.27:0/486563007 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec159190 con 0x7f15ec0c58c0 2024-03-28T19:00:32.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.319+0000 7f15f1e00640 1 --2- 172.21.15.27:0/486563007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0c5ca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009870 tx=0x7f15e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8db0b4f662665f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:32.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.319+0000 7f15f0a00640 1 -- 172.21.15.27:0/486563007 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0035020 con 0x7f15ec0c58c0 2024-03-28T19:00:32.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.320+0000 7f15f0a00640 1 -- 172.21.15.27:0/486563007 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e0031640 con 0x7f15ec0c58c0 2024-03-28T19:00:32.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.320+0000 7f15f0a00640 1 -- 172.21.15.27:0/486563007 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e002d040 con 0x7f15ec0c58c0 2024-03-28T19:00:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.320+0000 7f15f3200640 1 -- 172.21.15.27:0/486563007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 msgr2=0x7f15ec0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.320+0000 7f15f3200640 1 --2- 172.21.15.27:0/486563007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0c5ca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009870 tx=0x7f15e002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.320+0000 7f15f3200640 1 -- 172.21.15.27:0/486563007 shutdown_connections 2024-03-28T19:00:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.320+0000 7f15f3200640 1 --2- 172.21.15.27:0/486563007 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0be5d0 0x7f15ec155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.320+0000 7f15f3200640 1 --2- 172.21.15.27:0/486563007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0c5ca0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.320+0000 7f15f3200640 1 -- 172.21.15.27:0/486563007 >> 172.21.15.27:0/486563007 conn(0x7f15ec06b6f0 msgr2=0x7f15ec0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.321+0000 7f15f3200640 1 -- 172.21.15.27:0/486563007 shutdown_connections 2024-03-28T19:00:32.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.321+0000 7f15f3200640 1 -- 172.21.15.27:0/486563007 wait complete. 2024-03-28T19:00:32.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.322+0000 7f15f3200640 1 Processor -- start 2024-03-28T19:00:32.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.322+0000 7f15f3200640 1 -- start start 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.323+0000 7f15f3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15ec0be5d0 0x7f15ec0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.323+0000 7f15f3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.323+0000 7f15f3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0c8ed0 0x7f15ec1512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.323+0000 7f15f3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec0b0820 con 0x7f15ec0c58c0 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.323+0000 7f15f3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec159750 con 0x7f15ec0be5d0 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.323+0000 7f15f3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec0b09a0 con 0x7f15ec0c8ed0 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0bea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56086/0 (socket says 172.21.15.27:56086) 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1400640 1 -- 172.21.15.27:0/774727627 learned_addr learned my addr 172.21.15.27:0/774727627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f2800640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0c8ed0 0x7f15ec1512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1e00640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15ec0be5d0 0x7f15ec0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:32.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1400640 1 -- 172.21.15.27:0/774727627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0c8ed0 msgr2=0x7f15ec1512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:32.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1400640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0c8ed0 0x7f15ec1512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1400640 1 -- 172.21.15.27:0/774727627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15ec0be5d0 msgr2=0x7f15ec0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:32.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1400640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15ec0be5d0 0x7f15ec0ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.324+0000 7f15f1400640 1 -- 172.21.15.27:0/774727627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec0c19d0 con 0x7f15ec0c58c0 2024-03-28T19:00:32.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.325+0000 7f15f1400640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0bea70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00c470 tx=0x7f15dc00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:32.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.325+0000 7f15e6000640 1 -- 172.21.15.27:0/774727627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0052d0 con 0x7f15ec0c58c0 2024-03-28T19:00:32.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.325+0000 7f15e6000640 1 -- 172.21.15.27:0/774727627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc005490 con 0x7f15ec0c58c0 2024-03-28T19:00:32.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.325+0000 7f15e6000640 1 -- 172.21.15.27:0/774727627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc007260 con 0x7f15ec0c58c0 2024-03-28T19:00:32.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.326+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec0c2260 con 0x7f15ec0c58c0 2024-03-28T19:00:32.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.326+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec0aec70 con 0x7f15ec0c58c0 2024-03-28T19:00:32.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.327+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ec069640 con 0x7f15ec0c58c0 2024-03-28T19:00:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.330+0000 7f15e6000640 1 -- 172.21.15.27:0/774727627 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f15dc007400 con 0x7f15ec0c58c0 2024-03-28T19:00:32.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.331+0000 7f15e6000640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f15d006bbd0 0x7f15d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:32.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.331+0000 7f15e6000640 1 -- 172.21.15.27:0/774727627 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f15dc08a940 con 0x7f15ec0c58c0 2024-03-28T19:00:32.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.331+0000 7f15f1e00640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f15d006bbd0 0x7f15d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.332+0000 7f15f1e00640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f15d006bbd0 0x7f15d006e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f15e0036000 tx=0x7f15e0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:32.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.334+0000 7f15e6000640 1 -- 172.21.15.27:0/774727627 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f15dc058f10 con 0x7f15ec0c58c0 2024-03-28T19:00:32.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.677+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15ec156580 con 0x7f15ec0c58c0 2024-03-28T19:00:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.678+0000 7f15e6000640 1 -- 172.21.15.27:0/774727627 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6248 (secure 0 0 0) 0x7f15dc058810 con 0x7f15ec0c58c0 2024-03-28T19:00:32.678 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:32.678 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 43 2024-03-28T19:00:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f15d006bbd0 msgr2=0x7f15d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f15d006bbd0 0x7f15d006e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f15e0036000 tx=0x7f15e0008030 comp rx=0 tx=0).stop 2024-03-28T19:00:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 msgr2=0x7f15ec0bea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0bea70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00c470 tx=0x7f15dc00c930 comp rx=0 tx=0).stop 2024-03-28T19:00:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 shutdown_connections 2024-03-28T19:00:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f15d006bbd0 0x7f15d006e090 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15ec0c8ed0 0x7f15ec1512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15ec0c58c0 0x7f15ec0bea70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 --2- 172.21.15.27:0/774727627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15ec0be5d0 0x7f15ec0ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.681+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 >> 172.21.15.27:0/774727627 conn(0x7f15ec06b6f0 msgr2=0x7f15ec072570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.682+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 shutdown_connections 2024-03-28T19:00:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:32.682+0000 7f15f3200640 1 -- 172.21.15.27:0/774727627 wait complete. 2024-03-28T19:00:32.701 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:00:33.163 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/39' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/10' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/34' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/38' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/11' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/16' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/32' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/30' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/14' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/18' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/13' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/33' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/17' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/35' 2024-03-28T19:00:33.164 INFO:tasks.workunit.client.0.smithi027.stdout:removed './119/37' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './119' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/19' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/15' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/31' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/36' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/20' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/12' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/39' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/10' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/34' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/38' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/11' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/16' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/32' 2024-03-28T19:00:33.165 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/30' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/14' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/18' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/13' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/33' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/17' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/35' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './120/37' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './120' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/19' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/15' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/31' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/36' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/20' 2024-03-28T19:00:33.166 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/12' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/39' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/10' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/34' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/38' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/11' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/16' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/32' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/30' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/14' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/18' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/13' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/33' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/17' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/35' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed './121/37' 2024-03-28T19:00:33.167 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './121' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/19' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/15' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/31' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/36' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/20' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/12' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/39' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/10' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/34' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/38' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/11' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/16' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/32' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/30' 2024-03-28T19:00:33.168 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/14' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/18' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/13' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/33' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/17' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/35' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './122/37' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './122' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/19' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/15' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/31' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/36' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/20' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/12' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/39' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/10' 2024-03-28T19:00:33.169 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/34' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/38' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/11' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/16' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/32' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/30' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/14' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/18' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/13' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/33' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/17' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/35' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './123/37' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './123' 2024-03-28T19:00:33.170 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/19' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/15' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/31' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/36' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/20' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/12' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/39' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/10' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/34' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/38' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/11' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/16' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/32' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/30' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/14' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/18' 2024-03-28T19:00:33.171 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/13' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/33' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/17' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/35' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './124/37' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './124' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/19' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/15' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/31' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/36' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/20' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/12' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/39' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/10' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/34' 2024-03-28T19:00:33.172 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/38' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/11' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/16' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/32' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/30' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/14' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/18' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/13' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/33' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/17' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/35' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './125/37' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './125' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/19' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/15' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/31' 2024-03-28T19:00:33.173 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/36' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/20' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/12' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/39' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/10' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/34' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/38' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/11' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/16' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/32' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/30' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/14' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/18' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/13' 2024-03-28T19:00:33.174 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/33' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/17' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/35' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './126/37' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './126' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/19' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/15' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/31' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/36' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/20' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/12' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/39' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/10' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/34' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/38' 2024-03-28T19:00:33.175 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/11' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/16' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/32' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/30' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/14' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/18' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/13' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/33' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/17' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/35' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './127/37' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './127' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/19' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/15' 2024-03-28T19:00:33.176 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/31' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/36' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/20' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/12' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/39' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/10' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/34' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/38' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/11' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/16' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/32' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/30' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/14' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/18' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/13' 2024-03-28T19:00:33.177 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/33' 2024-03-28T19:00:33.178 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/17' 2024-03-28T19:00:33.178 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/35' 2024-03-28T19:00:34.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:34.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.944+0000 7efea0a00640 1 Processor -- start 2024-03-28T19:00:34.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.944+0000 7efea0a00640 1 -- start start 2024-03-28T19:00:34.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.946+0000 7efea0a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efe9c06ce20 0x7efe9c0a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:34.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.946+0000 7efea0a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c0c9a40 0x7efe9c0c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:34.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.946+0000 7efea0a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c159bf0 con 0x7efe9c159760 2024-03-28T19:00:34.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.946+0000 7efea0a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe9c0c5040 con 0x7efe9c06ce20 2024-03-28T19:00:34.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.946+0000 7efea0a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c74b0 con 0x7efe9c0c9a40 2024-03-28T19:00:34.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.947+0000 7efe99400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efe9c159760 0x7efe9c0c8cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41924/0 (socket says 172.21.15.27:41924) 2024-03-28T19:00:34.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.947+0000 7efe99400640 1 -- 172.21.15.27:0/999151024 learned_addr learned my addr 172.21.15.27:0/999151024 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:34.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.947+0000 7efe99e00640 1 --2- 172.21.15.27:0/999151024 >> v2:172.21.15.155:3300/0 conn(0x7efe9c06ce20 0x7efe9c0a7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:34.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.947+0000 7efe9a800640 1 --2- 172.21.15.27:0/999151024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c0c9a40 0x7efe9c0c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:34.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.947+0000 7efe98a00640 1 -- 172.21.15.27:0/999151024 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648752696 0 0) 0x7efe9c159bf0 con 0x7efe9c159760 2024-03-28T19:00:34.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.947+0000 7efe98a00640 1 -- 172.21.15.27:0/999151024 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe80003520 con 0x7efe9c159760 2024-03-28T19:00:34.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.948+0000 7efe99e00640 1 -- 172.21.15.27:0/999151024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c0c9a40 msgr2=0x7efe9c0c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:34.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.948+0000 7efe99e00640 1 --2- 172.21.15.27:0/999151024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c0c9a40 0x7efe9c0c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:34.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.948+0000 7efe99e00640 1 -- 172.21.15.27:0/999151024 >> v1:172.21.15.27:6789/0 conn(0x7efe9c159760 legacy=0x7efe9c0c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:34.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.948+0000 7efe98a00640 1 -- 172.21.15.27:0/999151024 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 412105985 0 0) 0x7efe80003520 con 0x7efe9c159760 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.948+0000 7efe99e00640 1 -- 172.21.15.27:0/999151024 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c0c7790 con 0x7efe9c06ce20 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.949+0000 7efe99e00640 1 --2- 172.21.15.27:0/999151024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c06ce20 0x7efe9c0a7b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efe8c0097a0 tx=0x7efe8c02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=619b1fdc9826b79a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.950+0000 7efe98a00640 1 -- 172.21.15.27:0/999151024 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c037070 con 0x7efe9c06ce20 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.950+0000 7efe98a00640 1 -- 172.21.15.27:0/999151024 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c030510 con 0x7efe9c06ce20 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.950+0000 7efe98a00640 1 -- 172.21.15.27:0/999151024 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88001670 con 0x7efe9c06ce20 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.951+0000 7efea0a00640 1 -- 172.21.15.27:0/999151024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c06ce20 msgr2=0x7efe9c0a7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.951+0000 7efea0a00640 1 --2- 172.21.15.27:0/999151024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c06ce20 0x7efe9c0a7b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efe8c0097a0 tx=0x7efe8c02b710 comp rx=0 tx=0).stop 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.951+0000 7efea0a00640 1 -- 172.21.15.27:0/999151024 shutdown_connections 2024-03-28T19:00:34.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.951+0000 7efea0a00640 1 --2- 172.21.15.27:0/999151024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c0c9a40 0x7efe9c0c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:34.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.951+0000 7efea0a00640 1 --2- 172.21.15.27:0/999151024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c06ce20 0x7efe9c0a7b00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:34.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.951+0000 7efea0a00640 1 -- 172.21.15.27:0/999151024 >> 172.21.15.27:0/999151024 conn(0x7efe9c0aaab0 msgr2=0x7efe9c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:34.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.951+0000 7efea0a00640 1 -- 172.21.15.27:0/999151024 shutdown_connections 2024-03-28T19:00:34.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.951+0000 7efea0a00640 1 -- 172.21.15.27:0/999151024 wait complete. 2024-03-28T19:00:34.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.953+0000 7efea0a00640 1 Processor -- start 2024-03-28T19:00:34.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.953+0000 7efea0a00640 1 -- start start 2024-03-28T19:00:34.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.954+0000 7efea0a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c06ce20 0x7efe9c1ef230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.954+0000 7efea0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0c9a40 0x7efe9c1551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.954+0000 7efea0a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c14d0c0 0x7efe9c14c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.954+0000 7efea0a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c9560 con 0x7efe9c0c9a40 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.954+0000 7efea0a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c93e0 con 0x7efe9c14d0c0 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.954+0000 7efea0a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9c0c4290 con 0x7efe9c06ce20 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0c9a40 0x7efe9c1551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0c9a40 0x7efe9c1551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56100/0 (socket says 172.21.15.27:56100) 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99400640 1 -- 172.21.15.27:0/3506820980 learned_addr learned my addr 172.21.15.27:0/3506820980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99e00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c06ce20 0x7efe9c1ef230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe9a800640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c14d0c0 0x7efe9c14c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99400640 1 -- 172.21.15.27:0/3506820980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c06ce20 msgr2=0x7efe9c1ef230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99400640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c06ce20 0x7efe9c1ef230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99400640 1 -- 172.21.15.27:0/3506820980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c14d0c0 msgr2=0x7efe9c14c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99400640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c14d0c0 0x7efe9c14c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:34.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.955+0000 7efe99400640 1 -- 172.21.15.27:0/3506820980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c157010 con 0x7efe9c0c9a40 2024-03-28T19:00:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.956+0000 7efe99400640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0c9a40 0x7efe9c1551d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efe8800c750 tx=0x7efe8800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.956+0000 7efe92000640 1 -- 172.21.15.27:0/3506820980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88017070 con 0x7efe9c0c9a40 2024-03-28T19:00:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.956+0000 7efe92000640 1 -- 172.21.15.27:0/3506820980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe88005ad0 con 0x7efe9c0c9a40 2024-03-28T19:00:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.956+0000 7efe92000640 1 -- 172.21.15.27:0/3506820980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88007070 con 0x7efe9c0c9a40 2024-03-28T19:00:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.956+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c1572f0 con 0x7efe9c0c9a40 2024-03-28T19:00:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.956+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c1578b0 con 0x7efe9c0c9a40 2024-03-28T19:00:34.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.957+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c04f400 con 0x7efe9c0c9a40 2024-03-28T19:00:34.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.959+0000 7efe92000640 1 -- 172.21.15.27:0/3506820980 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efe8801a070 con 0x7efe9c0c9a40 2024-03-28T19:00:34.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.960+0000 7efe92000640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bbd0 0x7efe7c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:34.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.960+0000 7efe92000640 1 -- 172.21.15.27:0/3506820980 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7095+0+0 (secure 0 0 0) 0x7efe8808a300 con 0x7efe9c0c9a40 2024-03-28T19:00:34.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.961+0000 7efe99e00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bbd0 0x7efe7c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:34.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.962+0000 7efe99e00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bbd0 0x7efe7c06e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efe8c034000 tx=0x7efe8c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:34.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:34.967+0000 7efe92000640 1 -- 172.21.15.27:0/3506820980 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efe88058230 con 0x7efe9c0c9a40 2024-03-28T19:00:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.313+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe9c0c3790 con 0x7efe9c0c9a40 2024-03-28T19:00:35.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.315+0000 7efe92000640 1 -- 172.21.15.27:0/3506820980 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6248 (secure 0 0 0) 0x7efe8805c2d0 con 0x7efe9c0c9a40 2024-03-28T19:00:35.315 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:35.315 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:35.315 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 43 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bbd0 msgr2=0x7efe7c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bbd0 0x7efe7c06e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efe8c034000 tx=0x7efe8c008030 comp rx=0 tx=0).stop 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0c9a40 msgr2=0x7efe9c1551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0c9a40 0x7efe9c1551d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efe8800c750 tx=0x7efe8800cc10 comp rx=0 tx=0).stop 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 shutdown_connections 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bbd0 0x7efe7c06e090 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe9c14d0c0 0x7efe9c14c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe9c0c9a40 0x7efe9c1551d0 secure :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efe8800c750 tx=0x7efe8800cc10 comp rx=0 tx=0).stop 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 --2- 172.21.15.27:0/3506820980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe9c06ce20 0x7efe9c1ef230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.319+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 >> 172.21.15.27:0/3506820980 conn(0x7efe9c0aaab0 msgr2=0x7efe9c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.320+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 shutdown_connections 2024-03-28T19:00:35.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:35.320+0000 7efea0a00640 1 -- 172.21.15.27:0/3506820980 wait complete. 2024-03-28T19:00:35.343 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:00:35.723 INFO:tasks.workunit.client.0.smithi027.stdout:removed './128/37' 2024-03-28T19:00:35.723 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './128' 2024-03-28T19:00:35.723 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/19' 2024-03-28T19:00:35.723 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/15' 2024-03-28T19:00:35.723 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/31' 2024-03-28T19:00:35.723 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/36' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/20' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/12' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/39' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/10' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/34' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/38' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/11' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/16' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/32' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/30' 2024-03-28T19:00:35.724 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/14' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/18' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/13' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/33' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/17' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/35' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './129/37' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './129' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/19' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/15' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/31' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/36' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/20' 2024-03-28T19:00:35.725 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/12' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/39' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/10' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/34' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/38' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/11' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/16' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/32' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/30' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/14' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/18' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/13' 2024-03-28T19:00:35.726 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/33' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/17' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/35' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './130/37' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './130' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/19' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/15' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/31' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/36' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/20' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/12' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/39' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/10' 2024-03-28T19:00:35.727 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/34' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/38' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/11' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/16' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/32' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/30' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/14' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/18' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/13' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/33' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/17' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/35' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed './131/37' 2024-03-28T19:00:35.728 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './131' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/19' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/15' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/31' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/36' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/20' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/12' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/39' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/10' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/34' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/38' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/11' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/16' 2024-03-28T19:00:35.729 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/32' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/30' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/14' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/18' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/13' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/33' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/17' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/35' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './132/37' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './132' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/19' 2024-03-28T19:00:35.730 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/15' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/31' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/36' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/20' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/12' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/39' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/10' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/34' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/38' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/11' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/16' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/32' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/30' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/14' 2024-03-28T19:00:35.731 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/18' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/13' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/33' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/17' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/35' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './133/37' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './133' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/19' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/15' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/31' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/36' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/20' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/12' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/39' 2024-03-28T19:00:35.732 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/10' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/34' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/38' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/11' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/16' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/32' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/30' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/14' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/18' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/13' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/33' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/17' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/35' 2024-03-28T19:00:35.733 INFO:tasks.workunit.client.0.smithi027.stdout:removed './134/37' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './134' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/19' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/15' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/31' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/36' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/20' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/12' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/39' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/10' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/34' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/38' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/11' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/16' 2024-03-28T19:00:35.734 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/32' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/30' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/14' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/18' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/13' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/33' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/17' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/35' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './135/37' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './135' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/19' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/15' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/31' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/36' 2024-03-28T19:00:35.735 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/20' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/12' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/39' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/10' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/34' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/38' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/11' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/16' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/32' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/30' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/14' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/18' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/13' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/33' 2024-03-28T19:00:35.736 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/17' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/35' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './136/37' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './136' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/19' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/15' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/31' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/36' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/20' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/12' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/39' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/10' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/34' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/38' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/11' 2024-03-28T19:00:35.737 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/16' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/32' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/30' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/14' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/18' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/13' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/33' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/17' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/35' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './137/37' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './137' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/19' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/15' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/31' 2024-03-28T19:00:35.738 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/36' 2024-03-28T19:00:35.739 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/20' 2024-03-28T19:00:35.739 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/12' 2024-03-28T19:00:35.739 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/39' 2024-03-28T19:00:35.739 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/10' 2024-03-28T19:00:35.739 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/34' 2024-03-28T19:00:35.739 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/38' 2024-03-28T19:00:35.739 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/11' 2024-03-28T19:00:35.739 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/16' 2024-03-28T19:00:37.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:37.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.591+0000 7f9ce7e00640 1 Processor -- start 2024-03-28T19:00:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.591+0000 7f9ce7e00640 1 -- start start 2024-03-28T19:00:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.593+0000 7f9ce7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce8070820 0x7f9ce8153280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.593+0000 7f9ce7e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce81576f0 con 0x7f9ce80aaae0 2024-03-28T19:00:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.593+0000 7f9ce7e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce80a7f60 con 0x7f9ce80a82d0 2024-03-28T19:00:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.593+0000 7f9ce7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8156870 con 0x7f9ce8070820 2024-03-28T19:00:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.593+0000 7f9ce6a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9ce80aaae0 0x7f9ce8156fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54384/0 (socket says 172.21.15.27:54384) 2024-03-28T19:00:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.593+0000 7f9ce6a00640 1 -- 172.21.15.27:0/1010243312 learned_addr learned my addr 172.21.15.27:0/1010243312 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.593+0000 7f9ce7400640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce8070820 0x7f9ce8153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001758151 0 0) 0x7f9ce81576f0 con 0x7f9ce80aaae0 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ccc003520 con 0x7f9ce80aaae0 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4057567781 0 0) 0x7f9ccc003520 con 0x7f9ce80aaae0 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce8070820 msgr2=0x7f9ce8153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce8070820 0x7f9ce8153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 >> v1:172.21.15.155:6789/0 conn(0x7f9ce80a82d0 legacy=0x7f9ce815bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce80a7c80 con 0x7f9ce80aaae0 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9cd8001690 con 0x7f9ce80aaae0 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 >> v1:172.21.15.27:6789/0 conn(0x7f9ce80aaae0 legacy=0x7f9ce8156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ccc0039e0 0x7f9ccc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce8070820 0x7f9ccc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ccc004d40 0x7f9ccc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8156870 con 0x7f9ccc0039e0 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc00baf0 con 0x7f9ccc004d40 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.594+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc010100 con 0x7f9ce8070820 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce6a00640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ccc0039e0 0x7f9ccc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce7400640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ccc004d40 0x7f9ccc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce6000640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce8070820 0x7f9ccc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce6a00640 1 -- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce8070820 msgr2=0x7f9ccc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce6a00640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce8070820 0x7f9ccc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce6a00640 1 -- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ccc004d40 msgr2=0x7f9ccc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce6a00640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ccc004d40 0x7f9ccc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce6a00640 1 -- 172.21.15.27:0/1010243312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc003520 con 0x7f9ccc0039e0 2024-03-28T19:00:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.595+0000 7f9ce7400640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ccc004d40 0x7f9ccc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:00:37.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.596+0000 7f9ce6a00640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ccc0039e0 0x7f9ccc003e00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8007fd0 tx=0x7f9ccc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97b5c9a3e0fa64 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:37.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.597+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8007d50 con 0x7f9ccc0039e0 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.597+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd8030070 con 0x7f9ccc0039e0 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.597+0000 7f9ce5600640 1 -- 172.21.15.27:0/1010243312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8005d30 con 0x7f9ccc0039e0 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.597+0000 7f9ce7e00640 1 -- 172.21.15.27:0/1010243312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ccc0039e0 msgr2=0x7f9ccc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.597+0000 7f9ce7e00640 1 --2- 172.21.15.27:0/1010243312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ccc0039e0 0x7f9ccc003e00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8007fd0 tx=0x7f9ccc005830 comp rx=0 tx=0).stop 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.598+0000 7f9ce7400640 1 -- 172.21.15.27:0/1010243312 reap_dead start 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.598+0000 7f9ce7e00640 1 -- 172.21.15.27:0/1010243312 shutdown_connections 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.598+0000 7f9ce7e00640 1 -- 172.21.15.27:0/1010243312 >> 172.21.15.27:0/1010243312 conn(0x7f9ce8072bd0 msgr2=0x7f9ce8072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.598+0000 7f9ce7e00640 1 -- 172.21.15.27:0/1010243312 shutdown_connections 2024-03-28T19:00:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.598+0000 7f9ce7e00640 1 -- 172.21.15.27:0/1010243312 wait complete. 2024-03-28T19:00:37.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.600+0000 7f9ce7e00640 1 Processor -- start 2024-03-28T19:00:37.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.600+0000 7f9ce7e00640 1 -- start start 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce814bee0 0x7f9ce81e7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce8151bc0 0x7f9ce8152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce81ea270 0x7f9ce81ea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8156ac0 con 0x7f9ce814bee0 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80c97d0 con 0x7f9ce8151bc0 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80c6460 con 0x7f9ce81ea270 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce814bee0 0x7f9ce81e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce814bee0 0x7f9ce81e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40442/0 (socket says 172.21.15.27:40442) 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.601+0000 7f9ce6a00640 1 -- 172.21.15.27:0/2034695517 learned_addr learned my addr 172.21.15.27:0/2034695517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce6000640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce8151bc0 0x7f9ce8152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce7400640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce81ea270 0x7f9ce81ea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:37.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce6a00640 1 -- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce81ea270 msgr2=0x7f9ce81ea6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce6a00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce81ea270 0x7f9ce81ea6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce6a00640 1 -- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce8151bc0 msgr2=0x7f9ce8152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce6a00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce8151bc0 0x7f9ce8152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce6a00640 1 -- 172.21.15.27:0/2034695517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce81528d0 con 0x7f9ce814bee0 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce6000640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce8151bc0 0x7f9ce8152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.602+0000 7f9ce6a00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce814bee0 0x7f9ce81e7780 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9cd802c480 tx=0x7f9cd802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.603+0000 7f9cdea00640 1 -- 172.21.15.27:0/2034695517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8007d50 con 0x7f9ce814bee0 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.603+0000 7f9cdea00640 1 -- 172.21.15.27:0/2034695517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd802cc20 con 0x7f9ce814bee0 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.603+0000 7f9cdea00640 1 -- 172.21.15.27:0/2034695517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8007070 con 0x7f9ce814bee0 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.603+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce8152bb0 con 0x7f9ce814bee0 2024-03-28T19:00:37.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.603+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce81513c0 con 0x7f9ce814bee0 2024-03-28T19:00:37.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.604+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce8069640 con 0x7f9ce814bee0 2024-03-28T19:00:37.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.605+0000 7f9cdea00640 1 -- 172.21.15.27:0/2034695517 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9cd8048020 con 0x7f9ce814bee0 2024-03-28T19:00:37.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.607+0000 7f9cdea00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc806bbd0 0x7f9cc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:37.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.607+0000 7f9cdea00640 1 -- 172.21.15.27:0/2034695517 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7095+0+0 (secure 0 0 0) 0x7f9cd80ac440 con 0x7f9ce814bee0 2024-03-28T19:00:37.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.608+0000 7f9ce6000640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc806bbd0 0x7f9cc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:37.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.608+0000 7f9ce6000640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc806bbd0 0x7f9cc806e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80a9530 tx=0x7f9cd4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.612+0000 7f9cdea00640 1 -- 172.21.15.27:0/2034695517 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9cd807a920 con 0x7f9ce814bee0 2024-03-28T19:00:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.972+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ce814dba0 con 0x7f9ce814bee0 2024-03-28T19:00:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.973+0000 7f9cdea00640 1 -- 172.21.15.27:0/2034695517 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6248 (secure 0 0 0) 0x7f9cd8034e80 con 0x7f9ce814bee0 2024-03-28T19:00:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:37.973 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 43 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc806bbd0 msgr2=0x7f9cc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc806bbd0 0x7f9cc806e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80a9530 tx=0x7f9cd4009040 comp rx=0 tx=0).stop 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce814bee0 msgr2=0x7f9ce81e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce814bee0 0x7f9ce81e7780 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9cd802c480 tx=0x7f9cd802e070 comp rx=0 tx=0).stop 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 shutdown_connections 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9cc806bbd0 0x7f9cc806e090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ce81ea270 0x7f9ce81ea6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ce8151bc0 0x7f9ce8152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 --2- 172.21.15.27:0/2034695517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce814bee0 0x7f9ce81e7780 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 >> 172.21.15.27:0/2034695517 conn(0x7f9ce8072bd0 msgr2=0x7f9ce80a9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 shutdown_connections 2024-03-28T19:00:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:37.976+0000 7f9ce7e00640 1 -- 172.21.15.27:0/2034695517 wait complete. 2024-03-28T19:00:37.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/32' 2024-03-28T19:00:37.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/30' 2024-03-28T19:00:37.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/14' 2024-03-28T19:00:37.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/18' 2024-03-28T19:00:37.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/13' 2024-03-28T19:00:37.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/33' 2024-03-28T19:00:37.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/17' 2024-03-28T19:00:37.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/35' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './138/37' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './138' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/19' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/15' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/31' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/36' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/20' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/12' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/39' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/10' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/34' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/38' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/11' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/16' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/32' 2024-03-28T19:00:37.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/30' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/14' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/18' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/13' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/33' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/17' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/35' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './139/37' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './139' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/19' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/15' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/31' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/36' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/20' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/12' 2024-03-28T19:00:37.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/39' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/10' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/34' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/38' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/11' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/16' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/32' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/30' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/14' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/18' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/13' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/33' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/17' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/35' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './140/37' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './140' 2024-03-28T19:00:37.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/19' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/15' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/31' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/36' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/20' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/12' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/39' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/10' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/34' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/38' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/11' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/16' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/32' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/30' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/14' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/18' 2024-03-28T19:00:37.988 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/13' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/33' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/17' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/35' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './141/37' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './141' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/19' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/15' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/31' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/36' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/20' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/12' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/39' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/10' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/34' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/38' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/11' 2024-03-28T19:00:37.989 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/16' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/32' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/30' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/14' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/18' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/13' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/33' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/17' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/35' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './142/37' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './142' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/19' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/15' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/31' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/36' 2024-03-28T19:00:37.990 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/20' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/12' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/39' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/10' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/34' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/38' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/11' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/16' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/32' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/30' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/14' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/18' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/13' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/33' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/17' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/35' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed './143/37' 2024-03-28T19:00:37.991 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './143' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/19' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/15' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/31' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/36' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/20' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/12' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/39' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/10' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/34' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/38' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/11' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/16' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/32' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/30' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/14' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/18' 2024-03-28T19:00:37.992 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/13' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/33' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/17' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/35' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './144/37' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './144' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/19' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/15' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/31' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/36' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/20' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/12' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/39' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/10' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/34' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/38' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/11' 2024-03-28T19:00:37.993 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/16' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/32' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/30' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/14' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/18' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/13' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/33' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/17' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/35' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './145/37' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './145' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/19' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/15' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/31' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/36' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/20' 2024-03-28T19:00:37.994 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/12' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/39' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/10' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/34' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/38' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/11' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/16' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/32' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/30' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/14' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/18' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/13' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/33' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/17' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/35' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed './146/37' 2024-03-28T19:00:37.995 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './146' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/19' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/15' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/31' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/36' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/20' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/12' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/39' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/10' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/34' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/38' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/11' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/16' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/32' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/30' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/14' 2024-03-28T19:00:37.996 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/18' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/13' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/33' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/17' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/35' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './147/37' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './147' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/19' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/15' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/31' 2024-03-28T19:00:37.997 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/36' 2024-03-28T19:00:37.998 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:00:39.999 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:40.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.218+0000 7f71dac00640 1 Processor -- start 2024-03-28T19:00:40.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.218+0000 7f71dac00640 1 -- start start 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71dac00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f71d40c74e0 0x7f71d40c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71dac00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71d40c2290 0x7f71d4152760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71dac00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d4152ca0 0x7f71d41556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71dac00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d4153110 con 0x7f71d40c2290 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71dac00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d4158d00 con 0x7f71d40c74e0 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71dac00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d4158fe0 con 0x7f71d4152ca0 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71d8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71d40c2290 0x7f71d4152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71d8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71d40c2290 0x7f71d4152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40446/0 (socket says 172.21.15.27:40446) 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.220+0000 7f71d8e00640 1 -- 172.21.15.27:0/1160500638 learned_addr learned my addr 172.21.15.27:0/1160500638 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.221+0000 7f71d9800640 1 --2- 172.21.15.27:0/1160500638 >> v2:172.21.15.155:3300/0 conn(0x7f71d40c74e0 0x7f71d40c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.221+0000 7f71da200640 1 --2- 172.21.15.27:0/1160500638 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d4152ca0 0x7f71d41556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.221+0000 7f71d8e00640 1 -- 172.21.15.27:0/1160500638 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d4152ca0 msgr2=0x7f71d41556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:40.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.221+0000 7f71d8e00640 1 --2- 172.21.15.27:0/1160500638 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d4152ca0 0x7f71d41556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.221+0000 7f71d8e00640 1 -- 172.21.15.27:0/1160500638 >> v2:172.21.15.155:3300/0 conn(0x7f71d40c74e0 msgr2=0x7f71d40c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:40.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.221+0000 7f71d8e00640 1 --2- 172.21.15.27:0/1160500638 >> v2:172.21.15.155:3300/0 conn(0x7f71d40c74e0 0x7f71d40c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.221+0000 7f71d8e00640 1 -- 172.21.15.27:0/1160500638 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d41592c0 con 0x7f71d40c2290 2024-03-28T19:00:40.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.221+0000 7f71da200640 1 --2- 172.21.15.27:0/1160500638 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d4152ca0 0x7f71d41556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.222+0000 7f71d8e00640 1 --2- 172.21.15.27:0/1160500638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d4152760 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f71d40c7240 tx=0x7f71c002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bdd7294dfe8ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.222+0000 7f71d3e00640 1 -- 172.21.15.27:0/1160500638 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c00317d0 con 0x7f71d40c2290 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.222+0000 7f71d3e00640 1 -- 172.21.15.27:0/1160500638 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c0037070 con 0x7f71d40c2290 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71d3e00640 1 -- 172.21.15.27:0/1160500638 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0031b90 con 0x7f71d40c2290 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71dac00640 1 -- 172.21.15.27:0/1160500638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 msgr2=0x7f71d4152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71dac00640 1 --2- 172.21.15.27:0/1160500638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d4152760 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f71d40c7240 tx=0x7f71c002b870 comp rx=0 tx=0).stop 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71dac00640 1 -- 172.21.15.27:0/1160500638 shutdown_connections 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71dac00640 1 --2- 172.21.15.27:0/1160500638 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d4152ca0 0x7f71d41556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71dac00640 1 --2- 172.21.15.27:0/1160500638 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d4152760 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71dac00640 1 --2- 172.21.15.27:0/1160500638 >> v2:172.21.15.155:3300/0 conn(0x7f71d40c74e0 0x7f71d40c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71dac00640 1 -- 172.21.15.27:0/1160500638 >> 172.21.15.27:0/1160500638 conn(0x7f71d406d980 msgr2=0x7f71d40a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:40.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.223+0000 7f71dac00640 1 -- 172.21.15.27:0/1160500638 shutdown_connections 2024-03-28T19:00:40.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.224+0000 7f71dac00640 1 -- 172.21.15.27:0/1160500638 wait complete. 2024-03-28T19:00:40.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.226+0000 7f71dac00640 1 Processor -- start 2024-03-28T19:00:40.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.226+0000 7f71dac00640 1 -- start start 2024-03-28T19:00:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71dac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d41ebbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71dac00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d40c74e0 0x7f71d41ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71dac00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71d4152ca0 0x7f71d41546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71dac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d41589f0 con 0x7f71d40c2290 2024-03-28T19:00:40.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71dac00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d4159880 con 0x7f71d4152ca0 2024-03-28T19:00:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71dac00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d40c78c0 con 0x7f71d40c74e0 2024-03-28T19:00:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71d9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d41ebbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71d9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d41ebbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40456/0 (socket says 172.21.15.27:40456) 2024-03-28T19:00:40.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71d9800640 1 -- 172.21.15.27:0/3566245427 learned_addr learned my addr 172.21.15.27:0/3566245427 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71d9800640 1 -- 172.21.15.27:0/3566245427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d40c74e0 msgr2=0x7f71d41ee620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71d9800640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d40c74e0 0x7f71d41ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71d9800640 1 -- 172.21.15.27:0/3566245427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71d4152ca0 msgr2=0x7f71d41546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71d9800640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71d4152ca0 0x7f71d41546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.227+0000 7f71d9800640 1 -- 172.21.15.27:0/3566245427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d4148090 con 0x7f71d40c2290 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.228+0000 7f71d9800640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d41ebbe0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f71c800d800 tx=0x7f71c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.228+0000 7f71d1600640 1 -- 172.21.15.27:0/3566245427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8018030 con 0x7f71d40c2290 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.228+0000 7f71d1600640 1 -- 172.21.15.27:0/3566245427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c80096a0 con 0x7f71d40c2290 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.228+0000 7f71d1600640 1 -- 172.21.15.27:0/3566245427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8009870 con 0x7f71d40c2290 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.228+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d41eb3e0 con 0x7f71d40c2290 2024-03-28T19:00:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.228+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d41eeb60 con 0x7f71d40c2290 2024-03-28T19:00:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.230+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d4067dd0 con 0x7f71d40c2290 2024-03-28T19:00:40.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.231+0000 7f71d1600640 1 -- 172.21.15.27:0/3566245427 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f71c801b070 con 0x7f71d40c2290 2024-03-28T19:00:40.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.232+0000 7f71d1600640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71b406bbd0 0x7f71b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:40.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.232+0000 7f71d1600640 1 -- 172.21.15.27:0/3566245427 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7095+0+0 (secure 0 0 0) 0x7f71c808b2a0 con 0x7f71d40c2290 2024-03-28T19:00:40.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.232+0000 7f71d8e00640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71b406bbd0 0x7f71b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:40.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.233+0000 7f71d8e00640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71b406bbd0 0x7f71b406e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f71c0036000 tx=0x7f71c0030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:40.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.237+0000 7f71d1600640 1 -- 172.21.15.27:0/3566245427 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f71c8059810 con 0x7f71d40c2290 2024-03-28T19:00:40.438 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/20' 2024-03-28T19:00:40.438 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/12' 2024-03-28T19:00:40.438 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/39' 2024-03-28T19:00:40.438 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/10' 2024-03-28T19:00:40.438 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/34' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/38' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/11' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/16' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/32' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/30' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/14' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/18' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/13' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/33' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/17' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/35' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './148/37' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './148' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/19' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/15' 2024-03-28T19:00:40.439 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/31' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/36' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/20' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/12' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/39' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/10' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/34' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/38' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/11' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/16' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/32' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/30' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/14' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/18' 2024-03-28T19:00:40.440 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/13' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/33' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/17' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/35' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './149/37' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './149' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/19' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/15' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/31' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/36' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/20' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/12' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/39' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/10' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/34' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/38' 2024-03-28T19:00:40.441 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/11' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/16' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/32' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/30' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/14' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/18' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/13' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/33' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/17' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/35' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './150/37' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './150' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/19' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/15' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/31' 2024-03-28T19:00:40.442 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/36' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/20' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/12' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/39' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/10' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/34' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/38' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/11' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/16' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/32' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/30' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/14' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/18' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/13' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/33' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/17' 2024-03-28T19:00:40.443 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/35' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './151/37' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './151' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/19' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/15' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/31' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/36' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/20' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/12' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/39' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/10' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/34' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/38' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/11' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/16' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/32' 2024-03-28T19:00:40.444 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/30' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/14' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/18' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/13' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/33' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/17' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/35' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './152/37' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './152' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/19' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/15' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/31' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/36' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/20' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/12' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/39' 2024-03-28T19:00:40.445 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/10' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/34' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/38' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/11' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/16' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/32' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/30' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/14' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/18' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/13' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/33' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/17' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/35' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './153/37' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './153' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/19' 2024-03-28T19:00:40.446 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/15' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/31' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/36' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/20' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/12' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/39' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/10' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/34' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/38' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/11' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/16' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/32' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/30' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/14' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/18' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/13' 2024-03-28T19:00:40.447 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/33' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/17' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/35' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './154/37' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './154' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/19' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/15' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/31' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/36' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/20' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/12' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/39' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/10' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/34' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/38' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/11' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/16' 2024-03-28T19:00:40.448 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/32' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/30' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/14' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/18' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/13' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/33' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/17' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/35' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './155/37' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './155' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/19' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/15' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/31' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/36' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/20' 2024-03-28T19:00:40.449 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/12' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/39' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/10' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/34' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/38' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/11' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/16' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/32' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/30' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/14' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/18' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/13' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/33' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/17' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/35' 2024-03-28T19:00:40.450 INFO:tasks.workunit.client.0.smithi027.stdout:removed './156/37' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './156' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/19' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/15' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/31' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/36' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/20' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/12' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/39' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/10' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/34' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/38' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/11' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/16' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/32' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/30' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/14' 2024-03-28T19:00:40.451 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/18' 2024-03-28T19:00:40.452 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/13' 2024-03-28T19:00:40.452 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/33' 2024-03-28T19:00:40.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.563+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71d4069640 con 0x7f71d40c2290 2024-03-28T19:00:40.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.564+0000 7f71d1600640 1 -- 172.21.15.27:0/3566245427 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6248 (secure 0 0 0) 0x7f71c8059110 con 0x7f71d40c2290 2024-03-28T19:00:40.563 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:40.564 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:40.564 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 43 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71b406bbd0 msgr2=0x7f71b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71b406bbd0 0x7f71b406e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f71c0036000 tx=0x7f71c0030000 comp rx=0 tx=0).stop 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 msgr2=0x7f71d41ebbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d41ebbe0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f71c800d800 tx=0x7f71c800dcc0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 shutdown_connections 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71b406bbd0 0x7f71b406e090 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71d4152ca0 0x7f71d41546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71d40c74e0 0x7f71d41ee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 --2- 172.21.15.27:0/3566245427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d40c2290 0x7f71d41ebbe0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 >> 172.21.15.27:0/3566245427 conn(0x7f71d406d980 msgr2=0x7f71d41509b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 shutdown_connections 2024-03-28T19:00:40.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:40.567+0000 7f71dac00640 1 -- 172.21.15.27:0/3566245427 wait complete. 2024-03-28T19:00:40.588 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:00:42.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.817+0000 7fd744000640 1 Processor -- start 2024-03-28T19:00:42.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.817+0000 7fd744000640 1 -- start start 2024-03-28T19:00:42.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.819+0000 7fd744000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c153ed0 0x7fd73c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:42.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.819+0000 7fd744000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd73c06eb80 0x7fd73c15a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:42.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.819+0000 7fd744000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c06efb0 con 0x7fd73c06e6f0 2024-03-28T19:00:42.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.819+0000 7fd744000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c0c6990 con 0x7fd73c06eb80 2024-03-28T19:00:42.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.819+0000 7fd744000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0c3830 con 0x7fd73c153ed0 2024-03-28T19:00:42.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.821+0000 7fd741600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c153ed0 0x7fd73c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:42.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.822+0000 7fd742000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd73c06eb80 0x7fd73c15a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:42.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.822+0000 7fd742000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd73c06eb80 0x7fd73c15a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:34040/0 (socket says 172.21.15.27:34040) 2024-03-28T19:00:42.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.822+0000 7fd742000640 1 -- 172.21.15.27:0/2688416132 learned_addr learned my addr 172.21.15.27:0/2688416132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.825+0000 7fd741600640 1 -- 172.21.15.27:0/2688416132 >> v2:172.21.15.155:3300/0 conn(0x7fd73c06eb80 msgr2=0x7fd73c15a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.825+0000 7fd741600640 1 --2- 172.21.15.27:0/2688416132 >> v2:172.21.15.155:3300/0 conn(0x7fd73c06eb80 0x7fd73c15a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.825+0000 7fd741600640 1 -- 172.21.15.27:0/2688416132 >> v1:172.21.15.155:6789/0 conn(0x7fd73c06e6f0 legacy=0x7fd73c159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:42.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.825+0000 7fd741600640 1 -- 172.21.15.27:0/2688416132 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c06d1f0 con 0x7fd73c153ed0 2024-03-28T19:00:42.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.825+0000 7fd742000640 1 --2- 172.21.15.27:0/2688416132 >> v2:172.21.15.155:3300/0 conn(0x7fd73c06eb80 0x7fd73c15a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:00:42.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.829+0000 7fd741600640 1 --2- 172.21.15.27:0/2688416132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c153ed0 0x7fd73c06ccb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd7300097a0 tx=0x7fd73002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fff68f01a8773545 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:42.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.830+0000 7fd73be00640 1 -- 172.21.15.27:0/2688416132 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730035020 con 0x7fd73c153ed0 2024-03-28T19:00:42.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.830+0000 7fd73be00640 1 -- 172.21.15.27:0/2688416132 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd730031810 con 0x7fd73c153ed0 2024-03-28T19:00:42.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.830+0000 7fd73be00640 1 -- 172.21.15.27:0/2688416132 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730031be0 con 0x7fd73c153ed0 2024-03-28T19:00:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.831+0000 7fd744000640 1 -- 172.21.15.27:0/2688416132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c153ed0 msgr2=0x7fd73c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:42.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.831+0000 7fd744000640 1 --2- 172.21.15.27:0/2688416132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c153ed0 0x7fd73c06ccb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd7300097a0 tx=0x7fd73002b8b0 comp rx=0 tx=0).stop 2024-03-28T19:00:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.833+0000 7fd744000640 1 -- 172.21.15.27:0/2688416132 shutdown_connections 2024-03-28T19:00:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.833+0000 7fd744000640 1 --2- 172.21.15.27:0/2688416132 >> v2:172.21.15.155:3300/0 conn(0x7fd73c06eb80 0x7fd73c15a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.833+0000 7fd744000640 1 --2- 172.21.15.27:0/2688416132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c153ed0 0x7fd73c06ccb0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd7300097a0 tx=0x7fd73002b8b0 comp rx=0 tx=0).stop 2024-03-28T19:00:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.833+0000 7fd744000640 1 -- 172.21.15.27:0/2688416132 >> 172.21.15.27:0/2688416132 conn(0x7fd73c06b9d0 msgr2=0x7fd73c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:42.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.835+0000 7fd744000640 1 -- 172.21.15.27:0/2688416132 shutdown_connections 2024-03-28T19:00:42.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.836+0000 7fd744000640 1 -- 172.21.15.27:0/2688416132 wait complete. 2024-03-28T19:00:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.839+0000 7fd744000640 1 Processor -- start 2024-03-28T19:00:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.840+0000 7fd744000640 1 -- start start 2024-03-28T19:00:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.841+0000 7fd744000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd73c06e6f0 0x7fd73c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.841+0000 7fd744000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c06eb80 0x7fd73c0aafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.841+0000 7fd744000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73c155610 0x7fd73c152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.842+0000 7fd741600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd73c06e6f0 0x7fd73c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.842+0000 7fd741600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd73c06e6f0 0x7fd73c15d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:34054/0 (socket says 172.21.15.27:34054) 2024-03-28T19:00:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.843+0000 7fd740c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c06eb80 0x7fd73c0aafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.843+0000 7fd742000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73c155610 0x7fd73c152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.843+0000 7fd740c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c06eb80 0x7fd73c0aafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:54120/0 (socket says 172.21.15.27:54120) 2024-03-28T19:00:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.843+0000 7fd742000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73c155610 0x7fd73c152100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40462/0 (socket says 172.21.15.27:40462) 2024-03-28T19:00:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.843+0000 7fd744000640 1 -- 172.21.15.27:0/3273838456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0c3db0 con 0x7fd73c155610 2024-03-28T19:00:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.843+0000 7fd744000640 1 -- 172.21.15.27:0/3273838456 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0c3ad0 con 0x7fd73c06e6f0 2024-03-28T19:00:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.843+0000 7fd744000640 1 -- 172.21.15.27:0/3273838456 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c15a810 con 0x7fd73c06eb80 2024-03-28T19:00:42.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.843+0000 7fd741600640 1 -- 172.21.15.27:0/3273838456 learned_addr learned my addr 172.21.15.27:0/3273838456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:42.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.845+0000 7fd741600640 1 -- 172.21.15.27:0/3273838456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c06eb80 msgr2=0x7fd73c0aafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:42.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.845+0000 7fd741600640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c06eb80 0x7fd73c0aafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:42.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.845+0000 7fd741600640 1 -- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73c155610 msgr2=0x7fd73c152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:42.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.845+0000 7fd741600640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73c155610 0x7fd73c152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:42.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.845+0000 7fd741600640 1 -- 172.21.15.27:0/3273838456 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c1572d0 con 0x7fd73c06e6f0 2024-03-28T19:00:42.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.846+0000 7fd741600640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd73c06e6f0 0x7fd73c15d9e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd7300098c0 tx=0x7fd7300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:42.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.847+0000 7fd739600640 1 -- 172.21.15.27:0/3273838456 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730041070 con 0x7fd73c06e6f0 2024-03-28T19:00:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.847+0000 7fd739600640 1 -- 172.21.15.27:0/3273838456 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73003ad00 con 0x7fd73c06e6f0 2024-03-28T19:00:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.847+0000 7fd744000640 1 -- 172.21.15.27:0/3273838456 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c154620 con 0x7fd73c06e6f0 2024-03-28T19:00:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.847+0000 7fd744000640 1 -- 172.21.15.27:0/3273838456 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c15c5c0 con 0x7fd73c06e6f0 2024-03-28T19:00:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.847+0000 7fd739600640 1 -- 172.21.15.27:0/3273838456 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730035040 con 0x7fd73c06e6f0 2024-03-28T19:00:42.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.849+0000 7fd72ea00640 1 -- 172.21.15.27:0/3273838456 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73c0c5ec0 con 0x7fd73c06e6f0 2024-03-28T19:00:42.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.851+0000 7fd739600640 1 -- 172.21.15.27:0/3273838456 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd73004c060 con 0x7fd73c06e6f0 2024-03-28T19:00:42.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.852+0000 7fd739600640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd71c06bc20 0x7fd71c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:42.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.852+0000 7fd739600640 1 -- 172.21.15.27:0/3273838456 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7095+0+0 (secure 0 0 0) 0x7fd7300ac850 con 0x7fd73c06e6f0 2024-03-28T19:00:42.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.853+0000 7fd740c00640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd71c06bc20 0x7fd71c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:42.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.853+0000 7fd740c00640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd71c06bc20 0x7fd71c06e0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0a9220 tx=0x7fd728009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:42.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:42.859+0000 7fd739600640 1 -- 172.21.15.27:0/3273838456 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd73c0c5ec0 con 0x7fd73c06e6f0 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/17' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/35' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './157/37' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './157' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/19' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/15' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/31' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/36' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/20' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/12' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/39' 2024-03-28T19:00:42.928 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/10' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/34' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/38' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/11' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/16' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/32' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/30' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/14' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/18' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/13' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/33' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/17' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/35' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed './158/37' 2024-03-28T19:00:42.929 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './158' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/19' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/15' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/31' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/36' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/20' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/12' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/39' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/10' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/34' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/38' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/11' 2024-03-28T19:00:42.930 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/16' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/32' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/30' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/14' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/18' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/13' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/33' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/17' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/35' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './159/37' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './159' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/19' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/15' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/31' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/36' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/20' 2024-03-28T19:00:42.931 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/12' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/39' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/10' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/34' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/38' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/11' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/16' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/32' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/30' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/14' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/18' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/13' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/33' 2024-03-28T19:00:42.932 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/17' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/35' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './160/37' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './160' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/19' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/15' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/31' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/36' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/20' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/12' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/39' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/10' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/34' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/38' 2024-03-28T19:00:42.933 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/11' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/16' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/32' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/30' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/14' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/18' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/13' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/33' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/17' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/35' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './161/37' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './161' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/19' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/15' 2024-03-28T19:00:42.934 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/31' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/36' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/20' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/12' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/39' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/10' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/34' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/38' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/11' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/16' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/32' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/30' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/14' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/18' 2024-03-28T19:00:42.935 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/13' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/33' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/17' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/35' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './162/37' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './162' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/19' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/15' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/31' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/36' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/20' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/12' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/39' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/10' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/34' 2024-03-28T19:00:42.936 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/38' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/11' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/16' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/32' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/30' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/14' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/18' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/13' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/33' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/17' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/35' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './163/37' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './163' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/19' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/15' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/31' 2024-03-28T19:00:42.937 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/36' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/20' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/12' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/39' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/10' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/34' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/38' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/11' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/16' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/32' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/30' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/14' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/18' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/13' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/33' 2024-03-28T19:00:42.938 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/17' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/35' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './164/37' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './164' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/19' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/15' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/31' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/36' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/20' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/12' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/39' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/10' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/34' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/38' 2024-03-28T19:00:42.939 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/11' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/16' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/32' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/30' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/14' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/18' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/13' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/33' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/17' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/35' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './165/37' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './165' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/19' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/15' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/31' 2024-03-28T19:00:42.940 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/36' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/20' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/12' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/39' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/10' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/34' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/38' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/11' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/16' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/32' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/30' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/14' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/18' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/13' 2024-03-28T19:00:42.941 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/33' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/17' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/35' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './166/37' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './166' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/19' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/15' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/31' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/36' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/20' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/12' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/39' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/10' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/34' 2024-03-28T19:00:42.942 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/38' 2024-03-28T19:00:43.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.154+0000 7fd72ea00640 1 -- 172.21.15.27:0/3273838456 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd73c04f400 con 0x7fd73c06e6f0 2024-03-28T19:00:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.155+0000 7fd739600640 1 -- 172.21.15.27:0/3273838456 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6248 (secure 0 0 0) 0x7fd73007a8f0 con 0x7fd73c06e6f0 2024-03-28T19:00:43.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:43.156 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5236,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:14.296650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5433":{"gid":5433,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":7,"addr":"172.21.15.27:6837/700464224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":700464224},{"type":"v1","addr":"172.21.15.27:6837","nonce":700464224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5433,"qdb_cluster":[5433]},"id":2}]} 2024-03-28T19:00:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 43 2024-03-28T19:00:43.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.158+0000 7fd72ea00640 1 -- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd71c06bc20 msgr2=0x7fd71c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:43.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.158+0000 7fd72ea00640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd71c06bc20 0x7fd71c06e0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0a9220 tx=0x7fd728009040 comp rx=0 tx=0).stop 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.158+0000 7fd72ea00640 1 -- 172.21.15.27:0/3273838456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd73c06e6f0 msgr2=0x7fd73c15d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.158+0000 7fd72ea00640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd73c06e6f0 0x7fd73c15d9e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd7300098c0 tx=0x7fd7300046c0 comp rx=0 tx=0).stop 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.159+0000 7fd72ea00640 1 -- 172.21.15.27:0/3273838456 shutdown_connections 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.159+0000 7fd72ea00640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd71c06bc20 0x7fd71c06e0e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.159+0000 7fd72ea00640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd73c155610 0x7fd73c152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.159+0000 7fd72ea00640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd73c06eb80 0x7fd73c0aafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.159+0000 7fd72ea00640 1 --2- 172.21.15.27:0/3273838456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd73c06e6f0 0x7fd73c15d9e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.159+0000 7fd72ea00640 1 -- 172.21.15.27:0/3273838456 >> 172.21.15.27:0/3273838456 conn(0x7fd73c06b9d0 msgr2=0x7fd73c0a9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.159+0000 7fd72ea00640 1 -- 172.21.15.27:0/3273838456 shutdown_connections 2024-03-28T19:00:43.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:43.159+0000 7fd72ea00640 1 -- 172.21.15.27:0/3273838456 wait complete. 2024-03-28T19:00:43.178 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:00:44.972 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/11' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/16' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/32' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/30' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/14' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/18' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/13' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/33' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/17' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/35' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './167/37' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './167' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/19' 2024-03-28T19:00:44.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/15' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/31' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/36' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/20' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/12' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/39' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/10' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/34' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/38' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/11' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/16' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/32' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/30' 2024-03-28T19:00:44.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/14' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/18' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/13' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/33' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/17' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/35' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './168/37' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './168' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/19' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/15' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/31' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/36' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/20' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/12' 2024-03-28T19:00:44.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/39' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/10' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/34' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/38' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/11' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/16' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/32' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/30' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/14' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/18' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/13' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/33' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/17' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/35' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './169/37' 2024-03-28T19:00:44.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './169' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/19' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/15' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/31' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/36' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/20' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/12' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/39' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/10' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/34' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/38' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/11' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/16' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/32' 2024-03-28T19:00:44.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/30' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/14' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/18' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/13' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/33' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/17' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/35' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './170/37' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './170' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/19' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/15' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/31' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/36' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/20' 2024-03-28T19:00:44.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/12' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/39' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/10' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/34' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/38' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/11' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/16' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/32' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/30' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/14' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/18' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/13' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/33' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/17' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/35' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './171/37' 2024-03-28T19:00:44.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './171' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/19' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/15' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/31' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/36' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/20' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/12' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/39' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/10' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/34' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/38' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/11' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/16' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/32' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/30' 2024-03-28T19:00:44.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/14' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/18' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/13' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/33' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/17' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/35' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './172/37' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './172' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/19' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/15' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/31' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/36' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/20' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/12' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/39' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/10' 2024-03-28T19:00:44.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/34' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/38' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/11' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/16' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/32' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/30' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/14' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/18' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/13' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/33' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/17' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/35' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed './173/37' 2024-03-28T19:00:44.982 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './173' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/19' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/15' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/31' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/36' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/20' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/12' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/39' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/10' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/34' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/38' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/11' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/16' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/32' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/30' 2024-03-28T19:00:44.983 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/14' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/18' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/13' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/33' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/17' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/35' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './174/37' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './174' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/19' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/15' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/31' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/36' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/20' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/12' 2024-03-28T19:00:44.984 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/39' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/10' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/34' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/38' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/11' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/16' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/32' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/30' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/14' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/18' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/13' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/33' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/17' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/35' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed './175/37' 2024-03-28T19:00:44.985 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './175' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/19' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/15' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/31' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/36' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/20' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/12' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/39' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/10' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/34' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/38' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/11' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/16' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/32' 2024-03-28T19:00:44.986 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/30' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/14' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/18' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/13' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/33' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/17' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/35' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './176/37' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './176' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/19' 2024-03-28T19:00:44.987 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/15' 2024-03-28T19:00:45.179 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.423+0000 7fceac200640 1 Processor -- start 2024-03-28T19:00:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.424+0000 7fceac200640 1 -- start start 2024-03-28T19:00:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.425+0000 7fceac200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c3ea0 0x7fcea4159610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.425+0000 7fceac200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40c1600 con 0x7fcea406ee00 2024-03-28T19:00:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.425+0000 7fceac200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40c18e0 con 0x7fcea40c60e0 2024-03-28T19:00:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.425+0000 7fceac200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40c23b0 con 0x7fcea40c3ea0 2024-03-28T19:00:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.426+0000 7fcea8e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcea406ee00 0x7fcea4158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54386/0 (socket says 172.21.15.27:54386) 2024-03-28T19:00:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.426+0000 7fcea8e00640 1 -- 172.21.15.27:0/1036977331 learned_addr learned my addr 172.21.15.27:0/1036977331 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.426+0000 7fceaa200640 1 --2- 172.21.15.27:0/1036977331 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c3ea0 0x7fcea4159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.426+0000 7fcea3e00640 1 -- 172.21.15.27:0/1036977331 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994486394 0 0) 0x7fcea40c1600 con 0x7fcea406ee00 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.426+0000 7fcea3e00640 1 -- 172.21.15.27:0/1036977331 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c003520 con 0x7fcea406ee00 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.427+0000 7fcea3e00640 1 -- 172.21.15.27:0/1036977331 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309170460 0 0) 0x7fcea40c18e0 con 0x7fcea40c60e0 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.427+0000 7fcea3e00640 1 -- 172.21.15.27:0/1036977331 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea40c1600 con 0x7fcea40c60e0 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.427+0000 7fceaa200640 1 -- 172.21.15.27:0/1036977331 >> v1:172.21.15.155:6789/0 conn(0x7fcea40c60e0 legacy=0x7fcea406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.427+0000 7fceaa200640 1 -- 172.21.15.27:0/1036977331 >> v1:172.21.15.27:6789/0 conn(0x7fcea406ee00 legacy=0x7fcea4158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.427+0000 7fcea3e00640 1 -- 172.21.15.27:0/1036977331 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322223231 0 0) 0x7fce8c003520 con 0x7fcea406ee00 2024-03-28T19:00:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.427+0000 7fceaa200640 1 -- 172.21.15.27:0/1036977331 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40c1bc0 con 0x7fcea40c3ea0 2024-03-28T19:00:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.428+0000 7fceaa200640 1 --2- 172.21.15.27:0/1036977331 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c3ea0 0x7fcea4159610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcea40c2530 tx=0x7fce9c02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d1a3f5dd803caad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.429+0000 7fcea3e00640 1 -- 172.21.15.27:0/1036977331 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c039070 con 0x7fcea40c3ea0 2024-03-28T19:00:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.429+0000 7fcea3e00640 1 -- 172.21.15.27:0/1036977331 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c02c880 con 0x7fcea40c3ea0 2024-03-28T19:00:45.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.429+0000 7fcea3e00640 1 -- 172.21.15.27:0/1036977331 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90001670 con 0x7fcea40c3ea0 2024-03-28T19:00:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.430+0000 7fceac200640 1 -- 172.21.15.27:0/1036977331 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c3ea0 msgr2=0x7fcea4159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.430+0000 7fceac200640 1 --2- 172.21.15.27:0/1036977331 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c3ea0 0x7fcea4159610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcea40c2530 tx=0x7fce9c02bd50 comp rx=0 tx=0).stop 2024-03-28T19:00:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.430+0000 7fceac200640 1 -- 172.21.15.27:0/1036977331 shutdown_connections 2024-03-28T19:00:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.430+0000 7fceac200640 1 --2- 172.21.15.27:0/1036977331 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c3ea0 0x7fcea4159610 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.430+0000 7fceac200640 1 -- 172.21.15.27:0/1036977331 >> 172.21.15.27:0/1036977331 conn(0x7fcea406b9d0 msgr2=0x7fcea40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.430+0000 7fceac200640 1 -- 172.21.15.27:0/1036977331 shutdown_connections 2024-03-28T19:00:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.430+0000 7fceac200640 1 -- 172.21.15.27:0/1036977331 wait complete. 2024-03-28T19:00:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.432+0000 7fceac200640 1 Processor -- start 2024-03-28T19:00:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.432+0000 7fceac200640 1 -- start start 2024-03-28T19:00:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fceac200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea406ee00 0x7fcea4157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fceac200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcea40c3ea0 0x7fcea4152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fceac200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c60e0 0x7fcea4153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fceac200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40c64c0 con 0x7fcea406ee00 2024-03-28T19:00:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fceac200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4159b50 con 0x7fcea40c3ea0 2024-03-28T19:00:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fceac200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40c6640 con 0x7fcea40c60e0 2024-03-28T19:00:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fcea9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea406ee00 0x7fcea4157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fcea9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea406ee00 0x7fcea4157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40470/0 (socket says 172.21.15.27:40470) 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fcea9800640 1 -- 172.21.15.27:0/1547747538 learned_addr learned my addr 172.21.15.27:0/1547747538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.434+0000 7fcea8e00640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcea40c3ea0 0x7fcea4152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fceaa200640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c60e0 0x7fcea4153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fcea9800640 1 -- 172.21.15.27:0/1547747538 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c60e0 msgr2=0x7fcea4153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fcea9800640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c60e0 0x7fcea4153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fcea9800640 1 -- 172.21.15.27:0/1547747538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcea40c3ea0 msgr2=0x7fcea4152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fcea9800640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcea40c3ea0 0x7fcea4152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fcea9800640 1 -- 172.21.15.27:0/1547747538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea41ef0b0 con 0x7fcea406ee00 2024-03-28T19:00:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fcea9800640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea406ee00 0x7fcea4157eb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c490 tx=0x7fce9800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fcea1600640 1 -- 172.21.15.27:0/1547747538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98017070 con 0x7fcea406ee00 2024-03-28T19:00:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.435+0000 7fcea1600640 1 -- 172.21.15.27:0/1547747538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce980052d0 con 0x7fcea406ee00 2024-03-28T19:00:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.436+0000 7fcea1600640 1 -- 172.21.15.27:0/1547747538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98003830 con 0x7fcea406ee00 2024-03-28T19:00:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.436+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea40aad10 con 0x7fcea406ee00 2024-03-28T19:00:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.436+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea41583f0 con 0x7fcea406ee00 2024-03-28T19:00:45.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.437+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce80005290 con 0x7fcea406ee00 2024-03-28T19:00:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.438+0000 7fcea1600640 1 -- 172.21.15.27:0/1547747538 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fce9801a070 con 0x7fcea406ee00 2024-03-28T19:00:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.439+0000 7fcea1600640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce8406bc00 0x7fce8406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.440+0000 7fcea1600640 1 -- 172.21.15.27:0/1547747538 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7153+0+0 (secure 0 0 0) 0x7fce9808a510 con 0x7fcea406ee00 2024-03-28T19:00:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.440+0000 7fcea8e00640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce8406bc00 0x7fce8406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.441+0000 7fcea8e00640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce8406bc00 0x7fce8406e0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcea40a7060 tx=0x7fce90009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.445+0000 7fcea1600640 1 -- 172.21.15.27:0/1547747538 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fce98058930 con 0x7fcea406ee00 2024-03-28T19:00:45.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.802+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce80005580 con 0x7fcea406ee00 2024-03-28T19:00:45.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.803+0000 7fcea1600640 1 -- 172.21.15.27:0/1547747538 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5456 (secure 0 0 0) 0x7fce98058230 con 0x7fcea406ee00 2024-03-28T19:00:45.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:45.803 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:45.624489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:00:45.804 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 45 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.805+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce8406bc00 msgr2=0x7fce8406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.805+0000 7fceac200640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce8406bc00 0x7fce8406e0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcea40a7060 tx=0x7fce90009040 comp rx=0 tx=0).stop 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.805+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea406ee00 msgr2=0x7fcea4157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.805+0000 7fceac200640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea406ee00 0x7fcea4157eb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fce9800c490 tx=0x7fce9800c950 comp rx=0 tx=0).stop 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.806+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 shutdown_connections 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.806+0000 7fceac200640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce8406bc00 0x7fce8406e0c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.806+0000 7fceac200640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcea40c60e0 0x7fcea4153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.806+0000 7fceac200640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcea40c3ea0 0x7fcea4152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:45.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.806+0000 7fceac200640 1 --2- 172.21.15.27:0/1547747538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcea406ee00 0x7fcea4157eb0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:45.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.806+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 >> 172.21.15.27:0/1547747538 conn(0x7fcea406b9d0 msgr2=0x7fcea40a7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:45.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.806+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 shutdown_connections 2024-03-28T19:00:45.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:45.806+0000 7fceac200640 1 -- 172.21.15.27:0/1547747538 wait complete. 2024-03-28T19:00:45.826 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-03-28T19:00:45.826 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:00:45.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.065+0000 7f3fe8800640 1 Processor -- start 2024-03-28T19:00:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.065+0000 7f3fe8800640 1 -- start start 2024-03-28T19:00:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.067+0000 7f3fe8800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fe00c1f50 0x7f3fe00ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.067+0000 7f3fe8800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3fe00abd90 0x7f3fe00aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.067+0000 7f3fe8800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe00ac1c0 con 0x7f3fe00c5e80 2024-03-28T19:00:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.067+0000 7f3fe8800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe006d650 con 0x7f3fe00c1f50 2024-03-28T19:00:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.067+0000 7f3fe8800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe006e1d0 con 0x7f3fe00abd90 2024-03-28T19:00:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.067+0000 7f3fe5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fe00c1f50 0x7f3fe00ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.067+0000 7f3fe5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fe00c1f50 0x7f3fe00ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40486/0 (socket says 172.21.15.27:40486) 2024-03-28T19:00:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.067+0000 7f3fe5400640 1 -- 172.21.15.27:0/3514633845 learned_addr learned my addr 172.21.15.27:0/3514633845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.068+0000 7f3fe6800640 1 --2- 172.21.15.27:0/3514633845 >> v2:172.21.15.155:3300/0 conn(0x7f3fe00abd90 0x7f3fe00aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.068+0000 7f3fe5400640 1 -- 172.21.15.27:0/3514633845 >> v2:172.21.15.155:3300/0 conn(0x7f3fe00abd90 msgr2=0x7f3fe00aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.068+0000 7f3fe5400640 1 --2- 172.21.15.27:0/3514633845 >> v2:172.21.15.155:3300/0 conn(0x7f3fe00abd90 0x7f3fe00aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.068+0000 7f3fe5400640 1 -- 172.21.15.27:0/3514633845 >> v1:172.21.15.155:6789/0 conn(0x7f3fe00c5e80 legacy=0x7f3fe00c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.068+0000 7f3fe5400640 1 -- 172.21.15.27:0/3514633845 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe006d8a0 con 0x7f3fe00c1f50 2024-03-28T19:00:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.069+0000 7f3fe5400640 1 --2- 172.21.15.27:0/3514633845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00ab850 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3fe006eff0 tx=0x7f3fd002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d63f7de26beeece server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.069+0000 7f3fe4a00640 1 -- 172.21.15.27:0/3514633845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0038070 con 0x7f3fe00c1f50 2024-03-28T19:00:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.069+0000 7f3fe4a00640 1 -- 172.21.15.27:0/3514633845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd002f050 con 0x7f3fe00c1f50 2024-03-28T19:00:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.070+0000 7f3fe4a00640 1 -- 172.21.15.27:0/3514633845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0031970 con 0x7f3fe00c1f50 2024-03-28T19:00:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.070+0000 7f3fe8800640 1 -- 172.21.15.27:0/3514633845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 msgr2=0x7f3fe00ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.070+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3514633845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00ab850 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3fe006eff0 tx=0x7f3fd002b7f0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.070+0000 7f3fe8800640 1 -- 172.21.15.27:0/3514633845 shutdown_connections 2024-03-28T19:00:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.070+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3514633845 >> v2:172.21.15.155:3300/0 conn(0x7f3fe00abd90 0x7f3fe00aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.070+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3514633845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00ab850 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.070+0000 7f3fe8800640 1 -- 172.21.15.27:0/3514633845 >> 172.21.15.27:0/3514633845 conn(0x7f3fe006b9d0 msgr2=0x7f3fe00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.071+0000 7f3fe8800640 1 -- 172.21.15.27:0/3514633845 shutdown_connections 2024-03-28T19:00:46.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.071+0000 7f3fe8800640 1 -- 172.21.15.27:0/3514633845 wait complete. 2024-03-28T19:00:46.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.073+0000 7f3fe8800640 1 Processor -- start 2024-03-28T19:00:46.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.073+0000 7f3fe8800640 1 -- start start 2024-03-28T19:00:46.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.074+0000 7f3fe8800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe00abd90 0x7f3fe01eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.074+0000 7f3fe8800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.074+0000 7f3fe8800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fe00c5e80 0x7f3fe01512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.074+0000 7f3fe8800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00c6260 con 0x7f3fe00c1f50 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.074+0000 7f3fe8800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe006de60 con 0x7f3fe00c5e80 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.074+0000 7f3fe8800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00c63e0 con 0x7f3fe00abd90 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00a6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40500/0 (socket says 172.21.15.27:40500) 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5400640 1 -- 172.21.15.27:0/3252822783 learned_addr learned my addr 172.21.15.27:0/3252822783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5e00640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe00abd90 0x7f3fe01eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe6800640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fe00c5e80 0x7f3fe01512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5400640 1 -- 172.21.15.27:0/3252822783 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe00abd90 msgr2=0x7f3fe01eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5400640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe00abd90 0x7f3fe01eeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5400640 1 -- 172.21.15.27:0/3252822783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fe00c5e80 msgr2=0x7f3fe01512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5400640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fe00c5e80 0x7f3fe01512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.075+0000 7f3fe5400640 1 -- 172.21.15.27:0/3252822783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe0153460 con 0x7f3fe00c1f50 2024-03-28T19:00:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.076+0000 7f3fe5400640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00a6920 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009960 tx=0x7f3fd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.076+0000 7f3fda000640 1 -- 172.21.15.27:0/3252822783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0042020 con 0x7f3fe00c1f50 2024-03-28T19:00:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.076+0000 7f3fda000640 1 -- 172.21.15.27:0/3252822783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd003d890 con 0x7f3fe00c1f50 2024-03-28T19:00:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.076+0000 7f3fda000640 1 -- 172.21.15.27:0/3252822783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0038040 con 0x7f3fe00c1f50 2024-03-28T19:00:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.076+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe0153740 con 0x7f3fe00c1f50 2024-03-28T19:00:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.077+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe00a6e60 con 0x7f3fe00c1f50 2024-03-28T19:00:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.078+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe00a9cb0 con 0x7f3fe00c1f50 2024-03-28T19:00:46.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.079+0000 7f3fda000640 1 -- 172.21.15.27:0/3252822783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3fd0045020 con 0x7f3fe00c1f50 2024-03-28T19:00:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.080+0000 7f3fda000640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3fc406bbd0 0x7f3fc406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.080+0000 7f3fda000640 1 -- 172.21.15.27:0/3252822783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7153+0+0 (secure 0 0 0) 0x7f3fd00aca60 con 0x7f3fe00c1f50 2024-03-28T19:00:46.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.081+0000 7f3fe5e00640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3fc406bbd0 0x7f3fc406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:46.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.082+0000 7f3fe5e00640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3fc406bbd0 0x7f3fc406e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4006fd0 tx=0x7f3fd4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:46.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.087+0000 7f3fda000640 1 -- 172.21.15.27:0/3252822783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3fe00a9cb0 con 0x7f3fe00c1f50 2024-03-28T19:00:46.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/31' 2024-03-28T19:00:46.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/36' 2024-03-28T19:00:46.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/20' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/12' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/39' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/10' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/34' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/38' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/11' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/16' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/32' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/30' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/14' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/18' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/13' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/33' 2024-03-28T19:00:46.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/17' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/35' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './177/37' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './177' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/19' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/15' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/31' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/36' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/20' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/12' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/39' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/10' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/34' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/38' 2024-03-28T19:00:46.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/11' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/16' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/32' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/30' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/14' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/18' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/13' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/33' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/17' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/35' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './178/37' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './178' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/19' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/15' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/31' 2024-03-28T19:00:46.421 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/36' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/20' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/12' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/39' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/10' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/34' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/38' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/11' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/16' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/32' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/30' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/14' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/18' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/13' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/33' 2024-03-28T19:00:46.422 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/17' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/35' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './179/37' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './179' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/19' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/15' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/31' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/36' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/20' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/12' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/39' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/10' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/34' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/38' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/11' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/16' 2024-03-28T19:00:46.423 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/32' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/30' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/14' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/18' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/13' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/33' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/17' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/35' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './180/37' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './180' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/19' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/15' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/31' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/36' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/20' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/12' 2024-03-28T19:00:46.424 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/39' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/10' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/34' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/38' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/11' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/16' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/32' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/30' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/14' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/18' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/13' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/33' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/17' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/35' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './181/37' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './181' 2024-03-28T19:00:46.425 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/19' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/15' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/31' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/36' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/20' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/12' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/39' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/10' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/34' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/38' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/11' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/16' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/32' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/30' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/14' 2024-03-28T19:00:46.426 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/18' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/13' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/33' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/17' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/35' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './182/37' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './182' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/19' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/15' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/31' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/36' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/20' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/12' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/39' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/10' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/34' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/38' 2024-03-28T19:00:46.427 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/11' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/16' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/32' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/30' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/14' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/18' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/13' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/33' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/17' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/35' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './183/37' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './183' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/19' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/15' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/31' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/36' 2024-03-28T19:00:46.428 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/20' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/12' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/39' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/10' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/34' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/38' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/11' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/16' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/32' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/30' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/14' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/18' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/13' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/33' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/17' 2024-03-28T19:00:46.429 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/35' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './184/37' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './184' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/19' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/15' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/31' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/36' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/20' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/12' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/39' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/10' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/34' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/38' 2024-03-28T19:00:46.430 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/11' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/16' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/32' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/30' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/14' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/18' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/13' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/33' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/17' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/35' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './185/37' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './185' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/19' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/15' 2024-03-28T19:00:46.431 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/31' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/36' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/20' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/12' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/39' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/10' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/34' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/38' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/11' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/16' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/32' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/30' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/14' 2024-03-28T19:00:46.432 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/18' 2024-03-28T19:00:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.454+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fe004f400 con 0x7f3fe00c1f50 2024-03-28T19:00:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.455+0000 7f3fda000640 1 -- 172.21.15.27:0/3252822783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5456 (secure 0 0 0) 0x7f3fd004c030 con 0x7f3fe00c1f50 2024-03-28T19:00:46.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:46.456 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:45.624489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:00:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 45 2024-03-28T19:00:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3fc406bbd0 msgr2=0x7f3fc406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3fc406bbd0 0x7f3fc406e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4006fd0 tx=0x7f3fd4008040 comp rx=0 tx=0).stop 2024-03-28T19:00:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 msgr2=0x7f3fe00a6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00a6920 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009960 tx=0x7f3fd00046c0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 shutdown_connections 2024-03-28T19:00:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3fc406bbd0 0x7f3fc406e090 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fe00c5e80 0x7f3fe01512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fe00c1f50 0x7f3fe00a6920 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 --2- 172.21.15.27:0/3252822783 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe00abd90 0x7f3fe01eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.458+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 >> 172.21.15.27:0/3252822783 conn(0x7f3fe006b9d0 msgr2=0x7f3fe0155230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.459+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 shutdown_connections 2024-03-28T19:00:46.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:46.459+0000 7f3fe8800640 1 -- 172.21.15.27:0/3252822783 wait complete. 2024-03-28T19:00:46.480 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5325": { "addr": "172.21.15.27:6836/181571966", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 181571966, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 181571966, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5325, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T18:59:54.159296+0000", "qdb_cluster": [ 5325 ], "qdb_leader": 5325, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5325 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5236": { "addr": "172.21.15.155:6834/620038090", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 620038090, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 620038090, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:00:45.624489+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5236 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:00:48.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:48.557 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/13' 2024-03-28T19:00:48.610 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/33' 2024-03-28T19:00:48.610 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/17' 2024-03-28T19:00:48.610 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/35' 2024-03-28T19:00:48.610 INFO:tasks.workunit.client.0.smithi027.stdout:removed './186/37' 2024-03-28T19:00:48.610 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './186' 2024-03-28T19:00:48.610 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/19' 2024-03-28T19:00:48.610 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/15' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/31' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/36' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/20' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/12' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/39' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/10' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/34' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/38' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/11' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/16' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/32' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/30' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/14' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/18' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/13' 2024-03-28T19:00:48.611 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/33' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/17' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/35' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './187/37' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './187' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/19' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/15' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/31' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/36' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/20' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/12' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/39' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/10' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/34' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/38' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/11' 2024-03-28T19:00:48.612 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/16' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/32' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/30' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/14' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/18' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/13' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/33' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/17' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/35' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './188/37' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './188' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/19' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/15' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/31' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/36' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/20' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/12' 2024-03-28T19:00:48.613 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/39' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/10' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/34' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/38' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/11' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/16' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/32' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/30' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/14' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/18' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/13' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/33' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/17' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/35' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './189/37' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './189' 2024-03-28T19:00:48.614 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/19' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/15' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/31' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/36' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/20' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/12' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/39' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/10' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/34' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/38' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/11' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/16' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/32' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/30' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/14' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/18' 2024-03-28T19:00:48.615 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/13' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/33' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/17' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/35' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './190/37' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './190' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/19' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/15' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/31' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/36' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/20' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/12' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/39' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/10' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/34' 2024-03-28T19:00:48.616 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/38' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/11' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/16' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/32' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/30' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/14' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/18' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/13' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/33' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/17' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/35' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './191/37' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './191' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/19' 2024-03-28T19:00:48.617 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/15' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/31' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/36' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/20' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/12' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/39' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/10' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/34' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/38' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/11' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/16' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/32' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/30' 2024-03-28T19:00:48.618 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/14' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/18' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/13' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/33' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/17' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/35' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './192/37' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './192' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/19' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/15' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/31' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/36' 2024-03-28T19:00:48.619 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/20' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/12' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/39' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/10' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/34' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/38' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/11' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/16' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/32' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/30' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/14' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/18' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/13' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/33' 2024-03-28T19:00:48.620 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/17' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/35' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './193/37' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './193' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/19' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/15' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/31' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/36' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/20' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/12' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/39' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/10' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/34' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/38' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/11' 2024-03-28T19:00:48.621 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/16' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/32' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/30' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/14' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/18' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/13' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/33' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/17' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/35' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './194/37' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './194' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/19' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/15' 2024-03-28T19:00:48.622 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/31' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/36' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/20' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/12' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/39' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/10' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/34' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/38' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/11' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/16' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/32' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/30' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/14' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/18' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/13' 2024-03-28T19:00:48.623 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/33' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/17' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/35' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './195/37' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './195' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/19' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/15' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/31' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/36' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/20' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/12' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/39' 2024-03-28T19:00:48.624 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/10' 2024-03-28T19:00:48.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.715+0000 7f94c4000640 1 Processor -- start 2024-03-28T19:00:48.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.715+0000 7f94c4000640 1 -- start start 2024-03-28T19:00:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c4000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ac0 0x7f94bc072fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c4000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f94bc0c5290 0x7f94bc15fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c4000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc0c4300 con 0x7f94bc0c3ea0 2024-03-28T19:00:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c4000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0c5c80 con 0x7f94bc0c5290 2024-03-28T19:00:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c4000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc073580 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c0c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f94bc0c3ea0 0x7f94bc0ab3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52370/0 (socket says 172.21.15.27:52370) 2024-03-28T19:00:48.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c0c00640 1 -- 172.21.15.27:0/2560663715 learned_addr learned my addr 172.21.15.27:0/2560663715 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:48.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c1600640 1 --2- 172.21.15.27:0/2560663715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ac0 0x7f94bc072fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:48.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.717+0000 7f94c2000640 1 --2- 172.21.15.27:0/2560663715 >> v2:172.21.15.155:3300/0 conn(0x7f94bc0c5290 0x7f94bc15fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:48.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.718+0000 7f94c1600640 1 -- 172.21.15.27:0/2560663715 >> v2:172.21.15.155:3300/0 conn(0x7f94bc0c5290 msgr2=0x7f94bc15fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:48.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.718+0000 7f94c1600640 1 --2- 172.21.15.27:0/2560663715 >> v2:172.21.15.155:3300/0 conn(0x7f94bc0c5290 0x7f94bc15fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:48.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.718+0000 7f94c1600640 1 -- 172.21.15.27:0/2560663715 >> v1:172.21.15.27:6789/0 conn(0x7f94bc0c3ea0 legacy=0x7f94bc0ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:48.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.718+0000 7f94c1600640 1 -- 172.21.15.27:0/2560663715 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc0aa630 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.720+0000 7f94c1600640 1 --2- 172.21.15.27:0/2560663715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ac0 0x7f94bc072fa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94b00097a0 tx=0x7f94b002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d76581013464625 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:48.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.723+0000 7f94bbe00640 1 -- 172.21.15.27:0/2560663715 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b00315a0 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.723+0000 7f94bbe00640 1 -- 172.21.15.27:0/2560663715 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0037070 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.723+0000 7f94bbe00640 1 -- 172.21.15.27:0/2560663715 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0031960 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.723+0000 7f94c4000640 1 -- 172.21.15.27:0/2560663715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ac0 msgr2=0x7f94bc072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:48.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.723+0000 7f94c4000640 1 --2- 172.21.15.27:0/2560663715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ac0 0x7f94bc072fa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94b00097a0 tx=0x7f94b002b840 comp rx=0 tx=0).stop 2024-03-28T19:00:48.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.724+0000 7f94c4000640 1 -- 172.21.15.27:0/2560663715 shutdown_connections 2024-03-28T19:00:48.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.724+0000 7f94c4000640 1 --2- 172.21.15.27:0/2560663715 >> v2:172.21.15.155:3300/0 conn(0x7f94bc0c5290 0x7f94bc15fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:48.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.724+0000 7f94c4000640 1 --2- 172.21.15.27:0/2560663715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ac0 0x7f94bc072fa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:48.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.724+0000 7f94c4000640 1 -- 172.21.15.27:0/2560663715 >> 172.21.15.27:0/2560663715 conn(0x7f94bc0c6c00 msgr2=0x7f94bc0c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:48.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.724+0000 7f94c4000640 1 -- 172.21.15.27:0/2560663715 shutdown_connections 2024-03-28T19:00:48.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.724+0000 7f94c4000640 1 -- 172.21.15.27:0/2560663715 wait complete. 2024-03-28T19:00:48.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.726+0000 7f94c4000640 1 Processor -- start 2024-03-28T19:00:48.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.726+0000 7f94c4000640 1 -- start start 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c4000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94bc0c3ac0 0x7f94bc1e0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c4000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ea0 0x7f94bc1debc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c4000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc0c5290 0x7f94bc0a8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c4000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0aaed0 con 0x7f94bc0c5290 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c4000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0aabf0 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c4000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0c4e10 con 0x7f94bc0c3ea0 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc0c5290 0x7f94bc0a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94bc0c3ac0 0x7f94bc1e0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc0c5290 0x7f94bc0a8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35152/0 (socket says 172.21.15.27:35152) 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c2000640 1 -- 172.21.15.27:0/2686637924 learned_addr learned my addr 172.21.15.27:0/2686637924 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:48.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.727+0000 7f94c0c00640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ea0 0x7f94bc1debc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:48.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.728+0000 7f94c1600640 1 -- 172.21.15.27:0/2686637924 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ea0 msgr2=0x7f94bc1debc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.728+0000 7f94c1600640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ea0 0x7f94bc1debc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.728+0000 7f94c1600640 1 -- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc0c5290 msgr2=0x7f94bc0a8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.728+0000 7f94c1600640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc0c5290 0x7f94bc0a8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.728+0000 7f94c1600640 1 -- 172.21.15.27:0/2686637924 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc1dc8c0 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.729+0000 7f94c1600640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94bc0c3ac0 0x7f94bc1e0800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f94b0033000 tx=0x7f94b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.729+0000 7f94b9600640 1 -- 172.21.15.27:0/2686637924 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0032630 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.729+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc1dbf90 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.729+0000 7f94b9600640 1 -- 172.21.15.27:0/2686637924 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0032810 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.729+0000 7f94b9600640 1 -- 172.21.15.27:0/2686637924 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0032c00 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.729+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94bc1dc550 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.731+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94bc04f400 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.733+0000 7f94b9600640 1 -- 172.21.15.27:0/2686637924 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f94b003b3f0 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.734+0000 7f94b9600640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f949c06bbd0 0x7f949c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.734+0000 7f94b9600640 1 -- 172.21.15.27:0/2686637924 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7153+0+0 (secure 0 0 0) 0x7f94b002f030 con 0x7f94bc0c3ac0 2024-03-28T19:00:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.735+0000 7f94c0c00640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f949c06bbd0 0x7f949c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.735+0000 7f94c0c00640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f949c06bbd0 0x7f949c06e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f94bc1d5910 tx=0x7f94a8007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:48.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:48.742+0000 7f94b9600640 1 -- 172.21.15.27:0/2686637924 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f94b0040070 con 0x7f94bc0c3ac0 2024-03-28T19:00:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.091+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94bc0c5070 con 0x7f94bc0c3ac0 2024-03-28T19:00:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.092+0000 7f94b9600640 1 -- 172.21.15.27:0/2686637924 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5461 (secure 0 0 0) 0x7f94b007add0 con 0x7f94bc0c3ac0 2024-03-28T19:00:49.093 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:49.093 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:00:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 48 2024-03-28T19:00:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.095+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f949c06bbd0 msgr2=0x7f949c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.095+0000 7f94c4000640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f949c06bbd0 0x7f949c06e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f94bc1d5910 tx=0x7f94a8007e80 comp rx=0 tx=0).stop 2024-03-28T19:00:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.095+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94bc0c3ac0 msgr2=0x7f94bc1e0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.095+0000 7f94c4000640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94bc0c3ac0 0x7f94bc1e0800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f94b0033000 tx=0x7f94b00046c0 comp rx=0 tx=0).stop 2024-03-28T19:00:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.096+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 shutdown_connections 2024-03-28T19:00:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.096+0000 7f94c4000640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f949c06bbd0 0x7f949c06e090 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:49.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.096+0000 7f94c4000640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94bc0c5290 0x7f94bc0a8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:49.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.096+0000 7f94c4000640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94bc0c3ea0 0x7f94bc1debc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:49.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.096+0000 7f94c4000640 1 --2- 172.21.15.27:0/2686637924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94bc0c3ac0 0x7f94bc1e0800 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:49.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.096+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 >> 172.21.15.27:0/2686637924 conn(0x7f94bc0c6c00 msgr2=0x7f94bc1da350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:49.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.096+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 shutdown_connections 2024-03-28T19:00:49.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:49.096+0000 7f94c4000640 1 -- 172.21.15.27:0/2686637924 wait complete. 2024-03-28T19:00:49.115 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5433 2024-03-28T19:00:49.115 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.c 2024-03-28T19:00:49.295 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-03-28T19:00:49.295 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-03-28T19:00:49.295 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-03-28T19:00:49.295 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-03-28T19:00:49.295 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-03-28T19:00:49.295 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-03-28T19:00:49.296 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-03-28T19:00:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-03-28T19:00:49.298 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-03-28T19:00:49.299 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-03-28T19:00:49.300 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-03-28T19:00:50.230 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-03-28T19:00:50.231 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-03-28T19:00:50.232 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-03-28T19:00:50.233 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-03-28T19:00:50.234 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-03-28T19:00:50.235 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-03-28T19:00:50.235 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-03-28T19:00:50.235 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-03-28T19:00:50.235 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-03-28T19:00:50.235 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-03-28T19:00:50.235 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-03-28T19:00:50.235 INFO:tasks.workunit.client.1.smithi155.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-03-28T19:00:50.235 INFO:tasks.workunit.client.1.smithi155.stdout:Take third snapshot .snap/test3 2024-03-28T19:00:50.274 INFO:tasks.workunit.client.1.smithi155.stdout:Delete the snapshots... 2024-03-28T19:00:50.342 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/34' 2024-03-28T19:00:50.342 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/38' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/11' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/16' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/32' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/30' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/14' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/18' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/13' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/33' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/17' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/35' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './196/37' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './196' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/19' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/15' 2024-03-28T19:00:50.343 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/31' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/36' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/20' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/12' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/39' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/10' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/34' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/38' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/11' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/16' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/32' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/30' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/14' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/18' 2024-03-28T19:00:50.344 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/13' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/33' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/17' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/35' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './197/37' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './197' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/19' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/15' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/31' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/36' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/20' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/12' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/39' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/10' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/34' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/38' 2024-03-28T19:00:50.345 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/11' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/16' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/32' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/30' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/14' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/18' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/13' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/33' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/17' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/35' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './198/37' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './198' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/19' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/15' 2024-03-28T19:00:50.346 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/31' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/36' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/20' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/12' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/39' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/10' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/34' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/38' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/11' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/16' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/32' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/30' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/14' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/18' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/13' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/33' 2024-03-28T19:00:50.347 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/17' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/35' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './199/37' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './199' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/21' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/27' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/31' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/36' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/20' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/39' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/34' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/38' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/28' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/24' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/29' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/32' 2024-03-28T19:00:50.348 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/22' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/25' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/30' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/33' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/23' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/35' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/26' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './200/37' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './200' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/21' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/27' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/31' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/36' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/20' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/39' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/34' 2024-03-28T19:00:50.349 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/38' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/28' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/24' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/29' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/32' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/22' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/25' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/30' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/33' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/23' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/35' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/26' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed './201/37' 2024-03-28T19:00:50.350 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './201' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/21' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/27' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/31' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/36' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/20' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/39' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/34' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/38' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/28' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/24' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/29' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/32' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/22' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/25' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/30' 2024-03-28T19:00:50.351 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/33' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/23' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/35' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/26' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './202/37' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './202' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/21' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/27' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/31' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/36' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/20' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/39' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/34' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/38' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/28' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/24' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/29' 2024-03-28T19:00:50.352 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/32' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/22' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/25' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/30' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/33' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/23' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/35' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/26' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './203/37' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './203' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/21' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/27' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/31' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/36' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/20' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/39' 2024-03-28T19:00:50.353 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/34' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/38' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/28' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/24' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/29' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/32' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/22' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/25' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/30' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/33' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/23' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/35' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/26' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed './204/37' 2024-03-28T19:00:50.354 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './204' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/21' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/27' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/31' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/36' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/20' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/39' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/34' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/38' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/28' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/24' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/29' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/32' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/22' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/25' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/30' 2024-03-28T19:00:50.355 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/33' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/23' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/35' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/26' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './205/37' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './205' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/21' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/27' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/31' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/36' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/20' 2024-03-28T19:00:50.356 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/39' 2024-03-28T19:00:50.356 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T19:00:50.350+0000 7f47c7400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:00:52.343 INFO:tasks.workunit.client.1.smithi155.stdout:.//.snap/test1 2024-03-28T19:00:52.344 INFO:tasks.workunit.client.1.smithi155.stdout:.//.snap/test2 2024-03-28T19:00:52.344 INFO:tasks.workunit.client.1.smithi155.stdout:.//.snap/test3 2024-03-28T19:00:52.358 INFO:tasks.workunit.client.1.smithi155.stdout:./274/.snap/snap-subdir-test 2024-03-28T19:00:52.375 INFO:tasks.workunit.client.1.smithi155.stdout:./166/.snap/snap-subdir-test 2024-03-28T19:00:52.390 INFO:tasks.workunit.client.1.smithi155.stdout:./256/.snap/snap-subdir-test 2024-03-28T19:00:52.404 INFO:tasks.workunit.client.1.smithi155.stdout:./247/.snap/snap-subdir-test 2024-03-28T19:00:52.418 INFO:tasks.workunit.client.1.smithi155.stdout:./248/.snap/snap-subdir-test 2024-03-28T19:00:52.436 INFO:tasks.workunit.client.1.smithi155.stdout:./147/.snap/snap-subdir-test 2024-03-28T19:00:52.453 INFO:tasks.workunit.client.1.smithi155.stdout:./151/.snap/snap-subdir-test 2024-03-28T19:00:52.469 INFO:tasks.workunit.client.1.smithi155.stdout:./170/.snap/snap-subdir-test 2024-03-28T19:00:52.486 INFO:tasks.workunit.client.1.smithi155.stdout:./167/.snap/snap-subdir-test 2024-03-28T19:00:52.501 INFO:tasks.workunit.client.1.smithi155.stdout:./140/.snap/snap-subdir-test 2024-03-28T19:00:52.506 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/34' 2024-03-28T19:00:52.506 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/38' 2024-03-28T19:00:52.506 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/28' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/24' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/29' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/32' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/22' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/25' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/30' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/33' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/23' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/35' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/26' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './206/37' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './206' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/21' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/27' 2024-03-28T19:00:52.507 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/31' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/36' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/20' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/39' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/34' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/38' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/28' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/24' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/29' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/32' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/22' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/25' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/30' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/33' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/23' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/35' 2024-03-28T19:00:52.508 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/26' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './207/37' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './207' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/21' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/27' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/31' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/36' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/20' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/39' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/34' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/38' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/28' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/24' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/29' 2024-03-28T19:00:52.509 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/32' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/22' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/25' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/30' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/33' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/23' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/35' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/26' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './208/37' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './208' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/21' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/27' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/31' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/36' 2024-03-28T19:00:52.510 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/20' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/39' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/34' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/38' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/28' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/24' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/29' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/32' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/22' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/25' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/30' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/33' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/23' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/35' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/26' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed './209/37' 2024-03-28T19:00:52.511 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './209' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/21' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/27' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/31' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/36' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/20' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/39' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/34' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/38' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/28' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/24' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/29' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/32' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/22' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/25' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/30' 2024-03-28T19:00:52.512 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/33' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/23' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/35' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/26' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './210/37' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './210' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/21' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/27' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/31' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/36' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/20' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/39' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/34' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/38' 2024-03-28T19:00:52.513 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/28' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/24' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/29' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/32' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/22' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/25' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/30' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/33' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/23' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/35' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/26' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './211/37' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './211' 2024-03-28T19:00:52.514 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/21' 2024-03-28T19:00:52.515 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/27' 2024-03-28T19:00:52.515 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/31' 2024-03-28T19:00:52.515 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/36' 2024-03-28T19:00:52.515 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/20' 2024-03-28T19:00:52.517 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/39' 2024-03-28T19:00:52.517 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/34' 2024-03-28T19:00:52.517 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/38' 2024-03-28T19:00:52.517 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/28' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/24' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/29' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/32' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/22' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/25' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/30' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/33' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/23' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/35' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/26' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './212/37' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './212' 2024-03-28T19:00:52.518 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/21' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/27' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/31' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/36' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/20' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/39' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/34' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/38' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/28' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/24' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/29' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/32' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/22' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/25' 2024-03-28T19:00:52.519 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/30' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/33' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/23' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/35' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/26' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './213/37' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './213' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/21' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/27' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/31' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/36' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/20' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/39' 2024-03-28T19:00:52.520 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/34' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/38' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/28' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/24' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/29' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/32' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/22' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/25' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/30' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/33' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/23' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/35' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/26' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed './214/37' 2024-03-28T19:00:52.521 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './214' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/21' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/27' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/31' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/36' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/20' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/39' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/34' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/38' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/28' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/24' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/29' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/32' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/22' 2024-03-28T19:00:52.522 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/25' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/30' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/33' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/23' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/35' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/26' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './215/37' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './215' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/21' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/27' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/31' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/36' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/20' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/39' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/34' 2024-03-28T19:00:52.523 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/38' 2024-03-28T19:00:52.524 INFO:tasks.workunit.client.1.smithi155.stdout:./142/.snap/snap-subdir-test 2024-03-28T19:00:52.535 INFO:tasks.workunit.client.1.smithi155.stdout:./150/.snap/snap-subdir-test 2024-03-28T19:00:52.550 INFO:tasks.workunit.client.1.smithi155.stdout:./273/.snap/snap-subdir-test 2024-03-28T19:00:52.564 INFO:tasks.workunit.client.1.smithi155.stdout:./269/.snap/snap-subdir-test 2024-03-28T19:00:52.584 INFO:tasks.workunit.client.1.smithi155.stdout:./105/.snap/snap-subdir-test 2024-03-28T19:00:52.600 INFO:tasks.workunit.client.1.smithi155.stdout:./210/.snap/snap-subdir-test 2024-03-28T19:00:52.618 INFO:tasks.workunit.client.1.smithi155.stdout:./265/.snap/snap-subdir-test 2024-03-28T19:00:52.632 INFO:tasks.workunit.client.1.smithi155.stdout:./218/.snap/snap-subdir-test 2024-03-28T19:00:52.646 INFO:tasks.workunit.client.1.smithi155.stdout:./290/.snap/snap-subdir-test 2024-03-28T19:00:52.667 INFO:tasks.workunit.client.1.smithi155.stdout:./149/.snap/snap-subdir-test 2024-03-28T19:00:52.683 INFO:tasks.workunit.client.1.smithi155.stdout:./109/.snap/snap-subdir-test 2024-03-28T19:00:52.700 INFO:tasks.workunit.client.1.smithi155.stdout:./292/.snap/snap-subdir-test 2024-03-28T19:00:52.716 INFO:tasks.workunit.client.1.smithi155.stdout:./242/.snap/snap-subdir-test 2024-03-28T19:00:52.733 INFO:tasks.workunit.client.1.smithi155.stdout:./158/.snap/snap-subdir-test 2024-03-28T19:00:52.752 INFO:tasks.workunit.client.1.smithi155.stdout:./243/.snap/snap-subdir-test 2024-03-28T19:00:52.768 INFO:tasks.workunit.client.1.smithi155.stdout:./106/.snap/snap-subdir-test 2024-03-28T19:00:52.785 INFO:tasks.workunit.client.1.smithi155.stdout:./250/.snap/snap-subdir-test 2024-03-28T19:00:52.800 INFO:tasks.workunit.client.1.smithi155.stdout:./221/.snap/snap-subdir-test 2024-03-28T19:00:52.815 INFO:tasks.workunit.client.1.smithi155.stdout:./244/.snap/snap-subdir-test 2024-03-28T19:00:52.832 INFO:tasks.workunit.client.1.smithi155.stdout:./258/.snap/snap-subdir-test 2024-03-28T19:00:52.850 INFO:tasks.workunit.client.1.smithi155.stdout:./268/.snap/snap-subdir-test 2024-03-28T19:00:52.867 INFO:tasks.workunit.client.1.smithi155.stdout:./189/.snap/snap-subdir-test 2024-03-28T19:00:52.885 INFO:tasks.workunit.client.1.smithi155.stdout:./285/.snap/snap-subdir-test 2024-03-28T19:00:52.899 INFO:tasks.workunit.client.1.smithi155.stdout:./208/.snap/snap-subdir-test 2024-03-28T19:00:52.914 INFO:tasks.workunit.client.1.smithi155.stdout:./288/.snap/snap-subdir-test 2024-03-28T19:00:52.956 INFO:tasks.workunit.client.1.smithi155.stdout:./209/.snap/snap-subdir-test 2024-03-28T19:00:52.971 INFO:tasks.workunit.client.1.smithi155.stdout:./253/.snap/snap-subdir-test 2024-03-28T19:00:52.988 INFO:tasks.workunit.client.1.smithi155.stdout:./249/.snap/snap-subdir-test 2024-03-28T19:00:53.005 INFO:tasks.workunit.client.1.smithi155.stdout:./133/.snap/snap-subdir-test 2024-03-28T19:00:53.020 INFO:tasks.workunit.client.1.smithi155.stdout:./291/.snap/snap-subdir-test 2024-03-28T19:00:53.035 INFO:tasks.workunit.client.1.smithi155.stdout:./226/.snap/snap-subdir-test 2024-03-28T19:00:53.051 INFO:tasks.workunit.client.1.smithi155.stdout:./173/.snap/snap-subdir-test 2024-03-28T19:00:53.069 INFO:tasks.workunit.client.1.smithi155.stdout:./174/.snap/snap-subdir-test 2024-03-28T19:00:53.084 INFO:tasks.workunit.client.1.smithi155.stdout:./299/.snap/snap-subdir-test 2024-03-28T19:00:53.101 INFO:tasks.workunit.client.1.smithi155.stdout:./262/.snap/snap-subdir-test 2024-03-28T19:00:53.116 INFO:tasks.workunit.client.1.smithi155.stdout:./231/.snap/snap-subdir-test 2024-03-28T19:00:53.132 INFO:tasks.workunit.client.1.smithi155.stdout:./214/.snap/snap-subdir-test 2024-03-28T19:00:53.148 INFO:tasks.workunit.client.1.smithi155.stdout:./281/.snap/snap-subdir-test 2024-03-28T19:00:53.161 INFO:tasks.workunit.client.1.smithi155.stdout:./289/.snap/snap-subdir-test 2024-03-28T19:00:53.176 INFO:tasks.workunit.client.1.smithi155.stdout:./283/.snap/snap-subdir-test 2024-03-28T19:00:53.192 INFO:tasks.workunit.client.1.smithi155.stdout:./232/.snap/snap-subdir-test 2024-03-28T19:00:53.207 INFO:tasks.workunit.client.1.smithi155.stdout:./298/.snap/snap-subdir-test 2024-03-28T19:00:53.223 INFO:tasks.workunit.client.1.smithi155.stdout:./233/.snap/snap-subdir-test 2024-03-28T19:00:53.239 INFO:tasks.workunit.client.1.smithi155.stdout:./155/.snap/snap-subdir-test 2024-03-28T19:00:53.257 INFO:tasks.workunit.client.1.smithi155.stdout:./114/.snap/snap-subdir-test 2024-03-28T19:00:53.274 INFO:tasks.workunit.client.1.smithi155.stdout:./159/.snap/snap-subdir-test 2024-03-28T19:00:53.292 INFO:tasks.workunit.client.1.smithi155.stdout:./187/.snap/snap-subdir-test 2024-03-28T19:00:53.310 INFO:tasks.workunit.client.1.smithi155.stdout:./185/.snap/snap-subdir-test 2024-03-28T19:00:53.324 INFO:tasks.workunit.client.1.smithi155.stdout:./293/.snap/snap-subdir-test 2024-03-28T19:00:53.339 INFO:tasks.workunit.client.1.smithi155.stdout:./228/.snap/snap-subdir-test 2024-03-28T19:00:53.358 INFO:tasks.workunit.client.1.smithi155.stdout:./120/.snap/snap-subdir-test 2024-03-28T19:00:53.376 INFO:tasks.workunit.client.1.smithi155.stdout:./110/.snap/snap-subdir-test 2024-03-28T19:00:53.392 INFO:tasks.workunit.client.1.smithi155.stdout:./267/.snap/snap-subdir-test 2024-03-28T19:00:53.408 INFO:tasks.workunit.client.1.smithi155.stdout:./202/.snap/snap-subdir-test 2024-03-28T19:00:53.424 INFO:tasks.workunit.client.1.smithi155.stdout:./137/.snap/snap-subdir-test 2024-03-28T19:00:53.441 INFO:tasks.workunit.client.1.smithi155.stdout:./246/.snap/snap-subdir-test 2024-03-28T19:00:53.453 INFO:tasks.workunit.client.1.smithi155.stdout:./219/.snap/snap-subdir-test 2024-03-28T19:00:53.472 INFO:tasks.workunit.client.1.smithi155.stdout:./198/.snap/snap-subdir-test 2024-03-28T19:00:53.489 INFO:tasks.workunit.client.1.smithi155.stdout:./136/.snap/snap-subdir-test 2024-03-28T19:00:53.505 INFO:tasks.workunit.client.1.smithi155.stdout:./270/.snap/snap-subdir-test 2024-03-28T19:00:53.522 INFO:tasks.workunit.client.1.smithi155.stdout:./125/.snap/snap-subdir-test 2024-03-28T19:00:53.538 INFO:tasks.workunit.client.1.smithi155.stdout:./194/.snap/snap-subdir-test 2024-03-28T19:00:53.553 INFO:tasks.workunit.client.1.smithi155.stdout:./119/.snap/snap-subdir-test 2024-03-28T19:00:53.568 INFO:tasks.workunit.client.1.smithi155.stdout:./297/.snap/snap-subdir-test 2024-03-28T19:00:53.583 INFO:tasks.workunit.client.1.smithi155.stdout:./230/.snap/snap-subdir-test 2024-03-28T19:00:53.599 INFO:tasks.workunit.client.1.smithi155.stdout:./260/.snap/snap-subdir-test 2024-03-28T19:00:53.619 INFO:tasks.workunit.client.1.smithi155.stdout:./157/.snap/snap-subdir-test 2024-03-28T19:00:53.638 INFO:tasks.workunit.client.1.smithi155.stdout:./172/.snap/snap-subdir-test 2024-03-28T19:00:53.653 INFO:tasks.workunit.client.1.smithi155.stdout:./277/.snap/snap-subdir-test 2024-03-28T19:00:53.671 INFO:tasks.workunit.client.1.smithi155.stdout:./186/.snap/snap-subdir-test 2024-03-28T19:00:53.685 INFO:tasks.workunit.client.1.smithi155.stdout:./237/.snap/snap-subdir-test 2024-03-28T19:00:53.704 INFO:tasks.workunit.client.1.smithi155.stdout:./190/.snap/snap-subdir-test 2024-03-28T19:00:53.719 INFO:tasks.workunit.client.1.smithi155.stdout:./213/.snap/snap-subdir-test 2024-03-28T19:00:53.735 INFO:tasks.workunit.client.1.smithi155.stdout:./104/.snap/snap-subdir-test 2024-03-28T19:00:53.753 INFO:tasks.workunit.client.1.smithi155.stdout:./164/.snap/snap-subdir-test 2024-03-28T19:00:53.770 INFO:tasks.workunit.client.1.smithi155.stdout:./112/.snap/snap-subdir-test 2024-03-28T19:00:53.784 INFO:tasks.workunit.client.1.smithi155.stdout:./295/.snap/snap-subdir-test 2024-03-28T19:00:53.798 INFO:tasks.workunit.client.1.smithi155.stdout:./272/.snap/snap-subdir-test 2024-03-28T19:00:53.814 INFO:tasks.workunit.client.1.smithi155.stdout:./100/.snap/snap-subdir-test 2024-03-28T19:00:53.833 INFO:tasks.workunit.client.1.smithi155.stdout:./134/.snap/snap-subdir-test 2024-03-28T19:00:53.853 INFO:tasks.workunit.client.1.smithi155.stdout:./200/.snap/snap-subdir-test 2024-03-28T19:00:53.871 INFO:tasks.workunit.client.1.smithi155.stdout:./115/.snap/snap-subdir-test 2024-03-28T19:00:53.886 INFO:tasks.workunit.client.1.smithi155.stdout:./284/.snap/snap-subdir-test 2024-03-28T19:00:53.899 INFO:tasks.workunit.client.1.smithi155.stdout:./280/.snap/snap-subdir-test 2024-03-28T19:00:53.916 INFO:tasks.workunit.client.1.smithi155.stdout:./116/.snap/snap-subdir-test 2024-03-28T19:00:53.945 INFO:tasks.workunit.client.1.smithi155.stdout:./245/.snap/snap-subdir-test 2024-03-28T19:00:53.967 INFO:tasks.workunit.client.1.smithi155.stdout:./195/.snap/snap-subdir-test 2024-03-28T19:00:53.985 INFO:tasks.workunit.client.1.smithi155.stdout:./138/.snap/snap-subdir-test 2024-03-28T19:00:54.001 INFO:tasks.workunit.client.1.smithi155.stdout:./135/.snap/snap-subdir-test 2024-03-28T19:00:54.018 INFO:tasks.workunit.client.1.smithi155.stdout:./117/.snap/snap-subdir-test 2024-03-28T19:00:54.034 INFO:tasks.workunit.client.1.smithi155.stdout:./241/.snap/snap-subdir-test 2024-03-28T19:00:54.051 INFO:tasks.workunit.client.1.smithi155.stdout:./143/.snap/snap-subdir-test 2024-03-28T19:00:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:./180/.snap/snap-subdir-test 2024-03-28T19:00:54.085 INFO:tasks.workunit.client.1.smithi155.stdout:./126/.snap/snap-subdir-test 2024-03-28T19:00:54.101 INFO:tasks.workunit.client.1.smithi155.stdout:./161/.snap/snap-subdir-test 2024-03-28T19:00:54.116 INFO:tasks.workunit.client.1.smithi155.stdout:./204/.snap/snap-subdir-test 2024-03-28T19:00:54.133 INFO:tasks.workunit.client.1.smithi155.stdout:./176/.snap/snap-subdir-test 2024-03-28T19:00:54.148 INFO:tasks.workunit.client.1.smithi155.stdout:./255/.snap/snap-subdir-test 2024-03-28T19:00:54.165 INFO:tasks.workunit.client.1.smithi155.stdout:./215/.snap/snap-subdir-test 2024-03-28T19:00:54.183 INFO:tasks.workunit.client.1.smithi155.stdout:./111/.snap/snap-subdir-test 2024-03-28T19:00:54.198 INFO:tasks.workunit.client.1.smithi155.stdout:./238/.snap/snap-subdir-test 2024-03-28T19:00:54.215 INFO:tasks.workunit.client.1.smithi155.stdout:./191/.snap/snap-subdir-test 2024-03-28T19:00:54.230 INFO:tasks.workunit.client.1.smithi155.stdout:./184/.snap/snap-subdir-test 2024-03-28T19:00:54.246 INFO:tasks.workunit.client.1.smithi155.stdout:./192/.snap/snap-subdir-test 2024-03-28T19:00:54.262 INFO:tasks.workunit.client.1.smithi155.stdout:./103/.snap/snap-subdir-test 2024-03-28T19:00:54.279 INFO:tasks.workunit.client.1.smithi155.stdout:./128/.snap/snap-subdir-test 2024-03-28T19:00:54.294 INFO:tasks.workunit.client.1.smithi155.stdout:./229/.snap/snap-subdir-test 2024-03-28T19:00:54.312 INFO:tasks.workunit.client.1.smithi155.stdout:./144/.snap/snap-subdir-test 2024-03-28T19:00:54.329 INFO:tasks.workunit.client.1.smithi155.stdout:./146/.snap/snap-subdir-test 2024-03-28T19:00:54.344 INFO:tasks.workunit.client.1.smithi155.stdout:./227/.snap/snap-subdir-test 2024-03-28T19:00:54.359 INFO:tasks.workunit.client.1.smithi155.stdout:./278/.snap/snap-subdir-test 2024-03-28T19:00:54.375 INFO:tasks.workunit.client.1.smithi155.stdout:./141/.snap/snap-subdir-test 2024-03-28T19:00:54.391 INFO:tasks.workunit.client.1.smithi155.stdout:./234/.snap/snap-subdir-test 2024-03-28T19:00:54.408 INFO:tasks.workunit.client.1.smithi155.stdout:./225/.snap/snap-subdir-test 2024-03-28T19:00:54.425 INFO:tasks.workunit.client.1.smithi155.stdout:./182/.snap/snap-subdir-test 2024-03-28T19:00:54.443 INFO:tasks.workunit.client.1.smithi155.stdout:./107/.snap/snap-subdir-test 2024-03-28T19:00:54.490 INFO:tasks.workunit.client.1.smithi155.stdout:./178/.snap/snap-subdir-test 2024-03-28T19:00:54.511 INFO:tasks.workunit.client.1.smithi155.stdout:./266/.snap/snap-subdir-test 2024-03-28T19:00:54.527 INFO:tasks.workunit.client.1.smithi155.stdout:./240/.snap/snap-subdir-test 2024-03-28T19:00:54.542 INFO:tasks.workunit.client.1.smithi155.stdout:./156/.snap/snap-subdir-test 2024-03-28T19:00:54.558 INFO:tasks.workunit.client.1.smithi155.stdout:./131/.snap/snap-subdir-test 2024-03-28T19:00:54.575 INFO:tasks.workunit.client.1.smithi155.stdout:./123/.snap/snap-subdir-test 2024-03-28T19:00:54.578 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/28' 2024-03-28T19:00:54.578 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/24' 2024-03-28T19:00:54.578 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/29' 2024-03-28T19:00:54.578 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/32' 2024-03-28T19:00:54.578 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/22' 2024-03-28T19:00:54.578 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/25' 2024-03-28T19:00:54.578 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/30' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/33' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/23' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/35' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/26' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './216/37' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './216' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/21' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/27' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/31' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/36' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/20' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/39' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/34' 2024-03-28T19:00:54.579 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/38' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/28' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/24' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/29' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/32' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/22' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/25' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/30' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/33' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/23' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/35' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/26' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './217/37' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './217' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/21' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/27' 2024-03-28T19:00:54.580 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/31' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/36' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/20' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/39' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/34' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/38' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/28' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/24' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/29' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/32' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/22' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/25' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/30' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/33' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/23' 2024-03-28T19:00:54.581 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/35' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/26' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './218/37' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './218' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/21' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/27' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/31' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/36' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/20' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/39' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/34' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/38' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/28' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/24' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/29' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/32' 2024-03-28T19:00:54.582 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/22' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/25' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/30' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/33' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/23' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/35' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/26' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './219/37' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './219' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/21' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/27' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/31' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/36' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/20' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/39' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/34' 2024-03-28T19:00:54.583 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/38' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/28' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/24' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/29' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/32' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/22' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/25' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/30' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/33' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/23' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/35' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/26' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './220/37' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './220' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/21' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/27' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/31' 2024-03-28T19:00:54.584 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/36' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/20' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/39' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/34' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/38' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/28' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/24' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/29' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/32' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/22' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/25' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/30' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/33' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/23' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/35' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/26' 2024-03-28T19:00:54.585 INFO:tasks.workunit.client.0.smithi027.stdout:removed './221/37' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './221' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/21' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/27' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/31' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/36' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/20' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/39' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/34' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/38' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/28' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/24' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/29' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/32' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/22' 2024-03-28T19:00:54.586 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/25' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/30' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/33' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/23' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/35' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/26' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './222/37' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './222' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/21' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/27' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/31' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/36' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/20' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/39' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/34' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/38' 2024-03-28T19:00:54.587 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/28' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/24' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/29' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/32' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/22' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/25' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/30' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/33' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/23' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/35' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/26' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './223/37' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './223' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/21' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/27' 2024-03-28T19:00:54.588 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/31' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/36' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/20' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/39' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/34' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/38' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/28' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/24' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/29' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/32' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/22' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/25' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/30' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/33' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/23' 2024-03-28T19:00:54.589 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/35' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/26' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './224/37' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './224' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/21' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/27' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/31' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/36' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/20' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/39' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/34' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/38' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/28' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/24' 2024-03-28T19:00:54.590 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/29' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/32' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/22' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/25' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/30' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/33' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/23' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/35' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/26' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './225/37' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './225' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/21' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/27' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/31' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/36' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/20' 2024-03-28T19:00:54.591 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/39' 2024-03-28T19:00:54.592 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/34' 2024-03-28T19:00:54.592 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/38' 2024-03-28T19:00:54.592 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/28' 2024-03-28T19:00:54.592 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/24' 2024-03-28T19:00:54.592 INFO:tasks.workunit.client.1.smithi155.stdout:./160/.snap/snap-subdir-test 2024-03-28T19:00:54.607 INFO:tasks.workunit.client.1.smithi155.stdout:./165/.snap/snap-subdir-test 2024-03-28T19:00:54.624 INFO:tasks.workunit.client.1.smithi155.stdout:./101/.snap/snap-subdir-test 2024-03-28T19:00:54.641 INFO:tasks.workunit.client.1.smithi155.stdout:./148/.snap/snap-subdir-test 2024-03-28T19:00:54.658 INFO:tasks.workunit.client.1.smithi155.stdout:./168/.snap/snap-subdir-test 2024-03-28T19:00:54.675 INFO:tasks.workunit.client.1.smithi155.stdout:./179/.snap/snap-subdir-test 2024-03-28T19:00:54.691 INFO:tasks.workunit.client.1.smithi155.stdout:./257/.snap/snap-subdir-test 2024-03-28T19:00:54.705 INFO:tasks.workunit.client.1.smithi155.stdout:./252/.snap/snap-subdir-test 2024-03-28T19:00:54.720 INFO:tasks.workunit.client.1.smithi155.stdout:./132/.snap/snap-subdir-test 2024-03-28T19:00:54.736 INFO:tasks.workunit.client.1.smithi155.stdout:./236/.snap/snap-subdir-test 2024-03-28T19:00:54.754 INFO:tasks.workunit.client.1.smithi155.stdout:./211/.snap/snap-subdir-test 2024-03-28T19:00:54.770 INFO:tasks.workunit.client.1.smithi155.stdout:./199/.snap/snap-subdir-test 2024-03-28T19:00:54.785 INFO:tasks.workunit.client.1.smithi155.stdout:./205/.snap/snap-subdir-test 2024-03-28T19:00:54.803 INFO:tasks.workunit.client.1.smithi155.stdout:./163/.snap/snap-subdir-test 2024-03-28T19:00:54.818 INFO:tasks.workunit.client.1.smithi155.stdout:./261/.snap/snap-subdir-test 2024-03-28T19:00:54.835 INFO:tasks.workunit.client.1.smithi155.stdout:./201/.snap/snap-subdir-test 2024-03-28T19:00:54.850 INFO:tasks.workunit.client.1.smithi155.stdout:./282/.snap/snap-subdir-test 2024-03-28T19:00:54.868 INFO:tasks.workunit.client.1.smithi155.stdout:./296/.snap/snap-subdir-test 2024-03-28T19:00:54.885 INFO:tasks.workunit.client.1.smithi155.stdout:./217/.snap/snap-subdir-test 2024-03-28T19:00:54.901 INFO:tasks.workunit.client.1.smithi155.stdout:./153/.snap/snap-subdir-test 2024-03-28T19:00:54.918 INFO:tasks.workunit.client.1.smithi155.stdout:./206/.snap/snap-subdir-test 2024-03-28T19:00:54.936 INFO:tasks.workunit.client.1.smithi155.stdout:./122/.snap/snap-subdir-test 2024-03-28T19:00:54.954 INFO:tasks.workunit.client.1.smithi155.stdout:./162/.snap/snap-subdir-test 2024-03-28T19:00:55.010 INFO:tasks.workunit.client.1.smithi155.stdout:./197/.snap/snap-subdir-test 2024-03-28T19:00:55.026 INFO:tasks.workunit.client.1.smithi155.stdout:./216/.snap/snap-subdir-test 2024-03-28T19:00:55.116 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-03-28T19:00:55.116 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:00:55.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:00:55.122 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:00:55.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:55.320 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:00:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.365+0000 7f8290400640 1 Processor -- start 2024-03-28T19:00:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.365+0000 7f8290400640 1 -- start start 2024-03-28T19:00:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.366+0000 7f8290400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8288152bd0 0x7f828806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.366+0000 7f8290400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f828806f1e0 0x7f8288070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.366+0000 7f8290400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82880c52e0 con 0x7f82880c4dc0 2024-03-28T19:00:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.366+0000 7f8290400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82881557e0 con 0x7f8288152bd0 2024-03-28T19:00:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.366+0000 7f8290400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8288155ac0 con 0x7f828806f1e0 2024-03-28T19:00:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8288152bd0 0x7f828806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8288152bd0 0x7f828806eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35172/0 (socket says 172.21.15.27:35172) 2024-03-28T19:00:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828e400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f82880c4dc0 0x7f8288158c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:48400/0 (socket says 172.21.15.27:48400) 2024-03-28T19:00:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828da00640 1 -- 172.21.15.27:0/3281028952 learned_addr learned my addr 172.21.15.27:0/3281028952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828d000640 1 --2- 172.21.15.27:0/3281028952 >> v2:172.21.15.155:3300/0 conn(0x7f828806f1e0 0x7f8288070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828da00640 1 -- 172.21.15.27:0/3281028952 >> v2:172.21.15.155:3300/0 conn(0x7f828806f1e0 msgr2=0x7f8288070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828da00640 1 --2- 172.21.15.27:0/3281028952 >> v2:172.21.15.155:3300/0 conn(0x7f828806f1e0 0x7f8288070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f8287e00640 1 -- 172.21.15.27:0/3281028952 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208652245 0 0) 0x7f82880c52e0 con 0x7f82880c4dc0 2024-03-28T19:00:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828da00640 1 -- 172.21.15.27:0/3281028952 >> v1:172.21.15.155:6789/0 conn(0x7f82880c4dc0 legacy=0x7f8288158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.367+0000 7f828da00640 1 -- 172.21.15.27:0/3281028952 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8288155da0 con 0x7f8288152bd0 2024-03-28T19:00:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.369+0000 7f828da00640 1 --2- 172.21.15.27:0/3281028952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 0x7f828806eca0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f82880c9e10 tx=0x7f827c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3a561138eb46ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.370+0000 7f8287e00640 1 -- 172.21.15.27:0/3281028952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c034070 con 0x7f8288152bd0 2024-03-28T19:00:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.370+0000 7f8287e00640 1 -- 172.21.15.27:0/3281028952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c030860 con 0x7f8288152bd0 2024-03-28T19:00:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.370+0000 7f8287e00640 1 -- 172.21.15.27:0/3281028952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c030c30 con 0x7f8288152bd0 2024-03-28T19:00:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.371+0000 7f8290400640 1 -- 172.21.15.27:0/3281028952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 msgr2=0x7f828806eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.371+0000 7f8290400640 1 --2- 172.21.15.27:0/3281028952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 0x7f828806eca0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f82880c9e10 tx=0x7f827c02b9a0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.371+0000 7f8290400640 1 -- 172.21.15.27:0/3281028952 shutdown_connections 2024-03-28T19:00:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.371+0000 7f8290400640 1 --2- 172.21.15.27:0/3281028952 >> v2:172.21.15.155:3300/0 conn(0x7f828806f1e0 0x7f8288070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.371+0000 7f8290400640 1 --2- 172.21.15.27:0/3281028952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 0x7f828806eca0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.371+0000 7f8290400640 1 -- 172.21.15.27:0/3281028952 >> 172.21.15.27:0/3281028952 conn(0x7f8288066dd0 msgr2=0x7f82880671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.371+0000 7f8290400640 1 -- 172.21.15.27:0/3281028952 shutdown_connections 2024-03-28T19:00:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.371+0000 7f8290400640 1 -- 172.21.15.27:0/3281028952 wait complete. 2024-03-28T19:00:55.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.373+0000 7f8290400640 1 Processor -- start 2024-03-28T19:00:55.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.373+0000 7f8290400640 1 -- start start 2024-03-28T19:00:55.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.374+0000 7f8290400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828806f1e0 0x7f82881589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:55.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.374+0000 7f8290400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82880c4dc0 0x7f828809c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:55.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.374+0000 7f8290400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 0x7f828809c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:55.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.374+0000 7f8290400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880c6df0 con 0x7f8288152bd0 2024-03-28T19:00:55.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.374+0000 7f8290400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880c6c70 con 0x7f828806f1e0 2024-03-28T19:00:55.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.374+0000 7f8290400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8288070a80 con 0x7f82880c4dc0 2024-03-28T19:00:55.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 0x7f828809c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:55.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828806f1e0 0x7f82881589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:55.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828806f1e0 0x7f82881589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:33688/0 (socket says 172.21.15.27:33688) 2024-03-28T19:00:55.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828da00640 1 -- 172.21.15.27:0/301455691 learned_addr learned my addr 172.21.15.27:0/301455691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:55.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828d000640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82880c4dc0 0x7f828809c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:55.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828e400640 1 -- 172.21.15.27:0/301455691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82880c4dc0 msgr2=0x7f828809c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:55.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828e400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82880c4dc0 0x7f828809c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828e400640 1 -- 172.21.15.27:0/301455691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828806f1e0 msgr2=0x7f82881589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:55.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828e400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828806f1e0 0x7f82881589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828e400640 1 -- 172.21.15.27:0/301455691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8288156e50 con 0x7f8288152bd0 2024-03-28T19:00:55.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.375+0000 7f828da00640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828806f1e0 0x7f82881589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:00:55.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.376+0000 7f828e400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 0x7f828809c6e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f828000c6d0 tx=0x7f828000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:55.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.376+0000 7f8285600640 1 -- 172.21.15.27:0/301455691 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8280018070 con 0x7f8288152bd0 2024-03-28T19:00:55.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.376+0000 7f8285600640 1 -- 172.21.15.27:0/301455691 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8280004d60 con 0x7f8288152bd0 2024-03-28T19:00:55.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.376+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8288157130 con 0x7f8288152bd0 2024-03-28T19:00:55.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.376+0000 7f8285600640 1 -- 172.21.15.27:0/301455691 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828000a820 con 0x7f8288152bd0 2024-03-28T19:00:55.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.377+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82881576f0 con 0x7f8288152bd0 2024-03-28T19:00:55.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.378+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828804f400 con 0x7f8288152bd0 2024-03-28T19:00:55.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.380+0000 7f8285600640 1 -- 172.21.15.27:0/301455691 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f828001b070 con 0x7f8288152bd0 2024-03-28T19:00:55.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.381+0000 7f8285600640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f826806bc20 0x7f826806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:55.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.381+0000 7f8285600640 1 -- 172.21.15.27:0/301455691 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7153+0+0 (secure 0 0 0) 0x7f828008a790 con 0x7f8288152bd0 2024-03-28T19:00:55.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.382+0000 7f828da00640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f826806bc20 0x7f826806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:55.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.383+0000 7f828da00640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f826806bc20 0x7f826806e0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f827c009630 tx=0x7f827c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:55.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.388+0000 7f8285600640 1 -- 172.21.15.27:0/301455691 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f82800586f0 con 0x7f8288152bd0 2024-03-28T19:00:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.723+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f828806f800 con 0x7f8288152bd0 2024-03-28T19:00:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.724+0000 7f8285600640 1 -- 172.21.15.27:0/301455691 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5461 (secure 0 0 0) 0x7f828005c790 con 0x7f8288152bd0 2024-03-28T19:00:55.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:55.724 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:00:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 48 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f826806bc20 msgr2=0x7f826806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f826806bc20 0x7f826806e0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f827c009630 tx=0x7f827c008030 comp rx=0 tx=0).stop 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 msgr2=0x7f828809c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 0x7f828809c6e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f828000c6d0 tx=0x7f828000cb90 comp rx=0 tx=0).stop 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 shutdown_connections 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f826806bc20 0x7f826806e0e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8288152bd0 0x7f828809c6e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82880c4dc0 0x7f828809c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 --2- 172.21.15.27:0/301455691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828806f1e0 0x7f82881589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.727+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 >> 172.21.15.27:0/301455691 conn(0x7f8288066dd0 msgr2=0x7f828806db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.728+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 shutdown_connections 2024-03-28T19:00:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:55.728+0000 7f8290400640 1 -- 172.21.15.27:0/301455691 wait complete. 2024-03-28T19:00:55.748 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/29' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/32' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/22' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/25' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/30' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/33' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/23' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/35' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/26' 2024-03-28T19:00:56.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './226/37' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './226' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/21' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/27' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/31' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/36' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/20' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/39' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/34' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/38' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/28' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/24' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/29' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/32' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/22' 2024-03-28T19:00:56.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/25' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/30' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/33' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/23' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/35' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/26' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './227/37' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './227' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/21' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/27' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/31' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/36' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/20' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/39' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/34' 2024-03-28T19:00:56.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/38' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/28' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/24' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/29' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/32' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/22' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/25' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/30' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/33' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/23' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/35' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/26' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './228/37' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './228' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/21' 2024-03-28T19:00:56.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/27' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/31' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/36' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/20' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/39' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/34' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/38' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/28' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/24' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/29' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/32' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/22' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/25' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/30' 2024-03-28T19:00:56.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/33' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/23' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/35' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/26' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './229/37' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './229' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/21' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/27' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/31' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/36' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/20' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/39' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/34' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/38' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/28' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/24' 2024-03-28T19:00:56.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/29' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/32' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/22' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/25' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/30' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/33' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/23' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/35' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/26' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './230/37' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './230' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/21' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/27' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/31' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/36' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/20' 2024-03-28T19:00:56.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/39' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/34' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/38' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/28' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/24' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/29' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/32' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/22' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/25' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/30' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/33' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/23' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/35' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/26' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './231/37' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './231' 2024-03-28T19:00:56.192 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/21' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/27' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/31' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/36' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/20' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/39' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/34' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/38' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/28' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/24' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/29' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/32' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/22' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/25' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/30' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/33' 2024-03-28T19:00:56.193 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/23' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/35' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/26' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './232/37' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './232' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/21' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/27' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/31' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/36' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/20' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/39' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/34' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/38' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/28' 2024-03-28T19:00:56.194 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/24' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/29' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/32' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/22' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/25' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/30' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/33' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/23' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/35' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/26' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './233/37' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './233' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/21' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/27' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/31' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/36' 2024-03-28T19:00:56.195 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/20' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/39' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/34' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/38' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/28' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/24' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/29' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/32' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/22' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/25' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/30' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/33' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/23' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/35' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/26' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed './234/37' 2024-03-28T19:00:56.196 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './234' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/21' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/27' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/31' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/36' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/20' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/39' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/34' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/38' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/28' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/24' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/29' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/32' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/22' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/25' 2024-03-28T19:00:56.197 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/30' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/33' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/23' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/35' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/26' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './235/37' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './235' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/21' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/27' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/31' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/36' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/20' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/39' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/34' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/38' 2024-03-28T19:00:56.198 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/28' 2024-03-28T19:00:56.199 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/24' 2024-03-28T19:00:56.199 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/29' 2024-03-28T19:00:56.199 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/32' 2024-03-28T19:00:56.537 INFO:tasks.workunit.client.1.smithi155.stdout:./254/.snap/snap-subdir-test 2024-03-28T19:00:56.555 INFO:tasks.workunit.client.1.smithi155.stdout:./113/.snap/snap-subdir-test 2024-03-28T19:00:56.575 INFO:tasks.workunit.client.1.smithi155.stdout:./152/.snap/snap-subdir-test 2024-03-28T19:00:56.592 INFO:tasks.workunit.client.1.smithi155.stdout:./121/.snap/snap-subdir-test 2024-03-28T19:00:56.610 INFO:tasks.workunit.client.1.smithi155.stdout:./118/.snap/snap-subdir-test 2024-03-28T19:00:56.629 INFO:tasks.workunit.client.1.smithi155.stdout:./196/.snap/snap-subdir-test 2024-03-28T19:00:56.646 INFO:tasks.workunit.client.1.smithi155.stdout:./124/.snap/snap-subdir-test 2024-03-28T19:00:56.661 INFO:tasks.workunit.client.1.smithi155.stdout:./239/.snap/snap-subdir-test 2024-03-28T19:00:56.679 INFO:tasks.workunit.client.1.smithi155.stdout:./188/.snap/snap-subdir-test 2024-03-28T19:00:56.700 INFO:tasks.workunit.client.1.smithi155.stdout:./287/.snap/snap-subdir-test 2024-03-28T19:00:56.715 INFO:tasks.workunit.client.1.smithi155.stdout:./275/.snap/snap-subdir-test 2024-03-28T19:00:56.730 INFO:tasks.workunit.client.1.smithi155.stdout:./259/.snap/snap-subdir-test 2024-03-28T19:00:56.747 INFO:tasks.workunit.client.1.smithi155.stdout:./224/.snap/snap-subdir-test 2024-03-28T19:00:56.762 INFO:tasks.workunit.client.1.smithi155.stdout:./264/.snap/snap-subdir-test 2024-03-28T19:00:56.778 INFO:tasks.workunit.client.1.smithi155.stdout:./145/.snap/snap-subdir-test 2024-03-28T19:00:56.794 INFO:tasks.workunit.client.1.smithi155.stdout:./279/.snap/snap-subdir-test 2024-03-28T19:00:56.811 INFO:tasks.workunit.client.1.smithi155.stdout:./263/.snap/snap-subdir-test 2024-03-28T19:00:56.829 INFO:tasks.workunit.client.1.smithi155.stdout:./130/.snap/snap-subdir-test 2024-03-28T19:00:56.847 INFO:tasks.workunit.client.1.smithi155.stdout:./193/.snap/snap-subdir-test 2024-03-28T19:00:56.865 INFO:tasks.workunit.client.1.smithi155.stdout:./223/.snap/snap-subdir-test 2024-03-28T19:00:56.883 INFO:tasks.workunit.client.1.smithi155.stdout:./207/.snap/snap-subdir-test 2024-03-28T19:00:56.898 INFO:tasks.workunit.client.1.smithi155.stdout:./276/.snap/snap-subdir-test 2024-03-28T19:00:56.915 INFO:tasks.workunit.client.1.smithi155.stdout:./271/.snap/snap-subdir-test 2024-03-28T19:00:56.930 INFO:tasks.workunit.client.1.smithi155.stdout:./251/.snap/snap-subdir-test 2024-03-28T19:00:56.947 INFO:tasks.workunit.client.1.smithi155.stdout:./175/.snap/snap-subdir-test 2024-03-28T19:00:56.973 INFO:tasks.workunit.client.1.smithi155.stdout:./222/.snap/snap-subdir-test 2024-03-28T19:00:56.994 INFO:tasks.workunit.client.1.smithi155.stdout:./220/.snap/snap-subdir-test 2024-03-28T19:00:57.010 INFO:tasks.workunit.client.1.smithi155.stdout:./203/.snap/snap-subdir-test 2024-03-28T19:00:57.027 INFO:tasks.workunit.client.1.smithi155.stdout:./181/.snap/snap-subdir-test 2024-03-28T19:00:57.041 INFO:tasks.workunit.client.1.smithi155.stdout:./286/.snap/snap-subdir-test 2024-03-28T19:00:57.058 INFO:tasks.workunit.client.1.smithi155.stdout:./127/.snap/snap-subdir-test 2024-03-28T19:00:57.077 INFO:tasks.workunit.client.1.smithi155.stdout:./108/.snap/snap-subdir-test 2024-03-28T19:00:57.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:00:57.089 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:00:57.097 INFO:tasks.workunit.client.1.smithi155.stdout:./183/.snap/snap-subdir-test 2024-03-28T19:00:57.113 INFO:tasks.workunit.client.1.smithi155.stdout:./294/.snap/snap-subdir-test 2024-03-28T19:00:57.130 INFO:tasks.workunit.client.1.smithi155.stdout:./235/.snap/snap-subdir-test 2024-03-28T19:00:57.147 INFO:tasks.workunit.client.1.smithi155.stdout:./177/.snap/snap-subdir-test 2024-03-28T19:00:57.165 INFO:tasks.workunit.client.1.smithi155.stdout:./169/.snap/snap-subdir-test 2024-03-28T19:00:57.180 INFO:tasks.workunit.client.1.smithi155.stdout:./212/.snap/snap-subdir-test 2024-03-28T19:00:57.195 INFO:tasks.workunit.client.1.smithi155.stdout:./154/.snap/snap-subdir-test 2024-03-28T19:00:57.212 INFO:tasks.workunit.client.1.smithi155.stdout:./171/.snap/snap-subdir-test 2024-03-28T19:00:57.229 INFO:tasks.workunit.client.1.smithi155.stdout:./139/.snap/snap-subdir-test 2024-03-28T19:00:57.246 INFO:tasks.workunit.client.1.smithi155.stdout:./102/.snap/snap-subdir-test 2024-03-28T19:00:57.267 INFO:tasks.workunit.client.1.smithi155.stdout:./129/.snap/snap-subdir-test 2024-03-28T19:00:57.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:00:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.989+0000 7f38c0200640 1 Processor -- start 2024-03-28T19:00:57.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.990+0000 7f38c0200640 1 -- start start 2024-03-28T19:00:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.992+0000 7f38c0200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b806e6f0 0x7f38b806eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:57.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.992+0000 7f38c0200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f38b8159bc0 0x7f38b815a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:57.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.992+0000 7f38c0200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b80c3830 con 0x7f38b8153ed0 2024-03-28T19:00:57.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.992+0000 7f38c0200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b80c6990 con 0x7f38b8159bc0 2024-03-28T19:00:57.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.992+0000 7f38c0200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80c66b0 con 0x7f38b806e6f0 2024-03-28T19:00:57.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.993+0000 7f38be200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f38b8159bc0 0x7f38b815a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:57.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.993+0000 7f38bd800640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f38b8153ed0 0x7f38b806ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:53392/0 (socket says 172.21.15.27:53392) 2024-03-28T19:00:57.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.993+0000 7f38be200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f38b8159bc0 0x7f38b815a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57028/0 (socket says 172.21.15.27:57028) 2024-03-28T19:00:57.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.994+0000 7f38bd800640 1 -- 172.21.15.27:0/3528609158 learned_addr learned my addr 172.21.15.27:0/3528609158 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:00:57.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.994+0000 7f38bce00640 1 --2- 172.21.15.27:0/3528609158 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b806e6f0 0x7f38b806eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:57.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.995+0000 7f38be200640 1 -- 172.21.15.27:0/3528609158 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b806e6f0 msgr2=0x7f38b806eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:00:57.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.995+0000 7f38be200640 1 --2- 172.21.15.27:0/3528609158 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b806e6f0 0x7f38b806eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:57.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.995+0000 7f38be200640 1 -- 172.21.15.27:0/3528609158 >> v1:172.21.15.155:6789/0 conn(0x7f38b8153ed0 legacy=0x7f38b806ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:57.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.995+0000 7f38be200640 1 -- 172.21.15.27:0/3528609158 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b80c3ad0 con 0x7f38b8159bc0 2024-03-28T19:00:57.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:57.999+0000 7f38be200640 1 --2- 172.21.15.27:0/3528609158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8159bc0 0x7f38b815a010 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009d90 tx=0x7f38b002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30cdcabe6a3ccbed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.000+0000 7f38b7e00640 1 -- 172.21.15.27:0/3528609158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0039070 con 0x7f38b8159bc0 2024-03-28T19:00:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.000+0000 7f38b7e00640 1 -- 172.21.15.27:0/3528609158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b002cc50 con 0x7f38b8159bc0 2024-03-28T19:00:57.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.000+0000 7f38b7e00640 1 -- 172.21.15.27:0/3528609158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0034440 con 0x7f38b8159bc0 2024-03-28T19:00:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.001+0000 7f38c0200640 1 -- 172.21.15.27:0/3528609158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8159bc0 msgr2=0x7f38b815a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:58.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.001+0000 7f38c0200640 1 --2- 172.21.15.27:0/3528609158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8159bc0 0x7f38b815a010 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009d90 tx=0x7f38b002bea0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.003+0000 7f38c0200640 1 -- 172.21.15.27:0/3528609158 shutdown_connections 2024-03-28T19:00:58.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.003+0000 7f38c0200640 1 --2- 172.21.15.27:0/3528609158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8159bc0 0x7f38b815a010 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.003+0000 7f38c0200640 1 --2- 172.21.15.27:0/3528609158 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b806e6f0 0x7f38b806eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.003+0000 7f38c0200640 1 -- 172.21.15.27:0/3528609158 >> 172.21.15.27:0/3528609158 conn(0x7f38b806b9d0 msgr2=0x7f38b80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:58.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.003+0000 7f38c0200640 1 -- 172.21.15.27:0/3528609158 shutdown_connections 2024-03-28T19:00:58.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.004+0000 7f38c0200640 1 -- 172.21.15.27:0/3528609158 wait complete. 2024-03-28T19:00:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.007+0000 7f38c0200640 1 Processor -- start 2024-03-28T19:00:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.008+0000 7f38c0200640 1 -- start start 2024-03-28T19:00:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.009+0000 7f38c0200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b806e6f0 0x7f38b8150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.009+0000 7f38c0200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8153ed0 0x7f38b81519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.009+0000 7f38c0200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b8159bc0 0x7f38b81f0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.010+0000 7f38c0200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80c1160 con 0x7f38b8153ed0 2024-03-28T19:00:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.010+0000 7f38c0200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80c6380 con 0x7f38b806e6f0 2024-03-28T19:00:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.010+0000 7f38c0200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80c12e0 con 0x7f38b8159bc0 2024-03-28T19:00:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.010+0000 7f38bce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8153ed0 0x7f38b81519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.010+0000 7f38be200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b8159bc0 0x7f38b81f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.010+0000 7f38be200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b8159bc0 0x7f38b81f0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45372/0 (socket says 172.21.15.27:45372) 2024-03-28T19:00:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.010+0000 7f38be200640 1 -- 172.21.15.27:0/423704912 learned_addr learned my addr 172.21.15.27:0/423704912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:00:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.010+0000 7f38bd800640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b806e6f0 0x7f38b8150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.011+0000 7f38be200640 1 -- 172.21.15.27:0/423704912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b806e6f0 msgr2=0x7f38b8150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.011+0000 7f38be200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b806e6f0 0x7f38b8150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.011+0000 7f38be200640 1 -- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8153ed0 msgr2=0x7f38b81519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.011+0000 7f38be200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8153ed0 0x7f38b81519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.011+0000 7f38be200640 1 -- 172.21.15.27:0/423704912 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b80a9d50 con 0x7f38b8159bc0 2024-03-28T19:00:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.011+0000 7f38bce00640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8153ed0 0x7f38b81519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:00:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.011+0000 7f38be200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b8159bc0 0x7f38b81f0500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009eb0 tx=0x7f38b0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.012+0000 7f38b5600640 1 -- 172.21.15.27:0/423704912 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0044020 con 0x7f38b8159bc0 2024-03-28T19:00:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.012+0000 7f38b5600640 1 -- 172.21.15.27:0/423704912 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b002ca50 con 0x7f38b8159bc0 2024-03-28T19:00:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.012+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b80a9a70 con 0x7f38b8159bc0 2024-03-28T19:00:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.012+0000 7f38b5600640 1 -- 172.21.15.27:0/423704912 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0039040 con 0x7f38b8159bc0 2024-03-28T19:00:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.012+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b80ab850 con 0x7f38b8159bc0 2024-03-28T19:00:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.013+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b804f400 con 0x7f38b8159bc0 2024-03-28T19:00:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.016+0000 7f38b5600640 1 -- 172.21.15.27:0/423704912 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f38b0047020 con 0x7f38b8159bc0 2024-03-28T19:00:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.017+0000 7f38b5600640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f389806bbd0 0x7f389806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:00:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.017+0000 7f38b5600640 1 -- 172.21.15.27:0/423704912 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7153+0+0 (secure 0 0 0) 0x7f38b00adb60 con 0x7f38b8159bc0 2024-03-28T19:00:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.017+0000 7f38bd800640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f389806bbd0 0x7f389806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:00:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.018+0000 7f38bd800640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f389806bbd0 0x7f389806e090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f38b8158790 tx=0x7f38ac007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:00:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.022+0000 7f38b5600640 1 -- 172.21.15.27:0/423704912 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f38b004e030 con 0x7f38b8159bc0 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/22' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/25' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/30' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/33' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/23' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/35' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/26' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './236/37' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './236' 2024-03-28T19:00:58.091 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/21' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/27' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/31' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/36' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/20' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/39' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/34' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/38' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/28' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/24' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/29' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/32' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/22' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/25' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/30' 2024-03-28T19:00:58.092 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/33' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/23' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/35' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/26' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './237/37' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './237' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/21' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/27' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/31' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/36' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/20' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/39' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/34' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/38' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/28' 2024-03-28T19:00:58.093 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/24' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/29' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/32' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/22' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/25' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/30' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/33' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/23' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/35' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/26' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './238/37' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './238' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/21' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/27' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/31' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/36' 2024-03-28T19:00:58.094 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/20' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/39' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/34' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/38' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/28' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/24' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/29' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/32' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/22' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/25' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/30' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/33' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/23' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/35' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/26' 2024-03-28T19:00:58.095 INFO:tasks.workunit.client.0.smithi027.stdout:removed './239/37' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './239' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/21' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/27' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/31' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/36' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/20' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/39' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/34' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/38' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/28' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/24' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/29' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/32' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/22' 2024-03-28T19:00:58.096 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/25' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/30' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/33' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/23' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/35' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/26' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './240/37' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './240' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/21' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/27' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/31' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/36' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/20' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/39' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/34' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/38' 2024-03-28T19:00:58.097 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/28' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/24' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/29' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/32' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/22' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/25' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/30' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/33' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/23' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/35' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/26' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './241/37' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './241' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/21' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/27' 2024-03-28T19:00:58.098 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/31' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/36' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/20' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/39' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/34' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/38' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/28' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/24' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/29' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/32' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/22' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/25' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/30' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/33' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/23' 2024-03-28T19:00:58.099 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/35' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/26' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './242/37' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './242' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/21' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/27' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/31' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/36' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/20' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/39' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/34' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/38' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/28' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/24' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/29' 2024-03-28T19:00:58.100 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/32' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/22' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/25' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/30' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/33' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/23' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/35' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/26' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './243/37' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './243' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/21' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/27' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/31' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/36' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/20' 2024-03-28T19:00:58.101 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/39' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/34' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/38' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/28' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/24' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/29' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/32' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/22' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/25' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/30' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/33' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/23' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/35' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/26' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed './244/37' 2024-03-28T19:00:58.102 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './244' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/21' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/27' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/31' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/36' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/20' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/39' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/34' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/38' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/28' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/24' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/29' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/32' 2024-03-28T19:00:58.103 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/22' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/25' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/30' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/33' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/23' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/35' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/26' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './245/37' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './245' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/21' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/27' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/31' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/36' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/20' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/39' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/34' 2024-03-28T19:00:58.104 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/38' 2024-03-28T19:00:58.105 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/28' 2024-03-28T19:00:58.105 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/24' 2024-03-28T19:00:58.105 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/29' 2024-03-28T19:00:58.105 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/32' 2024-03-28T19:00:58.105 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/22' 2024-03-28T19:00:58.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.384+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38b80ac190 con 0x7f38b8159bc0 2024-03-28T19:00:58.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.386+0000 7f38b5600640 1 -- 172.21.15.27:0/423704912 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f38b007bbe0 con 0x7f38b8159bc0 2024-03-28T19:00:58.386 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:00:58.386 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:00:58.386 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:00:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.389+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f389806bbd0 msgr2=0x7f389806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.389+0000 7f38c0200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f389806bbd0 0x7f389806e090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f38b8158790 tx=0x7f38ac007e80 comp rx=0 tx=0).stop 2024-03-28T19:00:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b8159bc0 msgr2=0x7f38b81f0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:00:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b8159bc0 0x7f38b81f0500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009eb0 tx=0x7f38b0004a40 comp rx=0 tx=0).stop 2024-03-28T19:00:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 shutdown_connections 2024-03-28T19:00:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f389806bbd0 0x7f389806e090 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b8159bc0 0x7f38b81f0500 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b8153ed0 0x7f38b81519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 --2- 172.21.15.27:0/423704912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b806e6f0 0x7f38b8150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:00:58.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 >> 172.21.15.27:0/423704912 conn(0x7f38b806b9d0 msgr2=0x7f38b8159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:00:58.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 shutdown_connections 2024-03-28T19:00:58.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:00:58.390+0000 7f38c0200640 1 -- 172.21.15.27:0/423704912 wait complete. 2024-03-28T19:00:58.409 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-03-28T19:00:58.409 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/25' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/30' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/33' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/23' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/35' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/26' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './246/37' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './246' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/21' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/27' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/31' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/36' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/20' 2024-03-28T19:00:59.864 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/39' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/34' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/38' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/28' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/24' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/29' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/32' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/22' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/25' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/30' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/33' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/23' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/35' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/26' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './247/37' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './247' 2024-03-28T19:00:59.865 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/21' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/27' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/31' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/36' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/20' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/39' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/34' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/38' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/28' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/24' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/29' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/32' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/22' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/25' 2024-03-28T19:00:59.866 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/30' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/33' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/23' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/35' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/26' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './248/37' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './248' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/21' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/27' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/31' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/36' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/20' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/39' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/34' 2024-03-28T19:00:59.867 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/38' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/28' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/24' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/29' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/32' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/22' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/25' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/30' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/33' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/23' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/35' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/26' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './249/37' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './249' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/21' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/27' 2024-03-28T19:00:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/31' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/36' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/20' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/39' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/34' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/38' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/28' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/24' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/29' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/32' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/22' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/25' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/30' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/33' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/23' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/35' 2024-03-28T19:00:59.869 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/26' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './250/37' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './250' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/21' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/27' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/31' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/36' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/20' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/39' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/34' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/38' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/28' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/24' 2024-03-28T19:00:59.870 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/29' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/32' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/22' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/25' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/30' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/33' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/23' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/35' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/26' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './251/37' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './251' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/21' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/27' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/31' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/36' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/20' 2024-03-28T19:00:59.871 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/39' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/34' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/38' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/28' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/24' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/29' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/32' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/22' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/25' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/30' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/33' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/23' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/35' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/26' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './252/37' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './252' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/21' 2024-03-28T19:00:59.872 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/27' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/31' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/36' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/20' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/39' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/34' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/38' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/28' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/24' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/29' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/32' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/22' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/25' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/30' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/33' 2024-03-28T19:00:59.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/23' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/35' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/26' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './253/37' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './253' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/21' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/27' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/31' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/36' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/20' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/39' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/34' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/38' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/28' 2024-03-28T19:00:59.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/24' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/29' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/32' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/22' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/25' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/30' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/33' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/23' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/35' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/26' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './254/37' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './254' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/21' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/27' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/31' 2024-03-28T19:00:59.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/36' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/20' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/39' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/34' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/38' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/28' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/24' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/29' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/32' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/22' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/25' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/30' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/33' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/23' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/35' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/26' 2024-03-28T19:00:59.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './255/37' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './255' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/21' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/27' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/31' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/36' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/20' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/39' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/34' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/38' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/28' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/24' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/29' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/32' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/22' 2024-03-28T19:00:59.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/25' 2024-03-28T19:00:59.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/30' 2024-03-28T19:01:03.811 INFO:tasks.workunit.client.1.smithi155.stdout:Delete all the files and directories ... 2024-03-28T19:01:05.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/19' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/15' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/31' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/36' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/20' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/12' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/39' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/10' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/34' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/38' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/11' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/16' 2024-03-28T19:01:05.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/32' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/30' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/14' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/18' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/13' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/33' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/17' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/35' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './100/37' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './100' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/19' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/15' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/31' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/36' 2024-03-28T19:01:05.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/20' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/12' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/39' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/10' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/34' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/38' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/11' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/16' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/32' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/30' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/14' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/18' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/13' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/33' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/17' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/35' 2024-03-28T19:01:05.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './101/37' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './101' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/19' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/15' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/31' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/36' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/20' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/12' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/39' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/10' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/34' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/38' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/11' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/16' 2024-03-28T19:01:05.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/32' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/30' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/14' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/18' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/13' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/33' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/17' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/35' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './102/37' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './102' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/19' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/15' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/31' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/36' 2024-03-28T19:01:05.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/20' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/12' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/39' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/10' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/34' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/38' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/11' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/16' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/32' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/30' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/14' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/18' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/13' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/33' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/17' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/35' 2024-03-28T19:01:05.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './103/37' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './103' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/19' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/15' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/31' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/36' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/20' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/12' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/39' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/10' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/34' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/38' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/11' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/16' 2024-03-28T19:01:05.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/32' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/30' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/14' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/18' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/13' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/33' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/17' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/35' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './104/37' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './104' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/19' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/15' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/31' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/36' 2024-03-28T19:01:05.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/20' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/12' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/39' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/10' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/34' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/38' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/11' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/16' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/32' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/30' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/14' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/18' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/13' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/33' 2024-03-28T19:01:05.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/17' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/35' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './105/37' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './105' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/19' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/15' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/31' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/36' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/20' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/12' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/39' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/10' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/34' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/38' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/11' 2024-03-28T19:01:05.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/16' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/32' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/30' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/14' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/18' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/13' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/33' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/17' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/35' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './106/37' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './106' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/19' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/15' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/31' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/36' 2024-03-28T19:01:05.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/20' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/12' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/39' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/10' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/34' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/38' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/11' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/16' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/32' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/30' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/14' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/18' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/13' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/33' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/17' 2024-03-28T19:01:05.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/35' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './107/37' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './107' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/19' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/15' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/31' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/36' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/20' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/12' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/39' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/10' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/34' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/38' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/11' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/16' 2024-03-28T19:01:05.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/32' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/30' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/14' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/18' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/13' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/33' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/17' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/35' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './108/37' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './108' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/19' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/15' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/31' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/36' 2024-03-28T19:01:05.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/20' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/12' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/39' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/10' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/34' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/38' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/11' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/16' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/32' 2024-03-28T19:01:05.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/30' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/33' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/23' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/35' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/26' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './256/37' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './256' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/21' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/27' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/31' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/36' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/20' 2024-03-28T19:01:05.682 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/39' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/34' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/38' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/28' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/24' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/29' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/32' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/22' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/25' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/30' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/33' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/23' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/35' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/26' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed './257/37' 2024-03-28T19:01:05.683 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './257' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/21' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/27' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/31' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/36' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/20' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/39' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/34' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/38' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/28' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/24' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/29' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/32' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/22' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/25' 2024-03-28T19:01:05.684 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/30' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/33' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/23' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/35' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/26' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './258/37' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './258' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/21' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/27' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/31' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/36' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/20' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/39' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/34' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/38' 2024-03-28T19:01:05.685 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/28' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/24' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/29' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/32' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/22' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/25' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/30' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/33' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/23' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/35' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/26' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './259/37' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './259' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/21' 2024-03-28T19:01:05.686 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/27' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/31' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/36' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/20' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/39' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/34' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/38' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/28' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/24' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/29' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/32' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/22' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/25' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/30' 2024-03-28T19:01:05.687 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/33' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/23' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/35' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/26' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './260/37' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './260' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/21' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/27' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/31' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/36' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/20' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/39' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/34' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/38' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/28' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/24' 2024-03-28T19:01:05.688 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/29' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/32' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/22' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/25' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/30' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/33' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/23' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/35' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/26' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './261/37' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './261' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/21' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/27' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/31' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/36' 2024-03-28T19:01:05.689 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/20' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/39' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/34' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/38' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/28' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/24' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/29' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/32' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/22' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/25' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/30' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/33' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/23' 2024-03-28T19:01:05.690 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/35' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/26' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './262/37' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './262' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/21' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/27' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/31' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/36' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/20' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/39' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/34' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/38' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/28' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/24' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/29' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/32' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/22' 2024-03-28T19:01:05.691 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/25' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/30' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/33' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/23' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/35' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/26' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './263/37' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './263' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/21' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/27' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/31' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/36' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/20' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/39' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/34' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/38' 2024-03-28T19:01:05.692 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/28' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/24' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/29' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/32' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/22' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/25' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/30' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/33' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/23' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/35' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/26' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './264/37' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './264' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/21' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/27' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/31' 2024-03-28T19:01:05.693 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/36' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/20' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/39' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/34' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/38' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/28' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/24' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/29' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/32' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/22' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/25' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/30' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/33' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/23' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/35' 2024-03-28T19:01:05.694 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/26' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './265/37' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './265' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/21' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/27' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/31' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/36' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/20' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/39' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/34' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/38' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/28' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/24' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/29' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/32' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/22' 2024-03-28T19:01:05.695 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/25' 2024-03-28T19:01:05.696 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/30' 2024-03-28T19:01:05.696 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/33' 2024-03-28T19:01:05.696 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/23' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/14' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/18' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/13' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/33' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/17' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/35' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './109/37' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './109' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/19' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/15' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/31' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/36' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/20' 2024-03-28T19:01:07.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/12' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/39' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/10' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/34' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/38' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/11' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/16' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/32' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/30' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/14' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/18' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/13' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/33' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/17' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/35' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './110/37' 2024-03-28T19:01:07.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './110' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/19' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/15' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/31' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/36' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/20' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/12' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/39' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/10' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/34' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/38' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/11' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/16' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/32' 2024-03-28T19:01:07.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/30' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/14' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/18' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/13' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/33' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/17' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/35' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './111/37' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './111' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/19' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/15' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/31' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/36' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/20' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/12' 2024-03-28T19:01:07.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/39' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/10' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/34' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/38' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/11' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/16' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/32' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/30' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/14' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/18' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/13' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/33' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/17' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/35' 2024-03-28T19:01:07.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './112/37' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './112' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/19' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/15' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/31' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/36' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/20' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/12' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/39' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/10' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/34' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/38' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/11' 2024-03-28T19:01:07.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/16' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/32' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/30' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/14' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/18' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/13' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/33' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/17' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/35' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './113/37' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './113' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/19' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/15' 2024-03-28T19:01:07.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/31' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/36' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/20' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/12' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/39' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/10' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/34' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/38' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/11' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/16' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/32' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/30' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/14' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/18' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/13' 2024-03-28T19:01:07.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/33' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/17' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/35' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './114/37' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './114' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/19' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/15' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/31' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/36' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/20' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/12' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/39' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/10' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/34' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/38' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/11' 2024-03-28T19:01:07.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/16' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/32' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/30' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/14' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/18' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/13' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/33' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/17' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/35' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './115/37' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './115' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/19' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/15' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/31' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/36' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/20' 2024-03-28T19:01:07.117 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/12' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/39' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/10' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/34' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/38' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/11' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/16' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/32' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/30' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/14' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/18' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/13' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/33' 2024-03-28T19:01:07.118 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/17' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/35' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './116/37' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './116' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/19' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/15' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/31' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/36' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/20' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/12' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/39' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/10' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/34' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/38' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/11' 2024-03-28T19:01:07.119 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/16' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/32' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/30' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/14' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/18' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/13' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/33' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/17' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/35' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './117/37' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './117' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/19' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/15' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/31' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/36' 2024-03-28T19:01:07.120 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/20' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/12' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/39' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/10' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/34' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/38' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/11' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/16' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/32' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/30' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/14' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/18' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/13' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/33' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/17' 2024-03-28T19:01:07.121 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/35' 2024-03-28T19:01:07.122 INFO:tasks.workunit.client.1.smithi155.stdout:removed './118/37' 2024-03-28T19:01:07.122 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './118' 2024-03-28T19:01:07.122 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/19' 2024-03-28T19:01:07.122 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/15' 2024-03-28T19:01:07.122 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/31' 2024-03-28T19:01:07.122 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/36' 2024-03-28T19:01:07.122 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/20' 2024-03-28T19:01:07.122 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/12' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/35' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/26' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './266/37' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './266' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/21' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/27' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/31' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/36' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/20' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/39' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/34' 2024-03-28T19:01:07.477 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/38' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/28' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/24' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/29' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/32' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/22' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/25' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/30' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/33' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/23' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/35' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/26' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './267/37' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './267' 2024-03-28T19:01:07.478 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/21' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/27' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/31' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/36' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/20' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/39' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/34' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/38' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/28' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/24' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/29' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/32' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/22' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/25' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/30' 2024-03-28T19:01:07.479 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/33' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/23' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/35' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/26' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './268/37' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './268' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/21' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/27' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/31' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/36' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/20' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/39' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/34' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/38' 2024-03-28T19:01:07.480 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/28' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/24' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/29' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/32' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/22' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/25' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/30' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/33' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/23' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/35' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/26' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './269/37' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './269' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/21' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/27' 2024-03-28T19:01:07.481 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/31' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/36' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/20' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/39' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/34' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/38' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/28' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/24' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/29' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/32' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/22' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/25' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/30' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/33' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/23' 2024-03-28T19:01:07.482 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/35' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/26' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './270/37' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './270' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/21' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/27' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/31' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/36' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/20' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/39' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/34' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/38' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/28' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/24' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/29' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/32' 2024-03-28T19:01:07.483 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/22' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/25' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/30' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/33' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/23' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/35' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/26' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './271/37' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './271' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/21' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/27' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/31' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/36' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/20' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/39' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/34' 2024-03-28T19:01:07.484 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/38' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/28' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/24' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/29' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/32' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/22' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/25' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/30' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/33' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/23' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/35' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/26' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './272/37' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './272' 2024-03-28T19:01:07.485 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/21' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/27' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/31' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/36' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/20' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/39' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/34' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/38' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/28' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/24' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/29' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/32' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/22' 2024-03-28T19:01:07.486 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/25' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/30' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/33' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/23' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/35' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/26' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './273/37' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './273' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/21' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/27' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/31' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/36' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/20' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/39' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/34' 2024-03-28T19:01:07.487 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/38' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/28' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/24' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/29' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/32' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/22' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/25' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/30' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/33' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/23' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/35' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/26' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './274/37' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './274' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/21' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/27' 2024-03-28T19:01:07.488 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/31' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/36' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/20' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/39' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/34' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/38' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/28' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/24' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/29' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/32' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/22' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/25' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/30' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/33' 2024-03-28T19:01:07.489 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/23' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/35' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/26' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './275/37' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './275' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/21' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/27' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/31' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/36' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/20' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/39' 2024-03-28T19:01:07.490 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/34' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/38' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/28' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/24' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/29' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/32' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/22' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/25' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/30' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/33' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/23' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/35' 2024-03-28T19:01:07.491 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/26' 2024-03-28T19:01:08.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/39' 2024-03-28T19:01:08.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/10' 2024-03-28T19:01:08.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/34' 2024-03-28T19:01:08.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/38' 2024-03-28T19:01:08.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/11' 2024-03-28T19:01:08.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/16' 2024-03-28T19:01:08.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/32' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/30' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/14' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/18' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/13' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/33' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/17' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/35' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './119/37' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './119' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/19' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/15' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/31' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/36' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/20' 2024-03-28T19:01:08.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/12' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/39' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/10' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/34' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/38' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/11' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/16' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/32' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/30' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/14' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/18' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/13' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/33' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/17' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/35' 2024-03-28T19:01:08.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './120/37' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './120' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/19' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/15' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/31' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/36' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/20' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/12' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/39' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/10' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/34' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/38' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/11' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/16' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/32' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/30' 2024-03-28T19:01:08.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/14' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/18' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/13' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/33' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/17' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/35' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './121/37' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './121' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/19' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/15' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/31' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/36' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/20' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/12' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/39' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/10' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/34' 2024-03-28T19:01:08.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/38' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/11' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/16' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/32' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/30' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/14' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/18' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/13' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/33' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/17' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/35' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './122/37' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './122' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/19' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/15' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/31' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/36' 2024-03-28T19:01:08.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/20' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/12' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/39' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/10' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/34' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/38' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/11' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/16' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/32' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/30' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/14' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/18' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/13' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/33' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/17' 2024-03-28T19:01:08.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/35' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './123/37' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './123' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/19' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/15' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/31' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/36' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/20' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/12' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/39' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/10' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/34' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/38' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/11' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/16' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/32' 2024-03-28T19:01:08.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/30' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/14' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/18' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/13' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/33' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/17' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/35' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './124/37' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './124' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/19' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/15' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/31' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/36' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/20' 2024-03-28T19:01:08.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/12' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/39' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/10' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/34' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/38' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/11' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/16' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/32' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/30' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/14' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/18' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/13' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/33' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/17' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/35' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './125/37' 2024-03-28T19:01:08.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './125' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/19' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/15' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/31' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/36' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/20' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/12' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/39' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/10' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/34' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/38' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/11' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/16' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/32' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/30' 2024-03-28T19:01:08.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/14' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/18' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/13' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/33' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/17' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/35' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './126/37' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './126' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/19' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/15' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/31' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/36' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/20' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/12' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/39' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/10' 2024-03-28T19:01:08.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/34' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/38' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/11' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/16' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/32' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/30' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/14' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/18' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/13' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/33' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/17' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/35' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './127/37' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './127' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/19' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/15' 2024-03-28T19:01:08.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/31' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/36' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/20' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/12' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/39' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/10' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/34' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/38' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/11' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/16' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/32' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/30' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/14' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/18' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/13' 2024-03-28T19:01:08.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/33' 2024-03-28T19:01:08.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/17' 2024-03-28T19:01:08.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/35' 2024-03-28T19:01:09.274 INFO:tasks.workunit.client.0.smithi027.stdout:removed './276/37' 2024-03-28T19:01:09.274 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './276' 2024-03-28T19:01:09.274 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/21' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/27' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/31' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/36' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/20' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/39' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/34' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/38' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/28' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/24' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/29' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/32' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/22' 2024-03-28T19:01:09.275 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/25' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/30' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/33' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/23' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/35' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/26' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './277/37' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './277' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/21' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/27' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/31' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/36' 2024-03-28T19:01:09.277 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/20' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/39' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/34' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/38' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/28' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/24' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/29' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/32' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/22' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/25' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/30' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/33' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/23' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/35' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/26' 2024-03-28T19:01:09.278 INFO:tasks.workunit.client.0.smithi027.stdout:removed './278/37' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './278' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/21' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/27' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/31' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/36' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/20' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/39' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/34' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/38' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/28' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/24' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/29' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/32' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/22' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/25' 2024-03-28T19:01:09.279 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/30' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/33' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/23' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/35' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/26' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './279/37' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './279' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/21' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/27' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/31' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/36' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/20' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/39' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/34' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/38' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/28' 2024-03-28T19:01:09.280 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/24' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/29' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/32' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/22' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/25' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/30' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/33' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/23' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/35' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/26' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './280/37' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './280' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/21' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/27' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/31' 2024-03-28T19:01:09.281 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/36' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/20' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/39' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/34' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/38' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/28' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/24' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/29' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/32' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/22' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/25' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/30' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/33' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/23' 2024-03-28T19:01:09.282 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/35' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/26' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './281/37' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './281' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/21' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/27' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/31' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/36' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/20' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/39' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/34' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/38' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/28' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/24' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/29' 2024-03-28T19:01:09.283 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/32' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/22' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/25' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/30' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/33' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/23' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/35' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/26' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './282/37' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './282' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/21' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/27' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/31' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/36' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/20' 2024-03-28T19:01:09.284 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/39' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/34' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/38' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/28' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/24' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/29' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/32' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/22' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/25' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/30' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/33' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/23' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/35' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/26' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './283/37' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './283' 2024-03-28T19:01:09.285 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/21' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/27' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/31' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/36' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/20' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/39' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/34' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/38' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/28' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/24' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/29' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/32' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/22' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/25' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/30' 2024-03-28T19:01:09.286 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/33' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/23' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/35' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/26' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './284/37' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './284' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/21' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/27' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/31' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/36' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/20' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/39' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/34' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/38' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/28' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/24' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/29' 2024-03-28T19:01:09.287 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/32' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/22' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/25' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/30' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/33' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/23' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/35' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/26' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './285/37' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './285' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/21' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/27' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/31' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/36' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/20' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/39' 2024-03-28T19:01:09.288 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/34' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/38' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/28' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/24' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/29' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/32' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/22' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/25' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/30' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/33' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/23' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/35' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/26' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed './286/37' 2024-03-28T19:01:09.289 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './286' 2024-03-28T19:01:10.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './128/37' 2024-03-28T19:01:10.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './128' 2024-03-28T19:01:10.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/19' 2024-03-28T19:01:10.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/15' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/31' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/36' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/20' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/12' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/39' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/10' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/34' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/38' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/11' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/16' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/32' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/30' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/14' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/18' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/13' 2024-03-28T19:01:10.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/33' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/17' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/35' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './129/37' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './129' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/19' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/15' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/31' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/36' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/20' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/12' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/39' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/10' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/34' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/38' 2024-03-28T19:01:10.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/11' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/16' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/32' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/30' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/14' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/18' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/13' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/33' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/17' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/35' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './130/37' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './130' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/19' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/15' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/31' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/36' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/20' 2024-03-28T19:01:10.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/12' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/39' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/10' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/34' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/38' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/11' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/16' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/32' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/30' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/14' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/18' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/13' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/33' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/17' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/35' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './131/37' 2024-03-28T19:01:10.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './131' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/19' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/15' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/31' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/36' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/20' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/12' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/39' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/10' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/34' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/38' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/11' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/16' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/32' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/30' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/14' 2024-03-28T19:01:10.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/18' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/13' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/33' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/17' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/35' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './132/37' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './132' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/19' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/15' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/31' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/36' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/20' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/12' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/39' 2024-03-28T19:01:10.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/10' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/34' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/38' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/11' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/16' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/32' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/30' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/14' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/18' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/13' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/33' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/17' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/35' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './133/37' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './133' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/19' 2024-03-28T19:01:10.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/15' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/31' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/36' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/20' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/12' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/39' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/10' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/34' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/38' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/11' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/16' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/32' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/30' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/14' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/18' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/13' 2024-03-28T19:01:10.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/33' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/17' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/35' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './134/37' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './134' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/19' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/15' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/31' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/36' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/20' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/12' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/39' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/10' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/34' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/38' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/11' 2024-03-28T19:01:10.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/16' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/32' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/30' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/14' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/18' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/13' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/33' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/17' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/35' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './135/37' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './135' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/19' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/15' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/31' 2024-03-28T19:01:10.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/36' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/20' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/12' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/39' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/10' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/34' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/38' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/11' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/16' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/32' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/30' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/14' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/18' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/13' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/33' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/17' 2024-03-28T19:01:10.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/35' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './136/37' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './136' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/19' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/15' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/31' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/36' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/20' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/12' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/39' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/10' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/34' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/38' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/11' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/16' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/32' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/30' 2024-03-28T19:01:10.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/14' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/18' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/13' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/33' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/17' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/35' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './137/37' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './137' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/19' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/15' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/31' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/36' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/20' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/12' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/39' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/10' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/34' 2024-03-28T19:01:10.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/38' 2024-03-28T19:01:10.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/11' 2024-03-28T19:01:10.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/16' 2024-03-28T19:01:11.065 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/21' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/27' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/31' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/36' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/20' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/39' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/34' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/38' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/28' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/24' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/29' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/32' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/22' 2024-03-28T19:01:11.066 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/25' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/30' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/33' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/23' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/35' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/26' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './287/37' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './287' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/21' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/27' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/31' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/36' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/20' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/39' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/34' 2024-03-28T19:01:11.067 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/38' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/28' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/24' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/29' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/32' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/22' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/25' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/30' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/33' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/23' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/35' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/26' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './288/37' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './288' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/21' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/27' 2024-03-28T19:01:11.068 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/31' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/36' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/20' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/39' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/34' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/38' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/28' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/24' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/29' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/32' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/22' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/25' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/30' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/33' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/23' 2024-03-28T19:01:11.069 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/35' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/26' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './289/37' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './289' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/21' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/27' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/31' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/36' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/20' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/39' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/34' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/38' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/28' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/24' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/29' 2024-03-28T19:01:11.070 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/32' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/22' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/25' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/30' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/33' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/23' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/35' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/26' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './290/37' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './290' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/21' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/27' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/31' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/36' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/20' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/39' 2024-03-28T19:01:11.071 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/34' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/38' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/28' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/24' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/29' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/32' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/22' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/25' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/30' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/33' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/23' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/35' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/26' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './291/37' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './291' 2024-03-28T19:01:11.072 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/21' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/27' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/31' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/36' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/20' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/39' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/34' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/38' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/28' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/24' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/29' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/32' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/22' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/25' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/30' 2024-03-28T19:01:11.073 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/33' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/23' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/35' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/26' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './292/37' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './292' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/21' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/27' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/31' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/36' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/20' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/39' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/34' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/38' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/28' 2024-03-28T19:01:11.074 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/24' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/29' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/32' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/22' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/25' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/30' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/33' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/23' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/35' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/26' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './293/37' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './293' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/21' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/27' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/31' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/36' 2024-03-28T19:01:11.075 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/20' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/39' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/34' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/38' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/28' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/24' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/29' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/32' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/22' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/25' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/30' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/33' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/23' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/35' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/26' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './294/37' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './294' 2024-03-28T19:01:11.076 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/21' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/27' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/31' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/36' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/20' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/39' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/34' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/38' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/28' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/24' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/29' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/32' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/22' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/25' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/30' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/33' 2024-03-28T19:01:11.077 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/23' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/35' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/26' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './295/37' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './295' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/21' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/27' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/31' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/36' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/20' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/39' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/34' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/38' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/28' 2024-03-28T19:01:11.078 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/24' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/29' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/32' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/22' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/25' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/30' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/33' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/23' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/35' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/26' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './296/37' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './296' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/21' 2024-03-28T19:01:11.079 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/27' 2024-03-28T19:01:11.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/32' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/30' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/14' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/18' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/13' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/33' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/17' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/35' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './138/37' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './138' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/19' 2024-03-28T19:01:11.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/15' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/31' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/36' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/20' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/12' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/39' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/10' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/34' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/38' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/11' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/16' 2024-03-28T19:01:11.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/32' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/30' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/14' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/18' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/13' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/33' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/17' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/35' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './139/37' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './139' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/19' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/15' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/31' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/36' 2024-03-28T19:01:11.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/20' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/12' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/39' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/10' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/34' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/38' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/11' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/16' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/32' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/30' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/14' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/18' 2024-03-28T19:01:11.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/13' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/33' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/17' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/35' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './140/37' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './140' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/19' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/15' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/31' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/36' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/20' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/12' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/39' 2024-03-28T19:01:11.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/10' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/34' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/38' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/11' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/16' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/32' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/30' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/14' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/18' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/13' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/33' 2024-03-28T19:01:11.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/17' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/35' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './141/37' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './141' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/19' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/15' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/31' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/36' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/20' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/12' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/39' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/10' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/34' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/38' 2024-03-28T19:01:11.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/11' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/16' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/32' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/30' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/14' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/18' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/13' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/33' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/17' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/35' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './142/37' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './142' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/19' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/15' 2024-03-28T19:01:11.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/31' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/36' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/20' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/12' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/39' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/10' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/34' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/38' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/11' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/16' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/32' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/30' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/14' 2024-03-28T19:01:11.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/18' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/13' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/33' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/17' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/35' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './143/37' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './143' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/19' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/15' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/31' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/36' 2024-03-28T19:01:11.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/20' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/12' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/39' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/10' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/34' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/38' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/11' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/16' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/32' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/30' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/14' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/18' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/13' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/33' 2024-03-28T19:01:11.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/17' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/35' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './144/37' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './144' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/19' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/15' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/31' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/36' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/20' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/12' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/39' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/10' 2024-03-28T19:01:11.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/34' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/38' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/11' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/16' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/32' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/30' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/14' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/18' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/13' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/33' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/17' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/35' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './145/37' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './145' 2024-03-28T19:01:11.537 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/19' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/15' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/31' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/36' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/20' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/12' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/39' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/10' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/34' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/38' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/11' 2024-03-28T19:01:11.538 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/16' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/32' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/30' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/14' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/18' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/13' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/33' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/17' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/35' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './146/37' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './146' 2024-03-28T19:01:11.539 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/19' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/15' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/31' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/36' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/20' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/12' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/39' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/10' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/34' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/38' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/11' 2024-03-28T19:01:11.540 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/16' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/32' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/30' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/14' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/18' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/13' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/33' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/17' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/35' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './147/37' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './147' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/19' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/15' 2024-03-28T19:01:11.541 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/31' 2024-03-28T19:01:11.542 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/36' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/31' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/36' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/20' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/39' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/34' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/38' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/28' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/24' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/29' 2024-03-28T19:01:12.829 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/32' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/22' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/25' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/30' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/33' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/23' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/35' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/26' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './297/37' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './297' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/21' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/27' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/31' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/36' 2024-03-28T19:01:12.830 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/20' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/39' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/34' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/38' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/28' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/24' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/29' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/32' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/22' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/25' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/30' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/33' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/23' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/35' 2024-03-28T19:01:12.831 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/26' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './298/37' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './298' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/21' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/27' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/31' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/36' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/20' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/39' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/34' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/38' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/28' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/24' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/29' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/32' 2024-03-28T19:01:12.832 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/22' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/25' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/30' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/33' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/23' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/35' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/26' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './299/37' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './299' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/19' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/15' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/20' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/12' 2024-03-28T19:01:12.833 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/10' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/11' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/16' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/14' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/18' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/13' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/166/17' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/166' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/19' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/15' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/20' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/12' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/10' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/11' 2024-03-28T19:01:12.834 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/16' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/14' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/18' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/13' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/147/17' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/147' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/19' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/15' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/20' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/12' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/10' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/11' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/16' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/14' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/18' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/13' 2024-03-28T19:01:12.835 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/151/17' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/151' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/19' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/15' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/20' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/12' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/10' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/11' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/16' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/14' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/18' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/13' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/170/17' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/170' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/19' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/15' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/20' 2024-03-28T19:01:12.836 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/12' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/10' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/11' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/16' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/14' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/18' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/13' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/167/17' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/167' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/19' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/15' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/20' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/12' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/10' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/11' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/16' 2024-03-28T19:01:12.837 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/14' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/18' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/13' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/140/17' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/140' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/19' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/15' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/20' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/12' 2024-03-28T19:01:12.838 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/10' 2024-03-28T19:01:13.287 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/20' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/12' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/39' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/10' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/34' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/38' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/11' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/16' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/32' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/30' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/14' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/18' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/13' 2024-03-28T19:01:13.288 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/33' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/17' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/35' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './148/37' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './148' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/19' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/15' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/31' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/36' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/20' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/12' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/39' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/10' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/34' 2024-03-28T19:01:13.289 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/38' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/11' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/16' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/32' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/30' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/14' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/18' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/13' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/33' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/17' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/35' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './149/37' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './149' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/19' 2024-03-28T19:01:13.290 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/15' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/31' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/36' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/20' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/12' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/39' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/10' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/34' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/38' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/11' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/16' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/32' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/30' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/14' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/18' 2024-03-28T19:01:13.291 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/13' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/33' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/17' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/35' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './150/37' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './150' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/19' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/15' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/31' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/36' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/20' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/12' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/39' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/10' 2024-03-28T19:01:13.292 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/34' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/38' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/11' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/16' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/32' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/30' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/14' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/18' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/13' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/33' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/17' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/35' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './151/37' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './151' 2024-03-28T19:01:13.293 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/19' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/15' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/31' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/36' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/20' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/12' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/39' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/10' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/34' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/38' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/11' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/16' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/32' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/30' 2024-03-28T19:01:13.294 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/14' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/18' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/13' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/33' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/17' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/35' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './152/37' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './152' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/19' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/15' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/31' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/36' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/20' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/12' 2024-03-28T19:01:13.295 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/39' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/10' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/34' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/38' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/11' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/16' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/32' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/30' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/14' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/18' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/13' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/33' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/17' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/35' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed './153/37' 2024-03-28T19:01:13.296 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './153' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/19' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/15' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/31' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/36' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/20' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/12' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/39' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/10' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/34' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/38' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/11' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/16' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/32' 2024-03-28T19:01:13.297 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/30' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/14' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/18' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/13' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/33' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/17' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/35' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './154/37' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './154' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/19' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/15' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/31' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/36' 2024-03-28T19:01:13.298 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/20' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/12' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/39' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/10' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/34' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/38' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/11' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/16' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/32' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/30' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/14' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/18' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/13' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/33' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/17' 2024-03-28T19:01:13.299 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/35' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './155/37' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './155' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/19' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/15' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/31' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/36' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/20' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/12' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/39' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/10' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/34' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/38' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/11' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/16' 2024-03-28T19:01:13.300 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/32' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/30' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/14' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/18' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/13' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/33' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/17' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/35' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './156/37' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './156' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/19' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/15' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/31' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/36' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/20' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/12' 2024-03-28T19:01:13.301 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/39' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/10' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/34' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/38' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/11' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/16' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/32' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/30' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/14' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/18' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/13' 2024-03-28T19:01:13.302 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/33' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/11' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/16' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/14' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/18' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/13' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/142/17' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/142' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/19' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/15' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/20' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/12' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/10' 2024-03-28T19:01:13.919 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/11' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/16' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/14' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/18' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/13' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/150/17' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/150' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/19' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/15' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/20' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/12' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/10' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/11' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/16' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/14' 2024-03-28T19:01:13.920 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/18' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/13' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/105/17' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/105' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/19' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/15' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/20' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/12' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/10' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/11' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/16' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/14' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/18' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/13' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/149/17' 2024-03-28T19:01:13.921 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/149' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/19' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/15' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/20' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/12' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/10' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/11' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/16' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/14' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/18' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/13' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/109/17' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/109' 2024-03-28T19:01:13.922 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/19' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/15' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/20' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/12' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/10' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/11' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/16' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/14' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/18' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/13' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/158/17' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/158' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/19' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/15' 2024-03-28T19:01:13.923 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/20' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/12' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/10' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/11' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/16' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/14' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/18' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/13' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/106/17' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/106' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/19' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/15' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/20' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/12' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/10' 2024-03-28T19:01:13.924 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/11' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/16' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/14' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/18' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/13' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/189/17' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/189' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/19' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/15' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/20' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/12' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/10' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/11' 2024-03-28T19:01:13.925 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/16' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/14' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/18' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/13' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/133/17' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/133' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/19' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/15' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/20' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/12' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/10' 2024-03-28T19:01:13.926 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/11' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/17' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/35' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './157/37' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './157' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/19' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/15' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/31' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/36' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/20' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/12' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/39' 2024-03-28T19:01:14.516 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/10' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/34' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/38' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/11' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/16' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/32' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/30' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/14' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/18' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/13' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/33' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/17' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/35' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './158/37' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './158' 2024-03-28T19:01:14.517 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/19' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/15' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/31' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/36' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/20' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/12' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/39' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/10' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/34' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/38' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/11' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/16' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/32' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/30' 2024-03-28T19:01:14.518 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/14' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/18' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/13' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/33' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/17' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/35' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './159/37' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './159' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/19' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/15' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/31' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/36' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/20' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/12' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/39' 2024-03-28T19:01:14.519 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/10' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/34' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/38' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/11' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/16' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/32' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/30' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/14' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/18' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/13' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/33' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/17' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/35' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed './160/37' 2024-03-28T19:01:14.520 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './160' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/19' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/15' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/31' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/36' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/20' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/12' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/39' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/10' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/34' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/38' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/11' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/16' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/32' 2024-03-28T19:01:14.521 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/30' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/14' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/18' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/13' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/33' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/17' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/35' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './161/37' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './161' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/19' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/15' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/31' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/36' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/20' 2024-03-28T19:01:14.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/12' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/39' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/10' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/34' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/38' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/11' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/16' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/32' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/30' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/14' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/18' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/13' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/33' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/17' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/35' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './162/37' 2024-03-28T19:01:14.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './162' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/19' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/15' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/31' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/36' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/20' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/12' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/39' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/10' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/34' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/38' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/11' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/16' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/32' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/30' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/14' 2024-03-28T19:01:14.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/18' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/13' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/33' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/17' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/35' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './163/37' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './163' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/19' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/15' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/31' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/36' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/20' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/12' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/39' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/10' 2024-03-28T19:01:14.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/34' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/38' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/11' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/16' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/32' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/30' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/14' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/18' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/13' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/33' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/17' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/35' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './164/37' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './164' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/19' 2024-03-28T19:01:14.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/15' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/31' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/36' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/20' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/12' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/39' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/10' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/34' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/38' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/11' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/16' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/32' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/30' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/14' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/18' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/13' 2024-03-28T19:01:14.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/33' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/17' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/35' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './165/37' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './165' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/19' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/15' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/31' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/36' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/20' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/12' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/39' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/10' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/34' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/38' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/11' 2024-03-28T19:01:14.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/16' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/32' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/30' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/14' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/18' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/13' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/33' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/17' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/35' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './166/37' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './166' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/19' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/15' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/31' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/36' 2024-03-28T19:01:14.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/20' 2024-03-28T19:01:14.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/12' 2024-03-28T19:01:14.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/39' 2024-03-28T19:01:14.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/10' 2024-03-28T19:01:14.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/34' 2024-03-28T19:01:14.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/38' 2024-03-28T19:01:15.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/16' 2024-03-28T19:01:15.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/14' 2024-03-28T19:01:15.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/18' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/13' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/173/17' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/173' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/19' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/15' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/20' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/12' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/10' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/11' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/16' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/14' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/18' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/13' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/174/17' 2024-03-28T19:01:15.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/174' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/19' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/15' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/20' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/12' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/10' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/11' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/16' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/14' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/18' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/13' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/155/17' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/155' 2024-03-28T19:01:15.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/19' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/15' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/20' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/12' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/10' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/11' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/16' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/14' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/18' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/13' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/114/17' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/114' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/19' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/15' 2024-03-28T19:01:15.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/20' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/12' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/10' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/11' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/16' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/14' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/18' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/13' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/159/17' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/159' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/19' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/15' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/20' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/12' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/10' 2024-03-28T19:01:15.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/11' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/16' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/14' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/18' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/13' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/187/17' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/187' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/19' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/15' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/20' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/12' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/10' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/11' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/16' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/14' 2024-03-28T19:01:15.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/18' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/13' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/185/17' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/185' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/19' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/15' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/20' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/12' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/10' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/11' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/16' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/14' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/18' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/13' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/120/17' 2024-03-28T19:01:15.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/120' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/19' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/15' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/20' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/12' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/10' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/11' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/16' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/14' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/18' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/13' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/110/17' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/110' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/19' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/15' 2024-03-28T19:01:15.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/20' 2024-03-28T19:01:15.118 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/12' 2024-03-28T19:01:15.118 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/10' 2024-03-28T19:01:15.118 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/11' 2024-03-28T19:01:15.118 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/16' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/11' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/16' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/32' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/30' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/14' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/18' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/13' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/33' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/17' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/35' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed './167/37' 2024-03-28T19:01:15.860 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './167' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/19' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/15' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/31' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/36' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/20' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/12' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/39' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/10' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/34' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/38' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/11' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/16' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/32' 2024-03-28T19:01:15.861 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/30' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/14' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/18' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/13' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/33' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/17' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/35' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './168/37' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './168' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/19' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/15' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/31' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/36' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/20' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/12' 2024-03-28T19:01:15.862 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/39' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/10' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/34' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/38' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/11' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/16' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/32' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/30' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/14' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/18' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/13' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/33' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/17' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/35' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './169/37' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './169' 2024-03-28T19:01:15.863 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/19' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/15' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/31' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/36' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/20' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/12' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/39' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/10' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/34' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/38' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/11' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/16' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/32' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/30' 2024-03-28T19:01:15.864 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/14' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/18' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/13' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/33' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/17' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/35' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './170/37' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './170' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/19' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/15' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/31' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/36' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/20' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/12' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/39' 2024-03-28T19:01:15.865 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/10' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/34' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/38' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/11' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/16' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/32' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/30' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/14' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/18' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/13' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/33' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/17' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/35' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed './171/37' 2024-03-28T19:01:15.866 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './171' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/19' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/15' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/31' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/36' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/20' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/12' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/39' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/10' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/34' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/38' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/11' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/16' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/32' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/30' 2024-03-28T19:01:15.867 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/14' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/18' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/13' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/33' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/17' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/35' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './172/37' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './172' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/19' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/15' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/31' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/36' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/20' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/12' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/39' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/10' 2024-03-28T19:01:15.868 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/34' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/38' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/11' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/16' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/32' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/30' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/14' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/18' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/13' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/33' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/17' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/35' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './173/37' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './173' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/19' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/15' 2024-03-28T19:01:15.869 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/31' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/36' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/20' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/12' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/39' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/10' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/34' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/38' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/11' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/16' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/32' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/30' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/14' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/18' 2024-03-28T19:01:15.870 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/13' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/33' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/17' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/35' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './174/37' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './174' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/19' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/15' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/31' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/36' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/20' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/12' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/39' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/10' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/34' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/38' 2024-03-28T19:01:15.871 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/11' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/16' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/32' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/30' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/14' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/18' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/13' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/33' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/17' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/35' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './175/37' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './175' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/19' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/15' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/31' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/36' 2024-03-28T19:01:15.872 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/20' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/12' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/39' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/10' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/34' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/38' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/11' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/16' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/32' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/30' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/14' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/18' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/13' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/33' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/17' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/35' 2024-03-28T19:01:15.873 INFO:tasks.workunit.client.1.smithi155.stdout:removed './176/37' 2024-03-28T19:01:15.874 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './176' 2024-03-28T19:01:15.874 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/19' 2024-03-28T19:01:15.874 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/15' 2024-03-28T19:01:16.184 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/14' 2024-03-28T19:01:16.184 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/18' 2024-03-28T19:01:16.184 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/13' 2024-03-28T19:01:16.184 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/137/17' 2024-03-28T19:01:16.184 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/137' 2024-03-28T19:01:16.184 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/19' 2024-03-28T19:01:16.184 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/15' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/20' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/12' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/10' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/11' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/16' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/14' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/18' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/13' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/198/17' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/198' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/19' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/15' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/20' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/12' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/10' 2024-03-28T19:01:16.185 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/11' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/16' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/14' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/18' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/13' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/136/17' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/136' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/19' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/15' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/20' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/12' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/10' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/11' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/16' 2024-03-28T19:01:16.186 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/14' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/18' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/13' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/125/17' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/125' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/19' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/15' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/20' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/12' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/10' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/11' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/16' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/14' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/18' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/13' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/194/17' 2024-03-28T19:01:16.187 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/194' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/19' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/15' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/20' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/12' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/10' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/11' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/16' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/14' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/18' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/13' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/119/17' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/119' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/19' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/15' 2024-03-28T19:01:16.188 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/20' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/12' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/10' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/11' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/16' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/14' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/18' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/13' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/157/17' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/157' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/19' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/15' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/20' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/12' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/10' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/11' 2024-03-28T19:01:16.189 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/16' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/14' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/18' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/13' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/172/17' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/172' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/19' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/15' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/20' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/12' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/10' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/11' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/16' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/14' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/18' 2024-03-28T19:01:16.190 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/13' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/186/17' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/186' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/19' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/15' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/20' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/12' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/10' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/11' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/16' 2024-03-28T19:01:16.191 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/14' 2024-03-28T19:01:16.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:16.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.639+0000 7f80c3200640 1 Processor -- start 2024-03-28T19:01:16.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.639+0000 7f80c3200640 1 -- start start 2024-03-28T19:01:16.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.640+0000 7f80c3200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f80bc0c6a90 0x7f80bc06b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:16.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.640+0000 7f80c3200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80bc06d080 con 0x7f80bc0c6e70 2024-03-28T19:01:16.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.640+0000 7f80c3200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80bc06cb00 con 0x7f80bc0c72f0 2024-03-28T19:01:16.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.640+0000 7f80c3200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80bc159980 con 0x7f80bc0c6a90 2024-03-28T19:01:16.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.641+0000 7f80c1400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f80bc0c6e70 0x7f80bc156ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37862/0 (socket says 172.21.15.27:37862) 2024-03-28T19:01:16.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.641+0000 7f80c1400640 1 -- 172.21.15.27:0/1171232409 learned_addr learned my addr 172.21.15.27:0/1171232409 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:16.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.641+0000 7f80c1e00640 1 --2- 172.21.15.27:0/1171232409 >> v2:172.21.15.155:3300/0 conn(0x7f80bc0c6a90 0x7f80bc06b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:16.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.641+0000 7f80c0a00640 1 -- 172.21.15.27:0/1171232409 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200449425 0 0) 0x7f80bc06d080 con 0x7f80bc0c6e70 2024-03-28T19:01:16.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.641+0000 7f80c0a00640 1 -- 172.21.15.27:0/1171232409 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a4003520 con 0x7f80bc0c6e70 2024-03-28T19:01:16.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.641+0000 7f80c0a00640 1 -- 172.21.15.27:0/1171232409 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646921626 0 0) 0x7f80bc06cb00 con 0x7f80bc0c72f0 2024-03-28T19:01:16.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.641+0000 7f80c0a00640 1 -- 172.21.15.27:0/1171232409 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80bc06d080 con 0x7f80bc0c72f0 2024-03-28T19:01:16.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.642+0000 7f80c1e00640 1 -- 172.21.15.27:0/1171232409 >> v1:172.21.15.155:6789/0 conn(0x7f80bc0c72f0 legacy=0x7f80bc1572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:16.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.642+0000 7f80c1e00640 1 -- 172.21.15.27:0/1171232409 >> v1:172.21.15.27:6789/0 conn(0x7f80bc0c6e70 legacy=0x7f80bc156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:16.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.642+0000 7f80c0a00640 1 -- 172.21.15.27:0/1171232409 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127881606 0 0) 0x7f80a4003520 con 0x7f80bc0c6e70 2024-03-28T19:01:16.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.642+0000 7f80c1e00640 1 -- 172.21.15.27:0/1171232409 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc0c7780 con 0x7f80bc0c6a90 2024-03-28T19:01:16.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.643+0000 7f80c1e00640 1 --2- 172.21.15.27:0/1171232409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6a90 0x7f80bc06b0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f80b00097a0 tx=0x7f80b002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ad51cd8cf5fcc17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:16.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.644+0000 7f80c0a00640 1 -- 172.21.15.27:0/1171232409 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0037070 con 0x7f80bc0c6a90 2024-03-28T19:01:16.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.644+0000 7f80c0a00640 1 -- 172.21.15.27:0/1171232409 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b0030510 con 0x7f80bc0c6a90 2024-03-28T19:01:16.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.644+0000 7f80c0a00640 1 -- 172.21.15.27:0/1171232409 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac001670 con 0x7f80bc0c6a90 2024-03-28T19:01:16.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.644+0000 7f80c3200640 1 -- 172.21.15.27:0/1171232409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6a90 msgr2=0x7f80bc06b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:16.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.644+0000 7f80c3200640 1 --2- 172.21.15.27:0/1171232409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6a90 0x7f80bc06b0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f80b00097a0 tx=0x7f80b002b790 comp rx=0 tx=0).stop 2024-03-28T19:01:16.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.644+0000 7f80c3200640 1 -- 172.21.15.27:0/1171232409 shutdown_connections 2024-03-28T19:01:16.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.644+0000 7f80c3200640 1 --2- 172.21.15.27:0/1171232409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6a90 0x7f80bc06b0c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:16.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.644+0000 7f80c3200640 1 -- 172.21.15.27:0/1171232409 >> 172.21.15.27:0/1171232409 conn(0x7f80bc0aa7c0 msgr2=0x7f80bc0aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:16.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.645+0000 7f80c3200640 1 -- 172.21.15.27:0/1171232409 shutdown_connections 2024-03-28T19:01:16.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.645+0000 7f80c3200640 1 -- 172.21.15.27:0/1171232409 wait complete. 2024-03-28T19:01:16.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.646+0000 7f80c3200640 1 Processor -- start 2024-03-28T19:01:16.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.647+0000 7f80c3200640 1 -- start start 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc0c6a90 0x7f80bc1ec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6e70 0x7f80bc1eefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80bc0c72f0 0x7f80bc14f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc0c8630 con 0x7f80bc0c6a90 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc159610 con 0x7f80bc0c6e70 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc152e90 con 0x7f80bc0c72f0 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc0c6a90 0x7f80bc1ec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc0c6a90 0x7f80bc1ec4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55886/0 (socket says 172.21.15.27:55886) 2024-03-28T19:01:16.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c1e00640 1 -- 172.21.15.27:0/1533145395 learned_addr learned my addr 172.21.15.27:0/1533145395 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:16.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.648+0000 7f80c1400640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6e70 0x7f80bc1eefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:16.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.649+0000 7f80c2800640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80bc0c72f0 0x7f80bc14f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:16.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.649+0000 7f80c1e00640 1 -- 172.21.15.27:0/1533145395 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80bc0c72f0 msgr2=0x7f80bc14f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.649+0000 7f80c1e00640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80bc0c72f0 0x7f80bc14f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.649+0000 7f80c1e00640 1 -- 172.21.15.27:0/1533145395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6e70 msgr2=0x7f80bc1eefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.649+0000 7f80c1e00640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6e70 0x7f80bc1eefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.649+0000 7f80c1e00640 1 -- 172.21.15.27:0/1533145395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc0c0090 con 0x7f80bc0c6a90 2024-03-28T19:01:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.650+0000 7f80c1e00640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc0c6a90 0x7f80bc1ec4b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f80b002b7d0 tx=0x7f80b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.650+0000 7f80b6000640 1 -- 172.21.15.27:0/1533145395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0041030 con 0x7f80bc0c6a90 2024-03-28T19:01:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.650+0000 7f80b6000640 1 -- 172.21.15.27:0/1533145395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b003c670 con 0x7f80bc0c6a90 2024-03-28T19:01:16.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.650+0000 7f80b6000640 1 -- 172.21.15.27:0/1533145395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b0037040 con 0x7f80bc0c6a90 2024-03-28T19:01:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.651+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80bc0c0370 con 0x7f80bc0c6a90 2024-03-28T19:01:16.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.651+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80bc0c20c0 con 0x7f80bc0c6a90 2024-03-28T19:01:16.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.652+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc067dd0 con 0x7f80bc0c6a90 2024-03-28T19:01:16.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.654+0000 7f80b6000640 1 -- 172.21.15.27:0/1533145395 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f80b0044070 con 0x7f80bc0c6a90 2024-03-28T19:01:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.655+0000 7f80b6000640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80a006bbd0 0x7f80a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.655+0000 7f80b6000640 1 -- 172.21.15.27:0/1533145395 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7f80b00ad180 con 0x7f80bc0c6a90 2024-03-28T19:01:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.655+0000 7f80c1400640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80a006bbd0 0x7f80a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:16.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.656+0000 7f80c1400640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80a006bbd0 0x7f80a006e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f80bc157310 tx=0x7f80ac001710 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:16.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.664+0000 7f80b6000640 1 -- 172.21.15.27:0/1533145395 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f80b007aed0 con 0x7f80bc0c6a90 2024-03-28T19:01:16.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.996+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80bc069640 con 0x7f80bc0c6a90 2024-03-28T19:01:16.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:16.998+0000 7f80b6000640 1 -- 172.21.15.27:0/1533145395 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f80b007ef70 con 0x7f80bc0c6a90 2024-03-28T19:01:16.997 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:16.997 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:16.997 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:17.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.001+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80a006bbd0 msgr2=0x7f80a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.001+0000 7f80c3200640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80a006bbd0 0x7f80a006e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f80bc157310 tx=0x7f80ac001710 comp rx=0 tx=0).stop 2024-03-28T19:01:17.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.001+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc0c6a90 msgr2=0x7f80bc1ec4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.001+0000 7f80c3200640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc0c6a90 0x7f80bc1ec4b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f80b002b7d0 tx=0x7f80b00046c0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.002+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 shutdown_connections 2024-03-28T19:01:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.002+0000 7f80c3200640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f80a006bbd0 0x7f80a006e090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.002+0000 7f80c3200640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80bc0c72f0 0x7f80bc14f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.002+0000 7f80c3200640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80bc0c6e70 0x7f80bc1eefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.002+0000 7f80c3200640 1 --2- 172.21.15.27:0/1533145395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc0c6a90 0x7f80bc1ec4b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.002+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 >> 172.21.15.27:0/1533145395 conn(0x7f80bc0aa7c0 msgr2=0x7f80bc06e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.002+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 shutdown_connections 2024-03-28T19:01:17.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.002+0000 7f80c3200640 1 -- 172.21.15.27:0/1533145395 wait complete. 2024-03-28T19:01:17.024 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-03-28T19:01:17.025 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T19:01:17.025 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:01:17.064 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T19:01:17.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/18' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/13' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/190/17' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/190' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/19' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/15' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/20' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/12' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/10' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/11' 2024-03-28T19:01:17.110 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/16' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/14' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/18' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/13' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/104/17' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/104' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/19' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/15' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/20' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/12' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/10' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/11' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/16' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/14' 2024-03-28T19:01:17.111 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/18' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/13' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/164/17' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/164' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/19' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/15' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/20' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/12' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/10' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/11' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/16' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/14' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/18' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/13' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/112/17' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/112' 2024-03-28T19:01:17.112 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/19' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/15' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/20' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/12' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/10' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/11' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/16' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/14' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/18' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/13' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/100/17' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/100' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/19' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/15' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/20' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/12' 2024-03-28T19:01:17.113 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/10' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/11' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/16' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/14' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/18' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/13' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/134/17' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/134' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/19' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/15' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/20' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/12' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/10' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/11' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/16' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/14' 2024-03-28T19:01:17.114 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/18' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/13' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/115/17' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/115' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/19' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/15' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/20' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/12' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/10' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/11' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/16' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/14' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/18' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/13' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/116/17' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/116' 2024-03-28T19:01:17.115 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/19' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/15' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/20' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/12' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/10' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/11' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/16' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/14' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/18' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/13' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/195/17' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/195' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/19' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/15' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/20' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/12' 2024-03-28T19:01:17.116 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/10' 2024-03-28T19:01:17.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/11' 2024-03-28T19:01:17.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/16' 2024-03-28T19:01:17.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/14' 2024-03-28T19:01:17.117 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/18' 2024-03-28T19:01:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.309+0000 7f3d69200640 1 Processor -- start 2024-03-28T19:01:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.309+0000 7f3d69200640 1 -- start start 2024-03-28T19:01:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.311+0000 7f3d69200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d640c69a0 0x7f3d640c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.311+0000 7f3d69200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d64150ec0 con 0x7f3d64150ae0 2024-03-28T19:01:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.311+0000 7f3d69200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6406dc70 con 0x7f3d64153a30 2024-03-28T19:01:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.311+0000 7f3d69200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d64153550 con 0x7f3d640c69a0 2024-03-28T19:01:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.311+0000 7f3d63400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d64150ae0 0x7f3d640a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37864/0 (socket says 172.21.15.27:37864) 2024-03-28T19:01:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.311+0000 7f3d63400640 1 -- 172.21.15.27:0/663635981 learned_addr learned my addr 172.21.15.27:0/663635981 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.311+0000 7f3d5be00640 1 --2- 172.21.15.27:0/663635981 >> v2:172.21.15.27:3300/0 conn(0x7f3d640c69a0 0x7f3d640c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.312+0000 7f3d62a00640 1 -- 172.21.15.27:0/663635981 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813215324 0 0) 0x7f3d54001110 con 0x7f3d64150ae0 2024-03-28T19:01:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.312+0000 7f3d62a00640 1 -- 172.21.15.27:0/663635981 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48003520 con 0x7f3d64150ae0 2024-03-28T19:01:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.312+0000 7f3d5be00640 1 -- 172.21.15.27:0/663635981 >> v1:172.21.15.155:6789/0 conn(0x7f3d64153a30 legacy=0x7f3d6415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.312+0000 7f3d5be00640 1 -- 172.21.15.27:0/663635981 >> v1:172.21.15.27:6789/0 conn(0x7f3d64150ae0 legacy=0x7f3d640a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.312+0000 7f3d5be00640 1 -- 172.21.15.27:0/663635981 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6406df10 con 0x7f3d640c69a0 2024-03-28T19:01:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.314+0000 7f3d5be00640 1 --2- 172.21.15.27:0/663635981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d640c69a0 0x7f3d640c6e30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0097a0 tx=0x7f3d4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44686ea136cdaa4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.314+0000 7f3d62a00640 1 -- 172.21.15.27:0/663635981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c035020 con 0x7f3d640c69a0 2024-03-28T19:01:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.314+0000 7f3d62a00640 1 -- 172.21.15.27:0/663635981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c031830 con 0x7f3d640c69a0 2024-03-28T19:01:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.314+0000 7f3d62a00640 1 -- 172.21.15.27:0/663635981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c031c00 con 0x7f3d640c69a0 2024-03-28T19:01:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.315+0000 7f3d69200640 1 -- 172.21.15.27:0/663635981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d640c69a0 msgr2=0x7f3d640c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.315+0000 7f3d69200640 1 --2- 172.21.15.27:0/663635981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d640c69a0 0x7f3d640c6e30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0097a0 tx=0x7f3d4c02b8d0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.315+0000 7f3d69200640 1 -- 172.21.15.27:0/663635981 shutdown_connections 2024-03-28T19:01:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.315+0000 7f3d69200640 1 --2- 172.21.15.27:0/663635981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d640c69a0 0x7f3d640c6e30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.315+0000 7f3d69200640 1 -- 172.21.15.27:0/663635981 >> 172.21.15.27:0/663635981 conn(0x7f3d6406b0c0 msgr2=0x7f3d6406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.315+0000 7f3d69200640 1 -- 172.21.15.27:0/663635981 shutdown_connections 2024-03-28T19:01:17.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.315+0000 7f3d69200640 1 -- 172.21.15.27:0/663635981 wait complete. 2024-03-28T19:01:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.317+0000 7f3d69200640 1 Processor -- start 2024-03-28T19:01:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.317+0000 7f3d69200640 1 -- start start 2024-03-28T19:01:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d69200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d64150ae0 0x7f3d640bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d69200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d64153a30 0x7f3d6414bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d69200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d641ee3f0 0x7f3d6414c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d69200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6406e7b0 con 0x7f3d64150ae0 2024-03-28T19:01:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d69200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6406e4d0 con 0x7f3d641ee3f0 2024-03-28T19:01:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d69200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640a6370 con 0x7f3d64153a30 2024-03-28T19:01:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d63400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d64150ae0 0x7f3d640bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d63400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d64150ae0 0x7f3d640bff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55900/0 (socket says 172.21.15.27:55900) 2024-03-28T19:01:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.318+0000 7f3d63400640 1 -- 172.21.15.27:0/4263236925 learned_addr learned my addr 172.21.15.27:0/4263236925 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d5be00640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d64153a30 0x7f3d6414bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d63e00640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d641ee3f0 0x7f3d6414c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d63400640 1 -- 172.21.15.27:0/4263236925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d64153a30 msgr2=0x7f3d6414bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d63400640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d64153a30 0x7f3d6414bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d63400640 1 -- 172.21.15.27:0/4263236925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d641ee3f0 msgr2=0x7f3d6414c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d63400640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d641ee3f0 0x7f3d6414c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d63400640 1 -- 172.21.15.27:0/4263236925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d640a79e0 con 0x7f3d64150ae0 2024-03-28T19:01:17.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d63400640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d64150ae0 0x7f3d640bff90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400c760 tx=0x7f3d5400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:17.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d5b400640 1 -- 172.21.15.27:0/4263236925 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d54017070 con 0x7f3d64150ae0 2024-03-28T19:01:17.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.319+0000 7f3d5b400640 1 -- 172.21.15.27:0/4263236925 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d54005ab0 con 0x7f3d64150ae0 2024-03-28T19:01:17.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.320+0000 7f3d5b400640 1 -- 172.21.15.27:0/4263236925 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d54007070 con 0x7f3d64150ae0 2024-03-28T19:01:17.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.320+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d641ee880 con 0x7f3d64150ae0 2024-03-28T19:01:17.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.320+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d641eee40 con 0x7f3d64150ae0 2024-03-28T19:01:17.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.321+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d64067dd0 con 0x7f3d64150ae0 2024-03-28T19:01:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.323+0000 7f3d5b400640 1 -- 172.21.15.27:0/4263236925 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d5401a070 con 0x7f3d64150ae0 2024-03-28T19:01:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.324+0000 7f3d5b400640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d4006bbd0 0x7f3d4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.324+0000 7f3d5b400640 1 -- 172.21.15.27:0/4263236925 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7f3d5408a590 con 0x7f3d64150ae0 2024-03-28T19:01:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.324+0000 7f3d5be00640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d4006bbd0 0x7f3d4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:17.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.325+0000 7f3d5be00640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d4006bbd0 0x7f3d4006e090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c036000 tx=0x7f3d4c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:17.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.329+0000 7f3d5b400640 1 -- 172.21.15.27:0/4263236925 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d64150ec0 con 0x7f3d64150ae0 2024-03-28T19:01:17.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.667+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d64069640 con 0x7f3d64150ae0 2024-03-28T19:01:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.669+0000 7f3d5b400640 1 -- 172.21.15.27:0/4263236925 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f3d54058940 con 0x7f3d64150ae0 2024-03-28T19:01:17.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:17.668 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d4006bbd0 msgr2=0x7f3d4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d4006bbd0 0x7f3d4006e090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c036000 tx=0x7f3d4c008030 comp rx=0 tx=0).stop 2024-03-28T19:01:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d64150ae0 msgr2=0x7f3d640bff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d64150ae0 0x7f3d640bff90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400c760 tx=0x7f3d5400cc20 comp rx=0 tx=0).stop 2024-03-28T19:01:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 shutdown_connections 2024-03-28T19:01:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d4006bbd0 0x7f3d4006e090 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d641ee3f0 0x7f3d6414c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d64153a30 0x7f3d6414bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 --2- 172.21.15.27:0/4263236925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d64150ae0 0x7f3d640bff90 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:17.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.671+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 >> 172.21.15.27:0/4263236925 conn(0x7f3d6406b0c0 msgr2=0x7f3d641547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:17.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.672+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 shutdown_connections 2024-03-28T19:01:17.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:17.672+0000 7f3d69200640 1 -- 172.21.15.27:0/4263236925 wait complete. 2024-03-28T19:01:17.693 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:01:18.648 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/13' 2024-03-28T19:01:18.648 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/138/17' 2024-03-28T19:01:18.648 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/138' 2024-03-28T19:01:18.648 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/19' 2024-03-28T19:01:18.648 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/15' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/20' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/12' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/10' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/11' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/16' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/14' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/18' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/13' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/135/17' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/135' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/19' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/15' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/20' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/12' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/10' 2024-03-28T19:01:18.649 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/11' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/16' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/14' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/18' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/13' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/117/17' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/117' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/19' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/15' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/20' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/12' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/10' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/11' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/16' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/14' 2024-03-28T19:01:18.650 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/18' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/13' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/143/17' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/143' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/19' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/15' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/20' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/12' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/10' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/11' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/16' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/14' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/18' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/13' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/180/17' 2024-03-28T19:01:18.651 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/180' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/19' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/15' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/20' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/12' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/10' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/11' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/16' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/14' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/18' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/13' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/126/17' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/126' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/19' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/15' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/20' 2024-03-28T19:01:18.652 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/12' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/10' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/11' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/16' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/14' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/18' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/13' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/161/17' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/161' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/19' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/15' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/20' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/12' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/10' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/11' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/16' 2024-03-28T19:01:18.653 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/14' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/18' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/13' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/176/17' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/176' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/19' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/15' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/20' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/12' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/10' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/11' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/16' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/14' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/18' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/13' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/111/17' 2024-03-28T19:01:18.654 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/111' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/19' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/15' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/20' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/12' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/10' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/11' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/16' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/14' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/18' 2024-03-28T19:01:18.655 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/13' 2024-03-28T19:01:19.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.926+0000 7f940fe00640 1 Processor -- start 2024-03-28T19:01:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.926+0000 7f940fe00640 1 -- start start 2024-03-28T19:01:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.928+0000 7f940fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9410158d20 0x7f94100c6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.928+0000 7f940fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9410159300 con 0x7f94100c3e40 2024-03-28T19:01:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.928+0000 7f940fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94101595e0 con 0x7f9410156180 2024-03-28T19:01:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.928+0000 7f940fe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94101598c0 con 0x7f9410158d20 2024-03-28T19:01:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.928+0000 7f940e000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f94100c3e40 0x7f94100c5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37870/0 (socket says 172.21.15.27:37870) 2024-03-28T19:01:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.928+0000 7f940e000640 1 -- 172.21.15.27:0/3608584312 learned_addr learned my addr 172.21.15.27:0/3608584312 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940f400640 1 --2- 172.21.15.27:0/3608584312 >> v2:172.21.15.155:3300/0 conn(0x7f9410158d20 0x7f94100c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885045338 0 0) 0x7f9410159300 con 0x7f94100c3e40 2024-03-28T19:01:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f4003520 con 0x7f94100c3e40 2024-03-28T19:01:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2212721666 0 0) 0x7f93f4003520 con 0x7f94100c3e40 2024-03-28T19:01:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 >> v2:172.21.15.155:3300/0 conn(0x7f9410158d20 msgr2=0x7f94100c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 --2- 172.21.15.27:0/3608584312 >> v2:172.21.15.155:3300/0 conn(0x7f9410158d20 0x7f94100c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 >> v1:172.21.15.155:6789/0 conn(0x7f9410156180 legacy=0x7f94100c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410159ba0 con 0x7f94100c3e40 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f93fc001670 con 0x7f94100c3e40 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 >> v1:172.21.15.27:6789/0 conn(0x7f94100c3e40 legacy=0x7f94100c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f93f40039b0 0x7f93f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93f4004800 0x7f93f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.929+0000 7f940d600640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f93f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.930+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94101598c0 con 0x7f93f40051c0 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.930+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f400c0c0 con 0x7f93f4004800 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.930+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f400c2c0 con 0x7f93f40039b0 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.930+0000 7f940f400640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f93f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.930+0000 7f940e000640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93f4004800 0x7f93f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.930+0000 7f940ea00640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f93f40039b0 0x7f93f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.930+0000 7f940f400640 1 -- 172.21.15.27:0/3608584312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f93f40039b0 msgr2=0x7f93f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.930+0000 7f940f400640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f93f40039b0 0x7f93f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.931+0000 7f940f400640 1 -- 172.21.15.27:0/3608584312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93f4004800 msgr2=0x7f93f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.931+0000 7f940f400640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93f4004800 0x7f93f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.931+0000 7f940f400640 1 -- 172.21.15.27:0/3608584312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410159ba0 con 0x7f93f40051c0 2024-03-28T19:01:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.932+0000 7f940f400640 1 -- 172.21.15.27:0/3608584312 reap_dead start 2024-03-28T19:01:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.932+0000 7f940f400640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f93f400b9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9408009fc0 tx=0x7f93f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed53eaa88b59d3c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.932+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408009c10 con 0x7f93f40051c0 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.932+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940802cc30 con 0x7f93f40051c0 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.932+0000 7f940d600640 1 -- 172.21.15.27:0/3608584312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408003990 con 0x7f93f40051c0 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.933+0000 7f940fe00640 1 -- 172.21.15.27:0/3608584312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 msgr2=0x7f93f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.933+0000 7f940fe00640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f93f400b9c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9408009fc0 tx=0x7f93f4005830 comp rx=0 tx=0).stop 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.933+0000 7f940fe00640 1 -- 172.21.15.27:0/3608584312 shutdown_connections 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.933+0000 7f940fe00640 1 --2- 172.21.15.27:0/3608584312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f93f400b9c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.933+0000 7f940fe00640 1 -- 172.21.15.27:0/3608584312 >> 172.21.15.27:0/3608584312 conn(0x7f94100c4500 msgr2=0x7f94100c48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.933+0000 7f940fe00640 1 -- 172.21.15.27:0/3608584312 shutdown_connections 2024-03-28T19:01:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.933+0000 7f940fe00640 1 -- 172.21.15.27:0/3608584312 wait complete. 2024-03-28T19:01:19.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.936+0000 7f940fe00640 1 Processor -- start 2024-03-28T19:01:19.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.936+0000 7f940fe00640 1 -- start start 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.938+0000 7f940fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f94101508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.938+0000 7f940fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94100be7e0 0x7f9410072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.938+0000 7f940fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94101eec50 0x7f94101eaf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.938+0000 7f940fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9410156560 con 0x7f93f40051c0 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.938+0000 7f940fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94100c5160 con 0x7f94100be7e0 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.938+0000 7f940fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94101566e0 con 0x7f94101eec50 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.938+0000 7f940ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f94101508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f94101508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55904/0 (socket says 172.21.15.27:55904) 2024-03-28T19:01:19.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940ea00640 1 -- 172.21.15.27:0/864625194 learned_addr learned my addr 172.21.15.27:0/864625194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:19.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940e000640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94100be7e0 0x7f9410072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:19.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940f400640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94101eec50 0x7f94101eaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940ea00640 1 -- 172.21.15.27:0/864625194 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94101eec50 msgr2=0x7f94101eaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940ea00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94101eec50 0x7f94101eaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940ea00640 1 -- 172.21.15.27:0/864625194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94100be7e0 msgr2=0x7f9410072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940ea00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94100be7e0 0x7f9410072cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.939+0000 7f940ea00640 1 -- 172.21.15.27:0/864625194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94100be400 con 0x7f93f40051c0 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.940+0000 7f940ea00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f94101508b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f940000e480 tx=0x7f940000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.940+0000 7f9406a00640 1 -- 172.21.15.27:0/864625194 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000052e0 con 0x7f93f40051c0 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.940+0000 7f9406a00640 1 -- 172.21.15.27:0/864625194 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9400018070 con 0x7f93f40051c0 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.940+0000 7f9406a00640 1 -- 172.21.15.27:0/864625194 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400003830 con 0x7f93f40051c0 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.940+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94100c0bb0 con 0x7f93f40051c0 2024-03-28T19:01:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.940+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94100bdad0 con 0x7f93f40051c0 2024-03-28T19:01:19.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.941+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9410069640 con 0x7f93f40051c0 2024-03-28T19:01:19.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.943+0000 7f9406a00640 1 -- 172.21.15.27:0/864625194 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f94000039d0 con 0x7f93f40051c0 2024-03-28T19:01:19.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.944+0000 7f9406a00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93f006bbd0 0x7f93f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:19.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.944+0000 7f9406a00640 1 -- 172.21.15.27:0/864625194 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7f940008aee0 con 0x7f93f40051c0 2024-03-28T19:01:19.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.944+0000 7f940e000640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93f006bbd0 0x7f93f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:19.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.946+0000 7f940e000640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93f006bbd0 0x7f93f006e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f94100bd550 tx=0x7f93fc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:19.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:19.950+0000 7f9406a00640 1 -- 172.21.15.27:0/864625194 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9400059210 con 0x7f93f40051c0 2024-03-28T19:01:19.973 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/191/17' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/191' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/19' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/15' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/20' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/12' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/10' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/11' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/16' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/14' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/18' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/13' 2024-03-28T19:01:19.974 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/184/17' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/184' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/19' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/15' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/20' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/12' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/10' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/11' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/16' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/14' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/18' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/13' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/192/17' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/192' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/19' 2024-03-28T19:01:19.975 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/15' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/20' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/12' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/10' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/11' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/16' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/14' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/18' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/13' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/103/17' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/103' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/19' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/15' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/20' 2024-03-28T19:01:19.976 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/12' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/10' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/11' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/16' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/14' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/18' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/13' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/128/17' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/128' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/19' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/15' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/20' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/12' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/10' 2024-03-28T19:01:19.977 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/11' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/16' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/14' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/18' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/13' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/144/17' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/144' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/19' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/15' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/20' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/12' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/10' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/11' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/16' 2024-03-28T19:01:19.978 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/14' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/18' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/13' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/146/17' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/146' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/19' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/15' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/20' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/12' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/10' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/11' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/16' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/14' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/18' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/13' 2024-03-28T19:01:19.979 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/141/17' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/141' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/19' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/15' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/20' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/12' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/10' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/11' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/16' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/14' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/18' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/13' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/182/17' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/182' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/19' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/15' 2024-03-28T19:01:19.980 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/20' 2024-03-28T19:01:19.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/12' 2024-03-28T19:01:19.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/10' 2024-03-28T19:01:19.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/11' 2024-03-28T19:01:19.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/16' 2024-03-28T19:01:19.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/14' 2024-03-28T19:01:19.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/18' 2024-03-28T19:01:19.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/13' 2024-03-28T19:01:19.981 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/107/17' 2024-03-28T19:01:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.275+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94100c2950 con 0x7f93f40051c0 2024-03-28T19:01:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.276+0000 7f9406a00640 1 -- 172.21.15.27:0/864625194 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f9400058b10 con 0x7f93f40051c0 2024-03-28T19:01:20.277 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.279+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93f006bbd0 msgr2=0x7f93f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.279+0000 7f940fe00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93f006bbd0 0x7f93f006e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f94100bd550 tx=0x7f93fc009040 comp rx=0 tx=0).stop 2024-03-28T19:01:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.279+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 msgr2=0x7f94101508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.279+0000 7f940fe00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f94101508b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f940000e480 tx=0x7f940000e940 comp rx=0 tx=0).stop 2024-03-28T19:01:20.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.280+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 shutdown_connections 2024-03-28T19:01:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.280+0000 7f940fe00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93f006bbd0 0x7f93f006e090 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.280+0000 7f940fe00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94101eec50 0x7f94101eaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.280+0000 7f940fe00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94100be7e0 0x7f9410072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.280+0000 7f940fe00640 1 --2- 172.21.15.27:0/864625194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93f40051c0 0x7f94101508b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.280+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 >> 172.21.15.27:0/864625194 conn(0x7f94100c4500 msgr2=0x7f94100a8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.280+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 shutdown_connections 2024-03-28T19:01:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:20.280+0000 7f940fe00640 1 -- 172.21.15.27:0/864625194 wait complete. 2024-03-28T19:01:20.302 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:01:21.232 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/107' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/19' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/15' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/20' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/12' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/10' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/11' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/16' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/14' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/18' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/13' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/178/17' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/178' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/19' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/15' 2024-03-28T19:01:21.233 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/20' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/12' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/10' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/11' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/16' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/14' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/18' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/13' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/156/17' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/156' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/19' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/15' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/20' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/12' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/10' 2024-03-28T19:01:21.234 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/11' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/16' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/14' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/18' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/13' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/131/17' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/131' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/19' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/15' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/20' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/12' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/10' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/11' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/16' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/14' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/18' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/13' 2024-03-28T19:01:21.235 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/123/17' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/123' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/19' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/15' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/20' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/12' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/10' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/11' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/16' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/14' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/18' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/13' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/160/17' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/160' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/19' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/15' 2024-03-28T19:01:21.236 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/20' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/12' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/10' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/11' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/16' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/14' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/18' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/13' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/165/17' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/165' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/19' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/15' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/20' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/12' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/10' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/11' 2024-03-28T19:01:21.237 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/16' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/14' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/18' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/13' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/101/17' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/101' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/19' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/15' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/20' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/12' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/10' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/11' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/16' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/14' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/18' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/13' 2024-03-28T19:01:21.238 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/148/17' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/148' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/19' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/15' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/20' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/12' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/10' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/11' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/16' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/14' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/18' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/13' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/168/17' 2024-03-28T19:01:21.239 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/168' 2024-03-28T19:01:22.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:22.412 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/19' 2024-03-28T19:01:22.412 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/15' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/20' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/12' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/10' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/11' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/16' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/14' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/18' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/13' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/179/17' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/179' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/19' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/15' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/20' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/12' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/10' 2024-03-28T19:01:22.413 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/11' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/16' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/14' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/18' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/13' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/132/17' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/132' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/19' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/15' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/20' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/12' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/10' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/11' 2024-03-28T19:01:22.414 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/16' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/14' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/18' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/13' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/199/17' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/199' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/19' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/15' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/20' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/12' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/10' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/11' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/16' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/14' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/18' 2024-03-28T19:01:22.415 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/13' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/163/17' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/163' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/19' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/15' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/20' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/12' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/10' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/11' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/16' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/14' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/18' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/13' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/153/17' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/153' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/19' 2024-03-28T19:01:22.416 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/15' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/20' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/12' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/10' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/11' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/16' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/14' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/18' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/13' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/122/17' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/122' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/19' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/15' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/20' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/12' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/10' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/11' 2024-03-28T19:01:22.417 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/16' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/14' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/18' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/13' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/162/17' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/162' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/19' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/15' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/20' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/12' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/10' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/11' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/16' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/14' 2024-03-28T19:01:22.418 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/18' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/13' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/197/17' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/197' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/19' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/15' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/20' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/12' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/10' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/11' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/16' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/14' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/18' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/13' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/113/17' 2024-03-28T19:01:22.419 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/113' 2024-03-28T19:01:22.420 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/19' 2024-03-28T19:01:22.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.516+0000 7fe302000640 1 Processor -- start 2024-03-28T19:01:22.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.517+0000 7fe302000640 1 -- start start 2024-03-28T19:01:22.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.518+0000 7fe302000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.518+0000 7fe302000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe2fc0c54c0 0x7fe2fc156c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.518+0000 7fe302000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc0c5c40 con 0x7fe2fc0c8530 2024-03-28T19:01:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.518+0000 7fe302000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc0c3480 con 0x7fe2fc0c54c0 2024-03-28T19:01:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.518+0000 7fe302000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0c5f20 con 0x7fe2fc0c4510 2024-03-28T19:01:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.519+0000 7fe2fbe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe2fc0c54c0 0x7fe2fc156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.519+0000 7fe2fb400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:22.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.519+0000 7fe2fbe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe2fc0c54c0 0x7fe2fc156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:60094/0 (socket says 172.21.15.27:60094) 2024-03-28T19:01:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.519+0000 7fe2fbe00640 1 -- 172.21.15.27:0/2310915400 learned_addr learned my addr 172.21.15.27:0/2310915400 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.520+0000 7fe2fb400640 1 -- 172.21.15.27:0/2310915400 >> v2:172.21.15.155:3300/0 conn(0x7fe2fc0c54c0 msgr2=0x7fe2fc156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.520+0000 7fe2fb400640 1 --2- 172.21.15.27:0/2310915400 >> v2:172.21.15.155:3300/0 conn(0x7fe2fc0c54c0 0x7fe2fc156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.520+0000 7fe2fb400640 1 -- 172.21.15.27:0/2310915400 >> v1:172.21.15.155:6789/0 conn(0x7fe2fc0c8530 legacy=0x7fe2fc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:22.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.520+0000 7fe2fb400640 1 -- 172.21.15.27:0/2310915400 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc0c6200 con 0x7fe2fc0c4510 2024-03-28T19:01:22.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.522+0000 7fe2fb400640 1 --2- 172.21.15.27:0/2310915400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc0c70d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec0097a0 tx=0x7fe2ec02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=231bc9a75e396695 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:22.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.523+0000 7fe2fa000640 1 -- 172.21.15.27:0/2310915400 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec0315a0 con 0x7fe2fc0c4510 2024-03-28T19:01:22.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.523+0000 7fe2fa000640 1 -- 172.21.15.27:0/2310915400 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec037070 con 0x7fe2fc0c4510 2024-03-28T19:01:22.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.523+0000 7fe2fa000640 1 -- 172.21.15.27:0/2310915400 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec031960 con 0x7fe2fc0c4510 2024-03-28T19:01:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.523+0000 7fe302000640 1 -- 172.21.15.27:0/2310915400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 msgr2=0x7fe2fc0c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.523+0000 7fe302000640 1 --2- 172.21.15.27:0/2310915400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc0c70d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec0097a0 tx=0x7fe2ec02b840 comp rx=0 tx=0).stop 2024-03-28T19:01:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.524+0000 7fe302000640 1 -- 172.21.15.27:0/2310915400 shutdown_connections 2024-03-28T19:01:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.524+0000 7fe302000640 1 --2- 172.21.15.27:0/2310915400 >> v2:172.21.15.155:3300/0 conn(0x7fe2fc0c54c0 0x7fe2fc156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.524+0000 7fe302000640 1 --2- 172.21.15.27:0/2310915400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc0c70d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.524+0000 7fe302000640 1 -- 172.21.15.27:0/2310915400 >> 172.21.15.27:0/2310915400 conn(0x7fe2fc066dd0 msgr2=0x7fe2fc0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.524+0000 7fe302000640 1 -- 172.21.15.27:0/2310915400 shutdown_connections 2024-03-28T19:01:22.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.524+0000 7fe302000640 1 -- 172.21.15.27:0/2310915400 wait complete. 2024-03-28T19:01:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.526+0000 7fe302000640 1 Processor -- start 2024-03-28T19:01:22.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.526+0000 7fe302000640 1 -- start start 2024-03-28T19:01:22.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe302000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc1ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:22.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe302000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2fc0c54c0 0x7fe2fc09b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe302000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2fc0c8530 0x7fe2fc1dce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe302000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0c5930 con 0x7fe2fc0c8530 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe302000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc06fca0 con 0x7fe2fc0c54c0 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe302000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0c48f0 con 0x7fe2fc0c4510 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe2fbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2fc0c8530 0x7fe2fc1dce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe2fbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2fc0c8530 0x7fe2fc1dce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55910/0 (socket says 172.21.15.27:55910) 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.527+0000 7fe2fbe00640 1 -- 172.21.15.27:0/152053754 learned_addr learned my addr 172.21.15.27:0/152053754 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.528+0000 7fe2faa00640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2fc0c54c0 0x7fe2fc09b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.528+0000 7fe2fb400640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc1ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.528+0000 7fe2fbe00640 1 -- 172.21.15.27:0/152053754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 msgr2=0x7fe2fc1ebce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.528+0000 7fe2fbe00640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc1ebce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.528+0000 7fe2fbe00640 1 -- 172.21.15.27:0/152053754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2fc0c54c0 msgr2=0x7fe2fc09b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.528+0000 7fe2fbe00640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2fc0c54c0 0x7fe2fc09b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.528+0000 7fe2fbe00640 1 -- 172.21.15.27:0/152053754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc09b8e0 con 0x7fe2fc0c8530 2024-03-28T19:01:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.528+0000 7fe2fbe00640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2fc0c8530 0x7fe2fc1dce90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400efc0 tx=0x7fe2f400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.529+0000 7fe2f3400640 1 -- 172.21.15.27:0/152053754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400cd20 con 0x7fe2fc0c8530 2024-03-28T19:01:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.529+0000 7fe2f3400640 1 -- 172.21.15.27:0/152053754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4003990 con 0x7fe2fc0c8530 2024-03-28T19:01:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.529+0000 7fe2f3400640 1 -- 172.21.15.27:0/152053754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4003d80 con 0x7fe2fc0c8530 2024-03-28T19:01:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.529+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc1ee920 con 0x7fe2fc0c8530 2024-03-28T19:01:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.529+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc1eeee0 con 0x7fe2fc0c8530 2024-03-28T19:01:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.531+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc04f400 con 0x7fe2fc0c8530 2024-03-28T19:01:22.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.532+0000 7fe2f3400640 1 -- 172.21.15.27:0/152053754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe2f4012050 con 0x7fe2fc0c8530 2024-03-28T19:01:22.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.533+0000 7fe2f3400640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2dc06bc00 0x7fe2dc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.533+0000 7fe2f3400640 1 -- 172.21.15.27:0/152053754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7fe2f4092420 con 0x7fe2fc0c8530 2024-03-28T19:01:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.534+0000 7fe2fb400640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2dc06bc00 0x7fe2dc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.535+0000 7fe2fb400640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2dc06bc00 0x7fe2dc06e0c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec036000 tx=0x7fe2ec03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.539+0000 7fe2f3400640 1 -- 172.21.15.27:0/152053754 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe2f4010040 con 0x7fe2fc0c8530 2024-03-28T19:01:22.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.912+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2fc0c79d0 con 0x7fe2fc0c8530 2024-03-28T19:01:22.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.913+0000 7fe2f3400640 1 -- 172.21.15.27:0/152053754 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7fe2f4059270 con 0x7fe2fc0c8530 2024-03-28T19:01:22.914 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:22.914 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:22.914 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2dc06bc00 msgr2=0x7fe2dc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2dc06bc00 0x7fe2dc06e0c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec036000 tx=0x7fe2ec03d040 comp rx=0 tx=0).stop 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2fc0c8530 msgr2=0x7fe2fc1dce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2fc0c8530 0x7fe2fc1dce90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400efc0 tx=0x7fe2f400c490 comp rx=0 tx=0).stop 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 shutdown_connections 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2dc06bc00 0x7fe2dc06e0c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2fc0c8530 0x7fe2fc1dce90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2fc0c54c0 0x7fe2fc09b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 --2- 172.21.15.27:0/152053754 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2fc0c4510 0x7fe2fc1ebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 >> 172.21.15.27:0/152053754 conn(0x7fe2fc066dd0 msgr2=0x7fe2fc0c2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 shutdown_connections 2024-03-28T19:01:22.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:22.916+0000 7fe302000640 1 -- 172.21.15.27:0/152053754 wait complete. 2024-03-28T19:01:22.936 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:01:23.631 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/15' 2024-03-28T19:01:23.631 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/20' 2024-03-28T19:01:23.631 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/12' 2024-03-28T19:01:23.631 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/10' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/11' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/16' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/14' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/18' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/13' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/152/17' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/152' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/19' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/15' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/20' 2024-03-28T19:01:23.632 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/12' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/10' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/11' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/16' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/14' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/18' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/13' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/121/17' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/121' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/19' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/15' 2024-03-28T19:01:23.633 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/20' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/12' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/10' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/11' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/16' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/14' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/18' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/13' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/118/17' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/118' 2024-03-28T19:01:23.634 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/19' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/15' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/20' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/12' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/10' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/11' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/16' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/14' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/18' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/13' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/196/17' 2024-03-28T19:01:23.635 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/196' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/19' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/15' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/20' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/12' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/10' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/11' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/16' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/14' 2024-03-28T19:01:23.636 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/18' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/13' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/124/17' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/124' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/19' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/15' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/20' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/12' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/10' 2024-03-28T19:01:23.637 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/11' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/16' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/14' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/18' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/13' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/188/17' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/188' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/19' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/15' 2024-03-28T19:01:23.638 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/20' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/12' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/10' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/11' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/16' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/14' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/18' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/13' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/145/17' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/145' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/19' 2024-03-28T19:01:23.639 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/15' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/20' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/12' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/10' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/11' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/16' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/14' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/18' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/13' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/130/17' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/130' 2024-03-28T19:01:23.640 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/19' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/15' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/20' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/12' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/10' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/11' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/16' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/14' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/18' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/13' 2024-03-28T19:01:23.641 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/193/17' 2024-03-28T19:01:23.642 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/193' 2024-03-28T19:01:23.642 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/19' 2024-03-28T19:01:23.642 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/15' 2024-03-28T19:01:24.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/20' 2024-03-28T19:01:24.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/12' 2024-03-28T19:01:24.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/10' 2024-03-28T19:01:24.873 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/11' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/16' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/14' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/18' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/13' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/175/17' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/175' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/19' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/15' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/20' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/12' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/10' 2024-03-28T19:01:24.874 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/11' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/16' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/14' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/18' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/13' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/181/17' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/181' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/19' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/15' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/20' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/12' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/10' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/11' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/16' 2024-03-28T19:01:24.875 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/14' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/18' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/13' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/127/17' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/127' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/19' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/15' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/20' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/12' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/10' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/11' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/16' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/14' 2024-03-28T19:01:24.876 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/18' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/13' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/108/17' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/108' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/19' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/15' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/20' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/12' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/10' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/11' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/16' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/14' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/18' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/13' 2024-03-28T19:01:24.877 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/183/17' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/183' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/19' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/15' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/20' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/12' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/10' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/11' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/16' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/14' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/18' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/13' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/177/17' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/177' 2024-03-28T19:01:24.878 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/19' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/15' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/20' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/12' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/10' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/11' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/16' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/14' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/18' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/13' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/169/17' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/169' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/19' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/15' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/20' 2024-03-28T19:01:24.879 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/12' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/10' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/11' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/16' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/14' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/18' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/13' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/154/17' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/154' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/19' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/15' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/20' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/12' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/10' 2024-03-28T19:01:24.880 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/11' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/16' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/14' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/18' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/13' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/171/17' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/171' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/19' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/15' 2024-03-28T19:01:24.881 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/20' 2024-03-28T19:01:24.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.177+0000 7f4466c00640 1 Processor -- start 2024-03-28T19:01:25.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.177+0000 7f4466c00640 1 -- start start 2024-03-28T19:01:25.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.179+0000 7f4466c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4460072fa0 0x7f44600c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.179+0000 7f4466c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44600c5290 0x7f446015fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.179+0000 7f4466c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44600c5c80 con 0x7f44600c3ac0 2024-03-28T19:01:25.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.179+0000 7f4466c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4460073580 con 0x7f44600c5290 2024-03-28T19:01:25.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.179+0000 7f4466c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44600aa630 con 0x7f4460072fa0 2024-03-28T19:01:25.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.181+0000 7f4464c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44600c5290 0x7f446015fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.181+0000 7f445fe00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f44600c3ac0 0x7f44600ab3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:58332/0 (socket says 172.21.15.27:58332) 2024-03-28T19:01:25.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.181+0000 7f445fe00640 1 -- 172.21.15.27:0/2310941210 learned_addr learned my addr 172.21.15.27:0/2310941210 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:25.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.182+0000 7f445f400640 1 --2- 172.21.15.27:0/2310941210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4460072fa0 0x7f44600c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.183+0000 7f4464c00640 1 -- 172.21.15.27:0/2310941210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4460072fa0 msgr2=0x7f44600c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:25.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.183+0000 7f4464c00640 1 --2- 172.21.15.27:0/2310941210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4460072fa0 0x7f44600c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.183+0000 7f4464c00640 1 -- 172.21.15.27:0/2310941210 >> v1:172.21.15.155:6789/0 conn(0x7f44600c3ac0 legacy=0x7f44600ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.183+0000 7f4464c00640 1 -- 172.21.15.27:0/2310941210 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44600aa910 con 0x7f44600c5290 2024-03-28T19:01:25.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.186+0000 7f4464c00640 1 --2- 172.21.15.27:0/2310941210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44600c5290 0x7f446015fac0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4458009d90 tx=0x7f445802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea7a8ee27e9418a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:25.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.186+0000 7f445ea00640 1 -- 172.21.15.27:0/2310941210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458039070 con 0x7f44600c5290 2024-03-28T19:01:25.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.186+0000 7f445ea00640 1 -- 172.21.15.27:0/2310941210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445802cbe0 con 0x7f44600c5290 2024-03-28T19:01:25.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.186+0000 7f445ea00640 1 -- 172.21.15.27:0/2310941210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458034440 con 0x7f44600c5290 2024-03-28T19:01:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.187+0000 7f4466c00640 1 -- 172.21.15.27:0/2310941210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44600c5290 msgr2=0x7f446015fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.187+0000 7f4466c00640 1 --2- 172.21.15.27:0/2310941210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44600c5290 0x7f446015fac0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4458009d90 tx=0x7f445802be30 comp rx=0 tx=0).stop 2024-03-28T19:01:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.187+0000 7f4466c00640 1 -- 172.21.15.27:0/2310941210 shutdown_connections 2024-03-28T19:01:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.187+0000 7f4466c00640 1 --2- 172.21.15.27:0/2310941210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44600c5290 0x7f446015fac0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.187+0000 7f4466c00640 1 --2- 172.21.15.27:0/2310941210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4460072fa0 0x7f44600c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.187+0000 7f4466c00640 1 -- 172.21.15.27:0/2310941210 >> 172.21.15.27:0/2310941210 conn(0x7f44600c6c00 msgr2=0x7f44600c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.188+0000 7f4466c00640 1 -- 172.21.15.27:0/2310941210 shutdown_connections 2024-03-28T19:01:25.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.188+0000 7f4466c00640 1 -- 172.21.15.27:0/2310941210 wait complete. 2024-03-28T19:01:25.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.190+0000 7f4466c00640 1 Processor -- start 2024-03-28T19:01:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.190+0000 7f4466c00640 1 -- start start 2024-03-28T19:01:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.192+0000 7f4466c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4460072fa0 0x7f44601f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.192+0000 7f4466c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44600c3ac0 0x7f44601f0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.193+0000 7f4466c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44600c5290 0x7f44601d7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.193+0000 7f4466c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44600c4e10 con 0x7f4460072fa0 2024-03-28T19:01:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.193+0000 7f4466c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44600aaed0 con 0x7f44600c3ac0 2024-03-28T19:01:25.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.193+0000 7f4466c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44600c5720 con 0x7f44600c5290 2024-03-28T19:01:25.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.193+0000 7f4464c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44600c5290 0x7f44601d7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.194+0000 7f445fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4460072fa0 0x7f44601f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.194+0000 7f4464c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44600c5290 0x7f44601d7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:59850/0 (socket says 172.21.15.27:59850) 2024-03-28T19:01:25.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.194+0000 7f4464c00640 1 -- 172.21.15.27:0/476204742 learned_addr learned my addr 172.21.15.27:0/476204742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.196+0000 7f4464c00640 1 -- 172.21.15.27:0/476204742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44600c3ac0 msgr2=0x7f44601f0000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T19:01:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.196+0000 7f4464c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44600c3ac0 0x7f44601f0000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.196+0000 7f4464c00640 1 -- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4460072fa0 msgr2=0x7f44601f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.196+0000 7f4464c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4460072fa0 0x7f44601f36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.196+0000 7f4464c00640 1 -- 172.21.15.27:0/476204742 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44601df230 con 0x7f44600c5290 2024-03-28T19:01:25.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.197+0000 7f4464c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44600c5290 0x7f44601d7730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4458009eb0 tx=0x7f4458004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.198+0000 7f4457e00640 1 -- 172.21.15.27:0/476204742 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458044020 con 0x7f44600c5290 2024-03-28T19:01:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.198+0000 7f4457e00640 1 -- 172.21.15.27:0/476204742 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445803a040 con 0x7f44600c5290 2024-03-28T19:01:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.198+0000 7f4457e00640 1 -- 172.21.15.27:0/476204742 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458039070 con 0x7f44600c5290 2024-03-28T19:01:25.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.198+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44601f3be0 con 0x7f44600c5290 2024-03-28T19:01:25.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.199+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44601dff40 con 0x7f44600c5290 2024-03-28T19:01:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.201+0000 7f4457e00640 1 -- 172.21.15.27:0/476204742 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4458047020 con 0x7f44600c5290 2024-03-28T19:01:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.202+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446004f400 con 0x7f44600c5290 2024-03-28T19:01:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.202+0000 7f4457e00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444006bc20 0x7f444006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.202+0000 7f4457e00640 1 -- 172.21.15.27:0/476204742 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7f44580ad440 con 0x7f44600c5290 2024-03-28T19:01:25.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.203+0000 7f445fe00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444006bc20 0x7f444006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.205+0000 7f445fe00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444006bc20 0x7f444006e0e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4450005880 tx=0x7f4450009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:25.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.210+0000 7f4457e00640 1 -- 172.21.15.27:0/476204742 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f44601df230 con 0x7f44600c5290 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/12' 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/10' 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/11' 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/16' 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/14' 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/18' 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/13' 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/139/17' 2024-03-28T19:01:25.229 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/139' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/19' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/15' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/20' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/12' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/10' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/11' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/16' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/14' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/18' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/13' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/102/17' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/102' 2024-03-28T19:01:25.230 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/19' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/15' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/20' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/12' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/10' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/11' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/16' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/14' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/18' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/13' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed './copyofsnap1/test1/129/17' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1/129' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1/test1' 2024-03-28T19:01:25.231 INFO:tasks.workunit.client.0.smithi027.stdout:removed directory './copyofsnap1' 2024-03-28T19:01:25.232 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-03-28T19:01:25.233 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:01:25.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T19:01:25.319 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-03-28T19:01:25.319 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-03-28T19:01:25.398 INFO:tasks.workunit.client.0.smithi027.stderr:+ set -e 2024-03-28T19:01:25.398 INFO:tasks.workunit.client.0.smithi027.stderr:+ touch foo 2024-03-28T19:01:25.404 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod +x foo 2024-03-28T19:01:25.407 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/s 2024-03-28T19:01:25.429 INFO:tasks.workunit.client.0.smithi027.stderr:+ find .snap/s/foo -executable 2024-03-28T19:01:25.429 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep foo 2024-03-28T19:01:25.440 INFO:tasks.workunit.client.0.smithi027.stdout:.snap/s/foo 2024-03-28T19:01:25.440 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir .snap/s 2024-03-28T19:01:25.456 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm foo 2024-03-28T19:01:25.463 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo OK 2024-03-28T19:01:25.464 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-03-28T19:01:25.464 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:01:25.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T19:01:25.514 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:25.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.534+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44600c5070 con 0x7f44600c5290 2024-03-28T19:01:25.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.535+0000 7f4457e00640 1 -- 172.21.15.27:0/476204742 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f445807b5b0 con 0x7f44600c5290 2024-03-28T19:01:25.536 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:25.536 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:25.536 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:25.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444006bc20 msgr2=0x7f444006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444006bc20 0x7f444006e0e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4450005880 tx=0x7f4450009040 comp rx=0 tx=0).stop 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44600c5290 msgr2=0x7f44601d7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44600c5290 0x7f44601d7730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4458009eb0 tx=0x7f4458004a40 comp rx=0 tx=0).stop 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 shutdown_connections 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444006bc20 0x7f444006e0e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44600c5290 0x7f44601d7730 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44600c3ac0 0x7f44601f0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 --2- 172.21.15.27:0/476204742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4460072fa0 0x7f44601f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 >> 172.21.15.27:0/476204742 conn(0x7f44600c6c00 msgr2=0x7f44600c6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 shutdown_connections 2024-03-28T19:01:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.539+0000 7f4466c00640 1 -- 172.21.15.27:0/476204742 wait complete. 2024-03-28T19:01:25.554 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-03-28T19:01:25.555 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-03-28T19:01:25.562 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:01:25.676 INFO:tasks.workunit.client.0.smithi027.stderr:+ set -e 2024-03-28T19:01:25.676 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir foo 2024-03-28T19:01:25.681 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo x 2024-03-28T19:01:25.684 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.727+0000 7fb006600640 1 Processor -- start 2024-03-28T19:01:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.727+0000 7fb006600640 1 -- start start 2024-03-28T19:01:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.728+0000 7fb006600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 0x7fb0000c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.728+0000 7fb006600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0001534c0 0x7fb000153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.728+0000 7fb006600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0000c3d90 con 0x7fb000150c70 2024-03-28T19:01:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.729+0000 7fb006600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0000ab4e0 con 0x7fb0001534c0 2024-03-28T19:01:25.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.729+0000 7fb006600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000ac1f0 con 0x7fb0000a9cb0 2024-03-28T19:01:25.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.729+0000 7fafffe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0001534c0 0x7fb000153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.729+0000 7fafffe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0001534c0 0x7fb000153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55944/0 (socket says 172.21.15.27:55944) 2024-03-28T19:01:25.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.729+0000 7fafffe00640 1 -- 172.21.15.27:0/2962414485 learned_addr learned my addr 172.21.15.27:0/2962414485 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.729+0000 7faffea00640 1 --2- 172.21.15.27:0/2962414485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 0x7fb0000c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.729+0000 7fafffe00640 1 -- 172.21.15.27:0/2962414485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 msgr2=0x7fb0000c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.729+0000 7fafffe00640 1 --2- 172.21.15.27:0/2962414485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 0x7fb0000c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.730+0000 7fafffe00640 1 -- 172.21.15.27:0/2962414485 >> v1:172.21.15.155:6789/0 conn(0x7fb000150c70 legacy=0x7fb0000c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.730+0000 7fafffe00640 1 -- 172.21.15.27:0/2962414485 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0000ab710 con 0x7fb0001534c0 2024-03-28T19:01:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.731+0000 7fafffe00640 1 --2- 172.21.15.27:0/2962414485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000153950 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7faff800f3a0 tx=0x7faff80302c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca2b2042821bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.732+0000 7faffe000640 1 -- 172.21.15.27:0/2962414485 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff800d2d0 con 0x7fb0001534c0 2024-03-28T19:01:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.732+0000 7faffe000640 1 -- 172.21.15.27:0/2962414485 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff800d470 con 0x7fb0001534c0 2024-03-28T19:01:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.732+0000 7faffe000640 1 -- 172.21.15.27:0/2962414485 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff800b800 con 0x7fb0001534c0 2024-03-28T19:01:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.732+0000 7fb006600640 1 -- 172.21.15.27:0/2962414485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 msgr2=0x7fb000153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.732+0000 7fb006600640 1 --2- 172.21.15.27:0/2962414485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000153950 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7faff800f3a0 tx=0x7faff80302c0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.733+0000 7fb006600640 1 -- 172.21.15.27:0/2962414485 shutdown_connections 2024-03-28T19:01:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.733+0000 7fb006600640 1 --2- 172.21.15.27:0/2962414485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000153950 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.733+0000 7fb006600640 1 --2- 172.21.15.27:0/2962414485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 0x7fb0000c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.733+0000 7fb006600640 1 -- 172.21.15.27:0/2962414485 >> 172.21.15.27:0/2962414485 conn(0x7fb00006b9d0 msgr2=0x7fb0000c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.733+0000 7fb006600640 1 -- 172.21.15.27:0/2962414485 shutdown_connections 2024-03-28T19:01:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.733+0000 7fb006600640 1 -- 172.21.15.27:0/2962414485 wait complete. 2024-03-28T19:01:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.736+0000 7fb006600640 1 Processor -- start 2024-03-28T19:01:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.736+0000 7fb006600640 1 -- start start 2024-03-28T19:01:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.737+0000 7fb006600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 0x7fb000149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.737+0000 7fb006600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb000150c70 0x7fb000149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.737+0000 7fb006600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.737+0000 7fb006600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000c1810 con 0x7fb0001534c0 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.737+0000 7fb006600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000abcd0 con 0x7fb000150c70 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.737+0000 7fb006600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000c6830 con 0x7fb0000a9cb0 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000071a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55954/0 (socket says 172.21.15.27:55954) 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafffe00640 1 -- 172.21.15.27:0/743876660 learned_addr learned my addr 172.21.15.27:0/743876660 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7faffea00640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb000150c70 0x7fb000149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafff400640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 0x7fb000149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafffe00640 1 -- 172.21.15.27:0/743876660 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 msgr2=0x7fb000149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafffe00640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 0x7fb000149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafffe00640 1 -- 172.21.15.27:0/743876660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb000150c70 msgr2=0x7fb000149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafffe00640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb000150c70 0x7fb000149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.738+0000 7fafffe00640 1 -- 172.21.15.27:0/743876660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0001e7970 con 0x7fb0001534c0 2024-03-28T19:01:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.739+0000 7fafffe00640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000071a30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faff800f400 tx=0x7faff8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.739+0000 7faff7400640 1 -- 172.21.15.27:0/743876660 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff8047020 con 0x7fb0001534c0 2024-03-28T19:01:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.739+0000 7faff7400640 1 -- 172.21.15.27:0/743876660 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff800d620 con 0x7fb0001534c0 2024-03-28T19:01:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.739+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0001e7c10 con 0x7fb0001534c0 2024-03-28T19:01:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.739+0000 7faff7400640 1 -- 172.21.15.27:0/743876660 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff8042880 con 0x7fb0001534c0 2024-03-28T19:01:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.739+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb000071230 con 0x7fb0001534c0 2024-03-28T19:01:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.740+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00004f400 con 0x7fb0001534c0 2024-03-28T19:01:25.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.742+0000 7faff7400640 1 -- 172.21.15.27:0/743876660 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faff804a020 con 0x7fb0001534c0 2024-03-28T19:01:25.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.742+0000 7faff7400640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe006bbd0 0x7fafe006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:25.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.743+0000 7faff7400640 1 -- 172.21.15.27:0/743876660 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7faff80b1de0 con 0x7fb0001534c0 2024-03-28T19:01:25.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.743+0000 7fafff400640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe006bbd0 0x7fafe006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:25.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.743+0000 7fafff400640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe006bbd0 0x7fafe006e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faff0001640 tx=0x7faff0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:25.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:25.748+0000 7faff7400640 1 -- 172.21.15.27:0/743876660 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faff8051030 con 0x7fb0001534c0 2024-03-28T19:01:26.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.004+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb000072ec0 con 0x7fb0001534c0 2024-03-28T19:01:26.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.005+0000 7faff7400640 1 -- 172.21.15.27:0/743876660 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7faff8080110 con 0x7fb0001534c0 2024-03-28T19:01:26.006 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:26.006 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:26.006 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.007+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe006bbd0 msgr2=0x7fafe006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.007+0000 7fb006600640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe006bbd0 0x7fafe006e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faff0001640 tx=0x7faff0009040 comp rx=0 tx=0).stop 2024-03-28T19:01:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 msgr2=0x7fb000071a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:26.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000071a30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faff800f400 tx=0x7faff8004a40 comp rx=0 tx=0).stop 2024-03-28T19:01:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 shutdown_connections 2024-03-28T19:01:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fafe006bbd0 0x7fafe006e090 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0001534c0 0x7fb000071a30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb000150c70 0x7fb000149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 --2- 172.21.15.27:0/743876660 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0000a9cb0 0x7fb000149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 >> 172.21.15.27:0/743876660 conn(0x7fb00006b9d0 msgr2=0x7fb0000c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 shutdown_connections 2024-03-28T19:01:26.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.008+0000 7fb006600640 1 -- 172.21.15.27:0/743876660 wait complete. 2024-03-28T19:01:26.026 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-03-28T19:01:26.027 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:01:26.027 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:01:26.065 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:01:26.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.338+0000 7fdc6b200640 1 Processor -- start 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.338+0000 7fdc6b200640 1 -- start start 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.339+0000 7fdc6b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc6c08b880 0x7fdc6c0c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.339+0000 7fdc6b200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdc6c0c4910 0x7fdc6c0a99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.339+0000 7fdc6b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c0c42a0 con 0x7fdc6c0ab350 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.339+0000 7fdc6b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c35f0 con 0x7fdc6c08b880 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.339+0000 7fdc6b200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6c06e0f0 con 0x7fdc6c0c4910 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.340+0000 7fdc69e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc6c08b880 0x7fdc6c0c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.340+0000 7fdc69e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc6c08b880 0x7fdc6c0c3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45454/0 (socket says 172.21.15.27:45454) 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.340+0000 7fdc69e00640 1 -- 172.21.15.27:0/2973986265 learned_addr learned my addr 172.21.15.27:0/2973986265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.341+0000 7fdc68a00640 1 -- 172.21.15.27:0/2973986265 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078441357 0 0) 0x7fdc6c0c42a0 con 0x7fdc6c0ab350 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.340+0000 7fdc69400640 1 --2- 172.21.15.27:0/2973986265 >> v2:172.21.15.155:3300/0 conn(0x7fdc6c0c4910 0x7fdc6c0a99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.461+0000 7fdc68a00640 1 -- 172.21.15.27:0/2973986265 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc50003500 con 0x7fdc6c0ab350 2024-03-28T19:01:26.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.462+0000 7fdc69e00640 1 -- 172.21.15.27:0/2973986265 >> v2:172.21.15.155:3300/0 conn(0x7fdc6c0c4910 msgr2=0x7fdc6c0a99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:26.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.462+0000 7fdc69e00640 1 --2- 172.21.15.27:0/2973986265 >> v2:172.21.15.155:3300/0 conn(0x7fdc6c0c4910 0x7fdc6c0a99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.462+0000 7fdc69e00640 1 -- 172.21.15.27:0/2973986265 >> v1:172.21.15.155:6789/0 conn(0x7fdc6c0ab350 legacy=0x7fdc6c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:26.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.462+0000 7fdc69e00640 1 -- 172.21.15.27:0/2973986265 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c06e3d0 con 0x7fdc6c08b880 2024-03-28T19:01:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.465+0000 7fdc69e00640 1 --2- 172.21.15.27:0/2973986265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c08b880 0x7fdc6c0c3d60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0c4530 tx=0x7fdc5c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b957ff679ef03c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.466+0000 7fdc68a00640 1 -- 172.21.15.27:0/2973986265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c039070 con 0x7fdc6c08b880 2024-03-28T19:01:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.466+0000 7fdc68a00640 1 -- 172.21.15.27:0/2973986265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c02cbf0 con 0x7fdc6c08b880 2024-03-28T19:01:26.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.466+0000 7fdc68a00640 1 -- 172.21.15.27:0/2973986265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c034440 con 0x7fdc6c08b880 2024-03-28T19:01:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.467+0000 7fdc6b200640 1 -- 172.21.15.27:0/2973986265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c08b880 msgr2=0x7fdc6c0c3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.467+0000 7fdc6b200640 1 --2- 172.21.15.27:0/2973986265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c08b880 0x7fdc6c0c3d60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0c4530 tx=0x7fdc5c02be40 comp rx=0 tx=0).stop 2024-03-28T19:01:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.467+0000 7fdc6b200640 1 -- 172.21.15.27:0/2973986265 shutdown_connections 2024-03-28T19:01:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.467+0000 7fdc6b200640 1 --2- 172.21.15.27:0/2973986265 >> v2:172.21.15.155:3300/0 conn(0x7fdc6c0c4910 0x7fdc6c0a99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.467+0000 7fdc6b200640 1 --2- 172.21.15.27:0/2973986265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c08b880 0x7fdc6c0c3d60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.467+0000 7fdc6b200640 1 -- 172.21.15.27:0/2973986265 >> 172.21.15.27:0/2973986265 conn(0x7fdc6c06cd50 msgr2=0x7fdc6c06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.468+0000 7fdc6b200640 1 -- 172.21.15.27:0/2973986265 shutdown_connections 2024-03-28T19:01:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.468+0000 7fdc6b200640 1 -- 172.21.15.27:0/2973986265 wait complete. 2024-03-28T19:01:26.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-28T19:01:26.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.470+0000 7fdc6b200640 1 Processor -- start 2024-03-28T19:01:26.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.471+0000 7fdc6b200640 1 -- start start 2024-03-28T19:01:26.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.472+0000 7fdc6b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc6c08b880 0x7fdc6c1f7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:26.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.472+0000 7fdc6b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c0ab350 0x7fdc6c0bd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.472+0000 7fdc6b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc6c0c4910 0x7fdc6c072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.472+0000 7fdc6b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0a9ee0 con 0x7fdc6c0ab350 2024-03-28T19:01:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.472+0000 7fdc6b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c3840 con 0x7fdc6c08b880 2024-03-28T19:01:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.472+0000 7fdc6b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c08bc60 con 0x7fdc6c0c4910 2024-03-28T19:01:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.472+0000 7fdc69400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c0ab350 0x7fdc6c0bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.473+0000 7fdc69400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c0ab350 0x7fdc6c0bd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45458/0 (socket says 172.21.15.27:45458) 2024-03-28T19:01:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.473+0000 7fdc69400640 1 -- 172.21.15.27:0/241760624 learned_addr learned my addr 172.21.15.27:0/241760624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.473+0000 7fdc69e00640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc6c08b880 0x7fdc6c1f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.473+0000 7fdc6a800640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc6c0c4910 0x7fdc6c072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.473+0000 7fdc69400640 1 -- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc6c0c4910 msgr2=0x7fdc6c072bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.473+0000 7fdc69400640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc6c0c4910 0x7fdc6c072bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.473+0000 7fdc69400640 1 -- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc6c08b880 msgr2=0x7fdc6c1f7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.473+0000 7fdc69400640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc6c08b880 0x7fdc6c1f7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.474+0000 7fdc69400640 1 -- 172.21.15.27:0/241760624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c165090 con 0x7fdc6c0ab350 2024-03-28T19:01:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.474+0000 7fdc69e00640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc6c08b880 0x7fdc6c1f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:01:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.474+0000 7fdc69400640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c0ab350 0x7fdc6c0bd740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdc58006630 tx=0x7fdc58012510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.475+0000 7fdc62000640 1 -- 172.21.15.27:0/241760624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58012e20 con 0x7fdc6c0ab350 2024-03-28T19:01:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.475+0000 7fdc62000640 1 -- 172.21.15.27:0/241760624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5801e070 con 0x7fdc6c0ab350 2024-03-28T19:01:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.475+0000 7fdc62000640 1 -- 172.21.15.27:0/241760624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58019630 con 0x7fdc6c0ab350 2024-03-28T19:01:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.475+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c165370 con 0x7fdc6c0ab350 2024-03-28T19:01:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.475+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c165930 con 0x7fdc6c0ab350 2024-03-28T19:01:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.476+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c067dd0 con 0x7fdc6c0ab350 2024-03-28T19:01:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.480+0000 7fdc62000640 1 -- 172.21.15.27:0/241760624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdc58016030 con 0x7fdc6c0ab350 2024-03-28T19:01:26.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.481+0000 7fdc62000640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4c06bbd0 0x7fdc4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:26.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.481+0000 7fdc62000640 1 -- 172.21.15.27:0/241760624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7fdc5808f6c0 con 0x7fdc6c0ab350 2024-03-28T19:01:26.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.482+0000 7fdc69e00640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4c06bbd0 0x7fdc4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:26.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.483+0000 7fdc69e00640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4c06bbd0 0x7fdc4c06e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c036000 tx=0x7fdc5c030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:26.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.487+0000 7fdc62000640 1 -- 172.21.15.27:0/241760624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdc58014040 con 0x7fdc6c0ab350 2024-03-28T19:01:26.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.737+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc6c069640 con 0x7fdc6c0ab350 2024-03-28T19:01:26.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.738+0000 7fdc62000640 1 -- 172.21.15.27:0/241760624 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7fdc5805d440 con 0x7fdc6c0ab350 2024-03-28T19:01:26.739 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:26.739 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4c06bbd0 msgr2=0x7fdc4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4c06bbd0 0x7fdc4c06e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c036000 tx=0x7fdc5c030000 comp rx=0 tx=0).stop 2024-03-28T19:01:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c0ab350 msgr2=0x7fdc6c0bd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c0ab350 0x7fdc6c0bd740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdc58006630 tx=0x7fdc58012510 comp rx=0 tx=0).stop 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 shutdown_connections 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4c06bbd0 0x7fdc4c06e090 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc6c0c4910 0x7fdc6c072bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc6c0ab350 0x7fdc6c0bd740 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 --2- 172.21.15.27:0/241760624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc6c08b880 0x7fdc6c1f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 >> 172.21.15.27:0/241760624 conn(0x7fdc6c06cd50 msgr2=0x7fdc6c0c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 shutdown_connections 2024-03-28T19:01:26.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:26.741+0000 7fdc6b200640 1 -- 172.21.15.27:0/241760624 wait complete. 2024-03-28T19:01:26.762 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:01:27.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:01:27.223 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:01:27.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:27.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.800+0000 7fb997000640 1 Processor -- start 2024-03-28T19:01:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.800+0000 7fb997000640 1 -- start start 2024-03-28T19:01:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.802+0000 7fb997000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb990150ae0 0x7fb9900a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.802+0000 7fb997000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb9900a61a0 0x7fb9900c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.802+0000 7fb997000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb990150ec0 con 0x7fb9900c6ee0 2024-03-28T19:01:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.802+0000 7fb997000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99006dc70 con 0x7fb9900a61a0 2024-03-28T19:01:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.802+0000 7fb997000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb990153550 con 0x7fb990150ae0 2024-03-28T19:01:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.802+0000 7fb996600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb9900c6ee0 0x7fb990153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53688/0 (socket says 172.21.15.27:53688) 2024-03-28T19:01:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.802+0000 7fb996600640 1 -- 172.21.15.27:0/2706521547 learned_addr learned my addr 172.21.15.27:0/2706521547 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb995c00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb990150ae0 0x7fb9900a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb995200640 1 --2- 172.21.15.27:0/2706521547 >> v2:172.21.15.155:3300/0 conn(0x7fb9900a61a0 0x7fb9900c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667659343 0 0) 0x7fb988001490 con 0x7fb9900c6ee0 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb978003520 con 0x7fb9900c6ee0 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786207511 0 0) 0x7fb978003520 con 0x7fb9900c6ee0 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb990150ae0 msgr2=0x7fb9900a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb990150ae0 0x7fb9900a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 >> v2:172.21.15.155:3300/0 conn(0x7fb9900a61a0 msgr2=0x7fb9900c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 --2- 172.21.15.27:0/2706521547 >> v2:172.21.15.155:3300/0 conn(0x7fb9900a61a0 0x7fb9900c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99006df10 con 0x7fb9900c6ee0 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fb988001bc0 con 0x7fb9900c6ee0 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 >> v1:172.21.15.27:6789/0 conn(0x7fb9900c6ee0 legacy=0x7fb990153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9780039b0 0x7fb978003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb990150ae0 0x7fb978004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb978004d40 0x7fb9780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99006dc70 con 0x7fb9780039b0 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb990153550 con 0x7fb978004d40 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.803+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97800b9f0 con 0x7fb990150ae0 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.804+0000 7fb995c00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9780039b0 0x7fb978003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.804+0000 7fb996600640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb978004d40 0x7fb9780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.804+0000 7fb995200640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb990150ae0 0x7fb978004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.804+0000 7fb995c00640 1 -- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb990150ae0 msgr2=0x7fb978004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.804+0000 7fb995c00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb990150ae0 0x7fb978004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.804+0000 7fb995c00640 1 -- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb978004d40 msgr2=0x7fb9780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.804+0000 7fb995c00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb978004d40 0x7fb9780051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.804+0000 7fb995c00640 1 -- 172.21.15.27:0/2706521547 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb978003520 con 0x7fb9780039b0 2024-03-28T19:01:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.805+0000 7fb995c00640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9780039b0 0x7fb978003dd0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb984009ac0 tx=0x7fb978005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1548b28ada6114aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.806+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984039070 con 0x7fb9780039b0 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.806+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98402c6b0 con 0x7fb9780039b0 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.806+0000 7fb98fe00640 1 -- 172.21.15.27:0/2706521547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984009690 con 0x7fb9780039b0 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.806+0000 7fb997000640 1 -- 172.21.15.27:0/2706521547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9780039b0 msgr2=0x7fb978003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.806+0000 7fb997000640 1 --2- 172.21.15.27:0/2706521547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9780039b0 0x7fb978003dd0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb984009ac0 tx=0x7fb978005830 comp rx=0 tx=0).stop 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.806+0000 7fb996600640 1 -- 172.21.15.27:0/2706521547 reap_dead start 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.806+0000 7fb997000640 1 -- 172.21.15.27:0/2706521547 shutdown_connections 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.806+0000 7fb997000640 1 -- 172.21.15.27:0/2706521547 >> 172.21.15.27:0/2706521547 conn(0x7fb99006b0c0 msgr2=0x7fb99006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.807+0000 7fb997000640 1 -- 172.21.15.27:0/2706521547 shutdown_connections 2024-03-28T19:01:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.807+0000 7fb997000640 1 -- 172.21.15.27:0/2706521547 wait complete. 2024-03-28T19:01:27.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.808+0000 7fb997000640 1 Processor -- start 2024-03-28T19:01:27.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.808+0000 7fb997000640 1 -- start start 2024-03-28T19:01:27.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.809+0000 7fb997000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9780039b0 0x7fb990073030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.809+0000 7fb997000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9900a7f40 0x7fb9901e9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.809+0000 7fb997000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9901ec6e0 0x7fb9901e9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.809+0000 7fb997000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99006e7b0 con 0x7fb9901ec6e0 2024-03-28T19:01:27.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.809+0000 7fb997000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99006e4d0 con 0x7fb9900a7f40 2024-03-28T19:01:27.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.809+0000 7fb997000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9901569d0 con 0x7fb9780039b0 2024-03-28T19:01:27.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb996600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9901ec6e0 0x7fb9901e9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb996600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9901ec6e0 0x7fb9901e9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45468/0 (socket says 172.21.15.27:45468) 2024-03-28T19:01:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb996600640 1 -- 172.21.15.27:0/460967967 learned_addr learned my addr 172.21.15.27:0/460967967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb995200640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9900a7f40 0x7fb9901e9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb995c00640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9780039b0 0x7fb990073030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb996600640 1 -- 172.21.15.27:0/460967967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9780039b0 msgr2=0x7fb990073030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb996600640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9780039b0 0x7fb990073030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb996600640 1 -- 172.21.15.27:0/460967967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9900a7f40 msgr2=0x7fb9901e9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb996600640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9900a7f40 0x7fb9901e9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.810+0000 7fb996600640 1 -- 172.21.15.27:0/460967967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9901ea0f0 con 0x7fb9901ec6e0 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.811+0000 7fb996600640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9901ec6e0 0x7fb9901e9bb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb988001330 tx=0x7fb98800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.811+0000 7fb98d600640 1 -- 172.21.15.27:0/460967967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988004d60 con 0x7fb9901ec6e0 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.811+0000 7fb98d600640 1 -- 172.21.15.27:0/460967967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98800bb80 con 0x7fb9901ec6e0 2024-03-28T19:01:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.811+0000 7fb98d600640 1 -- 172.21.15.27:0/460967967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988007110 con 0x7fb9901ec6e0 2024-03-28T19:01:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.811+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9901e9170 con 0x7fb9901ec6e0 2024-03-28T19:01:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.811+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9901ecec0 con 0x7fb9901ec6e0 2024-03-28T19:01:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.812+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb990067dd0 con 0x7fb9901ec6e0 2024-03-28T19:01:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.813+0000 7fb98d600640 1 -- 172.21.15.27:0/460967967 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb98801b020 con 0x7fb9901ec6e0 2024-03-28T19:01:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.814+0000 7fb98d600640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb97006bc00 0x7fb97006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:27.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.814+0000 7fb98d600640 1 -- 172.21.15.27:0/460967967 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7fb98808b450 con 0x7fb9901ec6e0 2024-03-28T19:01:27.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.814+0000 7fb995c00640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb97006bc00 0x7fb97006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:27.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.815+0000 7fb995c00640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb97006bc00 0x7fb97006e0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb984036000 tx=0x7fb984030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:27.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:27.819+0000 7fb98d600640 1 -- 172.21.15.27:0/460967967 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb9901ea0f0 con 0x7fb9901ec6e0 2024-03-28T19:01:28.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.064+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb990069640 con 0x7fb9901ec6e0 2024-03-28T19:01:28.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.065+0000 7fb98d600640 1 -- 172.21.15.27:0/460967967 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7fb990150ec0 con 0x7fb9901ec6e0 2024-03-28T19:01:28.066 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:28.066 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:28.066 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:28.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.067+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb97006bc00 msgr2=0x7fb97006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.067+0000 7fb997000640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb97006bc00 0x7fb97006e0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb984036000 tx=0x7fb984030000 comp rx=0 tx=0).stop 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.067+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9901ec6e0 msgr2=0x7fb9901e9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.067+0000 7fb997000640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9901ec6e0 0x7fb9901e9bb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb988001330 tx=0x7fb98800b4a0 comp rx=0 tx=0).stop 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.068+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 shutdown_connections 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.068+0000 7fb997000640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb97006bc00 0x7fb97006e0c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.068+0000 7fb997000640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9901ec6e0 0x7fb9901e9bb0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.068+0000 7fb997000640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9900a7f40 0x7fb9901e9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.068+0000 7fb997000640 1 --2- 172.21.15.27:0/460967967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9780039b0 0x7fb990073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.068+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 >> 172.21.15.27:0/460967967 conn(0x7fb99006b0c0 msgr2=0x7fb99014ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.068+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 shutdown_connections 2024-03-28T19:01:28.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.068+0000 7fb997000640 1 -- 172.21.15.27:0/460967967 wait complete. 2024-03-28T19:01:28.087 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:01:28.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:28.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.998+0000 7fb80fe00640 1 Processor -- start 2024-03-28T19:01:28.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.998+0000 7fb80fe00640 1 -- start start 2024-03-28T19:01:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.999+0000 7fb80fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8100727e0 0x7fb8100c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.999+0000 7fb80fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb810155fe0 con 0x7fb8100c59d0 2024-03-28T19:01:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.999+0000 7fb80fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8101562c0 con 0x7fb810155600 2024-03-28T19:01:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:28.999+0000 7fb80fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8101565a0 con 0x7fb8100727e0 2024-03-28T19:01:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.000+0000 7fb80ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb8100c59d0 0x7fb810159520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53696/0 (socket says 172.21.15.27:53696) 2024-03-28T19:01:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.000+0000 7fb80ea00640 1 -- 172.21.15.27:0/4154436663 learned_addr learned my addr 172.21.15.27:0/4154436663 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:29.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.000+0000 7fb80e000640 1 --2- 172.21.15.27:0/4154436663 >> v2:172.21.15.27:3300/0 conn(0x7fb8100727e0 0x7fb8100c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:29.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.000+0000 7fb80e000640 1 -- 172.21.15.27:0/4154436663 >> v1:172.21.15.155:6789/0 conn(0x7fb810155600 legacy=0x7fb81014f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:29.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.000+0000 7fb80e000640 1 -- 172.21.15.27:0/4154436663 >> v1:172.21.15.27:6789/0 conn(0x7fb8100c59d0 legacy=0x7fb810159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:29.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.000+0000 7fb80e000640 1 -- 172.21.15.27:0/4154436663 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb810158d20 con 0x7fb8100727e0 2024-03-28T19:01:29.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.000+0000 7fb80d600640 1 -- 172.21.15.27:0/4154436663 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099657541 0 0) 0x7fb810155fe0 con 0x7fb8100c59d0 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.001+0000 7fb80e000640 1 --2- 172.21.15.27:0/4154436663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c8f00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80097a0 tx=0x7fb7f802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29abd58bf9f30c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80d600640 1 -- 172.21.15.27:0/4154436663 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8038070 con 0x7fb8100727e0 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80d600640 1 -- 172.21.15.27:0/4154436663 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f8031700 con 0x7fb8100727e0 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80d600640 1 -- 172.21.15.27:0/4154436663 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8031ad0 con 0x7fb8100727e0 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80fe00640 1 -- 172.21.15.27:0/4154436663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 msgr2=0x7fb8100c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80fe00640 1 --2- 172.21.15.27:0/4154436663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c8f00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80097a0 tx=0x7fb7f802b7a0 comp rx=0 tx=0).stop 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80fe00640 1 -- 172.21.15.27:0/4154436663 shutdown_connections 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80fe00640 1 --2- 172.21.15.27:0/4154436663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c8f00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80fe00640 1 -- 172.21.15.27:0/4154436663 >> 172.21.15.27:0/4154436663 conn(0x7fb8100ac910 msgr2=0x7fb8100accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80fe00640 1 -- 172.21.15.27:0/4154436663 shutdown_connections 2024-03-28T19:01:29.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.002+0000 7fb80fe00640 1 -- 172.21.15.27:0/4154436663 wait complete. 2024-03-28T19:01:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.004+0000 7fb80fe00640 1 Processor -- start 2024-03-28T19:01:29.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.004+0000 7fb80fe00640 1 -- start start 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.005+0000 7fb80fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.005+0000 7fb80fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8100c59d0 0x7fb810158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.005+0000 7fb80fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb810155600 0x7fb810143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.005+0000 7fb80fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100c5db0 con 0x7fb8100727e0 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.005+0000 7fb80fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb810159090 con 0x7fb810155600 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.005+0000 7fb80fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100c5f30 con 0x7fb8100c59d0 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.005+0000 7fb80ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45484/0 (socket says 172.21.15.27:45484) 2024-03-28T19:01:29.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80ea00640 1 -- 172.21.15.27:0/2138327690 learned_addr learned my addr 172.21.15.27:0/2138327690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:29.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80ea00640 1 -- 172.21.15.27:0/2138327690 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8100c59d0 msgr2=0x7fb810158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80ea00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8100c59d0 0x7fb810158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80ea00640 1 -- 172.21.15.27:0/2138327690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb810155600 msgr2=0x7fb810143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80f400640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb810155600 0x7fb810143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80ea00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb810155600 0x7fb810143c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80ea00640 1 -- 172.21.15.27:0/2138327690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100c4f00 con 0x7fb8100727e0 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.006+0000 7fb80ea00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c28c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb80000c890 tx=0x7fb80000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.007+0000 7fb806a00640 1 -- 172.21.15.27:0/2138327690 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800018070 con 0x7fb8100727e0 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.007+0000 7fb806a00640 1 -- 172.21.15.27:0/2138327690 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80000a5e0 con 0x7fb8100727e0 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.007+0000 7fb806a00640 1 -- 172.21.15.27:0/2138327690 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80000a9d0 con 0x7fb8100727e0 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.007+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8100c21f0 con 0x7fb8100727e0 2024-03-28T19:01:29.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.007+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8100ac160 con 0x7fb8100727e0 2024-03-28T19:01:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.008+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb810067dd0 con 0x7fb8100727e0 2024-03-28T19:01:29.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.009+0000 7fb806a00640 1 -- 172.21.15.27:0/2138327690 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb80001b070 con 0x7fb8100727e0 2024-03-28T19:01:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.010+0000 7fb806a00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ec06bbd0 0x7fb7ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.010+0000 7fb806a00640 1 -- 172.21.15.27:0/2138327690 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7fb80008a8e0 con 0x7fb8100727e0 2024-03-28T19:01:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.010+0000 7fb80e000640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ec06bbd0 0x7fb7ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:29.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.011+0000 7fb80e000640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ec06bbd0 0x7fb7ec06e090 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035000 tx=0x7fb7f8030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:29.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.015+0000 7fb806a00640 1 -- 172.21.15.27:0/2138327690 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb800058c10 con 0x7fb8100727e0 2024-03-28T19:01:29.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.261+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb810069640 con 0x7fb8100727e0 2024-03-28T19:01:29.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.262+0000 7fb806a00640 1 -- 172.21.15.27:0/2138327690 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7fb800058510 con 0x7fb8100727e0 2024-03-28T19:01:29.263 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:29.263 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:29.263 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:29.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ec06bbd0 msgr2=0x7fb7ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ec06bbd0 0x7fb7ec06e090 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035000 tx=0x7fb7f8030000 comp rx=0 tx=0).stop 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 msgr2=0x7fb8100c28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c28c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb80000c890 tx=0x7fb80000cd50 comp rx=0 tx=0).stop 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 shutdown_connections 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ec06bbd0 0x7fb7ec06e090 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb810155600 0x7fb810143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8100c59d0 0x7fb810158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 --2- 172.21.15.27:0/2138327690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8100727e0 0x7fb8100c28c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 >> 172.21.15.27:0/2138327690 conn(0x7fb8100ac910 msgr2=0x7fb81014cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:29.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 shutdown_connections 2024-03-28T19:01:29.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:29.266+0000 7fb80fe00640 1 -- 172.21.15.27:0/2138327690 wait complete. 2024-03-28T19:01:29.288 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:01:29.934 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir foo/.snap/ss 2024-03-28T19:01:30.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:30.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.326+0000 7fa8cb000640 1 Processor -- start 2024-03-28T19:01:30.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.326+0000 7fa8cb000640 1 -- start start 2024-03-28T19:01:30.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.327+0000 7fa8cb000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa8c4154140 0x7fa8c40c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:30.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.327+0000 7fa8cb000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c4150ec0 con 0x7fa8c4150ae0 2024-03-28T19:01:30.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.327+0000 7fa8cb000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c406dc70 con 0x7fa8c40c69a0 2024-03-28T19:01:30.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.327+0000 7fa8cb000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c4153550 con 0x7fa8c4154140 2024-03-28T19:01:30.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.328+0000 7fa8c9c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa8c4150ae0 0x7fa8c40a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53712/0 (socket says 172.21.15.27:53712) 2024-03-28T19:01:30.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.328+0000 7fa8c9c00640 1 -- 172.21.15.27:0/4199955926 learned_addr learned my addr 172.21.15.27:0/4199955926 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.328+0000 7fa8ca600640 1 --2- 172.21.15.27:0/4199955926 >> v2:172.21.15.155:3300/0 conn(0x7fa8c4154140 0x7fa8c40c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.328+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929792580 0 0) 0x7fa8b8001110 con 0x7fa8c4150ae0 2024-03-28T19:01:30.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.328+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ac003520 con 0x7fa8c4150ae0 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820194929 0 0) 0x7fa8ac003520 con 0x7fa8c4150ae0 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 >> v2:172.21.15.155:3300/0 conn(0x7fa8c4154140 msgr2=0x7fa8c40c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 --2- 172.21.15.27:0/4199955926 >> v2:172.21.15.155:3300/0 conn(0x7fa8c4154140 0x7fa8c40c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 >> v1:172.21.15.155:6789/0 conn(0x7fa8c40c69a0 legacy=0x7fa8c4153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c406df10 con 0x7fa8c4150ae0 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fa8b8001840 con 0x7fa8c4150ae0 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 >> v1:172.21.15.27:6789/0 conn(0x7fa8c4150ae0 legacy=0x7fa8c40a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8ac003a00 0x7fa8ac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac004800 0x7fa8ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac005210 0x7fa8ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c4153550 con 0x7fa8ac005210 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac00c160 con 0x7fa8ac004800 2024-03-28T19:01:30.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac00c360 con 0x7fa8ac003a00 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.329+0000 7fa8ca600640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac005210 0x7fa8ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.330+0000 7fa8c9200640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac004800 0x7fa8ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.330+0000 7fa8c9c00640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8ac003a00 0x7fa8ac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.330+0000 7fa8ca600640 1 -- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8ac003a00 msgr2=0x7fa8ac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.330+0000 7fa8ca600640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8ac003a00 0x7fa8ac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.330+0000 7fa8ca600640 1 -- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac004800 msgr2=0x7fa8ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.330+0000 7fa8ca600640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac004800 0x7fa8ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.330+0000 7fa8ca600640 1 -- 172.21.15.27:0/4199955926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac003520 con 0x7fa8ac005210 2024-03-28T19:01:30.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.330+0000 7fa8c9c00640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8ac003a00 0x7fa8ac003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.331+0000 7fa8ca600640 1 -- 172.21.15.27:0/4199955926 reap_dead start 2024-03-28T19:01:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.331+0000 7fa8ca600640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac005210 0x7fa8ac00b9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc009f90 tx=0x7fa8ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dfa68214dab9e96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.331+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc009c10 con 0x7fa8ac005210 2024-03-28T19:01:30.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.331+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc02cc20 con 0x7fa8ac005210 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.331+0000 7fa8c3e00640 1 -- 172.21.15.27:0/4199955926 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc003990 con 0x7fa8ac005210 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.332+0000 7fa8cb000640 1 -- 172.21.15.27:0/4199955926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac005210 msgr2=0x7fa8ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.332+0000 7fa8cb000640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac005210 0x7fa8ac00b9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc009f90 tx=0x7fa8ac005830 comp rx=0 tx=0).stop 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.332+0000 7fa8cb000640 1 -- 172.21.15.27:0/4199955926 shutdown_connections 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.332+0000 7fa8cb000640 1 --2- 172.21.15.27:0/4199955926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac005210 0x7fa8ac00b9c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.332+0000 7fa8cb000640 1 -- 172.21.15.27:0/4199955926 >> 172.21.15.27:0/4199955926 conn(0x7fa8c406b0c0 msgr2=0x7fa8c406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.332+0000 7fa8cb000640 1 -- 172.21.15.27:0/4199955926 shutdown_connections 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.332+0000 7fa8cb000640 1 -- 172.21.15.27:0/4199955926 wait complete. 2024-03-28T19:01:30.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.334+0000 7fa8cb000640 1 Processor -- start 2024-03-28T19:01:30.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.334+0000 7fa8cb000640 1 -- start start 2024-03-28T19:01:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8cb000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac005210 0x7fa8c41ef1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8cb000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8c40c2a20 0x7fa8c414e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8cb000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c414eac0 0x7fa8c40717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8cb000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c406e7b0 con 0x7fa8c414eac0 2024-03-28T19:01:30.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8cb000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c406e4d0 con 0x7fa8ac005210 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8cb000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c41569d0 con 0x7fa8c40c2a20 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8ca600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c414eac0 0x7fa8c40717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8ca600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c414eac0 0x7fa8c40717c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45514/0 (socket says 172.21.15.27:45514) 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.335+0000 7fa8ca600640 1 -- 172.21.15.27:0/2616748268 learned_addr learned my addr 172.21.15.27:0/2616748268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8c9c00640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac005210 0x7fa8c41ef1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8c9200640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8c40c2a20 0x7fa8c414e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8ca600640 1 -- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8c40c2a20 msgr2=0x7fa8c414e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8ca600640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8c40c2a20 0x7fa8c414e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8ca600640 1 -- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac005210 msgr2=0x7fa8c41ef1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8ca600640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac005210 0x7fa8c41ef1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8ca600640 1 -- 172.21.15.27:0/2616748268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c4072a70 con 0x7fa8c414eac0 2024-03-28T19:01:30.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8c9c00640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac005210 0x7fa8c41ef1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:30.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.336+0000 7fa8ca600640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c414eac0 0x7fa8c40717c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00e200 tx=0x7fa8bc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:30.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.337+0000 7fa8c1600640 1 -- 172.21.15.27:0/2616748268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc009c10 con 0x7fa8c414eac0 2024-03-28T19:01:30.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.337+0000 7fa8c1600640 1 -- 172.21.15.27:0/2616748268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc044070 con 0x7fa8c414eac0 2024-03-28T19:01:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.337+0000 7fa8c1600640 1 -- 172.21.15.27:0/2616748268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc003dc0 con 0x7fa8c414eac0 2024-03-28T19:01:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.337+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c4072d10 con 0x7fa8c414eac0 2024-03-28T19:01:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.337+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c40732d0 con 0x7fa8c414eac0 2024-03-28T19:01:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.338+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c4067dd0 con 0x7fa8c414eac0 2024-03-28T19:01:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.339+0000 7fa8c1600640 1 -- 172.21.15.27:0/2616748268 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa8bc005d10 con 0x7fa8c414eac0 2024-03-28T19:01:30.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.340+0000 7fa8c1600640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8a406bc00 0x7fa8a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:30.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.340+0000 7fa8c1600640 1 -- 172.21.15.27:0/2616748268 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7fa8bc0ad910 con 0x7fa8c414eac0 2024-03-28T19:01:30.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.340+0000 7fa8c9c00640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8a406bc00 0x7fa8a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:30.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.341+0000 7fa8c9c00640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8a406bc00 0x7fa8a406e0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8001eb0 tx=0x7fa8b8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.345+0000 7fa8c1600640 1 -- 172.21.15.27:0/2616748268 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa8bc07bd60 con 0x7fa8c414eac0 2024-03-28T19:01:30.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.593+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8c4152b50 con 0x7fa8c414eac0 2024-03-28T19:01:30.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.594+0000 7fa8c1600640 1 -- 172.21.15.27:0/2616748268 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7fa8c4152b50 con 0x7fa8c414eac0 2024-03-28T19:01:30.593 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:30.593 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:30.594 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.596+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8a406bc00 msgr2=0x7fa8a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.596+0000 7fa8cb000640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8a406bc00 0x7fa8a406e0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8001eb0 tx=0x7fa8b8009040 comp rx=0 tx=0).stop 2024-03-28T19:01:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.596+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c414eac0 msgr2=0x7fa8c40717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.596+0000 7fa8cb000640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c414eac0 0x7fa8c40717c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00e200 tx=0x7fa8bc009750 comp rx=0 tx=0).stop 2024-03-28T19:01:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.597+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 shutdown_connections 2024-03-28T19:01:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.597+0000 7fa8cb000640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8a406bc00 0x7fa8a406e0c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.597+0000 7fa8cb000640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c414eac0 0x7fa8c40717c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.597+0000 7fa8cb000640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8c40c2a20 0x7fa8c414e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.597+0000 7fa8cb000640 1 --2- 172.21.15.27:0/2616748268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8ac005210 0x7fa8c41ef1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.597+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 >> 172.21.15.27:0/2616748268 conn(0x7fa8c406b0c0 msgr2=0x7fa8c406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.597+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 shutdown_connections 2024-03-28T19:01:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:30.597+0000 7fa8cb000640 1 -- 172.21.15.27:0/2616748268 wait complete. 2024-03-28T19:01:30.618 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:01:31.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.534+0000 7f7cd0a00640 1 Processor -- start 2024-03-28T19:01:31.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.534+0000 7f7cd0a00640 1 -- start start 2024-03-28T19:01:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.535+0000 7f7cd0a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ccc150ae0 0x7f7ccc0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.535+0000 7f7cd0a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ccc0a61a0 0x7f7ccc0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.535+0000 7f7cd0a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc150ec0 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.535+0000 7f7cd0a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc06dc70 con 0x7f7ccc0a61a0 2024-03-28T19:01:31.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.535+0000 7f7cd0a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc153550 con 0x7f7ccc150ae0 2024-03-28T19:01:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.536+0000 7f7ccaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ccc0a61a0 0x7f7ccc0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.536+0000 7f7ccaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ccc0a61a0 0x7f7ccc0c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45522/0 (socket says 172.21.15.27:45522) 2024-03-28T19:01:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.536+0000 7f7ccaa00640 1 -- 172.21.15.27:0/3463894153 learned_addr learned my addr 172.21.15.27:0/3463894153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:31.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.536+0000 7f7ccb400640 1 --2- 172.21.15.27:0/3463894153 >> v2:172.21.15.155:3300/0 conn(0x7f7ccc150ae0 0x7f7ccc0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.536+0000 7f7ccaa00640 1 -- 172.21.15.27:0/3463894153 >> v2:172.21.15.155:3300/0 conn(0x7f7ccc150ae0 msgr2=0x7f7ccc0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.536+0000 7f7ccaa00640 1 --2- 172.21.15.27:0/3463894153 >> v2:172.21.15.155:3300/0 conn(0x7f7ccc150ae0 0x7f7ccc0a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.537+0000 7f7ccaa00640 1 -- 172.21.15.27:0/3463894153 >> v1:172.21.15.155:6789/0 conn(0x7f7ccc0c6ee0 legacy=0x7f7ccc153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:31.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.537+0000 7f7ccaa00640 1 -- 172.21.15.27:0/3463894153 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc06df10 con 0x7f7ccc0a61a0 2024-03-28T19:01:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.538+0000 7f7ccaa00640 1 --2- 172.21.15.27:0/3463894153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0a61a0 0x7f7ccc0c69a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0a9760 tx=0x7f7cb802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644d2766f978c7e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.538+0000 7f7cca000640 1 -- 172.21.15.27:0/3463894153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb802c9f0 con 0x7f7ccc0a61a0 2024-03-28T19:01:31.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.538+0000 7f7cca000640 1 -- 172.21.15.27:0/3463894153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb802cb90 con 0x7f7ccc0a61a0 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.538+0000 7f7cca000640 1 -- 172.21.15.27:0/3463894153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80343f0 con 0x7f7ccc0a61a0 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.539+0000 7f7cd0a00640 1 -- 172.21.15.27:0/3463894153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0a61a0 msgr2=0x7f7ccc0c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.539+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/3463894153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0a61a0 0x7f7ccc0c69a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0a9760 tx=0x7f7cb802be50 comp rx=0 tx=0).stop 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.539+0000 7f7cd0a00640 1 -- 172.21.15.27:0/3463894153 shutdown_connections 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.539+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/3463894153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0a61a0 0x7f7ccc0c69a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.539+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/3463894153 >> v2:172.21.15.155:3300/0 conn(0x7f7ccc150ae0 0x7f7ccc0a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.539+0000 7f7cd0a00640 1 -- 172.21.15.27:0/3463894153 >> 172.21.15.27:0/3463894153 conn(0x7f7ccc06b0c0 msgr2=0x7f7ccc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.539+0000 7f7cd0a00640 1 -- 172.21.15.27:0/3463894153 shutdown_connections 2024-03-28T19:01:31.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.539+0000 7f7cd0a00640 1 -- 172.21.15.27:0/3463894153 wait complete. 2024-03-28T19:01:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.541+0000 7f7cd0a00640 1 Processor -- start 2024-03-28T19:01:31.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.541+0000 7f7cd0a00640 1 -- start start 2024-03-28T19:01:31.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.542+0000 7f7cd0a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ccc0a61a0 0x7f7ccc1eeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.542+0000 7f7cd0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0c6ee0 0x7f7ccc0bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.542+0000 7f7cd0a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc150ae0 0x7f7ccc148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.542+0000 7f7cd0a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc06e7b0 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.542+0000 7f7cd0a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc06e4d0 con 0x7f7ccc150ae0 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.542+0000 7f7cd0a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1569d0 con 0x7f7ccc0a61a0 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0c6ee0 0x7f7ccc0bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0c6ee0 0x7f7ccc0bff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45528/0 (socket says 172.21.15.27:45528) 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccaa00640 1 -- 172.21.15.27:0/2658197069 learned_addr learned my addr 172.21.15.27:0/2658197069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccaa00640 1 -- 172.21.15.27:0/2658197069 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ccc0a61a0 msgr2=0x7f7ccc1eeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccaa00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ccc0a61a0 0x7f7ccc1eeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccbe00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc150ae0 0x7f7ccc148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:31.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccaa00640 1 -- 172.21.15.27:0/2658197069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc150ae0 msgr2=0x7f7ccc148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:31.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccaa00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc150ae0 0x7f7ccc148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccaa00640 1 -- 172.21.15.27:0/2658197069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc070810 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.543+0000 7f7ccbe00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc150ae0 0x7f7ccc148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:31.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.544+0000 7f7ccaa00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0c6ee0 0x7f7ccc0bff90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8000f80 tx=0x7f7cb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:31.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.544+0000 7f7cc3400640 1 -- 172.21.15.27:0/2658197069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8043020 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.544+0000 7f7cc3400640 1 -- 172.21.15.27:0/2658197069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb803e450 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.544+0000 7f7cc3400640 1 -- 172.21.15.27:0/2658197069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8009510 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.544+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc070af0 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.544+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc153140 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.546+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc067dd0 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.547+0000 7f7cc3400640 1 -- 172.21.15.27:0/2658197069 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7cb8046020 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.547+0000 7f7cc3400640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7cac06bbd0 0x7f7cac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.548+0000 7f7cc3400640 1 -- 172.21.15.27:0/2658197069 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7f7cb80ada20 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.548+0000 7f7ccb400640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7cac06bbd0 0x7f7cac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:31.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.549+0000 7f7ccb400640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7cac06bbd0 0x7f7cac06e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc006fd0 tx=0x7f7cbc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:31.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.553+0000 7f7cc3400640 1 -- 172.21.15.27:0/2658197069 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7cb804d030 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.801+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ccc069640 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.802+0000 7f7cc3400640 1 -- 172.21.15.27:0/2658197069 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f7ccc150ec0 con 0x7f7ccc0c6ee0 2024-03-28T19:01:31.801 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:31.802 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:31.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7cac06bbd0 msgr2=0x7f7cac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7cac06bbd0 0x7f7cac06e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc006fd0 tx=0x7f7cbc008040 comp rx=0 tx=0).stop 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0c6ee0 msgr2=0x7f7ccc0bff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0c6ee0 0x7f7ccc0bff90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8000f80 tx=0x7f7cb8004a40 comp rx=0 tx=0).stop 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 shutdown_connections 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7cac06bbd0 0x7f7cac06e090 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc150ae0 0x7f7ccc148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ccc0c6ee0 0x7f7ccc0bff90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 --2- 172.21.15.27:0/2658197069 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ccc0a61a0 0x7f7ccc1eeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 >> 172.21.15.27:0/2658197069 conn(0x7f7ccc06b0c0 msgr2=0x7f7ccc06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 shutdown_connections 2024-03-28T19:01:31.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:31.805+0000 7f7cd0a00640 1 -- 172.21.15.27:0/2658197069 wait complete. 2024-03-28T19:01:31.825 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:01:32.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.854+0000 7f2c5d200640 1 Processor -- start 2024-03-28T19:01:32.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.854+0000 7f2c5d200640 1 -- start start 2024-03-28T19:01:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.855+0000 7f2c5d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c58150ae0 0x7f2c580a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.855+0000 7f2c5d200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c580a61a0 0x7f2c580c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.855+0000 7f2c5d200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2c580c6ee0 0x7f2c58153a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:32.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.855+0000 7f2c5d200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c58150ec0 con 0x7f2c58150ae0 2024-03-28T19:01:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.855+0000 7f2c5d200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5806dc70 con 0x7f2c580c6ee0 2024-03-28T19:01:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.855+0000 7f2c5d200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58153550 con 0x7f2c580a61a0 2024-03-28T19:01:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.856+0000 7f2c57400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c58150ae0 0x7f2c580a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.856+0000 7f2c57400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c58150ae0 0x7f2c580a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45532/0 (socket says 172.21.15.27:45532) 2024-03-28T19:01:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.856+0000 7f2c57400640 1 -- 172.21.15.27:0/1106466379 learned_addr learned my addr 172.21.15.27:0/1106466379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.856+0000 7f2c57e00640 1 --2- 172.21.15.27:0/1106466379 >> v2:172.21.15.155:3300/0 conn(0x7f2c580c6ee0 0x7f2c58153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:32.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.856+0000 7f2c4fe00640 1 --2- 172.21.15.27:0/1106466379 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c580a61a0 0x7f2c580c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:32.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.857+0000 7f2c57400640 1 -- 172.21.15.27:0/1106466379 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c580a61a0 msgr2=0x7f2c580c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:32.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.857+0000 7f2c57400640 1 --2- 172.21.15.27:0/1106466379 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c580a61a0 0x7f2c580c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:32.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.857+0000 7f2c57400640 1 -- 172.21.15.27:0/1106466379 >> v2:172.21.15.155:3300/0 conn(0x7f2c580c6ee0 msgr2=0x7f2c58153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:32.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.857+0000 7f2c57400640 1 --2- 172.21.15.27:0/1106466379 >> v2:172.21.15.155:3300/0 conn(0x7f2c580c6ee0 0x7f2c58153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:32.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.857+0000 7f2c57400640 1 -- 172.21.15.27:0/1106466379 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5806df10 con 0x7f2c58150ae0 2024-03-28T19:01:32.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.858+0000 7f2c57400640 1 --2- 172.21.15.27:0/1106466379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c58150ae0 0x7f2c580a5c60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2c5806d840 tx=0x7f2c4802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475d556163af316 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.859+0000 7f2c56a00640 1 -- 172.21.15.27:0/1106466379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4802e070 con 0x7f2c58150ae0 2024-03-28T19:01:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.859+0000 7f2c56a00640 1 -- 172.21.15.27:0/1106466379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c48037070 con 0x7f2c58150ae0 2024-03-28T19:01:32.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.859+0000 7f2c56a00640 1 -- 172.21.15.27:0/1106466379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48031a00 con 0x7f2c58150ae0 2024-03-28T19:01:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.859+0000 7f2c5d200640 1 -- 172.21.15.27:0/1106466379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c58150ae0 msgr2=0x7f2c580a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.859+0000 7f2c5d200640 1 --2- 172.21.15.27:0/1106466379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c58150ae0 0x7f2c580a5c60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2c5806d840 tx=0x7f2c4802b880 comp rx=0 tx=0).stop 2024-03-28T19:01:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.860+0000 7f2c5d200640 1 -- 172.21.15.27:0/1106466379 shutdown_connections 2024-03-28T19:01:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.860+0000 7f2c5d200640 1 --2- 172.21.15.27:0/1106466379 >> v2:172.21.15.155:3300/0 conn(0x7f2c580c6ee0 0x7f2c58153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.860+0000 7f2c5d200640 1 --2- 172.21.15.27:0/1106466379 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c580a61a0 0x7f2c580c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.860+0000 7f2c5d200640 1 --2- 172.21.15.27:0/1106466379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c58150ae0 0x7f2c580a5c60 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.860+0000 7f2c5d200640 1 -- 172.21.15.27:0/1106466379 >> 172.21.15.27:0/1106466379 conn(0x7f2c5806b0c0 msgr2=0x7f2c5806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:32.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.860+0000 7f2c5d200640 1 -- 172.21.15.27:0/1106466379 shutdown_connections 2024-03-28T19:01:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.860+0000 7f2c5d200640 1 -- 172.21.15.27:0/1106466379 wait complete. 2024-03-28T19:01:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.861+0000 7f2c5d200640 1 Processor -- start 2024-03-28T19:01:32.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.861+0000 7f2c5d200640 1 -- start start 2024-03-28T19:01:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.862+0000 7f2c5d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c580a61a0 0x7f2c58154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.862+0000 7f2c5d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c580c6ee0 0x7f2c581e6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.862+0000 7f2c5d200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c58150ae0 0x7f2c581e73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.862+0000 7f2c5d200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5806e7b0 con 0x7f2c580c6ee0 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.862+0000 7f2c5d200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5806e4d0 con 0x7f2c580a61a0 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.862+0000 7f2c5d200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c581569d0 con 0x7f2c58150ae0 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c4fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c580c6ee0 0x7f2c581e6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c4fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c580c6ee0 0x7f2c581e6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45534/0 (socket says 172.21.15.27:45534) 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c4fe00640 1 -- 172.21.15.27:0/4250723898 learned_addr learned my addr 172.21.15.27:0/4250723898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c57400640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c580a61a0 0x7f2c58154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c57e00640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c58150ae0 0x7f2c581e73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c4fe00640 1 -- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c58150ae0 msgr2=0x7f2c581e73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c4fe00640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c58150ae0 0x7f2c581e73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c4fe00640 1 -- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c580a61a0 msgr2=0x7f2c58154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c4fe00640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c580a61a0 0x7f2c58154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c4fe00640 1 -- 172.21.15.27:0/4250723898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c580ab830 con 0x7f2c580c6ee0 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.863+0000 7f2c57400640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c580a61a0 0x7f2c58154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.864+0000 7f2c4fe00640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c580c6ee0 0x7f2c581e6e90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000e870 tx=0x7f2c4000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.864+0000 7f2c4f400640 1 -- 172.21.15.27:0/4250723898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c40017070 con 0x7f2c580c6ee0 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.864+0000 7f2c4f400640 1 -- 172.21.15.27:0/4250723898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4000ee60 con 0x7f2c580c6ee0 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.864+0000 7f2c4f400640 1 -- 172.21.15.27:0/4250723898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4000cb80 con 0x7f2c580c6ee0 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.864+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c581eaae0 con 0x7f2c580c6ee0 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.864+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c581ea790 con 0x7f2c580c6ee0 2024-03-28T19:01:32.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.865+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c58067dd0 con 0x7f2c580c6ee0 2024-03-28T19:01:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.866+0000 7f2c4f400640 1 -- 172.21.15.27:0/4250723898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2c4001a070 con 0x7f2c580c6ee0 2024-03-28T19:01:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.867+0000 7f2c4f400640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2c3406bbd0 0x7f2c3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.867+0000 7f2c4f400640 1 -- 172.21.15.27:0/4250723898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7f2c4008b120 con 0x7f2c580c6ee0 2024-03-28T19:01:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.868+0000 7f2c57400640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2c3406bbd0 0x7f2c3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:32.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.868+0000 7f2c57400640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2c3406bbd0 0x7f2c3406e090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2c48036000 tx=0x7f2c48008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:32.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:32.872+0000 7f2c4f400640 1 -- 172.21.15.27:0/4250723898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2c400593c0 con 0x7f2c580c6ee0 2024-03-28T19:01:32.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-28T19:01:33.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.114+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c5806da50 con 0x7f2c580c6ee0 2024-03-28T19:01:33.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.116+0000 7f2c4f400640 1 -- 172.21.15.27:0/4250723898 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f2c40058cc0 con 0x7f2c580c6ee0 2024-03-28T19:01:33.115 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:33.116 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:33.116 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:33.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.118+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2c3406bbd0 msgr2=0x7f2c3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.118+0000 7f2c5d200640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2c3406bbd0 0x7f2c3406e090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2c48036000 tx=0x7f2c48008030 comp rx=0 tx=0).stop 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.118+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c580c6ee0 msgr2=0x7f2c581e6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.118+0000 7f2c5d200640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c580c6ee0 0x7f2c581e6e90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000e870 tx=0x7f2c4000ed30 comp rx=0 tx=0).stop 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.119+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 shutdown_connections 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.119+0000 7f2c5d200640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2c3406bbd0 0x7f2c3406e090 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.119+0000 7f2c5d200640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2c58150ae0 0x7f2c581e73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.119+0000 7f2c5d200640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c580c6ee0 0x7f2c581e6e90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.119+0000 7f2c5d200640 1 --2- 172.21.15.27:0/4250723898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c580a61a0 0x7f2c58154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.119+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 >> 172.21.15.27:0/4250723898 conn(0x7f2c5806b0c0 msgr2=0x7f2c58155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.119+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 shutdown_connections 2024-03-28T19:01:33.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:33.119+0000 7f2c5d200640 1 -- 172.21.15.27:0/4250723898 wait complete. 2024-03-28T19:01:33.137 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:01:33.825 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.063+0000 7f9252800640 1 Processor -- start 2024-03-28T19:01:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.063+0000 7f9252800640 1 -- start start 2024-03-28T19:01:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.065+0000 7f9252800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0c69a0 0x7f924c0c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.065+0000 7f9252800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f924c150ec0 con 0x7f924c153a30 2024-03-28T19:01:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.065+0000 7f9252800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f924c06dc70 con 0x7f924c150ae0 2024-03-28T19:01:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.065+0000 7f9252800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c153550 con 0x7f924c0c69a0 2024-03-28T19:01:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.065+0000 7f9251e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f924c153a30 0x7f924c15dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53716/0 (socket says 172.21.15.27:53716) 2024-03-28T19:01:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.065+0000 7f9251e00640 1 -- 172.21.15.27:0/3172791144 learned_addr learned my addr 172.21.15.27:0/3172791144 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f9250a00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0c69a0 0x7f924c0c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367402329 0 0) 0x7f9244001490 con 0x7f924c153a30 2024-03-28T19:01:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9234003520 con 0x7f924c153a30 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1509303690 0 0) 0x7f9234003520 con 0x7f924c153a30 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0c69a0 msgr2=0x7f924c0c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0c69a0 0x7f924c0c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 >> v1:172.21.15.155:6789/0 conn(0x7f924c150ae0 legacy=0x7f924c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c06df10 con 0x7f924c153a30 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9244001bc0 con 0x7f924c153a30 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 >> v1:172.21.15.27:6789/0 conn(0x7f924c153a30 legacy=0x7f924c15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0c69a0 0x7f92340039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.066+0000 7f924be00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234003f60 0x7f9234004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f924be00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234004d40 0x7f92340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c153550 con 0x7f9234003f60 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923400bb50 con 0x7f9234004d40 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923400bd50 con 0x7f924c0c69a0 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f9250a00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234003f60 0x7f9234004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f9251e00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234004d40 0x7f92340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f9250a00640 1 -- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0c69a0 msgr2=0x7f92340039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f9250a00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0c69a0 0x7f92340039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f9250a00640 1 -- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234004d40 msgr2=0x7f92340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f9250a00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234004d40 0x7f92340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f9250a00640 1 -- 172.21.15.27:0/3172791144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9234003520 con 0x7f9234003f60 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.067+0000 7f9251e00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234004d40 0x7f92340051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.068+0000 7f9250a00640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234003f60 0x7f9234004800 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9238009ac0 tx=0x7f9234005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22aae9e37b40734 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.068+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238039070 con 0x7f9234003f60 2024-03-28T19:01:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.069+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923802c710 con 0x7f9234003f60 2024-03-28T19:01:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.069+0000 7f924be00640 1 -- 172.21.15.27:0/3172791144 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238009690 con 0x7f9234003f60 2024-03-28T19:01:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.069+0000 7f9252800640 1 -- 172.21.15.27:0/3172791144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234003f60 msgr2=0x7f9234004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.069+0000 7f9252800640 1 --2- 172.21.15.27:0/3172791144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234003f60 0x7f9234004800 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9238009ac0 tx=0x7f9234005830 comp rx=0 tx=0).stop 2024-03-28T19:01:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.069+0000 7f9251e00640 1 -- 172.21.15.27:0/3172791144 reap_dead start 2024-03-28T19:01:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.069+0000 7f9252800640 1 -- 172.21.15.27:0/3172791144 shutdown_connections 2024-03-28T19:01:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.069+0000 7f9252800640 1 -- 172.21.15.27:0/3172791144 >> 172.21.15.27:0/3172791144 conn(0x7f924c06b0c0 msgr2=0x7f924c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.070+0000 7f9252800640 1 -- 172.21.15.27:0/3172791144 shutdown_connections 2024-03-28T19:01:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.070+0000 7f9252800640 1 -- 172.21.15.27:0/3172791144 wait complete. 2024-03-28T19:01:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.071+0000 7f9252800640 1 Processor -- start 2024-03-28T19:01:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.071+0000 7f9252800640 1 -- start start 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.072+0000 7f9252800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234003f60 0x7f924c147590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.072+0000 7f9252800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924c14fb00 0x7f924c0c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.072+0000 7f9252800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0aa4a0 0x7f924c0aa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.072+0000 7f9252800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c06e7b0 con 0x7f924c14fb00 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.072+0000 7f9252800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c06e4d0 con 0x7f9234003f60 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.072+0000 7f9252800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924c0a6370 con 0x7f924c0aa4a0 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.072+0000 7f9250a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924c14fb00 0x7f924c0c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9250a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924c14fb00 0x7f924c0c4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45548/0 (socket says 172.21.15.27:45548) 2024-03-28T19:01:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9250a00640 1 -- 172.21.15.27:0/3333762640 learned_addr learned my addr 172.21.15.27:0/3333762640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9250a00640 1 -- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0aa4a0 msgr2=0x7f924c0aa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9251e00640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0aa4a0 0x7f924c0aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9250a00640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0aa4a0 0x7f924c0aa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9251400640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234003f60 0x7f924c147590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9250a00640 1 -- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234003f60 msgr2=0x7f924c147590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9250a00640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234003f60 0x7f924c147590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9250a00640 1 -- 172.21.15.27:0/3333762640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c0aae60 con 0x7f924c14fb00 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9251e00640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0aa4a0 0x7f924c0aa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9251400640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234003f60 0x7f924c147590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.073+0000 7f9250a00640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924c14fb00 0x7f924c0c4f40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9238000c00 tx=0x7f9238004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.074+0000 7f9249600640 1 -- 172.21.15.27:0/3333762640 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238042020 con 0x7f924c14fb00 2024-03-28T19:01:34.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.074+0000 7f9249600640 1 -- 172.21.15.27:0/3333762640 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923803a040 con 0x7f924c14fb00 2024-03-28T19:01:34.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.074+0000 7f9249600640 1 -- 172.21.15.27:0/3333762640 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238039070 con 0x7f924c14fb00 2024-03-28T19:01:34.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.074+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924c0c5820 con 0x7f924c14fb00 2024-03-28T19:01:34.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.074+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c1481f0 con 0x7f924c14fb00 2024-03-28T19:01:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.075+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924c067dd0 con 0x7f924c14fb00 2024-03-28T19:01:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.076+0000 7f9249600640 1 -- 172.21.15.27:0/3333762640 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9238045020 con 0x7f924c14fb00 2024-03-28T19:01:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.077+0000 7f9249600640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f922c06bbd0 0x7f922c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.077+0000 7f9249600640 1 -- 172.21.15.27:0/3333762640 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7677+0+0 (secure 0 0 0) 0x7f92380ac700 con 0x7f924c14fb00 2024-03-28T19:01:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.077+0000 7f9251400640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f922c06bbd0 0x7f922c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.078+0000 7f9251400640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f922c06bbd0 0x7f922c06e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9240001710 tx=0x7f9240009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:34.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.082+0000 7f9249600640 1 -- 172.21.15.27:0/3333762640 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f923804c070 con 0x7f924c14fb00 2024-03-28T19:01:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.319+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f924c069640 con 0x7f924c14fb00 2024-03-28T19:01:34.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.320+0000 7f9249600640 1 -- 172.21.15.27:0/3333762640 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f924c0aae60 con 0x7f924c14fb00 2024-03-28T19:01:34.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:34.320 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:00:48.706563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/620038090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":620038090},{"type":"v1","addr":"172.21.15.155:6834","nonce":620038090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":2}]} 2024-03-28T19:01:34.320 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 49 2024-03-28T19:01:34.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.322+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f922c06bbd0 msgr2=0x7f922c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.322+0000 7f9252800640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f922c06bbd0 0x7f922c06e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9240001710 tx=0x7f9240009040 comp rx=0 tx=0).stop 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924c14fb00 msgr2=0x7f924c0c4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924c14fb00 0x7f924c0c4f40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9238000c00 tx=0x7f9238004730 comp rx=0 tx=0).stop 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 shutdown_connections 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f922c06bbd0 0x7f922c06e090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f924c0aa4a0 0x7f924c0aa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924c14fb00 0x7f924c0c4f40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 --2- 172.21.15.27:0/3333762640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9234003f60 0x7f924c147590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 >> 172.21.15.27:0/3333762640 conn(0x7f924c06b0c0 msgr2=0x7f924c0c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 shutdown_connections 2024-03-28T19:01:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:34.323+0000 7f9252800640 1 -- 172.21.15.27:0/3333762640 wait complete. 2024-03-28T19:01:34.342 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:01:35.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.360+0000 7eff33200640 1 Processor -- start 2024-03-28T19:01:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.360+0000 7eff33200640 1 -- start start 2024-03-28T19:01:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.361+0000 7eff33200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.361+0000 7eff33200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff2c0c7df0 0x7eff2c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.361+0000 7eff33200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c0c7310 con 0x7eff2c0c7970 2024-03-28T19:01:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.361+0000 7eff33200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2c158d40 con 0x7eff2c0c7df0 2024-03-28T19:01:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.361+0000 7eff33200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c159020 con 0x7eff2c152f80 2024-03-28T19:01:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.362+0000 7eff31200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff2c0c7df0 0x7eff2c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.362+0000 7eff31200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff2c0c7df0 0x7eff2c155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45558/0 (socket says 172.21.15.27:45558) 2024-03-28T19:01:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.362+0000 7eff31200640 1 -- 172.21.15.27:0/2600270881 learned_addr learned my addr 172.21.15.27:0/2600270881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.362+0000 7eff2be00640 1 --2- 172.21.15.27:0/2600270881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.363+0000 7eff2be00640 1 -- 172.21.15.27:0/2600270881 >> v2:172.21.15.27:3300/0 conn(0x7eff2c0c7df0 msgr2=0x7eff2c155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:35.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.363+0000 7eff2be00640 1 --2- 172.21.15.27:0/2600270881 >> v2:172.21.15.27:3300/0 conn(0x7eff2c0c7df0 0x7eff2c155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.363+0000 7eff2be00640 1 -- 172.21.15.27:0/2600270881 >> v1:172.21.15.155:6789/0 conn(0x7eff2c0c7970 legacy=0x7eff2c155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.363+0000 7eff2be00640 1 -- 172.21.15.27:0/2600270881 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c159300 con 0x7eff2c152f80 2024-03-28T19:01:35.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.363+0000 7eff2aa00640 1 -- 172.21.15.27:0/2600270881 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329994458 0 0) 0x7eff2c0c7310 con 0x7eff2c0c7970 2024-03-28T19:01:35.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.363+0000 7eff31200640 1 --2- 172.21.15.27:0/2600270881 >> v2:172.21.15.27:3300/0 conn(0x7eff2c0c7df0 0x7eff2c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:01:35.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.364+0000 7eff2be00640 1 --2- 172.21.15.27:0/2600270881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c0c58e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0097a0 tx=0x7eff1c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b07c62d6ac0972 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.365+0000 7eff2aa00640 1 -- 172.21.15.27:0/2600270881 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c031780 con 0x7eff2c152f80 2024-03-28T19:01:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.365+0000 7eff2aa00640 1 -- 172.21.15.27:0/2600270881 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c037070 con 0x7eff2c152f80 2024-03-28T19:01:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.365+0000 7eff2aa00640 1 -- 172.21.15.27:0/2600270881 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c031b40 con 0x7eff2c152f80 2024-03-28T19:01:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.365+0000 7eff33200640 1 -- 172.21.15.27:0/2600270881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 msgr2=0x7eff2c0c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.365+0000 7eff33200640 1 --2- 172.21.15.27:0/2600270881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c0c58e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0097a0 tx=0x7eff1c02b820 comp rx=0 tx=0).stop 2024-03-28T19:01:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.366+0000 7eff33200640 1 -- 172.21.15.27:0/2600270881 shutdown_connections 2024-03-28T19:01:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.366+0000 7eff33200640 1 --2- 172.21.15.27:0/2600270881 >> v2:172.21.15.27:3300/0 conn(0x7eff2c0c7df0 0x7eff2c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.366+0000 7eff33200640 1 --2- 172.21.15.27:0/2600270881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c0c58e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.366+0000 7eff33200640 1 -- 172.21.15.27:0/2600270881 >> 172.21.15.27:0/2600270881 conn(0x7eff2c06b6f0 msgr2=0x7eff2c0caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.366+0000 7eff33200640 1 -- 172.21.15.27:0/2600270881 shutdown_connections 2024-03-28T19:01:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.366+0000 7eff33200640 1 -- 172.21.15.27:0/2600270881 wait complete. 2024-03-28T19:01:35.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.368+0000 7eff33200640 1 Processor -- start 2024-03-28T19:01:35.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.368+0000 7eff33200640 1 -- start start 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.369+0000 7eff33200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2c0c7970 0x7eff2c1eb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.369+0000 7eff33200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2c0c7df0 0x7eff2c150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.369+0000 7eff33200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.369+0000 7eff33200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c159b00 con 0x7eff2c0c7970 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.369+0000 7eff33200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c0c5e20 con 0x7eff2c0c7df0 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.369+0000 7eff33200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c0cbca0 con 0x7eff2c152f80 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.369+0000 7eff2be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2c0c7970 0x7eff2c1eb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff2be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2c0c7970 0x7eff2c1eb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45574/0 (socket says 172.21.15.27:45574) 2024-03-28T19:01:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff2be00640 1 -- 172.21.15.27:0/1691197774 learned_addr learned my addr 172.21.15.27:0/1691197774 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff2b400640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2c0c7df0 0x7eff2c150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff31200640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff2be00640 1 -- 172.21.15.27:0/1691197774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 msgr2=0x7eff2c154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff2be00640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff2be00640 1 -- 172.21.15.27:0/1691197774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2c0c7df0 msgr2=0x7eff2c150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff2be00640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2c0c7df0 0x7eff2c150fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.370+0000 7eff2be00640 1 -- 172.21.15.27:0/1691197774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c152060 con 0x7eff2c0c7970 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.371+0000 7eff2be00640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2c0c7970 0x7eff2c1eb7b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0097a0 tx=0x7eff1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.371+0000 7eff23e00640 1 -- 172.21.15.27:0/1691197774 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c032610 con 0x7eff2c0c7970 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.371+0000 7eff23e00640 1 -- 172.21.15.27:0/1691197774 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c0327b0 con 0x7eff2c0c7970 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.371+0000 7eff23e00640 1 -- 172.21.15.27:0/1691197774 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c032b80 con 0x7eff2c0c7970 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.371+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2c152340 con 0x7eff2c0c7970 2024-03-28T19:01:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.371+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2c1507b0 con 0x7eff2c0c7970 2024-03-28T19:01:35.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.372+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2c0c6130 con 0x7eff2c0c7970 2024-03-28T19:01:35.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.373+0000 7eff23e00640 1 -- 172.21.15.27:0/1691197774 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7eff1c04e440 con 0x7eff2c0c7970 2024-03-28T19:01:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.374+0000 7eff23e00640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0c06bc00 0x7eff0c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.374+0000 7eff23e00640 1 -- 172.21.15.27:0/1691197774 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7239+0+0 (secure 0 0 0) 0x7eff1c0acdf0 con 0x7eff2c0c7970 2024-03-28T19:01:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.375+0000 7eff2b400640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0c06bc00 0x7eff0c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.376+0000 7eff2b400640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0c06bc00 0x7eff0c06e0c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7eff18007fd0 tx=0x7eff18009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:35.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.379+0000 7eff23e00640 1 -- 172.21.15.27:0/1691197774 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7eff2c0c6130 con 0x7eff2c0c7970 2024-03-28T19:01:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.622+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff2c04f400 con 0x7eff2c0c7970 2024-03-28T19:01:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.623+0000 7eff23e00640 1 -- 172.21.15.27:0/1691197774 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5453 (secure 0 0 0) 0x7eff1c040070 con 0x7eff2c0c7970 2024-03-28T19:01:35.623 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:35.623 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:34.660049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:01:35.623 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-03-28T19:01:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0c06bc00 msgr2=0x7eff0c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0c06bc00 0x7eff0c06e0c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7eff18007fd0 tx=0x7eff18009040 comp rx=0 tx=0).stop 2024-03-28T19:01:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2c0c7970 msgr2=0x7eff2c1eb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2c0c7970 0x7eff2c1eb7b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0097a0 tx=0x7eff1c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 shutdown_connections 2024-03-28T19:01:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0c06bc00 0x7eff0c06e0c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2c152f80 0x7eff2c154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2c0c7df0 0x7eff2c150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 --2- 172.21.15.27:0/1691197774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2c0c7970 0x7eff2c1eb7b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 >> 172.21.15.27:0/1691197774 conn(0x7eff2c06b6f0 msgr2=0x7eff2c14da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:35.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 shutdown_connections 2024-03-28T19:01:35.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.626+0000 7eff33200640 1 -- 172.21.15.27:0/1691197774 wait complete. 2024-03-28T19:01:35.643 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-03-28T19:01:35.643 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:01:35.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.850+0000 7f68d8c00640 1 Processor -- start 2024-03-28T19:01:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.850+0000 7f68d8c00640 1 -- start start 2024-03-28T19:01:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.851+0000 7f68d8c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68d40c58c0 0x7f68d40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.851+0000 7f68d8c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68d40c8ed0 0x7f68d41555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.851+0000 7f68d8c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40be5d0 0x7f68d4155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.851+0000 7f68d8c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d4158bd0 con 0x7f68d40c8ed0 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.851+0000 7f68d8c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d4158eb0 con 0x7f68d40c58c0 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.851+0000 7f68d8c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4159190 con 0x7f68d40be5d0 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.852+0000 7f68d2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68d40c8ed0 0x7f68d41555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.852+0000 7f68d2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68d40c8ed0 0x7f68d41555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45590/0 (socket says 172.21.15.27:45590) 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.852+0000 7f68d2a00640 1 -- 172.21.15.27:0/1641039934 learned_addr learned my addr 172.21.15.27:0/1641039934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.852+0000 7f68d2a00640 1 -- 172.21.15.27:0/1641039934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40be5d0 msgr2=0x7f68d4155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.852+0000 7f68d2a00640 1 --2- 172.21.15.27:0/1641039934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40be5d0 0x7f68d4155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.852+0000 7f68d2a00640 1 -- 172.21.15.27:0/1641039934 >> v2:172.21.15.155:3300/0 conn(0x7f68d40c58c0 msgr2=0x7f68d40c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.852+0000 7f68d2a00640 1 --2- 172.21.15.27:0/1641039934 >> v2:172.21.15.155:3300/0 conn(0x7f68d40c58c0 0x7f68d40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.852+0000 7f68d2a00640 1 -- 172.21.15.27:0/1641039934 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d4159470 con 0x7f68d40c8ed0 2024-03-28T19:01:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.853+0000 7f68d2a00640 1 --2- 172.21.15.27:0/1641039934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40c8ed0 0x7f68d41555e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f68d406c060 tx=0x7f68bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c41770b4ad7badf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d2000640 1 -- 172.21.15.27:0/1641039934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc02e070 con 0x7f68d40c8ed0 2024-03-28T19:01:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d2000640 1 -- 172.21.15.27:0/1641039934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68bc037070 con 0x7f68d40c8ed0 2024-03-28T19:01:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d2000640 1 -- 172.21.15.27:0/1641039934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc031a00 con 0x7f68d40c8ed0 2024-03-28T19:01:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d8c00640 1 -- 172.21.15.27:0/1641039934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40c8ed0 msgr2=0x7f68d41555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d8c00640 1 --2- 172.21.15.27:0/1641039934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40c8ed0 0x7f68d41555e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f68d406c060 tx=0x7f68bc02b880 comp rx=0 tx=0).stop 2024-03-28T19:01:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d8c00640 1 -- 172.21.15.27:0/1641039934 shutdown_connections 2024-03-28T19:01:35.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d8c00640 1 --2- 172.21.15.27:0/1641039934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40be5d0 0x7f68d4155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d8c00640 1 --2- 172.21.15.27:0/1641039934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40c8ed0 0x7f68d41555e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d8c00640 1 --2- 172.21.15.27:0/1641039934 >> v2:172.21.15.155:3300/0 conn(0x7f68d40c58c0 0x7f68d40c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d8c00640 1 -- 172.21.15.27:0/1641039934 >> 172.21.15.27:0/1641039934 conn(0x7f68d406b6f0 msgr2=0x7f68d40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:35.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.854+0000 7f68d8c00640 1 -- 172.21.15.27:0/1641039934 shutdown_connections 2024-03-28T19:01:35.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.855+0000 7f68d8c00640 1 -- 172.21.15.27:0/1641039934 wait complete. 2024-03-28T19:01:35.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.856+0000 7f68d8c00640 1 Processor -- start 2024-03-28T19:01:35.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.856+0000 7f68d8c00640 1 -- start start 2024-03-28T19:01:35.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40be5d0 0x7f68d40c2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d8c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40c58c0 0x7f68d40c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d8c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d40c8ed0 0x7f68d414c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d8c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40b09a0 con 0x7f68d40be5d0 2024-03-28T19:01:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d8c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40b0820 con 0x7f68d40c8ed0 2024-03-28T19:01:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d8c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40ad7d0 con 0x7f68d40c58c0 2024-03-28T19:01:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40be5d0 0x7f68d40c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40be5d0 0x7f68d40c2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45596/0 (socket says 172.21.15.27:45596) 2024-03-28T19:01:35.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.857+0000 7f68d3400640 1 -- 172.21.15.27:0/3861540887 learned_addr learned my addr 172.21.15.27:0/3861540887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d3e00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d40c8ed0 0x7f68d414c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d2a00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40c58c0 0x7f68d40c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d3400640 1 -- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40c58c0 msgr2=0x7f68d40c2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d3400640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40c58c0 0x7f68d40c2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d3400640 1 -- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d40c8ed0 msgr2=0x7f68d414c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d3400640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d40c8ed0 0x7f68d414c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d3400640 1 -- 172.21.15.27:0/3861540887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40c46e0 con 0x7f68d40be5d0 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d2a00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40c58c0 0x7f68d40c2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68d3400640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40be5d0 0x7f68d40c2010 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f68c400ba80 tx=0x7f68c400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:35.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68cb400640 1 -- 172.21.15.27:0/3861540887 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c4019070 con 0x7f68d40be5d0 2024-03-28T19:01:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.858+0000 7f68cb400640 1 -- 172.21.15.27:0/3861540887 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c400cc20 con 0x7f68d40be5d0 2024-03-28T19:01:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.859+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d40c4cc0 con 0x7f68d40be5d0 2024-03-28T19:01:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.859+0000 7f68cb400640 1 -- 172.21.15.27:0/3861540887 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c4014440 con 0x7f68d40be5d0 2024-03-28T19:01:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.859+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d40c3410 con 0x7f68d40be5d0 2024-03-28T19:01:35.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.860+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d40ab580 con 0x7f68d40be5d0 2024-03-28T19:01:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.861+0000 7f68cb400640 1 -- 172.21.15.27:0/3861540887 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f68c401c070 con 0x7f68d40be5d0 2024-03-28T19:01:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.862+0000 7f68cb400640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b006bc00 0x7f68b006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:35.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.862+0000 7f68cb400640 1 -- 172.21.15.27:0/3861540887 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7239+0+0 (secure 0 0 0) 0x7f68c408b1b0 con 0x7f68d40be5d0 2024-03-28T19:01:35.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.863+0000 7f68d2a00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b006bc00 0x7f68b006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:35.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.864+0000 7f68d2a00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b006bc00 0x7f68b006e0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f68bc036000 tx=0x7f68bc008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:35.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:35.866+0000 7f68cb400640 1 -- 172.21.15.27:0/3861540887 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f68d40ab580 con 0x7f68d40be5d0 2024-03-28T19:01:36.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.098+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68d4069640 con 0x7f68d40be5d0 2024-03-28T19:01:36.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.099+0000 7f68cb400640 1 -- 172.21.15.27:0/3861540887 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5453 (secure 0 0 0) 0x7f68c4059030 con 0x7f68d40be5d0 2024-03-28T19:01:36.099 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:36.099 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:34.660049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:01:36.099 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-03-28T19:01:36.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b006bc00 msgr2=0x7f68b006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b006bc00 0x7f68b006e0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f68bc036000 tx=0x7f68bc008030 comp rx=0 tx=0).stop 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40be5d0 msgr2=0x7f68d40c2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40be5d0 0x7f68d40c2010 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f68c400ba80 tx=0x7f68c400bf40 comp rx=0 tx=0).stop 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 shutdown_connections 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b006bc00 0x7f68b006e0c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d40c8ed0 0x7f68d414c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d40c58c0 0x7f68d40c2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 --2- 172.21.15.27:0/3861540887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d40be5d0 0x7f68d40c2010 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 >> 172.21.15.27:0/3861540887 conn(0x7f68d406b6f0 msgr2=0x7f68d4072570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:36.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 shutdown_connections 2024-03-28T19:01:36.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.102+0000 7f68d8c00640 1 -- 172.21.15.27:0/3861540887 wait complete. 2024-03-28T19:01:36.120 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5325": { "addr": "172.21.15.27:6836/181571966", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 181571966, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 181571966, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5325, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T18:59:54.159296+0000", "qdb_cluster": [ 5325 ], "qdb_leader": 5325, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5325 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5302": { "addr": "172.21.15.27:6837/4097237428", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 4097237428, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 4097237428, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5302, "incarnation": 50, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:01:34.660049+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5302 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:01:36.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:36.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.575+0000 7f9eae400640 1 Processor -- start 2024-03-28T19:01:36.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.575+0000 7f9eae400640 1 -- start start 2024-03-28T19:01:36.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.577+0000 7f9eae400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 0x7f9ea8150580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:36.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.577+0000 7f9eae400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9ea80c4630 0x7f9ea8157190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:36.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.577+0000 7f9eae400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea81543f0 con 0x7f9ea80c2430 2024-03-28T19:01:36.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.577+0000 7f9eae400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea80c2d50 con 0x7f9ea80c4630 2024-03-28T19:01:36.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.577+0000 7f9eae400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8153630 con 0x7f9ea80c5210 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9ea7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9ea80c4630 0x7f9ea8157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9e9ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9ea80c2430 0x7f9ea8156a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36552/0 (socket says 172.21.15.27:36552) 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9e9ea00640 1 -- 172.21.15.27:0/1862945579 learned_addr learned my addr 172.21.15.27:0/1862945579 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9ea7400640 1 --2- 172.21.15.27:0/1862945579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 0x7f9ea8150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9ea7e00640 1 -- 172.21.15.27:0/1862945579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 msgr2=0x7f9ea8150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9ea7e00640 1 --2- 172.21.15.27:0/1862945579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 0x7f9ea8150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9ea7e00640 1 -- 172.21.15.27:0/1862945579 >> v1:172.21.15.27:6789/0 conn(0x7f9ea80c2430 legacy=0x7f9ea8156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9ea7e00640 1 -- 172.21.15.27:0/1862945579 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea8153910 con 0x7f9ea80c4630 2024-03-28T19:01:36.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.578+0000 7f9ea6a00640 1 -- 172.21.15.27:0/1862945579 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840729094 0 0) 0x7f9ea81543f0 con 0x7f9ea80c2430 2024-03-28T19:01:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.579+0000 7f9ea7e00640 1 --2- 172.21.15.27:0/1862945579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea8157190 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00097a0 tx=0x7f9ea002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb5d7c94ffd3d4b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9ea6a00640 1 -- 172.21.15.27:0/1862945579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0035020 con 0x7f9ea80c4630 2024-03-28T19:01:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9ea6a00640 1 -- 172.21.15.27:0/1862945579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea0031830 con 0x7f9ea80c4630 2024-03-28T19:01:36.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9ea6a00640 1 -- 172.21.15.27:0/1862945579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0031c00 con 0x7f9ea80c4630 2024-03-28T19:01:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9eae400640 1 -- 172.21.15.27:0/1862945579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 msgr2=0x7f9ea8157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9eae400640 1 --2- 172.21.15.27:0/1862945579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea8157190 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00097a0 tx=0x7f9ea002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9eae400640 1 -- 172.21.15.27:0/1862945579 shutdown_connections 2024-03-28T19:01:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9eae400640 1 --2- 172.21.15.27:0/1862945579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea8157190 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9eae400640 1 --2- 172.21.15.27:0/1862945579 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 0x7f9ea8150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9eae400640 1 -- 172.21.15.27:0/1862945579 >> 172.21.15.27:0/1862945579 conn(0x7f9ea806c5f0 msgr2=0x7f9ea806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9eae400640 1 -- 172.21.15.27:0/1862945579 shutdown_connections 2024-03-28T19:01:36.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.580+0000 7f9eae400640 1 -- 172.21.15.27:0/1862945579 wait complete. 2024-03-28T19:01:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.582+0000 7f9eae400640 1 Processor -- start 2024-03-28T19:01:36.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.582+0000 7f9eae400640 1 -- start start 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.583+0000 7f9eae400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea80c2430 0x7f9ea81518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.583+0000 7f9eae400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea80ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.583+0000 7f9eae400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 0x7f9ea81ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.583+0000 7f9eae400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea80c2a00 con 0x7f9ea80c4630 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.583+0000 7f9eae400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8153ed0 con 0x7f9ea80c2430 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.583+0000 7f9eae400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea80c55f0 con 0x7f9ea80c5210 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea80ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea80ab490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40238/0 (socket says 172.21.15.27:40238) 2024-03-28T19:01:36.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 -- 172.21.15.27:0/397911029 learned_addr learned my addr 172.21.15.27:0/397911029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:36.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9ea7400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea80c2430 0x7f9ea81518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9ea7e00640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 0x7f9ea81ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 -- 172.21.15.27:0/397911029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 msgr2=0x7f9ea81ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 0x7f9ea81ee680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 -- 172.21.15.27:0/397911029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea80c2430 msgr2=0x7f9ea81518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea80c2430 0x7f9ea81518c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 -- 172.21.15.27:0/397911029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea80ad200 con 0x7f9ea80c4630 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.584+0000 7f9e9ea00640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea80ab490 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000c4e0 tx=0x7f9e9000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.585+0000 7f9e9fe00640 1 -- 172.21.15.27:0/397911029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90018070 con 0x7f9ea80c4630 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.585+0000 7f9e9fe00640 1 -- 172.21.15.27:0/397911029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e90007500 con 0x7f9ea80c4630 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.585+0000 7f9e9fe00640 1 -- 172.21.15.27:0/397911029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90004a20 con 0x7f9ea80c4630 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.585+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea80ac960 con 0x7f9ea80c4630 2024-03-28T19:01:36.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.585+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea80acf20 con 0x7f9ea80c4630 2024-03-28T19:01:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.586+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea804f400 con 0x7f9ea80c4630 2024-03-28T19:01:36.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.587+0000 7f9e9fe00640 1 -- 172.21.15.27:0/397911029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9e9001b070 con 0x7f9ea80c4630 2024-03-28T19:01:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.589+0000 7f9e9fe00640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8406bc00 0x7f9e8406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.589+0000 7f9e9fe00640 1 -- 172.21.15.27:0/397911029 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7239+0+0 (secure 0 0 0) 0x7f9e9008a2a0 con 0x7f9ea80c4630 2024-03-28T19:01:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.589+0000 7f9ea7400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8406bc00 0x7f9e8406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:36.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.590+0000 7f9ea7400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8406bc00 0x7f9e8406e0c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009e90 tx=0x7f9e9800b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:36.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.594+0000 7f9e9fe00640 1 -- 172.21.15.27:0/397911029 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9e900588e0 con 0x7f9ea80c4630 2024-03-28T19:01:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.832+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ea8149880 con 0x7f9ea80c4630 2024-03-28T19:01:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.833+0000 7f9e9fe00640 1 -- 172.21.15.27:0/397911029 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f9e900581e0 con 0x7f9ea80c4630 2024-03-28T19:01:36.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:36.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:36.066013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:01:36.834 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 51 2024-03-28T19:01:36.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8406bc00 msgr2=0x7f9e8406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:36.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8406bc00 0x7f9e8406e0c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009e90 tx=0x7f9e9800b040 comp rx=0 tx=0).stop 2024-03-28T19:01:36.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 msgr2=0x7f9ea80ab490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:36.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea80ab490 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000c4e0 tx=0x7f9e9000c9a0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 shutdown_connections 2024-03-28T19:01:36.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e8406bc00 0x7f9e8406e0c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ea80c5210 0x7f9ea81ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea80c4630 0x7f9ea80ab490 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 --2- 172.21.15.27:0/397911029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea80c2430 0x7f9ea81518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:36.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 >> 172.21.15.27:0/397911029 conn(0x7f9ea806c5f0 msgr2=0x7f9ea814fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:36.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 shutdown_connections 2024-03-28T19:01:36.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:36.836+0000 7f9eae400640 1 -- 172.21.15.27:0/397911029 wait complete. 2024-03-28T19:01:36.852 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:01:38.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.363+0000 7fa536c00640 1 Processor -- start 2024-03-28T19:01:38.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.363+0000 7fa536c00640 1 -- start start 2024-03-28T19:01:38.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.365+0000 7fa536c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa53006c340 0x7fa530156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:38.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.365+0000 7fa536c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa530071a00 0x7fa530157520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:38.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.365+0000 7fa536c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5300c5c70 con 0x7fa5300731c0 2024-03-28T19:01:38.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.365+0000 7fa536c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5300c63e0 con 0x7fa530071a00 2024-03-28T19:01:38.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.365+0000 7fa536c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300c69a0 con 0x7fa53006c340 2024-03-28T19:01:38.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.365+0000 7fa534c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa530071a00 0x7fa530157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:38.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.366+0000 7fa534c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa530071a00 0x7fa530157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40248/0 (socket says 172.21.15.27:40248) 2024-03-28T19:01:38.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.366+0000 7fa534c00640 1 -- 172.21.15.27:0/4115818132 learned_addr learned my addr 172.21.15.27:0/4115818132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:38.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.366+0000 7fa52fe00640 1 --2- 172.21.15.27:0/4115818132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa53006c340 0x7fa530156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:38.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.366+0000 7fa534c00640 1 -- 172.21.15.27:0/4115818132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa53006c340 msgr2=0x7fa530156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:38.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.366+0000 7fa534c00640 1 --2- 172.21.15.27:0/4115818132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa53006c340 0x7fa530156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.366+0000 7fa534c00640 1 -- 172.21.15.27:0/4115818132 >> v1:172.21.15.155:6789/0 conn(0x7fa5300731c0 legacy=0x7fa530156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:38.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.366+0000 7fa534c00640 1 -- 172.21.15.27:0/4115818132 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5300c5f10 con 0x7fa530071a00 2024-03-28T19:01:38.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.367+0000 7fa534c00640 1 --2- 172.21.15.27:0/4115818132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa530071a00 0x7fa530157520 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa528009e50 tx=0x7fa52802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21ab55626873755 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.367+0000 7fa52f400640 1 -- 172.21.15.27:0/4115818132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52802c9e0 con 0x7fa530071a00 2024-03-28T19:01:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa52f400640 1 -- 172.21.15.27:0/4115818132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52802cb80 con 0x7fa530071a00 2024-03-28T19:01:38.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa52f400640 1 -- 172.21.15.27:0/4115818132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5280343f0 con 0x7fa530071a00 2024-03-28T19:01:38.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa536c00640 1 -- 172.21.15.27:0/4115818132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa530071a00 msgr2=0x7fa530157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:38.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa536c00640 1 --2- 172.21.15.27:0/4115818132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa530071a00 0x7fa530157520 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa528009e50 tx=0x7fa52802be40 comp rx=0 tx=0).stop 2024-03-28T19:01:38.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa536c00640 1 -- 172.21.15.27:0/4115818132 shutdown_connections 2024-03-28T19:01:38.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa536c00640 1 --2- 172.21.15.27:0/4115818132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa530071a00 0x7fa530157520 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa536c00640 1 --2- 172.21.15.27:0/4115818132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa53006c340 0x7fa530156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa536c00640 1 -- 172.21.15.27:0/4115818132 >> 172.21.15.27:0/4115818132 conn(0x7fa5300c4750 msgr2=0x7fa5300c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:38.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.368+0000 7fa536c00640 1 -- 172.21.15.27:0/4115818132 shutdown_connections 2024-03-28T19:01:38.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.369+0000 7fa536c00640 1 -- 172.21.15.27:0/4115818132 wait complete. 2024-03-28T19:01:38.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.371+0000 7fa536c00640 1 Processor -- start 2024-03-28T19:01:38.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.371+0000 7fa536c00640 1 -- start start 2024-03-28T19:01:38.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa536c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa53006c340 0x7fa5300a8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa536c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa530071a00 0x7fa53014e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa536c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5300731c0 0x7fa53014e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa536c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300c5580 con 0x7fa5300731c0 2024-03-28T19:01:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa536c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa530157a60 con 0x7fa53006c340 2024-03-28T19:01:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa536c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300c5700 con 0x7fa530071a00 2024-03-28T19:01:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa534c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5300731c0 0x7fa53014e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa534c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5300731c0 0x7fa53014e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40256/0 (socket says 172.21.15.27:40256) 2024-03-28T19:01:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.372+0000 7fa534c00640 1 -- 172.21.15.27:0/3699886606 learned_addr learned my addr 172.21.15.27:0/3699886606 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:38.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa52fe00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa53006c340 0x7fa5300a8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:38.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa527e00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa530071a00 0x7fa53014e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:38.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa534c00640 1 -- 172.21.15.27:0/3699886606 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa530071a00 msgr2=0x7fa53014e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:38.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa534c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa530071a00 0x7fa53014e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa534c00640 1 -- 172.21.15.27:0/3699886606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa53006c340 msgr2=0x7fa5300a8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa534c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa53006c340 0x7fa5300a8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa534c00640 1 -- 172.21.15.27:0/3699886606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53014ed60 con 0x7fa5300731c0 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa534c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5300731c0 0x7fa53014e820 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa528036040 tx=0x7fa528004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.373+0000 7fa52cc00640 1 -- 172.21.15.27:0/3699886606 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528043020 con 0x7fa5300731c0 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.374+0000 7fa52cc00640 1 -- 172.21.15.27:0/3699886606 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52803e450 con 0x7fa5300731c0 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.374+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5300c0ae0 con 0x7fa5300731c0 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.374+0000 7fa52cc00640 1 -- 172.21.15.27:0/3699886606 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528009510 con 0x7fa5300731c0 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.374+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5300c10a0 con 0x7fa5300731c0 2024-03-28T19:01:38.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.375+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5300a6d20 con 0x7fa5300731c0 2024-03-28T19:01:38.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.376+0000 7fa52cc00640 1 -- 172.21.15.27:0/3699886606 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa528046020 con 0x7fa5300731c0 2024-03-28T19:01:38.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.377+0000 7fa52cc00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa50c06bbd0 0x7fa50c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:38.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.377+0000 7fa52cc00640 1 -- 172.21.15.27:0/3699886606 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7239+0+0 (secure 0 0 0) 0x7fa5280ad8e0 con 0x7fa5300731c0 2024-03-28T19:01:38.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.377+0000 7fa52fe00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa50c06bbd0 0x7fa50c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:38.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.378+0000 7fa52fe00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa50c06bbd0 0x7fa50c06e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa520006fd0 tx=0x7fa520008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:38.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.381+0000 7fa52cc00640 1 -- 172.21.15.27:0/3699886606 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa5300a6d20 con 0x7fa5300731c0 2024-03-28T19:01:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.624+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa53006c8c0 con 0x7fa5300731c0 2024-03-28T19:01:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.626+0000 7fa52cc00640 1 -- 172.21.15.27:0/3699886606 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5454 (secure 0 0 0) 0x7fa52804d030 con 0x7fa5300731c0 2024-03-28T19:01:38.625 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:38.625 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:36.823049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:01:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 52 2024-03-28T19:01:38.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.628+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa50c06bbd0 msgr2=0x7fa50c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:38.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.628+0000 7fa536c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa50c06bbd0 0x7fa50c06e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa520006fd0 tx=0x7fa520008040 comp rx=0 tx=0).stop 2024-03-28T19:01:38.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5300731c0 msgr2=0x7fa53014e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5300731c0 0x7fa53014e820 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa528036040 tx=0x7fa528004a40 comp rx=0 tx=0).stop 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 shutdown_connections 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa50c06bbd0 0x7fa50c06e090 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5300731c0 0x7fa53014e820 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa530071a00 0x7fa53014e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 --2- 172.21.15.27:0/3699886606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa53006c340 0x7fa5300a8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 >> 172.21.15.27:0/3699886606 conn(0x7fa5300c4750 msgr2=0x7fa530157350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 shutdown_connections 2024-03-28T19:01:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:38.629+0000 7fa536c00640 1 -- 172.21.15.27:0/3699886606 wait complete. 2024-03-28T19:01:38.645 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:01:38.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.096+0000 7f0b53800640 1 Processor -- start 2024-03-28T19:01:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.097+0000 7f0b53800640 1 -- start start 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b53800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b4c0c5e80 0x7f0b4c0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b53800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c06d8a0 con 0x7f0b4c0abf60 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b53800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c06db80 con 0x7f0b4c0c1f50 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b53800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c06d650 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b50e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b4c0c5e80 0x7f0b4c0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b50e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b4c0c5e80 0x7f0b4c0c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40270/0 (socket says 172.21.15.27:40270) 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b50e00640 1 -- 172.21.15.27:0/125559024 learned_addr learned my addr 172.21.15.27:0/125559024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b51800640 1 --1- 172.21.15.27:0/125559024 >> v1:172.21.15.27:6789/0 conn(0x7f0b4c0abf60 0x7f0b4c0aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36568/0 (socket says 172.21.15.27:36568) 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.098+0000 7f0b50e00640 1 -- 172.21.15.27:0/125559024 >> v1:172.21.15.155:6789/0 conn(0x7f0b4c0c1f50 legacy=0x7f0b4c0ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.099+0000 7f0b50e00640 1 -- 172.21.15.27:0/125559024 >> v1:172.21.15.27:6789/0 conn(0x7f0b4c0abf60 legacy=0x7f0b4c0aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.099+0000 7f0b4be00640 1 -- 172.21.15.27:0/125559024 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146368527 0 0) 0x7f0b4c06d8a0 con 0x7f0b4c0abf60 2024-03-28T19:01:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.099+0000 7f0b50e00640 1 -- 172.21.15.27:0/125559024 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c06de60 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.100+0000 7f0b50e00640 1 --2- 172.21.15.27:0/125559024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c2cb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c009870 tx=0x7f0b3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151277db0aac0d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.100+0000 7f0b4be00640 1 -- 172.21.15.27:0/125559024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c02e070 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.100+0000 7f0b4be00640 1 -- 172.21.15.27:0/125559024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c037070 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.100+0000 7f0b4be00640 1 -- 172.21.15.27:0/125559024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c031a00 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.101+0000 7f0b53800640 1 -- 172.21.15.27:0/125559024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 msgr2=0x7f0b4c0c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.101+0000 7f0b53800640 1 --2- 172.21.15.27:0/125559024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c2cb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c009870 tx=0x7f0b3c02b880 comp rx=0 tx=0).stop 2024-03-28T19:01:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.101+0000 7f0b53800640 1 -- 172.21.15.27:0/125559024 shutdown_connections 2024-03-28T19:01:39.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.101+0000 7f0b53800640 1 --2- 172.21.15.27:0/125559024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c2cb0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:39.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.101+0000 7f0b53800640 1 -- 172.21.15.27:0/125559024 >> 172.21.15.27:0/125559024 conn(0x7f0b4c06b9d0 msgr2=0x7f0b4c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:39.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.101+0000 7f0b53800640 1 -- 172.21.15.27:0/125559024 shutdown_connections 2024-03-28T19:01:39.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.101+0000 7f0b53800640 1 -- 172.21.15.27:0/125559024 wait complete. 2024-03-28T19:01:39.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.102+0000 7f0b53800640 1 Processor -- start 2024-03-28T19:01:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.102+0000 7f0b53800640 1 -- start start 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.103+0000 7f0b53800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b4c0abf60 0x7f0b4c145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.103+0000 7f0b53800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b4c0c1f50 0x7f0b4c0bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.103+0000 7f0b53800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.103+0000 7f0b53800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c0c6260 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.103+0000 7f0b53800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c06e1d0 con 0x7f0b4c0abf60 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.103+0000 7f0b53800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c0c63e0 con 0x7f0b4c0c1f50 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40286/0 (socket says 172.21.15.27:40286) 2024-03-28T19:01:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 -- 172.21.15.27:0/4259675577 learned_addr learned my addr 172.21.15.27:0/4259675577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b50e00640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b4c0abf60 0x7f0b4c145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b43e00640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b4c0c1f50 0x7f0b4c0bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 -- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b4c0c1f50 msgr2=0x7f0b4c0bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b4c0c1f50 0x7f0b4c0bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 -- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b4c0abf60 msgr2=0x7f0b4c145dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b4c0abf60 0x7f0b4c145dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 -- 172.21.15.27:0/4259675577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c071db0 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b50e00640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b4c0abf60 0x7f0b4c145dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.104+0000 7f0b51800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c02f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400c730 tx=0x7f0b4400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.105+0000 7f0b49600640 1 -- 172.21.15.27:0/4259675577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b44018070 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.105+0000 7f0b49600640 1 -- 172.21.15.27:0/4259675577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4400ce40 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.105+0000 7f0b49600640 1 -- 172.21.15.27:0/4259675577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4400a860 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.105+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c0bd220 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.105+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c0bd7e0 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.106+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c0bf680 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.107+0000 7f0b49600640 1 -- 172.21.15.27:0/4259675577 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0b4401b070 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.108+0000 7f0b49600640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b2806bbd0 0x7f0b2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.108+0000 7f0b49600640 1 -- 172.21.15.27:0/4259675577 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7239+0+0 (secure 0 0 0) 0x7f0b4408a680 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.109+0000 7f0b50e00640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b2806bbd0 0x7f0b2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.109+0000 7f0b50e00640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b2806bbd0 0x7f0b2806e090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c036000 tx=0x7f0b3c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:39.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.113+0000 7f0b49600640 1 -- 172.21.15.27:0/4259675577 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0b44058b60 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.357+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b4c0a8830 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.358+0000 7f0b49600640 1 -- 172.21.15.27:0/4259675577 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5460 (secure 0 0 0) 0x7f0b44058460 con 0x7f0b4c0c5e80 2024-03-28T19:01:39.358 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:39.358 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:37.934279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:clientreplay","state_seq":14,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:01:39.358 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 53 2024-03-28T19:01:39.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b2806bbd0 msgr2=0x7f0b2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b2806bbd0 0x7f0b2806e090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c036000 tx=0x7f0b3c008030 comp rx=0 tx=0).stop 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 msgr2=0x7f0b4c0c02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c02f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400c730 tx=0x7f0b4400cbf0 comp rx=0 tx=0).stop 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 shutdown_connections 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b2806bbd0 0x7f0b2806e090 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b4c0c5e80 0x7f0b4c0c02f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b4c0c1f50 0x7f0b4c0bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 --2- 172.21.15.27:0/4259675577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b4c0abf60 0x7f0b4c145dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 >> 172.21.15.27:0/4259675577 conn(0x7f0b4c06b9d0 msgr2=0x7f0b4c0714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 shutdown_connections 2024-03-28T19:01:39.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:39.361+0000 7f0b53800640 1 -- 172.21.15.27:0/4259675577 wait complete. 2024-03-28T19:01:39.380 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:01:40.065 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/31' 2024-03-28T19:01:40.065 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/36' 2024-03-28T19:01:40.065 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/20' 2024-03-28T19:01:40.065 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/12' 2024-03-28T19:01:40.065 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/39' 2024-03-28T19:01:40.065 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/10' 2024-03-28T19:01:40.065 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/34' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/38' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/11' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/16' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/32' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/30' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/14' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/18' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/13' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/33' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/17' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/35' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './177/37' 2024-03-28T19:01:40.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './177' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/19' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/15' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/31' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/36' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/20' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/12' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/39' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/10' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/34' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/38' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/11' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/16' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/32' 2024-03-28T19:01:40.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/30' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/14' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/18' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/13' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/33' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/17' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/35' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './178/37' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './178' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/19' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/15' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/31' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/36' 2024-03-28T19:01:40.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/20' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/12' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/39' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/10' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/34' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/38' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/11' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/16' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/32' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/30' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/14' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/18' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/13' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/33' 2024-03-28T19:01:40.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/17' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/35' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './179/37' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './179' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/19' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/15' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/31' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/36' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/20' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/12' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/39' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/10' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/34' 2024-03-28T19:01:40.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/38' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/11' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/16' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/32' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/30' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/14' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/18' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/13' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/33' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/17' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/35' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './180/37' 2024-03-28T19:01:40.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './180' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/19' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/15' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/31' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/36' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/20' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/12' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/39' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/10' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/34' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/38' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/11' 2024-03-28T19:01:40.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/16' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/32' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/30' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/14' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/18' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/13' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/33' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/17' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/35' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './181/37' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './181' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/19' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/15' 2024-03-28T19:01:40.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/31' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/36' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/20' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/12' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/39' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/10' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/34' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/38' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/11' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/16' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/32' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/30' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/14' 2024-03-28T19:01:40.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/18' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/13' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/33' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/17' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/35' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './182/37' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './182' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/19' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/15' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/31' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/36' 2024-03-28T19:01:40.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/20' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/12' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/39' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/10' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/34' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/38' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/11' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/16' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/32' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/30' 2024-03-28T19:01:40.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/14' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/18' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/13' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/33' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/17' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/35' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './183/37' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './183' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/19' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/15' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/31' 2024-03-28T19:01:40.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/36' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/20' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/12' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/39' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/10' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/34' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/38' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/11' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/16' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/32' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/30' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/14' 2024-03-28T19:01:40.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/18' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/13' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/33' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/17' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/35' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './184/37' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './184' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/19' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/15' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/31' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/36' 2024-03-28T19:01:40.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/20' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/12' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/39' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/10' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/34' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/38' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/11' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/16' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/32' 2024-03-28T19:01:40.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/30' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/14' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/18' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/13' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/33' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/17' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/35' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './185/37' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './185' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/19' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/15' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/31' 2024-03-28T19:01:40.081 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/36' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/20' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/12' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/39' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/10' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/34' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/38' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/11' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/16' 2024-03-28T19:01:40.082 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/32' 2024-03-28T19:01:40.083 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/30' 2024-03-28T19:01:40.083 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/14' 2024-03-28T19:01:40.083 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/18' 2024-03-28T19:01:40.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:40.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.887+0000 7fa903200640 1 Processor -- start 2024-03-28T19:01:40.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.887+0000 7fa903200640 1 -- start start 2024-03-28T19:01:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.889+0000 7fa903200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 0x7fa9040c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.889+0000 7fa903200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa904155fe0 con 0x7fa904155600 2024-03-28T19:01:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.889+0000 7fa903200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9041562c0 con 0x7fa9040727e0 2024-03-28T19:01:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.889+0000 7fa903200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9041565a0 con 0x7fa9040c59d0 2024-03-28T19:01:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.889+0000 7fa902800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa904155600 0x7fa90414f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36580/0 (socket says 172.21.15.27:36580) 2024-03-28T19:01:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.889+0000 7fa902800640 1 -- 172.21.15.27:0/3803807406 learned_addr learned my addr 172.21.15.27:0/3803807406 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.889+0000 7fa901e00640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 0x7fa9040c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969354348 0 0) 0x7fa904155fe0 con 0x7fa904155600 2024-03-28T19:01:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e8003520 con 0x7fa904155600 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896660121 0 0) 0x7fa8e8003520 con 0x7fa904155600 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 msgr2=0x7fa9040c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 0x7fa9040c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 >> v1:172.21.15.155:6789/0 conn(0x7fa9040727e0 legacy=0x7fa904159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa904158d20 con 0x7fa904155600 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fa8fc0019f0 con 0x7fa904155600 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 >> v1:172.21.15.27:6789/0 conn(0x7fa904155600 legacy=0x7fa90414f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 0x7fa8e8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa8e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e8004d40 0x7fa8e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9041565a0 con 0x7fa8e8003f80 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e800bc00 con 0x7fa8e8004d40 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.890+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e800be00 con 0x7fa9040c59d0 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.891+0000 7fa901400640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa8e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.891+0000 7fa901e00640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 0x7fa8e8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.891+0000 7fa901400640 1 -- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 msgr2=0x7fa8e8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.891+0000 7fa901400640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 0x7fa8e8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.891+0000 7fa901400640 1 -- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e8004d40 msgr2=0x7fa8e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.891+0000 7fa901400640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e8004d40 0x7fa8e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.891+0000 7fa901400640 1 -- 172.21.15.27:0/3803807406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e8003520 con 0x7fa8e8003f80 2024-03-28T19:01:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.891+0000 7fa901e00640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9040c59d0 0x7fa8e8003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.892+0000 7fa901400640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa8e8004800 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0001720 tx=0x7fa8e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82055714d11cfa56 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.893+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0001370 con 0x7fa8e8003f80 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.893+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f00052d0 con 0x7fa8e8003f80 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.893+0000 7fa900a00640 1 -- 172.21.15.27:0/3803807406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0003830 con 0x7fa8e8003f80 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.893+0000 7fa903200640 1 -- 172.21.15.27:0/3803807406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 msgr2=0x7fa8e8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.893+0000 7fa903200640 1 --2- 172.21.15.27:0/3803807406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa8e8004800 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0001720 tx=0x7fa8e8005830 comp rx=0 tx=0).stop 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.893+0000 7fa902800640 1 -- 172.21.15.27:0/3803807406 reap_dead start 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.894+0000 7fa903200640 1 -- 172.21.15.27:0/3803807406 shutdown_connections 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.894+0000 7fa903200640 1 -- 172.21.15.27:0/3803807406 >> 172.21.15.27:0/3803807406 conn(0x7fa9040ac910 msgr2=0x7fa9040accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.894+0000 7fa903200640 1 -- 172.21.15.27:0/3803807406 shutdown_connections 2024-03-28T19:01:40.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.894+0000 7fa903200640 1 -- 172.21.15.27:0/3803807406 wait complete. 2024-03-28T19:01:40.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.896+0000 7fa903200640 1 Processor -- start 2024-03-28T19:01:40.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.897+0000 7fa903200640 1 -- start start 2024-03-28T19:01:40.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.897+0000 7fa903200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa9041eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:40.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.897+0000 7fa903200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9041536e0 0x7fa90414b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:40.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa903200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa90414b6f0 0x7fa9040bf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:40.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa903200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9040c5db0 con 0x7fa8e8003f80 2024-03-28T19:01:40.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa903200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa904159090 con 0x7fa9041536e0 2024-03-28T19:01:40.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa903200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9040c5f30 con 0x7fa90414b6f0 2024-03-28T19:01:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa9041eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa9041eebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40310/0 (socket says 172.21.15.27:40310) 2024-03-28T19:01:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901e00640 1 -- 172.21.15.27:0/2816784730 learned_addr learned my addr 172.21.15.27:0/2816784730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901400640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9041536e0 0x7fa90414b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901e00640 1 -- 172.21.15.27:0/2816784730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa90414b6f0 msgr2=0x7fa9040bf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901e00640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa90414b6f0 0x7fa9040bf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901e00640 1 -- 172.21.15.27:0/2816784730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9041536e0 msgr2=0x7fa90414b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901e00640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9041536e0 0x7fa90414b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901e00640 1 -- 172.21.15.27:0/2816784730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9040be9b0 con 0x7fa8e8003f80 2024-03-28T19:01:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.898+0000 7fa901400640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9041536e0 0x7fa90414b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.899+0000 7fa901e00640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa9041eebd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa8f40108e0 tx=0x7fa8f4010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.899+0000 7fa8fa000640 1 -- 172.21.15.27:0/2816784730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f400f040 con 0x7fa8e8003f80 2024-03-28T19:01:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.899+0000 7fa8fa000640 1 -- 172.21.15.27:0/2816784730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f400ebc0 con 0x7fa8e8003f80 2024-03-28T19:01:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.899+0000 7fa8fa000640 1 -- 172.21.15.27:0/2816784730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f401c400 con 0x7fa8e8003f80 2024-03-28T19:01:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.899+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9041ef110 con 0x7fa8e8003f80 2024-03-28T19:01:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.899+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa904151700 con 0x7fa8e8003f80 2024-03-28T19:01:40.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.901+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa904067dd0 con 0x7fa8e8003f80 2024-03-28T19:01:40.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.904+0000 7fa8fa000640 1 -- 172.21.15.27:0/2816784730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa8f4005990 con 0x7fa8e8003f80 2024-03-28T19:01:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.905+0000 7fa8fa000640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8e406bc00 0x7fa8e406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.905+0000 7fa8fa000640 1 -- 172.21.15.27:0/2816784730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7239+0+0 (secure 0 0 0) 0x7fa8f408ba00 con 0x7fa8e8003f80 2024-03-28T19:01:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.905+0000 7fa901400640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8e406bc00 0x7fa8e406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:40.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.906+0000 7fa901400640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8e406bc00 0x7fa8e406e0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0033000 tx=0x7fa8f0009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:40.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:40.909+0000 7fa8fa000640 1 -- 172.21.15.27:0/2816784730 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa8f4018030 con 0x7fa8e8003f80 2024-03-28T19:01:41.082 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:01:41.082+0000 7fd4eaa00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:01:41.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.201+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa904069640 con 0x7fa8e8003f80 2024-03-28T19:01:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.202+0000 7fa8fa000640 1 -- 172.21.15.27:0/2816784730 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5461 (secure 0 0 0) 0x7fa8f4059ed0 con 0x7fa8e8003f80 2024-03-28T19:01:41.202 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:41.202 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 54 2024-03-28T19:01:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.205+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8e406bc00 msgr2=0x7fa8e406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.205+0000 7fa903200640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8e406bc00 0x7fa8e406e0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0033000 tx=0x7fa8f0009030 comp rx=0 tx=0).stop 2024-03-28T19:01:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.205+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 msgr2=0x7fa9041eebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.205+0000 7fa903200640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa9041eebd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa8f40108e0 tx=0x7fa8f4010da0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.206+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 shutdown_connections 2024-03-28T19:01:41.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.206+0000 7fa903200640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa8e406bc00 0x7fa8e406e0c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.206+0000 7fa903200640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa90414b6f0 0x7fa9040bf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.206+0000 7fa903200640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9041536e0 0x7fa90414b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.206+0000 7fa903200640 1 --2- 172.21.15.27:0/2816784730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8e8003f80 0x7fa9041eebd0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.206+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 >> 172.21.15.27:0/2816784730 conn(0x7fa9040ac910 msgr2=0x7fa9041584e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:41.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.206+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 shutdown_connections 2024-03-28T19:01:41.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.206+0000 7fa903200640 1 -- 172.21.15.27:0/2816784730 wait complete. 2024-03-28T19:01:41.228 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5236 2024-03-28T19:01:41.228 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before reviving mds.d 2024-03-28T19:01:41.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:41.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.617+0000 7f1aeb600640 1 Processor -- start 2024-03-28T19:01:41.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.617+0000 7f1aeb600640 1 -- start start 2024-03-28T19:01:41.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.619+0000 7f1aeb600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ae406cf80 0x7f1ae40c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:41.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.619+0000 7f1aeb600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae4159c60 con 0x7f1ae4152e70 2024-03-28T19:01:41.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.619+0000 7f1aeb600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae40c29d0 con 0x7f1ae4158ce0 2024-03-28T19:01:41.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.619+0000 7f1aeb600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae40c0fb0 con 0x7f1ae406cf80 2024-03-28T19:01:41.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.619+0000 7f1ae3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ae406cf80 0x7f1ae40c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:41.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.619+0000 7f1ae8c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1ae4152e70 0x7f1ae40c22c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36590/0 (socket says 172.21.15.27:36590) 2024-03-28T19:01:41.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.619+0000 7f1ae3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ae406cf80 0x7f1ae40c34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40316/0 (socket says 172.21.15.27:40316) 2024-03-28T19:01:41.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.619+0000 7f1ae8c00640 1 -- 172.21.15.27:0/2882163371 learned_addr learned my addr 172.21.15.27:0/2882163371 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:41.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.620+0000 7f1ae3e00640 1 -- 172.21.15.27:0/2882163371 >> v1:172.21.15.155:6789/0 conn(0x7f1ae4158ce0 legacy=0x7f1ae4159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.620+0000 7f1ae3e00640 1 -- 172.21.15.27:0/2882163371 >> v1:172.21.15.27:6789/0 conn(0x7f1ae4152e70 legacy=0x7f1ae40c22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.620+0000 7f1ae3e00640 1 -- 172.21.15.27:0/2882163371 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae406d770 con 0x7f1ae406cf80 2024-03-28T19:01:41.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.620+0000 7f1ae3400640 1 -- 172.21.15.27:0/2882163371 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898189578 0 0) 0x7f1ae4159c60 con 0x7f1ae4152e70 2024-03-28T19:01:41.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.622+0000 7f1ae3e00640 1 --2- 172.21.15.27:0/2882163371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae40c34b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1ad00097a0 tx=0x7f1ad002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb33155d40cfb94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.622+0000 7f1ae3400640 1 -- 172.21.15.27:0/2882163371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0035020 con 0x7f1ae406cf80 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.622+0000 7f1ae3400640 1 -- 172.21.15.27:0/2882163371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0031830 con 0x7f1ae406cf80 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.622+0000 7f1ae3400640 1 -- 172.21.15.27:0/2882163371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0031c00 con 0x7f1ae406cf80 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.623+0000 7f1aeb600640 1 -- 172.21.15.27:0/2882163371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 msgr2=0x7f1ae40c34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.623+0000 7f1aeb600640 1 --2- 172.21.15.27:0/2882163371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae40c34b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1ad00097a0 tx=0x7f1ad002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.623+0000 7f1aeb600640 1 -- 172.21.15.27:0/2882163371 shutdown_connections 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.623+0000 7f1aeb600640 1 --2- 172.21.15.27:0/2882163371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae40c34b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.623+0000 7f1aeb600640 1 -- 172.21.15.27:0/2882163371 >> 172.21.15.27:0/2882163371 conn(0x7f1ae406b0c0 msgr2=0x7f1ae406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.623+0000 7f1aeb600640 1 -- 172.21.15.27:0/2882163371 shutdown_connections 2024-03-28T19:01:41.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.623+0000 7f1aeb600640 1 -- 172.21.15.27:0/2882163371 wait complete. 2024-03-28T19:01:41.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.625+0000 7f1aeb600640 1 Processor -- start 2024-03-28T19:01:41.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.625+0000 7f1aeb600640 1 -- start start 2024-03-28T19:01:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1aeb600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae414d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1aeb600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ae4152e70 0x7f1ae4155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1aeb600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ae4158ce0 0x7f1ae41559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:41.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1aeb600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae40c2e30 con 0x7f1ae406cf80 2024-03-28T19:01:41.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1aeb600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae40c1290 con 0x7f1ae4158ce0 2024-03-28T19:01:41.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1aeb600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae40c2fb0 con 0x7f1ae4152e70 2024-03-28T19:01:41.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1ae8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae414d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:41.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1ae8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae414d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40320/0 (socket says 172.21.15.27:40320) 2024-03-28T19:01:41.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.626+0000 7f1ae8c00640 1 -- 172.21.15.27:0/2687617752 learned_addr learned my addr 172.21.15.27:0/2687617752 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae9600640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ae4158ce0 0x7f1ae41559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae3e00640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ae4152e70 0x7f1ae4155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae8c00640 1 -- 172.21.15.27:0/2687617752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ae4152e70 msgr2=0x7f1ae4155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae8c00640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ae4152e70 0x7f1ae4155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae8c00640 1 -- 172.21.15.27:0/2687617752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ae4158ce0 msgr2=0x7f1ae41559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae8c00640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ae4158ce0 0x7f1ae41559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae8c00640 1 -- 172.21.15.27:0/2687617752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae414dc70 con 0x7f1ae406cf80 2024-03-28T19:01:41.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae8c00640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae414d380 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800c430 tx=0x7f1ad800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:41.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.627+0000 7f1ae0c00640 1 -- 172.21.15.27:0/2687617752 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad80074e0 con 0x7f1ae406cf80 2024-03-28T19:01:41.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.628+0000 7f1ae0c00640 1 -- 172.21.15.27:0/2687617752 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad80047e0 con 0x7f1ae406cf80 2024-03-28T19:01:41.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.628+0000 7f1ae0c00640 1 -- 172.21.15.27:0/2687617752 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8004bd0 con 0x7f1ae406cf80 2024-03-28T19:01:41.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.628+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae414cd60 con 0x7f1ae406cf80 2024-03-28T19:01:41.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.628+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae414c510 con 0x7f1ae406cf80 2024-03-28T19:01:41.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.630+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae404f400 con 0x7f1ae406cf80 2024-03-28T19:01:41.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.630+0000 7f1ae0c00640 1 -- 172.21.15.27:0/2687617752 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1ad8011070 con 0x7f1ae406cf80 2024-03-28T19:01:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.632+0000 7f1ae0c00640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ac406bbd0 0x7f1ac406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.632+0000 7f1ae0c00640 1 -- 172.21.15.27:0/2687617752 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7239+0+0 (secure 0 0 0) 0x7f1ad808ad40 con 0x7f1ae406cf80 2024-03-28T19:01:41.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.632+0000 7f1ae3e00640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ac406bbd0 0x7f1ac406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:41.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.633+0000 7f1ae3e00640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ac406bbd0 0x7f1ac406e090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0036000 tx=0x7f1ad0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:41.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.637+0000 7f1ae0c00640 1 -- 172.21.15.27:0/2687617752 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1ae414dc70 con 0x7f1ae406cf80 2024-03-28T19:01:41.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.911+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ae40c09d0 con 0x7f1ae406cf80 2024-03-28T19:01:41.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.912+0000 7f1ae0c00640 1 -- 172.21.15.27:0/2687617752 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5461 (secure 0 0 0) 0x7f1ad8059220 con 0x7f1ae406cf80 2024-03-28T19:01:41.911 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:41.911 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:41.912 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 54 2024-03-28T19:01:41.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ac406bbd0 msgr2=0x7f1ac406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ac406bbd0 0x7f1ac406e090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0036000 tx=0x7f1ad0008030 comp rx=0 tx=0).stop 2024-03-28T19:01:41.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 msgr2=0x7f1ae414d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:41.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae414d380 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800c430 tx=0x7f1ad800c8f0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 shutdown_connections 2024-03-28T19:01:41.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ac406bbd0 0x7f1ac406e090 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ae4158ce0 0x7f1ae41559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ae4152e70 0x7f1ae4155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.914+0000 7f1aeb600640 1 --2- 172.21.15.27:0/2687617752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ae406cf80 0x7f1ae414d380 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:41.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.915+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 >> 172.21.15.27:0/2687617752 conn(0x7f1ae406b0c0 msgr2=0x7f1ae40c3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:41.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.915+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 shutdown_connections 2024-03-28T19:01:41.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:41.915+0000 7f1aeb600640 1 -- 172.21.15.27:0/2687617752 wait complete. 2024-03-28T19:01:41.933 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:01:42.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/13' 2024-03-28T19:01:42.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/33' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/17' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/35' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './186/37' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './186' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/19' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/15' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/31' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/36' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/20' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/12' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/39' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/10' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/34' 2024-03-28T19:01:42.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/38' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/11' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/16' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/32' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/30' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/14' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/18' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/13' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/33' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/17' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/35' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './187/37' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './187' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/19' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/15' 2024-03-28T19:01:42.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/31' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/36' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/20' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/12' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/39' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/10' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/34' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/38' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/11' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/16' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/32' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/30' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/14' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/18' 2024-03-28T19:01:42.665 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/13' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/33' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/17' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/35' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './188/37' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './188' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/19' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/15' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/31' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/36' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/20' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/12' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/39' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/10' 2024-03-28T19:01:42.666 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/34' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/38' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/11' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/16' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/32' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/30' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/14' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/18' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/13' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/33' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/17' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/35' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './189/37' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './189' 2024-03-28T19:01:42.667 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/19' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/15' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/31' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/36' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/20' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/12' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/39' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/10' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/34' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/38' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/11' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/16' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/32' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/30' 2024-03-28T19:01:42.668 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/14' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/18' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/13' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/33' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/17' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/35' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './190/37' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './190' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/19' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/15' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/31' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/36' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/20' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/12' 2024-03-28T19:01:42.669 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/39' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/10' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/34' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/38' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/11' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/16' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/32' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/30' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/14' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/18' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/13' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/33' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/17' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/35' 2024-03-28T19:01:42.670 INFO:tasks.workunit.client.1.smithi155.stdout:removed './191/37' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './191' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/19' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/15' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/31' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/36' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/20' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/12' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/39' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/10' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/34' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/38' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/11' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/16' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/32' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/30' 2024-03-28T19:01:42.671 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/14' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/18' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/13' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/33' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/17' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/35' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './192/37' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './192' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/19' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/15' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/31' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/36' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/20' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/12' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/39' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/10' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/34' 2024-03-28T19:01:42.672 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/38' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/11' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/16' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/32' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/30' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/14' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/18' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/13' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/33' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/17' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/35' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './193/37' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './193' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/19' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/15' 2024-03-28T19:01:42.673 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/31' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/36' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/20' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/12' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/39' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/10' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/34' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/38' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/11' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/16' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/32' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/30' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/14' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/18' 2024-03-28T19:01:42.674 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/13' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/33' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/17' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/35' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './194/37' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './194' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/19' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/15' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/31' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/36' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/20' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/12' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/39' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/10' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/34' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/38' 2024-03-28T19:01:42.675 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/11' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/16' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/32' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/30' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/14' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/18' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/13' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/33' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/17' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/35' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './195/37' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './195' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/19' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/15' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/31' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/36' 2024-03-28T19:01:42.676 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/20' 2024-03-28T19:01:42.677 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/12' 2024-03-28T19:01:42.677 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/39' 2024-03-28T19:01:42.677 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/10' 2024-03-28T19:01:43.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.181+0000 7fa447e00640 1 Processor -- start 2024-03-28T19:01:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.181+0000 7fa447e00640 1 -- start start 2024-03-28T19:01:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa447e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa440150b60 0x7fa4400ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa447e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4400c8470 0x7fa440156b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa447e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4400ad680 con 0x7fa4400c8c70 2024-03-28T19:01:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa447e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4400ad9b0 con 0x7fa4400c8470 2024-03-28T19:01:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa447e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4400c4de0 con 0x7fa440150b60 2024-03-28T19:01:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa444a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4400c8470 0x7fa440156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa445e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa4400c8c70 0x7fa4400c3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36602/0 (socket says 172.21.15.27:36602) 2024-03-28T19:01:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa445e00640 1 -- 172.21.15.27:0/3008428386 learned_addr learned my addr 172.21.15.27:0/3008428386 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.183+0000 7fa444a00640 1 --2- 172.21.15.27:0/3008428386 >> v2:172.21.15.27:3300/0 conn(0x7fa4400c8470 0x7fa440156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40326/0 (socket says 172.21.15.27:40326) 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.184+0000 7fa445400640 1 --2- 172.21.15.27:0/3008428386 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa440150b60 0x7fa4400ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.184+0000 7fa444a00640 1 -- 172.21.15.27:0/3008428386 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa440150b60 msgr2=0x7fa4400ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.184+0000 7fa444a00640 1 --2- 172.21.15.27:0/3008428386 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa440150b60 0x7fa4400ade40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.184+0000 7fa444a00640 1 -- 172.21.15.27:0/3008428386 >> v1:172.21.15.27:6789/0 conn(0x7fa4400c8c70 legacy=0x7fa4400c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.184+0000 7fa444a00640 1 -- 172.21.15.27:0/3008428386 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4400c50c0 con 0x7fa4400c8470 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.184+0000 7fa43fe00640 1 -- 172.21.15.27:0/3008428386 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333646694 0 0) 0x7fa4400ad680 con 0x7fa4400c8c70 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.186+0000 7fa444a00640 1 --2- 172.21.15.27:0/3008428386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4400c8470 0x7fa440156b00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0097a0 tx=0x7fa42c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e441b1df9c721 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.186+0000 7fa43fe00640 1 -- 172.21.15.27:0/3008428386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c0315a0 con 0x7fa4400c8470 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.186+0000 7fa43fe00640 1 -- 172.21.15.27:0/3008428386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c037070 con 0x7fa4400c8470 2024-03-28T19:01:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.186+0000 7fa43fe00640 1 -- 172.21.15.27:0/3008428386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c031960 con 0x7fa4400c8470 2024-03-28T19:01:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.187+0000 7fa447e00640 1 -- 172.21.15.27:0/3008428386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4400c8470 msgr2=0x7fa440156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.187+0000 7fa447e00640 1 --2- 172.21.15.27:0/3008428386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4400c8470 0x7fa440156b00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0097a0 tx=0x7fa42c02b840 comp rx=0 tx=0).stop 2024-03-28T19:01:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.187+0000 7fa447e00640 1 -- 172.21.15.27:0/3008428386 shutdown_connections 2024-03-28T19:01:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.187+0000 7fa447e00640 1 --2- 172.21.15.27:0/3008428386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4400c8470 0x7fa440156b00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.187+0000 7fa447e00640 1 --2- 172.21.15.27:0/3008428386 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa440150b60 0x7fa4400ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.187+0000 7fa447e00640 1 -- 172.21.15.27:0/3008428386 >> 172.21.15.27:0/3008428386 conn(0x7fa4400aaab0 msgr2=0x7fa4400aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:44.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.187+0000 7fa447e00640 1 -- 172.21.15.27:0/3008428386 shutdown_connections 2024-03-28T19:01:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.187+0000 7fa447e00640 1 -- 172.21.15.27:0/3008428386 wait complete. 2024-03-28T19:01:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.189+0000 7fa447e00640 1 Processor -- start 2024-03-28T19:01:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.189+0000 7fa447e00640 1 -- start start 2024-03-28T19:01:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa447e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4400c8470 0x7fa4401ecc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa447e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4400c8c70 0x7fa44006fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa447e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa440150b60 0x7fa440155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa447e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4400a85a0 con 0x7fa440150b60 2024-03-28T19:01:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa447e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa440150f40 con 0x7fa4400c8c70 2024-03-28T19:01:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa447e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4400a8720 con 0x7fa4400c8470 2024-03-28T19:01:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa445e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa440150b60 0x7fa440155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa445e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa440150b60 0x7fa440155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40334/0 (socket says 172.21.15.27:40334) 2024-03-28T19:01:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa445e00640 1 -- 172.21.15.27:0/2943302132 learned_addr learned my addr 172.21.15.27:0/2943302132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa445e00640 1 -- 172.21.15.27:0/2943302132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4400c8470 msgr2=0x7fa4401ecc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa445e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4400c8470 0x7fa4401ecc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.190+0000 7fa444a00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4400c8c70 0x7fa44006fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa445e00640 1 -- 172.21.15.27:0/2943302132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4400c8c70 msgr2=0x7fa44006fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa445e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4400c8c70 0x7fa44006fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa445e00640 1 -- 172.21.15.27:0/2943302132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4401536a0 con 0x7fa440150b60 2024-03-28T19:01:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa444a00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4400c8c70 0x7fa44006fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:01:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa445e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa440150b60 0x7fa440155630 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa43800c7a0 tx=0x7fa43800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa43d600640 1 -- 172.21.15.27:0/2943302132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa438018070 con 0x7fa440150b60 2024-03-28T19:01:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa43d600640 1 -- 172.21.15.27:0/2943302132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43800a640 con 0x7fa440150b60 2024-03-28T19:01:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4401e9740 con 0x7fa440150b60 2024-03-28T19:01:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.191+0000 7fa43d600640 1 -- 172.21.15.27:0/2943302132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43800aab0 con 0x7fa440150b60 2024-03-28T19:01:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.192+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4401ec460 con 0x7fa440150b60 2024-03-28T19:01:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.193+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44004f400 con 0x7fa440150b60 2024-03-28T19:01:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.194+0000 7fa43d600640 1 -- 172.21.15.27:0/2943302132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa43801b070 con 0x7fa440150b60 2024-03-28T19:01:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.195+0000 7fa43d600640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa42006bc00 0x7fa42006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.195+0000 7fa43d600640 1 -- 172.21.15.27:0/2943302132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7239+0+0 (secure 0 0 0) 0x7fa43808a6a0 con 0x7fa440150b60 2024-03-28T19:01:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.195+0000 7fa445400640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa42006bc00 0x7fa42006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.196+0000 7fa445400640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa42006bc00 0x7fa42006e0c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa4340098f0 tx=0x7fa434008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:44.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.200+0000 7fa43d600640 1 -- 172.21.15.27:0/2943302132 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa438058b80 con 0x7fa440150b60 2024-03-28T19:01:44.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.494+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4400c4a20 con 0x7fa440150b60 2024-03-28T19:01:44.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.495+0000 7fa43d600640 1 -- 172.21.15.27:0/2943302132 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5461 (secure 0 0 0) 0x7fa438058480 con 0x7fa440150b60 2024-03-28T19:01:44.494 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:44.495 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T18:59:54.159296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6836/181571966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":181571966},{"type":"v1","addr":"172.21.15.27:6836","nonce":181571966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:44.495 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 54 2024-03-28T19:01:44.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa42006bc00 msgr2=0x7fa42006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:44.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa42006bc00 0x7fa42006e0c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa4340098f0 tx=0x7fa434008040 comp rx=0 tx=0).stop 2024-03-28T19:01:44.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa440150b60 msgr2=0x7fa440155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:44.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa440150b60 0x7fa440155630 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa43800c7a0 tx=0x7fa43800cc60 comp rx=0 tx=0).stop 2024-03-28T19:01:44.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 shutdown_connections 2024-03-28T19:01:44.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa42006bc00 0x7fa42006e0c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa440150b60 0x7fa440155630 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4400c8c70 0x7fa44006fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 --2- 172.21.15.27:0/2943302132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4400c8470 0x7fa4401ecc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:44.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 >> 172.21.15.27:0/2943302132 conn(0x7fa4400aaab0 msgr2=0x7fa4400aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:44.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.498+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 shutdown_connections 2024-03-28T19:01:44.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:44.499+0000 7fa447e00640 1 -- 172.21.15.27:0/2943302132 wait complete. 2024-03-28T19:01:44.517 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/34' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/38' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/11' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/16' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/32' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/30' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/14' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/18' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/13' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/33' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/17' 2024-03-28T19:01:45.008 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/35' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './196/37' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './196' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/19' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/15' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/31' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/36' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/20' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/12' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/39' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/10' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/34' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/38' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/11' 2024-03-28T19:01:45.009 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/16' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/32' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/30' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/14' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/18' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/13' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/33' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/17' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/35' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './197/37' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './197' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/19' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/15' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/31' 2024-03-28T19:01:45.010 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/36' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/20' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/12' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/39' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/10' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/34' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/38' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/11' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/16' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/32' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/30' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/14' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/18' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/13' 2024-03-28T19:01:45.011 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/33' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/17' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/35' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './198/37' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './198' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/19' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/15' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/31' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/36' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/20' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/12' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/39' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/10' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/34' 2024-03-28T19:01:45.012 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/38' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/11' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/16' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/32' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/30' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/14' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/18' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/13' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/33' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/17' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/35' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './199/37' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './199' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/21' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/27' 2024-03-28T19:01:45.013 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/31' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/36' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/20' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/39' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/34' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/38' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/28' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/24' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/29' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/32' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/22' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/25' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/30' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/33' 2024-03-28T19:01:45.014 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/23' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/35' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/26' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './200/37' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './200' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/21' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/27' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/31' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/36' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/20' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/39' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/34' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/38' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/28' 2024-03-28T19:01:45.015 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/24' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/29' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/32' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/22' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/25' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/30' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/33' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/23' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/35' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/26' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './201/37' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './201' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/21' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/27' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/31' 2024-03-28T19:01:45.016 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/36' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/20' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/39' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/34' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/38' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/28' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/24' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/29' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/32' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/22' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/25' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/30' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/33' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/23' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/35' 2024-03-28T19:01:45.017 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/26' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './202/37' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './202' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/21' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/27' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/31' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/36' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/20' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/39' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/34' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/38' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/28' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/24' 2024-03-28T19:01:45.018 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/29' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/32' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/22' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/25' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/30' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/33' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/23' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/35' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/26' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './203/37' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './203' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/21' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/27' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/31' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/36' 2024-03-28T19:01:45.019 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/20' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/39' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/34' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/38' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/28' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/24' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/29' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/32' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/22' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/25' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/30' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/33' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/23' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/35' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/26' 2024-03-28T19:01:45.020 INFO:tasks.workunit.client.1.smithi155.stdout:removed './204/37' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './204' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/21' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/27' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/31' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/36' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/20' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/39' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/34' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/38' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/28' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/24' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/29' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/32' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/22' 2024-03-28T19:01:45.021 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/25' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/30' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/33' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/23' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/35' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/26' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './205/37' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './205' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/21' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/27' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/31' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/36' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/20' 2024-03-28T19:01:45.022 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/39' 2024-03-28T19:01:46.229 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-03-28T19:01:46.229 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T19:01:46.230 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T19:01:46.237 INFO:tasks.ceph.mds.d:Started 2024-03-28T19:01:46.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:46.416 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T19:01:46.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.477+0000 7f1c93200640 1 Processor -- start 2024-03-28T19:01:46.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.477+0000 7f1c93200640 1 -- start start 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.479+0000 7f1c93200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1c941529b0 0x7f1c94155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.479+0000 7f1c93200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c94152d90 0x7f1c940c0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.479+0000 7f1c93200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c940c2290 con 0x7f1c94073380 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.479+0000 7f1c93200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c940c09b0 con 0x7f1c94152d90 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.479+0000 7f1c93200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c94156500 con 0x7f1c941529b0 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.480+0000 7f1c91400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c94152d90 0x7f1c940c0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.480+0000 7f1c91400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c94152d90 0x7f1c940c0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34126/0 (socket says 172.21.15.27:34126) 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.480+0000 7f1c91400640 1 -- 172.21.15.27:0/3476218134 learned_addr learned my addr 172.21.15.27:0/3476218134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:46.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.480+0000 7f1c91e00640 1 --2- 172.21.15.27:0/3476218134 >> v2:172.21.15.155:3300/0 conn(0x7f1c941529b0 0x7f1c94155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.480+0000 7f1c91400640 1 -- 172.21.15.27:0/3476218134 >> v2:172.21.15.155:3300/0 conn(0x7f1c941529b0 msgr2=0x7f1c94155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:46.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.480+0000 7f1c91400640 1 --2- 172.21.15.27:0/3476218134 >> v2:172.21.15.155:3300/0 conn(0x7f1c941529b0 0x7f1c94155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.480+0000 7f1c91400640 1 -- 172.21.15.27:0/3476218134 >> v1:172.21.15.155:6789/0 conn(0x7f1c94073380 legacy=0x7f1c9406e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:46.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.480+0000 7f1c91400640 1 -- 172.21.15.27:0/3476218134 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c940c4610 con 0x7f1c94152d90 2024-03-28T19:01:46.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.482+0000 7f1c91400640 1 --2- 172.21.15.27:0/3476218134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 0x7f1c940c0c20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1c9406efb0 tx=0x7f1c8002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf57cbb29cdd442 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:46.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.482+0000 7f1c90a00640 1 -- 172.21.15.27:0/3476218134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80038070 con 0x7f1c94152d90 2024-03-28T19:01:46.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.482+0000 7f1c90a00640 1 -- 172.21.15.27:0/3476218134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8002f050 con 0x7f1c94152d90 2024-03-28T19:01:46.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.482+0000 7f1c90a00640 1 -- 172.21.15.27:0/3476218134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80031970 con 0x7f1c94152d90 2024-03-28T19:01:46.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.483+0000 7f1c93200640 1 -- 172.21.15.27:0/3476218134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 msgr2=0x7f1c940c0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.483+0000 7f1c93200640 1 --2- 172.21.15.27:0/3476218134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 0x7f1c940c0c20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1c9406efb0 tx=0x7f1c8002b7f0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.483+0000 7f1c93200640 1 -- 172.21.15.27:0/3476218134 shutdown_connections 2024-03-28T19:01:46.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.483+0000 7f1c93200640 1 --2- 172.21.15.27:0/3476218134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 0x7f1c940c0c20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.483+0000 7f1c93200640 1 --2- 172.21.15.27:0/3476218134 >> v2:172.21.15.155:3300/0 conn(0x7f1c941529b0 0x7f1c94155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.483+0000 7f1c93200640 1 -- 172.21.15.27:0/3476218134 >> 172.21.15.27:0/3476218134 conn(0x7f1c940c72e0 msgr2=0x7f1c940c76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:46.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.484+0000 7f1c93200640 1 -- 172.21.15.27:0/3476218134 shutdown_connections 2024-03-28T19:01:46.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.484+0000 7f1c93200640 1 -- 172.21.15.27:0/3476218134 wait complete. 2024-03-28T19:01:46.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.486+0000 7f1c93200640 1 Processor -- start 2024-03-28T19:01:46.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.486+0000 7f1c93200640 1 -- start start 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c93200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c94073380 0x7f1c9414bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c93200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c941529b0 0x7f1c9414d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c93200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 0x7f1c9414b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c93200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940c4330 con 0x7f1c94152d90 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c93200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940c4050 con 0x7f1c941529b0 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c93200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9406de80 con 0x7f1c94073380 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c92800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 0x7f1c9414b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c91e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c94073380 0x7f1c9414bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c91e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c94073380 0x7f1c9414bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:44472/0 (socket says 172.21.15.27:44472) 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c91e00640 1 -- 172.21.15.27:0/1920862334 learned_addr learned my addr 172.21.15.27:0/1920862334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:46.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.487+0000 7f1c91400640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c941529b0 0x7f1c9414d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.488+0000 7f1c91400640 1 -- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c94073380 msgr2=0x7f1c9414bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.488+0000 7f1c91400640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c94073380 0x7f1c9414bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.488+0000 7f1c91400640 1 -- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 msgr2=0x7f1c9414b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.488+0000 7f1c91400640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 0x7f1c9414b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.488+0000 7f1c91400640 1 -- 172.21.15.27:0/1920862334 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c94157620 con 0x7f1c941529b0 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.488+0000 7f1c91e00640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c94073380 0x7f1c9414bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.489+0000 7f1c91400640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c941529b0 0x7f1c9414d630 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1c80033000 tx=0x7f1c800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.489+0000 7f1c8a000640 1 -- 172.21.15.27:0/1920862334 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80042070 con 0x7f1c941529b0 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.489+0000 7f1c8a000640 1 -- 172.21.15.27:0/1920862334 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8003d870 con 0x7f1c941529b0 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.489+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c941ebd80 con 0x7f1c941529b0 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.489+0000 7f1c8a000640 1 -- 172.21.15.27:0/1920862334 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80038040 con 0x7f1c941529b0 2024-03-28T19:01:46.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.489+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c940c6d10 con 0x7f1c941529b0 2024-03-28T19:01:46.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.491+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c94067dd0 con 0x7f1c941529b0 2024-03-28T19:01:46.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.493+0000 7f1c8a000640 1 -- 172.21.15.27:0/1920862334 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1c80045070 con 0x7f1c941529b0 2024-03-28T19:01:46.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.494+0000 7f1c8a000640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c7406bbd0 0x7f1c7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.494+0000 7f1c8a000640 1 -- 172.21.15.27:0/1920862334 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7325+0+0 (secure 0 0 0) 0x7f1c800acd50 con 0x7f1c941529b0 2024-03-28T19:01:46.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.494+0000 7f1c91e00640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c7406bbd0 0x7f1c7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.495+0000 7f1c91e00640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c7406bbd0 0x7f1c7406e090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1c84006fd0 tx=0x7f1c84008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.509+0000 7f1c8a000640 1 -- 172.21.15.27:0/1920862334 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1c800b0030 con 0x7f1c941529b0 2024-03-28T19:01:46.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:46.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.780+0000 7f6656600640 1 Processor -- start 2024-03-28T19:01:46.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.780+0000 7f6656600640 1 -- start start 2024-03-28T19:01:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.781+0000 7f6656600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6650150c70 0x7f66500a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.782+0000 7f6656600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66500c3af0 con 0x7f66500c3690 2024-03-28T19:01:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.782+0000 7f6656600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6650154020 con 0x7f66501534c0 2024-03-28T19:01:46.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.782+0000 7f6656600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500ab4e0 con 0x7f6650150c70 2024-03-28T19:01:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.782+0000 7f664f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6650150c70 0x7f66500a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.782+0000 7f664f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6650150c70 0x7f66500a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34156/0 (socket says 172.21.15.27:34156) 2024-03-28T19:01:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.782+0000 7f664ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f66500c3690 0x7f66500c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40270/0 (socket says 172.21.15.27:40270) 2024-03-28T19:01:46.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.782+0000 7f664f400640 1 -- 172.21.15.27:0/1695596911 learned_addr learned my addr 172.21.15.27:0/1695596911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.783+0000 7f664f400640 1 -- 172.21.15.27:0/1695596911 >> v1:172.21.15.155:6789/0 conn(0x7f66501534c0 legacy=0x7f6650153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.783+0000 7f664f400640 1 -- 172.21.15.27:0/1695596911 >> v1:172.21.15.27:6789/0 conn(0x7f66500c3690 legacy=0x7f66500c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.783+0000 7f664f400640 1 -- 172.21.15.27:0/1695596911 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500c3d90 con 0x7f6650150c70 2024-03-28T19:01:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.784+0000 7f664f400640 1 --2- 172.21.15.27:0/1695596911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6650150c70 0x7f66500a9cb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f66501541f0 tx=0x7f664002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71e620c5af73414 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.785+0000 7f664e000640 1 -- 172.21.15.27:0/1695596911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640034070 con 0x7f6650150c70 2024-03-28T19:01:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.785+0000 7f664e000640 1 -- 172.21.15.27:0/1695596911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640030840 con 0x7f6650150c70 2024-03-28T19:01:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.785+0000 7f664e000640 1 -- 172.21.15.27:0/1695596911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640030c10 con 0x7f6650150c70 2024-03-28T19:01:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.786+0000 7f6656600640 1 -- 172.21.15.27:0/1695596911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6650150c70 msgr2=0x7f66500a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.786+0000 7f6656600640 1 --2- 172.21.15.27:0/1695596911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6650150c70 0x7f66500a9cb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f66501541f0 tx=0x7f664002b980 comp rx=0 tx=0).stop 2024-03-28T19:01:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.786+0000 7f6656600640 1 -- 172.21.15.27:0/1695596911 shutdown_connections 2024-03-28T19:01:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.786+0000 7f6656600640 1 --2- 172.21.15.27:0/1695596911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6650150c70 0x7f66500a9cb0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.786+0000 7f6656600640 1 -- 172.21.15.27:0/1695596911 >> 172.21.15.27:0/1695596911 conn(0x7f665006b9d0 msgr2=0x7f66500c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.786+0000 7f6656600640 1 -- 172.21.15.27:0/1695596911 shutdown_connections 2024-03-28T19:01:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.786+0000 7f6656600640 1 -- 172.21.15.27:0/1695596911 wait complete. 2024-03-28T19:01:46.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.789+0000 7f6656600640 1 Processor -- start 2024-03-28T19:01:46.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.789+0000 7f6656600640 1 -- start start 2024-03-28T19:01:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f6656600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66500c3690 0x7f66501ec900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f6656600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6650150c70 0x7f6650148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f6656600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66501534c0 0x7f6650144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f6656600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500ab9f0 con 0x7f66501534c0 2024-03-28T19:01:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f6656600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500ac1f0 con 0x7f66500c3690 2024-03-28T19:01:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f6656600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500abcd0 con 0x7f6650150c70 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f664f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66500c3690 0x7f66501ec900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f664f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66500c3690 0x7f66501ec900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:55384/0 (socket says 172.21.15.27:55384) 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f664f400640 1 -- 172.21.15.27:0/3232629751 learned_addr learned my addr 172.21.15.27:0/3232629751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.791+0000 7f664ea00640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6650150c70 0x7f6650148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.792+0000 7f664fe00640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66501534c0 0x7f6650144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.792+0000 7f664f400640 1 -- 172.21.15.27:0/3232629751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6650150c70 msgr2=0x7f6650148670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.792+0000 7f664f400640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6650150c70 0x7f6650148670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.792+0000 7f664f400640 1 -- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66501534c0 msgr2=0x7f6650144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.792+0000 7f664f400640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66501534c0 0x7f6650144940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.792+0000 7f664f400640 1 -- 172.21.15.27:0/3232629751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66501ecfe0 con 0x7f66500c3690 2024-03-28T19:01:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.792+0000 7f664fe00640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66501534c0 0x7f6650144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:01:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.793+0000 7f664f400640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66500c3690 0x7f66501ec900 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f664002b9c0 tx=0x7f6640004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.793+0000 7f6647400640 1 -- 172.21.15.27:0/3232629751 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640048070 con 0x7f66500c3690 2024-03-28T19:01:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.793+0000 7f6647400640 1 -- 172.21.15.27:0/3232629751 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6640030640 con 0x7f66500c3690 2024-03-28T19:01:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.793+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66501ed2c0 con 0x7f66500c3690 2024-03-28T19:01:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.793+0000 7f6647400640 1 -- 172.21.15.27:0/3232629751 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640034040 con 0x7f66500c3690 2024-03-28T19:01:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.793+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6650148bb0 con 0x7f66500c3690 2024-03-28T19:01:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.795+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66501ed7e0 con 0x7f66500c3690 2024-03-28T19:01:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.796+0000 7f6647400640 1 -- 172.21.15.27:0/3232629751 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f664004b070 con 0x7f66500c3690 2024-03-28T19:01:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.797+0000 7f6647400640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f663006bbd0 0x7f663006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.797+0000 7f6647400640 1 -- 172.21.15.27:0/3232629751 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7325+0+0 (secure 0 0 0) 0x7f66400ad240 con 0x7f66500c3690 2024-03-28T19:01:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.798+0000 7f664ea00640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f663006bbd0 0x7f663006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.799+0000 7f664ea00640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f663006bbd0 0x7f663006e090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66500c4f90 tx=0x7f663c000ed0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.803+0000 7f6647400640 1 -- 172.21.15.27:0/3232629751 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f66501ed7e0 con 0x7f66500c3690 2024-03-28T19:01:46.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.870+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c94069640 con 0x7f1c941529b0 2024-03-28T19:01:46.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.871+0000 7f1c8a000640 1 -- 172.21.15.27:0/1920862334 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4667 (secure 0 0 0) 0x7f1c940c2290 con 0x7f1c941529b0 2024-03-28T19:01:46.870 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:46.871 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:44.667001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:46.871 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 55 2024-03-28T19:01:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.874+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c7406bbd0 msgr2=0x7f1c7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.874+0000 7f1c93200640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c7406bbd0 0x7f1c7406e090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1c84006fd0 tx=0x7f1c84008040 comp rx=0 tx=0).stop 2024-03-28T19:01:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.875+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c941529b0 msgr2=0x7f1c9414d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:46.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.875+0000 7f1c93200640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c941529b0 0x7f1c9414d630 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1c80033000 tx=0x7f1c800046c0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.876+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 shutdown_connections 2024-03-28T19:01:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.876+0000 7f1c93200640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c7406bbd0 0x7f1c7406e090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.876+0000 7f1c93200640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c94152d90 0x7f1c9414b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.876+0000 7f1c93200640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c941529b0 0x7f1c9414d630 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.876+0000 7f1c93200640 1 --2- 172.21.15.27:0/1920862334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c94073380 0x7f1c9414bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.876+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 >> 172.21.15.27:0/1920862334 conn(0x7f1c940c72e0 msgr2=0x7f1c9406e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.876+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 shutdown_connections 2024-03-28T19:01:46.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:46.876+0000 7f1c93200640 1 -- 172.21.15.27:0/1920862334 wait complete. 2024-03-28T19:01:46.896 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-03-28T19:01:47.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.089+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66501541f0 con 0x7f66500c3690 2024-03-28T19:01:47.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.090+0000 7f6647400640 1 -- 172.21.15.27:0/3232629751 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5464 (secure 0 0 0) 0x7f66501541f0 con 0x7f66500c3690 2024-03-28T19:01:47.090 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:47.090 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:46.871381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:reconnect","state_seq":96,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:47.090 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 56 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f663006bbd0 msgr2=0x7f663006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f663006bbd0 0x7f663006e090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66500c4f90 tx=0x7f663c000ed0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66500c3690 msgr2=0x7f66501ec900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66500c3690 0x7f66501ec900 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f664002b9c0 tx=0x7f6640004730 comp rx=0 tx=0).stop 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 shutdown_connections 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f663006bbd0 0x7f663006e090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66501534c0 0x7f6650144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6650150c70 0x7f6650148670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 --2- 172.21.15.27:0/3232629751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66500c3690 0x7f66501ec900 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 >> 172.21.15.27:0/3232629751 conn(0x7f665006b9d0 msgr2=0x7f6650153910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:47.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 shutdown_connections 2024-03-28T19:01:47.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.093+0000 7f6656600640 1 -- 172.21.15.27:0/3232629751 wait complete. 2024-03-28T19:01:47.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/34' 2024-03-28T19:01:47.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/38' 2024-03-28T19:01:47.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/28' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/24' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/29' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/32' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/22' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/25' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/30' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/33' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/23' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/35' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/26' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './206/37' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './206' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/21' 2024-03-28T19:01:47.103 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/27' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/31' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/36' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/20' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/39' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/34' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/38' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/28' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/24' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/29' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/32' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/22' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/25' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/30' 2024-03-28T19:01:47.104 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/33' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/23' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/35' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/26' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './207/37' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './207' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/21' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/27' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/31' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/36' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/20' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/39' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/34' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/38' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/28' 2024-03-28T19:01:47.105 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/24' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/29' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/32' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/22' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/25' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/30' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/33' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/23' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/35' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/26' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './208/37' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './208' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/21' 2024-03-28T19:01:47.106 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/27' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/31' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/36' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/20' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/39' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/34' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/38' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/28' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/24' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/29' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/32' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/22' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/25' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/30' 2024-03-28T19:01:47.107 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/33' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/23' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/35' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/26' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './209/37' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './209' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/21' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/27' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/31' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/36' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/20' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/39' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/34' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/38' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/28' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/24' 2024-03-28T19:01:47.108 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/29' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/32' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/22' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/25' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/30' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/33' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/23' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/35' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/26' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './210/37' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './210' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/21' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/27' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/31' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/36' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/20' 2024-03-28T19:01:47.109 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/39' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/34' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/38' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/28' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/24' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/29' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/32' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/22' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/25' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/30' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/33' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/23' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/35' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/26' 2024-03-28T19:01:47.110 INFO:tasks.workunit.client.1.smithi155.stdout:removed './211/37' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './211' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/21' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/27' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/31' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/36' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/20' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/39' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/34' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/38' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/28' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/24' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/29' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/32' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/22' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/25' 2024-03-28T19:01:47.111 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/30' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/33' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/23' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/35' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/26' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './212/37' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './212' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/21' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/27' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/31' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/36' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/20' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/39' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/34' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/38' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/28' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/24' 2024-03-28T19:01:47.112 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/29' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/32' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/22' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/25' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/30' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/33' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/23' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/35' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/26' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './213/37' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './213' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/21' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/27' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/31' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/36' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/20' 2024-03-28T19:01:47.113 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/39' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/34' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/38' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/28' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/24' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/29' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/32' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/22' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/25' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/30' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/33' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/23' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/35' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/26' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './214/37' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './214' 2024-03-28T19:01:47.114 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/21' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/27' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/31' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/36' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/20' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/39' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/34' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/38' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/28' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/24' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/29' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/32' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/22' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/25' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/30' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/33' 2024-03-28T19:01:47.115 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/23' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/35' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/26' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './215/37' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './215' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/21' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/27' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/31' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/36' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/20' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/39' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/34' 2024-03-28T19:01:47.116 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/38' 2024-03-28T19:01:47.117 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-03-28T19:01:47.117 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:01:47.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:47.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.352+0000 7f3f60000640 1 Processor -- start 2024-03-28T19:01:47.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.352+0000 7f3f60000640 1 -- start start 2024-03-28T19:01:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.353+0000 7f3f60000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f580a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.353+0000 7f3f60000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f580c3690 0x7f3f580c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.353+0000 7f3f60000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3f580c6660 0x7f3f581534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.353+0000 7f3f60000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f580ab4e0 con 0x7f3f580c3690 2024-03-28T19:01:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.353+0000 7f3f60000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f580ac1f0 con 0x7f3f580c6660 2024-03-28T19:01:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.353+0000 7f3f60000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580c3af0 con 0x7f3f58150c70 2024-03-28T19:01:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f580c3690 0x7f3f580c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f580c3690 0x7f3f580c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34178/0 (socket says 172.21.15.27:34178) 2024-03-28T19:01:47.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5cc00640 1 -- 172.21.15.27:0/2698499559 learned_addr learned my addr 172.21.15.27:0/2698499559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:47.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5d600640 1 --2- 172.21.15.27:0/2698499559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f580a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:47.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5e000640 1 --2- 172.21.15.27:0/2698499559 >> v2:172.21.15.155:3300/0 conn(0x7f3f580c6660 0x7f3f581534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5cc00640 1 -- 172.21.15.27:0/2698499559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 msgr2=0x7f3f580a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5cc00640 1 --2- 172.21.15.27:0/2698499559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f580a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5cc00640 1 -- 172.21.15.27:0/2698499559 >> v2:172.21.15.155:3300/0 conn(0x7f3f580c6660 msgr2=0x7f3f581534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5cc00640 1 --2- 172.21.15.27:0/2698499559 >> v2:172.21.15.155:3300/0 conn(0x7f3f580c6660 0x7f3f581534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.354+0000 7f3f5cc00640 1 -- 172.21.15.27:0/2698499559 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f580c3d90 con 0x7f3f580c3690 2024-03-28T19:01:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.356+0000 7f3f5cc00640 1 --2- 172.21.15.27:0/2698499559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c3690 0x7f3f580c6120 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3f5806c6a0 tx=0x7f3f4402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7237dfff9c02b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.356+0000 7f3f57e00640 1 -- 172.21.15.27:0/2698499559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f440317c0 con 0x7f3f580c3690 2024-03-28T19:01:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.356+0000 7f3f57e00640 1 -- 172.21.15.27:0/2698499559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44037070 con 0x7f3f580c3690 2024-03-28T19:01:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.356+0000 7f3f57e00640 1 -- 172.21.15.27:0/2698499559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44031b80 con 0x7f3f580c3690 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 -- 172.21.15.27:0/2698499559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c3690 msgr2=0x7f3f580c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 --2- 172.21.15.27:0/2698499559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c3690 0x7f3f580c6120 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3f5806c6a0 tx=0x7f3f4402b860 comp rx=0 tx=0).stop 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 -- 172.21.15.27:0/2698499559 shutdown_connections 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 --2- 172.21.15.27:0/2698499559 >> v2:172.21.15.155:3300/0 conn(0x7f3f580c6660 0x7f3f581534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 --2- 172.21.15.27:0/2698499559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c3690 0x7f3f580c6120 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 --2- 172.21.15.27:0/2698499559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f580a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 -- 172.21.15.27:0/2698499559 >> 172.21.15.27:0/2698499559 conn(0x7f3f5806b9d0 msgr2=0x7f3f580c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 -- 172.21.15.27:0/2698499559 shutdown_connections 2024-03-28T19:01:47.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.357+0000 7f3f60000640 1 -- 172.21.15.27:0/2698499559 wait complete. 2024-03-28T19:01:47.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.359+0000 7f3f60000640 1 Processor -- start 2024-03-28T19:01:47.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.359+0000 7f3f60000640 1 -- start start 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f60000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f580c3690 0x7f3f58157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f60000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c6660 0x7f3f580c0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f60000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f58153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f60000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580abcd0 con 0x7f3f580c6660 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f60000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580ab9f0 con 0x7f3f580c3690 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f60000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580c1810 con 0x7f3f58150c70 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f5cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c6660 0x7f3f580c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f5cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c6660 0x7f3f580c0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34190/0 (socket says 172.21.15.27:34190) 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f5cc00640 1 -- 172.21.15.27:0/2598754641 learned_addr learned my addr 172.21.15.27:0/2598754641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.360+0000 7f3f5d600640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f580c3690 0x7f3f58157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:47.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.361+0000 7f3f5e000640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f58153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.361+0000 7f3f5cc00640 1 -- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 msgr2=0x7f3f58153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.361+0000 7f3f5cc00640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f58153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.361+0000 7f3f5cc00640 1 -- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f580c3690 msgr2=0x7f3f58157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.361+0000 7f3f5cc00640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f580c3690 0x7f3f58157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.361+0000 7f3f5cc00640 1 -- 172.21.15.27:0/2598754641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f58157850 con 0x7f3f580c6660 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.361+0000 7f3f5e000640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f58153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.362+0000 7f3f5cc00640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c6660 0x7f3f580c0120 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009870 tx=0x7f3f440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.362+0000 7f3f55600640 1 -- 172.21.15.27:0/2598754641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44032650 con 0x7f3f580c6660 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.362+0000 7f3f55600640 1 -- 172.21.15.27:0/2598754641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f440327f0 con 0x7f3f580c6660 2024-03-28T19:01:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.362+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f58155c00 con 0x7f3f580c6660 2024-03-28T19:01:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.362+0000 7f3f55600640 1 -- 172.21.15.27:0/2598754641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44032be0 con 0x7f3f580c6660 2024-03-28T19:01:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.362+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f58151e90 con 0x7f3f580c6660 2024-03-28T19:01:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.364+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f58151cb0 con 0x7f3f580c6660 2024-03-28T19:01:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.365+0000 7f3f55600640 1 -- 172.21.15.27:0/2598754641 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f4404c060 con 0x7f3f580c6660 2024-03-28T19:01:47.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.366+0000 7f3f55600640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3806bbd0 0x7f3f3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.366+0000 7f3f55600640 1 -- 172.21.15.27:0/2598754641 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7325+0+0 (secure 0 0 0) 0x7f3f440ac920 con 0x7f3f580c6660 2024-03-28T19:01:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.366+0000 7f3f5d600640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3806bbd0 0x7f3f3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.367+0000 7f3f5d600640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3806bbd0 0x7f3f3806e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c007920 tx=0x7f3f4c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:47.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.371+0000 7f3f55600640 1 -- 172.21.15.27:0/2598754641 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f58151cb0 con 0x7f3f580c6660 2024-03-28T19:01:47.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.713+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f5804f400 con 0x7f3f580c6660 2024-03-28T19:01:47.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.714+0000 7f3f55600640 1 -- 172.21.15.27:0/2598754641 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5464 (secure 0 0 0) 0x7f3f44040070 con 0x7f3f580c6660 2024-03-28T19:01:47.714 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:47.714 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:46.871381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:reconnect","state_seq":96,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:47.714 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 56 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3806bbd0 msgr2=0x7f3f3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3806bbd0 0x7f3f3806e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c007920 tx=0x7f3f4c008040 comp rx=0 tx=0).stop 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c6660 msgr2=0x7f3f580c0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c6660 0x7f3f580c0120 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009870 tx=0x7f3f440046c0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 shutdown_connections 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3806bbd0 0x7f3f3806e090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f58150c70 0x7f3f58153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f580c6660 0x7f3f580c0120 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 --2- 172.21.15.27:0/2598754641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f580c3690 0x7f3f58157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 >> 172.21.15.27:0/2598754641 conn(0x7f3f5806b9d0 msgr2=0x7f3f58150270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:47.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 shutdown_connections 2024-03-28T19:01:47.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:47.717+0000 7f3f60000640 1 -- 172.21.15.27:0/2598754641 wait complete. 2024-03-28T19:01:47.734 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4247": { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 55, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 96 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:01:46.871381+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4247 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5302": { "addr": "172.21.15.27:6837/4097237428", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 4097237428, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 4097237428, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5302, "incarnation": 50, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:01:39.837636+0000", "qdb_cluster": [ 5302 ], "qdb_leader": 5302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5302 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:01:48.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.137+0000 7f6589000640 1 Processor -- start 2024-03-28T19:01:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.137+0000 7f6589000640 1 -- start start 2024-03-28T19:01:49.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.138+0000 7f6589000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6584150c70 0x7f65840a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.138+0000 7f6589000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f65840c3690 0x7f65840c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.138+0000 7f6589000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c6660 0x7f65841534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.138+0000 7f6589000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65840ab4e0 con 0x7f6584150c70 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.138+0000 7f6589000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65840ac1f0 con 0x7f65840c3690 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.138+0000 7f6589000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65840c3af0 con 0x7f65840c6660 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6581e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6584150c70 0x7f65840a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6581e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6584150c70 0x7f65840a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34196/0 (socket says 172.21.15.27:34196) 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6581e00640 1 -- 172.21.15.27:0/2044939626 learned_addr learned my addr 172.21.15.27:0/2044939626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6582800640 1 --2- 172.21.15.27:0/2044939626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c6660 0x7f65841534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6581e00640 1 -- 172.21.15.27:0/2044939626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c6660 msgr2=0x7f65841534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6581e00640 1 --2- 172.21.15.27:0/2044939626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c6660 0x7f65841534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6581e00640 1 -- 172.21.15.27:0/2044939626 >> v2:172.21.15.155:3300/0 conn(0x7f65840c3690 msgr2=0x7f65840c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:01:49.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6581e00640 1 --2- 172.21.15.27:0/2044939626 >> v2:172.21.15.155:3300/0 conn(0x7f65840c3690 0x7f65840c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.139+0000 7f6581e00640 1 -- 172.21.15.27:0/2044939626 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65840c3d90 con 0x7f6584150c70 2024-03-28T19:01:49.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.140+0000 7f6581e00640 1 --2- 172.21.15.27:0/2044939626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f65840a9cb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f65840c4180 tx=0x7f657402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a068829059081f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:49.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.141+0000 7f6580a00640 1 -- 172.21.15.27:0/2044939626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65740317c0 con 0x7f6584150c70 2024-03-28T19:01:49.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.141+0000 7f6580a00640 1 -- 172.21.15.27:0/2044939626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574037070 con 0x7f6584150c70 2024-03-28T19:01:49.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.141+0000 7f6580a00640 1 -- 172.21.15.27:0/2044939626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574031b80 con 0x7f6584150c70 2024-03-28T19:01:49.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 -- 172.21.15.27:0/2044939626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 msgr2=0x7f65840a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 --2- 172.21.15.27:0/2044939626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f65840a9cb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f65840c4180 tx=0x7f657402b860 comp rx=0 tx=0).stop 2024-03-28T19:01:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 -- 172.21.15.27:0/2044939626 shutdown_connections 2024-03-28T19:01:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 --2- 172.21.15.27:0/2044939626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c6660 0x7f65841534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 --2- 172.21.15.27:0/2044939626 >> v2:172.21.15.155:3300/0 conn(0x7f65840c3690 0x7f65840c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 --2- 172.21.15.27:0/2044939626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f65840a9cb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 -- 172.21.15.27:0/2044939626 >> 172.21.15.27:0/2044939626 conn(0x7f658406b9d0 msgr2=0x7f65840c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 -- 172.21.15.27:0/2044939626 shutdown_connections 2024-03-28T19:01:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.142+0000 7f6589000640 1 -- 172.21.15.27:0/2044939626 wait complete. 2024-03-28T19:01:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.144+0000 7f6589000640 1 Processor -- start 2024-03-28T19:01:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.144+0000 7f6589000640 1 -- start start 2024-03-28T19:01:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.145+0000 7f6589000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c3690 0x7f6584153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.145+0000 7f6589000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c6660 0x7f6584155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.145+0000 7f6589000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f6584156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.145+0000 7f6589000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65840abcd0 con 0x7f6584150c70 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.145+0000 7f6589000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65840ab9f0 con 0x7f65840c6660 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.145+0000 7f6589000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65840c1810 con 0x7f65840c3690 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6582800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f6584156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6582800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f6584156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34212/0 (socket says 172.21.15.27:34212) 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6582800640 1 -- 172.21.15.27:0/3616112267 learned_addr learned my addr 172.21.15.27:0/3616112267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6581e00640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c3690 0x7f6584153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6581400640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c6660 0x7f6584155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6582800640 1 -- 172.21.15.27:0/3616112267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c3690 msgr2=0x7f6584153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6582800640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c3690 0x7f6584153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6582800640 1 -- 172.21.15.27:0/3616112267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c6660 msgr2=0x7f6584155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:49.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6582800640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c6660 0x7f6584155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.146+0000 7f6582800640 1 -- 172.21.15.27:0/3616112267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584154840 con 0x7f6584150c70 2024-03-28T19:01:49.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.147+0000 7f6582800640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f6584156030 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f657c00cda0 tx=0x7f657c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:49.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.147+0000 7f657a000640 1 -- 172.21.15.27:0/3616112267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c003b90 con 0x7f6584150c70 2024-03-28T19:01:49.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.147+0000 7f657a000640 1 -- 172.21.15.27:0/3616112267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c003d30 con 0x7f6584150c70 2024-03-28T19:01:49.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.147+0000 7f657a000640 1 -- 172.21.15.27:0/3616112267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c004240 con 0x7f6584150c70 2024-03-28T19:01:49.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.147+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6584154b20 con 0x7f6584150c70 2024-03-28T19:01:49.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.147+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65841550e0 con 0x7f6584150c70 2024-03-28T19:01:49.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.149+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658404f400 con 0x7f6584150c70 2024-03-28T19:01:49.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.150+0000 7f657a000640 1 -- 172.21.15.27:0/3616112267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f657c0043e0 con 0x7f6584150c70 2024-03-28T19:01:49.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.151+0000 7f657a000640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bc00 0x7f656406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:49.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.151+0000 7f657a000640 1 -- 172.21.15.27:0/3616112267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7325+0+0 (secure 0 0 0) 0x7f657c08bbd0 con 0x7f6584150c70 2024-03-28T19:01:49.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.151+0000 7f6581e00640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bc00 0x7f656406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:49.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.152+0000 7f6581e00640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bc00 0x7f656406e0c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6574036000 tx=0x7f657403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.156+0000 7f657a000640 1 -- 172.21.15.27:0/3616112267 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f657c05a060 con 0x7f6584150c70 2024-03-28T19:01:49.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.458+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65840c5f00 con 0x7f6584150c70 2024-03-28T19:01:49.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.459+0000 7f657a000640 1 -- 172.21.15.27:0/3616112267 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5467 (secure 0 0 0) 0x7f657c059960 con 0x7f6584150c70 2024-03-28T19:01:49.459 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:49.459 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:47.970960+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":98,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:49.459 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 58 2024-03-28T19:01:49.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.463+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bc00 msgr2=0x7f656406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:49.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.463+0000 7f6589000640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bc00 0x7f656406e0c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6574036000 tx=0x7f657403d040 comp rx=0 tx=0).stop 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.463+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 msgr2=0x7f6584156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.463+0000 7f6589000640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f6584156030 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f657c00cda0 tx=0x7f657c00a660 comp rx=0 tx=0).stop 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.464+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 shutdown_connections 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.464+0000 7f6589000640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bc00 0x7f656406e0c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.464+0000 7f6589000640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6584150c70 0x7f6584156030 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.464+0000 7f6589000640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c6660 0x7f6584155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.464+0000 7f6589000640 1 --2- 172.21.15.27:0/3616112267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c3690 0x7f6584153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.464+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 >> 172.21.15.27:0/3616112267 conn(0x7f658406b9d0 msgr2=0x7f6584072580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.464+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 shutdown_connections 2024-03-28T19:01:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.464+0000 7f6589000640 1 -- 172.21.15.27:0/3616112267 wait complete. 2024-03-28T19:01:49.483 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-03-28T19:01:49.483 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/28' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/24' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/29' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/32' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/22' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/25' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/30' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/33' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/23' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/35' 2024-03-28T19:01:49.522 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/26' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './216/37' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './216' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/21' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/27' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/31' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/36' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/20' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/39' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/34' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/38' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/28' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/24' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/29' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/32' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/22' 2024-03-28T19:01:49.523 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/25' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/30' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/33' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/23' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/35' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/26' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './217/37' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './217' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/21' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/27' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/31' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/36' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/20' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/39' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/34' 2024-03-28T19:01:49.524 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/38' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/28' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/24' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/29' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/32' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/22' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/25' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/30' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/33' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/23' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/35' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/26' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './218/37' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './218' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/21' 2024-03-28T19:01:49.525 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/27' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/31' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/36' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/20' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/39' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/34' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/38' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/28' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/24' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/29' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/32' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/22' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/25' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/30' 2024-03-28T19:01:49.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/33' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/23' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/35' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/26' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './219/37' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './219' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/21' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/27' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/31' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/36' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/20' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/39' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/34' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/38' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/28' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/24' 2024-03-28T19:01:49.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/29' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/32' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/22' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/25' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/30' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/33' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/23' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/35' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/26' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './220/37' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './220' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/21' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/27' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/31' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/36' 2024-03-28T19:01:49.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/20' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/39' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/34' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/38' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/28' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/24' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/29' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/32' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/22' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/25' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/30' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/33' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/23' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/35' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/26' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './221/37' 2024-03-28T19:01:49.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './221' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/21' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/27' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/31' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/36' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/20' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/39' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/34' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/38' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/28' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/24' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/29' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/32' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/22' 2024-03-28T19:01:49.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/25' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/30' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/33' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/23' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/35' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/26' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './222/37' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './222' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/21' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/27' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/31' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/36' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/20' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/39' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/34' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/38' 2024-03-28T19:01:49.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/28' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/24' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/29' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/32' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/22' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/25' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/30' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/33' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/23' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/35' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/26' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './223/37' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './223' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/21' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/27' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/31' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/36' 2024-03-28T19:01:49.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/20' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/39' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/34' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/38' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/28' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/24' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/29' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/32' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/22' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/25' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/30' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/33' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/23' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/35' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/26' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './224/37' 2024-03-28T19:01:49.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './224' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/21' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/27' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/31' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/36' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/20' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/39' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/34' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/38' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/28' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/24' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/29' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/32' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/22' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/25' 2024-03-28T19:01:49.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/30' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/33' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/23' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/35' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/26' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './225/37' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './225' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/21' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/27' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/31' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/36' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/20' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/39' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/34' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/38' 2024-03-28T19:01:49.535 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/28' 2024-03-28T19:01:49.536 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/24' 2024-03-28T19:01:49.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:49.941 INFO:tasks.workunit.client.0.smithi027.stderr:+ ln foo/x foo/xx 2024-03-28T19:01:49.957 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat foo/.snap/ss/x 2024-03-28T19:01:49.967 INFO:tasks.workunit.client.0.smithi027.stdout:x 2024-03-28T19:01:49.967 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir foo/.snap/ss 2024-03-28T19:01:49.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.989+0000 7f53c2e00640 1 Processor -- start 2024-03-28T19:01:50.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.989+0000 7f53c2e00640 1 -- start start 2024-03-28T19:01:50.069 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo a 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.990+0000 7f53c2e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53bc153bd0 0x7f53bc0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.990+0000 7f53c2e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc154020 con 0x7f53bc0a9cb0 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.990+0000 7f53c2e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0c3d90 con 0x7f53bc150c70 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.990+0000 7f53c2e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0ab4e0 con 0x7f53bc153bd0 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.991+0000 7f53c0e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53bc153bd0 0x7f53bc0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.991+0000 7f53c0e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53bc153bd0 0x7f53bc0c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34214/0 (socket says 172.21.15.27:34214) 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.991+0000 7f53c0e00640 1 -- 172.21.15.27:0/3228891564 learned_addr learned my addr 172.21.15.27:0/3228891564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.991+0000 7f53c0e00640 1 -- 172.21.15.27:0/3228891564 >> v1:172.21.15.155:6789/0 conn(0x7f53bc150c70 legacy=0x7f53bc0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.992+0000 7f53c0e00640 1 -- 172.21.15.27:0/3228891564 >> v1:172.21.15.27:6789/0 conn(0x7f53bc0a9cb0 legacy=0x7f53bc1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.992+0000 7f53c0e00640 1 -- 172.21.15.27:0/3228891564 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0ab710 con 0x7f53bc153bd0 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.993+0000 7f53c0e00640 1 --2- 172.21.15.27:0/3228891564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc153bd0 0x7f53bc0c3690 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009e50 tx=0x7f53b402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d849c6f44b7fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.994+0000 7f53baa00640 1 -- 172.21.15.27:0/3228891564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b402cb10 con 0x7f53bc153bd0 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.994+0000 7f53baa00640 1 -- 172.21.15.27:0/3228891564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b402ccb0 con 0x7f53bc153bd0 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.994+0000 7f53baa00640 1 -- 172.21.15.27:0/3228891564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b402ce60 con 0x7f53bc153bd0 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.995+0000 7f53c2e00640 1 -- 172.21.15.27:0/3228891564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc153bd0 msgr2=0x7f53bc0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.995+0000 7f53c2e00640 1 --2- 172.21.15.27:0/3228891564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc153bd0 0x7f53bc0c3690 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009e50 tx=0x7f53b402bf70 comp rx=0 tx=0).stop 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.995+0000 7f53c2e00640 1 -- 172.21.15.27:0/3228891564 shutdown_connections 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.995+0000 7f53c2e00640 1 --2- 172.21.15.27:0/3228891564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc153bd0 0x7f53bc0c3690 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.995+0000 7f53c2e00640 1 -- 172.21.15.27:0/3228891564 >> 172.21.15.27:0/3228891564 conn(0x7f53bc06b9d0 msgr2=0x7f53bc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.995+0000 7f53c2e00640 1 -- 172.21.15.27:0/3228891564 shutdown_connections 2024-03-28T19:01:50.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.995+0000 7f53c2e00640 1 -- 172.21.15.27:0/3228891564 wait complete. 2024-03-28T19:01:50.072 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo b 2024-03-28T19:01:50.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.997+0000 7f53c2e00640 1 Processor -- start 2024-03-28T19:01:50.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.997+0000 7f53c2e00640 1 -- start start 2024-03-28T19:01:50.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.998+0000 7f53c2e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53bc0a9cb0 0x7f53bc1525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.998+0000 7f53c2e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc150c70 0x7f53bc1f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.998+0000 7f53c2e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53bc153bd0 0x7f53bc1f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.998+0000 7f53c2e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0abcd0 con 0x7f53bc150c70 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.998+0000 7f53c2e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0ac1f0 con 0x7f53bc0a9cb0 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.998+0000 7f53c2e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0c1810 con 0x7f53bc153bd0 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.999+0000 7f53bb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc150c70 0x7f53bc1f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.999+0000 7f53bb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc150c70 0x7f53bc1f22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34230/0 (socket says 172.21.15.27:34230) 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.999+0000 7f53bb400640 1 -- 172.21.15.27:0/557932361 learned_addr learned my addr 172.21.15.27:0/557932361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.999+0000 7f53c0e00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53bc153bd0 0x7f53bc1f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:49.999+0000 7f53bbe00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53bc0a9cb0 0x7f53bc1525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.000+0000 7f53bb400640 1 -- 172.21.15.27:0/557932361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53bc153bd0 msgr2=0x7f53bc1f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.000+0000 7f53bb400640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53bc153bd0 0x7f53bc1f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.000+0000 7f53bb400640 1 -- 172.21.15.27:0/557932361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53bc0a9cb0 msgr2=0x7f53bc1525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.000+0000 7f53bb400640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53bc0a9cb0 0x7f53bc1525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.000+0000 7f53bb400640 1 -- 172.21.15.27:0/557932361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc154a20 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.000+0000 7f53bb400640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc150c70 0x7f53bc1f22a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f53a800c430 tx=0x7f53a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.001+0000 7f53b3e00640 1 -- 172.21.15.27:0/557932361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a80052d0 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.001+0000 7f53b3e00640 1 -- 172.21.15.27:0/557932361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a8005490 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.001+0000 7f53b3e00640 1 -- 172.21.15.27:0/557932361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8007260 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.001+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc155030 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.001+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc157120 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.003+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc04f400 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.004+0000 7f53b3e00640 1 -- 172.21.15.27:0/557932361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f53a8007400 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.005+0000 7f53b3e00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f539c06bcd0 0x7f539c06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.005+0000 7f53b3e00640 1 -- 172.21.15.27:0/557932361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7325+0+0 (secure 0 0 0) 0x7f53a808a580 con 0x7f53bc150c70 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.005+0000 7f53bbe00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f539c06bcd0 0x7f539c06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.006+0000 7f53bbe00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f539c06bcd0 0x7f539c06e190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f53ac000f60 tx=0x7f53ac009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:50.077 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir foo/.snap/s 2024-03-28T19:01:50.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.015+0000 7f53b3e00640 1 -- 172.21.15.27:0/557932361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f53a80583b0 con 0x7f53bc150c70 2024-03-28T19:01:50.077 INFO:tasks.workunit.client.0.smithi027.stderr:++ cat foo/.snap/s/a 2024-03-28T19:01:50.078 INFO:tasks.workunit.client.0.smithi027.stderr:+ r=a 2024-03-28T19:01:50.078 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' -z a ']' 2024-03-28T19:01:50.078 INFO:tasks.workunit.client.0.smithi027.stderr:+ ln foo/b foo/b2 2024-03-28T19:01:50.078 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat foo/.snap/s/b 2024-03-28T19:01:50.078 INFO:tasks.workunit.client.0.smithi027.stdout:b 2024-03-28T19:01:50.078 INFO:tasks.workunit.client.0.smithi027.stdout:this used to hang: 2024-03-28T19:01:50.079 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 'this used to hang:' 2024-03-28T19:01:50.079 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo more 2024-03-28T19:01:50.079 INFO:tasks.workunit.client.0.smithi027.stdout:oh, it didn't hang! good job. 2024-03-28T19:01:50.079 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2024-03-28T19:01:50.079 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat foo/b 2024-03-28T19:01:50.080 INFO:tasks.workunit.client.0.smithi027.stdout:b 2024-03-28T19:01:50.080 INFO:tasks.workunit.client.0.smithi027.stdout:more 2024-03-28T19:01:50.080 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir foo/.snap/s 2024-03-28T19:01:50.094 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm -r foo 2024-03-28T19:01:50.167 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-03-28T19:01:50.168 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo OK 2024-03-28T19:01:50.169 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:01:50.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T19:01:50.188 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:01:50.189+0000 7f9c3e400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:01:50.275 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-03-28T19:01:50.275 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-03-28T19:01:50.357 INFO:tasks.workunit.client.0.smithi027.stderr:+ set -e 2024-03-28T19:01:50.358 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:50.364 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/bar 2024-03-28T19:01:50.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.383+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53bc1f0f20 con 0x7f53bc150c70 2024-03-28T19:01:50.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.384+0000 7f53b3e00640 1 -- 172.21.15.27:0/557932361 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5468 (secure 0 0 0) 0x7f53a805c450 con 0x7f53bc150c70 2024-03-28T19:01:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:50.384 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:50.384 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 59 2024-03-28T19:01:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.387+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f539c06bcd0 msgr2=0x7f539c06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.387+0000 7f53c2e00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f539c06bcd0 0x7f539c06e190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f53ac000f60 tx=0x7f53ac009040 comp rx=0 tx=0).stop 2024-03-28T19:01:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.387+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc150c70 msgr2=0x7f53bc1f22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.387+0000 7f53c2e00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc150c70 0x7f53bc1f22a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f53a800c430 tx=0x7f53a800c8f0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.388+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 shutdown_connections 2024-03-28T19:01:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.388+0000 7f53c2e00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f539c06bcd0 0x7f539c06e190 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.388+0000 7f53c2e00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53bc153bd0 0x7f53bc1f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.388+0000 7f53c2e00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53bc150c70 0x7f53bc1f22a0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.388+0000 7f53c2e00640 1 --2- 172.21.15.27:0/557932361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53bc0a9cb0 0x7f53bc1525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.388+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 >> 172.21.15.27:0/557932361 conn(0x7f53bc06b9d0 msgr2=0x7f53bc0c0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:50.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.388+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 shutdown_connections 2024-03-28T19:01:50.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.388+0000 7f53c2e00640 1 -- 172.21.15.27:0/557932361 wait complete. 2024-03-28T19:01:50.388 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/foo 2024-03-28T19:01:50.405 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/foo 2024-03-28T19:01:50.408 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=5325 2024-03-28T19:01:50.408 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.a 2024-03-28T19:01:50.408 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-03-28T19:01:50.408 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:01:50.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:01:50.413 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:01:50.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:01:50.423 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:50.433 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/bar/a 2024-03-28T19:01:50.451 INFO:tasks.workunit.client.0.smithi027.stdout: File: .snap/bar/a 2024-03-28T19:01:50.451 INFO:tasks.workunit.client.0.smithi027.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-28T19:01:50.451 INFO:tasks.workunit.client.0.smithi027.stdout:Device: 29h/41d Inode: 845524441765765 Links: 2 2024-03-28T19:01:50.451 INFO:tasks.workunit.client.0.smithi027.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-28T19:01:50.451 INFO:tasks.workunit.client.0.smithi027.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-28T19:01:50.451 INFO:tasks.workunit.client.0.smithi027.stdout:Access: 2024-03-28 19:01:50.362281269 +0000 2024-03-28T19:01:50.452 INFO:tasks.workunit.client.0.smithi027.stdout:Modify: 2024-03-28 19:01:50.362281269 +0000 2024-03-28T19:01:50.452 INFO:tasks.workunit.client.0.smithi027.stdout:Change: 2024-03-28 19:01:50.362281269 +0000 2024-03-28T19:01:50.452 INFO:tasks.workunit.client.0.smithi027.stdout: Birth: - 2024-03-28T19:01:50.452 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir .snap/bar 2024-03-28T19:01:50.471 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-03-28T19:01:50.471 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo OK 2024-03-28T19:01:50.472 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:01:50.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T19:01:50.612 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:01:50.615 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-03-28T19:01:50.616 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-03-28T19:01:50.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.651+0000 7feb49e00640 1 Processor -- start 2024-03-28T19:01:50.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.652+0000 7feb49e00640 1 -- start start 2024-03-28T19:01:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.653+0000 7feb49e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7feb440c7df0 0x7feb44155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.653+0000 7feb49e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb440c7310 con 0x7feb44152f80 2024-03-28T19:01:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.654+0000 7feb49e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb44158d40 con 0x7feb440c7970 2024-03-28T19:01:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.654+0000 7feb49e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb44159020 con 0x7feb440c7df0 2024-03-28T19:01:50.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.654+0000 7feb42a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feb44152f80 0x7feb440c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40306/0 (socket says 172.21.15.27:40306) 2024-03-28T19:01:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.654+0000 7feb42a00640 1 -- 172.21.15.27:0/435297996 learned_addr learned my addr 172.21.15.27:0/435297996 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:01:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.654+0000 7feb43400640 1 --2- 172.21.15.27:0/435297996 >> v2:172.21.15.155:3300/0 conn(0x7feb440c7df0 0x7feb44155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.655+0000 7feb42000640 1 -- 172.21.15.27:0/435297996 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043422851 0 0) 0x7feb440c7310 con 0x7feb44152f80 2024-03-28T19:01:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.655+0000 7feb42000640 1 -- 172.21.15.27:0/435297996 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb28003520 con 0x7feb44152f80 2024-03-28T19:01:50.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.655+0000 7feb43400640 1 -- 172.21.15.27:0/435297996 >> v1:172.21.15.155:6789/0 conn(0x7feb440c7970 legacy=0x7feb44155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.655+0000 7feb43400640 1 -- 172.21.15.27:0/435297996 >> v1:172.21.15.27:6789/0 conn(0x7feb44152f80 legacy=0x7feb440c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.655+0000 7feb43400640 1 -- 172.21.15.27:0/435297996 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb44159300 con 0x7feb440c7df0 2024-03-28T19:01:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.657+0000 7feb43400640 1 --2- 172.21.15.27:0/435297996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7df0 0x7feb44155e40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feb3c0097a0 tx=0x7feb3c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f41165bf743c85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.658+0000 7feb42000640 1 -- 172.21.15.27:0/435297996 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c035020 con 0x7feb440c7df0 2024-03-28T19:01:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.658+0000 7feb42000640 1 -- 172.21.15.27:0/435297996 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c031830 con 0x7feb440c7df0 2024-03-28T19:01:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.658+0000 7feb42000640 1 -- 172.21.15.27:0/435297996 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c031be0 con 0x7feb440c7df0 2024-03-28T19:01:50.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.659+0000 7feb49e00640 1 -- 172.21.15.27:0/435297996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7df0 msgr2=0x7feb44155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.659+0000 7feb49e00640 1 --2- 172.21.15.27:0/435297996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7df0 0x7feb44155e40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feb3c0097a0 tx=0x7feb3c02b8d0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.659+0000 7feb49e00640 1 -- 172.21.15.27:0/435297996 shutdown_connections 2024-03-28T19:01:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.659+0000 7feb49e00640 1 --2- 172.21.15.27:0/435297996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7df0 0x7feb44155e40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.659+0000 7feb49e00640 1 -- 172.21.15.27:0/435297996 >> 172.21.15.27:0/435297996 conn(0x7feb4406b6f0 msgr2=0x7feb440caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.659+0000 7feb49e00640 1 -- 172.21.15.27:0/435297996 shutdown_connections 2024-03-28T19:01:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.659+0000 7feb49e00640 1 -- 172.21.15.27:0/435297996 wait complete. 2024-03-28T19:01:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.661+0000 7feb49e00640 1 Processor -- start 2024-03-28T19:01:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.662+0000 7feb49e00640 1 -- start start 2024-03-28T19:01:50.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.663+0000 7feb49e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7970 0x7feb440c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.663+0000 7feb49e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb440c7df0 0x7feb440c1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.663+0000 7feb49e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb44152f80 0x7feb440c22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.663+0000 7feb49e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440cbca0 con 0x7feb440c7df0 2024-03-28T19:01:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.663+0000 7feb49e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb44159b00 con 0x7feb440c7970 2024-03-28T19:01:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.663+0000 7feb49e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb440c1690 con 0x7feb44152f80 2024-03-28T19:01:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.664+0000 7feb43400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb44152f80 0x7feb440c22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.664+0000 7feb42a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7970 0x7feb440c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.664+0000 7feb43400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb44152f80 0x7feb440c22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:44552/0 (socket says 172.21.15.27:44552) 2024-03-28T19:01:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.664+0000 7feb43400640 1 -- 172.21.15.27:0/1493710376 learned_addr learned my addr 172.21.15.27:0/1493710376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:01:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.665+0000 7feb3a000640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb440c7df0 0x7feb440c1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.665+0000 7feb42a00640 1 -- 172.21.15.27:0/1493710376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb44152f80 msgr2=0x7feb440c22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.665+0000 7feb42a00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb44152f80 0x7feb440c22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.665+0000 7feb42a00640 1 -- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb440c7df0 msgr2=0x7feb440c1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.665+0000 7feb42a00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb440c7df0 0x7feb440c1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.665+0000 7feb42a00640 1 -- 172.21.15.27:0/1493710376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb440c1dc0 con 0x7feb440c7970 2024-03-28T19:01:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.666+0000 7feb42a00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7970 0x7feb440c0c10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7feb3400cae0 tx=0x7feb3400a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.666+0000 7feb3be00640 1 -- 172.21.15.27:0/1493710376 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34017070 con 0x7feb440c7970 2024-03-28T19:01:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.666+0000 7feb3be00640 1 -- 172.21.15.27:0/1493710376 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3400ac40 con 0x7feb440c7970 2024-03-28T19:01:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.666+0000 7feb3be00640 1 -- 172.21.15.27:0/1493710376 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34012620 con 0x7feb440c7970 2024-03-28T19:01:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.667+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4414d2c0 con 0x7feb440c7970 2024-03-28T19:01:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.667+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4414e5c0 con 0x7feb440c7970 2024-03-28T19:01:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.668+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4414e110 con 0x7feb440c7970 2024-03-28T19:01:50.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.670+0000 7feb3be00640 1 -- 172.21.15.27:0/1493710376 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feb3401a070 con 0x7feb440c7970 2024-03-28T19:01:50.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.671+0000 7feb3be00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb2006bc00 0x7feb2006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:01:50.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.671+0000 7feb3be00640 1 -- 172.21.15.27:0/1493710376 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7325+0+0 (secure 0 0 0) 0x7feb3408b180 con 0x7feb440c7970 2024-03-28T19:01:50.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.671+0000 7feb3a000640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb2006bc00 0x7feb2006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:01:50.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.672+0000 7feb3a000640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb2006bc00 0x7feb2006e0c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feb44158d10 tx=0x7feb2c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:01:50.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:50.677+0000 7feb3be00640 1 -- 172.21.15.27:0/1493710376 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feb34021020 con 0x7feb440c7970 2024-03-28T19:01:50.698 INFO:tasks.workunit.client.0.smithi027.stderr:+ set -e 2024-03-28T19:01:50.699 INFO:tasks.workunit.client.0.smithi027.stderr:++ seq 1 20 2024-03-28T19:01:50.702 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:50.702 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:50.708 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:50.708 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:50.736 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:50.752 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:50.757 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:51.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.040+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb44153360 con 0x7feb440c7970 2024-03-28T19:01:51.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.041+0000 7feb3be00640 1 -- 172.21.15.27:0/1493710376 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7feb44153360 con 0x7feb440c7970 2024-03-28T19:01:51.041 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:01:51.041 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:01:51.041 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb2006bc00 msgr2=0x7feb2006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb2006bc00 0x7feb2006e0c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feb44158d10 tx=0x7feb2c009040 comp rx=0 tx=0).stop 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7970 msgr2=0x7feb440c0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7970 0x7feb440c0c10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7feb3400cae0 tx=0x7feb3400a060 comp rx=0 tx=0).stop 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 shutdown_connections 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb2006bc00 0x7feb2006e0c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb44152f80 0x7feb440c22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb440c7df0 0x7feb440c1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 --2- 172.21.15.27:0/1493710376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb440c7970 0x7feb440c0c10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:01:51.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 >> 172.21.15.27:0/1493710376 conn(0x7feb4406b6f0 msgr2=0x7feb44151190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:01:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 shutdown_connections 2024-03-28T19:01:51.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:01:51.044+0000 7feb49e00640 1 -- 172.21.15.27:0/1493710376 wait complete. 2024-03-28T19:01:51.063 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-03-28T19:01:51.063 INFO:tasks.mds_thrash.fs.[a]:waiting for 56 secs before thrashing 2024-03-28T19:01:51.648 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/29' 2024-03-28T19:01:51.648 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/32' 2024-03-28T19:01:51.648 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/22' 2024-03-28T19:01:51.648 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/25' 2024-03-28T19:01:51.648 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/30' 2024-03-28T19:01:51.648 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/33' 2024-03-28T19:01:51.648 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/23' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/35' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/26' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './226/37' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './226' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/21' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/27' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/31' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/36' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/20' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/39' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/34' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/38' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/28' 2024-03-28T19:01:51.649 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/24' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/29' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/32' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/22' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/25' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/30' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/33' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/23' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/35' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/26' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './227/37' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './227' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/21' 2024-03-28T19:01:51.650 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/27' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/31' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/36' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/20' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/39' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/34' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/38' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/28' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/24' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/29' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/32' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/22' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/25' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/30' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/33' 2024-03-28T19:01:51.651 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/23' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/35' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/26' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './228/37' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './228' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/21' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/27' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/31' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/36' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/20' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/39' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/34' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/38' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/28' 2024-03-28T19:01:51.652 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/24' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/29' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/32' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/22' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/25' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/30' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/33' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/23' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/35' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/26' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './229/37' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './229' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/21' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/27' 2024-03-28T19:01:51.653 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/31' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/36' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/20' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/39' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/34' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/38' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/28' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/24' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/29' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/32' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/22' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/25' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/30' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/33' 2024-03-28T19:01:51.654 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/23' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/35' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/26' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './230/37' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './230' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/21' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/27' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/31' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/36' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/20' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/39' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/34' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/38' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/28' 2024-03-28T19:01:51.655 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/24' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/29' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/32' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/22' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/25' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/30' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/33' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/23' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/35' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/26' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './231/37' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './231' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/21' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/27' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/31' 2024-03-28T19:01:51.656 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/36' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/20' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/39' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/34' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/38' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/28' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/24' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/29' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/32' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/22' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/25' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/30' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/33' 2024-03-28T19:01:51.657 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/23' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/35' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/26' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './232/37' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './232' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/21' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/27' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/31' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/36' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/20' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/39' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/34' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/38' 2024-03-28T19:01:51.658 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/28' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/24' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/29' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/32' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/22' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/25' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/30' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/33' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/23' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/35' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/26' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed './233/37' 2024-03-28T19:01:51.659 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './233' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/21' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/27' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/31' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/36' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/20' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/39' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/34' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/38' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/28' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/24' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/29' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/32' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/22' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/25' 2024-03-28T19:01:51.660 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/30' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/33' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/23' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/35' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/26' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './234/37' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './234' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/21' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/27' 2024-03-28T19:01:51.661 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/31' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/36' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/20' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/39' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/34' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/38' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/28' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/24' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/29' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/32' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/22' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/25' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/30' 2024-03-28T19:01:51.662 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/33' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/23' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/35' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/26' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './235/37' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './235' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/21' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/27' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/31' 2024-03-28T19:01:51.663 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/36' 2024-03-28T19:01:51.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/20' 2024-03-28T19:01:51.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/39' 2024-03-28T19:01:51.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/34' 2024-03-28T19:01:51.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/38' 2024-03-28T19:01:51.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/28' 2024-03-28T19:01:51.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/24' 2024-03-28T19:01:51.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/29' 2024-03-28T19:01:51.664 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/32' 2024-03-28T19:01:53.506 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:53.533 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:53.533 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:53.533 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:53.533 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:53.533 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:53.533 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:53.533 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:53.533 INFO:tasks.workunit.client.0.smithi027.stdout: 55018 ? 00:00:00 ps 2024-03-28T19:01:53.545 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:53.586 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:53.598 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:53.609 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:53.610 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:53.623 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:53.623 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:53.642 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:53.657 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:53.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:54.006 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:54.029 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:54.029 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:54.029 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:54.029 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:54.029 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:54.029 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:54.029 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:54.029 INFO:tasks.workunit.client.0.smithi027.stdout: 55028 ? 00:00:00 ps 2024-03-28T19:01:54.038 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:54.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/22' 2024-03-28T19:01:54.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/25' 2024-03-28T19:01:54.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/30' 2024-03-28T19:01:54.066 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/33' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/23' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/35' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/26' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './236/37' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './236' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/21' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/27' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/31' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/36' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/20' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/39' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/34' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/38' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/28' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/24' 2024-03-28T19:01:54.067 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/29' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/32' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/22' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/25' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/30' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/33' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/23' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/35' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/26' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './237/37' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './237' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/21' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/27' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/31' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/36' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/20' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/39' 2024-03-28T19:01:54.068 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/34' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/38' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/28' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/24' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/29' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/32' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/22' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/25' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/30' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/33' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/23' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/35' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/26' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './238/37' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './238' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/21' 2024-03-28T19:01:54.069 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/27' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/31' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/36' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/20' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/39' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/34' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/38' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/28' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/24' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/29' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/32' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/22' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/25' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/30' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/33' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/23' 2024-03-28T19:01:54.070 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/35' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/26' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './239/37' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './239' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/21' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/27' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/31' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/36' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/20' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/39' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/34' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/38' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/28' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/24' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/29' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/32' 2024-03-28T19:01:54.071 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/22' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/25' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/30' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/33' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/23' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/35' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/26' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './240/37' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './240' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/21' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/27' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/31' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/36' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/20' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/39' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/34' 2024-03-28T19:01:54.072 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/38' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/28' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/24' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/29' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/32' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/22' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/25' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/30' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/33' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/23' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/35' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/26' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './241/37' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './241' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/21' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/27' 2024-03-28T19:01:54.073 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/31' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/36' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/20' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/39' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/34' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/38' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/28' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/24' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/29' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/32' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/22' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/25' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/30' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/33' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/23' 2024-03-28T19:01:54.074 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/35' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/26' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './242/37' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './242' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/21' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/27' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/31' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/36' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/20' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/39' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/34' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/38' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/28' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/24' 2024-03-28T19:01:54.075 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/29' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/32' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/22' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/25' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/30' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/33' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/23' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/35' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/26' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './243/37' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './243' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/21' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/27' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/31' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/36' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/20' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/39' 2024-03-28T19:01:54.076 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/34' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/38' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/28' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/24' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/29' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/32' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/22' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/25' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/30' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/33' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/23' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/35' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/26' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './244/37' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './244' 2024-03-28T19:01:54.077 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/21' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/27' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/31' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/36' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/20' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/39' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/34' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/38' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/28' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/24' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/29' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/32' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/22' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/25' 2024-03-28T19:01:54.078 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/30' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/33' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/23' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/35' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/26' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './245/37' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './245' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/21' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/27' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/31' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/36' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/20' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/39' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/34' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/38' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/28' 2024-03-28T19:01:54.079 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/24' 2024-03-28T19:01:54.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/29' 2024-03-28T19:01:54.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/32' 2024-03-28T19:01:54.080 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/22' 2024-03-28T19:01:54.080 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:54.083 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:54.099 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:54.099 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:54.114 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:54.114 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:54.142 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:54.163 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:54.169 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:54.314 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:54.332 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:54.332 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:54.333 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:54.333 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:54.333 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:54.333 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:54.333 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:54.333 INFO:tasks.workunit.client.0.smithi027.stdout: 55038 ? 00:00:00 ps 2024-03-28T19:01:54.342 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:54.374 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:54.389 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:54.402 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:54.402 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:54.413 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:54.413 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:54.439 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:54.458 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:54.462 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:54.582 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:54.604 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:54.604 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:54.604 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:54.604 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:54.604 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:54.604 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:54.604 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:54.604 INFO:tasks.workunit.client.0.smithi027.stdout: 55048 ? 00:00:00 ps 2024-03-28T19:01:54.616 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:54.646 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:54.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:54.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:54.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:54.691 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:54.692 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:54.711 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:54.732 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:54.737 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:55.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:55.038 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:55.038 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:55.039 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:55.039 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:55.039 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:55.039 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:55.039 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:55.039 INFO:tasks.workunit.client.0.smithi027.stdout: 55058 ? 00:00:00 ps 2024-03-28T19:01:55.047 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:55.082 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:55.097 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:55.146 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:55.146 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:55.160 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:55.160 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:55.180 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:55.191 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:55.199 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:55.474 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:55.494 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:55.494 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:55.494 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:55.494 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:55.494 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:55.494 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:55.494 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:55.494 INFO:tasks.workunit.client.0.smithi027.stdout: 55068 ? 00:00:00 ps 2024-03-28T19:01:55.503 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:55.532 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:55.543 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:55.553 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:55.553 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:55.562 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:55.562 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:55.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:55.599 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:55.603 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:55.767 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:55.787 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:55.787 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:55.787 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:55.787 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:55.787 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:55.787 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:55.787 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:55.787 INFO:tasks.workunit.client.0.smithi027.stdout: 55078 ? 00:00:00 ps 2024-03-28T19:01:55.795 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:55.837 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:55.844 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:55.852 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:55.852 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:55.860 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:55.860 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:55.874 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:55.885 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:55.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:56.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:56.029 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:56.029 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:56.029 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:56.029 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:56.030 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:56.030 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:56.030 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:56.030 INFO:tasks.workunit.client.0.smithi027.stdout: 55088 ? 00:00:00 ps 2024-03-28T19:01:56.038 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:56.068 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:56.078 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/25' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/30' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/33' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/23' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/35' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/26' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './246/37' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './246' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/21' 2024-03-28T19:01:56.088 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/27' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/31' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/36' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/20' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/39' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/34' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/38' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/28' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/24' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/29' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/32' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/22' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/25' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/30' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/33' 2024-03-28T19:01:56.089 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/23' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/35' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/26' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './247/37' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './247' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/21' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/27' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/31' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/36' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/20' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/39' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/34' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/38' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/28' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/24' 2024-03-28T19:01:56.090 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/29' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/32' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/22' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/25' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/30' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/33' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/23' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/35' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/26' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './248/37' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './248' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/21' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/27' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/31' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/36' 2024-03-28T19:01:56.091 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/20' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/39' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/34' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/38' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/28' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/24' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/29' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/32' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/22' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/25' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/30' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/33' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/23' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/35' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/26' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed './249/37' 2024-03-28T19:01:56.092 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './249' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/21' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/27' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/31' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/36' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/20' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/39' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/34' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/38' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/28' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/24' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/29' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/32' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/22' 2024-03-28T19:01:56.093 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/25' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/30' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/33' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/23' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/35' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/26' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './250/37' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './250' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/21' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/27' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/31' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/36' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/20' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/39' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/34' 2024-03-28T19:01:56.094 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/38' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/28' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/24' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/29' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/32' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/22' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/25' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/30' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/33' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/23' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/35' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/26' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './251/37' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './251' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/21' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/27' 2024-03-28T19:01:56.095 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/31' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/36' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/20' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/39' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/34' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/38' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/28' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/24' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/29' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/32' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/22' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/25' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/30' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/33' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/23' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/35' 2024-03-28T19:01:56.096 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/26' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './252/37' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './252' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/21' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/27' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/31' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/36' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/20' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/39' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/34' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/38' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/28' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/24' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/29' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/32' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/22' 2024-03-28T19:01:56.097 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/25' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/30' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/33' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/23' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/35' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/26' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './253/37' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './253' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/21' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/27' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/31' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/36' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/20' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/39' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/34' 2024-03-28T19:01:56.098 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/38' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/28' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/24' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/29' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/32' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/22' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/25' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/30' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/33' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/23' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/35' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/26' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './254/37' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './254' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/21' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/27' 2024-03-28T19:01:56.099 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/31' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/36' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/20' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/39' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/34' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/38' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/28' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/24' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/29' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/32' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/22' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/25' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/30' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/33' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/23' 2024-03-28T19:01:56.100 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/35' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/26' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './255/37' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './255' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/21' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/27' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/31' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/36' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/20' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/39' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/34' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/38' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/28' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/24' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/29' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/32' 2024-03-28T19:01:56.101 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/22' 2024-03-28T19:01:56.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/25' 2024-03-28T19:01:56.102 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/30' 2024-03-28T19:01:56.102 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:56.102 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:56.106 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:56.106 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:56.156 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:56.174 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:56.178 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:56.317 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:56.333 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:56.333 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:56.333 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:56.333 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:56.333 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:56.333 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:56.333 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:56.333 INFO:tasks.workunit.client.0.smithi027.stdout: 55098 ? 00:00:00 ps 2024-03-28T19:01:56.344 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:56.372 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:56.386 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:56.397 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:56.397 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:56.407 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:56.407 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:56.428 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:56.444 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:56.448 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:56.576 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:56.606 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:56.607 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:56.607 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:56.607 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:56.607 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:56.607 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:56.607 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:56.607 INFO:tasks.workunit.client.0.smithi027.stdout: 55108 ? 00:00:00 ps 2024-03-28T19:01:56.615 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:56.642 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:56.650 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:56.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:56.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:56.671 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:56.671 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:56.693 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:56.706 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:56.710 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:56.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:56.855 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:56.855 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:56.855 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:56.855 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:56.855 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:56.855 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:56.855 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:56.855 INFO:tasks.workunit.client.0.smithi027.stdout: 55118 ? 00:00:00 ps 2024-03-28T19:01:56.859 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:56.879 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:56.885 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:56.893 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:56.893 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:56.901 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:56.902 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:56.915 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:56.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:56.931 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:57.052 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:57.075 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:57.076 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:57.076 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:57.076 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:57.076 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:57.076 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:57.076 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:57.076 INFO:tasks.workunit.client.0.smithi027.stdout: 55128 ? 00:00:00 ps 2024-03-28T19:01:57.084 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:57.117 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:57.130 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:57.159 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:57.159 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:57.170 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:57.170 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:57.186 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:57.197 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:57.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:57.355 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:57.374 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:57.374 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:57.374 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:57.374 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:57.374 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:57.374 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:57.374 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:57.374 INFO:tasks.workunit.client.0.smithi027.stdout: 55138 ? 00:00:00 ps 2024-03-28T19:01:57.382 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:57.411 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:57.420 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:57.427 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:57.428 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:57.437 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:57.437 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:57.455 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:57.467 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:57.473 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:57.602 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:57.621 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:57.622 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:57.622 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:57.622 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:57.622 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:57.622 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:57.622 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:57.622 INFO:tasks.workunit.client.0.smithi027.stdout: 55148 ? 00:00:00 ps 2024-03-28T19:01:57.629 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:57.660 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:57.673 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:57.687 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:57.687 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:57.696 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:57.696 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:57.713 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:57.727 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:57.731 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:57.910 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:57.928 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:57.929 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:57.929 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:57.929 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:57.929 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:57.929 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:57.929 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:57.929 INFO:tasks.workunit.client.0.smithi027.stdout: 55158 ? 00:00:00 ps 2024-03-28T19:01:57.934 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:57.956 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:57.963 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:57.972 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:57.972 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:57.982 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:57.982 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:57.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/33' 2024-03-28T19:01:57.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/23' 2024-03-28T19:01:57.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/35' 2024-03-28T19:01:57.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/26' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './256/37' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './256' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/21' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/27' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/31' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/36' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/20' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/39' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/34' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/38' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/28' 2024-03-28T19:01:57.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/24' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/29' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/32' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/22' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/25' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/30' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/33' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/23' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/35' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/26' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './257/37' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './257' 2024-03-28T19:01:57.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/21' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/27' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/31' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/36' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/20' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/39' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/34' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/38' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/28' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/24' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/29' 2024-03-28T19:01:57.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/32' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/22' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/25' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/30' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/33' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/23' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/35' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/26' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './258/37' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './258' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/21' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/27' 2024-03-28T19:01:57.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/31' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/36' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/20' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/39' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/34' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/38' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/28' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/24' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/29' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/32' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/22' 2024-03-28T19:01:57.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/25' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/30' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/33' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/23' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/35' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/26' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './259/37' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './259' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/21' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/27' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/31' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/36' 2024-03-28T19:01:57.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/20' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/39' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/34' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/38' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/28' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/24' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/29' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/32' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/22' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/25' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/30' 2024-03-28T19:01:57.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/33' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/23' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/35' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/26' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './260/37' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './260' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/21' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/27' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/31' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/36' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/20' 2024-03-28T19:01:57.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/39' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/34' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/38' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/28' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/24' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/29' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/32' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/22' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/25' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/30' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/33' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/23' 2024-03-28T19:01:57.996 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/35' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/26' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './261/37' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './261' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/21' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/27' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/31' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/36' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/20' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/39' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/34' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/38' 2024-03-28T19:01:57.997 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/28' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/24' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/29' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/32' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/22' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/25' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/30' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/33' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/23' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/35' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/26' 2024-03-28T19:01:57.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './262/37' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './262' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/21' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/27' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/31' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/36' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/20' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/39' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/34' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/38' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/28' 2024-03-28T19:01:57.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/24' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/29' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/32' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/22' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/25' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/30' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/33' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/23' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/35' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/26' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './263/37' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './263' 2024-03-28T19:01:58.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/21' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/27' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/31' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/36' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/20' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/39' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/34' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/38' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/28' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/24' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/29' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/32' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/22' 2024-03-28T19:01:58.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/25' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/30' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/33' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/23' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/35' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/26' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './264/37' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './264' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/21' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/27' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/31' 2024-03-28T19:01:58.002 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/36' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/20' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/39' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/34' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/38' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/28' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/24' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/29' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/32' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/22' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/25' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/30' 2024-03-28T19:01:58.003 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/33' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/23' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/35' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/26' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './265/37' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './265' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/21' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/27' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/31' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/36' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/20' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/39' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/34' 2024-03-28T19:01:58.004 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/38' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/28' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/24' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/29' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/32' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/22' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/25' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/30' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/33' 2024-03-28T19:01:58.005 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/23' 2024-03-28T19:01:58.006 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:58.011 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:58.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:58.156 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:58.186 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:58.186 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:58.186 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:58.186 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:58.186 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:58.186 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:58.186 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:58.186 INFO:tasks.workunit.client.0.smithi027.stdout: 55168 ? 00:00:00 ps 2024-03-28T19:01:58.191 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:58.221 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:58.230 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:58.238 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:58.238 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:58.246 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:58.246 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:58.261 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:58.272 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:58.276 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:58.436 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:58.459 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:58.459 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:58.459 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:58.459 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:58.459 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:58.459 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:58.459 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:58.459 INFO:tasks.workunit.client.0.smithi027.stdout: 55178 ? 00:00:00 ps 2024-03-28T19:01:58.464 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:58.482 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:58.489 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:58.497 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:58.497 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:58.506 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:58.507 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:58.523 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:58.534 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:58.538 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:58.697 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:58.715 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:58.715 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:58.715 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:58.715 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:58.715 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:58.715 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:58.716 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:58.716 INFO:tasks.workunit.client.0.smithi027.stdout: 55188 ? 00:00:00 ps 2024-03-28T19:01:58.725 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:58.759 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:58.769 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:58.781 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:58.782 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:58.794 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:58.795 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:58.816 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:58.831 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:58.836 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:58.945 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:58.961 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:58.961 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:58.961 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:58.962 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:58.962 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:58.962 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:58.962 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:58.962 INFO:tasks.workunit.client.0.smithi027.stdout: 55198 ? 00:00:00 ps 2024-03-28T19:01:58.969 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:59.001 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:59.012 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:59.023 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 20` 2024-03-28T19:01:59.023 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a 2024-03-28T19:01:59.037 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/one 2024-03-28T19:01:59.038 INFO:tasks.workunit.client.0.smithi027.stderr:+ cat 2024-03-28T19:01:59.058 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir a/.snap/two 2024-03-28T19:01:59.074 INFO:tasks.workunit.client.0.smithi027.stderr:+ chmod 777 a/foo 2024-03-28T19:01:59.081 INFO:tasks.workunit.client.0.smithi027.stderr:+ sync 2024-03-28T19:01:59.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:01:59.124 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:01:59.229 INFO:tasks.workunit.client.0.smithi027.stderr:+ ps 2024-03-28T19:01:59.245 INFO:tasks.workunit.client.0.smithi027.stdout: PID TTY TIME CMD 2024-03-28T19:01:59.245 INFO:tasks.workunit.client.0.smithi027.stdout: 2975 ? 00:00:00 systemd 2024-03-28T19:01:59.245 INFO:tasks.workunit.client.0.smithi027.stdout: 2977 ? 00:00:00 (sd-pam) 2024-03-28T19:01:59.245 INFO:tasks.workunit.client.0.smithi027.stdout: 23036 ? 00:00:03 sshd 2024-03-28T19:01:59.245 INFO:tasks.workunit.client.0.smithi027.stdout: 46658 ? 00:00:00 bash 2024-03-28T19:01:59.245 INFO:tasks.workunit.client.0.smithi027.stdout: 54967 ? 00:00:00 timeout 2024-03-28T19:01:59.245 INFO:tasks.workunit.client.0.smithi027.stdout: 55007 ? 00:00:00 snaptest-double 2024-03-28T19:01:59.246 INFO:tasks.workunit.client.0.smithi027.stdout: 55226 ? 00:00:00 ps 2024-03-28T19:01:59.251 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:01:59.269 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm a/foo 2024-03-28T19:01:59.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir a 2024-03-28T19:01:59.283 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo OK 2024-03-28T19:01:59.284 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-03-28T19:01:59.284 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:01:59.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T19:01:59.401 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-03-28T19:01:59.401 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-03-28T19:01:59.484 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/foo 2024-03-28T19:01:59.500 INFO:tasks.workunit.client.0.smithi027.stdout:We want ENOENT, not ESTALE, here. 2024-03-28T19:01:59.501 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2024-03-28T19:01:59.501 INFO:tasks.workunit.client.0.smithi027.stderr:++ seq 1 100 2024-03-28T19:01:59.503 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.503 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/1 2024-03-28T19:01:59.504 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.512 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-03-28T19:01:59.512 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.513 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/2 2024-03-28T19:01:59.514 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.520 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-03-28T19:01:59.520 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.521 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/3 2024-03-28T19:01:59.521 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.529 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-03-28T19:01:59.529 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.530 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/4 2024-03-28T19:01:59.531 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.537 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-03-28T19:01:59.537 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.538 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/5 2024-03-28T19:01:59.538 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.548 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-03-28T19:01:59.548 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.550 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/6 2024-03-28T19:01:59.551 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.561 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-03-28T19:01:59.561 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.562 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/7 2024-03-28T19:01:59.563 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.570 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-03-28T19:01:59.571 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.571 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/8 2024-03-28T19:01:59.572 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.579 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-03-28T19:01:59.579 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.580 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/9 2024-03-28T19:01:59.580 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.587 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-03-28T19:01:59.588 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.589 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/10 2024-03-28T19:01:59.590 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.595 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-03-28T19:01:59.595 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.596 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/11 2024-03-28T19:01:59.597 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.603 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-03-28T19:01:59.603 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.604 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/12 2024-03-28T19:01:59.604 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.611 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-03-28T19:01:59.611 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.612 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/13 2024-03-28T19:01:59.612 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.618 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-03-28T19:01:59.618 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.619 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/14 2024-03-28T19:01:59.619 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.625 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-03-28T19:01:59.625 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.626 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/15 2024-03-28T19:01:59.627 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.632 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-03-28T19:01:59.632 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.633 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/16 2024-03-28T19:01:59.633 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.639 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-03-28T19:01:59.639 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.640 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/17 2024-03-28T19:01:59.640 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.646 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-03-28T19:01:59.646 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.647 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/18 2024-03-28T19:01:59.647 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.653 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-03-28T19:01:59.653 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.654 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/19 2024-03-28T19:01:59.654 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.660 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-03-28T19:01:59.660 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/20 2024-03-28T19:01:59.662 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.668 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-03-28T19:01:59.668 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.669 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/21 2024-03-28T19:01:59.669 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.676 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-03-28T19:01:59.677 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.678 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/22 2024-03-28T19:01:59.679 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.693 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-03-28T19:01:59.696 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.696 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.696 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/23 2024-03-28T19:01:59.708 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-03-28T19:01:59.709 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.711 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/24 2024-03-28T19:01:59.714 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.723 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-03-28T19:01:59.724 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.726 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.727 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/25 2024-03-28T19:01:59.738 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-03-28T19:01:59.739 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.741 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/26 2024-03-28T19:01:59.742 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.760 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-03-28T19:01:59.763 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.764 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.765 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/27 2024-03-28T19:01:59.779 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-03-28T19:01:59.779 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.780 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/28 2024-03-28T19:01:59.781 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.791 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-03-28T19:01:59.792 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.792 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/29 2024-03-28T19:01:59.793 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.804 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-03-28T19:01:59.806 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/30 2024-03-28T19:01:59.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.817 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-03-28T19:01:59.818 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.819 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/31 2024-03-28T19:01:59.820 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.826 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-03-28T19:01:59.826 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.827 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/32 2024-03-28T19:01:59.827 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.833 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-03-28T19:01:59.833 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.834 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/33 2024-03-28T19:01:59.835 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.850 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-03-28T19:01:59.850 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.851 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.852 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/34 2024-03-28T19:01:59.868 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-03-28T19:01:59.868 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.869 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/35 2024-03-28T19:01:59.870 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.879 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-03-28T19:01:59.879 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.880 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/36 2024-03-28T19:01:59.880 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.888 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-03-28T19:01:59.888 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.889 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/37 2024-03-28T19:01:59.890 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.897 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-03-28T19:01:59.898 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.899 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/38 2024-03-28T19:01:59.899 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.906 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-03-28T19:01:59.906 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.907 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/35' 2024-03-28T19:01:59.907 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/26' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './266/37' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './266' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/21' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/27' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/31' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/36' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/20' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/39' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/34' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/38' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/28' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/24' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/29' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/32' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/22' 2024-03-28T19:01:59.908 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/25' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/30' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/33' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/23' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/35' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/26' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './267/37' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './267' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/21' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/27' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/31' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/36' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/20' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/39' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/34' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/38' 2024-03-28T19:01:59.909 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/28' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/24' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/29' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/32' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/22' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/25' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/30' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/33' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/23' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/35' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/26' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './268/37' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './268' 2024-03-28T19:01:59.910 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/21' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/27' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/31' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/36' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/20' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/39' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/34' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/38' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/28' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/24' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/29' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/32' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/22' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/25' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/30' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/33' 2024-03-28T19:01:59.911 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/23' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/35' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/26' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './269/37' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './269' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/21' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/27' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/31' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/36' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/20' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/39' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/34' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/38' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/28' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/24' 2024-03-28T19:01:59.912 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/29' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/32' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/22' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/25' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/30' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/33' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/23' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/35' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/26' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './270/37' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './270' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/21' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/27' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/31' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/36' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/20' 2024-03-28T19:01:59.913 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/39' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/34' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/38' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/28' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/24' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/29' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/32' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/22' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/25' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/30' 2024-03-28T19:01:59.914 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/33' 2024-03-28T19:01:59.917 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/23' 2024-03-28T19:01:59.917 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/35' 2024-03-28T19:01:59.917 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/26' 2024-03-28T19:01:59.917 INFO:tasks.workunit.client.1.smithi155.stdout:removed './271/37' 2024-03-28T19:01:59.917 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './271' 2024-03-28T19:01:59.917 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/21' 2024-03-28T19:01:59.917 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/27' 2024-03-28T19:01:59.917 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/31' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/36' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/20' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/39' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/34' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/38' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/28' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/24' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/29' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/32' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/22' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/25' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/30' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/33' 2024-03-28T19:01:59.918 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/23' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/35' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/26' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './272/37' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './272' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/21' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/27' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/31' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/36' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/20' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/39' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/34' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/38' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/28' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/24' 2024-03-28T19:01:59.919 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/29' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/32' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/22' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/25' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/30' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/33' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/23' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/35' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/26' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './273/37' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './273' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/21' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/27' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/31' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/36' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/20' 2024-03-28T19:01:59.920 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/39' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/34' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/38' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/28' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/24' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/29' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/32' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/22' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/25' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/30' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/33' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/23' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/35' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/26' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './274/37' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './274' 2024-03-28T19:01:59.921 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/21' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/27' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/31' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/36' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/20' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/39' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/34' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/38' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/28' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/24' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/29' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/32' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/22' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/25' 2024-03-28T19:01:59.922 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/30' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/33' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/23' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/35' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/26' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './275/37' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './275' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/21' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/27' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/31' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/36' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/20' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/39' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/34' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/38' 2024-03-28T19:01:59.923 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/28' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/24' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/29' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/32' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/22' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/25' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/30' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/33' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/23' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/35' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/26' 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/39 2024-03-28T19:01:59.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.925 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-03-28T19:01:59.925 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.925 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/40 2024-03-28T19:01:59.925 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-03-28T19:01:59.926 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.926 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/41 2024-03-28T19:01:59.926 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.926 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-03-28T19:01:59.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/42 2024-03-28T19:01:59.927 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.931 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-03-28T19:01:59.932 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.932 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/43 2024-03-28T19:01:59.933 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.939 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-03-28T19:01:59.939 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.940 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/44 2024-03-28T19:01:59.940 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.945 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-03-28T19:01:59.945 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.946 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/45 2024-03-28T19:01:59.947 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.952 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-03-28T19:01:59.952 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/46 2024-03-28T19:01:59.953 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.959 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-03-28T19:01:59.959 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.960 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/47 2024-03-28T19:01:59.961 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.966 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-03-28T19:01:59.967 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.967 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/48 2024-03-28T19:01:59.968 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.973 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-03-28T19:01:59.973 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.974 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/49 2024-03-28T19:01:59.974 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.980 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-03-28T19:01:59.980 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.981 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/50 2024-03-28T19:01:59.982 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.986 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-03-28T19:01:59.987 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.987 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/51 2024-03-28T19:01:59.988 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:01:59.993 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-03-28T19:01:59.993 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:01:59.994 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/52 2024-03-28T19:01:59.994 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.000 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-03-28T19:02:00.000 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.001 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/53 2024-03-28T19:02:00.001 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.006 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-03-28T19:02:00.007 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/54 2024-03-28T19:02:00.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.014 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-03-28T19:02:00.014 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/55 2024-03-28T19:02:00.015 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.022 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-03-28T19:02:00.022 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.023 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/56 2024-03-28T19:02:00.024 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.031 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-03-28T19:02:00.031 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.032 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/57 2024-03-28T19:02:00.032 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.039 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-03-28T19:02:00.040 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.040 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/58 2024-03-28T19:02:00.041 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.047 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-03-28T19:02:00.048 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.048 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/59 2024-03-28T19:02:00.049 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.055 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-03-28T19:02:00.055 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/60 2024-03-28T19:02:00.056 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.063 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-03-28T19:02:00.064 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.064 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/61 2024-03-28T19:02:00.065 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.071 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-03-28T19:02:00.072 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.072 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/62 2024-03-28T19:02:00.072 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.080 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-03-28T19:02:00.080 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.081 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/63 2024-03-28T19:02:00.081 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.088 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-03-28T19:02:00.089 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.089 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/64 2024-03-28T19:02:00.090 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.097 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-03-28T19:02:00.098 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.099 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/65 2024-03-28T19:02:00.099 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.106 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-03-28T19:02:00.107 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.108 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/66 2024-03-28T19:02:00.108 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.115 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-03-28T19:02:00.115 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.116 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/67 2024-03-28T19:02:00.116 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.122 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-03-28T19:02:00.122 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.123 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/68 2024-03-28T19:02:00.123 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.129 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-03-28T19:02:00.129 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.130 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/69 2024-03-28T19:02:00.130 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.137 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-03-28T19:02:00.137 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.138 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/70 2024-03-28T19:02:00.138 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.146 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-03-28T19:02:00.147 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.148 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/71 2024-03-28T19:02:00.148 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.155 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-03-28T19:02:00.155 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.156 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/72 2024-03-28T19:02:00.157 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.164 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-03-28T19:02:00.164 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.165 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/73 2024-03-28T19:02:00.165 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.173 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-03-28T19:02:00.174 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.174 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/74 2024-03-28T19:02:00.175 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.182 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-03-28T19:02:00.183 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.184 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/75 2024-03-28T19:02:00.184 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.190 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-03-28T19:02:00.191 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/76 2024-03-28T19:02:00.192 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.199 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-03-28T19:02:00.199 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.200 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/77 2024-03-28T19:02:00.201 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.208 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-03-28T19:02:00.208 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.209 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/78 2024-03-28T19:02:00.210 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.216 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-03-28T19:02:00.217 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.218 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/79 2024-03-28T19:02:00.218 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.225 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-03-28T19:02:00.225 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.226 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/80 2024-03-28T19:02:00.226 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.234 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-03-28T19:02:00.234 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.235 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/81 2024-03-28T19:02:00.236 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.241 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-03-28T19:02:00.241 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/82 2024-03-28T19:02:00.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.248 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-03-28T19:02:00.249 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.250 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/83 2024-03-28T19:02:00.250 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.257 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-03-28T19:02:00.257 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.258 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/84 2024-03-28T19:02:00.259 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.266 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-03-28T19:02:00.267 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.268 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/85 2024-03-28T19:02:00.269 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.274 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-03-28T19:02:00.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.276 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/86 2024-03-28T19:02:00.276 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.283 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-03-28T19:02:00.283 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.284 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/87 2024-03-28T19:02:00.284 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.290 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-03-28T19:02:00.291 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.291 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/88 2024-03-28T19:02:00.292 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.299 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-03-28T19:02:00.299 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/89 2024-03-28T19:02:00.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.306 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-03-28T19:02:00.307 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.307 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/90 2024-03-28T19:02:00.308 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.314 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-03-28T19:02:00.314 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.315 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/91 2024-03-28T19:02:00.315 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.322 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-03-28T19:02:00.322 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.323 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/92 2024-03-28T19:02:00.326 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.330 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-03-28T19:02:00.330 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.331 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/93 2024-03-28T19:02:00.331 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.338 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-03-28T19:02:00.338 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.339 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/94 2024-03-28T19:02:00.339 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.346 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-03-28T19:02:00.346 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.347 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/95 2024-03-28T19:02:00.348 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.355 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-03-28T19:02:00.355 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.356 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/96 2024-03-28T19:02:00.357 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.365 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-03-28T19:02:00.365 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.366 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/97 2024-03-28T19:02:00.366 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.372 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-03-28T19:02:00.373 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.374 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/98 2024-03-28T19:02:00.374 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.380 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-03-28T19:02:00.380 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.381 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/99 2024-03-28T19:02:00.381 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.387 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-03-28T19:02:00.387 INFO:tasks.workunit.client.0.smithi027.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:00.388 INFO:tasks.workunit.client.0.smithi027.stderr:+ stat .snap/foo/100 2024-03-28T19:02:00.389 INFO:tasks.workunit.client.0.smithi027.stderr:+ grep 'No such file' 2024-03-28T19:02:00.395 INFO:tasks.workunit.client.0.smithi027.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-03-28T19:02:00.395 INFO:tasks.workunit.client.0.smithi027.stderr:+ rmdir .snap/foo 2024-03-28T19:02:00.417 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo OK 2024-03-28T19:02:00.418 INFO:tasks.workunit.client.0.smithi027.stdout:OK 2024-03-28T19:02:00.419 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:02:00.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T19:02:00.511 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-03-28T19:02:00.512 DEBUG:teuthology.orchestra.run.smithi027:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-03-28T19:02:00.585 INFO:tasks.workunit.client.0.smithi027.stderr:+ set -e 2024-03-28T19:02:00.586 INFO:tasks.workunit.client.0.smithi027.stderr:+ sudo git config --global http.sslVerify false 2024-03-28T19:02:00.645 INFO:tasks.workunit.client.0.smithi027.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-03-28T19:02:00.711 INFO:tasks.workunit.client.0.smithi027.stderr:+ retried=false 2024-03-28T19:02:00.712 INFO:tasks.workunit.client.0.smithi027.stderr:+ trap -- retry EXIT 2024-03-28T19:02:00.712 INFO:tasks.workunit.client.0.smithi027.stderr:+ rm -rf ceph 2024-03-28T19:02:00.715 INFO:tasks.workunit.client.0.smithi027.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-03-28T19:02:00.726 INFO:tasks.workunit.client.0.smithi027.stderr:Cloning into 'ceph'... 2024-03-28T19:02:01.761 INFO:tasks.workunit.client.1.smithi155.stdout:removed './276/37' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './276' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/21' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/27' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/31' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/36' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/20' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/39' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/34' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/38' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/28' 2024-03-28T19:02:01.762 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/24' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/29' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/32' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/22' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/25' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/30' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/33' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/23' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/35' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/26' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './277/37' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './277' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/21' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/27' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/31' 2024-03-28T19:02:01.763 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/36' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/20' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/39' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/34' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/38' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/28' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/24' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/29' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/32' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/22' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/25' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/30' 2024-03-28T19:02:01.764 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/33' 2024-03-28T19:02:01.765 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/23' 2024-03-28T19:02:01.765 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/35' 2024-03-28T19:02:01.765 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/26' 2024-03-28T19:02:01.765 INFO:tasks.workunit.client.1.smithi155.stdout:removed './278/37' 2024-03-28T19:02:01.765 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './278' 2024-03-28T19:02:01.765 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/21' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/27' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/31' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/36' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/20' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/39' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/34' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/38' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/28' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/24' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/29' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/32' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/22' 2024-03-28T19:02:01.766 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/25' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/30' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/33' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/23' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/35' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/26' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './279/37' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './279' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/21' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/27' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/31' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/36' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/20' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/39' 2024-03-28T19:02:01.767 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/34' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/38' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/28' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/24' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/29' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/32' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/22' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/25' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/30' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/33' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/23' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/35' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/26' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed './280/37' 2024-03-28T19:02:01.768 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './280' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/21' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/27' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/31' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/36' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/20' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/39' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/34' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/38' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/28' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/24' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/29' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/32' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/22' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/25' 2024-03-28T19:02:01.769 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/30' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/33' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/23' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/35' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/26' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './281/37' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './281' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/21' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/27' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/31' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/36' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/20' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/39' 2024-03-28T19:02:01.770 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/34' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/38' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/28' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/24' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/29' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/32' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/22' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/25' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/30' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/33' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/23' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/35' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/26' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './282/37' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './282' 2024-03-28T19:02:01.771 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/21' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/27' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/31' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/36' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/20' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/39' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/34' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/38' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/28' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/24' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/29' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/32' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/22' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/25' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/30' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/33' 2024-03-28T19:02:01.772 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/23' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/35' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/26' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './283/37' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './283' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/21' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/27' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/31' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/36' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/20' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/39' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/34' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/38' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/28' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/24' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/29' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/32' 2024-03-28T19:02:01.773 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/22' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/25' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/30' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/33' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/23' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/35' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/26' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './284/37' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './284' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/21' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/27' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/31' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/36' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/20' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/39' 2024-03-28T19:02:01.774 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/34' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/38' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/28' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/24' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/29' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/32' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/22' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/25' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/30' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/33' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/23' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/35' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/26' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './285/37' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './285' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/21' 2024-03-28T19:02:01.775 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/27' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/31' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/36' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/20' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/39' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/34' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/38' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/28' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/24' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/29' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/32' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/22' 2024-03-28T19:02:01.776 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/25' 2024-03-28T19:02:01.777 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/30' 2024-03-28T19:02:01.777 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/33' 2024-03-28T19:02:01.777 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/23' 2024-03-28T19:02:01.777 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/35' 2024-03-28T19:02:01.777 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/26' 2024-03-28T19:02:01.777 INFO:tasks.workunit.client.1.smithi155.stdout:removed './286/37' 2024-03-28T19:02:01.777 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './286' 2024-03-28T19:02:03.979 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/21' 2024-03-28T19:02:03.979 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/27' 2024-03-28T19:02:03.979 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/31' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/36' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/20' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/39' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/34' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/38' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/28' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/24' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/29' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/32' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/22' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/25' 2024-03-28T19:02:03.980 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/30' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/33' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/23' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/35' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/26' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './287/37' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './287' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/21' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/27' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/31' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/36' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/20' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/39' 2024-03-28T19:02:03.981 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/34' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/38' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/28' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/24' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/29' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/32' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/22' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/25' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/30' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/33' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/23' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/35' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/26' 2024-03-28T19:02:03.982 INFO:tasks.workunit.client.1.smithi155.stdout:removed './288/37' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './288' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/21' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/27' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/31' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/36' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/20' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/39' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/34' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/38' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/28' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/24' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/29' 2024-03-28T19:02:03.983 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/32' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/22' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/25' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/30' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/33' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/23' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/35' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/26' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './289/37' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './289' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/21' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/27' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/31' 2024-03-28T19:02:03.984 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/36' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/20' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/39' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/34' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/38' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/28' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/24' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/29' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/32' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/22' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/25' 2024-03-28T19:02:03.985 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/30' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/33' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/23' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/35' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/26' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './290/37' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './290' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/21' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/27' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/31' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/36' 2024-03-28T19:02:03.986 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/20' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/39' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/34' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/38' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/28' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/24' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/29' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/32' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/22' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/25' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/30' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/33' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/23' 2024-03-28T19:02:03.987 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/35' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/26' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './291/37' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './291' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/21' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/27' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/31' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/36' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/20' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/39' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/34' 2024-03-28T19:02:03.988 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/38' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/28' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/24' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/29' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/32' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/22' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/25' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/30' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/33' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/23' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/35' 2024-03-28T19:02:03.989 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/26' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './292/37' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './292' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/21' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/27' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/31' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/36' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/20' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/39' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/34' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/38' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/28' 2024-03-28T19:02:03.990 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/24' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/29' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/32' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/22' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/25' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/30' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/33' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/23' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/35' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/26' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './293/37' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './293' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/21' 2024-03-28T19:02:03.991 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/27' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/31' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/36' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/20' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/39' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/34' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/38' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/28' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/24' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/29' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/32' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/22' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/25' 2024-03-28T19:02:03.992 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/30' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/33' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/23' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/35' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/26' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './294/37' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './294' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/21' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/27' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/31' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/36' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/20' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/39' 2024-03-28T19:02:03.993 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/34' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/38' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/28' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/24' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/29' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/32' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/22' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/25' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/30' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/33' 2024-03-28T19:02:03.994 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/23' 2024-03-28T19:02:03.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/35' 2024-03-28T19:02:03.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/26' 2024-03-28T19:02:03.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './295/37' 2024-03-28T19:02:03.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './295' 2024-03-28T19:02:03.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/21' 2024-03-28T19:02:03.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/27' 2024-03-28T19:02:03.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/31' 2024-03-28T19:02:03.995 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/36' 2024-03-28T19:02:03.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/20' 2024-03-28T19:02:03.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/39' 2024-03-28T19:02:03.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/34' 2024-03-28T19:02:03.998 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/38' 2024-03-28T19:02:03.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/28' 2024-03-28T19:02:03.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/24' 2024-03-28T19:02:03.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/29' 2024-03-28T19:02:03.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/32' 2024-03-28T19:02:03.999 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/22' 2024-03-28T19:02:04.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/25' 2024-03-28T19:02:04.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/30' 2024-03-28T19:02:04.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/33' 2024-03-28T19:02:04.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/23' 2024-03-28T19:02:04.000 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/35' 2024-03-28T19:02:04.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/26' 2024-03-28T19:02:04.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './296/37' 2024-03-28T19:02:04.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './296' 2024-03-28T19:02:04.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/21' 2024-03-28T19:02:04.001 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/27' 2024-03-28T19:02:05.447 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/31' 2024-03-28T19:02:05.447 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/36' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/20' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/39' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/34' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/38' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/28' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/24' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/29' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/32' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/22' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/25' 2024-03-28T19:02:05.448 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/30' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/33' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/23' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/35' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/26' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './297/37' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './297' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/21' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/27' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/31' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/36' 2024-03-28T19:02:05.449 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/20' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/39' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/34' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/38' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/28' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/24' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/29' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/32' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/22' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/25' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/30' 2024-03-28T19:02:05.450 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/33' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/23' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/35' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/26' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './298/37' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './298' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/21' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/27' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/31' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/36' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/20' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/39' 2024-03-28T19:02:05.451 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/34' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/38' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/28' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/24' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/29' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/32' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/22' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/25' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/30' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/33' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/23' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/35' 2024-03-28T19:02:05.452 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/26' 2024-03-28T19:02:05.453 INFO:tasks.workunit.client.1.smithi155.stdout:removed './299/37' 2024-03-28T19:02:05.453 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './299' 2024-03-28T19:02:05.453 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/19' 2024-03-28T19:02:05.453 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/15' 2024-03-28T19:02:05.453 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/20' 2024-03-28T19:02:05.453 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/12' 2024-03-28T19:02:05.453 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/10' 2024-03-28T19:02:05.453 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/11' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/16' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/14' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/18' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/13' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/166/17' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/166' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/19' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/15' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/20' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/12' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/10' 2024-03-28T19:02:05.454 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/11' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/16' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/14' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/18' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/13' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/147/17' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/147' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/19' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/15' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/20' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/12' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/10' 2024-03-28T19:02:05.455 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/11' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/16' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/14' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/18' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/13' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/151/17' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/151' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/19' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/15' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/20' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/12' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/10' 2024-03-28T19:02:05.456 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/11' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/16' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/14' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/18' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/13' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/170/17' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/170' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/19' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/15' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/20' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/12' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/10' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/11' 2024-03-28T19:02:05.457 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/16' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/14' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/18' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/13' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/167/17' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/167' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/19' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/15' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/20' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/12' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/10' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/11' 2024-03-28T19:02:05.458 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/16' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/14' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/18' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/13' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/140/17' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/140' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/19' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/15' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/20' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/12' 2024-03-28T19:02:05.459 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/10' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/11' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/16' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/14' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/18' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/13' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/142/17' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/142' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/19' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/15' 2024-03-28T19:02:06.526 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/20' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/12' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/10' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/11' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/16' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/14' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/18' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/13' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/150/17' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/150' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/19' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/15' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/20' 2024-03-28T19:02:06.527 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/12' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/10' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/11' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/16' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/14' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/18' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/13' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/105/17' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/105' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/19' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/15' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/20' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/12' 2024-03-28T19:02:06.528 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/10' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/11' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/16' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/14' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/18' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/13' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/149/17' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/149' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/19' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/15' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/20' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/12' 2024-03-28T19:02:06.529 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/10' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/11' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/16' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/14' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/18' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/13' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/109/17' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/109' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/19' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/15' 2024-03-28T19:02:06.530 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/20' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/12' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/10' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/11' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/16' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/14' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/18' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/13' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/158/17' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/158' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/19' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/15' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/20' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/12' 2024-03-28T19:02:06.531 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/10' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/11' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/16' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/14' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/18' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/13' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/106/17' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/106' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/19' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/15' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/20' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/12' 2024-03-28T19:02:06.532 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/10' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/11' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/16' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/14' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/18' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/13' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/189/17' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/189' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/19' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/15' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/20' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/12' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/10' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/11' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/16' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/14' 2024-03-28T19:02:06.533 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/18' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/13' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/133/17' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/133' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/19' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/15' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/20' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/12' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/10' 2024-03-28T19:02:06.534 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/11' 2024-03-28T19:02:07.592 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/16' 2024-03-28T19:02:07.592 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/14' 2024-03-28T19:02:07.592 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/18' 2024-03-28T19:02:07.592 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/13' 2024-03-28T19:02:07.592 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/173/17' 2024-03-28T19:02:07.592 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/173' 2024-03-28T19:02:07.592 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/19' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/15' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/20' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/12' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/10' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/11' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/16' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/14' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/18' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/13' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/174/17' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/174' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/19' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/15' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/20' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/12' 2024-03-28T19:02:07.593 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/10' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/11' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/16' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/14' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/18' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/13' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/155/17' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/155' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/19' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/15' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/20' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/12' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/10' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/11' 2024-03-28T19:02:07.594 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/16' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/14' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/18' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/13' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/114/17' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/114' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/19' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/15' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/20' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/12' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/10' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/11' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/16' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/14' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/18' 2024-03-28T19:02:07.595 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/13' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/159/17' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/159' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/19' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/15' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/20' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/12' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/10' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/11' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/16' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/14' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/18' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/13' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/187/17' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/187' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/19' 2024-03-28T19:02:07.596 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/15' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/20' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/12' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/10' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/11' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/16' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/14' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/18' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/13' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/185/17' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/185' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/19' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/15' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/20' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/12' 2024-03-28T19:02:07.597 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/10' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/11' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/16' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/14' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/18' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/13' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/120/17' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/120' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/19' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/15' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/20' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/12' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/10' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/11' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/16' 2024-03-28T19:02:07.598 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/14' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/18' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/13' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/110/17' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/110' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/19' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/15' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/20' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/12' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/10' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/11' 2024-03-28T19:02:07.599 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/16' 2024-03-28T19:02:08.622 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/14' 2024-03-28T19:02:08.622 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/18' 2024-03-28T19:02:08.622 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/13' 2024-03-28T19:02:08.622 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/137/17' 2024-03-28T19:02:08.622 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/137' 2024-03-28T19:02:08.622 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/19' 2024-03-28T19:02:08.622 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/15' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/20' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/12' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/10' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/11' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/16' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/14' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/18' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/13' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/198/17' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/198' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/19' 2024-03-28T19:02:08.623 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/15' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/20' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/12' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/10' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/11' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/16' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/14' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/18' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/13' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/136/17' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/136' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/19' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/15' 2024-03-28T19:02:08.624 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/20' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/12' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/10' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/11' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/16' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/14' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/18' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/13' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/125/17' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/125' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/19' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/15' 2024-03-28T19:02:08.625 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/20' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/12' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/10' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/11' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/16' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/14' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/18' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/13' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/194/17' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/194' 2024-03-28T19:02:08.626 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/19' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/15' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/20' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/12' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/10' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/11' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/16' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/14' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/18' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/13' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/119/17' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/119' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/19' 2024-03-28T19:02:08.627 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/15' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/20' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/12' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/10' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/11' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/16' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/14' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/18' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/13' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/157/17' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/157' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/19' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/15' 2024-03-28T19:02:08.628 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/20' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/12' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/10' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/11' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/16' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/14' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/18' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/13' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/172/17' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/172' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/19' 2024-03-28T19:02:08.629 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/15' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/20' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/12' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/10' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/11' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/16' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/14' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/18' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/13' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/186/17' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/186' 2024-03-28T19:02:08.630 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/19' 2024-03-28T19:02:08.631 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/15' 2024-03-28T19:02:08.631 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/20' 2024-03-28T19:02:08.631 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/12' 2024-03-28T19:02:08.631 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/10' 2024-03-28T19:02:08.631 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/11' 2024-03-28T19:02:08.631 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/16' 2024-03-28T19:02:08.631 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/14' 2024-03-28T19:02:09.562 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/18' 2024-03-28T19:02:09.562 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/13' 2024-03-28T19:02:09.562 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/190/17' 2024-03-28T19:02:09.562 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/190' 2024-03-28T19:02:09.562 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/19' 2024-03-28T19:02:09.562 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/15' 2024-03-28T19:02:09.562 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/20' 2024-03-28T19:02:09.562 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/12' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/10' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/11' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/16' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/14' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/18' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/13' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/104/17' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/104' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/19' 2024-03-28T19:02:09.563 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/15' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/20' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/12' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/10' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/11' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/16' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/14' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/18' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/13' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/164/17' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/164' 2024-03-28T19:02:09.564 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/19' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/15' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/20' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/12' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/10' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/11' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/16' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/14' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/18' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/13' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/112/17' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/112' 2024-03-28T19:02:09.565 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/19' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/15' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/20' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/12' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/10' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/11' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/16' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/14' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/18' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/13' 2024-03-28T19:02:09.566 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/100/17' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/100' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/19' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/15' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/20' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/12' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/10' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/11' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/16' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/14' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/18' 2024-03-28T19:02:09.567 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/13' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/134/17' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/134' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/19' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/15' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/20' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/12' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/10' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/11' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/16' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/14' 2024-03-28T19:02:09.568 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/18' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/13' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/115/17' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/115' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/19' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/15' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/20' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/12' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/10' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/11' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/16' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/14' 2024-03-28T19:02:09.569 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/18' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/13' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/116/17' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/116' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/19' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/15' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/20' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/12' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/10' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/11' 2024-03-28T19:02:09.570 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/16' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/14' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/18' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/13' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/195/17' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/195' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/19' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/15' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/20' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/12' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/10' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/11' 2024-03-28T19:02:09.571 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/16' 2024-03-28T19:02:09.572 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/14' 2024-03-28T19:02:09.572 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/18' 2024-03-28T19:02:10.493 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/13' 2024-03-28T19:02:10.493 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/138/17' 2024-03-28T19:02:10.493 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/138' 2024-03-28T19:02:10.493 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/19' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/15' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/20' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/12' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/10' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/11' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/16' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/14' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/18' 2024-03-28T19:02:10.494 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/13' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/135/17' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/135' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/19' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/15' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/20' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/12' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/10' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/11' 2024-03-28T19:02:10.495 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/16' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/14' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/18' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/13' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/117/17' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/117' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/19' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/15' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/20' 2024-03-28T19:02:10.496 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/12' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/10' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/11' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/16' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/14' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/18' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/13' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/143/17' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/143' 2024-03-28T19:02:10.497 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/19' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/15' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/20' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/12' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/10' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/11' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/16' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/14' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/18' 2024-03-28T19:02:10.498 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/13' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/180/17' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/180' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/19' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/15' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/20' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/12' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/10' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/11' 2024-03-28T19:02:10.499 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/16' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/14' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/18' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/13' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/126/17' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/126' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/19' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/15' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/20' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/12' 2024-03-28T19:02:10.500 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/10' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/11' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/16' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/14' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/18' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/13' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/161/17' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/161' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/19' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/15' 2024-03-28T19:02:10.501 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/20' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/12' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/10' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/11' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/16' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/14' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/18' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/13' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/176/17' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/176' 2024-03-28T19:02:10.502 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/19' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/15' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/20' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/12' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/10' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/11' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/16' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/14' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/18' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/13' 2024-03-28T19:02:10.503 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/111/17' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/111' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/19' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/15' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/20' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/12' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/10' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/11' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/16' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/14' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/18' 2024-03-28T19:02:10.504 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/13' 2024-03-28T19:02:11.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/191/17' 2024-03-28T19:02:11.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/191' 2024-03-28T19:02:11.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/19' 2024-03-28T19:02:11.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/15' 2024-03-28T19:02:11.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/20' 2024-03-28T19:02:11.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/12' 2024-03-28T19:02:11.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/10' 2024-03-28T19:02:11.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/11' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/16' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/14' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/18' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/13' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/184/17' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/184' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/19' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/15' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/20' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/12' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/10' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/11' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/16' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/14' 2024-03-28T19:02:11.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/18' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/13' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/192/17' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/192' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/19' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/15' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/20' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/12' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/10' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/11' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/16' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/14' 2024-03-28T19:02:11.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/18' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/13' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/103/17' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/103' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/19' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/15' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/20' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/12' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/10' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/11' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/16' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/14' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/18' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/13' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/128/17' 2024-03-28T19:02:11.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/128' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/19' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/15' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/20' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/12' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/10' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/11' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/16' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/14' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/18' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/13' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/144/17' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/144' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/19' 2024-03-28T19:02:11.488 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/15' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/20' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/12' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/10' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/11' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/16' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/14' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/18' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/13' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/146/17' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/146' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/19' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/15' 2024-03-28T19:02:11.489 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/20' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/12' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/10' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/11' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/16' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/14' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/18' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/13' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/141/17' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/141' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/19' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/15' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/20' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/12' 2024-03-28T19:02:11.490 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/10' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/11' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/16' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/14' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/18' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/13' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/182/17' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/182' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/19' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/15' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/20' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/12' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/10' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/11' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/16' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/14' 2024-03-28T19:02:11.491 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/18' 2024-03-28T19:02:11.492 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/13' 2024-03-28T19:02:11.492 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/107/17' 2024-03-28T19:02:12.480 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/107' 2024-03-28T19:02:12.480 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/19' 2024-03-28T19:02:12.480 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/15' 2024-03-28T19:02:12.480 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/20' 2024-03-28T19:02:12.480 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/12' 2024-03-28T19:02:12.480 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/10' 2024-03-28T19:02:12.480 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/11' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/16' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/14' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/18' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/13' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/178/17' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/178' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/19' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/15' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/20' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/12' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/10' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/11' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/16' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/14' 2024-03-28T19:02:12.481 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/18' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/13' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/156/17' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/156' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/19' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/15' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/20' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/12' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/10' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/11' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/16' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/14' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/18' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/13' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/131/17' 2024-03-28T19:02:12.482 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/131' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/19' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/15' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/20' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/12' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/10' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/11' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/16' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/14' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/18' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/13' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/123/17' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/123' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/19' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/15' 2024-03-28T19:02:12.483 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/20' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/12' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/10' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/11' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/16' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/14' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/18' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/13' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/160/17' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/160' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/19' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/15' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/20' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/12' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/10' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/11' 2024-03-28T19:02:12.484 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/16' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/14' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/18' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/13' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/165/17' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/165' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/19' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/15' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/20' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/12' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/10' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/11' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/16' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/14' 2024-03-28T19:02:12.485 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/18' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/13' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/101/17' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/101' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/19' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/15' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/20' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/12' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/10' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/11' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/16' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/14' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/18' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/13' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/148/17' 2024-03-28T19:02:12.486 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/148' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/19' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/15' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/20' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/12' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/10' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/11' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/16' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/14' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/18' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/13' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/168/17' 2024-03-28T19:02:12.487 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/168' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/19' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/15' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/20' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/12' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/10' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/11' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/16' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/14' 2024-03-28T19:02:13.437 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/18' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/13' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/179/17' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/179' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/19' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/15' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/20' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/12' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/10' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/11' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/16' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/14' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/18' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/13' 2024-03-28T19:02:13.438 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/132/17' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/132' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/19' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/15' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/20' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/12' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/10' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/11' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/16' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/14' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/18' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/13' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/199/17' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/199' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/19' 2024-03-28T19:02:13.439 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/15' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/20' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/12' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/10' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/11' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/16' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/14' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/18' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/13' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/163/17' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/163' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/19' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/15' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/20' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/12' 2024-03-28T19:02:13.440 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/10' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/11' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/16' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/14' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/18' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/13' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/153/17' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/153' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/19' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/15' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/20' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/12' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/10' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/11' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/16' 2024-03-28T19:02:13.441 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/14' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/18' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/13' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/122/17' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/122' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/19' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/15' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/20' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/12' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/10' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/11' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/16' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/14' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/18' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/13' 2024-03-28T19:02:13.442 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/162/17' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/162' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/19' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/15' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/20' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/12' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/10' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/11' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/16' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/14' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/18' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/13' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/197/17' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/197' 2024-03-28T19:02:13.443 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/19' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/15' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/20' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/12' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/10' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/11' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/16' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/14' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/18' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/13' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/113/17' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/113' 2024-03-28T19:02:13.444 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/19' 2024-03-28T19:02:14.409 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/15' 2024-03-28T19:02:14.409 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/20' 2024-03-28T19:02:14.409 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/12' 2024-03-28T19:02:14.409 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/10' 2024-03-28T19:02:14.409 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/11' 2024-03-28T19:02:14.409 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/16' 2024-03-28T19:02:14.409 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/14' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/18' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/13' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/152/17' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/152' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/19' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/15' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/20' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/12' 2024-03-28T19:02:14.410 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/10' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/11' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/16' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/14' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/18' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/13' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/121/17' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/121' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/19' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/15' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/20' 2024-03-28T19:02:14.411 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/12' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/10' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/11' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/16' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/14' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/18' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/13' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/118/17' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/118' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/19' 2024-03-28T19:02:14.412 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/15' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/20' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/12' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/10' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/11' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/16' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/14' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/18' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/13' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/196/17' 2024-03-28T19:02:14.413 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/196' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/19' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/15' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/20' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/12' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/10' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/11' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/16' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/14' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/18' 2024-03-28T19:02:14.414 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/13' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/124/17' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/124' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/19' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/15' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/20' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/12' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/10' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/11' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/16' 2024-03-28T19:02:14.415 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/14' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/18' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/13' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/188/17' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/188' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/19' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/15' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/20' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/12' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/10' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/11' 2024-03-28T19:02:14.416 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/16' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/14' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/18' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/13' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/145/17' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/145' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/19' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/15' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/20' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/12' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/10' 2024-03-28T19:02:14.417 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/11' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/16' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/14' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/18' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/13' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/130/17' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/130' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/19' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/15' 2024-03-28T19:02:14.418 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/20' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/12' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/10' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/11' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/16' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/14' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/18' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/13' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/193/17' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/193' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/19' 2024-03-28T19:02:14.419 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/15' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/20' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/12' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/10' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/11' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/16' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/14' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/18' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/13' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/175/17' 2024-03-28T19:02:15.383 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/175' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/19' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/15' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/20' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/12' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/10' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/11' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/16' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/14' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/18' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/13' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/181/17' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/181' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/19' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/15' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/20' 2024-03-28T19:02:15.384 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/12' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/10' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/11' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/16' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/14' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/18' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/13' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/127/17' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/127' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/19' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/15' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/20' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/12' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/10' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/11' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/16' 2024-03-28T19:02:15.385 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/14' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/18' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/13' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/108/17' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/108' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/19' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/15' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/20' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/12' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/10' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/11' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/16' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/14' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/18' 2024-03-28T19:02:15.386 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/13' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/183/17' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/183' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/19' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/15' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/20' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/12' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/10' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/11' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/16' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/14' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/18' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/13' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/177/17' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/177' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/19' 2024-03-28T19:02:15.387 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/15' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/20' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/12' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/10' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/11' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/16' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/14' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/18' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/13' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/169/17' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/169' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/19' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/15' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/20' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/12' 2024-03-28T19:02:15.388 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/10' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/11' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/16' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/14' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/18' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/13' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/154/17' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/154' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/19' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/15' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/20' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/12' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/10' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/11' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/16' 2024-03-28T19:02:15.389 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/14' 2024-03-28T19:02:15.390 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/18' 2024-03-28T19:02:15.390 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/13' 2024-03-28T19:02:15.390 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/171/17' 2024-03-28T19:02:15.390 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/171' 2024-03-28T19:02:15.390 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/19' 2024-03-28T19:02:15.390 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/15' 2024-03-28T19:02:15.390 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/20' 2024-03-28T19:02:15.677 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/12' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/10' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/11' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/16' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/14' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/18' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/13' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/139/17' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/139' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/19' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/15' 2024-03-28T19:02:15.678 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/20' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/12' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/10' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/11' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/16' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/14' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/18' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/13' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/102/17' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/102' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/19' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/15' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/20' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/12' 2024-03-28T19:02:15.679 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/10' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/11' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/16' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/14' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/18' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/13' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed './copyofsnap1/test1/129/17' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1/129' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1/test1' 2024-03-28T19:02:15.680 INFO:tasks.workunit.client.1.smithi155.stdout:removed directory './copyofsnap1' 2024-03-28T19:02:15.681 INFO:tasks.workunit.client.1.smithi155.stdout:OK 2024-03-28T19:02:15.681 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:02:15.682 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-28T19:02:15.793 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-03-28T19:02:15.793 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-03-28T19:02:15.893 INFO:tasks.workunit.client.1.smithi155.stderr:+ set -e 2024-03-28T19:02:15.893 INFO:tasks.workunit.client.1.smithi155.stderr:+ touch foo 2024-03-28T19:02:15.900 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod +x foo 2024-03-28T19:02:15.905 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/s 2024-03-28T19:02:15.929 INFO:tasks.workunit.client.1.smithi155.stderr:+ find .snap/s/foo -executable 2024-03-28T19:02:15.930 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep foo 2024-03-28T19:02:15.951 INFO:tasks.workunit.client.1.smithi155.stdout:.snap/s/foo 2024-03-28T19:02:15.952 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir .snap/s 2024-03-28T19:02:15.999 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm foo 2024-03-28T19:02:16.007 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo OK 2024-03-28T19:02:16.007 INFO:tasks.workunit.client.1.smithi155.stdout:OK 2024-03-28T19:02:16.007 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:02:16.008 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-28T19:02:16.092 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-03-28T19:02:16.092 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-03-28T19:02:16.163 INFO:tasks.workunit.client.1.smithi155.stderr:+ set -e 2024-03-28T19:02:16.163 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir foo 2024-03-28T19:02:16.169 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo x 2024-03-28T19:02:16.174 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:16.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:16.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.713+0000 7ff624600640 1 Processor -- start 2024-03-28T19:02:16.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.714+0000 7ff624600640 1 -- start start 2024-03-28T19:02:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.715+0000 7ff624600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff61c0a9cb0 0x7ff61c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.715+0000 7ff624600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c154020 con 0x7ff61c1534c0 2024-03-28T19:02:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.715+0000 7ff624600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c0c3d90 con 0x7ff61c150c70 2024-03-28T19:02:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.715+0000 7ff624600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61c0ab4e0 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.716+0000 7ff622600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff61c1534c0 0x7ff61c153910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52090/0 (socket says 172.21.15.27:52090) 2024-03-28T19:02:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.716+0000 7ff622600640 1 -- 172.21.15.27:0/215243361 learned_addr learned my addr 172.21.15.27:0/215243361 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.716+0000 7ff621200640 1 --2- 172.21.15.27:0/215243361 >> v2:172.21.15.155:3300/0 conn(0x7ff61c0a9cb0 0x7ff61c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.716+0000 7ff621200640 1 -- 172.21.15.27:0/215243361 >> v1:172.21.15.155:6789/0 conn(0x7ff61c150c70 legacy=0x7ff61c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.716+0000 7ff621200640 1 -- 172.21.15.27:0/215243361 >> v1:172.21.15.27:6789/0 conn(0x7ff61c1534c0 legacy=0x7ff61c153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.716+0000 7ff621200640 1 -- 172.21.15.27:0/215243361 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c0ab710 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.718+0000 7ff621200640 1 --2- 172.21.15.27:0/215243361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c0c3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff6080097a0 tx=0x7ff60802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ba93eb7ba05be38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.718+0000 7ff61be00640 1 -- 172.21.15.27:0/215243361 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608035020 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.718+0000 7ff61be00640 1 -- 172.21.15.27:0/215243361 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608031830 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.718+0000 7ff61be00640 1 -- 172.21.15.27:0/215243361 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608031c00 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.719+0000 7ff624600640 1 -- 172.21.15.27:0/215243361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c0a9cb0 msgr2=0x7ff61c0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.719+0000 7ff624600640 1 --2- 172.21.15.27:0/215243361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c0c3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff6080097a0 tx=0x7ff60802b8d0 comp rx=0 tx=0).stop 2024-03-28T19:02:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.719+0000 7ff624600640 1 -- 172.21.15.27:0/215243361 shutdown_connections 2024-03-28T19:02:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.719+0000 7ff624600640 1 --2- 172.21.15.27:0/215243361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c0c3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.719+0000 7ff624600640 1 -- 172.21.15.27:0/215243361 >> 172.21.15.27:0/215243361 conn(0x7ff61c06b9d0 msgr2=0x7ff61c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.719+0000 7ff624600640 1 -- 172.21.15.27:0/215243361 shutdown_connections 2024-03-28T19:02:16.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.719+0000 7ff624600640 1 -- 172.21.15.27:0/215243361 wait complete. 2024-03-28T19:02:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.721+0000 7ff624600640 1 Processor -- start 2024-03-28T19:02:16.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.721+0000 7ff624600640 1 -- start start 2024-03-28T19:02:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.722+0000 7ff624600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c1ee770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.722+0000 7ff624600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff61c150c70 0x7ff61c1e2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.722+0000 7ff624600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c1534c0 0x7ff61c06dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.722+0000 7ff624600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0abcd0 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.722+0000 7ff624600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0ac1f0 con 0x7ff61c1534c0 2024-03-28T19:02:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.722+0000 7ff624600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0c1810 con 0x7ff61c150c70 2024-03-28T19:02:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c1ee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:16.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c1ee770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33800/0 (socket says 172.21.15.27:33800) 2024-03-28T19:02:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621c00640 1 -- 172.21.15.27:0/3398808689 learned_addr learned my addr 172.21.15.27:0/3398808689 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff622600640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c1534c0 0x7ff61c06dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621200640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff61c150c70 0x7ff61c1e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621c00640 1 -- 172.21.15.27:0/3398808689 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff61c150c70 msgr2=0x7ff61c1e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621c00640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff61c150c70 0x7ff61c1e2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621c00640 1 -- 172.21.15.27:0/3398808689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c1534c0 msgr2=0x7ff61c06dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621c00640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c1534c0 0x7ff61c06dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.723+0000 7ff621c00640 1 -- 172.21.15.27:0/3398808689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c0a7120 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.724+0000 7ff621c00640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c1ee770 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff61000c480 tx=0x7ff61000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.724+0000 7ff619600640 1 -- 172.21.15.27:0/3398808689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6100052d0 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.724+0000 7ff619600640 1 -- 172.21.15.27:0/3398808689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610005470 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.724+0000 7ff619600640 1 -- 172.21.15.27:0/3398808689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610012870 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.724+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61c0a7400 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.724+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61c0a79c0 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.726+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61c04f400 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.727+0000 7ff619600640 1 -- 172.21.15.27:0/3398808689 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff610012a10 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.728+0000 7ff619600640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5fc06bbd0 0x7ff5fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.729+0000 7ff619600640 1 -- 172.21.15.27:0/3398808689 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7353+0+0 (secure 0 0 0) 0x7ff61008ad70 con 0x7ff61c0a9cb0 2024-03-28T19:02:16.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.729+0000 7ff621200640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5fc06bbd0 0x7ff5fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:16.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.730+0000 7ff621200640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5fc06bbd0 0x7ff5fc06e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff608036000 tx=0x7ff608008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:16.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:16.735+0000 7ff619600640 1 -- 172.21.15.27:0/3398808689 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff6100591f0 con 0x7ff61c0a9cb0 2024-03-28T19:02:17.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.050+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff61c0c5f00 con 0x7ff61c0a9cb0 2024-03-28T19:02:17.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.051+0000 7ff619600640 1 -- 172.21.15.27:0/3398808689 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7ff610058af0 con 0x7ff61c0a9cb0 2024-03-28T19:02:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:17.051 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:02:17.051 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:02:17.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5fc06bbd0 msgr2=0x7ff5fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5fc06bbd0 0x7ff5fc06e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff608036000 tx=0x7ff608008030 comp rx=0 tx=0).stop 2024-03-28T19:02:17.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff61c0a9cb0 msgr2=0x7ff61c1ee770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c1ee770 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff61000c480 tx=0x7ff61000c940 comp rx=0 tx=0).stop 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 shutdown_connections 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5fc06bbd0 0x7ff5fc06e090 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff61c1534c0 0x7ff61c06dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff61c150c70 0x7ff61c1e2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 --2- 172.21.15.27:0/3398808689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff61c0a9cb0 0x7ff61c1ee770 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 >> 172.21.15.27:0/3398808689 conn(0x7ff61c06b9d0 msgr2=0x7ff61c0c47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 shutdown_connections 2024-03-28T19:02:17.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.054+0000 7ff624600640 1 -- 172.21.15.27:0/3398808689 wait complete. 2024-03-28T19:02:17.078 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-03-28T19:02:17.078 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:02:17.078 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:02:17.101 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:02:17.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.358+0000 7fd151600640 1 Processor -- start 2024-03-28T19:02:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.358+0000 7fd151600640 1 -- start start 2024-03-28T19:02:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.360+0000 7fd151600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c06c0d0 0x7fd14c06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.360+0000 7fd151600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14c0a7400 con 0x7fd14c0a6e90 2024-03-28T19:02:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.360+0000 7fd151600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14c158b40 con 0x7fd14c0c1780 2024-03-28T19:02:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.360+0000 7fd151600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c159470 con 0x7fd14c06c0d0 2024-03-28T19:02:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.360+0000 7fd14b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd14c0a6e90 0x7fd14c155530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52094/0 (socket says 172.21.15.27:52094) 2024-03-28T19:02:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.360+0000 7fd14b400640 1 -- 172.21.15.27:0/1073352104 learned_addr learned my addr 172.21.15.27:0/1073352104 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:17.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14be00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c06c0d0 0x7fd14c06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:17.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595708281 0 0) 0x7fd14c0a7400 con 0x7fd14c0a6e90 2024-03-28T19:02:17.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd130003520 con 0x7fd14c0a6e90 2024-03-28T19:02:17.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137103171 0 0) 0x7fd14c158b40 con 0x7fd14c0c1780 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c0a7400 con 0x7fd14c0c1780 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776087596 0 0) 0x7fd130003520 con 0x7fd14c0a6e90 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c06c0d0 msgr2=0x7fd14c06c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14aa00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c06c0d0 0x7fd14c06c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.361+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 >> v1:172.21.15.155:6789/0 conn(0x7fd14c0c1780 legacy=0x7fd14c15fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c158e20 con 0x7fd14c0a6e90 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd134001670 con 0x7fd14c0a6e90 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 >> v1:172.21.15.27:6789/0 conn(0x7fd14c0a6e90 legacy=0x7fd14c155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd130003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c06c0d0 0x7fd130004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd130004d40 0x7fd1300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c159470 con 0x7fd130003b70 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13000bac0 con 0x7fd130004d40 2024-03-28T19:02:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13000bcc0 con 0x7fd14c06c0d0 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14be00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd130003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd14b400640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c06c0d0 0x7fd130004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.362+0000 7fd150c00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd130004d40 0x7fd1300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.363+0000 7fd14be00640 1 -- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c06c0d0 msgr2=0x7fd130004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.363+0000 7fd14be00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c06c0d0 0x7fd130004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.363+0000 7fd14be00640 1 -- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd130004d40 msgr2=0x7fd1300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.363+0000 7fd14be00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd130004d40 0x7fd1300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.363+0000 7fd14be00640 1 -- 172.21.15.27:0/1073352104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd130003520 con 0x7fd130003b70 2024-03-28T19:02:17.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.363+0000 7fd150c00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd130004d40 0x7fd1300051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:17.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.364+0000 7fd14be00640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd130003f70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009fc0 tx=0x7fd130005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03d682e9e134f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:17.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.364+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c039070 con 0x7fd130003b70 2024-03-28T19:02:17.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.364+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c02cc20 con 0x7fd130003b70 2024-03-28T19:02:17.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.364+0000 7fd14aa00640 1 -- 172.21.15.27:0/1073352104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c009c10 con 0x7fd130003b70 2024-03-28T19:02:17.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.365+0000 7fd151600640 1 -- 172.21.15.27:0/1073352104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 msgr2=0x7fd130003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.365+0000 7fd151600640 1 --2- 172.21.15.27:0/1073352104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd130003f70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009fc0 tx=0x7fd130005830 comp rx=0 tx=0).stop 2024-03-28T19:02:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.365+0000 7fd150c00640 1 -- 172.21.15.27:0/1073352104 reap_dead start 2024-03-28T19:02:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.365+0000 7fd151600640 1 -- 172.21.15.27:0/1073352104 shutdown_connections 2024-03-28T19:02:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.365+0000 7fd151600640 1 -- 172.21.15.27:0/1073352104 >> 172.21.15.27:0/1073352104 conn(0x7fd14c0c7aa0 msgr2=0x7fd14c0c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.365+0000 7fd151600640 1 -- 172.21.15.27:0/1073352104 shutdown_connections 2024-03-28T19:02:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.365+0000 7fd151600640 1 -- 172.21.15.27:0/1073352104 wait complete. 2024-03-28T19:02:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.367+0000 7fd151600640 1 Processor -- start 2024-03-28T19:02:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.367+0000 7fd151600640 1 -- start start 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd151600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd14c06f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd151600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd14c0a9030 0x7fd14c1e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd151600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c155f20 0x7fd14c1e8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd151600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c152e20 con 0x7fd130003b70 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd151600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c159900 con 0x7fd14c0a9030 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd151600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c152fa0 con 0x7fd14c155f20 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd14be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd14c06f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd14be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd14c06f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33810/0 (socket says 172.21.15.27:33810) 2024-03-28T19:02:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.368+0000 7fd14be00640 1 -- 172.21.15.27:0/80653005 learned_addr learned my addr 172.21.15.27:0/80653005 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:17.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd14b400640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd14c0a9030 0x7fd14c1e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:17.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd150c00640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c155f20 0x7fd14c1e8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd14be00640 1 -- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c155f20 msgr2=0x7fd14c1e8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd14be00640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c155f20 0x7fd14c1e8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd14be00640 1 -- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd14c0a9030 msgr2=0x7fd14c1e7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd14be00640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd14c0a9030 0x7fd14c1e7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd14be00640 1 -- 172.21.15.27:0/80653005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c0abb20 con 0x7fd130003b70 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd14b400640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd14c0a9030 0x7fd14c1e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.369+0000 7fd14be00640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd14c06f6f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd13c02c690 tx=0x7fd13c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.370+0000 7fd143e00640 1 -- 172.21.15.27:0/80653005 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c044020 con 0x7fd130003b70 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.370+0000 7fd143e00640 1 -- 172.21.15.27:0/80653005 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c03a040 con 0x7fd130003b70 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.370+0000 7fd143e00640 1 -- 172.21.15.27:0/80653005 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c039050 con 0x7fd130003b70 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.370+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14c0abe00 con 0x7fd130003b70 2024-03-28T19:02:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.370+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14c0ac0e0 con 0x7fd130003b70 2024-03-28T19:02:17.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.371+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14c069640 con 0x7fd130003b70 2024-03-28T19:02:17.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.372+0000 7fd143e00640 1 -- 172.21.15.27:0/80653005 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd13c005d10 con 0x7fd130003b70 2024-03-28T19:02:17.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.373+0000 7fd143e00640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:17.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.374+0000 7fd143e00640 1 -- 172.21.15.27:0/80653005 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7353+0+0 (secure 0 0 0) 0x7fd13c0ad600 con 0x7fd130003b70 2024-03-28T19:02:17.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.374+0000 7fd14b400640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.375+0000 7fd14b400640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd134000c00 tx=0x7fd134001d30 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:17.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.379+0000 7fd143e00640 1 -- 172.21.15.27:0/80653005 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd13c04e020 con 0x7fd130003b70 2024-03-28T19:02:17.409 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir foo/.snap/ss 2024-03-28T19:02:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.647+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd14c0c1560 con 0x7fd130003b70 2024-03-28T19:02:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.648+0000 7fd143e00640 1 -- 172.21.15.27:0/80653005 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fd13c034e20 con 0x7fd130003b70 2024-03-28T19:02:17.648 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:17.648 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:02:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:02:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 msgr2=0x7fd12806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd134000c00 tx=0x7fd134001d30 comp rx=0 tx=0).stop 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 msgr2=0x7fd14c06f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd14c06f6f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd13c02c690 tx=0x7fd13c009750 comp rx=0 tx=0).stop 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 shutdown_connections 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12806bc00 0x7fd12806e0c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14c155f20 0x7fd14c1e8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd14c0a9030 0x7fd14c1e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 --2- 172.21.15.27:0/80653005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd130003b70 0x7fd14c06f6f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 >> 172.21.15.27:0/80653005 conn(0x7fd14c0c7aa0 msgr2=0x7fd14c0c3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 shutdown_connections 2024-03-28T19:02:17.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:17.651+0000 7fd151600640 1 -- 172.21.15.27:0/80653005 wait complete. 2024-03-28T19:02:17.673 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:02:19.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.922+0000 7f32b4c00640 1 Processor -- start 2024-03-28T19:02:19.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.922+0000 7f32b4c00640 1 -- start start 2024-03-28T19:02:19.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.923+0000 7f32b4c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32b00c93c0 0x7f32b0155990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:19.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.923+0000 7f32b4c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c8cd0 0x7f32b0155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:19.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.923+0000 7f32b4c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b0158c30 con 0x7f32b0152e90 2024-03-28T19:02:19.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.923+0000 7f32b4c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b0158f10 con 0x7f32b00c93c0 2024-03-28T19:02:19.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.923+0000 7f32b4c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b01591f0 con 0x7f32b00c8cd0 2024-03-28T19:02:19.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.923+0000 7f32af400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32b0152e90 0x7f32b00c7710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52100/0 (socket says 172.21.15.27:52100) 2024-03-28T19:02:19.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.923+0000 7f32af400640 1 -- 172.21.15.27:0/2225239874 learned_addr learned my addr 172.21.15.27:0/2225239874 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:19.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.924+0000 7f32aea00640 1 --2- 172.21.15.27:0/2225239874 >> v2:172.21.15.27:3300/0 conn(0x7f32b00c93c0 0x7f32b0155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:19.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.924+0000 7f32afe00640 1 --2- 172.21.15.27:0/2225239874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c8cd0 0x7f32b0155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.924+0000 7f32afe00640 1 -- 172.21.15.27:0/2225239874 >> v2:172.21.15.27:3300/0 conn(0x7f32b00c93c0 msgr2=0x7f32b0155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.924+0000 7f32afe00640 1 --2- 172.21.15.27:0/2225239874 >> v2:172.21.15.27:3300/0 conn(0x7f32b00c93c0 0x7f32b0155990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.924+0000 7f32afe00640 1 -- 172.21.15.27:0/2225239874 >> v1:172.21.15.27:6789/0 conn(0x7f32b0152e90 legacy=0x7f32b00c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.924+0000 7f32ae000640 1 -- 172.21.15.27:0/2225239874 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259218223 0 0) 0x7f32b0158c30 con 0x7f32b0152e90 2024-03-28T19:02:19.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.924+0000 7f32afe00640 1 -- 172.21.15.27:0/2225239874 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b01594d0 con 0x7f32b00c8cd0 2024-03-28T19:02:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.926+0000 7f32afe00640 1 --2- 172.21.15.27:0/2225239874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c8cd0 0x7f32b0155ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009d90 tx=0x7f32a802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=564c4fa17ed0084d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32ae000640 1 -- 172.21.15.27:0/2225239874 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8039070 con 0x7f32b00c8cd0 2024-03-28T19:02:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32ae000640 1 -- 172.21.15.27:0/2225239874 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a802cc50 con 0x7f32b00c8cd0 2024-03-28T19:02:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32ae000640 1 -- 172.21.15.27:0/2225239874 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8034440 con 0x7f32b00c8cd0 2024-03-28T19:02:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32b4c00640 1 -- 172.21.15.27:0/2225239874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c8cd0 msgr2=0x7f32b0155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32b4c00640 1 --2- 172.21.15.27:0/2225239874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c8cd0 0x7f32b0155ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009d90 tx=0x7f32a802bea0 comp rx=0 tx=0).stop 2024-03-28T19:02:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32b4c00640 1 -- 172.21.15.27:0/2225239874 shutdown_connections 2024-03-28T19:02:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32b4c00640 1 --2- 172.21.15.27:0/2225239874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c8cd0 0x7f32b0155ed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32b4c00640 1 --2- 172.21.15.27:0/2225239874 >> v2:172.21.15.27:3300/0 conn(0x7f32b00c93c0 0x7f32b0155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.927+0000 7f32b4c00640 1 -- 172.21.15.27:0/2225239874 >> 172.21.15.27:0/2225239874 conn(0x7f32b00ac900 msgr2=0x7f32b00acd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.928+0000 7f32b4c00640 1 -- 172.21.15.27:0/2225239874 shutdown_connections 2024-03-28T19:02:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.928+0000 7f32b4c00640 1 -- 172.21.15.27:0/2225239874 wait complete. 2024-03-28T19:02:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.929+0000 7f32b4c00640 1 Processor -- start 2024-03-28T19:02:19.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.929+0000 7f32b4c00640 1 -- start start 2024-03-28T19:02:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.930+0000 7f32b4c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32b00c8cd0 0x7f32b0142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.930+0000 7f32b4c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c93c0 0x7f32b014b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.930+0000 7f32b4c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b0152e90 0x7f32b01ee610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32b4c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00c7e20 con 0x7f32b0152e90 2024-03-28T19:02:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32b4c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0159a90 con 0x7f32b00c8cd0 2024-03-28T19:02:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32b4c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00c7fa0 con 0x7f32b00c93c0 2024-03-28T19:02:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b0152e90 0x7f32b01ee610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b0152e90 0x7f32b01ee610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33814/0 (socket says 172.21.15.27:33814) 2024-03-28T19:02:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32afe00640 1 -- 172.21.15.27:0/856742730 learned_addr learned my addr 172.21.15.27:0/856742730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32aea00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c93c0 0x7f32b014b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32af400640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32b00c8cd0 0x7f32b0142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32afe00640 1 -- 172.21.15.27:0/856742730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c93c0 msgr2=0x7f32b014b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32afe00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c93c0 0x7f32b014b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32afe00640 1 -- 172.21.15.27:0/856742730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32b00c8cd0 msgr2=0x7f32b0142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32afe00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32b00c8cd0 0x7f32b0142b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.931+0000 7f32afe00640 1 -- 172.21.15.27:0/856742730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b01eae70 con 0x7f32b0152e90 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.932+0000 7f32afe00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b0152e90 0x7f32b01ee610 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009eb0 tx=0x7f32a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.932+0000 7f32a7400640 1 -- 172.21.15.27:0/856742730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8039050 con 0x7f32b0152e90 2024-03-28T19:02:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.932+0000 7f32a7400640 1 -- 172.21.15.27:0/856742730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a8045070 con 0x7f32b0152e90 2024-03-28T19:02:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.932+0000 7f32a7400640 1 -- 172.21.15.27:0/856742730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8009450 con 0x7f32b0152e90 2024-03-28T19:02:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.932+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b01eb130 con 0x7f32b0152e90 2024-03-28T19:02:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.932+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b01eb6f0 con 0x7f32b0152e90 2024-03-28T19:02:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.934+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b0150640 con 0x7f32b0152e90 2024-03-28T19:02:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.934+0000 7f32a7400640 1 -- 172.21.15.27:0/856742730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f32a803e610 con 0x7f32b0152e90 2024-03-28T19:02:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.935+0000 7f32a7400640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f329006bbd0 0x7f329006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.935+0000 7f32a7400640 1 -- 172.21.15.27:0/856742730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7353+0+0 (secure 0 0 0) 0x7f32a80adb70 con 0x7f32b0152e90 2024-03-28T19:02:19.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.936+0000 7f32af400640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f329006bbd0 0x7f329006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:19.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.936+0000 7f32af400640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f329006bbd0 0x7f329006e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f32a0000f60 tx=0x7f32a0007c00 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:19.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:19.941+0000 7f32a7400640 1 -- 172.21.15.27:0/856742730 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f32b0150640 con 0x7f32b0152e90 2024-03-28T19:02:20.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.215+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32b0069640 con 0x7f32b0152e90 2024-03-28T19:02:20.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.216+0000 7f32a7400640 1 -- 172.21.15.27:0/856742730 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f32a807bfe0 con 0x7f32b0152e90 2024-03-28T19:02:20.216 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:20.216 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:02:20.216 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f329006bbd0 msgr2=0x7f329006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f329006bbd0 0x7f329006e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f32a0000f60 tx=0x7f32a0007c00 comp rx=0 tx=0).stop 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b0152e90 msgr2=0x7f32b01ee610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b0152e90 0x7f32b01ee610 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009eb0 tx=0x7f32a8004a40 comp rx=0 tx=0).stop 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 shutdown_connections 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f329006bbd0 0x7f329006e090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32b0152e90 0x7f32b01ee610 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32b00c93c0 0x7f32b014b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:20.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 --2- 172.21.15.27:0/856742730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32b00c8cd0 0x7f32b0142b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 >> 172.21.15.27:0/856742730 conn(0x7f32b00ac900 msgr2=0x7f32b00acce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 shutdown_connections 2024-03-28T19:02:20.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:20.219+0000 7f32b4c00640 1 -- 172.21.15.27:0/856742730 wait complete. 2024-03-28T19:02:20.238 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:02:22.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:22.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.476+0000 7fd9e6c00640 1 Processor -- start 2024-03-28T19:02:22.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.476+0000 7fd9e6c00640 1 -- start start 2024-03-28T19:02:22.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.478+0000 7fd9e6c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9e0150ae0 0x7fd9e00a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.478+0000 7fd9e6c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e00c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.478+0000 7fd9e6c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e0150ec0 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.478+0000 7fd9e6c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e006dc70 con 0x7fd9e0150ae0 2024-03-28T19:02:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.478+0000 7fd9e6c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0153550 con 0x7fd9e00a61a0 2024-03-28T19:02:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.478+0000 7fd9e5800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9e0150ae0 0x7fd9e00a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.478+0000 7fd9e5800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9e0150ae0 0x7fd9e00a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33824/0 (socket says 172.21.15.27:33824) 2024-03-28T19:02:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.478+0000 7fd9e5800640 1 -- 172.21.15.27:0/1873628690 learned_addr learned my addr 172.21.15.27:0/1873628690 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.479+0000 7fd9e4e00640 1 --2- 172.21.15.27:0/1873628690 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e00c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.479+0000 7fd9e5800640 1 -- 172.21.15.27:0/1873628690 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 msgr2=0x7fd9e00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.479+0000 7fd9e5800640 1 --2- 172.21.15.27:0/1873628690 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e00c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.479+0000 7fd9e5800640 1 -- 172.21.15.27:0/1873628690 >> v1:172.21.15.155:6789/0 conn(0x7fd9e00c6ee0 legacy=0x7fd9e0153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.479+0000 7fd9e5800640 1 -- 172.21.15.27:0/1873628690 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e006df10 con 0x7fd9e0150ae0 2024-03-28T19:02:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.479+0000 7fd9e4e00640 1 --2- 172.21.15.27:0/1873628690 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e00c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.480+0000 7fd9e5800640 1 --2- 172.21.15.27:0/1873628690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e0150ae0 0x7fd9e00a5c60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4009870 tx=0x7fd9d402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fddabd8c5046c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:22.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.480+0000 7fd9dfe00640 1 -- 172.21.15.27:0/1873628690 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40317c0 con 0x7fd9e0150ae0 2024-03-28T19:02:22.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.480+0000 7fd9dfe00640 1 -- 172.21.15.27:0/1873628690 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d4037070 con 0x7fd9e0150ae0 2024-03-28T19:02:22.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.480+0000 7fd9dfe00640 1 -- 172.21.15.27:0/1873628690 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4031b80 con 0x7fd9e0150ae0 2024-03-28T19:02:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.481+0000 7fd9e6c00640 1 -- 172.21.15.27:0/1873628690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e0150ae0 msgr2=0x7fd9e00a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.481+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/1873628690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e0150ae0 0x7fd9e00a5c60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4009870 tx=0x7fd9d402b860 comp rx=0 tx=0).stop 2024-03-28T19:02:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.481+0000 7fd9e6c00640 1 -- 172.21.15.27:0/1873628690 shutdown_connections 2024-03-28T19:02:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.481+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/1873628690 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e00c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.481+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/1873628690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e0150ae0 0x7fd9e00a5c60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.481+0000 7fd9e6c00640 1 -- 172.21.15.27:0/1873628690 >> 172.21.15.27:0/1873628690 conn(0x7fd9e006b0c0 msgr2=0x7fd9e006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.481+0000 7fd9e6c00640 1 -- 172.21.15.27:0/1873628690 shutdown_connections 2024-03-28T19:02:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.481+0000 7fd9e6c00640 1 -- 172.21.15.27:0/1873628690 wait complete. 2024-03-28T19:02:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.483+0000 7fd9e6c00640 1 Processor -- start 2024-03-28T19:02:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.483+0000 7fd9e6c00640 1 -- start start 2024-03-28T19:02:22.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e6c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e0072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e6c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e00c6ee0 0x7fd9e00a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e6c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9e0150ae0 0x7fd9e00a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e6c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e006e7b0 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e6c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e006e4d0 con 0x7fd9e0150ae0 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e6c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e01569d0 con 0x7fd9e00a61a0 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e00c6ee0 0x7fd9e00a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e00c6ee0 0x7fd9e00a6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33830/0 (socket says 172.21.15.27:33830) 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.484+0000 7fd9e4e00640 1 -- 172.21.15.27:0/2127889519 learned_addr learned my addr 172.21.15.27:0/2127889519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e6200640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9e0150ae0 0x7fd9e00a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e4e00640 1 -- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 msgr2=0x7fd9e0072bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e5800640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e0072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e4e00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e0072bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e4e00640 1 -- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9e0150ae0 msgr2=0x7fd9e00a6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e4e00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9e0150ae0 0x7fd9e00a6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e4e00640 1 -- 172.21.15.27:0/2127889519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e01ecf80 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e6200640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9e0150ae0 0x7fd9e00a6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.485+0000 7fd9e4e00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e00c6ee0 0x7fd9e00a6710 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc00c7f0 tx=0x7fd9cc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.486+0000 7fd9dd600640 1 -- 172.21.15.27:0/2127889519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc015070 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.486+0000 7fd9dd600640 1 -- 172.21.15.27:0/2127889519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc007a30 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.486+0000 7fd9dd600640 1 -- 172.21.15.27:0/2127889519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc007e00 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.486+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e01ed260 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.486+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e00730f0 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.487+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e0067dd0 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.489+0000 7fd9dd600640 1 -- 172.21.15.27:0/2127889519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd9cc018070 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.490+0000 7fd9dd600640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9c006bc00 0x7fd9c006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:22.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.490+0000 7fd9dd600640 1 -- 172.21.15.27:0/2127889519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7353+0+0 (secure 0 0 0) 0x7fd9cc08b0d0 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.490+0000 7fd9e5800640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9c006bc00 0x7fd9c006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:22.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.491+0000 7fd9e5800640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9c006bc00 0x7fd9c006e0c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4036000 tx=0x7fd9d4008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:22.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.495+0000 7fd9dd600640 1 -- 172.21.15.27:0/2127889519 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd9cc0594b0 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.756+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9e0069640 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.757+0000 7fd9dd600640 1 -- 172.21.15.27:0/2127889519 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fd9cc05de20 con 0x7fd9e00c6ee0 2024-03-28T19:02:22.757 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:22.757 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:02:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:02:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9c006bc00 msgr2=0x7fd9c006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9c006bc00 0x7fd9c006e0c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4036000 tx=0x7fd9d4008030 comp rx=0 tx=0).stop 2024-03-28T19:02:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e00c6ee0 msgr2=0x7fd9e00a6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e00c6ee0 0x7fd9e00a6710 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc00c7f0 tx=0x7fd9cc00ccb0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 shutdown_connections 2024-03-28T19:02:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9c006bc00 0x7fd9c006e0c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9e0150ae0 0x7fd9e00a6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9e00c6ee0 0x7fd9e00a6710 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 --2- 172.21.15.27:0/2127889519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9e00a61a0 0x7fd9e0072bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.760+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 >> 172.21.15.27:0/2127889519 conn(0x7fd9e006b0c0 msgr2=0x7fd9e0154770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.761+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 shutdown_connections 2024-03-28T19:02:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:22.761+0000 7fd9e6c00640 1 -- 172.21.15.27:0/2127889519 wait complete. 2024-03-28T19:02:22.782 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:02:24.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:25.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.031+0000 7f29d9600640 1 Processor -- start 2024-03-28T19:02:25.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.031+0000 7f29d9600640 1 -- start start 2024-03-28T19:02:25.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.032+0000 7f29d9600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29d40c2730 0x7f29d4071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:25.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.033+0000 7f29d9600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f29d406de30 0x7f29d4072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:25.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.033+0000 7f29d9600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d4073260 con 0x7f29d4150820 2024-03-28T19:02:25.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.033+0000 7f29d9600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d4073030 con 0x7f29d40c2730 2024-03-28T19:02:25.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.033+0000 7f29d9600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d40c1ce0 con 0x7f29d406de30 2024-03-28T19:02:25.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.033+0000 7f29d3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29d40c2730 0x7f29d4071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:25.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.033+0000 7f29d3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29d40c2730 0x7f29d4071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33836/0 (socket says 172.21.15.27:33836) 2024-03-28T19:02:25.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.033+0000 7f29d3400640 1 -- 172.21.15.27:0/3859995796 learned_addr learned my addr 172.21.15.27:0/3859995796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.033+0000 7f29d8c00640 1 --2- 172.21.15.27:0/3859995796 >> v2:172.21.15.155:3300/0 conn(0x7f29d406de30 0x7f29d4072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.034+0000 7f29d3400640 1 -- 172.21.15.27:0/3859995796 >> v2:172.21.15.155:3300/0 conn(0x7f29d406de30 msgr2=0x7f29d4072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.034+0000 7f29d3400640 1 --2- 172.21.15.27:0/3859995796 >> v2:172.21.15.155:3300/0 conn(0x7f29d406de30 0x7f29d4072390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.034+0000 7f29d3400640 1 -- 172.21.15.27:0/3859995796 >> v1:172.21.15.155:6789/0 conn(0x7f29d4150820 legacy=0x7f29d40c7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.034+0000 7f29d3400640 1 -- 172.21.15.27:0/3859995796 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d4073540 con 0x7f29d40c2730 2024-03-28T19:02:25.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.035+0000 7f29d3400640 1 --2- 172.21.15.27:0/3859995796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d40c2730 0x7f29d4071e50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f29d406e2a0 tx=0x7f29c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f92dd8647d03e6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:25.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.035+0000 7f29d2a00640 1 -- 172.21.15.27:0/3859995796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c002e070 con 0x7f29d40c2730 2024-03-28T19:02:25.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.035+0000 7f29d2a00640 1 -- 172.21.15.27:0/3859995796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c0037070 con 0x7f29d40c2730 2024-03-28T19:02:25.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.035+0000 7f29d2a00640 1 -- 172.21.15.27:0/3859995796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c0031a00 con 0x7f29d40c2730 2024-03-28T19:02:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.036+0000 7f29d9600640 1 -- 172.21.15.27:0/3859995796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d40c2730 msgr2=0x7f29d4071e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.036+0000 7f29d9600640 1 --2- 172.21.15.27:0/3859995796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d40c2730 0x7f29d4071e50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f29d406e2a0 tx=0x7f29c002b880 comp rx=0 tx=0).stop 2024-03-28T19:02:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.036+0000 7f29d9600640 1 -- 172.21.15.27:0/3859995796 shutdown_connections 2024-03-28T19:02:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.036+0000 7f29d9600640 1 --2- 172.21.15.27:0/3859995796 >> v2:172.21.15.155:3300/0 conn(0x7f29d406de30 0x7f29d4072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.036+0000 7f29d9600640 1 --2- 172.21.15.27:0/3859995796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d40c2730 0x7f29d4071e50 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.036+0000 7f29d9600640 1 -- 172.21.15.27:0/3859995796 >> 172.21.15.27:0/3859995796 conn(0x7f29d406d980 msgr2=0x7f29d40a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.036+0000 7f29d9600640 1 -- 172.21.15.27:0/3859995796 shutdown_connections 2024-03-28T19:02:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.036+0000 7f29d9600640 1 -- 172.21.15.27:0/3859995796 wait complete. 2024-03-28T19:02:25.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.038+0000 7f29d9600640 1 Processor -- start 2024-03-28T19:02:25.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.038+0000 7f29d9600640 1 -- start start 2024-03-28T19:02:25.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.039+0000 7f29d9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d406de30 0x7f29d40abb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.039+0000 7f29d9600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29d4150820 0x7f29d4156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.039+0000 7f29d9600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29d40ac0b0 0x7f29d4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.039+0000 7f29d9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40c7980 con 0x7f29d406de30 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.039+0000 7f29d9600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d406f020 con 0x7f29d40ac0b0 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.039+0000 7f29d9600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d40c7b00 con 0x7f29d4150820 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.039+0000 7f29d3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d406de30 0x7f29d40abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.039+0000 7f29d3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d406de30 0x7f29d40abb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33838/0 (socket says 172.21.15.27:33838) 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d3e00640 1 -- 172.21.15.27:0/3083771393 learned_addr learned my addr 172.21.15.27:0/3083771393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d3400640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29d4150820 0x7f29d4156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d8c00640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29d40ac0b0 0x7f29d4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:25.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d3e00640 1 -- 172.21.15.27:0/3083771393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29d4150820 msgr2=0x7f29d4156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d3e00640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29d4150820 0x7f29d4156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d3e00640 1 -- 172.21.15.27:0/3083771393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29d40ac0b0 msgr2=0x7f29d4156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d3e00640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29d40ac0b0 0x7f29d4156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d3e00640 1 -- 172.21.15.27:0/3083771393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d41eafe0 con 0x7f29d406de30 2024-03-28T19:02:25.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.040+0000 7f29d3e00640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d406de30 0x7f29d40abb70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f29c400b3f0 tx=0x7f29c400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:25.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.041+0000 7f29cbe00640 1 -- 172.21.15.27:0/3083771393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4015020 con 0x7f29d406de30 2024-03-28T19:02:25.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.041+0000 7f29cbe00640 1 -- 172.21.15.27:0/3083771393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c4011830 con 0x7f29d406de30 2024-03-28T19:02:25.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.041+0000 7f29cbe00640 1 -- 172.21.15.27:0/3083771393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4011c20 con 0x7f29d406de30 2024-03-28T19:02:25.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.041+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d41eb2c0 con 0x7f29d406de30 2024-03-28T19:02:25.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.041+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d41eb880 con 0x7f29d406de30 2024-03-28T19:02:25.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.042+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d4067dd0 con 0x7f29d406de30 2024-03-28T19:02:25.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.043+0000 7f29cbe00640 1 -- 172.21.15.27:0/3083771393 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f29c4018020 con 0x7f29d406de30 2024-03-28T19:02:25.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.044+0000 7f29cbe00640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29b406bbd0 0x7f29b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:25.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.044+0000 7f29cbe00640 1 -- 172.21.15.27:0/3083771393 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7353+0+0 (secure 0 0 0) 0x7f29c408ad80 con 0x7f29d406de30 2024-03-28T19:02:25.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.044+0000 7f29d3400640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29b406bbd0 0x7f29b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:25.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.045+0000 7f29d3400640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29b406bbd0 0x7f29b406e090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f29c0036000 tx=0x7f29c003d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:25.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.049+0000 7f29cbe00640 1 -- 172.21.15.27:0/3083771393 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f29c401f030 con 0x7f29d406de30 2024-03-28T19:02:25.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.354+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d4069640 con 0x7f29d406de30 2024-03-28T19:02:25.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.356+0000 7f29cbe00640 1 -- 172.21.15.27:0/3083771393 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f29c40172c0 con 0x7f29d406de30 2024-03-28T19:02:25.356 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:25.356 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:02:25.356 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:02:25.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.358+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29b406bbd0 msgr2=0x7f29b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.358+0000 7f29d9600640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29b406bbd0 0x7f29b406e090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f29c0036000 tx=0x7f29c003d040 comp rx=0 tx=0).stop 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.358+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d406de30 msgr2=0x7f29d40abb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.358+0000 7f29d9600640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d406de30 0x7f29d40abb70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f29c400b3f0 tx=0x7f29c400b8b0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.358+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 shutdown_connections 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.358+0000 7f29d9600640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29b406bbd0 0x7f29b406e090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.358+0000 7f29d9600640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29d40ac0b0 0x7f29d4156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.359+0000 7f29d9600640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29d4150820 0x7f29d4156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.359+0000 7f29d9600640 1 --2- 172.21.15.27:0/3083771393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29d406de30 0x7f29d40abb70 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.359+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 >> 172.21.15.27:0/3083771393 conn(0x7f29d406d980 msgr2=0x7f29d40c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:25.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.359+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 shutdown_connections 2024-03-28T19:02:25.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:25.359+0000 7f29d9600640 1 -- 172.21.15.27:0/3083771393 wait complete. 2024-03-28T19:02:25.381 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:02:27.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:27.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.621+0000 7f3193200640 1 Processor -- start 2024-03-28T19:02:27.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.621+0000 7f3193200640 1 -- start start 2024-03-28T19:02:27.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.623+0000 7f3193200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31940ae3d0 0x7f31940c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:27.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.623+0000 7f3193200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31940c5460 con 0x7f31940aebe0 2024-03-28T19:02:27.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.623+0000 7f3193200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31940c4bc0 con 0x7f31940ae7b0 2024-03-28T19:02:27.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.623+0000 7f3193200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3194153630 con 0x7f31940ae3d0 2024-03-28T19:02:27.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.624+0000 7f3191e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31940ae3d0 0x7f31940c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:27.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.624+0000 7f3191e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31940ae3d0 0x7f31940c56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40944/0 (socket says 172.21.15.27:40944) 2024-03-28T19:02:27.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.624+0000 7f3191e00640 1 -- 172.21.15.27:0/2320831698 learned_addr learned my addr 172.21.15.27:0/2320831698 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:27.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.624+0000 7f3191400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31940aebe0 0x7f3194153ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38848/0 (socket says 172.21.15.27:38848) 2024-03-28T19:02:27.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.624+0000 7f3191e00640 1 -- 172.21.15.27:0/2320831698 >> v1:172.21.15.155:6789/0 conn(0x7f31940ae7b0 legacy=0x7f319415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:27.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.624+0000 7f3191e00640 1 -- 172.21.15.27:0/2320831698 >> v1:172.21.15.27:6789/0 conn(0x7f31940aebe0 legacy=0x7f3194153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:27.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.625+0000 7f3191e00640 1 -- 172.21.15.27:0/2320831698 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31940c4ea0 con 0x7f31940ae3d0 2024-03-28T19:02:27.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.626+0000 7f3191e00640 1 --2- 172.21.15.27:0/2320831698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940ae3d0 0x7f31940c56b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3184009870 tx=0x7f318402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be66fc7fbd10cad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:27.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.626+0000 7f3190a00640 1 -- 172.21.15.27:0/2320831698 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318402f850 con 0x7f31940ae3d0 2024-03-28T19:02:27.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.626+0000 7f3190a00640 1 -- 172.21.15.27:0/2320831698 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318402f9f0 con 0x7f31940ae3d0 2024-03-28T19:02:27.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.626+0000 7f3190a00640 1 -- 172.21.15.27:0/2320831698 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318402fdc0 con 0x7f31940ae3d0 2024-03-28T19:02:27.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.627+0000 7f3193200640 1 -- 172.21.15.27:0/2320831698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940ae3d0 msgr2=0x7f31940c56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:27.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.627+0000 7f3193200640 1 --2- 172.21.15.27:0/2320831698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940ae3d0 0x7f31940c56b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3184009870 tx=0x7f318402ba30 comp rx=0 tx=0).stop 2024-03-28T19:02:27.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.627+0000 7f3193200640 1 -- 172.21.15.27:0/2320831698 shutdown_connections 2024-03-28T19:02:27.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.627+0000 7f3193200640 1 --2- 172.21.15.27:0/2320831698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940ae3d0 0x7f31940c56b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:27.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.627+0000 7f3193200640 1 -- 172.21.15.27:0/2320831698 >> 172.21.15.27:0/2320831698 conn(0x7f3194072d10 msgr2=0x7f319406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:27.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.627+0000 7f3193200640 1 -- 172.21.15.27:0/2320831698 shutdown_connections 2024-03-28T19:02:27.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.627+0000 7f3193200640 1 -- 172.21.15.27:0/2320831698 wait complete. 2024-03-28T19:02:27.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.629+0000 7f3193200640 1 Processor -- start 2024-03-28T19:02:27.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.629+0000 7f3193200640 1 -- start start 2024-03-28T19:02:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.630+0000 7f3193200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31940ae3d0 0x7f31940ad130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.630+0000 7f3193200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31940ae7b0 0x7f319406c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.630+0000 7f3193200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940aebe0 0x7f31940aba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.630+0000 7f3193200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31941541d0 con 0x7f31940aebe0 2024-03-28T19:02:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.630+0000 7f3193200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3194070890 con 0x7f31940ae3d0 2024-03-28T19:02:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.630+0000 7f3193200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3194154350 con 0x7f31940ae7b0 2024-03-28T19:02:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3192800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940aebe0 0x7f31940aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:27.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3192800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940aebe0 0x7f31940aba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40956/0 (socket says 172.21.15.27:40956) 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3192800640 1 -- 172.21.15.27:0/1274867036 learned_addr learned my addr 172.21.15.27:0/1274867036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3191e00640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31940ae3d0 0x7f31940ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3191400640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31940ae7b0 0x7f319406c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3192800640 1 -- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31940ae7b0 msgr2=0x7f319406c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3192800640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31940ae7b0 0x7f319406c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3192800640 1 -- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31940ae3d0 msgr2=0x7f31940ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3192800640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31940ae3d0 0x7f31940ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3192800640 1 -- 172.21.15.27:0/1274867036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31941eb480 con 0x7f31940aebe0 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.631+0000 7f3191400640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31940ae7b0 0x7f319406c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:02:27.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.632+0000 7f3192800640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940aebe0 0x7f31940aba90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f318c00c6b0 tx=0x7f318c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.632+0000 7f318a000640 1 -- 172.21.15.27:0/1274867036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318c003b70 con 0x7f31940aebe0 2024-03-28T19:02:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.632+0000 7f318a000640 1 -- 172.21.15.27:0/1274867036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318c003d30 con 0x7f31940aebe0 2024-03-28T19:02:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.632+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31940ad960 con 0x7f31940aebe0 2024-03-28T19:02:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.632+0000 7f318a000640 1 -- 172.21.15.27:0/1274867036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318c005310 con 0x7f31940aebe0 2024-03-28T19:02:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.632+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31941eba20 con 0x7f31940aebe0 2024-03-28T19:02:27.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.633+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31940c4240 con 0x7f31940aebe0 2024-03-28T19:02:27.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.635+0000 7f318a000640 1 -- 172.21.15.27:0/1274867036 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f318c0054b0 con 0x7f31940aebe0 2024-03-28T19:02:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.636+0000 7f318a000640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f317406bbd0 0x7f317406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.636+0000 7f318a000640 1 -- 172.21.15.27:0/1274867036 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7353+0+0 (secure 0 0 0) 0x7f318c08aeb0 con 0x7f31940aebe0 2024-03-28T19:02:27.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.636+0000 7f3191e00640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f317406bbd0 0x7f317406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.637+0000 7f3191e00640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f317406bbd0 0x7f317406e090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3184036000 tx=0x7f3184008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:27.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.640+0000 7f318a000640 1 -- 172.21.15.27:0/1274867036 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f31940c4240 con 0x7f31940aebe0 2024-03-28T19:02:27.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.929+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3194156700 con 0x7f31940aebe0 2024-03-28T19:02:27.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.930+0000 7f318a000640 1 -- 172.21.15.27:0/1274867036 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f318c059330 con 0x7f31940aebe0 2024-03-28T19:02:27.930 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:02:27.930 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f317406bbd0 msgr2=0x7f317406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f317406bbd0 0x7f317406e090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3184036000 tx=0x7f3184008030 comp rx=0 tx=0).stop 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940aebe0 msgr2=0x7f31940aba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940aebe0 0x7f31940aba90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f318c00c6b0 tx=0x7f318c00cb70 comp rx=0 tx=0).stop 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 shutdown_connections 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f317406bbd0 0x7f317406e090 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31940aebe0 0x7f31940aba90 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31940ae7b0 0x7f319406c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 --2- 172.21.15.27:0/1274867036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31940ae3d0 0x7f31940ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 >> 172.21.15.27:0/1274867036 conn(0x7f3194072d10 msgr2=0x7f3194154790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 shutdown_connections 2024-03-28T19:02:27.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:27.933+0000 7f3193200640 1 -- 172.21.15.27:0/1274867036 wait complete. 2024-03-28T19:02:27.952 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:02:29.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:02:29.312 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:02:29.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:30.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.189+0000 7f9801600640 1 Processor -- start 2024-03-28T19:02:30.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.189+0000 7f9801600640 1 -- start start 2024-03-28T19:02:30.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.191+0000 7f9801600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0c8ed0 0x7f97fc0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:30.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.191+0000 7f9801600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f97fc0be5d0 0x7f97fc155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:30.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.191+0000 7f9801600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc156230 con 0x7f97fc0c58c0 2024-03-28T19:02:30.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.191+0000 7f9801600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97fc158bd0 con 0x7f97fc0be5d0 2024-03-28T19:02:30.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.191+0000 7f9801600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc158eb0 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.191+0000 7f9800c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f97fc0be5d0 0x7f97fc155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:30.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.192+0000 7f9800c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f97fc0be5d0 0x7f97fc155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:44672/0 (socket says 172.21.15.27:44672) 2024-03-28T19:02:30.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.192+0000 7f9800c00640 1 -- 172.21.15.27:0/1432552320 learned_addr learned my addr 172.21.15.27:0/1432552320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:30.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.192+0000 7f97fb400640 1 --2- 172.21.15.27:0/1432552320 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0c8ed0 0x7f97fc0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:30.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.192+0000 7f9800c00640 1 -- 172.21.15.27:0/1432552320 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0c8ed0 msgr2=0x7f97fc0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:30.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.192+0000 7f9800c00640 1 --2- 172.21.15.27:0/1432552320 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0c8ed0 0x7f97fc0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.192+0000 7f9800c00640 1 -- 172.21.15.27:0/1432552320 >> v1:172.21.15.155:6789/0 conn(0x7f97fc0c58c0 legacy=0x7f97fc1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:30.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.192+0000 7f9800c00640 1 -- 172.21.15.27:0/1432552320 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc159190 con 0x7f97fc0be5d0 2024-03-28T19:02:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.194+0000 7f9800c00640 1 --2- 172.21.15.27:0/1432552320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0be5d0 0x7f97fc155cf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009e50 tx=0x7f97f402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9185acecf823a452 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:30.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.194+0000 7f97faa00640 1 -- 172.21.15.27:0/1432552320 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f402ca70 con 0x7f97fc0be5d0 2024-03-28T19:02:30.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f97faa00640 1 -- 172.21.15.27:0/1432552320 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f402cc10 con 0x7f97fc0be5d0 2024-03-28T19:02:30.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f97faa00640 1 -- 172.21.15.27:0/1432552320 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f40343f0 con 0x7f97fc0be5d0 2024-03-28T19:02:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f9801600640 1 -- 172.21.15.27:0/1432552320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0be5d0 msgr2=0x7f97fc155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f9801600640 1 --2- 172.21.15.27:0/1432552320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0be5d0 0x7f97fc155cf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009e50 tx=0x7f97f402bed0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f9801600640 1 -- 172.21.15.27:0/1432552320 shutdown_connections 2024-03-28T19:02:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f9801600640 1 --2- 172.21.15.27:0/1432552320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0be5d0 0x7f97fc155cf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f9801600640 1 --2- 172.21.15.27:0/1432552320 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0c8ed0 0x7f97fc0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f9801600640 1 -- 172.21.15.27:0/1432552320 >> 172.21.15.27:0/1432552320 conn(0x7f97fc06b6f0 msgr2=0x7f97fc0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f9801600640 1 -- 172.21.15.27:0/1432552320 shutdown_connections 2024-03-28T19:02:30.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.195+0000 7f9801600640 1 -- 172.21.15.27:0/1432552320 wait complete. 2024-03-28T19:02:30.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.197+0000 7f9801600640 1 Processor -- start 2024-03-28T19:02:30.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.197+0000 7f9801600640 1 -- start start 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.198+0000 7f9801600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0be5d0 0x7f97fc1ebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.198+0000 7f9801600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0c58c0 0x7f97fc0c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.198+0000 7f9801600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.198+0000 7f9801600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc0b0820 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.198+0000 7f9801600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc159750 con 0x7f97fc0c58c0 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.198+0000 7f9801600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc0b09a0 con 0x7f97fc0be5d0 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f9800c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f9800c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40964/0 (socket says 172.21.15.27:40964) 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f9800c00640 1 -- 172.21.15.27:0/3825120232 learned_addr learned my addr 172.21.15.27:0/3825120232 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f97fbe00640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0be5d0 0x7f97fc1ebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f97fb400640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0c58c0 0x7f97fc0c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:30.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f9800c00640 1 -- 172.21.15.27:0/3825120232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0be5d0 msgr2=0x7f97fc1ebfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f9800c00640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0be5d0 0x7f97fc1ebfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f9800c00640 1 -- 172.21.15.27:0/3825120232 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0c58c0 msgr2=0x7f97fc0c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f9800c00640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0c58c0 0x7f97fc0c3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.199+0000 7f9800c00640 1 -- 172.21.15.27:0/3825120232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc1df6d0 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.200+0000 7f9800c00640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c2e50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009e50 tx=0x7f97f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.200+0000 7f97f3e00640 1 -- 172.21.15.27:0/3825120232 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4043020 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.200+0000 7f97f3e00640 1 -- 172.21.15.27:0/3825120232 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f403e450 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.200+0000 7f97f3e00640 1 -- 172.21.15.27:0/3825120232 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4009510 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.200+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97fc1df9b0 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.200+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc1eb7d0 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.202+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc069640 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.203+0000 7f97f3e00640 1 -- 172.21.15.27:0/3825120232 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f97f4046020 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.204+0000 7f97f3e00640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f97dc06bbd0 0x7f97dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.204+0000 7f97f3e00640 1 -- 172.21.15.27:0/3825120232 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7353+0+0 (secure 0 0 0) 0x7f97f40ad7f0 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.204+0000 7f97fbe00640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f97dc06bbd0 0x7f97dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:30.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.205+0000 7f97fbe00640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f97dc06bbd0 0x7f97dc06e090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0ad160 tx=0x7f97ec009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:30.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.210+0000 7f97f3e00640 1 -- 172.21.15.27:0/3825120232 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f97f404d030 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.472+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97fc156580 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.473+0000 7f97f3e00640 1 -- 172.21.15.27:0/3825120232 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f97f407bd00 con 0x7f97fc0c8ed0 2024-03-28T19:02:30.473 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:30.473 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:02:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:02:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.476+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f97dc06bbd0 msgr2=0x7f97dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.476+0000 7f9801600640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f97dc06bbd0 0x7f97dc06e090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0ad160 tx=0x7f97ec009040 comp rx=0 tx=0).stop 2024-03-28T19:02:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.476+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97fc0c8ed0 msgr2=0x7f97fc0c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.476+0000 7f9801600640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c2e50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009e50 tx=0x7f97f4004a40 comp rx=0 tx=0).stop 2024-03-28T19:02:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.477+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 shutdown_connections 2024-03-28T19:02:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.477+0000 7f9801600640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f97dc06bbd0 0x7f97dc06e090 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.477+0000 7f9801600640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97fc0c8ed0 0x7f97fc0c2e50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.477+0000 7f9801600640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97fc0c58c0 0x7f97fc0c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.477+0000 7f9801600640 1 --2- 172.21.15.27:0/3825120232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97fc0be5d0 0x7f97fc1ebfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.477+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 >> 172.21.15.27:0/3825120232 conn(0x7f97fc06b6f0 msgr2=0x7f97fc157c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.477+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 shutdown_connections 2024-03-28T19:02:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:30.477+0000 7f9801600640 1 -- 172.21.15.27:0/3825120232 wait complete. 2024-03-28T19:02:30.501 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:02:32.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:32.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.747+0000 7f767d600640 1 Processor -- start 2024-03-28T19:02:32.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.747+0000 7f767d600640 1 -- start start 2024-03-28T19:02:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.748+0000 7f767d600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 0x7f76780c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.748+0000 7f767d600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7678158bd0 con 0x7f76780c58c0 2024-03-28T19:02:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.749+0000 7f767d600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7678158eb0 con 0x7f76780c8ed0 2024-03-28T19:02:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.749+0000 7f767d600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7678159190 con 0x7f76780be5d0 2024-03-28T19:02:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.749+0000 7f7677e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f76780c58c0 0x7f76781555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38860/0 (socket says 172.21.15.27:38860) 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.749+0000 7f7677e00640 1 -- 172.21.15.27:0/1177996198 learned_addr learned my addr 172.21.15.27:0/1177996198 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.749+0000 7f767cc00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 0x7f76780c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.749+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589614478 0 0) 0x7f7678158bd0 con 0x7f76780c58c0 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.749+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f765c003520 con 0x7f76780c58c0 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4113335462 0 0) 0x7f765c003520 con 0x7f76780c58c0 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 msgr2=0x7f76780c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 0x7f76780c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 >> v1:172.21.15.155:6789/0 conn(0x7f76780c8ed0 legacy=0x7f7678155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7678159470 con 0x7f76780c58c0 2024-03-28T19:02:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f7668001670 con 0x7f76780c58c0 2024-03-28T19:02:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 >> v1:172.21.15.27:6789/0 conn(0x7f76780c58c0 legacy=0x7f76781555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 0x7f765c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c003f20 0x7f765c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f765c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7678159190 con 0x7f765c004d40 2024-03-28T19:02:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c00bad0 con 0x7f765c003f20 2024-03-28T19:02:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c00bcd0 con 0x7f76780be5d0 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.750+0000 7f767cc00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f765c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.751+0000 7f7677400640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c003f20 0x7f765c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.751+0000 7f7677e00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 0x7f765c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.751+0000 7f767cc00640 1 -- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 msgr2=0x7f765c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.751+0000 7f767cc00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 0x7f765c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.751+0000 7f767cc00640 1 -- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c003f20 msgr2=0x7f765c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.751+0000 7f767cc00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c003f20 0x7f765c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.751+0000 7f767cc00640 1 -- 172.21.15.27:0/1177996198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c003520 con 0x7f765c004d40 2024-03-28T19:02:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.751+0000 7f7677e00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76780be5d0 0x7f765c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.752+0000 7f767cc00640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f765c0051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7670009fc0 tx=0x7f765c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb0748976a833b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.752+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670039070 con 0x7f765c004d40 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.752+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767002cc20 con 0x7f765c004d40 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.752+0000 7f7676a00640 1 -- 172.21.15.27:0/1177996198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670009c10 con 0x7f765c004d40 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.753+0000 7f767d600640 1 -- 172.21.15.27:0/1177996198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 msgr2=0x7f765c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.753+0000 7f767d600640 1 --2- 172.21.15.27:0/1177996198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f765c0051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7670009fc0 tx=0x7f765c005830 comp rx=0 tx=0).stop 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.753+0000 7f767cc00640 1 -- 172.21.15.27:0/1177996198 reap_dead start 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.753+0000 7f767d600640 1 -- 172.21.15.27:0/1177996198 shutdown_connections 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.753+0000 7f767d600640 1 -- 172.21.15.27:0/1177996198 >> 172.21.15.27:0/1177996198 conn(0x7f767806b6f0 msgr2=0x7f76780acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.753+0000 7f767d600640 1 -- 172.21.15.27:0/1177996198 shutdown_connections 2024-03-28T19:02:32.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.753+0000 7f767d600640 1 -- 172.21.15.27:0/1177996198 wait complete. 2024-03-28T19:02:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.755+0000 7f767d600640 1 Processor -- start 2024-03-28T19:02:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.755+0000 7f767d600640 1 -- start start 2024-03-28T19:02:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f767d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f76780abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f767d600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76781eb160 0x7f76781eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f767d600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7678151ac0 0x7f7678151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f767d600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76780b09a0 con 0x7f765c004d40 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f767d600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76780b0820 con 0x7f7678151ac0 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f767d600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76780ad7d0 con 0x7f76781eb160 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f7677e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f76780abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f7677e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f76780abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40992/0 (socket says 172.21.15.27:40992) 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f7677e00640 1 -- 172.21.15.27:0/2578445445 learned_addr learned my addr 172.21.15.27:0/2578445445 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f767cc00640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7678151ac0 0x7f7678151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f7677e00640 1 -- 172.21.15.27:0/2578445445 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76781eb160 msgr2=0x7f76781eb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f7677e00640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76781eb160 0x7f76781eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f7677e00640 1 -- 172.21.15.27:0/2578445445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7678151ac0 msgr2=0x7f7678151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f7677e00640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7678151ac0 0x7f7678151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.757+0000 7f7677e00640 1 -- 172.21.15.27:0/2578445445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76781543b0 con 0x7f765c004d40 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.758+0000 7f7677e00640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f76780abd00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f766800d8e0 tx=0x7f766800dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.758+0000 7f766fe00640 1 -- 172.21.15.27:0/2578445445 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668019070 con 0x7f765c004d40 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.758+0000 7f766fe00640 1 -- 172.21.15.27:0/2578445445 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76680074e0 con 0x7f765c004d40 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.758+0000 7f766fe00640 1 -- 172.21.15.27:0/2578445445 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76680049e0 con 0x7f765c004d40 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.758+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76780acde0 con 0x7f765c004d40 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.758+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76781ef3c0 con 0x7f765c004d40 2024-03-28T19:02:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.759+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7678069640 con 0x7f765c004d40 2024-03-28T19:02:32.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.761+0000 7f766fe00640 1 -- 172.21.15.27:0/2578445445 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7668004b80 con 0x7f765c004d40 2024-03-28T19:02:32.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.761+0000 7f766fe00640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f765806bc00 0x7f765806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:32.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.762+0000 7f766fe00640 1 -- 172.21.15.27:0/2578445445 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7353+0+0 (secure 0 0 0) 0x7f766808abc0 con 0x7f765c004d40 2024-03-28T19:02:32.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.762+0000 7f7677400640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f765806bc00 0x7f765806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:32.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.763+0000 7f7677400640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f765806bc00 0x7f765806e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7664007fd0 tx=0x7f7664009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:32.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:32.767+0000 7f766fe00640 1 -- 172.21.15.27:0/2578445445 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7668059030 con 0x7f765c004d40 2024-03-28T19:02:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.038+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7678156580 con 0x7f765c004d40 2024-03-28T19:02:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.039+0000 7f766fe00640 1 -- 172.21.15.27:0/2578445445 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f7668058930 con 0x7f765c004d40 2024-03-28T19:02:33.039 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:33.039 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:01:39.837636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":15,"addr":"172.21.15.27:6837/4097237428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":4097237428},{"type":"v1","addr":"172.21.15.27:6837","nonce":4097237428}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":2}]} 2024-03-28T19:02:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-03-28T19:02:33.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f765806bc00 msgr2=0x7f765806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f765806bc00 0x7f765806e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7664007fd0 tx=0x7f7664009040 comp rx=0 tx=0).stop 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 msgr2=0x7f76780abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f76780abd00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f766800d8e0 tx=0x7f766800dda0 comp rx=0 tx=0).stop 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 shutdown_connections 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f765806bc00 0x7f765806e0c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7678151ac0 0x7f7678151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76781eb160 0x7f76781eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 --2- 172.21.15.27:0/2578445445 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c004d40 0x7f76780abd00 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 >> 172.21.15.27:0/2578445445 conn(0x7f767806b6f0 msgr2=0x7f76780c95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.042+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 shutdown_connections 2024-03-28T19:02:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:33.043+0000 7f767d600640 1 -- 172.21.15.27:0/2578445445 wait complete. 2024-03-28T19:02:33.065 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:02:35.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:35.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.299+0000 7ffab5c00640 1 Processor -- start 2024-03-28T19:02:35.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.299+0000 7ffab5c00640 1 -- start start 2024-03-28T19:02:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.300+0000 7ffab5c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffab00c59d0 0x7ffab00c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.300+0000 7ffab5c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffab0155600 0x7ffab014f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffab5c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab0155fe0 con 0x7ffab00727e0 2024-03-28T19:02:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffab5c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab01565a0 con 0x7ffab00c59d0 2024-03-28T19:02:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffab5c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab0159090 con 0x7ffab0155600 2024-03-28T19:02:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffaafe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffab00c59d0 0x7ffab00c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffaafe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffab00c59d0 0x7ffab00c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41006/0 (socket says 172.21.15.27:41006) 2024-03-28T19:02:35.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffaafe00640 1 -- 172.21.15.27:0/1192036462 learned_addr learned my addr 172.21.15.27:0/1192036462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:35.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffab5200640 1 --2- 172.21.15.27:0/1192036462 >> v2:172.21.15.155:3300/0 conn(0x7ffab0155600 0x7ffab014f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffaafe00640 1 -- 172.21.15.27:0/1192036462 >> v2:172.21.15.155:3300/0 conn(0x7ffab0155600 msgr2=0x7ffab014f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:35.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffaafe00640 1 --2- 172.21.15.27:0/1192036462 >> v2:172.21.15.155:3300/0 conn(0x7ffab0155600 0x7ffab014f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffaafe00640 1 -- 172.21.15.27:0/1192036462 >> v1:172.21.15.155:6789/0 conn(0x7ffab00727e0 legacy=0x7ffab0159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:35.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.301+0000 7ffaafe00640 1 -- 172.21.15.27:0/1192036462 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab01562c0 con 0x7ffab00c59d0 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.303+0000 7ffaafe00640 1 --2- 172.21.15.27:0/1192036462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab00c59d0 0x7ffab00c8f00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffab00c6bf0 tx=0x7ffaa002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af9755d132c39db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.303+0000 7ffaaea00640 1 -- 172.21.15.27:0/1192036462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0038070 con 0x7ffab00c59d0 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.303+0000 7ffaaea00640 1 -- 172.21.15.27:0/1192036462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa002f050 con 0x7ffab00c59d0 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.303+0000 7ffaaea00640 1 -- 172.21.15.27:0/1192036462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0031970 con 0x7ffab00c59d0 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.304+0000 7ffab5c00640 1 -- 172.21.15.27:0/1192036462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab00c59d0 msgr2=0x7ffab00c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.304+0000 7ffab5c00640 1 --2- 172.21.15.27:0/1192036462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab00c59d0 0x7ffab00c8f00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffab00c6bf0 tx=0x7ffaa002b7f0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.304+0000 7ffab5c00640 1 -- 172.21.15.27:0/1192036462 shutdown_connections 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.304+0000 7ffab5c00640 1 --2- 172.21.15.27:0/1192036462 >> v2:172.21.15.155:3300/0 conn(0x7ffab0155600 0x7ffab014f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.304+0000 7ffab5c00640 1 --2- 172.21.15.27:0/1192036462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab00c59d0 0x7ffab00c8f00 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.304+0000 7ffab5c00640 1 -- 172.21.15.27:0/1192036462 >> 172.21.15.27:0/1192036462 conn(0x7ffab00ac910 msgr2=0x7ffab00accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:35.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.304+0000 7ffab5c00640 1 -- 172.21.15.27:0/1192036462 shutdown_connections 2024-03-28T19:02:35.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.304+0000 7ffab5c00640 1 -- 172.21.15.27:0/1192036462 wait complete. 2024-03-28T19:02:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.306+0000 7ffab5c00640 1 Processor -- start 2024-03-28T19:02:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.306+0000 7ffab5c00640 1 -- start start 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.307+0000 7ffab5c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffab00727e0 0x7ffab01e1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.307+0000 7ffab5c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab00c59d0 0x7ffab01ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.307+0000 7ffab5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab0155600 0x7ffab00ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.307+0000 7ffab5c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00c5f30 con 0x7ffab0155600 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.307+0000 7ffab5c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00c5db0 con 0x7ffab00c59d0 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.307+0000 7ffab5c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab0155a50 con 0x7ffab00727e0 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffab5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab0155600 0x7ffab00ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffab5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab0155600 0x7ffab00ab900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41014/0 (socket says 172.21.15.27:41014) 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffab5200640 1 -- 172.21.15.27:0/246677783 learned_addr learned my addr 172.21.15.27:0/246677783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:35.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffaafe00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffab00727e0 0x7ffab01e1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffaaf400640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab00c59d0 0x7ffab01ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffab5200640 1 -- 172.21.15.27:0/246677783 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffab00727e0 msgr2=0x7ffab01e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffab5200640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffab00727e0 0x7ffab01e1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffab5200640 1 -- 172.21.15.27:0/246677783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab00c59d0 msgr2=0x7ffab01ebce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffab5200640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab00c59d0 0x7ffab01ebce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.308+0000 7ffab5200640 1 -- 172.21.15.27:0/246677783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab01ef590 con 0x7ffab0155600 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.309+0000 7ffab5200640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab0155600 0x7ffab00ab900 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8007e60 tx=0x7ffaa800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.309+0000 7ffaa7e00640 1 -- 172.21.15.27:0/246677783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8014420 con 0x7ffab0155600 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.309+0000 7ffaa7e00640 1 -- 172.21.15.27:0/246677783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa8019070 con 0x7ffab0155600 2024-03-28T19:02:35.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.309+0000 7ffaa7e00640 1 -- 172.21.15.27:0/246677783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8014820 con 0x7ffab0155600 2024-03-28T19:02:35.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.309+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab01ef870 con 0x7ffab0155600 2024-03-28T19:02:35.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.309+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab00c4b70 con 0x7ffab0155600 2024-03-28T19:02:35.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.311+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab0067dd0 con 0x7ffab0155600 2024-03-28T19:02:35.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.312+0000 7ffaa7e00640 1 -- 172.21.15.27:0/246677783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffaa80149c0 con 0x7ffab0155600 2024-03-28T19:02:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.313+0000 7ffaa7e00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffa9006bc20 0x7ffa9006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.313+0000 7ffaa7e00640 1 -- 172.21.15.27:0/246677783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7411+0+0 (secure 0 0 0) 0x7ffaa808b8b0 con 0x7ffab0155600 2024-03-28T19:02:35.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.316+0000 7ffaafe00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffa9006bc20 0x7ffa9006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.317+0000 7ffaafe00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffa9006bc20 0x7ffa9006e0e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0035000 tx=0x7ffaa0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:35.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.318+0000 7ffaa7e00640 1 -- 172.21.15.27:0/246677783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffaa8059de0 con 0x7ffab0155600 2024-03-28T19:02:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.602+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffab00733c0 con 0x7ffab0155600 2024-03-28T19:02:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.603+0000 7ffaa7e00640 1 -- 172.21.15.27:0/246677783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5460 (secure 0 0 0) 0x7ffab00733c0 con 0x7ffab0155600 2024-03-28T19:02:35.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:35.603 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:34.697300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:02:35.603 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 61 2024-03-28T19:02:35.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.605+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffa9006bc20 msgr2=0x7ffa9006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:35.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffa9006bc20 0x7ffa9006e0e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0035000 tx=0x7ffaa0008030 comp rx=0 tx=0).stop 2024-03-28T19:02:35.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab0155600 msgr2=0x7ffab00ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:35.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab0155600 0x7ffab00ab900 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8007e60 tx=0x7ffaa800e4b0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 shutdown_connections 2024-03-28T19:02:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffa9006bc20 0x7ffa9006e0e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab0155600 0x7ffab00ab900 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffab00c59d0 0x7ffab01ebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 --2- 172.21.15.27:0/246677783 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffab00727e0 0x7ffab01e1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 >> 172.21.15.27:0/246677783 conn(0x7ffab00ac910 msgr2=0x7ffab0153c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 shutdown_connections 2024-03-28T19:02:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.606+0000 7ffab5c00640 1 -- 172.21.15.27:0/246677783 wait complete. 2024-03-28T19:02:35.628 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-03-28T19:02:35.628 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:02:35.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.840+0000 7f47f1a00640 1 Processor -- start 2024-03-28T19:02:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.840+0000 7f47f1a00640 1 -- start start 2024-03-28T19:02:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47f1a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f47ec152bd0 0x7f47ec06eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47f1a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec06f1e0 0x7f47ec070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47f1a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47ec0c4dc0 0x7f47ec158c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47f1a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec0c52e0 con 0x7f47ec0c4dc0 2024-03-28T19:02:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47f1a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec1557e0 con 0x7f47ec152bd0 2024-03-28T19:02:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47f1a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec155ac0 con 0x7f47ec06f1e0 2024-03-28T19:02:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47eb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47ec0c4dc0 0x7f47ec158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47eb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47ec0c4dc0 0x7f47ec158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41020/0 (socket says 172.21.15.27:41020) 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47eaa00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f47ec152bd0 0x7f47ec06eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47eb400640 1 -- 172.21.15.27:0/3162184803 learned_addr learned my addr 172.21.15.27:0/3162184803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.842+0000 7f47ea000640 1 --2- 172.21.15.27:0/3162184803 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec06f1e0 0x7f47ec070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.843+0000 7f47eb400640 1 -- 172.21.15.27:0/3162184803 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec06f1e0 msgr2=0x7f47ec070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.843+0000 7f47eb400640 1 --2- 172.21.15.27:0/3162184803 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec06f1e0 0x7f47ec070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.843+0000 7f47eb400640 1 -- 172.21.15.27:0/3162184803 >> v2:172.21.15.155:3300/0 conn(0x7f47ec152bd0 msgr2=0x7f47ec06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.843+0000 7f47eb400640 1 --2- 172.21.15.27:0/3162184803 >> v2:172.21.15.155:3300/0 conn(0x7f47ec152bd0 0x7f47ec06eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.843+0000 7f47eb400640 1 -- 172.21.15.27:0/3162184803 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec155da0 con 0x7f47ec0c4dc0 2024-03-28T19:02:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.844+0000 7f47eb400640 1 --2- 172.21.15.27:0/3162184803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec0c4dc0 0x7f47ec158c30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06ec70 tx=0x7f47e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f2088cd456f4d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.845+0000 7f47e9600640 1 -- 172.21.15.27:0/3162184803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e402ca70 con 0x7f47ec0c4dc0 2024-03-28T19:02:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.845+0000 7f47e9600640 1 -- 172.21.15.27:0/3162184803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e402cc10 con 0x7f47ec0c4dc0 2024-03-28T19:02:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.845+0000 7f47e9600640 1 -- 172.21.15.27:0/3162184803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e40343f0 con 0x7f47ec0c4dc0 2024-03-28T19:02:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.845+0000 7f47f1a00640 1 -- 172.21.15.27:0/3162184803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec0c4dc0 msgr2=0x7f47ec158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:35.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.845+0000 7f47f1a00640 1 --2- 172.21.15.27:0/3162184803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec0c4dc0 0x7f47ec158c30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06ec70 tx=0x7f47e402bed0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.846+0000 7f47f1a00640 1 -- 172.21.15.27:0/3162184803 shutdown_connections 2024-03-28T19:02:35.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.846+0000 7f47f1a00640 1 --2- 172.21.15.27:0/3162184803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec0c4dc0 0x7f47ec158c30 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.846+0000 7f47f1a00640 1 --2- 172.21.15.27:0/3162184803 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec06f1e0 0x7f47ec070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.846+0000 7f47f1a00640 1 --2- 172.21.15.27:0/3162184803 >> v2:172.21.15.155:3300/0 conn(0x7f47ec152bd0 0x7f47ec06eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.846+0000 7f47f1a00640 1 -- 172.21.15.27:0/3162184803 >> 172.21.15.27:0/3162184803 conn(0x7f47ec066dd0 msgr2=0x7f47ec0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.846+0000 7f47f1a00640 1 -- 172.21.15.27:0/3162184803 shutdown_connections 2024-03-28T19:02:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.846+0000 7f47f1a00640 1 -- 172.21.15.27:0/3162184803 wait complete. 2024-03-28T19:02:35.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.848+0000 7f47f1a00640 1 Processor -- start 2024-03-28T19:02:35.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.848+0000 7f47f1a00640 1 -- start start 2024-03-28T19:02:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47f1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec06f1e0 0x7f47ec14e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47f1a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec0c4dc0 0x7f47ec0c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47f1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47ec152bd0 0x7f47ec1e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47f1a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec0c6df0 con 0x7f47ec06f1e0 2024-03-28T19:02:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47f1a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec0c6c70 con 0x7f47ec152bd0 2024-03-28T19:02:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47f1a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec070a80 con 0x7f47ec0c4dc0 2024-03-28T19:02:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec06f1e0 0x7f47ec14e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec06f1e0 0x7f47ec14e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41034/0 (socket says 172.21.15.27:41034) 2024-03-28T19:02:35.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47eaa00640 1 -- 172.21.15.27:0/2861689818 learned_addr learned my addr 172.21.15.27:0/2861689818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:35.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47eaa00640 1 -- 172.21.15.27:0/2861689818 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec0c4dc0 msgr2=0x7f47ec0c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:35.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47eaa00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec0c4dc0 0x7f47ec0c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47eaa00640 1 -- 172.21.15.27:0/2861689818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47ec152bd0 msgr2=0x7f47ec1e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:35.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47eaa00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47ec152bd0 0x7f47ec1e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:35.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.849+0000 7f47eaa00640 1 -- 172.21.15.27:0/2861689818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec1ef3d0 con 0x7f47ec06f1e0 2024-03-28T19:02:35.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.850+0000 7f47eaa00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec06f1e0 0x7f47ec14e1b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00e9e0 tx=0x7f47dc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.850+0000 7f47e2a00640 1 -- 172.21.15.27:0/2861689818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc017070 con 0x7f47ec06f1e0 2024-03-28T19:02:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.850+0000 7f47e2a00640 1 -- 172.21.15.27:0/2861689818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc00c9d0 con 0x7f47ec06f1e0 2024-03-28T19:02:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.850+0000 7f47e2a00640 1 -- 172.21.15.27:0/2861689818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc00cdc0 con 0x7f47ec06f1e0 2024-03-28T19:02:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.850+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec0adcb0 con 0x7f47ec06f1e0 2024-03-28T19:02:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.850+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec1ef060 con 0x7f47ec06f1e0 2024-03-28T19:02:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.852+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec04f400 con 0x7f47ec06f1e0 2024-03-28T19:02:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.853+0000 7f47e2a00640 1 -- 172.21.15.27:0/2861689818 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f47dc01a070 con 0x7f47ec06f1e0 2024-03-28T19:02:35.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.854+0000 7f47e2a00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f47cc06bc20 0x7f47cc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.854+0000 7f47e2a00640 1 -- 172.21.15.27:0/2861689818 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7411+0+0 (secure 0 0 0) 0x7f47dc08b360 con 0x7f47ec06f1e0 2024-03-28T19:02:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.854+0000 7f47ea000640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f47cc06bc20 0x7f47cc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:35.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.855+0000 7f47ea000640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f47cc06bc20 0x7f47cc06e0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06f5c0 tx=0x7f47d8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:35.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:35.859+0000 7f47e2a00640 1 -- 172.21.15.27:0/2861689818 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f47dc059910 con 0x7f47ec06f1e0 2024-03-28T19:02:36.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.138+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47ec154e40 con 0x7f47ec06f1e0 2024-03-28T19:02:36.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.139+0000 7f47e2a00640 1 -- 172.21.15.27:0/2861689818 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5460 (secure 0 0 0) 0x7f47dc059210 con 0x7f47ec06f1e0 2024-03-28T19:02:36.138 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:36.139 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:34.697300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:02:36.139 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 61 2024-03-28T19:02:36.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f47cc06bc20 msgr2=0x7f47cc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:36.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f47cc06bc20 0x7f47cc06e0e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06f5c0 tx=0x7f47d8008040 comp rx=0 tx=0).stop 2024-03-28T19:02:36.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec06f1e0 msgr2=0x7f47ec14e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:36.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec06f1e0 0x7f47ec14e1b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00e9e0 tx=0x7f47dc00eea0 comp rx=0 tx=0).stop 2024-03-28T19:02:36.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 shutdown_connections 2024-03-28T19:02:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f47cc06bc20 0x7f47cc06e0e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47ec152bd0 0x7f47ec1e1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47ec0c4dc0 0x7f47ec0c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 --2- 172.21.15.27:0/2861689818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47ec06f1e0 0x7f47ec14e1b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 >> 172.21.15.27:0/2861689818 conn(0x7f47ec066dd0 msgr2=0x7f47ec061bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 shutdown_connections 2024-03-28T19:02:36.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:36.142+0000 7f47f1a00640 1 -- 172.21.15.27:0/2861689818 wait complete. 2024-03-28T19:02:36.161 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 61, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 59, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4247": { "addr": "172.21.15.155:6835/1254917384", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1254917384, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1254917384, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 55, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 99 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:01:49.910084+0000", "qdb_cluster": [ 4247 ], "qdb_leader": 4247, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4247 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 61, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5769": { "addr": "172.21.15.27:6836/1001046540", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1001046540, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 1001046540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5769, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:02:34.697300+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5769 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:02:38.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:38.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.396+0000 7fedfd000640 1 Processor -- start 2024-03-28T19:02:38.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.396+0000 7fedfd000640 1 -- start start 2024-03-28T19:02:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedfd000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf8156120 0x7fedf806e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedfd000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fedf80c5320 0x7fedf80c0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedfd000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fedf8156500 0x7fedf80c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedfd000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf80c4d70 con 0x7fedf8156500 2024-03-28T19:02:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedfd000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf8159c50 con 0x7fedf80c5320 2024-03-28T19:02:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedfd000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c4a40 con 0x7fedf8156120 2024-03-28T19:02:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedf7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fedf8156500 0x7fedf80c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedf7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fedf8156500 0x7fedf80c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34902/0 (socket says 172.21.15.27:34902) 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.398+0000 7fedf7e00640 1 -- 172.21.15.27:0/3206491096 learned_addr learned my addr 172.21.15.27:0/3206491096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.399+0000 7fedf7400640 1 --2- 172.21.15.27:0/3206491096 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf8156120 0x7fedf806e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.399+0000 7fedefe00640 1 --2- 172.21.15.27:0/3206491096 >> v2:172.21.15.155:3300/0 conn(0x7fedf80c5320 0x7fedf80c0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.399+0000 7fedf7e00640 1 -- 172.21.15.27:0/3206491096 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf8156120 msgr2=0x7fedf806e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.399+0000 7fedf7e00640 1 --2- 172.21.15.27:0/3206491096 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf8156120 0x7fedf806e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.399+0000 7fedf7e00640 1 -- 172.21.15.27:0/3206491096 >> v2:172.21.15.155:3300/0 conn(0x7fedf80c5320 msgr2=0x7fedf80c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.399+0000 7fedf7e00640 1 --2- 172.21.15.27:0/3206491096 >> v2:172.21.15.155:3300/0 conn(0x7fedf80c5320 0x7fedf80c0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.399+0000 7fedf7e00640 1 -- 172.21.15.27:0/3206491096 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf8159970 con 0x7fedf8156500 2024-03-28T19:02:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.400+0000 7fedf7e00640 1 --2- 172.21.15.27:0/3206491096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156500 0x7fedf80c6730 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009e50 tx=0x7fedf002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24feeb6f242a8455 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedf6a00640 1 -- 172.21.15.27:0/3206491096 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf002c960 con 0x7fedf8156500 2024-03-28T19:02:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedf6a00640 1 -- 172.21.15.27:0/3206491096 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf002cb00 con 0x7fedf8156500 2024-03-28T19:02:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedf6a00640 1 -- 172.21.15.27:0/3206491096 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf00343f0 con 0x7fedf8156500 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 -- 172.21.15.27:0/3206491096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156500 msgr2=0x7fedf80c6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 --2- 172.21.15.27:0/3206491096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156500 0x7fedf80c6730 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009e50 tx=0x7fedf002bdc0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 -- 172.21.15.27:0/3206491096 shutdown_connections 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 --2- 172.21.15.27:0/3206491096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156500 0x7fedf80c6730 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 --2- 172.21.15.27:0/3206491096 >> v2:172.21.15.155:3300/0 conn(0x7fedf80c5320 0x7fedf80c0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 --2- 172.21.15.27:0/3206491096 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf8156120 0x7fedf806e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 -- 172.21.15.27:0/3206491096 >> 172.21.15.27:0/3206491096 conn(0x7fedf806b0c0 msgr2=0x7fedf806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 -- 172.21.15.27:0/3206491096 shutdown_connections 2024-03-28T19:02:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.401+0000 7fedfd000640 1 -- 172.21.15.27:0/3206491096 wait complete. 2024-03-28T19:02:38.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.403+0000 7fedfd000640 1 Processor -- start 2024-03-28T19:02:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.404+0000 7fedfd000640 1 -- start start 2024-03-28T19:02:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedfd000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf80c5320 0x7fedf81ec270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedfd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156120 0x7fedf8150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedfd000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fedf8156500 0x7fedf8151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedfd000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c73d0 con 0x7fedf8156120 2024-03-28T19:02:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedfd000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c7250 con 0x7fedf8156500 2024-03-28T19:02:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedfd000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c29a0 con 0x7fedf80c5320 2024-03-28T19:02:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedefe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156120 0x7fedf8150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedefe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156120 0x7fedf8150c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34910/0 (socket says 172.21.15.27:34910) 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedefe00640 1 -- 172.21.15.27:0/423858672 learned_addr learned my addr 172.21.15.27:0/423858672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedf7400640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf80c5320 0x7fedf81ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedf7e00640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fedf8156500 0x7fedf8151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedefe00640 1 -- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf80c5320 msgr2=0x7fedf81ec270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedefe00640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf80c5320 0x7fedf81ec270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedefe00640 1 -- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fedf8156500 msgr2=0x7fedf8151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedefe00640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fedf8156500 0x7fedf8151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedefe00640 1 -- 172.21.15.27:0/423858672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80a8d40 con 0x7fedf8156120 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.405+0000 7fedf7400640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf80c5320 0x7fedf81ec270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:38.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.406+0000 7fedefe00640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156120 0x7fedf8150c50 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fede000c7f0 tx=0x7fede000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.406+0000 7fedef400640 1 -- 172.21.15.27:0/423858672 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0015070 con 0x7fedf8156120 2024-03-28T19:02:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.406+0000 7fedef400640 1 -- 172.21.15.27:0/423858672 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede0004240 con 0x7fedf8156120 2024-03-28T19:02:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.406+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf8072300 con 0x7fedf8156120 2024-03-28T19:02:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.406+0000 7fedef400640 1 -- 172.21.15.27:0/423858672 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0007620 con 0x7fedf8156120 2024-03-28T19:02:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.406+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf8154230 con 0x7fedf8156120 2024-03-28T19:02:38.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.408+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf8067dd0 con 0x7fedf8156120 2024-03-28T19:02:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.409+0000 7fedef400640 1 -- 172.21.15.27:0/423858672 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fede00079d0 con 0x7fedf8156120 2024-03-28T19:02:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.409+0000 7fedef400640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fedd406bbd0 0x7fedd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.410+0000 7fedef400640 1 -- 172.21.15.27:0/423858672 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7411+0+0 (secure 0 0 0) 0x7fede008b180 con 0x7fedf8156120 2024-03-28T19:02:38.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.410+0000 7fedf7400640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fedd406bbd0 0x7fedd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:38.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.411+0000 7fedf7400640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fedd406bbd0 0x7fedd406e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fede8006fd0 tx=0x7fede8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:38.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.414+0000 7fedef400640 1 -- 172.21.15.27:0/423858672 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fede0059650 con 0x7fedf8156120 2024-03-28T19:02:38.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.704+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedf8069640 con 0x7fedf8156120 2024-03-28T19:02:38.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.706+0000 7fedef400640 1 -- 172.21.15.27:0/423858672 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5464 (secure 0 0 0) 0x7fede0058f50 con 0x7fedf8156120 2024-03-28T19:02:38.706 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:38.706 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:37.285730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:02:38.706 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 62 2024-03-28T19:02:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.708+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fedd406bbd0 msgr2=0x7fedd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.708+0000 7fedfd000640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fedd406bbd0 0x7fedd406e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fede8006fd0 tx=0x7fede8008040 comp rx=0 tx=0).stop 2024-03-28T19:02:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156120 msgr2=0x7fedf8150c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:38.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156120 0x7fedf8150c50 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fede000c7f0 tx=0x7fede000ccb0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 shutdown_connections 2024-03-28T19:02:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fedd406bbd0 0x7fedd406e090 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fedf8156500 0x7fedf8151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf8156120 0x7fedf8150c50 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 --2- 172.21.15.27:0/423858672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fedf80c5320 0x7fedf81ec270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 >> 172.21.15.27:0/423858672 conn(0x7fedf806b0c0 msgr2=0x7fedf8155330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 shutdown_connections 2024-03-28T19:02:38.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:38.709+0000 7fedfd000640 1 -- 172.21.15.27:0/423858672 wait complete. 2024-03-28T19:02:38.727 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:02:40.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:40.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.960+0000 7fdd2ca00640 1 Processor -- start 2024-03-28T19:02:40.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.960+0000 7fdd2ca00640 1 -- start start 2024-03-28T19:02:40.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.961+0000 7fdd2ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd280c58c0 0x7fdd280c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.961+0000 7fdd2ca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd28158bd0 con 0x7fdd280be5d0 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd2ca00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd28158eb0 con 0x7fdd280c8ed0 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd2ca00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd28159190 con 0x7fdd280c58c0 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd26800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd280c58c0 0x7fdd280c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd26800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd280c58c0 0x7fdd280c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34924/0 (socket says 172.21.15.27:34924) 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd26800640 1 -- 172.21.15.27:0/3345080132 learned_addr learned my addr 172.21.15.27:0/3345080132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd27200640 1 --1- 172.21.15.27:0/3345080132 >> v1:172.21.15.27:6789/0 conn(0x7fdd280be5d0 0x7fdd28155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60778/0 (socket says 172.21.15.27:60778) 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd26800640 1 -- 172.21.15.27:0/3345080132 >> v1:172.21.15.155:6789/0 conn(0x7fdd280c8ed0 legacy=0x7fdd281555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd26800640 1 -- 172.21.15.27:0/3345080132 >> v1:172.21.15.27:6789/0 conn(0x7fdd280be5d0 legacy=0x7fdd28155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:40.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.962+0000 7fdd26800640 1 -- 172.21.15.27:0/3345080132 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28159470 con 0x7fdd280c58c0 2024-03-28T19:02:40.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.964+0000 7fdd26800640 1 --2- 172.21.15.27:0/3345080132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280c58c0 0x7fdd280c5ca0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009870 tx=0x7fdd1802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47105e04ead90bf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:40.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.964+0000 7fdd25400640 1 -- 172.21.15.27:0/3345080132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1802e070 con 0x7fdd280c58c0 2024-03-28T19:02:40.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.964+0000 7fdd25400640 1 -- 172.21.15.27:0/3345080132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd18037070 con 0x7fdd280c58c0 2024-03-28T19:02:40.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.964+0000 7fdd25400640 1 -- 172.21.15.27:0/3345080132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18031a00 con 0x7fdd280c58c0 2024-03-28T19:02:40.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.964+0000 7fdd2ca00640 1 -- 172.21.15.27:0/3345080132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280c58c0 msgr2=0x7fdd280c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:40.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.964+0000 7fdd2ca00640 1 --2- 172.21.15.27:0/3345080132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280c58c0 0x7fdd280c5ca0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009870 tx=0x7fdd1802b880 comp rx=0 tx=0).stop 2024-03-28T19:02:40.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.965+0000 7fdd2ca00640 1 -- 172.21.15.27:0/3345080132 shutdown_connections 2024-03-28T19:02:40.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.965+0000 7fdd2ca00640 1 --2- 172.21.15.27:0/3345080132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280c58c0 0x7fdd280c5ca0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:40.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.965+0000 7fdd2ca00640 1 -- 172.21.15.27:0/3345080132 >> 172.21.15.27:0/3345080132 conn(0x7fdd2806b6f0 msgr2=0x7fdd280acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:40.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.965+0000 7fdd2ca00640 1 -- 172.21.15.27:0/3345080132 shutdown_connections 2024-03-28T19:02:40.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.965+0000 7fdd2ca00640 1 -- 172.21.15.27:0/3345080132 wait complete. 2024-03-28T19:02:40.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.967+0000 7fdd2ca00640 1 Processor -- start 2024-03-28T19:02:40.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.967+0000 7fdd2ca00640 1 -- start start 2024-03-28T19:02:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd2ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280be5d0 0x7fdd280c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd2ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd280c58c0 0x7fdd280c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd2ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd280c8ed0 0x7fdd28153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd2ca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280b09a0 con 0x7fdd280be5d0 2024-03-28T19:02:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd2ca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280b0820 con 0x7fdd280c8ed0 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd2ca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280ad7d0 con 0x7fdd280c58c0 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd26800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280be5d0 0x7fdd280c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd26800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280be5d0 0x7fdd280c2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34936/0 (socket says 172.21.15.27:34936) 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.968+0000 7fdd26800640 1 -- 172.21.15.27:0/4049387498 learned_addr learned my addr 172.21.15.27:0/4049387498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd25e00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd280c58c0 0x7fdd280c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd26800640 1 -- 172.21.15.27:0/4049387498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd280c58c0 msgr2=0x7fdd280c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd27200640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd280c8ed0 0x7fdd28153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd26800640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd280c58c0 0x7fdd280c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd26800640 1 -- 172.21.15.27:0/4049387498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd280c8ed0 msgr2=0x7fdd28153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd26800640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd280c8ed0 0x7fdd28153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd26800640 1 -- 172.21.15.27:0/4049387498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28154f00 con 0x7fdd280be5d0 2024-03-28T19:02:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd26800640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280be5d0 0x7fdd280c2670 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fdd18000c00 tx=0x7fdd180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.969+0000 7fdd1ea00640 1 -- 172.21.15.27:0/4049387498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18032880 con 0x7fdd280be5d0 2024-03-28T19:02:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.970+0000 7fdd1ea00640 1 -- 172.21.15.27:0/4049387498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd18032a40 con 0x7fdd280be5d0 2024-03-28T19:02:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.970+0000 7fdd1ea00640 1 -- 172.21.15.27:0/4049387498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18032e30 con 0x7fdd280be5d0 2024-03-28T19:02:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.970+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd281551e0 con 0x7fdd280be5d0 2024-03-28T19:02:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.970+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd281ef330 con 0x7fdd280be5d0 2024-03-28T19:02:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.971+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd28069640 con 0x7fdd280be5d0 2024-03-28T19:02:40.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.972+0000 7fdd1ea00640 1 -- 172.21.15.27:0/4049387498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdd1804c060 con 0x7fdd280be5d0 2024-03-28T19:02:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.973+0000 7fdd1ea00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd0806bbd0 0x7fdd0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.973+0000 7fdd1ea00640 1 -- 172.21.15.27:0/4049387498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7411+0+0 (secure 0 0 0) 0x7fdd180aca40 con 0x7fdd280be5d0 2024-03-28T19:02:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.974+0000 7fdd25e00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd0806bbd0 0x7fdd0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:40.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.975+0000 7fdd25e00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd0806bbd0 0x7fdd0806e090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdd14006fd0 tx=0x7fdd14008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:40.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:40.979+0000 7fdd1ea00640 1 -- 172.21.15.27:0/4049387498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdd18040070 con 0x7fdd280be5d0 2024-03-28T19:02:41.220 INFO:tasks.workunit.client.1.smithi155.stderr:+ ln foo/x foo/xx 2024-03-28T19:02:41.238 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat foo/.snap/ss/x 2024-03-28T19:02:41.250 INFO:tasks.workunit.client.1.smithi155.stdout:x 2024-03-28T19:02:41.251 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir foo/.snap/ss 2024-03-28T19:02:41.272 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo a 2024-03-28T19:02:41.278 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo b 2024-03-28T19:02:41.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.280+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd280cbc70 con 0x7fdd280be5d0 2024-03-28T19:02:41.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.281+0000 7fdd1ea00640 1 -- 172.21.15.27:0/4049387498 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5468 (secure 0 0 0) 0x7fdd1807a820 con 0x7fdd280be5d0 2024-03-28T19:02:41.281 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:41.281 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:41.282 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 65 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd0806bbd0 msgr2=0x7fdd0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd0806bbd0 0x7fdd0806e090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdd14006fd0 tx=0x7fdd14008040 comp rx=0 tx=0).stop 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280be5d0 msgr2=0x7fdd280c2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280be5d0 0x7fdd280c2670 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fdd18000c00 tx=0x7fdd180046c0 comp rx=0 tx=0).stop 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 shutdown_connections 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd0806bbd0 0x7fdd0806e090 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd280c8ed0 0x7fdd28153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd280c58c0 0x7fdd280c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 --2- 172.21.15.27:0/4049387498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd280be5d0 0x7fdd280c2670 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:41.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 >> 172.21.15.27:0/4049387498 conn(0x7fdd2806b6f0 msgr2=0x7fdd280acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:41.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 shutdown_connections 2024-03-28T19:02:41.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:41.284+0000 7fdd2ca00640 1 -- 172.21.15.27:0/4049387498 wait complete. 2024-03-28T19:02:41.289 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir foo/.snap/s 2024-03-28T19:02:41.307 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5302 2024-03-28T19:02:41.308 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.c 2024-03-28T19:02:41.310 INFO:tasks.workunit.client.1.smithi155.stderr:++ cat foo/.snap/s/a 2024-03-28T19:02:41.328 INFO:tasks.workunit.client.1.smithi155.stderr:+ r=a 2024-03-28T19:02:41.328 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' -z a ']' 2024-03-28T19:02:41.328 INFO:tasks.workunit.client.1.smithi155.stderr:+ ln foo/b foo/b2 2024-03-28T19:02:41.338 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat foo/.snap/s/b 2024-03-28T19:02:41.351 INFO:tasks.workunit.client.1.smithi155.stdout:b 2024-03-28T19:02:41.352 INFO:tasks.workunit.client.1.smithi155.stdout:this used to hang: 2024-03-28T19:02:41.352 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 'this used to hang:' 2024-03-28T19:02:41.352 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo more 2024-03-28T19:02:41.357 INFO:tasks.workunit.client.1.smithi155.stdout:oh, it didn't hang! good job. 2024-03-28T19:02:41.357 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2024-03-28T19:02:41.357 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat foo/b 2024-03-28T19:02:41.365 INFO:tasks.workunit.client.1.smithi155.stdout:b 2024-03-28T19:02:41.365 INFO:tasks.workunit.client.1.smithi155.stdout:more 2024-03-28T19:02:41.366 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir foo/.snap/s 2024-03-28T19:02:41.386 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm -r foo 2024-03-28T19:02:41.446 INFO:tasks.workunit.client.1.smithi155.stdout:OK 2024-03-28T19:02:41.446 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo OK 2024-03-28T19:02:41.448 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:02:41.448 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-28T19:02:41.561 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-03-28T19:02:41.561 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-03-28T19:02:41.644 INFO:tasks.workunit.client.1.smithi155.stderr:+ set -e 2024-03-28T19:02:41.645 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:41.649 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/bar 2024-03-28T19:02:41.667 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/foo 2024-03-28T19:02:41.681 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/foo 2024-03-28T19:02:41.697 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:41.703 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/bar/a 2024-03-28T19:02:41.717 INFO:tasks.workunit.client.1.smithi155.stdout: File: .snap/bar/a 2024-03-28T19:02:41.717 INFO:tasks.workunit.client.1.smithi155.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-28T19:02:41.717 INFO:tasks.workunit.client.1.smithi155.stdout:Device: 2ah/42d Inode: 845524441765765 Links: 2 2024-03-28T19:02:41.717 INFO:tasks.workunit.client.1.smithi155.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-28T19:02:41.717 INFO:tasks.workunit.client.1.smithi155.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-28T19:02:41.717 INFO:tasks.workunit.client.1.smithi155.stdout:Access: 2024-03-28 19:02:41.648954430 +0000 2024-03-28T19:02:41.717 INFO:tasks.workunit.client.1.smithi155.stdout:Modify: 2024-03-28 19:02:41.648954430 +0000 2024-03-28T19:02:41.718 INFO:tasks.workunit.client.1.smithi155.stdout:Change: 2024-03-28 19:02:41.648954430 +0000 2024-03-28T19:02:41.718 INFO:tasks.workunit.client.1.smithi155.stdout: Birth: - 2024-03-28T19:02:41.718 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir .snap/bar 2024-03-28T19:02:41.734 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo OK 2024-03-28T19:02:41.735 INFO:tasks.workunit.client.1.smithi155.stdout:OK 2024-03-28T19:02:41.736 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:02:41.736 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-28T19:02:41.822 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-03-28T19:02:41.823 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-03-28T19:02:41.902 INFO:tasks.workunit.client.1.smithi155.stderr:+ set -e 2024-03-28T19:02:41.902 INFO:tasks.workunit.client.1.smithi155.stderr:++ seq 1 20 2024-03-28T19:02:41.904 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:41.905 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:41.913 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:41.913 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:41.938 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:41.957 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:41.964 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:42.308 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-03-28T19:02:42.308 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:02:42.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:02:42.318 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:02:42.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:42.495 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:02:42.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.539+0000 7f6077200640 1 Processor -- start 2024-03-28T19:02:42.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.540+0000 7f6077200640 1 -- start start 2024-03-28T19:02:42.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.541+0000 7f6077200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60780be5d0 0x7f60780c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:42.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.541+0000 7f6077200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078158bd0 con 0x7f60780c8ed0 2024-03-28T19:02:42.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.541+0000 7f6077200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6078158eb0 con 0x7f60780c58c0 2024-03-28T19:02:42.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.541+0000 7f6077200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6078159190 con 0x7f60780be5d0 2024-03-28T19:02:42.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.541+0000 7f6075400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f60780c8ed0 0x7f6078155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60800/0 (socket says 172.21.15.27:60800) 2024-03-28T19:02:42.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.541+0000 7f6075400640 1 -- 172.21.15.27:0/1673069256 learned_addr learned my addr 172.21.15.27:0/1673069256 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:42.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6076800640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60780be5d0 0x7f60780c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:42.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641202719 0 0) 0x7f6078158bd0 con 0x7f60780c8ed0 2024-03-28T19:02:42.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f605c003520 con 0x7f60780c8ed0 2024-03-28T19:02:42.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750518166 0 0) 0x7f605c003520 con 0x7f60780c8ed0 2024-03-28T19:02:42.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60780be5d0 msgr2=0x7f60780c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60780be5d0 0x7f60780c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 >> v1:172.21.15.155:6789/0 conn(0x7f60780c58c0 legacy=0x7f60781555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078159470 con 0x7f60780c8ed0 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f6064001670 con 0x7f60780c8ed0 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.542+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 >> v1:172.21.15.27:6789/0 conn(0x7f60780c8ed0 legacy=0x7f6078155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6074a00640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f605c0039e0 0x7f605c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6074a00640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60780be5d0 0x7f605c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6074a00640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c004d40 0x7f605c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6078159190 con 0x7f605c0039e0 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c00baf0 con 0x7f605c004d40 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c00bcf0 con 0x7f60780be5d0 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6075e00640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f605c0039e0 0x7f605c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6076800640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c004d40 0x7f605c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:42.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6075e00640 1 -- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60780be5d0 msgr2=0x7f605c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:42.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.543+0000 7f6075400640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60780be5d0 0x7f605c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:42.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.544+0000 7f6075e00640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60780be5d0 0x7f605c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.544+0000 7f6075e00640 1 -- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c004d40 msgr2=0x7f605c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:42.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.544+0000 7f6075e00640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c004d40 0x7f605c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.544+0000 7f6075e00640 1 -- 172.21.15.27:0/1673069256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c003520 con 0x7f605c0039e0 2024-03-28T19:02:42.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.544+0000 7f6076800640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c004d40 0x7f605c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:42.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.545+0000 7f6075e00640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f605c0039e0 0x7f605c003e00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6068001750 tx=0x7f605c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68990a308c75770e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:42.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.545+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068001370 con 0x7f605c0039e0 2024-03-28T19:02:42.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.545+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60680052d0 con 0x7f605c0039e0 2024-03-28T19:02:42.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.545+0000 7f6074a00640 1 -- 172.21.15.27:0/1673069256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068003830 con 0x7f605c0039e0 2024-03-28T19:02:42.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.546+0000 7f6077200640 1 -- 172.21.15.27:0/1673069256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f605c0039e0 msgr2=0x7f605c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:42.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.546+0000 7f6077200640 1 --2- 172.21.15.27:0/1673069256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f605c0039e0 0x7f605c003e00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6068001750 tx=0x7f605c005830 comp rx=0 tx=0).stop 2024-03-28T19:02:42.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.546+0000 7f6076800640 1 -- 172.21.15.27:0/1673069256 reap_dead start 2024-03-28T19:02:42.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.546+0000 7f6077200640 1 -- 172.21.15.27:0/1673069256 shutdown_connections 2024-03-28T19:02:42.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.546+0000 7f6077200640 1 -- 172.21.15.27:0/1673069256 >> 172.21.15.27:0/1673069256 conn(0x7f607806b6f0 msgr2=0x7f60780acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:42.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.546+0000 7f6077200640 1 -- 172.21.15.27:0/1673069256 shutdown_connections 2024-03-28T19:02:42.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.546+0000 7f6077200640 1 -- 172.21.15.27:0/1673069256 wait complete. 2024-03-28T19:02:42.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.548+0000 7f6077200640 1 Processor -- start 2024-03-28T19:02:42.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.549+0000 7f6077200640 1 -- start start 2024-03-28T19:02:42.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.549+0000 7f6077200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f605c0039e0 0x7f60781530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:42.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6077200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60781528a0 0x7f6078157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:42.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6077200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60781ebc60 0x7f60781ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:42.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6077200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60780b09a0 con 0x7f60781ebc60 2024-03-28T19:02:42.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6077200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60780b0820 con 0x7f60781528a0 2024-03-28T19:02:42.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6077200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60780ad7d0 con 0x7f605c0039e0 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6076800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60781ebc60 0x7f60781ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6076800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60781ebc60 0x7f60781ec0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34970/0 (socket says 172.21.15.27:34970) 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6076800640 1 -- 172.21.15.27:0/4081088017 learned_addr learned my addr 172.21.15.27:0/4081088017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6075400640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60781528a0 0x7f6078157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6075e00640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f605c0039e0 0x7f60781530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6076800640 1 -- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f605c0039e0 msgr2=0x7f60781530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6076800640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f605c0039e0 0x7f60781530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6076800640 1 -- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60781528a0 msgr2=0x7f6078157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6076800640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60781528a0 0x7f6078157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.550+0000 7f6076800640 1 -- 172.21.15.27:0/4081088017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60780acbe0 con 0x7f60781ebc60 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.551+0000 7f6075e00640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f605c0039e0 0x7f60781530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.551+0000 7f6076800640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60781ebc60 0x7f60781ec0e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6078062b00 tx=0x7f607000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.551+0000 7f606e000640 1 -- 172.21.15.27:0/4081088017 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607001c070 con 0x7f60781ebc60 2024-03-28T19:02:42.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.551+0000 7f606e000640 1 -- 172.21.15.27:0/4081088017 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6070017400 con 0x7f60781ebc60 2024-03-28T19:02:42.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.551+0000 7f606e000640 1 -- 172.21.15.27:0/4081088017 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60700177d0 con 0x7f60781ebc60 2024-03-28T19:02:42.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.551+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60780c21b0 con 0x7f60781ebc60 2024-03-28T19:02:42.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.552+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60781eb740 con 0x7f60781ebc60 2024-03-28T19:02:42.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.553+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6078069640 con 0x7f60781ebc60 2024-03-28T19:02:42.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.554+0000 7f606e000640 1 -- 172.21.15.27:0/4081088017 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6070005d10 con 0x7f60781ebc60 2024-03-28T19:02:42.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.555+0000 7f606e000640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f605806bbd0 0x7f605806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:42.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.555+0000 7f606e000640 1 -- 172.21.15.27:0/4081088017 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7411+0+0 (secure 0 0 0) 0x7f607008b3a0 con 0x7f60781ebc60 2024-03-28T19:02:42.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.556+0000 7f6075e00640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f605806bbd0 0x7f605806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:42.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.556+0000 7f6075e00640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f605806bbd0 0x7f605806e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6068033000 tx=0x7f6068009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:42.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.560+0000 7f606e000640 1 -- 172.21.15.27:0/4081088017 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6070012070 con 0x7f60781ebc60 2024-03-28T19:02:42.768 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:42.797 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:42.797 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:42.797 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:42.797 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:42.797 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:42.797 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:42.797 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:42.797 INFO:tasks.workunit.client.1.smithi155.stdout: 45484 ? 00:00:00 ps 2024-03-28T19:02:42.805 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:42.828 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:42.842 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:42.853 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:42.854 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:42.866 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:42.866 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:42.886 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:42.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.891+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6078156580 con 0x7f60781ebc60 2024-03-28T19:02:42.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.892+0000 7f606e000640 1 -- 172.21.15.27:0/4081088017 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5468 (secure 0 0 0) 0x7f60700597d0 con 0x7f60781ebc60 2024-03-28T19:02:42.893 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:42.893 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:42.893 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 65 2024-03-28T19:02:42.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.895+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f605806bbd0 msgr2=0x7f605806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:42.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.895+0000 7f6077200640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f605806bbd0 0x7f605806e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6068033000 tx=0x7f6068009030 comp rx=0 tx=0).stop 2024-03-28T19:02:42.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.895+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60781ebc60 msgr2=0x7f60781ec0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:42.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.895+0000 7f6077200640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60781ebc60 0x7f60781ec0e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6078062b00 tx=0x7f607000f4a0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.896+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 shutdown_connections 2024-03-28T19:02:42.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.896+0000 7f6077200640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f605806bbd0 0x7f605806e090 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.896+0000 7f6077200640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60781ebc60 0x7f60781ec0e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.896+0000 7f6077200640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60781528a0 0x7f6078157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.896+0000 7f6077200640 1 --2- 172.21.15.27:0/4081088017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f605c0039e0 0x7f60781530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:42.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.896+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 >> 172.21.15.27:0/4081088017 conn(0x7f607806b6f0 msgr2=0x7f60780aedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:42.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.896+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 shutdown_connections 2024-03-28T19:02:42.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:42.896+0000 7f6077200640 1 -- 172.21.15.27:0/4081088017 wait complete. 2024-03-28T19:02:42.901 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:42.906 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:42.916 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:02:42.999 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:43.019 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:43.019 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:43.019 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:43.019 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:43.019 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:43.019 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:43.019 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:43.019 INFO:tasks.workunit.client.1.smithi155.stdout: 45494 ? 00:00:00 ps 2024-03-28T19:02:43.029 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:43.062 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:43.073 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:43.083 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:43.083 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:43.095 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:43.095 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:43.119 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:43.136 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:43.140 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:43.204 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:43.229 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:43.229 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:43.229 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:43.229 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:43.229 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:43.229 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:43.229 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:43.229 INFO:tasks.workunit.client.1.smithi155.stdout: 45504 ? 00:00:00 ps 2024-03-28T19:02:43.236 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:43.260 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:43.268 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:43.278 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:43.278 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:43.288 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:43.288 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:43.309 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:43.325 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:43.329 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:43.406 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:43.424 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:43.424 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:43.424 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:43.424 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:43.424 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:43.424 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:43.424 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:43.424 INFO:tasks.workunit.client.1.smithi155.stdout: 45514 ? 00:00:00 ps 2024-03-28T19:02:43.434 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:43.463 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:43.477 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:43.490 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:43.490 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:43.503 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:43.504 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:43.528 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:43.544 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:43.549 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:43.605 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:43.623 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:43.623 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:43.623 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:43.624 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:43.624 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:43.624 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:43.624 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:43.624 INFO:tasks.workunit.client.1.smithi155.stdout: 45524 ? 00:00:00 ps 2024-03-28T19:02:43.632 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:43.660 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:43.677 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:43.690 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:43.690 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:43.703 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:43.703 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:43.724 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:43.739 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:43.745 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:43.823 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:43.842 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:43.842 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:43.842 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:43.842 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:43.842 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:43.843 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:43.843 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:43.843 INFO:tasks.workunit.client.1.smithi155.stdout: 45534 ? 00:00:00 ps 2024-03-28T19:02:43.850 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:43.880 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:43.895 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:43.909 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:43.909 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:43.921 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:43.921 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:43.942 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:43.958 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:43.962 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:44.040 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:44.059 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:44.059 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:44.059 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:44.059 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:44.059 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:44.059 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:44.059 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:44.059 INFO:tasks.workunit.client.1.smithi155.stdout: 45544 ? 00:00:00 ps 2024-03-28T19:02:44.067 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:44.093 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:44.104 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:44.114 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:44.114 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:44.127 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:44.128 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:44.151 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:44.170 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:44.176 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:44.240 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:44.259 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:44.259 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:44.259 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:44.259 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:44.259 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:44.259 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:44.259 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:44.259 INFO:tasks.workunit.client.1.smithi155.stdout: 45554 ? 00:00:00 ps 2024-03-28T19:02:44.269 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:44.303 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:44.314 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:44.333 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:44.333 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:44.343 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:44.343 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:44.366 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:44.384 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:44.389 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:44.483 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:44.502 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:44.502 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:44.503 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:44.503 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:44.503 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:44.503 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:44.503 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:44.503 INFO:tasks.workunit.client.1.smithi155.stdout: 45564 ? 00:00:00 ps 2024-03-28T19:02:44.512 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:44.545 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:44.558 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:44.571 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:44.571 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:44.585 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:44.585 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:44.611 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:44.632 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:44.638 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:44.736 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:44.768 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:44.768 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:44.768 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:44.768 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:44.768 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:44.768 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:44.768 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:44.768 INFO:tasks.workunit.client.1.smithi155.stdout: 45574 ? 00:00:00 ps 2024-03-28T19:02:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:44.809 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:44.821 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:44.833 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:44.833 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:44.843 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:44.843 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:44.862 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:44.880 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:44.885 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:44.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:44.967 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:44.986 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:44.986 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:44.986 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:44.986 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:44.986 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:44.986 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:44.986 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:44.986 INFO:tasks.workunit.client.1.smithi155.stdout: 45584 ? 00:00:00 ps 2024-03-28T19:02:44.993 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:45.023 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:45.033 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:45.043 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:45.043 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:45.056 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:45.056 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:45.077 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:45.092 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:45.097 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:45.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.145+0000 7fa043e00640 1 Processor -- start 2024-03-28T19:02:45.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.145+0000 7fa043e00640 1 -- start start 2024-03-28T19:02:45.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.147+0000 7fa043e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa0440c58c0 0x7fa0440c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.147+0000 7fa043e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa044158bd0 con 0x7fa0440c8ed0 2024-03-28T19:02:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.147+0000 7fa043e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa044158eb0 con 0x7fa0440be5d0 2024-03-28T19:02:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.147+0000 7fa043e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa044159190 con 0x7fa0440c58c0 2024-03-28T19:02:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.147+0000 7fa03be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa0440c8ed0 0x7fa0441555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60810/0 (socket says 172.21.15.27:60810) 2024-03-28T19:02:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.147+0000 7fa03be00640 1 -- 172.21.15.27:0/4065293110 learned_addr learned my addr 172.21.15.27:0/4065293110 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:45.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.147+0000 7fa042a00640 1 --2- 172.21.15.27:0/4065293110 >> v2:172.21.15.155:3300/0 conn(0x7fa0440c58c0 0x7fa0440c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.148+0000 7fa042000640 1 -- 172.21.15.27:0/4065293110 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318119679 0 0) 0x7fa044158bd0 con 0x7fa0440c8ed0 2024-03-28T19:02:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.148+0000 7fa042000640 1 -- 172.21.15.27:0/4065293110 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa028003520 con 0x7fa0440c8ed0 2024-03-28T19:02:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.148+0000 7fa042a00640 1 -- 172.21.15.27:0/4065293110 >> v1:172.21.15.155:6789/0 conn(0x7fa0440be5d0 legacy=0x7fa044155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.148+0000 7fa042a00640 1 -- 172.21.15.27:0/4065293110 >> v1:172.21.15.27:6789/0 conn(0x7fa0440c8ed0 legacy=0x7fa0441555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.148+0000 7fa042a00640 1 -- 172.21.15.27:0/4065293110 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044159470 con 0x7fa0440c58c0 2024-03-28T19:02:45.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.149+0000 7fa042a00640 1 --2- 172.21.15.27:0/4065293110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c58c0 0x7fa0440c5ca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa0340097a0 tx=0x7fa03402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=667d11522005a913 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.150+0000 7fa042000640 1 -- 172.21.15.27:0/4065293110 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0340315a0 con 0x7fa0440c58c0 2024-03-28T19:02:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.150+0000 7fa042000640 1 -- 172.21.15.27:0/4065293110 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa034037070 con 0x7fa0440c58c0 2024-03-28T19:02:45.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.150+0000 7fa042000640 1 -- 172.21.15.27:0/4065293110 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa034031940 con 0x7fa0440c58c0 2024-03-28T19:02:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.150+0000 7fa043e00640 1 -- 172.21.15.27:0/4065293110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c58c0 msgr2=0x7fa0440c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.150+0000 7fa043e00640 1 --2- 172.21.15.27:0/4065293110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c58c0 0x7fa0440c5ca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa0340097a0 tx=0x7fa03402b840 comp rx=0 tx=0).stop 2024-03-28T19:02:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.151+0000 7fa043e00640 1 -- 172.21.15.27:0/4065293110 shutdown_connections 2024-03-28T19:02:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.151+0000 7fa043e00640 1 --2- 172.21.15.27:0/4065293110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c58c0 0x7fa0440c5ca0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:45.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.151+0000 7fa043e00640 1 -- 172.21.15.27:0/4065293110 >> 172.21.15.27:0/4065293110 conn(0x7fa04406b6f0 msgr2=0x7fa0440acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:45.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.151+0000 7fa043e00640 1 -- 172.21.15.27:0/4065293110 shutdown_connections 2024-03-28T19:02:45.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.151+0000 7fa043e00640 1 -- 172.21.15.27:0/4065293110 wait complete. 2024-03-28T19:02:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.153+0000 7fa043e00640 1 Processor -- start 2024-03-28T19:02:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.153+0000 7fa043e00640 1 -- start start 2024-03-28T19:02:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa043e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0440be5d0 0x7fa0441519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa043e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0440c58c0 0x7fa0440acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa043e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c8ed0 0x7fa0440c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa043e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0440b09a0 con 0x7fa0440c58c0 2024-03-28T19:02:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa043e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0440b0820 con 0x7fa0440c8ed0 2024-03-28T19:02:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa043e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0440ad7d0 con 0x7fa0440be5d0 2024-03-28T19:02:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa03be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0440c58c0 0x7fa0440acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa03be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0440c58c0 0x7fa0440acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34984/0 (socket says 172.21.15.27:34984) 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa03be00640 1 -- 172.21.15.27:0/1342147541 learned_addr learned my addr 172.21.15.27:0/1342147541 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa042a00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0440be5d0 0x7fa0441519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.154+0000 7fa043400640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c8ed0 0x7fa0440c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03be00640 1 -- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0440be5d0 msgr2=0x7fa0441519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03be00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0440be5d0 0x7fa0441519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03be00640 1 -- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c8ed0 msgr2=0x7fa0440c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03be00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c8ed0 0x7fa0440c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03be00640 1 -- 172.21.15.27:0/1342147541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044152230 con 0x7fa0440c58c0 2024-03-28T19:02:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa042a00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0440be5d0 0x7fa0441519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:02:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03be00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0440c58c0 0x7fa0440acc60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00c410 tx=0x7fa02c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03aa00640 1 -- 172.21.15.27:0/1342147541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c017070 con 0x7fa0440c58c0 2024-03-28T19:02:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03aa00640 1 -- 172.21.15.27:0/1342147541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c005ce0 con 0x7fa0440c58c0 2024-03-28T19:02:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0440c2430 con 0x7fa0440c58c0 2024-03-28T19:02:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.155+0000 7fa03aa00640 1 -- 172.21.15.27:0/1342147541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0072d0 con 0x7fa0440c58c0 2024-03-28T19:02:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.156+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0440aaad0 con 0x7fa0440c58c0 2024-03-28T19:02:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.157+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0440c2220 con 0x7fa0440c58c0 2024-03-28T19:02:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.158+0000 7fa03aa00640 1 -- 172.21.15.27:0/1342147541 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa02c01a070 con 0x7fa0440c58c0 2024-03-28T19:02:45.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.159+0000 7fa03aa00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa02006bbd0 0x7fa02006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:45.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.159+0000 7fa03aa00640 1 -- 172.21.15.27:0/1342147541 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7411+0+0 (secure 0 0 0) 0x7fa02c08a410 con 0x7fa0440c58c0 2024-03-28T19:02:45.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.160+0000 7fa042a00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa02006bbd0 0x7fa02006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:45.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.161+0000 7fa042a00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa02006bbd0 0x7fa02006e090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa034036000 tx=0x7fa034008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:45.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.166+0000 7fa03aa00640 1 -- 172.21.15.27:0/1342147541 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa02c0581f0 con 0x7fa0440c58c0 2024-03-28T19:02:45.167 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:45.185 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:45.185 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:45.186 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:45.186 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:45.186 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:45.186 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:45.186 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:45.186 INFO:tasks.workunit.client.1.smithi155.stdout: 45594 ? 00:00:00 ps 2024-03-28T19:02:45.193 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:45.219 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:45.228 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:45.239 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:45.239 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:45.249 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:45.249 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:45.268 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:45.280 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:45.283 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:45.343 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:45.363 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:45.363 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:45.363 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:45.363 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:45.363 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:45.363 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:45.363 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:45.363 INFO:tasks.workunit.client.1.smithi155.stdout: 45604 ? 00:00:00 ps 2024-03-28T19:02:45.371 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:45.401 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:45.416 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:45.429 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:45.429 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:45.442 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:45.442 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:45.466 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:45.480 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:45.485 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.489+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa044069640 con 0x7fa0440c58c0 2024-03-28T19:02:45.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.490+0000 7fa03aa00640 1 -- 172.21.15.27:0/1342147541 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fa02c05e070 con 0x7fa0440c58c0 2024-03-28T19:02:45.491 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:45.491 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:45.491 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:02:45.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa02006bbd0 msgr2=0x7fa02006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:45.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa02006bbd0 0x7fa02006e090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa034036000 tx=0x7fa034008030 comp rx=0 tx=0).stop 2024-03-28T19:02:45.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0440c58c0 msgr2=0x7fa0440acc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:45.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0440c58c0 0x7fa0440acc60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00c410 tx=0x7fa02c00c8d0 comp rx=0 tx=0).stop 2024-03-28T19:02:45.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 shutdown_connections 2024-03-28T19:02:45.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa02006bbd0 0x7fa02006e090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:45.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0440c8ed0 0x7fa0440c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:45.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0440c58c0 0x7fa0440acc60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:45.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 --2- 172.21.15.27:0/1342147541 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0440be5d0 0x7fa0441519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:45.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 >> 172.21.15.27:0/1342147541 conn(0x7fa04406b6f0 msgr2=0x7fa0441581a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:45.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.494+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 shutdown_connections 2024-03-28T19:02:45.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:45.495+0000 7fa043e00640 1 -- 172.21.15.27:0/1342147541 wait complete. 2024-03-28T19:02:45.520 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-03-28T19:02:45.520 INFO:tasks.mds_thrash.fs.[b]:waiting for 98 secs before thrashing 2024-03-28T19:02:45.567 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:45.586 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:45.586 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:45.586 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:45.586 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:45.586 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:45.586 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:45.586 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:45.586 INFO:tasks.workunit.client.1.smithi155.stdout: 45614 ? 00:00:00 ps 2024-03-28T19:02:45.594 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:45.618 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:45.645 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:45.653 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:45.653 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:45.662 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:45.662 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:45.686 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:45.702 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:45.708 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:45.786 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:45.803 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:45.804 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:45.804 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:45.804 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:45.804 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:45.804 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:45.804 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:45.804 INFO:tasks.workunit.client.1.smithi155.stdout: 45624 ? 00:00:00 ps 2024-03-28T19:02:45.813 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:45.847 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:45.860 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:45.872 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:45.872 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:45.884 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:45.884 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:45.907 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:45.927 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:45.932 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:45.934 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:02:45.934+0000 7f51d8e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:02:46.002 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:46.020 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:46.020 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:46.020 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:46.020 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:46.020 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:46.020 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:46.021 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:46.021 INFO:tasks.workunit.client.1.smithi155.stdout: 45634 ? 00:00:00 ps 2024-03-28T19:02:46.031 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:46.070 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:46.081 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:46.097 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:46.097 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:46.113 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:46.113 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:46.137 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:46.156 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:46.161 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:46.218 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:46.239 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:46.240 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:46.240 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:46.240 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:46.240 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:46.240 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:46.240 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:46.240 INFO:tasks.workunit.client.1.smithi155.stdout: 45644 ? 00:00:00 ps 2024-03-28T19:02:46.252 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:46.302 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:46.317 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:46.341 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:46.342 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:46.360 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:46.360 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:46.421 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:46.453 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:46.458 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:46.788 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:46.816 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:46.816 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:46.816 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:46.816 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:46.816 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:46.816 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:46.817 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:46.817 INFO:tasks.workunit.client.1.smithi155.stdout: 45654 ? 00:00:00 ps 2024-03-28T19:02:46.827 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:46.860 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:46.870 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:46.882 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:46.882 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:46.894 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:46.894 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:46.918 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:46.935 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:46.940 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:47.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:47.146 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:47.165 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:47.165 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:47.165 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:47.165 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:47.165 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:47.165 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:47.165 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:47.166 INFO:tasks.workunit.client.1.smithi155.stdout: 45664 ? 00:00:00 ps 2024-03-28T19:02:47.171 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:47.193 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:47.202 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:47.212 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 20` 2024-03-28T19:02:47.212 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a 2024-03-28T19:02:47.222 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/one 2024-03-28T19:02:47.223 INFO:tasks.workunit.client.1.smithi155.stderr:+ cat 2024-03-28T19:02:47.243 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir a/.snap/two 2024-03-28T19:02:47.274 INFO:tasks.workunit.client.1.smithi155.stderr:+ chmod 777 a/foo 2024-03-28T19:02:47.279 INFO:tasks.workunit.client.1.smithi155.stderr:+ sync 2024-03-28T19:02:47.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.291+0000 7efcc0a00640 1 Processor -- start 2024-03-28T19:02:47.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.292+0000 7efcc0a00640 1 -- start start 2024-03-28T19:02:47.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.293+0000 7efcc0a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efcbc0ab2b0 0x7efcbc0c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.293+0000 7efcc0a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0c26d0 con 0x7efcbc1551a0 2024-03-28T19:02:47.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.293+0000 7efcc0a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc156870 con 0x7efcbc0aada0 2024-03-28T19:02:47.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.293+0000 7efcc0a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcbc155cf0 con 0x7efcbc0ab2b0 2024-03-28T19:02:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.293+0000 7efcbbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efcbc0ab2b0 0x7efcbc0c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.293+0000 7efcbb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efcbc1551a0 0x7efcbc0c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59294/0 (socket says 172.21.15.27:59294) 2024-03-28T19:02:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.293+0000 7efcbb400640 1 -- 172.21.15.27:0/2791050551 learned_addr learned my addr 172.21.15.27:0/2791050551 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.294+0000 7efcbbe00640 1 -- 172.21.15.27:0/2791050551 >> v1:172.21.15.155:6789/0 conn(0x7efcbc0aada0 legacy=0x7efcbc0c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.294+0000 7efcbbe00640 1 -- 172.21.15.27:0/2791050551 >> v1:172.21.15.27:6789/0 conn(0x7efcbc1551a0 legacy=0x7efcbc0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.294+0000 7efcba000640 1 -- 172.21.15.27:0/2791050551 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158462432 0 0) 0x7efcbc156870 con 0x7efcbc0aada0 2024-03-28T19:02:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.294+0000 7efcbbe00640 1 -- 172.21.15.27:0/2791050551 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc155fd0 con 0x7efcbc0ab2b0 2024-03-28T19:02:47.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.295+0000 7efcbbe00640 1 --2- 172.21.15.27:0/2791050551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0ab2b0 0x7efcbc0c2190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009dc0 tx=0x7efcb402bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22de3047c71b0c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:47.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.296+0000 7efcba000640 1 -- 172.21.15.27:0/2791050551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4039070 con 0x7efcbc0ab2b0 2024-03-28T19:02:47.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.296+0000 7efcba000640 1 -- 172.21.15.27:0/2791050551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb402c980 con 0x7efcbc0ab2b0 2024-03-28T19:02:47.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.296+0000 7efcba000640 1 -- 172.21.15.27:0/2791050551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb402cd50 con 0x7efcbc0ab2b0 2024-03-28T19:02:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.296+0000 7efcc0a00640 1 -- 172.21.15.27:0/2791050551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0ab2b0 msgr2=0x7efcbc0c2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.296+0000 7efcc0a00640 1 --2- 172.21.15.27:0/2791050551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0ab2b0 0x7efcbc0c2190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009dc0 tx=0x7efcb402bc50 comp rx=0 tx=0).stop 2024-03-28T19:02:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.296+0000 7efcc0a00640 1 -- 172.21.15.27:0/2791050551 shutdown_connections 2024-03-28T19:02:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.296+0000 7efcc0a00640 1 --2- 172.21.15.27:0/2791050551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0ab2b0 0x7efcbc0c2190 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.296+0000 7efcc0a00640 1 -- 172.21.15.27:0/2791050551 >> 172.21.15.27:0/2791050551 conn(0x7efcbc06b0c0 msgr2=0x7efcbc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.297+0000 7efcc0a00640 1 -- 172.21.15.27:0/2791050551 shutdown_connections 2024-03-28T19:02:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.297+0000 7efcc0a00640 1 -- 172.21.15.27:0/2791050551 wait complete. 2024-03-28T19:02:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.298+0000 7efcc0a00640 1 Processor -- start 2024-03-28T19:02:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.298+0000 7efcc0a00640 1 -- start start 2024-03-28T19:02:47.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.299+0000 7efcc0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0aada0 0x7efcbc0a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.299+0000 7efcc0a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efcbc0ab2b0 0x7efcbc153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.299+0000 7efcc0a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcbc1551a0 0x7efcbc1542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.299+0000 7efcc0a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1559a0 con 0x7efcbc0aada0 2024-03-28T19:02:47.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.299+0000 7efcc0a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc156590 con 0x7efcbc1551a0 2024-03-28T19:02:47.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.299+0000 7efcc0a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0c5750 con 0x7efcbc0ab2b0 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.300+0000 7efcbb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0aada0 0x7efcbc0a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.300+0000 7efcbb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0aada0 0x7efcbc0a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40764/0 (socket says 172.21.15.27:40764) 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.300+0000 7efcbb400640 1 -- 172.21.15.27:0/1158184057 learned_addr learned my addr 172.21.15.27:0/1158184057 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.300+0000 7efcbaa00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efcbc0ab2b0 0x7efcbc153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.300+0000 7efcbbe00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcbc1551a0 0x7efcbc1542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.300+0000 7efcbb400640 1 -- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efcbc0ab2b0 msgr2=0x7efcbc153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.300+0000 7efcbb400640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efcbc0ab2b0 0x7efcbc153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.301+0000 7efcbb400640 1 -- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcbc1551a0 msgr2=0x7efcbc1542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.301+0000 7efcbb400640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcbc1551a0 0x7efcbc1542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.301+0000 7efcbb400640 1 -- 172.21.15.27:0/1158184057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc1e5260 con 0x7efcbc0aada0 2024-03-28T19:02:47.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.301+0000 7efcbbe00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcbc1551a0 0x7efcbc1542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:02:47.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.301+0000 7efcbb400640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0aada0 0x7efcbc0a46b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7efcac00c5f0 tx=0x7efcac00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:47.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.302+0000 7efcb3400640 1 -- 172.21.15.27:0/1158184057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac018070 con 0x7efcbc0aada0 2024-03-28T19:02:47.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.302+0000 7efcb3400640 1 -- 172.21.15.27:0/1158184057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac0074e0 con 0x7efcbc0aada0 2024-03-28T19:02:47.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.302+0000 7efcb3400640 1 -- 172.21.15.27:0/1158184057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac004a00 con 0x7efcbc0aada0 2024-03-28T19:02:47.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.302+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc1f36c0 con 0x7efcbc0aada0 2024-03-28T19:02:47.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.302+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc06b690 con 0x7efcbc0aada0 2024-03-28T19:02:47.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.305+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc067dd0 con 0x7efcbc0aada0 2024-03-28T19:02:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.307+0000 7efcb3400640 1 -- 172.21.15.27:0/1158184057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efcac01b070 con 0x7efcbc0aada0 2024-03-28T19:02:47.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.309+0000 7efcb3400640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc9c06bca0 0x7efc9c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.309+0000 7efcb3400640 1 -- 172.21.15.27:0/1158184057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7483+0+0 (secure 0 0 0) 0x7efcac08a610 con 0x7efcbc0aada0 2024-03-28T19:02:47.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.310+0000 7efcbaa00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc9c06bca0 0x7efc9c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.312+0000 7efcbaa00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc9c06bca0 0x7efc9c06e160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0a4ee0 tx=0x7efca8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:47.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.318+0000 7efcb3400640 1 -- 172.21.15.27:0/1158184057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efcac058430 con 0x7efcbc0aada0 2024-03-28T19:02:47.412 INFO:tasks.workunit.client.1.smithi155.stderr:+ ps 2024-03-28T19:02:47.427 INFO:tasks.workunit.client.1.smithi155.stdout: PID TTY TIME CMD 2024-03-28T19:02:47.428 INFO:tasks.workunit.client.1.smithi155.stdout: 4285 ? 00:00:00 systemd 2024-03-28T19:02:47.428 INFO:tasks.workunit.client.1.smithi155.stdout: 4287 ? 00:00:00 (sd-pam) 2024-03-28T19:02:47.428 INFO:tasks.workunit.client.1.smithi155.stdout: 23040 ? 00:00:01 sshd 2024-03-28T19:02:47.428 INFO:tasks.workunit.client.1.smithi155.stdout: 42723 ? 00:00:00 bash 2024-03-28T19:02:47.428 INFO:tasks.workunit.client.1.smithi155.stdout: 45457 ? 00:00:00 timeout 2024-03-28T19:02:47.428 INFO:tasks.workunit.client.1.smithi155.stdout: 45475 ? 00:00:00 snaptest-double 2024-03-28T19:02:47.428 INFO:tasks.workunit.client.1.smithi155.stdout: 45674 ? 00:00:00 ps 2024-03-28T19:02:47.432 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a/.snap/one a/.snap/two 2024-03-28T19:02:47.451 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm a/foo 2024-03-28T19:02:47.457 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir a 2024-03-28T19:02:47.465 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo OK 2024-03-28T19:02:47.466 INFO:tasks.workunit.client.1.smithi155.stdout:OK 2024-03-28T19:02:47.467 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:02:47.467 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-28T19:02:47.551 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-03-28T19:02:47.551 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-estale.sh 2024-03-28T19:02:47.623 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/foo 2024-03-28T19:02:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.635+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcbc069640 con 0x7efcbc0aada0 2024-03-28T19:02:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.636+0000 7efcb3400640 1 -- 172.21.15.27:0/1158184057 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7efcac05c4d0 con 0x7efcbc0aada0 2024-03-28T19:02:47.637 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:47.637 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:02:47.638 INFO:tasks.workunit.client.1.smithi155.stdout:We want ENOENT, not ESTALE, here. 2024-03-28T19:02:47.639 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2024-03-28T19:02:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.638+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc9c06bca0 msgr2=0x7efc9c06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.638+0000 7efcc0a00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc9c06bca0 0x7efc9c06e160 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0a4ee0 tx=0x7efca8009040 comp rx=0 tx=0).stop 2024-03-28T19:02:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.638+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0aada0 msgr2=0x7efcbc0a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.638+0000 7efcc0a00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0aada0 0x7efcbc0a46b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7efcac00c5f0 tx=0x7efcac00cab0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.639+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 shutdown_connections 2024-03-28T19:02:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.639+0000 7efcc0a00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc9c06bca0 0x7efc9c06e160 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.639+0000 7efcc0a00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcbc1551a0 0x7efcbc1542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.639+0000 7efcc0a00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efcbc0ab2b0 0x7efcbc153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.639+0000 7efcc0a00640 1 --2- 172.21.15.27:0/1158184057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efcbc0aada0 0x7efcbc0a46b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.639+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 >> 172.21.15.27:0/1158184057 conn(0x7efcbc06b0c0 msgr2=0x7efcbc0c65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.639+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 shutdown_connections 2024-03-28T19:02:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.639+0000 7efcc0a00640 1 -- 172.21.15.27:0/1158184057 wait complete. 2024-03-28T19:02:47.641 INFO:tasks.workunit.client.1.smithi155.stderr:++ seq 1 100 2024-03-28T19:02:47.642 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.643 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/1 2024-03-28T19:02:47.644 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.649 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-03-28T19:02:47.650 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.650 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/2 2024-03-28T19:02:47.651 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.655 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-03-28T19:02:47.656 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.656 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/3 2024-03-28T19:02:47.657 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.658 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-03-28T19:02:47.659 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:02:47.659 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:02:47.661 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-03-28T19:02:47.662 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.662 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/4 2024-03-28T19:02:47.663 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.668 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-03-28T19:02:47.668 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.669 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/5 2024-03-28T19:02:47.669 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.675 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-03-28T19:02:47.676 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.676 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/6 2024-03-28T19:02:47.677 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.680 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:02:47.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:47.683 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-03-28T19:02:47.683 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.684 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/7 2024-03-28T19:02:47.684 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.689 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-03-28T19:02:47.690 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.690 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/8 2024-03-28T19:02:47.691 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.695 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-03-28T19:02:47.696 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.696 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/9 2024-03-28T19:02:47.697 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.702 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-03-28T19:02:47.702 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.703 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.703 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/10 2024-03-28T19:02:47.709 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-03-28T19:02:47.709 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.710 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/11 2024-03-28T19:02:47.710 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.714 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-03-28T19:02:47.715 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.716 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/12 2024-03-28T19:02:47.716 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.721 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-03-28T19:02:47.721 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.722 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/13 2024-03-28T19:02:47.722 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.727 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-03-28T19:02:47.727 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.728 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/14 2024-03-28T19:02:47.728 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.735 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-03-28T19:02:47.735 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.736 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/15 2024-03-28T19:02:47.736 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.742 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-03-28T19:02:47.742 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.743 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.743 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/16 2024-03-28T19:02:47.748 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-03-28T19:02:47.749 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.750 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.750 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/17 2024-03-28T19:02:47.757 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-03-28T19:02:47.757 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.758 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/18 2024-03-28T19:02:47.758 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.762 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-03-28T19:02:47.763 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.763 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/19 2024-03-28T19:02:47.764 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.768 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-03-28T19:02:47.769 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.769 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/20 2024-03-28T19:02:47.770 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.774 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-03-28T19:02:47.775 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.775 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/21 2024-03-28T19:02:47.776 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.780 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-03-28T19:02:47.781 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.781 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/22 2024-03-28T19:02:47.781 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.787 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-03-28T19:02:47.788 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.788 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/23 2024-03-28T19:02:47.788 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.794 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-03-28T19:02:47.795 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.795 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/24 2024-03-28T19:02:47.795 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.801 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-03-28T19:02:47.802 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.802 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/25 2024-03-28T19:02:47.803 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.808 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-03-28T19:02:47.809 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.809 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/26 2024-03-28T19:02:47.810 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.816 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-03-28T19:02:47.816 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.817 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/27 2024-03-28T19:02:47.818 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.824 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-03-28T19:02:47.824 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.825 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/28 2024-03-28T19:02:47.825 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.831 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-03-28T19:02:47.831 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.832 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/29 2024-03-28T19:02:47.833 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.839 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-03-28T19:02:47.840 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.841 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/30 2024-03-28T19:02:47.842 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.849 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-03-28T19:02:47.850 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.852 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/31 2024-03-28T19:02:47.852 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.860 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-03-28T19:02:47.860 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.861 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/32 2024-03-28T19:02:47.861 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.868 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-03-28T19:02:47.868 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.869 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/33 2024-03-28T19:02:47.870 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.876 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-03-28T19:02:47.877 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.878 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.878 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/34 2024-03-28T19:02:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.880+0000 7fdbc9a00640 1 Processor -- start 2024-03-28T19:02:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.880+0000 7fdbc9a00640 1 -- start start 2024-03-28T19:02:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.882+0000 7fdbc9a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdbc4153bd0 0x7fdbc40c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.882+0000 7fdbc9a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc4154020 con 0x7fdbc4150c70 2024-03-28T19:02:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.882+0000 7fdbc9a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40c3d90 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.882+0000 7fdbc9a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc40ab4e0 con 0x7fdbc4153bd0 2024-03-28T19:02:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.882+0000 7fdbc2a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdbc4150c70 0x7fdbc40c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59306/0 (socket says 172.21.15.27:59306) 2024-03-28T19:02:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.882+0000 7fdbc2a00640 1 -- 172.21.15.27:0/912627151 learned_addr learned my addr 172.21.15.27:0/912627151 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.882+0000 7fdbc3400640 1 --2- 172.21.15.27:0/912627151 >> v2:172.21.15.155:3300/0 conn(0x7fdbc4153bd0 0x7fdbc40c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.883+0000 7fdbc1600640 1 -- 172.21.15.27:0/912627151 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002836779 0 0) 0x7fdbc4154020 con 0x7fdbc4150c70 2024-03-28T19:02:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.883+0000 7fdbc1600640 1 -- 172.21.15.27:0/912627151 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba8003520 con 0x7fdbc4150c70 2024-03-28T19:02:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.883+0000 7fdbc1600640 1 -- 172.21.15.27:0/912627151 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886035807 0 0) 0x7fdbc40c3d90 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.883+0000 7fdbc1600640 1 -- 172.21.15.27:0/912627151 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc4154020 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.883+0000 7fdbc3400640 1 -- 172.21.15.27:0/912627151 >> v1:172.21.15.155:6789/0 conn(0x7fdbc40a9cb0 legacy=0x7fdbc41534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.883+0000 7fdbc3400640 1 -- 172.21.15.27:0/912627151 >> v1:172.21.15.27:6789/0 conn(0x7fdbc4150c70 legacy=0x7fdbc40c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.883+0000 7fdbc3400640 1 -- 172.21.15.27:0/912627151 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40ab710 con 0x7fdbc4153bd0 2024-03-28T19:02:47.886 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-03-28T19:02:47.887 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.887 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/35 2024-03-28T19:02:47.887 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.884+0000 7fdbc3400640 1 --2- 172.21.15.27:0/912627151 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 0x7fdbc40c3690 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009d90 tx=0x7fdbbc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3e37ea8499549dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:47.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.884+0000 7fdbc1600640 1 -- 172.21.15.27:0/912627151 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc039070 con 0x7fdbc4153bd0 2024-03-28T19:02:47.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc1600640 1 -- 172.21.15.27:0/912627151 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc02cc60 con 0x7fdbc4153bd0 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc1600640 1 -- 172.21.15.27:0/912627151 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc034640 con 0x7fdbc4153bd0 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc9a00640 1 -- 172.21.15.27:0/912627151 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 msgr2=0x7fdbc40c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc9a00640 1 --2- 172.21.15.27:0/912627151 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 0x7fdbc40c3690 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009d90 tx=0x7fdbbc02beb0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc9a00640 1 -- 172.21.15.27:0/912627151 shutdown_connections 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc9a00640 1 --2- 172.21.15.27:0/912627151 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 0x7fdbc40c3690 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc9a00640 1 -- 172.21.15.27:0/912627151 >> 172.21.15.27:0/912627151 conn(0x7fdbc406b9d0 msgr2=0x7fdbc40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc9a00640 1 -- 172.21.15.27:0/912627151 shutdown_connections 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.885+0000 7fdbc9a00640 1 -- 172.21.15.27:0/912627151 wait complete. 2024-03-28T19:02:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.888+0000 7fdbc9a00640 1 Processor -- start 2024-03-28T19:02:47.890 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-03-28T19:02:47.891 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.892 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/36 2024-03-28T19:02:47.892 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.888+0000 7fdbc9a00640 1 -- start start 2024-03-28T19:02:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40a9cb0 0x7fdbc41ece90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc9a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbc4150c70 0x7fdbc41ec540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 0x7fdbc4151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc9a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40abcd0 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc9a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40ac1f0 con 0x7fdbc4153bd0 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc9a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40c1810 con 0x7fdbc4150c70 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40a9cb0 0x7fdbc41ece90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40a9cb0 0x7fdbc41ece90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40780/0 (socket says 172.21.15.27:40780) 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc2a00640 1 -- 172.21.15.27:0/4169679466 learned_addr learned my addr 172.21.15.27:0/4169679466 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc2000640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbc4150c70 0x7fdbc41ec540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.890+0000 7fdbc3400640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 0x7fdbc4151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbc2a00640 1 -- 172.21.15.27:0/4169679466 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbc4150c70 msgr2=0x7fdbc41ec540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbc2a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbc4150c70 0x7fdbc41ec540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbc2a00640 1 -- 172.21.15.27:0/4169679466 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 msgr2=0x7fdbc4151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbc2a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 0x7fdbc4151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbc2a00640 1 -- 172.21.15.27:0/4169679466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc4153700 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbc2a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40a9cb0 0x7fdbc41ece90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400c500 tx=0x7fdbb400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbbaa00640 1 -- 172.21.15.27:0/4169679466 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4017070 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbbaa00640 1 -- 172.21.15.27:0/4169679466 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb4005d00 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc4153420 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbbaa00640 1 -- 172.21.15.27:0/4169679466 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4007330 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.891+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc41eca80 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.892+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc404f400 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.893+0000 7fdbbaa00640 1 -- 172.21.15.27:0/4169679466 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdbb401a070 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.897 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-03-28T19:02:47.898 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.898 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/37 2024-03-28T19:02:47.898 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.895+0000 7fdbbaa00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdba406bbd0 0x7fdba406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:47.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.895+0000 7fdbbaa00640 1 -- 172.21.15.27:0/4169679466 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7483+0+0 (secure 0 0 0) 0x7fdbb408a580 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.895+0000 7fdbc2000640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdba406bbd0 0x7fdba406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:47.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.896+0000 7fdbc2000640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdba406bbd0 0x7fdba406e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4071710 tx=0x7fdbb0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:47.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:47.900+0000 7fdbbaa00640 1 -- 172.21.15.27:0/4169679466 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdbb40583e0 con 0x7fdbc40a9cb0 2024-03-28T19:02:47.901 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-03-28T19:02:47.901 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.901 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/38 2024-03-28T19:02:47.902 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.907 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-03-28T19:02:47.907 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.908 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/39 2024-03-28T19:02:47.908 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.914 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-03-28T19:02:47.914 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.915 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/40 2024-03-28T19:02:47.915 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.921 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-03-28T19:02:47.921 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.922 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/41 2024-03-28T19:02:47.922 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.928 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-03-28T19:02:47.928 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.928 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/42 2024-03-28T19:02:47.929 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.935 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-03-28T19:02:47.935 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.936 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/43 2024-03-28T19:02:47.936 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.944 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-03-28T19:02:47.944 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.945 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.946 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/44 2024-03-28T19:02:47.953 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-03-28T19:02:47.953 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.954 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/45 2024-03-28T19:02:47.954 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.962 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-03-28T19:02:47.963 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.963 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/46 2024-03-28T19:02:47.964 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.970 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-03-28T19:02:47.971 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.972 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/47 2024-03-28T19:02:47.972 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.981 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-03-28T19:02:47.981 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.982 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/48 2024-03-28T19:02:47.983 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:47.992 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-03-28T19:02:47.992 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:47.993 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/49 2024-03-28T19:02:47.993 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.002 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-03-28T19:02:48.002 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.003 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/50 2024-03-28T19:02:48.004 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.013 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-03-28T19:02:48.013 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.015 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/51 2024-03-28T19:02:48.015 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.023 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-03-28T19:02:48.024 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.024 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/52 2024-03-28T19:02:48.024 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.032 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-03-28T19:02:48.033 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.033 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/53 2024-03-28T19:02:48.034 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.042 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-03-28T19:02:48.042 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.042 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/54 2024-03-28T19:02:48.043 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.051 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-03-28T19:02:48.051 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.052 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/55 2024-03-28T19:02:48.052 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.059 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-03-28T19:02:48.060 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.060 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/56 2024-03-28T19:02:48.061 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.068 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-03-28T19:02:48.069 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.069 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/57 2024-03-28T19:02:48.070 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.079 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-03-28T19:02:48.079 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.079 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/58 2024-03-28T19:02:48.080 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.088 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-03-28T19:02:48.088 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.089 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/59 2024-03-28T19:02:48.090 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.099 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-03-28T19:02:48.099 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.100 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/60 2024-03-28T19:02:48.100 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.107 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-03-28T19:02:48.107 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.108 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/61 2024-03-28T19:02:48.108 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.115 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-03-28T19:02:48.115 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.116 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/62 2024-03-28T19:02:48.116 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.123 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-03-28T19:02:48.124 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.126 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/63 2024-03-28T19:02:48.126 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.133 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-03-28T19:02:48.133 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.134 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/64 2024-03-28T19:02:48.134 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.141 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-03-28T19:02:48.142 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.143 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/65 2024-03-28T19:02:48.145 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.151 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-03-28T19:02:48.151 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.152 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/66 2024-03-28T19:02:48.154 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.160 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-03-28T19:02:48.160 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.161 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/67 2024-03-28T19:02:48.162 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.169 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-03-28T19:02:48.169 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.170 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/68 2024-03-28T19:02:48.172 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.179 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-03-28T19:02:48.179 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.179+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbc40c5f00 con 0x7fdbc40a9cb0 2024-03-28T19:02:48.180 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/69 2024-03-28T19:02:48.180 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.180+0000 7fdbbaa00640 1 -- 172.21.15.27:0/4169679466 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fdbb405c480 con 0x7fdbc40a9cb0 2024-03-28T19:02:48.181 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:48.181 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:48.182 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:02:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdba406bbd0 msgr2=0x7fdba406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdba406bbd0 0x7fdba406e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4071710 tx=0x7fdbb0009040 comp rx=0 tx=0).stop 2024-03-28T19:02:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40a9cb0 msgr2=0x7fdbc41ece90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40a9cb0 0x7fdbc41ece90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400c500 tx=0x7fdbb400c9c0 comp rx=0 tx=0).stop 2024-03-28T19:02:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 shutdown_connections 2024-03-28T19:02:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdba406bbd0 0x7fdba406e090 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbc4153bd0 0x7fdbc4151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbc4150c70 0x7fdbc41ec540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 --2- 172.21.15.27:0/4169679466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbc40a9cb0 0x7fdbc41ece90 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 >> 172.21.15.27:0/4169679466 conn(0x7fdbc406b9d0 msgr2=0x7fdbc4072020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 shutdown_connections 2024-03-28T19:02:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:48.183+0000 7fdbc9a00640 1 -- 172.21.15.27:0/4169679466 wait complete. 2024-03-28T19:02:48.186 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-03-28T19:02:48.186 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.187 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.187 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/70 2024-03-28T19:02:48.194 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-03-28T19:02:48.194 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.195 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/71 2024-03-28T19:02:48.195 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.201 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-03-28T19:02:48.201 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.202 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:02:48.203 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.203 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/72 2024-03-28T19:02:48.210 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-03-28T19:02:48.210 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.211 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/73 2024-03-28T19:02:48.211 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.217 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-03-28T19:02:48.218 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.218 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/74 2024-03-28T19:02:48.219 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.225 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-03-28T19:02:48.226 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.226 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/75 2024-03-28T19:02:48.227 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.233 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-03-28T19:02:48.234 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.235 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/76 2024-03-28T19:02:48.235 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.241 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-03-28T19:02:48.242 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.243 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.243 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/77 2024-03-28T19:02:48.250 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-03-28T19:02:48.251 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.253 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/78 2024-03-28T19:02:48.254 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.273 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-03-28T19:02:48.274 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.276 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/79 2024-03-28T19:02:48.276 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.285 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-03-28T19:02:48.287 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.288 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/80 2024-03-28T19:02:48.288 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.295 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-03-28T19:02:48.296 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.296 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/81 2024-03-28T19:02:48.296 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.301 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-03-28T19:02:48.301 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.302 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/82 2024-03-28T19:02:48.302 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.308 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-03-28T19:02:48.308 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.309 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/83 2024-03-28T19:02:48.309 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.314 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-03-28T19:02:48.315 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.315 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/84 2024-03-28T19:02:48.315 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.322 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-03-28T19:02:48.322 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.323 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/85 2024-03-28T19:02:48.323 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.329 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-03-28T19:02:48.329 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.329 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/86 2024-03-28T19:02:48.330 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.336 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-03-28T19:02:48.337 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.337 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/87 2024-03-28T19:02:48.337 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.343 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-03-28T19:02:48.344 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.344 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/88 2024-03-28T19:02:48.344 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.351 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-03-28T19:02:48.351 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.352 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/89 2024-03-28T19:02:48.352 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.359 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-03-28T19:02:48.359 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.360 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/90 2024-03-28T19:02:48.360 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.367 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-03-28T19:02:48.367 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.368 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/91 2024-03-28T19:02:48.368 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.375 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-03-28T19:02:48.375 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.376 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.376 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/92 2024-03-28T19:02:48.385 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-03-28T19:02:48.386 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.386 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/93 2024-03-28T19:02:48.387 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.393 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-03-28T19:02:48.393 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.394 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/94 2024-03-28T19:02:48.394 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.400 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-03-28T19:02:48.401 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.401 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/95 2024-03-28T19:02:48.402 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.407 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-03-28T19:02:48.408 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.408 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/96 2024-03-28T19:02:48.409 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.415 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-03-28T19:02:48.415 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.416 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/97 2024-03-28T19:02:48.417 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.423 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-03-28T19:02:48.423 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.424 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/98 2024-03-28T19:02:48.424 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.430 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-03-28T19:02:48.431 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.431 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/99 2024-03-28T19:02:48.432 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.438 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-03-28T19:02:48.438 INFO:tasks.workunit.client.1.smithi155.stderr:+ for f in `seq 1 100` 2024-03-28T19:02:48.439 INFO:tasks.workunit.client.1.smithi155.stderr:+ stat .snap/foo/100 2024-03-28T19:02:48.439 INFO:tasks.workunit.client.1.smithi155.stderr:+ grep 'No such file' 2024-03-28T19:02:48.446 INFO:tasks.workunit.client.1.smithi155.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-03-28T19:02:48.446 INFO:tasks.workunit.client.1.smithi155.stderr:+ rmdir .snap/foo 2024-03-28T19:02:48.470 INFO:tasks.workunit.client.1.smithi155.stdout:OK 2024-03-28T19:02:48.471 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo OK 2024-03-28T19:02:48.472 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T19:02:48.472 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-28T19:02:48.605 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-03-28T19:02:48.606 DEBUG:teuthology.orchestra.run.smithi155:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-03-28T19:02:48.694 INFO:tasks.workunit.client.1.smithi155.stderr:+ set -e 2024-03-28T19:02:48.695 INFO:tasks.workunit.client.1.smithi155.stderr:+ sudo git config --global http.sslVerify false 2024-03-28T19:02:48.765 INFO:tasks.workunit.client.1.smithi155.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-03-28T19:02:48.810 INFO:tasks.workunit.client.1.smithi155.stderr:+ retried=false 2024-03-28T19:02:48.810 INFO:tasks.workunit.client.1.smithi155.stderr:+ trap -- retry EXIT 2024-03-28T19:02:48.810 INFO:tasks.workunit.client.1.smithi155.stderr:+ rm -rf ceph 2024-03-28T19:02:48.812 INFO:tasks.workunit.client.1.smithi155.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-03-28T19:02:48.825 INFO:tasks.workunit.client.1.smithi155.stderr:Cloning into 'ceph'... 2024-03-28T19:02:50.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.425+0000 7f5656a00640 1 Processor -- start 2024-03-28T19:02:50.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.425+0000 7f5656a00640 1 -- start start 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.426+0000 7f5656a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5650152e90 0x7f56500c7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.426+0000 7f5656a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c93c0 0x7f56500c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.426+0000 7f5656a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f56500c8cd0 0x7f5650155990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.426+0000 7f5656a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5650158c30 con 0x7f5650152e90 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.426+0000 7f5656a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5650158f10 con 0x7f56500c8cd0 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.426+0000 7f5656a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56501591f0 con 0x7f56500c93c0 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.427+0000 7f5655600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5650152e90 0x7f56500c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.427+0000 7f5655600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5650152e90 0x7f56500c7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40792/0 (socket says 172.21.15.27:40792) 2024-03-28T19:02:50.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.427+0000 7f5655600640 1 -- 172.21.15.27:0/3972865692 learned_addr learned my addr 172.21.15.27:0/3972865692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.427+0000 7f5655600640 1 -- 172.21.15.27:0/3972865692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c93c0 msgr2=0x7f56500c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.427+0000 7f5655600640 1 --2- 172.21.15.27:0/3972865692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c93c0 0x7f56500c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.427+0000 7f5655600640 1 -- 172.21.15.27:0/3972865692 >> v2:172.21.15.155:3300/0 conn(0x7f56500c8cd0 msgr2=0x7f5650155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.427+0000 7f5655600640 1 --2- 172.21.15.27:0/3972865692 >> v2:172.21.15.155:3300/0 conn(0x7f56500c8cd0 0x7f5650155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.427+0000 7f5655600640 1 -- 172.21.15.27:0/3972865692 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56501594d0 con 0x7f5650152e90 2024-03-28T19:02:50.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.428+0000 7f5655600640 1 --2- 172.21.15.27:0/3972865692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650152e90 0x7f56500c7710 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f565006c570 tx=0x7f564402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fe2d6296b5e294 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:50.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.429+0000 7f564fe00640 1 -- 172.21.15.27:0/3972865692 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644035020 con 0x7f5650152e90 2024-03-28T19:02:50.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.429+0000 7f564fe00640 1 -- 172.21.15.27:0/3972865692 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5644031870 con 0x7f5650152e90 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.429+0000 7f564fe00640 1 -- 172.21.15.27:0/3972865692 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644031c40 con 0x7f5650152e90 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.429+0000 7f5656a00640 1 -- 172.21.15.27:0/3972865692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650152e90 msgr2=0x7f56500c7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.429+0000 7f5656a00640 1 --2- 172.21.15.27:0/3972865692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650152e90 0x7f56500c7710 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f565006c570 tx=0x7f564402b910 comp rx=0 tx=0).stop 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.429+0000 7f5656a00640 1 -- 172.21.15.27:0/3972865692 shutdown_connections 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.430+0000 7f5656a00640 1 --2- 172.21.15.27:0/3972865692 >> v2:172.21.15.155:3300/0 conn(0x7f56500c8cd0 0x7f5650155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.430+0000 7f5656a00640 1 --2- 172.21.15.27:0/3972865692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c93c0 0x7f56500c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.430+0000 7f5656a00640 1 --2- 172.21.15.27:0/3972865692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5650152e90 0x7f56500c7710 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.430+0000 7f5656a00640 1 -- 172.21.15.27:0/3972865692 >> 172.21.15.27:0/3972865692 conn(0x7f56500ac900 msgr2=0x7f56500acd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.430+0000 7f5656a00640 1 -- 172.21.15.27:0/3972865692 shutdown_connections 2024-03-28T19:02:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.430+0000 7f5656a00640 1 -- 172.21.15.27:0/3972865692 wait complete. 2024-03-28T19:02:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.432+0000 7f5656a00640 1 Processor -- start 2024-03-28T19:02:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.433+0000 7f5656a00640 1 -- start start 2024-03-28T19:02:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.434+0000 7f5656a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c8cd0 0x7f56500ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.434+0000 7f5656a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56500c93c0 0x7f5650157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.434+0000 7f5656a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5650152e90 0x7f56501ecb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.434+0000 7f5656a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650155ed0 con 0x7f56500c93c0 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.434+0000 7f5656a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650159a90 con 0x7f5650152e90 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.434+0000 7f5656a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5650156050 con 0x7f56500c8cd0 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.434+0000 7f5654c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56500c93c0 0x7f5650157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5654c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56500c93c0 0x7f5650157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40808/0 (socket says 172.21.15.27:40808) 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5654c00640 1 -- 172.21.15.27:0/2569273198 learned_addr learned my addr 172.21.15.27:0/2569273198 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5655600640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c8cd0 0x7f56500ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5656000640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5650152e90 0x7f56501ecb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5654c00640 1 -- 172.21.15.27:0/2569273198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c8cd0 msgr2=0x7f56500ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5654c00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c8cd0 0x7f56500ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5654c00640 1 -- 172.21.15.27:0/2569273198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5650152e90 msgr2=0x7f56501ecb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5654c00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5650152e90 0x7f56501ecb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5654c00640 1 -- 172.21.15.27:0/2569273198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56500cb1f0 con 0x7f56500c93c0 2024-03-28T19:02:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.435+0000 7f5654c00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56500c93c0 0x7f5650157cd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f563c00b4d0 tx=0x7f563c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.436+0000 7f564d600640 1 -- 172.21.15.27:0/2569273198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c014070 con 0x7f56500c93c0 2024-03-28T19:02:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.436+0000 7f564d600640 1 -- 172.21.15.27:0/2569273198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c010870 con 0x7f56500c93c0 2024-03-28T19:02:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.436+0000 7f564d600640 1 -- 172.21.15.27:0/2569273198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c010c60 con 0x7f56500c93c0 2024-03-28T19:02:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.436+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56500cb4b0 con 0x7f56500c93c0 2024-03-28T19:02:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.436+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56500c9dd0 con 0x7f56500c93c0 2024-03-28T19:02:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.437+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5650067dd0 con 0x7f56500c93c0 2024-03-28T19:02:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.438+0000 7f564d600640 1 -- 172.21.15.27:0/2569273198 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f563c017070 con 0x7f56500c93c0 2024-03-28T19:02:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.439+0000 7f564d600640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f563006bc00 0x7f563006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.439+0000 7f564d600640 1 -- 172.21.15.27:0/2569273198 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7495+0+0 (secure 0 0 0) 0x7f563c08af10 con 0x7f56500c93c0 2024-03-28T19:02:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.439+0000 7f5655600640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f563006bc00 0x7f563006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.440+0000 7f5655600640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f563006bc00 0x7f563006e0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5644036000 tx=0x7f564403c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:50.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.444+0000 7f564d600640 1 -- 172.21.15.27:0/2569273198 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f563c0592f0 con 0x7f56500c93c0 2024-03-28T19:02:50.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.694+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5650069640 con 0x7f56500c93c0 2024-03-28T19:02:50.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.695+0000 7f564d600640 1 -- 172.21.15.27:0/2569273198 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f563c058bf0 con 0x7f56500c93c0 2024-03-28T19:02:50.695 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:50.695 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:50.696 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.698+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f563006bc00 msgr2=0x7f563006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.698+0000 7f5656a00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f563006bc00 0x7f563006e0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5644036000 tx=0x7f564403c040 comp rx=0 tx=0).stop 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.698+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56500c93c0 msgr2=0x7f5650157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.698+0000 7f5656a00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56500c93c0 0x7f5650157cd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f563c00b4d0 tx=0x7f563c00b990 comp rx=0 tx=0).stop 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.698+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 shutdown_connections 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.698+0000 7f5656a00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f563006bc00 0x7f563006e0c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.698+0000 7f5656a00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5650152e90 0x7f56501ecb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.698+0000 7f5656a00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56500c93c0 0x7f5650157cd0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.699+0000 7f5656a00640 1 --2- 172.21.15.27:0/2569273198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56500c8cd0 0x7f56500ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.699+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 >> 172.21.15.27:0/2569273198 conn(0x7f56500ac900 msgr2=0x7f5650154900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.699+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 shutdown_connections 2024-03-28T19:02:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:50.699+0000 7f5656a00640 1 -- 172.21.15.27:0/2569273198 wait complete. 2024-03-28T19:02:50.721 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:02:52.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.956+0000 7fdf5b200640 1 Processor -- start 2024-03-28T19:02:52.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.956+0000 7fdf5b200640 1 -- start start 2024-03-28T19:02:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.957+0000 7fdf5b200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdf5c0c8ed0 0x7fdf5c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.957+0000 7fdf5b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf5c0be5d0 0x7fdf5c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.957+0000 7fdf5b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c156230 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.957+0000 7fdf5b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5c158bd0 con 0x7fdf5c0be5d0 2024-03-28T19:02:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.957+0000 7fdf5b200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5c158eb0 con 0x7fdf5c0c8ed0 2024-03-28T19:02:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.957+0000 7fdf59e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdf5c0c58c0 0x7fdf5c1555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59310/0 (socket says 172.21.15.27:59310) 2024-03-28T19:02:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf59e00640 1 -- 172.21.15.27:0/704939523 learned_addr learned my addr 172.21.15.27:0/704939523 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:02:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf5a800640 1 --2- 172.21.15.27:0/704939523 >> v2:172.21.15.27:3300/0 conn(0x7fdf5c0be5d0 0x7fdf5c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:52.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf51400640 1 --2- 172.21.15.27:0/704939523 >> v2:172.21.15.155:3300/0 conn(0x7fdf5c0c8ed0 0x7fdf5c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf59400640 1 -- 172.21.15.27:0/704939523 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256725501 0 0) 0x7fdf5c156230 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf59400640 1 -- 172.21.15.27:0/704939523 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf40003520 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf5a800640 1 -- 172.21.15.27:0/704939523 >> v2:172.21.15.155:3300/0 conn(0x7fdf5c0c8ed0 msgr2=0x7fdf5c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf5a800640 1 --2- 172.21.15.27:0/704939523 >> v2:172.21.15.155:3300/0 conn(0x7fdf5c0c8ed0 0x7fdf5c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf5a800640 1 -- 172.21.15.27:0/704939523 >> v1:172.21.15.27:6789/0 conn(0x7fdf5c0c58c0 legacy=0x7fdf5c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.958+0000 7fdf5a800640 1 -- 172.21.15.27:0/704939523 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c159190 con 0x7fdf5c0be5d0 2024-03-28T19:02:52.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.960+0000 7fdf5a800640 1 --2- 172.21.15.27:0/704939523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0be5d0 0x7fdf5c155cf0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0c83d0 tx=0x7fdf5402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd45004d05532d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:52.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.960+0000 7fdf59400640 1 -- 172.21.15.27:0/704939523 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54038070 con 0x7fdf5c0be5d0 2024-03-28T19:02:52.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.960+0000 7fdf59400640 1 -- 172.21.15.27:0/704939523 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf54031710 con 0x7fdf5c0be5d0 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.960+0000 7fdf59400640 1 -- 172.21.15.27:0/704939523 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54031ae0 con 0x7fdf5c0be5d0 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.960+0000 7fdf5b200640 1 -- 172.21.15.27:0/704939523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0be5d0 msgr2=0x7fdf5c155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.960+0000 7fdf5b200640 1 --2- 172.21.15.27:0/704939523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0be5d0 0x7fdf5c155cf0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0c83d0 tx=0x7fdf5402b7b0 comp rx=0 tx=0).stop 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.961+0000 7fdf5b200640 1 -- 172.21.15.27:0/704939523 shutdown_connections 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.961+0000 7fdf5b200640 1 --2- 172.21.15.27:0/704939523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0be5d0 0x7fdf5c155cf0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.961+0000 7fdf5b200640 1 --2- 172.21.15.27:0/704939523 >> v2:172.21.15.155:3300/0 conn(0x7fdf5c0c8ed0 0x7fdf5c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.961+0000 7fdf5b200640 1 -- 172.21.15.27:0/704939523 >> 172.21.15.27:0/704939523 conn(0x7fdf5c06b6f0 msgr2=0x7fdf5c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.961+0000 7fdf5b200640 1 -- 172.21.15.27:0/704939523 shutdown_connections 2024-03-28T19:02:52.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.961+0000 7fdf5b200640 1 -- 172.21.15.27:0/704939523 wait complete. 2024-03-28T19:02:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.962+0000 7fdf5b200640 1 Processor -- start 2024-03-28T19:02:52.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.963+0000 7fdf5b200640 1 -- start start 2024-03-28T19:02:52.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.963+0000 7fdf5b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf5c0be5d0 0x7fdf5c0c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.963+0000 7fdf5b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0c58c0 0x7fdf5c151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.963+0000 7fdf5b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf5c0c8ed0 0x7fdf5c1523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.963+0000 7fdf5b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c0b0820 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.963+0000 7fdf5b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c159750 con 0x7fdf5c0be5d0 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.963+0000 7fdf5b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c0b09a0 con 0x7fdf5c0c8ed0 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0c58c0 0x7fdf5c151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0c58c0 0x7fdf5c151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40832/0 (socket says 172.21.15.27:40832) 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 -- 172.21.15.27:0/4065653397 learned_addr learned my addr 172.21.15.27:0/4065653397 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf59e00640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf5c0be5d0 0x7fdf5c0c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 -- 172.21.15.27:0/4065653397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf5c0c8ed0 msgr2=0x7fdf5c1523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf5c0c8ed0 0x7fdf5c1523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 -- 172.21.15.27:0/4065653397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf5c0be5d0 msgr2=0x7fdf5c0c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf5c0be5d0 0x7fdf5c0c2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 -- 172.21.15.27:0/4065653397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0c4a50 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.964+0000 7fdf51400640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0c58c0 0x7fdf5c151e70 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400e870 tx=0x7fdf4400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.965+0000 7fdf52a00640 1 -- 172.21.15.27:0/4065653397 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf44017070 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.965+0000 7fdf52a00640 1 -- 172.21.15.27:0/4065653397 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4400c940 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.965+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c0c4d30 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.965+0000 7fdf52a00640 1 -- 172.21.15.27:0/4065653397 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4400cd10 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.965+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c0aad00 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.966+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c0ab0e0 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.967+0000 7fdf52a00640 1 -- 172.21.15.27:0/4065653397 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf4401a070 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.968+0000 7fdf52a00640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf3806bbd0 0x7fdf3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.968+0000 7fdf52a00640 1 -- 172.21.15.27:0/4065653397 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7495+0+0 (secure 0 0 0) 0x7fdf4408b2f0 con 0x7fdf5c0c58c0 2024-03-28T19:02:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.968+0000 7fdf59e00640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf3806bbd0 0x7fdf3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.969+0000 7fdf59e00640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf3806bbd0 0x7fdf3806e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c1582b0 tx=0x7fdf4c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:52.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:52.973+0000 7fdf52a00640 1 -- 172.21.15.27:0/4065653397 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf5c0ab0e0 con 0x7fdf5c0c58c0 2024-03-28T19:02:53.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.219+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf5c069640 con 0x7fdf5c0c58c0 2024-03-28T19:02:53.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.219+0000 7fdf52a00640 1 -- 172.21.15.27:0/4065653397 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fdf440596d0 con 0x7fdf5c0c58c0 2024-03-28T19:02:53.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:53.220 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:53.220 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:02:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.222+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf3806bbd0 msgr2=0x7fdf3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.222+0000 7fdf5b200640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf3806bbd0 0x7fdf3806e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c1582b0 tx=0x7fdf4c009040 comp rx=0 tx=0).stop 2024-03-28T19:02:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.222+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0c58c0 msgr2=0x7fdf5c151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.222+0000 7fdf5b200640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0c58c0 0x7fdf5c151e70 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400e870 tx=0x7fdf4400ed30 comp rx=0 tx=0).stop 2024-03-28T19:02:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.223+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 shutdown_connections 2024-03-28T19:02:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.223+0000 7fdf5b200640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdf3806bbd0 0x7fdf3806e090 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.223+0000 7fdf5b200640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdf5c0c8ed0 0x7fdf5c1523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.223+0000 7fdf5b200640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf5c0c58c0 0x7fdf5c151e70 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.223+0000 7fdf5b200640 1 --2- 172.21.15.27:0/4065653397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf5c0be5d0 0x7fdf5c0c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.223+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 >> 172.21.15.27:0/4065653397 conn(0x7fdf5c06b6f0 msgr2=0x7fdf5c0ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.223+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 shutdown_connections 2024-03-28T19:02:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:53.223+0000 7fdf5b200640 1 -- 172.21.15.27:0/4065653397 wait complete. 2024-03-28T19:02:53.240 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:02:55.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.468+0000 7f5c81a00640 1 Processor -- start 2024-03-28T19:02:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.468+0000 7f5c81a00640 1 -- start start 2024-03-28T19:02:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.470+0000 7f5c81a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c7c0711d0 0x7f5c7c0c5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.470+0000 7f5c81a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0a7b90 0x7f5c7c0c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.470+0000 7f5c81a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c159800 con 0x7f5c7c0c91f0 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.470+0000 7f5c81a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7c0c5680 con 0x7f5c7c0711d0 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.470+0000 7f5c81a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0715b0 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.470+0000 7f5c7aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c7c0711d0 0x7f5c7c0c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.470+0000 7f5c7aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c7c0711d0 0x7f5c7c0c5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40836/0 (socket says 172.21.15.27:40836) 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.470+0000 7f5c7aa00640 1 -- 172.21.15.27:0/376437465 learned_addr learned my addr 172.21.15.27:0/376437465 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.471+0000 7f5c7a000640 1 --2- 172.21.15.27:0/376437465 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0a7b90 0x7f5c7c0c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.471+0000 7f5c7aa00640 1 -- 172.21.15.27:0/376437465 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0a7b90 msgr2=0x7f5c7c0c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:55.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.471+0000 7f5c7aa00640 1 --2- 172.21.15.27:0/376437465 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0a7b90 0x7f5c7c0c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.471+0000 7f5c7aa00640 1 -- 172.21.15.27:0/376437465 >> v1:172.21.15.27:6789/0 conn(0x7f5c7c0c91f0 legacy=0x7f5c7c0c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.471+0000 7f5c7aa00640 1 -- 172.21.15.27:0/376437465 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c159ae0 con 0x7f5c7c0711d0 2024-03-28T19:02:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.471+0000 7f5c79600640 1 -- 172.21.15.27:0/376437465 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487123632 0 0) 0x7f5c7c159800 con 0x7f5c7c0c91f0 2024-03-28T19:02:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.473+0000 7f5c7aa00640 1 --2- 172.21.15.27:0/376437465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0711d0 0x7f5c7c0c5140 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c009870 tx=0x7f5c6c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357bef65e155a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.473+0000 7f5c79600640 1 -- 172.21.15.27:0/376437465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c034070 con 0x7f5c7c0711d0 2024-03-28T19:02:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.473+0000 7f5c79600640 1 -- 172.21.15.27:0/376437465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c030840 con 0x7f5c7c0711d0 2024-03-28T19:02:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.473+0000 7f5c79600640 1 -- 172.21.15.27:0/376437465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c030c10 con 0x7f5c7c0711d0 2024-03-28T19:02:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.474+0000 7f5c81a00640 1 -- 172.21.15.27:0/376437465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0711d0 msgr2=0x7f5c7c0c5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.474+0000 7f5c81a00640 1 --2- 172.21.15.27:0/376437465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0711d0 0x7f5c7c0c5140 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c009870 tx=0x7f5c6c02b980 comp rx=0 tx=0).stop 2024-03-28T19:02:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.474+0000 7f5c81a00640 1 -- 172.21.15.27:0/376437465 shutdown_connections 2024-03-28T19:02:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.474+0000 7f5c81a00640 1 --2- 172.21.15.27:0/376437465 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0a7b90 0x7f5c7c0c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.474+0000 7f5c81a00640 1 --2- 172.21.15.27:0/376437465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0711d0 0x7f5c7c0c5140 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.474+0000 7f5c81a00640 1 -- 172.21.15.27:0/376437465 >> 172.21.15.27:0/376437465 conn(0x7f5c7c0aaab0 msgr2=0x7f5c7c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.474+0000 7f5c81a00640 1 -- 172.21.15.27:0/376437465 shutdown_connections 2024-03-28T19:02:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.474+0000 7f5c81a00640 1 -- 172.21.15.27:0/376437465 wait complete. 2024-03-28T19:02:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.476+0000 7f5c81a00640 1 Processor -- start 2024-03-28T19:02:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.476+0000 7f5c81a00640 1 -- start start 2024-03-28T19:02:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c81a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c7c0711d0 0x7f5c7c0ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c81a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0a7b90 0x7f5c7c1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c81a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0c91f0 0x7f5c7c153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c81a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0c74a0 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c81a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0c9cb0 con 0x7f5c7c0711d0 2024-03-28T19:02:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c81a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0c4760 con 0x7f5c7c0c91f0 2024-03-28T19:02:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c7a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0a7b90 0x7f5c7c1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c7a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0a7b90 0x7f5c7c1535b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40838/0 (socket says 172.21.15.27:40838) 2024-03-28T19:02:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c7a000640 1 -- 172.21.15.27:0/1633785334 learned_addr learned my addr 172.21.15.27:0/1633785334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c7aa00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c7c0711d0 0x7f5c7c0ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c7a000640 1 -- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0c91f0 msgr2=0x7f5c7c153af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.478+0000 7f5c7b400640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0c91f0 0x7f5c7c153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c7a000640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0c91f0 0x7f5c7c153af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c7a000640 1 -- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c7c0711d0 msgr2=0x7f5c7c0ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c7a000640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c7c0711d0 0x7f5c7c0ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c7a000640 1 -- 172.21.15.27:0/1633785334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c0adca0 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c7aa00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c7c0711d0 0x7f5c7c0ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c7b400640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0c91f0 0x7f5c7c153af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c7a000640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0a7b90 0x7f5c7c1535b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800d830 tx=0x7f5c6800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c72a00640 1 -- 172.21.15.27:0/1633785334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6800b6c0 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c72a00640 1 -- 172.21.15.27:0/1633785334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c68019070 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.479+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7c152fc0 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.480+0000 7f5c72a00640 1 -- 172.21.15.27:0/1633785334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6800ba80 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.480+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7c0ad3a0 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.481+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7c150e00 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.483+0000 7f5c72a00640 1 -- 172.21.15.27:0/1633785334 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5c6801e460 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.484+0000 7f5c72a00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5c06bbd0 0x7f5c5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.484+0000 7f5c72a00640 1 -- 172.21.15.27:0/1633785334 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7439+0+0 (secure 0 0 0) 0x7f5c6808b790 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.484+0000 7f5c7aa00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5c06bbd0 0x7f5c5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.485+0000 7f5c7aa00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5c06bbd0 0x7f5c5c06e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c036000 tx=0x7f5c6c03b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.494+0000 7f5c72a00640 1 -- 172.21.15.27:0/1633785334 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5c7c150e00 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.755+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c7c1509b0 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.756+0000 7f5c72a00640 1 -- 172.21.15.27:0/1633785334 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f5c68059c50 con 0x7f5c7c0a7b90 2024-03-28T19:02:55.755 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:55.756 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:55.756 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:02:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5c06bbd0 msgr2=0x7f5c5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5c06bbd0 0x7f5c5c06e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c036000 tx=0x7f5c6c03b040 comp rx=0 tx=0).stop 2024-03-28T19:02:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0a7b90 msgr2=0x7f5c7c1535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0a7b90 0x7f5c7c1535b0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800d830 tx=0x7f5c6800dcf0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 shutdown_connections 2024-03-28T19:02:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5c06bbd0 0x7f5c5c06e090 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c7c0c91f0 0x7f5c7c153af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7c0a7b90 0x7f5c7c1535b0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 --2- 172.21.15.27:0/1633785334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c7c0711d0 0x7f5c7c0ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 >> 172.21.15.27:0/1633785334 conn(0x7f5c7c0aaab0 msgr2=0x7f5c7c0c4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 shutdown_connections 2024-03-28T19:02:55.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:55.759+0000 7f5c81a00640 1 -- 172.21.15.27:0/1633785334 wait complete. 2024-03-28T19:02:55.777 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:02:57.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:02:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.014+0000 7fe74ec00640 1 Processor -- start 2024-03-28T19:02:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.014+0000 7fe74ec00640 1 -- start start 2024-03-28T19:02:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.017+0000 7fe74ec00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7480a7060 0x7fe7480c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.017+0000 7fe74ec00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe748155700 0x7fe7480c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.017+0000 7fe74ec00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe748152ad0 0x7fe74806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.017+0000 7fe74ec00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe748072d40 con 0x7fe748155700 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.017+0000 7fe74ec00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe748073550 con 0x7fe748152ad0 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.017+0000 7fe74ec00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7480c42d0 con 0x7fe7480a7060 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.017+0000 7fe74ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe748155700 0x7fe7480c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe748155700 0x7fe7480c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47730/0 (socket says 172.21.15.27:47730) 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74ce00640 1 -- 172.21.15.27:0/3247222902 learned_addr learned my addr 172.21.15.27:0/3247222902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74d800640 1 --2- 172.21.15.27:0/3247222902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7480a7060 0x7fe7480c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74e200640 1 --2- 172.21.15.27:0/3247222902 >> v2:172.21.15.155:3300/0 conn(0x7fe748152ad0 0x7fe74806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74ce00640 1 -- 172.21.15.27:0/3247222902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7480a7060 msgr2=0x7fe7480c2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74ce00640 1 --2- 172.21.15.27:0/3247222902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7480a7060 0x7fe7480c2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74ce00640 1 -- 172.21.15.27:0/3247222902 >> v2:172.21.15.155:3300/0 conn(0x7fe748152ad0 msgr2=0x7fe74806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:02:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74ce00640 1 --2- 172.21.15.27:0/3247222902 >> v2:172.21.15.155:3300/0 conn(0x7fe748152ad0 0x7fe74806c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74ce00640 1 -- 172.21.15.27:0/3247222902 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748072f90 con 0x7fe748155700 2024-03-28T19:02:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.018+0000 7fe74e200640 1 --2- 172.21.15.27:0/3247222902 >> v2:172.21.15.155:3300/0 conn(0x7fe748152ad0 0x7fe74806c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:02:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.019+0000 7fe74ce00640 1 --2- 172.21.15.27:0/3247222902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c3d90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe748072c20 tx=0x7fe73402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1621ad972190cb00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe747e00640 1 -- 172.21.15.27:0/3247222902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734035020 con 0x7fe748155700 2024-03-28T19:02:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe747e00640 1 -- 172.21.15.27:0/3247222902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe734031870 con 0x7fe748155700 2024-03-28T19:02:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe747e00640 1 -- 172.21.15.27:0/3247222902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734031c40 con 0x7fe748155700 2024-03-28T19:02:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe74ec00640 1 -- 172.21.15.27:0/3247222902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 msgr2=0x7fe7480c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe74ec00640 1 --2- 172.21.15.27:0/3247222902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c3d90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe748072c20 tx=0x7fe73402b910 comp rx=0 tx=0).stop 2024-03-28T19:02:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe74ec00640 1 -- 172.21.15.27:0/3247222902 shutdown_connections 2024-03-28T19:02:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe74ec00640 1 --2- 172.21.15.27:0/3247222902 >> v2:172.21.15.155:3300/0 conn(0x7fe748152ad0 0x7fe74806c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe74ec00640 1 --2- 172.21.15.27:0/3247222902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c3d90 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe74ec00640 1 --2- 172.21.15.27:0/3247222902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7480a7060 0x7fe7480c2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe74ec00640 1 -- 172.21.15.27:0/3247222902 >> 172.21.15.27:0/3247222902 conn(0x7fe7480c7aa0 msgr2=0x7fe7480c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.020+0000 7fe74ec00640 1 -- 172.21.15.27:0/3247222902 shutdown_connections 2024-03-28T19:02:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.021+0000 7fe74ec00640 1 -- 172.21.15.27:0/3247222902 wait complete. 2024-03-28T19:02:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.022+0000 7fe74ec00640 1 Processor -- start 2024-03-28T19:02:58.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.022+0000 7fe74ec00640 1 -- start start 2024-03-28T19:02:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74ec00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7480a7060 0x7fe7481eb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74ec00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe748152ad0 0x7fe7481ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74ec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74ec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7481525b0 con 0x7fe748155700 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74ec00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7481522d0 con 0x7fe7480a7060 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74ec00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe748155b60 con 0x7fe748152ad0 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47738/0 (socket says 172.21.15.27:47738) 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74e200640 1 -- 172.21.15.27:0/1650466918 learned_addr learned my addr 172.21.15.27:0/1650466918 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74d800640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7480a7060 0x7fe7481eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.024+0000 7fe74ce00640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe748152ad0 0x7fe7481ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe74e200640 1 -- 172.21.15.27:0/1650466918 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe748152ad0 msgr2=0x7fe7481ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe74e200640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe748152ad0 0x7fe7481ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe74e200640 1 -- 172.21.15.27:0/1650466918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7480a7060 msgr2=0x7fe7481eb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe74e200640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7480a7060 0x7fe7481eb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe74e200640 1 -- 172.21.15.27:0/1650466918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7481ebd80 con 0x7fe748155700 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe74e200640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c1140 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe74000cf20 tx=0x7fe74000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe745600640 1 -- 172.21.15.27:0/1650466918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74001a070 con 0x7fe748155700 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe745600640 1 -- 172.21.15.27:0/1650466918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74000d840 con 0x7fe748155700 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe745600640 1 -- 172.21.15.27:0/1650466918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74000dc30 con 0x7fe748155700 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.025+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7481ee8e0 con 0x7fe748155700 2024-03-28T19:02:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.026+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7481eb040 con 0x7fe748155700 2024-03-28T19:02:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.027+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe748070200 con 0x7fe748155700 2024-03-28T19:02:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.028+0000 7fe745600640 1 -- 172.21.15.27:0/1650466918 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe74001d070 con 0x7fe748155700 2024-03-28T19:02:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.029+0000 7fe745600640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72806bbd0 0x7fe72806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:02:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.029+0000 7fe745600640 1 -- 172.21.15.27:0/1650466918 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7439+0+0 (secure 0 0 0) 0x7fe74008b950 con 0x7fe748155700 2024-03-28T19:02:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.029+0000 7fe74d800640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72806bbd0 0x7fe72806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:02:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.030+0000 7fe74d800640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72806bbd0 0x7fe72806e090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:02:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.035+0000 7fe745600640 1 -- 172.21.15.27:0/1650466918 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe740059e10 con 0x7fe748155700 2024-03-28T19:02:58.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.283+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7481eae30 con 0x7fe748155700 2024-03-28T19:02:58.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.284+0000 7fe745600640 1 -- 172.21.15.27:0/1650466918 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fe740059710 con 0x7fe748155700 2024-03-28T19:02:58.283 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:02:58.283 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:02:58.284 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:02:58.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.286+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72806bbd0 msgr2=0x7fe72806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:58.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.286+0000 7fe74ec00640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72806bbd0 0x7fe72806e090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe73c006fd0 tx=0x7fe73c008040 comp rx=0 tx=0).stop 2024-03-28T19:02:58.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.286+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 msgr2=0x7fe7480c1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:02:58.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.286+0000 7fe74ec00640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c1140 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe74000cf20 tx=0x7fe74000a7d0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.287+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 shutdown_connections 2024-03-28T19:02:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.287+0000 7fe74ec00640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72806bbd0 0x7fe72806e090 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.287+0000 7fe74ec00640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe748155700 0x7fe7480c1140 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.287+0000 7fe74ec00640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe748152ad0 0x7fe7481ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.287+0000 7fe74ec00640 1 --2- 172.21.15.27:0/1650466918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7480a7060 0x7fe7481eb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:02:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.287+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 >> 172.21.15.27:0/1650466918 conn(0x7fe7480c7aa0 msgr2=0x7fe74806bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:02:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.287+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 shutdown_connections 2024-03-28T19:02:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:02:58.287+0000 7fe74ec00640 1 -- 172.21.15.27:0/1650466918 wait complete. 2024-03-28T19:02:58.307 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:02:59.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:02:59.450 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:03:00.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.583+0000 7ff27be00640 1 Processor -- start 2024-03-28T19:03:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.583+0000 7ff27be00640 1 -- start start 2024-03-28T19:03:00.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.585+0000 7ff27be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c150ae0 0x7ff27c0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.585+0000 7ff27be00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff27c0a61a0 0x7ff27c0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.585+0000 7ff27be00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27c150ec0 con 0x7ff27c0c6ee0 2024-03-28T19:03:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.585+0000 7ff27be00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27c06dc70 con 0x7ff27c0a61a0 2024-03-28T19:03:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.585+0000 7ff27be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c153550 con 0x7ff27c150ae0 2024-03-28T19:03:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.586+0000 7ff27aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c150ae0 0x7ff27c0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.586+0000 7ff27a000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff27c0a61a0 0x7ff27c0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.586+0000 7ff27aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c150ae0 0x7ff27c0a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:40412/0 (socket says 172.21.15.27:40412) 2024-03-28T19:03:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.586+0000 7ff27aa00640 1 -- 172.21.15.27:0/3782530904 learned_addr learned my addr 172.21.15.27:0/3782530904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.587+0000 7ff27a000640 1 -- 172.21.15.27:0/3782530904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c150ae0 msgr2=0x7ff27c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.587+0000 7ff27a000640 1 --2- 172.21.15.27:0/3782530904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c150ae0 0x7ff27c0a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.587+0000 7ff27a000640 1 -- 172.21.15.27:0/3782530904 >> v1:172.21.15.155:6789/0 conn(0x7ff27c0c6ee0 legacy=0x7ff27c153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:00.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.587+0000 7ff27a000640 1 -- 172.21.15.27:0/3782530904 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c06df10 con 0x7ff27c0a61a0 2024-03-28T19:03:00.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.587+0000 7ff27aa00640 1 --2- 172.21.15.27:0/3782530904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c150ae0 0x7ff27c0a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:03:00.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.588+0000 7ff27a000640 1 --2- 172.21.15.27:0/3782530904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0c69a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff2680097a0 tx=0x7ff26802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b039b44078107cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.589+0000 7ff279600640 1 -- 172.21.15.27:0/3782530904 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2680315a0 con 0x7ff27c0a61a0 2024-03-28T19:03:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.589+0000 7ff279600640 1 -- 172.21.15.27:0/3782530904 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff268037070 con 0x7ff27c0a61a0 2024-03-28T19:03:00.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.589+0000 7ff279600640 1 -- 172.21.15.27:0/3782530904 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268031960 con 0x7ff27c0a61a0 2024-03-28T19:03:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.590+0000 7ff27be00640 1 -- 172.21.15.27:0/3782530904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c0a61a0 msgr2=0x7ff27c0c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.590+0000 7ff27be00640 1 --2- 172.21.15.27:0/3782530904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0c69a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff2680097a0 tx=0x7ff26802b840 comp rx=0 tx=0).stop 2024-03-28T19:03:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.590+0000 7ff27be00640 1 -- 172.21.15.27:0/3782530904 shutdown_connections 2024-03-28T19:03:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.590+0000 7ff27be00640 1 --2- 172.21.15.27:0/3782530904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0c69a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.590+0000 7ff27be00640 1 --2- 172.21.15.27:0/3782530904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c150ae0 0x7ff27c0a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.590+0000 7ff27be00640 1 -- 172.21.15.27:0/3782530904 >> 172.21.15.27:0/3782530904 conn(0x7ff27c06b0c0 msgr2=0x7ff27c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.590+0000 7ff27be00640 1 -- 172.21.15.27:0/3782530904 shutdown_connections 2024-03-28T19:03:00.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.590+0000 7ff27be00640 1 -- 172.21.15.27:0/3782530904 wait complete. 2024-03-28T19:03:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.593+0000 7ff27be00640 1 Processor -- start 2024-03-28T19:03:00.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.593+0000 7ff27be00640 1 -- start start 2024-03-28T19:03:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.594+0000 7ff27be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.594+0000 7ff27be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c0c6ee0 0x7ff27c072700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.594+0000 7ff27be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c150ae0 0x7ff27c072c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.594+0000 7ff27be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c06e7b0 con 0x7ff27c0a61a0 2024-03-28T19:03:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.594+0000 7ff27be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c06e4d0 con 0x7ff27c150ae0 2024-03-28T19:03:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.594+0000 7ff27be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c1569d0 con 0x7ff27c0c6ee0 2024-03-28T19:03:00.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:00.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0bff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47754/0 (socket says 172.21.15.27:47754) 2024-03-28T19:03:00.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27aa00640 1 -- 172.21.15.27:0/253256520 learned_addr learned my addr 172.21.15.27:0/253256520 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27a000640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c0c6ee0 0x7ff27c072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27b400640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c150ae0 0x7ff27c072c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27aa00640 1 -- 172.21.15.27:0/253256520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c0c6ee0 msgr2=0x7ff27c072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27aa00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c0c6ee0 0x7ff27c072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27aa00640 1 -- 172.21.15.27:0/253256520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c150ae0 msgr2=0x7ff27c072c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27aa00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c150ae0 0x7ff27c072c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.595+0000 7ff27aa00640 1 -- 172.21.15.27:0/253256520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c071f70 con 0x7ff27c0a61a0 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.596+0000 7ff27aa00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0bff90 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00e9d0 tx=0x7ff26c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:00.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.596+0000 7ff273400640 1 -- 172.21.15.27:0/253256520 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c017070 con 0x7ff27c0a61a0 2024-03-28T19:03:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.596+0000 7ff273400640 1 -- 172.21.15.27:0/253256520 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c00ca60 con 0x7ff27c0a61a0 2024-03-28T19:03:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.596+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27c070810 con 0x7ff27c0a61a0 2024-03-28T19:03:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.596+0000 7ff273400640 1 -- 172.21.15.27:0/253256520 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c00ce10 con 0x7ff27c0a61a0 2024-03-28T19:03:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.596+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27c070ab0 con 0x7ff27c0a61a0 2024-03-28T19:03:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.597+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27c067dd0 con 0x7ff27c0a61a0 2024-03-28T19:03:00.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.600+0000 7ff273400640 1 -- 172.21.15.27:0/253256520 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff26c01a070 con 0x7ff27c0a61a0 2024-03-28T19:03:00.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.601+0000 7ff273400640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff25c06bbd0 0x7ff25c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.601+0000 7ff273400640 1 -- 172.21.15.27:0/253256520 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7439+0+0 (secure 0 0 0) 0x7ff26c08b100 con 0x7ff27c0a61a0 2024-03-28T19:03:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.602+0000 7ff27a000640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff25c06bbd0 0x7ff25c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:00.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.602+0000 7ff27a000640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff25c06bbd0 0x7ff25c06e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff268036000 tx=0x7ff268008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.606+0000 7ff273400640 1 -- 172.21.15.27:0/253256520 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff27c071f70 con 0x7ff27c0a61a0 2024-03-28T19:03:00.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.862+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff27c069640 con 0x7ff27c0a61a0 2024-03-28T19:03:00.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.863+0000 7ff273400640 1 -- 172.21.15.27:0/253256520 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7ff26c059520 con 0x7ff27c0a61a0 2024-03-28T19:03:00.862 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:00.863 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:00.863 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:03:00.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.865+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff25c06bbd0 msgr2=0x7ff25c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:00.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.865+0000 7ff27be00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff25c06bbd0 0x7ff25c06e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff268036000 tx=0x7ff268008030 comp rx=0 tx=0).stop 2024-03-28T19:03:00.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.865+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff27c0a61a0 msgr2=0x7ff27c0bff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:00.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.865+0000 7ff27be00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0bff90 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00e9d0 tx=0x7ff26c00ee90 comp rx=0 tx=0).stop 2024-03-28T19:03:00.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.865+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 shutdown_connections 2024-03-28T19:03:00.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.865+0000 7ff27be00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff25c06bbd0 0x7ff25c06e090 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.865+0000 7ff27be00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff27c150ae0 0x7ff27c072c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.866+0000 7ff27be00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff27c0c6ee0 0x7ff27c072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.866+0000 7ff27be00640 1 --2- 172.21.15.27:0/253256520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff27c0a61a0 0x7ff27c0bff90 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:00.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.866+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 >> 172.21.15.27:0/253256520 conn(0x7ff27c06b0c0 msgr2=0x7ff27c0c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:00.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.866+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 shutdown_connections 2024-03-28T19:03:00.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:00.866+0000 7ff27be00640 1 -- 172.21.15.27:0/253256520 wait complete. 2024-03-28T19:03:00.883 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:03:02.884 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.121+0000 7f1712a00640 1 Processor -- start 2024-03-28T19:03:03.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.122+0000 7f1712a00640 1 -- start start 2024-03-28T19:03:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.124+0000 7f1712a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c153bd0 0x7f170c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.124+0000 7f1712a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170c0c3d90 con 0x7f170c150c70 2024-03-28T19:03:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.124+0000 7f1712a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170c0ab710 con 0x7f170c0a9cb0 2024-03-28T19:03:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.124+0000 7f1712a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0ab4e0 con 0x7f170c153bd0 2024-03-28T19:03:03.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.124+0000 7f170be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f170c150c70 0x7f170c0c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52500/0 (socket says 172.21.15.27:52500) 2024-03-28T19:03:03.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.124+0000 7f170be00640 1 -- 172.21.15.27:0/2884591444 learned_addr learned my addr 172.21.15.27:0/2884591444 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:03.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.125+0000 7f1710a00640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c153bd0 0x7f170c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:03.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.125+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777449740 0 0) 0x7f170c0c3d90 con 0x7f170c150c70 2024-03-28T19:03:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.125+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f0003520 con 0x7f170c150c70 2024-03-28T19:03:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1382517931 0 0) 0x7f16f0003520 con 0x7f170c150c70 2024-03-28T19:03:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c153bd0 msgr2=0x7f170c0c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c153bd0 0x7f170c0c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 >> v1:172.21.15.155:6789/0 conn(0x7f170c0a9cb0 legacy=0x7f170c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c0ab9f0 con 0x7f170c150c70 2024-03-28T19:03:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f16fc001670 con 0x7f170c150c70 2024-03-28T19:03:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 >> v1:172.21.15.27:6789/0 conn(0x7f170c150c70 legacy=0x7f170c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00039e0 0x7f16f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c153bd0 0x7f16f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f0004d40 0x7f16f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f000fed0 con 0x7f16f00039e0 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0010080 con 0x7f16f0004d40 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0010280 con 0x7f170c153bd0 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.126+0000 7f170be00640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00039e0 0x7f16f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.127+0000 7f1710a00640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f0004d40 0x7f16f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:03.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.127+0000 7f1703e00640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c153bd0 0x7f16f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.127+0000 7f170be00640 1 -- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c153bd0 msgr2=0x7f16f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.127+0000 7f170be00640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c153bd0 0x7f16f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.127+0000 7f170be00640 1 -- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f0004d40 msgr2=0x7f16f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.127+0000 7f170be00640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f0004d40 0x7f16f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.127+0000 7f170be00640 1 -- 172.21.15.27:0/2884591444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c0ab9f0 con 0x7f16f00039e0 2024-03-28T19:03:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.128+0000 7f170be00640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00039e0 0x7f16f0003e00 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f16fc001050 tx=0x7f16f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8937c40d34bebc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.128+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc007d10 con 0x7f16f00039e0 2024-03-28T19:03:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.128+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc030070 con 0x7f16f00039e0 2024-03-28T19:03:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.128+0000 7f170b400640 1 -- 172.21.15.27:0/2884591444 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc005d10 con 0x7f16f00039e0 2024-03-28T19:03:03.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.129+0000 7f1712a00640 1 -- 172.21.15.27:0/2884591444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00039e0 msgr2=0x7f16f0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:03.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.129+0000 7f1712a00640 1 --2- 172.21.15.27:0/2884591444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f00039e0 0x7f16f0003e00 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f16fc001050 tx=0x7f16f0005830 comp rx=0 tx=0).stop 2024-03-28T19:03:03.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.129+0000 7f1710a00640 1 -- 172.21.15.27:0/2884591444 reap_dead start 2024-03-28T19:03:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.129+0000 7f1712a00640 1 -- 172.21.15.27:0/2884591444 shutdown_connections 2024-03-28T19:03:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.129+0000 7f1712a00640 1 -- 172.21.15.27:0/2884591444 >> 172.21.15.27:0/2884591444 conn(0x7f170c06b9d0 msgr2=0x7f170c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.129+0000 7f1712a00640 1 -- 172.21.15.27:0/2884591444 shutdown_connections 2024-03-28T19:03:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.130+0000 7f1712a00640 1 -- 172.21.15.27:0/2884591444 wait complete. 2024-03-28T19:03:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.131+0000 7f1712a00640 1 Processor -- start 2024-03-28T19:03:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.132+0000 7f1712a00640 1 -- start start 2024-03-28T19:03:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.132+0000 7f1712a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f00039e0 0x7f170c14fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.132+0000 7f1712a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c14b3a0 0x7f170c0c0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1712a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c0c0740 0x7f170c1ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1712a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c154060 con 0x7f170c14b3a0 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1712a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0ac1f0 con 0x7f16f00039e0 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1712a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0c1810 con 0x7f170c0c0740 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1703e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c14b3a0 0x7f170c0c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1703e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c14b3a0 0x7f170c0c0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47768/0 (socket says 172.21.15.27:47768) 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1703e00640 1 -- 172.21.15.27:0/2004558960 learned_addr learned my addr 172.21.15.27:0/2004558960 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1710a00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c0c0740 0x7f170c1ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f170be00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f00039e0 0x7f170c14fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1703e00640 1 -- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c0c0740 msgr2=0x7f170c1ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1703e00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c0c0740 0x7f170c1ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1703e00640 1 -- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f00039e0 msgr2=0x7f170c14fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1703e00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f00039e0 0x7f170c14fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1703e00640 1 -- 172.21.15.27:0/2004558960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c1e7020 con 0x7f170c14b3a0 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.133+0000 7f1710a00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c0c0740 0x7f170c1ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.134+0000 7f1703e00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c14b3a0 0x7f170c0c0200 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e500 tx=0x7f16f400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.134+0000 7f1708c00640 1 -- 172.21.15.27:0/2004558960 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f40053a0 con 0x7f170c14b3a0 2024-03-28T19:03:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.134+0000 7f1708c00640 1 -- 172.21.15.27:0/2004558960 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f4018070 con 0x7f170c14b3a0 2024-03-28T19:03:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.134+0000 7f1708c00640 1 -- 172.21.15.27:0/2004558960 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4003810 con 0x7f170c14b3a0 2024-03-28T19:03:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.134+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c1e72e0 con 0x7f170c14b3a0 2024-03-28T19:03:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.134+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c1e78a0 con 0x7f170c14b3a0 2024-03-28T19:03:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.136+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c04f400 con 0x7f170c14b3a0 2024-03-28T19:03:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.137+0000 7f1708c00640 1 -- 172.21.15.27:0/2004558960 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f16f40039b0 con 0x7f170c14b3a0 2024-03-28T19:03:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.137+0000 7f1708c00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f16e806bc00 0x7f16e806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.138+0000 7f1708c00640 1 -- 172.21.15.27:0/2004558960 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7439+0+0 (secure 0 0 0) 0x7f16f408ae80 con 0x7f170c14b3a0 2024-03-28T19:03:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.138+0000 7f170be00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f16e806bc00 0x7f16e806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.139+0000 7f170be00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f16e806bc00 0x7f16e806e0c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f16fc034000 tx=0x7f16fc009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:03.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.142+0000 7f1708c00640 1 -- 172.21.15.27:0/2004558960 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f16f4059220 con 0x7f170c14b3a0 2024-03-28T19:03:03.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.429+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f170c0c5f00 con 0x7f170c14b3a0 2024-03-28T19:03:03.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.431+0000 7f1708c00640 1 -- 172.21.15.27:0/2004558960 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f16f4058b20 con 0x7f170c14b3a0 2024-03-28T19:03:03.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:03.431 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5862,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:01:49.910084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":99,"addr":"172.21.15.155:6835/1254917384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1254917384},{"type":"v1","addr":"172.21.15.155:6835","nonce":1254917384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4247,"qdb_cluster":[4247]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:03.431 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 66 2024-03-28T19:03:03.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.433+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f16e806bc00 msgr2=0x7f16e806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.433+0000 7f1712a00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f16e806bc00 0x7f16e806e0c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f16fc034000 tx=0x7f16fc009030 comp rx=0 tx=0).stop 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c14b3a0 msgr2=0x7f170c0c0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c14b3a0 0x7f170c0c0200 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e500 tx=0x7f16f400e9c0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 shutdown_connections 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f16e806bc00 0x7f16e806e0c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f170c0c0740 0x7f170c1ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f170c14b3a0 0x7f170c0c0200 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 --2- 172.21.15.27:0/2004558960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16f00039e0 0x7f170c14fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:03.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 >> 172.21.15.27:0/2004558960 conn(0x7f170c06b9d0 msgr2=0x7f170c0c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:03.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 shutdown_connections 2024-03-28T19:03:03.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:03.434+0000 7f1712a00640 1 -- 172.21.15.27:0/2004558960 wait complete. 2024-03-28T19:03:03.452 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:03:05.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.691+0000 7f70f9200640 1 Processor -- start 2024-03-28T19:03:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.691+0000 7f70f9200640 1 -- start start 2024-03-28T19:03:05.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.692+0000 7f70f9200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:05.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.692+0000 7f70f9200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f70f40abe80 0x7f70f40ac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:05.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.692+0000 7f70f9200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f40aa6d0 con 0x7f70f4066dd0 2024-03-28T19:03:05.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.692+0000 7f70f9200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f4150e30 con 0x7f70f40abe80 2024-03-28T19:03:05.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.692+0000 7f70f9200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f406f360 con 0x7f70f4150a50 2024-03-28T19:03:05.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.693+0000 7f70f2800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f70f40abe80 0x7f70f40ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:05.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.693+0000 7f70f1e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:05.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.693+0000 7f70f1e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:40450/0 (socket says 172.21.15.27:40450) 2024-03-28T19:03:05.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.693+0000 7f70f1e00640 1 -- 172.21.15.27:0/4218764520 learned_addr learned my addr 172.21.15.27:0/4218764520 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:05.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.694+0000 7f70f1e00640 1 -- 172.21.15.27:0/4218764520 >> v2:172.21.15.155:3300/0 conn(0x7f70f40abe80 msgr2=0x7f70f40ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:05.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.694+0000 7f70f1e00640 1 --2- 172.21.15.27:0/4218764520 >> v2:172.21.15.155:3300/0 conn(0x7f70f40abe80 0x7f70f40ac310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:05.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.694+0000 7f70f1e00640 1 -- 172.21.15.27:0/4218764520 >> v1:172.21.15.155:6789/0 conn(0x7f70f4066dd0 legacy=0x7f70f40ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:05.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.694+0000 7f70f1e00640 1 -- 172.21.15.27:0/4218764520 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f4154040 con 0x7f70f4150a50 2024-03-28T19:03:05.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.695+0000 7f70f1e00640 1 --2- 172.21.15.27:0/4218764520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f406b0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009870 tx=0x7f70e402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be74b522c67015ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f0a00640 1 -- 172.21.15.27:0/4218764520 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e40317c0 con 0x7f70f4150a50 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f0a00640 1 -- 172.21.15.27:0/4218764520 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e4037070 con 0x7f70f4150a50 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f0a00640 1 -- 172.21.15.27:0/4218764520 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4031b80 con 0x7f70f4150a50 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f9200640 1 -- 172.21.15.27:0/4218764520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 msgr2=0x7f70f406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f9200640 1 --2- 172.21.15.27:0/4218764520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f406b0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009870 tx=0x7f70e402b860 comp rx=0 tx=0).stop 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f9200640 1 -- 172.21.15.27:0/4218764520 shutdown_connections 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f9200640 1 --2- 172.21.15.27:0/4218764520 >> v2:172.21.15.155:3300/0 conn(0x7f70f40abe80 0x7f70f40ac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f9200640 1 --2- 172.21.15.27:0/4218764520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f406b0c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f9200640 1 -- 172.21.15.27:0/4218764520 >> 172.21.15.27:0/4218764520 conn(0x7f70f40c3ff0 msgr2=0x7f70f40c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.696+0000 7f70f9200640 1 -- 172.21.15.27:0/4218764520 shutdown_connections 2024-03-28T19:03:05.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.697+0000 7f70f9200640 1 -- 172.21.15.27:0/4218764520 wait complete. 2024-03-28T19:03:05.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.699+0000 7f70f9200640 1 Processor -- start 2024-03-28T19:03:05.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.699+0000 7f70f9200640 1 -- start start 2024-03-28T19:03:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f4066dd0 0x7f70f41ec680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70f40abe80 0x7f70f41ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f9200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f4147a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f9200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f40672e0 con 0x7f70f4066dd0 2024-03-28T19:03:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f9200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f40acbe0 con 0x7f70f40abe80 2024-03-28T19:03:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f9200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f4067460 con 0x7f70f4150a50 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f4066dd0 0x7f70f41ec680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f4066dd0 0x7f70f41ec680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47772/0 (socket says 172.21.15.27:47772) 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f1e00640 1 -- 172.21.15.27:0/761781435 learned_addr learned my addr 172.21.15.27:0/761781435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f1e00640 1 -- 172.21.15.27:0/761781435 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 msgr2=0x7f70f4147a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f1e00640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f4147a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f1e00640 1 -- 172.21.15.27:0/761781435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70f40abe80 msgr2=0x7f70f41ecbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.701+0000 7f70f1400640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70f40abe80 0x7f70f41ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.702+0000 7f70f1e00640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70f40abe80 0x7f70f41ecbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.702+0000 7f70f1e00640 1 -- 172.21.15.27:0/761781435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f41ed100 con 0x7f70f4066dd0 2024-03-28T19:03:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.702+0000 7f70f1400640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70f40abe80 0x7f70f41ecbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.702+0000 7f70f1e00640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f4066dd0 0x7f70f41ec680 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f70e4000c00 tx=0x7f70e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.702+0000 7f70ea000640 1 -- 172.21.15.27:0/761781435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4032650 con 0x7f70f4066dd0 2024-03-28T19:03:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.702+0000 7f70ea000640 1 -- 172.21.15.27:0/761781435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e4032810 con 0x7f70f4066dd0 2024-03-28T19:03:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.702+0000 7f70ea000640 1 -- 172.21.15.27:0/761781435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4032c00 con 0x7f70f4066dd0 2024-03-28T19:03:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.703+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f41de4c0 con 0x7f70f4066dd0 2024-03-28T19:03:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.703+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f41ded50 con 0x7f70f4066dd0 2024-03-28T19:03:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.704+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f404f400 con 0x7f70f4066dd0 2024-03-28T19:03:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.705+0000 7f70ea000640 1 -- 172.21.15.27:0/761781435 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f70e404c060 con 0x7f70f4066dd0 2024-03-28T19:03:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.706+0000 7f70ea000640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f70d406bc20 0x7f70d406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.707+0000 7f70ea000640 1 -- 172.21.15.27:0/761781435 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f70e40acc40 con 0x7f70f4066dd0 2024-03-28T19:03:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.707+0000 7f70f1400640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f70d406bc20 0x7f70d406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.708+0000 7f70f1400640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f70d406bc20 0x7f70d406e0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f70f4153940 tx=0x7f70e0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:05.711+0000 7f70ea000640 1 -- 172.21.15.27:0/761781435 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f70e4040070 con 0x7f70f4066dd0 2024-03-28T19:03:06.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.028+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70f40c4c90 con 0x7f70f4066dd0 2024-03-28T19:03:06.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.028+0000 7f70ea000640 1 -- 172.21.15.27:0/761781435 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f70e407b1a0 con 0x7f70f4066dd0 2024-03-28T19:03:06.028 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:06.028 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:04.718543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:06.029 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-28T19:03:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f70d406bc20 msgr2=0x7f70d406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f70d406bc20 0x7f70d406e0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f70f4153940 tx=0x7f70e0009040 comp rx=0 tx=0).stop 2024-03-28T19:03:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f4066dd0 msgr2=0x7f70f41ec680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f4066dd0 0x7f70f41ec680 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f70e4000c00 tx=0x7f70e40046c0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 shutdown_connections 2024-03-28T19:03:06.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f70d406bc20 0x7f70d406e0e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70f4150a50 0x7f70f4147a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70f40abe80 0x7f70f41ecbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 --2- 172.21.15.27:0/761781435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70f4066dd0 0x7f70f41ec680 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 >> 172.21.15.27:0/761781435 conn(0x7f70f40c3ff0 msgr2=0x7f70f40acd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:06.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 shutdown_connections 2024-03-28T19:03:06.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.035+0000 7f70f9200640 1 -- 172.21.15.27:0/761781435 wait complete. 2024-03-28T19:03:06.061 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-03-28T19:03:06.062 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:03:06.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:06.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.271+0000 7f0921600640 1 Processor -- start 2024-03-28T19:03:06.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.271+0000 7f0921600640 1 -- start start 2024-03-28T19:03:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.273+0000 7f0921600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.273+0000 7f0921600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f091c0be5d0 0x7f091c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.273+0000 7f0921600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c156230 con 0x7f091c0c58c0 2024-03-28T19:03:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.273+0000 7f0921600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c158bd0 con 0x7f091c0be5d0 2024-03-28T19:03:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.273+0000 7f0921600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c158eb0 con 0x7f091c0c8ed0 2024-03-28T19:03:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.274+0000 7f091b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.274+0000 7f091b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:58910/0 (socket says 172.21.15.27:58910) 2024-03-28T19:03:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.274+0000 7f091b400640 1 -- 172.21.15.27:0/1001377271 learned_addr learned my addr 172.21.15.27:0/1001377271 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.274+0000 7f0920c00640 1 --2- 172.21.15.27:0/1001377271 >> v2:172.21.15.155:3300/0 conn(0x7f091c0be5d0 0x7f091c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.275+0000 7f091b400640 1 -- 172.21.15.27:0/1001377271 >> v2:172.21.15.155:3300/0 conn(0x7f091c0be5d0 msgr2=0x7f091c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.275+0000 7f091b400640 1 --2- 172.21.15.27:0/1001377271 >> v2:172.21.15.155:3300/0 conn(0x7f091c0be5d0 0x7f091c155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.275+0000 7f091b400640 1 -- 172.21.15.27:0/1001377271 >> v1:172.21.15.155:6789/0 conn(0x7f091c0c58c0 legacy=0x7f091c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.275+0000 7f091b400640 1 -- 172.21.15.27:0/1001377271 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c159190 con 0x7f091c0c8ed0 2024-03-28T19:03:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.276+0000 7f091b400640 1 --2- 172.21.15.27:0/1001377271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c0c5ca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0908009870 tx=0x7f090802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d1cbe285f9f79c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.277+0000 7f091aa00640 1 -- 172.21.15.27:0/1001377271 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908035020 con 0x7f091c0c8ed0 2024-03-28T19:03:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.277+0000 7f091aa00640 1 -- 172.21.15.27:0/1001377271 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908031640 con 0x7f091c0c8ed0 2024-03-28T19:03:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.277+0000 7f091aa00640 1 -- 172.21.15.27:0/1001377271 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090802d040 con 0x7f091c0c8ed0 2024-03-28T19:03:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.278+0000 7f0921600640 1 -- 172.21.15.27:0/1001377271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 msgr2=0x7f091c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.278+0000 7f0921600640 1 --2- 172.21.15.27:0/1001377271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c0c5ca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0908009870 tx=0x7f090802b8f0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.278+0000 7f0921600640 1 -- 172.21.15.27:0/1001377271 shutdown_connections 2024-03-28T19:03:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.278+0000 7f0921600640 1 --2- 172.21.15.27:0/1001377271 >> v2:172.21.15.155:3300/0 conn(0x7f091c0be5d0 0x7f091c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.278+0000 7f0921600640 1 --2- 172.21.15.27:0/1001377271 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c0c5ca0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.278+0000 7f0921600640 1 -- 172.21.15.27:0/1001377271 >> 172.21.15.27:0/1001377271 conn(0x7f091c06b6f0 msgr2=0x7f091c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.278+0000 7f0921600640 1 -- 172.21.15.27:0/1001377271 shutdown_connections 2024-03-28T19:03:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.278+0000 7f0921600640 1 -- 172.21.15.27:0/1001377271 wait complete. 2024-03-28T19:03:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.281+0000 7f0921600640 1 Processor -- start 2024-03-28T19:03:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.281+0000 7f0921600640 1 -- start start 2024-03-28T19:03:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f0921600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f091c0be5d0 0x7f091c0ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f0921600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091c0c58c0 0x7f091c1eb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f0921600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c1ebe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f0921600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0b0820 con 0x7f091c0c58c0 2024-03-28T19:03:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f0921600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c159750 con 0x7f091c0be5d0 2024-03-28T19:03:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f0921600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0b09a0 con 0x7f091c0c8ed0 2024-03-28T19:03:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f091b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091c0c58c0 0x7f091c1eb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f091b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091c0c58c0 0x7f091c1eb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33178/0 (socket says 172.21.15.27:33178) 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f091b400640 1 -- 172.21.15.27:0/2488199697 learned_addr learned my addr 172.21.15.27:0/2488199697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f0920c00640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c1ebe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f091be00640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f091c0be5d0 0x7f091c0ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f091b400640 1 -- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 msgr2=0x7f091c1ebe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f091b400640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c1ebe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f091b400640 1 -- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f091c0be5d0 msgr2=0x7f091c0ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.282+0000 7f091b400640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f091c0be5d0 0x7f091c0ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.283+0000 7f091b400640 1 -- 172.21.15.27:0/2488199697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c0c3ca0 con 0x7f091c0c58c0 2024-03-28T19:03:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.283+0000 7f0920c00640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c1ebe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.283+0000 7f091b400640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091c0c58c0 0x7f091c1eb900 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f090802b930 tx=0x7f09080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.283+0000 7f0913e00640 1 -- 172.21.15.27:0/2488199697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908040070 con 0x7f091c0c58c0 2024-03-28T19:03:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.283+0000 7f0913e00640 1 -- 172.21.15.27:0/2488199697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090803e070 con 0x7f091c0c58c0 2024-03-28T19:03:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.283+0000 7f0913e00640 1 -- 172.21.15.27:0/2488199697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908035040 con 0x7f091c0c58c0 2024-03-28T19:03:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.283+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c0c4530 con 0x7f091c0c58c0 2024-03-28T19:03:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.283+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c0c4af0 con 0x7f091c0c58c0 2024-03-28T19:03:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.285+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091c069640 con 0x7f091c0c58c0 2024-03-28T19:03:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.286+0000 7f0913e00640 1 -- 172.21.15.27:0/2488199697 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0908038020 con 0x7f091c0c58c0 2024-03-28T19:03:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.287+0000 7f0913e00640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f08fc06bbd0 0x7f08fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.287+0000 7f0913e00640 1 -- 172.21.15.27:0/2488199697 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f09080acb40 con 0x7f091c0c58c0 2024-03-28T19:03:06.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.288+0000 7f091be00640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f08fc06bbd0 0x7f08fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.288+0000 7f091be00640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f08fc06bbd0 0x7f08fc06e090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f091c0c1dc0 tx=0x7f090c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:06.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.292+0000 7f0913e00640 1 -- 172.21.15.27:0/2488199697 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f090807ae90 con 0x7f091c0c58c0 2024-03-28T19:03:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.616+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f091c156580 con 0x7f091c0c58c0 2024-03-28T19:03:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.617+0000 7f0913e00640 1 -- 172.21.15.27:0/2488199697 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f090807a790 con 0x7f091c0c58c0 2024-03-28T19:03:06.616 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:06.616 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:04.718543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-03-28T19:03:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.620+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f08fc06bbd0 msgr2=0x7f08fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.620+0000 7f0921600640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f08fc06bbd0 0x7f08fc06e090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f091c0c1dc0 tx=0x7f090c009040 comp rx=0 tx=0).stop 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.620+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091c0c58c0 msgr2=0x7f091c1eb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.620+0000 7f0921600640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091c0c58c0 0x7f091c1eb900 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f090802b930 tx=0x7f09080046c0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.621+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 shutdown_connections 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.621+0000 7f0921600640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f08fc06bbd0 0x7f08fc06e090 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.621+0000 7f0921600640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f091c0c8ed0 0x7f091c1ebe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.621+0000 7f0921600640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091c0c58c0 0x7f091c1eb900 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.621+0000 7f0921600640 1 --2- 172.21.15.27:0/2488199697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f091c0be5d0 0x7f091c0ac930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.621+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 >> 172.21.15.27:0/2488199697 conn(0x7f091c06b6f0 msgr2=0x7f091c0aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.621+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 shutdown_connections 2024-03-28T19:03:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:06.621+0000 7f0921600640 1 -- 172.21.15.27:0/2488199697 wait complete. 2024-03-28T19:03:06.640 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 67, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 67, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5862": { "addr": "172.21.15.27:6837/1459475560", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 1459475560, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 1459475560, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5862, "incarnation": 67, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:03:04.718543+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5862 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5769": { "addr": "172.21.15.27:6836/1001046540", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1001046540, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 1001046540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5769, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:02:41.197840+0000", "qdb_cluster": [ 5769 ], "qdb_leader": 5769, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5769 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:03:08.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:08.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.871+0000 7faedea00640 1 Processor -- start 2024-03-28T19:03:08.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.871+0000 7faedea00640 1 -- start start 2024-03-28T19:03:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.872+0000 7faedea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faed8156320 0x7faed80c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.872+0000 7faedea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806ff90 0x7faed8158c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.872+0000 7faedea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed80c3df0 con 0x7faed806c3c0 2024-03-28T19:03:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.872+0000 7faedea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed8156700 con 0x7faed8156320 2024-03-28T19:03:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.872+0000 7faedea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed80c4120 con 0x7faed806ff90 2024-03-28T19:03:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faedd600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faed8156320 0x7faed80c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faede000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faed806c3c0 0x7faed80728e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43786/0 (socket says 172.21.15.27:43786) 2024-03-28T19:03:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faede000640 1 -- 172.21.15.27:0/4059112293 learned_addr learned my addr 172.21.15.27:0/4059112293 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faedcc00640 1 --2- 172.21.15.27:0/4059112293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806ff90 0x7faed8158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faedd600640 1 -- 172.21.15.27:0/4059112293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806ff90 msgr2=0x7faed8158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faedd600640 1 --2- 172.21.15.27:0/4059112293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806ff90 0x7faed8158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faedd600640 1 -- 172.21.15.27:0/4059112293 >> v1:172.21.15.27:6789/0 conn(0x7faed806c3c0 legacy=0x7faed80728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:08.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faedd600640 1 -- 172.21.15.27:0/4059112293 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed80c1470 con 0x7faed8156320 2024-03-28T19:03:08.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.873+0000 7faed7e00640 1 -- 172.21.15.27:0/4059112293 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439322901 0 0) 0x7faed80c3df0 con 0x7faed806c3c0 2024-03-28T19:03:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.875+0000 7faedd600640 1 --2- 172.21.15.27:0/4059112293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed80c0f30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7faecc0097a0 tx=0x7faecc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602ae4993f620a1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faed7e00640 1 -- 172.21.15.27:0/4059112293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc031780 con 0x7faed8156320 2024-03-28T19:03:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faed7e00640 1 -- 172.21.15.27:0/4059112293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc037070 con 0x7faed8156320 2024-03-28T19:03:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faed7e00640 1 -- 172.21.15.27:0/4059112293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc031b40 con 0x7faed8156320 2024-03-28T19:03:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faedea00640 1 -- 172.21.15.27:0/4059112293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 msgr2=0x7faed80c0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faedea00640 1 --2- 172.21.15.27:0/4059112293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed80c0f30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7faecc0097a0 tx=0x7faecc02b820 comp rx=0 tx=0).stop 2024-03-28T19:03:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faedea00640 1 -- 172.21.15.27:0/4059112293 shutdown_connections 2024-03-28T19:03:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faedea00640 1 --2- 172.21.15.27:0/4059112293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806ff90 0x7faed8158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:08.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faedea00640 1 --2- 172.21.15.27:0/4059112293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed80c0f30 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:08.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faedea00640 1 -- 172.21.15.27:0/4059112293 >> 172.21.15.27:0/4059112293 conn(0x7faed806b0c0 msgr2=0x7faed806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:08.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faedea00640 1 -- 172.21.15.27:0/4059112293 shutdown_connections 2024-03-28T19:03:08.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.876+0000 7faedea00640 1 -- 172.21.15.27:0/4059112293 wait complete. 2024-03-28T19:03:08.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.878+0000 7faedea00640 1 Processor -- start 2024-03-28T19:03:08.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.879+0000 7faedea00640 1 -- start start 2024-03-28T19:03:08.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faedea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806c3c0 0x7faed814d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faedea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed806ff90 0x7faed80a8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faedea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed8154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faedea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed80a6d00 con 0x7faed8156320 2024-03-28T19:03:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faedea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed80c3850 con 0x7faed806ff90 2024-03-28T19:03:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faedea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed80c7750 con 0x7faed806c3c0 2024-03-28T19:03:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faede000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed8154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faede000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed8154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33202/0 (socket says 172.21.15.27:33202) 2024-03-28T19:03:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faede000640 1 -- 172.21.15.27:0/1431140172 learned_addr learned my addr 172.21.15.27:0/1431140172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faedd600640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806c3c0 0x7faed814d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faedcc00640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed806ff90 0x7faed80a8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faede000640 1 -- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806c3c0 msgr2=0x7faed814d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faede000640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806c3c0 0x7faed814d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.880+0000 7faede000640 1 -- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed806ff90 msgr2=0x7faed80a8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faede000640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed806ff90 0x7faed80a8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faede000640 1 -- 172.21.15.27:0/1431140172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed8148ee0 con 0x7faed8156320 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faedcc00640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed806ff90 0x7faed80a8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faede000640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed8154830 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7faed000c700 tx=0x7faed000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faed5600640 1 -- 172.21.15.27:0/1431140172 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0018070 con 0x7faed8156320 2024-03-28T19:03:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faed5600640 1 -- 172.21.15.27:0/1431140172 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed000ce10 con 0x7faed8156320 2024-03-28T19:03:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faed5600640 1 -- 172.21.15.27:0/1431140172 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed000a880 con 0x7faed8156320 2024-03-28T19:03:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed8148a90 con 0x7faed8156320 2024-03-28T19:03:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.881+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed814f160 con 0x7faed8156320 2024-03-28T19:03:08.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.883+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed80a8ab0 con 0x7faed8156320 2024-03-28T19:03:08.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.884+0000 7faed5600640 1 -- 172.21.15.27:0/1431140172 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faed001b070 con 0x7faed8156320 2024-03-28T19:03:08.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.884+0000 7faed5600640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faeb806bbd0 0x7faeb806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:08.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.885+0000 7faed5600640 1 -- 172.21.15.27:0/1431140172 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7faed008a700 con 0x7faed8156320 2024-03-28T19:03:08.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.885+0000 7faedd600640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faeb806bbd0 0x7faeb806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:08.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.885+0000 7faedd600640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faeb806bbd0 0x7faeb806e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faecc036000 tx=0x7faecc03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:08.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:08.894+0000 7faed5600640 1 -- 172.21.15.27:0/1431140172 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faed0058af0 con 0x7faed8156320 2024-03-28T19:03:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.158+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faed8069640 con 0x7faed8156320 2024-03-28T19:03:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.159+0000 7faed5600640 1 -- 172.21.15.27:0/1431140172 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5457 (secure 0 0 0) 0x7faed00583f0 con 0x7faed8156320 2024-03-28T19:03:09.159 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:09.159 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:07.735554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 69 2024-03-28T19:03:09.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faeb806bbd0 msgr2=0x7faeb806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:09.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faeb806bbd0 0x7faeb806e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faecc036000 tx=0x7faecc03d040 comp rx=0 tx=0).stop 2024-03-28T19:03:09.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 msgr2=0x7faed8154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:09.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed8154830 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7faed000c700 tx=0x7faed000cbc0 comp rx=0 tx=0).stop 2024-03-28T19:03:09.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 shutdown_connections 2024-03-28T19:03:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faeb806bbd0 0x7faeb806e090 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed8156320 0x7faed8154830 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed806ff90 0x7faed80a8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 --2- 172.21.15.27:0/1431140172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faed806c3c0 0x7faed814d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.162+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 >> 172.21.15.27:0/1431140172 conn(0x7faed806b0c0 msgr2=0x7faed80717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.163+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 shutdown_connections 2024-03-28T19:03:09.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:09.163+0000 7faedea00640 1 -- 172.21.15.27:0/1431140172 wait complete. 2024-03-28T19:03:09.184 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:03:11.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.445+0000 7f297f200640 1 Processor -- start 2024-03-28T19:03:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.446+0000 7f297f200640 1 -- start start 2024-03-28T19:03:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.448+0000 7f297f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800be5d0 0x7f29800c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.448+0000 7f297f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2980158bd0 con 0x7f29800c58c0 2024-03-28T19:03:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.448+0000 7f297f200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2980158eb0 con 0x7f29800c8ed0 2024-03-28T19:03:11.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.448+0000 7f297f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2980159190 con 0x7f29800be5d0 2024-03-28T19:03:11.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.448+0000 7f297de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f29800c58c0 0x7f29801555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43792/0 (socket says 172.21.15.27:43792) 2024-03-28T19:03:11.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.448+0000 7f297de00640 1 -- 172.21.15.27:0/3023266140 learned_addr learned my addr 172.21.15.27:0/3023266140 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:11.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.448+0000 7f297e800640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800be5d0 0x7f29800c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197207608 0 0) 0x7f2980158bd0 con 0x7f29800c58c0 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2964003520 con 0x7f29800c58c0 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2294764057 0 0) 0x7f2964003520 con 0x7f29800c58c0 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800be5d0 msgr2=0x7f29800c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800be5d0 0x7f29800c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 >> v1:172.21.15.155:6789/0 conn(0x7f29800c8ed0 legacy=0x7f2980155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2980159470 con 0x7f29800c58c0 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f2970001670 con 0x7f29800c58c0 2024-03-28T19:03:11.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 >> v1:172.21.15.27:6789/0 conn(0x7f29800c58c0 legacy=0x7f29801555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800be5d0 0x7f29640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2964003fb0 0x7f2964004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2980159190 con 0x7f2964004d40 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296400bbb0 con 0x7f2964003fb0 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.449+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296400bdb0 con 0x7f29800be5d0 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.450+0000 7f297e800640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.450+0000 7f297de00640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800be5d0 0x7f29640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.450+0000 7f297d400640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2964003fb0 0x7f2964004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:11.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.450+0000 7f297e800640 1 -- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800be5d0 msgr2=0x7f29640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:11.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.450+0000 7f297e800640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800be5d0 0x7f29640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.450+0000 7f297e800640 1 -- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2964003fb0 msgr2=0x7f2964004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:11.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.450+0000 7f297e800640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2964003fb0 0x7f2964004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.450+0000 7f297e800640 1 -- 172.21.15.27:0/3023266140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2964003520 con 0x7f2964004d40 2024-03-28T19:03:11.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.452+0000 7f297e800640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29640051c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2978009fc0 tx=0x7f2964005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbe68e63cb9bf59 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:11.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.452+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978039070 con 0x7f2964004d40 2024-03-28T19:03:11.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.452+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297802cc20 con 0x7f2964004d40 2024-03-28T19:03:11.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.452+0000 7f297ca00640 1 -- 172.21.15.27:0/3023266140 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978009c10 con 0x7f2964004d40 2024-03-28T19:03:11.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.453+0000 7f297f200640 1 -- 172.21.15.27:0/3023266140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 msgr2=0x7f29640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:11.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.453+0000 7f297f200640 1 --2- 172.21.15.27:0/3023266140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29640051c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2978009fc0 tx=0x7f2964005830 comp rx=0 tx=0).stop 2024-03-28T19:03:11.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.453+0000 7f297e800640 1 -- 172.21.15.27:0/3023266140 reap_dead start 2024-03-28T19:03:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.453+0000 7f297f200640 1 -- 172.21.15.27:0/3023266140 shutdown_connections 2024-03-28T19:03:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.453+0000 7f297f200640 1 -- 172.21.15.27:0/3023266140 >> 172.21.15.27:0/3023266140 conn(0x7f298006b6f0 msgr2=0x7f29800acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.453+0000 7f297f200640 1 -- 172.21.15.27:0/3023266140 shutdown_connections 2024-03-28T19:03:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.453+0000 7f297f200640 1 -- 172.21.15.27:0/3023266140 wait complete. 2024-03-28T19:03:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.455+0000 7f297f200640 1 Processor -- start 2024-03-28T19:03:11.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.455+0000 7f297f200640 1 -- start start 2024-03-28T19:03:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.456+0000 7f297f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29800bef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29800c4560 0x7f29800c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800c1290 0x7f2980152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29800b09a0 con 0x7f2964004d40 2024-03-28T19:03:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29800b0820 con 0x7f29800c4560 2024-03-28T19:03:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29800ad7d0 con 0x7f29800c1290 2024-03-28T19:03:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29800bef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29800bef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33212/0 (socket says 172.21.15.27:33212) 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297de00640 1 -- 172.21.15.27:0/1136488013 learned_addr learned my addr 172.21.15.27:0/1136488013 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.457+0000 7f297d400640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29800c4560 0x7f29800c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.458+0000 7f297e800640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800c1290 0x7f2980152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.458+0000 7f297de00640 1 -- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800c1290 msgr2=0x7f2980152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.458+0000 7f297de00640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800c1290 0x7f2980152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.458+0000 7f297de00640 1 -- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29800c4560 msgr2=0x7f29800c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.458+0000 7f297de00640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29800c4560 0x7f29800c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.458+0000 7f297de00640 1 -- 172.21.15.27:0/1136488013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2980153db0 con 0x7f2964004d40 2024-03-28T19:03:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.458+0000 7f297d400640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29800c4560 0x7f29800c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:03:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.458+0000 7f297de00640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29800bef30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f29700072c0 tx=0x7f2970007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.459+0000 7f2976000640 1 -- 172.21.15.27:0/1136488013 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297000f960 con 0x7f2964004d40 2024-03-28T19:03:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.459+0000 7f2976000640 1 -- 172.21.15.27:0/1136488013 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970018070 con 0x7f2964004d40 2024-03-28T19:03:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.459+0000 7f2976000640 1 -- 172.21.15.27:0/1136488013 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297000fd60 con 0x7f2964004d40 2024-03-28T19:03:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.459+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29800c3870 con 0x7f2964004d40 2024-03-28T19:03:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.459+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29800c3590 con 0x7f2964004d40 2024-03-28T19:03:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.461+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2980069640 con 0x7f2964004d40 2024-03-28T19:03:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.462+0000 7f2976000640 1 -- 172.21.15.27:0/1136488013 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f297001d460 con 0x7f2964004d40 2024-03-28T19:03:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.463+0000 7f2976000640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f296006bbd0 0x7f296006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.464+0000 7f2976000640 1 -- 172.21.15.27:0/1136488013 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f297008b3a0 con 0x7f2964004d40 2024-03-28T19:03:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.464+0000 7f297d400640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f296006bbd0 0x7f296006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.465+0000 7f297d400640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f296006bbd0 0x7f296006e090 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f296c007fd0 tx=0x7f296c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.471+0000 7f2976000640 1 -- 172.21.15.27:0/1136488013 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2980153db0 con 0x7f2964004d40 2024-03-28T19:03:11.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.854+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2980156580 con 0x7f2964004d40 2024-03-28T19:03:11.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.855+0000 7f2976000640 1 -- 172.21.15.27:0/1136488013 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5465 (secure 0 0 0) 0x7f2970059780 con 0x7f2964004d40 2024-03-28T19:03:11.855 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:11.855 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:11.855 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 70 2024-03-28T19:03:11.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f296006bbd0 msgr2=0x7f296006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:11.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f296006bbd0 0x7f296006e090 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f296c007fd0 tx=0x7f296c009040 comp rx=0 tx=0).stop 2024-03-28T19:03:11.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 msgr2=0x7f29800bef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:11.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29800bef30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f29700072c0 tx=0x7f2970007780 comp rx=0 tx=0).stop 2024-03-28T19:03:11.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 shutdown_connections 2024-03-28T19:03:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f296006bbd0 0x7f296006e090 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29800c1290 0x7f2980152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29800c4560 0x7f29800c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 --2- 172.21.15.27:0/1136488013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2964004d40 0x7f29800bef30 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 >> 172.21.15.27:0/1136488013 conn(0x7f298006b6f0 msgr2=0x7f29800c9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.858+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 shutdown_connections 2024-03-28T19:03:11.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:11.859+0000 7f297f200640 1 -- 172.21.15.27:0/1136488013 wait complete. 2024-03-28T19:03:11.880 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4247 2024-03-28T19:03:11.880 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.b 2024-03-28T19:03:13.227 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:03:13.227+0000 7f5089200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:03:16.880 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-03-28T19:03:16.880 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:03:16.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:03:16.889 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:03:16.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:17.068 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:03:17.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.143+0000 7f53e9200640 1 Processor -- start 2024-03-28T19:03:17.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.144+0000 7f53e9200640 1 -- start start 2024-03-28T19:03:17.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.145+0000 7f53e9200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f53e40c69a0 0x7f53e40c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:17.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.145+0000 7f53e9200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e4150ec0 con 0x7f53e4153a30 2024-03-28T19:03:17.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.145+0000 7f53e9200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e406dc70 con 0x7f53e4150ae0 2024-03-28T19:03:17.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.145+0000 7f53e9200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e4153550 con 0x7f53e40c69a0 2024-03-28T19:03:17.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.146+0000 7f53e3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f53e4153a30 0x7f53e415dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33454/0 (socket says 172.21.15.27:33454) 2024-03-28T19:03:17.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.146+0000 7f53e3e00640 1 -- 172.21.15.27:0/3049557222 learned_addr learned my addr 172.21.15.27:0/3049557222 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:17.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.146+0000 7f53e2a00640 1 --2- 172.21.15.27:0/3049557222 >> v2:172.21.15.27:3300/0 conn(0x7f53e40c69a0 0x7f53e40c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:17.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.146+0000 7f53e2a00640 1 -- 172.21.15.27:0/3049557222 >> v1:172.21.15.155:6789/0 conn(0x7f53e4150ae0 legacy=0x7f53e40a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:17.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.146+0000 7f53e2a00640 1 -- 172.21.15.27:0/3049557222 >> v1:172.21.15.27:6789/0 conn(0x7f53e4153a30 legacy=0x7f53e415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:17.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.147+0000 7f53e2a00640 1 -- 172.21.15.27:0/3049557222 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e406df10 con 0x7f53e40c69a0 2024-03-28T19:03:17.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.147+0000 7f53e2000640 1 -- 172.21.15.27:0/3049557222 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766899287 0 0) 0x7f53dc001490 con 0x7f53e4153a30 2024-03-28T19:03:17.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.148+0000 7f53e2a00640 1 --2- 172.21.15.27:0/3049557222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e40c69a0 0x7f53e40c6e30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f53d00097a0 tx=0x7f53d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ecd2b1c647bea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.149+0000 7f53e2000640 1 -- 172.21.15.27:0/3049557222 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0035020 con 0x7f53e40c69a0 2024-03-28T19:03:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.149+0000 7f53e2000640 1 -- 172.21.15.27:0/3049557222 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d0031830 con 0x7f53e40c69a0 2024-03-28T19:03:17.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.149+0000 7f53e2000640 1 -- 172.21.15.27:0/3049557222 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0031c00 con 0x7f53e40c69a0 2024-03-28T19:03:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.150+0000 7f53e9200640 1 -- 172.21.15.27:0/3049557222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e40c69a0 msgr2=0x7f53e40c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.150+0000 7f53e9200640 1 --2- 172.21.15.27:0/3049557222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e40c69a0 0x7f53e40c6e30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f53d00097a0 tx=0x7f53d002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:03:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.150+0000 7f53e9200640 1 -- 172.21.15.27:0/3049557222 shutdown_connections 2024-03-28T19:03:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.150+0000 7f53e9200640 1 --2- 172.21.15.27:0/3049557222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e40c69a0 0x7f53e40c6e30 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.150+0000 7f53e9200640 1 -- 172.21.15.27:0/3049557222 >> 172.21.15.27:0/3049557222 conn(0x7f53e406b0c0 msgr2=0x7f53e406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.150+0000 7f53e9200640 1 -- 172.21.15.27:0/3049557222 shutdown_connections 2024-03-28T19:03:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.150+0000 7f53e9200640 1 -- 172.21.15.27:0/3049557222 wait complete. 2024-03-28T19:03:17.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.152+0000 7f53e9200640 1 Processor -- start 2024-03-28T19:03:17.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.152+0000 7f53e9200640 1 -- start start 2024-03-28T19:03:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.153+0000 7f53e9200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53e40c69a0 0x7f53e41eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.153+0000 7f53e9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53e4150ae0 0x7f53e40be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.153+0000 7f53e9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e4153a30 0x7f53e4142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.153+0000 7f53e9200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e406e7b0 con 0x7f53e4153a30 2024-03-28T19:03:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.154+0000 7f53e9200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e406e4d0 con 0x7f53e4150ae0 2024-03-28T19:03:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.154+0000 7f53e9200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e40a6370 con 0x7f53e40c69a0 2024-03-28T19:03:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.154+0000 7f53e3400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53e40c69a0 0x7f53e41eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.154+0000 7f53e2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53e4150ae0 0x7f53e40be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.154+0000 7f53e3400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53e40c69a0 0x7f53e41eedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:43906/0 (socket says 172.21.15.27:43906) 2024-03-28T19:03:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.154+0000 7f53e3400640 1 -- 172.21.15.27:0/3705369764 learned_addr learned my addr 172.21.15.27:0/3705369764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.154+0000 7f53e3e00640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e4153a30 0x7f53e4142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.155+0000 7f53e3e00640 1 -- 172.21.15.27:0/3705369764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53e40c69a0 msgr2=0x7f53e41eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.155+0000 7f53e3e00640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53e40c69a0 0x7f53e41eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.155+0000 7f53e3e00640 1 -- 172.21.15.27:0/3705369764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53e4150ae0 msgr2=0x7f53e40be970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.155+0000 7f53e3e00640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53e4150ae0 0x7f53e40be970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.155+0000 7f53e3e00640 1 -- 172.21.15.27:0/3705369764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e41ef7a0 con 0x7f53e4153a30 2024-03-28T19:03:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.155+0000 7f53e3e00640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e4153a30 0x7f53e4142ea0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00c750 tx=0x7f53dc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:17.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.156+0000 7f53db400640 1 -- 172.21.15.27:0/3705369764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc018070 con 0x7f53e4153a30 2024-03-28T19:03:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.156+0000 7f53db400640 1 -- 172.21.15.27:0/3705369764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53dc004d60 con 0x7f53e4153a30 2024-03-28T19:03:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.156+0000 7f53db400640 1 -- 172.21.15.27:0/3705369764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc00a880 con 0x7f53e4153a30 2024-03-28T19:03:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.156+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e40bd4a0 con 0x7f53e4153a30 2024-03-28T19:03:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.156+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e41520e0 con 0x7f53e4153a30 2024-03-28T19:03:17.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.158+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e4067dd0 con 0x7f53e4153a30 2024-03-28T19:03:17.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.161+0000 7f53db400640 1 -- 172.21.15.27:0/3705369764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f53dc01b070 con 0x7f53e4153a30 2024-03-28T19:03:17.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.162+0000 7f53db400640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53c406bc20 0x7f53c406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:17.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.162+0000 7f53db400640 1 -- 172.21.15.27:0/3705369764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f53dc08a5d0 con 0x7f53e4153a30 2024-03-28T19:03:17.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.162+0000 7f53e3400640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53c406bc20 0x7f53c406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:17.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.164+0000 7f53e3400640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53c406bc20 0x7f53c406e0e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f53e415f030 tx=0x7f53d4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:17.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.169+0000 7f53db400640 1 -- 172.21.15.27:0/3705369764 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f53e4150ec0 con 0x7f53e4153a30 2024-03-28T19:03:17.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.545+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53e4069640 con 0x7f53e4153a30 2024-03-28T19:03:17.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.546+0000 7f53db400640 1 -- 172.21.15.27:0/3705369764 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f53dc0584e0 con 0x7f53e4153a30 2024-03-28T19:03:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:17.547 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:17.547 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:17.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53c406bc20 msgr2=0x7f53c406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:17.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53c406bc20 0x7f53c406e0e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f53e415f030 tx=0x7f53d4009040 comp rx=0 tx=0).stop 2024-03-28T19:03:17.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e4153a30 msgr2=0x7f53e4142ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:17.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e4153a30 0x7f53e4142ea0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00c750 tx=0x7f53dc00cc10 comp rx=0 tx=0).stop 2024-03-28T19:03:17.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 shutdown_connections 2024-03-28T19:03:17.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53c406bc20 0x7f53c406e0e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:17.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53e4153a30 0x7f53e4142ea0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:17.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53e4150ae0 0x7f53e40be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:17.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 --2- 172.21.15.27:0/3705369764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53e40c69a0 0x7f53e41eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.548+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 >> 172.21.15.27:0/3705369764 conn(0x7f53e406b0c0 msgr2=0x7f53e4152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.549+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 shutdown_connections 2024-03-28T19:03:17.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:17.549+0000 7f53e9200640 1 -- 172.21.15.27:0/3705369764 wait complete. 2024-03-28T19:03:17.568 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-03-28T19:03:17.568 INFO:tasks.mds_thrash.fs.[a]:waiting for 20 secs before thrashing 2024-03-28T19:03:29.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:03:29.598 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:03:37.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.824+0000 7f4c33e00640 1 Processor -- start 2024-03-28T19:03:37.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.824+0000 7f4c33e00640 1 -- start start 2024-03-28T19:03:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.825+0000 7f4c33e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4c2c150c70 0x7f4c2c0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.825+0000 7f4c33e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0c3af0 con 0x7f4c2c0c3690 2024-03-28T19:03:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.825+0000 7f4c33e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c154020 con 0x7f4c2c1534c0 2024-03-28T19:03:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.825+0000 7f4c33e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c0ab4e0 con 0x7f4c2c150c70 2024-03-28T19:03:37.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.826+0000 7f4c31400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4c2c150c70 0x7f4c2c0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.826+0000 7f4c31e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4c2c1534c0 0x7f4c2c153910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:41308/0 (socket says 172.21.15.27:41308) 2024-03-28T19:03:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.826+0000 7f4c31400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4c2c150c70 0x7f4c2c0a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:45702/0 (socket says 172.21.15.27:45702) 2024-03-28T19:03:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.826+0000 7f4c31e00640 1 -- 172.21.15.27:0/3933979941 learned_addr learned my addr 172.21.15.27:0/3933979941 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.827+0000 7f4c2be00640 1 -- 172.21.15.27:0/3933979941 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446375750 0 0) 0x7f4c2c0c3af0 con 0x7f4c2c0c3690 2024-03-28T19:03:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.827+0000 7f4c2be00640 1 -- 172.21.15.27:0/3933979941 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c14003520 con 0x7f4c2c0c3690 2024-03-28T19:03:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.827+0000 7f4c31400640 1 -- 172.21.15.27:0/3933979941 >> v1:172.21.15.155:6789/0 conn(0x7f4c2c1534c0 legacy=0x7f4c2c153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.827+0000 7f4c2be00640 1 -- 172.21.15.27:0/3933979941 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789246977 0 0) 0x7f4c2c154020 con 0x7f4c2c1534c0 2024-03-28T19:03:37.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.827+0000 7f4c31400640 1 -- 172.21.15.27:0/3933979941 >> v1:172.21.15.27:6789/0 conn(0x7f4c2c0c3690 legacy=0x7f4c2c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:37.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.827+0000 7f4c31400640 1 -- 172.21.15.27:0/3933979941 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0c3d90 con 0x7f4c2c150c70 2024-03-28T19:03:37.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.829+0000 7f4c31400640 1 --2- 172.21.15.27:0/3933979941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 0x7f4c2c0a9cb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c1541f0 tx=0x7f4c2002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d43417a3ff6000 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.829+0000 7f4c2be00640 1 -- 172.21.15.27:0/3933979941 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20035020 con 0x7f4c2c150c70 2024-03-28T19:03:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.829+0000 7f4c2be00640 1 -- 172.21.15.27:0/3933979941 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c20031830 con 0x7f4c2c150c70 2024-03-28T19:03:37.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.830+0000 7f4c2be00640 1 -- 172.21.15.27:0/3933979941 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20031c00 con 0x7f4c2c150c70 2024-03-28T19:03:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.830+0000 7f4c33e00640 1 -- 172.21.15.27:0/3933979941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 msgr2=0x7f4c2c0a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.830+0000 7f4c33e00640 1 --2- 172.21.15.27:0/3933979941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 0x7f4c2c0a9cb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c1541f0 tx=0x7f4c2002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:03:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.830+0000 7f4c33e00640 1 -- 172.21.15.27:0/3933979941 shutdown_connections 2024-03-28T19:03:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.830+0000 7f4c33e00640 1 --2- 172.21.15.27:0/3933979941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 0x7f4c2c0a9cb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.830+0000 7f4c33e00640 1 -- 172.21.15.27:0/3933979941 >> 172.21.15.27:0/3933979941 conn(0x7f4c2c06b9d0 msgr2=0x7f4c2c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.831+0000 7f4c33e00640 1 -- 172.21.15.27:0/3933979941 shutdown_connections 2024-03-28T19:03:37.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.831+0000 7f4c33e00640 1 -- 172.21.15.27:0/3933979941 wait complete. 2024-03-28T19:03:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.833+0000 7f4c33e00640 1 Processor -- start 2024-03-28T19:03:37.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.833+0000 7f4c33e00640 1 -- start start 2024-03-28T19:03:37.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.834+0000 7f4c33e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c2c0c3690 0x7f4c2c14e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:37.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.834+0000 7f4c33e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 0x7f4c2c0713a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:37.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.834+0000 7f4c33e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c2c1534c0 0x7f4c2c0718e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:37.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.834+0000 7f4c33e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0ab9f0 con 0x7f4c2c1534c0 2024-03-28T19:03:37.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.834+0000 7f4c33e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0ac1f0 con 0x7f4c2c150c70 2024-03-28T19:03:37.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.834+0000 7f4c33e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0abcd0 con 0x7f4c2c0c3690 2024-03-28T19:03:37.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c2c1534c0 0x7f4c2c0718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:37.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c2c1534c0 0x7f4c2c0718e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60086/0 (socket says 172.21.15.27:60086) 2024-03-28T19:03:37.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31e00640 1 -- 172.21.15.27:0/1205354318 learned_addr learned my addr 172.21.15.27:0/1205354318 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:37.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c30a00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 0x7f4c2c0713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:37.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31400640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c2c0c3690 0x7f4c2c14e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:37.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31e00640 1 -- 172.21.15.27:0/1205354318 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c2c0c3690 msgr2=0x7f4c2c14e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c2c0c3690 0x7f4c2c14e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31e00640 1 -- 172.21.15.27:0/1205354318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 msgr2=0x7f4c2c0713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 0x7f4c2c0713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.835+0000 7f4c31e00640 1 -- 172.21.15.27:0/1205354318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0a7b00 con 0x7f4c2c1534c0 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.836+0000 7f4c31e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c2c1534c0 0x7f4c2c0718e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400c730 tx=0x7f4c2400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.836+0000 7f4c29600640 1 -- 172.21.15.27:0/1205354318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24018070 con 0x7f4c2c1534c0 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.836+0000 7f4c29600640 1 -- 172.21.15.27:0/1205354318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2400ce40 con 0x7f4c2c1534c0 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.836+0000 7f4c29600640 1 -- 172.21.15.27:0/1205354318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2400a880 con 0x7f4c2c1534c0 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.836+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c0a7de0 con 0x7f4c2c1534c0 2024-03-28T19:03:37.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.836+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c14e990 con 0x7f4c2c1534c0 2024-03-28T19:03:37.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.838+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c04f400 con 0x7f4c2c1534c0 2024-03-28T19:03:37.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.840+0000 7f4c29600640 1 -- 172.21.15.27:0/1205354318 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4c2401b070 con 0x7f4c2c1534c0 2024-03-28T19:03:37.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.840+0000 7f4c29600640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c0c06bc50 0x7f4c0c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:37.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.840+0000 7f4c29600640 1 -- 172.21.15.27:0/1205354318 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f4c2408ac90 con 0x7f4c2c1534c0 2024-03-28T19:03:37.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.841+0000 7f4c31400640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c0c06bc50 0x7f4c0c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:37.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.842+0000 7f4c31400640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c0c06bc50 0x7f4c0c06e110 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4c20036000 tx=0x7f4c20008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:37.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:37.849+0000 7f4c29600640 1 -- 172.21.15.27:0/1205354318 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4c24022020 con 0x7f4c2c1534c0 2024-03-28T19:03:38.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.213+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c2c0c66c0 con 0x7f4c2c1534c0 2024-03-28T19:03:38.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.214+0000 7f4c29600640 1 -- 172.21.15.27:0/1205354318 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f4c240590f0 con 0x7f4c2c1534c0 2024-03-28T19:03:38.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:38.214 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:38.214 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:38.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.217+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c0c06bc50 msgr2=0x7f4c0c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:38.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.217+0000 7f4c33e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c0c06bc50 0x7f4c0c06e110 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4c20036000 tx=0x7f4c20008030 comp rx=0 tx=0).stop 2024-03-28T19:03:38.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.217+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c2c1534c0 msgr2=0x7f4c2c0718e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:38.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.217+0000 7f4c33e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c2c1534c0 0x7f4c2c0718e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400c730 tx=0x7f4c2400cbf0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.218+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 shutdown_connections 2024-03-28T19:03:38.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.218+0000 7f4c33e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c0c06bc50 0x7f4c0c06e110 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.218+0000 7f4c33e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c2c1534c0 0x7f4c2c0718e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.218+0000 7f4c33e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c2c150c70 0x7f4c2c0713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.218+0000 7f4c33e00640 1 --2- 172.21.15.27:0/1205354318 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c2c0c3690 0x7f4c2c14e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.218+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 >> 172.21.15.27:0/1205354318 conn(0x7f4c2c06b9d0 msgr2=0x7f4c2c151840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:38.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.218+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 shutdown_connections 2024-03-28T19:03:38.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.218+0000 7f4c33e00640 1 -- 172.21.15.27:0/1205354318 wait complete. 2024-03-28T19:03:38.239 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-03-28T19:03:38.239 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:03:38.239 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:03:38.260 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:03:38.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.515+0000 7fa7d7e00640 1 Processor -- start 2024-03-28T19:03:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.515+0000 7fa7d7e00640 1 -- start start 2024-03-28T19:03:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.516+0000 7fa7d7e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa7d80c58c0 0x7fa7d80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.516+0000 7fa7d7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7d80be5d0 0x7fa7d8155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.516+0000 7fa7d7e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d8156230 con 0x7fa7d80c8ed0 2024-03-28T19:03:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.516+0000 7fa7d7e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d8158bd0 con 0x7fa7d80be5d0 2024-03-28T19:03:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.516+0000 7fa7d7e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d8158eb0 con 0x7fa7d80c58c0 2024-03-28T19:03:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.516+0000 7fa7d6000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa7d80c8ed0 0x7fa7d81555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56094/0 (socket says 172.21.15.27:56094) 2024-03-28T19:03:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.516+0000 7fa7d7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7d80be5d0 0x7fa7d8155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.516+0000 7fa7d6000640 1 -- 172.21.15.27:0/3519103325 learned_addr learned my addr 172.21.15.27:0/3519103325 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.517+0000 7fa7d6a00640 1 --2- 172.21.15.27:0/3519103325 >> v2:172.21.15.155:3300/0 conn(0x7fa7d80c58c0 0x7fa7d80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.517+0000 7fa7d7400640 1 -- 172.21.15.27:0/3519103325 >> v2:172.21.15.155:3300/0 conn(0x7fa7d80c58c0 msgr2=0x7fa7d80c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.517+0000 7fa7d7400640 1 --2- 172.21.15.27:0/3519103325 >> v2:172.21.15.155:3300/0 conn(0x7fa7d80c58c0 0x7fa7d80c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.517+0000 7fa7d7400640 1 -- 172.21.15.27:0/3519103325 >> v1:172.21.15.27:6789/0 conn(0x7fa7d80c8ed0 legacy=0x7fa7d81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.517+0000 7fa7d7400640 1 -- 172.21.15.27:0/3519103325 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8159190 con 0x7fa7d80be5d0 2024-03-28T19:03:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.517+0000 7fa7d5600640 1 -- 172.21.15.27:0/3519103325 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127141646 0 0) 0x7fa7d8156230 con 0x7fa7d80c8ed0 2024-03-28T19:03:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.518+0000 7fa7d7400640 1 --2- 172.21.15.27:0/3519103325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d8155cf0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa7d806c060 tx=0x7fa7d002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2489e866a1ca49e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.518+0000 7fa7d5600640 1 -- 172.21.15.27:0/3519103325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0039070 con 0x7fa7d80be5d0 2024-03-28T19:03:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.518+0000 7fa7d5600640 1 -- 172.21.15.27:0/3519103325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d002cb60 con 0x7fa7d80be5d0 2024-03-28T19:03:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.518+0000 7fa7d5600640 1 -- 172.21.15.27:0/3519103325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0034440 con 0x7fa7d80be5d0 2024-03-28T19:03:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.519+0000 7fa7d7e00640 1 -- 172.21.15.27:0/3519103325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 msgr2=0x7fa7d8155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.519+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/3519103325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d8155cf0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa7d806c060 tx=0x7fa7d002bdb0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.519+0000 7fa7d7e00640 1 -- 172.21.15.27:0/3519103325 shutdown_connections 2024-03-28T19:03:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.519+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/3519103325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d8155cf0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.519+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/3519103325 >> v2:172.21.15.155:3300/0 conn(0x7fa7d80c58c0 0x7fa7d80c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.519+0000 7fa7d7e00640 1 -- 172.21.15.27:0/3519103325 >> 172.21.15.27:0/3519103325 conn(0x7fa7d806b6f0 msgr2=0x7fa7d80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.519+0000 7fa7d7e00640 1 -- 172.21.15.27:0/3519103325 shutdown_connections 2024-03-28T19:03:38.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.519+0000 7fa7d7e00640 1 -- 172.21.15.27:0/3519103325 wait complete. 2024-03-28T19:03:38.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.521+0000 7fa7d7e00640 1 Processor -- start 2024-03-28T19:03:38.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.521+0000 7fa7d7e00640 1 -- start start 2024-03-28T19:03:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d80718c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7d80c58c0 0x7fa7d80ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7d80c8ed0 0x7fa7d81eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d80b0820 con 0x7fa7d80be5d0 2024-03-28T19:03:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d8159750 con 0x7fa7d80c8ed0 2024-03-28T19:03:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d80b09a0 con 0x7fa7d80c58c0 2024-03-28T19:03:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d80718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d80718c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60104/0 (socket says 172.21.15.27:60104) 2024-03-28T19:03:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d6a00640 1 -- 172.21.15.27:0/886007053 learned_addr learned my addr 172.21.15.27:0/886007053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.522+0000 7fa7d7400640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7d80c8ed0 0x7fa7d81eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d6a00640 1 -- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7d80c58c0 msgr2=0x7fa7d80ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d6000640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7d80c58c0 0x7fa7d80ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d6a00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7d80c58c0 0x7fa7d80ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d6a00640 1 -- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7d80c8ed0 msgr2=0x7fa7d81eed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d6a00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7d80c8ed0 0x7fa7d81eed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d6a00640 1 -- 172.21.15.27:0/886007053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d80c41c0 con 0x7fa7d80be5d0 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d7400640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7d80c8ed0 0x7fa7d81eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d6000640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7d80c58c0 0x7fa7d80ac930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d6a00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d80718c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800d5f0 tx=0x7fa7c800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7cea00640 1 -- 172.21.15.27:0/886007053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80125f0 con 0x7fa7d80be5d0 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7cea00640 1 -- 172.21.15.27:0/886007053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c8019070 con 0x7fa7d80be5d0 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d80c4430 con 0x7fa7d80be5d0 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.523+0000 7fa7cea00640 1 -- 172.21.15.27:0/886007053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80129d0 con 0x7fa7d80be5d0 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.524+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d80c49f0 con 0x7fa7d80be5d0 2024-03-28T19:03:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.525+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d8069640 con 0x7fa7d80be5d0 2024-03-28T19:03:38.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.526+0000 7fa7cea00640 1 -- 172.21.15.27:0/886007053 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa7c8012b70 con 0x7fa7d80be5d0 2024-03-28T19:03:38.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.527+0000 7fa7cea00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7b806bbd0 0x7fa7b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:38.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.528+0000 7fa7cea00640 1 -- 172.21.15.27:0/886007053 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fa7c808b630 con 0x7fa7d80be5d0 2024-03-28T19:03:38.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.528+0000 7fa7d6000640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7b806bbd0 0x7fa7b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:38.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.529+0000 7fa7d6000640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7b806bbd0 0x7fa7b806e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80afbc0 tx=0x7fa7c4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:38.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.531+0000 7fa7cea00640 1 -- 172.21.15.27:0/886007053 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa7c8059980 con 0x7fa7d80be5d0 2024-03-28T19:03:38.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.776+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7d8156580 con 0x7fa7d80be5d0 2024-03-28T19:03:38.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.777+0000 7fa7cea00640 1 -- 172.21.15.27:0/886007053 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7fa7c8059280 con 0x7fa7d80be5d0 2024-03-28T19:03:38.776 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:38.777 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:38.777 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:38.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.779+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7b806bbd0 msgr2=0x7fa7b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:38.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.779+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7b806bbd0 0x7fa7b806e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80afbc0 tx=0x7fa7c4009040 comp rx=0 tx=0).stop 2024-03-28T19:03:38.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.779+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 msgr2=0x7fa7d80718c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.779+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d80718c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800d5f0 tx=0x7fa7c800dab0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.780+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 shutdown_connections 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.780+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7b806bbd0 0x7fa7b806e090 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.780+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7d80c8ed0 0x7fa7d81eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.780+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7d80c58c0 0x7fa7d80ac930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.780+0000 7fa7d7e00640 1 --2- 172.21.15.27:0/886007053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7d80be5d0 0x7fa7d80718c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.780+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 >> 172.21.15.27:0/886007053 conn(0x7fa7d806b6f0 msgr2=0x7fa7d8157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.780+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 shutdown_connections 2024-03-28T19:03:38.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:38.780+0000 7fa7d7e00640 1 -- 172.21.15.27:0/886007053 wait complete. 2024-03-28T19:03:38.803 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:03:40.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.032+0000 7f35ed000640 1 Processor -- start 2024-03-28T19:03:41.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.032+0000 7f35ed000640 1 -- start start 2024-03-28T19:03:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.033+0000 7f35ed000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e814f6d0 0x7f35e8159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.034+0000 7f35ed000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e8072870 con 0x7f35e8071cf0 2024-03-28T19:03:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.034+0000 7f35ed000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e8156640 con 0x7f35e8072c60 2024-03-28T19:03:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.034+0000 7f35ed000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e8158db0 con 0x7f35e814f6d0 2024-03-28T19:03:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.034+0000 7f35e5400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f35e8071cf0 0x7f35e80c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56098/0 (socket says 172.21.15.27:56098) 2024-03-28T19:03:41.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.034+0000 7f35e5400640 1 -- 172.21.15.27:0/1369054809 learned_addr learned my addr 172.21.15.27:0/1369054809 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:41.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.034+0000 7f35e5e00640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e814f6d0 0x7f35e8159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:41.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248461664 0 0) 0x7f35d4001110 con 0x7f35e8071cf0 2024-03-28T19:03:41.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc003520 con 0x7f35e8071cf0 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1900922816 0 0) 0x7f35cc003520 con 0x7f35e8071cf0 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e814f6d0 msgr2=0x7f35e8159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e814f6d0 0x7f35e8159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 >> v1:172.21.15.155:6789/0 conn(0x7f35e8072c60 legacy=0x7f35e80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8156360 con 0x7f35e8071cf0 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f35d4001840 con 0x7f35e8071cf0 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 >> v1:172.21.15.27:6789/0 conn(0x7f35e8071cf0 legacy=0x7f35e80c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e814f6d0 0x7f35cc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35cc003f60 0x7f35cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35cc004d40 0x7f35cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e8158db0 con 0x7f35cc004d40 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc00bb00 con 0x7f35cc003f60 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.035+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc00bd00 con 0x7f35e814f6d0 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e6800640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35cc004d40 0x7f35cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e5e00640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e814f6d0 0x7f35cc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e5400640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35cc003f60 0x7f35cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e6800640 1 -- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e814f6d0 msgr2=0x7f35cc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e6800640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e814f6d0 0x7f35cc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e6800640 1 -- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35cc003f60 msgr2=0x7f35cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e6800640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35cc003f60 0x7f35cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e6800640 1 -- 172.21.15.27:0/1369054809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8156360 con 0x7f35cc004d40 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.036+0000 7f35e5400640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35cc003f60 0x7f35cc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.037+0000 7f35e6800640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35cc004d40 0x7f35cc0051c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f35e0001e50 tx=0x7f35cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600532246a2a9524 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.038+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e0001a70 con 0x7f35cc004d40 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.038+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0003b70 con 0x7f35cc004d40 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.038+0000 7f35e4a00640 1 -- 172.21.15.27:0/1369054809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e00050d0 con 0x7f35cc004d40 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.038+0000 7f35ed000640 1 -- 172.21.15.27:0/1369054809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35cc004d40 msgr2=0x7f35cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.038+0000 7f35ed000640 1 --2- 172.21.15.27:0/1369054809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35cc004d40 0x7f35cc0051c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f35e0001e50 tx=0x7f35cc005830 comp rx=0 tx=0).stop 2024-03-28T19:03:41.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.038+0000 7f35e6800640 1 -- 172.21.15.27:0/1369054809 reap_dead start 2024-03-28T19:03:41.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.038+0000 7f35ed000640 1 -- 172.21.15.27:0/1369054809 shutdown_connections 2024-03-28T19:03:41.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.038+0000 7f35ed000640 1 -- 172.21.15.27:0/1369054809 >> 172.21.15.27:0/1369054809 conn(0x7f35e806c5f0 msgr2=0x7f35e806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:41.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.039+0000 7f35ed000640 1 -- 172.21.15.27:0/1369054809 shutdown_connections 2024-03-28T19:03:41.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.039+0000 7f35ed000640 1 -- 172.21.15.27:0/1369054809 wait complete. 2024-03-28T19:03:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.040+0000 7f35ed000640 1 Processor -- start 2024-03-28T19:03:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.041+0000 7f35ed000640 1 -- start start 2024-03-28T19:03:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.041+0000 7f35ed000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e80aa110 0x7f35e81eb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.041+0000 7f35ed000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8154d10 0x7f35e81eb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.041+0000 7f35ed000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35e81ebec0 0x7f35e814c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:41.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35ed000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e814fab0 con 0x7f35e8154d10 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35ed000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e8159330 con 0x7f35e81ebec0 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35ed000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e80c3c70 con 0x7f35e80aa110 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8154d10 0x7f35e81eb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8154d10 0x7f35e81eb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60120/0 (socket says 172.21.15.27:60120) 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5400640 1 -- 172.21.15.27:0/3223483460 learned_addr learned my addr 172.21.15.27:0/3223483460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5e00640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e80aa110 0x7f35e81eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e6800640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35e81ebec0 0x7f35e814c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5400640 1 -- 172.21.15.27:0/3223483460 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e80aa110 msgr2=0x7f35e81eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:41.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5400640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e80aa110 0x7f35e81eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5400640 1 -- 172.21.15.27:0/3223483460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35e81ebec0 msgr2=0x7f35e814c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5400640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35e81ebec0 0x7f35e814c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.042+0000 7f35e5400640 1 -- 172.21.15.27:0/3223483460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e81ee990 con 0x7f35e8154d10 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.043+0000 7f35e5400640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8154d10 0x7f35e81eb980 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f35d400dbc0 tx=0x7f35d400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.043+0000 7f35de000640 1 -- 172.21.15.27:0/3223483460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4019070 con 0x7f35e8154d10 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.043+0000 7f35de000640 1 -- 172.21.15.27:0/3223483460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d400bdb0 con 0x7f35e8154d10 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.043+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e81eec70 con 0x7f35e8154d10 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.043+0000 7f35de000640 1 -- 172.21.15.27:0/3223483460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4004ae0 con 0x7f35e8154d10 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.043+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e81ef230 con 0x7f35e8154d10 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.045+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e81ef560 con 0x7f35e8154d10 2024-03-28T19:03:41.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.046+0000 7f35de000640 1 -- 172.21.15.27:0/3223483460 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35d401c070 con 0x7f35e8154d10 2024-03-28T19:03:41.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.047+0000 7f35de000640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f35c806bbd0 0x7f35c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:41.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.047+0000 7f35de000640 1 -- 172.21.15.27:0/3223483460 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f35d408ae30 con 0x7f35e8154d10 2024-03-28T19:03:41.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.047+0000 7f35e5e00640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f35c806bbd0 0x7f35c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:41.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.048+0000 7f35e5e00640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f35c806bbd0 0x7f35c806e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009a10 tx=0x7f35d8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:41.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.052+0000 7f35de000640 1 -- 172.21.15.27:0/3223483460 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35e8072870 con 0x7f35e8154d10 2024-03-28T19:03:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.307+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35e804f400 con 0x7f35e8154d10 2024-03-28T19:03:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.308+0000 7f35de000640 1 -- 172.21.15.27:0/3223483460 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f35d4059210 con 0x7f35e8154d10 2024-03-28T19:03:41.308 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:41.308 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:41.308 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f35c806bbd0 msgr2=0x7f35c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f35c806bbd0 0x7f35c806e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009a10 tx=0x7f35d8008040 comp rx=0 tx=0).stop 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8154d10 msgr2=0x7f35e81eb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8154d10 0x7f35e81eb980 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f35d400dbc0 tx=0x7f35d400b4a0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 shutdown_connections 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f35c806bbd0 0x7f35c806e090 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35e81ebec0 0x7f35e814c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35e8154d10 0x7f35e81eb980 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 --2- 172.21.15.27:0/3223483460 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35e80aa110 0x7f35e81eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 >> 172.21.15.27:0/3223483460 conn(0x7f35e806c5f0 msgr2=0x7f35e80a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 shutdown_connections 2024-03-28T19:03:41.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:41.311+0000 7f35ed000640 1 -- 172.21.15.27:0/3223483460 wait complete. 2024-03-28T19:03:41.330 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:03:43.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.558+0000 7fef83800640 1 Processor -- start 2024-03-28T19:03:43.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.558+0000 7fef83800640 1 -- start start 2024-03-28T19:03:43.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.560+0000 7fef83800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c155d60 0x7fef7c1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:43.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.560+0000 7fef83800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c0c7a10 con 0x7fef7c159650 2024-03-28T19:03:43.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.560+0000 7fef83800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c0c7170 con 0x7fef7c158cb0 2024-03-28T19:03:43.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.560+0000 7fef83800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c0c7450 con 0x7fef7c155d60 2024-03-28T19:03:43.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.560+0000 7fef80e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fef7c159650 0x7fef7c0c1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56100/0 (socket says 172.21.15.27:56100) 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.560+0000 7fef80e00640 1 -- 172.21.15.27:0/1907480627 learned_addr learned my addr 172.21.15.27:0/1907480627 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.560+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919416887 0 0) 0x7fef7c0c7a10 con 0x7fef7c159650 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef60003520 con 0x7fef7c159650 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef81800640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c155d60 0x7fef7c1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3924785611 0 0) 0x7fef60003520 con 0x7fef7c159650 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c155d60 msgr2=0x7fef7c1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c155d60 0x7fef7c1561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 >> v1:172.21.15.155:6789/0 conn(0x7fef7c158cb0 legacy=0x7fef7c06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c0c7730 con 0x7fef7c159650 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fef74001670 con 0x7fef7c159650 2024-03-28T19:03:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 >> v1:172.21.15.27:6789/0 conn(0x7fef7c159650 legacy=0x7fef7c0c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c155d60 0x7fef60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef60004d40 0x7fef600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c0c7450 con 0x7fef600039e0 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6000bb50 con 0x7fef60004d40 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.561+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6000bd50 con 0x7fef7c155d60 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.562+0000 7fef80e00640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.562+0000 7fef7be00640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c155d60 0x7fef60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.562+0000 7fef81800640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef60004d40 0x7fef600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.562+0000 7fef80e00640 1 -- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c155d60 msgr2=0x7fef60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.562+0000 7fef80e00640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c155d60 0x7fef60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.562+0000 7fef80e00640 1 -- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef60004d40 msgr2=0x7fef600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.562+0000 7fef80e00640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef60004d40 0x7fef600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.562+0000 7fef80e00640 1 -- 172.21.15.27:0/1907480627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c0c7730 con 0x7fef600039e0 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.563+0000 7fef80e00640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef60003e00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fef74001050 tx=0x7fef60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1206b41e241bbed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74007d10 con 0x7fef600039e0 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef74030070 con 0x7fef600039e0 2024-03-28T19:03:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef7b400640 1 -- 172.21.15.27:0/1907480627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7402cd40 con 0x7fef600039e0 2024-03-28T19:03:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef83800640 1 -- 172.21.15.27:0/1907480627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 msgr2=0x7fef60003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef83800640 1 --2- 172.21.15.27:0/1907480627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef60003e00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fef74001050 tx=0x7fef60005830 comp rx=0 tx=0).stop 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef81800640 1 -- 172.21.15.27:0/1907480627 reap_dead start 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef83800640 1 -- 172.21.15.27:0/1907480627 shutdown_connections 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef83800640 1 -- 172.21.15.27:0/1907480627 >> 172.21.15.27:0/1907480627 conn(0x7fef7c0c4750 msgr2=0x7fef7c0c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef83800640 1 -- 172.21.15.27:0/1907480627 shutdown_connections 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.564+0000 7fef83800640 1 -- 172.21.15.27:0/1907480627 wait complete. 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.566+0000 7fef83800640 1 Processor -- start 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.566+0000 7fef83800640 1 -- start start 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.567+0000 7fef83800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef7c1ec1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.567+0000 7fef83800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef7c0a6c70 0x7fef7c1df930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.567+0000 7fef83800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c151e80 0x7fef7c152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.567+0000 7fef83800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c06c960 con 0x7fef600039e0 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.567+0000 7fef83800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c06c7e0 con 0x7fef7c0a6c70 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.567+0000 7fef83800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c159a30 con 0x7fef7c151e80 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef80e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef7c1ec1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef80e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef7c1ec1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60140/0 (socket says 172.21.15.27:60140) 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef80e00640 1 -- 172.21.15.27:0/3030183540 learned_addr learned my addr 172.21.15.27:0/3030183540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef81800640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c151e80 0x7fef7c152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef7be00640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef7c0a6c70 0x7fef7c1df930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef80e00640 1 -- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c151e80 msgr2=0x7fef7c152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef80e00640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c151e80 0x7fef7c152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef80e00640 1 -- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef7c0a6c70 msgr2=0x7fef7c1df930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef80e00640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef7c0a6c70 0x7fef7c1df930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef80e00640 1 -- 172.21.15.27:0/3030183540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c1ded20 con 0x7fef600039e0 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.568+0000 7fef81800640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c151e80 0x7fef7c152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.569+0000 7fef80e00640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef7c1ec1c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fef7400c320 tx=0x7fef74000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.569+0000 7fef78c00640 1 -- 172.21.15.27:0/3030183540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74045070 con 0x7fef600039e0 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.569+0000 7fef78c00640 1 -- 172.21.15.27:0/3030183540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef74004bf0 con 0x7fef600039e0 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.569+0000 7fef78c00640 1 -- 172.21.15.27:0/3030183540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74007cf0 con 0x7fef600039e0 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.569+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c0a7150 con 0x7fef600039e0 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.569+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c1df2a0 con 0x7fef600039e0 2024-03-28T19:03:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.570+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c1df4a0 con 0x7fef600039e0 2024-03-28T19:03:43.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.573+0000 7fef78c00640 1 -- 172.21.15.27:0/3030183540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fef74007070 con 0x7fef600039e0 2024-03-28T19:03:43.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.573+0000 7fef78c00640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef5c06bc00 0x7fef5c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:43.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.574+0000 7fef78c00640 1 -- 172.21.15.27:0/3030183540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fef740ac340 con 0x7fef600039e0 2024-03-28T19:03:43.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.574+0000 7fef7be00640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef5c06bc00 0x7fef5c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:43.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.575+0000 7fef7be00640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef5c06bc00 0x7fef5c06e0c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef68007fd0 tx=0x7fef68009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:43.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.577+0000 7fef78c00640 1 -- 172.21.15.27:0/3030183540 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fef7407a810 con 0x7fef600039e0 2024-03-28T19:03:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.834+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef7c04f400 con 0x7fef600039e0 2024-03-28T19:03:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.835+0000 7fef78c00640 1 -- 172.21.15.27:0/3030183540 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7fef7407a110 con 0x7fef600039e0 2024-03-28T19:03:43.835 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:43.836 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef5c06bc00 msgr2=0x7fef5c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef5c06bc00 0x7fef5c06e0c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef68007fd0 tx=0x7fef68009040 comp rx=0 tx=0).stop 2024-03-28T19:03:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 msgr2=0x7fef7c1ec1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef7c1ec1c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fef7400c320 tx=0x7fef74000fe0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 shutdown_connections 2024-03-28T19:03:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef5c06bc00 0x7fef5c06e0c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef7c151e80 0x7fef7c152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef7c0a6c70 0x7fef7c1df930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 --2- 172.21.15.27:0/3030183540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef600039e0 0x7fef7c1ec1c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 >> 172.21.15.27:0/3030183540 conn(0x7fef7c0c4750 msgr2=0x7fef7c0c3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 shutdown_connections 2024-03-28T19:03:43.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:43.838+0000 7fef83800640 1 -- 172.21.15.27:0/3030183540 wait complete. 2024-03-28T19:03:43.859 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:03:45.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.087+0000 7f7e54a00640 1 Processor -- start 2024-03-28T19:03:46.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.087+0000 7f7e54a00640 1 -- start start 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e54a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 0x7f7e500733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e54a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e500c35c0 0x7f7e500c1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e54a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5006d7d0 con 0x7f7e500a6300 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e54a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e500c2f00 con 0x7f7e500c35c0 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e54a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c31e0 con 0x7f7e50070a50 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e4de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e500a6300 0x7f7e5006ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56106/0 (socket says 172.21.15.27:56106) 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e4de00640 1 -- 172.21.15.27:0/1024330348 learned_addr learned my addr 172.21.15.27:0/1024330348 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e4e800640 1 --2- 172.21.15.27:0/1024330348 >> v2:172.21.15.27:3300/0 conn(0x7f7e500c35c0 0x7f7e500c1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:46.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.089+0000 7f7e4d400640 1 --2- 172.21.15.27:0/1024330348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 0x7f7e500733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:46.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.090+0000 7f7e4ca00640 1 -- 172.21.15.27:0/1024330348 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976025565 0 0) 0x7f7e5006d7d0 con 0x7f7e500a6300 2024-03-28T19:03:46.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.090+0000 7f7e4ca00640 1 -- 172.21.15.27:0/1024330348 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e34003520 con 0x7f7e500a6300 2024-03-28T19:03:46.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.090+0000 7f7e4e800640 1 -- 172.21.15.27:0/1024330348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 msgr2=0x7f7e500733b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:46.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.090+0000 7f7e4e800640 1 --2- 172.21.15.27:0/1024330348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 0x7f7e500733b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.090+0000 7f7e4e800640 1 -- 172.21.15.27:0/1024330348 >> v1:172.21.15.27:6789/0 conn(0x7f7e500a6300 legacy=0x7f7e5006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:46.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.090+0000 7f7e4e800640 1 -- 172.21.15.27:0/1024330348 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5006dab0 con 0x7f7e500c35c0 2024-03-28T19:03:46.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.091+0000 7f7e4e800640 1 --2- 172.21.15.27:0/1024330348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e500c1c30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009d90 tx=0x7f7e4802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e2266a990948fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:46.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.091+0000 7f7e4ca00640 1 -- 172.21.15.27:0/1024330348 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48039070 con 0x7f7e500c35c0 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.091+0000 7f7e4ca00640 1 -- 172.21.15.27:0/1024330348 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4802cb40 con 0x7f7e500c35c0 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.091+0000 7f7e4ca00640 1 -- 172.21.15.27:0/1024330348 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48034440 con 0x7f7e500c35c0 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.092+0000 7f7e54a00640 1 -- 172.21.15.27:0/1024330348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 msgr2=0x7f7e500c1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.092+0000 7f7e54a00640 1 --2- 172.21.15.27:0/1024330348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e500c1c30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009d90 tx=0x7f7e4802bd90 comp rx=0 tx=0).stop 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.092+0000 7f7e54a00640 1 -- 172.21.15.27:0/1024330348 shutdown_connections 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.092+0000 7f7e54a00640 1 --2- 172.21.15.27:0/1024330348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e500c1c30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.092+0000 7f7e54a00640 1 --2- 172.21.15.27:0/1024330348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 0x7f7e500733b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.092+0000 7f7e54a00640 1 -- 172.21.15.27:0/1024330348 >> 172.21.15.27:0/1024330348 conn(0x7f7e5006b9d0 msgr2=0x7f7e500c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.092+0000 7f7e54a00640 1 -- 172.21.15.27:0/1024330348 shutdown_connections 2024-03-28T19:03:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.092+0000 7f7e54a00640 1 -- 172.21.15.27:0/1024330348 wait complete. 2024-03-28T19:03:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.094+0000 7f7e54a00640 1 Processor -- start 2024-03-28T19:03:46.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.094+0000 7f7e54a00640 1 -- start start 2024-03-28T19:03:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e54a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 0x7f7e50154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e54a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e500a6300 0x7f7e50156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e54a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e50156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e54a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c22f0 con 0x7f7e500c35c0 2024-03-28T19:03:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e54a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c2170 con 0x7f7e500a6300 2024-03-28T19:03:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e54a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c2470 con 0x7f7e50070a50 2024-03-28T19:03:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e4e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e50156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e4e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e50156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60156/0 (socket says 172.21.15.27:60156) 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e4e800640 1 -- 172.21.15.27:0/2268206599 learned_addr learned my addr 172.21.15.27:0/2268206599 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e4d400640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e500a6300 0x7f7e50156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.095+0000 7f7e4de00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 0x7f7e50154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.096+0000 7f7e4e800640 1 -- 172.21.15.27:0/2268206599 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 msgr2=0x7f7e50154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.096+0000 7f7e4e800640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 0x7f7e50154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.096+0000 7f7e4e800640 1 -- 172.21.15.27:0/2268206599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e500a6300 msgr2=0x7f7e50156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.096+0000 7f7e4e800640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e500a6300 0x7f7e50156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.096+0000 7f7e4e800640 1 -- 172.21.15.27:0/2268206599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e50153630 con 0x7f7e500c35c0 2024-03-28T19:03:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.096+0000 7f7e4e800640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e50156c40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7e4802bdd0 tx=0x7f7e48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.096+0000 7f7e46000640 1 -- 172.21.15.27:0/2268206599 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48044020 con 0x7f7e500c35c0 2024-03-28T19:03:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.097+0000 7f7e46000640 1 -- 172.21.15.27:0/2268206599 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48004280 con 0x7f7e500c35c0 2024-03-28T19:03:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.097+0000 7f7e46000640 1 -- 172.21.15.27:0/2268206599 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48039040 con 0x7f7e500c35c0 2024-03-28T19:03:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.097+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e50159530 con 0x7f7e500c35c0 2024-03-28T19:03:46.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.097+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e50159af0 con 0x7f7e500c35c0 2024-03-28T19:03:46.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.099+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e500c7520 con 0x7f7e500c35c0 2024-03-28T19:03:46.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.099+0000 7f7e46000640 1 -- 172.21.15.27:0/2268206599 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7e48047020 con 0x7f7e500c35c0 2024-03-28T19:03:46.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.100+0000 7f7e46000640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e3006bbd0 0x7f7e3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:46.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.100+0000 7f7e46000640 1 -- 172.21.15.27:0/2268206599 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f7e480ad980 con 0x7f7e500c35c0 2024-03-28T19:03:46.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.100+0000 7f7e4de00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e3006bbd0 0x7f7e3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:46.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.101+0000 7f7e4de00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e3006bbd0 0x7f7e3006e090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7e40005880 tx=0x7f7e40009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:46.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.105+0000 7f7e46000640 1 -- 172.21.15.27:0/2268206599 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7e500c7520 con 0x7f7e500c35c0 2024-03-28T19:03:46.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.348+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e5004f400 con 0x7f7e500c35c0 2024-03-28T19:03:46.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.349+0000 7f7e46000640 1 -- 172.21.15.27:0/2268206599 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f7e4804e030 con 0x7f7e500c35c0 2024-03-28T19:03:46.348 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:46.348 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:46.349 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e3006bbd0 msgr2=0x7f7e3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e3006bbd0 0x7f7e3006e090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7e40005880 tx=0x7f7e40009040 comp rx=0 tx=0).stop 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 msgr2=0x7f7e50156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e50156c40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7e4802bdd0 tx=0x7f7e48004a40 comp rx=0 tx=0).stop 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 shutdown_connections 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e3006bbd0 0x7f7e3006e090 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e500c35c0 0x7f7e50156c40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e500a6300 0x7f7e50156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 --2- 172.21.15.27:0/2268206599 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e50070a50 0x7f7e50154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.351+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 >> 172.21.15.27:0/2268206599 conn(0x7f7e5006b9d0 msgr2=0x7f7e5006ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:46.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.352+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 shutdown_connections 2024-03-28T19:03:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:46.352+0000 7f7e54a00640 1 -- 172.21.15.27:0/2268206599 wait complete. 2024-03-28T19:03:46.371 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:03:48.372 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:48.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.600+0000 7fb0ca400640 1 Processor -- start 2024-03-28T19:03:48.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.600+0000 7fb0ca400640 1 -- start start 2024-03-28T19:03:48.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.601+0000 7fb0ca400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb0c4154380 0x7fb0c406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:48.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.601+0000 7fb0ca400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0c40c26e0 0x7fb0c40c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:48.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.601+0000 7fb0ca400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40a7490 con 0x7fb0c406c610 2024-03-28T19:03:48.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.601+0000 7fb0ca400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c4154760 con 0x7fb0c40c26e0 2024-03-28T19:03:48.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.601+0000 7fb0ca400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c40a7240 con 0x7fb0c4154380 2024-03-28T19:03:48.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.602+0000 7fb0c3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0c40c26e0 0x7fb0c40c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:48.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.602+0000 7fb0c9a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0c406c610 0x7fb0c4066dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54638/0 (socket says 172.21.15.27:54638) 2024-03-28T19:03:48.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.602+0000 7fb0c9a00640 1 -- 172.21.15.27:0/2478737272 learned_addr learned my addr 172.21.15.27:0/2478737272 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:48.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.602+0000 7fb0c9000640 1 --2- 172.21.15.27:0/2478737272 >> v2:172.21.15.155:3300/0 conn(0x7fb0c4154380 0x7fb0c406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:48.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.602+0000 7fb0c3e00640 1 -- 172.21.15.27:0/2478737272 >> v2:172.21.15.155:3300/0 conn(0x7fb0c4154380 msgr2=0x7fb0c406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:48.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.602+0000 7fb0c3e00640 1 --2- 172.21.15.27:0/2478737272 >> v2:172.21.15.155:3300/0 conn(0x7fb0c4154380 0x7fb0c406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.602+0000 7fb0c3e00640 1 -- 172.21.15.27:0/2478737272 >> v1:172.21.15.27:6789/0 conn(0x7fb0c406c610 legacy=0x7fb0c4066dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:48.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.603+0000 7fb0c3e00640 1 -- 172.21.15.27:0/2478737272 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40725a0 con 0x7fb0c40c26e0 2024-03-28T19:03:48.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.604+0000 7fb0c3e00640 1 --2- 172.21.15.27:0/2478737272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40c26e0 0x7fb0c40c5c50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40714d0 tx=0x7fb0ac02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4ddb974fe16699 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:48.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.604+0000 7fb0c3400640 1 -- 172.21.15.27:0/2478737272 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac039070 con 0x7fb0c40c26e0 2024-03-28T19:03:48.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.604+0000 7fb0c3400640 1 -- 172.21.15.27:0/2478737272 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac02cc70 con 0x7fb0c40c26e0 2024-03-28T19:03:48.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.604+0000 7fb0c3400640 1 -- 172.21.15.27:0/2478737272 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac034660 con 0x7fb0c40c26e0 2024-03-28T19:03:48.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.605+0000 7fb0ca400640 1 -- 172.21.15.27:0/2478737272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40c26e0 msgr2=0x7fb0c40c5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:48.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.605+0000 7fb0ca400640 1 --2- 172.21.15.27:0/2478737272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40c26e0 0x7fb0c40c5c50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40714d0 tx=0x7fb0ac02bec0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.605+0000 7fb0ca400640 1 -- 172.21.15.27:0/2478737272 shutdown_connections 2024-03-28T19:03:48.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.605+0000 7fb0ca400640 1 --2- 172.21.15.27:0/2478737272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c40c26e0 0x7fb0c40c5c50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.605+0000 7fb0ca400640 1 --2- 172.21.15.27:0/2478737272 >> v2:172.21.15.155:3300/0 conn(0x7fb0c4154380 0x7fb0c406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.605+0000 7fb0ca400640 1 -- 172.21.15.27:0/2478737272 >> 172.21.15.27:0/2478737272 conn(0x7fb0c40c7aa0 msgr2=0x7fb0c40c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:48.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.605+0000 7fb0ca400640 1 -- 172.21.15.27:0/2478737272 shutdown_connections 2024-03-28T19:03:48.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.605+0000 7fb0ca400640 1 -- 172.21.15.27:0/2478737272 wait complete. 2024-03-28T19:03:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.607+0000 7fb0ca400640 1 Processor -- start 2024-03-28T19:03:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.607+0000 7fb0ca400640 1 -- start start 2024-03-28T19:03:48.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0ca400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406c610 0x7fb0c4150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0ca400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0c4154380 0x7fb0c40be670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0ca400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0c414e800 0x7fb0c40bebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0ca400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40c2e60 con 0x7fb0c406c610 2024-03-28T19:03:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0ca400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40c31d0 con 0x7fb0c4154380 2024-03-28T19:03:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0ca400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c40c77b0 con 0x7fb0c414e800 2024-03-28T19:03:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0c9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406c610 0x7fb0c4150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0c9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406c610 0x7fb0c4150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55722/0 (socket says 172.21.15.27:55722) 2024-03-28T19:03:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.608+0000 7fb0c9000640 1 -- 172.21.15.27:0/462300298 learned_addr learned my addr 172.21.15.27:0/462300298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:48.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c3e00640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0c4154380 0x7fb0c40be670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c9a00640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0c414e800 0x7fb0c40bebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c9000640 1 -- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0c414e800 msgr2=0x7fb0c40bebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c9000640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0c414e800 0x7fb0c40bebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c9000640 1 -- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0c4154380 msgr2=0x7fb0c40be670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c9000640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0c4154380 0x7fb0c40be670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c9000640 1 -- 172.21.15.27:0/462300298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40bf0f0 con 0x7fb0c406c610 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c3e00640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0c4154380 0x7fb0c40be670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c9a00640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0c414e800 0x7fb0c40bebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c9000640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406c610 0x7fb0c4150930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400d5f0 tx=0x7fb0b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.609+0000 7fb0c0c00640 1 -- 172.21.15.27:0/462300298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40125f0 con 0x7fb0c406c610 2024-03-28T19:03:48.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.610+0000 7fb0c0c00640 1 -- 172.21.15.27:0/462300298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b4019070 con 0x7fb0c406c610 2024-03-28T19:03:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.610+0000 7fb0c0c00640 1 -- 172.21.15.27:0/462300298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40129d0 con 0x7fb0c406c610 2024-03-28T19:03:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.610+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c4157280 con 0x7fb0c406c610 2024-03-28T19:03:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.610+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c4157520 con 0x7fb0c406c610 2024-03-28T19:03:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.611+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c4069640 con 0x7fb0c406c610 2024-03-28T19:03:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.612+0000 7fb0c0c00640 1 -- 172.21.15.27:0/462300298 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb0b4012b70 con 0x7fb0c406c610 2024-03-28T19:03:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.613+0000 7fb0c0c00640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb0a006bc00 0x7fb0a006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.613+0000 7fb0c0c00640 1 -- 172.21.15.27:0/462300298 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fb0b408b4a0 con 0x7fb0c406c610 2024-03-28T19:03:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.613+0000 7fb0c3e00640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb0a006bc00 0x7fb0a006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.614+0000 7fb0c3e00640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb0a006bc00 0x7fb0a006e0c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac036000 tx=0x7fb0ac030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:48.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.617+0000 7fb0c0c00640 1 -- 172.21.15.27:0/462300298 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb0c40bf0f0 con 0x7fb0c406c610 2024-03-28T19:03:48.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.906+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c40a7020 con 0x7fb0c406c610 2024-03-28T19:03:48.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.907+0000 7fb0c0c00640 1 -- 172.21.15.27:0/462300298 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7fb0b4059920 con 0x7fb0c406c610 2024-03-28T19:03:48.906 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:48.906 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:48.906 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb0a006bc00 msgr2=0x7fb0a006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb0a006bc00 0x7fb0a006e0c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac036000 tx=0x7fb0ac030000 comp rx=0 tx=0).stop 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406c610 msgr2=0x7fb0c4150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406c610 0x7fb0c4150930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400d5f0 tx=0x7fb0b400dab0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 shutdown_connections 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb0a006bc00 0x7fb0a006e0c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0c414e800 0x7fb0c40bebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0c4154380 0x7fb0c40be670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 --2- 172.21.15.27:0/462300298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406c610 0x7fb0c4150930 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 >> 172.21.15.27:0/462300298 conn(0x7fb0c40c7aa0 msgr2=0x7fb0c40a80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.909+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 shutdown_connections 2024-03-28T19:03:48.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:48.910+0000 7fb0ca400640 1 -- 172.21.15.27:0/462300298 wait complete. 2024-03-28T19:03:48.935 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:03:50.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:51.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.169+0000 7fef10e00640 1 Processor -- start 2024-03-28T19:03:51.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.169+0000 7fef10e00640 1 -- start start 2024-03-28T19:03:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.170+0000 7fef10e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef0c0c58c0 0x7fef0c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.170+0000 7fef10e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fef0c0c8ed0 0x7fef0c1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.170+0000 7fef10e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c156230 con 0x7fef0c0be5d0 2024-03-28T19:03:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.170+0000 7fef10e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0c158bd0 con 0x7fef0c0c58c0 2024-03-28T19:03:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.170+0000 7fef10e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0c158eb0 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.171+0000 7fef0b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef0c0c58c0 0x7fef0c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.171+0000 7fef0b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef0c0c58c0 0x7fef0c0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55726/0 (socket says 172.21.15.27:55726) 2024-03-28T19:03:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.171+0000 7fef0b400640 1 -- 172.21.15.27:0/1007793739 learned_addr learned my addr 172.21.15.27:0/1007793739 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.171+0000 7fef0aa00640 1 --2- 172.21.15.27:0/1007793739 >> v2:172.21.15.155:3300/0 conn(0x7fef0c0c8ed0 0x7fef0c1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.172+0000 7fef0b400640 1 -- 172.21.15.27:0/1007793739 >> v2:172.21.15.155:3300/0 conn(0x7fef0c0c8ed0 msgr2=0x7fef0c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.172+0000 7fef0b400640 1 --2- 172.21.15.27:0/1007793739 >> v2:172.21.15.155:3300/0 conn(0x7fef0c0c8ed0 0x7fef0c1555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.172+0000 7fef0b400640 1 -- 172.21.15.27:0/1007793739 >> v1:172.21.15.155:6789/0 conn(0x7fef0c0be5d0 legacy=0x7fef0c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.172+0000 7fef0b400640 1 -- 172.21.15.27:0/1007793739 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c159190 con 0x7fef0c0c58c0 2024-03-28T19:03:51.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.173+0000 7fef0b400640 1 --2- 172.21.15.27:0/1007793739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c58c0 0x7fef0c0c5ca0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0c83d0 tx=0x7feefc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959c1f58a32dbe16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:51.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.173+0000 7fef0a000640 1 -- 172.21.15.27:0/1007793739 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc02e070 con 0x7fef0c0c58c0 2024-03-28T19:03:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.174+0000 7fef0a000640 1 -- 172.21.15.27:0/1007793739 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc037070 con 0x7fef0c0c58c0 2024-03-28T19:03:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.174+0000 7fef0a000640 1 -- 172.21.15.27:0/1007793739 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc031a00 con 0x7fef0c0c58c0 2024-03-28T19:03:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.174+0000 7fef10e00640 1 -- 172.21.15.27:0/1007793739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c58c0 msgr2=0x7fef0c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.174+0000 7fef10e00640 1 --2- 172.21.15.27:0/1007793739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c58c0 0x7fef0c0c5ca0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0c83d0 tx=0x7feefc02b880 comp rx=0 tx=0).stop 2024-03-28T19:03:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.175+0000 7fef10e00640 1 -- 172.21.15.27:0/1007793739 shutdown_connections 2024-03-28T19:03:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.175+0000 7fef10e00640 1 --2- 172.21.15.27:0/1007793739 >> v2:172.21.15.155:3300/0 conn(0x7fef0c0c8ed0 0x7fef0c1555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.175+0000 7fef10e00640 1 --2- 172.21.15.27:0/1007793739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c58c0 0x7fef0c0c5ca0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.175+0000 7fef10e00640 1 -- 172.21.15.27:0/1007793739 >> 172.21.15.27:0/1007793739 conn(0x7fef0c06b6f0 msgr2=0x7fef0c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.175+0000 7fef10e00640 1 -- 172.21.15.27:0/1007793739 shutdown_connections 2024-03-28T19:03:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.175+0000 7fef10e00640 1 -- 172.21.15.27:0/1007793739 wait complete. 2024-03-28T19:03:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.178+0000 7fef10e00640 1 Processor -- start 2024-03-28T19:03:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.178+0000 7fef10e00640 1 -- start start 2024-03-28T19:03:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef10e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef0c0be5d0 0x7fef0c1525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef10e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef0c0c58c0 0x7fef0c0c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef10e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c8ed0 0x7fef0c0ac700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef10e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c0b0820 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef10e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c159750 con 0x7fef0c0be5d0 2024-03-28T19:03:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef10e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c0b09a0 con 0x7fef0c0c58c0 2024-03-28T19:03:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef0be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c8ed0 0x7fef0c0ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef0be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c8ed0 0x7fef0c0ac700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55734/0 (socket says 172.21.15.27:55734) 2024-03-28T19:03:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef0be00640 1 -- 172.21.15.27:0/182977664 learned_addr learned my addr 172.21.15.27:0/182977664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef0aa00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef0c0c58c0 0x7fef0c0c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.180+0000 7fef0b400640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef0c0be5d0 0x7fef0c1525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef0be00640 1 -- 172.21.15.27:0/182977664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef0c0c58c0 msgr2=0x7fef0c0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef0be00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef0c0c58c0 0x7fef0c0c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef0be00640 1 -- 172.21.15.27:0/182977664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef0c0be5d0 msgr2=0x7fef0c1525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef0be00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef0c0be5d0 0x7fef0c1525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef0be00640 1 -- 172.21.15.27:0/182977664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c0c2470 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef0be00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c8ed0 0x7fef0c0ac700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fef0400c6b0 tx=0x7fef0400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef03400640 1 -- 172.21.15.27:0/182977664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04003b70 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef03400640 1 -- 172.21.15.27:0/182977664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef04003d30 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0c0c2750 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef03400640 1 -- 172.21.15.27:0/182977664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04005310 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.181+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0c0acc40 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.183+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0c069640 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.184+0000 7fef03400640 1 -- 172.21.15.27:0/182977664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fef040054b0 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.185+0000 7fef03400640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feeec06bbd0 0x7feeec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:51.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.186+0000 7fef03400640 1 -- 172.21.15.27:0/182977664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fef0408aff0 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.186+0000 7fef0b400640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feeec06bbd0 0x7feeec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.187+0000 7fef0b400640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feeec06bbd0 0x7feeec06e090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feefc036000 tx=0x7feefc03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:51.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.190+0000 7fef03400640 1 -- 172.21.15.27:0/182977664 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fef040593f0 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.442+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef0c156580 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.443+0000 7fef03400640 1 -- 172.21.15.27:0/182977664 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7fef04058cf0 con 0x7fef0c0c8ed0 2024-03-28T19:03:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:51.443 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feeec06bbd0 msgr2=0x7feeec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feeec06bbd0 0x7feeec06e090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feefc036000 tx=0x7feefc03d040 comp rx=0 tx=0).stop 2024-03-28T19:03:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c8ed0 msgr2=0x7fef0c0ac700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c8ed0 0x7fef0c0ac700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fef0400c6b0 tx=0x7fef0400cb70 comp rx=0 tx=0).stop 2024-03-28T19:03:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 shutdown_connections 2024-03-28T19:03:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feeec06bbd0 0x7feeec06e090 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef0c0c8ed0 0x7fef0c0ac700 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef0c0c58c0 0x7fef0c0c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 --2- 172.21.15.27:0/182977664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef0c0be5d0 0x7fef0c1525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:51.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 >> 172.21.15.27:0/182977664 conn(0x7fef0c06b6f0 msgr2=0x7fef0c0ad9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:51.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 shutdown_connections 2024-03-28T19:03:51.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:51.446+0000 7fef10e00640 1 -- 172.21.15.27:0/182977664 wait complete. 2024-03-28T19:03:51.466 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:03:53.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.727+0000 7f0965000640 1 Processor -- start 2024-03-28T19:03:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.727+0000 7f0965000640 1 -- start start 2024-03-28T19:03:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.728+0000 7f0965000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f09600c59d0 0x7f09600c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.728+0000 7f0965000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09600727e0 0x7f0960159520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.729+0000 7f0965000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0960155fe0 con 0x7f0960155600 2024-03-28T19:03:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.729+0000 7f0965000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09601565a0 con 0x7f09600727e0 2024-03-28T19:03:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.729+0000 7f0965000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0960159090 con 0x7f09600c59d0 2024-03-28T19:03:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.729+0000 7f095ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09600727e0 0x7f0960159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.729+0000 7f095fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0960155600 0x7f096014f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54654/0 (socket says 172.21.15.27:54654) 2024-03-28T19:03:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.729+0000 7f095fe00640 1 -- 172.21.15.27:0/2008447209 learned_addr learned my addr 172.21.15.27:0/2008447209 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.729+0000 7f095f400640 1 --2- 172.21.15.27:0/2008447209 >> v2:172.21.15.155:3300/0 conn(0x7f09600c59d0 0x7f09600c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.730+0000 7f095ea00640 1 -- 172.21.15.27:0/2008447209 >> v2:172.21.15.155:3300/0 conn(0x7f09600c59d0 msgr2=0x7f09600c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.730+0000 7f095ea00640 1 --2- 172.21.15.27:0/2008447209 >> v2:172.21.15.155:3300/0 conn(0x7f09600c59d0 0x7f09600c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.730+0000 7f095ea00640 1 -- 172.21.15.27:0/2008447209 >> v1:172.21.15.27:6789/0 conn(0x7f0960155600 legacy=0x7f096014f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.730+0000 7f095ea00640 1 -- 172.21.15.27:0/2008447209 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09601562c0 con 0x7f09600727e0 2024-03-28T19:03:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.730+0000 7f095e000640 1 -- 172.21.15.27:0/2008447209 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592299211 0 0) 0x7f0960155fe0 con 0x7f0960155600 2024-03-28T19:03:53.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.731+0000 7f095ea00640 1 --2- 172.21.15.27:0/2008447209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09600727e0 0x7f0960159520 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f09600aa5c0 tx=0x7f094c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c695685dddbee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:53.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.731+0000 7f095e000640 1 -- 172.21.15.27:0/2008447209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c038070 con 0x7f09600727e0 2024-03-28T19:03:53.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.731+0000 7f095e000640 1 -- 172.21.15.27:0/2008447209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c031520 con 0x7f09600727e0 2024-03-28T19:03:53.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.731+0000 7f095e000640 1 -- 172.21.15.27:0/2008447209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0318f0 con 0x7f09600727e0 2024-03-28T19:03:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.732+0000 7f0965000640 1 -- 172.21.15.27:0/2008447209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09600727e0 msgr2=0x7f0960159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.732+0000 7f0965000640 1 --2- 172.21.15.27:0/2008447209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09600727e0 0x7f0960159520 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f09600aa5c0 tx=0x7f094c02b7c0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.732+0000 7f0965000640 1 -- 172.21.15.27:0/2008447209 shutdown_connections 2024-03-28T19:03:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.732+0000 7f0965000640 1 --2- 172.21.15.27:0/2008447209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09600727e0 0x7f0960159520 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.732+0000 7f0965000640 1 --2- 172.21.15.27:0/2008447209 >> v2:172.21.15.155:3300/0 conn(0x7f09600c59d0 0x7f09600c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.732+0000 7f0965000640 1 -- 172.21.15.27:0/2008447209 >> 172.21.15.27:0/2008447209 conn(0x7f09600ac910 msgr2=0x7f09600accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.732+0000 7f0965000640 1 -- 172.21.15.27:0/2008447209 shutdown_connections 2024-03-28T19:03:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.732+0000 7f0965000640 1 -- 172.21.15.27:0/2008447209 wait complete. 2024-03-28T19:03:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.734+0000 7f0965000640 1 Processor -- start 2024-03-28T19:03:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.734+0000 7f0965000640 1 -- start start 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.735+0000 7f0965000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09600727e0 0x7f09600c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.735+0000 7f0965000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09600c59d0 0x7f09600aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.735+0000 7f0965000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0960155600 0x7f09601569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.735+0000 7f0965000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096014f970 con 0x7f0960155600 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.735+0000 7f0965000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0960159a60 con 0x7f09600c59d0 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.735+0000 7f0965000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096014faf0 con 0x7f09600727e0 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.735+0000 7f095fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0960155600 0x7f09601569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0960155600 0x7f09601569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55746/0 (socket says 172.21.15.27:55746) 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095fe00640 1 -- 172.21.15.27:0/1958734910 learned_addr learned my addr 172.21.15.27:0/1958734910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095f400640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09600727e0 0x7f09600c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095ea00640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09600c59d0 0x7f09600aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:53.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095fe00640 1 -- 172.21.15.27:0/1958734910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09600727e0 msgr2=0x7f09600c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:53.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095fe00640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09600727e0 0x7f09600c2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095fe00640 1 -- 172.21.15.27:0/1958734910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09600c59d0 msgr2=0x7f09600aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:53.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095fe00640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09600c59d0 0x7f09600aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.736+0000 7f095fe00640 1 -- 172.21.15.27:0/1958734910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09600c1f20 con 0x7f0960155600 2024-03-28T19:03:53.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.737+0000 7f095fe00640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0960155600 0x7f09601569f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f095800c700 tx=0x7f095800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.737+0000 7f0957400640 1 -- 172.21.15.27:0/1958734910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0958018070 con 0x7f0960155600 2024-03-28T19:03:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.737+0000 7f0957400640 1 -- 172.21.15.27:0/1958734910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095800ce10 con 0x7f0960155600 2024-03-28T19:03:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.737+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0960153370 con 0x7f0960155600 2024-03-28T19:03:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.737+0000 7f0957400640 1 -- 172.21.15.27:0/1958734910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095800a860 con 0x7f0960155600 2024-03-28T19:03:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.737+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09600c4cf0 con 0x7f0960155600 2024-03-28T19:03:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.739+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0960067dd0 con 0x7f0960155600 2024-03-28T19:03:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.740+0000 7f0957400640 1 -- 172.21.15.27:0/1958734910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f095801b070 con 0x7f0960155600 2024-03-28T19:03:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.740+0000 7f0957400640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f094006bc00 0x7f094006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.741+0000 7f0957400640 1 -- 172.21.15.27:0/1958734910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f095808a530 con 0x7f0960155600 2024-03-28T19:03:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.741+0000 7f095f400640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f094006bc00 0x7f094006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.741+0000 7f095f400640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f094006bc00 0x7f094006e0c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0950006fd0 tx=0x7f0950008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:53.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.745+0000 7f0957400640 1 -- 172.21.15.27:0/1958734910 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0958058360 con 0x7f0960155600 2024-03-28T19:03:53.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.992+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0960069640 con 0x7f0960155600 2024-03-28T19:03:53.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.993+0000 7f0957400640 1 -- 172.21.15.27:0/1958734910 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f095805c400 con 0x7f0960155600 2024-03-28T19:03:53.993 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:53.993 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:09.742278+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/1459475560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1459475560},{"type":"v1","addr":"172.21.15.27:6837","nonce":1459475560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:53.993 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 71 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f094006bc00 msgr2=0x7f094006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f094006bc00 0x7f094006e0c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0950006fd0 tx=0x7f0950008040 comp rx=0 tx=0).stop 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0960155600 msgr2=0x7f09601569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0960155600 0x7f09601569f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f095800c700 tx=0x7f095800cbc0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 shutdown_connections 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f094006bc00 0x7f094006e0c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0960155600 0x7f09601569f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09600c59d0 0x7f09600aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 --2- 172.21.15.27:0/1958734910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09600727e0 0x7f09600c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 >> 172.21.15.27:0/1958734910 conn(0x7f09600ac910 msgr2=0x7f0960154520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:53.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 shutdown_connections 2024-03-28T19:03:53.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:53.996+0000 7f0965000640 1 -- 172.21.15.27:0/1958734910 wait complete. 2024-03-28T19:03:54.016 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:03:56.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.236+0000 7fae75e00640 1 Processor -- start 2024-03-28T19:03:56.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.237+0000 7fae75e00640 1 -- start start 2024-03-28T19:03:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.239+0000 7fae75e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fae700c4dc0 0x7fae70070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.239+0000 7fae75e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70070a80 0x7fae7015ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.239+0000 7fae75e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae700c52e0 con 0x7fae70152bd0 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.239+0000 7fae75e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae701557e0 con 0x7fae700c4dc0 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.239+0000 7fae75e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae70155ac0 con 0x7fae70070a80 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.239+0000 7fae6ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fae70152bd0 0x7fae7006eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51410/0 (socket says 172.21.15.27:51410) 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.239+0000 7fae6ea00640 1 -- 172.21.15.27:0/3621090993 learned_addr learned my addr 172.21.15.27:0/3621090993 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.239+0000 7fae6f400640 1 --2- 172.21.15.27:0/3621090993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70070a80 0x7fae7015ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.240+0000 7fae6e000640 1 --2- 172.21.15.27:0/3621090993 >> v2:172.21.15.155:3300/0 conn(0x7fae700c4dc0 0x7fae70070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.240+0000 7fae6d600640 1 -- 172.21.15.27:0/3621090993 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001431113 0 0) 0x7fae700c52e0 con 0x7fae70152bd0 2024-03-28T19:03:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.240+0000 7fae6d600640 1 -- 172.21.15.27:0/3621090993 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae54003520 con 0x7fae70152bd0 2024-03-28T19:03:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.240+0000 7fae6f400640 1 -- 172.21.15.27:0/3621090993 >> v2:172.21.15.155:3300/0 conn(0x7fae700c4dc0 msgr2=0x7fae70070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.240+0000 7fae6f400640 1 --2- 172.21.15.27:0/3621090993 >> v2:172.21.15.155:3300/0 conn(0x7fae700c4dc0 0x7fae70070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.240+0000 7fae6f400640 1 -- 172.21.15.27:0/3621090993 >> v1:172.21.15.27:6789/0 conn(0x7fae70152bd0 legacy=0x7fae7006eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.240+0000 7fae6f400640 1 -- 172.21.15.27:0/3621090993 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae70155da0 con 0x7fae70070a80 2024-03-28T19:03:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.241+0000 7fae6f400640 1 --2- 172.21.15.27:0/3621090993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70070a80 0x7fae7015ffc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae68009d90 tx=0x7fae6802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80a5d7cf0d619e57 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.242+0000 7fae6d600640 1 -- 172.21.15.27:0/3621090993 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68039070 con 0x7fae70070a80 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.242+0000 7fae6d600640 1 -- 172.21.15.27:0/3621090993 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6802cc50 con 0x7fae70070a80 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.242+0000 7fae6d600640 1 -- 172.21.15.27:0/3621090993 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68034440 con 0x7fae70070a80 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.243+0000 7fae75e00640 1 -- 172.21.15.27:0/3621090993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70070a80 msgr2=0x7fae7015ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.243+0000 7fae75e00640 1 --2- 172.21.15.27:0/3621090993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70070a80 0x7fae7015ffc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae68009d90 tx=0x7fae6802bea0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.243+0000 7fae75e00640 1 -- 172.21.15.27:0/3621090993 shutdown_connections 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.243+0000 7fae75e00640 1 --2- 172.21.15.27:0/3621090993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70070a80 0x7fae7015ffc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.243+0000 7fae75e00640 1 --2- 172.21.15.27:0/3621090993 >> v2:172.21.15.155:3300/0 conn(0x7fae700c4dc0 0x7fae70070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.243+0000 7fae75e00640 1 -- 172.21.15.27:0/3621090993 >> 172.21.15.27:0/3621090993 conn(0x7fae70066dd0 msgr2=0x7fae700671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.243+0000 7fae75e00640 1 -- 172.21.15.27:0/3621090993 shutdown_connections 2024-03-28T19:03:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.243+0000 7fae75e00640 1 -- 172.21.15.27:0/3621090993 wait complete. 2024-03-28T19:03:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.245+0000 7fae75e00640 1 Processor -- start 2024-03-28T19:03:56.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.245+0000 7fae75e00640 1 -- start start 2024-03-28T19:03:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.246+0000 7fae75e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae70070a80 0x7fae70158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.246+0000 7fae75e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae700c4dc0 0x7fae70145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae75e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70152bd0 0x7fae70146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae75e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae700c6c70 con 0x7fae700c4dc0 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae75e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae70152fb0 con 0x7fae70070a80 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae75e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae700c6df0 con 0x7fae70152bd0 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae6e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae700c4dc0 0x7fae70145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae6e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae700c4dc0 0x7fae70145fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38694/0 (socket says 172.21.15.27:38694) 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae6e000640 1 -- 172.21.15.27:0/2997960067 learned_addr learned my addr 172.21.15.27:0/2997960067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae6ea00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae70070a80 0x7fae70158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae6f400640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70152bd0 0x7fae70146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.247+0000 7fae6e000640 1 -- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70152bd0 msgr2=0x7fae70146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae6e000640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70152bd0 0x7fae70146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae6e000640 1 -- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae70070a80 msgr2=0x7fae70158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae6e000640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae70070a80 0x7fae70158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae6e000640 1 -- 172.21.15.27:0/2997960067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae701ebbf0 con 0x7fae700c4dc0 2024-03-28T19:03:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae6f400640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70152bd0 0x7fae70146500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:03:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae6e000640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae700c4dc0 0x7fae70145fc0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e9b0 tx=0x7fae5c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae67400640 1 -- 172.21.15.27:0/2997960067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c017070 con 0x7fae700c4dc0 2024-03-28T19:03:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae67400640 1 -- 172.21.15.27:0/2997960067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c00c9a0 con 0x7fae700c4dc0 2024-03-28T19:03:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae67400640 1 -- 172.21.15.27:0/2997960067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c00cd90 con 0x7fae700c4dc0 2024-03-28T19:03:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.248+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae701ebe90 con 0x7fae700c4dc0 2024-03-28T19:03:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.249+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7009adb0 con 0x7fae700c4dc0 2024-03-28T19:03:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.250+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7004f400 con 0x7fae700c4dc0 2024-03-28T19:03:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.251+0000 7fae67400640 1 -- 172.21.15.27:0/2997960067 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fae5c01a070 con 0x7fae700c4dc0 2024-03-28T19:03:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.252+0000 7fae67400640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae5006bc00 0x7fae5006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.252+0000 7fae67400640 1 -- 172.21.15.27:0/2997960067 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7fae5c08b410 con 0x7fae700c4dc0 2024-03-28T19:03:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.252+0000 7fae6ea00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae5006bc00 0x7fae5006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.253+0000 7fae6ea00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae5006bc00 0x7fae5006e0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fae60005880 tx=0x7fae60009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.258+0000 7fae67400640 1 -- 172.21.15.27:0/2997960067 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fae5c0591f0 con 0x7fae700c4dc0 2024-03-28T19:03:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.513+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae700c2860 con 0x7fae700c4dc0 2024-03-28T19:03:56.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.514+0000 7fae67400640 1 -- 172.21.15.27:0/2997960067 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5457 (secure 0 0 0) 0x7fae5c05f030 con 0x7fae700c4dc0 2024-03-28T19:03:56.515 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:56.515 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:54.747851+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:56.515 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 72 2024-03-28T19:03:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae5006bc00 msgr2=0x7fae5006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae5006bc00 0x7fae5006e0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fae60005880 tx=0x7fae60009040 comp rx=0 tx=0).stop 2024-03-28T19:03:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae700c4dc0 msgr2=0x7fae70145fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae700c4dc0 0x7fae70145fc0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e9b0 tx=0x7fae5c00ee70 comp rx=0 tx=0).stop 2024-03-28T19:03:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 shutdown_connections 2024-03-28T19:03:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fae5006bc00 0x7fae5006e0c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fae70152bd0 0x7fae70146500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae700c4dc0 0x7fae70145fc0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 --2- 172.21.15.27:0/2997960067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae70070a80 0x7fae70158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 >> 172.21.15.27:0/2997960067 conn(0x7fae70066dd0 msgr2=0x7fae70153230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 shutdown_connections 2024-03-28T19:03:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.517+0000 7fae75e00640 1 -- 172.21.15.27:0/2997960067 wait complete. 2024-03-28T19:03:56.538 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-03-28T19:03:56.538 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:03:56.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.773+0000 7f9b4a400640 1 Processor -- start 2024-03-28T19:03:56.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.773+0000 7f9b4a400640 1 -- start start 2024-03-28T19:03:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.775+0000 7f9b4a400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9b440a82d0 0x7f9b44153280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.775+0000 7f9b4a400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b44070820 0x7f9b440aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.775+0000 7f9b4a400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b441576f0 con 0x7f9b440aaae0 2024-03-28T19:03:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.775+0000 7f9b4a400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b44156870 con 0x7f9b44070820 2024-03-28T19:03:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.775+0000 7f9b4a400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b440a7f60 con 0x7f9b440a82d0 2024-03-28T19:03:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b49000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b440aaae0 0x7f9b44156fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51422/0 (socket says 172.21.15.27:51422) 2024-03-28T19:03:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b49000640 1 -- 172.21.15.27:0/4001847331 learned_addr learned my addr 172.21.15.27:0/4001847331 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b49a00640 1 --2- 172.21.15.27:0/4001847331 >> v2:172.21.15.27:3300/0 conn(0x7f9b44070820 0x7f9b440aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b3be00640 1 --2- 172.21.15.27:0/4001847331 >> v2:172.21.15.155:3300/0 conn(0x7f9b440a82d0 0x7f9b44153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b49a00640 1 -- 172.21.15.27:0/4001847331 >> v2:172.21.15.155:3300/0 conn(0x7f9b440a82d0 msgr2=0x7f9b44153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b49a00640 1 --2- 172.21.15.27:0/4001847331 >> v2:172.21.15.155:3300/0 conn(0x7f9b440a82d0 0x7f9b44153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b43e00640 1 -- 172.21.15.27:0/4001847331 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486085503 0 0) 0x7f9b441576f0 con 0x7f9b440aaae0 2024-03-28T19:03:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b49a00640 1 -- 172.21.15.27:0/4001847331 >> v1:172.21.15.27:6789/0 conn(0x7f9b440aaae0 legacy=0x7f9b44156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b49a00640 1 -- 172.21.15.27:0/4001847331 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440a7c80 con 0x7f9b44070820 2024-03-28T19:03:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.776+0000 7f9b3be00640 1 --2- 172.21.15.27:0/4001847331 >> v2:172.21.15.155:3300/0 conn(0x7f9b440a82d0 0x7f9b44153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:03:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.777+0000 7f9b49a00640 1 --2- 172.21.15.27:0/4001847331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b44070820 0x7f9b440aaec0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9b44070250 tx=0x7f9b3c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f00e4cd6b7f2b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b43e00640 1 -- 172.21.15.27:0/4001847331 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0315a0 con 0x7f9b44070820 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b43e00640 1 -- 172.21.15.27:0/4001847331 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c037070 con 0x7f9b44070820 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b43e00640 1 -- 172.21.15.27:0/4001847331 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c031960 con 0x7f9b44070820 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b4a400640 1 -- 172.21.15.27:0/4001847331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b44070820 msgr2=0x7f9b440aaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b4a400640 1 --2- 172.21.15.27:0/4001847331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b44070820 0x7f9b440aaec0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9b44070250 tx=0x7f9b3c02b840 comp rx=0 tx=0).stop 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b4a400640 1 -- 172.21.15.27:0/4001847331 shutdown_connections 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b4a400640 1 --2- 172.21.15.27:0/4001847331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b44070820 0x7f9b440aaec0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b4a400640 1 --2- 172.21.15.27:0/4001847331 >> v2:172.21.15.155:3300/0 conn(0x7f9b440a82d0 0x7f9b44153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b4a400640 1 -- 172.21.15.27:0/4001847331 >> 172.21.15.27:0/4001847331 conn(0x7f9b44072bd0 msgr2=0x7f9b44072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b4a400640 1 -- 172.21.15.27:0/4001847331 shutdown_connections 2024-03-28T19:03:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.778+0000 7f9b4a400640 1 -- 172.21.15.27:0/4001847331 wait complete. 2024-03-28T19:03:56.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.780+0000 7f9b4a400640 1 Processor -- start 2024-03-28T19:03:56.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.780+0000 7f9b4a400640 1 -- start start 2024-03-28T19:03:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.781+0000 7f9b4a400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b44070820 0x7f9b4406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.781+0000 7f9b4a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b440a82d0 0x7f9b441521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.781+0000 7f9b4a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b440aaae0 0x7f9b440ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.781+0000 7f9b4a400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44156ac0 con 0x7f9b440a82d0 2024-03-28T19:03:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.781+0000 7f9b4a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440c97d0 con 0x7f9b440aaae0 2024-03-28T19:03:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.781+0000 7f9b4a400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440c6460 con 0x7f9b44070820 2024-03-28T19:03:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.781+0000 7f9b3be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b440a82d0 0x7f9b441521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b3be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b440a82d0 0x7f9b441521d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38724/0 (socket says 172.21.15.27:38724) 2024-03-28T19:03:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b3be00640 1 -- 172.21.15.27:0/3969247542 learned_addr learned my addr 172.21.15.27:0/3969247542 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b49a00640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b440aaae0 0x7f9b440ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b49000640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b44070820 0x7f9b4406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b3be00640 1 -- 172.21.15.27:0/3969247542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b44070820 msgr2=0x7f9b4406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b3be00640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b44070820 0x7f9b4406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b3be00640 1 -- 172.21.15.27:0/3969247542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b440aaae0 msgr2=0x7f9b440ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b3be00640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b440aaae0 0x7f9b440ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b3be00640 1 -- 172.21.15.27:0/3969247542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4406bb00 con 0x7f9b440a82d0 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.782+0000 7f9b3be00640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b440a82d0 0x7f9b441521d0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00e810 tx=0x7f9b2c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.783+0000 7f9b41600640 1 -- 172.21.15.27:0/3969247542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c017070 con 0x7f9b440a82d0 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.783+0000 7f9b41600640 1 -- 172.21.15.27:0/3969247542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c00c6d0 con 0x7f9b440a82d0 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.783+0000 7f9b41600640 1 -- 172.21.15.27:0/3969247542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c00f040 con 0x7f9b440a82d0 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.783+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4406bde0 con 0x7f9b440a82d0 2024-03-28T19:03:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.783+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4406e0a0 con 0x7f9b440a82d0 2024-03-28T19:03:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.784+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b44069640 con 0x7f9b440a82d0 2024-03-28T19:03:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.785+0000 7f9b41600640 1 -- 172.21.15.27:0/3969247542 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9b2c01a070 con 0x7f9b440a82d0 2024-03-28T19:03:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.786+0000 7f9b41600640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b2006bbd0 0x7f9b2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.786+0000 7f9b41600640 1 -- 172.21.15.27:0/3969247542 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f9b2c08ad10 con 0x7f9b440a82d0 2024-03-28T19:03:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.786+0000 7f9b49000640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b2006bbd0 0x7f9b2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.787+0000 7f9b49000640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b2006bbd0 0x7f9b2006e090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9b340012e0 tx=0x7f9b34007ef0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:56.795+0000 7f9b41600640 1 -- 172.21.15.27:0/3969247542 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9b2c059140 con 0x7f9b440a82d0 2024-03-28T19:03:57.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.066+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b440c7680 con 0x7f9b440a82d0 2024-03-28T19:03:57.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.067+0000 7f9b41600640 1 -- 172.21.15.27:0/3969247542 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f9b2c058a40 con 0x7f9b440a82d0 2024-03-28T19:03:57.068 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:57.068 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:54.747851+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:57.068 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 72 2024-03-28T19:03:57.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b2006bbd0 msgr2=0x7f9b2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:57.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b2006bbd0 0x7f9b2006e090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9b340012e0 tx=0x7f9b34007ef0 comp rx=0 tx=0).stop 2024-03-28T19:03:57.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b440a82d0 msgr2=0x7f9b441521d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:57.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b440a82d0 0x7f9b441521d0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00e810 tx=0x7f9b2c00ecd0 comp rx=0 tx=0).stop 2024-03-28T19:03:57.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 shutdown_connections 2024-03-28T19:03:57.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b2006bbd0 0x7f9b2006e090 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:57.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b440aaae0 0x7f9b440ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:57.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b440a82d0 0x7f9b441521d0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:57.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 --2- 172.21.15.27:0/3969247542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b44070820 0x7f9b4406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:57.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 >> 172.21.15.27:0/3969247542 conn(0x7f9b44072bd0 msgr2=0x7f9b44072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:57.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 shutdown_connections 2024-03-28T19:03:57.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:57.070+0000 7f9b4a400640 1 -- 172.21.15.27:0/3969247542 wait complete. 2024-03-28T19:03:57.092 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 72, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 72, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5201": { "addr": "172.21.15.155:6835/331868181", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 331868181, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 331868181, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5201, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:03:54.747851+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5201 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5769": { "addr": "172.21.15.27:6836/1001046540", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1001046540, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 1001046540, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5769, "incarnation": 61, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:02:41.197840+0000", "qdb_cluster": [ 5769 ], "qdb_leader": 5769, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5769 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:03:59.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:03:59.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.336+0000 7fa809000640 1 Processor -- start 2024-03-28T19:03:59.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.336+0000 7fa809000640 1 -- start start 2024-03-28T19:03:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.338+0000 7fa809000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa804150c70 0x7fa8040a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.338+0000 7fa809000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa8041534c0 0x7fa804153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.338+0000 7fa809000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8040c3af0 con 0x7fa8040c3690 2024-03-28T19:03:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.338+0000 7fa809000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8040ab4e0 con 0x7fa804150c70 2024-03-28T19:03:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.338+0000 7fa809000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8040ac1f0 con 0x7fa8041534c0 2024-03-28T19:03:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.338+0000 7fa801400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa8040c3690 0x7fa8040c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51438/0 (socket says 172.21.15.27:51438) 2024-03-28T19:03:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.338+0000 7fa801400640 1 -- 172.21.15.27:0/937442448 learned_addr learned my addr 172.21.15.27:0/937442448 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:03:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.338+0000 7fa801e00640 1 --2- 172.21.15.27:0/937442448 >> v2:172.21.15.27:3300/0 conn(0x7fa804150c70 0x7fa8040a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.339+0000 7fa802800640 1 --2- 172.21.15.27:0/937442448 >> v2:172.21.15.155:3300/0 conn(0x7fa8041534c0 0x7fa804153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.339+0000 7fa801e00640 1 -- 172.21.15.27:0/937442448 >> v2:172.21.15.155:3300/0 conn(0x7fa8041534c0 msgr2=0x7fa804153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:03:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.339+0000 7fa801e00640 1 --2- 172.21.15.27:0/937442448 >> v2:172.21.15.155:3300/0 conn(0x7fa8041534c0 0x7fa804153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.339+0000 7fa800a00640 1 -- 172.21.15.27:0/937442448 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236724990 0 0) 0x7fa8040c3af0 con 0x7fa8040c3690 2024-03-28T19:03:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.339+0000 7fa801e00640 1 -- 172.21.15.27:0/937442448 >> v1:172.21.15.27:6789/0 conn(0x7fa8040c3690 legacy=0x7fa8040c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.339+0000 7fa801e00640 1 -- 172.21.15.27:0/937442448 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8040c3d90 con 0x7fa804150c70 2024-03-28T19:03:59.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.341+0000 7fa801e00640 1 --2- 172.21.15.27:0/937442448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa804150c70 0x7fa8040a9cb0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa8040c4610 tx=0x7fa7f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721202113045092a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:59.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.341+0000 7fa800a00640 1 -- 172.21.15.27:0/937442448 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40315a0 con 0x7fa804150c70 2024-03-28T19:03:59.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.341+0000 7fa800a00640 1 -- 172.21.15.27:0/937442448 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f4037070 con 0x7fa804150c70 2024-03-28T19:03:59.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.341+0000 7fa800a00640 1 -- 172.21.15.27:0/937442448 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4031960 con 0x7fa804150c70 2024-03-28T19:03:59.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.342+0000 7fa809000640 1 -- 172.21.15.27:0/937442448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa804150c70 msgr2=0x7fa8040a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.342+0000 7fa809000640 1 --2- 172.21.15.27:0/937442448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa804150c70 0x7fa8040a9cb0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa8040c4610 tx=0x7fa7f402b840 comp rx=0 tx=0).stop 2024-03-28T19:03:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.342+0000 7fa809000640 1 -- 172.21.15.27:0/937442448 shutdown_connections 2024-03-28T19:03:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.342+0000 7fa809000640 1 --2- 172.21.15.27:0/937442448 >> v2:172.21.15.155:3300/0 conn(0x7fa8041534c0 0x7fa804153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.342+0000 7fa809000640 1 --2- 172.21.15.27:0/937442448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa804150c70 0x7fa8040a9cb0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.342+0000 7fa809000640 1 -- 172.21.15.27:0/937442448 >> 172.21.15.27:0/937442448 conn(0x7fa80406b9d0 msgr2=0x7fa8040c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.342+0000 7fa809000640 1 -- 172.21.15.27:0/937442448 shutdown_connections 2024-03-28T19:03:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.342+0000 7fa809000640 1 -- 172.21.15.27:0/937442448 wait complete. 2024-03-28T19:03:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.344+0000 7fa809000640 1 Processor -- start 2024-03-28T19:03:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.345+0000 7fa809000640 1 -- start start 2024-03-28T19:03:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa809000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8040c3690 0x7fa8040c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa809000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa804150c70 0x7fa8041eac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa809000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8041534c0 0x7fa8041eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa809000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040abcd0 con 0x7fa8040c3690 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa809000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040ab9f0 con 0x7fa8041534c0 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa809000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8040c1810 con 0x7fa804150c70 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa801e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8040c3690 0x7fa8040c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa801e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8040c3690 0x7fa8040c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38746/0 (socket says 172.21.15.27:38746) 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.346+0000 7fa801e00640 1 -- 172.21.15.27:0/820051363 learned_addr learned my addr 172.21.15.27:0/820051363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa802800640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8041534c0 0x7fa8041eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa801400640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa804150c70 0x7fa8041eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa801e00640 1 -- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa804150c70 msgr2=0x7fa8041eac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa801e00640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa804150c70 0x7fa8041eac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa801e00640 1 -- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8041534c0 msgr2=0x7fa8041eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa801e00640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8041534c0 0x7fa8041eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa801e00640 1 -- 172.21.15.27:0/820051363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8041ef110 con 0x7fa8040c3690 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa801400640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa804150c70 0x7fa8041eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.347+0000 7fa801e00640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8040c3690 0x7fa8040c02a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009890 tx=0x7fa7f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.348+0000 7fa7fa000640 1 -- 172.21.15.27:0/820051363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4032630 con 0x7fa8040c3690 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.348+0000 7fa7fa000640 1 -- 172.21.15.27:0/820051363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f40327f0 con 0x7fa8040c3690 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.348+0000 7fa7fa000640 1 -- 172.21.15.27:0/820051363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4032be0 con 0x7fa8040c3690 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.348+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80406e120 con 0x7fa8040c3690 2024-03-28T19:03:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.348+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80406e6e0 con 0x7fa8040c3690 2024-03-28T19:03:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.350+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80404f400 con 0x7fa8040c3690 2024-03-28T19:03:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.350+0000 7fa7fa000640 1 -- 172.21.15.27:0/820051363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa7f404c020 con 0x7fa8040c3690 2024-03-28T19:03:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.352+0000 7fa7fa000640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7e406bbd0 0x7fa7e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:03:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.352+0000 7fa7fa000640 1 -- 172.21.15.27:0/820051363 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7fa7f402f020 con 0x7fa8040c3690 2024-03-28T19:03:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.352+0000 7fa801400640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7e406bbd0 0x7fa7e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:03:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.353+0000 7fa801400640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7e406bbd0 0x7fa7e406e090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa804068360 tx=0x7fa7f0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:03:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.357+0000 7fa7fa000640 1 -- 172.21.15.27:0/820051363 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa8041ef110 con 0x7fa8040c3690 2024-03-28T19:03:59.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.691+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa804150460 con 0x7fa8040c3690 2024-03-28T19:03:59.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.692+0000 7fa7fa000640 1 -- 172.21.15.27:0/820051363 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5465 (secure 0 0 0) 0x7fa7f4040070 con 0x7fa8040c3690 2024-03-28T19:03:59.693 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:03:59.693 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:03:59.693 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 75 2024-03-28T19:03:59.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7e406bbd0 msgr2=0x7fa7e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7e406bbd0 0x7fa7e406e090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa804068360 tx=0x7fa7f0009040 comp rx=0 tx=0).stop 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8040c3690 msgr2=0x7fa8040c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8040c3690 0x7fa8040c02a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009890 tx=0x7fa7f40046c0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 shutdown_connections 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa7e406bbd0 0x7fa7e406e090 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8041534c0 0x7fa8041eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa804150c70 0x7fa8041eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 --2- 172.21.15.27:0/820051363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8040c3690 0x7fa8040c02a0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.695+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 >> 172.21.15.27:0/820051363 conn(0x7fa80406b9d0 msgr2=0x7fa80414ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:03:59.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.696+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 shutdown_connections 2024-03-28T19:03:59.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:03:59.696+0000 7fa809000640 1 -- 172.21.15.27:0/820051363 wait complete. 2024-03-28T19:03:59.716 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=5862 2024-03-28T19:03:59.716 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.c 2024-03-28T19:03:59.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:03:59.747 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:04:02.153 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:04:02.153+0000 7f9c08a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:04:04.717 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-03-28T19:04:04.717 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:04:04.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:04:04.727 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:04:04.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:04.912 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:04:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.954+0000 7f3e16200640 1 Processor -- start 2024-03-28T19:04:04.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.954+0000 7f3e16200640 1 -- start start 2024-03-28T19:04:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e16200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e10152f80 0x7f3e100c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e16200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7df0 0x7f3e10155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e16200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e100c7310 con 0x7f3e100c7970 2024-03-28T19:04:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e16200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e10158d40 con 0x7f3e10152f80 2024-03-28T19:04:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e16200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e10159020 con 0x7f3e100c7df0 2024-03-28T19:04:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e0f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e10152f80 0x7f3e100c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e0f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e10152f80 0x7f3e100c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38784/0 (socket says 172.21.15.27:38784) 2024-03-28T19:04:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e0f400640 1 -- 172.21.15.27:0/117431892 learned_addr learned my addr 172.21.15.27:0/117431892 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.956+0000 7f3e0fe00640 1 --2- 172.21.15.27:0/117431892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7df0 0x7f3e10155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.957+0000 7f3e0f400640 1 -- 172.21.15.27:0/117431892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7df0 msgr2=0x7f3e10155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.957+0000 7f3e0f400640 1 --2- 172.21.15.27:0/117431892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7df0 0x7f3e10155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.957+0000 7f3e0f400640 1 -- 172.21.15.27:0/117431892 >> v1:172.21.15.155:6789/0 conn(0x7f3e100c7970 legacy=0x7f3e10155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.957+0000 7f3e0f400640 1 -- 172.21.15.27:0/117431892 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10159300 con 0x7f3e10152f80 2024-03-28T19:04:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.958+0000 7f3e0f400640 1 --2- 172.21.15.27:0/117431892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e10152f80 0x7f3e100c58e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009870 tx=0x7f3e0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c5ed14f389953d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.958+0000 7f3e0e000640 1 -- 172.21.15.27:0/117431892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00035020 con 0x7f3e10152f80 2024-03-28T19:04:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.958+0000 7f3e0e000640 1 -- 172.21.15.27:0/117431892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e00031640 con 0x7f3e10152f80 2024-03-28T19:04:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e0e000640 1 -- 172.21.15.27:0/117431892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0002d040 con 0x7f3e10152f80 2024-03-28T19:04:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e16200640 1 -- 172.21.15.27:0/117431892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e10152f80 msgr2=0x7f3e100c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e16200640 1 --2- 172.21.15.27:0/117431892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e10152f80 0x7f3e100c58e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009870 tx=0x7f3e0002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:04:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e16200640 1 -- 172.21.15.27:0/117431892 shutdown_connections 2024-03-28T19:04:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e16200640 1 --2- 172.21.15.27:0/117431892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7df0 0x7f3e10155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e16200640 1 --2- 172.21.15.27:0/117431892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e10152f80 0x7f3e100c58e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e16200640 1 -- 172.21.15.27:0/117431892 >> 172.21.15.27:0/117431892 conn(0x7f3e1006b6f0 msgr2=0x7f3e100caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e16200640 1 -- 172.21.15.27:0/117431892 shutdown_connections 2024-03-28T19:04:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.959+0000 7f3e16200640 1 -- 172.21.15.27:0/117431892 wait complete. 2024-03-28T19:04:04.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.961+0000 7f3e16200640 1 Processor -- start 2024-03-28T19:04:04.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.961+0000 7f3e16200640 1 -- start start 2024-03-28T19:04:04.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.962+0000 7f3e16200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7970 0x7f3e100c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:04.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.962+0000 7f3e16200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e100c7df0 0x7f3e101517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:04.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.962+0000 7f3e16200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e10152f80 0x7f3e101f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:04.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e16200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e10159b00 con 0x7f3e100c7df0 2024-03-28T19:04:04.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e16200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e100c5e20 con 0x7f3e10152f80 2024-03-28T19:04:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e16200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e100cbca0 con 0x7f3e100c7970 2024-03-28T19:04:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e100c7df0 0x7f3e101517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e100c7df0 0x7f3e101517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38794/0 (socket says 172.21.15.27:38794) 2024-03-28T19:04:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0ea00640 1 -- 172.21.15.27:0/326754631 learned_addr learned my addr 172.21.15.27:0/326754631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:04.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0fe00640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e10152f80 0x7f3e101f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0f400640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7970 0x7f3e100c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0ea00640 1 -- 172.21.15.27:0/326754631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7970 msgr2=0x7f3e100c4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0ea00640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7970 0x7f3e100c4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0ea00640 1 -- 172.21.15.27:0/326754631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e10152f80 msgr2=0x7f3e101f00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0ea00640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e10152f80 0x7f3e101f00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.963+0000 7f3e0ea00640 1 -- 172.21.15.27:0/326754631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e101527d0 con 0x7f3e100c7df0 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.964+0000 7f3e0ea00640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e100c7df0 0x7f3e101517f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00c3f0 tx=0x7f3dfc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.964+0000 7f3e07400640 1 -- 172.21.15.27:0/326754631 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0052d0 con 0x7f3e100c7df0 2024-03-28T19:04:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.964+0000 7f3e07400640 1 -- 172.21.15.27:0/326754631 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc005490 con 0x7f3e100c7df0 2024-03-28T19:04:04.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.964+0000 7f3e07400640 1 -- 172.21.15.27:0/326754631 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc007260 con 0x7f3e100c7df0 2024-03-28T19:04:04.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.964+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e100c2c10 con 0x7f3e100c7df0 2024-03-28T19:04:04.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.964+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e101f3be0 con 0x7f3e100c7df0 2024-03-28T19:04:04.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.966+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e101f3de0 con 0x7f3e100c7df0 2024-03-28T19:04:04.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.968+0000 7f3e07400640 1 -- 172.21.15.27:0/326754631 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3dfc007400 con 0x7f3e100c7df0 2024-03-28T19:04:04.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.969+0000 7f3e07400640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3df006bbd0 0x7f3df006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:04.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.969+0000 7f3e07400640 1 -- 172.21.15.27:0/326754631 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f3dfc08a5e0 con 0x7f3e100c7df0 2024-03-28T19:04:04.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.970+0000 7f3e0f400640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3df006bbd0 0x7f3df006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:04.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.971+0000 7f3e0f400640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3df006bbd0 0x7f3df006e090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3e00036000 tx=0x7f3e0003c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:04.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:04.976+0000 7f3e07400640 1 -- 172.21.15.27:0/326754631 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3e101f3de0 con 0x7f3e100c7df0 2024-03-28T19:04:05.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.282+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e1004f400 con 0x7f3e100c7df0 2024-03-28T19:04:05.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.284+0000 7f3e07400640 1 -- 172.21.15.27:0/326754631 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5465 (secure 0 0 0) 0x7f3dfc058340 con 0x7f3e100c7df0 2024-03-28T19:04:05.284 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:05.285 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 75 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3df006bbd0 msgr2=0x7f3df006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3df006bbd0 0x7f3df006e090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3e00036000 tx=0x7f3e0003c040 comp rx=0 tx=0).stop 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e100c7df0 msgr2=0x7f3e101517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e100c7df0 0x7f3e101517f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00c3f0 tx=0x7f3dfc00c8b0 comp rx=0 tx=0).stop 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 shutdown_connections 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3df006bbd0 0x7f3df006e090 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e10152f80 0x7f3e101f00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e100c7df0 0x7f3e101517f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 --2- 172.21.15.27:0/326754631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3e100c7970 0x7f3e100c4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 >> 172.21.15.27:0/326754631 conn(0x7f3e1006b6f0 msgr2=0x7f3e100c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.287+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 shutdown_connections 2024-03-28T19:04:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:05.288+0000 7f3e16200640 1 -- 172.21.15.27:0/326754631 wait complete. 2024-03-28T19:04:05.307 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:04:07.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:07.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.541+0000 7f5acca00640 1 Processor -- start 2024-03-28T19:04:07.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.541+0000 7f5acca00640 1 -- start start 2024-03-28T19:04:07.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.542+0000 7f5acca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5ac81551a0 0x7f5ac80c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.543+0000 7f5acca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ac80c5580 0x7f5ac80c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.543+0000 7f5acca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80c26d0 con 0x7f5ac80aada0 2024-03-28T19:04:07.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.543+0000 7f5acca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac8156870 con 0x7f5ac81551a0 2024-03-28T19:04:07.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.543+0000 7f5acca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8155cf0 con 0x7f5ac80c5580 2024-03-28T19:04:07.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.543+0000 7f5ac7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5ac80aada0 0x7f5ac80c1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59476/0 (socket says 172.21.15.27:59476) 2024-03-28T19:04:07.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.543+0000 7f5ac7e00640 1 -- 172.21.15.27:0/2069851942 learned_addr learned my addr 172.21.15.27:0/2069851942 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:07.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.543+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500960689 0 0) 0x7f5ac80c26d0 con 0x7f5ac80aada0 2024-03-28T19:04:07.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac7400640 1 --2- 172.21.15.27:0/2069851942 >> v2:172.21.15.155:3300/0 conn(0x7f5ac81551a0 0x7f5ac80c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aac003520 con 0x7f5ac80aada0 2024-03-28T19:04:07.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6a00640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ac80c5580 0x7f5ac80c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126462204 0 0) 0x7f5aac003520 con 0x7f5ac80aada0 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ac80c5580 msgr2=0x7f5ac80c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ac80c5580 0x7f5ac80c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 >> v2:172.21.15.155:3300/0 conn(0x7f5ac81551a0 msgr2=0x7f5ac80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 --2- 172.21.15.27:0/2069851942 >> v2:172.21.15.155:3300/0 conn(0x7f5ac81551a0 0x7f5ac80c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac8155fd0 con 0x7f5ac80aada0 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f5ac00019f0 con 0x7f5ac80aada0 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 >> v1:172.21.15.27:6789/0 conn(0x7f5ac80aada0 legacy=0x7f5ac80c1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5aac0039b0 0x7f5aac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aac004800 0x7f5aac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ac80c5580 0x7f5aac0052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8156870 con 0x7f5aac0039b0 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8155cf0 con 0x7f5aac004800 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.544+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac010050 con 0x7f5ac80c5580 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.545+0000 7f5ac7400640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5aac0039b0 0x7f5aac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.545+0000 7f5ac7e00640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ac80c5580 0x7f5aac0052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.545+0000 7f5ac6a00640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aac004800 0x7f5aac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.545+0000 7f5ac7400640 1 -- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ac80c5580 msgr2=0x7f5aac0052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.545+0000 7f5ac7400640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ac80c5580 0x7f5aac0052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.545+0000 7f5ac7400640 1 -- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aac004800 msgr2=0x7f5aac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.545+0000 7f5ac7400640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aac004800 0x7f5aac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.545+0000 7f5ac7400640 1 -- 172.21.15.27:0/2069851942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac003520 con 0x7f5aac0039b0 2024-03-28T19:04:07.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.546+0000 7f5ac7400640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5aac0039b0 0x7f5aac003dd0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ba30 tx=0x7f5aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b133afc0abe4ddb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:07.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.547+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab800b600 con 0x7f5aac0039b0 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.547+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab8038070 con 0x7f5aac0039b0 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.547+0000 7f5ac6000640 1 -- 172.21.15.27:0/2069851942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab802d040 con 0x7f5aac0039b0 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.548+0000 7f5acca00640 1 -- 172.21.15.27:0/2069851942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5aac0039b0 msgr2=0x7f5aac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.548+0000 7f5acca00640 1 --2- 172.21.15.27:0/2069851942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5aac0039b0 0x7f5aac003dd0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ba30 tx=0x7f5aac005830 comp rx=0 tx=0).stop 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.548+0000 7f5ac7e00640 1 -- 172.21.15.27:0/2069851942 reap_dead start 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.548+0000 7f5acca00640 1 -- 172.21.15.27:0/2069851942 shutdown_connections 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.548+0000 7f5acca00640 1 -- 172.21.15.27:0/2069851942 >> 172.21.15.27:0/2069851942 conn(0x7f5ac806b0c0 msgr2=0x7f5ac806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.548+0000 7f5acca00640 1 -- 172.21.15.27:0/2069851942 shutdown_connections 2024-03-28T19:04:07.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.548+0000 7f5acca00640 1 -- 172.21.15.27:0/2069851942 wait complete. 2024-03-28T19:04:07.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.550+0000 7f5acca00640 1 Processor -- start 2024-03-28T19:04:07.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.550+0000 7f5acca00640 1 -- start start 2024-03-28T19:04:07.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.551+0000 7f5acca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5aac0039b0 0x7f5ac814eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.551+0000 7f5acca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ac81dd820 0x7f5ac814d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.551+0000 7f5acca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ac8072d70 0x7f5ac81e7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.551+0000 7f5acca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81559a0 con 0x7f5ac81dd820 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.551+0000 7f5acca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8156590 con 0x7f5ac8072d70 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.551+0000 7f5acca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac8155580 con 0x7f5aac0039b0 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ac81dd820 0x7f5ac814d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ac81dd820 0x7f5ac814d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55104/0 (socket says 172.21.15.27:55104) 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac6a00640 1 -- 172.21.15.27:0/4192855995 learned_addr learned my addr 172.21.15.27:0/4192855995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac7400640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5aac0039b0 0x7f5ac814eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac7e00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ac8072d70 0x7f5ac81e7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac6a00640 1 -- 172.21.15.27:0/4192855995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5aac0039b0 msgr2=0x7f5ac814eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac6a00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5aac0039b0 0x7f5ac814eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac6a00640 1 -- 172.21.15.27:0/4192855995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ac8072d70 msgr2=0x7f5ac81e7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac6a00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ac8072d70 0x7f5ac81e7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.552+0000 7f5ac6a00640 1 -- 172.21.15.27:0/4192855995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac81dc560 con 0x7f5ac81dd820 2024-03-28T19:04:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.553+0000 7f5ac6a00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ac81dd820 0x7f5ac814d420 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000f580 tx=0x7f5ab000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.553+0000 7f5abf400640 1 -- 172.21.15.27:0/4192855995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0019020 con 0x7f5ac81dd820 2024-03-28T19:04:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.553+0000 7f5abf400640 1 -- 172.21.15.27:0/4192855995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab0014870 con 0x7f5ac81dd820 2024-03-28T19:04:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.553+0000 7f5abf400640 1 -- 172.21.15.27:0/4192855995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0014c60 con 0x7f5ac81dd820 2024-03-28T19:04:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.553+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac814e0b0 con 0x7f5ac81dd820 2024-03-28T19:04:07.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.553+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac814e670 con 0x7f5ac81dd820 2024-03-28T19:04:07.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.555+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac8067dd0 con 0x7f5ac81dd820 2024-03-28T19:04:07.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.556+0000 7f5abf400640 1 -- 172.21.15.27:0/4192855995 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5ab001c020 con 0x7f5ac81dd820 2024-03-28T19:04:07.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.557+0000 7f5abf400640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5aa406bc00 0x7f5aa406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:07.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.557+0000 7f5abf400640 1 -- 172.21.15.27:0/4192855995 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f5ab008b9a0 con 0x7f5ac81dd820 2024-03-28T19:04:07.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.557+0000 7f5ac7400640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5aa406bc00 0x7f5aa406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:07.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.558+0000 7f5ac7400640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5aa406bc00 0x7f5aa406e0c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ba00 tx=0x7f5ab8030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.566+0000 7f5abf400640 1 -- 172.21.15.27:0/4192855995 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5ab0023020 con 0x7f5ac81dd820 2024-03-28T19:04:07.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.863+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ac8069640 con 0x7f5ac81dd820 2024-03-28T19:04:07.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.864+0000 7f5abf400640 1 -- 172.21.15.27:0/4192855995 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f5ab0078070 con 0x7f5ac81dd820 2024-03-28T19:04:07.863 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:07.863 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:07.864 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:07.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5aa406bc00 msgr2=0x7f5aa406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:07.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5aa406bc00 0x7f5aa406e0c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800ba00 tx=0x7f5ab8030000 comp rx=0 tx=0).stop 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ac81dd820 msgr2=0x7f5ac814d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ac81dd820 0x7f5ac814d420 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5ab000f580 tx=0x7f5ab000fa40 comp rx=0 tx=0).stop 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 shutdown_connections 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5aa406bc00 0x7f5aa406e0c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ac8072d70 0x7f5ac81e7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ac81dd820 0x7f5ac814d420 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 --2- 172.21.15.27:0/4192855995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5aac0039b0 0x7f5ac814eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 >> 172.21.15.27:0/4192855995 conn(0x7f5ac806b0c0 msgr2=0x7f5ac80c61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 shutdown_connections 2024-03-28T19:04:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:07.867+0000 7f5acca00640 1 -- 172.21.15.27:0/4192855995 wait complete. 2024-03-28T19:04:07.891 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-03-28T19:04:07.891 INFO:tasks.mds_thrash.fs.[a]:waiting for 30 secs before thrashing 2024-03-28T19:04:23.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.741+0000 7fe039a00640 1 Processor -- start 2024-03-28T19:04:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.741+0000 7fe039a00640 1 -- start start 2024-03-28T19:04:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.744+0000 7fe039a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe0340aaae0 0x7fe034153280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.744+0000 7fe039a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0341576f0 con 0x7fe034070820 2024-03-28T19:04:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.744+0000 7fe039a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0340a7f60 con 0x7fe0340a82d0 2024-03-28T19:04:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.744+0000 7fe039a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe034156870 con 0x7fe0340aaae0 2024-03-28T19:04:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.744+0000 7fe039000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe034070820 0x7fe03415dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56618/0 (socket says 172.21.15.27:56618) 2024-03-28T19:04:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.744+0000 7fe039000640 1 -- 172.21.15.27:0/1321820024 learned_addr learned my addr 172.21.15.27:0/1321820024 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.744+0000 7fe033e00640 1 --2- 172.21.15.27:0/1321820024 >> v2:172.21.15.155:3300/0 conn(0x7fe0340aaae0 0x7fe034153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219112258 0 0) 0x7fe0341576f0 con 0x7fe034070820 2024-03-28T19:04:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe018003520 con 0x7fe034070820 2024-03-28T19:04:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381708134 0 0) 0x7fe018003520 con 0x7fe034070820 2024-03-28T19:04:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 >> v2:172.21.15.155:3300/0 conn(0x7fe0340aaae0 msgr2=0x7fe034153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 --2- 172.21.15.27:0/1321820024 >> v2:172.21.15.155:3300/0 conn(0x7fe0340aaae0 0x7fe034153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 >> v1:172.21.15.155:6789/0 conn(0x7fe0340a82d0 legacy=0x7fe034156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0340a7c80 con 0x7fe034070820 2024-03-28T19:04:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe02c0019f0 con 0x7fe034070820 2024-03-28T19:04:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe033e00640 1 --2- 172.21.15.27:0/1321820024 >> v2:172.21.15.155:3300/0 conn(0x7fe0340aaae0 0x7fe034153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:04:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 >> v1:172.21.15.27:6789/0 conn(0x7fe034070820 legacy=0x7fe03415dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180039b0 0x7fe018003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe018004800 0x7fe018004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.745+0000 7fe032a00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0180051c0 0x7fe01800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe034156870 con 0x7fe0180039b0 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01800c160 con 0x7fe0180051c0 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01800c360 con 0x7fe018004800 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe033e00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180039b0 0x7fe018003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe033400640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe018004800 0x7fe018004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe039000640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0180051c0 0x7fe01800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe033e00640 1 -- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe018004800 msgr2=0x7fe018004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe033e00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe018004800 0x7fe018004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe033e00640 1 -- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0180051c0 msgr2=0x7fe01800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe033e00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0180051c0 0x7fe01800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe033e00640 1 -- 172.21.15.27:0/1321820024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe018003520 con 0x7fe0180039b0 2024-03-28T19:04:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe039000640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0180051c0 0x7fe01800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.746+0000 7fe039000640 1 -- 172.21.15.27:0/1321820024 reap_dead start 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.748+0000 7fe033e00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180039b0 0x7fe018003dd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe024009c90 tx=0x7fe018005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7908c4147b93c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.748+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024039070 con 0x7fe0180039b0 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.748+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02402c8f0 con 0x7fe0180039b0 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.748+0000 7fe032a00640 1 -- 172.21.15.27:0/1321820024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024009860 con 0x7fe0180039b0 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.749+0000 7fe039a00640 1 -- 172.21.15.27:0/1321820024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180039b0 msgr2=0x7fe018003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.749+0000 7fe039a00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180039b0 0x7fe018003dd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe024009c90 tx=0x7fe018005830 comp rx=0 tx=0).stop 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.749+0000 7fe039a00640 1 -- 172.21.15.27:0/1321820024 shutdown_connections 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.749+0000 7fe039a00640 1 --2- 172.21.15.27:0/1321820024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180039b0 0x7fe018003dd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.749+0000 7fe039a00640 1 -- 172.21.15.27:0/1321820024 >> 172.21.15.27:0/1321820024 conn(0x7fe034072bd0 msgr2=0x7fe034072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.749+0000 7fe039a00640 1 -- 172.21.15.27:0/1321820024 shutdown_connections 2024-03-28T19:04:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.749+0000 7fe039a00640 1 -- 172.21.15.27:0/1321820024 wait complete. 2024-03-28T19:04:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.752+0000 7fe039a00640 1 Processor -- start 2024-03-28T19:04:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.752+0000 7fe039a00640 1 -- start start 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe039a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe034147ff0 0x7fe03414fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe039a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0340c2d00 0x7fe034140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe039a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0341478a0 0x7fe0341412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe039a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe034156ac0 con 0x7fe034147ff0 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe039a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340c97d0 con 0x7fe0341478a0 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe039a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340c6460 con 0x7fe0340c2d00 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe033e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe034147ff0 0x7fe03414fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe033e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe034147ff0 0x7fe03414fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60048/0 (socket says 172.21.15.27:60048) 2024-03-28T19:04:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe033e00640 1 -- 172.21.15.27:0/1159409337 learned_addr learned my addr 172.21.15.27:0/1159409337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe033400640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0340c2d00 0x7fe034140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.754+0000 7fe039000640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0341478a0 0x7fe0341412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe033e00640 1 -- 172.21.15.27:0/1159409337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0340c2d00 msgr2=0x7fe034140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe033e00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0340c2d00 0x7fe034140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe033e00640 1 -- 172.21.15.27:0/1159409337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0341478a0 msgr2=0x7fe0341412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe033e00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0341478a0 0x7fe0341412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe033e00640 1 -- 172.21.15.27:0/1159409337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe034150090 con 0x7fe034147ff0 2024-03-28T19:04:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe033e00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe034147ff0 0x7fe03414fa30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe024037040 tx=0x7fe024004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe02be00640 1 -- 172.21.15.27:0/1159409337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024039050 con 0x7fe034147ff0 2024-03-28T19:04:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe02be00640 1 -- 172.21.15.27:0/1159409337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02403a040 con 0x7fe034147ff0 2024-03-28T19:04:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe02be00640 1 -- 172.21.15.27:0/1159409337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024009860 con 0x7fe034147ff0 2024-03-28T19:04:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.755+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe034150330 con 0x7fe034147ff0 2024-03-28T19:04:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.756+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe034149f80 con 0x7fe034147ff0 2024-03-28T19:04:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.757+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe034069640 con 0x7fe034147ff0 2024-03-28T19:04:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.759+0000 7fe02be00640 1 -- 172.21.15.27:0/1159409337 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe024047020 con 0x7fe034147ff0 2024-03-28T19:04:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.760+0000 7fe02be00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe01406bbd0 0x7fe01406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.760+0000 7fe02be00640 1 -- 172.21.15.27:0/1159409337 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7fe0240ac920 con 0x7fe034147ff0 2024-03-28T19:04:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.761+0000 7fe033400640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe01406bbd0 0x7fe01406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.762+0000 7fe033400640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe01406bbd0 0x7fe01406e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe034147d30 tx=0x7fe020009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:23.768+0000 7fe02be00640 1 -- 172.21.15.27:0/1159409337 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe02404e020 con 0x7fe034147ff0 2024-03-28T19:04:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.095+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0340a9fe0 con 0x7fe034147ff0 2024-03-28T19:04:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.097+0000 7fe02be00640 1 -- 172.21.15.27:0/1159409337 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7fe02407a820 con 0x7fe034147ff0 2024-03-28T19:04:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:24.097 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:24.097 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe01406bbd0 msgr2=0x7fe01406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe01406bbd0 0x7fe01406e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe034147d30 tx=0x7fe020009040 comp rx=0 tx=0).stop 2024-03-28T19:04:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe034147ff0 msgr2=0x7fe03414fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe034147ff0 0x7fe03414fa30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe024037040 tx=0x7fe024004730 comp rx=0 tx=0).stop 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 shutdown_connections 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe01406bbd0 0x7fe01406e090 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0341478a0 0x7fe0341412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0340c2d00 0x7fe034140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 --2- 172.21.15.27:0/1159409337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe034147ff0 0x7fe03414fa30 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 >> 172.21.15.27:0/1159409337 conn(0x7fe034072bd0 msgr2=0x7fe034072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 shutdown_connections 2024-03-28T19:04:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.100+0000 7fe039a00640 1 -- 172.21.15.27:0/1159409337 wait complete. 2024-03-28T19:04:24.120 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-03-28T19:04:24.120 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:04:24.120 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:04:24.146 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:04:24.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.430+0000 7fbcc8e00640 1 Processor -- start 2024-03-28T19:04:24.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.430+0000 7fbcc8e00640 1 -- start start 2024-03-28T19:04:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.431+0000 7fbcc8e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbcc40a9cb0 0x7fbcc40c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.431+0000 7fbcc8e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc4154020 con 0x7fbcc4150c70 2024-03-28T19:04:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.431+0000 7fbcc8e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc40c3d90 con 0x7fbcc41534c0 2024-03-28T19:04:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.431+0000 7fbcc8e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc40ab4e0 con 0x7fbcc40a9cb0 2024-03-28T19:04:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.432+0000 7fbcc1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbcc4150c70 0x7fbcc40c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56630/0 (socket says 172.21.15.27:56630) 2024-03-28T19:04:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.432+0000 7fbcc1e00640 1 -- 172.21.15.27:0/3856026732 learned_addr learned my addr 172.21.15.27:0/3856026732 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.432+0000 7fbcc1400640 1 --2- 172.21.15.27:0/3856026732 >> v2:172.21.15.155:3300/0 conn(0x7fbcc40a9cb0 0x7fbcc40c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:24.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.432+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317011491 0 0) 0x7fbcc4154020 con 0x7fbcc4150c70 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.432+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca8003520 con 0x7fbcc4150c70 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386098645 0 0) 0x7fbca8003520 con 0x7fbcc4150c70 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 >> v2:172.21.15.155:3300/0 conn(0x7fbcc40a9cb0 msgr2=0x7fbcc40c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 --2- 172.21.15.27:0/3856026732 >> v2:172.21.15.155:3300/0 conn(0x7fbcc40a9cb0 0x7fbcc40c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 >> v1:172.21.15.155:6789/0 conn(0x7fbcc41534c0 legacy=0x7fbcc4153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc40ab710 con 0x7fbcc4150c70 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fbcb4001670 con 0x7fbcc4150c70 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 >> v1:172.21.15.27:6789/0 conn(0x7fbcc4150c70 legacy=0x7fbcc40c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbca8003a00 0x7fbca8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca8004800 0x7fbca8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8005210 0x7fbca800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca800c130 con 0x7fbca8003a00 2024-03-28T19:04:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca800c2e0 con 0x7fbca8004800 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca800c4e0 con 0x7fbca8005210 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.433+0000 7fbcc1e00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbca8003a00 0x7fbca8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc1400640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca8004800 0x7fbca8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc1e00640 1 -- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8005210 msgr2=0x7fbca800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc2800640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8005210 0x7fbca800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc1e00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8005210 0x7fbca800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc1e00640 1 -- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca8004800 msgr2=0x7fbca8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc1e00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca8004800 0x7fbca8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc1e00640 1 -- 172.21.15.27:0/3856026732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8003520 con 0x7fbca8003a00 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc2800640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8005210 0x7fbca800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:24.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.434+0000 7fbcc2800640 1 -- 172.21.15.27:0/3856026732 reap_dead start 2024-03-28T19:04:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.435+0000 7fbcc1e00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbca8003a00 0x7fbca8003e20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c0e0 tx=0x7fbca8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaae449e03807288 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4007c40 con 0x7fbca8003a00 2024-03-28T19:04:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb4030070 con 0x7fbca8003a00 2024-03-28T19:04:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc0a00640 1 -- 172.21.15.27:0/3856026732 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4005d70 con 0x7fbca8003a00 2024-03-28T19:04:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc8e00640 1 -- 172.21.15.27:0/3856026732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbca8003a00 msgr2=0x7fbca8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc8e00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbca8003a00 0x7fbca8003e20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400c0e0 tx=0x7fbca8005830 comp rx=0 tx=0).stop 2024-03-28T19:04:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc8e00640 1 -- 172.21.15.27:0/3856026732 shutdown_connections 2024-03-28T19:04:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc8e00640 1 --2- 172.21.15.27:0/3856026732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbca8003a00 0x7fbca8003e20 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc8e00640 1 -- 172.21.15.27:0/3856026732 >> 172.21.15.27:0/3856026732 conn(0x7fbcc406b9d0 msgr2=0x7fbcc40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc8e00640 1 -- 172.21.15.27:0/3856026732 shutdown_connections 2024-03-28T19:04:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.436+0000 7fbcc8e00640 1 -- 172.21.15.27:0/3856026732 wait complete. 2024-03-28T19:04:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.438+0000 7fbcc8e00640 1 Processor -- start 2024-03-28T19:04:24.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.438+0000 7fbcc8e00640 1 -- start start 2024-03-28T19:04:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.439+0000 7fbcc8e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8003a00 0x7fbcc4071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.439+0000 7fbcc8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcc40a7ec0 0x7fbcc41ea900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.439+0000 7fbcc8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcc414bf00 0x7fbcc41eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.439+0000 7fbcc8e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc40abcd0 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.439+0000 7fbcc8e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc40ac1f0 con 0x7fbcc414bf00 2024-03-28T19:04:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.439+0000 7fbcc8e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc40c1810 con 0x7fbca8003a00 2024-03-28T19:04:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.439+0000 7fbcc1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcc40a7ec0 0x7fbcc41ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:24.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcc40a7ec0 0x7fbcc41ea900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60058/0 (socket says 172.21.15.27:60058) 2024-03-28T19:04:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1400640 1 -- 172.21.15.27:0/2535487709 learned_addr learned my addr 172.21.15.27:0/2535487709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc2800640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcc414bf00 0x7fbcc41eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8003a00 0x7fbcc4071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1400640 1 -- 172.21.15.27:0/2535487709 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8003a00 msgr2=0x7fbcc4071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1400640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8003a00 0x7fbcc4071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1400640 1 -- 172.21.15.27:0/2535487709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcc414bf00 msgr2=0x7fbcc41eae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1400640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcc414bf00 0x7fbcc41eae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1400640 1 -- 172.21.15.27:0/2535487709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc4154a80 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.440+0000 7fbcc1400640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcc40a7ec0 0x7fbcc41ea900 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0011620 tx=0x7fbcb0011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.441+0000 7fbcba000640 1 -- 172.21.15.27:0/2535487709 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb000f920 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.441+0000 7fbcba000640 1 -- 172.21.15.27:0/2535487709 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb000fac0 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.441+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc4154d60 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.441+0000 7fbcba000640 1 -- 172.21.15.27:0/2535487709 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0015460 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.441+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc4072380 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.442+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc404f400 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.443+0000 7fbcba000640 1 -- 172.21.15.27:0/2535487709 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbcb0015600 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.444+0000 7fbcba000640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbca406bc00 0x7fbca406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.445+0000 7fbcba000640 1 -- 172.21.15.27:0/2535487709 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7fbcb008bb20 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.445+0000 7fbcc1e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbca406bc00 0x7fbca406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:24.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.446+0000 7fbcc1e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbca406bc00 0x7fbca406e0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4034000 tx=0x7fbcb4009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:24.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.449+0000 7fbcba000640 1 -- 172.21.15.27:0/2535487709 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbcb0019030 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.727+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcc41ea540 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.729+0000 7fbcba000640 1 -- 172.21.15.27:0/2535487709 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7fbcb0059eb0 con 0x7fbcc40a7ec0 2024-03-28T19:04:24.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:24.728 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:24.728 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbca406bc00 msgr2=0x7fbca406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbca406bc00 0x7fbca406e0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4034000 tx=0x7fbcb4009030 comp rx=0 tx=0).stop 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcc40a7ec0 msgr2=0x7fbcc41ea900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcc40a7ec0 0x7fbcc41ea900 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0011620 tx=0x7fbcb0011ae0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 shutdown_connections 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbca406bc00 0x7fbca406e0c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcc414bf00 0x7fbcc41eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbcc40a7ec0 0x7fbcc41ea900 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 --2- 172.21.15.27:0/2535487709 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbca8003a00 0x7fbcc4071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.731+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 >> 172.21.15.27:0/2535487709 conn(0x7fbcc406b9d0 msgr2=0x7fbcc40c47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:24.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.732+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 shutdown_connections 2024-03-28T19:04:24.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:24.732+0000 7fbcc8e00640 1 -- 172.21.15.27:0/2535487709 wait complete. 2024-03-28T19:04:24.752 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:04:26.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.001+0000 7f1a83e00640 1 Processor -- start 2024-03-28T19:04:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.002+0000 7f1a83e00640 1 -- start start 2024-03-28T19:04:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.003+0000 7f1a83e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a841551a0 0x7f1a840c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.003+0000 7f1a83e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1a840c5580 0x7f1a840c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.003+0000 7f1a83e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a840aada0 0x7f1a840c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.003+0000 7f1a83e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a84156870 con 0x7f1a840aada0 2024-03-28T19:04:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.003+0000 7f1a83e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a84155cf0 con 0x7f1a840c5580 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.003+0000 7f1a83e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84155fd0 con 0x7f1a841551a0 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a83400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a840aada0 0x7f1a840c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a83400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a840aada0 0x7f1a840c1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53376/0 (socket says 172.21.15.27:53376) 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a83400640 1 -- 172.21.15.27:0/1937171363 learned_addr learned my addr 172.21.15.27:0/1937171363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a82a00640 1 --2- 172.21.15.27:0/1937171363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a841551a0 0x7f1a840c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a82000640 1 --2- 172.21.15.27:0/1937171363 >> v2:172.21.15.155:3300/0 conn(0x7f1a840c5580 0x7f1a840c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a83400640 1 -- 172.21.15.27:0/1937171363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a841551a0 msgr2=0x7f1a840c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a83400640 1 --2- 172.21.15.27:0/1937171363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a841551a0 0x7f1a840c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a83400640 1 -- 172.21.15.27:0/1937171363 >> v2:172.21.15.155:3300/0 conn(0x7f1a840c5580 msgr2=0x7f1a840c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a83400640 1 --2- 172.21.15.27:0/1937171363 >> v2:172.21.15.155:3300/0 conn(0x7f1a840c5580 0x7f1a840c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.004+0000 7f1a83400640 1 -- 172.21.15.27:0/1937171363 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a841562b0 con 0x7f1a840aada0 2024-03-28T19:04:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.006+0000 7f1a83400640 1 --2- 172.21.15.27:0/1937171363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c1fc0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1a8406e900 tx=0x7f1a7c02dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8410c6ecc2dd366 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.006+0000 7f1a81600640 1 -- 172.21.15.27:0/1937171363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c02e970 con 0x7f1a840aada0 2024-03-28T19:04:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.006+0000 7f1a81600640 1 -- 172.21.15.27:0/1937171363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7c02eb10 con 0x7f1a840aada0 2024-03-28T19:04:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.006+0000 7f1a81600640 1 -- 172.21.15.27:0/1937171363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c009600 con 0x7f1a840aada0 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.006+0000 7f1a83e00640 1 -- 172.21.15.27:0/1937171363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 msgr2=0x7f1a840c1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.006+0000 7f1a83e00640 1 --2- 172.21.15.27:0/1937171363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c1fc0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1a8406e900 tx=0x7f1a7c02dfd0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.007+0000 7f1a83e00640 1 -- 172.21.15.27:0/1937171363 shutdown_connections 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.007+0000 7f1a83e00640 1 --2- 172.21.15.27:0/1937171363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c1fc0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.007+0000 7f1a83e00640 1 --2- 172.21.15.27:0/1937171363 >> v2:172.21.15.155:3300/0 conn(0x7f1a840c5580 0x7f1a840c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.007+0000 7f1a83e00640 1 --2- 172.21.15.27:0/1937171363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a841551a0 0x7f1a840c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.007+0000 7f1a83e00640 1 -- 172.21.15.27:0/1937171363 >> 172.21.15.27:0/1937171363 conn(0x7f1a8406b0c0 msgr2=0x7f1a8406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.007+0000 7f1a83e00640 1 -- 172.21.15.27:0/1937171363 shutdown_connections 2024-03-28T19:04:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.007+0000 7f1a83e00640 1 -- 172.21.15.27:0/1937171363 wait complete. 2024-03-28T19:04:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.008+0000 7f1a83e00640 1 Processor -- start 2024-03-28T19:04:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.009+0000 7f1a83e00640 1 -- start start 2024-03-28T19:04:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a83e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a83e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a840c5580 0x7f1a8414b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a83e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a841551a0 0x7f1a8414b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a83e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84155580 con 0x7f1a840aada0 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a83e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a841559a0 con 0x7f1a841551a0 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a83e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84155700 con 0x7f1a840c5580 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53380/0 (socket says 172.21.15.27:53380) 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82a00640 1 -- 172.21.15.27:0/2809192420 learned_addr learned my addr 172.21.15.27:0/2809192420 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82000640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a840c5580 0x7f1a8414b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a83400640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a841551a0 0x7f1a8414b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82a00640 1 -- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a840c5580 msgr2=0x7f1a8414b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82a00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a840c5580 0x7f1a8414b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82a00640 1 -- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a841551a0 msgr2=0x7f1a8414b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82a00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a841551a0 0x7f1a8414b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a82a00640 1 -- 172.21.15.27:0/2809192420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a84073140 con 0x7f1a840aada0 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.010+0000 7f1a83400640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a841551a0 0x7f1a8414b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.011+0000 7f1a82a00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c3aa0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400d800 tx=0x7f1a7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.011+0000 7f1a7aa00640 1 -- 172.21.15.27:0/2809192420 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7400b6c0 con 0x7f1a840aada0 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.011+0000 7f1a7aa00640 1 -- 172.21.15.27:0/2809192420 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74019070 con 0x7f1a840aada0 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.011+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a84073420 con 0x7f1a840aada0 2024-03-28T19:04:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.011+0000 7f1a7aa00640 1 -- 172.21.15.27:0/2809192420 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7400ba80 con 0x7f1a840aada0 2024-03-28T19:04:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.011+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a840c3fe0 con 0x7f1a840aada0 2024-03-28T19:04:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.013+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a84067dd0 con 0x7f1a840aada0 2024-03-28T19:04:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.014+0000 7f1a7aa00640 1 -- 172.21.15.27:0/2809192420 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1a7400bc20 con 0x7f1a840aada0 2024-03-28T19:04:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.015+0000 7f1a7aa00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a6406bbd0 0x7f1a6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.015+0000 7f1a7aa00640 1 -- 172.21.15.27:0/2809192420 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f1a7408b640 con 0x7f1a840aada0 2024-03-28T19:04:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.015+0000 7f1a82000640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a6406bbd0 0x7f1a6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.016+0000 7f1a82000640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a6406bbd0 0x7f1a6406e090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fd0 tx=0x7f1a70008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.020+0000 7f1a7aa00640 1 -- 172.21.15.27:0/2809192420 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1a740599d0 con 0x7f1a840aada0 2024-03-28T19:04:27.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.280+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a84069640 con 0x7f1a840aada0 2024-03-28T19:04:27.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.282+0000 7f1a7aa00640 1 -- 172.21.15.27:0/2809192420 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f1a740592d0 con 0x7f1a840aada0 2024-03-28T19:04:27.282 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:27.282 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:27.282 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:27.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a6406bbd0 msgr2=0x7f1a6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a6406bbd0 0x7f1a6406e090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fd0 tx=0x7f1a70008040 comp rx=0 tx=0).stop 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 msgr2=0x7f1a840c3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c3aa0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400d800 tx=0x7f1a7400dcc0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 shutdown_connections 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a6406bbd0 0x7f1a6406e090 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a841551a0 0x7f1a8414b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a840c5580 0x7f1a8414b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 --2- 172.21.15.27:0/2809192420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a840aada0 0x7f1a840c3aa0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 >> 172.21.15.27:0/2809192420 conn(0x7f1a8406b0c0 msgr2=0x7f1a840c1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 shutdown_connections 2024-03-28T19:04:27.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:27.285+0000 7f1a83e00640 1 -- 172.21.15.27:0/2809192420 wait complete. 2024-03-28T19:04:27.306 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:04:29.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:29.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.550+0000 7ff884e00640 1 Processor -- start 2024-03-28T19:04:29.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.550+0000 7ff884e00640 1 -- start start 2024-03-28T19:04:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.551+0000 7ff884e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff8800c5210 0x7ff880150580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.552+0000 7ff884e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8800c4630 0x7ff880157190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.552+0000 7ff884e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8801543f0 con 0x7ff8800c2430 2024-03-28T19:04:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.552+0000 7ff884e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8800c2d50 con 0x7ff8800c4630 2024-03-28T19:04:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.552+0000 7ff884e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff880153630 con 0x7ff8800c5210 2024-03-28T19:04:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.552+0000 7ff87e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8800c4630 0x7ff880157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.552+0000 7ff87e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff8800c4630 0x7ff880157190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53382/0 (socket says 172.21.15.27:53382) 2024-03-28T19:04:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.552+0000 7ff87e800640 1 -- 172.21.15.27:0/3379936465 learned_addr learned my addr 172.21.15.27:0/3379936465 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.552+0000 7ff87de00640 1 --2- 172.21.15.27:0/3379936465 >> v2:172.21.15.155:3300/0 conn(0x7ff8800c5210 0x7ff880150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:29.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.553+0000 7ff87e800640 1 -- 172.21.15.27:0/3379936465 >> v2:172.21.15.155:3300/0 conn(0x7ff8800c5210 msgr2=0x7ff880150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:29.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.553+0000 7ff87e800640 1 --2- 172.21.15.27:0/3379936465 >> v2:172.21.15.155:3300/0 conn(0x7ff8800c5210 0x7ff880150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.553+0000 7ff87e800640 1 -- 172.21.15.27:0/3379936465 >> v1:172.21.15.27:6789/0 conn(0x7ff8800c2430 legacy=0x7ff880156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:29.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.553+0000 7ff87e800640 1 -- 172.21.15.27:0/3379936465 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880153910 con 0x7ff8800c4630 2024-03-28T19:04:29.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.554+0000 7ff87e800640 1 --2- 172.21.15.27:0/3379936465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c4630 0x7ff880157190 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff8800c7f60 tx=0x7ff87802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded281dfa1529fcd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:29.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.554+0000 7ff87ca00640 1 -- 172.21.15.27:0/3379936465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87802cb10 con 0x7ff8800c4630 2024-03-28T19:04:29.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff87ca00640 1 -- 172.21.15.27:0/3379936465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87802ccb0 con 0x7ff8800c4630 2024-03-28T19:04:29.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff87ca00640 1 -- 172.21.15.27:0/3379936465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87802ce60 con 0x7ff8800c4630 2024-03-28T19:04:29.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff884e00640 1 -- 172.21.15.27:0/3379936465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c4630 msgr2=0x7ff880157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:29.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff884e00640 1 --2- 172.21.15.27:0/3379936465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c4630 0x7ff880157190 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff8800c7f60 tx=0x7ff87802bf70 comp rx=0 tx=0).stop 2024-03-28T19:04:29.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff884e00640 1 -- 172.21.15.27:0/3379936465 shutdown_connections 2024-03-28T19:04:29.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff884e00640 1 --2- 172.21.15.27:0/3379936465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c4630 0x7ff880157190 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff884e00640 1 --2- 172.21.15.27:0/3379936465 >> v2:172.21.15.155:3300/0 conn(0x7ff8800c5210 0x7ff880150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff884e00640 1 -- 172.21.15.27:0/3379936465 >> 172.21.15.27:0/3379936465 conn(0x7ff88006c5f0 msgr2=0x7ff88006ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:29.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff884e00640 1 -- 172.21.15.27:0/3379936465 shutdown_connections 2024-03-28T19:04:29.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.555+0000 7ff884e00640 1 -- 172.21.15.27:0/3379936465 wait complete. 2024-03-28T19:04:29.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.557+0000 7ff884e00640 1 Processor -- start 2024-03-28T19:04:29.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.557+0000 7ff884e00640 1 -- start start 2024-03-28T19:04:29.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.558+0000 7ff884e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c2430 0x7ff8800acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:29.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.558+0000 7ff884e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8800c4630 0x7ff8801474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:29.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.558+0000 7ff884e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8800c5210 0x7ff8801458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:29.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.558+0000 7ff884e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800c2a00 con 0x7ff8800c2430 2024-03-28T19:04:29.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.558+0000 7ff884e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff880153ed0 con 0x7ff8800c4630 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff884e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800c55f0 con 0x7ff8800c5210 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c2430 0x7ff8800acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c2430 0x7ff8800acc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53384/0 (socket says 172.21.15.27:53384) 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87de00640 1 -- 172.21.15.27:0/1315645894 learned_addr learned my addr 172.21.15.27:0/1315645894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87d400640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8800c4630 0x7ff8801474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87e800640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8800c5210 0x7ff8801458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87de00640 1 -- 172.21.15.27:0/1315645894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8800c5210 msgr2=0x7ff8801458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87de00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8800c5210 0x7ff8801458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87de00640 1 -- 172.21.15.27:0/1315645894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8800c4630 msgr2=0x7ff8801474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:29.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87de00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8800c4630 0x7ff8801474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.559+0000 7ff87de00640 1 -- 172.21.15.27:0/1315645894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8801ebd70 con 0x7ff8800c2430 2024-03-28T19:04:29.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.560+0000 7ff87de00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c2430 0x7ff8800acc80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff87000d5f0 tx=0x7ff87000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:29.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.560+0000 7ff876000640 1 -- 172.21.15.27:0/1315645894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8700125f0 con 0x7ff8800c2430 2024-03-28T19:04:29.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.560+0000 7ff876000640 1 -- 172.21.15.27:0/1315645894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff870019070 con 0x7ff8800c2430 2024-03-28T19:04:29.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.560+0000 7ff876000640 1 -- 172.21.15.27:0/1315645894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8700129d0 con 0x7ff8800c2430 2024-03-28T19:04:29.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.560+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88006dec0 con 0x7ff8800c2430 2024-03-28T19:04:29.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.560+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8801479f0 con 0x7ff8800c2430 2024-03-28T19:04:29.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.561+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88004f400 con 0x7ff8800c2430 2024-03-28T19:04:29.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.563+0000 7ff876000640 1 -- 172.21.15.27:0/1315645894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff870012b70 con 0x7ff8800c2430 2024-03-28T19:04:29.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.564+0000 7ff876000640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff86006bbd0 0x7ff86006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:29.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.564+0000 7ff876000640 1 -- 172.21.15.27:0/1315645894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7ff87008b5b0 con 0x7ff8800c2430 2024-03-28T19:04:29.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.564+0000 7ff87d400640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff86006bbd0 0x7ff86006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:29.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.565+0000 7ff87d400640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff86006bbd0 0x7ff86006e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff8801498c0 tx=0x7ff86c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:29.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.569+0000 7ff876000640 1 -- 172.21.15.27:0/1315645894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff870059940 con 0x7ff8800c2430 2024-03-28T19:04:29.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.847+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff88014cbb0 con 0x7ff8800c2430 2024-03-28T19:04:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.848+0000 7ff876000640 1 -- 172.21.15.27:0/1315645894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7ff870059240 con 0x7ff8800c2430 2024-03-28T19:04:29.848 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:29.848 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:29.849 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.851+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff86006bbd0 msgr2=0x7ff86006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.851+0000 7ff884e00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff86006bbd0 0x7ff86006e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff8801498c0 tx=0x7ff86c009040 comp rx=0 tx=0).stop 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.851+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c2430 msgr2=0x7ff8800acc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.851+0000 7ff884e00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c2430 0x7ff8800acc80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff87000d5f0 tx=0x7ff87000dab0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.852+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 shutdown_connections 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.852+0000 7ff884e00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff86006bbd0 0x7ff86006e090 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.852+0000 7ff884e00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8800c5210 0x7ff8801458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.852+0000 7ff884e00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8800c4630 0x7ff8801474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.852+0000 7ff884e00640 1 --2- 172.21.15.27:0/1315645894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8800c2430 0x7ff8800acc80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:29.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.852+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 >> 172.21.15.27:0/1315645894 conn(0x7ff88006c5f0 msgr2=0x7ff8800aa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:29.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.852+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 shutdown_connections 2024-03-28T19:04:29.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:29.852+0000 7ff884e00640 1 -- 172.21.15.27:0/1315645894 wait complete. 2024-03-28T19:04:29.873 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:04:29.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:04:29.880 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:04:31.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:32.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.122+0000 7f5e73e00640 1 Processor -- start 2024-03-28T19:04:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.123+0000 7f5e73e00640 1 -- start start 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.124+0000 7f5e73e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e740c58c0 0x7f5e740c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.124+0000 7f5e73e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5e740be5d0 0x7f5e74155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.124+0000 7f5e73e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e74156230 con 0x7f5e740c8ed0 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.124+0000 7f5e73e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e74158bd0 con 0x7f5e740c58c0 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.124+0000 7f5e73e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e74158eb0 con 0x7f5e740be5d0 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e72a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e740c58c0 0x7f5e740c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e72a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e740c58c0 0x7f5e740c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53386/0 (socket says 172.21.15.27:53386) 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e6be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5e740c8ed0 0x7f5e741555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60166/0 (socket says 172.21.15.27:60166) 2024-03-28T19:04:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e72a00640 1 -- 172.21.15.27:0/2783294449 learned_addr learned my addr 172.21.15.27:0/2783294449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e73400640 1 --2- 172.21.15.27:0/2783294449 >> v2:172.21.15.155:3300/0 conn(0x7f5e740be5d0 0x7f5e74155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e72a00640 1 -- 172.21.15.27:0/2783294449 >> v2:172.21.15.155:3300/0 conn(0x7f5e740be5d0 msgr2=0x7f5e74155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e72a00640 1 --2- 172.21.15.27:0/2783294449 >> v2:172.21.15.155:3300/0 conn(0x7f5e740be5d0 0x7f5e74155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e72a00640 1 -- 172.21.15.27:0/2783294449 >> v1:172.21.15.27:6789/0 conn(0x7f5e740c8ed0 legacy=0x7f5e741555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:32.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.125+0000 7f5e72a00640 1 -- 172.21.15.27:0/2783294449 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e74159190 con 0x7f5e740c58c0 2024-03-28T19:04:32.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.127+0000 7f5e72a00640 1 --2- 172.21.15.27:0/2783294449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740c58c0 0x7f5e740c5ca0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5e7408d050 tx=0x7f5e6402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deaf8a033261246c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.127+0000 7f5e72000640 1 -- 172.21.15.27:0/2783294449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6402e070 con 0x7f5e740c58c0 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.127+0000 7f5e72000640 1 -- 172.21.15.27:0/2783294449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e64037070 con 0x7f5e740c58c0 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.127+0000 7f5e72000640 1 -- 172.21.15.27:0/2783294449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64031a00 con 0x7f5e740c58c0 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.127+0000 7f5e73e00640 1 -- 172.21.15.27:0/2783294449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740c58c0 msgr2=0x7f5e740c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.127+0000 7f5e73e00640 1 --2- 172.21.15.27:0/2783294449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740c58c0 0x7f5e740c5ca0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5e7408d050 tx=0x7f5e6402b880 comp rx=0 tx=0).stop 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.128+0000 7f5e73e00640 1 -- 172.21.15.27:0/2783294449 shutdown_connections 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.128+0000 7f5e73e00640 1 --2- 172.21.15.27:0/2783294449 >> v2:172.21.15.155:3300/0 conn(0x7f5e740be5d0 0x7f5e74155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.128+0000 7f5e73e00640 1 --2- 172.21.15.27:0/2783294449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740c58c0 0x7f5e740c5ca0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.128+0000 7f5e73e00640 1 -- 172.21.15.27:0/2783294449 >> 172.21.15.27:0/2783294449 conn(0x7f5e7406b6f0 msgr2=0x7f5e740acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.128+0000 7f5e73e00640 1 -- 172.21.15.27:0/2783294449 shutdown_connections 2024-03-28T19:04:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.128+0000 7f5e73e00640 1 -- 172.21.15.27:0/2783294449 wait complete. 2024-03-28T19:04:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.130+0000 7f5e73e00640 1 Processor -- start 2024-03-28T19:04:32.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.130+0000 7f5e73e00640 1 -- start start 2024-03-28T19:04:32.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740be5d0 0x7f5e740c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:32.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e73e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740c58c0 0x7f5e741ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:32.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e73e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740c8ed0 0x7f5e74154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e73e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740b0820 con 0x7f5e740be5d0 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e73e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e74159750 con 0x7f5e740c58c0 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e73e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740b09a0 con 0x7f5e740c8ed0 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e72a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740be5d0 0x7f5e740c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e72a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740be5d0 0x7f5e740c2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53388/0 (socket says 172.21.15.27:53388) 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.131+0000 7f5e72a00640 1 -- 172.21.15.27:0/691098820 learned_addr learned my addr 172.21.15.27:0/691098820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e73400640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740c8ed0 0x7f5e74154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e6be00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740c58c0 0x7f5e741ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e72a00640 1 -- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740c8ed0 msgr2=0x7f5e74154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e72a00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740c8ed0 0x7f5e74154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e72a00640 1 -- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740c58c0 msgr2=0x7f5e741ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e72a00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740c58c0 0x7f5e741ee9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e72a00640 1 -- 172.21.15.27:0/691098820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740ab480 con 0x7f5e740be5d0 2024-03-28T19:04:32.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e6be00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740c58c0 0x7f5e741ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.132+0000 7f5e72a00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740be5d0 0x7f5e740c2280 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5e64000c00 tx=0x7f5e640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.133+0000 7f5e6b400640 1 -- 172.21.15.27:0/691098820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64032670 con 0x7f5e740be5d0 2024-03-28T19:04:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.133+0000 7f5e6b400640 1 -- 172.21.15.27:0/691098820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e64032850 con 0x7f5e740be5d0 2024-03-28T19:04:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.133+0000 7f5e6b400640 1 -- 172.21.15.27:0/691098820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64032c40 con 0x7f5e740be5d0 2024-03-28T19:04:32.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.133+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e740c4120 con 0x7f5e740be5d0 2024-03-28T19:04:32.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.133+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e740c27c0 con 0x7f5e740be5d0 2024-03-28T19:04:32.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.134+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e74069640 con 0x7f5e740be5d0 2024-03-28T19:04:32.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.135+0000 7f5e6b400640 1 -- 172.21.15.27:0/691098820 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5e6404c060 con 0x7f5e740be5d0 2024-03-28T19:04:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.136+0000 7f5e6b400640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e5006bbd0 0x7f5e5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.136+0000 7f5e6b400640 1 -- 172.21.15.27:0/691098820 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f5e640accd0 con 0x7f5e740be5d0 2024-03-28T19:04:32.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.136+0000 7f5e6be00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e5006bbd0 0x7f5e5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:32.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.137+0000 7f5e6be00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e5006bbd0 0x7f5e5006e090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5e741efab0 tx=0x7f5e5c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:32.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.141+0000 7f5e6b400640 1 -- 172.21.15.27:0/691098820 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5e64040070 con 0x7f5e740be5d0 2024-03-28T19:04:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.428+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e740ad5b0 con 0x7f5e740be5d0 2024-03-28T19:04:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.429+0000 7f5e6b400640 1 -- 172.21.15.27:0/691098820 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f5e6407afd0 con 0x7f5e740be5d0 2024-03-28T19:04:32.429 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:32.429 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e5006bbd0 msgr2=0x7f5e5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e5006bbd0 0x7f5e5006e090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5e741efab0 tx=0x7f5e5c009040 comp rx=0 tx=0).stop 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740be5d0 msgr2=0x7f5e740c2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740be5d0 0x7f5e740c2280 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5e64000c00 tx=0x7f5e640046c0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 shutdown_connections 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e5006bbd0 0x7f5e5006e090 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740c8ed0 0x7f5e74154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740c58c0 0x7f5e741ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 --2- 172.21.15.27:0/691098820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740be5d0 0x7f5e740c2280 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.432+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 >> 172.21.15.27:0/691098820 conn(0x7f5e7406b6f0 msgr2=0x7f5e740ca200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.433+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 shutdown_connections 2024-03-28T19:04:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:32.433+0000 7f5e73e00640 1 -- 172.21.15.27:0/691098820 wait complete. 2024-03-28T19:04:32.455 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:04:34.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:34.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.690+0000 7f3453000640 1 Processor -- start 2024-03-28T19:04:34.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.690+0000 7f3453000640 1 -- start start 2024-03-28T19:04:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.691+0000 7f3453000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f344c0c69a0 0x7f344c0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.691+0000 7f3453000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c150ec0 con 0x7f344c150ae0 2024-03-28T19:04:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.691+0000 7f3453000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c06dc70 con 0x7f344c153a30 2024-03-28T19:04:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.691+0000 7f3453000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344c153550 con 0x7f344c0c69a0 2024-03-28T19:04:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.691+0000 7f3451200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f344c0c69a0 0x7f344c0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.691+0000 7f3451200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f344c0c69a0 0x7f344c0c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53400/0 (socket says 172.21.15.27:53400) 2024-03-28T19:04:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.691+0000 7f3451200640 1 -- 172.21.15.27:0/2785628170 learned_addr learned my addr 172.21.15.27:0/2785628170 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.692+0000 7f3451200640 1 -- 172.21.15.27:0/2785628170 >> v1:172.21.15.155:6789/0 conn(0x7f344c153a30 legacy=0x7f344c15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.692+0000 7f3451200640 1 -- 172.21.15.27:0/2785628170 >> v1:172.21.15.27:6789/0 conn(0x7f344c150ae0 legacy=0x7f344c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.692+0000 7f3451200640 1 -- 172.21.15.27:0/2785628170 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c06df10 con 0x7f344c0c69a0 2024-03-28T19:04:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.693+0000 7f3451200640 1 --2- 172.21.15.27:0/2785628170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0c6e30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3438009870 tx=0x7f343802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e1e4b05f537b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:34.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.693+0000 7f344be00640 1 -- 172.21.15.27:0/2785628170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438035020 con 0x7f344c0c69a0 2024-03-28T19:04:34.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f344be00640 1 -- 172.21.15.27:0/2785628170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343802d040 con 0x7f344c0c69a0 2024-03-28T19:04:34.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f344be00640 1 -- 172.21.15.27:0/2785628170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438031a80 con 0x7f344c0c69a0 2024-03-28T19:04:34.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f3453000640 1 -- 172.21.15.27:0/2785628170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 msgr2=0x7f344c0c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f3453000640 1 --2- 172.21.15.27:0/2785628170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0c6e30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3438009870 tx=0x7f343802b900 comp rx=0 tx=0).stop 2024-03-28T19:04:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f3453000640 1 -- 172.21.15.27:0/2785628170 shutdown_connections 2024-03-28T19:04:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f3453000640 1 --2- 172.21.15.27:0/2785628170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0c6e30 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f3453000640 1 -- 172.21.15.27:0/2785628170 >> 172.21.15.27:0/2785628170 conn(0x7f344c06b0c0 msgr2=0x7f344c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f3453000640 1 -- 172.21.15.27:0/2785628170 shutdown_connections 2024-03-28T19:04:34.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.694+0000 7f3453000640 1 -- 172.21.15.27:0/2785628170 wait complete. 2024-03-28T19:04:34.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.696+0000 7f3453000640 1 Processor -- start 2024-03-28T19:04:34.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.696+0000 7f3453000640 1 -- start start 2024-03-28T19:04:34.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.697+0000 7f3453000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.697+0000 7f3453000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f344c150ae0 0x7f344c0a9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.697+0000 7f3453000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f344c153a30 0x7f344c1ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.697+0000 7f3453000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c06e7b0 con 0x7f344c0c69a0 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.697+0000 7f3453000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c06e4d0 con 0x7f344c153a30 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.697+0000 7f3453000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0a6370 con 0x7f344c150ae0 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0a8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0a8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53414/0 (socket says 172.21.15.27:53414) 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451c00640 1 -- 172.21.15.27:0/2952315124 learned_addr learned my addr 172.21.15.27:0/2952315124 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451200640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f344c150ae0 0x7f344c0a9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3452600640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f344c153a30 0x7f344c1ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451c00640 1 -- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f344c150ae0 msgr2=0x7f344c0a9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:34.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451c00640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f344c150ae0 0x7f344c0a9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:34.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451c00640 1 -- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f344c153a30 msgr2=0x7f344c1ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:34.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451c00640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f344c153a30 0x7f344c1ee670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:34.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451c00640 1 -- 172.21.15.27:0/2952315124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c148a10 con 0x7f344c0c69a0 2024-03-28T19:04:34.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.698+0000 7f3451200640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f344c150ae0 0x7f344c0a9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:34.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.699+0000 7f3451c00640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0a8c20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f344000c400 tx=0x7f344000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.699+0000 7f3449600640 1 -- 172.21.15.27:0/2952315124 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34400052d0 con 0x7f344c0c69a0 2024-03-28T19:04:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.699+0000 7f3449600640 1 -- 172.21.15.27:0/2952315124 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3440005490 con 0x7f344c0c69a0 2024-03-28T19:04:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.699+0000 7f3449600640 1 -- 172.21.15.27:0/2952315124 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3440007260 con 0x7f344c0c69a0 2024-03-28T19:04:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.699+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c148cd0 con 0x7f344c0c69a0 2024-03-28T19:04:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.699+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c149290 con 0x7f344c0c69a0 2024-03-28T19:04:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.700+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c067dd0 con 0x7f344c0c69a0 2024-03-28T19:04:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.701+0000 7f3449600640 1 -- 172.21.15.27:0/2952315124 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3440007400 con 0x7f344c0c69a0 2024-03-28T19:04:34.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.702+0000 7f3449600640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f342c06bbd0 0x7f342c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:34.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.702+0000 7f3449600640 1 -- 172.21.15.27:0/2952315124 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f344008a740 con 0x7f344c0c69a0 2024-03-28T19:04:34.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.703+0000 7f3451200640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f342c06bbd0 0x7f342c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:34.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.703+0000 7f3451200640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f342c06bbd0 0x7f342c06e090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3438036000 tx=0x7f3438008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:34.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:34.708+0000 7f3449600640 1 -- 172.21.15.27:0/2952315124 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f344c150ec0 con 0x7f344c0c69a0 2024-03-28T19:04:35.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.003+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f344c06da50 con 0x7f344c0c69a0 2024-03-28T19:04:35.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.003+0000 7f3449600640 1 -- 172.21.15.27:0/2952315124 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f3440058ad0 con 0x7f344c0c69a0 2024-03-28T19:04:35.003 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:35.003 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:35.003 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:35.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f342c06bbd0 msgr2=0x7f342c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:35.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f342c06bbd0 0x7f342c06e090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3438036000 tx=0x7f3438008030 comp rx=0 tx=0).stop 2024-03-28T19:04:35.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 msgr2=0x7f344c0a8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:35.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0a8c20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f344000c400 tx=0x7f344000c8c0 comp rx=0 tx=0).stop 2024-03-28T19:04:35.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 shutdown_connections 2024-03-28T19:04:35.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f342c06bbd0 0x7f342c06e090 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f344c153a30 0x7f344c1ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f344c150ae0 0x7f344c0a9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.006+0000 7f3453000640 1 --2- 172.21.15.27:0/2952315124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f344c0c69a0 0x7f344c0a8c20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.007+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 >> 172.21.15.27:0/2952315124 conn(0x7f344c06b0c0 msgr2=0x7f344c155ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.007+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 shutdown_connections 2024-03-28T19:04:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:35.007+0000 7f3453000640 1 -- 172.21.15.27:0/2952315124 wait complete. 2024-03-28T19:04:35.024 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:04:37.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:37.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.274+0000 7fcb6b200640 1 Processor -- start 2024-03-28T19:04:37.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.274+0000 7fcb6b200640 1 -- start start 2024-03-28T19:04:37.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.276+0000 7fcb6b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb640c69a0 0x7fcb640c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:37.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.276+0000 7fcb6b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb64150ec0 con 0x7fcb64150ae0 2024-03-28T19:04:37.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.276+0000 7fcb6b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6406dc70 con 0x7fcb64153a30 2024-03-28T19:04:37.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.276+0000 7fcb6b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb64153550 con 0x7fcb640c69a0 2024-03-28T19:04:37.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.276+0000 7fcb69e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcb64150ae0 0x7fcb640a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46418/0 (socket says 172.21.15.27:46418) 2024-03-28T19:04:37.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.276+0000 7fcb69e00640 1 -- 172.21.15.27:0/1011505636 learned_addr learned my addr 172.21.15.27:0/1011505636 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:37.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.276+0000 7fcb69400640 1 --2- 172.21.15.27:0/1011505636 >> v2:172.21.15.27:3300/0 conn(0x7fcb640c69a0 0x7fcb640c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:37.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.277+0000 7fcb69400640 1 -- 172.21.15.27:0/1011505636 >> v1:172.21.15.155:6789/0 conn(0x7fcb64153a30 legacy=0x7fcb6415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:37.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.277+0000 7fcb69400640 1 -- 172.21.15.27:0/1011505636 >> v1:172.21.15.27:6789/0 conn(0x7fcb64150ae0 legacy=0x7fcb640a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:37.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.277+0000 7fcb69400640 1 -- 172.21.15.27:0/1011505636 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6406df10 con 0x7fcb640c69a0 2024-03-28T19:04:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.279+0000 7fcb69400640 1 --2- 172.21.15.27:0/1011505636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb640c6e30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fcb540097a0 tx=0x7fcb5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7045e2e937528dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.279+0000 7fcb68a00640 1 -- 172.21.15.27:0/1011505636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54035020 con 0x7fcb640c69a0 2024-03-28T19:04:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.279+0000 7fcb68a00640 1 -- 172.21.15.27:0/1011505636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb54031830 con 0x7fcb640c69a0 2024-03-28T19:04:37.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.279+0000 7fcb68a00640 1 -- 172.21.15.27:0/1011505636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54031c00 con 0x7fcb640c69a0 2024-03-28T19:04:37.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.280+0000 7fcb6b200640 1 -- 172.21.15.27:0/1011505636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 msgr2=0x7fcb640c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.280+0000 7fcb6b200640 1 --2- 172.21.15.27:0/1011505636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb640c6e30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fcb540097a0 tx=0x7fcb5402b8d0 comp rx=0 tx=0).stop 2024-03-28T19:04:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.280+0000 7fcb6b200640 1 -- 172.21.15.27:0/1011505636 shutdown_connections 2024-03-28T19:04:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.280+0000 7fcb6b200640 1 --2- 172.21.15.27:0/1011505636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb640c6e30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.280+0000 7fcb6b200640 1 -- 172.21.15.27:0/1011505636 >> 172.21.15.27:0/1011505636 conn(0x7fcb6406b0c0 msgr2=0x7fcb6406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.280+0000 7fcb6b200640 1 -- 172.21.15.27:0/1011505636 shutdown_connections 2024-03-28T19:04:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.280+0000 7fcb6b200640 1 -- 172.21.15.27:0/1011505636 wait complete. 2024-03-28T19:04:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.282+0000 7fcb6b200640 1 Processor -- start 2024-03-28T19:04:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.282+0000 7fcb6b200640 1 -- start start 2024-03-28T19:04:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.283+0000 7fcb6b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb64072430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.283+0000 7fcb6b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb64150ae0 0x7fcb640bfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.283+0000 7fcb6b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb64153a30 0x7fcb641eb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.283+0000 7fcb6b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6406e7b0 con 0x7fcb640c69a0 2024-03-28T19:04:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.283+0000 7fcb6b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6406e4d0 con 0x7fcb64150ae0 2024-03-28T19:04:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.283+0000 7fcb6b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640a6370 con 0x7fcb64153a30 2024-03-28T19:04:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb64072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb64072430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56998/0 (socket says 172.21.15.27:56998) 2024-03-28T19:04:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69e00640 1 -- 172.21.15.27:0/887764358 learned_addr learned my addr 172.21.15.27:0/887764358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69400640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb64150ae0 0x7fcb640bfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb6a800640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb64153a30 0x7fcb641eb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69e00640 1 -- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb64153a30 msgr2=0x7fcb641eb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69e00640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb64153a30 0x7fcb641eb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69e00640 1 -- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb64150ae0 msgr2=0x7fcb640bfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69e00640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb64150ae0 0x7fcb640bfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb69e00640 1 -- 172.21.15.27:0/887764358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb64073110 con 0x7fcb640c69a0 2024-03-28T19:04:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.284+0000 7fcb6a800640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb64153a30 0x7fcb641eb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.285+0000 7fcb69e00640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb64072430 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c4a0 tx=0x7fcb5800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.285+0000 7fcb5e000640 1 -- 172.21.15.27:0/887764358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58017070 con 0x7fcb640c69a0 2024-03-28T19:04:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.285+0000 7fcb5e000640 1 -- 172.21.15.27:0/887764358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb58005ce0 con 0x7fcb640c69a0 2024-03-28T19:04:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.285+0000 7fcb5e000640 1 -- 172.21.15.27:0/887764358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58007270 con 0x7fcb640c69a0 2024-03-28T19:04:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.285+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb640a9000 con 0x7fcb640c69a0 2024-03-28T19:04:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.285+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb641ef8d0 con 0x7fcb640c69a0 2024-03-28T19:04:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.286+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb64067dd0 con 0x7fcb640c69a0 2024-03-28T19:04:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.287+0000 7fcb5e000640 1 -- 172.21.15.27:0/887764358 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcb5801a070 con 0x7fcb640c69a0 2024-03-28T19:04:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.289+0000 7fcb5e000640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb4806bbd0 0x7fcb4806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.289+0000 7fcb5e000640 1 -- 172.21.15.27:0/887764358 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7fcb5808a3f0 con 0x7fcb640c69a0 2024-03-28T19:04:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.289+0000 7fcb69400640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb4806bbd0 0x7fcb4806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.290+0000 7fcb69400640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb4806bbd0 0x7fcb4806e090 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcb54036000 tx=0x7fcb54008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.293+0000 7fcb5e000640 1 -- 172.21.15.27:0/887764358 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcb64150ec0 con 0x7fcb640c69a0 2024-03-28T19:04:37.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.595+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb64069640 con 0x7fcb640c69a0 2024-03-28T19:04:37.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.597+0000 7fcb5e000640 1 -- 172.21.15.27:0/887764358 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7fcb58058130 con 0x7fcb640c69a0 2024-03-28T19:04:37.598 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:37.598 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:37.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb4806bbd0 msgr2=0x7fcb4806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:37.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb4806bbd0 0x7fcb4806e090 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcb54036000 tx=0x7fcb54008030 comp rx=0 tx=0).stop 2024-03-28T19:04:37.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 msgr2=0x7fcb64072430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:37.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb64072430 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c4a0 tx=0x7fcb5800c960 comp rx=0 tx=0).stop 2024-03-28T19:04:37.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 shutdown_connections 2024-03-28T19:04:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb4806bbd0 0x7fcb4806e090 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb64153a30 0x7fcb641eb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb64150ae0 0x7fcb640bfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 --2- 172.21.15.27:0/887764358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb640c69a0 0x7fcb64072430 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.599+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 >> 172.21.15.27:0/887764358 conn(0x7fcb6406b0c0 msgr2=0x7fcb640c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.600+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 shutdown_connections 2024-03-28T19:04:37.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:37.600+0000 7fcb6b200640 1 -- 172.21.15.27:0/887764358 wait complete. 2024-03-28T19:04:37.623 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:04:37.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:38.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.127+0000 7f5f8f400640 1 Processor -- start 2024-03-28T19:04:38.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.127+0000 7f5f8f400640 1 -- start start 2024-03-28T19:04:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.129+0000 7f5f8f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c60e0 0x7f5f8806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.129+0000 7f5f8f400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5f880c3ea0 0x7f5f8806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.129+0000 7f5f8f400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f880c1600 con 0x7f5f88072df0 2024-03-28T19:04:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.129+0000 7f5f8f400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f880c23b0 con 0x7f5f880c3ea0 2024-03-28T19:04:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.129+0000 7f5f8f400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880c18e0 con 0x7f5f880c60e0 2024-03-28T19:04:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.130+0000 7f5f7fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5f880c3ea0 0x7f5f8806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.130+0000 7f5f8ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c60e0 0x7f5f8806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.130+0000 7f5f7fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5f880c3ea0 0x7f5f8806ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:55914/0 (socket says 172.21.15.27:55914) 2024-03-28T19:04:38.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.130+0000 7f5f7fe00640 1 -- 172.21.15.27:0/2427626344 learned_addr learned my addr 172.21.15.27:0/2427626344 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:38.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.131+0000 7f5f8ca00640 1 -- 172.21.15.27:0/2427626344 >> v2:172.21.15.155:3300/0 conn(0x7f5f880c3ea0 msgr2=0x7f5f8806ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:38.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.131+0000 7f5f8ca00640 1 --2- 172.21.15.27:0/2427626344 >> v2:172.21.15.155:3300/0 conn(0x7f5f880c3ea0 0x7f5f8806ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.131+0000 7f5f8ca00640 1 -- 172.21.15.27:0/2427626344 >> v1:172.21.15.155:6789/0 conn(0x7f5f88072df0 legacy=0x7f5f88158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.131+0000 7f5f8ca00640 1 -- 172.21.15.27:0/2427626344 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f880c1bc0 con 0x7f5f880c60e0 2024-03-28T19:04:38.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.131+0000 7f5f7fe00640 1 --2- 172.21.15.27:0/2427626344 >> v2:172.21.15.155:3300/0 conn(0x7f5f880c3ea0 0x7f5f8806ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:04:38.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.133+0000 7f5f8ca00640 1 --2- 172.21.15.27:0/2427626344 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c60e0 0x7f5f8806e6f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009d90 tx=0x7f5f7802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=457cb61c673e45f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.133+0000 7f5f87e00640 1 -- 172.21.15.27:0/2427626344 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78039070 con 0x7f5f880c60e0 2024-03-28T19:04:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.133+0000 7f5f87e00640 1 -- 172.21.15.27:0/2427626344 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7802cb30 con 0x7f5f880c60e0 2024-03-28T19:04:38.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.133+0000 7f5f87e00640 1 -- 172.21.15.27:0/2427626344 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78034440 con 0x7f5f880c60e0 2024-03-28T19:04:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.134+0000 7f5f8f400640 1 -- 172.21.15.27:0/2427626344 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c60e0 msgr2=0x7f5f8806e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.134+0000 7f5f8f400640 1 --2- 172.21.15.27:0/2427626344 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c60e0 0x7f5f8806e6f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009d90 tx=0x7f5f7802bd80 comp rx=0 tx=0).stop 2024-03-28T19:04:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.134+0000 7f5f8f400640 1 -- 172.21.15.27:0/2427626344 shutdown_connections 2024-03-28T19:04:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.134+0000 7f5f8f400640 1 --2- 172.21.15.27:0/2427626344 >> v2:172.21.15.155:3300/0 conn(0x7f5f880c3ea0 0x7f5f8806ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.134+0000 7f5f8f400640 1 --2- 172.21.15.27:0/2427626344 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c60e0 0x7f5f8806e6f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.134+0000 7f5f8f400640 1 -- 172.21.15.27:0/2427626344 >> 172.21.15.27:0/2427626344 conn(0x7f5f8806b9d0 msgr2=0x7f5f880c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.134+0000 7f5f8f400640 1 -- 172.21.15.27:0/2427626344 shutdown_connections 2024-03-28T19:04:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.134+0000 7f5f8f400640 1 -- 172.21.15.27:0/2427626344 wait complete. 2024-03-28T19:04:38.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.136+0000 7f5f8f400640 1 Processor -- start 2024-03-28T19:04:38.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.136+0000 7f5f8f400640 1 -- start start 2024-03-28T19:04:38.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f88072df0 0x7f5f8809a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c3ea0 0x7f5f88071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f880c60e0 0x7f5f880aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8f400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880c6640 con 0x7f5f880c60e0 2024-03-28T19:04:38.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8f400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880c64c0 con 0x7f5f88072df0 2024-03-28T19:04:38.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8f400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880c4f00 con 0x7f5f880c3ea0 2024-03-28T19:04:38.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f880c60e0 0x7f5f880aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f880c60e0 0x7f5f880aa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57008/0 (socket says 172.21.15.27:57008) 2024-03-28T19:04:38.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8d400640 1 -- 172.21.15.27:0/630584178 learned_addr learned my addr 172.21.15.27:0/630584178 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:38.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f7fe00640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c3ea0 0x7f5f88071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.137+0000 7f5f8ca00640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f88072df0 0x7f5f8809a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.138+0000 7f5f8d400640 1 -- 172.21.15.27:0/630584178 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c3ea0 msgr2=0x7f5f88071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.138+0000 7f5f8d400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c3ea0 0x7f5f88071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.138+0000 7f5f8d400640 1 -- 172.21.15.27:0/630584178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f88072df0 msgr2=0x7f5f8809a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.138+0000 7f5f8d400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f88072df0 0x7f5f8809a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.138+0000 7f5f8d400640 1 -- 172.21.15.27:0/630584178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f880abf40 con 0x7f5f880c60e0 2024-03-28T19:04:38.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.138+0000 7f5f8d400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f880c60e0 0x7f5f880aa5b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5f8000c3d0 tx=0x7f5f8000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:38.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.139+0000 7f5f85600640 1 -- 172.21.15.27:0/630584178 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80017070 con 0x7f5f880c60e0 2024-03-28T19:04:38.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.139+0000 7f5f85600640 1 -- 172.21.15.27:0/630584178 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f800052d0 con 0x7f5f880c60e0 2024-03-28T19:04:38.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.139+0000 7f5f85600640 1 -- 172.21.15.27:0/630584178 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80003830 con 0x7f5f880c60e0 2024-03-28T19:04:38.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.139+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f880a9b10 con 0x7f5f880c60e0 2024-03-28T19:04:38.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.139+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f880a9db0 con 0x7f5f880c60e0 2024-03-28T19:04:38.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.140+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8804f400 con 0x7f5f880c60e0 2024-03-28T19:04:38.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.141+0000 7f5f85600640 1 -- 172.21.15.27:0/630584178 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5f8001a070 con 0x7f5f880c60e0 2024-03-28T19:04:38.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.142+0000 7f5f85600640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5f6406bbd0 0x7f5f6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.142+0000 7f5f85600640 1 -- 172.21.15.27:0/630584178 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f5f8008a490 con 0x7f5f880c60e0 2024-03-28T19:04:38.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.142+0000 7f5f8ca00640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5f6406bbd0 0x7f5f6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.143+0000 7f5f8ca00640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5f6406bbd0 0x7f5f6406e090 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5f78036000 tx=0x7f5f78030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:38.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.148+0000 7f5f85600640 1 -- 172.21.15.27:0/630584178 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5f80058290 con 0x7f5f880c60e0 2024-03-28T19:04:38.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.420+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f880c2fd0 con 0x7f5f880c60e0 2024-03-28T19:04:38.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.421+0000 7f5f85600640 1 -- 172.21.15.27:0/630584178 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f5f8005c330 con 0x7f5f880c60e0 2024-03-28T19:04:38.422 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:38.422 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:38.422 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5f6406bbd0 msgr2=0x7f5f6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5f6406bbd0 0x7f5f6406e090 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5f78036000 tx=0x7f5f78030000 comp rx=0 tx=0).stop 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f880c60e0 msgr2=0x7f5f880aa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f880c60e0 0x7f5f880aa5b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5f8000c3d0 tx=0x7f5f8000c890 comp rx=0 tx=0).stop 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 shutdown_connections 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5f6406bbd0 0x7f5f6406e090 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f880c60e0 0x7f5f880aa5b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f880c3ea0 0x7f5f88071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 --2- 172.21.15.27:0/630584178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f88072df0 0x7f5f8809a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 >> 172.21.15.27:0/630584178 conn(0x7f5f8806b9d0 msgr2=0x7f5f880c3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:38.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.424+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 shutdown_connections 2024-03-28T19:04:38.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.425+0000 7f5f8f400640 1 -- 172.21.15.27:0/630584178 wait complete. 2024-03-28T19:04:38.446 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-03-28T19:04:38.446 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:04:38.446 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:04:38.495 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:04:38.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:38.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.722+0000 7f8b1fa00640 1 Processor -- start 2024-03-28T19:04:38.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.722+0000 7f8b1fa00640 1 -- start start 2024-03-28T19:04:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.723+0000 7f8b1fa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b18150c70 0x7f8b180a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.723+0000 7f8b1fa00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8b180c3690 0x7f8b180c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b1fa00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b180c3af0 con 0x7f8b180c6660 2024-03-28T19:04:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b1fa00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b180ab4e0 con 0x7f8b180c3690 2024-03-28T19:04:38.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b1fa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b180ac1f0 con 0x7f8b18150c70 2024-03-28T19:04:38.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b1d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b18150c70 0x7f8b180a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b0fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8b180c3690 0x7f8b180c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b1d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b18150c70 0x7f8b180a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:54462/0 (socket says 172.21.15.27:54462) 2024-03-28T19:04:38.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b0fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8b180c3690 0x7f8b180c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:55926/0 (socket says 172.21.15.27:55926) 2024-03-28T19:04:38.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b1d000640 1 -- 172.21.15.27:0/1086474371 learned_addr learned my addr 172.21.15.27:0/1086474371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:38.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.724+0000 7f8b1da00640 1 --1- 172.21.15.27:0/1086474371 >> v1:172.21.15.155:6789/0 conn(0x7f8b180c6660 0x7f8b181534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:58688/0 (socket says 172.21.15.27:58688) 2024-03-28T19:04:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.725+0000 7f8b0fe00640 1 -- 172.21.15.27:0/1086474371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b18150c70 msgr2=0x7f8b180a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.725+0000 7f8b0fe00640 1 --2- 172.21.15.27:0/1086474371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b18150c70 0x7f8b180a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.725+0000 7f8b0fe00640 1 -- 172.21.15.27:0/1086474371 >> v1:172.21.15.155:6789/0 conn(0x7f8b180c6660 legacy=0x7f8b181534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.725+0000 7f8b0fe00640 1 -- 172.21.15.27:0/1086474371 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b180c3d90 con 0x7f8b180c3690 2024-03-28T19:04:38.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.725+0000 7f8b1d000640 1 --2- 172.21.15.27:0/1086474371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b18150c70 0x7f8b180a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:04:38.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.726+0000 7f8b0fe00640 1 --2- 172.21.15.27:0/1086474371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b180c3690 0x7f8b180c6120 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8b000097a0 tx=0x7f8b0002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93802b2cb926e4da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:38.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.727+0000 7f8b17e00640 1 -- 172.21.15.27:0/1086474371 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00031780 con 0x7f8b180c3690 2024-03-28T19:04:38.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.727+0000 7f8b17e00640 1 -- 172.21.15.27:0/1086474371 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b00037070 con 0x7f8b180c3690 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.727+0000 7f8b17e00640 1 -- 172.21.15.27:0/1086474371 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00031b40 con 0x7f8b180c3690 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.728+0000 7f8b1fa00640 1 -- 172.21.15.27:0/1086474371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b180c3690 msgr2=0x7f8b180c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.728+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/1086474371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b180c3690 0x7f8b180c6120 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8b000097a0 tx=0x7f8b0002b820 comp rx=0 tx=0).stop 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.728+0000 7f8b1fa00640 1 -- 172.21.15.27:0/1086474371 shutdown_connections 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.728+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/1086474371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b180c3690 0x7f8b180c6120 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.728+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/1086474371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b18150c70 0x7f8b180a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.728+0000 7f8b1fa00640 1 -- 172.21.15.27:0/1086474371 >> 172.21.15.27:0/1086474371 conn(0x7f8b1806b9d0 msgr2=0x7f8b180c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.728+0000 7f8b1fa00640 1 -- 172.21.15.27:0/1086474371 shutdown_connections 2024-03-28T19:04:38.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.728+0000 7f8b1fa00640 1 -- 172.21.15.27:0/1086474371 wait complete. 2024-03-28T19:04:38.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.730+0000 7f8b1fa00640 1 Processor -- start 2024-03-28T19:04:38.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.730+0000 7f8b1fa00640 1 -- start start 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1fa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180c3690 0x7f8b18153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1fa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b180c6660 0x7f8b18154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1fa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b18150c70 0x7f8b1814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1fa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b180abcd0 con 0x7f8b180c3690 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1fa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b180ab9f0 con 0x7f8b18150c70 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1fa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b180c1810 con 0x7f8b180c6660 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180c3690 0x7f8b18153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180c3690 0x7f8b18153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57020/0 (socket says 172.21.15.27:57020) 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1d000640 1 -- 172.21.15.27:0/3484225943 learned_addr learned my addr 172.21.15.27:0/3484225943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b0fe00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b180c6660 0x7f8b18154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1da00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b18150c70 0x7f8b1814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1d000640 1 -- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b180c6660 msgr2=0x7f8b18154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1d000640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b180c6660 0x7f8b18154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1d000640 1 -- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b18150c70 msgr2=0x7f8b1814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1d000640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b18150c70 0x7f8b1814b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b1d000640 1 -- 172.21.15.27:0/3484225943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18154d70 con 0x7f8b180c3690 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.731+0000 7f8b0fe00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b180c6660 0x7f8b18154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.732+0000 7f8b1d000640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180c3690 0x7f8b18153580 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800e9b0 tx=0x7f8b0800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.732+0000 7f8b15600640 1 -- 172.21.15.27:0/3484225943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08017070 con 0x7f8b180c3690 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.732+0000 7f8b15600640 1 -- 172.21.15.27:0/3484225943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0800c9a0 con 0x7f8b180c3690 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.732+0000 7f8b15600640 1 -- 172.21.15.27:0/3484225943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0800cd50 con 0x7f8b180c3690 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.732+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b181ece70 con 0x7f8b180c3690 2024-03-28T19:04:38.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.732+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b181560b0 con 0x7f8b180c3690 2024-03-28T19:04:38.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.734+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1804f400 con 0x7f8b180c3690 2024-03-28T19:04:38.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.735+0000 7f8b15600640 1 -- 172.21.15.27:0/3484225943 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8b0801a070 con 0x7f8b180c3690 2024-03-28T19:04:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.735+0000 7f8b15600640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8af406bbd0 0x7f8af406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.736+0000 7f8b15600640 1 -- 172.21.15.27:0/3484225943 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7583+0+0 (secure 0 0 0) 0x7f8b0808b0c0 con 0x7f8b180c3690 2024-03-28T19:04:38.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.736+0000 7f8b0fe00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8af406bbd0 0x7f8af406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:38.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.736+0000 7f8b0fe00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8af406bbd0 0x7f8af406e090 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8b00036000 tx=0x7f8b00008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:38.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.740+0000 7f8b15600640 1 -- 172.21.15.27:0/3484225943 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8b08059450 con 0x7f8b180c3690 2024-03-28T19:04:38.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.984+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b180c5f00 con 0x7f8b180c3690 2024-03-28T19:04:38.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.985+0000 7f8b15600640 1 -- 172.21.15.27:0/3484225943 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6256 (secure 0 0 0) 0x7f8b08058d50 con 0x7f8b180c3690 2024-03-28T19:04:38.986 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:38.986 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:02:41.197840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6836/1001046540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1001046540},{"type":"v1","addr":"172.21.15.27:6836","nonce":1001046540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5769,"qdb_cluster":[5769]},"id":2}]} 2024-03-28T19:04:38.987 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 76 2024-03-28T19:04:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8af406bbd0 msgr2=0x7f8af406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8af406bbd0 0x7f8af406e090 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8b00036000 tx=0x7f8b00008030 comp rx=0 tx=0).stop 2024-03-28T19:04:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180c3690 msgr2=0x7f8b18153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180c3690 0x7f8b18153580 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800e9b0 tx=0x7f8b0800ee70 comp rx=0 tx=0).stop 2024-03-28T19:04:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 shutdown_connections 2024-03-28T19:04:38.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8af406bbd0 0x7f8af406e090 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b18150c70 0x7f8b1814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b180c6660 0x7f8b18154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 --2- 172.21.15.27:0/3484225943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b180c3690 0x7f8b18153580 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 >> 172.21.15.27:0/3484225943 conn(0x7f8b1806b9d0 msgr2=0x7f8b180712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.988+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 shutdown_connections 2024-03-28T19:04:38.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:38.989+0000 7f8b1fa00640 1 -- 172.21.15.27:0/3484225943 wait complete. 2024-03-28T19:04:39.009 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:04:39.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:39.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.887+0000 7ff6f9600640 1 Processor -- start 2024-03-28T19:04:39.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.888+0000 7ff6f9600640 1 -- start start 2024-03-28T19:04:39.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.889+0000 7ff6f9600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f406ee00 0x7ff6f4158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:39.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.889+0000 7ff6f9600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f40c1600 con 0x7ff6f40c60e0 2024-03-28T19:04:39.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.889+0000 7ff6f9600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f40c18e0 con 0x7ff6f40c3ea0 2024-03-28T19:04:39.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.889+0000 7ff6f9600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40c23b0 con 0x7ff6f406ee00 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.889+0000 7ff6f2a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff6f40c60e0 0x7ff6f406e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46430/0 (socket says 172.21.15.27:46430) 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.889+0000 7ff6f2a00640 1 -- 172.21.15.27:0/2482603349 learned_addr learned my addr 172.21.15.27:0/2482603349 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6ebe00640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f406ee00 0x7ff6f4158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743692389 0 0) 0x7ff6f40c1600 con 0x7ff6f40c60e0 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d8003520 con 0x7ff6f40c60e0 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2740072154 0 0) 0x7ff6d8003520 con 0x7ff6f40c60e0 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f406ee00 msgr2=0x7ff6f4158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f406ee00 0x7ff6f4158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 >> v1:172.21.15.155:6789/0 conn(0x7ff6f40c3ea0 legacy=0x7ff6f4159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f40c1bc0 con 0x7ff6f40c60e0 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7ff6e4001670 con 0x7ff6f40c60e0 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 >> v1:172.21.15.27:6789/0 conn(0x7ff6f40c60e0 legacy=0x7ff6f406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.890+0000 7ff6f2000640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2000640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6d8004800 0x7ff6d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2000640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f406ee00 0x7ff6d8005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40c23b0 con 0x7ff6d80039e0 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d800bb50 con 0x7ff6d8004800 2024-03-28T19:04:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d800bd50 con 0x7ff6f406ee00 2024-03-28T19:04:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2a00640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f3400640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f406ee00 0x7ff6d8005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6ebe00640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6d8004800 0x7ff6d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2a00640 1 -- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f406ee00 msgr2=0x7ff6d8005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2a00640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f406ee00 0x7ff6d8005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.891+0000 7ff6f2a00640 1 -- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6d8004800 msgr2=0x7ff6d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.892+0000 7ff6f2a00640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6d8004800 0x7ff6d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.892+0000 7ff6f2a00640 1 -- 172.21.15.27:0/2482603349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d8003520 con 0x7ff6d80039e0 2024-03-28T19:04:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.893+0000 7ff6f2a00640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6d8003e00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400c0e0 tx=0x7ff6d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e012df246dac007 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4007c20 con 0x7ff6d80039e0 2024-03-28T19:04:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e4030070 con 0x7ff6d80039e0 2024-03-28T19:04:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f2000640 1 -- 172.21.15.27:0/2482603349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4005d30 con 0x7ff6d80039e0 2024-03-28T19:04:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f9600640 1 -- 172.21.15.27:0/2482603349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 msgr2=0x7ff6d8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f9600640 1 --2- 172.21.15.27:0/2482603349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6d8003e00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400c0e0 tx=0x7ff6d8005830 comp rx=0 tx=0).stop 2024-03-28T19:04:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f3400640 1 -- 172.21.15.27:0/2482603349 reap_dead start 2024-03-28T19:04:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f9600640 1 -- 172.21.15.27:0/2482603349 shutdown_connections 2024-03-28T19:04:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f9600640 1 -- 172.21.15.27:0/2482603349 >> 172.21.15.27:0/2482603349 conn(0x7ff6f406b9d0 msgr2=0x7ff6f40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f9600640 1 -- 172.21.15.27:0/2482603349 shutdown_connections 2024-03-28T19:04:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.894+0000 7ff6f9600640 1 -- 172.21.15.27:0/2482603349 wait complete. 2024-03-28T19:04:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.897+0000 7ff6f9600640 1 Processor -- start 2024-03-28T19:04:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.897+0000 7ff6f9600640 1 -- start start 2024-03-28T19:04:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6f4153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f9600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f41e1120 0x7ff6f41579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f9600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6f41df1d0 0x7ff6f41df650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40c64c0 con 0x7ff6d80039e0 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f9600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4159b50 con 0x7ff6f41df1d0 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f9600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40c6640 con 0x7ff6f41e1120 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6f4153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6f4153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57028/0 (socket says 172.21.15.27:57028) 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.899+0000 7ff6f2a00640 1 -- 172.21.15.27:0/3919456182 learned_addr learned my addr 172.21.15.27:0/3919456182 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6ebe00640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f41e1120 0x7ff6f41579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f2a00640 1 -- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f41e1120 msgr2=0x7ff6f41579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f3400640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6f41df1d0 0x7ff6f41df650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f2a00640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f41e1120 0x7ff6f41579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:39.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f2a00640 1 -- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6f41df1d0 msgr2=0x7ff6f41df650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f2a00640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6f41df1d0 0x7ff6f41df650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f2a00640 1 -- 172.21.15.27:0/3919456182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f41eeea0 con 0x7ff6d80039e0 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6ebe00640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f41e1120 0x7ff6f41579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f3400640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6f41df1d0 0x7ff6f41df650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f2a00640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6f4153200 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400c1d0 tx=0x7ff6f40c1d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6eaa00640 1 -- 172.21.15.27:0/3919456182 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4045070 con 0x7ff6d80039e0 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6eaa00640 1 -- 172.21.15.27:0/3919456182 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e4004bf0 con 0x7ff6d80039e0 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6eaa00640 1 -- 172.21.15.27:0/3919456182 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4001b60 con 0x7ff6d80039e0 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.900+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f4155ac0 con 0x7ff6d80039e0 2024-03-28T19:04:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.901+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f4157f00 con 0x7ff6d80039e0 2024-03-28T19:04:39.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.902+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f404f400 con 0x7ff6d80039e0 2024-03-28T19:04:39.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.903+0000 7ff6eaa00640 1 -- 172.21.15.27:0/3919456182 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff6e4007070 con 0x7ff6d80039e0 2024-03-28T19:04:39.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.904+0000 7ff6eaa00640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff6d006bbd0 0x7ff6d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:39.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.904+0000 7ff6eaa00640 1 -- 172.21.15.27:0/3919456182 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7ff6e40ac2c0 con 0x7ff6d80039e0 2024-03-28T19:04:39.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.904+0000 7ff6ebe00640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff6d006bbd0 0x7ff6d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:39.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.905+0000 7ff6ebe00640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff6d006bbd0 0x7ff6d006e090 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40c5dc0 tx=0x7ff6dc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:39.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:39.913+0000 7ff6eaa00640 1 -- 172.21.15.27:0/3919456182 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff6e4007cc0 con 0x7ff6d80039e0 2024-03-28T19:04:40.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.179+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f40c4ce0 con 0x7ff6d80039e0 2024-03-28T19:04:40.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.180+0000 7ff6eaa00640 1 -- 172.21.15.27:0/3919456182 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5457 (secure 0 0 0) 0x7ff6e407a020 con 0x7ff6d80039e0 2024-03-28T19:04:40.181 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:40.181 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:39.773675+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:04:40.181 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-03-28T19:04:40.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.183+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff6d006bbd0 msgr2=0x7ff6d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:40.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.183+0000 7ff6f9600640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff6d006bbd0 0x7ff6d006e090 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40c5dc0 tx=0x7ff6dc008040 comp rx=0 tx=0).stop 2024-03-28T19:04:40.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.183+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 msgr2=0x7ff6f4153200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:40.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6f4153200 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400c1d0 tx=0x7ff6f40c1d90 comp rx=0 tx=0).stop 2024-03-28T19:04:40.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 shutdown_connections 2024-03-28T19:04:40.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff6d006bbd0 0x7ff6d006e090 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6f41df1d0 0x7ff6f41df650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6f41e1120 0x7ff6f41579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 --2- 172.21.15.27:0/3919456182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d80039e0 0x7ff6f4153200 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 >> 172.21.15.27:0/3919456182 conn(0x7ff6f406b9d0 msgr2=0x7ff6f4158790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:40.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 shutdown_connections 2024-03-28T19:04:40.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.184+0000 7ff6f9600640 1 -- 172.21.15.27:0/3919456182 wait complete. 2024-03-28T19:04:40.211 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-03-28T19:04:40.212 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:04:40.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.402+0000 7f4b16a00640 1 Processor -- start 2024-03-28T19:04:40.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.402+0000 7f4b16a00640 1 -- start start 2024-03-28T19:04:40.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.403+0000 7f4b16a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b100c1f50 0x7f4b100ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:40.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.403+0000 7f4b16a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4b100abd90 0x7f4b100aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.403+0000 7f4b16a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b100ac1c0 con 0x7f4b100c5e80 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.403+0000 7f4b16a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1006d650 con 0x7f4b100c1f50 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.403+0000 7f4b16a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1006e1d0 con 0x7f4b100abd90 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.404+0000 7f4b0f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b100c1f50 0x7f4b100ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.404+0000 7f4b0f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b100c1f50 0x7f4b100ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57030/0 (socket says 172.21.15.27:57030) 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.404+0000 7f4b0f400640 1 -- 172.21.15.27:0/1082859901 learned_addr learned my addr 172.21.15.27:0/1082859901 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.404+0000 7f4b14a00640 1 --2- 172.21.15.27:0/1082859901 >> v2:172.21.15.155:3300/0 conn(0x7f4b100abd90 0x7f4b100aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.404+0000 7f4b0f400640 1 -- 172.21.15.27:0/1082859901 >> v2:172.21.15.155:3300/0 conn(0x7f4b100abd90 msgr2=0x7f4b100aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.404+0000 7f4b0f400640 1 --2- 172.21.15.27:0/1082859901 >> v2:172.21.15.155:3300/0 conn(0x7f4b100abd90 0x7f4b100aa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.404+0000 7f4b0f400640 1 -- 172.21.15.27:0/1082859901 >> v1:172.21.15.27:6789/0 conn(0x7f4b100c5e80 legacy=0x7f4b100c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:40.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.404+0000 7f4b0f400640 1 -- 172.21.15.27:0/1082859901 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1006d8a0 con 0x7f4b100c1f50 2024-03-28T19:04:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.406+0000 7f4b0f400640 1 --2- 172.21.15.27:0/1082859901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100c1f50 0x7f4b100ab850 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4b1006eff0 tx=0x7f4afc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731591054db107 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.406+0000 7f4b0ea00640 1 -- 172.21.15.27:0/1082859901 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc02e070 con 0x7f4b100c1f50 2024-03-28T19:04:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.406+0000 7f4b0ea00640 1 -- 172.21.15.27:0/1082859901 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc037070 con 0x7f4b100c1f50 2024-03-28T19:04:40.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.406+0000 7f4b0ea00640 1 -- 172.21.15.27:0/1082859901 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc031a00 con 0x7f4b100c1f50 2024-03-28T19:04:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.407+0000 7f4b16a00640 1 -- 172.21.15.27:0/1082859901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100c1f50 msgr2=0x7f4b100ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.407+0000 7f4b16a00640 1 --2- 172.21.15.27:0/1082859901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100c1f50 0x7f4b100ab850 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4b1006eff0 tx=0x7f4afc02b880 comp rx=0 tx=0).stop 2024-03-28T19:04:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.407+0000 7f4b16a00640 1 -- 172.21.15.27:0/1082859901 shutdown_connections 2024-03-28T19:04:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.407+0000 7f4b16a00640 1 --2- 172.21.15.27:0/1082859901 >> v2:172.21.15.155:3300/0 conn(0x7f4b100abd90 0x7f4b100aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.407+0000 7f4b16a00640 1 --2- 172.21.15.27:0/1082859901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100c1f50 0x7f4b100ab850 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.407+0000 7f4b16a00640 1 -- 172.21.15.27:0/1082859901 >> 172.21.15.27:0/1082859901 conn(0x7f4b1006b9d0 msgr2=0x7f4b100c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.407+0000 7f4b16a00640 1 -- 172.21.15.27:0/1082859901 shutdown_connections 2024-03-28T19:04:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.407+0000 7f4b16a00640 1 -- 172.21.15.27:0/1082859901 wait complete. 2024-03-28T19:04:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.410+0000 7f4b16a00640 1 Processor -- start 2024-03-28T19:04:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.410+0000 7f4b16a00640 1 -- start start 2024-03-28T19:04:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.411+0000 7f4b16a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100abd90 0x7f4b101e0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.411+0000 7f4b16a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100c1f50 0x7f4b100a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.411+0000 7f4b16a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b100c5e80 0x7f4b101eea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.411+0000 7f4b16a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100c6260 con 0x7f4b100abd90 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.411+0000 7f4b16a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1006de60 con 0x7f4b100c1f50 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.411+0000 7f4b16a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100c63e0 con 0x7f4b100c5e80 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100abd90 0x7f4b101e0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100abd90 0x7f4b101e0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57034/0 (socket says 172.21.15.27:57034) 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0fe00640 1 -- 172.21.15.27:0/2261529956 learned_addr learned my addr 172.21.15.27:0/2261529956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0f400640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100c1f50 0x7f4b100a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0fe00640 1 -- 172.21.15.27:0/2261529956 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b100c5e80 msgr2=0x7f4b101eea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0fe00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b100c5e80 0x7f4b101eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0fe00640 1 -- 172.21.15.27:0/2261529956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100c1f50 msgr2=0x7f4b100a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0fe00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100c1f50 0x7f4b100a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0fe00640 1 -- 172.21.15.27:0/2261529956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b100c0cb0 con 0x7f4b100abd90 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.412+0000 7f4b0f400640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100c1f50 0x7f4b100a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.413+0000 7f4b0fe00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100abd90 0x7f4b101e0290 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000c3f0 tx=0x7f4b0000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.413+0000 7f4b07e00640 1 -- 172.21.15.27:0/2261529956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b000052d0 con 0x7f4b100abd90 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.413+0000 7f4b07e00640 1 -- 172.21.15.27:0/2261529956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b00005490 con 0x7f4b100abd90 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.413+0000 7f4b07e00640 1 -- 172.21.15.27:0/2261529956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00007260 con 0x7f4b100abd90 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.413+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b101ee5a0 con 0x7f4b100abd90 2024-03-28T19:04:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.413+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b100a82f0 con 0x7f4b100abd90 2024-03-28T19:04:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.414+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b101ee770 con 0x7f4b100abd90 2024-03-28T19:04:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.415+0000 7f4b07e00640 1 -- 172.21.15.27:0/2261529956 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4b00007400 con 0x7f4b100abd90 2024-03-28T19:04:40.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.417+0000 7f4b07e00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4af006bbd0 0x7f4af006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:40.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.417+0000 7f4b07e00640 1 -- 172.21.15.27:0/2261529956 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f4b0008a790 con 0x7f4b100abd90 2024-03-28T19:04:40.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.417+0000 7f4b0f400640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4af006bbd0 0x7f4af006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.418+0000 7f4b0f400640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4af006bbd0 0x7f4af006e090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4afc036000 tx=0x7f4afc008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.422+0000 7f4b07e00640 1 -- 172.21.15.27:0/2261529956 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4b00058a40 con 0x7f4b100abd90 2024-03-28T19:04:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.668+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b100c4ba0 con 0x7f4b100abd90 2024-03-28T19:04:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.669+0000 7f4b07e00640 1 -- 172.21.15.27:0/2261529956 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f4b00058340 con 0x7f4b100abd90 2024-03-28T19:04:40.670 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:40.670 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:39.773675+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:04:40.670 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-03-28T19:04:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.671+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4af006bbd0 msgr2=0x7f4af006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.671+0000 7f4b16a00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4af006bbd0 0x7f4af006e090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4afc036000 tx=0x7f4afc008030 comp rx=0 tx=0).stop 2024-03-28T19:04:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.671+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100abd90 msgr2=0x7f4b101e0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.671+0000 7f4b16a00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100abd90 0x7f4b101e0290 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4b0000c3f0 tx=0x7f4b0000c8b0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.672+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 shutdown_connections 2024-03-28T19:04:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.672+0000 7f4b16a00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4af006bbd0 0x7f4af006e090 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.672+0000 7f4b16a00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b100c5e80 0x7f4b101eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.672+0000 7f4b16a00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100c1f50 0x7f4b100a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.672+0000 7f4b16a00640 1 --2- 172.21.15.27:0/2261529956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b100abd90 0x7f4b101e0290 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:40.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.672+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 >> 172.21.15.27:0/2261529956 conn(0x7f4b1006b9d0 msgr2=0x7f4b100c5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:40.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.672+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 shutdown_connections 2024-03-28T19:04:40.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:40.672+0000 7f4b16a00640 1 -- 172.21.15.27:0/2261529956 wait complete. 2024-03-28T19:04:40.689 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 75, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5201": { "addr": "172.21.15.155:6835/331868181", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 331868181, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 331868181, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5201, "incarnation": 72, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:03:59.309496+0000", "qdb_cluster": [ 5201 ], "qdb_leader": 5201, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5201 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6054": { "addr": "172.21.15.27:6837/1815850547", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 1815850547, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 1815850547, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6054, "incarnation": 77, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:04:39.773675+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6054 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:04:41.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.239+0000 7f87f4800640 1 Processor -- start 2024-03-28T19:04:41.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.240+0000 7f87f4800640 1 -- start start 2024-03-28T19:04:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.241+0000 7f87f4800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87ec153ed0 0x7f87ec06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.241+0000 7f87f4800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec15a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.241+0000 7f87f4800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec06efb0 con 0x7f87ec06e6f0 2024-03-28T19:04:41.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.241+0000 7f87f4800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ec0c6990 con 0x7f87ec153ed0 2024-03-28T19:04:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.241+0000 7f87f4800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0c3830 con 0x7f87ec06eb80 2024-03-28T19:04:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.241+0000 7f87f1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87ec153ed0 0x7f87ec06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.241+0000 7f87f1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87ec153ed0 0x7f87ec06ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57036/0 (socket says 172.21.15.27:57036) 2024-03-28T19:04:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.241+0000 7f87f1e00640 1 -- 172.21.15.27:0/2340743774 learned_addr learned my addr 172.21.15.27:0/2340743774 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.242+0000 7f87f2800640 1 --2- 172.21.15.27:0/2340743774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec15a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.242+0000 7f87f1e00640 1 -- 172.21.15.27:0/2340743774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 msgr2=0x7f87ec15a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:41.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.242+0000 7f87f1e00640 1 --2- 172.21.15.27:0/2340743774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec15a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.242+0000 7f87f1e00640 1 -- 172.21.15.27:0/2340743774 >> v1:172.21.15.155:6789/0 conn(0x7f87ec06e6f0 legacy=0x7f87ec159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:41.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.242+0000 7f87f1e00640 1 -- 172.21.15.27:0/2340743774 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec06d1f0 con 0x7f87ec153ed0 2024-03-28T19:04:41.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.242+0000 7f87f2800640 1 --2- 172.21.15.27:0/2340743774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec15a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:41.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.243+0000 7f87f1e00640 1 --2- 172.21.15.27:0/2340743774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec153ed0 0x7f87ec06ccb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0c5ca0 tx=0x7f87e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b772c796c3772a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:41.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.243+0000 7f87f0a00640 1 -- 172.21.15.27:0/2340743774 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0035020 con 0x7f87ec153ed0 2024-03-28T19:04:41.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.243+0000 7f87f0a00640 1 -- 172.21.15.27:0/2340743774 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e0031830 con 0x7f87ec153ed0 2024-03-28T19:04:41.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.243+0000 7f87f0a00640 1 -- 172.21.15.27:0/2340743774 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0031c00 con 0x7f87ec153ed0 2024-03-28T19:04:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.244+0000 7f87f4800640 1 -- 172.21.15.27:0/2340743774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec153ed0 msgr2=0x7f87ec06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.244+0000 7f87f4800640 1 --2- 172.21.15.27:0/2340743774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec153ed0 0x7f87ec06ccb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0c5ca0 tx=0x7f87e002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.244+0000 7f87f4800640 1 -- 172.21.15.27:0/2340743774 shutdown_connections 2024-03-28T19:04:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.244+0000 7f87f4800640 1 --2- 172.21.15.27:0/2340743774 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec15a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.244+0000 7f87f4800640 1 --2- 172.21.15.27:0/2340743774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec153ed0 0x7f87ec06ccb0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.244+0000 7f87f4800640 1 -- 172.21.15.27:0/2340743774 >> 172.21.15.27:0/2340743774 conn(0x7f87ec06b9d0 msgr2=0x7f87ec0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.244+0000 7f87f4800640 1 -- 172.21.15.27:0/2340743774 shutdown_connections 2024-03-28T19:04:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.244+0000 7f87f4800640 1 -- 172.21.15.27:0/2340743774 wait complete. 2024-03-28T19:04:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.247+0000 7f87f4800640 1 Processor -- start 2024-03-28T19:04:41.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.247+0000 7f87f4800640 1 -- start start 2024-03-28T19:04:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.248+0000 7f87f4800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec06e6f0 0x7f87ec156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.248+0000 7f87f4800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec1e5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.248+0000 7f87f4800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87ec153ed0 0x7f87ec1e4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.248+0000 7f87f4800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0c3db0 con 0x7f87ec06e6f0 2024-03-28T19:04:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.248+0000 7f87f4800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0c3ad0 con 0x7f87ec153ed0 2024-03-28T19:04:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.248+0000 7f87f4800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec15a810 con 0x7f87ec06eb80 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec06e6f0 0x7f87ec156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec06e6f0 0x7f87ec156e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57048/0 (socket says 172.21.15.27:57048) 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 -- 172.21.15.27:0/1647359512 learned_addr learned my addr 172.21.15.27:0/1647359512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1400640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec1e5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f2800640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87ec153ed0 0x7f87ec1e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 -- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 msgr2=0x7f87ec1e5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec1e5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 -- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87ec153ed0 msgr2=0x7f87ec1e4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:41.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87ec153ed0 0x7f87ec1e4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 -- 172.21.15.27:0/1647359512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec156b80 con 0x7f87ec06e6f0 2024-03-28T19:04:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f2800640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87ec153ed0 0x7f87ec1e4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1400640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec1e5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.249+0000 7f87f1e00640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec06e6f0 0x7f87ec156e60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f87e0000c00 tx=0x7f87e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.250+0000 7f87e6000640 1 -- 172.21.15.27:0/1647359512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0040070 con 0x7f87ec06e6f0 2024-03-28T19:04:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.250+0000 7f87e6000640 1 -- 172.21.15.27:0/1647359512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e00399c0 con 0x7f87ec06e6f0 2024-03-28T19:04:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.250+0000 7f87e6000640 1 -- 172.21.15.27:0/1647359512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0035040 con 0x7f87ec06e6f0 2024-03-28T19:04:41.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.250+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec1e5ce0 con 0x7f87ec06e6f0 2024-03-28T19:04:41.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.250+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ec0a63f0 con 0x7f87ec06e6f0 2024-03-28T19:04:41.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.251+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ec1f0050 con 0x7f87ec06e6f0 2024-03-28T19:04:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.252+0000 7f87e6000640 1 -- 172.21.15.27:0/1647359512 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f87e004b060 con 0x7f87ec06e6f0 2024-03-28T19:04:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.254+0000 7f87e6000640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87d006bc00 0x7f87d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.254+0000 7f87e6000640 1 -- 172.21.15.27:0/1647359512 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f87e00acd10 con 0x7f87ec06e6f0 2024-03-28T19:04:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.254+0000 7f87f1400640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87d006bc00 0x7f87d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:41.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.255+0000 7f87f1400640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87d006bc00 0x7f87d006e0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f87ec153850 tx=0x7f87dc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:41.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.257+0000 7f87e6000640 1 -- 172.21.15.27:0/1647359512 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f87e00b2020 con 0x7f87ec06e6f0 2024-03-28T19:04:41.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.504+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87ec04f400 con 0x7f87ec06e6f0 2024-03-28T19:04:41.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.505+0000 7f87e6000640 1 -- 172.21.15.27:0/1647359512 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f87ec06efb0 con 0x7f87ec06e6f0 2024-03-28T19:04:41.507 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:41.507 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:39.773675+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:04:41.507 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-03-28T19:04:41.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87d006bc00 msgr2=0x7f87d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:41.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87d006bc00 0x7f87d006e0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f87ec153850 tx=0x7f87dc009040 comp rx=0 tx=0).stop 2024-03-28T19:04:41.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec06e6f0 msgr2=0x7f87ec156e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:41.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec06e6f0 0x7f87ec156e60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f87e0000c00 tx=0x7f87e00046c0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 shutdown_connections 2024-03-28T19:04:41.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87d006bc00 0x7f87d006e0c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87ec153ed0 0x7f87ec1e4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87ec06eb80 0x7f87ec1e5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 --2- 172.21.15.27:0/1647359512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87ec06e6f0 0x7f87ec156e60 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:41.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 >> 172.21.15.27:0/1647359512 conn(0x7f87ec06b9d0 msgr2=0x7f87ec0ab4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 shutdown_connections 2024-03-28T19:04:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:41.508+0000 7f87f4800640 1 -- 172.21.15.27:0/1647359512 wait complete. 2024-03-28T19:04:41.525 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:04:42.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:42.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.926+0000 7f9853e00640 1 Processor -- start 2024-03-28T19:04:42.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.926+0000 7f9853e00640 1 -- start start 2024-03-28T19:04:42.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.928+0000 7f9853e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98540c5320 0x7f985406e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:42.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.928+0000 7f9853e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9854156500 0x7f98540c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:42.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.928+0000 7f9853e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9854159c50 con 0x7f9854156120 2024-03-28T19:04:42.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.928+0000 7f9853e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98540c4d70 con 0x7f98540c5320 2024-03-28T19:04:42.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.928+0000 7f9853e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98540c1030 con 0x7f9854156500 2024-03-28T19:04:42.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.928+0000 7f984a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98540c5320 0x7f985406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:42.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.929+0000 7f984a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98540c5320 0x7f985406e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57052/0 (socket says 172.21.15.27:57052) 2024-03-28T19:04:42.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.929+0000 7f984a000640 1 -- 172.21.15.27:0/4271774481 learned_addr learned my addr 172.21.15.27:0/4271774481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:42.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.929+0000 7f9852a00640 1 --1- 172.21.15.27:0/4271774481 >> v1:172.21.15.27:6789/0 conn(0x7f9854156120 0x7f98540c0920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46454/0 (socket says 172.21.15.27:46454) 2024-03-28T19:04:42.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.929+0000 7f9853400640 1 --2- 172.21.15.27:0/4271774481 >> v2:172.21.15.155:3300/0 conn(0x7f9854156500 0x7f98540c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:42.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.929+0000 7f984a000640 1 -- 172.21.15.27:0/4271774481 >> v2:172.21.15.155:3300/0 conn(0x7f9854156500 msgr2=0x7f98540c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:42.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.929+0000 7f984a000640 1 --2- 172.21.15.27:0/4271774481 >> v2:172.21.15.155:3300/0 conn(0x7f9854156500 0x7f98540c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:42.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.929+0000 7f984a000640 1 -- 172.21.15.27:0/4271774481 >> v1:172.21.15.27:6789/0 conn(0x7f9854156120 legacy=0x7f98540c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:42.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.929+0000 7f984a000640 1 -- 172.21.15.27:0/4271774481 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98540c4a40 con 0x7f98540c5320 2024-03-28T19:04:42.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.930+0000 7f984a000640 1 --2- 172.21.15.27:0/4271774481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f985406e050 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f98540738f0 tx=0x7f983c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7723829b9c527a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:42.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9852000640 1 -- 172.21.15.27:0/4271774481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c02e070 con 0x7f98540c5320 2024-03-28T19:04:42.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9852000640 1 -- 172.21.15.27:0/4271774481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c037070 con 0x7f98540c5320 2024-03-28T19:04:42.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9852000640 1 -- 172.21.15.27:0/4271774481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983c031a00 con 0x7f98540c5320 2024-03-28T19:04:42.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9853e00640 1 -- 172.21.15.27:0/4271774481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 msgr2=0x7f985406e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:42.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9853e00640 1 --2- 172.21.15.27:0/4271774481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f985406e050 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f98540738f0 tx=0x7f983c02b880 comp rx=0 tx=0).stop 2024-03-28T19:04:42.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9853e00640 1 -- 172.21.15.27:0/4271774481 shutdown_connections 2024-03-28T19:04:42.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9853e00640 1 --2- 172.21.15.27:0/4271774481 >> v2:172.21.15.155:3300/0 conn(0x7f9854156500 0x7f98540c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:42.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9853e00640 1 --2- 172.21.15.27:0/4271774481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f985406e050 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:42.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9853e00640 1 -- 172.21.15.27:0/4271774481 >> 172.21.15.27:0/4271774481 conn(0x7f985406b0c0 msgr2=0x7f985406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:42.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9853e00640 1 -- 172.21.15.27:0/4271774481 shutdown_connections 2024-03-28T19:04:42.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.931+0000 7f9853e00640 1 -- 172.21.15.27:0/4271774481 wait complete. 2024-03-28T19:04:42.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.933+0000 7f9853e00640 1 Processor -- start 2024-03-28T19:04:42.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.933+0000 7f9853e00640 1 -- start start 2024-03-28T19:04:42.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.934+0000 7f9853e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f98541ef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.934+0000 7f9853e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854156120 0x7f9854158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.934+0000 7f9853e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9854156500 0x7f98541e1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.934+0000 7f9853e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98540c73d0 con 0x7f98540c5320 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.934+0000 7f9853e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98540c7250 con 0x7f9854156120 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.934+0000 7f9853e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98540c29a0 con 0x7f9854156500 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9852a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f98541ef1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9852a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f98541ef1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57062/0 (socket says 172.21.15.27:57062) 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9852a00640 1 -- 172.21.15.27:0/2059660174 learned_addr learned my addr 172.21.15.27:0/2059660174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f984a000640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854156120 0x7f9854158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9853400640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9854156500 0x7f98541e1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9852a00640 1 -- 172.21.15.27:0/2059660174 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9854156500 msgr2=0x7f98541e1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9852a00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9854156500 0x7f98541e1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9852a00640 1 -- 172.21.15.27:0/2059660174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854156120 msgr2=0x7f9854158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9852a00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854156120 0x7f9854158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.935+0000 7f9852a00640 1 -- 172.21.15.27:0/2059660174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98540aa890 con 0x7f98540c5320 2024-03-28T19:04:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.936+0000 7f9852a00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f98541ef1c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f984400c400 tx=0x7f984400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:42.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.936+0000 7f984b400640 1 -- 172.21.15.27:0/2059660174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98440074e0 con 0x7f98540c5320 2024-03-28T19:04:42.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.936+0000 7f984b400640 1 -- 172.21.15.27:0/2059660174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9844007680 con 0x7f98540c5320 2024-03-28T19:04:42.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.936+0000 7f984b400640 1 -- 172.21.15.27:0/2059660174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844004a00 con 0x7f98540c5320 2024-03-28T19:04:42.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.936+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98540aab70 con 0x7f98540c5320 2024-03-28T19:04:42.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.936+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98540ab8f0 con 0x7f98540c5320 2024-03-28T19:04:42.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.937+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9854067dd0 con 0x7f98540c5320 2024-03-28T19:04:42.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.938+0000 7f984b400640 1 -- 172.21.15.27:0/2059660174 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f984401a020 con 0x7f98540c5320 2024-03-28T19:04:42.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.939+0000 7f984b400640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f983006bbd0 0x7f983006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:42.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.939+0000 7f984b400640 1 -- 172.21.15.27:0/2059660174 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f984408aaf0 con 0x7f98540c5320 2024-03-28T19:04:42.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.939+0000 7f984a000640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f983006bbd0 0x7f983006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:42.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.940+0000 7f984a000640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f983006bbd0 0x7f983006e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f983c036000 tx=0x7f983c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:42.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:42.943+0000 7f984b400640 1 -- 172.21.15.27:0/2059660174 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9844010300 con 0x7f98540c5320 2024-03-28T19:04:43.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.186+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9854069640 con 0x7f98540c5320 2024-03-28T19:04:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.187+0000 7f984b400640 1 -- 172.21.15.27:0/2059660174 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5461 (secure 0 0 0) 0x7f9844058d90 con 0x7f98540c5320 2024-03-28T19:04:43.188 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:43.188 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:41.857649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:04:43.189 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 78 2024-03-28T19:04:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f983006bbd0 msgr2=0x7f983006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f983006bbd0 0x7f983006e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f983c036000 tx=0x7f983c03d040 comp rx=0 tx=0).stop 2024-03-28T19:04:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 msgr2=0x7f98541ef1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f98541ef1c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f984400c400 tx=0x7f984400c8c0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 shutdown_connections 2024-03-28T19:04:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f983006bbd0 0x7f983006e090 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9854156500 0x7f98541e1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854156120 0x7f9854158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 --2- 172.21.15.27:0/2059660174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98540c5320 0x7f98541ef1c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 >> 172.21.15.27:0/2059660174 conn(0x7f985406b0c0 msgr2=0x7f98540c6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 shutdown_connections 2024-03-28T19:04:43.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.190+0000 7f9853e00640 1 -- 172.21.15.27:0/2059660174 wait complete. 2024-03-28T19:04:43.208 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:04:43.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:43.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.744+0000 7f5c6fe00640 1 Processor -- start 2024-03-28T19:04:43.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.745+0000 7f5c6fe00640 1 -- start start 2024-03-28T19:04:43.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.746+0000 7f5c6fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5c700c8ed0 0x7f5c700c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:43.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.746+0000 7f5c6fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c700be5d0 0x7f5c70155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:43.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.746+0000 7f5c6fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c70156230 con 0x7f5c700c58c0 2024-03-28T19:04:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.746+0000 7f5c6fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c70158bd0 con 0x7f5c700be5d0 2024-03-28T19:04:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.746+0000 7f5c6fe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c70158eb0 con 0x7f5c700c8ed0 2024-03-28T19:04:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.747+0000 7f5c6f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c700be5d0 0x7f5c70155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.747+0000 7f5c6f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c700be5d0 0x7f5c70155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57064/0 (socket says 172.21.15.27:57064) 2024-03-28T19:04:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.747+0000 7f5c6f400640 1 -- 172.21.15.27:0/1914123423 learned_addr learned my addr 172.21.15.27:0/1914123423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.747+0000 7f5c6e000640 1 --2- 172.21.15.27:0/1914123423 >> v2:172.21.15.155:3300/0 conn(0x7f5c700c8ed0 0x7f5c700c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.747+0000 7f5c6f400640 1 -- 172.21.15.27:0/1914123423 >> v2:172.21.15.155:3300/0 conn(0x7f5c700c8ed0 msgr2=0x7f5c700c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.747+0000 7f5c6f400640 1 --2- 172.21.15.27:0/1914123423 >> v2:172.21.15.155:3300/0 conn(0x7f5c700c8ed0 0x7f5c700c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.747+0000 7f5c6f400640 1 -- 172.21.15.27:0/1914123423 >> v1:172.21.15.155:6789/0 conn(0x7f5c700c58c0 legacy=0x7f5c701555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:43.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.748+0000 7f5c6f400640 1 -- 172.21.15.27:0/1914123423 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c70159190 con 0x7f5c700be5d0 2024-03-28T19:04:43.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.748+0000 7f5c6e000640 1 --2- 172.21.15.27:0/1914123423 >> v2:172.21.15.155:3300/0 conn(0x7f5c700c8ed0 0x7f5c700c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:04:43.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.749+0000 7f5c6f400640 1 --2- 172.21.15.27:0/1914123423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c70155cf0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f5c700c83d0 tx=0x7f5c6802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438c0e1bb1b57255 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:43.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.749+0000 7f5c6d600640 1 -- 172.21.15.27:0/1914123423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6802c9f0 con 0x7f5c700be5d0 2024-03-28T19:04:43.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.749+0000 7f5c6d600640 1 -- 172.21.15.27:0/1914123423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6802cb90 con 0x7f5c700be5d0 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.749+0000 7f5c6d600640 1 -- 172.21.15.27:0/1914123423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680343f0 con 0x7f5c700be5d0 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.750+0000 7f5c6fe00640 1 -- 172.21.15.27:0/1914123423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 msgr2=0x7f5c70155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.750+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/1914123423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c70155cf0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f5c700c83d0 tx=0x7f5c6802be50 comp rx=0 tx=0).stop 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.750+0000 7f5c6fe00640 1 -- 172.21.15.27:0/1914123423 shutdown_connections 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.750+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/1914123423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c70155cf0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.750+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/1914123423 >> v2:172.21.15.155:3300/0 conn(0x7f5c700c8ed0 0x7f5c700c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.750+0000 7f5c6fe00640 1 -- 172.21.15.27:0/1914123423 >> 172.21.15.27:0/1914123423 conn(0x7f5c7006b6f0 msgr2=0x7f5c700acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.750+0000 7f5c6fe00640 1 -- 172.21.15.27:0/1914123423 shutdown_connections 2024-03-28T19:04:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.750+0000 7f5c6fe00640 1 -- 172.21.15.27:0/1914123423 wait complete. 2024-03-28T19:04:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.752+0000 7f5c6fe00640 1 Processor -- start 2024-03-28T19:04:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.752+0000 7f5c6fe00640 1 -- start start 2024-03-28T19:04:43.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c701e0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c700c58c0 0x7f5c700c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c700c8ed0 0x7f5c700c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700b0820 con 0x7f5c700be5d0 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70159750 con 0x7f5c700c8ed0 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700b09a0 con 0x7f5c700c58c0 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c701e0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c701e0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57068/0 (socket says 172.21.15.27:57068) 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.753+0000 7f5c6ea00640 1 -- 172.21.15.27:0/527774671 learned_addr learned my addr 172.21.15.27:0/527774671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6e000640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c700c58c0 0x7f5c700c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:43.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6f400640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c700c8ed0 0x7f5c700c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6ea00640 1 -- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c700c58c0 msgr2=0x7f5c700c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6ea00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c700c58c0 0x7f5c700c2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6ea00640 1 -- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c700c8ed0 msgr2=0x7f5c700c2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6ea00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c700c8ed0 0x7f5c700c2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6ea00640 1 -- 172.21.15.27:0/527774671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c701e1a40 con 0x7f5c700be5d0 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6f400640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c700c8ed0 0x7f5c700c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6ea00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c701e0f60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000c3f0 tx=0x7f5c6000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c66a00640 1 -- 172.21.15.27:0/527774671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c600052d0 con 0x7f5c700be5d0 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c66a00640 1 -- 172.21.15.27:0/527774671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c60005490 con 0x7f5c700be5d0 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c66a00640 1 -- 172.21.15.27:0/527774671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60007260 con 0x7f5c700be5d0 2024-03-28T19:04:43.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.754+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c701e17a0 con 0x7f5c700be5d0 2024-03-28T19:04:43.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.755+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c701ec250 con 0x7f5c700be5d0 2024-03-28T19:04:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.756+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c700c8360 con 0x7f5c700be5d0 2024-03-28T19:04:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.757+0000 7f5c66a00640 1 -- 172.21.15.27:0/527774671 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5c60007400 con 0x7f5c700be5d0 2024-03-28T19:04:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.757+0000 7f5c66a00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5006bc00 0x7f5c5006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.758+0000 7f5c66a00640 1 -- 172.21.15.27:0/527774671 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f5c6008a710 con 0x7f5c700be5d0 2024-03-28T19:04:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.758+0000 7f5c6e000640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5006bc00 0x7f5c5006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:43.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.758+0000 7f5c6e000640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5006bc00 0x7f5c5006e0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5c700ad520 tx=0x7f5c5c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:43.762+0000 7f5c66a00640 1 -- 172.21.15.27:0/527774671 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5c700c8360 con 0x7f5c700be5d0 2024-03-28T19:04:44.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.001+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c70069640 con 0x7f5c700be5d0 2024-03-28T19:04:44.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.002+0000 7f5c66a00640 1 -- 172.21.15.27:0/527774671 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5458 (secure 0 0 0) 0x7f5c60058a40 con 0x7f5c700be5d0 2024-03-28T19:04:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:44.003 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:42.511191+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:04:44.003 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 79 2024-03-28T19:04:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5006bc00 msgr2=0x7f5c5006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5006bc00 0x7f5c5006e0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5c700ad520 tx=0x7f5c5c008040 comp rx=0 tx=0).stop 2024-03-28T19:04:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 msgr2=0x7f5c701e0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c701e0f60 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000c3f0 tx=0x7f5c6000c8b0 comp rx=0 tx=0).stop 2024-03-28T19:04:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 shutdown_connections 2024-03-28T19:04:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c5006bc00 0x7f5c5006e0c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c700c8ed0 0x7f5c700c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c700c58c0 0x7f5c700c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 --2- 172.21.15.27:0/527774671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c700be5d0 0x7f5c701e0f60 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 >> 172.21.15.27:0/527774671 conn(0x7f5c7006b6f0 msgr2=0x7f5c700c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 shutdown_connections 2024-03-28T19:04:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:44.005+0000 7f5c6fe00640 1 -- 172.21.15.27:0/527774671 wait complete. 2024-03-28T19:04:44.024 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:04:45.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:45.355 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:04:45.355+0000 7f4d2e400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:04:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.437+0000 7fe6ef800640 1 Processor -- start 2024-03-28T19:04:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.437+0000 7fe6ef800640 1 -- start start 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.438+0000 7fe6ef800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e8150b60 0x7fe6e80ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.438+0000 7fe6ef800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6e80c8470 0x7fe6e8156b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.438+0000 7fe6ef800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6e80c8c70 0x7fe6e80c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.438+0000 7fe6ef800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80ad9b0 con 0x7fe6e80c8c70 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ef800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80ad680 con 0x7fe6e80c8470 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ef800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80c4de0 con 0x7fe6e8150b60 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ed800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6e80c8c70 0x7fe6e80c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ed800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6e80c8c70 0x7fe6e80c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57072/0 (socket says 172.21.15.27:57072) 2024-03-28T19:04:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ed800640 1 -- 172.21.15.27:0/3935711079 learned_addr learned my addr 172.21.15.27:0/3935711079 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ece00640 1 --2- 172.21.15.27:0/3935711079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e8150b60 0x7fe6e80ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6e7e00640 1 --2- 172.21.15.27:0/3935711079 >> v2:172.21.15.155:3300/0 conn(0x7fe6e80c8470 0x7fe6e8156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ed800640 1 -- 172.21.15.27:0/3935711079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e8150b60 msgr2=0x7fe6e80ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ed800640 1 --2- 172.21.15.27:0/3935711079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e8150b60 0x7fe6e80ade40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ed800640 1 -- 172.21.15.27:0/3935711079 >> v2:172.21.15.155:3300/0 conn(0x7fe6e80c8470 msgr2=0x7fe6e8156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.439+0000 7fe6ed800640 1 --2- 172.21.15.27:0/3935711079 >> v2:172.21.15.155:3300/0 conn(0x7fe6e80c8470 0x7fe6e8156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.440+0000 7fe6ed800640 1 -- 172.21.15.27:0/3935711079 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80c50c0 con 0x7fe6e80c8c70 2024-03-28T19:04:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.441+0000 7fe6ed800640 1 --2- 172.21.15.27:0/3935711079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e80c8c70 0x7fe6e80c9100 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80c5760 tx=0x7fe6e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e095c29c0de3cd99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.441+0000 7fe6e7400640 1 -- 172.21.15.27:0/3935711079 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e002c9e0 con 0x7fe6e80c8c70 2024-03-28T19:04:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.441+0000 7fe6e7400640 1 -- 172.21.15.27:0/3935711079 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e002cb80 con 0x7fe6e80c8c70 2024-03-28T19:04:45.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.441+0000 7fe6e7400640 1 -- 172.21.15.27:0/3935711079 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00343f0 con 0x7fe6e80c8c70 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 -- 172.21.15.27:0/3935711079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e80c8c70 msgr2=0x7fe6e80c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 --2- 172.21.15.27:0/3935711079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e80c8c70 0x7fe6e80c9100 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80c5760 tx=0x7fe6e002be40 comp rx=0 tx=0).stop 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 -- 172.21.15.27:0/3935711079 shutdown_connections 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 --2- 172.21.15.27:0/3935711079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e80c8c70 0x7fe6e80c9100 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 --2- 172.21.15.27:0/3935711079 >> v2:172.21.15.155:3300/0 conn(0x7fe6e80c8470 0x7fe6e8156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 --2- 172.21.15.27:0/3935711079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e8150b60 0x7fe6e80ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 -- 172.21.15.27:0/3935711079 >> 172.21.15.27:0/3935711079 conn(0x7fe6e80aaab0 msgr2=0x7fe6e80aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 -- 172.21.15.27:0/3935711079 shutdown_connections 2024-03-28T19:04:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.442+0000 7fe6ef800640 1 -- 172.21.15.27:0/3935711079 wait complete. 2024-03-28T19:04:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.445+0000 7fe6ef800640 1 Processor -- start 2024-03-28T19:04:45.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.445+0000 7fe6ef800640 1 -- start start 2024-03-28T19:04:45.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ef800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e80c8470 0x7fe6e806c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:45.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ef800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e80c8c70 0x7fe6e814f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:45.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ef800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e8150b60 0x7fe6e81474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:45.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ef800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80a85a0 con 0x7fe6e8150b60 2024-03-28T19:04:45.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ef800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8150f40 con 0x7fe6e80c8470 2024-03-28T19:04:45.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ef800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80a8720 con 0x7fe6e80c8c70 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ed800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e8150b60 0x7fe6e81474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ed800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e8150b60 0x7fe6e81474f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57080/0 (socket says 172.21.15.27:57080) 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.446+0000 7fe6ed800640 1 -- 172.21.15.27:0/4062900728 learned_addr learned my addr 172.21.15.27:0/4062900728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.447+0000 7fe6ece00640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e80c8470 0x7fe6e806c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.447+0000 7fe6e7e00640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e80c8c70 0x7fe6e814f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.447+0000 7fe6ed800640 1 -- 172.21.15.27:0/4062900728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e80c8c70 msgr2=0x7fe6e814f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.447+0000 7fe6ed800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e80c8c70 0x7fe6e814f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.447+0000 7fe6ed800640 1 -- 172.21.15.27:0/4062900728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e80c8470 msgr2=0x7fe6e806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.447+0000 7fe6ed800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e80c8470 0x7fe6e806c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.447+0000 7fe6ed800640 1 -- 172.21.15.27:0/4062900728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8070560 con 0x7fe6e8150b60 2024-03-28T19:04:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.447+0000 7fe6ed800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e8150b60 0x7fe6e81474f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009f40 tx=0x7fe6e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.448+0000 7fe6e4c00640 1 -- 172.21.15.27:0/4062900728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0043020 con 0x7fe6e8150b60 2024-03-28T19:04:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.448+0000 7fe6e4c00640 1 -- 172.21.15.27:0/4062900728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e003e450 con 0x7fe6e8150b60 2024-03-28T19:04:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.448+0000 7fe6e4c00640 1 -- 172.21.15.27:0/4062900728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0009510 con 0x7fe6e8150b60 2024-03-28T19:04:45.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.448+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e80707e0 con 0x7fe6e8150b60 2024-03-28T19:04:45.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.448+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e814e160 con 0x7fe6e8150b60 2024-03-28T19:04:45.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.449+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e814e360 con 0x7fe6e8150b60 2024-03-28T19:04:45.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.450+0000 7fe6e4c00640 1 -- 172.21.15.27:0/4062900728 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe6e0046020 con 0x7fe6e8150b60 2024-03-28T19:04:45.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.451+0000 7fe6e4c00640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe6c806bbd0 0x7fe6c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:45.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.451+0000 7fe6e4c00640 1 -- 172.21.15.27:0/4062900728 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7fe6e00ada70 con 0x7fe6e8150b60 2024-03-28T19:04:45.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.451+0000 7fe6ece00640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe6c806bbd0 0x7fe6c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:45.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.452+0000 7fe6ece00640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe6c806bbd0 0x7fe6c806e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc006fd0 tx=0x7fe6dc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:45.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.456+0000 7fe6e4c00640 1 -- 172.21.15.27:0/4062900728 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe6e004d030 con 0x7fe6e8150b60 2024-03-28T19:04:45.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.887+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6e804f400 con 0x7fe6e8150b60 2024-03-28T19:04:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.889+0000 7fe6e4c00640 1 -- 172.21.15.27:0/4062900728 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5465 (secure 0 0 0) 0x7fe6e8070560 con 0x7fe6e8150b60 2024-03-28T19:04:45.889 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:45.889 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:45.889 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 80 2024-03-28T19:04:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe6c806bbd0 msgr2=0x7fe6c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe6c806bbd0 0x7fe6c806e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc006fd0 tx=0x7fe6dc008040 comp rx=0 tx=0).stop 2024-03-28T19:04:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e8150b60 msgr2=0x7fe6e81474f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:45.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e8150b60 0x7fe6e81474f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009f40 tx=0x7fe6e0004a40 comp rx=0 tx=0).stop 2024-03-28T19:04:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 shutdown_connections 2024-03-28T19:04:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe6c806bbd0 0x7fe6c806e090 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6e8150b60 0x7fe6e81474f0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e80c8c70 0x7fe6e814f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 --2- 172.21.15.27:0/4062900728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e80c8470 0x7fe6e806c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 >> 172.21.15.27:0/4062900728 conn(0x7fe6e80aaab0 msgr2=0x7fe6e81503d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.891+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 shutdown_connections 2024-03-28T19:04:45.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:45.893+0000 7fe6ef800640 1 -- 172.21.15.27:0/4062900728 wait complete. 2024-03-28T19:04:45.910 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5769 2024-03-28T19:04:45.910 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.a 2024-03-28T19:04:46.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.251+0000 7f4aeb000640 1 Processor -- start 2024-03-28T19:04:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.251+0000 7f4aeb000640 1 -- start start 2024-03-28T19:04:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.253+0000 7f4aeb000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 0x7f4ae406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.253+0000 7f4aeb000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae414fa40 con 0x7f4ae40c21d0 2024-03-28T19:04:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.254+0000 7f4aeb000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae4155880 con 0x7f4ae40c1d50 2024-03-28T19:04:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.254+0000 7f4aeb000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae4156490 con 0x7f4ae414f660 2024-03-28T19:04:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.254+0000 7f4aea600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4ae40c21d0 0x7f4ae41594b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60150/0 (socket says 172.21.15.27:60150) 2024-03-28T19:04:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.254+0000 7f4aea600640 1 -- 172.21.15.27:0/2310412333 learned_addr learned my addr 172.21.15.27:0/2310412333 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:46.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.254+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 0x7f4ae406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:46.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.254+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652302534 0 0) 0x7f4ae414fa40 con 0x7f4ae40c21d0 2024-03-28T19:04:46.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.254+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4acc003520 con 0x7f4ae40c21d0 2024-03-28T19:04:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4262898289 0 0) 0x7f4acc003520 con 0x7f4ae40c21d0 2024-03-28T19:04:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 msgr2=0x7f4ae406ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 0x7f4ae406ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 >> v1:172.21.15.155:6789/0 conn(0x7f4ae40c1d50 legacy=0x7f4ae4158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae4155b60 con 0x7f4ae40c21d0 2024-03-28T19:04:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f4adc0019f0 con 0x7f4ae40c21d0 2024-03-28T19:04:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 0x7f4ae406ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:04:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 >> v1:172.21.15.27:6789/0 conn(0x7f4ae40c21d0 legacy=0x7f4ae41594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4acc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 0x7f4acc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc004d40 0x7f4acc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae4156490 con 0x7f4acc0039e0 2024-03-28T19:04:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc00bb50 con 0x7f4acc004d40 2024-03-28T19:04:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc00bd50 con 0x7f4ae414f660 2024-03-28T19:04:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.255+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4acc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.256+0000 7f4aea600640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc004d40 0x7f4acc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.256+0000 7f4ae9200640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 0x7f4acc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.256+0000 7f4ae9c00640 1 -- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 msgr2=0x7f4acc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.256+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae414f660 0x7f4acc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.256+0000 7f4ae9c00640 1 -- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc004d40 msgr2=0x7f4acc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.256+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc004d40 0x7f4acc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.256+0000 7f4ae9c00640 1 -- 172.21.15.27:0/2310412333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4acc003520 con 0x7f4acc0039e0 2024-03-28T19:04:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.256+0000 7f4aea600640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4acc004d40 0x7f4acc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:46.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.257+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4acc003e00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80098a0 tx=0x7f4acc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c66d5bf553613c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:46.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad800f040 con 0x7f4acc0039e0 2024-03-28T19:04:46.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad8004230 con 0x7f4acc0039e0 2024-03-28T19:04:46.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4ae3e00640 1 -- 172.21.15.27:0/2310412333 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8003610 con 0x7f4acc0039e0 2024-03-28T19:04:46.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4aeb000640 1 -- 172.21.15.27:0/2310412333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 msgr2=0x7f4acc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:46.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4aeb000640 1 --2- 172.21.15.27:0/2310412333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4acc003e00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80098a0 tx=0x7f4acc005830 comp rx=0 tx=0).stop 2024-03-28T19:04:46.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4aea600640 1 -- 172.21.15.27:0/2310412333 reap_dead start 2024-03-28T19:04:46.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4aeb000640 1 -- 172.21.15.27:0/2310412333 shutdown_connections 2024-03-28T19:04:46.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4aeb000640 1 -- 172.21.15.27:0/2310412333 >> 172.21.15.27:0/2310412333 conn(0x7f4ae406d060 msgr2=0x7f4ae406d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4aeb000640 1 -- 172.21.15.27:0/2310412333 shutdown_connections 2024-03-28T19:04:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.258+0000 7f4aeb000640 1 -- 172.21.15.27:0/2310412333 wait complete. 2024-03-28T19:04:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.260+0000 7f4aeb000640 1 Processor -- start 2024-03-28T19:04:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.260+0000 7f4aeb000640 1 -- start start 2024-03-28T19:04:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.261+0000 7f4aeb000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4ae414e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.261+0000 7f4aeb000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae41583b0 0x7f4ae41eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.261+0000 7f4aeb000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ae4158810 0x7f4ae41eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.261+0000 7f4aeb000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae4155570 con 0x7f4acc0039e0 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.261+0000 7f4aeb000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae4156120 con 0x7f4ae4158810 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.261+0000 7f4aeb000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae406e540 con 0x7f4ae41583b0 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4ae414e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4ae414e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43970/0 (socket says 172.21.15.27:43970) 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9c00640 1 -- 172.21.15.27:0/3798109933 learned_addr learned my addr 172.21.15.27:0/3798109933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9200640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae41583b0 0x7f4ae41eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4aea600640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ae4158810 0x7f4ae41eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9c00640 1 -- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae41583b0 msgr2=0x7f4ae41eb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae41583b0 0x7f4ae41eb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9c00640 1 -- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ae4158810 msgr2=0x7f4ae41eb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ae4158810 0x7f4ae41eb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4ae9c00640 1 -- 172.21.15.27:0/3798109933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae41ee620 con 0x7f4acc0039e0 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.262+0000 7f4aea600640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ae4158810 0x7f4ae41eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.263+0000 7f4ae9c00640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4ae414e140 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80099c0 tx=0x7f4ad8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.263+0000 7f4ae1600640 1 -- 172.21.15.27:0/3798109933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8048070 con 0x7f4acc0039e0 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.263+0000 7f4ae1600640 1 -- 172.21.15.27:0/3798109933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad8030430 con 0x7f4acc0039e0 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.263+0000 7f4ae1600640 1 -- 172.21.15.27:0/3798109933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8030800 con 0x7f4acc0039e0 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.263+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae41ee900 con 0x7f4acc0039e0 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.263+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae41eeec0 con 0x7f4acc0039e0 2024-03-28T19:04:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.264+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae4067dd0 con 0x7f4acc0039e0 2024-03-28T19:04:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.265+0000 7f4ae1600640 1 -- 172.21.15.27:0/3798109933 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4ad804b070 con 0x7f4acc0039e0 2024-03-28T19:04:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.266+0000 7f4ae1600640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ac406bbd0 0x7f4ac406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.266+0000 7f4ae1600640 1 -- 172.21.15.27:0/3798109933 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f4ad80acc20 con 0x7f4acc0039e0 2024-03-28T19:04:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.267+0000 7f4ae9200640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ac406bbd0 0x7f4ac406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.267+0000 7f4ae9200640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ac406bbd0 0x7f4ac406e090 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4ae414c850 tx=0x7f4ad0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.271+0000 7f4ae1600640 1 -- 172.21.15.27:0/3798109933 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4ad8034020 con 0x7f4acc0039e0 2024-03-28T19:04:46.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.593+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ae406e190 con 0x7f4acc0039e0 2024-03-28T19:04:46.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.594+0000 7f4ae1600640 1 -- 172.21.15.27:0/3798109933 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5465 (secure 0 0 0) 0x7f4ad807af60 con 0x7f4acc0039e0 2024-03-28T19:04:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:46.594 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:46.594 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 80 2024-03-28T19:04:46.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ac406bbd0 msgr2=0x7f4ac406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ac406bbd0 0x7f4ac406e090 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4ae414c850 tx=0x7f4ad0009040 comp rx=0 tx=0).stop 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 msgr2=0x7f4ae414e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4ae414e140 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80099c0 tx=0x7f4ad8004730 comp rx=0 tx=0).stop 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 shutdown_connections 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ac406bbd0 0x7f4ac406e090 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ae4158810 0x7f4ae41eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ae41583b0 0x7f4ae41eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 --2- 172.21.15.27:0/3798109933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4acc0039e0 0x7f4ae414e140 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 >> 172.21.15.27:0/3798109933 conn(0x7f4ae406d060 msgr2=0x7f4ae414a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 shutdown_connections 2024-03-28T19:04:46.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:46.597+0000 7f4aeb000640 1 -- 172.21.15.27:0/3798109933 wait complete. 2024-03-28T19:04:46.617 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:04:48.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:48.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.867+0000 7fe0aee00640 1 Processor -- start 2024-03-28T19:04:48.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.867+0000 7fe0aee00640 1 -- start start 2024-03-28T19:04:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.868+0000 7fe0aee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c58c0 0x7fe0a80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0aee00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe0a80be5d0 0x7fe0a8155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0aee00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a8156230 con 0x7fe0a80c8ed0 2024-03-28T19:04:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0aee00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a8158bd0 con 0x7fe0a80be5d0 2024-03-28T19:04:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0aee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a8158eb0 con 0x7fe0a80c58c0 2024-03-28T19:04:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0ada00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c58c0 0x7fe0a80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0ada00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c58c0 0x7fe0a80c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:33850/0 (socket says 172.21.15.27:33850) 2024-03-28T19:04:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0ada00640 1 -- 172.21.15.27:0/2555985338 learned_addr learned my addr 172.21.15.27:0/2555985338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0ad000640 1 --1- 172.21.15.27:0/2555985338 >> v1:172.21.15.155:6789/0 conn(0x7fe0a80c8ed0 0x7fe0a81555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:49274/0 (socket says 172.21.15.27:49274) 2024-03-28T19:04:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.869+0000 7fe0ae400640 1 --2- 172.21.15.27:0/2555985338 >> v2:172.21.15.155:3300/0 conn(0x7fe0a80be5d0 0x7fe0a8155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.870+0000 7fe0ada00640 1 -- 172.21.15.27:0/2555985338 >> v2:172.21.15.155:3300/0 conn(0x7fe0a80be5d0 msgr2=0x7fe0a8155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.870+0000 7fe0ada00640 1 --2- 172.21.15.27:0/2555985338 >> v2:172.21.15.155:3300/0 conn(0x7fe0a80be5d0 0x7fe0a8155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.870+0000 7fe0ada00640 1 -- 172.21.15.27:0/2555985338 >> v1:172.21.15.155:6789/0 conn(0x7fe0a80c8ed0 legacy=0x7fe0a81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.870+0000 7fe0ada00640 1 -- 172.21.15.27:0/2555985338 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a8159190 con 0x7fe0a80c58c0 2024-03-28T19:04:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.870+0000 7fe0a7e00640 1 -- 172.21.15.27:0/2555985338 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839117913 0 0) 0x7fe0a8156230 con 0x7fe0a80c8ed0 2024-03-28T19:04:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.872+0000 7fe0ada00640 1 --2- 172.21.15.27:0/2555985338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c58c0 0x7fe0a80c5ca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009870 tx=0x7fe09c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb7c5c91d5554c4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.872+0000 7fe0a7e00640 1 -- 172.21.15.27:0/2555985338 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c035020 con 0x7fe0a80c58c0 2024-03-28T19:04:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0a7e00640 1 -- 172.21.15.27:0/2555985338 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c031640 con 0x7fe0a80c58c0 2024-03-28T19:04:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0a7e00640 1 -- 172.21.15.27:0/2555985338 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c02d040 con 0x7fe0a80c58c0 2024-03-28T19:04:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0aee00640 1 -- 172.21.15.27:0/2555985338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c58c0 msgr2=0x7fe0a80c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0aee00640 1 --2- 172.21.15.27:0/2555985338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c58c0 0x7fe0a80c5ca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009870 tx=0x7fe09c02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:04:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0aee00640 1 -- 172.21.15.27:0/2555985338 shutdown_connections 2024-03-28T19:04:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0aee00640 1 --2- 172.21.15.27:0/2555985338 >> v2:172.21.15.155:3300/0 conn(0x7fe0a80be5d0 0x7fe0a8155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0aee00640 1 --2- 172.21.15.27:0/2555985338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c58c0 0x7fe0a80c5ca0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0aee00640 1 -- 172.21.15.27:0/2555985338 >> 172.21.15.27:0/2555985338 conn(0x7fe0a806b6f0 msgr2=0x7fe0a80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0aee00640 1 -- 172.21.15.27:0/2555985338 shutdown_connections 2024-03-28T19:04:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.873+0000 7fe0aee00640 1 -- 172.21.15.27:0/2555985338 wait complete. 2024-03-28T19:04:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.877+0000 7fe0aee00640 1 Processor -- start 2024-03-28T19:04:48.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.877+0000 7fe0aee00640 1 -- start start 2024-03-28T19:04:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.878+0000 7fe0aee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80be5d0 0x7fe0a8155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.878+0000 7fe0aee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0a80c58c0 0x7fe0a81deeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0aee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c8ed0 0x7fe0a80c2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:48.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0aee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a80b0820 con 0x7fe0a80be5d0 2024-03-28T19:04:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0aee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a8159750 con 0x7fe0a80c58c0 2024-03-28T19:04:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0aee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a80b09a0 con 0x7fe0a80c8ed0 2024-03-28T19:04:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0ada00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80be5d0 0x7fe0a8155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0ada00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80be5d0 0x7fe0a8155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43986/0 (socket says 172.21.15.27:43986) 2024-03-28T19:04:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0ada00640 1 -- 172.21.15.27:0/3245258122 learned_addr learned my addr 172.21.15.27:0/3245258122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0ad000640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0a80c58c0 0x7fe0a81deeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:48.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0ae400640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c8ed0 0x7fe0a80c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0ada00640 1 -- 172.21.15.27:0/3245258122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c8ed0 msgr2=0x7fe0a80c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.879+0000 7fe0ada00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c8ed0 0x7fe0a80c2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.880+0000 7fe0ada00640 1 -- 172.21.15.27:0/3245258122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0a80c58c0 msgr2=0x7fe0a81deeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.880+0000 7fe0ada00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0a80c58c0 0x7fe0a81deeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.880+0000 7fe0ada00640 1 -- 172.21.15.27:0/3245258122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a81e0be0 con 0x7fe0a80be5d0 2024-03-28T19:04:48.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.880+0000 7fe0ada00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80be5d0 0x7fe0a8155040 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0090b0 tx=0x7fe09c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.880+0000 7fe0a5600640 1 -- 172.21.15.27:0/3245258122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c040070 con 0x7fe0a80be5d0 2024-03-28T19:04:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.880+0000 7fe0a5600640 1 -- 172.21.15.27:0/3245258122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c0399f0 con 0x7fe0a80be5d0 2024-03-28T19:04:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.880+0000 7fe0a5600640 1 -- 172.21.15.27:0/3245258122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c035040 con 0x7fe0a80be5d0 2024-03-28T19:04:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.880+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a8155580 con 0x7fe0a80be5d0 2024-03-28T19:04:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.881+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a81e1160 con 0x7fe0a80be5d0 2024-03-28T19:04:48.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.882+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a8069640 con 0x7fe0a80be5d0 2024-03-28T19:04:48.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.883+0000 7fe0a5600640 1 -- 172.21.15.27:0/3245258122 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe09c038020 con 0x7fe0a80be5d0 2024-03-28T19:04:48.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.884+0000 7fe0a5600640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08806bbd0 0x7fe08806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:48.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.884+0000 7fe0a5600640 1 -- 172.21.15.27:0/3245258122 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7fe09c0acc90 con 0x7fe0a80be5d0 2024-03-28T19:04:48.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.884+0000 7fe0ad000640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08806bbd0 0x7fe08806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:48.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.885+0000 7fe0ad000640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08806bbd0 0x7fe08806e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80c1f40 tx=0x7fe094007c00 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:48.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:48.889+0000 7fe0a5600640 1 -- 172.21.15.27:0/3245258122 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe09c07aea0 con 0x7fe0a80be5d0 2024-03-28T19:04:49.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.189+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0a8156580 con 0x7fe0a80be5d0 2024-03-28T19:04:49.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.190+0000 7fe0a5600640 1 -- 172.21.15.27:0/3245258122 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5465 (secure 0 0 0) 0x7fe09c07a7a0 con 0x7fe0a80be5d0 2024-03-28T19:04:49.190 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:49.190 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:49.190 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 80 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08806bbd0 msgr2=0x7fe08806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08806bbd0 0x7fe08806e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80c1f40 tx=0x7fe094007c00 comp rx=0 tx=0).stop 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80be5d0 msgr2=0x7fe0a8155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80be5d0 0x7fe0a8155040 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0090b0 tx=0x7fe09c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 shutdown_connections 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08806bbd0 0x7fe08806e090 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0a80c8ed0 0x7fe0a80c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0a80c58c0 0x7fe0a81deeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 --2- 172.21.15.27:0/3245258122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80be5d0 0x7fe0a8155040 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:49.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.193+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 >> 172.21.15.27:0/3245258122 conn(0x7fe0a806b6f0 msgr2=0x7fe0a80c95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:49.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.194+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 shutdown_connections 2024-03-28T19:04:49.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:49.194+0000 7fe0aee00640 1 -- 172.21.15.27:0/3245258122 wait complete. 2024-03-28T19:04:49.214 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:04:51.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:51.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.438+0000 7fcc13200640 1 Processor -- start 2024-03-28T19:04:51.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.439+0000 7fcc13200640 1 -- start start 2024-03-28T19:04:51.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.440+0000 7fcc13200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcc140c58c0 0x7fcc140c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:51.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.440+0000 7fcc13200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc14158bd0 con 0x7fcc140c8ed0 2024-03-28T19:04:51.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.440+0000 7fcc13200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc14158eb0 con 0x7fcc140be5d0 2024-03-28T19:04:51.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.440+0000 7fcc13200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc14159190 con 0x7fcc140c58c0 2024-03-28T19:04:51.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.441+0000 7fcc0be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcc140c8ed0 0x7fcc141555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60156/0 (socket says 172.21.15.27:60156) 2024-03-28T19:04:51.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.441+0000 7fcc0be00640 1 -- 172.21.15.27:0/1576044784 learned_addr learned my addr 172.21.15.27:0/1576044784 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:51.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.441+0000 7fcc11e00640 1 --2- 172.21.15.27:0/1576044784 >> v2:172.21.15.155:3300/0 conn(0x7fcc140c58c0 0x7fcc140c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:51.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.441+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262063934 0 0) 0x7fcc14158bd0 con 0x7fcc140c8ed0 2024-03-28T19:04:51.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.441+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf8003520 con 0x7fcc140c8ed0 2024-03-28T19:04:51.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 744641330 0 0) 0x7fcbf8003520 con 0x7fcc140c8ed0 2024-03-28T19:04:51.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 >> v2:172.21.15.155:3300/0 conn(0x7fcc140c58c0 msgr2=0x7fcc140c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 --2- 172.21.15.27:0/1576044784 >> v2:172.21.15.155:3300/0 conn(0x7fcc140c58c0 0x7fcc140c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 >> v1:172.21.15.155:6789/0 conn(0x7fcc140be5d0 legacy=0x7fcc14155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc14159470 con 0x7fcc140c8ed0 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fcbfc001670 con 0x7fcc140c8ed0 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 >> v1:172.21.15.27:6789/0 conn(0x7fcc140c8ed0 legacy=0x7fcc141555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbf80039b0 0x7fcbf8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcbf8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbf80052e0 0x7fcbf800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc14159190 con 0x7fcbf8004800 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf800c1b0 con 0x7fcbf80052e0 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf800c3b0 con 0x7fcbf80039b0 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.442+0000 7fcc0be00640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcbf8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:51.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.443+0000 7fcc0be00640 1 -- 172.21.15.27:0/1576044784 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbf80039b0 msgr2=0x7fcbf8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:51.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.443+0000 7fcc0be00640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbf80039b0 0x7fcbf8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.443+0000 7fcc0be00640 1 -- 172.21.15.27:0/1576044784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbf80052e0 msgr2=0x7fcbf800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:51.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.443+0000 7fcc12800640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbf80052e0 0x7fcbf800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:51.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.443+0000 7fcc0be00640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbf80052e0 0x7fcbf800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.443+0000 7fcc0be00640 1 -- 172.21.15.27:0/1576044784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf8003520 con 0x7fcbf8004800 2024-03-28T19:04:51.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.443+0000 7fcc12800640 1 -- 172.21.15.27:0/1576044784 reap_dead start 2024-03-28T19:04:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.444+0000 7fcc0be00640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcbf8004c80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc001eb0 tx=0x7fcbf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f996a655bcade903 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.444+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc007d30 con 0x7fcbf8004800 2024-03-28T19:04:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.444+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc030070 con 0x7fcbf8004800 2024-03-28T19:04:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.444+0000 7fcc11400640 1 -- 172.21.15.27:0/1576044784 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc005d70 con 0x7fcbf8004800 2024-03-28T19:04:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.445+0000 7fcc13200640 1 -- 172.21.15.27:0/1576044784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 msgr2=0x7fcbf8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.445+0000 7fcc13200640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcbf8004c80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc001eb0 tx=0x7fcbf8005830 comp rx=0 tx=0).stop 2024-03-28T19:04:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.445+0000 7fcc13200640 1 -- 172.21.15.27:0/1576044784 shutdown_connections 2024-03-28T19:04:51.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.445+0000 7fcc13200640 1 --2- 172.21.15.27:0/1576044784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcbf8004c80 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.445+0000 7fcc13200640 1 -- 172.21.15.27:0/1576044784 >> 172.21.15.27:0/1576044784 conn(0x7fcc1406b6f0 msgr2=0x7fcc140acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.445+0000 7fcc13200640 1 -- 172.21.15.27:0/1576044784 shutdown_connections 2024-03-28T19:04:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.445+0000 7fcc13200640 1 -- 172.21.15.27:0/1576044784 wait complete. 2024-03-28T19:04:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.447+0000 7fcc13200640 1 Processor -- start 2024-03-28T19:04:51.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.447+0000 7fcc13200640 1 -- start start 2024-03-28T19:04:51.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc13200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcc141546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:51.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc13200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc1414c6d0 0x7fcc141eb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:51.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc13200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc141eb6a0 0x7fcc1414b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:51.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc13200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140b09a0 con 0x7fcbf8004800 2024-03-28T19:04:51.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc13200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140b0820 con 0x7fcc141eb6a0 2024-03-28T19:04:51.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc13200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140ad7d0 con 0x7fcc1414c6d0 2024-03-28T19:04:51.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc11e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcc141546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc11e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcc141546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44002/0 (socket says 172.21.15.27:44002) 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.448+0000 7fcc11e00640 1 -- 172.21.15.27:0/3474693834 learned_addr learned my addr 172.21.15.27:0/3474693834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc0be00640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc1414c6d0 0x7fcc141eb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc11e00640 1 -- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc1414c6d0 msgr2=0x7fcc141eb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc12800640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc141eb6a0 0x7fcc1414b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc11e00640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc1414c6d0 0x7fcc141eb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc11e00640 1 -- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc141eb6a0 msgr2=0x7fcc1414b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc11e00640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc141eb6a0 0x7fcc1414b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc11e00640 1 -- 172.21.15.27:0/3474693834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1414c3d0 con 0x7fcbf8004800 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc0be00640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc1414c6d0 0x7fcc141eb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:51.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc11e00640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcc141546f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400e890 tx=0x7fcc0400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:51.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc0a000640 1 -- 172.21.15.27:0/3474693834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04017070 con 0x7fcbf8004800 2024-03-28T19:04:51.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc0a000640 1 -- 172.21.15.27:0/3474693834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0400c8e0 con 0x7fcbf8004800 2024-03-28T19:04:51.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.449+0000 7fcc0a000640 1 -- 172.21.15.27:0/3474693834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400ccb0 con 0x7fcbf8004800 2024-03-28T19:04:51.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.450+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc141ef3c0 con 0x7fcbf8004800 2024-03-28T19:04:51.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.450+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc141552d0 con 0x7fcbf8004800 2024-03-28T19:04:51.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.451+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc140ac190 con 0x7fcbf8004800 2024-03-28T19:04:51.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.452+0000 7fcc0a000640 1 -- 172.21.15.27:0/3474693834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcc0401a070 con 0x7fcbf8004800 2024-03-28T19:04:51.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.453+0000 7fcc0a000640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:51.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.453+0000 7fcc0a000640 1 -- 172.21.15.27:0/3474693834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7fcc0408b560 con 0x7fcbf8004800 2024-03-28T19:04:51.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.453+0000 7fcc0be00640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:51.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.454+0000 7fcc0be00640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc034000 tx=0x7fcbfc039040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.463+0000 7fcc0a000640 1 -- 172.21.15.27:0/3474693834 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcc04059800 con 0x7fcbf8004800 2024-03-28T19:04:51.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.722+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc14069640 con 0x7fcbf8004800 2024-03-28T19:04:51.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.723+0000 7fcc0a000640 1 -- 172.21.15.27:0/3474693834 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5465 (secure 0 0 0) 0x7fcc04059100 con 0x7fcbf8004800 2024-03-28T19:04:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:51.723 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:51.723 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 80 2024-03-28T19:04:51.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 msgr2=0x7fcbf006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:51.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc034000 tx=0x7fcbfc039040 comp rx=0 tx=0).stop 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 msgr2=0x7fcc141546f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcc141546f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400e890 tx=0x7fcc0400ed50 comp rx=0 tx=0).stop 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 shutdown_connections 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc141eb6a0 0x7fcc1414b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc1414c6d0 0x7fcc141eb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 --2- 172.21.15.27:0/3474693834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbf8004800 0x7fcc141546f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 >> 172.21.15.27:0/3474693834 conn(0x7fcc1406b6f0 msgr2=0x7fcc140716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 shutdown_connections 2024-03-28T19:04:51.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:51.726+0000 7fcc13200640 1 -- 172.21.15.27:0/3474693834 wait complete. 2024-03-28T19:04:51.746 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:04:53.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:53.911 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-03-28T19:04:53.911 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:04:53.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:04:53.919 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:04:53.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.978+0000 7f431a000640 1 Processor -- start 2024-03-28T19:04:53.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.978+0000 7f431a000640 1 -- start start 2024-03-28T19:04:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.979+0000 7f431a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 0x7f43140c5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.979+0000 7f431a000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43140c7910 con 0x7f4314158580 2024-03-28T19:04:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.979+0000 7f431a000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43140c7bf0 con 0x7f43140c5e40 2024-03-28T19:04:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.979+0000 7f431a000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43140c76e0 con 0x7f43140be630 2024-03-28T19:04:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.979+0000 7f4313400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4314158580 0x7f43141560c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60158/0 (socket says 172.21.15.27:60158) 2024-03-28T19:04:53.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.979+0000 7f4313400640 1 -- 172.21.15.27:0/2808183921 learned_addr learned my addr 172.21.15.27:0/2808183921 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.980+0000 7f4312a00640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 0x7f43140c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:53.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.980+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087015131 0 0) 0x7f43140c7910 con 0x7f4314158580 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.980+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f8003520 con 0x7f4314158580 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.980+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204659999 0 0) 0x7f42f8003520 con 0x7f4314158580 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 msgr2=0x7f43140c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 0x7f43140c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 >> v1:172.21.15.155:6789/0 conn(0x7f43140c5e40 legacy=0x7f4314159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43140c7ed0 con 0x7f4314158580 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f4304001670 con 0x7f4314158580 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 >> v1:172.21.15.27:6789/0 conn(0x7f4314158580 legacy=0x7f43141560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42f80039e0 0x7f42f8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42f8004800 0x7f42f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 0x7f42f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f800ba70 con 0x7f42f80039e0 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f800bc20 con 0x7f42f8004800 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f800be20 con 0x7f43140be630 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4313400640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42f80039e0 0x7f42f8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:53.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.981+0000 7f4312a00640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42f8004800 0x7f42f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.982+0000 7f4313e00640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 0x7f42f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.982+0000 7f4313400640 1 -- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 msgr2=0x7f42f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.982+0000 7f4313400640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 0x7f42f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.982+0000 7f4313400640 1 -- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42f8004800 msgr2=0x7f42f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.982+0000 7f4313400640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42f8004800 0x7f42f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:53.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.982+0000 7f4313400640 1 -- 172.21.15.27:0/2808183921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f8003520 con 0x7f42f80039e0 2024-03-28T19:04:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.982+0000 7f4312a00640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42f8004800 0x7f42f8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:04:53.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.982+0000 7f4313e00640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43140be630 0x7f42f80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.983+0000 7f4313400640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42f80039e0 0x7f42f8003e00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4304007fd0 tx=0x7f42f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fb9bd8857f188 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.984+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304007c20 con 0x7f42f80039e0 2024-03-28T19:04:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.984+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4304030070 con 0x7f42f80039e0 2024-03-28T19:04:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.984+0000 7f4312000640 1 -- 172.21.15.27:0/2808183921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304005d30 con 0x7f42f80039e0 2024-03-28T19:04:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.984+0000 7f431a000640 1 -- 172.21.15.27:0/2808183921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42f80039e0 msgr2=0x7f42f8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.984+0000 7f431a000640 1 --2- 172.21.15.27:0/2808183921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42f80039e0 0x7f42f8003e00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4304007fd0 tx=0x7f42f8005830 comp rx=0 tx=0).stop 2024-03-28T19:04:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.984+0000 7f4313e00640 1 -- 172.21.15.27:0/2808183921 reap_dead start 2024-03-28T19:04:53.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.985+0000 7f431a000640 1 -- 172.21.15.27:0/2808183921 shutdown_connections 2024-03-28T19:04:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.985+0000 7f431a000640 1 -- 172.21.15.27:0/2808183921 >> 172.21.15.27:0/2808183921 conn(0x7f43140caf80 msgr2=0x7f43140cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.985+0000 7f431a000640 1 -- 172.21.15.27:0/2808183921 shutdown_connections 2024-03-28T19:04:53.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.985+0000 7f431a000640 1 -- 172.21.15.27:0/2808183921 wait complete. 2024-03-28T19:04:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.987+0000 7f431a000640 1 Processor -- start 2024-03-28T19:04:53.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.987+0000 7f431a000640 1 -- start start 2024-03-28T19:04:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f431a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42f80039e0 0x7f43141507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f431a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43140adba0 0x7f43140c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:53.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f431a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140c4670 0x7f43141ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f431a000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4314158cc0 con 0x7f43140c4670 2024-03-28T19:04:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f431a000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43140c6dd0 con 0x7f43140adba0 2024-03-28T19:04:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f431a000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4314158960 con 0x7f42f80039e0 2024-03-28T19:04:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f4313e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140c4670 0x7f43141ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f4313e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140c4670 0x7f43141ee670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44018/0 (socket says 172.21.15.27:44018) 2024-03-28T19:04:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.988+0000 7f4313e00640 1 -- 172.21.15.27:0/736270939 learned_addr learned my addr 172.21.15.27:0/736270939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4312a00640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43140adba0 0x7f43140c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4313400640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42f80039e0 0x7f43141507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4313e00640 1 -- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42f80039e0 msgr2=0x7f43141507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4313e00640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42f80039e0 0x7f43141507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4313e00640 1 -- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43140adba0 msgr2=0x7f43140c4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4313e00640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43140adba0 0x7f43140c4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4313e00640 1 -- 172.21.15.27:0/736270939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43141eb610 con 0x7f43140c4670 2024-03-28T19:04:53.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4312a00640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43140adba0 0x7f43140c4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:04:53.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.989+0000 7f4313e00640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140c4670 0x7f43141ee670 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f43140c8500 tx=0x7f430c00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:53.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.990+0000 7f430b400640 1 -- 172.21.15.27:0/736270939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c003970 con 0x7f43140c4670 2024-03-28T19:04:53.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.990+0000 7f430b400640 1 -- 172.21.15.27:0/736270939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c003b50 con 0x7f43140c4670 2024-03-28T19:04:53.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.990+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43140c4af0 con 0x7f43140c4670 2024-03-28T19:04:53.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.990+0000 7f430b400640 1 -- 172.21.15.27:0/736270939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c005070 con 0x7f43140c4670 2024-03-28T19:04:53.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.990+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43140c2180 con 0x7f43140c4670 2024-03-28T19:04:53.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.992+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431404f400 con 0x7f43140c4670 2024-03-28T19:04:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.993+0000 7f430b400640 1 -- 172.21.15.27:0/736270939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f430c01b020 con 0x7f43140c4670 2024-03-28T19:04:53.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.994+0000 7f430b400640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42f406bbd0 0x7f42f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:53.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.994+0000 7f430b400640 1 -- 172.21.15.27:0/736270939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f430c08abd0 con 0x7f43140c4670 2024-03-28T19:04:53.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.995+0000 7f4313400640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42f406bbd0 0x7f42f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:53.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:53.996+0000 7f4313400640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42f406bbd0 0x7f42f406e090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4304034000 tx=0x7f4304007000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:53.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.000+0000 7f430b400640 1 -- 172.21.15.27:0/736270939 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f43140c4af0 con 0x7f43140c4670 2024-03-28T19:04:54.089 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:04:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.140+0000 7f5a2cc00640 1 Processor -- start 2024-03-28T19:04:54.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.140+0000 7f5a2cc00640 1 -- start start 2024-03-28T19:04:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.141+0000 7f5a2cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280c58c0 0x7f5a280c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.141+0000 7f5a2cc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5a280be5d0 0x7f5a28155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.141+0000 7f5a2cc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a28156230 con 0x7f5a280c8ed0 2024-03-28T19:04:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.141+0000 7f5a2cc00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a28158bd0 con 0x7f5a280be5d0 2024-03-28T19:04:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.141+0000 7f5a2cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a28158eb0 con 0x7f5a280c58c0 2024-03-28T19:04:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.142+0000 7f5a26a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5a280c8ed0 0x7f5a281555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60176/0 (socket says 172.21.15.27:60176) 2024-03-28T19:04:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.142+0000 7f5a26a00640 1 -- 172.21.15.27:0/1959668133 learned_addr learned my addr 172.21.15.27:0/1959668133 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.142+0000 7f5a27400640 1 --2- 172.21.15.27:0/1959668133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280c58c0 0x7f5a280c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.142+0000 7f5a27e00640 1 --2- 172.21.15.27:0/1959668133 >> v2:172.21.15.155:3300/0 conn(0x7f5a280be5d0 0x7f5a28155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.142+0000 7f5a26000640 1 -- 172.21.15.27:0/1959668133 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610644684 0 0) 0x7f5a28156230 con 0x7f5a280c8ed0 2024-03-28T19:04:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.142+0000 7f5a26000640 1 -- 172.21.15.27:0/1959668133 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a0c003520 con 0x7f5a280c8ed0 2024-03-28T19:04:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.143+0000 7f5a27400640 1 -- 172.21.15.27:0/1959668133 >> v2:172.21.15.155:3300/0 conn(0x7f5a280be5d0 msgr2=0x7f5a28155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.143+0000 7f5a27400640 1 --2- 172.21.15.27:0/1959668133 >> v2:172.21.15.155:3300/0 conn(0x7f5a280be5d0 0x7f5a28155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.143+0000 7f5a27400640 1 -- 172.21.15.27:0/1959668133 >> v1:172.21.15.27:6789/0 conn(0x7f5a280c8ed0 legacy=0x7f5a281555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:54.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.143+0000 7f5a27400640 1 -- 172.21.15.27:0/1959668133 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a28159190 con 0x7f5a280c58c0 2024-03-28T19:04:54.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.144+0000 7f5a27400640 1 --2- 172.21.15.27:0/1959668133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280c58c0 0x7f5a280c5ca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5a180097a0 tx=0x7f5a1802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97b57a6bb8496b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a26000640 1 -- 172.21.15.27:0/1959668133 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18031780 con 0x7f5a280c58c0 2024-03-28T19:04:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a26000640 1 -- 172.21.15.27:0/1959668133 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a18037070 con 0x7f5a280c58c0 2024-03-28T19:04:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a26000640 1 -- 172.21.15.27:0/1959668133 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18031b40 con 0x7f5a280c58c0 2024-03-28T19:04:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a2cc00640 1 -- 172.21.15.27:0/1959668133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280c58c0 msgr2=0x7f5a280c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:54.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/1959668133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280c58c0 0x7f5a280c5ca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5a180097a0 tx=0x7f5a1802b820 comp rx=0 tx=0).stop 2024-03-28T19:04:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a2cc00640 1 -- 172.21.15.27:0/1959668133 shutdown_connections 2024-03-28T19:04:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/1959668133 >> v2:172.21.15.155:3300/0 conn(0x7f5a280be5d0 0x7f5a28155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/1959668133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280c58c0 0x7f5a280c5ca0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.145+0000 7f5a2cc00640 1 -- 172.21.15.27:0/1959668133 >> 172.21.15.27:0/1959668133 conn(0x7f5a2806b6f0 msgr2=0x7f5a280acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.146+0000 7f5a2cc00640 1 -- 172.21.15.27:0/1959668133 shutdown_connections 2024-03-28T19:04:54.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.146+0000 7f5a2cc00640 1 -- 172.21.15.27:0/1959668133 wait complete. 2024-03-28T19:04:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.147+0000 7f5a2cc00640 1 Processor -- start 2024-03-28T19:04:54.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.147+0000 7f5a2cc00640 1 -- start start 2024-03-28T19:04:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.148+0000 7f5a2cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280be5d0 0x7f5a280c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.148+0000 7f5a2cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a280c58c0 0x7f5a281521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.148+0000 7f5a2cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a280c8ed0 0x7f5a281e1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.148+0000 7f5a2cc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a280b0820 con 0x7f5a280c8ed0 2024-03-28T19:04:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.148+0000 7f5a2cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a28159750 con 0x7f5a280c58c0 2024-03-28T19:04:54.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.148+0000 7f5a2cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a280b09a0 con 0x7f5a280be5d0 2024-03-28T19:04:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a280c8ed0 0x7f5a281e1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a26a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a280c58c0 0x7f5a281521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a280c8ed0 0x7f5a281e1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44056/0 (socket says 172.21.15.27:44056) 2024-03-28T19:04:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a27e00640 1 -- 172.21.15.27:0/3643687890 learned_addr learned my addr 172.21.15.27:0/3643687890 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:54.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a27400640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280be5d0 0x7f5a280c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a26a00640 1 -- 172.21.15.27:0/3643687890 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280be5d0 msgr2=0x7f5a280c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a26a00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280be5d0 0x7f5a280c3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a26a00640 1 -- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a280c8ed0 msgr2=0x7f5a281e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.149+0000 7f5a26a00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a280c8ed0 0x7f5a281e1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.150+0000 7f5a26a00640 1 -- 172.21.15.27:0/3643687890 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a28153980 con 0x7f5a280c58c0 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.150+0000 7f5a27e00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a280c8ed0 0x7f5a281e1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.150+0000 7f5a26a00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a280c58c0 0x7f5a281521b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400c3c0 tx=0x7f5a14001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.150+0000 7f5a1f400640 1 -- 172.21.15.27:0/3643687890 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14017070 con 0x7f5a280c58c0 2024-03-28T19:04:54.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.150+0000 7f5a1f400640 1 -- 172.21.15.27:0/3643687890 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a14005ce0 con 0x7f5a280c58c0 2024-03-28T19:04:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.150+0000 7f5a1f400640 1 -- 172.21.15.27:0/3643687890 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14007310 con 0x7f5a280c58c0 2024-03-28T19:04:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.151+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a28151f10 con 0x7f5a280c58c0 2024-03-28T19:04:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.151+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a281ef180 con 0x7f5a280c58c0 2024-03-28T19:04:54.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.152+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a28069640 con 0x7f5a280c58c0 2024-03-28T19:04:54.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.154+0000 7f5a1f400640 1 -- 172.21.15.27:0/3643687890 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5a1401a070 con 0x7f5a280c58c0 2024-03-28T19:04:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.155+0000 7f5a1f400640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5a0806bc00 0x7f5a0806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.155+0000 7f5a1f400640 1 -- 172.21.15.27:0/3643687890 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7669+0+0 (secure 0 0 0) 0x7f5a1408a7f0 con 0x7f5a280c58c0 2024-03-28T19:04:54.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.155+0000 7f5a27400640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5a0806bc00 0x7f5a0806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:54.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.156+0000 7f5a27400640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5a0806bc00 0x7f5a0806e0c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5a18036000 tx=0x7f5a1803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:54.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.173+0000 7f5a1f400640 1 -- 172.21.15.27:0/3643687890 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5a14021020 con 0x7f5a280c58c0 2024-03-28T19:04:54.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.270+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43140729e0 con 0x7f43140c4670 2024-03-28T19:04:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.271+0000 7f430b400640 1 -- 172.21.15.27:0/736270939 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5465 (secure 0 0 0) 0x7f43141eb610 con 0x7f43140c4670 2024-03-28T19:04:54.270 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:54.270 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:54.270 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 80 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42f406bbd0 msgr2=0x7f42f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42f406bbd0 0x7f42f406e090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4304034000 tx=0x7f4304007000 comp rx=0 tx=0).stop 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140c4670 msgr2=0x7f43141ee670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140c4670 0x7f43141ee670 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f43140c8500 tx=0x7f430c00d9d0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 shutdown_connections 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42f406bbd0 0x7f42f406e090 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140c4670 0x7f43141ee670 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43140adba0 0x7f43140c4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 --2- 172.21.15.27:0/736270939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42f80039e0 0x7f43141507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.273+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 >> 172.21.15.27:0/736270939 conn(0x7f43140caf80 msgr2=0x7f4314154260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.274+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 shutdown_connections 2024-03-28T19:04:54.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.274+0000 7f431a000640 1 -- 172.21.15.27:0/736270939 wait complete. 2024-03-28T19:04:54.299 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:04:54.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.437+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a28156580 con 0x7f5a280c58c0 2024-03-28T19:04:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.439+0000 7f5a1f400640 1 -- 172.21.15.27:0/3643687890 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5465 (secure 0 0 0) 0x7f5a14058570 con 0x7f5a280c58c0 2024-03-28T19:04:54.438 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:54.439 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:03:59.309496+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"b","rank":0,"incarnation":72,"state":"up:active","state_seq":13,"addr":"172.21.15.155:6835/331868181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":331868181},{"type":"v1","addr":"172.21.15.155:6835","nonce":331868181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:54.439 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 80 2024-03-28T19:04:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.441+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5a0806bc00 msgr2=0x7f5a0806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.441+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5a0806bc00 0x7f5a0806e0c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5a18036000 tx=0x7f5a1803d040 comp rx=0 tx=0).stop 2024-03-28T19:04:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.441+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a280c58c0 msgr2=0x7f5a281521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.441+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a280c58c0 0x7f5a281521b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400c3c0 tx=0x7f5a14001e70 comp rx=0 tx=0).stop 2024-03-28T19:04:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.442+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 shutdown_connections 2024-03-28T19:04:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.442+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5a0806bc00 0x7f5a0806e0c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.442+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a280c8ed0 0x7f5a281e1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.442+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a280c58c0 0x7f5a281521b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.442+0000 7f5a2cc00640 1 --2- 172.21.15.27:0/3643687890 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a280be5d0 0x7f5a280c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.442+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 >> 172.21.15.27:0/3643687890 conn(0x7f5a2806b6f0 msgr2=0x7f5a280ca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.442+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 shutdown_connections 2024-03-28T19:04:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:54.442+0000 7f5a2cc00640 1 -- 172.21.15.27:0/3643687890 wait complete. 2024-03-28T19:04:54.468 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:04:56.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:56.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.517+0000 7f76f3600640 1 Processor -- start 2024-03-28T19:04:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.517+0000 7f76f3600640 1 -- start start 2024-03-28T19:04:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76f3600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76ec0c4dc0 0x7f76ec070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76f3600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f76ec070a80 0x7f76ec15ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76f3600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec0c52e0 con 0x7f76ec152bd0 2024-03-28T19:04:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76f3600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ec1557e0 con 0x7f76ec0c4dc0 2024-03-28T19:04:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76f3600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ec155ac0 con 0x7f76ec070a80 2024-03-28T19:04:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76f0c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f76ec152bd0 0x7f76ec06eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36422/0 (socket says 172.21.15.27:36422) 2024-03-28T19:04:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76f0c00640 1 -- 172.21.15.27:0/448730497 learned_addr learned my addr 172.21.15.27:0/448730497 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76ebe00640 1 --2- 172.21.15.27:0/448730497 >> v2:172.21.15.27:3300/0 conn(0x7f76ec0c4dc0 0x7f76ec070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.519+0000 7f76f1600640 1 --2- 172.21.15.27:0/448730497 >> v2:172.21.15.155:3300/0 conn(0x7f76ec070a80 0x7f76ec15ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.520+0000 7f76ebe00640 1 -- 172.21.15.27:0/448730497 >> v2:172.21.15.155:3300/0 conn(0x7f76ec070a80 msgr2=0x7f76ec15ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.520+0000 7f76ebe00640 1 --2- 172.21.15.27:0/448730497 >> v2:172.21.15.155:3300/0 conn(0x7f76ec070a80 0x7f76ec15ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.520+0000 7f76ebe00640 1 -- 172.21.15.27:0/448730497 >> v1:172.21.15.27:6789/0 conn(0x7f76ec152bd0 legacy=0x7f76ec06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.520+0000 7f76ebe00640 1 -- 172.21.15.27:0/448730497 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec155da0 con 0x7f76ec0c4dc0 2024-03-28T19:04:56.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.521+0000 7f76ebe00640 1 --2- 172.21.15.27:0/448730497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec0c4dc0 0x7f76ec070540 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f76ec0c4a50 tx=0x7f76d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ed1d2038727366 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.521+0000 7f76eb400640 1 -- 172.21.15.27:0/448730497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d8035020 con 0x7f76ec0c4dc0 2024-03-28T19:04:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.521+0000 7f76eb400640 1 -- 172.21.15.27:0/448730497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d8031830 con 0x7f76ec0c4dc0 2024-03-28T19:04:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.521+0000 7f76eb400640 1 -- 172.21.15.27:0/448730497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d8031c00 con 0x7f76ec0c4dc0 2024-03-28T19:04:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.522+0000 7f76f3600640 1 -- 172.21.15.27:0/448730497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec0c4dc0 msgr2=0x7f76ec070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.522+0000 7f76f3600640 1 --2- 172.21.15.27:0/448730497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec0c4dc0 0x7f76ec070540 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f76ec0c4a50 tx=0x7f76d802b8d0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.522+0000 7f76f3600640 1 -- 172.21.15.27:0/448730497 shutdown_connections 2024-03-28T19:04:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.522+0000 7f76f3600640 1 --2- 172.21.15.27:0/448730497 >> v2:172.21.15.155:3300/0 conn(0x7f76ec070a80 0x7f76ec15ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.522+0000 7f76f3600640 1 --2- 172.21.15.27:0/448730497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec0c4dc0 0x7f76ec070540 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.522+0000 7f76f3600640 1 -- 172.21.15.27:0/448730497 >> 172.21.15.27:0/448730497 conn(0x7f76ec066dd0 msgr2=0x7f76ec0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.522+0000 7f76f3600640 1 -- 172.21.15.27:0/448730497 shutdown_connections 2024-03-28T19:04:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.522+0000 7f76f3600640 1 -- 172.21.15.27:0/448730497 wait complete. 2024-03-28T19:04:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.524+0000 7f76f3600640 1 Processor -- start 2024-03-28T19:04:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.524+0000 7f76f3600640 1 -- start start 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.525+0000 7f76f3600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec070a80 0x7f76ec0c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.525+0000 7f76f3600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76ec0c4dc0 0x7f76ec14d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.525+0000 7f76f3600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76ec152bd0 0x7f76ec14d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.525+0000 7f76f3600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec0c6c70 con 0x7f76ec070a80 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.525+0000 7f76f3600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec152fb0 con 0x7f76ec0c4dc0 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.525+0000 7f76f3600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec0c6df0 con 0x7f76ec152bd0 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.525+0000 7f76f0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec070a80 0x7f76ec0c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76f0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec070a80 0x7f76ec0c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46270/0 (socket says 172.21.15.27:46270) 2024-03-28T19:04:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76f0c00640 1 -- 172.21.15.27:0/2217655393 learned_addr learned my addr 172.21.15.27:0/2217655393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76ebe00640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76ec0c4dc0 0x7f76ec14d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76f1600640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76ec152bd0 0x7f76ec14d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76f0c00640 1 -- 172.21.15.27:0/2217655393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76ec152bd0 msgr2=0x7f76ec14d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76f0c00640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76ec152bd0 0x7f76ec14d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76f0c00640 1 -- 172.21.15.27:0/2217655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76ec0c4dc0 msgr2=0x7f76ec14d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76f0c00640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76ec0c4dc0 0x7f76ec14d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.526+0000 7f76f0c00640 1 -- 172.21.15.27:0/2217655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec1e1770 con 0x7f76ec070a80 2024-03-28T19:04:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.528+0000 7f76f0c00640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec070a80 0x7f76ec0c8a40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f76e000c430 tx=0x7f76e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:56.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.528+0000 7f76e8c00640 1 -- 172.21.15.27:0/2217655393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e00058b0 con 0x7f76ec070a80 2024-03-28T19:04:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.528+0000 7f76e8c00640 1 -- 172.21.15.27:0/2217655393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e0019030 con 0x7f76ec070a80 2024-03-28T19:04:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.528+0000 7f76e8c00640 1 -- 172.21.15.27:0/2217655393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0005cd0 con 0x7f76ec070a80 2024-03-28T19:04:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.528+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ec1e1a50 con 0x7f76ec070a80 2024-03-28T19:04:56.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.529+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ec154db0 con 0x7f76ec070a80 2024-03-28T19:04:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.530+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ec04f400 con 0x7f76ec070a80 2024-03-28T19:04:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.532+0000 7f76e8c00640 1 -- 172.21.15.27:0/2217655393 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f76e0007070 con 0x7f76ec070a80 2024-03-28T19:04:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.534+0000 7f76e8c00640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f76cc06bbd0 0x7f76cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.534+0000 7f76e8c00640 1 -- 172.21.15.27:0/2217655393 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7f76e0092ec0 con 0x7f76ec070a80 2024-03-28T19:04:56.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.534+0000 7f76ebe00640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f76cc06bbd0 0x7f76cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.535+0000 7f76ebe00640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f76cc06bbd0 0x7f76cc06e090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f76d8036000 tx=0x7f76d8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:56.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.542+0000 7f76e8c00640 1 -- 172.21.15.27:0/2217655393 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f76e00611a0 con 0x7f76ec070a80 2024-03-28T19:04:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.707+0000 7f0261000640 1 Processor -- start 2024-03-28T19:04:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.707+0000 7f0261000640 1 -- start start 2024-03-28T19:04:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.708+0000 7f0261000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f025c0c5320 0x7f025c06e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.708+0000 7f0261000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c156500 0x7f025c0c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.708+0000 7f0261000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025c159c50 con 0x7f025c156120 2024-03-28T19:04:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.708+0000 7f0261000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f025c0c4d70 con 0x7f025c0c5320 2024-03-28T19:04:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.708+0000 7f0261000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0c1030 con 0x7f025c156500 2024-03-28T19:04:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.709+0000 7f025be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c156500 0x7f025c0c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.709+0000 7f025aa00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f025c0c5320 0x7f025c06e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.709+0000 7f025be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c156500 0x7f025c0c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:43876/0 (socket says 172.21.15.27:43876) 2024-03-28T19:04:56.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.709+0000 7f025be00640 1 -- 172.21.15.27:0/4272950828 learned_addr learned my addr 172.21.15.27:0/4272950828 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:56.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.710+0000 7f025aa00640 1 -- 172.21.15.27:0/4272950828 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c156500 msgr2=0x7f025c0c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:56.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.710+0000 7f025aa00640 1 --2- 172.21.15.27:0/4272950828 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c156500 0x7f025c0c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.710+0000 7f025aa00640 1 -- 172.21.15.27:0/4272950828 >> v1:172.21.15.155:6789/0 conn(0x7f025c156120 legacy=0x7f025c0c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.710+0000 7f025aa00640 1 -- 172.21.15.27:0/4272950828 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c0c4a40 con 0x7f025c0c5320 2024-03-28T19:04:56.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.711+0000 7f025aa00640 1 --2- 172.21.15.27:0/4272950828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c0c5320 0x7f025c06e050 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f02480097a0 tx=0x7f024802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b2f7ba187feb4e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:56.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f025a000640 1 -- 172.21.15.27:0/4272950828 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248039070 con 0x7f025c0c5320 2024-03-28T19:04:56.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f025a000640 1 -- 172.21.15.27:0/4272950828 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02480316f0 con 0x7f025c0c5320 2024-03-28T19:04:56.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f025a000640 1 -- 172.21.15.27:0/4272950828 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248031ac0 con 0x7f025c0c5320 2024-03-28T19:04:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f0261000640 1 -- 172.21.15.27:0/4272950828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c0c5320 msgr2=0x7f025c06e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f0261000640 1 --2- 172.21.15.27:0/4272950828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c0c5320 0x7f025c06e050 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f02480097a0 tx=0x7f024802b790 comp rx=0 tx=0).stop 2024-03-28T19:04:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f0261000640 1 -- 172.21.15.27:0/4272950828 shutdown_connections 2024-03-28T19:04:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f0261000640 1 --2- 172.21.15.27:0/4272950828 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c156500 0x7f025c0c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f0261000640 1 --2- 172.21.15.27:0/4272950828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c0c5320 0x7f025c06e050 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.712+0000 7f0261000640 1 -- 172.21.15.27:0/4272950828 >> 172.21.15.27:0/4272950828 conn(0x7f025c06b0c0 msgr2=0x7f025c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.713+0000 7f0261000640 1 -- 172.21.15.27:0/4272950828 shutdown_connections 2024-03-28T19:04:56.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.713+0000 7f0261000640 1 -- 172.21.15.27:0/4272950828 wait complete. 2024-03-28T19:04:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.714+0000 7f0261000640 1 Processor -- start 2024-03-28T19:04:56.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.714+0000 7f0261000640 1 -- start start 2024-03-28T19:04:56.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.715+0000 7f0261000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c0c5320 0x7f025c14a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.715+0000 7f0261000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c156120 0x7f025c14a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.715+0000 7f0261000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f025c156500 0x7f025c14dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.715+0000 7f0261000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0c73d0 con 0x7f025c156500 2024-03-28T19:04:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.715+0000 7f0261000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0c7250 con 0x7f025c156120 2024-03-28T19:04:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.715+0000 7f0261000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0c29a0 con 0x7f025c0c5320 2024-03-28T19:04:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.716+0000 7f025be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f025c156500 0x7f025c14dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.716+0000 7f025be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f025c156500 0x7f025c14dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46272/0 (socket says 172.21.15.27:46272) 2024-03-28T19:04:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.716+0000 7f025be00640 1 -- 172.21.15.27:0/149532363 learned_addr learned my addr 172.21.15.27:0/149532363 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.716+0000 7f025b400640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c0c5320 0x7f025c14a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.716+0000 7f025aa00640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c156120 0x7f025c14a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f025be00640 1 -- 172.21.15.27:0/149532363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c0c5320 msgr2=0x7f025c14a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f025be00640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c0c5320 0x7f025c14a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f025be00640 1 -- 172.21.15.27:0/149532363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c156120 msgr2=0x7f025c14a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f025be00640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c156120 0x7f025c14a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f025be00640 1 -- 172.21.15.27:0/149532363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c14cbb0 con 0x7f025c156500 2024-03-28T19:04:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f025be00640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f025c156500 0x7f025c14dc90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f025400efc0 tx=0x7f025400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f0253400640 1 -- 172.21.15.27:0/149532363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254009e60 con 0x7f025c156500 2024-03-28T19:04:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f0253400640 1 -- 172.21.15.27:0/149532363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025400cd20 con 0x7f025c156500 2024-03-28T19:04:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.717+0000 7f0253400640 1 -- 172.21.15.27:0/149532363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254003bd0 con 0x7f025c156500 2024-03-28T19:04:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.718+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025c14b020 con 0x7f025c156500 2024-03-28T19:04:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.718+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f025c070e50 con 0x7f025c156500 2024-03-28T19:04:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.719+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025c067dd0 con 0x7f025c156500 2024-03-28T19:04:56.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.720+0000 7f0253400640 1 -- 172.21.15.27:0/149532363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0254005d10 con 0x7f025c156500 2024-03-28T19:04:56.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.722+0000 7f0253400640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f023c06bbd0 0x7f023c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:56.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.722+0000 7f0253400640 1 -- 172.21.15.27:0/149532363 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7f025408b5d0 con 0x7f025c156500 2024-03-28T19:04:56.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.722+0000 7f025b400640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f023c06bbd0 0x7f023c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:56.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.724+0000 7f025b400640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f023c06bbd0 0x7f023c06e090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f024c001780 tx=0x7f024c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:56.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.727+0000 7f0253400640 1 -- 172.21.15.27:0/149532363 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0254022020 con 0x7f025c156500 2024-03-28T19:04:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.891+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76ec14dc70 con 0x7f76ec070a80 2024-03-28T19:04:56.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.891+0000 7f76e8c00640 1 -- 172.21.15.27:0/2217655393 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f76e0060aa0 con 0x7f76ec070a80 2024-03-28T19:04:56.891 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:54.587987+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:56.891 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 82 2024-03-28T19:04:56.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f76cc06bbd0 msgr2=0x7f76cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f76cc06bbd0 0x7f76cc06e090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f76d8036000 tx=0x7f76d8008030 comp rx=0 tx=0).stop 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec070a80 msgr2=0x7f76ec0c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec070a80 0x7f76ec0c8a40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f76e000c430 tx=0x7f76e000c8f0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 shutdown_connections 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f76cc06bbd0 0x7f76cc06e090 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76ec152bd0 0x7f76ec14d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76ec0c4dc0 0x7f76ec14d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 --2- 172.21.15.27:0/2217655393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76ec070a80 0x7f76ec0c8a40 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 >> 172.21.15.27:0/2217655393 conn(0x7f76ec066dd0 msgr2=0x7f76ec06eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 shutdown_connections 2024-03-28T19:04:56.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:56.894+0000 7f76f3600640 1 -- 172.21.15.27:0/2217655393 wait complete. 2024-03-28T19:04:56.909 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-03-28T19:04:56.910 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:04:56.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.135+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f025c06de30 con 0x7f025c156500 2024-03-28T19:04:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.135+0000 7f0253400640 1 -- 172.21.15.27:0/149532363 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f0254010040 con 0x7f025c156500 2024-03-28T19:04:57.135 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:57.135 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:54.587987+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 82 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f023c06bbd0 msgr2=0x7f023c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f023c06bbd0 0x7f023c06e090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f024c001780 tx=0x7f024c009040 comp rx=0 tx=0).stop 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f025c156500 msgr2=0x7f025c14dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f025c156500 0x7f025c14dc90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f025400efc0 tx=0x7f025400c490 comp rx=0 tx=0).stop 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 shutdown_connections 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f023c06bbd0 0x7f023c06e090 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f025c156500 0x7f025c14dc90 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f025c156120 0x7f025c14a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 --2- 172.21.15.27:0/149532363 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f025c0c5320 0x7f025c14a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 >> 172.21.15.27:0/149532363 conn(0x7f025c06b0c0 msgr2=0x7f025c0c06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 shutdown_connections 2024-03-28T19:04:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.139+0000 7f0261000640 1 -- 172.21.15.27:0/149532363 wait complete. 2024-03-28T19:04:57.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.149+0000 7f6edd600640 1 Processor -- start 2024-03-28T19:04:57.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.149+0000 7f6edd600640 1 -- start start 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.150+0000 7f6edd600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ed80c7970 0x7f6ed80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.150+0000 7f6edd600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6ed8155730 0x7f6ed8155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.150+0000 7f6edd600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed80c7310 con 0x7f6ed8152f80 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.150+0000 7f6edd600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed8158d40 con 0x7f6ed80c7970 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.150+0000 7f6edd600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed8159020 con 0x7f6ed8155730 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.151+0000 7f6ecfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ed80c7970 0x7f6ed80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.151+0000 7f6ecfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ed80c7970 0x7f6ed80c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46282/0 (socket says 172.21.15.27:46282) 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.151+0000 7f6ecfe00640 1 -- 172.21.15.27:0/2349901828 learned_addr learned my addr 172.21.15.27:0/2349901828 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.151+0000 7f6ed7400640 1 --2- 172.21.15.27:0/2349901828 >> v2:172.21.15.155:3300/0 conn(0x7f6ed8155730 0x7f6ed8155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.151+0000 7f6ecfe00640 1 -- 172.21.15.27:0/2349901828 >> v2:172.21.15.155:3300/0 conn(0x7f6ed8155730 msgr2=0x7f6ed8155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.151+0000 7f6ecfe00640 1 --2- 172.21.15.27:0/2349901828 >> v2:172.21.15.155:3300/0 conn(0x7f6ed8155730 0x7f6ed8155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.151+0000 7f6ecfe00640 1 -- 172.21.15.27:0/2349901828 >> v1:172.21.15.155:6789/0 conn(0x7f6ed8152f80 legacy=0x7f6ed80c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.151+0000 7f6ecfe00640 1 -- 172.21.15.27:0/2349901828 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed8159300 con 0x7f6ed80c7970 2024-03-28T19:04:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.152+0000 7f6ecfe00640 1 --2- 172.21.15.27:0/2349901828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed80c7970 0x7f6ed80c7df0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80c7850 tx=0x7f6ec002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f211c19c0f389588 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.153+0000 7f6ed6000640 1 -- 172.21.15.27:0/2349901828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec002e070 con 0x7f6ed80c7970 2024-03-28T19:04:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.153+0000 7f6ed6000640 1 -- 172.21.15.27:0/2349901828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec0037070 con 0x7f6ed80c7970 2024-03-28T19:04:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.153+0000 7f6ed6000640 1 -- 172.21.15.27:0/2349901828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0031a00 con 0x7f6ed80c7970 2024-03-28T19:04:57.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.154+0000 7f6edd600640 1 -- 172.21.15.27:0/2349901828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed80c7970 msgr2=0x7f6ed80c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.154+0000 7f6edd600640 1 --2- 172.21.15.27:0/2349901828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed80c7970 0x7f6ed80c7df0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80c7850 tx=0x7f6ec002b880 comp rx=0 tx=0).stop 2024-03-28T19:04:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.154+0000 7f6edd600640 1 -- 172.21.15.27:0/2349901828 shutdown_connections 2024-03-28T19:04:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.154+0000 7f6edd600640 1 --2- 172.21.15.27:0/2349901828 >> v2:172.21.15.155:3300/0 conn(0x7f6ed8155730 0x7f6ed8155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.154+0000 7f6edd600640 1 --2- 172.21.15.27:0/2349901828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed80c7970 0x7f6ed80c7df0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.154+0000 7f6edd600640 1 -- 172.21.15.27:0/2349901828 >> 172.21.15.27:0/2349901828 conn(0x7f6ed806b6f0 msgr2=0x7f6ed80caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.154+0000 7f6edd600640 1 -- 172.21.15.27:0/2349901828 shutdown_connections 2024-03-28T19:04:57.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.154+0000 7f6edd600640 1 -- 172.21.15.27:0/2349901828 wait complete. 2024-03-28T19:04:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.156+0000 7f6edd600640 1 Processor -- start 2024-03-28T19:04:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.157+0000 7f6edd600640 1 -- start start 2024-03-28T19:04:57.157 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:04:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6edd600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ed80c7970 0x7f6ed81ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6edd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed8152f80 0x7f6ed8148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6edd600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ed8155730 0x7f6ed8149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6edd600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed80cbca0 con 0x7f6ed8152f80 2024-03-28T19:04:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6edd600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed8159b00 con 0x7f6ed80c7970 2024-03-28T19:04:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6edd600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed81ef350 con 0x7f6ed8155730 2024-03-28T19:04:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6ecfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed8152f80 0x7f6ed8148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6ecfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed8152f80 0x7f6ed8148f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46292/0 (socket says 172.21.15.27:46292) 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.158+0000 7f6ecfe00640 1 -- 172.21.15.27:0/3304027559 learned_addr learned my addr 172.21.15.27:0/3304027559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ed6a00640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ed80c7970 0x7f6ed81ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ed7400640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ed8155730 0x7f6ed8149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecfe00640 1 -- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ed8155730 msgr2=0x7f6ed8149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecfe00640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ed8155730 0x7f6ed8149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecfe00640 1 -- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ed80c7970 msgr2=0x7f6ed81ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecfe00640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ed80c7970 0x7f6ed81ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecfe00640 1 -- 172.21.15.27:0/3304027559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed814fc30 con 0x7f6ed8152f80 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ed7400640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ed8155730 0x7f6ed8149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecfe00640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed8152f80 0x7f6ed8148f30 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0000c00 tx=0x7f6ec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecea00640 1 -- 172.21.15.27:0/3304027559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0032880 con 0x7f6ed8152f80 2024-03-28T19:04:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecea00640 1 -- 172.21.15.27:0/3304027559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec0032a20 con 0x7f6ed8152f80 2024-03-28T19:04:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6ecea00640 1 -- 172.21.15.27:0/3304027559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0032e10 con 0x7f6ed8152f80 2024-03-28T19:04:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.159+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed814ff10 con 0x7f6ed8152f80 2024-03-28T19:04:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.160+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed814ef90 con 0x7f6ed8152f80 2024-03-28T19:04:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.161+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed804f400 con 0x7f6ed8152f80 2024-03-28T19:04:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.162+0000 7f6ecea00640 1 -- 172.21.15.27:0/3304027559 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6ec004c060 con 0x7f6ed8152f80 2024-03-28T19:04:57.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.163+0000 7f6ecea00640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6eb406bbd0 0x7f6eb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:57.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.163+0000 7f6ecea00640 1 -- 172.21.15.27:0/3304027559 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7f6ec00acef0 con 0x7f6ed8152f80 2024-03-28T19:04:57.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.163+0000 7f6ed6a00640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6eb406bbd0 0x7f6eb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:57.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.164+0000 7f6ed6a00640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6eb406bbd0 0x7f6eb406e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8006fd0 tx=0x7f6ec8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.167+0000 7f6ecea00640 1 -- 172.21.15.27:0/3304027559 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6ec0040070 con 0x7f6ed8152f80 2024-03-28T19:04:57.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.405+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ed80cba80 con 0x7f6ed8152f80 2024-03-28T19:04:57.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.406+0000 7f6ecea00640 1 -- 172.21.15.27:0/3304027559 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5457 (secure 0 0 0) 0x7f6ec007b1d0 con 0x7f6ed8152f80 2024-03-28T19:04:57.406 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:57.406 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:54.587987+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:57.406 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 82 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6eb406bbd0 msgr2=0x7f6eb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6eb406bbd0 0x7f6eb406e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8006fd0 tx=0x7f6ec8008040 comp rx=0 tx=0).stop 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed8152f80 msgr2=0x7f6ed8148f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed8152f80 0x7f6ed8148f30 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0000c00 tx=0x7f6ec00046c0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 shutdown_connections 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6eb406bbd0 0x7f6eb406e090 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ed8155730 0x7f6ed8149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ed8152f80 0x7f6ed8148f30 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 --2- 172.21.15.27:0/3304027559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ed80c7970 0x7f6ed81ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 >> 172.21.15.27:0/3304027559 conn(0x7f6ed806b6f0 msgr2=0x7f6ed80caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:57.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 shutdown_connections 2024-03-28T19:04:57.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:57.409+0000 7f6edd600640 1 -- 172.21.15.27:0/3304027559 wait complete. 2024-03-28T19:04:57.426 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 82, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 82, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6192": { "addr": "172.21.15.27:6836/2888303230", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 2888303230, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 2888303230, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6192, "incarnation": 82, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:04:54.587987+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6192 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 80, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6054": { "addr": "172.21.15.27:6837/1815850547", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 1815850547, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 1815850547, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6054, "incarnation": 77, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:04:44.514032+0000", "qdb_cluster": [ 6054 ], "qdb_leader": 6054, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6054 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:04:59.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:59.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.394+0000 7f7c61400640 1 Processor -- start 2024-03-28T19:04:59.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.394+0000 7f7c61400640 1 -- start start 2024-03-28T19:04:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.396+0000 7f7c61400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c5c06c0d0 0x7f7c5c06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.396+0000 7f7c61400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7c5c0c1780 0x7f7c5c155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.396+0000 7f7c61400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c0a7400 con 0x7f7c5c0a6e90 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.396+0000 7f7c61400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c159470 con 0x7f7c5c06c0d0 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.396+0000 7f7c61400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c158b40 con 0x7f7c5c0c1780 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.396+0000 7f7c5be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c5c06c0d0 0x7f7c5c06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.396+0000 7f7c5be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c5c06c0d0 0x7f7c5c06c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46296/0 (socket says 172.21.15.27:46296) 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.396+0000 7f7c5be00640 1 -- 172.21.15.27:0/2398231111 learned_addr learned my addr 172.21.15.27:0/2398231111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.397+0000 7f7c60a00640 1 --2- 172.21.15.27:0/2398231111 >> v2:172.21.15.155:3300/0 conn(0x7f7c5c0c1780 0x7f7c5c155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.397+0000 7f7c5be00640 1 -- 172.21.15.27:0/2398231111 >> v2:172.21.15.155:3300/0 conn(0x7f7c5c0c1780 msgr2=0x7f7c5c155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.397+0000 7f7c5be00640 1 --2- 172.21.15.27:0/2398231111 >> v2:172.21.15.155:3300/0 conn(0x7f7c5c0c1780 0x7f7c5c155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.397+0000 7f7c5be00640 1 -- 172.21.15.27:0/2398231111 >> v1:172.21.15.155:6789/0 conn(0x7f7c5c0a6e90 legacy=0x7f7c5c155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.397+0000 7f7c5be00640 1 -- 172.21.15.27:0/2398231111 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c158e20 con 0x7f7c5c06c0d0 2024-03-28T19:04:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.397+0000 7f7c60a00640 1 --2- 172.21.15.27:0/2398231111 >> v2:172.21.15.155:3300/0 conn(0x7f7c5c0c1780 0x7f7c5c155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:59.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.398+0000 7f7c5be00640 1 --2- 172.21.15.27:0/2398231111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c06c0d0 0x7f7c5c06c4b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0c43e0 tx=0x7f7c4c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac75552d6483a5be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:59.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c5aa00640 1 -- 172.21.15.27:0/2398231111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c02e070 con 0x7f7c5c06c0d0 2024-03-28T19:04:59.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c5aa00640 1 -- 172.21.15.27:0/2398231111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c037070 con 0x7f7c5c06c0d0 2024-03-28T19:04:59.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c5aa00640 1 -- 172.21.15.27:0/2398231111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c031a00 con 0x7f7c5c06c0d0 2024-03-28T19:04:59.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c61400640 1 -- 172.21.15.27:0/2398231111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c06c0d0 msgr2=0x7f7c5c06c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c61400640 1 --2- 172.21.15.27:0/2398231111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c06c0d0 0x7f7c5c06c4b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0c43e0 tx=0x7f7c4c02b880 comp rx=0 tx=0).stop 2024-03-28T19:04:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c61400640 1 -- 172.21.15.27:0/2398231111 shutdown_connections 2024-03-28T19:04:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c61400640 1 --2- 172.21.15.27:0/2398231111 >> v2:172.21.15.155:3300/0 conn(0x7f7c5c0c1780 0x7f7c5c155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c61400640 1 --2- 172.21.15.27:0/2398231111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c06c0d0 0x7f7c5c06c4b0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c61400640 1 -- 172.21.15.27:0/2398231111 >> 172.21.15.27:0/2398231111 conn(0x7f7c5c0c7aa0 msgr2=0x7f7c5c0c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c61400640 1 -- 172.21.15.27:0/2398231111 shutdown_connections 2024-03-28T19:04:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.399+0000 7f7c61400640 1 -- 172.21.15.27:0/2398231111 wait complete. 2024-03-28T19:04:59.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.403+0000 7f7c61400640 1 Processor -- start 2024-03-28T19:04:59.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.404+0000 7f7c61400640 1 -- start start 2024-03-28T19:04:59.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c61400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c5c06c0d0 0x7f7c5c0a8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c61400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c5c0a6e90 0x7f7c5c1ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c61400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c0c1780 0x7f7c5c14d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c61400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c152e20 con 0x7f7c5c0c1780 2024-03-28T19:04:59.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c61400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c159900 con 0x7f7c5c0a6e90 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c61400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c152fa0 con 0x7f7c5c06c0d0 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c60a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c0c1780 0x7f7c5c14d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c60a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c0c1780 0x7f7c5c14d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46298/0 (socket says 172.21.15.27:46298) 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c60a00640 1 -- 172.21.15.27:0/29310904 learned_addr learned my addr 172.21.15.27:0/29310904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.406+0000 7f7c5be00640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c5c06c0d0 0x7f7c5c0a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c5b400640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c5c0a6e90 0x7f7c5c1ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c60a00640 1 -- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c5c06c0d0 msgr2=0x7f7c5c0a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c60a00640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c5c06c0d0 0x7f7c5c0a8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c60a00640 1 -- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c5c0a6e90 msgr2=0x7f7c5c1ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c60a00640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c5c0a6e90 0x7f7c5c1ee5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c60a00640 1 -- 172.21.15.27:0/29310904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c14e220 con 0x7f7c5c0c1780 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c5b400640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c5c0a6e90 0x7f7c5c1ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:04:59.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c60a00640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c0c1780 0x7f7c5c14d130 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400cd50 tx=0x7f7c5400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c53e00640 1 -- 172.21.15.27:0/29310904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400f680 con 0x7f7c5c0c1780 2024-03-28T19:04:59.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c53e00640 1 -- 172.21.15.27:0/29310904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5400f840 con 0x7f7c5c0c1780 2024-03-28T19:04:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c14eb30 con 0x7f7c5c0c1780 2024-03-28T19:04:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.407+0000 7f7c53e00640 1 -- 172.21.15.27:0/29310904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400fc30 con 0x7f7c5c0c1780 2024-03-28T19:04:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.408+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c14e850 con 0x7f7c5c0c1780 2024-03-28T19:04:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.409+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c0c2cb0 con 0x7f7c5c0c1780 2024-03-28T19:04:59.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.410+0000 7f7c53e00640 1 -- 172.21.15.27:0/29310904 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7c540213f0 con 0x7f7c5c0c1780 2024-03-28T19:04:59.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.411+0000 7f7c53e00640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c3806bbd0 0x7f7c3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.411+0000 7f7c53e00640 1 -- 172.21.15.27:0/29310904 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7f7c5408bfe0 con 0x7f7c5c0c1780 2024-03-28T19:04:59.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.411+0000 7f7c5be00640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c3806bbd0 0x7f7c3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.412+0000 7f7c5be00640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c3806bbd0 0x7f7c3806e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c036000 tx=0x7f7c4c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:59.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.416+0000 7f7c53e00640 1 -- 172.21.15.27:0/29310904 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7c5c0c2cb0 con 0x7f7c5c0c1780 2024-03-28T19:04:59.427 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:04:59.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.623+0000 7fb753e00640 1 Processor -- start 2024-03-28T19:04:59.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.623+0000 7fb753e00640 1 -- start start 2024-03-28T19:04:59.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb753e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c150c70 0x7fb74c0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb753e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb74c0c3690 0x7fb74c0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb753e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c0c3af0 con 0x7fb74c0c6660 2024-03-28T19:04:59.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb753e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0ab4e0 con 0x7fb74c0c3690 2024-03-28T19:04:59.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb753e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0ac1f0 con 0x7fb74c150c70 2024-03-28T19:04:59.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb750a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb74c0c3690 0x7fb74c0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb751e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb74c0c6660 0x7fb74c1534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36434/0 (socket says 172.21.15.27:36434) 2024-03-28T19:04:59.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb751e00640 1 -- 172.21.15.27:0/3111698920 learned_addr learned my addr 172.21.15.27:0/3111698920 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:04:59.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.625+0000 7fb751400640 1 --2- 172.21.15.27:0/3111698920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c150c70 0x7fb74c0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.626+0000 7fb750a00640 1 -- 172.21.15.27:0/3111698920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c150c70 msgr2=0x7fb74c0a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:04:59.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.626+0000 7fb750a00640 1 --2- 172.21.15.27:0/3111698920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c150c70 0x7fb74c0a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.626+0000 7fb750a00640 1 -- 172.21.15.27:0/3111698920 >> v1:172.21.15.27:6789/0 conn(0x7fb74c0c6660 legacy=0x7fb74c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.626+0000 7fb750a00640 1 -- 172.21.15.27:0/3111698920 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0c3d90 con 0x7fb74c0c3690 2024-03-28T19:04:59.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.628+0000 7fb750a00640 1 --2- 172.21.15.27:0/3111698920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c6120 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb7380097a0 tx=0x7fb73802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a0053b38f107a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:59.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.628+0000 7fb74be00640 1 -- 172.21.15.27:0/3111698920 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7380315a0 con 0x7fb74c0c3690 2024-03-28T19:04:59.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.628+0000 7fb74be00640 1 -- 172.21.15.27:0/3111698920 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb738037070 con 0x7fb74c0c3690 2024-03-28T19:04:59.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.628+0000 7fb74be00640 1 -- 172.21.15.27:0/3111698920 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738031960 con 0x7fb74c0c3690 2024-03-28T19:04:59.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.629+0000 7fb753e00640 1 -- 172.21.15.27:0/3111698920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 msgr2=0x7fb74c0c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.629+0000 7fb753e00640 1 --2- 172.21.15.27:0/3111698920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c6120 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb7380097a0 tx=0x7fb73802b840 comp rx=0 tx=0).stop 2024-03-28T19:04:59.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.629+0000 7fb753e00640 1 -- 172.21.15.27:0/3111698920 shutdown_connections 2024-03-28T19:04:59.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.629+0000 7fb753e00640 1 --2- 172.21.15.27:0/3111698920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c6120 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.629+0000 7fb753e00640 1 --2- 172.21.15.27:0/3111698920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c150c70 0x7fb74c0a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.629+0000 7fb753e00640 1 -- 172.21.15.27:0/3111698920 >> 172.21.15.27:0/3111698920 conn(0x7fb74c06b9d0 msgr2=0x7fb74c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.629+0000 7fb753e00640 1 -- 172.21.15.27:0/3111698920 shutdown_connections 2024-03-28T19:04:59.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.629+0000 7fb753e00640 1 -- 172.21.15.27:0/3111698920 wait complete. 2024-03-28T19:04:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.631+0000 7fb753e00640 1 Processor -- start 2024-03-28T19:04:59.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.631+0000 7fb753e00640 1 -- start start 2024-03-28T19:04:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb753e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb753e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c0c6660 0x7fb74c0a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb753e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb74c150c70 0x7fb74c0a7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb753e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0abcd0 con 0x7fb74c0c3690 2024-03-28T19:04:59.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb753e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0ab9f0 con 0x7fb74c150c70 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb753e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c0c1810 con 0x7fb74c0c6660 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb751400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb751400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46306/0 (socket says 172.21.15.27:46306) 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.632+0000 7fb751400640 1 -- 172.21.15.27:0/1570454086 learned_addr learned my addr 172.21.15.27:0/1570454086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.633+0000 7fb750a00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c0c6660 0x7fb74c0a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.633+0000 7fb751e00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb74c150c70 0x7fb74c0a7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.633+0000 7fb751400640 1 -- 172.21.15.27:0/1570454086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c0c6660 msgr2=0x7fb74c0a6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.633+0000 7fb751400640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c0c6660 0x7fb74c0a6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.633+0000 7fb751400640 1 -- 172.21.15.27:0/1570454086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb74c150c70 msgr2=0x7fb74c0a7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.633+0000 7fb751400640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb74c150c70 0x7fb74c0a7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.633+0000 7fb751400640 1 -- 172.21.15.27:0/1570454086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0a77a0 con 0x7fb74c0c3690 2024-03-28T19:04:59.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.633+0000 7fb751400640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c02a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d5f0 tx=0x7fb74000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.634+0000 7fb749600640 1 -- 172.21.15.27:0/1570454086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7400125f0 con 0x7fb74c0c3690 2024-03-28T19:04:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.634+0000 7fb749600640 1 -- 172.21.15.27:0/1570454086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740019070 con 0x7fb74c0c3690 2024-03-28T19:04:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.634+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c1eea20 con 0x7fb74c0c3690 2024-03-28T19:04:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.634+0000 7fb749600640 1 -- 172.21.15.27:0/1570454086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7400129d0 con 0x7fb74c0c3690 2024-03-28T19:04:59.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.634+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c1eefe0 con 0x7fb74c0c3690 2024-03-28T19:04:59.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.635+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c04f400 con 0x7fb74c0c3690 2024-03-28T19:04:59.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.639+0000 7fb749600640 1 -- 172.21.15.27:0/1570454086 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb740012b70 con 0x7fb74c0c3690 2024-03-28T19:04:59.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.640+0000 7fb749600640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb72c06bbd0 0x7fb72c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:04:59.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.640+0000 7fb749600640 1 -- 172.21.15.27:0/1570454086 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7fb74008b680 con 0x7fb74c0c3690 2024-03-28T19:04:59.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.640+0000 7fb750a00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb72c06bbd0 0x7fb72c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:04:59.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.642+0000 7fb750a00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb72c06bbd0 0x7fb72c06e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb738036000 tx=0x7fb73803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:04:59.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.647+0000 7fb749600640 1 -- 172.21.15.27:0/1570454086 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb740059960 con 0x7fb74c0c3690 2024-03-28T19:04:59.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.712+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c5c069640 con 0x7f7c5c0c1780 2024-03-28T19:04:59.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.713+0000 7f7c53e00640 1 -- 172.21.15.27:0/29310904 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5464 (secure 0 0 0) 0x7f7c54014030 con 0x7f7c5c0c1780 2024-03-28T19:04:59.712 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:04:59.712 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:04:59.713 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 85 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c3806bbd0 msgr2=0x7f7c3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c3806bbd0 0x7f7c3806e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c036000 tx=0x7f7c4c03d040 comp rx=0 tx=0).stop 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c0c1780 msgr2=0x7f7c5c14d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c0c1780 0x7f7c5c14d130 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400cd50 tx=0x7f7c5400a650 comp rx=0 tx=0).stop 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 shutdown_connections 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c3806bbd0 0x7f7c3806e090 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5c0c1780 0x7f7c5c14d130 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c5c0a6e90 0x7f7c5c1ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 --2- 172.21.15.27:0/29310904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c5c06c0d0 0x7f7c5c0a8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 >> 172.21.15.27:0/29310904 conn(0x7f7c5c0c7aa0 msgr2=0x7f7c5c071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 shutdown_connections 2024-03-28T19:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:04:59.716+0000 7f7c61400640 1 -- 172.21.15.27:0/29310904 wait complete. 2024-03-28T19:04:59.737 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:active state 2024-03-28T19:04:59.737 INFO:tasks.mds_thrash.fs.[b]:waiting for 111 secs before thrashing 2024-03-28T19:05:00.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:05:00.020 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:05:00.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.040+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb74c0c5f00 con 0x7fb74c0c3690 2024-03-28T19:05:00.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.041+0000 7fb749600640 1 -- 172.21.15.27:0/1570454086 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5464 (secure 0 0 0) 0x7fb740059260 con 0x7fb74c0c3690 2024-03-28T19:05:00.040 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:05:00.040 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:05:00.040 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 85 2024-03-28T19:05:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.044+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb72c06bbd0 msgr2=0x7fb72c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.044+0000 7fb753e00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb72c06bbd0 0x7fb72c06e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb738036000 tx=0x7fb73803d040 comp rx=0 tx=0).stop 2024-03-28T19:05:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 msgr2=0x7fb74c0c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:00.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c02a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d5f0 tx=0x7fb74000dab0 comp rx=0 tx=0).stop 2024-03-28T19:05:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 shutdown_connections 2024-03-28T19:05:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb72c06bbd0 0x7fb72c06e090 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb74c150c70 0x7fb74c0a7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb74c0c6660 0x7fb74c0a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 --2- 172.21.15.27:0/1570454086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb74c0c3690 0x7fb74c0c02a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 >> 172.21.15.27:0/1570454086 conn(0x7fb74c06b9d0 msgr2=0x7fb74c0a8990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:05:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 shutdown_connections 2024-03-28T19:05:00.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:00.045+0000 7fb753e00640 1 -- 172.21.15.27:0/1570454086 wait complete. 2024-03-28T19:05:00.069 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5201 2024-03-28T19:05:00.069 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.b 2024-03-28T19:05:04.604 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:05:04.604+0000 7f97d3c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:05:07.070 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-03-28T19:05:07.070 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:05:07.070 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:05:07.078 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:05:07.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:05:07.270 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:05:07.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.318+0000 7fde91c00640 1 Processor -- start 2024-03-28T19:05:07.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.318+0000 7fde91c00640 1 -- start start 2024-03-28T19:05:07.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.319+0000 7fde91c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:05:07.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.319+0000 7fde91c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c158bd0 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.319+0000 7fde91c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c158eb0 con 0x7fde8c0be5d0 2024-03-28T19:05:07.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.319+0000 7fde91c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c159190 con 0x7fde8c0c58c0 2024-03-28T19:05:07.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.320+0000 7fde8be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:05:07.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.320+0000 7fde8b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fde8c0c8ed0 0x7fde8c1555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47868/0 (socket says 172.21.15.27:47868) 2024-03-28T19:05:07.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.320+0000 7fde8b400640 1 -- 172.21.15.27:0/4027070617 learned_addr learned my addr 172.21.15.27:0/4027070617 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:05:07.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.320+0000 7fde8be00640 1 -- 172.21.15.27:0/4027070617 >> v1:172.21.15.155:6789/0 conn(0x7fde8c0be5d0 legacy=0x7fde8c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:07.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.320+0000 7fde8be00640 1 -- 172.21.15.27:0/4027070617 >> v1:172.21.15.27:6789/0 conn(0x7fde8c0c8ed0 legacy=0x7fde8c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:07.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.320+0000 7fde8aa00640 1 -- 172.21.15.27:0/4027070617 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334824108 0 0) 0x7fde8c158bd0 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.320+0000 7fde8be00640 1 -- 172.21.15.27:0/4027070617 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c159470 con 0x7fde8c0c58c0 2024-03-28T19:05:07.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.322+0000 7fde8be00640 1 --2- 172.21.15.27:0/4027070617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c5ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0097d0 tx=0x7fde7c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60e6a09e86dd34f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:05:07.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.322+0000 7fde8aa00640 1 -- 172.21.15.27:0/4027070617 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c035020 con 0x7fde8c0c58c0 2024-03-28T19:05:07.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.322+0000 7fde8aa00640 1 -- 172.21.15.27:0/4027070617 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c031800 con 0x7fde8c0c58c0 2024-03-28T19:05:07.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.322+0000 7fde8aa00640 1 -- 172.21.15.27:0/4027070617 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c031bd0 con 0x7fde8c0c58c0 2024-03-28T19:05:07.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.323+0000 7fde91c00640 1 -- 172.21.15.27:0/4027070617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 msgr2=0x7fde8c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:07.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.323+0000 7fde91c00640 1 --2- 172.21.15.27:0/4027070617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c5ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0097d0 tx=0x7fde7c02b8a0 comp rx=0 tx=0).stop 2024-03-28T19:05:07.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.323+0000 7fde91c00640 1 -- 172.21.15.27:0/4027070617 shutdown_connections 2024-03-28T19:05:07.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.323+0000 7fde91c00640 1 --2- 172.21.15.27:0/4027070617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c5ca0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:07.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.323+0000 7fde91c00640 1 -- 172.21.15.27:0/4027070617 >> 172.21.15.27:0/4027070617 conn(0x7fde8c06b6f0 msgr2=0x7fde8c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:05:07.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.323+0000 7fde91c00640 1 -- 172.21.15.27:0/4027070617 shutdown_connections 2024-03-28T19:05:07.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.323+0000 7fde91c00640 1 -- 172.21.15.27:0/4027070617 wait complete. 2024-03-28T19:05:07.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.326+0000 7fde91c00640 1 Processor -- start 2024-03-28T19:05:07.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.326+0000 7fde91c00640 1 -- start start 2024-03-28T19:05:07.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.327+0000 7fde91c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde8c0be5d0 0x7fde8c152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:05:07.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.327+0000 7fde91c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:05:07.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.327+0000 7fde91c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8c0c8ed0 0x7fde8c1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:05:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.327+0000 7fde91c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0b09a0 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.327+0000 7fde91c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0b0820 con 0x7fde8c0be5d0 2024-03-28T19:05:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.327+0000 7fde91c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0ad7d0 con 0x7fde8c0c58c0 2024-03-28T19:05:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.328+0000 7fde91200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8c0c8ed0 0x7fde8c1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:05:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.328+0000 7fde8be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde8c0be5d0 0x7fde8c152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:05:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.328+0000 7fde91200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8c0c8ed0 0x7fde8c1514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54554/0 (socket says 172.21.15.27:54554) 2024-03-28T19:05:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.328+0000 7fde8b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:05:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.328+0000 7fde91200640 1 -- 172.21.15.27:0/1704286596 learned_addr learned my addr 172.21.15.27:0/1704286596 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:05:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.329+0000 7fde91200640 1 -- 172.21.15.27:0/1704286596 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 msgr2=0x7fde8c0c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.329+0000 7fde91200640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.329+0000 7fde91200640 1 -- 172.21.15.27:0/1704286596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde8c0be5d0 msgr2=0x7fde8c152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.329+0000 7fde91200640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde8c0be5d0 0x7fde8c152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.329+0000 7fde91200640 1 -- 172.21.15.27:0/1704286596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c1ebdf0 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.330+0000 7fde91200640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8c0c8ed0 0x7fde8c1514e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c6b0 tx=0x7fde8400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:05:07.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.330+0000 7fde83e00640 1 -- 172.21.15.27:0/1704286596 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84003b70 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.330+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c152940 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.330+0000 7fde83e00640 1 -- 172.21.15.27:0/1704286596 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde84003d10 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.330+0000 7fde83e00640 1 -- 172.21.15.27:0/1704286596 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84005310 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.330+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c0aaed0 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.332+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8c069640 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.333+0000 7fde83e00640 1 -- 172.21.15.27:0/1704286596 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fde840054b0 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.335+0000 7fde83e00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde6c06bbd0 0x7fde6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:05:07.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.335+0000 7fde83e00640 1 -- 172.21.15.27:0/1704286596 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7fde8408af50 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.335+0000 7fde8be00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde6c06bbd0 0x7fde6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:05:07.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.337+0000 7fde8be00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde6c06bbd0 0x7fde6c06e090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fde7c036000 tx=0x7fde7c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:05:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.340+0000 7fde83e00640 1 -- 172.21.15.27:0/1704286596 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fde84058be0 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.725+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde8c156580 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.725+0000 7fde83e00640 1 -- 172.21.15.27:0/1704286596 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6258 (secure 0 0 0) 0x7fde8405cc80 con 0x7fde8c0c8ed0 2024-03-28T19:05:07.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:05:07.725 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5336,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:05:07.725 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.728+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde6c06bbd0 msgr2=0x7fde6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.728+0000 7fde91c00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde6c06bbd0 0x7fde6c06e090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fde7c036000 tx=0x7fde7c03c040 comp rx=0 tx=0).stop 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.728+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8c0c8ed0 msgr2=0x7fde8c1514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.728+0000 7fde91c00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8c0c8ed0 0x7fde8c1514e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c6b0 tx=0x7fde8400cb70 comp rx=0 tx=0).stop 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.729+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 shutdown_connections 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.729+0000 7fde91c00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde6c06bbd0 0x7fde6c06e090 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.729+0000 7fde91c00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde8c0c8ed0 0x7fde8c1514e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.729+0000 7fde91c00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde8c0c58c0 0x7fde8c0c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.729+0000 7fde91c00640 1 --2- 172.21.15.27:0/1704286596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde8c0be5d0 0x7fde8c152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.729+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 >> 172.21.15.27:0/1704286596 conn(0x7fde8c06b6f0 msgr2=0x7fde8c0ae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:05:07.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.729+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 shutdown_connections 2024-03-28T19:05:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:05:07.729+0000 7fde91c00640 1 -- 172.21.15.27:0/1704286596 wait complete. 2024-03-28T19:05:07.751 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-03-28T19:05:07.751 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2024-03-28T19:05:30.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:05:30.161 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:06:00.296 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:06:00.303 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:06:01.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.015+0000 7fc38f200640 1 Processor -- start 2024-03-28T19:06:02.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.015+0000 7fc38f200640 1 -- start start 2024-03-28T19:06:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.017+0000 7fc38f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3900be5d0 0x7fc3900c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.017+0000 7fc38f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc390158bd0 con 0x7fc3900c58c0 2024-03-28T19:06:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.017+0000 7fc38f200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc390158eb0 con 0x7fc3900c8ed0 2024-03-28T19:06:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.017+0000 7fc38f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc390159190 con 0x7fc3900be5d0 2024-03-28T19:06:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.017+0000 7fc38de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc3900c58c0 0x7fc3901555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36588/0 (socket says 172.21.15.27:36588) 2024-03-28T19:06:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.017+0000 7fc38e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3900be5d0 0x7fc3900c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.017+0000 7fc38de00640 1 -- 172.21.15.27:0/3411988950 learned_addr learned my addr 172.21.15.27:0/3411988950 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:06:02.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.018+0000 7fc38ca00640 1 -- 172.21.15.27:0/3411988950 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039805533 0 0) 0x7fc390158bd0 con 0x7fc3900c58c0 2024-03-28T19:06:02.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.018+0000 7fc38ca00640 1 -- 172.21.15.27:0/3411988950 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc374003520 con 0x7fc3900c58c0 2024-03-28T19:06:02.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.018+0000 7fc38e800640 1 -- 172.21.15.27:0/3411988950 >> v1:172.21.15.155:6789/0 conn(0x7fc3900c8ed0 legacy=0x7fc390155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.018+0000 7fc38e800640 1 -- 172.21.15.27:0/3411988950 >> v1:172.21.15.27:6789/0 conn(0x7fc3900c58c0 legacy=0x7fc3901555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.018+0000 7fc38e800640 1 -- 172.21.15.27:0/3411988950 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc390159470 con 0x7fc3900be5d0 2024-03-28T19:06:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.019+0000 7fc38e800640 1 --2- 172.21.15.27:0/3411988950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900be5d0 0x7fc3900c5ca0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc388009d90 tx=0x7fc38802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b2d5f2772d52ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.019+0000 7fc38ca00640 1 -- 172.21.15.27:0/3411988950 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388039070 con 0x7fc3900be5d0 2024-03-28T19:06:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.019+0000 7fc38ca00640 1 -- 172.21.15.27:0/3411988950 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38802cbd0 con 0x7fc3900be5d0 2024-03-28T19:06:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.019+0000 7fc38ca00640 1 -- 172.21.15.27:0/3411988950 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388034440 con 0x7fc3900be5d0 2024-03-28T19:06:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.020+0000 7fc38f200640 1 -- 172.21.15.27:0/3411988950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900be5d0 msgr2=0x7fc3900c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.020+0000 7fc38f200640 1 --2- 172.21.15.27:0/3411988950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900be5d0 0x7fc3900c5ca0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc388009d90 tx=0x7fc38802be20 comp rx=0 tx=0).stop 2024-03-28T19:06:02.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.020+0000 7fc38f200640 1 -- 172.21.15.27:0/3411988950 shutdown_connections 2024-03-28T19:06:02.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.020+0000 7fc38f200640 1 --2- 172.21.15.27:0/3411988950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900be5d0 0x7fc3900c5ca0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.020+0000 7fc38f200640 1 -- 172.21.15.27:0/3411988950 >> 172.21.15.27:0/3411988950 conn(0x7fc39006b6f0 msgr2=0x7fc3900acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:02.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.020+0000 7fc38f200640 1 -- 172.21.15.27:0/3411988950 shutdown_connections 2024-03-28T19:06:02.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.020+0000 7fc38f200640 1 -- 172.21.15.27:0/3411988950 wait complete. 2024-03-28T19:06:02.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.022+0000 7fc38f200640 1 Processor -- start 2024-03-28T19:06:02.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.022+0000 7fc38f200640 1 -- start start 2024-03-28T19:06:02.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3900be5d0 0x7fc390152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3900c58c0 0x7fc3900c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900c8ed0 0x7fc390151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900b09a0 con 0x7fc3900c8ed0 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900b0820 con 0x7fc3900be5d0 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3900ad7d0 con 0x7fc3900c58c0 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900c8ed0 0x7fc390151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900c8ed0 0x7fc390151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35482/0 (socket says 172.21.15.27:35482) 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38e800640 1 -- 172.21.15.27:0/1291509254 learned_addr learned my addr 172.21.15.27:0/1291509254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38de00640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3900be5d0 0x7fc390152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc38e800640 1 -- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3900c58c0 msgr2=0x7fc3900c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.023+0000 7fc38d400640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3900c58c0 0x7fc3900c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc38e800640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3900c58c0 0x7fc3900c2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc38e800640 1 -- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3900be5d0 msgr2=0x7fc390152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc38e800640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3900be5d0 0x7fc390152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc38e800640 1 -- 172.21.15.27:0/1291509254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3901ef4a0 con 0x7fc3900c8ed0 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc38de00640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3900be5d0 0x7fc390152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc38e800640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900c8ed0 0x7fc390151450 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc388037040 tx=0x7fc388004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc386000640 1 -- 172.21.15.27:0/1291509254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388044020 con 0x7fc3900c8ed0 2024-03-28T19:06:02.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc386000640 1 -- 172.21.15.27:0/1291509254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38803a040 con 0x7fc3900c8ed0 2024-03-28T19:06:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc386000640 1 -- 172.21.15.27:0/1291509254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388039050 con 0x7fc3900c8ed0 2024-03-28T19:06:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.024+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3900aacc0 con 0x7fc3900c8ed0 2024-03-28T19:06:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.025+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3900ab280 con 0x7fc3900c8ed0 2024-03-28T19:06:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.026+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc390069640 con 0x7fc3900c8ed0 2024-03-28T19:06:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.027+0000 7fc386000640 1 -- 172.21.15.27:0/1291509254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc388003bd0 con 0x7fc3900c8ed0 2024-03-28T19:06:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.028+0000 7fc386000640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc37006bc00 0x7fc37006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.028+0000 7fc386000640 1 -- 172.21.15.27:0/1291509254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7fc3880ad920 con 0x7fc3900c8ed0 2024-03-28T19:06:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.028+0000 7fc38de00640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc37006bc00 0x7fc37006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.029+0000 7fc38de00640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc37006bc00 0x7fc37006e0c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc380007fd0 tx=0x7fc380009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:02.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.037+0000 7fc386000640 1 -- 172.21.15.27:0/1291509254 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc38804e060 con 0x7fc3900c8ed0 2024-03-28T19:06:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.304+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc390156580 con 0x7fc3900c8ed0 2024-03-28T19:06:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.305+0000 7fc386000640 1 -- 172.21.15.27:0/1291509254 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6258 (secure 0 0 0) 0x7fc38807bc00 con 0x7fc3900c8ed0 2024-03-28T19:06:02.306 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:02.306 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5336,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-03-28T19:06:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc37006bc00 msgr2=0x7fc37006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc37006bc00 0x7fc37006e0c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc380007fd0 tx=0x7fc380009040 comp rx=0 tx=0).stop 2024-03-28T19:06:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900c8ed0 msgr2=0x7fc390151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900c8ed0 0x7fc390151450 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc388037040 tx=0x7fc388004a40 comp rx=0 tx=0).stop 2024-03-28T19:06:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 shutdown_connections 2024-03-28T19:06:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc37006bc00 0x7fc37006e0c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3900c8ed0 0x7fc390151450 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3900c58c0 0x7fc3900c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 --2- 172.21.15.27:0/1291509254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3900be5d0 0x7fc390152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 >> 172.21.15.27:0/1291509254 conn(0x7fc39006b6f0 msgr2=0x7fc3900cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 shutdown_connections 2024-03-28T19:06:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.308+0000 7fc38f200640 1 -- 172.21.15.27:0/1291509254 wait complete. 2024-03-28T19:06:02.327 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-03-28T19:06:02.328 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:06:02.328 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:06:02.349 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:06:02.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.615+0000 7fdbf1c00640 1 Processor -- start 2024-03-28T19:06:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.616+0000 7fdbf1c00640 1 -- start start 2024-03-28T19:06:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.617+0000 7fdbf1c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbec152a70 0x7fdbec159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.617+0000 7fdbf1c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 0x7fdbec0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.617+0000 7fdbf1c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdbec0cb620 0x7fdbec155b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.617+0000 7fdbf1c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbec1556b0 con 0x7fdbec152a70 2024-03-28T19:06:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.617+0000 7fdbf1c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbec0c5710 con 0x7fdbec0cb620 2024-03-28T19:06:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.617+0000 7fdbf1c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec0c59f0 con 0x7fdbec0c6cc0 2024-03-28T19:06:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbebe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbec152a70 0x7fdbec159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbebe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbec152a70 0x7fdbec159580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35488/0 (socket says 172.21.15.27:35488) 2024-03-28T19:06:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbebe00640 1 -- 172.21.15.27:0/1186330231 learned_addr learned my addr 172.21.15.27:0/1186330231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbeb400640 1 --2- 172.21.15.27:0/1186330231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 0x7fdbec0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbf1200640 1 --2- 172.21.15.27:0/1186330231 >> v2:172.21.15.155:3300/0 conn(0x7fdbec0cb620 0x7fdbec155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbebe00640 1 -- 172.21.15.27:0/1186330231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 msgr2=0x7fdbec0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbebe00640 1 --2- 172.21.15.27:0/1186330231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 0x7fdbec0cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbebe00640 1 -- 172.21.15.27:0/1186330231 >> v2:172.21.15.155:3300/0 conn(0x7fdbec0cb620 msgr2=0x7fdbec155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbebe00640 1 --2- 172.21.15.27:0/1186330231 >> v2:172.21.15.155:3300/0 conn(0x7fdbec0cb620 0x7fdbec155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbebe00640 1 -- 172.21.15.27:0/1186330231 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec0c5cd0 con 0x7fdbec152a70 2024-03-28T19:06:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.618+0000 7fdbf1200640 1 --2- 172.21.15.27:0/1186330231 >> v2:172.21.15.155:3300/0 conn(0x7fdbec0cb620 0x7fdbec155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.619+0000 7fdbebe00640 1 --2- 172.21.15.27:0/1186330231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec159580 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0727a0 tx=0x7fdbdc02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1e8ce698ef5da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.620+0000 7fdbeaa00640 1 -- 172.21.15.27:0/1186330231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc0317d0 con 0x7fdbec152a70 2024-03-28T19:06:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.620+0000 7fdbeaa00640 1 -- 172.21.15.27:0/1186330231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbdc037070 con 0x7fdbec152a70 2024-03-28T19:06:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.620+0000 7fdbeaa00640 1 -- 172.21.15.27:0/1186330231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc031b90 con 0x7fdbec152a70 2024-03-28T19:06:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.620+0000 7fdbf1c00640 1 -- 172.21.15.27:0/1186330231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 msgr2=0x7fdbec159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.620+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/1186330231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec159580 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0727a0 tx=0x7fdbdc02b870 comp rx=0 tx=0).stop 2024-03-28T19:06:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.621+0000 7fdbf1c00640 1 -- 172.21.15.27:0/1186330231 shutdown_connections 2024-03-28T19:06:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.621+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/1186330231 >> v2:172.21.15.155:3300/0 conn(0x7fdbec0cb620 0x7fdbec155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.621+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/1186330231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 0x7fdbec0cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.621+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/1186330231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec159580 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.621+0000 7fdbf1c00640 1 -- 172.21.15.27:0/1186330231 >> 172.21.15.27:0/1186330231 conn(0x7fdbec06b2a0 msgr2=0x7fdbec06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.621+0000 7fdbf1c00640 1 -- 172.21.15.27:0/1186330231 shutdown_connections 2024-03-28T19:06:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.621+0000 7fdbf1c00640 1 -- 172.21.15.27:0/1186330231 wait complete. 2024-03-28T19:06:02.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.623+0000 7fdbf1c00640 1 Processor -- start 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.623+0000 7fdbf1c00640 1 -- start start 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 0x7fdbec147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbec0cb620 0x7fdbec0abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec0c8f50 con 0x7fdbec152a70 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec0c8dd0 con 0x7fdbec0cb620 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbec0c5fb0 con 0x7fdbec0c6cc0 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec1540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35490/0 (socket says 172.21.15.27:35490) 2024-03-28T19:06:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbf1200640 1 -- 172.21.15.27:0/4012379875 learned_addr learned my addr 172.21.15.27:0/4012379875 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.625+0000 7fdbebe00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 0x7fdbec147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbeb400640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbec0cb620 0x7fdbec0abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbf1200640 1 -- 172.21.15.27:0/4012379875 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 msgr2=0x7fdbec147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbf1200640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 0x7fdbec147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbf1200640 1 -- 172.21.15.27:0/4012379875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbec0cb620 msgr2=0x7fdbec0abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbf1200640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbec0cb620 0x7fdbec0abdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbf1200640 1 -- 172.21.15.27:0/4012379875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec1eb230 con 0x7fdbec152a70 2024-03-28T19:06:02.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbf1200640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec1540c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400cef0 tx=0x7fdbe400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:02.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbe3e00640 1 -- 172.21.15.27:0/4012379875 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe401a070 con 0x7fdbec152a70 2024-03-28T19:06:02.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbe3e00640 1 -- 172.21.15.27:0/4012379875 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe400d860 con 0x7fdbec152a70 2024-03-28T19:06:02.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbe3e00640 1 -- 172.21.15.27:0/4012379875 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe400dc50 con 0x7fdbec152a70 2024-03-28T19:06:02.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.626+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbec1ee940 con 0x7fdbec152a70 2024-03-28T19:06:02.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.627+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbec1eb7d0 con 0x7fdbec152a70 2024-03-28T19:06:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.628+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbec04f400 con 0x7fdbec152a70 2024-03-28T19:06:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.629+0000 7fdbe3e00640 1 -- 172.21.15.27:0/4012379875 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdbe401d070 con 0x7fdbec152a70 2024-03-28T19:06:02.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.630+0000 7fdbe3e00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdbcc06bbd0 0x7fdbcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:02.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.630+0000 7fdbe3e00640 1 -- 172.21.15.27:0/4012379875 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7fdbe408bb90 con 0x7fdbec152a70 2024-03-28T19:06:02.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.630+0000 7fdbebe00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdbcc06bbd0 0x7fdbcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:02.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.631+0000 7fdbebe00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdbcc06bbd0 0x7fdbcc06e090 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc036000 tx=0x7fdbdc03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:02.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.635+0000 7fdbe3e00640 1 -- 172.21.15.27:0/4012379875 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdbec04f400 con 0x7fdbec152a70 2024-03-28T19:06:02.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.891+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbec0c1c20 con 0x7fdbec152a70 2024-03-28T19:06:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.926+0000 7fdbe3e00640 1 -- 172.21.15.27:0/4012379875 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6258 (secure 0 0 0) 0x7fdbe4059eb0 con 0x7fdbec152a70 2024-03-28T19:06:02.927 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:02.927 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5336,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-03-28T19:06:02.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdbcc06bbd0 msgr2=0x7fdbcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdbcc06bbd0 0x7fdbcc06e090 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc036000 tx=0x7fdbdc03d040 comp rx=0 tx=0).stop 2024-03-28T19:06:02.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 msgr2=0x7fdbec1540c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:02.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec1540c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400cef0 tx=0x7fdbe400a710 comp rx=0 tx=0).stop 2024-03-28T19:06:02.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 shutdown_connections 2024-03-28T19:06:02.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdbcc06bbd0 0x7fdbcc06e090 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbec152a70 0x7fdbec1540c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbec0cb620 0x7fdbec0abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 --2- 172.21.15.27:0/4012379875 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbec0c6cc0 0x7fdbec147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:02.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 >> 172.21.15.27:0/4012379875 conn(0x7fdbec06b2a0 msgr2=0x7fdbec14b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:02.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.929+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 shutdown_connections 2024-03-28T19:06:02.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:02.930+0000 7fdbf1c00640 1 -- 172.21.15.27:0/4012379875 wait complete. 2024-03-28T19:06:02.948 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:06:04.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.178+0000 7fa314a00640 1 Processor -- start 2024-03-28T19:06:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.178+0000 7fa314a00640 1 -- start start 2024-03-28T19:06:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.180+0000 7fa314a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa3100c8ed0 0x7fa3100c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.180+0000 7fa314a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa310158bd0 con 0x7fa3100c58c0 2024-03-28T19:06:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.180+0000 7fa314a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa310158eb0 con 0x7fa3100be5d0 2024-03-28T19:06:05.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.180+0000 7fa314a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa310159190 con 0x7fa3100c8ed0 2024-03-28T19:06:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.180+0000 7fa30e800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa3100c58c0 0x7fa3101555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36602/0 (socket says 172.21.15.27:36602) 2024-03-28T19:06:05.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.180+0000 7fa30e800640 1 -- 172.21.15.27:0/4121993177 learned_addr learned my addr 172.21.15.27:0/4121993177 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:06:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.180+0000 7fa307e00640 1 --2- 172.21.15.27:0/4121993177 >> v2:172.21.15.155:3300/0 conn(0x7fa3100c8ed0 0x7fa3100c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.180+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546004479 0 0) 0x7fa310158bd0 con 0x7fa3100c58c0 2024-03-28T19:06:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f4003520 con 0x7fa3100c58c0 2024-03-28T19:06:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1757325108 0 0) 0x7fa2f4003520 con 0x7fa3100c58c0 2024-03-28T19:06:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 >> v2:172.21.15.155:3300/0 conn(0x7fa3100c8ed0 msgr2=0x7fa3100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 --2- 172.21.15.27:0/4121993177 >> v2:172.21.15.155:3300/0 conn(0x7fa3100c8ed0 0x7fa3100c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 >> v1:172.21.15.155:6789/0 conn(0x7fa3100be5d0 legacy=0x7fa310155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:05.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa310159470 con 0x7fa3100c58c0 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fa300001670 con 0x7fa3100c58c0 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 >> v1:172.21.15.27:6789/0 conn(0x7fa3100c58c0 legacy=0x7fa3101555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f40039b0 0x7fa2f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f4004800 0x7fa2f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa2f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa310159190 con 0x7fa2f40052e0 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f400c1b0 con 0x7fa2f4004800 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.181+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f400c3b0 con 0x7fa2f40039b0 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa30f200640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa2f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa30e800640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f40039b0 0x7fa2f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa307e00640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f4004800 0x7fa2f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:05.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa30f200640 1 -- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f40039b0 msgr2=0x7fa2f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa30f200640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2f40039b0 0x7fa2f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa30f200640 1 -- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f4004800 msgr2=0x7fa2f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa30f200640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f4004800 0x7fa2f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa30f200640 1 -- 172.21.15.27:0/4121993177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4003520 con 0x7fa2f40052e0 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.182+0000 7fa307e00640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2f4004800 0x7fa2f4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.183+0000 7fa30f200640 1 -- 172.21.15.27:0/4121993177 reap_dead start 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.183+0000 7fa30f200640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa2f400b9c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa308001e20 tx=0x7fa2f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d58811805cacbe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.183+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308001a70 con 0x7fa2f40052e0 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.183+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa308003b70 con 0x7fa2f40052e0 2024-03-28T19:06:05.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.183+0000 7fa30de00640 1 -- 172.21.15.27:0/4121993177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3080050d0 con 0x7fa2f40052e0 2024-03-28T19:06:05.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.184+0000 7fa314a00640 1 -- 172.21.15.27:0/4121993177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 msgr2=0x7fa2f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:05.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.184+0000 7fa314a00640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa2f400b9c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa308001e20 tx=0x7fa2f4005830 comp rx=0 tx=0).stop 2024-03-28T19:06:05.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.184+0000 7fa314a00640 1 -- 172.21.15.27:0/4121993177 shutdown_connections 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.184+0000 7fa314a00640 1 --2- 172.21.15.27:0/4121993177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa2f400b9c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.184+0000 7fa314a00640 1 -- 172.21.15.27:0/4121993177 >> 172.21.15.27:0/4121993177 conn(0x7fa31006b6f0 msgr2=0x7fa3100acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.184+0000 7fa314a00640 1 -- 172.21.15.27:0/4121993177 shutdown_connections 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.184+0000 7fa314a00640 1 -- 172.21.15.27:0/4121993177 wait complete. 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.186+0000 7fa314a00640 1 Processor -- start 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.186+0000 7fa314a00640 1 -- start start 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.187+0000 7fa314a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa3100abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.187+0000 7fa314a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3101e1880 0x7fa3101519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:05.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.187+0000 7fa314a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa310152450 0x7fa3101eb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.187+0000 7fa314a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100b09a0 con 0x7fa2f40052e0 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.187+0000 7fa314a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100b0820 con 0x7fa310152450 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.187+0000 7fa314a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3100ad7d0 con 0x7fa3101e1880 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.187+0000 7fa30e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa3100abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.187+0000 7fa30e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa3100abce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35500/0 (socket says 172.21.15.27:35500) 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30e800640 1 -- 172.21.15.27:0/3276093175 learned_addr learned my addr 172.21.15.27:0/3276093175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa307e00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3101e1880 0x7fa3101519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30f200640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa310152450 0x7fa3101eb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30e800640 1 -- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3101e1880 msgr2=0x7fa3101519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:05.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30e800640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3101e1880 0x7fa3101519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30e800640 1 -- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa310152450 msgr2=0x7fa3101eb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30e800640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa310152450 0x7fa3101eb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30e800640 1 -- 172.21.15.27:0/3276093175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100acc10 con 0x7fa2f40052e0 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30f200640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa310152450 0x7fa3101eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.188+0000 7fa30e800640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa3100abce0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e780 tx=0x7fa30000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.189+0000 7fa306a00640 1 -- 172.21.15.27:0/3276093175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30000c660 con 0x7fa2f40052e0 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.189+0000 7fa306a00640 1 -- 172.21.15.27:0/3276093175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300018070 con 0x7fa2f40052e0 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.189+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3100c2600 con 0x7fa2f40052e0 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.189+0000 7fa306a00640 1 -- 172.21.15.27:0/3276093175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30000ca60 con 0x7fa2f40052e0 2024-03-28T19:06:05.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.189+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3100c28a0 con 0x7fa2f40052e0 2024-03-28T19:06:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.190+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3101e14c0 con 0x7fa2f40052e0 2024-03-28T19:06:05.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.191+0000 7fa306a00640 1 -- 172.21.15.27:0/3276093175 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa30000cc00 con 0x7fa2f40052e0 2024-03-28T19:06:05.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.192+0000 7fa306a00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2ec06bc00 0x7fa2ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:05.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.192+0000 7fa306a00640 1 -- 172.21.15.27:0/3276093175 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7fa30008ad60 con 0x7fa2f40052e0 2024-03-28T19:06:05.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.193+0000 7fa307e00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2ec06bc00 0x7fa2ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:05.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.193+0000 7fa307e00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2ec06bc00 0x7fa2ec06e0c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa3101e26c0 tx=0x7fa2f8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:05.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.197+0000 7fa306a00640 1 -- 172.21.15.27:0/3276093175 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa300059040 con 0x7fa2f40052e0 2024-03-28T19:06:05.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.562+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa310069640 con 0x7fa2f40052e0 2024-03-28T19:06:05.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.563+0000 7fa306a00640 1 -- 172.21.15.27:0/3276093175 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6258 (secure 0 0 0) 0x7fa300058940 con 0x7fa2f40052e0 2024-03-28T19:06:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:05.562 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5336,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:05.563 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.565+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2ec06bc00 msgr2=0x7fa2ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.565+0000 7fa314a00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2ec06bc00 0x7fa2ec06e0c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa3101e26c0 tx=0x7fa2f8008040 comp rx=0 tx=0).stop 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 msgr2=0x7fa3100abce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa3100abce0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa30000e780 tx=0x7fa30000ec40 comp rx=0 tx=0).stop 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 shutdown_connections 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2ec06bc00 0x7fa2ec06e0c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa310152450 0x7fa3101eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3101e1880 0x7fa3101519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 --2- 172.21.15.27:0/3276093175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2f40052e0 0x7fa3100abce0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:05.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 >> 172.21.15.27:0/3276093175 conn(0x7fa31006b6f0 msgr2=0x7fa3100c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:05.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 shutdown_connections 2024-03-28T19:06:05.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:05.566+0000 7fa314a00640 1 -- 172.21.15.27:0/3276093175 wait complete. 2024-03-28T19:06:05.585 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:06:07.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:07.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.823+0000 7f60d8a00640 1 Processor -- start 2024-03-28T19:06:07.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.824+0000 7f60d8a00640 1 -- start start 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.825+0000 7f60d8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d4150ae0 0x7f60d40a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.825+0000 7f60d8a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f60d40a61a0 0x7f60d40c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.825+0000 7f60d8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60d40c6ee0 0x7f60d4153a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.825+0000 7f60d8a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d4150ec0 con 0x7f60d40c6ee0 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.825+0000 7f60d8a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d406dc70 con 0x7f60d40a61a0 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.825+0000 7f60d8a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4153550 con 0x7f60d4150ae0 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60d40c6ee0 0x7f60d4153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60d40c6ee0 0x7f60d4153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35680/0 (socket says 172.21.15.27:35680) 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3e00640 1 -- 172.21.15.27:0/1895862088 learned_addr learned my addr 172.21.15.27:0/1895862088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3400640 1 --2- 172.21.15.27:0/1895862088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d4150ae0 0x7f60d40a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60cbe00640 1 --2- 172.21.15.27:0/1895862088 >> v2:172.21.15.155:3300/0 conn(0x7f60d40a61a0 0x7f60d40c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3e00640 1 -- 172.21.15.27:0/1895862088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d4150ae0 msgr2=0x7f60d40a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3e00640 1 --2- 172.21.15.27:0/1895862088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d4150ae0 0x7f60d40a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3e00640 1 -- 172.21.15.27:0/1895862088 >> v2:172.21.15.155:3300/0 conn(0x7f60d40a61a0 msgr2=0x7f60d40c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3e00640 1 --2- 172.21.15.27:0/1895862088 >> v2:172.21.15.155:3300/0 conn(0x7f60d40a61a0 0x7f60d40c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.826+0000 7f60d3e00640 1 -- 172.21.15.27:0/1895862088 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d406df10 con 0x7f60d40c6ee0 2024-03-28T19:06:07.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.828+0000 7f60d3e00640 1 --2- 172.21.15.27:0/1895862088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d40c6ee0 0x7f60d4153a30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f60d40abd20 tx=0x7f60cc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631cccabe31099fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:07.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.828+0000 7f60d2a00640 1 -- 172.21.15.27:0/1895862088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc02c9e0 con 0x7f60d40c6ee0 2024-03-28T19:06:07.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.828+0000 7f60d2a00640 1 -- 172.21.15.27:0/1895862088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc02cb80 con 0x7f60d40c6ee0 2024-03-28T19:06:07.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.828+0000 7f60d2a00640 1 -- 172.21.15.27:0/1895862088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0343f0 con 0x7f60d40c6ee0 2024-03-28T19:06:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 -- 172.21.15.27:0/1895862088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d40c6ee0 msgr2=0x7f60d4153a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 --2- 172.21.15.27:0/1895862088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d40c6ee0 0x7f60d4153a30 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f60d40abd20 tx=0x7f60cc02be40 comp rx=0 tx=0).stop 2024-03-28T19:06:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 -- 172.21.15.27:0/1895862088 shutdown_connections 2024-03-28T19:06:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 --2- 172.21.15.27:0/1895862088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d40c6ee0 0x7f60d4153a30 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 --2- 172.21.15.27:0/1895862088 >> v2:172.21.15.155:3300/0 conn(0x7f60d40a61a0 0x7f60d40c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 --2- 172.21.15.27:0/1895862088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d4150ae0 0x7f60d40a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:07.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 -- 172.21.15.27:0/1895862088 >> 172.21.15.27:0/1895862088 conn(0x7f60d406b0c0 msgr2=0x7f60d406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:07.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 -- 172.21.15.27:0/1895862088 shutdown_connections 2024-03-28T19:06:07.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.829+0000 7f60d8a00640 1 -- 172.21.15.27:0/1895862088 wait complete. 2024-03-28T19:06:07.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.831+0000 7f60d8a00640 1 Processor -- start 2024-03-28T19:06:07.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.831+0000 7f60d8a00640 1 -- start start 2024-03-28T19:06:07.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d8a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d40a61a0 0x7f60d40ab740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:07.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d40c6ee0 0x7f60d40a9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:07.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d4150ae0 0x7f60d40728b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:07.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d8a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d406e7b0 con 0x7f60d4150ae0 2024-03-28T19:06:07.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d8a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d406e4d0 con 0x7f60d40a61a0 2024-03-28T19:06:07.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d8a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d41569d0 con 0x7f60d40c6ee0 2024-03-28T19:06:07.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d4150ae0 0x7f60d40728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:07.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d4150ae0 0x7f60d40728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35688/0 (socket says 172.21.15.27:35688) 2024-03-28T19:06:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.832+0000 7f60d3e00640 1 -- 172.21.15.27:0/3235202596 learned_addr learned my addr 172.21.15.27:0/3235202596 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.833+0000 7f60d3400640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d40a61a0 0x7f60d40ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.833+0000 7f60cbe00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d40c6ee0 0x7f60d40a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.833+0000 7f60d3e00640 1 -- 172.21.15.27:0/3235202596 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d40c6ee0 msgr2=0x7f60d40a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.833+0000 7f60d3e00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d40c6ee0 0x7f60d40a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.833+0000 7f60d3e00640 1 -- 172.21.15.27:0/3235202596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d40a61a0 msgr2=0x7f60d40ab740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.833+0000 7f60d3e00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d40a61a0 0x7f60d40ab740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.833+0000 7f60d3e00640 1 -- 172.21.15.27:0/3235202596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d41e77e0 con 0x7f60d4150ae0 2024-03-28T19:06:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.833+0000 7f60d3e00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d4150ae0 0x7f60d40728b0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009f40 tx=0x7f60cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.834+0000 7f60cb400640 1 -- 172.21.15.27:0/3235202596 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc043020 con 0x7f60d4150ae0 2024-03-28T19:06:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.834+0000 7f60cb400640 1 -- 172.21.15.27:0/3235202596 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc03e450 con 0x7f60d4150ae0 2024-03-28T19:06:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.834+0000 7f60cb400640 1 -- 172.21.15.27:0/3235202596 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc009510 con 0x7f60d4150ae0 2024-03-28T19:06:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.834+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d41e7a80 con 0x7f60d4150ae0 2024-03-28T19:06:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.834+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d4148770 con 0x7f60d4150ae0 2024-03-28T19:06:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.835+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d4067dd0 con 0x7f60d4150ae0 2024-03-28T19:06:07.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.836+0000 7f60cb400640 1 -- 172.21.15.27:0/3235202596 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f60cc046020 con 0x7f60d4150ae0 2024-03-28T19:06:07.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.837+0000 7f60cb400640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60b006bbd0 0x7f60b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:07.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.837+0000 7f60cb400640 1 -- 172.21.15.27:0/3235202596 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7f60cc0adc80 con 0x7f60d4150ae0 2024-03-28T19:06:07.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.838+0000 7f60d3400640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60b006bbd0 0x7f60b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:07.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.838+0000 7f60d3400640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60b006bbd0 0x7f60b006e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f60c4006fd0 tx=0x7f60c4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:07.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:07.843+0000 7f60cb400640 1 -- 172.21.15.27:0/3235202596 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f60cc04d020 con 0x7f60d4150ae0 2024-03-28T19:06:08.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.097+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d4069640 con 0x7f60d4150ae0 2024-03-28T19:06:08.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.098+0000 7f60cb400640 1 -- 172.21.15.27:0/3235202596 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6258 (secure 0 0 0) 0x7f60cc07c120 con 0x7f60d4150ae0 2024-03-28T19:06:08.097 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:08.097 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5336,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:08.098 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-03-28T19:06:08.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60b006bbd0 msgr2=0x7f60b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:08.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60b006bbd0 0x7f60b006e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f60c4006fd0 tx=0x7f60c4008040 comp rx=0 tx=0).stop 2024-03-28T19:06:08.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d4150ae0 msgr2=0x7f60d40728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:08.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d4150ae0 0x7f60d40728b0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009f40 tx=0x7f60cc004a40 comp rx=0 tx=0).stop 2024-03-28T19:06:08.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 shutdown_connections 2024-03-28T19:06:08.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60b006bbd0 0x7f60b006e090 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:08.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60d4150ae0 0x7f60d40728b0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:08.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60d40c6ee0 0x7f60d40a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:08.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 --2- 172.21.15.27:0/3235202596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d40a61a0 0x7f60d40ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:08.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 >> 172.21.15.27:0/3235202596 conn(0x7f60d406b0c0 msgr2=0x7f60d40c0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:08.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 shutdown_connections 2024-03-28T19:06:08.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:08.101+0000 7f60d8a00640 1 -- 172.21.15.27:0/3235202596 wait complete. 2024-03-28T19:06:08.119 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:06:10.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.343+0000 7f30fc200640 1 Processor -- start 2024-03-28T19:06:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.343+0000 7f30fc200640 1 -- start start 2024-03-28T19:06:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.345+0000 7f30fc200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 0x7f30f406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.345+0000 7f30fc200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f30f406f1e0 0x7f30f4070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.345+0000 7f30fc200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40c52e0 con 0x7f30f40c4dc0 2024-03-28T19:06:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.345+0000 7f30fc200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f41557e0 con 0x7f30f406f1e0 2024-03-28T19:06:10.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.345+0000 7f30fc200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f4155ac0 con 0x7f30f4152bd0 2024-03-28T19:06:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.346+0000 7f30f8e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f30f406f1e0 0x7f30f4070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.346+0000 7f30f9800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 0x7f30f406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.346+0000 7f30f8e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f30f406f1e0 0x7f30f4070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:45018/0 (socket says 172.21.15.27:45018) 2024-03-28T19:06:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.346+0000 7f30f8e00640 1 -- 172.21.15.27:0/585676869 learned_addr learned my addr 172.21.15.27:0/585676869 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.346+0000 7f30f8e00640 1 -- 172.21.15.27:0/585676869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 msgr2=0x7f30f406eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.346+0000 7f30f8e00640 1 --2- 172.21.15.27:0/585676869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 0x7f30f406eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.346+0000 7f30f8e00640 1 -- 172.21.15.27:0/585676869 >> v1:172.21.15.155:6789/0 conn(0x7f30f40c4dc0 legacy=0x7f30f4158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.346+0000 7f30f8e00640 1 -- 172.21.15.27:0/585676869 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f4155da0 con 0x7f30f406f1e0 2024-03-28T19:06:10.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.348+0000 7f30f8e00640 1 --2- 172.21.15.27:0/585676869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f406f1e0 0x7f30f4070540 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009870 tx=0x7f30e002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a22c21eb764118d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:10.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.348+0000 7f30f3e00640 1 -- 172.21.15.27:0/585676869 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0034070 con 0x7f30f406f1e0 2024-03-28T19:06:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.348+0000 7f30f3e00640 1 -- 172.21.15.27:0/585676869 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e0030840 con 0x7f30f406f1e0 2024-03-28T19:06:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.349+0000 7f30f3e00640 1 -- 172.21.15.27:0/585676869 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0030c10 con 0x7f30f406f1e0 2024-03-28T19:06:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.349+0000 7f30fc200640 1 -- 172.21.15.27:0/585676869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f406f1e0 msgr2=0x7f30f4070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.349+0000 7f30fc200640 1 --2- 172.21.15.27:0/585676869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f406f1e0 0x7f30f4070540 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009870 tx=0x7f30e002b980 comp rx=0 tx=0).stop 2024-03-28T19:06:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.349+0000 7f30fc200640 1 -- 172.21.15.27:0/585676869 shutdown_connections 2024-03-28T19:06:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.349+0000 7f30fc200640 1 --2- 172.21.15.27:0/585676869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f406f1e0 0x7f30f4070540 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.349+0000 7f30fc200640 1 --2- 172.21.15.27:0/585676869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 0x7f30f406eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.349+0000 7f30fc200640 1 -- 172.21.15.27:0/585676869 >> 172.21.15.27:0/585676869 conn(0x7f30f4066dd0 msgr2=0x7f30f40671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.350+0000 7f30fc200640 1 -- 172.21.15.27:0/585676869 shutdown_connections 2024-03-28T19:06:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.350+0000 7f30fc200640 1 -- 172.21.15.27:0/585676869 wait complete. 2024-03-28T19:06:10.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.352+0000 7f30fc200640 1 Processor -- start 2024-03-28T19:06:10.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.352+0000 7f30fc200640 1 -- start start 2024-03-28T19:06:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.353+0000 7f30fc200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30f406f1e0 0x7f30f4156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.353+0000 7f30fc200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f40c4dc0 0x7f30f41571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.353+0000 7f30fc200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 0x7f30f41e20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.353+0000 7f30fc200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f40c6df0 con 0x7f30f406f1e0 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.353+0000 7f30fc200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f40c6c70 con 0x7f30f40c4dc0 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.353+0000 7f30fc200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f4070a80 con 0x7f30f4152bd0 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30f406f1e0 0x7f30f4156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30f406f1e0 0x7f30f4156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35700/0 (socket says 172.21.15.27:35700) 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f9800640 1 -- 172.21.15.27:0/3408840053 learned_addr learned my addr 172.21.15.27:0/3408840053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f8e00640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f40c4dc0 0x7f30f41571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30fa200640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 0x7f30f41e20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f9800640 1 -- 172.21.15.27:0/3408840053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 msgr2=0x7f30f41e20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f9800640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 0x7f30f41e20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f9800640 1 -- 172.21.15.27:0/3408840053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f40c4dc0 msgr2=0x7f30f41571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f9800640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f40c4dc0 0x7f30f41571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.354+0000 7f30f9800640 1 -- 172.21.15.27:0/3408840053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f41eba40 con 0x7f30f406f1e0 2024-03-28T19:06:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.355+0000 7f30f9800640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30f406f1e0 0x7f30f4156c80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f30e800e9e0 tx=0x7f30e800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.355+0000 7f30f1600640 1 -- 172.21.15.27:0/3408840053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8017070 con 0x7f30f406f1e0 2024-03-28T19:06:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.355+0000 7f30f1600640 1 -- 172.21.15.27:0/3408840053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e800c9d0 con 0x7f30f406f1e0 2024-03-28T19:06:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.355+0000 7f30f1600640 1 -- 172.21.15.27:0/3408840053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e800cdc0 con 0x7f30f406f1e0 2024-03-28T19:06:10.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.355+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f40abcf0 con 0x7f30f406f1e0 2024-03-28T19:06:10.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.355+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f40ac2b0 con 0x7f30f406f1e0 2024-03-28T19:06:10.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.357+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f40c4370 con 0x7f30f406f1e0 2024-03-28T19:06:10.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.358+0000 7f30f1600640 1 -- 172.21.15.27:0/3408840053 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f30e801a070 con 0x7f30f406f1e0 2024-03-28T19:06:10.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.359+0000 7f30f1600640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30d406bbd0 0x7f30d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:10.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.359+0000 7f30f1600640 1 -- 172.21.15.27:0/3408840053 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7f30e808b6a0 con 0x7f30f406f1e0 2024-03-28T19:06:10.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.359+0000 7f30f8e00640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30d406bbd0 0x7f30d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:10.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.360+0000 7f30f8e00640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30d406bbd0 0x7f30d406e090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f30e0036000 tx=0x7f30e003b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:10.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.369+0000 7f30f1600640 1 -- 172.21.15.27:0/3408840053 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f30f40c4370 con 0x7f30f406f1e0 2024-03-28T19:06:10.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.626+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30f404f400 con 0x7f30f406f1e0 2024-03-28T19:06:10.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.627+0000 7f30f1600640 1 -- 172.21.15.27:0/3408840053 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6258 (secure 0 0 0) 0x7f30e8059980 con 0x7f30f406f1e0 2024-03-28T19:06:10.627 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:10.627 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5336,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:10.627 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30d406bbd0 msgr2=0x7f30d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30d406bbd0 0x7f30d406e090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f30e0036000 tx=0x7f30e003b040 comp rx=0 tx=0).stop 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30f406f1e0 msgr2=0x7f30f4156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30f406f1e0 0x7f30f4156c80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f30e800e9e0 tx=0x7f30e800eea0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 shutdown_connections 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f30d406bbd0 0x7f30d406e090 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30f4152bd0 0x7f30f41e20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30f40c4dc0 0x7f30f41571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 --2- 172.21.15.27:0/3408840053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30f406f1e0 0x7f30f4156c80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 >> 172.21.15.27:0/3408840053 conn(0x7f30f4066dd0 msgr2=0x7f30f40c2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 shutdown_connections 2024-03-28T19:06:10.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:10.630+0000 7f30fc200640 1 -- 172.21.15.27:0/3408840053 wait complete. 2024-03-28T19:06:10.656 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:06:12.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.891+0000 7fcfb1400640 1 Processor -- start 2024-03-28T19:06:12.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.891+0000 7fcfb1400640 1 -- start start 2024-03-28T19:06:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.892+0000 7fcfb1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcfac150c70 0x7fcfac0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.892+0000 7fcfb1400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 0x7fcfac0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.892+0000 7fcfb1400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfac0c3af0 con 0x7fcfac0c6660 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.892+0000 7fcfb1400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfac0ab4e0 con 0x7fcfac150c70 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.892+0000 7fcfb1400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0ac1f0 con 0x7fcfac0c3690 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.893+0000 7fcfaa800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcfac150c70 0x7fcfac0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.893+0000 7fcfaa800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcfac150c70 0x7fcfac0a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35702/0 (socket says 172.21.15.27:35702) 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.893+0000 7fcfaa800640 1 -- 172.21.15.27:0/1936274125 learned_addr learned my addr 172.21.15.27:0/1936274125 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.893+0000 7fcfa3e00640 1 --2- 172.21.15.27:0/1936274125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 0x7fcfac0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.893+0000 7fcfaa800640 1 -- 172.21.15.27:0/1936274125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 msgr2=0x7fcfac0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.893+0000 7fcfaa800640 1 --2- 172.21.15.27:0/1936274125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 0x7fcfac0c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.893+0000 7fcfaa800640 1 -- 172.21.15.27:0/1936274125 >> v1:172.21.15.155:6789/0 conn(0x7fcfac0c6660 legacy=0x7fcfac1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:12.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.893+0000 7fcfaa800640 1 -- 172.21.15.27:0/1936274125 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac0c3d90 con 0x7fcfac150c70 2024-03-28T19:06:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.894+0000 7fcfaa800640 1 --2- 172.21.15.27:0/1936274125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac0a9cb0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009870 tx=0x7fcf9c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f89b479352bdde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.895+0000 7fcfa9e00640 1 -- 172.21.15.27:0/1936274125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0317c0 con 0x7fcfac150c70 2024-03-28T19:06:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.895+0000 7fcfa9e00640 1 -- 172.21.15.27:0/1936274125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c037070 con 0x7fcfac150c70 2024-03-28T19:06:12.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.895+0000 7fcfa9e00640 1 -- 172.21.15.27:0/1936274125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c031b80 con 0x7fcfac150c70 2024-03-28T19:06:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.896+0000 7fcfb1400640 1 -- 172.21.15.27:0/1936274125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 msgr2=0x7fcfac0a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.896+0000 7fcfb1400640 1 --2- 172.21.15.27:0/1936274125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac0a9cb0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009870 tx=0x7fcf9c02b860 comp rx=0 tx=0).stop 2024-03-28T19:06:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.896+0000 7fcfb1400640 1 -- 172.21.15.27:0/1936274125 shutdown_connections 2024-03-28T19:06:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.896+0000 7fcfb1400640 1 --2- 172.21.15.27:0/1936274125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 0x7fcfac0c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.896+0000 7fcfb1400640 1 --2- 172.21.15.27:0/1936274125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac0a9cb0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.896+0000 7fcfb1400640 1 -- 172.21.15.27:0/1936274125 >> 172.21.15.27:0/1936274125 conn(0x7fcfac06b9d0 msgr2=0x7fcfac0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.896+0000 7fcfb1400640 1 -- 172.21.15.27:0/1936274125 shutdown_connections 2024-03-28T19:06:12.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.896+0000 7fcfb1400640 1 -- 172.21.15.27:0/1936274125 wait complete. 2024-03-28T19:06:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.898+0000 7fcfb1400640 1 Processor -- start 2024-03-28T19:06:12.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.898+0000 7fcfb1400640 1 -- start start 2024-03-28T19:06:12.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfb1400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 0x7fcfac0a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:12.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfb1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0c6660 0x7fcfac157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfb1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfb1400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0abcd0 con 0x7fcfac150c70 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfb1400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0ab9f0 con 0x7fcfac0c6660 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfb1400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac0c1810 con 0x7fcfac0c3690 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfab200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfab200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac146b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35712/0 (socket says 172.21.15.27:35712) 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfab200640 1 -- 172.21.15.27:0/3269496223 learned_addr learned my addr 172.21.15.27:0/3269496223 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfa3e00640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0c6660 0x7fcfac157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:12.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.899+0000 7fcfaa800640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 0x7fcfac0a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:12.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfab200640 1 -- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 msgr2=0x7fcfac0a7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfab200640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 0x7fcfac0a7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfab200640 1 -- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0c6660 msgr2=0x7fcfac157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfab200640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0c6660 0x7fcfac157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfab200640 1 -- 172.21.15.27:0/3269496223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac1ee3c0 con 0x7fcfac150c70 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfa3e00640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0c6660 0x7fcfac157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfab200640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac146b90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400c780 tx=0x7fcfa400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfa2a00640 1 -- 172.21.15.27:0/3269496223 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4004b60 con 0x7fcfac150c70 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfa2a00640 1 -- 172.21.15.27:0/3269496223 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa4004d00 con 0x7fcfac150c70 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfa2a00640 1 -- 172.21.15.27:0/3269496223 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa400aa90 con 0x7fcfac150c70 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.900+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac151c90 con 0x7fcfac150c70 2024-03-28T19:06:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.901+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac152250 con 0x7fcfac150c70 2024-03-28T19:06:12.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.902+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac04f400 con 0x7fcfac150c70 2024-03-28T19:06:12.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.903+0000 7fcfa2a00640 1 -- 172.21.15.27:0/3269496223 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcfa400ac30 con 0x7fcfac150c70 2024-03-28T19:06:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.905+0000 7fcfa2a00640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf8806bbd0 0x7fcf8806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.905+0000 7fcfa2a00640 1 -- 172.21.15.27:0/3269496223 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7755+0+0 (secure 0 0 0) 0x7fcfa408b500 con 0x7fcfac150c70 2024-03-28T19:06:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.905+0000 7fcfaa800640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf8806bbd0 0x7fcf8806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:12.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.906+0000 7fcfaa800640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf8806bbd0 0x7fcf8806e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c036000 tx=0x7fcf9c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:12.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:12.912+0000 7fcfa2a00640 1 -- 172.21.15.27:0/3269496223 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcfa4020070 con 0x7fcfac150c70 2024-03-28T19:06:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.219+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfac1ee590 con 0x7fcfac150c70 2024-03-28T19:06:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.221+0000 7fcfa2a00640 1 -- 172.21.15.27:0/3269496223 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6258 (secure 0 0 0) 0x7fcfa40597e0 con 0x7fcfac150c70 2024-03-28T19:06:13.220 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:13.220 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":5336,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:04:59.622490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6192},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6192":{"gid":6192,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":4,"addr":"172.21.15.27:6836/2888303230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2888303230},{"type":"v1","addr":"172.21.15.27:6836","nonce":2888303230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6192,"qdb_cluster":[6192]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 86 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf8806bbd0 msgr2=0x7fcf8806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf8806bbd0 0x7fcf8806e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c036000 tx=0x7fcf9c03d040 comp rx=0 tx=0).stop 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 msgr2=0x7fcfac146b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac146b90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400c780 tx=0x7fcfa400cc40 comp rx=0 tx=0).stop 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 shutdown_connections 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf8806bbd0 0x7fcf8806e090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfac150c70 0x7fcfac146b90 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0c6660 0x7fcfac157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 --2- 172.21.15.27:0/3269496223 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfac0c3690 0x7fcfac0a7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 >> 172.21.15.27:0/3269496223 conn(0x7fcfac06b9d0 msgr2=0x7fcfac0c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 shutdown_connections 2024-03-28T19:06:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:13.223+0000 7fcfb1400640 1 -- 172.21.15.27:0/3269496223 wait complete. 2024-03-28T19:06:13.245 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:06:15.247 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:15.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.478+0000 7fbb23e00640 1 Processor -- start 2024-03-28T19:06:15.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.479+0000 7fbb23e00640 1 -- start start 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.480+0000 7fbb23e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbb24152a70 0x7fbb24159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.480+0000 7fbb23e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb24155b40 0x7fbb24155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.480+0000 7fbb23e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb240c5710 con 0x7fbb240c6cc0 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.480+0000 7fbb23e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb241556b0 con 0x7fbb24155b40 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.480+0000 7fbb23e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb240c59f0 con 0x7fbb24152a70 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.480+0000 7fbb23400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb24155b40 0x7fbb24155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.480+0000 7fbb23400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb24155b40 0x7fbb24155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35722/0 (socket says 172.21.15.27:35722) 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.481+0000 7fbb23400640 1 -- 172.21.15.27:0/1133412660 learned_addr learned my addr 172.21.15.27:0/1133412660 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:15.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.481+0000 7fbb22a00640 1 --2- 172.21.15.27:0/1133412660 >> v2:172.21.15.155:3300/0 conn(0x7fbb24152a70 0x7fbb24159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:15.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.481+0000 7fbb23400640 1 -- 172.21.15.27:0/1133412660 >> v2:172.21.15.155:3300/0 conn(0x7fbb24152a70 msgr2=0x7fbb24159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:15.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.481+0000 7fbb23400640 1 --2- 172.21.15.27:0/1133412660 >> v2:172.21.15.155:3300/0 conn(0x7fbb24152a70 0x7fbb24159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.481+0000 7fbb23400640 1 -- 172.21.15.27:0/1133412660 >> v1:172.21.15.27:6789/0 conn(0x7fbb240c6cc0 legacy=0x7fbb240cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:15.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.481+0000 7fbb23400640 1 -- 172.21.15.27:0/1133412660 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb240c5cd0 con 0x7fbb24155b40 2024-03-28T19:06:15.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.482+0000 7fbb23400640 1 --2- 172.21.15.27:0/1133412660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb24155fb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbb240c8870 tx=0x7fbb1c02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a8254a6bb2b1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:15.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.483+0000 7fbb22000640 1 -- 172.21.15.27:0/1133412660 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c035020 con 0x7fbb24155b40 2024-03-28T19:06:15.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.483+0000 7fbb22000640 1 -- 172.21.15.27:0/1133412660 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c031890 con 0x7fbb24155b40 2024-03-28T19:06:15.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.483+0000 7fbb22000640 1 -- 172.21.15.27:0/1133412660 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c031c60 con 0x7fbb24155b40 2024-03-28T19:06:15.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.483+0000 7fbb23e00640 1 -- 172.21.15.27:0/1133412660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 msgr2=0x7fbb24155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:15.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.483+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1133412660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb24155fb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbb240c8870 tx=0x7fbb1c02b930 comp rx=0 tx=0).stop 2024-03-28T19:06:15.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.484+0000 7fbb23e00640 1 -- 172.21.15.27:0/1133412660 shutdown_connections 2024-03-28T19:06:15.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.484+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1133412660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb24155fb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.484+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1133412660 >> v2:172.21.15.155:3300/0 conn(0x7fbb24152a70 0x7fbb24159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.484+0000 7fbb23e00640 1 -- 172.21.15.27:0/1133412660 >> 172.21.15.27:0/1133412660 conn(0x7fbb2406b2a0 msgr2=0x7fbb2406b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:15.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.484+0000 7fbb23e00640 1 -- 172.21.15.27:0/1133412660 shutdown_connections 2024-03-28T19:06:15.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.484+0000 7fbb23e00640 1 -- 172.21.15.27:0/1133412660 wait complete. 2024-03-28T19:06:15.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.485+0000 7fbb23e00640 1 Processor -- start 2024-03-28T19:06:15.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.485+0000 7fbb23e00640 1 -- start start 2024-03-28T19:06:15.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.486+0000 7fbb23e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb240c6cc0 0x7fbb24147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.486+0000 7fbb23e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb24152a70 0x7fbb24147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.486+0000 7fbb23e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb240aa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.486+0000 7fbb23e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb240cb7f0 con 0x7fbb24155b40 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.486+0000 7fbb23e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb24072a70 con 0x7fbb240c6cc0 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.486+0000 7fbb23e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb240cb970 con 0x7fbb24152a70 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb23400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb240aa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb23400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb240aa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35730/0 (socket says 172.21.15.27:35730) 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb23400640 1 -- 172.21.15.27:0/1252361847 learned_addr learned my addr 172.21.15.27:0/1252361847 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb22a00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb240c6cc0 0x7fbb24147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb23400640 1 -- 172.21.15.27:0/1252361847 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb24152a70 msgr2=0x7fbb24147ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T19:06:15.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb23400640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb24152a70 0x7fbb24147ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb23400640 1 -- 172.21.15.27:0/1252361847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb240c6cc0 msgr2=0x7fbb24147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:15.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb23400640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb240c6cc0 0x7fbb24147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb23400640 1 -- 172.21.15.27:0/1252361847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb241507c0 con 0x7fbb24155b40 2024-03-28T19:06:15.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.487+0000 7fbb22a00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb240c6cc0 0x7fbb24147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:06:15.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.488+0000 7fbb23400640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb240aa8a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c02b970 tx=0x7fbb1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.488+0000 7fbb1b400640 1 -- 172.21.15.27:0/1252361847 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c040070 con 0x7fbb24155b40 2024-03-28T19:06:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.488+0000 7fbb1b400640 1 -- 172.21.15.27:0/1252361847 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c030d20 con 0x7fbb24155b40 2024-03-28T19:06:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.488+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb241499a0 con 0x7fbb24155b40 2024-03-28T19:06:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.488+0000 7fbb1b400640 1 -- 172.21.15.27:0/1252361847 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c035040 con 0x7fbb24155b40 2024-03-28T19:06:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.488+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb240ac380 con 0x7fbb24155b40 2024-03-28T19:06:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.489+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb24067dd0 con 0x7fbb24155b40 2024-03-28T19:06:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.490+0000 7fbb1b400640 1 -- 172.21.15.27:0/1252361847 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbb1c038020 con 0x7fbb24155b40 2024-03-28T19:06:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.492+0000 7fbb1b400640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbb0006bbd0 0x7fbb0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.492+0000 7fbb1b400640 1 -- 172.21.15.27:0/1252361847 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7fbb1c0ad100 con 0x7fbb24155b40 2024-03-28T19:06:15.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.492+0000 7fbb22a00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbb0006bbd0 0x7fbb0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.493+0000 7fbb22a00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbb0006bbd0 0x7fbb0006e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbb140074c0 tx=0x7fbb1400b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:15.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.496+0000 7fbb1b400640 1 -- 172.21.15.27:0/1252361847 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbb1c02d040 con 0x7fbb24155b40 2024-03-28T19:06:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.760+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb241511f0 con 0x7fbb24155b40 2024-03-28T19:06:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.761+0000 7fbb1b400640 1 -- 172.21.15.27:0/1252361847 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5460 (secure 0 0 0) 0x7fbb241511f0 con 0x7fbb24155b40 2024-03-28T19:06:15.760 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:15.760 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:14.812095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 87 2024-03-28T19:06:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbb0006bbd0 msgr2=0x7fbb0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbb0006bbd0 0x7fbb0006e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbb140074c0 tx=0x7fbb1400b040 comp rx=0 tx=0).stop 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 msgr2=0x7fbb240aa8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb240aa8a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c02b970 tx=0x7fbb1c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 shutdown_connections 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbb0006bbd0 0x7fbb0006e090 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb24155b40 0x7fbb240aa8a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb24152a70 0x7fbb24147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 --2- 172.21.15.27:0/1252361847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb240c6cc0 0x7fbb24147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 >> 172.21.15.27:0/1252361847 conn(0x7fbb2406b2a0 msgr2=0x7fbb24061ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 shutdown_connections 2024-03-28T19:06:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:15.764+0000 7fbb23e00640 1 -- 172.21.15.27:0/1252361847 wait complete. 2024-03-28T19:06:15.783 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-03-28T19:06:15.784 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:06:15.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:16.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.004+0000 7f6583e00640 1 Processor -- start 2024-03-28T19:06:16.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.004+0000 7f6583e00640 1 -- start start 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.006+0000 7f6583e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c58c0 0x7f65840c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.006+0000 7f6583e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65840c8ed0 0x7f65841555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.006+0000 7f6583e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6584156230 con 0x7f65840be5d0 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.006+0000 7f6583e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6584158bd0 con 0x7f65840c8ed0 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.006+0000 7f6583e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6584158eb0 con 0x7f65840c58c0 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.006+0000 7f6582000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65840c8ed0 0x7f65841555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.007+0000 7f6582000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65840c8ed0 0x7f65841555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35742/0 (socket says 172.21.15.27:35742) 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.007+0000 7f6582000640 1 -- 172.21.15.27:0/1187014388 learned_addr learned my addr 172.21.15.27:0/1187014388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.007+0000 7f6582a00640 1 --2- 172.21.15.27:0/1187014388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c58c0 0x7f65840c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:16.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.007+0000 7f6582000640 1 -- 172.21.15.27:0/1187014388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c58c0 msgr2=0x7f65840c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:16.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.007+0000 7f6582000640 1 --2- 172.21.15.27:0/1187014388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c58c0 0x7f65840c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.007+0000 7f6582000640 1 -- 172.21.15.27:0/1187014388 >> v1:172.21.15.27:6789/0 conn(0x7f65840be5d0 legacy=0x7f6584155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:16.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.007+0000 7f6582000640 1 -- 172.21.15.27:0/1187014388 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584159190 con 0x7f65840c8ed0 2024-03-28T19:06:16.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.007+0000 7f6582a00640 1 --2- 172.21.15.27:0/1187014388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c58c0 0x7f65840c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:06:16.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.008+0000 7f6582000640 1 --2- 172.21.15.27:0/1187014388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65841555e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6570009870 tx=0x7f657002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650ac00bb56315b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:16.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.009+0000 7f6581600640 1 -- 172.21.15.27:0/1187014388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570035020 con 0x7f65840c8ed0 2024-03-28T19:06:16.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.009+0000 7f6581600640 1 -- 172.21.15.27:0/1187014388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6570031640 con 0x7f65840c8ed0 2024-03-28T19:06:16.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.009+0000 7f6581600640 1 -- 172.21.15.27:0/1187014388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657002d040 con 0x7f65840c8ed0 2024-03-28T19:06:16.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.010+0000 7f6583e00640 1 -- 172.21.15.27:0/1187014388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 msgr2=0x7f65841555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:16.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.010+0000 7f6583e00640 1 --2- 172.21.15.27:0/1187014388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65841555e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6570009870 tx=0x7f657002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.010+0000 7f6583e00640 1 -- 172.21.15.27:0/1187014388 shutdown_connections 2024-03-28T19:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.010+0000 7f6583e00640 1 --2- 172.21.15.27:0/1187014388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65841555e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.010+0000 7f6583e00640 1 --2- 172.21.15.27:0/1187014388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840c58c0 0x7f65840c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.010+0000 7f6583e00640 1 -- 172.21.15.27:0/1187014388 >> 172.21.15.27:0/1187014388 conn(0x7f658406b6f0 msgr2=0x7f65840acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.010+0000 7f6583e00640 1 -- 172.21.15.27:0/1187014388 shutdown_connections 2024-03-28T19:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.010+0000 7f6583e00640 1 -- 172.21.15.27:0/1187014388 wait complete. 2024-03-28T19:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.012+0000 7f6583e00640 1 Processor -- start 2024-03-28T19:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.012+0000 7f6583e00640 1 -- start start 2024-03-28T19:06:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.013+0000 7f6583e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840be5d0 0x7f65841ecb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.013+0000 7f6583e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c58c0 0x7f65841ec0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.013+0000 7f6583e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65840ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.013+0000 7f6583e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65840b0820 con 0x7f65840c8ed0 2024-03-28T19:06:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.013+0000 7f6583e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584159750 con 0x7f65840c58c0 2024-03-28T19:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.013+0000 7f6583e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65840b09a0 con 0x7f65840be5d0 2024-03-28T19:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.014+0000 7f6582a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840be5d0 0x7f65841ecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.014+0000 7f6583400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65840ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.014+0000 7f6583400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65840ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35752/0 (socket says 172.21.15.27:35752) 2024-03-28T19:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.014+0000 7f6583400640 1 -- 172.21.15.27:0/1568714295 learned_addr learned my addr 172.21.15.27:0/1568714295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.014+0000 7f6582000640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c58c0 0x7f65841ec0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f6583400640 1 -- 172.21.15.27:0/1568714295 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840be5d0 msgr2=0x7f65841ecb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f6583400640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840be5d0 0x7f65841ecb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f6583400640 1 -- 172.21.15.27:0/1568714295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c58c0 msgr2=0x7f65841ec0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f6583400640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c58c0 0x7f65841ec0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f6583400640 1 -- 172.21.15.27:0/1568714295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65841586b0 con 0x7f65840c8ed0 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f6582a00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840be5d0 0x7f65841ecb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f6583400640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65840ac1c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f657c00c680 tx=0x7f657c00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f657aa00640 1 -- 172.21.15.27:0/1568714295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c003b70 con 0x7f65840c8ed0 2024-03-28T19:06:16.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f657aa00640 1 -- 172.21.15.27:0/1568714295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c003d10 con 0x7f65840c8ed0 2024-03-28T19:06:16.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f657aa00640 1 -- 172.21.15.27:0/1568714295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c005310 con 0x7f65840c8ed0 2024-03-28T19:06:16.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.015+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65841ec610 con 0x7f65840c8ed0 2024-03-28T19:06:16.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.016+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65840c1570 con 0x7f65840c8ed0 2024-03-28T19:06:16.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.017+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6584069640 con 0x7f65840c8ed0 2024-03-28T19:06:16.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.018+0000 7f657aa00640 1 -- 172.21.15.27:0/1568714295 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f657c0054b0 con 0x7f65840c8ed0 2024-03-28T19:06:16.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.019+0000 7f657aa00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bbd0 0x7f656406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:16.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.019+0000 7f657aa00640 1 -- 172.21.15.27:0/1568714295 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f657c08b100 con 0x7f65840c8ed0 2024-03-28T19:06:16.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.019+0000 7f6582a00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bbd0 0x7f656406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:16.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.020+0000 7f6582a00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bbd0 0x7f656406e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6574006fd0 tx=0x7f6574008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:16.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.024+0000 7f657aa00640 1 -- 172.21.15.27:0/1568714295 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f65841586b0 con 0x7f65840c8ed0 2024-03-28T19:06:16.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.317+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6584156010 con 0x7f65840c8ed0 2024-03-28T19:06:16.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.318+0000 7f657aa00640 1 -- 172.21.15.27:0/1568714295 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5460 (secure 0 0 0) 0x7f657c058c10 con 0x7f65840c8ed0 2024-03-28T19:06:16.318 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:16.318 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:14.812095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:16.318 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 87 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bbd0 msgr2=0x7f656406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bbd0 0x7f656406e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6574006fd0 tx=0x7f6574008040 comp rx=0 tx=0).stop 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 msgr2=0x7f65840ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65840ac1c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f657c00c680 tx=0x7f657c00cb40 comp rx=0 tx=0).stop 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 shutdown_connections 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f656406bbd0 0x7f656406e090 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65840c8ed0 0x7f65840ac1c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65840c58c0 0x7f65841ec0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 --2- 172.21.15.27:0/1568714295 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65840be5d0 0x7f65841ecb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:16.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 >> 172.21.15.27:0/1568714295 conn(0x7f658406b6f0 msgr2=0x7f65840ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:16.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.321+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 shutdown_connections 2024-03-28T19:06:16.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:16.322+0000 7f6583e00640 1 -- 172.21.15.27:0/1568714295 wait complete. 2024-03-28T19:06:16.349 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 87, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 87, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5336": { "addr": "172.21.15.155:6835/3884653650", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 3884653650, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 3884653650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5336, "incarnation": 87, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:06:14.812095+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5336 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 80, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6054": { "addr": "172.21.15.27:6837/1815850547", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 1815850547, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 1815850547, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6054, "incarnation": 77, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:04:44.514032+0000", "qdb_cluster": [ 6054 ], "qdb_leader": 6054, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6054 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:06:18.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:18.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.580+0000 7efef3200640 1 Processor -- start 2024-03-28T19:06:18.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.580+0000 7efef3200640 1 -- start start 2024-03-28T19:06:18.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.581+0000 7efef3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40abca0 0x7efef4153c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:18.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.581+0000 7efef3200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efef40aa520 0x7efef40ac080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:18.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.581+0000 7efef3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efef40ac5c0 0x7efef406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:18.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.581+0000 7efef3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef4062370 con 0x7efef40ac5c0 2024-03-28T19:06:18.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.581+0000 7efef3200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef4062650 con 0x7efef40aa520 2024-03-28T19:06:18.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.581+0000 7efef3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef40acf90 con 0x7efef40abca0 2024-03-28T19:06:18.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efef40ac5c0 0x7efef406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:18.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efef40ac5c0 0x7efef406dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34930/0 (socket says 172.21.15.27:34930) 2024-03-28T19:06:18.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef2800640 1 -- 172.21.15.27:0/401692674 learned_addr learned my addr 172.21.15.27:0/401692674 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efeebe00640 1 --2- 172.21.15.27:0/401692674 >> v2:172.21.15.155:3300/0 conn(0x7efef40aa520 0x7efef40ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef2800640 1 -- 172.21.15.27:0/401692674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40abca0 msgr2=0x7efef4153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef1e00640 1 --2- 172.21.15.27:0/401692674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40abca0 0x7efef4153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef2800640 1 --2- 172.21.15.27:0/401692674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40abca0 0x7efef4153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef2800640 1 -- 172.21.15.27:0/401692674 >> v2:172.21.15.155:3300/0 conn(0x7efef40aa520 msgr2=0x7efef40ac080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef2800640 1 --2- 172.21.15.27:0/401692674 >> v2:172.21.15.155:3300/0 conn(0x7efef40aa520 0x7efef40ac080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef2800640 1 -- 172.21.15.27:0/401692674 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef40ad270 con 0x7efef40ac5c0 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efeebe00640 1 --2- 172.21.15.27:0/401692674 >> v2:172.21.15.155:3300/0 conn(0x7efef40aa520 0x7efef40ac080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:18.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.582+0000 7efef1e00640 1 --2- 172.21.15.27:0/401692674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40abca0 0x7efef4153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.584+0000 7efef2800640 1 --2- 172.21.15.27:0/401692674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40ac5c0 0x7efef406dee0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7efef40c4f10 tx=0x7efeec02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6798127f436e0837 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.584+0000 7efef1400640 1 -- 172.21.15.27:0/401692674 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec02c9e0 con 0x7efef40ac5c0 2024-03-28T19:06:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.584+0000 7efef1400640 1 -- 172.21.15.27:0/401692674 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec02cb80 con 0x7efef40ac5c0 2024-03-28T19:06:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.584+0000 7efef1400640 1 -- 172.21.15.27:0/401692674 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0343f0 con 0x7efef40ac5c0 2024-03-28T19:06:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 -- 172.21.15.27:0/401692674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40ac5c0 msgr2=0x7efef406dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 --2- 172.21.15.27:0/401692674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40ac5c0 0x7efef406dee0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7efef40c4f10 tx=0x7efeec02be40 comp rx=0 tx=0).stop 2024-03-28T19:06:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 -- 172.21.15.27:0/401692674 shutdown_connections 2024-03-28T19:06:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 --2- 172.21.15.27:0/401692674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40ac5c0 0x7efef406dee0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 --2- 172.21.15.27:0/401692674 >> v2:172.21.15.155:3300/0 conn(0x7efef40aa520 0x7efef40ac080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 --2- 172.21.15.27:0/401692674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40abca0 0x7efef4153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 -- 172.21.15.27:0/401692674 >> 172.21.15.27:0/401692674 conn(0x7efef406b0c0 msgr2=0x7efef406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 -- 172.21.15.27:0/401692674 shutdown_connections 2024-03-28T19:06:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.585+0000 7efef3200640 1 -- 172.21.15.27:0/401692674 wait complete. 2024-03-28T19:06:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.587+0000 7efef3200640 1 Processor -- start 2024-03-28T19:06:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.587+0000 7efef3200640 1 -- start start 2024-03-28T19:06:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efef3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40aa520 0x7efef41dd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efef3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40abca0 0x7efef41ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efef3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efef40ac5c0 0x7efef4155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efef3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef406e420 con 0x7efef40abca0 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efef3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef414a0a0 con 0x7efef40ac5c0 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efef3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef406e5a0 con 0x7efef40aa520 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efeebe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40abca0 0x7efef41ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efeebe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40abca0 0x7efef41ee210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34936/0 (socket says 172.21.15.27:34936) 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efeebe00640 1 -- 172.21.15.27:0/3828276021 learned_addr learned my addr 172.21.15.27:0/3828276021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:18.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efef1e00640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40aa520 0x7efef41dd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efeebe00640 1 -- 172.21.15.27:0/3828276021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40aa520 msgr2=0x7efef41dd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efef2800640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efef40ac5c0 0x7efef4155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efeebe00640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40aa520 0x7efef41dd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efeebe00640 1 -- 172.21.15.27:0/3828276021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efef40ac5c0 msgr2=0x7efef4155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efeebe00640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efef40ac5c0 0x7efef4155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.588+0000 7efeebe00640 1 -- 172.21.15.27:0/3828276021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef41dcb90 con 0x7efef40abca0 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.589+0000 7efeebe00640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40abca0 0x7efef41ee210 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7efedc00c7c0 tx=0x7efedc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.589+0000 7efeea000640 1 -- 172.21.15.27:0/3828276021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc015070 con 0x7efef40abca0 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.589+0000 7efeea000640 1 -- 172.21.15.27:0/3828276021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc007a30 con 0x7efef40abca0 2024-03-28T19:06:18.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.589+0000 7efeea000640 1 -- 172.21.15.27:0/3828276021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc007e00 con 0x7efef40abca0 2024-03-28T19:06:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.589+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef41eea50 con 0x7efef40abca0 2024-03-28T19:06:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.589+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef41ee750 con 0x7efef40abca0 2024-03-28T19:06:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.591+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef4067dd0 con 0x7efef40abca0 2024-03-28T19:06:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.591+0000 7efeea000640 1 -- 172.21.15.27:0/3828276021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efedc018070 con 0x7efef40abca0 2024-03-28T19:06:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.592+0000 7efeea000640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efed006bbd0 0x7efed006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.592+0000 7efeea000640 1 -- 172.21.15.27:0/3828276021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7efedc08b290 con 0x7efef40abca0 2024-03-28T19:06:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.593+0000 7efef1e00640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efed006bbd0 0x7efed006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.593+0000 7efef1e00640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efed006bbd0 0x7efed006e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7efee400a990 tx=0x7efee4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:18.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.597+0000 7efeea000640 1 -- 172.21.15.27:0/3828276021 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efedc059510 con 0x7efef40abca0 2024-03-28T19:06:18.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.866+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efef4069640 con 0x7efef40abca0 2024-03-28T19:06:18.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.867+0000 7efeea000640 1 -- 172.21.15.27:0/3828276021 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5461 (secure 0 0 0) 0x7efedc058e10 con 0x7efef40abca0 2024-03-28T19:06:18.867 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:18.867 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:17.264157+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:18.868 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 89 2024-03-28T19:06:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efed006bbd0 msgr2=0x7efed006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efed006bbd0 0x7efed006e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7efee400a990 tx=0x7efee4008040 comp rx=0 tx=0).stop 2024-03-28T19:06:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40abca0 msgr2=0x7efef41ee210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40abca0 0x7efef41ee210 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7efedc00c7c0 tx=0x7efedc00cc80 comp rx=0 tx=0).stop 2024-03-28T19:06:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 shutdown_connections 2024-03-28T19:06:18.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efed006bbd0 0x7efed006e090 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efef40ac5c0 0x7efef4155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efef40abca0 0x7efef41ee210 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 --2- 172.21.15.27:0/3828276021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efef40aa520 0x7efef41dd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 >> 172.21.15.27:0/3828276021 conn(0x7efef406b0c0 msgr2=0x7efef40c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 shutdown_connections 2024-03-28T19:06:18.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:18.870+0000 7efef3200640 1 -- 172.21.15.27:0/3828276021 wait complete. 2024-03-28T19:06:18.892 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:06:20.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:21.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.132+0000 7f57d5e00640 1 Processor -- start 2024-03-28T19:06:21.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.132+0000 7f57d5e00640 1 -- start start 2024-03-28T19:06:21.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57d5e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f57d006da20 0x7f57d006de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:21.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57d5e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f57d006e340 0x7f57d0154e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:21.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57d5e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00aa430 con 0x7f57d00a5ef0 2024-03-28T19:06:21.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57d5e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d00c21e0 con 0x7f57d006e340 2024-03-28T19:06:21.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57d5e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d00c4980 con 0x7f57d006da20 2024-03-28T19:06:21.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57cfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f57d006e340 0x7f57d0154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:21.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57cfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f57d006e340 0x7f57d0154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34950/0 (socket says 172.21.15.27:34950) 2024-03-28T19:06:21.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57cfe00640 1 -- 172.21.15.27:0/908056059 learned_addr learned my addr 172.21.15.27:0/908056059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:21.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.134+0000 7f57d4a00640 1 --2- 172.21.15.27:0/908056059 >> v2:172.21.15.155:3300/0 conn(0x7f57d006da20 0x7f57d006de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:21.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.135+0000 7f57cfe00640 1 -- 172.21.15.27:0/908056059 >> v2:172.21.15.155:3300/0 conn(0x7f57d006da20 msgr2=0x7f57d006de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:21.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.135+0000 7f57cfe00640 1 --2- 172.21.15.27:0/908056059 >> v2:172.21.15.155:3300/0 conn(0x7f57d006da20 0x7f57d006de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.135+0000 7f57cfe00640 1 -- 172.21.15.27:0/908056059 >> v1:172.21.15.155:6789/0 conn(0x7f57d00a5ef0 legacy=0x7f57d00aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:21.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.135+0000 7f57cfe00640 1 -- 172.21.15.27:0/908056059 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d00c2430 con 0x7f57d006e340 2024-03-28T19:06:21.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.137+0000 7f57cfe00640 1 --2- 172.21.15.27:0/908056059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d006e340 0x7f57d0154e80 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f57d00c2360 tx=0x7f57bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94115f76b0e1e0d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:21.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.137+0000 7f57cf400640 1 -- 172.21.15.27:0/908056059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc034070 con 0x7f57d006e340 2024-03-28T19:06:21.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.137+0000 7f57cf400640 1 -- 172.21.15.27:0/908056059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc030860 con 0x7f57d006e340 2024-03-28T19:06:21.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.137+0000 7f57cf400640 1 -- 172.21.15.27:0/908056059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc030c30 con 0x7f57d006e340 2024-03-28T19:06:21.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.138+0000 7f57d5e00640 1 -- 172.21.15.27:0/908056059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d006e340 msgr2=0x7f57d0154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:21.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.138+0000 7f57d5e00640 1 --2- 172.21.15.27:0/908056059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d006e340 0x7f57d0154e80 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f57d00c2360 tx=0x7f57bc02b9a0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.138+0000 7f57d5e00640 1 -- 172.21.15.27:0/908056059 shutdown_connections 2024-03-28T19:06:21.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.138+0000 7f57d5e00640 1 --2- 172.21.15.27:0/908056059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d006e340 0x7f57d0154e80 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.138+0000 7f57d5e00640 1 --2- 172.21.15.27:0/908056059 >> v2:172.21.15.155:3300/0 conn(0x7f57d006da20 0x7f57d006de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.138+0000 7f57d5e00640 1 -- 172.21.15.27:0/908056059 >> 172.21.15.27:0/908056059 conn(0x7f57d006b0c0 msgr2=0x7f57d006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:21.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.138+0000 7f57d5e00640 1 -- 172.21.15.27:0/908056059 shutdown_connections 2024-03-28T19:06:21.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.138+0000 7f57d5e00640 1 -- 172.21.15.27:0/908056059 wait complete. 2024-03-28T19:06:21.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.140+0000 7f57d5e00640 1 Processor -- start 2024-03-28T19:06:21.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.140+0000 7f57d5e00640 1 -- start start 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f57d006da20 0x7f57d00c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57d006e340 0x7f57d0152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00a5ef0 0x7f57d01eedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00c2890 con 0x7f57d00a5ef0 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00c2710 con 0x7f57d006e340 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00c5060 con 0x7f57d006da20 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00a5ef0 0x7f57d01eedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00a5ef0 0x7f57d01eedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34964/0 (socket says 172.21.15.27:34964) 2024-03-28T19:06:21.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d5400640 1 -- 172.21.15.27:0/3217392576 learned_addr learned my addr 172.21.15.27:0/3217392576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57cfe00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57d006e340 0x7f57d0152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.142+0000 7f57d4a00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f57d006da20 0x7f57d00c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.143+0000 7f57d5400640 1 -- 172.21.15.27:0/3217392576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f57d006da20 msgr2=0x7f57d00c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.143+0000 7f57d5400640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f57d006da20 0x7f57d00c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.143+0000 7f57d5400640 1 -- 172.21.15.27:0/3217392576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57d006e340 msgr2=0x7f57d0152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.143+0000 7f57d5400640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57d006e340 0x7f57d0152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.143+0000 7f57d5400640 1 -- 172.21.15.27:0/3217392576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d0140e00 con 0x7f57d00a5ef0 2024-03-28T19:06:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.143+0000 7f57d5400640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00a5ef0 0x7f57d01eedc0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f57c800c6b0 tx=0x7f57c800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.143+0000 7f57ccc00640 1 -- 172.21.15.27:0/3217392576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c8003b70 con 0x7f57d00a5ef0 2024-03-28T19:06:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.144+0000 7f57ccc00640 1 -- 172.21.15.27:0/3217392576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c8003d30 con 0x7f57d00a5ef0 2024-03-28T19:06:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.144+0000 7f57ccc00640 1 -- 172.21.15.27:0/3217392576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c8005310 con 0x7f57d00a5ef0 2024-03-28T19:06:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.144+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d00c5530 con 0x7f57d00a5ef0 2024-03-28T19:06:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.144+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d00c5af0 con 0x7f57d00a5ef0 2024-03-28T19:06:21.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.145+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d0067dd0 con 0x7f57d00a5ef0 2024-03-28T19:06:21.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.146+0000 7f57ccc00640 1 -- 172.21.15.27:0/3217392576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f57c80054b0 con 0x7f57d00a5ef0 2024-03-28T19:06:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.147+0000 7f57ccc00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f57b006bc00 0x7f57b006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.147+0000 7f57ccc00640 1 -- 172.21.15.27:0/3217392576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f57c808b150 con 0x7f57d00a5ef0 2024-03-28T19:06:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.147+0000 7f57d4a00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f57b006bc00 0x7f57b006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.148+0000 7f57d4a00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f57b006bc00 0x7f57b006e0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f57d0141f00 tx=0x7f57c4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.157+0000 7f57ccc00640 1 -- 172.21.15.27:0/3217392576 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f57c80593d0 con 0x7f57d00a5ef0 2024-03-28T19:06:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.429+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57d00a7d60 con 0x7f57d00a5ef0 2024-03-28T19:06:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.430+0000 7f57ccc00640 1 -- 172.21.15.27:0/3217392576 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5468 (secure 0 0 0) 0x7f57c8058cd0 con 0x7f57d00a5ef0 2024-03-28T19:06:21.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:21.430 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 91 2024-03-28T19:06:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.433+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f57b006bc00 msgr2=0x7f57b006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.433+0000 7f57d5e00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f57b006bc00 0x7f57b006e0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f57d0141f00 tx=0x7f57c4008040 comp rx=0 tx=0).stop 2024-03-28T19:06:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.433+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00a5ef0 msgr2=0x7f57d01eedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.433+0000 7f57d5e00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00a5ef0 0x7f57d01eedc0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f57c800c6b0 tx=0x7f57c800cb70 comp rx=0 tx=0).stop 2024-03-28T19:06:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.434+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 shutdown_connections 2024-03-28T19:06:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.434+0000 7f57d5e00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f57b006bc00 0x7f57b006e0c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.434+0000 7f57d5e00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57d00a5ef0 0x7f57d01eedc0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.434+0000 7f57d5e00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57d006e340 0x7f57d0152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.434+0000 7f57d5e00640 1 --2- 172.21.15.27:0/3217392576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f57d006da20 0x7f57d00c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.434+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 >> 172.21.15.27:0/3217392576 conn(0x7f57d006b0c0 msgr2=0x7f57d00aae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.434+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 shutdown_connections 2024-03-28T19:06:21.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:21.434+0000 7f57d5e00640 1 -- 172.21.15.27:0/3217392576 wait complete. 2024-03-28T19:06:21.452 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=6192 2024-03-28T19:06:21.452 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.a 2024-03-28T19:06:22.661 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:06:22.662+0000 7fddc6a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:06:25.453 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-03-28T19:06:25.453 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:06:25.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:06:25.461 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:06:25.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:25.658 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:06:25.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.682+0000 7f9ec2000640 1 Processor -- start 2024-03-28T19:06:25.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.682+0000 7f9ec2000640 1 -- start start 2024-03-28T19:06:25.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.683+0000 7f9ec2000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9ebc0c69a0 0x7f9ebc0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.683+0000 7f9ec2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc153a30 0x7f9ebc153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.683+0000 7f9ec2000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ebc150ec0 con 0x7f9ebc150ae0 2024-03-28T19:06:25.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.683+0000 7f9ec2000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ebc06dc70 con 0x7f9ebc0c69a0 2024-03-28T19:06:25.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.683+0000 7f9ec2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc153550 con 0x7f9ebc153a30 2024-03-28T19:06:25.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.684+0000 7f9ec0c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9ebc150ae0 0x7f9ebc0a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51924/0 (socket says 172.21.15.27:51924) 2024-03-28T19:06:25.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.684+0000 7f9ec0c00640 1 -- 172.21.15.27:0/2035157198 learned_addr learned my addr 172.21.15.27:0/2035157198 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:06:25.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.684+0000 7f9ec1600640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc153a30 0x7f9ebc153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.684+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/2035157198 >> v2:172.21.15.155:3300/0 conn(0x7f9ebc0c69a0 0x7f9ebc0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.684+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385084717 0 0) 0x7f9eb0001110 con 0x7f9ebc150ae0 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.684+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea4003520 con 0x7f9ebc150ae0 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1829832420 0 0) 0x7f9ea4003520 con 0x7f9ebc150ae0 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc153a30 msgr2=0x7f9ebc153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc153a30 0x7f9ebc153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 >> v2:172.21.15.155:3300/0 conn(0x7f9ebc0c69a0 msgr2=0x7f9ebc0c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 --2- 172.21.15.27:0/2035157198 >> v2:172.21.15.155:3300/0 conn(0x7f9ebc0c69a0 0x7f9ebc0c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc06df10 con 0x7f9ebc150ae0 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9eb0001840 con 0x7f9ebc150ae0 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 >> v1:172.21.15.27:6789/0 conn(0x7f9ebc150ae0 legacy=0x7f9ebc0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea40039b0 0x7f9ea4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc153a30 0x7f9ea4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea4004d40 0x7f9ea40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc06dc70 con 0x7f9ea40039b0 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc153550 con 0x7f9ea4004d40 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea400b9f0 con 0x7f9ebc153a30 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/2035157198 >> v2:172.21.15.155:3300/0 conn(0x7f9ebc0c69a0 0x7f9ebc0c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.685+0000 7f9ec0c00640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea40039b0 0x7f9ea4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.686+0000 7f9ec1600640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea4004d40 0x7f9ea40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.686+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc153a30 0x7f9ea4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.686+0000 7f9ec0c00640 1 -- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc153a30 msgr2=0x7f9ea4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.686+0000 7f9ec0c00640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc153a30 0x7f9ea4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.686+0000 7f9ec0c00640 1 -- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea4004d40 msgr2=0x7f9ea40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.686+0000 7f9ec0c00640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea4004d40 0x7f9ea40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.686+0000 7f9ec0c00640 1 -- 172.21.15.27:0/2035157198 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea4003520 con 0x7f9ea40039b0 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.688+0000 7f9ec0c00640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea40039b0 0x7f9ea4003dd0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000c320 tx=0x7f9ea4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728bd9492c9ee7d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.688+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb002cc40 con 0x7f9ea40039b0 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.688+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0030070 con 0x7f9ea40039b0 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.688+0000 7f9ebb400640 1 -- 172.21.15.27:0/2035157198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00058d0 con 0x7f9ea40039b0 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.689+0000 7f9ec2000640 1 -- 172.21.15.27:0/2035157198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea40039b0 msgr2=0x7f9ea4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.689+0000 7f9ec2000640 1 --2- 172.21.15.27:0/2035157198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ea40039b0 0x7f9ea4003dd0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000c320 tx=0x7f9ea4005830 comp rx=0 tx=0).stop 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.689+0000 7f9ec1600640 1 -- 172.21.15.27:0/2035157198 reap_dead start 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.689+0000 7f9ec2000640 1 -- 172.21.15.27:0/2035157198 shutdown_connections 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.689+0000 7f9ec2000640 1 -- 172.21.15.27:0/2035157198 >> 172.21.15.27:0/2035157198 conn(0x7f9ebc06b0c0 msgr2=0x7f9ebc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.689+0000 7f9ec2000640 1 -- 172.21.15.27:0/2035157198 shutdown_connections 2024-03-28T19:06:25.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.689+0000 7f9ec2000640 1 -- 172.21.15.27:0/2035157198 wait complete. 2024-03-28T19:06:25.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.691+0000 7f9ec2000640 1 Processor -- start 2024-03-28T19:06:25.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.691+0000 7f9ec2000640 1 -- start start 2024-03-28T19:06:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ec2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea40039b0 0x7f9ebc0a7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ec2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc14dc90 0x7f9ebc14d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ec2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc072170 0x7f9ebc0c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ec2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc06e7b0 con 0x7f9ebc14dc90 2024-03-28T19:06:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ec2000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc06e4d0 con 0x7f9ea40039b0 2024-03-28T19:06:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ec2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc0a6370 con 0x7f9ebc072170 2024-03-28T19:06:25.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ebbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc14dc90 0x7f9ebc14d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ebbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc14dc90 0x7f9ebc14d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35012/0 (socket says 172.21.15.27:35012) 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ebbe00640 1 -- 172.21.15.27:0/1624094237 learned_addr learned my addr 172.21.15.27:0/1624094237 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.692+0000 7f9ec0c00640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea40039b0 0x7f9ebc0a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9ec1600640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc072170 0x7f9ebc0c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9ebbe00640 1 -- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc072170 msgr2=0x7f9ebc0c45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc072170 0x7f9ebc0c45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9ebbe00640 1 -- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea40039b0 msgr2=0x7f9ebc0a7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea40039b0 0x7f9ebc0a7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9ebbe00640 1 -- 172.21.15.27:0/1624094237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc0a70c0 con 0x7f9ebc14dc90 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9ec1600640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc072170 0x7f9ebc0c45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc14dc90 0x7f9ebc14d240 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8011960 tx=0x7f9ea8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:25.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9eb8c00640 1 -- 172.21.15.27:0/1624094237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea800fbd0 con 0x7f9ebc14dc90 2024-03-28T19:06:25.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9eb8c00640 1 -- 172.21.15.27:0/1624094237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea800fd70 con 0x7f9ebc14dc90 2024-03-28T19:06:25.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.693+0000 7f9eb8c00640 1 -- 172.21.15.27:0/1624094237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea8015660 con 0x7f9ebc14dc90 2024-03-28T19:06:25.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.694+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ebc073090 con 0x7f9ebc14dc90 2024-03-28T19:06:25.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.694+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ebc14e0f0 con 0x7f9ebc14dc90 2024-03-28T19:06:25.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.695+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ebc067dd0 con 0x7f9ebc14dc90 2024-03-28T19:06:25.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.696+0000 7f9eb8c00640 1 -- 172.21.15.27:0/1624094237 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ea8015800 con 0x7f9ebc14dc90 2024-03-28T19:06:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.697+0000 7f9eb8c00640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e9c06bc00 0x7f9e9c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.697+0000 7f9eb8c00640 1 -- 172.21.15.27:0/1624094237 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f9ea808b880 con 0x7f9ebc14dc90 2024-03-28T19:06:25.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.697+0000 7f9ec0c00640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e9c06bc00 0x7f9e9c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:25.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.698+0000 7f9ec0c00640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e9c06bc00 0x7f9e9c06e0c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0034000 tx=0x7f9eb0007000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:25.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.702+0000 7f9eb8c00640 1 -- 172.21.15.27:0/1624094237 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9ea8019030 con 0x7f9ebc14dc90 2024-03-28T19:06:25.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.961+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ebc069640 con 0x7f9ebc14dc90 2024-03-28T19:06:25.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.961+0000 7f9eb8c00640 1 -- 172.21.15.27:0/1624094237 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5468 (secure 0 0 0) 0x7f9ea8059b00 con 0x7f9ebc14dc90 2024-03-28T19:06:25.961 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:25.961 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:25.961 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 91 2024-03-28T19:06:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e9c06bc00 msgr2=0x7f9e9c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:25.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e9c06bc00 0x7f9e9c06e0c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0034000 tx=0x7f9eb0007000 comp rx=0 tx=0).stop 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc14dc90 msgr2=0x7f9ebc14d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc14dc90 0x7f9ebc14d240 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8011960 tx=0x7f9ea8011e20 comp rx=0 tx=0).stop 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 shutdown_connections 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e9c06bc00 0x7f9e9c06e0c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ebc072170 0x7f9ebc0c45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc14dc90 0x7f9ebc14d240 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 --2- 172.21.15.27:0/1624094237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ea40039b0 0x7f9ebc0a7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 >> 172.21.15.27:0/1624094237 conn(0x7f9ebc06b0c0 msgr2=0x7f9ebc0c0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 shutdown_connections 2024-03-28T19:06:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:25.964+0000 7f9ec2000640 1 -- 172.21.15.27:0/1624094237 wait complete. 2024-03-28T19:06:25.983 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:06:27.985 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:28.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.218+0000 7f42f1a00640 1 Processor -- start 2024-03-28T19:06:28.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.219+0000 7f42f1a00640 1 -- start start 2024-03-28T19:06:28.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.220+0000 7f42f1a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42ec06c0d0 0x7f42ec06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:28.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.220+0000 7f42f1a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f42ec0c1780 0x7f42ec155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:28.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.220+0000 7f42f1a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec0a7400 con 0x7f42ec0a6e90 2024-03-28T19:06:28.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.220+0000 7f42f1a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ec159470 con 0x7f42ec06c0d0 2024-03-28T19:06:28.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.220+0000 7f42f1a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ec158b40 con 0x7f42ec0c1780 2024-03-28T19:06:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.220+0000 7f42ebe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42ec06c0d0 0x7f42ec06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.220+0000 7f42eb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f42ec0a6e90 0x7f42ec155530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39838/0 (socket says 172.21.15.27:39838) 2024-03-28T19:06:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.220+0000 7f42eb400640 1 -- 172.21.15.27:0/823523986 learned_addr learned my addr 172.21.15.27:0/823523986 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:06:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.221+0000 7f42f1000640 1 --2- 172.21.15.27:0/823523986 >> v2:172.21.15.155:3300/0 conn(0x7f42ec0c1780 0x7f42ec155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.221+0000 7f42ebe00640 1 -- 172.21.15.27:0/823523986 >> v2:172.21.15.155:3300/0 conn(0x7f42ec0c1780 msgr2=0x7f42ec155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.221+0000 7f42ebe00640 1 --2- 172.21.15.27:0/823523986 >> v2:172.21.15.155:3300/0 conn(0x7f42ec0c1780 0x7f42ec155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.221+0000 7f42ebe00640 1 -- 172.21.15.27:0/823523986 >> v1:172.21.15.27:6789/0 conn(0x7f42ec0a6e90 legacy=0x7f42ec155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:28.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.221+0000 7f42ebe00640 1 -- 172.21.15.27:0/823523986 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec158e20 con 0x7f42ec06c0d0 2024-03-28T19:06:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.222+0000 7f42ebe00640 1 --2- 172.21.15.27:0/823523986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec06c0d0 0x7f42ec06c4b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0c43e0 tx=0x7f42dc02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475f920b095d27fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.222+0000 7f42eaa00640 1 -- 172.21.15.27:0/823523986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc039070 con 0x7f42ec06c0d0 2024-03-28T19:06:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.222+0000 7f42eaa00640 1 -- 172.21.15.27:0/823523986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc02cbe0 con 0x7f42ec06c0d0 2024-03-28T19:06:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.222+0000 7f42eaa00640 1 -- 172.21.15.27:0/823523986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc034440 con 0x7f42ec06c0d0 2024-03-28T19:06:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.223+0000 7f42f1a00640 1 -- 172.21.15.27:0/823523986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec06c0d0 msgr2=0x7f42ec06c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.223+0000 7f42f1a00640 1 --2- 172.21.15.27:0/823523986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec06c0d0 0x7f42ec06c4b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0c43e0 tx=0x7f42dc02be30 comp rx=0 tx=0).stop 2024-03-28T19:06:28.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.223+0000 7f42f1a00640 1 -- 172.21.15.27:0/823523986 shutdown_connections 2024-03-28T19:06:28.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.223+0000 7f42f1a00640 1 --2- 172.21.15.27:0/823523986 >> v2:172.21.15.155:3300/0 conn(0x7f42ec0c1780 0x7f42ec155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.223+0000 7f42f1a00640 1 --2- 172.21.15.27:0/823523986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec06c0d0 0x7f42ec06c4b0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.223+0000 7f42f1a00640 1 -- 172.21.15.27:0/823523986 >> 172.21.15.27:0/823523986 conn(0x7f42ec0c7aa0 msgr2=0x7f42ec0c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:28.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.223+0000 7f42f1a00640 1 -- 172.21.15.27:0/823523986 shutdown_connections 2024-03-28T19:06:28.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.223+0000 7f42f1a00640 1 -- 172.21.15.27:0/823523986 wait complete. 2024-03-28T19:06:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.225+0000 7f42f1a00640 1 Processor -- start 2024-03-28T19:06:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.225+0000 7f42f1a00640 1 -- start start 2024-03-28T19:06:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42ec06c0d0 0x7f42ec1585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42ec0a6e90 0x7f42ec1ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec0c1780 0x7f42ec0a8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec152e20 con 0x7f42ec0c1780 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec159900 con 0x7f42ec0a6e90 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec152fa0 con 0x7f42ec06c0d0 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec0c1780 0x7f42ec0a8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec0c1780 0x7f42ec0a8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51426/0 (socket says 172.21.15.27:51426) 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42f1000640 1 -- 172.21.15.27:0/2510349953 learned_addr learned my addr 172.21.15.27:0/2510349953 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.226+0000 7f42ebe00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42ec06c0d0 0x7f42ec1585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.227+0000 7f42eb400640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42ec0a6e90 0x7f42ec1ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.227+0000 7f42f1000640 1 -- 172.21.15.27:0/2510349953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42ec06c0d0 msgr2=0x7f42ec1585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:28.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.227+0000 7f42f1000640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42ec06c0d0 0x7f42ec1585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.227+0000 7f42f1000640 1 -- 172.21.15.27:0/2510349953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42ec0a6e90 msgr2=0x7f42ec1ef000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.227+0000 7f42f1000640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42ec0a6e90 0x7f42ec1ef000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.227+0000 7f42f1000640 1 -- 172.21.15.27:0/2510349953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec06f350 con 0x7f42ec0c1780 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.227+0000 7f42f1000640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec0c1780 0x7f42ec0a8890 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f42e400d5d0 tx=0x7f42e400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.227+0000 7f42e3e00640 1 -- 172.21.15.27:0/2510349953 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e40125f0 con 0x7f42ec0c1780 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.228+0000 7f42e3e00640 1 -- 172.21.15.27:0/2510349953 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e4019070 con 0x7f42ec0c1780 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.228+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec06f630 con 0x7f42ec0c1780 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.228+0000 7f42e3e00640 1 -- 172.21.15.27:0/2510349953 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e40129f0 con 0x7f42ec0c1780 2024-03-28T19:06:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.228+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec1ee800 con 0x7f42ec0c1780 2024-03-28T19:06:28.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.229+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ec069640 con 0x7f42ec0c1780 2024-03-28T19:06:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.230+0000 7f42e3e00640 1 -- 172.21.15.27:0/2510349953 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f42e4012b90 con 0x7f42ec0c1780 2024-03-28T19:06:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.231+0000 7f42e3e00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42c806bbd0 0x7f42c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.231+0000 7f42e3e00640 1 -- 172.21.15.27:0/2510349953 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f42e408b650 con 0x7f42ec0c1780 2024-03-28T19:06:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.231+0000 7f42ebe00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42c806bbd0 0x7f42c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.232+0000 7f42ebe00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42c806bbd0 0x7f42c806e090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f42dc036000 tx=0x7f42dc030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:28.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.236+0000 7f42e3e00640 1 -- 172.21.15.27:0/2510349953 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f42e4059850 con 0x7f42ec0c1780 2024-03-28T19:06:28.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.494+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42ec1561e0 con 0x7f42ec0c1780 2024-03-28T19:06:28.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.495+0000 7f42e3e00640 1 -- 172.21.15.27:0/2510349953 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f42e4059150 con 0x7f42ec0c1780 2024-03-28T19:06:28.495 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:28.495 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:28.495 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-28T19:06:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42c806bbd0 msgr2=0x7f42c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42c806bbd0 0x7f42c806e090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f42dc036000 tx=0x7f42dc030000 comp rx=0 tx=0).stop 2024-03-28T19:06:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec0c1780 msgr2=0x7f42ec0a8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec0c1780 0x7f42ec0a8890 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f42e400d5d0 tx=0x7f42e400da90 comp rx=0 tx=0).stop 2024-03-28T19:06:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 shutdown_connections 2024-03-28T19:06:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f42c806bbd0 0x7f42c806e090 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ec0c1780 0x7f42ec0a8890 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42ec0a6e90 0x7f42ec1ef000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 --2- 172.21.15.27:0/2510349953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f42ec06c0d0 0x7f42ec1585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 >> 172.21.15.27:0/2510349953 conn(0x7f42ec0c7aa0 msgr2=0x7f42ec066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 shutdown_connections 2024-03-28T19:06:28.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:28.498+0000 7f42f1a00640 1 -- 172.21.15.27:0/2510349953 wait complete. 2024-03-28T19:06:28.519 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-03-28T19:06:28.519 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2024-03-28T19:06:30.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:06:30.442 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:06:50.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.985+0000 7f027e400640 1 Processor -- start 2024-03-28T19:06:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.985+0000 7f027e400640 1 -- start start 2024-03-28T19:06:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f027e400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f02780c1f50 0x7f02780ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f027e400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02780abd90 0x7f02780aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f027e400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02780ac1c0 con 0x7f02780c5e80 2024-03-28T19:06:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f027e400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027806d650 con 0x7f02780abd90 2024-03-28T19:06:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f027e400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027806e1d0 con 0x7f02780c1f50 2024-03-28T19:06:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f0277400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f02780c5e80 0x7f02780c2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46336/0 (socket says 172.21.15.27:46336) 2024-03-28T19:06:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f0277400640 1 -- 172.21.15.27:0/1055853732 learned_addr learned my addr 172.21.15.27:0/1055853732 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:06:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f0277e00640 1 --2- 172.21.15.27:0/1055853732 >> v2:172.21.15.27:3300/0 conn(0x7f02780abd90 0x7f02780aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.987+0000 7f0276a00640 1 --2- 172.21.15.27:0/1055853732 >> v2:172.21.15.155:3300/0 conn(0x7f02780c1f50 0x7f02780ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.988+0000 7f0277e00640 1 -- 172.21.15.27:0/1055853732 >> v2:172.21.15.155:3300/0 conn(0x7f02780c1f50 msgr2=0x7f02780ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.988+0000 7f0277e00640 1 --2- 172.21.15.27:0/1055853732 >> v2:172.21.15.155:3300/0 conn(0x7f02780c1f50 0x7f02780ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.988+0000 7f0277e00640 1 -- 172.21.15.27:0/1055853732 >> v1:172.21.15.27:6789/0 conn(0x7f02780c5e80 legacy=0x7f02780c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.988+0000 7f0277e00640 1 -- 172.21.15.27:0/1055853732 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027806d8a0 con 0x7f02780abd90 2024-03-28T19:06:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.990+0000 7f0277e00640 1 --2- 172.21.15.27:0/1055853732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780abd90 0x7f02780aa570 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0278073150 tx=0x7f027002bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b00d6239bd3644b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.990+0000 7f0276000640 1 -- 172.21.15.27:0/1055853732 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270039070 con 0x7f02780abd90 2024-03-28T19:06:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.990+0000 7f0276000640 1 -- 172.21.15.27:0/1055853732 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027002cb50 con 0x7f02780abd90 2024-03-28T19:06:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.990+0000 7f0276000640 1 -- 172.21.15.27:0/1055853732 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270034440 con 0x7f02780abd90 2024-03-28T19:06:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.990+0000 7f027e400640 1 -- 172.21.15.27:0/1055853732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780abd90 msgr2=0x7f02780aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:50.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.990+0000 7f027e400640 1 --2- 172.21.15.27:0/1055853732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780abd90 0x7f02780aa570 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0278073150 tx=0x7f027002bda0 comp rx=0 tx=0).stop 2024-03-28T19:06:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.991+0000 7f027e400640 1 -- 172.21.15.27:0/1055853732 shutdown_connections 2024-03-28T19:06:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.991+0000 7f027e400640 1 --2- 172.21.15.27:0/1055853732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780abd90 0x7f02780aa570 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.991+0000 7f027e400640 1 --2- 172.21.15.27:0/1055853732 >> v2:172.21.15.155:3300/0 conn(0x7f02780c1f50 0x7f02780ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.991+0000 7f027e400640 1 -- 172.21.15.27:0/1055853732 >> 172.21.15.27:0/1055853732 conn(0x7f027806b9d0 msgr2=0x7f02780c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.991+0000 7f027e400640 1 -- 172.21.15.27:0/1055853732 shutdown_connections 2024-03-28T19:06:50.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.991+0000 7f027e400640 1 -- 172.21.15.27:0/1055853732 wait complete. 2024-03-28T19:06:50.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.993+0000 7f027e400640 1 Processor -- start 2024-03-28T19:06:50.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.993+0000 7f027e400640 1 -- start start 2024-03-28T19:06:50.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.994+0000 7f027e400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02780abd90 0x7f027814fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.994+0000 7f027e400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02780c1f50 0x7f027814f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.994+0000 7f027e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780c5e80 0x7f02780c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.994+0000 7f027e400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02780c6260 con 0x7f02780c5e80 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.994+0000 7f027e400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027806de60 con 0x7f02780c1f50 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.994+0000 7f027e400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02780c63e0 con 0x7f02780abd90 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780c5e80 0x7f02780c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780c5e80 0x7f02780c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53474/0 (socket says 172.21.15.27:53474) 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277e00640 1 -- 172.21.15.27:0/3967427993 learned_addr learned my addr 172.21.15.27:0/3967427993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0276a00640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02780c1f50 0x7f027814f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02780abd90 0x7f027814fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277e00640 1 -- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02780abd90 msgr2=0x7f027814fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:50.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277e00640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02780abd90 0x7f027814fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:50.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277e00640 1 -- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02780c1f50 msgr2=0x7f027814f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:50.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277e00640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02780c1f50 0x7f027814f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:50.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277e00640 1 -- 172.21.15.27:0/3967427993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02780a8180 con 0x7f02780c5e80 2024-03-28T19:06:50.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.995+0000 7f0277400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02780abd90 0x7f027814fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:50.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.996+0000 7f0277e00640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780c5e80 0x7f02780c38b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f027002c3b0 tx=0x7f0270004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:50.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.996+0000 7f026f400640 1 -- 172.21.15.27:0/3967427993 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270039050 con 0x7f02780c5e80 2024-03-28T19:06:50.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.996+0000 7f026f400640 1 -- 172.21.15.27:0/3967427993 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0270045070 con 0x7f02780c5e80 2024-03-28T19:06:50.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.996+0000 7f026f400640 1 -- 172.21.15.27:0/3967427993 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027003e830 con 0x7f02780c5e80 2024-03-28T19:06:50.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.996+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02780a8460 con 0x7f02780c5e80 2024-03-28T19:06:50.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.996+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02780c3df0 con 0x7f02780c5e80 2024-03-28T19:06:50.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:50.998+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027804f400 con 0x7f02780c5e80 2024-03-28T19:06:50.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.000+0000 7f026f400640 1 -- 172.21.15.27:0/3967427993 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f027003e9d0 con 0x7f02780c5e80 2024-03-28T19:06:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.001+0000 7f026f400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025806bbd0 0x7f025806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.001+0000 7f026f400640 1 -- 172.21.15.27:0/3967427993 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f02700adf80 con 0x7f02780c5e80 2024-03-28T19:06:51.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.001+0000 7f0277400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025806bbd0 0x7f025806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:51.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.002+0000 7f0277400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025806bbd0 0x7f025806e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f027814ced0 tx=0x7f0268009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.006+0000 7f026f400640 1 -- 172.21.15.27:0/3967427993 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0270032e20 con 0x7f02780c5e80 2024-03-28T19:06:51.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.315+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0278155820 con 0x7f02780c5e80 2024-03-28T19:06:51.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.316+0000 7f026f400640 1 -- 172.21.15.27:0/3967427993 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f027007c2a0 con 0x7f02780c5e80 2024-03-28T19:06:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:51.342 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:51.344 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-28T19:06:51.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025806bbd0 msgr2=0x7f025806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:51.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025806bbd0 0x7f025806e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f027814ced0 tx=0x7f0268009040 comp rx=0 tx=0).stop 2024-03-28T19:06:51.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780c5e80 msgr2=0x7f02780c38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:51.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780c5e80 0x7f02780c38b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f027002c3b0 tx=0x7f0270004a40 comp rx=0 tx=0).stop 2024-03-28T19:06:51.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 shutdown_connections 2024-03-28T19:06:51.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f025806bbd0 0x7f025806e090 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02780c5e80 0x7f02780c38b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02780c1f50 0x7f027814f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 --2- 172.21.15.27:0/3967427993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02780abd90 0x7f027814fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.319+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 >> 172.21.15.27:0/3967427993 conn(0x7f027806b9d0 msgr2=0x7f02780c08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:51.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.320+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 shutdown_connections 2024-03-28T19:06:51.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.320+0000 7f027e400640 1 -- 172.21.15.27:0/3967427993 wait complete. 2024-03-28T19:06:51.345 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-03-28T19:06:51.346 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:06:51.346 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:06:51.380 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:06:51.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:51.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.643+0000 7f6b0da00640 1 Processor -- start 2024-03-28T19:06:51.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.643+0000 7f6b0da00640 1 -- start start 2024-03-28T19:06:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6b0da00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6b08152a70 0x7f6b08159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6b0da00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b080cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6b0da00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080c5710 con 0x7f6b080cb620 2024-03-28T19:06:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6b0da00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b081556b0 con 0x7f6b08152a70 2024-03-28T19:06:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6b0da00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080c59f0 con 0x7f6b080c6cc0 2024-03-28T19:06:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6b0d000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6b080cb620 0x7f6b08155b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46352/0 (socket says 172.21.15.27:46352) 2024-03-28T19:06:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6b0d000640 1 -- 172.21.15.27:0/2709723901 learned_addr learned my addr 172.21.15.27:0/2709723901 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:06:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6affe00640 1 --2- 172.21.15.27:0/2709723901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b080cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.645+0000 7f6b07e00640 1 --2- 172.21.15.27:0/2709723901 >> v2:172.21.15.155:3300/0 conn(0x7f6b08152a70 0x7f6b08159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.646+0000 7f6b07400640 1 -- 172.21.15.27:0/2709723901 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853736048 0 0) 0x7f6b080c5710 con 0x7f6b080cb620 2024-03-28T19:06:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.646+0000 7f6b07400640 1 -- 172.21.15.27:0/2709723901 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aec003520 con 0x7f6b080cb620 2024-03-28T19:06:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.646+0000 7f6affe00640 1 -- 172.21.15.27:0/2709723901 >> v2:172.21.15.155:3300/0 conn(0x7f6b08152a70 msgr2=0x7f6b08159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.646+0000 7f6affe00640 1 --2- 172.21.15.27:0/2709723901 >> v2:172.21.15.155:3300/0 conn(0x7f6b08152a70 0x7f6b08159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.646+0000 7f6affe00640 1 -- 172.21.15.27:0/2709723901 >> v1:172.21.15.27:6789/0 conn(0x7f6b080cb620 legacy=0x7f6b08155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.646+0000 7f6affe00640 1 -- 172.21.15.27:0/2709723901 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b080c5cd0 con 0x7f6b080c6cc0 2024-03-28T19:06:51.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.648+0000 7f6affe00640 1 --2- 172.21.15.27:0/2709723901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b080cb0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097a0 tx=0x7f6af002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7b9449449a94e9b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:51.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b07400640 1 -- 172.21.15.27:0/2709723901 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0031780 con 0x7f6b080c6cc0 2024-03-28T19:06:51.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b07400640 1 -- 172.21.15.27:0/2709723901 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af0037070 con 0x7f6b080c6cc0 2024-03-28T19:06:51.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b07400640 1 -- 172.21.15.27:0/2709723901 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0031b40 con 0x7f6b080c6cc0 2024-03-28T19:06:51.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b0da00640 1 -- 172.21.15.27:0/2709723901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 msgr2=0x7f6b080cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:51.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2709723901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b080cb0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097a0 tx=0x7f6af002b820 comp rx=0 tx=0).stop 2024-03-28T19:06:51.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b0da00640 1 -- 172.21.15.27:0/2709723901 shutdown_connections 2024-03-28T19:06:51.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2709723901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b080cb0e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2709723901 >> v2:172.21.15.155:3300/0 conn(0x7f6b08152a70 0x7f6b08159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.649+0000 7f6b0da00640 1 -- 172.21.15.27:0/2709723901 >> 172.21.15.27:0/2709723901 conn(0x7f6b0806b2a0 msgr2=0x7f6b0806b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:51.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.650+0000 7f6b0da00640 1 -- 172.21.15.27:0/2709723901 shutdown_connections 2024-03-28T19:06:51.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.650+0000 7f6b0da00640 1 -- 172.21.15.27:0/2709723901 wait complete. 2024-03-28T19:06:51.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.652+0000 7f6b0da00640 1 Processor -- start 2024-03-28T19:06:51.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.652+0000 7f6b0da00640 1 -- start start 2024-03-28T19:06:51.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6b0da00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b08153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:51.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6b0da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080cb620 0x7f6b08156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:51.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6b0da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b08152a70 0x7f6b08156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:51.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6b0da00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080c8f50 con 0x7f6b080cb620 2024-03-28T19:06:51.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6b0da00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080c8dd0 con 0x7f6b08152a70 2024-03-28T19:06:51.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6b0da00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b080c5fb0 con 0x7f6b080c6cc0 2024-03-28T19:06:51.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6affe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080cb620 0x7f6b08156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:51.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6affe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080cb620 0x7f6b08156a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53476/0 (socket says 172.21.15.27:53476) 2024-03-28T19:06:51.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.653+0000 7f6affe00640 1 -- 172.21.15.27:0/2164387263 learned_addr learned my addr 172.21.15.27:0/2164387263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:51.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.654+0000 7f6b0d000640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b08152a70 0x7f6b08156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:51.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.654+0000 7f6b07e00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b08153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:51.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.654+0000 7f6affe00640 1 -- 172.21.15.27:0/2164387263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 msgr2=0x7f6b08153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:51.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.654+0000 7f6affe00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b08153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.654+0000 7f6affe00640 1 -- 172.21.15.27:0/2164387263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b08152a70 msgr2=0x7f6b08156fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.654+0000 7f6affe00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b08152a70 0x7f6b08156fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.654+0000 7f6affe00640 1 -- 172.21.15.27:0/2164387263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08159270 con 0x7f6b080cb620 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.654+0000 7f6affe00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080cb620 0x7f6b08156a70 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6af0034040 tx=0x7f6af00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.655+0000 7f6b04c00640 1 -- 172.21.15.27:0/2164387263 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0032610 con 0x7f6b080cb620 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.655+0000 7f6b04c00640 1 -- 172.21.15.27:0/2164387263 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af00327d0 con 0x7f6b080cb620 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.655+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b081ebfa0 con 0x7f6b080cb620 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.655+0000 7f6b04c00640 1 -- 172.21.15.27:0/2164387263 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0032bc0 con 0x7f6b080cb620 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.655+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b081540b0 con 0x7f6b080cb620 2024-03-28T19:06:51.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.656+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b08067dd0 con 0x7f6b080cb620 2024-03-28T19:06:51.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.657+0000 7f6b04c00640 1 -- 172.21.15.27:0/2164387263 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6af004e440 con 0x7f6b080cb620 2024-03-28T19:06:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.658+0000 7f6b04c00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6ae406bc00 0x7f6ae406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:51.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.658+0000 7f6b04c00640 1 -- 172.21.15.27:0/2164387263 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f6af00acf30 con 0x7f6b080cb620 2024-03-28T19:06:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.659+0000 7f6b07e00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6ae406bc00 0x7f6ae406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:51.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.659+0000 7f6b07e00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6ae406bc00 0x7f6ae406e0c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6b0814b8e0 tx=0x7f6af8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:51.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.664+0000 7f6b04c00640 1 -- 172.21.15.27:0/2164387263 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6af0040070 con 0x7f6b080cb620 2024-03-28T19:06:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.934+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b08069640 con 0x7f6b080cb620 2024-03-28T19:06:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.935+0000 7f6b04c00640 1 -- 172.21.15.27:0/2164387263 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f6af002f050 con 0x7f6b080cb620 2024-03-28T19:06:51.935 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:51.935 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:51.935 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-28T19:06:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.938+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6ae406bc00 msgr2=0x7f6ae406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.938+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6ae406bc00 0x7f6ae406e0c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6b0814b8e0 tx=0x7f6af8008040 comp rx=0 tx=0).stop 2024-03-28T19:06:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.938+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080cb620 msgr2=0x7f6b08156a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.938+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080cb620 0x7f6b08156a70 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6af0034040 tx=0x7f6af00046c0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.939+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 shutdown_connections 2024-03-28T19:06:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.939+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6ae406bc00 0x7f6ae406e0c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.939+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b08152a70 0x7f6b08156fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.939+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080cb620 0x7f6b08156a70 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.939+0000 7f6b0da00640 1 --2- 172.21.15.27:0/2164387263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b080c6cc0 0x7f6b08153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.939+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 >> 172.21.15.27:0/2164387263 conn(0x7f6b0806b2a0 msgr2=0x7f6b0814cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.939+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 shutdown_connections 2024-03-28T19:06:51.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:51.939+0000 7f6b0da00640 1 -- 172.21.15.27:0/2164387263 wait complete. 2024-03-28T19:06:51.956 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:06:53.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.200+0000 7f4c9fe00640 1 Processor -- start 2024-03-28T19:06:54.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.200+0000 7f4c9fe00640 1 -- start start 2024-03-28T19:06:54.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.202+0000 7f4c9fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ca00c6a90 0x7f4ca006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:54.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.202+0000 7f4c9fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca006d080 con 0x7f4ca00c6e70 2024-03-28T19:06:54.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.202+0000 7f4c9fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca006cb00 con 0x7f4ca00c72f0 2024-03-28T19:06:54.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.202+0000 7f4c9fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca0159980 con 0x7f4ca00c6a90 2024-03-28T19:06:54.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.202+0000 7f4c9ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ca00c6a90 0x7f4ca006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:54.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.202+0000 7f4c9ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ca00c6a90 0x7f4ca006b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53478/0 (socket says 172.21.15.27:53478) 2024-03-28T19:06:54.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.202+0000 7f4c9ea00640 1 -- 172.21.15.27:0/2327907723 learned_addr learned my addr 172.21.15.27:0/2327907723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:54.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.203+0000 7f4c9ea00640 1 -- 172.21.15.27:0/2327907723 >> v1:172.21.15.155:6789/0 conn(0x7f4ca00c72f0 legacy=0x7f4ca01572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:54.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.203+0000 7f4c9ea00640 1 -- 172.21.15.27:0/2327907723 >> v1:172.21.15.27:6789/0 conn(0x7f4ca00c6e70 legacy=0x7f4ca0156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:54.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.203+0000 7f4c9ea00640 1 -- 172.21.15.27:0/2327907723 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca00c7780 con 0x7f4ca00c6a90 2024-03-28T19:06:54.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.204+0000 7f4c9ea00640 1 --2- 172.21.15.27:0/2327907723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006b0c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009870 tx=0x7f4c9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e09de60dbdc362 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:54.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.205+0000 7f4c9d600640 1 -- 172.21.15.27:0/2327907723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90035020 con 0x7f4ca00c6a90 2024-03-28T19:06:54.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.205+0000 7f4c9d600640 1 -- 172.21.15.27:0/2327907723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c90031870 con 0x7f4ca00c6a90 2024-03-28T19:06:54.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.205+0000 7f4c9d600640 1 -- 172.21.15.27:0/2327907723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90031c40 con 0x7f4ca00c6a90 2024-03-28T19:06:54.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.205+0000 7f4c9fe00640 1 -- 172.21.15.27:0/2327907723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 msgr2=0x7f4ca006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:54.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.205+0000 7f4c9fe00640 1 --2- 172.21.15.27:0/2327907723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006b0c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009870 tx=0x7f4c9002b910 comp rx=0 tx=0).stop 2024-03-28T19:06:54.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.206+0000 7f4c9fe00640 1 -- 172.21.15.27:0/2327907723 shutdown_connections 2024-03-28T19:06:54.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.206+0000 7f4c9fe00640 1 --2- 172.21.15.27:0/2327907723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006b0c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:54.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.206+0000 7f4c9fe00640 1 -- 172.21.15.27:0/2327907723 >> 172.21.15.27:0/2327907723 conn(0x7f4ca00aa7c0 msgr2=0x7f4ca00aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:54.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.206+0000 7f4c9fe00640 1 -- 172.21.15.27:0/2327907723 shutdown_connections 2024-03-28T19:06:54.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.206+0000 7f4c9fe00640 1 -- 172.21.15.27:0/2327907723 wait complete. 2024-03-28T19:06:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.207+0000 7f4c9fe00640 1 Processor -- start 2024-03-28T19:06:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.207+0000 7f4c9fe00640 1 -- start start 2024-03-28T19:06:54.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.208+0000 7f4c9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.208+0000 7f4c9fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ca00c6e70 0x7f4ca01519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.208+0000 7f4c9fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ca00c72f0 0x7f4ca0151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.208+0000 7f4c9fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca00c8630 con 0x7f4ca00c6a90 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.208+0000 7f4c9fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca0159610 con 0x7f4ca00c6e70 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.208+0000 7f4c9fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca0152e90 con 0x7f4ca00c72f0 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53494/0 (socket says 172.21.15.27:53494) 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9ea00640 1 -- 172.21.15.27:0/3888889297 learned_addr learned my addr 172.21.15.27:0/3888889297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9e000640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ca00c6e70 0x7f4ca01519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9f400640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ca00c72f0 0x7f4ca0151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9ea00640 1 -- 172.21.15.27:0/3888889297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ca00c72f0 msgr2=0x7f4ca0151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9ea00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ca00c72f0 0x7f4ca0151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:54.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9ea00640 1 -- 172.21.15.27:0/3888889297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ca00c6e70 msgr2=0x7f4ca01519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9ea00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ca00c6e70 0x7f4ca01519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.209+0000 7f4c9ea00640 1 -- 172.21.15.27:0/3888889297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0153f90 con 0x7f4ca00c6a90 2024-03-28T19:06:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.210+0000 7f4c9ea00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006f6c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4c90000c00 tx=0x7f4c900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:54.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.210+0000 7f4c96a00640 1 -- 172.21.15.27:0/3888889297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90040070 con 0x7f4ca00c6a90 2024-03-28T19:06:54.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.210+0000 7f4c96a00640 1 -- 172.21.15.27:0/3888889297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9003a420 con 0x7f4ca00c6a90 2024-03-28T19:06:54.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.210+0000 7f4c96a00640 1 -- 172.21.15.27:0/3888889297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90035040 con 0x7f4ca00c6a90 2024-03-28T19:06:54.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.210+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca0154270 con 0x7f4ca00c6a90 2024-03-28T19:06:54.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.210+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca006f100 con 0x7f4ca00c6a90 2024-03-28T19:06:54.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.212+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca0067dd0 con 0x7f4ca00c6a90 2024-03-28T19:06:54.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.213+0000 7f4c96a00640 1 -- 172.21.15.27:0/3888889297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4c9004b060 con 0x7f4ca00c6a90 2024-03-28T19:06:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.214+0000 7f4c96a00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c8006bbd0 0x7f4c8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.214+0000 7f4c96a00640 1 -- 172.21.15.27:0/3888889297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f4c900ad1b0 con 0x7f4ca00c6a90 2024-03-28T19:06:54.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.214+0000 7f4c9e000640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c8006bbd0 0x7f4c8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:54.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.215+0000 7f4c9e000640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c8006bbd0 0x7f4c8006e090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0156880 tx=0x7f4c8c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:54.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.220+0000 7f4c96a00640 1 -- 172.21.15.27:0/3888889297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4c9007b440 con 0x7f4ca00c6a90 2024-03-28T19:06:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.508+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ca0154790 con 0x7f4ca00c6a90 2024-03-28T19:06:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.509+0000 7f4c96a00640 1 -- 172.21.15.27:0/3888889297 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f4ca0154790 con 0x7f4ca00c6a90 2024-03-28T19:06:54.508 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:54.509 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-28T19:06:54.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c8006bbd0 msgr2=0x7f4c8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c8006bbd0 0x7f4c8006e090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0156880 tx=0x7f4c8c009040 comp rx=0 tx=0).stop 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 msgr2=0x7f4ca006f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006f6c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4c90000c00 tx=0x7f4c900046c0 comp rx=0 tx=0).stop 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 shutdown_connections 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4c8006bbd0 0x7f4c8006e090 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ca00c72f0 0x7f4ca0151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ca00c6e70 0x7f4ca01519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 --2- 172.21.15.27:0/3888889297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ca00c6a90 0x7f4ca006f6c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:54.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.512+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 >> 172.21.15.27:0/3888889297 conn(0x7f4ca00aa7c0 msgr2=0x7f4ca00bf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.513+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 shutdown_connections 2024-03-28T19:06:54.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:54.513+0000 7f4c9fe00640 1 -- 172.21.15.27:0/3888889297 wait complete. 2024-03-28T19:06:54.534 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:06:56.535 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.776+0000 7fe9f7400640 1 Processor -- start 2024-03-28T19:06:56.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.776+0000 7fe9f7400640 1 -- start start 2024-03-28T19:06:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.777+0000 7fe9f7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9f00be630 0x7fe9f00c5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.777+0000 7fe9f7400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe9f00c5e40 0x7fe9f0159150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.777+0000 7fe9f7400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f00c7910 con 0x7fe9f0158580 2024-03-28T19:06:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.777+0000 7fe9f7400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f00c76e0 con 0x7fe9f00be630 2024-03-28T19:06:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.777+0000 7fe9f7400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f00c6dd0 con 0x7fe9f00c5e40 2024-03-28T19:06:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.778+0000 7fe9efe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9f00be630 0x7fe9f00c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.778+0000 7fe9efe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9f00be630 0x7fe9f00c5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59266/0 (socket says 172.21.15.27:59266) 2024-03-28T19:06:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.778+0000 7fe9efe00640 1 -- 172.21.15.27:0/1155894574 learned_addr learned my addr 172.21.15.27:0/1155894574 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.778+0000 7fe9f5400640 1 --2- 172.21.15.27:0/1155894574 >> v2:172.21.15.155:3300/0 conn(0x7fe9f00c5e40 0x7fe9f0159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:56.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.778+0000 7fe9efe00640 1 -- 172.21.15.27:0/1155894574 >> v2:172.21.15.155:3300/0 conn(0x7fe9f00c5e40 msgr2=0x7fe9f0159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.778+0000 7fe9efe00640 1 --2- 172.21.15.27:0/1155894574 >> v2:172.21.15.155:3300/0 conn(0x7fe9f00c5e40 0x7fe9f0159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.778+0000 7fe9efe00640 1 -- 172.21.15.27:0/1155894574 >> v1:172.21.15.155:6789/0 conn(0x7fe9f0158580 legacy=0x7fe9f01560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.779+0000 7fe9efe00640 1 -- 172.21.15.27:0/1155894574 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00c7bf0 con 0x7fe9f00be630 2024-03-28T19:06:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.779+0000 7fe9f5400640 1 --2- 172.21.15.27:0/1155894574 >> v2:172.21.15.155:3300/0 conn(0x7fe9f00c5e40 0x7fe9f0159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:06:56.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.780+0000 7fe9efe00640 1 --2- 172.21.15.27:0/1155894574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00be630 0x7fe9f00c5900 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe9f00c8aa0 tx=0x7fe9dc02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3450b223ea7ca35e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.780+0000 7fe9ef400640 1 -- 172.21.15.27:0/1155894574 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc034070 con 0x7fe9f00be630 2024-03-28T19:06:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.780+0000 7fe9ef400640 1 -- 172.21.15.27:0/1155894574 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc030870 con 0x7fe9f00be630 2024-03-28T19:06:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.780+0000 7fe9ef400640 1 -- 172.21.15.27:0/1155894574 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc030c40 con 0x7fe9f00be630 2024-03-28T19:06:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.781+0000 7fe9f7400640 1 -- 172.21.15.27:0/1155894574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00be630 msgr2=0x7fe9f00c5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.781+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1155894574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00be630 0x7fe9f00c5900 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe9f00c8aa0 tx=0x7fe9dc02b9b0 comp rx=0 tx=0).stop 2024-03-28T19:06:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.781+0000 7fe9f7400640 1 -- 172.21.15.27:0/1155894574 shutdown_connections 2024-03-28T19:06:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.781+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1155894574 >> v2:172.21.15.155:3300/0 conn(0x7fe9f00c5e40 0x7fe9f0159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:56.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.781+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1155894574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00be630 0x7fe9f00c5900 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.781+0000 7fe9f7400640 1 -- 172.21.15.27:0/1155894574 >> 172.21.15.27:0/1155894574 conn(0x7fe9f00caf80 msgr2=0x7fe9f00cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.781+0000 7fe9f7400640 1 -- 172.21.15.27:0/1155894574 shutdown_connections 2024-03-28T19:06:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.781+0000 7fe9f7400640 1 -- 172.21.15.27:0/1155894574 wait complete. 2024-03-28T19:06:56.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.782+0000 7fe9f7400640 1 Processor -- start 2024-03-28T19:06:56.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.783+0000 7fe9f7400640 1 -- start start 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.783+0000 7fe9f7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9f00be630 0x7fe9f01ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.783+0000 7fe9f7400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9f00c5e40 0x7fe9f01eceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.783+0000 7fe9f7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f0158580 0x7fe9f01ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f7400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00c81b0 con 0x7fe9f0158580 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f7400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f0158cc0 con 0x7fe9f00be630 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f7400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f0158960 con 0x7fe9f00c5e40 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f0158580 0x7fe9f01ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f0158580 0x7fe9f01ed3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59278/0 (socket says 172.21.15.27:59278) 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f5400640 1 -- 172.21.15.27:0/1011367908 learned_addr learned my addr 172.21.15.27:0/1011367908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f4a00640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9f00be630 0x7fe9f01ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9efe00640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9f00c5e40 0x7fe9f01eceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:56.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f5400640 1 -- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9f00c5e40 msgr2=0x7fe9f01eceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f5400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9f00c5e40 0x7fe9f01eceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f5400640 1 -- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9f00be630 msgr2=0x7fe9f01ee5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f5400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9f00be630 0x7fe9f01ee5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f5400640 1 -- 172.21.15.27:0/1011367908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00c4ce0 con 0x7fe9f0158580 2024-03-28T19:06:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9efe00640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9f00c5e40 0x7fe9f01eceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.784+0000 7fe9f4a00640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9f00be630 0x7fe9f01ee5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:06:56.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.785+0000 7fe9f5400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f0158580 0x7fe9f01ed3f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8007e60 tx=0x7fe9e800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.785+0000 7fe9ecc00640 1 -- 172.21.15.27:0/1011367908 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e801a070 con 0x7fe9f0158580 2024-03-28T19:06:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.785+0000 7fe9ecc00640 1 -- 172.21.15.27:0/1011367908 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e8003bb0 con 0x7fe9f0158580 2024-03-28T19:06:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.785+0000 7fe9ecc00640 1 -- 172.21.15.27:0/1011367908 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8004030 con 0x7fe9f0158580 2024-03-28T19:06:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.785+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f01ef000 con 0x7fe9f0158580 2024-03-28T19:06:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.785+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f01ec6b0 con 0x7fe9f0158580 2024-03-28T19:06:56.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.787+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f01ec9e0 con 0x7fe9f0158580 2024-03-28T19:06:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.787+0000 7fe9ecc00640 1 -- 172.21.15.27:0/1011367908 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe9e801d070 con 0x7fe9f0158580 2024-03-28T19:06:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.788+0000 7fe9ecc00640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9d006bbd0 0x7fe9d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.788+0000 7fe9ecc00640 1 -- 172.21.15.27:0/1011367908 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7fe9e808b570 con 0x7fe9f0158580 2024-03-28T19:06:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.789+0000 7fe9f4a00640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9d006bbd0 0x7fe9d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.789+0000 7fe9f4a00640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9d006bbd0 0x7fe9d006e090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40015e0 tx=0x7fe9e4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:56.794+0000 7fe9ecc00640 1 -- 172.21.15.27:0/1011367908 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe9e8010070 con 0x7fe9f0158580 2024-03-28T19:06:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.078+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9f004f400 con 0x7fe9f0158580 2024-03-28T19:06:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.079+0000 7fe9ecc00640 1 -- 172.21.15.27:0/1011367908 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fe9e80597f0 con 0x7fe9f0158580 2024-03-28T19:06:57.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:57.079 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-28T19:06:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.082+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9d006bbd0 msgr2=0x7fe9d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.082+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9d006bbd0 0x7fe9d006e090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40015e0 tx=0x7fe9e4009040 comp rx=0 tx=0).stop 2024-03-28T19:06:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.082+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f0158580 msgr2=0x7fe9f01ed3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.082+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f0158580 0x7fe9f01ed3f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8007e60 tx=0x7fe9e800d4a0 comp rx=0 tx=0).stop 2024-03-28T19:06:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.083+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 shutdown_connections 2024-03-28T19:06:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.083+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9d006bbd0 0x7fe9d006e090 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.083+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f0158580 0x7fe9f01ed3f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.083+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9f00c5e40 0x7fe9f01eceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.083+0000 7fe9f7400640 1 --2- 172.21.15.27:0/1011367908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9f00be630 0x7fe9f01ee5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.083+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 >> 172.21.15.27:0/1011367908 conn(0x7fe9f00caf80 msgr2=0x7fe9f00aabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.083+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 shutdown_connections 2024-03-28T19:06:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:57.083+0000 7fe9f7400640 1 -- 172.21.15.27:0/1011367908 wait complete. 2024-03-28T19:06:57.100 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:06:59.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:06:59.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.346+0000 7f8211600640 1 Processor -- start 2024-03-28T19:06:59.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.346+0000 7f8211600640 1 -- start start 2024-03-28T19:06:59.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.348+0000 7f8211600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f820c0c58c0 0x7f820c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.348+0000 7f8211600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f820c0be5d0 0x7f820c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.348+0000 7f8211600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820c156230 con 0x7f820c0c8ed0 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.348+0000 7f8211600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c158bd0 con 0x7f820c0be5d0 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.348+0000 7f8211600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c158eb0 con 0x7f820c0c58c0 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.348+0000 7f8210c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f820c0be5d0 0x7f820c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.348+0000 7f8210c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f820c0be5d0 0x7f820c155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59292/0 (socket says 172.21.15.27:59292) 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.348+0000 7f8210c00640 1 -- 172.21.15.27:0/201199015 learned_addr learned my addr 172.21.15.27:0/201199015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.349+0000 7f820be00640 1 --2- 172.21.15.27:0/201199015 >> v2:172.21.15.155:3300/0 conn(0x7f820c0c58c0 0x7f820c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.349+0000 7f8210c00640 1 -- 172.21.15.27:0/201199015 >> v2:172.21.15.155:3300/0 conn(0x7f820c0c58c0 msgr2=0x7f820c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.349+0000 7f8210c00640 1 --2- 172.21.15.27:0/201199015 >> v2:172.21.15.155:3300/0 conn(0x7f820c0c58c0 0x7f820c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.349+0000 7f8210c00640 1 -- 172.21.15.27:0/201199015 >> v1:172.21.15.155:6789/0 conn(0x7f820c0c8ed0 legacy=0x7f820c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:06:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.349+0000 7f8210c00640 1 -- 172.21.15.27:0/201199015 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c159190 con 0x7f820c0be5d0 2024-03-28T19:06:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.349+0000 7f820be00640 1 --2- 172.21.15.27:0/201199015 >> v2:172.21.15.155:3300/0 conn(0x7f820c0c58c0 0x7f820c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:06:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.350+0000 7f8210c00640 1 --2- 172.21.15.27:0/201199015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c155cf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f820c06c060 tx=0x7f820402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1022b6beed8ce26a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f820b400640 1 -- 172.21.15.27:0/201199015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820402c960 con 0x7f820c0be5d0 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f820b400640 1 -- 172.21.15.27:0/201199015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820402cb00 con 0x7f820c0be5d0 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f820b400640 1 -- 172.21.15.27:0/201199015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040343f0 con 0x7f820c0be5d0 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f8211600640 1 -- 172.21.15.27:0/201199015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 msgr2=0x7f820c155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f8211600640 1 --2- 172.21.15.27:0/201199015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c155cf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f820c06c060 tx=0x7f820402bdc0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f8211600640 1 -- 172.21.15.27:0/201199015 shutdown_connections 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f8211600640 1 --2- 172.21.15.27:0/201199015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c155cf0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f8211600640 1 --2- 172.21.15.27:0/201199015 >> v2:172.21.15.155:3300/0 conn(0x7f820c0c58c0 0x7f820c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.351+0000 7f8211600640 1 -- 172.21.15.27:0/201199015 >> 172.21.15.27:0/201199015 conn(0x7f820c06b6f0 msgr2=0x7f820c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.352+0000 7f8211600640 1 -- 172.21.15.27:0/201199015 shutdown_connections 2024-03-28T19:06:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.352+0000 7f8211600640 1 -- 172.21.15.27:0/201199015 wait complete. 2024-03-28T19:06:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.353+0000 7f8211600640 1 Processor -- start 2024-03-28T19:06:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.353+0000 7f8211600640 1 -- start start 2024-03-28T19:06:59.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.354+0000 7f8211600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c0c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.354+0000 7f8211600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c58c0 0x7f820c1ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.354+0000 7f8211600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0c8ed0 0x7f820c1ec080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.354+0000 7f8211600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0b0820 con 0x7f820c0be5d0 2024-03-28T19:06:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.354+0000 7f8211600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c159750 con 0x7f820c0c8ed0 2024-03-28T19:06:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.354+0000 7f8211600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0b09a0 con 0x7f820c0c58c0 2024-03-28T19:06:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.354+0000 7f820be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c0c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.354+0000 7f820be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c0c2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59298/0 (socket says 172.21.15.27:59298) 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.355+0000 7f820be00640 1 -- 172.21.15.27:0/2192079140 learned_addr learned my addr 172.21.15.27:0/2192079140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.355+0000 7f8210c00640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0c8ed0 0x7f820c1ec080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.355+0000 7f820be00640 1 -- 172.21.15.27:0/2192079140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c58c0 msgr2=0x7f820c1ebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.355+0000 7f820be00640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c58c0 0x7f820c1ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.355+0000 7f820be00640 1 -- 172.21.15.27:0/2192079140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0c8ed0 msgr2=0x7f820c1ec080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.355+0000 7f820be00640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0c8ed0 0x7f820c1ec080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.355+0000 7f820be00640 1 -- 172.21.15.27:0/2192079140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c1ef3d0 con 0x7f820c0be5d0 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.355+0000 7f820be00640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c0c2280 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00d7d0 tx=0x7f81fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:59.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.356+0000 7f8208c00640 1 -- 172.21.15.27:0/2192079140 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc004030 con 0x7f820c0be5d0 2024-03-28T19:06:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.356+0000 7f8208c00640 1 -- 172.21.15.27:0/2192079140 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc0041d0 con 0x7f820c0be5d0 2024-03-28T19:06:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.356+0000 7f8208c00640 1 -- 172.21.15.27:0/2192079140 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc0096a0 con 0x7f820c0be5d0 2024-03-28T19:06:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.356+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c0aad20 con 0x7f820c0be5d0 2024-03-28T19:06:59.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.356+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c0c27c0 con 0x7f820c0be5d0 2024-03-28T19:06:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.357+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c069640 con 0x7f820c0be5d0 2024-03-28T19:06:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.358+0000 7f8208c00640 1 -- 172.21.15.27:0/2192079140 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f81fc009840 con 0x7f820c0be5d0 2024-03-28T19:06:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.360+0000 7f8208c00640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81e806bbd0 0x7f81e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:06:59.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.360+0000 7f8208c00640 1 -- 172.21.15.27:0/2192079140 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f81fc08b690 con 0x7f820c0be5d0 2024-03-28T19:06:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.360+0000 7f8203e00640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81e806bbd0 0x7f81e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:06:59.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.361+0000 7f8203e00640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81e806bbd0 0x7f81e806e090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f81f4006fd0 tx=0x7f81f4006bc0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:06:59.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.365+0000 7f8208c00640 1 -- 172.21.15.27:0/2192079140 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f820c1ef3d0 con 0x7f820c0be5d0 2024-03-28T19:06:59.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.660+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f820c156580 con 0x7f820c0be5d0 2024-03-28T19:06:59.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.662+0000 7f8208c00640 1 -- 172.21.15.27:0/2192079140 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f81fc059880 con 0x7f820c0be5d0 2024-03-28T19:06:59.662 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:06:59.662 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:06:59.663 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-28T19:06:59.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81e806bbd0 msgr2=0x7f81e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:59.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81e806bbd0 0x7f81e806e090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f81f4006fd0 tx=0x7f81f4006bc0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 msgr2=0x7f820c0c2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:06:59.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c0c2280 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00d7d0 tx=0x7f81fc00dc90 comp rx=0 tx=0).stop 2024-03-28T19:06:59.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 shutdown_connections 2024-03-28T19:06:59.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81e806bbd0 0x7f81e806e090 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0c8ed0 0x7f820c1ec080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c58c0 0x7f820c1ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 --2- 172.21.15.27:0/2192079140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0be5d0 0x7f820c0c2280 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:06:59.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.665+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 >> 172.21.15.27:0/2192079140 conn(0x7f820c06b6f0 msgr2=0x7f820c072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:06:59.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.666+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 shutdown_connections 2024-03-28T19:06:59.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:06:59.666+0000 7f8211600640 1 -- 172.21.15.27:0/2192079140 wait complete. 2024-03-28T19:06:59.683 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:07:00.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:07:00.591 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:07:01.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:01.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.921+0000 7f9e4f200640 1 Processor -- start 2024-03-28T19:07:01.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.921+0000 7f9e4f200640 1 -- start start 2024-03-28T19:07:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.922+0000 7f9e4f200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9e50152c80 0x7f9e500c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.922+0000 7f9e4f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e501596b0 0x7f9e50156060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.922+0000 7f9e4f200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e501558a0 con 0x7f9e500c79b0 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.922+0000 7f9e4f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e500c5cc0 con 0x7f9e501596b0 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.922+0000 7f9e4f200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e500c5fa0 con 0x7f9e50152c80 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.923+0000 7f9e4d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e501596b0 0x7f9e50156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.923+0000 7f9e4d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e501596b0 0x7f9e50156060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59308/0 (socket says 172.21.15.27:59308) 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.923+0000 7f9e4d400640 1 -- 172.21.15.27:0/2837259109 learned_addr learned my addr 172.21.15.27:0/2837259109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.923+0000 7f9e4de00640 1 --2- 172.21.15.27:0/2837259109 >> v2:172.21.15.155:3300/0 conn(0x7f9e50152c80 0x7f9e500c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.923+0000 7f9e4d400640 1 -- 172.21.15.27:0/2837259109 >> v2:172.21.15.155:3300/0 conn(0x7f9e50152c80 msgr2=0x7f9e500c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.923+0000 7f9e4d400640 1 --2- 172.21.15.27:0/2837259109 >> v2:172.21.15.155:3300/0 conn(0x7f9e50152c80 0x7f9e500c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.923+0000 7f9e4d400640 1 -- 172.21.15.27:0/2837259109 >> v1:172.21.15.155:6789/0 conn(0x7f9e500c79b0 legacy=0x7f9e500716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.923+0000 7f9e4d400640 1 -- 172.21.15.27:0/2837259109 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50155b40 con 0x7f9e501596b0 2024-03-28T19:07:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.925+0000 7f9e4d400640 1 --2- 172.21.15.27:0/2837259109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50156060 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9e500c8630 tx=0x7f9e3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f03e24f5415652 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.925+0000 7f9e4ca00640 1 -- 172.21.15.27:0/2837259109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c02e070 con 0x7f9e501596b0 2024-03-28T19:07:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.925+0000 7f9e4ca00640 1 -- 172.21.15.27:0/2837259109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3c037070 con 0x7f9e501596b0 2024-03-28T19:07:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.925+0000 7f9e4ca00640 1 -- 172.21.15.27:0/2837259109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c031a00 con 0x7f9e501596b0 2024-03-28T19:07:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.926+0000 7f9e4f200640 1 -- 172.21.15.27:0/2837259109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 msgr2=0x7f9e50156060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.926+0000 7f9e4f200640 1 --2- 172.21.15.27:0/2837259109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50156060 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9e500c8630 tx=0x7f9e3c02b880 comp rx=0 tx=0).stop 2024-03-28T19:07:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.926+0000 7f9e4f200640 1 -- 172.21.15.27:0/2837259109 shutdown_connections 2024-03-28T19:07:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.926+0000 7f9e4f200640 1 --2- 172.21.15.27:0/2837259109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50156060 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.926+0000 7f9e4f200640 1 --2- 172.21.15.27:0/2837259109 >> v2:172.21.15.155:3300/0 conn(0x7f9e50152c80 0x7f9e500c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.926+0000 7f9e4f200640 1 -- 172.21.15.27:0/2837259109 >> 172.21.15.27:0/2837259109 conn(0x7f9e5006b6c0 msgr2=0x7f9e500cafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.926+0000 7f9e4f200640 1 -- 172.21.15.27:0/2837259109 shutdown_connections 2024-03-28T19:07:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.926+0000 7f9e4f200640 1 -- 172.21.15.27:0/2837259109 wait complete. 2024-03-28T19:07:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.928+0000 7f9e4f200640 1 Processor -- start 2024-03-28T19:07:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.928+0000 7f9e4f200640 1 -- start start 2024-03-28T19:07:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e500c79b0 0x7f9e501eb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50152c80 0x7f9e50154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e501565a0 con 0x7f9e501596b0 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e500cbd70 con 0x7f9e50152c80 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e50159b30 con 0x7f9e500c79b0 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59320/0 (socket says 172.21.15.27:59320) 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4e800640 1 -- 172.21.15.27:0/4117801278 learned_addr learned my addr 172.21.15.27:0/4117801278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4de00640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e500c79b0 0x7f9e501eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4d400640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50152c80 0x7f9e50154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4e800640 1 -- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e500c79b0 msgr2=0x7f9e501eb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4e800640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e500c79b0 0x7f9e501eb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4e800640 1 -- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50152c80 msgr2=0x7f9e50154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4e800640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50152c80 0x7f9e50154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.930+0000 7f9e4e800640 1 -- 172.21.15.27:0/4117801278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e501580b0 con 0x7f9e501596b0 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.931+0000 7f9e4d400640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50152c80 0x7f9e50154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.931+0000 7f9e4e800640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50157390 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9e4800c780 tx=0x7f9e4800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.931+0000 7f9e46000640 1 -- 172.21.15.27:0/4117801278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48004b60 con 0x7f9e501596b0 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.931+0000 7f9e46000640 1 -- 172.21.15.27:0/4117801278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e48004d00 con 0x7f9e501596b0 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.931+0000 7f9e46000640 1 -- 172.21.15.27:0/4117801278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4800aa90 con 0x7f9e501596b0 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.931+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e50158390 con 0x7f9e501596b0 2024-03-28T19:07:01.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.931+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e50156e70 con 0x7f9e501596b0 2024-03-28T19:07:01.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.933+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e50069640 con 0x7f9e501596b0 2024-03-28T19:07:01.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.934+0000 7f9e46000640 1 -- 172.21.15.27:0/4117801278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9e4800ac30 con 0x7f9e501596b0 2024-03-28T19:07:01.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.935+0000 7f9e46000640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e3006bbd0 0x7f9e3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:01.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.935+0000 7f9e46000640 1 -- 172.21.15.27:0/4117801278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f9e4808b5d0 con 0x7f9e501596b0 2024-03-28T19:07:01.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.935+0000 7f9e4de00640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e3006bbd0 0x7f9e3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:01.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.936+0000 7f9e4de00640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e3006bbd0 0x7f9e3006e090 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9e40006fd0 tx=0x7f9e40008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:01.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:01.940+0000 7f9e46000640 1 -- 172.21.15.27:0/4117801278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9e48020070 con 0x7f9e501596b0 2024-03-28T19:07:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.205+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e500cbae0 con 0x7f9e501596b0 2024-03-28T19:07:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.206+0000 7f9e46000640 1 -- 172.21.15.27:0/4117801278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f9e48059850 con 0x7f9e501596b0 2024-03-28T19:07:02.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:02.206 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:07:02.206 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-28T19:07:02.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.210+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e3006bbd0 msgr2=0x7f9e3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:02.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.210+0000 7f9e4f200640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e3006bbd0 0x7f9e3006e090 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9e40006fd0 tx=0x7f9e40008040 comp rx=0 tx=0).stop 2024-03-28T19:07:02.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.210+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 msgr2=0x7f9e50157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:02.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.210+0000 7f9e4f200640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50157390 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9e4800c780 tx=0x7f9e4800cc40 comp rx=0 tx=0).stop 2024-03-28T19:07:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.211+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 shutdown_connections 2024-03-28T19:07:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.211+0000 7f9e4f200640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e3006bbd0 0x7f9e3006e090 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.211+0000 7f9e4f200640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e501596b0 0x7f9e50157390 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.211+0000 7f9e4f200640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e50152c80 0x7f9e50154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.211+0000 7f9e4f200640 1 --2- 172.21.15.27:0/4117801278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e500c79b0 0x7f9e501eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.211+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 >> 172.21.15.27:0/4117801278 conn(0x7f9e5006b6c0 msgr2=0x7f9e5014e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.211+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 shutdown_connections 2024-03-28T19:07:02.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:02.211+0000 7f9e4f200640 1 -- 172.21.15.27:0/4117801278 wait complete. 2024-03-28T19:07:02.228 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:07:04.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:04.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.455+0000 7f44ac200640 1 Processor -- start 2024-03-28T19:07:04.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.455+0000 7f44ac200640 1 -- start start 2024-03-28T19:07:04.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.456+0000 7f44ac200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44a4150c70 0x7f44a40a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:04.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.456+0000 7f44ac200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a40c3af0 con 0x7f44a41534c0 2024-03-28T19:07:04.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.456+0000 7f44ac200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a4154020 con 0x7f44a40c3690 2024-03-28T19:07:04.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.456+0000 7f44ac200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a40ab4e0 con 0x7f44a4150c70 2024-03-28T19:07:04.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.457+0000 7f44a9800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44a4150c70 0x7f44a40a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:04.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.457+0000 7f44a9800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44a4150c70 0x7f44a40a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59332/0 (socket says 172.21.15.27:59332) 2024-03-28T19:07:04.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.457+0000 7f44a9800640 1 -- 172.21.15.27:0/3711409626 learned_addr learned my addr 172.21.15.27:0/3711409626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:04.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.457+0000 7f44aa200640 1 --1- 172.21.15.27:0/3711409626 >> v1:172.21.15.27:6789/0 conn(0x7f44a41534c0 0x7f44a4153910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41604/0 (socket says 172.21.15.27:41604) 2024-03-28T19:07:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.457+0000 7f44a9800640 1 -- 172.21.15.27:0/3711409626 >> v1:172.21.15.155:6789/0 conn(0x7f44a40c3690 legacy=0x7f44a40c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.457+0000 7f44a9800640 1 -- 172.21.15.27:0/3711409626 >> v1:172.21.15.27:6789/0 conn(0x7f44a41534c0 legacy=0x7f44a4153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.457+0000 7f44a9800640 1 -- 172.21.15.27:0/3711409626 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a40c3d90 con 0x7f44a4150c70 2024-03-28T19:07:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.459+0000 7f44a9800640 1 --2- 172.21.15.27:0/3711409626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a4150c70 0x7f44a40a9cb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f44a41541f0 tx=0x7f449802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c319dcb6216e4739 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.459+0000 7f44a3e00640 1 -- 172.21.15.27:0/3711409626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498034070 con 0x7f44a4150c70 2024-03-28T19:07:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.459+0000 7f44a3e00640 1 -- 172.21.15.27:0/3711409626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4498030840 con 0x7f44a4150c70 2024-03-28T19:07:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.459+0000 7f44a3e00640 1 -- 172.21.15.27:0/3711409626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498030c10 con 0x7f44a4150c70 2024-03-28T19:07:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.460+0000 7f44ac200640 1 -- 172.21.15.27:0/3711409626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a4150c70 msgr2=0x7f44a40a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.460+0000 7f44ac200640 1 --2- 172.21.15.27:0/3711409626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a4150c70 0x7f44a40a9cb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f44a41541f0 tx=0x7f449802b980 comp rx=0 tx=0).stop 2024-03-28T19:07:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.460+0000 7f44ac200640 1 -- 172.21.15.27:0/3711409626 shutdown_connections 2024-03-28T19:07:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.460+0000 7f44ac200640 1 --2- 172.21.15.27:0/3711409626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a4150c70 0x7f44a40a9cb0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.460+0000 7f44ac200640 1 -- 172.21.15.27:0/3711409626 >> 172.21.15.27:0/3711409626 conn(0x7f44a406b9d0 msgr2=0x7f44a40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.460+0000 7f44ac200640 1 -- 172.21.15.27:0/3711409626 shutdown_connections 2024-03-28T19:07:04.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.460+0000 7f44ac200640 1 -- 172.21.15.27:0/3711409626 wait complete. 2024-03-28T19:07:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.462+0000 7f44ac200640 1 Processor -- start 2024-03-28T19:07:04.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.462+0000 7f44ac200640 1 -- start start 2024-03-28T19:07:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44ac200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44a40c3690 0x7f44a41eb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44ac200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44a4150c70 0x7f44a406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44ac200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a41534c0 0x7f44a40a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44ac200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a40ab9f0 con 0x7f44a41534c0 2024-03-28T19:07:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44ac200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a40ac1f0 con 0x7f44a4150c70 2024-03-28T19:07:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44ac200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a40abcd0 con 0x7f44a40c3690 2024-03-28T19:07:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44aa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a41534c0 0x7f44a40a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44aa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a41534c0 0x7f44a40a6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59338/0 (socket says 172.21.15.27:59338) 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.463+0000 7f44aa200640 1 -- 172.21.15.27:0/2639378703 learned_addr learned my addr 172.21.15.27:0/2639378703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44a9800640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44a40c3690 0x7f44a41eb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44a8e00640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44a4150c70 0x7f44a406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44aa200640 1 -- 172.21.15.27:0/2639378703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44a40c3690 msgr2=0x7f44a41eb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44aa200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44a40c3690 0x7f44a41eb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44aa200640 1 -- 172.21.15.27:0/2639378703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44a4150c70 msgr2=0x7f44a406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44aa200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44a4150c70 0x7f44a406e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44aa200640 1 -- 172.21.15.27:0/2639378703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a4154f80 con 0x7f44a41534c0 2024-03-28T19:07:04.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44aa200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a41534c0 0x7f44a40a6980 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f449c00c760 tx=0x7f449c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44a1600640 1 -- 172.21.15.27:0/2639378703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c017070 con 0x7f44a41534c0 2024-03-28T19:07:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.464+0000 7f44a1600640 1 -- 172.21.15.27:0/2639378703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449c00a640 con 0x7f44a41534c0 2024-03-28T19:07:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.465+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a41eb770 con 0x7f44a41534c0 2024-03-28T19:07:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.465+0000 7f44a1600640 1 -- 172.21.15.27:0/2639378703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c00a9f0 con 0x7f44a41534c0 2024-03-28T19:07:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.465+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a41ebd30 con 0x7f44a41534c0 2024-03-28T19:07:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.466+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a404f400 con 0x7f44a41534c0 2024-03-28T19:07:04.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.468+0000 7f44a1600640 1 -- 172.21.15.27:0/2639378703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f449c01a070 con 0x7f44a41534c0 2024-03-28T19:07:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.469+0000 7f44a1600640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f448406bbd0 0x7f448406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.469+0000 7f44a1600640 1 -- 172.21.15.27:0/2639378703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7841+0+0 (secure 0 0 0) 0x7f449c08a7c0 con 0x7f44a41534c0 2024-03-28T19:07:04.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.469+0000 7f44a9800640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f448406bbd0 0x7f448406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:04.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.470+0000 7f44a9800640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f448406bbd0 0x7f448406e090 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4498036000 tx=0x7f449803b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.475+0000 7f44a1600640 1 -- 172.21.15.27:0/2639378703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f449c0583f0 con 0x7f44a41534c0 2024-03-28T19:07:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.759+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44a40c66c0 con 0x7f44a41534c0 2024-03-28T19:07:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.760+0000 7f44a1600640 1 -- 172.21.15.27:0/2639378703 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f449c05c490 con 0x7f44a41534c0 2024-03-28T19:07:04.759 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:04.759 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:04:44.514032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":13,"addr":"172.21.15.27:6837/1815850547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":1815850547},{"type":"v1","addr":"172.21.15.27:6837","nonce":1815850547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-03-28T19:07:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 92 2024-03-28T19:07:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.763+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f448406bbd0 msgr2=0x7f448406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.763+0000 7f44ac200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f448406bbd0 0x7f448406e090 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4498036000 tx=0x7f449803b040 comp rx=0 tx=0).stop 2024-03-28T19:07:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.763+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a41534c0 msgr2=0x7f44a40a6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.763+0000 7f44ac200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a41534c0 0x7f44a40a6980 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f449c00c760 tx=0x7f449c00cc20 comp rx=0 tx=0).stop 2024-03-28T19:07:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.764+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 shutdown_connections 2024-03-28T19:07:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.764+0000 7f44ac200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f448406bbd0 0x7f448406e090 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.764+0000 7f44ac200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a41534c0 0x7f44a40a6980 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.764+0000 7f44ac200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44a4150c70 0x7f44a406e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.764+0000 7f44ac200640 1 --2- 172.21.15.27:0/2639378703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44a40c3690 0x7f44a41eb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.764+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 >> 172.21.15.27:0/2639378703 conn(0x7f44a406b9d0 msgr2=0x7f44a40c2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.764+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 shutdown_connections 2024-03-28T19:07:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:04.764+0000 7f44ac200640 1 -- 172.21.15.27:0/2639378703 wait complete. 2024-03-28T19:07:04.784 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:07:06.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:07.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.021+0000 7f022a000640 1 Processor -- start 2024-03-28T19:07:07.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.022+0000 7f022a000640 1 -- start start 2024-03-28T19:07:07.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.023+0000 7f022a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02240c59d0 0x7f02240c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.023+0000 7f022a000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0224155fe0 con 0x7f02240727e0 2024-03-28T19:07:07.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.023+0000 7f022a000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02241562c0 con 0x7f0224155600 2024-03-28T19:07:07.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.023+0000 7f022a000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02241565a0 con 0x7f02240c59d0 2024-03-28T19:07:07.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.023+0000 7f0223e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f02240727e0 0x7f0224159520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53326/0 (socket says 172.21.15.27:53326) 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.023+0000 7f0223e00640 1 -- 172.21.15.27:0/3524318612 learned_addr learned my addr 172.21.15.27:0/3524318612 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.024+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128047996 0 0) 0x7f0224155fe0 con 0x7f02240727e0 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.024+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f020c003520 con 0x7f02240727e0 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2348568741 0 0) 0x7f020c003520 con 0x7f02240727e0 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02240c59d0 msgr2=0x7f02240c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0228c00640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02240c59d0 0x7f02240c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02240c59d0 0x7f02240c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 >> v1:172.21.15.155:6789/0 conn(0x7f0224155600 legacy=0x7f022414f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0224158d20 con 0x7f02240727e0 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f0210001670 con 0x7f02240727e0 2024-03-28T19:07:07.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 >> v1:172.21.15.27:6789/0 conn(0x7f02240727e0 legacy=0x7f0224159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f020c0039e0 0x7f020c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c004800 0x7f020c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c0052e0 0x7f020c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02241565a0 con 0x7f020c0039e0 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c00c1e0 con 0x7f020c0052e0 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c00c3e0 con 0x7f020c004800 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0228c00640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02240c59d0 0x7f02240c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.025+0000 7f0228c00640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f020c0039e0 0x7f020c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.026+0000 7f0229600640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c0052e0 0x7f020c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.026+0000 7f0223e00640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c004800 0x7f020c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.026+0000 7f0228c00640 1 -- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c004800 msgr2=0x7f020c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:07.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.026+0000 7f0228c00640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c004800 0x7f020c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.026+0000 7f0228c00640 1 -- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c0052e0 msgr2=0x7f020c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:07.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.026+0000 7f0228c00640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c0052e0 0x7f020c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.026+0000 7f0228c00640 1 -- 172.21.15.27:0/3524318612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0224158d20 con 0x7f020c0039e0 2024-03-28T19:07:07.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.026+0000 7f0223e00640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c004800 0x7f020c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:07:07.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.028+0000 7f0228c00640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f020c0039e0 0x7f020c003de0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f02240cacc0 tx=0x7f020c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3d4a9f90ec907 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:07.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.028+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021800a4b0 con 0x7f020c0039e0 2024-03-28T19:07:07.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.029+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218037070 con 0x7f020c0039e0 2024-03-28T19:07:07.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.029+0000 7f0223400640 1 -- 172.21.15.27:0/3524318612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218032880 con 0x7f020c0039e0 2024-03-28T19:07:07.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.029+0000 7f022a000640 1 -- 172.21.15.27:0/3524318612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f020c0039e0 msgr2=0x7f020c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.029+0000 7f022a000640 1 --2- 172.21.15.27:0/3524318612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f020c0039e0 0x7f020c003de0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f02240cacc0 tx=0x7f020c005830 comp rx=0 tx=0).stop 2024-03-28T19:07:07.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.029+0000 7f0229600640 1 -- 172.21.15.27:0/3524318612 reap_dead start 2024-03-28T19:07:07.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.029+0000 7f022a000640 1 -- 172.21.15.27:0/3524318612 shutdown_connections 2024-03-28T19:07:07.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.029+0000 7f022a000640 1 -- 172.21.15.27:0/3524318612 >> 172.21.15.27:0/3524318612 conn(0x7f02240ac910 msgr2=0x7f02240accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:07.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.030+0000 7f022a000640 1 -- 172.21.15.27:0/3524318612 shutdown_connections 2024-03-28T19:07:07.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.030+0000 7f022a000640 1 -- 172.21.15.27:0/3524318612 wait complete. 2024-03-28T19:07:07.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.032+0000 7f022a000640 1 Processor -- start 2024-03-28T19:07:07.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.033+0000 7f022a000640 1 -- start start 2024-03-28T19:07:07.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f022a000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c0039e0 0x7f02241ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f022a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02240c2150 0x7f02240abf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f022a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02240aad40 0x7f02240ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f022a000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02240c5db0 con 0x7f02240c2150 2024-03-28T19:07:07.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f022a000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0224159090 con 0x7f02240aad40 2024-03-28T19:07:07.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f022a000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02240c5f30 con 0x7f020c0039e0 2024-03-28T19:07:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f0223e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02240c2150 0x7f02240abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f0223e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02240c2150 0x7f02240abf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48648/0 (socket says 172.21.15.27:48648) 2024-03-28T19:07:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.034+0000 7f0223e00640 1 -- 172.21.15.27:0/1750394792 learned_addr learned my addr 172.21.15.27:0/1750394792 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:07.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.035+0000 7f0229600640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02240aad40 0x7f02240ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.035+0000 7f0228c00640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c0039e0 0x7f02241ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.035+0000 7f0223e00640 1 -- 172.21.15.27:0/1750394792 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c0039e0 msgr2=0x7f02241ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.035+0000 7f0223e00640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c0039e0 0x7f02241ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.035+0000 7f0223e00640 1 -- 172.21.15.27:0/1750394792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02240aad40 msgr2=0x7f02240ab1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.035+0000 7f0223e00640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02240aad40 0x7f02240ab1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.035+0000 7f0223e00640 1 -- 172.21.15.27:0/1750394792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02240c4ef0 con 0x7f02240c2150 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.036+0000 7f0223e00640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02240c2150 0x7f02240abf40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f021000e800 tx=0x7f021000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.036+0000 7f0220c00640 1 -- 172.21.15.27:0/1750394792 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021000c870 con 0x7f02240c2150 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.036+0000 7f0220c00640 1 -- 172.21.15.27:0/1750394792 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0210018070 con 0x7f02240c2150 2024-03-28T19:07:07.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.036+0000 7f0220c00640 1 -- 172.21.15.27:0/1750394792 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021000cc10 con 0x7f02240c2150 2024-03-28T19:07:07.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.036+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02241ef160 con 0x7f02240c2150 2024-03-28T19:07:07.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.036+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02241eeba0 con 0x7f02240c2150 2024-03-28T19:07:07.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.038+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0224067dd0 con 0x7f02240c2150 2024-03-28T19:07:07.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.039+0000 7f0220c00640 1 -- 172.21.15.27:0/1750394792 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f021001d460 con 0x7f02240c2150 2024-03-28T19:07:07.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.041+0000 7f0220c00640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f020406bc20 0x7f020406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.041+0000 7f0220c00640 1 -- 172.21.15.27:0/1750394792 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f021008b0d0 con 0x7f02240c2150 2024-03-28T19:07:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.041+0000 7f0228c00640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f020406bc20 0x7f020406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.042+0000 7f0228c00640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f020406bc20 0x7f020406e0e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0218036000 tx=0x7f0218031000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.047+0000 7f0220c00640 1 -- 172.21.15.27:0/1750394792 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0210059380 con 0x7f02240c2150 2024-03-28T19:07:07.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.325+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0224069640 con 0x7f02240c2150 2024-03-28T19:07:07.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.325+0000 7f0220c00640 1 -- 172.21.15.27:0/1750394792 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5460 (secure 0 0 0) 0x7f021005f080 con 0x7f02240c2150 2024-03-28T19:07:07.325 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:07.325 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:04.840887+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:07:07.325 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 93 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f020406bc20 msgr2=0x7f020406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f020406bc20 0x7f020406e0e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0218036000 tx=0x7f0218031000 comp rx=0 tx=0).stop 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02240c2150 msgr2=0x7f02240abf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02240c2150 0x7f02240abf40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f021000e800 tx=0x7f021000ecc0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 shutdown_connections 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f020406bc20 0x7f020406e0e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02240aad40 0x7f02240ab1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02240c2150 0x7f02240abf40 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 --2- 172.21.15.27:0/1750394792 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f020c0039e0 0x7f02241ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 >> 172.21.15.27:0/1750394792 conn(0x7f02240ac910 msgr2=0x7f022414efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.328+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 shutdown_connections 2024-03-28T19:07:07.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.329+0000 7f022a000640 1 -- 172.21.15.27:0/1750394792 wait complete. 2024-03-28T19:07:07.347 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-03-28T19:07:07.347 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:07:07.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.559+0000 7f0cafe00640 1 Processor -- start 2024-03-28T19:07:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.560+0000 7f0cafe00640 1 -- start start 2024-03-28T19:07:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.561+0000 7f0cafe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0cb00c58c0 0x7f0cb00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.561+0000 7f0cafe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c8ed0 0x7f0cb01555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.561+0000 7f0cafe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb0156230 con 0x7f0cb00be5d0 2024-03-28T19:07:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.561+0000 7f0cafe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb0158bd0 con 0x7f0cb00c58c0 2024-03-28T19:07:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.561+0000 7f0cafe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0158eb0 con 0x7f0cb00c8ed0 2024-03-28T19:07:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.561+0000 7f0caf400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0cb00be5d0 0x7f0cb0155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53342/0 (socket says 172.21.15.27:53342) 2024-03-28T19:07:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.561+0000 7f0caf400640 1 -- 172.21.15.27:0/2084443338 learned_addr learned my addr 172.21.15.27:0/2084443338 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:07:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cae000640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c8ed0 0x7f0cb01555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0caea00640 1 --2- 172.21.15.27:0/2084443338 >> v2:172.21.15.155:3300/0 conn(0x7f0cb00c58c0 0x7f0cb00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623434889 0 0) 0x7f0cb0156230 con 0x7f0cb00be5d0 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c94003520 con 0x7f0cb00be5d0 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095778026 0 0) 0x7f0c94003520 con 0x7f0cb00be5d0 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c8ed0 msgr2=0x7f0cb01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c8ed0 0x7f0cb01555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 >> v2:172.21.15.155:3300/0 conn(0x7f0cb00c58c0 msgr2=0x7f0cb00c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 --2- 172.21.15.27:0/2084443338 >> v2:172.21.15.155:3300/0 conn(0x7f0cb00c58c0 0x7f0cb00c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0159190 con 0x7f0cb00be5d0 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f0ca80019f0 con 0x7f0cb00be5d0 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 >> v1:172.21.15.27:6789/0 conn(0x7f0cb00be5d0 legacy=0x7f0cb0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c940039b0 0x7f0c94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c8ed0 0x7f0c94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c94004d40 0x7f0c940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0158bd0 con 0x7f0c94004d40 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0158eb0 con 0x7f0c940039b0 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.562+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9400b9f0 con 0x7f0cb00c8ed0 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.563+0000 7f0caf400640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c94004d40 0x7f0c940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.563+0000 7f0caea00640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c940039b0 0x7f0c94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.563+0000 7f0cae000640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c8ed0 0x7f0c94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.563+0000 7f0caf400640 1 -- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c8ed0 msgr2=0x7f0c94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.563+0000 7f0caf400640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c8ed0 0x7f0c94004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.563+0000 7f0caf400640 1 -- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c940039b0 msgr2=0x7f0c94003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.563+0000 7f0caf400640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c940039b0 0x7f0c94003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.563+0000 7f0caf400640 1 -- 172.21.15.27:0/2084443338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c94003520 con 0x7f0c94004d40 2024-03-28T19:07:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.565+0000 7f0caf400640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c94004d40 0x7f0c940051c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800a7a0 tx=0x7f0c94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f58d96fa6f4a330 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.565+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800a3f0 con 0x7f0c94004d40 2024-03-28T19:07:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.565+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca8034630 con 0x7f0c94004d40 2024-03-28T19:07:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.565+0000 7f0cad600640 1 -- 172.21.15.27:0/2084443338 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8034a40 con 0x7f0c94004d40 2024-03-28T19:07:07.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.565+0000 7f0cafe00640 1 -- 172.21.15.27:0/2084443338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c94004d40 msgr2=0x7f0c940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.565+0000 7f0cafe00640 1 --2- 172.21.15.27:0/2084443338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c94004d40 0x7f0c940051c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800a7a0 tx=0x7f0c94005830 comp rx=0 tx=0).stop 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.565+0000 7f0caf400640 1 -- 172.21.15.27:0/2084443338 reap_dead start 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.566+0000 7f0cafe00640 1 -- 172.21.15.27:0/2084443338 shutdown_connections 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.566+0000 7f0cafe00640 1 -- 172.21.15.27:0/2084443338 >> 172.21.15.27:0/2084443338 conn(0x7f0cb006b6f0 msgr2=0x7f0cb00acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.566+0000 7f0cafe00640 1 -- 172.21.15.27:0/2084443338 shutdown_connections 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.566+0000 7f0cafe00640 1 -- 172.21.15.27:0/2084443338 wait complete. 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.568+0000 7f0cafe00640 1 Processor -- start 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.568+0000 7f0cafe00640 1 -- start start 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cafe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c94004d40 0x7f0cb01ec560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cafe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb0152450 0x7f0cb01546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cafe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c47a0 0x7f0cb00c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cafe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00b0820 con 0x7f0cb0152450 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cafe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0159750 con 0x7f0c94004d40 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cafe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00b09a0 con 0x7f0cb00c47a0 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cae000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb0152450 0x7f0cb01546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cae000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb0152450 0x7f0cb01546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48666/0 (socket says 172.21.15.27:48666) 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.569+0000 7f0cae000640 1 -- 172.21.15.27:0/1444613797 learned_addr learned my addr 172.21.15.27:0/1444613797 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0caea00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c94004d40 0x7f0cb01ec560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0caf400640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c47a0 0x7f0cb00c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0cae000640 1 -- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c47a0 msgr2=0x7f0cb00c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0cae000640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c47a0 0x7f0cb00c4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0cae000640 1 -- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c94004d40 msgr2=0x7f0cb01ec560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0cae000640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c94004d40 0x7f0cb01ec560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0cae000640 1 -- 172.21.15.27:0/1444613797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00c2ba0 con 0x7f0cb0152450 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0caf400640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c47a0 0x7f0cb00c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0cae000640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb0152450 0x7f0cb01546e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c010890 tx=0x7f0c9c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0ca6a00640 1 -- 172.21.15.27:0/1444613797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c00f040 con 0x7f0cb0152450 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.570+0000 7f0ca6a00640 1 -- 172.21.15.27:0/1444613797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c00eb60 con 0x7f0cb0152450 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.571+0000 7f0ca6a00640 1 -- 172.21.15.27:0/1444613797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c01c400 con 0x7f0cb0152450 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.571+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb00c2e80 con 0x7f0cb0152450 2024-03-28T19:07:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.571+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb01541c0 con 0x7f0cb0152450 2024-03-28T19:07:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.572+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb0069640 con 0x7f0cb0152450 2024-03-28T19:07:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.573+0000 7f0ca6a00640 1 -- 172.21.15.27:0/1444613797 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0c9c01c5a0 con 0x7f0cb0152450 2024-03-28T19:07:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.574+0000 7f0ca6a00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c9006bbd0 0x7f0c9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.574+0000 7f0ca6a00640 1 -- 172.21.15.27:0/1444613797 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f0c9c08b920 con 0x7f0cb0152450 2024-03-28T19:07:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.574+0000 7f0caea00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c9006bbd0 0x7f0c9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:07.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.575+0000 7f0caea00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c9006bbd0 0x7f0c9006e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009a10 tx=0x7f0ca0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:07.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.579+0000 7f0ca6a00640 1 -- 172.21.15.27:0/1444613797 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0c9c018030 con 0x7f0cb0152450 2024-03-28T19:07:07.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.849+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb01589b0 con 0x7f0cb0152450 2024-03-28T19:07:07.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.850+0000 7f0ca6a00640 1 -- 172.21.15.27:0/1444613797 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5460 (secure 0 0 0) 0x7f0c9c059bf0 con 0x7f0cb0152450 2024-03-28T19:07:07.850 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:07.850 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:04.840887+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:07:07.851 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 93 2024-03-28T19:07:07.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c9006bbd0 msgr2=0x7f0c9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c9006bbd0 0x7f0c9006e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009a10 tx=0x7f0ca0008040 comp rx=0 tx=0).stop 2024-03-28T19:07:07.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb0152450 msgr2=0x7f0cb01546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:07.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb0152450 0x7f0cb01546e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c010890 tx=0x7f0c9c010d50 comp rx=0 tx=0).stop 2024-03-28T19:07:07.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 shutdown_connections 2024-03-28T19:07:07.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c9006bbd0 0x7f0c9006e090 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cb00c47a0 0x7f0cb00c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cb0152450 0x7f0cb01546e0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 --2- 172.21.15.27:0/1444613797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c94004d40 0x7f0cb01ec560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 >> 172.21.15.27:0/1444613797 conn(0x7f0cb006b6f0 msgr2=0x7f0cb01dec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 shutdown_connections 2024-03-28T19:07:07.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:07.853+0000 7f0cafe00640 1 -- 172.21.15.27:0/1444613797 wait complete. 2024-03-28T19:07:07.871 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5336": { "addr": "172.21.15.155:6835/3884653650", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 3884653650, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 3884653650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5336, "incarnation": 87, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:06:20.279933+0000", "qdb_cluster": [ 5336 ], "qdb_leader": 5336, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5336 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6309": { "addr": "172.21.15.27:6836/1248101703", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1248101703, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 1248101703, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6309, "incarnation": 93, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:07:04.840887+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6309 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:07:09.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:10.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.096+0000 7fb7d3000640 1 Processor -- start 2024-03-28T19:07:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.096+0000 7fb7d3000640 1 -- start start 2024-03-28T19:07:10.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.097+0000 7fb7d3000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7cc0c3ea0 0x7fb7cc159610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.097+0000 7fb7d3000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc0c1600 con 0x7fb7cc0c60e0 2024-03-28T19:07:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.097+0000 7fb7d3000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc0c18e0 con 0x7fb7cc06ee00 2024-03-28T19:07:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.097+0000 7fb7d3000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7cc0c23b0 con 0x7fb7cc0c3ea0 2024-03-28T19:07:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.098+0000 7fb7d1000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7cc0c3ea0 0x7fb7cc159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.098+0000 7fb7cbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb7cc0c60e0 0x7fb7cc06e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53352/0 (socket says 172.21.15.27:53352) 2024-03-28T19:07:10.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.098+0000 7fb7cbe00640 1 -- 172.21.15.27:0/2839068292 learned_addr learned my addr 172.21.15.27:0/2839068292 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:07:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.098+0000 7fb7d1000640 1 -- 172.21.15.27:0/2839068292 >> v1:172.21.15.155:6789/0 conn(0x7fb7cc06ee00 legacy=0x7fb7cc158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.098+0000 7fb7d1000640 1 -- 172.21.15.27:0/2839068292 >> v1:172.21.15.27:6789/0 conn(0x7fb7cc0c60e0 legacy=0x7fb7cc06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.098+0000 7fb7d1000640 1 -- 172.21.15.27:0/2839068292 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0c1bc0 con 0x7fb7cc0c3ea0 2024-03-28T19:07:10.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.098+0000 7fb7caa00640 1 -- 172.21.15.27:0/2839068292 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854524395 0 0) 0x7fb7cc0c1600 con 0x7fb7cc0c60e0 2024-03-28T19:07:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.100+0000 7fb7d1000640 1 --2- 172.21.15.27:0/2839068292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc0c3ea0 0x7fb7cc159610 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0c2530 tx=0x7fb7c402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d74136a297dde9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.100+0000 7fb7caa00640 1 -- 172.21.15.27:0/2839068292 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4039070 con 0x7fb7cc0c3ea0 2024-03-28T19:07:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.100+0000 7fb7caa00640 1 -- 172.21.15.27:0/2839068292 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c402cc50 con 0x7fb7cc0c3ea0 2024-03-28T19:07:10.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.100+0000 7fb7caa00640 1 -- 172.21.15.27:0/2839068292 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4034440 con 0x7fb7cc0c3ea0 2024-03-28T19:07:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.100+0000 7fb7d3000640 1 -- 172.21.15.27:0/2839068292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc0c3ea0 msgr2=0x7fb7cc159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.101+0000 7fb7d3000640 1 --2- 172.21.15.27:0/2839068292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc0c3ea0 0x7fb7cc159610 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0c2530 tx=0x7fb7c402bea0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.101+0000 7fb7d3000640 1 -- 172.21.15.27:0/2839068292 shutdown_connections 2024-03-28T19:07:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.101+0000 7fb7d3000640 1 --2- 172.21.15.27:0/2839068292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc0c3ea0 0x7fb7cc159610 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.101+0000 7fb7d3000640 1 -- 172.21.15.27:0/2839068292 >> 172.21.15.27:0/2839068292 conn(0x7fb7cc06b9d0 msgr2=0x7fb7cc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.101+0000 7fb7d3000640 1 -- 172.21.15.27:0/2839068292 shutdown_connections 2024-03-28T19:07:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.101+0000 7fb7d3000640 1 -- 172.21.15.27:0/2839068292 wait complete. 2024-03-28T19:07:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.102+0000 7fb7d3000640 1 Processor -- start 2024-03-28T19:07:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.103+0000 7fb7d3000640 1 -- start start 2024-03-28T19:07:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7d3000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc06ee00 0x7fb7cc071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7d3000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7cc0c3ea0 0x7fb7cc153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7d3000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7cc0c60e0 0x7fb7cc153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7d3000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0c64c0 con 0x7fb7cc06ee00 2024-03-28T19:07:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7d3000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc159b50 con 0x7fb7cc0c3ea0 2024-03-28T19:07:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7d3000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc0c6640 con 0x7fb7cc0c60e0 2024-03-28T19:07:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7cbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc06ee00 0x7fb7cc071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7cbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc06ee00 0x7fb7cc071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48678/0 (socket says 172.21.15.27:48678) 2024-03-28T19:07:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7cbe00640 1 -- 172.21.15.27:0/2526048179 learned_addr learned my addr 172.21.15.27:0/2526048179 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7cb400640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7cc0c3ea0 0x7fb7cc153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.104+0000 7fb7d1000640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7cc0c60e0 0x7fb7cc153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7cbe00640 1 -- 172.21.15.27:0/2526048179 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7cc0c60e0 msgr2=0x7fb7cc153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7cbe00640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7cc0c60e0 0x7fb7cc153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7cbe00640 1 -- 172.21.15.27:0/2526048179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7cc0c3ea0 msgr2=0x7fb7cc153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7cbe00640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7cc0c3ea0 0x7fb7cc153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7cbe00640 1 -- 172.21.15.27:0/2526048179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc1eb2b0 con 0x7fb7cc06ee00 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7cbe00640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc06ee00 0x7fb7cc071d10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c420 tx=0x7fb7bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7c3e00640 1 -- 172.21.15.27:0/2526048179 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc0074e0 con 0x7fb7cc06ee00 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7c3e00640 1 -- 172.21.15.27:0/2526048179 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc0047e0 con 0x7fb7cc06ee00 2024-03-28T19:07:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7c3e00640 1 -- 172.21.15.27:0/2526048179 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc004c10 con 0x7fb7cc06ee00 2024-03-28T19:07:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.105+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc1eb570 con 0x7fb7cc06ee00 2024-03-28T19:07:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.106+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc1ebb30 con 0x7fb7cc06ee00 2024-03-28T19:07:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.107+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7cc073330 con 0x7fb7cc06ee00 2024-03-28T19:07:10.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.108+0000 7fb7c3e00640 1 -- 172.21.15.27:0/2526048179 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb7bc011070 con 0x7fb7cc06ee00 2024-03-28T19:07:10.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.109+0000 7fb7c3e00640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:10.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.110+0000 7fb7c3e00640 1 -- 172.21.15.27:0/2526048179 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7fb7bc08ae40 con 0x7fb7cc06ee00 2024-03-28T19:07:10.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.110+0000 7fb7cb400640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:10.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.111+0000 7fb7cb400640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc156ba0 tx=0x7fb7b8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:10.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.116+0000 7fb7c3e00640 1 -- 172.21.15.27:0/2526048179 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb7cc073330 con 0x7fb7cc06ee00 2024-03-28T19:07:10.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.392+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7cc0c1d90 con 0x7fb7cc06ee00 2024-03-28T19:07:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.393+0000 7fb7c3e00640 1 -- 172.21.15.27:0/2526048179 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5461 (secure 0 0 0) 0x7fb7cc0c1d90 con 0x7fb7cc06ee00 2024-03-28T19:07:10.393 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:10.393 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:08.507290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:07:10.393 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 95 2024-03-28T19:07:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.396+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ac06bbd0 msgr2=0x7fb7ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.396+0000 7fb7d3000640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc156ba0 tx=0x7fb7b8009040 comp rx=0 tx=0).stop 2024-03-28T19:07:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.396+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc06ee00 msgr2=0x7fb7cc071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.396+0000 7fb7d3000640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc06ee00 0x7fb7cc071d10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c420 tx=0x7fb7bc00c8e0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.397+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 shutdown_connections 2024-03-28T19:07:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.397+0000 7fb7d3000640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7ac06bbd0 0x7fb7ac06e090 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.397+0000 7fb7d3000640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7cc0c60e0 0x7fb7cc153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.397+0000 7fb7d3000640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7cc0c3ea0 0x7fb7cc153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.397+0000 7fb7d3000640 1 --2- 172.21.15.27:0/2526048179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7cc06ee00 0x7fb7cc071d10 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.397+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 >> 172.21.15.27:0/2526048179 conn(0x7fb7cc06b9d0 msgr2=0x7fb7cc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.397+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 shutdown_connections 2024-03-28T19:07:10.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:10.397+0000 7fb7d3000640 1 -- 172.21.15.27:0/2526048179 wait complete. 2024-03-28T19:07:10.416 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:07:11.337 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:07:11.338+0000 7f7bb4000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:07:12.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.663+0000 7f37b5e00640 1 Processor -- start 2024-03-28T19:07:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.664+0000 7f37b5e00640 1 -- start start 2024-03-28T19:07:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.665+0000 7f37b5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b0150c70 0x7f37b00a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.665+0000 7f37b5e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f37b00c3690 0x7f37b00c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.665+0000 7f37b5e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37b00c6660 0x7f37b01534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.665+0000 7f37b5e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b00ab4e0 con 0x7f37b00c6660 2024-03-28T19:07:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.665+0000 7f37b5e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b00ac1f0 con 0x7f37b00c3690 2024-03-28T19:07:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.665+0000 7f37b5e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b00c3af0 con 0x7f37b0150c70 2024-03-28T19:07:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37af400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37b00c6660 0x7f37b01534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37af400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37b00c6660 0x7f37b01534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48682/0 (socket says 172.21.15.27:48682) 2024-03-28T19:07:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37af400640 1 -- 172.21.15.27:0/1335992049 learned_addr learned my addr 172.21.15.27:0/1335992049 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37aea00640 1 --2- 172.21.15.27:0/1335992049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b0150c70 0x7f37b00a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37ae000640 1 --2- 172.21.15.27:0/1335992049 >> v2:172.21.15.155:3300/0 conn(0x7f37b00c3690 0x7f37b00c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37af400640 1 -- 172.21.15.27:0/1335992049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b0150c70 msgr2=0x7f37b00a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37af400640 1 --2- 172.21.15.27:0/1335992049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b0150c70 0x7f37b00a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37af400640 1 -- 172.21.15.27:0/1335992049 >> v2:172.21.15.155:3300/0 conn(0x7f37b00c3690 msgr2=0x7f37b00c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37af400640 1 --2- 172.21.15.27:0/1335992049 >> v2:172.21.15.155:3300/0 conn(0x7f37b00c3690 0x7f37b00c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37af400640 1 -- 172.21.15.27:0/1335992049 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b00c3d90 con 0x7f37b00c6660 2024-03-28T19:07:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.666+0000 7f37ae000640 1 --2- 172.21.15.27:0/1335992049 >> v2:172.21.15.155:3300/0 conn(0x7f37b00c3690 0x7f37b00c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:07:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.668+0000 7f37af400640 1 --2- 172.21.15.27:0/1335992049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b01534c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f37b006f180 tx=0x7f37a802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be0ac7cc73693da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.668+0000 7f37ad600640 1 -- 172.21.15.27:0/1335992049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a802c9e0 con 0x7f37b00c6660 2024-03-28T19:07:12.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.668+0000 7f37ad600640 1 -- 172.21.15.27:0/1335992049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a802cb80 con 0x7f37b00c6660 2024-03-28T19:07:12.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.668+0000 7f37ad600640 1 -- 172.21.15.27:0/1335992049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a80343f0 con 0x7f37b00c6660 2024-03-28T19:07:12.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 -- 172.21.15.27:0/1335992049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 msgr2=0x7f37b01534c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 --2- 172.21.15.27:0/1335992049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b01534c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f37b006f180 tx=0x7f37a802be40 comp rx=0 tx=0).stop 2024-03-28T19:07:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 -- 172.21.15.27:0/1335992049 shutdown_connections 2024-03-28T19:07:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 --2- 172.21.15.27:0/1335992049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b01534c0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 --2- 172.21.15.27:0/1335992049 >> v2:172.21.15.155:3300/0 conn(0x7f37b00c3690 0x7f37b00c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 --2- 172.21.15.27:0/1335992049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b0150c70 0x7f37b00a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 -- 172.21.15.27:0/1335992049 >> 172.21.15.27:0/1335992049 conn(0x7f37b006b9d0 msgr2=0x7f37b00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 -- 172.21.15.27:0/1335992049 shutdown_connections 2024-03-28T19:07:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.669+0000 7f37b5e00640 1 -- 172.21.15.27:0/1335992049 wait complete. 2024-03-28T19:07:12.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.672+0000 7f37b5e00640 1 Processor -- start 2024-03-28T19:07:12.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.672+0000 7f37b5e00640 1 -- start start 2024-03-28T19:07:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.673+0000 7f37b5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b00c3690 0x7f37b01eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.673+0000 7f37b5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b0154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.673+0000 7f37b5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37b0150c70 0x7f37b0157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.673+0000 7f37b5e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b00abcd0 con 0x7f37b00c6660 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.673+0000 7f37b5e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b00ab9f0 con 0x7f37b0150c70 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.673+0000 7f37b5e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b00c1810 con 0x7f37b00c3690 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37ae000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b0154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37aea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b00c3690 0x7f37b01eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37ae000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b0154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48690/0 (socket says 172.21.15.27:48690) 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37ae000640 1 -- 172.21.15.27:0/120630840 learned_addr learned my addr 172.21.15.27:0/120630840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37af400640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37b0150c70 0x7f37b0157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37ae000640 1 -- 172.21.15.27:0/120630840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b00c3690 msgr2=0x7f37b01eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:12.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37ae000640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b00c3690 0x7f37b01eb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37ae000640 1 -- 172.21.15.27:0/120630840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37b0150c70 msgr2=0x7f37b0157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37ae000640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37b0150c70 0x7f37b0157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.674+0000 7f37ae000640 1 -- 172.21.15.27:0/120630840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b0154d70 con 0x7f37b00c6660 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.675+0000 7f37ae000640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b0154830 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f379c00d7d0 tx=0x7f379c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.675+0000 7f37a7400640 1 -- 172.21.15.27:0/120630840 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c004030 con 0x7f37b00c6660 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.675+0000 7f37a7400640 1 -- 172.21.15.27:0/120630840 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c0041d0 con 0x7f37b00c6660 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.675+0000 7f37a7400640 1 -- 172.21.15.27:0/120630840 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c00b6c0 con 0x7f37b00c6660 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.675+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b006cf90 con 0x7f37b00c6660 2024-03-28T19:07:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.675+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b006d550 con 0x7f37b00c6660 2024-03-28T19:07:12.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.677+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b004f400 con 0x7f37b00c6660 2024-03-28T19:07:12.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.678+0000 7f37a7400640 1 -- 172.21.15.27:0/120630840 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f379c00b860 con 0x7f37b00c6660 2024-03-28T19:07:12.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.680+0000 7f37a7400640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f379006bc00 0x7f379006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:12.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.680+0000 7f37a7400640 1 -- 172.21.15.27:0/120630840 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f379c08b360 con 0x7f37b00c6660 2024-03-28T19:07:12.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.680+0000 7f37aea00640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f379006bc00 0x7f379006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:12.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.682+0000 7f37aea00640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f379006bc00 0x7f379006e0c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f37a0006fd0 tx=0x7f37a0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:12.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.685+0000 7f37a7400640 1 -- 172.21.15.27:0/120630840 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f379c058fe0 con 0x7f37b00c6660 2024-03-28T19:07:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.998+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37b00aa7d0 con 0x7f37b00c6660 2024-03-28T19:07:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:12.999+0000 7f37a7400640 1 -- 172.21.15.27:0/120630840 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5468 (secure 0 0 0) 0x7f379c05d080 con 0x7f37b00c6660 2024-03-28T19:07:12.998 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:12.998 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:12.998 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 96 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.002+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f379006bc00 msgr2=0x7f379006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.002+0000 7f37b5e00640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f379006bc00 0x7f379006e0c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f37a0006fd0 tx=0x7f37a0008040 comp rx=0 tx=0).stop 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.002+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 msgr2=0x7f37b0154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.002+0000 7f37b5e00640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b0154830 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f379c00d7d0 tx=0x7f379c00dc90 comp rx=0 tx=0).stop 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.003+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 shutdown_connections 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.003+0000 7f37b5e00640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f379006bc00 0x7f379006e0c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.003+0000 7f37b5e00640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37b0150c70 0x7f37b0157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.003+0000 7f37b5e00640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37b00c6660 0x7f37b0154830 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.003+0000 7f37b5e00640 1 --2- 172.21.15.27:0/120630840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f37b00c3690 0x7f37b01eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.003+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 >> 172.21.15.27:0/120630840 conn(0x7f37b006b9d0 msgr2=0x7f37b00c1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.003+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 shutdown_connections 2024-03-28T19:07:13.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.003+0000 7f37b5e00640 1 -- 172.21.15.27:0/120630840 wait complete. 2024-03-28T19:07:13.028 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6054 2024-03-28T19:07:13.029 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.c 2024-03-28T19:07:13.029 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-03-28T19:07:13.029 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:07:13.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:07:13.035 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:07:13.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:13.234 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:07:13.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.271+0000 7f54dbe00640 1 Processor -- start 2024-03-28T19:07:13.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.272+0000 7f54dbe00640 1 -- start start 2024-03-28T19:07:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.273+0000 7f54dbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54dc0c58c0 0x7f54dc0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.273+0000 7f54dbe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c8ed0 0x7f54dc1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:13.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.273+0000 7f54dbe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc156230 con 0x7f54dc0be5d0 2024-03-28T19:07:13.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.273+0000 7f54dbe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc158bd0 con 0x7f54dc0c58c0 2024-03-28T19:07:13.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.273+0000 7f54dbe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc158eb0 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.273+0000 7f54daa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54dc0c58c0 0x7f54dc0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.274+0000 7f54daa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54dc0c58c0 0x7f54dc0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48724/0 (socket says 172.21.15.27:48724) 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.274+0000 7f54daa00640 1 -- 172.21.15.27:0/2353337744 learned_addr learned my addr 172.21.15.27:0/2353337744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.274+0000 7f54da000640 1 --2- 172.21.15.27:0/2353337744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c8ed0 0x7f54dc1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.274+0000 7f54daa00640 1 -- 172.21.15.27:0/2353337744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c8ed0 msgr2=0x7f54dc1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.274+0000 7f54daa00640 1 --2- 172.21.15.27:0/2353337744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c8ed0 0x7f54dc1555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.274+0000 7f54daa00640 1 -- 172.21.15.27:0/2353337744 >> v1:172.21.15.155:6789/0 conn(0x7f54dc0be5d0 legacy=0x7f54dc155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.274+0000 7f54daa00640 1 -- 172.21.15.27:0/2353337744 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc159190 con 0x7f54dc0c58c0 2024-03-28T19:07:13.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.274+0000 7f54da000640 1 --2- 172.21.15.27:0/2353337744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c8ed0 0x7f54dc1555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:07:13.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.276+0000 7f54daa00640 1 --2- 172.21.15.27:0/2353337744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c58c0 0x7f54dc0c5ca0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f54cc009870 tx=0x7f54cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fbace0d2398cb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:13.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.276+0000 7f54d9600640 1 -- 172.21.15.27:0/2353337744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc035020 con 0x7f54dc0c58c0 2024-03-28T19:07:13.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.277+0000 7f54d9600640 1 -- 172.21.15.27:0/2353337744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc031640 con 0x7f54dc0c58c0 2024-03-28T19:07:13.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.277+0000 7f54d9600640 1 -- 172.21.15.27:0/2353337744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc02d040 con 0x7f54dc0c58c0 2024-03-28T19:07:13.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.277+0000 7f54dbe00640 1 -- 172.21.15.27:0/2353337744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c58c0 msgr2=0x7f54dc0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:13.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.277+0000 7f54dbe00640 1 --2- 172.21.15.27:0/2353337744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c58c0 0x7f54dc0c5ca0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f54cc009870 tx=0x7f54cc02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.278+0000 7f54dbe00640 1 -- 172.21.15.27:0/2353337744 shutdown_connections 2024-03-28T19:07:13.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.278+0000 7f54dbe00640 1 --2- 172.21.15.27:0/2353337744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c8ed0 0x7f54dc1555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.278+0000 7f54dbe00640 1 --2- 172.21.15.27:0/2353337744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c58c0 0x7f54dc0c5ca0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.278+0000 7f54dbe00640 1 -- 172.21.15.27:0/2353337744 >> 172.21.15.27:0/2353337744 conn(0x7f54dc06b6f0 msgr2=0x7f54dc0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:13.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.278+0000 7f54dbe00640 1 -- 172.21.15.27:0/2353337744 shutdown_connections 2024-03-28T19:07:13.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.278+0000 7f54dbe00640 1 -- 172.21.15.27:0/2353337744 wait complete. 2024-03-28T19:07:13.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.280+0000 7f54dbe00640 1 Processor -- start 2024-03-28T19:07:13.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.281+0000 7f54dbe00640 1 -- start start 2024-03-28T19:07:13.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54dbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54dc0be5d0 0x7f54dc0c1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:13.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54dbe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c58c0 0x7f54dc14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:13.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54dbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c8ed0 0x7f54dc150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:13.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54dbe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0b0820 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54dbe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc159750 con 0x7f54dc0be5d0 2024-03-28T19:07:13.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54dbe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0b09a0 con 0x7f54dc0c58c0 2024-03-28T19:07:13.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c8ed0 0x7f54dc150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:13.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54daa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54dc0be5d0 0x7f54dc0c1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:13.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c8ed0 0x7f54dc150290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48736/0 (socket says 172.21.15.27:48736) 2024-03-28T19:07:13.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.282+0000 7f54db400640 1 -- 172.21.15.27:0/1621824092 learned_addr learned my addr 172.21.15.27:0/1621824092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:13.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.283+0000 7f54da000640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c58c0 0x7f54dc14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:13.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.283+0000 7f54db400640 1 -- 172.21.15.27:0/1621824092 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c58c0 msgr2=0x7f54dc14fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:13.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.283+0000 7f54db400640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c58c0 0x7f54dc14fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.283+0000 7f54db400640 1 -- 172.21.15.27:0/1621824092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54dc0be5d0 msgr2=0x7f54dc0c1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:13.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.283+0000 7f54db400640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54dc0be5d0 0x7f54dc0c1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.283+0000 7f54db400640 1 -- 172.21.15.27:0/1621824092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0c0620 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.284+0000 7f54db400640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c8ed0 0x7f54dc150290 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f54d400c730 tx=0x7f54d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:13.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.284+0000 7f54d2a00640 1 -- 172.21.15.27:0/1621824092 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4016020 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.284+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc0c0900 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.285+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc0c0ec0 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.285+0000 7f54d2a00640 1 -- 172.21.15.27:0/1621824092 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4006d50 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.285+0000 7f54d2a00640 1 -- 172.21.15.27:0/1621824092 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4003bb0 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.286+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b4005290 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.288+0000 7f54d2a00640 1 -- 172.21.15.27:0/1621824092 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f54d40052e0 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.290+0000 7f54d2a00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f54bc06bbd0 0x7f54bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.290+0000 7f54d2a00640 1 -- 172.21.15.27:0/1621824092 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f54d408b050 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.290+0000 7f54daa00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f54bc06bbd0 0x7f54bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:13.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.291+0000 7f54daa00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f54bc06bbd0 0x7f54bc06e090 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f54cc036000 tx=0x7f54cc03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.295+0000 7f54d2a00640 1 -- 172.21.15.27:0/1621824092 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f54d4020020 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.644+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54b4005580 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.645+0000 7f54d2a00640 1 -- 172.21.15.27:0/1621824092 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f54d4059280 con 0x7f54dc0c8ed0 2024-03-28T19:07:13.646 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:13.646 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f54bc06bbd0 msgr2=0x7f54bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f54bc06bbd0 0x7f54bc06e090 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f54cc036000 tx=0x7f54cc03c040 comp rx=0 tx=0).stop 2024-03-28T19:07:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c8ed0 msgr2=0x7f54dc150290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c8ed0 0x7f54dc150290 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f54d400c730 tx=0x7f54d400cbf0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 shutdown_connections 2024-03-28T19:07:13.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f54bc06bbd0 0x7f54bc06e090 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54dc0c8ed0 0x7f54dc150290 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54dc0c58c0 0x7f54dc14fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.648+0000 7f54dbe00640 1 --2- 172.21.15.27:0/1621824092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54dc0be5d0 0x7f54dc0c1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.649+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 >> 172.21.15.27:0/1621824092 conn(0x7f54dc06b6f0 msgr2=0x7f54dc0ca200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.649+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 shutdown_connections 2024-03-28T19:07:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:13.649+0000 7f54dbe00640 1 -- 172.21.15.27:0/1621824092 wait complete. 2024-03-28T19:07:13.669 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-03-28T19:07:13.669 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before thrashing 2024-03-28T19:07:21.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.900+0000 7f506e400640 1 Processor -- start 2024-03-28T19:07:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.900+0000 7f506e400640 1 -- start start 2024-03-28T19:07:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.901+0000 7f506e400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5068150ae0 0x7f50680a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.901+0000 7f506e400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f50680a61a0 0x7f50680c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.902+0000 7f506e400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5068150ec0 con 0x7f50680c6ee0 2024-03-28T19:07:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.902+0000 7f506e400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506806dc70 con 0x7f5068150ae0 2024-03-28T19:07:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.902+0000 7f506e400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5068153550 con 0x7f50680a61a0 2024-03-28T19:07:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.902+0000 7f506d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5068150ae0 0x7f50680a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.902+0000 7f506d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5068150ae0 0x7f50680a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49016/0 (socket says 172.21.15.27:49016) 2024-03-28T19:07:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.902+0000 7f506d000640 1 -- 172.21.15.27:0/2683349418 learned_addr learned my addr 172.21.15.27:0/2683349418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.902+0000 7f5067e00640 1 --2- 172.21.15.27:0/2683349418 >> v2:172.21.15.155:3300/0 conn(0x7f50680a61a0 0x7f50680c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.903+0000 7f506d000640 1 -- 172.21.15.27:0/2683349418 >> v2:172.21.15.155:3300/0 conn(0x7f50680a61a0 msgr2=0x7f50680c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.903+0000 7f506d000640 1 --2- 172.21.15.27:0/2683349418 >> v2:172.21.15.155:3300/0 conn(0x7f50680a61a0 0x7f50680c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.903+0000 7f506d000640 1 -- 172.21.15.27:0/2683349418 >> v1:172.21.15.155:6789/0 conn(0x7f50680c6ee0 legacy=0x7f5068153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.903+0000 7f506d000640 1 -- 172.21.15.27:0/2683349418 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506806df10 con 0x7f5068150ae0 2024-03-28T19:07:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.904+0000 7f506d000640 1 --2- 172.21.15.27:0/2683349418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068150ae0 0x7f50680a5c60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f50680c44a0 tx=0x7f505c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661a575f7e54d115 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.904+0000 7f5067400640 1 -- 172.21.15.27:0/2683349418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c02e070 con 0x7f5068150ae0 2024-03-28T19:07:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.904+0000 7f5067400640 1 -- 172.21.15.27:0/2683349418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c037070 con 0x7f5068150ae0 2024-03-28T19:07:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.904+0000 7f5067400640 1 -- 172.21.15.27:0/2683349418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c031a00 con 0x7f5068150ae0 2024-03-28T19:07:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.905+0000 7f506e400640 1 -- 172.21.15.27:0/2683349418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068150ae0 msgr2=0x7f50680a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.905+0000 7f506e400640 1 --2- 172.21.15.27:0/2683349418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068150ae0 0x7f50680a5c60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f50680c44a0 tx=0x7f505c02b880 comp rx=0 tx=0).stop 2024-03-28T19:07:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.905+0000 7f506e400640 1 -- 172.21.15.27:0/2683349418 shutdown_connections 2024-03-28T19:07:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.905+0000 7f506e400640 1 --2- 172.21.15.27:0/2683349418 >> v2:172.21.15.155:3300/0 conn(0x7f50680a61a0 0x7f50680c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.905+0000 7f506e400640 1 --2- 172.21.15.27:0/2683349418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5068150ae0 0x7f50680a5c60 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.905+0000 7f506e400640 1 -- 172.21.15.27:0/2683349418 >> 172.21.15.27:0/2683349418 conn(0x7f506806b0c0 msgr2=0x7f506806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.905+0000 7f506e400640 1 -- 172.21.15.27:0/2683349418 shutdown_connections 2024-03-28T19:07:21.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.905+0000 7f506e400640 1 -- 172.21.15.27:0/2683349418 wait complete. 2024-03-28T19:07:21.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.907+0000 7f506e400640 1 Processor -- start 2024-03-28T19:07:21.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.908+0000 7f506e400640 1 -- start start 2024-03-28T19:07:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50680a61a0 0x7f50680ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506e400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50680c6ee0 0x7f50680a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:21.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506e400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5068150ae0 0x7f50680a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506e400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506806e7b0 con 0x7f50680a61a0 2024-03-28T19:07:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506e400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506806e4d0 con 0x7f5068150ae0 2024-03-28T19:07:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506e400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50681569d0 con 0x7f50680c6ee0 2024-03-28T19:07:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50680a61a0 0x7f50680ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50680a61a0 0x7f50680ab490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49022/0 (socket says 172.21.15.27:49022) 2024-03-28T19:07:21.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.909+0000 7f506d000640 1 -- 172.21.15.27:0/239959249 learned_addr learned my addr 172.21.15.27:0/239959249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.910+0000 7f5067e00640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50680c6ee0 0x7f50680a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.910+0000 7f506da00640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5068150ae0 0x7f50680a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.910+0000 7f506d000640 1 -- 172.21.15.27:0/239959249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50680c6ee0 msgr2=0x7f50680a57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.910+0000 7f506d000640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50680c6ee0 0x7f50680a57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.910+0000 7f506d000640 1 -- 172.21.15.27:0/239959249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5068150ae0 msgr2=0x7f50680a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.910+0000 7f506d000640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5068150ae0 0x7f50680a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:21.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.910+0000 7f506d000640 1 -- 172.21.15.27:0/239959249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50680aba90 con 0x7f50680a61a0 2024-03-28T19:07:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.910+0000 7f506d000640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50680a61a0 0x7f50680ab490 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f505c02b8c0 tx=0x7f505c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.911+0000 7f5064c00640 1 -- 172.21.15.27:0/239959249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c032880 con 0x7f50680a61a0 2024-03-28T19:07:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.911+0000 7f5064c00640 1 -- 172.21.15.27:0/239959249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c032a20 con 0x7f50680a61a0 2024-03-28T19:07:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.911+0000 7f5064c00640 1 -- 172.21.15.27:0/239959249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c032e10 con 0x7f50680a61a0 2024-03-28T19:07:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.911+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50681eaf40 con 0x7f50680a61a0 2024-03-28T19:07:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.911+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50681eb500 con 0x7f50680a61a0 2024-03-28T19:07:21.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.912+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5068067dd0 con 0x7f50680a61a0 2024-03-28T19:07:21.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.914+0000 7f5064c00640 1 -- 172.21.15.27:0/239959249 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f505c0453f0 con 0x7f50680a61a0 2024-03-28T19:07:21.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.916+0000 7f5064c00640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f504806bbd0 0x7f504806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:21.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.916+0000 7f5064c00640 1 -- 172.21.15.27:0/239959249 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f505c0ad3b0 con 0x7f50680a61a0 2024-03-28T19:07:21.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.916+0000 7f5067e00640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f504806bbd0 0x7f504806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:21.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.917+0000 7f5067e00640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f504806bbd0 0x7f504806e090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5068073800 tx=0x7f5054008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:21.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:21.922+0000 7f5064c00640 1 -- 172.21.15.27:0/239959249 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f505c040070 con 0x7f50680a61a0 2024-03-28T19:07:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.254+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5068069640 con 0x7f50680a61a0 2024-03-28T19:07:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.254+0000 7f5064c00640 1 -- 172.21.15.27:0/239959249 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f5068150ec0 con 0x7f50680a61a0 2024-03-28T19:07:22.255 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:22.255 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.257+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f504806bbd0 msgr2=0x7f504806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f504806bbd0 0x7f504806e090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5068073800 tx=0x7f5054008040 comp rx=0 tx=0).stop 2024-03-28T19:07:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50680a61a0 msgr2=0x7f50680ab490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50680a61a0 0x7f50680ab490 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f505c02b8c0 tx=0x7f505c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 shutdown_connections 2024-03-28T19:07:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f504806bbd0 0x7f504806e090 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5068150ae0 0x7f50680a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50680c6ee0 0x7f50680a57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 --2- 172.21.15.27:0/239959249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50680a61a0 0x7f50680ab490 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 >> 172.21.15.27:0/239959249 conn(0x7f506806b0c0 msgr2=0x7f5068154440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 shutdown_connections 2024-03-28T19:07:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.258+0000 7f506e400640 1 -- 172.21.15.27:0/239959249 wait complete. 2024-03-28T19:07:22.279 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-03-28T19:07:22.279 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:07:22.279 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:07:22.303 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:07:22.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.559+0000 7f779e400640 1 Processor -- start 2024-03-28T19:07:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.559+0000 7f779e400640 1 -- start start 2024-03-28T19:07:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f779e400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7798152d10 0x7f77980c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f779e400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77981562f0 0x7f779815ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f779e400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77981530f0 con 0x7f77980c8e10 2024-03-28T19:07:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f779e400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779806d3d0 con 0x7f77981562f0 2024-03-28T19:07:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f779e400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77980c92f0 con 0x7f7798152d10 2024-03-28T19:07:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f7797e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77981562f0 0x7f779815ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f7797e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77981562f0 0x7f779815ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49038/0 (socket says 172.21.15.27:49038) 2024-03-28T19:07:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f7797e00640 1 -- 172.21.15.27:0/565718882 learned_addr learned my addr 172.21.15.27:0/565718882 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f7796a00640 1 --1- 172.21.15.27:0/565718882 >> v1:172.21.15.27:6789/0 conn(0x7f77980c8e10 0x7f7798158be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44098/0 (socket says 172.21.15.27:44098) 2024-03-28T19:07:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.561+0000 7f7797400640 1 --2- 172.21.15.27:0/565718882 >> v2:172.21.15.155:3300/0 conn(0x7f7798152d10 0x7f77980c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.562+0000 7f7797e00640 1 -- 172.21.15.27:0/565718882 >> v2:172.21.15.155:3300/0 conn(0x7f7798152d10 msgr2=0x7f77980c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.562+0000 7f7797e00640 1 --2- 172.21.15.27:0/565718882 >> v2:172.21.15.155:3300/0 conn(0x7f7798152d10 0x7f77980c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.562+0000 7f7797e00640 1 -- 172.21.15.27:0/565718882 >> v1:172.21.15.27:6789/0 conn(0x7f77980c8e10 legacy=0x7f7798158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.562+0000 7f7797e00640 1 -- 172.21.15.27:0/565718882 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798155810 con 0x7f77981562f0 2024-03-28T19:07:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.563+0000 7f7797e00640 1 --2- 172.21.15.27:0/565718882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77981562f0 0x7f779815ff50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f779806d170 tx=0x7f779002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1855e1fcff3eaa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.564+0000 7f7796000640 1 -- 172.21.15.27:0/565718882 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779002c9f0 con 0x7f77981562f0 2024-03-28T19:07:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.564+0000 7f7796000640 1 -- 172.21.15.27:0/565718882 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779002cb90 con 0x7f77981562f0 2024-03-28T19:07:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.564+0000 7f7796000640 1 -- 172.21.15.27:0/565718882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77900343f0 con 0x7f77981562f0 2024-03-28T19:07:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.564+0000 7f779e400640 1 -- 172.21.15.27:0/565718882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77981562f0 msgr2=0x7f779815ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.564+0000 7f779e400640 1 --2- 172.21.15.27:0/565718882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77981562f0 0x7f779815ff50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f779806d170 tx=0x7f779002be50 comp rx=0 tx=0).stop 2024-03-28T19:07:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.565+0000 7f779e400640 1 -- 172.21.15.27:0/565718882 shutdown_connections 2024-03-28T19:07:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.565+0000 7f779e400640 1 --2- 172.21.15.27:0/565718882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77981562f0 0x7f779815ff50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.565+0000 7f779e400640 1 --2- 172.21.15.27:0/565718882 >> v2:172.21.15.155:3300/0 conn(0x7f7798152d10 0x7f77980c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.565+0000 7f779e400640 1 -- 172.21.15.27:0/565718882 >> 172.21.15.27:0/565718882 conn(0x7f7798066fb0 msgr2=0x7f77980673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.565+0000 7f779e400640 1 -- 172.21.15.27:0/565718882 shutdown_connections 2024-03-28T19:07:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.565+0000 7f779e400640 1 -- 172.21.15.27:0/565718882 wait complete. 2024-03-28T19:07:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.567+0000 7f779e400640 1 Processor -- start 2024-03-28T19:07:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.567+0000 7f779e400640 1 -- start start 2024-03-28T19:07:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f779e400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77980c8e10 0x7f77980c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f779e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798152d10 0x7f779814ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f779e400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77981562f0 0x7f7798157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f779e400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77981592f0 con 0x7f7798152d10 2024-03-28T19:07:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f779e400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779806d850 con 0x7f77980c8e10 2024-03-28T19:07:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f779e400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77981ebec0 con 0x7f77981562f0 2024-03-28T19:07:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f7796a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798152d10 0x7f779814ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f7796a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798152d10 0x7f779814ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49046/0 (socket says 172.21.15.27:49046) 2024-03-28T19:07:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f7796a00640 1 -- 172.21.15.27:0/1179764740 learned_addr learned my addr 172.21.15.27:0/1179764740 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f7797400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77980c8e10 0x7f77980c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.568+0000 7f7797e00640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77981562f0 0x7f7798157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f7796a00640 1 -- 172.21.15.27:0/1179764740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77981562f0 msgr2=0x7f7798157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f7796a00640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77981562f0 0x7f7798157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f7796a00640 1 -- 172.21.15.27:0/1179764740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77980c8e10 msgr2=0x7f77980c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f7796a00640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77980c8e10 0x7f77980c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f7796a00640 1 -- 172.21.15.27:0/1179764740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779814fb30 con 0x7f7798152d10 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f7796a00640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798152d10 0x7f779814ed90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f778400c3c0 tx=0x7f778400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f778f400640 1 -- 172.21.15.27:0/1179764740 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77840074e0 con 0x7f7798152d10 2024-03-28T19:07:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f778f400640 1 -- 172.21.15.27:0/1179764740 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7784007680 con 0x7f7798152d10 2024-03-28T19:07:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.569+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779814fdd0 con 0x7f7798152d10 2024-03-28T19:07:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.570+0000 7f778f400640 1 -- 172.21.15.27:0/1179764740 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784004a00 con 0x7f7798152d10 2024-03-28T19:07:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.570+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77981581e0 con 0x7f7798152d10 2024-03-28T19:07:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.571+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779804f400 con 0x7f7798152d10 2024-03-28T19:07:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.572+0000 7f778f400640 1 -- 172.21.15.27:0/1179764740 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f778401a020 con 0x7f7798152d10 2024-03-28T19:07:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.573+0000 7f778f400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f777806bbd0 0x7f777806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.574+0000 7f778f400640 1 -- 172.21.15.27:0/1179764740 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f778408ab30 con 0x7f7798152d10 2024-03-28T19:07:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.574+0000 7f7797400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f777806bbd0 0x7f777806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.575+0000 7f7797400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f777806bbd0 0x7f777806e090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7788009ad0 tx=0x7f7788008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:22.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.579+0000 7f778f400640 1 -- 172.21.15.27:0/1179764740 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f77981530f0 con 0x7f7798152d10 2024-03-28T19:07:22.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.833+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7798159ac0 con 0x7f7798152d10 2024-03-28T19:07:22.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.834+0000 7f778f400640 1 -- 172.21.15.27:0/1179764740 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f7784058d60 con 0x7f7798152d10 2024-03-28T19:07:22.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:22.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:22.835 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:22.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f777806bbd0 msgr2=0x7f777806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:22.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f777806bbd0 0x7f777806e090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7788009ad0 tx=0x7f7788008040 comp rx=0 tx=0).stop 2024-03-28T19:07:22.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798152d10 msgr2=0x7f779814ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:22.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798152d10 0x7f779814ed90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f778400c3c0 tx=0x7f778400c880 comp rx=0 tx=0).stop 2024-03-28T19:07:22.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 shutdown_connections 2024-03-28T19:07:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f777806bbd0 0x7f777806e090 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77981562f0 0x7f7798157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7798152d10 0x7f779814ed90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 --2- 172.21.15.27:0/1179764740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77980c8e10 0x7f77980c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 >> 172.21.15.27:0/1179764740 conn(0x7f7798066fb0 msgr2=0x7f7798154980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 shutdown_connections 2024-03-28T19:07:22.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:22.837+0000 7f779e400640 1 -- 172.21.15.27:0/1179764740 wait complete. 2024-03-28T19:07:22.859 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:07:24.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:25.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.102+0000 7f9b3a600640 1 Processor -- start 2024-03-28T19:07:25.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.102+0000 7f9b3a600640 1 -- start start 2024-03-28T19:07:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b3a600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9b34152f80 0x7f9b340c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b3a600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b340c7970 0x7f9b340c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b3a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b34155730 0x7f9b34155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b3a600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b340c7310 con 0x7f9b340c7970 2024-03-28T19:07:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b3a600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b34158d40 con 0x7f9b34152f80 2024-03-28T19:07:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b3a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b34159020 con 0x7f9b34155730 2024-03-28T19:07:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b32a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b340c7970 0x7f9b340c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b32a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b340c7970 0x7f9b340c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49054/0 (socket says 172.21.15.27:49054) 2024-03-28T19:07:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b32a00640 1 -- 172.21.15.27:0/4174495819 learned_addr learned my addr 172.21.15.27:0/4174495819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b33400640 1 --2- 172.21.15.27:0/4174495819 >> v2:172.21.15.155:3300/0 conn(0x7f9b34152f80 0x7f9b340c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.104+0000 7f9b33e00640 1 --2- 172.21.15.27:0/4174495819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b34155730 0x7f9b34155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.105+0000 7f9b32a00640 1 -- 172.21.15.27:0/4174495819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b34155730 msgr2=0x7f9b34155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.105+0000 7f9b32a00640 1 --2- 172.21.15.27:0/4174495819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b34155730 0x7f9b34155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.105+0000 7f9b32a00640 1 -- 172.21.15.27:0/4174495819 >> v2:172.21.15.155:3300/0 conn(0x7f9b34152f80 msgr2=0x7f9b340c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.105+0000 7f9b32a00640 1 --2- 172.21.15.27:0/4174495819 >> v2:172.21.15.155:3300/0 conn(0x7f9b34152f80 0x7f9b340c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.105+0000 7f9b32a00640 1 -- 172.21.15.27:0/4174495819 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b34159300 con 0x7f9b340c7970 2024-03-28T19:07:25.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.106+0000 7f9b32a00640 1 --2- 172.21.15.27:0/4174495819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b340c7970 0x7f9b340c7df0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9b3406c000 tx=0x7f9b2002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cbf6df004c432e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.106+0000 7f9b32000640 1 -- 172.21.15.27:0/4174495819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2002e070 con 0x7f9b340c7970 2024-03-28T19:07:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b32000640 1 -- 172.21.15.27:0/4174495819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b20037070 con 0x7f9b340c7970 2024-03-28T19:07:25.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b32000640 1 -- 172.21.15.27:0/4174495819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20031a00 con 0x7f9b340c7970 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b3a600640 1 -- 172.21.15.27:0/4174495819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b340c7970 msgr2=0x7f9b340c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b3a600640 1 --2- 172.21.15.27:0/4174495819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b340c7970 0x7f9b340c7df0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9b3406c000 tx=0x7f9b2002b880 comp rx=0 tx=0).stop 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b3a600640 1 -- 172.21.15.27:0/4174495819 shutdown_connections 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b3a600640 1 --2- 172.21.15.27:0/4174495819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b34155730 0x7f9b34155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b3a600640 1 --2- 172.21.15.27:0/4174495819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b340c7970 0x7f9b340c7df0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b3a600640 1 --2- 172.21.15.27:0/4174495819 >> v2:172.21.15.155:3300/0 conn(0x7f9b34152f80 0x7f9b340c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.107+0000 7f9b3a600640 1 -- 172.21.15.27:0/4174495819 >> 172.21.15.27:0/4174495819 conn(0x7f9b3406b6f0 msgr2=0x7f9b340caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.108+0000 7f9b3a600640 1 -- 172.21.15.27:0/4174495819 shutdown_connections 2024-03-28T19:07:25.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.108+0000 7f9b3a600640 1 -- 172.21.15.27:0/4174495819 wait complete. 2024-03-28T19:07:25.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.109+0000 7f9b3a600640 1 Processor -- start 2024-03-28T19:07:25.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.110+0000 7f9b3a600640 1 -- start start 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b3a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b340c7970 0x7f9b3414b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b3a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34152f80 0x7f9b34141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b3a600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b34155730 0x7f9b34141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b3a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b34159b00 con 0x7f9b34152f80 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b3a600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340c5e20 con 0x7f9b34155730 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b3a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340cbca0 con 0x7f9b340c7970 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b32a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34152f80 0x7f9b34141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b32a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34152f80 0x7f9b34141040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49066/0 (socket says 172.21.15.27:49066) 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b32a00640 1 -- 172.21.15.27:0/184194462 learned_addr learned my addr 172.21.15.27:0/184194462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b33e00640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b34155730 0x7f9b34141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:25.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.111+0000 7f9b33400640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b340c7970 0x7f9b3414b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:25.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.112+0000 7f9b32a00640 1 -- 172.21.15.27:0/184194462 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b340c7970 msgr2=0x7f9b3414b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:25.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.112+0000 7f9b32a00640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b340c7970 0x7f9b3414b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.112+0000 7f9b32a00640 1 -- 172.21.15.27:0/184194462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b34155730 msgr2=0x7f9b34141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:25.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.112+0000 7f9b32a00640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b34155730 0x7f9b34141580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.112+0000 7f9b32a00640 1 -- 172.21.15.27:0/184194462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b34141ac0 con 0x7f9b34152f80 2024-03-28T19:07:25.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.112+0000 7f9b32a00640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34152f80 0x7f9b34141040 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9b200097c0 tx=0x7f9b200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:25.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.113+0000 7f9b2b400640 1 -- 172.21.15.27:0/184194462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20032670 con 0x7f9b34152f80 2024-03-28T19:07:25.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.113+0000 7f9b2b400640 1 -- 172.21.15.27:0/184194462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b20032810 con 0x7f9b34152f80 2024-03-28T19:07:25.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.113+0000 7f9b2b400640 1 -- 172.21.15.27:0/184194462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20032c00 con 0x7f9b34152f80 2024-03-28T19:07:25.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.113+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b341eb810 con 0x7f9b34152f80 2024-03-28T19:07:25.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.113+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b341ebdb0 con 0x7f9b34152f80 2024-03-28T19:07:25.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.114+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b340cb5b0 con 0x7f9b34152f80 2024-03-28T19:07:25.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.115+0000 7f9b2b400640 1 -- 172.21.15.27:0/184194462 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9b2004d3f0 con 0x7f9b34152f80 2024-03-28T19:07:25.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.116+0000 7f9b2b400640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b1406bbd0 0x7f9b1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:25.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.116+0000 7f9b2b400640 1 -- 172.21.15.27:0/184194462 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f9b200acdb0 con 0x7f9b34152f80 2024-03-28T19:07:25.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.117+0000 7f9b33400640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b1406bbd0 0x7f9b1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:25.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.117+0000 7f9b33400640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b1406bbd0 0x7f9b1406e090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9b24006fd0 tx=0x7f9b24008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.122+0000 7f9b2b400640 1 -- 172.21.15.27:0/184194462 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9b340cb5b0 con 0x7f9b34152f80 2024-03-28T19:07:25.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.378+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b3404f400 con 0x7f9b34152f80 2024-03-28T19:07:25.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.379+0000 7f9b2b400640 1 -- 172.21.15.27:0/184194462 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f9b20040070 con 0x7f9b34152f80 2024-03-28T19:07:25.378 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:25.378 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:25.379 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:25.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.381+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b1406bbd0 msgr2=0x7f9b1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:25.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.381+0000 7f9b3a600640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b1406bbd0 0x7f9b1406e090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9b24006fd0 tx=0x7f9b24008040 comp rx=0 tx=0).stop 2024-03-28T19:07:25.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.381+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34152f80 msgr2=0x7f9b34141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:25.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.381+0000 7f9b3a600640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34152f80 0x7f9b34141040 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9b200097c0 tx=0x7f9b200046c0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.382+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 shutdown_connections 2024-03-28T19:07:25.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.382+0000 7f9b3a600640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b1406bbd0 0x7f9b1406e090 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.382+0000 7f9b3a600640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b34155730 0x7f9b34141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.382+0000 7f9b3a600640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b34152f80 0x7f9b34141040 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.382+0000 7f9b3a600640 1 --2- 172.21.15.27:0/184194462 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b340c7970 0x7f9b3414b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.382+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 >> 172.21.15.27:0/184194462 conn(0x7f9b3406b6f0 msgr2=0x7f9b341548e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.382+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 shutdown_connections 2024-03-28T19:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:25.382+0000 7f9b3a600640 1 -- 172.21.15.27:0/184194462 wait complete. 2024-03-28T19:07:25.400 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:07:27.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:27.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.647+0000 7fb899000640 1 Processor -- start 2024-03-28T19:07:27.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.647+0000 7fb899000640 1 -- start start 2024-03-28T19:07:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.648+0000 7fb899000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb8940c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.648+0000 7fb899000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb894071240 0x7fb8941591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.648+0000 7fb899000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89406b2f0 con 0x7fb8940c7610 2024-03-28T19:07:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.648+0000 7fb899000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8940c4340 con 0x7fb894071240 2024-03-28T19:07:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.648+0000 7fb899000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940c4620 con 0x7fb8940c3970 2024-03-28T19:07:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.649+0000 7fb891e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb8940c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.649+0000 7fb891e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb8940c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:58712/0 (socket says 172.21.15.27:58712) 2024-03-28T19:07:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.649+0000 7fb892800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb894071240 0x7fb8941591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.649+0000 7fb891e00640 1 -- 172.21.15.27:0/170495019 learned_addr learned my addr 172.21.15.27:0/170495019 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.650+0000 7fb891e00640 1 -- 172.21.15.27:0/170495019 >> v2:172.21.15.155:3300/0 conn(0x7fb894071240 msgr2=0x7fb8941591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.650+0000 7fb891e00640 1 --2- 172.21.15.27:0/170495019 >> v2:172.21.15.155:3300/0 conn(0x7fb894071240 0x7fb8941591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.650+0000 7fb891e00640 1 -- 172.21.15.27:0/170495019 >> v1:172.21.15.155:6789/0 conn(0x7fb8940c7610 legacy=0x7fb894070b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.650+0000 7fb891e00640 1 -- 172.21.15.27:0/170495019 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8940c4900 con 0x7fb8940c3970 2024-03-28T19:07:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.652+0000 7fb891e00640 1 --2- 172.21.15.27:0/170495019 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb8940c50d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb884009870 tx=0x7fb88402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62226fa77d18a9d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.652+0000 7fb891400640 1 -- 172.21.15.27:0/170495019 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb884034070 con 0x7fb8940c3970 2024-03-28T19:07:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.652+0000 7fb891400640 1 -- 172.21.15.27:0/170495019 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb884030840 con 0x7fb8940c3970 2024-03-28T19:07:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.652+0000 7fb891400640 1 -- 172.21.15.27:0/170495019 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb884030c10 con 0x7fb8940c3970 2024-03-28T19:07:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.653+0000 7fb899000640 1 -- 172.21.15.27:0/170495019 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 msgr2=0x7fb8940c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.653+0000 7fb899000640 1 --2- 172.21.15.27:0/170495019 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb8940c50d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb884009870 tx=0x7fb88402b980 comp rx=0 tx=0).stop 2024-03-28T19:07:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.653+0000 7fb899000640 1 -- 172.21.15.27:0/170495019 shutdown_connections 2024-03-28T19:07:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.653+0000 7fb899000640 1 --2- 172.21.15.27:0/170495019 >> v2:172.21.15.155:3300/0 conn(0x7fb894071240 0x7fb8941591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.653+0000 7fb899000640 1 --2- 172.21.15.27:0/170495019 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb8940c50d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.653+0000 7fb899000640 1 -- 172.21.15.27:0/170495019 >> 172.21.15.27:0/170495019 conn(0x7fb8940aaab0 msgr2=0x7fb8940aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.653+0000 7fb899000640 1 -- 172.21.15.27:0/170495019 shutdown_connections 2024-03-28T19:07:27.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.653+0000 7fb899000640 1 -- 172.21.15.27:0/170495019 wait complete. 2024-03-28T19:07:27.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.655+0000 7fb899000640 1 Processor -- start 2024-03-28T19:07:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.655+0000 7fb899000640 1 -- start start 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.656+0000 7fb899000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb894071240 0x7fb8941551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.656+0000 7fb899000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb89409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.656+0000 7fb899000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8940c7610 0x7fb89409b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.656+0000 7fb899000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940c3ed0 con 0x7fb8940c7610 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.656+0000 7fb899000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940c3d50 con 0x7fb894071240 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.656+0000 7fb899000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8941596e0 con 0x7fb8940c3970 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb892800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8940c7610 0x7fb89409b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb892800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8940c7610 0x7fb89409b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38138/0 (socket says 172.21.15.27:38138) 2024-03-28T19:07:27.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb892800640 1 -- 172.21.15.27:0/2690652425 learned_addr learned my addr 172.21.15.27:0/2690652425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:27.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb88be00640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb89409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:27.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb891e00640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb894071240 0x7fb8941551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb892800640 1 -- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 msgr2=0x7fb89409b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb892800640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb89409b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb892800640 1 -- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb894071240 msgr2=0x7fb8941551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb892800640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb894071240 0x7fb8941551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb892800640 1 -- 172.21.15.27:0/2690652425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8941e2270 con 0x7fb8940c7610 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.657+0000 7fb88be00640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb89409b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.658+0000 7fb892800640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8940c7610 0x7fb89409b9f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00efc0 tx=0x7fb88c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.658+0000 7fb88a000640 1 -- 172.21.15.27:0/2690652425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c018020 con 0x7fb8940c7610 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.658+0000 7fb88a000640 1 -- 172.21.15.27:0/2690652425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c014480 con 0x7fb8940c7610 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.658+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8941e2550 con 0x7fb8940c7610 2024-03-28T19:07:27.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.658+0000 7fb88a000640 1 -- 172.21.15.27:0/2690652425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c014870 con 0x7fb8940c7610 2024-03-28T19:07:27.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.658+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89409af90 con 0x7fb8940c7610 2024-03-28T19:07:27.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.659+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89409b190 con 0x7fb8940c7610 2024-03-28T19:07:27.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.661+0000 7fb88a000640 1 -- 172.21.15.27:0/2690652425 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb88c005d10 con 0x7fb8940c7610 2024-03-28T19:07:27.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.662+0000 7fb88a000640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb87006bc00 0x7fb87006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.662+0000 7fb88a000640 1 -- 172.21.15.27:0/2690652425 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7fb88c08c260 con 0x7fb8940c7610 2024-03-28T19:07:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.663+0000 7fb891e00640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb87006bc00 0x7fb87006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:27.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.663+0000 7fb891e00640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb87006bc00 0x7fb87006e0c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb884036000 tx=0x7fb88403b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:27.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.668+0000 7fb88a000640 1 -- 172.21.15.27:0/2690652425 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb89409b190 con 0x7fb8940c7610 2024-03-28T19:07:27.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.957+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb89404f400 con 0x7fb8940c7610 2024-03-28T19:07:27.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.958+0000 7fb88a000640 1 -- 172.21.15.27:0/2690652425 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fb88c022030 con 0x7fb8940c7610 2024-03-28T19:07:27.957 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:27.957 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:27.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.960+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb87006bc00 msgr2=0x7fb87006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:27.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.960+0000 7fb899000640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb87006bc00 0x7fb87006e0c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb884036000 tx=0x7fb88403b040 comp rx=0 tx=0).stop 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.960+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8940c7610 msgr2=0x7fb89409b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.960+0000 7fb899000640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8940c7610 0x7fb89409b9f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00efc0 tx=0x7fb88c00c490 comp rx=0 tx=0).stop 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.961+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 shutdown_connections 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.961+0000 7fb899000640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb87006bc00 0x7fb87006e0c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.961+0000 7fb899000640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8940c7610 0x7fb89409b9f0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.961+0000 7fb899000640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8940c3970 0x7fb89409b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.961+0000 7fb899000640 1 --2- 172.21.15.27:0/2690652425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb894071240 0x7fb8941551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.961+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 >> 172.21.15.27:0/2690652425 conn(0x7fb8940aaab0 msgr2=0x7fb89406c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.961+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 shutdown_connections 2024-03-28T19:07:27.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:27.961+0000 7fb899000640 1 -- 172.21.15.27:0/2690652425 wait complete. 2024-03-28T19:07:27.985 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:07:29.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:30.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.218+0000 7fea9ca00640 1 Processor -- start 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.218+0000 7fea9ca00640 1 -- start start 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.219+0000 7fea9ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea980aa300 0x7fea9816dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.219+0000 7fea9ca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea980ac150 con 0x7fea980c6760 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.219+0000 7fea9ca00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea980c12d0 con 0x7fea981662a0 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.219+0000 7fea9ca00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980733f0 con 0x7fea980aa300 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.220+0000 7fea96800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea980aa300 0x7fea9816dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.220+0000 7fea96800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea980aa300 0x7fea9816dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38154/0 (socket says 172.21.15.27:38154) 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.220+0000 7fea96800640 1 -- 172.21.15.27:0/4118390846 learned_addr learned my addr 172.21.15.27:0/4118390846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.220+0000 7fea94a00640 1 -- 172.21.15.27:0/4118390846 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278706532 0 0) 0x7fea84001110 con 0x7fea980c6760 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.435+0000 7fea94a00640 1 -- 172.21.15.27:0/4118390846 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c003500 con 0x7fea980c6760 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.435+0000 7fea94a00640 1 -- 172.21.15.27:0/4118390846 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844369119 0 0) 0x7fea980c12d0 con 0x7fea981662a0 2024-03-28T19:07:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.435+0000 7fea94a00640 1 -- 172.21.15.27:0/4118390846 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea84001110 con 0x7fea981662a0 2024-03-28T19:07:30.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.436+0000 7fea96800640 1 -- 172.21.15.27:0/4118390846 >> v1:172.21.15.155:6789/0 conn(0x7fea981662a0 legacy=0x7fea980a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:30.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.436+0000 7fea96800640 1 -- 172.21.15.27:0/4118390846 >> v1:172.21.15.27:6789/0 conn(0x7fea980c6760 legacy=0x7fea980c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:30.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.436+0000 7fea96800640 1 -- 172.21.15.27:0/4118390846 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98073640 con 0x7fea980aa300 2024-03-28T19:07:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.439+0000 7fea96800640 1 --2- 172.21.15.27:0/4118390846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980aa300 0x7fea9816dc80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fea90007730 tx=0x7fea9002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4004185899cf681 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.439+0000 7fea94a00640 1 -- 172.21.15.27:0/4118390846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9002cd10 con 0x7fea980aa300 2024-03-28T19:07:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.439+0000 7fea94a00640 1 -- 172.21.15.27:0/4118390846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea90003990 con 0x7fea980aa300 2024-03-28T19:07:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.440+0000 7fea94a00640 1 -- 172.21.15.27:0/4118390846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea90003d40 con 0x7fea980aa300 2024-03-28T19:07:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.441+0000 7fea9ca00640 1 -- 172.21.15.27:0/4118390846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980aa300 msgr2=0x7fea9816dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.441+0000 7fea9ca00640 1 --2- 172.21.15.27:0/4118390846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980aa300 0x7fea9816dc80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fea90007730 tx=0x7fea9002bf60 comp rx=0 tx=0).stop 2024-03-28T19:07:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.441+0000 7fea9ca00640 1 -- 172.21.15.27:0/4118390846 shutdown_connections 2024-03-28T19:07:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.441+0000 7fea9ca00640 1 --2- 172.21.15.27:0/4118390846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea980aa300 0x7fea9816dc80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.441+0000 7fea9ca00640 1 -- 172.21.15.27:0/4118390846 >> 172.21.15.27:0/4118390846 conn(0x7fea980c6c00 msgr2=0x7fea980c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.442+0000 7fea9ca00640 1 -- 172.21.15.27:0/4118390846 shutdown_connections 2024-03-28T19:07:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.442+0000 7fea9ca00640 1 -- 172.21.15.27:0/4118390846 wait complete. 2024-03-28T19:07:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.444+0000 7fea9ca00640 1 Processor -- start 2024-03-28T19:07:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.445+0000 7fea9ca00640 1 -- start start 2024-03-28T19:07:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea9ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea980aa300 0x7fea980beaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea9ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c6760 0x7fea9808b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea9ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981662a0 0x7fea980c0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea9ca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea980abb00 con 0x7fea981662a0 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea9ca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea980a9de0 con 0x7fea980c6760 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea9ca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea980a6370 con 0x7fea980aa300 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea96800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981662a0 0x7fea980c0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea96800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981662a0 0x7fea980c0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38170/0 (socket says 172.21.15.27:38170) 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea95e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea980aa300 0x7fea980beaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:30.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.446+0000 7fea96800640 1 -- 172.21.15.27:0/1729254619 learned_addr learned my addr 172.21.15.27:0/1729254619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:30.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.447+0000 7fea95400640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c6760 0x7fea9808b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:30.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.447+0000 7fea95e00640 1 -- 172.21.15.27:0/1729254619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c6760 msgr2=0x7fea9808b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:30.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.447+0000 7fea95e00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c6760 0x7fea9808b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.448+0000 7fea95e00640 1 -- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981662a0 msgr2=0x7fea980c0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.448+0000 7fea95e00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981662a0 0x7fea980c0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.448+0000 7fea95e00640 1 -- 172.21.15.27:0/1729254619 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980befe0 con 0x7fea980aa300 2024-03-28T19:07:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.448+0000 7fea96800640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981662a0 0x7fea980c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:07:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.448+0000 7fea95e00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea980aa300 0x7fea980beaa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fea8800c3f0 tx=0x7fea8800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:30.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.449+0000 7fea8e000640 1 -- 172.21.15.27:0/1729254619 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88017070 con 0x7fea980aa300 2024-03-28T19:07:30.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.449+0000 7fea8e000640 1 -- 172.21.15.27:0/1729254619 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea880052f0 con 0x7fea980aa300 2024-03-28T19:07:30.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.449+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea980a6760 con 0x7fea980aa300 2024-03-28T19:07:30.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.449+0000 7fea8e000640 1 -- 172.21.15.27:0/1729254619 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88003830 con 0x7fea980aa300 2024-03-28T19:07:30.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.449+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea980a6a20 con 0x7fea980aa300 2024-03-28T19:07:30.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.451+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea980a97e0 con 0x7fea980aa300 2024-03-28T19:07:30.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.453+0000 7fea8e000640 1 -- 172.21.15.27:0/1729254619 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fea8801a070 con 0x7fea980aa300 2024-03-28T19:07:30.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.454+0000 7fea8e000640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7806bbd0 0x7fea7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:30.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.454+0000 7fea8e000640 1 -- 172.21.15.27:0/1729254619 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7fea8808ac40 con 0x7fea980aa300 2024-03-28T19:07:30.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.454+0000 7fea95400640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7806bbd0 0x7fea7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:30.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.455+0000 7fea95400640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7806bbd0 0x7fea7806e090 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fea980bbe20 tx=0x7fea84009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:30.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.466+0000 7fea8e000640 1 -- 172.21.15.27:0/1729254619 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fea980a97e0 con 0x7fea980aa300 2024-03-28T19:07:30.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:07:30.736 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:07:30.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.767+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea9804f400 con 0x7fea980aa300 2024-03-28T19:07:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.769+0000 7fea8e000640 1 -- 172.21.15.27:0/1729254619 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7fea88021070 con 0x7fea980aa300 2024-03-28T19:07:30.768 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:30.768 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7806bbd0 msgr2=0x7fea7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7806bbd0 0x7fea7806e090 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fea980bbe20 tx=0x7fea84009040 comp rx=0 tx=0).stop 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea980aa300 msgr2=0x7fea980beaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea980aa300 0x7fea980beaa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fea8800c3f0 tx=0x7fea8800c8b0 comp rx=0 tx=0).stop 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 shutdown_connections 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea7806bbd0 0x7fea7806e090 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea981662a0 0x7fea980c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea980c6760 0x7fea9808b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 --2- 172.21.15.27:0/1729254619 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea980aa300 0x7fea980beaa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.771+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 >> 172.21.15.27:0/1729254619 conn(0x7fea980c6c00 msgr2=0x7fea980c55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:30.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.772+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 shutdown_connections 2024-03-28T19:07:30.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:30.772+0000 7fea9ca00640 1 -- 172.21.15.27:0/1729254619 wait complete. 2024-03-28T19:07:30.790 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:07:32.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.030+0000 7f11c9400640 1 Processor -- start 2024-03-28T19:07:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.030+0000 7f11c9400640 1 -- start start 2024-03-28T19:07:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.032+0000 7f11c9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11c4152da0 0x7f11c40c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.032+0000 7f11c9400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c406b160 con 0x7f11c40c2e40 2024-03-28T19:07:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.032+0000 7f11c9400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c4156450 con 0x7f11c40732e0 2024-03-28T19:07:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.032+0000 7f11c9400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c40c2100 con 0x7f11c4152da0 2024-03-28T19:07:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.032+0000 7f11c2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11c4152da0 0x7f11c40c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.032+0000 7f11c2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f11c4152da0 0x7f11c40c2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38184/0 (socket says 172.21.15.27:38184) 2024-03-28T19:07:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.032+0000 7f11c2800640 1 -- 172.21.15.27:0/3537623861 learned_addr learned my addr 172.21.15.27:0/3537623861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.033+0000 7f11c2800640 1 -- 172.21.15.27:0/3537623861 >> v1:172.21.15.155:6789/0 conn(0x7f11c40732e0 legacy=0x7f11c415ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.033+0000 7f11c2800640 1 -- 172.21.15.27:0/3537623861 >> v1:172.21.15.27:6789/0 conn(0x7f11c40c2e40 legacy=0x7f11c406b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.033+0000 7f11c2800640 1 -- 172.21.15.27:0/3537623861 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40c2350 con 0x7f11c4152da0 2024-03-28T19:07:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.034+0000 7f11c2800640 1 --2- 172.21.15.27:0/3537623861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c4152da0 0x7f11c40c2900 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009870 tx=0x7f11b402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f368493aba7e70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.034+0000 7f11c1400640 1 -- 172.21.15.27:0/3537623861 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b40317d0 con 0x7f11c4152da0 2024-03-28T19:07:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.034+0000 7f11c1400640 1 -- 172.21.15.27:0/3537623861 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b4037070 con 0x7f11c4152da0 2024-03-28T19:07:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.034+0000 7f11c1400640 1 -- 172.21.15.27:0/3537623861 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4031b90 con 0x7f11c4152da0 2024-03-28T19:07:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.035+0000 7f11c9400640 1 -- 172.21.15.27:0/3537623861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c4152da0 msgr2=0x7f11c40c2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.035+0000 7f11c9400640 1 --2- 172.21.15.27:0/3537623861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c4152da0 0x7f11c40c2900 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009870 tx=0x7f11b402b870 comp rx=0 tx=0).stop 2024-03-28T19:07:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.035+0000 7f11c9400640 1 -- 172.21.15.27:0/3537623861 shutdown_connections 2024-03-28T19:07:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.035+0000 7f11c9400640 1 --2- 172.21.15.27:0/3537623861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c4152da0 0x7f11c40c2900 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.035+0000 7f11c9400640 1 -- 172.21.15.27:0/3537623861 >> 172.21.15.27:0/3537623861 conn(0x7f11c40c3ff0 msgr2=0x7f11c40c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.035+0000 7f11c9400640 1 -- 172.21.15.27:0/3537623861 shutdown_connections 2024-03-28T19:07:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.035+0000 7f11c9400640 1 -- 172.21.15.27:0/3537623861 wait complete. 2024-03-28T19:07:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.037+0000 7f11c9400640 1 Processor -- start 2024-03-28T19:07:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.037+0000 7f11c9400640 1 -- start start 2024-03-28T19:07:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40732e0 0x7f11c40c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c40c2e40 0x7f11c40c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c9400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c4152da0 0x7f11c41448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c9400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4067130 con 0x7f11c40732e0 2024-03-28T19:07:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c9400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c41560e0 con 0x7f11c40c2e40 2024-03-28T19:07:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c9400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40672b0 con 0x7f11c4152da0 2024-03-28T19:07:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40732e0 0x7f11c40c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40732e0 0x7f11c40c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38186/0 (socket says 172.21.15.27:38186) 2024-03-28T19:07:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c2800640 1 -- 172.21.15.27:0/2113304088 learned_addr learned my addr 172.21.15.27:0/2113304088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c3200640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c4152da0 0x7f11c41448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c1e00640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c40c2e40 0x7f11c40c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c2800640 1 -- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c4152da0 msgr2=0x7f11c41448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c2800640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c4152da0 0x7f11c41448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c2800640 1 -- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c40c2e40 msgr2=0x7f11c40c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c2800640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c40c2e40 0x7f11c40c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:33.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.039+0000 7f11c2800640 1 -- 172.21.15.27:0/2113304088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c4157300 con 0x7f11c40732e0 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.040+0000 7f11c3200640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c4152da0 0x7f11c41448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.040+0000 7f11c1e00640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c40c2e40 0x7f11c40c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.040+0000 7f11c2800640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40732e0 0x7f11c40c60b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009870 tx=0x7f11b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.040+0000 7f11baa00640 1 -- 172.21.15.27:0/2113304088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4032870 con 0x7f11c40732e0 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.040+0000 7f11baa00640 1 -- 172.21.15.27:0/2113304088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b4032a30 con 0x7f11c40732e0 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.040+0000 7f11baa00640 1 -- 172.21.15.27:0/2113304088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4032e20 con 0x7f11c40732e0 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.040+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c4156d40 con 0x7f11c40732e0 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.041+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c40c5b90 con 0x7f11c40732e0 2024-03-28T19:07:33.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.042+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c404f400 con 0x7f11c40732e0 2024-03-28T19:07:33.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.043+0000 7f11baa00640 1 -- 172.21.15.27:0/2113304088 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11b404c060 con 0x7f11c40732e0 2024-03-28T19:07:33.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.044+0000 7f11baa00640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11a406bbd0 0x7f11a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.044+0000 7f11baa00640 1 -- 172.21.15.27:0/2113304088 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f11b40acc80 con 0x7f11c40732e0 2024-03-28T19:07:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.045+0000 7f11c1e00640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11a406bbd0 0x7f11a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:33.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.046+0000 7f11c1e00640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11a406bbd0 0x7f11a406e090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f11c41520a0 tx=0x7f11b0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:33.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.049+0000 7f11baa00640 1 -- 172.21.15.27:0/2113304088 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11b40b2020 con 0x7f11c40732e0 2024-03-28T19:07:33.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.349+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11c4066f10 con 0x7f11c40732e0 2024-03-28T19:07:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.351+0000 7f11baa00640 1 -- 172.21.15.27:0/2113304088 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f11b4040070 con 0x7f11c40732e0 2024-03-28T19:07:33.350 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:33.350 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:33.350 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:33.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11a406bbd0 msgr2=0x7f11a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11a406bbd0 0x7f11a406e090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f11c41520a0 tx=0x7f11b0009040 comp rx=0 tx=0).stop 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40732e0 msgr2=0x7f11c40c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40732e0 0x7f11c40c60b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009870 tx=0x7f11b40046c0 comp rx=0 tx=0).stop 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 shutdown_connections 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11a406bbd0 0x7f11a406e090 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c4152da0 0x7f11c41448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c40c2e40 0x7f11c40c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 --2- 172.21.15.27:0/2113304088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c40732e0 0x7f11c40c60b0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 >> 172.21.15.27:0/2113304088 conn(0x7f11c40c3ff0 msgr2=0x7f11c40c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.354+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 shutdown_connections 2024-03-28T19:07:33.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:33.355+0000 7f11c9400640 1 -- 172.21.15.27:0/2113304088 wait complete. 2024-03-28T19:07:33.372 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:07:35.372 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:35.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.606+0000 7f3d73e00640 1 Processor -- start 2024-03-28T19:07:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.606+0000 7f3d73e00640 1 -- start start 2024-03-28T19:07:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.608+0000 7f3d73e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d740a6e90 0x7f3d74155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.608+0000 7f3d73e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740a7400 con 0x7f3d7406c0d0 2024-03-28T19:07:35.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.608+0000 7f3d73e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d74158b40 con 0x7f3d740c1780 2024-03-28T19:07:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.608+0000 7f3d73e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d74159470 con 0x7f3d740a6e90 2024-03-28T19:07:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.608+0000 7f3d6a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d740a6e90 0x7f3d74155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.608+0000 7f3d6a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d740a6e90 0x7f3d74155c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38200/0 (socket says 172.21.15.27:38200) 2024-03-28T19:07:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.608+0000 7f3d6a000640 1 -- 172.21.15.27:0/663685525 learned_addr learned my addr 172.21.15.27:0/663685525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.609+0000 7f3d6a000640 1 -- 172.21.15.27:0/663685525 >> v1:172.21.15.155:6789/0 conn(0x7f3d740c1780 legacy=0x7f3d7415fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.609+0000 7f3d6a000640 1 -- 172.21.15.27:0/663685525 >> v1:172.21.15.27:6789/0 conn(0x7f3d7406c0d0 legacy=0x7f3d74155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:35.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.609+0000 7f3d6a000640 1 -- 172.21.15.27:0/663685525 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74158e20 con 0x7f3d740a6e90 2024-03-28T19:07:35.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.610+0000 7f3d6a000640 1 --2- 172.21.15.27:0/663685525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740a6e90 0x7f3d74155c40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009870 tx=0x7f3d5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3818d9f2eef18df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.611+0000 7f3d72000640 1 -- 172.21.15.27:0/663685525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c035020 con 0x7f3d740a6e90 2024-03-28T19:07:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.611+0000 7f3d72000640 1 -- 172.21.15.27:0/663685525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5c031870 con 0x7f3d740a6e90 2024-03-28T19:07:35.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.611+0000 7f3d72000640 1 -- 172.21.15.27:0/663685525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c031c40 con 0x7f3d740a6e90 2024-03-28T19:07:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.611+0000 7f3d73e00640 1 -- 172.21.15.27:0/663685525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740a6e90 msgr2=0x7f3d74155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.611+0000 7f3d73e00640 1 --2- 172.21.15.27:0/663685525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740a6e90 0x7f3d74155c40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009870 tx=0x7f3d5c02b910 comp rx=0 tx=0).stop 2024-03-28T19:07:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.611+0000 7f3d73e00640 1 -- 172.21.15.27:0/663685525 shutdown_connections 2024-03-28T19:07:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.611+0000 7f3d73e00640 1 --2- 172.21.15.27:0/663685525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740a6e90 0x7f3d74155c40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.611+0000 7f3d73e00640 1 -- 172.21.15.27:0/663685525 >> 172.21.15.27:0/663685525 conn(0x7f3d740c7aa0 msgr2=0x7f3d740c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.612+0000 7f3d73e00640 1 -- 172.21.15.27:0/663685525 shutdown_connections 2024-03-28T19:07:35.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.612+0000 7f3d73e00640 1 -- 172.21.15.27:0/663685525 wait complete. 2024-03-28T19:07:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.614+0000 7f3d73e00640 1 Processor -- start 2024-03-28T19:07:35.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.614+0000 7f3d73e00640 1 -- start start 2024-03-28T19:07:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d7406c0d0 0x7f3d74158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d740a6e90 0x7f3d741543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:35.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740c1780 0x7f3d7406f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d74152e20 con 0x7f3d740c1780 2024-03-28T19:07:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d74159900 con 0x7f3d7406c0d0 2024-03-28T19:07:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d74152fa0 con 0x7f3d740a6e90 2024-03-28T19:07:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740c1780 0x7f3d7406f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740c1780 0x7f3d7406f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38216/0 (socket says 172.21.15.27:38216) 2024-03-28T19:07:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d73400640 1 -- 172.21.15.27:0/1867859880 learned_addr learned my addr 172.21.15.27:0/1867859880 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d72a00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d7406c0d0 0x7f3d74158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:35.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.615+0000 7f3d6a000640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d740a6e90 0x7f3d741543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d73400640 1 -- 172.21.15.27:0/1867859880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d740a6e90 msgr2=0x7f3d741543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d73400640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d740a6e90 0x7f3d741543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d73400640 1 -- 172.21.15.27:0/1867859880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d7406c0d0 msgr2=0x7f3d74158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d73400640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d7406c0d0 0x7f3d74158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:35.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d73400640 1 -- 172.21.15.27:0/1867859880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74154930 con 0x7f3d740c1780 2024-03-28T19:07:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d73400640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740c1780 0x7f3d7406f5c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00b9a0 tx=0x7f3d6c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d6be00640 1 -- 172.21.15.27:0/1867859880 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c019070 con 0x7f3d740c1780 2024-03-28T19:07:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d6be00640 1 -- 172.21.15.27:0/1867859880 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c00cba0 con 0x7f3d740c1780 2024-03-28T19:07:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.616+0000 7f3d6be00640 1 -- 172.21.15.27:0/1867859880 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c014440 con 0x7f3d740c1780 2024-03-28T19:07:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.617+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d741ee8f0 con 0x7f3d740c1780 2024-03-28T19:07:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.617+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d741eeeb0 con 0x7f3d740c1780 2024-03-28T19:07:35.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.618+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d74072030 con 0x7f3d740c1780 2024-03-28T19:07:35.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.619+0000 7f3d6be00640 1 -- 172.21.15.27:0/1867859880 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d6c01c070 con 0x7f3d740c1780 2024-03-28T19:07:35.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.620+0000 7f3d6be00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d5006bc00 0x7f3d5006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:35.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.620+0000 7f3d6be00640 1 -- 172.21.15.27:0/1867859880 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f3d6c08af70 con 0x7f3d740c1780 2024-03-28T19:07:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.620+0000 7f3d72a00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d5006bc00 0x7f3d5006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.621+0000 7f3d72a00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d5006bc00 0x7f3d5006e0c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3d64001640 tx=0x7f3d64009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:35.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.626+0000 7f3d6be00640 1 -- 172.21.15.27:0/1867859880 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d74072030 con 0x7f3d740c1780 2024-03-28T19:07:35.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.912+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d74069640 con 0x7f3d740c1780 2024-03-28T19:07:35.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.913+0000 7f3d6be00640 1 -- 172.21.15.27:0/1867859880 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f3d6c059110 con 0x7f3d740c1780 2024-03-28T19:07:35.913 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:35.913 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:35.913 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.916+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d5006bc00 msgr2=0x7f3d5006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.916+0000 7f3d73e00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d5006bc00 0x7f3d5006e0c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3d64001640 tx=0x7f3d64009040 comp rx=0 tx=0).stop 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.916+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740c1780 msgr2=0x7f3d7406f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.916+0000 7f3d73e00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740c1780 0x7f3d7406f5c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00b9a0 tx=0x7f3d6c00be60 comp rx=0 tx=0).stop 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.917+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 shutdown_connections 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.917+0000 7f3d73e00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d5006bc00 0x7f3d5006e0c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.917+0000 7f3d73e00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d740c1780 0x7f3d7406f5c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.917+0000 7f3d73e00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d740a6e90 0x7f3d741543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.917+0000 7f3d73e00640 1 --2- 172.21.15.27:0/1867859880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d7406c0d0 0x7f3d74158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:35.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.917+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 >> 172.21.15.27:0/1867859880 conn(0x7f3d740c7aa0 msgr2=0x7f3d74066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:35.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.917+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 shutdown_connections 2024-03-28T19:07:35.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:35.917+0000 7f3d73e00640 1 -- 172.21.15.27:0/1867859880 wait complete. 2024-03-28T19:07:35.935 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:07:37.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.183+0000 7f6523e00640 1 Processor -- start 2024-03-28T19:07:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.183+0000 7f6523e00640 1 -- start start 2024-03-28T19:07:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.184+0000 7f6523e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f65240a7060 0x7f65240c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.185+0000 7f6523e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f652406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.185+0000 7f6523e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6524072f90 con 0x7f6524155700 2024-03-28T19:07:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.185+0000 7f6523e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6524072d40 con 0x7f65240a7060 2024-03-28T19:07:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.185+0000 7f6523e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6524073550 con 0x7f6524152ad0 2024-03-28T19:07:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.185+0000 7f6522000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6524155700 0x7f65240c3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47934/0 (socket says 172.21.15.27:47934) 2024-03-28T19:07:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.185+0000 7f6522000640 1 -- 172.21.15.27:0/787590846 learned_addr learned my addr 172.21.15.27:0/787590846 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:07:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.185+0000 7f6522a00640 1 --2- 172.21.15.27:0/787590846 >> v2:172.21.15.155:3300/0 conn(0x7f65240a7060 0x7f65240c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.185+0000 7f6523400640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f652406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714840261 0 0) 0x7f6524072f90 con 0x7f6524155700 2024-03-28T19:07:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6508003520 con 0x7f6524155700 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737113825 0 0) 0x7f6508003520 con 0x7f6524155700 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 msgr2=0x7f652406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f652406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 >> v2:172.21.15.155:3300/0 conn(0x7f65240a7060 msgr2=0x7f65240c2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 --2- 172.21.15.27:0/787590846 >> v2:172.21.15.155:3300/0 conn(0x7f65240a7060 0x7f65240c2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6524073270 con 0x7f6524155700 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f6510001670 con 0x7f6524155700 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 >> v1:172.21.15.27:6789/0 conn(0x7f6524155700 legacy=0x7f65240c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65080039b0 0x7f6508003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f6508004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6508004d40 0x7f65080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6524072d40 con 0x7f6508004d40 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6524073550 con 0x7f65080039b0 2024-03-28T19:07:38.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650800fdd0 con 0x7f6524152ad0 2024-03-28T19:07:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.186+0000 7f6522a00640 1 --2- 172.21.15.27:0/787590846 >> v2:172.21.15.155:3300/0 conn(0x7f65240a7060 0x7f65240c2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:07:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.187+0000 7f6522000640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f6508004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.187+0000 7f6523400640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6508004d40 0x7f65080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.187+0000 7f6522a00640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65080039b0 0x7f6508003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.188+0000 7f6522000640 1 -- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65080039b0 msgr2=0x7f6508003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.188+0000 7f6522000640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65080039b0 0x7f6508003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.188+0000 7f6522000640 1 -- 172.21.15.27:0/787590846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6508004d40 msgr2=0x7f65080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.188+0000 7f6522000640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6508004d40 0x7f65080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.188+0000 7f6522000640 1 -- 172.21.15.27:0/787590846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6508003520 con 0x7f6524152ad0 2024-03-28T19:07:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.189+0000 7f6522000640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f6508004800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f651000c160 tx=0x7f6508005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fe5ff1b0a09922 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.190+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510007c60 con 0x7f6524152ad0 2024-03-28T19:07:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.190+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6510030070 con 0x7f6524152ad0 2024-03-28T19:07:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.190+0000 7f6521600640 1 -- 172.21.15.27:0/787590846 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510005d30 con 0x7f6524152ad0 2024-03-28T19:07:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.190+0000 7f6523e00640 1 -- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 msgr2=0x7f6508004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.190+0000 7f6523e00640 1 --2- 172.21.15.27:0/787590846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f6508004800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f651000c160 tx=0x7f6508005830 comp rx=0 tx=0).stop 2024-03-28T19:07:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.191+0000 7f6523400640 1 -- 172.21.15.27:0/787590846 reap_dead start 2024-03-28T19:07:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.191+0000 7f6523e00640 1 -- 172.21.15.27:0/787590846 shutdown_connections 2024-03-28T19:07:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.191+0000 7f6523e00640 1 -- 172.21.15.27:0/787590846 >> 172.21.15.27:0/787590846 conn(0x7f65240c7aa0 msgr2=0x7f65240c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.191+0000 7f6523e00640 1 -- 172.21.15.27:0/787590846 shutdown_connections 2024-03-28T19:07:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.191+0000 7f6523e00640 1 -- 172.21.15.27:0/787590846 wait complete. 2024-03-28T19:07:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.193+0000 7f6523e00640 1 Processor -- start 2024-03-28T19:07:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.193+0000 7f6523e00640 1 -- start start 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f652406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6524146ba0 0x7f65241e7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65241475d0 0x7f6524154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6524155b60 con 0x7f65241475d0 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65241525b0 con 0x7f6524146ba0 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6524152f60 con 0x7f6524152ad0 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65241475d0 0x7f6524154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65241475d0 0x7f6524154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53522/0 (socket says 172.21.15.27:53522) 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.194+0000 7f6523400640 1 -- 172.21.15.27:0/3469493843 learned_addr learned my addr 172.21.15.27:0/3469493843 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.195+0000 7f6522a00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f652406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.195+0000 7f6522000640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6524146ba0 0x7f65241e7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.195+0000 7f6523400640 1 -- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 msgr2=0x7f652406fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.195+0000 7f6523400640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f652406fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.195+0000 7f6523400640 1 -- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6524146ba0 msgr2=0x7f65241e7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.195+0000 7f6523400640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6524146ba0 0x7f65241e7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.195+0000 7f6523400640 1 -- 172.21.15.27:0/3469493843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65241eb0c0 con 0x7f65241475d0 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.195+0000 7f6522000640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6524146ba0 0x7f65241e7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.196+0000 7f6523400640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65241475d0 0x7f6524154a30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6524156820 tx=0x7f651c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.196+0000 7f651b400640 1 -- 172.21.15.27:0/3469493843 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c00fe10 con 0x7f65241475d0 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.196+0000 7f651b400640 1 -- 172.21.15.27:0/3469493843 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c01d070 con 0x7f65241475d0 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.196+0000 7f651b400640 1 -- 172.21.15.27:0/3469493843 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c015670 con 0x7f65241475d0 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.196+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6524142e20 con 0x7f65241475d0 2024-03-28T19:07:38.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.196+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652406f1f0 con 0x7f65241475d0 2024-03-28T19:07:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.197+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65240c3620 con 0x7f65241475d0 2024-03-28T19:07:38.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.198+0000 7f651b400640 1 -- 172.21.15.27:0/3469493843 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f651c015810 con 0x7f65241475d0 2024-03-28T19:07:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.200+0000 7f651b400640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f650406bbd0 0x7f650406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.200+0000 7f651b400640 1 -- 172.21.15.27:0/3469493843 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7927+0+0 (secure 0 0 0) 0x7f651c08bb50 con 0x7f65241475d0 2024-03-28T19:07:38.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.200+0000 7f6522a00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f650406bbd0 0x7f650406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:38.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.201+0000 7f6522a00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f650406bbd0 0x7f650406e090 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f65140093d0 tx=0x7f6514009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:38.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.205+0000 7f651b400640 1 -- 172.21.15.27:0/3469493843 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f65240c3620 con 0x7f65241475d0 2024-03-28T19:07:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.516+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6524069640 con 0x7f65241475d0 2024-03-28T19:07:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.518+0000 7f651b400640 1 -- 172.21.15.27:0/3469493843 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6259 (secure 0 0 0) 0x7f651c010040 con 0x7f65241475d0 2024-03-28T19:07:38.517 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:38.518 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:10.529093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1248101703","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1248101703},{"type":"v1","addr":"172.21.15.27:6836","nonce":1248101703}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-03-28T19:07:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 97 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.521+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f650406bbd0 msgr2=0x7f650406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.521+0000 7f6523e00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f650406bbd0 0x7f650406e090 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f65140093d0 tx=0x7f6514009190 comp rx=0 tx=0).stop 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.521+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65241475d0 msgr2=0x7f6524154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.521+0000 7f6523e00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65241475d0 0x7f6524154a30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6524156820 tx=0x7f651c00f580 comp rx=0 tx=0).stop 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.522+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 shutdown_connections 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.522+0000 7f6523e00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f650406bbd0 0x7f650406e090 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.522+0000 7f6523e00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65241475d0 0x7f6524154a30 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.522+0000 7f6523e00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6524146ba0 0x7f65241e7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.522+0000 7f6523e00640 1 --2- 172.21.15.27:0/3469493843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6524152ad0 0x7f652406fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.522+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 >> 172.21.15.27:0/3469493843 conn(0x7f65240c7aa0 msgr2=0x7f6524155ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.522+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 shutdown_connections 2024-03-28T19:07:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:38.522+0000 7f6523e00640 1 -- 172.21.15.27:0/3469493843 wait complete. 2024-03-28T19:07:38.542 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:07:40.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:40.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.771+0000 7fc5c7200640 1 Processor -- start 2024-03-28T19:07:40.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.771+0000 7fc5c7200640 1 -- start start 2024-03-28T19:07:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.772+0000 7fc5c7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8150ae0 0x7fc5c80a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.772+0000 7fc5c7200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc5c80a61a0 0x7fc5c80c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c7200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5c80c6ee0 0x7fc5c8153a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c7200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c8150ec0 con 0x7fc5c80c6ee0 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c7200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c806dc70 con 0x7fc5c80a61a0 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8153550 con 0x7fc5c8150ae0 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c6800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5c80c6ee0 0x7fc5c8153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c6800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5c80c6ee0 0x7fc5c8153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53530/0 (socket says 172.21.15.27:53530) 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c6800640 1 -- 172.21.15.27:0/3626270844 learned_addr learned my addr 172.21.15.27:0/3626270844 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5bfe00640 1 --2- 172.21.15.27:0/3626270844 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80a61a0 0x7fc5c80c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c5e00640 1 --2- 172.21.15.27:0/3626270844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8150ae0 0x7fc5c80a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c6800640 1 -- 172.21.15.27:0/3626270844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8150ae0 msgr2=0x7fc5c80a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.773+0000 7fc5c6800640 1 --2- 172.21.15.27:0/3626270844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8150ae0 0x7fc5c80a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.774+0000 7fc5c6800640 1 -- 172.21.15.27:0/3626270844 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80a61a0 msgr2=0x7fc5c80c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.774+0000 7fc5c6800640 1 --2- 172.21.15.27:0/3626270844 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80a61a0 0x7fc5c80c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.774+0000 7fc5c6800640 1 -- 172.21.15.27:0/3626270844 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c806df10 con 0x7fc5c80c6ee0 2024-03-28T19:07:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.774+0000 7fc5c5e00640 1 --2- 172.21.15.27:0/3626270844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8150ae0 0x7fc5c80a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:07:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.775+0000 7fc5c6800640 1 --2- 172.21.15.27:0/3626270844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c80c6ee0 0x7fc5c8153a30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80abd20 tx=0x7fc5c002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b619f4c46bc41648 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.775+0000 7fc5c5400640 1 -- 172.21.15.27:0/3626270844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c002c9f0 con 0x7fc5c80c6ee0 2024-03-28T19:07:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.775+0000 7fc5c5400640 1 -- 172.21.15.27:0/3626270844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c002cb90 con 0x7fc5c80c6ee0 2024-03-28T19:07:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.775+0000 7fc5c5400640 1 -- 172.21.15.27:0/3626270844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c00343f0 con 0x7fc5c80c6ee0 2024-03-28T19:07:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.776+0000 7fc5c7200640 1 -- 172.21.15.27:0/3626270844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c80c6ee0 msgr2=0x7fc5c8153a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.776+0000 7fc5c7200640 1 --2- 172.21.15.27:0/3626270844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c80c6ee0 0x7fc5c8153a30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80abd20 tx=0x7fc5c002be50 comp rx=0 tx=0).stop 2024-03-28T19:07:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.776+0000 7fc5c7200640 1 -- 172.21.15.27:0/3626270844 shutdown_connections 2024-03-28T19:07:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.776+0000 7fc5c7200640 1 --2- 172.21.15.27:0/3626270844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c80c6ee0 0x7fc5c8153a30 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:40.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.776+0000 7fc5c7200640 1 --2- 172.21.15.27:0/3626270844 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80a61a0 0x7fc5c80c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.776+0000 7fc5c7200640 1 --2- 172.21.15.27:0/3626270844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8150ae0 0x7fc5c80a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.776+0000 7fc5c7200640 1 -- 172.21.15.27:0/3626270844 >> 172.21.15.27:0/3626270844 conn(0x7fc5c806b0c0 msgr2=0x7fc5c806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.777+0000 7fc5c7200640 1 -- 172.21.15.27:0/3626270844 shutdown_connections 2024-03-28T19:07:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.777+0000 7fc5c7200640 1 -- 172.21.15.27:0/3626270844 wait complete. 2024-03-28T19:07:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.779+0000 7fc5c7200640 1 Processor -- start 2024-03-28T19:07:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.779+0000 7fc5c7200640 1 -- start start 2024-03-28T19:07:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.780+0000 7fc5c7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80a61a0 0x7fc5c81ec8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.780+0000 7fc5c7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c80c6ee0 0x7fc5c81487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.780+0000 7fc5c7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8150ae0 0x7fc5c8148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.780+0000 7fc5c7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c806e7b0 con 0x7fc5c8150ae0 2024-03-28T19:07:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.780+0000 7fc5c7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c806e4d0 con 0x7fc5c80c6ee0 2024-03-28T19:07:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.780+0000 7fc5c7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c81569d0 con 0x7fc5c80a61a0 2024-03-28T19:07:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8150ae0 0x7fc5c8148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8150ae0 0x7fc5c8148d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53538/0 (socket says 172.21.15.27:53538) 2024-03-28T19:07:40.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c6800640 1 -- 172.21.15.27:0/4109087856 learned_addr learned my addr 172.21.15.27:0/4109087856 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:40.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5bfe00640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c80c6ee0 0x7fc5c81487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:40.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c5e00640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80a61a0 0x7fc5c81ec8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:40.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c6800640 1 -- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80a61a0 msgr2=0x7fc5c81ec8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:40.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c6800640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80a61a0 0x7fc5c81ec8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:40.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c6800640 1 -- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c80c6ee0 msgr2=0x7fc5c81487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:40.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c6800640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c80c6ee0 0x7fc5c81487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:40.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c6800640 1 -- 172.21.15.27:0/4109087856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c814add0 con 0x7fc5c8150ae0 2024-03-28T19:07:40.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.781+0000 7fc5c5e00640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80a61a0 0x7fc5c81ec8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:07:40.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.782+0000 7fc5c6800640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8150ae0 0x7fc5c8148d30 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0036040 tx=0x7fc5c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:40.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.782+0000 7fc5be000640 1 -- 172.21.15.27:0/4109087856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0043020 con 0x7fc5c8150ae0 2024-03-28T19:07:40.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.782+0000 7fc5be000640 1 -- 172.21.15.27:0/4109087856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c003e450 con 0x7fc5c8150ae0 2024-03-28T19:07:40.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.782+0000 7fc5be000640 1 -- 172.21.15.27:0/4109087856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0009510 con 0x7fc5c8150ae0 2024-03-28T19:07:40.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.782+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c81ece10 con 0x7fc5c8150ae0 2024-03-28T19:07:40.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.782+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c81ed0b0 con 0x7fc5c8150ae0 2024-03-28T19:07:40.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.784+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c8062500 con 0x7fc5c8150ae0 2024-03-28T19:07:40.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.786+0000 7fc5be000640 1 -- 172.21.15.27:0/4109087856 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc5c0046020 con 0x7fc5c8150ae0 2024-03-28T19:07:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.787+0000 7fc5be000640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a406bc00 0x7fc5a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.787+0000 7fc5be000640 1 -- 172.21.15.27:0/4109087856 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7fc5c00adbc0 con 0x7fc5c8150ae0 2024-03-28T19:07:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.787+0000 7fc5c5e00640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a406bc00 0x7fc5a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:40.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.788+0000 7fc5c5e00640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a406bc00 0x7fc5a406e0c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8006fd0 tx=0x7fc5b8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:40.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:40.793+0000 7fc5be000640 1 -- 172.21.15.27:0/4109087856 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc5c8062500 con 0x7fc5c8150ae0 2024-03-28T19:07:41.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.112+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5c8069640 con 0x7fc5c8150ae0 2024-03-28T19:07:41.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.112+0000 7fc5be000640 1 -- 172.21.15.27:0/4109087856 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5460 (secure 0 0 0) 0x7fc5c004d030 con 0x7fc5c8150ae0 2024-03-28T19:07:41.112 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:41.112 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:39.867402+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:07:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 98 2024-03-28T19:07:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.116+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a406bc00 msgr2=0x7fc5a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.116+0000 7fc5c7200640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a406bc00 0x7fc5a406e0c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8006fd0 tx=0x7fc5b8008040 comp rx=0 tx=0).stop 2024-03-28T19:07:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.116+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8150ae0 msgr2=0x7fc5c8148d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.116+0000 7fc5c7200640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8150ae0 0x7fc5c8148d30 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0036040 tx=0x7fc5c0004a40 comp rx=0 tx=0).stop 2024-03-28T19:07:41.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.117+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 shutdown_connections 2024-03-28T19:07:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.117+0000 7fc5c7200640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a406bc00 0x7fc5a406e0c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.117+0000 7fc5c7200640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8150ae0 0x7fc5c8148d30 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.117+0000 7fc5c7200640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c80c6ee0 0x7fc5c81487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.117+0000 7fc5c7200640 1 --2- 172.21.15.27:0/4109087856 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80a61a0 0x7fc5c81ec8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.117+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 >> 172.21.15.27:0/4109087856 conn(0x7fc5c806b0c0 msgr2=0x7fc5c8152000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.117+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 shutdown_connections 2024-03-28T19:07:41.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.117+0000 7fc5c7200640 1 -- 172.21.15.27:0/4109087856 wait complete. 2024-03-28T19:07:41.139 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-03-28T19:07:41.139 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:07:41.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:41.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.366+0000 7f8dd6600640 1 Processor -- start 2024-03-28T19:07:41.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.366+0000 7f8dd6600640 1 -- start start 2024-03-28T19:07:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.368+0000 7f8dd6600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8dd00c58c0 0x7f8dd00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.368+0000 7f8dd6600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 0x7f8dd0155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:41.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.368+0000 7f8dd6600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd0156230 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.368+0000 7f8dd6600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd0158bd0 con 0x7f8dd00c58c0 2024-03-28T19:07:41.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.368+0000 7f8dd6600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0158eb0 con 0x7f8dd00be5d0 2024-03-28T19:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.368+0000 7f8dd5200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8dd00c58c0 0x7f8dd00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.368+0000 7f8dd5200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8dd00c58c0 0x7f8dd00c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53550/0 (socket says 172.21.15.27:53550) 2024-03-28T19:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.368+0000 7f8dd5200640 1 -- 172.21.15.27:0/433829277 learned_addr learned my addr 172.21.15.27:0/433829277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.369+0000 7f8dd5c00640 1 --2- 172.21.15.27:0/433829277 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 0x7f8dd0155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.369+0000 7f8dd5200640 1 -- 172.21.15.27:0/433829277 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 msgr2=0x7f8dd0155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.369+0000 7f8dd5200640 1 --2- 172.21.15.27:0/433829277 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 0x7f8dd0155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.369+0000 7f8dd5200640 1 -- 172.21.15.27:0/433829277 >> v1:172.21.15.155:6789/0 conn(0x7f8dd00c8ed0 legacy=0x7f8dd01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:41.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.369+0000 7f8dd5200640 1 -- 172.21.15.27:0/433829277 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0159190 con 0x7f8dd00c58c0 2024-03-28T19:07:41.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.371+0000 7f8dd5200640 1 --2- 172.21.15.27:0/433829277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c58c0 0x7f8dd00c5ca0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009870 tx=0x7f8dc402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adae0fd1466b44c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:41.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.371+0000 7f8dcf400640 1 -- 172.21.15.27:0/433829277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4035020 con 0x7f8dd00c58c0 2024-03-28T19:07:41.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.371+0000 7f8dcf400640 1 -- 172.21.15.27:0/433829277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc4031640 con 0x7f8dd00c58c0 2024-03-28T19:07:41.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.371+0000 7f8dcf400640 1 -- 172.21.15.27:0/433829277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc402d040 con 0x7f8dd00c58c0 2024-03-28T19:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.372+0000 7f8dd6600640 1 -- 172.21.15.27:0/433829277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c58c0 msgr2=0x7f8dd00c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.372+0000 7f8dd6600640 1 --2- 172.21.15.27:0/433829277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c58c0 0x7f8dd00c5ca0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009870 tx=0x7f8dc402b8f0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.372+0000 7f8dd6600640 1 -- 172.21.15.27:0/433829277 shutdown_connections 2024-03-28T19:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.372+0000 7f8dd6600640 1 --2- 172.21.15.27:0/433829277 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 0x7f8dd0155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.372+0000 7f8dd6600640 1 --2- 172.21.15.27:0/433829277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c58c0 0x7f8dd00c5ca0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.372+0000 7f8dd6600640 1 -- 172.21.15.27:0/433829277 >> 172.21.15.27:0/433829277 conn(0x7f8dd006b6f0 msgr2=0x7f8dd00acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:41.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.372+0000 7f8dd6600640 1 -- 172.21.15.27:0/433829277 shutdown_connections 2024-03-28T19:07:41.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.372+0000 7f8dd6600640 1 -- 172.21.15.27:0/433829277 wait complete. 2024-03-28T19:07:41.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.375+0000 7f8dd6600640 1 Processor -- start 2024-03-28T19:07:41.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.375+0000 7f8dd6600640 1 -- start start 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd6600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 0x7f8dd00c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd6600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8dd00c58c0 0x7f8dd00c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd6600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c8ed0 0x7f8dd00aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd6600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00b0820 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd6600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0159750 con 0x7f8dd00c58c0 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd6600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00b09a0 con 0x7f8dd00be5d0 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c8ed0 0x7f8dd00aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c8ed0 0x7f8dd00aae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53560/0 (socket says 172.21.15.27:53560) 2024-03-28T19:07:41.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.376+0000 7f8dd5c00640 1 -- 172.21.15.27:0/2344174654 learned_addr learned my addr 172.21.15.27:0/2344174654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.377+0000 7f8dd5200640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 0x7f8dd00c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.377+0000 7f8dcfe00640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8dd00c58c0 0x7f8dd00c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.377+0000 7f8dd5c00640 1 -- 172.21.15.27:0/2344174654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 msgr2=0x7f8dd00c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.377+0000 7f8dd5c00640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 0x7f8dd00c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.377+0000 7f8dd5c00640 1 -- 172.21.15.27:0/2344174654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8dd00c58c0 msgr2=0x7f8dd00c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.377+0000 7f8dd5c00640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8dd00c58c0 0x7f8dd00c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.377+0000 7f8dd5c00640 1 -- 172.21.15.27:0/2344174654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00acb50 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.378+0000 7f8dd5c00640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c8ed0 0x7f8dd00aae70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8007e90 tx=0x7f8dc800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:41.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.378+0000 7f8dccc00640 1 -- 172.21.15.27:0/2344174654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8014420 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.378+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd00ace30 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.378+0000 7f8dccc00640 1 -- 172.21.15.27:0/2344174654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8019070 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.378+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd00c2690 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.378+0000 7f8dccc00640 1 -- 172.21.15.27:0/2344174654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8014800 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.380+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dac005290 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.381+0000 7f8dccc00640 1 -- 172.21.15.27:0/2344174654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8dc80149a0 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.382+0000 7f8dccc00640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8db006bc50 0x7f8db006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:41.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.382+0000 7f8dccc00640 1 -- 172.21.15.27:0/2344174654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7f8dc808ba50 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.383+0000 7f8dd5200640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8db006bc50 0x7f8db006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:41.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.384+0000 7f8dd5200640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8db006bc50 0x7f8db006e110 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4036000 tx=0x7f8dc403c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:41.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.387+0000 7f8dccc00640 1 -- 172.21.15.27:0/2344174654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8dc8059cb0 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.709+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dac005580 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.709+0000 7f8dccc00640 1 -- 172.21.15.27:0/2344174654 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5460 (secure 0 0 0) 0x7f8dc80595b0 con 0x7f8dd00c8ed0 2024-03-28T19:07:41.709 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:41.709 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:39.867402+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:07:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 98 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.712+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8db006bc50 msgr2=0x7f8db006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.712+0000 7f8dd6600640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8db006bc50 0x7f8db006e110 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4036000 tx=0x7f8dc403c040 comp rx=0 tx=0).stop 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.712+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c8ed0 msgr2=0x7f8dd00aae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.712+0000 7f8dd6600640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c8ed0 0x7f8dd00aae70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8007e90 tx=0x7f8dc800e4b0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.713+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 shutdown_connections 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.713+0000 7f8dd6600640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8db006bc50 0x7f8db006e110 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.713+0000 7f8dd6600640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00c8ed0 0x7f8dd00aae70 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.713+0000 7f8dd6600640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8dd00c58c0 0x7f8dd00c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.713+0000 7f8dd6600640 1 --2- 172.21.15.27:0/2344174654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8dd00be5d0 0x7f8dd00c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.713+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 >> 172.21.15.27:0/2344174654 conn(0x7f8dd006b6f0 msgr2=0x7f8dd00aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.713+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 shutdown_connections 2024-03-28T19:07:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:41.713+0000 7f8dd6600640 1 -- 172.21.15.27:0/2344174654 wait complete. 2024-03-28T19:07:41.733 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5336": { "addr": "172.21.15.155:6835/3884653650", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 3884653650, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 3884653650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5336, "incarnation": 87, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:06:20.279933+0000", "qdb_cluster": [ 5336 ], "qdb_leader": 5336, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5336 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 98, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6411": { "addr": "172.21.15.27:6837/2647193511", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 2647193511, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 2647193511, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6411, "incarnation": 98, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:07:39.867402+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6411 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:07:43.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:43.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.980+0000 7f32db000640 1 Processor -- start 2024-03-28T19:07:43.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.980+0000 7f32db000640 1 -- start start 2024-03-28T19:07:43.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.981+0000 7f32db000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f32d406c0d0 0x7f32d406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:43.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.982+0000 7f32db000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32d40a6e90 0x7f32d4155530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:43.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.982+0000 7f32db000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40a7400 con 0x7f32d40c1780 2024-03-28T19:07:43.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.982+0000 7f32db000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d4159470 con 0x7f32d40a6e90 2024-03-28T19:07:43.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.982+0000 7f32db000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d4158b40 con 0x7f32d406c0d0 2024-03-28T19:07:43.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.982+0000 7f32d9200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32d40a6e90 0x7f32d4155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:43.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.982+0000 7f32d9200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32d40a6e90 0x7f32d4155530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53568/0 (socket says 172.21.15.27:53568) 2024-03-28T19:07:43.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.982+0000 7f32d9200640 1 -- 172.21.15.27:0/3220231072 learned_addr learned my addr 172.21.15.27:0/3220231072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:43.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.982+0000 7f32d9c00640 1 --2- 172.21.15.27:0/3220231072 >> v2:172.21.15.155:3300/0 conn(0x7f32d406c0d0 0x7f32d406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:43.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.983+0000 7f32d9200640 1 -- 172.21.15.27:0/3220231072 >> v2:172.21.15.155:3300/0 conn(0x7f32d406c0d0 msgr2=0x7f32d406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:43.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.983+0000 7f32d9200640 1 --2- 172.21.15.27:0/3220231072 >> v2:172.21.15.155:3300/0 conn(0x7f32d406c0d0 0x7f32d406c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:43.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.983+0000 7f32d9200640 1 -- 172.21.15.27:0/3220231072 >> v1:172.21.15.155:6789/0 conn(0x7f32d40c1780 legacy=0x7f32d4155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:43.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.983+0000 7f32d9200640 1 -- 172.21.15.27:0/3220231072 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4158e20 con 0x7f32d40a6e90 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.984+0000 7f32d9200640 1 --2- 172.21.15.27:0/3220231072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d4155530 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f32d4050360 tx=0x7f32c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30545710a0ef5124 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.984+0000 7f32d3e00640 1 -- 172.21.15.27:0/3220231072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c002e070 con 0x7f32d40a6e90 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.984+0000 7f32d3e00640 1 -- 172.21.15.27:0/3220231072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c0037070 con 0x7f32d40a6e90 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.984+0000 7f32d3e00640 1 -- 172.21.15.27:0/3220231072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0031a00 con 0x7f32d40a6e90 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.985+0000 7f32db000640 1 -- 172.21.15.27:0/3220231072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 msgr2=0x7f32d4155530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.985+0000 7f32db000640 1 --2- 172.21.15.27:0/3220231072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d4155530 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f32d4050360 tx=0x7f32c002b880 comp rx=0 tx=0).stop 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.985+0000 7f32db000640 1 -- 172.21.15.27:0/3220231072 shutdown_connections 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.985+0000 7f32db000640 1 --2- 172.21.15.27:0/3220231072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d4155530 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.985+0000 7f32db000640 1 --2- 172.21.15.27:0/3220231072 >> v2:172.21.15.155:3300/0 conn(0x7f32d406c0d0 0x7f32d406c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.985+0000 7f32db000640 1 -- 172.21.15.27:0/3220231072 >> 172.21.15.27:0/3220231072 conn(0x7f32d40c7aa0 msgr2=0x7f32d40c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.985+0000 7f32db000640 1 -- 172.21.15.27:0/3220231072 shutdown_connections 2024-03-28T19:07:43.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.985+0000 7f32db000640 1 -- 172.21.15.27:0/3220231072 wait complete. 2024-03-28T19:07:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.987+0000 7f32db000640 1 Processor -- start 2024-03-28T19:07:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.988+0000 7f32db000640 1 -- start start 2024-03-28T19:07:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32db000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d406c0d0 0x7f32d4154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32db000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d406f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32db000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32d40c1780 0x7f32d40a8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32db000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4152e20 con 0x7f32d40a6e90 2024-03-28T19:07:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32db000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4159900 con 0x7f32d406c0d0 2024-03-28T19:07:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32db000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d4152fa0 con 0x7f32d40c1780 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32d9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d406f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32d9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d406f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53576/0 (socket says 172.21.15.27:53576) 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32d9200640 1 -- 172.21.15.27:0/407606567 learned_addr learned my addr 172.21.15.27:0/407606567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.989+0000 7f32d9c00640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d406c0d0 0x7f32d4154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.990+0000 7f32da600640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32d40c1780 0x7f32d40a8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.990+0000 7f32d9200640 1 -- 172.21.15.27:0/407606567 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32d40c1780 msgr2=0x7f32d40a8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.990+0000 7f32d9200640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32d40c1780 0x7f32d40a8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.990+0000 7f32d9200640 1 -- 172.21.15.27:0/407606567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d406c0d0 msgr2=0x7f32d4154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.990+0000 7f32d9200640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d406c0d0 0x7f32d4154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.990+0000 7f32d9200640 1 -- 172.21.15.27:0/407606567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4153350 con 0x7f32d40a6e90 2024-03-28T19:07:43.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.990+0000 7f32d9200640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d406f5c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f32c002b8c0 tx=0x7f32c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:43.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.991+0000 7f32d1600640 1 -- 172.21.15.27:0/407606567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0032670 con 0x7f32d40a6e90 2024-03-28T19:07:43.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.991+0000 7f32d1600640 1 -- 172.21.15.27:0/407606567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c0032830 con 0x7f32d40a6e90 2024-03-28T19:07:43.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.991+0000 7f32d1600640 1 -- 172.21.15.27:0/407606567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0032c20 con 0x7f32d40a6e90 2024-03-28T19:07:43.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.991+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d41eef10 con 0x7f32d40a6e90 2024-03-28T19:07:43.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.991+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d40a9460 con 0x7f32d40a6e90 2024-03-28T19:07:43.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.992+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d4069640 con 0x7f32d40a6e90 2024-03-28T19:07:43.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.993+0000 7f32d1600640 1 -- 172.21.15.27:0/407606567 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f32c00453f0 con 0x7f32d40a6e90 2024-03-28T19:07:43.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.994+0000 7f32d1600640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f32b406bbd0 0x7f32b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:43.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.994+0000 7f32d1600640 1 -- 172.21.15.27:0/407606567 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7f32c00ad410 con 0x7f32d40a6e90 2024-03-28T19:07:43.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.995+0000 7f32d9c00640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f32b406bbd0 0x7f32b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:43.996+0000 7f32d9c00640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f32b406bbd0 0x7f32b406e090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f32c8006fd0 tx=0x7f32c8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:43.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.000+0000 7f32d1600640 1 -- 172.21.15.27:0/407606567 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f32c0040070 con 0x7f32d40a6e90 2024-03-28T19:07:44.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.270+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32d40c1560 con 0x7f32d40a6e90 2024-03-28T19:07:44.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.271+0000 7f32d1600640 1 -- 172.21.15.27:0/407606567 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5463 (secure 0 0 0) 0x7f32c007b5f0 con 0x7f32d40a6e90 2024-03-28T19:07:44.271 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:44.271 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:42.851938+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:07:44.271 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 99 2024-03-28T19:07:44.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f32b406bbd0 msgr2=0x7f32b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:44.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f32b406bbd0 0x7f32b406e090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f32c8006fd0 tx=0x7f32c8008040 comp rx=0 tx=0).stop 2024-03-28T19:07:44.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 msgr2=0x7f32d406f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:44.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d406f5c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f32c002b8c0 tx=0x7f32c00046c0 comp rx=0 tx=0).stop 2024-03-28T19:07:44.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 shutdown_connections 2024-03-28T19:07:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f32b406bbd0 0x7f32b406e090 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32d40c1780 0x7f32d40a8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32d40a6e90 0x7f32d406f5c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 --2- 172.21.15.27:0/407606567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d406c0d0 0x7f32d4154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.274+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 >> 172.21.15.27:0/407606567 conn(0x7f32d40c7aa0 msgr2=0x7f32d4073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.275+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 shutdown_connections 2024-03-28T19:07:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:44.275+0000 7f32db000640 1 -- 172.21.15.27:0/407606567 wait complete. 2024-03-28T19:07:44.298 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:07:46.299 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:46.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.555+0000 7fd0ef000640 1 Processor -- start 2024-03-28T19:07:46.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.556+0000 7fd0ef000640 1 -- start start 2024-03-28T19:07:46.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.557+0000 7fd0ef000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e80c6a90 0x7fd0e806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.557+0000 7fd0ef000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd0e80c72f0 0x7fd0e81572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.557+0000 7fd0ef000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e806d080 con 0x7fd0e80c6e70 2024-03-28T19:07:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.557+0000 7fd0ef000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e8159980 con 0x7fd0e80c72f0 2024-03-28T19:07:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.557+0000 7fd0ef000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e806cb00 con 0x7fd0e80c6a90 2024-03-28T19:07:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.557+0000 7fd0ed200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd0e80c6e70 0x7fd0e8156ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37154/0 (socket says 172.21.15.27:37154) 2024-03-28T19:07:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.558+0000 7fd0ed200640 1 -- 172.21.15.27:0/2785688442 learned_addr learned my addr 172.21.15.27:0/2785688442 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:07:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.558+0000 7fd0edc00640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e80c6a90 0x7fd0e806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.558+0000 7fd0ee600640 1 --2- 172.21.15.27:0/2785688442 >> v2:172.21.15.155:3300/0 conn(0x7fd0e80c72f0 0x7fd0e81572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.558+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000736884 0 0) 0x7fd0e806d080 con 0x7fd0e80c6e70 2024-03-28T19:07:46.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.558+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d0003520 con 0x7fd0e80c6e70 2024-03-28T19:07:46.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215112611 0 0) 0x7fd0d0003520 con 0x7fd0e80c6e70 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e80c6a90 msgr2=0x7fd0e806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e80c6a90 0x7fd0e806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 >> v2:172.21.15.155:3300/0 conn(0x7fd0e80c72f0 msgr2=0x7fd0e81572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 --2- 172.21.15.27:0/2785688442 >> v2:172.21.15.155:3300/0 conn(0x7fd0e80c72f0 0x7fd0e81572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e806cda0 con 0x7fd0e80c6e70 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd0d4001670 con 0x7fd0e80c6e70 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 >> v1:172.21.15.27:6789/0 conn(0x7fd0e80c6e70 legacy=0x7fd0e8156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e80c6a90 0x7fd0d00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0d0004d40 0x7fd0d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e806cb00 con 0x7fd0d0003f30 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d000b9c0 con 0x7fd0d0004d40 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d000bb40 con 0x7fd0e80c6a90 2024-03-28T19:07:46.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0ee600640 1 --2- 172.21.15.27:0/2785688442 >> v2:172.21.15.155:3300/0 conn(0x7fd0e80c72f0 0x7fd0e81572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:07:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0ed200640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.559+0000 7fd0edc00640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e80c6a90 0x7fd0d00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.560+0000 7fd0ee600640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0d0004d40 0x7fd0d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.560+0000 7fd0ed200640 1 -- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e80c6a90 msgr2=0x7fd0d00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.560+0000 7fd0ed200640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e80c6a90 0x7fd0d00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.560+0000 7fd0ed200640 1 -- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0d0004d40 msgr2=0x7fd0d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.560+0000 7fd0ed200640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0d0004d40 0x7fd0d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.560+0000 7fd0ed200640 1 -- 172.21.15.27:0/2785688442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d0003520 con 0x7fd0d0003f30 2024-03-28T19:07:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.560+0000 7fd0ee600640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0d0004d40 0x7fd0d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:07:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.562+0000 7fd0ed200640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0d0004800 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4007fd0 tx=0x7fd0d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c596f86f622cca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.562+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4007d50 con 0x7fd0d0003f30 2024-03-28T19:07:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.562+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d4030070 con 0x7fd0d0003f30 2024-03-28T19:07:46.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.562+0000 7fd0e7e00640 1 -- 172.21.15.27:0/2785688442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d402cd40 con 0x7fd0d0003f30 2024-03-28T19:07:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.563+0000 7fd0ef000640 1 -- 172.21.15.27:0/2785688442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 msgr2=0x7fd0d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.563+0000 7fd0ef000640 1 --2- 172.21.15.27:0/2785688442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0d0004800 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4007fd0 tx=0x7fd0d0005830 comp rx=0 tx=0).stop 2024-03-28T19:07:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.563+0000 7fd0ee600640 1 -- 172.21.15.27:0/2785688442 reap_dead start 2024-03-28T19:07:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.563+0000 7fd0ef000640 1 -- 172.21.15.27:0/2785688442 shutdown_connections 2024-03-28T19:07:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.563+0000 7fd0ef000640 1 -- 172.21.15.27:0/2785688442 >> 172.21.15.27:0/2785688442 conn(0x7fd0e80aa7c0 msgr2=0x7fd0e80aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.563+0000 7fd0ef000640 1 -- 172.21.15.27:0/2785688442 shutdown_connections 2024-03-28T19:07:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.563+0000 7fd0ef000640 1 -- 172.21.15.27:0/2785688442 wait complete. 2024-03-28T19:07:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.566+0000 7fd0ef000640 1 Processor -- start 2024-03-28T19:07:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.566+0000 7fd0ef000640 1 -- start start 2024-03-28T19:07:46.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0ef000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0e806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0ef000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e814e560 0x7fd0e80c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0ef000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e814cee0 0x7fd0e814d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0ef000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e81577f0 con 0x7fd0d0003f30 2024-03-28T19:07:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0ef000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e80c7750 con 0x7fd0e814cee0 2024-03-28T19:07:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0ef000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e80c8630 con 0x7fd0e814e560 2024-03-28T19:07:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0edc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0e806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0edc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0e806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37158/0 (socket says 172.21.15.27:37158) 2024-03-28T19:07:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0edc00640 1 -- 172.21.15.27:0/23781220 learned_addr learned my addr 172.21.15.27:0/23781220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0ed200640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e814e560 0x7fd0e80c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.567+0000 7fd0ee600640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e814cee0 0x7fd0e814d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.568+0000 7fd0edc00640 1 -- 172.21.15.27:0/23781220 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e814e560 msgr2=0x7fd0e80c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.568+0000 7fd0edc00640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e814e560 0x7fd0e80c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.568+0000 7fd0edc00640 1 -- 172.21.15.27:0/23781220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e814cee0 msgr2=0x7fd0e814d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.568+0000 7fd0edc00640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e814cee0 0x7fd0e814d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.568+0000 7fd0edc00640 1 -- 172.21.15.27:0/23781220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e814d8a0 con 0x7fd0d0003f30 2024-03-28T19:07:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.568+0000 7fd0edc00640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0e806fd00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0119a0 tx=0x7fd0dc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.569+0000 7fd0e5600640 1 -- 172.21.15.27:0/23781220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc00faf0 con 0x7fd0d0003f30 2024-03-28T19:07:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.569+0000 7fd0e5600640 1 -- 172.21.15.27:0/23781220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc00fc90 con 0x7fd0d0003f30 2024-03-28T19:07:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.569+0000 7fd0e5600640 1 -- 172.21.15.27:0/23781220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc014440 con 0x7fd0d0003f30 2024-03-28T19:07:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.569+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e81eb3d0 con 0x7fd0d0003f30 2024-03-28T19:07:46.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.569+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e81eb990 con 0x7fd0d0003f30 2024-03-28T19:07:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.571+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e8069640 con 0x7fd0d0003f30 2024-03-28T19:07:46.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.573+0000 7fd0e5600640 1 -- 172.21.15.27:0/23781220 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd0dc012040 con 0x7fd0d0003f30 2024-03-28T19:07:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.574+0000 7fd0e5600640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c806bc50 0x7fd0c806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.574+0000 7fd0e5600640 1 -- 172.21.15.27:0/23781220 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7fd0dc08b6f0 con 0x7fd0d0003f30 2024-03-28T19:07:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.574+0000 7fd0ed200640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c806bc50 0x7fd0c806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:46.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.575+0000 7fd0ed200640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c806bc50 0x7fd0c806e110 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4034000 tx=0x7fd0d4007000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.581+0000 7fd0e5600640 1 -- 172.21.15.27:0/23781220 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd0dc019030 con 0x7fd0d0003f30 2024-03-28T19:07:46.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.893+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0e80c81f0 con 0x7fd0d0003f30 2024-03-28T19:07:46.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.894+0000 7fd0e5600640 1 -- 172.21.15.27:0/23781220 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5470 (secure 0 0 0) 0x7fd0dc059300 con 0x7fd0d0003f30 2024-03-28T19:07:46.893 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:46.893 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:07:46.893 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-28T19:07:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.896+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c806bc50 msgr2=0x7fd0c806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.896+0000 7fd0ef000640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c806bc50 0x7fd0c806e110 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4034000 tx=0x7fd0d4007000 comp rx=0 tx=0).stop 2024-03-28T19:07:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.896+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 msgr2=0x7fd0e806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.896+0000 7fd0ef000640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0e806fd00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0119a0 tx=0x7fd0dc011e60 comp rx=0 tx=0).stop 2024-03-28T19:07:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.897+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 shutdown_connections 2024-03-28T19:07:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.897+0000 7fd0ef000640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c806bc50 0x7fd0c806e110 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.897+0000 7fd0ef000640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e814cee0 0x7fd0e814d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.897+0000 7fd0ef000640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e814e560 0x7fd0e80c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.897+0000 7fd0ef000640 1 --2- 172.21.15.27:0/23781220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0d0003f30 0x7fd0e806fd00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.897+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 >> 172.21.15.27:0/23781220 conn(0x7fd0e80aa7c0 msgr2=0x7fd0e8156aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.897+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 shutdown_connections 2024-03-28T19:07:46.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:46.897+0000 7fd0ef000640 1 -- 172.21.15.27:0/23781220 wait complete. 2024-03-28T19:07:46.917 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6309 2024-03-28T19:07:46.917 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.a 2024-03-28T19:07:46.917 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-03-28T19:07:46.917 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:07:46.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:07:46.922 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:07:46.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:47.124 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:07:47.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.168+0000 7fab64c00640 1 Processor -- start 2024-03-28T19:07:47.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.169+0000 7fab64c00640 1 -- start start 2024-03-28T19:07:47.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab64c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fab600c5e80 0x7fab600c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab64c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab600c1f50 0x7fab600ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab64c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600abd90 0x7fab600aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab64c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6006d650 con 0x7fab600c1f50 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab64c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6006e1d0 con 0x7fab600c5e80 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab64c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600ac200 con 0x7fab600abd90 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab5d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab600c1f50 0x7fab600ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab5d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab600c1f50 0x7fab600ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37170/0 (socket says 172.21.15.27:37170) 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab5d400640 1 -- 172.21.15.27:0/1537904627 learned_addr learned my addr 172.21.15.27:0/1537904627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab5e800640 1 --2- 172.21.15.27:0/1537904627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600abd90 0x7fab600aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.171+0000 7fab5de00640 1 --2- 172.21.15.27:0/1537904627 >> v2:172.21.15.155:3300/0 conn(0x7fab600c5e80 0x7fab600c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:47.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.172+0000 7fab5d400640 1 -- 172.21.15.27:0/1537904627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600abd90 msgr2=0x7fab600aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:47.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.172+0000 7fab5d400640 1 --2- 172.21.15.27:0/1537904627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600abd90 0x7fab600aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.172+0000 7fab5d400640 1 -- 172.21.15.27:0/1537904627 >> v2:172.21.15.155:3300/0 conn(0x7fab600c5e80 msgr2=0x7fab600c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:47.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.172+0000 7fab5d400640 1 --2- 172.21.15.27:0/1537904627 >> v2:172.21.15.155:3300/0 conn(0x7fab600c5e80 0x7fab600c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.172+0000 7fab5d400640 1 -- 172.21.15.27:0/1537904627 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6006d8a0 con 0x7fab600c1f50 2024-03-28T19:07:47.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.172+0000 7fab5de00640 1 --2- 172.21.15.27:0/1537904627 >> v2:172.21.15.155:3300/0 conn(0x7fab600c5e80 0x7fab600c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:07:47.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.174+0000 7fab5d400640 1 --2- 172.21.15.27:0/1537904627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c1f50 0x7fab600ab850 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fab600c6670 tx=0x7fab4c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07a6cf80929197d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:47.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.174+0000 7fab5ca00640 1 -- 172.21.15.27:0/1537904627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c038070 con 0x7fab600c1f50 2024-03-28T19:07:47.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.174+0000 7fab5ca00640 1 -- 172.21.15.27:0/1537904627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4c02f050 con 0x7fab600c1f50 2024-03-28T19:07:47.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.174+0000 7fab5ca00640 1 -- 172.21.15.27:0/1537904627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c031970 con 0x7fab600c1f50 2024-03-28T19:07:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 -- 172.21.15.27:0/1537904627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c1f50 msgr2=0x7fab600ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:47.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 --2- 172.21.15.27:0/1537904627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c1f50 0x7fab600ab850 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fab600c6670 tx=0x7fab4c02b7f0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 -- 172.21.15.27:0/1537904627 shutdown_connections 2024-03-28T19:07:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 --2- 172.21.15.27:0/1537904627 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600abd90 0x7fab600aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 --2- 172.21.15.27:0/1537904627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c1f50 0x7fab600ab850 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 --2- 172.21.15.27:0/1537904627 >> v2:172.21.15.155:3300/0 conn(0x7fab600c5e80 0x7fab600c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 -- 172.21.15.27:0/1537904627 >> 172.21.15.27:0/1537904627 conn(0x7fab6006b9d0 msgr2=0x7fab600c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 -- 172.21.15.27:0/1537904627 shutdown_connections 2024-03-28T19:07:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.175+0000 7fab64c00640 1 -- 172.21.15.27:0/1537904627 wait complete. 2024-03-28T19:07:47.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.178+0000 7fab64c00640 1 Processor -- start 2024-03-28T19:07:47.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.178+0000 7fab64c00640 1 -- start start 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.180+0000 7fab64c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab600abd90 0x7fab600c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.180+0000 7fab64c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600c1f50 0x7fab600bca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.180+0000 7fab64c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c5e80 0x7fab600bcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.180+0000 7fab64c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600c31f0 con 0x7fab600c5e80 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.180+0000 7fab64c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006de60 con 0x7fab600abd90 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.180+0000 7fab64c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600c3370 con 0x7fab600c1f50 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.180+0000 7fab5e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c5e80 0x7fab600bcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c5e80 0x7fab600bcfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37180/0 (socket says 172.21.15.27:37180) 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5e800640 1 -- 172.21.15.27:0/3604243691 learned_addr learned my addr 172.21.15.27:0/3604243691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:47.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5de00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab600abd90 0x7fab600c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:47.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5d400640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600c1f50 0x7fab600bca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5e800640 1 -- 172.21.15.27:0/3604243691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600c1f50 msgr2=0x7fab600bca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5e800640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600c1f50 0x7fab600bca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5e800640 1 -- 172.21.15.27:0/3604243691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab600abd90 msgr2=0x7fab600c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5e800640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab600abd90 0x7fab600c4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.181+0000 7fab5e800640 1 -- 172.21.15.27:0/3604243691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab601eb0f0 con 0x7fab600c5e80 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.182+0000 7fab5e800640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c5e80 0x7fab600bcfc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fab5800eed0 tx=0x7fab5800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.182+0000 7fab56000640 1 -- 172.21.15.27:0/3604243691 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58018020 con 0x7fab600c5e80 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.182+0000 7fab56000640 1 -- 172.21.15.27:0/3604243691 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5800bd20 con 0x7fab600c5e80 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.182+0000 7fab56000640 1 -- 172.21.15.27:0/3604243691 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58014640 con 0x7fab600c5e80 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.182+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab600bbcc0 con 0x7fab600c5e80 2024-03-28T19:07:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.182+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab600bc280 con 0x7fab600c5e80 2024-03-28T19:07:47.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.184+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab600a9cb0 con 0x7fab600c5e80 2024-03-28T19:07:47.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.186+0000 7fab56000640 1 -- 172.21.15.27:0/3604243691 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fab58005d10 con 0x7fab600c5e80 2024-03-28T19:07:47.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.187+0000 7fab56000640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fab4006bbd0 0x7fab4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:47.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.188+0000 7fab56000640 1 -- 172.21.15.27:0/3604243691 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7fab5808bb50 con 0x7fab600c5e80 2024-03-28T19:07:47.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.188+0000 7fab5de00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fab4006bbd0 0x7fab4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:47.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.189+0000 7fab5de00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fab4006bbd0 0x7fab4006e090 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fab50006e80 tx=0x7fab50008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:47.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.192+0000 7fab56000640 1 -- 172.21.15.27:0/3604243691 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fab600a9cb0 con 0x7fab600c5e80 2024-03-28T19:07:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.561+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab6004f400 con 0x7fab600c5e80 2024-03-28T19:07:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.562+0000 7fab56000640 1 -- 172.21.15.27:0/3604243691 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5470 (secure 0 0 0) 0x7fab58022030 con 0x7fab600c5e80 2024-03-28T19:07:47.562 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:47.562 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:07:47.562 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 101 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.565+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fab4006bbd0 msgr2=0x7fab4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.565+0000 7fab64c00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fab4006bbd0 0x7fab4006e090 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fab50006e80 tx=0x7fab50008040 comp rx=0 tx=0).stop 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.565+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c5e80 msgr2=0x7fab600bcfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.565+0000 7fab64c00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c5e80 0x7fab600bcfc0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fab5800eed0 tx=0x7fab5800b480 comp rx=0 tx=0).stop 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.565+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 shutdown_connections 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.565+0000 7fab64c00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fab4006bbd0 0x7fab4006e090 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.565+0000 7fab64c00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab600c5e80 0x7fab600bcfc0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.565+0000 7fab64c00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab600c1f50 0x7fab600bca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.566+0000 7fab64c00640 1 --2- 172.21.15.27:0/3604243691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab600abd90 0x7fab600c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.566+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 >> 172.21.15.27:0/3604243691 conn(0x7fab6006b9d0 msgr2=0x7fab600c6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:47.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.566+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 shutdown_connections 2024-03-28T19:07:47.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:47.566+0000 7fab64c00640 1 -- 172.21.15.27:0/3604243691 wait complete. 2024-03-28T19:07:47.585 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:07:48.558 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:07:48.559+0000 7fc5f4000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:07:49.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:07:49.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.832+0000 7f8212a00640 1 Processor -- start 2024-03-28T19:07:49.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.832+0000 7f8212a00640 1 -- start start 2024-03-28T19:07:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.834+0000 7f8212a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 0x7f820c160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.834+0000 7f8212a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820c0c52e0 con 0x7f820c0c4dc0 2024-03-28T19:07:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.834+0000 7f8212a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820c1557e0 con 0x7f820c152bd0 2024-03-28T19:07:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.834+0000 7f8212a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c155ac0 con 0x7f820c0c2a80 2024-03-28T19:07:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.835+0000 7f820b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f820c0c4dc0 0x7f820c070540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37174/0 (socket says 172.21.15.27:37174) 2024-03-28T19:07:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.835+0000 7f820b400640 1 -- 172.21.15.27:0/2477607683 learned_addr learned my addr 172.21.15.27:0/2477607683 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:07:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.835+0000 7f8210a00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 0x7f820c160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:49.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.835+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098832944 0 0) 0x7f820c0c52e0 con 0x7f820c0c4dc0 2024-03-28T19:07:49.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f0003520 con 0x7f820c0c4dc0 2024-03-28T19:07:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3471043845 0 0) 0x7f81f0003520 con 0x7f820c0c4dc0 2024-03-28T19:07:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 msgr2=0x7f820c160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 0x7f820c160fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 >> v1:172.21.15.155:6789/0 conn(0x7f820c152bd0 legacy=0x7f820c06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c155da0 con 0x7f820c0c4dc0 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f81f8001670 con 0x7f820c0c4dc0 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 >> v1:172.21.15.27:6789/0 conn(0x7f820c0c4dc0 legacy=0x7f820c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 0x7f81f00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81f0003f20 0x7f81f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81f0004d40 0x7f81f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c155ac0 con 0x7f81f0004d40 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f000ba20 con 0x7f81f0003f20 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.836+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f000bc20 con 0x7f820c0c2a80 2024-03-28T19:07:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f8210a00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81f0004d40 0x7f81f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f820be00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 0x7f81f00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f820b400640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81f0003f20 0x7f81f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f8210a00640 1 -- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 msgr2=0x7f81f00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f8210a00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 0x7f81f00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f8210a00640 1 -- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81f0003f20 msgr2=0x7f81f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f8210a00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81f0003f20 0x7f81f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f8210a00640 1 -- 172.21.15.27:0/2477607683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f0003520 con 0x7f81f0004d40 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.837+0000 7f820be00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f820c0c2a80 0x7f81f00039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.838+0000 7f8210a00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81f0004d40 0x7f81f00051c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8204009ac0 tx=0x7f81f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5328d6186f5dc87b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.839+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204039070 con 0x7f81f0004d40 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.839+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820402c6b0 con 0x7f81f0004d40 2024-03-28T19:07:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.839+0000 7f820aa00640 1 -- 172.21.15.27:0/2477607683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204009690 con 0x7f81f0004d40 2024-03-28T19:07:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.839+0000 7f8212a00640 1 -- 172.21.15.27:0/2477607683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81f0004d40 msgr2=0x7f81f00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.839+0000 7f8212a00640 1 --2- 172.21.15.27:0/2477607683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81f0004d40 0x7f81f00051c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8204009ac0 tx=0x7f81f0005830 comp rx=0 tx=0).stop 2024-03-28T19:07:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.840+0000 7f8210a00640 1 -- 172.21.15.27:0/2477607683 reap_dead start 2024-03-28T19:07:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.840+0000 7f8212a00640 1 -- 172.21.15.27:0/2477607683 shutdown_connections 2024-03-28T19:07:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.840+0000 7f8212a00640 1 -- 172.21.15.27:0/2477607683 >> 172.21.15.27:0/2477607683 conn(0x7f820c066dd0 msgr2=0x7f820c0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.840+0000 7f8212a00640 1 -- 172.21.15.27:0/2477607683 shutdown_connections 2024-03-28T19:07:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.840+0000 7f8212a00640 1 -- 172.21.15.27:0/2477607683 wait complete. 2024-03-28T19:07:49.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.843+0000 7f8212a00640 1 Processor -- start 2024-03-28T19:07:49.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.843+0000 7f8212a00640 1 -- start start 2024-03-28T19:07:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.844+0000 7f8212a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81f0004d40 0x7f820c156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.844+0000 7f8212a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0ab530 0x7f820c158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8212a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0ab9b0 0x7f820c158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8212a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c070c50 con 0x7f820c0ab9b0 2024-03-28T19:07:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8212a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c152fb0 con 0x7f820c0ab530 2024-03-28T19:07:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8212a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c070dd0 con 0x7f81f0004d40 2024-03-28T19:07:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8210a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0ab9b0 0x7f820c158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:49.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8210a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0ab9b0 0x7f820c158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37208/0 (socket says 172.21.15.27:37208) 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8210a00640 1 -- 172.21.15.27:0/685667446 learned_addr learned my addr 172.21.15.27:0/685667446 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f820be00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81f0004d40 0x7f820c156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f820b400640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0ab530 0x7f820c158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8210a00640 1 -- 172.21.15.27:0/685667446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81f0004d40 msgr2=0x7f820c156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8210a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81f0004d40 0x7f820c156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8210a00640 1 -- 172.21.15.27:0/685667446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0ab530 msgr2=0x7f820c158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8210a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0ab530 0x7f820c158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.845+0000 7f8210a00640 1 -- 172.21.15.27:0/685667446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c0c91f0 con 0x7f820c0ab9b0 2024-03-28T19:07:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.846+0000 7f8210a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0ab9b0 0x7f820c158670 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f820402c320 tx=0x7f8204004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.846+0000 7f8203e00640 1 -- 172.21.15.27:0/685667446 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204039040 con 0x7f820c0ab9b0 2024-03-28T19:07:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.846+0000 7f8203e00640 1 -- 172.21.15.27:0/685667446 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8204043070 con 0x7f820c0ab9b0 2024-03-28T19:07:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.846+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c0c8220 con 0x7f820c0ab9b0 2024-03-28T19:07:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.846+0000 7f8203e00640 1 -- 172.21.15.27:0/685667446 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8204004470 con 0x7f820c0ab9b0 2024-03-28T19:07:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.847+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c0c87e0 con 0x7f820c0ab9b0 2024-03-28T19:07:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.848+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c04f400 con 0x7f820c0ab9b0 2024-03-28T19:07:49.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.849+0000 7f8203e00640 1 -- 172.21.15.27:0/685667446 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8204003610 con 0x7f820c0ab9b0 2024-03-28T19:07:49.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.850+0000 7f8203e00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81ec06bbd0 0x7f81ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:07:49.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.851+0000 7f8203e00640 1 -- 172.21.15.27:0/685667446 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7f82040ac4f0 con 0x7f820c0ab9b0 2024-03-28T19:07:49.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.851+0000 7f820be00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81ec06bbd0 0x7f81ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:07:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.852+0000 7f820be00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81ec06bbd0 0x7f81ec06e090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f820c14e280 tx=0x7f81fc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:07:49.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:49.856+0000 7f8203e00640 1 -- 172.21.15.27:0/685667446 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f820404c020 con 0x7f820c0ab9b0 2024-03-28T19:07:50.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.163+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f820c0c9ca0 con 0x7f820c0ab9b0 2024-03-28T19:07:50.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.164+0000 7f8203e00640 1 -- 172.21.15.27:0/685667446 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6262 (secure 0 0 0) 0x7f820407a080 con 0x7f820c0ab9b0 2024-03-28T19:07:50.163 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:07:50.164 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6501,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:07:50.164 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-28T19:07:50.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81ec06bbd0 msgr2=0x7f81ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:50.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81ec06bbd0 0x7f81ec06e090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f820c14e280 tx=0x7f81fc009040 comp rx=0 tx=0).stop 2024-03-28T19:07:50.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0ab9b0 msgr2=0x7f820c158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0ab9b0 0x7f820c158670 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f820402c320 tx=0x7f8204004730 comp rx=0 tx=0).stop 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 shutdown_connections 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81ec06bbd0 0x7f81ec06e090 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f820c0ab9b0 0x7f820c158670 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f820c0ab530 0x7f820c158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 --2- 172.21.15.27:0/685667446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81f0004d40 0x7f820c156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 >> 172.21.15.27:0/685667446 conn(0x7f820c066dd0 msgr2=0x7f820c070330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 shutdown_connections 2024-03-28T19:07:50.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:07:50.167+0000 7f8212a00640 1 -- 172.21.15.27:0/685667446 wait complete. 2024-03-28T19:07:50.189 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-03-28T19:07:50.189 INFO:tasks.mds_thrash.fs.[b]:waiting for 19 secs before thrashing 2024-03-28T19:08:00.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:08:00.967 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:08:09.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.415+0000 7f1a99200640 1 Processor -- start 2024-03-28T19:08:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.415+0000 7f1a99200640 1 -- start start 2024-03-28T19:08:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a99200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a94150c70 0x7f1a940a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a99200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1a940c3690 0x7f1a940c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a99200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a940c3af0 con 0x7f1a940c6660 2024-03-28T19:08:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a99200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a940ab4e0 con 0x7f1a94150c70 2024-03-28T19:08:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a99200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a940ac1f0 con 0x7f1a940c3690 2024-03-28T19:08:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a91e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a94150c70 0x7f1a940a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a92800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1a940c6660 0x7f1a941534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55850/0 (socket says 172.21.15.27:55850) 2024-03-28T19:08:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a91e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a94150c70 0x7f1a940a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55402/0 (socket says 172.21.15.27:55402) 2024-03-28T19:08:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.417+0000 7f1a92800640 1 -- 172.21.15.27:0/738615781 learned_addr learned my addr 172.21.15.27:0/738615781 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.418+0000 7f1a91400640 1 --2- 172.21.15.27:0/738615781 >> v2:172.21.15.155:3300/0 conn(0x7f1a940c3690 0x7f1a940c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.418+0000 7f1a91e00640 1 -- 172.21.15.27:0/738615781 >> v2:172.21.15.155:3300/0 conn(0x7f1a940c3690 msgr2=0x7f1a940c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.418+0000 7f1a91e00640 1 --2- 172.21.15.27:0/738615781 >> v2:172.21.15.155:3300/0 conn(0x7f1a940c3690 0x7f1a940c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.418+0000 7f1a91e00640 1 -- 172.21.15.27:0/738615781 >> v1:172.21.15.27:6789/0 conn(0x7f1a940c6660 legacy=0x7f1a941534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.418+0000 7f1a91e00640 1 -- 172.21.15.27:0/738615781 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a940c3d90 con 0x7f1a94150c70 2024-03-28T19:08:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.419+0000 7f1a91e00640 1 --2- 172.21.15.27:0/738615781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a94150c70 0x7f1a940a9cb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1a940c4180 tx=0x7f1a8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45812aa2b028c85d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.420+0000 7f1a90a00640 1 -- 172.21.15.27:0/738615781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a840315a0 con 0x7f1a94150c70 2024-03-28T19:08:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.420+0000 7f1a90a00640 1 -- 172.21.15.27:0/738615781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a84037070 con 0x7f1a94150c70 2024-03-28T19:08:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.420+0000 7f1a90a00640 1 -- 172.21.15.27:0/738615781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84031960 con 0x7f1a94150c70 2024-03-28T19:08:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.420+0000 7f1a99200640 1 -- 172.21.15.27:0/738615781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a94150c70 msgr2=0x7f1a940a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.420+0000 7f1a99200640 1 --2- 172.21.15.27:0/738615781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a94150c70 0x7f1a940a9cb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1a940c4180 tx=0x7f1a8402b840 comp rx=0 tx=0).stop 2024-03-28T19:08:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.421+0000 7f1a99200640 1 -- 172.21.15.27:0/738615781 shutdown_connections 2024-03-28T19:08:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.421+0000 7f1a99200640 1 --2- 172.21.15.27:0/738615781 >> v2:172.21.15.155:3300/0 conn(0x7f1a940c3690 0x7f1a940c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.421+0000 7f1a99200640 1 --2- 172.21.15.27:0/738615781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a94150c70 0x7f1a940a9cb0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.421+0000 7f1a99200640 1 -- 172.21.15.27:0/738615781 >> 172.21.15.27:0/738615781 conn(0x7f1a9406b9d0 msgr2=0x7f1a940c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.421+0000 7f1a99200640 1 -- 172.21.15.27:0/738615781 shutdown_connections 2024-03-28T19:08:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.421+0000 7f1a99200640 1 -- 172.21.15.27:0/738615781 wait complete. 2024-03-28T19:08:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.423+0000 7f1a99200640 1 Processor -- start 2024-03-28T19:08:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.423+0000 7f1a99200640 1 -- start start 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a99200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a940c3690 0x7f1a9406e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a99200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940c6660 0x7f1a940c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a99200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a94150c70 0x7f1a941eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a99200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940abcd0 con 0x7f1a940c6660 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a99200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940ab9f0 con 0x7f1a940c3690 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a99200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940c1810 con 0x7f1a94150c70 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a91400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940c6660 0x7f1a940c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a91400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940c6660 0x7f1a940c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55406/0 (socket says 172.21.15.27:55406) 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a91400640 1 -- 172.21.15.27:0/2964371837 learned_addr learned my addr 172.21.15.27:0/2964371837 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a91e00640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a940c3690 0x7f1a9406e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.424+0000 7f1a92800640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a94150c70 0x7f1a941eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a91400640 1 -- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a94150c70 msgr2=0x7f1a941eb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a91400640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a94150c70 0x7f1a941eb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a91400640 1 -- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a940c3690 msgr2=0x7f1a9406e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a91400640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a940c3690 0x7f1a9406e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a91400640 1 -- 172.21.15.27:0/2964371837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9406eb00 con 0x7f1a940c6660 2024-03-28T19:08:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a91e00640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a940c3690 0x7f1a9406e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:08:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a91400640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940c6660 0x7f1a940c02a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000e9e0 tx=0x7f1a8000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a8a000640 1 -- 172.21.15.27:0/2964371837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80017070 con 0x7f1a940c6660 2024-03-28T19:08:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a8a000640 1 -- 172.21.15.27:0/2964371837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8000c9d0 con 0x7f1a940c6660 2024-03-28T19:08:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.425+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a941eebd0 con 0x7f1a940c6660 2024-03-28T19:08:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.426+0000 7f1a8a000640 1 -- 172.21.15.27:0/2964371837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000cdc0 con 0x7f1a940c6660 2024-03-28T19:08:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.426+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a940c07e0 con 0x7f1a940c6660 2024-03-28T19:08:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.427+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9404f400 con 0x7f1a940c6660 2024-03-28T19:08:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.428+0000 7f1a8a000640 1 -- 172.21.15.27:0/2964371837 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1a8001a070 con 0x7f1a940c6660 2024-03-28T19:08:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.430+0000 7f1a8a000640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a7406bbd0 0x7f1a7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.430+0000 7f1a8a000640 1 -- 172.21.15.27:0/2964371837 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7f1a8008b670 con 0x7f1a940c6660 2024-03-28T19:08:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.430+0000 7f1a91e00640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a7406bbd0 0x7f1a7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.431+0000 7f1a91e00640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a7406bbd0 0x7f1a7406e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1a84036000 tx=0x7f1a8403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:09.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.436+0000 7f1a8a000640 1 -- 172.21.15.27:0/2964371837 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1a800598f0 con 0x7f1a940c6660 2024-03-28T19:08:09.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.840+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a94070ae0 con 0x7f1a940c6660 2024-03-28T19:08:09.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.842+0000 7f1a8a000640 1 -- 172.21.15.27:0/2964371837 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6262 (secure 0 0 0) 0x7f1a800591f0 con 0x7f1a940c6660 2024-03-28T19:08:09.841 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:09.841 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6501,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:08:09.841 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-28T19:08:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.844+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a7406bbd0 msgr2=0x7f1a7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.844+0000 7f1a99200640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a7406bbd0 0x7f1a7406e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1a84036000 tx=0x7f1a8403d040 comp rx=0 tx=0).stop 2024-03-28T19:08:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.844+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940c6660 msgr2=0x7f1a940c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.844+0000 7f1a99200640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940c6660 0x7f1a940c02a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000e9e0 tx=0x7f1a8000eea0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.845+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 shutdown_connections 2024-03-28T19:08:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.845+0000 7f1a99200640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1a7406bbd0 0x7f1a7406e090 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.845+0000 7f1a99200640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1a94150c70 0x7f1a941eb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.845+0000 7f1a99200640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940c6660 0x7f1a940c02a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.845+0000 7f1a99200640 1 --2- 172.21.15.27:0/2964371837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a940c3690 0x7f1a9406e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:09.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.845+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 >> 172.21.15.27:0/2964371837 conn(0x7f1a9406b9d0 msgr2=0x7f1a940c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:09.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.845+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 shutdown_connections 2024-03-28T19:08:09.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:09.845+0000 7f1a99200640 1 -- 172.21.15.27:0/2964371837 wait complete. 2024-03-28T19:08:09.864 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-03-28T19:08:09.864 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:08:09.864 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:08:09.893 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:08:09.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.160+0000 7fb2f6e00640 1 Processor -- start 2024-03-28T19:08:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.160+0000 7fb2f6e00640 1 -- start start 2024-03-28T19:08:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.162+0000 7fb2f6e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f00a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.162+0000 7fb2f6e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb2f01534c0 0x7fb2f0153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.162+0000 7fb2f6e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f00c3af0 con 0x7fb2f00c3690 2024-03-28T19:08:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.162+0000 7fb2f6e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f00ab4e0 con 0x7fb2f01534c0 2024-03-28T19:08:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.162+0000 7fb2f6e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00ac1f0 con 0x7fb2f0150c70 2024-03-28T19:08:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.162+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f00a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.162+0000 7fb2f4e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb2f01534c0 0x7fb2f0153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.162+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f00a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:34014/0 (socket says 172.21.15.27:34014) 2024-03-28T19:08:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.163+0000 7fb2efe00640 1 -- 172.21.15.27:0/3066204497 learned_addr learned my addr 172.21.15.27:0/3066204497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.163+0000 7fb2efe00640 1 -- 172.21.15.27:0/3066204497 >> v2:172.21.15.155:3300/0 conn(0x7fb2f01534c0 msgr2=0x7fb2f0153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.163+0000 7fb2efe00640 1 --2- 172.21.15.27:0/3066204497 >> v2:172.21.15.155:3300/0 conn(0x7fb2f01534c0 0x7fb2f0153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.163+0000 7fb2efe00640 1 -- 172.21.15.27:0/3066204497 >> v1:172.21.15.155:6789/0 conn(0x7fb2f00c3690 legacy=0x7fb2f00c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.163+0000 7fb2efe00640 1 -- 172.21.15.27:0/3066204497 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f00c3d90 con 0x7fb2f0150c70 2024-03-28T19:08:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.163+0000 7fb2f4e00640 1 --2- 172.21.15.27:0/3066204497 >> v2:172.21.15.155:3300/0 conn(0x7fb2f01534c0 0x7fb2f0153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:08:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.165+0000 7fb2efe00640 1 --2- 172.21.15.27:0/3066204497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f00a9cb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009870 tx=0x7fb2e002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0b85fe4ce605799 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.165+0000 7fb2eea00640 1 -- 172.21.15.27:0/3066204497 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0035020 con 0x7fb2f0150c70 2024-03-28T19:08:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.165+0000 7fb2eea00640 1 -- 172.21.15.27:0/3066204497 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e0031640 con 0x7fb2f0150c70 2024-03-28T19:08:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.165+0000 7fb2eea00640 1 -- 172.21.15.27:0/3066204497 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e002d040 con 0x7fb2f0150c70 2024-03-28T19:08:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.166+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3066204497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 msgr2=0x7fb2f00a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.166+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3066204497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f00a9cb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009870 tx=0x7fb2e002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.166+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3066204497 shutdown_connections 2024-03-28T19:08:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.166+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3066204497 >> v2:172.21.15.155:3300/0 conn(0x7fb2f01534c0 0x7fb2f0153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.166+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3066204497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f00a9cb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.166+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3066204497 >> 172.21.15.27:0/3066204497 conn(0x7fb2f006b9d0 msgr2=0x7fb2f00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.166+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3066204497 shutdown_connections 2024-03-28T19:08:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.166+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3066204497 wait complete. 2024-03-28T19:08:10.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.168+0000 7fb2f6e00640 1 Processor -- start 2024-03-28T19:08:10.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.168+0000 7fb2f6e00640 1 -- start start 2024-03-28T19:08:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.169+0000 7fb2f6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2f00c3690 0x7fb2f0072350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.169+0000 7fb2f6e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f01ec0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.169+0000 7fb2f6e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2f01534c0 0x7fb2f01ec600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.169+0000 7fb2f6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00abcd0 con 0x7fb2f00c3690 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.169+0000 7fb2f6e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00ab9f0 con 0x7fb2f01534c0 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.169+0000 7fb2f6e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00c1810 con 0x7fb2f0150c70 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2f00c3690 0x7fb2f0072350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2f00c3690 0x7fb2f0072350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55416/0 (socket says 172.21.15.27:55416) 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2efe00640 1 -- 172.21.15.27:0/3826514443 learned_addr learned my addr 172.21.15.27:0/3826514443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2f4e00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2f01534c0 0x7fb2f01ec600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2ef400640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f01ec0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2efe00640 1 -- 172.21.15.27:0/3826514443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 msgr2=0x7fb2f01ec0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2efe00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f01ec0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2efe00640 1 -- 172.21.15.27:0/3826514443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2f01534c0 msgr2=0x7fb2f01ec600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2efe00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2f01534c0 0x7fb2f01ec600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.170+0000 7fb2efe00640 1 -- 172.21.15.27:0/3826514443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f01ecb40 con 0x7fb2f00c3690 2024-03-28T19:08:10.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.171+0000 7fb2efe00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2f00c3690 0x7fb2f0072350 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0000c00 tx=0x7fb2e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:10.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.171+0000 7fb2e7e00640 1 -- 172.21.15.27:0/3826514443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e003aa60 con 0x7fb2f00c3690 2024-03-28T19:08:10.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.171+0000 7fb2e7e00640 1 -- 172.21.15.27:0/3826514443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e003ac20 con 0x7fb2f00c3690 2024-03-28T19:08:10.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.171+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f01ef020 con 0x7fb2f00c3690 2024-03-28T19:08:10.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.171+0000 7fb2e7e00640 1 -- 172.21.15.27:0/3826514443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00443f0 con 0x7fb2f00c3690 2024-03-28T19:08:10.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.171+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f014f7d0 con 0x7fb2f00c3690 2024-03-28T19:08:10.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.173+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f004f400 con 0x7fb2f00c3690 2024-03-28T19:08:10.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.174+0000 7fb2e7e00640 1 -- 172.21.15.27:0/3826514443 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb2e0038020 con 0x7fb2f00c3690 2024-03-28T19:08:10.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.175+0000 7fb2e7e00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2d006bc00 0x7fb2d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:10.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.175+0000 7fb2e7e00640 1 -- 172.21.15.27:0/3826514443 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7fb2e00acf30 con 0x7fb2f00c3690 2024-03-28T19:08:10.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.176+0000 7fb2ef400640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2d006bc00 0x7fb2d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:10.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.176+0000 7fb2ef400640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2d006bc00 0x7fb2d006e0c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0072b80 tx=0x7fb2dc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:10.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.180+0000 7fb2e7e00640 1 -- 172.21.15.27:0/3826514443 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb2e003f070 con 0x7fb2f00c3690 2024-03-28T19:08:10.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.460+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2f00c5f00 con 0x7fb2f00c3690 2024-03-28T19:08:10.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.461+0000 7fb2e7e00640 1 -- 172.21.15.27:0/3826514443 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6262 (secure 0 0 0) 0x7fb2e007b2d0 con 0x7fb2f00c3690 2024-03-28T19:08:10.460 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:10.460 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6501,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:08:10.461 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-28T19:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.463+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2d006bc00 msgr2=0x7fb2d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2d006bc00 0x7fb2d006e0c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0072b80 tx=0x7fb2dc009040 comp rx=0 tx=0).stop 2024-03-28T19:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2f00c3690 msgr2=0x7fb2f0072350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2f00c3690 0x7fb2f0072350 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0000c00 tx=0x7fb2e00046c0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 shutdown_connections 2024-03-28T19:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2d006bc00 0x7fb2d006e0c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2f01534c0 0x7fb2f01ec600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2f0150c70 0x7fb2f01ec0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 --2- 172.21.15.27:0/3826514443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2f00c3690 0x7fb2f0072350 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 >> 172.21.15.27:0/3826514443 conn(0x7fb2f006b9d0 msgr2=0x7fb2f01518c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 shutdown_connections 2024-03-28T19:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:10.464+0000 7fb2f6e00640 1 -- 172.21.15.27:0/3826514443 wait complete. 2024-03-28T19:08:10.486 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:08:12.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:12.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.739+0000 7f8a1da00640 1 Processor -- start 2024-03-28T19:08:12.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.739+0000 7f8a1da00640 1 -- start start 2024-03-28T19:08:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.741+0000 7f8a1da00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a18153bd0 0x7f8a180c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.741+0000 7f8a1da00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180c3d90 con 0x7f8a180a9cb0 2024-03-28T19:08:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.741+0000 7f8a1da00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180ab710 con 0x7f8a18150c70 2024-03-28T19:08:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.741+0000 7f8a1da00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180ab4e0 con 0x7f8a18153bd0 2024-03-28T19:08:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.741+0000 7f8a16000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8a180a9cb0 0x7f8a181534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55858/0 (socket says 172.21.15.27:55858) 2024-03-28T19:08:12.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.741+0000 7f8a16000640 1 -- 172.21.15.27:0/2274965119 learned_addr learned my addr 172.21.15.27:0/2274965119 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:12.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.741+0000 7f8a17400640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a18153bd0 0x7f8a180c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463865275 0 0) 0x7f8a180c3d90 con 0x7f8a180a9cb0 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc003520 con 0x7f8a180a9cb0 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1649782730 0 0) 0x7f89fc003520 con 0x7f8a180a9cb0 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a18153bd0 msgr2=0x7f8a180c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a18153bd0 0x7f8a180c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 >> v1:172.21.15.155:6789/0 conn(0x7f8a18150c70 legacy=0x7f8a180c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a180ab9f0 con 0x7f8a180a9cb0 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f8a04001670 con 0x7f8a180a9cb0 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 >> v1:172.21.15.27:6789/0 conn(0x7f8a180a9cb0 legacy=0x7f8a181534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc0039e0 0x7f89fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc004800 0x7f89fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a18153bd0 0x7f89fc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc00bac0 con 0x7f89fc004800 2024-03-28T19:08:12.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc00bc70 con 0x7f89fc0039e0 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.742+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89fc00be70 con 0x7f8a18153bd0 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.743+0000 7f8a16000640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc004800 0x7f89fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.743+0000 7f8a16a00640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc0039e0 0x7f89fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.743+0000 7f8a17400640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a18153bd0 0x7f89fc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.743+0000 7f8a16000640 1 -- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a18153bd0 msgr2=0x7f89fc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.743+0000 7f8a16000640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a18153bd0 0x7f89fc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.743+0000 7f8a16000640 1 -- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc0039e0 msgr2=0x7f89fc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.743+0000 7f8a16000640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc0039e0 0x7f89fc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.743+0000 7f8a16000640 1 -- 172.21.15.27:0/2274965119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc003520 con 0x7f89fc004800 2024-03-28T19:08:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.744+0000 7f8a16000640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc004800 0x7f89fc004c80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8a0400c240 tx=0x7f89fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26dcc151b2bd4138 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.745+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04007d10 con 0x7f89fc004800 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.745+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a04030070 con 0x7f89fc004800 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.745+0000 7f8a15600640 1 -- 172.21.15.27:0/2274965119 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0402cd60 con 0x7f89fc004800 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.745+0000 7f8a1da00640 1 -- 172.21.15.27:0/2274965119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc004800 msgr2=0x7f89fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.745+0000 7f8a1da00640 1 --2- 172.21.15.27:0/2274965119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89fc004800 0x7f89fc004c80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f8a0400c240 tx=0x7f89fc005830 comp rx=0 tx=0).stop 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.745+0000 7f8a17400640 1 -- 172.21.15.27:0/2274965119 reap_dead start 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.745+0000 7f8a1da00640 1 -- 172.21.15.27:0/2274965119 shutdown_connections 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.745+0000 7f8a1da00640 1 -- 172.21.15.27:0/2274965119 >> 172.21.15.27:0/2274965119 conn(0x7f8a1806b9d0 msgr2=0x7f8a180c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.746+0000 7f8a1da00640 1 -- 172.21.15.27:0/2274965119 shutdown_connections 2024-03-28T19:08:12.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.746+0000 7f8a1da00640 1 -- 172.21.15.27:0/2274965119 wait complete. 2024-03-28T19:08:12.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.747+0000 7f8a1da00640 1 Processor -- start 2024-03-28T19:08:12.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.748+0000 7f8a1da00640 1 -- start start 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.748+0000 7f8a1da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc004800 0x7f8a180a88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a1da00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a181ee350 0x7f8a181eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a1da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a18157170 0x7f8a181eb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a1da00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18154060 con 0x7f8a18157170 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a1da00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180ac1f0 con 0x7f89fc004800 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a1da00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180c1810 con 0x7f8a181ee350 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a17400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a18157170 0x7f8a181eb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a17400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a18157170 0x7f8a181eb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55426/0 (socket says 172.21.15.27:55426) 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a17400640 1 -- 172.21.15.27:0/2460978899 learned_addr learned my addr 172.21.15.27:0/2460978899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a16000640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a181ee350 0x7f8a181eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:12.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a16a00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc004800 0x7f8a180a88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a17400640 1 -- 172.21.15.27:0/2460978899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a181ee350 msgr2=0x7f8a181eb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a17400640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a181ee350 0x7f8a181eb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a17400640 1 -- 172.21.15.27:0/2460978899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc004800 msgr2=0x7f8a180a88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.749+0000 7f8a17400640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc004800 0x7f8a180a88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.750+0000 7f8a17400640 1 -- 172.21.15.27:0/2460978899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a181ebd60 con 0x7f8a18157170 2024-03-28T19:08:12.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.750+0000 7f8a17400640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a18157170 0x7f8a181eb820 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8a180c5c30 tx=0x7f8a10011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.750+0000 7f8a0ea00640 1 -- 172.21.15.27:0/2460978899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10019020 con 0x7f8a18157170 2024-03-28T19:08:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.750+0000 7f8a0ea00640 1 -- 172.21.15.27:0/2460978899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a10003610 con 0x7f8a18157170 2024-03-28T19:08:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.750+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a181eade0 con 0x7f8a18157170 2024-03-28T19:08:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.750+0000 7f8a0ea00640 1 -- 172.21.15.27:0/2460978899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10003a40 con 0x7f8a18157170 2024-03-28T19:08:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.750+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a180c3990 con 0x7f8a18157170 2024-03-28T19:08:12.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.752+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1804f400 con 0x7f8a18157170 2024-03-28T19:08:12.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.753+0000 7f8a0ea00640 1 -- 172.21.15.27:0/2460978899 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8a1001c020 con 0x7f8a18157170 2024-03-28T19:08:12.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.754+0000 7f8a0ea00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f89f806bbd0 0x7f89f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:12.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.754+0000 7f8a0ea00640 1 -- 172.21.15.27:0/2460978899 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7f8a1008b3a0 con 0x7f8a18157170 2024-03-28T19:08:12.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.754+0000 7f8a16a00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f89f806bbd0 0x7f89f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:12.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.755+0000 7f8a16a00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f89f806bbd0 0x7f89f806e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8a08007fd0 tx=0x7f8a08009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:12.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:12.760+0000 7f8a0ea00640 1 -- 172.21.15.27:0/2460978899 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8a10023070 con 0x7f8a18157170 2024-03-28T19:08:13.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.022+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a180c5f00 con 0x7f8a18157170 2024-03-28T19:08:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.024+0000 7f8a0ea00640 1 -- 172.21.15.27:0/2460978899 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6262 (secure 0 0 0) 0x7f8a181ebd60 con 0x7f8a18157170 2024-03-28T19:08:13.023 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:13.023 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6501,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:08:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-28T19:08:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.026+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f89f806bbd0 msgr2=0x7f89f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.026+0000 7f8a1da00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f89f806bbd0 0x7f89f806e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8a08007fd0 tx=0x7f8a08009040 comp rx=0 tx=0).stop 2024-03-28T19:08:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.026+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a18157170 msgr2=0x7f8a181eb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:13.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a18157170 0x7f8a181eb820 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8a180c5c30 tx=0x7f8a10011920 comp rx=0 tx=0).stop 2024-03-28T19:08:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 shutdown_connections 2024-03-28T19:08:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f89f806bbd0 0x7f89f806e090 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a18157170 0x7f8a181eb820 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a181ee350 0x7f8a181eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 --2- 172.21.15.27:0/2460978899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89fc004800 0x7f8a180a88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 >> 172.21.15.27:0/2460978899 conn(0x7f8a1806b9d0 msgr2=0x7f8a18153340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 shutdown_connections 2024-03-28T19:08:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:13.027+0000 7f8a1da00640 1 -- 172.21.15.27:0/2460978899 wait complete. 2024-03-28T19:08:13.046 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:08:15.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.277+0000 7fe66b200640 1 Processor -- start 2024-03-28T19:08:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.277+0000 7fe66b200640 1 -- start start 2024-03-28T19:08:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.278+0000 7fe66b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe664153bd0 0x7fe6640c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.278+0000 7fe66b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6640c3d90 con 0x7fe664150c70 2024-03-28T19:08:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.279+0000 7fe66b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6640ab710 con 0x7fe6640a9cb0 2024-03-28T19:08:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.279+0000 7fe66b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640ab4e0 con 0x7fe664153bd0 2024-03-28T19:08:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.279+0000 7fe663e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe664150c70 0x7fe6640c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55864/0 (socket says 172.21.15.27:55864) 2024-03-28T19:08:15.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.279+0000 7fe663e00640 1 -- 172.21.15.27:0/2088784214 learned_addr learned my addr 172.21.15.27:0/2088784214 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:15.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.279+0000 7fe669200640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe664153bd0 0x7fe6640c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:15.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.279+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965284180 0 0) 0x7fe6640c3d90 con 0x7fe664150c70 2024-03-28T19:08:15.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.279+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe648007930 con 0x7fe664150c70 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266426061 0 0) 0x7fe648007930 con 0x7fe664150c70 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe664153bd0 msgr2=0x7fe6640c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe664153bd0 0x7fe6640c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 >> v1:172.21.15.155:6789/0 conn(0x7fe6640a9cb0 legacy=0x7fe6641534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6640ab9f0 con 0x7fe664150c70 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe654001670 con 0x7fe664150c70 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 >> v1:172.21.15.27:6789/0 conn(0x7fe664150c70 legacy=0x7fe6640c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648007df0 0x7fe648008210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe664153bd0 0x7fe648008c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe648009150 0x7fe6480095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64800ff60 con 0x7fe648009150 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480100e0 con 0x7fe648007df0 2024-03-28T19:08:15.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.280+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480102e0 con 0x7fe664153bd0 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe669200640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe648009150 0x7fe6480095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe663e00640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648007df0 0x7fe648008210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe663400640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe664153bd0 0x7fe648008c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe669200640 1 -- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe664153bd0 msgr2=0x7fe648008c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe669200640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe664153bd0 0x7fe648008c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe669200640 1 -- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648007df0 msgr2=0x7fe648008210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe669200640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648007df0 0x7fe648008210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe669200640 1 -- 172.21.15.27:0/2088784214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe648007930 con 0x7fe648009150 2024-03-28T19:08:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.281+0000 7fe663e00640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648007df0 0x7fe648008210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:08:15.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.282+0000 7fe669200640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe648009150 0x7fe6480095d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009fc0 tx=0x7fe648009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c315531c4db42f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:15.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.283+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c039070 con 0x7fe648009150 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.283+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c02cc20 con 0x7fe648009150 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.283+0000 7fe662a00640 1 -- 172.21.15.27:0/2088784214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c009c10 con 0x7fe648009150 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.283+0000 7fe66b200640 1 -- 172.21.15.27:0/2088784214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe648009150 msgr2=0x7fe6480095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.283+0000 7fe66b200640 1 --2- 172.21.15.27:0/2088784214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe648009150 0x7fe6480095d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe65c009fc0 tx=0x7fe648009c40 comp rx=0 tx=0).stop 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.284+0000 7fe669200640 1 -- 172.21.15.27:0/2088784214 reap_dead start 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.284+0000 7fe66b200640 1 -- 172.21.15.27:0/2088784214 shutdown_connections 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.284+0000 7fe66b200640 1 -- 172.21.15.27:0/2088784214 >> 172.21.15.27:0/2088784214 conn(0x7fe66406b9d0 msgr2=0x7fe6640c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.284+0000 7fe66b200640 1 -- 172.21.15.27:0/2088784214 shutdown_connections 2024-03-28T19:08:15.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.284+0000 7fe66b200640 1 -- 172.21.15.27:0/2088784214 wait complete. 2024-03-28T19:08:15.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.286+0000 7fe66b200640 1 Processor -- start 2024-03-28T19:08:15.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.286+0000 7fe66b200640 1 -- start start 2024-03-28T19:08:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.287+0000 7fe66b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe66414f9e0 0x7fe6641eae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.287+0000 7fe66b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6641541e0 0x7fe66414af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.287+0000 7fe66b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe66414b4d0 0x7fe6640a8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.287+0000 7fe66b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe664154060 con 0x7fe66414b4d0 2024-03-28T19:08:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.287+0000 7fe66b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640ac1f0 con 0x7fe66414f9e0 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.287+0000 7fe66b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640c1810 con 0x7fe6641541e0 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe669200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe66414b4d0 0x7fe6640a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe669200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe66414b4d0 0x7fe6640a8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55440/0 (socket says 172.21.15.27:55440) 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe669200640 1 -- 172.21.15.27:0/1082672334 learned_addr learned my addr 172.21.15.27:0/1082672334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe663e00640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe66414f9e0 0x7fe6641eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe663400640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6641541e0 0x7fe66414af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe669200640 1 -- 172.21.15.27:0/1082672334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6641541e0 msgr2=0x7fe66414af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe669200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6641541e0 0x7fe66414af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe669200640 1 -- 172.21.15.27:0/1082672334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe66414f9e0 msgr2=0x7fe6641eae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe669200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe66414f9e0 0x7fe6641eae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.288+0000 7fe669200640 1 -- 172.21.15.27:0/1082672334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6641eb3b0 con 0x7fe66414b4d0 2024-03-28T19:08:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.289+0000 7fe669200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe66414b4d0 0x7fe6640a8940 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00e230 tx=0x7fe65c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.289+0000 7fe65be00640 1 -- 172.21.15.27:0/1082672334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c039070 con 0x7fe66414b4d0 2024-03-28T19:08:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.289+0000 7fe65be00640 1 -- 172.21.15.27:0/1082672334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c03a040 con 0x7fe66414b4d0 2024-03-28T19:08:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.289+0000 7fe65be00640 1 -- 172.21.15.27:0/1082672334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c009c10 con 0x7fe66414b4d0 2024-03-28T19:08:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.289+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6641555b0 con 0x7fe66414b4d0 2024-03-28T19:08:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.289+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6640a7260 con 0x7fe66414b4d0 2024-03-28T19:08:15.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.291+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66404f400 con 0x7fe66414b4d0 2024-03-28T19:08:15.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.292+0000 7fe65be00640 1 -- 172.21.15.27:0/1082672334 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe65c046030 con 0x7fe66414b4d0 2024-03-28T19:08:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.293+0000 7fe65be00640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64406bc50 0x7fe64406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.293+0000 7fe65be00640 1 -- 172.21.15.27:0/1082672334 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7fe65c0ad2a0 con 0x7fe66414b4d0 2024-03-28T19:08:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.293+0000 7fe663e00640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64406bc50 0x7fe64406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:15.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.294+0000 7fe663e00640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64406bc50 0x7fe64406e110 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe654000ff0 tx=0x7fe654009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:15.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.298+0000 7fe65be00640 1 -- 172.21.15.27:0/1082672334 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe65c04d020 con 0x7fe66414b4d0 2024-03-28T19:08:15.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.574+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6640c5f00 con 0x7fe66414b4d0 2024-03-28T19:08:15.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.575+0000 7fe65be00640 1 -- 172.21.15.27:0/1082672334 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6262 (secure 0 0 0) 0x7fe65c07b500 con 0x7fe66414b4d0 2024-03-28T19:08:15.574 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:15.575 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6501,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:08:15.575 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64406bc50 msgr2=0x7fe64406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64406bc50 0x7fe64406e110 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe654000ff0 tx=0x7fe654009040 comp rx=0 tx=0).stop 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe66414b4d0 msgr2=0x7fe6640a8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe66414b4d0 0x7fe6640a8940 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00e230 tx=0x7fe65c009750 comp rx=0 tx=0).stop 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 shutdown_connections 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64406bc50 0x7fe64406e110 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe66414b4d0 0x7fe6640a8940 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6641541e0 0x7fe66414af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 --2- 172.21.15.27:0/1082672334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe66414f9e0 0x7fe6641eae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 >> 172.21.15.27:0/1082672334 conn(0x7fe66406b9d0 msgr2=0x7fe6640711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:15.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.578+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 shutdown_connections 2024-03-28T19:08:15.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:15.579+0000 7fe66b200640 1 -- 172.21.15.27:0/1082672334 wait complete. 2024-03-28T19:08:15.599 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:08:17.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.825+0000 7fe0b2000640 1 Processor -- start 2024-03-28T19:08:17.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.825+0000 7fe0b2000640 1 -- start start 2024-03-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.826+0000 7fe0b2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0ac152f80 0x7fe0ac0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.826+0000 7fe0b2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 0x7fe0ac0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.826+0000 7fe0b2000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0c7310 con 0x7fe0ac155730 2024-03-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.826+0000 7fe0b2000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ac158d40 con 0x7fe0ac152f80 2024-03-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.826+0000 7fe0b2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac159020 con 0x7fe0ac0c7970 2024-03-28T19:08:17.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.827+0000 7fe0ab400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0ac152f80 0x7fe0ac0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.827+0000 7fe0ab400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0ac152f80 0x7fe0ac0c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58890/0 (socket says 172.21.15.27:58890) 2024-03-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.827+0000 7fe0ab400640 1 -- 172.21.15.27:0/385410530 learned_addr learned my addr 172.21.15.27:0/385410530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.827+0000 7fe0ab400640 1 -- 172.21.15.27:0/385410530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 msgr2=0x7fe0ac0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.827+0000 7fe0ab400640 1 --2- 172.21.15.27:0/385410530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 0x7fe0ac0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.827+0000 7fe0ab400640 1 -- 172.21.15.27:0/385410530 >> v1:172.21.15.27:6789/0 conn(0x7fe0ac155730 legacy=0x7fe0ac155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.827+0000 7fe0ab400640 1 -- 172.21.15.27:0/385410530 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac159300 con 0x7fe0ac152f80 2024-03-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.829+0000 7fe0ab400640 1 --2- 172.21.15.27:0/385410530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac152f80 0x7fe0ac0c58e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009870 tx=0x7fe09c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc007b8a977ffb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.829+0000 7fe0aa000640 1 -- 172.21.15.27:0/385410530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0317c0 con 0x7fe0ac152f80 2024-03-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.829+0000 7fe0aa000640 1 -- 172.21.15.27:0/385410530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c037070 con 0x7fe0ac152f80 2024-03-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.829+0000 7fe0aa000640 1 -- 172.21.15.27:0/385410530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c031b80 con 0x7fe0ac152f80 2024-03-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.830+0000 7fe0b2000640 1 -- 172.21.15.27:0/385410530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac152f80 msgr2=0x7fe0ac0c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:17.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.830+0000 7fe0b2000640 1 --2- 172.21.15.27:0/385410530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac152f80 0x7fe0ac0c58e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009870 tx=0x7fe09c02b860 comp rx=0 tx=0).stop 2024-03-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.830+0000 7fe0b2000640 1 -- 172.21.15.27:0/385410530 shutdown_connections 2024-03-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.830+0000 7fe0b2000640 1 --2- 172.21.15.27:0/385410530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 0x7fe0ac0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.830+0000 7fe0b2000640 1 --2- 172.21.15.27:0/385410530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac152f80 0x7fe0ac0c58e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.830+0000 7fe0b2000640 1 -- 172.21.15.27:0/385410530 >> 172.21.15.27:0/385410530 conn(0x7fe0ac06b6f0 msgr2=0x7fe0ac0caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.830+0000 7fe0b2000640 1 -- 172.21.15.27:0/385410530 shutdown_connections 2024-03-28T19:08:17.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.830+0000 7fe0b2000640 1 -- 172.21.15.27:0/385410530 wait complete. 2024-03-28T19:08:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.832+0000 7fe0b2000640 1 Processor -- start 2024-03-28T19:08:17.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.832+0000 7fe0b2000640 1 -- start start 2024-03-28T19:08:17.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.833+0000 7fe0b2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 0x7fe0ac0c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.833+0000 7fe0b2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0ac152f80 0x7fe0ac0caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.833+0000 7fe0b2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac155730 0x7fe0ac0c3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.833+0000 7fe0b2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac159b00 con 0x7fe0ac155730 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.833+0000 7fe0b2000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac0c5e20 con 0x7fe0ac152f80 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.833+0000 7fe0b2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac0cbca0 con 0x7fe0ac0c7970 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac155730 0x7fe0ac0c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac155730 0x7fe0ac0c3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58900/0 (socket says 172.21.15.27:58900) 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0abe00640 1 -- 172.21.15.27:0/3594509205 learned_addr learned my addr 172.21.15.27:0/3594509205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:17.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0ab400640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 0x7fe0ac0c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0aaa00640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0ac152f80 0x7fe0ac0caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0abe00640 1 -- 172.21.15.27:0/3594509205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 msgr2=0x7fe0ac0c18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0abe00640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 0x7fe0ac0c18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0abe00640 1 -- 172.21.15.27:0/3594509205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0ac152f80 msgr2=0x7fe0ac0caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0abe00640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0ac152f80 0x7fe0ac0caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.834+0000 7fe0abe00640 1 -- 172.21.15.27:0/3594509205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac14fd30 con 0x7fe0ac155730 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.835+0000 7fe0abe00640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac155730 0x7fe0ac0c3680 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400c780 tx=0x7fe0a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.835+0000 7fe0a3400640 1 -- 172.21.15.27:0/3594509205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4004b10 con 0x7fe0ac155730 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.835+0000 7fe0a3400640 1 -- 172.21.15.27:0/3594509205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a4004cb0 con 0x7fe0ac155730 2024-03-28T19:08:17.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.835+0000 7fe0a3400640 1 -- 172.21.15.27:0/3594509205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a400aa90 con 0x7fe0ac155730 2024-03-28T19:08:17.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.835+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac0c46d0 con 0x7fe0ac155730 2024-03-28T19:08:17.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.835+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac0cafe0 con 0x7fe0ac155730 2024-03-28T19:08:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.837+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac04f400 con 0x7fe0ac155730 2024-03-28T19:08:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.838+0000 7fe0a3400640 1 -- 172.21.15.27:0/3594509205 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe0a400ac30 con 0x7fe0ac155730 2024-03-28T19:08:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.838+0000 7fe0a3400640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08c06bc50 0x7fe08c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.839+0000 7fe0a3400640 1 -- 172.21.15.27:0/3594509205 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7fe0a408b3c0 con 0x7fe0ac155730 2024-03-28T19:08:17.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.839+0000 7fe0ab400640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08c06bc50 0x7fe08c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:17.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.840+0000 7fe0ab400640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08c06bc50 0x7fe08c06e110 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe09c036000 tx=0x7fe09c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:17.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:17.845+0000 7fe0a3400640 1 -- 172.21.15.27:0/3594509205 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe0a4020070 con 0x7fe0ac155730 2024-03-28T19:08:18.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.120+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0ac0cba80 con 0x7fe0ac155730 2024-03-28T19:08:18.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.120+0000 7fe0a3400640 1 -- 172.21.15.27:0/3594509205 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6262 (secure 0 0 0) 0x7fe0a4059720 con 0x7fe0ac155730 2024-03-28T19:08:18.121 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-28T19:08:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:18.123 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6501,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:08:18.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08c06bc50 msgr2=0x7fe08c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:18.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08c06bc50 0x7fe08c06e110 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe09c036000 tx=0x7fe09c03d040 comp rx=0 tx=0).stop 2024-03-28T19:08:18.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac155730 msgr2=0x7fe0ac0c3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac155730 0x7fe0ac0c3680 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400c780 tx=0x7fe0a400cc40 comp rx=0 tx=0).stop 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 shutdown_connections 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe08c06bc50 0x7fe08c06e110 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0ac155730 0x7fe0ac0c3680 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0ac152f80 0x7fe0ac0caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 --2- 172.21.15.27:0/3594509205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0ac0c7970 0x7fe0ac0c18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 >> 172.21.15.27:0/3594509205 conn(0x7fe0ac06b6f0 msgr2=0x7fe0ac151130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 shutdown_connections 2024-03-28T19:08:18.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:18.124+0000 7fe0b2000640 1 -- 172.21.15.27:0/3594509205 wait complete. 2024-03-28T19:08:18.145 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:08:20.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.398+0000 7f89b2000640 1 Processor -- start 2024-03-28T19:08:20.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.398+0000 7f89b2000640 1 -- start start 2024-03-28T19:08:20.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89b2000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f89ac0ae3d0 0x7f89ac0c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:20.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89b2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89ac0aebe0 0x7f89ac153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89b2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae7b0 0x7f89ac154000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89b2000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac153630 con 0x7f89ac0aebe0 2024-03-28T19:08:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89b2000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0c5460 con 0x7f89ac0ae3d0 2024-03-28T19:08:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89b2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac0c4bc0 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89abe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89ac0aebe0 0x7f89ac153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89abe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89ac0aebe0 0x7f89ac153ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58916/0 (socket says 172.21.15.27:58916) 2024-03-28T19:08:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.400+0000 7f89abe00640 1 -- 172.21.15.27:0/44692552 learned_addr learned my addr 172.21.15.27:0/44692552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:20.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.401+0000 7f89b1600640 1 --2- 172.21.15.27:0/44692552 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae7b0 0x7f89ac154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:20.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.401+0000 7f89b0c00640 1 --2- 172.21.15.27:0/44692552 >> v2:172.21.15.155:3300/0 conn(0x7f89ac0ae3d0 0x7f89ac0c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:20.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.401+0000 7f89abe00640 1 -- 172.21.15.27:0/44692552 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae7b0 msgr2=0x7f89ac154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:20.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.401+0000 7f89abe00640 1 --2- 172.21.15.27:0/44692552 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae7b0 0x7f89ac154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.401+0000 7f89abe00640 1 -- 172.21.15.27:0/44692552 >> v2:172.21.15.155:3300/0 conn(0x7f89ac0ae3d0 msgr2=0x7f89ac0c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:20.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.401+0000 7f89abe00640 1 --2- 172.21.15.27:0/44692552 >> v2:172.21.15.155:3300/0 conn(0x7f89ac0ae3d0 0x7f89ac0c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.401+0000 7f89abe00640 1 -- 172.21.15.27:0/44692552 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac0c4ea0 con 0x7f89ac0aebe0 2024-03-28T19:08:20.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.401+0000 7f89b0c00640 1 --2- 172.21.15.27:0/44692552 >> v2:172.21.15.155:3300/0 conn(0x7f89ac0ae3d0 0x7f89ac0c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:08:20.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.402+0000 7f89abe00640 1 --2- 172.21.15.27:0/44692552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0aebe0 0x7f89ac153ac0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f89ac06e1d0 tx=0x7f899802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf21ff1e457c9f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.402+0000 7f89ab400640 1 -- 172.21.15.27:0/44692552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998035020 con 0x7f89ac0aebe0 2024-03-28T19:08:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89ab400640 1 -- 172.21.15.27:0/44692552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899802d040 con 0x7f89ac0aebe0 2024-03-28T19:08:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89ab400640 1 -- 172.21.15.27:0/44692552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998031a80 con 0x7f89ac0aebe0 2024-03-28T19:08:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 -- 172.21.15.27:0/44692552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0aebe0 msgr2=0x7f89ac153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 --2- 172.21.15.27:0/44692552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0aebe0 0x7f89ac153ac0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f89ac06e1d0 tx=0x7f899802b900 comp rx=0 tx=0).stop 2024-03-28T19:08:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 -- 172.21.15.27:0/44692552 shutdown_connections 2024-03-28T19:08:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 --2- 172.21.15.27:0/44692552 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae7b0 0x7f89ac154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 --2- 172.21.15.27:0/44692552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0aebe0 0x7f89ac153ac0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 --2- 172.21.15.27:0/44692552 >> v2:172.21.15.155:3300/0 conn(0x7f89ac0ae3d0 0x7f89ac0c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 -- 172.21.15.27:0/44692552 >> 172.21.15.27:0/44692552 conn(0x7f89ac072d10 msgr2=0x7f89ac06d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 -- 172.21.15.27:0/44692552 shutdown_connections 2024-03-28T19:08:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.403+0000 7f89b2000640 1 -- 172.21.15.27:0/44692552 wait complete. 2024-03-28T19:08:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.405+0000 7f89b2000640 1 Processor -- start 2024-03-28T19:08:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.406+0000 7f89b2000640 1 -- start start 2024-03-28T19:08:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89b2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae3d0 0x7f89ac14e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89b2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0ae7b0 0x7f89ac0ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89b2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89ac0aebe0 0x7f89ac06c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89b2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac154890 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89b2000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac070890 con 0x7f89ac0aebe0 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89b2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac154a10 con 0x7f89ac0ae3d0 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0ae7b0 0x7f89ac0ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0ae7b0 0x7f89ac0ad240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58922/0 (socket says 172.21.15.27:58922) 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89abe00640 1 -- 172.21.15.27:0/2857890761 learned_addr learned my addr 172.21.15.27:0/2857890761 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89b0c00640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae3d0 0x7f89ac14e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.407+0000 7f89b1600640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89ac0aebe0 0x7f89ac06c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.408+0000 7f89abe00640 1 -- 172.21.15.27:0/2857890761 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae3d0 msgr2=0x7f89ac14e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.408+0000 7f89abe00640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae3d0 0x7f89ac14e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.408+0000 7f89abe00640 1 -- 172.21.15.27:0/2857890761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89ac0aebe0 msgr2=0x7f89ac06c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.408+0000 7f89abe00640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89ac0aebe0 0x7f89ac06c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.408+0000 7f89abe00640 1 -- 172.21.15.27:0/2857890761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac0ad780 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.408+0000 7f89abe00640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0ae7b0 0x7f89ac0ad240 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f899802b940 tx=0x7f89980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.408+0000 7f89a8c00640 1 -- 172.21.15.27:0/2857890761 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998040070 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.409+0000 7f89a8c00640 1 -- 172.21.15.27:0/2857890761 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8998039c10 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.409+0000 7f89a8c00640 1 -- 172.21.15.27:0/2857890761 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998035040 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.409+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac0ac320 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.409+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac0ac8e0 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.410+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac067dd0 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.411+0000 7f89a8c00640 1 -- 172.21.15.27:0/2857890761 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f899804b060 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.412+0000 7f89a8c00640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f898c06bbd0 0x7f898c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:20.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.412+0000 7f89a8c00640 1 -- 172.21.15.27:0/2857890761 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7f89980acea0 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.413+0000 7f89b0c00640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f898c06bbd0 0x7f898c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:20.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.414+0000 7f89b0c00640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f898c06bbd0 0x7f898c06e090 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f89a000a980 tx=0x7f89a0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:20.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.419+0000 7f89a8c00640 1 -- 172.21.15.27:0/2857890761 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f899807b120 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.693+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89ac069640 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.694+0000 7f89a8c00640 1 -- 172.21.15.27:0/2857890761 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6262 (secure 0 0 0) 0x7f899807aa20 con 0x7f89ac0ae7b0 2024-03-28T19:08:20.693 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:20.694 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6501,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:08:20.694 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-28T19:08:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.696+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f898c06bbd0 msgr2=0x7f898c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.696+0000 7f89b2000640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f898c06bbd0 0x7f898c06e090 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f89a000a980 tx=0x7f89a0008040 comp rx=0 tx=0).stop 2024-03-28T19:08:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.696+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0ae7b0 msgr2=0x7f89ac0ad240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.696+0000 7f89b2000640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0ae7b0 0x7f89ac0ad240 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f899802b940 tx=0x7f89980046c0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.697+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 shutdown_connections 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.697+0000 7f89b2000640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f898c06bbd0 0x7f898c06e090 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.697+0000 7f89b2000640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89ac0aebe0 0x7f89ac06c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.697+0000 7f89b2000640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89ac0ae7b0 0x7f89ac0ad240 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.697+0000 7f89b2000640 1 --2- 172.21.15.27:0/2857890761 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89ac0ae3d0 0x7f89ac14e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.697+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 >> 172.21.15.27:0/2857890761 conn(0x7f89ac072d10 msgr2=0x7f89ac153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.697+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 shutdown_connections 2024-03-28T19:08:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:20.697+0000 7f89b2000640 1 -- 172.21.15.27:0/2857890761 wait complete. 2024-03-28T19:08:20.718 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:08:22.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:22.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.958+0000 7f7826e00640 1 Processor -- start 2024-03-28T19:08:22.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.959+0000 7f7826e00640 1 -- start start 2024-03-28T19:08:22.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.960+0000 7f7826e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7820150ae0 0x7f78200a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:22.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.960+0000 7f7826e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78200a61a0 0x7f78200c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:22.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.960+0000 7f7826e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7820150ec0 con 0x7f78200c6ee0 2024-03-28T19:08:22.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.960+0000 7f7826e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782006dc70 con 0x7f78200a61a0 2024-03-28T19:08:22.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.960+0000 7f7826e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7820153550 con 0x7f7820150ae0 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.961+0000 7f7825000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78200a61a0 0x7f78200c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.961+0000 7f7825000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78200a61a0 0x7f78200c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58934/0 (socket says 172.21.15.27:58934) 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.961+0000 7f7825000640 1 -- 172.21.15.27:0/2958762861 learned_addr learned my addr 172.21.15.27:0/2958762861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.961+0000 7f7825a00640 1 --2- 172.21.15.27:0/2958762861 >> v2:172.21.15.155:3300/0 conn(0x7f7820150ae0 0x7f78200a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.961+0000 7f7825000640 1 -- 172.21.15.27:0/2958762861 >> v2:172.21.15.155:3300/0 conn(0x7f7820150ae0 msgr2=0x7f78200a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.961+0000 7f7825000640 1 --2- 172.21.15.27:0/2958762861 >> v2:172.21.15.155:3300/0 conn(0x7f7820150ae0 0x7f78200a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.961+0000 7f7825000640 1 -- 172.21.15.27:0/2958762861 >> v1:172.21.15.155:6789/0 conn(0x7f78200c6ee0 legacy=0x7f7820153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.961+0000 7f7825000640 1 -- 172.21.15.27:0/2958762861 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782006df10 con 0x7f78200a61a0 2024-03-28T19:08:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.963+0000 7f7825000640 1 --2- 172.21.15.27:0/2958762861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200a61a0 0x7f78200c69a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f78200a9760 tx=0x7f780c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf00e85ead5303b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:22.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.963+0000 7f781fe00640 1 -- 172.21.15.27:0/2958762861 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c02e070 con 0x7f78200a61a0 2024-03-28T19:08:22.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.963+0000 7f781fe00640 1 -- 172.21.15.27:0/2958762861 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c037070 con 0x7f78200a61a0 2024-03-28T19:08:22.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.963+0000 7f781fe00640 1 -- 172.21.15.27:0/2958762861 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c031a00 con 0x7f78200a61a0 2024-03-28T19:08:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.964+0000 7f7826e00640 1 -- 172.21.15.27:0/2958762861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200a61a0 msgr2=0x7f78200c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.964+0000 7f7826e00640 1 --2- 172.21.15.27:0/2958762861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200a61a0 0x7f78200c69a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f78200a9760 tx=0x7f780c02b880 comp rx=0 tx=0).stop 2024-03-28T19:08:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.964+0000 7f7826e00640 1 -- 172.21.15.27:0/2958762861 shutdown_connections 2024-03-28T19:08:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.964+0000 7f7826e00640 1 --2- 172.21.15.27:0/2958762861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200a61a0 0x7f78200c69a0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.964+0000 7f7826e00640 1 --2- 172.21.15.27:0/2958762861 >> v2:172.21.15.155:3300/0 conn(0x7f7820150ae0 0x7f78200a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.964+0000 7f7826e00640 1 -- 172.21.15.27:0/2958762861 >> 172.21.15.27:0/2958762861 conn(0x7f782006b0c0 msgr2=0x7f782006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.964+0000 7f7826e00640 1 -- 172.21.15.27:0/2958762861 shutdown_connections 2024-03-28T19:08:22.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.964+0000 7f7826e00640 1 -- 172.21.15.27:0/2958762861 wait complete. 2024-03-28T19:08:22.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.966+0000 7f7826e00640 1 Processor -- start 2024-03-28T19:08:22.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.966+0000 7f7826e00640 1 -- start start 2024-03-28T19:08:22.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7826e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78200a61a0 0x7f78200a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:22.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7826e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200c6ee0 0x7f78200a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7826e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7820150ae0 0x7f78200a7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7826e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782006e7b0 con 0x7f78200c6ee0 2024-03-28T19:08:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7826e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782006e4d0 con 0x7f78200a61a0 2024-03-28T19:08:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7826e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78201569d0 con 0x7f7820150ae0 2024-03-28T19:08:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7825000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200c6ee0 0x7f78200a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7825000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200c6ee0 0x7f78200a9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58936/0 (socket says 172.21.15.27:58936) 2024-03-28T19:08:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.967+0000 7f7825000640 1 -- 172.21.15.27:0/3329723829 learned_addr learned my addr 172.21.15.27:0/3329723829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:22.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.968+0000 7f7825a00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78200a61a0 0x7f78200a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:22.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.968+0000 7f7826400640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7820150ae0 0x7f78200a7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:22.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.968+0000 7f7825000640 1 -- 172.21.15.27:0/3329723829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7820150ae0 msgr2=0x7f78200a7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:22.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.968+0000 7f7825000640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7820150ae0 0x7f78200a7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.968+0000 7f7825000640 1 -- 172.21.15.27:0/3329723829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78200a61a0 msgr2=0x7f78200a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.968+0000 7f7825000640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78200a61a0 0x7f78200a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.968+0000 7f7825000640 1 -- 172.21.15.27:0/3329723829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78200ab670 con 0x7f78200c6ee0 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.968+0000 7f7825000640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200c6ee0 0x7f78200a9ca0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f780c02b8c0 tx=0x7f780c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.969+0000 7f781d600640 1 -- 172.21.15.27:0/3329723829 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c032880 con 0x7f78200c6ee0 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.969+0000 7f781d600640 1 -- 172.21.15.27:0/3329723829 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c032a20 con 0x7f78200c6ee0 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.969+0000 7f781d600640 1 -- 172.21.15.27:0/3329723829 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c032e10 con 0x7f78200c6ee0 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.969+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78200ab950 con 0x7f78200c6ee0 2024-03-28T19:08:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.969+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78200a5e50 con 0x7f78200c6ee0 2024-03-28T19:08:22.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.970+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7820067dd0 con 0x7f78200c6ee0 2024-03-28T19:08:22.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.972+0000 7f781d600640 1 -- 172.21.15.27:0/3329723829 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f780c0453f0 con 0x7f78200c6ee0 2024-03-28T19:08:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.973+0000 7f781d600640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780006bbd0 0x7f780006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.973+0000 7f781d600640 1 -- 172.21.15.27:0/3329723829 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7f780c0ad360 con 0x7f78200c6ee0 2024-03-28T19:08:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.973+0000 7f7825a00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780006bbd0 0x7f780006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:22.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.974+0000 7f7825a00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780006bbd0 0x7f780006e090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7814006fd0 tx=0x7f7814008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:22.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:22.978+0000 7f781d600640 1 -- 172.21.15.27:0/3329723829 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f780c040070 con 0x7f78200c6ee0 2024-03-28T19:08:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.254+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7820069640 con 0x7f78200c6ee0 2024-03-28T19:08:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.256+0000 7f781d600640 1 -- 172.21.15.27:0/3329723829 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6262 (secure 0 0 0) 0x7f7820150ec0 con 0x7f78200c6ee0 2024-03-28T19:08:23.255 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:23.255 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6501,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:07:45.701547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6411},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6411":{"gid":6411,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":11,"addr":"172.21.15.27:6837/2647193511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":2647193511},{"type":"v1","addr":"172.21.15.27:6837","nonce":2647193511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6411,"qdb_cluster":[6411]},"id":2}]} 2024-03-28T19:08:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 102 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780006bbd0 msgr2=0x7f780006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780006bbd0 0x7f780006e090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7814006fd0 tx=0x7f7814008040 comp rx=0 tx=0).stop 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200c6ee0 msgr2=0x7f78200a9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200c6ee0 0x7f78200a9ca0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f780c02b8c0 tx=0x7f780c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 shutdown_connections 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f780006bbd0 0x7f780006e090 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7820150ae0 0x7f78200a7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78200c6ee0 0x7f78200a9ca0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 --2- 172.21.15.27:0/3329723829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78200a61a0 0x7f78200a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 >> 172.21.15.27:0/3329723829 conn(0x7f782006b0c0 msgr2=0x7f78200c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 shutdown_connections 2024-03-28T19:08:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:23.259+0000 7f7826e00640 1 -- 172.21.15.27:0/3329723829 wait complete. 2024-03-28T19:08:23.281 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:08:24.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.770+0000 7f6cee400640 1 Processor -- start 2024-03-28T19:08:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.770+0000 7f6cee400640 1 -- start start 2024-03-28T19:08:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6cee400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6ce80c59d0 0x7f6ce80c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6cee400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ce80727e0 0x7f6ce8159520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6cee400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce8155fe0 con 0x7f6ce8155600 2024-03-28T19:08:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6cee400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce81565a0 con 0x7f6ce80727e0 2024-03-28T19:08:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6cee400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce8159090 con 0x7f6ce80c59d0 2024-03-28T19:08:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6ce7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ce80727e0 0x7f6ce8159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6ce7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ce80727e0 0x7f6ce8159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58948/0 (socket says 172.21.15.27:58948) 2024-03-28T19:08:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6ce7e00640 1 -- 172.21.15.27:0/786910410 learned_addr learned my addr 172.21.15.27:0/786910410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.772+0000 7f6ced000640 1 --2- 172.21.15.27:0/786910410 >> v2:172.21.15.155:3300/0 conn(0x7f6ce80c59d0 0x7f6ce80c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.773+0000 7f6ce7e00640 1 -- 172.21.15.27:0/786910410 >> v2:172.21.15.155:3300/0 conn(0x7f6ce80c59d0 msgr2=0x7f6ce80c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.773+0000 7f6ce7e00640 1 --2- 172.21.15.27:0/786910410 >> v2:172.21.15.155:3300/0 conn(0x7f6ce80c59d0 0x7f6ce80c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.773+0000 7f6ce7e00640 1 -- 172.21.15.27:0/786910410 >> v1:172.21.15.155:6789/0 conn(0x7f6ce8155600 legacy=0x7f6ce814f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.773+0000 7f6ce7e00640 1 -- 172.21.15.27:0/786910410 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce81562c0 con 0x7f6ce80727e0 2024-03-28T19:08:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.775+0000 7f6ce7e00640 1 --2- 172.21.15.27:0/786910410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80727e0 0x7f6ce8159520 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80aa5c0 tx=0x7f6cd402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9938b1b94d57b4d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.775+0000 7f6ce7400640 1 -- 172.21.15.27:0/786910410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd402e070 con 0x7f6ce80727e0 2024-03-28T19:08:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.775+0000 7f6ce7400640 1 -- 172.21.15.27:0/786910410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4037070 con 0x7f6ce80727e0 2024-03-28T19:08:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.775+0000 7f6ce7400640 1 -- 172.21.15.27:0/786910410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4031a00 con 0x7f6ce80727e0 2024-03-28T19:08:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.776+0000 7f6cee400640 1 -- 172.21.15.27:0/786910410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80727e0 msgr2=0x7f6ce8159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.776+0000 7f6cee400640 1 --2- 172.21.15.27:0/786910410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80727e0 0x7f6ce8159520 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80aa5c0 tx=0x7f6cd402b880 comp rx=0 tx=0).stop 2024-03-28T19:08:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.776+0000 7f6cee400640 1 -- 172.21.15.27:0/786910410 shutdown_connections 2024-03-28T19:08:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.776+0000 7f6cee400640 1 --2- 172.21.15.27:0/786910410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80727e0 0x7f6ce8159520 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.776+0000 7f6cee400640 1 --2- 172.21.15.27:0/786910410 >> v2:172.21.15.155:3300/0 conn(0x7f6ce80c59d0 0x7f6ce80c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.776+0000 7f6cee400640 1 -- 172.21.15.27:0/786910410 >> 172.21.15.27:0/786910410 conn(0x7f6ce80ac910 msgr2=0x7f6ce80accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.776+0000 7f6cee400640 1 -- 172.21.15.27:0/786910410 shutdown_connections 2024-03-28T19:08:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.776+0000 7f6cee400640 1 -- 172.21.15.27:0/786910410 wait complete. 2024-03-28T19:08:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.778+0000 7f6cee400640 1 Processor -- start 2024-03-28T19:08:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.778+0000 7f6cee400640 1 -- start start 2024-03-28T19:08:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.779+0000 7f6cee400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ce80727e0 0x7f6ce8154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.779+0000 7f6cee400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80c59d0 0x7f6ce80c0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.779+0000 7f6cee400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ce8155600 0x7f6ce80c0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.779+0000 7f6cee400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce814f970 con 0x7f6ce80c59d0 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.779+0000 7f6cee400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce8159a60 con 0x7f6ce8155600 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.779+0000 7f6cee400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce814faf0 con 0x7f6ce80727e0 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ce7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80c59d0 0x7f6ce80c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ce7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80c59d0 0x7f6ce80c0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58954/0 (socket says 172.21.15.27:58954) 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ce7e00640 1 -- 172.21.15.27:0/137023443 learned_addr learned my addr 172.21.15.27:0/137023443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ced000640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ce80727e0 0x7f6ce8154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ceda00640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ce8155600 0x7f6ce80c0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ce7e00640 1 -- 172.21.15.27:0/137023443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ce80727e0 msgr2=0x7f6ce8154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ce7e00640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ce80727e0 0x7f6ce8154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ce7e00640 1 -- 172.21.15.27:0/137023443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ce8155600 msgr2=0x7f6ce80c0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ce7e00640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ce8155600 0x7f6ce80c0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.780+0000 7f6ce7e00640 1 -- 172.21.15.27:0/137023443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce81eef70 con 0x7f6ce80c59d0 2024-03-28T19:08:24.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.781+0000 7f6ce7e00640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80c59d0 0x7f6ce80c0350 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4000c00 tx=0x7f6cd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:24.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.781+0000 7f6ce4c00640 1 -- 172.21.15.27:0/137023443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4032670 con 0x7f6ce80c59d0 2024-03-28T19:08:24.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.781+0000 7f6ce4c00640 1 -- 172.21.15.27:0/137023443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4032830 con 0x7f6ce80c59d0 2024-03-28T19:08:24.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.781+0000 7f6ce4c00640 1 -- 172.21.15.27:0/137023443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4032c20 con 0x7f6ce80c59d0 2024-03-28T19:08:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.781+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce80c23d0 con 0x7f6ce80c59d0 2024-03-28T19:08:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.782+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce8154d40 con 0x7f6ce80c59d0 2024-03-28T19:08:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.783+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce8067dd0 con 0x7f6ce80c59d0 2024-03-28T19:08:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.784+0000 7f6ce4c00640 1 -- 172.21.15.27:0/137023443 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6cd40453f0 con 0x7f6ce80c59d0 2024-03-28T19:08:24.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.785+0000 7f6ce4c00640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cc806bc00 0x7f6cc806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:24.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.785+0000 7f6ce4c00640 1 -- 172.21.15.27:0/137023443 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 8013+0+0 (secure 0 0 0) 0x7f6cd40ad130 con 0x7f6ce80c59d0 2024-03-28T19:08:24.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.785+0000 7f6ced000640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cc806bc00 0x7f6cc806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:24.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.786+0000 7f6ced000640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cc806bc00 0x7f6cc806e0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009ad0 tx=0x7f6cdc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:24.791+0000 7f6ce4c00640 1 -- 172.21.15.27:0/137023443 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6cd4040070 con 0x7f6ce80c59d0 2024-03-28T19:08:25.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.135+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ce80c8ce0 con 0x7f6ce80c59d0 2024-03-28T19:08:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.136+0000 7f6ce4c00640 1 -- 172.21.15.27:0/137023443 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f6cd407b430 con 0x7f6ce80c59d0 2024-03-28T19:08:25.136 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:25.136 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:24.892464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:08:25.136 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 103 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cc806bc00 msgr2=0x7f6cc806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cc806bc00 0x7f6cc806e0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009ad0 tx=0x7f6cdc008040 comp rx=0 tx=0).stop 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80c59d0 msgr2=0x7f6ce80c0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80c59d0 0x7f6ce80c0350 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4000c00 tx=0x7f6cd40046c0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 shutdown_connections 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cc806bc00 0x7f6cc806e0c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ce8155600 0x7f6ce80c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce80c59d0 0x7f6ce80c0350 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 --2- 172.21.15.27:0/137023443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ce80727e0 0x7f6ce8154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 >> 172.21.15.27:0/137023443 conn(0x7f6ce80ac910 msgr2=0x7f6ce80c94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.139+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 shutdown_connections 2024-03-28T19:08:25.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.140+0000 7f6cee400640 1 -- 172.21.15.27:0/137023443 wait complete. 2024-03-28T19:08:25.161 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-03-28T19:08:25.162 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:08:25.162 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:08:25.190 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:08:25.191 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:25.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:25.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.420+0000 7fd649200640 1 Processor -- start 2024-03-28T19:08:25.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.420+0000 7fd649200640 1 -- start start 2024-03-28T19:08:25.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.422+0000 7fd649200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd6440c7970 0x7fd6440c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.422+0000 7fd649200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6440c7310 con 0x7fd644152f80 2024-03-28T19:08:25.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.422+0000 7fd649200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd644158d40 con 0x7fd644155730 2024-03-28T19:08:25.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.422+0000 7fd649200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd644159020 con 0x7fd6440c7970 2024-03-28T19:08:25.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.422+0000 7fd641400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd6440c7970 0x7fd6440c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.423+0000 7fd641e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd644152f80 0x7fd6440c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39944/0 (socket says 172.21.15.27:39944) 2024-03-28T19:08:25.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.423+0000 7fd641e00640 1 -- 172.21.15.27:0/280175424 learned_addr learned my addr 172.21.15.27:0/280175424 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:25.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.423+0000 7fd641400640 1 -- 172.21.15.27:0/280175424 >> v1:172.21.15.155:6789/0 conn(0x7fd644155730 legacy=0x7fd644155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.423+0000 7fd641400640 1 -- 172.21.15.27:0/280175424 >> v1:172.21.15.27:6789/0 conn(0x7fd644152f80 legacy=0x7fd6440c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.423+0000 7fd641400640 1 -- 172.21.15.27:0/280175424 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd644159300 con 0x7fd6440c7970 2024-03-28T19:08:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.425+0000 7fd641400640 1 --2- 172.21.15.27:0/280175424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6440c7970 0x7fd6440c7df0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd6300097a0 tx=0x7fd63002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fdf94ba5fdb74c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.426+0000 7fd640a00640 1 -- 172.21.15.27:0/280175424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6300315a0 con 0x7fd6440c7970 2024-03-28T19:08:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.426+0000 7fd640a00640 1 -- 172.21.15.27:0/280175424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630037070 con 0x7fd6440c7970 2024-03-28T19:08:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.426+0000 7fd640a00640 1 -- 172.21.15.27:0/280175424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630031960 con 0x7fd6440c7970 2024-03-28T19:08:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.427+0000 7fd649200640 1 -- 172.21.15.27:0/280175424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6440c7970 msgr2=0x7fd6440c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.427+0000 7fd649200640 1 --2- 172.21.15.27:0/280175424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6440c7970 0x7fd6440c7df0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd6300097a0 tx=0x7fd63002b840 comp rx=0 tx=0).stop 2024-03-28T19:08:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.427+0000 7fd649200640 1 -- 172.21.15.27:0/280175424 shutdown_connections 2024-03-28T19:08:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.427+0000 7fd649200640 1 --2- 172.21.15.27:0/280175424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6440c7970 0x7fd6440c7df0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.427+0000 7fd649200640 1 -- 172.21.15.27:0/280175424 >> 172.21.15.27:0/280175424 conn(0x7fd64406b6f0 msgr2=0x7fd6440caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.427+0000 7fd649200640 1 -- 172.21.15.27:0/280175424 shutdown_connections 2024-03-28T19:08:25.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.427+0000 7fd649200640 1 -- 172.21.15.27:0/280175424 wait complete. 2024-03-28T19:08:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.429+0000 7fd649200640 1 Processor -- start 2024-03-28T19:08:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.430+0000 7fd649200640 1 -- start start 2024-03-28T19:08:25.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd649200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6440c7970 0x7fd644156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd649200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd644152f80 0x7fd6440caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd649200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd644155730 0x7fd6441ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd649200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6440cbca0 con 0x7fd644155730 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd649200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd644159b00 con 0x7fd6440c7970 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd649200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6440cb3e0 con 0x7fd644152f80 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd642800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd644155730 0x7fd6441ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd642800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd644155730 0x7fd6441ec5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58974/0 (socket says 172.21.15.27:58974) 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.431+0000 7fd642800640 1 -- 172.21.15.27:0/2123900233 learned_addr learned my addr 172.21.15.27:0/2123900233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.432+0000 7fd641e00640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6440c7970 0x7fd644156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.432+0000 7fd641400640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd644152f80 0x7fd6440caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.432+0000 7fd642800640 1 -- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd644152f80 msgr2=0x7fd6440caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.432+0000 7fd642800640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd644152f80 0x7fd6440caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.432+0000 7fd642800640 1 -- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6440c7970 msgr2=0x7fd644156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.432+0000 7fd642800640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6440c7970 0x7fd644156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.432+0000 7fd642800640 1 -- 172.21.15.27:0/2123900233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd644157cb0 con 0x7fd644155730 2024-03-28T19:08:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.432+0000 7fd641e00640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6440c7970 0x7fd644156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:08:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.433+0000 7fd642800640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd644155730 0x7fd6441ec5d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00c780 tx=0x7fd63c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.434+0000 7fd63a000640 1 -- 172.21.15.27:0/2123900233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c004b10 con 0x7fd644155730 2024-03-28T19:08:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.434+0000 7fd63a000640 1 -- 172.21.15.27:0/2123900233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63c004cb0 con 0x7fd644155730 2024-03-28T19:08:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.434+0000 7fd63a000640 1 -- 172.21.15.27:0/2123900233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c004e60 con 0x7fd644155730 2024-03-28T19:08:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.434+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd644157130 con 0x7fd644155730 2024-03-28T19:08:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.434+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6441576f0 con 0x7fd644155730 2024-03-28T19:08:25.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.436+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64404f400 con 0x7fd644155730 2024-03-28T19:08:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.439+0000 7fd63a000640 1 -- 172.21.15.27:0/2123900233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd63c006b50 con 0x7fd644155730 2024-03-28T19:08:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.441+0000 7fd63a000640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd62406bbd0 0x7fd62406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.441+0000 7fd63a000640 1 -- 172.21.15.27:0/2123900233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7fd63c08be00 con 0x7fd644155730 2024-03-28T19:08:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.442+0000 7fd641e00640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd62406bbd0 0x7fd62406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.443+0000 7fd641e00640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd62406bbd0 0x7fd62406e090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd6440cb700 tx=0x7fd634009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.446+0000 7fd63a000640 1 -- 172.21.15.27:0/2123900233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd63c059ab0 con 0x7fd644155730 2024-03-28T19:08:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.518+0000 7f414ea00640 1 Processor -- start 2024-03-28T19:08:25.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.518+0000 7f414ea00640 1 -- start start 2024-03-28T19:08:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.519+0000 7f414ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4148152a70 0x7f4148159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.519+0000 7f414ea00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4148155b40 0x7f4148155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.519+0000 7f414ea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41480c5710 con 0x7f41480c6cc0 2024-03-28T19:08:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.519+0000 7f414ea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41481556b0 con 0x7f4148152a70 2024-03-28T19:08:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.519+0000 7f414ea00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41480c59f0 con 0x7f4148155b40 2024-03-28T19:08:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.520+0000 7f414d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4148152a70 0x7f4148159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.520+0000 7f414cc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f41480c6cc0 0x7f41480cb0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39946/0 (socket says 172.21.15.27:39946) 2024-03-28T19:08:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.520+0000 7f414cc00640 1 -- 172.21.15.27:0/2260732468 learned_addr learned my addr 172.21.15.27:0/2260732468 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.520+0000 7f414e000640 1 --2- 172.21.15.27:0/2260732468 >> v2:172.21.15.155:3300/0 conn(0x7f4148155b40 0x7f4148155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.521+0000 7f414d600640 1 -- 172.21.15.27:0/2260732468 >> v2:172.21.15.155:3300/0 conn(0x7f4148155b40 msgr2=0x7f4148155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.521+0000 7f414d600640 1 --2- 172.21.15.27:0/2260732468 >> v2:172.21.15.155:3300/0 conn(0x7f4148155b40 0x7f4148155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.521+0000 7f414d600640 1 -- 172.21.15.27:0/2260732468 >> v1:172.21.15.27:6789/0 conn(0x7f41480c6cc0 legacy=0x7f41480cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.521+0000 7f414d600640 1 -- 172.21.15.27:0/2260732468 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41480c5cd0 con 0x7f4148152a70 2024-03-28T19:08:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.522+0000 7f414d600640 1 --2- 172.21.15.27:0/2260732468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 0x7f4148159580 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f41480727a0 tx=0x7f413c02b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f66d231b15130c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.523+0000 7f4147e00640 1 -- 172.21.15.27:0/2260732468 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c037070 con 0x7f4148152a70 2024-03-28T19:08:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.523+0000 7f4147e00640 1 -- 172.21.15.27:0/2260732468 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413c030710 con 0x7f4148152a70 2024-03-28T19:08:25.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.523+0000 7f4147e00640 1 -- 172.21.15.27:0/2260732468 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c030ae0 con 0x7f4148152a70 2024-03-28T19:08:25.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.524+0000 7f414ea00640 1 -- 172.21.15.27:0/2260732468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 msgr2=0x7f4148159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.524+0000 7f414ea00640 1 --2- 172.21.15.27:0/2260732468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 0x7f4148159580 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f41480727a0 tx=0x7f413c02b680 comp rx=0 tx=0).stop 2024-03-28T19:08:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.525+0000 7f414ea00640 1 -- 172.21.15.27:0/2260732468 shutdown_connections 2024-03-28T19:08:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.525+0000 7f414ea00640 1 --2- 172.21.15.27:0/2260732468 >> v2:172.21.15.155:3300/0 conn(0x7f4148155b40 0x7f4148155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.525+0000 7f414ea00640 1 --2- 172.21.15.27:0/2260732468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 0x7f4148159580 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.525+0000 7f414ea00640 1 -- 172.21.15.27:0/2260732468 >> 172.21.15.27:0/2260732468 conn(0x7f414806b2a0 msgr2=0x7f414806b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.525+0000 7f414ea00640 1 -- 172.21.15.27:0/2260732468 shutdown_connections 2024-03-28T19:08:25.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.526+0000 7f414ea00640 1 -- 172.21.15.27:0/2260732468 wait complete. 2024-03-28T19:08:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.529+0000 7f414ea00640 1 Processor -- start 2024-03-28T19:08:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.529+0000 7f414ea00640 1 -- start start 2024-03-28T19:08:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.530+0000 7f414ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41480c6cc0 0x7f4148150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.530+0000 7f414ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 0x7f41481ec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.531+0000 7f414ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4148155b40 0x7f41481ec810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.531+0000 7f414ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41480cb7f0 con 0x7f4148152a70 2024-03-28T19:08:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.531+0000 7f414ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4148072a70 con 0x7f41480c6cc0 2024-03-28T19:08:25.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.531+0000 7f414ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41480cb970 con 0x7f4148155b40 2024-03-28T19:08:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.531+0000 7f414d600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41480c6cc0 0x7f4148150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.531+0000 7f414d600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41480c6cc0 0x7f4148150500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43456/0 (socket says 172.21.15.27:43456) 2024-03-28T19:08:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.531+0000 7f414d600640 1 -- 172.21.15.27:0/1316615105 learned_addr learned my addr 172.21.15.27:0/1316615105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.532+0000 7f414cc00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 0x7f41481ec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.532+0000 7f414e000640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4148155b40 0x7f41481ec810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.532+0000 7f414d600640 1 -- 172.21.15.27:0/1316615105 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4148155b40 msgr2=0x7f41481ec810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.532+0000 7f414d600640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4148155b40 0x7f41481ec810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.532+0000 7f414d600640 1 -- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 msgr2=0x7f41481ec2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.532+0000 7f414d600640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 0x7f41481ec2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.532+0000 7f414d600640 1 -- 172.21.15.27:0/1316615105 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41481ef1b0 con 0x7f41480c6cc0 2024-03-28T19:08:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.533+0000 7f414cc00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 0x7f41481ec2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:08:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.533+0000 7f414d600640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41480c6cc0 0x7f4148150500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f413c02bc90 tx=0x7f413c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.534+0000 7f4145600640 1 -- 172.21.15.27:0/1316615105 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c041020 con 0x7f41480c6cc0 2024-03-28T19:08:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.534+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41481ef490 con 0x7f41480c6cc0 2024-03-28T19:08:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.534+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41481efa50 con 0x7f41480c6cc0 2024-03-28T19:08:25.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.535+0000 7f4145600640 1 -- 172.21.15.27:0/1316615105 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413c030690 con 0x7f41480c6cc0 2024-03-28T19:08:25.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.535+0000 7f4145600640 1 -- 172.21.15.27:0/1316615105 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c037040 con 0x7f41480c6cc0 2024-03-28T19:08:25.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.536+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4124005290 con 0x7f41480c6cc0 2024-03-28T19:08:25.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.540+0000 7f4145600640 1 -- 172.21.15.27:0/1316615105 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f413c044020 con 0x7f41480c6cc0 2024-03-28T19:08:25.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.541+0000 7f4145600640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f412806bbd0 0x7f412806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:25.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.542+0000 7f4145600640 1 -- 172.21.15.27:0/1316615105 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7f413c0ad500 con 0x7f41480c6cc0 2024-03-28T19:08:25.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.542+0000 7f414cc00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f412806bbd0 0x7f412806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:25.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.543+0000 7f414cc00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f412806bbd0 0x7f412806e090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f414814e030 tx=0x7f4134007e00 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:25.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.554+0000 7f4145600640 1 -- 172.21.15.27:0/1316615105 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f413c04b030 con 0x7f41480c6cc0 2024-03-28T19:08:25.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.890+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6441ec330 con 0x7fd644155730 2024-03-28T19:08:25.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.891+0000 7fd63a000640 1 -- 172.21.15.27:0/2123900233 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5463 (secure 0 0 0) 0x7fd63c05db50 con 0x7fd644155730 2024-03-28T19:08:25.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.891+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4124005580 con 0x7f41480c6cc0 2024-03-28T19:08:25.890 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:25.891 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:24.892464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:08:25.891 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 103 2024-03-28T19:08:25.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.892+0000 7f4145600640 1 -- 172.21.15.27:0/1316615105 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f413c07b890 con 0x7f41480c6cc0 2024-03-28T19:08:25.892 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:25.892 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:24.892464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:08:25.893 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 103 2024-03-28T19:08:25.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd62406bbd0 msgr2=0x7fd62406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd62406bbd0 0x7fd62406e090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd6440cb700 tx=0x7fd634009040 comp rx=0 tx=0).stop 2024-03-28T19:08:25.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd644155730 msgr2=0x7fd6441ec5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd644155730 0x7fd6441ec5d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00c780 tx=0x7fd63c00cc40 comp rx=0 tx=0).stop 2024-03-28T19:08:25.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 shutdown_connections 2024-03-28T19:08:25.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd62406bbd0 0x7fd62406e090 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd644155730 0x7fd6441ec5d0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd644152f80 0x7fd6440caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 --2- 172.21.15.27:0/2123900233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6440c7970 0x7fd644156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 >> 172.21.15.27:0/2123900233 conn(0x7fd64406b6f0 msgr2=0x7fd644150ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:25.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.894+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 shutdown_connections 2024-03-28T19:08:25.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.895+0000 7fd649200640 1 -- 172.21.15.27:0/2123900233 wait complete. 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.895+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f412806bbd0 msgr2=0x7f412806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.895+0000 7f414ea00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f412806bbd0 0x7f412806e090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f414814e030 tx=0x7f4134007e00 comp rx=0 tx=0).stop 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.895+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41480c6cc0 msgr2=0x7f4148150500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.895+0000 7f414ea00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41480c6cc0 0x7f4148150500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f413c02bc90 tx=0x7f413c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.896+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 shutdown_connections 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.896+0000 7f414ea00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f412806bbd0 0x7f412806e090 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.896+0000 7f414ea00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4148155b40 0x7f41481ec810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.896+0000 7f414ea00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4148152a70 0x7f41481ec2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.896+0000 7f414ea00640 1 --2- 172.21.15.27:0/1316615105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41480c6cc0 0x7f4148150500 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.896+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 >> 172.21.15.27:0/1316615105 conn(0x7f414806b2a0 msgr2=0x7f414806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.896+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 shutdown_connections 2024-03-28T19:08:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:25.896+0000 7f414ea00640 1 -- 172.21.15.27:0/1316615105 wait complete. 2024-03-28T19:08:25.917 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-03-28T19:08:25.917 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:08:25.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:25.918 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:08:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.173+0000 7f1cb1200640 1 Processor -- start 2024-03-28T19:08:26.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.173+0000 7f1cb1200640 1 -- start start 2024-03-28T19:08:26.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.175+0000 7f1cb1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cac152f80 0x7f1cac0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:26.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.175+0000 7f1cb1200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1cac0c7970 0x7f1cac0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.175+0000 7f1cb1200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac155730 0x7f1cac155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.175+0000 7f1cb1200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac0c7310 con 0x7f1cac152f80 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.175+0000 7f1cb1200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac158d40 con 0x7f1cac0c7970 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.175+0000 7f1cb1200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac159020 con 0x7f1cac155730 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cac152f80 0x7f1cac0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cac152f80 0x7f1cac0c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39108/0 (socket says 172.21.15.27:39108) 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9e00640 1 -- 172.21.15.27:0/3417710284 learned_addr learned my addr 172.21.15.27:0/3417710284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1caa800640 1 --2- 172.21.15.27:0/3417710284 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac155730 0x7f1cac155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9400640 1 --2- 172.21.15.27:0/3417710284 >> v2:172.21.15.155:3300/0 conn(0x7f1cac0c7970 0x7f1cac0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9e00640 1 -- 172.21.15.27:0/3417710284 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac155730 msgr2=0x7f1cac155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:26.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9e00640 1 --2- 172.21.15.27:0/3417710284 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac155730 0x7f1cac155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9e00640 1 -- 172.21.15.27:0/3417710284 >> v2:172.21.15.155:3300/0 conn(0x7f1cac0c7970 msgr2=0x7f1cac0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:26.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9e00640 1 --2- 172.21.15.27:0/3417710284 >> v2:172.21.15.155:3300/0 conn(0x7f1cac0c7970 0x7f1cac0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.176+0000 7f1ca9e00640 1 -- 172.21.15.27:0/3417710284 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac159300 con 0x7f1cac152f80 2024-03-28T19:08:26.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.178+0000 7f1ca9e00640 1 --2- 172.21.15.27:0/3417710284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac152f80 0x7f1cac0c58e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0c8480 tx=0x7f1c9c02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c4cdaa8825123d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:26.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.178+0000 7f1ca8a00640 1 -- 172.21.15.27:0/3417710284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c0317f0 con 0x7f1cac152f80 2024-03-28T19:08:26.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.179+0000 7f1ca8a00640 1 -- 172.21.15.27:0/3417710284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c037070 con 0x7f1cac152f80 2024-03-28T19:08:26.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.179+0000 7f1ca8a00640 1 -- 172.21.15.27:0/3417710284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c031bb0 con 0x7f1cac152f80 2024-03-28T19:08:26.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.179+0000 7f1cb1200640 1 -- 172.21.15.27:0/3417710284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac152f80 msgr2=0x7f1cac0c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:26.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.179+0000 7f1cb1200640 1 --2- 172.21.15.27:0/3417710284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac152f80 0x7f1cac0c58e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0c8480 tx=0x7f1c9c02b890 comp rx=0 tx=0).stop 2024-03-28T19:08:26.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.180+0000 7f1cb1200640 1 -- 172.21.15.27:0/3417710284 shutdown_connections 2024-03-28T19:08:26.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.180+0000 7f1cb1200640 1 --2- 172.21.15.27:0/3417710284 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac155730 0x7f1cac155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.180+0000 7f1cb1200640 1 --2- 172.21.15.27:0/3417710284 >> v2:172.21.15.155:3300/0 conn(0x7f1cac0c7970 0x7f1cac0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.180+0000 7f1cb1200640 1 --2- 172.21.15.27:0/3417710284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac152f80 0x7f1cac0c58e0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.180+0000 7f1cb1200640 1 -- 172.21.15.27:0/3417710284 >> 172.21.15.27:0/3417710284 conn(0x7f1cac06b6f0 msgr2=0x7f1cac0caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:26.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.180+0000 7f1cb1200640 1 -- 172.21.15.27:0/3417710284 shutdown_connections 2024-03-28T19:08:26.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.180+0000 7f1cb1200640 1 -- 172.21.15.27:0/3417710284 wait complete. 2024-03-28T19:08:26.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.182+0000 7f1cb1200640 1 Processor -- start 2024-03-28T19:08:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.183+0000 7f1cb1200640 1 -- start start 2024-03-28T19:08:26.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1cb1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cac0c7970 0x7f1cac1ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:26.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1cb1200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac152f80 0x7f1cac1eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1cb1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac155730 0x7f1cac0caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1cb1200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac159b00 con 0x7f1cac155730 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1cb1200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0c5e20 con 0x7f1cac0c7970 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1cb1200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0cbca0 con 0x7f1cac152f80 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1caa800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac155730 0x7f1cac0caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1caa800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac155730 0x7f1cac0caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39110/0 (socket says 172.21.15.27:39110) 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.184+0000 7f1caa800640 1 -- 172.21.15.27:0/4245615299 learned_addr learned my addr 172.21.15.27:0/4245615299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.185+0000 7f1ca9e00640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cac0c7970 0x7f1cac1ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.185+0000 7f1ca9400640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac152f80 0x7f1cac1eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.185+0000 7f1caa800640 1 -- 172.21.15.27:0/4245615299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac152f80 msgr2=0x7f1cac1eb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.185+0000 7f1caa800640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac152f80 0x7f1cac1eb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.185+0000 7f1caa800640 1 -- 172.21.15.27:0/4245615299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cac0c7970 msgr2=0x7f1cac1ef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.185+0000 7f1caa800640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cac0c7970 0x7f1cac1ef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.185+0000 7f1caa800640 1 -- 172.21.15.27:0/4245615299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac148520 con 0x7f1cac155730 2024-03-28T19:08:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.186+0000 7f1caa800640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac155730 0x7f1cac0caaa0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400cc80 tx=0x7f1ca400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.186+0000 7f1ca2000640 1 -- 172.21.15.27:0/4245615299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca400ce30 con 0x7f1cac155730 2024-03-28T19:08:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.186+0000 7f1ca2000640 1 -- 172.21.15.27:0/4245615299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca4003990 con 0x7f1cac155730 2024-03-28T19:08:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.186+0000 7f1ca2000640 1 -- 172.21.15.27:0/4245615299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4003d60 con 0x7f1cac155730 2024-03-28T19:08:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.186+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac148040 con 0x7f1cac155730 2024-03-28T19:08:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.186+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac1decb0 con 0x7f1cac155730 2024-03-28T19:08:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.188+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac154a10 con 0x7f1cac155730 2024-03-28T19:08:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.189+0000 7f1ca2000640 1 -- 172.21.15.27:0/4245615299 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1ca4003f00 con 0x7f1cac155730 2024-03-28T19:08:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.190+0000 7f1ca2000640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c8c06bc20 0x7f1c8c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.191+0000 7f1ca2000640 1 -- 172.21.15.27:0/4245615299 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7f1ca408b6c0 con 0x7f1cac155730 2024-03-28T19:08:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.191+0000 7f1ca9e00640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c8c06bc20 0x7f1c8c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.192+0000 7f1ca9e00640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c8c06bc20 0x7f1c8c06e0e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c036000 tx=0x7f1c9c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.195+0000 7f1ca2000640 1 -- 172.21.15.27:0/4245615299 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1ca4014030 con 0x7f1cac155730 2024-03-28T19:08:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.516+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cac04f400 con 0x7f1cac155730 2024-03-28T19:08:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.517+0000 7f1ca2000640 1 -- 172.21.15.27:0/4245615299 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f1ca4059270 con 0x7f1cac155730 2024-03-28T19:08:26.516 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:26.517 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:24.892464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:08:26.517 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 103 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c8c06bc20 msgr2=0x7f1c8c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c8c06bc20 0x7f1c8c06e0e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c036000 tx=0x7f1c9c03d040 comp rx=0 tx=0).stop 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac155730 msgr2=0x7f1cac0caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac155730 0x7f1cac0caaa0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400cc80 tx=0x7f1ca400a5c0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 shutdown_connections 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1c8c06bc20 0x7f1c8c06e0e0 secure :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c036000 tx=0x7f1c9c03d040 comp rx=0 tx=0).stop 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cac155730 0x7f1cac0caaa0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cac152f80 0x7f1cac1eb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 --2- 172.21.15.27:0/4245615299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cac0c7970 0x7f1cac1ef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:26.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 >> 172.21.15.27:0/4245615299 conn(0x7f1cac06b6f0 msgr2=0x7f1cac151e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 shutdown_connections 2024-03-28T19:08:26.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:26.520+0000 7f1cb1200640 1 -- 172.21.15.27:0/4245615299 wait complete. 2024-03-28T19:08:26.539 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 103, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5336": { "addr": "172.21.15.155:6835/3884653650", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 3884653650, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 3884653650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5336, "incarnation": 87, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:06:20.279933+0000", "qdb_cluster": [ 5336 ], "qdb_leader": 5336, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5336 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 103, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6501": { "addr": "172.21.15.27:6836/1398280334", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1398280334, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 1398280334, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6501, "incarnation": 103, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:08:24.892464+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6501 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:08:27.920 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:28.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.157+0000 7fef2ca00640 1 Processor -- start 2024-03-28T19:08:28.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.157+0000 7fef2ca00640 1 -- start start 2024-03-28T19:08:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.159+0000 7fef2ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef280c2460 0x7fef28152760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.159+0000 7fef2ca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fef28152ca0 0x7fef281556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.159+0000 7fef2ca00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef28158d00 con 0x7fef280c74e0 2024-03-28T19:08:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.159+0000 7fef2ca00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef28158fe0 con 0x7fef280c2460 2024-03-28T19:08:28.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.159+0000 7fef2ca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef281592c0 con 0x7fef28152ca0 2024-03-28T19:08:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.159+0000 7fef25e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef280c2460 0x7fef28152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.160+0000 7fef27200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fef28152ca0 0x7fef281556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.160+0000 7fef27200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fef28152ca0 0x7fef281556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:32854/0 (socket says 172.21.15.27:32854) 2024-03-28T19:08:28.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.160+0000 7fef25e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef280c2460 0x7fef28152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39118/0 (socket says 172.21.15.27:39118) 2024-03-28T19:08:28.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.160+0000 7fef27200640 1 -- 172.21.15.27:0/4007617112 learned_addr learned my addr 172.21.15.27:0/4007617112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:28.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.160+0000 7fef25e00640 1 -- 172.21.15.27:0/4007617112 >> v2:172.21.15.155:3300/0 conn(0x7fef28152ca0 msgr2=0x7fef281556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:28.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.160+0000 7fef25e00640 1 --2- 172.21.15.27:0/4007617112 >> v2:172.21.15.155:3300/0 conn(0x7fef28152ca0 0x7fef281556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.160+0000 7fef25e00640 1 -- 172.21.15.27:0/4007617112 >> v1:172.21.15.155:6789/0 conn(0x7fef280c74e0 legacy=0x7fef280c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.160+0000 7fef25e00640 1 -- 172.21.15.27:0/4007617112 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef281595a0 con 0x7fef280c2460 2024-03-28T19:08:28.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.162+0000 7fef25e00640 1 --2- 172.21.15.27:0/4007617112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c2460 0x7fef28152760 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fef140097a0 tx=0x7fef1402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbaa9a570c4ef66e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:28.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.162+0000 7fef25400640 1 -- 172.21.15.27:0/4007617112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef140315a0 con 0x7fef280c2460 2024-03-28T19:08:28.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.162+0000 7fef25400640 1 -- 172.21.15.27:0/4007617112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef14037070 con 0x7fef280c2460 2024-03-28T19:08:28.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.162+0000 7fef25400640 1 -- 172.21.15.27:0/4007617112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14031960 con 0x7fef280c2460 2024-03-28T19:08:28.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.163+0000 7fef2ca00640 1 -- 172.21.15.27:0/4007617112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c2460 msgr2=0x7fef28152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.163+0000 7fef2ca00640 1 --2- 172.21.15.27:0/4007617112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c2460 0x7fef28152760 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fef140097a0 tx=0x7fef1402b840 comp rx=0 tx=0).stop 2024-03-28T19:08:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.163+0000 7fef2ca00640 1 -- 172.21.15.27:0/4007617112 shutdown_connections 2024-03-28T19:08:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.163+0000 7fef2ca00640 1 --2- 172.21.15.27:0/4007617112 >> v2:172.21.15.155:3300/0 conn(0x7fef28152ca0 0x7fef281556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.163+0000 7fef2ca00640 1 --2- 172.21.15.27:0/4007617112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c2460 0x7fef28152760 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.163+0000 7fef2ca00640 1 -- 172.21.15.27:0/4007617112 >> 172.21.15.27:0/4007617112 conn(0x7fef2806d980 msgr2=0x7fef280a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.163+0000 7fef2ca00640 1 -- 172.21.15.27:0/4007617112 shutdown_connections 2024-03-28T19:08:28.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.163+0000 7fef2ca00640 1 -- 172.21.15.27:0/4007617112 wait complete. 2024-03-28T19:08:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.165+0000 7fef2ca00640 1 Processor -- start 2024-03-28T19:08:28.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.165+0000 7fef2ca00640 1 -- start start 2024-03-28T19:08:28.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.166+0000 7fef2ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef280c2460 0x7fef280c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.166+0000 7fef2ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c74e0 0x7fef2814d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.166+0000 7fef2ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef28152ca0 0x7fef2814dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.166+0000 7fef2ca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef280c78c0 con 0x7fef280c74e0 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.166+0000 7fef2ca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef281589f0 con 0x7fef28152ca0 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.166+0000 7fef2ca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef280c7a40 con 0x7fef280c2460 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef25e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c74e0 0x7fef2814d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef25e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c74e0 0x7fef2814d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39132/0 (socket says 172.21.15.27:39132) 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef25e00640 1 -- 172.21.15.27:0/1890371015 learned_addr learned my addr 172.21.15.27:0/1890371015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef26800640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef280c2460 0x7fef280c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef27200640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef28152ca0 0x7fef2814dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef25e00640 1 -- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef280c2460 msgr2=0x7fef280c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef25e00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef280c2460 0x7fef280c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef25e00640 1 -- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef28152ca0 msgr2=0x7fef2814dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef25e00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef28152ca0 0x7fef2814dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef25e00640 1 -- 172.21.15.27:0/1890371015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef281eea00 con 0x7fef280c74e0 2024-03-28T19:08:28.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.167+0000 7fef27200640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef28152ca0 0x7fef2814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:08:28.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.168+0000 7fef25e00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c74e0 0x7fef2814d7a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fef14034040 tx=0x7fef140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.168+0000 7fef1ea00640 1 -- 172.21.15.27:0/1890371015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14032630 con 0x7fef280c74e0 2024-03-28T19:08:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.168+0000 7fef1ea00640 1 -- 172.21.15.27:0/1890371015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef140327f0 con 0x7fef280c74e0 2024-03-28T19:08:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.168+0000 7fef1ea00640 1 -- 172.21.15.27:0/1890371015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14032be0 con 0x7fef280c74e0 2024-03-28T19:08:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.168+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2814ccc0 con 0x7fef280c74e0 2024-03-28T19:08:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.168+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2814d280 con 0x7fef280c74e0 2024-03-28T19:08:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.170+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef28069640 con 0x7fef280c74e0 2024-03-28T19:08:28.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.170+0000 7fef1ea00640 1 -- 172.21.15.27:0/1890371015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fef1404e430 con 0x7fef280c74e0 2024-03-28T19:08:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.172+0000 7fef1ea00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef0806bbd0 0x7fef0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.172+0000 7fef1ea00640 1 -- 172.21.15.27:0/1890371015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7fef1402f030 con 0x7fef280c74e0 2024-03-28T19:08:28.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.172+0000 7fef26800640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef0806bbd0 0x7fef0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.173+0000 7fef26800640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef0806bbd0 0x7fef0806e090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fef18000f60 tx=0x7fef18009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:28.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.177+0000 7fef1ea00640 1 -- 172.21.15.27:0/1890371015 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fef14040070 con 0x7fef280c74e0 2024-03-28T19:08:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.455+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef28151c20 con 0x7fef280c74e0 2024-03-28T19:08:28.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.456+0000 7fef1ea00640 1 -- 172.21.15.27:0/1890371015 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5467 (secure 0 0 0) 0x7fef1407ad70 con 0x7fef280c74e0 2024-03-28T19:08:28.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:28.456 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:27.544939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:08:28.456 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 104 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef0806bbd0 msgr2=0x7fef0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef0806bbd0 0x7fef0806e090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fef18000f60 tx=0x7fef18009040 comp rx=0 tx=0).stop 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c74e0 msgr2=0x7fef2814d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c74e0 0x7fef2814d7a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fef14034040 tx=0x7fef140046c0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 shutdown_connections 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef0806bbd0 0x7fef0806e090 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef28152ca0 0x7fef2814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef280c74e0 0x7fef2814d7a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 --2- 172.21.15.27:0/1890371015 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef280c2460 0x7fef280c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.459+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 >> 172.21.15.27:0/1890371015 conn(0x7fef2806d980 msgr2=0x7fef280c1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:28.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.460+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 shutdown_connections 2024-03-28T19:08:28.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.460+0000 7fef2ca00640 1 -- 172.21.15.27:0/1890371015 wait complete. 2024-03-28T19:08:28.479 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:08:28.540 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:28.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.763+0000 7f0c3a600640 1 Processor -- start 2024-03-28T19:08:28.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.763+0000 7f0c3a600640 1 -- start start 2024-03-28T19:08:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.764+0000 7f0c3a600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c34150ae0 0x7f0c340a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.765+0000 7f0c3a600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c34150ec0 con 0x7f0c34153a30 2024-03-28T19:08:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.765+0000 7f0c3a600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3406dc70 con 0x7f0c340a61a0 2024-03-28T19:08:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.765+0000 7f0c3a600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c34153550 con 0x7f0c34150ae0 2024-03-28T19:08:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.765+0000 7f0c39200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c34150ae0 0x7f0c340a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.765+0000 7f0c39c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0c34153a30 0x7f0c3415dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59922/0 (socket says 172.21.15.27:59922) 2024-03-28T19:08:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.765+0000 7f0c39200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0c34150ae0 0x7f0c340a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39136/0 (socket says 172.21.15.27:39136) 2024-03-28T19:08:28.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.765+0000 7f0c39c00640 1 -- 172.21.15.27:0/938468231 learned_addr learned my addr 172.21.15.27:0/938468231 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.766+0000 7f0c39200640 1 -- 172.21.15.27:0/938468231 >> v1:172.21.15.155:6789/0 conn(0x7f0c340a61a0 legacy=0x7f0c340c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.766+0000 7f0c39200640 1 -- 172.21.15.27:0/938468231 >> v1:172.21.15.27:6789/0 conn(0x7f0c34153a30 legacy=0x7f0c3415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.766+0000 7f0c39200640 1 -- 172.21.15.27:0/938468231 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3406df10 con 0x7f0c34150ae0 2024-03-28T19:08:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.767+0000 7f0c39200640 1 --2- 172.21.15.27:0/938468231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c340a5c60 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0c280097a0 tx=0x7f0c2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d572edfd0304ad91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.767+0000 7f0c33400640 1 -- 172.21.15.27:0/938468231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28035020 con 0x7f0c34150ae0 2024-03-28T19:08:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.767+0000 7f0c33400640 1 -- 172.21.15.27:0/938468231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28031830 con 0x7f0c34150ae0 2024-03-28T19:08:28.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.767+0000 7f0c33400640 1 -- 172.21.15.27:0/938468231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28031c00 con 0x7f0c34150ae0 2024-03-28T19:08:28.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.768+0000 7f0c3a600640 1 -- 172.21.15.27:0/938468231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 msgr2=0x7f0c340a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.768+0000 7f0c3a600640 1 --2- 172.21.15.27:0/938468231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c340a5c60 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0c280097a0 tx=0x7f0c2802b8d0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.768+0000 7f0c3a600640 1 -- 172.21.15.27:0/938468231 shutdown_connections 2024-03-28T19:08:28.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.768+0000 7f0c3a600640 1 --2- 172.21.15.27:0/938468231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c340a5c60 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.768+0000 7f0c3a600640 1 -- 172.21.15.27:0/938468231 >> 172.21.15.27:0/938468231 conn(0x7f0c3406b0c0 msgr2=0x7f0c3406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:28.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.768+0000 7f0c3a600640 1 -- 172.21.15.27:0/938468231 shutdown_connections 2024-03-28T19:08:28.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.768+0000 7f0c3a600640 1 -- 172.21.15.27:0/938468231 wait complete. 2024-03-28T19:08:28.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.770+0000 7f0c3a600640 1 Processor -- start 2024-03-28T19:08:28.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.770+0000 7f0c3a600640 1 -- start start 2024-03-28T19:08:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.771+0000 7f0c3a600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c340a61a0 0x7f0c3414ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.771+0000 7f0c3a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c34149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.771+0000 7f0c3a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c34153a30 0x7f0c34149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.771+0000 7f0c3a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3406e7b0 con 0x7f0c34150ae0 2024-03-28T19:08:28.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.771+0000 7f0c3a600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3406e4d0 con 0x7f0c340a61a0 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.771+0000 7f0c3a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340c70b0 con 0x7f0c34153a30 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c33e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c34149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c33e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c34149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39138/0 (socket says 172.21.15.27:39138) 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c33e00640 1 -- 172.21.15.27:0/3314312943 learned_addr learned my addr 172.21.15.27:0/3314312943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c39200640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c340a61a0 0x7f0c3414ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c39c00640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c34153a30 0x7f0c34149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c33e00640 1 -- 172.21.15.27:0/3314312943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c34153a30 msgr2=0x7f0c34149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c33e00640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c34153a30 0x7f0c34149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c33e00640 1 -- 172.21.15.27:0/3314312943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c340a61a0 msgr2=0x7f0c3414ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c33e00640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c340a61a0 0x7f0c3414ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.772+0000 7f0c33e00640 1 -- 172.21.15.27:0/3314312943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340708a0 con 0x7f0c34150ae0 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.773+0000 7f0c33e00640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c34149a10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000c480 tx=0x7f0c2000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.773+0000 7f0c30c00640 1 -- 172.21.15.27:0/3314312943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c200052d0 con 0x7f0c34150ae0 2024-03-28T19:08:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.773+0000 7f0c30c00640 1 -- 172.21.15.27:0/3314312943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c20005470 con 0x7f0c34150ae0 2024-03-28T19:08:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.773+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c34071830 con 0x7f0c34150ae0 2024-03-28T19:08:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.773+0000 7f0c30c00640 1 -- 172.21.15.27:0/3314312943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20012870 con 0x7f0c34150ae0 2024-03-28T19:08:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.773+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3414b250 con 0x7f0c34150ae0 2024-03-28T19:08:28.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.775+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c34067dd0 con 0x7f0c34150ae0 2024-03-28T19:08:28.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.776+0000 7f0c30c00640 1 -- 172.21.15.27:0/3314312943 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0c20012a10 con 0x7f0c34150ae0 2024-03-28T19:08:28.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.777+0000 7f0c30c00640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c1406bc00 0x7f0c1406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:28.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.777+0000 7f0c30c00640 1 -- 172.21.15.27:0/3314312943 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7f0c2008aff0 con 0x7f0c34150ae0 2024-03-28T19:08:28.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.778+0000 7f0c39200640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c1406bc00 0x7f0c1406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:28.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.778+0000 7f0c39200640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c1406bc00 0x7f0c1406e0c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0c28036000 tx=0x7f0c28008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:28.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:28.782+0000 7f0c30c00640 1 -- 172.21.15.27:0/3314312943 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0c200591f0 con 0x7f0c34150ae0 2024-03-28T19:08:29.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.051+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c34069640 con 0x7f0c34150ae0 2024-03-28T19:08:29.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.053+0000 7f0c30c00640 1 -- 172.21.15.27:0/3314312943 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5467 (secure 0 0 0) 0x7f0c20058af0 con 0x7f0c34150ae0 2024-03-28T19:08:29.053 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:29.053 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:27.544939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:08:29.053 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 104 2024-03-28T19:08:29.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.055+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c1406bc00 msgr2=0x7f0c1406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:29.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.055+0000 7f0c3a600640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c1406bc00 0x7f0c1406e0c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0c28036000 tx=0x7f0c28008030 comp rx=0 tx=0).stop 2024-03-28T19:08:29.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.055+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 msgr2=0x7f0c34149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:29.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.055+0000 7f0c3a600640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c34149a10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000c480 tx=0x7f0c2000c940 comp rx=0 tx=0).stop 2024-03-28T19:08:29.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.056+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 shutdown_connections 2024-03-28T19:08:29.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.056+0000 7f0c3a600640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0c1406bc00 0x7f0c1406e0c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:29.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.056+0000 7f0c3a600640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0c34153a30 0x7f0c34149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:29.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.056+0000 7f0c3a600640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c34150ae0 0x7f0c34149a10 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:29.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.056+0000 7f0c3a600640 1 --2- 172.21.15.27:0/3314312943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c340a61a0 0x7f0c3414ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:29.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.056+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 >> 172.21.15.27:0/3314312943 conn(0x7f0c3406b0c0 msgr2=0x7f0c340c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:29.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.056+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 shutdown_connections 2024-03-28T19:08:29.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:29.056+0000 7f0c3a600640 1 -- 172.21.15.27:0/3314312943 wait complete. 2024-03-28T19:08:29.075 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:08:30.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.719+0000 7ff101600640 1 Processor -- start 2024-03-28T19:08:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.720+0000 7ff101600640 1 -- start start 2024-03-28T19:08:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.721+0000 7ff101600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc0c8ed0 0x7ff0fc0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.721+0000 7ff101600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff0fc0be5d0 0x7ff0fc155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.721+0000 7ff101600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0fc156230 con 0x7ff0fc0c58c0 2024-03-28T19:08:30.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.721+0000 7ff101600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0fc158bd0 con 0x7ff0fc0be5d0 2024-03-28T19:08:30.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.721+0000 7ff101600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc158eb0 con 0x7ff0fc0c8ed0 2024-03-28T19:08:30.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.721+0000 7ff0fbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff0fc0c58c0 0x7ff0fc1555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59926/0 (socket says 172.21.15.27:59926) 2024-03-28T19:08:30.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.721+0000 7ff0fbe00640 1 -- 172.21.15.27:0/3485500327 learned_addr learned my addr 172.21.15.27:0/3485500327 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:30.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0fb400640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc0c8ed0 0x7ff0fc0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff100c00640 1 --2- 172.21.15.27:0/3485500327 >> v2:172.21.15.155:3300/0 conn(0x7ff0fc0be5d0 0x7ff0fc155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403485358 0 0) 0x7ff0fc156230 con 0x7ff0fc0c58c0 2024-03-28T19:08:30.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e0003520 con 0x7ff0fc0c58c0 2024-03-28T19:08:30.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3408550071 0 0) 0x7ff0e0003520 con 0x7ff0fc0c58c0 2024-03-28T19:08:30.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc0c8ed0 msgr2=0x7ff0fc0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc0c8ed0 0x7ff0fc0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 >> v2:172.21.15.155:3300/0 conn(0x7ff0fc0be5d0 msgr2=0x7ff0fc155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 --2- 172.21.15.27:0/3485500327 >> v2:172.21.15.155:3300/0 conn(0x7ff0fc0be5d0 0x7ff0fc155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc159190 con 0x7ff0fc0c58c0 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7ff0ec001670 con 0x7ff0fc0c58c0 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 >> v1:172.21.15.27:6789/0 conn(0x7ff0fc0c58c0 legacy=0x7ff0fc1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.722+0000 7ff0faa00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0faa00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc0c8ed0 0x7ff0e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0faa00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0e0004d40 0x7ff0e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc158bd0 con 0x7ff0e00039b0 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc158eb0 con 0x7ff0e0004d40 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e000b9c0 con 0x7ff0fc0c8ed0 2024-03-28T19:08:30.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0fbe00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0fbe00640 1 -- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc0c8ed0 msgr2=0x7ff0e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0fb400640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc0c8ed0 0x7ff0e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff100c00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0e0004d40 0x7ff0e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0fbe00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc0c8ed0 0x7ff0e0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0fbe00640 1 -- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0e0004d40 msgr2=0x7ff0e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0fbe00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0e0004d40 0x7ff0e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.723+0000 7ff0fbe00640 1 -- 172.21.15.27:0/3485500327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e0003520 con 0x7ff0e00039b0 2024-03-28T19:08:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.724+0000 7ff100c00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0e0004d40 0x7ff0e00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:08:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.725+0000 7ff0fbe00640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0e0003dd0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec001050 tx=0x7ff0e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2897455ca4390f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.725+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec007d70 con 0x7ff0e00039b0 2024-03-28T19:08:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.725+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ec030070 con 0x7ff0e00039b0 2024-03-28T19:08:30.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.725+0000 7ff0faa00640 1 -- 172.21.15.27:0/3485500327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec02cd80 con 0x7ff0e00039b0 2024-03-28T19:08:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.726+0000 7ff101600640 1 -- 172.21.15.27:0/3485500327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 msgr2=0x7ff0e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.726+0000 7ff101600640 1 --2- 172.21.15.27:0/3485500327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0e0003dd0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec001050 tx=0x7ff0e0005830 comp rx=0 tx=0).stop 2024-03-28T19:08:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.726+0000 7ff100c00640 1 -- 172.21.15.27:0/3485500327 reap_dead start 2024-03-28T19:08:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.726+0000 7ff101600640 1 -- 172.21.15.27:0/3485500327 shutdown_connections 2024-03-28T19:08:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.726+0000 7ff101600640 1 -- 172.21.15.27:0/3485500327 >> 172.21.15.27:0/3485500327 conn(0x7ff0fc06b6f0 msgr2=0x7ff0fc0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.726+0000 7ff101600640 1 -- 172.21.15.27:0/3485500327 shutdown_connections 2024-03-28T19:08:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.726+0000 7ff101600640 1 -- 172.21.15.27:0/3485500327 wait complete. 2024-03-28T19:08:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.728+0000 7ff101600640 1 Processor -- start 2024-03-28T19:08:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.728+0000 7ff101600640 1 -- start start 2024-03-28T19:08:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.729+0000 7ff101600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0fc1df1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.729+0000 7ff101600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0fc1df730 0x7ff0fc0c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.729+0000 7ff101600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc152000 0x7ff0fc0c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.729+0000 7ff101600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc0b0820 con 0x7ff0e00039b0 2024-03-28T19:08:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.729+0000 7ff101600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc159750 con 0x7ff0fc1df730 2024-03-28T19:08:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.729+0000 7ff101600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc0b09a0 con 0x7ff0fc152000 2024-03-28T19:08:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0fc1df1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0fc1df1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39148/0 (socket says 172.21.15.27:39148) 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fbe00640 1 -- 172.21.15.27:0/3326207563 learned_addr learned my addr 172.21.15.27:0/3326207563 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff100c00640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc152000 0x7ff0fc0c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fb400640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0fc1df730 0x7ff0fc0c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fbe00640 1 -- 172.21.15.27:0/3326207563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc152000 msgr2=0x7ff0fc0c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fbe00640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc152000 0x7ff0fc0c4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fbe00640 1 -- 172.21.15.27:0/3326207563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0fc1df730 msgr2=0x7ff0fc0c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fbe00640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0fc1df730 0x7ff0fc0c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.730+0000 7ff0fbe00640 1 -- 172.21.15.27:0/3326207563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc0c4880 con 0x7ff0e00039b0 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.731+0000 7ff0fbe00640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0fc1df1f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00c300 tx=0x7ff0ec000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.731+0000 7ff0f3e00640 1 -- 172.21.15.27:0/3326207563 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec045070 con 0x7ff0e00039b0 2024-03-28T19:08:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.731+0000 7ff0f3e00640 1 -- 172.21.15.27:0/3326207563 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ec004bf0 con 0x7ff0e00039b0 2024-03-28T19:08:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.731+0000 7ff0f3e00640 1 -- 172.21.15.27:0/3326207563 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec001cc0 con 0x7ff0e00039b0 2024-03-28T19:08:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.731+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0fc1ebdc0 con 0x7ff0e00039b0 2024-03-28T19:08:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.731+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0fc152aa0 con 0x7ff0e00039b0 2024-03-28T19:08:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.733+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0fc1ebbe0 con 0x7ff0e00039b0 2024-03-28T19:08:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.734+0000 7ff0f3e00640 1 -- 172.21.15.27:0/3326207563 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff0ec007070 con 0x7ff0e00039b0 2024-03-28T19:08:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.735+0000 7ff0f3e00640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff0dc06bc00 0x7ff0dc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.735+0000 7ff0f3e00640 1 -- 172.21.15.27:0/3326207563 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7ff0ec0ac410 con 0x7ff0e00039b0 2024-03-28T19:08:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.736+0000 7ff0fb400640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff0dc06bc00 0x7ff0dc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.736+0000 7ff0fb400640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff0dc06bc00 0x7ff0dc06e0c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc1dfb30 tx=0x7ff0e8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:30.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:30.741+0000 7ff0f3e00640 1 -- 172.21.15.27:0/3326207563 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff0fc1ebbe0 con 0x7ff0e00039b0 2024-03-28T19:08:31.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:31.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:08:31.088 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:08:31.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.096+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0fc069640 con 0x7ff0e00039b0 2024-03-28T19:08:31.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.097+0000 7ff0f3e00640 1 -- 172.21.15.27:0/3326207563 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5471 (secure 0 0 0) 0x7ff0ec07a610 con 0x7ff0e00039b0 2024-03-28T19:08:31.096 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:31.096 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:31.097 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 106 2024-03-28T19:08:31.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff0dc06bc00 msgr2=0x7ff0dc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff0dc06bc00 0x7ff0dc06e0c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc1dfb30 tx=0x7ff0e8008040 comp rx=0 tx=0).stop 2024-03-28T19:08:31.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 msgr2=0x7ff0fc1df1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0fc1df1f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00c300 tx=0x7ff0ec000fe0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 shutdown_connections 2024-03-28T19:08:31.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff0dc06bc00 0x7ff0dc06e0c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0fc152000 0x7ff0fc0c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0fc1df730 0x7ff0fc0c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 --2- 172.21.15.27:0/3326207563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0e00039b0 0x7ff0fc1df1f0 secure :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00c300 tx=0x7ff0ec000fe0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.100+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 >> 172.21.15.27:0/3326207563 conn(0x7ff0fc06b6f0 msgr2=0x7ff0fc0ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:31.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.101+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 shutdown_connections 2024-03-28T19:08:31.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.101+0000 7ff101600640 1 -- 172.21.15.27:0/3326207563 wait complete. 2024-03-28T19:08:31.123 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:08:31.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.314+0000 7f38d0c00640 1 Processor -- start 2024-03-28T19:08:31.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.314+0000 7f38d0c00640 1 -- start start 2024-03-28T19:08:31.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.315+0000 7f38d0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc1551a0 0x7f38cc0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:31.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.315+0000 7f38d0c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc156870 con 0x7f38cc0c5580 2024-03-28T19:08:31.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.315+0000 7f38d0c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc155cf0 con 0x7f38cc0aada0 2024-03-28T19:08:31.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.315+0000 7f38d0c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc155fd0 con 0x7f38cc1551a0 2024-03-28T19:08:31.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.316+0000 7f38caa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f38cc0c5580 0x7f38cc0c1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59934/0 (socket says 172.21.15.27:59934) 2024-03-28T19:08:31.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.316+0000 7f38caa00640 1 -- 172.21.15.27:0/1111517687 learned_addr learned my addr 172.21.15.27:0/1111517687 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:31.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.316+0000 7f38cb400640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc1551a0 0x7f38cc0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:31.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.316+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682181741 0 0) 0x7f38cc156870 con 0x7f38cc0c5580 2024-03-28T19:08:31.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.316+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b0003520 con 0x7f38cc0c5580 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2572159754 0 0) 0x7f38b0003520 con 0x7f38cc0c5580 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc1551a0 msgr2=0x7f38cc0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc1551a0 0x7f38cc0c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 >> v1:172.21.15.155:6789/0 conn(0x7f38cc0aada0 legacy=0x7f38cc0c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc1562b0 con 0x7f38cc0c5580 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f38b8001670 con 0x7f38cc0c5580 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 >> v1:172.21.15.27:6789/0 conn(0x7f38cc0c5580 legacy=0x7f38cc0c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b00039e0 0x7f38b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b0004800 0x7f38b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b0005230 0x7f38b000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc155fd0 con 0x7f38b0004800 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b000e350 con 0x7f38b0005230 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b000e550 con 0x7f38b00039e0 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38cb400640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc1551a0 0x7f38cc0c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.317+0000 7f38caa00640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b0004800 0x7f38b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.318+0000 7f38caa00640 1 -- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b00039e0 msgr2=0x7f38b0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.318+0000 7f38caa00640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b00039e0 0x7f38b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.318+0000 7f38caa00640 1 -- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b0005230 msgr2=0x7f38b000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.318+0000 7f38caa00640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b0005230 0x7f38b000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.318+0000 7f38caa00640 1 -- 172.21.15.27:0/1111517687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b0003520 con 0x7f38b0004800 2024-03-28T19:08:31.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.319+0000 7f38caa00640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b0004800 0x7f38b0004c80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c110 tx=0x7f38b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17dd25ba103b6fe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.319+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8007c40 con 0x7f38b0004800 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.320+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b8030070 con 0x7f38b0004800 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.320+0000 7f38ca000640 1 -- 172.21.15.27:0/1111517687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8005d70 con 0x7f38b0004800 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.320+0000 7f38d0c00640 1 -- 172.21.15.27:0/1111517687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b0004800 msgr2=0x7f38b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.320+0000 7f38d0c00640 1 --2- 172.21.15.27:0/1111517687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b0004800 0x7f38b0004c80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c110 tx=0x7f38b0005830 comp rx=0 tx=0).stop 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.320+0000 7f38cbe00640 1 -- 172.21.15.27:0/1111517687 reap_dead start 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.320+0000 7f38d0c00640 1 -- 172.21.15.27:0/1111517687 shutdown_connections 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.320+0000 7f38d0c00640 1 -- 172.21.15.27:0/1111517687 >> 172.21.15.27:0/1111517687 conn(0x7f38cc06b0c0 msgr2=0x7f38cc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.321+0000 7f38d0c00640 1 -- 172.21.15.27:0/1111517687 shutdown_connections 2024-03-28T19:08:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.321+0000 7f38d0c00640 1 -- 172.21.15.27:0/1111517687 wait complete. 2024-03-28T19:08:31.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.323+0000 7f38d0c00640 1 Processor -- start 2024-03-28T19:08:31.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.323+0000 7f38d0c00640 1 -- start start 2024-03-28T19:08:31.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38d0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc0bd9d0 0x7f38cc0bddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:31.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38d0c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38cc14ef30 0x7f38cc1520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:31.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38d0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc152620 0x7f38cc0717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38d0c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc155580 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38d0c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc1559a0 con 0x7f38cc14ef30 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38d0c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc155700 con 0x7f38cc152620 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc0bd9d0 0x7f38cc0bddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc0bd9d0 0x7f38cc0bddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39168/0 (socket says 172.21.15.27:39168) 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.324+0000 7f38cb400640 1 -- 172.21.15.27:0/1099039423 learned_addr learned my addr 172.21.15.27:0/1099039423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38caa00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38cc14ef30 0x7f38cc1520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38cbe00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc152620 0x7f38cc0717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38cb400640 1 -- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc152620 msgr2=0x7f38cc0717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38cb400640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc152620 0x7f38cc0717c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38cb400640 1 -- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38cc14ef30 msgr2=0x7f38cc1520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38cb400640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38cc14ef30 0x7f38cc1520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38cb400640 1 -- 172.21.15.27:0/1099039423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc073010 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38caa00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38cc14ef30 0x7f38cc1520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:08:31.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38cbe00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc152620 0x7f38cc0717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:08:31.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38cb400640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc0bd9d0 0x7f38cc0bddb0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00eb10 tx=0x7f38bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:31.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.325+0000 7f38c3400640 1 -- 172.21.15.27:0/1099039423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc004030 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.326+0000 7f38c3400640 1 -- 172.21.15.27:0/1099039423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc0041f0 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.326+0000 7f38c3400640 1 -- 172.21.15.27:0/1099039423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc00c870 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.326+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc1e7d90 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.326+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc0bbab0 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.327+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38cc067dd0 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.329+0000 7f38c3400640 1 -- 172.21.15.27:0/1099039423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f38bc00ca10 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.330+0000 7f38c3400640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38ac06bc00 0x7f38ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:31.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.330+0000 7f38c3400640 1 -- 172.21.15.27:0/1099039423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7f38bc08b310 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.330+0000 7f38caa00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38ac06bc00 0x7f38ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:31.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.331+0000 7f38caa00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38ac06bc00 0x7f38ac06e0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f38b8034000 tx=0x7f38b8039040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:31.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.336+0000 7f38c3400640 1 -- 172.21.15.27:0/1099039423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f38bc017020 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.663+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38cc073510 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.664+0000 7f38c3400640 1 -- 172.21.15.27:0/1099039423 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f38cc073510 con 0x7f38cc0bd9d0 2024-03-28T19:08:31.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:31.664 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:31.664 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 106 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38ac06bc00 msgr2=0x7f38ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38ac06bc00 0x7f38ac06e0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f38b8034000 tx=0x7f38b8039040 comp rx=0 tx=0).stop 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc0bd9d0 msgr2=0x7f38cc0bddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc0bd9d0 0x7f38cc0bddb0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00eb10 tx=0x7f38bc009e30 comp rx=0 tx=0).stop 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 shutdown_connections 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38ac06bc00 0x7f38ac06e0c0 secure :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f38b8034000 tx=0x7f38b8039040 comp rx=0 tx=0).stop 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38cc152620 0x7f38cc0717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38cc14ef30 0x7f38cc1520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 --2- 172.21.15.27:0/1099039423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38cc0bd9d0 0x7f38cc0bddb0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.667+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 >> 172.21.15.27:0/1099039423 conn(0x7f38cc06b0c0 msgr2=0x7f38cc153ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.668+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 shutdown_connections 2024-03-28T19:08:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:31.668+0000 7f38d0c00640 1 -- 172.21.15.27:0/1099039423 wait complete. 2024-03-28T19:08:31.688 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6411 2024-03-28T19:08:31.688 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.c 2024-03-28T19:08:32.730 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:08:32.730+0000 7f68e2e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:08:33.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.371+0000 7f1d1f000640 1 Processor -- start 2024-03-28T19:08:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.371+0000 7f1d1f000640 1 -- start start 2024-03-28T19:08:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.372+0000 7f1d1f000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c8ed0 0x7f1d180c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.372+0000 7f1d1f000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d18158bd0 con 0x7f1d180be5d0 2024-03-28T19:08:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.372+0000 7f1d1f000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d18158eb0 con 0x7f1d180c58c0 2024-03-28T19:08:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.372+0000 7f1d1f000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d18159190 con 0x7f1d180c8ed0 2024-03-28T19:08:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.373+0000 7f1d1e600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1d180be5d0 0x7f1d18155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59942/0 (socket says 172.21.15.27:59942) 2024-03-28T19:08:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.373+0000 7f1d1e600640 1 -- 172.21.15.27:0/892236952 learned_addr learned my addr 172.21.15.27:0/892236952 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.373+0000 7f1d1d200640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c8ed0 0x7f1d180c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.373+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33610921 0 0) 0x7f1d18158bd0 con 0x7f1d180be5d0 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d00003520 con 0x7f1d180be5d0 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3034520851 0 0) 0x7f1d00003520 con 0x7f1d180be5d0 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c8ed0 msgr2=0x7f1d180c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c8ed0 0x7f1d180c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 >> v1:172.21.15.155:6789/0 conn(0x7f1d180c58c0 legacy=0x7f1d181555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d18159470 con 0x7f1d180be5d0 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f1d100019f0 con 0x7f1d180be5d0 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 >> v1:172.21.15.27:6789/0 conn(0x7f1d180be5d0 legacy=0x7f1d18155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c8ed0 0x7f1d000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d00003f60 0x7f1d00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d18159190 con 0x7f1d00004d40 2024-03-28T19:08:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0000bb50 con 0x7f1d00003f60 2024-03-28T19:08:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.374+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0000bd50 con 0x7f1d180c8ed0 2024-03-28T19:08:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.375+0000 7f1d1e600640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.375+0000 7f1d1d200640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d00003f60 0x7f1d00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.375+0000 7f1d1dc00640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c8ed0 0x7f1d000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.375+0000 7f1d1e600640 1 -- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c8ed0 msgr2=0x7f1d000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.375+0000 7f1d1e600640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c8ed0 0x7f1d000039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.375+0000 7f1d1e600640 1 -- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d00003f60 msgr2=0x7f1d00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.375+0000 7f1d1e600640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d00003f60 0x7f1d00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.375+0000 7f1d1e600640 1 -- 172.21.15.27:0/892236952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d00003520 con 0x7f1d00004d40 2024-03-28T19:08:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.377+0000 7f1d1e600640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d000051c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1d180bea40 tx=0x7f1d00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7612143b63d060 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:33.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.377+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1000a4a0 con 0x7f1d00004d40 2024-03-28T19:08:33.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.377+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d10034630 con 0x7f1d00004d40 2024-03-28T19:08:33.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.377+0000 7f1d17e00640 1 -- 172.21.15.27:0/892236952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d100349e0 con 0x7f1d00004d40 2024-03-28T19:08:33.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.378+0000 7f1d1f000640 1 -- 172.21.15.27:0/892236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 msgr2=0x7f1d000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.378+0000 7f1d1f000640 1 --2- 172.21.15.27:0/892236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d000051c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1d180bea40 tx=0x7f1d00005830 comp rx=0 tx=0).stop 2024-03-28T19:08:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.378+0000 7f1d1e600640 1 -- 172.21.15.27:0/892236952 reap_dead start 2024-03-28T19:08:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.378+0000 7f1d1f000640 1 -- 172.21.15.27:0/892236952 shutdown_connections 2024-03-28T19:08:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.378+0000 7f1d1f000640 1 -- 172.21.15.27:0/892236952 >> 172.21.15.27:0/892236952 conn(0x7f1d1806b6f0 msgr2=0x7f1d180acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.378+0000 7f1d1f000640 1 -- 172.21.15.27:0/892236952 shutdown_connections 2024-03-28T19:08:33.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.378+0000 7f1d1f000640 1 -- 172.21.15.27:0/892236952 wait complete. 2024-03-28T19:08:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.380+0000 7f1d1f000640 1 Processor -- start 2024-03-28T19:08:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.380+0000 7f1d1f000640 1 -- start start 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.381+0000 7f1d1f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d180c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.381+0000 7f1d1f000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c26b0 0x7f1d180c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.381+0000 7f1d1f000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d181ee890 0x7f1d181eed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.381+0000 7f1d1f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d180b09a0 con 0x7f1d00004d40 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.381+0000 7f1d1f000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d180b0820 con 0x7f1d181ee890 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.381+0000 7f1d1f000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d180ad7d0 con 0x7f1d180c26b0 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d180c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d180c4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39196/0 (socket says 172.21.15.27:39196) 2024-03-28T19:08:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1dc00640 1 -- 172.21.15.27:0/2172823715 learned_addr learned my addr 172.21.15.27:0/2172823715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1d200640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c26b0 0x7f1d180c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1e600640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d181ee890 0x7f1d181eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1dc00640 1 -- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c26b0 msgr2=0x7f1d180c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1dc00640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c26b0 0x7f1d180c2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1dc00640 1 -- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d181ee890 msgr2=0x7f1d181eed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1dc00640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d181ee890 0x7f1d181eed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1dc00640 1 -- 172.21.15.27:0/2172823715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d180c20b0 con 0x7f1d00004d40 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.382+0000 7f1d1e600640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d181ee890 0x7f1d181eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.383+0000 7f1d1dc00640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d180c4a90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00e5b0 tx=0x7f1d0c00ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.383+0000 7f1d15600640 1 -- 172.21.15.27:0/2172823715 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0052e0 con 0x7f1d00004d40 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.383+0000 7f1d15600640 1 -- 172.21.15.27:0/2172823715 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c018070 con 0x7f1d00004d40 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.383+0000 7f1d15600640 1 -- 172.21.15.27:0/2172823715 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c003830 con 0x7f1d00004d40 2024-03-28T19:08:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.383+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d180c2390 con 0x7f1d00004d40 2024-03-28T19:08:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.383+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d180c33b0 con 0x7f1d00004d40 2024-03-28T19:08:33.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.385+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d18069640 con 0x7f1d00004d40 2024-03-28T19:08:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.386+0000 7f1d15600640 1 -- 172.21.15.27:0/2172823715 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1d0c0039d0 con 0x7f1d00004d40 2024-03-28T19:08:33.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.387+0000 7f1d15600640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1cf806bc50 0x7f1cf806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.387+0000 7f1d15600640 1 -- 172.21.15.27:0/2172823715 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7f1d0c08afc0 con 0x7f1d00004d40 2024-03-28T19:08:33.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.388+0000 7f1d1d200640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1cf806bc50 0x7f1cf806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:33.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.389+0000 7f1d1d200640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1cf806bc50 0x7f1cf806e110 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1d18156000 tx=0x7f1d04008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:33.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.395+0000 7f1d15600640 1 -- 172.21.15.27:0/2172823715 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1d0c059130 con 0x7f1d00004d40 2024-03-28T19:08:33.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.733+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d180c5ca0 con 0x7f1d00004d40 2024-03-28T19:08:33.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.734+0000 7f1d15600640 1 -- 172.21.15.27:0/2172823715 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f1d0c058a30 con 0x7f1d00004d40 2024-03-28T19:08:33.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:33.789 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 106 2024-03-28T19:08:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1cf806bc50 msgr2=0x7f1cf806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1cf806bc50 0x7f1cf806e110 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1d18156000 tx=0x7f1d04008040 comp rx=0 tx=0).stop 2024-03-28T19:08:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 msgr2=0x7f1d180c4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:33.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d180c4a90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00e5b0 tx=0x7f1d0c00ea70 comp rx=0 tx=0).stop 2024-03-28T19:08:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 shutdown_connections 2024-03-28T19:08:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1cf806bc50 0x7f1cf806e110 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d181ee890 0x7f1d181eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d180c26b0 0x7f1d180c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 --2- 172.21.15.27:0/2172823715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d00004d40 0x7f1d180c4a90 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.737+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 >> 172.21.15.27:0/2172823715 conn(0x7f1d1806b6f0 msgr2=0x7f1d180caf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.738+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 shutdown_connections 2024-03-28T19:08:33.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:33.738+0000 7f1d1f000640 1 -- 172.21.15.27:0/2172823715 wait complete. 2024-03-28T19:08:33.793 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:08:35.689 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-03-28T19:08:35.689 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:08:35.689 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:08:35.698 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:08:35.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:35.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:35.910 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:08:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.955+0000 7fb12fe00640 1 Processor -- start 2024-03-28T19:08:35.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.955+0000 7fb12fe00640 1 -- start start 2024-03-28T19:08:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.957+0000 7fb12fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300c69a0 0x7fb1300c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.957+0000 7fb12fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb130150ec0 con 0x7fb130150ae0 2024-03-28T19:08:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.957+0000 7fb12fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13006dc70 con 0x7fb130153a30 2024-03-28T19:08:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.957+0000 7fb12fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb130153550 con 0x7fb1300c69a0 2024-03-28T19:08:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.958+0000 7fb12ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb130150ae0 0x7fb1300a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59952/0 (socket says 172.21.15.27:59952) 2024-03-28T19:08:35.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.958+0000 7fb12ea00640 1 -- 172.21.15.27:0/959105488 learned_addr learned my addr 172.21.15.27:0/959105488 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.958+0000 7fb12e000640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300c69a0 0x7fb1300c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.958+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636506403 0 0) 0x7fb120001110 con 0x7fb130150ae0 2024-03-28T19:08:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb114003520 con 0x7fb130150ae0 2024-03-28T19:08:35.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153951203 0 0) 0x7fb13006dc70 con 0x7fb130153a30 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb120001110 con 0x7fb130153a30 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3451508847 0 0) 0x7fb114003520 con 0x7fb130150ae0 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300c69a0 msgr2=0x7fb1300c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300c69a0 0x7fb1300c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 >> v1:172.21.15.155:6789/0 conn(0x7fb130153a30 legacy=0x7fb13015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13006df10 con 0x7fb130150ae0 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fb1200017c0 con 0x7fb130150ae0 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 >> v1:172.21.15.27:6789/0 conn(0x7fb130150ae0 legacy=0x7fb1300a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb114003b70 0x7fb114003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb114004800 0x7fb114004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300c69a0 0x7fb1140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb130153550 con 0x7fb114004800 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11400fe80 con 0x7fb114003b70 2024-03-28T19:08:35.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.959+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb114010080 con 0x7fb1300c69a0 2024-03-28T19:08:35.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.960+0000 7fb12e000640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb114004800 0x7fb114004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:35.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.960+0000 7fb12f400640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300c69a0 0x7fb1140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:35.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.961+0000 7fb12e000640 1 -- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300c69a0 msgr2=0x7fb1140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:35.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.961+0000 7fb12ea00640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb114003b70 0x7fb114003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:35.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.961+0000 7fb12e000640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300c69a0 0x7fb1140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:35.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.961+0000 7fb12e000640 1 -- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb114003b70 msgr2=0x7fb114003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:35.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.961+0000 7fb12e000640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb114003b70 0x7fb114003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:35.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.961+0000 7fb12e000640 1 -- 172.21.15.27:0/959105488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13006df10 con 0x7fb114004800 2024-03-28T19:08:35.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.961+0000 7fb12ea00640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb114003b70 0x7fb114003f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:08:35.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.963+0000 7fb12e000640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb114004800 0x7fb114004c80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009a90 tx=0x7fb114005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd636a80de403fde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.963+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c039070 con 0x7fb114004800 2024-03-28T19:08:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.963+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c02c650 con 0x7fb114004800 2024-03-28T19:08:35.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.963+0000 7fb12d600640 1 -- 172.21.15.27:0/959105488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c009690 con 0x7fb114004800 2024-03-28T19:08:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.964+0000 7fb12fe00640 1 -- 172.21.15.27:0/959105488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb114004800 msgr2=0x7fb114004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.964+0000 7fb12fe00640 1 --2- 172.21.15.27:0/959105488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb114004800 0x7fb114004c80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009a90 tx=0x7fb114005830 comp rx=0 tx=0).stop 2024-03-28T19:08:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.964+0000 7fb12f400640 1 -- 172.21.15.27:0/959105488 reap_dead start 2024-03-28T19:08:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.965+0000 7fb12fe00640 1 -- 172.21.15.27:0/959105488 shutdown_connections 2024-03-28T19:08:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.965+0000 7fb12fe00640 1 -- 172.21.15.27:0/959105488 >> 172.21.15.27:0/959105488 conn(0x7fb13006b0c0 msgr2=0x7fb13006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.965+0000 7fb12fe00640 1 -- 172.21.15.27:0/959105488 shutdown_connections 2024-03-28T19:08:35.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.965+0000 7fb12fe00640 1 -- 172.21.15.27:0/959105488 wait complete. 2024-03-28T19:08:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.968+0000 7fb12fe00640 1 Processor -- start 2024-03-28T19:08:35.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.968+0000 7fb12fe00640 1 -- start start 2024-03-28T19:08:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.969+0000 7fb12fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb13014f0e0 0x7fb1300a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:35.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.969+0000 7fb12fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300aac00 0x7fb1300a9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.969+0000 7fb12fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb130149800 0x7fb130149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.969+0000 7fb12fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13006e7b0 con 0x7fb130149800 2024-03-28T19:08:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.969+0000 7fb12fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13006e4d0 con 0x7fb13014f0e0 2024-03-28T19:08:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.969+0000 7fb12fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300a6370 con 0x7fb1300aac00 2024-03-28T19:08:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.970+0000 7fb12f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb130149800 0x7fb130149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.970+0000 7fb12e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300aac00 0x7fb1300a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:35.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.970+0000 7fb12ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb13014f0e0 0x7fb1300a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.970+0000 7fb12e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300aac00 0x7fb1300a9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:57334/0 (socket says 172.21.15.27:57334) 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.970+0000 7fb12f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb130149800 0x7fb130149cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39226/0 (socket says 172.21.15.27:39226) 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.970+0000 7fb12e000640 1 -- 172.21.15.27:0/1649383257 learned_addr learned my addr 172.21.15.27:0/1649383257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.971+0000 7fb12e000640 1 -- 172.21.15.27:0/1649383257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb13014f0e0 msgr2=0x7fb1300a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.971+0000 7fb12e000640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb13014f0e0 0x7fb1300a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.971+0000 7fb12e000640 1 -- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb130149800 msgr2=0x7fb130149cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.971+0000 7fb12e000640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb130149800 0x7fb130149cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.971+0000 7fb12e000640 1 -- 172.21.15.27:0/1649383257 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300ab090 con 0x7fb1300aac00 2024-03-28T19:08:35.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.971+0000 7fb12f400640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb130149800 0x7fb130149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:08:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.972+0000 7fb12e000640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300aac00 0x7fb1300a9720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb11c037040 tx=0x7fb11c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.972+0000 7fb127400640 1 -- 172.21.15.27:0/1649383257 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c039070 con 0x7fb1300aac00 2024-03-28T19:08:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.972+0000 7fb127400640 1 -- 172.21.15.27:0/1649383257 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c043070 con 0x7fb1300aac00 2024-03-28T19:08:35.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.972+0000 7fb127400640 1 -- 172.21.15.27:0/1649383257 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c003a50 con 0x7fb1300aac00 2024-03-28T19:08:35.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.972+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1300a6980 con 0x7fb1300aac00 2024-03-28T19:08:35.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.973+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1300abab0 con 0x7fb1300aac00 2024-03-28T19:08:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.975+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb130067dd0 con 0x7fb1300aac00 2024-03-28T19:08:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.977+0000 7fb127400640 1 -- 172.21.15.27:0/1649383257 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb11c003bf0 con 0x7fb1300aac00 2024-03-28T19:08:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.978+0000 7fb127400640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb11006bc00 0x7fb11006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.978+0000 7fb127400640 1 -- 172.21.15.27:0/1649383257 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7fb11c0ac500 con 0x7fb1300aac00 2024-03-28T19:08:35.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.978+0000 7fb12ea00640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb11006bc00 0x7fb11006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.979+0000 7fb12ea00640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb11006bc00 0x7fb11006e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb120000fd0 tx=0x7fb120009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:35.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:35.995+0000 7fb127400640 1 -- 172.21.15.27:0/1649383257 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb11c04c020 con 0x7fb1300aac00 2024-03-28T19:08:36.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.039+0000 7ffb23200640 1 Processor -- start 2024-03-28T19:08:36.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.039+0000 7ffb23200640 1 -- start start 2024-03-28T19:08:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.041+0000 7ffb23200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffb240be5d0 0x7ffb240c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.041+0000 7ffb23200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb24158bd0 con 0x7ffb240c58c0 2024-03-28T19:08:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.041+0000 7ffb23200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb24158eb0 con 0x7ffb240c8ed0 2024-03-28T19:08:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.041+0000 7ffb23200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb24159190 con 0x7ffb240be5d0 2024-03-28T19:08:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.041+0000 7ffb21e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffb240c58c0 0x7ffb241555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59954/0 (socket says 172.21.15.27:59954) 2024-03-28T19:08:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.041+0000 7ffb21e00640 1 -- 172.21.15.27:0/3149734844 learned_addr learned my addr 172.21.15.27:0/3149734844 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.042+0000 7ffb22800640 1 --2- 172.21.15.27:0/3149734844 >> v2:172.21.15.155:3300/0 conn(0x7ffb240be5d0 0x7ffb240c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.042+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670222201 0 0) 0x7ffb24158bd0 con 0x7ffb240c58c0 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.042+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb08003520 con 0x7ffb240c58c0 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192653885 0 0) 0x7ffb24158eb0 con 0x7ffb240c8ed0 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb24158bd0 con 0x7ffb240c8ed0 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3789110049 0 0) 0x7ffb08003520 con 0x7ffb240c58c0 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 >> v2:172.21.15.155:3300/0 conn(0x7ffb240be5d0 msgr2=0x7ffb240c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 --2- 172.21.15.27:0/3149734844 >> v2:172.21.15.155:3300/0 conn(0x7ffb240be5d0 0x7ffb240c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 >> v1:172.21.15.155:6789/0 conn(0x7ffb240c8ed0 legacy=0x7ffb24155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24159470 con 0x7ffb240c58c0 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7ffb140015f0 con 0x7ffb240c58c0 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 >> v1:172.21.15.27:6789/0 conn(0x7ffb240c58c0 legacy=0x7ffb241555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb08003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb08004800 0x7ffb08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb08005230 0x7ffb0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb24159190 con 0x7ffb08003b40 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb080149a0 con 0x7ffb08005230 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb08014ba0 con 0x7ffb08004800 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.043+0000 7ffb21e00640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb08003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.044+0000 7ffb21400640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb08004800 0x7ffb08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.044+0000 7ffb22800640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb08005230 0x7ffb0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.044+0000 7ffb21e00640 1 -- 172.21.15.27:0/3149734844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb08004800 msgr2=0x7ffb08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.044+0000 7ffb21e00640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb08004800 0x7ffb08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.044+0000 7ffb21e00640 1 -- 172.21.15.27:0/3149734844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb08005230 msgr2=0x7ffb0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.044+0000 7ffb21e00640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb08005230 0x7ffb0800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.044+0000 7ffb21e00640 1 -- 172.21.15.27:0/3149734844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb08003520 con 0x7ffb08003b40 2024-03-28T19:08:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.044+0000 7ffb22800640 1 -- 172.21.15.27:0/3149734844 reap_dead start 2024-03-28T19:08:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.046+0000 7ffb21e00640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb08003f40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400c190 tx=0x7ffb08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f41a7a901c85057 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.046+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14007c40 con 0x7ffb08003b40 2024-03-28T19:08:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.046+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14030070 con 0x7ffb08003b40 2024-03-28T19:08:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.046+0000 7ffb20a00640 1 -- 172.21.15.27:0/3149734844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14005d50 con 0x7ffb08003b40 2024-03-28T19:08:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.047+0000 7ffb23200640 1 -- 172.21.15.27:0/3149734844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 msgr2=0x7ffb08003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.047+0000 7ffb23200640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb08003f40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400c190 tx=0x7ffb08005830 comp rx=0 tx=0).stop 2024-03-28T19:08:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.047+0000 7ffb23200640 1 -- 172.21.15.27:0/3149734844 shutdown_connections 2024-03-28T19:08:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.048+0000 7ffb23200640 1 --2- 172.21.15.27:0/3149734844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb08003f40 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.048+0000 7ffb23200640 1 -- 172.21.15.27:0/3149734844 >> 172.21.15.27:0/3149734844 conn(0x7ffb2406b6f0 msgr2=0x7ffb240acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.048+0000 7ffb23200640 1 -- 172.21.15.27:0/3149734844 shutdown_connections 2024-03-28T19:08:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.048+0000 7ffb23200640 1 -- 172.21.15.27:0/3149734844 wait complete. 2024-03-28T19:08:36.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.051+0000 7ffb23200640 1 Processor -- start 2024-03-28T19:08:36.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.051+0000 7ffb23200640 1 -- start start 2024-03-28T19:08:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.052+0000 7ffb23200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb2414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.052+0000 7ffb23200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb240c4b70 0x7ffb2414b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.053+0000 7ffb23200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb2414b950 0x7ffb241e7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.053+0000 7ffb23200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb240b09a0 con 0x7ffb08003b40 2024-03-28T19:08:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.053+0000 7ffb23200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb240b0820 con 0x7ffb240c4b70 2024-03-28T19:08:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.053+0000 7ffb23200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb240ad7d0 con 0x7ffb2414b950 2024-03-28T19:08:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.053+0000 7ffb21400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb240c4b70 0x7ffb2414b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.053+0000 7ffb21400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb240c4b70 0x7ffb2414b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:32996/0 (socket says 172.21.15.27:32996) 2024-03-28T19:08:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.053+0000 7ffb21400640 1 -- 172.21.15.27:0/2401599854 learned_addr learned my addr 172.21.15.27:0/2401599854 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.054+0000 7ffb21e00640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb2414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.054+0000 7ffb22800640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb2414b950 0x7ffb241e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.054+0000 7ffb21e00640 1 -- 172.21.15.27:0/2401599854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb2414b950 msgr2=0x7ffb241e7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.054+0000 7ffb21e00640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb2414b950 0x7ffb241e7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.054+0000 7ffb21e00640 1 -- 172.21.15.27:0/2401599854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb240c4b70 msgr2=0x7ffb2414b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.054+0000 7ffb21e00640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb240c4b70 0x7ffb2414b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.054+0000 7ffb21e00640 1 -- 172.21.15.27:0/2401599854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2414e170 con 0x7ffb08003b40 2024-03-28T19:08:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.055+0000 7ffb21e00640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb2414d7f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ffb14031000 tx=0x7ffb140019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.056+0000 7ffb1a000640 1 -- 172.21.15.27:0/2401599854 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14001b90 con 0x7ffb08003b40 2024-03-28T19:08:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.056+0000 7ffb1a000640 1 -- 172.21.15.27:0/2401599854 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14004a00 con 0x7ffb08003b40 2024-03-28T19:08:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.056+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb240ac4a0 con 0x7ffb08003b40 2024-03-28T19:08:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.056+0000 7ffb1a000640 1 -- 172.21.15.27:0/2401599854 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14004df0 con 0x7ffb08003b40 2024-03-28T19:08:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.056+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb240ac740 con 0x7ffb08003b40 2024-03-28T19:08:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.059+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffafc005290 con 0x7ffb08003b40 2024-03-28T19:08:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.060+0000 7ffb1a000640 1 -- 172.21.15.27:0/2401599854 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffb14007070 con 0x7ffb08003b40 2024-03-28T19:08:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.061+0000 7ffb1a000640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb0406bbd0 0x7ffb0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:36.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.062+0000 7ffb1a000640 1 -- 172.21.15.27:0/2401599854 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7ffb140ac820 con 0x7ffb08003b40 2024-03-28T19:08:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.063+0000 7ffb21400640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb0406bbd0 0x7ffb0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.064+0000 7ffb21400640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb0406bbd0 0x7ffb0406e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffb2414f370 tx=0x7ffb10009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:36.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.073+0000 7ffb1a000640 1 -- 172.21.15.27:0/2401599854 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffb1407a9a0 con 0x7ffb08003b40 2024-03-28T19:08:36.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.297+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb13006da50 con 0x7fb1300aac00 2024-03-28T19:08:36.299 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:36.299 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:36.299 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 106 2024-03-28T19:08:36.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.360+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffafc005580 con 0x7ffb08003b40 2024-03-28T19:08:36.361 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:36.361 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:36.362 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 106 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.299+0000 7fb127400640 1 -- 172.21.15.27:0/1649383257 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5471 (secure 0 0 0) 0x7fb11c07a030 con 0x7fb1300aac00 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb11006bc00 msgr2=0x7fb11006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb11006bc00 0x7fb11006e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb120000fd0 tx=0x7fb120009040 comp rx=0 tx=0).stop 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300aac00 msgr2=0x7fb1300a9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300aac00 0x7fb1300a9720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb11c037040 tx=0x7fb11c004730 comp rx=0 tx=0).stop 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 shutdown_connections 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb11006bc00 0x7fb11006e0c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb130149800 0x7fb130149cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1300aac00 0x7fb1300a9720 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 --2- 172.21.15.27:0/1649383257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb13014f0e0 0x7fb1300a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.302+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 >> 172.21.15.27:0/1649383257 conn(0x7fb13006b0c0 msgr2=0x7fb130155040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.303+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 shutdown_connections 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.303+0000 7fb12fe00640 1 -- 172.21.15.27:0/1649383257 wait complete. 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.361+0000 7ffb1a000640 1 -- 172.21.15.27:0/2401599854 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5471 (secure 0 0 0) 0x7ffb1407a2a0 con 0x7ffb08003b40 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb0406bbd0 msgr2=0x7ffb0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb0406bbd0 0x7ffb0406e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffb2414f370 tx=0x7ffb10009040 comp rx=0 tx=0).stop 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 msgr2=0x7ffb2414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb2414d7f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ffb14031000 tx=0x7ffb140019b0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 shutdown_connections 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb0406bbd0 0x7ffb0406e090 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb2414b950 0x7ffb241e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb240c4b70 0x7ffb2414b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 --2- 172.21.15.27:0/2401599854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb08003b40 0x7ffb2414d7f0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 >> 172.21.15.27:0/2401599854 conn(0x7ffb2406b6f0 msgr2=0x7ffb240c9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 shutdown_connections 2024-03-28T19:08:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:36.364+0000 7ffb23200640 1 -- 172.21.15.27:0/2401599854 wait complete. 2024-03-28T19:08:36.426 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:08:36.427 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:08:38.427 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:38.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.674+0000 7f4a4a200640 1 Processor -- start 2024-03-28T19:08:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.674+0000 7f7c39c00640 1 Processor -- start 2024-03-28T19:08:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.674+0000 7f4a4a200640 1 -- start start 2024-03-28T19:08:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.674+0000 7f7c39c00640 1 -- start start 2024-03-28T19:08:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.675+0000 7f4a4a200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a4414f6d0 0x7f4a44159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.675+0000 7f4a4a200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4a44071cf0 0x7f4a440c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.675+0000 7f7c39c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c34152f80 0x7f7c340c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.675+0000 7f7c39c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7c340c7df0 0x7f7c34155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f7c39c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c340c7310 con 0x7f7c340c7970 2024-03-28T19:08:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f7c39c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c34158d40 con 0x7f7c340c7df0 2024-03-28T19:08:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f7c39c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34159020 con 0x7f7c34152f80 2024-03-28T19:08:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.675+0000 7f4a4a200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a44072870 con 0x7f4a440c3aa0 2024-03-28T19:08:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.675+0000 7f4a4a200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a44156640 con 0x7f4a44071cf0 2024-03-28T19:08:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.675+0000 7f4a4a200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a44158db0 con 0x7f4a4414f6d0 2024-03-28T19:08:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f7c32000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7c340c7970 0x7f7c34155730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45304/0 (socket says 172.21.15.27:45304) 2024-03-28T19:08:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f7c32000640 1 -- 172.21.15.27:0/2659585525 learned_addr learned my addr 172.21.15.27:0/2659585525 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f4a43e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a440c3aa0 0x7f4a440c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45292/0 (socket says 172.21.15.27:45292) 2024-03-28T19:08:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f4a43e00640 1 -- 172.21.15.27:0/1941171402 learned_addr learned my addr 172.21.15.27:0/1941171402 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f4a43400640 1 --2- 172.21.15.27:0/1941171402 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a4414f6d0 0x7f4a44159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f4a42a00640 1 --2- 172.21.15.27:0/1941171402 >> v2:172.21.15.155:3300/0 conn(0x7f4a44071cf0 0x7f4a440c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f7c33400640 1 --2- 172.21.15.27:0/2659585525 >> v2:172.21.15.155:3300/0 conn(0x7f7c340c7df0 0x7f7c34155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.676+0000 7f7c32a00640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c34152f80 0x7f7c340c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925946966 0 0) 0x7f7c340c7310 con 0x7f7c340c7970 2024-03-28T19:08:38.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c18003520 con 0x7f7c340c7970 2024-03-28T19:08:38.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f4a43400640 1 -- 172.21.15.27:0/1941171402 >> v2:172.21.15.155:3300/0 conn(0x7f4a44071cf0 msgr2=0x7f4a440c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:38.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f4a43400640 1 --2- 172.21.15.27:0/1941171402 >> v2:172.21.15.155:3300/0 conn(0x7f4a44071cf0 0x7f4a440c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f4a43400640 1 -- 172.21.15.27:0/1941171402 >> v1:172.21.15.27:6789/0 conn(0x7f4a440c3aa0 legacy=0x7f4a440c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:38.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f4a43400640 1 -- 172.21.15.27:0/1941171402 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a44156360 con 0x7f4a4414f6d0 2024-03-28T19:08:38.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f4a42000640 1 -- 172.21.15.27:0/1941171402 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970318708 0 0) 0x7f4a3c001490 con 0x7f4a440c3aa0 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.677+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539244204 0 0) 0x7f7c18003520 con 0x7f7c340c7970 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c34152f80 msgr2=0x7f7c340c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c34152f80 0x7f7c340c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 >> v2:172.21.15.155:3300/0 conn(0x7f7c340c7df0 msgr2=0x7f7c34155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 --2- 172.21.15.27:0/2659585525 >> v2:172.21.15.155:3300/0 conn(0x7f7c340c7df0 0x7f7c34155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c34159300 con 0x7f7c340c7970 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f7c20001670 con 0x7f7c340c7970 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 >> v1:172.21.15.27:6789/0 conn(0x7f7c340c7970 legacy=0x7f7c34155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c180039b0 0x7f7c18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c18004800 0x7f7c18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c180051c0 0x7f7c1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34158d40 con 0x7f7c180039b0 2024-03-28T19:08:38.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34159020 con 0x7f7c180051c0 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1800bf80 con 0x7f7c18004800 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.678+0000 7f7c32a00640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c180039b0 0x7f7c18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.679+0000 7f7c33400640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c180051c0 0x7f7c1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.679+0000 7f7c32000640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c18004800 0x7f7c18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.679+0000 7f7c32a00640 1 -- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c18004800 msgr2=0x7f7c18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.679+0000 7f7c32a00640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c18004800 0x7f7c18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.679+0000 7f7c32a00640 1 -- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c180051c0 msgr2=0x7f7c1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.679+0000 7f7c32a00640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c180051c0 0x7f7c1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.679+0000 7f7c32a00640 1 -- 172.21.15.27:0/2659585525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c18003520 con 0x7f7c180039b0 2024-03-28T19:08:38.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.679+0000 7f7c33400640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c180051c0 0x7f7c1800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:08:38.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.680+0000 7f4a43400640 1 --2- 172.21.15.27:0/1941171402 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a4414f6d0 0x7f4a44159850 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4a340097a0 tx=0x7f4a3402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a83c688d8b14484 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:38.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.680+0000 7f4a42000640 1 -- 172.21.15.27:0/1941171402 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34031780 con 0x7f4a4414f6d0 2024-03-28T19:08:38.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.681+0000 7f4a42000640 1 -- 172.21.15.27:0/1941171402 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a34037070 con 0x7f4a4414f6d0 2024-03-28T19:08:38.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.681+0000 7f4a42000640 1 -- 172.21.15.27:0/1941171402 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34031b40 con 0x7f4a4414f6d0 2024-03-28T19:08:38.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.681+0000 7f7c32a00640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c180039b0 0x7f7c18003dd0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f7c340c7940 tx=0x7f7c18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d7cfb0ebadaa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:38.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.681+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24039070 con 0x7f7c180039b0 2024-03-28T19:08:38.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.681+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2402c550 con 0x7f7c180039b0 2024-03-28T19:08:38.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.681+0000 7f7c31600640 1 -- 172.21.15.27:0/2659585525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24009630 con 0x7f7c180039b0 2024-03-28T19:08:38.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f4a4a200640 1 -- 172.21.15.27:0/1941171402 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a4414f6d0 msgr2=0x7f4a44159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:38.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f4a4a200640 1 --2- 172.21.15.27:0/1941171402 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a4414f6d0 0x7f4a44159850 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4a340097a0 tx=0x7f4a3402b820 comp rx=0 tx=0).stop 2024-03-28T19:08:38.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f4a4a200640 1 -- 172.21.15.27:0/1941171402 shutdown_connections 2024-03-28T19:08:38.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f4a4a200640 1 --2- 172.21.15.27:0/1941171402 >> v2:172.21.15.155:3300/0 conn(0x7f4a44071cf0 0x7f4a440c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f4a4a200640 1 --2- 172.21.15.27:0/1941171402 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a4414f6d0 0x7f4a44159850 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f4a4a200640 1 -- 172.21.15.27:0/1941171402 >> 172.21.15.27:0/1941171402 conn(0x7f4a4406c5f0 msgr2=0x7f4a4406ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:38.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f7c39c00640 1 -- 172.21.15.27:0/2659585525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c180039b0 msgr2=0x7f7c18003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:38.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f7c39c00640 1 --2- 172.21.15.27:0/2659585525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c180039b0 0x7f7c18003dd0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f7c340c7940 tx=0x7f7c18005830 comp rx=0 tx=0).stop 2024-03-28T19:08:38.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f7c33400640 1 -- 172.21.15.27:0/2659585525 reap_dead start 2024-03-28T19:08:38.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f4a4a200640 1 -- 172.21.15.27:0/1941171402 shutdown_connections 2024-03-28T19:08:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f7c39c00640 1 -- 172.21.15.27:0/2659585525 shutdown_connections 2024-03-28T19:08:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f7c39c00640 1 -- 172.21.15.27:0/2659585525 >> 172.21.15.27:0/2659585525 conn(0x7f7c3406b6f0 msgr2=0x7f7c340caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.682+0000 7f4a4a200640 1 -- 172.21.15.27:0/1941171402 wait complete. 2024-03-28T19:08:38.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.683+0000 7f7c39c00640 1 -- 172.21.15.27:0/2659585525 shutdown_connections 2024-03-28T19:08:38.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.683+0000 7f7c39c00640 1 -- 172.21.15.27:0/2659585525 wait complete. 2024-03-28T19:08:38.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.684+0000 7f4a4a200640 1 Processor -- start 2024-03-28T19:08:38.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.685+0000 7f4a4a200640 1 -- start start 2024-03-28T19:08:38.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.685+0000 7f7c39c00640 1 Processor -- start 2024-03-28T19:08:38.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f7c39c00640 1 -- start start 2024-03-28T19:08:38.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a4a200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a44071cf0 0x7f4a44155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a4a200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a440c3aa0 0x7f4a440ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a4a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a4414f6d0 0x7f4a440bf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a4a200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4414fab0 con 0x7f4a4414f6d0 2024-03-28T19:08:38.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a4a200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a44159330 con 0x7f4a44071cf0 2024-03-28T19:08:38.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a4a200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440c56b0 con 0x7f4a440c3aa0 2024-03-28T19:08:38.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a43e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a4414f6d0 0x7f4a440bf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a43e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a4414f6d0 0x7f4a440bf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49084/0 (socket says 172.21.15.27:49084) 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.686+0000 7f4a43e00640 1 -- 172.21.15.27:0/2861169407 learned_addr learned my addr 172.21.15.27:0/2861169407 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.687+0000 7f4a42a00640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a440c3aa0 0x7f4a440ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.687+0000 7f4a43400640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a44071cf0 0x7f4a44155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.687+0000 7f4a43e00640 1 -- 172.21.15.27:0/2861169407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a440c3aa0 msgr2=0x7f4a440ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.687+0000 7f4a43e00640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a440c3aa0 0x7f4a440ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.687+0000 7f4a43e00640 1 -- 172.21.15.27:0/2861169407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a44071cf0 msgr2=0x7f4a44155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.687+0000 7f4a43e00640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a44071cf0 0x7f4a44155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.687+0000 7f4a43e00640 1 -- 172.21.15.27:0/2861169407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440bda60 con 0x7f4a4414f6d0 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.687+0000 7f4a43e00640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a4414f6d0 0x7f4a440bf400 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c001730 tx=0x7f4a3c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:38.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c39c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c340c0410 0x7f7c340c07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c39c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c341545f0 0x7f7c34154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c39c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c341502d0 0x7f7c34150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c39c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c340cbca0 con 0x7f7c341545f0 2024-03-28T19:08:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c39c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34159b00 con 0x7f7c341502d0 2024-03-28T19:08:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c39c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34155470 con 0x7f7c340c0410 2024-03-28T19:08:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f4a3b400640 1 -- 172.21.15.27:0/2861169407 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c018070 con 0x7f4a4414f6d0 2024-03-28T19:08:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f4a3b400640 1 -- 172.21.15.27:0/2861169407 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c004d60 con 0x7f4a4414f6d0 2024-03-28T19:08:38.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f4a3b400640 1 -- 172.21.15.27:0/2861169407 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c00a860 con 0x7f4a4414f6d0 2024-03-28T19:08:38.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c32000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c341545f0 0x7f7c34154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a441efe00 con 0x7f4a4414f6d0 2024-03-28T19:08:38.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a441f00a0 con 0x7f4a4414f6d0 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c32000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c341545f0 0x7f7c34154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49100/0 (socket says 172.21.15.27:49100) 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c32000640 1 -- 172.21.15.27:0/549875295 learned_addr learned my addr 172.21.15.27:0/549875295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.688+0000 7f7c33400640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c341502d0 0x7f7c34150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.689+0000 7f7c32a00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c340c0410 0x7f7c340c07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.689+0000 7f7c33400640 1 -- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c340c0410 msgr2=0x7f7c340c07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.689+0000 7f7c33400640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c340c0410 0x7f7c340c07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.689+0000 7f7c33400640 1 -- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c341545f0 msgr2=0x7f7c34154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.689+0000 7f7c33400640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c341545f0 0x7f7c34154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.690+0000 7f7c33400640 1 -- 172.21.15.27:0/549875295 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c341eb300 con 0x7f7c341502d0 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.690+0000 7f7c33400640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c341502d0 0x7f7c34150750 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7c340c8800 tx=0x7f7c2c010590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:38.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.690+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4404f400 con 0x7f4a4414f6d0 2024-03-28T19:08:38.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.691+0000 7f7c2aa00640 1 -- 172.21.15.27:0/549875295 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c01a030 con 0x7f7c341502d0 2024-03-28T19:08:38.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.691+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c341eb5e0 con 0x7f7c341502d0 2024-03-28T19:08:38.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.691+0000 7f7c2aa00640 1 -- 172.21.15.27:0/549875295 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2c010da0 con 0x7f7c341502d0 2024-03-28T19:08:38.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.691+0000 7f7c2aa00640 1 -- 172.21.15.27:0/549875295 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c003bd0 con 0x7f7c341502d0 2024-03-28T19:08:38.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.691+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c341ebba0 con 0x7f7c341502d0 2024-03-28T19:08:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.692+0000 7f4a3b400640 1 -- 172.21.15.27:0/2861169407 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a3c01b070 con 0x7f4a4414f6d0 2024-03-28T19:08:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.693+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3404f400 con 0x7f7c341502d0 2024-03-28T19:08:38.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.693+0000 7f4a3b400640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a2406bc20 0x7f4a2406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.693+0000 7f4a3b400640 1 -- 172.21.15.27:0/2861169407 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7f4a3c08a9d0 con 0x7f4a4414f6d0 2024-03-28T19:08:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.694+0000 7f4a43400640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a2406bc20 0x7f4a2406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.695+0000 7f4a43400640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a2406bc20 0x7f4a2406e0e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4a34036000 tx=0x7f4a34008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.695+0000 7f7c2aa00640 1 -- 172.21.15.27:0/549875295 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7c2c005d10 con 0x7f7c341502d0 2024-03-28T19:08:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.696+0000 7f7c2aa00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c1406bc00 0x7f7c1406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.696+0000 7f7c2aa00640 1 -- 172.21.15.27:0/549875295 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 8099+0+0 (secure 0 0 0) 0x7f7c2c08c380 con 0x7f7c341502d0 2024-03-28T19:08:38.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.697+0000 7f7c32a00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c1406bc00 0x7f7c1406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:38.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.698+0000 7f7c32a00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c1406bc00 0x7f7c1406e0c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7c340c7940 tx=0x7f7c24030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:38.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.699+0000 7f4a3b400640 1 -- 172.21.15.27:0/2861169407 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a44072870 con 0x7f4a4414f6d0 2024-03-28T19:08:38.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:38.713+0000 7f7c2aa00640 1 -- 172.21.15.27:0/549875295 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7c2c059eb0 con 0x7f7c341502d0 2024-03-28T19:08:39.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.029+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a44158a60 con 0x7f4a4414f6d0 2024-03-28T19:08:39.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.029+0000 7f4a3b400640 1 -- 172.21.15.27:0/2861169407 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f4a3c058bd0 con 0x7f4a4414f6d0 2024-03-28T19:08:39.030 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:39.030 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6061,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 107 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.032+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a2406bc20 msgr2=0x7f4a2406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.032+0000 7f4a4a200640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a2406bc20 0x7f4a2406e0e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4a34036000 tx=0x7f4a34008030 comp rx=0 tx=0).stop 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.032+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a4414f6d0 msgr2=0x7f4a440bf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.032+0000 7f4a4a200640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a4414f6d0 0x7f4a440bf400 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c001730 tx=0x7f4a3c00cb70 comp rx=0 tx=0).stop 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.032+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 shutdown_connections 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.032+0000 7f4a4a200640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a2406bc20 0x7f4a2406e0e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.033+0000 7f4a4a200640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a4414f6d0 0x7f4a440bf400 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.033+0000 7f4a4a200640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a440c3aa0 0x7f4a440ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.033+0000 7f4a4a200640 1 --2- 172.21.15.27:0/2861169407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a44071cf0 0x7f4a44155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.033+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 >> 172.21.15.27:0/2861169407 conn(0x7f4a4406c5f0 msgr2=0x7f4a4406f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.033+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 shutdown_connections 2024-03-28T19:08:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.033+0000 7f4a4a200640 1 -- 172.21.15.27:0/2861169407 wait complete. 2024-03-28T19:08:39.055 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:08:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.064+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c340cba80 con 0x7f7c341502d0 2024-03-28T19:08:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.066+0000 7f7c2aa00640 1 -- 172.21.15.27:0/549875295 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f7c2c05df50 con 0x7f7c341502d0 2024-03-28T19:08:39.066 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:39.066 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6061,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:06:20.279933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5336":{"gid":5336,"name":"b","rank":0,"incarnation":87,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/3884653650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3884653650},{"type":"v1","addr":"172.21.15.155:6835","nonce":3884653650}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5336,"qdb_cluster":[5336]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 107 2024-03-28T19:08:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c1406bc00 msgr2=0x7f7c1406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c1406bc00 0x7f7c1406e0c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7c340c7940 tx=0x7f7c24030000 comp rx=0 tx=0).stop 2024-03-28T19:08:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c341502d0 msgr2=0x7f7c34150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c341502d0 0x7f7c34150750 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7c340c8800 tx=0x7f7c2c010590 comp rx=0 tx=0).stop 2024-03-28T19:08:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 shutdown_connections 2024-03-28T19:08:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7c1406bc00 0x7f7c1406e0c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c341502d0 0x7f7c34150750 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c341545f0 0x7f7c34154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 --2- 172.21.15.27:0/549875295 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c340c0410 0x7f7c340c07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 >> 172.21.15.27:0/549875295 conn(0x7f7c3406b6f0 msgr2=0x7f7c3414d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 shutdown_connections 2024-03-28T19:08:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:39.069+0000 7f7c39c00640 1 -- 172.21.15.27:0/549875295 wait complete. 2024-03-28T19:08:39.088 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-03-28T19:08:39.088 INFO:tasks.mds_thrash.fs.[b]:waiting for 52 secs before thrashing 2024-03-28T19:08:41.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.288+0000 7f84e7200640 1 Processor -- start 2024-03-28T19:08:41.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.288+0000 7f84e7200640 1 -- start start 2024-03-28T19:08:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.289+0000 7f84e7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c58c0 0x7f84e80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:41.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.289+0000 7f84e7200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84e80be5d0 0x7f84e8155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.289+0000 7f84e7200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e8156230 con 0x7f84e80c8ed0 2024-03-28T19:08:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.289+0000 7f84e7200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e8158bd0 con 0x7f84e80be5d0 2024-03-28T19:08:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.289+0000 7f84e7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8158eb0 con 0x7f84e80c58c0 2024-03-28T19:08:41.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.290+0000 7f84e5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c58c0 0x7f84e80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.290+0000 7f84e5400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f84e80c8ed0 0x7f84e81555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:52128/0 (socket says 172.21.15.27:52128) 2024-03-28T19:08:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.290+0000 7f84e5400640 1 -- 172.21.15.27:0/1105181034 learned_addr learned my addr 172.21.15.27:0/1105181034 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.290+0000 7f84e6800640 1 --2- 172.21.15.27:0/1105181034 >> v2:172.21.15.155:3300/0 conn(0x7f84e80be5d0 0x7f84e8155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:41.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.291+0000 7f84e5e00640 1 -- 172.21.15.27:0/1105181034 >> v2:172.21.15.155:3300/0 conn(0x7f84e80be5d0 msgr2=0x7f84e8155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.291+0000 7f84e5e00640 1 --2- 172.21.15.27:0/1105181034 >> v2:172.21.15.155:3300/0 conn(0x7f84e80be5d0 0x7f84e8155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.291+0000 7f84e5e00640 1 -- 172.21.15.27:0/1105181034 >> v1:172.21.15.155:6789/0 conn(0x7f84e80c8ed0 legacy=0x7f84e81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:41.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.291+0000 7f84e5e00640 1 -- 172.21.15.27:0/1105181034 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e8159190 con 0x7f84e80c58c0 2024-03-28T19:08:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.293+0000 7f84e5e00640 1 --2- 172.21.15.27:0/1105181034 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c58c0 0x7f84e80c5ca0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f84d80097a0 tx=0x7f84d802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f15cd9b84844228c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.293+0000 7f84e4a00640 1 -- 172.21.15.27:0/1105181034 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d80315a0 con 0x7f84e80c58c0 2024-03-28T19:08:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.293+0000 7f84e4a00640 1 -- 172.21.15.27:0/1105181034 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d8037070 con 0x7f84e80c58c0 2024-03-28T19:08:41.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.293+0000 7f84e4a00640 1 -- 172.21.15.27:0/1105181034 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8031960 con 0x7f84e80c58c0 2024-03-28T19:08:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.294+0000 7f84e7200640 1 -- 172.21.15.27:0/1105181034 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c58c0 msgr2=0x7f84e80c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.294+0000 7f84e7200640 1 --2- 172.21.15.27:0/1105181034 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c58c0 0x7f84e80c5ca0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f84d80097a0 tx=0x7f84d802b840 comp rx=0 tx=0).stop 2024-03-28T19:08:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.294+0000 7f84e7200640 1 -- 172.21.15.27:0/1105181034 shutdown_connections 2024-03-28T19:08:41.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.294+0000 7f84e7200640 1 --2- 172.21.15.27:0/1105181034 >> v2:172.21.15.155:3300/0 conn(0x7f84e80be5d0 0x7f84e8155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.294+0000 7f84e7200640 1 --2- 172.21.15.27:0/1105181034 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c58c0 0x7f84e80c5ca0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.294+0000 7f84e7200640 1 -- 172.21.15.27:0/1105181034 >> 172.21.15.27:0/1105181034 conn(0x7f84e806b6f0 msgr2=0x7f84e80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:41.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.294+0000 7f84e7200640 1 -- 172.21.15.27:0/1105181034 shutdown_connections 2024-03-28T19:08:41.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.294+0000 7f84e7200640 1 -- 172.21.15.27:0/1105181034 wait complete. 2024-03-28T19:08:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.296+0000 7f84e7200640 1 Processor -- start 2024-03-28T19:08:41.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.296+0000 7f84e7200640 1 -- start start 2024-03-28T19:08:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.297+0000 7f84e7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e80be5d0 0x7f84e8152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.297+0000 7f84e7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e80c58c0 0x7f84e80c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.297+0000 7f84e7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c8ed0 0x7f84e8151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.297+0000 7f84e7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80b0820 con 0x7f84e80be5d0 2024-03-28T19:08:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.297+0000 7f84e7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8159750 con 0x7f84e80c58c0 2024-03-28T19:08:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.297+0000 7f84e7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80b09a0 con 0x7f84e80c8ed0 2024-03-28T19:08:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.298+0000 7f84e5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e80be5d0 0x7f84e8152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:41.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.298+0000 7f84e5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e80be5d0 0x7f84e8152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49116/0 (socket says 172.21.15.27:49116) 2024-03-28T19:08:41.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.298+0000 7f84e5e00640 1 -- 172.21.15.27:0/3700880441 learned_addr learned my addr 172.21.15.27:0/3700880441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:41.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.298+0000 7f84e5400640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e80c58c0 0x7f84e80c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:41.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.298+0000 7f84e6800640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c8ed0 0x7f84e8151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:41.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.299+0000 7f84e5e00640 1 -- 172.21.15.27:0/3700880441 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c8ed0 msgr2=0x7f84e8151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:41.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.299+0000 7f84e5e00640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c8ed0 0x7f84e8151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.299+0000 7f84e5e00640 1 -- 172.21.15.27:0/3700880441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e80c58c0 msgr2=0x7f84e80c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:41.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.299+0000 7f84e5e00640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e80c58c0 0x7f84e80c2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.299+0000 7f84e5e00640 1 -- 172.21.15.27:0/3700880441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e80aaf70 con 0x7f84e80be5d0 2024-03-28T19:08:41.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.300+0000 7f84e5e00640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e80be5d0 0x7f84e8152580 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f84d80098c0 tx=0x7f84d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:41.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.300+0000 7f84de000640 1 -- 172.21.15.27:0/3700880441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8032630 con 0x7f84e80be5d0 2024-03-28T19:08:41.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.300+0000 7f84de000640 1 -- 172.21.15.27:0/3700880441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d80327d0 con 0x7f84e80be5d0 2024-03-28T19:08:41.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.300+0000 7f84de000640 1 -- 172.21.15.27:0/3700880441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8032bc0 con 0x7f84e80be5d0 2024-03-28T19:08:41.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.300+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e81ef750 con 0x7f84e80be5d0 2024-03-28T19:08:41.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.300+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e81efa30 con 0x7f84e80be5d0 2024-03-28T19:08:41.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.302+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e8069640 con 0x7f84e80be5d0 2024-03-28T19:08:41.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.304+0000 7f84de000640 1 -- 172.21.15.27:0/3700880441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f84d803b3f0 con 0x7f84e80be5d0 2024-03-28T19:08:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.305+0000 7f84de000640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c806bbd0 0x7f84c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.305+0000 7f84de000640 1 -- 172.21.15.27:0/3700880441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f84d802f030 con 0x7f84e80be5d0 2024-03-28T19:08:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.306+0000 7f84e5400640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c806bbd0 0x7f84c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.307+0000 7f84e5400640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c806bbd0 0x7f84c806e090 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f84e80aab70 tx=0x7f84d4007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.311+0000 7f84de000640 1 -- 172.21.15.27:0/3700880441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f84d8040070 con 0x7f84e80be5d0 2024-03-28T19:08:41.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.805+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e80b0ce0 con 0x7f84e80be5d0 2024-03-28T19:08:41.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.806+0000 7f84de000640 1 -- 172.21.15.27:0/3700880441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5462 (secure 0 0 0) 0x7f84d807ae00 con 0x7f84e80be5d0 2024-03-28T19:08:41.806 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:41.807 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:39.903321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:41.807 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 108 2024-03-28T19:08:41.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c806bbd0 msgr2=0x7f84c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c806bbd0 0x7f84c806e090 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f84e80aab70 tx=0x7f84d4007e80 comp rx=0 tx=0).stop 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e80be5d0 msgr2=0x7f84e8152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e80be5d0 0x7f84e8152580 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f84d80098c0 tx=0x7f84d80046c0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 shutdown_connections 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84c806bbd0 0x7f84c806e090 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84e80c8ed0 0x7f84e8151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84e80c58c0 0x7f84e80c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.809+0000 7f84e7200640 1 --2- 172.21.15.27:0/3700880441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84e80be5d0 0x7f84e8152580 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.810+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 >> 172.21.15.27:0/3700880441 conn(0x7f84e806b6f0 msgr2=0x7f84e81569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:41.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.810+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 shutdown_connections 2024-03-28T19:08:41.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:41.810+0000 7f84e7200640 1 -- 172.21.15.27:0/3700880441 wait complete. 2024-03-28T19:08:41.831 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-03-28T19:08:41.832 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:08:41.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:42.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.096+0000 7f2fd9800640 1 Processor -- start 2024-03-28T19:08:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.096+0000 7f2fd9800640 1 -- start start 2024-03-28T19:08:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.098+0000 7f2fd9800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2fd4153ed0 0x7f2fd406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.098+0000 7f2fd9800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fd406e6f0 0x7f2fd406eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.098+0000 7f2fd9800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40c3830 con 0x7f2fd4159bc0 2024-03-28T19:08:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.098+0000 7f2fd9800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd40c6990 con 0x7f2fd406e6f0 2024-03-28T19:08:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.098+0000 7f2fd9800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd406d1f0 con 0x7f2fd4153ed0 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.099+0000 7f2fd2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2fd406e6f0 0x7f2fd406eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.099+0000 7f2fd3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2fd4159bc0 0x7f2fd4159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45308/0 (socket says 172.21.15.27:45308) 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.099+0000 7f2fd3400640 1 -- 172.21.15.27:0/2043680665 learned_addr learned my addr 172.21.15.27:0/2043680665 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.099+0000 7f2fd2a00640 1 --2- 172.21.15.27:0/2043680665 >> v2:172.21.15.155:3300/0 conn(0x7f2fd4153ed0 0x7f2fd406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.100+0000 7f2fd2000640 1 -- 172.21.15.27:0/2043680665 >> v2:172.21.15.155:3300/0 conn(0x7f2fd4153ed0 msgr2=0x7f2fd406ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.100+0000 7f2fd2000640 1 --2- 172.21.15.27:0/2043680665 >> v2:172.21.15.155:3300/0 conn(0x7f2fd4153ed0 0x7f2fd406ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.100+0000 7f2fd2000640 1 -- 172.21.15.27:0/2043680665 >> v1:172.21.15.27:6789/0 conn(0x7f2fd4159bc0 legacy=0x7f2fd4159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.100+0000 7f2fd2000640 1 -- 172.21.15.27:0/2043680665 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40c66b0 con 0x7f2fd406e6f0 2024-03-28T19:08:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.100+0000 7f2fd1600640 1 -- 172.21.15.27:0/2043680665 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397511443 0 0) 0x7f2fd40c3830 con 0x7f2fd4159bc0 2024-03-28T19:08:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.102+0000 7f2fd2000640 1 --2- 172.21.15.27:0/2043680665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd406e6f0 0x7f2fd406eb80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40c5ca0 tx=0x7f2fc002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da3b310d5f5e328 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:42.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.102+0000 7f2fd1600640 1 -- 172.21.15.27:0/2043680665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0035020 con 0x7f2fd406e6f0 2024-03-28T19:08:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.102+0000 7f2fd1600640 1 -- 172.21.15.27:0/2043680665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc0031830 con 0x7f2fd406e6f0 2024-03-28T19:08:42.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.102+0000 7f2fd1600640 1 -- 172.21.15.27:0/2043680665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0031c00 con 0x7f2fd406e6f0 2024-03-28T19:08:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.104+0000 7f2fd9800640 1 -- 172.21.15.27:0/2043680665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd406e6f0 msgr2=0x7f2fd406eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.104+0000 7f2fd9800640 1 --2- 172.21.15.27:0/2043680665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd406e6f0 0x7f2fd406eb80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40c5ca0 tx=0x7f2fc002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.104+0000 7f2fd9800640 1 -- 172.21.15.27:0/2043680665 shutdown_connections 2024-03-28T19:08:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.104+0000 7f2fd9800640 1 --2- 172.21.15.27:0/2043680665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd406e6f0 0x7f2fd406eb80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.104+0000 7f2fd9800640 1 --2- 172.21.15.27:0/2043680665 >> v2:172.21.15.155:3300/0 conn(0x7f2fd4153ed0 0x7f2fd406ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.104+0000 7f2fd9800640 1 -- 172.21.15.27:0/2043680665 >> 172.21.15.27:0/2043680665 conn(0x7f2fd406b9d0 msgr2=0x7f2fd40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.104+0000 7f2fd9800640 1 -- 172.21.15.27:0/2043680665 shutdown_connections 2024-03-28T19:08:42.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.104+0000 7f2fd9800640 1 -- 172.21.15.27:0/2043680665 wait complete. 2024-03-28T19:08:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.107+0000 7f2fd9800640 1 Processor -- start 2024-03-28T19:08:42.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.107+0000 7f2fd9800640 1 -- start start 2024-03-28T19:08:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.108+0000 7f2fd9800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fd406e6f0 0x7f2fd41e6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.108+0000 7f2fd9800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2fd4153ed0 0x7f2fd4152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.108+0000 7f2fd9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd4159bc0 0x7f2fd4152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.108+0000 7f2fd9800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40c6380 con 0x7f2fd4159bc0 2024-03-28T19:08:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.108+0000 7f2fd9800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40c3db0 con 0x7f2fd406e6f0 2024-03-28T19:08:42.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.108+0000 7f2fd9800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40c1160 con 0x7f2fd4153ed0 2024-03-28T19:08:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.109+0000 7f2fd3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd4159bc0 0x7f2fd4152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.109+0000 7f2fd3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd4159bc0 0x7f2fd4152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49130/0 (socket says 172.21.15.27:49130) 2024-03-28T19:08:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.109+0000 7f2fd3400640 1 -- 172.21.15.27:0/3833326060 learned_addr learned my addr 172.21.15.27:0/3833326060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.109+0000 7f2fd2a00640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fd406e6f0 0x7f2fd41e6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.109+0000 7f2fd2000640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2fd4153ed0 0x7f2fd4152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:42.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.110+0000 7f2fd3400640 1 -- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2fd4153ed0 msgr2=0x7f2fd4152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:42.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.110+0000 7f2fd3400640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2fd4153ed0 0x7f2fd4152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.110+0000 7f2fd3400640 1 -- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fd406e6f0 msgr2=0x7f2fd41e6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:42.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.110+0000 7f2fd3400640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fd406e6f0 0x7f2fd41e6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.110+0000 7f2fd3400640 1 -- 172.21.15.27:0/3833326060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40abdd0 con 0x7f2fd4159bc0 2024-03-28T19:08:42.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.110+0000 7f2fd2a00640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fd406e6f0 0x7f2fd41e6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:08:42.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.110+0000 7f2fd3400640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd4159bc0 0x7f2fd4152f60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00c7b0 tx=0x7f2fcc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:42.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.111+0000 7f2fcaa00640 1 -- 172.21.15.27:0/3833326060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc018070 con 0x7f2fd4159bc0 2024-03-28T19:08:42.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.111+0000 7f2fcaa00640 1 -- 172.21.15.27:0/3833326060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc00a640 con 0x7f2fd4159bc0 2024-03-28T19:08:42.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.111+0000 7f2fcaa00640 1 -- 172.21.15.27:0/3833326060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc00aa30 con 0x7f2fd4159bc0 2024-03-28T19:08:42.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.111+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd4154620 con 0x7f2fd4159bc0 2024-03-28T19:08:42.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.111+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd40aba80 con 0x7f2fd4159bc0 2024-03-28T19:08:42.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.112+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd404f400 con 0x7f2fd4159bc0 2024-03-28T19:08:42.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.115+0000 7f2fcaa00640 1 -- 172.21.15.27:0/3833326060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2fcc00abd0 con 0x7f2fd4159bc0 2024-03-28T19:08:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.116+0000 7f2fcaa00640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2fb406bbd0 0x7f2fb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:42.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.116+0000 7f2fcaa00640 1 -- 172.21.15.27:0/3833326060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f2fcc08aa50 con 0x7f2fd4159bc0 2024-03-28T19:08:42.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.117+0000 7f2fd2a00640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2fb406bbd0 0x7f2fb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:42.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.118+0000 7f2fd2a00640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2fb406bbd0 0x7f2fb406e090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4006fd0 tx=0x7f2fc4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:42.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.122+0000 7f2fcaa00640 1 -- 172.21.15.27:0/3833326060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2fcc058b70 con 0x7f2fd4159bc0 2024-03-28T19:08:42.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.487+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fd40ac190 con 0x7f2fd4159bc0 2024-03-28T19:08:42.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.487+0000 7f2fcaa00640 1 -- 172.21.15.27:0/3833326060 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5462 (secure 0 0 0) 0x7f2fcc058470 con 0x7f2fd4159bc0 2024-03-28T19:08:42.488 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:42.488 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:39.903321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 108 2024-03-28T19:08:42.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2fb406bbd0 msgr2=0x7f2fb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:42.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2fb406bbd0 0x7f2fb406e090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4006fd0 tx=0x7f2fc4008040 comp rx=0 tx=0).stop 2024-03-28T19:08:42.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd4159bc0 msgr2=0x7f2fd4152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:42.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd4159bc0 0x7f2fd4152f60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00c7b0 tx=0x7f2fcc00cc70 comp rx=0 tx=0).stop 2024-03-28T19:08:42.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 shutdown_connections 2024-03-28T19:08:42.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2fb406bbd0 0x7f2fb406e090 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fd4159bc0 0x7f2fd4152f60 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2fd4153ed0 0x7f2fd4152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 --2- 172.21.15.27:0/3833326060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fd406e6f0 0x7f2fd41e6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:42.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 >> 172.21.15.27:0/3833326060 conn(0x7f2fd406b9d0 msgr2=0x7f2fd4070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:42.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 shutdown_connections 2024-03-28T19:08:42.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:42.491+0000 7f2fd9800640 1 -- 172.21.15.27:0/3833326060 wait complete. 2024-03-28T19:08:42.513 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6061": { "addr": "172.21.15.27:6837/3585028907", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 3585028907, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 3585028907, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6061, "incarnation": 108, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:08:39.903321+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6061 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 106, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6501": { "addr": "172.21.15.27:6836/1398280334", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1398280334, "type": "v2" }, { "addr": "172.21.15.27:6836", "nonce": 1398280334, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6501, "incarnation": 103, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:08:30.359626+0000", "qdb_cluster": [ 6501 ], "qdb_leader": 6501, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6501 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:08:44.514 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:44.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.736+0000 7f2aebe00640 1 Processor -- start 2024-03-28T19:08:44.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.736+0000 7f2aebe00640 1 -- start start 2024-03-28T19:08:44.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.738+0000 7f2aebe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2aec150ae0 0x7f2aec0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.738+0000 7f2aebe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2aec153a30 0x7f2aec153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.738+0000 7f2aebe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec150ec0 con 0x7f2aec0a61a0 2024-03-28T19:08:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.738+0000 7f2aebe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aec1543c0 con 0x7f2aec153a30 2024-03-28T19:08:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.738+0000 7f2aebe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aec06dc70 con 0x7f2aec150ae0 2024-03-28T19:08:44.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2aeb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2aec153a30 0x7f2aec153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:44.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2ae3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2aec0a61a0 0x7f2aec0c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45318/0 (socket says 172.21.15.27:45318) 2024-03-28T19:08:44.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2ae3e00640 1 -- 172.21.15.27:0/3304328797 learned_addr learned my addr 172.21.15.27:0/3304328797 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:44.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2aeaa00640 1 --2- 172.21.15.27:0/3304328797 >> v2:172.21.15.155:3300/0 conn(0x7f2aec150ae0 0x7f2aec0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:44.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2aeb400640 1 -- 172.21.15.27:0/3304328797 >> v2:172.21.15.155:3300/0 conn(0x7f2aec150ae0 msgr2=0x7f2aec0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:44.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2aeb400640 1 --2- 172.21.15.27:0/3304328797 >> v2:172.21.15.155:3300/0 conn(0x7f2aec150ae0 0x7f2aec0a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:44.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2aeb400640 1 -- 172.21.15.27:0/3304328797 >> v1:172.21.15.27:6789/0 conn(0x7f2aec0a61a0 legacy=0x7f2aec0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:44.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2aeb400640 1 -- 172.21.15.27:0/3304328797 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec153550 con 0x7f2aec153a30 2024-03-28T19:08:44.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.739+0000 7f2aea000640 1 -- 172.21.15.27:0/3304328797 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106192350 0 0) 0x7f2ad4001110 con 0x7f2aec0a61a0 2024-03-28T19:08:44.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.740+0000 7f2aeaa00640 1 --2- 172.21.15.27:0/3304328797 >> v2:172.21.15.155:3300/0 conn(0x7f2aec150ae0 0x7f2aec0a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:08:44.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.741+0000 7f2aeb400640 1 --2- 172.21.15.27:0/3304328797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec153e80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0a9760 tx=0x7f2ae402be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5df05c2f27406d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.741+0000 7f2aea000640 1 -- 172.21.15.27:0/3304328797 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4039070 con 0x7f2aec153a30 2024-03-28T19:08:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.741+0000 7f2aea000640 1 -- 172.21.15.27:0/3304328797 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae402cbe0 con 0x7f2aec153a30 2024-03-28T19:08:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.741+0000 7f2aea000640 1 -- 172.21.15.27:0/3304328797 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4034440 con 0x7f2aec153a30 2024-03-28T19:08:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.742+0000 7f2aebe00640 1 -- 172.21.15.27:0/3304328797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 msgr2=0x7f2aec153e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.742+0000 7f2aebe00640 1 --2- 172.21.15.27:0/3304328797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec153e80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0a9760 tx=0x7f2ae402be30 comp rx=0 tx=0).stop 2024-03-28T19:08:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.743+0000 7f2aebe00640 1 -- 172.21.15.27:0/3304328797 shutdown_connections 2024-03-28T19:08:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.743+0000 7f2aebe00640 1 --2- 172.21.15.27:0/3304328797 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec153e80 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.743+0000 7f2aebe00640 1 --2- 172.21.15.27:0/3304328797 >> v2:172.21.15.155:3300/0 conn(0x7f2aec150ae0 0x7f2aec0a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.743+0000 7f2aebe00640 1 -- 172.21.15.27:0/3304328797 >> 172.21.15.27:0/3304328797 conn(0x7f2aec06b0c0 msgr2=0x7f2aec06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.743+0000 7f2aebe00640 1 -- 172.21.15.27:0/3304328797 shutdown_connections 2024-03-28T19:08:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.743+0000 7f2aebe00640 1 -- 172.21.15.27:0/3304328797 wait complete. 2024-03-28T19:08:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.745+0000 7f2aebe00640 1 Processor -- start 2024-03-28T19:08:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.745+0000 7f2aebe00640 1 -- start start 2024-03-28T19:08:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2aec0a61a0 0x7f2aec0bfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2aec150ae0 0x7f2aec1eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aebe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec06e4d0 con 0x7f2aec153a30 2024-03-28T19:08:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aebe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec06e1f0 con 0x7f2aec150ae0 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aebe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec06e7b0 con 0x7f2aec0a61a0 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aeb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aeb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec149150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49154/0 (socket says 172.21.15.27:49154) 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.746+0000 7f2aeb400640 1 -- 172.21.15.27:0/2209106942 learned_addr learned my addr 172.21.15.27:0/2209106942 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.747+0000 7f2aeaa00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2aec0a61a0 0x7f2aec0bfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.747+0000 7f2ae3e00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2aec150ae0 0x7f2aec1eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.747+0000 7f2aeb400640 1 -- 172.21.15.27:0/2209106942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2aec0a61a0 msgr2=0x7f2aec0bfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.747+0000 7f2aeb400640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2aec0a61a0 0x7f2aec0bfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.747+0000 7f2aeb400640 1 -- 172.21.15.27:0/2209106942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2aec150ae0 msgr2=0x7f2aec1eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.747+0000 7f2aeb400640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2aec150ae0 0x7f2aec1eb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.747+0000 7f2aeb400640 1 -- 172.21.15.27:0/2209106942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec152dc0 con 0x7f2aec153a30 2024-03-28T19:08:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.748+0000 7f2aeb400640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec149150 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009e80 tx=0x7f2ae4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.748+0000 7f2ae2a00640 1 -- 172.21.15.27:0/2209106942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4039050 con 0x7f2aec153a30 2024-03-28T19:08:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.748+0000 7f2ae2a00640 1 -- 172.21.15.27:0/2209106942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae4045070 con 0x7f2aec153a30 2024-03-28T19:08:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.748+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec1530a0 con 0x7f2aec153a30 2024-03-28T19:08:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.748+0000 7f2ae2a00640 1 -- 172.21.15.27:0/2209106942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae403e830 con 0x7f2aec153a30 2024-03-28T19:08:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.749+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec141db0 con 0x7f2aec153a30 2024-03-28T19:08:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.750+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aec067dd0 con 0x7f2aec153a30 2024-03-28T19:08:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.752+0000 7f2ae2a00640 1 -- 172.21.15.27:0/2209106942 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2ae403e9d0 con 0x7f2aec153a30 2024-03-28T19:08:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.753+0000 7f2ae2a00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ac806bc20 0x7f2ac806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.754+0000 7f2ae2a00640 1 -- 172.21.15.27:0/2209106942 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f2ae40ae190 con 0x7f2aec153a30 2024-03-28T19:08:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.754+0000 7f2aeaa00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ac806bc20 0x7f2ac806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.755+0000 7f2aeaa00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ac806bc20 0x7f2ac806e0e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2adc006fd0 tx=0x7f2adc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:44.762+0000 7f2ae2a00640 1 -- 172.21.15.27:0/2209106942 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2ae407c450 con 0x7f2aec153a30 2024-03-28T19:08:45.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.062+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aec1eaec0 con 0x7f2aec153a30 2024-03-28T19:08:45.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.063+0000 7f2ae2a00640 1 -- 172.21.15.27:0/2209106942 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5462 (secure 0 0 0) 0x7f2ae407bd50 con 0x7f2aec153a30 2024-03-28T19:08:45.062 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:45.063 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:43.454954+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:45.063 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 110 2024-03-28T19:08:45.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.066+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ac806bc20 msgr2=0x7f2ac806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:45.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.066+0000 7f2aebe00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ac806bc20 0x7f2ac806e0e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2adc006fd0 tx=0x7f2adc008040 comp rx=0 tx=0).stop 2024-03-28T19:08:45.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.066+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 msgr2=0x7f2aec149150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:45.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.066+0000 7f2aebe00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec149150 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009e80 tx=0x7f2ae4004a40 comp rx=0 tx=0).stop 2024-03-28T19:08:45.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.067+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 shutdown_connections 2024-03-28T19:08:45.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.067+0000 7f2aebe00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2ac806bc20 0x7f2ac806e0e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:45.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.067+0000 7f2aebe00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2aec153a30 0x7f2aec149150 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:45.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.067+0000 7f2aebe00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2aec150ae0 0x7f2aec1eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:45.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.067+0000 7f2aebe00640 1 --2- 172.21.15.27:0/2209106942 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2aec0a61a0 0x7f2aec0bfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:45.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.067+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 >> 172.21.15.27:0/2209106942 conn(0x7f2aec06b0c0 msgr2=0x7f2aec0c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:45.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.067+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 shutdown_connections 2024-03-28T19:08:45.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:45.067+0000 7f2aebe00640 1 -- 172.21.15.27:0/2209106942 wait complete. 2024-03-28T19:08:45.086 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:08:46.419 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:08:46.420+0000 7f2a9d800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:08:47.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:47.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.350+0000 7fe1dbe00640 1 Processor -- start 2024-03-28T19:08:47.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.350+0000 7fe1dbe00640 1 -- start start 2024-03-28T19:08:47.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1dbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1dc0c8ed0 0x7fe1dc0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:47.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1dbe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe1dc0be5d0 0x7fe1dc155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:47.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1dbe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc156230 con 0x7fe1dc0c58c0 2024-03-28T19:08:47.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1dbe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc158bd0 con 0x7fe1dc0c8ed0 2024-03-28T19:08:47.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1dbe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc158eb0 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1da000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1dc0c8ed0 0x7fe1dc0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:47.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1da000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1dc0c8ed0 0x7fe1dc0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57122/0 (socket says 172.21.15.27:57122) 2024-03-28T19:08:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1da000640 1 -- 172.21.15.27:0/2840459836 learned_addr learned my addr 172.21.15.27:0/2840459836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.352+0000 7fe1daa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe1dc0c58c0 0x7fe1dc1555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36682/0 (socket says 172.21.15.27:36682) 2024-03-28T19:08:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.353+0000 7fe1db400640 1 --2- 172.21.15.27:0/2840459836 >> v2:172.21.15.155:3300/0 conn(0x7fe1dc0be5d0 0x7fe1dc155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.353+0000 7fe1da000640 1 -- 172.21.15.27:0/2840459836 >> v2:172.21.15.155:3300/0 conn(0x7fe1dc0be5d0 msgr2=0x7fe1dc155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:47.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.353+0000 7fe1da000640 1 --2- 172.21.15.27:0/2840459836 >> v2:172.21.15.155:3300/0 conn(0x7fe1dc0be5d0 0x7fe1dc155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.353+0000 7fe1da000640 1 -- 172.21.15.27:0/2840459836 >> v1:172.21.15.27:6789/0 conn(0x7fe1dc0c58c0 legacy=0x7fe1dc1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:47.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.353+0000 7fe1da000640 1 -- 172.21.15.27:0/2840459836 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc159190 con 0x7fe1dc0c8ed0 2024-03-28T19:08:47.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.353+0000 7fe1db400640 1 --2- 172.21.15.27:0/2840459836 >> v2:172.21.15.155:3300/0 conn(0x7fe1dc0be5d0 0x7fe1dc155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:08:47.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.353+0000 7fe1d9600640 1 -- 172.21.15.27:0/2840459836 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120362705 0 0) 0x7fe1dc156230 con 0x7fe1dc0c58c0 2024-03-28T19:08:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.355+0000 7fe1da000640 1 --2- 172.21.15.27:0/2840459836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0c8ed0 0x7fe1dc0c5ca0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc08d050 tx=0x7fe1c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf539fa738e2006d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.355+0000 7fe1d9600640 1 -- 172.21.15.27:0/2840459836 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c802e070 con 0x7fe1dc0c8ed0 2024-03-28T19:08:47.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.355+0000 7fe1d9600640 1 -- 172.21.15.27:0/2840459836 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c8037070 con 0x7fe1dc0c8ed0 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.355+0000 7fe1d9600640 1 -- 172.21.15.27:0/2840459836 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8031a00 con 0x7fe1dc0c8ed0 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.356+0000 7fe1dbe00640 1 -- 172.21.15.27:0/2840459836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0c8ed0 msgr2=0x7fe1dc0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.356+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/2840459836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0c8ed0 0x7fe1dc0c5ca0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc08d050 tx=0x7fe1c802b880 comp rx=0 tx=0).stop 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.356+0000 7fe1dbe00640 1 -- 172.21.15.27:0/2840459836 shutdown_connections 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.356+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/2840459836 >> v2:172.21.15.155:3300/0 conn(0x7fe1dc0be5d0 0x7fe1dc155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.356+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/2840459836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0c8ed0 0x7fe1dc0c5ca0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.356+0000 7fe1dbe00640 1 -- 172.21.15.27:0/2840459836 >> 172.21.15.27:0/2840459836 conn(0x7fe1dc06b6f0 msgr2=0x7fe1dc0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.356+0000 7fe1dbe00640 1 -- 172.21.15.27:0/2840459836 shutdown_connections 2024-03-28T19:08:47.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.356+0000 7fe1dbe00640 1 -- 172.21.15.27:0/2840459836 wait complete. 2024-03-28T19:08:47.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.359+0000 7fe1dbe00640 1 Processor -- start 2024-03-28T19:08:47.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.359+0000 7fe1dbe00640 1 -- start start 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1dbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0be5d0 0x7fe1dc1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1dbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc0c58c0 0x7fe1dc0bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1dbe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1dc0c8ed0 0x7fe1dc152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1dbe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b0820 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1dbe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc159750 con 0x7fe1dc0c58c0 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1dbe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b09a0 con 0x7fe1dc0c8ed0 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1daa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0be5d0 0x7fe1dc1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1daa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0be5d0 0x7fe1dc1579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57134/0 (socket says 172.21.15.27:57134) 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.360+0000 7fe1daa00640 1 -- 172.21.15.27:0/491239921 learned_addr learned my addr 172.21.15.27:0/491239921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:47.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.361+0000 7fe1da000640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc0c58c0 0x7fe1dc0bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.361+0000 7fe1db400640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1dc0c8ed0 0x7fe1dc152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.361+0000 7fe1daa00640 1 -- 172.21.15.27:0/491239921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1dc0c8ed0 msgr2=0x7fe1dc152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.361+0000 7fe1daa00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1dc0c8ed0 0x7fe1dc152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.361+0000 7fe1daa00640 1 -- 172.21.15.27:0/491239921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc0c58c0 msgr2=0x7fe1dc0bffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.361+0000 7fe1daa00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc0c58c0 0x7fe1dc0bffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.361+0000 7fe1daa00640 1 -- 172.21.15.27:0/491239921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc1e1fe0 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.361+0000 7fe1daa00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0be5d0 0x7fe1dc1579a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00c460 tx=0x7fe1cc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.362+0000 7fe1d2a00640 1 -- 172.21.15.27:0/491239921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc018070 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.362+0000 7fe1d2a00640 1 -- 172.21.15.27:0/491239921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc007500 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.362+0000 7fe1d2a00640 1 -- 172.21.15.27:0/491239921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc004a20 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.362+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc1e22c0 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.362+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc1e2880 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.364+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc069640 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.365+0000 7fe1d2a00640 1 -- 172.21.15.27:0/491239921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe1cc01b070 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.367+0000 7fe1d2a00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe1bc06bc50 0x7fe1bc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.367+0000 7fe1d2a00640 1 -- 172.21.15.27:0/491239921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7fe1cc08a6f0 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.367+0000 7fe1da000640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe1bc06bc50 0x7fe1bc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.368+0000 7fe1da000640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe1bc06bc50 0x7fe1bc06e110 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8036000 tx=0x7fe1c8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:47.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.372+0000 7fe1d2a00640 1 -- 172.21.15.27:0/491239921 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe1cc0588f0 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.707+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1dc156580 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.708+0000 7fe1d2a00640 1 -- 172.21.15.27:0/491239921 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5469 (secure 0 0 0) 0x7fe1cc0581f0 con 0x7fe1dc0be5d0 2024-03-28T19:08:47.707 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:47.708 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:47.708 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 111 2024-03-28T19:08:47.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe1bc06bc50 msgr2=0x7fe1bc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:47.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe1bc06bc50 0x7fe1bc06e110 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8036000 tx=0x7fe1c8008030 comp rx=0 tx=0).stop 2024-03-28T19:08:47.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0be5d0 msgr2=0x7fe1dc1579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:47.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0be5d0 0x7fe1dc1579a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00c460 tx=0x7fe1cc00c920 comp rx=0 tx=0).stop 2024-03-28T19:08:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 shutdown_connections 2024-03-28T19:08:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe1bc06bc50 0x7fe1bc06e110 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1dc0c8ed0 0x7fe1dc152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1dc0c58c0 0x7fe1dc0bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 --2- 172.21.15.27:0/491239921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1dc0be5d0 0x7fe1dc1579a0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 >> 172.21.15.27:0/491239921 conn(0x7fe1dc06b6f0 msgr2=0x7fe1dc0ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 shutdown_connections 2024-03-28T19:08:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:47.711+0000 7fe1dbe00640 1 -- 172.21.15.27:0/491239921 wait complete. 2024-03-28T19:08:47.735 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5336 2024-03-28T19:08:47.735 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before reviving mds.b 2024-03-28T19:08:56.736 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-03-28T19:08:56.736 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:08:56.736 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:08:56.746 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:08:56.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:56.928 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:08:56.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.979+0000 7fe743e00640 1 Processor -- start 2024-03-28T19:08:56.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.980+0000 7fe743e00640 1 -- start start 2024-03-28T19:08:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.981+0000 7fe743e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe7440c59d0 0x7fe7440c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.981+0000 7fe743e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7440727e0 0x7fe744159520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.981+0000 7fe743e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe744155fe0 con 0x7fe744155600 2024-03-28T19:08:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.981+0000 7fe743e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7441565a0 con 0x7fe7440727e0 2024-03-28T19:08:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.981+0000 7fe743e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe744159090 con 0x7fe7440c59d0 2024-03-28T19:08:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.981+0000 7fe742000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7440727e0 0x7fe744159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.982+0000 7fe743400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe744155600 0x7fe74414f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:58526/0 (socket says 172.21.15.27:58526) 2024-03-28T19:08:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.982+0000 7fe743400640 1 -- 172.21.15.27:0/236875619 learned_addr learned my addr 172.21.15.27:0/236875619 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.981+0000 7fe742a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe7440c59d0 0x7fe7440c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.982+0000 7fe742000640 1 -- 172.21.15.27:0/236875619 >> v2:172.21.15.155:3300/0 conn(0x7fe7440c59d0 msgr2=0x7fe7440c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:08:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.982+0000 7fe742000640 1 --2- 172.21.15.27:0/236875619 >> v2:172.21.15.155:3300/0 conn(0x7fe7440c59d0 0x7fe7440c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.982+0000 7fe742000640 1 -- 172.21.15.27:0/236875619 >> v1:172.21.15.155:6789/0 conn(0x7fe744155600 legacy=0x7fe74414f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.982+0000 7fe742000640 1 -- 172.21.15.27:0/236875619 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7441562c0 con 0x7fe7440727e0 2024-03-28T19:08:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.984+0000 7fe742000640 1 --2- 172.21.15.27:0/236875619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744159520 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe7440aa5c0 tx=0x7fe73002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c78d44a8fbec99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.985+0000 7fe741600640 1 -- 172.21.15.27:0/236875619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730038070 con 0x7fe7440727e0 2024-03-28T19:08:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.985+0000 7fe741600640 1 -- 172.21.15.27:0/236875619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe730031710 con 0x7fe7440727e0 2024-03-28T19:08:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.985+0000 7fe741600640 1 -- 172.21.15.27:0/236875619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730031ae0 con 0x7fe7440727e0 2024-03-28T19:08:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.985+0000 7fe743e00640 1 -- 172.21.15.27:0/236875619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 msgr2=0x7fe744159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.985+0000 7fe743e00640 1 --2- 172.21.15.27:0/236875619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744159520 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe7440aa5c0 tx=0x7fe73002b7b0 comp rx=0 tx=0).stop 2024-03-28T19:08:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.986+0000 7fe743e00640 1 -- 172.21.15.27:0/236875619 shutdown_connections 2024-03-28T19:08:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.986+0000 7fe743e00640 1 --2- 172.21.15.27:0/236875619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744159520 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.986+0000 7fe743e00640 1 --2- 172.21.15.27:0/236875619 >> v2:172.21.15.155:3300/0 conn(0x7fe7440c59d0 0x7fe7440c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.986+0000 7fe743e00640 1 -- 172.21.15.27:0/236875619 >> 172.21.15.27:0/236875619 conn(0x7fe7440ac910 msgr2=0x7fe7440accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.986+0000 7fe743e00640 1 -- 172.21.15.27:0/236875619 shutdown_connections 2024-03-28T19:08:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.986+0000 7fe743e00640 1 -- 172.21.15.27:0/236875619 wait complete. 2024-03-28T19:08:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.989+0000 7fe743e00640 1 Processor -- start 2024-03-28T19:08:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.989+0000 7fe743e00640 1 -- start start 2024-03-28T19:08:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe743e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe743e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7440c59d0 0x7fe7441f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe743e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe744155600 0x7fe74414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe743e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74414f970 con 0x7fe7440727e0 2024-03-28T19:08:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe743e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe744159a60 con 0x7fe7440c59d0 2024-03-28T19:08:56.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe743e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74414faf0 con 0x7fe744155600 2024-03-28T19:08:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe742a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe742a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59414/0 (socket says 172.21.15.27:59414) 2024-03-28T19:08:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe742a00640 1 -- 172.21.15.27:0/3466457455 learned_addr learned my addr 172.21.15.27:0/3466457455 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe742000640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7440c59d0 0x7fe7441f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.990+0000 7fe743400640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe744155600 0x7fe74414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.991+0000 7fe742a00640 1 -- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe744155600 msgr2=0x7fe74414c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.991+0000 7fe742a00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe744155600 0x7fe74414c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.991+0000 7fe742a00640 1 -- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7440c59d0 msgr2=0x7fe7441f0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.991+0000 7fe742a00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7440c59d0 0x7fe7441f0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.991+0000 7fe742a00640 1 -- 172.21.15.27:0/3466457455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7441f2cc0 con 0x7fe7440727e0 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.991+0000 7fe742000640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7440c59d0 0x7fe7441f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.991+0000 7fe742a00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744153990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe73400e9e0 tx=0x7fe73400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.992+0000 7fe73aa00640 1 -- 172.21.15.27:0/3466457455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734017070 con 0x7fe7440727e0 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.992+0000 7fe73aa00640 1 -- 172.21.15.27:0/3466457455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73400c9d0 con 0x7fe7440727e0 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.992+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7441ef840 con 0x7fe7440727e0 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.992+0000 7fe73aa00640 1 -- 172.21.15.27:0/3466457455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73400cdc0 con 0x7fe7440727e0 2024-03-28T19:08:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.992+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe744153ed0 con 0x7fe7440727e0 2024-03-28T19:08:56.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.994+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe744067dd0 con 0x7fe7440727e0 2024-03-28T19:08:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.995+0000 7fe73aa00640 1 -- 172.21.15.27:0/3466457455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe73401a070 con 0x7fe7440727e0 2024-03-28T19:08:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.996+0000 7fe73aa00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72406bca0 0x7fe72406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.996+0000 7fe73aa00640 1 -- 172.21.15.27:0/3466457455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7fe73408b710 con 0x7fe7440727e0 2024-03-28T19:08:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.996+0000 7fe742000640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72406bca0 0x7fe72406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:56.997+0000 7fe742000640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72406bca0 0x7fe72406e160 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe730035000 tx=0x7fe730008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.003+0000 7fe73aa00640 1 -- 172.21.15.27:0/3466457455 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe734059260 con 0x7fe7440727e0 2024-03-28T19:08:57.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.337+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7441ef630 con 0x7fe7440727e0 2024-03-28T19:08:57.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:08:57.337 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:08:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 111 2024-03-28T19:08:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.338+0000 7fe73aa00640 1 -- 172.21.15.27:0/3466457455 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5469 (secure 0 0 0) 0x7fe73405d300 con 0x7fe7440727e0 2024-03-28T19:08:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72406bca0 msgr2=0x7fe72406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72406bca0 0x7fe72406e160 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe730035000 tx=0x7fe730008030 comp rx=0 tx=0).stop 2024-03-28T19:08:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 msgr2=0x7fe744153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744153990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe73400e9e0 tx=0x7fe73400eea0 comp rx=0 tx=0).stop 2024-03-28T19:08:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 shutdown_connections 2024-03-28T19:08:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe72406bca0 0x7fe72406e160 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe744155600 0x7fe74414c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7440c59d0 0x7fe7441f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 --2- 172.21.15.27:0/3466457455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7440727e0 0x7fe744153990 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 >> 172.21.15.27:0/3466457455 conn(0x7fe7440ac910 msgr2=0x7fe744154600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 shutdown_connections 2024-03-28T19:08:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:57.341+0000 7fe743e00640 1 -- 172.21.15.27:0/3466457455 wait complete. 2024-03-28T19:08:57.504 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-03-28T19:08:59.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:08:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.746+0000 7f0ba3e00640 1 Processor -- start 2024-03-28T19:08:59.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.747+0000 7f0ba3e00640 1 -- start start 2024-03-28T19:08:59.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.748+0000 7f0ba3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c158580 0x7f0b9c1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:59.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.748+0000 7f0ba3e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c156600 con 0x7f0b9c0be630 2024-03-28T19:08:59.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.748+0000 7f0ba3e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c0c7910 con 0x7f0b9c159150 2024-03-28T19:08:59.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.748+0000 7f0ba3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0c76e0 con 0x7f0b9c158580 2024-03-28T19:08:59.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.749+0000 7f0ba0a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b9c0be630 0x7f0b9c0c5900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60374/0 (socket says 172.21.15.27:60374) 2024-03-28T19:08:59.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.749+0000 7f0ba0a00640 1 -- 172.21.15.27:0/1580347601 learned_addr learned my addr 172.21.15.27:0/1580347601 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:08:59.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.749+0000 7f0ba1400640 1 --2- 172.21.15.27:0/1580347601 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c158580 0x7f0b9c1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.749+0000 7f0b9be00640 1 -- 172.21.15.27:0/1580347601 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968983625 0 0) 0x7f0b88001110 con 0x7f0b9c0be630 2024-03-28T19:08:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.749+0000 7f0b9be00640 1 -- 172.21.15.27:0/1580347601 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b84003520 con 0x7f0b9c0be630 2024-03-28T19:08:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.750+0000 7f0b9be00640 1 -- 172.21.15.27:0/1580347601 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489911638 0 0) 0x7f0b9c0c7910 con 0x7f0b9c159150 2024-03-28T19:08:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.750+0000 7f0b9be00640 1 -- 172.21.15.27:0/1580347601 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b88001110 con 0x7f0b9c159150 2024-03-28T19:08:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.750+0000 7f0ba1400640 1 -- 172.21.15.27:0/1580347601 >> v1:172.21.15.155:6789/0 conn(0x7f0b9c159150 legacy=0x7f0b9c1595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.750+0000 7f0ba1400640 1 -- 172.21.15.27:0/1580347601 >> v1:172.21.15.27:6789/0 conn(0x7f0b9c0be630 legacy=0x7f0b9c0c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:59.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.750+0000 7f0ba1400640 1 -- 172.21.15.27:0/1580347601 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c0c7bf0 con 0x7f0b9c158580 2024-03-28T19:08:59.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.751+0000 7f0ba1400640 1 --2- 172.21.15.27:0/1580347601 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c158580 0x7f0b9c1560c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b900097d0 tx=0x7f0b9002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcd759dac93dd933 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:59.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.752+0000 7f0b9be00640 1 -- 172.21.15.27:0/1580347601 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90035020 con 0x7f0b9c158580 2024-03-28T19:08:59.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.752+0000 7f0b9be00640 1 -- 172.21.15.27:0/1580347601 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b90031810 con 0x7f0b9c158580 2024-03-28T19:08:59.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.752+0000 7f0b9be00640 1 -- 172.21.15.27:0/1580347601 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b90031bc0 con 0x7f0b9c158580 2024-03-28T19:08:59.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.753+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1580347601 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c158580 msgr2=0x7f0b9c1560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:59.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.753+0000 7f0ba3e00640 1 --2- 172.21.15.27:0/1580347601 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c158580 0x7f0b9c1560c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b900097d0 tx=0x7f0b9002b8b0 comp rx=0 tx=0).stop 2024-03-28T19:08:59.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.753+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1580347601 shutdown_connections 2024-03-28T19:08:59.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.753+0000 7f0ba3e00640 1 --2- 172.21.15.27:0/1580347601 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c158580 0x7f0b9c1560c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:59.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.753+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1580347601 >> 172.21.15.27:0/1580347601 conn(0x7f0b9c0caf80 msgr2=0x7f0b9c0cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:08:59.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.753+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1580347601 shutdown_connections 2024-03-28T19:08:59.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.753+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1580347601 wait complete. 2024-03-28T19:08:59.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.756+0000 7f0ba3e00640 1 Processor -- start 2024-03-28T19:08:59.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.756+0000 7f0ba3e00640 1 -- start start 2024-03-28T19:08:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.757+0000 7f0ba3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b9c0be630 0x7f0b9c0acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.757+0000 7f0ba3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b9c158580 0x7f0b9c0c3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c159150 0x7f0b9c1eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba3e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c158cc0 con 0x7f0b9c158580 2024-03-28T19:08:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba3e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0c6dd0 con 0x7f0b9c0be630 2024-03-28T19:08:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0c81b0 con 0x7f0b9c159150 2024-03-28T19:08:59.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b9c158580 0x7f0b9c0c3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b9c158580 0x7f0b9c0c3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59428/0 (socket says 172.21.15.27:59428) 2024-03-28T19:08:59.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba0a00640 1 -- 172.21.15.27:0/1455348803 learned_addr learned my addr 172.21.15.27:0/1455348803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba1400640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b9c0be630 0x7f0b9c0acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.758+0000 7f0ba1e00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c159150 0x7f0b9c1eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0ba0a00640 1 -- 172.21.15.27:0/1455348803 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c159150 msgr2=0x7f0b9c1eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0ba0a00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c159150 0x7f0b9c1eb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0ba0a00640 1 -- 172.21.15.27:0/1455348803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b9c0be630 msgr2=0x7f0b9c0acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0ba0a00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b9c0be630 0x7f0b9c0acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0ba0a00640 1 -- 172.21.15.27:0/1455348803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c1ead30 con 0x7f0b9c158580 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0ba0a00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b9c158580 0x7f0b9c0c3310 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800c6b0 tx=0x7f0b8800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0b99600640 1 -- 172.21.15.27:0/1455348803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88017070 con 0x7f0b9c158580 2024-03-28T19:08:59.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0b99600640 1 -- 172.21.15.27:0/1455348803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b88005ab0 con 0x7f0b9c158580 2024-03-28T19:08:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0b99600640 1 -- 172.21.15.27:0/1455348803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88007070 con 0x7f0b9c158580 2024-03-28T19:08:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.759+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c0c1700 con 0x7f0b9c158580 2024-03-28T19:08:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.760+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c0c1cc0 con 0x7f0b9c158580 2024-03-28T19:08:59.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.761+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c04f400 con 0x7f0b9c158580 2024-03-28T19:08:59.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.762+0000 7f0b99600640 1 -- 172.21.15.27:0/1455348803 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0b8801a070 con 0x7f0b9c158580 2024-03-28T19:08:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.764+0000 7f0b99600640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b7c06bc20 0x7f0b7c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:08:59.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.764+0000 7f0b99600640 1 -- 172.21.15.27:0/1455348803 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f0b8808a9b0 con 0x7f0b9c158580 2024-03-28T19:08:59.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.765+0000 7f0ba1400640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b7c06bc20 0x7f0b7c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:08:59.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.766+0000 7f0ba1400640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b7c06bc20 0x7f0b7c06e0e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0b90036000 tx=0x7f0b90008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:08:59.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:08:59.769+0000 7f0b99600640 1 -- 172.21.15.27:0/1455348803 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0b9c156600 con 0x7f0b9c158580 2024-03-28T19:09:00.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.136+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b9c0729e0 con 0x7f0b9c158580 2024-03-28T19:09:00.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.137+0000 7f0b99600640 1 -- 172.21.15.27:0/1455348803 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f0b88058b50 con 0x7f0b9c158580 2024-03-28T19:09:00.137 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:00.137 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:00.137 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:00.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.140+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b7c06bc20 msgr2=0x7f0b7c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.140+0000 7f0ba3e00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b7c06bc20 0x7f0b7c06e0e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0b90036000 tx=0x7f0b90008030 comp rx=0 tx=0).stop 2024-03-28T19:09:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.140+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b9c158580 msgr2=0x7f0b9c0c3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.140+0000 7f0ba3e00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b9c158580 0x7f0b9c0c3310 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800c6b0 tx=0x7f0b8800cb70 comp rx=0 tx=0).stop 2024-03-28T19:09:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.141+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 shutdown_connections 2024-03-28T19:09:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.141+0000 7f0ba3e00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b7c06bc20 0x7f0b7c06e0e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:00.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.141+0000 7f0ba3e00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b9c159150 0x7f0b9c1eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:00.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.141+0000 7f0ba3e00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b9c158580 0x7f0b9c0c3310 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:00.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.141+0000 7f0ba3e00640 1 --2- 172.21.15.27:0/1455348803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b9c0be630 0x7f0b9c0acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:00.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.141+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 >> 172.21.15.27:0/1455348803 conn(0x7f0b9c0caf80 msgr2=0x7f0b9c151460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:00.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.141+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 shutdown_connections 2024-03-28T19:09:00.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:00.141+0000 7f0ba3e00640 1 -- 172.21.15.27:0/1455348803 wait complete. 2024-03-28T19:09:00.160 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-03-28T19:09:00.161 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2024-03-28T19:09:01.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:09:01.218 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:09:31.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.336+0000 7f3bcf200640 1 Processor -- start 2024-03-28T19:09:31.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.337+0000 7f3bcf200640 1 -- start start 2024-03-28T19:09:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bcf200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3bd0150ae0 0x7f3bd00a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:31.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bcf200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bd00a61a0 0x7f3bd00c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:31.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bcf200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd0150ec0 con 0x7f3bd00c6ee0 2024-03-28T19:09:31.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bcf200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd006dc70 con 0x7f3bd00a61a0 2024-03-28T19:09:31.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bcf200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd0153550 con 0x7f3bd0150ae0 2024-03-28T19:09:31.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bce800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3bd00c6ee0 0x7f3bd0153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39774/0 (socket says 172.21.15.27:39774) 2024-03-28T19:09:31.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bc7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bd00a61a0 0x7f3bd00c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:31.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bce800640 1 -- 172.21.15.27:0/2793537038 learned_addr learned my addr 172.21.15.27:0/2793537038 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:09:31.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.338+0000 7f3bcde00640 1 --2- 172.21.15.27:0/2793537038 >> v2:172.21.15.155:3300/0 conn(0x7f3bd0150ae0 0x7f3bd00a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.339+0000 7f3bc7e00640 1 -- 172.21.15.27:0/2793537038 >> v2:172.21.15.155:3300/0 conn(0x7f3bd0150ae0 msgr2=0x7f3bd00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.339+0000 7f3bc7e00640 1 --2- 172.21.15.27:0/2793537038 >> v2:172.21.15.155:3300/0 conn(0x7f3bd0150ae0 0x7f3bd00a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.339+0000 7f3bc7e00640 1 -- 172.21.15.27:0/2793537038 >> v1:172.21.15.27:6789/0 conn(0x7f3bd00c6ee0 legacy=0x7f3bd0153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.339+0000 7f3bc7e00640 1 -- 172.21.15.27:0/2793537038 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd006df10 con 0x7f3bd00a61a0 2024-03-28T19:09:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.340+0000 7f3bc7e00640 1 --2- 172.21.15.27:0/2793537038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd00a61a0 0x7f3bd00c69a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3bd00a9760 tx=0x7f3bb802b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3be3fe1afd3ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.341+0000 7f3bcd400640 1 -- 172.21.15.27:0/2793537038 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8037070 con 0x7f3bd00a61a0 2024-03-28T19:09:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.341+0000 7f3bcd400640 1 -- 172.21.15.27:0/2793537038 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb8030710 con 0x7f3bd00a61a0 2024-03-28T19:09:31.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.341+0000 7f3bcd400640 1 -- 172.21.15.27:0/2793537038 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8030ae0 con 0x7f3bd00a61a0 2024-03-28T19:09:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.342+0000 7f3bcf200640 1 -- 172.21.15.27:0/2793537038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd00a61a0 msgr2=0x7f3bd00c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.342+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2793537038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd00a61a0 0x7f3bd00c69a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3bd00a9760 tx=0x7f3bb802b680 comp rx=0 tx=0).stop 2024-03-28T19:09:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.342+0000 7f3bcf200640 1 -- 172.21.15.27:0/2793537038 shutdown_connections 2024-03-28T19:09:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.342+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2793537038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd00a61a0 0x7f3bd00c69a0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.342+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2793537038 >> v2:172.21.15.155:3300/0 conn(0x7f3bd0150ae0 0x7f3bd00a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.342+0000 7f3bcf200640 1 -- 172.21.15.27:0/2793537038 >> 172.21.15.27:0/2793537038 conn(0x7f3bd006b0c0 msgr2=0x7f3bd006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:31.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.342+0000 7f3bcf200640 1 -- 172.21.15.27:0/2793537038 shutdown_connections 2024-03-28T19:09:31.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.342+0000 7f3bcf200640 1 -- 172.21.15.27:0/2793537038 wait complete. 2024-03-28T19:09:31.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.345+0000 7f3bcf200640 1 Processor -- start 2024-03-28T19:09:31.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.345+0000 7f3bcf200640 1 -- start start 2024-03-28T19:09:31.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.346+0000 7f3bcf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bd00a61a0 0x7f3bd01f0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:31.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bcf200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bd00c6ee0 0x7f3bd00aa400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:31.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bcf200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd0150ae0 0x7f3bd0151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:31.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bcf200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd006e7b0 con 0x7f3bd0150ae0 2024-03-28T19:09:31.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bcf200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd006e4d0 con 0x7f3bd00c6ee0 2024-03-28T19:09:31.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bcf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01569d0 con 0x7f3bd00a61a0 2024-03-28T19:09:31.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd0150ae0 0x7f3bd0151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:31.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd0150ae0 0x7f3bd0151ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46644/0 (socket says 172.21.15.27:46644) 2024-03-28T19:09:31.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bce800640 1 -- 172.21.15.27:0/2174531265 learned_addr learned my addr 172.21.15.27:0/2174531265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bc7e00640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bd00c6ee0 0x7f3bd00aa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.347+0000 7f3bcde00640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bd00a61a0 0x7f3bd01f0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bce800640 1 -- 172.21.15.27:0/2174531265 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bd00a61a0 msgr2=0x7f3bd01f0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bce800640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bd00a61a0 0x7f3bd01f0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bce800640 1 -- 172.21.15.27:0/2174531265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bd00c6ee0 msgr2=0x7f3bd00aa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bce800640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bd00c6ee0 0x7f3bd00aa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bce800640 1 -- 172.21.15.27:0/2174531265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd01f05c0 con 0x7f3bd0150ae0 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bce800640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd0150ae0 0x7f3bd0151ec0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800cb60 tx=0x7f3bc800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bc6000640 1 -- 172.21.15.27:0/2174531265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8018070 con 0x7f3bd0150ae0 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bc6000640 1 -- 172.21.15.27:0/2174531265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc8004d60 con 0x7f3bd0150ae0 2024-03-28T19:09:31.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.348+0000 7f3bc6000640 1 -- 172.21.15.27:0/2174531265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc800ad20 con 0x7f3bd0150ae0 2024-03-28T19:09:31.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.349+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd00aae40 con 0x7f3bd0150ae0 2024-03-28T19:09:31.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.349+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd01513e0 con 0x7f3bd0150ae0 2024-03-28T19:09:31.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.350+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd0067dd0 con 0x7f3bd0150ae0 2024-03-28T19:09:31.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:09:31.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.352+0000 7f3bc6000640 1 -- 172.21.15.27:0/2174531265 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3bc80070d0 con 0x7f3bd0150ae0 2024-03-28T19:09:31.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.353+0000 7f3bc6000640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bac06bbd0 0x7f3bac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:31.358 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:09:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.353+0000 7f3bc6000640 1 -- 172.21.15.27:0/2174531265 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f3bc808ad60 con 0x7f3bd0150ae0 2024-03-28T19:09:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.354+0000 7f3bcde00640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bac06bbd0 0x7f3bac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:31.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.355+0000 7f3bcde00640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bac06bbd0 0x7f3bac06e090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0006fd0 tx=0x7f3bc0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.361+0000 7f3bc6000640 1 -- 172.21.15.27:0/2174531265 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3bd0150ec0 con 0x7f3bd0150ae0 2024-03-28T19:09:31.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.717+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bd006da50 con 0x7f3bd0150ae0 2024-03-28T19:09:31.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.717+0000 7f3bc6000640 1 -- 172.21.15.27:0/2174531265 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f3bc8058ee0 con 0x7f3bd0150ae0 2024-03-28T19:09:31.717 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:31.717 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:31.717 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:31.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bac06bbd0 msgr2=0x7f3bac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bac06bbd0 0x7f3bac06e090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0006fd0 tx=0x7f3bc0008040 comp rx=0 tx=0).stop 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd0150ae0 msgr2=0x7f3bd0151ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd0150ae0 0x7f3bd0151ec0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800cb60 tx=0x7f3bc800a430 comp rx=0 tx=0).stop 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 shutdown_connections 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3bac06bbd0 0x7f3bac06e090 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bd0150ae0 0x7f3bd0151ec0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bd00c6ee0 0x7f3bd00aa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 --2- 172.21.15.27:0/2174531265 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bd00a61a0 0x7f3bd01f0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.720+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 >> 172.21.15.27:0/2174531265 conn(0x7f3bd006b0c0 msgr2=0x7f3bd00c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:31.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.721+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 shutdown_connections 2024-03-28T19:09:31.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:31.721+0000 7f3bcf200640 1 -- 172.21.15.27:0/2174531265 wait complete. 2024-03-28T19:09:31.741 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-03-28T19:09:31.741 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:09:31.741 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:09:31.768 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:09:31.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.034+0000 7f91e4000640 1 Processor -- start 2024-03-28T19:09:32.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.034+0000 7f91e4000640 1 -- start start 2024-03-28T19:09:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.035+0000 7f91e4000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f91dc0a9cb0 0x7f91dc0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.035+0000 7f91e4000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc1534c0 0x7f91dc153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.035+0000 7f91e4000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc0c3d90 con 0x7f91dc150c70 2024-03-28T19:09:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.035+0000 7f91e4000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc0ab4e0 con 0x7f91dc0a9cb0 2024-03-28T19:09:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.035+0000 7f91e4000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0ac1f0 con 0x7f91dc1534c0 2024-03-28T19:09:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.036+0000 7f91e0c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f91dc0a9cb0 0x7f91dc0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:32.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.036+0000 7f91e0c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f91dc0a9cb0 0x7f91dc0c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46660/0 (socket says 172.21.15.27:46660) 2024-03-28T19:09:32.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.036+0000 7f91e0c00640 1 -- 172.21.15.27:0/3416669143 learned_addr learned my addr 172.21.15.27:0/3416669143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:32.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.036+0000 7f91e2000640 1 --2- 172.21.15.27:0/3416669143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc1534c0 0x7f91dc153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:32.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.036+0000 7f91e0c00640 1 -- 172.21.15.27:0/3416669143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc1534c0 msgr2=0x7f91dc153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:32.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.036+0000 7f91e0c00640 1 --2- 172.21.15.27:0/3416669143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc1534c0 0x7f91dc153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.037+0000 7f91e0c00640 1 -- 172.21.15.27:0/3416669143 >> v1:172.21.15.155:6789/0 conn(0x7f91dc150c70 legacy=0x7f91dc0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:32.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.037+0000 7f91e0c00640 1 -- 172.21.15.27:0/3416669143 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0ab710 con 0x7f91dc0a9cb0 2024-03-28T19:09:32.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.038+0000 7f91e0c00640 1 --2- 172.21.15.27:0/3416669143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc0a9cb0 0x7f91dc0c3690 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f91c8009870 tx=0x7f91c802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9672fd2854cb4d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:32.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.038+0000 7f91dbe00640 1 -- 172.21.15.27:0/3416669143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c80317c0 con 0x7f91dc0a9cb0 2024-03-28T19:09:32.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.038+0000 7f91dbe00640 1 -- 172.21.15.27:0/3416669143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c8037070 con 0x7f91dc0a9cb0 2024-03-28T19:09:32.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.038+0000 7f91dbe00640 1 -- 172.21.15.27:0/3416669143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8031b80 con 0x7f91dc0a9cb0 2024-03-28T19:09:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.039+0000 7f91e4000640 1 -- 172.21.15.27:0/3416669143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc0a9cb0 msgr2=0x7f91dc0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:32.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.039+0000 7f91e4000640 1 --2- 172.21.15.27:0/3416669143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc0a9cb0 0x7f91dc0c3690 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f91c8009870 tx=0x7f91c802b860 comp rx=0 tx=0).stop 2024-03-28T19:09:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.039+0000 7f91e4000640 1 -- 172.21.15.27:0/3416669143 shutdown_connections 2024-03-28T19:09:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.039+0000 7f91e4000640 1 --2- 172.21.15.27:0/3416669143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc1534c0 0x7f91dc153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.039+0000 7f91e4000640 1 --2- 172.21.15.27:0/3416669143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc0a9cb0 0x7f91dc0c3690 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.039+0000 7f91e4000640 1 -- 172.21.15.27:0/3416669143 >> 172.21.15.27:0/3416669143 conn(0x7f91dc06b9d0 msgr2=0x7f91dc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.039+0000 7f91e4000640 1 -- 172.21.15.27:0/3416669143 shutdown_connections 2024-03-28T19:09:32.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.039+0000 7f91e4000640 1 -- 172.21.15.27:0/3416669143 wait complete. 2024-03-28T19:09:32.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.041+0000 7f91e4000640 1 Processor -- start 2024-03-28T19:09:32.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.041+0000 7f91e4000640 1 -- start start 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.042+0000 7f91e4000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91dc0a9cb0 0x7f91dc156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.042+0000 7f91e4000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc150c70 0x7f91dc071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.042+0000 7f91e4000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc1534c0 0x7f91dc14dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.042+0000 7f91e4000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0c1810 con 0x7f91dc1534c0 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.042+0000 7f91e4000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0abcd0 con 0x7f91dc0a9cb0 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.042+0000 7f91e4000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0c6830 con 0x7f91dc150c70 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc1534c0 0x7f91dc14dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc1534c0 0x7f91dc14dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46674/0 (socket says 172.21.15.27:46674) 2024-03-28T19:09:32.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e2000640 1 -- 172.21.15.27:0/3452985710 learned_addr learned my addr 172.21.15.27:0/3452985710 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e0c00640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc150c70 0x7f91dc071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e1600640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91dc0a9cb0 0x7f91dc156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e2000640 1 -- 172.21.15.27:0/3452985710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc150c70 msgr2=0x7f91dc071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e2000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc150c70 0x7f91dc071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e2000640 1 -- 172.21.15.27:0/3452985710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91dc0a9cb0 msgr2=0x7f91dc156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e2000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91dc0a9cb0 0x7f91dc156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.043+0000 7f91e2000640 1 -- 172.21.15.27:0/3452985710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0a7c00 con 0x7f91dc1534c0 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.044+0000 7f91e2000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc1534c0 0x7f91dc14dc00 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f91d4007e60 tx=0x7f91d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.044+0000 7f91d9600640 1 -- 172.21.15.27:0/3452985710 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4014420 con 0x7f91dc1534c0 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.044+0000 7f91d9600640 1 -- 172.21.15.27:0/3452985710 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d4019070 con 0x7f91dc1534c0 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.044+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91dc1eecb0 con 0x7f91dc1534c0 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.044+0000 7f91d9600640 1 -- 172.21.15.27:0/3452985710 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4014820 con 0x7f91dc1534c0 2024-03-28T19:09:32.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.044+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91dc154990 con 0x7f91dc1534c0 2024-03-28T19:09:32.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.046+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91dc04f400 con 0x7f91dc1534c0 2024-03-28T19:09:32.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.047+0000 7f91d9600640 1 -- 172.21.15.27:0/3452985710 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f91d40149c0 con 0x7f91dc1534c0 2024-03-28T19:09:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.048+0000 7f91d9600640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91bc06bbd0 0x7f91bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.048+0000 7f91d9600640 1 -- 172.21.15.27:0/3452985710 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f91d408bd50 con 0x7f91dc1534c0 2024-03-28T19:09:32.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.048+0000 7f91e1600640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91bc06bbd0 0x7f91bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:32.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.049+0000 7f91e1600640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91bc06bbd0 0x7f91bc06e090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f91d0004ea0 tx=0x7f91d0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:32.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.057+0000 7f91d9600640 1 -- 172.21.15.27:0/3452985710 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f91d4059df0 con 0x7f91dc1534c0 2024-03-28T19:09:32.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.353+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91dc0c5f00 con 0x7f91dc1534c0 2024-03-28T19:09:32.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.354+0000 7f91d9600640 1 -- 172.21.15.27:0/3452985710 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f91d40596f0 con 0x7f91dc1534c0 2024-03-28T19:09:32.353 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:32.353 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:32.353 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:32.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91bc06bbd0 msgr2=0x7f91bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:32.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91bc06bbd0 0x7f91bc06e090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f91d0004ea0 tx=0x7f91d0009040 comp rx=0 tx=0).stop 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc1534c0 msgr2=0x7f91dc14dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc1534c0 0x7f91dc14dc00 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f91d4007e60 tx=0x7f91d400e4b0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 shutdown_connections 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91bc06bbd0 0x7f91bc06e090 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91dc1534c0 0x7f91dc14dc00 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91dc150c70 0x7f91dc071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 --2- 172.21.15.27:0/3452985710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91dc0a9cb0 0x7f91dc156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 >> 172.21.15.27:0/3452985710 conn(0x7f91dc06b9d0 msgr2=0x7f91dc0c1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 shutdown_connections 2024-03-28T19:09:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:32.357+0000 7f91e4000640 1 -- 172.21.15.27:0/3452985710 wait complete. 2024-03-28T19:09:32.375 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:09:34.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:34.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.617+0000 7f0aa7000640 1 Processor -- start 2024-03-28T19:09:34.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.617+0000 7f0aa7000640 1 -- start start 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0aa7000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa0150c70 0x7f0aa00a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0aa7000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0aa00c3690 0x7f0aa00c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0aa7000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0aa00c6660 0x7f0aa01534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0aa7000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa00ab4e0 con 0x7f0aa00c3690 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0aa7000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa00ac1f0 con 0x7f0aa00c6660 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0aa7000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00c3af0 con 0x7f0aa0150c70 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0a9f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0aa00c3690 0x7f0aa00c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0a9f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0aa00c3690 0x7f0aa00c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46682/0 (socket says 172.21.15.27:46682) 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.619+0000 7f0a9f400640 1 -- 172.21.15.27:0/639076523 learned_addr learned my addr 172.21.15.27:0/639076523 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:34.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.620+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/639076523 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa0150c70 0x7f0aa00a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:34.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.620+0000 7f0aa5000640 1 --2- 172.21.15.27:0/639076523 >> v2:172.21.15.155:3300/0 conn(0x7f0aa00c6660 0x7f0aa01534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.620+0000 7f0a9f400640 1 -- 172.21.15.27:0/639076523 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa0150c70 msgr2=0x7f0aa00a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.620+0000 7f0a9f400640 1 --2- 172.21.15.27:0/639076523 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa0150c70 0x7f0aa00a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.620+0000 7f0a9f400640 1 -- 172.21.15.27:0/639076523 >> v2:172.21.15.155:3300/0 conn(0x7f0aa00c6660 msgr2=0x7f0aa01534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.620+0000 7f0a9f400640 1 --2- 172.21.15.27:0/639076523 >> v2:172.21.15.155:3300/0 conn(0x7f0aa00c6660 0x7f0aa01534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.620+0000 7f0a9f400640 1 -- 172.21.15.27:0/639076523 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa00c3d90 con 0x7f0aa00c3690 2024-03-28T19:09:34.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.620+0000 7f0aa5000640 1 --2- 172.21.15.27:0/639076523 >> v2:172.21.15.155:3300/0 conn(0x7f0aa00c6660 0x7f0aa01534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:09:34.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.621+0000 7f0a9f400640 1 --2- 172.21.15.27:0/639076523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c3690 0x7f0aa00c6120 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f0aa006c6a0 tx=0x7f0a8c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27914721c7656011 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:34.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.622+0000 7f0a9ea00640 1 -- 172.21.15.27:0/639076523 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0317c0 con 0x7f0aa00c3690 2024-03-28T19:09:34.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.622+0000 7f0a9ea00640 1 -- 172.21.15.27:0/639076523 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c037070 con 0x7f0aa00c3690 2024-03-28T19:09:34.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.622+0000 7f0a9ea00640 1 -- 172.21.15.27:0/639076523 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c031b80 con 0x7f0aa00c3690 2024-03-28T19:09:34.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.622+0000 7f0aa7000640 1 -- 172.21.15.27:0/639076523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c3690 msgr2=0x7f0aa00c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:34.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.622+0000 7f0aa7000640 1 --2- 172.21.15.27:0/639076523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c3690 0x7f0aa00c6120 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f0aa006c6a0 tx=0x7f0a8c02b860 comp rx=0 tx=0).stop 2024-03-28T19:09:34.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.623+0000 7f0aa7000640 1 -- 172.21.15.27:0/639076523 shutdown_connections 2024-03-28T19:09:34.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.623+0000 7f0aa7000640 1 --2- 172.21.15.27:0/639076523 >> v2:172.21.15.155:3300/0 conn(0x7f0aa00c6660 0x7f0aa01534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.623+0000 7f0aa7000640 1 --2- 172.21.15.27:0/639076523 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c3690 0x7f0aa00c6120 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.623+0000 7f0aa7000640 1 --2- 172.21.15.27:0/639076523 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa0150c70 0x7f0aa00a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.623+0000 7f0aa7000640 1 -- 172.21.15.27:0/639076523 >> 172.21.15.27:0/639076523 conn(0x7f0aa006b9d0 msgr2=0x7f0aa00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:34.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.623+0000 7f0aa7000640 1 -- 172.21.15.27:0/639076523 shutdown_connections 2024-03-28T19:09:34.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.623+0000 7f0aa7000640 1 -- 172.21.15.27:0/639076523 wait complete. 2024-03-28T19:09:34.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.626+0000 7f0aa7000640 1 Processor -- start 2024-03-28T19:09:34.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.626+0000 7f0aa7000640 1 -- start start 2024-03-28T19:09:34.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0aa7000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa00c3690 0x7f0aa01ef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:34.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0aa7000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c6660 0x7f0aa0148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:34.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0aa7000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0aa0150c70 0x7f0aa01eb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:34.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0aa7000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00abcd0 con 0x7f0aa00c6660 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0aa7000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00ab9f0 con 0x7f0aa0150c70 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0aa7000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00c1810 con 0x7f0aa00c3690 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0a9f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c6660 0x7f0aa0148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0a9f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c6660 0x7f0aa0148e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46684/0 (socket says 172.21.15.27:46684) 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0a9f400640 1 -- 172.21.15.27:0/3821359766 learned_addr learned my addr 172.21.15.27:0/3821359766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa00c3690 0x7f0aa01ef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.627+0000 7f0aa5000640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0aa0150c70 0x7f0aa01eb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.628+0000 7f0a9f400640 1 -- 172.21.15.27:0/3821359766 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa00c3690 msgr2=0x7f0aa01ef0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.628+0000 7f0a9f400640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa00c3690 0x7f0aa01ef0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.628+0000 7f0a9f400640 1 -- 172.21.15.27:0/3821359766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0aa0150c70 msgr2=0x7f0aa01eb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.628+0000 7f0a9f400640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0aa0150c70 0x7f0aa01eb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.628+0000 7f0a9f400640 1 -- 172.21.15.27:0/3821359766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa006e150 con 0x7f0aa00c6660 2024-03-28T19:09:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.628+0000 7f0a9f400640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c6660 0x7f0aa0148e20 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009870 tx=0x7f0a8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.629+0000 7f0a97e00640 1 -- 172.21.15.27:0/3821359766 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c032650 con 0x7f0aa00c6660 2024-03-28T19:09:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.629+0000 7f0a97e00640 1 -- 172.21.15.27:0/3821359766 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c0327f0 con 0x7f0aa00c6660 2024-03-28T19:09:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.629+0000 7f0a97e00640 1 -- 172.21.15.27:0/3821359766 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c032be0 con 0x7f0aa00c6660 2024-03-28T19:09:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.629+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa006e430 con 0x7f0aa00c6660 2024-03-28T19:09:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.629+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa01eeb90 con 0x7f0aa00c6660 2024-03-28T19:09:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.630+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa004f400 con 0x7f0aa00c6660 2024-03-28T19:09:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.631+0000 7f0a97e00640 1 -- 172.21.15.27:0/3821359766 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0a8c04c060 con 0x7f0aa00c6660 2024-03-28T19:09:34.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.632+0000 7f0a97e00640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a8006bbd0 0x7f0a8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:34.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.632+0000 7f0a97e00640 1 -- 172.21.15.27:0/3821359766 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f0a8c0acbe0 con 0x7f0aa00c6660 2024-03-28T19:09:34.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.633+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a8006bbd0 0x7f0a8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:34.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.634+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a8006bbd0 0x7f0a8006e090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0a90006fd0 tx=0x7f0a90008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:34.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.637+0000 7f0a97e00640 1 -- 172.21.15.27:0/3821359766 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0a8c040070 con 0x7f0aa00c6660 2024-03-28T19:09:34.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.915+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aa00c0e20 con 0x7f0aa00c6660 2024-03-28T19:09:34.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.915+0000 7f0a97e00640 1 -- 172.21.15.27:0/3821359766 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f0a8c07ad00 con 0x7f0aa00c6660 2024-03-28T19:09:34.915 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:34.915 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:34.916 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:34.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a8006bbd0 msgr2=0x7f0a8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:34.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a8006bbd0 0x7f0a8006e090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0a90006fd0 tx=0x7f0a90008040 comp rx=0 tx=0).stop 2024-03-28T19:09:34.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c6660 msgr2=0x7f0aa0148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:34.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c6660 0x7f0aa0148e20 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009870 tx=0x7f0a8c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 shutdown_connections 2024-03-28T19:09:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a8006bbd0 0x7f0a8006e090 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0aa0150c70 0x7f0aa01eb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0aa00c6660 0x7f0aa0148e20 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.919+0000 7f0aa7000640 1 --2- 172.21.15.27:0/3821359766 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0aa00c3690 0x7f0aa01ef0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.920+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 >> 172.21.15.27:0/3821359766 conn(0x7f0aa006b9d0 msgr2=0x7f0aa00712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.920+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 shutdown_connections 2024-03-28T19:09:34.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:34.920+0000 7f0aa7000640 1 -- 172.21.15.27:0/3821359766 wait complete. 2024-03-28T19:09:34.941 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:09:35.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:35.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.422+0000 7f5468400640 1 Processor -- start 2024-03-28T19:09:35.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.422+0000 7f5468400640 1 -- start start 2024-03-28T19:09:35.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.425+0000 7f5468400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5460150c70 0x7f54600a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:35.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.425+0000 7f5468400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600c3af0 con 0x7f54601534c0 2024-03-28T19:09:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.425+0000 7f5468400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5460154020 con 0x7f54600c3690 2024-03-28T19:09:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.425+0000 7f5468400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54600ab4e0 con 0x7f5460150c70 2024-03-28T19:09:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.425+0000 7f5465a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5460150c70 0x7f54600a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.425+0000 7f5465a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5460150c70 0x7f54600a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46692/0 (socket says 172.21.15.27:46692) 2024-03-28T19:09:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.425+0000 7f5466400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f54601534c0 0x7f5460153910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39784/0 (socket says 172.21.15.27:39784) 2024-03-28T19:09:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.425+0000 7f5465a00640 1 -- 172.21.15.27:0/1684345280 learned_addr learned my addr 172.21.15.27:0/1684345280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:35.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.426+0000 7f545fe00640 1 -- 172.21.15.27:0/1684345280 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439208625 0 0) 0x7f54600c3af0 con 0x7f54601534c0 2024-03-28T19:09:35.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.426+0000 7f5465a00640 1 -- 172.21.15.27:0/1684345280 >> v1:172.21.15.155:6789/0 conn(0x7f54600c3690 legacy=0x7f54600c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:35.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.426+0000 7f5465a00640 1 -- 172.21.15.27:0/1684345280 >> v1:172.21.15.27:6789/0 conn(0x7f54601534c0 legacy=0x7f5460153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:35.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.426+0000 7f5465a00640 1 -- 172.21.15.27:0/1684345280 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54600c3d90 con 0x7f5460150c70 2024-03-28T19:09:35.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.427+0000 7f5465a00640 1 --2- 172.21.15.27:0/1684345280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f54600a9cb0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f54601541f0 tx=0x7f545402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c37e655c30902a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:35.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.428+0000 7f545fe00640 1 -- 172.21.15.27:0/1684345280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454034070 con 0x7f5460150c70 2024-03-28T19:09:35.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.428+0000 7f545fe00640 1 -- 172.21.15.27:0/1684345280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5454030840 con 0x7f5460150c70 2024-03-28T19:09:35.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.428+0000 7f545fe00640 1 -- 172.21.15.27:0/1684345280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454030c10 con 0x7f5460150c70 2024-03-28T19:09:35.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.428+0000 7f5468400640 1 -- 172.21.15.27:0/1684345280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 msgr2=0x7f54600a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:35.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.428+0000 7f5468400640 1 --2- 172.21.15.27:0/1684345280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f54600a9cb0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f54601541f0 tx=0x7f545402b980 comp rx=0 tx=0).stop 2024-03-28T19:09:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.428+0000 7f5468400640 1 -- 172.21.15.27:0/1684345280 shutdown_connections 2024-03-28T19:09:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.428+0000 7f5468400640 1 --2- 172.21.15.27:0/1684345280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f54600a9cb0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.428+0000 7f5468400640 1 -- 172.21.15.27:0/1684345280 >> 172.21.15.27:0/1684345280 conn(0x7f546006b9d0 msgr2=0x7f54600c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.429+0000 7f5468400640 1 -- 172.21.15.27:0/1684345280 shutdown_connections 2024-03-28T19:09:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.429+0000 7f5468400640 1 -- 172.21.15.27:0/1684345280 wait complete. 2024-03-28T19:09:35.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.431+0000 7f5468400640 1 Processor -- start 2024-03-28T19:09:35.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.431+0000 7f5468400640 1 -- start start 2024-03-28T19:09:35.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5468400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54600c3690 0x7f546014dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:35.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5468400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f5460144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:35.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5468400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54601534c0 0x7f5460144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:35.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5468400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54600ab9f0 con 0x7f5460150c70 2024-03-28T19:09:35.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5468400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54600ac1f0 con 0x7f54600c3690 2024-03-28T19:09:35.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5468400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54600abcd0 con 0x7f54601534c0 2024-03-28T19:09:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5465000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f5460144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5465000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f5460144940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46702/0 (socket says 172.21.15.27:46702) 2024-03-28T19:09:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.432+0000 7f5465000640 1 -- 172.21.15.27:0/1487875899 learned_addr learned my addr 172.21.15.27:0/1487875899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5465a00640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54600c3690 0x7f546014dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5466400640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54601534c0 0x7f5460144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5465000640 1 -- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54601534c0 msgr2=0x7f5460144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5465000640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54601534c0 0x7f5460144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5465000640 1 -- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54600c3690 msgr2=0x7f546014dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5465000640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54600c3690 0x7f546014dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5465000640 1 -- 172.21.15.27:0/1487875899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54601ece90 con 0x7f5460150c70 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5466400640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54601534c0 0x7f5460144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5465a00640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54600c3690 0x7f546014dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.433+0000 7f5465000640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f5460144940 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f544c00c450 tx=0x7f544c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.434+0000 7f545d600640 1 -- 172.21.15.27:0/1487875899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c0052d0 con 0x7f5460150c70 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.434+0000 7f545d600640 1 -- 172.21.15.27:0/1487875899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c005470 con 0x7f5460150c70 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.434+0000 7f545d600640 1 -- 172.21.15.27:0/1487875899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c012870 con 0x7f5460150c70 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.434+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546014a050 con 0x7f5460150c70 2024-03-28T19:09:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.434+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546014e2b0 con 0x7f5460150c70 2024-03-28T19:09:35.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.436+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546004f400 con 0x7f5460150c70 2024-03-28T19:09:35.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.438+0000 7f545d600640 1 -- 172.21.15.27:0/1487875899 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f544c012a10 con 0x7f5460150c70 2024-03-28T19:09:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.439+0000 7f545d600640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544006bc50 0x7f544006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.439+0000 7f545d600640 1 -- 172.21.15.27:0/1487875899 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f544c08af60 con 0x7f5460150c70 2024-03-28T19:09:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.440+0000 7f5465a00640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544006bc50 0x7f544006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:35.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.441+0000 7f5465a00640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544006bc50 0x7f544006e110 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5454036000 tx=0x7f545403b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:35.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.445+0000 7f545d600640 1 -- 172.21.15.27:0/1487875899 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f544c059160 con 0x7f5460150c70 2024-03-28T19:09:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.750+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54600c5f00 con 0x7f5460150c70 2024-03-28T19:09:35.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.751+0000 7f545d600640 1 -- 172.21.15.27:0/1487875899 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f544c058a60 con 0x7f5460150c70 2024-03-28T19:09:35.751 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:35.752 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:35.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544006bc50 msgr2=0x7f544006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:35.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544006bc50 0x7f544006e110 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5454036000 tx=0x7f545403b040 comp rx=0 tx=0).stop 2024-03-28T19:09:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 msgr2=0x7f5460144940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f5460144940 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f544c00c450 tx=0x7f544c00c910 comp rx=0 tx=0).stop 2024-03-28T19:09:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 shutdown_connections 2024-03-28T19:09:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544006bc50 0x7f544006e110 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54601534c0 0x7f5460144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5460150c70 0x7f5460144940 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.754+0000 7f5468400640 1 --2- 172.21.15.27:0/1487875899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54600c3690 0x7f546014dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.755+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 >> 172.21.15.27:0/1487875899 conn(0x7f546006b9d0 msgr2=0x7f54601525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:35.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.755+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 shutdown_connections 2024-03-28T19:09:35.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:35.755+0000 7f5468400640 1 -- 172.21.15.27:0/1487875899 wait complete. 2024-03-28T19:09:35.776 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-03-28T19:09:35.777 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:09:35.777 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:09:35.802 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:09:35.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.065+0000 7f5d82800640 1 Processor -- start 2024-03-28T19:09:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.065+0000 7f5d82800640 1 -- start start 2024-03-28T19:09:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.067+0000 7f5d82800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d7c1554f0 0x7f5d7c06fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.067+0000 7f5d82800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c155bf0 con 0x7f5d7c0c2e20 2024-03-28T19:09:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.067+0000 7f5d82800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c155ed0 con 0x7f5d7c0c2050 2024-03-28T19:09:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.067+0000 7f5d82800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c0703c0 con 0x7f5d7c1554f0 2024-03-28T19:09:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.067+0000 7f5d81e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d7c1554f0 0x7f5d7c06fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.067+0000 7f5d81e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d7c1554f0 0x7f5d7c06fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46714/0 (socket says 172.21.15.27:46714) 2024-03-28T19:09:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.067+0000 7f5d81e00640 1 -- 172.21.15.27:0/3558190586 learned_addr learned my addr 172.21.15.27:0/3558190586 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.068+0000 7f5d81e00640 1 -- 172.21.15.27:0/3558190586 >> v1:172.21.15.155:6789/0 conn(0x7f5d7c0c2050 legacy=0x7f5d7c0a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.068+0000 7f5d81e00640 1 -- 172.21.15.27:0/3558190586 >> v1:172.21.15.27:6789/0 conn(0x7f5d7c0c2e20 legacy=0x7f5d7c1591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.068+0000 7f5d7be00640 1 -- 172.21.15.27:0/3558190586 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598054296 0 0) 0x7f5d7c155bf0 con 0x7f5d7c0c2e20 2024-03-28T19:09:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.068+0000 7f5d81e00640 1 -- 172.21.15.27:0/3558190586 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c1561b0 con 0x7f5d7c1554f0 2024-03-28T19:09:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.070+0000 7f5d81e00640 1 --2- 172.21.15.27:0/3558190586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c1554f0 0x7f5d7c06fe80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009e50 tx=0x7f5d7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8eeda0baea81df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.070+0000 7f5d7be00640 1 -- 172.21.15.27:0/3558190586 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7402ca80 con 0x7f5d7c1554f0 2024-03-28T19:09:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.070+0000 7f5d7be00640 1 -- 172.21.15.27:0/3558190586 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7402cc20 con 0x7f5d7c1554f0 2024-03-28T19:09:36.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.070+0000 7f5d7be00640 1 -- 172.21.15.27:0/3558190586 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d740343f0 con 0x7f5d7c1554f0 2024-03-28T19:09:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.071+0000 7f5d82800640 1 -- 172.21.15.27:0/3558190586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c1554f0 msgr2=0x7f5d7c06fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.071+0000 7f5d82800640 1 --2- 172.21.15.27:0/3558190586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c1554f0 0x7f5d7c06fe80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009e50 tx=0x7f5d7402bee0 comp rx=0 tx=0).stop 2024-03-28T19:09:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.071+0000 7f5d82800640 1 -- 172.21.15.27:0/3558190586 shutdown_connections 2024-03-28T19:09:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.071+0000 7f5d82800640 1 --2- 172.21.15.27:0/3558190586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c1554f0 0x7f5d7c06fe80 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.071+0000 7f5d82800640 1 -- 172.21.15.27:0/3558190586 >> 172.21.15.27:0/3558190586 conn(0x7f5d7c072d10 msgr2=0x7f5d7c06d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.071+0000 7f5d82800640 1 -- 172.21.15.27:0/3558190586 shutdown_connections 2024-03-28T19:09:36.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.071+0000 7f5d82800640 1 -- 172.21.15.27:0/3558190586 wait complete. 2024-03-28T19:09:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.073+0000 7f5d82800640 1 Processor -- start 2024-03-28T19:09:36.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.073+0000 7f5d82800640 1 -- start start 2024-03-28T19:09:36.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.074+0000 7f5d82800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c2050 0x7f5d7c14b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:36.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.074+0000 7f5d82800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c2e20 0x7f5d7c1eb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.074+0000 7f5d82800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c1554f0 0x7f5d7c0adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.074+0000 7f5d82800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0c3200 con 0x7f5d7c0c2050 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.074+0000 7f5d82800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c155900 con 0x7f5d7c0c2e20 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.074+0000 7f5d82800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c1598b0 con 0x7f5d7c1554f0 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c2050 0x7f5d7c14b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c2050 0x7f5d7c14b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46718/0 (socket says 172.21.15.27:46718) 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81400640 1 -- 172.21.15.27:0/211742370 learned_addr learned my addr 172.21.15.27:0/211742370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d80a00640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c2e20 0x7f5d7c1eb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81e00640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c1554f0 0x7f5d7c0adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81400640 1 -- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c1554f0 msgr2=0x7f5d7c0adbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81400640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c1554f0 0x7f5d7c0adbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81400640 1 -- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c2e20 msgr2=0x7f5d7c1eb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81400640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c2e20 0x7f5d7c1eb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d81400640 1 -- 172.21.15.27:0/211742370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c1ef2a0 con 0x7f5d7c0c2050 2024-03-28T19:09:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.075+0000 7f5d80a00640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c2e20 0x7f5d7c1eb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:09:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.076+0000 7f5d81400640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c2050 0x7f5d7c14b1d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c440 tx=0x7f5d7000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.076+0000 7f5d79600640 1 -- 172.21.15.27:0/211742370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70017070 con 0x7f5d7c0c2050 2024-03-28T19:09:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.076+0000 7f5d79600640 1 -- 172.21.15.27:0/211742370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d70005480 con 0x7f5d7c0c2050 2024-03-28T19:09:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.076+0000 7f5d79600640 1 -- 172.21.15.27:0/211742370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70003830 con 0x7f5d7c0c2050 2024-03-28T19:09:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.076+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c149280 con 0x7f5d7c0c2050 2024-03-28T19:09:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.076+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c1ef560 con 0x7f5d7c0c2050 2024-03-28T19:09:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.078+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c067dd0 con 0x7f5d7c0c2050 2024-03-28T19:09:36.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.079+0000 7f5d79600640 1 -- 172.21.15.27:0/211742370 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5d7001a070 con 0x7f5d7c0c2050 2024-03-28T19:09:36.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.080+0000 7f5d79600640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d6006bbd0 0x7f5d6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:36.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.081+0000 7f5d79600640 1 -- 172.21.15.27:0/211742370 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f5d7008a770 con 0x7f5d7c0c2050 2024-03-28T19:09:36.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.081+0000 7f5d80a00640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d6006bbd0 0x7f5d6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:36.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.082+0000 7f5d80a00640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d6006bbd0 0x7f5d6006e090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c14c9c0 tx=0x7f5d6c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:36.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.085+0000 7f5d79600640 1 -- 172.21.15.27:0/211742370 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5d70058890 con 0x7f5d7c0c2050 2024-03-28T19:09:36.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.363+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d7c068890 con 0x7f5d7c0c2050 2024-03-28T19:09:36.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.364+0000 7f5d79600640 1 -- 172.21.15.27:0/211742370 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f5d7c068890 con 0x7f5d7c0c2050 2024-03-28T19:09:36.363 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:36.363 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:36.363 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.366+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d6006bbd0 msgr2=0x7f5d6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.366+0000 7f5d82800640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d6006bbd0 0x7f5d6006e090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c14c9c0 tx=0x7f5d6c008040 comp rx=0 tx=0).stop 2024-03-28T19:09:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.366+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c2050 msgr2=0x7f5d7c14b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.366+0000 7f5d82800640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c2050 0x7f5d7c14b1d0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c440 tx=0x7f5d7000c900 comp rx=0 tx=0).stop 2024-03-28T19:09:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.367+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 shutdown_connections 2024-03-28T19:09:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.367+0000 7f5d82800640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d6006bbd0 0x7f5d6006e090 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.367+0000 7f5d82800640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d7c1554f0 0x7f5d7c0adbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:36.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.367+0000 7f5d82800640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d7c0c2e20 0x7f5d7c1eb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.367+0000 7f5d82800640 1 --2- 172.21.15.27:0/211742370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d7c0c2050 0x7f5d7c14b1d0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.367+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 >> 172.21.15.27:0/211742370 conn(0x7f5d7c072d10 msgr2=0x7f5d7c06d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.367+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 shutdown_connections 2024-03-28T19:09:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:36.367+0000 7f5d82800640 1 -- 172.21.15.27:0/211742370 wait complete. 2024-03-28T19:09:36.385 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:09:36.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:37.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.190+0000 7fe911e00640 1 Processor -- start 2024-03-28T19:09:37.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.191+0000 7fe911e00640 1 -- start start 2024-03-28T19:09:37.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.192+0000 7fe911e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe90c070a50 0x7fe90c0733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:37.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.192+0000 7fe911e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90c06d7d0 con 0x7fe90c0a6300 2024-03-28T19:09:37.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.192+0000 7fe911e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90c0c2f00 con 0x7fe90c06ceb0 2024-03-28T19:09:37.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.192+0000 7fe911e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c31e0 con 0x7fe90c070a50 2024-03-28T19:09:37.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.193+0000 7fe90a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe90c0a6300 0x7fe90c155a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56312/0 (socket says 172.21.15.27:56312) 2024-03-28T19:09:37.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.193+0000 7fe90a000640 1 -- 172.21.15.27:0/3612843084 learned_addr learned my addr 172.21.15.27:0/3612843084 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:09:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.193+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796592129 0 0) 0x7fe90c06d7d0 con 0x7fe90c0a6300 2024-03-28T19:09:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.193+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f0003520 con 0x7fe90c0a6300 2024-03-28T19:09:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.193+0000 7fe90b400640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe90c070a50 0x7fe90c0733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1344014677 0 0) 0x7fe8f0003520 con 0x7fe90c0a6300 2024-03-28T19:09:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe90c070a50 msgr2=0x7fe90c0733b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe90c070a50 0x7fe90c0733b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 >> v1:172.21.15.155:6789/0 conn(0x7fe90c06ceb0 legacy=0x7fe90c0c1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:37.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c06dab0 con 0x7fe90c0a6300 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe8f8001670 con 0x7fe90c0a6300 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 >> v1:172.21.15.27:6789/0 conn(0x7fe90c0a6300 legacy=0x7fe90c155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8f00039e0 0x7fe8f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f0004800 0x7fe8f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe90c070a50 0x7fe8f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c31e0 con 0x7fe8f0004800 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f0005520 con 0x7fe8f00039e0 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f000e110 con 0x7fe90c070a50 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.194+0000 7fe90a000640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f0004800 0x7fe8f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.195+0000 7fe90aa00640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8f00039e0 0x7fe8f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.195+0000 7fe90b400640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe90c070a50 0x7fe8f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.195+0000 7fe90a000640 1 -- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe90c070a50 msgr2=0x7fe8f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.195+0000 7fe90a000640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe90c070a50 0x7fe8f000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.195+0000 7fe90a000640 1 -- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8f00039e0 msgr2=0x7fe8f0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:37.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.195+0000 7fe90a000640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8f00039e0 0x7fe8f0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.195+0000 7fe90a000640 1 -- 172.21.15.27:0/3612843084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0003520 con 0x7fe8f0004800 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.196+0000 7fe90a000640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f0004800 0x7fe8f0004c80 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800c210 tx=0x7fe8f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546ca71200e7de66 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.196+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8007d30 con 0x7fe8f0004800 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.196+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f8030070 con 0x7fe8f0004800 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.197+0000 7fe909600640 1 -- 172.21.15.27:0/3612843084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f802cd60 con 0x7fe8f0004800 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.197+0000 7fe911e00640 1 -- 172.21.15.27:0/3612843084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f0004800 msgr2=0x7fe8f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.197+0000 7fe911e00640 1 --2- 172.21.15.27:0/3612843084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f0004800 0x7fe8f0004c80 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800c210 tx=0x7fe8f0005830 comp rx=0 tx=0).stop 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.197+0000 7fe90b400640 1 -- 172.21.15.27:0/3612843084 reap_dead start 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.197+0000 7fe911e00640 1 -- 172.21.15.27:0/3612843084 shutdown_connections 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.197+0000 7fe911e00640 1 -- 172.21.15.27:0/3612843084 >> 172.21.15.27:0/3612843084 conn(0x7fe90c06b9d0 msgr2=0x7fe90c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.197+0000 7fe911e00640 1 -- 172.21.15.27:0/3612843084 shutdown_connections 2024-03-28T19:09:37.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.197+0000 7fe911e00640 1 -- 172.21.15.27:0/3612843084 wait complete. 2024-03-28T19:09:37.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.200+0000 7fe911e00640 1 Processor -- start 2024-03-28T19:09:37.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.200+0000 7fe911e00640 1 -- start start 2024-03-28T19:09:37.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe911e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe8f0004800 0x7fe90c1dbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:37.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe911e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe90c1d1a90 0x7fe90c1d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:37.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe911e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe90c1da030 0x7fe90c1e7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:37.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe911e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c45b0 con 0x7fe90c1d1a90 2024-03-28T19:09:37.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe911e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c4430 con 0x7fe90c1da030 2024-03-28T19:09:37.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe911e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c06d290 con 0x7fe8f0004800 2024-03-28T19:09:37.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe90a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe90c1d1a90 0x7fe90c1d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:37.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe90a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe90c1d1a90 0x7fe90c1d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33418/0 (socket says 172.21.15.27:33418) 2024-03-28T19:09:37.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe90a000640 1 -- 172.21.15.27:0/1033100530 learned_addr learned my addr 172.21.15.27:0/1033100530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:37.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe90b400640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe90c1da030 0x7fe90c1e7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:37.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.201+0000 7fe90aa00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe8f0004800 0x7fe90c1dbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.202+0000 7fe90a000640 1 -- 172.21.15.27:0/1033100530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe8f0004800 msgr2=0x7fe90c1dbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.202+0000 7fe90a000640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe8f0004800 0x7fe90c1dbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.202+0000 7fe90a000640 1 -- 172.21.15.27:0/1033100530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe90c1da030 msgr2=0x7fe90c1e7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.202+0000 7fe90a000640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe90c1da030 0x7fe90c1e7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.202+0000 7fe90a000640 1 -- 172.21.15.27:0/1033100530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c1eb690 con 0x7fe90c1d1a90 2024-03-28T19:09:37.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.202+0000 7fe90a000640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe90c1d1a90 0x7fe90c1d0ed0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800c300 tx=0x7fe8f8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:37.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.202+0000 7fe903400640 1 -- 172.21.15.27:0/1033100530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f80049e0 con 0x7fe90c1d1a90 2024-03-28T19:09:37.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.203+0000 7fe903400640 1 -- 172.21.15.27:0/1033100530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f8043030 con 0x7fe90c1d1a90 2024-03-28T19:09:37.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.203+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c1eb970 con 0x7fe90c1d1a90 2024-03-28T19:09:37.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.203+0000 7fe903400640 1 -- 172.21.15.27:0/1033100530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8004dc0 con 0x7fe90c1d1a90 2024-03-28T19:09:37.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.203+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c1d0ba0 con 0x7fe90c1d1a90 2024-03-28T19:09:37.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.204+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c04f400 con 0x7fe90c1d1a90 2024-03-28T19:09:37.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.205+0000 7fe903400640 1 -- 172.21.15.27:0/1033100530 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe8f8007070 con 0x7fe90c1d1a90 2024-03-28T19:09:37.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.206+0000 7fe903400640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8ec06bc00 0x7fe8ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:37.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.207+0000 7fe903400640 1 -- 172.21.15.27:0/1033100530 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7fe8f80b0c00 con 0x7fe90c1d1a90 2024-03-28T19:09:37.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.207+0000 7fe90aa00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8ec06bc00 0x7fe8ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:37.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.208+0000 7fe90aa00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8ec06bc00 0x7fe8ec06e0c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0a5f30 tx=0x7fe8fc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:37.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.211+0000 7fe903400640 1 -- 172.21.15.27:0/1033100530 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe8f80b7080 con 0x7fe90c1d1a90 2024-03-28T19:09:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.524+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe90c0c63b0 con 0x7fe90c1d1a90 2024-03-28T19:09:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.526+0000 7fe903400640 1 -- 172.21.15.27:0/1033100530 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fe8f804d020 con 0x7fe90c1d1a90 2024-03-28T19:09:37.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:37.525 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8ec06bc00 msgr2=0x7fe8ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8ec06bc00 0x7fe8ec06e0c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0a5f30 tx=0x7fe8fc009040 comp rx=0 tx=0).stop 2024-03-28T19:09:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe90c1d1a90 msgr2=0x7fe90c1d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe90c1d1a90 0x7fe90c1d0ed0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800c300 tx=0x7fe8f8001a40 comp rx=0 tx=0).stop 2024-03-28T19:09:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 shutdown_connections 2024-03-28T19:09:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe8ec06bc00 0x7fe8ec06e0c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe90c1da030 0x7fe90c1e7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe90c1d1a90 0x7fe90c1d0ed0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 --2- 172.21.15.27:0/1033100530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe8f0004800 0x7fe90c1dbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 >> 172.21.15.27:0/1033100530 conn(0x7fe90c06b9d0 msgr2=0x7fe90c070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 shutdown_connections 2024-03-28T19:09:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:37.529+0000 7fe911e00640 1 -- 172.21.15.27:0/1033100530 wait complete. 2024-03-28T19:09:37.550 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:09:38.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:38.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.613+0000 7fa309400640 1 Processor -- start 2024-03-28T19:09:38.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.614+0000 7fa309400640 1 -- start start 2024-03-28T19:09:38.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.615+0000 7fa309400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3040a7060 0x7fa3040c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:38.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.615+0000 7fa309400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa304155700 0x7fa3040c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:38.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.615+0000 7fa309400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa304152ad0 0x7fa30406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.615+0000 7fa309400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa304072d40 con 0x7fa3040a7060 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.615+0000 7fa309400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa304073550 con 0x7fa304152ad0 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.615+0000 7fa309400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040c42d0 con 0x7fa304155700 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3040a7060 0x7fa3040c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3040a7060 0x7fa3040c2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33428/0 (socket says 172.21.15.27:33428) 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303e00640 1 -- 172.21.15.27:0/964646240 learned_addr learned my addr 172.21.15.27:0/964646240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa308a00640 1 --2- 172.21.15.27:0/964646240 >> v2:172.21.15.155:3300/0 conn(0x7fa304152ad0 0x7fa30406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303400640 1 --2- 172.21.15.27:0/964646240 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa304155700 0x7fa3040c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303e00640 1 -- 172.21.15.27:0/964646240 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa304155700 msgr2=0x7fa3040c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303e00640 1 --2- 172.21.15.27:0/964646240 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa304155700 0x7fa3040c3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303e00640 1 -- 172.21.15.27:0/964646240 >> v2:172.21.15.155:3300/0 conn(0x7fa304152ad0 msgr2=0x7fa30406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303e00640 1 --2- 172.21.15.27:0/964646240 >> v2:172.21.15.155:3300/0 conn(0x7fa304152ad0 0x7fa30406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.616+0000 7fa303e00640 1 -- 172.21.15.27:0/964646240 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304072f90 con 0x7fa3040a7060 2024-03-28T19:09:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.618+0000 7fa303e00640 1 --2- 172.21.15.27:0/964646240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3040a7060 0x7fa3040c2530 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa304072c20 tx=0x7fa2fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a293a78d0d6e49b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.618+0000 7fa302a00640 1 -- 172.21.15.27:0/964646240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc02ca70 con 0x7fa3040a7060 2024-03-28T19:09:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.618+0000 7fa302a00640 1 -- 172.21.15.27:0/964646240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc02cc10 con 0x7fa3040a7060 2024-03-28T19:09:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.618+0000 7fa302a00640 1 -- 172.21.15.27:0/964646240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0343f0 con 0x7fa3040a7060 2024-03-28T19:09:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 -- 172.21.15.27:0/964646240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3040a7060 msgr2=0x7fa3040c2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 --2- 172.21.15.27:0/964646240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3040a7060 0x7fa3040c2530 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa304072c20 tx=0x7fa2fc02bed0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 -- 172.21.15.27:0/964646240 shutdown_connections 2024-03-28T19:09:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 --2- 172.21.15.27:0/964646240 >> v2:172.21.15.155:3300/0 conn(0x7fa304152ad0 0x7fa30406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 --2- 172.21.15.27:0/964646240 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa304155700 0x7fa3040c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 --2- 172.21.15.27:0/964646240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3040a7060 0x7fa3040c2530 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 -- 172.21.15.27:0/964646240 >> 172.21.15.27:0/964646240 conn(0x7fa3040c7aa0 msgr2=0x7fa3040c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 -- 172.21.15.27:0/964646240 shutdown_connections 2024-03-28T19:09:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.619+0000 7fa309400640 1 -- 172.21.15.27:0/964646240 wait complete. 2024-03-28T19:09:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.621+0000 7fa309400640 1 Processor -- start 2024-03-28T19:09:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.621+0000 7fa309400640 1 -- start start 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.622+0000 7fa309400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3040a7060 0x7fa304150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.622+0000 7fa309400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304152ad0 0x7fa30406fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.622+0000 7fa309400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa304155700 0x7fa3040ab4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.622+0000 7fa309400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3041525b0 con 0x7fa304152ad0 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.622+0000 7fa309400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3041522d0 con 0x7fa304155700 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.622+0000 7fa309400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa304155b60 con 0x7fa3040a7060 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.622+0000 7fa303400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304152ad0 0x7fa30406fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.622+0000 7fa303400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304152ad0 0x7fa30406fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33434/0 (socket says 172.21.15.27:33434) 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa303400640 1 -- 172.21.15.27:0/3056672665 learned_addr learned my addr 172.21.15.27:0/3056672665 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa303e00640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3040a7060 0x7fa304150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa308a00640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa304155700 0x7fa3040ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:38.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa303400640 1 -- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3040a7060 msgr2=0x7fa304150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:38.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa303400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3040a7060 0x7fa304150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa303400640 1 -- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa304155700 msgr2=0x7fa3040ab4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa303400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa304155700 0x7fa3040ab4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa303400640 1 -- 172.21.15.27:0/3056672665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3041553a0 con 0x7fa304152ad0 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa308a00640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa304155700 0x7fa3040ab4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.623+0000 7fa303400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304152ad0 0x7fa30406fa90 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400d600 tx=0x7fa2f400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.624+0000 7fa2fbe00640 1 -- 172.21.15.27:0/3056672665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4004240 con 0x7fa304152ad0 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.624+0000 7fa2fbe00640 1 -- 172.21.15.27:0/3056672665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f40043e0 con 0x7fa304152ad0 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.624+0000 7fa2fbe00640 1 -- 172.21.15.27:0/3056672665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4003830 con 0x7fa304152ad0 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.624+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3041550c0 con 0x7fa304152ad0 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.624+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa304150320 con 0x7fa304152ad0 2024-03-28T19:09:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.625+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa304069640 con 0x7fa304152ad0 2024-03-28T19:09:38.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.626+0000 7fa2fbe00640 1 -- 172.21.15.27:0/3056672665 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa2f40039d0 con 0x7fa304152ad0 2024-03-28T19:09:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.627+0000 7fa2fbe00640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2e406bbd0 0x7fa2e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:38.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.627+0000 7fa2fbe00640 1 -- 172.21.15.27:0/3056672665 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7fa2f408b7c0 con 0x7fa304152ad0 2024-03-28T19:09:38.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.628+0000 7fa303e00640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2e406bbd0 0x7fa2e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:38.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.629+0000 7fa303e00640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2e406bbd0 0x7fa2e406e090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc035000 tx=0x7fa2fc030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:38.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.634+0000 7fa2fbe00640 1 -- 172.21.15.27:0/3056672665 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa2f401c360 con 0x7fa304152ad0 2024-03-28T19:09:38.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.922+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3040ad030 con 0x7fa304152ad0 2024-03-28T19:09:38.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.923+0000 7fa2fbe00640 1 -- 172.21.15.27:0/3056672665 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fa2f4059850 con 0x7fa304152ad0 2024-03-28T19:09:38.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:38.923 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:38.923 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2e406bbd0 msgr2=0x7fa2e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2e406bbd0 0x7fa2e406e090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc035000 tx=0x7fa2fc030000 comp rx=0 tx=0).stop 2024-03-28T19:09:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304152ad0 msgr2=0x7fa30406fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304152ad0 0x7fa30406fa90 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400d600 tx=0x7fa2f400dac0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 shutdown_connections 2024-03-28T19:09:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa2e406bbd0 0x7fa2e406e090 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa304155700 0x7fa3040ab4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304152ad0 0x7fa30406fa90 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 --2- 172.21.15.27:0/3056672665 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3040a7060 0x7fa304150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:38.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.926+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 >> 172.21.15.27:0/3056672665 conn(0x7fa3040c7aa0 msgr2=0x7fa3040c6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:38.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.927+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 shutdown_connections 2024-03-28T19:09:38.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:38.927+0000 7fa309400640 1 -- 172.21.15.27:0/3056672665 wait complete. 2024-03-28T19:09:38.948 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:09:39.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:39.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.781+0000 7f5b2ea00640 1 Processor -- start 2024-03-28T19:09:39.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.781+0000 7f5b2ea00640 1 -- start start 2024-03-28T19:09:39.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.782+0000 7f5b2ea00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5b280a9cb0 0x7f5b280c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:39.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.782+0000 7f5b2ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b281534c0 0x7f5b28153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:39.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.782+0000 7f5b2ea00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b280c3d90 con 0x7f5b28150c70 2024-03-28T19:09:39.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.782+0000 7f5b2ea00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b280ab4e0 con 0x7f5b280a9cb0 2024-03-28T19:09:39.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.782+0000 7f5b2ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280ac1f0 con 0x7f5b281534c0 2024-03-28T19:09:39.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.783+0000 7f5b2ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b281534c0 0x7f5b28153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:39.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.783+0000 7f5b2ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b281534c0 0x7f5b28153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:35678/0 (socket says 172.21.15.27:35678) 2024-03-28T19:09:39.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.783+0000 7f5b2ca00640 1 -- 172.21.15.27:0/1151613233 learned_addr learned my addr 172.21.15.27:0/1151613233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:39.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.783+0000 7f5b1fe00640 1 --2- 172.21.15.27:0/1151613233 >> v2:172.21.15.155:3300/0 conn(0x7f5b280a9cb0 0x7f5b280c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:39.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.784+0000 7f5b2ca00640 1 -- 172.21.15.27:0/1151613233 >> v2:172.21.15.155:3300/0 conn(0x7f5b280a9cb0 msgr2=0x7f5b280c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:39.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.784+0000 7f5b2ca00640 1 --2- 172.21.15.27:0/1151613233 >> v2:172.21.15.155:3300/0 conn(0x7f5b280a9cb0 0x7f5b280c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:39.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.784+0000 7f5b2ca00640 1 -- 172.21.15.27:0/1151613233 >> v1:172.21.15.155:6789/0 conn(0x7f5b28150c70 legacy=0x7f5b280c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:39.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.784+0000 7f5b2ca00640 1 -- 172.21.15.27:0/1151613233 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b280ab710 con 0x7f5b281534c0 2024-03-28T19:09:39.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.785+0000 7f5b2ca00640 1 --2- 172.21.15.27:0/1151613233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b281534c0 0x7f5b28153950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009e50 tx=0x7f5b2002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9c820b05cbd6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:39.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.786+0000 7f5b27400640 1 -- 172.21.15.27:0/1151613233 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2002ca70 con 0x7f5b281534c0 2024-03-28T19:09:39.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.786+0000 7f5b27400640 1 -- 172.21.15.27:0/1151613233 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2002cc10 con 0x7f5b281534c0 2024-03-28T19:09:39.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.786+0000 7f5b27400640 1 -- 172.21.15.27:0/1151613233 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b200343f0 con 0x7f5b281534c0 2024-03-28T19:09:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.787+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1151613233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b281534c0 msgr2=0x7f5b28153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.787+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1151613233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b281534c0 0x7f5b28153950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009e50 tx=0x7f5b2002bed0 comp rx=0 tx=0).stop 2024-03-28T19:09:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.787+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1151613233 shutdown_connections 2024-03-28T19:09:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.787+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1151613233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b281534c0 0x7f5b28153950 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.787+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1151613233 >> v2:172.21.15.155:3300/0 conn(0x7f5b280a9cb0 0x7f5b280c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.787+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1151613233 >> 172.21.15.27:0/1151613233 conn(0x7f5b2806b9d0 msgr2=0x7f5b280c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.787+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1151613233 shutdown_connections 2024-03-28T19:09:39.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.787+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1151613233 wait complete. 2024-03-28T19:09:39.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.789+0000 7f5b2ea00640 1 Processor -- start 2024-03-28T19:09:39.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.789+0000 7f5b2ea00640 1 -- start start 2024-03-28T19:09:39.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.790+0000 7f5b2ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b280a9cb0 0x7f5b280a8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.790+0000 7f5b2ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b28150c70 0x7f5b2806e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.790+0000 7f5b2ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b281534c0 0x7f5b28146e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.790+0000 7f5b2ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280c1810 con 0x7f5b280a9cb0 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.790+0000 7f5b2ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280abcd0 con 0x7f5b281534c0 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.790+0000 7f5b2ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280c6830 con 0x7f5b28150c70 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.790+0000 7f5b27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b280a9cb0 0x7f5b280a8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b280a9cb0 0x7f5b280a8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33446/0 (socket says 172.21.15.27:33446) 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b27e00640 1 -- 172.21.15.27:0/1354952752 learned_addr learned my addr 172.21.15.27:0/1354952752 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b1fe00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b28150c70 0x7f5b2806e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:39.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b2ca00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b281534c0 0x7f5b28146e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b27e00640 1 -- 172.21.15.27:0/1354952752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b28150c70 msgr2=0x7f5b2806e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b27e00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b28150c70 0x7f5b2806e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b27e00640 1 -- 172.21.15.27:0/1354952752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b281534c0 msgr2=0x7f5b28146e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b27e00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b281534c0 0x7f5b28146e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b27e00640 1 -- 172.21.15.27:0/1354952752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b280a95d0 con 0x7f5b280a9cb0 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.791+0000 7f5b27e00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b280a9cb0 0x7f5b280a8500 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800c400 tx=0x7f5b1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.792+0000 7f5b24c00640 1 -- 172.21.15.27:0/1354952752 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180052d0 con 0x7f5b280a9cb0 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.792+0000 7f5b24c00640 1 -- 172.21.15.27:0/1354952752 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18005490 con 0x7f5b280a9cb0 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.792+0000 7f5b24c00640 1 -- 172.21.15.27:0/1354952752 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18007260 con 0x7f5b280a9cb0 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.792+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b28156090 con 0x7f5b280a9cb0 2024-03-28T19:09:39.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.792+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b280a92f0 con 0x7f5b280a9cb0 2024-03-28T19:09:39.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.793+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2804f400 con 0x7f5b280a9cb0 2024-03-28T19:09:39.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.794+0000 7f5b24c00640 1 -- 172.21.15.27:0/1354952752 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b18007400 con 0x7f5b280a9cb0 2024-03-28T19:09:39.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.795+0000 7f5b24c00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b0406bbd0 0x7f5b0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:39.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.795+0000 7f5b24c00640 1 -- 172.21.15.27:0/1354952752 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f5b1808a9b0 con 0x7f5b280a9cb0 2024-03-28T19:09:39.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.796+0000 7f5b1fe00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b0406bbd0 0x7f5b0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:39.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.796+0000 7f5b1fe00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b0406bbd0 0x7f5b0406e090 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5b280c0af0 tx=0x7f5b10008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:39.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:39.801+0000 7f5b24c00640 1 -- 172.21.15.27:0/1354952752 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b18058ad0 con 0x7f5b280a9cb0 2024-03-28T19:09:40.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.066+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b280c5f00 con 0x7f5b280a9cb0 2024-03-28T19:09:40.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.067+0000 7f5b24c00640 1 -- 172.21.15.27:0/1354952752 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f5b180583d0 con 0x7f5b280a9cb0 2024-03-28T19:09:40.067 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:40.067 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:40.067 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b0406bbd0 msgr2=0x7f5b0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b0406bbd0 0x7f5b0406e090 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5b280c0af0 tx=0x7f5b10008040 comp rx=0 tx=0).stop 2024-03-28T19:09:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b280a9cb0 msgr2=0x7f5b280a8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b280a9cb0 0x7f5b280a8500 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800c400 tx=0x7f5b1800c8c0 comp rx=0 tx=0).stop 2024-03-28T19:09:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 shutdown_connections 2024-03-28T19:09:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b0406bbd0 0x7f5b0406e090 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b281534c0 0x7f5b28146e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b28150c70 0x7f5b2806e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 --2- 172.21.15.27:0/1354952752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b280a9cb0 0x7f5b280a8500 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 >> 172.21.15.27:0/1354952752 conn(0x7f5b2806b9d0 msgr2=0x7f5b280c2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 shutdown_connections 2024-03-28T19:09:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:40.070+0000 7f5b2ea00640 1 -- 172.21.15.27:0/1354952752 wait complete. 2024-03-28T19:09:40.091 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:09:40.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:41.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.172+0000 7fcbb8c00640 1 Processor -- start 2024-03-28T19:09:41.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.172+0000 7fcbb8c00640 1 -- start start 2024-03-28T19:09:41.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.174+0000 7fcbb8c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcbb414fb10 0x7fcbb40c2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:41.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.174+0000 7fcbb8c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcbb414fef0 0x7fcbb406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.174+0000 7fcbb8c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb406c610 0x7fcbb41529f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.174+0000 7fcbb8c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb4152560 con 0x7fcbb414fef0 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.174+0000 7fcbb8c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb40c78e0 con 0x7fcbb414fb10 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.174+0000 7fcbb8c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40c3e80 con 0x7fcbb406c610 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.174+0000 7fcba9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcbb414fef0 0x7fcbb406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.174+0000 7fcba9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcbb414fef0 0x7fcbb406c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33452/0 (socket says 172.21.15.27:33452) 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.175+0000 7fcba9400640 1 -- 172.21.15.27:0/357146367 learned_addr learned my addr 172.21.15.27:0/357146367 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.175+0000 7fcbb2800640 1 --2- 172.21.15.27:0/357146367 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb406c610 0x7fcbb41529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.175+0000 7fcbb1e00640 1 --2- 172.21.15.27:0/357146367 >> v2:172.21.15.155:3300/0 conn(0x7fcbb414fb10 0x7fcbb40c2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.175+0000 7fcba9400640 1 -- 172.21.15.27:0/357146367 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb406c610 msgr2=0x7fcbb41529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.175+0000 7fcba9400640 1 --2- 172.21.15.27:0/357146367 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb406c610 0x7fcbb41529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.175+0000 7fcba9400640 1 -- 172.21.15.27:0/357146367 >> v2:172.21.15.155:3300/0 conn(0x7fcbb414fb10 msgr2=0x7fcbb40c2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:41.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.175+0000 7fcba9400640 1 --2- 172.21.15.27:0/357146367 >> v2:172.21.15.155:3300/0 conn(0x7fcbb414fb10 0x7fcbb40c2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.175+0000 7fcba9400640 1 -- 172.21.15.27:0/357146367 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40c4160 con 0x7fcbb414fef0 2024-03-28T19:09:41.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.176+0000 7fcba9400640 1 --2- 172.21.15.27:0/357146367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb414fef0 0x7fcbb406c0d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40c5dd0 tx=0x7fcb9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5ea7090ed95415 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:41.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.177+0000 7fcbb1400640 1 -- 172.21.15.27:0/357146367 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c035020 con 0x7fcbb414fef0 2024-03-28T19:09:41.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.177+0000 7fcbb1400640 1 -- 172.21.15.27:0/357146367 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c031870 con 0x7fcbb414fef0 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.177+0000 7fcbb1400640 1 -- 172.21.15.27:0/357146367 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c031c40 con 0x7fcbb414fef0 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.177+0000 7fcbb8c00640 1 -- 172.21.15.27:0/357146367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb414fef0 msgr2=0x7fcbb406c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.177+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/357146367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb414fef0 0x7fcbb406c0d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40c5dd0 tx=0x7fcb9c02b910 comp rx=0 tx=0).stop 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.178+0000 7fcbb8c00640 1 -- 172.21.15.27:0/357146367 shutdown_connections 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.178+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/357146367 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb406c610 0x7fcbb41529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.178+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/357146367 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb414fef0 0x7fcbb406c0d0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.178+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/357146367 >> v2:172.21.15.155:3300/0 conn(0x7fcbb414fb10 0x7fcbb40c2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.178+0000 7fcbb8c00640 1 -- 172.21.15.27:0/357146367 >> 172.21.15.27:0/357146367 conn(0x7fcbb40c4750 msgr2=0x7fcbb40c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.178+0000 7fcbb8c00640 1 -- 172.21.15.27:0/357146367 shutdown_connections 2024-03-28T19:09:41.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.178+0000 7fcbb8c00640 1 -- 172.21.15.27:0/357146367 wait complete. 2024-03-28T19:09:41.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.180+0000 7fcbb8c00640 1 Processor -- start 2024-03-28T19:09:41.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.180+0000 7fcbb8c00640 1 -- start start 2024-03-28T19:09:41.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb406c610 0x7fcbb40ab6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:41.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb8c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbb414fb10 0x7fcbb415de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:41.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb8c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb414fef0 0x7fcbb4146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb8c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb4071a00 con 0x7fcbb406c610 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb8c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40c7570 con 0x7fcbb414fb10 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb8c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb4071b80 con 0x7fcbb414fef0 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb406c610 0x7fcbb40ab6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb406c610 0x7fcbb40ab6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33468/0 (socket says 172.21.15.27:33468) 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.181+0000 7fcbb1e00640 1 -- 172.21.15.27:0/1948568000 learned_addr learned my addr 172.21.15.27:0/1948568000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcba9400640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbb414fb10 0x7fcbb415de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcbb2800640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb414fef0 0x7fcbb4146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcbb1e00640 1 -- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb414fef0 msgr2=0x7fcbb4146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcbb1e00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb414fef0 0x7fcbb4146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcbb1e00640 1 -- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbb414fb10 msgr2=0x7fcbb415de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcbb1e00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbb414fb10 0x7fcbb415de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcbb1e00640 1 -- 172.21.15.27:0/1948568000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40abc10 con 0x7fcbb406c610 2024-03-28T19:09:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcbb2800640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb414fef0 0x7fcbb4146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:09:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.182+0000 7fcbb1e00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb406c610 0x7fcbb40ab6d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcba400d800 tx=0x7fcba400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.183+0000 7fcbaaa00640 1 -- 172.21.15.27:0/1948568000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba400b6c0 con 0x7fcbb406c610 2024-03-28T19:09:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.183+0000 7fcbaaa00640 1 -- 172.21.15.27:0/1948568000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba4019070 con 0x7fcbb406c610 2024-03-28T19:09:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.183+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb40ace60 con 0x7fcbb406c610 2024-03-28T19:09:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.183+0000 7fcbaaa00640 1 -- 172.21.15.27:0/1948568000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba400ba80 con 0x7fcbb406c610 2024-03-28T19:09:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.183+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb40ac3b0 con 0x7fcbb406c610 2024-03-28T19:09:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.184+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb404f400 con 0x7fcbb406c610 2024-03-28T19:09:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.185+0000 7fcbaaa00640 1 -- 172.21.15.27:0/1948568000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcba400bc20 con 0x7fcbb406c610 2024-03-28T19:09:41.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.187+0000 7fcbaaa00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb9006bbd0 0x7fcb9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:41.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.187+0000 7fcbaaa00640 1 -- 172.21.15.27:0/1948568000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7fcba408b910 con 0x7fcbb406c610 2024-03-28T19:09:41.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.187+0000 7fcba9400640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb9006bbd0 0x7fcb9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:41.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.188+0000 7fcba9400640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb9006bbd0 0x7fcb9006e090 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c036000 tx=0x7fcb9c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:41.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.192+0000 7fcbaaa00640 1 -- 172.21.15.27:0/1948568000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcbb40abc10 con 0x7fcbb406c610 2024-03-28T19:09:41.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.464+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbb40c1fc0 con 0x7fcbb406c610 2024-03-28T19:09:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.465+0000 7fcbaaa00640 1 -- 172.21.15.27:0/1948568000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fcba4059a30 con 0x7fcbb406c610 2024-03-28T19:09:41.465 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:41.465 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:41.466 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:41.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb9006bbd0 msgr2=0x7fcb9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:41.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb9006bbd0 0x7fcb9006e090 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c036000 tx=0x7fcb9c03c040 comp rx=0 tx=0).stop 2024-03-28T19:09:41.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb406c610 msgr2=0x7fcbb40ab6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:41.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb406c610 0x7fcbb40ab6d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcba400d800 tx=0x7fcba400dcc0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 shutdown_connections 2024-03-28T19:09:41.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb9006bbd0 0x7fcb9006e090 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcbb414fef0 0x7fcbb4146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbb414fb10 0x7fcbb415de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 --2- 172.21.15.27:0/1948568000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcbb406c610 0x7fcbb40ab6d0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 >> 172.21.15.27:0/1948568000 conn(0x7fcbb40c4750 msgr2=0x7fcbb406b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 shutdown_connections 2024-03-28T19:09:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:41.469+0000 7fcbb8c00640 1 -- 172.21.15.27:0/1948568000 wait complete. 2024-03-28T19:09:41.491 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:09:42.092 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:42.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.313+0000 7f858b400640 1 Processor -- start 2024-03-28T19:09:42.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.313+0000 7f858b400640 1 -- start start 2024-03-28T19:09:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.314+0000 7f858b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8584152bd0 0x7f858406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.315+0000 7f858b400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85840c52e0 con 0x7f85840c4dc0 2024-03-28T19:09:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.315+0000 7f858b400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85841557e0 con 0x7f858406f1e0 2024-03-28T19:09:42.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.315+0000 7f858b400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8584155ac0 con 0x7f8584152bd0 2024-03-28T19:09:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.315+0000 7f8588a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8584152bd0 0x7f858406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.315+0000 7f8588a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8584152bd0 0x7f858406eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33484/0 (socket says 172.21.15.27:33484) 2024-03-28T19:09:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.315+0000 7f8588a00640 1 -- 172.21.15.27:0/3838624069 learned_addr learned my addr 172.21.15.27:0/3838624069 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.316+0000 7f8588a00640 1 -- 172.21.15.27:0/3838624069 >> v1:172.21.15.155:6789/0 conn(0x7f858406f1e0 legacy=0x7f8584070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.316+0000 7f8588a00640 1 -- 172.21.15.27:0/3838624069 >> v1:172.21.15.27:6789/0 conn(0x7f85840c4dc0 legacy=0x7f858415ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:42.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.316+0000 7f8588a00640 1 -- 172.21.15.27:0/3838624069 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8584155da0 con 0x7f8584152bd0 2024-03-28T19:09:42.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.317+0000 7f8588a00640 1 --2- 172.21.15.27:0/3838624069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584152bd0 0x7f858406eca0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8578009870 tx=0x7f857802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35eba01d2938fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:42.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.318+0000 7f8583400640 1 -- 172.21.15.27:0/3838624069 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857802f850 con 0x7f8584152bd0 2024-03-28T19:09:42.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.318+0000 7f8583400640 1 -- 172.21.15.27:0/3838624069 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857802f9f0 con 0x7f8584152bd0 2024-03-28T19:09:42.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.318+0000 7f8583400640 1 -- 172.21.15.27:0/3838624069 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857802fdc0 con 0x7f8584152bd0 2024-03-28T19:09:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.318+0000 7f858b400640 1 -- 172.21.15.27:0/3838624069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584152bd0 msgr2=0x7f858406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.318+0000 7f858b400640 1 --2- 172.21.15.27:0/3838624069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584152bd0 0x7f858406eca0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8578009870 tx=0x7f857802ba30 comp rx=0 tx=0).stop 2024-03-28T19:09:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.318+0000 7f858b400640 1 -- 172.21.15.27:0/3838624069 shutdown_connections 2024-03-28T19:09:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.318+0000 7f858b400640 1 --2- 172.21.15.27:0/3838624069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8584152bd0 0x7f858406eca0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.318+0000 7f858b400640 1 -- 172.21.15.27:0/3838624069 >> 172.21.15.27:0/3838624069 conn(0x7f8584066dd0 msgr2=0x7f85840671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.319+0000 7f858b400640 1 -- 172.21.15.27:0/3838624069 shutdown_connections 2024-03-28T19:09:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.319+0000 7f858b400640 1 -- 172.21.15.27:0/3838624069 wait complete. 2024-03-28T19:09:42.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.320+0000 7f858b400640 1 Processor -- start 2024-03-28T19:09:42.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.320+0000 7f858b400640 1 -- start start 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f858b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f858406f1e0 0x7f8584154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f858b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85840c4dc0 0x7f85840c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f858b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8584152bd0 0x7f85840c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f858b400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8584070dd0 con 0x7f858406f1e0 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f858b400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8584070c50 con 0x7f85840c4dc0 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f858b400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85840c6c70 con 0x7f8584152bd0 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8588a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f858406f1e0 0x7f8584154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8588a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f858406f1e0 0x7f8584154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33498/0 (socket says 172.21.15.27:33498) 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8588a00640 1 -- 172.21.15.27:0/2535564548 learned_addr learned my addr 172.21.15.27:0/2535564548 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8583e00640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85840c4dc0 0x7f85840c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8589400640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8584152bd0 0x7f85840c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8588a00640 1 -- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8584152bd0 msgr2=0x7f85840c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8588a00640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8584152bd0 0x7f85840c7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8588a00640 1 -- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85840c4dc0 msgr2=0x7f85840c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8588a00640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85840c4dc0 0x7f85840c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8588a00640 1 -- 172.21.15.27:0/2535564548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8584153a20 con 0x7f858406f1e0 2024-03-28T19:09:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.322+0000 7f8583e00640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85840c4dc0 0x7f85840c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:09:42.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.323+0000 7f8588a00640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f858406f1e0 0x7f8584154fc0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f857802ba70 tx=0x7f8578004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:42.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.323+0000 7f8580c00640 1 -- 172.21.15.27:0/2535564548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8578047070 con 0x7f858406f1e0 2024-03-28T19:09:42.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.323+0000 7f8580c00640 1 -- 172.21.15.27:0/2535564548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857802f840 con 0x7f858406f1e0 2024-03-28T19:09:42.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.323+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8584153d00 con 0x7f858406f1e0 2024-03-28T19:09:42.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.323+0000 7f8580c00640 1 -- 172.21.15.27:0/2535564548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85780426c0 con 0x7f858406f1e0 2024-03-28T19:09:42.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.323+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85841542c0 con 0x7f858406f1e0 2024-03-28T19:09:42.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.325+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85840ae380 con 0x7f858406f1e0 2024-03-28T19:09:42.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.325+0000 7f8580c00640 1 -- 172.21.15.27:0/2535564548 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f857804a070 con 0x7f858406f1e0 2024-03-28T19:09:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.326+0000 7f8580c00640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f856406bbd0 0x7f856406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.327+0000 7f8580c00640 1 -- 172.21.15.27:0/2535564548 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f85780ad960 con 0x7f858406f1e0 2024-03-28T19:09:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.327+0000 7f8583e00640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f856406bbd0 0x7f856406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:42.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.328+0000 7f8583e00640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f856406bbd0 0x7f856406e090 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f85840c9280 tx=0x7f8570009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:42.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.332+0000 7f8580c00640 1 -- 172.21.15.27:0/2535564548 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85840ae380 con 0x7f858406f1e0 2024-03-28T19:09:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.624+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f858404f400 con 0x7f858406f1e0 2024-03-28T19:09:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.626+0000 7f8580c00640 1 -- 172.21.15.27:0/2535564548 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f8578033030 con 0x7f858406f1e0 2024-03-28T19:09:42.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:42.626 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:42.626 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.629+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f856406bbd0 msgr2=0x7f856406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.629+0000 7f858b400640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f856406bbd0 0x7f856406e090 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f85840c9280 tx=0x7f8570009040 comp rx=0 tx=0).stop 2024-03-28T19:09:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.629+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f858406f1e0 msgr2=0x7f8584154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:42.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.629+0000 7f858b400640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f858406f1e0 0x7f8584154fc0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f857802ba70 tx=0x7f8578004730 comp rx=0 tx=0).stop 2024-03-28T19:09:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.630+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 shutdown_connections 2024-03-28T19:09:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.630+0000 7f858b400640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f856406bbd0 0x7f856406e090 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.630+0000 7f858b400640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8584152bd0 0x7f85840c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.630+0000 7f858b400640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85840c4dc0 0x7f85840c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.630+0000 7f858b400640 1 --2- 172.21.15.27:0/2535564548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f858406f1e0 0x7f8584154fc0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.630+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 >> 172.21.15.27:0/2535564548 conn(0x7f8584066dd0 msgr2=0x7f858406ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.630+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 shutdown_connections 2024-03-28T19:09:42.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:42.630+0000 7f858b400640 1 -- 172.21.15.27:0/2535564548 wait complete. 2024-03-28T19:09:42.651 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:09:43.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.729+0000 7f3576c00640 1 Processor -- start 2024-03-28T19:09:43.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.729+0000 7f3576c00640 1 -- start start 2024-03-28T19:09:43.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.730+0000 7f3576c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c58c0 0x7f35700c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.731+0000 7f3576c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f35700c8ed0 0x7f35701555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.731+0000 7f3576c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3570156230 con 0x7f35700be5d0 2024-03-28T19:09:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.731+0000 7f3576c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3570158bd0 con 0x7f35700c8ed0 2024-03-28T19:09:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.731+0000 7f3576c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3570158eb0 con 0x7f35700c58c0 2024-03-28T19:09:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.731+0000 7f3576200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f35700be5d0 0x7f3570155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56332/0 (socket says 172.21.15.27:56332) 2024-03-28T19:09:43.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.731+0000 7f3576200640 1 -- 172.21.15.27:0/2780656853 learned_addr learned my addr 172.21.15.27:0/2780656853 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:09:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.731+0000 7f3575800640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c58c0 0x7f35700c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.731+0000 7f3574e00640 1 --2- 172.21.15.27:0/2780656853 >> v2:172.21.15.155:3300/0 conn(0x7f35700c8ed0 0x7f35701555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427128035 0 0) 0x7f3570156230 con 0x7f35700be5d0 2024-03-28T19:09:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3558003520 con 0x7f35700be5d0 2024-03-28T19:09:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2814645035 0 0) 0x7f3558003520 con 0x7f35700be5d0 2024-03-28T19:09:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c58c0 msgr2=0x7f35700c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:43.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c58c0 0x7f35700c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 >> v2:172.21.15.155:3300/0 conn(0x7f35700c8ed0 msgr2=0x7f35701555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 --2- 172.21.15.27:0/2780656853 >> v2:172.21.15.155:3300/0 conn(0x7f35700c8ed0 0x7f35701555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570159190 con 0x7f35700be5d0 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f35680019f0 con 0x7f35700be5d0 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 >> v1:172.21.15.27:6789/0 conn(0x7f35700be5d0 legacy=0x7f3570155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580039b0 0x7f3558003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3558004800 0x7f3558004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c58c0 0x7f35580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570158bd0 con 0x7f35580039b0 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570158eb0 con 0x7f3558004800 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.732+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355800ba10 con 0x7f35700c58c0 2024-03-28T19:09:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3575800640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580039b0 0x7f3558003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3574e00640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3558004800 0x7f3558004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3576200640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c58c0 0x7f35580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3575800640 1 -- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c58c0 msgr2=0x7f35580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3575800640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c58c0 0x7f35580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3575800640 1 -- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3558004800 msgr2=0x7f3558004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3575800640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3558004800 0x7f3558004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3575800640 1 -- 172.21.15.27:0/2780656853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3558003520 con 0x7f35580039b0 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.733+0000 7f3574e00640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3558004800 0x7f3558004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.734+0000 7f3575800640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580039b0 0x7f3558003dd0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f3564009af0 tx=0x7f3558005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178d1431001d42f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.735+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564039070 con 0x7f35580039b0 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.735+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356402c6b0 con 0x7f35580039b0 2024-03-28T19:09:43.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.735+0000 7f356fe00640 1 -- 172.21.15.27:0/2780656853 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564009690 con 0x7f35580039b0 2024-03-28T19:09:43.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.735+0000 7f3576c00640 1 -- 172.21.15.27:0/2780656853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580039b0 msgr2=0x7f3558003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:43.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.735+0000 7f3576c00640 1 --2- 172.21.15.27:0/2780656853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580039b0 0x7f3558003dd0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f3564009af0 tx=0x7f3558005830 comp rx=0 tx=0).stop 2024-03-28T19:09:43.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.735+0000 7f3576200640 1 -- 172.21.15.27:0/2780656853 reap_dead start 2024-03-28T19:09:43.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.735+0000 7f3576c00640 1 -- 172.21.15.27:0/2780656853 shutdown_connections 2024-03-28T19:09:43.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.735+0000 7f3576c00640 1 -- 172.21.15.27:0/2780656853 >> 172.21.15.27:0/2780656853 conn(0x7f357006b6f0 msgr2=0x7f35700acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:43.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.736+0000 7f3576c00640 1 -- 172.21.15.27:0/2780656853 shutdown_connections 2024-03-28T19:09:43.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.736+0000 7f3576c00640 1 -- 172.21.15.27:0/2780656853 wait complete. 2024-03-28T19:09:43.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.738+0000 7f3576c00640 1 Processor -- start 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.738+0000 7f3576c00640 1 -- start start 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35580039b0 0x7f35700aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c3490 0x7f3570151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35701e12e0 0x7f35701523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35700b0820 con 0x7f35701e12e0 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570159750 con 0x7f35580039b0 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35700b09a0 con 0x7f35700c3490 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35701e12e0 0x7f35701523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35701e12e0 0x7f35701523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33516/0 (socket says 172.21.15.27:33516) 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.739+0000 7f3576200640 1 -- 172.21.15.27:0/2224051955 learned_addr learned my addr 172.21.15.27:0/2224051955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:43.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3574e00640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c3490 0x7f3570151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3575800640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35580039b0 0x7f35700aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3576200640 1 -- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c3490 msgr2=0x7f3570151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3576200640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c3490 0x7f3570151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3576200640 1 -- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35580039b0 msgr2=0x7f35700aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3576200640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35580039b0 0x7f35700aedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3576200640 1 -- 172.21.15.27:0/2224051955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35701e0180 con 0x7f35701e12e0 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3574e00640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c3490 0x7f3570151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f3576200640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35701e12e0 0x7f35701523c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f35680013d0 tx=0x7f356800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:43.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f356d600640 1 -- 172.21.15.27:0/2224051955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356800eb90 con 0x7f35701e12e0 2024-03-28T19:09:43.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.740+0000 7f356d600640 1 -- 172.21.15.27:0/2224051955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356800ed30 con 0x7f35701e12e0 2024-03-28T19:09:43.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.741+0000 7f356d600640 1 -- 172.21.15.27:0/2224051955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568014670 con 0x7f35701e12e0 2024-03-28T19:09:43.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.741+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35701dece0 con 0x7f35701e12e0 2024-03-28T19:09:43.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.741+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35700c29e0 con 0x7f35701e12e0 2024-03-28T19:09:43.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.742+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570069640 con 0x7f35701e12e0 2024-03-28T19:09:43.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.744+0000 7f356d600640 1 -- 172.21.15.27:0/2224051955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3568014810 con 0x7f35701e12e0 2024-03-28T19:09:43.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.745+0000 7f356d600640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f355006bbd0 0x7f355006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:43.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.745+0000 7f356d600640 1 -- 172.21.15.27:0/2224051955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f356808b1c0 con 0x7f35701e12e0 2024-03-28T19:09:43.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.745+0000 7f3575800640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f355006bbd0 0x7f355006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:43.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.746+0000 7f3575800640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f355006bbd0 0x7f355006e090 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3564036000 tx=0x7f3564030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:43.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:43.750+0000 7f356d600640 1 -- 172.21.15.27:0/2224051955 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3568059340 con 0x7f35701e12e0 2024-03-28T19:09:44.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.001+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3570156580 con 0x7f35701e12e0 2024-03-28T19:09:44.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.002+0000 7f356d600640 1 -- 172.21.15.27:0/2224051955 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f356808a090 con 0x7f35701e12e0 2024-03-28T19:09:44.002 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:44.002 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6118,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:08:30.359626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6501},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6501":{"gid":6501,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":14,"addr":"172.21.15.27:6836/1398280334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1398280334},{"type":"v1","addr":"172.21.15.27:6836","nonce":1398280334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6501,"qdb_cluster":[6501]},"id":2}]} 2024-03-28T19:09:44.002 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 112 2024-03-28T19:09:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f355006bbd0 msgr2=0x7f355006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f355006bbd0 0x7f355006e090 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3564036000 tx=0x7f3564030000 comp rx=0 tx=0).stop 2024-03-28T19:09:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35701e12e0 msgr2=0x7f35701523c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35701e12e0 0x7f35701523c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f35680013d0 tx=0x7f356800e530 comp rx=0 tx=0).stop 2024-03-28T19:09:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 shutdown_connections 2024-03-28T19:09:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f355006bbd0 0x7f355006e090 secure :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3564036000 tx=0x7f3564030000 comp rx=0 tx=0).stop 2024-03-28T19:09:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35701e12e0 0x7f35701523c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700c3490 0x7f3570151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 --2- 172.21.15.27:0/2224051955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35580039b0 0x7f35700aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 >> 172.21.15.27:0/2224051955 conn(0x7f357006b6f0 msgr2=0x7f35700716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 shutdown_connections 2024-03-28T19:09:44.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.006+0000 7f3576c00640 1 -- 172.21.15.27:0/2224051955 wait complete. 2024-03-28T19:09:44.028 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:09:44.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.893+0000 7f7738c00640 1 Processor -- start 2024-03-28T19:09:44.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.893+0000 7f7738c00640 1 -- start start 2024-03-28T19:09:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7738c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f77340c8e10 0x7f7734158be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:44.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7738c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77341562f0 0x7f773415ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7738c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77341530f0 con 0x7f7734152d10 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7738c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773406d3d0 con 0x7f77341562f0 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7738c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77340c92f0 con 0x7f77340c8e10 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7732800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77341562f0 0x7f773415ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7732800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77341562f0 0x7f773415ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33520/0 (socket says 172.21.15.27:33520) 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7732800640 1 -- 172.21.15.27:0/1806114435 learned_addr learned my addr 172.21.15.27:0/1806114435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.895+0000 7f7731400640 1 --2- 172.21.15.27:0/1806114435 >> v2:172.21.15.155:3300/0 conn(0x7f77340c8e10 0x7f7734158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.896+0000 7f7732800640 1 -- 172.21.15.27:0/1806114435 >> v2:172.21.15.155:3300/0 conn(0x7f77340c8e10 msgr2=0x7f7734158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:44.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.896+0000 7f7732800640 1 --2- 172.21.15.27:0/1806114435 >> v2:172.21.15.155:3300/0 conn(0x7f77340c8e10 0x7f7734158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.896+0000 7f7732800640 1 -- 172.21.15.27:0/1806114435 >> v1:172.21.15.155:6789/0 conn(0x7f7734152d10 legacy=0x7f77340c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:44.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.896+0000 7f7732800640 1 -- 172.21.15.27:0/1806114435 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7734155810 con 0x7f77341562f0 2024-03-28T19:09:44.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.897+0000 7f7732800640 1 --2- 172.21.15.27:0/1806114435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f773415ff50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f77340c89d0 tx=0x7f772c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c345b07d1b8bd14 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:44.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.897+0000 7f7730a00640 1 -- 172.21.15.27:0/1806114435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c02c960 con 0x7f77341562f0 2024-03-28T19:09:44.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.898+0000 7f7730a00640 1 -- 172.21.15.27:0/1806114435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c02cb00 con 0x7f77341562f0 2024-03-28T19:09:44.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.898+0000 7f7730a00640 1 -- 172.21.15.27:0/1806114435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c0343f0 con 0x7f77341562f0 2024-03-28T19:09:44.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.898+0000 7f7738c00640 1 -- 172.21.15.27:0/1806114435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 msgr2=0x7f773415ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:44.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.898+0000 7f7738c00640 1 --2- 172.21.15.27:0/1806114435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f773415ff50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f77340c89d0 tx=0x7f772c02bdc0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.899+0000 7f7738c00640 1 -- 172.21.15.27:0/1806114435 shutdown_connections 2024-03-28T19:09:44.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.899+0000 7f7738c00640 1 --2- 172.21.15.27:0/1806114435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f773415ff50 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.899+0000 7f7738c00640 1 --2- 172.21.15.27:0/1806114435 >> v2:172.21.15.155:3300/0 conn(0x7f77340c8e10 0x7f7734158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.899+0000 7f7738c00640 1 -- 172.21.15.27:0/1806114435 >> 172.21.15.27:0/1806114435 conn(0x7f7734066fb0 msgr2=0x7f77340673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.899+0000 7f7738c00640 1 -- 172.21.15.27:0/1806114435 shutdown_connections 2024-03-28T19:09:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.899+0000 7f7738c00640 1 -- 172.21.15.27:0/1806114435 wait complete. 2024-03-28T19:09:44.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.901+0000 7f7738c00640 1 Processor -- start 2024-03-28T19:09:44.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.901+0000 7f7738c00640 1 -- start start 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.902+0000 7f7738c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77340c8e10 0x7f773414c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.902+0000 7f7738c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7734152d10 0x7f773414c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.902+0000 7f7738c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f77341ec8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.902+0000 7f7738c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77340c5fa0 con 0x7f77341562f0 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.902+0000 7f7738c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773406d850 con 0x7f7734152d10 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.902+0000 7f7738c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77340c6120 con 0x7f77340c8e10 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7732800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f77341ec8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7732800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f77341ec8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33534/0 (socket says 172.21.15.27:33534) 2024-03-28T19:09:44.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7732800640 1 -- 172.21.15.27:0/2770335820 learned_addr learned my addr 172.21.15.27:0/2770335820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7731400640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7734152d10 0x7f773414c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7731e00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77340c8e10 0x7f773414c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7732800640 1 -- 172.21.15.27:0/2770335820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77340c8e10 msgr2=0x7f773414c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7732800640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77340c8e10 0x7f773414c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7732800640 1 -- 172.21.15.27:0/2770335820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7734152d10 msgr2=0x7f773414c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7732800640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7734152d10 0x7f773414c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.903+0000 7f7732800640 1 -- 172.21.15.27:0/2770335820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77340c0a10 con 0x7f77341562f0 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.904+0000 7f7732800640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f77341ec8b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f772c036040 tx=0x7f772c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.904+0000 7f772a000640 1 -- 172.21.15.27:0/2770335820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c043020 con 0x7f77341562f0 2024-03-28T19:09:44.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.904+0000 7f772a000640 1 -- 172.21.15.27:0/2770335820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c03e450 con 0x7f77341562f0 2024-03-28T19:09:44.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.904+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77340c1560 con 0x7f77341562f0 2024-03-28T19:09:44.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.904+0000 7f772a000640 1 -- 172.21.15.27:0/2770335820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c03e820 con 0x7f77341562f0 2024-03-28T19:09:44.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.905+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77340bf5d0 con 0x7f77341562f0 2024-03-28T19:09:44.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.906+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7734073200 con 0x7f77341562f0 2024-03-28T19:09:44.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.908+0000 7f772a000640 1 -- 172.21.15.27:0/2770335820 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f772c046020 con 0x7f77341562f0 2024-03-28T19:09:44.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.909+0000 7f772a000640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f771406bbd0 0x7f771406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:44.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.909+0000 7f772a000640 1 -- 172.21.15.27:0/2770335820 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8185+0+0 (secure 0 0 0) 0x7f772c0adcc0 con 0x7f77341562f0 2024-03-28T19:09:44.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.909+0000 7f7731e00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f771406bbd0 0x7f771406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:44.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.910+0000 7f7731e00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f771406bbd0 0x7f771406e090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7734150c20 tx=0x7f7724009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:44.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:44.915+0000 7f772a000640 1 -- 172.21.15.27:0/2770335820 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7734073200 con 0x7f77341562f0 2024-03-28T19:09:45.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.223+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77341598f0 con 0x7f77341562f0 2024-03-28T19:09:45.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.224+0000 7f772a000640 1 -- 172.21.15.27:0/2770335820 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5464 (secure 0 0 0) 0x7f77341598f0 con 0x7f77341562f0 2024-03-28T19:09:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:45.224 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:44.939519+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:09:45.225 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 113 2024-03-28T19:09:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f771406bbd0 msgr2=0x7f771406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f771406bbd0 0x7f771406e090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7734150c20 tx=0x7f7724009040 comp rx=0 tx=0).stop 2024-03-28T19:09:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 msgr2=0x7f77341ec8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f77341ec8b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f772c036040 tx=0x7f772c004a40 comp rx=0 tx=0).stop 2024-03-28T19:09:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 shutdown_connections 2024-03-28T19:09:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f771406bbd0 0x7f771406e090 secure :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7734150c20 tx=0x7f7724009040 comp rx=0 tx=0).stop 2024-03-28T19:09:45.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77341562f0 0x7f77341ec8b0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7734152d10 0x7f773414c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 --2- 172.21.15.27:0/2770335820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77340c8e10 0x7f773414c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.228+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 >> 172.21.15.27:0/2770335820 conn(0x7f7734066fb0 msgr2=0x7f7734151f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.229+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 shutdown_connections 2024-03-28T19:09:45.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.229+0000 7f7738c00640 1 -- 172.21.15.27:0/2770335820 wait complete. 2024-03-28T19:09:45.251 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-03-28T19:09:45.251 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:09:45.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:45.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.470+0000 7f3f36c00640 1 Processor -- start 2024-03-28T19:09:45.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.471+0000 7f3f36c00640 1 -- start start 2024-03-28T19:09:45.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f36c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f30150ae0 0x7f3f300a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:45.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f36c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f300a61a0 0x7f3f300c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:45.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f36c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f30150ec0 con 0x7f3f300c6ee0 2024-03-28T19:09:45.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f36c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3006dc70 con 0x7f3f30150ae0 2024-03-28T19:09:45.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f36c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30153550 con 0x7f3f300a61a0 2024-03-28T19:09:45.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f35800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f30150ae0 0x7f3f300a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:45.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f35800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f30150ae0 0x7f3f300a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33540/0 (socket says 172.21.15.27:33540) 2024-03-28T19:09:45.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f35800640 1 -- 172.21.15.27:0/961269730 learned_addr learned my addr 172.21.15.27:0/961269730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:45.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.472+0000 7f3f34e00640 1 --2- 172.21.15.27:0/961269730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f300a61a0 0x7f3f300c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:45.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.473+0000 7f3f35800640 1 -- 172.21.15.27:0/961269730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f300a61a0 msgr2=0x7f3f300c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:45.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.473+0000 7f3f35800640 1 --2- 172.21.15.27:0/961269730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f300a61a0 0x7f3f300c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.473+0000 7f3f35800640 1 -- 172.21.15.27:0/961269730 >> v1:172.21.15.155:6789/0 conn(0x7f3f300c6ee0 legacy=0x7f3f30153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:45.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.473+0000 7f3f35800640 1 -- 172.21.15.27:0/961269730 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3006df10 con 0x7f3f30150ae0 2024-03-28T19:09:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.474+0000 7f3f35800640 1 --2- 172.21.15.27:0/961269730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30150ae0 0x7f3f300a5c60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009870 tx=0x7f3f2402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefd23f3de95a557 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:45.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.475+0000 7f3f2fe00640 1 -- 172.21.15.27:0/961269730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24035020 con 0x7f3f30150ae0 2024-03-28T19:09:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.475+0000 7f3f2fe00640 1 -- 172.21.15.27:0/961269730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f24031640 con 0x7f3f30150ae0 2024-03-28T19:09:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.475+0000 7f3f2fe00640 1 -- 172.21.15.27:0/961269730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2402d040 con 0x7f3f30150ae0 2024-03-28T19:09:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.475+0000 7f3f36c00640 1 -- 172.21.15.27:0/961269730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30150ae0 msgr2=0x7f3f300a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:45.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.475+0000 7f3f36c00640 1 --2- 172.21.15.27:0/961269730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30150ae0 0x7f3f300a5c60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009870 tx=0x7f3f2402b8f0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.476+0000 7f3f36c00640 1 -- 172.21.15.27:0/961269730 shutdown_connections 2024-03-28T19:09:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.476+0000 7f3f36c00640 1 --2- 172.21.15.27:0/961269730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f300a61a0 0x7f3f300c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.476+0000 7f3f36c00640 1 --2- 172.21.15.27:0/961269730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f30150ae0 0x7f3f300a5c60 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.476+0000 7f3f36c00640 1 -- 172.21.15.27:0/961269730 >> 172.21.15.27:0/961269730 conn(0x7f3f3006b0c0 msgr2=0x7f3f3006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.476+0000 7f3f36c00640 1 -- 172.21.15.27:0/961269730 shutdown_connections 2024-03-28T19:09:45.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.476+0000 7f3f36c00640 1 -- 172.21.15.27:0/961269730 wait complete. 2024-03-28T19:09:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.477+0000 7f3f36c00640 1 Processor -- start 2024-03-28T19:09:45.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.478+0000 7f3f36c00640 1 -- start start 2024-03-28T19:09:45.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f36c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f300a61a0 0x7f3f301ec020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:45.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f36c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f300c6ee0 0x7f3f301eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:45.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f36c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f30150ae0 0x7f3f300bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f36c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3006e7b0 con 0x7f3f300a61a0 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f36c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3006e4d0 con 0x7f3f300c6ee0 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f36c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f301569d0 con 0x7f3f30150ae0 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f35800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f300a61a0 0x7f3f301ec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f35800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f300a61a0 0x7f3f301ec020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33546/0 (socket says 172.21.15.27:33546) 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f35800640 1 -- 172.21.15.27:0/2786096497 learned_addr learned my addr 172.21.15.27:0/2786096497 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f34e00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f300c6ee0 0x7f3f301eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.479+0000 7f3f36200640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f30150ae0 0x7f3f300bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f35800640 1 -- 172.21.15.27:0/2786096497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f30150ae0 msgr2=0x7f3f300bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f35800640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f30150ae0 0x7f3f300bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f35800640 1 -- 172.21.15.27:0/2786096497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f300c6ee0 msgr2=0x7f3f301eb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f35800640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f300c6ee0 0x7f3f301eb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f35800640 1 -- 172.21.15.27:0/2786096497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f301ebaf0 con 0x7f3f300a61a0 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f35800640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f300a61a0 0x7f3f301ec020 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f3f2402b930 tx=0x7f3f240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f2d600640 1 -- 172.21.15.27:0/2786096497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24040070 con 0x7f3f300a61a0 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f2d600640 1 -- 172.21.15.27:0/2786096497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2403e070 con 0x7f3f300a61a0 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.480+0000 7f3f2d600640 1 -- 172.21.15.27:0/2786096497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24035040 con 0x7f3f300a61a0 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.481+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f300ab2d0 con 0x7f3f300a61a0 2024-03-28T19:09:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.481+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f300ab890 con 0x7f3f300a61a0 2024-03-28T19:09:45.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.482+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f30067dd0 con 0x7f3f300a61a0 2024-03-28T19:09:45.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.483+0000 7f3f2d600640 1 -- 172.21.15.27:0/2786096497 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f24038020 con 0x7f3f300a61a0 2024-03-28T19:09:45.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.484+0000 7f3f2d600640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f1006bbd0 0x7f3f1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:45.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.485+0000 7f3f2d600640 1 -- 172.21.15.27:0/2786096497 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8271+0+0 (secure 0 0 0) 0x7f3f240acd50 con 0x7f3f300a61a0 2024-03-28T19:09:45.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.485+0000 7f3f34e00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f1006bbd0 0x7f3f1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.486+0000 7f3f34e00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f1006bbd0 0x7f3f1006e090 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3f300aad40 tx=0x7f3f1c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.489+0000 7f3f2d600640 1 -- 172.21.15.27:0/2786096497 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f301ebaf0 con 0x7f3f300a61a0 2024-03-28T19:09:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.777+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f30155590 con 0x7f3f300a61a0 2024-03-28T19:09:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.778+0000 7f3f2d600640 1 -- 172.21.15.27:0/2786096497 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5464 (secure 0 0 0) 0x7f3f2407ae20 con 0x7f3f300a61a0 2024-03-28T19:09:45.778 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:45.778 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:44.939519+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:09:45.779 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 113 2024-03-28T19:09:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.781+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f1006bbd0 msgr2=0x7f3f1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.781+0000 7f3f36c00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f1006bbd0 0x7f3f1006e090 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3f300aad40 tx=0x7f3f1c009190 comp rx=0 tx=0).stop 2024-03-28T19:09:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.781+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f300a61a0 msgr2=0x7f3f301ec020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:45.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.781+0000 7f3f36c00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f300a61a0 0x7f3f301ec020 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f3f2402b930 tx=0x7f3f240046c0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.781+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 shutdown_connections 2024-03-28T19:09:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.781+0000 7f3f36c00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f1006bbd0 0x7f3f1006e090 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.781+0000 7f3f36c00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f30150ae0 0x7f3f300bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.782+0000 7f3f36c00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f300c6ee0 0x7f3f301eb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.782+0000 7f3f36c00640 1 --2- 172.21.15.27:0/2786096497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f300a61a0 0x7f3f301ec020 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.782+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 >> 172.21.15.27:0/2786096497 conn(0x7f3f3006b0c0 msgr2=0x7f3f30153930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.782+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 shutdown_connections 2024-03-28T19:09:45.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:45.782+0000 7f3f36c00640 1 -- 172.21.15.27:0/2786096497 wait complete. 2024-03-28T19:09:45.803 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 113, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 111, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6061": { "addr": "172.21.15.27:6837/3585028907", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6835", "nonce": 3585028907, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 3585028907, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6061, "incarnation": 108, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:08:45.461368+0000", "qdb_cluster": [ 6061 ], "qdb_leader": 6061, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6061 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 113, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6118": { "addr": "172.21.15.155:6835/3599950884", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 3599950884, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 3599950884, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6118, "incarnation": 113, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:09:44.939519+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6118 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:09:46.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.264+0000 7fef70400640 1 Processor -- start 2024-03-28T19:09:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.265+0000 7fef70400640 1 -- start start 2024-03-28T19:09:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.266+0000 7fef70400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef680c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.266+0000 7fef70400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef680c7310 con 0x7fef68152f80 2024-03-28T19:09:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.266+0000 7fef70400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef68158d40 con 0x7fef68155730 2024-03-28T19:09:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.266+0000 7fef70400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef68159020 con 0x7fef680c7970 2024-03-28T19:09:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.267+0000 7fef6da00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fef68152f80 0x7fef680c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34890/0 (socket says 172.21.15.27:34890) 2024-03-28T19:09:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.267+0000 7fef6da00640 1 -- 172.21.15.27:0/2217604806 learned_addr learned my addr 172.21.15.27:0/2217604806 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:09:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.267+0000 7fef6d000640 1 --2- 172.21.15.27:0/2217604806 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef680c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.267+0000 7fef67e00640 1 -- 172.21.15.27:0/2217604806 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279855069 0 0) 0x7fef680c7310 con 0x7fef68152f80 2024-03-28T19:09:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.267+0000 7fef67e00640 1 -- 172.21.15.27:0/2217604806 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef50003520 con 0x7fef68152f80 2024-03-28T19:09:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.267+0000 7fef67e00640 1 -- 172.21.15.27:0/2217604806 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200775890 0 0) 0x7fef68158d40 con 0x7fef68155730 2024-03-28T19:09:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.267+0000 7fef67e00640 1 -- 172.21.15.27:0/2217604806 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef680c7310 con 0x7fef68155730 2024-03-28T19:09:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.268+0000 7fef6d000640 1 -- 172.21.15.27:0/2217604806 >> v1:172.21.15.155:6789/0 conn(0x7fef68155730 legacy=0x7fef68155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.268+0000 7fef6d000640 1 -- 172.21.15.27:0/2217604806 >> v1:172.21.15.27:6789/0 conn(0x7fef68152f80 legacy=0x7fef680c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:46.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.268+0000 7fef6d000640 1 -- 172.21.15.27:0/2217604806 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68159300 con 0x7fef680c7970 2024-03-28T19:09:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.269+0000 7fef6d000640 1 --2- 172.21.15.27:0/2217604806 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef680c7df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef540097d0 tx=0x7fef5402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dd99a46bad9b044 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.270+0000 7fef67e00640 1 -- 172.21.15.27:0/2217604806 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54035020 con 0x7fef680c7970 2024-03-28T19:09:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.270+0000 7fef67e00640 1 -- 172.21.15.27:0/2217604806 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef54031810 con 0x7fef680c7970 2024-03-28T19:09:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.270+0000 7fef67e00640 1 -- 172.21.15.27:0/2217604806 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54031be0 con 0x7fef680c7970 2024-03-28T19:09:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.270+0000 7fef70400640 1 -- 172.21.15.27:0/2217604806 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 msgr2=0x7fef680c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.270+0000 7fef70400640 1 --2- 172.21.15.27:0/2217604806 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef680c7df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef540097d0 tx=0x7fef5402b8b0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.271+0000 7fef70400640 1 -- 172.21.15.27:0/2217604806 shutdown_connections 2024-03-28T19:09:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.271+0000 7fef70400640 1 --2- 172.21.15.27:0/2217604806 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef680c7df0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.271+0000 7fef70400640 1 -- 172.21.15.27:0/2217604806 >> 172.21.15.27:0/2217604806 conn(0x7fef6806b6f0 msgr2=0x7fef680caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.271+0000 7fef70400640 1 -- 172.21.15.27:0/2217604806 shutdown_connections 2024-03-28T19:09:46.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.271+0000 7fef70400640 1 -- 172.21.15.27:0/2217604806 wait complete. 2024-03-28T19:09:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.273+0000 7fef70400640 1 Processor -- start 2024-03-28T19:09:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.273+0000 7fef70400640 1 -- start start 2024-03-28T19:09:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef70400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef681ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef70400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68152f80 0x7fef681eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef70400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68155730 0x7fef6814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef70400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef680cbca0 con 0x7fef68155730 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef70400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef68159b00 con 0x7fef68152f80 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef70400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef680c3850 con 0x7fef680c7970 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef6e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68155730 0x7fef6814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef6e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68155730 0x7fef6814a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49192/0 (socket says 172.21.15.27:49192) 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef6e400640 1 -- 172.21.15.27:0/822123176 learned_addr learned my addr 172.21.15.27:0/822123176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef6d000640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68152f80 0x7fef681eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.274+0000 7fef6da00640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef681ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.275+0000 7fef6e400640 1 -- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 msgr2=0x7fef681ee910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.275+0000 7fef6e400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef681ee910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.275+0000 7fef6e400640 1 -- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68152f80 msgr2=0x7fef681eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.275+0000 7fef6e400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68152f80 0x7fef681eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.275+0000 7fef6e400640 1 -- 172.21.15.27:0/822123176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef681ef390 con 0x7fef68155730 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.275+0000 7fef6da00640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef681ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.275+0000 7fef6e400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68155730 0x7fef6814a0f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fef6000c700 tx=0x7fef6000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.275+0000 7fef65600640 1 -- 172.21.15.27:0/822123176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef60003b70 con 0x7fef68155730 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.276+0000 7fef65600640 1 -- 172.21.15.27:0/822123176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef60003d10 con 0x7fef68155730 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.276+0000 7fef65600640 1 -- 172.21.15.27:0/822123176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef600052d0 con 0x7fef68155730 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.276+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef680c3520 con 0x7fef68155730 2024-03-28T19:09:46.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.276+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef680c27a0 con 0x7fef68155730 2024-03-28T19:09:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.277+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6804f400 con 0x7fef68155730 2024-03-28T19:09:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.278+0000 7fef65600640 1 -- 172.21.15.27:0/822123176 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fef60005470 con 0x7fef68155730 2024-03-28T19:09:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.279+0000 7fef65600640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef4806bbd0 0x7fef4806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.279+0000 7fef65600640 1 -- 172.21.15.27:0/822123176 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8271+0+0 (secure 0 0 0) 0x7fef6008b1f0 con 0x7fef68155730 2024-03-28T19:09:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.280+0000 7fef6da00640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef4806bbd0 0x7fef4806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:46.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.280+0000 7fef6da00640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef4806bbd0 0x7fef4806e090 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0011e0 tx=0x7fef5c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:46.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.284+0000 7fef65600640 1 -- 172.21.15.27:0/822123176 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fef60059320 con 0x7fef68155730 2024-03-28T19:09:46.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.562+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef680cba80 con 0x7fef68155730 2024-03-28T19:09:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.563+0000 7fef65600640 1 -- 172.21.15.27:0/822123176 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5464 (secure 0 0 0) 0x7fef60058c20 con 0x7fef68155730 2024-03-28T19:09:46.563 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:46.563 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:08:45.461368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6061":{"gid":6061,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6837/3585028907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6835","nonce":3585028907},{"type":"v1","addr":"172.21.15.27:6837","nonce":3585028907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6061,"qdb_cluster":[6061]},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:44.939519+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:09:46.563 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 113 2024-03-28T19:09:46.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.566+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef4806bbd0 msgr2=0x7fef4806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.566+0000 7fef70400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef4806bbd0 0x7fef4806e090 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0011e0 tx=0x7fef5c009040 comp rx=0 tx=0).stop 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.566+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68155730 msgr2=0x7fef6814a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.566+0000 7fef70400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68155730 0x7fef6814a0f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fef6000c700 tx=0x7fef6000cbc0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.567+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 shutdown_connections 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.567+0000 7fef70400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef4806bbd0 0x7fef4806e090 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.567+0000 7fef70400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef68155730 0x7fef6814a0f0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.567+0000 7fef70400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68152f80 0x7fef681eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.567+0000 7fef70400640 1 --2- 172.21.15.27:0/822123176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef680c7970 0x7fef681ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.567+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 >> 172.21.15.27:0/822123176 conn(0x7fef6806b6f0 msgr2=0x7fef68155bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.567+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 shutdown_connections 2024-03-28T19:09:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:46.567+0000 7fef70400640 1 -- 172.21.15.27:0/822123176 wait complete. 2024-03-28T19:09:46.589 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:09:47.804 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:48.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.029+0000 7fd0ec200640 1 Processor -- start 2024-03-28T19:09:48.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.029+0000 7fd0ec200640 1 -- start start 2024-03-28T19:09:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.030+0000 7fd0ec200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0e414f6d0 0x7fd0e4159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.030+0000 7fd0ec200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 0x7fd0e40c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.030+0000 7fd0ec200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd0e40c3aa0 0x7fd0e40c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.030+0000 7fd0ec200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e4072870 con 0x7fd0e414f6d0 2024-03-28T19:09:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.030+0000 7fd0ec200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e4156640 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.030+0000 7fd0ec200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4158db0 con 0x7fd0e4071cf0 2024-03-28T19:09:48.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e9800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0e414f6d0 0x7fd0e4159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e9800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0e414f6d0 0x7fd0e4159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49196/0 (socket says 172.21.15.27:49196) 2024-03-28T19:09:48.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e9800640 1 -- 172.21.15.27:0/736964765 learned_addr learned my addr 172.21.15.27:0/736964765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:48.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e8e00640 1 --2- 172.21.15.27:0/736964765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 0x7fd0e40c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0ea200640 1 --2- 172.21.15.27:0/736964765 >> v2:172.21.15.155:3300/0 conn(0x7fd0e40c3aa0 0x7fd0e40c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e9800640 1 -- 172.21.15.27:0/736964765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 msgr2=0x7fd0e40c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:48.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e9800640 1 --2- 172.21.15.27:0/736964765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 0x7fd0e40c3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e9800640 1 -- 172.21.15.27:0/736964765 >> v2:172.21.15.155:3300/0 conn(0x7fd0e40c3aa0 msgr2=0x7fd0e40c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:48.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e9800640 1 --2- 172.21.15.27:0/736964765 >> v2:172.21.15.155:3300/0 conn(0x7fd0e40c3aa0 0x7fd0e40c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0e9800640 1 -- 172.21.15.27:0/736964765 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e4156360 con 0x7fd0e414f6d0 2024-03-28T19:09:48.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.031+0000 7fd0ea200640 1 --2- 172.21.15.27:0/736964765 >> v2:172.21.15.155:3300/0 conn(0x7fd0e40c3aa0 0x7fd0e40c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:09:48.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.033+0000 7fd0e9800640 1 --2- 172.21.15.27:0/736964765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e414f6d0 0x7fd0e4159850 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40c48a0 tx=0x7fd0d802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f2fc58dd5ee02d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:48.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.033+0000 7fd0e3e00640 1 -- 172.21.15.27:0/736964765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80317d0 con 0x7fd0e414f6d0 2024-03-28T19:09:48.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.033+0000 7fd0e3e00640 1 -- 172.21.15.27:0/736964765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d8037070 con 0x7fd0e414f6d0 2024-03-28T19:09:48.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.033+0000 7fd0e3e00640 1 -- 172.21.15.27:0/736964765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8031b90 con 0x7fd0e414f6d0 2024-03-28T19:09:48.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 -- 172.21.15.27:0/736964765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e414f6d0 msgr2=0x7fd0e4159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 --2- 172.21.15.27:0/736964765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e414f6d0 0x7fd0e4159850 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40c48a0 tx=0x7fd0d802b870 comp rx=0 tx=0).stop 2024-03-28T19:09:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 -- 172.21.15.27:0/736964765 shutdown_connections 2024-03-28T19:09:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 --2- 172.21.15.27:0/736964765 >> v2:172.21.15.155:3300/0 conn(0x7fd0e40c3aa0 0x7fd0e40c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 --2- 172.21.15.27:0/736964765 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 0x7fd0e40c3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 --2- 172.21.15.27:0/736964765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e414f6d0 0x7fd0e4159850 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 -- 172.21.15.27:0/736964765 >> 172.21.15.27:0/736964765 conn(0x7fd0e406c5f0 msgr2=0x7fd0e406ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 -- 172.21.15.27:0/736964765 shutdown_connections 2024-03-28T19:09:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.035+0000 7fd0ec200640 1 -- 172.21.15.27:0/736964765 wait complete. 2024-03-28T19:09:48.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.037+0000 7fd0ec200640 1 Processor -- start 2024-03-28T19:09:48.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.037+0000 7fd0ec200640 1 -- start start 2024-03-28T19:09:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.038+0000 7fd0ec200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 0x7fd0e40ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.038+0000 7fd0ec200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e40c3aa0 0x7fd0e40c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.038+0000 7fd0ec200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e414f6d0 0x7fd0e40aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.038+0000 7fd0ec200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e414fab0 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.039+0000 7fd0ec200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4159330 con 0x7fd0e414f6d0 2024-03-28T19:09:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.039+0000 7fd0ec200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4155fa0 con 0x7fd0e4071cf0 2024-03-28T19:09:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.039+0000 7fd0e8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e40c3aa0 0x7fd0e40c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.039+0000 7fd0e8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e40c3aa0 0x7fd0e40c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49200/0 (socket says 172.21.15.27:49200) 2024-03-28T19:09:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.039+0000 7fd0e8e00640 1 -- 172.21.15.27:0/689034714 learned_addr learned my addr 172.21.15.27:0/689034714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.039+0000 7fd0e9800640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 0x7fd0e40ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.039+0000 7fd0ea200640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e414f6d0 0x7fd0e40aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.040+0000 7fd0e8e00640 1 -- 172.21.15.27:0/689034714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 msgr2=0x7fd0e40ac5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.040+0000 7fd0e8e00640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 0x7fd0e40ac5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.040+0000 7fd0e8e00640 1 -- 172.21.15.27:0/689034714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e414f6d0 msgr2=0x7fd0e40aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.040+0000 7fd0e8e00640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e414f6d0 0x7fd0e40aafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.040+0000 7fd0e8e00640 1 -- 172.21.15.27:0/689034714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e41537b0 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.040+0000 7fd0e8e00640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e40c3aa0 0x7fd0e40c82e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000e860 tx=0x7fd0d000ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.040+0000 7fd0e1600640 1 -- 172.21.15.27:0/689034714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0017070 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.041+0000 7fd0e1600640 1 -- 172.21.15.27:0/689034714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d000c940 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.041+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e41eefb0 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.041+0000 7fd0e1600640 1 -- 172.21.15.27:0/689034714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000cd10 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.041+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e41ef290 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.042+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e404f400 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.044+0000 7fd0e1600640 1 -- 172.21.15.27:0/689034714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd0d001a070 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.045+0000 7fd0e1600640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c406bbd0 0x7fd0c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.045+0000 7fd0e1600640 1 -- 172.21.15.27:0/689034714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7fd0d008b470 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.045+0000 7fd0e9800640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c406bbd0 0x7fd0c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.046+0000 7fd0e9800640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c406bbd0 0x7fd0c406e090 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8036000 tx=0x7fd0d803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.051+0000 7fd0e1600640 1 -- 172.21.15.27:0/689034714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd0d0058e90 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.316+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0e40c2330 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.318+0000 7fd0e1600640 1 -- 172.21.15.27:0/689034714 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4671 (secure 0 0 0) 0x7fd0d005cf30 con 0x7fd0e40c3aa0 2024-03-28T19:09:48.317 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:48.317 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:47.701472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:47.048431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:09:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 115 2024-03-28T19:09:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.320+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c406bbd0 msgr2=0x7fd0c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.320+0000 7fd0ec200640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c406bbd0 0x7fd0c406e090 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8036000 tx=0x7fd0d803d040 comp rx=0 tx=0).stop 2024-03-28T19:09:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.320+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e40c3aa0 msgr2=0x7fd0e40c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.320+0000 7fd0ec200640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e40c3aa0 0x7fd0e40c82e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000e860 tx=0x7fd0d000ed20 comp rx=0 tx=0).stop 2024-03-28T19:09:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.321+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 shutdown_connections 2024-03-28T19:09:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.321+0000 7fd0ec200640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd0c406bbd0 0x7fd0c406e090 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.321+0000 7fd0ec200640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0e414f6d0 0x7fd0e40aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.321+0000 7fd0ec200640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e40c3aa0 0x7fd0e40c82e0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.321+0000 7fd0ec200640 1 --2- 172.21.15.27:0/689034714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd0e4071cf0 0x7fd0e40ac5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.321+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 >> 172.21.15.27:0/689034714 conn(0x7fd0e406c5f0 msgr2=0x7fd0e406de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.321+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 shutdown_connections 2024-03-28T19:09:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.321+0000 7fd0ec200640 1 -- 172.21.15.27:0/689034714 wait complete. 2024-03-28T19:09:48.338 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:09:48.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:48.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.831+0000 7fc163c00640 1 Processor -- start 2024-03-28T19:09:48.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.831+0000 7fc163c00640 1 -- start start 2024-03-28T19:09:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.833+0000 7fc163c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc15c06ee00 0x7fc15c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.833+0000 7fc163c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc15c0c3ea0 0x7fc15c159440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.833+0000 7fc163c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15c0c1600 con 0x7fc15c0c60e0 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.833+0000 7fc163c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc15c0c23b0 con 0x7fc15c06ee00 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.833+0000 7fc163c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc15c0c18e0 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.833+0000 7fc15be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc15c06ee00 0x7fc15c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.833+0000 7fc15be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc15c06ee00 0x7fc15c158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49204/0 (socket says 172.21.15.27:49204) 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.833+0000 7fc15be00640 1 -- 172.21.15.27:0/2018834881 learned_addr learned my addr 172.21.15.27:0/2018834881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.834+0000 7fc161c00640 1 --2- 172.21.15.27:0/2018834881 >> v2:172.21.15.155:3300/0 conn(0x7fc15c0c3ea0 0x7fc15c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.834+0000 7fc15be00640 1 -- 172.21.15.27:0/2018834881 >> v2:172.21.15.155:3300/0 conn(0x7fc15c0c3ea0 msgr2=0x7fc15c159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.834+0000 7fc15be00640 1 --2- 172.21.15.27:0/2018834881 >> v2:172.21.15.155:3300/0 conn(0x7fc15c0c3ea0 0x7fc15c159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.834+0000 7fc15be00640 1 -- 172.21.15.27:0/2018834881 >> v1:172.21.15.27:6789/0 conn(0x7fc15c0c60e0 legacy=0x7fc15c06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.834+0000 7fc15be00640 1 -- 172.21.15.27:0/2018834881 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c0c1bc0 con 0x7fc15c06ee00 2024-03-28T19:09:48.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.835+0000 7fc15be00640 1 --2- 172.21.15.27:0/2018834881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c06ee00 0x7fc15c158f00 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0c2320 tx=0x7fc15002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b82e58a8f503ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:48.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.835+0000 7fc15b400640 1 -- 172.21.15.27:0/2018834881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150035020 con 0x7fc15c06ee00 2024-03-28T19:09:48.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.835+0000 7fc15b400640 1 -- 172.21.15.27:0/2018834881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc150031870 con 0x7fc15c06ee00 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.835+0000 7fc15b400640 1 -- 172.21.15.27:0/2018834881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150031c40 con 0x7fc15c06ee00 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.836+0000 7fc163c00640 1 -- 172.21.15.27:0/2018834881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c06ee00 msgr2=0x7fc15c158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.836+0000 7fc163c00640 1 --2- 172.21.15.27:0/2018834881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c06ee00 0x7fc15c158f00 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0c2320 tx=0x7fc15002b910 comp rx=0 tx=0).stop 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.836+0000 7fc163c00640 1 -- 172.21.15.27:0/2018834881 shutdown_connections 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.836+0000 7fc163c00640 1 --2- 172.21.15.27:0/2018834881 >> v2:172.21.15.155:3300/0 conn(0x7fc15c0c3ea0 0x7fc15c159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.836+0000 7fc163c00640 1 --2- 172.21.15.27:0/2018834881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c06ee00 0x7fc15c158f00 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.836+0000 7fc163c00640 1 -- 172.21.15.27:0/2018834881 >> 172.21.15.27:0/2018834881 conn(0x7fc15c06b9d0 msgr2=0x7fc15c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.836+0000 7fc163c00640 1 -- 172.21.15.27:0/2018834881 shutdown_connections 2024-03-28T19:09:48.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.836+0000 7fc163c00640 1 -- 172.21.15.27:0/2018834881 wait complete. 2024-03-28T19:09:48.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.838+0000 7fc163c00640 1 Processor -- start 2024-03-28T19:09:48.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.838+0000 7fc163c00640 1 -- start start 2024-03-28T19:09:48.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.839+0000 7fc163c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc15c06ee00 0x7fc15c1e1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.839+0000 7fc163c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c0c3ea0 0x7fc15c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.839+0000 7fc163c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc15c0c60e0 0x7fc15c0aa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.839+0000 7fc163c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0c6640 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.839+0000 7fc163c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0c64c0 con 0x7fc15c0c60e0 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.839+0000 7fc163c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c159980 con 0x7fc15c06ee00 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc15be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c0c3ea0 0x7fc15c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc15be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c0c3ea0 0x7fc15c071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49216/0 (socket says 172.21.15.27:49216) 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc15be00640 1 -- 172.21.15.27:0/3065187836 learned_addr learned my addr 172.21.15.27:0/3065187836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc161200640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc15c06ee00 0x7fc15c1e1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc161c00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc15c0c60e0 0x7fc15c0aa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc15be00640 1 -- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc15c06ee00 msgr2=0x7fc15c1e1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc15be00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc15c06ee00 0x7fc15c1e1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc15be00640 1 -- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc15c0c60e0 msgr2=0x7fc15c0aa1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:48.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc15be00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc15c0c60e0 0x7fc15c0aa1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:48.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc15be00640 1 -- 172.21.15.27:0/3065187836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c0aae80 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.840+0000 7fc161200640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc15c06ee00 0x7fc15c1e1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:09:48.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.841+0000 7fc15be00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c0c3ea0 0x7fc15c071d10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc15002b950 tx=0x7fc1500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:48.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.841+0000 7fc158c00640 1 -- 172.21.15.27:0/3065187836 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150040070 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.841+0000 7fc158c00640 1 -- 172.21.15.27:0/3065187836 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc150030d00 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.841+0000 7fc158c00640 1 -- 172.21.15.27:0/3065187836 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150035040 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.841+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15c0ab120 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.841+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15c072250 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.843+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c0c6ef0 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.844+0000 7fc158c00640 1 -- 172.21.15.27:0/3065187836 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc150038020 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.845+0000 7fc158c00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.846+0000 7fc158c00640 1 -- 172.21.15.27:0/3065187836 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7fc1500ad330 con 0x7fc15c0c3ea0 2024-03-28T19:09:48.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.846+0000 7fc161200640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:48.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.847+0000 7fc161200640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc15c153bd0 tx=0x7fc14c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:48.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:48.850+0000 7fc158c00640 1 -- 172.21.15.27:0/3065187836 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc15c0c6ef0 con 0x7fc15c0c3ea0 2024-03-28T19:09:49.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.125+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc15c0a9a70 con 0x7fc15c0c3ea0 2024-03-28T19:09:49.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.126+0000 7fc158c00640 1 -- 172.21.15.27:0/3065187836 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4668 (secure 0 0 0) 0x7fc15007b3a0 con 0x7fc15c0c3ea0 2024-03-28T19:09:49.126 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:49.126 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:47.701472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:47.712810+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:09:49.126 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 116 2024-03-28T19:09:49.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 msgr2=0x7fc14006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:49.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc15c153bd0 tx=0x7fc14c009040 comp rx=0 tx=0).stop 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c0c3ea0 msgr2=0x7fc15c071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c0c3ea0 0x7fc15c071d10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc15002b950 tx=0x7fc1500046c0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 shutdown_connections 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc15c0c60e0 0x7fc15c0aa1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc15c0c3ea0 0x7fc15c071d10 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 --2- 172.21.15.27:0/3065187836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc15c06ee00 0x7fc15c1e1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 >> 172.21.15.27:0/3065187836 conn(0x7fc15c06b9d0 msgr2=0x7fc15c0c34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 shutdown_connections 2024-03-28T19:09:49.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.129+0000 7fc163c00640 1 -- 172.21.15.27:0/3065187836 wait complete. 2024-03-28T19:09:49.147 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-03-28T19:09:49.147 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:09:49.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.358+0000 7ff379000640 1 Processor -- start 2024-03-28T19:09:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.358+0000 7ff379000640 1 -- start start 2024-03-28T19:09:49.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff379000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff374156120 0x7ff37406e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:49.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff379000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff374156500 0x7ff3740c6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:49.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff379000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff374159c50 con 0x7ff3740c5320 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff379000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740c4d70 con 0x7ff374156120 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff379000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740c1030 con 0x7ff374156500 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff373400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff374156120 0x7ff37406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff373400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff374156120 0x7ff37406e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49224/0 (socket says 172.21.15.27:49224) 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff373400640 1 -- 172.21.15.27:0/706106054 learned_addr learned my addr 172.21.15.27:0/706106054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.360+0000 7ff373e00640 1 --2- 172.21.15.27:0/706106054 >> v2:172.21.15.155:3300/0 conn(0x7ff374156500 0x7ff3740c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.361+0000 7ff373400640 1 -- 172.21.15.27:0/706106054 >> v2:172.21.15.155:3300/0 conn(0x7ff374156500 msgr2=0x7ff3740c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.361+0000 7ff373400640 1 --2- 172.21.15.27:0/706106054 >> v2:172.21.15.155:3300/0 conn(0x7ff374156500 0x7ff3740c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.361+0000 7ff373400640 1 -- 172.21.15.27:0/706106054 >> v1:172.21.15.155:6789/0 conn(0x7ff3740c5320 legacy=0x7ff3740c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.361+0000 7ff373400640 1 -- 172.21.15.27:0/706106054 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740c4a40 con 0x7ff374156120 2024-03-28T19:09:49.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.362+0000 7ff373400640 1 --2- 172.21.15.27:0/706106054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff37406e050 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff3740738f0 tx=0x7ff36402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc1a52b878242d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:49.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.362+0000 7ff372a00640 1 -- 172.21.15.27:0/706106054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364038070 con 0x7ff374156120 2024-03-28T19:09:49.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.362+0000 7ff372a00640 1 -- 172.21.15.27:0/706106054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36402f050 con 0x7ff374156120 2024-03-28T19:09:49.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.362+0000 7ff372a00640 1 -- 172.21.15.27:0/706106054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364031970 con 0x7ff374156120 2024-03-28T19:09:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.363+0000 7ff379000640 1 -- 172.21.15.27:0/706106054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 msgr2=0x7ff37406e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.363+0000 7ff379000640 1 --2- 172.21.15.27:0/706106054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff37406e050 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff3740738f0 tx=0x7ff36402b7f0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.363+0000 7ff379000640 1 -- 172.21.15.27:0/706106054 shutdown_connections 2024-03-28T19:09:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.363+0000 7ff379000640 1 --2- 172.21.15.27:0/706106054 >> v2:172.21.15.155:3300/0 conn(0x7ff374156500 0x7ff3740c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.363+0000 7ff379000640 1 --2- 172.21.15.27:0/706106054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff37406e050 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.363+0000 7ff379000640 1 -- 172.21.15.27:0/706106054 >> 172.21.15.27:0/706106054 conn(0x7ff37406b0c0 msgr2=0x7ff37406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.363+0000 7ff379000640 1 -- 172.21.15.27:0/706106054 shutdown_connections 2024-03-28T19:09:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.363+0000 7ff379000640 1 -- 172.21.15.27:0/706106054 wait complete. 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.365+0000 7ff379000640 1 Processor -- start 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.365+0000 7ff379000640 1 -- start start 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff379000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3740c5320 0x7ff3740ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff379000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff3741ee710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff379000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff374156500 0x7ff374150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff379000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740c73d0 con 0x7ff374156120 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff379000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740c7250 con 0x7ff3740c5320 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff379000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740c29a0 con 0x7ff374156500 2024-03-28T19:09:49.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff36be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff3741ee710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff36be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff3741ee710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49230/0 (socket says 172.21.15.27:49230) 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff36be00640 1 -- 172.21.15.27:0/364465707 learned_addr learned my addr 172.21.15.27:0/364465707 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.366+0000 7ff373400640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3740c5320 0x7ff3740ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff373e00640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff374156500 0x7ff374150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36be00640 1 -- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff374156500 msgr2=0x7ff374150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36be00640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff374156500 0x7ff374150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36be00640 1 -- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3740c5320 msgr2=0x7ff3740ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36be00640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3740c5320 0x7ff3740ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36be00640 1 -- 172.21.15.27:0/364465707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3741ef1c0 con 0x7ff374156120 2024-03-28T19:09:49.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff373400640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3740c5320 0x7ff3740ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:09:49.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36be00640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff3741ee710 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00c430 tx=0x7ff35c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:49.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36b400640 1 -- 172.21.15.27:0/364465707 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c011790 con 0x7ff374156120 2024-03-28T19:09:49.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36b400640 1 -- 172.21.15.27:0/364465707 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c017070 con 0x7ff374156120 2024-03-28T19:09:49.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3741ef4a0 con 0x7ff374156120 2024-03-28T19:09:49.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff36b400640 1 -- 172.21.15.27:0/364465707 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c011bb0 con 0x7ff374156120 2024-03-28T19:09:49.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.367+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3740a9420 con 0x7ff374156120 2024-03-28T19:09:49.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.369+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff374067dd0 con 0x7ff374156120 2024-03-28T19:09:49.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.370+0000 7ff36b400640 1 -- 172.21.15.27:0/364465707 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff35c01c460 con 0x7ff374156120 2024-03-28T19:09:49.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.370+0000 7ff36b400640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff35006bbd0 0x7ff35006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:49.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.371+0000 7ff36b400640 1 -- 172.21.15.27:0/364465707 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7ff35c08b410 con 0x7ff374156120 2024-03-28T19:09:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.371+0000 7ff373400640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff35006bbd0 0x7ff35006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:49.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.371+0000 7ff373400640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff35006bbd0 0x7ff35006e090 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff364035000 tx=0x7ff364008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:49.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.375+0000 7ff36b400640 1 -- 172.21.15.27:0/364465707 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff35c059480 con 0x7ff374156120 2024-03-28T19:09:49.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.621+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff374069640 con 0x7ff374156120 2024-03-28T19:09:49.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.622+0000 7ff36b400640 1 -- 172.21.15.27:0/364465707 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+4668 (secure 0 0 0) 0x7ff35c058d80 con 0x7ff374156120 2024-03-28T19:09:49.622 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:49.622 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:47.701472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:47.712810+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:09:49.623 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 116 2024-03-28T19:09:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff35006bbd0 msgr2=0x7ff35006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff35006bbd0 0x7ff35006e090 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff364035000 tx=0x7ff364008030 comp rx=0 tx=0).stop 2024-03-28T19:09:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 msgr2=0x7ff3741ee710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff3741ee710 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00c430 tx=0x7ff35c00c8f0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 shutdown_connections 2024-03-28T19:09:49.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff35006bbd0 0x7ff35006e090 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff374156500 0x7ff374150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff374156120 0x7ff3741ee710 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 --2- 172.21.15.27:0/364465707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3740c5320 0x7ff3740ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:49.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 >> 172.21.15.27:0/364465707 conn(0x7ff37406b0c0 msgr2=0x7ff3740c6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:49.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.625+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 shutdown_connections 2024-03-28T19:09:49.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:49.626+0000 7ff379000640 1 -- 172.21.15.27:0/364465707 wait complete. 2024-03-28T19:09:49.645 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 116, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 115, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5078": { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 115, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:09:47.701472+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5078 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 116, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6118": { "addr": "172.21.15.155:6835/3599950884", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 3599950884, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 3599950884, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6118, "incarnation": 113, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:09:47.712810+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6118 } } } ], "standbys": [] } 2024-03-28T19:09:50.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:50.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.583+0000 7fc169600640 1 Processor -- start 2024-03-28T19:09:50.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.583+0000 7fc169600640 1 -- start start 2024-03-28T19:09:50.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.584+0000 7fc169600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc1640c60e0 0x7fc16406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:50.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.584+0000 7fc169600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc164072df0 0x7fc16406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:50.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.584+0000 7fc169600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640c1600 con 0x7fc1640c3ea0 2024-03-28T19:09:50.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.584+0000 7fc169600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1640c23b0 con 0x7fc1640c60e0 2024-03-28T19:09:50.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.584+0000 7fc169600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640c18e0 con 0x7fc164072df0 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.584+0000 7fc162a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc1640c60e0 0x7fc16406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.584+0000 7fc15a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc1640c3ea0 0x7fc164158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34914/0 (socket says 172.21.15.27:34914) 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.584+0000 7fc15a000640 1 -- 172.21.15.27:0/3030489791 learned_addr learned my addr 172.21.15.27:0/3030489791 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.585+0000 7fc163400640 1 --2- 172.21.15.27:0/3030489791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc164072df0 0x7fc16406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.585+0000 7fc162a00640 1 -- 172.21.15.27:0/3030489791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc164072df0 msgr2=0x7fc16406ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.585+0000 7fc162a00640 1 --2- 172.21.15.27:0/3030489791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc164072df0 0x7fc16406ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.585+0000 7fc162a00640 1 -- 172.21.15.27:0/3030489791 >> v1:172.21.15.27:6789/0 conn(0x7fc1640c3ea0 legacy=0x7fc164158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.585+0000 7fc162000640 1 -- 172.21.15.27:0/3030489791 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395153239 0 0) 0x7fc1640c1600 con 0x7fc1640c3ea0 2024-03-28T19:09:50.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.585+0000 7fc162a00640 1 -- 172.21.15.27:0/3030489791 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1640c1bc0 con 0x7fc1640c60e0 2024-03-28T19:09:50.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.586+0000 7fc162a00640 1 --2- 172.21.15.27:0/3030489791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c60e0 0x7fc16406e6f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc1540097a0 tx=0x7fc15402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5d909ad20c75d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:50.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.586+0000 7fc162000640 1 -- 172.21.15.27:0/3030489791 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154038070 con 0x7fc1640c60e0 2024-03-28T19:09:50.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.586+0000 7fc162000640 1 -- 172.21.15.27:0/3030489791 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc154031700 con 0x7fc1640c60e0 2024-03-28T19:09:50.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.586+0000 7fc162000640 1 -- 172.21.15.27:0/3030489791 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154031ad0 con 0x7fc1640c60e0 2024-03-28T19:09:50.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.587+0000 7fc169600640 1 -- 172.21.15.27:0/3030489791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c60e0 msgr2=0x7fc16406e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.587+0000 7fc169600640 1 --2- 172.21.15.27:0/3030489791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c60e0 0x7fc16406e6f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc1540097a0 tx=0x7fc15402b7a0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.587+0000 7fc169600640 1 -- 172.21.15.27:0/3030489791 shutdown_connections 2024-03-28T19:09:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.587+0000 7fc169600640 1 --2- 172.21.15.27:0/3030489791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc164072df0 0x7fc16406ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.587+0000 7fc169600640 1 --2- 172.21.15.27:0/3030489791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c60e0 0x7fc16406e6f0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.587+0000 7fc169600640 1 -- 172.21.15.27:0/3030489791 >> 172.21.15.27:0/3030489791 conn(0x7fc16406b9d0 msgr2=0x7fc1640c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.587+0000 7fc169600640 1 -- 172.21.15.27:0/3030489791 shutdown_connections 2024-03-28T19:09:50.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.587+0000 7fc169600640 1 -- 172.21.15.27:0/3030489791 wait complete. 2024-03-28T19:09:50.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.589+0000 7fc169600640 1 Processor -- start 2024-03-28T19:09:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.589+0000 7fc169600640 1 -- start start 2024-03-28T19:09:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc169600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc164072df0 0x7fc1641eba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc169600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c3ea0 0x7fc1641eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc169600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1640c60e0 0x7fc1640c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc169600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640c6640 con 0x7fc1640c3ea0 2024-03-28T19:09:50.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc169600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640c64c0 con 0x7fc164072df0 2024-03-28T19:09:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc169600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640c4f00 con 0x7fc1640c60e0 2024-03-28T19:09:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc15a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c3ea0 0x7fc1641eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc15a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c3ea0 0x7fc1641eee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49260/0 (socket says 172.21.15.27:49260) 2024-03-28T19:09:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.590+0000 7fc15a000640 1 -- 172.21.15.27:0/866900437 learned_addr learned my addr 172.21.15.27:0/866900437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc163400640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1640c60e0 0x7fc1640c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15a000640 1 -- 172.21.15.27:0/866900437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1640c60e0 msgr2=0x7fc1640c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15a000640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1640c60e0 0x7fc1640c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc162a00640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc164072df0 0x7fc1641eba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15a000640 1 -- 172.21.15.27:0/866900437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc164072df0 msgr2=0x7fc1641eba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15a000640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc164072df0 0x7fc1641eba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15a000640 1 -- 172.21.15.27:0/866900437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16409d470 con 0x7fc1640c3ea0 2024-03-28T19:09:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15a000640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c3ea0 0x7fc1641eee90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00c820 tx=0x7fc14c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15b400640 1 -- 172.21.15.27:0/866900437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c018070 con 0x7fc1640c3ea0 2024-03-28T19:09:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15b400640 1 -- 172.21.15.27:0/866900437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c00a640 con 0x7fc1640c3ea0 2024-03-28T19:09:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc15b400640 1 -- 172.21.15.27:0/866900437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c00aa30 con 0x7fc1640c3ea0 2024-03-28T19:09:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.591+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1641ef3d0 con 0x7fc1640c3ea0 2024-03-28T19:09:50.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.592+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1640ab0b0 con 0x7fc1640c3ea0 2024-03-28T19:09:50.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.593+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16404f400 con 0x7fc1640c3ea0 2024-03-28T19:09:50.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.594+0000 7fc15b400640 1 -- 172.21.15.27:0/866900437 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc14c00abd0 con 0x7fc1640c3ea0 2024-03-28T19:09:50.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.595+0000 7fc15b400640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:50.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.595+0000 7fc15b400640 1 -- 172.21.15.27:0/866900437 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7fc14c08aa90 con 0x7fc1640c3ea0 2024-03-28T19:09:50.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.595+0000 7fc162a00640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:50.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.596+0000 7fc162a00640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc154035000 tx=0x7fc154008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:50.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.600+0000 7fc15b400640 1 -- 172.21.15.27:0/866900437 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc14c058b00 con 0x7fc1640c3ea0 2024-03-28T19:09:50.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.843+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1640c4ce0 con 0x7fc1640c3ea0 2024-03-28T19:09:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.844+0000 7fc15b400640 1 -- 172.21.15.27:0/866900437 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+4680 (secure 0 0 0) 0x7fc14c058400 con 0x7fc1640c3ea0 2024-03-28T19:09:50.843 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:50.844 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:49.907970+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:reconnect","state_seq":122,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:09:50.844 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 118 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 msgr2=0x7fc14006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc154035000 tx=0x7fc154008030 comp rx=0 tx=0).stop 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c3ea0 msgr2=0x7fc1641eee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c3ea0 0x7fc1641eee90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00c820 tx=0x7fc14c00cce0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 shutdown_connections 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc14006bbd0 0x7fc14006e090 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1640c60e0 0x7fc1640c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1640c3ea0 0x7fc1641eee90 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 --2- 172.21.15.27:0/866900437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc164072df0 0x7fc1641eba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:50.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 >> 172.21.15.27:0/866900437 conn(0x7fc16406b9d0 msgr2=0x7fc164158750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:50.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 shutdown_connections 2024-03-28T19:09:50.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:50.847+0000 7fc169600640 1 -- 172.21.15.27:0/866900437 wait complete. 2024-03-28T19:09:50.865 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=6501 2024-03-28T19:09:50.865 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before reviving mds.a 2024-03-28T19:09:51.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.883+0000 7f4f86e00640 1 Processor -- start 2024-03-28T19:09:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.883+0000 7f4f86e00640 1 -- start start 2024-03-28T19:09:51.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.885+0000 7f4f86e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4f800c5e80 0x7f4f800c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:51.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.885+0000 7f4f86e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f800abf60 0x7f4f800aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:51.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.885+0000 7f4f86e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8006d8a0 con 0x7f4f800c1f50 2024-03-28T19:09:51.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.885+0000 7f4f86e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8006d650 con 0x7f4f800abf60 2024-03-28T19:09:51.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.885+0000 7f4f86e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8006e1d0 con 0x7f4f800c5e80 2024-03-28T19:09:51.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.885+0000 7f4f84e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f800abf60 0x7f4f800aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:51.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.885+0000 7f4f7f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4f800c1f50 0x7f4f800ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34922/0 (socket says 172.21.15.27:34922) 2024-03-28T19:09:51.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.885+0000 7f4f7f400640 1 -- 172.21.15.27:0/798244698 learned_addr learned my addr 172.21.15.27:0/798244698 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:09:51.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.886+0000 7f4f7fe00640 1 --2- 172.21.15.27:0/798244698 >> v2:172.21.15.155:3300/0 conn(0x7f4f800c5e80 0x7f4f800c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.886+0000 7f4f84e00640 1 -- 172.21.15.27:0/798244698 >> v2:172.21.15.155:3300/0 conn(0x7f4f800c5e80 msgr2=0x7f4f800c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.886+0000 7f4f84e00640 1 --2- 172.21.15.27:0/798244698 >> v2:172.21.15.155:3300/0 conn(0x7f4f800c5e80 0x7f4f800c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.886+0000 7f4f84e00640 1 -- 172.21.15.27:0/798244698 >> v1:172.21.15.27:6789/0 conn(0x7f4f800c1f50 legacy=0x7f4f800ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.886+0000 7f4f84e00640 1 -- 172.21.15.27:0/798244698 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8006db80 con 0x7f4f800abf60 2024-03-28T19:09:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.886+0000 7f4f7ea00640 1 -- 172.21.15.27:0/798244698 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974098862 0 0) 0x7f4f8006d8a0 con 0x7f4f800c1f50 2024-03-28T19:09:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.887+0000 7f4f84e00640 1 --2- 172.21.15.27:0/798244698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f800aa570 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4f800c6670 tx=0x7f4f7802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f47694895852bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.887+0000 7f4f7ea00640 1 -- 172.21.15.27:0/798244698 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78039070 con 0x7f4f800abf60 2024-03-28T19:09:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.887+0000 7f4f7ea00640 1 -- 172.21.15.27:0/798244698 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7802cb60 con 0x7f4f800abf60 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.887+0000 7f4f7ea00640 1 -- 172.21.15.27:0/798244698 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78034440 con 0x7f4f800abf60 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.888+0000 7f4f86e00640 1 -- 172.21.15.27:0/798244698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 msgr2=0x7f4f800aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.888+0000 7f4f86e00640 1 --2- 172.21.15.27:0/798244698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f800aa570 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4f800c6670 tx=0x7f4f7802bdb0 comp rx=0 tx=0).stop 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.888+0000 7f4f86e00640 1 -- 172.21.15.27:0/798244698 shutdown_connections 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.888+0000 7f4f86e00640 1 --2- 172.21.15.27:0/798244698 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f800aa570 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.888+0000 7f4f86e00640 1 --2- 172.21.15.27:0/798244698 >> v2:172.21.15.155:3300/0 conn(0x7f4f800c5e80 0x7f4f800c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.888+0000 7f4f86e00640 1 -- 172.21.15.27:0/798244698 >> 172.21.15.27:0/798244698 conn(0x7f4f8006b9d0 msgr2=0x7f4f800c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.888+0000 7f4f86e00640 1 -- 172.21.15.27:0/798244698 shutdown_connections 2024-03-28T19:09:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.888+0000 7f4f86e00640 1 -- 172.21.15.27:0/798244698 wait complete. 2024-03-28T19:09:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.890+0000 7f4f86e00640 1 Processor -- start 2024-03-28T19:09:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.890+0000 7f4f86e00640 1 -- start start 2024-03-28T19:09:51.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f86e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f80152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f86e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f800c1f50 0x7f4f80151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f86e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f800c5e80 0x7f4f80152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f86e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f800c3370 con 0x7f4f800abf60 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f86e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f800c31f0 con 0x7f4f800c1f50 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f86e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f800c6260 con 0x7f4f800c5e80 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f7fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f80152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f7fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f80152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49278/0 (socket says 172.21.15.27:49278) 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f7fe00640 1 -- 172.21.15.27:0/1002524507 learned_addr learned my addr 172.21.15.27:0/1002524507 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f7f400640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f800c1f50 0x7f4f80151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f7fe00640 1 -- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f800c5e80 msgr2=0x7f4f80152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.891+0000 7f4f84e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f800c5e80 0x7f4f80152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:51.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f7fe00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f800c5e80 0x7f4f80152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f7fe00640 1 -- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f800c1f50 msgr2=0x7f4f80151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f7fe00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f800c1f50 0x7f4f80151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f7fe00640 1 -- 172.21.15.27:0/1002524507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f80151090 con 0x7f4f800abf60 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f84e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f800c5e80 0x7f4f80152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f7fe00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f80152fe0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000d620 tx=0x7f4f7000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f77e00640 1 -- 172.21.15.27:0/1002524507 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000b620 con 0x7f4f800abf60 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f77e00640 1 -- 172.21.15.27:0/1002524507 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f70019070 con 0x7f4f800abf60 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f77e00640 1 -- 172.21.15.27:0/1002524507 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000ba00 con 0x7f4f800abf60 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f80151330 con 0x7f4f800abf60 2024-03-28T19:09:51.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.892+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f801518f0 con 0x7f4f800abf60 2024-03-28T19:09:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.893+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8004f400 con 0x7f4f800abf60 2024-03-28T19:09:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.895+0000 7f4f77e00640 1 -- 172.21.15.27:0/1002524507 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4f7000bba0 con 0x7f4f800abf60 2024-03-28T19:09:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.896+0000 7f4f77e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f6006bc00 0x7f4f6006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.896+0000 7f4f77e00640 1 -- 172.21.15.27:0/1002524507 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f4f7008b870 con 0x7f4f800abf60 2024-03-28T19:09:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.896+0000 7f4f7f400640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f6006bc00 0x7f4f6006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:51.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.897+0000 7f4f7f400640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f6006bc00 0x7f4f6006e0c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4f800c2e90 tx=0x7f4f6c007b50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:51.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:51.902+0000 7f4f77e00640 1 -- 172.21.15.27:0/1002524507 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4f70059940 con 0x7f4f800abf60 2024-03-28T19:09:52.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.157+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f800c78b0 con 0x7f4f800abf60 2024-03-28T19:09:52.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.159+0000 7f4f77e00640 1 -- 172.21.15.27:0/1002524507 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4677 (secure 0 0 0) 0x7f4f70059240 con 0x7f4f800abf60 2024-03-28T19:09:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:52.159 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:50.743330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:rejoin","state_seq":123,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:09:52.159 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 119 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f6006bc00 msgr2=0x7f4f6006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f6006bc00 0x7f4f6006e0c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4f800c2e90 tx=0x7f4f6c007b50 comp rx=0 tx=0).stop 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 msgr2=0x7f4f80152fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f80152fe0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000d620 tx=0x7f4f7000dae0 comp rx=0 tx=0).stop 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 shutdown_connections 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f6006bc00 0x7f4f6006e0c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f800c5e80 0x7f4f80152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f800c1f50 0x7f4f80151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 --2- 172.21.15.27:0/1002524507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f800abf60 0x7f4f80152fe0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 >> 172.21.15.27:0/1002524507 conn(0x7f4f8006b9d0 msgr2=0x7f4f800ab620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:52.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.161+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 shutdown_connections 2024-03-28T19:09:52.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:52.162+0000 7f4f86e00640 1 -- 172.21.15.27:0/1002524507 wait complete. 2024-03-28T19:09:52.180 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:09:52.523 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:09:52.523+0000 7f13c3200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:09:54.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.434+0000 7f18d9800640 1 Processor -- start 2024-03-28T19:09:54.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.435+0000 7f18d9800640 1 -- start start 2024-03-28T19:09:54.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.436+0000 7f18d9800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d414f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:54.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.436+0000 7f18d9800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18d40c26c0 0x7f18d40c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.436+0000 7f18d9800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d4155f50 con 0x7f18d40c1ef0 2024-03-28T19:09:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.436+0000 7f18d9800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d40674e0 con 0x7f18d40c26c0 2024-03-28T19:09:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.436+0000 7f18d9800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4155ac0 con 0x7f18d4071b70 2024-03-28T19:09:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.437+0000 7f18d2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d414f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.437+0000 7f18d2a00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f18d40c1ef0 0x7f18d4066dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:46940/0 (socket says 172.21.15.27:46940) 2024-03-28T19:09:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.437+0000 7f18d2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d414f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:50422/0 (socket says 172.21.15.27:50422) 2024-03-28T19:09:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.437+0000 7f18d2a00640 1 -- 172.21.15.27:0/1229163065 learned_addr learned my addr 172.21.15.27:0/1229163065 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:09:54.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.437+0000 7f18d3400640 1 --2- 172.21.15.27:0/1229163065 >> v2:172.21.15.27:3300/0 conn(0x7f18d40c26c0 0x7f18d40c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:54.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.438+0000 7f18d2000640 1 -- 172.21.15.27:0/1229163065 >> v2:172.21.15.27:3300/0 conn(0x7f18d40c26c0 msgr2=0x7f18d40c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:54.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.438+0000 7f18d2000640 1 --2- 172.21.15.27:0/1229163065 >> v2:172.21.15.27:3300/0 conn(0x7f18d40c26c0 0x7f18d40c2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.438+0000 7f18d2000640 1 -- 172.21.15.27:0/1229163065 >> v1:172.21.15.155:6789/0 conn(0x7f18d40c1ef0 legacy=0x7f18d4066dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:54.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.438+0000 7f18d2000640 1 -- 172.21.15.27:0/1229163065 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4155190 con 0x7f18d4071b70 2024-03-28T19:09:54.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.439+0000 7f18d2000640 1 --2- 172.21.15.27:0/1229163065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d414f340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18c00097a0 tx=0x7f18c002b5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c628c0d022877b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:54.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.440+0000 7f18d1600640 1 -- 172.21.15.27:0/1229163065 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0037070 con 0x7f18d4071b70 2024-03-28T19:09:54.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.440+0000 7f18d1600640 1 -- 172.21.15.27:0/1229163065 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c0030710 con 0x7f18d4071b70 2024-03-28T19:09:54.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.440+0000 7f18d1600640 1 -- 172.21.15.27:0/1229163065 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0030ae0 con 0x7f18d4071b70 2024-03-28T19:09:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.440+0000 7f18d9800640 1 -- 172.21.15.27:0/1229163065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 msgr2=0x7f18d414f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.440+0000 7f18d9800640 1 --2- 172.21.15.27:0/1229163065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d414f340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18c00097a0 tx=0x7f18c002b5e0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.441+0000 7f18d9800640 1 -- 172.21.15.27:0/1229163065 shutdown_connections 2024-03-28T19:09:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.441+0000 7f18d9800640 1 --2- 172.21.15.27:0/1229163065 >> v2:172.21.15.27:3300/0 conn(0x7f18d40c26c0 0x7f18d40c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.441+0000 7f18d9800640 1 --2- 172.21.15.27:0/1229163065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d414f340 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.441+0000 7f18d9800640 1 -- 172.21.15.27:0/1229163065 >> 172.21.15.27:0/1229163065 conn(0x7f18d40c3ff0 msgr2=0x7f18d40c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:54.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.441+0000 7f18d9800640 1 -- 172.21.15.27:0/1229163065 shutdown_connections 2024-03-28T19:09:54.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.441+0000 7f18d9800640 1 -- 172.21.15.27:0/1229163065 wait complete. 2024-03-28T19:09:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.442+0000 7f18d9800640 1 Processor -- start 2024-03-28T19:09:54.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.443+0000 7f18d9800640 1 -- start start 2024-03-28T19:09:54.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.444+0000 7f18d9800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d413ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.444+0000 7f18d9800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40c1ef0 0x7f18d415ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.444+0000 7f18d9800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18d40c26c0 0x7f18d4140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.444+0000 7f18d9800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40712d0 con 0x7f18d40c26c0 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.444+0000 7f18d9800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4155750 con 0x7f18d40c1ef0 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.444+0000 7f18d9800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4072150 con 0x7f18d4071b70 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.444+0000 7f18d2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d413ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.444+0000 7f18d3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18d40c26c0 0x7f18d4140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.445+0000 7f18d2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d413ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:50426/0 (socket says 172.21.15.27:50426) 2024-03-28T19:09:54.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.445+0000 7f18d2a00640 1 -- 172.21.15.27:0/2536360851 learned_addr learned my addr 172.21.15.27:0/2536360851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.445+0000 7f18d2000640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40c1ef0 0x7f18d415ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.445+0000 7f18d3400640 1 -- 172.21.15.27:0/2536360851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 msgr2=0x7f18d413ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.445+0000 7f18d3400640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d413ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.445+0000 7f18d3400640 1 -- 172.21.15.27:0/2536360851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40c1ef0 msgr2=0x7f18d415ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.445+0000 7f18d3400640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40c1ef0 0x7f18d415ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.445+0000 7f18d3400640 1 -- 172.21.15.27:0/2536360851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4144a00 con 0x7f18d40c26c0 2024-03-28T19:09:54.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.446+0000 7f18d3400640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18d40c26c0 0x7f18d4140250 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00efc0 tx=0x7f18cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.446+0000 7f18caa00640 1 -- 172.21.15.27:0/2536360851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc018020 con 0x7f18d40c26c0 2024-03-28T19:09:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.446+0000 7f18caa00640 1 -- 172.21.15.27:0/2536360851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc014480 con 0x7f18d40c26c0 2024-03-28T19:09:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.446+0000 7f18caa00640 1 -- 172.21.15.27:0/2536360851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc014870 con 0x7f18d40c26c0 2024-03-28T19:09:54.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.446+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d4144ce0 con 0x7f18d40c26c0 2024-03-28T19:09:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.446+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d41f2c30 con 0x7f18d40c26c0 2024-03-28T19:09:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.448+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d404f400 con 0x7f18d40c26c0 2024-03-28T19:09:54.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.449+0000 7f18caa00640 1 -- 172.21.15.27:0/2536360851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f18cc01b020 con 0x7f18d40c26c0 2024-03-28T19:09:54.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.450+0000 7f18caa00640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18b406bc20 0x7f18b406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:54.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.450+0000 7f18caa00640 1 -- 172.21.15.27:0/2536360851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f18cc08c1c0 con 0x7f18d40c26c0 2024-03-28T19:09:54.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.451+0000 7f18d2a00640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18b406bc20 0x7f18b406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:54.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.452+0000 7f18d2a00640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18b406bc20 0x7f18b406e0e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f18c4000f60 tx=0x7f18c4000bc0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:54.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.457+0000 7f18caa00640 1 -- 172.21.15.27:0/2536360851 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f18cc022020 con 0x7f18d40c26c0 2024-03-28T19:09:54.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.817+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18d4154e40 con 0x7f18d40c26c0 2024-03-28T19:09:54.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.817+0000 7f18caa00640 1 -- 172.21.15.27:0/2536360851 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4684 (secure 0 0 0) 0x7f18cc010040 con 0x7f18d40c26c0 2024-03-28T19:09:54.818 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:54.819 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:09:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 120 2024-03-28T19:09:54.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18b406bc20 msgr2=0x7f18b406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:54.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18b406bc20 0x7f18b406e0e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f18c4000f60 tx=0x7f18c4000bc0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18d40c26c0 msgr2=0x7f18d4140250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18d40c26c0 0x7f18d4140250 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00efc0 tx=0x7f18cc00c490 comp rx=0 tx=0).stop 2024-03-28T19:09:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 shutdown_connections 2024-03-28T19:09:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18b406bc20 0x7f18b406e0e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18d40c26c0 0x7f18d4140250 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40c1ef0 0x7f18d415ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 --2- 172.21.15.27:0/2536360851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d4071b70 0x7f18d413ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.820+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 >> 172.21.15.27:0/2536360851 conn(0x7f18d40c3ff0 msgr2=0x7f18d41456f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.821+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 shutdown_connections 2024-03-28T19:09:54.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:54.821+0000 7f18d9800640 1 -- 172.21.15.27:0/2536360851 wait complete. 2024-03-28T19:09:54.843 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=6061 2024-03-28T19:09:54.843 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.c 2024-03-28T19:09:56.914 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T19:09:56.914+0000 7ff052600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:09:58.844 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-03-28T19:09:58.845 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:09:58.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:09:58.856 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:09:58.856 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:09:59.067 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:09:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.103+0000 7f1b0c600640 1 Processor -- start 2024-03-28T19:09:59.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.103+0000 7f1b0c600640 1 -- start start 2024-03-28T19:09:59.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.104+0000 7f1b0c600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b040a9cb0 0x7f1b040c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:59.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.104+0000 7f1b0c600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b041534c0 0x7f1b04153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:59.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.104+0000 7f1b0c600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b040c3d90 con 0x7f1b04150c70 2024-03-28T19:09:59.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.104+0000 7f1b0c600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b040ab4e0 con 0x7f1b041534c0 2024-03-28T19:09:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.104+0000 7f1b0c600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040ac1f0 con 0x7f1b040a9cb0 2024-03-28T19:09:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b09c00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f1b04150c70 0x7f1b040c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:45896/0 (socket says 172.21.15.27:45896) 2024-03-28T19:09:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b09c00640 1 -- 172.21.15.27:0/3434816197 learned_addr learned my addr 172.21.15.27:0/3434816197 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:09:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b09200640 1 --2- 172.21.15.27:0/3434816197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b040a9cb0 0x7f1b040c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b0a600640 1 --2- 172.21.15.27:0/3434816197 >> v2:172.21.15.27:3300/0 conn(0x7f1b041534c0 0x7f1b04153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b03e00640 1 -- 172.21.15.27:0/3434816197 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960287259 0 0) 0x7f1b040c3d90 con 0x7f1b04150c70 2024-03-28T19:09:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b03e00640 1 -- 172.21.15.27:0/3434816197 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aec003520 con 0x7f1b04150c70 2024-03-28T19:09:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b09200640 1 -- 172.21.15.27:0/3434816197 >> v2:172.21.15.27:3300/0 conn(0x7f1b041534c0 msgr2=0x7f1b04153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:09:59.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b09200640 1 --2- 172.21.15.27:0/3434816197 >> v2:172.21.15.27:3300/0 conn(0x7f1b041534c0 0x7f1b04153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.105+0000 7f1b09200640 1 -- 172.21.15.27:0/3434816197 >> v1:172.21.15.155:6789/0 conn(0x7f1b04150c70 legacy=0x7f1b040c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:59.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.106+0000 7f1b09200640 1 -- 172.21.15.27:0/3434816197 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b040ab710 con 0x7f1b040a9cb0 2024-03-28T19:09:59.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.107+0000 7f1b09200640 1 --2- 172.21.15.27:0/3434816197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b040a9cb0 0x7f1b040c3690 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097a0 tx=0x7f1af002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d741dc3c6cc00c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:59.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.107+0000 7f1b03e00640 1 -- 172.21.15.27:0/3434816197 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0031780 con 0x7f1b040a9cb0 2024-03-28T19:09:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.107+0000 7f1b03e00640 1 -- 172.21.15.27:0/3434816197 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af0037070 con 0x7f1b040a9cb0 2024-03-28T19:09:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.107+0000 7f1b03e00640 1 -- 172.21.15.27:0/3434816197 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0031b40 con 0x7f1b040a9cb0 2024-03-28T19:09:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.108+0000 7f1b0c600640 1 -- 172.21.15.27:0/3434816197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b040a9cb0 msgr2=0x7f1b040c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.108+0000 7f1b0c600640 1 --2- 172.21.15.27:0/3434816197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b040a9cb0 0x7f1b040c3690 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097a0 tx=0x7f1af002b820 comp rx=0 tx=0).stop 2024-03-28T19:09:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.108+0000 7f1b0c600640 1 -- 172.21.15.27:0/3434816197 shutdown_connections 2024-03-28T19:09:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.108+0000 7f1b0c600640 1 --2- 172.21.15.27:0/3434816197 >> v2:172.21.15.27:3300/0 conn(0x7f1b041534c0 0x7f1b04153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.108+0000 7f1b0c600640 1 --2- 172.21.15.27:0/3434816197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b040a9cb0 0x7f1b040c3690 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.108+0000 7f1b0c600640 1 -- 172.21.15.27:0/3434816197 >> 172.21.15.27:0/3434816197 conn(0x7f1b0406b9d0 msgr2=0x7f1b040c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:59.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.108+0000 7f1b0c600640 1 -- 172.21.15.27:0/3434816197 shutdown_connections 2024-03-28T19:09:59.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.108+0000 7f1b0c600640 1 -- 172.21.15.27:0/3434816197 wait complete. 2024-03-28T19:09:59.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.110+0000 7f1b0c600640 1 Processor -- start 2024-03-28T19:09:59.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.110+0000 7f1b0c600640 1 -- start start 2024-03-28T19:09:59.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b0c600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040a9cb0 0x7f1b040c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:59.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b0c600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b04150c70 0x7f1b04156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b0c600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b041534c0 0x7f1b04157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b0c600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040c1810 con 0x7f1b040a9cb0 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b0c600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040abcd0 con 0x7f1b04150c70 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b0c600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b040c6830 con 0x7f1b041534c0 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b09200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b04150c70 0x7f1b04156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b09200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b04150c70 0x7f1b04156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:35386/0 (socket says 172.21.15.27:35386) 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b09200640 1 -- 172.21.15.27:0/413352887 learned_addr learned my addr 172.21.15.27:0/413352887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b09c00640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040a9cb0 0x7f1b040c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:59.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.111+0000 7f1b0a600640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b041534c0 0x7f1b04157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.112+0000 7f1b09200640 1 -- 172.21.15.27:0/413352887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b041534c0 msgr2=0x7f1b04157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.112+0000 7f1b09200640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b041534c0 0x7f1b04157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.112+0000 7f1b09200640 1 -- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040a9cb0 msgr2=0x7f1b040c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.112+0000 7f1b09200640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040a9cb0 0x7f1b040c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.112+0000 7f1b09200640 1 -- 172.21.15.27:0/413352887 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0414de10 con 0x7f1b04150c70 2024-03-28T19:09:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.112+0000 7f1b09200640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b04150c70 0x7f1b04156e50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1af0000c00 tx=0x7f1af00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.113+0000 7f1b01600640 1 -- 172.21.15.27:0/413352887 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0032610 con 0x7f1b04150c70 2024-03-28T19:09:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.113+0000 7f1b01600640 1 -- 172.21.15.27:0/413352887 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af00327d0 con 0x7f1b04150c70 2024-03-28T19:09:59.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.113+0000 7f1b01600640 1 -- 172.21.15.27:0/413352887 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0032bc0 con 0x7f1b04150c70 2024-03-28T19:09:59.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.113+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0414e0b0 con 0x7f1b04150c70 2024-03-28T19:09:59.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.113+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b040a72d0 con 0x7f1b04150c70 2024-03-28T19:09:59.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.114+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b040a6de0 con 0x7f1b04150c70 2024-03-28T19:09:59.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.115+0000 7f1b01600640 1 -- 172.21.15.27:0/413352887 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1af004c060 con 0x7f1b04150c70 2024-03-28T19:09:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.116+0000 7f1b01600640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ae406bbd0 0x7f1ae406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:09:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.116+0000 7f1b01600640 1 -- 172.21.15.27:0/413352887 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f1af00acfb0 con 0x7f1b04150c70 2024-03-28T19:09:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.117+0000 7f1b09c00640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ae406bbd0 0x7f1ae406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:09:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.117+0000 7f1b09c00640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ae406bbd0 0x7f1ae406e090 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1af8005880 tx=0x7f1af8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:09:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.123+0000 7f1b01600640 1 -- 172.21.15.27:0/413352887 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1af0040070 con 0x7f1b04150c70 2024-03-28T19:09:59.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.459+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b0404f400 con 0x7f1b04150c70 2024-03-28T19:09:59.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.459+0000 7f1b01600640 1 -- 172.21.15.27:0/413352887 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+4684 (secure 0 0 0) 0x7f1af002f050 con 0x7f1b04150c70 2024-03-28T19:09:59.460 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:09:59.460 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:09:59.461 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 120 2024-03-28T19:09:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ae406bbd0 msgr2=0x7f1ae406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ae406bbd0 0x7f1ae406e090 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1af8005880 tx=0x7f1af8009040 comp rx=0 tx=0).stop 2024-03-28T19:09:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b04150c70 msgr2=0x7f1b04156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:09:59.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b04150c70 0x7f1b04156e50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1af0000c00 tx=0x7f1af00046c0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 shutdown_connections 2024-03-28T19:09:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ae406bbd0 0x7f1ae406e090 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b041534c0 0x7f1b04157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b04150c70 0x7f1b04156e50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 --2- 172.21.15.27:0/413352887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b040a9cb0 0x7f1b040c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:09:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 >> 172.21.15.27:0/413352887 conn(0x7f1b0406b9d0 msgr2=0x7f1b041519d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:09:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.463+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 shutdown_connections 2024-03-28T19:09:59.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:09:59.464+0000 7f1b0c600640 1 -- 172.21.15.27:0/413352887 wait complete. 2024-03-28T19:09:59.487 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:09:59.866 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-03-28T19:09:59.866 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:09:59.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:09:59.876 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:09:59.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:00.089 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:10:00.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.131+0000 7f777a600640 1 Processor -- start 2024-03-28T19:10:00.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.132+0000 7f777a600640 1 -- start start 2024-03-28T19:10:00.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.133+0000 7f777a600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7774152da0 0x7f77740c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:00.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.133+0000 7f777a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 0x7f777406bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:00.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.133+0000 7f777a600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777406b160 con 0x7f77740c2e40 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.133+0000 7f777a600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777406c260 con 0x7f7774152da0 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.133+0000 7f777a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77740c2100 con 0x7f77740732e0 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.133+0000 7f7773400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7774152da0 0x7f77740c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.134+0000 7f7773400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7774152da0 0x7f77740c2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:35396/0 (socket says 172.21.15.27:35396) 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.134+0000 7f7773400640 1 -- 172.21.15.27:0/1816087346 learned_addr learned my addr 172.21.15.27:0/1816087346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.134+0000 7f7773e00640 1 --2- 172.21.15.27:0/1816087346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 0x7f777406bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.134+0000 7f7773400640 1 -- 172.21.15.27:0/1816087346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 msgr2=0x7f777406bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.134+0000 7f7773400640 1 --2- 172.21.15.27:0/1816087346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 0x7f777406bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.134+0000 7f7773400640 1 -- 172.21.15.27:0/1816087346 >> v1:172.21.15.27:6789/0 conn(0x7f77740c2e40 legacy=0x7f777406b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:00.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.134+0000 7f7773400640 1 -- 172.21.15.27:0/1816087346 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774156450 con 0x7f7774152da0 2024-03-28T19:10:00.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.136+0000 7f7773400640 1 --2- 172.21.15.27:0/1816087346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7774152da0 0x7f77740c2900 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7764009870 tx=0x7f776402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62dcbeaa0961c8cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:00.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.136+0000 7f7772000640 1 -- 172.21.15.27:0/1816087346 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77640317c0 con 0x7f7774152da0 2024-03-28T19:10:00.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.136+0000 7f7772000640 1 -- 172.21.15.27:0/1816087346 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7764037070 con 0x7f7774152da0 2024-03-28T19:10:00.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.136+0000 7f7772000640 1 -- 172.21.15.27:0/1816087346 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764031b80 con 0x7f7774152da0 2024-03-28T19:10:00.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.137+0000 7f777a600640 1 -- 172.21.15.27:0/1816087346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7774152da0 msgr2=0x7f77740c2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:00.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.137+0000 7f777a600640 1 --2- 172.21.15.27:0/1816087346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7774152da0 0x7f77740c2900 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7764009870 tx=0x7f776402b860 comp rx=0 tx=0).stop 2024-03-28T19:10:00.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.137+0000 7f777a600640 1 -- 172.21.15.27:0/1816087346 shutdown_connections 2024-03-28T19:10:00.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.137+0000 7f777a600640 1 --2- 172.21.15.27:0/1816087346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 0x7f777406bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.137+0000 7f777a600640 1 --2- 172.21.15.27:0/1816087346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7774152da0 0x7f77740c2900 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.137+0000 7f777a600640 1 -- 172.21.15.27:0/1816087346 >> 172.21.15.27:0/1816087346 conn(0x7f77740c3ff0 msgr2=0x7f77740c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:00.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.137+0000 7f777a600640 1 -- 172.21.15.27:0/1816087346 shutdown_connections 2024-03-28T19:10:00.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.137+0000 7f777a600640 1 -- 172.21.15.27:0/1816087346 wait complete. 2024-03-28T19:10:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.139+0000 7f777a600640 1 Processor -- start 2024-03-28T19:10:00.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.140+0000 7f777a600640 1 -- start start 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f777a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 0x7f77740c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f777a600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77740c2e40 0x7f77741eb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f777a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774152da0 0x7f7774151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f777a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77741560e0 con 0x7f7774152da0 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f777a600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77740c2630 con 0x7f77740c2e40 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f777a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7774067130 con 0x7f77740732e0 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f7772a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77740c2e40 0x7f77741eb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f7773400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 0x7f77740c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f7772a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77740c2e40 0x7f77741eb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:35412/0 (socket says 172.21.15.27:35412) 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.141+0000 7f7772a00640 1 -- 172.21.15.27:0/1974429832 learned_addr learned my addr 172.21.15.27:0/1974429832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:00.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.142+0000 7f7773e00640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774152da0 0x7f7774151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:00.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.142+0000 7f7772a00640 1 -- 172.21.15.27:0/1974429832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 msgr2=0x7f77740c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.142+0000 7f7772a00640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 0x7f77740c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.142+0000 7f7772a00640 1 -- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774152da0 msgr2=0x7f7774151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.142+0000 7f7772a00640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774152da0 0x7f7774151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.142+0000 7f7772a00640 1 -- 172.21.15.27:0/1974429832 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774159a00 con 0x7f77740c2e40 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.142+0000 7f7773e00640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774152da0 0x7f7774151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.142+0000 7f7772a00640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77740c2e40 0x7f77741eb0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f776000c400 tx=0x7f776000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.143+0000 7f776b400640 1 -- 172.21.15.27:0/1974429832 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77600052d0 con 0x7f77740c2e40 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.143+0000 7f776b400640 1 -- 172.21.15.27:0/1974429832 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760005490 con 0x7f77740c2e40 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.143+0000 7f776b400640 1 -- 172.21.15.27:0/1974429832 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760007260 con 0x7f77740c2e40 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.143+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77740c64f0 con 0x7f77740c2e40 2024-03-28T19:10:00.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.143+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77740c6210 con 0x7f77740c2e40 2024-03-28T19:10:00.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.144+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777404f400 con 0x7f77740c2e40 2024-03-28T19:10:00.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.146+0000 7f776b400640 1 -- 172.21.15.27:0/1974429832 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7760007400 con 0x7f77740c2e40 2024-03-28T19:10:00.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.148+0000 7f776b400640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f775406bc50 0x7f775406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:00.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.148+0000 7f776b400640 1 -- 172.21.15.27:0/1974429832 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f776008ade0 con 0x7f77740c2e40 2024-03-28T19:10:00.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.148+0000 7f7773400640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f775406bc50 0x7f775406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:00.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.149+0000 7f7773400640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f775406bc50 0x7f775406e110 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7764036000 tx=0x7f776403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:00.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.155+0000 7f776b400640 1 -- 172.21.15.27:0/1974429832 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7760058880 con 0x7f77740c2e40 2024-03-28T19:10:00.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.507+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77740c2900 con 0x7f77740c2e40 2024-03-28T19:10:00.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.508+0000 7f776b400640 1 -- 172.21.15.27:0/1974429832 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5475 (secure 0 0 0) 0x7f776005c920 con 0x7f77740c2e40 2024-03-28T19:10:00.509 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:00.509 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:00.509 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 121 2024-03-28T19:10:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f775406bc50 msgr2=0x7f775406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f775406bc50 0x7f775406e110 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7764036000 tx=0x7f776403d040 comp rx=0 tx=0).stop 2024-03-28T19:10:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77740c2e40 msgr2=0x7f77741eb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77740c2e40 0x7f77741eb0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f776000c400 tx=0x7f776000c8c0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 shutdown_connections 2024-03-28T19:10:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f775406bc50 0x7f775406e110 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7774152da0 0x7f7774151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77740c2e40 0x7f77741eb0e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 --2- 172.21.15.27:0/1974429832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77740732e0 0x7f77740c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:00.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 >> 172.21.15.27:0/1974429832 conn(0x7f77740c3ff0 msgr2=0x7f77740723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:00.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 shutdown_connections 2024-03-28T19:10:00.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:00.511+0000 7f777a600640 1 -- 172.21.15.27:0/1974429832 wait complete. 2024-03-28T19:10:00.532 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:10:01.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:01.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:10:01.504 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:10:01.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.761+0000 7f06c8a00640 1 Processor -- start 2024-03-28T19:10:01.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.761+0000 7f06c8a00640 1 -- start start 2024-03-28T19:10:01.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.763+0000 7f06c8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06c40aebe0 0x7f06c4153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:01.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.763+0000 7f06c8a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c40c5460 con 0x7f06c40ae3d0 2024-03-28T19:10:01.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.763+0000 7f06c8a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c40c4bc0 con 0x7f06c40ae7b0 2024-03-28T19:10:01.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.763+0000 7f06c8a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c4153630 con 0x7f06c40aebe0 2024-03-28T19:10:01.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.763+0000 7f06c3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f06c40ae3d0 0x7f06c40c56b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53944/0 (socket says 172.21.15.27:53944) 2024-03-28T19:10:01.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.763+0000 7f06c3400640 1 -- 172.21.15.27:0/3301361210 learned_addr learned my addr 172.21.15.27:0/3301361210 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:10:01.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.763+0000 7f06c2a00640 1 --2- 172.21.15.27:0/3301361210 >> v2:172.21.15.27:3300/0 conn(0x7f06c40aebe0 0x7f06c4153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:01.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.764+0000 7f06c2000640 1 -- 172.21.15.27:0/3301361210 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350806794 0 0) 0x7f06c40c5460 con 0x7f06c40ae3d0 2024-03-28T19:10:01.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.764+0000 7f06c2000640 1 -- 172.21.15.27:0/3301361210 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a8003520 con 0x7f06c40ae3d0 2024-03-28T19:10:01.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.764+0000 7f06c2a00640 1 -- 172.21.15.27:0/3301361210 >> v1:172.21.15.155:6789/0 conn(0x7f06c40ae7b0 legacy=0x7f06c415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:01.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.764+0000 7f06c2a00640 1 -- 172.21.15.27:0/3301361210 >> v1:172.21.15.27:6789/0 conn(0x7f06c40ae3d0 legacy=0x7f06c40c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:01.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.764+0000 7f06c2000640 1 -- 172.21.15.27:0/3301361210 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1126555603 0 0) 0x7f06a8003520 con 0x7f06c40ae3d0 2024-03-28T19:10:01.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.764+0000 7f06c2a00640 1 -- 172.21.15.27:0/3301361210 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c40c4ea0 con 0x7f06c40aebe0 2024-03-28T19:10:01.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.766+0000 7f06c2a00640 1 --2- 172.21.15.27:0/3301361210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40aebe0 0x7f06c4153ac0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f06b00097a0 tx=0x7f06b002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=febc768893263609 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:01.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.766+0000 7f06c2000640 1 -- 172.21.15.27:0/3301361210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0038070 con 0x7f06c40aebe0 2024-03-28T19:10:01.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.766+0000 7f06c2000640 1 -- 172.21.15.27:0/3301361210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b0031500 con 0x7f06c40aebe0 2024-03-28T19:10:01.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.766+0000 7f06c2000640 1 -- 172.21.15.27:0/3301361210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4001670 con 0x7f06c40aebe0 2024-03-28T19:10:01.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.767+0000 7f06c8a00640 1 -- 172.21.15.27:0/3301361210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40aebe0 msgr2=0x7f06c4153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:01.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.767+0000 7f06c8a00640 1 --2- 172.21.15.27:0/3301361210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40aebe0 0x7f06c4153ac0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f06b00097a0 tx=0x7f06b002b820 comp rx=0 tx=0).stop 2024-03-28T19:10:01.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.767+0000 7f06c8a00640 1 -- 172.21.15.27:0/3301361210 shutdown_connections 2024-03-28T19:10:01.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.767+0000 7f06c8a00640 1 --2- 172.21.15.27:0/3301361210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40aebe0 0x7f06c4153ac0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:01.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.767+0000 7f06c8a00640 1 -- 172.21.15.27:0/3301361210 >> 172.21.15.27:0/3301361210 conn(0x7f06c4072d10 msgr2=0x7f06c406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:01.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.767+0000 7f06c8a00640 1 -- 172.21.15.27:0/3301361210 shutdown_connections 2024-03-28T19:10:01.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.767+0000 7f06c8a00640 1 -- 172.21.15.27:0/3301361210 wait complete. 2024-03-28T19:10:01.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.769+0000 7f06c8a00640 1 Processor -- start 2024-03-28T19:10:01.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.769+0000 7f06c8a00640 1 -- start start 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.770+0000 7f06c8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40ae3d0 0x7f06c406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.770+0000 7f06c8a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c40ae7b0 0x7f06c40c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.770+0000 7f06c8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06c40aebe0 0x7f06c414ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.770+0000 7f06c8a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40c5dc0 con 0x7f06c40ae3d0 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.770+0000 7f06c8a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4070890 con 0x7f06c40ae7b0 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.770+0000 7f06c8a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40c5f40 con 0x7f06c40aebe0 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.771+0000 7f06c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40ae3d0 0x7f06c406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.771+0000 7f06c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40ae3d0 0x7f06c406c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54970/0 (socket says 172.21.15.27:54970) 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.771+0000 7f06c3400640 1 -- 172.21.15.27:0/3596067879 learned_addr learned my addr 172.21.15.27:0/3596067879 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.771+0000 7f06c2a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c40ae7b0 0x7f06c40c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:01.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.771+0000 7f06c3e00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06c40aebe0 0x7f06c414ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:01.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.771+0000 7f06c3400640 1 -- 172.21.15.27:0/3596067879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06c40aebe0 msgr2=0x7f06c414ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:01.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.771+0000 7f06c3400640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06c40aebe0 0x7f06c414ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:01.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.772+0000 7f06c3400640 1 -- 172.21.15.27:0/3596067879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c40ae7b0 msgr2=0x7f06c40c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:01.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.772+0000 7f06c3400640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c40ae7b0 0x7f06c40c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:01.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.772+0000 7f06c3400640 1 -- 172.21.15.27:0/3596067879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c40c6b10 con 0x7f06c40ae3d0 2024-03-28T19:10:01.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.772+0000 7f06c3400640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40ae3d0 0x7f06c406c4b0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f06b400c700 tx=0x7f06b400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:01.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.772+0000 7f06bb400640 1 -- 172.21.15.27:0/3596067879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4005ad0 con 0x7f06c40ae3d0 2024-03-28T19:10:01.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.773+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c40c6dd0 con 0x7f06c40ae3d0 2024-03-28T19:10:01.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.773+0000 7f06bb400640 1 -- 172.21.15.27:0/3596067879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b4005c70 con 0x7f06c40ae3d0 2024-03-28T19:10:01.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.773+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c406bcb0 con 0x7f06c40ae3d0 2024-03-28T19:10:01.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.773+0000 7f06bb400640 1 -- 172.21.15.27:0/3596067879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4005e60 con 0x7f06c40ae3d0 2024-03-28T19:10:01.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.775+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c005290 con 0x7f06c40ae3d0 2024-03-28T19:10:01.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.776+0000 7f06bb400640 1 -- 172.21.15.27:0/3596067879 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f06b4007290 con 0x7f06c40ae3d0 2024-03-28T19:10:01.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.777+0000 7f06bb400640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f06a406bbd0 0x7f06a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:01.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.777+0000 7f06bb400640 1 -- 172.21.15.27:0/3596067879 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f06b408b140 con 0x7f06c40ae3d0 2024-03-28T19:10:01.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.777+0000 7f06c2a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f06a406bbd0 0x7f06a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.778+0000 7f06c2a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f06a406bbd0 0x7f06a406e090 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f06b0035000 tx=0x7f06b0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:01.782+0000 7f06bb400640 1 -- 172.21.15.27:0/3596067879 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f06b4058b30 con 0x7f06c40ae3d0 2024-03-28T19:10:02.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.088+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f069c005580 con 0x7f06c40ae3d0 2024-03-28T19:10:02.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.089+0000 7f06bb400640 1 -- 172.21.15.27:0/3596067879 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f06b405cbd0 con 0x7f06c40ae3d0 2024-03-28T19:10:02.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:02.089 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":6846,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:02.089 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-28T19:10:02.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.092+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f06a406bbd0 msgr2=0x7f06a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:02.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.092+0000 7f06c8a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f06a406bbd0 0x7f06a406e090 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f06b0035000 tx=0x7f06b0008030 comp rx=0 tx=0).stop 2024-03-28T19:10:02.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.092+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40ae3d0 msgr2=0x7f06c406c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:02.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.092+0000 7f06c8a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40ae3d0 0x7f06c406c4b0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f06b400c700 tx=0x7f06b400cbc0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.093+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 shutdown_connections 2024-03-28T19:10:02.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.093+0000 7f06c8a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f06a406bbd0 0x7f06a406e090 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.093+0000 7f06c8a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06c40aebe0 0x7f06c414ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.093+0000 7f06c8a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c40ae7b0 0x7f06c40c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.093+0000 7f06c8a00640 1 --2- 172.21.15.27:0/3596067879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c40ae3d0 0x7f06c406c4b0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.093+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 >> 172.21.15.27:0/3596067879 conn(0x7f06c4072d10 msgr2=0x7f06c4155c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:02.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.093+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 shutdown_connections 2024-03-28T19:10:02.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.093+0000 7f06c8a00640 1 -- 172.21.15.27:0/3596067879 wait complete. 2024-03-28T19:10:02.116 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-03-28T19:10:02.116 INFO:tasks.mds_thrash.fs.[a]:waiting for 119 secs before thrashing 2024-03-28T19:10:02.533 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:02.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.787+0000 7f4b6f200640 1 Processor -- start 2024-03-28T19:10:02.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.787+0000 7f4b6f200640 1 -- start start 2024-03-28T19:10:02.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b700c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:02.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6f200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4b70073180 0x7f4b70155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b70155fe0 0x7f4b700c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700ab1e0 con 0x7f4b70155fe0 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6f200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b70152d30 con 0x7f4b70073180 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70153010 con 0x7f4b700c74e0 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b70155fe0 0x7f4b700c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6de00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b700c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6de00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b700c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:53468/0 (socket says 172.21.15.27:53468) 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6de00640 1 -- 172.21.15.27:0/3839333234 learned_addr learned my addr 172.21.15.27:0/3839333234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b70155fe0 0x7f4b700c1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54986/0 (socket says 172.21.15.27:54986) 2024-03-28T19:10:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.789+0000 7f4b6d400640 1 --2- 172.21.15.27:0/3839333234 >> v2:172.21.15.155:3300/0 conn(0x7f4b70073180 0x7f4b70155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.790+0000 7f4b6de00640 1 -- 172.21.15.27:0/3839333234 >> v2:172.21.15.155:3300/0 conn(0x7f4b70073180 msgr2=0x7f4b70155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.790+0000 7f4b6de00640 1 --2- 172.21.15.27:0/3839333234 >> v2:172.21.15.155:3300/0 conn(0x7f4b70073180 0x7f4b70155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.790+0000 7f4b6de00640 1 -- 172.21.15.27:0/3839333234 >> v2:172.21.15.27:3300/0 conn(0x7f4b70155fe0 msgr2=0x7f4b700c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.790+0000 7f4b6de00640 1 --2- 172.21.15.27:0/3839333234 >> v2:172.21.15.27:3300/0 conn(0x7f4b70155fe0 0x7f4b700c1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.790+0000 7f4b6de00640 1 -- 172.21.15.27:0/3839333234 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700aae70 con 0x7f4b700c74e0 2024-03-28T19:10:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.790+0000 7f4b6e800640 1 --2- 172.21.15.27:0/3839333234 >> v2:172.21.15.27:3300/0 conn(0x7f4b70155fe0 0x7f4b700c1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:10:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.792+0000 7f4b6de00640 1 --2- 172.21.15.27:0/3839333234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b700c2e50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b700aa0a0 tx=0x7f4b6002b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e97968589d98cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.792+0000 7f4b6ca00640 1 -- 172.21.15.27:0/3839333234 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6002e070 con 0x7f4b700c74e0 2024-03-28T19:10:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.792+0000 7f4b6ca00640 1 -- 172.21.15.27:0/3839333234 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60037070 con 0x7f4b700c74e0 2024-03-28T19:10:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.792+0000 7f4b6ca00640 1 -- 172.21.15.27:0/3839333234 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60031a00 con 0x7f4b700c74e0 2024-03-28T19:10:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 -- 172.21.15.27:0/3839333234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 msgr2=0x7f4b700c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 --2- 172.21.15.27:0/3839333234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b700c2e50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b700aa0a0 tx=0x7f4b6002b880 comp rx=0 tx=0).stop 2024-03-28T19:10:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 -- 172.21.15.27:0/3839333234 shutdown_connections 2024-03-28T19:10:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 --2- 172.21.15.27:0/3839333234 >> v2:172.21.15.27:3300/0 conn(0x7f4b70155fe0 0x7f4b700c1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 --2- 172.21.15.27:0/3839333234 >> v2:172.21.15.155:3300/0 conn(0x7f4b70073180 0x7f4b70155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 --2- 172.21.15.27:0/3839333234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b700c2e50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 -- 172.21.15.27:0/3839333234 >> 172.21.15.27:0/3839333234 conn(0x7f4b7006d980 msgr2=0x7f4b700a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 -- 172.21.15.27:0/3839333234 shutdown_connections 2024-03-28T19:10:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.793+0000 7f4b6f200640 1 -- 172.21.15.27:0/3839333234 wait complete. 2024-03-28T19:10:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.795+0000 7f4b6f200640 1 Processor -- start 2024-03-28T19:10:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.795+0000 7f4b6f200640 1 -- start start 2024-03-28T19:10:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.796+0000 7f4b6f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b70073180 0x7f4b70144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.796+0000 7f4b6f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b70071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.796+0000 7f4b6f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b70155fe0 0x7f4b70149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.796+0000 7f4b6f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70156450 con 0x7f4b70073180 2024-03-28T19:10:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.796+0000 7f4b6f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700c7ad0 con 0x7f4b70155fe0 2024-03-28T19:10:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.796+0000 7f4b6f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701565d0 con 0x7f4b700c74e0 2024-03-28T19:10:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.797+0000 7f4b6de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b70073180 0x7f4b70144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.797+0000 7f4b6d400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b70071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.797+0000 7f4b6d400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b70071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:53476/0 (socket says 172.21.15.27:53476) 2024-03-28T19:10:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.797+0000 7f4b6d400640 1 -- 172.21.15.27:0/2054259390 learned_addr learned my addr 172.21.15.27:0/2054259390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.797+0000 7f4b6e800640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b70155fe0 0x7f4b70149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.798+0000 7f4b6de00640 1 -- 172.21.15.27:0/2054259390 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 msgr2=0x7f4b70071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.798+0000 7f4b6de00640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b70071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.798+0000 7f4b6de00640 1 -- 172.21.15.27:0/2054259390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b70155fe0 msgr2=0x7f4b70149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.798+0000 7f4b6de00640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b70155fe0 0x7f4b70149920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:02.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.798+0000 7f4b6de00640 1 -- 172.21.15.27:0/2054259390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b701ebdf0 con 0x7f4b70073180 2024-03-28T19:10:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.799+0000 7f4b6de00640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b70073180 0x7f4b70144130 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f4b60033000 tx=0x7f4b600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.799+0000 7f4b66000640 1 -- 172.21.15.27:0/2054259390 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60032670 con 0x7f4b70073180 2024-03-28T19:10:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.799+0000 7f4b66000640 1 -- 172.21.15.27:0/2054259390 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60032850 con 0x7f4b70073180 2024-03-28T19:10:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.799+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b701ec0d0 con 0x7f4b70073180 2024-03-28T19:10:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.799+0000 7f4b66000640 1 -- 172.21.15.27:0/2054259390 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60032c40 con 0x7f4b70073180 2024-03-28T19:10:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.799+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b70149e60 con 0x7f4b70073180 2024-03-28T19:10:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.801+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b48005290 con 0x7f4b70073180 2024-03-28T19:10:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.802+0000 7f4b66000640 1 -- 172.21.15.27:0/2054259390 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4b6004c060 con 0x7f4b70073180 2024-03-28T19:10:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.802+0000 7f4b66000640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b5006bc00 0x7f4b5006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.803+0000 7f4b66000640 1 -- 172.21.15.27:0/2054259390 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f4b600acc00 con 0x7f4b70073180 2024-03-28T19:10:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.803+0000 7f4b6d400640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b5006bc00 0x7f4b5006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:02.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.804+0000 7f4b6d400640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b5006bc00 0x7f4b5006e0c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4b701473e0 tx=0x7f4b5c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:02.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:02.810+0000 7f4b66000640 1 -- 172.21.15.27:0/2054259390 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4b60040070 con 0x7f4b70073180 2024-03-28T19:10:03.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.166+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b48005580 con 0x7f4b70073180 2024-03-28T19:10:03.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.167+0000 7f4b66000640 1 -- 172.21.15.27:0/2054259390 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f4b6007a6b0 con 0x7f4b70073180 2024-03-28T19:10:03.167 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:03.167 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":6846,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:03.167 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-28T19:10:03.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.170+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b5006bc00 msgr2=0x7f4b5006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:03.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.170+0000 7f4b6f200640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b5006bc00 0x7f4b5006e0c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4b701473e0 tx=0x7f4b5c009190 comp rx=0 tx=0).stop 2024-03-28T19:10:03.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.170+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b70073180 msgr2=0x7f4b70144130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.170+0000 7f4b6f200640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b70073180 0x7f4b70144130 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f4b60033000 tx=0x7f4b600046c0 comp rx=0 tx=0).stop 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.171+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 shutdown_connections 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.171+0000 7f4b6f200640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b5006bc00 0x7f4b5006e0c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.171+0000 7f4b6f200640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b70155fe0 0x7f4b70149920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.171+0000 7f4b6f200640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b700c74e0 0x7f4b70071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.171+0000 7f4b6f200640 1 --2- 172.21.15.27:0/2054259390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b70073180 0x7f4b70144130 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.171+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 >> 172.21.15.27:0/2054259390 conn(0x7f4b7006d980 msgr2=0x7f4b70149160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.171+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 shutdown_connections 2024-03-28T19:10:03.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:03.171+0000 7f4b6f200640 1 -- 172.21.15.27:0/2054259390 wait complete. 2024-03-28T19:10:03.192 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-03-28T19:10:03.192 INFO:tasks.mds_thrash.fs.[b]:waiting for 44 secs before thrashing 2024-03-28T19:10:31.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:10:31.661 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:10:47.193 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.418+0000 7faf6d200640 1 Processor -- start 2024-03-28T19:10:47.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.419+0000 7faf6d200640 1 -- start start 2024-03-28T19:10:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.420+0000 7faf6d200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7faf6806ce20 0x7faf680a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.420+0000 7faf6d200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf680c9a40 0x7faf680c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.420+0000 7faf6d200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf68159bf0 con 0x7faf68159760 2024-03-28T19:10:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.420+0000 7faf6d200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf680c5040 con 0x7faf6806ce20 2024-03-28T19:10:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.420+0000 7faf6d200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf680c74b0 con 0x7faf680c9a40 2024-03-28T19:10:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.421+0000 7faf65400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faf68159760 0x7faf680c8cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58156/0 (socket says 172.21.15.27:58156) 2024-03-28T19:10:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.421+0000 7faf65400640 1 -- 172.21.15.27:0/3706156381 learned_addr learned my addr 172.21.15.27:0/3706156381 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:10:47.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.421+0000 7faf66800640 1 --2- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf680c9a40 0x7faf680c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.421+0000 7faf65e00640 1 --2- 172.21.15.27:0/3706156381 >> v2:172.21.15.155:3300/0 conn(0x7faf6806ce20 0x7faf680a7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.422+0000 7faf65e00640 1 -- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf680c9a40 msgr2=0x7faf680c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.422+0000 7faf65e00640 1 --2- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf680c9a40 0x7faf680c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.422+0000 7faf65e00640 1 -- 172.21.15.27:0/3706156381 >> v1:172.21.15.27:6789/0 conn(0x7faf68159760 legacy=0x7faf680c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.422+0000 7faf65e00640 1 -- 172.21.15.27:0/3706156381 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf680c7790 con 0x7faf6806ce20 2024-03-28T19:10:47.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.422+0000 7faf66800640 1 --2- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf680c9a40 0x7faf680c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:10:47.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.423+0000 7faf65e00640 1 --2- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6806ce20 0x7faf680a7b00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7faf580097a0 tx=0x7faf5802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c011a80f8e1dd87c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.426+0000 7faf64a00640 1 -- 172.21.15.27:0/3706156381 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58031780 con 0x7faf6806ce20 2024-03-28T19:10:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.426+0000 7faf64a00640 1 -- 172.21.15.27:0/3706156381 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf58037070 con 0x7faf6806ce20 2024-03-28T19:10:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.426+0000 7faf64a00640 1 -- 172.21.15.27:0/3706156381 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58031b40 con 0x7faf6806ce20 2024-03-28T19:10:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.426+0000 7faf6d200640 1 -- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6806ce20 msgr2=0x7faf680a7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.426+0000 7faf6d200640 1 --2- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6806ce20 0x7faf680a7b00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7faf580097a0 tx=0x7faf5802b820 comp rx=0 tx=0).stop 2024-03-28T19:10:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.427+0000 7faf6d200640 1 -- 172.21.15.27:0/3706156381 shutdown_connections 2024-03-28T19:10:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.427+0000 7faf6d200640 1 --2- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf680c9a40 0x7faf680c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.427+0000 7faf6d200640 1 --2- 172.21.15.27:0/3706156381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6806ce20 0x7faf680a7b00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.427+0000 7faf6d200640 1 -- 172.21.15.27:0/3706156381 >> 172.21.15.27:0/3706156381 conn(0x7faf680aaab0 msgr2=0x7faf680aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.427+0000 7faf6d200640 1 -- 172.21.15.27:0/3706156381 shutdown_connections 2024-03-28T19:10:47.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.427+0000 7faf6d200640 1 -- 172.21.15.27:0/3706156381 wait complete. 2024-03-28T19:10:47.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.428+0000 7faf6d200640 1 Processor -- start 2024-03-28T19:10:47.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.429+0000 7faf6d200640 1 -- start start 2024-03-28T19:10:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.429+0000 7faf6d200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf6806ce20 0x7faf681533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.429+0000 7faf6d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf680c9a40 0x7faf68151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf6d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6814f270 0x7faf68157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf6d200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf680c9560 con 0x7faf680c9a40 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf6d200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf680c93e0 con 0x7faf6814f270 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf6d200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf680c4290 con 0x7faf6806ce20 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf680c9a40 0x7faf68151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf680c9a40 0x7faf68151a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34260/0 (socket says 172.21.15.27:34260) 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65400640 1 -- 172.21.15.27:0/651588320 learned_addr learned my addr 172.21.15.27:0/651588320 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf66800640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6814f270 0x7faf68157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65e00640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf6806ce20 0x7faf681533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65400640 1 -- 172.21.15.27:0/651588320 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf6806ce20 msgr2=0x7faf681533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65400640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf6806ce20 0x7faf681533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65400640 1 -- 172.21.15.27:0/651588320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6814f270 msgr2=0x7faf68157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65400640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6814f270 0x7faf68157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.430+0000 7faf65400640 1 -- 172.21.15.27:0/651588320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6814f720 con 0x7faf680c9a40 2024-03-28T19:10:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.431+0000 7faf65400640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf680c9a40 0x7faf68151a70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7faf5400c470 tx=0x7faf5400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.431+0000 7faf5e000640 1 -- 172.21.15.27:0/651588320 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf54017070 con 0x7faf680c9a40 2024-03-28T19:10:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.431+0000 7faf5e000640 1 -- 172.21.15.27:0/651588320 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf54005ce0 con 0x7faf680c9a40 2024-03-28T19:10:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.431+0000 7faf5e000640 1 -- 172.21.15.27:0/651588320 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf540072d0 con 0x7faf680c9a40 2024-03-28T19:10:47.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.431+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf68150990 con 0x7faf680c9a40 2024-03-28T19:10:47.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.431+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6814ef00 con 0x7faf680c9a40 2024-03-28T19:10:47.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.433+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf680a92b0 con 0x7faf680c9a40 2024-03-28T19:10:47.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.433+0000 7faf5e000640 1 -- 172.21.15.27:0/651588320 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faf5401a070 con 0x7faf680c9a40 2024-03-28T19:10:47.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.434+0000 7faf5e000640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf4806bbd0 0x7faf4806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.434+0000 7faf5e000640 1 -- 172.21.15.27:0/651588320 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7faf5408a680 con 0x7faf680c9a40 2024-03-28T19:10:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.435+0000 7faf65e00640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf4806bbd0 0x7faf4806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.435+0000 7faf65e00640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf4806bbd0 0x7faf4806e090 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faf58036000 tx=0x7faf5803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.439+0000 7faf5e000640 1 -- 172.21.15.27:0/651588320 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faf680a92b0 con 0x7faf680c9a40 2024-03-28T19:10:47.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.688+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf6804f400 con 0x7faf680c9a40 2024-03-28T19:10:47.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.689+0000 7faf5e000640 1 -- 172.21.15.27:0/651588320 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6264 (secure 0 0 0) 0x7faf540580a0 con 0x7faf680c9a40 2024-03-28T19:10:47.689 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:47.689 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":6846,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:47.689 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf4806bbd0 msgr2=0x7faf4806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf4806bbd0 0x7faf4806e090 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faf58036000 tx=0x7faf5803d040 comp rx=0 tx=0).stop 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf680c9a40 msgr2=0x7faf68151a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf680c9a40 0x7faf68151a70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7faf5400c470 tx=0x7faf5400c870 comp rx=0 tx=0).stop 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 shutdown_connections 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf4806bbd0 0x7faf4806e090 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf6814f270 0x7faf68157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf680c9a40 0x7faf68151a70 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 --2- 172.21.15.27:0/651588320 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf6806ce20 0x7faf681533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.692+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 >> 172.21.15.27:0/651588320 conn(0x7faf680aaab0 msgr2=0x7faf680aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:47.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.693+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 shutdown_connections 2024-03-28T19:10:47.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.693+0000 7faf6d200640 1 -- 172.21.15.27:0/651588320 wait complete. 2024-03-28T19:10:47.712 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-03-28T19:10:47.713 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:10:47.713 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:10:47.736 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:10:47.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:47.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.955+0000 7f58bbe00640 1 Processor -- start 2024-03-28T19:10:47.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.955+0000 7f58bbe00640 1 -- start start 2024-03-28T19:10:47.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.957+0000 7f58bbe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc152a70 0x7f58bc159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.957+0000 7f58bbe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0c5710 con 0x7f58bc155b40 2024-03-28T19:10:47.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.957+0000 7f58bbe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0c59f0 con 0x7f58bc0c6cc0 2024-03-28T19:10:47.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.957+0000 7f58bbe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1556b0 con 0x7f58bc152a70 2024-03-28T19:10:47.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58ba000640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f58bc0c6cc0 0x7f58bc0cb0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:39814/0 (socket says 172.21.15.27:39814) 2024-03-28T19:10:47.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58ba000640 1 -- 172.21.15.27:0/938055945 learned_addr learned my addr 172.21.15.27:0/938055945 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:10:47.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58bb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f58bc155b40 0x7f58bc155fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58166/0 (socket says 172.21.15.27:58166) 2024-03-28T19:10:47.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58baa00640 1 --2- 172.21.15.27:0/938055945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc152a70 0x7f58bc159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58b9600640 1 -- 172.21.15.27:0/938055945 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738352065 0 0) 0x7f58bc0c5710 con 0x7f58bc155b40 2024-03-28T19:10:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58b9600640 1 -- 172.21.15.27:0/938055945 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a0003520 con 0x7f58bc155b40 2024-03-28T19:10:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58b9600640 1 -- 172.21.15.27:0/938055945 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11958959 0 0) 0x7f58bc0c59f0 con 0x7f58bc0c6cc0 2024-03-28T19:10:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58b9600640 1 -- 172.21.15.27:0/938055945 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58bc0c5710 con 0x7f58bc0c6cc0 2024-03-28T19:10:47.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58baa00640 1 -- 172.21.15.27:0/938055945 >> v1:172.21.15.155:6789/0 conn(0x7f58bc0c6cc0 legacy=0x7f58bc0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58baa00640 1 -- 172.21.15.27:0/938055945 >> v1:172.21.15.27:6789/0 conn(0x7f58bc155b40 legacy=0x7f58bc155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.958+0000 7f58baa00640 1 -- 172.21.15.27:0/938055945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc0c5cd0 con 0x7f58bc152a70 2024-03-28T19:10:47.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.960+0000 7f58baa00640 1 --2- 172.21.15.27:0/938055945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc152a70 0x7f58bc159580 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f58bc155830 tx=0x7f58ac02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ea6da2d1de275e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:47.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.960+0000 7f58b9600640 1 -- 172.21.15.27:0/938055945 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac035020 con 0x7f58bc152a70 2024-03-28T19:10:47.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.960+0000 7f58b9600640 1 -- 172.21.15.27:0/938055945 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac031800 con 0x7f58bc152a70 2024-03-28T19:10:47.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.960+0000 7f58b9600640 1 -- 172.21.15.27:0/938055945 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac031bd0 con 0x7f58bc152a70 2024-03-28T19:10:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.961+0000 7f58bbe00640 1 -- 172.21.15.27:0/938055945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc152a70 msgr2=0x7f58bc159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.961+0000 7f58bbe00640 1 --2- 172.21.15.27:0/938055945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc152a70 0x7f58bc159580 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f58bc155830 tx=0x7f58ac02b8a0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.961+0000 7f58bbe00640 1 -- 172.21.15.27:0/938055945 shutdown_connections 2024-03-28T19:10:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.961+0000 7f58bbe00640 1 --2- 172.21.15.27:0/938055945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc152a70 0x7f58bc159580 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.961+0000 7f58bbe00640 1 -- 172.21.15.27:0/938055945 >> 172.21.15.27:0/938055945 conn(0x7f58bc06b2a0 msgr2=0x7f58bc06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.962+0000 7f58bbe00640 1 -- 172.21.15.27:0/938055945 shutdown_connections 2024-03-28T19:10:47.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.962+0000 7f58bbe00640 1 -- 172.21.15.27:0/938055945 wait complete. 2024-03-28T19:10:47.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.964+0000 7f58bbe00640 1 Processor -- start 2024-03-28T19:10:47.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.964+0000 7f58bbe00640 1 -- start start 2024-03-28T19:10:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58bbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58bc0c6cc0 0x7f58bc157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58bc152a70 0x7f58bc158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58bbe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc155b40 0x7f58bc156e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58bbe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0cb970 con 0x7f58bc152a70 2024-03-28T19:10:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58bbe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0cb7f0 con 0x7f58bc0c6cc0 2024-03-28T19:10:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58bbe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0c8dd0 con 0x7f58bc155b40 2024-03-28T19:10:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58ba000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58bc152a70 0x7f58bc158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58baa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58bc0c6cc0 0x7f58bc157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58bb400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc155b40 0x7f58bc156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58ba000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58bc152a70 0x7f58bc158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34266/0 (socket says 172.21.15.27:34266) 2024-03-28T19:10:47.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.965+0000 7f58ba000640 1 -- 172.21.15.27:0/3823948453 learned_addr learned my addr 172.21.15.27:0/3823948453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58ba000640 1 -- 172.21.15.27:0/3823948453 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc155b40 msgr2=0x7f58bc156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58ba000640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc155b40 0x7f58bc156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58ba000640 1 -- 172.21.15.27:0/3823948453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58bc0c6cc0 msgr2=0x7f58bc157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58ba000640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58bc0c6cc0 0x7f58bc157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58ba000640 1 -- 172.21.15.27:0/3823948453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc153fb0 con 0x7f58bc152a70 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58baa00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58bc0c6cc0 0x7f58bc157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58ba000640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58bc152a70 0x7f58bc158d90 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f58a800c530 tx=0x7f58a800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58b2a00640 1 -- 172.21.15.27:0/3823948453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8017070 con 0x7f58bc152a70 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58b2a00640 1 -- 172.21.15.27:0/3823948453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a8005ce0 con 0x7f58bc152a70 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58b2a00640 1 -- 172.21.15.27:0/3823948453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a80072b0 con 0x7f58bc152a70 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.966+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc157610 con 0x7f58bc152a70 2024-03-28T19:10:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.967+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc154530 con 0x7f58bc152a70 2024-03-28T19:10:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.968+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc067dd0 con 0x7f58bc152a70 2024-03-28T19:10:47.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.969+0000 7f58b2a00640 1 -- 172.21.15.27:0/3823948453 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f58a801a070 con 0x7f58bc152a70 2024-03-28T19:10:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.970+0000 7f58b2a00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f589c06bbd0 0x7f589c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.970+0000 7f58b2a00640 1 -- 172.21.15.27:0/3823948453 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f58a808a850 con 0x7f58bc152a70 2024-03-28T19:10:47.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.970+0000 7f58baa00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f589c06bbd0 0x7f589c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:47.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.971+0000 7f58baa00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f589c06bbd0 0x7f589c06e090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f58ac036000 tx=0x7f58ac008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:47.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:47.975+0000 7f58b2a00640 1 -- 172.21.15.27:0/3823948453 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f58a8058330 con 0x7f58bc152a70 2024-03-28T19:10:48.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.261+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58bc069640 con 0x7f58bc152a70 2024-03-28T19:10:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.262+0000 7f58b2a00640 1 -- 172.21.15.27:0/3823948453 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f58a805c3d0 con 0x7f58bc152a70 2024-03-28T19:10:48.262 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:48.262 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":6846,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-28T19:10:48.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f589c06bbd0 msgr2=0x7f589c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f589c06bbd0 0x7f589c06e090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f58ac036000 tx=0x7f58ac008030 comp rx=0 tx=0).stop 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58bc152a70 msgr2=0x7f58bc158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58bc152a70 0x7f58bc158d90 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f58a800c530 tx=0x7f58a800c9f0 comp rx=0 tx=0).stop 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 shutdown_connections 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f589c06bbd0 0x7f589c06e090 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58bc155b40 0x7f58bc156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58bc152a70 0x7f58bc158d90 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 --2- 172.21.15.27:0/3823948453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58bc0c6cc0 0x7f58bc157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 >> 172.21.15.27:0/3823948453 conn(0x7f58bc06b2a0 msgr2=0x7f58bc0c81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 shutdown_connections 2024-03-28T19:10:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:48.265+0000 7f58bbe00640 1 -- 172.21.15.27:0/3823948453 wait complete. 2024-03-28T19:10:48.288 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:10:50.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.521+0000 7f3f58c00640 1 Processor -- start 2024-03-28T19:10:50.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.521+0000 7f3f58c00640 1 -- start start 2024-03-28T19:10:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.522+0000 7f3f58c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54159650 0x7f3f540c1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.523+0000 7f3f58c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f54155d60 0x7f3f541561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.523+0000 7f3f58c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f540c7a10 con 0x7f3f54158cb0 2024-03-28T19:10:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.523+0000 7f3f58c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f540c7170 con 0x7f3f54155d60 2024-03-28T19:10:50.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.523+0000 7f3f58c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540c7450 con 0x7f3f54159650 2024-03-28T19:10:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.523+0000 7f3f52800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f54155d60 0x7f3f541561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.523+0000 7f3f52800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3f54155d60 0x7f3f541561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34268/0 (socket says 172.21.15.27:34268) 2024-03-28T19:10:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.523+0000 7f3f52800640 1 -- 172.21.15.27:0/1537731404 learned_addr learned my addr 172.21.15.27:0/1537731404 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.523+0000 7f3f51e00640 1 --2- 172.21.15.27:0/1537731404 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54159650 0x7f3f540c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.524+0000 7f3f52800640 1 -- 172.21.15.27:0/1537731404 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54159650 msgr2=0x7f3f540c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.524+0000 7f3f52800640 1 --2- 172.21.15.27:0/1537731404 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54159650 0x7f3f540c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.524+0000 7f3f52800640 1 -- 172.21.15.27:0/1537731404 >> v1:172.21.15.155:6789/0 conn(0x7f3f54158cb0 legacy=0x7f3f5406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.524+0000 7f3f52800640 1 -- 172.21.15.27:0/1537731404 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540c7730 con 0x7f3f54155d60 2024-03-28T19:10:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.525+0000 7f3f52800640 1 --2- 172.21.15.27:0/1537731404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541561f0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009870 tx=0x7f3f4402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631a8ac6bab15ec7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.525+0000 7f3f50a00640 1 -- 172.21.15.27:0/1537731404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44034070 con 0x7f3f54155d60 2024-03-28T19:10:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.525+0000 7f3f50a00640 1 -- 172.21.15.27:0/1537731404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44030840 con 0x7f3f54155d60 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.525+0000 7f3f50a00640 1 -- 172.21.15.27:0/1537731404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44030c10 con 0x7f3f54155d60 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.526+0000 7f3f58c00640 1 -- 172.21.15.27:0/1537731404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 msgr2=0x7f3f541561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.526+0000 7f3f58c00640 1 --2- 172.21.15.27:0/1537731404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541561f0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009870 tx=0x7f3f4402b980 comp rx=0 tx=0).stop 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.526+0000 7f3f58c00640 1 -- 172.21.15.27:0/1537731404 shutdown_connections 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.526+0000 7f3f58c00640 1 --2- 172.21.15.27:0/1537731404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541561f0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.526+0000 7f3f58c00640 1 --2- 172.21.15.27:0/1537731404 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54159650 0x7f3f540c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.526+0000 7f3f58c00640 1 -- 172.21.15.27:0/1537731404 >> 172.21.15.27:0/1537731404 conn(0x7f3f540c4750 msgr2=0x7f3f540c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.526+0000 7f3f58c00640 1 -- 172.21.15.27:0/1537731404 shutdown_connections 2024-03-28T19:10:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.526+0000 7f3f58c00640 1 -- 172.21.15.27:0/1537731404 wait complete. 2024-03-28T19:10:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.528+0000 7f3f58c00640 1 Processor -- start 2024-03-28T19:10:50.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.528+0000 7f3f58c00640 1 -- start start 2024-03-28T19:10:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f58c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541ebd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f58c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54158cb0 0x7f3f54151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f58c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54159650 0x7f3f541defc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f58c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540c1c30 con 0x7f3f54155d60 2024-03-28T19:10:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f58c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540c1ab0 con 0x7f3f54159650 2024-03-28T19:10:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f58c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5406c7e0 con 0x7f3f54158cb0 2024-03-28T19:10:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f51e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541ebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f51e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541ebd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34280/0 (socket says 172.21.15.27:34280) 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f51e00640 1 -- 172.21.15.27:0/3130236952 learned_addr learned my addr 172.21.15.27:0/3130236952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f51400640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54158cb0 0x7f3f54151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f52800640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54159650 0x7f3f541defc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f51e00640 1 -- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54158cb0 msgr2=0x7f3f54151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f51e00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54158cb0 0x7f3f54151b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.529+0000 7f3f51e00640 1 -- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54159650 msgr2=0x7f3f541defc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f51e00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54159650 0x7f3f541defc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f51e00640 1 -- 172.21.15.27:0/3130236952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f541ec270 con 0x7f3f54155d60 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f52800640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54159650 0x7f3f541defc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f51e00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541ebd30 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00d800 tx=0x7f3f4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f4a000640 1 -- 172.21.15.27:0/3130236952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c00b6c0 con 0x7f3f54155d60 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f4a000640 1 -- 172.21.15.27:0/3130236952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4c019070 con 0x7f3f54155d60 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f4a000640 1 -- 172.21.15.27:0/3130236952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c00ba80 con 0x7f3f54155d60 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f541585d0 con 0x7f3f54155d60 2024-03-28T19:10:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.530+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f54156b30 con 0x7f3f54155d60 2024-03-28T19:10:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.532+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f54156930 con 0x7f3f54155d60 2024-03-28T19:10:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.533+0000 7f3f4a000640 1 -- 172.21.15.27:0/3130236952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f4c00bc20 con 0x7f3f54155d60 2024-03-28T19:10:50.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.534+0000 7f3f4a000640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3406bc00 0x7f3f3406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:50.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.534+0000 7f3f4a000640 1 -- 172.21.15.27:0/3130236952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f3f4c08b9b0 con 0x7f3f54155d60 2024-03-28T19:10:50.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.534+0000 7f3f51400640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3406bc00 0x7f3f3406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.535+0000 7f3f51400640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3406bc00 0x7f3f3406e0c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3f40007fd0 tx=0x7f3f40009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:50.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.539+0000 7f3f4a000640 1 -- 172.21.15.27:0/3130236952 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f4c01c360 con 0x7f3f54155d60 2024-03-28T19:10:50.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.825+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f5404f400 con 0x7f3f54155d60 2024-03-28T19:10:50.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.826+0000 7f3f4a000640 1 -- 172.21.15.27:0/3130236952 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f3f4c059a20 con 0x7f3f54155d60 2024-03-28T19:10:50.825 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:50.825 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":6846,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:50.826 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3406bc00 msgr2=0x7f3f3406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3406bc00 0x7f3f3406e0c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3f40007fd0 tx=0x7f3f40009040 comp rx=0 tx=0).stop 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 msgr2=0x7f3f541ebd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541ebd30 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00d800 tx=0x7f3f4c00dcc0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 shutdown_connections 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3f3406bc00 0x7f3f3406e0c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f54159650 0x7f3f541defc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f54158cb0 0x7f3f54151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 --2- 172.21.15.27:0/3130236952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f54155d60 0x7f3f541ebd30 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 >> 172.21.15.27:0/3130236952 conn(0x7f3f540c4750 msgr2=0x7f3f5406e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.829+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 shutdown_connections 2024-03-28T19:10:50.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:50.830+0000 7f3f58c00640 1 -- 172.21.15.27:0/3130236952 wait complete. 2024-03-28T19:10:50.852 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:10:52.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.090+0000 7f0ad9e00640 1 Processor -- start 2024-03-28T19:10:53.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.090+0000 7f0ad9e00640 1 -- start start 2024-03-28T19:10:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.092+0000 7f0ad9e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ad40c8ed0 0x7f0ad40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.092+0000 7f0ad9e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40be5d0 0x7f0ad4155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.092+0000 7f0ad9e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad4156230 con 0x7f0ad40c58c0 2024-03-28T19:10:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.092+0000 7f0ad9e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad4158bd0 con 0x7f0ad40c8ed0 2024-03-28T19:10:53.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.092+0000 7f0ad9e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4158eb0 con 0x7f0ad40be5d0 2024-03-28T19:10:53.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.093+0000 7f0acbe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ad40c8ed0 0x7f0ad40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:53.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.093+0000 7f0acbe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ad40c8ed0 0x7f0ad40c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:46146/0 (socket says 172.21.15.27:46146) 2024-03-28T19:10:53.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.093+0000 7f0acbe00640 1 -- 172.21.15.27:0/4143857026 learned_addr learned my addr 172.21.15.27:0/4143857026 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:53.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.093+0000 7f0ad9400640 1 --2- 172.21.15.27:0/4143857026 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40be5d0 0x7f0ad4155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:53.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.094+0000 7f0acbe00640 1 -- 172.21.15.27:0/4143857026 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40be5d0 msgr2=0x7f0ad4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:53.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.094+0000 7f0acbe00640 1 --2- 172.21.15.27:0/4143857026 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40be5d0 0x7f0ad4155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.094+0000 7f0acbe00640 1 -- 172.21.15.27:0/4143857026 >> v1:172.21.15.155:6789/0 conn(0x7f0ad40c58c0 legacy=0x7f0ad41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:53.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.094+0000 7f0acbe00640 1 -- 172.21.15.27:0/4143857026 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad4159190 con 0x7f0ad40c8ed0 2024-03-28T19:10:53.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.095+0000 7f0acbe00640 1 --2- 172.21.15.27:0/4143857026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40c8ed0 0x7f0ad40c5ca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0abc009870 tx=0x7f0abc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0c3cbfcc9ff373c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:53.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.095+0000 7f0ad3e00640 1 -- 172.21.15.27:0/4143857026 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc035020 con 0x7f0ad40c8ed0 2024-03-28T19:10:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.095+0000 7f0ad3e00640 1 -- 172.21.15.27:0/4143857026 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc031640 con 0x7f0ad40c8ed0 2024-03-28T19:10:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.095+0000 7f0ad3e00640 1 -- 172.21.15.27:0/4143857026 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc02d040 con 0x7f0ad40c8ed0 2024-03-28T19:10:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.096+0000 7f0ad9e00640 1 -- 172.21.15.27:0/4143857026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40c8ed0 msgr2=0x7f0ad40c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.096+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/4143857026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40c8ed0 0x7f0ad40c5ca0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0abc009870 tx=0x7f0abc02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.096+0000 7f0ad9e00640 1 -- 172.21.15.27:0/4143857026 shutdown_connections 2024-03-28T19:10:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.096+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/4143857026 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40be5d0 0x7f0ad4155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.096+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/4143857026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40c8ed0 0x7f0ad40c5ca0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.096+0000 7f0ad9e00640 1 -- 172.21.15.27:0/4143857026 >> 172.21.15.27:0/4143857026 conn(0x7f0ad406b6f0 msgr2=0x7f0ad40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.096+0000 7f0ad9e00640 1 -- 172.21.15.27:0/4143857026 shutdown_connections 2024-03-28T19:10:53.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.096+0000 7f0ad9e00640 1 -- 172.21.15.27:0/4143857026 wait complete. 2024-03-28T19:10:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.098+0000 7f0ad9e00640 1 Processor -- start 2024-03-28T19:10:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.098+0000 7f0ad9e00640 1 -- start start 2024-03-28T19:10:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.099+0000 7f0ad9e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40be5d0 0x7f0ad40c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.099+0000 7f0ad9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad40c58c0 0x7f0ad4152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.099+0000 7f0ad9e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40c8ed0 0x7f0ad40c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.099+0000 7f0ad9e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40b0820 con 0x7f0ad40c58c0 2024-03-28T19:10:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.099+0000 7f0ad9e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4159750 con 0x7f0ad40be5d0 2024-03-28T19:10:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.099+0000 7f0ad9e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40b09a0 con 0x7f0ad40c8ed0 2024-03-28T19:10:53.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0acbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad40c58c0 0x7f0ad4152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:53.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0acbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad40c58c0 0x7f0ad4152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34290/0 (socket says 172.21.15.27:34290) 2024-03-28T19:10:53.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0acbe00640 1 -- 172.21.15.27:0/1990292900 learned_addr learned my addr 172.21.15.27:0/1990292900 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:53.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0ad8a00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40be5d0 0x7f0ad40c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0acbe00640 1 -- 172.21.15.27:0/1990292900 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40c8ed0 msgr2=0x7f0ad40c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0acbe00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40c8ed0 0x7f0ad40c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0acbe00640 1 -- 172.21.15.27:0/1990292900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40be5d0 msgr2=0x7f0ad40c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0acbe00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40be5d0 0x7f0ad40c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.100+0000 7f0acbe00640 1 -- 172.21.15.27:0/1990292900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad4152570 con 0x7f0ad40c58c0 2024-03-28T19:10:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.101+0000 7f0acbe00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad40c58c0 0x7f0ad4152030 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0abc000c00 tx=0x7f0abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:53.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.101+0000 7f0ad1600640 1 -- 172.21.15.27:0/1990292900 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc040070 con 0x7f0ad40c58c0 2024-03-28T19:10:53.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.101+0000 7f0ad1600640 1 -- 172.21.15.27:0/1990292900 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc0399f0 con 0x7f0ad40c58c0 2024-03-28T19:10:53.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.101+0000 7f0ad1600640 1 -- 172.21.15.27:0/1990292900 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc035040 con 0x7f0ad40c58c0 2024-03-28T19:10:53.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.101+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad40ab950 con 0x7f0ad40c58c0 2024-03-28T19:10:53.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.101+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad40ac220 con 0x7f0ad40c58c0 2024-03-28T19:10:53.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.103+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad4069640 con 0x7f0ad40c58c0 2024-03-28T19:10:53.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.104+0000 7f0ad1600640 1 -- 172.21.15.27:0/1990292900 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0abc04b060 con 0x7f0ad40c58c0 2024-03-28T19:10:53.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.105+0000 7f0ad1600640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ab006bbd0 0x7f0ab006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:53.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.105+0000 7f0ad1600640 1 -- 172.21.15.27:0/1990292900 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f0abc0acd30 con 0x7f0ad40c58c0 2024-03-28T19:10:53.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.105+0000 7f0ad8a00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ab006bbd0 0x7f0ab006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:53.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.106+0000 7f0ad8a00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ab006bbd0 0x7f0ab006e090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4001ca0 tx=0x7f0ac4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:53.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.110+0000 7f0ad1600640 1 -- 172.21.15.27:0/1990292900 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0abc07ada0 con 0x7f0ad40c58c0 2024-03-28T19:10:53.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.384+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad4156580 con 0x7f0ad40c58c0 2024-03-28T19:10:53.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.385+0000 7f0ad1600640 1 -- 172.21.15.27:0/1990292900 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f0abc07a6a0 con 0x7f0ad40c58c0 2024-03-28T19:10:53.384 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:53.384 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":6846,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:53.384 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.387+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ab006bbd0 msgr2=0x7f0ab006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.387+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ab006bbd0 0x7f0ab006e090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4001ca0 tx=0x7f0ac4009040 comp rx=0 tx=0).stop 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.387+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad40c58c0 msgr2=0x7f0ad4152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.387+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad40c58c0 0x7f0ad4152030 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0abc000c00 tx=0x7f0abc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.388+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 shutdown_connections 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.388+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ab006bbd0 0x7f0ab006e090 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.388+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ad40c8ed0 0x7f0ad40c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.388+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad40c58c0 0x7f0ad4152030 secure :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0abc000c00 tx=0x7f0abc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.388+0000 7f0ad9e00640 1 --2- 172.21.15.27:0/1990292900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ad40be5d0 0x7f0ad40c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.388+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 >> 172.21.15.27:0/1990292900 conn(0x7f0ad406b6f0 msgr2=0x7f0ad41569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.388+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 shutdown_connections 2024-03-28T19:10:53.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:53.388+0000 7f0ad9e00640 1 -- 172.21.15.27:0/1990292900 wait complete. 2024-03-28T19:10:53.409 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:10:55.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.643+0000 7fa8ade00640 1 Processor -- start 2024-03-28T19:10:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.643+0000 7fa8ade00640 1 -- start start 2024-03-28T19:10:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.644+0000 7fa8ade00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa8a80c59d0 0x7fa8a80c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.644+0000 7fa8ade00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80727e0 0x7fa8a8159520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.644+0000 7fa8ade00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a8155fe0 con 0x7fa8a8155600 2024-03-28T19:10:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.645+0000 7fa8ade00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a81565a0 con 0x7fa8a80c59d0 2024-03-28T19:10:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.645+0000 7fa8ade00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8159090 con 0x7fa8a80727e0 2024-03-28T19:10:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.645+0000 7fa89fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80727e0 0x7fa8a8159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.645+0000 7fa8aca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa8a80c59d0 0x7fa8a80c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.645+0000 7fa8aca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa8a80c59d0 0x7fa8a80c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:46174/0 (socket says 172.21.15.27:46174) 2024-03-28T19:10:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.645+0000 7fa8aca00640 1 -- 172.21.15.27:0/1205813780 learned_addr learned my addr 172.21.15.27:0/1205813780 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.646+0000 7fa8aca00640 1 -- 172.21.15.27:0/1205813780 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80727e0 msgr2=0x7fa8a8159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.646+0000 7fa8aca00640 1 --2- 172.21.15.27:0/1205813780 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80727e0 0x7fa8a8159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.646+0000 7fa8aca00640 1 -- 172.21.15.27:0/1205813780 >> v1:172.21.15.155:6789/0 conn(0x7fa8a8155600 legacy=0x7fa8a814f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.646+0000 7fa8aca00640 1 -- 172.21.15.27:0/1205813780 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a81562c0 con 0x7fa8a80c59d0 2024-03-28T19:10:55.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.648+0000 7fa8aca00640 1 --2- 172.21.15.27:0/1205813780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a80c59d0 0x7fa8a80c8f00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa898009870 tx=0x7fa89802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfa71c4629c9b5c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.648+0000 7fa8a7e00640 1 -- 172.21.15.27:0/1205813780 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8980317c0 con 0x7fa8a80c59d0 2024-03-28T19:10:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.648+0000 7fa8a7e00640 1 -- 172.21.15.27:0/1205813780 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa898037070 con 0x7fa8a80c59d0 2024-03-28T19:10:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.648+0000 7fa8a7e00640 1 -- 172.21.15.27:0/1205813780 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898031b80 con 0x7fa8a80c59d0 2024-03-28T19:10:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.649+0000 7fa8ade00640 1 -- 172.21.15.27:0/1205813780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a80c59d0 msgr2=0x7fa8a80c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.649+0000 7fa8ade00640 1 --2- 172.21.15.27:0/1205813780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a80c59d0 0x7fa8a80c8f00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa898009870 tx=0x7fa89802b860 comp rx=0 tx=0).stop 2024-03-28T19:10:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.649+0000 7fa8ade00640 1 -- 172.21.15.27:0/1205813780 shutdown_connections 2024-03-28T19:10:55.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.649+0000 7fa8ade00640 1 --2- 172.21.15.27:0/1205813780 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80727e0 0x7fa8a8159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.649+0000 7fa8ade00640 1 --2- 172.21.15.27:0/1205813780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a80c59d0 0x7fa8a80c8f00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.649+0000 7fa8ade00640 1 -- 172.21.15.27:0/1205813780 >> 172.21.15.27:0/1205813780 conn(0x7fa8a80ac910 msgr2=0x7fa8a80accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:55.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.649+0000 7fa8ade00640 1 -- 172.21.15.27:0/1205813780 shutdown_connections 2024-03-28T19:10:55.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.649+0000 7fa8ade00640 1 -- 172.21.15.27:0/1205813780 wait complete. 2024-03-28T19:10:55.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.651+0000 7fa8ade00640 1 Processor -- start 2024-03-28T19:10:55.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.651+0000 7fa8ade00640 1 -- start start 2024-03-28T19:10:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.652+0000 7fa8ade00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8a80727e0 0x7fa8a80ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.652+0000 7fa8ade00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80c59d0 0x7fa8a80aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.652+0000 7fa8ade00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a8155600 0x7fa8a80ab230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.652+0000 7fa8ade00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a814f970 con 0x7fa8a80727e0 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.652+0000 7fa8ade00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8159a60 con 0x7fa8a8155600 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.652+0000 7fa8ade00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a814faf0 con 0x7fa8a80c59d0 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8aca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8a80727e0 0x7fa8a80ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8aca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8a80727e0 0x7fa8a80ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34298/0 (socket says 172.21.15.27:34298) 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8aca00640 1 -- 172.21.15.27:0/403850822 learned_addr learned my addr 172.21.15.27:0/403850822 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa89fe00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80c59d0 0x7fa8a80aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8ad400640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a8155600 0x7fa8a80ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8aca00640 1 -- 172.21.15.27:0/403850822 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80c59d0 msgr2=0x7fa8a80aacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8aca00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80c59d0 0x7fa8a80aacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8aca00640 1 -- 172.21.15.27:0/403850822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a8155600 msgr2=0x7fa8a80ab230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8aca00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a8155600 0x7fa8a80ab230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.653+0000 7fa8aca00640 1 -- 172.21.15.27:0/403850822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a80c1820 con 0x7fa8a80727e0 2024-03-28T19:10:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.654+0000 7fa8aca00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8a80727e0 0x7fa8a80ca290 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa898000c00 tx=0x7fa8980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.654+0000 7fa8a5600640 1 -- 172.21.15.27:0/403850822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898032650 con 0x7fa8a80727e0 2024-03-28T19:10:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.654+0000 7fa8a5600640 1 -- 172.21.15.27:0/403850822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa898032810 con 0x7fa8a80727e0 2024-03-28T19:10:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.654+0000 7fa8a5600640 1 -- 172.21.15.27:0/403850822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898032c00 con 0x7fa8a80727e0 2024-03-28T19:10:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.654+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a80c1260 con 0x7fa8a80727e0 2024-03-28T19:10:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.654+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a80aa7d0 con 0x7fa8a80727e0 2024-03-28T19:10:55.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.656+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a8067dd0 con 0x7fa8a80727e0 2024-03-28T19:10:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.657+0000 7fa8a5600640 1 -- 172.21.15.27:0/403850822 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa89804c060 con 0x7fa8a80727e0 2024-03-28T19:10:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.658+0000 7fa8a5600640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa88806bbd0 0x7fa88806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.658+0000 7fa8a5600640 1 -- 172.21.15.27:0/403850822 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7fa8980acab0 con 0x7fa8a80727e0 2024-03-28T19:10:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.658+0000 7fa89fe00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa88806bbd0 0x7fa88806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:55.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.659+0000 7fa89fe00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa88806bbd0 0x7fa88806e090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80ad000 tx=0x7fa894009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:55.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.664+0000 7fa8a5600640 1 -- 172.21.15.27:0/403850822 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa898040070 con 0x7fa8a80727e0 2024-03-28T19:10:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.918+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8a8069640 con 0x7fa8a80727e0 2024-03-28T19:10:55.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.919+0000 7fa8a5600640 1 -- 172.21.15.27:0/403850822 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fa89807a650 con 0x7fa8a80727e0 2024-03-28T19:10:55.918 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:55.919 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":6846,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-28T19:10:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa88806bbd0 msgr2=0x7fa88806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa88806bbd0 0x7fa88806e090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80ad000 tx=0x7fa894009190 comp rx=0 tx=0).stop 2024-03-28T19:10:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8a80727e0 msgr2=0x7fa8a80ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8a80727e0 0x7fa8a80ca290 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa898000c00 tx=0x7fa8980046c0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 shutdown_connections 2024-03-28T19:10:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa88806bbd0 0x7fa88806e090 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8a8155600 0x7fa8a80ab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8a80c59d0 0x7fa8a80aacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 --2- 172.21.15.27:0/403850822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8a80727e0 0x7fa8a80ca290 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 >> 172.21.15.27:0/403850822 conn(0x7fa8a80ac910 msgr2=0x7fa8a8154b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.922+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 shutdown_connections 2024-03-28T19:10:55.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:55.923+0000 7fa8ade00640 1 -- 172.21.15.27:0/403850822 wait complete. 2024-03-28T19:10:55.940 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:10:57.941 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:10:58.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.195+0000 7f9d47200640 1 Processor -- start 2024-03-28T19:10:58.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.196+0000 7f9d47200640 1 -- start start 2024-03-28T19:10:58.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.197+0000 7f9d47200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9d48152a70 0x7f9d48159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.197+0000 7f9d47200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d480c5710 con 0x7f9d480c6cc0 2024-03-28T19:10:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.198+0000 7f9d47200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d480c59f0 con 0x7f9d48155b40 2024-03-28T19:10:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.198+0000 7f9d47200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d481556b0 con 0x7f9d48152a70 2024-03-28T19:10:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.198+0000 7f9d45400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9d480c6cc0 0x7f9d480cb0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41370/0 (socket says 172.21.15.27:41370) 2024-03-28T19:10:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.198+0000 7f9d45400640 1 -- 172.21.15.27:0/2396499085 learned_addr learned my addr 172.21.15.27:0/2396499085 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:10:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.198+0000 7f9d45e00640 1 --2- 172.21.15.27:0/2396499085 >> v2:172.21.15.155:3300/0 conn(0x7f9d48152a70 0x7f9d48159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:58.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.198+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118449021 0 0) 0x7f9d480c5710 con 0x7f9d480c6cc0 2024-03-28T19:10:58.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2c003520 con 0x7f9d480c6cc0 2024-03-28T19:10:58.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806410302 0 0) 0x7f9d2c003520 con 0x7f9d480c6cc0 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 >> v2:172.21.15.155:3300/0 conn(0x7f9d48152a70 msgr2=0x7f9d48159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 --2- 172.21.15.27:0/2396499085 >> v2:172.21.15.155:3300/0 conn(0x7f9d48152a70 0x7f9d48159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 >> v1:172.21.15.155:6789/0 conn(0x7f9d48155b40 legacy=0x7f9d48155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d480c5cd0 con 0x7f9d480c6cc0 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9d30001670 con 0x7f9d480c6cc0 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 >> v1:172.21.15.27:6789/0 conn(0x7f9d480c6cc0 legacy=0x7f9d480cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d2c0039b0 0x7f9d2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d2c004800 0x7f9d2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d481556b0 con 0x7f9d2c0051c0 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c00c160 con 0x7f9d2c0039b0 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.199+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c00c360 con 0x7f9d2c004800 2024-03-28T19:10:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d45400640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d2c004800 0x7f9d2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d46800640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d45e00640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d2c0039b0 0x7f9d2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d46800640 1 -- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d2c004800 msgr2=0x7f9d2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d46800640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d2c004800 0x7f9d2c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d46800640 1 -- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d2c0039b0 msgr2=0x7f9d2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:10:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d46800640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d2c0039b0 0x7f9d2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d46800640 1 -- 172.21.15.27:0/2396499085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c003520 con 0x7f9d2c0051c0 2024-03-28T19:10:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.200+0000 7f9d45e00640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d2c0039b0 0x7f9d2c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:10:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.202+0000 7f9d46800640 1 -- 172.21.15.27:0/2396499085 reap_dead start 2024-03-28T19:10:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.202+0000 7f9d46800640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d2c00b9c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f9d40001e50 tx=0x7f9d2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b372c083995ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.202+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d40001a70 con 0x7f9d2c0051c0 2024-03-28T19:10:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.202+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d40003b90 con 0x7f9d2c0051c0 2024-03-28T19:10:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.202+0000 7f9d44a00640 1 -- 172.21.15.27:0/2396499085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d400050d0 con 0x7f9d2c0051c0 2024-03-28T19:10:58.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.203+0000 7f9d47200640 1 -- 172.21.15.27:0/2396499085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 msgr2=0x7f9d2c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:58.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.203+0000 7f9d47200640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d2c00b9c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f9d40001e50 tx=0x7f9d2c005830 comp rx=0 tx=0).stop 2024-03-28T19:10:58.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.203+0000 7f9d47200640 1 -- 172.21.15.27:0/2396499085 shutdown_connections 2024-03-28T19:10:58.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.203+0000 7f9d47200640 1 --2- 172.21.15.27:0/2396499085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d2c00b9c0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.203+0000 7f9d47200640 1 -- 172.21.15.27:0/2396499085 >> 172.21.15.27:0/2396499085 conn(0x7f9d4806b2a0 msgr2=0x7f9d4806b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:58.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.203+0000 7f9d47200640 1 -- 172.21.15.27:0/2396499085 shutdown_connections 2024-03-28T19:10:58.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.203+0000 7f9d47200640 1 -- 172.21.15.27:0/2396499085 wait complete. 2024-03-28T19:10:58.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.205+0000 7f9d47200640 1 Processor -- start 2024-03-28T19:10:58.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.205+0000 7f9d47200640 1 -- start start 2024-03-28T19:10:58.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d47200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d48151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d47200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d481ec500 0x7f9d48149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d47200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d481ec960 0x7f9d48149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d47200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d480c8f50 con 0x7f9d2c0051c0 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d47200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d480c8dd0 con 0x7f9d481ec500 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d47200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d480c5fb0 con 0x7f9d481ec960 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d48151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d48151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59490/0 (socket says 172.21.15.27:59490) 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45e00640 1 -- 172.21.15.27:0/2396165901 learned_addr learned my addr 172.21.15.27:0/2396165901 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:10:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45400640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d481ec500 0x7f9d48149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d46800640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d481ec960 0x7f9d48149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45e00640 1 -- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d481ec960 msgr2=0x7f9d48149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45e00640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d481ec960 0x7f9d48149880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45e00640 1 -- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d481ec500 msgr2=0x7f9d48149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45e00640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d481ec500 0x7f9d48149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.207+0000 7f9d45e00640 1 -- 172.21.15.27:0/2396165901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d480c20b0 con 0x7f9d2c0051c0 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.208+0000 7f9d45400640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d481ec500 0x7f9d48149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.208+0000 7f9d45e00640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d48151780 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f9d38011540 tx=0x7f9d38011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.208+0000 7f9d3e000640 1 -- 172.21.15.27:0/2396165901 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38015870 con 0x7f9d2c0051c0 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.208+0000 7f9d3e000640 1 -- 172.21.15.27:0/2396165901 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d38015a10 con 0x7f9d2c0051c0 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.208+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d480c2390 con 0x7f9d2c0051c0 2024-03-28T19:10:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.208+0000 7f9d3e000640 1 -- 172.21.15.27:0/2396165901 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38015dc0 con 0x7f9d2c0051c0 2024-03-28T19:10:58.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.208+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d48150f80 con 0x7f9d2c0051c0 2024-03-28T19:10:58.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.210+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d48067dd0 con 0x7f9d2c0051c0 2024-03-28T19:10:58.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.211+0000 7f9d3e000640 1 -- 172.21.15.27:0/2396165901 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9d3801d400 con 0x7f9d2c0051c0 2024-03-28T19:10:58.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.212+0000 7f9d3e000640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d2406bc00 0x7f9d2406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:10:58.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.212+0000 7f9d3e000640 1 -- 172.21.15.27:0/2396165901 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8357+0+0 (secure 0 0 0) 0x7f9d3808bd80 con 0x7f9d2c0051c0 2024-03-28T19:10:58.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.212+0000 7f9d45400640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d2406bc00 0x7f9d2406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:10:58.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.213+0000 7f9d45400640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d2406bc00 0x7f9d2406e0c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9d48159b80 tx=0x7f9d30009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:10:58.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.218+0000 7f9d3e000640 1 -- 172.21.15.27:0/2396165901 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9d38019030 con 0x7f9d2c0051c0 2024-03-28T19:10:58.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.515+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d48069640 con 0x7f9d2c0051c0 2024-03-28T19:10:58.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.516+0000 7f9d3e000640 1 -- 172.21.15.27:0/2396165901 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f9d38059dc0 con 0x7f9d2c0051c0 2024-03-28T19:10:58.515 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:10:58.516 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":6846,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:09:49.709830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6118":{"gid":6118,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6835/3599950884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":3599950884},{"type":"v1","addr":"172.21.15.155:6835","nonce":3599950884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6118,"qdb_cluster":[6118]},"id":2}]} 2024-03-28T19:10:58.516 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 122 2024-03-28T19:10:58.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d2406bc00 msgr2=0x7f9d2406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:58.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d2406bc00 0x7f9d2406e0c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9d48159b80 tx=0x7f9d30009040 comp rx=0 tx=0).stop 2024-03-28T19:10:58.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 msgr2=0x7f9d48151780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:10:58.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d48151780 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f9d38011540 tx=0x7f9d38011a00 comp rx=0 tx=0).stop 2024-03-28T19:10:58.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 shutdown_connections 2024-03-28T19:10:58.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d2406bc00 0x7f9d2406e0c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d481ec960 0x7f9d48149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d481ec500 0x7f9d48149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 --2- 172.21.15.27:0/2396165901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c0051c0 0x7f9d48151780 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:10:58.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 >> 172.21.15.27:0/2396165901 conn(0x7f9d4806b2a0 msgr2=0x7f9d48153fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:10:58.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 shutdown_connections 2024-03-28T19:10:58.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:10:58.519+0000 7f9d47200640 1 -- 172.21.15.27:0/2396165901 wait complete. 2024-03-28T19:10:58.539 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:11:00.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:11:00.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.771+0000 7fb46b600640 1 Processor -- start 2024-03-28T19:11:00.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.771+0000 7fb46b600640 1 -- start start 2024-03-28T19:11:00.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.772+0000 7fb46b600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb464152da0 0x7fb4640c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:00.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.772+0000 7fb46b600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4640732e0 0x7fb46406bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:00.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.772+0000 7fb46b600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46406b160 con 0x7fb4640c2e40 2024-03-28T19:11:00.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.772+0000 7fb46b600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb46406c260 con 0x7fb4640732e0 2024-03-28T19:11:00.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.772+0000 7fb46b600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4640c2100 con 0x7fb464152da0 2024-03-28T19:11:00.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.773+0000 7fb469600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4640732e0 0x7fb46406bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:00.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.773+0000 7fb463e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb4640c2e40 0x7fb46406b610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41384/0 (socket says 172.21.15.27:41384) 2024-03-28T19:11:00.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.773+0000 7fb463e00640 1 -- 172.21.15.27:0/3900299236 learned_addr learned my addr 172.21.15.27:0/3900299236 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:11:00.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.774+0000 7fb469600640 1 -- 172.21.15.27:0/3900299236 >> v2:172.21.15.155:3300/0 conn(0x7fb464152da0 msgr2=0x7fb4640c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:11:00.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.774+0000 7fb469600640 1 --2- 172.21.15.27:0/3900299236 >> v2:172.21.15.155:3300/0 conn(0x7fb464152da0 0x7fb4640c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:00.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.774+0000 7fb469600640 1 -- 172.21.15.27:0/3900299236 >> v1:172.21.15.27:6789/0 conn(0x7fb4640c2e40 legacy=0x7fb46406b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:00.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.774+0000 7fb463400640 1 -- 172.21.15.27:0/3900299236 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214086363 0 0) 0x7fb46406b160 con 0x7fb4640c2e40 2024-03-28T19:11:00.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.774+0000 7fb469600640 1 -- 172.21.15.27:0/3900299236 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464156450 con 0x7fb4640732e0 2024-03-28T19:11:00.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.775+0000 7fb469600640 1 --2- 172.21.15.27:0/3900299236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640732e0 0x7fb46406bd20 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb464068d60 tx=0x7fb45c02bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a27d0ba4f55255 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:00.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.776+0000 7fb463400640 1 -- 172.21.15.27:0/3900299236 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c039070 con 0x7fb4640732e0 2024-03-28T19:11:00.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.776+0000 7fb463400640 1 -- 172.21.15.27:0/3900299236 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c02cb50 con 0x7fb4640732e0 2024-03-28T19:11:00.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.776+0000 7fb463400640 1 -- 172.21.15.27:0/3900299236 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c034440 con 0x7fb4640732e0 2024-03-28T19:11:00.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.777+0000 7fb46b600640 1 -- 172.21.15.27:0/3900299236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640732e0 msgr2=0x7fb46406bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:00.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.777+0000 7fb46b600640 1 --2- 172.21.15.27:0/3900299236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640732e0 0x7fb46406bd20 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb464068d60 tx=0x7fb45c02bda0 comp rx=0 tx=0).stop 2024-03-28T19:11:00.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.777+0000 7fb46b600640 1 -- 172.21.15.27:0/3900299236 shutdown_connections 2024-03-28T19:11:00.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.777+0000 7fb46b600640 1 --2- 172.21.15.27:0/3900299236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640732e0 0x7fb46406bd20 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:00.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.777+0000 7fb46b600640 1 --2- 172.21.15.27:0/3900299236 >> v2:172.21.15.155:3300/0 conn(0x7fb464152da0 0x7fb4640c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:00.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.777+0000 7fb46b600640 1 -- 172.21.15.27:0/3900299236 >> 172.21.15.27:0/3900299236 conn(0x7fb4640c3ff0 msgr2=0x7fb4640c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:00.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.777+0000 7fb46b600640 1 -- 172.21.15.27:0/3900299236 shutdown_connections 2024-03-28T19:11:00.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.777+0000 7fb46b600640 1 -- 172.21.15.27:0/3900299236 wait complete. 2024-03-28T19:11:00.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.779+0000 7fb46b600640 1 Processor -- start 2024-03-28T19:11:00.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.779+0000 7fb46b600640 1 -- start start 2024-03-28T19:11:00.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.780+0000 7fb46b600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4640732e0 0x7fb4641477e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:00.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.780+0000 7fb46b600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640c2e40 0x7fb464144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:00.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.780+0000 7fb46b600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb464152da0 0x7fb464145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:00.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.780+0000 7fb46b600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4641560e0 con 0x7fb4640c2e40 2024-03-28T19:11:00.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.780+0000 7fb46b600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4640c2630 con 0x7fb4640732e0 2024-03-28T19:11:00.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.780+0000 7fb46b600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb464067130 con 0x7fb464152da0 2024-03-28T19:11:00.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb463e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640c2e40 0x7fb464144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:00.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb463e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640c2e40 0x7fb464144e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59518/0 (socket says 172.21.15.27:59518) 2024-03-28T19:11:00.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb463e00640 1 -- 172.21.15.27:0/1161234833 learned_addr learned my addr 172.21.15.27:0/1161234833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:11:00.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb469600640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb464152da0 0x7fb464145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:00.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb468c00640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4640732e0 0x7fb4641477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb463e00640 1 -- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb464152da0 msgr2=0x7fb464145340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb463e00640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb464152da0 0x7fb464145340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb463e00640 1 -- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4640732e0 msgr2=0x7fb4641477e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb463e00640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4640732e0 0x7fb4641477e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb463e00640 1 -- 172.21.15.27:0/1161234833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464150ea0 con 0x7fb4640c2e40 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.781+0000 7fb468c00640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4640732e0 0x7fb4641477e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.782+0000 7fb463e00640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640c2e40 0x7fb464144e00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb45000c4a0 tx=0x7fb45000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.782+0000 7fb460c00640 1 -- 172.21.15.27:0/1161234833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450018070 con 0x7fb4640c2e40 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.782+0000 7fb460c00640 1 -- 172.21.15.27:0/1161234833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4500074e0 con 0x7fb4640c2e40 2024-03-28T19:11:00.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.782+0000 7fb460c00640 1 -- 172.21.15.27:0/1161234833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450004a20 con 0x7fb4640c2e40 2024-03-28T19:11:00.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.782+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb464145bf0 con 0x7fb4640c2e40 2024-03-28T19:11:00.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.783+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb464151160 con 0x7fb4640c2e40 2024-03-28T19:11:00.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.784+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb464150390 con 0x7fb4640c2e40 2024-03-28T19:11:00.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.785+0000 7fb460c00640 1 -- 172.21.15.27:0/1161234833 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb45001b070 con 0x7fb4640c2e40 2024-03-28T19:11:00.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.786+0000 7fb460c00640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb44406bbd0 0x7fb44406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:00.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.787+0000 7fb460c00640 1 -- 172.21.15.27:0/1161234833 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7fb45008a730 con 0x7fb4640c2e40 2024-03-28T19:11:00.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.789+0000 7fb468c00640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb44406bbd0 0x7fb44406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:00.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.790+0000 7fb468c00640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb44406bbd0 0x7fb44406e090 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb458009890 tx=0x7fb458008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:00.793+0000 7fb460c00640 1 -- 172.21.15.27:0/1161234833 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb464150390 con 0x7fb4640c2e40 2024-03-28T19:11:01.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.111+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb46404f400 con 0x7fb4640c2e40 2024-03-28T19:11:01.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.112+0000 7fb460c00640 1 -- 172.21.15.27:0/1161234833 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+5461 (secure 0 0 0) 0x7fb450058160 con 0x7fb4640c2e40 2024-03-28T19:11:01.111 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:11:01.112 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:10:59.975310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:11:01.112 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-28T19:11:01.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.115+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb44406bbd0 msgr2=0x7fb44406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:01.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.115+0000 7fb46b600640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb44406bbd0 0x7fb44406e090 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb458009890 tx=0x7fb458008040 comp rx=0 tx=0).stop 2024-03-28T19:11:01.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.115+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640c2e40 msgr2=0x7fb464144e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:01.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.115+0000 7fb46b600640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640c2e40 0x7fb464144e00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fb45000c4a0 tx=0x7fb45000c960 comp rx=0 tx=0).stop 2024-03-28T19:11:01.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.116+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 shutdown_connections 2024-03-28T19:11:01.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.116+0000 7fb46b600640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb44406bbd0 0x7fb44406e090 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.116+0000 7fb46b600640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb464152da0 0x7fb464145340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.116+0000 7fb46b600640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4640c2e40 0x7fb464144e00 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.116+0000 7fb46b600640 1 --2- 172.21.15.27:0/1161234833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4640732e0 0x7fb4641477e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.116+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 >> 172.21.15.27:0/1161234833 conn(0x7fb4640c3ff0 msgr2=0x7fb46406ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:01.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.116+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 shutdown_connections 2024-03-28T19:11:01.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.116+0000 7fb46b600640 1 -- 172.21.15.27:0/1161234833 wait complete. 2024-03-28T19:11:01.138 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-03-28T19:11:01.138 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:11:01.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:11:01.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.365+0000 7fc0dc800640 1 Processor -- start 2024-03-28T19:11:01.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.365+0000 7fc0dc800640 1 -- start start 2024-03-28T19:11:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.366+0000 7fc0dc800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d4152bd0 0x7fc0d406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.366+0000 7fc0dc800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc0d406f1e0 0x7fc0d4070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.367+0000 7fc0dc800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d40c52e0 con 0x7fc0d40c4dc0 2024-03-28T19:11:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.367+0000 7fc0dc800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d41557e0 con 0x7fc0d406f1e0 2024-03-28T19:11:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.367+0000 7fc0dc800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d4155ac0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.367+0000 7fc0da800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc0d40c4dc0 0x7fc0d4158c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41388/0 (socket says 172.21.15.27:41388) 2024-03-28T19:11:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.367+0000 7fc0da800640 1 -- 172.21.15.27:0/211057430 learned_addr learned my addr 172.21.15.27:0/211057430 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:11:01.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.367+0000 7fc0d9e00640 1 --2- 172.21.15.27:0/211057430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d4152bd0 0x7fc0d406eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:01.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.367+0000 7fc0d9400640 1 --2- 172.21.15.27:0/211057430 >> v2:172.21.15.155:3300/0 conn(0x7fc0d406f1e0 0x7fc0d4070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:01.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.368+0000 7fc0d8a00640 1 -- 172.21.15.27:0/211057430 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104973077 0 0) 0x7fc0d40c52e0 con 0x7fc0d40c4dc0 2024-03-28T19:11:01.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.368+0000 7fc0d8a00640 1 -- 172.21.15.27:0/211057430 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc003520 con 0x7fc0d40c4dc0 2024-03-28T19:11:01.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.368+0000 7fc0d9e00640 1 -- 172.21.15.27:0/211057430 >> v2:172.21.15.155:3300/0 conn(0x7fc0d406f1e0 msgr2=0x7fc0d4070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:11:01.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.368+0000 7fc0d9e00640 1 --2- 172.21.15.27:0/211057430 >> v2:172.21.15.155:3300/0 conn(0x7fc0d406f1e0 0x7fc0d4070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.368+0000 7fc0d9e00640 1 -- 172.21.15.27:0/211057430 >> v1:172.21.15.27:6789/0 conn(0x7fc0d40c4dc0 legacy=0x7fc0d4158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:01.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.368+0000 7fc0d9e00640 1 -- 172.21.15.27:0/211057430 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d4155da0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.368+0000 7fc0d9400640 1 --2- 172.21.15.27:0/211057430 >> v2:172.21.15.155:3300/0 conn(0x7fc0d406f1e0 0x7fc0d4070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:11:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.369+0000 7fc0d9e00640 1 --2- 172.21.15.27:0/211057430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d4152bd0 0x7fc0d406eca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80097a0 tx=0x7fc0c802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f484fd8e3c3503a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.369+0000 7fc0d8a00640 1 -- 172.21.15.27:0/211057430 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8035020 con 0x7fc0d4152bd0 2024-03-28T19:11:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.369+0000 7fc0d8a00640 1 -- 172.21.15.27:0/211057430 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c8031810 con 0x7fc0d4152bd0 2024-03-28T19:11:01.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.370+0000 7fc0d8a00640 1 -- 172.21.15.27:0/211057430 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8031be0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.370+0000 7fc0dc800640 1 -- 172.21.15.27:0/211057430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d4152bd0 msgr2=0x7fc0d406eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.370+0000 7fc0dc800640 1 --2- 172.21.15.27:0/211057430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d4152bd0 0x7fc0d406eca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80097a0 tx=0x7fc0c802b8b0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.370+0000 7fc0dc800640 1 -- 172.21.15.27:0/211057430 shutdown_connections 2024-03-28T19:11:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.370+0000 7fc0dc800640 1 --2- 172.21.15.27:0/211057430 >> v2:172.21.15.155:3300/0 conn(0x7fc0d406f1e0 0x7fc0d4070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.370+0000 7fc0dc800640 1 --2- 172.21.15.27:0/211057430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d4152bd0 0x7fc0d406eca0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.370+0000 7fc0dc800640 1 -- 172.21.15.27:0/211057430 >> 172.21.15.27:0/211057430 conn(0x7fc0d4066dd0 msgr2=0x7fc0d40671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.370+0000 7fc0dc800640 1 -- 172.21.15.27:0/211057430 shutdown_connections 2024-03-28T19:11:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.371+0000 7fc0dc800640 1 -- 172.21.15.27:0/211057430 wait complete. 2024-03-28T19:11:01.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.372+0000 7fc0dc800640 1 Processor -- start 2024-03-28T19:11:01.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.372+0000 7fc0dc800640 1 -- start start 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.373+0000 7fc0dc800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d406f1e0 0x7fc0d41eaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.373+0000 7fc0dc800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d40c4dc0 0x7fc0d41eb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.373+0000 7fc0dc800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d4152bd0 0x7fc0d41ec070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.373+0000 7fc0dc800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d40c6df0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.373+0000 7fc0dc800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d40c6c70 con 0x7fc0d406f1e0 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.373+0000 7fc0dc800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d4070a80 con 0x7fc0d40c4dc0 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0da800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d4152bd0 0x7fc0d41ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0d9400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d40c4dc0 0x7fc0d41eb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0d9e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d406f1e0 0x7fc0d41eaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0d9e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d406f1e0 0x7fc0d41eaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:39026/0 (socket says 172.21.15.27:39026) 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0da800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d4152bd0 0x7fc0d41ec070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59532/0 (socket says 172.21.15.27:59532) 2024-03-28T19:11:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0d9e00640 1 -- 172.21.15.27:0/4001135138 learned_addr learned my addr 172.21.15.27:0/4001135138 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0da800640 1 -- 172.21.15.27:0/4001135138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d40c4dc0 msgr2=0x7fc0d41eb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0da800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d40c4dc0 0x7fc0d41eb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0da800640 1 -- 172.21.15.27:0/4001135138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d406f1e0 msgr2=0x7fc0d41eaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0da800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d406f1e0 0x7fc0d41eaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0da800640 1 -- 172.21.15.27:0/4001135138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d409ac60 con 0x7fc0d4152bd0 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0d9400640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d40c4dc0 0x7fc0d41eb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.374+0000 7fc0da800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d4152bd0 0x7fc0d41ec070 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000ca50 tx=0x7fc0d000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.375+0000 7fc0ce000640 1 -- 172.21.15.27:0/4001135138 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0017070 con 0x7fc0d4152bd0 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.375+0000 7fc0ce000640 1 -- 172.21.15.27:0/4001135138 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d000abb0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.375+0000 7fc0ce000640 1 -- 172.21.15.27:0/4001135138 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0012400 con 0x7fc0d4152bd0 2024-03-28T19:11:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.375+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d40abcf0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.375+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d40abfb0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.376+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d404f400 con 0x7fc0d4152bd0 2024-03-28T19:11:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.377+0000 7fc0ce000640 1 -- 172.21.15.27:0/4001135138 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc0d001a070 con 0x7fc0d4152bd0 2024-03-28T19:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.379+0000 7fc0ce000640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0b806bc00 0x7fc0b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.379+0000 7fc0ce000640 1 -- 172.21.15.27:0/4001135138 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7fc0d008b0f0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.380+0000 7fc0d9e00640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0b806bc00 0x7fc0b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.381+0000 7fc0d9e00640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0b806bc00 0x7fc0b806e0c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009560 tx=0x7fc0c8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.385+0000 7fc0ce000640 1 -- 172.21.15.27:0/4001135138 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc0d0058ac0 con 0x7fc0d4152bd0 2024-03-28T19:11:01.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.777+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0d4152750 con 0x7fc0d4152bd0 2024-03-28T19:11:01.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.778+0000 7fc0ce000640 1 -- 172.21.15.27:0/4001135138 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+5461 (secure 0 0 0) 0x7fc0d005cb60 con 0x7fc0d4152bd0 2024-03-28T19:11:01.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:11:01.778 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:10:59.975310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:11:01.778 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 123 2024-03-28T19:11:01.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0b806bc00 msgr2=0x7fc0b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:01.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0b806bc00 0x7fc0b806e0c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009560 tx=0x7fc0c8008030 comp rx=0 tx=0).stop 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d4152bd0 msgr2=0x7fc0d41ec070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d4152bd0 0x7fc0d41ec070 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000ca50 tx=0x7fc0d000cf10 comp rx=0 tx=0).stop 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 shutdown_connections 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc0b806bc00 0x7fc0b806e0c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0d4152bd0 0x7fc0d41ec070 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc0d40c4dc0 0x7fc0d41eb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 --2- 172.21.15.27:0/4001135138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0d406f1e0 0x7fc0d41eaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 >> 172.21.15.27:0/4001135138 conn(0x7fc0d4066dd0 msgr2=0x7fc0d40c2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 shutdown_connections 2024-03-28T19:11:01.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:01.781+0000 7fc0dc800640 1 -- 172.21.15.27:0/4001135138 wait complete. 2024-03-28T19:11:01.800 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 123, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 120, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5078": { "addr": "172.21.15.155:6834/3692798710", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3692798710, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3692798710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5078, "incarnation": 115, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 124 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:09:52.749310+0000", "qdb_cluster": [ 5078 ], "qdb_leader": 5078, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5078 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 123, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6846": { "addr": "172.21.15.27:6837/242334139", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 242334139, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 242334139, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6846, "incarnation": 123, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:10:59.975310+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6846 } } } ], "standbys": [ { "addr": "172.21.15.27:6835/2858357568", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 2858357568, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 2858357568, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 121, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6834, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:11:01.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:11:01.817 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:11:03.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:11:04.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.043+0000 7fa84dc00640 1 Processor -- start 2024-03-28T19:11:04.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.043+0000 7fa84dc00640 1 -- start start 2024-03-28T19:11:04.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa84dc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa8480c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:04.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa84dc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa8480be5d0 0x7fa848155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:04.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa84dc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa848156230 con 0x7fa8480c8ed0 2024-03-28T19:11:04.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa84dc00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa848158bd0 con 0x7fa8480be5d0 2024-03-28T19:11:04.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa84dc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa848158eb0 con 0x7fa8480c58c0 2024-03-28T19:11:04.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa84d200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa8480be5d0 0x7fa848155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:04.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa847e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa8480c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:04.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa84d200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa8480be5d0 0x7fa848155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:39034/0 (socket says 172.21.15.27:39034) 2024-03-28T19:11:04.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.045+0000 7fa84d200640 1 -- 172.21.15.27:0/1119467023 learned_addr learned my addr 172.21.15.27:0/1119467023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:11:04.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.046+0000 7fa847e00640 1 -- 172.21.15.27:0/1119467023 >> v2:172.21.15.155:3300/0 conn(0x7fa8480be5d0 msgr2=0x7fa848155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:11:04.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.046+0000 7fa847e00640 1 --2- 172.21.15.27:0/1119467023 >> v2:172.21.15.155:3300/0 conn(0x7fa8480be5d0 0x7fa848155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.046+0000 7fa847e00640 1 -- 172.21.15.27:0/1119467023 >> v1:172.21.15.155:6789/0 conn(0x7fa8480c8ed0 legacy=0x7fa8481555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:04.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.046+0000 7fa847e00640 1 -- 172.21.15.27:0/1119467023 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa848159190 con 0x7fa8480c58c0 2024-03-28T19:11:04.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.046+0000 7fa84d200640 1 --2- 172.21.15.27:0/1119467023 >> v2:172.21.15.155:3300/0 conn(0x7fa8480be5d0 0x7fa848155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:11:04.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.047+0000 7fa847e00640 1 --2- 172.21.15.27:0/1119467023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa8480c5ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa8380097a0 tx=0x7fa83802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa3acf536d1b7b72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:04.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.048+0000 7fa846a00640 1 -- 172.21.15.27:0/1119467023 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838031780 con 0x7fa8480c58c0 2024-03-28T19:11:04.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.048+0000 7fa846a00640 1 -- 172.21.15.27:0/1119467023 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838037070 con 0x7fa8480c58c0 2024-03-28T19:11:04.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.048+0000 7fa846a00640 1 -- 172.21.15.27:0/1119467023 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838031b40 con 0x7fa8480c58c0 2024-03-28T19:11:04.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.049+0000 7fa84dc00640 1 -- 172.21.15.27:0/1119467023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 msgr2=0x7fa8480c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:04.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.049+0000 7fa84dc00640 1 --2- 172.21.15.27:0/1119467023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa8480c5ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa8380097a0 tx=0x7fa83802b820 comp rx=0 tx=0).stop 2024-03-28T19:11:04.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.049+0000 7fa84dc00640 1 -- 172.21.15.27:0/1119467023 shutdown_connections 2024-03-28T19:11:04.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.049+0000 7fa84dc00640 1 --2- 172.21.15.27:0/1119467023 >> v2:172.21.15.155:3300/0 conn(0x7fa8480be5d0 0x7fa848155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.049+0000 7fa84dc00640 1 --2- 172.21.15.27:0/1119467023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa8480c5ca0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.049+0000 7fa84dc00640 1 -- 172.21.15.27:0/1119467023 >> 172.21.15.27:0/1119467023 conn(0x7fa84806b6f0 msgr2=0x7fa8480acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:04.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.049+0000 7fa84dc00640 1 -- 172.21.15.27:0/1119467023 shutdown_connections 2024-03-28T19:11:04.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.049+0000 7fa84dc00640 1 -- 172.21.15.27:0/1119467023 wait complete. 2024-03-28T19:11:04.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.051+0000 7fa84dc00640 1 Processor -- start 2024-03-28T19:11:04.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.051+0000 7fa84dc00640 1 -- start start 2024-03-28T19:11:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa84dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8480be5d0 0x7fa84814c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa84dc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa84814c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa84dc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8480c8ed0 0x7fa8480c2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa84dc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8480b0820 con 0x7fa8480be5d0 2024-03-28T19:11:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa84dc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa848159750 con 0x7fa8480c8ed0 2024-03-28T19:11:04.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa84dc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8480b09a0 con 0x7fa8480c58c0 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa847e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8480be5d0 0x7fa84814c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa847e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8480be5d0 0x7fa84814c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59534/0 (socket says 172.21.15.27:59534) 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa847e00640 1 -- 172.21.15.27:0/2192864017 learned_addr learned my addr 172.21.15.27:0/2192864017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.052+0000 7fa847400640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa84814c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.053+0000 7fa84d200640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8480c8ed0 0x7fa8480c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.053+0000 7fa847e00640 1 -- 172.21.15.27:0/2192864017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 msgr2=0x7fa84814c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.053+0000 7fa847e00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa84814c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.053+0000 7fa847e00640 1 -- 172.21.15.27:0/2192864017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8480c8ed0 msgr2=0x7fa8480c2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.053+0000 7fa847e00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8480c8ed0 0x7fa8480c2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.053+0000 7fa847e00640 1 -- 172.21.15.27:0/2192864017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8481ebcf0 con 0x7fa8480be5d0 2024-03-28T19:11:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.053+0000 7fa847e00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8480be5d0 0x7fa84814c400 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fa8380098c0 tx=0x7fa8380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:04.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.054+0000 7fa83fe00640 1 -- 172.21.15.27:0/2192864017 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838032610 con 0x7fa8480be5d0 2024-03-28T19:11:04.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.054+0000 7fa83fe00640 1 -- 172.21.15.27:0/2192864017 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8380327b0 con 0x7fa8480be5d0 2024-03-28T19:11:04.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.054+0000 7fa83fe00640 1 -- 172.21.15.27:0/2192864017 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838032ba0 con 0x7fa8480be5d0 2024-03-28T19:11:04.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.054+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8480c4090 con 0x7fa8480be5d0 2024-03-28T19:11:04.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.054+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8480c4700 con 0x7fa8480be5d0 2024-03-28T19:11:04.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.055+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa848069640 con 0x7fa8480be5d0 2024-03-28T19:11:04.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.056+0000 7fa83fe00640 1 -- 172.21.15.27:0/2192864017 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa83804e440 con 0x7fa8480be5d0 2024-03-28T19:11:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.057+0000 7fa83fe00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa82806bbd0 0x7fa82806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.057+0000 7fa83fe00640 1 -- 172.21.15.27:0/2192864017 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7fa8380ad140 con 0x7fa8480be5d0 2024-03-28T19:11:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.058+0000 7fa847400640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa82806bbd0 0x7fa82806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:04.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.058+0000 7fa847400640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa82806bbd0 0x7fa82806e090 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa8480c8cf0 tx=0x7fa834009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:04.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.062+0000 7fa83fe00640 1 -- 172.21.15.27:0/2192864017 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa8480c4090 con 0x7fa8480be5d0 2024-03-28T19:11:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.339+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa848156580 con 0x7fa8480be5d0 2024-03-28T19:11:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.340+0000 7fa83fe00640 1 -- 172.21.15.27:0/2192864017 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5465 (secure 0 0 0) 0x7fa838040070 con 0x7fa8480be5d0 2024-03-28T19:11:04.339 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:11:04.339 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:02.588239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:11:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 124 2024-03-28T19:11:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.343+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa82806bbd0 msgr2=0x7fa82806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.343+0000 7fa84dc00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa82806bbd0 0x7fa82806e090 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa8480c8cf0 tx=0x7fa834009040 comp rx=0 tx=0).stop 2024-03-28T19:11:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.343+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8480be5d0 msgr2=0x7fa84814c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.343+0000 7fa84dc00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8480be5d0 0x7fa84814c400 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fa8380098c0 tx=0x7fa8380046c0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.344+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 shutdown_connections 2024-03-28T19:11:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.344+0000 7fa84dc00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa82806bbd0 0x7fa82806e090 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.344+0000 7fa84dc00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8480c8ed0 0x7fa8480c2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.344+0000 7fa84dc00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa8480c58c0 0x7fa84814c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.344+0000 7fa84dc00640 1 --2- 172.21.15.27:0/2192864017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8480be5d0 0x7fa84814c400 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.344+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 >> 172.21.15.27:0/2192864017 conn(0x7fa84806b6f0 msgr2=0x7fa8480aec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.344+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 shutdown_connections 2024-03-28T19:11:04.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:04.344+0000 7fa84dc00640 1 -- 172.21.15.27:0/2192864017 wait complete. 2024-03-28T19:11:04.364 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:11:05.829 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:11:05.829+0000 7f8f90000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:11:06.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:11:06.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.601+0000 7f2a6d200640 1 Processor -- start 2024-03-28T19:11:06.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.601+0000 7f2a6d200640 1 -- start start 2024-03-28T19:11:06.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.602+0000 7f2a6d200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2a680c58c0 0x7f2a680c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:06.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.602+0000 7f2a6d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a680c8ed0 0x7f2a681555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:06.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.602+0000 7f2a6d200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a68156230 con 0x7f2a680be5d0 2024-03-28T19:11:06.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.602+0000 7f2a6d200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a68158bd0 con 0x7f2a680c8ed0 2024-03-28T19:11:06.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.602+0000 7f2a6d200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a68158eb0 con 0x7f2a680c58c0 2024-03-28T19:11:06.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.602+0000 7f2a66a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a680c8ed0 0x7f2a681555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:06.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.603+0000 7f2a67400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2a680c58c0 0x7f2a680c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:06.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.603+0000 7f2a66a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a680c8ed0 0x7f2a681555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37366/0 (socket says 172.21.15.27:37366) 2024-03-28T19:11:06.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.603+0000 7f2a66a00640 1 -- 172.21.15.27:0/4191089421 learned_addr learned my addr 172.21.15.27:0/4191089421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:11:06.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.603+0000 7f2a66a00640 1 -- 172.21.15.27:0/4191089421 >> v2:172.21.15.155:3300/0 conn(0x7f2a680c58c0 msgr2=0x7f2a680c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:11:06.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.603+0000 7f2a66000640 1 -- 172.21.15.27:0/4191089421 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142034024 0 0) 0x7f2a68156230 con 0x7f2a680be5d0 2024-03-28T19:11:06.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.603+0000 7f2a66a00640 1 --2- 172.21.15.27:0/4191089421 >> v2:172.21.15.155:3300/0 conn(0x7f2a680c58c0 0x7f2a680c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.603+0000 7f2a66a00640 1 -- 172.21.15.27:0/4191089421 >> v1:172.21.15.155:6789/0 conn(0x7f2a680be5d0 legacy=0x7f2a68155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:06.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.603+0000 7f2a66a00640 1 -- 172.21.15.27:0/4191089421 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a68159190 con 0x7f2a680c8ed0 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.605+0000 7f2a66a00640 1 --2- 172.21.15.27:0/4191089421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681555e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2a6806c060 tx=0x7f2a5402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06b04d6626c3db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.605+0000 7f2a66000640 1 -- 172.21.15.27:0/4191089421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5402e070 con 0x7f2a680c8ed0 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.605+0000 7f2a66000640 1 -- 172.21.15.27:0/4191089421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a54037070 con 0x7f2a680c8ed0 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.605+0000 7f2a66000640 1 -- 172.21.15.27:0/4191089421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54031a00 con 0x7f2a680c8ed0 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.606+0000 7f2a6d200640 1 -- 172.21.15.27:0/4191089421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 msgr2=0x7f2a681555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.606+0000 7f2a6d200640 1 --2- 172.21.15.27:0/4191089421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681555e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2a6806c060 tx=0x7f2a5402b880 comp rx=0 tx=0).stop 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.606+0000 7f2a6d200640 1 -- 172.21.15.27:0/4191089421 shutdown_connections 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.606+0000 7f2a6d200640 1 --2- 172.21.15.27:0/4191089421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681555e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.606+0000 7f2a6d200640 1 --2- 172.21.15.27:0/4191089421 >> v2:172.21.15.155:3300/0 conn(0x7f2a680c58c0 0x7f2a680c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.606+0000 7f2a6d200640 1 -- 172.21.15.27:0/4191089421 >> 172.21.15.27:0/4191089421 conn(0x7f2a6806b6f0 msgr2=0x7f2a680acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.606+0000 7f2a6d200640 1 -- 172.21.15.27:0/4191089421 shutdown_connections 2024-03-28T19:11:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.606+0000 7f2a6d200640 1 -- 172.21.15.27:0/4191089421 wait complete. 2024-03-28T19:11:06.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.608+0000 7f2a6d200640 1 Processor -- start 2024-03-28T19:11:06.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.608+0000 7f2a6d200640 1 -- start start 2024-03-28T19:11:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a6d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a680be5d0 0x7f2a680c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a6d200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a680c58c0 0x7f2a681539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a6d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681ec270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a6d200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680b0820 con 0x7f2a680c8ed0 2024-03-28T19:11:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a6d200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68159750 con 0x7f2a680be5d0 2024-03-28T19:11:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a6d200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a680b09a0 con 0x7f2a680c58c0 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a67e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a67400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a680be5d0 0x7f2a680c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a67e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681ec270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37368/0 (socket says 172.21.15.27:37368) 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a67e00640 1 -- 172.21.15.27:0/950194706 learned_addr learned my addr 172.21.15.27:0/950194706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.609+0000 7f2a66a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a680c58c0 0x7f2a681539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.610+0000 7f2a67e00640 1 -- 172.21.15.27:0/950194706 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a680c58c0 msgr2=0x7f2a681539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.610+0000 7f2a67e00640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a680c58c0 0x7f2a681539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.610+0000 7f2a67e00640 1 -- 172.21.15.27:0/950194706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a680be5d0 msgr2=0x7f2a680c2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.610+0000 7f2a67e00640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a680be5d0 0x7f2a680c2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.610+0000 7f2a67e00640 1 -- 172.21.15.27:0/950194706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a681ec7b0 con 0x7f2a680c8ed0 2024-03-28T19:11:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.610+0000 7f2a67e00640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681ec270 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c7b0 tx=0x7f2a6000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.611+0000 7f2a5f400640 1 -- 172.21.15.27:0/950194706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60003ba0 con 0x7f2a680c8ed0 2024-03-28T19:11:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.611+0000 7f2a5f400640 1 -- 172.21.15.27:0/950194706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a60003d40 con 0x7f2a680c8ed0 2024-03-28T19:11:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.611+0000 7f2a5f400640 1 -- 172.21.15.27:0/950194706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60005270 con 0x7f2a680c8ed0 2024-03-28T19:11:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.611+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a681528b0 con 0x7f2a680c8ed0 2024-03-28T19:11:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.611+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a680c26a0 con 0x7f2a680c8ed0 2024-03-28T19:11:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.612+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a68069640 con 0x7f2a680c8ed0 2024-03-28T19:11:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.613+0000 7f2a5f400640 1 -- 172.21.15.27:0/950194706 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2a60005410 con 0x7f2a680c8ed0 2024-03-28T19:11:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.615+0000 7f2a5f400640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a4806bbd0 0x7f2a4806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.615+0000 7f2a5f400640 1 -- 172.21.15.27:0/950194706 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f2a6008b300 con 0x7f2a680c8ed0 2024-03-28T19:11:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.615+0000 7f2a67400640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a4806bbd0 0x7f2a4806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.616+0000 7f2a67400640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a4806bbd0 0x7f2a4806e090 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009aa0 tx=0x7f2a58009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:06.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.620+0000 7f2a5f400640 1 -- 172.21.15.27:0/950194706 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2a60058c40 con 0x7f2a680c8ed0 2024-03-28T19:11:06.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.944+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a680b0e70 con 0x7f2a680c8ed0 2024-03-28T19:11:06.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.945+0000 7f2a5f400640 1 -- 172.21.15.27:0/950194706 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5469 (secure 0 0 0) 0x7f2a6005cce0 con 0x7f2a680c8ed0 2024-03-28T19:11:06.944 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:11:06.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:11:06.945 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 126 2024-03-28T19:11:06.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a4806bbd0 msgr2=0x7f2a4806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a4806bbd0 0x7f2a4806e090 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009aa0 tx=0x7f2a58009190 comp rx=0 tx=0).stop 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 msgr2=0x7f2a681ec270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681ec270 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c7b0 tx=0x7f2a6000cbb0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 shutdown_connections 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2a4806bbd0 0x7f2a4806e090 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a680c8ed0 0x7f2a681ec270 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a680c58c0 0x7f2a681539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 --2- 172.21.15.27:0/950194706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a680be5d0 0x7f2a680c2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 >> 172.21.15.27:0/950194706 conn(0x7f2a6806b6f0 msgr2=0x7f2a681e2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:06.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.948+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 shutdown_connections 2024-03-28T19:11:06.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:06.949+0000 7f2a6d200640 1 -- 172.21.15.27:0/950194706 wait complete. 2024-03-28T19:11:06.975 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6118 2024-03-28T19:11:06.976 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.b 2024-03-28T19:11:07.977 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-03-28T19:11:07.977 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:11:07.978 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:11:07.982 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:11:07.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:11:08.177 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:11:08.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.230+0000 7fd449400640 1 Processor -- start 2024-03-28T19:11:08.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.231+0000 7fd449400640 1 -- start start 2024-03-28T19:11:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd449400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4440c74e0 0x7fd4440c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd449400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444152e70 0x7fd4441556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd449400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd444158d00 con 0x7fd4440c2290 2024-03-28T19:11:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd449400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd444158fe0 con 0x7fd4440c74e0 2024-03-28T19:11:08.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd449400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441592c0 con 0x7fd444152e70 2024-03-28T19:11:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd443e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4440c74e0 0x7fd4440c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd443400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd4440c2290 0x7fd444152760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48612/0 (socket says 172.21.15.27:48612) 2024-03-28T19:11:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd443400640 1 -- 172.21.15.27:0/3025852649 learned_addr learned my addr 172.21.15.27:0/3025852649 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:11:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.233+0000 7fd448a00640 1 --2- 172.21.15.27:0/3025852649 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444152e70 0x7fd4441556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.234+0000 7fd443e00640 1 -- 172.21.15.27:0/3025852649 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444152e70 msgr2=0x7fd4441556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:11:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.234+0000 7fd443e00640 1 --2- 172.21.15.27:0/3025852649 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444152e70 0x7fd4441556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.234+0000 7fd443e00640 1 -- 172.21.15.27:0/3025852649 >> v1:172.21.15.27:6789/0 conn(0x7fd4440c2290 legacy=0x7fd444152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:08.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.234+0000 7fd443e00640 1 -- 172.21.15.27:0/3025852649 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4441595a0 con 0x7fd4440c74e0 2024-03-28T19:11:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.234+0000 7fd442a00640 1 -- 172.21.15.27:0/3025852649 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210023850 0 0) 0x7fd444158d00 con 0x7fd4440c2290 2024-03-28T19:11:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.235+0000 7fd443e00640 1 --2- 172.21.15.27:0/3025852649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c74e0 0x7fd4440c1d50 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd4340097a0 tx=0x7fd43402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959f1a42aa704563 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.235+0000 7fd442a00640 1 -- 172.21.15.27:0/3025852649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4340315a0 con 0x7fd4440c74e0 2024-03-28T19:11:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.235+0000 7fd442a00640 1 -- 172.21.15.27:0/3025852649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd434037070 con 0x7fd4440c74e0 2024-03-28T19:11:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.235+0000 7fd442a00640 1 -- 172.21.15.27:0/3025852649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434031960 con 0x7fd4440c74e0 2024-03-28T19:11:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.236+0000 7fd449400640 1 -- 172.21.15.27:0/3025852649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c74e0 msgr2=0x7fd4440c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.236+0000 7fd449400640 1 --2- 172.21.15.27:0/3025852649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c74e0 0x7fd4440c1d50 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd4340097a0 tx=0x7fd43402b840 comp rx=0 tx=0).stop 2024-03-28T19:11:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.236+0000 7fd449400640 1 -- 172.21.15.27:0/3025852649 shutdown_connections 2024-03-28T19:11:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.236+0000 7fd449400640 1 --2- 172.21.15.27:0/3025852649 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd444152e70 0x7fd4441556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.236+0000 7fd449400640 1 --2- 172.21.15.27:0/3025852649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c74e0 0x7fd4440c1d50 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.236+0000 7fd449400640 1 -- 172.21.15.27:0/3025852649 >> 172.21.15.27:0/3025852649 conn(0x7fd44406d980 msgr2=0x7fd4440a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.236+0000 7fd449400640 1 -- 172.21.15.27:0/3025852649 shutdown_connections 2024-03-28T19:11:08.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.236+0000 7fd449400640 1 -- 172.21.15.27:0/3025852649 wait complete. 2024-03-28T19:11:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.238+0000 7fd449400640 1 Processor -- start 2024-03-28T19:11:08.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.238+0000 7fd449400640 1 -- start start 2024-03-28T19:11:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd449400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c2290 0x7fd4441472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd449400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4440c74e0 0x7fd444071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd449400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152e70 0x7fd444156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd449400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4440c78c0 con 0x7fd4440c2290 2024-03-28T19:11:08.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd449400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441589f0 con 0x7fd444152e70 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd449400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4440c7a40 con 0x7fd4440c74e0 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd443e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c2290 0x7fd4441472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd448a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152e70 0x7fd444156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd443e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c2290 0x7fd4441472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37380/0 (socket says 172.21.15.27:37380) 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.239+0000 7fd443e00640 1 -- 172.21.15.27:0/3067164905 learned_addr learned my addr 172.21.15.27:0/3067164905 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.240+0000 7fd443400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4440c74e0 0x7fd444071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.240+0000 7fd443e00640 1 -- 172.21.15.27:0/3067164905 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4440c74e0 msgr2=0x7fd444071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.240+0000 7fd443e00640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4440c74e0 0x7fd444071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.240+0000 7fd443e00640 1 -- 172.21.15.27:0/3067164905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152e70 msgr2=0x7fd444156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.240+0000 7fd443e00640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152e70 0x7fd444156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.240+0000 7fd443e00640 1 -- 172.21.15.27:0/3067164905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4441587e0 con 0x7fd4440c2290 2024-03-28T19:11:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.241+0000 7fd443e00640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c2290 0x7fd4441472f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd434034040 tx=0x7fd4340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.241+0000 7fd43be00640 1 -- 172.21.15.27:0/3067164905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434032630 con 0x7fd4440c2290 2024-03-28T19:11:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.241+0000 7fd43be00640 1 -- 172.21.15.27:0/3067164905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4340327f0 con 0x7fd4440c2290 2024-03-28T19:11:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.241+0000 7fd43be00640 1 -- 172.21.15.27:0/3067164905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434032be0 con 0x7fd4440c2290 2024-03-28T19:11:08.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.241+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd444072e40 con 0x7fd4440c2290 2024-03-28T19:11:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.241+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4441ef280 con 0x7fd4440c2290 2024-03-28T19:11:08.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.243+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd444067dd0 con 0x7fd4440c2290 2024-03-28T19:11:08.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.244+0000 7fd43be00640 1 -- 172.21.15.27:0/3067164905 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd43404e430 con 0x7fd4440c2290 2024-03-28T19:11:08.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.245+0000 7fd43be00640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:11:08.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.245+0000 7fd43be00640 1 -- 172.21.15.27:0/3067164905 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7fd43402f030 con 0x7fd4440c2290 2024-03-28T19:11:08.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.245+0000 7fd443400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:11:08.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.246+0000 7fd443400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd4441e8d10 tx=0x7fd430007d50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:11:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.250+0000 7fd43be00640 1 -- 172.21.15.27:0/3067164905 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd434040070 con 0x7fd4440c2290 2024-03-28T19:11:08.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.590+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd444069640 con 0x7fd4440c2290 2024-03-28T19:11:08.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.590+0000 7fd43be00640 1 -- 172.21.15.27:0/3067164905 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fd43407ac90 con 0x7fd4440c2290 2024-03-28T19:11:08.590 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:11:08.591 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:11:08.591 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:11:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 msgr2=0x7fd42406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd4441e8d10 tx=0x7fd430007d50 comp rx=0 tx=0).stop 2024-03-28T19:11:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c2290 msgr2=0x7fd4441472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:11:08.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c2290 0x7fd4441472f0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd434034040 tx=0x7fd4340046c0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 shutdown_connections 2024-03-28T19:11:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd42406bbd0 0x7fd42406e090 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd444152e70 0x7fd444156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4440c74e0 0x7fd444071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 --2- 172.21.15.27:0/3067164905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4440c2290 0x7fd4441472f0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:11:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 >> 172.21.15.27:0/3067164905 conn(0x7fd44406d980 msgr2=0x7fd44414ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:11:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 shutdown_connections 2024-03-28T19:11:08.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:11:08.593+0000 7fd449400640 1 -- 172.21.15.27:0/3067164905 wait complete. 2024-03-28T19:11:08.614 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-03-28T19:11:08.615 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2024-03-28T19:11:31.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:11:31.964 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:12:01.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:01.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.345+0000 7fd148800640 1 Processor -- start 2024-03-28T19:12:01.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.345+0000 7fd148800640 1 -- start start 2024-03-28T19:12:01.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd148800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd140071d30 0x7fd140154590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd148800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1400c5bb0 0x7fd1401570b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd148800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1400c6b70 con 0x7fd14006c340 2024-03-28T19:12:01.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd148800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1400c79f0 con 0x7fd1400c5bb0 2024-03-28T19:12:01.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd148800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1400c65f0 con 0x7fd140071d30 2024-03-28T19:12:01.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd145400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd140071d30 0x7fd140154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd146800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1400c5bb0 0x7fd1401570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd146800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1400c5bb0 0x7fd1401570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40082/0 (socket says 172.21.15.27:40082) 2024-03-28T19:12:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd145400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd140071d30 0x7fd140154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:40042/0 (socket says 172.21.15.27:40042) 2024-03-28T19:12:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.347+0000 7fd146800640 1 -- 172.21.15.27:0/4233194027 learned_addr learned my addr 172.21.15.27:0/4233194027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.348+0000 7fd146800640 1 -- 172.21.15.27:0/4233194027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd140071d30 msgr2=0x7fd140154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.348+0000 7fd146800640 1 --2- 172.21.15.27:0/4233194027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd140071d30 0x7fd140154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.348+0000 7fd146800640 1 -- 172.21.15.27:0/4233194027 >> v1:172.21.15.155:6789/0 conn(0x7fd14006c340 legacy=0x7fd140153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.348+0000 7fd146800640 1 -- 172.21.15.27:0/4233194027 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400c2a30 con 0x7fd1400c5bb0 2024-03-28T19:12:01.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.348+0000 7fd144a00640 1 -- 172.21.15.27:0/4233194027 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065402160 0 0) 0x7fd134001110 con 0x7fd14006c340 2024-03-28T19:12:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.349+0000 7fd146800640 1 --2- 172.21.15.27:0/4233194027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1400c5bb0 0x7fd1401570b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009e50 tx=0x7fd13c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1e86f06be1d887 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.349+0000 7fd144a00640 1 -- 172.21.15.27:0/4233194027 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c02cb00 con 0x7fd1400c5bb0 2024-03-28T19:12:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.350+0000 7fd144a00640 1 -- 172.21.15.27:0/4233194027 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c02cca0 con 0x7fd1400c5bb0 2024-03-28T19:12:01.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.350+0000 7fd144a00640 1 -- 172.21.15.27:0/4233194027 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c034610 con 0x7fd1400c5bb0 2024-03-28T19:12:01.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.350+0000 7fd148800640 1 -- 172.21.15.27:0/4233194027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1400c5bb0 msgr2=0x7fd1401570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.350+0000 7fd148800640 1 --2- 172.21.15.27:0/4233194027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1400c5bb0 0x7fd1401570b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009e50 tx=0x7fd13c02bf60 comp rx=0 tx=0).stop 2024-03-28T19:12:01.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.351+0000 7fd148800640 1 -- 172.21.15.27:0/4233194027 shutdown_connections 2024-03-28T19:12:01.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.351+0000 7fd148800640 1 --2- 172.21.15.27:0/4233194027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1400c5bb0 0x7fd1401570b0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.351+0000 7fd148800640 1 --2- 172.21.15.27:0/4233194027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd140071d30 0x7fd140154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.351+0000 7fd148800640 1 -- 172.21.15.27:0/4233194027 >> 172.21.15.27:0/4233194027 conn(0x7fd1400c4750 msgr2=0x7fd1400c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:01.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.351+0000 7fd148800640 1 -- 172.21.15.27:0/4233194027 shutdown_connections 2024-03-28T19:12:01.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.351+0000 7fd148800640 1 -- 172.21.15.27:0/4233194027 wait complete. 2024-03-28T19:12:01.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.353+0000 7fd148800640 1 Processor -- start 2024-03-28T19:12:01.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.353+0000 7fd148800640 1 -- start start 2024-03-28T19:12:01.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.354+0000 7fd148800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14006c340 0x7fd140155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.354+0000 7fd148800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd140071d30 0x7fd1401dc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.354+0000 7fd148800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1400c5bb0 0x7fd1401ee490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.354+0000 7fd148800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1400736c0 con 0x7fd140071d30 2024-03-28T19:12:01.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.354+0000 7fd148800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1400c6890 con 0x7fd1400c5bb0 2024-03-28T19:12:01.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.354+0000 7fd148800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd140071a20 con 0x7fd14006c340 2024-03-28T19:12:01.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd145400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd140071d30 0x7fd1401dc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd146800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1400c5bb0 0x7fd1401ee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd145e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14006c340 0x7fd140155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd146800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1400c5bb0 0x7fd1401ee490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:46150/0 (socket says 172.21.15.27:46150) 2024-03-28T19:12:01.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd146800640 1 -- 172.21.15.27:0/850807109 learned_addr learned my addr 172.21.15.27:0/850807109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:01.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd145400640 1 -- 172.21.15.27:0/850807109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14006c340 msgr2=0x7fd140155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd145400640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14006c340 0x7fd140155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd145400640 1 -- 172.21.15.27:0/850807109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1400c5bb0 msgr2=0x7fd1401ee490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd145400640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1400c5bb0 0x7fd1401ee490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.355+0000 7fd145400640 1 -- 172.21.15.27:0/850807109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400accb0 con 0x7fd140071d30 2024-03-28T19:12:01.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.356+0000 7fd145400640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd140071d30 0x7fd1401dc380 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e820 tx=0x7fd13000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:01.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.356+0000 7fd13a000640 1 -- 172.21.15.27:0/850807109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130017070 con 0x7fd140071d30 2024-03-28T19:12:01.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.356+0000 7fd13a000640 1 -- 172.21.15.27:0/850807109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13000c940 con 0x7fd140071d30 2024-03-28T19:12:01.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.356+0000 7fd13a000640 1 -- 172.21.15.27:0/850807109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13000cd10 con 0x7fd140071d30 2024-03-28T19:12:01.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.356+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1400c3c00 con 0x7fd140071d30 2024-03-28T19:12:01.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.356+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1401dce40 con 0x7fd140071d30 2024-03-28T19:12:01.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.357+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd140068ce0 con 0x7fd140071d30 2024-03-28T19:12:01.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.360+0000 7fd13a000640 1 -- 172.21.15.27:0/850807109 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd13001a070 con 0x7fd140071d30 2024-03-28T19:12:01.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.361+0000 7fd13a000640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12406bc20 0x7fd12406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.361+0000 7fd13a000640 1 -- 172.21.15.27:0/850807109 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7fd13008b6f0 con 0x7fd140071d30 2024-03-28T19:12:01.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.362+0000 7fd145e00640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12406bc20 0x7fd12406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.363+0000 7fd145e00640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12406bc20 0x7fd12406e0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd134000f60 tx=0x7fd134009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:01.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.372+0000 7fd13a000640 1 -- 172.21.15.27:0/850807109 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd130059140 con 0x7fd140071d30 2024-03-28T19:12:01.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:01.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.715+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd14004f400 con 0x7fd140071d30 2024-03-28T19:12:01.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.716+0000 7fd13a000640 1 -- 172.21.15.27:0/850807109 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fd13005d1e0 con 0x7fd140071d30 2024-03-28T19:12:01.716 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:01.716 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:01.716 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:01.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12406bc20 msgr2=0x7fd12406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12406bc20 0x7fd12406e0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd134000f60 tx=0x7fd134009040 comp rx=0 tx=0).stop 2024-03-28T19:12:01.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd140071d30 msgr2=0x7fd1401dc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd140071d30 0x7fd1401dc380 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e820 tx=0x7fd13000ece0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 shutdown_connections 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd12406bc20 0x7fd12406e0e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1400c5bb0 0x7fd1401ee490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd140071d30 0x7fd1401dc380 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 --2- 172.21.15.27:0/850807109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd14006c340 0x7fd140155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.719+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 >> 172.21.15.27:0/850807109 conn(0x7fd1400c4750 msgr2=0x7fd140152170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.720+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 shutdown_connections 2024-03-28T19:12:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.720+0000 7fd148800640 1 -- 172.21.15.27:0/850807109 wait complete. 2024-03-28T19:12:01.742 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-03-28T19:12:01.743 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T19:12:01.743 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:12:01.768 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T19:12:01.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:01.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.835+0000 7f1ff8e00640 1 Processor -- start 2024-03-28T19:12:01.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.835+0000 7f1ff8e00640 1 -- start start 2024-03-28T19:12:01.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.836+0000 7f1ff8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ff40a6300 0x7f1ff40733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.836+0000 7f1ff8e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1ff4070a50 0x7f1ff4155a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.836+0000 7f1ff8e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff406d7d0 con 0x7f1ff406ceb0 2024-03-28T19:12:01.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.836+0000 7f1ff8e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff40c2f00 con 0x7f1ff40a6300 2024-03-28T19:12:01.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.836+0000 7f1ff8e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff40c31e0 con 0x7f1ff4070a50 2024-03-28T19:12:01.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.837+0000 7f1ff1e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f1ff406ceb0 0x7f1ff40c1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:37260/0 (socket says 172.21.15.27:37260) 2024-03-28T19:12:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.837+0000 7f1ff1e00640 1 -- 172.21.15.27:0/2092377019 learned_addr learned my addr 172.21.15.27:0/2092377019 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.837+0000 7f1ff1400640 1 --2- 172.21.15.27:0/2092377019 >> v2:172.21.15.27:3300/0 conn(0x7f1ff40a6300 0x7f1ff40733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.837+0000 7f1ff2800640 1 --2- 172.21.15.27:0/2092377019 >> v2:172.21.15.155:3300/0 conn(0x7f1ff4070a50 0x7f1ff4155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.837+0000 7f1ff1400640 1 -- 172.21.15.27:0/2092377019 >> v2:172.21.15.155:3300/0 conn(0x7f1ff4070a50 msgr2=0x7f1ff4155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.837+0000 7f1ff1400640 1 --2- 172.21.15.27:0/2092377019 >> v2:172.21.15.155:3300/0 conn(0x7f1ff4070a50 0x7f1ff4155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.837+0000 7f1ff1400640 1 -- 172.21.15.27:0/2092377019 >> v1:172.21.15.155:6789/0 conn(0x7f1ff406ceb0 legacy=0x7f1ff40c1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.837+0000 7f1ff1400640 1 -- 172.21.15.27:0/2092377019 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff406dab0 con 0x7f1ff40a6300 2024-03-28T19:12:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.838+0000 7f1ff1400640 1 --2- 172.21.15.27:0/2092377019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40a6300 0x7f1ff40733b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40709e0 tx=0x7f1fe002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c040568c809a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.839+0000 7f1ff0a00640 1 -- 172.21.15.27:0/2092377019 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe00315a0 con 0x7f1ff40a6300 2024-03-28T19:12:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.839+0000 7f1ff0a00640 1 -- 172.21.15.27:0/2092377019 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe0037070 con 0x7f1ff40a6300 2024-03-28T19:12:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.839+0000 7f1ff0a00640 1 -- 172.21.15.27:0/2092377019 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0031960 con 0x7f1ff40a6300 2024-03-28T19:12:01.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.839+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2092377019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40a6300 msgr2=0x7f1ff40733b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.839+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2092377019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40a6300 0x7f1ff40733b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40709e0 tx=0x7f1fe002b840 comp rx=0 tx=0).stop 2024-03-28T19:12:01.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.840+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2092377019 shutdown_connections 2024-03-28T19:12:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.840+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2092377019 >> v2:172.21.15.155:3300/0 conn(0x7f1ff4070a50 0x7f1ff4155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.840+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2092377019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40a6300 0x7f1ff40733b0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.840+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2092377019 >> 172.21.15.27:0/2092377019 conn(0x7f1ff406b9d0 msgr2=0x7f1ff40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.840+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2092377019 shutdown_connections 2024-03-28T19:12:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.840+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2092377019 wait complete. 2024-03-28T19:12:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.842+0000 7f1ff8e00640 1 Processor -- start 2024-03-28T19:12:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.842+0000 7f1ff8e00640 1 -- start start 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.843+0000 7f1ff8e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ff406ceb0 0x7f1ff414c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.843+0000 7f1ff8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff4070a50 0x7f1ff4150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.843+0000 7f1ff8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff40a6300 0x7f1ff4155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.843+0000 7f1ff8e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40c45b0 con 0x7f1ff4070a50 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.843+0000 7f1ff8e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40c4430 con 0x7f1ff40a6300 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.843+0000 7f1ff8e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff406d290 con 0x7f1ff406ceb0 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff4070a50 0x7f1ff4150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff4070a50 0x7f1ff4150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40110/0 (socket says 172.21.15.27:40110) 2024-03-28T19:12:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1400640 1 -- 172.21.15.27:0/2279223677 learned_addr learned my addr 172.21.15.27:0/2279223677 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ff406ceb0 0x7f1ff414c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff2800640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff40a6300 0x7f1ff4155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1400640 1 -- 172.21.15.27:0/2279223677 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ff406ceb0 msgr2=0x7f1ff414c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1400640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ff406ceb0 0x7f1ff414c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1400640 1 -- 172.21.15.27:0/2279223677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff40a6300 msgr2=0x7f1ff4155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1400640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff40a6300 0x7f1ff4155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.844+0000 7f1ff1400640 1 -- 172.21.15.27:0/2279223677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40bdce0 con 0x7f1ff4070a50 2024-03-28T19:12:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.845+0000 7f1ff1400640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff4070a50 0x7f1ff4150fe0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f1fe002be50 tx=0x7f1fe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:01.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.845+0000 7f1fea000640 1 -- 172.21.15.27:0/2279223677 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0032630 con 0x7f1ff4070a50 2024-03-28T19:12:01.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.845+0000 7f1fea000640 1 -- 172.21.15.27:0/2279223677 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe00327d0 con 0x7f1ff4070a50 2024-03-28T19:12:01.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.845+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff40becb0 con 0x7f1ff4070a50 2024-03-28T19:12:01.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.845+0000 7f1fea000640 1 -- 172.21.15.27:0/2279223677 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0032bc0 con 0x7f1ff4070a50 2024-03-28T19:12:01.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.845+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff41ec070 con 0x7f1ff4070a50 2024-03-28T19:12:01.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.847+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff404f400 con 0x7f1ff4070a50 2024-03-28T19:12:01.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.848+0000 7f1fea000640 1 -- 172.21.15.27:0/2279223677 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1fe004c020 con 0x7f1ff4070a50 2024-03-28T19:12:01.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.850+0000 7f1fea000640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1fd406bbd0 0x7f1fd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.850+0000 7f1fea000640 1 -- 172.21.15.27:0/2279223677 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f1fe002f020 con 0x7f1ff4070a50 2024-03-28T19:12:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.850+0000 7f1ff1e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1fd406bbd0 0x7f1fd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:01.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.851+0000 7f1ff1e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1fd406bbd0 0x7f1fd406e090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4072220 tx=0x7f1fe4007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:01.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:01.857+0000 7f1fea000640 1 -- 172.21.15.27:0/2279223677 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1fe0040070 con 0x7f1ff4070a50 2024-03-28T19:12:02.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.002+0000 7f7656000640 1 Processor -- start 2024-03-28T19:12:02.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.002+0000 7f7656000640 1 -- start start 2024-03-28T19:12:02.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f7656000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f765006ceb0 0x7f76500733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f7656000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7650070a50 0x7f7650155a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f7656000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765006d7d0 con 0x7f76500a6300 2024-03-28T19:12:02.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f7656000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76500c2f00 con 0x7f765006ceb0 2024-03-28T19:12:02.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f7656000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76500c31e0 con 0x7f7650070a50 2024-03-28T19:12:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f764f400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f765006ceb0 0x7f76500733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f764fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7650070a50 0x7f7650155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f764f400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f765006ceb0 0x7f76500733b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:46162/0 (socket says 172.21.15.27:46162) 2024-03-28T19:12:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f764f400640 1 -- 172.21.15.27:0/2827613350 learned_addr learned my addr 172.21.15.27:0/2827613350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.004+0000 7f764ea00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f76500a6300 0x7f76500c1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:37274/0 (socket says 172.21.15.27:37274) 2024-03-28T19:12:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.005+0000 7f764fe00640 1 -- 172.21.15.27:0/2827613350 >> v2:172.21.15.155:3300/0 conn(0x7f765006ceb0 msgr2=0x7f76500733b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.005+0000 7f764fe00640 1 --2- 172.21.15.27:0/2827613350 >> v2:172.21.15.155:3300/0 conn(0x7f765006ceb0 0x7f76500733b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.005+0000 7f764fe00640 1 -- 172.21.15.27:0/2827613350 >> v1:172.21.15.155:6789/0 conn(0x7f76500a6300 legacy=0x7f76500c1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.005+0000 7f764fe00640 1 -- 172.21.15.27:0/2827613350 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765006dab0 con 0x7f7650070a50 2024-03-28T19:12:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.006+0000 7f764fe00640 1 --2- 172.21.15.27:0/2827613350 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7650070a50 0x7f7650155a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7648009d90 tx=0x7f764802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40d16f19e86572f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.007+0000 7f764e000640 1 -- 172.21.15.27:0/2827613350 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648039070 con 0x7f7650070a50 2024-03-28T19:12:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.007+0000 7f764e000640 1 -- 172.21.15.27:0/2827613350 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764802cbc0 con 0x7f7650070a50 2024-03-28T19:12:02.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.007+0000 7f764e000640 1 -- 172.21.15.27:0/2827613350 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648034440 con 0x7f7650070a50 2024-03-28T19:12:02.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.008+0000 7f7656000640 1 -- 172.21.15.27:0/2827613350 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7650070a50 msgr2=0x7f7650155a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.008+0000 7f7656000640 1 --2- 172.21.15.27:0/2827613350 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7650070a50 0x7f7650155a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7648009d90 tx=0x7f764802be10 comp rx=0 tx=0).stop 2024-03-28T19:12:02.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.008+0000 7f7656000640 1 -- 172.21.15.27:0/2827613350 shutdown_connections 2024-03-28T19:12:02.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.008+0000 7f7656000640 1 --2- 172.21.15.27:0/2827613350 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7650070a50 0x7f7650155a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.008+0000 7f7656000640 1 --2- 172.21.15.27:0/2827613350 >> v2:172.21.15.155:3300/0 conn(0x7f765006ceb0 0x7f76500733b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.008+0000 7f7656000640 1 -- 172.21.15.27:0/2827613350 >> 172.21.15.27:0/2827613350 conn(0x7f765006b9d0 msgr2=0x7f76500c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:02.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.008+0000 7f7656000640 1 -- 172.21.15.27:0/2827613350 shutdown_connections 2024-03-28T19:12:02.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.008+0000 7f7656000640 1 -- 172.21.15.27:0/2827613350 wait complete. 2024-03-28T19:12:02.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.011+0000 7f7656000640 1 Processor -- start 2024-03-28T19:12:02.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.011+0000 7f7656000640 1 -- start start 2024-03-28T19:12:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f7656000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765006ceb0 0x7f76500bbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f7656000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7650070a50 0x7f76501dbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f7656000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76500a6300 0x7f76501dc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f7656000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76500c45b0 con 0x7f765006ceb0 2024-03-28T19:12:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f7656000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76500c4430 con 0x7f7650070a50 2024-03-28T19:12:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f7656000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765006d290 con 0x7f76500a6300 2024-03-28T19:12:02.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f764f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765006ceb0 0x7f76500bbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f764f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765006ceb0 0x7f76500bbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40124/0 (socket says 172.21.15.27:40124) 2024-03-28T19:12:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f764f400640 1 -- 172.21.15.27:0/1747351699 learned_addr learned my addr 172.21.15.27:0/1747351699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f764fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76500a6300 0x7f76501dc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.013+0000 7f764ea00640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7650070a50 0x7f76501dbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.014+0000 7f764f400640 1 -- 172.21.15.27:0/1747351699 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76500a6300 msgr2=0x7f76501dc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.014+0000 7f764f400640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76500a6300 0x7f76501dc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.014+0000 7f764f400640 1 -- 172.21.15.27:0/1747351699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7650070a50 msgr2=0x7f76501dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.014+0000 7f764f400640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7650070a50 0x7f76501dbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.014+0000 7f764f400640 1 -- 172.21.15.27:0/1747351699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76501d58e0 con 0x7f765006ceb0 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.014+0000 7f764f400640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765006ceb0 0x7f76500bbe90 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7640009fa0 tx=0x7f764000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.014+0000 7f7647400640 1 -- 172.21.15.27:0/1747351699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764001e070 con 0x7f765006ceb0 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.015+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76501e75f0 con 0x7f765006ceb0 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.015+0000 7f7647400640 1 -- 172.21.15.27:0/1747351699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764000c910 con 0x7f765006ceb0 2024-03-28T19:12:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.015+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76500bc3d0 con 0x7f765006ceb0 2024-03-28T19:12:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.015+0000 7f7647400640 1 -- 172.21.15.27:0/1747351699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764000cd00 con 0x7f765006ceb0 2024-03-28T19:12:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.016+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7628005290 con 0x7f765006ceb0 2024-03-28T19:12:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.018+0000 7f7647400640 1 -- 172.21.15.27:0/1747351699 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7640021070 con 0x7f765006ceb0 2024-03-28T19:12:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.019+0000 7f7647400640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763006bbd0 0x7f763006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.019+0000 7f7647400640 1 -- 172.21.15.27:0/1747351699 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f764008cfd0 con 0x7f765006ceb0 2024-03-28T19:12:02.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.019+0000 7f764ea00640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763006bbd0 0x7f763006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.020+0000 7f764ea00640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763006bbd0 0x7f763006e090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f76501d5b00 tx=0x7f763c008f90 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:02.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.024+0000 7f7647400640 1 -- 172.21.15.27:0/1747351699 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f764005a9a0 con 0x7f765006ceb0 2024-03-28T19:12:02.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:12:02.109 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:12:02.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.124+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ff4068390 con 0x7f1ff4070a50 2024-03-28T19:12:02.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.125+0000 7f1fea000640 1 -- 172.21.15.27:0/2279223677 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f1fe007a900 con 0x7f1ff4070a50 2024-03-28T19:12:02.125 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:02.125 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:02.125 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:02.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1fd406bbd0 msgr2=0x7f1fd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1fd406bbd0 0x7f1fd406e090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4072220 tx=0x7f1fe4007e80 comp rx=0 tx=0).stop 2024-03-28T19:12:02.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff4070a50 msgr2=0x7f1ff4150fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff4070a50 0x7f1ff4150fe0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f1fe002be50 tx=0x7f1fe00046c0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 shutdown_connections 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1fd406bbd0 0x7f1fd406e090 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff40a6300 0x7f1ff4155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff4070a50 0x7f1ff4150fe0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 --2- 172.21.15.27:0/2279223677 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ff406ceb0 0x7f1ff414c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 >> 172.21.15.27:0/2279223677 conn(0x7f1ff406b9d0 msgr2=0x7f1ff40aadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 shutdown_connections 2024-03-28T19:12:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.128+0000 7f1ff8e00640 1 -- 172.21.15.27:0/2279223677 wait complete. 2024-03-28T19:12:02.146 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-03-28T19:12:02.146 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:12:02.147 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:12:02.215 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:12:02.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:02.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.296+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7628005580 con 0x7f765006ceb0 2024-03-28T19:12:02.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.297+0000 7f7647400640 1 -- 172.21.15.27:0/1747351699 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f764005ea40 con 0x7f765006ceb0 2024-03-28T19:12:02.296 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:02.296 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:02.296 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.299+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763006bbd0 msgr2=0x7f763006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763006bbd0 0x7f763006e090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f76501d5b00 tx=0x7f763c008f90 comp rx=0 tx=0).stop 2024-03-28T19:12:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765006ceb0 msgr2=0x7f76500bbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765006ceb0 0x7f76500bbe90 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7640009fa0 tx=0x7f764000ecd0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 shutdown_connections 2024-03-28T19:12:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763006bbd0 0x7f763006e090 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f76500a6300 0x7f76501dc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7650070a50 0x7f76501dbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 --2- 172.21.15.27:0/1747351699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765006ceb0 0x7f76500bbe90 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 >> 172.21.15.27:0/1747351699 conn(0x7f765006b9d0 msgr2=0x7f76500aa070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 shutdown_connections 2024-03-28T19:12:02.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.300+0000 7f7656000640 1 -- 172.21.15.27:0/1747351699 wait complete. 2024-03-28T19:12:02.317 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:12:02.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.465+0000 7f9081600640 1 Processor -- start 2024-03-28T19:12:02.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.466+0000 7f9081600640 1 -- start start 2024-03-28T19:12:02.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.467+0000 7f9081600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f907c150ae0 0x7f907c0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.467+0000 7f9081600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f907c153a30 0x7f907c153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.467+0000 7f9081600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c150ec0 con 0x7f907c0a61a0 2024-03-28T19:12:02.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.467+0000 7f9081600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907c1543c0 con 0x7f907c150ae0 2024-03-28T19:12:02.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.467+0000 7f9081600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907c06dc70 con 0x7f907c153a30 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f907be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f907c150ae0 0x7f907c0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f9073e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f907c0a61a0 0x7f907c0c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:37276/0 (socket says 172.21.15.27:37276) 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f9073e00640 1 -- 172.21.15.27:0/917040763 learned_addr learned my addr 172.21.15.27:0/917040763 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f9080c00640 1 --2- 172.21.15.27:0/917040763 >> v2:172.21.15.155:3300/0 conn(0x7f907c153a30 0x7f907c153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f907be00640 1 -- 172.21.15.27:0/917040763 >> v2:172.21.15.155:3300/0 conn(0x7f907c153a30 msgr2=0x7f907c153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f907be00640 1 --2- 172.21.15.27:0/917040763 >> v2:172.21.15.155:3300/0 conn(0x7f907c153a30 0x7f907c153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f907be00640 1 -- 172.21.15.27:0/917040763 >> v1:172.21.15.155:6789/0 conn(0x7f907c0a61a0 legacy=0x7f907c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f907be00640 1 -- 172.21.15.27:0/917040763 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c153550 con 0x7f907c150ae0 2024-03-28T19:12:02.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.468+0000 7f907b400640 1 -- 172.21.15.27:0/917040763 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325394009 0 0) 0x7f9064001110 con 0x7f907c0a61a0 2024-03-28T19:12:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.469+0000 7f907be00640 1 --2- 172.21.15.27:0/917040763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c150ae0 0x7f907c0a5c60 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f907c073830 tx=0x7f906c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4edbb2973811d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.470+0000 7f907b400640 1 -- 172.21.15.27:0/917040763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c0315a0 con 0x7f907c150ae0 2024-03-28T19:12:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.470+0000 7f907b400640 1 -- 172.21.15.27:0/917040763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c037070 con 0x7f907c150ae0 2024-03-28T19:12:02.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.470+0000 7f907b400640 1 -- 172.21.15.27:0/917040763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c031960 con 0x7f907c150ae0 2024-03-28T19:12:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.470+0000 7f9081600640 1 -- 172.21.15.27:0/917040763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c150ae0 msgr2=0x7f907c0a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.470+0000 7f9081600640 1 --2- 172.21.15.27:0/917040763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c150ae0 0x7f907c0a5c60 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f907c073830 tx=0x7f906c02b840 comp rx=0 tx=0).stop 2024-03-28T19:12:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.471+0000 7f9081600640 1 -- 172.21.15.27:0/917040763 shutdown_connections 2024-03-28T19:12:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.471+0000 7f9081600640 1 --2- 172.21.15.27:0/917040763 >> v2:172.21.15.155:3300/0 conn(0x7f907c153a30 0x7f907c153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.471+0000 7f9081600640 1 --2- 172.21.15.27:0/917040763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c150ae0 0x7f907c0a5c60 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.471+0000 7f9081600640 1 -- 172.21.15.27:0/917040763 >> 172.21.15.27:0/917040763 conn(0x7f907c06b0c0 msgr2=0x7f907c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:02.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.471+0000 7f9081600640 1 -- 172.21.15.27:0/917040763 shutdown_connections 2024-03-28T19:12:02.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.471+0000 7f9081600640 1 -- 172.21.15.27:0/917040763 wait complete. 2024-03-28T19:12:02.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.472+0000 7f9081600640 1 Processor -- start 2024-03-28T19:12:02.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.473+0000 7f9081600640 1 -- start start 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.473+0000 7f9081600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c0a61a0 0x7f907c14abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.473+0000 7f9081600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f907c150ae0 0x7f907c14b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.473+0000 7f9081600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f907c153a30 0x7f907c14ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.473+0000 7f9081600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c06e4d0 con 0x7f907c0a61a0 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.473+0000 7f9081600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c06e1f0 con 0x7f907c153a30 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.473+0000 7f9081600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c06e7b0 con 0x7f907c150ae0 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f907be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c0a61a0 0x7f907c14abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f907be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c0a61a0 0x7f907c14abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40132/0 (socket says 172.21.15.27:40132) 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f907be00640 1 -- 172.21.15.27:0/4135577064 learned_addr learned my addr 172.21.15.27:0/4135577064 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:02.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f9073e00640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f907c150ae0 0x7f907c14b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f9080c00640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f907c153a30 0x7f907c14ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f907be00640 1 -- 172.21.15.27:0/4135577064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f907c150ae0 msgr2=0x7f907c14b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f907be00640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f907c150ae0 0x7f907c14b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f907be00640 1 -- 172.21.15.27:0/4135577064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f907c153a30 msgr2=0x7f907c14ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f907be00640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f907c153a30 0x7f907c14ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.474+0000 7f907be00640 1 -- 172.21.15.27:0/4135577064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c0bdc90 con 0x7f907c0a61a0 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.475+0000 7f907be00640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c0a61a0 0x7f907c14abd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f906c000c00 tx=0x7f906c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.475+0000 7f9078c00640 1 -- 172.21.15.27:0/4135577064 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c032630 con 0x7f907c0a61a0 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.475+0000 7f9078c00640 1 -- 172.21.15.27:0/4135577064 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c0327f0 con 0x7f907c0a61a0 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.475+0000 7f9078c00640 1 -- 172.21.15.27:0/4135577064 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c032be0 con 0x7f907c0a61a0 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.475+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907c14b650 con 0x7f907c0a61a0 2024-03-28T19:12:02.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.475+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c0bc430 con 0x7f907c0a61a0 2024-03-28T19:12:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.477+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907c067dd0 con 0x7f907c0a61a0 2024-03-28T19:12:02.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.477+0000 7f9078c00640 1 -- 172.21.15.27:0/4135577064 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f906c04d3f0 con 0x7f907c0a61a0 2024-03-28T19:12:02.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.478+0000 7f9078c00640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f905806bbd0 0x7f905806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:02.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.478+0000 7f9078c00640 1 -- 172.21.15.27:0/4135577064 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f906c02f030 con 0x7f907c0a61a0 2024-03-28T19:12:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.479+0000 7f9073e00640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f905806bbd0 0x7f905806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:02.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.479+0000 7f9073e00640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f905806bbd0 0x7f905806e090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f907c1556d0 tx=0x7f9064007b50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:02.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.483+0000 7f9078c00640 1 -- 172.21.15.27:0/4135577064 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f906c040070 con 0x7f907c0a61a0 2024-03-28T19:12:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.737+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f907c0bcaf0 con 0x7f907c0a61a0 2024-03-28T19:12:02.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.738+0000 7f9078c00640 1 -- 172.21.15.27:0/4135577064 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f907c0bcaf0 con 0x7f907c0a61a0 2024-03-28T19:12:02.737 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:02.737 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:02.738 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f905806bbd0 msgr2=0x7f905806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f905806bbd0 0x7f905806e090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f907c1556d0 tx=0x7f9064007b50 comp rx=0 tx=0).stop 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c0a61a0 msgr2=0x7f907c14abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c0a61a0 0x7f907c14abd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f906c000c00 tx=0x7f906c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 shutdown_connections 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f905806bbd0 0x7f905806e090 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f907c153a30 0x7f907c14ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f907c150ae0 0x7f907c14b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 --2- 172.21.15.27:0/4135577064 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f907c0a61a0 0x7f907c14abd0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 >> 172.21.15.27:0/4135577064 conn(0x7f907c06b0c0 msgr2=0x7f907c0c47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 shutdown_connections 2024-03-28T19:12:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:02.741+0000 7f9081600640 1 -- 172.21.15.27:0/4135577064 wait complete. 2024-03-28T19:12:02.759 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:12:04.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:04.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.559+0000 7f2b78200640 1 Processor -- start 2024-03-28T19:12:04.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.560+0000 7f2b78200640 1 -- start start 2024-03-28T19:12:04.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.561+0000 7f2b78200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2b7014fef0 0x7f2b700c2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:04.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.561+0000 7f2b78200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700c78e0 con 0x7f2b700a7400 2024-03-28T19:12:04.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.561+0000 7f2b78200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700c3e80 con 0x7f2b7014fb10 2024-03-28T19:12:04.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.561+0000 7f2b78200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b70152560 con 0x7f2b7014fef0 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.561+0000 7f2b76200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2b700a7400 0x7f2b701529f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47476/0 (socket says 172.21.15.27:47476) 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.561+0000 7f2b76200640 1 -- 172.21.15.27:0/2095737321 learned_addr learned my addr 172.21.15.27:0/2095737321 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.561+0000 7f2b74e00640 1 --2- 172.21.15.27:0/2095737321 >> v2:172.21.15.27:3300/0 conn(0x7f2b7014fef0 0x7f2b700c2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.562+0000 7f2b6fe00640 1 -- 172.21.15.27:0/2095737321 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782899354 0 0) 0x7f2b700c78e0 con 0x7f2b700a7400 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.562+0000 7f2b6fe00640 1 -- 172.21.15.27:0/2095737321 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b58003520 con 0x7f2b700a7400 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.562+0000 7f2b6fe00640 1 -- 172.21.15.27:0/2095737321 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582791643 0 0) 0x7f2b700c3e80 con 0x7f2b7014fb10 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.562+0000 7f2b74e00640 1 -- 172.21.15.27:0/2095737321 >> v1:172.21.15.155:6789/0 conn(0x7f2b7014fb10 legacy=0x7f2b7006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.562+0000 7f2b74e00640 1 -- 172.21.15.27:0/2095737321 >> v1:172.21.15.27:6789/0 conn(0x7f2b700a7400 legacy=0x7f2b701529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.562+0000 7f2b74e00640 1 -- 172.21.15.27:0/2095737321 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700c4160 con 0x7f2b7014fef0 2024-03-28T19:12:04.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.564+0000 7f2b74e00640 1 --2- 172.21.15.27:0/2095737321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fef0 0x7f2b700c2650 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0097a0 tx=0x7f2b5c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2afb156ce8c30c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:04.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.564+0000 7f2b6fe00640 1 -- 172.21.15.27:0/2095737321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c034070 con 0x7f2b7014fef0 2024-03-28T19:12:04.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.564+0000 7f2b6fe00640 1 -- 172.21.15.27:0/2095737321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c030610 con 0x7f2b7014fef0 2024-03-28T19:12:04.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.564+0000 7f2b6fe00640 1 -- 172.21.15.27:0/2095737321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0309e0 con 0x7f2b7014fef0 2024-03-28T19:12:04.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.564+0000 7f2b78200640 1 -- 172.21.15.27:0/2095737321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fef0 msgr2=0x7f2b700c2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.564+0000 7f2b78200640 1 --2- 172.21.15.27:0/2095737321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fef0 0x7f2b700c2650 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0097a0 tx=0x7f2b5c02b960 comp rx=0 tx=0).stop 2024-03-28T19:12:04.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.565+0000 7f2b78200640 1 -- 172.21.15.27:0/2095737321 shutdown_connections 2024-03-28T19:12:04.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.565+0000 7f2b78200640 1 --2- 172.21.15.27:0/2095737321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fef0 0x7f2b700c2650 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.565+0000 7f2b78200640 1 -- 172.21.15.27:0/2095737321 >> 172.21.15.27:0/2095737321 conn(0x7f2b700c4750 msgr2=0x7f2b700c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:04.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.565+0000 7f2b78200640 1 -- 172.21.15.27:0/2095737321 shutdown_connections 2024-03-28T19:12:04.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.565+0000 7f2b78200640 1 -- 172.21.15.27:0/2095737321 wait complete. 2024-03-28T19:12:04.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.566+0000 7f2b78200640 1 Processor -- start 2024-03-28T19:12:04.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.567+0000 7f2b78200640 1 -- start start 2024-03-28T19:12:04.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.567+0000 7f2b78200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b700a7400 0x7f2b7009b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:04.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.567+0000 7f2b78200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fb10 0x7f2b7009de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:04.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.567+0000 7f2b78200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b7014fef0 0x7f2b701ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:04.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.567+0000 7f2b78200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7006c7e0 con 0x7f2b7014fb10 2024-03-28T19:12:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.567+0000 7f2b78200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700c7570 con 0x7f2b7014fef0 2024-03-28T19:12:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.567+0000 7f2b78200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7006c960 con 0x7f2b700a7400 2024-03-28T19:12:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b74e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fb10 0x7f2b7009de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b74e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fb10 0x7f2b7009de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40160/0 (socket says 172.21.15.27:40160) 2024-03-28T19:12:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b74e00640 1 -- 172.21.15.27:0/2344807911 learned_addr learned my addr 172.21.15.27:0/2344807911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b75800640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b700a7400 0x7f2b7009b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b76200640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b7014fef0 0x7f2b701ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:04.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b74e00640 1 -- 172.21.15.27:0/2344807911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b700a7400 msgr2=0x7f2b7009b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b74e00640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b700a7400 0x7f2b7009b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b74e00640 1 -- 172.21.15.27:0/2344807911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b7014fef0 msgr2=0x7f2b701ee5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b74e00640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b7014fef0 0x7f2b701ee5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.568+0000 7f2b74e00640 1 -- 172.21.15.27:0/2344807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b701edde0 con 0x7f2b7014fb10 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.569+0000 7f2b74e00640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fb10 0x7f2b7009de80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0098c0 tx=0x7f2b5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.569+0000 7f2b6d600640 1 -- 172.21.15.27:0/2344807911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c049070 con 0x7f2b7014fb10 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.569+0000 7f2b6d600640 1 -- 172.21.15.27:0/2344807911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c03a890 con 0x7f2b7014fb10 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.569+0000 7f2b6d600640 1 -- 172.21.15.27:0/2344807911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c034040 con 0x7f2b7014fb10 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.569+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b701eb390 con 0x7f2b7014fb10 2024-03-28T19:12:04.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.569+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b701eeae0 con 0x7f2b7014fb10 2024-03-28T19:12:04.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.570+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7004f400 con 0x7f2b7014fb10 2024-03-28T19:12:04.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.571+0000 7f2b6d600640 1 -- 172.21.15.27:0/2344807911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2b5c04c070 con 0x7f2b7014fb10 2024-03-28T19:12:04.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.572+0000 7f2b6d600640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b5006bbd0 0x7f2b5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:04.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.572+0000 7f2b6d600640 1 -- 172.21.15.27:0/2344807911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f2b5c0ad200 con 0x7f2b7014fb10 2024-03-28T19:12:04.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.573+0000 7f2b75800640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b5006bbd0 0x7f2b5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:04.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.573+0000 7f2b75800640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b5006bbd0 0x7f2b5006e090 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000f60 tx=0x7f2b64009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:04.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.577+0000 7f2b6d600640 1 -- 172.21.15.27:0/2344807911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2b5c03f070 con 0x7f2b7014fb10 2024-03-28T19:12:04.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:04.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.817+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b700a6d00 con 0x7f2b7014fb10 2024-03-28T19:12:04.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.818+0000 7f2b6d600640 1 -- 172.21.15.27:0/2344807911 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f2b5c07ad50 con 0x7f2b7014fb10 2024-03-28T19:12:04.818 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:04.818 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:04.818 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:04.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b5006bbd0 msgr2=0x7f2b5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b5006bbd0 0x7f2b5006e090 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000f60 tx=0x7f2b64009040 comp rx=0 tx=0).stop 2024-03-28T19:12:04.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fb10 msgr2=0x7f2b7009de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fb10 0x7f2b7009de80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0098c0 tx=0x7f2b5c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 shutdown_connections 2024-03-28T19:12:04.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b5006bbd0 0x7f2b5006e090 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b7014fef0 0x7f2b701ee5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b7014fb10 0x7f2b7009de80 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 --2- 172.21.15.27:0/2344807911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b700a7400 0x7f2b7009b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 >> 172.21.15.27:0/2344807911 conn(0x7f2b700c4750 msgr2=0x7f2b7006f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:04.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 shutdown_connections 2024-03-28T19:12:04.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.821+0000 7f2b78200640 1 -- 172.21.15.27:0/2344807911 wait complete. 2024-03-28T19:12:04.840 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:12:04.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.993+0000 7f2303600640 1 Processor -- start 2024-03-28T19:12:04.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.993+0000 7f2303600640 1 -- start start 2024-03-28T19:12:04.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.994+0000 7f2303600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f22fc150c70 0x7f22fc0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:04.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.994+0000 7f2303600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0c3af0 con 0x7f22fc0c3690 2024-03-28T19:12:04.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.994+0000 7f2303600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc154020 con 0x7f22fc1534c0 2024-03-28T19:12:04.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.994+0000 7f2303600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc0ab4e0 con 0x7f22fc150c70 2024-03-28T19:12:04.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.995+0000 7f22fbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f22fc0c3690 0x7f22fc0c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47478/0 (socket says 172.21.15.27:47478) 2024-03-28T19:12:04.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.995+0000 7f22fbe00640 1 -- 172.21.15.27:0/668520117 learned_addr learned my addr 172.21.15.27:0/668520117 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:04.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.995+0000 7f2300c00640 1 --2- 172.21.15.27:0/668520117 >> v2:172.21.15.155:3300/0 conn(0x7f22fc150c70 0x7f22fc0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.995+0000 7f22fb400640 1 -- 172.21.15.27:0/668520117 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830928259 0 0) 0x7f22fc0c3af0 con 0x7f22fc0c3690 2024-03-28T19:12:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.995+0000 7f22fb400640 1 -- 172.21.15.27:0/668520117 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e0003520 con 0x7f22fc0c3690 2024-03-28T19:12:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.995+0000 7f22fb400640 1 -- 172.21.15.27:0/668520117 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538184634 0 0) 0x7f22fc154020 con 0x7f22fc1534c0 2024-03-28T19:12:04.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.995+0000 7f22fb400640 1 -- 172.21.15.27:0/668520117 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22fc0c3af0 con 0x7f22fc1534c0 2024-03-28T19:12:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.996+0000 7f2300c00640 1 -- 172.21.15.27:0/668520117 >> v1:172.21.15.155:6789/0 conn(0x7f22fc1534c0 legacy=0x7f22fc153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.996+0000 7f2300c00640 1 -- 172.21.15.27:0/668520117 >> v1:172.21.15.27:6789/0 conn(0x7f22fc0c3690 legacy=0x7f22fc0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.996+0000 7f2300c00640 1 -- 172.21.15.27:0/668520117 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0c3d90 con 0x7f22fc150c70 2024-03-28T19:12:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.997+0000 7f2300c00640 1 --2- 172.21.15.27:0/668520117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc150c70 0x7f22fc0a9cb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22fc1541f0 tx=0x7f22ec02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31b1f20e6c4e19f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:04.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f22fb400640 1 -- 172.21.15.27:0/668520117 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec035020 con 0x7f22fc150c70 2024-03-28T19:12:04.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f22fb400640 1 -- 172.21.15.27:0/668520117 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec031830 con 0x7f22fc150c70 2024-03-28T19:12:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f22fb400640 1 -- 172.21.15.27:0/668520117 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec031c00 con 0x7f22fc150c70 2024-03-28T19:12:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f2303600640 1 -- 172.21.15.27:0/668520117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc150c70 msgr2=0x7f22fc0a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f2303600640 1 --2- 172.21.15.27:0/668520117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc150c70 0x7f22fc0a9cb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22fc1541f0 tx=0x7f22ec02b8d0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f2303600640 1 -- 172.21.15.27:0/668520117 shutdown_connections 2024-03-28T19:12:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f2303600640 1 --2- 172.21.15.27:0/668520117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc150c70 0x7f22fc0a9cb0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f2303600640 1 -- 172.21.15.27:0/668520117 >> 172.21.15.27:0/668520117 conn(0x7f22fc06b9d0 msgr2=0x7f22fc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f2303600640 1 -- 172.21.15.27:0/668520117 shutdown_connections 2024-03-28T19:12:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:04.998+0000 7f2303600640 1 -- 172.21.15.27:0/668520117 wait complete. 2024-03-28T19:12:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.000+0000 7f2303600640 1 Processor -- start 2024-03-28T19:12:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.000+0000 7f2303600640 1 -- start start 2024-03-28T19:12:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f2303600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc0c3690 0x7f22fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f2303600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc150c70 0x7f22fc1eaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f2303600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22fc1534c0 0x7f22fc1eb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f2303600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0ab9f0 con 0x7f22fc150c70 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f2303600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0ac1f0 con 0x7f22fc0c3690 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f2303600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0abcd0 con 0x7f22fc1534c0 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f22fbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc150c70 0x7f22fc1eaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f2300c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc0c3690 0x7f22fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.001+0000 7f22fbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc150c70 0x7f22fc1eaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40172/0 (socket says 172.21.15.27:40172) 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.002+0000 7f22fbe00640 1 -- 172.21.15.27:0/188952297 learned_addr learned my addr 172.21.15.27:0/188952297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.002+0000 7f2301600640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22fc1534c0 0x7f22fc1eb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.002+0000 7f22fbe00640 1 -- 172.21.15.27:0/188952297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22fc1534c0 msgr2=0x7f22fc1eb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.002+0000 7f22fbe00640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22fc1534c0 0x7f22fc1eb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.002+0000 7f22fbe00640 1 -- 172.21.15.27:0/188952297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc0c3690 msgr2=0x7f22fc06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.002+0000 7f22fbe00640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc0c3690 0x7f22fc06d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.002+0000 7f22fbe00640 1 -- 172.21.15.27:0/188952297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc157300 con 0x7f22fc150c70 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.002+0000 7f22fbe00640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc150c70 0x7f22fc1eaea0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f22e800c530 tx=0x7f22e800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.003+0000 7f22f8c00640 1 -- 172.21.15.27:0/188952297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8017070 con 0x7f22fc150c70 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.003+0000 7f22f8c00640 1 -- 172.21.15.27:0/188952297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e8005ce0 con 0x7f22fc150c70 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.003+0000 7f22f8c00640 1 -- 172.21.15.27:0/188952297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8007270 con 0x7f22fc150c70 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.003+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc1575e0 con 0x7f22fc150c70 2024-03-28T19:12:05.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.003+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc1ee820 con 0x7f22fc150c70 2024-03-28T19:12:05.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.004+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc04f400 con 0x7f22fc150c70 2024-03-28T19:12:05.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.005+0000 7f22f8c00640 1 -- 172.21.15.27:0/188952297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f22e801a070 con 0x7f22fc150c70 2024-03-28T19:12:05.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.006+0000 7f22f8c00640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f22dc06bbd0 0x7f22dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:05.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.006+0000 7f22f8c00640 1 -- 172.21.15.27:0/188952297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f22e808a810 con 0x7f22fc150c70 2024-03-28T19:12:05.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.006+0000 7f2300c00640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f22dc06bbd0 0x7f22dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:05.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.007+0000 7f2300c00640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f22dc06bbd0 0x7f22dc06e090 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f22ec036000 tx=0x7f22ec008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.010+0000 7f22f8c00640 1 -- 172.21.15.27:0/188952297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f22e80581e0 con 0x7f22fc150c70 2024-03-28T19:12:05.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.254+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22fc0c5f00 con 0x7f22fc150c70 2024-03-28T19:12:05.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.255+0000 7f22f8c00640 1 -- 172.21.15.27:0/188952297 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f22e805c280 con 0x7f22fc150c70 2024-03-28T19:12:05.255 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:05.255 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:05.255 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:05.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f22dc06bbd0 msgr2=0x7f22dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:05.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f22dc06bbd0 0x7f22dc06e090 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f22ec036000 tx=0x7f22ec008030 comp rx=0 tx=0).stop 2024-03-28T19:12:05.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc150c70 msgr2=0x7f22fc1eaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:05.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc150c70 0x7f22fc1eaea0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f22e800c530 tx=0x7f22e800c9f0 comp rx=0 tx=0).stop 2024-03-28T19:12:05.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 shutdown_connections 2024-03-28T19:12:05.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f22dc06bbd0 0x7f22dc06e090 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:05.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22fc1534c0 0x7f22fc1eb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:05.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22fc150c70 0x7f22fc1eaea0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:05.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 --2- 172.21.15.27:0/188952297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22fc0c3690 0x7f22fc06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:05.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 >> 172.21.15.27:0/188952297 conn(0x7f22fc06b9d0 msgr2=0x7f22fc0c2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:05.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 shutdown_connections 2024-03-28T19:12:05.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:05.258+0000 7f2303600640 1 -- 172.21.15.27:0/188952297 wait complete. 2024-03-28T19:12:05.280 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:12:06.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:07.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.077+0000 7f9874a00640 1 Processor -- start 2024-03-28T19:12:07.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.077+0000 7f9874a00640 1 -- start start 2024-03-28T19:12:07.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.078+0000 7f9874a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987014be30 0x7f9870158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.078+0000 7f9874a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9870155df0 0x7f987006fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.078+0000 7f9874a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9870159010 con 0x7f987006ffb0 2024-03-28T19:12:07.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.078+0000 7f9874a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9870155960 con 0x7f9870155df0 2024-03-28T19:12:07.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.078+0000 7f9874a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98700c6bb0 con 0x7f987014be30 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.079+0000 7f986e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987014be30 0x7f9870158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.079+0000 7f986e800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987014be30 0x7f9870158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:48534/0 (socket says 172.21.15.27:48534) 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.079+0000 7f986e800640 1 -- 172.21.15.27:0/444881451 learned_addr learned my addr 172.21.15.27:0/444881451 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.079+0000 7f986de00640 1 --2- 172.21.15.27:0/444881451 >> v2:172.21.15.155:3300/0 conn(0x7f9870155df0 0x7f987006fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.080+0000 7f986e800640 1 -- 172.21.15.27:0/444881451 >> v2:172.21.15.155:3300/0 conn(0x7f9870155df0 msgr2=0x7f987006fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.080+0000 7f986e800640 1 --2- 172.21.15.27:0/444881451 >> v2:172.21.15.155:3300/0 conn(0x7f9870155df0 0x7f987006fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.080+0000 7f986e800640 1 -- 172.21.15.27:0/444881451 >> v1:172.21.15.155:6789/0 conn(0x7f987006ffb0 legacy=0x7f98700a7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.080+0000 7f986e800640 1 -- 172.21.15.27:0/444881451 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700c6e50 con 0x7f987014be30 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.080+0000 7f986de00640 1 --2- 172.21.15.27:0/444881451 >> v2:172.21.15.155:3300/0 conn(0x7f9870155df0 0x7f987006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:12:07.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.080+0000 7f986d400640 1 -- 172.21.15.27:0/444881451 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864272710 0 0) 0x7f9870159010 con 0x7f987006ffb0 2024-03-28T19:12:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.081+0000 7f986e800640 1 --2- 172.21.15.27:0/444881451 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987014be30 0x7f9870158ad0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9860009870 tx=0x7f986002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a236a106d84dfcc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.081+0000 7f986d400640 1 -- 172.21.15.27:0/444881451 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9860034070 con 0x7f987014be30 2024-03-28T19:12:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.081+0000 7f986d400640 1 -- 172.21.15.27:0/444881451 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9860030840 con 0x7f987014be30 2024-03-28T19:12:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.081+0000 7f986d400640 1 -- 172.21.15.27:0/444881451 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9860030c10 con 0x7f987014be30 2024-03-28T19:12:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.082+0000 7f9874a00640 1 -- 172.21.15.27:0/444881451 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987014be30 msgr2=0x7f9870158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.082+0000 7f9874a00640 1 --2- 172.21.15.27:0/444881451 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987014be30 0x7f9870158ad0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9860009870 tx=0x7f986002b980 comp rx=0 tx=0).stop 2024-03-28T19:12:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.082+0000 7f9874a00640 1 -- 172.21.15.27:0/444881451 shutdown_connections 2024-03-28T19:12:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.082+0000 7f9874a00640 1 --2- 172.21.15.27:0/444881451 >> v2:172.21.15.155:3300/0 conn(0x7f9870155df0 0x7f987006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.082+0000 7f9874a00640 1 --2- 172.21.15.27:0/444881451 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987014be30 0x7f9870158ad0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.082+0000 7f9874a00640 1 -- 172.21.15.27:0/444881451 >> 172.21.15.27:0/444881451 conn(0x7f9870072bd0 msgr2=0x7f9870072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:07.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.082+0000 7f9874a00640 1 -- 172.21.15.27:0/444881451 shutdown_connections 2024-03-28T19:12:07.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.082+0000 7f9874a00640 1 -- 172.21.15.27:0/444881451 wait complete. 2024-03-28T19:12:07.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.084+0000 7f9874a00640 1 Processor -- start 2024-03-28T19:12:07.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.084+0000 7f9874a00640 1 -- start start 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f9874a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987006ffb0 0x7f98701e8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f9874a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987014be30 0x7f98701493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f9874a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870155df0 0x7f9870149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f9874a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700a8420 con 0x7f987014be30 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f9874a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700c6400 con 0x7f9870155df0 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f9874a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9870149e40 con 0x7f987006ffb0 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987014be30 0x7f98701493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987014be30 0x7f98701493c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47386/0 (socket says 172.21.15.27:47386) 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986de00640 1 -- 172.21.15.27:0/2345954283 learned_addr learned my addr 172.21.15.27:0/2345954283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986e800640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987006ffb0 0x7f98701e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986f200640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870155df0 0x7f9870149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986de00640 1 -- 172.21.15.27:0/2345954283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987006ffb0 msgr2=0x7f98701e8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986de00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987006ffb0 0x7f98701e8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986de00640 1 -- 172.21.15.27:0/2345954283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870155df0 msgr2=0x7f9870149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986de00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870155df0 0x7f9870149900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.085+0000 7f986de00640 1 -- 172.21.15.27:0/2345954283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987014a0b0 con 0x7f987014be30 2024-03-28T19:12:07.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.086+0000 7f986de00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987014be30 0x7f98701493c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9858009fc0 tx=0x7f985800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:07.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.086+0000 7f9866a00640 1 -- 172.21.15.27:0/2345954283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985801a070 con 0x7f987014be30 2024-03-28T19:12:07.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.086+0000 7f9866a00640 1 -- 172.21.15.27:0/2345954283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9858003bb0 con 0x7f987014be30 2024-03-28T19:12:07.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.086+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98701eeaa0 con 0x7f987014be30 2024-03-28T19:12:07.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.086+0000 7f9866a00640 1 -- 172.21.15.27:0/2345954283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858004030 con 0x7f987014be30 2024-03-28T19:12:07.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.086+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9870154e60 con 0x7f987014be30 2024-03-28T19:12:07.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.087+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98701ee8b0 con 0x7f987014be30 2024-03-28T19:12:07.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.088+0000 7f9866a00640 1 -- 172.21.15.27:0/2345954283 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f985801d070 con 0x7f987014be30 2024-03-28T19:12:07.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.089+0000 7f9866a00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f984c06bbd0 0x7f984c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.089+0000 7f9866a00640 1 -- 172.21.15.27:0/2345954283 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f985808b8a0 con 0x7f987014be30 2024-03-28T19:12:07.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.089+0000 7f986e800640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f984c06bbd0 0x7f984c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.090+0000 7f986e800640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f984c06bbd0 0x7f984c06e090 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9860036000 tx=0x7f9860008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:07.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.094+0000 7f9866a00640 1 -- 172.21.15.27:0/2345954283 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9858010070 con 0x7f987014be30 2024-03-28T19:12:07.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:07.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.337+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9870069640 con 0x7f987014be30 2024-03-28T19:12:07.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.337+0000 7f9866a00640 1 -- 172.21.15.27:0/2345954283 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f98580591e0 con 0x7f987014be30 2024-03-28T19:12:07.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:07.337 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:07.337 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f984c06bbd0 msgr2=0x7f984c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f984c06bbd0 0x7f984c06e090 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9860036000 tx=0x7f9860008030 comp rx=0 tx=0).stop 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987014be30 msgr2=0x7f98701493c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987014be30 0x7f98701493c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9858009fc0 tx=0x7f985800d560 comp rx=0 tx=0).stop 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 shutdown_connections 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f984c06bbd0 0x7f984c06e090 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870155df0 0x7f9870149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987014be30 0x7f98701493c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 --2- 172.21.15.27:0/2345954283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f987006ffb0 0x7f98701e8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.340+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 >> 172.21.15.27:0/2345954283 conn(0x7f9870072bd0 msgr2=0x7f98700c1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.341+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 shutdown_connections 2024-03-28T19:12:07.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.341+0000 7f9874a00640 1 -- 172.21.15.27:0/2345954283 wait complete. 2024-03-28T19:12:07.362 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:12:07.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.508+0000 7f7e45e00640 1 Processor -- start 2024-03-28T19:12:07.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.509+0000 7f7e45e00640 1 -- start start 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.510+0000 7f7e45e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7e400c8ed0 0x7f7e400c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.510+0000 7f7e45e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400be5d0 0x7f7e40155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.510+0000 7f7e45e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e40156230 con 0x7f7e400c58c0 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.510+0000 7f7e45e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e40158bd0 con 0x7f7e400c8ed0 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.510+0000 7f7e45e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e40158eb0 con 0x7f7e400be5d0 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e44a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e400c58c0 0x7f7e401555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50756/0 (socket says 172.21.15.27:50756) 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e44a00640 1 -- 172.21.15.27:0/1495684550 learned_addr learned my addr 172.21.15.27:0/1495684550 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e45400640 1 --2- 172.21.15.27:0/1495684550 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400be5d0 0x7f7e40155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e37e00640 1 --2- 172.21.15.27:0/1495684550 >> v2:172.21.15.155:3300/0 conn(0x7f7e400c8ed0 0x7f7e400c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e3fe00640 1 -- 172.21.15.27:0/1495684550 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449774900 0 0) 0x7f7e40156230 con 0x7f7e400c58c0 2024-03-28T19:12:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e3fe00640 1 -- 172.21.15.27:0/1495684550 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e24003520 con 0x7f7e400c58c0 2024-03-28T19:12:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e45400640 1 -- 172.21.15.27:0/1495684550 >> v2:172.21.15.155:3300/0 conn(0x7f7e400c8ed0 msgr2=0x7f7e400c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:07.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e45400640 1 --2- 172.21.15.27:0/1495684550 >> v2:172.21.15.155:3300/0 conn(0x7f7e400c8ed0 0x7f7e400c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e45400640 1 -- 172.21.15.27:0/1495684550 >> v1:172.21.15.27:6789/0 conn(0x7f7e400c58c0 legacy=0x7f7e401555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e45400640 1 -- 172.21.15.27:0/1495684550 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40159190 con 0x7f7e400be5d0 2024-03-28T19:12:07.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.511+0000 7f7e37e00640 1 --2- 172.21.15.27:0/1495684550 >> v2:172.21.15.155:3300/0 conn(0x7f7e400c8ed0 0x7f7e400c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:12:07.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.513+0000 7f7e45400640 1 --2- 172.21.15.27:0/1495684550 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400be5d0 0x7f7e40155cf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009d90 tx=0x7f7e3802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93eddbd57a422f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:07.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.513+0000 7f7e3fe00640 1 -- 172.21.15.27:0/1495684550 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38039070 con 0x7f7e400be5d0 2024-03-28T19:12:07.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.513+0000 7f7e3fe00640 1 -- 172.21.15.27:0/1495684550 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3802cbd0 con 0x7f7e400be5d0 2024-03-28T19:12:07.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.513+0000 7f7e3fe00640 1 -- 172.21.15.27:0/1495684550 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38034440 con 0x7f7e400be5d0 2024-03-28T19:12:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.514+0000 7f7e45e00640 1 -- 172.21.15.27:0/1495684550 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400be5d0 msgr2=0x7f7e40155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.514+0000 7f7e45e00640 1 --2- 172.21.15.27:0/1495684550 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400be5d0 0x7f7e40155cf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009d90 tx=0x7f7e3802be20 comp rx=0 tx=0).stop 2024-03-28T19:12:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.514+0000 7f7e45e00640 1 -- 172.21.15.27:0/1495684550 shutdown_connections 2024-03-28T19:12:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.514+0000 7f7e45e00640 1 --2- 172.21.15.27:0/1495684550 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400be5d0 0x7f7e40155cf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.514+0000 7f7e45e00640 1 --2- 172.21.15.27:0/1495684550 >> v2:172.21.15.155:3300/0 conn(0x7f7e400c8ed0 0x7f7e400c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.514+0000 7f7e45e00640 1 -- 172.21.15.27:0/1495684550 >> 172.21.15.27:0/1495684550 conn(0x7f7e4006b6f0 msgr2=0x7f7e400acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.514+0000 7f7e45e00640 1 -- 172.21.15.27:0/1495684550 shutdown_connections 2024-03-28T19:12:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.514+0000 7f7e45e00640 1 -- 172.21.15.27:0/1495684550 wait complete. 2024-03-28T19:12:07.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.516+0000 7f7e45e00640 1 Processor -- start 2024-03-28T19:12:07.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.516+0000 7f7e45e00640 1 -- start start 2024-03-28T19:12:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.517+0000 7f7e45e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e400be5d0 0x7f7e40153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.517+0000 7f7e45e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400c58c0 0x7f7e400c2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.517+0000 7f7e45e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400c8ed0 0x7f7e400c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.517+0000 7f7e45e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e400b0820 con 0x7f7e400be5d0 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.517+0000 7f7e45e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e40159750 con 0x7f7e400c8ed0 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.517+0000 7f7e45e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e400b09a0 con 0x7f7e400c58c0 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e44a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e400be5d0 0x7f7e40153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e37e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400c58c0 0x7f7e400c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e44a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e400be5d0 0x7f7e40153470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47392/0 (socket says 172.21.15.27:47392) 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e44a00640 1 -- 172.21.15.27:0/3755583969 learned_addr learned my addr 172.21.15.27:0/3755583969 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e45400640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400c8ed0 0x7f7e400c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e44a00640 1 -- 172.21.15.27:0/3755583969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400c58c0 msgr2=0x7f7e400c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e44a00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400c58c0 0x7f7e400c2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e44a00640 1 -- 172.21.15.27:0/3755583969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400c8ed0 msgr2=0x7f7e400c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e44a00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400c8ed0 0x7f7e400c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.518+0000 7f7e44a00640 1 -- 172.21.15.27:0/3755583969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e401539b0 con 0x7f7e400be5d0 2024-03-28T19:12:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.519+0000 7f7e44a00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e400be5d0 0x7f7e40153470 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000c390 tx=0x7f7e3000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.519+0000 7f7e3d600640 1 -- 172.21.15.27:0/3755583969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30017070 con 0x7f7e400be5d0 2024-03-28T19:12:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.519+0000 7f7e3d600640 1 -- 172.21.15.27:0/3755583969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e30005ce0 con 0x7f7e400be5d0 2024-03-28T19:12:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.519+0000 7f7e3d600640 1 -- 172.21.15.27:0/3755583969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e300072f0 con 0x7f7e400be5d0 2024-03-28T19:12:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.519+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e401521a0 con 0x7f7e400be5d0 2024-03-28T19:12:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.519+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e400ac9a0 con 0x7f7e400be5d0 2024-03-28T19:12:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.521+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4006c810 con 0x7f7e400be5d0 2024-03-28T19:12:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.522+0000 7f7e3d600640 1 -- 172.21.15.27:0/3755583969 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7e3001a070 con 0x7f7e400be5d0 2024-03-28T19:12:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.523+0000 7f7e3d600640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e1c06bc00 0x7f7e1c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.523+0000 7f7e3d600640 1 -- 172.21.15.27:0/3755583969 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f7e3008a870 con 0x7f7e400be5d0 2024-03-28T19:12:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.523+0000 7f7e37e00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e1c06bc00 0x7f7e1c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.524+0000 7f7e37e00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e1c06bc00 0x7f7e1c06e0c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009560 tx=0x7f7e28009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.527+0000 7f7e3d600640 1 -- 172.21.15.27:0/3755583969 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7e4006c810 con 0x7f7e400be5d0 2024-03-28T19:12:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.770+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e40069640 con 0x7f7e400be5d0 2024-03-28T19:12:07.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.772+0000 7f7e3d600640 1 -- 172.21.15.27:0/3755583969 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f7e30058240 con 0x7f7e400be5d0 2024-03-28T19:12:07.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:07.771 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:07.771 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e1c06bc00 msgr2=0x7f7e1c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e1c06bc00 0x7f7e1c06e0c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009560 tx=0x7f7e28009190 comp rx=0 tx=0).stop 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e400be5d0 msgr2=0x7f7e40153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e400be5d0 0x7f7e40153470 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000c390 tx=0x7f7e3000c850 comp rx=0 tx=0).stop 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 shutdown_connections 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e1c06bc00 0x7f7e1c06e0c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e400c8ed0 0x7f7e400c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e400c58c0 0x7f7e400c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.774+0000 7f7e45e00640 1 --2- 172.21.15.27:0/3755583969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e400be5d0 0x7f7e40153470 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:07.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.775+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 >> 172.21.15.27:0/3755583969 conn(0x7f7e4006b6f0 msgr2=0x7f7e401583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.775+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 shutdown_connections 2024-03-28T19:12:07.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:07.775+0000 7f7e45e00640 1 -- 172.21.15.27:0/3755583969 wait complete. 2024-03-28T19:12:07.794 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:12:09.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:09.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.600+0000 7f8799200640 1 Processor -- start 2024-03-28T19:12:09.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.600+0000 7f8799200640 1 -- start start 2024-03-28T19:12:09.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8799200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f87940c3e70 0x7f87941534b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:09.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8799200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 0x7f87940aba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:09.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8799200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87940abf90 0x7f87940c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:09.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8799200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8794072ec0 con 0x7f87940abf90 2024-03-28T19:12:09.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8799200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87940722b0 con 0x7f87940c3e70 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8799200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8794072590 con 0x7f87940c15a0 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8792800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87940abf90 0x7f87940c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8792800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87940abf90 0x7f87940c1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47394/0 (socket says 172.21.15.27:47394) 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.602+0000 7f8792800640 1 -- 172.21.15.27:0/2723324080 learned_addr learned my addr 172.21.15.27:0/2723324080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.603+0000 7f8791e00640 1 --2- 172.21.15.27:0/2723324080 >> v2:172.21.15.155:3300/0 conn(0x7f87940c3e70 0x7f87941534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.603+0000 7f8791400640 1 --2- 172.21.15.27:0/2723324080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 0x7f87940aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.603+0000 7f8792800640 1 -- 172.21.15.27:0/2723324080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 msgr2=0x7f87940aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.603+0000 7f8792800640 1 --2- 172.21.15.27:0/2723324080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 0x7f87940aba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.603+0000 7f8792800640 1 -- 172.21.15.27:0/2723324080 >> v2:172.21.15.155:3300/0 conn(0x7f87940c3e70 msgr2=0x7f87941534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.603+0000 7f8792800640 1 --2- 172.21.15.27:0/2723324080 >> v2:172.21.15.155:3300/0 conn(0x7f87940c3e70 0x7f87941534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.603+0000 7f8792800640 1 -- 172.21.15.27:0/2723324080 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8794072870 con 0x7f87940abf90 2024-03-28T19:12:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.604+0000 7f8792800640 1 --2- 172.21.15.27:0/2723324080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940abf90 0x7f87940c1db0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8794067ae0 tx=0x7f878c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a578c64922636306 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.605+0000 7f8790a00640 1 -- 172.21.15.27:0/2723324080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c02ca70 con 0x7f87940abf90 2024-03-28T19:12:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.605+0000 7f8790a00640 1 -- 172.21.15.27:0/2723324080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c02cc10 con 0x7f87940abf90 2024-03-28T19:12:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.605+0000 7f8790a00640 1 -- 172.21.15.27:0/2723324080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c0343f0 con 0x7f87940abf90 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.605+0000 7f8799200640 1 -- 172.21.15.27:0/2723324080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940abf90 msgr2=0x7f87940c1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.605+0000 7f8799200640 1 --2- 172.21.15.27:0/2723324080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940abf90 0x7f87940c1db0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8794067ae0 tx=0x7f878c02bed0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.606+0000 7f8799200640 1 -- 172.21.15.27:0/2723324080 shutdown_connections 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.606+0000 7f8799200640 1 --2- 172.21.15.27:0/2723324080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940abf90 0x7f87940c1db0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.606+0000 7f8799200640 1 --2- 172.21.15.27:0/2723324080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 0x7f87940aba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.606+0000 7f8799200640 1 --2- 172.21.15.27:0/2723324080 >> v2:172.21.15.155:3300/0 conn(0x7f87940c3e70 0x7f87941534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.606+0000 7f8799200640 1 -- 172.21.15.27:0/2723324080 >> 172.21.15.27:0/2723324080 conn(0x7f879406b0c0 msgr2=0x7f879406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.606+0000 7f8799200640 1 -- 172.21.15.27:0/2723324080 shutdown_connections 2024-03-28T19:12:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.606+0000 7f8799200640 1 -- 172.21.15.27:0/2723324080 wait complete. 2024-03-28T19:12:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.607+0000 7f8799200640 1 Processor -- start 2024-03-28T19:12:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.608+0000 7f8799200640 1 -- start start 2024-03-28T19:12:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.608+0000 7f8799200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87940abf90 0x7f87941530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.608+0000 7f8799200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 0x7f87940bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.608+0000 7f8799200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940c3e70 0x7f8794144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.608+0000 7f8799200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87940ac420 con 0x7f87940c3e70 2024-03-28T19:12:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.608+0000 7f8799200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8794071ed0 con 0x7f87940abf90 2024-03-28T19:12:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.608+0000 7f8799200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87941535e0 con 0x7f87940c15a0 2024-03-28T19:12:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8792800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940c3e70 0x7f8794144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8792800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940c3e70 0x7f8794144530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47396/0 (socket says 172.21.15.27:47396) 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8792800640 1 -- 172.21.15.27:0/2859198254 learned_addr learned my addr 172.21.15.27:0/2859198254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8791400640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 0x7f87940bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8791e00640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87940abf90 0x7f87941530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8792800640 1 -- 172.21.15.27:0/2859198254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 msgr2=0x7f87940bd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8792800640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 0x7f87940bd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8792800640 1 -- 172.21.15.27:0/2859198254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87940abf90 msgr2=0x7f87941530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8792800640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87940abf90 0x7f87941530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.609+0000 7f8792800640 1 -- 172.21.15.27:0/2859198254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8794153810 con 0x7f87940c3e70 2024-03-28T19:12:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.610+0000 7f8792800640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940c3e70 0x7f8794144530 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f878c009f40 tx=0x7f878c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.610+0000 7f878a000640 1 -- 172.21.15.27:0/2859198254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c043020 con 0x7f87940c3e70 2024-03-28T19:12:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.610+0000 7f878a000640 1 -- 172.21.15.27:0/2859198254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c03e450 con 0x7f87940c3e70 2024-03-28T19:12:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.610+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87941ea3a0 con 0x7f87940c3e70 2024-03-28T19:12:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.610+0000 7f878a000640 1 -- 172.21.15.27:0/2859198254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c009510 con 0x7f87940c3e70 2024-03-28T19:12:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.610+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87941ea960 con 0x7f87940c3e70 2024-03-28T19:12:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.611+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879404f400 con 0x7f87940c3e70 2024-03-28T19:12:09.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.613+0000 7f878a000640 1 -- 172.21.15.27:0/2859198254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f878c046020 con 0x7f87940c3e70 2024-03-28T19:12:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.613+0000 7f878a000640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f877406bbd0 0x7f877406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.614+0000 7f878a000640 1 -- 172.21.15.27:0/2859198254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f878c0adab0 con 0x7f87940c3e70 2024-03-28T19:12:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.614+0000 7f8791e00640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f877406bbd0 0x7f877406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.614+0000 7f8791e00640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f877406bbd0 0x7f877406e090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f878400df40 tx=0x7f878400f040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.618+0000 7f878a000640 1 -- 172.21.15.27:0/2859198254 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f878c04d030 con 0x7f87940c3e70 2024-03-28T19:12:09.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:09.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.876+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8794154680 con 0x7f87940c3e70 2024-03-28T19:12:09.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.877+0000 7f878a000640 1 -- 172.21.15.27:0/2859198254 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f878c07b480 con 0x7f87940c3e70 2024-03-28T19:12:09.876 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:09.876 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:09.876 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:09.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f877406bbd0 msgr2=0x7f877406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:09.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f877406bbd0 0x7f877406e090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f878400df40 tx=0x7f878400f040 comp rx=0 tx=0).stop 2024-03-28T19:12:09.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940c3e70 msgr2=0x7f8794144530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:09.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940c3e70 0x7f8794144530 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f878c009f40 tx=0x7f878c004a40 comp rx=0 tx=0).stop 2024-03-28T19:12:09.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 shutdown_connections 2024-03-28T19:12:09.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f877406bbd0 0x7f877406e090 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87940c3e70 0x7f8794144530 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87940c15a0 0x7f87940bd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 --2- 172.21.15.27:0/2859198254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87940abf90 0x7f87941530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:09.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 >> 172.21.15.27:0/2859198254 conn(0x7f879406b0c0 msgr2=0x7f8794149e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:09.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.880+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 shutdown_connections 2024-03-28T19:12:09.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:09.881+0000 7f8799200640 1 -- 172.21.15.27:0/2859198254 wait complete. 2024-03-28T19:12:09.900 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:12:10.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.024+0000 7fa22fe00640 1 Processor -- start 2024-03-28T19:12:10.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.024+0000 7fa22fe00640 1 -- start start 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.025+0000 7fa22fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa228150c70 0x7fa2280a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.025+0000 7fa22fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2280c3690 0x7fa2280c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.025+0000 7fa22fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c6660 0x7fa2281534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.025+0000 7fa22fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2280ab4e0 con 0x7fa2280c3690 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.025+0000 7fa22fe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2280ac1f0 con 0x7fa228150c70 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.025+0000 7fa22fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280c3af0 con 0x7fa2280c6660 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2280c3690 0x7fa2280c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2280c3690 0x7fa2280c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47406/0 (socket says 172.21.15.27:47406) 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22ca00640 1 -- 172.21.15.27:0/2912620648 learned_addr learned my addr 172.21.15.27:0/2912620648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:10.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22d400640 1 --2- 172.21.15.27:0/2912620648 >> v2:172.21.15.155:3300/0 conn(0x7fa228150c70 0x7fa2280a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22de00640 1 --2- 172.21.15.27:0/2912620648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c6660 0x7fa2281534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22ca00640 1 -- 172.21.15.27:0/2912620648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c6660 msgr2=0x7fa2281534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22ca00640 1 --2- 172.21.15.27:0/2912620648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c6660 0x7fa2281534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22ca00640 1 -- 172.21.15.27:0/2912620648 >> v2:172.21.15.155:3300/0 conn(0x7fa228150c70 msgr2=0x7fa2280a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:10.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22ca00640 1 --2- 172.21.15.27:0/2912620648 >> v2:172.21.15.155:3300/0 conn(0x7fa228150c70 0x7fa2280a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.026+0000 7fa22ca00640 1 -- 172.21.15.27:0/2912620648 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2280c3d90 con 0x7fa2280c3690 2024-03-28T19:12:10.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.027+0000 7fa22ca00640 1 --2- 172.21.15.27:0/2912620648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c3690 0x7fa2280c6120 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa2280c7950 tx=0x7fa21402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6383afe5a714f729 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa227e00640 1 -- 172.21.15.27:0/2912620648 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2140317c0 con 0x7fa2280c3690 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa227e00640 1 -- 172.21.15.27:0/2912620648 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa214037070 con 0x7fa2280c3690 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa227e00640 1 -- 172.21.15.27:0/2912620648 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214031b80 con 0x7fa2280c3690 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa22fe00640 1 -- 172.21.15.27:0/2912620648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c3690 msgr2=0x7fa2280c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa22fe00640 1 --2- 172.21.15.27:0/2912620648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c3690 0x7fa2280c6120 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa2280c7950 tx=0x7fa21402b860 comp rx=0 tx=0).stop 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa22fe00640 1 -- 172.21.15.27:0/2912620648 shutdown_connections 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa22fe00640 1 --2- 172.21.15.27:0/2912620648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c6660 0x7fa2281534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa22fe00640 1 --2- 172.21.15.27:0/2912620648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c3690 0x7fa2280c6120 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa22fe00640 1 --2- 172.21.15.27:0/2912620648 >> v2:172.21.15.155:3300/0 conn(0x7fa228150c70 0x7fa2280a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.028+0000 7fa22fe00640 1 -- 172.21.15.27:0/2912620648 >> 172.21.15.27:0/2912620648 conn(0x7fa22806b9d0 msgr2=0x7fa2280c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.029+0000 7fa22fe00640 1 -- 172.21.15.27:0/2912620648 shutdown_connections 2024-03-28T19:12:10.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.029+0000 7fa22fe00640 1 -- 172.21.15.27:0/2912620648 wait complete. 2024-03-28T19:12:10.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.030+0000 7fa22fe00640 1 Processor -- start 2024-03-28T19:12:10.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.030+0000 7fa22fe00640 1 -- start start 2024-03-28T19:12:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.031+0000 7fa22fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c3690 0x7fa2281dce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.031+0000 7fa22fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c6660 0x7fa2281e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.031+0000 7fa22fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa228150c70 0x7fa228149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.031+0000 7fa22fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280abcd0 con 0x7fa2280c6660 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.031+0000 7fa22fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280ab9f0 con 0x7fa228150c70 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.031+0000 7fa22fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280c1810 con 0x7fa2280c3690 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22d400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c3690 0x7fa2281dce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22d400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c3690 0x7fa2281dce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:48598/0 (socket says 172.21.15.27:48598) 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22d400640 1 -- 172.21.15.27:0/1830274366 learned_addr learned my addr 172.21.15.27:0/1830274366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22ca00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c6660 0x7fa2281e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22de00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa228150c70 0x7fa228149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22ca00640 1 -- 172.21.15.27:0/1830274366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c3690 msgr2=0x7fa2281dce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22ca00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c3690 0x7fa2281dce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22ca00640 1 -- 172.21.15.27:0/1830274366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa228150c70 msgr2=0x7fa228149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22ca00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa228150c70 0x7fa228149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22ca00640 1 -- 172.21.15.27:0/1830274366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22814b250 con 0x7fa2280c6660 2024-03-28T19:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.032+0000 7fa22d400640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c3690 0x7fa2281dce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:12:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.033+0000 7fa22ca00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c6660 0x7fa2281e02a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa214033000 tx=0x7fa2140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.033+0000 7fa225600640 1 -- 172.21.15.27:0/1830274366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214032860 con 0x7fa2280c6660 2024-03-28T19:12:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.033+0000 7fa225600640 1 -- 172.21.15.27:0/1830274366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa214031ec0 con 0x7fa2280c6660 2024-03-28T19:12:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.033+0000 7fa225600640 1 -- 172.21.15.27:0/1830274366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214032c80 con 0x7fa2280c6660 2024-03-28T19:12:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.033+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22814b530 con 0x7fa2280c6660 2024-03-28T19:12:10.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.034+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2281499a0 con 0x7fa2280c6660 2024-03-28T19:12:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.035+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2281494f0 con 0x7fa2280c6660 2024-03-28T19:12:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.036+0000 7fa225600640 1 -- 172.21.15.27:0/1830274366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa21404c060 con 0x7fa2280c6660 2024-03-28T19:12:10.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.037+0000 7fa225600640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa20806bbd0 0x7fa20806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:10.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.037+0000 7fa225600640 1 -- 172.21.15.27:0/1830274366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7fa2140acfe0 con 0x7fa2280c6660 2024-03-28T19:12:10.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.037+0000 7fa22d400640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa20806bbd0 0x7fa20806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:10.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.038+0000 7fa22d400640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa20806bbd0 0x7fa20806e090 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa21c006fd0 tx=0x7fa21c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.042+0000 7fa225600640 1 -- 172.21.15.27:0/1830274366 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa214040070 con 0x7fa2280c6660 2024-03-28T19:12:10.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.298+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa22804f400 con 0x7fa2280c6660 2024-03-28T19:12:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.298+0000 7fa225600640 1 -- 172.21.15.27:0/1830274366 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fa214032e20 con 0x7fa2280c6660 2024-03-28T19:12:10.298 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:10.298 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:10.298 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa20806bbd0 msgr2=0x7fa20806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa20806bbd0 0x7fa20806e090 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa21c006fd0 tx=0x7fa21c008040 comp rx=0 tx=0).stop 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c6660 msgr2=0x7fa2281e02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c6660 0x7fa2281e02a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa214033000 tx=0x7fa2140046c0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 shutdown_connections 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa20806bbd0 0x7fa20806e090 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa228150c70 0x7fa228149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2280c6660 0x7fa2281e02a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 --2- 172.21.15.27:0/1830274366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2280c3690 0x7fa2281dce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 >> 172.21.15.27:0/1830274366 conn(0x7fa22806b9d0 msgr2=0x7fa22814f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 shutdown_connections 2024-03-28T19:12:10.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:10.301+0000 7fa22fe00640 1 -- 172.21.15.27:0/1830274366 wait complete. 2024-03-28T19:12:10.319 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:12:11.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.136+0000 7f19dc600640 1 Processor -- start 2024-03-28T19:12:12.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.136+0000 7f19dc600640 1 -- start start 2024-03-28T19:12:12.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19dc600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19d40a9cb0 0x7f19d40c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19dc600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d41534c0 0x7f19d4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19dc600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d40c3d90 con 0x7f19d4150c70 2024-03-28T19:12:12.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19dc600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d40ab4e0 con 0x7f19d40a9cb0 2024-03-28T19:12:12.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19dc600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d40ac1f0 con 0x7f19d41534c0 2024-03-28T19:12:12.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19d9200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19d40a9cb0 0x7f19d40c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19da600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d41534c0 0x7f19d4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19d9200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19d40a9cb0 0x7f19d40c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47430/0 (socket says 172.21.15.27:47430) 2024-03-28T19:12:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.138+0000 7f19d9200640 1 -- 172.21.15.27:0/525597904 learned_addr learned my addr 172.21.15.27:0/525597904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.139+0000 7f19d9200640 1 -- 172.21.15.27:0/525597904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d41534c0 msgr2=0x7f19d4153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.139+0000 7f19d9200640 1 --2- 172.21.15.27:0/525597904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d41534c0 0x7f19d4153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.139+0000 7f19d3e00640 1 -- 172.21.15.27:0/525597904 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373935526 0 0) 0x7f19d40c3d90 con 0x7f19d4150c70 2024-03-28T19:12:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.139+0000 7f19d9200640 1 -- 172.21.15.27:0/525597904 >> v1:172.21.15.155:6789/0 conn(0x7f19d4150c70 legacy=0x7f19d40c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.139+0000 7f19d9200640 1 -- 172.21.15.27:0/525597904 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d40ab710 con 0x7f19d40a9cb0 2024-03-28T19:12:12.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.140+0000 7f19d9200640 1 --2- 172.21.15.27:0/525597904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d40a9cb0 0x7f19d40c3690 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009870 tx=0x7f19c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10dba95b476391c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:12.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.140+0000 7f19d3e00640 1 -- 172.21.15.27:0/525597904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0035020 con 0x7f19d40a9cb0 2024-03-28T19:12:12.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.140+0000 7f19d3e00640 1 -- 172.21.15.27:0/525597904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c0031640 con 0x7f19d40a9cb0 2024-03-28T19:12:12.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.140+0000 7f19d3e00640 1 -- 172.21.15.27:0/525597904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c002d040 con 0x7f19d40a9cb0 2024-03-28T19:12:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.141+0000 7f19dc600640 1 -- 172.21.15.27:0/525597904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d40a9cb0 msgr2=0x7f19d40c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.141+0000 7f19dc600640 1 --2- 172.21.15.27:0/525597904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d40a9cb0 0x7f19d40c3690 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009870 tx=0x7f19c002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.141+0000 7f19dc600640 1 -- 172.21.15.27:0/525597904 shutdown_connections 2024-03-28T19:12:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.141+0000 7f19dc600640 1 --2- 172.21.15.27:0/525597904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d41534c0 0x7f19d4153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.141+0000 7f19dc600640 1 --2- 172.21.15.27:0/525597904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d40a9cb0 0x7f19d40c3690 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.141+0000 7f19dc600640 1 -- 172.21.15.27:0/525597904 >> 172.21.15.27:0/525597904 conn(0x7f19d406b9d0 msgr2=0x7f19d40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.141+0000 7f19dc600640 1 -- 172.21.15.27:0/525597904 shutdown_connections 2024-03-28T19:12:12.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.141+0000 7f19dc600640 1 -- 172.21.15.27:0/525597904 wait complete. 2024-03-28T19:12:12.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.143+0000 7f19dc600640 1 Processor -- start 2024-03-28T19:12:12.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.143+0000 7f19dc600640 1 -- start start 2024-03-28T19:12:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.144+0000 7f19dc600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d40a9cb0 0x7f19d40c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.144+0000 7f19dc600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d4150c70 0x7f19d4147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.144+0000 7f19dc600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19d41534c0 0x7f19d4147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.144+0000 7f19dc600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d40c1810 con 0x7f19d4150c70 2024-03-28T19:12:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.144+0000 7f19dc600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d40abcd0 con 0x7f19d41534c0 2024-03-28T19:12:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.144+0000 7f19dc600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d40c6830 con 0x7f19d40a9cb0 2024-03-28T19:12:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d4150c70 0x7f19d4147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d4150c70 0x7f19d4147a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47442/0 (socket says 172.21.15.27:47442) 2024-03-28T19:12:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 -- 172.21.15.27:0/1758633499 learned_addr learned my addr 172.21.15.27:0/1758633499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9c00640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d40a9cb0 0x7f19d40c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19da600640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19d41534c0 0x7f19d4147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 -- 172.21.15.27:0/1758633499 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d40a9cb0 msgr2=0x7f19d40c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d40a9cb0 0x7f19d40c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 -- 172.21.15.27:0/1758633499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19d41534c0 msgr2=0x7f19d4147fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19d41534c0 0x7f19d4147fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 -- 172.21.15.27:0/1758633499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d4149120 con 0x7f19d4150c70 2024-03-28T19:12:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.145+0000 7f19d9200640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d4150c70 0x7f19d4147a90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f19c002b930 tx=0x7f19c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.146+0000 7f19d1600640 1 -- 172.21.15.27:0/1758633499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0040070 con 0x7f19d4150c70 2024-03-28T19:12:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.146+0000 7f19d1600640 1 -- 172.21.15.27:0/1758633499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c003e070 con 0x7f19d4150c70 2024-03-28T19:12:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.146+0000 7f19d1600640 1 -- 172.21.15.27:0/1758633499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0035040 con 0x7f19d4150c70 2024-03-28T19:12:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.146+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d41ee830 con 0x7f19d4150c70 2024-03-28T19:12:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.146+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d40c07e0 con 0x7f19d4150c70 2024-03-28T19:12:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.147+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d414f170 con 0x7f19d4150c70 2024-03-28T19:12:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.148+0000 7f19d1600640 1 -- 172.21.15.27:0/1758633499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f19c0038020 con 0x7f19d4150c70 2024-03-28T19:12:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.149+0000 7f19d1600640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19b406bc00 0x7f19b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.149+0000 7f19d1600640 1 -- 172.21.15.27:0/1758633499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7f19c00acda0 con 0x7f19d4150c70 2024-03-28T19:12:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.149+0000 7f19d9c00640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19b406bc00 0x7f19b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.150+0000 7f19d9c00640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19b406bc00 0x7f19b406e0c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f19c8000f60 tx=0x7f19c8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.153+0000 7f19d1600640 1 -- 172.21.15.27:0/1758633499 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f19d414f170 con 0x7f19d4150c70 2024-03-28T19:12:12.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.420+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19d404f400 con 0x7f19d4150c70 2024-03-28T19:12:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.421+0000 7f19d1600640 1 -- 172.21.15.27:0/1758633499 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f19c007a770 con 0x7f19d4150c70 2024-03-28T19:12:12.420 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:12.421 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:12.421 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19b406bc00 msgr2=0x7f19b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19b406bc00 0x7f19b406e0c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f19c8000f60 tx=0x7f19c8009040 comp rx=0 tx=0).stop 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d4150c70 msgr2=0x7f19d4147a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d4150c70 0x7f19d4147a90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f19c002b930 tx=0x7f19c00046c0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 shutdown_connections 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19b406bc00 0x7f19b406e0c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19d41534c0 0x7f19d4147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19d4150c70 0x7f19d4147a90 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 --2- 172.21.15.27:0/1758633499 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19d40a9cb0 0x7f19d40c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 >> 172.21.15.27:0/1758633499 conn(0x7f19d406b9d0 msgr2=0x7f19d40a9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 shutdown_connections 2024-03-28T19:12:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.424+0000 7f19dc600640 1 -- 172.21.15.27:0/1758633499 wait complete. 2024-03-28T19:12:12.442 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:12:12.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.545+0000 7fdc6b400640 1 Processor -- start 2024-03-28T19:12:12.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.545+0000 7fdc6b400640 1 -- start start 2024-03-28T19:12:12.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 0x7fdc640c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc6b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc641534c0 0x7fdc64153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc6b400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc640c3d90 con 0x7fdc64150c70 2024-03-28T19:12:12.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc6b400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc640ab4e0 con 0x7fdc641534c0 2024-03-28T19:12:12.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc6b400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc640ac1f0 con 0x7fdc640a9cb0 2024-03-28T19:12:12.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc69400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc641534c0 0x7fdc64153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc5be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 0x7fdc640c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc69400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdc641534c0 0x7fdc64153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47446/0 (socket says 172.21.15.27:47446) 2024-03-28T19:12:12.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.547+0000 7fdc69400640 1 -- 172.21.15.27:0/514063400 learned_addr learned my addr 172.21.15.27:0/514063400 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:12.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.548+0000 7fdc69400640 1 -- 172.21.15.27:0/514063400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 msgr2=0x7fdc640c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:12.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.548+0000 7fdc69400640 1 --2- 172.21.15.27:0/514063400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 0x7fdc640c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.548+0000 7fdc69400640 1 -- 172.21.15.27:0/514063400 >> v1:172.21.15.155:6789/0 conn(0x7fdc64150c70 legacy=0x7fdc640c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.548+0000 7fdc69400640 1 -- 172.21.15.27:0/514063400 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc640ab710 con 0x7fdc641534c0 2024-03-28T19:12:12.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.549+0000 7fdc69400640 1 --2- 172.21.15.27:0/514063400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64153950 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009e80 tx=0x7fdc5c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4913a5662f281f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:12.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc63e00640 1 -- 172.21.15.27:0/514063400 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c02ca50 con 0x7fdc641534c0 2024-03-28T19:12:12.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc63e00640 1 -- 172.21.15.27:0/514063400 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c02cbf0 con 0x7fdc641534c0 2024-03-28T19:12:12.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc63e00640 1 -- 172.21.15.27:0/514063400 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0343f0 con 0x7fdc641534c0 2024-03-28T19:12:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc6b400640 1 -- 172.21.15.27:0/514063400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 msgr2=0x7fdc64153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc6b400640 1 --2- 172.21.15.27:0/514063400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64153950 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009e80 tx=0x7fdc5c02beb0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc6b400640 1 -- 172.21.15.27:0/514063400 shutdown_connections 2024-03-28T19:12:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc6b400640 1 --2- 172.21.15.27:0/514063400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64153950 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc6b400640 1 --2- 172.21.15.27:0/514063400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 0x7fdc640c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.550+0000 7fdc6b400640 1 -- 172.21.15.27:0/514063400 >> 172.21.15.27:0/514063400 conn(0x7fdc6406b9d0 msgr2=0x7fdc640c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.551+0000 7fdc6b400640 1 -- 172.21.15.27:0/514063400 shutdown_connections 2024-03-28T19:12:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.551+0000 7fdc6b400640 1 -- 172.21.15.27:0/514063400 wait complete. 2024-03-28T19:12:12.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.552+0000 7fdc6b400640 1 Processor -- start 2024-03-28T19:12:12.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.552+0000 7fdc6b400640 1 -- start start 2024-03-28T19:12:12.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.553+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 0x7fdc641e02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.553+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc64150c70 0x7fdc641571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.553+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.553+0000 7fdc6b400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc640c1810 con 0x7fdc641534c0 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.553+0000 7fdc6b400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc640abcd0 con 0x7fdc64150c70 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.553+0000 7fdc6b400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc640c6830 con 0x7fdc640a9cb0 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc69400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc69400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47450/0 (socket says 172.21.15.27:47450) 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc69400640 1 -- 172.21.15.27:0/349621203 learned_addr learned my addr 172.21.15.27:0/349621203 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc68a00640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 0x7fdc641e02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc5be00640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc64150c70 0x7fdc641571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc69400640 1 -- 172.21.15.27:0/349621203 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 msgr2=0x7fdc641e02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc69400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 0x7fdc641e02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc69400640 1 -- 172.21.15.27:0/349621203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc64150c70 msgr2=0x7fdc641571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc69400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc64150c70 0x7fdc641571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.554+0000 7fdc69400640 1 -- 172.21.15.27:0/349621203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc640a8720 con 0x7fdc641534c0 2024-03-28T19:12:12.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.555+0000 7fdc69400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64156360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c02c4c0 tx=0x7fdc5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:12.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.555+0000 7fdc61600640 1 -- 172.21.15.27:0/349621203 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c043020 con 0x7fdc641534c0 2024-03-28T19:12:12.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.555+0000 7fdc61600640 1 -- 172.21.15.27:0/349621203 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c03e450 con 0x7fdc641534c0 2024-03-28T19:12:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.555+0000 7fdc61600640 1 -- 172.21.15.27:0/349621203 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c009510 con 0x7fdc641534c0 2024-03-28T19:12:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.555+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc641ecc80 con 0x7fdc641534c0 2024-03-28T19:12:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.555+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc641ed240 con 0x7fdc641534c0 2024-03-28T19:12:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.557+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc640c5830 con 0x7fdc641534c0 2024-03-28T19:12:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.557+0000 7fdc61600640 1 -- 172.21.15.27:0/349621203 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdc5c046020 con 0x7fdc641534c0 2024-03-28T19:12:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.558+0000 7fdc61600640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4006bbd0 0x7fdc4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:12.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.558+0000 7fdc61600640 1 -- 172.21.15.27:0/349621203 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7fdc5c0add10 con 0x7fdc641534c0 2024-03-28T19:12:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.559+0000 7fdc68a00640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4006bbd0 0x7fdc4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.559+0000 7fdc68a00640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4006bbd0 0x7fdc4006e090 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001640 tx=0x7fdc54009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.563+0000 7fdc61600640 1 -- 172.21.15.27:0/349621203 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdc640c5830 con 0x7fdc641534c0 2024-03-28T19:12:12.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.845+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc6404f400 con 0x7fdc641534c0 2024-03-28T19:12:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.846+0000 7fdc61600640 1 -- 172.21.15.27:0/349621203 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fdc5c04d030 con 0x7fdc641534c0 2024-03-28T19:12:12.846 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:12.846 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4006bbd0 msgr2=0x7fdc4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4006bbd0 0x7fdc4006e090 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001640 tx=0x7fdc54009040 comp rx=0 tx=0).stop 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 msgr2=0x7fdc64156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64156360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c02c4c0 tx=0x7fdc5c004a40 comp rx=0 tx=0).stop 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 shutdown_connections 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc4006bbd0 0x7fdc4006e090 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc641534c0 0x7fdc64156360 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc64150c70 0x7fdc641571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 --2- 172.21.15.27:0/349621203 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc640a9cb0 0x7fdc641e02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 >> 172.21.15.27:0/349621203 conn(0x7fdc6406b9d0 msgr2=0x7fdc640c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.849+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 shutdown_connections 2024-03-28T19:12:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:12.850+0000 7fdc6b400640 1 -- 172.21.15.27:0/349621203 wait complete. 2024-03-28T19:12:12.866 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:12:14.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:14.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.678+0000 7fbc1fe00640 1 Processor -- start 2024-03-28T19:12:14.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.678+0000 7fbc1fe00640 1 -- start start 2024-03-28T19:12:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.679+0000 7fbc1fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbc20150ae0 0x7fbc200a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.680+0000 7fbc1fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc200c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.680+0000 7fbc1fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc20150ec0 con 0x7fbc200c6ee0 2024-03-28T19:12:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.680+0000 7fbc1fe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2006dc70 con 0x7fbc20150ae0 2024-03-28T19:12:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.680+0000 7fbc1fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc20153550 con 0x7fbc200a61a0 2024-03-28T19:12:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.680+0000 7fbc1f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbc200c6ee0 0x7fbc20153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50768/0 (socket says 172.21.15.27:50768) 2024-03-28T19:12:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.680+0000 7fbc1f400640 1 -- 172.21.15.27:0/3320118880 learned_addr learned my addr 172.21.15.27:0/3320118880 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.680+0000 7fbc1ea00640 1 --2- 172.21.15.27:0/3320118880 >> v2:172.21.15.155:3300/0 conn(0x7fbc20150ae0 0x7fbc200a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.680+0000 7fbc17e00640 1 --2- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc200c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.681+0000 7fbc1e000640 1 -- 172.21.15.27:0/3320118880 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390915920 0 0) 0x7fbc18001490 con 0x7fbc200c6ee0 2024-03-28T19:12:14.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.681+0000 7fbc1e000640 1 -- 172.21.15.27:0/3320118880 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc04003520 con 0x7fbc200c6ee0 2024-03-28T19:12:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.681+0000 7fbc1ea00640 1 -- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 msgr2=0x7fbc200c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.681+0000 7fbc1ea00640 1 --2- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc200c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.681+0000 7fbc1ea00640 1 -- 172.21.15.27:0/3320118880 >> v1:172.21.15.27:6789/0 conn(0x7fbc200c6ee0 legacy=0x7fbc20153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.681+0000 7fbc1ea00640 1 -- 172.21.15.27:0/3320118880 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2006df10 con 0x7fbc20150ae0 2024-03-28T19:12:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.681+0000 7fbc17e00640 1 --2- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc200c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:12:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.682+0000 7fbc1ea00640 1 --2- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 0x7fbc200a5c60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbc100097a0 tx=0x7fbc1002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83deeba5b1082126 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:14.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.682+0000 7fbc1e000640 1 -- 172.21.15.27:0/3320118880 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc100315a0 con 0x7fbc20150ae0 2024-03-28T19:12:14.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.682+0000 7fbc1e000640 1 -- 172.21.15.27:0/3320118880 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc10037070 con 0x7fbc20150ae0 2024-03-28T19:12:14.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.682+0000 7fbc1e000640 1 -- 172.21.15.27:0/3320118880 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10031960 con 0x7fbc20150ae0 2024-03-28T19:12:14.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.683+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 msgr2=0x7fbc200a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:14.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.683+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 0x7fbc200a5c60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbc100097a0 tx=0x7fbc1002b840 comp rx=0 tx=0).stop 2024-03-28T19:12:14.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.683+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3320118880 shutdown_connections 2024-03-28T19:12:14.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.683+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc200c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.683+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3320118880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 0x7fbc200a5c60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.683+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3320118880 >> 172.21.15.27:0/3320118880 conn(0x7fbc2006b0c0 msgr2=0x7fbc2006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:14.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.683+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3320118880 shutdown_connections 2024-03-28T19:12:14.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.683+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3320118880 wait complete. 2024-03-28T19:12:14.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.685+0000 7fbc1fe00640 1 Processor -- start 2024-03-28T19:12:14.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.685+0000 7fbc1fe00640 1 -- start start 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc1fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc201eb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc1fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc200c6ee0 0x7fbc20072180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc1fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 0x7fbc200726c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc1fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2006e7b0 con 0x7fbc200c6ee0 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc1fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2006e4d0 con 0x7fbc20150ae0 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc1fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc201569d0 con 0x7fbc200a61a0 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc17e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc200c6ee0 0x7fbc20072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc17e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc200c6ee0 0x7fbc20072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47462/0 (socket says 172.21.15.27:47462) 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc17e00640 1 -- 172.21.15.27:0/3777371361 learned_addr learned my addr 172.21.15.27:0/3777371361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc1ea00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc201eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:14.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.686+0000 7fbc1f400640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 0x7fbc200726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc17e00640 1 -- 172.21.15.27:0/3777371361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 msgr2=0x7fbc201eb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc17e00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc201eb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc17e00640 1 -- 172.21.15.27:0/3777371361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 msgr2=0x7fbc200726c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc17e00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 0x7fbc200726c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc17e00640 1 -- 172.21.15.27:0/3777371361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc201e28c0 con 0x7fbc200c6ee0 2024-03-28T19:12:14.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc17e00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc200c6ee0 0x7fbc20072180 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbc0800e810 tx=0x7fbc0800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc16a00640 1 -- 172.21.15.27:0/3777371361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0800f040 con 0x7fbc200c6ee0 2024-03-28T19:12:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc16a00640 1 -- 172.21.15.27:0/3777371361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc08004030 con 0x7fbc200c6ee0 2024-03-28T19:12:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc16a00640 1 -- 172.21.15.27:0/3777371361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08004400 con 0x7fbc200c6ee0 2024-03-28T19:12:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.687+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc200ab4b0 con 0x7fbc200c6ee0 2024-03-28T19:12:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.688+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc201ebf40 con 0x7fbc200c6ee0 2024-03-28T19:12:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.689+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc20067dd0 con 0x7fbc200c6ee0 2024-03-28T19:12:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.690+0000 7fbc16a00640 1 -- 172.21.15.27:0/3777371361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbc0800c6d0 con 0x7fbc200c6ee0 2024-03-28T19:12:14.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.691+0000 7fbc16a00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbfc06bc00 0x7fbbfc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:14.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.691+0000 7fbc16a00640 1 -- 172.21.15.27:0/3777371361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8443+0+0 (secure 0 0 0) 0x7fbc0808b140 con 0x7fbc200c6ee0 2024-03-28T19:12:14.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.691+0000 7fbc1ea00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbfc06bc00 0x7fbbfc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.692+0000 7fbc1ea00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbfc06bc00 0x7fbbfc06e0c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbc10036000 tx=0x7fbc1003d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:14.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.695+0000 7fbc16a00640 1 -- 172.21.15.27:0/3777371361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbc201e28c0 con 0x7fbc200c6ee0 2024-03-28T19:12:14.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.944+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc20069640 con 0x7fbc200c6ee0 2024-03-28T19:12:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.945+0000 7fbc16a00640 1 -- 172.21.15.27:0/3777371361 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fbc08017020 con 0x7fbc200c6ee0 2024-03-28T19:12:14.944 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:14.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127},{"gid":6834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:09:52.749310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":115,"state":"up:active","state_seq":124,"addr":"172.21.15.155:6834/3692798710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3692798710},{"type":"v1","addr":"172.21.15.155:6834","nonce":3692798710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5078,"qdb_cluster":[5078]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:14.944 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 127 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbfc06bc00 msgr2=0x7fbbfc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbfc06bc00 0x7fbbfc06e0c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbc10036000 tx=0x7fbc1003d040 comp rx=0 tx=0).stop 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc200c6ee0 msgr2=0x7fbc20072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc200c6ee0 0x7fbc20072180 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbc0800e810 tx=0x7fbc0800ecd0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 shutdown_connections 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbfc06bc00 0x7fbbfc06e0c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc20150ae0 0x7fbc200726c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc200c6ee0 0x7fbc20072180 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.947+0000 7fbc1fe00640 1 --2- 172.21.15.27:0/3777371361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc200a61a0 0x7fbc201eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:14.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.948+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 >> 172.21.15.27:0/3777371361 conn(0x7fbc2006b0c0 msgr2=0x7fbc201545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:14.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.948+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 shutdown_connections 2024-03-28T19:12:14.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:14.948+0000 7fbc1fe00640 1 -- 172.21.15.27:0/3777371361 wait complete. 2024-03-28T19:12:14.964 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:12:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.106+0000 7fde6d800640 1 Processor -- start 2024-03-28T19:12:15.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.106+0000 7fde6d800640 1 -- start start 2024-03-28T19:12:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.108+0000 7fde6d800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68152e70 0x7fde68156c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.108+0000 7fde6d800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fde68156060 0x7fde680c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.108+0000 7fde6d800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde68070510 con 0x7fde680c2d80 2024-03-28T19:12:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.108+0000 7fde6d800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde680707f0 con 0x7fde68156060 2024-03-28T19:12:15.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.108+0000 7fde6d800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde68070ad0 con 0x7fde68152e70 2024-03-28T19:12:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.109+0000 7fde67400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fde68156060 0x7fde680c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.109+0000 7fde67400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fde68156060 0x7fde680c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:55294/0 (socket says 172.21.15.27:55294) 2024-03-28T19:12:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.109+0000 7fde67400640 1 -- 172.21.15.27:0/1202423435 learned_addr learned my addr 172.21.15.27:0/1202423435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.109+0000 7fde66000640 1 --1- 172.21.15.27:0/1202423435 >> v1:172.21.15.155:6789/0 conn(0x7fde680c2d80 0x7fde6806fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:52346/0 (socket says 172.21.15.27:52346) 2024-03-28T19:12:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.109+0000 7fde66a00640 1 --2- 172.21.15.27:0/1202423435 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68152e70 0x7fde68156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.110+0000 7fde67400640 1 -- 172.21.15.27:0/1202423435 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68152e70 msgr2=0x7fde68156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.110+0000 7fde67400640 1 --2- 172.21.15.27:0/1202423435 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68152e70 0x7fde68156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.110+0000 7fde67400640 1 -- 172.21.15.27:0/1202423435 >> v1:172.21.15.155:6789/0 conn(0x7fde680c2d80 legacy=0x7fde6806fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.110+0000 7fde67400640 1 -- 172.21.15.27:0/1202423435 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde68155860 con 0x7fde68156060 2024-03-28T19:12:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.110+0000 7fde65600640 1 -- 172.21.15.27:0/1202423435 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774406223 0 0) 0x7fde68070510 con 0x7fde680c2d80 2024-03-28T19:12:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.112+0000 7fde67400640 1 --2- 172.21.15.27:0/1202423435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde68156060 0x7fde680c8d70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fde60009e50 tx=0x7fde6002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2595f279186cabd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.112+0000 7fde65600640 1 -- 172.21.15.27:0/1202423435 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6002ca80 con 0x7fde68156060 2024-03-28T19:12:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.112+0000 7fde65600640 1 -- 172.21.15.27:0/1202423435 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6002cc20 con 0x7fde68156060 2024-03-28T19:12:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.112+0000 7fde65600640 1 -- 172.21.15.27:0/1202423435 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde600343f0 con 0x7fde68156060 2024-03-28T19:12:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.113+0000 7fde6d800640 1 -- 172.21.15.27:0/1202423435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde68156060 msgr2=0x7fde680c8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:15.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.113+0000 7fde6d800640 1 --2- 172.21.15.27:0/1202423435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde68156060 0x7fde680c8d70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fde60009e50 tx=0x7fde6002bee0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.113+0000 7fde6d800640 1 -- 172.21.15.27:0/1202423435 shutdown_connections 2024-03-28T19:12:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.113+0000 7fde6d800640 1 --2- 172.21.15.27:0/1202423435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde68156060 0x7fde680c8d70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.113+0000 7fde6d800640 1 --2- 172.21.15.27:0/1202423435 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68152e70 0x7fde68156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.113+0000 7fde6d800640 1 -- 172.21.15.27:0/1202423435 >> 172.21.15.27:0/1202423435 conn(0x7fde68066dd0 msgr2=0x7fde680671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.113+0000 7fde6d800640 1 -- 172.21.15.27:0/1202423435 shutdown_connections 2024-03-28T19:12:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.114+0000 7fde6d800640 1 -- 172.21.15.27:0/1202423435 wait complete. 2024-03-28T19:12:15.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.116+0000 7fde6d800640 1 Processor -- start 2024-03-28T19:12:15.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.117+0000 7fde6d800640 1 -- start start 2024-03-28T19:12:15.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.118+0000 7fde6d800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde680c2d80 0x7fde68157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.118+0000 7fde6d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde68152e70 0x7fde6814f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.118+0000 7fde6d800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68156060 0x7fde6814f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.118+0000 7fde6d800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde681533d0 con 0x7fde68152e70 2024-03-28T19:12:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.118+0000 7fde6d800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68153250 con 0x7fde680c2d80 2024-03-28T19:12:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.118+0000 7fde6d800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde68156510 con 0x7fde68156060 2024-03-28T19:12:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.119+0000 7fde66a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde680c2d80 0x7fde68157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.119+0000 7fde66a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde680c2d80 0x7fde68157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:55302/0 (socket says 172.21.15.27:55302) 2024-03-28T19:12:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.119+0000 7fde66a00640 1 -- 172.21.15.27:0/3616263172 learned_addr learned my addr 172.21.15.27:0/3616263172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.119+0000 7fde66000640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde68152e70 0x7fde6814f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.119+0000 7fde67400640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68156060 0x7fde6814f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:15.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.120+0000 7fde67400640 1 -- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde680c2d80 msgr2=0x7fde68157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.120+0000 7fde67400640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde680c2d80 0x7fde68157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.120+0000 7fde67400640 1 -- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde68152e70 msgr2=0x7fde6814f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.120+0000 7fde67400640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde68152e70 0x7fde6814f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.120+0000 7fde67400640 1 -- 172.21.15.27:0/3616263172 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde681e23e0 con 0x7fde68156060 2024-03-28T19:12:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.120+0000 7fde66a00640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde680c2d80 0x7fde68157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:12:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.120+0000 7fde66000640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde68152e70 0x7fde6814f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:12:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.120+0000 7fde67400640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68156060 0x7fde6814f550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde6806bf30 tx=0x7fde60004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.121+0000 7fde5ea00640 1 -- 172.21.15.27:0/3616263172 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60043020 con 0x7fde68156060 2024-03-28T19:12:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.121+0000 7fde5ea00640 1 -- 172.21.15.27:0/3616263172 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6003e450 con 0x7fde68156060 2024-03-28T19:12:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.121+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6814ff00 con 0x7fde68156060 2024-03-28T19:12:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.121+0000 7fde5ea00640 1 -- 172.21.15.27:0/3616263172 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6003e880 con 0x7fde68156060 2024-03-28T19:12:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.121+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde680ab5b0 con 0x7fde68156060 2024-03-28T19:12:15.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.122+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6804f400 con 0x7fde68156060 2024-03-28T19:12:15.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.125+0000 7fde5ea00640 1 -- 172.21.15.27:0/3616263172 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fde60046020 con 0x7fde68156060 2024-03-28T19:12:15.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.127+0000 7fde5ea00640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde4806bc20 0x7fde4806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.127+0000 7fde5ea00640 1 -- 172.21.15.27:0/3616263172 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8529+0+0 (secure 0 0 0) 0x7fde600ae0f0 con 0x7fde68156060 2024-03-28T19:12:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.128+0000 7fde66a00640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde4806bc20 0x7fde4806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.128+0000 7fde66a00640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde4806bc20 0x7fde4806e0e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fde680c4bf0 tx=0x7fde58008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:15.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.129+0000 7fde5ea00640 1 -- 172.21.15.27:0/3616263172 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fde6004d030 con 0x7fde68156060 2024-03-28T19:12:15.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.455+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde680c2730 con 0x7fde68156060 2024-03-28T19:12:15.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.456+0000 7fde5ea00640 1 -- 172.21.15.27:0/3616263172 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fde6007bc00 con 0x7fde68156060 2024-03-28T19:12:15.471 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:15.471 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:15.019980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:15.471 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 128 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.459+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde4806bc20 msgr2=0x7fde4806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.459+0000 7fde6d800640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde4806bc20 0x7fde4806e0e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fde680c4bf0 tx=0x7fde58008040 comp rx=0 tx=0).stop 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.459+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68156060 msgr2=0x7fde6814f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.459+0000 7fde6d800640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68156060 0x7fde6814f550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde6806bf30 tx=0x7fde60004a40 comp rx=0 tx=0).stop 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.460+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 shutdown_connections 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.460+0000 7fde6d800640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fde4806bc20 0x7fde4806e0e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.460+0000 7fde6d800640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fde68156060 0x7fde6814f550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.460+0000 7fde6d800640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde68152e70 0x7fde6814f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.460+0000 7fde6d800640 1 --2- 172.21.15.27:0/3616263172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde680c2d80 0x7fde68157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.460+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 >> 172.21.15.27:0/3616263172 conn(0x7fde68066dd0 msgr2=0x7fde680671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.460+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 shutdown_connections 2024-03-28T19:12:15.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:15.460+0000 7fde6d800640 1 -- 172.21.15.27:0/3616263172 wait complete. 2024-03-28T19:12:15.484 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:12:16.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.175+0000 7ff56ec00640 1 Processor -- start 2024-03-28T19:12:17.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.176+0000 7ff56ec00640 1 -- start start 2024-03-28T19:12:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.177+0000 7ff56ec00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 0x7ff5680c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.177+0000 7ff56ec00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff568158bd0 con 0x7ff5680c8ed0 2024-03-28T19:12:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.177+0000 7ff56ec00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff568158eb0 con 0x7ff5680c58c0 2024-03-28T19:12:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.177+0000 7ff56ec00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff568159190 con 0x7ff5680be5d0 2024-03-28T19:12:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff56ce00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff5680c8ed0 0x7ff568155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58458/0 (socket says 172.21.15.27:58458) 2024-03-28T19:12:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff56ce00640 1 -- 172.21.15.27:0/112701878 learned_addr learned my addr 172.21.15.27:0/112701878 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff56d800640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff5680c58c0 0x7ff5681555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:41550/0 (socket says 172.21.15.27:41550) 2024-03-28T19:12:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff56e200640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 0x7ff5680c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885264812 0 0) 0x7ff568158bd0 con 0x7ff5680c8ed0 2024-03-28T19:12:17.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff550003520 con 0x7ff5680c8ed0 2024-03-28T19:12:17.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302983242 0 0) 0x7ff568158eb0 con 0x7ff5680c58c0 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff568158bd0 con 0x7ff5680c58c0 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2258792795 0 0) 0x7ff550003520 con 0x7ff5680c8ed0 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 msgr2=0x7ff5680c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 0x7ff5680c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 >> v1:172.21.15.155:6789/0 conn(0x7ff5680c58c0 legacy=0x7ff5681555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff568159470 con 0x7ff5680c8ed0 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.178+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7ff554001670 con 0x7ff5680c8ed0 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 >> v1:172.21.15.27:6789/0 conn(0x7ff5680c8ed0 legacy=0x7ff568155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff567e00640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550003b70 0x7ff550003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff567e00640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550004800 0x7ff550004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff567e00640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 0x7ff550005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff568159190 con 0x7ff550003b70 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55000bb50 con 0x7ff550004800 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55000bd20 con 0x7ff5680be5d0 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56d800640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550003b70 0x7ff550003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56e200640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 0x7ff550005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56ce00640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550004800 0x7ff550004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56d800640 1 -- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 msgr2=0x7ff550005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56d800640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 0x7ff550005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56d800640 1 -- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550004800 msgr2=0x7ff550004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56d800640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550004800 0x7ff550004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56d800640 1 -- 172.21.15.27:0/112701878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff568159470 con 0x7ff550003b70 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56ce00640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550004800 0x7ff550004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.179+0000 7ff56e200640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680be5d0 0x7ff550005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:12:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.180+0000 7ff56d800640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550003b70 0x7ff550003f70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff560001f60 tx=0x7ff550005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d5755f38c53608 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.181+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560001b50 con 0x7ff550003b70 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.181+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff560004d60 con 0x7ff550003b70 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.181+0000 7ff567e00640 1 -- 172.21.15.27:0/112701878 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56002d730 con 0x7ff550003b70 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.181+0000 7ff56ec00640 1 -- 172.21.15.27:0/112701878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550003b70 msgr2=0x7ff550003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.181+0000 7ff56ec00640 1 --2- 172.21.15.27:0/112701878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff550003b70 0x7ff550003f70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff560001f60 tx=0x7ff550005830 comp rx=0 tx=0).stop 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.181+0000 7ff56e200640 1 -- 172.21.15.27:0/112701878 reap_dead start 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.182+0000 7ff56ec00640 1 -- 172.21.15.27:0/112701878 shutdown_connections 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.182+0000 7ff56ec00640 1 -- 172.21.15.27:0/112701878 >> 172.21.15.27:0/112701878 conn(0x7ff56806b6f0 msgr2=0x7ff5680acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.182+0000 7ff56ec00640 1 -- 172.21.15.27:0/112701878 shutdown_connections 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.182+0000 7ff56ec00640 1 -- 172.21.15.27:0/112701878 wait complete. 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.183+0000 7ff56ec00640 1 Processor -- start 2024-03-28T19:12:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.184+0000 7ff56ec00640 1 -- start start 2024-03-28T19:12:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.184+0000 7ff56ec00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550003b70 0x7ff5681f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.184+0000 7ff56ec00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680bea70 0x7ff5680ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56ec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff568151d70 0x7ff5681521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56ec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5680b09a0 con 0x7ff568151d70 2024-03-28T19:12:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56ec00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5680b0820 con 0x7ff550003b70 2024-03-28T19:12:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56ec00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5680ad7d0 con 0x7ff5680bea70 2024-03-28T19:12:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56d800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550003b70 0x7ff5681f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56ce00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680bea70 0x7ff5680ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56d800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550003b70 0x7ff5681f01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:52312/0 (socket says 172.21.15.27:52312) 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56d800640 1 -- 172.21.15.27:0/81019043 learned_addr learned my addr 172.21.15.27:0/81019043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.185+0000 7ff56e200640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff568151d70 0x7ff5681521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff56d800640 1 -- 172.21.15.27:0/81019043 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680bea70 msgr2=0x7ff5680ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff56d800640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680bea70 0x7ff5680ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff56d800640 1 -- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff568151d70 msgr2=0x7ff5681521f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff56d800640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff568151d70 0x7ff5681521f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff56d800640 1 -- 172.21.15.27:0/81019043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5680bd5f0 con 0x7ff550003b70 2024-03-28T19:12:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff56d800640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550003b70 0x7ff5681f01d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff560002080 tx=0x7ff560001580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff565600640 1 -- 172.21.15.27:0/81019043 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560001b50 con 0x7ff550003b70 2024-03-28T19:12:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff565600640 1 -- 172.21.15.27:0/81019043 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff560007280 con 0x7ff550003b70 2024-03-28T19:12:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff565600640 1 -- 172.21.15.27:0/81019043 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560007670 con 0x7ff550003b70 2024-03-28T19:12:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.186+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5680bd8d0 con 0x7ff550003b70 2024-03-28T19:12:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.187+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5680c4f80 con 0x7ff550003b70 2024-03-28T19:12:17.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.188+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff568069640 con 0x7ff550003b70 2024-03-28T19:12:17.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.189+0000 7ff565600640 1 -- 172.21.15.27:0/81019043 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff560041070 con 0x7ff550003b70 2024-03-28T19:12:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.190+0000 7ff565600640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff54806bbd0 0x7ff54806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.190+0000 7ff565600640 1 -- 172.21.15.27:0/81019043 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8529+0+0 (secure 0 0 0) 0x7ff5600ad580 con 0x7ff550003b70 2024-03-28T19:12:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.190+0000 7ff56ce00640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff54806bbd0 0x7ff54806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.191+0000 7ff56ce00640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff54806bbd0 0x7ff54806e090 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff568072520 tx=0x7ff554009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.198+0000 7ff565600640 1 -- 172.21.15.27:0/81019043 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff560031070 con 0x7ff550003b70 2024-03-28T19:12:17.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.462+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5680c5150 con 0x7ff550003b70 2024-03-28T19:12:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.463+0000 7ff565600640 1 -- 172.21.15.27:0/81019043 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5460 (secure 0 0 0) 0x7ff56007ae70 con 0x7ff550003b70 2024-03-28T19:12:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:17.462 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5756,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:15.019980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:11:05.517146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6846},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6846":{"gid":6846,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":19,"addr":"172.21.15.27:6837/242334139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":242334139},{"type":"v1","addr":"172.21.15.27:6837","nonce":242334139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6846,"qdb_cluster":[6846]},"id":2}]} 2024-03-28T19:12:17.462 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 128 2024-03-28T19:12:17.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.466+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff54806bbd0 msgr2=0x7ff54806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.466+0000 7ff56ec00640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff54806bbd0 0x7ff54806e090 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff568072520 tx=0x7ff554009040 comp rx=0 tx=0).stop 2024-03-28T19:12:17.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.466+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550003b70 msgr2=0x7ff5681f01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.466+0000 7ff56ec00640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550003b70 0x7ff5681f01d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff560002080 tx=0x7ff560001580 comp rx=0 tx=0).stop 2024-03-28T19:12:17.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.467+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 shutdown_connections 2024-03-28T19:12:17.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.467+0000 7ff56ec00640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff54806bbd0 0x7ff54806e090 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.467+0000 7ff56ec00640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff568151d70 0x7ff5681521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.467+0000 7ff56ec00640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff5680bea70 0x7ff5680ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.467+0000 7ff56ec00640 1 --2- 172.21.15.27:0/81019043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff550003b70 0x7ff5681f01d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.467+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 >> 172.21.15.27:0/81019043 conn(0x7ff56806b6f0 msgr2=0x7ff568155b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:17.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.467+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 shutdown_connections 2024-03-28T19:12:17.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.467+0000 7ff56ec00640 1 -- 172.21.15.27:0/81019043 wait complete. 2024-03-28T19:12:17.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:17.493 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-03-28T19:12:17.493 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:12:17.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:17.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.714+0000 7f1472a00640 1 Processor -- start 2024-03-28T19:12:17.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.715+0000 7f1472a00640 1 -- start start 2024-03-28T19:12:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.716+0000 7f1472a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f146c071b70 0x7f146c06b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.716+0000 7f1472a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146c14f720 con 0x7f146c0c1ef0 2024-03-28T19:12:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.716+0000 7f1472a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146c155f50 con 0x7f146c14f340 2024-03-28T19:12:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.716+0000 7f1472a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f146c155ac0 con 0x7f146c071b70 2024-03-28T19:12:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.716+0000 7f1463400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f146c0c1ef0 0x7f146c06b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58472/0 (socket says 172.21.15.27:58472) 2024-03-28T19:12:17.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.716+0000 7f1463400640 1 -- 172.21.15.27:0/1052828511 learned_addr learned my addr 172.21.15.27:0/1052828511 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.716+0000 7f1470a00640 1 --2- 172.21.15.27:0/1052828511 >> v2:172.21.15.155:3300/0 conn(0x7f146c071b70 0x7f146c06b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.717+0000 7f146b400640 1 -- 172.21.15.27:0/1052828511 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672033978 0 0) 0x7f146c14f720 con 0x7f146c0c1ef0 2024-03-28T19:12:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.717+0000 7f146b400640 1 -- 172.21.15.27:0/1052828511 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1450003520 con 0x7f146c0c1ef0 2024-03-28T19:12:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.717+0000 7f1470a00640 1 -- 172.21.15.27:0/1052828511 >> v1:172.21.15.155:6789/0 conn(0x7f146c14f340 legacy=0x7f146c0c26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.717+0000 7f1470a00640 1 -- 172.21.15.27:0/1052828511 >> v1:172.21.15.27:6789/0 conn(0x7f146c0c1ef0 legacy=0x7f146c06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.717+0000 7f1470a00640 1 -- 172.21.15.27:0/1052828511 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c155190 con 0x7f146c071b70 2024-03-28T19:12:17.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.718+0000 7f1470a00640 1 --2- 172.21.15.27:0/1052828511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c071b70 0x7f146c06b7d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f146400aea0 tx=0x7f146402cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f73fcd5f9e8d0bdb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.718+0000 7f146b400640 1 -- 172.21.15.27:0/1052828511 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146402dac0 con 0x7f146c071b70 2024-03-28T19:12:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.718+0000 7f146b400640 1 -- 172.21.15.27:0/1052828511 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146402dc60 con 0x7f146c071b70 2024-03-28T19:12:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.718+0000 7f146b400640 1 -- 172.21.15.27:0/1052828511 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464008660 con 0x7f146c071b70 2024-03-28T19:12:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.719+0000 7f1472a00640 1 -- 172.21.15.27:0/1052828511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c071b70 msgr2=0x7f146c06b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.719+0000 7f1472a00640 1 --2- 172.21.15.27:0/1052828511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c071b70 0x7f146c06b7d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f146400aea0 tx=0x7f146402cf20 comp rx=0 tx=0).stop 2024-03-28T19:12:17.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.719+0000 7f1472a00640 1 -- 172.21.15.27:0/1052828511 shutdown_connections 2024-03-28T19:12:17.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.719+0000 7f1472a00640 1 --2- 172.21.15.27:0/1052828511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c071b70 0x7f146c06b7d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.719+0000 7f1472a00640 1 -- 172.21.15.27:0/1052828511 >> 172.21.15.27:0/1052828511 conn(0x7f146c0c3ff0 msgr2=0x7f146c0c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:17.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.719+0000 7f1472a00640 1 -- 172.21.15.27:0/1052828511 shutdown_connections 2024-03-28T19:12:17.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.719+0000 7f1472a00640 1 -- 172.21.15.27:0/1052828511 wait complete. 2024-03-28T19:12:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.721+0000 7f1472a00640 1 Processor -- start 2024-03-28T19:12:17.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.721+0000 7f1472a00640 1 -- start start 2024-03-28T19:12:17.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f1472a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f146c071b70 0x7f146c14e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f1472a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c0c1ef0 0x7f146c149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f1472a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f146c14f340 0x7f146c14a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f1472a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c0712d0 con 0x7f146c071b70 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f1472a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c155750 con 0x7f146c0c1ef0 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f1472a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146c072150 con 0x7f146c14f340 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f146be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f146c071b70 0x7f146c14e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f146be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f146c071b70 0x7f146c14e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40058/0 (socket says 172.21.15.27:40058) 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f146be00640 1 -- 172.21.15.27:0/3739779162 learned_addr learned my addr 172.21.15.27:0/3739779162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f1463400640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c0c1ef0 0x7f146c149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f1470a00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f146c14f340 0x7f146c14a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f146be00640 1 -- 172.21.15.27:0/3739779162 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f146c14f340 msgr2=0x7f146c14a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f146be00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f146c14f340 0x7f146c14a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f146be00640 1 -- 172.21.15.27:0/3739779162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c0c1ef0 msgr2=0x7f146c149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f146be00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c0c1ef0 0x7f146c149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.722+0000 7f146be00640 1 -- 172.21.15.27:0/3739779162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c1ddb70 con 0x7f146c071b70 2024-03-28T19:12:17.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.723+0000 7f146be00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f146c071b70 0x7f146c14e250 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f145c00c430 tx=0x7f145c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.723+0000 7f1468c00640 1 -- 172.21.15.27:0/3739779162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c017070 con 0x7f146c071b70 2024-03-28T19:12:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.723+0000 7f1468c00640 1 -- 172.21.15.27:0/3739779162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c0052d0 con 0x7f146c071b70 2024-03-28T19:12:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.723+0000 7f1468c00640 1 -- 172.21.15.27:0/3739779162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c003830 con 0x7f146c071b70 2024-03-28T19:12:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.723+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146c1dde10 con 0x7f146c071b70 2024-03-28T19:12:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.723+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146c0c11b0 con 0x7f146c071b70 2024-03-28T19:12:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.724+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146c14f720 con 0x7f146c071b70 2024-03-28T19:12:17.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.726+0000 7f1468c00640 1 -- 172.21.15.27:0/3739779162 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f145c01a070 con 0x7f146c071b70 2024-03-28T19:12:17.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.727+0000 7f1468c00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f144806bbd0 0x7f144806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.727+0000 7f1468c00640 1 -- 172.21.15.27:0/3739779162 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8529+0+0 (secure 0 0 0) 0x7f145c08a8a0 con 0x7f146c071b70 2024-03-28T19:12:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.727+0000 7f1463400640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f144806bbd0 0x7f144806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.727+0000 7f4b28400640 1 Processor -- start 2024-03-28T19:12:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.727+0000 7f4b28400640 1 -- start start 2024-03-28T19:12:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.728+0000 7f1463400640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f144806bbd0 0x7f144806e090 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f146c14ebf0 tx=0x7f1454009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.729+0000 7f4b28400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4b200c3ea0 0x7f4b20159610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.729+0000 7f4b28400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200c1600 con 0x7f4b200c60e0 2024-03-28T19:12:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.729+0000 7f4b28400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200c18e0 con 0x7f4b2006ee00 2024-03-28T19:12:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.729+0000 7f4b28400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b200c23b0 con 0x7f4b200c3ea0 2024-03-28T19:12:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.729+0000 7f4b25a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4b200c60e0 0x7f4b2006e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58478/0 (socket says 172.21.15.27:58478) 2024-03-28T19:12:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.729+0000 7f4b25a00640 1 -- 172.21.15.27:0/1508130853 learned_addr learned my addr 172.21.15.27:0/1508130853 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.729+0000 7f4b26400640 1 --2- 172.21.15.27:0/1508130853 >> v2:172.21.15.155:3300/0 conn(0x7f4b200c3ea0 0x7f4b20159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.730+0000 7f4b1fe00640 1 -- 172.21.15.27:0/1508130853 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009019323 0 0) 0x7f4b200c18e0 con 0x7f4b2006ee00 2024-03-28T19:12:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.730+0000 7f4b1fe00640 1 -- 172.21.15.27:0/1508130853 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b08003520 con 0x7f4b2006ee00 2024-03-28T19:12:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.730+0000 7f4b26400640 1 -- 172.21.15.27:0/1508130853 >> v1:172.21.15.155:6789/0 conn(0x7f4b2006ee00 legacy=0x7f4b20158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.730+0000 7f4b26400640 1 -- 172.21.15.27:0/1508130853 >> v1:172.21.15.27:6789/0 conn(0x7f4b200c60e0 legacy=0x7f4b2006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.730+0000 7f4b26400640 1 -- 172.21.15.27:0/1508130853 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200c1bc0 con 0x7f4b200c3ea0 2024-03-28T19:12:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.731+0000 7f4b26400640 1 --2- 172.21.15.27:0/1508130853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 0x7f4b20159610 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4b200c2530 tx=0x7f4b1802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed329fcc786b9c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.731+0000 7f4b1fe00640 1 -- 172.21.15.27:0/1508130853 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18039070 con 0x7f4b200c3ea0 2024-03-28T19:12:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.731+0000 7f4b1fe00640 1 -- 172.21.15.27:0/1508130853 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1802cc50 con 0x7f4b200c3ea0 2024-03-28T19:12:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.731+0000 7f4b1fe00640 1 -- 172.21.15.27:0/1508130853 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18034440 con 0x7f4b200c3ea0 2024-03-28T19:12:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.732+0000 7f4b28400640 1 -- 172.21.15.27:0/1508130853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 msgr2=0x7f4b20159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.732+0000 7f4b28400640 1 --2- 172.21.15.27:0/1508130853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 0x7f4b20159610 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4b200c2530 tx=0x7f4b1802bea0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.732+0000 7f4b28400640 1 -- 172.21.15.27:0/1508130853 shutdown_connections 2024-03-28T19:12:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.732+0000 7f4b28400640 1 --2- 172.21.15.27:0/1508130853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 0x7f4b20159610 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.732+0000 7f1468c00640 1 -- 172.21.15.27:0/3739779162 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f146c14f720 con 0x7f146c071b70 2024-03-28T19:12:17.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.732+0000 7f4b28400640 1 -- 172.21.15.27:0/1508130853 >> 172.21.15.27:0/1508130853 conn(0x7f4b2006b9d0 msgr2=0x7f4b200c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:17.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.732+0000 7f4b28400640 1 -- 172.21.15.27:0/1508130853 shutdown_connections 2024-03-28T19:12:17.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.732+0000 7f4b28400640 1 -- 172.21.15.27:0/1508130853 wait complete. 2024-03-28T19:12:17.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.734+0000 7f4b28400640 1 Processor -- start 2024-03-28T19:12:17.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.735+0000 7f4b28400640 1 -- start start 2024-03-28T19:12:17.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.736+0000 7f4b28400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b2006ee00 0x7f4b200c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.736+0000 7f4b28400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 0x7f4b200ab090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.736+0000 7f4b28400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b200c60e0 0x7f4b20153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.736+0000 7f4b28400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200c64c0 con 0x7f4b200c60e0 2024-03-28T19:12:17.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.736+0000 7f4b28400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20159b50 con 0x7f4b200c3ea0 2024-03-28T19:12:17.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.736+0000 7f4b28400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200c6640 con 0x7f4b2006ee00 2024-03-28T19:12:17.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b25a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b2006ee00 0x7f4b200c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b26400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b200c60e0 0x7f4b20153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b25a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b2006ee00 0x7f4b200c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:48396/0 (socket says 172.21.15.27:48396) 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b25a00640 1 -- 172.21.15.27:0/3301288117 learned_addr learned my addr 172.21.15.27:0/3301288117 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b25000640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 0x7f4b200ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b26400640 1 -- 172.21.15.27:0/3301288117 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b2006ee00 msgr2=0x7f4b200c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b26400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b2006ee00 0x7f4b200c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b26400640 1 -- 172.21.15.27:0/3301288117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 msgr2=0x7f4b200ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b26400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 0x7f4b200ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b26400640 1 -- 172.21.15.27:0/3301288117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200aa6d0 con 0x7f4b200c60e0 2024-03-28T19:12:17.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b25a00640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b2006ee00 0x7f4b200c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:12:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.737+0000 7f4b26400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b200c60e0 0x7f4b20153330 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f4b18033000 tx=0x7f4b18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.738+0000 7f4b1d600640 1 -- 172.21.15.27:0/3301288117 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18039050 con 0x7f4b200c60e0 2024-03-28T19:12:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.738+0000 7f4b1d600640 1 -- 172.21.15.27:0/3301288117 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18045070 con 0x7f4b200c60e0 2024-03-28T19:12:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.738+0000 7f4b1d600640 1 -- 172.21.15.27:0/3301288117 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18009450 con 0x7f4b200c60e0 2024-03-28T19:12:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.738+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b200aa9b0 con 0x7f4b200c60e0 2024-03-28T19:12:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.738+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2009ad20 con 0x7f4b200c60e0 2024-03-28T19:12:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.739+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2004f400 con 0x7f4b200c60e0 2024-03-28T19:12:17.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.741+0000 7f4b1d600640 1 -- 172.21.15.27:0/3301288117 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4b1803e610 con 0x7f4b200c60e0 2024-03-28T19:12:17.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.742+0000 7f4b1d600640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b0006bc00 0x7f4b0006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:17.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.742+0000 7f4b1d600640 1 -- 172.21.15.27:0/3301288117 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8529+0+0 (secure 0 0 0) 0x7f4b180aded0 con 0x7f4b200c60e0 2024-03-28T19:12:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.743+0000 7f4b25a00640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b0006bc00 0x7f4b0006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:17.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.744+0000 7f4b25a00640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b0006bc00 0x7f4b0006e0c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4b14005880 tx=0x7f4b14009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:17.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:17.747+0000 7f4b1d600640 1 -- 172.21.15.27:0/3301288117 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4b1807b7c0 con 0x7f4b200c60e0 2024-03-28T19:12:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.016+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f146c068890 con 0x7f146c071b70 2024-03-28T19:12:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.018+0000 7f1468c00640 1 -- 172.21.15.27:0/3739779162 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4668 (secure 0 0 0) 0x7f146c068890 con 0x7f146c071b70 2024-03-28T19:12:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:18.017 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:17.706180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:17.775097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:12:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 130 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f144806bbd0 msgr2=0x7f144806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f144806bbd0 0x7f144806e090 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f146c14ebf0 tx=0x7f1454009040 comp rx=0 tx=0).stop 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f146c071b70 msgr2=0x7f146c14e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f146c071b70 0x7f146c14e250 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f145c00c430 tx=0x7f145c00c8f0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 shutdown_connections 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f144806bbd0 0x7f144806e090 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f146c14f340 0x7f146c14a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f146c0c1ef0 0x7f146c149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 --2- 172.21.15.27:0/3739779162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f146c071b70 0x7f146c14e250 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.020+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 >> 172.21.15.27:0/3739779162 conn(0x7f146c0c3ff0 msgr2=0x7f146c14c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:18.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.021+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 shutdown_connections 2024-03-28T19:12:18.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.021+0000 7f1472a00640 1 -- 172.21.15.27:0/3739779162 wait complete. 2024-03-28T19:12:18.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.028+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b200a5c40 con 0x7f4b200c60e0 2024-03-28T19:12:18.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.028+0000 7f4b1d600640 1 -- 172.21.15.27:0/3301288117 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4668 (secure 0 0 0) 0x7f4b1807f860 con 0x7f4b200c60e0 2024-03-28T19:12:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:18.028 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:17.706180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:17.775097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:12:18.028 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 130 2024-03-28T19:12:18.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b0006bc00 msgr2=0x7f4b0006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b0006bc00 0x7f4b0006e0c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4b14005880 tx=0x7f4b14009040 comp rx=0 tx=0).stop 2024-03-28T19:12:18.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b200c60e0 msgr2=0x7f4b20153330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b200c60e0 0x7f4b20153330 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f4b18033000 tx=0x7f4b18004a40 comp rx=0 tx=0).stop 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 shutdown_connections 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4b0006bc00 0x7f4b0006e0c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b200c60e0 0x7f4b20153330 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b200c3ea0 0x7f4b200ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 --2- 172.21.15.27:0/3301288117 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4b2006ee00 0x7f4b200c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 >> 172.21.15.27:0/3301288117 conn(0x7f4b2006b9d0 msgr2=0x7f4b200c5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 shutdown_connections 2024-03-28T19:12:18.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.031+0000 7f4b28400640 1 -- 172.21.15.27:0/3301288117 wait complete. 2024-03-28T19:12:18.036 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-03-28T19:12:18.037 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:12:18.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:18.050 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 130, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6834": { "addr": "172.21.15.27:6835/2858357568", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 2858357568, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 2858357568, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6834, "incarnation": 128, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:12:17.706180+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6834 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 130, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5756": { "addr": "172.21.15.155:6835/2989591316", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 2989591316, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 2989591316, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5756, "incarnation": 130, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:12:17.775097+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5756 } } } ], "standbys": [] } 2024-03-28T19:12:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.245+0000 7fc443e00640 1 Processor -- start 2024-03-28T19:12:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.245+0000 7fc443e00640 1 -- start start 2024-03-28T19:12:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.247+0000 7fc443e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc44414f3d0 0x7fc4440c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.247+0000 7fc443e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc444151d20 0x7fc4441521b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.247+0000 7fc443e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc444155250 con 0x7fc4440c5200 2024-03-28T19:12:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.247+0000 7fc443e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4440c1f30 con 0x7fc444151d20 2024-03-28T19:12:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.247+0000 7fc443e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc444155530 con 0x7fc44414f3d0 2024-03-28T19:12:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.247+0000 7fc442a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc44414f3d0 0x7fc4440c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:18.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.248+0000 7fc442a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc44414f3d0 0x7fc4440c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:48408/0 (socket says 172.21.15.27:48408) 2024-03-28T19:12:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.248+0000 7fc442a00640 1 -- 172.21.15.27:0/1138189216 learned_addr learned my addr 172.21.15.27:0/1138189216 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.248+0000 7fc443400640 1 --2- 172.21.15.27:0/1138189216 >> v2:172.21.15.155:3300/0 conn(0x7fc444151d20 0x7fc4441521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.248+0000 7fc442a00640 1 -- 172.21.15.27:0/1138189216 >> v2:172.21.15.155:3300/0 conn(0x7fc444151d20 msgr2=0x7fc4441521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.248+0000 7fc442a00640 1 --2- 172.21.15.27:0/1138189216 >> v2:172.21.15.155:3300/0 conn(0x7fc444151d20 0x7fc4441521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.248+0000 7fc442a00640 1 -- 172.21.15.27:0/1138189216 >> v1:172.21.15.155:6789/0 conn(0x7fc4440c5200 legacy=0x7fc4440c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.248+0000 7fc442a00640 1 -- 172.21.15.27:0/1138189216 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444155810 con 0x7fc44414f3d0 2024-03-28T19:12:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.249+0000 7fc442a00640 1 --2- 172.21.15.27:0/1138189216 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc44414f3d0 0x7fc4440c4910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc434009870 tx=0x7fc43402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9b355f672d6c86d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.250+0000 7fc442000640 1 -- 172.21.15.27:0/1138189216 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4340317c0 con 0x7fc44414f3d0 2024-03-28T19:12:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.250+0000 7fc442000640 1 -- 172.21.15.27:0/1138189216 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc434037070 con 0x7fc44414f3d0 2024-03-28T19:12:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.250+0000 7fc442000640 1 -- 172.21.15.27:0/1138189216 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434031b80 con 0x7fc44414f3d0 2024-03-28T19:12:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.251+0000 7fc443e00640 1 -- 172.21.15.27:0/1138189216 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc44414f3d0 msgr2=0x7fc4440c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.251+0000 7fc443e00640 1 --2- 172.21.15.27:0/1138189216 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc44414f3d0 0x7fc4440c4910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc434009870 tx=0x7fc43402b860 comp rx=0 tx=0).stop 2024-03-28T19:12:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.251+0000 7fc443e00640 1 -- 172.21.15.27:0/1138189216 shutdown_connections 2024-03-28T19:12:18.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.251+0000 7fc443e00640 1 --2- 172.21.15.27:0/1138189216 >> v2:172.21.15.155:3300/0 conn(0x7fc444151d20 0x7fc4441521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.251+0000 7fc443e00640 1 --2- 172.21.15.27:0/1138189216 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc44414f3d0 0x7fc4440c4910 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.251+0000 7fc443e00640 1 -- 172.21.15.27:0/1138189216 >> 172.21.15.27:0/1138189216 conn(0x7fc44406b0c0 msgr2=0x7fc44406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.251+0000 7fc443e00640 1 -- 172.21.15.27:0/1138189216 shutdown_connections 2024-03-28T19:12:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.251+0000 7fc443e00640 1 -- 172.21.15.27:0/1138189216 wait complete. 2024-03-28T19:12:18.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.253+0000 7fc443e00640 1 Processor -- start 2024-03-28T19:12:18.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.253+0000 7fc443e00640 1 -- start start 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4440c5200 0x7fc4441ebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc44414f3d0 0x7fc4441de820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc444151d20 0x7fc44414d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440c3ad0 con 0x7fc444151d20 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc444155dd0 con 0x7fc44414f3d0 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440c3c50 con 0x7fc4440c5200 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc444151d20 0x7fc44414d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc444151d20 0x7fc44414d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40080/0 (socket says 172.21.15.27:40080) 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443400640 1 -- 172.21.15.27:0/2067884431 learned_addr learned my addr 172.21.15.27:0/2067884431 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc43a000640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc44414f3d0 0x7fc4441de820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc442a00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4440c5200 0x7fc4441ebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443400640 1 -- 172.21.15.27:0/2067884431 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4440c5200 msgr2=0x7fc4441ebcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443400640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4440c5200 0x7fc4441ebcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443400640 1 -- 172.21.15.27:0/2067884431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc44414f3d0 msgr2=0x7fc4441de820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443400640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc44414f3d0 0x7fc4441de820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.254+0000 7fc443400640 1 -- 172.21.15.27:0/2067884431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44414ec10 con 0x7fc444151d20 2024-03-28T19:12:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.255+0000 7fc443400640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc444151d20 0x7fc44414d800 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009d30 tx=0x7fc43c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:18.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.255+0000 7fc43b400640 1 -- 172.21.15.27:0/2067884431 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c018020 con 0x7fc444151d20 2024-03-28T19:12:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.255+0000 7fc43b400640 1 -- 172.21.15.27:0/2067884431 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43c014480 con 0x7fc444151d20 2024-03-28T19:12:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.255+0000 7fc43b400640 1 -- 172.21.15.27:0/2067884431 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c014870 con 0x7fc444151d20 2024-03-28T19:12:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.255+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44414e650 con 0x7fc444151d20 2024-03-28T19:12:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.255+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44414cd20 con 0x7fc444151d20 2024-03-28T19:12:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.256+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc444067dd0 con 0x7fc444151d20 2024-03-28T19:12:18.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.257+0000 7fc43b400640 1 -- 172.21.15.27:0/2067884431 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc43c005d10 con 0x7fc444151d20 2024-03-28T19:12:18.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.258+0000 7fc43b400640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc42006bbd0 0x7fc42006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:18.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.258+0000 7fc43b400640 1 -- 172.21.15.27:0/2067884431 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7fc43c08bf40 con 0x7fc444151d20 2024-03-28T19:12:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.259+0000 7fc442a00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc42006bbd0 0x7fc42006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:18.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.259+0000 7fc442a00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc42006bbd0 0x7fc42006e090 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc434036000 tx=0x7fc43403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:18.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.263+0000 7fc43b400640 1 -- 172.21.15.27:0/2067884431 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc43c022070 con 0x7fc444151d20 2024-03-28T19:12:18.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.501+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc444069640 con 0x7fc444151d20 2024-03-28T19:12:18.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.501+0000 7fc43b400640 1 -- 172.21.15.27:0/2067884431 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4668 (secure 0 0 0) 0x7fc43c010040 con 0x7fc444151d20 2024-03-28T19:12:18.501 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:18.501 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:17.706180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:17.775097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:12:18.501 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 130 2024-03-28T19:12:18.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc42006bbd0 msgr2=0x7fc42006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc42006bbd0 0x7fc42006e090 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc434036000 tx=0x7fc43403d040 comp rx=0 tx=0).stop 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc444151d20 msgr2=0x7fc44414d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc444151d20 0x7fc44414d800 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009d30 tx=0x7fc43c00c490 comp rx=0 tx=0).stop 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 shutdown_connections 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc42006bbd0 0x7fc42006e090 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc444151d20 0x7fc44414d800 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc44414f3d0 0x7fc4441de820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 --2- 172.21.15.27:0/2067884431 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4440c5200 0x7fc4441ebcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 >> 172.21.15.27:0/2067884431 conn(0x7fc44406b0c0 msgr2=0x7fc44406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 shutdown_connections 2024-03-28T19:12:18.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:18.504+0000 7fc443e00640 1 -- 172.21.15.27:0/2067884431 wait complete. 2024-03-28T19:12:18.523 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 130, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6834": { "addr": "172.21.15.27:6835/2858357568", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 2858357568, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 2858357568, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6834, "incarnation": 128, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:12:17.706180+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6834 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 130, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5756": { "addr": "172.21.15.155:6835/2989591316", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 2989591316, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 2989591316, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5756, "incarnation": 130, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:12:17.775097+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5756 } } } ], "standbys": [] } 2024-03-28T19:12:20.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:20.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.256+0000 7fecb7000640 1 Processor -- start 2024-03-28T19:12:20.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.256+0000 7fecb7000640 1 -- start start 2024-03-28T19:12:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.257+0000 7fecb7000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0150ae0 0x7fecb00a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.257+0000 7fecb7000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fecb00a61a0 0x7fecb00c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.257+0000 7fecb7000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb0150ec0 con 0x7fecb00c6ee0 2024-03-28T19:12:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.257+0000 7fecb7000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb006dc70 con 0x7fecb00a61a0 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.257+0000 7fecb7000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb0153550 con 0x7fecb0150ae0 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecb6600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fecb00c6ee0 0x7fecb0153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58482/0 (socket says 172.21.15.27:58482) 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecb6600640 1 -- 172.21.15.27:0/3878877268 learned_addr learned my addr 172.21.15.27:0/3878877268 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecb5c00640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0150ae0 0x7fecb00a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350554754 0 0) 0x7feca8001490 con 0x7fecb00c6ee0 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec98003520 con 0x7fecb00c6ee0 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecb5200640 1 --2- 172.21.15.27:0/3878877268 >> v2:172.21.15.155:3300/0 conn(0x7fecb00a61a0 0x7fecb00c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2386214012 0 0) 0x7fec98003520 con 0x7fecb00c6ee0 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0150ae0 msgr2=0x7fecb00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.258+0000 7fecafe00640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0150ae0 0x7fecb00a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 >> v2:172.21.15.155:3300/0 conn(0x7fecb00a61a0 msgr2=0x7fecb00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 --2- 172.21.15.27:0/3878877268 >> v2:172.21.15.155:3300/0 conn(0x7fecb00a61a0 0x7fecb00c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb006df10 con 0x7fecb00c6ee0 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7feca8001bc0 con 0x7fecb00c6ee0 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 >> v1:172.21.15.27:6789/0 conn(0x7fecb00c6ee0 legacy=0x7fecb0153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0150ae0 0x7fec980039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec98003f30 0x7fec98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fec980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb006dc70 con 0x7fec98004d40 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb0153550 con 0x7fec98003f30 2024-03-28T19:12:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9800bac0 con 0x7fecb0150ae0 2024-03-28T19:12:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecb6600640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fec980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecb5c00640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0150ae0 0x7fec980039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecb5200640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec98003f30 0x7fec98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecb6600640 1 -- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0150ae0 msgr2=0x7fec980039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecb6600640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0150ae0 0x7fec980039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecb6600640 1 -- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec98003f30 msgr2=0x7fec98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecb6600640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec98003f30 0x7fec98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.259+0000 7fecb6600640 1 -- 172.21.15.27:0/3878877268 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec98003520 con 0x7fec98004d40 2024-03-28T19:12:20.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.261+0000 7fecb6600640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fec980051c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7feca800aa90 tx=0x7fec98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427e12933bf3e24b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:20.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.261+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca8038070 con 0x7fec98004d40 2024-03-28T19:12:20.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.261+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca80072d0 con 0x7fec98004d40 2024-03-28T19:12:20.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.261+0000 7fecafe00640 1 -- 172.21.15.27:0/3878877268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca800a6b0 con 0x7fec98004d40 2024-03-28T19:12:20.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.261+0000 7fecb7000640 1 -- 172.21.15.27:0/3878877268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 msgr2=0x7fec980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.261+0000 7fecb7000640 1 --2- 172.21.15.27:0/3878877268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fec980051c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7feca800aa90 tx=0x7fec98005830 comp rx=0 tx=0).stop 2024-03-28T19:12:20.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.261+0000 7fecb6600640 1 -- 172.21.15.27:0/3878877268 reap_dead start 2024-03-28T19:12:20.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.262+0000 7fecb7000640 1 -- 172.21.15.27:0/3878877268 shutdown_connections 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.262+0000 7fecb7000640 1 -- 172.21.15.27:0/3878877268 >> 172.21.15.27:0/3878877268 conn(0x7fecb006b0c0 msgr2=0x7fecb006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.262+0000 7fecb7000640 1 -- 172.21.15.27:0/3878877268 shutdown_connections 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.262+0000 7fecb7000640 1 -- 172.21.15.27:0/3878877268 wait complete. 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.263+0000 7fecb7000640 1 Processor -- start 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.263+0000 7fecb7000640 1 -- start start 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.264+0000 7fecb7000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fecb0072e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.264+0000 7fecb7000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0147f30 0x7fecb0152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.264+0000 7fecb7000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecb0070770 0x7fecb0070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.264+0000 7fecb7000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb006e7b0 con 0x7fec98004d40 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.264+0000 7fecb7000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb006e4d0 con 0x7fecb0070770 2024-03-28T19:12:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.264+0000 7fecb7000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb01569d0 con 0x7fecb0147f30 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fecb0072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fecb0072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40098/0 (socket says 172.21.15.27:40098) 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 -- 172.21.15.27:0/3907795246 learned_addr learned my addr 172.21.15.27:0/3907795246 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb6600640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecb0070770 0x7fecb0070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5200640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0147f30 0x7fecb0152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 -- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0147f30 msgr2=0x7fecb0152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0147f30 0x7fecb0152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 -- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecb0070770 msgr2=0x7fecb0070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecb0070770 0x7fecb0070bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 -- 172.21.15.27:0/3907795246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb00a71c0 con 0x7fec98004d40 2024-03-28T19:12:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb6600640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecb0070770 0x7fecb0070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:12:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.265+0000 7fecb5c00640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fecb0072e40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7feca40119a0 tx=0x7feca4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.266+0000 7fecad600640 1 -- 172.21.15.27:0/3907795246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca400faf0 con 0x7fec98004d40 2024-03-28T19:12:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.266+0000 7fecad600640 1 -- 172.21.15.27:0/3907795246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca400fc90 con 0x7fec98004d40 2024-03-28T19:12:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.266+0000 7fecad600640 1 -- 172.21.15.27:0/3907795246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca4014440 con 0x7fec98004d40 2024-03-28T19:12:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.266+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb00a88b0 con 0x7fec98004d40 2024-03-28T19:12:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.266+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb00a8390 con 0x7fec98004d40 2024-03-28T19:12:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.267+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb0067dd0 con 0x7fec98004d40 2024-03-28T19:12:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.268+0000 7fecad600640 1 -- 172.21.15.27:0/3907795246 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feca4012040 con 0x7fec98004d40 2024-03-28T19:12:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.269+0000 7fecad600640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec9006bc00 0x7fec9006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.270+0000 7fecad600640 1 -- 172.21.15.27:0/3907795246 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7feca408bc90 con 0x7fec98004d40 2024-03-28T19:12:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.270+0000 7fecb5200640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec9006bc00 0x7fec9006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.270+0000 7fecb5200640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec9006bc00 0x7fec9006e0c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fecb0155c10 tx=0x7fec9c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.275+0000 7fecad600640 1 -- 172.21.15.27:0/3907795246 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feca4019030 con 0x7fec98004d40 2024-03-28T19:12:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.523+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecb0069640 con 0x7fec98004d40 2024-03-28T19:12:20.523 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:20.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.524+0000 7fecad600640 1 -- 172.21.15.27:0/3907795246 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+4671 (secure 0 0 0) 0x7feca4059c00 con 0x7fec98004d40 2024-03-28T19:12:20.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:20.525 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:18.900981+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:clientreplay","state_seq":38,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:17.775097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:12:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 132 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec9006bc00 msgr2=0x7fec9006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec9006bc00 0x7fec9006e0c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fecb0155c10 tx=0x7fec9c008040 comp rx=0 tx=0).stop 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 msgr2=0x7fecb0072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fecb0072e40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7feca40119a0 tx=0x7feca4011e60 comp rx=0 tx=0).stop 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 shutdown_connections 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec9006bc00 0x7fec9006e0c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecb0070770 0x7fecb0070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecb0147f30 0x7fecb0152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 --2- 172.21.15.27:0/3907795246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec98004d40 0x7fecb0072e40 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.527+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 >> 172.21.15.27:0/3907795246 conn(0x7fecb006b0c0 msgr2=0x7fecb0154560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.528+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 shutdown_connections 2024-03-28T19:12:20.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.528+0000 7fecb7000640 1 -- 172.21.15.27:0/3907795246 wait complete. 2024-03-28T19:12:20.547 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:12:20.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.736+0000 7fa253c00640 1 Processor -- start 2024-03-28T19:12:20.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.736+0000 7fa253c00640 1 -- start start 2024-03-28T19:12:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.738+0000 7fa253c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa24c153bd0 0x7fa24c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.738+0000 7fa253c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c154020 con 0x7fa24c0a9cb0 2024-03-28T19:12:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.738+0000 7fa253c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c0c3d90 con 0x7fa24c150c70 2024-03-28T19:12:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.738+0000 7fa253c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24c0ab4e0 con 0x7fa24c153bd0 2024-03-28T19:12:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.738+0000 7fa24be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa24c0a9cb0 0x7fa24c1534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58498/0 (socket says 172.21.15.27:58498) 2024-03-28T19:12:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.738+0000 7fa24be00640 1 -- 172.21.15.27:0/1203744593 learned_addr learned my addr 172.21.15.27:0/1203744593 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:20.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.738+0000 7fa251c00640 1 --2- 172.21.15.27:0/1203744593 >> v2:172.21.15.155:3300/0 conn(0x7fa24c153bd0 0x7fa24c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.739+0000 7fa24b400640 1 -- 172.21.15.27:0/1203744593 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578842196 0 0) 0x7fa24c154020 con 0x7fa24c0a9cb0 2024-03-28T19:12:20.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.739+0000 7fa24b400640 1 -- 172.21.15.27:0/1203744593 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa230003520 con 0x7fa24c0a9cb0 2024-03-28T19:12:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.739+0000 7fa251c00640 1 -- 172.21.15.27:0/1203744593 >> v1:172.21.15.155:6789/0 conn(0x7fa24c150c70 legacy=0x7fa24c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.739+0000 7fa251c00640 1 -- 172.21.15.27:0/1203744593 >> v1:172.21.15.27:6789/0 conn(0x7fa24c0a9cb0 legacy=0x7fa24c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.739+0000 7fa251c00640 1 -- 172.21.15.27:0/1203744593 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c0ab710 con 0x7fa24c153bd0 2024-03-28T19:12:20.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.740+0000 7fa251c00640 1 --2- 172.21.15.27:0/1203744593 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c153bd0 0x7fa24c0c3690 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa244009d90 tx=0x7fa24402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27d9261da346c933 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.741+0000 7fa24b400640 1 -- 172.21.15.27:0/1203744593 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244039070 con 0x7fa24c153bd0 2024-03-28T19:12:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.741+0000 7fa24b400640 1 -- 172.21.15.27:0/1203744593 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24402cc60 con 0x7fa24c153bd0 2024-03-28T19:12:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.741+0000 7fa24b400640 1 -- 172.21.15.27:0/1203744593 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244034640 con 0x7fa24c153bd0 2024-03-28T19:12:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.742+0000 7fa253c00640 1 -- 172.21.15.27:0/1203744593 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c153bd0 msgr2=0x7fa24c0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.742+0000 7fa253c00640 1 --2- 172.21.15.27:0/1203744593 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c153bd0 0x7fa24c0c3690 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa244009d90 tx=0x7fa24402beb0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.742+0000 7fa253c00640 1 -- 172.21.15.27:0/1203744593 shutdown_connections 2024-03-28T19:12:20.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.742+0000 7fa253c00640 1 --2- 172.21.15.27:0/1203744593 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c153bd0 0x7fa24c0c3690 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.742+0000 7fa253c00640 1 -- 172.21.15.27:0/1203744593 >> 172.21.15.27:0/1203744593 conn(0x7fa24c06b9d0 msgr2=0x7fa24c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.742+0000 7fa253c00640 1 -- 172.21.15.27:0/1203744593 shutdown_connections 2024-03-28T19:12:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.742+0000 7fa253c00640 1 -- 172.21.15.27:0/1203744593 wait complete. 2024-03-28T19:12:20.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.744+0000 7fa253c00640 1 Processor -- start 2024-03-28T19:12:20.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.744+0000 7fa253c00640 1 -- start start 2024-03-28T19:12:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.745+0000 7fa253c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa24c0a9cb0 0x7fa24c0c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.745+0000 7fa253c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c150c70 0x7fa24c144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.745+0000 7fa253c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c153bd0 0x7fa24c1452d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.745+0000 7fa253c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0abcd0 con 0x7fa24c153bd0 2024-03-28T19:12:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.745+0000 7fa253c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0ac1f0 con 0x7fa24c150c70 2024-03-28T19:12:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.745+0000 7fa253c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0c1810 con 0x7fa24c0a9cb0 2024-03-28T19:12:20.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.745+0000 7fa251c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c153bd0 0x7fa24c1452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa251c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c153bd0 0x7fa24c1452d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40114/0 (socket says 172.21.15.27:40114) 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa251c00640 1 -- 172.21.15.27:0/454031637 learned_addr learned my addr 172.21.15.27:0/454031637 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa24be00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c150c70 0x7fa24c144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa251c00640 1 -- 172.21.15.27:0/454031637 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa24c0a9cb0 msgr2=0x7fa24c0c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa251c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa24c0a9cb0 0x7fa24c0c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa251c00640 1 -- 172.21.15.27:0/454031637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c150c70 msgr2=0x7fa24c144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa251c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c150c70 0x7fa24c144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa251c00640 1 -- 172.21.15.27:0/454031637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c1eae90 con 0x7fa24c153bd0 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.746+0000 7fa251c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c153bd0 0x7fa24c1452d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa244009d90 tx=0x7fa244004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:20.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.747+0000 7fa248c00640 1 -- 172.21.15.27:0/454031637 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244042030 con 0x7fa24c153bd0 2024-03-28T19:12:20.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.747+0000 7fa248c00640 1 -- 172.21.15.27:0/454031637 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24403a040 con 0x7fa24c153bd0 2024-03-28T19:12:20.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.747+0000 7fa248c00640 1 -- 172.21.15.27:0/454031637 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244039070 con 0x7fa24c153bd0 2024-03-28T19:12:20.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.747+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24c153970 con 0x7fa24c153bd0 2024-03-28T19:12:20.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.747+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24c06e390 con 0x7fa24c153bd0 2024-03-28T19:12:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.748+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24c04f400 con 0x7fa24c153bd0 2024-03-28T19:12:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.749+0000 7fa248c00640 1 -- 172.21.15.27:0/454031637 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa244045070 con 0x7fa24c153bd0 2024-03-28T19:12:20.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.750+0000 7fa248c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa22c06bbd0 0x7fa22c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:20.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.750+0000 7fa248c00640 1 -- 172.21.15.27:0/454031637 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7fa2440ad410 con 0x7fa24c153bd0 2024-03-28T19:12:20.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.750+0000 7fa251200640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa22c06bbd0 0x7fa22c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:20.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.751+0000 7fa251200640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa22c06bbd0 0x7fa22c06e090 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa23c000f60 tx=0x7fa23c0079b0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:20.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:20.755+0000 7fa248c00640 1 -- 172.21.15.27:0/454031637 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa24404c070 con 0x7fa24c153bd0 2024-03-28T19:12:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.067+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa24c0aacc0 con 0x7fa24c153bd0 2024-03-28T19:12:21.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.068+0000 7fa248c00640 1 -- 172.21.15.27:0/454031637 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+4672 (secure 0 0 0) 0x7fa24c1eae90 con 0x7fa24c153bd0 2024-03-28T19:12:21.067 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:21.068 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:17.775097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:12:21.068 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 133 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa22c06bbd0 msgr2=0x7fa22c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa22c06bbd0 0x7fa22c06e090 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa23c000f60 tx=0x7fa23c0079b0 comp rx=0 tx=0).stop 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c153bd0 msgr2=0x7fa24c1452d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c153bd0 0x7fa24c1452d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa244009d90 tx=0x7fa244004a40 comp rx=0 tx=0).stop 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 shutdown_connections 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa22c06bbd0 0x7fa22c06e090 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa24c153bd0 0x7fa24c1452d0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa24c150c70 0x7fa24c144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 --2- 172.21.15.27:0/454031637 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa24c0a9cb0 0x7fa24c0c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.071+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 >> 172.21.15.27:0/454031637 conn(0x7fa24c06b9d0 msgr2=0x7fa24c070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:21.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.072+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 shutdown_connections 2024-03-28T19:12:21.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:21.072+0000 7fa253c00640 1 -- 172.21.15.27:0/454031637 wait complete. 2024-03-28T19:12:21.093 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:12:22.548 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:22.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.796+0000 7fd673800640 1 Processor -- start 2024-03-28T19:12:22.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.796+0000 7fd673800640 1 -- start start 2024-03-28T19:12:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.798+0000 7fd673800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd66c153bd0 0x7fd66c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.798+0000 7fd673800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c154020 con 0x7fd66c0a9cb0 2024-03-28T19:12:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.798+0000 7fd673800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0c3d90 con 0x7fd66c150c70 2024-03-28T19:12:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.798+0000 7fd673800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c0ab4e0 con 0x7fd66c153bd0 2024-03-28T19:12:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.798+0000 7fd66be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd66c0a9cb0 0x7fd66c1534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58502/0 (socket says 172.21.15.27:58502) 2024-03-28T19:12:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.798+0000 7fd66be00640 1 -- 172.21.15.27:0/1244635809 learned_addr learned my addr 172.21.15.27:0/1244635809 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.798+0000 7fd671800640 1 --2- 172.21.15.27:0/1244635809 >> v2:172.21.15.155:3300/0 conn(0x7fd66c153bd0 0x7fd66c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.799+0000 7fd66b400640 1 -- 172.21.15.27:0/1244635809 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203642023 0 0) 0x7fd66c154020 con 0x7fd66c0a9cb0 2024-03-28T19:12:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.799+0000 7fd66b400640 1 -- 172.21.15.27:0/1244635809 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd650003520 con 0x7fd66c0a9cb0 2024-03-28T19:12:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.799+0000 7fd671800640 1 -- 172.21.15.27:0/1244635809 >> v1:172.21.15.155:6789/0 conn(0x7fd66c150c70 legacy=0x7fd66c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.799+0000 7fd671800640 1 -- 172.21.15.27:0/1244635809 >> v1:172.21.15.27:6789/0 conn(0x7fd66c0a9cb0 legacy=0x7fd66c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.799+0000 7fd671800640 1 -- 172.21.15.27:0/1244635809 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0ab710 con 0x7fd66c153bd0 2024-03-28T19:12:22.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.801+0000 7fd671800640 1 --2- 172.21.15.27:0/1244635809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c153bd0 0x7fd66c0c3690 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0097a0 tx=0x7fd65c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b171b261792837e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.801+0000 7fd66b400640 1 -- 172.21.15.27:0/1244635809 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c035020 con 0x7fd66c153bd0 2024-03-28T19:12:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.801+0000 7fd66b400640 1 -- 172.21.15.27:0/1244635809 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c031830 con 0x7fd66c153bd0 2024-03-28T19:12:22.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.801+0000 7fd66b400640 1 -- 172.21.15.27:0/1244635809 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c031be0 con 0x7fd66c153bd0 2024-03-28T19:12:22.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.802+0000 7fd673800640 1 -- 172.21.15.27:0/1244635809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c153bd0 msgr2=0x7fd66c0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:22.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.802+0000 7fd673800640 1 --2- 172.21.15.27:0/1244635809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c153bd0 0x7fd66c0c3690 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0097a0 tx=0x7fd65c02b8d0 comp rx=0 tx=0).stop 2024-03-28T19:12:22.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.802+0000 7fd673800640 1 -- 172.21.15.27:0/1244635809 shutdown_connections 2024-03-28T19:12:22.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.802+0000 7fd673800640 1 --2- 172.21.15.27:0/1244635809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c153bd0 0x7fd66c0c3690 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:22.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.802+0000 7fd673800640 1 -- 172.21.15.27:0/1244635809 >> 172.21.15.27:0/1244635809 conn(0x7fd66c06b9d0 msgr2=0x7fd66c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:22.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.802+0000 7fd673800640 1 -- 172.21.15.27:0/1244635809 shutdown_connections 2024-03-28T19:12:22.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.802+0000 7fd673800640 1 -- 172.21.15.27:0/1244635809 wait complete. 2024-03-28T19:12:22.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.804+0000 7fd673800640 1 Processor -- start 2024-03-28T19:12:22.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.805+0000 7fd673800640 1 -- start start 2024-03-28T19:12:22.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd673800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c0a9cb0 0x7fd66c1525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:22.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd673800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd66c150c70 0x7fd66c14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:22.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd673800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66c153bd0 0x7fd66c14f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:22.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd673800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0abcd0 con 0x7fd66c153bd0 2024-03-28T19:12:22.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd673800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0ac1f0 con 0x7fd66c0a9cb0 2024-03-28T19:12:22.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd673800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0c1810 con 0x7fd66c150c70 2024-03-28T19:12:22.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd671800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66c153bd0 0x7fd66c14f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:22.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd671800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66c153bd0 0x7fd66c14f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40128/0 (socket says 172.21.15.27:40128) 2024-03-28T19:12:22.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd671800640 1 -- 172.21.15.27:0/1925103840 learned_addr learned my addr 172.21.15.27:0/1925103840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:22.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd670e00640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c0a9cb0 0x7fd66c1525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:22.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.806+0000 7fd66be00640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd66c150c70 0x7fd66c14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:22.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.807+0000 7fd66be00640 1 -- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c0a9cb0 msgr2=0x7fd66c1525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:22.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.807+0000 7fd66be00640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c0a9cb0 0x7fd66c1525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:22.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.807+0000 7fd66be00640 1 -- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66c153bd0 msgr2=0x7fd66c14f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:22.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.807+0000 7fd66be00640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66c153bd0 0x7fd66c14f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:22.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.807+0000 7fd66be00640 1 -- 172.21.15.27:0/1925103840 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c154830 con 0x7fd66c150c70 2024-03-28T19:12:22.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.808+0000 7fd66be00640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd66c150c70 0x7fd66c14eea0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd65800c460 tx=0x7fd65800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:22.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.808+0000 7fd668c00640 1 -- 172.21.15.27:0/1925103840 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658017070 con 0x7fd66c150c70 2024-03-28T19:12:22.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.808+0000 7fd668c00640 1 -- 172.21.15.27:0/1925103840 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd658005ce0 con 0x7fd66c150c70 2024-03-28T19:12:22.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.808+0000 7fd668c00640 1 -- 172.21.15.27:0/1925103840 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658007270 con 0x7fd66c150c70 2024-03-28T19:12:22.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.808+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c1ef230 con 0x7fd66c150c70 2024-03-28T19:12:22.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.808+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c1530f0 con 0x7fd66c150c70 2024-03-28T19:12:22.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.809+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c152ee0 con 0x7fd66c150c70 2024-03-28T19:12:22.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.811+0000 7fd668c00640 1 -- 172.21.15.27:0/1925103840 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd65801a070 con 0x7fd66c150c70 2024-03-28T19:12:22.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.812+0000 7fd668c00640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd64c06bbd0 0x7fd64c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:22.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.813+0000 7fd668c00640 1 -- 172.21.15.27:0/1925103840 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7fd65808ac90 con 0x7fd66c150c70 2024-03-28T19:12:22.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.813+0000 7fd670e00640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd64c06bbd0 0x7fd64c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:22.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.814+0000 7fd670e00640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd64c06bbd0 0x7fd64c06e090 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd664001f10 tx=0x7fd664009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:22.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:22.816+0000 7fd668c00640 1 -- 172.21.15.27:0/1925103840 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd658058690 con 0x7fd66c150c70 2024-03-28T19:12:23.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:23.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.185+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd66c04f400 con 0x7fd66c150c70 2024-03-28T19:12:23.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.186+0000 7fd668c00640 1 -- 172.21.15.27:0/1925103840 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4673 (secure 0 0 0) 0x7fd65805c730 con 0x7fd66c150c70 2024-03-28T19:12:23.186 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:23.186 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:21.820966+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:12:23.187 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 135 2024-03-28T19:12:23.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd64c06bbd0 msgr2=0x7fd64c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd64c06bbd0 0x7fd64c06e090 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd664001f10 tx=0x7fd664009040 comp rx=0 tx=0).stop 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd66c150c70 msgr2=0x7fd66c14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd66c150c70 0x7fd66c14eea0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd65800c460 tx=0x7fd65800c920 comp rx=0 tx=0).stop 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 shutdown_connections 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd64c06bbd0 0x7fd64c06e090 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd66c153bd0 0x7fd66c14f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd66c150c70 0x7fd66c14eea0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 --2- 172.21.15.27:0/1925103840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd66c0a9cb0 0x7fd66c1525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 >> 172.21.15.27:0/1925103840 conn(0x7fd66c06b9d0 msgr2=0x7fd66c0c1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 shutdown_connections 2024-03-28T19:12:23.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.190+0000 7fd673800640 1 -- 172.21.15.27:0/1925103840 wait complete. 2024-03-28T19:12:23.211 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5078 2024-03-28T19:12:23.212 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.d 2024-03-28T19:12:23.212 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-03-28T19:12:23.212 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T19:12:23.212 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T19:12:23.219 INFO:tasks.ceph.mds.d:Started 2024-03-28T19:12:23.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:23.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.349+0000 7fc5cf200640 1 Processor -- start 2024-03-28T19:12:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.350+0000 7fc5cf200640 1 -- start start 2024-03-28T19:12:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.352+0000 7fc5cf200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc5c80c7970 0x7fc5c80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.352+0000 7fc5cf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8155730 0x7fc5c8155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.352+0000 7fc5cf200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80c7310 con 0x7fc5c8152f80 2024-03-28T19:12:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.352+0000 7fc5cf200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c8158d40 con 0x7fc5c80c7970 2024-03-28T19:12:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.352+0000 7fc5cf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8159020 con 0x7fc5c8155730 2024-03-28T19:12:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.352+0000 7fc5cd200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8155730 0x7fc5c8155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.352+0000 7fc5c7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc5c8152f80 0x7fc5c80c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58508/0 (socket says 172.21.15.27:58508) 2024-03-28T19:12:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.353+0000 7fc5c7e00640 1 -- 172.21.15.27:0/2302520562 learned_addr learned my addr 172.21.15.27:0/2302520562 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:12:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.353+0000 7fc5c7400640 1 --2- 172.21.15.27:0/2302520562 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80c7970 0x7fc5c80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.353+0000 7fc5cd200640 1 -- 172.21.15.27:0/2302520562 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80c7970 msgr2=0x7fc5c80c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.353+0000 7fc5cd200640 1 --2- 172.21.15.27:0/2302520562 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80c7970 0x7fc5c80c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.353+0000 7fc5cd200640 1 -- 172.21.15.27:0/2302520562 >> v1:172.21.15.27:6789/0 conn(0x7fc5c8152f80 legacy=0x7fc5c80c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.354+0000 7fc5cd200640 1 -- 172.21.15.27:0/2302520562 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8159300 con 0x7fc5c8155730 2024-03-28T19:12:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.354+0000 7fc5c7400640 1 --2- 172.21.15.27:0/2302520562 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80c7970 0x7fc5c80c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:12:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.355+0000 7fc5cd200640 1 --2- 172.21.15.27:0/2302520562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8155730 0x7fc5c8155bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009d90 tx=0x7fc5c002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f46f3f9215b3e311 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.356+0000 7fc5c6a00640 1 -- 172.21.15.27:0/2302520562 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0039070 con 0x7fc5c8155730 2024-03-28T19:12:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.356+0000 7fc5c6a00640 1 -- 172.21.15.27:0/2302520562 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c002cbc0 con 0x7fc5c8155730 2024-03-28T19:12:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.356+0000 7fc5c6a00640 1 -- 172.21.15.27:0/2302520562 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0034440 con 0x7fc5c8155730 2024-03-28T19:12:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.357+0000 7fc5cf200640 1 -- 172.21.15.27:0/2302520562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8155730 msgr2=0x7fc5c8155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.357+0000 7fc5cf200640 1 --2- 172.21.15.27:0/2302520562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8155730 0x7fc5c8155bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009d90 tx=0x7fc5c002be10 comp rx=0 tx=0).stop 2024-03-28T19:12:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.358+0000 7fc5cf200640 1 -- 172.21.15.27:0/2302520562 shutdown_connections 2024-03-28T19:12:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.358+0000 7fc5cf200640 1 --2- 172.21.15.27:0/2302520562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c8155730 0x7fc5c8155bc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.358+0000 7fc5cf200640 1 --2- 172.21.15.27:0/2302520562 >> v2:172.21.15.155:3300/0 conn(0x7fc5c80c7970 0x7fc5c80c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.358+0000 7fc5cf200640 1 -- 172.21.15.27:0/2302520562 >> 172.21.15.27:0/2302520562 conn(0x7fc5c806b6f0 msgr2=0x7fc5c80caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.358+0000 7fc5cf200640 1 -- 172.21.15.27:0/2302520562 shutdown_connections 2024-03-28T19:12:23.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.358+0000 7fc5cf200640 1 -- 172.21.15.27:0/2302520562 wait complete. 2024-03-28T19:12:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.361+0000 7fc5cf200640 1 Processor -- start 2024-03-28T19:12:23.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.361+0000 7fc5cf200640 1 -- start start 2024-03-28T19:12:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80c7970 0x7fc5c80c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cf200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c8152f80 0x7fc5c81575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cf200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8155730 0x7fc5c8148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cf200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80cbca0 con 0x7fc5c8155730 2024-03-28T19:12:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cf200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8159b00 con 0x7fc5c8152f80 2024-03-28T19:12:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80c2690 con 0x7fc5c80c7970 2024-03-28T19:12:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8155730 0x7fc5c8148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8155730 0x7fc5c8148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40134/0 (socket says 172.21.15.27:40134) 2024-03-28T19:12:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c8152f80 0x7fc5c81575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5cd200640 1 -- 172.21.15.27:0/3863409919 learned_addr learned my addr 172.21.15.27:0/3863409919 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.363+0000 7fc5c7e00640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80c7970 0x7fc5c80c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.364+0000 7fc5c7400640 1 -- 172.21.15.27:0/3863409919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80c7970 msgr2=0x7fc5c80c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.364+0000 7fc5c7400640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80c7970 0x7fc5c80c4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.364+0000 7fc5c7400640 1 -- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8155730 msgr2=0x7fc5c8148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.364+0000 7fc5c7400640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8155730 0x7fc5c8148140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.364+0000 7fc5c7400640 1 -- 172.21.15.27:0/3863409919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80c21b0 con 0x7fc5c8152f80 2024-03-28T19:12:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.364+0000 7fc5cd200640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8155730 0x7fc5c8148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:12:23.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.365+0000 7fc5c7400640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c8152f80 0x7fc5c81575d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400e870 tx=0x7fc5b400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.365+0000 7fc5bfe00640 1 -- 172.21.15.27:0/3863409919 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4017070 con 0x7fc5c8152f80 2024-03-28T19:12:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.365+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c81582b0 con 0x7fc5c8152f80 2024-03-28T19:12:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.366+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c8158830 con 0x7fc5c8152f80 2024-03-28T19:12:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.366+0000 7fc5bfe00640 1 -- 172.21.15.27:0/3863409919 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b400c940 con 0x7fc5c8152f80 2024-03-28T19:12:23.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.366+0000 7fc5bfe00640 1 -- 172.21.15.27:0/3863409919 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400cd10 con 0x7fc5c8152f80 2024-03-28T19:12:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.367+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a0005290 con 0x7fc5c8152f80 2024-03-28T19:12:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.370+0000 7fc5bfe00640 1 -- 172.21.15.27:0/3863409919 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc5b401a070 con 0x7fc5c8152f80 2024-03-28T19:12:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.371+0000 7fc5bfe00640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a806bbd0 0x7fc5a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.371+0000 7fc5bfe00640 1 -- 172.21.15.27:0/3863409919 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7fc5b408b7e0 con 0x7fc5c8152f80 2024-03-28T19:12:23.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.372+0000 7fc5c7e00640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a806bbd0 0x7fc5a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.373+0000 7fc5c7e00640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a806bbd0 0x7fc5a806e090 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8007fd0 tx=0x7fc5b8007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:23.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.380+0000 7fc5bfe00640 1 -- 172.21.15.27:0/3863409919 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc5b4058fe0 con 0x7fc5c8152f80 2024-03-28T19:12:23.403 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T19:12:23.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.469+0000 7f3030200640 1 Processor -- start 2024-03-28T19:12:23.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.469+0000 7f3030200640 1 -- start start 2024-03-28T19:12:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f3030200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3028150c70 0x7f30280a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f3030200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30281534c0 0x7f3028153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f3030200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30280c3af0 con 0x7f30280c3690 2024-03-28T19:12:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f3030200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30280ab4e0 con 0x7f3028150c70 2024-03-28T19:12:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f3030200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30280ac1f0 con 0x7f30281534c0 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f302ce00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f30280c3690 0x7f30280c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:41614/0 (socket says 172.21.15.27:41614) 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f302e200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30281534c0 0x7f3028153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f302e200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30281534c0 0x7f3028153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:48510/0 (socket says 172.21.15.27:48510) 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f302e200640 1 -- 172.21.15.27:0/2341401656 learned_addr learned my addr 172.21.15.27:0/2341401656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.472+0000 7f302e200640 1 -- 172.21.15.27:0/2341401656 >> v2:172.21.15.27:3300/0 conn(0x7f3028150c70 msgr2=0x7f30280a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.471+0000 7f302d800640 1 --2- 172.21.15.27:0/2341401656 >> v2:172.21.15.27:3300/0 conn(0x7f3028150c70 0x7f30280a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.472+0000 7f302e200640 1 --2- 172.21.15.27:0/2341401656 >> v2:172.21.15.27:3300/0 conn(0x7f3028150c70 0x7f30280a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.472+0000 7f302e200640 1 -- 172.21.15.27:0/2341401656 >> v1:172.21.15.155:6789/0 conn(0x7f30280c3690 legacy=0x7f30280c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.472+0000 7f302e200640 1 -- 172.21.15.27:0/2341401656 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280c3d90 con 0x7f30281534c0 2024-03-28T19:12:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.472+0000 7f3027e00640 1 -- 172.21.15.27:0/2341401656 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919043534 0 0) 0x7f30280c3af0 con 0x7f30280c3690 2024-03-28T19:12:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.472+0000 7f302d800640 1 --2- 172.21.15.27:0/2341401656 >> v2:172.21.15.27:3300/0 conn(0x7f3028150c70 0x7f30280a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:12:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.473+0000 7f302e200640 1 --2- 172.21.15.27:0/2341401656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30281534c0 0x7f3028153950 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3020009e50 tx=0x7f302002bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a93d96296fd4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.474+0000 7f3027e00640 1 -- 172.21.15.27:0/2341401656 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302002ca80 con 0x7f30281534c0 2024-03-28T19:12:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.474+0000 7f3027e00640 1 -- 172.21.15.27:0/2341401656 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302002cc20 con 0x7f30281534c0 2024-03-28T19:12:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.474+0000 7f3027e00640 1 -- 172.21.15.27:0/2341401656 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30200343f0 con 0x7f30281534c0 2024-03-28T19:12:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.475+0000 7f3030200640 1 -- 172.21.15.27:0/2341401656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30281534c0 msgr2=0x7f3028153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.475+0000 7f3030200640 1 --2- 172.21.15.27:0/2341401656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30281534c0 0x7f3028153950 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3020009e50 tx=0x7f302002bee0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.475+0000 7f3030200640 1 -- 172.21.15.27:0/2341401656 shutdown_connections 2024-03-28T19:12:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.475+0000 7f3030200640 1 --2- 172.21.15.27:0/2341401656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30281534c0 0x7f3028153950 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.475+0000 7f3030200640 1 --2- 172.21.15.27:0/2341401656 >> v2:172.21.15.27:3300/0 conn(0x7f3028150c70 0x7f30280a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.475+0000 7f3030200640 1 -- 172.21.15.27:0/2341401656 >> 172.21.15.27:0/2341401656 conn(0x7f302806b9d0 msgr2=0x7f30280c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.475+0000 7f3030200640 1 -- 172.21.15.27:0/2341401656 shutdown_connections 2024-03-28T19:12:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.476+0000 7f3030200640 1 -- 172.21.15.27:0/2341401656 wait complete. 2024-03-28T19:12:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.479+0000 7f3030200640 1 Processor -- start 2024-03-28T19:12:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.479+0000 7f3030200640 1 -- start start 2024-03-28T19:12:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.480+0000 7f3030200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280c3690 0x7f30281f3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.480+0000 7f3030200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3028150c70 0x7f30280a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.480+0000 7f3030200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30281534c0 0x7f3028156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.480+0000 7f3030200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30280abcd0 con 0x7f30280c3690 2024-03-28T19:12:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.480+0000 7f3030200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30280ab9f0 con 0x7f30281534c0 2024-03-28T19:12:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.480+0000 7f3030200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30280c1810 con 0x7f3028150c70 2024-03-28T19:12:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.480+0000 7f302d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280c3690 0x7f30281f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302ce00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3028150c70 0x7f30280a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302ce00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3028150c70 0x7f30280a6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:48514/0 (socket says 172.21.15.27:48514) 2024-03-28T19:12:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302ce00640 1 -- 172.21.15.27:0/244407553 learned_addr learned my addr 172.21.15.27:0/244407553 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302e200640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30281534c0 0x7f3028156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302d800640 1 -- 172.21.15.27:0/244407553 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3028150c70 msgr2=0x7f30280a6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302d800640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3028150c70 0x7f30280a6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302d800640 1 -- 172.21.15.27:0/244407553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30281534c0 msgr2=0x7f3028156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302d800640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30281534c0 0x7f3028156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.481+0000 7f302d800640 1 -- 172.21.15.27:0/244407553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30281549d0 con 0x7f30280c3690 2024-03-28T19:12:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.482+0000 7f302d800640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280c3690 0x7f30281f3320 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f301c00c820 tx=0x7f301c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.483+0000 7f3025600640 1 -- 172.21.15.27:0/244407553 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c015070 con 0x7f30280c3690 2024-03-28T19:12:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.483+0000 7f3025600640 1 -- 172.21.15.27:0/244407553 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c004220 con 0x7f30280c3690 2024-03-28T19:12:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.483+0000 7f3025600640 1 -- 172.21.15.27:0/244407553 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c00ce10 con 0x7f30280c3690 2024-03-28T19:12:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.483+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30281f3bf0 con 0x7f30280c3690 2024-03-28T19:12:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.483+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3028154cb0 con 0x7f30280c3690 2024-03-28T19:12:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.484+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302804f400 con 0x7f30280c3690 2024-03-28T19:12:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.486+0000 7f3025600640 1 -- 172.21.15.27:0/244407553 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f301c0077d0 con 0x7f30280c3690 2024-03-28T19:12:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.488+0000 7f3025600640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f300806bbd0 0x7f300806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.488+0000 7f3025600640 1 -- 172.21.15.27:0/244407553 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f301c08b3c0 con 0x7f30280c3690 2024-03-28T19:12:23.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.488+0000 7f302ce00640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f300806bbd0 0x7f300806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:23.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.490+0000 7f302ce00640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f300806bbd0 0x7f300806e090 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3014000f60 tx=0x7f3014007e10 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:23.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.495+0000 7f3025600640 1 -- 172.21.15.27:0/244407553 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f301c058ce0 con 0x7f30280c3690 2024-03-28T19:12:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.741+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5a0005580 con 0x7fc5c8152f80 2024-03-28T19:12:23.742 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:23.742 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:21.820966+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:12:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 135 2024-03-28T19:12:23.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.845+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30280aa7d0 con 0x7f30280c3690 2024-03-28T19:12:23.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.742+0000 7fc5bfe00640 1 -- 172.21.15.27:0/3863409919 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4673 (secure 0 0 0) 0x7fc5b405d080 con 0x7fc5c8152f80 2024-03-28T19:12:23.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a806bbd0 msgr2=0x7fc5a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a806bbd0 0x7fc5a806e090 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8007fd0 tx=0x7fc5b8007e80 comp rx=0 tx=0).stop 2024-03-28T19:12:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c8152f80 msgr2=0x7fc5c81575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c8152f80 0x7fc5c81575d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400e870 tx=0x7fc5b400ed30 comp rx=0 tx=0).stop 2024-03-28T19:12:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 shutdown_connections 2024-03-28T19:12:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc5a806bbd0 0x7fc5a806e090 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8155730 0x7fc5c8148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5c8152f80 0x7fc5c81575d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 --2- 172.21.15.27:0/3863409919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5c80c7970 0x7fc5c80c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 >> 172.21.15.27:0/3863409919 conn(0x7fc5c806b6f0 msgr2=0x7fc5c8151020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:23.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 shutdown_connections 2024-03-28T19:12:23.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.745+0000 7fc5cf200640 1 -- 172.21.15.27:0/3863409919 wait complete. 2024-03-28T19:12:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.907+0000 7f3025600640 1 -- 172.21.15.27:0/244407553 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5474 (secure 0 0 0) 0x7f301c05cd80 con 0x7f30280c3690 2024-03-28T19:12:23.906 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:23.907 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:12:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 136 2024-03-28T19:12:23.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f300806bbd0 msgr2=0x7f300806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f300806bbd0 0x7f300806e090 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3014000f60 tx=0x7f3014007e10 comp rx=0 tx=0).stop 2024-03-28T19:12:23.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280c3690 msgr2=0x7f30281f3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:23.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280c3690 0x7f30281f3320 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f301c00c820 tx=0x7f301c00cce0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 shutdown_connections 2024-03-28T19:12:23.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f300806bbd0 0x7f300806e090 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30281534c0 0x7f3028156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3028150c70 0x7f30280a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 --2- 172.21.15.27:0/244407553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30280c3690 0x7f30281f3320 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:23.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 >> 172.21.15.27:0/244407553 conn(0x7f302806b9d0 msgr2=0x7f30280c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:23.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 shutdown_connections 2024-03-28T19:12:23.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:23.910+0000 7f3030200640 1 -- 172.21.15.27:0/244407553 wait complete. 2024-03-28T19:12:23.924 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:12:23.932 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-03-28T19:12:23.932 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2024-03-28T19:12:24.815 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:12:24.815+0000 7f5ad3000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:12:25.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.183+0000 7faadf800640 1 Processor -- start 2024-03-28T19:12:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.184+0000 7faadf800640 1 -- start start 2024-03-28T19:12:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.185+0000 7faadf800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faad8152f80 0x7faad80c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.185+0000 7faadf800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7faad80c7970 0x7faad80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.185+0000 7faadf800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad80c7310 con 0x7faad8155730 2024-03-28T19:12:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.185+0000 7faadf800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad8158d40 con 0x7faad8152f80 2024-03-28T19:12:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.185+0000 7faadf800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad8159020 con 0x7faad80c7970 2024-03-28T19:12:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.186+0000 7faadce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faad8152f80 0x7faad80c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.186+0000 7faad7e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7faad80c7970 0x7faad80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.186+0000 7faadce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faad8152f80 0x7faad80c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48670/0 (socket says 172.21.15.27:48670) 2024-03-28T19:12:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.186+0000 7faadce00640 1 -- 172.21.15.27:0/3416637537 learned_addr learned my addr 172.21.15.27:0/3416637537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.186+0000 7faadce00640 1 -- 172.21.15.27:0/3416637537 >> v2:172.21.15.155:3300/0 conn(0x7faad80c7970 msgr2=0x7faad80c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:26.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.186+0000 7faadce00640 1 --2- 172.21.15.27:0/3416637537 >> v2:172.21.15.155:3300/0 conn(0x7faad80c7970 0x7faad80c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.186+0000 7faadce00640 1 -- 172.21.15.27:0/3416637537 >> v1:172.21.15.155:6789/0 conn(0x7faad8155730 legacy=0x7faad8155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:26.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.186+0000 7faadce00640 1 -- 172.21.15.27:0/3416637537 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8159300 con 0x7faad8152f80 2024-03-28T19:12:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.188+0000 7faadce00640 1 --2- 172.21.15.27:0/3416637537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 0x7faad80c58e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7faad80c8480 tx=0x7faacc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cfd5fc6a968309 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.188+0000 7faad7400640 1 -- 172.21.15.27:0/3416637537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc035020 con 0x7faad8152f80 2024-03-28T19:12:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.188+0000 7faad7400640 1 -- 172.21.15.27:0/3416637537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faacc031870 con 0x7faad8152f80 2024-03-28T19:12:26.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.188+0000 7faad7400640 1 -- 172.21.15.27:0/3416637537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faacc031c40 con 0x7faad8152f80 2024-03-28T19:12:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.189+0000 7faadf800640 1 -- 172.21.15.27:0/3416637537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 msgr2=0x7faad80c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.189+0000 7faadf800640 1 --2- 172.21.15.27:0/3416637537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 0x7faad80c58e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7faad80c8480 tx=0x7faacc02b910 comp rx=0 tx=0).stop 2024-03-28T19:12:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.189+0000 7faadf800640 1 -- 172.21.15.27:0/3416637537 shutdown_connections 2024-03-28T19:12:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.189+0000 7faadf800640 1 --2- 172.21.15.27:0/3416637537 >> v2:172.21.15.155:3300/0 conn(0x7faad80c7970 0x7faad80c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.189+0000 7faadf800640 1 --2- 172.21.15.27:0/3416637537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 0x7faad80c58e0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.189+0000 7faadf800640 1 -- 172.21.15.27:0/3416637537 >> 172.21.15.27:0/3416637537 conn(0x7faad806b6f0 msgr2=0x7faad80caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.189+0000 7faadf800640 1 -- 172.21.15.27:0/3416637537 shutdown_connections 2024-03-28T19:12:26.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.189+0000 7faadf800640 1 -- 172.21.15.27:0/3416637537 wait complete. 2024-03-28T19:12:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.191+0000 7faadf800640 1 Processor -- start 2024-03-28T19:12:26.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.192+0000 7faadf800640 1 -- start start 2024-03-28T19:12:26.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadf800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faad80c7970 0x7faad8154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:26.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadf800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 0x7faad81ef460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:26.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadf800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faad8155730 0x7faad814dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:26.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadf800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad8159b00 con 0x7faad8152f80 2024-03-28T19:12:26.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadf800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad80c5e20 con 0x7faad80c7970 2024-03-28T19:12:26.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadf800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad80cbca0 con 0x7faad8155730 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faad80c7970 0x7faad8154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faad80c7970 0x7faad8154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:52478/0 (socket says 172.21.15.27:52478) 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadce00640 1 -- 172.21.15.27:0/3654402549 learned_addr learned my addr 172.21.15.27:0/3654402549 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faad7e00640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 0x7faad81ef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.193+0000 7faadd800640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faad8155730 0x7faad814dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.194+0000 7faad7e00640 1 -- 172.21.15.27:0/3654402549 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faad8155730 msgr2=0x7faad814dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.194+0000 7faad7e00640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faad8155730 0x7faad814dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.194+0000 7faad7e00640 1 -- 172.21.15.27:0/3654402549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faad80c7970 msgr2=0x7faad8154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:26.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.194+0000 7faad7e00640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faad80c7970 0x7faad8154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.194+0000 7faad7e00640 1 -- 172.21.15.27:0/3654402549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad81ef9a0 con 0x7faad8152f80 2024-03-28T19:12:26.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.195+0000 7faad7e00640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 0x7faad81ef460 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7faac400e8a0 tx=0x7faac400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.195+0000 7faad4c00640 1 -- 172.21.15.27:0/3654402549 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4017070 con 0x7faad8152f80 2024-03-28T19:12:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.195+0000 7faad4c00640 1 -- 172.21.15.27:0/3654402549 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac400c940 con 0x7faad8152f80 2024-03-28T19:12:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.195+0000 7faad4c00640 1 -- 172.21.15.27:0/3654402549 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4009e60 con 0x7faad8152f80 2024-03-28T19:12:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.195+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad8155010 con 0x7faad8152f80 2024-03-28T19:12:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.195+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad81eec60 con 0x7faad8152f80 2024-03-28T19:12:26.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.197+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad8153360 con 0x7faad8152f80 2024-03-28T19:12:26.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.198+0000 7faad4c00640 1 -- 172.21.15.27:0/3654402549 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faac401a070 con 0x7faad8152f80 2024-03-28T19:12:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.199+0000 7faad4c00640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faab806bc00 0x7faab806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:26.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.199+0000 7faad4c00640 1 -- 172.21.15.27:0/3654402549 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7faac408b2a0 con 0x7faad8152f80 2024-03-28T19:12:26.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.199+0000 7faadce00640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faab806bc00 0x7faab806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:26.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.200+0000 7faadce00640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faab806bc00 0x7faab806e0c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faacc034040 tx=0x7faacc03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:26.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.204+0000 7faad4c00640 1 -- 172.21.15.27:0/3654402549 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faad8153360 con 0x7faad8152f80 2024-03-28T19:12:26.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.549+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faad804f400 con 0x7faad8152f80 2024-03-28T19:12:26.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.550+0000 7faad4c00640 1 -- 172.21.15.27:0/3654402549 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5474 (secure 0 0 0) 0x7faac4058c20 con 0x7faad8152f80 2024-03-28T19:12:26.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:26.550 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:12:26.551 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 136 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.553+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faab806bc00 msgr2=0x7faab806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faab806bc00 0x7faab806e0c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faacc034040 tx=0x7faacc03c040 comp rx=0 tx=0).stop 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 msgr2=0x7faad81ef460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 0x7faad81ef460 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7faac400e8a0 tx=0x7faac400ed60 comp rx=0 tx=0).stop 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 shutdown_connections 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faab806bc00 0x7faab806e0c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faad8155730 0x7faad814dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad8152f80 0x7faad81ef460 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 --2- 172.21.15.27:0/3654402549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faad80c7970 0x7faad8154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 >> 172.21.15.27:0/3654402549 conn(0x7faad806b6f0 msgr2=0x7faad8151de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 shutdown_connections 2024-03-28T19:12:26.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:26.554+0000 7faadf800640 1 -- 172.21.15.27:0/3654402549 wait complete. 2024-03-28T19:12:26.575 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=6846 2024-03-28T19:12:26.576 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.a 2024-03-28T19:12:27.576 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-03-28T19:12:27.576 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:12:27.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:12:27.582 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:12:27.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:12:27.794 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:12:27.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.839+0000 7f3b3ae00640 1 Processor -- start 2024-03-28T19:12:27.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.839+0000 7f3b3ae00640 1 -- start start 2024-03-28T19:12:27.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.840+0000 7f3b3ae00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b340a9cb0 0x7f3b340c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:27.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.841+0000 7f3b3ae00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3b341534c0 0x7f3b34153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:27.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.841+0000 7f3b3ae00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b340c3d90 con 0x7f3b34150c70 2024-03-28T19:12:27.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.841+0000 7f3b3ae00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b340ab4e0 con 0x7f3b341534c0 2024-03-28T19:12:27.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.841+0000 7f3b3ae00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340ac1f0 con 0x7f3b340a9cb0 2024-03-28T19:12:27.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.841+0000 7f3b33400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b340a9cb0 0x7f3b340c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:27.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.841+0000 7f3b38e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3b341534c0 0x7f3b34153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:27.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.841+0000 7f3b38e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3b341534c0 0x7f3b34153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:52482/0 (socket says 172.21.15.27:52482) 2024-03-28T19:12:27.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.841+0000 7f3b38e00640 1 -- 172.21.15.27:0/1469652602 learned_addr learned my addr 172.21.15.27:0/1469652602 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:27.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.842+0000 7f3b33400640 1 -- 172.21.15.27:0/1469652602 >> v2:172.21.15.155:3300/0 conn(0x7f3b341534c0 msgr2=0x7f3b34153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:12:27.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.842+0000 7f3b33400640 1 --2- 172.21.15.27:0/1469652602 >> v2:172.21.15.155:3300/0 conn(0x7f3b341534c0 0x7f3b34153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:27.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.842+0000 7f3b32a00640 1 -- 172.21.15.27:0/1469652602 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139413569 0 0) 0x7f3b340c3d90 con 0x7f3b34150c70 2024-03-28T19:12:27.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.842+0000 7f3b33400640 1 -- 172.21.15.27:0/1469652602 >> v1:172.21.15.155:6789/0 conn(0x7f3b34150c70 legacy=0x7f3b340c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:27.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.842+0000 7f3b33400640 1 -- 172.21.15.27:0/1469652602 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b340ab710 con 0x7f3b340a9cb0 2024-03-28T19:12:27.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.842+0000 7f3b38e00640 1 --2- 172.21.15.27:0/1469652602 >> v2:172.21.15.155:3300/0 conn(0x7f3b341534c0 0x7f3b34153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:12:27.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.843+0000 7f3b33400640 1 --2- 172.21.15.27:0/1469652602 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b340a9cb0 0x7f3b340c3690 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3b200097a0 tx=0x7f3b2002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=374058dce61382db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:27.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.844+0000 7f3b32a00640 1 -- 172.21.15.27:0/1469652602 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20031780 con 0x7f3b340a9cb0 2024-03-28T19:12:27.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.844+0000 7f3b32a00640 1 -- 172.21.15.27:0/1469652602 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b20037070 con 0x7f3b340a9cb0 2024-03-28T19:12:27.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.844+0000 7f3b32a00640 1 -- 172.21.15.27:0/1469652602 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20031b40 con 0x7f3b340a9cb0 2024-03-28T19:12:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.845+0000 7f3b3ae00640 1 -- 172.21.15.27:0/1469652602 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b340a9cb0 msgr2=0x7f3b340c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.845+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/1469652602 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b340a9cb0 0x7f3b340c3690 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3b200097a0 tx=0x7f3b2002b820 comp rx=0 tx=0).stop 2024-03-28T19:12:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.845+0000 7f3b3ae00640 1 -- 172.21.15.27:0/1469652602 shutdown_connections 2024-03-28T19:12:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.845+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/1469652602 >> v2:172.21.15.155:3300/0 conn(0x7f3b341534c0 0x7f3b34153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.845+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/1469652602 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b340a9cb0 0x7f3b340c3690 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.845+0000 7f3b3ae00640 1 -- 172.21.15.27:0/1469652602 >> 172.21.15.27:0/1469652602 conn(0x7f3b3406b9d0 msgr2=0x7f3b340c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.845+0000 7f3b3ae00640 1 -- 172.21.15.27:0/1469652602 shutdown_connections 2024-03-28T19:12:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.845+0000 7f3b3ae00640 1 -- 172.21.15.27:0/1469652602 wait complete. 2024-03-28T19:12:27.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.849+0000 7f3b3ae00640 1 Processor -- start 2024-03-28T19:12:27.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.849+0000 7f3b3ae00640 1 -- start start 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.850+0000 7f3b3ae00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b340a9cb0 0x7f3b34154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.850+0000 7f3b3ae00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b34150c70 0x7f3b34147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.850+0000 7f3b3ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b341534c0 0x7f3b341e9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.850+0000 7f3b3ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340c1810 con 0x7f3b341534c0 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.850+0000 7f3b3ae00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340abcd0 con 0x7f3b340a9cb0 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.850+0000 7f3b3ae00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b340c6830 con 0x7f3b34150c70 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b38e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b341534c0 0x7f3b341e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b38e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b341534c0 0x7f3b341e9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48704/0 (socket says 172.21.15.27:48704) 2024-03-28T19:12:27.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b38e00640 1 -- 172.21.15.27:0/2715288894 learned_addr learned my addr 172.21.15.27:0/2715288894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:12:27.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b33400640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b34150c70 0x7f3b34147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:27.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b33e00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b340a9cb0 0x7f3b34154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:27.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b38e00640 1 -- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b34150c70 msgr2=0x7f3b34147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:27.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b38e00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b34150c70 0x7f3b34147fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:27.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b38e00640 1 -- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b340a9cb0 msgr2=0x7f3b34154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:27.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b38e00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b340a9cb0 0x7f3b34154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:27.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.851+0000 7f3b38e00640 1 -- 172.21.15.27:0/2715288894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b340a7d80 con 0x7f3b341534c0 2024-03-28T19:12:27.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.852+0000 7f3b33400640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b34150c70 0x7f3b34147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:12:27.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.852+0000 7f3b38e00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b341534c0 0x7f3b341e9900 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009f10 tx=0x7f3b2c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.854+0000 7f3b2be00640 1 -- 172.21.15.27:0/2715288894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c00cbe0 con 0x7f3b341534c0 2024-03-28T19:12:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.854+0000 7f3b2be00640 1 -- 172.21.15.27:0/2715288894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c00cd80 con 0x7f3b341534c0 2024-03-28T19:12:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.854+0000 7f3b2be00640 1 -- 172.21.15.27:0/2715288894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c003bb0 con 0x7f3b341534c0 2024-03-28T19:12:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.854+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b340a8060 con 0x7f3b341534c0 2024-03-28T19:12:27.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.854+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b341e90d0 con 0x7f3b341534c0 2024-03-28T19:12:27.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.856+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3404f400 con 0x7f3b341534c0 2024-03-28T19:12:27.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.857+0000 7f3b2be00640 1 -- 172.21.15.27:0/2715288894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3b2c01b030 con 0x7f3b341534c0 2024-03-28T19:12:27.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.858+0000 7f3b2be00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1406bc50 0x7f3b1406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:12:27.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.858+0000 7f3b2be00640 1 -- 172.21.15.27:0/2715288894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f3b2c08b880 con 0x7f3b341534c0 2024-03-28T19:12:27.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.859+0000 7f3b33e00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1406bc50 0x7f3b1406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:12:27.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.860+0000 7f3b33e00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1406bc50 0x7f3b1406e110 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3b24000f60 tx=0x7f3b24009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:12:27.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:27.866+0000 7f3b2be00640 1 -- 172.21.15.27:0/2715288894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3b2c022020 con 0x7f3b341534c0 2024-03-28T19:12:28.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.219+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b340a8720 con 0x7f3b341534c0 2024-03-28T19:12:28.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.220+0000 7f3b2be00640 1 -- 172.21.15.27:0/2715288894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f3b2c010040 con 0x7f3b341534c0 2024-03-28T19:12:28.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:12:28.219 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":6532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:12:28.220 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 137 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.223+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1406bc50 msgr2=0x7f3b1406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.223+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1406bc50 0x7f3b1406e110 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3b24000f60 tx=0x7f3b24009040 comp rx=0 tx=0).stop 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.223+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b341534c0 msgr2=0x7f3b341e9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.223+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b341534c0 0x7f3b341e9900 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c009f10 tx=0x7f3b2c00c490 comp rx=0 tx=0).stop 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.224+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 shutdown_connections 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.224+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1406bc50 0x7f3b1406e110 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.224+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b341534c0 0x7f3b341e9900 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.224+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b34150c70 0x7f3b34147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.224+0000 7f3b3ae00640 1 --2- 172.21.15.27:0/2715288894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b340a9cb0 0x7f3b34154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.224+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 >> 172.21.15.27:0/2715288894 conn(0x7f3b3406b9d0 msgr2=0x7f3b34152720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.224+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 shutdown_connections 2024-03-28T19:12:28.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:12:28.224+0000 7f3b3ae00640 1 -- 172.21.15.27:0/2715288894 wait complete. 2024-03-28T19:12:28.248 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-03-28T19:12:28.248 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2024-03-28T19:12:28.563 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:12:28.563+0000 7f4282e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:12:32.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:12:32.278 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:13:02.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:13:02.407 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:13:32.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:13:32.530 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:13:42.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:43.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.175+0000 7f313c800640 1 Processor -- start 2024-03-28T19:13:43.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.175+0000 7f313c800640 1 -- start start 2024-03-28T19:13:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.176+0000 7f313c800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f313406eb80 0x7f313415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.176+0000 7f313c800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31340c3830 con 0x7f313406e6f0 2024-03-28T19:13:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.176+0000 7f313c800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313406ef60 con 0x7f3134153ed0 2024-03-28T19:13:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.176+0000 7f313c800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31340c6990 con 0x7f313406eb80 2024-03-28T19:13:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.176+0000 7f3139400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f313406e6f0 0x7f3134159bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35048/0 (socket says 172.21.15.27:35048) 2024-03-28T19:13:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.176+0000 7f3139400640 1 -- 172.21.15.27:0/2736733075 learned_addr learned my addr 172.21.15.27:0/2736733075 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:13:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.177+0000 7f313a800640 1 --2- 172.21.15.27:0/2736733075 >> v2:172.21.15.155:3300/0 conn(0x7f313406eb80 0x7f313415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.177+0000 7f3138a00640 1 -- 172.21.15.27:0/2736733075 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200741061 0 0) 0x7f31340c3830 con 0x7f313406e6f0 2024-03-28T19:13:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.177+0000 7f3138a00640 1 -- 172.21.15.27:0/2736733075 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f311c003520 con 0x7f313406e6f0 2024-03-28T19:13:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.177+0000 7f3138a00640 1 -- 172.21.15.27:0/2736733075 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416597387 0 0) 0x7f313406ef60 con 0x7f3134153ed0 2024-03-28T19:13:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.177+0000 7f3138a00640 1 -- 172.21.15.27:0/2736733075 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31340c3830 con 0x7f3134153ed0 2024-03-28T19:13:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.178+0000 7f313a800640 1 -- 172.21.15.27:0/2736733075 >> v1:172.21.15.155:6789/0 conn(0x7f3134153ed0 legacy=0x7f313406ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.178+0000 7f3138a00640 1 -- 172.21.15.27:0/2736733075 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803781026 0 0) 0x7f311c003520 con 0x7f313406e6f0 2024-03-28T19:13:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.178+0000 7f313a800640 1 -- 172.21.15.27:0/2736733075 >> v1:172.21.15.27:6789/0 conn(0x7f313406e6f0 legacy=0x7f3134159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.178+0000 7f313a800640 1 -- 172.21.15.27:0/2736733075 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31340c66b0 con 0x7f313406eb80 2024-03-28T19:13:43.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.179+0000 7f313a800640 1 --2- 172.21.15.27:0/2736733075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 0x7f313415a2d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f313002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5d7a0562763c6b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.180+0000 7f3138a00640 1 -- 172.21.15.27:0/2736733075 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130039070 con 0x7f313406eb80 2024-03-28T19:13:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.180+0000 7f3138a00640 1 -- 172.21.15.27:0/2736733075 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313002c940 con 0x7f313406eb80 2024-03-28T19:13:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.180+0000 7f3138a00640 1 -- 172.21.15.27:0/2736733075 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124001670 con 0x7f313406eb80 2024-03-28T19:13:43.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.181+0000 7f313c800640 1 -- 172.21.15.27:0/2736733075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 msgr2=0x7f313415a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.181+0000 7f313c800640 1 --2- 172.21.15.27:0/2736733075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 0x7f313415a2d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f313002be10 comp rx=0 tx=0).stop 2024-03-28T19:13:43.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.181+0000 7f313c800640 1 -- 172.21.15.27:0/2736733075 shutdown_connections 2024-03-28T19:13:43.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.181+0000 7f313c800640 1 --2- 172.21.15.27:0/2736733075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 0x7f313415a2d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.181+0000 7f313c800640 1 -- 172.21.15.27:0/2736733075 >> 172.21.15.27:0/2736733075 conn(0x7f313406b9d0 msgr2=0x7f31340c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.181+0000 7f313c800640 1 -- 172.21.15.27:0/2736733075 shutdown_connections 2024-03-28T19:13:43.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.181+0000 7f313c800640 1 -- 172.21.15.27:0/2736733075 wait complete. 2024-03-28T19:13:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.183+0000 7f313c800640 1 Processor -- start 2024-03-28T19:13:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.183+0000 7f313c800640 1 -- start start 2024-03-28T19:13:43.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.185+0000 7f313c800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f313406e6f0 0x7f313414f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.185+0000 7f313c800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 0x7f3134070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.185+0000 7f313c800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3134153ed0 0x7f313415d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.185+0000 7f313c800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313415a810 con 0x7f3134153ed0 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.185+0000 7f313c800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31340c3db0 con 0x7f313406eb80 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.185+0000 7f313c800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31340c6380 con 0x7f313406e6f0 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.185+0000 7f313a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3134153ed0 0x7f313415d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.185+0000 7f313a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3134153ed0 0x7f313415d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47204/0 (socket says 172.21.15.27:47204) 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f313a800640 1 -- 172.21.15.27:0/780711463 learned_addr learned my addr 172.21.15.27:0/780711463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f3139e00640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f313406e6f0 0x7f313414f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f3139400640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 0x7f3134070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f313a800640 1 -- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f313406e6f0 msgr2=0x7f313414f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f313a800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f313406e6f0 0x7f313414f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f313a800640 1 -- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 msgr2=0x7f3134070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f313a800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 0x7f3134070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f313a800640 1 -- 172.21.15.27:0/780711463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3134071460 con 0x7f3134153ed0 2024-03-28T19:13:43.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.186+0000 7f3139400640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 0x7f3134070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:13:43.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.187+0000 7f313a800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3134153ed0 0x7f313415d8e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f3130004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.187+0000 7f312e000640 1 -- 172.21.15.27:0/780711463 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130039050 con 0x7f3134153ed0 2024-03-28T19:13:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.187+0000 7f312e000640 1 -- 172.21.15.27:0/780711463 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313003a040 con 0x7f3134153ed0 2024-03-28T19:13:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.187+0000 7f312e000640 1 -- 172.21.15.27:0/780711463 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130009450 con 0x7f3134153ed0 2024-03-28T19:13:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.187+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3134071700 con 0x7f3134153ed0 2024-03-28T19:13:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.187+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31341efca0 con 0x7f3134153ed0 2024-03-28T19:13:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.189+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313404f400 con 0x7f3134153ed0 2024-03-28T19:13:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.189+0000 7f312e000640 1 -- 172.21.15.27:0/780711463 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3130047020 con 0x7f3134153ed0 2024-03-28T19:13:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.190+0000 7f312e000640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311806bbd0 0x7f311806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.191+0000 7f312e000640 1 -- 172.21.15.27:0/780711463 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f31300ad630 con 0x7f3134153ed0 2024-03-28T19:13:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.191+0000 7f3139e00640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311806bbd0 0x7f311806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.192+0000 7f3139e00640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311806bbd0 0x7f311806e090 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3128007fd0 tx=0x7f3128009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:43.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.200+0000 7f312e000640 1 -- 172.21.15.27:0/780711463 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f313007b5a0 con 0x7f3134153ed0 2024-03-28T19:13:43.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.547+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31340ac190 con 0x7f3134153ed0 2024-03-28T19:13:43.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.548+0000 7f312e000640 1 -- 172.21.15.27:0/780711463 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f313007aea0 con 0x7f3134153ed0 2024-03-28T19:13:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:43.547 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":6532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:43.547 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 137 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.551+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311806bbd0 msgr2=0x7f311806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.551+0000 7f313c800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311806bbd0 0x7f311806e090 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3128007fd0 tx=0x7f3128009040 comp rx=0 tx=0).stop 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.551+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3134153ed0 msgr2=0x7f313415d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.551+0000 7f313c800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3134153ed0 0x7f313415d8e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f3130009d90 tx=0x7f3130004a40 comp rx=0 tx=0).stop 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.552+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 shutdown_connections 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.552+0000 7f313c800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311806bbd0 0x7f311806e090 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.552+0000 7f313c800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3134153ed0 0x7f313415d8e0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.552+0000 7f313c800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313406eb80 0x7f3134070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.552+0000 7f313c800640 1 --2- 172.21.15.27:0/780711463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f313406e6f0 0x7f313414f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.552+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 >> 172.21.15.27:0/780711463 conn(0x7f313406b9d0 msgr2=0x7f31341581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.552+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 shutdown_connections 2024-03-28T19:13:43.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.552+0000 7f313c800640 1 -- 172.21.15.27:0/780711463 wait complete. 2024-03-28T19:13:43.572 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-03-28T19:13:43.572 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:13:43.572 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:13:43.599 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:13:43.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:43.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.873+0000 7f8ea7e00640 1 Processor -- start 2024-03-28T19:13:43.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.873+0000 7f8ea7e00640 1 -- start start 2024-03-28T19:13:43.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.874+0000 7f8ea7e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8ea81551a0 0x7f8ea80c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.874+0000 7f8ea7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80c5580 0x7f8ea80c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.874+0000 7f8ea7e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea80c26d0 con 0x7f8ea80aada0 2024-03-28T19:13:43.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.874+0000 7f8ea7e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea8156870 con 0x7f8ea81551a0 2024-03-28T19:13:43.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.874+0000 7f8ea7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8155cf0 con 0x7f8ea80c5580 2024-03-28T19:13:43.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.875+0000 7f8ea6a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8ea81551a0 0x7f8ea80c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.875+0000 7f8ea6a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8ea81551a0 0x7f8ea80c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43392/0 (socket says 172.21.15.27:43392) 2024-03-28T19:13:43.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.875+0000 7f8ea6a00640 1 -- 172.21.15.27:0/2394584378 learned_addr learned my addr 172.21.15.27:0/2394584378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:43.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.875+0000 7f8ea6000640 1 --2- 172.21.15.27:0/2394584378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80c5580 0x7f8ea80c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.876+0000 7f8ea6a00640 1 -- 172.21.15.27:0/2394584378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80c5580 msgr2=0x7f8ea80c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:43.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.876+0000 7f8ea6a00640 1 --2- 172.21.15.27:0/2394584378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80c5580 0x7f8ea80c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.876+0000 7f8ea6a00640 1 -- 172.21.15.27:0/2394584378 >> v1:172.21.15.155:6789/0 conn(0x7f8ea80aada0 legacy=0x7f8ea80c1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.876+0000 7f8ea6a00640 1 -- 172.21.15.27:0/2394584378 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea8155fd0 con 0x7f8ea81551a0 2024-03-28T19:13:43.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.878+0000 7f8ea6a00640 1 --2- 172.21.15.27:0/2394584378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 0x7f8ea80c5040 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009870 tx=0x7f8e9802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19b243c52636039a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:43.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.878+0000 7f8ea5600640 1 -- 172.21.15.27:0/2394584378 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e980317c0 con 0x7f8ea81551a0 2024-03-28T19:13:43.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.878+0000 7f8ea5600640 1 -- 172.21.15.27:0/2394584378 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98037070 con 0x7f8ea81551a0 2024-03-28T19:13:43.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.878+0000 7f8ea5600640 1 -- 172.21.15.27:0/2394584378 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98031b80 con 0x7f8ea81551a0 2024-03-28T19:13:43.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.879+0000 7f8ea7e00640 1 -- 172.21.15.27:0/2394584378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 msgr2=0x7f8ea80c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.879+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/2394584378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 0x7f8ea80c5040 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009870 tx=0x7f8e9802b860 comp rx=0 tx=0).stop 2024-03-28T19:13:43.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.879+0000 7f8ea7e00640 1 -- 172.21.15.27:0/2394584378 shutdown_connections 2024-03-28T19:13:43.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.879+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/2394584378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80c5580 0x7f8ea80c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.879+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/2394584378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 0x7f8ea80c5040 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.879+0000 7f8ea7e00640 1 -- 172.21.15.27:0/2394584378 >> 172.21.15.27:0/2394584378 conn(0x7f8ea806b0c0 msgr2=0x7f8ea806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:43.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.879+0000 7f8ea7e00640 1 -- 172.21.15.27:0/2394584378 shutdown_connections 2024-03-28T19:13:43.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.879+0000 7f8ea7e00640 1 -- 172.21.15.27:0/2394584378 wait complete. 2024-03-28T19:13:43.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.881+0000 7f8ea7e00640 1 Processor -- start 2024-03-28T19:13:43.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.881+0000 7f8ea7e00640 1 -- start start 2024-03-28T19:13:43.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80aada0 0x7f8ea80a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea80c5580 0x7f8ea814a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 0x7f8ea8146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea81559a0 con 0x7f8ea80c5580 2024-03-28T19:13:43.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8156590 con 0x7f8ea81551a0 2024-03-28T19:13:43.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8155580 con 0x7f8ea80aada0 2024-03-28T19:13:43.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea80c5580 0x7f8ea814a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea80c5580 0x7f8ea814a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47214/0 (socket says 172.21.15.27:47214) 2024-03-28T19:13:43.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea6000640 1 -- 172.21.15.27:0/274455719 learned_addr learned my addr 172.21.15.27:0/274455719 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:43.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea6a00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80aada0 0x7f8ea80a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.882+0000 7f8ea7400640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 0x7f8ea8146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.883+0000 7f8ea6000640 1 -- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80aada0 msgr2=0x7f8ea80a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.883+0000 7f8ea6000640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80aada0 0x7f8ea80a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.883+0000 7f8ea6000640 1 -- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 msgr2=0x7f8ea8146cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:43.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.883+0000 7f8ea6000640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 0x7f8ea8146cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:43.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.883+0000 7f8ea6000640 1 -- 172.21.15.27:0/274455719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea81ec350 con 0x7f8ea80c5580 2024-03-28T19:13:43.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.883+0000 7f8ea6a00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80aada0 0x7f8ea80a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:13:43.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.883+0000 7f8ea6000640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea80c5580 0x7f8ea814a010 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400e9b0 tx=0x7f8e9400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:43.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.884+0000 7f8e9ea00640 1 -- 172.21.15.27:0/274455719 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94017070 con 0x7f8ea80c5580 2024-03-28T19:13:43.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.884+0000 7f8e9ea00640 1 -- 172.21.15.27:0/274455719 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9400c9a0 con 0x7f8ea80c5580 2024-03-28T19:13:43.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.884+0000 7f8e9ea00640 1 -- 172.21.15.27:0/274455719 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400cd90 con 0x7f8ea80c5580 2024-03-28T19:13:43.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.884+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea81ef360 con 0x7f8ea80c5580 2024-03-28T19:13:43.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.884+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea81500c0 con 0x7f8ea80c5580 2024-03-28T19:13:43.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.885+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea8067dd0 con 0x7f8ea80c5580 2024-03-28T19:13:43.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.886+0000 7f8e9ea00640 1 -- 172.21.15.27:0/274455719 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8e9401a070 con 0x7f8ea80c5580 2024-03-28T19:13:43.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.887+0000 7f8e9ea00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e8806bc00 0x7f8e8806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:43.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.887+0000 7f8e9ea00640 1 -- 172.21.15.27:0/274455719 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f8e9408b610 con 0x7f8ea80c5580 2024-03-28T19:13:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.888+0000 7f8ea6a00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e8806bc00 0x7f8e8806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.888+0000 7f8ea6a00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e8806bc00 0x7f8e8806e0c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8e98036000 tx=0x7f8e9803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:43.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:43.892+0000 7f8e9ea00640 1 -- 172.21.15.27:0/274455719 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8e94059580 con 0x7f8ea80c5580 2024-03-28T19:13:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.149+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ea8150290 con 0x7f8ea80c5580 2024-03-28T19:13:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.150+0000 7f8e9ea00640 1 -- 172.21.15.27:0/274455719 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f8e94058e80 con 0x7f8ea80c5580 2024-03-28T19:13:44.149 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:44.150 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":6532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 137 2024-03-28T19:13:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e8806bc00 msgr2=0x7f8e8806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e8806bc00 0x7f8e8806e0c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8e98036000 tx=0x7f8e9803d040 comp rx=0 tx=0).stop 2024-03-28T19:13:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea80c5580 msgr2=0x7f8ea814a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:44.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea80c5580 0x7f8ea814a010 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400e9b0 tx=0x7f8e9400ee70 comp rx=0 tx=0).stop 2024-03-28T19:13:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 shutdown_connections 2024-03-28T19:13:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8e8806bc00 0x7f8e8806e0c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ea81551a0 0x7f8ea8146cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ea80c5580 0x7f8ea814a010 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 --2- 172.21.15.27:0/274455719 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ea80aada0 0x7f8ea80a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 >> 172.21.15.27:0/274455719 conn(0x7f8ea806b0c0 msgr2=0x7f8ea806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 shutdown_connections 2024-03-28T19:13:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:44.154+0000 7f8ea7e00640 1 -- 172.21.15.27:0/274455719 wait complete. 2024-03-28T19:13:44.173 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:13:46.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.417+0000 7fa4fee00640 1 Processor -- start 2024-03-28T19:13:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.417+0000 7fa4fee00640 1 -- start start 2024-03-28T19:13:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.419+0000 7fa4fee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4f814f6d0 0x7fa4f8159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.419+0000 7fa4fee00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa4f8072c60 0x7fa4f80c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.419+0000 7fa4fee00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f8072870 con 0x7fa4f8071cf0 2024-03-28T19:13:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.419+0000 7fa4fee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f8156640 con 0x7fa4f814f6d0 2024-03-28T19:13:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.419+0000 7fa4fee00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f8158db0 con 0x7fa4f8072c60 2024-03-28T19:13:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.419+0000 7fa4f7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa4f8071cf0 0x7fa4f80c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37124/0 (socket says 172.21.15.27:37124) 2024-03-28T19:13:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.419+0000 7fa4f7400640 1 -- 172.21.15.27:0/3723037745 learned_addr learned my addr 172.21.15.27:0/3723037745 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:13:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.419+0000 7fa4f7e00640 1 --2- 172.21.15.27:0/3723037745 >> v2:172.21.15.27:3300/0 conn(0x7fa4f814f6d0 0x7fa4f8159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.420+0000 7fa4fce00640 1 --2- 172.21.15.27:0/3723037745 >> v2:172.21.15.155:3300/0 conn(0x7fa4f8072c60 0x7fa4f80c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.420+0000 7fa4f6a00640 1 -- 172.21.15.27:0/3723037745 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820378163 0 0) 0x7fa4e0001110 con 0x7fa4f8071cf0 2024-03-28T19:13:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.420+0000 7fa4f6a00640 1 -- 172.21.15.27:0/3723037745 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc003520 con 0x7fa4f8071cf0 2024-03-28T19:13:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.420+0000 7fa4f7e00640 1 -- 172.21.15.27:0/3723037745 >> v2:172.21.15.155:3300/0 conn(0x7fa4f8072c60 msgr2=0x7fa4f80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.420+0000 7fa4f7e00640 1 --2- 172.21.15.27:0/3723037745 >> v2:172.21.15.155:3300/0 conn(0x7fa4f8072c60 0x7fa4f80c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.420+0000 7fa4f7e00640 1 -- 172.21.15.27:0/3723037745 >> v1:172.21.15.27:6789/0 conn(0x7fa4f8071cf0 legacy=0x7fa4f80c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.420+0000 7fa4f7e00640 1 -- 172.21.15.27:0/3723037745 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8156360 con 0x7fa4f814f6d0 2024-03-28T19:13:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.421+0000 7fa4f7e00640 1 --2- 172.21.15.27:0/3723037745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f8159850 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80c28a0 tx=0x7fa4e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f46c67ae24075a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:46.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.421+0000 7fa4f6a00640 1 -- 172.21.15.27:0/3723037745 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e80315a0 con 0x7fa4f814f6d0 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.421+0000 7fa4f6a00640 1 -- 172.21.15.27:0/3723037745 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e8037070 con 0x7fa4f814f6d0 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.421+0000 7fa4f6a00640 1 -- 172.21.15.27:0/3723037745 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8031960 con 0x7fa4f814f6d0 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.422+0000 7fa4fee00640 1 -- 172.21.15.27:0/3723037745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 msgr2=0x7fa4f8159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.422+0000 7fa4fee00640 1 --2- 172.21.15.27:0/3723037745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f8159850 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80c28a0 tx=0x7fa4e802b840 comp rx=0 tx=0).stop 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.422+0000 7fa4fee00640 1 -- 172.21.15.27:0/3723037745 shutdown_connections 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.422+0000 7fa4fee00640 1 --2- 172.21.15.27:0/3723037745 >> v2:172.21.15.155:3300/0 conn(0x7fa4f8072c60 0x7fa4f80c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.422+0000 7fa4fee00640 1 --2- 172.21.15.27:0/3723037745 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f8159850 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.422+0000 7fa4fee00640 1 -- 172.21.15.27:0/3723037745 >> 172.21.15.27:0/3723037745 conn(0x7fa4f806c5f0 msgr2=0x7fa4f806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.422+0000 7fa4fee00640 1 -- 172.21.15.27:0/3723037745 shutdown_connections 2024-03-28T19:13:46.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.422+0000 7fa4fee00640 1 -- 172.21.15.27:0/3723037745 wait complete. 2024-03-28T19:13:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.424+0000 7fa4fee00640 1 Processor -- start 2024-03-28T19:13:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.424+0000 7fa4fee00640 1 -- start start 2024-03-28T19:13:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.425+0000 7fa4fee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4f8071cf0 0x7fa4f81eb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.425+0000 7fa4fee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4f8072c60 0x7fa4f814de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.425+0000 7fa4fee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f81eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:46.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.425+0000 7fa4fee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f814fab0 con 0x7fa4f814f6d0 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.425+0000 7fa4fee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f8159330 con 0x7fa4f8071cf0 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.425+0000 7fa4fee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f80c3c70 con 0x7fa4f8072c60 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f81eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4f7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4f8071cf0 0x7fa4f81eb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f81eee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54588/0 (socket says 172.21.15.27:54588) 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 -- 172.21.15.27:0/1739788987 learned_addr learned my addr 172.21.15.27:0/1739788987 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4f7400640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4f8072c60 0x7fa4f814de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 -- 172.21.15.27:0/1739788987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4f8072c60 msgr2=0x7fa4f814de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4f8072c60 0x7fa4f814de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 -- 172.21.15.27:0/1739788987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4f8071cf0 msgr2=0x7fa4f81eb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4f8071cf0 0x7fa4f81eb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 -- 172.21.15.27:0/1739788987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80aca50 con 0x7fa4f814f6d0 2024-03-28T19:13:46.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.426+0000 7fa4fce00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f81eee00 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0007cf0 tx=0x7fa4f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.427+0000 7fa4efe00640 1 -- 172.21.15.27:0/1739788987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0014420 con 0x7fa4f814f6d0 2024-03-28T19:13:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.427+0000 7fa4efe00640 1 -- 172.21.15.27:0/1739788987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f0019070 con 0x7fa4f814f6d0 2024-03-28T19:13:46.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.427+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f80acd30 con 0x7fa4f814f6d0 2024-03-28T19:13:46.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.427+0000 7fa4efe00640 1 -- 172.21.15.27:0/1739788987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0014820 con 0x7fa4f814f6d0 2024-03-28T19:13:46.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.427+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f814e3b0 con 0x7fa4f814f6d0 2024-03-28T19:13:46.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.428+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f80ad1a0 con 0x7fa4f814f6d0 2024-03-28T19:13:46.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.429+0000 7fa4efe00640 1 -- 172.21.15.27:0/1739788987 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa4f00149c0 con 0x7fa4f814f6d0 2024-03-28T19:13:46.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.430+0000 7fa4efe00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa4d406bbd0 0x7fa4d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:46.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.430+0000 7fa4efe00640 1 -- 172.21.15.27:0/1739788987 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7fa4f008bce0 con 0x7fa4f814f6d0 2024-03-28T19:13:46.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.430+0000 7fa4f7e00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa4d406bbd0 0x7fa4d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:46.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.431+0000 7fa4f7e00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa4d406bbd0 0x7fa4d406e090 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8036000 tx=0x7fa4e803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:46.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.435+0000 7fa4efe00640 1 -- 172.21.15.27:0/1739788987 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa4f80ad1a0 con 0x7fa4f814f6d0 2024-03-28T19:13:46.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.688+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4f804f400 con 0x7fa4f814f6d0 2024-03-28T19:13:46.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.690+0000 7fa4efe00640 1 -- 172.21.15.27:0/1739788987 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fa4f0059c50 con 0x7fa4f814f6d0 2024-03-28T19:13:46.690 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:46.690 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":6532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:46.690 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 137 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa4d406bbd0 msgr2=0x7fa4d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa4d406bbd0 0x7fa4d406e090 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8036000 tx=0x7fa4e803d040 comp rx=0 tx=0).stop 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 msgr2=0x7fa4f81eee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f81eee00 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0007cf0 tx=0x7fa4f000e4b0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 shutdown_connections 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa4d406bbd0 0x7fa4d406e090 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4f814f6d0 0x7fa4f81eee00 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4f8072c60 0x7fa4f814de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 --2- 172.21.15.27:0/1739788987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4f8071cf0 0x7fa4f81eb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.693+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 >> 172.21.15.27:0/1739788987 conn(0x7fa4f806c5f0 msgr2=0x7fa4f80a9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:46.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.694+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 shutdown_connections 2024-03-28T19:13:46.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:46.694+0000 7fa4fee00640 1 -- 172.21.15.27:0/1739788987 wait complete. 2024-03-28T19:13:46.711 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:13:48.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:48.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.950+0000 7f4894e00640 1 Processor -- start 2024-03-28T19:13:48.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.951+0000 7f4894e00640 1 -- start start 2024-03-28T19:13:48.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.952+0000 7f4894e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f48900c3780 0x7f4890160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.952+0000 7f4894e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4890159630 con 0x7f48900c3e50 2024-03-28T19:13:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.952+0000 7f4894e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4890159960 con 0x7f48901562f0 2024-03-28T19:13:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.952+0000 7f4894e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48900c1760 con 0x7f48900c3780 2024-03-28T19:13:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.952+0000 7f488ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f48900c3e50 0x7f48900c3070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37132/0 (socket says 172.21.15.27:37132) 2024-03-28T19:13:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.952+0000 7f488ea00640 1 -- 172.21.15.27:0/4092309945 learned_addr learned my addr 172.21.15.27:0/4092309945 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:13:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488fe00640 1 --2- 172.21.15.27:0/4092309945 >> v2:172.21.15.155:3300/0 conn(0x7f48900c3780 0x7f4890160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631712243 0 0) 0x7f4890159630 con 0x7f48900c3e50 2024-03-28T19:13:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4874003520 con 0x7f48900c3e50 2024-03-28T19:13:48.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636326344 0 0) 0x7f4874003520 con 0x7f48900c3e50 2024-03-28T19:13:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 >> v2:172.21.15.155:3300/0 conn(0x7f48900c3780 msgr2=0x7f4890160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 --2- 172.21.15.27:0/4092309945 >> v2:172.21.15.155:3300/0 conn(0x7f48900c3780 0x7f4890160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 >> v1:172.21.15.155:6789/0 conn(0x7f48901562f0 legacy=0x7f489006bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48901566d0 con 0x7f48900c3e50 2024-03-28T19:13:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f4878001670 con 0x7f48900c3e50 2024-03-28T19:13:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.953+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 >> v1:172.21.15.27:6789/0 conn(0x7f48900c3e50 legacy=0x7f48900c3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:48.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488e000640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4874003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488e000640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4874004800 0x7f4874004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488e000640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4874005230 0x7f487400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900c1760 con 0x7f4874003a00 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487400c190 con 0x7f4874005230 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487400c390 con 0x7f4874004800 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488f400640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4874003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488fe00640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4874005230 0x7f487400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488ea00640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4874004800 0x7f4874004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:48.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488f400640 1 -- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4874004800 msgr2=0x7f4874004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488f400640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4874004800 0x7f4874004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488f400640 1 -- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4874005230 msgr2=0x7f487400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488f400640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4874005230 0x7f487400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488f400640 1 -- 172.21.15.27:0/4092309945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4874003520 con 0x7f4874003a00 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488fe00640 1 -- 172.21.15.27:0/4092309945 reap_dead start 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.954+0000 7f488ea00640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4874004800 0x7f4874004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.956+0000 7f488f400640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4874003e40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f4880001e50 tx=0x7f4874005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8add5d226aa7ed75 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.956+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880001a70 con 0x7f4874003a00 2024-03-28T19:13:48.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.956+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880003b70 con 0x7f4874003a00 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.956+0000 7f488e000640 1 -- 172.21.15.27:0/4092309945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800050d0 con 0x7f4874003a00 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.956+0000 7f4894e00640 1 -- 172.21.15.27:0/4092309945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 msgr2=0x7f4874003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.956+0000 7f4894e00640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4874003e40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f4880001e50 tx=0x7f4874005830 comp rx=0 tx=0).stop 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.957+0000 7f4894e00640 1 -- 172.21.15.27:0/4092309945 shutdown_connections 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.957+0000 7f4894e00640 1 --2- 172.21.15.27:0/4092309945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4874003e40 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.957+0000 7f4894e00640 1 -- 172.21.15.27:0/4092309945 >> 172.21.15.27:0/4092309945 conn(0x7f489006b0c0 msgr2=0x7f489006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.957+0000 7f4894e00640 1 -- 172.21.15.27:0/4092309945 shutdown_connections 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.957+0000 7f4894e00640 1 -- 172.21.15.27:0/4092309945 wait complete. 2024-03-28T19:13:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.959+0000 7f4894e00640 1 Processor -- start 2024-03-28T19:13:48.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.959+0000 7f4894e00640 1 -- start start 2024-03-28T19:13:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f4894e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4890157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f4894e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48900aace0 0x7f489006dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f4894e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4890157e80 0x7f489006e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f4894e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900c1480 con 0x7f4874003a00 2024-03-28T19:13:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f4894e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890067cc0 con 0x7f4890157e80 2024-03-28T19:13:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f4894e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48900c19e0 con 0x7f48900aace0 2024-03-28T19:13:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4890157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4890157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54594/0 (socket says 172.21.15.27:54594) 2024-03-28T19:13:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488f400640 1 -- 172.21.15.27:0/481952141 learned_addr learned my addr 172.21.15.27:0/481952141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488fe00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4890157e80 0x7f489006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488ea00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48900aace0 0x7f489006dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488f400640 1 -- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48900aace0 msgr2=0x7f489006dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488f400640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48900aace0 0x7f489006dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488f400640 1 -- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4890157e80 msgr2=0x7f489006e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488f400640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4890157e80 0x7f489006e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488f400640 1 -- 172.21.15.27:0/481952141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48900ad7f0 con 0x7f4874003a00 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.960+0000 7f488fe00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4890157e80 0x7f489006e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.961+0000 7f488f400640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4890157940 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f4880001eb0 tx=0x7f48800014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.961+0000 7f4887400640 1 -- 172.21.15.27:0/481952141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880006d60 con 0x7f4874003a00 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.961+0000 7f4887400640 1 -- 172.21.15.27:0/481952141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880005480 con 0x7f4874003a00 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.961+0000 7f4887400640 1 -- 172.21.15.27:0/481952141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880005830 con 0x7f4874003a00 2024-03-28T19:13:48.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.961+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489006eed0 con 0x7f4874003a00 2024-03-28T19:13:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.961+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489006f470 con 0x7f4874003a00 2024-03-28T19:13:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.962+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4890062890 con 0x7f4874003a00 2024-03-28T19:13:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.963+0000 7f4887400640 1 -- 172.21.15.27:0/481952141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4880007030 con 0x7f4874003a00 2024-03-28T19:13:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.965+0000 7f4887400640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f486c06bc00 0x7f486c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.965+0000 7f4887400640 1 -- 172.21.15.27:0/481952141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f48800ad6e0 con 0x7f4874003a00 2024-03-28T19:13:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.965+0000 7f488ea00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f486c06bc00 0x7f486c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.966+0000 7f488ea00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f486c06bc00 0x7f486c06e0c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f48900c6a90 tx=0x7f4878009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:48.969+0000 7f4887400640 1 -- 172.21.15.27:0/481952141 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f488007b650 con 0x7f4874003a00 2024-03-28T19:13:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.220+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f489006f640 con 0x7f4874003a00 2024-03-28T19:13:49.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.221+0000 7f4887400640 1 -- 172.21.15.27:0/481952141 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f488007af50 con 0x7f4874003a00 2024-03-28T19:13:49.221 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:49.221 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":6532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 137 2024-03-28T19:13:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f486c06bc00 msgr2=0x7f486c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f486c06bc00 0x7f486c06e0c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f48900c6a90 tx=0x7f4878009040 comp rx=0 tx=0).stop 2024-03-28T19:13:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 msgr2=0x7f4890157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4890157940 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f4880001eb0 tx=0x7f48800014a0 comp rx=0 tx=0).stop 2024-03-28T19:13:49.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 shutdown_connections 2024-03-28T19:13:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f486c06bc00 0x7f486c06e0c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4890157e80 0x7f489006e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48900aace0 0x7f489006dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 --2- 172.21.15.27:0/481952141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874003a00 0x7f4890157940 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 >> 172.21.15.27:0/481952141 conn(0x7f489006b0c0 msgr2=0x7f48900c5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.224+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 shutdown_connections 2024-03-28T19:13:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:49.225+0000 7f4894e00640 1 -- 172.21.15.27:0/481952141 wait complete. 2024-03-28T19:13:49.244 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:13:51.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:51.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.484+0000 7fea73200640 1 Processor -- start 2024-03-28T19:13:51.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.484+0000 7fea73200640 1 -- start start 2024-03-28T19:13:51.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea73200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea741551a0 0x7fea740c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:51.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea73200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea740aada0 0x7fea740c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:51.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea73200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740c26d0 con 0x7fea740c5580 2024-03-28T19:13:51.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea73200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea74156870 con 0x7fea740aada0 2024-03-28T19:13:51.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea73200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea74155cf0 con 0x7fea741551a0 2024-03-28T19:13:51.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea72800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea740aada0 0x7fea740c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:51.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea72800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea740aada0 0x7fea740c2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54604/0 (socket says 172.21.15.27:54604) 2024-03-28T19:13:51.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea72800640 1 -- 172.21.15.27:0/3822682661 learned_addr learned my addr 172.21.15.27:0/3822682661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:51.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.486+0000 7fea71e00640 1 --2- 172.21.15.27:0/3822682661 >> v2:172.21.15.155:3300/0 conn(0x7fea741551a0 0x7fea740c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:51.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.487+0000 7fea72800640 1 -- 172.21.15.27:0/3822682661 >> v2:172.21.15.155:3300/0 conn(0x7fea741551a0 msgr2=0x7fea740c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:51.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.487+0000 7fea72800640 1 --2- 172.21.15.27:0/3822682661 >> v2:172.21.15.155:3300/0 conn(0x7fea741551a0 0x7fea740c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.487+0000 7fea72800640 1 -- 172.21.15.27:0/3822682661 >> v1:172.21.15.155:6789/0 conn(0x7fea740c5580 legacy=0x7fea740c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:51.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.487+0000 7fea72800640 1 -- 172.21.15.27:0/3822682661 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74155fd0 con 0x7fea740aada0 2024-03-28T19:13:51.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.488+0000 7fea72800640 1 --2- 172.21.15.27:0/3822682661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea740aada0 0x7fea740c2190 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fea7406d290 tx=0x7fea6c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9fe8f7f072fcec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:51.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea70a00640 1 -- 172.21.15.27:0/3822682661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c02c9f0 con 0x7fea740aada0 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea70a00640 1 -- 172.21.15.27:0/3822682661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6c02cb90 con 0x7fea740aada0 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea70a00640 1 -- 172.21.15.27:0/3822682661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c0343f0 con 0x7fea740aada0 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea73200640 1 -- 172.21.15.27:0/3822682661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea740aada0 msgr2=0x7fea740c2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea73200640 1 --2- 172.21.15.27:0/3822682661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea740aada0 0x7fea740c2190 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fea7406d290 tx=0x7fea6c02be50 comp rx=0 tx=0).stop 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea73200640 1 -- 172.21.15.27:0/3822682661 shutdown_connections 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea73200640 1 --2- 172.21.15.27:0/3822682661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea740aada0 0x7fea740c2190 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea73200640 1 --2- 172.21.15.27:0/3822682661 >> v2:172.21.15.155:3300/0 conn(0x7fea741551a0 0x7fea740c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.489+0000 7fea73200640 1 -- 172.21.15.27:0/3822682661 >> 172.21.15.27:0/3822682661 conn(0x7fea7406b0c0 msgr2=0x7fea7406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.490+0000 7fea73200640 1 -- 172.21.15.27:0/3822682661 shutdown_connections 2024-03-28T19:13:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.490+0000 7fea73200640 1 -- 172.21.15.27:0/3822682661 wait complete. 2024-03-28T19:13:51.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.491+0000 7fea73200640 1 Processor -- start 2024-03-28T19:13:51.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.491+0000 7fea73200640 1 -- start start 2024-03-28T19:13:51.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.492+0000 7fea73200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea740aada0 0x7fea740a9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.492+0000 7fea73200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea740c5580 0x7fea740bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.492+0000 7fea73200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea741551a0 0x7fea741eb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.492+0000 7fea73200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea741559a0 con 0x7fea741551a0 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.492+0000 7fea73200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea74156590 con 0x7fea740c5580 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.492+0000 7fea73200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea74155580 con 0x7fea740aada0 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea741551a0 0x7fea741eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea741551a0 0x7fea741eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54620/0 (socket says 172.21.15.27:54620) 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea71e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea740aada0 0x7fea740a9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea72800640 1 -- 172.21.15.27:0/2963940073 learned_addr learned my addr 172.21.15.27:0/2963940073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea71400640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea740c5580 0x7fea740bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea72800640 1 -- 172.21.15.27:0/2963940073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea740aada0 msgr2=0x7fea740a9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea72800640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea740aada0 0x7fea740a9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea72800640 1 -- 172.21.15.27:0/2963940073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea740c5580 msgr2=0x7fea740bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea72800640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea740c5580 0x7fea740bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.493+0000 7fea72800640 1 -- 172.21.15.27:0/2963940073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740a9970 con 0x7fea741551a0 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.494+0000 7fea72800640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea741551a0 0x7fea741eb400 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009f40 tx=0x7fea6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.494+0000 7fea6a000640 1 -- 172.21.15.27:0/2963940073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c044020 con 0x7fea741551a0 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.494+0000 7fea6a000640 1 -- 172.21.15.27:0/2963940073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6c040470 con 0x7fea741551a0 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.494+0000 7fea6a000640 1 -- 172.21.15.27:0/2963940073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c040860 con 0x7fea741551a0 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.494+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea741eeb10 con 0x7fea741551a0 2024-03-28T19:13:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.494+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea740c04d0 con 0x7fea741551a0 2024-03-28T19:13:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.496+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea74067dd0 con 0x7fea741551a0 2024-03-28T19:13:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.496+0000 7fea6a000640 1 -- 172.21.15.27:0/2963940073 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fea6c047020 con 0x7fea741551a0 2024-03-28T19:13:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.497+0000 7fea6a000640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5406bc00 0x7fea5406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.497+0000 7fea6a000640 1 -- 172.21.15.27:0/2963940073 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7fea6c0adf20 con 0x7fea741551a0 2024-03-28T19:13:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.498+0000 7fea71e00640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5406bc00 0x7fea5406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.498+0000 7fea71e00640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5406bc00 0x7fea5406e0c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fea64005be0 tx=0x7fea6400b750 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.503+0000 7fea6a000640 1 -- 172.21.15.27:0/2963940073 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fea6c04e030 con 0x7fea741551a0 2024-03-28T19:13:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.756+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea740aabc0 con 0x7fea741551a0 2024-03-28T19:13:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.757+0000 7fea6a000640 1 -- 172.21.15.27:0/2963940073 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fea740aabc0 con 0x7fea741551a0 2024-03-28T19:13:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:51.758 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":6532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:51.758 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 137 2024-03-28T19:13:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5406bc00 msgr2=0x7fea5406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5406bc00 0x7fea5406e0c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fea64005be0 tx=0x7fea6400b750 comp rx=0 tx=0).stop 2024-03-28T19:13:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea741551a0 msgr2=0x7fea741eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea741551a0 0x7fea741eb400 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009f40 tx=0x7fea6c004a40 comp rx=0 tx=0).stop 2024-03-28T19:13:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 shutdown_connections 2024-03-28T19:13:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fea5406bc00 0x7fea5406e0c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea741551a0 0x7fea741eb400 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea740c5580 0x7fea740bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.759+0000 7fea73200640 1 --2- 172.21.15.27:0/2963940073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea740aada0 0x7fea740a9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.760+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 >> 172.21.15.27:0/2963940073 conn(0x7fea7406b0c0 msgr2=0x7fea74153930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.760+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 shutdown_connections 2024-03-28T19:13:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:51.760+0000 7fea73200640 1 -- 172.21.15.27:0/2963940073 wait complete. 2024-03-28T19:13:51.781 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:13:53.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.021+0000 7f390bc00640 1 Processor -- start 2024-03-28T19:13:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.021+0000 7f390bc00640 1 -- start start 2024-03-28T19:13:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.022+0000 7f390bc00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f39040a9cb0 0x7f39040c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.022+0000 7f390bc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3904154020 con 0x7f39041534c0 2024-03-28T19:13:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.022+0000 7f390bc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39040c3d90 con 0x7f3904150c70 2024-03-28T19:13:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.022+0000 7f390bc00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39040ab4e0 con 0x7f39040a9cb0 2024-03-28T19:13:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.023+0000 7f3909c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f39041534c0 0x7f3904153910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37134/0 (socket says 172.21.15.27:37134) 2024-03-28T19:13:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.023+0000 7f3909c00640 1 -- 172.21.15.27:0/672307429 learned_addr learned my addr 172.21.15.27:0/672307429 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:13:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.023+0000 7f3903e00640 1 --2- 172.21.15.27:0/672307429 >> v2:172.21.15.155:3300/0 conn(0x7f39040a9cb0 0x7f39040c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.023+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192602533 0 0) 0x7f3904154020 con 0x7f39041534c0 2024-03-28T19:13:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.023+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ec003520 con 0x7f39041534c0 2024-03-28T19:13:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2873939092 0 0) 0x7f38ec003520 con 0x7f39041534c0 2024-03-28T19:13:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 >> v2:172.21.15.155:3300/0 conn(0x7f39040a9cb0 msgr2=0x7f39040c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 --2- 172.21.15.27:0/672307429 >> v2:172.21.15.155:3300/0 conn(0x7f39040a9cb0 0x7f39040c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 >> v1:172.21.15.155:6789/0 conn(0x7f3904150c70 legacy=0x7f39040c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040ab710 con 0x7f39041534c0 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f38fc0019f0 con 0x7f39041534c0 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 >> v1:172.21.15.27:6789/0 conn(0x7f39041534c0 legacy=0x7f3904153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec0039b0 0x7f38ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec004800 0x7f38ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38ec0051c0 0x7f38ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec00c130 con 0x7f38ec0039b0 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec00c2e0 con 0x7f38ec004800 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec00c4e0 con 0x7f38ec0051c0 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3909200640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec0039b0 0x7f38ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.024+0000 7f3903e00640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec004800 0x7f38ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.025+0000 7f3909c00640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38ec0051c0 0x7f38ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.025+0000 7f3909200640 1 -- 172.21.15.27:0/672307429 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38ec0051c0 msgr2=0x7f38ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.025+0000 7f3909200640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38ec0051c0 0x7f38ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.025+0000 7f3909200640 1 -- 172.21.15.27:0/672307429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec004800 msgr2=0x7f38ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.025+0000 7f3909200640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec004800 0x7f38ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.025+0000 7f3909200640 1 -- 172.21.15.27:0/672307429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec003520 con 0x7f38ec0039b0 2024-03-28T19:13:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.025+0000 7f3909c00640 1 -- 172.21.15.27:0/672307429 reap_dead start 2024-03-28T19:13:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.026+0000 7f3909200640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec0039b0 0x7f38ec003dd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f38f4001750 tx=0x7f38ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8072324990f4421 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.026+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f4001370 con 0x7f38ec0039b0 2024-03-28T19:13:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.026+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f40052d0 con 0x7f38ec0039b0 2024-03-28T19:13:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.026+0000 7f3903400640 1 -- 172.21.15.27:0/672307429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f4003830 con 0x7f38ec0039b0 2024-03-28T19:13:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.027+0000 7f390bc00640 1 -- 172.21.15.27:0/672307429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec0039b0 msgr2=0x7f38ec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.027+0000 7f390bc00640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec0039b0 0x7f38ec003dd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f38f4001750 tx=0x7f38ec005830 comp rx=0 tx=0).stop 2024-03-28T19:13:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.027+0000 7f390bc00640 1 -- 172.21.15.27:0/672307429 shutdown_connections 2024-03-28T19:13:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.027+0000 7f390bc00640 1 --2- 172.21.15.27:0/672307429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec0039b0 0x7f38ec003dd0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.027+0000 7f390bc00640 1 -- 172.21.15.27:0/672307429 >> 172.21.15.27:0/672307429 conn(0x7f390406b9d0 msgr2=0x7f39040c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.027+0000 7f390bc00640 1 -- 172.21.15.27:0/672307429 shutdown_connections 2024-03-28T19:13:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.027+0000 7f390bc00640 1 -- 172.21.15.27:0/672307429 wait complete. 2024-03-28T19:13:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.029+0000 7f390bc00640 1 Processor -- start 2024-03-28T19:13:54.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.029+0000 7f390bc00640 1 -- start start 2024-03-28T19:13:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f390bc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec0039b0 0x7f39041eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f390bc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3904072100 0x7f3904144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f390bc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3904145470 0x7f39041553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f390bc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39040abcd0 con 0x7f3904145470 2024-03-28T19:13:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f390bc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39040ac1f0 con 0x7f38ec0039b0 2024-03-28T19:13:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f390bc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040c1810 con 0x7f3904072100 2024-03-28T19:13:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f3909c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3904145470 0x7f39041553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f3909c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3904145470 0x7f39041553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54642/0 (socket says 172.21.15.27:54642) 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f3909c00640 1 -- 172.21.15.27:0/3753917609 learned_addr learned my addr 172.21.15.27:0/3753917609 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f3903e00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3904072100 0x7f3904144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.030+0000 7f3909200640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec0039b0 0x7f39041eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.031+0000 7f3909c00640 1 -- 172.21.15.27:0/3753917609 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3904072100 msgr2=0x7f3904144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.031+0000 7f3909c00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3904072100 0x7f3904144f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.031+0000 7f3909c00640 1 -- 172.21.15.27:0/3753917609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec0039b0 msgr2=0x7f39041eb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.031+0000 7f3909c00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec0039b0 0x7f39041eb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.031+0000 7f3909c00640 1 -- 172.21.15.27:0/3753917609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904071230 con 0x7f3904145470 2024-03-28T19:13:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.031+0000 7f3909c00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3904145470 0x7f39041553f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f38fc005fd0 tx=0x7f38fc007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.031+0000 7f3900c00640 1 -- 172.21.15.27:0/3753917609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00ea50 con 0x7f3904145470 2024-03-28T19:13:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.032+0000 7f3900c00640 1 -- 172.21.15.27:0/3753917609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc00ebf0 con 0x7f3904145470 2024-03-28T19:13:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.032+0000 7f3900c00640 1 -- 172.21.15.27:0/3753917609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00ede0 con 0x7f3904145470 2024-03-28T19:13:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.032+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390406eaa0 con 0x7f3904145470 2024-03-28T19:13:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.032+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39041ebd10 con 0x7f3904145470 2024-03-28T19:13:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.033+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390404f400 con 0x7f3904145470 2024-03-28T19:13:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.034+0000 7f3900c00640 1 -- 172.21.15.27:0/3753917609 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f38fc01a020 con 0x7f3904145470 2024-03-28T19:13:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.035+0000 7f3900c00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38e806bbd0 0x7f38e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.035+0000 7f3900c00640 1 -- 172.21.15.27:0/3753917609 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8615+0+0 (secure 0 0 0) 0x7f38fc08af50 con 0x7f3904145470 2024-03-28T19:13:54.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.035+0000 7f3909200640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38e806bbd0 0x7f38e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:54.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.036+0000 7f3909200640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38e806bbd0 0x7f38e806e090 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f38f4033000 tx=0x7f38f4009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:54.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.039+0000 7f3900c00640 1 -- 172.21.15.27:0/3753917609 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f38fc021020 con 0x7f3904145470 2024-03-28T19:13:54.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.287+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3904071400 con 0x7f3904145470 2024-03-28T19:13:54.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.289+0000 7f3900c00640 1 -- 172.21.15.27:0/3753917609 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f38fc058ec0 con 0x7f3904145470 2024-03-28T19:13:54.289 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:54.289 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":6532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:12:20.796615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6834":{"gid":6834,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.27:6835/2858357568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2858357568},{"type":"v1","addr":"172.21.15.27:6835","nonce":2858357568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6834,"qdb_cluster":[6834]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:54.289 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 137 2024-03-28T19:13:54.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38e806bbd0 msgr2=0x7f38e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:54.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38e806bbd0 0x7f38e806e090 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f38f4033000 tx=0x7f38f4009030 comp rx=0 tx=0).stop 2024-03-28T19:13:54.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3904145470 msgr2=0x7f39041553f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:54.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3904145470 0x7f39041553f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f38fc005fd0 tx=0x7f38fc007bc0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 shutdown_connections 2024-03-28T19:13:54.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f38e806bbd0 0x7f38e806e090 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3904145470 0x7f39041553f0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3904072100 0x7f3904144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 --2- 172.21.15.27:0/3753917609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38ec0039b0 0x7f39041eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:54.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.291+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 >> 172.21.15.27:0/3753917609 conn(0x7f390406b9d0 msgr2=0x7f39040c33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:54.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.292+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 shutdown_connections 2024-03-28T19:13:54.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:54.292+0000 7f390bc00640 1 -- 172.21.15.27:0/3753917609 wait complete. 2024-03-28T19:13:54.312 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:13:56.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.536+0000 7f39bea00640 1 Processor -- start 2024-03-28T19:13:56.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.536+0000 7f39bea00640 1 -- start start 2024-03-28T19:13:56.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.537+0000 7f39bea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39b8152e70 0x7f39b8156c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.537+0000 7f39bea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 0x7f39b806fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.537+0000 7f39bea00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f39b8156060 0x7f39b80c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.537+0000 7f39bea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b8070270 con 0x7f39b8152e70 2024-03-28T19:13:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.537+0000 7f39bea00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b8070510 con 0x7f39b8156060 2024-03-28T19:13:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.537+0000 7f39bea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80707f0 con 0x7f39b80c2d80 2024-03-28T19:13:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.537+0000 7f39b7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39b8152e70 0x7f39b8156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39b7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39b8152e70 0x7f39b8156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45778/0 (socket says 172.21.15.27:45778) 2024-03-28T19:13:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39b7e00640 1 -- 172.21.15.27:0/372009299 learned_addr learned my addr 172.21.15.27:0/372009299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:56.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39bca00640 1 --2- 172.21.15.27:0/372009299 >> v2:172.21.15.155:3300/0 conn(0x7f39b8156060 0x7f39b80c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:56.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39af400640 1 --2- 172.21.15.27:0/372009299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 0x7f39b806fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:56.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39b7e00640 1 -- 172.21.15.27:0/372009299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 msgr2=0x7f39b806fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:56.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39b7e00640 1 --2- 172.21.15.27:0/372009299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 0x7f39b806fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39b7e00640 1 -- 172.21.15.27:0/372009299 >> v2:172.21.15.155:3300/0 conn(0x7f39b8156060 msgr2=0x7f39b80c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:56.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39b7e00640 1 --2- 172.21.15.27:0/372009299 >> v2:172.21.15.155:3300/0 conn(0x7f39b8156060 0x7f39b80c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.538+0000 7f39b7e00640 1 -- 172.21.15.27:0/372009299 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b8070ad0 con 0x7f39b8152e70 2024-03-28T19:13:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.539+0000 7f39b7e00640 1 --2- 172.21.15.27:0/372009299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b8156c40 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f39b80c9df0 tx=0x7f39a802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc5e9850db5500e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.539+0000 7f39b7400640 1 -- 172.21.15.27:0/372009299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a80317d0 con 0x7f39b8152e70 2024-03-28T19:13:56.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39b7400640 1 -- 172.21.15.27:0/372009299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a8037070 con 0x7f39b8152e70 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39b7400640 1 -- 172.21.15.27:0/372009299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8031b90 con 0x7f39b8152e70 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 -- 172.21.15.27:0/372009299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 msgr2=0x7f39b8156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 --2- 172.21.15.27:0/372009299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b8156c40 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f39b80c9df0 tx=0x7f39a802b870 comp rx=0 tx=0).stop 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 -- 172.21.15.27:0/372009299 shutdown_connections 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 --2- 172.21.15.27:0/372009299 >> v2:172.21.15.155:3300/0 conn(0x7f39b8156060 0x7f39b80c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 --2- 172.21.15.27:0/372009299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 0x7f39b806fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 --2- 172.21.15.27:0/372009299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b8156c40 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 -- 172.21.15.27:0/372009299 >> 172.21.15.27:0/372009299 conn(0x7f39b8066dd0 msgr2=0x7f39b80671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 -- 172.21.15.27:0/372009299 shutdown_connections 2024-03-28T19:13:56.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.540+0000 7f39bea00640 1 -- 172.21.15.27:0/372009299 wait complete. 2024-03-28T19:13:56.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.542+0000 7f39bea00640 1 Processor -- start 2024-03-28T19:13:56.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.542+0000 7f39bea00640 1 -- start start 2024-03-28T19:13:56.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.543+0000 7f39bea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 0x7f39b814d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.543+0000 7f39bea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b814b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.543+0000 7f39bea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39b8156060 0x7f39b814b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.543+0000 7f39bea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8153250 con 0x7f39b8152e70 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.543+0000 7f39bea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8155b40 con 0x7f39b8156060 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.543+0000 7f39bea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b81533d0 con 0x7f39b80c2d80 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39af400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b814b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39af400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b814b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45794/0 (socket says 172.21.15.27:45794) 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39af400640 1 -- 172.21.15.27:0/3643874997 learned_addr learned my addr 172.21.15.27:0/3643874997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39b7e00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 0x7f39b814d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:56.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39bca00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39b8156060 0x7f39b814b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39af400640 1 -- 172.21.15.27:0/3643874997 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 msgr2=0x7f39b814d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39af400640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 0x7f39b814d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39af400640 1 -- 172.21.15.27:0/3643874997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39b8156060 msgr2=0x7f39b814b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39af400640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39b8156060 0x7f39b814b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.544+0000 7f39af400640 1 -- 172.21.15.27:0/3643874997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80abfc0 con 0x7f39b8152e70 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.545+0000 7f39af400640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b814b490 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f39a000d830 tx=0x7f39a000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.545+0000 7f39b4c00640 1 -- 172.21.15.27:0/3643874997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0018030 con 0x7f39b8152e70 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.545+0000 7f39b4c00640 1 -- 172.21.15.27:0/3643874997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a000be10 con 0x7f39b8152e70 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.545+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b80ac960 con 0x7f39b8152e70 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.545+0000 7f39b4c00640 1 -- 172.21.15.27:0/3643874997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00098b0 con 0x7f39b8152e70 2024-03-28T19:13:56.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.545+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b80acc40 con 0x7f39b8152e70 2024-03-28T19:13:56.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.546+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b804f400 con 0x7f39b8152e70 2024-03-28T19:13:56.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.547+0000 7f39b4c00640 1 -- 172.21.15.27:0/3643874997 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f39a001b070 con 0x7f39b8152e70 2024-03-28T19:13:56.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.548+0000 7f39b4c00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f399406bbd0 0x7f399406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:56.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.548+0000 7f39b4c00640 1 -- 172.21.15.27:0/3643874997 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f39a008b7d0 con 0x7f39b8152e70 2024-03-28T19:13:56.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.549+0000 7f39b7e00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f399406bbd0 0x7f399406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:56.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.549+0000 7f39b7e00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f399406bbd0 0x7f399406e090 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f39a8036000 tx=0x7f39a803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:56.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.553+0000 7f39b4c00640 1 -- 172.21.15.27:0/3643874997 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f39a00596f0 con 0x7f39b8152e70 2024-03-28T19:13:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.799+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39b806e290 con 0x7f39b8152e70 2024-03-28T19:13:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.799+0000 7f39b4c00640 1 -- 172.21.15.27:0/3643874997 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f39a0058ff0 con 0x7f39b8152e70 2024-03-28T19:13:56.800 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:13:55.072117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 138 2024-03-28T19:13:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f399406bbd0 msgr2=0x7f399406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f399406bbd0 0x7f399406e090 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f39a8036000 tx=0x7f39a803d040 comp rx=0 tx=0).stop 2024-03-28T19:13:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 msgr2=0x7f39b814b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b814b490 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f39a000d830 tx=0x7f39a000dcf0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 shutdown_connections 2024-03-28T19:13:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f399406bbd0 0x7f399406e090 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39b8156060 0x7f39b814b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39b8152e70 0x7f39b814b490 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 --2- 172.21.15.27:0/3643874997 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39b80c2d80 0x7f39b814d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 >> 172.21.15.27:0/3643874997 conn(0x7f39b8066dd0 msgr2=0x7f39b80671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 shutdown_connections 2024-03-28T19:13:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:56.802+0000 7f39bea00640 1 -- 172.21.15.27:0/3643874997 wait complete. 2024-03-28T19:13:56.820 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-03-28T19:13:56.820 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:13:56.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.013+0000 7f804d400640 1 Processor -- start 2024-03-28T19:13:57.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.013+0000 7f804d400640 1 -- start start 2024-03-28T19:13:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.014+0000 7f804d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8048150c70 0x7f80480a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:57.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.014+0000 7f804d400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f80480c3690 0x7f80480c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f804d400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80480c3af0 con 0x7f80480c6660 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f804d400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80480ab4e0 con 0x7f8048150c70 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f804d400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80480ac1f0 con 0x7f80480c3690 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f8046800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8048150c70 0x7f80480a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f8046800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8048150c70 0x7f80480a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45800/0 (socket says 172.21.15.27:45800) 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f8046800640 1 -- 172.21.15.27:0/3305422382 learned_addr learned my addr 172.21.15.27:0/3305422382 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f8045e00640 1 --2- 172.21.15.27:0/3305422382 >> v2:172.21.15.155:3300/0 conn(0x7f80480c3690 0x7f80480c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f8046800640 1 -- 172.21.15.27:0/3305422382 >> v2:172.21.15.155:3300/0 conn(0x7f80480c3690 msgr2=0x7f80480c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f8046800640 1 --2- 172.21.15.27:0/3305422382 >> v2:172.21.15.155:3300/0 conn(0x7f80480c3690 0x7f80480c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f8046800640 1 -- 172.21.15.27:0/3305422382 >> v1:172.21.15.155:6789/0 conn(0x7f80480c6660 legacy=0x7f80481534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:57.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.015+0000 7f8046800640 1 -- 172.21.15.27:0/3305422382 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80480c3d90 con 0x7f8048150c70 2024-03-28T19:13:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.016+0000 7f8046800640 1 --2- 172.21.15.27:0/3305422382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8048150c70 0x7f80480a9cb0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f80480c4180 tx=0x7f803802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8db8d85c8e8596 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.017+0000 7f8045400640 1 -- 172.21.15.27:0/3305422382 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038035020 con 0x7f8048150c70 2024-03-28T19:13:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.017+0000 7f8045400640 1 -- 172.21.15.27:0/3305422382 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8038031870 con 0x7f8048150c70 2024-03-28T19:13:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.017+0000 7f8045400640 1 -- 172.21.15.27:0/3305422382 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038031c40 con 0x7f8048150c70 2024-03-28T19:13:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.017+0000 7f804d400640 1 -- 172.21.15.27:0/3305422382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8048150c70 msgr2=0x7f80480a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.017+0000 7f804d400640 1 --2- 172.21.15.27:0/3305422382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8048150c70 0x7f80480a9cb0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f80480c4180 tx=0x7f803802b910 comp rx=0 tx=0).stop 2024-03-28T19:13:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.018+0000 7f804d400640 1 -- 172.21.15.27:0/3305422382 shutdown_connections 2024-03-28T19:13:57.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.018+0000 7f804d400640 1 --2- 172.21.15.27:0/3305422382 >> v2:172.21.15.155:3300/0 conn(0x7f80480c3690 0x7f80480c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.018+0000 7f804d400640 1 --2- 172.21.15.27:0/3305422382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8048150c70 0x7f80480a9cb0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.018+0000 7f804d400640 1 -- 172.21.15.27:0/3305422382 >> 172.21.15.27:0/3305422382 conn(0x7f804806b9d0 msgr2=0x7f80480c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:57.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.018+0000 7f804d400640 1 -- 172.21.15.27:0/3305422382 shutdown_connections 2024-03-28T19:13:57.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.018+0000 7f804d400640 1 -- 172.21.15.27:0/3305422382 wait complete. 2024-03-28T19:13:57.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.019+0000 7f804d400640 1 Processor -- start 2024-03-28T19:13:57.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.019+0000 7f804d400640 1 -- start start 2024-03-28T19:13:57.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.020+0000 7f804d400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80480c3690 0x7f804806e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:57.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.020+0000 7f804d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480c6660 0x7f80481eb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.020+0000 7f804d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8048150c70 0x7f80481eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.020+0000 7f804d400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80480abcd0 con 0x7f80480c6660 2024-03-28T19:13:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.020+0000 7f804d400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80480ab9f0 con 0x7f8048150c70 2024-03-28T19:13:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.020+0000 7f804d400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80480c1810 con 0x7f80480c3690 2024-03-28T19:13:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8045e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480c6660 0x7f80481eb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8045e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480c6660 0x7f80481eb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45804/0 (socket says 172.21.15.27:45804) 2024-03-28T19:13:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8045e00640 1 -- 172.21.15.27:0/3072133327 learned_addr learned my addr 172.21.15.27:0/3072133327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:57.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8046800640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80480c3690 0x7f804806e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:57.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8047200640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8048150c70 0x7f80481eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:57.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8045e00640 1 -- 172.21.15.27:0/3072133327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80480c3690 msgr2=0x7f804806e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:57.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8045e00640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80480c3690 0x7f804806e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8045e00640 1 -- 172.21.15.27:0/3072133327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8048150c70 msgr2=0x7f80481eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:57.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8045e00640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8048150c70 0x7f80481eb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.021+0000 7f8045e00640 1 -- 172.21.15.27:0/3072133327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80481ee620 con 0x7f80480c6660 2024-03-28T19:13:57.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.022+0000 7f8045e00640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480c6660 0x7f80481eb0b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f803400d800 tx=0x7f803400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:57.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.022+0000 7f803ea00640 1 -- 172.21.15.27:0/3072133327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803400b660 con 0x7f80480c6660 2024-03-28T19:13:57.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.022+0000 7f803ea00640 1 -- 172.21.15.27:0/3072133327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8034019070 con 0x7f80480c6660 2024-03-28T19:13:57.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.022+0000 7f803ea00640 1 -- 172.21.15.27:0/3072133327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803400ba20 con 0x7f80480c6660 2024-03-28T19:13:57.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.022+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80481e24a0 con 0x7f80480c6660 2024-03-28T19:13:57.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.022+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80481ee8e0 con 0x7f80480c6660 2024-03-28T19:13:57.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.024+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8048071d30 con 0x7f80480c6660 2024-03-28T19:13:57.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.025+0000 7f803ea00640 1 -- 172.21.15.27:0/3072133327 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f803400bbc0 con 0x7f80480c6660 2024-03-28T19:13:57.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.026+0000 7f803ea00640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f802806bbd0 0x7f802806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:57.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.026+0000 7f803ea00640 1 -- 172.21.15.27:0/3072133327 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f803408b8e0 con 0x7f80480c6660 2024-03-28T19:13:57.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.026+0000 7f8046800640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f802806bbd0 0x7f802806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:57.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.027+0000 7f8046800640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f802806bbd0 0x7f802806e090 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8038036000 tx=0x7f803803c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:57.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.032+0000 7f803ea00640 1 -- 172.21.15.27:0/3072133327 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8048071d30 con 0x7f80480c6660 2024-03-28T19:13:57.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.282+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f804804f400 con 0x7f80480c6660 2024-03-28T19:13:57.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.283+0000 7f803ea00640 1 -- 172.21.15.27:0/3072133327 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f8034059210 con 0x7f80480c6660 2024-03-28T19:13:57.283 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:57.283 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:13:55.072117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:57.284 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 138 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.285+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f802806bbd0 msgr2=0x7f802806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.285+0000 7f804d400640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f802806bbd0 0x7f802806e090 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8038036000 tx=0x7f803803c040 comp rx=0 tx=0).stop 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.285+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480c6660 msgr2=0x7f80481eb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.285+0000 7f804d400640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480c6660 0x7f80481eb0b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f803400d800 tx=0x7f803400dcc0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.286+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 shutdown_connections 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.286+0000 7f804d400640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f802806bbd0 0x7f802806e090 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.286+0000 7f804d400640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8048150c70 0x7f80481eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.286+0000 7f804d400640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480c6660 0x7f80481eb0b0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.286+0000 7f804d400640 1 --2- 172.21.15.27:0/3072133327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80480c3690 0x7f804806e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.286+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 >> 172.21.15.27:0/3072133327 conn(0x7f804806b9d0 msgr2=0x7f8048061ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.286+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 shutdown_connections 2024-03-28T19:13:57.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:57.286+0000 7f804d400640 1 -- 172.21.15.27:0/3072133327 wait complete. 2024-03-28T19:13:57.303 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 138, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 138, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6532": { "addr": "172.21.15.27:6837/552369694", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 552369694, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 552369694, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6532, "incarnation": 138, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:13:55.072117+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6532 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 136, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5756": { "addr": "172.21.15.155:6835/2989591316", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 2989591316, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 2989591316, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5756, "incarnation": 130, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:12:23.824811+0000", "qdb_cluster": [ 5756 ], "qdb_leader": 5756, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5756 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/2504098667", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 2504098667, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 2504098667, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 136, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5870, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:13:59.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:13:59.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.552+0000 7fd952c00640 1 Processor -- start 2024-03-28T19:13:59.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.552+0000 7fd952c00640 1 -- start start 2024-03-28T19:13:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.553+0000 7fd952c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd94c159650 0x7fd94c0c1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.553+0000 7fd952c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c158cb0 0x7fd94c06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:59.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd952c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94c0c7a10 con 0x7fd94c06c610 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd952c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd94c0c7170 con 0x7fd94c159650 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd952c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c0c7450 con 0x7fd94c158cb0 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd950c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd94c06c610 0x7fd94c155d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58896/0 (socket says 172.21.15.27:58896) 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd950c00640 1 -- 172.21.15.27:0/2287837245 learned_addr learned my addr 172.21.15.27:0/2287837245 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd94be00640 1 --2- 172.21.15.27:0/2287837245 >> v2:172.21.15.155:3300/0 conn(0x7fd94c159650 0x7fd94c0c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd943400640 1 --2- 172.21.15.27:0/2287837245 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c158cb0 0x7fd94c06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd94b400640 1 -- 172.21.15.27:0/2287837245 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865953385 0 0) 0x7fd94c0c7a10 con 0x7fd94c06c610 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.554+0000 7fd94b400640 1 -- 172.21.15.27:0/2287837245 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd930003520 con 0x7fd94c06c610 2024-03-28T19:13:59.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.555+0000 7fd94be00640 1 -- 172.21.15.27:0/2287837245 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c158cb0 msgr2=0x7fd94c06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:13:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.555+0000 7fd94be00640 1 --2- 172.21.15.27:0/2287837245 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c158cb0 0x7fd94c06c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.555+0000 7fd94b400640 1 -- 172.21.15.27:0/2287837245 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3175134080 0 0) 0x7fd930003520 con 0x7fd94c06c610 2024-03-28T19:13:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.555+0000 7fd94be00640 1 -- 172.21.15.27:0/2287837245 >> v1:172.21.15.27:6789/0 conn(0x7fd94c06c610 legacy=0x7fd94c155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.555+0000 7fd94be00640 1 -- 172.21.15.27:0/2287837245 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c0c7730 con 0x7fd94c159650 2024-03-28T19:13:59.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.556+0000 7fd94be00640 1 --2- 172.21.15.27:0/2287837245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 0x7fd94c0c1570 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009d90 tx=0x7fd93c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b02aeba2e2ff4803 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.559+0000 7fd94b400640 1 -- 172.21.15.27:0/2287837245 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c039070 con 0x7fd94c159650 2024-03-28T19:13:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.559+0000 7fd94b400640 1 -- 172.21.15.27:0/2287837245 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c02c8a0 con 0x7fd94c159650 2024-03-28T19:13:59.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.559+0000 7fd94b400640 1 -- 172.21.15.27:0/2287837245 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944001670 con 0x7fd94c159650 2024-03-28T19:13:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.559+0000 7fd952c00640 1 -- 172.21.15.27:0/2287837245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 msgr2=0x7fd94c0c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.559+0000 7fd952c00640 1 --2- 172.21.15.27:0/2287837245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 0x7fd94c0c1570 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009d90 tx=0x7fd93c02bd70 comp rx=0 tx=0).stop 2024-03-28T19:13:59.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.560+0000 7fd952c00640 1 -- 172.21.15.27:0/2287837245 shutdown_connections 2024-03-28T19:13:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.560+0000 7fd952c00640 1 --2- 172.21.15.27:0/2287837245 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c158cb0 0x7fd94c06c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.560+0000 7fd952c00640 1 --2- 172.21.15.27:0/2287837245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 0x7fd94c0c1570 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.560+0000 7fd952c00640 1 -- 172.21.15.27:0/2287837245 >> 172.21.15.27:0/2287837245 conn(0x7fd94c0c4750 msgr2=0x7fd94c0c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.560+0000 7fd952c00640 1 -- 172.21.15.27:0/2287837245 shutdown_connections 2024-03-28T19:13:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.560+0000 7fd952c00640 1 -- 172.21.15.27:0/2287837245 wait complete. 2024-03-28T19:13:59.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.562+0000 7fd952c00640 1 Processor -- start 2024-03-28T19:13:59.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.562+0000 7fd952c00640 1 -- start start 2024-03-28T19:13:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd952c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c06c610 0x7fd94c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd952c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd94c158cb0 0x7fd94c158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd952c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 0x7fd94c0aab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd952c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c0c1c30 con 0x7fd94c158cb0 2024-03-28T19:13:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd952c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c0c1ab0 con 0x7fd94c159650 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd952c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c156470 con 0x7fd94c06c610 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd943400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd94c158cb0 0x7fd94c158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd943400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd94c158cb0 0x7fd94c158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45812/0 (socket says 172.21.15.27:45812) 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd943400640 1 -- 172.21.15.27:0/749009909 learned_addr learned my addr 172.21.15.27:0/749009909 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd94be00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c06c610 0x7fd94c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.563+0000 7fd950c00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 0x7fd94c0aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.564+0000 7fd943400640 1 -- 172.21.15.27:0/749009909 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c06c610 msgr2=0x7fd94c06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.564+0000 7fd943400640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c06c610 0x7fd94c06ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.564+0000 7fd943400640 1 -- 172.21.15.27:0/749009909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 msgr2=0x7fd94c0aab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.564+0000 7fd943400640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 0x7fd94c0aab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.564+0000 7fd943400640 1 -- 172.21.15.27:0/749009909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c0a8930 con 0x7fd94c158cb0 2024-03-28T19:13:59.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.564+0000 7fd943400640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd94c158cb0 0x7fd94c158240 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd93400e8d0 tx=0x7fd93400ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.564+0000 7fd948c00640 1 -- 172.21.15.27:0/749009909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd934017070 con 0x7fd94c158cb0 2024-03-28T19:13:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.565+0000 7fd948c00640 1 -- 172.21.15.27:0/749009909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93400c940 con 0x7fd94c158cb0 2024-03-28T19:13:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.565+0000 7fd948c00640 1 -- 172.21.15.27:0/749009909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93400cd30 con 0x7fd94c158cb0 2024-03-28T19:13:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.565+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94c0a8ba0 con 0x7fd94c158cb0 2024-03-28T19:13:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.565+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94c0a9160 con 0x7fd94c158cb0 2024-03-28T19:13:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.567+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd94c068890 con 0x7fd94c158cb0 2024-03-28T19:13:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.568+0000 7fd948c00640 1 -- 172.21.15.27:0/749009909 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd93401a070 con 0x7fd94c158cb0 2024-03-28T19:13:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.569+0000 7fd948c00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd92806bc00 0x7fd92806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:13:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.569+0000 7fd948c00640 1 -- 172.21.15.27:0/749009909 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7fd93408b6c0 con 0x7fd94c158cb0 2024-03-28T19:13:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.569+0000 7fd94be00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd92806bc00 0x7fd92806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:13:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.570+0000 7fd94be00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd92806bc00 0x7fd92806e0c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd93c036000 tx=0x7fd93c030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:13:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.574+0000 7fd948c00640 1 -- 172.21.15.27:0/749009909 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd94c068890 con 0x7fd94c158cb0 2024-03-28T19:13:59.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.846+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd94c04f400 con 0x7fd94c158cb0 2024-03-28T19:13:59.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.847+0000 7fd948c00640 1 -- 172.21.15.27:0/749009909 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5464 (secure 0 0 0) 0x7fd934058e70 con 0x7fd94c158cb0 2024-03-28T19:13:59.848 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:13:59.848 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:13:58.575906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:13:59.848 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 140 2024-03-28T19:13:59.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd92806bc00 msgr2=0x7fd92806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:59.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd92806bc00 0x7fd92806e0c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd93c036000 tx=0x7fd93c030000 comp rx=0 tx=0).stop 2024-03-28T19:13:59.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd94c158cb0 msgr2=0x7fd94c158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:13:59.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd94c158cb0 0x7fd94c158240 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd93400e8d0 tx=0x7fd93400ed90 comp rx=0 tx=0).stop 2024-03-28T19:13:59.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 shutdown_connections 2024-03-28T19:13:59.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd92806bc00 0x7fd92806e0c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd94c159650 0x7fd94c0aab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd94c158cb0 0x7fd94c158240 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 --2- 172.21.15.27:0/749009909 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd94c06c610 0x7fd94c06ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:13:59.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 >> 172.21.15.27:0/749009909 conn(0x7fd94c0c4750 msgr2=0x7fd94c0c2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:13:59.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 shutdown_connections 2024-03-28T19:13:59.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:13:59.850+0000 7fd952c00640 1 -- 172.21.15.27:0/749009909 wait complete. 2024-03-28T19:13:59.872 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:14:01.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:02.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.120+0000 7f65da400640 1 Processor -- start 2024-03-28T19:14:02.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.120+0000 7f65da400640 1 -- start start 2024-03-28T19:14:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.122+0000 7f65da400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65d4158580 0x7f65d41560c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.122+0000 7f65da400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f65d4159150 0x7f65d4159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.122+0000 7f65da400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d4156600 con 0x7f65d40be630 2024-03-28T19:14:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.122+0000 7f65da400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d40c76e0 con 0x7f65d4158580 2024-03-28T19:14:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.122+0000 7f65da400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d40c6dd0 con 0x7f65d4159150 2024-03-28T19:14:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.122+0000 7f65d3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65d4158580 0x7f65d41560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.122+0000 7f65d3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65d4158580 0x7f65d41560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45822/0 (socket says 172.21.15.27:45822) 2024-03-28T19:14:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.122+0000 7f65d3400640 1 -- 172.21.15.27:0/518105302 learned_addr learned my addr 172.21.15.27:0/518105302 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:02.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.123+0000 7f65d3e00640 1 --2- 172.21.15.27:0/518105302 >> v2:172.21.15.155:3300/0 conn(0x7f65d4159150 0x7f65d4159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:02.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.123+0000 7f65d3400640 1 -- 172.21.15.27:0/518105302 >> v2:172.21.15.155:3300/0 conn(0x7f65d4159150 msgr2=0x7f65d4159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:02.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.123+0000 7f65d3400640 1 --2- 172.21.15.27:0/518105302 >> v2:172.21.15.155:3300/0 conn(0x7f65d4159150 0x7f65d4159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.123+0000 7f65d3400640 1 -- 172.21.15.27:0/518105302 >> v1:172.21.15.155:6789/0 conn(0x7f65d40be630 legacy=0x7f65d40c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:02.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.123+0000 7f65d3400640 1 -- 172.21.15.27:0/518105302 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40c7910 con 0x7f65d4158580 2024-03-28T19:14:02.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.125+0000 7f65d3400640 1 --2- 172.21.15.27:0/518105302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 0x7f65d41560c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f65d40c8aa0 tx=0x7f65c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d737a549b120e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:02.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.125+0000 7f65d2000640 1 -- 172.21.15.27:0/518105302 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4035020 con 0x7f65d4158580 2024-03-28T19:14:02.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.125+0000 7f65d2000640 1 -- 172.21.15.27:0/518105302 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4031870 con 0x7f65d4158580 2024-03-28T19:14:02.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.125+0000 7f65d2000640 1 -- 172.21.15.27:0/518105302 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4031c40 con 0x7f65d4158580 2024-03-28T19:14:02.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.126+0000 7f65da400640 1 -- 172.21.15.27:0/518105302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 msgr2=0x7f65d41560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:02.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.126+0000 7f65da400640 1 --2- 172.21.15.27:0/518105302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 0x7f65d41560c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f65d40c8aa0 tx=0x7f65c402b910 comp rx=0 tx=0).stop 2024-03-28T19:14:02.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.126+0000 7f65da400640 1 -- 172.21.15.27:0/518105302 shutdown_connections 2024-03-28T19:14:02.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.126+0000 7f65da400640 1 --2- 172.21.15.27:0/518105302 >> v2:172.21.15.155:3300/0 conn(0x7f65d4159150 0x7f65d4159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.126+0000 7f65da400640 1 --2- 172.21.15.27:0/518105302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 0x7f65d41560c0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.126+0000 7f65da400640 1 -- 172.21.15.27:0/518105302 >> 172.21.15.27:0/518105302 conn(0x7f65d40caf80 msgr2=0x7f65d40cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:02.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.127+0000 7f65da400640 1 -- 172.21.15.27:0/518105302 shutdown_connections 2024-03-28T19:14:02.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.127+0000 7f65da400640 1 -- 172.21.15.27:0/518105302 wait complete. 2024-03-28T19:14:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.129+0000 7f65da400640 1 Processor -- start 2024-03-28T19:14:02.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.130+0000 7f65da400640 1 -- start start 2024-03-28T19:14:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.131+0000 7f65da400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d40be630 0x7f65d414dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.131+0000 7f65da400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 0x7f65d40c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.131+0000 7f65da400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d4159150 0x7f65d40c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.131+0000 7f65da400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d40c7ed0 con 0x7f65d4158580 2024-03-28T19:14:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.131+0000 7f65da400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d4158cc0 con 0x7f65d40be630 2024-03-28T19:14:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.131+0000 7f65da400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d40c81b0 con 0x7f65d4159150 2024-03-28T19:14:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.132+0000 7f65d3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d40be630 0x7f65d414dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.132+0000 7f65d2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 0x7f65d40c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.132+0000 7f65d3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d40be630 0x7f65d414dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:36314/0 (socket says 172.21.15.27:36314) 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.132+0000 7f65d3400640 1 -- 172.21.15.27:0/2659731966 learned_addr learned my addr 172.21.15.27:0/2659731966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.132+0000 7f65d3e00640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d4159150 0x7f65d40c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.132+0000 7f65d3400640 1 -- 172.21.15.27:0/2659731966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d4159150 msgr2=0x7f65d40c47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.132+0000 7f65d3400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d4159150 0x7f65d40c47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.132+0000 7f65d3400640 1 -- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 msgr2=0x7f65d40c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.133+0000 7f65d3400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 0x7f65d40c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.133+0000 7f65d3400640 1 -- 172.21.15.27:0/2659731966 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40c2a60 con 0x7f65d40be630 2024-03-28T19:14:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.133+0000 7f65d2a00640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 0x7f65d40c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:14:02.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.133+0000 7f65d3400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d40be630 0x7f65d414dd50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009960 tx=0x7f65c40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:02.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.133+0000 7f65cb400640 1 -- 172.21.15.27:0/2659731966 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4035040 con 0x7f65d40be630 2024-03-28T19:14:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.134+0000 7f65cb400640 1 -- 172.21.15.27:0/2659731966 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4030d00 con 0x7f65d40be630 2024-03-28T19:14:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.134+0000 7f65cb400640 1 -- 172.21.15.27:0/2659731966 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c40456a0 con 0x7f65d40be630 2024-03-28T19:14:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.134+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d40c2d40 con 0x7f65d40be630 2024-03-28T19:14:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.134+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d414e290 con 0x7f65d40be630 2024-03-28T19:14:02.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.136+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d40c2f80 con 0x7f65d40be630 2024-03-28T19:14:02.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.137+0000 7f65cb400640 1 -- 172.21.15.27:0/2659731966 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f65c4038020 con 0x7f65d40be630 2024-03-28T19:14:02.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.138+0000 7f65cb400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bc00 0x7f65b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:02.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.138+0000 7f65cb400640 1 -- 172.21.15.27:0/2659731966 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f65c40ad790 con 0x7f65d40be630 2024-03-28T19:14:02.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.139+0000 7f65d2a00640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bc00 0x7f65b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.140+0000 7f65d2a00640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bc00 0x7f65b406e0c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f65d414c8a0 tx=0x7f65c0007e30 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:02.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.156+0000 7f65cb400640 1 -- 172.21.15.27:0/2659731966 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f65d40c2f80 con 0x7f65d40be630 2024-03-28T19:14:02.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.511+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65d404f400 con 0x7f65d40be630 2024-03-28T19:14:02.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.512+0000 7f65cb400640 1 -- 172.21.15.27:0/2659731966 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f65d4156600 con 0x7f65d40be630 2024-03-28T19:14:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:02.512 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:02.512 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 142 2024-03-28T19:14:02.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.515+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bc00 msgr2=0x7f65b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.515+0000 7f65da400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bc00 0x7f65b406e0c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f65d414c8a0 tx=0x7f65c0007e30 comp rx=0 tx=0).stop 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.515+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d40be630 msgr2=0x7f65d414dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.515+0000 7f65da400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d40be630 0x7f65d414dd50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009960 tx=0x7f65c40046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.515+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 shutdown_connections 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.515+0000 7f65da400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bc00 0x7f65b406e0c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.515+0000 7f65da400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d4159150 0x7f65d40c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.515+0000 7f65da400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d4158580 0x7f65d40c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.516+0000 7f65da400640 1 --2- 172.21.15.27:0/2659731966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d40be630 0x7f65d414dd50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:02.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.516+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 >> 172.21.15.27:0/2659731966 conn(0x7f65d40caf80 msgr2=0x7f65d40c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.516+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 shutdown_connections 2024-03-28T19:14:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:02.516+0000 7f65da400640 1 -- 172.21.15.27:0/2659731966 wait complete. 2024-03-28T19:14:02.535 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6834 2024-03-28T19:14:02.536 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.c 2024-03-28T19:14:02.655 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:14:02.659 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:14:02.861 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:14:02.860+0000 7f7ba4800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:14:06.536 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-03-28T19:14:06.537 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:14:06.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:14:06.545 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:14:06.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:06.758 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:14:06.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.798+0000 7fb31c400640 1 Processor -- start 2024-03-28T19:14:06.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.798+0000 7fb31c400640 1 -- start start 2024-03-28T19:14:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.799+0000 7fb31c400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb314153ed0 0x7fb31406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.799+0000 7fb31c400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb31406eb80 0x7fb31415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.800+0000 7fb31c400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140c3830 con 0x7fb31406e6f0 2024-03-28T19:14:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.800+0000 7fb31c400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3140c6990 con 0x7fb314153ed0 2024-03-28T19:14:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.800+0000 7fb31c400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31406d1f0 con 0x7fb31406eb80 2024-03-28T19:14:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.800+0000 7fb319a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb314153ed0 0x7fb31406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.800+0000 7fb319a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb314153ed0 0x7fb31406ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37516/0 (socket says 172.21.15.27:37516) 2024-03-28T19:14:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.800+0000 7fb319a00640 1 -- 172.21.15.27:0/847230444 learned_addr learned my addr 172.21.15.27:0/847230444 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.800+0000 7fb31a400640 1 --2- 172.21.15.27:0/847230444 >> v2:172.21.15.155:3300/0 conn(0x7fb31406eb80 0x7fb31415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.801+0000 7fb319a00640 1 -- 172.21.15.27:0/847230444 >> v2:172.21.15.155:3300/0 conn(0x7fb31406eb80 msgr2=0x7fb31415a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.801+0000 7fb319a00640 1 --2- 172.21.15.27:0/847230444 >> v2:172.21.15.155:3300/0 conn(0x7fb31406eb80 0x7fb31415a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.801+0000 7fb319a00640 1 -- 172.21.15.27:0/847230444 >> v1:172.21.15.155:6789/0 conn(0x7fb31406e6f0 legacy=0x7fb314159bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T19:14:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.801+0000 7fb319a00640 1 -- 172.21.15.27:0/847230444 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31406ef60 con 0x7fb314153ed0 2024-03-28T19:14:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.802+0000 7fb319a00640 1 --2- 172.21.15.27:0/847230444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb314153ed0 0x7fb31406ccb0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb3140c5ca0 tx=0x7fb30802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25bcbe507e2e9ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:06.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.803+0000 7fb313e00640 1 -- 172.21.15.27:0/847230444 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308035020 con 0x7fb314153ed0 2024-03-28T19:14:06.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.803+0000 7fb313e00640 1 -- 172.21.15.27:0/847230444 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb308031870 con 0x7fb314153ed0 2024-03-28T19:14:06.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.803+0000 7fb313e00640 1 -- 172.21.15.27:0/847230444 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308031c40 con 0x7fb314153ed0 2024-03-28T19:14:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.804+0000 7fb31c400640 1 -- 172.21.15.27:0/847230444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb314153ed0 msgr2=0x7fb31406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.804+0000 7fb31c400640 1 --2- 172.21.15.27:0/847230444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb314153ed0 0x7fb31406ccb0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb3140c5ca0 tx=0x7fb30802b910 comp rx=0 tx=0).stop 2024-03-28T19:14:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.804+0000 7fb31c400640 1 -- 172.21.15.27:0/847230444 shutdown_connections 2024-03-28T19:14:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.804+0000 7fb31c400640 1 --2- 172.21.15.27:0/847230444 >> v2:172.21.15.155:3300/0 conn(0x7fb31406eb80 0x7fb31415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.804+0000 7fb31c400640 1 --2- 172.21.15.27:0/847230444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb314153ed0 0x7fb31406ccb0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.804+0000 7fb31c400640 1 -- 172.21.15.27:0/847230444 >> 172.21.15.27:0/847230444 conn(0x7fb31406b9d0 msgr2=0x7fb3140c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.804+0000 7fb31c400640 1 -- 172.21.15.27:0/847230444 shutdown_connections 2024-03-28T19:14:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.804+0000 7fb31c400640 1 -- 172.21.15.27:0/847230444 wait complete. 2024-03-28T19:14:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.806+0000 7fb31c400640 1 Processor -- start 2024-03-28T19:14:06.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.806+0000 7fb31c400640 1 -- start start 2024-03-28T19:14:06.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.807+0000 7fb31c400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31406e6f0 0x7fb314152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:06.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.807+0000 7fb31c400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb31406eb80 0x7fb3140a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:06.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.807+0000 7fb31c400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb314153ed0 0x7fb3140a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:06.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.807+0000 7fb31c400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3140c3db0 con 0x7fb31406e6f0 2024-03-28T19:14:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.807+0000 7fb31c400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3140c3ad0 con 0x7fb314153ed0 2024-03-28T19:14:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.807+0000 7fb31c400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31415a810 con 0x7fb31406eb80 2024-03-28T19:14:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31406e6f0 0x7fb314152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31406e6f0 0x7fb314152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37526/0 (socket says 172.21.15.27:37526) 2024-03-28T19:14:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319a00640 1 -- 172.21.15.27:0/1839224692 learned_addr learned my addr 172.21.15.27:0/1839224692 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319000640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb31406eb80 0x7fb3140a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb31a400640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb314153ed0 0x7fb3140a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319a00640 1 -- 172.21.15.27:0/1839224692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb31406eb80 msgr2=0x7fb3140a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319a00640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb31406eb80 0x7fb3140a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319a00640 1 -- 172.21.15.27:0/1839224692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb314153ed0 msgr2=0x7fb3140a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319a00640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb314153ed0 0x7fb3140a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.808+0000 7fb319a00640 1 -- 172.21.15.27:0/1839224692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3141544b0 con 0x7fb31406e6f0 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.809+0000 7fb319a00640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31406e6f0 0x7fb314152a00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fb308000c00 tx=0x7fb3080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.809+0000 7fb311600640 1 -- 172.21.15.27:0/1839224692 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308040070 con 0x7fb31406e6f0 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.809+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb314157360 con 0x7fb31406e6f0 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.809+0000 7fb311600640 1 -- 172.21.15.27:0/1839224692 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb308030d00 con 0x7fb31406e6f0 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.809+0000 7fb311600640 1 -- 172.21.15.27:0/1839224692 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308035040 con 0x7fb31406e6f0 2024-03-28T19:14:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.810+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3140a4880 con 0x7fb31406e6f0 2024-03-28T19:14:06.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.811+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb314068180 con 0x7fb31406e6f0 2024-03-28T19:14:06.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.813+0000 7fb311600640 1 -- 172.21.15.27:0/1839224692 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb308038020 con 0x7fb31406e6f0 2024-03-28T19:14:06.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.814+0000 7fb311600640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2f406bbd0 0x7fb2f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:06.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.815+0000 7fb311600640 1 -- 172.21.15.27:0/1839224692 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7fb3080ad270 con 0x7fb31406e6f0 2024-03-28T19:14:06.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.815+0000 7fb319000640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2f406bbd0 0x7fb2f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:06.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.816+0000 7fb319000640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2f406bbd0 0x7fb2f406e090 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb31414de40 tx=0x7fb3000090e0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:06.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:06.821+0000 7fb311600640 1 -- 172.21.15.27:0/1839224692 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb314068180 con 0x7fb31406e6f0 2024-03-28T19:14:07.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.171+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb31404f400 con 0x7fb31406e6f0 2024-03-28T19:14:07.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.172+0000 7fb311600640 1 -- 172.21.15.27:0/1839224692 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5471 (secure 0 0 0) 0x7fb30807aba0 con 0x7fb31406e6f0 2024-03-28T19:14:07.172 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:07.172 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:07.172 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 142 2024-03-28T19:14:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.175+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2f406bbd0 msgr2=0x7fb2f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:07.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2f406bbd0 0x7fb2f406e090 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb31414de40 tx=0x7fb3000090e0 comp rx=0 tx=0).stop 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31406e6f0 msgr2=0x7fb314152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31406e6f0 0x7fb314152a00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fb308000c00 tx=0x7fb3080046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 shutdown_connections 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb2f406bbd0 0x7fb2f406e090 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb314153ed0 0x7fb3140a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb31406eb80 0x7fb3140a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 --2- 172.21.15.27:0/1839224692 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31406e6f0 0x7fb314152a00 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 >> 172.21.15.27:0/1839224692 conn(0x7fb31406b9d0 msgr2=0x7fb3140c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 shutdown_connections 2024-03-28T19:14:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:07.176+0000 7fb31c400640 1 -- 172.21.15.27:0/1839224692 wait complete. 2024-03-28T19:14:07.197 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:14:09.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.441+0000 7f0f7f200640 1 Processor -- start 2024-03-28T19:14:09.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.441+0000 7f0f7f200640 1 -- start start 2024-03-28T19:14:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.443+0000 7f0f7f200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0f780c66e0 0x7f0f780c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.443+0000 7f0f7f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7806b540 con 0x7f0f780c6f60 2024-03-28T19:14:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.443+0000 7f0f7f200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7806b820 con 0x7f0f78156220 2024-03-28T19:14:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.443+0000 7f0f7f200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f78156700 con 0x7f0f780c66e0 2024-03-28T19:14:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.443+0000 7f0f77e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0f780c6f60 0x7f0f7806bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60118/0 (socket says 172.21.15.27:60118) 2024-03-28T19:14:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.443+0000 7f0f77e00640 1 -- 172.21.15.27:0/3967511727 learned_addr learned my addr 172.21.15.27:0/3967511727 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:09.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.443+0000 7f0f77400640 1 --2- 172.21.15.27:0/3967511727 >> v2:172.21.15.155:3300/0 conn(0x7f0f780c66e0 0x7f0f780c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.444+0000 7f0f76a00640 1 -- 172.21.15.27:0/3967511727 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372888199 0 0) 0x7f0f7806b540 con 0x7f0f780c6f60 2024-03-28T19:14:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.444+0000 7f0f76a00640 1 -- 172.21.15.27:0/3967511727 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f5c003520 con 0x7f0f780c6f60 2024-03-28T19:14:09.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.444+0000 7f0f77400640 1 -- 172.21.15.27:0/3967511727 >> v1:172.21.15.155:6789/0 conn(0x7f0f78156220 legacy=0x7f0f78152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.444+0000 7f0f77400640 1 -- 172.21.15.27:0/3967511727 >> v1:172.21.15.27:6789/0 conn(0x7f0f780c6f60 legacy=0x7f0f7806bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.444+0000 7f0f77400640 1 -- 172.21.15.27:0/3967511727 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780c58b0 con 0x7f0f780c66e0 2024-03-28T19:14:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.446+0000 7f0f77400640 1 --2- 172.21.15.27:0/3967511727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c66e0 0x7f0f780c8e30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0f640097a0 tx=0x7f0f6402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a00db84d495944e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.446+0000 7f0f76a00640 1 -- 172.21.15.27:0/3967511727 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f640315a0 con 0x7f0f780c66e0 2024-03-28T19:14:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.447+0000 7f0f76a00640 1 -- 172.21.15.27:0/3967511727 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f64037070 con 0x7f0f780c66e0 2024-03-28T19:14:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.447+0000 7f0f76a00640 1 -- 172.21.15.27:0/3967511727 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64031960 con 0x7f0f780c66e0 2024-03-28T19:14:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.447+0000 7f0f7f200640 1 -- 172.21.15.27:0/3967511727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c66e0 msgr2=0x7f0f780c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.447+0000 7f0f7f200640 1 --2- 172.21.15.27:0/3967511727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c66e0 0x7f0f780c8e30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0f640097a0 tx=0x7f0f6402b840 comp rx=0 tx=0).stop 2024-03-28T19:14:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.448+0000 7f0f7f200640 1 -- 172.21.15.27:0/3967511727 shutdown_connections 2024-03-28T19:14:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.448+0000 7f0f7f200640 1 --2- 172.21.15.27:0/3967511727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c66e0 0x7f0f780c8e30 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.448+0000 7f0f7f200640 1 -- 172.21.15.27:0/3967511727 >> 172.21.15.27:0/3967511727 conn(0x7f0f780caf30 msgr2=0x7f0f780cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:09.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.448+0000 7f0f7f200640 1 -- 172.21.15.27:0/3967511727 shutdown_connections 2024-03-28T19:14:09.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.448+0000 7f0f7f200640 1 -- 172.21.15.27:0/3967511727 wait complete. 2024-03-28T19:14:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.450+0000 7f0f7f200640 1 Processor -- start 2024-03-28T19:14:09.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.451+0000 7f0f7f200640 1 -- start start 2024-03-28T19:14:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.452+0000 7f0f7f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f780c66e0 0x7f0f7814c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.452+0000 7f0f7f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c6f60 0x7f0f7814bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.452+0000 7f0f7f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f78156220 0x7f0f780bf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.452+0000 7f0f7f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7806c410 con 0x7f0f780c66e0 2024-03-28T19:14:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.452+0000 7f0f7f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7806c290 con 0x7f0f780c6f60 2024-03-28T19:14:09.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.452+0000 7f0f7f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f781532f0 con 0x7f0f78156220 2024-03-28T19:14:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f780c66e0 0x7f0f7814c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f780c66e0 0x7f0f7814c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37532/0 (socket says 172.21.15.27:37532) 2024-03-28T19:14:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f7d200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f78156220 0x7f0f780bf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77e00640 1 -- 172.21.15.27:0/984251622 learned_addr learned my addr 172.21.15.27:0/984251622 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:09.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77400640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c6f60 0x7f0f7814bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77e00640 1 -- 172.21.15.27:0/984251622 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f78156220 msgr2=0x7f0f780bf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77e00640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f78156220 0x7f0f780bf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77e00640 1 -- 172.21.15.27:0/984251622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c6f60 msgr2=0x7f0f7814bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77e00640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c6f60 0x7f0f7814bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.453+0000 7f0f77e00640 1 -- 172.21.15.27:0/984251622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780ca610 con 0x7f0f780c66e0 2024-03-28T19:14:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.454+0000 7f0f77e00640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f780c66e0 0x7f0f7814c710 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800c430 tx=0x7f0f6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.454+0000 7f0f6fe00640 1 -- 172.21.15.27:0/984251622 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f68017070 con 0x7f0f780c66e0 2024-03-28T19:14:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.454+0000 7f0f6fe00640 1 -- 172.21.15.27:0/984251622 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f68005ce0 con 0x7f0f780c66e0 2024-03-28T19:14:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.454+0000 7f0f6fe00640 1 -- 172.21.15.27:0/984251622 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f680072d0 con 0x7f0f780c66e0 2024-03-28T19:14:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.454+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f781ebbe0 con 0x7f0f780c66e0 2024-03-28T19:14:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.454+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f780bec80 con 0x7f0f780c66e0 2024-03-28T19:14:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.456+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7804f400 con 0x7f0f780c66e0 2024-03-28T19:14:09.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.458+0000 7f0f6fe00640 1 -- 172.21.15.27:0/984251622 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0f6801a070 con 0x7f0f780c66e0 2024-03-28T19:14:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.459+0000 7f0f6fe00640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f5806bbd0 0x7f0f5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.459+0000 7f0f6fe00640 1 -- 172.21.15.27:0/984251622 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f0f6808a8e0 con 0x7f0f780c66e0 2024-03-28T19:14:09.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.459+0000 7f0f77400640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f5806bbd0 0x7f0f5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:09.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.460+0000 7f0f77400640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f5806bbd0 0x7f0f5806e090 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0f64036000 tx=0x7f0f6403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:09.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.467+0000 7f0f6fe00640 1 -- 172.21.15.27:0/984251622 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0f68058210 con 0x7f0f780c66e0 2024-03-28T19:14:09.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.839+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f7814c4b0 con 0x7f0f780c66e0 2024-03-28T19:14:09.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.840+0000 7f0f6fe00640 1 -- 172.21.15.27:0/984251622 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f0f6805c2b0 con 0x7f0f780c66e0 2024-03-28T19:14:09.840 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:09.840 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:09.840 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:09.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.843+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f5806bbd0 msgr2=0x7f0f5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:09.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.843+0000 7f0f7f200640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f5806bbd0 0x7f0f5806e090 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0f64036000 tx=0x7f0f6403d040 comp rx=0 tx=0).stop 2024-03-28T19:14:09.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.843+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f780c66e0 msgr2=0x7f0f7814c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:09.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.843+0000 7f0f7f200640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f780c66e0 0x7f0f7814c710 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800c430 tx=0x7f0f6800c8f0 comp rx=0 tx=0).stop 2024-03-28T19:14:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.844+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 shutdown_connections 2024-03-28T19:14:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.844+0000 7f0f7f200640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f5806bbd0 0x7f0f5806e090 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.844+0000 7f0f7f200640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f78156220 0x7f0f780bf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.844+0000 7f0f7f200640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f780c6f60 0x7f0f7814bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.844+0000 7f0f7f200640 1 --2- 172.21.15.27:0/984251622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f780c66e0 0x7f0f7814c710 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.844+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 >> 172.21.15.27:0/984251622 conn(0x7f0f780caf30 msgr2=0x7f0f780abf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.844+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 shutdown_connections 2024-03-28T19:14:09.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:09.844+0000 7f0f7f200640 1 -- 172.21.15.27:0/984251622 wait complete. 2024-03-28T19:14:09.865 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-03-28T19:14:09.865 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2024-03-28T19:14:19.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:20.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.113+0000 7f0b4a800640 1 Processor -- start 2024-03-28T19:14:20.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.114+0000 7f0b4a800640 1 -- start start 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.115+0000 7f0b4a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b44150c70 0x7f0b440a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.115+0000 7f0b4a800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0b440c3690 0x7f0b440c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.115+0000 7f0b4a800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b440c3af0 con 0x7f0b440c6660 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.115+0000 7f0b4a800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b440ab4e0 con 0x7f0b44150c70 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.115+0000 7f0b4a800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b440ac1f0 con 0x7f0b440c3690 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b42a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0b440c3690 0x7f0b440c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b43400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b44150c70 0x7f0b440a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b43e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b440c6660 0x7f0b441534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52220/0 (socket says 172.21.15.27:52220) 2024-03-28T19:14:20.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b43e00640 1 -- 172.21.15.27:0/2603023629 learned_addr learned my addr 172.21.15.27:0/2603023629 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:20.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b42a00640 1 -- 172.21.15.27:0/2603023629 >> v2:172.21.15.27:3300/0 conn(0x7f0b44150c70 msgr2=0x7f0b440a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:20.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b42a00640 1 --2- 172.21.15.27:0/2603023629 >> v2:172.21.15.27:3300/0 conn(0x7f0b44150c70 0x7f0b440a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b42a00640 1 -- 172.21.15.27:0/2603023629 >> v1:172.21.15.27:6789/0 conn(0x7f0b440c6660 legacy=0x7f0b441534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b42a00640 1 -- 172.21.15.27:0/2603023629 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b440c3d90 con 0x7f0b440c3690 2024-03-28T19:14:20.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.116+0000 7f0b42000640 1 -- 172.21.15.27:0/2603023629 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293256259 0 0) 0x7f0b440c3af0 con 0x7f0b440c6660 2024-03-28T19:14:20.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.117+0000 7f0b43400640 1 --2- 172.21.15.27:0/2603023629 >> v2:172.21.15.27:3300/0 conn(0x7f0b44150c70 0x7f0b440a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:14:20.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.118+0000 7f0b42a00640 1 --2- 172.21.15.27:0/2603023629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b440c3690 0x7f0b440c6120 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0b440c4180 tx=0x7f0b3002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37632895406ef7d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:20.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.119+0000 7f0b42000640 1 -- 172.21.15.27:0/2603023629 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b300315a0 con 0x7f0b440c3690 2024-03-28T19:14:20.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.119+0000 7f0b42000640 1 -- 172.21.15.27:0/2603023629 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b30037070 con 0x7f0b440c3690 2024-03-28T19:14:20.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.119+0000 7f0b42000640 1 -- 172.21.15.27:0/2603023629 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30031960 con 0x7f0b440c3690 2024-03-28T19:14:20.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.120+0000 7f0b4a800640 1 -- 172.21.15.27:0/2603023629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b440c3690 msgr2=0x7f0b440c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.120+0000 7f0b4a800640 1 --2- 172.21.15.27:0/2603023629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b440c3690 0x7f0b440c6120 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0b440c4180 tx=0x7f0b3002b840 comp rx=0 tx=0).stop 2024-03-28T19:14:20.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.120+0000 7f0b4a800640 1 -- 172.21.15.27:0/2603023629 shutdown_connections 2024-03-28T19:14:20.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.120+0000 7f0b4a800640 1 --2- 172.21.15.27:0/2603023629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b440c3690 0x7f0b440c6120 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.120+0000 7f0b4a800640 1 --2- 172.21.15.27:0/2603023629 >> v2:172.21.15.27:3300/0 conn(0x7f0b44150c70 0x7f0b440a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.120+0000 7f0b4a800640 1 -- 172.21.15.27:0/2603023629 >> 172.21.15.27:0/2603023629 conn(0x7f0b4406b9d0 msgr2=0x7f0b440c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:20.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.120+0000 7f0b4a800640 1 -- 172.21.15.27:0/2603023629 shutdown_connections 2024-03-28T19:14:20.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.120+0000 7f0b4a800640 1 -- 172.21.15.27:0/2603023629 wait complete. 2024-03-28T19:14:20.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.122+0000 7f0b4a800640 1 Processor -- start 2024-03-28T19:14:20.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.122+0000 7f0b4a800640 1 -- start start 2024-03-28T19:14:20.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.123+0000 7f0b4a800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b440c3690 0x7f0b441516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.123+0000 7f0b4a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b440c6660 0x7f0b44155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.123+0000 7f0b4a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b44150c70 0x7f0b44156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.123+0000 7f0b4a800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b440abcd0 con 0x7f0b440c6660 2024-03-28T19:14:20.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.123+0000 7f0b4a800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b440ab9f0 con 0x7f0b44150c70 2024-03-28T19:14:20.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.123+0000 7f0b4a800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b440c1810 con 0x7f0b440c3690 2024-03-28T19:14:20.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.124+0000 7f0b43e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b44150c70 0x7f0b44156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.124+0000 7f0b43400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b440c3690 0x7f0b441516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.124+0000 7f0b43e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b44150c70 0x7f0b44156340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:58134/0 (socket says 172.21.15.27:58134) 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.124+0000 7f0b43e00640 1 -- 172.21.15.27:0/1202958275 learned_addr learned my addr 172.21.15.27:0/1202958275 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.124+0000 7f0b42a00640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b440c6660 0x7f0b44155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.125+0000 7f0b42a00640 1 -- 172.21.15.27:0/1202958275 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b440c3690 msgr2=0x7f0b441516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.125+0000 7f0b42a00640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b440c3690 0x7f0b441516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.125+0000 7f0b42a00640 1 -- 172.21.15.27:0/1202958275 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b44150c70 msgr2=0x7f0b44156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.125+0000 7f0b42a00640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b44150c70 0x7f0b44156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.125+0000 7f0b42a00640 1 -- 172.21.15.27:0/1202958275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4414bf50 con 0x7f0b440c6660 2024-03-28T19:14:20.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.125+0000 7f0b42a00640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b440c6660 0x7f0b44155e00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f0b30033000 tx=0x7f0b300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:20.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.126+0000 7f0b3b400640 1 -- 172.21.15.27:0/1202958275 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30032630 con 0x7f0b440c6660 2024-03-28T19:14:20.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.126+0000 7f0b3b400640 1 -- 172.21.15.27:0/1202958275 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b300327d0 con 0x7f0b440c6660 2024-03-28T19:14:20.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.126+0000 7f0b3b400640 1 -- 172.21.15.27:0/1202958275 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30032ba0 con 0x7f0b440c6660 2024-03-28T19:14:20.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.126+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b441ec1d0 con 0x7f0b440c6660 2024-03-28T19:14:20.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.126+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b441ec790 con 0x7f0b440c6660 2024-03-28T19:14:20.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.128+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4404f400 con 0x7f0b440c6660 2024-03-28T19:14:20.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.128+0000 7f0b3b400640 1 -- 172.21.15.27:0/1202958275 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0b3003b3f0 con 0x7f0b440c6660 2024-03-28T19:14:20.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.129+0000 7f0b3b400640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b24070060 0x7f0b24072520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.130+0000 7f0b3b400640 1 -- 172.21.15.27:0/1202958275 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f0b3002f030 con 0x7f0b440c6660 2024-03-28T19:14:20.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.131+0000 7f0b43400640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b24070060 0x7f0b24072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.133+0000 7f0b43400640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b24070060 0x7f0b24072520 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0b440c3580 tx=0x7f0b34009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:20.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.137+0000 7f0b3b400640 1 -- 172.21.15.27:0/1202958275 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0b30040070 con 0x7f0b440c6660 2024-03-28T19:14:20.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.466+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b440c5f00 con 0x7f0b440c6660 2024-03-28T19:14:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.467+0000 7f0b3b400640 1 -- 172.21.15.27:0/1202958275 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f0b3007b500 con 0x7f0b440c6660 2024-03-28T19:14:20.466 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:20.466 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:20.466 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.469+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b24070060 msgr2=0x7f0b24072520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.469+0000 7f0b4a800640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b24070060 0x7f0b24072520 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0b440c3580 tx=0x7f0b34009190 comp rx=0 tx=0).stop 2024-03-28T19:14:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.469+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b440c6660 msgr2=0x7f0b44155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.469+0000 7f0b4a800640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b440c6660 0x7f0b44155e00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f0b30033000 tx=0x7f0b300046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.470+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 shutdown_connections 2024-03-28T19:14:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.470+0000 7f0b4a800640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0b24070060 0x7f0b24072520 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.470+0000 7f0b4a800640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b44150c70 0x7f0b44156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.470+0000 7f0b4a800640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b440c6660 0x7f0b44155e00 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.470+0000 7f0b4a800640 1 --2- 172.21.15.27:0/1202958275 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b440c3690 0x7f0b441516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.470+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 >> 172.21.15.27:0/1202958275 conn(0x7f0b4406b9d0 msgr2=0x7f0b440c2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.470+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 shutdown_connections 2024-03-28T19:14:20.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.470+0000 7f0b4a800640 1 -- 172.21.15.27:0/1202958275 wait complete. 2024-03-28T19:14:20.494 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-03-28T19:14:20.494 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:14:20.494 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:14:20.522 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:14:20.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.794+0000 7f63a4e00640 1 Processor -- start 2024-03-28T19:14:20.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.794+0000 7f63a4e00640 1 -- start start 2024-03-28T19:14:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f63a4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63a0155cc0 0x7f63a00a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f63a4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a00c1a40 0x7f63a00aa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f63a4e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a0073300 con 0x7f63a0070a50 2024-03-28T19:14:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f63a4e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a00aa020 con 0x7f63a0155cc0 2024-03-28T19:14:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f63a4e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00735e0 con 0x7f63a00c1a40 2024-03-28T19:14:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f639de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63a0155cc0 0x7f63a00a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f639de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63a0155cc0 0x7f63a00a5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40642/0 (socket says 172.21.15.27:40642) 2024-03-28T19:14:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f639de00640 1 -- 172.21.15.27:0/3779752808 learned_addr learned my addr 172.21.15.27:0/3779752808 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.795+0000 7f639e800640 1 --1- 172.21.15.27:0/3779752808 >> v1:172.21.15.27:6789/0 conn(0x7f63a0070a50 0x7f63a01560a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52226/0 (socket says 172.21.15.27:52226) 2024-03-28T19:14:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.796+0000 7f639d400640 1 --2- 172.21.15.27:0/3779752808 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a00c1a40 0x7f63a00aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.796+0000 7f639de00640 1 -- 172.21.15.27:0/3779752808 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a00c1a40 msgr2=0x7f63a00aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.796+0000 7f639de00640 1 --2- 172.21.15.27:0/3779752808 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a00c1a40 0x7f63a00aa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.796+0000 7f639de00640 1 -- 172.21.15.27:0/3779752808 >> v1:172.21.15.27:6789/0 conn(0x7f63a0070a50 legacy=0x7f63a01560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.796+0000 7f639de00640 1 -- 172.21.15.27:0/3779752808 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a00a9cb0 con 0x7f63a0155cc0 2024-03-28T19:14:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.796+0000 7f639ca00640 1 -- 172.21.15.27:0/3779752808 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299150935 0 0) 0x7f63a0073300 con 0x7f63a0070a50 2024-03-28T19:14:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.797+0000 7f639de00640 1 --2- 172.21.15.27:0/3779752808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a0155cc0 0x7f63a00a5d90 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f638c009870 tx=0x7f638c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781d4689b390c558 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.797+0000 7f639ca00640 1 -- 172.21.15.27:0/3779752808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c035020 con 0x7f63a0155cc0 2024-03-28T19:14:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.797+0000 7f639ca00640 1 -- 172.21.15.27:0/3779752808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c031640 con 0x7f63a0155cc0 2024-03-28T19:14:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.797+0000 7f639ca00640 1 -- 172.21.15.27:0/3779752808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c02d040 con 0x7f63a0155cc0 2024-03-28T19:14:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.798+0000 7f63a4e00640 1 -- 172.21.15.27:0/3779752808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a0155cc0 msgr2=0x7f63a00a5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.798+0000 7f63a4e00640 1 --2- 172.21.15.27:0/3779752808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a0155cc0 0x7f63a00a5d90 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f638c009870 tx=0x7f638c02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.798+0000 7f63a4e00640 1 -- 172.21.15.27:0/3779752808 shutdown_connections 2024-03-28T19:14:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.798+0000 7f63a4e00640 1 --2- 172.21.15.27:0/3779752808 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a00c1a40 0x7f63a00aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.798+0000 7f63a4e00640 1 --2- 172.21.15.27:0/3779752808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a0155cc0 0x7f63a00a5d90 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.798+0000 7f63a4e00640 1 -- 172.21.15.27:0/3779752808 >> 172.21.15.27:0/3779752808 conn(0x7f63a00c6c00 msgr2=0x7f63a00c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.798+0000 7f63a4e00640 1 -- 172.21.15.27:0/3779752808 shutdown_connections 2024-03-28T19:14:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.798+0000 7f63a4e00640 1 -- 172.21.15.27:0/3779752808 wait complete. 2024-03-28T19:14:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.800+0000 7f63a4e00640 1 Processor -- start 2024-03-28T19:14:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.800+0000 7f63a4e00640 1 -- start start 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f63a4e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a0070a50 0x7f63a01ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f63a4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a00c1a40 0x7f63a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f63a4e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63a0155cc0 0x7f63a014ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f63a4e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a01567b0 con 0x7f63a00c1a40 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f63a4e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00c3460 con 0x7f63a0155cc0 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f63a4e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00c30f0 con 0x7f63a0070a50 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a00c1a40 0x7f63a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a00c1a40 0x7f63a006d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40648/0 (socket says 172.21.15.27:40648) 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639d400640 1 -- 172.21.15.27:0/2088087974 learned_addr learned my addr 172.21.15.27:0/2088087974 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639de00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a0070a50 0x7f63a01ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639e800640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63a0155cc0 0x7f63a014ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639d400640 1 -- 172.21.15.27:0/2088087974 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a0070a50 msgr2=0x7f63a01ef120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639d400640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a0070a50 0x7f63a01ef120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639d400640 1 -- 172.21.15.27:0/2088087974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63a0155cc0 msgr2=0x7f63a014ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639d400640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63a0155cc0 0x7f63a014ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.802+0000 7f639d400640 1 -- 172.21.15.27:0/2088087974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a01ef660 con 0x7f63a00c1a40 2024-03-28T19:14:20.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.803+0000 7f639d400640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a00c1a40 0x7f63a006d3c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f639000e910 tx=0x7f639000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.803+0000 7f6396000640 1 -- 172.21.15.27:0/2088087974 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6390016070 con 0x7f63a00c1a40 2024-03-28T19:14:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.803+0000 7f6396000640 1 -- 172.21.15.27:0/2088087974 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639000cbe0 con 0x7f63a00c1a40 2024-03-28T19:14:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.803+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a00a6340 con 0x7f63a00c1a40 2024-03-28T19:14:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.803+0000 7f6396000640 1 -- 172.21.15.27:0/2088087974 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639000b040 con 0x7f63a00c1a40 2024-03-28T19:14:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.803+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a00a6060 con 0x7f63a00c1a40 2024-03-28T19:14:20.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.804+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a004f400 con 0x7f63a00c1a40 2024-03-28T19:14:20.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.805+0000 7f6396000640 1 -- 172.21.15.27:0/2088087974 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6390019070 con 0x7f63a00c1a40 2024-03-28T19:14:20.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.807+0000 7f6396000640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f638006bbd0 0x7f638006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:20.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.807+0000 7f6396000640 1 -- 172.21.15.27:0/2088087974 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f639008b170 con 0x7f63a00c1a40 2024-03-28T19:14:20.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.807+0000 7f639de00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f638006bbd0 0x7f638006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.808+0000 7f639de00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f638006bbd0 0x7f638006e090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f638c036000 tx=0x7f638c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:20.811+0000 7f6396000640 1 -- 172.21.15.27:0/2088087974 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6390058a40 con 0x7f63a00c1a40 2024-03-28T19:14:21.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.070+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63a00c5060 con 0x7f63a00c1a40 2024-03-28T19:14:21.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.071+0000 7f6396000640 1 -- 172.21.15.27:0/2088087974 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f639005cae0 con 0x7f63a00c1a40 2024-03-28T19:14:21.070 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:21.071 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:21.071 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:21.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.073+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f638006bbd0 msgr2=0x7f638006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.073+0000 7f63a4e00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f638006bbd0 0x7f638006e090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f638c036000 tx=0x7f638c03c040 comp rx=0 tx=0).stop 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a00c1a40 msgr2=0x7f63a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a00c1a40 0x7f63a006d3c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f639000e910 tx=0x7f639000edd0 comp rx=0 tx=0).stop 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 shutdown_connections 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f638006bbd0 0x7f638006e090 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63a0155cc0 0x7f63a014ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63a00c1a40 0x7f63a006d3c0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 --2- 172.21.15.27:0/2088087974 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63a0070a50 0x7f63a01ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 >> 172.21.15.27:0/2088087974 conn(0x7f63a00c6c00 msgr2=0x7f63a00c6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:21.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 shutdown_connections 2024-03-28T19:14:21.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:21.074+0000 7f63a4e00640 1 -- 172.21.15.27:0/2088087974 wait complete. 2024-03-28T19:14:21.092 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:14:23.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:23.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.341+0000 7f85b4800640 1 Processor -- start 2024-03-28T19:14:23.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.341+0000 7f85b4800640 1 -- start start 2024-03-28T19:14:23.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.342+0000 7f85b4800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac0c5840 0x7f85ac161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.342+0000 7f85b4800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0c6250 con 0x7f85ac152ba0 2024-03-28T19:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.342+0000 7f85b4800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0c5cd0 con 0x7f85ac156390 2024-03-28T19:14:23.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.342+0000 7f85b4800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac155f00 con 0x7f85ac0c5840 2024-03-28T19:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.343+0000 7f85b1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f85ac152ba0 0x7f85ac158c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52232/0 (socket says 172.21.15.27:52232) 2024-03-28T19:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.343+0000 7f85b1e00640 1 -- 172.21.15.27:0/3184457261 learned_addr learned my addr 172.21.15.27:0/3184457261 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.343+0000 7f85b2800640 1 --2- 172.21.15.27:0/3184457261 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac0c5840 0x7f85ac161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.343+0000 7f85b0a00640 1 -- 172.21.15.27:0/3184457261 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187016978 0 0) 0x7f85a0001110 con 0x7f85ac152ba0 2024-03-28T19:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.343+0000 7f85b0a00640 1 -- 172.21.15.27:0/3184457261 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8594003520 con 0x7f85ac152ba0 2024-03-28T19:14:23.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.344+0000 7f85b0a00640 1 -- 172.21.15.27:0/3184457261 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177922503 0 0) 0x7f85ac0c5cd0 con 0x7f85ac156390 2024-03-28T19:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.344+0000 7f85b0a00640 1 -- 172.21.15.27:0/3184457261 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a0001110 con 0x7f85ac156390 2024-03-28T19:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.344+0000 7f85b2800640 1 -- 172.21.15.27:0/3184457261 >> v1:172.21.15.155:6789/0 conn(0x7f85ac156390 legacy=0x7f85ac0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.344+0000 7f85b2800640 1 -- 172.21.15.27:0/3184457261 >> v1:172.21.15.27:6789/0 conn(0x7f85ac152ba0 legacy=0x7f85ac158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.344+0000 7f85b2800640 1 -- 172.21.15.27:0/3184457261 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0c5f70 con 0x7f85ac0c5840 2024-03-28T19:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.345+0000 7f85b2800640 1 --2- 172.21.15.27:0/3184457261 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac0c5840 0x7f85ac161030 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f85ac156080 tx=0x7f85a802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b720a31622e62f64 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.346+0000 7f85b0a00640 1 -- 172.21.15.27:0/3184457261 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8039070 con 0x7f85ac0c5840 2024-03-28T19:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.346+0000 7f85b0a00640 1 -- 172.21.15.27:0/3184457261 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a802cbb0 con 0x7f85ac0c5840 2024-03-28T19:14:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.346+0000 7f85b0a00640 1 -- 172.21.15.27:0/3184457261 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8034440 con 0x7f85ac0c5840 2024-03-28T19:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.346+0000 7f85b4800640 1 -- 172.21.15.27:0/3184457261 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac0c5840 msgr2=0x7f85ac161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.346+0000 7f85b4800640 1 --2- 172.21.15.27:0/3184457261 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac0c5840 0x7f85ac161030 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f85ac156080 tx=0x7f85a802be00 comp rx=0 tx=0).stop 2024-03-28T19:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.346+0000 7f85b4800640 1 -- 172.21.15.27:0/3184457261 shutdown_connections 2024-03-28T19:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.346+0000 7f85b4800640 1 --2- 172.21.15.27:0/3184457261 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac0c5840 0x7f85ac161030 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.346+0000 7f85b4800640 1 -- 172.21.15.27:0/3184457261 >> 172.21.15.27:0/3184457261 conn(0x7f85ac066fb0 msgr2=0x7f85ac0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.347+0000 7f85b4800640 1 -- 172.21.15.27:0/3184457261 shutdown_connections 2024-03-28T19:14:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.347+0000 7f85b4800640 1 -- 172.21.15.27:0/3184457261 wait complete. 2024-03-28T19:14:23.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.348+0000 7f85b4800640 1 Processor -- start 2024-03-28T19:14:23.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.349+0000 7f85b4800640 1 -- start start 2024-03-28T19:14:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.349+0000 7f85b4800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85ac0c5840 0x7f85ac157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.349+0000 7f85b4800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85ac152ba0 0x7f85ac14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.349+0000 7f85b4800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac156390 0x7f85ac0c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b4800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac06d700 con 0x7f85ac152ba0 2024-03-28T19:14:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b4800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac155b90 con 0x7f85ac0c5840 2024-03-28T19:14:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b4800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac1593a0 con 0x7f85ac156390 2024-03-28T19:14:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85ac0c5840 0x7f85ac157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85ac152ba0 0x7f85ac14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85ac0c5840 0x7f85ac157450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:58142/0 (socket says 172.21.15.27:58142) 2024-03-28T19:14:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b1e00640 1 -- 172.21.15.27:0/1485622175 learned_addr learned my addr 172.21.15.27:0/1485622175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b2800640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac156390 0x7f85ac0c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b1e00640 1 -- 172.21.15.27:0/1485622175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac156390 msgr2=0x7f85ac0c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b1e00640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac156390 0x7f85ac0c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.350+0000 7f85b1e00640 1 -- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85ac152ba0 msgr2=0x7f85ac14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.351+0000 7f85b1e00640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85ac152ba0 0x7f85ac14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.351+0000 7f85b1e00640 1 -- 172.21.15.27:0/1485622175 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac1532a0 con 0x7f85ac0c5840 2024-03-28T19:14:23.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.351+0000 7f85b1400640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85ac152ba0 0x7f85ac14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.351+0000 7f85b1e00640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85ac0c5840 0x7f85ac157450 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c6b0 tx=0x7f85a000cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.351+0000 7f85a6000640 1 -- 172.21.15.27:0/1485622175 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0017070 con 0x7f85ac0c5840 2024-03-28T19:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.351+0000 7f85a6000640 1 -- 172.21.15.27:0/1485622175 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a0005af0 con 0x7f85ac0c5840 2024-03-28T19:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.351+0000 7f85a6000640 1 -- 172.21.15.27:0/1485622175 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0007070 con 0x7f85ac0c5840 2024-03-28T19:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.351+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac1583f0 con 0x7f85ac0c5840 2024-03-28T19:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.352+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac1589b0 con 0x7f85ac0c5840 2024-03-28T19:14:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.353+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac04f400 con 0x7f85ac0c5840 2024-03-28T19:14:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.354+0000 7f85a6000640 1 -- 172.21.15.27:0/1485622175 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f85a001a070 con 0x7f85ac0c5840 2024-03-28T19:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.355+0000 7f85a6000640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f859006bbd0 0x7f859006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.355+0000 7f85a6000640 1 -- 172.21.15.27:0/1485622175 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f85a008b2e0 con 0x7f85ac0c5840 2024-03-28T19:14:23.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.356+0000 7f85b1400640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f859006bbd0 0x7f859006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.356+0000 7f85b1400640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f859006bbd0 0x7f859006e090 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f859c007fd0 tx=0x7f859c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.359+0000 7f85a6000640 1 -- 172.21.15.27:0/1485622175 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85ac0c6250 con 0x7f85ac0c5840 2024-03-28T19:14:23.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.466+0000 7fc3cea00640 1 Processor -- start 2024-03-28T19:14:23.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.466+0000 7fc3cea00640 1 -- start start 2024-03-28T19:14:23.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.468+0000 7fc3cea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3c8153bd0 0x7fc3c80c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.468+0000 7fc3cea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c8154020 con 0x7fc3c8150c70 2024-03-28T19:14:23.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.468+0000 7fc3cea00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c80c3d90 con 0x7fc3c80a9cb0 2024-03-28T19:14:23.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.468+0000 7fc3cea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3c80ab4e0 con 0x7fc3c8153bd0 2024-03-28T19:14:23.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.468+0000 7fc3c7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc3c8150c70 0x7fc3c80c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52244/0 (socket says 172.21.15.27:52244) 2024-03-28T19:14:23.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.468+0000 7fc3c7e00640 1 -- 172.21.15.27:0/734065656 learned_addr learned my addr 172.21.15.27:0/734065656 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:23.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.468+0000 7fc3cca00640 1 --2- 172.21.15.27:0/734065656 >> v2:172.21.15.27:3300/0 conn(0x7fc3c8153bd0 0x7fc3c80c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.469+0000 7fc3cca00640 1 -- 172.21.15.27:0/734065656 >> v1:172.21.15.155:6789/0 conn(0x7fc3c80a9cb0 legacy=0x7fc3c81534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.469+0000 7fc3cca00640 1 -- 172.21.15.27:0/734065656 >> v1:172.21.15.27:6789/0 conn(0x7fc3c8150c70 legacy=0x7fc3c80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.469+0000 7fc3cca00640 1 -- 172.21.15.27:0/734065656 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80ab710 con 0x7fc3c8153bd0 2024-03-28T19:14:23.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.470+0000 7fc3cca00640 1 --2- 172.21.15.27:0/734065656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c80c3690 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009d90 tx=0x7fc3c002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e77e10166ce648 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:23.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.470+0000 7fc3c6a00640 1 -- 172.21.15.27:0/734065656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0039070 con 0x7fc3c8153bd0 2024-03-28T19:14:23.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.470+0000 7fc3c6a00640 1 -- 172.21.15.27:0/734065656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c002cc60 con 0x7fc3c8153bd0 2024-03-28T19:14:23.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.470+0000 7fc3c6a00640 1 -- 172.21.15.27:0/734065656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0034660 con 0x7fc3c8153bd0 2024-03-28T19:14:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.471+0000 7fc3cea00640 1 -- 172.21.15.27:0/734065656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 msgr2=0x7fc3c80c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.471+0000 7fc3cea00640 1 --2- 172.21.15.27:0/734065656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c80c3690 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009d90 tx=0x7fc3c002beb0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.471+0000 7fc3cea00640 1 -- 172.21.15.27:0/734065656 shutdown_connections 2024-03-28T19:14:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.471+0000 7fc3cea00640 1 --2- 172.21.15.27:0/734065656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c80c3690 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.471+0000 7fc3cea00640 1 -- 172.21.15.27:0/734065656 >> 172.21.15.27:0/734065656 conn(0x7fc3c806b9d0 msgr2=0x7fc3c80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.471+0000 7fc3cea00640 1 -- 172.21.15.27:0/734065656 shutdown_connections 2024-03-28T19:14:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.471+0000 7fc3cea00640 1 -- 172.21.15.27:0/734065656 wait complete. 2024-03-28T19:14:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.473+0000 7fc3cea00640 1 Processor -- start 2024-03-28T19:14:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.473+0000 7fc3cea00640 1 -- start start 2024-03-28T19:14:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.474+0000 7fc3cea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3c80a9cb0 0x7fc3c81e2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.474+0000 7fc3cea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3c8150c70 0x7fc3c8151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.474+0000 7fc3cea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c8151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.474+0000 7fc3cea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80abcd0 con 0x7fc3c8153bd0 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.474+0000 7fc3cea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80ac1f0 con 0x7fc3c8150c70 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.474+0000 7fc3cea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80c1810 con 0x7fc3c80a9cb0 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.474+0000 7fc3cca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c8151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3cca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c8151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40670/0 (socket says 172.21.15.27:40670) 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3cca00640 1 -- 172.21.15.27:0/6394742 learned_addr learned my addr 172.21.15.27:0/6394742 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3c7e00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3c80a9cb0 0x7fc3c81e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3c7400640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3c8150c70 0x7fc3c8151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3cca00640 1 -- 172.21.15.27:0/6394742 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3c80a9cb0 msgr2=0x7fc3c81e2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3cca00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3c80a9cb0 0x7fc3c81e2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3cca00640 1 -- 172.21.15.27:0/6394742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3c8150c70 msgr2=0x7fc3c8151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3cca00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3c8150c70 0x7fc3c8151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3cca00640 1 -- 172.21.15.27:0/6394742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c81eb240 con 0x7fc3c8153bd0 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.475+0000 7fc3cca00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c8151870 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009eb0 tx=0x7fc3c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.476+0000 7fc3bfe00640 1 -- 172.21.15.27:0/6394742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0039050 con 0x7fc3c8153bd0 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.476+0000 7fc3bfe00640 1 -- 172.21.15.27:0/6394742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0045070 con 0x7fc3c8153bd0 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.476+0000 7fc3bfe00640 1 -- 172.21.15.27:0/6394742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c003e830 con 0x7fc3c8153bd0 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.476+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c81eb520 con 0x7fc3c8153bd0 2024-03-28T19:14:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.476+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c81ebae0 con 0x7fc3c8153bd0 2024-03-28T19:14:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.477+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c80a9a70 con 0x7fc3c8153bd0 2024-03-28T19:14:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.478+0000 7fc3bfe00640 1 -- 172.21.15.27:0/6394742 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc3c003e9d0 con 0x7fc3c8153bd0 2024-03-28T19:14:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.479+0000 7fc3bfe00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3a806bbd0 0x7fc3a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.479+0000 7fc3bfe00640 1 -- 172.21.15.27:0/6394742 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7fc3c00ae680 con 0x7fc3c8153bd0 2024-03-28T19:14:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.479+0000 7fc3c7e00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3a806bbd0 0x7fc3a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.480+0000 7fc3c7e00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3a806bbd0 0x7fc3a806e090 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc3c81dcaf0 tx=0x7fc3b8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.485+0000 7fc3bfe00640 1 -- 172.21.15.27:0/6394742 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc3c80a9a70 con 0x7fc3c8153bd0 2024-03-28T19:14:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.627+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85ac158f10 con 0x7f85ac0c5840 2024-03-28T19:14:23.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.629+0000 7f85a6000640 1 -- 172.21.15.27:0/1485622175 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f85a0058cf0 con 0x7f85ac0c5840 2024-03-28T19:14:23.628 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:23.628 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:23.628 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f859006bbd0 msgr2=0x7f859006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f859006bbd0 0x7f859006e090 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f859c007fd0 tx=0x7f859c009040 comp rx=0 tx=0).stop 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85ac0c5840 msgr2=0x7f85ac157450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85ac0c5840 0x7f85ac157450 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c6b0 tx=0x7f85a000cb70 comp rx=0 tx=0).stop 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 shutdown_connections 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f859006bbd0 0x7f859006e090 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85ac156390 0x7f85ac0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85ac152ba0 0x7f85ac14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 --2- 172.21.15.27:0/1485622175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85ac0c5840 0x7f85ac157450 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 >> 172.21.15.27:0/1485622175 conn(0x7f85ac066fb0 msgr2=0x7f85ac0cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 shutdown_connections 2024-03-28T19:14:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.632+0000 7f85b4800640 1 -- 172.21.15.27:0/1485622175 wait complete. 2024-03-28T19:14:23.650 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:14:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.748+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3c804f400 con 0x7fc3c8153bd0 2024-03-28T19:14:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.749+0000 7fc3bfe00640 1 -- 172.21.15.27:0/6394742 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fc3c007c6c0 con 0x7fc3c8153bd0 2024-03-28T19:14:23.748 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:23.749 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3a806bbd0 msgr2=0x7fc3a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3a806bbd0 0x7fc3a806e090 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc3c81dcaf0 tx=0x7fc3b8009040 comp rx=0 tx=0).stop 2024-03-28T19:14:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 msgr2=0x7fc3c8151870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c8151870 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009eb0 tx=0x7fc3c0004a40 comp rx=0 tx=0).stop 2024-03-28T19:14:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 shutdown_connections 2024-03-28T19:14:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3a806bbd0 0x7fc3a806e090 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3c8153bd0 0x7fc3c8151870 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3c8150c70 0x7fc3c8151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 --2- 172.21.15.27:0/6394742 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3c80a9cb0 0x7fc3c81e2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 >> 172.21.15.27:0/6394742 conn(0x7fc3c806b9d0 msgr2=0x7fc3c80c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.752+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 shutdown_connections 2024-03-28T19:14:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:23.753+0000 7fc3cea00640 1 -- 172.21.15.27:0/6394742 wait complete. 2024-03-28T19:14:23.771 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-03-28T19:14:23.771 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:14:23.771 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:14:23.824 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:14:23.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:24.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.045+0000 7f9420a00640 1 Processor -- start 2024-03-28T19:14:24.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.045+0000 7f9420a00640 1 -- start start 2024-03-28T19:14:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.047+0000 7f9420a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f941c0c69a0 0x7f941c0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.047+0000 7f9420a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.047+0000 7f9420a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f941c150ec0 con 0x7f941c150ae0 2024-03-28T19:14:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.047+0000 7f9420a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941c06dc70 con 0x7f941c0c69a0 2024-03-28T19:14:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.047+0000 7f9420a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941c153550 con 0x7f941c153a30 2024-03-28T19:14:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.048+0000 7f941b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.048+0000 7f9419e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f941c0c69a0 0x7f941c0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.048+0000 7f941b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c153ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:60244/0 (socket says 172.21.15.27:60244) 2024-03-28T19:14:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.048+0000 7f941b200640 1 -- 172.21.15.27:0/4250367211 learned_addr learned my addr 172.21.15.27:0/4250367211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.049+0000 7f9419e00640 1 -- 172.21.15.27:0/4250367211 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 msgr2=0x7f941c153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.049+0000 7f9419e00640 1 --2- 172.21.15.27:0/4250367211 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.049+0000 7f9419e00640 1 -- 172.21.15.27:0/4250367211 >> v1:172.21.15.155:6789/0 conn(0x7f941c150ae0 legacy=0x7f941c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.049+0000 7f9419e00640 1 -- 172.21.15.27:0/4250367211 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c06df10 con 0x7f941c0c69a0 2024-03-28T19:14:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.049+0000 7f941b200640 1 --2- 172.21.15.27:0/4250367211 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:14:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.050+0000 7f9419e00640 1 --2- 172.21.15.27:0/4250367211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 0x7f941c0c6e30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f94080097a0 tx=0x7f940802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c422c0907abc16d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9419400640 1 -- 172.21.15.27:0/4250367211 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408031780 con 0x7f941c0c69a0 2024-03-28T19:14:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9419400640 1 -- 172.21.15.27:0/4250367211 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408037070 con 0x7f941c0c69a0 2024-03-28T19:14:24.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9419400640 1 -- 172.21.15.27:0/4250367211 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408031b40 con 0x7f941c0c69a0 2024-03-28T19:14:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9420a00640 1 -- 172.21.15.27:0/4250367211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 msgr2=0x7f941c0c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9420a00640 1 --2- 172.21.15.27:0/4250367211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 0x7f941c0c6e30 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f94080097a0 tx=0x7f940802b820 comp rx=0 tx=0).stop 2024-03-28T19:14:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9420a00640 1 -- 172.21.15.27:0/4250367211 shutdown_connections 2024-03-28T19:14:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9420a00640 1 --2- 172.21.15.27:0/4250367211 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9420a00640 1 --2- 172.21.15.27:0/4250367211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 0x7f941c0c6e30 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.051+0000 7f9420a00640 1 -- 172.21.15.27:0/4250367211 >> 172.21.15.27:0/4250367211 conn(0x7f941c06b0c0 msgr2=0x7f941c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.052+0000 7f9420a00640 1 -- 172.21.15.27:0/4250367211 shutdown_connections 2024-03-28T19:14:24.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.052+0000 7f9420a00640 1 -- 172.21.15.27:0/4250367211 wait complete. 2024-03-28T19:14:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.053+0000 7f9420a00640 1 Processor -- start 2024-03-28T19:14:24.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.053+0000 7f9420a00640 1 -- start start 2024-03-28T19:14:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.054+0000 7f9420a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 0x7f941c1ec560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:24.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.054+0000 7f9420a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941c150ae0 0x7f941c0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.054+0000 7f9420a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.054+0000 7f9420a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c06e7b0 con 0x7f941c150ae0 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.054+0000 7f9420a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c06e4d0 con 0x7f941c0c69a0 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.054+0000 7f9420a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941c0a6370 con 0x7f941c153a30 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f9419e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941c150ae0 0x7f941c0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f9419e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941c150ae0 0x7f941c0709d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40686/0 (socket says 172.21.15.27:40686) 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f9419e00640 1 -- 172.21.15.27:0/1343514143 learned_addr learned my addr 172.21.15.27:0/1343514143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f941b200640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f941a800640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 0x7f941c1ec560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f9419e00640 1 -- 172.21.15.27:0/1343514143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 msgr2=0x7f941c145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f9419e00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f9419e00640 1 -- 172.21.15.27:0/1343514143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 msgr2=0x7f941c1ec560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f9419e00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 0x7f941c1ec560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.055+0000 7f9419e00640 1 -- 172.21.15.27:0/1343514143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c073190 con 0x7f941c150ae0 2024-03-28T19:14:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.056+0000 7f9419e00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941c150ae0 0x7f941c0709d0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f94080098c0 tx=0x7f94080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.056+0000 7f9412a00640 1 -- 172.21.15.27:0/1343514143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408032610 con 0x7f941c150ae0 2024-03-28T19:14:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.056+0000 7f9412a00640 1 -- 172.21.15.27:0/1343514143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94080327b0 con 0x7f941c150ae0 2024-03-28T19:14:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.056+0000 7f9412a00640 1 -- 172.21.15.27:0/1343514143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408032ba0 con 0x7f941c150ae0 2024-03-28T19:14:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.056+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941c14c240 con 0x7f941c150ae0 2024-03-28T19:14:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.056+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941c073430 con 0x7f941c150ae0 2024-03-28T19:14:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.057+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941c067dd0 con 0x7f941c150ae0 2024-03-28T19:14:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.058+0000 7f9412a00640 1 -- 172.21.15.27:0/1343514143 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f940804e440 con 0x7f941c150ae0 2024-03-28T19:14:24.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.060+0000 7f9412a00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93fc06bc00 0x7f93fc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:24.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.060+0000 7f9412a00640 1 -- 172.21.15.27:0/1343514143 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f94080ad430 con 0x7f941c150ae0 2024-03-28T19:14:24.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.060+0000 7f941a800640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93fc06bc00 0x7f93fc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.061+0000 7f941a800640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93fc06bc00 0x7f93fc06e0c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f941c0c16a0 tx=0x7f940c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.065+0000 7f9412a00640 1 -- 172.21.15.27:0/1343514143 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9408040070 con 0x7f941c150ae0 2024-03-28T19:14:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.306+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f941c154cb0 con 0x7f941c150ae0 2024-03-28T19:14:24.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.308+0000 7f9412a00640 1 -- 172.21.15.27:0/1343514143 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f940802f050 con 0x7f941c150ae0 2024-03-28T19:14:24.307 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:24.307 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:24.308 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93fc06bc00 msgr2=0x7f93fc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93fc06bc00 0x7f93fc06e0c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f941c0c16a0 tx=0x7f940c009040 comp rx=0 tx=0).stop 2024-03-28T19:14:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941c150ae0 msgr2=0x7f941c0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941c150ae0 0x7f941c0709d0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f94080098c0 tx=0x7f94080046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 shutdown_connections 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f93fc06bc00 0x7f93fc06e0c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f941c153a30 0x7f941c145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f941c150ae0 0x7f941c0709d0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 --2- 172.21.15.27:0/1343514143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f941c0c69a0 0x7f941c1ec560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 >> 172.21.15.27:0/1343514143 conn(0x7f941c06b0c0 msgr2=0x7f941c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 shutdown_connections 2024-03-28T19:14:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:24.311+0000 7f9420a00640 1 -- 172.21.15.27:0/1343514143 wait complete. 2024-03-28T19:14:24.327 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:14:25.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:25.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.884+0000 7fa0b7200640 1 Processor -- start 2024-03-28T19:14:25.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.885+0000 7fa0b7200640 1 -- start start 2024-03-28T19:14:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0b7200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa0b80c58c0 0x7fa0b80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0b7200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0b80c8ed0 0x7fa0b81555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0b7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80be5d0 0x7fa0b8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0b7200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b8158bd0 con 0x7fa0b80c8ed0 2024-03-28T19:14:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0b7200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b8158eb0 con 0x7fa0b80c58c0 2024-03-28T19:14:25.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0b7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8159190 con 0x7fa0b80be5d0 2024-03-28T19:14:25.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0ad400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0b80c8ed0 0x7fa0b81555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:25.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0ad400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0b80c8ed0 0x7fa0b81555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40702/0 (socket says 172.21.15.27:40702) 2024-03-28T19:14:25.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.886+0000 7fa0ad400640 1 -- 172.21.15.27:0/140441480 learned_addr learned my addr 172.21.15.27:0/140441480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:25.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.887+0000 7fa0b6800640 1 --2- 172.21.15.27:0/140441480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80be5d0 0x7fa0b8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:25.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.887+0000 7fa0ad400640 1 -- 172.21.15.27:0/140441480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80be5d0 msgr2=0x7fa0b8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:25.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.887+0000 7fa0ad400640 1 --2- 172.21.15.27:0/140441480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80be5d0 0x7fa0b8155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:25.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.887+0000 7fa0ad400640 1 -- 172.21.15.27:0/140441480 >> v2:172.21.15.155:3300/0 conn(0x7fa0b80c58c0 msgr2=0x7fa0b80c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:25.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.887+0000 7fa0ad400640 1 --2- 172.21.15.27:0/140441480 >> v2:172.21.15.155:3300/0 conn(0x7fa0b80c58c0 0x7fa0b80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:25.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.887+0000 7fa0ad400640 1 -- 172.21.15.27:0/140441480 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8159470 con 0x7fa0b80c8ed0 2024-03-28T19:14:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.888+0000 7fa0ad400640 1 --2- 172.21.15.27:0/140441480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80c8ed0 0x7fa0b81555e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa0b806c060 tx=0x7fa0a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4d2768d379753e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.888+0000 7fa0b5400640 1 -- 172.21.15.27:0/140441480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a002e070 con 0x7fa0b80c8ed0 2024-03-28T19:14:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.888+0000 7fa0b5400640 1 -- 172.21.15.27:0/140441480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a0037070 con 0x7fa0b80c8ed0 2024-03-28T19:14:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.888+0000 7fa0b5400640 1 -- 172.21.15.27:0/140441480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0031a00 con 0x7fa0b80c8ed0 2024-03-28T19:14:25.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 -- 172.21.15.27:0/140441480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80c8ed0 msgr2=0x7fa0b81555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 --2- 172.21.15.27:0/140441480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80c8ed0 0x7fa0b81555e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa0b806c060 tx=0x7fa0a002b880 comp rx=0 tx=0).stop 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 -- 172.21.15.27:0/140441480 shutdown_connections 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 --2- 172.21.15.27:0/140441480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80be5d0 0x7fa0b8155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 --2- 172.21.15.27:0/140441480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80c8ed0 0x7fa0b81555e0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 --2- 172.21.15.27:0/140441480 >> v2:172.21.15.155:3300/0 conn(0x7fa0b80c58c0 0x7fa0b80c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 -- 172.21.15.27:0/140441480 >> 172.21.15.27:0/140441480 conn(0x7fa0b806b6f0 msgr2=0x7fa0b80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 -- 172.21.15.27:0/140441480 shutdown_connections 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.889+0000 7fa0b7200640 1 -- 172.21.15.27:0/140441480 wait complete. 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.891+0000 7fa0b7200640 1 Processor -- start 2024-03-28T19:14:25.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.891+0000 7fa0b7200640 1 -- start start 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80be5d0 0x7fa0b81ebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80c58c0 0x7fa0b80ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0b80c8ed0 0x7fa0b80c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80b09a0 con 0x7fa0b80be5d0 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80b0820 con 0x7fa0b80c8ed0 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80ad7d0 con 0x7fa0b80c58c0 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80be5d0 0x7fa0b81ebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80be5d0 0x7fa0b81ebcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40706/0 (socket says 172.21.15.27:40706) 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.892+0000 7fa0b5e00640 1 -- 172.21.15.27:0/1404111212 learned_addr learned my addr 172.21.15.27:0/1404111212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:25.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b6800640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0b80c8ed0 0x7fa0b80c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b5e00640 1 -- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80c58c0 msgr2=0x7fa0b80ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0ad400640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80c58c0 0x7fa0b80ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b5e00640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80c58c0 0x7fa0b80ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b5e00640 1 -- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0b80c8ed0 msgr2=0x7fa0b80c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b5e00640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0b80c8ed0 0x7fa0b80c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b5e00640 1 -- 172.21.15.27:0/1404111212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80cb9d0 con 0x7fa0b80be5d0 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b6800640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0b80c8ed0 0x7fa0b80c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0ad400640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80c58c0 0x7fa0b80ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:25.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b5e00640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80be5d0 0x7fa0b81ebcc0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800b4d0 tx=0x7fa0a800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0aea00640 1 -- 172.21.15.27:0/1404111212 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8014070 con 0x7fa0b80be5d0 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0aea00640 1 -- 172.21.15.27:0/1404111212 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a8010870 con 0x7fa0b80be5d0 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0aea00640 1 -- 172.21.15.27:0/1404111212 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8010c60 con 0x7fa0b80be5d0 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b80c15e0 con 0x7fa0b80be5d0 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.893+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b80c46c0 con 0x7fa0b80be5d0 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.895+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b8069640 con 0x7fa0b80be5d0 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.896+0000 7fa0aea00640 1 -- 172.21.15.27:0/1404111212 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa0a8017070 con 0x7fa0b80be5d0 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.896+0000 7fa0aea00640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa09406bbd0 0x7fa09406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:25.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.897+0000 7fa0aea00640 1 -- 172.21.15.27:0/1404111212 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7fa0a808b3e0 con 0x7fa0b80be5d0 2024-03-28T19:14:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.897+0000 7fa0ad400640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa09406bbd0 0x7fa09406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.897+0000 7fa0ad400640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa09406bbd0 0x7fa09406e090 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0036000 tx=0x7fa0a0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:25.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:25.901+0000 7fa0aea00640 1 -- 172.21.15.27:0/1404111212 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa0b80cb9d0 con 0x7fa0b80be5d0 2024-03-28T19:14:26.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.143+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0b8156580 con 0x7fa0b80be5d0 2024-03-28T19:14:26.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.144+0000 7fa0aea00640 1 -- 172.21.15.27:0/1404111212 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fa0a8059360 con 0x7fa0b80be5d0 2024-03-28T19:14:26.144 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:26.144 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:26.145 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa09406bbd0 msgr2=0x7fa09406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa09406bbd0 0x7fa09406e090 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0036000 tx=0x7fa0a0008030 comp rx=0 tx=0).stop 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80be5d0 msgr2=0x7fa0b81ebcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80be5d0 0x7fa0b81ebcc0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800b4d0 tx=0x7fa0a800b990 comp rx=0 tx=0).stop 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 shutdown_connections 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa09406bbd0 0x7fa09406e090 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0b80c8ed0 0x7fa0b80c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0b80c58c0 0x7fa0b80ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 --2- 172.21.15.27:0/1404111212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0b80be5d0 0x7fa0b81ebcc0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 >> 172.21.15.27:0/1404111212 conn(0x7fa0b806b6f0 msgr2=0x7fa0b80ae5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:26.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 shutdown_connections 2024-03-28T19:14:26.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.148+0000 7fa0b7200640 1 -- 172.21.15.27:0/1404111212 wait complete. 2024-03-28T19:14:26.167 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:14:26.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.566+0000 7f5916a00640 1 Processor -- start 2024-03-28T19:14:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.567+0000 7f5916a00640 1 -- start start 2024-03-28T19:14:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.568+0000 7f5916a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5910150a30 0x7f5910150e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.568+0000 7f5916a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f59100a7d10 0x7f59100ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.568+0000 7f5916a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59100c9820 con 0x7f59100ae6f0 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.568+0000 7f5916a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5910154130 con 0x7f5910150a30 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.568+0000 7f5916a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5910153800 con 0x7f59100a7d10 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.568+0000 7f5915600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5910150a30 0x7f5910150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.569+0000 7f5915600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5910150a30 0x7f5910150e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47896/0 (socket says 172.21.15.27:47896) 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.569+0000 7f5915600640 1 -- 172.21.15.27:0/3637931863 learned_addr learned my addr 172.21.15.27:0/3637931863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.569+0000 7f5916000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59100ae6f0 0x7f59100c6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46848/0 (socket says 172.21.15.27:46848) 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.569+0000 7f5914c00640 1 --2- 172.21.15.27:0/3637931863 >> v2:172.21.15.155:3300/0 conn(0x7f59100a7d10 0x7f59100ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.569+0000 7f5915600640 1 -- 172.21.15.27:0/3637931863 >> v2:172.21.15.155:3300/0 conn(0x7f59100a7d10 msgr2=0x7f59100ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.569+0000 7f5915600640 1 --2- 172.21.15.27:0/3637931863 >> v2:172.21.15.155:3300/0 conn(0x7f59100a7d10 0x7f59100ae1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.569+0000 7f5915600640 1 -- 172.21.15.27:0/3637931863 >> v1:172.21.15.27:6789/0 conn(0x7f59100ae6f0 legacy=0x7f59100c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:26.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.569+0000 7f5915600640 1 -- 172.21.15.27:0/3637931863 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910153ae0 con 0x7f5910150a30 2024-03-28T19:14:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.570+0000 7f5915600640 1 --2- 172.21.15.27:0/3637931863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5910150a30 0x7f5910150e10 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f59101569c0 tx=0x7f590002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5bff37a4317df20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f590fe00640 1 -- 172.21.15.27:0/3637931863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590002cb10 con 0x7f5910150a30 2024-03-28T19:14:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f590fe00640 1 -- 172.21.15.27:0/3637931863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590002ccb0 con 0x7f5910150a30 2024-03-28T19:14:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f590fe00640 1 -- 172.21.15.27:0/3637931863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590002ce60 con 0x7f5910150a30 2024-03-28T19:14:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f5916a00640 1 -- 172.21.15.27:0/3637931863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5910150a30 msgr2=0x7f5910150e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f5916a00640 1 --2- 172.21.15.27:0/3637931863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5910150a30 0x7f5910150e10 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f59101569c0 tx=0x7f590002bf70 comp rx=0 tx=0).stop 2024-03-28T19:14:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f5916a00640 1 -- 172.21.15.27:0/3637931863 shutdown_connections 2024-03-28T19:14:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f5916a00640 1 --2- 172.21.15.27:0/3637931863 >> v2:172.21.15.155:3300/0 conn(0x7f59100a7d10 0x7f59100ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f5916a00640 1 --2- 172.21.15.27:0/3637931863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5910150a30 0x7f5910150e10 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f5916a00640 1 -- 172.21.15.27:0/3637931863 >> 172.21.15.27:0/3637931863 conn(0x7f59100aa960 msgr2=0x7f59100aad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f5916a00640 1 -- 172.21.15.27:0/3637931863 shutdown_connections 2024-03-28T19:14:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.571+0000 7f5916a00640 1 -- 172.21.15.27:0/3637931863 wait complete. 2024-03-28T19:14:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.573+0000 7f5916a00640 1 Processor -- start 2024-03-28T19:14:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.573+0000 7f5916a00640 1 -- start start 2024-03-28T19:14:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5916a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59100a7d10 0x7f59100c2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5916a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59100ae6f0 0x7f59101ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5916a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5910150a30 0x7f59101ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5916a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591006e990 con 0x7f59100a7d10 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5916a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59101534f0 con 0x7f5910150a30 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5916a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591006eb10 con 0x7f59100ae6f0 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5915600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59100a7d10 0x7f59100c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5915600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59100a7d10 0x7f59100c2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47910/0 (socket says 172.21.15.27:47910) 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.574+0000 7f5915600640 1 -- 172.21.15.27:0/4197217791 learned_addr learned my addr 172.21.15.27:0/4197217791 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.575+0000 7f5914c00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59100ae6f0 0x7f59101ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.575+0000 7f5916000640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5910150a30 0x7f59101ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.575+0000 7f5915600640 1 -- 172.21.15.27:0/4197217791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59100ae6f0 msgr2=0x7f59101ee280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.575+0000 7f5915600640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59100ae6f0 0x7f59101ee280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.575+0000 7f5915600640 1 -- 172.21.15.27:0/4197217791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5910150a30 msgr2=0x7f59101ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.575+0000 7f5915600640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5910150a30 0x7f59101ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.575+0000 7f5915600640 1 -- 172.21.15.27:0/4197217791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59100ac320 con 0x7f59100a7d10 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.575+0000 7f5915600640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59100a7d10 0x7f59100c2250 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5900009f40 tx=0x7f5900004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.576+0000 7f590d600640 1 -- 172.21.15.27:0/4197217791 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900043020 con 0x7f59100a7d10 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.576+0000 7f590d600640 1 -- 172.21.15.27:0/4197217791 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590003e450 con 0x7f59100a7d10 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.576+0000 7f590d600640 1 -- 172.21.15.27:0/4197217791 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590002ce60 con 0x7f59100a7d10 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.576+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59100ac040 con 0x7f59100a7d10 2024-03-28T19:14:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.576+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59100ac8c0 con 0x7f59100a7d10 2024-03-28T19:14:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.577+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5910067dd0 con 0x7f59100a7d10 2024-03-28T19:14:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.578+0000 7f590d600640 1 -- 172.21.15.27:0/4197217791 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5900046020 con 0x7f59100a7d10 2024-03-28T19:14:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.579+0000 7f590d600640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f58f006bbd0 0x7f58f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.579+0000 7f590d600640 1 -- 172.21.15.27:0/4197217791 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f59000adcf0 con 0x7f59100a7d10 2024-03-28T19:14:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.579+0000 7f5914c00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f58f006bbd0 0x7f58f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.580+0000 7f5914c00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f58f006bbd0 0x7f58f006e090 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5910154db0 tx=0x7f58fc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.583+0000 7f590d600640 1 -- 172.21.15.27:0/4197217791 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f590004d030 con 0x7f59100a7d10 2024-03-28T19:14:26.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.828+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5910069640 con 0x7f59100a7d10 2024-03-28T19:14:26.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.829+0000 7f590d600640 1 -- 172.21.15.27:0/4197217791 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f590007bc10 con 0x7f59100a7d10 2024-03-28T19:14:26.829 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:26.829 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:26.830 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:26.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.832+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f58f006bbd0 msgr2=0x7f58f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.832+0000 7f5916a00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f58f006bbd0 0x7f58f006e090 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5910154db0 tx=0x7f58fc008040 comp rx=0 tx=0).stop 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.832+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59100a7d10 msgr2=0x7f59100c2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.832+0000 7f5916a00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59100a7d10 0x7f59100c2250 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5900009f40 tx=0x7f5900004a40 comp rx=0 tx=0).stop 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.833+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 shutdown_connections 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.833+0000 7f5916a00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f58f006bbd0 0x7f58f006e090 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.833+0000 7f5916a00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5910150a30 0x7f59101ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.833+0000 7f5916a00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59100ae6f0 0x7f59101ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.833+0000 7f5916a00640 1 --2- 172.21.15.27:0/4197217791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59100a7d10 0x7f59100c2250 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:26.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.833+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 >> 172.21.15.27:0/4197217791 conn(0x7f59100aa960 msgr2=0x7f5910151360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:26.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.833+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 shutdown_connections 2024-03-28T19:14:26.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:26.833+0000 7f5916a00640 1 -- 172.21.15.27:0/4197217791 wait complete. 2024-03-28T19:14:26.851 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:14:28.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:28.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.407+0000 7f176d000640 1 Processor -- start 2024-03-28T19:14:28.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.407+0000 7f176d000640 1 -- start start 2024-03-28T19:14:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.409+0000 7f176d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17680a9cb0 0x7f17680c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.409+0000 7f176d000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f17681534c0 0x7f1768153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.409+0000 7f176d000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17680c3d90 con 0x7f1768150c70 2024-03-28T19:14:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.409+0000 7f176d000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17680ab4e0 con 0x7f17680a9cb0 2024-03-28T19:14:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.409+0000 7f176d000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17680ac1f0 con 0x7f17681534c0 2024-03-28T19:14:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.409+0000 7f175d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17680a9cb0 0x7f17680c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.409+0000 7f175d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17680a9cb0 0x7f17680c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47918/0 (socket says 172.21.15.27:47918) 2024-03-28T19:14:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.409+0000 7f175d400640 1 -- 172.21.15.27:0/3063463571 learned_addr learned my addr 172.21.15.27:0/3063463571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.410+0000 7f1766800640 1 --2- 172.21.15.27:0/3063463571 >> v2:172.21.15.155:3300/0 conn(0x7f17681534c0 0x7f1768153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.410+0000 7f175d400640 1 -- 172.21.15.27:0/3063463571 >> v2:172.21.15.155:3300/0 conn(0x7f17681534c0 msgr2=0x7f1768153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.410+0000 7f175d400640 1 --2- 172.21.15.27:0/3063463571 >> v2:172.21.15.155:3300/0 conn(0x7f17681534c0 0x7f1768153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.410+0000 7f175d400640 1 -- 172.21.15.27:0/3063463571 >> v1:172.21.15.155:6789/0 conn(0x7f1768150c70 legacy=0x7f17680c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.410+0000 7f175d400640 1 -- 172.21.15.27:0/3063463571 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17680ab710 con 0x7f17680a9cb0 2024-03-28T19:14:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.412+0000 7f175d400640 1 --2- 172.21.15.27:0/3063463571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17680a9cb0 0x7f17680c3690 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f17680c4610 tx=0x7f175002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02529a88f1299ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.412+0000 7f1765400640 1 -- 172.21.15.27:0/3063463571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175002e070 con 0x7f17680a9cb0 2024-03-28T19:14:28.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.412+0000 7f1765400640 1 -- 172.21.15.27:0/3063463571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1750037070 con 0x7f17680a9cb0 2024-03-28T19:14:28.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.412+0000 7f1765400640 1 -- 172.21.15.27:0/3063463571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750031a00 con 0x7f17680a9cb0 2024-03-28T19:14:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.413+0000 7f176d000640 1 -- 172.21.15.27:0/3063463571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17680a9cb0 msgr2=0x7f17680c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.413+0000 7f176d000640 1 --2- 172.21.15.27:0/3063463571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17680a9cb0 0x7f17680c3690 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f17680c4610 tx=0x7f175002b880 comp rx=0 tx=0).stop 2024-03-28T19:14:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.413+0000 7f176d000640 1 -- 172.21.15.27:0/3063463571 shutdown_connections 2024-03-28T19:14:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.413+0000 7f176d000640 1 --2- 172.21.15.27:0/3063463571 >> v2:172.21.15.155:3300/0 conn(0x7f17681534c0 0x7f1768153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.413+0000 7f176d000640 1 --2- 172.21.15.27:0/3063463571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17680a9cb0 0x7f17680c3690 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.413+0000 7f176d000640 1 -- 172.21.15.27:0/3063463571 >> 172.21.15.27:0/3063463571 conn(0x7f176806b9d0 msgr2=0x7f17680c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.413+0000 7f176d000640 1 -- 172.21.15.27:0/3063463571 shutdown_connections 2024-03-28T19:14:28.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.413+0000 7f176d000640 1 -- 172.21.15.27:0/3063463571 wait complete. 2024-03-28T19:14:28.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.415+0000 7f176d000640 1 Processor -- start 2024-03-28T19:14:28.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.415+0000 7f176d000640 1 -- start start 2024-03-28T19:14:28.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f176d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17680a9cb0 0x7f176806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:28.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f176d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1768150c70 0x7f1768147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:28.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f176d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17681534c0 0x7f1768148370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:28.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f176d000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17680c1810 con 0x7f1768150c70 2024-03-28T19:14:28.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f176d000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17680abcd0 con 0x7f17680a9cb0 2024-03-28T19:14:28.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f176d000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17680c6830 con 0x7f17681534c0 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f175d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1768150c70 0x7f1768147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f175d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1768150c70 0x7f1768147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47928/0 (socket says 172.21.15.27:47928) 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f175d400640 1 -- 172.21.15.27:0/77711016 learned_addr learned my addr 172.21.15.27:0/77711016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f1765e00640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17680a9cb0 0x7f176806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.416+0000 7f1766800640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17681534c0 0x7f1768148370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175d400640 1 -- 172.21.15.27:0/77711016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17681534c0 msgr2=0x7f1768148370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175d400640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17681534c0 0x7f1768148370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175d400640 1 -- 172.21.15.27:0/77711016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17680a9cb0 msgr2=0x7f176806e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175d400640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17680a9cb0 0x7f176806e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175d400640 1 -- 172.21.15.27:0/77711016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176814f820 con 0x7f1768150c70 2024-03-28T19:14:28.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175d400640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1768150c70 0x7f1768147e30 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f1750000c00 tx=0x7f17500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:28.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175ea00640 1 -- 172.21.15.27:0/77711016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750032670 con 0x7f1768150c70 2024-03-28T19:14:28.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175ea00640 1 -- 172.21.15.27:0/77711016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1750032810 con 0x7f1768150c70 2024-03-28T19:14:28.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f175ea00640 1 -- 172.21.15.27:0/77711016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750032c00 con 0x7f1768150c70 2024-03-28T19:14:28.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176814fb00 con 0x7f1768150c70 2024-03-28T19:14:28.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.417+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176806e6f0 con 0x7f1768150c70 2024-03-28T19:14:28.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.419+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17681532e0 con 0x7f1768150c70 2024-03-28T19:14:28.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.419+0000 7f175ea00640 1 -- 172.21.15.27:0/77711016 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f175004c060 con 0x7f1768150c70 2024-03-28T19:14:28.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.420+0000 7f175ea00640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f174406bbd0 0x7f174406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:28.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.420+0000 7f175ea00640 1 -- 172.21.15.27:0/77711016 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f17500acec0 con 0x7f1768150c70 2024-03-28T19:14:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.421+0000 7f1765e00640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f174406bbd0 0x7f174406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.421+0000 7f1765e00640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f174406bbd0 0x7f174406e090 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1758005520 tx=0x7f1758009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:28.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.425+0000 7f175ea00640 1 -- 172.21.15.27:0/77711016 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f17681532e0 con 0x7f1768150c70 2024-03-28T19:14:28.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.668+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f176804f400 con 0x7f1768150c70 2024-03-28T19:14:28.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.669+0000 7f175ea00640 1 -- 172.21.15.27:0/77711016 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f1750040070 con 0x7f1768150c70 2024-03-28T19:14:28.669 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:28.669 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:28.670 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:28.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.672+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f174406bbd0 msgr2=0x7f174406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f174406bbd0 0x7f174406e090 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1758005520 tx=0x7f1758009040 comp rx=0 tx=0).stop 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1768150c70 msgr2=0x7f1768147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1768150c70 0x7f1768147e30 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f1750000c00 tx=0x7f17500046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 shutdown_connections 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f174406bbd0 0x7f174406e090 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17681534c0 0x7f1768148370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1768150c70 0x7f1768147e30 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 --2- 172.21.15.27:0/77711016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17680a9cb0 0x7f176806e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 >> 172.21.15.27:0/77711016 conn(0x7f176806b9d0 msgr2=0x7f1768151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:28.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 shutdown_connections 2024-03-28T19:14:28.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:28.673+0000 7f176d000640 1 -- 172.21.15.27:0/77711016 wait complete. 2024-03-28T19:14:28.691 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:14:28.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:29.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.082+0000 7f898b400640 1 Processor -- start 2024-03-28T19:14:29.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.082+0000 7f898b400640 1 -- start start 2024-03-28T19:14:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.084+0000 7f898b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89840732e0 0x7f898406b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.084+0000 7f898b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8984067130 0x7f898406bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.084+0000 7f898b400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898406b160 con 0x7f8984152da0 2024-03-28T19:14:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.084+0000 7f898b400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89840c2100 con 0x7f8984067130 2024-03-28T19:14:29.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.084+0000 7f898b400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8984156450 con 0x7f89840732e0 2024-03-28T19:14:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.084+0000 7f8988a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8984152da0 0x7f89840c2900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46850/0 (socket says 172.21.15.27:46850) 2024-03-28T19:14:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.084+0000 7f8988a00640 1 -- 172.21.15.27:0/3657148273 learned_addr learned my addr 172.21.15.27:0/3657148273 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.084+0000 7f8989400640 1 --2- 172.21.15.27:0/3657148273 >> v2:172.21.15.27:3300/0 conn(0x7f8984067130 0x7f898406bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.085+0000 7f8983e00640 1 --2- 172.21.15.27:0/3657148273 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89840732e0 0x7f898406b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.085+0000 7f8989400640 1 -- 172.21.15.27:0/3657148273 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89840732e0 msgr2=0x7f898406b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.085+0000 7f8989400640 1 --2- 172.21.15.27:0/3657148273 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89840732e0 0x7f898406b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.085+0000 7f8989400640 1 -- 172.21.15.27:0/3657148273 >> v1:172.21.15.27:6789/0 conn(0x7f8984152da0 legacy=0x7f89840c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:29.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.085+0000 7f8989400640 1 -- 172.21.15.27:0/3657148273 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89840c2350 con 0x7f8984067130 2024-03-28T19:14:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.086+0000 7f8989400640 1 --2- 172.21.15.27:0/3657148273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8984067130 0x7f898406bb50 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f897c0097a0 tx=0x7f897c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb8d9f2a4f92804 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.086+0000 7f8983400640 1 -- 172.21.15.27:0/3657148273 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c039070 con 0x7f8984067130 2024-03-28T19:14:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.086+0000 7f8983400640 1 -- 172.21.15.27:0/3657148273 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897c0316f0 con 0x7f8984067130 2024-03-28T19:14:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.086+0000 7f8983400640 1 -- 172.21.15.27:0/3657148273 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c031ac0 con 0x7f8984067130 2024-03-28T19:14:29.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.087+0000 7f898b400640 1 -- 172.21.15.27:0/3657148273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8984067130 msgr2=0x7f898406bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:29.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.087+0000 7f898b400640 1 --2- 172.21.15.27:0/3657148273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8984067130 0x7f898406bb50 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f897c0097a0 tx=0x7f897c02b790 comp rx=0 tx=0).stop 2024-03-28T19:14:29.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.087+0000 7f898b400640 1 -- 172.21.15.27:0/3657148273 shutdown_connections 2024-03-28T19:14:29.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.087+0000 7f898b400640 1 --2- 172.21.15.27:0/3657148273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8984067130 0x7f898406bb50 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.087+0000 7f898b400640 1 --2- 172.21.15.27:0/3657148273 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89840732e0 0x7f898406b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.087+0000 7f898b400640 1 -- 172.21.15.27:0/3657148273 >> 172.21.15.27:0/3657148273 conn(0x7f89840c3ff0 msgr2=0x7f89840c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:29.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.087+0000 7f898b400640 1 -- 172.21.15.27:0/3657148273 shutdown_connections 2024-03-28T19:14:29.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.087+0000 7f898b400640 1 -- 172.21.15.27:0/3657148273 wait complete. 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.089+0000 7f898b400640 1 Processor -- start 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.089+0000 7f898b400640 1 -- start start 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f898b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8984067130 0x7f8984148490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f898b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89840732e0 0x7f898414a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f898b400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8984152da0 0x7f89841eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f898b400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840c3010 con 0x7f89840732e0 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f898b400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89841560e0 con 0x7f8984067130 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f898b400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89840c3190 con 0x7f8984152da0 2024-03-28T19:14:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f8983e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89840732e0 0x7f898414a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f8983e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89840732e0 0x7f898414a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47948/0 (socket says 172.21.15.27:47948) 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.090+0000 7f8983e00640 1 -- 172.21.15.27:0/2182470954 learned_addr learned my addr 172.21.15.27:0/2182470954 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8988a00640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8984067130 0x7f8984148490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8989400640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8984152da0 0x7f89841eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8983e00640 1 -- 172.21.15.27:0/2182470954 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8984152da0 msgr2=0x7f89841eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8983e00640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8984152da0 0x7f89841eeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8983e00640 1 -- 172.21.15.27:0/2182470954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8984067130 msgr2=0x7f8984148490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8983e00640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8984067130 0x7f8984148490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8983e00640 1 -- 172.21.15.27:0/2182470954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984153f80 con 0x7f89840732e0 2024-03-28T19:14:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8983e00640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89840732e0 0x7f898414a270 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f897000c820 tx=0x7f897000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8980c00640 1 -- 172.21.15.27:0/2182470954 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8970015070 con 0x7f89840732e0 2024-03-28T19:14:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8980c00640 1 -- 172.21.15.27:0/2182470954 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8970004020 con 0x7f89840732e0 2024-03-28T19:14:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89841ef450 con 0x7f89840732e0 2024-03-28T19:14:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.091+0000 7f8980c00640 1 -- 172.21.15.27:0/2182470954 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89700043f0 con 0x7f89840732e0 2024-03-28T19:14:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.092+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89841ef730 con 0x7f89840732e0 2024-03-28T19:14:29.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.093+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898404f400 con 0x7f89840732e0 2024-03-28T19:14:29.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.094+0000 7f8980c00640 1 -- 172.21.15.27:0/2182470954 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8970007830 con 0x7f89840732e0 2024-03-28T19:14:29.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.095+0000 7f8980c00640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f896406bbd0 0x7f896406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:29.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.095+0000 7f8980c00640 1 -- 172.21.15.27:0/2182470954 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f897008b3d0 con 0x7f89840732e0 2024-03-28T19:14:29.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.095+0000 7f8988a00640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f896406bbd0 0x7f896406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:29.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.096+0000 7f8988a00640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f896406bbd0 0x7f896406e090 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f89740012e0 tx=0x7f8974009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:29.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.099+0000 7f8980c00640 1 -- 172.21.15.27:0/2182470954 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f89700592f0 con 0x7f89840732e0 2024-03-28T19:14:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.341+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89840715a0 con 0x7f89840732e0 2024-03-28T19:14:29.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.343+0000 7f8980c00640 1 -- 172.21.15.27:0/2182470954 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f8970058bf0 con 0x7f89840732e0 2024-03-28T19:14:29.343 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:29.343 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.345+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f896406bbd0 msgr2=0x7f896406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.345+0000 7f898b400640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f896406bbd0 0x7f896406e090 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f89740012e0 tx=0x7f8974009040 comp rx=0 tx=0).stop 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.345+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89840732e0 msgr2=0x7f898414a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.345+0000 7f898b400640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89840732e0 0x7f898414a270 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f897000c820 tx=0x7f897000cce0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.346+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 shutdown_connections 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.346+0000 7f898b400640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f896406bbd0 0x7f896406e090 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.346+0000 7f898b400640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8984152da0 0x7f89841eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.346+0000 7f898b400640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89840732e0 0x7f898414a270 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.346+0000 7f898b400640 1 --2- 172.21.15.27:0/2182470954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8984067130 0x7f8984148490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.346+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 >> 172.21.15.27:0/2182470954 conn(0x7f89840c3ff0 msgr2=0x7f898406e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:29.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.346+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 shutdown_connections 2024-03-28T19:14:29.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:29.346+0000 7f898b400640 1 -- 172.21.15.27:0/2182470954 wait complete. 2024-03-28T19:14:29.365 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:14:30.692 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.926+0000 7f4e3ee00640 1 Processor -- start 2024-03-28T19:14:30.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.926+0000 7f4e3ee00640 1 -- start start 2024-03-28T19:14:30.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3ee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3ee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e38153a30 0x7f4e38153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3ee00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e38150ec0 con 0x7f4e38150ae0 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3ee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e3806dc70 con 0x7f4e38153a30 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3ee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e38153550 con 0x7f4e380c69a0 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3da00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4e38150ae0 0x7f4e380a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46856/0 (socket says 172.21.15.27:46856) 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3da00640 1 -- 172.21.15.27:0/697502558 learned_addr learned my addr 172.21.15.27:0/697502558 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3e400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e38153a30 0x7f4e38153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.928+0000 7f4e3d000640 1 --2- 172.21.15.27:0/697502558 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.929+0000 7f4e3e400640 1 -- 172.21.15.27:0/697502558 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 msgr2=0x7f4e380c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.929+0000 7f4e3e400640 1 --2- 172.21.15.27:0/697502558 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.929+0000 7f4e3e400640 1 -- 172.21.15.27:0/697502558 >> v1:172.21.15.27:6789/0 conn(0x7f4e38150ae0 legacy=0x7f4e380a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.929+0000 7f4e3e400640 1 -- 172.21.15.27:0/697502558 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3806df10 con 0x7f4e38153a30 2024-03-28T19:14:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.930+0000 7f4e3e400640 1 --2- 172.21.15.27:0/697502558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38153a30 0x7f4e38153ea0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f4e30009d90 tx=0x7f4e3002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c23775a485ce64 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.930+0000 7f4e37e00640 1 -- 172.21.15.27:0/697502558 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30039070 con 0x7f4e38153a30 2024-03-28T19:14:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.930+0000 7f4e37e00640 1 -- 172.21.15.27:0/697502558 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3002cbd0 con 0x7f4e38153a30 2024-03-28T19:14:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.930+0000 7f4e37e00640 1 -- 172.21.15.27:0/697502558 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30034440 con 0x7f4e38153a30 2024-03-28T19:14:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.931+0000 7f4e3ee00640 1 -- 172.21.15.27:0/697502558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38153a30 msgr2=0x7f4e38153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:30.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.931+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/697502558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38153a30 0x7f4e38153ea0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f4e30009d90 tx=0x7f4e3002be20 comp rx=0 tx=0).stop 2024-03-28T19:14:30.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.931+0000 7f4e3ee00640 1 -- 172.21.15.27:0/697502558 shutdown_connections 2024-03-28T19:14:30.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.931+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/697502558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38153a30 0x7f4e38153ea0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:30.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.931+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/697502558 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:30.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.931+0000 7f4e3ee00640 1 -- 172.21.15.27:0/697502558 >> 172.21.15.27:0/697502558 conn(0x7f4e3806b0c0 msgr2=0x7f4e3806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:30.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.931+0000 7f4e3ee00640 1 -- 172.21.15.27:0/697502558 shutdown_connections 2024-03-28T19:14:30.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.931+0000 7f4e3ee00640 1 -- 172.21.15.27:0/697502558 wait complete. 2024-03-28T19:14:30.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.932+0000 7f4e3ee00640 1 Processor -- start 2024-03-28T19:14:30.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.933+0000 7f4e3ee00640 1 -- start start 2024-03-28T19:14:30.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.933+0000 7f4e3ee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380a7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:30.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.933+0000 7f4e3ee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38150ae0 0x7f4e380a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:30.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3ee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e38153a30 0x7f4e380a6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:30.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3ee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3806e7b0 con 0x7f4e38150ae0 2024-03-28T19:14:30.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3ee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3806e4d0 con 0x7f4e38153a30 2024-03-28T19:14:30.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3ee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e380a6370 con 0x7f4e380c69a0 2024-03-28T19:14:30.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38150ae0 0x7f4e380a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:30.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38150ae0 0x7f4e380a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47974/0 (socket says 172.21.15.27:47974) 2024-03-28T19:14:30.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3d000640 1 -- 172.21.15.27:0/1627344446 learned_addr learned my addr 172.21.15.27:0/1627344446 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:30.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3d000640 1 -- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 msgr2=0x7f4e380a7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:30.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3da00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380a7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:30.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3e400640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e38153a30 0x7f4e380a6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:30.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3d000640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380a7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:30.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3d000640 1 -- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e38153a30 msgr2=0x7f4e380a6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3d000640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e38153a30 0x7f4e380a6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3d000640 1 -- 172.21.15.27:0/1627344446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e380a8890 con 0x7f4e38150ae0 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.934+0000 7f4e3da00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.935+0000 7f4e3d000640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38150ae0 0x7f4e380a68b0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400d800 tx=0x7f4e2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.935+0000 7f4e35600640 1 -- 172.21.15.27:0/1627344446 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400b6c0 con 0x7f4e38150ae0 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.935+0000 7f4e35600640 1 -- 172.21.15.27:0/1627344446 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e24019070 con 0x7f4e38150ae0 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.935+0000 7f4e35600640 1 -- 172.21.15.27:0/1627344446 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400ba80 con 0x7f4e38150ae0 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.935+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e381ec080 con 0x7f4e38150ae0 2024-03-28T19:14:30.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.935+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3814f240 con 0x7f4e38150ae0 2024-03-28T19:14:30.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.937+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e38067dd0 con 0x7f4e38150ae0 2024-03-28T19:14:30.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.937+0000 7f4e35600640 1 -- 172.21.15.27:0/1627344446 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4e2400bc20 con 0x7f4e38150ae0 2024-03-28T19:14:30.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.938+0000 7f4e35600640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e1806bbd0 0x7f4e1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:30.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.938+0000 7f4e35600640 1 -- 172.21.15.27:0/1627344446 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f4e2408bab0 con 0x7f4e38150ae0 2024-03-28T19:14:30.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.939+0000 7f4e3da00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e1806bbd0 0x7f4e1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:30.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.939+0000 7f4e3da00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e1806bbd0 0x7f4e1806e090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4e380a9650 tx=0x7f4e2c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:30.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:30.943+0000 7f4e35600640 1 -- 172.21.15.27:0/1627344446 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4e380a8890 con 0x7f4e38150ae0 2024-03-28T19:14:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.187+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e38154590 con 0x7f4e38150ae0 2024-03-28T19:14:31.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.189+0000 7f4e35600640 1 -- 172.21.15.27:0/1627344446 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f4e38154590 con 0x7f4e38150ae0 2024-03-28T19:14:31.189 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:31.190 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:31.190 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.191+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e1806bbd0 msgr2=0x7f4e1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.191+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e1806bbd0 0x7f4e1806e090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4e380a9650 tx=0x7f4e2c009040 comp rx=0 tx=0).stop 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.191+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38150ae0 msgr2=0x7f4e380a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.191+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38150ae0 0x7f4e380a68b0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400d800 tx=0x7f4e2400dcc0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.192+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 shutdown_connections 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.192+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e1806bbd0 0x7f4e1806e090 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.192+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e38153a30 0x7f4e380a6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.192+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e38150ae0 0x7f4e380a68b0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.192+0000 7f4e3ee00640 1 --2- 172.21.15.27:0/1627344446 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e380c69a0 0x7f4e380a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.192+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 >> 172.21.15.27:0/1627344446 conn(0x7f4e3806b0c0 msgr2=0x7f4e381556d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.192+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 shutdown_connections 2024-03-28T19:14:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.192+0000 7f4e3ee00640 1 -- 172.21.15.27:0/1627344446 wait complete. 2024-03-28T19:14:31.211 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:14:31.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.599+0000 7fd8bca00640 1 Processor -- start 2024-03-28T19:14:31.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.599+0000 7fd8bca00640 1 -- start start 2024-03-28T19:14:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.600+0000 7fd8bca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8b80c5200 0x7fd8b80c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.600+0000 7fd8bca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd8b8151d20 0x7fd8b81521b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.600+0000 7fd8bca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b8155250 con 0x7fd8b814f3d0 2024-03-28T19:14:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8bca00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b80c1f30 con 0x7fd8b80c5200 2024-03-28T19:14:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8bca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b8155530 con 0x7fd8b8151d20 2024-03-28T19:14:31.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8b6800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd8b814f3d0 0x7fd8b80c33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46872/0 (socket says 172.21.15.27:46872) 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8b6800640 1 -- 172.21.15.27:0/2234888425 learned_addr learned my addr 172.21.15.27:0/2234888425 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8ade00640 1 --2- 172.21.15.27:0/2234888425 >> v2:172.21.15.27:3300/0 conn(0x7fd8b80c5200 0x7fd8b80c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8b7200640 1 --2- 172.21.15.27:0/2234888425 >> v2:172.21.15.155:3300/0 conn(0x7fd8b8151d20 0x7fd8b81521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8b5e00640 1 -- 172.21.15.27:0/2234888425 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311535756 0 0) 0x7fd8b8155250 con 0x7fd8b814f3d0 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8b5e00640 1 -- 172.21.15.27:0/2234888425 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd898003520 con 0x7fd8b814f3d0 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8ade00640 1 -- 172.21.15.27:0/2234888425 >> v2:172.21.15.155:3300/0 conn(0x7fd8b8151d20 msgr2=0x7fd8b81521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8ade00640 1 --2- 172.21.15.27:0/2234888425 >> v2:172.21.15.155:3300/0 conn(0x7fd8b8151d20 0x7fd8b81521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.601+0000 7fd8ade00640 1 -- 172.21.15.27:0/2234888425 >> v1:172.21.15.27:6789/0 conn(0x7fd8b814f3d0 legacy=0x7fd8b80c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.602+0000 7fd8ade00640 1 -- 172.21.15.27:0/2234888425 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b8155810 con 0x7fd8b80c5200 2024-03-28T19:14:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.603+0000 7fd8ade00640 1 --2- 172.21.15.27:0/2234888425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b80c5200 0x7fd8b80c4910 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8070700 tx=0x7fd8a002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5364ad071fa5f529 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.603+0000 7fd8b5e00640 1 -- 172.21.15.27:0/2234888425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0038070 con 0x7fd8b80c5200 2024-03-28T19:14:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.603+0000 7fd8b5e00640 1 -- 172.21.15.27:0/2234888425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0031710 con 0x7fd8b80c5200 2024-03-28T19:14:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.603+0000 7fd8b5e00640 1 -- 172.21.15.27:0/2234888425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0031ae0 con 0x7fd8b80c5200 2024-03-28T19:14:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.603+0000 7fd8bca00640 1 -- 172.21.15.27:0/2234888425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b80c5200 msgr2=0x7fd8b80c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:31.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.603+0000 7fd8bca00640 1 --2- 172.21.15.27:0/2234888425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b80c5200 0x7fd8b80c4910 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8070700 tx=0x7fd8a002b7b0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.604+0000 7fd8bca00640 1 -- 172.21.15.27:0/2234888425 shutdown_connections 2024-03-28T19:14:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.604+0000 7fd8bca00640 1 --2- 172.21.15.27:0/2234888425 >> v2:172.21.15.155:3300/0 conn(0x7fd8b8151d20 0x7fd8b81521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.604+0000 7fd8bca00640 1 --2- 172.21.15.27:0/2234888425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b80c5200 0x7fd8b80c4910 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.604+0000 7fd8bca00640 1 -- 172.21.15.27:0/2234888425 >> 172.21.15.27:0/2234888425 conn(0x7fd8b806b0c0 msgr2=0x7fd8b806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.604+0000 7fd8bca00640 1 -- 172.21.15.27:0/2234888425 shutdown_connections 2024-03-28T19:14:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.604+0000 7fd8bca00640 1 -- 172.21.15.27:0/2234888425 wait complete. 2024-03-28T19:14:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.605+0000 7fd8bca00640 1 Processor -- start 2024-03-28T19:14:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.606+0000 7fd8bca00640 1 -- start start 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8bca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8b80c5200 0x7fd8b8153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8bca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b814f3d0 0x7fd8b814cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8bca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b8151d20 0x7fd8b8150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8bca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b80c3ad0 con 0x7fd8b814f3d0 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8bca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b8155dd0 con 0x7fd8b8151d20 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8bca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b80c3c50 con 0x7fd8b80c5200 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b814f3d0 0x7fd8b814cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b814f3d0 0x7fd8b814cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47984/0 (socket says 172.21.15.27:47984) 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8ade00640 1 -- 172.21.15.27:0/1014833253 learned_addr learned my addr 172.21.15.27:0/1014833253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8ade00640 1 -- 172.21.15.27:0/1014833253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8b80c5200 msgr2=0x7fd8b8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8ade00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8b80c5200 0x7fd8b8153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8ade00640 1 -- 172.21.15.27:0/1014833253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b8151d20 msgr2=0x7fd8b8150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8ade00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b8151d20 0x7fd8b8150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.607+0000 7fd8ade00640 1 -- 172.21.15.27:0/1014833253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b81ebe50 con 0x7fd8b814f3d0 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.608+0000 7fd8ade00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b814f3d0 0x7fd8b814cbf0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009890 tx=0x7fd8a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.608+0000 7fd8af400640 1 -- 172.21.15.27:0/1014833253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0042070 con 0x7fd8b814f3d0 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.608+0000 7fd8af400640 1 -- 172.21.15.27:0/1014833253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a003d870 con 0x7fd8b814f3d0 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.608+0000 7fd8af400640 1 -- 172.21.15.27:0/1014833253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0038040 con 0x7fd8b814f3d0 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.608+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b8154360 con 0x7fd8b814f3d0 2024-03-28T19:14:31.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.608+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b81ebae0 con 0x7fd8b814f3d0 2024-03-28T19:14:31.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.609+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b81eb910 con 0x7fd8b814f3d0 2024-03-28T19:14:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.610+0000 7fd8af400640 1 -- 172.21.15.27:0/1014833253 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd8a0045070 con 0x7fd8b814f3d0 2024-03-28T19:14:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.611+0000 7fd8af400640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd89406bbd0 0x7fd89406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.611+0000 7fd8af400640 1 -- 172.21.15.27:0/1014833253 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7fd8a00ad220 con 0x7fd8b814f3d0 2024-03-28T19:14:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.612+0000 7fd8b6800640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd89406bbd0 0x7fd89406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.612+0000 7fd8b6800640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd89406bbd0 0x7fd89406e090 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8000f60 tx=0x7fd8a80078f0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:31.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.616+0000 7fd8af400640 1 -- 172.21.15.27:0/1014833253 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd8a007b1a0 con 0x7fd8b814f3d0 2024-03-28T19:14:31.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.862+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8b8067dd0 con 0x7fd8b814f3d0 2024-03-28T19:14:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.863+0000 7fd8af400640 1 -- 172.21.15.27:0/1014833253 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fd8a007aaa0 con 0x7fd8b814f3d0 2024-03-28T19:14:31.864 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:31.864 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:31.864 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd89406bbd0 msgr2=0x7fd89406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd89406bbd0 0x7fd89406e090 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8000f60 tx=0x7fd8a80078f0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b814f3d0 msgr2=0x7fd8b814cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b814f3d0 0x7fd8b814cbf0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009890 tx=0x7fd8a00046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 shutdown_connections 2024-03-28T19:14:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd89406bbd0 0x7fd89406e090 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b8151d20 0x7fd8b8150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b814f3d0 0x7fd8b814cbf0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 --2- 172.21.15.27:0/1014833253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8b80c5200 0x7fd8b8153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 >> 172.21.15.27:0/1014833253 conn(0x7fd8b806b0c0 msgr2=0x7fd8b80a8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 shutdown_connections 2024-03-28T19:14:31.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:31.866+0000 7fd8bca00640 1 -- 172.21.15.27:0/1014833253 wait complete. 2024-03-28T19:14:31.886 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:14:33.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:33.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:14:33.265 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:14:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.438+0000 7feb03e00640 1 Processor -- start 2024-03-28T19:14:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.438+0000 7feb03e00640 1 -- start start 2024-03-28T19:14:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.440+0000 7feb03e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb040c7580 0x7feb0406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.440+0000 7feb03e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7feb04156ba0 0x7feb04157050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.440+0000 7feb03e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0406d080 con 0x7feb040c6a90 2024-03-28T19:14:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.440+0000 7feb03e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb04159980 con 0x7feb040c7580 2024-03-28T19:14:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.440+0000 7feb03e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0406cb00 con 0x7feb04156ba0 2024-03-28T19:14:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.440+0000 7feb02a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feb040c6a90 0x7feb040c6e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46884/0 (socket says 172.21.15.27:46884) 2024-03-28T19:14:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.440+0000 7feb02a00640 1 -- 172.21.15.27:0/1221790795 learned_addr learned my addr 172.21.15.27:0/1221790795 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.440+0000 7feafa000640 1 --2- 172.21.15.27:0/1221790795 >> v2:172.21.15.27:3300/0 conn(0x7feb040c7580 0x7feb0406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.441+0000 7feb03400640 1 --2- 172.21.15.27:0/1221790795 >> v2:172.21.15.155:3300/0 conn(0x7feb04156ba0 0x7feb04157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:33.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.441+0000 7feafa000640 1 -- 172.21.15.27:0/1221790795 >> v2:172.21.15.155:3300/0 conn(0x7feb04156ba0 msgr2=0x7feb04157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:33.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.441+0000 7feb02000640 1 -- 172.21.15.27:0/1221790795 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649196812 0 0) 0x7feb0406d080 con 0x7feb040c6a90 2024-03-28T19:14:33.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.441+0000 7feafa000640 1 --2- 172.21.15.27:0/1221790795 >> v2:172.21.15.155:3300/0 conn(0x7feb04156ba0 0x7feb04157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.441+0000 7feafa000640 1 -- 172.21.15.27:0/1221790795 >> v1:172.21.15.27:6789/0 conn(0x7feb040c6a90 legacy=0x7feb040c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:33.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.441+0000 7feafa000640 1 -- 172.21.15.27:0/1221790795 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0406cda0 con 0x7feb040c7580 2024-03-28T19:14:33.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.442+0000 7feafa000640 1 --2- 172.21.15.27:0/1221790795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb040c7580 0x7feb0406b0c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7feb0406c220 tx=0x7feaec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fd8dda26e54b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:33.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.442+0000 7feb02000640 1 -- 172.21.15.27:0/1221790795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec035020 con 0x7feb040c7580 2024-03-28T19:14:33.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.442+0000 7feb02000640 1 -- 172.21.15.27:0/1221790795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaec031830 con 0x7feb040c7580 2024-03-28T19:14:33.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.442+0000 7feb02000640 1 -- 172.21.15.27:0/1221790795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec031c00 con 0x7feb040c7580 2024-03-28T19:14:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.443+0000 7feb03e00640 1 -- 172.21.15.27:0/1221790795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb040c7580 msgr2=0x7feb0406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.443+0000 7feb03e00640 1 --2- 172.21.15.27:0/1221790795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb040c7580 0x7feb0406b0c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7feb0406c220 tx=0x7feaec02b8d0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.443+0000 7feb03e00640 1 -- 172.21.15.27:0/1221790795 shutdown_connections 2024-03-28T19:14:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.443+0000 7feb03e00640 1 --2- 172.21.15.27:0/1221790795 >> v2:172.21.15.155:3300/0 conn(0x7feb04156ba0 0x7feb04157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.443+0000 7feb03e00640 1 --2- 172.21.15.27:0/1221790795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb040c7580 0x7feb0406b0c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.443+0000 7feb03e00640 1 -- 172.21.15.27:0/1221790795 >> 172.21.15.27:0/1221790795 conn(0x7feb040aa7c0 msgr2=0x7feb040aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.443+0000 7feb03e00640 1 -- 172.21.15.27:0/1221790795 shutdown_connections 2024-03-28T19:14:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.443+0000 7feb03e00640 1 -- 172.21.15.27:0/1221790795 wait complete. 2024-03-28T19:14:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.445+0000 7feb03e00640 1 Processor -- start 2024-03-28T19:14:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.445+0000 7feb03e00640 1 -- start start 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb040c6a90 0x7feb040c0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb040c7580 0x7feb041544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb04156ba0 0x7feb04154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb04152e90 con 0x7feb04156ba0 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb040c8630 con 0x7feb040c6a90 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb04157590 con 0x7feb040c7580 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb04156ba0 0x7feb04154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb04156ba0 0x7feb04154a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47992/0 (socket says 172.21.15.27:47992) 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.446+0000 7feb03400640 1 -- 172.21.15.27:0/3768506463 learned_addr learned my addr 172.21.15.27:0/3768506463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feb02a00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb040c6a90 0x7feb040c0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feafa000640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb040c7580 0x7feb041544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feb03400640 1 -- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb040c7580 msgr2=0x7feb041544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:33.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feb03400640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb040c7580 0x7feb041544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feb03400640 1 -- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb040c6a90 msgr2=0x7feb040c0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feb03400640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb040c6a90 0x7feb040c0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feb03400640 1 -- 172.21.15.27:0/3768506463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb040ae310 con 0x7feb04156ba0 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feafa000640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb040c7580 0x7feb041544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.447+0000 7feb03400640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb04156ba0 0x7feb04154a10 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7feafc00c580 tx=0x7feafc00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.448+0000 7feafb400640 1 -- 172.21.15.27:0/3768506463 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc015070 con 0x7feb04156ba0 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.448+0000 7feafb400640 1 -- 172.21.15.27:0/3768506463 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc004220 con 0x7feb04156ba0 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.448+0000 7feafb400640 1 -- 172.21.15.27:0/3768506463 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc003610 con 0x7feb04156ba0 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.448+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb040ad270 con 0x7feb04156ba0 2024-03-28T19:14:33.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.448+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb040ad830 con 0x7feb04156ba0 2024-03-28T19:14:33.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.449+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb04067dd0 con 0x7feb04156ba0 2024-03-28T19:14:33.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.450+0000 7feafb400640 1 -- 172.21.15.27:0/3768506463 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feafc0037b0 con 0x7feb04156ba0 2024-03-28T19:14:33.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.451+0000 7feafb400640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feae006bc00 0x7feae006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:33.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.451+0000 7feafb400640 1 -- 172.21.15.27:0/3768506463 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7feafc08b450 con 0x7feb04156ba0 2024-03-28T19:14:33.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.451+0000 7feb02a00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feae006bc00 0x7feae006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:33.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.452+0000 7feb02a00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feae006bc00 0x7feae006e0c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7feaf40012e0 tx=0x7feaf4007ed0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:33.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.455+0000 7feafb400640 1 -- 172.21.15.27:0/3768506463 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feafc0592f0 con 0x7feb04156ba0 2024-03-28T19:14:33.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.707+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb04069640 con 0x7feb04156ba0 2024-03-28T19:14:33.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.708+0000 7feafb400640 1 -- 172.21.15.27:0/3768506463 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7feafc058bf0 con 0x7feb04156ba0 2024-03-28T19:14:33.708 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:33.708 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:33.709 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.710+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feae006bc00 msgr2=0x7feae006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.710+0000 7feb03e00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feae006bc00 0x7feae006e0c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7feaf40012e0 tx=0x7feaf4007ed0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.710+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb04156ba0 msgr2=0x7feb04154a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.710+0000 7feb03e00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb04156ba0 0x7feb04154a10 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7feafc00c580 tx=0x7feafc00ca40 comp rx=0 tx=0).stop 2024-03-28T19:14:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.711+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 shutdown_connections 2024-03-28T19:14:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.711+0000 7feb03e00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feae006bc00 0x7feae006e0c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.711+0000 7feb03e00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb04156ba0 0x7feb04154a10 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.711+0000 7feb03e00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb040c7580 0x7feb041544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.711+0000 7feb03e00640 1 --2- 172.21.15.27:0/3768506463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb040c6a90 0x7feb040c0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.711+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 >> 172.21.15.27:0/3768506463 conn(0x7feb040aa7c0 msgr2=0x7feb041587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.711+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 shutdown_connections 2024-03-28T19:14:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:33.711+0000 7feb03e00640 1 -- 172.21.15.27:0/3768506463 wait complete. 2024-03-28T19:14:33.730 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:14:33.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:34.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.122+0000 7f69a5800640 1 Processor -- start 2024-03-28T19:14:34.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.122+0000 7f69a5800640 1 -- start start 2024-03-28T19:14:34.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.124+0000 7f69a5800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f69a00727e0 0x7f69a00c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:34.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.124+0000 7f69a5800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a0155fe0 con 0x7f69a00c59d0 2024-03-28T19:14:34.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.124+0000 7f69a5800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a01562c0 con 0x7f69a0155600 2024-03-28T19:14:34.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.124+0000 7f69a5800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a01565a0 con 0x7f69a00727e0 2024-03-28T19:14:34.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.124+0000 7f699fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f69a00c59d0 0x7f69a0159520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46898/0 (socket says 172.21.15.27:46898) 2024-03-28T19:14:34.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.124+0000 7f699fe00640 1 -- 172.21.15.27:0/2146323591 learned_addr learned my addr 172.21.15.27:0/2146323591 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:34.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699f400640 1 --2- 172.21.15.27:0/2146323591 >> v2:172.21.15.155:3300/0 conn(0x7f69a00727e0 0x7f69a00c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:34.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262794445 0 0) 0x7f69a0155fe0 con 0x7f69a00c59d0 2024-03-28T19:14:34.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6984003520 con 0x7f69a00c59d0 2024-03-28T19:14:34.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759891485 0 0) 0x7f69a01562c0 con 0x7f69a0155600 2024-03-28T19:14:34.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a0155fe0 con 0x7f69a0155600 2024-03-28T19:14:34.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306369486 0 0) 0x7f6984003520 con 0x7f69a00c59d0 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 >> v2:172.21.15.155:3300/0 conn(0x7f69a00727e0 msgr2=0x7f69a00c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 --2- 172.21.15.27:0/2146323591 >> v2:172.21.15.155:3300/0 conn(0x7f69a00727e0 0x7f69a00c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 >> v1:172.21.15.155:6789/0 conn(0x7f69a0155600 legacy=0x7f69a014f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a0158d20 con 0x7f69a00c59d0 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f6990001670 con 0x7f69a00c59d0 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.125+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 >> v1:172.21.15.27:6789/0 conn(0x7f69a00c59d0 legacy=0x7f69a0159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699ea00640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984003b90 0x7f6984003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699ea00640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6984004800 0x7f6984004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:34.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699ea00640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6984005280 0x7f698400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:34.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a01565a0 con 0x7f6984004800 2024-03-28T19:14:34.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698400c1c0 con 0x7f6984003b90 2024-03-28T19:14:34.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698400c3c0 con 0x7f6984005280 2024-03-28T19:14:34.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699f400640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6984004800 0x7f6984004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:34.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699fe00640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984003b90 0x7f6984003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:34.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f69a4e00640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6984005280 0x7f698400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:34.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.126+0000 7f699f400640 1 -- 172.21.15.27:0/2146323591 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6984005280 msgr2=0x7f698400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.127+0000 7f699f400640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6984005280 0x7f698400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.127+0000 7f699f400640 1 -- 172.21.15.27:0/2146323591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984003b90 msgr2=0x7f6984003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.127+0000 7f699f400640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984003b90 0x7f6984003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.127+0000 7f699f400640 1 -- 172.21.15.27:0/2146323591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6984003520 con 0x7f6984004800 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.127+0000 7f69a4e00640 1 -- 172.21.15.27:0/2146323591 reap_dead start 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.128+0000 7f699f400640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6984004800 0x7f6984004c80 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f698c009a90 tx=0x7f6984005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5c1f45caa33336 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.128+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c039070 con 0x7f6984004800 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.128+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c02c680 con 0x7f6984004800 2024-03-28T19:14:34.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.128+0000 7f699ea00640 1 -- 172.21.15.27:0/2146323591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c009690 con 0x7f6984004800 2024-03-28T19:14:34.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.129+0000 7f69a5800640 1 -- 172.21.15.27:0/2146323591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6984004800 msgr2=0x7f6984004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:34.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.129+0000 7f69a5800640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6984004800 0x7f6984004c80 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f698c009a90 tx=0x7f6984005830 comp rx=0 tx=0).stop 2024-03-28T19:14:34.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.129+0000 7f69a5800640 1 -- 172.21.15.27:0/2146323591 shutdown_connections 2024-03-28T19:14:34.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.129+0000 7f69a5800640 1 --2- 172.21.15.27:0/2146323591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6984004800 0x7f6984004c80 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.129+0000 7f69a5800640 1 -- 172.21.15.27:0/2146323591 >> 172.21.15.27:0/2146323591 conn(0x7f69a00ac910 msgr2=0x7f69a00accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:34.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.129+0000 7f69a5800640 1 -- 172.21.15.27:0/2146323591 shutdown_connections 2024-03-28T19:14:34.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.129+0000 7f69a5800640 1 -- 172.21.15.27:0/2146323591 wait complete. 2024-03-28T19:14:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.131+0000 7f69a5800640 1 Processor -- start 2024-03-28T19:14:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.131+0000 7f69a5800640 1 -- start start 2024-03-28T19:14:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984004800 0x7f69a00beb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a5800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69a014a250 0x7f69a00bf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69a0149200 0x7f69a0149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a5800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a0159090 con 0x7f69a0149200 2024-03-28T19:14:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a5800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a00c5fc0 con 0x7f6984004800 2024-03-28T19:14:34.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a5800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a0155a50 con 0x7f69a014a250 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69a0149200 0x7f69a0149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69a0149200 0x7f69a0149690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48002/0 (socket says 172.21.15.27:48002) 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f69a4e00640 1 -- 172.21.15.27:0/382844562 learned_addr learned my addr 172.21.15.27:0/382844562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f699fe00640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984004800 0x7f69a00beb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.132+0000 7f699f400640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69a014a250 0x7f69a00bf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f69a4e00640 1 -- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69a014a250 msgr2=0x7f69a00bf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f69a4e00640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69a014a250 0x7f69a00bf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f69a4e00640 1 -- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984004800 msgr2=0x7f69a00beb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f69a4e00640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984004800 0x7f69a00beb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f69a4e00640 1 -- 172.21.15.27:0/382844562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a00c2090 con 0x7f69a0149200 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f699fe00640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984004800 0x7f69a00beb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f69a4e00640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69a0149200 0x7f69a0149690 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f69a0156770 tx=0x7f699800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f6997e00640 1 -- 172.21.15.27:0/382844562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699800ee50 con 0x7f69a0149200 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f6997e00640 1 -- 172.21.15.27:0/382844562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6998018070 con 0x7f69a0149200 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f6997e00640 1 -- 172.21.15.27:0/382844562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998003dc0 con 0x7f69a0149200 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.133+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a01ebae0 con 0x7f69a0149200 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.134+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a00c4980 con 0x7f69a0149200 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.135+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a0067dd0 con 0x7f69a0149200 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.135+0000 7f6997e00640 1 -- 172.21.15.27:0/382844562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f69980050d0 con 0x7f69a0149200 2024-03-28T19:14:34.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.137+0000 7f6997e00640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f698006bc00 0x7f698006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.137+0000 7f6997e00640 1 -- 172.21.15.27:0/382844562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8701+0+0 (secure 0 0 0) 0x7f699808b430 con 0x7f69a0149200 2024-03-28T19:14:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.137+0000 7f699fe00640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f698006bc00 0x7f698006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:34.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.138+0000 7f699fe00640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f698006bc00 0x7f698006e0c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6990001eb0 tx=0x7f6990001cc0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:34.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.143+0000 7f6997e00640 1 -- 172.21.15.27:0/382844562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6998059350 con 0x7f69a0149200 2024-03-28T19:14:34.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.384+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69a0069640 con 0x7f69a0149200 2024-03-28T19:14:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.385+0000 7f6997e00640 1 -- 172.21.15.27:0/382844562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f6998058c50 con 0x7f69a0149200 2024-03-28T19:14:34.386 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:34.386 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136},{"gid":7146,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:01.574607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6532":{"gid":6532,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6837/552369694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":552369694},{"type":"v1","addr":"172.21.15.27:6837","nonce":552369694}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6532,"qdb_cluster":[6532]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 143 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f698006bc00 msgr2=0x7f698006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f698006bc00 0x7f698006e0c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6990001eb0 tx=0x7f6990001cc0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69a0149200 msgr2=0x7f69a0149690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69a0149200 0x7f69a0149690 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f69a0156770 tx=0x7f699800ec00 comp rx=0 tx=0).stop 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 shutdown_connections 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f698006bc00 0x7f698006e0c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69a0149200 0x7f69a0149690 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69a014a250 0x7f69a00bf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 --2- 172.21.15.27:0/382844562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6984004800 0x7f69a00beb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.388+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 >> 172.21.15.27:0/382844562 conn(0x7f69a00ac910 msgr2=0x7f69a0154e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.389+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 shutdown_connections 2024-03-28T19:14:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:34.389+0000 7f69a5800640 1 -- 172.21.15.27:0/382844562 wait complete. 2024-03-28T19:14:34.409 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:14:35.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:35.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.920+0000 7f94cf200640 1 Processor -- start 2024-03-28T19:14:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.920+0000 7f94cf200640 1 -- start start 2024-03-28T19:14:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.922+0000 7f94cf200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94c80c58c0 0x7f94c80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.922+0000 7f94cf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80c8ed0 0x7f94c81555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.922+0000 7f94cf200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c8156230 con 0x7f94c80be5d0 2024-03-28T19:14:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.922+0000 7f94cf200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c8158bd0 con 0x7f94c80c58c0 2024-03-28T19:14:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.922+0000 7f94cf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c8158eb0 con 0x7f94c80c8ed0 2024-03-28T19:14:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.922+0000 7f94cde00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94c80c58c0 0x7f94c80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.922+0000 7f94ce800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f94c80be5d0 0x7f94c8155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46908/0 (socket says 172.21.15.27:46908) 2024-03-28T19:14:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.922+0000 7f94ce800640 1 -- 172.21.15.27:0/1387013065 learned_addr learned my addr 172.21.15.27:0/1387013065 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.923+0000 7f94cd400640 1 --2- 172.21.15.27:0/1387013065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80c8ed0 0x7f94c81555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.923+0000 7f94cde00640 1 -- 172.21.15.27:0/1387013065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80c8ed0 msgr2=0x7f94c81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.923+0000 7f94cde00640 1 --2- 172.21.15.27:0/1387013065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80c8ed0 0x7f94c81555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.923+0000 7f94cde00640 1 -- 172.21.15.27:0/1387013065 >> v1:172.21.15.27:6789/0 conn(0x7f94c80be5d0 legacy=0x7f94c8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.923+0000 7f94cde00640 1 -- 172.21.15.27:0/1387013065 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c8159190 con 0x7f94c80c58c0 2024-03-28T19:14:35.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.923+0000 7f94cca00640 1 -- 172.21.15.27:0/1387013065 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158813237 0 0) 0x7f94c8156230 con 0x7f94c80be5d0 2024-03-28T19:14:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.925+0000 7f94cde00640 1 --2- 172.21.15.27:0/1387013065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c58c0 0x7f94c80c5ca0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0097a0 tx=0x7f94bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4528c49fda67c224 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.925+0000 7f94cca00640 1 -- 172.21.15.27:0/1387013065 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc0315a0 con 0x7f94c80c58c0 2024-03-28T19:14:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.925+0000 7f94cca00640 1 -- 172.21.15.27:0/1387013065 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc037070 con 0x7f94c80c58c0 2024-03-28T19:14:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.925+0000 7f94cca00640 1 -- 172.21.15.27:0/1387013065 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc031960 con 0x7f94c80c58c0 2024-03-28T19:14:35.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.926+0000 7f94cf200640 1 -- 172.21.15.27:0/1387013065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c58c0 msgr2=0x7f94c80c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.926+0000 7f94cf200640 1 --2- 172.21.15.27:0/1387013065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c58c0 0x7f94c80c5ca0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0097a0 tx=0x7f94bc02b840 comp rx=0 tx=0).stop 2024-03-28T19:14:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.926+0000 7f94cf200640 1 -- 172.21.15.27:0/1387013065 shutdown_connections 2024-03-28T19:14:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.926+0000 7f94cf200640 1 --2- 172.21.15.27:0/1387013065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80c8ed0 0x7f94c81555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.926+0000 7f94cf200640 1 --2- 172.21.15.27:0/1387013065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c58c0 0x7f94c80c5ca0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.926+0000 7f94cf200640 1 -- 172.21.15.27:0/1387013065 >> 172.21.15.27:0/1387013065 conn(0x7f94c806b6f0 msgr2=0x7f94c80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.926+0000 7f94cf200640 1 -- 172.21.15.27:0/1387013065 shutdown_connections 2024-03-28T19:14:35.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.926+0000 7f94cf200640 1 -- 172.21.15.27:0/1387013065 wait complete. 2024-03-28T19:14:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.928+0000 7f94cf200640 1 Processor -- start 2024-03-28T19:14:35.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.928+0000 7f94cf200640 1 -- start start 2024-03-28T19:14:35.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94cf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80be5d0 0x7f94c8152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:35.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94cf200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94c80c58c0 0x7f94c80c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:35.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94cf200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c8ed0 0x7f94c8151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:35.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94cf200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80b0820 con 0x7f94c80c8ed0 2024-03-28T19:14:35.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94cf200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c8159750 con 0x7f94c80c58c0 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94cf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80b09a0 con 0x7f94c80be5d0 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94ce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c8ed0 0x7f94c8151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94ce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c8ed0 0x7f94c8151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48010/0 (socket says 172.21.15.27:48010) 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94ce800640 1 -- 172.21.15.27:0/766297165 learned_addr learned my addr 172.21.15.27:0/766297165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94cde00640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80be5d0 0x7f94c8152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.929+0000 7f94cd400640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94c80c58c0 0x7f94c80c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94ce800640 1 -- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80be5d0 msgr2=0x7f94c8152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94ce800640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80be5d0 0x7f94c8152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94ce800640 1 -- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94c80c58c0 msgr2=0x7f94c80c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:35.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94ce800640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94c80c58c0 0x7f94c80c2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94ce800640 1 -- 172.21.15.27:0/766297165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c81ec220 con 0x7f94c80c8ed0 2024-03-28T19:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94cde00640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80be5d0 0x7f94c8152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94cd400640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94c80c58c0 0x7f94c80c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:14:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94ce800640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c8ed0 0x7f94c8151600 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f94c400c700 tx=0x7f94c400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94c2000640 1 -- 172.21.15.27:0/766297165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4018070 con 0x7f94c80c8ed0 2024-03-28T19:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.930+0000 7f94c2000640 1 -- 172.21.15.27:0/766297165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c400ce10 con 0x7f94c80c8ed0 2024-03-28T19:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.931+0000 7f94c2000640 1 -- 172.21.15.27:0/766297165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c400a820 con 0x7f94c80c8ed0 2024-03-28T19:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.931+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c8153360 con 0x7f94c80c8ed0 2024-03-28T19:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.931+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c8152280 con 0x7f94c80c8ed0 2024-03-28T19:14:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.932+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c8072c20 con 0x7f94c80c8ed0 2024-03-28T19:14:35.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.934+0000 7f94c2000640 1 -- 172.21.15.27:0/766297165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f94c401b070 con 0x7f94c80c8ed0 2024-03-28T19:14:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.935+0000 7f94c2000640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94ac06bbd0 0x7f94ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.935+0000 7f94c2000640 1 -- 172.21.15.27:0/766297165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8787+0+0 (secure 0 0 0) 0x7f94c408ab30 con 0x7f94c80c8ed0 2024-03-28T19:14:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.935+0000 7f94cde00640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94ac06bbd0 0x7f94ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.936+0000 7f94cde00640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94ac06bbd0 0x7f94ac06e090 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f94bc036000 tx=0x7f94bc03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:35.944+0000 7f94c2000640 1 -- 172.21.15.27:0/766297165 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f94c8072c20 con 0x7f94c80c8ed0 2024-03-28T19:14:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.197+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94c8069640 con 0x7f94c80c8ed0 2024-03-28T19:14:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.198+0000 7f94c2000640 1 -- 172.21.15.27:0/766297165 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f94c4058a50 con 0x7f94c80c8ed0 2024-03-28T19:14:36.199 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:36.199 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:35.095404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 144 2024-03-28T19:14:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94ac06bbd0 msgr2=0x7f94ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94ac06bbd0 0x7f94ac06e090 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f94bc036000 tx=0x7f94bc03d040 comp rx=0 tx=0).stop 2024-03-28T19:14:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c8ed0 msgr2=0x7f94c8151600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c8ed0 0x7f94c8151600 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f94c400c700 tx=0x7f94c400cbc0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 shutdown_connections 2024-03-28T19:14:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94ac06bbd0 0x7f94ac06e090 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94c80c8ed0 0x7f94c8151600 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94c80c58c0 0x7f94c80c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 --2- 172.21.15.27:0/766297165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94c80be5d0 0x7f94c8152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.201+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 >> 172.21.15.27:0/766297165 conn(0x7f94c806b6f0 msgr2=0x7f94c8071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.202+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 shutdown_connections 2024-03-28T19:14:36.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.202+0000 7f94cf200640 1 -- 172.21.15.27:0/766297165 wait complete. 2024-03-28T19:14:36.221 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-03-28T19:14:36.221 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:14:36.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:36.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.429+0000 7f5ce9600640 1 Processor -- start 2024-03-28T19:14:36.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.429+0000 7f5ce9600640 1 -- start start 2024-03-28T19:14:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.431+0000 7f5ce9600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5ce40be5d0 0x7f5ce40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.431+0000 7f5ce9600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce4158bd0 con 0x7f5ce40c8ed0 2024-03-28T19:14:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.431+0000 7f5ce9600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce4158eb0 con 0x7f5ce40c58c0 2024-03-28T19:14:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.431+0000 7f5ce9600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce4159190 con 0x7f5ce40be5d0 2024-03-28T19:14:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.431+0000 7f5ce3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5ce40c8ed0 0x7f5ce4155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56452/0 (socket says 172.21.15.27:56452) 2024-03-28T19:14:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.431+0000 7f5ce3400640 1 -- 172.21.15.27:0/3873824626 learned_addr learned my addr 172.21.15.27:0/3873824626 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:36.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce8c00640 1 --2- 172.21.15.27:0/3873824626 >> v2:172.21.15.155:3300/0 conn(0x7f5ce40be5d0 0x7f5ce40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464468484 0 0) 0x7f5ce4158bd0 con 0x7f5ce40c8ed0 2024-03-28T19:14:36.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cc8003520 con 0x7f5ce40c8ed0 2024-03-28T19:14:36.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4224587714 0 0) 0x7f5cc8003520 con 0x7f5ce40c8ed0 2024-03-28T19:14:36.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 >> v2:172.21.15.155:3300/0 conn(0x7f5ce40be5d0 msgr2=0x7f5ce40c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 --2- 172.21.15.27:0/3873824626 >> v2:172.21.15.155:3300/0 conn(0x7f5ce40be5d0 0x7f5ce40c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 >> v1:172.21.15.155:6789/0 conn(0x7f5ce40c58c0 legacy=0x7f5ce41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce4159470 con 0x7f5ce40c8ed0 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f5cd0001670 con 0x7f5ce40c8ed0 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 >> v1:172.21.15.27:6789/0 conn(0x7f5ce40c8ed0 legacy=0x7f5ce4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.432+0000 7f5ce2a00640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8003a00 0x7f5cc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce2a00640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cc8004800 0x7f5cc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce2a00640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc80052e0 0x7f5cc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4159190 con 0x7f5cc8003a00 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc800c1e0 con 0x7f5cc8004800 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc800c3e0 con 0x7f5cc80052e0 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce3e00640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8003a00 0x7f5cc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce3400640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cc8004800 0x7f5cc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce3e00640 1 -- 172.21.15.27:0/3873824626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc80052e0 msgr2=0x7f5cc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce3e00640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc80052e0 0x7f5cc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce3e00640 1 -- 172.21.15.27:0/3873824626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cc8004800 msgr2=0x7f5cc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce3e00640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cc8004800 0x7f5cc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce3e00640 1 -- 172.21.15.27:0/3873824626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc8003520 con 0x7f5cc8003a00 2024-03-28T19:14:36.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.433+0000 7f5ce3400640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cc8004800 0x7f5cc8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.434+0000 7f5ce8c00640 1 -- 172.21.15.27:0/3873824626 reap_dead start 2024-03-28T19:14:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.435+0000 7f5ce3e00640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8003a00 0x7f5cc8003e20 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4001750 tx=0x7f5cc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e002df563bf7c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.435+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4001370 con 0x7f5cc8003a00 2024-03-28T19:14:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.435+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd40052d0 con 0x7f5cc8003a00 2024-03-28T19:14:36.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.435+0000 7f5ce2a00640 1 -- 172.21.15.27:0/3873824626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4003830 con 0x7f5cc8003a00 2024-03-28T19:14:36.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.436+0000 7f5ce9600640 1 -- 172.21.15.27:0/3873824626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8003a00 msgr2=0x7f5cc8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.436+0000 7f5ce9600640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8003a00 0x7f5cc8003e20 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4001750 tx=0x7f5cc8005830 comp rx=0 tx=0).stop 2024-03-28T19:14:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.436+0000 7f5ce9600640 1 -- 172.21.15.27:0/3873824626 shutdown_connections 2024-03-28T19:14:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.436+0000 7f5ce9600640 1 --2- 172.21.15.27:0/3873824626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cc8003a00 0x7f5cc8003e20 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.436+0000 7f5ce9600640 1 -- 172.21.15.27:0/3873824626 >> 172.21.15.27:0/3873824626 conn(0x7f5ce406b6f0 msgr2=0x7f5ce40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.436+0000 7f5ce9600640 1 -- 172.21.15.27:0/3873824626 shutdown_connections 2024-03-28T19:14:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.437+0000 7f5ce9600640 1 -- 172.21.15.27:0/3873824626 wait complete. 2024-03-28T19:14:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.438+0000 7f5ce9600640 1 Processor -- start 2024-03-28T19:14:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.439+0000 7f5ce9600640 1 -- start start 2024-03-28T19:14:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce9600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc8003a00 0x7f5ce41ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce41ef1c0 0x7f5ce40ac190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce9600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ce40ac6d0 0x7f5ce4151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40b09a0 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce9600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40b0820 con 0x7f5ce40ac6d0 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce9600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40ad7d0 con 0x7f5cc8003a00 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce41ef1c0 0x7f5ce40ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce41ef1c0 0x7f5ce40ac190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55296/0 (socket says 172.21.15.27:55296) 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce3400640 1 -- 172.21.15.27:0/3507619910 learned_addr learned my addr 172.21.15.27:0/3507619910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.440+0000 7f5ce8c00640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ce40ac6d0 0x7f5ce4151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5ce3e00640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc8003a00 0x7f5ce41ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5ce3400640 1 -- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc8003a00 msgr2=0x7f5ce41ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5ce3400640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc8003a00 0x7f5ce41ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5ce3400640 1 -- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ce40ac6d0 msgr2=0x7f5ce4151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5ce3400640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ce40ac6d0 0x7f5ce4151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5ce3400640 1 -- 172.21.15.27:0/3507619910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40aae30 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5ce3e00640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc8003a00 0x7f5ce41ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5ce3400640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce41ef1c0 0x7f5ce40ac190 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000d770 tx=0x7f5cd000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.441+0000 7f5cdbe00640 1 -- 172.21.15.27:0/3507619910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0019070 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.442+0000 7f5cdbe00640 1 -- 172.21.15.27:0/3507619910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd00074e0 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.442+0000 7f5cdbe00640 1 -- 172.21.15.27:0/3507619910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd00049e0 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.442+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce40acbf0 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.442+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce40aaac0 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.443+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce4069640 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.446+0000 7f5cdbe00640 1 -- 172.21.15.27:0/3507619910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5cd001c070 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.447+0000 7f5cdbe00640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cc406bc20 0x7f5cc406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.447+0000 7f5cdbe00640 1 -- 172.21.15.27:0/3507619910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8787+0+0 (secure 0 0 0) 0x7f5cd008b000 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.447+0000 7f5ce3e00640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cc406bc20 0x7f5cc406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.448+0000 7f5ce3e00640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cc406bc20 0x7f5cc406e0e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4033000 tx=0x7f5cd4009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:36.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.453+0000 7f5cdbe00640 1 -- 172.21.15.27:0/3507619910 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5cd000de20 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.601+0000 7f6646000640 1 Processor -- start 2024-03-28T19:14:36.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.601+0000 7f6646000640 1 -- start start 2024-03-28T19:14:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.603+0000 7f6646000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f66400c69a0 0x7f66400c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.603+0000 7f6646000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6640153a30 0x7f6640153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.603+0000 7f6646000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6640150ec0 con 0x7f6640150ae0 2024-03-28T19:14:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.603+0000 7f6646000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66401543c0 con 0x7f6640153a30 2024-03-28T19:14:36.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.603+0000 7f6646000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664006dc70 con 0x7f66400c69a0 2024-03-28T19:14:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.603+0000 7f6645600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6640153a30 0x7f6640153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.603+0000 7f6644c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6640150ae0 0x7f66400a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56460/0 (socket says 172.21.15.27:56460) 2024-03-28T19:14:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.603+0000 7f6644c00640 1 -- 172.21.15.27:0/1578944566 learned_addr learned my addr 172.21.15.27:0/1578944566 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.604+0000 7f663fe00640 1 --2- 172.21.15.27:0/1578944566 >> v2:172.21.15.155:3300/0 conn(0x7f66400c69a0 0x7f66400c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.604+0000 7f6645600640 1 -- 172.21.15.27:0/1578944566 >> v2:172.21.15.155:3300/0 conn(0x7f66400c69a0 msgr2=0x7f66400c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.604+0000 7f6645600640 1 --2- 172.21.15.27:0/1578944566 >> v2:172.21.15.155:3300/0 conn(0x7f66400c69a0 0x7f66400c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.604+0000 7f6645600640 1 -- 172.21.15.27:0/1578944566 >> v1:172.21.15.27:6789/0 conn(0x7f6640150ae0 legacy=0x7f66400a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.604+0000 7f6645600640 1 -- 172.21.15.27:0/1578944566 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640153550 con 0x7f6640153a30 2024-03-28T19:14:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.604+0000 7f663fe00640 1 --2- 172.21.15.27:0/1578944566 >> v2:172.21.15.155:3300/0 conn(0x7f66400c69a0 0x7f66400c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:36.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.604+0000 7f663f400640 1 -- 172.21.15.27:0/1578944566 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927496594 0 0) 0x7f6634001110 con 0x7f6640150ae0 2024-03-28T19:14:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.605+0000 7f6645600640 1 --2- 172.21.15.27:0/1578944566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6640153a30 0x7f6640153e80 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f66400c44a0 tx=0x7f663802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606435dba4888f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.605+0000 7f663f400640 1 -- 172.21.15.27:0/1578944566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638039070 con 0x7f6640153a30 2024-03-28T19:14:36.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.605+0000 7f663f400640 1 -- 172.21.15.27:0/1578944566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663802cbf0 con 0x7f6640153a30 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.605+0000 7f663f400640 1 -- 172.21.15.27:0/1578944566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638034440 con 0x7f6640153a30 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.606+0000 7f6646000640 1 -- 172.21.15.27:0/1578944566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6640153a30 msgr2=0x7f6640153e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.606+0000 7f6646000640 1 --2- 172.21.15.27:0/1578944566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6640153a30 0x7f6640153e80 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f66400c44a0 tx=0x7f663802be40 comp rx=0 tx=0).stop 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.606+0000 7f6646000640 1 -- 172.21.15.27:0/1578944566 shutdown_connections 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.606+0000 7f6646000640 1 --2- 172.21.15.27:0/1578944566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6640153a30 0x7f6640153e80 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.607+0000 7f6646000640 1 --2- 172.21.15.27:0/1578944566 >> v2:172.21.15.155:3300/0 conn(0x7f66400c69a0 0x7f66400c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.607+0000 7f6646000640 1 -- 172.21.15.27:0/1578944566 >> 172.21.15.27:0/1578944566 conn(0x7f664006b0c0 msgr2=0x7f664006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.607+0000 7f6646000640 1 -- 172.21.15.27:0/1578944566 shutdown_connections 2024-03-28T19:14:36.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.607+0000 7f6646000640 1 -- 172.21.15.27:0/1578944566 wait complete. 2024-03-28T19:14:36.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.609+0000 7f6646000640 1 Processor -- start 2024-03-28T19:14:36.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.609+0000 7f6646000640 1 -- start start 2024-03-28T19:14:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6646000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66400c69a0 0x7f6640072060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6646000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6640150ae0 0x7f66400bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6646000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6640153a30 0x7f664014dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6646000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664006e4d0 con 0x7f66400c69a0 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6646000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664006e1f0 con 0x7f6640150ae0 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6646000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664006e7b0 con 0x7f6640153a30 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6644c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66400c69a0 0x7f6640072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6644c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66400c69a0 0x7f6640072060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55322/0 (socket says 172.21.15.27:55322) 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.611+0000 7f6644c00640 1 -- 172.21.15.27:0/1642656594 learned_addr learned my addr 172.21.15.27:0/1642656594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f663fe00640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6640150ae0 0x7f66400bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f6644c00640 1 -- 172.21.15.27:0/1642656594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6640153a30 msgr2=0x7f664014dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f6644c00640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6640153a30 0x7f664014dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f6644c00640 1 -- 172.21.15.27:0/1642656594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6640150ae0 msgr2=0x7f66400bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f6644c00640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6640150ae0 0x7f66400bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f6644c00640 1 -- 172.21.15.27:0/1642656594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66400725a0 con 0x7f66400c69a0 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f663fe00640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6640150ae0 0x7f66400bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f6644c00640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66400c69a0 0x7f6640072060 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f663400c780 tx=0x7f663400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f663cc00640 1 -- 172.21.15.27:0/1642656594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6634018070 con 0x7f66400c69a0 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f663cc00640 1 -- 172.21.15.27:0/1642656594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66340072d0 con 0x7f66400c69a0 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f663cc00640 1 -- 172.21.15.27:0/1642656594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66340047e0 con 0x7f66400c69a0 2024-03-28T19:14:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.612+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66401ead40 con 0x7f66400c69a0 2024-03-28T19:14:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.613+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66401eb300 con 0x7f66400c69a0 2024-03-28T19:14:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.614+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6640067dd0 con 0x7f66400c69a0 2024-03-28T19:14:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.616+0000 7f663cc00640 1 -- 172.21.15.27:0/1642656594 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6634004980 con 0x7f66400c69a0 2024-03-28T19:14:36.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.617+0000 7f663cc00640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f662006bbd0 0x7f662006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:36.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.617+0000 7f663cc00640 1 -- 172.21.15.27:0/1642656594 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8787+0+0 (secure 0 0 0) 0x7f663408aa30 con 0x7f66400c69a0 2024-03-28T19:14:36.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.618+0000 7f663fe00640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f662006bbd0 0x7f662006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.619+0000 7f663fe00640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f662006bbd0 0x7f662006e090 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f662c006fd0 tx=0x7f662c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:36.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.622+0000 7f663cc00640 1 -- 172.21.15.27:0/1642656594 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6640150ec0 con 0x7f66400c69a0 2024-03-28T19:14:36.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.745+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce4156580 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.745+0000 7f5cdbe00640 1 -- 172.21.15.27:0/3507619910 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f5cd0058830 con 0x7f5ce41ef1c0 2024-03-28T19:14:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:36.746 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:35.095404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:36.746 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 144 2024-03-28T19:14:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cc406bc20 msgr2=0x7f5cc406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cc406bc20 0x7f5cc406e0e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4033000 tx=0x7f5cd4009030 comp rx=0 tx=0).stop 2024-03-28T19:14:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce41ef1c0 msgr2=0x7f5ce40ac190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce41ef1c0 0x7f5ce40ac190 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5cd000d770 tx=0x7f5cd000dc30 comp rx=0 tx=0).stop 2024-03-28T19:14:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 shutdown_connections 2024-03-28T19:14:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5cc406bc20 0x7f5cc406e0e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ce40ac6d0 0x7f5ce4151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ce41ef1c0 0x7f5ce40ac190 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 --2- 172.21.15.27:0/3507619910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5cc8003a00 0x7f5ce41ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 >> 172.21.15.27:0/3507619910 conn(0x7f5ce406b6f0 msgr2=0x7f5ce40716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 shutdown_connections 2024-03-28T19:14:36.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.748+0000 7f5ce9600640 1 -- 172.21.15.27:0/3507619910 wait complete. 2024-03-28T19:14:36.766 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 144, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7146": { "addr": "172.21.15.27:6835/3913435536", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 3913435536, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 3913435536, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7146, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:14:35.095404+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7146 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 136, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5756": { "addr": "172.21.15.155:6835/2989591316", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 2989591316, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 2989591316, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5756, "incarnation": 130, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:12:23.824811+0000", "qdb_cluster": [ 5756 ], "qdb_leader": 5756, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5756 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/2504098667", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 2504098667, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 2504098667, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 136, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5870, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:14:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.932+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664006da50 con 0x7f66400c69a0 2024-03-28T19:14:36.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.933+0000 7f663cc00640 1 -- 172.21.15.27:0/1642656594 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f6634004c70 con 0x7f66400c69a0 2024-03-28T19:14:36.934 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:36.934 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:35.095404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:12:23.824811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5756},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5756":{"gid":5756,"name":"b","rank":0,"incarnation":130,"state":"up:active","state_seq":22,"addr":"172.21.15.155:6835/2989591316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2989591316},{"type":"v1","addr":"172.21.15.155:6835","nonce":2989591316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5756,"qdb_cluster":[5756]},"id":2}]} 2024-03-28T19:14:36.934 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 144 2024-03-28T19:14:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.936+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f662006bbd0 msgr2=0x7f662006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.936+0000 7f6646000640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f662006bbd0 0x7f662006e090 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f662c006fd0 tx=0x7f662c008040 comp rx=0 tx=0).stop 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.936+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66400c69a0 msgr2=0x7f6640072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.936+0000 7f6646000640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66400c69a0 0x7f6640072060 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f663400c780 tx=0x7f663400cc40 comp rx=0 tx=0).stop 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.937+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 shutdown_connections 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.937+0000 7f6646000640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f662006bbd0 0x7f662006e090 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.937+0000 7f6646000640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6640153a30 0x7f664014dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.937+0000 7f6646000640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6640150ae0 0x7f66400bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.937+0000 7f6646000640 1 --2- 172.21.15.27:0/1642656594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66400c69a0 0x7f6640072060 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.937+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 >> 172.21.15.27:0/1642656594 conn(0x7f664006b0c0 msgr2=0x7f664006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:36.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.937+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 shutdown_connections 2024-03-28T19:14:36.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:36.937+0000 7f6646000640 1 -- 172.21.15.27:0/1642656594 wait complete. 2024-03-28T19:14:36.961 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:14:38.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:38.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.000+0000 7f860e000640 1 Processor -- start 2024-03-28T19:14:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.000+0000 7f860e000640 1 -- start start 2024-03-28T19:14:39.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.001+0000 7f860e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080c58c0 0x7f86080c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.001+0000 7f860e000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f86080c8ed0 0x7f86081555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.001+0000 7f860e000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8608156230 con 0x7f86080be5d0 2024-03-28T19:14:39.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.002+0000 7f860e000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8608158bd0 con 0x7f86080c8ed0 2024-03-28T19:14:39.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.002+0000 7f860e000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8608158eb0 con 0x7f86080c58c0 2024-03-28T19:14:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.002+0000 7f860cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080c58c0 0x7f86080c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.002+0000 7f8607e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f86080c8ed0 0x7f86081555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.002+0000 7f860cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080c58c0 0x7f86080c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:56698/0 (socket says 172.21.15.27:56698) 2024-03-28T19:14:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.002+0000 7f8607e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f86080c8ed0 0x7f86081555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:60498/0 (socket says 172.21.15.27:60498) 2024-03-28T19:14:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.002+0000 7f860cc00640 1 -- 172.21.15.27:0/1242066350 learned_addr learned my addr 172.21.15.27:0/1242066350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.003+0000 7f8607400640 1 -- 172.21.15.27:0/1242066350 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186531018 0 0) 0x7f8608156230 con 0x7f86080be5d0 2024-03-28T19:14:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.003+0000 7f8607400640 1 -- 172.21.15.27:0/1242066350 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f0003520 con 0x7f86080be5d0 2024-03-28T19:14:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.003+0000 7f860cc00640 1 -- 172.21.15.27:0/1242066350 >> v2:172.21.15.155:3300/0 conn(0x7f86080c8ed0 msgr2=0x7f86081555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.003+0000 7f860cc00640 1 --2- 172.21.15.27:0/1242066350 >> v2:172.21.15.155:3300/0 conn(0x7f86080c8ed0 0x7f86081555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.003+0000 7f860cc00640 1 -- 172.21.15.27:0/1242066350 >> v1:172.21.15.155:6789/0 conn(0x7f86080be5d0 legacy=0x7f8608155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.003+0000 7f860cc00640 1 -- 172.21.15.27:0/1242066350 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8608159190 con 0x7f86080c58c0 2024-03-28T19:14:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.003+0000 7f8607e00640 1 --2- 172.21.15.27:0/1242066350 >> v2:172.21.15.155:3300/0 conn(0x7f86080c8ed0 0x7f86081555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:14:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.004+0000 7f860cc00640 1 --2- 172.21.15.27:0/1242066350 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080c58c0 0x7f86080c5ca0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009870 tx=0x7f85fc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86dcfb14105e25a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.005+0000 7f8607400640 1 -- 172.21.15.27:0/1242066350 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc035020 con 0x7f86080c58c0 2024-03-28T19:14:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.005+0000 7f8607400640 1 -- 172.21.15.27:0/1242066350 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc031640 con 0x7f86080c58c0 2024-03-28T19:14:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.005+0000 7f8607400640 1 -- 172.21.15.27:0/1242066350 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc02d040 con 0x7f86080c58c0 2024-03-28T19:14:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.005+0000 7f860e000640 1 -- 172.21.15.27:0/1242066350 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080c58c0 msgr2=0x7f86080c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.005+0000 7f860e000640 1 --2- 172.21.15.27:0/1242066350 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080c58c0 0x7f86080c5ca0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009870 tx=0x7f85fc02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.006+0000 7f860e000640 1 -- 172.21.15.27:0/1242066350 shutdown_connections 2024-03-28T19:14:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.006+0000 7f860e000640 1 --2- 172.21.15.27:0/1242066350 >> v2:172.21.15.155:3300/0 conn(0x7f86080c8ed0 0x7f86081555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.006+0000 7f860e000640 1 --2- 172.21.15.27:0/1242066350 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080c58c0 0x7f86080c5ca0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.006+0000 7f860e000640 1 -- 172.21.15.27:0/1242066350 >> 172.21.15.27:0/1242066350 conn(0x7f860806b6f0 msgr2=0x7f86080acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.006+0000 7f860e000640 1 -- 172.21.15.27:0/1242066350 shutdown_connections 2024-03-28T19:14:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.006+0000 7f860e000640 1 -- 172.21.15.27:0/1242066350 wait complete. 2024-03-28T19:14:39.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.008+0000 7f860e000640 1 Processor -- start 2024-03-28T19:14:39.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.008+0000 7f860e000640 1 -- start start 2024-03-28T19:14:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f860e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080be5d0 0x7f8608158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f860e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86080c58c0 0x7f86081519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f860e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86080c8ed0 0x7f8608151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f860e000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86080b0820 con 0x7f86080c58c0 2024-03-28T19:14:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f860e000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8608159750 con 0x7f86080c8ed0 2024-03-28T19:14:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f860e000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86080b09a0 con 0x7f86080be5d0 2024-03-28T19:14:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f8607e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86080c58c0 0x7f86081519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f8607e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86080c58c0 0x7f86081519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55334/0 (socket says 172.21.15.27:55334) 2024-03-28T19:14:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f8607e00640 1 -- 172.21.15.27:0/2149382888 learned_addr learned my addr 172.21.15.27:0/2149382888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f860cc00640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080be5d0 0x7f8608158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.009+0000 7f860d600640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86080c8ed0 0x7f8608151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8607e00640 1 -- 172.21.15.27:0/2149382888 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080be5d0 msgr2=0x7f8608158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8607e00640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080be5d0 0x7f8608158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8607e00640 1 -- 172.21.15.27:0/2149382888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86080c8ed0 msgr2=0x7f8608151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8607e00640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86080c8ed0 0x7f8608151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8607e00640 1 -- 172.21.15.27:0/2149382888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86080c4850 con 0x7f86080c58c0 2024-03-28T19:14:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8607e00640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86080c58c0 0x7f86081519a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f85f400e960 tx=0x7f85f400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8604c00640 1 -- 172.21.15.27:0/2149382888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4017070 con 0x7f86080c58c0 2024-03-28T19:14:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8604c00640 1 -- 172.21.15.27:0/2149382888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f400ca00 con 0x7f86080c58c0 2024-03-28T19:14:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86080c4b30 con 0x7f86080c58c0 2024-03-28T19:14:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f8604c00640 1 -- 172.21.15.27:0/2149382888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f400cdd0 con 0x7f86080c58c0 2024-03-28T19:14:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.010+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86080c2390 con 0x7f86080c58c0 2024-03-28T19:14:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.012+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8608069640 con 0x7f86080c58c0 2024-03-28T19:14:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.013+0000 7f8604c00640 1 -- 172.21.15.27:0/2149382888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f85f401a070 con 0x7f86080c58c0 2024-03-28T19:14:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.014+0000 7f8604c00640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85e806bc00 0x7f85e806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.014+0000 7f8604c00640 1 -- 172.21.15.27:0/2149382888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f85f408b860 con 0x7f86080c58c0 2024-03-28T19:14:39.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.014+0000 7f860cc00640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85e806bc00 0x7f85e806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.015+0000 7f860cc00640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85e806bc00 0x7f85e806e0c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f85fc036000 tx=0x7f85fc03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.018+0000 7f8604c00640 1 -- 172.21.15.27:0/2149382888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85f4059080 con 0x7f86080c58c0 2024-03-28T19:14:39.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.161+0000 7fc951000640 1 Processor -- start 2024-03-28T19:14:39.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.161+0000 7fc951000640 1 -- start start 2024-03-28T19:14:39.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.162+0000 7fc951000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc94c0c5e80 0x7fc94c0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.162+0000 7fc951000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc94c0c1f50 0x7fc94c0ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.162+0000 7fc951000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c0ac1c0 con 0x7fc94c0abd90 2024-03-28T19:14:39.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.162+0000 7fc951000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94c06d650 con 0x7fc94c0c1f50 2024-03-28T19:14:39.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.162+0000 7fc951000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94c06e1d0 con 0x7fc94c0c5e80 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc949400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc94c0c1f50 0x7fc94c0ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc949400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc94c0c1f50 0x7fc94c0ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55338/0 (socket says 172.21.15.27:55338) 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc949400640 1 -- 172.21.15.27:0/3190049603 learned_addr learned my addr 172.21.15.27:0/3190049603 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc94a800640 1 --1- 172.21.15.27:0/3190049603 >> v1:172.21.15.27:6789/0 conn(0x7fc94c0abd90 0x7fc94c0aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56472/0 (socket says 172.21.15.27:56472) 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc949e00640 1 --2- 172.21.15.27:0/3190049603 >> v2:172.21.15.155:3300/0 conn(0x7fc94c0c5e80 0x7fc94c0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc949400640 1 -- 172.21.15.27:0/3190049603 >> v2:172.21.15.155:3300/0 conn(0x7fc94c0c5e80 msgr2=0x7fc94c0c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc949400640 1 --2- 172.21.15.27:0/3190049603 >> v2:172.21.15.155:3300/0 conn(0x7fc94c0c5e80 0x7fc94c0c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc949400640 1 -- 172.21.15.27:0/3190049603 >> v1:172.21.15.27:6789/0 conn(0x7fc94c0abd90 legacy=0x7fc94c0aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.163+0000 7fc949400640 1 -- 172.21.15.27:0/3190049603 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c06d8a0 con 0x7fc94c0c1f50 2024-03-28T19:14:39.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.164+0000 7fc949400640 1 --2- 172.21.15.27:0/3190049603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0c1f50 0x7fc94c0ab850 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0c6670 tx=0x7fc93802e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f9e935ca977275 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.164+0000 7fc948a00640 1 -- 172.21.15.27:0/3190049603 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93803b070 con 0x7fc94c0c1f50 2024-03-28T19:14:39.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc948a00640 1 -- 172.21.15.27:0/3190049603 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93802ee10 con 0x7fc94c0c1f50 2024-03-28T19:14:39.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc948a00640 1 -- 172.21.15.27:0/3190049603 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938036660 con 0x7fc94c0c1f50 2024-03-28T19:14:39.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc951000640 1 -- 172.21.15.27:0/3190049603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0c1f50 msgr2=0x7fc94c0ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc951000640 1 --2- 172.21.15.27:0/3190049603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0c1f50 0x7fc94c0ab850 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0c6670 tx=0x7fc93802e060 comp rx=0 tx=0).stop 2024-03-28T19:14:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc951000640 1 -- 172.21.15.27:0/3190049603 shutdown_connections 2024-03-28T19:14:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc951000640 1 --2- 172.21.15.27:0/3190049603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0c1f50 0x7fc94c0ab850 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc951000640 1 --2- 172.21.15.27:0/3190049603 >> v2:172.21.15.155:3300/0 conn(0x7fc94c0c5e80 0x7fc94c0c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc951000640 1 -- 172.21.15.27:0/3190049603 >> 172.21.15.27:0/3190049603 conn(0x7fc94c06b9d0 msgr2=0x7fc94c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc951000640 1 -- 172.21.15.27:0/3190049603 shutdown_connections 2024-03-28T19:14:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.165+0000 7fc951000640 1 -- 172.21.15.27:0/3190049603 wait complete. 2024-03-28T19:14:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.167+0000 7fc951000640 1 Processor -- start 2024-03-28T19:14:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.167+0000 7fc951000640 1 -- start start 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc951000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0abd90 0x7fc94c1dda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc951000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc94c0c1f50 0x7fc94c0c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc951000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc94c0c5e80 0x7fc94c09a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc951000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c0c31f0 con 0x7fc94c0abd90 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc951000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c06de60 con 0x7fc94c0c1f50 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc951000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c0c3370 con 0x7fc94c0c5e80 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc949e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0abd90 0x7fc94c1dda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc949e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0abd90 0x7fc94c1dda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55348/0 (socket says 172.21.15.27:55348) 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc949e00640 1 -- 172.21.15.27:0/3777151023 learned_addr learned my addr 172.21.15.27:0/3777151023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.168+0000 7fc949400640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc94c0c1f50 0x7fc94c0c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc94a800640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc94c0c5e80 0x7fc94c09a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc949e00640 1 -- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc94c0c5e80 msgr2=0x7fc94c09a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc949e00640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc94c0c5e80 0x7fc94c09a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc949e00640 1 -- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc94c0c1f50 msgr2=0x7fc94c0c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc949e00640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc94c0c1f50 0x7fc94c0c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc949e00640 1 -- 172.21.15.27:0/3777151023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c09d550 con 0x7fc94c0abd90 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc949400640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc94c0c1f50 0x7fc94c0c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc94a800640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc94c0c5e80 0x7fc94c09a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc949e00640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0abd90 0x7fc94c1dda50 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00d830 tx=0x7fc93c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc942000640 1 -- 172.21.15.27:0/3777151023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00b6c0 con 0x7fc94c0abd90 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc942000640 1 -- 172.21.15.27:0/3777151023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c019070 con 0x7fc94c0abd90 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc942000640 1 -- 172.21.15.27:0/3777151023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00ba80 con 0x7fc94c0abd90 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.169+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c1dc9b0 con 0x7fc94c0abd90 2024-03-28T19:14:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.170+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94c1dcf70 con 0x7fc94c0abd90 2024-03-28T19:14:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.171+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94c0c7890 con 0x7fc94c0abd90 2024-03-28T19:14:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.172+0000 7fc942000640 1 -- 172.21.15.27:0/3777151023 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc93c01e460 con 0x7fc94c0abd90 2024-03-28T19:14:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.173+0000 7fc942000640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc92c06bbd0 0x7fc92c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.173+0000 7fc942000640 1 -- 172.21.15.27:0/3777151023 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7fc93c08bdd0 con 0x7fc94c0abd90 2024-03-28T19:14:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.173+0000 7fc949400640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc92c06bbd0 0x7fc92c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.174+0000 7fc949400640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc92c06bbd0 0x7fc92c06e090 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc938038000 tx=0x7fc938008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.177+0000 7fc942000640 1 -- 172.21.15.27:0/3777151023 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc94c0c7890 con 0x7fc94c0abd90 2024-03-28T19:14:39.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.264+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8608068300 con 0x7f86080c58c0 2024-03-28T19:14:39.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.265+0000 7f8604c00640 1 -- 172.21.15.27:0/2149382888 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4667 (secure 0 0 0) 0x7f85f405d120 con 0x7f86080c58c0 2024-03-28T19:14:39.265 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:39.265 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:38.630909+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:38.754409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:14:39.265 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 146 2024-03-28T19:14:39.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.267+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85e806bc00 msgr2=0x7f85e806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.267+0000 7f860e000640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85e806bc00 0x7f85e806e0c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f85fc036000 tx=0x7f85fc03c040 comp rx=0 tx=0).stop 2024-03-28T19:14:39.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86080c58c0 msgr2=0x7f86081519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86080c58c0 0x7f86081519a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f85f400e960 tx=0x7f85f400ee20 comp rx=0 tx=0).stop 2024-03-28T19:14:39.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 shutdown_connections 2024-03-28T19:14:39.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85e806bc00 0x7f85e806e0c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86080c8ed0 0x7f8608151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86080c58c0 0x7f86081519a0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 --2- 172.21.15.27:0/2149382888 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86080be5d0 0x7f8608158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 >> 172.21.15.27:0/2149382888 conn(0x7f860806b6f0 msgr2=0x7f86080ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:39.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 shutdown_connections 2024-03-28T19:14:39.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.268+0000 7f860e000640 1 -- 172.21.15.27:0/2149382888 wait complete. 2024-03-28T19:14:39.286 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:14:39.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.410+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc94c154930 con 0x7fc94c0abd90 2024-03-28T19:14:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.411+0000 7fc942000640 1 -- 172.21.15.27:0/3777151023 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+4667 (secure 0 0 0) 0x7fc93c059c40 con 0x7fc94c0abd90 2024-03-28T19:14:39.412 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:39.412 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:38.630909+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:38.754409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:14:39.412 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 146 2024-03-28T19:14:39.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc92c06bbd0 msgr2=0x7fc92c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc92c06bbd0 0x7fc92c06e090 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc938038000 tx=0x7fc938008040 comp rx=0 tx=0).stop 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0abd90 msgr2=0x7fc94c1dda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0abd90 0x7fc94c1dda50 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00d830 tx=0x7fc93c00dcf0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 shutdown_connections 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc92c06bbd0 0x7fc92c06e090 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc94c0c5e80 0x7fc94c09a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc94c0c1f50 0x7fc94c0c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 --2- 172.21.15.27:0/3777151023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94c0abd90 0x7fc94c1dda50 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 >> 172.21.15.27:0/3777151023 conn(0x7fc94c06b9d0 msgr2=0x7fc94c0c2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 shutdown_connections 2024-03-28T19:14:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.415+0000 7fc951000640 1 -- 172.21.15.27:0/3777151023 wait complete. 2024-03-28T19:14:39.430 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-03-28T19:14:39.430 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:14:39.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:39.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.649+0000 7fc426c00640 1 Processor -- start 2024-03-28T19:14:39.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.649+0000 7fc426c00640 1 -- start start 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.650+0000 7fc426c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc4200c8e90 0x7fc420156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.650+0000 7fc426c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4200c5e30 0x7fc420156550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.650+0000 7fc426c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4200c6f30 con 0x7fc4200c6380 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.650+0000 7fc426c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4200c7820 con 0x7fc4200c5e30 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.650+0000 7fc426c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc420157910 con 0x7fc4200c8e90 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.650+0000 7fc426200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4200c5e30 0x7fc420156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.652+0000 7fc426200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4200c5e30 0x7fc420156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55358/0 (socket says 172.21.15.27:55358) 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.652+0000 7fc426200640 1 -- 172.21.15.27:0/3412569105 learned_addr learned my addr 172.21.15.27:0/3412569105 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.652+0000 7fc424e00640 1 --1- 172.21.15.27:0/3412569105 >> v1:172.21.15.155:6789/0 conn(0x7fc4200c6380 0x7fc42006fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:56606/0 (socket says 172.21.15.27:56606) 2024-03-28T19:14:39.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.652+0000 7fc425800640 1 --2- 172.21.15.27:0/3412569105 >> v2:172.21.15.155:3300/0 conn(0x7fc4200c8e90 0x7fc420156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.652+0000 7fc426200640 1 -- 172.21.15.27:0/3412569105 >> v2:172.21.15.155:3300/0 conn(0x7fc4200c8e90 msgr2=0x7fc420156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.652+0000 7fc426200640 1 --2- 172.21.15.27:0/3412569105 >> v2:172.21.15.155:3300/0 conn(0x7fc4200c8e90 0x7fc420156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.652+0000 7fc426200640 1 -- 172.21.15.27:0/3412569105 >> v1:172.21.15.155:6789/0 conn(0x7fc4200c6380 legacy=0x7fc42006fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.652+0000 7fc426200640 1 -- 172.21.15.27:0/3412569105 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4200c71d0 con 0x7fc4200c5e30 2024-03-28T19:14:39.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.653+0000 7fc426200640 1 --2- 172.21.15.27:0/3412569105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c5e30 0x7fc420156550 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc42006e370 tx=0x7fc41802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c970a7fee1461c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.654+0000 7fc41fe00640 1 -- 172.21.15.27:0/3412569105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418035020 con 0x7fc4200c5e30 2024-03-28T19:14:39.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.654+0000 7fc41fe00640 1 -- 172.21.15.27:0/3412569105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418031870 con 0x7fc4200c5e30 2024-03-28T19:14:39.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.654+0000 7fc41fe00640 1 -- 172.21.15.27:0/3412569105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418031c40 con 0x7fc4200c5e30 2024-03-28T19:14:39.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.654+0000 7fc426c00640 1 -- 172.21.15.27:0/3412569105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c5e30 msgr2=0x7fc420156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.654+0000 7fc426c00640 1 --2- 172.21.15.27:0/3412569105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c5e30 0x7fc420156550 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc42006e370 tx=0x7fc41802b910 comp rx=0 tx=0).stop 2024-03-28T19:14:39.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.655+0000 7fc426c00640 1 -- 172.21.15.27:0/3412569105 shutdown_connections 2024-03-28T19:14:39.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.655+0000 7fc426c00640 1 --2- 172.21.15.27:0/3412569105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c5e30 0x7fc420156550 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.655+0000 7fc426c00640 1 --2- 172.21.15.27:0/3412569105 >> v2:172.21.15.155:3300/0 conn(0x7fc4200c8e90 0x7fc420156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.655+0000 7fc426c00640 1 -- 172.21.15.27:0/3412569105 >> 172.21.15.27:0/3412569105 conn(0x7fc42006ea70 msgr2=0x7fc42006ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:39.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.655+0000 7fc426c00640 1 -- 172.21.15.27:0/3412569105 shutdown_connections 2024-03-28T19:14:39.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.655+0000 7fc426c00640 1 -- 172.21.15.27:0/3412569105 wait complete. 2024-03-28T19:14:39.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.656+0000 7fc426c00640 1 Processor -- start 2024-03-28T19:14:39.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.656+0000 7fc426c00640 1 -- start start 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.657+0000 7fc426c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4200c5e30 0x7fc420151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.657+0000 7fc426c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4200c6380 0x7fc42014ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.657+0000 7fc426c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c8e90 0x7fc420150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.657+0000 7fc426c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4201576d0 con 0x7fc4200c8e90 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.657+0000 7fc426c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc420157550 con 0x7fc4200c5e30 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.657+0000 7fc426c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4200c9270 con 0x7fc4200c6380 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c8e90 0x7fc420150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c8e90 0x7fc420150480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55374/0 (socket says 172.21.15.27:55374) 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 -- 172.21.15.27:0/110592410 learned_addr learned my addr 172.21.15.27:0/110592410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc424e00640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4200c6380 0x7fc42014ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc425800640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4200c5e30 0x7fc420151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 -- 172.21.15.27:0/110592410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4200c6380 msgr2=0x7fc42014ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4200c6380 0x7fc42014ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 -- 172.21.15.27:0/110592410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4200c5e30 msgr2=0x7fc420151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4200c5e30 0x7fc420151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 -- 172.21.15.27:0/110592410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4200ac490 con 0x7fc4200c8e90 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.658+0000 7fc426200640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c8e90 0x7fc420150480 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc418009960 tx=0x7fc4180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.660+0000 7fc41d600640 1 -- 172.21.15.27:0/110592410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418040070 con 0x7fc4200c8e90 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.660+0000 7fc41d600640 1 -- 172.21.15.27:0/110592410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418030c50 con 0x7fc4200c8e90 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.660+0000 7fc41d600640 1 -- 172.21.15.27:0/110592410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418035040 con 0x7fc4200c8e90 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.660+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4201eee50 con 0x7fc4200c8e90 2024-03-28T19:14:39.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.660+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4200ac730 con 0x7fc4200c8e90 2024-03-28T19:14:39.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.661+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc420069640 con 0x7fc4200c8e90 2024-03-28T19:14:39.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.662+0000 7fc41d600640 1 -- 172.21.15.27:0/110592410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc418038020 con 0x7fc4200c8e90 2024-03-28T19:14:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.663+0000 7fc41d600640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc40006bc00 0x7fc40006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.663+0000 7fc41d600640 1 -- 172.21.15.27:0/110592410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7fc4180ad470 con 0x7fc4200c8e90 2024-03-28T19:14:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.663+0000 7fc425800640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc40006bc00 0x7fc40006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:39.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.663+0000 7fc425800640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc40006bc00 0x7fc40006e0c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4100099c0 tx=0x7fc410003990 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:39.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.667+0000 7fc41d600640 1 -- 172.21.15.27:0/110592410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc41807b340 con 0x7fc4200c8e90 2024-03-28T19:14:39.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.920+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4201ef370 con 0x7fc4200c8e90 2024-03-28T19:14:39.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.921+0000 7fc41d600640 1 -- 172.21.15.27:0/110592410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+4665 (secure 0 0 0) 0x7fc4180ad250 con 0x7fc4200c8e90 2024-03-28T19:14:39.920 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:39.920 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:38.783431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:38.754409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:14:39.921 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 147 2024-03-28T19:14:39.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc40006bc00 msgr2=0x7fc40006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc40006bc00 0x7fc40006e0c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4100099c0 tx=0x7fc410003990 comp rx=0 tx=0).stop 2024-03-28T19:14:39.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c8e90 msgr2=0x7fc420150480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:39.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c8e90 0x7fc420150480 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc418009960 tx=0x7fc4180046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 shutdown_connections 2024-03-28T19:14:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc40006bc00 0x7fc40006e0c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4200c8e90 0x7fc420150480 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4200c6380 0x7fc42014ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 --2- 172.21.15.27:0/110592410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4200c5e30 0x7fc420151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.923+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 >> 172.21.15.27:0/110592410 conn(0x7fc42006ea70 msgr2=0x7fc4200c20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.924+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 shutdown_connections 2024-03-28T19:14:39.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:39.924+0000 7fc426c00640 1 -- 172.21.15.27:0/110592410 wait complete. 2024-03-28T19:14:39.946 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 147, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 147, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7146": { "addr": "172.21.15.27:6835/3913435536", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 3913435536, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 3913435536, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7146, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:14:38.783431+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7146 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 146, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5870": { "addr": "172.21.15.155:6834/2504098667", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 2504098667, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 2504098667, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5870, "incarnation": 146, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:14:38.754409+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5870 } } } ], "standbys": [] } 2024-03-28T19:14:41.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.526+0000 7fc1fa200640 1 Processor -- start 2024-03-28T19:14:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.526+0000 7fc1fa200640 1 -- start start 2024-03-28T19:14:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1fa200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 0x7fc1f40ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1fa200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc1f40abd90 0x7fc1f40aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1fa200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f406d8a0 con 0x7fc1f40c5e80 2024-03-28T19:14:41.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1fa200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f40ac220 con 0x7fc1f40abd90 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1fa200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f406d650 con 0x7fc1f40c1f50 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1f3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc1f40c5e80 0x7fc1f40c2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56478/0 (socket says 172.21.15.27:56478) 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1f3400640 1 -- 172.21.15.27:0/3641771917 learned_addr learned my addr 172.21.15.27:0/3641771917 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1f3e00640 1 --2- 172.21.15.27:0/3641771917 >> v2:172.21.15.155:3300/0 conn(0x7fc1f40abd90 0x7fc1f40aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1f2a00640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 0x7fc1f40ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199264611 0 0) 0x7fc1f406d8a0 con 0x7fc1f40c5e80 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.528+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d8003520 con 0x7fc1f40c5e80 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 794734044 0 0) 0x7fc1d8003520 con 0x7fc1f40c5e80 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 msgr2=0x7fc1f40ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 0x7fc1f40ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 >> v2:172.21.15.155:3300/0 conn(0x7fc1f40abd90 msgr2=0x7fc1f40aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 --2- 172.21.15.27:0/3641771917 >> v2:172.21.15.155:3300/0 conn(0x7fc1f40abd90 0x7fc1f40aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f406db80 con 0x7fc1f40c5e80 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fc1e4001670 con 0x7fc1f40c5e80 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 >> v1:172.21.15.27:6789/0 conn(0x7fc1f40c5e80 legacy=0x7fc1f40c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 0x7fc1d80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1d8003f80 0x7fc1d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8004d40 0x7fc1d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f406d650 con 0x7fc1d8003f80 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d800ba70 con 0x7fc1d8004d40 2024-03-28T19:14:41.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d800bc70 con 0x7fc1f40c1f50 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.529+0000 7fc1f2a00640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1d8003f80 0x7fc1d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f3e00640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8004d40 0x7fc1d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f3400640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 0x7fc1d80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f2a00640 1 -- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 msgr2=0x7fc1d80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f2a00640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 0x7fc1d80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f2a00640 1 -- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8004d40 msgr2=0x7fc1d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f2a00640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8004d40 0x7fc1d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f2a00640 1 -- 172.21.15.27:0/3641771917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d8003520 con 0x7fc1d8003f80 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f3400640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f40c1f50 0x7fc1d80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.530+0000 7fc1f3e00640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8004d40 0x7fc1d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.531+0000 7fc1f2a00640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1d8003f80 0x7fc1d8004800 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc009a90 tx=0x7fc1d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4349be4a2fb4384 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.531+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc039070 con 0x7fc1d8003f80 2024-03-28T19:14:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.531+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc02c680 con 0x7fc1d8003f80 2024-03-28T19:14:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.531+0000 7fc1f2000640 1 -- 172.21.15.27:0/3641771917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc009690 con 0x7fc1d8003f80 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.532+0000 7fc1fa200640 1 -- 172.21.15.27:0/3641771917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1d8003f80 msgr2=0x7fc1d8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.532+0000 7fc1fa200640 1 --2- 172.21.15.27:0/3641771917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1d8003f80 0x7fc1d8004800 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc009a90 tx=0x7fc1d8005830 comp rx=0 tx=0).stop 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.532+0000 7fc1f3e00640 1 -- 172.21.15.27:0/3641771917 reap_dead start 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.532+0000 7fc1fa200640 1 -- 172.21.15.27:0/3641771917 shutdown_connections 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.532+0000 7fc1fa200640 1 -- 172.21.15.27:0/3641771917 >> 172.21.15.27:0/3641771917 conn(0x7fc1f406b9d0 msgr2=0x7fc1f40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.532+0000 7fc1fa200640 1 -- 172.21.15.27:0/3641771917 shutdown_connections 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.532+0000 7fc1fa200640 1 -- 172.21.15.27:0/3641771917 wait complete. 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.534+0000 7fc1fa200640 1 Processor -- start 2024-03-28T19:14:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.534+0000 7fc1fa200640 1 -- start start 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1fa200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8003f80 0x7fc1f41501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1fa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1f40a9100 0x7fc1f41457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1fa200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f4145ce0 0x7fc1f41ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1fa200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40c6260 con 0x7fc1f40a9100 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1fa200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f406e1d0 con 0x7fc1d8003f80 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1fa200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40c63e0 con 0x7fc1f4145ce0 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1f40a9100 0x7fc1f41457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1f40a9100 0x7fc1f41457a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55394/0 (socket says 172.21.15.27:55394) 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f2a00640 1 -- 172.21.15.27:0/3089784103 learned_addr learned my addr 172.21.15.27:0/3089784103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:41.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f3400640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8003f80 0x7fc1f41501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f3e00640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f4145ce0 0x7fc1f41ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f2a00640 1 -- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f4145ce0 msgr2=0x7fc1f41ee0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f2a00640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f4145ce0 0x7fc1f41ee0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f2a00640 1 -- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8003f80 msgr2=0x7fc1f41501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f2a00640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8003f80 0x7fc1f41501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f2a00640 1 -- 172.21.15.27:0/3089784103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f40a8a70 con 0x7fc1f40a9100 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.535+0000 7fc1f3e00640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f4145ce0 0x7fc1f41ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.536+0000 7fc1f2a00640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1f40a9100 0x7fc1f41457a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc037040 tx=0x7fc1dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.536+0000 7fc1eb400640 1 -- 172.21.15.27:0/3089784103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc039050 con 0x7fc1f40a9100 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.536+0000 7fc1eb400640 1 -- 172.21.15.27:0/3089784103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc045070 con 0x7fc1f40a9100 2024-03-28T19:14:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.536+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f40a7a40 con 0x7fc1f40a9100 2024-03-28T19:14:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.536+0000 7fc1eb400640 1 -- 172.21.15.27:0/3089784103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc0044e0 con 0x7fc1f40a9100 2024-03-28T19:14:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.536+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f40a6c60 con 0x7fc1f40a9100 2024-03-28T19:14:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.538+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f404f400 con 0x7fc1f40a9100 2024-03-28T19:14:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.539+0000 7fc1eb400640 1 -- 172.21.15.27:0/3089784103 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc1dc003850 con 0x7fc1f40a9100 2024-03-28T19:14:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.539+0000 7fc1eb400640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc1d006bbd0 0x7fc1d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.540+0000 7fc1eb400640 1 -- 172.21.15.27:0/3089784103 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7fc1dc0acbd0 con 0x7fc1f40a9100 2024-03-28T19:14:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.540+0000 7fc1f3400640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc1d006bbd0 0x7fc1d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.540+0000 7fc1f3400640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc1d006bbd0 0x7fc1d006e090 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40018a0 tx=0x7fc1e4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.544+0000 7fc1eb400640 1 -- 172.21.15.27:0/3089784103 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc1dc07aa40 con 0x7fc1f40a9100 2024-03-28T19:14:41.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.808+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1f4070b80 con 0x7fc1f40a9100 2024-03-28T19:14:41.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.809+0000 7fc1eb400640 1 -- 172.21.15.27:0/3089784103 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4676 (secure 0 0 0) 0x7fc1dc07a340 con 0x7fc1f40a9100 2024-03-28T19:14:41.808 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:41.808 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:41.692570+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:14:41.809 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 149 2024-03-28T19:14:41.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.812+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc1d006bbd0 msgr2=0x7fc1d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:41.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.812+0000 7fc1fa200640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc1d006bbd0 0x7fc1d006e090 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40018a0 tx=0x7fc1e4009040 comp rx=0 tx=0).stop 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.812+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1f40a9100 msgr2=0x7fc1f41457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.812+0000 7fc1fa200640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1f40a9100 0x7fc1f41457a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc037040 tx=0x7fc1dc004730 comp rx=0 tx=0).stop 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.813+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 shutdown_connections 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.813+0000 7fc1fa200640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc1d006bbd0 0x7fc1d006e090 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.813+0000 7fc1fa200640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1f4145ce0 0x7fc1f41ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.813+0000 7fc1fa200640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1f40a9100 0x7fc1f41457a0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.813+0000 7fc1fa200640 1 --2- 172.21.15.27:0/3089784103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1d8003f80 0x7fc1f41501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.813+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 >> 172.21.15.27:0/3089784103 conn(0x7fc1f406b9d0 msgr2=0x7fc1f40ab620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.813+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 shutdown_connections 2024-03-28T19:14:41.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:41.813+0000 7fc1fa200640 1 -- 172.21.15.27:0/3089784103 wait complete. 2024-03-28T19:14:41.837 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6532 2024-03-28T19:14:41.838 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.a 2024-03-28T19:14:41.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.172+0000 7fd7aac00640 1 Processor -- start 2024-03-28T19:14:42.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.172+0000 7fd7aac00640 1 -- start start 2024-03-28T19:14:42.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.174+0000 7fd7aac00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd7a406e6f0 0x7fd7a406eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:42.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.174+0000 7fd7aac00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40c3830 con 0x7fd7a4153ed0 2024-03-28T19:14:42.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.174+0000 7fd7aac00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40c66b0 con 0x7fd7a4159bc0 2024-03-28T19:14:42.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.174+0000 7fd7aac00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a40c6990 con 0x7fd7a406e6f0 2024-03-28T19:14:42.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.174+0000 7fd7a3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd7a4153ed0 0x7fd7a406ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56490/0 (socket says 172.21.15.27:56490) 2024-03-28T19:14:42.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.175+0000 7fd7a3e00640 1 -- 172.21.15.27:0/2526898641 learned_addr learned my addr 172.21.15.27:0/2526898641 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:42.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.175+0000 7fd79be00640 1 --2- 172.21.15.27:0/2526898641 >> v2:172.21.15.155:3300/0 conn(0x7fd7a406e6f0 0x7fd7a406eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:42.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.175+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920093431 0 0) 0x7fd7a40c3830 con 0x7fd7a4153ed0 2024-03-28T19:14:42.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.175+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788003520 con 0x7fd7a4153ed0 2024-03-28T19:14:42.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728707466 0 0) 0x7fd788003520 con 0x7fd7a4153ed0 2024-03-28T19:14:42.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 >> v2:172.21.15.155:3300/0 conn(0x7fd7a406e6f0 msgr2=0x7fd7a406eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:42.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 --2- 172.21.15.27:0/2526898641 >> v2:172.21.15.155:3300/0 conn(0x7fd7a406e6f0 0x7fd7a406eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 >> v1:172.21.15.155:6789/0 conn(0x7fd7a4159bc0 legacy=0x7fd7a4159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:42.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a40c3ad0 con 0x7fd7a4153ed0 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd794001670 con 0x7fd7a4153ed0 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 >> v1:172.21.15.27:6789/0 conn(0x7fd7a4153ed0 legacy=0x7fd7a406ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd7880039b0 0x7fd788003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd788004800 0x7fd788004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7880051c0 0x7fd78800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40c6990 con 0x7fd788004800 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78800c060 con 0x7fd7880051c0 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78800c260 con 0x7fd7880039b0 2024-03-28T19:14:42.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd79be00640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd788004800 0x7fd788004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:42.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.176+0000 7fd79be00640 1 -- 172.21.15.27:0/2526898641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd7880039b0 msgr2=0x7fd788003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:42.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.177+0000 7fd79be00640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd7880039b0 0x7fd788003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.177+0000 7fd79be00640 1 -- 172.21.15.27:0/2526898641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7880051c0 msgr2=0x7fd78800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:42.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.177+0000 7fd79be00640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7880051c0 0x7fd78800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.177+0000 7fd79be00640 1 -- 172.21.15.27:0/2526898641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788003520 con 0x7fd788004800 2024-03-28T19:14:42.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.177+0000 7fd7a8c00640 1 -- 172.21.15.27:0/2526898641 reap_dead start 2024-03-28T19:14:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.178+0000 7fd79be00640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd788004800 0x7fd788004c80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009ac0 tx=0x7fd788005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda74342e9c890b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.179+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c039070 con 0x7fd788004800 2024-03-28T19:14:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.179+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c02c6d0 con 0x7fd788004800 2024-03-28T19:14:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.179+0000 7fd7a3400640 1 -- 172.21.15.27:0/2526898641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c009690 con 0x7fd788004800 2024-03-28T19:14:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.180+0000 7fd7aac00640 1 -- 172.21.15.27:0/2526898641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd788004800 msgr2=0x7fd788004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.180+0000 7fd7aac00640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd788004800 0x7fd788004c80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009ac0 tx=0x7fd788005830 comp rx=0 tx=0).stop 2024-03-28T19:14:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.180+0000 7fd7aac00640 1 -- 172.21.15.27:0/2526898641 shutdown_connections 2024-03-28T19:14:42.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.180+0000 7fd7aac00640 1 --2- 172.21.15.27:0/2526898641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd788004800 0x7fd788004c80 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.180+0000 7fd7aac00640 1 -- 172.21.15.27:0/2526898641 >> 172.21.15.27:0/2526898641 conn(0x7fd7a406b9d0 msgr2=0x7fd7a40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:42.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.180+0000 7fd7aac00640 1 -- 172.21.15.27:0/2526898641 shutdown_connections 2024-03-28T19:14:42.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.180+0000 7fd7aac00640 1 -- 172.21.15.27:0/2526898641 wait complete. 2024-03-28T19:14:42.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.182+0000 7fd7aac00640 1 Processor -- start 2024-03-28T19:14:42.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.182+0000 7fd7aac00640 1 -- start start 2024-03-28T19:14:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd7aac00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd788004800 0x7fd7a415bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd7aac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7a40ab2a0 0x7fd7a41f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd7aac00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a41f37f0 0x7fd7a4152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd7aac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40c1160 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd7aac00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40c6380 con 0x7fd7a41f37f0 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd7aac00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40c12e0 con 0x7fd788004800 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd79be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7a40ab2a0 0x7fd7a41f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd79be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7a40ab2a0 0x7fd7a41f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55410/0 (socket says 172.21.15.27:55410) 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.183+0000 7fd79be00640 1 -- 172.21.15.27:0/447797781 learned_addr learned my addr 172.21.15.27:0/447797781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.184+0000 7fd7a3e00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd788004800 0x7fd7a415bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:42.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.184+0000 7fd7a8c00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a41f37f0 0x7fd7a4152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.184+0000 7fd79be00640 1 -- 172.21.15.27:0/447797781 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd788004800 msgr2=0x7fd7a415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.184+0000 7fd79be00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd788004800 0x7fd7a415bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.184+0000 7fd79be00640 1 -- 172.21.15.27:0/447797781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a41f37f0 msgr2=0x7fd7a4152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.184+0000 7fd79be00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a41f37f0 0x7fd7a4152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.184+0000 7fd79be00640 1 -- 172.21.15.27:0/447797781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a41531a0 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.184+0000 7fd79be00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7a40ab2a0 0x7fd7a41f32b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009be0 tx=0x7fd78c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.185+0000 7fd7a0c00640 1 -- 172.21.15.27:0/447797781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c042020 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.185+0000 7fd7a0c00640 1 -- 172.21.15.27:0/447797781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c03a040 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.185+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a40a7110 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.185+0000 7fd7a0c00640 1 -- 172.21.15.27:0/447797781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c039070 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.185+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a41534a0 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.186+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a404f400 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.188+0000 7fd7a0c00640 1 -- 172.21.15.27:0/447797781 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd78c045020 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.189+0000 7fd7a0c00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd78006bbd0 0x7fd78006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.189+0000 7fd7a0c00640 1 -- 172.21.15.27:0/447797781 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7fd78c0acb80 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.189+0000 7fd7a3e00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd78006bbd0 0x7fd78006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:42.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.190+0000 7fd7a3e00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd78006bbd0 0x7fd78006e090 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd7940011e0 tx=0x7fd794009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.194+0000 7fd7a0c00640 1 -- 172.21.15.27:0/447797781 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd78c04c070 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.484+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7a40ac190 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.485+0000 7fd7a0c00640 1 -- 172.21.15.27:0/447797781 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4676 (secure 0 0 0) 0x7fd78c07aa50 con 0x7fd7a40ab2a0 2024-03-28T19:14:42.485 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:42.485 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:41.692570+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:14:42.485 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 149 2024-03-28T19:14:42.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd78006bbd0 msgr2=0x7fd78006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd78006bbd0 0x7fd78006e090 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd7940011e0 tx=0x7fd794009040 comp rx=0 tx=0).stop 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7a40ab2a0 msgr2=0x7fd7a41f32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7a40ab2a0 0x7fd7a41f32b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009be0 tx=0x7fd78c004730 comp rx=0 tx=0).stop 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 shutdown_connections 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd78006bbd0 0x7fd78006e090 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a41f37f0 0x7fd7a4152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7a40ab2a0 0x7fd7a41f32b0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 --2- 172.21.15.27:0/447797781 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd788004800 0x7fd7a415bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:42.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.488+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 >> 172.21.15.27:0/447797781 conn(0x7fd7a406b9d0 msgr2=0x7fd7a40715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.489+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 shutdown_connections 2024-03-28T19:14:42.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:42.489+0000 7fd7aac00640 1 -- 172.21.15.27:0/447797781 wait complete. 2024-03-28T19:14:42.509 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:14:42.620 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:14:42.621+0000 7f60ca400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:14:43.838 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-03-28T19:14:43.838 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:14:43.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:14:43.847 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:14:43.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:44.076 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:14:44.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.131+0000 7f6fb6000640 1 Processor -- start 2024-03-28T19:14:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.131+0000 7f6fb6000640 1 -- start start 2024-03-28T19:14:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.133+0000 7f6fb6000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0152f80 0x7f6fb00c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.133+0000 7f6fb6000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6fb00c7970 0x7f6fb00c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.133+0000 7f6fb6000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb00c7310 con 0x7f6fb0155730 2024-03-28T19:14:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.133+0000 7f6fb6000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb0158d40 con 0x7f6fb00c7970 2024-03-28T19:14:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.133+0000 7f6fb6000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0159020 con 0x7f6fb0152f80 2024-03-28T19:14:44.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.133+0000 7f6faf400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0152f80 0x7f6fb00c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.134+0000 7f6faf400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0152f80 0x7f6fb00c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:56808/0 (socket says 172.21.15.27:56808) 2024-03-28T19:14:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.134+0000 7f6faf400640 1 -- 172.21.15.27:0/505054073 learned_addr learned my addr 172.21.15.27:0/505054073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.135+0000 7f6faf400640 1 -- 172.21.15.27:0/505054073 >> v2:172.21.15.155:3300/0 conn(0x7f6fb00c7970 msgr2=0x7f6fb00c7df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T19:14:44.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.135+0000 7f6faf400640 1 --2- 172.21.15.27:0/505054073 >> v2:172.21.15.155:3300/0 conn(0x7f6fb00c7970 0x7f6fb00c7df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.135+0000 7f6faf400640 1 -- 172.21.15.27:0/505054073 >> v1:172.21.15.27:6789/0 conn(0x7f6fb0155730 legacy=0x7f6fb0155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.135+0000 7f6faf400640 1 -- 172.21.15.27:0/505054073 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb0159300 con 0x7f6fb0152f80 2024-03-28T19:14:44.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.138+0000 7f6faf400640 1 --2- 172.21.15.27:0/505054073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0152f80 0x7f6fb00c58e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009870 tx=0x7f6f9802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=458185105e88f484 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.139+0000 7f6fae000640 1 -- 172.21.15.27:0/505054073 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98035020 con 0x7f6fb0152f80 2024-03-28T19:14:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.139+0000 7f6fae000640 1 -- 172.21.15.27:0/505054073 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f98031870 con 0x7f6fb0152f80 2024-03-28T19:14:44.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.139+0000 7f6fae000640 1 -- 172.21.15.27:0/505054073 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98031c40 con 0x7f6fb0152f80 2024-03-28T19:14:44.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.141+0000 7f6fb6000640 1 -- 172.21.15.27:0/505054073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0152f80 msgr2=0x7f6fb00c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.141+0000 7f6fb6000640 1 --2- 172.21.15.27:0/505054073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0152f80 0x7f6fb00c58e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009870 tx=0x7f6f9802b910 comp rx=0 tx=0).stop 2024-03-28T19:14:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.141+0000 7f6fb6000640 1 -- 172.21.15.27:0/505054073 shutdown_connections 2024-03-28T19:14:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.141+0000 7f6fb6000640 1 --2- 172.21.15.27:0/505054073 >> v2:172.21.15.155:3300/0 conn(0x7f6fb00c7970 0x7f6fb00c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.141+0000 7f6fb6000640 1 --2- 172.21.15.27:0/505054073 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0152f80 0x7f6fb00c58e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.141+0000 7f6fb6000640 1 -- 172.21.15.27:0/505054073 >> 172.21.15.27:0/505054073 conn(0x7f6fb006b6f0 msgr2=0x7f6fb00caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.141+0000 7f6fb6000640 1 -- 172.21.15.27:0/505054073 shutdown_connections 2024-03-28T19:14:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.141+0000 7f6fb6000640 1 -- 172.21.15.27:0/505054073 wait complete. 2024-03-28T19:14:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.143+0000 7f6fb6000640 1 Processor -- start 2024-03-28T19:14:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.144+0000 7f6fb6000640 1 -- start start 2024-03-28T19:14:44.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.146+0000 7f6fb6000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fb00c7970 0x7f6fb01f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.146+0000 7f6fb6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fb0152f80 0x7f6fb00c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.146+0000 7f6fb6000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0155730 0x7f6fb0148370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.146+0000 7f6fb6000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0159b00 con 0x7f6fb0152f80 2024-03-28T19:14:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.146+0000 7f6fb6000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00c5e20 con 0x7f6fb00c7970 2024-03-28T19:14:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6fb6000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00cbca0 con 0x7f6fb0155730 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fb00c7970 0x7f6fb01f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fb00c7970 0x7f6fb01f00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:60618/0 (socket says 172.21.15.27:60618) 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faf400640 1 -- 172.21.15.27:0/2097241820 learned_addr learned my addr 172.21.15.27:0/2097241820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6fafe00640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0155730 0x7f6fb0148370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faf400640 1 -- 172.21.15.27:0/2097241820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0155730 msgr2=0x7f6fb0148370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faf400640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0155730 0x7f6fb0148370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faf400640 1 -- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fb0152f80 msgr2=0x7f6fb00c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faea00640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fb0152f80 0x7f6fb00c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faf400640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fb0152f80 0x7f6fb00c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.147+0000 7f6faf400640 1 -- 172.21.15.27:0/2097241820 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00c4fe0 con 0x7f6fb00c7970 2024-03-28T19:14:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.148+0000 7f6faea00640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fb0152f80 0x7f6fb00c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:14:44.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.148+0000 7f6faf400640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fb00c7970 0x7f6fb01f00f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6f98000c00 tx=0x7f6f980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.149+0000 7f6fa7400640 1 -- 172.21.15.27:0/2097241820 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98047070 con 0x7f6fb00c7970 2024-03-28T19:14:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.149+0000 7f6fa7400640 1 -- 172.21.15.27:0/2097241820 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9803ad60 con 0x7f6fb00c7970 2024-03-28T19:14:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.149+0000 7f6fa7400640 1 -- 172.21.15.27:0/2097241820 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98042630 con 0x7f6fb00c7970 2024-03-28T19:14:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.150+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb00c4a20 con 0x7f6fb00c7970 2024-03-28T19:14:44.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.150+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb0149540 con 0x7f6fb00c7970 2024-03-28T19:14:44.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.152+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb004f400 con 0x7f6fb00c7970 2024-03-28T19:14:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.153+0000 7f6fa7400640 1 -- 172.21.15.27:0/2097241820 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6f9804a070 con 0x7f6fb00c7970 2024-03-28T19:14:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.155+0000 7f6fa7400640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8c06bc20 0x7f6f8c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.155+0000 7f6fa7400640 1 -- 172.21.15.27:0/2097241820 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f6f980ad640 con 0x7f6fb00c7970 2024-03-28T19:14:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.156+0000 7f6faea00640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8c06bc20 0x7f6f8c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.157+0000 7f6faea00640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8c06bc20 0x7f6f8c06e0e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6fb014b010 tx=0x7f6fa0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.160+0000 7f6fa7400640 1 -- 172.21.15.27:0/2097241820 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6f9803f070 con 0x7f6fb00c7970 2024-03-28T19:14:44.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:44.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.524+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fb00cba80 con 0x7f6fb00c7970 2024-03-28T19:14:44.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.526+0000 7f6fa7400640 1 -- 172.21.15.27:0/2097241820 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+4679 (secure 0 0 0) 0x7f6f9807b530 con 0x7f6fb00c7970 2024-03-28T19:14:44.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:44.526 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:42.972326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:clientreplay","state_seq":38,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:14:44.526 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 151 2024-03-28T19:14:44.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.528+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8c06bc20 msgr2=0x7f6f8c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.528+0000 7f6fb6000640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8c06bc20 0x7f6f8c06e0e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6fb014b010 tx=0x7f6fa0008040 comp rx=0 tx=0).stop 2024-03-28T19:14:44.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.528+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fb00c7970 msgr2=0x7f6fb01f00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.528+0000 7f6fb6000640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fb00c7970 0x7f6fb01f00f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6f98000c00 tx=0x7f6f980046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.529+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 shutdown_connections 2024-03-28T19:14:44.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.529+0000 7f6fb6000640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6f8c06bc20 0x7f6f8c06e0e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.529+0000 7f6fb6000640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fb0155730 0x7f6fb0148370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.529+0000 7f6fb6000640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fb0152f80 0x7f6fb00c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.529+0000 7f6fb6000640 1 --2- 172.21.15.27:0/2097241820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fb00c7970 0x7f6fb01f00f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.529+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 >> 172.21.15.27:0/2097241820 conn(0x7f6fb006b6f0 msgr2=0x7f6fb0155b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:44.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.529+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 shutdown_connections 2024-03-28T19:14:44.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.529+0000 7f6fb6000640 1 -- 172.21.15.27:0/2097241820 wait complete. 2024-03-28T19:14:44.550 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:14:44.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.738+0000 7f9d1fe00640 1 Processor -- start 2024-03-28T19:14:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.739+0000 7f9d1fe00640 1 -- start start 2024-03-28T19:14:44.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.741+0000 7f9d1fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9d200c1780 0x7f9d20155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.741+0000 7f9d1fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d200a7400 con 0x7f9d2006c0d0 2024-03-28T19:14:44.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.741+0000 7f9d1fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d20158b40 con 0x7f9d200a6e90 2024-03-28T19:14:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.741+0000 7f9d1fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d20159470 con 0x7f9d200c1780 2024-03-28T19:14:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.741+0000 7f9d1ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9d2006c0d0 0x7f9d20155530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56532/0 (socket says 172.21.15.27:56532) 2024-03-28T19:14:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.741+0000 7f9d1ea00640 1 -- 172.21.15.27:0/3392703271 learned_addr learned my addr 172.21.15.27:0/3392703271 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.741+0000 7f9d1f400640 1 --2- 172.21.15.27:0/3392703271 >> v2:172.21.15.27:3300/0 conn(0x7f9d200c1780 0x7f9d20155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.742+0000 7f9d1f400640 1 -- 172.21.15.27:0/3392703271 >> v1:172.21.15.155:6789/0 conn(0x7f9d200a6e90 legacy=0x7f9d2015ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.742+0000 7f9d1f400640 1 -- 172.21.15.27:0/3392703271 >> v1:172.21.15.27:6789/0 conn(0x7f9d2006c0d0 legacy=0x7f9d20155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.742+0000 7f9d1f400640 1 -- 172.21.15.27:0/3392703271 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d20158e20 con 0x7f9d200c1780 2024-03-28T19:14:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.742+0000 7f9d1d600640 1 -- 172.21.15.27:0/3392703271 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080810023 0 0) 0x7f9d200a7400 con 0x7f9d2006c0d0 2024-03-28T19:14:44.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.743+0000 7f9d1f400640 1 --2- 172.21.15.27:0/3392703271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d200c1780 0x7f9d20155c40 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009d90 tx=0x7f9d1802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b0bb57fcf7f148 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1d600640 1 -- 172.21.15.27:0/3392703271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18039070 con 0x7f9d200c1780 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1d600640 1 -- 172.21.15.27:0/3392703271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1802cc70 con 0x7f9d200c1780 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1d600640 1 -- 172.21.15.27:0/3392703271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18034660 con 0x7f9d200c1780 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1fe00640 1 -- 172.21.15.27:0/3392703271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d200c1780 msgr2=0x7f9d20155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1fe00640 1 --2- 172.21.15.27:0/3392703271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d200c1780 0x7f9d20155c40 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009d90 tx=0x7f9d1802bec0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1fe00640 1 -- 172.21.15.27:0/3392703271 shutdown_connections 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1fe00640 1 --2- 172.21.15.27:0/3392703271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d200c1780 0x7f9d20155c40 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1fe00640 1 -- 172.21.15.27:0/3392703271 >> 172.21.15.27:0/3392703271 conn(0x7f9d200c7aa0 msgr2=0x7f9d200c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.744+0000 7f9d1fe00640 1 -- 172.21.15.27:0/3392703271 shutdown_connections 2024-03-28T19:14:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.745+0000 7f9d1fe00640 1 -- 172.21.15.27:0/3392703271 wait complete. 2024-03-28T19:14:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.747+0000 7f9d1fe00640 1 Processor -- start 2024-03-28T19:14:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.747+0000 7f9d1fe00640 1 -- start start 2024-03-28T19:14:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.748+0000 7f9d1fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2006c0d0 0x7f9d201eee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.748+0000 7f9d1fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d200a6e90 0x7f9d2014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.748+0000 7f9d1fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d200c1780 0x7f9d200a7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.748+0000 7f9d1fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d20152e20 con 0x7f9d2006c0d0 2024-03-28T19:14:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.748+0000 7f9d1fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d20159900 con 0x7f9d200a6e90 2024-03-28T19:14:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.748+0000 7f9d1fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d20152fa0 con 0x7f9d200c1780 2024-03-28T19:14:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.748+0000 7f9d1ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2006c0d0 0x7f9d201eee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2006c0d0 0x7f9d201eee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55456/0 (socket says 172.21.15.27:55456) 2024-03-28T19:14:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1ea00640 1 -- 172.21.15.27:0/1899319561 learned_addr learned my addr 172.21.15.27:0/1899319561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:44.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1e000640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d200a6e90 0x7f9d2014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1f400640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d200c1780 0x7f9d200a7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1ea00640 1 -- 172.21.15.27:0/1899319561 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d200c1780 msgr2=0x7f9d200a7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1ea00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d200c1780 0x7f9d200a7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1ea00640 1 -- 172.21.15.27:0/1899319561 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d200a6e90 msgr2=0x7f9d2014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1ea00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d200a6e90 0x7f9d2014fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.749+0000 7f9d1ea00640 1 -- 172.21.15.27:0/1899319561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d201eb240 con 0x7f9d2006c0d0 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.750+0000 7f9d1ea00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2006c0d0 0x7f9d201eee70 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9d1000c4a0 tx=0x7f9d1000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.750+0000 7f9d16a00640 1 -- 172.21.15.27:0/1899319561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10018070 con 0x7f9d2006c0d0 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.750+0000 7f9d16a00640 1 -- 172.21.15.27:0/1899319561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d100074e0 con 0x7f9d2006c0d0 2024-03-28T19:14:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.750+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d201eb520 con 0x7f9d2006c0d0 2024-03-28T19:14:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.750+0000 7f9d16a00640 1 -- 172.21.15.27:0/1899319561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10004a20 con 0x7f9d2006c0d0 2024-03-28T19:14:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.750+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d201502e0 con 0x7f9d2006c0d0 2024-03-28T19:14:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.752+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d20069640 con 0x7f9d2006c0d0 2024-03-28T19:14:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.753+0000 7f9d16a00640 1 -- 172.21.15.27:0/1899319561 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9d1001b070 con 0x7f9d2006c0d0 2024-03-28T19:14:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.754+0000 7f9d16a00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d0006bbd0 0x7f9d0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.754+0000 7f9d16a00640 1 -- 172.21.15.27:0/1899319561 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f9d1008aa00 con 0x7f9d2006c0d0 2024-03-28T19:14:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.754+0000 7f9d1e000640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d0006bbd0 0x7f9d0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.755+0000 7f9d1e000640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d0006bbd0 0x7f9d0006e090 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9d200ac4e0 tx=0x7f9d0c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:44.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:44.759+0000 7f9d16a00640 1 -- 172.21.15.27:0/1899319561 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9d10058870 con 0x7f9d2006c0d0 2024-03-28T19:14:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.097+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d2014e690 con 0x7f9d2006c0d0 2024-03-28T19:14:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.097+0000 7f9d16a00640 1 -- 172.21.15.27:0/1899319561 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f9d10058170 con 0x7f9d2006c0d0 2024-03-28T19:14:45.097 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:45.097 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:14:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 152 2024-03-28T19:14:45.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.100+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d0006bbd0 msgr2=0x7f9d0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:45.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.100+0000 7f9d1fe00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d0006bbd0 0x7f9d0006e090 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9d200ac4e0 tx=0x7f9d0c009040 comp rx=0 tx=0).stop 2024-03-28T19:14:45.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.100+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2006c0d0 msgr2=0x7f9d201eee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:45.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.100+0000 7f9d1fe00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2006c0d0 0x7f9d201eee70 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9d1000c4a0 tx=0x7f9d1000c960 comp rx=0 tx=0).stop 2024-03-28T19:14:45.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.101+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 shutdown_connections 2024-03-28T19:14:45.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.101+0000 7f9d1fe00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d0006bbd0 0x7f9d0006e090 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:45.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.101+0000 7f9d1fe00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d200c1780 0x7f9d200a7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:45.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.101+0000 7f9d1fe00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d200a6e90 0x7f9d2014fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:45.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.101+0000 7f9d1fe00640 1 --2- 172.21.15.27:0/1899319561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2006c0d0 0x7f9d201eee70 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:45.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.101+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 >> 172.21.15.27:0/1899319561 conn(0x7f9d200c7aa0 msgr2=0x7f9d200c6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:45.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.101+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 shutdown_connections 2024-03-28T19:14:45.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:45.101+0000 7f9d1fe00640 1 -- 172.21.15.27:0/1899319561 wait complete. 2024-03-28T19:14:45.122 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5756 2024-03-28T19:14:45.122 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.b 2024-03-28T19:14:46.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:46.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.784+0000 7ff1a6000640 1 Processor -- start 2024-03-28T19:14:46.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.784+0000 7ff1a6000640 1 -- start start 2024-03-28T19:14:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.786+0000 7ff1a6000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1a0152da0 0x7ff1a00c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.786+0000 7ff1a6000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a006b160 con 0x7ff1a00732e0 2024-03-28T19:14:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.786+0000 7ff1a6000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a0156450 con 0x7ff1a00c2e40 2024-03-28T19:14:46.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.786+0000 7ff1a6000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a00c2100 con 0x7ff1a0152da0 2024-03-28T19:14:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.786+0000 7ff19fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff1a00732e0 0x7ff1a015ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54596/0 (socket says 172.21.15.27:54596) 2024-03-28T19:14:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.786+0000 7ff19f400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1a0152da0 0x7ff1a00c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:46.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.786+0000 7ff19fe00640 1 -- 172.21.15.27:0/1859404497 learned_addr learned my addr 172.21.15.27:0/1859404497 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.787+0000 7ff19e000640 1 -- 172.21.15.27:0/1859404497 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312613976 0 0) 0x7ff1a006b160 con 0x7ff1a00732e0 2024-03-28T19:14:46.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.787+0000 7ff19e000640 1 -- 172.21.15.27:0/1859404497 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff184003520 con 0x7ff1a00732e0 2024-03-28T19:14:46.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.787+0000 7ff19f400640 1 -- 172.21.15.27:0/1859404497 >> v1:172.21.15.155:6789/0 conn(0x7ff1a00c2e40 legacy=0x7ff1a006b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.787+0000 7ff19f400640 1 -- 172.21.15.27:0/1859404497 >> v1:172.21.15.27:6789/0 conn(0x7ff1a00732e0 legacy=0x7ff1a015ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.787+0000 7ff19f400640 1 -- 172.21.15.27:0/1859404497 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a00c2350 con 0x7ff1a0152da0 2024-03-28T19:14:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.789+0000 7ff19f400640 1 --2- 172.21.15.27:0/1859404497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a0152da0 0x7ff1a00c2900 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff1900097a0 tx=0x7ff19002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0ea3209db510c16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:46.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.790+0000 7ff19e000640 1 -- 172.21.15.27:0/1859404497 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1900315a0 con 0x7ff1a0152da0 2024-03-28T19:14:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.790+0000 7ff19e000640 1 -- 172.21.15.27:0/1859404497 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff190037070 con 0x7ff1a0152da0 2024-03-28T19:14:46.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.790+0000 7ff19e000640 1 -- 172.21.15.27:0/1859404497 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190031960 con 0x7ff1a0152da0 2024-03-28T19:14:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.791+0000 7ff1a6000640 1 -- 172.21.15.27:0/1859404497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a0152da0 msgr2=0x7ff1a00c2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.791+0000 7ff1a6000640 1 --2- 172.21.15.27:0/1859404497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a0152da0 0x7ff1a00c2900 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff1900097a0 tx=0x7ff19002b840 comp rx=0 tx=0).stop 2024-03-28T19:14:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.791+0000 7ff1a6000640 1 -- 172.21.15.27:0/1859404497 shutdown_connections 2024-03-28T19:14:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.791+0000 7ff1a6000640 1 --2- 172.21.15.27:0/1859404497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a0152da0 0x7ff1a00c2900 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.791+0000 7ff1a6000640 1 -- 172.21.15.27:0/1859404497 >> 172.21.15.27:0/1859404497 conn(0x7ff1a00c3ff0 msgr2=0x7ff1a00c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.791+0000 7ff1a6000640 1 -- 172.21.15.27:0/1859404497 shutdown_connections 2024-03-28T19:14:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.791+0000 7ff1a6000640 1 -- 172.21.15.27:0/1859404497 wait complete. 2024-03-28T19:14:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.794+0000 7ff1a6000640 1 Processor -- start 2024-03-28T19:14:46.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.794+0000 7ff1a6000640 1 -- start start 2024-03-28T19:14:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.796+0000 7ff1a6000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00732e0 0x7ff1a01eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.796+0000 7ff1a6000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c2e40 0x7ff1a0158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.796+0000 7ff1a6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a0152da0 0x7ff1a01591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.796+0000 7ff1a6000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a0067130 con 0x7ff1a0152da0 2024-03-28T19:14:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.796+0000 7ff1a6000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a01560e0 con 0x7ff1a00c2e40 2024-03-28T19:14:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.796+0000 7ff1a6000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00672b0 con 0x7ff1a00732e0 2024-03-28T19:14:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.796+0000 7ff19f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00732e0 0x7ff1a01eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.797+0000 7ff19f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00732e0 0x7ff1a01eeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:46054/0 (socket says 172.21.15.27:46054) 2024-03-28T19:14:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.797+0000 7ff19fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a0152da0 0x7ff1a01591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.797+0000 7ff19f400640 1 -- 172.21.15.27:0/1578229017 learned_addr learned my addr 172.21.15.27:0/1578229017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.797+0000 7ff19ea00640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c2e40 0x7ff1a0158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.798+0000 7ff19fe00640 1 -- 172.21.15.27:0/1578229017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00732e0 msgr2=0x7ff1a01eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.798+0000 7ff19fe00640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00732e0 0x7ff1a01eeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.798+0000 7ff19fe00640 1 -- 172.21.15.27:0/1578229017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c2e40 msgr2=0x7ff1a0158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.798+0000 7ff19fe00640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c2e40 0x7ff1a0158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.798+0000 7ff19fe00640 1 -- 172.21.15.27:0/1578229017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a0157530 con 0x7ff1a0152da0 2024-03-28T19:14:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.799+0000 7ff19fe00640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a0152da0 0x7ff1a01591f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff19800caa0 tx=0x7ff19800cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.799+0000 7ff197400640 1 -- 172.21.15.27:0/1578229017 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198017070 con 0x7ff1a0152da0 2024-03-28T19:14:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.799+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a0151ec0 con 0x7ff1a0152da0 2024-03-28T19:14:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.799+0000 7ff197400640 1 -- 172.21.15.27:0/1578229017 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19800ab40 con 0x7ff1a0152da0 2024-03-28T19:14:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.800+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a01ef5f0 con 0x7ff1a0152da0 2024-03-28T19:14:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.800+0000 7ff197400640 1 -- 172.21.15.27:0/1578229017 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198012400 con 0x7ff1a0152da0 2024-03-28T19:14:46.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.801+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff178005290 con 0x7ff1a0152da0 2024-03-28T19:14:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.803+0000 7ff197400640 1 -- 172.21.15.27:0/1578229017 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff19801a070 con 0x7ff1a0152da0 2024-03-28T19:14:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.805+0000 7ff197400640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bc20 0x7ff18006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.805+0000 7ff197400640 1 -- 172.21.15.27:0/1578229017 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7ff19808b2c0 con 0x7ff1a0152da0 2024-03-28T19:14:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.805+0000 7ff19f400640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bc20 0x7ff18006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.806+0000 7ff19f400640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bc20 0x7ff18006e0e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff1900097a0 tx=0x7ff19003d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:46.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:46.810+0000 7ff197400640 1 -- 172.21.15.27:0/1578229017 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff198058b60 con 0x7ff1a0152da0 2024-03-28T19:14:47.123 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-03-28T19:14:47.124 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:14:47.124 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:14:47.130 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:14:47.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:47.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.154+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff178005580 con 0x7ff1a0152da0 2024-03-28T19:14:47.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.155+0000 7ff197400640 1 -- 172.21.15.27:0/1578229017 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5471 (secure 0 0 0) 0x7ff19805cc00 con 0x7ff1a0152da0 2024-03-28T19:14:47.155 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:47.155 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:14:47.155 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 152 2024-03-28T19:14:47.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bc20 msgr2=0x7ff18006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:47.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bc20 0x7ff18006e0e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff1900097a0 tx=0x7ff19003d040 comp rx=0 tx=0).stop 2024-03-28T19:14:47.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a0152da0 msgr2=0x7ff1a01591f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:47.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a0152da0 0x7ff1a01591f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff19800caa0 tx=0x7ff19800cea0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 shutdown_connections 2024-03-28T19:14:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff18006bc20 0x7ff18006e0e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1a0152da0 0x7ff1a01591f0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a00c2e40 0x7ff1a0158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 --2- 172.21.15.27:0/1578229017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a00732e0 0x7ff1a01eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.159+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 >> 172.21.15.27:0/1578229017 conn(0x7ff1a00c3ff0 msgr2=0x7ff1a006b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.160+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 shutdown_connections 2024-03-28T19:14:47.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.160+0000 7ff1a6000640 1 -- 172.21.15.27:0/1578229017 wait complete. 2024-03-28T19:14:47.181 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-03-28T19:14:47.181 INFO:tasks.mds_thrash.fs.[a]:waiting for 69 secs before thrashing 2024-03-28T19:14:47.356 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:14:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.380+0000 7f9e69000640 1 Processor -- start 2024-03-28T19:14:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.381+0000 7f9e69000640 1 -- start start 2024-03-28T19:14:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.382+0000 7f9e69000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e640c8ea0 0x7f9e64156060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.382+0000 7f9e69000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e640c9320 con 0x7f9e64152c80 2024-03-28T19:14:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.382+0000 7f9e69000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e641558a0 con 0x7f9e640c79b0 2024-03-28T19:14:47.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.382+0000 7f9e69000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e640c5cc0 con 0x7f9e640c8ea0 2024-03-28T19:14:47.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.382+0000 7f9e5be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9e640c8ea0 0x7f9e64156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:47.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.382+0000 7f9e63400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9e64152c80 0x7f9e641596b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54600/0 (socket says 172.21.15.27:54600) 2024-03-28T19:14:47.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.382+0000 7f9e63400640 1 -- 172.21.15.27:0/1110698344 learned_addr learned my addr 172.21.15.27:0/1110698344 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:47.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.383+0000 7f9e5be00640 1 -- 172.21.15.27:0/1110698344 >> v1:172.21.15.155:6789/0 conn(0x7f9e640c79b0 legacy=0x7f9e64160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:47.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.383+0000 7f9e5be00640 1 -- 172.21.15.27:0/1110698344 >> v1:172.21.15.27:6789/0 conn(0x7f9e64152c80 legacy=0x7f9e641596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:47.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.383+0000 7f9e5be00640 1 -- 172.21.15.27:0/1110698344 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e640c5fa0 con 0x7f9e640c8ea0 2024-03-28T19:14:47.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.383+0000 7f9e62a00640 1 -- 172.21.15.27:0/1110698344 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849343885 0 0) 0x7f9e54001110 con 0x7f9e64152c80 2024-03-28T19:14:47.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.385+0000 7f9e5be00640 1 --2- 172.21.15.27:0/1110698344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156060 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0097a0 tx=0x7f9e4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1001dfab9629ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:47.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.385+0000 7f9e62a00640 1 -- 172.21.15.27:0/1110698344 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0315a0 con 0x7f9e640c8ea0 2024-03-28T19:14:47.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.385+0000 7f9e62a00640 1 -- 172.21.15.27:0/1110698344 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c037070 con 0x7f9e640c8ea0 2024-03-28T19:14:47.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.385+0000 7f9e62a00640 1 -- 172.21.15.27:0/1110698344 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c031960 con 0x7f9e640c8ea0 2024-03-28T19:14:47.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.385+0000 7f9e69000640 1 -- 172.21.15.27:0/1110698344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 msgr2=0x7f9e64156060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:47.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.385+0000 7f9e69000640 1 --2- 172.21.15.27:0/1110698344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156060 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0097a0 tx=0x7f9e4c02b840 comp rx=0 tx=0).stop 2024-03-28T19:14:47.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.386+0000 7f9e69000640 1 -- 172.21.15.27:0/1110698344 shutdown_connections 2024-03-28T19:14:47.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.386+0000 7f9e69000640 1 --2- 172.21.15.27:0/1110698344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156060 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.386+0000 7f9e69000640 1 -- 172.21.15.27:0/1110698344 >> 172.21.15.27:0/1110698344 conn(0x7f9e6406b6c0 msgr2=0x7f9e640cafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:47.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.386+0000 7f9e69000640 1 -- 172.21.15.27:0/1110698344 shutdown_connections 2024-03-28T19:14:47.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.386+0000 7f9e69000640 1 -- 172.21.15.27:0/1110698344 wait complete. 2024-03-28T19:14:47.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.388+0000 7f9e69000640 1 Processor -- start 2024-03-28T19:14:47.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.388+0000 7f9e69000640 1 -- start start 2024-03-28T19:14:47.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e69000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e640c79b0 0x7f9e64146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:47.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e69000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:47.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e69000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e64152c80 0x7f9e641e8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:47.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e69000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e640cbd70 con 0x7f9e640c8ea0 2024-03-28T19:14:47.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e69000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e640c6280 con 0x7f9e64152c80 2024-03-28T19:14:47.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e69000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e641565a0 con 0x7f9e640c79b0 2024-03-28T19:14:47.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e5be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e63e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e64152c80 0x7f9e641e8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e63400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e640c79b0 0x7f9e64146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e63e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e64152c80 0x7f9e641e8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:60198/0 (socket says 172.21.15.27:60198) 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e63e00640 1 -- 172.21.15.27:0/1076179957 learned_addr learned my addr 172.21.15.27:0/1076179957 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.390+0000 7f9e5be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33630/0 (socket says 172.21.15.27:33630) 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.391+0000 7f9e5be00640 1 -- 172.21.15.27:0/1076179957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e640c79b0 msgr2=0x7f9e64146360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.391+0000 7f9e5be00640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e640c79b0 0x7f9e64146360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.391+0000 7f9e5be00640 1 -- 172.21.15.27:0/1076179957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e64152c80 msgr2=0x7f9e641e8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.391+0000 7f9e5be00640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e64152c80 0x7f9e641e8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.391+0000 7f9e5be00640 1 -- 172.21.15.27:0/1076179957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64147040 con 0x7f9e640c8ea0 2024-03-28T19:14:47.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.391+0000 7f9e5be00640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156d60 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c033000 tx=0x7f9e4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:47.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.392+0000 7f9e5b400640 1 -- 172.21.15.27:0/1076179957 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c032840 con 0x7f9e640c8ea0 2024-03-28T19:14:47.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.392+0000 7f9e5b400640 1 -- 172.21.15.27:0/1076179957 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c032a20 con 0x7f9e640c8ea0 2024-03-28T19:14:47.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.392+0000 7f9e5b400640 1 -- 172.21.15.27:0/1076179957 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c032e10 con 0x7f9e640c8ea0 2024-03-28T19:14:47.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.392+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e64158d70 con 0x7f9e640c8ea0 2024-03-28T19:14:47.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.392+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e641ef070 con 0x7f9e640c8ea0 2024-03-28T19:14:47.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.393+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e64147210 con 0x7f9e640c8ea0 2024-03-28T19:14:47.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.394+0000 7f9e5b400640 1 -- 172.21.15.27:0/1076179957 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9e4c03b3f0 con 0x7f9e640c8ea0 2024-03-28T19:14:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.395+0000 7f9e5b400640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e4006bc00 0x7f9e4006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.395+0000 7f9e5b400640 1 -- 172.21.15.27:0/1076179957 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f9e4c02f030 con 0x7f9e640c8ea0 2024-03-28T19:14:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.396+0000 7f9e63400640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e4006bc00 0x7f9e4006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:47.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.397+0000 7f9e63400640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e4006bc00 0x7f9e4006e0c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9e54007fd0 tx=0x7f9e54007de0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:47.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.401+0000 7f9e5b400640 1 -- 172.21.15.27:0/1076179957 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9e4c040070 con 0x7f9e640c8ea0 2024-03-28T19:14:47.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.742+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e64069640 con 0x7f9e640c8ea0 2024-03-28T19:14:47.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.743+0000 7f9e5b400640 1 -- 172.21.15.27:0/1076179957 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f9e4c07aad0 con 0x7f9e640c8ea0 2024-03-28T19:14:47.743 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:47.743 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:14:47.743 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 152 2024-03-28T19:14:47.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.746+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e4006bc00 msgr2=0x7f9e4006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.746+0000 7f9e69000640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e4006bc00 0x7f9e4006e0c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9e54007fd0 tx=0x7f9e54007de0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.746+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 msgr2=0x7f9e64156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.746+0000 7f9e69000640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156d60 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c033000 tx=0x7f9e4c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.747+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 shutdown_connections 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.747+0000 7f9e69000640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9e4006bc00 0x7f9e4006e0c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.747+0000 7f9e69000640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e64152c80 0x7f9e641e8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.747+0000 7f9e69000640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e640c8ea0 0x7f9e64156d60 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.747+0000 7f9e69000640 1 --2- 172.21.15.27:0/1076179957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9e640c79b0 0x7f9e64146360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:47.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.747+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 >> 172.21.15.27:0/1076179957 conn(0x7f9e6406b6c0 msgr2=0x7f9e6414c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.747+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 shutdown_connections 2024-03-28T19:14:47.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:47.747+0000 7f9e69000640 1 -- 172.21.15.27:0/1076179957 wait complete. 2024-03-28T19:14:47.765 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-03-28T19:14:48.844 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T19:14:48.844+0000 7fb99a400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:14:49.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:50.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.014+0000 7fd4ab200640 1 Processor -- start 2024-03-28T19:14:50.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.014+0000 7fd4ab200640 1 -- start start 2024-03-28T19:14:50.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4ab200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4ac0c39c0 0x7fd4ac153430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:50.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4ab200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4ac0c5b20 0x7fd4ac0c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4ab200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac1569a0 con 0x7fd4ac0c7350 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4ab200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac157870 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4ab200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac156c60 con 0x7fd4ac0c39c0 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4aa800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4ac0c5b20 0x7fd4ac0c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4a9e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4ac0c39c0 0x7fd4ac153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4a9e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4ac0c39c0 0x7fd4ac153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:60212/0 (socket says 172.21.15.27:60212) 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4a9e00640 1 -- 172.21.15.27:0/2900389924 learned_addr learned my addr 172.21.15.27:0/2900389924 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4aa800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4ac0c5b20 0x7fd4ac0c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33640/0 (socket says 172.21.15.27:33640) 2024-03-28T19:14:50.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.016+0000 7fd4a9400640 1 --1- 172.21.15.27:0/2900389924 >> v1:172.21.15.155:6789/0 conn(0x7fd4ac0c7350 0x7fd4ac066dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:47570/0 (socket says 172.21.15.27:47570) 2024-03-28T19:14:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.017+0000 7fd4aa800640 1 -- 172.21.15.27:0/2900389924 >> v2:172.21.15.155:3300/0 conn(0x7fd4ac0c39c0 msgr2=0x7fd4ac153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.017+0000 7fd4aa800640 1 --2- 172.21.15.27:0/2900389924 >> v2:172.21.15.155:3300/0 conn(0x7fd4ac0c39c0 0x7fd4ac153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.017+0000 7fd4aa800640 1 -- 172.21.15.27:0/2900389924 >> v1:172.21.15.155:6789/0 conn(0x7fd4ac0c7350 legacy=0x7fd4ac066dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.017+0000 7fd4aa800640 1 -- 172.21.15.27:0/2900389924 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac156f40 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.017+0000 7fd4a9e00640 1 --2- 172.21.15.27:0/2900389924 >> v2:172.21.15.155:3300/0 conn(0x7fd4ac0c39c0 0x7fd4ac153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:14:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.018+0000 7fd4aa800640 1 --2- 172.21.15.27:0/2900389924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac0c5f50 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac062400 tx=0x7fd4a402bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af873e967221dc3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.019+0000 7fd4a8a00640 1 -- 172.21.15.27:0/2900389924 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4039070 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.019+0000 7fd4a8a00640 1 -- 172.21.15.27:0/2900389924 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a402c9a0 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.019+0000 7fd4a8a00640 1 -- 172.21.15.27:0/2900389924 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a402cd70 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.019+0000 7fd4ab200640 1 -- 172.21.15.27:0/2900389924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 msgr2=0x7fd4ac0c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.019+0000 7fd4ab200640 1 --2- 172.21.15.27:0/2900389924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac0c5f50 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac062400 tx=0x7fd4a402bc70 comp rx=0 tx=0).stop 2024-03-28T19:14:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.020+0000 7fd4ab200640 1 -- 172.21.15.27:0/2900389924 shutdown_connections 2024-03-28T19:14:50.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.020+0000 7fd4ab200640 1 --2- 172.21.15.27:0/2900389924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac0c5f50 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:50.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.020+0000 7fd4ab200640 1 --2- 172.21.15.27:0/2900389924 >> v2:172.21.15.155:3300/0 conn(0x7fd4ac0c39c0 0x7fd4ac153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:50.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.020+0000 7fd4ab200640 1 -- 172.21.15.27:0/2900389924 >> 172.21.15.27:0/2900389924 conn(0x7fd4ac06b0c0 msgr2=0x7fd4ac06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:50.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.020+0000 7fd4ab200640 1 -- 172.21.15.27:0/2900389924 shutdown_connections 2024-03-28T19:14:50.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.020+0000 7fd4ab200640 1 -- 172.21.15.27:0/2900389924 wait complete. 2024-03-28T19:14:50.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.022+0000 7fd4ab200640 1 Processor -- start 2024-03-28T19:14:50.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.022+0000 7fd4ab200640 1 -- start start 2024-03-28T19:14:50.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4ab200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4ac0c39c0 0x7fd4ac0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:50.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4ab200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:50.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4ab200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4ac0c7350 0x7fd4ac150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:50.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4ab200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0674e0 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4ab200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac157500 con 0x7fd4ac0c7350 2024-03-28T19:14:50.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4ab200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac067660 con 0x7fd4ac0c39c0 2024-03-28T19:14:50.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4a9e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4ac0c39c0 0x7fd4ac0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:50.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4a9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:50.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4aa800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4ac0c7350 0x7fd4ac150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:50.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4a9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33648/0 (socket says 172.21.15.27:33648) 2024-03-28T19:14:50.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.023+0000 7fd4a9400640 1 -- 172.21.15.27:0/3658430362 learned_addr learned my addr 172.21.15.27:0/3658430362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:50.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.024+0000 7fd4a9400640 1 -- 172.21.15.27:0/3658430362 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4ac0c39c0 msgr2=0x7fd4ac0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:50.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.024+0000 7fd4a9400640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4ac0c39c0 0x7fd4ac0ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:50.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.024+0000 7fd4a9400640 1 -- 172.21.15.27:0/3658430362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4ac0c7350 msgr2=0x7fd4ac150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:50.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.024+0000 7fd4a9400640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4ac0c7350 0x7fd4ac150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:50.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.024+0000 7fd4a9400640 1 -- 172.21.15.27:0/3658430362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac06efb0 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.025+0000 7fd4a9400640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac150700 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c780 tx=0x7fd49800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:50.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.025+0000 7fd4a2000640 1 -- 172.21.15.27:0/3658430362 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498018070 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.025+0000 7fd4a2000640 1 -- 172.21.15.27:0/3658430362 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4980072d0 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.025+0000 7fd4a2000640 1 -- 172.21.15.27:0/3658430362 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980047e0 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.025+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac06f290 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.025+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac0ad610 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.027+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac069640 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.028+0000 7fd4a2000640 1 -- 172.21.15.27:0/3658430362 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd49801b070 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.029+0000 7fd4a2000640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd48c06bc50 0x7fd48c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.029+0000 7fd4a2000640 1 -- 172.21.15.27:0/3658430362 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7fd49808aa80 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.030+0000 7fd4a9e00640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd48c06bc50 0x7fd48c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:50.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.031+0000 7fd4a9e00640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd48c06bc50 0x7fd48c06e110 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd49c006ca0 tx=0x7fd49c009240 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.034+0000 7fd4a2000640 1 -- 172.21.15.27:0/3658430362 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd498058440 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.352+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4ac0a6f20 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.353+0000 7fd4a2000640 1 -- 172.21.15.27:0/3658430362 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fd49808ae20 con 0x7fd4ac0c5b20 2024-03-28T19:14:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:50.354 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:14:50.354 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 153 2024-03-28T19:14:50.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd48c06bc50 msgr2=0x7fd48c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:50.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd48c06bc50 0x7fd48c06e110 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd49c006ca0 tx=0x7fd49c009240 comp rx=0 tx=0).stop 2024-03-28T19:14:50.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 msgr2=0x7fd4ac150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:50.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac150700 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c780 tx=0x7fd49800cc40 comp rx=0 tx=0).stop 2024-03-28T19:14:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 shutdown_connections 2024-03-28T19:14:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd48c06bc50 0x7fd48c06e110 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4ac0c7350 0x7fd4ac150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4ac0c5b20 0x7fd4ac150700 secure :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c780 tx=0x7fd49800cc40 comp rx=0 tx=0).stop 2024-03-28T19:14:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 --2- 172.21.15.27:0/3658430362 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4ac0c39c0 0x7fd4ac0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.356+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 >> 172.21.15.27:0/3658430362 conn(0x7fd4ac06b0c0 msgr2=0x7fd4ac0c5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.357+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 shutdown_connections 2024-03-28T19:14:50.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:50.357+0000 7fd4ab200640 1 -- 172.21.15.27:0/3658430362 wait complete. 2024-03-28T19:14:50.377 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-03-28T19:14:50.377 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before thrashing 2024-03-28T19:14:55.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:55.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.612+0000 7f77da600640 1 Processor -- start 2024-03-28T19:14:55.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.612+0000 7f77da600640 1 -- start start 2024-03-28T19:14:55.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.614+0000 7f77da600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f77d40c7970 0x7f77d40c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:55.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.614+0000 7f77da600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d40c7310 con 0x7f77d4155730 2024-03-28T19:14:55.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.614+0000 7f77da600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d4158d40 con 0x7f77d4152f80 2024-03-28T19:14:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.614+0000 7f77da600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d4159020 con 0x7f77d40c7970 2024-03-28T19:14:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.614+0000 7f77d3400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f77d4152f80 0x7f77d40c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:47586/0 (socket says 172.21.15.27:47586) 2024-03-28T19:14:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.614+0000 7f77d3400640 1 -- 172.21.15.27:0/2028479071 learned_addr learned my addr 172.21.15.27:0/2028479071 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.615+0000 7f77d2a00640 1 --2- 172.21.15.27:0/2028479071 >> v2:172.21.15.155:3300/0 conn(0x7f77d40c7970 0x7f77d40c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.615+0000 7f77d2000640 1 -- 172.21.15.27:0/2028479071 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106258804 0 0) 0x7f77d4158d40 con 0x7f77d4152f80 2024-03-28T19:14:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.615+0000 7f77d2000640 1 -- 172.21.15.27:0/2028479071 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b8003520 con 0x7f77d4152f80 2024-03-28T19:14:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.615+0000 7f77d2000640 1 -- 172.21.15.27:0/2028479071 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643419703 0 0) 0x7f77d40c7310 con 0x7f77d4155730 2024-03-28T19:14:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.615+0000 7f77d2000640 1 -- 172.21.15.27:0/2028479071 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d4158d40 con 0x7f77d4155730 2024-03-28T19:14:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.615+0000 7f77d2a00640 1 -- 172.21.15.27:0/2028479071 >> v1:172.21.15.155:6789/0 conn(0x7f77d4152f80 legacy=0x7f77d40c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.615+0000 7f77d2a00640 1 -- 172.21.15.27:0/2028479071 >> v1:172.21.15.27:6789/0 conn(0x7f77d4155730 legacy=0x7f77d4155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.615+0000 7f77d2a00640 1 -- 172.21.15.27:0/2028479071 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d4159300 con 0x7f77d40c7970 2024-03-28T19:14:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.617+0000 7f77d2a00640 1 --2- 172.21.15.27:0/2028479071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d40c7970 0x7f77d40c7df0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f77c00097a0 tx=0x7f77c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cee06536a0c5707 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.618+0000 7f77d2000640 1 -- 172.21.15.27:0/2028479071 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c0035020 con 0x7f77d40c7970 2024-03-28T19:14:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.618+0000 7f77d2000640 1 -- 172.21.15.27:0/2028479071 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c0031830 con 0x7f77d40c7970 2024-03-28T19:14:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.618+0000 7f77d2000640 1 -- 172.21.15.27:0/2028479071 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c0031c00 con 0x7f77d40c7970 2024-03-28T19:14:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.618+0000 7f77da600640 1 -- 172.21.15.27:0/2028479071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d40c7970 msgr2=0x7f77d40c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.618+0000 7f77da600640 1 --2- 172.21.15.27:0/2028479071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d40c7970 0x7f77d40c7df0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f77c00097a0 tx=0x7f77c002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:14:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.619+0000 7f77da600640 1 -- 172.21.15.27:0/2028479071 shutdown_connections 2024-03-28T19:14:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.619+0000 7f77da600640 1 --2- 172.21.15.27:0/2028479071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d40c7970 0x7f77d40c7df0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.619+0000 7f77da600640 1 -- 172.21.15.27:0/2028479071 >> 172.21.15.27:0/2028479071 conn(0x7f77d406b6f0 msgr2=0x7f77d40caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.619+0000 7f77da600640 1 -- 172.21.15.27:0/2028479071 shutdown_connections 2024-03-28T19:14:55.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.619+0000 7f77da600640 1 -- 172.21.15.27:0/2028479071 wait complete. 2024-03-28T19:14:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.621+0000 7f77da600640 1 Processor -- start 2024-03-28T19:14:55.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.621+0000 7f77da600640 1 -- start start 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77da600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77d40c7970 0x7f77d41413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77da600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77d4152f80 0x7f77d40cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77da600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d4155730 0x7f77d4140710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77da600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40cbca0 con 0x7f77d40c7970 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77da600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d4159b00 con 0x7f77d4155730 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77da600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d4141930 con 0x7f77d4152f80 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77d3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77d40c7970 0x7f77d41413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77d3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d4155730 0x7f77d4140710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:55.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77d2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77d4152f80 0x7f77d40cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77d3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77d40c7970 0x7f77d41413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33662/0 (socket says 172.21.15.27:33662) 2024-03-28T19:14:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77d3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d4155730 0x7f77d4140710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:60230/0 (socket says 172.21.15.27:60230) 2024-03-28T19:14:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.622+0000 7f77d3400640 1 -- 172.21.15.27:0/1068643791 learned_addr learned my addr 172.21.15.27:0/1068643791 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.623+0000 7f77d3400640 1 -- 172.21.15.27:0/1068643791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77d4152f80 msgr2=0x7f77d40cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.623+0000 7f77d3400640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77d4152f80 0x7f77d40cb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:55.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.623+0000 7f77d3400640 1 -- 172.21.15.27:0/1068643791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d4155730 msgr2=0x7f77d4140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.623+0000 7f77d3400640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d4155730 0x7f77d4140710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.623+0000 7f77d3400640 1 -- 172.21.15.27:0/1068643791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d414fc60 con 0x7f77d40c7970 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.623+0000 7f77d3e00640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d4155730 0x7f77d4140710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.623+0000 7f77d3400640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77d40c7970 0x7f77d41413f0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f77c400c500 tx=0x7f77c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.624+0000 7f77cb400640 1 -- 172.21.15.27:0/1068643791 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4017070 con 0x7f77d40c7970 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.624+0000 7f77cb400640 1 -- 172.21.15.27:0/1068643791 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c4005d00 con 0x7f77d40c7970 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.624+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d4150200 con 0x7f77d40c7970 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.624+0000 7f77cb400640 1 -- 172.21.15.27:0/1068643791 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4007330 con 0x7f77d40c7970 2024-03-28T19:14:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.625+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d414d510 con 0x7f77d40c7970 2024-03-28T19:14:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.626+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d404f400 con 0x7f77d40c7970 2024-03-28T19:14:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.627+0000 7f77cb400640 1 -- 172.21.15.27:0/1068643791 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f77c401a070 con 0x7f77d40c7970 2024-03-28T19:14:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.629+0000 7f77cb400640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f77b406bbd0 0x7f77b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.629+0000 7f77cb400640 1 -- 172.21.15.27:0/1068643791 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f77c4092350 con 0x7f77d40c7970 2024-03-28T19:14:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.629+0000 7f77d2a00640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f77b406bbd0 0x7f77b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:55.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.630+0000 7f77d2a00640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f77b406bbd0 0x7f77b406e090 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f77c0036000 tx=0x7f77c003c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.635+0000 7f77cb400640 1 -- 172.21.15.27:0/1068643791 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f77c40580e0 con 0x7f77d40c7970 2024-03-28T19:14:55.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.997+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77d4155220 con 0x7f77d40c7970 2024-03-28T19:14:55.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:55.998+0000 7f77cb400640 1 -- 172.21.15.27:0/1068643791 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f77c405c180 con 0x7f77d40c7970 2024-03-28T19:14:55.998 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:55.998 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:14:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 153 2024-03-28T19:14:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f77b406bbd0 msgr2=0x7f77b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f77b406bbd0 0x7f77b406e090 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f77c0036000 tx=0x7f77c003c040 comp rx=0 tx=0).stop 2024-03-28T19:14:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77d40c7970 msgr2=0x7f77d41413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77d40c7970 0x7f77d41413f0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f77c400c500 tx=0x7f77c400c9c0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 shutdown_connections 2024-03-28T19:14:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f77b406bbd0 0x7f77b406e090 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77d4155730 0x7f77d4140710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77d4152f80 0x7f77d40cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 --2- 172.21.15.27:0/1068643791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77d40c7970 0x7f77d41413f0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.001+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 >> 172.21.15.27:0/1068643791 conn(0x7f77d406b6f0 msgr2=0x7f77d40c54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.002+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 shutdown_connections 2024-03-28T19:14:56.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.002+0000 7f77da600640 1 -- 172.21.15.27:0/1068643791 wait complete. 2024-03-28T19:14:56.023 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-03-28T19:14:56.024 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T19:14:56.024 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:14:56.047 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T19:14:56.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:56.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.282+0000 7f156ac00640 1 Processor -- start 2024-03-28T19:14:56.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.283+0000 7f156ac00640 1 -- start start 2024-03-28T19:14:56.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.284+0000 7f156ac00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1564070a50 0x7f15640733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:56.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.284+0000 7f156ac00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156406d7d0 con 0x7f15640a6300 2024-03-28T19:14:56.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.284+0000 7f156ac00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15640c2f00 con 0x7f156406ceb0 2024-03-28T19:14:56.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.284+0000 7f156ac00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15640c31e0 con 0x7f1564070a50 2024-03-28T19:14:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.284+0000 7f1563400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f15640a6300 0x7f1564155a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36826/0 (socket says 172.21.15.27:36826) 2024-03-28T19:14:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.284+0000 7f1563400640 1 -- 172.21.15.27:0/2408122529 learned_addr learned my addr 172.21.15.27:0/2408122529 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:14:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.284+0000 7f1568c00640 1 --2- 172.21.15.27:0/2408122529 >> v2:172.21.15.155:3300/0 conn(0x7f1564070a50 0x7f15640733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.285+0000 7f1562a00640 1 -- 172.21.15.27:0/2408122529 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019595877 0 0) 0x7f156406d7d0 con 0x7f15640a6300 2024-03-28T19:14:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.285+0000 7f1562a00640 1 -- 172.21.15.27:0/2408122529 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1548003520 con 0x7f15640a6300 2024-03-28T19:14:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.285+0000 7f1568c00640 1 -- 172.21.15.27:0/2408122529 >> v1:172.21.15.155:6789/0 conn(0x7f156406ceb0 legacy=0x7f15640c1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.285+0000 7f1568c00640 1 -- 172.21.15.27:0/2408122529 >> v1:172.21.15.27:6789/0 conn(0x7f15640a6300 legacy=0x7f1564155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.285+0000 7f1568c00640 1 -- 172.21.15.27:0/2408122529 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156406dab0 con 0x7f1564070a50 2024-03-28T19:14:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.287+0000 7f1568c00640 1 --2- 172.21.15.27:0/2408122529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 0x7f15640733b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f155c00aea0 tx=0x7f155c02cfc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5df6821a4dd69224 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.287+0000 7f1562a00640 1 -- 172.21.15.27:0/2408122529 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c02db60 con 0x7f1564070a50 2024-03-28T19:14:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.288+0000 7f1562a00640 1 -- 172.21.15.27:0/2408122529 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c02dd00 con 0x7f1564070a50 2024-03-28T19:14:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.288+0000 7f1562a00640 1 -- 172.21.15.27:0/2408122529 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c008640 con 0x7f1564070a50 2024-03-28T19:14:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.288+0000 7f156ac00640 1 -- 172.21.15.27:0/2408122529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 msgr2=0x7f15640733b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.288+0000 7f156ac00640 1 --2- 172.21.15.27:0/2408122529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 0x7f15640733b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f155c00aea0 tx=0x7f155c02cfc0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.289+0000 7f156ac00640 1 -- 172.21.15.27:0/2408122529 shutdown_connections 2024-03-28T19:14:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.289+0000 7f156ac00640 1 --2- 172.21.15.27:0/2408122529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 0x7f15640733b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.289+0000 7f156ac00640 1 -- 172.21.15.27:0/2408122529 >> 172.21.15.27:0/2408122529 conn(0x7f156406b9d0 msgr2=0x7f15640c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.289+0000 7f156ac00640 1 -- 172.21.15.27:0/2408122529 shutdown_connections 2024-03-28T19:14:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.289+0000 7f156ac00640 1 -- 172.21.15.27:0/2408122529 wait complete. 2024-03-28T19:14:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.292+0000 7f156ac00640 1 Processor -- start 2024-03-28T19:14:56.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.292+0000 7f156ac00640 1 -- start start 2024-03-28T19:14:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f156ac00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f156406ceb0 0x7f15641ec440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f156ac00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 0x7f15641da2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f156ac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15640a6300 0x7f15641e05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f156ac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640c45b0 con 0x7f15640a6300 2024-03-28T19:14:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f156ac00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640c4430 con 0x7f1564070a50 2024-03-28T19:14:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f156ac00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156406d290 con 0x7f156406ceb0 2024-03-28T19:14:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f1568c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15640a6300 0x7f15641e05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f1568c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15640a6300 0x7f15641e05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35342/0 (socket says 172.21.15.27:35342) 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f1568c00640 1 -- 172.21.15.27:0/259686594 learned_addr learned my addr 172.21.15.27:0/259686594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f1563e00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f156406ceb0 0x7f15641ec440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.294+0000 7f1563400640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 0x7f15641da2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.295+0000 7f1568c00640 1 -- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f156406ceb0 msgr2=0x7f15641ec440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.295+0000 7f1568c00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f156406ceb0 0x7f15641ec440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.295+0000 7f1568c00640 1 -- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 msgr2=0x7f15641da2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.295+0000 7f1568c00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 0x7f15641da2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.295+0000 7f1568c00640 1 -- 172.21.15.27:0/259686594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15640a8510 con 0x7f15640a6300 2024-03-28T19:14:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.295+0000 7f1563400640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 0x7f15641da2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:14:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.295+0000 7f1568c00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15640a6300 0x7f15641e05e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f155c00aea0 tx=0x7f155c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.296+0000 7f155be00640 1 -- 172.21.15.27:0/259686594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c042020 con 0x7f15640a6300 2024-03-28T19:14:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.296+0000 7f155be00640 1 -- 172.21.15.27:0/259686594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c004800 con 0x7f15640a6300 2024-03-28T19:14:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.296+0000 7f155be00640 1 -- 172.21.15.27:0/259686594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c00a560 con 0x7f15640a6300 2024-03-28T19:14:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.296+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15641dfb00 con 0x7f15640a6300 2024-03-28T19:14:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.296+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15641da7f0 con 0x7f15640a6300 2024-03-28T19:14:56.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.297+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1564073730 con 0x7f15640a6300 2024-03-28T19:14:56.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.299+0000 7f155be00640 1 -- 172.21.15.27:0/259686594 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f155c045020 con 0x7f15640a6300 2024-03-28T19:14:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.300+0000 7f155be00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f154406bbd0 0x7f154406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.301+0000 7f155be00640 1 -- 172.21.15.27:0/259686594 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f155c0adc60 con 0x7f15640a6300 2024-03-28T19:14:56.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.301+0000 7f1563e00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f154406bbd0 0x7f154406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:56.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.302+0000 7f1563e00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f154406bbd0 0x7f154406e090 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f15540016e0 tx=0x7f1554009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:56.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.307+0000 7f155be00640 1 -- 172.21.15.27:0/259686594 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f155c04c030 con 0x7f15640a6300 2024-03-28T19:14:56.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.650+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f156404f400 con 0x7f15640a6300 2024-03-28T19:14:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.651+0000 7f155be00640 1 -- 172.21.15.27:0/259686594 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f155c07baa0 con 0x7f15640a6300 2024-03-28T19:14:56.650 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:56.650 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:14:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 153 2024-03-28T19:14:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.654+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f154406bbd0 msgr2=0x7f154406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.654+0000 7f156ac00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f154406bbd0 0x7f154406e090 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f15540016e0 tx=0x7f1554009040 comp rx=0 tx=0).stop 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.654+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15640a6300 msgr2=0x7f15641e05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.654+0000 7f156ac00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15640a6300 0x7f15641e05e0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f155c00aea0 tx=0x7f155c004a40 comp rx=0 tx=0).stop 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.655+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 shutdown_connections 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.655+0000 7f156ac00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f154406bbd0 0x7f154406e090 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.655+0000 7f156ac00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15640a6300 0x7f15641e05e0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.655+0000 7f156ac00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564070a50 0x7f15641da2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.655+0000 7f156ac00640 1 --2- 172.21.15.27:0/259686594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f156406ceb0 0x7f15641ec440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.655+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 >> 172.21.15.27:0/259686594 conn(0x7f156406b9d0 msgr2=0x7f15641dce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.655+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 shutdown_connections 2024-03-28T19:14:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:56.655+0000 7f156ac00640 1 -- 172.21.15.27:0/259686594 wait complete. 2024-03-28T19:14:56.674 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:14:58.675 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:14:58.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.911+0000 7feea2600640 1 Processor -- start 2024-03-28T19:14:58.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.912+0000 7feea2600640 1 -- start start 2024-03-28T19:14:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.913+0000 7feea2600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c152ba0 0x7fee9c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.913+0000 7feea2600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fee9c156390 0x7fee9c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.913+0000 7feea2600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee9c1591d0 0x7fee9c0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.913+0000 7feea2600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c155f00 con 0x7fee9c1591d0 2024-03-28T19:14:58.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.913+0000 7feea2600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c0c6250 con 0x7fee9c156390 2024-03-28T19:14:58.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.913+0000 7feea2600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0c5f70 con 0x7fee9c152ba0 2024-03-28T19:14:58.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee9c1591d0 0x7fee9c0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:58.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fee9c1591d0 0x7fee9c0cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35356/0 (socket says 172.21.15.27:35356) 2024-03-28T19:14:58.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9be00640 1 -- 172.21.15.27:0/580412669 learned_addr learned my addr 172.21.15.27:0/580412669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:58.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9aa00640 1 --2- 172.21.15.27:0/580412669 >> v2:172.21.15.155:3300/0 conn(0x7fee9c156390 0x7fee9c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:58.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9b400640 1 --2- 172.21.15.27:0/580412669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c152ba0 0x7fee9c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9be00640 1 -- 172.21.15.27:0/580412669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c152ba0 msgr2=0x7fee9c0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9be00640 1 --2- 172.21.15.27:0/580412669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c152ba0 0x7fee9c0c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9be00640 1 -- 172.21.15.27:0/580412669 >> v2:172.21.15.155:3300/0 conn(0x7fee9c156390 msgr2=0x7fee9c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:14:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9be00640 1 --2- 172.21.15.27:0/580412669 >> v2:172.21.15.155:3300/0 conn(0x7fee9c156390 0x7fee9c158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.914+0000 7fee9be00640 1 -- 172.21.15.27:0/580412669 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c1558b0 con 0x7fee9c1591d0 2024-03-28T19:14:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.915+0000 7fee9aa00640 1 --2- 172.21.15.27:0/580412669 >> v2:172.21.15.155:3300/0 conn(0x7fee9c156390 0x7fee9c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:14:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.916+0000 7fee9be00640 1 --2- 172.21.15.27:0/580412669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 0x7fee9c0cb1a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0c8a40 tx=0x7fee9402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ba498d7d7e6954 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.917+0000 7fee9a000640 1 -- 172.21.15.27:0/580412669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9402c950 con 0x7fee9c1591d0 2024-03-28T19:14:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.917+0000 7fee9a000640 1 -- 172.21.15.27:0/580412669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9402caf0 con 0x7fee9c1591d0 2024-03-28T19:14:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.917+0000 7fee9a000640 1 -- 172.21.15.27:0/580412669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee940343f0 con 0x7fee9c1591d0 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 -- 172.21.15.27:0/580412669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 msgr2=0x7fee9c0cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 --2- 172.21.15.27:0/580412669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 0x7fee9c0cb1a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0c8a40 tx=0x7fee9402bdb0 comp rx=0 tx=0).stop 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 -- 172.21.15.27:0/580412669 shutdown_connections 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 --2- 172.21.15.27:0/580412669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 0x7fee9c0cb1a0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 --2- 172.21.15.27:0/580412669 >> v2:172.21.15.155:3300/0 conn(0x7fee9c156390 0x7fee9c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 --2- 172.21.15.27:0/580412669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c152ba0 0x7fee9c0c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 -- 172.21.15.27:0/580412669 >> 172.21.15.27:0/580412669 conn(0x7fee9c066fb0 msgr2=0x7fee9c0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 -- 172.21.15.27:0/580412669 shutdown_connections 2024-03-28T19:14:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.918+0000 7feea2600640 1 -- 172.21.15.27:0/580412669 wait complete. 2024-03-28T19:14:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.920+0000 7feea2600640 1 Processor -- start 2024-03-28T19:14:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.921+0000 7feea2600640 1 -- start start 2024-03-28T19:14:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.922+0000 7feea2600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c152ba0 0x7fee9c153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.922+0000 7feea2600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c156390 0x7fee9c0abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.922+0000 7feea2600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 0x7fee9c0c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.922+0000 7feea2600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0c9780 con 0x7fee9c1591d0 2024-03-28T19:14:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.922+0000 7feea2600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c06d700 con 0x7fee9c152ba0 2024-03-28T19:14:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.922+0000 7feea2600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c0c9900 con 0x7fee9c156390 2024-03-28T19:14:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.923+0000 7fee9be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 0x7fee9c0c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.923+0000 7fee9be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 0x7fee9c0c3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35358/0 (socket says 172.21.15.27:35358) 2024-03-28T19:14:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.923+0000 7fee9be00640 1 -- 172.21.15.27:0/2646112701 learned_addr learned my addr 172.21.15.27:0/2646112701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:14:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.923+0000 7fee9aa00640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c156390 0x7fee9c0abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:58.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.923+0000 7fee9b400640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c152ba0 0x7fee9c153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.924+0000 7fee9aa00640 1 -- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c152ba0 msgr2=0x7fee9c153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.924+0000 7fee9aa00640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c152ba0 0x7fee9c153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.924+0000 7fee9aa00640 1 -- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 msgr2=0x7fee9c0c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.924+0000 7fee9aa00640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 0x7fee9c0c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:58.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.924+0000 7fee9aa00640 1 -- 172.21.15.27:0/2646112701 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c156c80 con 0x7fee9c156390 2024-03-28T19:14:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.924+0000 7fee9aa00640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c156390 0x7fee9c0abb80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fee88009e10 tx=0x7fee8800ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.925+0000 7fee93400640 1 -- 172.21.15.27:0/2646112701 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88024070 con 0x7fee9c156390 2024-03-28T19:14:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.925+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c154460 con 0x7fee9c156390 2024-03-28T19:14:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.925+0000 7fee93400640 1 -- 172.21.15.27:0/2646112701 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee88017bf0 con 0x7fee9c156390 2024-03-28T19:14:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.925+0000 7fee93400640 1 -- 172.21.15.27:0/2646112701 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8800f040 con 0x7fee9c156390 2024-03-28T19:14:58.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.925+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c154b10 con 0x7fee9c156390 2024-03-28T19:14:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.927+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9c04f400 con 0x7fee9c156390 2024-03-28T19:14:58.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.928+0000 7fee93400640 1 -- 172.21.15.27:0/2646112701 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fee88027070 con 0x7fee9c156390 2024-03-28T19:14:58.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.930+0000 7fee93400640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee7c06bbd0 0x7fee7c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:14:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.930+0000 7fee93400640 1 -- 172.21.15.27:0/2646112701 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7fee88093510 con 0x7fee9c156390 2024-03-28T19:14:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.930+0000 7fee9b400640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee7c06bbd0 0x7fee7c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:14:58.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.931+0000 7fee9b400640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee7c06bbd0 0x7fee7c06e090 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fee8c009a20 tx=0x7fee8c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:14:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:58.948+0000 7fee93400640 1 -- 172.21.15.27:0/2646112701 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fee88060c20 con 0x7fee9c156390 2024-03-28T19:14:59.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.243+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee9c06d4e0 con 0x7fee9c156390 2024-03-28T19:14:59.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.244+0000 7fee93400640 1 -- 172.21.15.27:0/2646112701 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fee88064cc0 con 0x7fee9c156390 2024-03-28T19:14:59.244 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:14:59.244 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:14:59.244 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 153 2024-03-28T19:14:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee7c06bbd0 msgr2=0x7fee7c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee7c06bbd0 0x7fee7c06e090 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fee8c009a20 tx=0x7fee8c009190 comp rx=0 tx=0).stop 2024-03-28T19:14:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c156390 msgr2=0x7fee9c0abb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:14:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c156390 0x7fee9c0abb80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fee88009e10 tx=0x7fee8800ebb0 comp rx=0 tx=0).stop 2024-03-28T19:14:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 shutdown_connections 2024-03-28T19:14:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fee7c06bbd0 0x7fee7c06e090 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee9c1591d0 0x7fee9c0c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:59.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee9c156390 0x7fee9c0abb80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:59.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 --2- 172.21.15.27:0/2646112701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c152ba0 0x7fee9c153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:14:59.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 >> 172.21.15.27:0/2646112701 conn(0x7fee9c066fb0 msgr2=0x7fee9c14fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:14:59.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.247+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 shutdown_connections 2024-03-28T19:14:59.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:14:59.248+0000 7feea2600640 1 -- 172.21.15.27:0/2646112701 wait complete. 2024-03-28T19:14:59.268 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:15:01.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:01.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.500+0000 7fb1f3e00640 1 Processor -- start 2024-03-28T19:15:01.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.500+0000 7fb1f3e00640 1 -- start start 2024-03-28T19:15:01.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.502+0000 7fb1f3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1ec071cf0 0x7fb1ec159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:01.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.502+0000 7fb1f3e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ec072870 con 0x7fb1ec14f6d0 2024-03-28T19:15:01.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.502+0000 7fb1f3e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ec156640 con 0x7fb1ec072c60 2024-03-28T19:15:01.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.502+0000 7fb1f3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec158db0 con 0x7fb1ec071cf0 2024-03-28T19:15:01.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.502+0000 7fb1f1400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb1ec14f6d0 0x7fb1ec0c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36842/0 (socket says 172.21.15.27:36842) 2024-03-28T19:15:01.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.502+0000 7fb1f1400640 1 -- 172.21.15.27:0/3033404067 learned_addr learned my addr 172.21.15.27:0/3033404067 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.503+0000 7fb1f0a00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1ec071cf0 0x7fb1ec159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:01.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.503+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149724774 0 0) 0x7fb1e0001110 con 0x7fb1ec14f6d0 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.503+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d4003520 con 0x7fb1ec14f6d0 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.503+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 634312626 0 0) 0x7fb1d4003520 con 0x7fb1ec14f6d0 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.503+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1ec071cf0 msgr2=0x7fb1ec159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.503+0000 7fb1ebe00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1ec071cf0 0x7fb1ec159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 >> v1:172.21.15.155:6789/0 conn(0x7fb1ec072c60 legacy=0x7fb1ec0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec156360 con 0x7fb1ec14f6d0 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fb1e0001840 con 0x7fb1ec14f6d0 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 >> v1:172.21.15.27:6789/0 conn(0x7fb1ec14f6d0 legacy=0x7fb1ec0c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1d40039e0 0x7fb1d4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:01.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1d4004800 0x7fb1d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1ec071cf0 0x7fb1d4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec158db0 con 0x7fb1d4004800 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d400bb20 con 0x7fb1d40039e0 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d400bd20 con 0x7fb1ec071cf0 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1f0a00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1d4004800 0x7fb1d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1f1400640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1d40039e0 0x7fb1d4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.504+0000 7fb1f1e00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1ec071cf0 0x7fb1d4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.505+0000 7fb1f0a00640 1 -- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1ec071cf0 msgr2=0x7fb1d4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:01.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.505+0000 7fb1f0a00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1ec071cf0 0x7fb1d4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.505+0000 7fb1f0a00640 1 -- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1d40039e0 msgr2=0x7fb1d4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:01.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.505+0000 7fb1f0a00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1d40039e0 0x7fb1d4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.505+0000 7fb1f0a00640 1 -- 172.21.15.27:0/3033404067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4003520 con 0x7fb1d4004800 2024-03-28T19:15:01.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.506+0000 7fb1f0a00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1d4004800 0x7fb1d4004c80 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009a90 tx=0x7fb1d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b42dd71b8a99fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:01.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.506+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8039070 con 0x7fb1d4004800 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.506+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d802c680 con 0x7fb1d4004800 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.506+0000 7fb1ebe00640 1 -- 172.21.15.27:0/3033404067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8009690 con 0x7fb1d4004800 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.507+0000 7fb1f3e00640 1 -- 172.21.15.27:0/3033404067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1d4004800 msgr2=0x7fb1d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.507+0000 7fb1f3e00640 1 --2- 172.21.15.27:0/3033404067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1d4004800 0x7fb1d4004c80 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009a90 tx=0x7fb1d4005830 comp rx=0 tx=0).stop 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.507+0000 7fb1f1e00640 1 -- 172.21.15.27:0/3033404067 reap_dead start 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.507+0000 7fb1f3e00640 1 -- 172.21.15.27:0/3033404067 shutdown_connections 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.507+0000 7fb1f3e00640 1 -- 172.21.15.27:0/3033404067 >> 172.21.15.27:0/3033404067 conn(0x7fb1ec06c5f0 msgr2=0x7fb1ec06ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.507+0000 7fb1f3e00640 1 -- 172.21.15.27:0/3033404067 shutdown_connections 2024-03-28T19:15:01.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.507+0000 7fb1f3e00640 1 -- 172.21.15.27:0/3033404067 wait complete. 2024-03-28T19:15:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.509+0000 7fb1f3e00640 1 Processor -- start 2024-03-28T19:15:01.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.510+0000 7fb1f3e00640 1 -- start start 2024-03-28T19:15:01.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.510+0000 7fb1f3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1d4004800 0x7fb1ec14e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.510+0000 7fb1f3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec14a0f0 0x7fb1ec1483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1ec152e70 0x7fb1ec148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f3e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec14fab0 con 0x7fb1ec152e70 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f3e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec159330 con 0x7fb1ec14a0f0 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec0c3c70 con 0x7fb1d4004800 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1ec152e70 0x7fb1ec148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1ec152e70 0x7fb1ec148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35374/0 (socket says 172.21.15.27:35374) 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1e00640 1 -- 172.21.15.27:0/2895702560 learned_addr learned my addr 172.21.15.27:0/2895702560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1400640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1d4004800 0x7fb1ec14e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:01.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f0a00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec14a0f0 0x7fb1ec1483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1e00640 1 -- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1d4004800 msgr2=0x7fb1ec14e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1d4004800 0x7fb1ec14e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1e00640 1 -- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec14a0f0 msgr2=0x7fb1ec1483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec14a0f0 0x7fb1ec1483f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1e00640 1 -- 172.21.15.27:0/2895702560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec1ee450 con 0x7fb1ec152e70 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.511+0000 7fb1f1400640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1d4004800 0x7fb1ec14e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.512+0000 7fb1f1e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1ec152e70 0x7fb1ec148930 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0c28d0 tx=0x7fb1e400e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.512+0000 7fb1e9600640 1 -- 172.21.15.27:0/2895702560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4003b80 con 0x7fb1ec152e70 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.512+0000 7fb1e9600640 1 -- 172.21.15.27:0/2895702560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e4018070 con 0x7fb1ec152e70 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.512+0000 7fb1e9600640 1 -- 172.21.15.27:0/2895702560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e40050d0 con 0x7fb1ec152e70 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.512+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ec1ee730 con 0x7fb1ec152e70 2024-03-28T19:15:01.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.512+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ec1eecf0 con 0x7fb1ec152e70 2024-03-28T19:15:01.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.514+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ec04f400 con 0x7fb1ec152e70 2024-03-28T19:15:01.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.515+0000 7fb1e9600640 1 -- 172.21.15.27:0/2895702560 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb1e4005270 con 0x7fb1ec152e70 2024-03-28T19:15:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.517+0000 7fb1e9600640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb1cc06bc50 0x7fb1cc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.517+0000 7fb1e9600640 1 -- 172.21.15.27:0/2895702560 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7fb1e408b310 con 0x7fb1ec152e70 2024-03-28T19:15:01.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.517+0000 7fb1f1400640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb1cc06bc50 0x7fb1cc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:01.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.518+0000 7fb1f1400640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb1cc06bc50 0x7fb1cc06e110 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00012b0 tx=0x7fb1e0000ed0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:01.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.523+0000 7fb1e9600640 1 -- 172.21.15.27:0/2895702560 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb1e40591d0 con 0x7fb1ec152e70 2024-03-28T19:15:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.838+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1ec0c2330 con 0x7fb1ec152e70 2024-03-28T19:15:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.839+0000 7fb1e9600640 1 -- 172.21.15.27:0/2895702560 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fb1e4058ad0 con 0x7fb1ec152e70 2024-03-28T19:15:01.839 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:01.839 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:15:01.839 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 153 2024-03-28T19:15:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb1cc06bc50 msgr2=0x7fb1cc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb1cc06bc50 0x7fb1cc06e110 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00012b0 tx=0x7fb1e0000ed0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1ec152e70 msgr2=0x7fb1ec148930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1ec152e70 0x7fb1ec148930 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0c28d0 tx=0x7fb1e400e9e0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 shutdown_connections 2024-03-28T19:15:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb1cc06bc50 0x7fb1cc06e110 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1ec152e70 0x7fb1ec148930 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec14a0f0 0x7fb1ec1483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 --2- 172.21.15.27:0/2895702560 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1d4004800 0x7fb1ec14e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:01.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 >> 172.21.15.27:0/2895702560 conn(0x7fb1ec06c5f0 msgr2=0x7fb1ec06ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:01.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 shutdown_connections 2024-03-28T19:15:01.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:01.842+0000 7fb1f3e00640 1 -- 172.21.15.27:0/2895702560 wait complete. 2024-03-28T19:15:01.863 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:15:03.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:15:03.405 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:15:03.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:04.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.103+0000 7f1552200640 1 Processor -- start 2024-03-28T19:15:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.103+0000 7f1552200640 1 -- start start 2024-03-28T19:15:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.105+0000 7f1552200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f154c0a9cb0 0x7f154c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.105+0000 7f1552200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f154c1534c0 0x7f154c153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.105+0000 7f1552200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c0c3d90 con 0x7f154c150c70 2024-03-28T19:15:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.105+0000 7f1552200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c0ab4e0 con 0x7f154c0a9cb0 2024-03-28T19:15:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.105+0000 7f1552200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c0ac1f0 con 0x7f154c1534c0 2024-03-28T19:15:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.105+0000 7f154b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f154c150c70 0x7f154c0c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36844/0 (socket says 172.21.15.27:36844) 2024-03-28T19:15:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.105+0000 7f154b400640 1 -- 172.21.15.27:0/3041239418 learned_addr learned my addr 172.21.15.27:0/3041239418 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.105+0000 7f154aa00640 1 --2- 172.21.15.27:0/3041239418 >> v2:172.21.15.27:3300/0 conn(0x7f154c0a9cb0 0x7f154c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.106+0000 7f154be00640 1 --2- 172.21.15.27:0/3041239418 >> v2:172.21.15.155:3300/0 conn(0x7f154c1534c0 0x7f154c153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.106+0000 7f154aa00640 1 -- 172.21.15.27:0/3041239418 >> v2:172.21.15.155:3300/0 conn(0x7f154c1534c0 msgr2=0x7f154c153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.106+0000 7f154aa00640 1 --2- 172.21.15.27:0/3041239418 >> v2:172.21.15.155:3300/0 conn(0x7f154c1534c0 0x7f154c153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.106+0000 7f154aa00640 1 -- 172.21.15.27:0/3041239418 >> v1:172.21.15.27:6789/0 conn(0x7f154c150c70 legacy=0x7f154c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.106+0000 7f154aa00640 1 -- 172.21.15.27:0/3041239418 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0ab710 con 0x7f154c0a9cb0 2024-03-28T19:15:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.106+0000 7f154a000640 1 -- 172.21.15.27:0/3041239418 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437785441 0 0) 0x7f154c0c3d90 con 0x7f154c150c70 2024-03-28T19:15:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.108+0000 7f154aa00640 1 --2- 172.21.15.27:0/3041239418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0a9cb0 0x7f154c0c3690 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f154c0c4610 tx=0x7f153802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fc015b518d4dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.108+0000 7f154a000640 1 -- 172.21.15.27:0/3041239418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15380315a0 con 0x7f154c0a9cb0 2024-03-28T19:15:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.108+0000 7f154a000640 1 -- 172.21.15.27:0/3041239418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1538037070 con 0x7f154c0a9cb0 2024-03-28T19:15:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.108+0000 7f154a000640 1 -- 172.21.15.27:0/3041239418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1538031960 con 0x7f154c0a9cb0 2024-03-28T19:15:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.109+0000 7f1552200640 1 -- 172.21.15.27:0/3041239418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0a9cb0 msgr2=0x7f154c0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.109+0000 7f1552200640 1 --2- 172.21.15.27:0/3041239418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0a9cb0 0x7f154c0c3690 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f154c0c4610 tx=0x7f153802b840 comp rx=0 tx=0).stop 2024-03-28T19:15:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.109+0000 7f1552200640 1 -- 172.21.15.27:0/3041239418 shutdown_connections 2024-03-28T19:15:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.109+0000 7f1552200640 1 --2- 172.21.15.27:0/3041239418 >> v2:172.21.15.155:3300/0 conn(0x7f154c1534c0 0x7f154c153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.109+0000 7f1552200640 1 --2- 172.21.15.27:0/3041239418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c0a9cb0 0x7f154c0c3690 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.109+0000 7f1552200640 1 -- 172.21.15.27:0/3041239418 >> 172.21.15.27:0/3041239418 conn(0x7f154c06b9d0 msgr2=0x7f154c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.109+0000 7f1552200640 1 -- 172.21.15.27:0/3041239418 shutdown_connections 2024-03-28T19:15:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.109+0000 7f1552200640 1 -- 172.21.15.27:0/3041239418 wait complete. 2024-03-28T19:15:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.111+0000 7f1552200640 1 Processor -- start 2024-03-28T19:15:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.111+0000 7f1552200640 1 -- start start 2024-03-28T19:15:04.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f1552200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f154c0a9cb0 0x7f154c1eb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f1552200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c150c70 0x7f154c0c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f1552200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f154c1534c0 0x7f154c0a7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f1552200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0c1810 con 0x7f154c150c70 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f1552200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0abcd0 con 0x7f154c1534c0 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f1552200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0c6830 con 0x7f154c0a9cb0 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f154aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c150c70 0x7f154c0c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f154aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c150c70 0x7f154c0c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35388/0 (socket says 172.21.15.27:35388) 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f154aa00640 1 -- 172.21.15.27:0/1715656238 learned_addr learned my addr 172.21.15.27:0/1715656238 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f154b400640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f154c0a9cb0 0x7f154c1eb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.113+0000 7f154be00640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f154c1534c0 0x7f154c0a7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.114+0000 7f154b400640 1 -- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f154c1534c0 msgr2=0x7f154c0a7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.114+0000 7f154b400640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f154c1534c0 0x7f154c0a7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.114+0000 7f154b400640 1 -- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c150c70 msgr2=0x7f154c0c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.114+0000 7f154b400640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c150c70 0x7f154c0c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.114+0000 7f154b400640 1 -- 172.21.15.27:0/1715656238 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c14e0c0 con 0x7f154c0a9cb0 2024-03-28T19:15:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.115+0000 7f154b400640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f154c0a9cb0 0x7f154c1eb7e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c450 tx=0x7f153c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.115+0000 7f1543400640 1 -- 172.21.15.27:0/1715656238 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c018070 con 0x7f154c0a9cb0 2024-03-28T19:15:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.115+0000 7f1543400640 1 -- 172.21.15.27:0/1715656238 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c0074e0 con 0x7f154c0a9cb0 2024-03-28T19:15:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.115+0000 7f1543400640 1 -- 172.21.15.27:0/1715656238 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c0049e0 con 0x7f154c0a9cb0 2024-03-28T19:15:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.115+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c1ebd20 con 0x7f154c0a9cb0 2024-03-28T19:15:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.116+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c14e610 con 0x7f154c0a9cb0 2024-03-28T19:15:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.117+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c04f400 con 0x7f154c0a9cb0 2024-03-28T19:15:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.119+0000 7f1543400640 1 -- 172.21.15.27:0/1715656238 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f153c004b80 con 0x7f154c0a9cb0 2024-03-28T19:15:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.120+0000 7f1543400640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f152c06bbd0 0x7f152c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.120+0000 7f1543400640 1 -- 172.21.15.27:0/1715656238 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f153c08ad50 con 0x7f154c0a9cb0 2024-03-28T19:15:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.120+0000 7f154aa00640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f152c06bbd0 0x7f152c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.121+0000 7f154aa00640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f152c06bbd0 0x7f152c06e090 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f153802b880 tx=0x7f153803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:04.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.137+0000 7f1543400640 1 -- 172.21.15.27:0/1715656238 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f154c1ebd20 con 0x7f154c0a9cb0 2024-03-28T19:15:04.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.451+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f154c0c5f00 con 0x7f154c0a9cb0 2024-03-28T19:15:04.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.454+0000 7f1543400640 1 -- 172.21.15.27:0/1715656238 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f153c058650 con 0x7f154c0a9cb0 2024-03-28T19:15:04.453 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:04.454 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:15:04.454 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 153 2024-03-28T19:15:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f152c06bbd0 msgr2=0x7f152c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f152c06bbd0 0x7f152c06e090 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f153802b880 tx=0x7f153803d040 comp rx=0 tx=0).stop 2024-03-28T19:15:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f154c0a9cb0 msgr2=0x7f154c1eb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f154c0a9cb0 0x7f154c1eb7e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c450 tx=0x7f153c00c910 comp rx=0 tx=0).stop 2024-03-28T19:15:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 shutdown_connections 2024-03-28T19:15:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f152c06bbd0 0x7f152c06e090 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f154c1534c0 0x7f154c0a7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f154c150c70 0x7f154c0c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 --2- 172.21.15.27:0/1715656238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f154c0a9cb0 0x7f154c1eb7e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 >> 172.21.15.27:0/1715656238 conn(0x7f154c06b9d0 msgr2=0x7f154c1501e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.457+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 shutdown_connections 2024-03-28T19:15:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:04.458+0000 7f1552200640 1 -- 172.21.15.27:0/1715656238 wait complete. 2024-03-28T19:15:04.481 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:15:06.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:06.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.725+0000 7f7b98a00640 1 Processor -- start 2024-03-28T19:15:06.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.725+0000 7f7b98a00640 1 -- start start 2024-03-28T19:15:06.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.727+0000 7f7b98a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b940c69a0 0x7f7b940c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.728+0000 7f7b98a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b94150ec0 con 0x7f7b94153a30 2024-03-28T19:15:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.728+0000 7f7b98a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9406dc70 con 0x7f7b94150ae0 2024-03-28T19:15:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.728+0000 7f7b98a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b94153550 con 0x7f7b940c69a0 2024-03-28T19:15:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.728+0000 7f7b93e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7b94153a30 0x7f7b9415dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59178/0 (socket says 172.21.15.27:59178) 2024-03-28T19:15:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.728+0000 7f7b93e00640 1 -- 172.21.15.27:0/1500865420 learned_addr learned my addr 172.21.15.27:0/1500865420 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.728+0000 7f7b92a00640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b940c69a0 0x7f7b940c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.728+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284889883 0 0) 0x7f7b8c001490 con 0x7f7b94153a30 2024-03-28T19:15:06.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b78003520 con 0x7f7b94153a30 2024-03-28T19:15:06.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1911021910 0 0) 0x7f7b78003520 con 0x7f7b94153a30 2024-03-28T19:15:06.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b940c69a0 msgr2=0x7f7b940c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:06.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b940c69a0 0x7f7b940c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:06.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 >> v1:172.21.15.155:6789/0 conn(0x7f7b94150ae0 legacy=0x7f7b940a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:06.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9406df10 con 0x7f7b94153a30 2024-03-28T19:15:06.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f7b8c001bc0 con 0x7f7b94153a30 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 >> v1:172.21.15.27:6789/0 conn(0x7f7b94153a30 legacy=0x7f7b9415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b940c69a0 0x7f7b780039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b78003f20 0x7f7b78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78004d40 0x7f7b780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b94153550 con 0x7f7b78003f20 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7800bb00 con 0x7f7b78004d40 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.729+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7800bd00 con 0x7f7b940c69a0 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.730+0000 7f7b92a00640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b78003f20 0x7f7b78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.730+0000 7f7b93400640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b940c69a0 0x7f7b780039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.730+0000 7f7b93e00640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78004d40 0x7f7b780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.730+0000 7f7b92a00640 1 -- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b940c69a0 msgr2=0x7f7b780039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:06.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.730+0000 7f7b92a00640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b940c69a0 0x7f7b780039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:06.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.730+0000 7f7b92a00640 1 -- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78004d40 msgr2=0x7f7b780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:06.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.730+0000 7f7b92a00640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78004d40 0x7f7b780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:06.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.730+0000 7f7b92a00640 1 -- 172.21.15.27:0/1500865420 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b78003520 con 0x7f7b78003f20 2024-03-28T19:15:06.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.731+0000 7f7b92a00640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b78003f20 0x7f7b78004800 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009ac0 tx=0x7f7b78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56d8c85e095a5f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:06.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.732+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80039070 con 0x7f7b78003f20 2024-03-28T19:15:06.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.732+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8002c6b0 con 0x7f7b78003f20 2024-03-28T19:15:06.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.732+0000 7f7b92000640 1 -- 172.21.15.27:0/1500865420 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80009690 con 0x7f7b78003f20 2024-03-28T19:15:06.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.733+0000 7f7b98a00640 1 -- 172.21.15.27:0/1500865420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b78003f20 msgr2=0x7f7b78004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:06.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.733+0000 7f7b98a00640 1 --2- 172.21.15.27:0/1500865420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b78003f20 0x7f7b78004800 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009ac0 tx=0x7f7b78005830 comp rx=0 tx=0).stop 2024-03-28T19:15:06.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.733+0000 7f7b93e00640 1 -- 172.21.15.27:0/1500865420 reap_dead start 2024-03-28T19:15:06.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.733+0000 7f7b98a00640 1 -- 172.21.15.27:0/1500865420 shutdown_connections 2024-03-28T19:15:06.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.733+0000 7f7b98a00640 1 -- 172.21.15.27:0/1500865420 >> 172.21.15.27:0/1500865420 conn(0x7f7b9406b0c0 msgr2=0x7f7b9406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:06.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.733+0000 7f7b98a00640 1 -- 172.21.15.27:0/1500865420 shutdown_connections 2024-03-28T19:15:06.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.733+0000 7f7b98a00640 1 -- 172.21.15.27:0/1500865420 wait complete. 2024-03-28T19:15:06.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.735+0000 7f7b98a00640 1 Processor -- start 2024-03-28T19:15:06.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.735+0000 7f7b98a00640 1 -- start start 2024-03-28T19:15:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.736+0000 7f7b98a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b78003f20 0x7f7b941ec3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.736+0000 7f7b98a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b94155b40 0x7f7b94071890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.736+0000 7f7b98a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b94070b20 0x7f7b940a8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.736+0000 7f7b98a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9406e7b0 con 0x7f7b94155b40 2024-03-28T19:15:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.736+0000 7f7b98a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9406e4d0 con 0x7f7b94070b20 2024-03-28T19:15:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.736+0000 7f7b98a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940a6370 con 0x7f7b78003f20 2024-03-28T19:15:06.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.737+0000 7f7b92a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b94155b40 0x7f7b94071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.737+0000 7f7b92a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b94155b40 0x7f7b94071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56648/0 (socket says 172.21.15.27:56648) 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.737+0000 7f7b92a00640 1 -- 172.21.15.27:0/2112410744 learned_addr learned my addr 172.21.15.27:0/2112410744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.737+0000 7f7b93400640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b78003f20 0x7f7b941ec3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.737+0000 7f7b93e00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b94070b20 0x7f7b940a8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.738+0000 7f7b92a00640 1 -- 172.21.15.27:0/2112410744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b78003f20 msgr2=0x7f7b941ec3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.738+0000 7f7b92a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b78003f20 0x7f7b941ec3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.738+0000 7f7b92a00640 1 -- 172.21.15.27:0/2112410744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b94070b20 msgr2=0x7f7b940a8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.738+0000 7f7b92a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b94070b20 0x7f7b940a8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.738+0000 7f7b92a00640 1 -- 172.21.15.27:0/2112410744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940a7370 con 0x7f7b94155b40 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.738+0000 7f7b92a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b94155b40 0x7f7b94071890 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f7b80037040 tx=0x7f7b80004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.739+0000 7f7b8b400640 1 -- 172.21.15.27:0/2112410744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80042020 con 0x7f7b94155b40 2024-03-28T19:15:06.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.739+0000 7f7b8b400640 1 -- 172.21.15.27:0/2112410744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8003a040 con 0x7f7b94155b40 2024-03-28T19:15:06.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.739+0000 7f7b8b400640 1 -- 172.21.15.27:0/2112410744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80039070 con 0x7f7b94155b40 2024-03-28T19:15:06.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.739+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b941eccb0 con 0x7f7b94155b40 2024-03-28T19:15:06.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.739+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b94155060 con 0x7f7b94155b40 2024-03-28T19:15:06.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.740+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b94067dd0 con 0x7f7b94155b40 2024-03-28T19:15:06.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.742+0000 7f7b8b400640 1 -- 172.21.15.27:0/2112410744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7b80045020 con 0x7f7b94155b40 2024-03-28T19:15:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.743+0000 7f7b8b400640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b7406bbd0 0x7f7b7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.743+0000 7f7b8b400640 1 -- 172.21.15.27:0/2112410744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f7b800acbd0 con 0x7f7b94155b40 2024-03-28T19:15:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.744+0000 7f7b93400640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b7406bbd0 0x7f7b7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:06.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.745+0000 7f7b93400640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b7406bbd0 0x7f7b7406e090 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7b84007ac0 tx=0x7f7b84007a50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:06.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:06.749+0000 7f7b8b400640 1 -- 172.21.15.27:0/2112410744 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7b8004c070 con 0x7f7b94155b40 2024-03-28T19:15:07.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.086+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b940a6130 con 0x7f7b94155b40 2024-03-28T19:15:07.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.088+0000 7f7b8b400640 1 -- 172.21.15.27:0/2112410744 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f7b8007a360 con 0x7f7b94155b40 2024-03-28T19:15:07.087 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:07.087 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:15:07.087 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 153 2024-03-28T19:15:07.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.090+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b7406bbd0 msgr2=0x7f7b7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:07.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.090+0000 7f7b98a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b7406bbd0 0x7f7b7406e090 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7b84007ac0 tx=0x7f7b84007a50 comp rx=0 tx=0).stop 2024-03-28T19:15:07.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.090+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b94155b40 msgr2=0x7f7b94071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:07.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.090+0000 7f7b98a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b94155b40 0x7f7b94071890 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f7b80037040 tx=0x7f7b80004730 comp rx=0 tx=0).stop 2024-03-28T19:15:07.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.091+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 shutdown_connections 2024-03-28T19:15:07.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.091+0000 7f7b98a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b7406bbd0 0x7f7b7406e090 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:07.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.091+0000 7f7b98a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b94070b20 0x7f7b940a8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:07.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.091+0000 7f7b98a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b94155b40 0x7f7b94071890 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:07.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.091+0000 7f7b98a00640 1 --2- 172.21.15.27:0/2112410744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b78003f20 0x7f7b941ec3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:07.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.091+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 >> 172.21.15.27:0/2112410744 conn(0x7f7b9406b0c0 msgr2=0x7f7b940c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:07.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.091+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 shutdown_connections 2024-03-28T19:15:07.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:07.091+0000 7f7b98a00640 1 -- 172.21.15.27:0/2112410744 wait complete. 2024-03-28T19:15:07.116 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:15:09.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:09.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.352+0000 7f7481a00640 1 Processor -- start 2024-03-28T19:15:09.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.352+0000 7f7481a00640 1 -- start start 2024-03-28T19:15:09.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.353+0000 7f7481a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 0x7f747c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.353+0000 7f7481a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c158bd0 con 0x7f747c0c8ed0 2024-03-28T19:15:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.353+0000 7f7481a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c158eb0 con 0x7f747c0c58c0 2024-03-28T19:15:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.354+0000 7f7481a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c159190 con 0x7f747c0be5d0 2024-03-28T19:15:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.354+0000 7f747b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f747c0c8ed0 0x7f747c155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59184/0 (socket says 172.21.15.27:59184) 2024-03-28T19:15:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.354+0000 7f747b400640 1 -- 172.21.15.27:0/2390264700 learned_addr learned my addr 172.21.15.27:0/2390264700 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.354+0000 7f7481000640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 0x7f747c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:09.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.354+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043593923 0 0) 0x7f747c158bd0 con 0x7f747c0c8ed0 2024-03-28T19:15:09.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.354+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460003520 con 0x7f747c0c8ed0 2024-03-28T19:15:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 764809457 0 0) 0x7f7460003520 con 0x7f747c0c8ed0 2024-03-28T19:15:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 msgr2=0x7f747c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:09.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 0x7f747c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 >> v1:172.21.15.155:6789/0 conn(0x7f747c0c58c0 legacy=0x7f747c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c159470 con 0x7f747c0c8ed0 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f7468001670 con 0x7f747c0c8ed0 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 >> v1:172.21.15.27:6789/0 conn(0x7f747c0c8ed0 legacy=0x7f747c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f7460003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7460004800 0x7f7460004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 0x7f7460005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c159190 con 0x7f74600039e0 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746000bb50 con 0x7f7460004800 2024-03-28T19:15:09.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746000bd50 con 0x7f747c0be5d0 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.355+0000 7f747be00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f7460003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.356+0000 7f747b400640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7460004800 0x7f7460004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.356+0000 7f7481000640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 0x7f7460005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.356+0000 7f747be00640 1 -- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 msgr2=0x7f7460005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.356+0000 7f747be00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 0x7f7460005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.356+0000 7f747be00640 1 -- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7460004800 msgr2=0x7f7460004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.356+0000 7f747be00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7460004800 0x7f7460004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.356+0000 7f747be00640 1 -- 172.21.15.27:0/2390264700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7460003520 con 0x7f74600039e0 2024-03-28T19:15:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.356+0000 7f7481000640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c0be5d0 0x7f7460005270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:15:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.357+0000 7f747be00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f7460003e00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f746c001750 tx=0x7f7460005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30065c04a2ed9b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.358+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c001370 con 0x7f74600039e0 2024-03-28T19:15:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.358+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c0052d0 con 0x7f74600039e0 2024-03-28T19:15:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.358+0000 7f747aa00640 1 -- 172.21.15.27:0/2390264700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c003830 con 0x7f74600039e0 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.359+0000 7f7481a00640 1 -- 172.21.15.27:0/2390264700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 msgr2=0x7f7460003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.359+0000 7f7481a00640 1 --2- 172.21.15.27:0/2390264700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f7460003e00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f746c001750 tx=0x7f7460005830 comp rx=0 tx=0).stop 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.359+0000 7f7481000640 1 -- 172.21.15.27:0/2390264700 reap_dead start 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.359+0000 7f7481a00640 1 -- 172.21.15.27:0/2390264700 shutdown_connections 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.359+0000 7f7481a00640 1 -- 172.21.15.27:0/2390264700 >> 172.21.15.27:0/2390264700 conn(0x7f747c06b6f0 msgr2=0x7f747c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.359+0000 7f7481a00640 1 -- 172.21.15.27:0/2390264700 shutdown_connections 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.359+0000 7f7481a00640 1 -- 172.21.15.27:0/2390264700 wait complete. 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.361+0000 7f7481a00640 1 Processor -- start 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.361+0000 7f7481a00640 1 -- start start 2024-03-28T19:15:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.362+0000 7f7481a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f747c0c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.362+0000 7f7481a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c1eea50 0x7f747c153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.362+0000 7f7481a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f747c1543a0 0x7f747c0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.362+0000 7f7481a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0b09a0 con 0x7f74600039e0 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.362+0000 7f7481a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0b0820 con 0x7f747c1543a0 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.362+0000 7f7481a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0ad7d0 con 0x7f747c1eea50 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f747c0c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f747c0c4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56674/0 (socket says 172.21.15.27:56674) 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747be00640 1 -- 172.21.15.27:0/4104526049 learned_addr learned my addr 172.21.15.27:0/4104526049 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:09.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747b400640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c1eea50 0x7f747c153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f7481000640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f747c1543a0 0x7f747c0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747be00640 1 -- 172.21.15.27:0/4104526049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c1eea50 msgr2=0x7f747c153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747be00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c1eea50 0x7f747c153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747be00640 1 -- 172.21.15.27:0/4104526049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f747c1543a0 msgr2=0x7f747c0ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747be00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f747c1543a0 0x7f747c0ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.363+0000 7f747be00640 1 -- 172.21.15.27:0/4104526049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c0c4d40 con 0x7f74600039e0 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.364+0000 7f747be00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f747c0c4800 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f746c001870 tx=0x7f746c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.364+0000 7f7473e00640 1 -- 172.21.15.27:0/4104526049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c041070 con 0x7f74600039e0 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.364+0000 7f7473e00640 1 -- 172.21.15.27:0/4104526049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c005480 con 0x7f74600039e0 2024-03-28T19:15:09.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.364+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c151f20 con 0x7f74600039e0 2024-03-28T19:15:09.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.364+0000 7f7473e00640 1 -- 172.21.15.27:0/4104526049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c001370 con 0x7f74600039e0 2024-03-28T19:15:09.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.364+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c155270 con 0x7f74600039e0 2024-03-28T19:15:09.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.366+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c06c810 con 0x7f74600039e0 2024-03-28T19:15:09.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.368+0000 7f7473e00640 1 -- 172.21.15.27:0/4104526049 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f746c044070 con 0x7f74600039e0 2024-03-28T19:15:09.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.369+0000 7f7473e00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f745c06bbd0 0x7f745c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:09.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.369+0000 7f7473e00640 1 -- 172.21.15.27:0/4104526049 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8873+0+0 (secure 0 0 0) 0x7f746c0ad4a0 con 0x7f74600039e0 2024-03-28T19:15:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.369+0000 7f747b400640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f745c06bbd0 0x7f745c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:09.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.370+0000 7f747b400640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f745c06bbd0 0x7f745c06e090 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f747c152600 tx=0x7f7468009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:09.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.372+0000 7f7473e00640 1 -- 172.21.15.27:0/4104526049 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f747c06c810 con 0x7f74600039e0 2024-03-28T19:15:09.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.685+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f747c069640 con 0x7f74600039e0 2024-03-28T19:15:09.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.686+0000 7f7473e00640 1 -- 172.21.15.27:0/4104526049 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f746c0052a0 con 0x7f74600039e0 2024-03-28T19:15:09.686 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:09.687 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":7317,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:14:44.824999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5870":{"gid":5870,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":39,"addr":"172.21.15.155:6834/2504098667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2504098667},{"type":"v1","addr":"172.21.15.155:6834","nonce":2504098667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5870,"qdb_cluster":[5870]},"id":2}]} 2024-03-28T19:15:09.687 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 153 2024-03-28T19:15:09.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f745c06bbd0 msgr2=0x7f745c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f745c06bbd0 0x7f745c06e090 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f747c152600 tx=0x7f7468009040 comp rx=0 tx=0).stop 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 msgr2=0x7f747c0c4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f747c0c4800 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f746c001870 tx=0x7f746c001120 comp rx=0 tx=0).stop 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 shutdown_connections 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f745c06bbd0 0x7f745c06e090 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f747c1543a0 0x7f747c0ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f747c1eea50 0x7f747c153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 --2- 172.21.15.27:0/4104526049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74600039e0 0x7f747c0c4800 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:09.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 >> 172.21.15.27:0/4104526049 conn(0x7f747c06b6f0 msgr2=0x7f747c157d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:09.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 shutdown_connections 2024-03-28T19:15:09.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:09.689+0000 7f7481a00640 1 -- 172.21.15.27:0/4104526049 wait complete. 2024-03-28T19:15:09.712 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:15:11.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.952+0000 7efe9f200640 1 Processor -- start 2024-03-28T19:15:11.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.953+0000 7efe9f200640 1 -- start start 2024-03-28T19:15:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.954+0000 7efe9f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe981555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.954+0000 7efe9f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe98156000 con 0x7efe980c2e30 2024-03-28T19:15:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.954+0000 7efe9f200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe981562e0 con 0x7efe98155af0 2024-03-28T19:15:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.954+0000 7efe9f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe98158d40 con 0x7efe98159540 2024-03-28T19:15:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.955+0000 7efe9de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efe980c2e30 0x7efe980c2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59192/0 (socket says 172.21.15.27:59192) 2024-03-28T19:15:11.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.955+0000 7efe9de00640 1 -- 172.21.15.27:0/274706584 learned_addr learned my addr 172.21.15.27:0/274706584 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.955+0000 7efe9d400640 1 --2- 172.21.15.27:0/274706584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe981555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.956+0000 7efe9ca00640 1 -- 172.21.15.27:0/274706584 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944492282 0 0) 0x7efe98156000 con 0x7efe980c2e30 2024-03-28T19:15:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.956+0000 7efe9ca00640 1 -- 172.21.15.27:0/274706584 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe80003520 con 0x7efe980c2e30 2024-03-28T19:15:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.956+0000 7efe9d400640 1 -- 172.21.15.27:0/274706584 >> v1:172.21.15.155:6789/0 conn(0x7efe98155af0 legacy=0x7efe980a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.956+0000 7efe9ca00640 1 -- 172.21.15.27:0/274706584 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023392053 0 0) 0x7efe981562e0 con 0x7efe98155af0 2024-03-28T19:15:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.956+0000 7efe9d400640 1 -- 172.21.15.27:0/274706584 >> v1:172.21.15.27:6789/0 conn(0x7efe980c2e30 legacy=0x7efe980c2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:11.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.956+0000 7efe9d400640 1 -- 172.21.15.27:0/274706584 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe980a85b0 con 0x7efe98159540 2024-03-28T19:15:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.958+0000 7efe9d400640 1 --2- 172.21.15.27:0/274706584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe981555b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efe880097d0 tx=0x7efe8802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c72ce2f58e810e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.959+0000 7efe9ca00640 1 -- 172.21.15.27:0/274706584 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe880315a0 con 0x7efe98159540 2024-03-28T19:15:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.959+0000 7efe9ca00640 1 -- 172.21.15.27:0/274706584 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe88037070 con 0x7efe98159540 2024-03-28T19:15:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.959+0000 7efe9ca00640 1 -- 172.21.15.27:0/274706584 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88031960 con 0x7efe98159540 2024-03-28T19:15:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.960+0000 7efe9f200640 1 -- 172.21.15.27:0/274706584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 msgr2=0x7efe981555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.960+0000 7efe9f200640 1 --2- 172.21.15.27:0/274706584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe981555b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efe880097d0 tx=0x7efe8802b840 comp rx=0 tx=0).stop 2024-03-28T19:15:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.960+0000 7efe9f200640 1 -- 172.21.15.27:0/274706584 shutdown_connections 2024-03-28T19:15:11.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.960+0000 7efe9f200640 1 --2- 172.21.15.27:0/274706584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe981555b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.960+0000 7efe9f200640 1 -- 172.21.15.27:0/274706584 >> 172.21.15.27:0/274706584 conn(0x7efe98072d10 msgr2=0x7efe9806d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.960+0000 7efe9f200640 1 -- 172.21.15.27:0/274706584 shutdown_connections 2024-03-28T19:15:11.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.961+0000 7efe9f200640 1 -- 172.21.15.27:0/274706584 wait complete. 2024-03-28T19:15:11.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.963+0000 7efe9f200640 1 Processor -- start 2024-03-28T19:15:11.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.963+0000 7efe9f200640 1 -- start start 2024-03-28T19:15:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.964+0000 7efe9f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe980c2e30 0x7efe9814e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.964+0000 7efe9f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155af0 0x7efe9814bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.964+0000 7efe9f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe9814c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.964+0000 7efe9f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe98159b40 con 0x7efe98155af0 2024-03-28T19:15:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.964+0000 7efe9f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe981599c0 con 0x7efe980c2e30 2024-03-28T19:15:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.964+0000 7efe9f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9806fe80 con 0x7efe98159540 2024-03-28T19:15:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.965+0000 7efe9d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155af0 0x7efe9814bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.965+0000 7efe9d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155af0 0x7efe9814bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56678/0 (socket says 172.21.15.27:56678) 2024-03-28T19:15:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.965+0000 7efe9d400640 1 -- 172.21.15.27:0/861584003 learned_addr learned my addr 172.21.15.27:0/861584003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.965+0000 7efe9e800640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe9814c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.966+0000 7efe9d400640 1 -- 172.21.15.27:0/861584003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 msgr2=0x7efe9814c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.966+0000 7efe9d400640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe9814c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.966+0000 7efe9d400640 1 -- 172.21.15.27:0/861584003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe980c2e30 msgr2=0x7efe9814e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.966+0000 7efe9d400640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe980c2e30 0x7efe9814e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.966+0000 7efe9d400640 1 -- 172.21.15.27:0/861584003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9814c9e0 con 0x7efe98155af0 2024-03-28T19:15:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.967+0000 7efe9d400640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155af0 0x7efe9814bf60 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7efe880097d0 tx=0x7efe880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.967+0000 7efe92000640 1 -- 172.21.15.27:0/861584003 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88049070 con 0x7efe98155af0 2024-03-28T19:15:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.967+0000 7efe92000640 1 -- 172.21.15.27:0/861584003 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe88032430 con 0x7efe98155af0 2024-03-28T19:15:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.967+0000 7efe92000640 1 -- 172.21.15.27:0/861584003 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe88032820 con 0x7efe98155af0 2024-03-28T19:15:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.968+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe981eee30 con 0x7efe98155af0 2024-03-28T19:15:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.968+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe981520b0 con 0x7efe98155af0 2024-03-28T19:15:11.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.970+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe98067dd0 con 0x7efe98155af0 2024-03-28T19:15:11.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.973+0000 7efe92000640 1 -- 172.21.15.27:0/861584003 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efe880329c0 con 0x7efe98155af0 2024-03-28T19:15:11.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.975+0000 7efe92000640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bca0 0x7efe7c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:11.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.975+0000 7efe92000640 1 -- 172.21.15.27:0/861584003 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7efe8802f020 con 0x7efe98155af0 2024-03-28T19:15:11.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.976+0000 7efe9de00640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bca0 0x7efe7c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:11.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.977+0000 7efe9de00640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bca0 0x7efe7c06e160 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efe8c000f60 tx=0x7efe8c0078f0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:11.981+0000 7efe92000640 1 -- 172.21.15.27:0/861584003 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efe8807a460 con 0x7efe98155af0 2024-03-28T19:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.273+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe98153050 con 0x7efe98155af0 2024-03-28T19:15:12.275 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:12.275 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:10.114978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:15:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 154 2024-03-28T19:15:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.274+0000 7efe92000640 1 -- 172.21.15.27:0/861584003 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5460 (secure 0 0 0) 0x7efe98153050 con 0x7efe98155af0 2024-03-28T19:15:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.276+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bca0 msgr2=0x7efe7c06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.276+0000 7efe9f200640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bca0 0x7efe7c06e160 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efe8c000f60 tx=0x7efe8c0078f0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.276+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155af0 msgr2=0x7efe9814bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.276+0000 7efe9f200640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155af0 0x7efe9814bf60 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7efe880097d0 tx=0x7efe880046c0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.277+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 shutdown_connections 2024-03-28T19:15:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.277+0000 7efe9f200640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efe7c06bca0 0x7efe7c06e160 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.277+0000 7efe9f200640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe98159540 0x7efe9814c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.277+0000 7efe9f200640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe98155af0 0x7efe9814bf60 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.277+0000 7efe9f200640 1 --2- 172.21.15.27:0/861584003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe980c2e30 0x7efe9814e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.277+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 >> 172.21.15.27:0/861584003 conn(0x7efe98072d10 msgr2=0x7efe981537c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.277+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 shutdown_connections 2024-03-28T19:15:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.277+0000 7efe9f200640 1 -- 172.21.15.27:0/861584003 wait complete. 2024-03-28T19:15:12.394 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-03-28T19:15:12.394 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:15:12.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:12.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.621+0000 7f174fa00640 1 Processor -- start 2024-03-28T19:15:12.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.621+0000 7f174fa00640 1 -- start start 2024-03-28T19:15:12.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.623+0000 7f174fa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1748150c70 0x7f17480a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:12.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.623+0000 7f174fa00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f17480c3690 0x7f17480c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:12.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.623+0000 7f174fa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c6660 0x7f17481534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:12.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.624+0000 7f174fa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17480ab4e0 con 0x7f1748150c70 2024-03-28T19:15:12.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.624+0000 7f174fa00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17480ac1f0 con 0x7f17480c3690 2024-03-28T19:15:12.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.624+0000 7f174fa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480c3af0 con 0x7f17480c6660 2024-03-28T19:15:12.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.624+0000 7f174d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1748150c70 0x7f17480a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:12.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.624+0000 7f174d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1748150c70 0x7f17480a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56694/0 (socket says 172.21.15.27:56694) 2024-03-28T19:15:12.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.624+0000 7f174d000640 1 -- 172.21.15.27:0/2699772779 learned_addr learned my addr 172.21.15.27:0/2699772779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:12.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.624+0000 7f174da00640 1 --2- 172.21.15.27:0/2699772779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c6660 0x7f17481534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:12.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.625+0000 7f1747e00640 1 --2- 172.21.15.27:0/2699772779 >> v2:172.21.15.155:3300/0 conn(0x7f17480c3690 0x7f17480c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:12.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.625+0000 7f174d000640 1 -- 172.21.15.27:0/2699772779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c6660 msgr2=0x7f17481534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:12.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.625+0000 7f174d000640 1 --2- 172.21.15.27:0/2699772779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c6660 0x7f17481534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.625+0000 7f174d000640 1 -- 172.21.15.27:0/2699772779 >> v2:172.21.15.155:3300/0 conn(0x7f17480c3690 msgr2=0x7f17480c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:12.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.625+0000 7f174d000640 1 --2- 172.21.15.27:0/2699772779 >> v2:172.21.15.155:3300/0 conn(0x7f17480c3690 0x7f17480c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.625+0000 7f174d000640 1 -- 172.21.15.27:0/2699772779 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17480c3d90 con 0x7f1748150c70 2024-03-28T19:15:12.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.625+0000 7f1747e00640 1 --2- 172.21.15.27:0/2699772779 >> v2:172.21.15.155:3300/0 conn(0x7f17480c3690 0x7f17480c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:15:12.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.626+0000 7f174d000640 1 --2- 172.21.15.27:0/2699772779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1748150c70 0x7f17480a9cb0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f17480c4180 tx=0x7f173c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b446c0f055ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:12.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.627+0000 7f1747400640 1 -- 172.21.15.27:0/2699772779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c0317c0 con 0x7f1748150c70 2024-03-28T19:15:12.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.627+0000 7f1747400640 1 -- 172.21.15.27:0/2699772779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c037070 con 0x7f1748150c70 2024-03-28T19:15:12.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.627+0000 7f1747400640 1 -- 172.21.15.27:0/2699772779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c031b80 con 0x7f1748150c70 2024-03-28T19:15:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.628+0000 7f174fa00640 1 -- 172.21.15.27:0/2699772779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1748150c70 msgr2=0x7f17480a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.628+0000 7f174fa00640 1 --2- 172.21.15.27:0/2699772779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1748150c70 0x7f17480a9cb0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f17480c4180 tx=0x7f173c02b860 comp rx=0 tx=0).stop 2024-03-28T19:15:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.628+0000 7f174fa00640 1 -- 172.21.15.27:0/2699772779 shutdown_connections 2024-03-28T19:15:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.628+0000 7f174fa00640 1 --2- 172.21.15.27:0/2699772779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c6660 0x7f17481534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.628+0000 7f174fa00640 1 --2- 172.21.15.27:0/2699772779 >> v2:172.21.15.155:3300/0 conn(0x7f17480c3690 0x7f17480c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.628+0000 7f174fa00640 1 --2- 172.21.15.27:0/2699772779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1748150c70 0x7f17480a9cb0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.628+0000 7f174fa00640 1 -- 172.21.15.27:0/2699772779 >> 172.21.15.27:0/2699772779 conn(0x7f174806b9d0 msgr2=0x7f17480c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.629+0000 7f174fa00640 1 -- 172.21.15.27:0/2699772779 shutdown_connections 2024-03-28T19:15:12.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.629+0000 7f174fa00640 1 -- 172.21.15.27:0/2699772779 wait complete. 2024-03-28T19:15:12.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.632+0000 7f174fa00640 1 Processor -- start 2024-03-28T19:15:12.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.632+0000 7f174fa00640 1 -- start start 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.633+0000 7f174fa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c3690 0x7f1748156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.633+0000 7f174fa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17480c6660 0x7f17480c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.633+0000 7f174fa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1748150c70 0x7f174814bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.633+0000 7f174fa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17480abcd0 con 0x7f17480c6660 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.633+0000 7f174fa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17480ab9f0 con 0x7f1748150c70 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.633+0000 7f174fa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480c1810 con 0x7f17480c3690 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f1747e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17480c6660 0x7f17480c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f1747e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17480c6660 0x7f17480c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56698/0 (socket says 172.21.15.27:56698) 2024-03-28T19:15:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f1747e00640 1 -- 172.21.15.27:0/2567521582 learned_addr learned my addr 172.21.15.27:0/2567521582 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f174d000640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c3690 0x7f1748156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f174da00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1748150c70 0x7f174814bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f1747e00640 1 -- 172.21.15.27:0/2567521582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c3690 msgr2=0x7f1748156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f1747e00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c3690 0x7f1748156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f1747e00640 1 -- 172.21.15.27:0/2567521582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1748150c70 msgr2=0x7f174814bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f1747e00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1748150c70 0x7f174814bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.634+0000 7f1747e00640 1 -- 172.21.15.27:0/2567521582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174806ea30 con 0x7f17480c6660 2024-03-28T19:15:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.635+0000 7f1747e00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17480c6660 0x7f17480c02a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f173400c7f0 tx=0x7f173400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:12.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.635+0000 7f1744c00640 1 -- 172.21.15.27:0/2567521582 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734015070 con 0x7f17480c6660 2024-03-28T19:15:12.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.635+0000 7f1744c00640 1 -- 172.21.15.27:0/2567521582 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1734007820 con 0x7f17480c6660 2024-03-28T19:15:12.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.635+0000 7f1744c00640 1 -- 172.21.15.27:0/2567521582 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734007bd0 con 0x7f17480c6660 2024-03-28T19:15:12.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.635+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1748154690 con 0x7f17480c6660 2024-03-28T19:15:12.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.635+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1748152420 con 0x7f17480c6660 2024-03-28T19:15:12.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.637+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17480c6420 con 0x7f17480c6660 2024-03-28T19:15:12.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.638+0000 7f1744c00640 1 -- 172.21.15.27:0/2567521582 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1734018070 con 0x7f17480c6660 2024-03-28T19:15:12.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.640+0000 7f1744c00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172806bbd0 0x7f172806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:12.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.640+0000 7f1744c00640 1 -- 172.21.15.27:0/2567521582 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f173408b8c0 con 0x7f17480c6660 2024-03-28T19:15:12.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.641+0000 7f174d000640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172806bbd0 0x7f172806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:12.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.642+0000 7f174d000640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172806bbd0 0x7f172806e090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f173c036000 tx=0x7f173c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:12.648+0000 7f1744c00640 1 -- 172.21.15.27:0/2567521582 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f17480c6420 con 0x7f17480c6660 2024-03-28T19:15:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.028+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1748152620 con 0x7f17480c6660 2024-03-28T19:15:13.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.030+0000 7f1744c00640 1 -- 172.21.15.27:0/2567521582 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f1748152620 con 0x7f17480c6660 2024-03-28T19:15:13.030 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:13.030 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:10.114978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:15:13.030 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 154 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172806bbd0 msgr2=0x7f172806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172806bbd0 0x7f172806e090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f173c036000 tx=0x7f173c008030 comp rx=0 tx=0).stop 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17480c6660 msgr2=0x7f17480c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17480c6660 0x7f17480c02a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f173400c7f0 tx=0x7f173400ccb0 comp rx=0 tx=0).stop 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 shutdown_connections 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f172806bbd0 0x7f172806e090 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1748150c70 0x7f174814bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17480c6660 0x7f17480c02a0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 --2- 172.21.15.27:0/2567521582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17480c3690 0x7f1748156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 >> 172.21.15.27:0/2567521582 conn(0x7f174806b9d0 msgr2=0x7f174814ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.033+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 shutdown_connections 2024-03-28T19:15:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:13.034+0000 7f174fa00640 1 -- 172.21.15.27:0/2567521582 wait complete. 2024-03-28T19:15:13.059 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7146": { "addr": "172.21.15.27:6835/3913435536", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 3913435536, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 3913435536, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7146, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:14:41.786270+0000", "qdb_cluster": [ 7146 ], "qdb_leader": 7146, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7146 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7317": { "addr": "172.21.15.27:6837/3534117308", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 3534117308, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 3534117308, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7317, "incarnation": 154, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:15:10.114978+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7317 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/938167944", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 938167944, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 938167944, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 153, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6014, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:15:15.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.287+0000 7f1212a00640 1 Processor -- start 2024-03-28T19:15:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.287+0000 7f1212a00640 1 -- start start 2024-03-28T19:15:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.288+0000 7f1212a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f120c152b60 0x7f120c1593e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.288+0000 7f1212a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f120c0c1820 0x7f120c0a6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.288+0000 7f1212a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c155b30 con 0x7f120c06c0d0 2024-03-28T19:15:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.288+0000 7f1212a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120c158990 con 0x7f120c0c1820 2024-03-28T19:15:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.288+0000 7f1212a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120c06c560 con 0x7f120c152b60 2024-03-28T19:15:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.288+0000 7f1212000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f120c0c1820 0x7f120c0a6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.288+0000 7f1212000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f120c0c1820 0x7f120c0a6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56706/0 (socket says 172.21.15.27:56706) 2024-03-28T19:15:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.288+0000 7f1212000640 1 -- 172.21.15.27:0/1748784878 learned_addr learned my addr 172.21.15.27:0/1748784878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.289+0000 7f1211600640 1 --2- 172.21.15.27:0/1748784878 >> v2:172.21.15.155:3300/0 conn(0x7f120c152b60 0x7f120c1593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:15.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.289+0000 7f1212000640 1 -- 172.21.15.27:0/1748784878 >> v2:172.21.15.155:3300/0 conn(0x7f120c152b60 msgr2=0x7f120c1593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:15.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.289+0000 7f1212000640 1 --2- 172.21.15.27:0/1748784878 >> v2:172.21.15.155:3300/0 conn(0x7f120c152b60 0x7f120c1593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.289+0000 7f1212000640 1 -- 172.21.15.27:0/1748784878 >> v1:172.21.15.155:6789/0 conn(0x7f120c06c0d0 legacy=0x7f120c155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:15.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.289+0000 7f1212000640 1 -- 172.21.15.27:0/1748784878 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c06c840 con 0x7f120c0c1820 2024-03-28T19:15:15.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.291+0000 7f1212000640 1 --2- 172.21.15.27:0/1748784878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c0c1820 0x7f120c0a6c70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f120c067de0 tx=0x7f120402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f65c303a636d39c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:15.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.291+0000 7f120be00640 1 -- 172.21.15.27:0/1748784878 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120402c9f0 con 0x7f120c0c1820 2024-03-28T19:15:15.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.291+0000 7f120be00640 1 -- 172.21.15.27:0/1748784878 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120402cb90 con 0x7f120c0c1820 2024-03-28T19:15:15.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.291+0000 7f120be00640 1 -- 172.21.15.27:0/1748784878 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12040343f0 con 0x7f120c0c1820 2024-03-28T19:15:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.292+0000 7f1212a00640 1 -- 172.21.15.27:0/1748784878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c0c1820 msgr2=0x7f120c0a6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.292+0000 7f1212a00640 1 --2- 172.21.15.27:0/1748784878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c0c1820 0x7f120c0a6c70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f120c067de0 tx=0x7f120402be50 comp rx=0 tx=0).stop 2024-03-28T19:15:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.292+0000 7f1212a00640 1 -- 172.21.15.27:0/1748784878 shutdown_connections 2024-03-28T19:15:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.292+0000 7f1212a00640 1 --2- 172.21.15.27:0/1748784878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c0c1820 0x7f120c0a6c70 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.292+0000 7f1212a00640 1 --2- 172.21.15.27:0/1748784878 >> v2:172.21.15.155:3300/0 conn(0x7f120c152b60 0x7f120c1593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.292+0000 7f1212a00640 1 -- 172.21.15.27:0/1748784878 >> 172.21.15.27:0/1748784878 conn(0x7f120c0c7aa0 msgr2=0x7f120c0c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:15.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.292+0000 7f1212a00640 1 -- 172.21.15.27:0/1748784878 shutdown_connections 2024-03-28T19:15:15.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.292+0000 7f1212a00640 1 -- 172.21.15.27:0/1748784878 wait complete. 2024-03-28T19:15:15.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.295+0000 7f1212a00640 1 Processor -- start 2024-03-28T19:15:15.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.295+0000 7f1212a00640 1 -- start start 2024-03-28T19:15:15.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1212a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f120c0c1820 0x7f120c1e2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:15.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1212a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c152b60 0x7f120c06fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:15.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1212a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f120c14fb50 0x7f120c0a8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:15.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1212a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c0c1ff0 con 0x7f120c152b60 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1212a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c158ec0 con 0x7f120c14fb50 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1212a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120c0c2170 con 0x7f120c0c1820 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1210c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c152b60 0x7f120c06fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1210c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c152b60 0x7f120c06fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56710/0 (socket says 172.21.15.27:56710) 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1210c00640 1 -- 172.21.15.27:0/3987688255 learned_addr learned my addr 172.21.15.27:0/3987688255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1212000640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f120c14fb50 0x7f120c0a8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1211600640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f120c0c1820 0x7f120c1e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1210c00640 1 -- 172.21.15.27:0/3987688255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f120c0c1820 msgr2=0x7f120c1e2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.297+0000 7f1210c00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f120c0c1820 0x7f120c1e2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1210c00640 1 -- 172.21.15.27:0/3987688255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f120c14fb50 msgr2=0x7f120c0a8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1210c00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f120c14fb50 0x7f120c0a8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1210c00640 1 -- 172.21.15.27:0/3987688255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c0aaa10 con 0x7f120c152b60 2024-03-28T19:15:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1210c00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c152b60 0x7f120c06fc50 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f11f800c460 tx=0x7f11f800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:15.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1209600640 1 -- 172.21.15.27:0/3987688255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8011790 con 0x7f120c152b60 2024-03-28T19:15:15.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1209600640 1 -- 172.21.15.27:0/3987688255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f8017070 con 0x7f120c152b60 2024-03-28T19:15:15.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120c0aacb0 con 0x7f120c152b60 2024-03-28T19:15:15.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1209600640 1 -- 172.21.15.27:0/3987688255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8011bb0 con 0x7f120c152b60 2024-03-28T19:15:15.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.298+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120c0a8ec0 con 0x7f120c152b60 2024-03-28T19:15:15.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.299+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120c069640 con 0x7f120c152b60 2024-03-28T19:15:15.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.301+0000 7f1209600640 1 -- 172.21.15.27:0/3987688255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11f801c460 con 0x7f120c152b60 2024-03-28T19:15:15.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.301+0000 7f1209600640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11ec06bc00 0x7f11ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:15.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.302+0000 7f1209600640 1 -- 172.21.15.27:0/3987688255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f11f808b610 con 0x7f120c152b60 2024-03-28T19:15:15.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.302+0000 7f1211600640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11ec06bc00 0x7f11ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:15.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.303+0000 7f1211600640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11ec06bc00 0x7f11ec06e0c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1200006fd0 tx=0x7f1200008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:15.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.308+0000 7f1209600640 1 -- 172.21.15.27:0/3987688255 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11f8059430 con 0x7f120c152b60 2024-03-28T19:15:15.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.631+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f120c0c74b0 con 0x7f120c152b60 2024-03-28T19:15:15.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.632+0000 7f1209600640 1 -- 172.21.15.27:0/3987688255 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f11f8058d30 con 0x7f120c152b60 2024-03-28T19:15:15.632 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:15.633 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:14.589449+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:15:15.633 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 155 2024-03-28T19:15:15.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11ec06bc00 msgr2=0x7f11ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:15.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11ec06bc00 0x7f11ec06e0c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1200006fd0 tx=0x7f1200008040 comp rx=0 tx=0).stop 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c152b60 msgr2=0x7f120c06fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c152b60 0x7f120c06fc50 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f11f800c460 tx=0x7f11f800c920 comp rx=0 tx=0).stop 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 shutdown_connections 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11ec06bc00 0x7f11ec06e0c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f120c14fb50 0x7f120c0a8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f120c152b60 0x7f120c06fc50 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 --2- 172.21.15.27:0/3987688255 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f120c0c1820 0x7f120c1e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 >> 172.21.15.27:0/3987688255 conn(0x7f120c0c7aa0 msgr2=0x7f120c066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 shutdown_connections 2024-03-28T19:15:15.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:15.635+0000 7f1212a00640 1 -- 172.21.15.27:0/3987688255 wait complete. 2024-03-28T19:15:15.655 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:15:17.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:17.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.954+0000 7f5d33a00640 1 Processor -- start 2024-03-28T19:15:17.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.954+0000 7f5d33a00640 1 -- start start 2024-03-28T19:15:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.956+0000 7f5d33a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0c4060 0x7f5d2c06f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.956+0000 7f5d33a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5d2c0a9f50 0x7f5d2c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.956+0000 7f5d33a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c152560 con 0x7f5d2c0c4440 2024-03-28T19:15:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.956+0000 7f5d33a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2c153170 con 0x7f5d2c0a9f50 2024-03-28T19:15:17.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.956+0000 7f5d33a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c152840 con 0x7f5d2c0c4060 2024-03-28T19:15:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.957+0000 7f5d31a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5d2c0a9f50 0x7f5d2c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.957+0000 7f5d31a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5d2c0a9f50 0x7f5d2c155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:60046/0 (socket says 172.21.15.27:60046) 2024-03-28T19:15:17.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.957+0000 7f5d31a00640 1 -- 172.21.15.27:0/1326045520 learned_addr learned my addr 172.21.15.27:0/1326045520 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.957+0000 7f5d31000640 1 --2- 172.21.15.27:0/1326045520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0c4060 0x7f5d2c06f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.958+0000 7f5d31a00640 1 -- 172.21.15.27:0/1326045520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0c4060 msgr2=0x7f5d2c06f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.958+0000 7f5d31a00640 1 --2- 172.21.15.27:0/1326045520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0c4060 0x7f5d2c06f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.958+0000 7f5d31a00640 1 -- 172.21.15.27:0/1326045520 >> v1:172.21.15.155:6789/0 conn(0x7f5d2c0c4440 legacy=0x7f5d2c0c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:17.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.958+0000 7f5d31a00640 1 -- 172.21.15.27:0/1326045520 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c152b20 con 0x7f5d2c0a9f50 2024-03-28T19:15:17.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.960+0000 7f5d31a00640 1 --2- 172.21.15.27:0/1326045520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0a9f50 0x7f5d2c155ab0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009e50 tx=0x7f5d2402e0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d60d6b46bde470 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:17.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.962+0000 7f5d2b400640 1 -- 172.21.15.27:0/1326045520 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2402ec70 con 0x7f5d2c0a9f50 2024-03-28T19:15:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.962+0000 7f5d2b400640 1 -- 172.21.15.27:0/1326045520 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2402ee10 con 0x7f5d2c0a9f50 2024-03-28T19:15:17.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.962+0000 7f5d2b400640 1 -- 172.21.15.27:0/1326045520 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24036610 con 0x7f5d2c0a9f50 2024-03-28T19:15:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.963+0000 7f5d33a00640 1 -- 172.21.15.27:0/1326045520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0a9f50 msgr2=0x7f5d2c155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:17.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.963+0000 7f5d33a00640 1 --2- 172.21.15.27:0/1326045520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0a9f50 0x7f5d2c155ab0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009e50 tx=0x7f5d2402e0f0 comp rx=0 tx=0).stop 2024-03-28T19:15:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.964+0000 7f5d33a00640 1 -- 172.21.15.27:0/1326045520 shutdown_connections 2024-03-28T19:15:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.964+0000 7f5d33a00640 1 --2- 172.21.15.27:0/1326045520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0a9f50 0x7f5d2c155ab0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.964+0000 7f5d33a00640 1 --2- 172.21.15.27:0/1326045520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0c4060 0x7f5d2c06f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.964+0000 7f5d33a00640 1 -- 172.21.15.27:0/1326045520 >> 172.21.15.27:0/1326045520 conn(0x7f5d2c06c5f0 msgr2=0x7f5d2c06ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.965+0000 7f5d33a00640 1 -- 172.21.15.27:0/1326045520 shutdown_connections 2024-03-28T19:15:17.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.965+0000 7f5d33a00640 1 -- 172.21.15.27:0/1326045520 wait complete. 2024-03-28T19:15:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.970+0000 7f5d33a00640 1 Processor -- start 2024-03-28T19:15:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.970+0000 7f5d33a00640 1 -- start start 2024-03-28T19:15:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.974+0000 7f5d33a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0a9f50 0x7f5d2c1e7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.974+0000 7f5d33a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0c4060 0x7f5d2c149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.974+0000 7f5d33a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c0c4440 0x7f5d2c1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.974+0000 7f5d31000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0a9f50 0x7f5d2c1e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.975+0000 7f5d31a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c0c4440 0x7f5d2c1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.975+0000 7f5d31000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0a9f50 0x7f5d2c1e7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:48946/0 (socket says 172.21.15.27:48946) 2024-03-28T19:15:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.975+0000 7f5d31a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c0c4440 0x7f5d2c1509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37126/0 (socket says 172.21.15.27:37126) 2024-03-28T19:15:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.975+0000 7f5d31a00640 1 -- 172.21.15.27:0/253541405 learned_addr learned my addr 172.21.15.27:0/253541405 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.975+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c58e0 con 0x7f5d2c0c4440 2024-03-28T19:15:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.975+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c5760 con 0x7f5d2c0c4060 2024-03-28T19:15:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.975+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c155ff0 con 0x7f5d2c0a9f50 2024-03-28T19:15:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.975+0000 7f5d2be00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0c4060 0x7f5d2c149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.976+0000 7f5d31a00640 1 -- 172.21.15.27:0/253541405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0a9f50 msgr2=0x7f5d2c1e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.976+0000 7f5d31a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0a9f50 0x7f5d2c1e7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.976+0000 7f5d31a00640 1 -- 172.21.15.27:0/253541405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0c4060 msgr2=0x7f5d2c149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.976+0000 7f5d31a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0c4060 0x7f5d2c149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.976+0000 7f5d31a00640 1 -- 172.21.15.27:0/253541405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c151b30 con 0x7f5d2c0c4440 2024-03-28T19:15:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.976+0000 7f5d31a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c0c4440 0x7f5d2c1509a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f5d24038040 tx=0x7f5d24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.977+0000 7f5d28c00640 1 -- 172.21.15.27:0/253541405 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24045070 con 0x7f5d2c0c4440 2024-03-28T19:15:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.977+0000 7f5d28c00640 1 -- 172.21.15.27:0/253541405 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d24041660 con 0x7f5d2c0c4440 2024-03-28T19:15:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.977+0000 7f5d28c00640 1 -- 172.21.15.27:0/253541405 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24041a50 con 0x7f5d2c0c4440 2024-03-28T19:15:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.977+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c151e10 con 0x7f5d2c0c4440 2024-03-28T19:15:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.978+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c150480 con 0x7f5d2c0c4440 2024-03-28T19:15:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.980+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c04f400 con 0x7f5d2c0c4440 2024-03-28T19:15:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.981+0000 7f5d28c00640 1 -- 172.21.15.27:0/253541405 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5d24048070 con 0x7f5d2c0c4440 2024-03-28T19:15:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.982+0000 7f5d28c00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d0c06bc20 0x7f5d0c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.982+0000 7f5d28c00640 1 -- 172.21.15.27:0/253541405 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f5d240afdb0 con 0x7f5d2c0c4440 2024-03-28T19:15:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.983+0000 7f5d31000640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d0c06bc20 0x7f5d0c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.986+0000 7f5d31000640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d0c06bc20 0x7f5d0c06e0e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5d20006fd0 tx=0x7f5d20008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:17.991+0000 7f5d28c00640 1 -- 172.21.15.27:0/253541405 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5d2407d660 con 0x7f5d2c0c4440 2024-03-28T19:15:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.328+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d2c0a9d30 con 0x7f5d2c0c4440 2024-03-28T19:15:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.329+0000 7f5d28c00640 1 -- 172.21.15.27:0/253541405 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+5467 (secure 0 0 0) 0x7f5d24081700 con 0x7f5d2c0c4440 2024-03-28T19:15:18.329 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:18.329 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:16.657896+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:15:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 157 2024-03-28T19:15:18.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.331+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d0c06bc20 msgr2=0x7f5d0c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:18.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.331+0000 7f5d33a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d0c06bc20 0x7f5d0c06e0e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5d20006fd0 tx=0x7f5d20008040 comp rx=0 tx=0).stop 2024-03-28T19:15:18.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c0c4440 msgr2=0x7f5d2c1509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c0c4440 0x7f5d2c1509a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f5d24038040 tx=0x7f5d24004a40 comp rx=0 tx=0).stop 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 shutdown_connections 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5d0c06bc20 0x7f5d0c06e0e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d2c0c4440 0x7f5d2c1509a0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d2c0c4060 0x7f5d2c149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 --2- 172.21.15.27:0/253541405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5d2c0a9f50 0x7f5d2c1e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 >> 172.21.15.27:0/253541405 conn(0x7f5d2c06c5f0 msgr2=0x7f5d2c0c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 shutdown_connections 2024-03-28T19:15:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:18.332+0000 7f5d33a00640 1 -- 172.21.15.27:0/253541405 wait complete. 2024-03-28T19:15:18.359 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:15:19.849 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:15:19.849+0000 7f79d8600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:15:20.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.618+0000 7f9f5a600640 1 Processor -- start 2024-03-28T19:15:20.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.619+0000 7f9f5a600640 1 -- start start 2024-03-28T19:15:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.621+0000 7f9f5a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54156250 0x7f9f54159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.621+0000 7f9f5a600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f540c5710 con 0x7f9f540c6cc0 2024-03-28T19:15:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.621+0000 7f9f5a600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f540c59f0 con 0x7f9f54152a70 2024-03-28T19:15:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.621+0000 7f9f5a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f541556b0 con 0x7f9f54156250 2024-03-28T19:15:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.621+0000 7f9f53e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9f540c6cc0 0x7f9f54155b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58556/0 (socket says 172.21.15.27:58556) 2024-03-28T19:15:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.621+0000 7f9f53e00640 1 -- 172.21.15.27:0/1159558741 learned_addr learned my addr 172.21.15.27:0/1159558741 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:20.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.621+0000 7f9f59c00640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54156250 0x7f9f54159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848785675 0 0) 0x7f9f540c5710 con 0x7f9f540c6cc0 2024-03-28T19:15:20.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f3c003520 con 0x7f9f540c6cc0 2024-03-28T19:15:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2279991732 0 0) 0x7f9f3c003520 con 0x7f9f540c6cc0 2024-03-28T19:15:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54156250 msgr2=0x7f9f54159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54156250 0x7f9f54159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 >> v1:172.21.15.155:6789/0 conn(0x7f9f54152a70 legacy=0x7f9f540cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f540c5cd0 con 0x7f9f540c6cc0 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9f40001670 con 0x7f9f540c6cc0 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 >> v1:172.21.15.27:6789/0 conn(0x7f9f540c6cc0 legacy=0x7f9f54155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f3c0039e0 0x7f9f3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.622+0000 7f9f53400640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f3c004800 0x7f9f3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f53400640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f3c0051c0 0x7f9f3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f541556b0 con 0x7f9f3c0039e0 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c00c090 con 0x7f9f3c004800 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c00c240 con 0x7f9f3c0051c0 2024-03-28T19:15:20.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f59c00640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54156250 0x7f9f54159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:15:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f59200640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f3c0039e0 0x7f9f3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f59200640 1 -- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f3c0051c0 msgr2=0x7f9f3c00b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T19:15:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f59200640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f3c0051c0 0x7f9f3c00b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f59200640 1 -- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f3c004800 msgr2=0x7f9f3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f59200640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f3c004800 0x7f9f3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:20.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.623+0000 7f9f59200640 1 -- 172.21.15.27:0/1159558741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c003520 con 0x7f9f3c0039e0 2024-03-28T19:15:20.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.625+0000 7f9f59200640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f3c0039e0 0x7f9f3c003dc0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f9f48001860 tx=0x7f9f3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7667f9d46ee7b6a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:20.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.626+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48001430 con 0x7f9f3c0039e0 2024-03-28T19:15:20.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.626+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f48032620 con 0x7f9f3c0039e0 2024-03-28T19:15:20.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.626+0000 7f9f53400640 1 -- 172.21.15.27:0/1159558741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48032a10 con 0x7f9f3c0039e0 2024-03-28T19:15:20.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.627+0000 7f9f5a600640 1 -- 172.21.15.27:0/1159558741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f3c0039e0 msgr2=0x7f9f3c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:20.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.627+0000 7f9f5a600640 1 --2- 172.21.15.27:0/1159558741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f3c0039e0 0x7f9f3c003dc0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f9f48001860 tx=0x7f9f3c005830 comp rx=0 tx=0).stop 2024-03-28T19:15:20.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.627+0000 7f9f59c00640 1 -- 172.21.15.27:0/1159558741 reap_dead start 2024-03-28T19:15:20.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.627+0000 7f9f5a600640 1 -- 172.21.15.27:0/1159558741 shutdown_connections 2024-03-28T19:15:20.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.627+0000 7f9f5a600640 1 -- 172.21.15.27:0/1159558741 >> 172.21.15.27:0/1159558741 conn(0x7f9f5406b2a0 msgr2=0x7f9f5406b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:20.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.627+0000 7f9f5a600640 1 -- 172.21.15.27:0/1159558741 shutdown_connections 2024-03-28T19:15:20.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.627+0000 7f9f5a600640 1 -- 172.21.15.27:0/1159558741 wait complete. 2024-03-28T19:15:20.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.630+0000 7f9f5a600640 1 Processor -- start 2024-03-28T19:15:20.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.631+0000 7f9f5a600640 1 -- start start 2024-03-28T19:15:20.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.632+0000 7f9f5a600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54148b30 0x7f9f54148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:20.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.632+0000 7f9f5a600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f54151a60 0x7f9f54157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:20.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.632+0000 7f9f5a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540be8a0 0x7f9f541590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:20.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.632+0000 7f9f5a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540cb970 con 0x7f9f540be8a0 2024-03-28T19:15:20.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.632+0000 7f9f5a600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540cb7f0 con 0x7f9f54151a60 2024-03-28T19:15:20.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.632+0000 7f9f5a600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540c8dd0 con 0x7f9f54148b30 2024-03-28T19:15:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.633+0000 7f9f59200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54148b30 0x7f9f54148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.633+0000 7f9f59200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54148b30 0x7f9f54148f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:48970/0 (socket says 172.21.15.27:48970) 2024-03-28T19:15:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.633+0000 7f9f59200640 1 -- 172.21.15.27:0/2716132819 learned_addr learned my addr 172.21.15.27:0/2716132819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:20.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.633+0000 7f9f53e00640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f54151a60 0x7f9f54157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.633+0000 7f9f59c00640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540be8a0 0x7f9f541590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.635+0000 7f9f59200640 1 -- 172.21.15.27:0/2716132819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f54151a60 msgr2=0x7f9f54157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.635+0000 7f9f59200640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f54151a60 0x7f9f54157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.635+0000 7f9f59200640 1 -- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540be8a0 msgr2=0x7f9f541590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.635+0000 7f9f59200640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540be8a0 0x7f9f541590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:20.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.635+0000 7f9f59200640 1 -- 172.21.15.27:0/2716132819 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54149b30 con 0x7f9f54148b30 2024-03-28T19:15:20.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.637+0000 7f9f59200640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54148b30 0x7f9f54148f10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9f48001980 tx=0x7f9f48001230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:20.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.637+0000 7f9f50c00640 1 -- 172.21.15.27:0/2716132819 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48032e30 con 0x7f9f54148b30 2024-03-28T19:15:20.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.638+0000 7f9f50c00640 1 -- 172.21.15.27:0/2716132819 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f48041020 con 0x7f9f54148b30 2024-03-28T19:15:20.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.638+0000 7f9f50c00640 1 -- 172.21.15.27:0/2716132819 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48007050 con 0x7f9f54148b30 2024-03-28T19:15:20.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.638+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f541496e0 con 0x7f9f54148b30 2024-03-28T19:15:20.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.638+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f54158420 con 0x7f9f54148b30 2024-03-28T19:15:20.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.640+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f54067dd0 con 0x7f9f54148b30 2024-03-28T19:15:20.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.643+0000 7f9f50c00640 1 -- 172.21.15.27:0/2716132819 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9f480071f0 con 0x7f9f54148b30 2024-03-28T19:15:20.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.645+0000 7f9f50c00640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f3406bc00 0x7f9f3406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:20.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.645+0000 7f9f50c00640 1 -- 172.21.15.27:0/2716132819 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f9f480ad680 con 0x7f9f54148b30 2024-03-28T19:15:20.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.645+0000 7f9f53e00640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f3406bc00 0x7f9f3406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:20.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.646+0000 7f9f53e00640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f3406bc00 0x7f9f3406e0c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9f54155050 tx=0x7f9f400017b0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:20.661+0000 7f9f50c00640 1 -- 172.21.15.27:0/2716132819 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9f4804b030 con 0x7f9f54148b30 2024-03-28T19:15:21.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.040+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f541498b0 con 0x7f9f54148b30 2024-03-28T19:15:21.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.041+0000 7f9f50c00640 1 -- 172.21.15.27:0/2716132819 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f9f541498b0 con 0x7f9f54148b30 2024-03-28T19:15:21.041 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:21.041 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:21.041 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 158 2024-03-28T19:15:21.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f3406bc00 msgr2=0x7f9f3406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:21.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f3406bc00 0x7f9f3406e0c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9f54155050 tx=0x7f9f400017b0 comp rx=0 tx=0).stop 2024-03-28T19:15:21.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54148b30 msgr2=0x7f9f54148f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:21.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54148b30 0x7f9f54148f10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9f48001980 tx=0x7f9f48001230 comp rx=0 tx=0).stop 2024-03-28T19:15:21.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 shutdown_connections 2024-03-28T19:15:21.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f3406bc00 0x7f9f3406e0c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:21.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f540be8a0 0x7f9f541590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:21.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f54151a60 0x7f9f54157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:21.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 --2- 172.21.15.27:0/2716132819 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f54148b30 0x7f9f54148f10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:21.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.044+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 >> 172.21.15.27:0/2716132819 conn(0x7f9f5406b2a0 msgr2=0x7f9f5406b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:21.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.045+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 shutdown_connections 2024-03-28T19:15:21.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:21.045+0000 7f9f5a600640 1 -- 172.21.15.27:0/2716132819 wait complete. 2024-03-28T19:15:21.067 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5870 2024-03-28T19:15:21.067 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.d 2024-03-28T19:15:29.068 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-03-28T19:15:29.068 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T19:15:29.068 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T19:15:29.078 INFO:tasks.ceph.mds.d:Started 2024-03-28T19:15:29.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:29.255 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T19:15:29.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.355+0000 7fdab5c00640 1 Processor -- start 2024-03-28T19:15:29.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.355+0000 7fdab5c00640 1 -- start start 2024-03-28T19:15:29.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.357+0000 7fdab5c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdab00c5e80 0x7fdab006cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:29.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.357+0000 7fdab5c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab006d5b0 con 0x7fdab00a5e50 2024-03-28T19:15:29.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.357+0000 7fdab5c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab006d890 con 0x7fdab00abdc0 2024-03-28T19:15:29.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.357+0000 7fdab5c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab006db70 con 0x7fdab00c5e80 2024-03-28T19:15:29.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.358+0000 7fdaaea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdab00c5e80 0x7fdab006cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:29.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.358+0000 7fdaaea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdab00c5e80 0x7fdab006cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58962/0 (socket says 172.21.15.27:58962) 2024-03-28T19:15:29.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.358+0000 7fdaa6000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdab00a5e50 0x7fdab00ab6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39900/0 (socket says 172.21.15.27:39900) 2024-03-28T19:15:29.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.358+0000 7fdaaea00640 1 -- 172.21.15.27:0/145497844 learned_addr learned my addr 172.21.15.27:0/145497844 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:29.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.358+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991354207 0 0) 0x7fdab006d5b0 con 0x7fdab00a5e50 2024-03-28T19:15:29.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.358+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda94003520 con 0x7fdab00a5e50 2024-03-28T19:15:29.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3065971560 0 0) 0x7fda94003520 con 0x7fdab00a5e50 2024-03-28T19:15:29.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 >> v2:172.21.15.27:3300/0 conn(0x7fdab00c5e80 msgr2=0x7fdab006cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:29.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 --2- 172.21.15.27:0/145497844 >> v2:172.21.15.27:3300/0 conn(0x7fdab00c5e80 0x7fdab006cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 >> v1:172.21.15.155:6789/0 conn(0x7fdab00abdc0 legacy=0x7fdab00c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:29.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab006de50 con 0x7fdab00a5e50 2024-03-28T19:15:29.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fda98001670 con 0x7fdab00a5e50 2024-03-28T19:15:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 >> v1:172.21.15.27:6789/0 conn(0x7fdab00a5e50 legacy=0x7fdab00ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda940039b0 0x7fda94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda940043b0 0x7fda94004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda94005120 0x7fda9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab006db70 con 0x7fda940039b0 2024-03-28T19:15:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9400c060 con 0x7fda940043b0 2024-03-28T19:15:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9400c260 con 0x7fda94005120 2024-03-28T19:15:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaaea00640 1 --2- 172.21.15.27:0/145497844 >> v2:172.21.15.27:3300/0 conn(0x7fdab00c5e80 0x7fdab006cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:15:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.359+0000 7fdaaea00640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda940039b0 0x7fda94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaa6000640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda940043b0 0x7fda94004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaaf400640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda94005120 0x7fda9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaaea00640 1 -- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda94005120 msgr2=0x7fda9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaaea00640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda94005120 0x7fda9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaaea00640 1 -- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda940043b0 msgr2=0x7fda94004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:29.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaaea00640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda940043b0 0x7fda94004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaaea00640 1 -- 172.21.15.27:0/145497844 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda94003520 con 0x7fda940039b0 2024-03-28T19:15:29.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaaf400640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda94005120 0x7fda9400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:15:29.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.360+0000 7fdaaf400640 1 -- 172.21.15.27:0/145497844 reap_dead start 2024-03-28T19:15:29.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.362+0000 7fdaaea00640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda940039b0 0x7fda94003d90 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000e4f0 tx=0x7fda94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b526f181cada1c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:29.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.362+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa000e140 con 0x7fda940039b0 2024-03-28T19:15:29.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.362+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa002cc20 con 0x7fda940039b0 2024-03-28T19:15:29.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.363+0000 7fdab5c00640 1 -- 172.21.15.27:0/145497844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda940039b0 msgr2=0x7fda94003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:29.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.363+0000 7fdaae000640 1 -- 172.21.15.27:0/145497844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa000e140 con 0x7fda940039b0 2024-03-28T19:15:29.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.363+0000 7fdab5c00640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda940039b0 0x7fda94003d90 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000e4f0 tx=0x7fda94005830 comp rx=0 tx=0).stop 2024-03-28T19:15:29.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.364+0000 7fdab5c00640 1 -- 172.21.15.27:0/145497844 shutdown_connections 2024-03-28T19:15:29.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.364+0000 7fdab5c00640 1 --2- 172.21.15.27:0/145497844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda940039b0 0x7fda94003d90 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.364+0000 7fdab5c00640 1 -- 172.21.15.27:0/145497844 >> 172.21.15.27:0/145497844 conn(0x7fdab006b9d0 msgr2=0x7fdab00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:29.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.364+0000 7fdab5c00640 1 -- 172.21.15.27:0/145497844 shutdown_connections 2024-03-28T19:15:29.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.364+0000 7fdab5c00640 1 -- 172.21.15.27:0/145497844 wait complete. 2024-03-28T19:15:29.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.367+0000 7fdab5c00640 1 Processor -- start 2024-03-28T19:15:29.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.367+0000 7fdab5c00640 1 -- start start 2024-03-28T19:15:29.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdab5c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda940039b0 0x7fdab014b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:29.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdab5c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab00a5530 0x7fdab0154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:29.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdab5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab01efc30 0x7fdab00a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:29.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdab5c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00c63e0 con 0x7fdab01efc30 2024-03-28T19:15:29.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdab5c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00c6260 con 0x7fdab00a5530 2024-03-28T19:15:29.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdab5c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab00ab6b0 con 0x7fda940039b0 2024-03-28T19:15:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdaaf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab01efc30 0x7fdab00a8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdaaf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab01efc30 0x7fdab00a8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58982/0 (socket says 172.21.15.27:58982) 2024-03-28T19:15:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.369+0000 7fdaaf400640 1 -- 172.21.15.27:0/3208028724 learned_addr learned my addr 172.21.15.27:0/3208028724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.370+0000 7fdaaea00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda940039b0 0x7fdab014b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.370+0000 7fdaa6000640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab00a5530 0x7fdab0154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.370+0000 7fdaaf400640 1 -- 172.21.15.27:0/3208028724 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda940039b0 msgr2=0x7fdab014b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.370+0000 7fdaaf400640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda940039b0 0x7fdab014b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.370+0000 7fdaaf400640 1 -- 172.21.15.27:0/3208028724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab00a5530 msgr2=0x7fdab0154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:29.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.370+0000 7fdaaf400640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab00a5530 0x7fdab0154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.370+0000 7fdaaf400640 1 -- 172.21.15.27:0/3208028724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab00a4cf0 con 0x7fdab01efc30 2024-03-28T19:15:29.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.371+0000 7fdaaf400640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab01efc30 0x7fdab00a8c20 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fdab006e500 tx=0x7fdaa800d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:29.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.372+0000 7fdaa7400640 1 -- 172.21.15.27:0/3208028724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8018070 con 0x7fdab01efc30 2024-03-28T19:15:29.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.372+0000 7fdaa7400640 1 -- 172.21.15.27:0/3208028724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa80035f0 con 0x7fdab01efc30 2024-03-28T19:15:29.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.372+0000 7fdaa7400640 1 -- 172.21.15.27:0/3208028724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80039e0 con 0x7fdab01efc30 2024-03-28T19:15:29.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.372+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab00a4f70 con 0x7fdab01efc30 2024-03-28T19:15:29.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.372+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab01ef430 con 0x7fdab01efc30 2024-03-28T19:15:29.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.374+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab004f400 con 0x7fdab01efc30 2024-03-28T19:15:29.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.375+0000 7fdaa7400640 1 -- 172.21.15.27:0/3208028724 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdaa801b070 con 0x7fdab01efc30 2024-03-28T19:15:29.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.377+0000 7fdaa7400640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda8c06bcd0 0x7fda8c06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:29.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.377+0000 7fdaa7400640 1 -- 172.21.15.27:0/3208028724 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7fdaa808ac20 con 0x7fdab01efc30 2024-03-28T19:15:29.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.378+0000 7fdaaea00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda8c06bcd0 0x7fda8c06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:29.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.379+0000 7fdaaea00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda8c06bcd0 0x7fda8c06e190 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0034000 tx=0x7fdaa002e000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:29.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.383+0000 7fdaa7400640 1 -- 172.21.15.27:0/3208028724 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdab00a4cf0 con 0x7fdab01efc30 2024-03-28T19:15:29.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.783+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdab00c6fd0 con 0x7fdab01efc30 2024-03-28T19:15:29.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.783+0000 7fdaa7400640 1 -- 172.21.15.27:0/3208028724 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fdaa8058ac0 con 0x7fdab01efc30 2024-03-28T19:15:29.783 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:29.783 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:29.783 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 159 2024-03-28T19:15:29.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda8c06bcd0 msgr2=0x7fda8c06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:29.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda8c06bcd0 0x7fda8c06e190 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0034000 tx=0x7fdaa002e000 comp rx=0 tx=0).stop 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab01efc30 msgr2=0x7fdab00a8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab01efc30 0x7fdab00a8c20 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fdab006e500 tx=0x7fdaa800d7d0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 shutdown_connections 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda8c06bcd0 0x7fda8c06e190 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdab01efc30 0x7fdab00a8c20 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab00a5530 0x7fdab0154e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 --2- 172.21.15.27:0/3208028724 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda940039b0 0x7fdab014b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 >> 172.21.15.27:0/3208028724 conn(0x7fdab006b9d0 msgr2=0x7fdab00729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 shutdown_connections 2024-03-28T19:15:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:29.787+0000 7fdab5c00640 1 -- 172.21.15.27:0/3208028724 wait complete. 2024-03-28T19:15:29.808 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-03-28T19:15:29.808 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2024-03-28T19:15:33.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.065+0000 7fda41c00640 1 Processor -- start 2024-03-28T19:15:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.066+0000 7fda41c00640 1 -- start start 2024-03-28T19:15:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.068+0000 7fda41c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0c58c0 0x7fda3c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.068+0000 7fda41c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda3c0c8ed0 0x7fda3c1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.068+0000 7fda41c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c156230 con 0x7fda3c0be5d0 2024-03-28T19:15:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.068+0000 7fda41c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c158bd0 con 0x7fda3c0c8ed0 2024-03-28T19:15:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.068+0000 7fda41c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c158eb0 con 0x7fda3c0c58c0 2024-03-28T19:15:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.068+0000 7fda3b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda3c0c8ed0 0x7fda3c1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.069+0000 7fda3b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda3c0c8ed0 0x7fda3c1555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58984/0 (socket says 172.21.15.27:58984) 2024-03-28T19:15:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.069+0000 7fda3b400640 1 -- 172.21.15.27:0/3524799695 learned_addr learned my addr 172.21.15.27:0/3524799695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.069+0000 7fda3be00640 1 --2- 172.21.15.27:0/3524799695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0c58c0 0x7fda3c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.069+0000 7fda3b400640 1 -- 172.21.15.27:0/3524799695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0c58c0 msgr2=0x7fda3c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.069+0000 7fda3b400640 1 --2- 172.21.15.27:0/3524799695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0c58c0 0x7fda3c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.069+0000 7fda3b400640 1 -- 172.21.15.27:0/3524799695 >> v1:172.21.15.27:6789/0 conn(0x7fda3c0be5d0 legacy=0x7fda3c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.069+0000 7fda3b400640 1 -- 172.21.15.27:0/3524799695 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c159190 con 0x7fda3c0c8ed0 2024-03-28T19:15:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.071+0000 7fda3b400640 1 --2- 172.21.15.27:0/3524799695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c8ed0 0x7fda3c1555e0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fda28009870 tx=0x7fda2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e6dd5ab8034e7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:34.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.071+0000 7fda3aa00640 1 -- 172.21.15.27:0/3524799695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28035020 con 0x7fda3c0c8ed0 2024-03-28T19:15:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.071+0000 7fda3aa00640 1 -- 172.21.15.27:0/3524799695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda28031640 con 0x7fda3c0c8ed0 2024-03-28T19:15:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.071+0000 7fda3aa00640 1 -- 172.21.15.27:0/3524799695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2802d040 con 0x7fda3c0c8ed0 2024-03-28T19:15:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.072+0000 7fda41c00640 1 -- 172.21.15.27:0/3524799695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c8ed0 msgr2=0x7fda3c1555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.072+0000 7fda41c00640 1 --2- 172.21.15.27:0/3524799695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c8ed0 0x7fda3c1555e0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fda28009870 tx=0x7fda2802b8f0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.072+0000 7fda41c00640 1 -- 172.21.15.27:0/3524799695 shutdown_connections 2024-03-28T19:15:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.072+0000 7fda41c00640 1 --2- 172.21.15.27:0/3524799695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c8ed0 0x7fda3c1555e0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.072+0000 7fda41c00640 1 --2- 172.21.15.27:0/3524799695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0c58c0 0x7fda3c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.072+0000 7fda41c00640 1 -- 172.21.15.27:0/3524799695 >> 172.21.15.27:0/3524799695 conn(0x7fda3c06b6f0 msgr2=0x7fda3c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.073+0000 7fda41c00640 1 -- 172.21.15.27:0/3524799695 shutdown_connections 2024-03-28T19:15:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.073+0000 7fda41c00640 1 -- 172.21.15.27:0/3524799695 wait complete. 2024-03-28T19:15:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.075+0000 7fda41c00640 1 Processor -- start 2024-03-28T19:15:34.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.075+0000 7fda41c00640 1 -- start start 2024-03-28T19:15:34.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda41c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0be5d0 0x7fda3c0aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda41c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c58c0 0x7fda3c1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda41c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda3c0c8ed0 0x7fda3c151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda41c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0b0820 con 0x7fda3c0c58c0 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda41c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c159750 con 0x7fda3c0c8ed0 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda41c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0b09a0 con 0x7fda3c0be5d0 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda3b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c58c0 0x7fda3c1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda3b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c58c0 0x7fda3c1514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58986/0 (socket says 172.21.15.27:58986) 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.076+0000 7fda3b400640 1 -- 172.21.15.27:0/1025778505 learned_addr learned my addr 172.21.15.27:0/1025778505 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.077+0000 7fda3be00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0be5d0 0x7fda3c0aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.077+0000 7fda41200640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda3c0c8ed0 0x7fda3c151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.077+0000 7fda3b400640 1 -- 172.21.15.27:0/1025778505 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0be5d0 msgr2=0x7fda3c0aca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.077+0000 7fda3b400640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0be5d0 0x7fda3c0aca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.077+0000 7fda3b400640 1 -- 172.21.15.27:0/1025778505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda3c0c8ed0 msgr2=0x7fda3c151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.077+0000 7fda3b400640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda3c0c8ed0 0x7fda3c151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.077+0000 7fda3b400640 1 -- 172.21.15.27:0/1025778505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0c4360 con 0x7fda3c0c58c0 2024-03-28T19:15:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.078+0000 7fda3b400640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c58c0 0x7fda3c1514e0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fda280090b0 tx=0x7fda280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.079+0000 7fda33e00640 1 -- 172.21.15.27:0/1025778505 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28040070 con 0x7fda3c0c58c0 2024-03-28T19:15:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.079+0000 7fda33e00640 1 -- 172.21.15.27:0/1025778505 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda28039c10 con 0x7fda3c0c58c0 2024-03-28T19:15:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.079+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c0c4640 con 0x7fda3c0c58c0 2024-03-28T19:15:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.079+0000 7fda33e00640 1 -- 172.21.15.27:0/1025778505 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28035040 con 0x7fda3c0c58c0 2024-03-28T19:15:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.079+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c0acfb0 con 0x7fda3c0c58c0 2024-03-28T19:15:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.081+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c0c6eb0 con 0x7fda3c0c58c0 2024-03-28T19:15:34.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.082+0000 7fda33e00640 1 -- 172.21.15.27:0/1025778505 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fda2804b060 con 0x7fda3c0c58c0 2024-03-28T19:15:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.083+0000 7fda33e00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda1c06bc00 0x7fda1c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.084+0000 7fda33e00640 1 -- 172.21.15.27:0/1025778505 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7fda280ace90 con 0x7fda3c0c58c0 2024-03-28T19:15:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.084+0000 7fda3be00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda1c06bc00 0x7fda1c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.085+0000 7fda3be00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda1c06bc00 0x7fda1c06e0c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fda2c004090 tx=0x7fda2c017040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:34.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.089+0000 7fda33e00640 1 -- 172.21.15.27:0/1025778505 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fda3c0c6eb0 con 0x7fda3c0c58c0 2024-03-28T19:15:34.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.507+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda3c069640 con 0x7fda3c0c58c0 2024-03-28T19:15:34.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.508+0000 7fda33e00640 1 -- 172.21.15.27:0/1025778505 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fda280ae020 con 0x7fda3c0c58c0 2024-03-28T19:15:34.508 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:34.508 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:34.508 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 159 2024-03-28T19:15:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda1c06bc00 msgr2=0x7fda1c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda1c06bc00 0x7fda1c06e0c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fda2c004090 tx=0x7fda2c017040 comp rx=0 tx=0).stop 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c58c0 msgr2=0x7fda3c1514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c58c0 0x7fda3c1514e0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fda280090b0 tx=0x7fda280046c0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 shutdown_connections 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fda1c06bc00 0x7fda1c06e0c0 secure :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fda2c004090 tx=0x7fda2c017040 comp rx=0 tx=0).stop 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda3c0c8ed0 0x7fda3c151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3c0c58c0 0x7fda3c1514e0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 --2- 172.21.15.27:0/1025778505 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda3c0be5d0 0x7fda3c0aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.512+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 >> 172.21.15.27:0/1025778505 conn(0x7fda3c06b6f0 msgr2=0x7fda3c0c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:34.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.513+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 shutdown_connections 2024-03-28T19:15:34.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.514+0000 7fda41c00640 1 -- 172.21.15.27:0/1025778505 wait complete. 2024-03-28T19:15:34.538 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-03-28T19:15:34.538 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:15:34.538 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:15:34.566 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:15:34.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:34.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:15:34.658 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:15:34.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.842+0000 7f71cca00640 1 Processor -- start 2024-03-28T19:15:34.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.842+0000 7f71cca00640 1 -- start start 2024-03-28T19:15:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.843+0000 7f71cca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f71c80be5d0 0x7f71c80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.843+0000 7f71cca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c8158bd0 con 0x7f71c80c58c0 2024-03-28T19:15:34.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.843+0000 7f71cca00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c8158eb0 con 0x7f71c80c8ed0 2024-03-28T19:15:34.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.843+0000 7f71cca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c8159190 con 0x7f71c80be5d0 2024-03-28T19:15:34.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.843+0000 7f71c7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f71c80c58c0 0x7f71c81555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39910/0 (socket says 172.21.15.27:39910) 2024-03-28T19:15:34.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.843+0000 7f71c7400640 1 -- 172.21.15.27:0/2892829072 learned_addr learned my addr 172.21.15.27:0/2892829072 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:34.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.844+0000 7f71c7e00640 1 --2- 172.21.15.27:0/2892829072 >> v2:172.21.15.155:3300/0 conn(0x7f71c80be5d0 0x7f71c80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.844+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649196977 0 0) 0x7f71c8158bd0 con 0x7f71c80c58c0 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.844+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ac003520 con 0x7f71c80c58c0 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.844+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478519585 0 0) 0x7f71ac003520 con 0x7f71c80c58c0 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 >> v2:172.21.15.155:3300/0 conn(0x7f71c80be5d0 msgr2=0x7f71c80c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 --2- 172.21.15.27:0/2892829072 >> v2:172.21.15.155:3300/0 conn(0x7f71c80be5d0 0x7f71c80c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 >> v1:172.21.15.155:6789/0 conn(0x7f71c80c8ed0 legacy=0x7f71c8155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8159470 con 0x7f71c80c58c0 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f71b8001670 con 0x7f71c80c58c0 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 >> v1:172.21.15.27:6789/0 conn(0x7f71c80c58c0 legacy=0x7f71c81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71ac0039b0 0x7f71ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0052e0 0x7f71ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8159190 con 0x7f71ac004800 2024-03-28T19:15:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac00c1b0 con 0x7f71ac0052e0 2024-03-28T19:15:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac00c3b0 con 0x7f71ac0039b0 2024-03-28T19:15:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.845+0000 7f71c6a00640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.846+0000 7f71c7e00640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0052e0 0x7f71ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.846+0000 7f71c6a00640 1 -- 172.21.15.27:0/2892829072 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71ac0039b0 msgr2=0x7f71ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:34.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.846+0000 7f71c6a00640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71ac0039b0 0x7f71ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.846+0000 7f71c6a00640 1 -- 172.21.15.27:0/2892829072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0052e0 msgr2=0x7f71ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:34.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.846+0000 7f71c6a00640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71ac0052e0 0x7f71ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.846+0000 7f71c6a00640 1 -- 172.21.15.27:0/2892829072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8159470 con 0x7f71ac004800 2024-03-28T19:15:34.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.846+0000 7f71c7e00640 1 -- 172.21.15.27:0/2892829072 reap_dead start 2024-03-28T19:15:34.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.847+0000 7f71c6a00640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71ac004c80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f71c80c9300 tx=0x7f71ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0a59663ac2ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.848+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4001350 con 0x7f71ac004800 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.848+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b40052f0 con 0x7f71ac004800 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.848+0000 7f71c6000640 1 -- 172.21.15.27:0/2892829072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4003830 con 0x7f71ac004800 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.849+0000 7f71cca00640 1 -- 172.21.15.27:0/2892829072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 msgr2=0x7f71ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.849+0000 7f71cca00640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71ac004c80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f71c80c9300 tx=0x7f71ac005830 comp rx=0 tx=0).stop 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.849+0000 7f71cca00640 1 -- 172.21.15.27:0/2892829072 shutdown_connections 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.849+0000 7f71cca00640 1 --2- 172.21.15.27:0/2892829072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71ac004c80 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.849+0000 7f71cca00640 1 -- 172.21.15.27:0/2892829072 >> 172.21.15.27:0/2892829072 conn(0x7f71c806b6f0 msgr2=0x7f71c80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:34.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.849+0000 7f71cca00640 1 -- 172.21.15.27:0/2892829072 shutdown_connections 2024-03-28T19:15:34.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.849+0000 7f71cca00640 1 -- 172.21.15.27:0/2892829072 wait complete. 2024-03-28T19:15:34.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.851+0000 7f71cca00640 1 Processor -- start 2024-03-28T19:15:34.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.851+0000 7f71cca00640 1 -- start start 2024-03-28T19:15:34.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.852+0000 7f71cca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71c81ef280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.852+0000 7f71cca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c8152860 0x7f71c80ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.852+0000 7f71cca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71c80c1960 0x7f71c81eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.852+0000 7f71cca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c80b09a0 con 0x7f71ac004800 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.852+0000 7f71cca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c80b0820 con 0x7f71c8152860 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.852+0000 7f71cca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c80ad7d0 con 0x7f71c80c1960 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.852+0000 7f71c7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71c81ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71c81ef280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59002/0 (socket says 172.21.15.27:59002) 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7400640 1 -- 172.21.15.27:0/1438039118 learned_addr learned my addr 172.21.15.27:0/1438039118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c6a00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c8152860 0x7f71c80ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7e00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71c80c1960 0x7f71c81eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7400640 1 -- 172.21.15.27:0/1438039118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71c80c1960 msgr2=0x7f71c81eb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7400640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71c80c1960 0x7f71c81eb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7400640 1 -- 172.21.15.27:0/1438039118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c8152860 msgr2=0x7f71c80ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7400640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c8152860 0x7f71c80ac790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7400640 1 -- 172.21.15.27:0/1438039118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c81ef7c0 con 0x7f71ac004800 2024-03-28T19:15:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.853+0000 7f71c7400640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71c81ef280 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f71b800c760 tx=0x7f71b800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:34.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.854+0000 7f71bf400640 1 -- 172.21.15.27:0/1438039118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8018070 con 0x7f71ac004800 2024-03-28T19:15:34.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.854+0000 7f71bf400640 1 -- 172.21.15.27:0/1438039118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b80072d0 con 0x7f71ac004800 2024-03-28T19:15:34.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.854+0000 7f71bf400640 1 -- 172.21.15.27:0/1438039118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b80047e0 con 0x7f71ac004800 2024-03-28T19:15:34.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.854+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c80c3bb0 con 0x7f71ac004800 2024-03-28T19:15:34.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.854+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c81eea80 con 0x7f71ac004800 2024-03-28T19:15:34.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.855+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c80c40d0 con 0x7f71ac004800 2024-03-28T19:15:34.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.857+0000 7f71bf400640 1 -- 172.21.15.27:0/1438039118 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f71b8004980 con 0x7f71ac004800 2024-03-28T19:15:34.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.858+0000 7f71bf400640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71a806bbd0 0x7f71a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:34.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.858+0000 7f71bf400640 1 -- 172.21.15.27:0/1438039118 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f71b808aa70 con 0x7f71ac004800 2024-03-28T19:15:34.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.858+0000 7f71c6a00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71a806bbd0 0x7f71a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:34.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.859+0000 7f71c6a00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71a806bbd0 0x7f71a806e090 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f71b4033000 tx=0x7f71b4009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:34.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:34.863+0000 7f71bf400640 1 -- 172.21.15.27:0/1438039118 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f71c80c40d0 con 0x7f71ac004800 2024-03-28T19:15:35.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.218+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71c8069640 con 0x7f71ac004800 2024-03-28T19:15:35.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.219+0000 7f71bf400640 1 -- 172.21.15.27:0/1438039118 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f71b8004c70 con 0x7f71ac004800 2024-03-28T19:15:35.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:35.219 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:35.219 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 159 2024-03-28T19:15:35.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71a806bbd0 msgr2=0x7f71a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71a806bbd0 0x7f71a806e090 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f71b4033000 tx=0x7f71b4009030 comp rx=0 tx=0).stop 2024-03-28T19:15:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 msgr2=0x7f71c81ef280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71c81ef280 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f71b800c760 tx=0x7f71b800cc20 comp rx=0 tx=0).stop 2024-03-28T19:15:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 shutdown_connections 2024-03-28T19:15:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71a806bbd0 0x7f71a806e090 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71c80c1960 0x7f71c81eb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c8152860 0x7f71c80ac790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:35.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 --2- 172.21.15.27:0/1438039118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71ac004800 0x7f71c81ef280 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:35.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 >> 172.21.15.27:0/1438039118 conn(0x7f71c806b6f0 msgr2=0x7f71c8155410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:35.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 shutdown_connections 2024-03-28T19:15:35.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:35.222+0000 7f71cca00640 1 -- 172.21.15.27:0/1438039118 wait complete. 2024-03-28T19:15:35.241 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:15:37.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:37.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.485+0000 7fb8bc000640 1 Processor -- start 2024-03-28T19:15:37.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.486+0000 7fb8bc000640 1 -- start start 2024-03-28T19:15:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.487+0000 7fb8bc000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8b40be630 0x7fb8b40c5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.487+0000 7fb8bc000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb8b40c5e40 0x7fb8b4159150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.487+0000 7fb8bc000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b40c7910 con 0x7fb8b4158580 2024-03-28T19:15:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.487+0000 7fb8bc000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b40c76e0 con 0x7fb8b40be630 2024-03-28T19:15:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.487+0000 7fb8bc000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b40c6dd0 con 0x7fb8b40c5e40 2024-03-28T19:15:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.487+0000 7fb8b8c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb8b40be630 0x7fb8b40c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.488+0000 7fb8b9600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb8b4158580 0x7fb8b41560c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43376/0 (socket says 172.21.15.27:43376) 2024-03-28T19:15:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.488+0000 7fb8b9600640 1 -- 172.21.15.27:0/992151623 learned_addr learned my addr 172.21.15.27:0/992151623 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.488+0000 7fb8ba000640 1 --2- 172.21.15.27:0/992151623 >> v2:172.21.15.155:3300/0 conn(0x7fb8b40c5e40 0x7fb8b4159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.488+0000 7fb8b8c00640 1 -- 172.21.15.27:0/992151623 >> v2:172.21.15.155:3300/0 conn(0x7fb8b40c5e40 msgr2=0x7fb8b4159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.488+0000 7fb8b8c00640 1 --2- 172.21.15.27:0/992151623 >> v2:172.21.15.155:3300/0 conn(0x7fb8b40c5e40 0x7fb8b4159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.488+0000 7fb8b8c00640 1 -- 172.21.15.27:0/992151623 >> v1:172.21.15.27:6789/0 conn(0x7fb8b4158580 legacy=0x7fb8b41560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.488+0000 7fb8b8c00640 1 -- 172.21.15.27:0/992151623 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b40c7bf0 con 0x7fb8b40be630 2024-03-28T19:15:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.490+0000 7fb8b8c00640 1 --2- 172.21.15.27:0/992151623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40be630 0x7fb8b40c5900 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40c8aa0 tx=0x7fb8a00340e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600c2ccafe723ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.490+0000 7fb8b3e00640 1 -- 172.21.15.27:0/992151623 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0041070 con 0x7fb8b40be630 2024-03-28T19:15:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.490+0000 7fb8b3e00640 1 -- 172.21.15.27:0/992151623 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a003c440 con 0x7fb8b40be630 2024-03-28T19:15:37.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.490+0000 7fb8b3e00640 1 -- 172.21.15.27:0/992151623 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a003c810 con 0x7fb8b40be630 2024-03-28T19:15:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.491+0000 7fb8bc000640 1 -- 172.21.15.27:0/992151623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40be630 msgr2=0x7fb8b40c5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.491+0000 7fb8bc000640 1 --2- 172.21.15.27:0/992151623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40be630 0x7fb8b40c5900 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb8b40c8aa0 tx=0x7fb8a00340e0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.491+0000 7fb8bc000640 1 -- 172.21.15.27:0/992151623 shutdown_connections 2024-03-28T19:15:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.491+0000 7fb8bc000640 1 --2- 172.21.15.27:0/992151623 >> v2:172.21.15.155:3300/0 conn(0x7fb8b40c5e40 0x7fb8b4159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.491+0000 7fb8bc000640 1 --2- 172.21.15.27:0/992151623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40be630 0x7fb8b40c5900 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.491+0000 7fb8bc000640 1 -- 172.21.15.27:0/992151623 >> 172.21.15.27:0/992151623 conn(0x7fb8b40caf80 msgr2=0x7fb8b40cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.491+0000 7fb8bc000640 1 -- 172.21.15.27:0/992151623 shutdown_connections 2024-03-28T19:15:37.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.491+0000 7fb8bc000640 1 -- 172.21.15.27:0/992151623 wait complete. 2024-03-28T19:15:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.494+0000 7fb8bc000640 1 Processor -- start 2024-03-28T19:15:37.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.494+0000 7fb8bc000640 1 -- start start 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8bc000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8b40be630 0x7fb8b40c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8bc000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40c5e40 0x7fb8b40bef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8bc000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8b4158580 0x7fb8b40bf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8bc000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b40c81b0 con 0x7fb8b40c5e40 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8bc000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4158cc0 con 0x7fb8b4158580 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8bc000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b4158960 con 0x7fb8b40be630 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8b8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40c5e40 0x7fb8b40bef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8b8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40c5e40 0x7fb8b40bef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54252/0 (socket says 172.21.15.27:54252) 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.496+0000 7fb8b8c00640 1 -- 172.21.15.27:0/459938467 learned_addr learned my addr 172.21.15.27:0/459938467 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.497+0000 7fb8b9600640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8b40be630 0x7fb8b40c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.497+0000 7fb8ba000640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8b4158580 0x7fb8b40bf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.497+0000 7fb8b8c00640 1 -- 172.21.15.27:0/459938467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8b40be630 msgr2=0x7fb8b40c4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.497+0000 7fb8b8c00640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8b40be630 0x7fb8b40c4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.497+0000 7fb8b8c00640 1 -- 172.21.15.27:0/459938467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8b4158580 msgr2=0x7fb8b40bf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.497+0000 7fb8b8c00640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8b4158580 0x7fb8b40bf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.497+0000 7fb8b8c00640 1 -- 172.21.15.27:0/459938467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b414a320 con 0x7fb8b40c5e40 2024-03-28T19:15:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.497+0000 7fb8b8c00640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40c5e40 0x7fb8b40bef10 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0000c00 tx=0x7fb8a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.498+0000 7fb8b1600640 1 -- 172.21.15.27:0/459938467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0041040 con 0x7fb8b40c5e40 2024-03-28T19:15:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.498+0000 7fb8b1600640 1 -- 172.21.15.27:0/459938467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a004c070 con 0x7fb8b40c5e40 2024-03-28T19:15:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.498+0000 7fb8b1600640 1 -- 172.21.15.27:0/459938467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a003abd0 con 0x7fb8b40c5e40 2024-03-28T19:15:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.498+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b414eff0 con 0x7fb8b40c5e40 2024-03-28T19:15:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.498+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b40c4c80 con 0x7fb8b40c5e40 2024-03-28T19:15:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.499+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b404f400 con 0x7fb8b40c5e40 2024-03-28T19:15:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.500+0000 7fb8b1600640 1 -- 172.21.15.27:0/459938467 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb8a0051460 con 0x7fb8b40c5e40 2024-03-28T19:15:37.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.502+0000 7fb8b1600640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb89406bbd0 0x7fb89406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.502+0000 7fb8b1600640 1 -- 172.21.15.27:0/459938467 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7fb8a00b6340 con 0x7fb8b40c5e40 2024-03-28T19:15:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.503+0000 7fb8b9600640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb89406bbd0 0x7fb89406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.504+0000 7fb8b9600640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb89406bbd0 0x7fb89406e090 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8000f60 tx=0x7fb8a8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:37.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.508+0000 7fb8b1600640 1 -- 172.21.15.27:0/459938467 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb8a003ce20 con 0x7fb8b40c5e40 2024-03-28T19:15:37.594 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 0% (94/12435) Updating files: 1% (125/12435) Updating files: 1% (166/12435) Updating files: 2% (249/12435) Updating files: 2% (250/12435) Updating files: 2% (305/12435) Updating files: 3% (374/12435) Updating files: 3% (379/12435) Updating files: 3% (437/12435) Updating files: 4% (498/12435) Updating files: 4% (508/12435) Updating files: 4% (578/12435) Updating files: 5% (622/12435) Updating files: 5% (625/12435) Updating files: 5% (693/12435) Updating files: 6% (747/12435) Updating files: 6% (755/12435) Updating files: 6% (819/12435) Updating files: 7% (871/12435) Updating files: 7% (886/12435) Updating files: 7% (935/12435) Updating files: 7% (990/12435) Updating files: 8% (995/12435) Updating files: 8% (1027/12435) Updating files: 8% (1088/12435) Updating files: 9% (1120/12435) Updating files: 9% (1128/12435) Updating files: 9% (1183/12435) Updating files: 9% (1232/12435) Updating files: 10% (1244/12435) Updating files: 10% (1293/12435) Updating files: 10% (1347/12435) Updating files: 11% (1368/12435) Updating files: 11% (1401/12435) Updating files: 11% (1459/12435) Updating files: 12% (1493/12435) Updating files: 12% (1512/12435) Updating files: 12% (1558/12435) Updating files: 12% (1597/12435) Updating files: 13% (1617/12435) Updating files: 13% (1637/12435) Updating files: 13% (1680/12435) Updating files: 13% (1725/12435) Updating files: 14% (1741/12435) Updating files: 14% (1769/12435) Updating files: 14% (1807/12435) Updating files: 14% (1851/12435) Updating files: 15% (1866/12435) Updating files: 15% (1897/12435) Updating files: 15% (1942/12435) Updating files: 15% (1986/12435) Updating files: 16% (1990/12435) Updating files: 16% (2029/12435) Updating files: 16% (2071/12435) Updating files: 16% (2110/12435) Updating files: 17% (2114/12435) Updating files: 17% (2145/12435) Updating files: 17% (2185/12435) Updating files: 17% (2227/12435) Updating files: 18% (2239/12435) Updating files: 18% (2263/12435) Updating files: 18% (2300/12435) Updating files: 18% (2339/12435) Updating files: 19% (2363/12435) Updating files: 19% (2375/12435) Updating files: 19% (2412/12435) Updating files: 19% (2454/12435) Updating files: 20% (2487/12435) Updating files: 20% (2499/12435) Updating files: 20% (2557/12435) Updating files: 20% (2608/12435) Updating files: 21% (2612/12435) Updating files: 21% (2663/12435) Updating files: 21% (2718/12435) Updating files: 22% (2736/12435) Updating files: 22% (2770/12435) Updating files: 22% (2821/12435) Updating files: 23% (2861/12435) Updating files: 23% (2873/12435) Updating files: 23% (2922/12435) Updating files: 23% (2971/12435) Updating files: 24% (2985/12435) Updating files: 24% (3023/12435) Updating files: 24% (3083/12435) Updating files: 24% (3098/12435) Updating files: 25% (3109/12435) Updating files: 25% (3142/12435) Updating files: 25% (3213/12435) Updating files: 26% (3234/12435) Updating files: 26% (3301/12435) Updating files: 26% (3337/12435) Updating files: 27% (3358/12435) Updating files: 27% (3395/12435) Updating files: 27% (3466/12435) Updating files: 28% (3482/12435) Updating files: 28% (3524/12435) Updating files: 28% (3592/12435) Updating files: 29% (3607/12435) Updating files: 29% (3647/12435) Updating files: 29% (3702/12435) Updating files: 30% (3731/12435) Updating files: 30% (3754/12435) Updating files: 30% (3804/12435) Updating files: 31% (3855/12435) Updating files: 31% (3858/12435) Updating files: 31% (3909/12435) Updating files: 31% (3961/12435) Updating files: 32% (3980/12435) Updating files: 32% (4011/12435) Updating files: 32% (4057/12435) Updating files: 33% (4104/12435) Updating files: 33% (4109/12435) Updating files: 33% (4155/12435) Updating files: 33% (4202/12435) Updating files: 34% (4228/12435) Updating files: 34% (4247/12435) Updating files: 34% (4292/12435) Updating files: 34% (4340/12435) Updating files: 35% (4353/12435) Updating files: 35% (4384/12435) Updating files: 35% (4427/12435) Updating files: 35% (4472/12435) Updating files: 36% (4477/12435) Updating files: 36% (4518/12435) Updating files: 36% (4570/12435) Updating files: 37% (4601/12435) Updating files: 37% (4662/12435) Updating files: 38% (4726/12435) Updating files: 38% (4769/12435) Updating files: 39% (4850/12435) Updating files: 39% (4856/12435) Updating files: 39% (4931/12435) Updating files: 40% (4974/12435) Updating files: 40% (5019/12435) Updating files: 40% (5081/12435) Updating files: 41% (5099/12435) Updating files: 41% (5166/12435) Updating files: 42% (5223/12435) Updating files: 42% (5236/12435) Updating files: 42% (5287/12435) Updating files: 42% (5327/12435) Updating files: 43% (5348/12435) Updating files: 43% (5369/12435) Updating files: 43% (5435/12435) Updating files: 44% (5472/12435) Updating files: 44% (5499/12435) Updating files: 44% (5578/12435) Updating files: 45% (5596/12435) Updating files: 45% (5658/12435) Updating files: 46% (5721/12435) Updating files: 46% (5744/12435) Updating files: 46% (5837/12435) Updating files: 47% (5845/12435) Updating files: 47% (5934/12435) Updating files: 48% (5969/12435) Updating files: 48% (6028/12435) Updating files: 49% (6094/12435) Updating files: 49% (6106/12435) Updating files: 49% (6173/12435) Updating files: 50% (6218/12435) Updating files: 50% (6223/12435) Updating files: 50% (6275/12435) Updating files: 50% (6340/12435) Updating files: 51% (6342/12435) Updating files: 51% (6409/12435) Updating files: 52% (6467/12435) Updating files: 52% (6488/12435) Updating files: 52% (6566/12435) Updating files: 53% (6591/12435) Updating files: 53% (6631/12435) Updating files: 53% (6686/12435) Updating files: 54% (6715/12435) Updating files: 54% (6735/12435) Updating files: 54% (6801/12435) Updating files: 55% (6840/12435) Updating files: 55% (6859/12435) Updating files: 55% (6922/12435) Updating files: 56% (6964/12435) Updating files: 56% (6978/12435) Updating files: 56% (7022/12435) Updating files: 56% (7081/12435) Updating files: 57% (7088/12435) Updating files: 57% (7142/12435) Updating files: 57% (7212/12435) Updating files: 58% (7213/12435) Updating files: 58% (7287/12435) Updating files: 59% (7337/12435) Updating files: 59% (7344/12435) Updating files: 59% (7422/12435) Updating files: 60% (7461/12435) Updating files: 60% (7505/12435) Updating files: 61% (7586/12435) Updating files: 61% (7664/12435) Updating files: 62% (7710/12435) Updating files: 62% (7758/12435) Updating files: 62% (7832/12435) Updating files: 63% (7835/12435) Updating files: 63% (7920/12435) Updating files: 64% (7959/12435) Updating files: 64% (7997/12435) Updating files: 64% (8069/12435) Updating files: 65% (8083/12435) Updating files: 65% (8124/12435) Updating files: 65% (8186/12435) Updating files: 66% (8208/12435) Updating files: 66% (8250/12435) Updating files: 66% (8303/12435) Updating files: 67% (8332/12435) Updating files: 67% (8354/12435) Updating files: 67% (8403/12435) Updating files: 67% (8404/12435) Updating files: 68% (8456/12435) Updating files: 68% (8457/12435) Updating files: 68% (8492/12435) Updating files: 68% (8528/12435) Updating files: 68% (8568/12435) Updating files: 69% (8581/12435) Updating files: 69% (8608/12435) Updating files: 69% (8657/12435) Updating files: 69% (8692/12435) Updating files: 70% (8705/12435) Updating files: 70% (8738/12435) Updating files: 70% (8783/12435) Updating files: 70% (8820/12435) Updating files: 71% (8829/12435) Updating files: 71% (8856/12435) Updating files: 71% (8899/12435) Updating files: 71% (8933/12435) Updating files: 72% (8954/12435) Updating files: 72% (8964/12435) Updating files: 72% (8997/12435) Updating files: 72% (9028/12435) Updating files: 72% (9073/12435) Updating files: 73% (9078/12435) Updating files: 73% (9109/12435) Updating files: 73% (9165/12435) Updating files: 74% (9202/12435) Updating files: 74% (9236/12435) Updating files: 74% (9279/12435) Updating files: 74% (9324/12435) Updating files: 75% (9327/12435) Updating files: 75% (9368/12435) Updating files: 75% (9383/12435) Updating files: 75% (9426/12435) Updating files: 76% (9451/12435) Updating files: 76% (9464/12435) Updating files: 76% (9512/12435) Updating files: 76% (9559/12435) Updating files: 77% (9575/12435) Updating files: 77% (9612/12435) Updating files: 77% (9658/12435) Updating files: 78% (9700/12435) Updating files: 78% (9704/12435) Updating files: 78% (9746/12435) Updating files: 78% (9820/12435) Updating files: 79% (9824/12435) Updating files: 79% (9859/12435) Updating files: 79% (9885/12435) Updating files: 79% (9914/12435) Updating files: 79% (9928/12435) Updating files: 79% (9936/12435) Updating files: 80% (9948/12435) Updating files: 80% (9985/12435) Updating files: 80% (10041/12435) Updating files: 81% (10073/12435) Updating files: 81% (10082/12435) Updating files: 81% (10131/12435) Updating files: 81% (10190/12435) Updating files: 82% (10197/12435) Updating files: 82% (10237/12435) Updating files: 82% (10284/12435) Updating files: 83% (10322/12435) Updating files: 83% (10336/12435) Updating files: 83% (10399/12435) Updating files: 84% (10446/12435) Updating files: 84% (10451/12435) Updating files: 84% (10522/12435) Updating files: 85% (10570/12435) Updating files: 85% (10589/12435) Updating files: 85% (10651/12435) Updating files: 86% (10695/12435) Updating files: 86% (10705/12435) Updating files: 86% (10754/12435) Updating files: 86% (10814/12435) Updating files: 87% (10819/12435) Updating files: 87% (10875/12435) Updating files: 87% (10934/12435) Updating files: 88% (10943/12435) Updating files: 88% (10996/12435) Updating files: 89% (11068/12435) Updating files: 89% (11075/12435) Updating files: 89% (11159/12435) Updating files: 90% (11192/12435) Updating files: 90% (11246/12435) Updating files: 91% (11316/12435) Updating files: 91% (11350/12435) Updating files: 92% (11441/12435) Updating files: 92% (11449/12435) Updating files: 92% (11538/12435) Updating files: 93% (11565/12435) Updating files: 93% (11626/12435) Updating files: 94% (11689/12435) Updating files: 94% (11715/12435) Updating files: 94% (11805/12435) Updating files: 95% (11814/12435) Updating files: 95% (11904/12435) Updating files: 96% (11938/12435) Updating files: 96% (12005/12435) Updating files: 97% (12062/12435) Updating files: 97% (12107/12435) Updating files: 98% (12187/12435) Updating files: 98% (12197/12435) Updating files: 98% (12294/12435) Updating files: 99% (12311/12435) Updating files: 99% (12344/12435) Updating files: 99% (12431/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-03-28T19:15:37.699 INFO:tasks.workunit.client.0.smithi027.stderr:+ trap - EXIT 2024-03-28T19:15:37.699 INFO:tasks.workunit.client.0.smithi027.stderr:+ cd ceph 2024-03-28T19:15:37.705 INFO:tasks.workunit.client.0.smithi027.stderr:++ seq 1 90 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:+ versions='1 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:2 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:3 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:4 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:5 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:6 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:7 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:8 2024-03-28T19:15:37.708 INFO:tasks.workunit.client.0.smithi027.stderr:9 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:10 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:11 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:12 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:13 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:14 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:15 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:16 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:17 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:18 2024-03-28T19:15:37.709 INFO:tasks.workunit.client.0.smithi027.stderr:19 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:20 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:21 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:22 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:23 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:24 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:25 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:26 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:27 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:28 2024-03-28T19:15:37.710 INFO:tasks.workunit.client.0.smithi027.stderr:29 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:30 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:31 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:32 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:33 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:34 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:35 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:36 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:37 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:38 2024-03-28T19:15:37.711 INFO:tasks.workunit.client.0.smithi027.stderr:39 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:40 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:41 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:42 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:43 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:44 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:45 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:46 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:47 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:48 2024-03-28T19:15:37.712 INFO:tasks.workunit.client.0.smithi027.stderr:49 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:50 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:51 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:52 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:53 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:54 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:55 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:56 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:57 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:58 2024-03-28T19:15:37.713 INFO:tasks.workunit.client.0.smithi027.stderr:59 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:60 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:61 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:62 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:63 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:64 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:65 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:66 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:67 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:68 2024-03-28T19:15:37.714 INFO:tasks.workunit.client.0.smithi027.stderr:69 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:70 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:71 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:72 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:73 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:74 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:75 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:76 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:77 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:78 2024-03-28T19:15:37.715 INFO:tasks.workunit.client.0.smithi027.stderr:79 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:80 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:81 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:82 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:83 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:84 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:85 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:86 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:87 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:88 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:89 2024-03-28T19:15:37.716 INFO:tasks.workunit.client.0.smithi027.stderr:90' 2024-03-28T19:15:37.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:15:37.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 1 -eq 48 ']' 2024-03-28T19:15:37.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.1 2024-03-28T19:15:37.717 INFO:tasks.workunit.client.0.smithi027.stdout:v0.1 2024-03-28T19:15:37.718 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.1 2024-03-28T19:15:37.718 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.1 2024-03-28T19:15:37.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.838+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8b40729e0 con 0x7fb8b40c5e40 2024-03-28T19:15:37.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.839+0000 7fb8b1600640 1 -- 172.21.15.27:0/459938467 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fb8a0084130 con 0x7fb8b40c5e40 2024-03-28T19:15:37.839 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:37.839 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:37.839 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 159 2024-03-28T19:15:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb89406bbd0 msgr2=0x7fb89406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb89406bbd0 0x7fb89406e090 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8000f60 tx=0x7fb8a8009040 comp rx=0 tx=0).stop 2024-03-28T19:15:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40c5e40 msgr2=0x7fb8b40bef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40c5e40 0x7fb8b40bef10 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0000c00 tx=0x7fb8a00046c0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 shutdown_connections 2024-03-28T19:15:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb89406bbd0 0x7fb89406e090 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8b4158580 0x7fb8b40bf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8b40c5e40 0x7fb8b40bef10 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 --2- 172.21.15.27:0/459938467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8b40be630 0x7fb8b40c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 >> 172.21.15.27:0/459938467 conn(0x7fb8b40caf80 msgr2=0x7fb8b40c24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.843+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 shutdown_connections 2024-03-28T19:15:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:37.844+0000 7fb8bc000640 1 -- 172.21.15.27:0/459938467 wait complete. 2024-03-28T19:15:37.862 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:15:39.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.100+0000 7f12dfe00640 1 Processor -- start 2024-03-28T19:15:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.100+0000 7f12dfe00640 1 -- start start 2024-03-28T19:15:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.102+0000 7f12dfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12e006e3b0 0x7f12e00c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.102+0000 7f12dfe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f12e00c2290 0x7f12e01555a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.102+0000 7f12dfe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0158b30 con 0x7f12e014bf40 2024-03-28T19:15:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.102+0000 7f12dfe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e01596d0 con 0x7f12e006e3b0 2024-03-28T19:15:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.102+0000 7f12dfe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e014c320 con 0x7f12e00c2290 2024-03-28T19:15:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.102+0000 7f12de000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12e006e3b0 0x7f12e00c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.102+0000 7f12de000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12e006e3b0 0x7f12e00c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54268/0 (socket says 172.21.15.27:54268) 2024-03-28T19:15:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.102+0000 7f12de000640 1 -- 172.21.15.27:0/1569842282 learned_addr learned my addr 172.21.15.27:0/1569842282 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.103+0000 7f12de000640 1 -- 172.21.15.27:0/1569842282 >> v2:172.21.15.155:3300/0 conn(0x7f12e00c2290 msgr2=0x7f12e01555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.103+0000 7f12de000640 1 --2- 172.21.15.27:0/1569842282 >> v2:172.21.15.155:3300/0 conn(0x7f12e00c2290 0x7f12e01555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.103+0000 7f12de000640 1 -- 172.21.15.27:0/1569842282 >> v1:172.21.15.155:6789/0 conn(0x7f12e014bf40 legacy=0x7f12e006ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.103+0000 7f12de000640 1 -- 172.21.15.27:0/1569842282 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e0158da0 con 0x7f12e006e3b0 2024-03-28T19:15:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.104+0000 7f12de000640 1 --2- 172.21.15.27:0/1569842282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e006e3b0 0x7f12e00c1d50 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f12e00738d0 tx=0x7f12cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343ec09e36b58235 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.104+0000 7f12dd600640 1 -- 172.21.15.27:0/1569842282 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc035020 con 0x7f12e006e3b0 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.104+0000 7f12dd600640 1 -- 172.21.15.27:0/1569842282 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12cc031870 con 0x7f12e006e3b0 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.104+0000 7f12dd600640 1 -- 172.21.15.27:0/1569842282 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc031c40 con 0x7f12e006e3b0 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.105+0000 7f12dfe00640 1 -- 172.21.15.27:0/1569842282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e006e3b0 msgr2=0x7f12e00c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.105+0000 7f12dfe00640 1 --2- 172.21.15.27:0/1569842282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e006e3b0 0x7f12e00c1d50 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f12e00738d0 tx=0x7f12cc02b910 comp rx=0 tx=0).stop 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.105+0000 7f12dfe00640 1 -- 172.21.15.27:0/1569842282 shutdown_connections 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.105+0000 7f12dfe00640 1 --2- 172.21.15.27:0/1569842282 >> v2:172.21.15.155:3300/0 conn(0x7f12e00c2290 0x7f12e01555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.105+0000 7f12dfe00640 1 --2- 172.21.15.27:0/1569842282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e006e3b0 0x7f12e00c1d50 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.105+0000 7f12dfe00640 1 -- 172.21.15.27:0/1569842282 >> 172.21.15.27:0/1569842282 conn(0x7f12e006d060 msgr2=0x7f12e006d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:40.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.105+0000 7f12dfe00640 1 -- 172.21.15.27:0/1569842282 shutdown_connections 2024-03-28T19:15:40.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.105+0000 7f12dfe00640 1 -- 172.21.15.27:0/1569842282 wait complete. 2024-03-28T19:15:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.107+0000 7f12dfe00640 1 Processor -- start 2024-03-28T19:15:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.107+0000 7f12dfe00640 1 -- start start 2024-03-28T19:15:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e006e3b0 0x7f12e01eec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00c2290 0x7f12e01dec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12dfe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e014bf40 0x7f12e00c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12dfe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0155ae0 con 0x7f12e00c2290 2024-03-28T19:15:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12dfe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0159360 con 0x7f12e014bf40 2024-03-28T19:15:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12dfe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0155c60 con 0x7f12e006e3b0 2024-03-28T19:15:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12de000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00c2290 0x7f12e01dec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12de000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00c2290 0x7f12e01dec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54270/0 (socket says 172.21.15.27:54270) 2024-03-28T19:15:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12de000640 1 -- 172.21.15.27:0/4066294418 learned_addr learned my addr 172.21.15.27:0/4066294418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.108+0000 7f12dea00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e006e3b0 0x7f12e01eec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.109+0000 7f12df400640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e014bf40 0x7f12e00c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.109+0000 7f12de000640 1 -- 172.21.15.27:0/4066294418 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e006e3b0 msgr2=0x7f12e01eec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.109+0000 7f12de000640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e006e3b0 0x7f12e01eec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.109+0000 7f12de000640 1 -- 172.21.15.27:0/4066294418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e014bf40 msgr2=0x7f12e00c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.109+0000 7f12de000640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e014bf40 0x7f12e00c6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.109+0000 7f12de000640 1 -- 172.21.15.27:0/4066294418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e01df1c0 con 0x7f12e00c2290 2024-03-28T19:15:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.109+0000 7f12de000640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00c2290 0x7f12e01dec80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f12cc02b950 tx=0x7f12cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.110+0000 7f12d6a00640 1 -- 172.21.15.27:0/4066294418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc03ad60 con 0x7f12e00c2290 2024-03-28T19:15:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.110+0000 7f12d6a00640 1 -- 172.21.15.27:0/4066294418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12cc0443f0 con 0x7f12e00c2290 2024-03-28T19:15:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.110+0000 7f12d6a00640 1 -- 172.21.15.27:0/4066294418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc0447e0 con 0x7f12e00c2290 2024-03-28T19:15:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.110+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e00c60d0 con 0x7f12e00c2290 2024-03-28T19:15:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.110+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e01ee140 con 0x7f12e00c2290 2024-03-28T19:15:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.111+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e0069640 con 0x7f12e00c2290 2024-03-28T19:15:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.112+0000 7f12d6a00640 1 -- 172.21.15.27:0/4066294418 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f12cc038020 con 0x7f12e00c2290 2024-03-28T19:15:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.113+0000 7f12d6a00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bbd0 0x7f12c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.113+0000 7f12d6a00640 1 -- 172.21.15.27:0/4066294418 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f12cc0ada00 con 0x7f12e00c2290 2024-03-28T19:15:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.113+0000 7f12dea00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bbd0 0x7f12c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.114+0000 7f12dea00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bbd0 0x7f12c006e090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f12d0006fd0 tx=0x7f12d0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:40.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.118+0000 7f12d6a00640 1 -- 172.21.15.27:0/4066294418 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f12e01df1c0 con 0x7f12e00c2290 2024-03-28T19:15:40.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.456+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e00c6fb0 con 0x7f12e00c2290 2024-03-28T19:15:40.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.457+0000 7f12d6a00640 1 -- 172.21.15.27:0/4066294418 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f12cc03f070 con 0x7f12e00c2290 2024-03-28T19:15:40.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:40.456 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:40.456 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 159 2024-03-28T19:15:40.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bbd0 msgr2=0x7f12c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bbd0 0x7f12c006e090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f12d0006fd0 tx=0x7f12d0008040 comp rx=0 tx=0).stop 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00c2290 msgr2=0x7f12e01dec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00c2290 0x7f12e01dec80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f12cc02b950 tx=0x7f12cc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 shutdown_connections 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f12c006bbd0 0x7f12c006e090 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12e014bf40 0x7f12e00c6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12e00c2290 0x7f12e01dec80 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 --2- 172.21.15.27:0/4066294418 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12e006e3b0 0x7f12e01eec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:40.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 >> 172.21.15.27:0/4066294418 conn(0x7f12e006d060 msgr2=0x7f12e006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:40.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 shutdown_connections 2024-03-28T19:15:40.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:40.460+0000 7f12dfe00640 1 -- 172.21.15.27:0/4066294418 wait complete. 2024-03-28T19:15:40.478 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:15:42.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.705+0000 7f7d4f800640 1 Processor -- start 2024-03-28T19:15:42.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.705+0000 7f7d4f800640 1 -- start start 2024-03-28T19:15:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.706+0000 7f7d4f800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d4806c340 0x7f7d48156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.706+0000 7f7d4f800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d480731c0 0x7f7d48156e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.706+0000 7f7d4f800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d480c5c70 con 0x7f7d48071a00 2024-03-28T19:15:42.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.706+0000 7f7d4f800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480c63e0 con 0x7f7d480731c0 2024-03-28T19:15:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.706+0000 7f7d4f800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480c69a0 con 0x7f7d4806c340 2024-03-28T19:15:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.706+0000 7f7d47e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d480731c0 0x7f7d48156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.706+0000 7f7d47e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7d480731c0 0x7f7d48156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54282/0 (socket says 172.21.15.27:54282) 2024-03-28T19:15:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.706+0000 7f7d47e00640 1 -- 172.21.15.27:0/1119968225 learned_addr learned my addr 172.21.15.27:0/1119968225 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.707+0000 7f7d4ce00640 1 --2- 172.21.15.27:0/1119968225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d4806c340 0x7f7d48156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.707+0000 7f7d47e00640 1 -- 172.21.15.27:0/1119968225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d4806c340 msgr2=0x7f7d48156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.707+0000 7f7d47e00640 1 --2- 172.21.15.27:0/1119968225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d4806c340 0x7f7d48156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:42.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.707+0000 7f7d47e00640 1 -- 172.21.15.27:0/1119968225 >> v1:172.21.15.155:6789/0 conn(0x7f7d48071a00 legacy=0x7f7d48157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:42.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.707+0000 7f7d47e00640 1 -- 172.21.15.27:0/1119968225 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480c5f10 con 0x7f7d480731c0 2024-03-28T19:15:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.708+0000 7f7d47e00640 1 --2- 172.21.15.27:0/1119968225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d480731c0 0x7f7d48156e10 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009870 tx=0x7f7d3402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faecef69c704462a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:42.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.708+0000 7f7d47400640 1 -- 172.21.15.27:0/1119968225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d340317c0 con 0x7f7d480731c0 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d47400640 1 -- 172.21.15.27:0/1119968225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34037070 con 0x7f7d480731c0 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d47400640 1 -- 172.21.15.27:0/1119968225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34031b80 con 0x7f7d480731c0 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d4f800640 1 -- 172.21.15.27:0/1119968225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d480731c0 msgr2=0x7f7d48156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d4f800640 1 --2- 172.21.15.27:0/1119968225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d480731c0 0x7f7d48156e10 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009870 tx=0x7f7d3402b860 comp rx=0 tx=0).stop 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d4f800640 1 -- 172.21.15.27:0/1119968225 shutdown_connections 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d4f800640 1 --2- 172.21.15.27:0/1119968225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d480731c0 0x7f7d48156e10 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d4f800640 1 --2- 172.21.15.27:0/1119968225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d4806c340 0x7f7d48156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d4f800640 1 -- 172.21.15.27:0/1119968225 >> 172.21.15.27:0/1119968225 conn(0x7f7d480c4750 msgr2=0x7f7d480c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:42.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d4f800640 1 -- 172.21.15.27:0/1119968225 shutdown_connections 2024-03-28T19:15:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.709+0000 7f7d4f800640 1 -- 172.21.15.27:0/1119968225 wait complete. 2024-03-28T19:15:42.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.712+0000 7f7d4f800640 1 Processor -- start 2024-03-28T19:15:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.712+0000 7f7d4f800640 1 -- start start 2024-03-28T19:15:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4f800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4806c340 0x7f7d481e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:42.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4f800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d48071a00 0x7f7d4806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4f800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d480731c0 0x7f7d48152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4f800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480c5580 con 0x7f7d4806c340 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4f800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d48157a60 con 0x7f7d480731c0 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4f800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480c5700 con 0x7f7d48071a00 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4806c340 0x7f7d481e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4806c340 0x7f7d481e1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54288/0 (socket says 172.21.15.27:54288) 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.713+0000 7f7d4ce00640 1 -- 172.21.15.27:0/4173143225 learned_addr learned my addr 172.21.15.27:0/4173143225 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.714+0000 7f7d47e00640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d48071a00 0x7f7d4806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.714+0000 7f7d4d800640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d480731c0 0x7f7d48152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:42.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.714+0000 7f7d4ce00640 1 -- 172.21.15.27:0/4173143225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d48071a00 msgr2=0x7f7d4806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.714+0000 7f7d4ce00640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d48071a00 0x7f7d4806e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.714+0000 7f7d4ce00640 1 -- 172.21.15.27:0/4173143225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d480731c0 msgr2=0x7f7d48152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.714+0000 7f7d4ce00640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d480731c0 0x7f7d48152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.714+0000 7f7d4ce00640 1 -- 172.21.15.27:0/4173143225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d481e2a20 con 0x7f7d4806c340 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.714+0000 7f7d4ce00640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4806c340 0x7f7d481e1ec0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00c7f0 tx=0x7f7d3c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.715+0000 7f7d44c00640 1 -- 172.21.15.27:0/4173143225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c015070 con 0x7f7d4806c340 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.715+0000 7f7d44c00640 1 -- 172.21.15.27:0/4173143225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c004220 con 0x7f7d4806c340 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.715+0000 7f7d44c00640 1 -- 172.21.15.27:0/4173143225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c007620 con 0x7f7d4806c340 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.715+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d481dcc70 con 0x7f7d4806c340 2024-03-28T19:15:42.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.715+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4806f440 con 0x7f7d4806c340 2024-03-28T19:15:42.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.716+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4804f400 con 0x7f7d4806c340 2024-03-28T19:15:42.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.717+0000 7f7d44c00640 1 -- 172.21.15.27:0/4173143225 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7d3c0079d0 con 0x7f7d4806c340 2024-03-28T19:15:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.719+0000 7f7d44c00640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2806bc00 0x7f7d2806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:42.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.719+0000 7f7d44c00640 1 -- 172.21.15.27:0/4173143225 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f7d3c08b710 con 0x7f7d4806c340 2024-03-28T19:15:42.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.719+0000 7f7d47e00640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2806bc00 0x7f7d2806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:42.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.720+0000 7f7d47e00640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2806bc00 0x7f7d2806e0c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7d34036000 tx=0x7f7d3403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:42.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:42.724+0000 7f7d44c00640 1 -- 172.21.15.27:0/4173143225 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7d481dcc70 con 0x7f7d4806c340 2024-03-28T19:15:43.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.021+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d48156a40 con 0x7f7d4806c340 2024-03-28T19:15:43.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.022+0000 7f7d44c00640 1 -- 172.21.15.27:0/4173143225 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f7d481e2a20 con 0x7f7d4806c340 2024-03-28T19:15:43.022 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:43.022 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:43.023 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 159 2024-03-28T19:15:43.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.025+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2806bc00 msgr2=0x7f7d2806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:43.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2806bc00 0x7f7d2806e0c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7d34036000 tx=0x7f7d3403d040 comp rx=0 tx=0).stop 2024-03-28T19:15:43.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4806c340 msgr2=0x7f7d481e1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:43.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4806c340 0x7f7d481e1ec0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00c7f0 tx=0x7f7d3c00ccb0 comp rx=0 tx=0).stop 2024-03-28T19:15:43.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 shutdown_connections 2024-03-28T19:15:43.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d2806bc00 0x7f7d2806e0c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:43.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d480731c0 0x7f7d48152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:43.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d48071a00 0x7f7d4806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:43.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 --2- 172.21.15.27:0/4173143225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d4806c340 0x7f7d481e1ec0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:43.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 >> 172.21.15.27:0/4173143225 conn(0x7f7d480c4750 msgr2=0x7f7d480c18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:43.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 shutdown_connections 2024-03-28T19:15:43.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:43.026+0000 7f7d4f800640 1 -- 172.21.15.27:0/4173143225 wait complete. 2024-03-28T19:15:43.045 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:15:45.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:45.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.289+0000 7f17fda00640 1 Processor -- start 2024-03-28T19:15:45.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.289+0000 7f17fda00640 1 -- start start 2024-03-28T19:15:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.291+0000 7f17fda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17f8150ae0 0x7f17f80a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.291+0000 7f17fda00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f8153a30 0x7f17f8153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.291+0000 7f17fda00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f8150ec0 con 0x7f17f80a61a0 2024-03-28T19:15:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.291+0000 7f17fda00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f806dc70 con 0x7f17f8150ae0 2024-03-28T19:15:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.291+0000 7f17fda00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8153550 con 0x7f17f8153a30 2024-03-28T19:15:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.291+0000 7f17f7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17f8150ae0 0x7f17f80a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.292+0000 7f17f7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17f8150ae0 0x7f17f80a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54304/0 (socket says 172.21.15.27:54304) 2024-03-28T19:15:45.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.292+0000 7f17f7e00640 1 -- 172.21.15.27:0/1200087095 learned_addr learned my addr 172.21.15.27:0/1200087095 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.292+0000 7f17fd000640 1 --2- 172.21.15.27:0/1200087095 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f8153a30 0x7f17f8153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.292+0000 7f17f7e00640 1 -- 172.21.15.27:0/1200087095 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f8153a30 msgr2=0x7f17f8153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.292+0000 7f17f7e00640 1 --2- 172.21.15.27:0/1200087095 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f8153a30 0x7f17f8153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.292+0000 7f17f7e00640 1 -- 172.21.15.27:0/1200087095 >> v1:172.21.15.155:6789/0 conn(0x7f17f80a61a0 legacy=0x7f17f80c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:45.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.292+0000 7f17f7e00640 1 -- 172.21.15.27:0/1200087095 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f806df10 con 0x7f17f8150ae0 2024-03-28T19:15:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.294+0000 7f17f7e00640 1 --2- 172.21.15.27:0/1200087095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8150ae0 0x7f17f80a5c60 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009870 tx=0x7f17e802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6786f6f37d63af53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.294+0000 7f17f6a00640 1 -- 172.21.15.27:0/1200087095 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e80317c0 con 0x7f17f8150ae0 2024-03-28T19:15:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.294+0000 7f17f6a00640 1 -- 172.21.15.27:0/1200087095 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e8037070 con 0x7f17f8150ae0 2024-03-28T19:15:45.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.294+0000 7f17f6a00640 1 -- 172.21.15.27:0/1200087095 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8031b80 con 0x7f17f8150ae0 2024-03-28T19:15:45.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.295+0000 7f17fda00640 1 -- 172.21.15.27:0/1200087095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8150ae0 msgr2=0x7f17f80a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.295+0000 7f17fda00640 1 --2- 172.21.15.27:0/1200087095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8150ae0 0x7f17f80a5c60 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009870 tx=0x7f17e802b860 comp rx=0 tx=0).stop 2024-03-28T19:15:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.295+0000 7f17fda00640 1 -- 172.21.15.27:0/1200087095 shutdown_connections 2024-03-28T19:15:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.295+0000 7f17fda00640 1 --2- 172.21.15.27:0/1200087095 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f8153a30 0x7f17f8153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.295+0000 7f17fda00640 1 --2- 172.21.15.27:0/1200087095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8150ae0 0x7f17f80a5c60 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.295+0000 7f17fda00640 1 -- 172.21.15.27:0/1200087095 >> 172.21.15.27:0/1200087095 conn(0x7f17f806b0c0 msgr2=0x7f17f806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.295+0000 7f17fda00640 1 -- 172.21.15.27:0/1200087095 shutdown_connections 2024-03-28T19:15:45.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.295+0000 7f17fda00640 1 -- 172.21.15.27:0/1200087095 wait complete. 2024-03-28T19:15:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.297+0000 7f17fda00640 1 Processor -- start 2024-03-28T19:15:45.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.297+0000 7f17fda00640 1 -- start start 2024-03-28T19:15:45.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.298+0000 7f17fda00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f80a61a0 0x7f17f814df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.298+0000 7f17fda00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17f8150ae0 0x7f17f8145250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.298+0000 7f17fda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8153a30 0x7f17f8145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.298+0000 7f17fda00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f806e7b0 con 0x7f17f8153a30 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.298+0000 7f17fda00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f806e4d0 con 0x7f17f8150ae0 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.298+0000 7f17fda00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f80c70b0 con 0x7f17f80a61a0 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17fd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8153a30 0x7f17f8145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17fd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8153a30 0x7f17f8145790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54306/0 (socket says 172.21.15.27:54306) 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17fd000640 1 -- 172.21.15.27:0/411481509 learned_addr learned my addr 172.21.15.27:0/411481509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17f7e00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f80a61a0 0x7f17f814df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17f7400640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17f8150ae0 0x7f17f8145250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:45.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17fd000640 1 -- 172.21.15.27:0/411481509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f80a61a0 msgr2=0x7f17f814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:45.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17fd000640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f80a61a0 0x7f17f814df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17fd000640 1 -- 172.21.15.27:0/411481509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17f8150ae0 msgr2=0x7f17f8145250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:45.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17fd000640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17f8150ae0 0x7f17f8145250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.299+0000 7f17fd000640 1 -- 172.21.15.27:0/411481509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f8145cd0 con 0x7f17f8153a30 2024-03-28T19:15:45.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.300+0000 7f17fd000640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8153a30 0x7f17f8145790 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f17f0007e60 tx=0x7f17f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.300+0000 7f17efe00640 1 -- 172.21.15.27:0/411481509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0014420 con 0x7f17f8153a30 2024-03-28T19:15:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.300+0000 7f17efe00640 1 -- 172.21.15.27:0/411481509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f0019070 con 0x7f17f8153a30 2024-03-28T19:15:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.300+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f80a7670 con 0x7f17f8153a30 2024-03-28T19:15:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.300+0000 7f17efe00640 1 -- 172.21.15.27:0/411481509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0014820 con 0x7f17f8153a30 2024-03-28T19:15:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.300+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f81ece00 con 0x7f17f8153a30 2024-03-28T19:15:45.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.302+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f8067dd0 con 0x7f17f8153a30 2024-03-28T19:15:45.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.303+0000 7f17efe00640 1 -- 172.21.15.27:0/411481509 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f17f00149c0 con 0x7f17f8153a30 2024-03-28T19:15:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.304+0000 7f17efe00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f17d806bbd0 0x7f17d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.304+0000 7f17efe00640 1 -- 172.21.15.27:0/411481509 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f17f008bf50 con 0x7f17f8153a30 2024-03-28T19:15:45.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.304+0000 7f17f7e00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f17d806bbd0 0x7f17d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:45.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.305+0000 7f17f7e00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f17d806bbd0 0x7f17d806e090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f17e8036000 tx=0x7f17e803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:45.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.309+0000 7f17efe00640 1 -- 172.21.15.27:0/411481509 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f17f0059dd0 con 0x7f17f8153a30 2024-03-28T19:15:45.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.610+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17f806da50 con 0x7f17f8153a30 2024-03-28T19:15:45.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.611+0000 7f17efe00640 1 -- 172.21.15.27:0/411481509 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f17f00596d0 con 0x7f17f8153a30 2024-03-28T19:15:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:45.611 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:45.611 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 159 2024-03-28T19:15:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.613+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f17d806bbd0 msgr2=0x7f17d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.613+0000 7f17fda00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f17d806bbd0 0x7f17d806e090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f17e8036000 tx=0x7f17e803d040 comp rx=0 tx=0).stop 2024-03-28T19:15:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.613+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8153a30 msgr2=0x7f17f8145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.613+0000 7f17fda00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8153a30 0x7f17f8145790 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f17f0007e60 tx=0x7f17f000e4b0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.614+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 shutdown_connections 2024-03-28T19:15:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.614+0000 7f17fda00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f17d806bbd0 0x7f17d806e090 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.614+0000 7f17fda00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17f8153a30 0x7f17f8145790 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.614+0000 7f17fda00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17f8150ae0 0x7f17f8145250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.614+0000 7f17fda00640 1 --2- 172.21.15.27:0/411481509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f17f80a61a0 0x7f17f814df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.614+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 >> 172.21.15.27:0/411481509 conn(0x7f17f806b0c0 msgr2=0x7f17f80c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.614+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 shutdown_connections 2024-03-28T19:15:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:45.614+0000 7f17fda00640 1 -- 172.21.15.27:0/411481509 wait complete. 2024-03-28T19:15:45.637 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:15:47.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:47.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.872+0000 7f921aa00640 1 Processor -- start 2024-03-28T19:15:47.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.872+0000 7f921aa00640 1 -- start start 2024-03-28T19:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.873+0000 7f921aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92140c1f50 0x7f92140ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.874+0000 7f921aa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92140ac1c0 con 0x7f92140abd90 2024-03-28T19:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.874+0000 7f921aa00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921406d8a0 con 0x7f92140c5e80 2024-03-28T19:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.874+0000 7f921aa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f921406d650 con 0x7f92140c1f50 2024-03-28T19:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.874+0000 7f9213400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92140c1f50 0x7f92140ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:47.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.874+0000 7f9213400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92140c1f50 0x7f92140ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40460/0 (socket says 172.21.15.27:40460) 2024-03-28T19:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.874+0000 7f9218a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f92140abd90 0x7f92140aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34108/0 (socket says 172.21.15.27:34108) 2024-03-28T19:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.874+0000 7f9213400640 1 -- 172.21.15.27:0/314762717 learned_addr learned my addr 172.21.15.27:0/314762717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.875+0000 7f9213400640 1 -- 172.21.15.27:0/314762717 >> v1:172.21.15.155:6789/0 conn(0x7f92140c5e80 legacy=0x7f92140c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.875+0000 7f9213400640 1 -- 172.21.15.27:0/314762717 >> v1:172.21.15.27:6789/0 conn(0x7f92140abd90 legacy=0x7f92140aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.875+0000 7f9213400640 1 -- 172.21.15.27:0/314762717 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921406db80 con 0x7f92140c1f50 2024-03-28T19:15:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.875+0000 7f9212a00640 1 -- 172.21.15.27:0/314762717 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123147678 0 0) 0x7f92140ac1c0 con 0x7f92140abd90 2024-03-28T19:15:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.876+0000 7f9213400640 1 --2- 172.21.15.27:0/314762717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92140ab850 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f9200009870 tx=0x7f920002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77067176b4292d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:47.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.876+0000 7f9212a00640 1 -- 172.21.15.27:0/314762717 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920002e070 con 0x7f92140c1f50 2024-03-28T19:15:47.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.876+0000 7f9212a00640 1 -- 172.21.15.27:0/314762717 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9200037070 con 0x7f92140c1f50 2024-03-28T19:15:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.876+0000 7f9212a00640 1 -- 172.21.15.27:0/314762717 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200031a00 con 0x7f92140c1f50 2024-03-28T19:15:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.877+0000 7f921aa00640 1 -- 172.21.15.27:0/314762717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 msgr2=0x7f92140ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.877+0000 7f921aa00640 1 --2- 172.21.15.27:0/314762717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92140ab850 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f9200009870 tx=0x7f920002b880 comp rx=0 tx=0).stop 2024-03-28T19:15:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.877+0000 7f921aa00640 1 -- 172.21.15.27:0/314762717 shutdown_connections 2024-03-28T19:15:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.877+0000 7f921aa00640 1 --2- 172.21.15.27:0/314762717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92140ab850 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.877+0000 7f921aa00640 1 -- 172.21.15.27:0/314762717 >> 172.21.15.27:0/314762717 conn(0x7f921406b9d0 msgr2=0x7f92140c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.877+0000 7f921aa00640 1 -- 172.21.15.27:0/314762717 shutdown_connections 2024-03-28T19:15:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.877+0000 7f921aa00640 1 -- 172.21.15.27:0/314762717 wait complete. 2024-03-28T19:15:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.879+0000 7f921aa00640 1 Processor -- start 2024-03-28T19:15:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.879+0000 7f921aa00640 1 -- start start 2024-03-28T19:15:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f921aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92140abd90 0x7f92140c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f921aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92141dc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f921aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f92140c5e80 0x7f92141dcc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f921aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92140c6260 con 0x7f92140c1f50 2024-03-28T19:15:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f921aa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921406e1d0 con 0x7f92140abd90 2024-03-28T19:15:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f921aa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92140c63e0 con 0x7f92140c5e80 2024-03-28T19:15:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f9213400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92141dc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f9213400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92141dc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40470/0 (socket says 172.21.15.27:40470) 2024-03-28T19:15:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.880+0000 7f9213400640 1 -- 172.21.15.27:0/2943985063 learned_addr learned my addr 172.21.15.27:0/2943985063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.881+0000 7f9213e00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92140abd90 0x7f92140c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.881+0000 7f9218a00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f92140c5e80 0x7f92141dcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.881+0000 7f9213400640 1 -- 172.21.15.27:0/2943985063 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f92140c5e80 msgr2=0x7f92141dcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.881+0000 7f9213400640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f92140c5e80 0x7f92141dcc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.881+0000 7f9213400640 1 -- 172.21.15.27:0/2943985063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92140abd90 msgr2=0x7f92140c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.881+0000 7f9213400640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92140abd90 0x7f92140c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.881+0000 7f9213400640 1 -- 172.21.15.27:0/2943985063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92140bd5f0 con 0x7f92140c1f50 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.882+0000 7f9213400640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92141dc730 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f9200000c00 tx=0x7f92000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.882+0000 7f920be00640 1 -- 172.21.15.27:0/2943985063 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200032880 con 0x7f92140c1f50 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.882+0000 7f920be00640 1 -- 172.21.15.27:0/2943985063 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9200032a40 con 0x7f92140c1f50 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.882+0000 7f920be00640 1 -- 172.21.15.27:0/2943985063 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9200032e30 con 0x7f92140c1f50 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.882+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92140bd860 con 0x7f92140c1f50 2024-03-28T19:15:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.882+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92140bbf00 con 0x7f92140c1f50 2024-03-28T19:15:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.884+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921404f400 con 0x7f92140c1f50 2024-03-28T19:15:47.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.885+0000 7f920be00640 1 -- 172.21.15.27:0/2943985063 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f920004c060 con 0x7f92140c1f50 2024-03-28T19:15:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.886+0000 7f920be00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91f406bbd0 0x7f91f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.886+0000 7f920be00640 1 -- 172.21.15.27:0/2943985063 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8959+0+0 (secure 0 0 0) 0x7f92000acff0 con 0x7f92140c1f50 2024-03-28T19:15:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.887+0000 7f9213e00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91f406bbd0 0x7f91f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.888+0000 7f9213e00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91f406bbd0 0x7f91f406e090 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9204004ea0 tx=0x7f9204009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:47.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:47.893+0000 7f920be00640 1 -- 172.21.15.27:0/2943985063 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9200040070 con 0x7f92140c1f50 2024-03-28T19:15:48.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.213+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f921406c0d0 con 0x7f92140c1f50 2024-03-28T19:15:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.213+0000 7f920be00640 1 -- 172.21.15.27:0/2943985063 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f920007ae70 con 0x7f92140c1f50 2024-03-28T19:15:48.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:48.213 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6077,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:18.543129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7317},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7317":{"gid":7317,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6837/3534117308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3534117308},{"type":"v1","addr":"172.21.15.27:6837","nonce":3534117308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7317,"qdb_cluster":[7317]},"id":2}]} 2024-03-28T19:15:48.214 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 159 2024-03-28T19:15:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91f406bbd0 msgr2=0x7f91f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91f406bbd0 0x7f91f406e090 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9204004ea0 tx=0x7f9204009040 comp rx=0 tx=0).stop 2024-03-28T19:15:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 msgr2=0x7f92141dc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92141dc730 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f9200000c00 tx=0x7f92000046c0 comp rx=0 tx=0).stop 2024-03-28T19:15:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 shutdown_connections 2024-03-28T19:15:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f91f406bbd0 0x7f91f406e090 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:48.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f92140c5e80 0x7f92141dcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:48.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92140c1f50 0x7f92141dc730 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:48.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 --2- 172.21.15.27:0/2943985063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92140abd90 0x7f92140c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:48.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.216+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 >> 172.21.15.27:0/2943985063 conn(0x7f921406b9d0 msgr2=0x7f9214154540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:48.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.217+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 shutdown_connections 2024-03-28T19:15:48.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:48.217+0000 7f921aa00640 1 -- 172.21.15.27:0/2943985063 wait complete. 2024-03-28T19:15:48.237 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:15:50.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:50.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.473+0000 7fd086800640 1 Processor -- start 2024-03-28T19:15:50.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.473+0000 7fd086800640 1 -- start start 2024-03-28T19:15:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.474+0000 7fd086800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd080152a70 0x7fd080159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:50.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.474+0000 7fd086800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd0800c6cc0 0x7fd0800cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.474+0000 7fd086800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0800c5710 con 0x7fd0800cb620 2024-03-28T19:15:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.474+0000 7fd086800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0801556b0 con 0x7fd080152a70 2024-03-28T19:15:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.474+0000 7fd086800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0800c59f0 con 0x7fd0800c6cc0 2024-03-28T19:15:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.475+0000 7fd085400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd080152a70 0x7fd080159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.475+0000 7fd084a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd0800c6cc0 0x7fd0800cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.475+0000 7fd085400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd080152a70 0x7fd080159580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40474/0 (socket says 172.21.15.27:40474) 2024-03-28T19:15:50.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.475+0000 7fd085400640 1 -- 172.21.15.27:0/1977533221 learned_addr learned my addr 172.21.15.27:0/1977533221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.475+0000 7fd085400640 1 -- 172.21.15.27:0/1977533221 >> v2:172.21.15.155:3300/0 conn(0x7fd0800c6cc0 msgr2=0x7fd0800cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.475+0000 7fd085400640 1 --2- 172.21.15.27:0/1977533221 >> v2:172.21.15.155:3300/0 conn(0x7fd0800c6cc0 0x7fd0800cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.476+0000 7fd085400640 1 -- 172.21.15.27:0/1977533221 >> v1:172.21.15.27:6789/0 conn(0x7fd0800cb620 legacy=0x7fd080155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:50.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.476+0000 7fd085400640 1 -- 172.21.15.27:0/1977533221 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0800c5cd0 con 0x7fd080152a70 2024-03-28T19:15:50.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.477+0000 7fd085400640 1 --2- 172.21.15.27:0/1977533221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd080152a70 0x7fd080159580 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fd0800cbb80 tx=0x7fd07402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cdc83548ef3895d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:50.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.477+0000 7fd07fe00640 1 -- 172.21.15.27:0/1977533221 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074035020 con 0x7fd080152a70 2024-03-28T19:15:50.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.477+0000 7fd07fe00640 1 -- 172.21.15.27:0/1977533221 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074031870 con 0x7fd080152a70 2024-03-28T19:15:50.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.478+0000 7fd07fe00640 1 -- 172.21.15.27:0/1977533221 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074031c40 con 0x7fd080152a70 2024-03-28T19:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.478+0000 7fd086800640 1 -- 172.21.15.27:0/1977533221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd080152a70 msgr2=0x7fd080159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:50.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.478+0000 7fd086800640 1 --2- 172.21.15.27:0/1977533221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd080152a70 0x7fd080159580 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fd0800cbb80 tx=0x7fd07402b910 comp rx=0 tx=0).stop 2024-03-28T19:15:50.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.479+0000 7fd086800640 1 -- 172.21.15.27:0/1977533221 shutdown_connections 2024-03-28T19:15:50.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.479+0000 7fd086800640 1 --2- 172.21.15.27:0/1977533221 >> v2:172.21.15.155:3300/0 conn(0x7fd0800c6cc0 0x7fd0800cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.479+0000 7fd086800640 1 --2- 172.21.15.27:0/1977533221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd080152a70 0x7fd080159580 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.479+0000 7fd086800640 1 -- 172.21.15.27:0/1977533221 >> 172.21.15.27:0/1977533221 conn(0x7fd08006b2a0 msgr2=0x7fd08006b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:50.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.479+0000 7fd086800640 1 -- 172.21.15.27:0/1977533221 shutdown_connections 2024-03-28T19:15:50.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.479+0000 7fd086800640 1 -- 172.21.15.27:0/1977533221 wait complete. 2024-03-28T19:15:50.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.481+0000 7fd086800640 1 Processor -- start 2024-03-28T19:15:50.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.481+0000 7fd086800640 1 -- start start 2024-03-28T19:15:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd086800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0800c6cc0 0x7fd0800ab240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:50.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd086800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0800cb620 0x7fd080147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd086800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd080152a70 0x7fd0801481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd086800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800c8f50 con 0x7fd0800cb620 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd086800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800c8dd0 con 0x7fd0800c6cc0 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd086800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800c5fb0 con 0x7fd080152a70 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd084a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0800cb620 0x7fd080147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd084a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0800cb620 0x7fd080147ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40482/0 (socket says 172.21.15.27:40482) 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.482+0000 7fd084a00640 1 -- 172.21.15.27:0/2099002553 learned_addr learned my addr 172.21.15.27:0/2099002553 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.483+0000 7fd085e00640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd080152a70 0x7fd0801481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.483+0000 7fd085400640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0800c6cc0 0x7fd0800ab240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:50.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.483+0000 7fd084a00640 1 -- 172.21.15.27:0/2099002553 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd080152a70 msgr2=0x7fd0801481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.483+0000 7fd084a00640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd080152a70 0x7fd0801481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.483+0000 7fd084a00640 1 -- 172.21.15.27:0/2099002553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0800c6cc0 msgr2=0x7fd0800ab240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.483+0000 7fd084a00640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0800c6cc0 0x7fd0800ab240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.483+0000 7fd084a00640 1 -- 172.21.15.27:0/2099002553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd080148720 con 0x7fd0800cb620 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.483+0000 7fd084a00640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0800cb620 0x7fd080147ca0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00e870 tx=0x7fd06c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.484+0000 7fd07d600640 1 -- 172.21.15.27:0/2099002553 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c017070 con 0x7fd0800cb620 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.484+0000 7fd07d600640 1 -- 172.21.15.27:0/2099002553 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c00ee60 con 0x7fd0800cb620 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.484+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd080154ce0 con 0x7fd0800cb620 2024-03-28T19:15:50.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.484+0000 7fd07d600640 1 -- 172.21.15.27:0/2099002553 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c00cb80 con 0x7fd0800cb620 2024-03-28T19:15:50.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.484+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd080154f80 con 0x7fd0800cb620 2024-03-28T19:15:50.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.486+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd080067dd0 con 0x7fd0800cb620 2024-03-28T19:15:50.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.487+0000 7fd07d600640 1 -- 172.21.15.27:0/2099002553 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd06c01a070 con 0x7fd0800cb620 2024-03-28T19:15:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.489+0000 7fd07d600640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd06006bbd0 0x7fd06006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.489+0000 7fd07d600640 1 -- 172.21.15.27:0/2099002553 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7fd06c08b6b0 con 0x7fd0800cb620 2024-03-28T19:15:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.489+0000 7fd085400640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd06006bbd0 0x7fd06006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:50.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.494+0000 7fd085400640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd06006bbd0 0x7fd06006e090 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd074036000 tx=0x7fd07403c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.496+0000 7fd07d600640 1 -- 172.21.15.27:0/2099002553 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd06c058e20 con 0x7fd0800cb620 2024-03-28T19:15:50.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.830+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd080071bd0 con 0x7fd0800cb620 2024-03-28T19:15:50.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.831+0000 7fd07d600640 1 -- 172.21.15.27:0/2099002553 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5463 (secure 0 0 0) 0x7fd06c05cec0 con 0x7fd0800cb620 2024-03-28T19:15:50.832 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:50.832 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:50.132498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:15:50.832 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 160 2024-03-28T19:15:50.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd06006bbd0 msgr2=0x7fd06006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:50.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd06006bbd0 0x7fd06006e090 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd074036000 tx=0x7fd07403c040 comp rx=0 tx=0).stop 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0800cb620 msgr2=0x7fd080147ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0800cb620 0x7fd080147ca0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00e870 tx=0x7fd06c00ed30 comp rx=0 tx=0).stop 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 shutdown_connections 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd06006bbd0 0x7fd06006e090 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd080152a70 0x7fd0801481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0800cb620 0x7fd080147ca0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 --2- 172.21.15.27:0/2099002553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0800c6cc0 0x7fd0800ab240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 >> 172.21.15.27:0/2099002553 conn(0x7fd08006b2a0 msgr2=0x7fd08014de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:50.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 shutdown_connections 2024-03-28T19:15:50.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:50.834+0000 7fd086800640 1 -- 172.21.15.27:0/2099002553 wait complete. 2024-03-28T19:15:50.856 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-03-28T19:15:50.857 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:15:50.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:51.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.092+0000 7f6acda00640 1 Processor -- start 2024-03-28T19:15:51.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.092+0000 7f6acda00640 1 -- start start 2024-03-28T19:15:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.094+0000 7f6acda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ac80733b0 0x7f6ac81572f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.094+0000 7f6acda00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6ac806bbb0 0x7f6ac806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.094+0000 7f6acda00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac80a7050 con 0x7f6ac8150b30 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.094+0000 7f6acda00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac80a7330 con 0x7f6ac80733b0 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.094+0000 7f6acda00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac8156810 con 0x7f6ac806bbb0 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.094+0000 7f6ac7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ac80733b0 0x7f6ac81572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.095+0000 7f6ac7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ac80733b0 0x7f6ac81572f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40490/0 (socket says 172.21.15.27:40490) 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.095+0000 7f6ac7400640 1 -- 172.21.15.27:0/3109415894 learned_addr learned my addr 172.21.15.27:0/3109415894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.095+0000 7f6acd000640 1 --2- 172.21.15.27:0/3109415894 >> v2:172.21.15.155:3300/0 conn(0x7f6ac806bbb0 0x7f6ac806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.095+0000 7f6ac7400640 1 -- 172.21.15.27:0/3109415894 >> v2:172.21.15.155:3300/0 conn(0x7f6ac806bbb0 msgr2=0x7f6ac806fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:51.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.095+0000 7f6ac7400640 1 --2- 172.21.15.27:0/3109415894 >> v2:172.21.15.155:3300/0 conn(0x7f6ac806bbb0 0x7f6ac806fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.095+0000 7f6ac7400640 1 -- 172.21.15.27:0/3109415894 >> v1:172.21.15.155:6789/0 conn(0x7f6ac8150b30 legacy=0x7f6ac8153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:51.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.095+0000 7f6ac7400640 1 -- 172.21.15.27:0/3109415894 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac8156af0 con 0x7f6ac80733b0 2024-03-28T19:15:51.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.095+0000 7f6acd000640 1 --2- 172.21.15.27:0/3109415894 >> v2:172.21.15.155:3300/0 conn(0x7f6ac806bbb0 0x7f6ac806fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:15:51.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.097+0000 7f6ac7400640 1 --2- 172.21.15.27:0/3109415894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80733b0 0x7f6ac81572f0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6ac806d430 tx=0x7f6ab402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd613d0f205db67e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.097+0000 7f6ac6a00640 1 -- 172.21.15.27:0/3109415894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4035020 con 0x7f6ac80733b0 2024-03-28T19:15:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.097+0000 7f6ac6a00640 1 -- 172.21.15.27:0/3109415894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab4031870 con 0x7f6ac80733b0 2024-03-28T19:15:51.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.097+0000 7f6ac6a00640 1 -- 172.21.15.27:0/3109415894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4031c40 con 0x7f6ac80733b0 2024-03-28T19:15:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.098+0000 7f6acda00640 1 -- 172.21.15.27:0/3109415894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80733b0 msgr2=0x7f6ac81572f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.098+0000 7f6acda00640 1 --2- 172.21.15.27:0/3109415894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80733b0 0x7f6ac81572f0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6ac806d430 tx=0x7f6ab402b910 comp rx=0 tx=0).stop 2024-03-28T19:15:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.099+0000 7f6acda00640 1 -- 172.21.15.27:0/3109415894 shutdown_connections 2024-03-28T19:15:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.099+0000 7f6acda00640 1 --2- 172.21.15.27:0/3109415894 >> v2:172.21.15.155:3300/0 conn(0x7f6ac806bbb0 0x7f6ac806fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.099+0000 7f6acda00640 1 --2- 172.21.15.27:0/3109415894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80733b0 0x7f6ac81572f0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.099+0000 7f6acda00640 1 -- 172.21.15.27:0/3109415894 >> 172.21.15.27:0/3109415894 conn(0x7f6ac806b0c0 msgr2=0x7f6ac806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:51.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.099+0000 7f6acda00640 1 -- 172.21.15.27:0/3109415894 shutdown_connections 2024-03-28T19:15:51.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.099+0000 7f6acda00640 1 -- 172.21.15.27:0/3109415894 wait complete. 2024-03-28T19:15:51.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.102+0000 7f6acda00640 1 Processor -- start 2024-03-28T19:15:51.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.102+0000 7f6acda00640 1 -- start start 2024-03-28T19:15:51.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acda00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ac806bbb0 0x7f6ac81efa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:51.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acda00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ac8150b30 0x7f6ac8152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:51.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80c1cb0 0x7f6ac80abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:51.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acda00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80a6c70 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acda00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac8153910 con 0x7f6ac806bbb0 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acda00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80a6df0 con 0x7f6ac8150b30 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80c1cb0 0x7f6ac80abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80c1cb0 0x7f6ac80abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40500/0 (socket says 172.21.15.27:40500) 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6acd000640 1 -- 172.21.15.27:0/1356650062 learned_addr learned my addr 172.21.15.27:0/1356650062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.104+0000 7f6ac7e00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ac806bbb0 0x7f6ac81efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.105+0000 7f6acd000640 1 -- 172.21.15.27:0/1356650062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ac8150b30 msgr2=0x7f6ac8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.105+0000 7f6acd000640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ac8150b30 0x7f6ac8152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.105+0000 7f6acd000640 1 -- 172.21.15.27:0/1356650062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ac806bbb0 msgr2=0x7f6ac81efa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:51.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.105+0000 7f6acd000640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ac806bbb0 0x7f6ac81efa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.105+0000 7f6acd000640 1 -- 172.21.15.27:0/1356650062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac80722c0 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.105+0000 7f6acd000640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80c1cb0 0x7f6ac80abe40 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0007ef0 tx=0x7f6ac000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:51.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.105+0000 7f6abfe00640 1 -- 172.21.15.27:0/1356650062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac001a070 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.105+0000 7f6abfe00640 1 -- 172.21.15.27:0/1356650062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac0003b90 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.106+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac806ee50 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.106+0000 7f6abfe00640 1 -- 172.21.15.27:0/1356650062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac000de20 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.106+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac806f410 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.108+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac8069640 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.109+0000 7f6abfe00640 1 -- 172.21.15.27:0/1356650062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6ac001d070 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.110+0000 7f6abfe00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6aa806bbd0 0x7f6aa806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:51.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.110+0000 7f6abfe00640 1 -- 172.21.15.27:0/1356650062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7f6ac008b940 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.111+0000 7f6ac7e00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6aa806bbd0 0x7f6aa806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:51.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.111+0000 7f6ac7e00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6aa806bbd0 0x7f6aa806e090 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80016e0 tx=0x7f6ab8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:51.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.115+0000 7f6abfe00640 1 -- 172.21.15.27:0/1356650062 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6ac80722c0 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.491+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ac80c2790 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.493+0000 7f6abfe00640 1 -- 172.21.15.27:0/1356650062 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f6ac0010070 con 0x7f6ac80c1cb0 2024-03-28T19:15:51.493 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:51.493 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:50.132498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:15:51.494 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 160 2024-03-28T19:15:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.496+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6aa806bbd0 msgr2=0x7f6aa806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.496+0000 7f6acda00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6aa806bbd0 0x7f6aa806e090 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80016e0 tx=0x7f6ab8009040 comp rx=0 tx=0).stop 2024-03-28T19:15:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.496+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80c1cb0 msgr2=0x7f6ac80abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.496+0000 7f6acda00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80c1cb0 0x7f6ac80abe40 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0007ef0 tx=0x7f6ac000d4a0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.497+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 shutdown_connections 2024-03-28T19:15:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.497+0000 7f6acda00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6aa806bbd0 0x7f6aa806e090 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.497+0000 7f6acda00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ac80c1cb0 0x7f6ac80abe40 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.497+0000 7f6acda00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ac8150b30 0x7f6ac8152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.497+0000 7f6acda00640 1 --2- 172.21.15.27:0/1356650062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ac806bbb0 0x7f6ac81efa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.497+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 >> 172.21.15.27:0/1356650062 conn(0x7f6ac806b0c0 msgr2=0x7f6ac814e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.497+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 shutdown_connections 2024-03-28T19:15:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:51.497+0000 7f6acda00640 1 -- 172.21.15.27:0/1356650062 wait complete. 2024-03-28T19:15:51.527 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 160, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7146": { "addr": "172.21.15.27:6835/3913435536", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 3913435536, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 3913435536, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7146, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:14:41.786270+0000", "qdb_cluster": [ 7146 ], "qdb_leader": 7146, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7146 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 160, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6077": { "addr": "172.21.15.155:6834/2865796032", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 2865796032, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 2865796032, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6077, "incarnation": 160, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:15:50.132498+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6077 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/938167944", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 938167944, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 938167944, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 153, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6014, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:15:53.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:53.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.785+0000 7f2089000640 1 Processor -- start 2024-03-28T19:15:53.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.786+0000 7f2089000640 1 -- start start 2024-03-28T19:15:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2089000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20840c58c0 0x7f20840c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2089000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c8ed0 0x7f20841555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2089000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f20840be5d0 0x7f2084155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2089000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2084158bd0 con 0x7f20840be5d0 2024-03-28T19:15:53.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2089000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2084158eb0 con 0x7f20840c58c0 2024-03-28T19:15:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2089000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2084159190 con 0x7f20840c8ed0 2024-03-28T19:15:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2083e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f20840be5d0 0x7f2084155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2082a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c8ed0 0x7f20841555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:53.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.787+0000 7f2083400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20840c58c0 0x7f20840c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:53.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.788+0000 7f2083400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20840c58c0 0x7f20840c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:47148/0 (socket says 172.21.15.27:47148) 2024-03-28T19:15:53.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.788+0000 7f2083400640 1 -- 172.21.15.27:0/1464767139 learned_addr learned my addr 172.21.15.27:0/1464767139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:53.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.788+0000 7f2083e00640 1 -- 172.21.15.27:0/1464767139 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c8ed0 msgr2=0x7f20841555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:53.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.788+0000 7f2083e00640 1 --2- 172.21.15.27:0/1464767139 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c8ed0 0x7f20841555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:53.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.788+0000 7f2083e00640 1 -- 172.21.15.27:0/1464767139 >> v2:172.21.15.155:3300/0 conn(0x7f20840c58c0 msgr2=0x7f20840c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:53.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.788+0000 7f2083e00640 1 --2- 172.21.15.27:0/1464767139 >> v2:172.21.15.155:3300/0 conn(0x7f20840c58c0 0x7f20840c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:53.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.788+0000 7f2083e00640 1 -- 172.21.15.27:0/1464767139 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084159470 con 0x7f20840be5d0 2024-03-28T19:15:53.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.789+0000 7f2083e00640 1 --2- 172.21.15.27:0/1464767139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f2084155b20 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f208406c060 tx=0x7f207c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfe3950eed3d270 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:53.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2082000640 1 -- 172.21.15.27:0/1464767139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c039070 con 0x7f20840be5d0 2024-03-28T19:15:53.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2082000640 1 -- 172.21.15.27:0/1464767139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c02cb40 con 0x7f20840be5d0 2024-03-28T19:15:53.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2082000640 1 -- 172.21.15.27:0/1464767139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c034440 con 0x7f20840be5d0 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2089000640 1 -- 172.21.15.27:0/1464767139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 msgr2=0x7f2084155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2089000640 1 --2- 172.21.15.27:0/1464767139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f2084155b20 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f208406c060 tx=0x7f207c02bd90 comp rx=0 tx=0).stop 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2089000640 1 -- 172.21.15.27:0/1464767139 shutdown_connections 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2089000640 1 --2- 172.21.15.27:0/1464767139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f2084155b20 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2089000640 1 --2- 172.21.15.27:0/1464767139 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c8ed0 0x7f20841555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2089000640 1 --2- 172.21.15.27:0/1464767139 >> v2:172.21.15.155:3300/0 conn(0x7f20840c58c0 0x7f20840c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.790+0000 7f2089000640 1 -- 172.21.15.27:0/1464767139 >> 172.21.15.27:0/1464767139 conn(0x7f208406b6f0 msgr2=0x7f20840acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.791+0000 7f2089000640 1 -- 172.21.15.27:0/1464767139 shutdown_connections 2024-03-28T19:15:53.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.791+0000 7f2089000640 1 -- 172.21.15.27:0/1464767139 wait complete. 2024-03-28T19:15:53.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.792+0000 7f2089000640 1 Processor -- start 2024-03-28T19:15:53.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.793+0000 7f2089000640 1 -- start start 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2089000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f20840ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2089000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c58c0 0x7f20840c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2089000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c8ed0 0x7f2084154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2089000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840b09a0 con 0x7f20840be5d0 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2089000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840b0820 con 0x7f20840c8ed0 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2089000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20840ad7d0 con 0x7f20840c58c0 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2083400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f20840ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2083400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f20840ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40516/0 (socket says 172.21.15.27:40516) 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2083400640 1 -- 172.21.15.27:0/1625026657 learned_addr learned my addr 172.21.15.27:0/1625026657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2082a00640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c58c0 0x7f20840c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2083e00640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c8ed0 0x7f2084154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:53.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.794+0000 7f2083400640 1 -- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c58c0 msgr2=0x7f20840c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:53.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2083400640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c58c0 0x7f20840c3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:53.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2083400640 1 -- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c8ed0 msgr2=0x7f2084154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:53.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2083400640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c8ed0 0x7f2084154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:53.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2083400640 1 -- 172.21.15.27:0/1625026657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20841eb220 con 0x7f20840be5d0 2024-03-28T19:15:53.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2082a00640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c58c0 0x7f20840c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:15:53.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2083e00640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c8ed0 0x7f2084154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:15:53.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2083400640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f20840ca5a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f207400e870 tx=0x7f207400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:53.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f207b400640 1 -- 172.21.15.27:0/1625026657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074017070 con 0x7f20840be5d0 2024-03-28T19:15:53.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f207b400640 1 -- 172.21.15.27:0/1625026657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207400ee60 con 0x7f20840be5d0 2024-03-28T19:15:53.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20841eb4e0 con 0x7f20840be5d0 2024-03-28T19:15:53.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f207b400640 1 -- 172.21.15.27:0/1625026657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207400cb40 con 0x7f20840be5d0 2024-03-28T19:15:53.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.795+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20841ebaa0 con 0x7f20840be5d0 2024-03-28T19:15:53.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.797+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2084069640 con 0x7f20840be5d0 2024-03-28T19:15:53.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.799+0000 7f207b400640 1 -- 172.21.15.27:0/1625026657 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f207401a070 con 0x7f20840be5d0 2024-03-28T19:15:53.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.801+0000 7f207b400640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bbd0 0x7f206406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:53.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.801+0000 7f207b400640 1 -- 172.21.15.27:0/1625026657 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7f207408bd60 con 0x7f20840be5d0 2024-03-28T19:15:53.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.801+0000 7f2082a00640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bbd0 0x7f206406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:53.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.802+0000 7f2082a00640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bbd0 0x7f206406e090 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f20841ef8d0 tx=0x7f2070008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:53.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:53.804+0000 7f207b400640 1 -- 172.21.15.27:0/1625026657 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2074059ba0 con 0x7f20840be5d0 2024-03-28T19:15:54.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.072+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2084156580 con 0x7f20840be5d0 2024-03-28T19:15:54.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.073+0000 7f207b400640 1 -- 172.21.15.27:0/1625026657 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f20740594a0 con 0x7f20840be5d0 2024-03-28T19:15:54.074 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:54.074 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:53.555186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:15:54.074 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 161 2024-03-28T19:15:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bbd0 msgr2=0x7f206406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:54.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bbd0 0x7f206406e090 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f20841ef8d0 tx=0x7f2070008040 comp rx=0 tx=0).stop 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 msgr2=0x7f20840ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f20840ca5a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f207400e870 tx=0x7f207400ed30 comp rx=0 tx=0).stop 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 shutdown_connections 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bbd0 0x7f206406e090 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c8ed0 0x7f2084154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c58c0 0x7f20840c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 --2- 172.21.15.27:0/1625026657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840be5d0 0x7f20840ca5a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 >> 172.21.15.27:0/1625026657 conn(0x7f208406b6f0 msgr2=0x7f20840afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.076+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 shutdown_connections 2024-03-28T19:15:54.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:54.077+0000 7f2089000640 1 -- 172.21.15.27:0/1625026657 wait complete. 2024-03-28T19:15:54.094 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:15:56.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:56.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:56.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.328+0000 7fc259800640 1 Processor -- start 2024-03-28T19:15:56.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.328+0000 7fc259800640 1 -- start start 2024-03-28T19:15:56.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.330+0000 7fc259800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2540727e0 0x7fc2540c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.330+0000 7fc259800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc254155600 0x7fc25414f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.330+0000 7fc259800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc254155fe0 con 0x7fc2540c59d0 2024-03-28T19:15:56.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.330+0000 7fc259800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2541565a0 con 0x7fc2540727e0 2024-03-28T19:15:56.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.330+0000 7fc259800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc254159090 con 0x7fc254155600 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.330+0000 7fc253400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2540727e0 0x7fc2540c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.330+0000 7fc253e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc2540c59d0 0x7fc254159520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54372/0 (socket says 172.21.15.27:54372) 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.330+0000 7fc253e00640 1 -- 172.21.15.27:0/78429572 learned_addr learned my addr 172.21.15.27:0/78429572 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.331+0000 7fc258e00640 1 --2- 172.21.15.27:0/78429572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc254155600 0x7fc25414f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.331+0000 7fc252a00640 1 -- 172.21.15.27:0/78429572 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460700579 0 0) 0x7fc254155fe0 con 0x7fc2540c59d0 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.331+0000 7fc252a00640 1 -- 172.21.15.27:0/78429572 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc238003520 con 0x7fc2540c59d0 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.331+0000 7fc253400640 1 -- 172.21.15.27:0/78429572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc254155600 msgr2=0x7fc25414f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.331+0000 7fc253400640 1 --2- 172.21.15.27:0/78429572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc254155600 0x7fc25414f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.331+0000 7fc253400640 1 -- 172.21.15.27:0/78429572 >> v1:172.21.15.27:6789/0 conn(0x7fc2540c59d0 legacy=0x7fc254159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.331+0000 7fc253400640 1 -- 172.21.15.27:0/78429572 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2541562c0 con 0x7fc2540727e0 2024-03-28T19:15:56.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.333+0000 7fc253400640 1 --2- 172.21.15.27:0/78429572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540727e0 0x7fc2540c8f00 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc2400097a0 tx=0x7fc24002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4a435acdf18593 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:56.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.333+0000 7fc252a00640 1 -- 172.21.15.27:0/78429572 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240038070 con 0x7fc2540727e0 2024-03-28T19:15:56.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.333+0000 7fc252a00640 1 -- 172.21.15.27:0/78429572 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc240031700 con 0x7fc2540727e0 2024-03-28T19:15:56.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.333+0000 7fc252a00640 1 -- 172.21.15.27:0/78429572 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240031ad0 con 0x7fc2540727e0 2024-03-28T19:15:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.334+0000 7fc259800640 1 -- 172.21.15.27:0/78429572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540727e0 msgr2=0x7fc2540c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.334+0000 7fc259800640 1 --2- 172.21.15.27:0/78429572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540727e0 0x7fc2540c8f00 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc2400097a0 tx=0x7fc24002b7a0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.334+0000 7fc259800640 1 -- 172.21.15.27:0/78429572 shutdown_connections 2024-03-28T19:15:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.334+0000 7fc259800640 1 --2- 172.21.15.27:0/78429572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc254155600 0x7fc25414f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.334+0000 7fc259800640 1 --2- 172.21.15.27:0/78429572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540727e0 0x7fc2540c8f00 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.334+0000 7fc259800640 1 -- 172.21.15.27:0/78429572 >> 172.21.15.27:0/78429572 conn(0x7fc2540ac910 msgr2=0x7fc2540accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.334+0000 7fc259800640 1 -- 172.21.15.27:0/78429572 shutdown_connections 2024-03-28T19:15:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.334+0000 7fc259800640 1 -- 172.21.15.27:0/78429572 wait complete. 2024-03-28T19:15:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.336+0000 7fc259800640 1 Processor -- start 2024-03-28T19:15:56.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.336+0000 7fc259800640 1 -- start start 2024-03-28T19:15:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc259800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2540727e0 0x7fc2540ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc259800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540c59d0 0x7fc2540c4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc259800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc254155600 0x7fc254147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc259800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2540c5f30 con 0x7fc2540c59d0 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc259800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2540c5db0 con 0x7fc254155600 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc259800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc254155a50 con 0x7fc2540727e0 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc253400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540c59d0 0x7fc2540c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc253400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540c59d0 0x7fc2540c4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37024/0 (socket says 172.21.15.27:37024) 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.337+0000 7fc253400640 1 -- 172.21.15.27:0/997775322 learned_addr learned my addr 172.21.15.27:0/997775322 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc258e00640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc254155600 0x7fc254147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc253e00640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2540727e0 0x7fc2540ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc253400640 1 -- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2540727e0 msgr2=0x7fc2540ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc253400640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2540727e0 0x7fc2540ab9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc253400640 1 -- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc254155600 msgr2=0x7fc254147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc253400640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc254155600 0x7fc254147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc253400640 1 -- 172.21.15.27:0/997775322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2540ac550 con 0x7fc2540c59d0 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc258e00640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc254155600 0x7fc254147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc253e00640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2540727e0 0x7fc2540ab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.338+0000 7fc253400640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540c59d0 0x7fc2540c4270 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc240036040 tx=0x7fc2400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.339+0000 7fc24be00640 1 -- 172.21.15.27:0/997775322 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240042070 con 0x7fc2540c59d0 2024-03-28T19:15:56.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.339+0000 7fc24be00640 1 -- 172.21.15.27:0/997775322 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24003d890 con 0x7fc2540c59d0 2024-03-28T19:15:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.339+0000 7fc24be00640 1 -- 172.21.15.27:0/997775322 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240038040 con 0x7fc2540c59d0 2024-03-28T19:15:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.339+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2541eab80 con 0x7fc2540c59d0 2024-03-28T19:15:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.339+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25414a590 con 0x7fc2540c59d0 2024-03-28T19:15:56.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.340+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc254067dd0 con 0x7fc2540c59d0 2024-03-28T19:15:56.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.342+0000 7fc24be00640 1 -- 172.21.15.27:0/997775322 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc24003b070 con 0x7fc2540c59d0 2024-03-28T19:15:56.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.343+0000 7fc24be00640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc23406bbd0 0x7fc23406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.344+0000 7fc24be00640 1 -- 172.21.15.27:0/997775322 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7fc2400acef0 con 0x7fc2540c59d0 2024-03-28T19:15:56.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.344+0000 7fc253e00640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc23406bbd0 0x7fc23406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.345+0000 7fc253e00640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc23406bbd0 0x7fc23406e090 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc244005880 tx=0x7fc244009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:56.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.348+0000 7fc24be00640 1 -- 172.21.15.27:0/997775322 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc2540ac550 con 0x7fc2540c59d0 2024-03-28T19:15:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.392+0000 7fb823200640 1 Processor -- start 2024-03-28T19:15:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.392+0000 7fb823200640 1 -- start start 2024-03-28T19:15:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.393+0000 7fb823200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb8240a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.393+0000 7fb823200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb824150ec0 con 0x7fb824153a30 2024-03-28T19:15:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.393+0000 7fb823200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82406dc70 con 0x7fb8240a61a0 2024-03-28T19:15:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.393+0000 7fb823200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb824153550 con 0x7fb824150ae0 2024-03-28T19:15:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.394+0000 7fb822800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb824153a30 0x7fb82415dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54382/0 (socket says 172.21.15.27:54382) 2024-03-28T19:15:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.394+0000 7fb822800640 1 -- 172.21.15.27:0/32062832 learned_addr learned my addr 172.21.15.27:0/32062832 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.394+0000 7fb821e00640 1 --2- 172.21.15.27:0/32062832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb8240a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.395+0000 7fb820a00640 1 -- 172.21.15.27:0/32062832 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614891965 0 0) 0x7fb81c001490 con 0x7fb824153a30 2024-03-28T19:15:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.395+0000 7fb820a00640 1 -- 172.21.15.27:0/32062832 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb808003520 con 0x7fb824153a30 2024-03-28T19:15:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.395+0000 7fb820a00640 1 -- 172.21.15.27:0/32062832 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186105118 0 0) 0x7fb82406dc70 con 0x7fb8240a61a0 2024-03-28T19:15:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.395+0000 7fb820a00640 1 -- 172.21.15.27:0/32062832 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb81c001490 con 0x7fb8240a61a0 2024-03-28T19:15:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.395+0000 7fb821e00640 1 -- 172.21.15.27:0/32062832 >> v1:172.21.15.155:6789/0 conn(0x7fb8240a61a0 legacy=0x7fb8240c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.395+0000 7fb821e00640 1 -- 172.21.15.27:0/32062832 >> v1:172.21.15.27:6789/0 conn(0x7fb824153a30 legacy=0x7fb82415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.395+0000 7fb821e00640 1 -- 172.21.15.27:0/32062832 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82406df10 con 0x7fb824150ae0 2024-03-28T19:15:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.396+0000 7fb821e00640 1 --2- 172.21.15.27:0/32062832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb8240a5c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb8140097d0 tx=0x7fb81402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e11426085f6ff13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.397+0000 7fb820a00640 1 -- 172.21.15.27:0/32062832 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814035020 con 0x7fb824150ae0 2024-03-28T19:15:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.397+0000 7fb820a00640 1 -- 172.21.15.27:0/32062832 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb814031810 con 0x7fb824150ae0 2024-03-28T19:15:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.397+0000 7fb820a00640 1 -- 172.21.15.27:0/32062832 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814031bc0 con 0x7fb824150ae0 2024-03-28T19:15:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.398+0000 7fb823200640 1 -- 172.21.15.27:0/32062832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 msgr2=0x7fb8240a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.398+0000 7fb823200640 1 --2- 172.21.15.27:0/32062832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb8240a5c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb8140097d0 tx=0x7fb81402b8b0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.398+0000 7fb823200640 1 -- 172.21.15.27:0/32062832 shutdown_connections 2024-03-28T19:15:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.398+0000 7fb823200640 1 --2- 172.21.15.27:0/32062832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb8240a5c60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.398+0000 7fb823200640 1 -- 172.21.15.27:0/32062832 >> 172.21.15.27:0/32062832 conn(0x7fb82406b0c0 msgr2=0x7fb82406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.398+0000 7fb823200640 1 -- 172.21.15.27:0/32062832 shutdown_connections 2024-03-28T19:15:56.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.398+0000 7fb823200640 1 -- 172.21.15.27:0/32062832 wait complete. 2024-03-28T19:15:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.400+0000 7fb823200640 1 Processor -- start 2024-03-28T19:15:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.400+0000 7fb823200640 1 -- start start 2024-03-28T19:15:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb823200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8240a61a0 0x7fb82414af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb823200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb824148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb823200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb824153a30 0x7fb824149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb823200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82406e7b0 con 0x7fb8240a61a0 2024-03-28T19:15:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb823200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82406e4d0 con 0x7fb824153a30 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb823200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8240c70b0 con 0x7fb824150ae0 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb821e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8240a61a0 0x7fb82414af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb821e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8240a61a0 0x7fb82414af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37034/0 (socket says 172.21.15.27:37034) 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb821e00640 1 -- 172.21.15.27:0/4182851474 learned_addr learned my addr 172.21.15.27:0/4182851474 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.401+0000 7fb821400640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb824148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.402+0000 7fb822800640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb824153a30 0x7fb824149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.402+0000 7fb821e00640 1 -- 172.21.15.27:0/4182851474 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 msgr2=0x7fb824148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.402+0000 7fb821e00640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb824148c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.402+0000 7fb821e00640 1 -- 172.21.15.27:0/4182851474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb824153a30 msgr2=0x7fb824149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.402+0000 7fb821e00640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb824153a30 0x7fb824149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.402+0000 7fb821e00640 1 -- 172.21.15.27:0/4182851474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb824073160 con 0x7fb8240a61a0 2024-03-28T19:15:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.402+0000 7fb821e00640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8240a61a0 0x7fb82414af20 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fb8140098f0 tx=0x7fb8140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.403+0000 7fb81a000640 1 -- 172.21.15.27:0/4182851474 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814035040 con 0x7fb8240a61a0 2024-03-28T19:15:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.403+0000 7fb81a000640 1 -- 172.21.15.27:0/4182851474 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb814003820 con 0x7fb8240a61a0 2024-03-28T19:15:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.403+0000 7fb81a000640 1 -- 172.21.15.27:0/4182851474 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814003bd0 con 0x7fb8240a61a0 2024-03-28T19:15:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.403+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb824149a50 con 0x7fb8240a61a0 2024-03-28T19:15:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.403+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8240723a0 con 0x7fb8240a61a0 2024-03-28T19:15:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.405+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb824069640 con 0x7fb8240a61a0 2024-03-28T19:15:56.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.405+0000 7fb81a000640 1 -- 172.21.15.27:0/4182851474 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb814038020 con 0x7fb8240a61a0 2024-03-28T19:15:56.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.407+0000 7fb81a000640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb80406bbd0 0x7fb80406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.407+0000 7fb81a000640 1 -- 172.21.15.27:0/4182851474 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7fb8140ad780 con 0x7fb8240a61a0 2024-03-28T19:15:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.408+0000 7fb821400640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb80406bbd0 0x7fb80406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:56.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.409+0000 7fb821400640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb80406bbd0 0x7fb80406e090 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb810007fd0 tx=0x7fb810009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:56.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.412+0000 7fb81a000640 1 -- 172.21.15.27:0/4182851474 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb814044390 con 0x7fb8240a61a0 2024-03-28T19:15:56.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.677+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2540c8ce0 con 0x7fc2540c59d0 2024-03-28T19:15:56.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.679+0000 7fc24be00640 1 -- 172.21.15.27:0/997775322 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5470 (secure 0 0 0) 0x7fc240030070 con 0x7fc2540c59d0 2024-03-28T19:15:56.679 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:56.680 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:54.926666+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:clientreplay","state_seq":10,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:15:56.680 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 163 2024-03-28T19:15:56.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.681+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc23406bbd0 msgr2=0x7fc23406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.681+0000 7fc259800640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc23406bbd0 0x7fc23406e090 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc244005880 tx=0x7fc244009040 comp rx=0 tx=0).stop 2024-03-28T19:15:56.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.681+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540c59d0 msgr2=0x7fc2540c4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.681+0000 7fc259800640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540c59d0 0x7fc2540c4270 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc240036040 tx=0x7fc2400046c0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.682+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 shutdown_connections 2024-03-28T19:15:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.682+0000 7fc259800640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc23406bbd0 0x7fc23406e090 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.682+0000 7fc259800640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc254155600 0x7fc254147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.682+0000 7fc259800640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2540c59d0 0x7fc2540c4270 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.682+0000 7fc259800640 1 --2- 172.21.15.27:0/997775322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2540727e0 0x7fc2540ab9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.682+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 >> 172.21.15.27:0/997775322 conn(0x7fc2540ac910 msgr2=0x7fc2540cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.682+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 shutdown_connections 2024-03-28T19:15:56.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.682+0000 7fc259800640 1 -- 172.21.15.27:0/997775322 wait complete. 2024-03-28T19:15:56.701 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:15:56.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.770+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb82414b460 con 0x7fb8240a61a0 2024-03-28T19:15:56.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.772+0000 7fb81a000640 1 -- 172.21.15.27:0/4182851474 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5471 (secure 0 0 0) 0x7fb81407b4c0 con 0x7fb8240a61a0 2024-03-28T19:15:56.773 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:15:56.773 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 164 2024-03-28T19:15:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb80406bbd0 msgr2=0x7fb80406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb80406bbd0 0x7fb80406e090 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb810007fd0 tx=0x7fb810009040 comp rx=0 tx=0).stop 2024-03-28T19:15:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8240a61a0 msgr2=0x7fb82414af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8240a61a0 0x7fb82414af20 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fb8140098f0 tx=0x7fb8140046c0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 shutdown_connections 2024-03-28T19:15:56.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb80406bbd0 0x7fb80406e090 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb824153a30 0x7fb824149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb824150ae0 0x7fb824148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 --2- 172.21.15.27:0/4182851474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8240a61a0 0x7fb82414af20 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 >> 172.21.15.27:0/4182851474 conn(0x7fb82406b0c0 msgr2=0x7fb8240bfd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 shutdown_connections 2024-03-28T19:15:56.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:56.775+0000 7fb823200640 1 -- 172.21.15.27:0/4182851474 wait complete. 2024-03-28T19:15:56.793 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-03-28T19:15:56.793 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:15:56.793 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:15:56.822 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:15:56.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:57.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.074+0000 7fdc41400640 1 Processor -- start 2024-03-28T19:15:57.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.074+0000 7fdc41400640 1 -- start start 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.076+0000 7fdc41400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdc3c0c58c0 0x7fdc3c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.076+0000 7fdc41400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c8ed0 0x7fdc3c1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.076+0000 7fdc41400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c156230 con 0x7fdc3c0be5d0 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.076+0000 7fdc41400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc3c158bd0 con 0x7fdc3c0c58c0 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.076+0000 7fdc41400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c158eb0 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.077+0000 7fdc3be00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdc3c0c58c0 0x7fdc3c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.077+0000 7fdc33e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c8ed0 0x7fdc3c1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.077+0000 7fdc3be00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdc3c0c58c0 0x7fdc3c0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:38010/0 (socket says 172.21.15.27:38010) 2024-03-28T19:15:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.077+0000 7fdc3be00640 1 -- 172.21.15.27:0/3514769725 learned_addr learned my addr 172.21.15.27:0/3514769725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.077+0000 7fdc3be00640 1 -- 172.21.15.27:0/3514769725 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c8ed0 msgr2=0x7fdc3c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.077+0000 7fdc3be00640 1 --2- 172.21.15.27:0/3514769725 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c8ed0 0x7fdc3c1555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.077+0000 7fdc3be00640 1 -- 172.21.15.27:0/3514769725 >> v1:172.21.15.155:6789/0 conn(0x7fdc3c0be5d0 legacy=0x7fdc3c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.077+0000 7fdc3be00640 1 -- 172.21.15.27:0/3514769725 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c159190 con 0x7fdc3c0c58c0 2024-03-28T19:15:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.078+0000 7fdc3be00640 1 --2- 172.21.15.27:0/3514769725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0c58c0 0x7fdc3c0c5ca0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009870 tx=0x7fdc2c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c246709a33feb248 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.079+0000 7fdc3b400640 1 -- 172.21.15.27:0/3514769725 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c035020 con 0x7fdc3c0c58c0 2024-03-28T19:15:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.079+0000 7fdc3b400640 1 -- 172.21.15.27:0/3514769725 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2c031640 con 0x7fdc3c0c58c0 2024-03-28T19:15:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.079+0000 7fdc3b400640 1 -- 172.21.15.27:0/3514769725 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c02d040 con 0x7fdc3c0c58c0 2024-03-28T19:15:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.080+0000 7fdc41400640 1 -- 172.21.15.27:0/3514769725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0c58c0 msgr2=0x7fdc3c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.080+0000 7fdc41400640 1 --2- 172.21.15.27:0/3514769725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0c58c0 0x7fdc3c0c5ca0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009870 tx=0x7fdc2c02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.080+0000 7fdc41400640 1 -- 172.21.15.27:0/3514769725 shutdown_connections 2024-03-28T19:15:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.080+0000 7fdc41400640 1 --2- 172.21.15.27:0/3514769725 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c8ed0 0x7fdc3c1555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.080+0000 7fdc41400640 1 --2- 172.21.15.27:0/3514769725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0c58c0 0x7fdc3c0c5ca0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.080+0000 7fdc41400640 1 -- 172.21.15.27:0/3514769725 >> 172.21.15.27:0/3514769725 conn(0x7fdc3c06b6f0 msgr2=0x7fdc3c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.080+0000 7fdc41400640 1 -- 172.21.15.27:0/3514769725 shutdown_connections 2024-03-28T19:15:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.080+0000 7fdc41400640 1 -- 172.21.15.27:0/3514769725 wait complete. 2024-03-28T19:15:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.081+0000 7fdc41400640 1 Processor -- start 2024-03-28T19:15:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.082+0000 7fdc41400640 1 -- start start 2024-03-28T19:15:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.082+0000 7fdc41400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0be5d0 0x7fdc3c0c2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.082+0000 7fdc41400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c58c0 0x7fdc3c0aa960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.082+0000 7fdc41400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3c0c8ed0 0x7fdc3c0aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.082+0000 7fdc41400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c0b0820 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.082+0000 7fdc41400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c159750 con 0x7fdc3c0be5d0 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.082+0000 7fdc41400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c0b09a0 con 0x7fdc3c0c58c0 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc3be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0be5d0 0x7fdc3c0c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc40a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3c0c8ed0 0x7fdc3c0aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc3be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0be5d0 0x7fdc3c0c2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:38014/0 (socket says 172.21.15.27:38014) 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc3be00640 1 -- 172.21.15.27:0/3299976201 learned_addr learned my addr 172.21.15.27:0/3299976201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc33e00640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c58c0 0x7fdc3c0aa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc40a00640 1 -- 172.21.15.27:0/3299976201 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c58c0 msgr2=0x7fdc3c0aa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc40a00640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c58c0 0x7fdc3c0aa960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc40a00640 1 -- 172.21.15.27:0/3299976201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0be5d0 msgr2=0x7fdc3c0c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc40a00640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0be5d0 0x7fdc3c0c2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc40a00640 1 -- 172.21.15.27:0/3299976201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c0cac30 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.083+0000 7fdc3be00640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0be5d0 0x7fdc3c0c2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:15:57.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.084+0000 7fdc40a00640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3c0c8ed0 0x7fdc3c0aaea0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c0bea40 tx=0x7fdc3400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.084+0000 7fdc38c00640 1 -- 172.21.15.27:0/3299976201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34003b70 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.084+0000 7fdc38c00640 1 -- 172.21.15.27:0/3299976201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc34003d10 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.084+0000 7fdc38c00640 1 -- 172.21.15.27:0/3299976201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34005310 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.084+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c153b20 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.084+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c153d90 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.085+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c069640 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.086+0000 7fdc38c00640 1 -- 172.21.15.27:0/3299976201 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdc340054b0 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.087+0000 7fdc38c00640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc1806bc50 0x7fdc1806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:57.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.087+0000 7fdc38c00640 1 -- 172.21.15.27:0/3299976201 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7fdc3408b3b0 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.088+0000 7fdc3be00640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc1806bc50 0x7fdc1806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:57.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.088+0000 7fdc3be00640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc1806bc50 0x7fdc1806e110 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009870 tx=0x7fdc2c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:57.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.092+0000 7fdc38c00640 1 -- 172.21.15.27:0/3299976201 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdc34058b10 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.338+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc3c156580 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.339+0000 7fdc38c00640 1 -- 172.21.15.27:0/3299976201 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5471 (secure 0 0 0) 0x7fdc3405cbb0 con 0x7fdc3c0c8ed0 2024-03-28T19:15:57.339 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:57.339 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 164 2024-03-28T19:15:57.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.341+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc1806bc50 msgr2=0x7fdc1806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:57.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.341+0000 7fdc41400640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc1806bc50 0x7fdc1806e110 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009870 tx=0x7fdc2c03c040 comp rx=0 tx=0).stop 2024-03-28T19:15:57.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.341+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3c0c8ed0 msgr2=0x7fdc3c0aaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:57.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.341+0000 7fdc41400640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3c0c8ed0 0x7fdc3c0aaea0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c0bea40 tx=0x7fdc3400cb90 comp rx=0 tx=0).stop 2024-03-28T19:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.342+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 shutdown_connections 2024-03-28T19:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.342+0000 7fdc41400640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc1806bc50 0x7fdc1806e110 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.342+0000 7fdc41400640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc3c0c8ed0 0x7fdc3c0aaea0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.342+0000 7fdc41400640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc3c0c58c0 0x7fdc3c0aa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.342+0000 7fdc41400640 1 --2- 172.21.15.27:0/3299976201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc3c0be5d0 0x7fdc3c0c2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.342+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 >> 172.21.15.27:0/3299976201 conn(0x7fdc3c06b6f0 msgr2=0x7fdc3c0cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.342+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 shutdown_connections 2024-03-28T19:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:57.342+0000 7fdc41400640 1 -- 172.21.15.27:0/3299976201 wait complete. 2024-03-28T19:15:57.361 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:15:58.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.934+0000 7f82e1600640 1 Processor -- start 2024-03-28T19:15:58.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.935+0000 7f82e1600640 1 -- start start 2024-03-28T19:15:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82e1600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82dc154140 0x7f82dc0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82e1600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82dc150ec0 con 0x7f82dc0c69a0 2024-03-28T19:15:58.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82e1600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82dc06dc70 con 0x7f82dc150ae0 2024-03-28T19:15:58.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82e1600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82dc153550 con 0x7f82dc154140 2024-03-28T19:15:58.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82db400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f82dc0c69a0 0x7f82dc153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54386/0 (socket says 172.21.15.27:54386) 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82db400640 1 -- 172.21.15.27:0/332722728 learned_addr learned my addr 172.21.15.27:0/332722728 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82e0c00640 1 --2- 172.21.15.27:0/332722728 >> v2:172.21.15.27:3300/0 conn(0x7f82dc154140 0x7f82dc0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598617518 0 0) 0x7f82c4001110 con 0x7f82dc0c69a0 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c0003520 con 0x7f82dc0c69a0 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27726954 0 0) 0x7f82c0003520 con 0x7f82dc0c69a0 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 >> v2:172.21.15.27:3300/0 conn(0x7f82dc154140 msgr2=0x7f82dc0c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82daa00640 1 --2- 172.21.15.27:0/332722728 >> v2:172.21.15.27:3300/0 conn(0x7f82dc154140 0x7f82dc0c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 >> v1:172.21.15.155:6789/0 conn(0x7f82dc150ae0 legacy=0x7f82dc0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc06df10 con 0x7f82dc0c69a0 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.937+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f82c4001840 con 0x7f82dc0c69a0 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 >> v1:172.21.15.27:6789/0 conn(0x7f82dc0c69a0 legacy=0x7f82dc153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:58.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82daa00640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00039b0 0x7f82c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82daa00640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82c0004800 0x7f82c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82daa00640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c00051c0 0x7f82c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc153550 con 0x7f82c00051c0 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c000c0c0 con 0x7f82c00039b0 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c000c2c0 con 0x7f82c0004800 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82e0c00640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c00051c0 0x7f82c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82dbe00640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00039b0 0x7f82c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82db400640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82c0004800 0x7f82c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82e0c00640 1 -- 172.21.15.27:0/332722728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82c0004800 msgr2=0x7f82c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82e0c00640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82c0004800 0x7f82c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82e0c00640 1 -- 172.21.15.27:0/332722728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00039b0 msgr2=0x7f82c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82e0c00640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00039b0 0x7f82c0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.938+0000 7f82e0c00640 1 -- 172.21.15.27:0/332722728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0003520 con 0x7f82c00051c0 2024-03-28T19:15:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.939+0000 7f82e0c00640 1 -- 172.21.15.27:0/332722728 reap_dead start 2024-03-28T19:15:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.940+0000 7f82e0c00640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c00051c0 0x7f82c000b9c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009ac0 tx=0x7f82c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebfcf48aa86c4e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.940+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc009690 con 0x7f82c00051c0 2024-03-28T19:15:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.940+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc004020 con 0x7f82c00051c0 2024-03-28T19:15:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.940+0000 7f82daa00640 1 -- 172.21.15.27:0/332722728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0043d0 con 0x7f82c00051c0 2024-03-28T19:15:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.941+0000 7f82e1600640 1 -- 172.21.15.27:0/332722728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c00051c0 msgr2=0x7f82c000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.941+0000 7f82e1600640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c00051c0 0x7f82c000b9c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009ac0 tx=0x7f82c0005830 comp rx=0 tx=0).stop 2024-03-28T19:15:58.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.941+0000 7f82e1600640 1 -- 172.21.15.27:0/332722728 shutdown_connections 2024-03-28T19:15:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.941+0000 7f82e1600640 1 --2- 172.21.15.27:0/332722728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c00051c0 0x7f82c000b9c0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.941+0000 7f82e1600640 1 -- 172.21.15.27:0/332722728 >> 172.21.15.27:0/332722728 conn(0x7f82dc06b0c0 msgr2=0x7f82dc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.941+0000 7f82e1600640 1 -- 172.21.15.27:0/332722728 shutdown_connections 2024-03-28T19:15:58.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.941+0000 7f82e1600640 1 -- 172.21.15.27:0/332722728 wait complete. 2024-03-28T19:15:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.943+0000 7f82e1600640 1 Processor -- start 2024-03-28T19:15:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.943+0000 7f82e1600640 1 -- start start 2024-03-28T19:15:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00051c0 0x7f82dc1ec670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e1600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82dc1dca40 0x7f82dc1e99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82dc14ca30 0x7f82dc155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e1600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc06e7b0 con 0x7f82dc14ca30 2024-03-28T19:15:58.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e1600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc06e4d0 con 0x7f82c00051c0 2024-03-28T19:15:58.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e1600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1569d0 con 0x7f82dc1dca40 2024-03-28T19:15:58.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82dc14ca30 0x7f82dc155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:58.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82dc14ca30 0x7f82dc155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37058/0 (socket says 172.21.15.27:37058) 2024-03-28T19:15:58.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82e0c00640 1 -- 172.21.15.27:0/3058440167 learned_addr learned my addr 172.21.15.27:0/3058440167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:58.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82db400640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82dc1dca40 0x7f82dc1e99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:58.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.944+0000 7f82dbe00640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00051c0 0x7f82dc1ec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:58.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82e0c00640 1 -- 172.21.15.27:0/3058440167 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82dc1dca40 msgr2=0x7f82dc1e99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:58.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82e0c00640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82dc1dca40 0x7f82dc1e99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:58.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82e0c00640 1 -- 172.21.15.27:0/3058440167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00051c0 msgr2=0x7f82dc1ec670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:58.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82e0c00640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00051c0 0x7f82dc1ec670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82e0c00640 1 -- 172.21.15.27:0/3058440167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc1e9210 con 0x7f82dc14ca30 2024-03-28T19:15:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82e0c00640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82dc14ca30 0x7f82dc155950 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009be0 tx=0x7f82cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82d3e00640 1 -- 172.21.15.27:0/3058440167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc02ce50 con 0x7f82dc14ca30 2024-03-28T19:15:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82d3e00640 1 -- 172.21.15.27:0/3058440167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc003610 con 0x7f82dc14ca30 2024-03-28T19:15:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82dc1dced0 con 0x7f82dc14ca30 2024-03-28T19:15:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.945+0000 7f82d3e00640 1 -- 172.21.15.27:0/3058440167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0039e0 con 0x7f82dc14ca30 2024-03-28T19:15:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.946+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc1e94b0 con 0x7f82dc14ca30 2024-03-28T19:15:58.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.947+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82dc067dd0 con 0x7f82dc14ca30 2024-03-28T19:15:58.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.947+0000 7f82d3e00640 1 -- 172.21.15.27:0/3058440167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f82cc003b80 con 0x7f82dc14ca30 2024-03-28T19:15:58.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.948+0000 7f82d3e00640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82b806bda0 0x7f82b806e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:58.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.949+0000 7f82d3e00640 1 -- 172.21.15.27:0/3058440167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7f82cc0aca80 con 0x7f82dc14ca30 2024-03-28T19:15:58.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.949+0000 7f82dbe00640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82b806bda0 0x7f82b806e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:58.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.949+0000 7f82dbe00640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82b806bda0 0x7f82b806e260 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f82d4005240 tx=0x7f82d4007f10 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:58.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:58.953+0000 7f82d3e00640 1 -- 172.21.15.27:0/3058440167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f82cc0aee20 con 0x7f82dc14ca30 2024-03-28T19:15:59.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.208+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82dc1531a0 con 0x7f82dc14ca30 2024-03-28T19:15:59.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.210+0000 7f82d3e00640 1 -- 172.21.15.27:0/3058440167 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f82cc080020 con 0x7f82dc14ca30 2024-03-28T19:15:59.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:59.210 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:15:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 164 2024-03-28T19:15:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82b806bda0 msgr2=0x7f82b806e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82b806bda0 0x7f82b806e260 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f82d4005240 tx=0x7f82d4007f10 comp rx=0 tx=0).stop 2024-03-28T19:15:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82dc14ca30 msgr2=0x7f82dc155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82dc14ca30 0x7f82dc155950 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009be0 tx=0x7f82cc004730 comp rx=0 tx=0).stop 2024-03-28T19:15:59.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 shutdown_connections 2024-03-28T19:15:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82b806bda0 0x7f82b806e260 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82dc14ca30 0x7f82dc155950 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82dc1dca40 0x7f82dc1e99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 --2- 172.21.15.27:0/3058440167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82c00051c0 0x7f82dc1ec670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 >> 172.21.15.27:0/3058440167 conn(0x7f82dc06b0c0 msgr2=0x7f82dc06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 shutdown_connections 2024-03-28T19:15:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.213+0000 7f82e1600640 1 -- 172.21.15.27:0/3058440167 wait complete. 2024-03-28T19:15:59.233 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=7317 2024-03-28T19:15:59.233 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before reviving mds.a 2024-03-28T19:15:59.362 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:15:59.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.601+0000 7f2046000640 1 Processor -- start 2024-03-28T19:15:59.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.602+0000 7f2046000640 1 -- start start 2024-03-28T19:15:59.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.603+0000 7f2046000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20400a9cb0 0x7f20400c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:59.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.603+0000 7f2046000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20401534c0 0x7f2040153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:59.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.603+0000 7f2046000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20400c3d90 con 0x7f2040150c70 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.603+0000 7f2046000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20400ab4e0 con 0x7f20400a9cb0 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.603+0000 7f2046000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20400ac1f0 con 0x7f20401534c0 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.603+0000 7f203f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2040150c70 0x7f20400c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54402/0 (socket says 172.21.15.27:54402) 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.603+0000 7f203f400640 1 -- 172.21.15.27:0/3664718883 learned_addr learned my addr 172.21.15.27:0/3664718883 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.604+0000 7f203fe00640 1 --2- 172.21.15.27:0/3664718883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20401534c0 0x7f2040153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.604+0000 7f2037e00640 1 --2- 172.21.15.27:0/3664718883 >> v2:172.21.15.155:3300/0 conn(0x7f20400a9cb0 0x7f20400c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.604+0000 7f203ea00640 1 -- 172.21.15.27:0/3664718883 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542342826 0 0) 0x7f20400c3d90 con 0x7f2040150c70 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.604+0000 7f203ea00640 1 -- 172.21.15.27:0/3664718883 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2024003520 con 0x7f2040150c70 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.604+0000 7f2037e00640 1 -- 172.21.15.27:0/3664718883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20401534c0 msgr2=0x7f2040153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:15:59.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.604+0000 7f2037e00640 1 --2- 172.21.15.27:0/3664718883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20401534c0 0x7f2040153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.604+0000 7f2037e00640 1 -- 172.21.15.27:0/3664718883 >> v1:172.21.15.27:6789/0 conn(0x7f2040150c70 legacy=0x7f20400c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:59.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.604+0000 7f2037e00640 1 -- 172.21.15.27:0/3664718883 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20400ab710 con 0x7f20400a9cb0 2024-03-28T19:15:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.605+0000 7f2037e00640 1 --2- 172.21.15.27:0/3664718883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20400a9cb0 0x7f20400c3690 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f20280097a0 tx=0x7f202802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f92f928cdfc2557f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f203ea00640 1 -- 172.21.15.27:0/3664718883 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2028031780 con 0x7f20400a9cb0 2024-03-28T19:15:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f203ea00640 1 -- 172.21.15.27:0/3664718883 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2028037070 con 0x7f20400a9cb0 2024-03-28T19:15:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f203ea00640 1 -- 172.21.15.27:0/3664718883 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2028031b40 con 0x7f20400a9cb0 2024-03-28T19:15:59.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f2046000640 1 -- 172.21.15.27:0/3664718883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20400a9cb0 msgr2=0x7f20400c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f2046000640 1 --2- 172.21.15.27:0/3664718883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20400a9cb0 0x7f20400c3690 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f20280097a0 tx=0x7f202802b820 comp rx=0 tx=0).stop 2024-03-28T19:15:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f2046000640 1 -- 172.21.15.27:0/3664718883 shutdown_connections 2024-03-28T19:15:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f2046000640 1 --2- 172.21.15.27:0/3664718883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20401534c0 0x7f2040153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f2046000640 1 --2- 172.21.15.27:0/3664718883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20400a9cb0 0x7f20400c3690 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f2046000640 1 -- 172.21.15.27:0/3664718883 >> 172.21.15.27:0/3664718883 conn(0x7f204006b9d0 msgr2=0x7f20400c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.606+0000 7f2046000640 1 -- 172.21.15.27:0/3664718883 shutdown_connections 2024-03-28T19:15:59.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.607+0000 7f2046000640 1 -- 172.21.15.27:0/3664718883 wait complete. 2024-03-28T19:15:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.608+0000 7f2046000640 1 Processor -- start 2024-03-28T19:15:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.608+0000 7f2046000640 1 -- start start 2024-03-28T19:15:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.609+0000 7f2046000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20400a9cb0 0x7f20400c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.609+0000 7f2046000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2040150c70 0x7f204006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:59.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.609+0000 7f2046000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20401534c0 0x7f20401eb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.609+0000 7f2046000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20400c1810 con 0x7f2040150c70 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.609+0000 7f2046000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20400abcd0 con 0x7f20401534c0 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.609+0000 7f2046000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20400c6830 con 0x7f20400a9cb0 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f2037e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2040150c70 0x7f204006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f203f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20400a9cb0 0x7f20400c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f2037e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2040150c70 0x7f204006d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37064/0 (socket says 172.21.15.27:37064) 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f2037e00640 1 -- 172.21.15.27:0/164543024 learned_addr learned my addr 172.21.15.27:0/164543024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f203fe00640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20401534c0 0x7f20401eb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:59.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f2037e00640 1 -- 172.21.15.27:0/164543024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20400a9cb0 msgr2=0x7f20400c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f2037e00640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20400a9cb0 0x7f20400c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f2037e00640 1 -- 172.21.15.27:0/164543024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20401534c0 msgr2=0x7f20401eb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f2037e00640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20401534c0 0x7f20401eb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.610+0000 7f2037e00640 1 -- 172.21.15.27:0/164543024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20401eebd0 con 0x7f2040150c70 2024-03-28T19:15:59.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.611+0000 7f2037e00640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2040150c70 0x7f204006d7d0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2028034040 tx=0x7f20280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:59.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.611+0000 7f2037400640 1 -- 172.21.15.27:0/164543024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2028032610 con 0x7f2040150c70 2024-03-28T19:15:59.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.611+0000 7f2037400640 1 -- 172.21.15.27:0/164543024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20280327d0 con 0x7f2040150c70 2024-03-28T19:15:59.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.611+0000 7f2037400640 1 -- 172.21.15.27:0/164543024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2028032bc0 con 0x7f2040150c70 2024-03-28T19:15:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.611+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20401eeeb0 con 0x7f2040150c70 2024-03-28T19:15:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.611+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20401ef470 con 0x7f2040150c70 2024-03-28T19:15:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.612+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204004f400 con 0x7f2040150c70 2024-03-28T19:15:59.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.613+0000 7f2037400640 1 -- 172.21.15.27:0/164543024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f202804e3f0 con 0x7f2040150c70 2024-03-28T19:15:59.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.614+0000 7f2037400640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f201c06bbd0 0x7f201c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:15:59.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.614+0000 7f2037400640 1 -- 172.21.15.27:0/164543024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7f20280ad410 con 0x7f2040150c70 2024-03-28T19:15:59.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.614+0000 7f203f400640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f201c06bbd0 0x7f201c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:15:59.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.615+0000 7f203f400640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f201c06bbd0 0x7f201c06e090 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2030005880 tx=0x7f2030009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:15:59.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.619+0000 7f2037400640 1 -- 172.21.15.27:0/164543024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2028040070 con 0x7f2040150c70 2024-03-28T19:15:59.623 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T19:15:59.623+0000 7fc331800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:15:59.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.867+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20400c5f00 con 0x7f2040150c70 2024-03-28T19:15:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.868+0000 7f2037400640 1 -- 172.21.15.27:0/164543024 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f202802f050 con 0x7f2040150c70 2024-03-28T19:15:59.867 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:15:59.867 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:15:59.867 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 164 2024-03-28T19:15:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f201c06bbd0 msgr2=0x7f201c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:59.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f201c06bbd0 0x7f201c06e090 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2030005880 tx=0x7f2030009040 comp rx=0 tx=0).stop 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2040150c70 msgr2=0x7f204006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2040150c70 0x7f204006d7d0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2028034040 tx=0x7f20280046c0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 shutdown_connections 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f201c06bbd0 0x7f201c06e090 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20401534c0 0x7f20401eb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2040150c70 0x7f204006d7d0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 --2- 172.21.15.27:0/164543024 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20400a9cb0 0x7f20400c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 >> 172.21.15.27:0/164543024 conn(0x7f204006b9d0 msgr2=0x7f20400c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 shutdown_connections 2024-03-28T19:15:59.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:15:59.871+0000 7f2046000640 1 -- 172.21.15.27:0/164543024 wait complete. 2024-03-28T19:15:59.889 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:16:01.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.129+0000 7f8397000640 1 Processor -- start 2024-03-28T19:16:02.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.129+0000 7f8397000640 1 -- start start 2024-03-28T19:16:02.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8397000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900c8ed0 0x7f83900c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:02.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8397000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83900be5d0 0x7f8390155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:02.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8397000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8390156230 con 0x7f83900c58c0 2024-03-28T19:16:02.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8397000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8390158bd0 con 0x7f83900be5d0 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8397000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8390158eb0 con 0x7f83900c8ed0 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8396600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83900be5d0 0x7f8390155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8395c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f83900c58c0 0x7f83901555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54412/0 (socket says 172.21.15.27:54412) 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8395c00640 1 -- 172.21.15.27:0/3895820616 learned_addr learned my addr 172.21.15.27:0/3895820616 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.131+0000 7f8395200640 1 --2- 172.21.15.27:0/3895820616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900c8ed0 0x7f83900c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.132+0000 7f838fe00640 1 -- 172.21.15.27:0/3895820616 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087721675 0 0) 0x7f8390156230 con 0x7f83900c58c0 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.132+0000 7f838fe00640 1 -- 172.21.15.27:0/3895820616 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8378003520 con 0x7f83900c58c0 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.132+0000 7f8396600640 1 -- 172.21.15.27:0/3895820616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900c8ed0 msgr2=0x7f83900c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.132+0000 7f8396600640 1 --2- 172.21.15.27:0/3895820616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900c8ed0 0x7f83900c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.132+0000 7f8396600640 1 -- 172.21.15.27:0/3895820616 >> v1:172.21.15.27:6789/0 conn(0x7f83900c58c0 legacy=0x7f83901555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.132+0000 7f8396600640 1 -- 172.21.15.27:0/3895820616 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390159190 con 0x7f83900be5d0 2024-03-28T19:16:02.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.133+0000 7f8396600640 1 --2- 172.21.15.27:0/3895820616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900be5d0 0x7f8390155cf0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f8388009d90 tx=0x7f838802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41012ff9e5e5d3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:02.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.134+0000 7f838fe00640 1 -- 172.21.15.27:0/3895820616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388039070 con 0x7f83900be5d0 2024-03-28T19:16:02.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.134+0000 7f838fe00640 1 -- 172.21.15.27:0/3895820616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838802cbc0 con 0x7f83900be5d0 2024-03-28T19:16:02.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.134+0000 7f838fe00640 1 -- 172.21.15.27:0/3895820616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388034440 con 0x7f83900be5d0 2024-03-28T19:16:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.134+0000 7f8397000640 1 -- 172.21.15.27:0/3895820616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900be5d0 msgr2=0x7f8390155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.134+0000 7f8397000640 1 --2- 172.21.15.27:0/3895820616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900be5d0 0x7f8390155cf0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f8388009d90 tx=0x7f838802be10 comp rx=0 tx=0).stop 2024-03-28T19:16:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.135+0000 7f8397000640 1 -- 172.21.15.27:0/3895820616 shutdown_connections 2024-03-28T19:16:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.135+0000 7f8397000640 1 --2- 172.21.15.27:0/3895820616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900be5d0 0x7f8390155cf0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.135+0000 7f8397000640 1 --2- 172.21.15.27:0/3895820616 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900c8ed0 0x7f83900c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.135+0000 7f8397000640 1 -- 172.21.15.27:0/3895820616 >> 172.21.15.27:0/3895820616 conn(0x7f839006b6f0 msgr2=0x7f83900acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.135+0000 7f8397000640 1 -- 172.21.15.27:0/3895820616 shutdown_connections 2024-03-28T19:16:02.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.135+0000 7f8397000640 1 -- 172.21.15.27:0/3895820616 wait complete. 2024-03-28T19:16:02.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.136+0000 7f8397000640 1 Processor -- start 2024-03-28T19:16:02.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.137+0000 7f8397000640 1 -- start start 2024-03-28T19:16:02.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8397000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900be5d0 0x7f8390151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:02.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8397000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900c58c0 0x7f83900ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:02.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8397000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83900c8ed0 0x7f83901ef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8397000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83900b0820 con 0x7f83900c58c0 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8397000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8390159750 con 0x7f83900c8ed0 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8397000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83900b09a0 con 0x7f83900be5d0 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8395200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900c58c0 0x7f83900ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8395200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900c58c0 0x7f83900ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37082/0 (socket says 172.21.15.27:37082) 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8395200640 1 -- 172.21.15.27:0/940458759 learned_addr learned my addr 172.21.15.27:0/940458759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8396600640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83900c8ed0 0x7f83901ef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.138+0000 7f8395c00640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900be5d0 0x7f8390151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f8395200640 1 -- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900be5d0 msgr2=0x7f8390151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:02.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f8395200640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900be5d0 0x7f8390151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f8395200640 1 -- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83900c8ed0 msgr2=0x7f83901ef8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f8395200640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83900c8ed0 0x7f83901ef8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f8395200640 1 -- 172.21.15.27:0/940458759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83901f36c0 con 0x7f83900c58c0 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f8396600640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83900c8ed0 0x7f83901ef8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f8395200640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900c58c0 0x7f83900ac1b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f837c00e870 tx=0x7f837c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f838d600640 1 -- 172.21.15.27:0/940458759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c017070 con 0x7f83900c58c0 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f838d600640 1 -- 172.21.15.27:0/940458759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c00c940 con 0x7f83900c58c0 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83901f39a0 con 0x7f83900c58c0 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.139+0000 7f838d600640 1 -- 172.21.15.27:0/940458759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c00cd30 con 0x7f83900c58c0 2024-03-28T19:16:02.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.140+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83900c4200 con 0x7f83900c58c0 2024-03-28T19:16:02.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.141+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8390069640 con 0x7f83900c58c0 2024-03-28T19:16:02.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.142+0000 7f838d600640 1 -- 172.21.15.27:0/940458759 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f837c01a070 con 0x7f83900c58c0 2024-03-28T19:16:02.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.142+0000 7f838d600640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f837006bbd0 0x7f837006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:02.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.143+0000 7f838d600640 1 -- 172.21.15.27:0/940458759 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7f837c08ba40 con 0x7f83900c58c0 2024-03-28T19:16:02.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.143+0000 7f8395c00640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f837006bbd0 0x7f837006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:02.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.144+0000 7f8395c00640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f837006bbd0 0x7f837006e090 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8384005880 tx=0x7f8384009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:02.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.147+0000 7f838d600640 1 -- 172.21.15.27:0/940458759 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f837c059800 con 0x7f83900c58c0 2024-03-28T19:16:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.403+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8390156580 con 0x7f83900c58c0 2024-03-28T19:16:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.404+0000 7f838d600640 1 -- 172.21.15.27:0/940458759 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f837c059100 con 0x7f83900c58c0 2024-03-28T19:16:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:02.404 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 164 2024-03-28T19:16:02.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f837006bbd0 msgr2=0x7f837006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f837006bbd0 0x7f837006e090 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8384005880 tx=0x7f8384009040 comp rx=0 tx=0).stop 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900c58c0 msgr2=0x7f83900ac1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900c58c0 0x7f83900ac1b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f837c00e870 tx=0x7f837c00ed30 comp rx=0 tx=0).stop 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 shutdown_connections 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f837006bbd0 0x7f837006e090 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83900c8ed0 0x7f83901ef8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83900c58c0 0x7f83900ac1b0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 --2- 172.21.15.27:0/940458759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83900be5d0 0x7f8390151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 >> 172.21.15.27:0/940458759 conn(0x7f839006b6f0 msgr2=0x7f83901569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 shutdown_connections 2024-03-28T19:16:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:02.407+0000 7f8397000640 1 -- 172.21.15.27:0/940458759 wait complete. 2024-03-28T19:16:02.428 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:16:04.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:04.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.667+0000 7f1187400640 1 Processor -- start 2024-03-28T19:16:04.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.668+0000 7f1187400640 1 -- start start 2024-03-28T19:16:04.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.669+0000 7f1187400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1180158580 0x7f11801560c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:04.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.669+0000 7f1187400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1180156600 con 0x7f11800be630 2024-03-28T19:16:04.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.669+0000 7f1187400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11800c7910 con 0x7f1180159150 2024-03-28T19:16:04.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.669+0000 7f1187400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11800c76e0 con 0x7f1180158580 2024-03-28T19:16:04.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.669+0000 7f1184a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1180158580 0x7f11801560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:04.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.669+0000 7f1184a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1180158580 0x7f11801560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37090/0 (socket says 172.21.15.27:37090) 2024-03-28T19:16:04.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.669+0000 7f1184a00640 1 -- 172.21.15.27:0/1009845771 learned_addr learned my addr 172.21.15.27:0/1009845771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:04.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.670+0000 7f1184a00640 1 -- 172.21.15.27:0/1009845771 >> v1:172.21.15.155:6789/0 conn(0x7f1180159150 legacy=0x7f1180159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.670+0000 7f1184a00640 1 -- 172.21.15.27:0/1009845771 >> v1:172.21.15.27:6789/0 conn(0x7f11800be630 legacy=0x7f11800c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:04.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.670+0000 7f1184a00640 1 -- 172.21.15.27:0/1009845771 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11800c7bf0 con 0x7f1180158580 2024-03-28T19:16:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.671+0000 7f1184a00640 1 --2- 172.21.15.27:0/1009845771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f11801560c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f1170009e50 tx=0x7f117002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1d910c61c91772 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.672+0000 7f117f400640 1 -- 172.21.15.27:0/1009845771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117002cbb0 con 0x7f1180158580 2024-03-28T19:16:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.672+0000 7f117f400640 1 -- 172.21.15.27:0/1009845771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117002cd50 con 0x7f1180158580 2024-03-28T19:16:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.672+0000 7f117f400640 1 -- 172.21.15.27:0/1009845771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170034610 con 0x7f1180158580 2024-03-28T19:16:04.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.672+0000 7f1187400640 1 -- 172.21.15.27:0/1009845771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 msgr2=0x7f11801560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.672+0000 7f1187400640 1 --2- 172.21.15.27:0/1009845771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f11801560c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f1170009e50 tx=0x7f117002c010 comp rx=0 tx=0).stop 2024-03-28T19:16:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.673+0000 7f1187400640 1 -- 172.21.15.27:0/1009845771 shutdown_connections 2024-03-28T19:16:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.673+0000 7f1187400640 1 --2- 172.21.15.27:0/1009845771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f11801560c0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.673+0000 7f1187400640 1 -- 172.21.15.27:0/1009845771 >> 172.21.15.27:0/1009845771 conn(0x7f11800caf80 msgr2=0x7f11800cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.673+0000 7f1187400640 1 -- 172.21.15.27:0/1009845771 shutdown_connections 2024-03-28T19:16:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.673+0000 7f1187400640 1 -- 172.21.15.27:0/1009845771 wait complete. 2024-03-28T19:16:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.674+0000 7f1187400640 1 Processor -- start 2024-03-28T19:16:04.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.675+0000 7f1187400640 1 -- start start 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.675+0000 7f1187400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11800be630 0x7f1180154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f1187400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f118014e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f1187400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180159150 0x7f1180153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f1187400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1180158cc0 con 0x7f1180158580 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f1187400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11800c6dd0 con 0x7f1180159150 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f1187400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11800c81b0 con 0x7f11800be630 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f117fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f118014e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f117fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f118014e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37104/0 (socket says 172.21.15.27:37104) 2024-03-28T19:16:04.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f117fe00640 1 -- 172.21.15.27:0/3452495554 learned_addr learned my addr 172.21.15.27:0/3452495554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f1184a00640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11800be630 0x7f1180154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f1185400640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180159150 0x7f1180153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f117fe00640 1 -- 172.21.15.27:0/3452495554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11800be630 msgr2=0x7f1180154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f117fe00640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11800be630 0x7f1180154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f117fe00640 1 -- 172.21.15.27:0/3452495554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180159150 msgr2=0x7f1180153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f117fe00640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180159150 0x7f1180153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:04.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.676+0000 7f117fe00640 1 -- 172.21.15.27:0/3452495554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1180159740 con 0x7f1180158580 2024-03-28T19:16:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.677+0000 7f117fe00640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f118014e2c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f116800c450 tx=0x7f116800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.677+0000 7f117cc00640 1 -- 172.21.15.27:0/3452495554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680052d0 con 0x7f1180158580 2024-03-28T19:16:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.677+0000 7f117cc00640 1 -- 172.21.15.27:0/3452495554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1168005470 con 0x7f1180158580 2024-03-28T19:16:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.677+0000 7f117cc00640 1 -- 172.21.15.27:0/3452495554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168012870 con 0x7f1180158580 2024-03-28T19:16:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.677+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11801eed40 con 0x7f1180158580 2024-03-28T19:16:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.677+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11800ad750 con 0x7f1180158580 2024-03-28T19:16:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.678+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11800ad560 con 0x7f1180158580 2024-03-28T19:16:04.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.679+0000 7f117cc00640 1 -- 172.21.15.27:0/3452495554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1168012a10 con 0x7f1180158580 2024-03-28T19:16:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.680+0000 7f117cc00640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f115c06bbd0 0x7f115c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.680+0000 7f117cc00640 1 -- 172.21.15.27:0/3452495554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7f116808b310 con 0x7f1180158580 2024-03-28T19:16:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.681+0000 7f1184a00640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f115c06bbd0 0x7f115c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.681+0000 7f1184a00640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f115c06bbd0 0x7f115c06e090 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1170035000 tx=0x7f1170030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:04.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.685+0000 7f117cc00640 1 -- 172.21.15.27:0/3452495554 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1180156600 con 0x7f1180158580 2024-03-28T19:16:04.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:16:04.878 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:16:04.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.934+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11800729e0 con 0x7f1180158580 2024-03-28T19:16:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.935+0000 7f117cc00640 1 -- 172.21.15.27:0/3452495554 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f11680589f0 con 0x7f1180158580 2024-03-28T19:16:04.935 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:04.935 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 164 2024-03-28T19:16:04.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.937+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f115c06bbd0 msgr2=0x7f115c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.937+0000 7f1187400640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f115c06bbd0 0x7f115c06e090 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1170035000 tx=0x7f1170030000 comp rx=0 tx=0).stop 2024-03-28T19:16:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.937+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 msgr2=0x7f118014e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.937+0000 7f1187400640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f118014e2c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f116800c450 tx=0x7f116800c910 comp rx=0 tx=0).stop 2024-03-28T19:16:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.938+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 shutdown_connections 2024-03-28T19:16:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.938+0000 7f1187400640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f115c06bbd0 0x7f115c06e090 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.938+0000 7f1187400640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180159150 0x7f1180153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:04.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.938+0000 7f1187400640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1180158580 0x7f118014e2c0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:04.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.938+0000 7f1187400640 1 --2- 172.21.15.27:0/3452495554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11800be630 0x7f1180154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:04.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.938+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 >> 172.21.15.27:0/3452495554 conn(0x7f11800caf80 msgr2=0x7f11800c9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:04.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.938+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 shutdown_connections 2024-03-28T19:16:04.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:04.938+0000 7f1187400640 1 -- 172.21.15.27:0/3452495554 wait complete. 2024-03-28T19:16:04.955 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:16:06.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.190+0000 7f68df600640 1 Processor -- start 2024-03-28T19:16:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.190+0000 7f68df600640 1 -- start start 2024-03-28T19:16:07.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.191+0000 7f68df600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68d80a9cb0 0x7f68d80c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:07.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.191+0000 7f68df600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d8154020 con 0x7f68d8150c70 2024-03-28T19:16:07.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68df600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d80c3d90 con 0x7f68d81534c0 2024-03-28T19:16:07.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68df600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d80ab4e0 con 0x7f68d80a9cb0 2024-03-28T19:16:07.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68d7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68d80a9cb0 0x7f68d80c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:07.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68d7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68d80a9cb0 0x7f68d80c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48288/0 (socket says 172.21.15.27:48288) 2024-03-28T19:16:07.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68d7e00640 1 -- 172.21.15.27:0/263281010 learned_addr learned my addr 172.21.15.27:0/263281010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68d7e00640 1 -- 172.21.15.27:0/263281010 >> v1:172.21.15.155:6789/0 conn(0x7f68d81534c0 legacy=0x7f68d8153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68d7e00640 1 -- 172.21.15.27:0/263281010 >> v1:172.21.15.27:6789/0 conn(0x7f68d8150c70 legacy=0x7f68d80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68d7400640 1 -- 172.21.15.27:0/263281010 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126213434 0 0) 0x7f68d8154020 con 0x7f68d8150c70 2024-03-28T19:16:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.192+0000 7f68d7e00640 1 -- 172.21.15.27:0/263281010 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d80ab710 con 0x7f68d80a9cb0 2024-03-28T19:16:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.194+0000 7f68d7e00640 1 --2- 172.21.15.27:0/263281010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d80a9cb0 0x7f68d80c3690 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009870 tx=0x7f68c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fae25f8e5c71fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.194+0000 7f68d7400640 1 -- 172.21.15.27:0/263281010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0035020 con 0x7f68d80a9cb0 2024-03-28T19:16:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.194+0000 7f68d7400640 1 -- 172.21.15.27:0/263281010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0031870 con 0x7f68d80a9cb0 2024-03-28T19:16:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.194+0000 7f68d7400640 1 -- 172.21.15.27:0/263281010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0031c40 con 0x7f68d80a9cb0 2024-03-28T19:16:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.194+0000 7f68df600640 1 -- 172.21.15.27:0/263281010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d80a9cb0 msgr2=0x7f68d80c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.194+0000 7f68df600640 1 --2- 172.21.15.27:0/263281010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d80a9cb0 0x7f68d80c3690 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009870 tx=0x7f68c002b910 comp rx=0 tx=0).stop 2024-03-28T19:16:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.195+0000 7f68df600640 1 -- 172.21.15.27:0/263281010 shutdown_connections 2024-03-28T19:16:07.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.195+0000 7f68df600640 1 --2- 172.21.15.27:0/263281010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d80a9cb0 0x7f68d80c3690 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:07.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.195+0000 7f68df600640 1 -- 172.21.15.27:0/263281010 >> 172.21.15.27:0/263281010 conn(0x7f68d806b9d0 msgr2=0x7f68d80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:07.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.195+0000 7f68df600640 1 -- 172.21.15.27:0/263281010 shutdown_connections 2024-03-28T19:16:07.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.195+0000 7f68df600640 1 -- 172.21.15.27:0/263281010 wait complete. 2024-03-28T19:16:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.196+0000 7f68df600640 1 Processor -- start 2024-03-28T19:16:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.196+0000 7f68df600640 1 -- start start 2024-03-28T19:16:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.197+0000 7f68df600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d80a9cb0 0x7f68d814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.197+0000 7f68df600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d8150c70 0x7f68d80a7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.197+0000 7f68df600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d81534c0 0x7f68d80a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.197+0000 7f68df600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80abcd0 con 0x7f68d8150c70 2024-03-28T19:16:07.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.197+0000 7f68df600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80ac1f0 con 0x7f68d81534c0 2024-03-28T19:16:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.197+0000 7f68df600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80c1810 con 0x7f68d80a9cb0 2024-03-28T19:16:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68d7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d8150c70 0x7f68d80a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68d7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d8150c70 0x7f68d80a7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48302/0 (socket says 172.21.15.27:48302) 2024-03-28T19:16:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68d7e00640 1 -- 172.21.15.27:0/1836639324 learned_addr learned my addr 172.21.15.27:0/1836639324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68dcc00640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d80a9cb0 0x7f68d814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68dd600640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d81534c0 0x7f68d80a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:07.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68d7e00640 1 -- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d80a9cb0 msgr2=0x7f68d814dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68d7e00640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d80a9cb0 0x7f68d814dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68d7e00640 1 -- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d81534c0 msgr2=0x7f68d80a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68d7e00640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d81534c0 0x7f68d80a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68d7e00640 1 -- 172.21.15.27:0/1836639324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d80a5890 con 0x7f68d8150c70 2024-03-28T19:16:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.198+0000 7f68dcc00640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d80a9cb0 0x7f68d814dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:16:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.199+0000 7f68d7e00640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d8150c70 0x7f68d80a7310 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f68c00090b0 tx=0x7f68c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.199+0000 7f68d4c00640 1 -- 172.21.15.27:0/1836639324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c003bb60 con 0x7f68d8150c70 2024-03-28T19:16:07.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.199+0000 7f68d4c00640 1 -- 172.21.15.27:0/1836639324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c003bd20 con 0x7f68d8150c70 2024-03-28T19:16:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.199+0000 7f68d4c00640 1 -- 172.21.15.27:0/1836639324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0004270 con 0x7f68d8150c70 2024-03-28T19:16:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.199+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d80a7c50 con 0x7f68d8150c70 2024-03-28T19:16:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.199+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d81f3180 con 0x7f68d8150c70 2024-03-28T19:16:07.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.200+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d81f3560 con 0x7f68d8150c70 2024-03-28T19:16:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.201+0000 7f68d4c00640 1 -- 172.21.15.27:0/1836639324 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f68c0038020 con 0x7f68d8150c70 2024-03-28T19:16:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.202+0000 7f68d4c00640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b406bc20 0x7f68b406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.202+0000 7f68d4c00640 1 -- 172.21.15.27:0/1836639324 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7f68c00ad860 con 0x7f68d8150c70 2024-03-28T19:16:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.203+0000 7f68dcc00640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b406bc20 0x7f68b406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.203+0000 7f68dcc00640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b406bc20 0x7f68b406e0e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f68c80012e0 tx=0x7f68c8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.207+0000 7f68d4c00640 1 -- 172.21.15.27:0/1836639324 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f68d81f3560 con 0x7f68d8150c70 2024-03-28T19:16:07.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.458+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68d804f400 con 0x7f68d8150c70 2024-03-28T19:16:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.459+0000 7f68d4c00640 1 -- 172.21.15.27:0/1836639324 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f68c0040070 con 0x7f68d8150c70 2024-03-28T19:16:07.458 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:07.459 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:07.459 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 164 2024-03-28T19:16:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b406bc20 msgr2=0x7f68b406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:07.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b406bc20 0x7f68b406e0e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f68c80012e0 tx=0x7f68c8009040 comp rx=0 tx=0).stop 2024-03-28T19:16:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d8150c70 msgr2=0x7f68d80a7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d8150c70 0x7f68d80a7310 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f68c00090b0 tx=0x7f68c00046c0 comp rx=0 tx=0).stop 2024-03-28T19:16:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 shutdown_connections 2024-03-28T19:16:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f68b406bc20 0x7f68b406e0e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68d81534c0 0x7f68d80a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68d8150c70 0x7f68d80a7310 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 --2- 172.21.15.27:0/1836639324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68d80a9cb0 0x7f68d814dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 >> 172.21.15.27:0/1836639324 conn(0x7f68d806b9d0 msgr2=0x7f68d8151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 shutdown_connections 2024-03-28T19:16:07.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:07.462+0000 7f68df600640 1 -- 172.21.15.27:0/1836639324 wait complete. 2024-03-28T19:16:07.480 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:16:08.233 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-03-28T19:16:08.234 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:16:08.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:16:08.242 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:16:08.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:08.424 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:16:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.464+0000 7f3b45400640 1 Processor -- start 2024-03-28T19:16:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.464+0000 7f3b45400640 1 -- start start 2024-03-28T19:16:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.465+0000 7f3b45400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3b400c60e0 0x7f3b4006e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.465+0000 7f3b45400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b40072df0 0x7f3b4006ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.465+0000 7f3b45400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b400c1600 con 0x7f3b400c3ea0 2024-03-28T19:16:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.465+0000 7f3b45400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b400c23b0 con 0x7f3b40072df0 2024-03-28T19:16:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.465+0000 7f3b45400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b400c18e0 con 0x7f3b400c60e0 2024-03-28T19:16:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b35e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3b400c3ea0 0x7f3b40158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56512/0 (socket says 172.21.15.27:56512) 2024-03-28T19:16:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b35e00640 1 -- 172.21.15.27:0/3130882415 learned_addr learned my addr 172.21.15.27:0/3130882415 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b3f200640 1 --2- 172.21.15.27:0/3130882415 >> v2:172.21.15.27:3300/0 conn(0x7f3b40072df0 0x7f3b4006ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b3e800640 1 --2- 172.21.15.27:0/3130882415 >> v2:172.21.15.155:3300/0 conn(0x7f3b400c60e0 0x7f3b4006e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b3de00640 1 -- 172.21.15.27:0/3130882415 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678539163 0 0) 0x7f3b400c1600 con 0x7f3b400c3ea0 2024-03-28T19:16:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b3de00640 1 -- 172.21.15.27:0/3130882415 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b24003520 con 0x7f3b400c3ea0 2024-03-28T19:16:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b3f200640 1 -- 172.21.15.27:0/3130882415 >> v2:172.21.15.155:3300/0 conn(0x7f3b400c60e0 msgr2=0x7f3b4006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b3f200640 1 --2- 172.21.15.27:0/3130882415 >> v2:172.21.15.155:3300/0 conn(0x7f3b400c60e0 0x7f3b4006e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b3f200640 1 -- 172.21.15.27:0/3130882415 >> v1:172.21.15.27:6789/0 conn(0x7f3b400c3ea0 legacy=0x7f3b40158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.466+0000 7f3b3f200640 1 -- 172.21.15.27:0/3130882415 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400c1bc0 con 0x7f3b40072df0 2024-03-28T19:16:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.467+0000 7f3b3f200640 1 --2- 172.21.15.27:0/3130882415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b40072df0 0x7f3b4006ec30 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f3b400c10d0 tx=0x7f3b3802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3059f5700456b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b3de00640 1 -- 172.21.15.27:0/3130882415 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b380315a0 con 0x7f3b40072df0 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b3de00640 1 -- 172.21.15.27:0/3130882415 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b38037070 con 0x7f3b40072df0 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b3de00640 1 -- 172.21.15.27:0/3130882415 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38031960 con 0x7f3b40072df0 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b45400640 1 -- 172.21.15.27:0/3130882415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b40072df0 msgr2=0x7f3b4006ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b45400640 1 --2- 172.21.15.27:0/3130882415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b40072df0 0x7f3b4006ec30 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f3b400c10d0 tx=0x7f3b3802b840 comp rx=0 tx=0).stop 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b45400640 1 -- 172.21.15.27:0/3130882415 shutdown_connections 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b45400640 1 --2- 172.21.15.27:0/3130882415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b40072df0 0x7f3b4006ec30 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b45400640 1 --2- 172.21.15.27:0/3130882415 >> v2:172.21.15.155:3300/0 conn(0x7f3b400c60e0 0x7f3b4006e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b45400640 1 -- 172.21.15.27:0/3130882415 >> 172.21.15.27:0/3130882415 conn(0x7f3b4006b9d0 msgr2=0x7f3b400c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.468+0000 7f3b45400640 1 -- 172.21.15.27:0/3130882415 shutdown_connections 2024-03-28T19:16:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.469+0000 7f3b45400640 1 -- 172.21.15.27:0/3130882415 wait complete. 2024-03-28T19:16:08.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.470+0000 7f3b45400640 1 Processor -- start 2024-03-28T19:16:08.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.470+0000 7f3b45400640 1 -- start start 2024-03-28T19:16:08.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.471+0000 7f3b45400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b40072df0 0x7f3b401ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:08.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.471+0000 7f3b45400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b400c3ea0 0x7f3b400722b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:08.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.471+0000 7f3b45400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b400c60e0 0x7f3b401eb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:08.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.471+0000 7f3b45400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400c6640 con 0x7f3b400c60e0 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.471+0000 7f3b45400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400c64c0 con 0x7f3b400c3ea0 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.471+0000 7f3b45400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400c4f00 con 0x7f3b40072df0 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b400c60e0 0x7f3b401eb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b400c60e0 0x7f3b401eb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48326/0 (socket says 172.21.15.27:48326) 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 -- 172.21.15.27:0/4071620750 learned_addr learned my addr 172.21.15.27:0/4071620750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b35e00640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b400c3ea0 0x7f3b400722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3e800640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b40072df0 0x7f3b401ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 -- 172.21.15.27:0/4071620750 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b40072df0 msgr2=0x7f3b401ef2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b40072df0 0x7f3b401ef2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 -- 172.21.15.27:0/4071620750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b400c3ea0 msgr2=0x7f3b400722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:08.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b400c3ea0 0x7f3b400722b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 -- 172.21.15.27:0/4071620750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400ab560 con 0x7f3b400c60e0 2024-03-28T19:16:08.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.472+0000 7f3b3f200640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b400c60e0 0x7f3b401eb960 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009890 tx=0x7f3b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:08.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.473+0000 7f3b37400640 1 -- 172.21.15.27:0/4071620750 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38032630 con 0x7f3b400c60e0 2024-03-28T19:16:08.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.473+0000 7f3b37400640 1 -- 172.21.15.27:0/4071620750 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b380327f0 con 0x7f3b400c60e0 2024-03-28T19:16:08.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.473+0000 7f3b37400640 1 -- 172.21.15.27:0/4071620750 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38032be0 con 0x7f3b400c60e0 2024-03-28T19:16:08.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.473+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b400ab840 con 0x7f3b400c60e0 2024-03-28T19:16:08.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.473+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b400abe00 con 0x7f3b400c60e0 2024-03-28T19:16:08.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.474+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4004f400 con 0x7f3b400c60e0 2024-03-28T19:16:08.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.475+0000 7f3b37400640 1 -- 172.21.15.27:0/4071620750 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3b3804c020 con 0x7f3b400c60e0 2024-03-28T19:16:08.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.476+0000 7f3b37400640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1c06bbd0 0x7f3b1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:08.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.476+0000 7f3b37400640 1 -- 172.21.15.27:0/4071620750 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 9045+0+0 (secure 0 0 0) 0x7f3b3802f020 con 0x7f3b400c60e0 2024-03-28T19:16:08.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.476+0000 7f3b3e800640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1c06bbd0 0x7f3b1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:08.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.477+0000 7f3b3e800640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1c06bbd0 0x7f3b1c06e090 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3b400a6bb0 tx=0x7f3b3000b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.481+0000 7f3b37400640 1 -- 172.21.15.27:0/4071620750 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3b38040070 con 0x7f3b400c60e0 2024-03-28T19:16:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.728+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b400c4ce0 con 0x7f3b400c60e0 2024-03-28T19:16:08.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.729+0000 7f3b37400640 1 -- 172.21.15.27:0/4071620750 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f3b3807b140 con 0x7f3b400c60e0 2024-03-28T19:16:08.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:08.728 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:14:41.786270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7146},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7146":{"gid":7146,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":12,"addr":"172.21.15.27:6835/3913435536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3913435536},{"type":"v1","addr":"172.21.15.27:6835","nonce":3913435536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7146,"qdb_cluster":[7146]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:08.729 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 164 2024-03-28T19:16:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1c06bbd0 msgr2=0x7f3b1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1c06bbd0 0x7f3b1c06e090 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3b400a6bb0 tx=0x7f3b3000b040 comp rx=0 tx=0).stop 2024-03-28T19:16:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b400c60e0 msgr2=0x7f3b401eb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b400c60e0 0x7f3b401eb960 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009890 tx=0x7f3b380046c0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 shutdown_connections 2024-03-28T19:16:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b1c06bbd0 0x7f3b1c06e090 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b400c60e0 0x7f3b401eb960 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b400c3ea0 0x7f3b400722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.731+0000 7f3b45400640 1 --2- 172.21.15.27:0/4071620750 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3b40072df0 0x7f3b401ef2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.732+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 >> 172.21.15.27:0/4071620750 conn(0x7f3b4006b9d0 msgr2=0x7f3b400c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.732+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 shutdown_connections 2024-03-28T19:16:08.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:08.732+0000 7f3b45400640 1 -- 172.21.15.27:0/4071620750 wait complete. 2024-03-28T19:16:08.752 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:16:09.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.700+0000 7fd4e6800640 1 Processor -- start 2024-03-28T19:16:09.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.701+0000 7fd4e6800640 1 -- start start 2024-03-28T19:16:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.702+0000 7fd4e6800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c8ed0 0x7fd4e00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.702+0000 7fd4e6800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4e00be5d0 0x7fd4e0155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.702+0000 7fd4e6800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e0156230 con 0x7fd4e00c58c0 2024-03-28T19:16:09.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.702+0000 7fd4e6800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e0158bd0 con 0x7fd4e00be5d0 2024-03-28T19:16:09.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.702+0000 7fd4e6800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e0158eb0 con 0x7fd4e00c8ed0 2024-03-28T19:16:09.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.703+0000 7fd4e5e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4e00be5d0 0x7fd4e0155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:09.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.703+0000 7fd4e4a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c8ed0 0x7fd4e00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:09.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.703+0000 7fd4e5e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4e00be5d0 0x7fd4e0155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:56978/0 (socket says 172.21.15.27:56978) 2024-03-28T19:16:09.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.703+0000 7fd4e5e00640 1 -- 172.21.15.27:0/1152953371 learned_addr learned my addr 172.21.15.27:0/1152953371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:09.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.703+0000 7fd4e5e00640 1 -- 172.21.15.27:0/1152953371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c8ed0 msgr2=0x7fd4e00c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:09.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.703+0000 7fd4e5e00640 1 --2- 172.21.15.27:0/1152953371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c8ed0 0x7fd4e00c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:09.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.703+0000 7fd4e5e00640 1 -- 172.21.15.27:0/1152953371 >> v1:172.21.15.155:6789/0 conn(0x7fd4e00c58c0 legacy=0x7fd4e01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:09.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.703+0000 7fd4e5e00640 1 -- 172.21.15.27:0/1152953371 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e0159190 con 0x7fd4e00be5d0 2024-03-28T19:16:09.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.705+0000 7fd4e5e00640 1 --2- 172.21.15.27:0/1152953371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e0155cf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009e50 tx=0x7fd4d802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0c6e5c8c116b790 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:09.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.706+0000 7fd4dfe00640 1 -- 172.21.15.27:0/1152953371 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d802ca80 con 0x7fd4e00be5d0 2024-03-28T19:16:09.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.706+0000 7fd4dfe00640 1 -- 172.21.15.27:0/1152953371 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d802cc20 con 0x7fd4e00be5d0 2024-03-28T19:16:09.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.706+0000 7fd4dfe00640 1 -- 172.21.15.27:0/1152953371 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80343f0 con 0x7fd4e00be5d0 2024-03-28T19:16:09.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.706+0000 7fd4e6800640 1 -- 172.21.15.27:0/1152953371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 msgr2=0x7fd4e0155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:09.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.706+0000 7fd4e6800640 1 --2- 172.21.15.27:0/1152953371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e0155cf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009e50 tx=0x7fd4d802bee0 comp rx=0 tx=0).stop 2024-03-28T19:16:09.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.707+0000 7fd4e6800640 1 -- 172.21.15.27:0/1152953371 shutdown_connections 2024-03-28T19:16:09.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.707+0000 7fd4e6800640 1 --2- 172.21.15.27:0/1152953371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e0155cf0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:09.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.707+0000 7fd4e6800640 1 --2- 172.21.15.27:0/1152953371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c8ed0 0x7fd4e00c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.707+0000 7fd4e6800640 1 -- 172.21.15.27:0/1152953371 >> 172.21.15.27:0/1152953371 conn(0x7fd4e006b6f0 msgr2=0x7fd4e00acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.707+0000 7fd4e6800640 1 -- 172.21.15.27:0/1152953371 shutdown_connections 2024-03-28T19:16:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.707+0000 7fd4e6800640 1 -- 172.21.15.27:0/1152953371 wait complete. 2024-03-28T19:16:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.708+0000 7fd4e6800640 1 Processor -- start 2024-03-28T19:16:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.709+0000 7fd4e6800640 1 -- start start 2024-03-28T19:16:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.709+0000 7fd4e6800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e00c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.709+0000 7fd4e6800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c58c0 0x7fd4e0154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e00c8ed0 0x7fd4e0151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e6800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e00b0820 con 0x7fd4e00c8ed0 2024-03-28T19:16:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e6800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e0159750 con 0x7fd4e00be5d0 2024-03-28T19:16:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e6800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e00b09a0 con 0x7fd4e00c58c0 2024-03-28T19:16:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e5400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e00c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e5400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e00c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:56988/0 (socket says 172.21.15.27:56988) 2024-03-28T19:16:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e00c8ed0 0x7fd4e0151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e5400640 1 -- 172.21.15.27:0/222240327 learned_addr learned my addr 172.21.15.27:0/222240327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.710+0000 7fd4e4a00640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c58c0 0x7fd4e0154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.711+0000 7fd4e5400640 1 -- 172.21.15.27:0/222240327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c58c0 msgr2=0x7fd4e0154790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.711+0000 7fd4e5400640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c58c0 0x7fd4e0154790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.711+0000 7fd4e5400640 1 -- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e00c8ed0 msgr2=0x7fd4e0151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.711+0000 7fd4e5400640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e00c8ed0 0x7fd4e0151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.711+0000 7fd4e5400640 1 -- 172.21.15.27:0/222240327 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e01543a0 con 0x7fd4e00be5d0 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.711+0000 7fd4e5e00640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e00c8ed0 0x7fd4e0151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.711+0000 7fd4e5400640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e00c46f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400c3c0 tx=0x7fd4d400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.711+0000 7fd4dd600640 1 -- 172.21.15.27:0/222240327 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40052d0 con 0x7fd4e00be5d0 2024-03-28T19:16:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.712+0000 7fd4dd600640 1 -- 172.21.15.27:0/222240327 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d4005490 con 0x7fd4e00be5d0 2024-03-28T19:16:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.712+0000 7fd4dd600640 1 -- 172.21.15.27:0/222240327 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4007260 con 0x7fd4e00be5d0 2024-03-28T19:16:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.712+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e01eb6f0 con 0x7fd4e00be5d0 2024-03-28T19:16:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.712+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e01eb990 con 0x7fd4e00be5d0 2024-03-28T19:16:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.713+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e0069640 con 0x7fd4e00be5d0 2024-03-28T19:16:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.714+0000 7fd4dd600640 1 -- 172.21.15.27:0/222240327 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd4d4007400 con 0x7fd4e00be5d0 2024-03-28T19:16:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.715+0000 7fd4dd600640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4c006bbd0 0x7fd4c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.716+0000 7fd4dd600640 1 -- 172.21.15.27:0/222240327 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7fd4d408ac60 con 0x7fd4e00be5d0 2024-03-28T19:16:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.716+0000 7fd4e4a00640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4c006bbd0 0x7fd4c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.717+0000 7fd4e4a00640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4c006bbd0 0x7fd4c006e090 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00ac520 tx=0x7fd4cc009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:09.721+0000 7fd4dd600640 1 -- 172.21.15.27:0/222240327 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd4e01543a0 con 0x7fd4e00be5d0 2024-03-28T19:16:10.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.014+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4e0156580 con 0x7fd4e00be5d0 2024-03-28T19:16:10.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.016+0000 7fd4dd600640 1 -- 172.21.15.27:0/222240327 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fd4d4058380 con 0x7fd4e00be5d0 2024-03-28T19:16:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:10.015 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:08.833799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:10.016 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 166 2024-03-28T19:16:10.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4c006bbd0 msgr2=0x7fd4c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4c006bbd0 0x7fd4c006e090 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00ac520 tx=0x7fd4cc009190 comp rx=0 tx=0).stop 2024-03-28T19:16:10.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 msgr2=0x7fd4e00c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e00c46f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400c3c0 tx=0x7fd4d400c880 comp rx=0 tx=0).stop 2024-03-28T19:16:10.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 shutdown_connections 2024-03-28T19:16:10.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd4c006bbd0 0x7fd4c006e090 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4e00c8ed0 0x7fd4e0151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4e00c58c0 0x7fd4e0154790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 --2- 172.21.15.27:0/222240327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4e00be5d0 0x7fd4e00c46f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 >> 172.21.15.27:0/222240327 conn(0x7fd4e006b6f0 msgr2=0x7fd4e00ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:10.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.018+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 shutdown_connections 2024-03-28T19:16:10.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.019+0000 7fd4e6800640 1 -- 172.21.15.27:0/222240327 wait complete. 2024-03-28T19:16:10.034 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-03-28T19:16:10.034 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:16:10.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:10.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.243+0000 7fce41600640 1 Processor -- start 2024-03-28T19:16:10.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.244+0000 7fce41600640 1 -- start start 2024-03-28T19:16:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.245+0000 7fce41600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fce3c153bd0 0x7fce3c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.245+0000 7fce41600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3c154020 con 0x7fce3c0a9cb0 2024-03-28T19:16:10.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.245+0000 7fce41600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3c0c3d90 con 0x7fce3c150c70 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.245+0000 7fce41600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3c0ab4e0 con 0x7fce3c153bd0 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.246+0000 7fce3a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fce3c0a9cb0 0x7fce3c1534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56516/0 (socket says 172.21.15.27:56516) 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.246+0000 7fce3a000640 1 -- 172.21.15.27:0/3159334547 learned_addr learned my addr 172.21.15.27:0/3159334547 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.246+0000 7fce3b400640 1 --2- 172.21.15.27:0/3159334547 >> v2:172.21.15.155:3300/0 conn(0x7fce3c153bd0 0x7fce3c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.246+0000 7fce39600640 1 -- 172.21.15.27:0/3159334547 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300883130 0 0) 0x7fce3c154020 con 0x7fce3c0a9cb0 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.246+0000 7fce39600640 1 -- 172.21.15.27:0/3159334547 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce20003520 con 0x7fce3c0a9cb0 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.246+0000 7fce3b400640 1 -- 172.21.15.27:0/3159334547 >> v1:172.21.15.155:6789/0 conn(0x7fce3c150c70 legacy=0x7fce3c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.246+0000 7fce3b400640 1 -- 172.21.15.27:0/3159334547 >> v1:172.21.15.27:6789/0 conn(0x7fce3c0a9cb0 legacy=0x7fce3c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.247+0000 7fce3b400640 1 -- 172.21.15.27:0/3159334547 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c0ab710 con 0x7fce3c153bd0 2024-03-28T19:16:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.248+0000 7fce3b400640 1 --2- 172.21.15.27:0/3159334547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c153bd0 0x7fce3c0c3690 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fce34009d90 tx=0x7fce3402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cac58f6fb59820e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.248+0000 7fce39600640 1 -- 172.21.15.27:0/3159334547 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34039070 con 0x7fce3c153bd0 2024-03-28T19:16:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.248+0000 7fce39600640 1 -- 172.21.15.27:0/3159334547 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3402cc60 con 0x7fce3c153bd0 2024-03-28T19:16:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.248+0000 7fce39600640 1 -- 172.21.15.27:0/3159334547 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34034660 con 0x7fce3c153bd0 2024-03-28T19:16:10.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.249+0000 7fce41600640 1 -- 172.21.15.27:0/3159334547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c153bd0 msgr2=0x7fce3c0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.249+0000 7fce41600640 1 --2- 172.21.15.27:0/3159334547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c153bd0 0x7fce3c0c3690 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fce34009d90 tx=0x7fce3402beb0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.249+0000 7fce41600640 1 -- 172.21.15.27:0/3159334547 shutdown_connections 2024-03-28T19:16:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.249+0000 7fce41600640 1 --2- 172.21.15.27:0/3159334547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c153bd0 0x7fce3c0c3690 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.249+0000 7fce41600640 1 -- 172.21.15.27:0/3159334547 >> 172.21.15.27:0/3159334547 conn(0x7fce3c06b9d0 msgr2=0x7fce3c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.249+0000 7fce41600640 1 -- 172.21.15.27:0/3159334547 shutdown_connections 2024-03-28T19:16:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.249+0000 7fce41600640 1 -- 172.21.15.27:0/3159334547 wait complete. 2024-03-28T19:16:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.251+0000 7fce41600640 1 Processor -- start 2024-03-28T19:16:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.251+0000 7fce41600640 1 -- start start 2024-03-28T19:16:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce41600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c0a9cb0 0x7fce3c152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce41600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce3c150c70 0x7fce3c154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce41600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c153bd0 0x7fce3c154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce41600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0abcd0 con 0x7fce3c153bd0 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce41600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0ac1f0 con 0x7fce3c0a9cb0 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce41600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0c1810 con 0x7fce3c150c70 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce3b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c153bd0 0x7fce3c154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce3b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c153bd0 0x7fce3c154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48352/0 (socket says 172.21.15.27:48352) 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce3b400640 1 -- 172.21.15.27:0/2760403528 learned_addr learned my addr 172.21.15.27:0/2760403528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce3aa00640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c0a9cb0 0x7fce3c152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.252+0000 7fce3a000640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce3c150c70 0x7fce3c154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce3b400640 1 -- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce3c150c70 msgr2=0x7fce3c154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce3b400640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce3c150c70 0x7fce3c154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce3b400640 1 -- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c0a9cb0 msgr2=0x7fce3c152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce3b400640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c0a9cb0 0x7fce3c152940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce3b400640 1 -- 172.21.15.27:0/2760403528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c157170 con 0x7fce3c153bd0 2024-03-28T19:16:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce3a000640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce3c150c70 0x7fce3c154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:16:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce3b400640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c153bd0 0x7fce3c154760 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fce34009eb0 tx=0x7fce34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce32a00640 1 -- 172.21.15.27:0/2760403528 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34039050 con 0x7fce3c153bd0 2024-03-28T19:16:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce32a00640 1 -- 172.21.15.27:0/2760403528 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce34045070 con 0x7fce3c153bd0 2024-03-28T19:16:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce32a00640 1 -- 172.21.15.27:0/2760403528 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34009450 con 0x7fce3c153bd0 2024-03-28T19:16:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.253+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c1dcbe0 con 0x7fce3c153bd0 2024-03-28T19:16:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.254+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c152e80 con 0x7fce3c153bd0 2024-03-28T19:16:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.255+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c04f400 con 0x7fce3c153bd0 2024-03-28T19:16:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.257+0000 7fce32a00640 1 -- 172.21.15.27:0/2760403528 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fce3403e850 con 0x7fce3c153bd0 2024-03-28T19:16:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.258+0000 7fce32a00640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce1c06bbd0 0x7fce1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.258+0000 7fce32a00640 1 -- 172.21.15.27:0/2760403528 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7fce340ae5a0 con 0x7fce3c153bd0 2024-03-28T19:16:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.258+0000 7fce3aa00640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce1c06bbd0 0x7fce1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.259+0000 7fce3aa00640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce1c06bbd0 0x7fce1c06e090 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fce3c150790 tx=0x7fce2c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.263+0000 7fce32a00640 1 -- 172.21.15.27:0/2760403528 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fce3407c490 con 0x7fce3c153bd0 2024-03-28T19:16:10.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.560+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce3c157340 con 0x7fce3c153bd0 2024-03-28T19:16:10.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.561+0000 7fce32a00640 1 -- 172.21.15.27:0/2760403528 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fce3407bd90 con 0x7fce3c153bd0 2024-03-28T19:16:10.560 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:10.560 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:08.833799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 166 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce1c06bbd0 msgr2=0x7fce1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce1c06bbd0 0x7fce1c06e090 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fce3c150790 tx=0x7fce2c009040 comp rx=0 tx=0).stop 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c153bd0 msgr2=0x7fce3c154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c153bd0 0x7fce3c154760 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fce34009eb0 tx=0x7fce34004a40 comp rx=0 tx=0).stop 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 shutdown_connections 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fce1c06bbd0 0x7fce1c06e090 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce3c153bd0 0x7fce3c154760 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce3c150c70 0x7fce3c154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 --2- 172.21.15.27:0/2760403528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce3c0a9cb0 0x7fce3c152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.563+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 >> 172.21.15.27:0/2760403528 conn(0x7fce3c06b9d0 msgr2=0x7fce3c0c19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.564+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 shutdown_connections 2024-03-28T19:16:10.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.564+0000 7fce41600640 1 -- 172.21.15.27:0/2760403528 wait complete. 2024-03-28T19:16:10.580 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 166, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 166, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7575": { "addr": "172.21.15.27:6835/148837773", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 148837773, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 148837773, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7575, "incarnation": 166, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:16:08.833799+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7575 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 164, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6077": { "addr": "172.21.15.155:6834/2865796032", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 2865796032, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 2865796032, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6077, "incarnation": 160, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:15:56.760391+0000", "qdb_cluster": [ 6077 ], "qdb_leader": 6077, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6077 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/938167944", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 938167944, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 938167944, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 153, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6014, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:16:10.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:10.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.960+0000 7fbacda00640 1 Processor -- start 2024-03-28T19:16:10.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.961+0000 7fbacda00640 1 -- start start 2024-03-28T19:16:10.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.963+0000 7fbacda00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.963+0000 7fbacda00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac80c1600 con 0x7fbac8072df0 2024-03-28T19:16:10.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.963+0000 7fbacda00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac80c18e0 con 0x7fbac80c3ea0 2024-03-28T19:16:10.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.963+0000 7fbacda00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80c23b0 con 0x7fbac80c60e0 2024-03-28T19:16:10.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.963+0000 7fbac6a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.963+0000 7fbac7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbac8072df0 0x7fbac8159610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56530/0 (socket says 172.21.15.27:56530) 2024-03-28T19:16:10.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.963+0000 7fbac7400640 1 -- 172.21.15.27:0/2743210604 learned_addr learned my addr 172.21.15.27:0/2743210604 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:10.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.964+0000 7fbac6a00640 1 -- 172.21.15.27:0/2743210604 >> v1:172.21.15.155:6789/0 conn(0x7fbac80c3ea0 legacy=0x7fbac8158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.964+0000 7fbac6a00640 1 -- 172.21.15.27:0/2743210604 >> v1:172.21.15.27:6789/0 conn(0x7fbac8072df0 legacy=0x7fbac8159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.964+0000 7fbac5600640 1 -- 172.21.15.27:0/2743210604 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628164795 0 0) 0x7fbac80c1600 con 0x7fbac8072df0 2024-03-28T19:16:10.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.964+0000 7fbac6a00640 1 -- 172.21.15.27:0/2743210604 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac80c1bc0 con 0x7fbac80c60e0 2024-03-28T19:16:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.965+0000 7fbac6a00640 1 --2- 172.21.15.27:0/2743210604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac806e6f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbab80097d0 tx=0x7fbab802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ad9c5a7b31c923 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.966+0000 7fbac5600640 1 -- 172.21.15.27:0/2743210604 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8031770 con 0x7fbac80c60e0 2024-03-28T19:16:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.966+0000 7fbac5600640 1 -- 172.21.15.27:0/2743210604 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab8037070 con 0x7fbac80c60e0 2024-03-28T19:16:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.966+0000 7fbac5600640 1 -- 172.21.15.27:0/2743210604 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8031b30 con 0x7fbac80c60e0 2024-03-28T19:16:10.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.966+0000 7fbacda00640 1 -- 172.21.15.27:0/2743210604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 msgr2=0x7fbac806e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.966+0000 7fbacda00640 1 --2- 172.21.15.27:0/2743210604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac806e6f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbab80097d0 tx=0x7fbab802b810 comp rx=0 tx=0).stop 2024-03-28T19:16:10.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.967+0000 7fbacda00640 1 -- 172.21.15.27:0/2743210604 shutdown_connections 2024-03-28T19:16:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.967+0000 7fbacda00640 1 --2- 172.21.15.27:0/2743210604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac806e6f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.967+0000 7fbacda00640 1 -- 172.21.15.27:0/2743210604 >> 172.21.15.27:0/2743210604 conn(0x7fbac806b9d0 msgr2=0x7fbac80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.967+0000 7fbacda00640 1 -- 172.21.15.27:0/2743210604 shutdown_connections 2024-03-28T19:16:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.967+0000 7fbacda00640 1 -- 172.21.15.27:0/2743210604 wait complete. 2024-03-28T19:16:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.968+0000 7fbacda00640 1 Processor -- start 2024-03-28T19:16:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.968+0000 7fbacda00640 1 -- start start 2024-03-28T19:16:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.969+0000 7fbacda00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbac8072df0 0x7fbac80aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.969+0000 7fbacda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac80c3ea0 0x7fbac81eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.969+0000 7fbacda00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac8071f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.969+0000 7fbacda00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80c6640 con 0x7fbac80c3ea0 2024-03-28T19:16:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.969+0000 7fbacda00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80c64c0 con 0x7fbac8072df0 2024-03-28T19:16:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.969+0000 7fbacda00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80c4f00 con 0x7fbac80c60e0 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac80c3ea0 0x7fbac81eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac7400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac8071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac80c3ea0 0x7fbac81eb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48354/0 (socket says 172.21.15.27:48354) 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6000640 1 -- 172.21.15.27:0/1515644993 learned_addr learned my addr 172.21.15.27:0/1515644993 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6a00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbac8072df0 0x7fbac80aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6000640 1 -- 172.21.15.27:0/1515644993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 msgr2=0x7fbac8071f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6000640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac8071f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6000640 1 -- 172.21.15.27:0/1515644993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbac8072df0 msgr2=0x7fbac80aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6000640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbac8072df0 0x7fbac80aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.970+0000 7fbac6000640 1 -- 172.21.15.27:0/1515644993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac8072ba0 con 0x7fbac80c3ea0 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.971+0000 7fbac6000640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac80c3ea0 0x7fbac81eb8e0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fbab400c400 tx=0x7fbab400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.971+0000 7fbabea00640 1 -- 172.21.15.27:0/1515644993 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4017070 con 0x7fbac80c3ea0 2024-03-28T19:16:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.971+0000 7fbabea00640 1 -- 172.21.15.27:0/1515644993 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab40052d0 con 0x7fbac80c3ea0 2024-03-28T19:16:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.971+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac81ebe20 con 0x7fbac80c3ea0 2024-03-28T19:16:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.971+0000 7fbabea00640 1 -- 172.21.15.27:0/1515644993 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4003830 con 0x7fbac80c3ea0 2024-03-28T19:16:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.971+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac80a9ba0 con 0x7fbac80c3ea0 2024-03-28T19:16:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.973+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac804f400 con 0x7fbac80c3ea0 2024-03-28T19:16:10.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.973+0000 7fbabea00640 1 -- 172.21.15.27:0/1515644993 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbab401a070 con 0x7fbac80c3ea0 2024-03-28T19:16:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.975+0000 7fbabea00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbaa806bbd0 0x7fbaa806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.975+0000 7fbabea00640 1 -- 172.21.15.27:0/1515644993 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7fbab408ab40 con 0x7fbac80c3ea0 2024-03-28T19:16:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.975+0000 7fbac6a00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbaa806bbd0 0x7fbaa806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:10.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.976+0000 7fbac6a00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbaa806bbd0 0x7fbaa806e090 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbab8036000 tx=0x7fbab8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:10.978+0000 7fbabea00640 1 -- 172.21.15.27:0/1515644993 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbac81ebe20 con 0x7fbac80c3ea0 2024-03-28T19:16:11.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.243+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac80c4ce0 con 0x7fbac80c3ea0 2024-03-28T19:16:11.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.243+0000 7fbabea00640 1 -- 172.21.15.27:0/1515644993 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fbac8072ba0 con 0x7fbac80c3ea0 2024-03-28T19:16:11.243 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:11.243 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:08.833799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:11.243 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 166 2024-03-28T19:16:11.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbaa806bbd0 msgr2=0x7fbaa806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:11.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbaa806bbd0 0x7fbaa806e090 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbab8036000 tx=0x7fbab8008030 comp rx=0 tx=0).stop 2024-03-28T19:16:11.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac80c3ea0 msgr2=0x7fbac81eb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:11.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac80c3ea0 0x7fbac81eb8e0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fbab400c400 tx=0x7fbab400c8c0 comp rx=0 tx=0).stop 2024-03-28T19:16:11.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 shutdown_connections 2024-03-28T19:16:11.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbaa806bbd0 0x7fbaa806e090 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:11.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbac80c60e0 0x7fbac8071f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:11.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbac80c3ea0 0x7fbac81eb8e0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:11.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 --2- 172.21.15.27:0/1515644993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbac8072df0 0x7fbac80aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:11.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.247+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 >> 172.21.15.27:0/1515644993 conn(0x7fbac806b9d0 msgr2=0x7fbac81586a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:11.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.248+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 shutdown_connections 2024-03-28T19:16:11.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:11.248+0000 7fbacda00640 1 -- 172.21.15.27:0/1515644993 wait complete. 2024-03-28T19:16:11.272 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:16:12.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:12.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.796+0000 7f5c37e00640 1 Processor -- start 2024-03-28T19:16:12.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.796+0000 7f5c37e00640 1 -- start start 2024-03-28T19:16:12.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.797+0000 7f5c37e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5c380c58c0 0x7f5c380c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:12.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.798+0000 7f5c37e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c380be5d0 0x7f5c38155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:12.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.798+0000 7f5c37e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c38156230 con 0x7f5c380c8ed0 2024-03-28T19:16:12.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.798+0000 7f5c37e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c38158bd0 con 0x7f5c380be5d0 2024-03-28T19:16:12.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.798+0000 7f5c37e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c38158eb0 con 0x7f5c380c58c0 2024-03-28T19:16:12.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.798+0000 7f5c37400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c380be5d0 0x7f5c38155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:12.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.798+0000 7f5c36000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5c380c8ed0 0x7f5c381555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56540/0 (socket says 172.21.15.27:56540) 2024-03-28T19:16:12.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.798+0000 7f5c36000640 1 -- 172.21.15.27:0/2713673393 learned_addr learned my addr 172.21.15.27:0/2713673393 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:12.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.798+0000 7f5c36a00640 1 --2- 172.21.15.27:0/2713673393 >> v2:172.21.15.155:3300/0 conn(0x7f5c380c58c0 0x7f5c380c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:12.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.799+0000 7f5c35600640 1 -- 172.21.15.27:0/2713673393 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098700112 0 0) 0x7f5c38156230 con 0x7f5c380c8ed0 2024-03-28T19:16:12.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.799+0000 7f5c35600640 1 -- 172.21.15.27:0/2713673393 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c1c003520 con 0x7f5c380c8ed0 2024-03-28T19:16:12.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.799+0000 7f5c37400640 1 -- 172.21.15.27:0/2713673393 >> v2:172.21.15.155:3300/0 conn(0x7f5c380c58c0 msgr2=0x7f5c380c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:12.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.799+0000 7f5c37400640 1 --2- 172.21.15.27:0/2713673393 >> v2:172.21.15.155:3300/0 conn(0x7f5c380c58c0 0x7f5c380c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:12.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.799+0000 7f5c37400640 1 -- 172.21.15.27:0/2713673393 >> v1:172.21.15.27:6789/0 conn(0x7f5c380c8ed0 legacy=0x7f5c381555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:12.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.799+0000 7f5c37400640 1 -- 172.21.15.27:0/2713673393 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c38159190 con 0x7f5c380be5d0 2024-03-28T19:16:12.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.801+0000 7f5c37400640 1 --2- 172.21.15.27:0/2713673393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380be5d0 0x7f5c38155cf0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5c3806c060 tx=0x7f5c3002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec965b11544bf3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:12.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.801+0000 7f5c35600640 1 -- 172.21.15.27:0/2713673393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30039070 con 0x7f5c380be5d0 2024-03-28T19:16:12.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.801+0000 7f5c35600640 1 -- 172.21.15.27:0/2713673393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3002cb60 con 0x7f5c380be5d0 2024-03-28T19:16:12.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.801+0000 7f5c35600640 1 -- 172.21.15.27:0/2713673393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30034440 con 0x7f5c380be5d0 2024-03-28T19:16:12.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.802+0000 7f5c37e00640 1 -- 172.21.15.27:0/2713673393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380be5d0 msgr2=0x7f5c38155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:12.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.802+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2713673393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380be5d0 0x7f5c38155cf0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5c3806c060 tx=0x7f5c3002bdb0 comp rx=0 tx=0).stop 2024-03-28T19:16:12.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.802+0000 7f5c37e00640 1 -- 172.21.15.27:0/2713673393 shutdown_connections 2024-03-28T19:16:12.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.802+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2713673393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380be5d0 0x7f5c38155cf0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:12.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.802+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2713673393 >> v2:172.21.15.155:3300/0 conn(0x7f5c380c58c0 0x7f5c380c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:12.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.802+0000 7f5c37e00640 1 -- 172.21.15.27:0/2713673393 >> 172.21.15.27:0/2713673393 conn(0x7f5c3806b6f0 msgr2=0x7f5c380acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:12.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.802+0000 7f5c37e00640 1 -- 172.21.15.27:0/2713673393 shutdown_connections 2024-03-28T19:16:12.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.802+0000 7f5c37e00640 1 -- 172.21.15.27:0/2713673393 wait complete. 2024-03-28T19:16:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.804+0000 7f5c37e00640 1 Processor -- start 2024-03-28T19:16:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.804+0000 7f5c37e00640 1 -- start start 2024-03-28T19:16:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c380be5d0 0x7f5c381ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c380c58c0 0x7f5c38151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380c8ed0 0x7f5c38151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:12.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380b0820 con 0x7f5c380c8ed0 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c38159750 con 0x7f5c380c58c0 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380b09a0 con 0x7f5c380be5d0 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380c8ed0 0x7f5c38151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380c8ed0 0x7f5c38151bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48364/0 (socket says 172.21.15.27:48364) 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c37400640 1 -- 172.21.15.27:0/2396106834 learned_addr learned my addr 172.21.15.27:0/2396106834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c36a00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c380be5d0 0x7f5c381ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.805+0000 7f5c36000640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c380c58c0 0x7f5c38151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c37400640 1 -- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c380be5d0 msgr2=0x7f5c381ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c37400640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c380be5d0 0x7f5c381ee970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:12.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c37400640 1 -- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c380c58c0 msgr2=0x7f5c38151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c37400640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c380c58c0 0x7f5c38151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c37400640 1 -- 172.21.15.27:0/2396106834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c38152310 con 0x7f5c380c8ed0 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c36000640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c380c58c0 0x7f5c38151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c37400640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380c8ed0 0x7f5c38151bd0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009e80 tx=0x7f5c30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c2ea00640 1 -- 172.21.15.27:0/2396106834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30039050 con 0x7f5c380c8ed0 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c2ea00640 1 -- 172.21.15.27:0/2396106834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c30045070 con 0x7f5c380c8ed0 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c2ea00640 1 -- 172.21.15.27:0/2396106834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30009450 con 0x7f5c380c8ed0 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.806+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c381ef360 con 0x7f5c380c8ed0 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.807+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c380af6f0 con 0x7f5c380c8ed0 2024-03-28T19:16:12.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.808+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c38069640 con 0x7f5c380c8ed0 2024-03-28T19:16:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.810+0000 7f5c2ea00640 1 -- 172.21.15.27:0/2396106834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5c3002cd10 con 0x7f5c380c8ed0 2024-03-28T19:16:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.810+0000 7f5c2ea00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c1806bbd0 0x7f5c1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.811+0000 7f5c2ea00640 1 -- 172.21.15.27:0/2396106834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f5c300ae0e0 con 0x7f5c380c8ed0 2024-03-28T19:16:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.811+0000 7f5c36a00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c1806bbd0 0x7f5c1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.811+0000 7f5c36a00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c1806bbd0 0x7f5c1806e090 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009a30 tx=0x7f5c28008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:12.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:12.820+0000 7f5c2ea00640 1 -- 172.21.15.27:0/2396106834 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5c3007bf70 con 0x7f5c380c8ed0 2024-03-28T19:16:13.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.086+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c380c5370 con 0x7f5c380c8ed0 2024-03-28T19:16:13.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.087+0000 7f5c2ea00640 1 -- 172.21.15.27:0/2396106834 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f5c3007b870 con 0x7f5c380c8ed0 2024-03-28T19:16:13.087 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:13.087 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:08.833799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:13.087 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 166 2024-03-28T19:16:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c1806bbd0 msgr2=0x7f5c1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c1806bbd0 0x7f5c1806e090 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009a30 tx=0x7f5c28008040 comp rx=0 tx=0).stop 2024-03-28T19:16:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380c8ed0 msgr2=0x7f5c38151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380c8ed0 0x7f5c38151bd0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009e80 tx=0x7f5c30004a40 comp rx=0 tx=0).stop 2024-03-28T19:16:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 shutdown_connections 2024-03-28T19:16:13.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5c1806bbd0 0x7f5c1806e090 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c380c8ed0 0x7f5c38151bd0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c380c58c0 0x7f5c38151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 --2- 172.21.15.27:0/2396106834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c380be5d0 0x7f5c381ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 >> 172.21.15.27:0/2396106834 conn(0x7f5c3806b6f0 msgr2=0x7f5c380ae380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.091+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 shutdown_connections 2024-03-28T19:16:13.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.092+0000 7f5c37e00640 1 -- 172.21.15.27:0/2396106834 wait complete. 2024-03-28T19:16:13.107 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:16:13.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:13.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.512+0000 7f6de9400640 1 Processor -- start 2024-03-28T19:16:13.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.512+0000 7f6de9400640 1 -- start start 2024-03-28T19:16:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.513+0000 7f6de9400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6de4155df0 0x7f6de406fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.513+0000 7f6de9400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de40c6bb0 con 0x7f6de414be30 2024-03-28T19:16:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.513+0000 7f6de9400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de40c6e50 con 0x7f6de406ffb0 2024-03-28T19:16:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.513+0000 7f6de9400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de4155960 con 0x7f6de4155df0 2024-03-28T19:16:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.514+0000 7f6de3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6de414be30 0x7f6de4158ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56554/0 (socket says 172.21.15.27:56554) 2024-03-28T19:16:13.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.514+0000 7f6de3e00640 1 -- 172.21.15.27:0/2187818360 learned_addr learned my addr 172.21.15.27:0/2187818360 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.514+0000 7f6de3400640 1 --2- 172.21.15.27:0/2187818360 >> v2:172.21.15.155:3300/0 conn(0x7f6de4155df0 0x7f6de406fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.514+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189934877 0 0) 0x7f6de40c6bb0 con 0x7f6de414be30 2024-03-28T19:16:13.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.514+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc8003520 con 0x7f6de414be30 2024-03-28T19:16:13.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 14231196 0 0) 0x7f6dc8003520 con 0x7f6de414be30 2024-03-28T19:16:13.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 >> v2:172.21.15.155:3300/0 conn(0x7f6de4155df0 msgr2=0x7f6de406fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:13.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 --2- 172.21.15.27:0/2187818360 >> v2:172.21.15.155:3300/0 conn(0x7f6de4155df0 0x7f6de406fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 >> v1:172.21.15.155:6789/0 conn(0x7f6de406ffb0 legacy=0x7f6de40a7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de41555f0 con 0x7f6de414be30 2024-03-28T19:16:13.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f6dd4001690 con 0x7f6de414be30 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 >> v1:172.21.15.27:6789/0 conn(0x7f6de414be30 legacy=0x7f6de4158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6dc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8004800 0x7f6dc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dc80051c0 0x7f6dc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de4155960 con 0x7f6dc8003a00 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc800c0c0 con 0x7f6dc8004800 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc800c2c0 con 0x7f6dc80051c0 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.515+0000 7f6de3e00640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6dc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de8a00640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dc80051c0 0x7f6dc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de3400640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8004800 0x7f6dc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:13.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de3e00640 1 -- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dc80051c0 msgr2=0x7f6dc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de3e00640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dc80051c0 0x7f6dc800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de3e00640 1 -- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8004800 msgr2=0x7f6dc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de3e00640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8004800 0x7f6dc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de3e00640 1 -- 172.21.15.27:0/2187818360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8003520 con 0x7f6dc8003a00 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de3400640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8004800 0x7f6dc8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.516+0000 7f6de8a00640 1 -- 172.21.15.27:0/2187818360 reap_dead start 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.517+0000 7f6de3e00640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6dc8003e20 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400c160 tx=0x7f6dc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ba692ef9b8370e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.517+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4007c60 con 0x7f6dc8003a00 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.517+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd4030070 con 0x7f6dc8003a00 2024-03-28T19:16:13.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.517+0000 7f6de2a00640 1 -- 172.21.15.27:0/2187818360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4005d10 con 0x7f6dc8003a00 2024-03-28T19:16:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.518+0000 7f6de9400640 1 -- 172.21.15.27:0/2187818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 msgr2=0x7f6dc8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.518+0000 7f6de9400640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6dc8003e20 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400c160 tx=0x7f6dc8005830 comp rx=0 tx=0).stop 2024-03-28T19:16:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.518+0000 7f6de9400640 1 -- 172.21.15.27:0/2187818360 shutdown_connections 2024-03-28T19:16:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.518+0000 7f6de9400640 1 --2- 172.21.15.27:0/2187818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6dc8003e20 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.518+0000 7f6de9400640 1 -- 172.21.15.27:0/2187818360 >> 172.21.15.27:0/2187818360 conn(0x7f6de4072bd0 msgr2=0x7f6de4072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.518+0000 7f6de9400640 1 -- 172.21.15.27:0/2187818360 shutdown_connections 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.518+0000 7f6de9400640 1 -- 172.21.15.27:0/2187818360 wait complete. 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.519+0000 7f6de9400640 1 Processor -- start 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.520+0000 7f6de9400640 1 -- start start 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.520+0000 7f6de9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6de41ee870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6de41564d0 0x7f6de414e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de9400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6de40ae650 0x7f6de414e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de9400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de40c72b0 con 0x7f6dc8003a00 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de9400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de40c7130 con 0x7f6de41564d0 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de9400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de414c210 con 0x7f6de40ae650 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6de41ee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6de41ee870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48380/0 (socket says 172.21.15.27:48380) 2024-03-28T19:16:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de3e00640 1 -- 172.21.15.27:0/1074494855 learned_addr learned my addr 172.21.15.27:0/1074494855 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de3e00640 1 -- 172.21.15.27:0/1074494855 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6de40ae650 msgr2=0x7f6de414e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de3e00640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6de40ae650 0x7f6de414e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de3e00640 1 -- 172.21.15.27:0/1074494855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6de41564d0 msgr2=0x7f6de414e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de3e00640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6de41564d0 0x7f6de414e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.521+0000 7f6de3e00640 1 -- 172.21.15.27:0/1074494855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de4159180 con 0x7f6dc8003a00 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.522+0000 7f6de3e00640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6de41ee870 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400c280 tx=0x7f6de40c83f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.522+0000 7f6ddbe00640 1 -- 172.21.15.27:0/1074494855 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4007c60 con 0x7f6dc8003a00 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.522+0000 7f6ddbe00640 1 -- 172.21.15.27:0/1074494855 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd40049f0 con 0x7f6dc8003a00 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.522+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de41ef1b0 con 0x7f6dc8003a00 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.522+0000 7f6ddbe00640 1 -- 172.21.15.27:0/1074494855 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4004de0 con 0x7f6dc8003a00 2024-03-28T19:16:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.522+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de41eedb0 con 0x7f6dc8003a00 2024-03-28T19:16:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.523+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de4069640 con 0x7f6dc8003a00 2024-03-28T19:16:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.524+0000 7f6ddbe00640 1 -- 172.21.15.27:0/1074494855 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6dd4007070 con 0x7f6dc8003a00 2024-03-28T19:16:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.525+0000 7f6ddbe00640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dc006bc00 0x7f6dc006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.525+0000 7f6ddbe00640 1 -- 172.21.15.27:0/1074494855 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f6dd40ac9a0 con 0x7f6dc8003a00 2024-03-28T19:16:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.525+0000 7f6de3400640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dc006bc00 0x7f6dc006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.526+0000 7f6de3400640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dc006bc00 0x7f6dc006e0c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6de40ae5d0 tx=0x7f6dcc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:13.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.530+0000 7f6ddbe00640 1 -- 172.21.15.27:0/1074494855 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6dd4046060 con 0x7f6dc8003a00 2024-03-28T19:16:13.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.766+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6de40689b0 con 0x7f6dc8003a00 2024-03-28T19:16:13.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.767+0000 7f6ddbe00640 1 -- 172.21.15.27:0/1074494855 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f6dd407a790 con 0x7f6dc8003a00 2024-03-28T19:16:13.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:13.766 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:08.833799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:13.767 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 166 2024-03-28T19:16:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.769+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dc006bc00 msgr2=0x7f6dc006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.769+0000 7f6de9400640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dc006bc00 0x7f6dc006e0c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6de40ae5d0 tx=0x7f6dcc008040 comp rx=0 tx=0).stop 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.769+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 msgr2=0x7f6de41ee870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6de41ee870 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400c280 tx=0x7f6de40c83f0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 shutdown_connections 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6dc006bc00 0x7f6dc006e0c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6de40ae650 0x7f6de414e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6de41564d0 0x7f6de414e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 --2- 172.21.15.27:0/1074494855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc8003a00 0x7f6de41ee870 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 >> 172.21.15.27:0/1074494855 conn(0x7f6de4072bd0 msgr2=0x7f6de40c8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 shutdown_connections 2024-03-28T19:16:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:13.770+0000 7f6de9400640 1 -- 172.21.15.27:0/1074494855 wait complete. 2024-03-28T19:16:13.786 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:16:15.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:15.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.322+0000 7f0d25800640 1 Processor -- start 2024-03-28T19:16:15.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.323+0000 7f0d25800640 1 -- start start 2024-03-28T19:16:15.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.324+0000 7f0d25800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d200be5d0 0x7f0d200c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:15.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.324+0000 7f0d25800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d20158bd0 con 0x7f0d200c58c0 2024-03-28T19:16:15.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.324+0000 7f0d25800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d20158eb0 con 0x7f0d200c8ed0 2024-03-28T19:16:15.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.324+0000 7f0d25800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d20159190 con 0x7f0d200be5d0 2024-03-28T19:16:15.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.324+0000 7f0d1fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0d200c58c0 0x7f0d201555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56562/0 (socket says 172.21.15.27:56562) 2024-03-28T19:16:15.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.324+0000 7f0d1fe00640 1 -- 172.21.15.27:0/4149797385 learned_addr learned my addr 172.21.15.27:0/4149797385 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:15.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.324+0000 7f0d24e00640 1 --2- 172.21.15.27:0/4149797385 >> v2:172.21.15.27:3300/0 conn(0x7f0d200be5d0 0x7f0d200c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:15.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.325+0000 7f0d24e00640 1 -- 172.21.15.27:0/4149797385 >> v1:172.21.15.155:6789/0 conn(0x7f0d200c8ed0 legacy=0x7f0d20155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:15.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.325+0000 7f0d24e00640 1 -- 172.21.15.27:0/4149797385 >> v1:172.21.15.27:6789/0 conn(0x7f0d200c58c0 legacy=0x7f0d201555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:15.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.325+0000 7f0d24e00640 1 -- 172.21.15.27:0/4149797385 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d20159470 con 0x7f0d200be5d0 2024-03-28T19:16:15.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.325+0000 7f0d1ea00640 1 -- 172.21.15.27:0/4149797385 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544696684 0 0) 0x7f0d20158bd0 con 0x7f0d200c58c0 2024-03-28T19:16:15.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.326+0000 7f0d24e00640 1 --2- 172.21.15.27:0/4149797385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200c5ca0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009d90 tx=0x7f0d1802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33f9eab5e996bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:15.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d1ea00640 1 -- 172.21.15.27:0/4149797385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18039070 con 0x7f0d200be5d0 2024-03-28T19:16:15.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d1ea00640 1 -- 172.21.15.27:0/4149797385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1802cbd0 con 0x7f0d200be5d0 2024-03-28T19:16:15.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d1ea00640 1 -- 172.21.15.27:0/4149797385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18034440 con 0x7f0d200be5d0 2024-03-28T19:16:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d25800640 1 -- 172.21.15.27:0/4149797385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 msgr2=0x7f0d200c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d25800640 1 --2- 172.21.15.27:0/4149797385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200c5ca0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009d90 tx=0x7f0d1802be20 comp rx=0 tx=0).stop 2024-03-28T19:16:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d25800640 1 -- 172.21.15.27:0/4149797385 shutdown_connections 2024-03-28T19:16:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d25800640 1 --2- 172.21.15.27:0/4149797385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200c5ca0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d25800640 1 -- 172.21.15.27:0/4149797385 >> 172.21.15.27:0/4149797385 conn(0x7f0d2006b6f0 msgr2=0x7f0d200acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d25800640 1 -- 172.21.15.27:0/4149797385 shutdown_connections 2024-03-28T19:16:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.327+0000 7f0d25800640 1 -- 172.21.15.27:0/4149797385 wait complete. 2024-03-28T19:16:15.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.329+0000 7f0d25800640 1 Processor -- start 2024-03-28T19:16:15.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.329+0000 7f0d25800640 1 -- start start 2024-03-28T19:16:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d25800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200abc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d25800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d200c58c0 0x7f0d201eecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d25800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d200c8ed0 0x7f0d201ef230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d25800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200b09a0 con 0x7f0d200be5d0 2024-03-28T19:16:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d25800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200b0820 con 0x7f0d200c8ed0 2024-03-28T19:16:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d25800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d200ad7d0 con 0x7f0d200c58c0 2024-03-28T19:16:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d1fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d1fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200abc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48410/0 (socket says 172.21.15.27:48410) 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.330+0000 7f0d1fe00640 1 -- 172.21.15.27:0/2297308173 learned_addr learned my addr 172.21.15.27:0/2297308173 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d1f400640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d200c58c0 0x7f0d201eecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d24e00640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d200c8ed0 0x7f0d201ef230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d1fe00640 1 -- 172.21.15.27:0/2297308173 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d200c58c0 msgr2=0x7f0d201eecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d1fe00640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d200c58c0 0x7f0d201eecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d1fe00640 1 -- 172.21.15.27:0/2297308173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d200c8ed0 msgr2=0x7f0d201ef230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d1fe00640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d200c8ed0 0x7f0d201ef230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d1fe00640 1 -- 172.21.15.27:0/2297308173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d201eb3e0 con 0x7f0d200be5d0 2024-03-28T19:16:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d1fe00640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200abc20 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c4a0 tx=0x7f0d1000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:15.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d17e00640 1 -- 172.21.15.27:0/2297308173 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10018070 con 0x7f0d200be5d0 2024-03-28T19:16:15.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.331+0000 7f0d17e00640 1 -- 172.21.15.27:0/2297308173 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d100074e0 con 0x7f0d200be5d0 2024-03-28T19:16:15.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.332+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d201ded70 con 0x7f0d200be5d0 2024-03-28T19:16:15.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.332+0000 7f0d17e00640 1 -- 172.21.15.27:0/2297308173 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10004a20 con 0x7f0d200be5d0 2024-03-28T19:16:15.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.332+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d201df330 con 0x7f0d200be5d0 2024-03-28T19:16:15.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.333+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d20069640 con 0x7f0d200be5d0 2024-03-28T19:16:15.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.334+0000 7f0d17e00640 1 -- 172.21.15.27:0/2297308173 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0d1001b070 con 0x7f0d200be5d0 2024-03-28T19:16:15.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.335+0000 7f0d17e00640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cfc06bbd0 0x7f0cfc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:15.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.335+0000 7f0d17e00640 1 -- 172.21.15.27:0/2297308173 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f0d1008aaf0 con 0x7f0d200be5d0 2024-03-28T19:16:15.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.335+0000 7f0d1f400640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cfc06bbd0 0x7f0cfc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:15.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.336+0000 7f0d1f400640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cfc06bbd0 0x7f0cfc06e090 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0d201e2da0 tx=0x7f0d08009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:15.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.340+0000 7f0d17e00640 1 -- 172.21.15.27:0/2297308173 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0d10058860 con 0x7f0d200be5d0 2024-03-28T19:16:15.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.587+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d201ee950 con 0x7f0d200be5d0 2024-03-28T19:16:15.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.588+0000 7f0d17e00640 1 -- 172.21.15.27:0/2297308173 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f0d10058160 con 0x7f0d200be5d0 2024-03-28T19:16:15.587 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:15.588 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:13.898784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:15.588 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 168 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cfc06bbd0 msgr2=0x7f0cfc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cfc06bbd0 0x7f0cfc06e090 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0d201e2da0 tx=0x7f0d08009040 comp rx=0 tx=0).stop 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 msgr2=0x7f0d200abc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200abc20 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c4a0 tx=0x7f0d1000c960 comp rx=0 tx=0).stop 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 shutdown_connections 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cfc06bbd0 0x7f0cfc06e090 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d200c8ed0 0x7f0d201ef230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0d200c58c0 0x7f0d201eecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 --2- 172.21.15.27:0/2297308173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d200be5d0 0x7f0d200abc20 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 >> 172.21.15.27:0/2297308173 conn(0x7f0d2006b6f0 msgr2=0x7f0d20157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.591+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 shutdown_connections 2024-03-28T19:16:15.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:15.592+0000 7f0d25800640 1 -- 172.21.15.27:0/2297308173 wait complete. 2024-03-28T19:16:15.610 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:16:15.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:16.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.050+0000 7f84d6e00640 1 Processor -- start 2024-03-28T19:16:16.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.050+0000 7f84d6e00640 1 -- start start 2024-03-28T19:16:16.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d6e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84d00c6a90 0x7f84d006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:16.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d6e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6e70 0x7f84d00c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d6e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d006d080 con 0x7f84d0156ba0 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d6e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d00c7830 con 0x7f84d00c6a90 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d6e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0159980 con 0x7f84d00c6e70 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d6400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f84d0156ba0 0x7f84d0157000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56578/0 (socket says 172.21.15.27:56578) 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d6400640 1 -- 172.21.15.27:0/1405370722 learned_addr learned my addr 172.21.15.27:0/1405370722 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d5000640 1 --2- 172.21.15.27:0/1405370722 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6e70 0x7f84d00c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.051+0000 7f84d5a00640 1 --2- 172.21.15.27:0/1405370722 >> v2:172.21.15.155:3300/0 conn(0x7f84d00c6a90 0x7f84d006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.052+0000 7f84cfe00640 1 -- 172.21.15.27:0/1405370722 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180909318 0 0) 0x7f84d006d080 con 0x7f84d0156ba0 2024-03-28T19:16:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.052+0000 7f84cfe00640 1 -- 172.21.15.27:0/1405370722 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b8003520 con 0x7f84d0156ba0 2024-03-28T19:16:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.052+0000 7f84d5000640 1 -- 172.21.15.27:0/1405370722 >> v2:172.21.15.155:3300/0 conn(0x7f84d00c6a90 msgr2=0x7f84d006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.052+0000 7f84d5000640 1 --2- 172.21.15.27:0/1405370722 >> v2:172.21.15.155:3300/0 conn(0x7f84d00c6a90 0x7f84d006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.052+0000 7f84d5000640 1 -- 172.21.15.27:0/1405370722 >> v1:172.21.15.27:6789/0 conn(0x7f84d0156ba0 legacy=0x7f84d0157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.052+0000 7f84d5000640 1 -- 172.21.15.27:0/1405370722 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d006cb00 con 0x7f84d00c6e70 2024-03-28T19:16:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.052+0000 7f84d5a00640 1 --2- 172.21.15.27:0/1405370722 >> v2:172.21.15.155:3300/0 conn(0x7f84d00c6a90 0x7f84d006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:16:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.053+0000 7f84d5000640 1 --2- 172.21.15.27:0/1405370722 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6e70 0x7f84d00c72f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0097a0 tx=0x7f84bc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59386a77a8cb90ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.054+0000 7f84cfe00640 1 -- 172.21.15.27:0/1405370722 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc035020 con 0x7f84d00c6e70 2024-03-28T19:16:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.054+0000 7f84cfe00640 1 -- 172.21.15.27:0/1405370722 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc031810 con 0x7f84d00c6e70 2024-03-28T19:16:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.054+0000 7f84cfe00640 1 -- 172.21.15.27:0/1405370722 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc031be0 con 0x7f84d00c6e70 2024-03-28T19:16:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.055+0000 7f84d6e00640 1 -- 172.21.15.27:0/1405370722 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6e70 msgr2=0x7f84d00c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:16.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.055+0000 7f84d6e00640 1 --2- 172.21.15.27:0/1405370722 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6e70 0x7f84d00c72f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0097a0 tx=0x7f84bc02b8b0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.055+0000 7f84d6e00640 1 -- 172.21.15.27:0/1405370722 shutdown_connections 2024-03-28T19:16:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.055+0000 7f84d6e00640 1 --2- 172.21.15.27:0/1405370722 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6e70 0x7f84d00c72f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.055+0000 7f84d6e00640 1 --2- 172.21.15.27:0/1405370722 >> v2:172.21.15.155:3300/0 conn(0x7f84d00c6a90 0x7f84d006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.055+0000 7f84d6e00640 1 -- 172.21.15.27:0/1405370722 >> 172.21.15.27:0/1405370722 conn(0x7f84d00aa7c0 msgr2=0x7f84d00aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.055+0000 7f84d6e00640 1 -- 172.21.15.27:0/1405370722 shutdown_connections 2024-03-28T19:16:16.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.055+0000 7f84d6e00640 1 -- 172.21.15.27:0/1405370722 wait complete. 2024-03-28T19:16:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.056+0000 7f84d6e00640 1 Processor -- start 2024-03-28T19:16:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.057+0000 7f84d6e00640 1 -- start start 2024-03-28T19:16:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.057+0000 7f84d6e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6a90 0x7f84d01dfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:16.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.057+0000 7f84d6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84d00c6e70 0x7f84d01519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d6e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0156ba0 0x7f84d0151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d00c8630 con 0x7f84d00c6e70 2024-03-28T19:16:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d6e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0159610 con 0x7f84d0156ba0 2024-03-28T19:16:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d6e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0152e90 con 0x7f84d00c6a90 2024-03-28T19:16:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84d00c6e70 0x7f84d01519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6a90 0x7f84d01dfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84d00c6e70 0x7f84d01519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48416/0 (socket says 172.21.15.27:48416) 2024-03-28T19:16:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5000640 1 -- 172.21.15.27:0/4087267504 learned_addr learned my addr 172.21.15.27:0/4087267504 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d6400640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0156ba0 0x7f84d0151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5000640 1 -- 172.21.15.27:0/4087267504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6a90 msgr2=0x7f84d01dfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5000640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6a90 0x7f84d01dfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5000640 1 -- 172.21.15.27:0/4087267504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0156ba0 msgr2=0x7f84d0151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5000640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0156ba0 0x7f84d0151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.058+0000 7f84d5000640 1 -- 172.21.15.27:0/4087267504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d01e13e0 con 0x7f84d00c6e70 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.059+0000 7f84d5000640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84d00c6e70 0x7f84d01519e0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f84bc02bec0 tx=0x7f84bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.059+0000 7f84cd600640 1 -- 172.21.15.27:0/4087267504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc041070 con 0x7f84d00c6e70 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.059+0000 7f84cd600640 1 -- 172.21.15.27:0/4087267504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc031e40 con 0x7f84d00c6e70 2024-03-28T19:16:16.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.059+0000 7f84cd600640 1 -- 172.21.15.27:0/4087267504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc035040 con 0x7f84d00c6e70 2024-03-28T19:16:16.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.059+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d01ef790 con 0x7f84d00c6e70 2024-03-28T19:16:16.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.060+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d01e1960 con 0x7f84d00c6e70 2024-03-28T19:16:16.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.061+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d0067dd0 con 0x7f84d00c6e70 2024-03-28T19:16:16.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.062+0000 7f84cd600640 1 -- 172.21.15.27:0/4087267504 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f84bc04c060 con 0x7f84d00c6e70 2024-03-28T19:16:16.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.063+0000 7f84cd600640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84b006bbd0 0x7f84b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:16.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.063+0000 7f84cd600640 1 -- 172.21.15.27:0/4087267504 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f84bc0ad170 con 0x7f84d00c6e70 2024-03-28T19:16:16.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.063+0000 7f84d5a00640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84b006bbd0 0x7f84b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:16.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.064+0000 7f84d5a00640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84b006bbd0 0x7f84b006e090 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009bd0 tx=0x7f84c400b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:16.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.067+0000 7f84cd600640 1 -- 172.21.15.27:0/4087267504 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f84bc07a8f0 con 0x7f84d00c6e70 2024-03-28T19:16:16.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.315+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84d0069640 con 0x7f84d00c6e70 2024-03-28T19:16:16.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.316+0000 7f84cd600640 1 -- 172.21.15.27:0/4087267504 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f84bc07e990 con 0x7f84d00c6e70 2024-03-28T19:16:16.315 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:16.315 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:15.116812+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:clientreplay","state_seq":5,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:16.316 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 169 2024-03-28T19:16:16.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84b006bbd0 msgr2=0x7f84b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:16.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84b006bbd0 0x7f84b006e090 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009bd0 tx=0x7f84c400b040 comp rx=0 tx=0).stop 2024-03-28T19:16:16.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84d00c6e70 msgr2=0x7f84d01519e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:16.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84d00c6e70 0x7f84d01519e0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f84bc02bec0 tx=0x7f84bc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 shutdown_connections 2024-03-28T19:16:16.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f84b006bbd0 0x7f84b006e090 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0156ba0 0x7f84d0151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84d00c6e70 0x7f84d01519e0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 --2- 172.21.15.27:0/4087267504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00c6a90 0x7f84d01dfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:16.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.318+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 >> 172.21.15.27:0/4087267504 conn(0x7f84d00aa7c0 msgr2=0x7f84d006ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:16.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.319+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 shutdown_connections 2024-03-28T19:16:16.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:16.319+0000 7f84d6e00640 1 -- 172.21.15.27:0/4087267504 wait complete. 2024-03-28T19:16:16.336 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:16:17.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:17.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.840+0000 7f6e3da00640 1 Processor -- start 2024-03-28T19:16:17.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.840+0000 7f6e3da00640 1 -- start start 2024-03-28T19:16:17.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.842+0000 7f6e3da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e380be5d0 0x7f6e380c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:17.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.842+0000 7f6e3da00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e38158bd0 con 0x7f6e380c8ed0 2024-03-28T19:16:17.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.842+0000 7f6e3da00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e38158eb0 con 0x7f6e380c58c0 2024-03-28T19:16:17.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.842+0000 7f6e3da00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e38159190 con 0x7f6e380be5d0 2024-03-28T19:16:17.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.842+0000 7f6e37400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6e380c8ed0 0x7f6e38155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49104/0 (socket says 172.21.15.27:49104) 2024-03-28T19:16:17.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.842+0000 7f6e37400640 1 -- 172.21.15.27:0/1379657297 learned_addr learned my addr 172.21.15.27:0/1379657297 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:17.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.842+0000 7f6e3d000640 1 --2- 172.21.15.27:0/1379657297 >> v2:172.21.15.27:3300/0 conn(0x7f6e380be5d0 0x7f6e380c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:17.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.843+0000 7f6e36a00640 1 -- 172.21.15.27:0/1379657297 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476548611 0 0) 0x7f6e38158bd0 con 0x7f6e380c8ed0 2024-03-28T19:16:17.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.843+0000 7f6e36a00640 1 -- 172.21.15.27:0/1379657297 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c003520 con 0x7f6e380c8ed0 2024-03-28T19:16:17.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.843+0000 7f6e36a00640 1 -- 172.21.15.27:0/1379657297 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992180821 0 0) 0x7f6e38158eb0 con 0x7f6e380c58c0 2024-03-28T19:16:17.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.843+0000 7f6e36a00640 1 -- 172.21.15.27:0/1379657297 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e38158bd0 con 0x7f6e380c58c0 2024-03-28T19:16:17.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.843+0000 7f6e3d000640 1 -- 172.21.15.27:0/1379657297 >> v1:172.21.15.155:6789/0 conn(0x7f6e380c58c0 legacy=0x7f6e381555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:17.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.843+0000 7f6e3d000640 1 -- 172.21.15.27:0/1379657297 >> v1:172.21.15.27:6789/0 conn(0x7f6e380c8ed0 legacy=0x7f6e38155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:17.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.843+0000 7f6e3d000640 1 -- 172.21.15.27:0/1379657297 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e38159470 con 0x7f6e380be5d0 2024-03-28T19:16:17.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.844+0000 7f6e3d000640 1 --2- 172.21.15.27:0/1379657297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380be5d0 0x7f6e380c5ca0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009d90 tx=0x7f6e3002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae93815bf20f9cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:17.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.845+0000 7f6e36a00640 1 -- 172.21.15.27:0/1379657297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30039070 con 0x7f6e380be5d0 2024-03-28T19:16:17.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.845+0000 7f6e36a00640 1 -- 172.21.15.27:0/1379657297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3002cb40 con 0x7f6e380be5d0 2024-03-28T19:16:17.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.845+0000 7f6e36a00640 1 -- 172.21.15.27:0/1379657297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30034440 con 0x7f6e380be5d0 2024-03-28T19:16:17.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.845+0000 7f6e3da00640 1 -- 172.21.15.27:0/1379657297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380be5d0 msgr2=0x7f6e380c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:17.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.845+0000 7f6e3da00640 1 --2- 172.21.15.27:0/1379657297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380be5d0 0x7f6e380c5ca0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009d90 tx=0x7f6e3002bd90 comp rx=0 tx=0).stop 2024-03-28T19:16:17.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.846+0000 7f6e3da00640 1 -- 172.21.15.27:0/1379657297 shutdown_connections 2024-03-28T19:16:17.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.846+0000 7f6e3da00640 1 --2- 172.21.15.27:0/1379657297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380be5d0 0x7f6e380c5ca0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:17.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.846+0000 7f6e3da00640 1 -- 172.21.15.27:0/1379657297 >> 172.21.15.27:0/1379657297 conn(0x7f6e3806b6f0 msgr2=0x7f6e380acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:17.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.846+0000 7f6e3da00640 1 -- 172.21.15.27:0/1379657297 shutdown_connections 2024-03-28T19:16:17.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.846+0000 7f6e3da00640 1 -- 172.21.15.27:0/1379657297 wait complete. 2024-03-28T19:16:17.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.848+0000 7f6e3da00640 1 Processor -- start 2024-03-28T19:16:17.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.848+0000 7f6e3da00640 1 -- start start 2024-03-28T19:16:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e3da00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e380be5d0 0x7f6e380c4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e3da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380c58c0 0x7f6e38156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e3da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e380c8ed0 0x7f6e38151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e3da00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e380b09a0 con 0x7f6e380c58c0 2024-03-28T19:16:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e3da00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e380b0820 con 0x7f6e380c8ed0 2024-03-28T19:16:17.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e3da00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e380ad7d0 con 0x7f6e380be5d0 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e37400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380c58c0 0x7f6e38156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e37400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380c58c0 0x7f6e38156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54174/0 (socket says 172.21.15.27:54174) 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e37400640 1 -- 172.21.15.27:0/2328589148 learned_addr learned my addr 172.21.15.27:0/2328589148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.850+0000 7f6e37e00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e380be5d0 0x7f6e380c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.851+0000 7f6e3d000640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e380c8ed0 0x7f6e38151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.851+0000 7f6e37400640 1 -- 172.21.15.27:0/2328589148 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e380be5d0 msgr2=0x7f6e380c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.851+0000 7f6e37400640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e380be5d0 0x7f6e380c4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.851+0000 7f6e37400640 1 -- 172.21.15.27:0/2328589148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e380c8ed0 msgr2=0x7f6e38151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.851+0000 7f6e37400640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e380c8ed0 0x7f6e38151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.851+0000 7f6e37400640 1 -- 172.21.15.27:0/2328589148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e380ac740 con 0x7f6e380c58c0 2024-03-28T19:16:17.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.851+0000 7f6e37400640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380c58c0 0x7f6e38156c40 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400c3e0 tx=0x7f6e2400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.851+0000 7f6e2fe00640 1 -- 172.21.15.27:0/2328589148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24017070 con 0x7f6e380c58c0 2024-03-28T19:16:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.852+0000 7f6e2fe00640 1 -- 172.21.15.27:0/2328589148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e24005ce0 con 0x7f6e380c58c0 2024-03-28T19:16:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.852+0000 7f6e2fe00640 1 -- 172.21.15.27:0/2328589148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e240072d0 con 0x7f6e380c58c0 2024-03-28T19:16:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.852+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e380aca20 con 0x7f6e380c58c0 2024-03-28T19:16:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.852+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e380c3b30 con 0x7f6e380c58c0 2024-03-28T19:16:17.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.853+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e38069640 con 0x7f6e380c58c0 2024-03-28T19:16:17.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.855+0000 7f6e2fe00640 1 -- 172.21.15.27:0/2328589148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e2401a070 con 0x7f6e380c58c0 2024-03-28T19:16:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.858+0000 7f6e2fe00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6e1806bbd0 0x7f6e1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.858+0000 7f6e2fe00640 1 -- 172.21.15.27:0/2328589148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f6e2408afe0 con 0x7f6e380c58c0 2024-03-28T19:16:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.859+0000 7f6e37e00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6e1806bbd0 0x7f6e1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:17.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.860+0000 7f6e37e00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6e1806bbd0 0x7f6e1806e090 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6e28001700 tx=0x7f6e28009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:17.860+0000 7f6e2fe00640 1 -- 172.21.15.27:0/2328589148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e240586f0 con 0x7f6e380c58c0 2024-03-28T19:16:18.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.204+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e38156580 con 0x7f6e380c58c0 2024-03-28T19:16:18.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.205+0000 7f6e2fe00640 1 -- 172.21.15.27:0/2328589148 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5467 (secure 0 0 0) 0x7f6e2405c790 con 0x7f6e380c58c0 2024-03-28T19:16:18.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:18.204 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:18.204 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 170 2024-03-28T19:16:18.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6e1806bbd0 msgr2=0x7f6e1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6e1806bbd0 0x7f6e1806e090 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6e28001700 tx=0x7f6e28009040 comp rx=0 tx=0).stop 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380c58c0 msgr2=0x7f6e38156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380c58c0 0x7f6e38156c40 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400c3e0 tx=0x7f6e2400c8a0 comp rx=0 tx=0).stop 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 shutdown_connections 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6e1806bbd0 0x7f6e1806e090 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e380c8ed0 0x7f6e38151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e380c58c0 0x7f6e38156c40 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 --2- 172.21.15.27:0/2328589148 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e380be5d0 0x7f6e380c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 >> 172.21.15.27:0/2328589148 conn(0x7f6e3806b6f0 msgr2=0x7f6e380ae360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 shutdown_connections 2024-03-28T19:16:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.208+0000 7f6e3da00640 1 -- 172.21.15.27:0/2328589148 wait complete. 2024-03-28T19:16:18.229 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7146 2024-03-28T19:16:18.229 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.c 2024-03-28T19:16:18.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.586+0000 7f208c600640 1 Processor -- start 2024-03-28T19:16:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.586+0000 7f208c600640 1 -- start start 2024-03-28T19:16:18.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.588+0000 7f208c600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20840abf60 0x7f20840aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:18.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.588+0000 7f208c600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208406d8a0 con 0x7f20840c5e80 2024-03-28T19:16:18.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.588+0000 7f208c600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208406db80 con 0x7f20840c1f50 2024-03-28T19:16:18.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.588+0000 7f208c600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208406d650 con 0x7f20840abf60 2024-03-28T19:16:18.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.588+0000 7f208a600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20840abf60 0x7f20840aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:18.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.589+0000 7f208a600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20840abf60 0x7f20840aa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:33628/0 (socket says 172.21.15.27:33628) 2024-03-28T19:16:18.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.589+0000 7f2089c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f20840c5e80 0x7f20840c2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49118/0 (socket says 172.21.15.27:49118) 2024-03-28T19:16:18.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.589+0000 7f208a600640 1 -- 172.21.15.27:0/3707097647 learned_addr learned my addr 172.21.15.27:0/3707097647 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.589+0000 7f208a600640 1 -- 172.21.15.27:0/3707097647 >> v1:172.21.15.155:6789/0 conn(0x7f20840c1f50 legacy=0x7f20840ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.589+0000 7f208a600640 1 -- 172.21.15.27:0/3707097647 >> v1:172.21.15.27:6789/0 conn(0x7f20840c5e80 legacy=0x7f20840c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:18.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.589+0000 7f208a600640 1 -- 172.21.15.27:0/3707097647 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208406de60 con 0x7f20840abf60 2024-03-28T19:16:18.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.591+0000 7f208a600640 1 --2- 172.21.15.27:0/3707097647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840abf60 0x7f20840aa570 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f207c009e50 tx=0x7f207c02bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=902b9dff7e77c9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.592+0000 7f2083e00640 1 -- 172.21.15.27:0/3707097647 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c02c850 con 0x7f20840abf60 2024-03-28T19:16:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.592+0000 7f2083e00640 1 -- 172.21.15.27:0/3707097647 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c02c9f0 con 0x7f20840abf60 2024-03-28T19:16:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.592+0000 7f2083e00640 1 -- 172.21.15.27:0/3707097647 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c02cdc0 con 0x7f20840abf60 2024-03-28T19:16:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.593+0000 7f208c600640 1 -- 172.21.15.27:0/3707097647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840abf60 msgr2=0x7f20840aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.593+0000 7f208c600640 1 --2- 172.21.15.27:0/3707097647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840abf60 0x7f20840aa570 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f207c009e50 tx=0x7f207c02bd30 comp rx=0 tx=0).stop 2024-03-28T19:16:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.594+0000 7f208c600640 1 -- 172.21.15.27:0/3707097647 shutdown_connections 2024-03-28T19:16:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.594+0000 7f208c600640 1 --2- 172.21.15.27:0/3707097647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840abf60 0x7f20840aa570 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:18.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.594+0000 7f208c600640 1 -- 172.21.15.27:0/3707097647 >> 172.21.15.27:0/3707097647 conn(0x7f208406b9d0 msgr2=0x7f20840c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.594+0000 7f208c600640 1 -- 172.21.15.27:0/3707097647 shutdown_connections 2024-03-28T19:16:18.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.594+0000 7f208c600640 1 -- 172.21.15.27:0/3707097647 wait complete. 2024-03-28T19:16:18.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.597+0000 7f208c600640 1 Processor -- start 2024-03-28T19:16:18.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.597+0000 7f208c600640 1 -- start start 2024-03-28T19:16:18.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.598+0000 7f208c600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840abf60 0x7f2084147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:18.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.598+0000 7f208c600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c1f50 0x7f20841481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:18.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.598+0000 7f208c600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c5e80 0x7f20840bbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:18.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.598+0000 7f208c600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840aaab0 con 0x7f20840abf60 2024-03-28T19:16:18.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.598+0000 7f208c600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208406e1d0 con 0x7f20840c5e80 2024-03-28T19:16:18.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.598+0000 7f208c600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20840aac30 con 0x7f20840c1f50 2024-03-28T19:16:18.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c1f50 0x7f20841481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:18.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c1f50 0x7f20841481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:44624/0 (socket says 172.21.15.27:44624) 2024-03-28T19:16:18.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089200640 1 -- 172.21.15.27:0/1365380237 learned_addr learned my addr 172.21.15.27:0/1365380237 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:18.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089c00640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840abf60 0x7f2084147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:18.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089200640 1 -- 172.21.15.27:0/1365380237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c5e80 msgr2=0x7f20840bbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:18.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089200640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c5e80 0x7f20840bbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:18.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089200640 1 -- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840abf60 msgr2=0x7f2084147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:18.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089200640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840abf60 0x7f2084147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:18.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.599+0000 7f2089200640 1 -- 172.21.15.27:0/1365380237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20840c0eb0 con 0x7f20840c1f50 2024-03-28T19:16:18.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.600+0000 7f2089200640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c1f50 0x7f20841481a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f207000c400 tx=0x7f207000c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:18.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.601+0000 7f2081600640 1 -- 172.21.15.27:0/1365380237 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070017070 con 0x7f20840c1f50 2024-03-28T19:16:18.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.601+0000 7f2081600640 1 -- 172.21.15.27:0/1365380237 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20700052f0 con 0x7f20840c1f50 2024-03-28T19:16:18.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.601+0000 7f2081600640 1 -- 172.21.15.27:0/1365380237 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070003830 con 0x7f20840c1f50 2024-03-28T19:16:18.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.601+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2084148810 con 0x7f20840c1f50 2024-03-28T19:16:18.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.601+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20840be8b0 con 0x7f20840c1f50 2024-03-28T19:16:18.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.603+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2060005290 con 0x7f20840c1f50 2024-03-28T19:16:18.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.605+0000 7f2081600640 1 -- 172.21.15.27:0/1365380237 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f207001a070 con 0x7f20840c1f50 2024-03-28T19:16:18.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.606+0000 7f2081600640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bc20 0x7f206406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:18.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.606+0000 7f2081600640 1 -- 172.21.15.27:0/1365380237 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f207008af70 con 0x7f20840c1f50 2024-03-28T19:16:18.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.607+0000 7f2089c00640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bc20 0x7f206406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:18.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.608+0000 7f2089c00640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bc20 0x7f206406e0e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2078007fd0 tx=0x7f2078007c20 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:18.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:18.613+0000 7f2081600640 1 -- 172.21.15.27:0/1365380237 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2070021020 con 0x7f20840c1f50 2024-03-28T19:16:18.855 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:16:18.856+0000 7f1cf8000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:16:19.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.013+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2060005580 con 0x7f20840c1f50 2024-03-28T19:16:19.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.013+0000 7f2081600640 1 -- 172.21.15.27:0/1365380237 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5467 (secure 0 0 0) 0x7f2070058da0 con 0x7f20840c1f50 2024-03-28T19:16:19.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:19.013 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:19.013 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 170 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.016+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bc20 msgr2=0x7f206406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.016+0000 7f208c600640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bc20 0x7f206406e0e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2078007fd0 tx=0x7f2078007c20 comp rx=0 tx=0).stop 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.016+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c1f50 msgr2=0x7f20841481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.016+0000 7f208c600640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c1f50 0x7f20841481a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f207000c400 tx=0x7f207000c8c0 comp rx=0 tx=0).stop 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.017+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 shutdown_connections 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.017+0000 7f208c600640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f206406bc20 0x7f206406e0e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.017+0000 7f208c600640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20840c5e80 0x7f20840bbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.017+0000 7f208c600640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20840c1f50 0x7f20841481a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.017+0000 7f208c600640 1 --2- 172.21.15.27:0/1365380237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840abf60 0x7f2084147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:19.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.017+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 >> 172.21.15.27:0/1365380237 conn(0x7f208406b9d0 msgr2=0x7f20840ab740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:19.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.017+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 shutdown_connections 2024-03-28T19:16:19.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:19.017+0000 7f208c600640 1 -- 172.21.15.27:0/1365380237 wait complete. 2024-03-28T19:16:19.038 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:active state 2024-03-28T19:16:19.038 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2024-03-28T19:16:25.230 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-03-28T19:16:25.230 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:16:25.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:16:25.236 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:16:25.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:25.427 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:16:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.481+0000 7fd9d3200640 1 Processor -- start 2024-03-28T19:16:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.481+0000 7fd9d3200640 1 -- start start 2024-03-28T19:16:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.482+0000 7fd9d3200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd9d4154140 0x7fd9d40c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:25.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.482+0000 7fd9d3200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4150ec0 con 0x7fd9d4150ae0 2024-03-28T19:16:25.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.482+0000 7fd9d3200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d406dc70 con 0x7fd9d40c69a0 2024-03-28T19:16:25.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.482+0000 7fd9d3200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d4153550 con 0x7fd9d4154140 2024-03-28T19:16:25.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.482+0000 7fd9d1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd9d4150ae0 0x7fd9d40a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49126/0 (socket says 172.21.15.27:49126) 2024-03-28T19:16:25.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.482+0000 7fd9d1e00640 1 -- 172.21.15.27:0/3639683923 learned_addr learned my addr 172.21.15.27:0/3639683923 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d2800640 1 --2- 172.21.15.27:0/3639683923 >> v2:172.21.15.155:3300/0 conn(0x7fd9d4154140 0x7fd9d40c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942843422 0 0) 0x7fd9c4001110 con 0x7fd9d4150ae0 2024-03-28T19:16:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b8003520 con 0x7fd9d4150ae0 2024-03-28T19:16:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3471946786 0 0) 0x7fd9b8003520 con 0x7fd9d4150ae0 2024-03-28T19:16:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 >> v2:172.21.15.155:3300/0 conn(0x7fd9d4154140 msgr2=0x7fd9d40c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d0a00640 1 --2- 172.21.15.27:0/3639683923 >> v2:172.21.15.155:3300/0 conn(0x7fd9d4154140 0x7fd9d40c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 >> v1:172.21.15.155:6789/0 conn(0x7fd9d40c69a0 legacy=0x7fd9d4153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d406df10 con 0x7fd9d4150ae0 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.483+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd9c4001840 con 0x7fd9d4150ae0 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.484+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 >> v1:172.21.15.27:6789/0 conn(0x7fd9d4150ae0 legacy=0x7fd9d40a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.484+0000 7fd9d0a00640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b80039b0 0x7fd9b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.484+0000 7fd9d0a00640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9b80043b0 0x7fd9b8004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.484+0000 7fd9d0a00640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b8005120 0x7fd9b800fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.484+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4153550 con 0x7fd9b80039b0 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.484+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b8010470 con 0x7fd9b8005120 2024-03-28T19:16:25.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.484+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b8010670 con 0x7fd9b80043b0 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d1e00640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b80039b0 0x7fd9b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d2800640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b8005120 0x7fd9b800fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d1400640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9b80043b0 0x7fd9b8004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d1e00640 1 -- 172.21.15.27:0/3639683923 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9b80043b0 msgr2=0x7fd9b8004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d1e00640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9b80043b0 0x7fd9b8004880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d1e00640 1 -- 172.21.15.27:0/3639683923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b8005120 msgr2=0x7fd9b800fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d1e00640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b8005120 0x7fd9b800fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d1e00640 1 -- 172.21.15.27:0/3639683923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b8003520 con 0x7fd9b80039b0 2024-03-28T19:16:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.485+0000 7fd9d2800640 1 -- 172.21.15.27:0/3639683923 reap_dead start 2024-03-28T19:16:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.487+0000 7fd9d1e00640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b80039b0 0x7fd9b8003d90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400c4c0 tx=0x7fd9b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5a04f22c91098d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.487+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c400c0e0 con 0x7fd9b80039b0 2024-03-28T19:16:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.487+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c4030070 con 0x7fd9b80039b0 2024-03-28T19:16:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.487+0000 7fd9d0a00640 1 -- 172.21.15.27:0/3639683923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40058d0 con 0x7fd9b80039b0 2024-03-28T19:16:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.488+0000 7fd9d3200640 1 -- 172.21.15.27:0/3639683923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b80039b0 msgr2=0x7fd9b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.488+0000 7fd9d3200640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b80039b0 0x7fd9b8003d90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400c4c0 tx=0x7fd9b8005830 comp rx=0 tx=0).stop 2024-03-28T19:16:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.488+0000 7fd9d3200640 1 -- 172.21.15.27:0/3639683923 shutdown_connections 2024-03-28T19:16:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.488+0000 7fd9d3200640 1 --2- 172.21.15.27:0/3639683923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b80039b0 0x7fd9b8003d90 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.488+0000 7fd9d3200640 1 -- 172.21.15.27:0/3639683923 >> 172.21.15.27:0/3639683923 conn(0x7fd9d406b0c0 msgr2=0x7fd9d406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.488+0000 7fd9d3200640 1 -- 172.21.15.27:0/3639683923 shutdown_connections 2024-03-28T19:16:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.488+0000 7fd9d3200640 1 -- 172.21.15.27:0/3639683923 wait complete. 2024-03-28T19:16:25.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.491+0000 7fd9d3200640 1 Processor -- start 2024-03-28T19:16:25.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.491+0000 7fd9d3200640 1 -- start start 2024-03-28T19:16:25.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b80039b0 0x7fd9d414f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9d41461d0 0x7fd9d4145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9d414b600 0x7fd9d414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d406e7b0 con 0x7fd9d414b600 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d406e4d0 con 0x7fd9b80039b0 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d41569d0 con 0x7fd9d41461d0 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d1400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9d41461d0 0x7fd9d4145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d1400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9d41461d0 0x7fd9d4145c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:44664/0 (socket says 172.21.15.27:44664) 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d1400640 1 -- 172.21.15.27:0/2131200741 learned_addr learned my addr 172.21.15.27:0/2131200741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.493+0000 7fd9d2800640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9d414b600 0x7fd9d414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.494+0000 7fd9d1e00640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b80039b0 0x7fd9d414f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.494+0000 7fd9d2800640 1 -- 172.21.15.27:0/2131200741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9d41461d0 msgr2=0x7fd9d4145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.494+0000 7fd9d2800640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9d41461d0 0x7fd9d4145c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.494+0000 7fd9d2800640 1 -- 172.21.15.27:0/2131200741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b80039b0 msgr2=0x7fd9d414f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.494+0000 7fd9d2800640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b80039b0 0x7fd9d414f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.494+0000 7fd9d2800640 1 -- 172.21.15.27:0/2131200741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d41f2b70 con 0x7fd9d414b600 2024-03-28T19:16:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.494+0000 7fd9d2800640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9d414b600 0x7fd9d414ba80 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fd9d406c8e0 tx=0x7fd9cc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.495+0000 7fd9ca000640 1 -- 172.21.15.27:0/2131200741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc00fde0 con 0x7fd9d414b600 2024-03-28T19:16:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.495+0000 7fd9ca000640 1 -- 172.21.15.27:0/2131200741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc01d070 con 0x7fd9d414b600 2024-03-28T19:16:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.495+0000 7fd9ca000640 1 -- 172.21.15.27:0/2131200741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc015670 con 0x7fd9d414b600 2024-03-28T19:16:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.495+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d41f2e30 con 0x7fd9d414b600 2024-03-28T19:16:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.495+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d41f33f0 con 0x7fd9d414b600 2024-03-28T19:16:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.496+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d4067dd0 con 0x7fd9d414b600 2024-03-28T19:16:25.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.499+0000 7fd9ca000640 1 -- 172.21.15.27:0/2131200741 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd9cc005d10 con 0x7fd9d414b600 2024-03-28T19:16:25.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.500+0000 7fd9ca000640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9b406bc50 0x7fd9b406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.500+0000 7fd9ca000640 1 -- 172.21.15.27:0/2131200741 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7fd9cc08cb70 con 0x7fd9d414b600 2024-03-28T19:16:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.500+0000 7fd9d1e00640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9b406bc50 0x7fd9b406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:25.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.502+0000 7fd9d1e00640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9b406bc50 0x7fd9b406e110 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4034000 tx=0x7fd9c4009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:25.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.504+0000 7fd9ca000640 1 -- 172.21.15.27:0/2131200741 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd9cc010040 con 0x7fd9d414b600 2024-03-28T19:16:25.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.842+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9d4069640 con 0x7fd9d414b600 2024-03-28T19:16:25.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.843+0000 7fd9ca000640 1 -- 172.21.15.27:0/2131200741 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7fd9cc05a8d0 con 0x7fd9d414b600 2024-03-28T19:16:25.842 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:25.842 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:25.842 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9b406bc50 msgr2=0x7fd9b406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9b406bc50 0x7fd9b406e110 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4034000 tx=0x7fd9c4009030 comp rx=0 tx=0).stop 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9d414b600 msgr2=0x7fd9d414ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9d414b600 0x7fd9d414ba80 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fd9d406c8e0 tx=0x7fd9cc00f580 comp rx=0 tx=0).stop 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 shutdown_connections 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd9b406bc50 0x7fd9b406e110 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9d414b600 0x7fd9d414ba80 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9d41461d0 0x7fd9d4145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 --2- 172.21.15.27:0/2131200741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b80039b0 0x7fd9d414f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 >> 172.21.15.27:0/2131200741 conn(0x7fd9d406b0c0 msgr2=0x7fd9d40c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:25.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 shutdown_connections 2024-03-28T19:16:25.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:25.846+0000 7fd9d3200640 1 -- 172.21.15.27:0/2131200741 wait complete. 2024-03-28T19:16:25.865 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-03-28T19:16:25.866 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2024-03-28T19:16:36.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:16:36.215 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:16:54.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:55.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.114+0000 7f84afe00640 1 Processor -- start 2024-03-28T19:16:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.114+0000 7f84afe00640 1 -- start start 2024-03-28T19:16:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84afe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84b00c69a0 0x7f84b00c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84afe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84b0153a30 0x7f84b0153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84afe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b0150ec0 con 0x7f84b0150ae0 2024-03-28T19:16:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84afe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b01543c0 con 0x7f84b00c69a0 2024-03-28T19:16:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84afe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b006dc70 con 0x7f84b0153a30 2024-03-28T19:16:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84a6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84b00c69a0 0x7f84b00c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84af400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84b0153a30 0x7f84b0153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84a6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84b00c69a0 0x7f84b00c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52706/0 (socket says 172.21.15.27:52706) 2024-03-28T19:16:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.116+0000 7f84a6000640 1 -- 172.21.15.27:0/4221310491 learned_addr learned my addr 172.21.15.27:0/4221310491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.117+0000 7f84a6000640 1 -- 172.21.15.27:0/4221310491 >> v2:172.21.15.155:3300/0 conn(0x7f84b0153a30 msgr2=0x7f84b0153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.117+0000 7f84a6000640 1 --2- 172.21.15.27:0/4221310491 >> v2:172.21.15.155:3300/0 conn(0x7f84b0153a30 0x7f84b0153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.117+0000 7f84a6000640 1 -- 172.21.15.27:0/4221310491 >> v1:172.21.15.155:6789/0 conn(0x7f84b0150ae0 legacy=0x7f84b00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.117+0000 7f84a6000640 1 -- 172.21.15.27:0/4221310491 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0153550 con 0x7f84b00c69a0 2024-03-28T19:16:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.117+0000 7f84af400640 1 --2- 172.21.15.27:0/4221310491 >> v2:172.21.15.155:3300/0 conn(0x7f84b0153a30 0x7f84b0153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:16:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.118+0000 7f84a6000640 1 --2- 172.21.15.27:0/4221310491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00c69a0 0x7f84b00c6e30 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f84b00c0af0 tx=0x7f849802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25ba7c2355a2c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.119+0000 7f84ae000640 1 -- 172.21.15.27:0/4221310491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498035020 con 0x7f84b00c69a0 2024-03-28T19:16:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.119+0000 7f84ae000640 1 -- 172.21.15.27:0/4221310491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498031870 con 0x7f84b00c69a0 2024-03-28T19:16:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.119+0000 7f84ae000640 1 -- 172.21.15.27:0/4221310491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498031c40 con 0x7f84b00c69a0 2024-03-28T19:16:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.120+0000 7f84afe00640 1 -- 172.21.15.27:0/4221310491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00c69a0 msgr2=0x7f84b00c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.120+0000 7f84afe00640 1 --2- 172.21.15.27:0/4221310491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00c69a0 0x7f84b00c6e30 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f84b00c0af0 tx=0x7f849802b910 comp rx=0 tx=0).stop 2024-03-28T19:16:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.120+0000 7f84afe00640 1 -- 172.21.15.27:0/4221310491 shutdown_connections 2024-03-28T19:16:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.120+0000 7f84afe00640 1 --2- 172.21.15.27:0/4221310491 >> v2:172.21.15.155:3300/0 conn(0x7f84b0153a30 0x7f84b0153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.120+0000 7f84afe00640 1 --2- 172.21.15.27:0/4221310491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00c69a0 0x7f84b00c6e30 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.120+0000 7f84afe00640 1 -- 172.21.15.27:0/4221310491 >> 172.21.15.27:0/4221310491 conn(0x7f84b006b0c0 msgr2=0x7f84b006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.121+0000 7f84afe00640 1 -- 172.21.15.27:0/4221310491 shutdown_connections 2024-03-28T19:16:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.121+0000 7f84afe00640 1 -- 172.21.15.27:0/4221310491 wait complete. 2024-03-28T19:16:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.123+0000 7f84afe00640 1 Processor -- start 2024-03-28T19:16:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.124+0000 7f84afe00640 1 -- start start 2024-03-28T19:16:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84afe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84b00c69a0 0x7f84b01eba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b0150ae0 0x7f84b014dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84afe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84b0153a30 0x7f84b00ab290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84afe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b006e4d0 con 0x7f84b0150ae0 2024-03-28T19:16:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84afe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b006e1f0 con 0x7f84b0153a30 2024-03-28T19:16:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84afe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b006e7b0 con 0x7f84b00c69a0 2024-03-28T19:16:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84a6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b0150ae0 0x7f84b014dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84aea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84b00c69a0 0x7f84b01eba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84a6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b0150ae0 0x7f84b014dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52714/0 (socket says 172.21.15.27:52714) 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84aea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84b00c69a0 0x7f84b01eba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:42758/0 (socket says 172.21.15.27:42758) 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84a6000640 1 -- 172.21.15.27:0/699371798 learned_addr learned my addr 172.21.15.27:0/699371798 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.125+0000 7f84af400640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84b0153a30 0x7f84b00ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.126+0000 7f84a6000640 1 -- 172.21.15.27:0/699371798 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84b00c69a0 msgr2=0x7f84b01eba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.126+0000 7f84a6000640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84b00c69a0 0x7f84b01eba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.126+0000 7f84a6000640 1 -- 172.21.15.27:0/699371798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84b0153a30 msgr2=0x7f84b00ab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.126+0000 7f84a6000640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84b0153a30 0x7f84b00ab290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.126+0000 7f84a6000640 1 -- 172.21.15.27:0/699371798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b01ebf40 con 0x7f84b0150ae0 2024-03-28T19:16:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.127+0000 7f84a6000640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b0150ae0 0x7f84b014dd70 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f849802b950 tx=0x7f84980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.127+0000 7f84a7e00640 1 -- 172.21.15.27:0/699371798 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498040070 con 0x7f84b0150ae0 2024-03-28T19:16:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.127+0000 7f84a7e00640 1 -- 172.21.15.27:0/699371798 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498030d00 con 0x7f84b0150ae0 2024-03-28T19:16:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.127+0000 7f84a7e00640 1 -- 172.21.15.27:0/699371798 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498035040 con 0x7f84b0150ae0 2024-03-28T19:16:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.127+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b014e2b0 con 0x7f84b0150ae0 2024-03-28T19:16:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.127+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b0072610 con 0x7f84b0150ae0 2024-03-28T19:16:55.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.129+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b0069640 con 0x7f84b0150ae0 2024-03-28T19:16:55.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.130+0000 7f84a7e00640 1 -- 172.21.15.27:0/699371798 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f849804b060 con 0x7f84b0150ae0 2024-03-28T19:16:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.131+0000 7f84a7e00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f848c06bbd0 0x7f848c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.131+0000 7f84a7e00640 1 -- 172.21.15.27:0/699371798 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f84980ad380 con 0x7f84b0150ae0 2024-03-28T19:16:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.132+0000 7f84aea00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f848c06bbd0 0x7f848c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.133+0000 7f84aea00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f848c06bbd0 0x7f848c06e090 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f84b01554e0 tx=0x7f84a0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.138+0000 7f84a7e00640 1 -- 172.21.15.27:0/699371798 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f84b014e2b0 con 0x7f84b0150ae0 2024-03-28T19:16:55.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.295+0000 7f3136400640 1 Processor -- start 2024-03-28T19:16:55.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.296+0000 7f3136400640 1 -- start start 2024-03-28T19:16:55.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f3136400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3130152bd0 0x7f313006eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f3136400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f31300c4dc0 0x7f313015ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f3136400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300c52e0 con 0x7f313006f1e0 2024-03-28T19:16:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f3136400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31301557e0 con 0x7f3130152bd0 2024-03-28T19:16:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f3136400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3130155ac0 con 0x7f31300c4dc0 2024-03-28T19:16:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f312f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3130152bd0 0x7f313006eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f312ea00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f313006f1e0 0x7f3130070540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:57738/0 (socket says 172.21.15.27:57738) 2024-03-28T19:16:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f312ea00640 1 -- 172.21.15.27:0/1029758971 learned_addr learned my addr 172.21.15.27:0/1029758971 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.298+0000 7f312fe00640 1 --2- 172.21.15.27:0/1029758971 >> v2:172.21.15.155:3300/0 conn(0x7f31300c4dc0 0x7f313015ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.299+0000 7f312e000640 1 -- 172.21.15.27:0/1029758971 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046456217 0 0) 0x7f31300c52e0 con 0x7f313006f1e0 2024-03-28T19:16:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.299+0000 7f312f400640 1 -- 172.21.15.27:0/1029758971 >> v2:172.21.15.155:3300/0 conn(0x7f31300c4dc0 msgr2=0x7f313015ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:55.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.299+0000 7f312f400640 1 --2- 172.21.15.27:0/1029758971 >> v2:172.21.15.155:3300/0 conn(0x7f31300c4dc0 0x7f313015ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.299+0000 7f312f400640 1 -- 172.21.15.27:0/1029758971 >> v1:172.21.15.155:6789/0 conn(0x7f313006f1e0 legacy=0x7f3130070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.299+0000 7f312f400640 1 -- 172.21.15.27:0/1029758971 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130155da0 con 0x7f3130152bd0 2024-03-28T19:16:55.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.301+0000 7f312f400640 1 --2- 172.21.15.27:0/1029758971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3130152bd0 0x7f313006eca0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f31300702d0 tx=0x7f311c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5577745f5515e05f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.301+0000 7f312e000640 1 -- 172.21.15.27:0/1029758971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c035020 con 0x7f3130152bd0 2024-03-28T19:16:55.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.301+0000 7f312e000640 1 -- 172.21.15.27:0/1029758971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311c031640 con 0x7f3130152bd0 2024-03-28T19:16:55.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.301+0000 7f312e000640 1 -- 172.21.15.27:0/1029758971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c02d040 con 0x7f3130152bd0 2024-03-28T19:16:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.302+0000 7f3136400640 1 -- 172.21.15.27:0/1029758971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3130152bd0 msgr2=0x7f313006eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.302+0000 7f3136400640 1 --2- 172.21.15.27:0/1029758971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3130152bd0 0x7f313006eca0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f31300702d0 tx=0x7f311c02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.302+0000 7f3136400640 1 -- 172.21.15.27:0/1029758971 shutdown_connections 2024-03-28T19:16:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.302+0000 7f3136400640 1 --2- 172.21.15.27:0/1029758971 >> v2:172.21.15.155:3300/0 conn(0x7f31300c4dc0 0x7f313015ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.302+0000 7f3136400640 1 --2- 172.21.15.27:0/1029758971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3130152bd0 0x7f313006eca0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.302+0000 7f3136400640 1 -- 172.21.15.27:0/1029758971 >> 172.21.15.27:0/1029758971 conn(0x7f3130066dd0 msgr2=0x7f31300671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:55.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.304+0000 7f3136400640 1 -- 172.21.15.27:0/1029758971 shutdown_connections 2024-03-28T19:16:55.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.304+0000 7f3136400640 1 -- 172.21.15.27:0/1029758971 wait complete. 2024-03-28T19:16:55.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.307+0000 7f3136400640 1 Processor -- start 2024-03-28T19:16:55.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.307+0000 7f3136400640 1 -- start start 2024-03-28T19:16:55.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.308+0000 7f3136400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313006f1e0 0x7f3130158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.308+0000 7f3136400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31300c4dc0 0x7f31300abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.308+0000 7f3136400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f313014dfb0 0x7f313014e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.309+0000 7f3136400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3130070dd0 con 0x7f313014dfb0 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.309+0000 7f3136400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3130070c50 con 0x7f313006f1e0 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.309+0000 7f3136400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31300c6c70 con 0x7f31300c4dc0 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.309+0000 7f312f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313006f1e0 0x7f3130158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.309+0000 7f312ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31300c4dc0 0x7f31300abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.309+0000 7f312ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31300c4dc0 0x7f31300abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:42772/0 (socket says 172.21.15.27:42772) 2024-03-28T19:16:55.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.309+0000 7f312ea00640 1 -- 172.21.15.27:0/3718584932 learned_addr learned my addr 172.21.15.27:0/3718584932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.309+0000 7f312fe00640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f313014dfb0 0x7f313014e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.310+0000 7f312ea00640 1 -- 172.21.15.27:0/3718584932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313006f1e0 msgr2=0x7f3130158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.310+0000 7f312ea00640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313006f1e0 0x7f3130158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.310+0000 7f312ea00640 1 -- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f313014dfb0 msgr2=0x7f313014e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.310+0000 7f312ea00640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f313014dfb0 0x7f313014e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.310+0000 7f312ea00640 1 -- 172.21.15.27:0/3718584932 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130158540 con 0x7f31300c4dc0 2024-03-28T19:16:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.310+0000 7f312fe00640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f313014dfb0 0x7f313014e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:16:55.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.311+0000 7f312ea00640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31300c4dc0 0x7f31300abe10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f312000c460 tx=0x7f312000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.311+0000 7f3127400640 1 -- 172.21.15.27:0/3718584932 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120018070 con 0x7f31300c4dc0 2024-03-28T19:16:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.311+0000 7f3127400640 1 -- 172.21.15.27:0/3718584932 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3120007520 con 0x7f31300c4dc0 2024-03-28T19:16:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.311+0000 7f3127400640 1 -- 172.21.15.27:0/3718584932 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120004a20 con 0x7f31300c4dc0 2024-03-28T19:16:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.311+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313014e970 con 0x7f31300c4dc0 2024-03-28T19:16:55.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.312+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31300aade0 con 0x7f31300c4dc0 2024-03-28T19:16:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.313+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313004f400 con 0x7f31300c4dc0 2024-03-28T19:16:55.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.314+0000 7f3127400640 1 -- 172.21.15.27:0/3718584932 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f312001b070 con 0x7f31300c4dc0 2024-03-28T19:16:55.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.315+0000 7f3127400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311006bc20 0x7f311006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.315+0000 7f3127400640 1 -- 172.21.15.27:0/3718584932 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f312008aaa0 con 0x7f31300c4dc0 2024-03-28T19:16:55.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.315+0000 7f312f400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311006bc20 0x7f311006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.316+0000 7f312f400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311006bc20 0x7f311006e0e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f311c036000 tx=0x7f311c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.322+0000 7f3127400640 1 -- 172.21.15.27:0/3718584932 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3120058240 con 0x7f31300c4dc0 2024-03-28T19:16:55.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.522+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84b00c1020 con 0x7f84b0150ae0 2024-03-28T19:16:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.523+0000 7f84a7e00640 1 -- 172.21.15.27:0/699371798 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f849807b0f0 con 0x7f84b0150ae0 2024-03-28T19:16:55.522 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:55.522 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:16:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f848c06bbd0 msgr2=0x7f848c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f848c06bbd0 0x7f848c06e090 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f84b01554e0 tx=0x7f84a0009040 comp rx=0 tx=0).stop 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b0150ae0 msgr2=0x7f84b014dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b0150ae0 0x7f84b014dd70 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f849802b950 tx=0x7f84980046c0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 shutdown_connections 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f848c06bbd0 0x7f848c06e090 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84b0153a30 0x7f84b00ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b0150ae0 0x7f84b014dd70 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 --2- 172.21.15.27:0/699371798 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84b00c69a0 0x7f84b01eba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.525+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 >> 172.21.15.27:0/699371798 conn(0x7f84b006b0c0 msgr2=0x7f84b0155b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.526+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 shutdown_connections 2024-03-28T19:16:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.526+0000 7f84afe00640 1 -- 172.21.15.27:0/699371798 wait complete. 2024-03-28T19:16:55.545 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-03-28T19:16:55.546 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:16:55.546 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:16:55.626 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:16:55.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.652+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31300c2860 con 0x7f31300c4dc0 2024-03-28T19:16:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.653+0000 7f3127400640 1 -- 172.21.15.27:0/3718584932 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f312005c2e0 con 0x7f31300c4dc0 2024-03-28T19:16:55.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:55.652 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:16:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311006bc20 msgr2=0x7f311006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311006bc20 0x7f311006e0e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f311c036000 tx=0x7f311c03c040 comp rx=0 tx=0).stop 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31300c4dc0 msgr2=0x7f31300abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31300c4dc0 0x7f31300abe10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f312000c460 tx=0x7f312000c920 comp rx=0 tx=0).stop 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 shutdown_connections 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f311006bc20 0x7f311006e0e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f313014dfb0 0x7f313014e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31300c4dc0 0x7f31300abe10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 --2- 172.21.15.27:0/3718584932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f313006f1e0 0x7f3130158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 >> 172.21.15.27:0/3718584932 conn(0x7f3130066dd0 msgr2=0x7f3130070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 shutdown_connections 2024-03-28T19:16:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.656+0000 7f3136400640 1 -- 172.21.15.27:0/3718584932 wait complete. 2024-03-28T19:16:55.676 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-03-28T19:16:55.676 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T19:16:55.677 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:16:55.701 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T19:16:55.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:55.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.853+0000 7f1877e00640 1 Processor -- start 2024-03-28T19:16:55.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.853+0000 7f1877e00640 1 -- start start 2024-03-28T19:16:55.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.854+0000 7f1877e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f187014fb10 0x7f18700c2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.855+0000 7f1877e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18700a7400 0x7f18701529f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.855+0000 7f1877e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700c78e0 con 0x7f187014fef0 2024-03-28T19:16:55.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.855+0000 7f1877e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1870152560 con 0x7f18700a7400 2024-03-28T19:16:55.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.855+0000 7f1877e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18700c3e80 con 0x7f187014fb10 2024-03-28T19:16:55.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.855+0000 7f1875e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18700a7400 0x7f18701529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.855+0000 7f1875e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18700a7400 0x7f18701529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52744/0 (socket says 172.21.15.27:52744) 2024-03-28T19:16:55.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.855+0000 7f1875e00640 1 -- 172.21.15.27:0/650390625 learned_addr learned my addr 172.21.15.27:0/650390625 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.855+0000 7f1875400640 1 --2- 172.21.15.27:0/650390625 >> v2:172.21.15.155:3300/0 conn(0x7f187014fb10 0x7f18700c2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.856+0000 7f1875e00640 1 -- 172.21.15.27:0/650390625 >> v2:172.21.15.155:3300/0 conn(0x7f187014fb10 msgr2=0x7f18700c2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.856+0000 7f1875e00640 1 --2- 172.21.15.27:0/650390625 >> v2:172.21.15.155:3300/0 conn(0x7f187014fb10 0x7f18700c2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.856+0000 7f1875e00640 1 -- 172.21.15.27:0/650390625 >> v1:172.21.15.155:6789/0 conn(0x7f187014fef0 legacy=0x7f187006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.856+0000 7f1875e00640 1 -- 172.21.15.27:0/650390625 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18700c4160 con 0x7f18700a7400 2024-03-28T19:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.857+0000 7f1875e00640 1 --2- 172.21.15.27:0/650390625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18701529f0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f18700c5dd0 tx=0x7f186802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe04a5e54e74cd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.857+0000 7f186fe00640 1 -- 172.21.15.27:0/650390625 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186802ca80 con 0x7f18700a7400 2024-03-28T19:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.857+0000 7f186fe00640 1 -- 172.21.15.27:0/650390625 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186802cc20 con 0x7f18700a7400 2024-03-28T19:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.857+0000 7f186fe00640 1 -- 172.21.15.27:0/650390625 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18680343f0 con 0x7f18700a7400 2024-03-28T19:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.858+0000 7f1877e00640 1 -- 172.21.15.27:0/650390625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 msgr2=0x7f18701529f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.858+0000 7f1877e00640 1 --2- 172.21.15.27:0/650390625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18701529f0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f18700c5dd0 tx=0x7f186802bee0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.858+0000 7f1877e00640 1 -- 172.21.15.27:0/650390625 shutdown_connections 2024-03-28T19:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.858+0000 7f1877e00640 1 --2- 172.21.15.27:0/650390625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18701529f0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.858+0000 7f1877e00640 1 --2- 172.21.15.27:0/650390625 >> v2:172.21.15.155:3300/0 conn(0x7f187014fb10 0x7f18700c2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.858+0000 7f1877e00640 1 -- 172.21.15.27:0/650390625 >> 172.21.15.27:0/650390625 conn(0x7f18700c4750 msgr2=0x7f18700c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.858+0000 7f1877e00640 1 -- 172.21.15.27:0/650390625 shutdown_connections 2024-03-28T19:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.858+0000 7f1877e00640 1 -- 172.21.15.27:0/650390625 wait complete. 2024-03-28T19:16:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.860+0000 7f1877e00640 1 Processor -- start 2024-03-28T19:16:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.860+0000 7f1877e00640 1 -- start start 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1877e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18700c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1877e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f187014fb10 0x7f1870143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1877e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f187014fef0 0x7f1870143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1877e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187006c7e0 con 0x7f18700a7400 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1877e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700c7570 con 0x7f187014fef0 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1877e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187006c960 con 0x7f187014fb10 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1875400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18700c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1875400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18700c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52756/0 (socket says 172.21.15.27:52756) 2024-03-28T19:16:55.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1875400640 1 -- 172.21.15.27:0/2624487234 learned_addr learned my addr 172.21.15.27:0/2624487234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:55.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.861+0000 7f1874a00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f187014fb10 0x7f1870143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1875e00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f187014fef0 0x7f1870143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1875400640 1 -- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f187014fb10 msgr2=0x7f1870143a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1875400640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f187014fb10 0x7f1870143a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1875400640 1 -- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f187014fef0 msgr2=0x7f1870143f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1875400640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f187014fef0 0x7f1870143f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1875400640 1 -- 172.21.15.27:0/2624487234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870146070 con 0x7f18700a7400 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1875e00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f187014fef0 0x7f1870143f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1875400640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18700c5c60 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f186400d800 tx=0x7f186400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f186d600640 1 -- 172.21.15.27:0/2624487234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186400b660 con 0x7f18700a7400 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f186d600640 1 -- 172.21.15.27:0/2624487234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864019070 con 0x7f18700a7400 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f186d600640 1 -- 172.21.15.27:0/2624487234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186400ba20 con 0x7f18700a7400 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.862+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18701ee560 con 0x7f18700a7400 2024-03-28T19:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.863+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18701eeb20 con 0x7f18700a7400 2024-03-28T19:16:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.864+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187004f400 con 0x7f18700a7400 2024-03-28T19:16:55.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.865+0000 7f186d600640 1 -- 172.21.15.27:0/2624487234 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f186400bbc0 con 0x7f18700a7400 2024-03-28T19:16:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.866+0000 7f186d600640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f185006bbd0 0x7f185006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.866+0000 7f186d600640 1 -- 172.21.15.27:0/2624487234 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f186408bd20 con 0x7f18700a7400 2024-03-28T19:16:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.866+0000 7f1874a00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f185006bbd0 0x7f185006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.867+0000 7f1874a00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f185006bbd0 0x7f185006e090 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f185c006fd0 tx=0x7f185c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.871+0000 7f186d600640 1 -- 172.21.15.27:0/2624487234 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1870146070 con 0x7f18700a7400 2024-03-28T19:16:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.948+0000 7f64aa600640 1 Processor -- start 2024-03-28T19:16:55.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.948+0000 7f64aa600640 1 -- start start 2024-03-28T19:16:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.949+0000 7f64aa600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c7df0 0x7f64a4155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.950+0000 7f64aa600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a40c7310 con 0x7f64a40c7970 2024-03-28T19:16:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.950+0000 7f64aa600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a4158d40 con 0x7f64a4152f80 2024-03-28T19:16:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.950+0000 7f64aa600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a4159020 con 0x7f64a40c7df0 2024-03-28T19:16:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.950+0000 7f64a2a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f64a40c7970 0x7f64a4155730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54594/0 (socket says 172.21.15.27:54594) 2024-03-28T19:16:55.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.950+0000 7f64a2a00640 1 -- 172.21.15.27:0/1128470346 learned_addr learned my addr 172.21.15.27:0/1128470346 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:16:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.950+0000 7f64a3e00640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c7df0 0x7f64a4155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.950+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173483316 0 0) 0x7f64a40c7310 con 0x7f64a40c7970 2024-03-28T19:16:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.950+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6488003520 con 0x7f64a40c7970 2024-03-28T19:16:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2486867386 0 0) 0x7f6488003520 con 0x7f64a40c7970 2024-03-28T19:16:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c7df0 msgr2=0x7f64a4155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c7df0 0x7f64a4155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 >> v1:172.21.15.155:6789/0 conn(0x7f64a4152f80 legacy=0x7f64a40c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4159300 con 0x7f64a40c7970 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f6490001670 con 0x7f64a40c7970 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 >> v1:172.21.15.27:6789/0 conn(0x7f64a40c7970 legacy=0x7f64a4155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f6488003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6488004800 0x7f6488004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c7df0 0x7f6488005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a4159020 con 0x7f64880039e0 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648800baf0 con 0x7f6488004800 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648800bcf0 con 0x7f64a40c7df0 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.951+0000 7f64a3400640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f6488003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.952+0000 7f64a3e00640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c7df0 0x7f6488005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.952+0000 7f64a2a00640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6488004800 0x7f6488004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.952+0000 7f64a3400640 1 -- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c7df0 msgr2=0x7f6488005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.952+0000 7f64a3400640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c7df0 0x7f6488005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.952+0000 7f64a3400640 1 -- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6488004800 msgr2=0x7f6488004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.952+0000 7f64a3400640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6488004800 0x7f6488004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.952+0000 7f64a3400640 1 -- 172.21.15.27:0/1128470346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6488003520 con 0x7f64880039e0 2024-03-28T19:16:55.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.953+0000 7f64a3400640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f6488003e00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f6494001750 tx=0x7f6488005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff1fcbe904c426a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.953+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494001370 con 0x7f64880039e0 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.953+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64940052d0 con 0x7f64880039e0 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.954+0000 7f64a2000640 1 -- 172.21.15.27:0/1128470346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494003830 con 0x7f64880039e0 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.954+0000 7f64aa600640 1 -- 172.21.15.27:0/1128470346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 msgr2=0x7f6488003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.954+0000 7f64aa600640 1 --2- 172.21.15.27:0/1128470346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f6488003e00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f6494001750 tx=0x7f6488005830 comp rx=0 tx=0).stop 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.954+0000 7f64a3e00640 1 -- 172.21.15.27:0/1128470346 reap_dead start 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.954+0000 7f64aa600640 1 -- 172.21.15.27:0/1128470346 shutdown_connections 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.954+0000 7f64aa600640 1 -- 172.21.15.27:0/1128470346 >> 172.21.15.27:0/1128470346 conn(0x7f64a406b6f0 msgr2=0x7f64a40caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.954+0000 7f64aa600640 1 -- 172.21.15.27:0/1128470346 shutdown_connections 2024-03-28T19:16:55.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.954+0000 7f64aa600640 1 -- 172.21.15.27:0/1128470346 wait complete. 2024-03-28T19:16:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.956+0000 7f64aa600640 1 Processor -- start 2024-03-28T19:16:55.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.956+0000 7f64aa600640 1 -- start start 2024-03-28T19:16:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.957+0000 7f64aa600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f64a40c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.957+0000 7f64aa600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a40b0450 0x7f64a4148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.957+0000 7f64aa600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c9d90 0x7f64a40ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.957+0000 7f64aa600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a40cbca0 con 0x7f64880039e0 2024-03-28T19:16:55.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.957+0000 7f64aa600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a4159b00 con 0x7f64a40b0450 2024-03-28T19:16:55.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.957+0000 7f64aa600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a40ad260 con 0x7f64a40c9d90 2024-03-28T19:16:55.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.957+0000 7f64a3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f64a40c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f64a40c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52764/0 (socket says 172.21.15.27:52764) 2024-03-28T19:16:55.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a3400640 1 -- 172.21.15.27:0/2335674886 learned_addr learned my addr 172.21.15.27:0/2335674886 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a2a00640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a40b0450 0x7f64a4148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a3400640 1 -- 172.21.15.27:0/2335674886 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c9d90 msgr2=0x7f64a40ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a3400640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c9d90 0x7f64a40ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a3400640 1 -- 172.21.15.27:0/2335674886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a40b0450 msgr2=0x7f64a4148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a3400640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a40b0450 0x7f64a4148770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a3400640 1 -- 172.21.15.27:0/2335674886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a40b0a30 con 0x7f64880039e0 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.958+0000 7f64a3400640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f64a40c4c10 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f6494005040 tx=0x7f6494001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.959+0000 7f649b400640 1 -- 172.21.15.27:0/2335674886 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494041070 con 0x7f64880039e0 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.959+0000 7f649b400640 1 -- 172.21.15.27:0/2335674886 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6494005480 con 0x7f64880039e0 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.959+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a40ad870 con 0x7f64880039e0 2024-03-28T19:16:55.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.959+0000 7f649b400640 1 -- 172.21.15.27:0/2335674886 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494001370 con 0x7f64880039e0 2024-03-28T19:16:55.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.959+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a40ad500 con 0x7f64880039e0 2024-03-28T19:16:55.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.960+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a404f400 con 0x7f64880039e0 2024-03-28T19:16:55.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.961+0000 7f649b400640 1 -- 172.21.15.27:0/2335674886 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6494044070 con 0x7f64880039e0 2024-03-28T19:16:55.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.962+0000 7f649b400640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f648406bbd0 0x7f648406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:55.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.962+0000 7f649b400640 1 -- 172.21.15.27:0/2335674886 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f64940ad2c0 con 0x7f64880039e0 2024-03-28T19:16:55.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.962+0000 7f64a2a00640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f648406bbd0 0x7f648406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:55.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.963+0000 7f64a2a00640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f648406bbd0 0x7f648406e090 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6490000c00 tx=0x7f6490001db0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:55.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:55.966+0000 7f649b400640 1 -- 172.21.15.27:0/2335674886 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f649407afb0 con 0x7f64880039e0 2024-03-28T19:16:56.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.156+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1870073590 con 0x7f18700a7400 2024-03-28T19:16:56.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.157+0000 7f186d600640 1 -- 172.21.15.27:0/2624487234 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f1864059a00 con 0x7f18700a7400 2024-03-28T19:16:56.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:56.156 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:56.156 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f185006bbd0 msgr2=0x7f185006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f185006bbd0 0x7f185006e090 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f185c006fd0 tx=0x7f185c008040 comp rx=0 tx=0).stop 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 msgr2=0x7f18700c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18700c5c60 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f186400d800 tx=0x7f186400dcc0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 shutdown_connections 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f185006bbd0 0x7f185006e090 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f187014fef0 0x7f1870143f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f187014fb10 0x7f1870143a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 --2- 172.21.15.27:0/2624487234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18700a7400 0x7f18700c5c60 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 >> 172.21.15.27:0/2624487234 conn(0x7f18700c4750 msgr2=0x7f1870150e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:56.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 shutdown_connections 2024-03-28T19:16:56.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.160+0000 7f1877e00640 1 -- 172.21.15.27:0/2624487234 wait complete. 2024-03-28T19:16:56.178 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:16:56.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.212+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a4068610 con 0x7f64880039e0 2024-03-28T19:16:56.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.214+0000 7f649b400640 1 -- 172.21.15.27:0/2335674886 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f649407a8b0 con 0x7f64880039e0 2024-03-28T19:16:56.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:56.213 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:56.213 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:16:56.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.216+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f648406bbd0 msgr2=0x7f648406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:56.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.216+0000 7f64aa600640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f648406bbd0 0x7f648406e090 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6490000c00 tx=0x7f6490001db0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 msgr2=0x7f64a40c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:56.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f64a40c4c10 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f6494005040 tx=0x7f6494001120 comp rx=0 tx=0).stop 2024-03-28T19:16:56.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 shutdown_connections 2024-03-28T19:16:56.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f648406bbd0 0x7f648406e090 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64a40c9d90 0x7f64a40ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a40b0450 0x7f64a4148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 --2- 172.21.15.27:0/2335674886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64880039e0 0x7f64a40c4c10 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:56.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 >> 172.21.15.27:0/2335674886 conn(0x7f64a406b6f0 msgr2=0x7f64a414b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:56.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 shutdown_connections 2024-03-28T19:16:56.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:56.217+0000 7f64aa600640 1 -- 172.21.15.27:0/2335674886 wait complete. 2024-03-28T19:16:56.234 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:16:58.179 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:58.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:16:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.409+0000 7f7662800640 1 Processor -- start 2024-03-28T19:16:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.409+0000 7f7662800640 1 -- start start 2024-03-28T19:16:58.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f7662800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f765c06ee00 0x7f765c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f7662800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c0c3ea0 0x7f765c159440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f7662800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765c0c1600 con 0x7f765c0c60e0 2024-03-28T19:16:58.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f7662800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765c0c23b0 con 0x7f765c06ee00 2024-03-28T19:16:58.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f7662800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c0c18e0 con 0x7f765c0c3ea0 2024-03-28T19:16:58.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f765aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f765c06ee00 0x7f765c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f765aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f765c06ee00 0x7f765c158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35430/0 (socket says 172.21.15.27:35430) 2024-03-28T19:16:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f765aa00640 1 -- 172.21.15.27:0/3664338139 learned_addr learned my addr 172.21.15.27:0/3664338139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.411+0000 7f765be00640 1 --2- 172.21.15.27:0/3664338139 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c0c3ea0 0x7f765c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.412+0000 7f765aa00640 1 -- 172.21.15.27:0/3664338139 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c0c3ea0 msgr2=0x7f765c159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.412+0000 7f765aa00640 1 --2- 172.21.15.27:0/3664338139 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c0c3ea0 0x7f765c159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.412+0000 7f765aa00640 1 -- 172.21.15.27:0/3664338139 >> v1:172.21.15.155:6789/0 conn(0x7f765c0c60e0 legacy=0x7f765c06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.412+0000 7f765aa00640 1 -- 172.21.15.27:0/3664338139 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c0c1bc0 con 0x7f765c06ee00 2024-03-28T19:16:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.413+0000 7f765aa00640 1 --2- 172.21.15.27:0/3664338139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c06ee00 0x7f765c158f00 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7648009870 tx=0x7f764802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430776b3b1e775bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.413+0000 7f765a000640 1 -- 172.21.15.27:0/3664338139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76480317c0 con 0x7f765c06ee00 2024-03-28T19:16:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.413+0000 7f765a000640 1 -- 172.21.15.27:0/3664338139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7648037070 con 0x7f765c06ee00 2024-03-28T19:16:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.413+0000 7f765a000640 1 -- 172.21.15.27:0/3664338139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648031b80 con 0x7f765c06ee00 2024-03-28T19:16:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.414+0000 7f7662800640 1 -- 172.21.15.27:0/3664338139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c06ee00 msgr2=0x7f765c158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.414+0000 7f7662800640 1 --2- 172.21.15.27:0/3664338139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c06ee00 0x7f765c158f00 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7648009870 tx=0x7f764802b860 comp rx=0 tx=0).stop 2024-03-28T19:16:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.414+0000 7f7662800640 1 -- 172.21.15.27:0/3664338139 shutdown_connections 2024-03-28T19:16:58.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.414+0000 7f7662800640 1 --2- 172.21.15.27:0/3664338139 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c0c3ea0 0x7f765c159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.414+0000 7f7662800640 1 --2- 172.21.15.27:0/3664338139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c06ee00 0x7f765c158f00 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.414+0000 7f7662800640 1 -- 172.21.15.27:0/3664338139 >> 172.21.15.27:0/3664338139 conn(0x7f765c06b9d0 msgr2=0x7f765c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:58.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.414+0000 7f7662800640 1 -- 172.21.15.27:0/3664338139 shutdown_connections 2024-03-28T19:16:58.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.414+0000 7f7662800640 1 -- 172.21.15.27:0/3664338139 wait complete. 2024-03-28T19:16:58.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.416+0000 7f7662800640 1 Processor -- start 2024-03-28T19:16:58.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.416+0000 7f7662800640 1 -- start start 2024-03-28T19:16:58.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.417+0000 7f7662800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c06ee00 0x7f765c1ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.417+0000 7f7662800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c0c3ea0 0x7f765c0a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.417+0000 7f7662800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0c60e0 0x7f765c155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.417+0000 7f7662800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c0c6640 con 0x7f765c0c3ea0 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.417+0000 7f7662800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c0c64c0 con 0x7f765c0c60e0 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.417+0000 7f7662800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c159980 con 0x7f765c06ee00 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.417+0000 7f765aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c0c3ea0 0x7f765c0a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c0c3ea0 0x7f765c0a5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35438/0 (socket says 172.21.15.27:35438) 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765aa00640 1 -- 172.21.15.27:0/856072303 learned_addr learned my addr 172.21.15.27:0/856072303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765b400640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c06ee00 0x7f765c1ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765be00640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0c60e0 0x7f765c155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765aa00640 1 -- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c06ee00 msgr2=0x7f765c1ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765aa00640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c06ee00 0x7f765c1ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765aa00640 1 -- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0c60e0 msgr2=0x7f765c155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765aa00640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0c60e0 0x7f765c155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765aa00640 1 -- 172.21.15.27:0/856072303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c09c1f0 con 0x7f765c0c3ea0 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.418+0000 7f765be00640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0c60e0 0x7f765c155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.419+0000 7f765aa00640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c0c3ea0 0x7f765c0a5970 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f7648009870 tx=0x7f76480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.419+0000 7f7653400640 1 -- 172.21.15.27:0/856072303 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648032650 con 0x7f765c0c3ea0 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.419+0000 7f7653400640 1 -- 172.21.15.27:0/856072303 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76480327f0 con 0x7f765c0c3ea0 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.419+0000 7f7653400640 1 -- 172.21.15.27:0/856072303 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648032be0 con 0x7f765c0c3ea0 2024-03-28T19:16:58.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.419+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c1eee30 con 0x7f765c0c3ea0 2024-03-28T19:16:58.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.419+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c1ef110 con 0x7f765c0c3ea0 2024-03-28T19:16:58.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.420+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765c04f400 con 0x7f765c0c3ea0 2024-03-28T19:16:58.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.422+0000 7f7653400640 1 -- 172.21.15.27:0/856072303 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f764804c060 con 0x7f765c0c3ea0 2024-03-28T19:16:58.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.423+0000 7f7653400640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763c06bbd0 0x7f763c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.423+0000 7f7653400640 1 -- 172.21.15.27:0/856072303 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f76480acf90 con 0x7f765c0c3ea0 2024-03-28T19:16:58.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.423+0000 7f765b400640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763c06bbd0 0x7f763c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.424+0000 7f765b400640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763c06bbd0 0x7f763c06e090 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f765c072ca0 tx=0x7f764c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:58.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.427+0000 7f7653400640 1 -- 172.21.15.27:0/856072303 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7648040070 con 0x7f765c0c3ea0 2024-03-28T19:16:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.447+0000 7fb622600640 1 Processor -- start 2024-03-28T19:16:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.447+0000 7fb622600640 1 -- start start 2024-03-28T19:16:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.448+0000 7fb622600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb61c153ed0 0x7fb61c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb622600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb61c06eb80 0x7fb61c15a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb622600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c0c3830 con 0x7fb61c06e6f0 2024-03-28T19:16:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb622600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61c0c6990 con 0x7fb61c153ed0 2024-03-28T19:16:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb622600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61c06d1f0 con 0x7fb61c06eb80 2024-03-28T19:16:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb61b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb61c153ed0 0x7fb61c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb61b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb61c153ed0 0x7fb61c06ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35452/0 (socket says 172.21.15.27:35452) 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb61b400640 1 -- 172.21.15.27:0/870778847 learned_addr learned my addr 172.21.15.27:0/870778847 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb613e00640 1 --1- 172.21.15.27:0/870778847 >> v1:172.21.15.27:6789/0 conn(0x7fb61c06e6f0 0x7fb61c159bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38056/0 (socket says 172.21.15.27:38056) 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb61be00640 1 --2- 172.21.15.27:0/870778847 >> v2:172.21.15.155:3300/0 conn(0x7fb61c06eb80 0x7fb61c15a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb61b400640 1 -- 172.21.15.27:0/870778847 >> v2:172.21.15.155:3300/0 conn(0x7fb61c06eb80 msgr2=0x7fb61c15a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb61b400640 1 --2- 172.21.15.27:0/870778847 >> v2:172.21.15.155:3300/0 conn(0x7fb61c06eb80 0x7fb61c15a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb61b400640 1 -- 172.21.15.27:0/870778847 >> v1:172.21.15.27:6789/0 conn(0x7fb61c06e6f0 legacy=0x7fb61c159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.449+0000 7fb61b400640 1 -- 172.21.15.27:0/870778847 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c06ef60 con 0x7fb61c153ed0 2024-03-28T19:16:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.450+0000 7fb61be00640 1 --2- 172.21.15.27:0/870778847 >> v2:172.21.15.155:3300/0 conn(0x7fb61c06eb80 0x7fb61c15a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:16:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.451+0000 7fb61b400640 1 --2- 172.21.15.27:0/870778847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c06ccb0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0c1920 tx=0x7fb60c02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73d891eb4f7ed3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.451+0000 7fb61aa00640 1 -- 172.21.15.27:0/870778847 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c034070 con 0x7fb61c153ed0 2024-03-28T19:16:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.451+0000 7fb61aa00640 1 -- 172.21.15.27:0/870778847 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb60c030880 con 0x7fb61c153ed0 2024-03-28T19:16:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.451+0000 7fb61aa00640 1 -- 172.21.15.27:0/870778847 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60c030c50 con 0x7fb61c153ed0 2024-03-28T19:16:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.451+0000 7fb622600640 1 -- 172.21.15.27:0/870778847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 msgr2=0x7fb61c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.451+0000 7fb622600640 1 --2- 172.21.15.27:0/870778847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c06ccb0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0c1920 tx=0x7fb60c02b9c0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.452+0000 7fb622600640 1 -- 172.21.15.27:0/870778847 shutdown_connections 2024-03-28T19:16:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.452+0000 7fb622600640 1 --2- 172.21.15.27:0/870778847 >> v2:172.21.15.155:3300/0 conn(0x7fb61c06eb80 0x7fb61c15a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.452+0000 7fb622600640 1 --2- 172.21.15.27:0/870778847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c06ccb0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.452+0000 7fb622600640 1 -- 172.21.15.27:0/870778847 >> 172.21.15.27:0/870778847 conn(0x7fb61c06b9d0 msgr2=0x7fb61c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.452+0000 7fb622600640 1 -- 172.21.15.27:0/870778847 shutdown_connections 2024-03-28T19:16:58.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.452+0000 7fb622600640 1 -- 172.21.15.27:0/870778847 wait complete. 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.453+0000 7fb622600640 1 Processor -- start 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.454+0000 7fb622600640 1 -- start start 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.454+0000 7fb622600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb61c06e6f0 0x7fb61c14c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.454+0000 7fb622600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb61c06eb80 0x7fb61c1f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.454+0000 7fb622600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c1f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.454+0000 7fb622600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c0c3db0 con 0x7fb61c153ed0 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.454+0000 7fb622600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c0c3ad0 con 0x7fb61c06eb80 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.454+0000 7fb622600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c15a810 con 0x7fb61c06e6f0 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c1f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c1f0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35454/0 (socket says 172.21.15.27:35454) 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61be00640 1 -- 172.21.15.27:0/4158378752 learned_addr learned my addr 172.21.15.27:0/4158378752 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61b400640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb61c06e6f0 0x7fb61c14c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb613e00640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb61c06eb80 0x7fb61c1f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61be00640 1 -- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb61c06e6f0 msgr2=0x7fb61c14c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61be00640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb61c06e6f0 0x7fb61c14c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61be00640 1 -- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb61c06eb80 msgr2=0x7fb61c1f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61be00640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb61c06eb80 0x7fb61c1f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61be00640 1 -- 172.21.15.27:0/4158378752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c1e5490 con 0x7fb61c153ed0 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.455+0000 7fb61b400640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb61c06e6f0 0x7fb61c14c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:16:58.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.456+0000 7fb61be00640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c1f0b50 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb61400cef0 tx=0x7fb61400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.456+0000 7fb613400640 1 -- 172.21.15.27:0/4158378752 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61401a070 con 0x7fb61c153ed0 2024-03-28T19:16:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.456+0000 7fb613400640 1 -- 172.21.15.27:0/4158378752 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61400d860 con 0x7fb61c153ed0 2024-03-28T19:16:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.456+0000 7fb613400640 1 -- 172.21.15.27:0/4158378752 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61400dc50 con 0x7fb61c153ed0 2024-03-28T19:16:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.456+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c1f39d0 con 0x7fb61c153ed0 2024-03-28T19:16:58.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.456+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c1e44a0 con 0x7fb61c153ed0 2024-03-28T19:16:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.457+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61c1e3fd0 con 0x7fb61c153ed0 2024-03-28T19:16:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.458+0000 7fb613400640 1 -- 172.21.15.27:0/4158378752 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb61401d070 con 0x7fb61c153ed0 2024-03-28T19:16:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.459+0000 7fb613400640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb5f806bbd0 0x7fb5f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:16:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.459+0000 7fb613400640 1 -- 172.21.15.27:0/4158378752 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7fb61408c0a0 con 0x7fb61c153ed0 2024-03-28T19:16:58.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.459+0000 7fb61b400640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb5f806bbd0 0x7fb5f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:16:58.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.460+0000 7fb61b400640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb5f806bbd0 0x7fb5f806e090 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb60c036000 tx=0x7fb60c03b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:16:58.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.464+0000 7fb613400640 1 -- 172.21.15.27:0/4158378752 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb614059ed0 con 0x7fb61c153ed0 2024-03-28T19:16:58.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.681+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f765c0c4ce0 con 0x7f765c0c3ea0 2024-03-28T19:16:58.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.682+0000 7f7653400640 1 -- 172.21.15.27:0/856072303 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f764807ad00 con 0x7f765c0c3ea0 2024-03-28T19:16:58.681 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:58.681 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:58.682 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:16:58.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763c06bbd0 msgr2=0x7f763c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763c06bbd0 0x7f763c06e090 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f765c072ca0 tx=0x7f764c009040 comp rx=0 tx=0).stop 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c0c3ea0 msgr2=0x7f765c0a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c0c3ea0 0x7f765c0a5970 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f7648009870 tx=0x7f76480046c0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 shutdown_connections 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f763c06bbd0 0x7f763c06e090 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f765c0c60e0 0x7f765c155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f765c0c3ea0 0x7f765c0a5970 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 --2- 172.21.15.27:0/856072303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f765c06ee00 0x7f765c1ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 >> 172.21.15.27:0/856072303 conn(0x7f765c06b9d0 msgr2=0x7f765c0c56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 shutdown_connections 2024-03-28T19:16:58.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.685+0000 7f7662800640 1 -- 172.21.15.27:0/856072303 wait complete. 2024-03-28T19:16:58.710 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:16:58.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.716+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb61c04f400 con 0x7fb61c153ed0 2024-03-28T19:16:58.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.717+0000 7fb613400640 1 -- 172.21.15.27:0/4158378752 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7fb6140597d0 con 0x7fb61c153ed0 2024-03-28T19:16:58.716 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:16:58.716 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:16:58.716 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:16:58.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb5f806bbd0 msgr2=0x7fb5f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb5f806bbd0 0x7fb5f806e090 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb60c036000 tx=0x7fb60c03b040 comp rx=0 tx=0).stop 2024-03-28T19:16:58.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 msgr2=0x7fb61c1f0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:16:58.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c1f0b50 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb61400cef0 tx=0x7fb61400a710 comp rx=0 tx=0).stop 2024-03-28T19:16:58.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 shutdown_connections 2024-03-28T19:16:58.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb5f806bbd0 0x7fb5f806e090 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb61c153ed0 0x7fb61c1f0b50 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb61c06eb80 0x7fb61c1f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 --2- 172.21.15.27:0/4158378752 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb61c06e6f0 0x7fb61c14c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:16:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.719+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 >> 172.21.15.27:0/4158378752 conn(0x7fb61c06b9d0 msgr2=0x7fb61c0c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:16:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.720+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 shutdown_connections 2024-03-28T19:16:58.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:16:58.720+0000 7fb622600640 1 -- 172.21.15.27:0/4158378752 wait complete. 2024-03-28T19:16:58.736 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:17:00.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:00.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.941+0000 7f5019800640 1 Processor -- start 2024-03-28T19:17:00.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.941+0000 7f5019800640 1 -- start start 2024-03-28T19:17:00.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5019800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50140a9cb0 0x7f50140c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5019800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50141534c0 0x7f5014153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5019800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140c3d90 con 0x7f5014150c70 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5019800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50140ab4e0 con 0x7f50141534c0 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5019800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50140ac1f0 con 0x7f50140a9cb0 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5013400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50141534c0 0x7f5014153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5012a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5014150c70 0x7f50140c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38072/0 (socket says 172.21.15.27:38072) 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5012a00640 1 -- 172.21.15.27:0/2271566333 learned_addr learned my addr 172.21.15.27:0/2271566333 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5013400640 1 --2- 172.21.15.27:0/2271566333 >> v2:172.21.15.27:3300/0 conn(0x7f50141534c0 0x7f5014153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35470/0 (socket says 172.21.15.27:35470) 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f500a000640 1 --2- 172.21.15.27:0/2271566333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50140a9cb0 0x7f50140c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.943+0000 7f5013400640 1 -- 172.21.15.27:0/2271566333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50140a9cb0 msgr2=0x7f50140c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.944+0000 7f5013400640 1 --2- 172.21.15.27:0/2271566333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50140a9cb0 0x7f50140c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.944+0000 7f5013400640 1 -- 172.21.15.27:0/2271566333 >> v1:172.21.15.27:6789/0 conn(0x7f5014150c70 legacy=0x7f50140c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.944+0000 7f5013400640 1 -- 172.21.15.27:0/2271566333 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140ab710 con 0x7f50141534c0 2024-03-28T19:17:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.945+0000 7f5013400640 1 --2- 172.21.15.27:0/2271566333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50141534c0 0x7f5014153950 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f500c009dc0 tx=0x7f500c02bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4b352e7df6fb53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.945+0000 7f5012000640 1 -- 172.21.15.27:0/2271566333 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c039070 con 0x7f50141534c0 2024-03-28T19:17:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.945+0000 7f5012000640 1 -- 172.21.15.27:0/2271566333 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c02cba0 con 0x7f50141534c0 2024-03-28T19:17:00.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.945+0000 7f5012000640 1 -- 172.21.15.27:0/2271566333 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c034440 con 0x7f50141534c0 2024-03-28T19:17:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.946+0000 7f5019800640 1 -- 172.21.15.27:0/2271566333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50141534c0 msgr2=0x7f5014153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.946+0000 7f5019800640 1 --2- 172.21.15.27:0/2271566333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50141534c0 0x7f5014153950 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f500c009dc0 tx=0x7f500c02bdf0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.946+0000 7f5019800640 1 -- 172.21.15.27:0/2271566333 shutdown_connections 2024-03-28T19:17:00.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.946+0000 7f5019800640 1 --2- 172.21.15.27:0/2271566333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50141534c0 0x7f5014153950 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.946+0000 7f5019800640 1 --2- 172.21.15.27:0/2271566333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50140a9cb0 0x7f50140c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.946+0000 7f5019800640 1 -- 172.21.15.27:0/2271566333 >> 172.21.15.27:0/2271566333 conn(0x7f501406b9d0 msgr2=0x7f50140c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:00.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.946+0000 7f5019800640 1 -- 172.21.15.27:0/2271566333 shutdown_connections 2024-03-28T19:17:00.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.946+0000 7f5019800640 1 -- 172.21.15.27:0/2271566333 wait complete. 2024-03-28T19:17:00.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.949+0000 7f5019800640 1 Processor -- start 2024-03-28T19:17:00.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.949+0000 7f5019800640 1 -- start start 2024-03-28T19:17:00.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5019800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50140a9cb0 0x7f50141ddf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5019800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5014150c70 0x7f501406e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5019800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50141534c0 0x7f50141eec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5019800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50140c1810 con 0x7f50140a9cb0 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5019800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50140abcd0 con 0x7f50141534c0 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5019800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50140c6830 con 0x7f5014150c70 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5012a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50140a9cb0 0x7f50141ddf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5012a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50140a9cb0 0x7f50141ddf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35486/0 (socket says 172.21.15.27:35486) 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5012a00640 1 -- 172.21.15.27:0/1336157022 learned_addr learned my addr 172.21.15.27:0/1336157022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f500a000640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5014150c70 0x7f501406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5012a00640 1 -- 172.21.15.27:0/1336157022 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5014150c70 msgr2=0x7f501406e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5012a00640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5014150c70 0x7f501406e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5012a00640 1 -- 172.21.15.27:0/1336157022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50141534c0 msgr2=0x7f50141eec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5012a00640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50141534c0 0x7f50141eec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.951+0000 7f5012a00640 1 -- 172.21.15.27:0/1336157022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140a6600 con 0x7f50140a9cb0 2024-03-28T19:17:00.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.952+0000 7f5012a00640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50140a9cb0 0x7f50141ddf80 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f500400c400 tx=0x7f500400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.952+0000 7f500b400640 1 -- 172.21.15.27:0/1336157022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004017070 con 0x7f50140a9cb0 2024-03-28T19:17:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.952+0000 7f500b400640 1 -- 172.21.15.27:0/1336157022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5004005ce0 con 0x7f50140a9cb0 2024-03-28T19:17:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.952+0000 7f500b400640 1 -- 172.21.15.27:0/1336157022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50040072f0 con 0x7f50140a9cb0 2024-03-28T19:17:00.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.952+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50141ee440 con 0x7f50140a9cb0 2024-03-28T19:17:00.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.952+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50141de4c0 con 0x7f50140a9cb0 2024-03-28T19:17:00.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.953+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501404f400 con 0x7f50140a9cb0 2024-03-28T19:17:00.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.955+0000 7f500b400640 1 -- 172.21.15.27:0/1336157022 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f500401a070 con 0x7f50140a9cb0 2024-03-28T19:17:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.956+0000 7f500b400640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ff006bc20 0x7f4ff006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.956+0000 7f500b400640 1 -- 172.21.15.27:0/1336157022 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f500408abe0 con 0x7f50140a9cb0 2024-03-28T19:17:00.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.957+0000 7f500a000640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ff006bc20 0x7f4ff006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.957+0000 7f500a000640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ff006bc20 0x7f4ff006e0e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f501406ed80 tx=0x7f4ffc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:00.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.961+0000 7f500b400640 1 -- 172.21.15.27:0/1336157022 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f50040589d0 con 0x7f50140a9cb0 2024-03-28T19:17:00.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.977+0000 7fd87be00640 1 Processor -- start 2024-03-28T19:17:00.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.978+0000 7fd87be00640 1 -- start start 2024-03-28T19:17:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.979+0000 7fd87be00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd8740a7ae0 0x7fd874159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.979+0000 7fd87be00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8740c9bd0 con 0x7fd8741591a0 2024-03-28T19:17:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.979+0000 7fd87be00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8740c7570 con 0x7fd8740c3e70 2024-03-28T19:17:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.979+0000 7fd87be00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8740c50c0 con 0x7fd8740a7ae0 2024-03-28T19:17:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.979+0000 7fd879400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd8741591a0 0x7fd8740c8cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38076/0 (socket says 172.21.15.27:38076) 2024-03-28T19:17:00.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.979+0000 7fd879400640 1 -- 172.21.15.27:0/974683358 learned_addr learned my addr 172.21.15.27:0/974683358 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.979+0000 7fd878a00640 1 --2- 172.21.15.27:0/974683358 >> v2:172.21.15.155:3300/0 conn(0x7fd8740a7ae0 0x7fd874159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58693401 0 0) 0x7fd8740c9bd0 con 0x7fd8741591a0 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd85c003520 con 0x7fd8741591a0 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2178868081 0 0) 0x7fd85c003520 con 0x7fd8741591a0 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 >> v2:172.21.15.155:3300/0 conn(0x7fd8740a7ae0 msgr2=0x7fd874159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 --2- 172.21.15.27:0/974683358 >> v2:172.21.15.155:3300/0 conn(0x7fd8740a7ae0 0x7fd874159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 >> v1:172.21.15.155:6789/0 conn(0x7fd8740c3e70 legacy=0x7fd8740c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8740c53a0 con 0x7fd8741591a0 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd868001670 con 0x7fd8741591a0 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 >> v1:172.21.15.27:6789/0 conn(0x7fd8741591a0 legacy=0x7fd8740c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd85c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd85c004800 0x7fd85c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd85c0051c0 0x7fd85c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8740c50c0 con 0x7fd85c0039b0 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c00c060 con 0x7fd85c004800 2024-03-28T19:17:00.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.980+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c00c260 con 0x7fd85c0051c0 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd879400640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd85c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd878a00640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd85c004800 0x7fd85c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd879e00640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd85c0051c0 0x7fd85c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd879400640 1 -- 172.21.15.27:0/974683358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd85c0051c0 msgr2=0x7fd85c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd879400640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd85c0051c0 0x7fd85c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd879400640 1 -- 172.21.15.27:0/974683358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd85c004800 msgr2=0x7fd85c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd879400640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd85c004800 0x7fd85c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd879400640 1 -- 172.21.15.27:0/974683358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c003520 con 0x7fd85c0039b0 2024-03-28T19:17:00.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.981+0000 7fd879e00640 1 -- 172.21.15.27:0/974683358 reap_dead start 2024-03-28T19:17:00.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.982+0000 7fd879400640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd85c003dd0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd86800c240 tx=0x7fd85c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d8fee5892aca6d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.982+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868007d10 con 0x7fd85c0039b0 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.982+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd868030070 con 0x7fd85c0039b0 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.982+0000 7fd873e00640 1 -- 172.21.15.27:0/974683358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86802cdc0 con 0x7fd85c0039b0 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.983+0000 7fd87be00640 1 -- 172.21.15.27:0/974683358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 msgr2=0x7fd85c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.983+0000 7fd87be00640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd85c003dd0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd86800c240 tx=0x7fd85c005830 comp rx=0 tx=0).stop 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.983+0000 7fd87be00640 1 -- 172.21.15.27:0/974683358 shutdown_connections 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.983+0000 7fd87be00640 1 --2- 172.21.15.27:0/974683358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd85c003dd0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.983+0000 7fd87be00640 1 -- 172.21.15.27:0/974683358 >> 172.21.15.27:0/974683358 conn(0x7fd8740aaab0 msgr2=0x7fd8740aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.983+0000 7fd87be00640 1 -- 172.21.15.27:0/974683358 shutdown_connections 2024-03-28T19:17:00.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.983+0000 7fd87be00640 1 -- 172.21.15.27:0/974683358 wait complete. 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.985+0000 7fd87be00640 1 Processor -- start 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.985+0000 7fd87be00640 1 -- start start 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd87be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd874150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd87be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8741ec3c0 0x7fd87406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd87be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8741ee730 0x7fd8741eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd87be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874159700 con 0x7fd85c0039b0 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd87be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874159580 con 0x7fd8741ec3c0 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd87be00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd874159880 con 0x7fd8741ee730 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd879400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd874150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd879400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd874150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35516/0 (socket says 172.21.15.27:35516) 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.986+0000 7fd879400640 1 -- 172.21.15.27:0/2108185360 learned_addr learned my addr 172.21.15.27:0/2108185360 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:00.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.987+0000 7fd879400640 1 -- 172.21.15.27:0/2108185360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8741ee730 msgr2=0x7fd8741eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.987+0000 7fd879400640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8741ee730 0x7fd8741eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.987+0000 7fd879400640 1 -- 172.21.15.27:0/2108185360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8741ec3c0 msgr2=0x7fd87406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:00.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.987+0000 7fd879400640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8741ec3c0 0x7fd87406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:00.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.987+0000 7fd879400640 1 -- 172.21.15.27:0/2108185360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8741ec840 con 0x7fd85c0039b0 2024-03-28T19:17:00.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.987+0000 7fd879400640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd874150b60 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd86800c240 tx=0x7fd8740729b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:00.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.988+0000 7fd871600640 1 -- 172.21.15.27:0/2108185360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868045070 con 0x7fd85c0039b0 2024-03-28T19:17:00.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.988+0000 7fd871600640 1 -- 172.21.15.27:0/2108185360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86802c900 con 0x7fd85c0039b0 2024-03-28T19:17:00.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.988+0000 7fd871600640 1 -- 172.21.15.27:0/2108185360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868007d10 con 0x7fd85c0039b0 2024-03-28T19:17:00.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.988+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8740708a0 con 0x7fd85c0039b0 2024-03-28T19:17:00.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.988+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd874070e60 con 0x7fd85c0039b0 2024-03-28T19:17:00.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.989+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87404f400 con 0x7fd85c0039b0 2024-03-28T19:17:00.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.990+0000 7fd871600640 1 -- 172.21.15.27:0/2108185360 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd868007070 con 0x7fd85c0039b0 2024-03-28T19:17:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.991+0000 7fd871600640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd85406bc00 0x7fd85406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.991+0000 7fd871600640 1 -- 172.21.15.27:0/2108185360 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7fd8680ac490 con 0x7fd85c0039b0 2024-03-28T19:17:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.991+0000 7fd878a00640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd85406bc00 0x7fd85406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:00.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:00.992+0000 7fd878a00640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd85406bc00 0x7fd85406e0c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd8740712a0 tx=0x7fd860008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:00.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.000+0000 7fd871600640 1 -- 172.21.15.27:0/2108185360 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd8741ec840 con 0x7fd85c0039b0 2024-03-28T19:17:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.219+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50141ee960 con 0x7f50140a9cb0 2024-03-28T19:17:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.221+0000 7f500b400640 1 -- 172.21.15.27:0/1336157022 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f50040582d0 con 0x7f50140a9cb0 2024-03-28T19:17:01.220 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:01.220 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:17:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:17:01.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ff006bc20 msgr2=0x7f4ff006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:01.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ff006bc20 0x7f4ff006e0e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f501406ed80 tx=0x7f4ffc008040 comp rx=0 tx=0).stop 2024-03-28T19:17:01.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50140a9cb0 msgr2=0x7f50141ddf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50140a9cb0 0x7f50141ddf80 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f500400c400 tx=0x7f500400c8c0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 shutdown_connections 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4ff006bc20 0x7f4ff006e0e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50141534c0 0x7f50141eec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5014150c70 0x7f501406e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 --2- 172.21.15.27:0/1336157022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50140a9cb0 0x7f50141ddf80 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 >> 172.21.15.27:0/1336157022 conn(0x7f501406b9d0 msgr2=0x7f50140c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 shutdown_connections 2024-03-28T19:17:01.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.224+0000 7f5019800640 1 -- 172.21.15.27:0/1336157022 wait complete. 2024-03-28T19:17:01.242 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:17:01.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.245+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd87406c900 con 0x7fd85c0039b0 2024-03-28T19:17:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.246+0000 7fd871600640 1 -- 172.21.15.27:0/2108185360 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7fd868079bb0 con 0x7fd85c0039b0 2024-03-28T19:17:01.245 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:01.245 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:17:01.245 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.248+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd85406bc00 msgr2=0x7fd85406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.248+0000 7fd87be00640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd85406bc00 0x7fd85406e0c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd8740712a0 tx=0x7fd860008040 comp rx=0 tx=0).stop 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.248+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 msgr2=0x7fd874150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.248+0000 7fd87be00640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd874150b60 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd86800c240 tx=0x7fd8740729b0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.249+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 shutdown_connections 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.249+0000 7fd87be00640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd85406bc00 0x7fd85406e0c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.249+0000 7fd87be00640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8741ee730 0x7fd8741eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.249+0000 7fd87be00640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8741ec3c0 0x7fd87406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.249+0000 7fd87be00640 1 --2- 172.21.15.27:0/2108185360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd85c0039b0 0x7fd874150b60 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:01.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.249+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 >> 172.21.15.27:0/2108185360 conn(0x7fd8740aaab0 msgr2=0x7fd8740c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.249+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 shutdown_connections 2024-03-28T19:17:01.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:01.249+0000 7fd87be00640 1 -- 172.21.15.27:0/2108185360 wait complete. 2024-03-28T19:17:01.266 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:17:03.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:03.267 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.469+0000 7fbf5ca00640 1 Processor -- start 2024-03-28T19:17:03.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.469+0000 7fbf5ca00640 1 -- start start 2024-03-28T19:17:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.470+0000 7fbf5ca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbf58150c70 0x7fbf580a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.471+0000 7fbf5ca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf580c3af0 con 0x7fbf581534c0 2024-03-28T19:17:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.471+0000 7fbf5ca00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf58154020 con 0x7fbf580c3690 2024-03-28T19:17:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.471+0000 7fbf5ca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf580ab4e0 con 0x7fbf58150c70 2024-03-28T19:17:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.471+0000 7fbf56800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbf581534c0 0x7fbf58153910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38084/0 (socket says 172.21.15.27:38084) 2024-03-28T19:17:03.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.471+0000 7fbf56800640 1 -- 172.21.15.27:0/1903848956 learned_addr learned my addr 172.21.15.27:0/1903848956 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.471+0000 7fbf55e00640 1 --2- 172.21.15.27:0/1903848956 >> v2:172.21.15.155:3300/0 conn(0x7fbf58150c70 0x7fbf580a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.471+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036179144 0 0) 0x7fbf580c3af0 con 0x7fbf581534c0 2024-03-28T19:17:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf3c003520 con 0x7fbf581534c0 2024-03-28T19:17:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3842468687 0 0) 0x7fbf3c003520 con 0x7fbf581534c0 2024-03-28T19:17:03.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 >> v2:172.21.15.155:3300/0 conn(0x7fbf58150c70 msgr2=0x7fbf580a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 --2- 172.21.15.27:0/1903848956 >> v2:172.21.15.155:3300/0 conn(0x7fbf58150c70 0x7fbf580a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 >> v1:172.21.15.155:6789/0 conn(0x7fbf580c3690 legacy=0x7fbf580c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf580c3d90 con 0x7fbf581534c0 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fbf500019f0 con 0x7fbf581534c0 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 >> v1:172.21.15.27:6789/0 conn(0x7fbf581534c0 legacy=0x7fbf58153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c003a00 0x7fbf3c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbf3c004800 0x7fbf3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf3c005230 0x7fbf3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c00c160 con 0x7fbf3c003a00 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c00c310 con 0x7fbf3c005230 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c00c510 con 0x7fbf3c004800 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.472+0000 7fbf55e00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c003a00 0x7fbf3c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.473+0000 7fbf55e00640 1 -- 172.21.15.27:0/1903848956 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbf3c004800 msgr2=0x7fbf3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:03.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.473+0000 7fbf55e00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbf3c004800 0x7fbf3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.473+0000 7fbf55e00640 1 -- 172.21.15.27:0/1903848956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf3c005230 msgr2=0x7fbf3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.473+0000 7fbf56800640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf3c005230 0x7fbf3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.473+0000 7fbf55e00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf3c005230 0x7fbf3c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.473+0000 7fbf55e00640 1 -- 172.21.15.27:0/1903848956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c003520 con 0x7fbf3c003a00 2024-03-28T19:17:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.473+0000 7fbf56800640 1 -- 172.21.15.27:0/1903848956 reap_dead start 2024-03-28T19:17:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.474+0000 7fbf55e00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c003a00 0x7fbf3c003e40 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009ac0 tx=0x7fbf3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36dc8a48b6bf8e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.474+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48039070 con 0x7fbf3c003a00 2024-03-28T19:17:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.474+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4802c600 con 0x7fbf3c003a00 2024-03-28T19:17:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.474+0000 7fbf54a00640 1 -- 172.21.15.27:0/1903848956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48009690 con 0x7fbf3c003a00 2024-03-28T19:17:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.475+0000 7fbf5ca00640 1 -- 172.21.15.27:0/1903848956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c003a00 msgr2=0x7fbf3c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.475+0000 7fbf5ca00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c003a00 0x7fbf3c003e40 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009ac0 tx=0x7fbf3c005830 comp rx=0 tx=0).stop 2024-03-28T19:17:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.475+0000 7fbf5ca00640 1 -- 172.21.15.27:0/1903848956 shutdown_connections 2024-03-28T19:17:03.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.475+0000 7fbf5ca00640 1 --2- 172.21.15.27:0/1903848956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c003a00 0x7fbf3c003e40 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.475+0000 7fbf5ca00640 1 -- 172.21.15.27:0/1903848956 >> 172.21.15.27:0/1903848956 conn(0x7fbf5806b9d0 msgr2=0x7fbf580c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.475+0000 7fbf5ca00640 1 -- 172.21.15.27:0/1903848956 shutdown_connections 2024-03-28T19:17:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.475+0000 7fbf5ca00640 1 -- 172.21.15.27:0/1903848956 wait complete. 2024-03-28T19:17:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.477+0000 7fbf5ca00640 1 Processor -- start 2024-03-28T19:17:03.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.477+0000 7fbf5ca00640 1 -- start start 2024-03-28T19:17:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf5ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbf3c003a00 0x7fbf580c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf5ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf58072500 0x7fbf58149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf5ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf58149640 0x7fbf581ec4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf5ca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580ab9f0 con 0x7fbf58149640 2024-03-28T19:17:03.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf5ca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580ac1f0 con 0x7fbf58072500 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf5ca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580abcd0 con 0x7fbf3c003a00 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf56800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf58149640 0x7fbf581ec4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf56800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf58149640 0x7fbf581ec4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35532/0 (socket says 172.21.15.27:35532) 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.478+0000 7fbf56800640 1 -- 172.21.15.27:0/591472207 learned_addr learned my addr 172.21.15.27:0/591472207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf55e00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbf3c003a00 0x7fbf580c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf55400640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf58072500 0x7fbf58149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf56800640 1 -- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbf3c003a00 msgr2=0x7fbf580c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf56800640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbf3c003a00 0x7fbf580c2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf56800640 1 -- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf58072500 msgr2=0x7fbf58149100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf56800640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf58072500 0x7fbf58149100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf56800640 1 -- 172.21.15.27:0/591472207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf58154830 con 0x7fbf58149640 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf55400640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf58072500 0x7fbf58149100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:17:03.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.479+0000 7fbf56800640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf58149640 0x7fbf581ec4a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fbf581538a0 tx=0x7fbf5000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:03.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.480+0000 7fbf4e000640 1 -- 172.21.15.27:0/591472207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50004d60 con 0x7fbf58149640 2024-03-28T19:17:03.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.480+0000 7fbf4e000640 1 -- 172.21.15.27:0/591472207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5000bb80 con 0x7fbf58149640 2024-03-28T19:17:03.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.480+0000 7fbf4e000640 1 -- 172.21.15.27:0/591472207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50007070 con 0x7fbf58149640 2024-03-28T19:17:03.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.480+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf581e9880 con 0x7fbf58149640 2024-03-28T19:17:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.480+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf58071700 con 0x7fbf58149640 2024-03-28T19:17:03.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.481+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5804f400 con 0x7fbf58149640 2024-03-28T19:17:03.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.483+0000 7fbf4e000640 1 -- 172.21.15.27:0/591472207 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbf5001b020 con 0x7fbf58149640 2024-03-28T19:17:03.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.484+0000 7fbf4e000640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbf3806bc00 0x7fbf3806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.484+0000 7fbf4e000640 1 -- 172.21.15.27:0/591472207 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7fbf5008b800 con 0x7fbf58149640 2024-03-28T19:17:03.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.484+0000 7fbf55e00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbf3806bc00 0x7fbf3806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.485+0000 7fbf55e00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbf3806bc00 0x7fbf3806e0c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbf48036000 tx=0x7fbf48030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:03.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.489+0000 7fbf4e000640 1 -- 172.21.15.27:0/591472207 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbf500595d0 con 0x7fbf58149640 2024-03-28T19:17:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.509+0000 7f280d000640 1 Processor -- start 2024-03-28T19:17:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.509+0000 7f280d000640 1 -- start start 2024-03-28T19:17:03.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f280d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2808150c70 0x7f28080a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f280d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28081534c0 0x7f2808153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f280d000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28080c3af0 con 0x7f28080c3690 2024-03-28T19:17:03.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f280d000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28080ab4e0 con 0x7f28081534c0 2024-03-28T19:17:03.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f280d000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28080ac1f0 con 0x7f2808150c70 2024-03-28T19:17:03.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f2806800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28081534c0 0x7f2808153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f2806800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28081534c0 0x7f2808153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35534/0 (socket says 172.21.15.27:35534) 2024-03-28T19:17:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f2806800640 1 -- 172.21.15.27:0/2984715835 learned_addr learned my addr 172.21.15.27:0/2984715835 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.511+0000 7f2805e00640 1 --2- 172.21.15.27:0/2984715835 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2808150c70 0x7f28080a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.512+0000 7f2806800640 1 -- 172.21.15.27:0/2984715835 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2808150c70 msgr2=0x7f28080a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.512+0000 7f2806800640 1 --2- 172.21.15.27:0/2984715835 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2808150c70 0x7f28080a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.512+0000 7f2806800640 1 -- 172.21.15.27:0/2984715835 >> v1:172.21.15.155:6789/0 conn(0x7f28080c3690 legacy=0x7f28080c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.512+0000 7f2806800640 1 -- 172.21.15.27:0/2984715835 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28080c3d90 con 0x7f28081534c0 2024-03-28T19:17:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.513+0000 7f2806800640 1 --2- 172.21.15.27:0/2984715835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28081534c0 0x7f2808153950 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f2800009e50 tx=0x7f280002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64993148788207e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.513+0000 7f2805400640 1 -- 172.21.15.27:0/2984715835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280002c9e0 con 0x7f28081534c0 2024-03-28T19:17:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.513+0000 7f2805400640 1 -- 172.21.15.27:0/2984715835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280002cb80 con 0x7f28081534c0 2024-03-28T19:17:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.513+0000 7f2805400640 1 -- 172.21.15.27:0/2984715835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28000343f0 con 0x7f28081534c0 2024-03-28T19:17:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.514+0000 7f280d000640 1 -- 172.21.15.27:0/2984715835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28081534c0 msgr2=0x7f2808153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.514+0000 7f280d000640 1 --2- 172.21.15.27:0/2984715835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28081534c0 0x7f2808153950 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f2800009e50 tx=0x7f280002be40 comp rx=0 tx=0).stop 2024-03-28T19:17:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.514+0000 7f280d000640 1 -- 172.21.15.27:0/2984715835 shutdown_connections 2024-03-28T19:17:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.514+0000 7f280d000640 1 --2- 172.21.15.27:0/2984715835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28081534c0 0x7f2808153950 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.514+0000 7f280d000640 1 --2- 172.21.15.27:0/2984715835 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2808150c70 0x7f28080a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.514+0000 7f280d000640 1 -- 172.21.15.27:0/2984715835 >> 172.21.15.27:0/2984715835 conn(0x7f280806b9d0 msgr2=0x7f28080c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.514+0000 7f280d000640 1 -- 172.21.15.27:0/2984715835 shutdown_connections 2024-03-28T19:17:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.514+0000 7f280d000640 1 -- 172.21.15.27:0/2984715835 wait complete. 2024-03-28T19:17:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.516+0000 7f280d000640 1 Processor -- start 2024-03-28T19:17:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.516+0000 7f280d000640 1 -- start start 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f280d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28080c3690 0x7f280806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f280d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808150c70 0x7f280806cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f280d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28081534c0 0x7f280814c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f280d000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080abcd0 con 0x7f2808150c70 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f280d000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080ab9f0 con 0x7f28080c3690 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f280d000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28080c1810 con 0x7f28081534c0 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f27fd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808150c70 0x7f280806cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f27fd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808150c70 0x7f280806cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35546/0 (socket says 172.21.15.27:35546) 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f27fd400640 1 -- 172.21.15.27:0/3933573284 learned_addr learned my addr 172.21.15.27:0/3933573284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f2805e00640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28080c3690 0x7f280806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.517+0000 7f2806800640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28081534c0 0x7f280814c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fd400640 1 -- 172.21.15.27:0/3933573284 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28081534c0 msgr2=0x7f280814c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fd400640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28081534c0 0x7f280814c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fd400640 1 -- 172.21.15.27:0/3933573284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28080c3690 msgr2=0x7f280806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fd400640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28080c3690 0x7f280806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fd400640 1 -- 172.21.15.27:0/3933573284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280814bd70 con 0x7f2808150c70 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fd400640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808150c70 0x7f280806cd40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f27f000b420 tx=0x7f27f000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fea00640 1 -- 172.21.15.27:0/3933573284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0011830 con 0x7f2808150c70 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fea00640 1 -- 172.21.15.27:0/3933573284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f0016070 con 0x7f2808150c70 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f27fea00640 1 -- 172.21.15.27:0/3933573284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0011c10 con 0x7f2808150c70 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.518+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28080c09d0 con 0x7f2808150c70 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.519+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28081ebaa0 con 0x7f2808150c70 2024-03-28T19:17:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.520+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280804f400 con 0x7f2808150c70 2024-03-28T19:17:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.521+0000 7f27fea00640 1 -- 172.21.15.27:0/3933573284 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f27f001b460 con 0x7f2808150c70 2024-03-28T19:17:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.522+0000 7f27fea00640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f27e406bbd0 0x7f27e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.522+0000 7f27fea00640 1 -- 172.21.15.27:0/3933573284 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f27f008b270 con 0x7f2808150c70 2024-03-28T19:17:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.523+0000 7f2805e00640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f27e406bbd0 0x7f27e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.523+0000 7f2805e00640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f27e406bbd0 0x7f27e406e090 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f27f8006fd0 tx=0x7f27f8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:03.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.527+0000 7f27fea00640 1 -- 172.21.15.27:0/3933573284 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f27f0020020 con 0x7f2808150c70 2024-03-28T19:17:03.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.737+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf580c5f00 con 0x7fbf58149640 2024-03-28T19:17:03.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.738+0000 7fbf4e000640 1 -- 172.21.15.27:0/591472207 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7fbf50058ed0 con 0x7fbf58149640 2024-03-28T19:17:03.738 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:03.738 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:17:03.738 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:17:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbf3806bc00 msgr2=0x7fbf3806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbf3806bc00 0x7fbf3806e0c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbf48036000 tx=0x7fbf48030000 comp rx=0 tx=0).stop 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf58149640 msgr2=0x7fbf581ec4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf58149640 0x7fbf581ec4a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fbf581538a0 tx=0x7fbf5000b4a0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 shutdown_connections 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbf3806bc00 0x7fbf3806e0c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf58149640 0x7fbf581ec4a0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf58072500 0x7fbf58149100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 --2- 172.21.15.27:0/591472207 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbf3c003a00 0x7fbf580c2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.741+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 >> 172.21.15.27:0/591472207 conn(0x7fbf5806b9d0 msgr2=0x7fbf580c47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:03.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.742+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 shutdown_connections 2024-03-28T19:17:03.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.742+0000 7fbf5ca00640 1 -- 172.21.15.27:0/591472207 wait complete. 2024-03-28T19:17:03.760 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:17:03.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.774+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28080c5f00 con 0x7f2808150c70 2024-03-28T19:17:03.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.775+0000 7f27fea00640 1 -- 172.21.15.27:0/3933573284 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f27f0059040 con 0x7f2808150c70 2024-03-28T19:17:03.774 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:03.774 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:17:03.775 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:17:03.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f27e406bbd0 msgr2=0x7f27e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f27e406bbd0 0x7f27e406e090 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f27f8006fd0 tx=0x7f27f8008040 comp rx=0 tx=0).stop 2024-03-28T19:17:03.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808150c70 msgr2=0x7f280806cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:03.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808150c70 0x7f280806cd40 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f27f000b420 tx=0x7f27f000b8e0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 shutdown_connections 2024-03-28T19:17:03.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f27e406bbd0 0x7f27e406e090 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28081534c0 0x7f280814c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808150c70 0x7f280806cd40 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 --2- 172.21.15.27:0/3933573284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28080c3690 0x7f280806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:03.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 >> 172.21.15.27:0/3933573284 conn(0x7f280806b9d0 msgr2=0x7f2808152950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:03.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 shutdown_connections 2024-03-28T19:17:03.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:03.778+0000 7f280d000640 1 -- 172.21.15.27:0/3933573284 wait complete. 2024-03-28T19:17:03.796 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:17:05.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:05.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:05.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.988+0000 7f66fd400640 1 Processor -- start 2024-03-28T19:17:05.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.988+0000 7f66fd400640 1 -- start start 2024-03-28T19:17:05.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.989+0000 7f66fd400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f66f80c5e80 0x7f66f80c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:05.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.990+0000 7f66fd400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66f80abf60 0x7f66f80aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:05.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.990+0000 7f66fd400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f806d8a0 con 0x7f66f80c1f50 2024-03-28T19:17:05.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.990+0000 7f66fd400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f806d650 con 0x7f66f80c5e80 2024-03-28T19:17:05.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.990+0000 7f66fd400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f806e1d0 con 0x7f66f80abf60 2024-03-28T19:17:05.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.990+0000 7f66ede00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f66f80c1f50 0x7f66f80ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38098/0 (socket says 172.21.15.27:38098) 2024-03-28T19:17:05.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.990+0000 7f66ede00640 1 -- 172.21.15.27:0/672762535 learned_addr learned my addr 172.21.15.27:0/672762535 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:05.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.990+0000 7f66f6800640 1 --2- 172.21.15.27:0/672762535 >> v2:172.21.15.155:3300/0 conn(0x7f66f80c5e80 0x7f66f80c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:05.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.990+0000 7f66f7200640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66f80abf60 0x7f66f80aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:05.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641492779 0 0) 0x7f66f806d8a0 con 0x7f66f80c1f50 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66dc005730 con 0x7f66f80c1f50 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 448049653 0 0) 0x7f66dc005730 con 0x7f66f80c1f50 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66f80abf60 msgr2=0x7f66f80aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66f80abf60 0x7f66f80aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 >> v2:172.21.15.155:3300/0 conn(0x7f66f80c5e80 msgr2=0x7f66f80c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 --2- 172.21.15.27:0/672762535 >> v2:172.21.15.155:3300/0 conn(0x7f66f80c5e80 0x7f66f80c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f806db80 con 0x7f66f80c1f50 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f66e0001670 con 0x7f66f80c1f50 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 >> v1:172.21.15.27:6789/0 conn(0x7f66f80c1f50 legacy=0x7f66f80ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc005bc0 0x7f66dc005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66f80abf60 0x7f66dc006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66dc006f50 0x7f66dc0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f806d650 con 0x7f66dc005bc0 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f806e1d0 con 0x7f66dc006f50 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc00dc50 con 0x7f66f80abf60 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.991+0000 7f66f6800640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc005bc0 0x7f66dc005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.992+0000 7f66f6800640 1 -- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66f80abf60 msgr2=0x7f66dc006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.992+0000 7f66f7200640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66dc006f50 0x7f66dc0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.992+0000 7f66f6800640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66f80abf60 0x7f66dc006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.992+0000 7f66f6800640 1 -- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66dc006f50 msgr2=0x7f66dc0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:05.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.992+0000 7f66f6800640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66dc006f50 0x7f66dc0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:05.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.992+0000 7f66f6800640 1 -- 172.21.15.27:0/672762535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc005730 con 0x7f66dc005bc0 2024-03-28T19:17:05.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.992+0000 7f66f7200640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66dc006f50 0x7f66dc0073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:17:05.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.993+0000 7f66f6800640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc005bc0 0x7f66dc005fe0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009ac0 tx=0x7f66dc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2280ee2bf325c1c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:05.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.993+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8039070 con 0x7f66dc005bc0 2024-03-28T19:17:05.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.993+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e802c710 con 0x7f66dc005bc0 2024-03-28T19:17:05.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.993+0000 7f66f5e00640 1 -- 172.21.15.27:0/672762535 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e8009690 con 0x7f66dc005bc0 2024-03-28T19:17:05.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.994+0000 7f66fd400640 1 -- 172.21.15.27:0/672762535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc005bc0 msgr2=0x7f66dc005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.994+0000 7f66fd400640 1 --2- 172.21.15.27:0/672762535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66dc005bc0 0x7f66dc005fe0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009ac0 tx=0x7f66dc007a40 comp rx=0 tx=0).stop 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.994+0000 7f66f7200640 1 -- 172.21.15.27:0/672762535 reap_dead start 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.994+0000 7f66fd400640 1 -- 172.21.15.27:0/672762535 shutdown_connections 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.994+0000 7f66fd400640 1 -- 172.21.15.27:0/672762535 >> 172.21.15.27:0/672762535 conn(0x7f66f806b9d0 msgr2=0x7f66f80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.994+0000 7f66fd400640 1 -- 172.21.15.27:0/672762535 shutdown_connections 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.994+0000 7f66fd400640 1 -- 172.21.15.27:0/672762535 wait complete. 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.996+0000 7f66fd400640 1 Processor -- start 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.996+0000 7f66fd400640 1 -- start start 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66fd400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66dc005bc0 0x7f66f806ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66fd400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66f81db850 0x7f66f81ec3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:05.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66fd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66f81ec930 0x7f66f81ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66fd400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f80c3370 con 0x7f66f81ec930 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66fd400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f80c31f0 con 0x7f66f81db850 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66fd400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f80c6260 con 0x7f66dc005bc0 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66f7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66f81ec930 0x7f66f81ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66f7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66f81ec930 0x7f66f81ee130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35572/0 (socket says 172.21.15.27:35572) 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.997+0000 7f66f7200640 1 -- 172.21.15.27:0/1568060496 learned_addr learned my addr 172.21.15.27:0/1568060496 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66f6800640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66dc005bc0 0x7f66f806ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66ede00640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66f81db850 0x7f66f81ec3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:06.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66f7200640 1 -- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66dc005bc0 msgr2=0x7f66f806ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66f7200640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66dc005bc0 0x7f66f806ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66f7200640 1 -- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66f81db850 msgr2=0x7f66f81ec3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66f7200640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66f81db850 0x7f66f81ec3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66f7200640 1 -- 172.21.15.27:0/1568060496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f80a9440 con 0x7f66f81ec930 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66f6800640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66dc005bc0 0x7f66f806ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66f7200640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66f81ec930 0x7f66f81ee130 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f66f80709e0 tx=0x7f66f000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66ef400640 1 -- 172.21.15.27:0/1568060496 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f001c070 con 0x7f66f81ec930 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66ef400640 1 -- 172.21.15.27:0/1568060496 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f000fe90 con 0x7f66f81ec930 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f80a8420 con 0x7f66f81ec930 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.998+0000 7f66ef400640 1 -- 172.21.15.27:0/1568060496 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0017620 con 0x7f66f81ec930 2024-03-28T19:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:05.999+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f80a89e0 con 0x7f66f81ec930 2024-03-28T19:17:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.000+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f804f400 con 0x7f66f81ec930 2024-03-28T19:17:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.001+0000 7f66ef400640 1 -- 172.21.15.27:0/1568060496 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f66f0005d10 con 0x7f66f81ec930 2024-03-28T19:17:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.002+0000 7f66ef400640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f66d406bc00 0x7f66d406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.002+0000 7f66ef400640 1 -- 172.21.15.27:0/1568060496 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f66f008b970 con 0x7f66f81ec930 2024-03-28T19:17:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.002+0000 7f66f6800640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f66d406bc00 0x7f66d406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:06.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.003+0000 7f66f6800640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f66d406bc00 0x7f66d406e0c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f66e8036000 tx=0x7f66e8030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:06.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.007+0000 7f66ef400640 1 -- 172.21.15.27:0/1568060496 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f66f0012070 con 0x7f66f81ec930 2024-03-28T19:17:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.014+0000 7f0cd8200640 1 Processor -- start 2024-03-28T19:17:06.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.014+0000 7f0cd8200640 1 -- start start 2024-03-28T19:17:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.015+0000 7f0cd8200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cd00abf60 0x7f0cd00aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.015+0000 7f0cd8200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd006d8a0 con 0x7f0cd00c1f50 2024-03-28T19:17:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.015+0000 7f0cd8200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd006db80 con 0x7f0cd00c5e80 2024-03-28T19:17:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.015+0000 7f0cd8200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd006d650 con 0x7f0cd00abf60 2024-03-28T19:17:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.015+0000 7f0cd6200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cd00abf60 0x7f0cd00aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.015+0000 7f0cd4e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0cd00c1f50 0x7f0cd00ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38102/0 (socket says 172.21.15.27:38102) 2024-03-28T19:17:06.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.015+0000 7f0cd4e00640 1 -- 172.21.15.27:0/2274121894 learned_addr learned my addr 172.21.15.27:0/2274121894 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.016+0000 7f0cd6200640 1 -- 172.21.15.27:0/2274121894 >> v1:172.21.15.155:6789/0 conn(0x7f0cd00c5e80 legacy=0x7f0cd00c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.016+0000 7f0cd6200640 1 -- 172.21.15.27:0/2274121894 >> v1:172.21.15.27:6789/0 conn(0x7f0cd00c1f50 legacy=0x7f0cd00ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.016+0000 7f0cd6200640 1 -- 172.21.15.27:0/2274121894 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd006de60 con 0x7f0cd00abf60 2024-03-28T19:17:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.016+0000 7f0ccfe00640 1 -- 172.21.15.27:0/2274121894 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517406147 0 0) 0x7f0cd006d8a0 con 0x7f0cd00c1f50 2024-03-28T19:17:06.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.018+0000 7f0cd6200640 1 --2- 172.21.15.27:0/2274121894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd00aa570 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009d90 tx=0x7f0cc802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c523aaa83caf5f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:06.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.018+0000 7f0ccfe00640 1 -- 172.21.15.27:0/2274121894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8039070 con 0x7f0cd00abf60 2024-03-28T19:17:06.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.018+0000 7f0ccfe00640 1 -- 172.21.15.27:0/2274121894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc802cbd0 con 0x7f0cd00abf60 2024-03-28T19:17:06.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.018+0000 7f0ccfe00640 1 -- 172.21.15.27:0/2274121894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8034440 con 0x7f0cd00abf60 2024-03-28T19:17:06.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.019+0000 7f0cd8200640 1 -- 172.21.15.27:0/2274121894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 msgr2=0x7f0cd00aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.019+0000 7f0cd8200640 1 --2- 172.21.15.27:0/2274121894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd00aa570 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009d90 tx=0x7f0cc802be20 comp rx=0 tx=0).stop 2024-03-28T19:17:06.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.019+0000 7f0cd8200640 1 -- 172.21.15.27:0/2274121894 shutdown_connections 2024-03-28T19:17:06.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.019+0000 7f0cd8200640 1 --2- 172.21.15.27:0/2274121894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd00aa570 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.019+0000 7f0cd8200640 1 -- 172.21.15.27:0/2274121894 >> 172.21.15.27:0/2274121894 conn(0x7f0cd006b9d0 msgr2=0x7f0cd00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:06.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.019+0000 7f0cd8200640 1 -- 172.21.15.27:0/2274121894 shutdown_connections 2024-03-28T19:17:06.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.019+0000 7f0cd8200640 1 -- 172.21.15.27:0/2274121894 wait complete. 2024-03-28T19:17:06.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.021+0000 7f0cd8200640 1 Processor -- start 2024-03-28T19:17:06.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.021+0000 7f0cd8200640 1 -- start start 2024-03-28T19:17:06.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd8200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd0152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd8200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cd00c1f50 0x7f0cd0144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd8200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cd00c5e80 0x7f0cd0145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd8200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00aaab0 con 0x7f0cd00abf60 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd8200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd006e1d0 con 0x7f0cd00c5e80 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd8200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00aac30 con 0x7f0cd00c1f50 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd0152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd0152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35586/0 (socket says 172.21.15.27:35586) 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd5800640 1 -- 172.21.15.27:0/295089910 learned_addr learned my addr 172.21.15.27:0/295089910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:06.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd5800640 1 -- 172.21.15.27:0/295089910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cd00c1f50 msgr2=0x7f0cd0144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd5800640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cd00c1f50 0x7f0cd0144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd5800640 1 -- 172.21.15.27:0/295089910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cd00c5e80 msgr2=0x7f0cd0145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd5800640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cd00c5e80 0x7f0cd0145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.022+0000 7f0cd5800640 1 -- 172.21.15.27:0/295089910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd01548b0 con 0x7f0cd00abf60 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.023+0000 7f0cd5800640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd0152cc0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400c400 tx=0x7f0cc400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.023+0000 7f0ccd600640 1 -- 172.21.15.27:0/295089910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4017070 con 0x7f0cd00abf60 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.023+0000 7f0ccd600640 1 -- 172.21.15.27:0/295089910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc40052d0 con 0x7f0cd00abf60 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.023+0000 7f0ccd600640 1 -- 172.21.15.27:0/295089910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4003830 con 0x7f0cd00abf60 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.023+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd00a6040 con 0x7f0cd00abf60 2024-03-28T19:17:06.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.023+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd01ebf00 con 0x7f0cd00abf60 2024-03-28T19:17:06.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.024+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd004f400 con 0x7f0cd00abf60 2024-03-28T19:17:06.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.025+0000 7f0ccd600640 1 -- 172.21.15.27:0/295089910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0cc401a070 con 0x7f0cd00abf60 2024-03-28T19:17:06.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.027+0000 7f0ccd600640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cb006bbd0 0x7f0cb006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:06.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.027+0000 7f0ccd600640 1 -- 172.21.15.27:0/295089910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f0cc408abc0 con 0x7f0cd00abf60 2024-03-28T19:17:06.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.027+0000 7f0cd4e00640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cb006bbd0 0x7f0cb006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:06.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.028+0000 7f0cd4e00640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cb006bbd0 0x7f0cb006e090 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc007fd0 tx=0x7f0cbc007900 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:06.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.032+0000 7f0ccd600640 1 -- 172.21.15.27:0/295089910 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0cc4058900 con 0x7f0cd00abf60 2024-03-28T19:17:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.258+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66f806d3d0 con 0x7f66f81ec930 2024-03-28T19:17:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.259+0000 7f66ef400640 1 -- 172.21.15.27:0/1568060496 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f66f00596e0 con 0x7f66f81ec930 2024-03-28T19:17:06.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:06.259 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:17:06.259 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:17:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f66d406bc00 msgr2=0x7f66d406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f66d406bc00 0x7f66d406e0c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f66e8036000 tx=0x7f66e8030000 comp rx=0 tx=0).stop 2024-03-28T19:17:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66f81ec930 msgr2=0x7f66f81ee130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66f81ec930 0x7f66f81ee130 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f66f80709e0 tx=0x7f66f000f4a0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 shutdown_connections 2024-03-28T19:17:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f66d406bc00 0x7f66d406e0c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66f81ec930 0x7f66f81ee130 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66f81db850 0x7f66f81ec3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 --2- 172.21.15.27:0/1568060496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66dc005bc0 0x7f66f806ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 >> 172.21.15.27:0/1568060496 conn(0x7f66f806b9d0 msgr2=0x7f66f80c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 shutdown_connections 2024-03-28T19:17:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.262+0000 7f66fd400640 1 -- 172.21.15.27:0/1568060496 wait complete. 2024-03-28T19:17:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.275+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cd0145780 con 0x7f0cd00abf60 2024-03-28T19:17:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.276+0000 7f0ccd600640 1 -- 172.21.15.27:0/295089910 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f0cc4058200 con 0x7f0cd00abf60 2024-03-28T19:17:06.276 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:06.276 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:17:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cb006bbd0 msgr2=0x7f0cb006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cb006bbd0 0x7f0cb006e090 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc007fd0 tx=0x7f0cbc007900 comp rx=0 tx=0).stop 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 msgr2=0x7f0cd0152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd0152cc0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400c400 tx=0x7f0cc400c8c0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 shutdown_connections 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0cb006bbd0 0x7f0cb006e090 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0cd00c5e80 0x7f0cd0145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0cd00c1f50 0x7f0cd0144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 --2- 172.21.15.27:0/295089910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd00abf60 0x7f0cd0152cc0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 >> 172.21.15.27:0/295089910 conn(0x7f0cd006b9d0 msgr2=0x7f0cd00c47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 shutdown_connections 2024-03-28T19:17:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:06.279+0000 7f0cd8200640 1 -- 172.21.15.27:0/295089910 wait complete. 2024-03-28T19:17:06.281 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:17:06.297 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:17:06.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:17:06.345 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:17:08.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:08.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:08.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.514+0000 7f3bab200640 1 Processor -- start 2024-03-28T19:17:08.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.515+0000 7f3bab200640 1 -- start start 2024-03-28T19:17:08.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.517+0000 7f3bab200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3ba40c8470 0x7f3ba40ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.517+0000 7f3bab200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 0x7f3ba40c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.517+0000 7f3bab200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba40ad680 con 0x7f3ba4150b60 2024-03-28T19:17:08.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.517+0000 7f3bab200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba4157210 con 0x7f3ba40c8470 2024-03-28T19:17:08.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.517+0000 7f3bab200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba40ad9b0 con 0x7f3ba40c8c70 2024-03-28T19:17:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.517+0000 7f3ba3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3ba4150b60 0x7f3ba4156b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57106/0 (socket says 172.21.15.27:57106) 2024-03-28T19:17:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.517+0000 7f3ba3e00640 1 -- 172.21.15.27:0/1303525224 learned_addr learned my addr 172.21.15.27:0/1303525224 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba3400640 1 --2- 172.21.15.27:0/1303525224 >> v2:172.21.15.155:3300/0 conn(0x7f3ba40c8470 0x7f3ba40ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba9200640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 0x7f3ba40c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820831428 0 0) 0x7f3ba40ad680 con 0x7f3ba4150b60 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b88003520 con 0x7f3ba4150b60 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2073368300 0 0) 0x7f3b88003520 con 0x7f3ba4150b60 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 msgr2=0x7f3ba40c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 0x7f3ba40c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 >> v2:172.21.15.155:3300/0 conn(0x7f3ba40c8470 msgr2=0x7f3ba40ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 --2- 172.21.15.27:0/1303525224 >> v2:172.21.15.155:3300/0 conn(0x7f3ba40c8470 0x7f3ba40ade40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba40c4de0 con 0x7f3ba4150b60 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f3b94001670 con 0x7f3ba4150b60 2024-03-28T19:17:08.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.518+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 >> v1:172.21.15.27:6789/0 conn(0x7f3ba4150b60 legacy=0x7f3ba4156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba2a00640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b880039b0 0x7f3b88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba2a00640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b88004800 0x7f3b88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba2a00640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 0x7f3b880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba4157210 con 0x7f3b880039b0 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba40ad9b0 con 0x7f3b88004800 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8800ba40 con 0x7f3ba40c8c70 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba3e00640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b880039b0 0x7f3b88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba3400640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b88004800 0x7f3b88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba9200640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 0x7f3b880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba3e00640 1 -- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 msgr2=0x7f3b880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba3e00640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 0x7f3b880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba3e00640 1 -- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b88004800 msgr2=0x7f3b88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba3e00640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b88004800 0x7f3b88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.519+0000 7f3ba3e00640 1 -- 172.21.15.27:0/1303525224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b88003520 con 0x7f3b880039b0 2024-03-28T19:17:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.520+0000 7f3ba9200640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba40c8c70 0x7f3b880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.521+0000 7f3ba3e00640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b880039b0 0x7f3b88003dd0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f3b9400c110 tx=0x7f3b88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b5ef0d1feb7b26 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.522+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94007c40 con 0x7f3b880039b0 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.522+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b94030070 con 0x7f3b880039b0 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.522+0000 7f3ba2a00640 1 -- 172.21.15.27:0/1303525224 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94005d30 con 0x7f3b880039b0 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.522+0000 7f3bab200640 1 -- 172.21.15.27:0/1303525224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b880039b0 msgr2=0x7f3b88003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.522+0000 7f3bab200640 1 --2- 172.21.15.27:0/1303525224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b880039b0 0x7f3b88003dd0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f3b9400c110 tx=0x7f3b88005830 comp rx=0 tx=0).stop 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.522+0000 7f3ba9200640 1 -- 172.21.15.27:0/1303525224 reap_dead start 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.522+0000 7f3bab200640 1 -- 172.21.15.27:0/1303525224 shutdown_connections 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.522+0000 7f3bab200640 1 -- 172.21.15.27:0/1303525224 >> 172.21.15.27:0/1303525224 conn(0x7f3ba40aaab0 msgr2=0x7f3ba40aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.523+0000 7f3bab200640 1 -- 172.21.15.27:0/1303525224 shutdown_connections 2024-03-28T19:17:08.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.523+0000 7f3bab200640 1 -- 172.21.15.27:0/1303525224 wait complete. 2024-03-28T19:17:08.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.524+0000 7f3bab200640 1 Processor -- start 2024-03-28T19:17:08.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.525+0000 7f3bab200640 1 -- start start 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.525+0000 7f3bab200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba4155bc0 0x7f3ba414d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.525+0000 7f3bab200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ba41e06c0 0x7f3ba414d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.525+0000 7f3bab200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ba41dcb30 0x7f3ba41e91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.525+0000 7f3bab200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba4150f40 con 0x7f3ba41dcb30 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.525+0000 7f3bab200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba40c53a0 con 0x7f3ba41e06c0 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.525+0000 7f3bab200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba40a85a0 con 0x7f3ba4155bc0 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ba41dcb30 0x7f3ba41e91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ba41dcb30 0x7f3ba41e91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36500/0 (socket says 172.21.15.27:36500) 2024-03-28T19:17:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba9200640 1 -- 172.21.15.27:0/149088980 learned_addr learned my addr 172.21.15.27:0/149088980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:08.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba3400640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ba41e06c0 0x7f3ba414d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba3e00640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba4155bc0 0x7f3ba414d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba9200640 1 -- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba4155bc0 msgr2=0x7f3ba414d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba9200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba4155bc0 0x7f3ba414d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba9200640 1 -- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ba41e06c0 msgr2=0x7f3ba414d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba9200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ba41e06c0 0x7f3ba414d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba9200640 1 -- 172.21.15.27:0/149088980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba41ed120 con 0x7f3ba41dcb30 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.526+0000 7f3ba3400640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ba41e06c0 0x7f3ba414d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.527+0000 7f3ba9200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ba41dcb30 0x7f3ba41e91f0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4067c90 tx=0x7f3b9c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.527+0000 7f3b9be00640 1 -- 172.21.15.27:0/149088980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c01c070 con 0x7f3ba41dcb30 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.527+0000 7f3b9be00640 1 -- 172.21.15.27:0/149088980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c017400 con 0x7f3ba41dcb30 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.527+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba41dda90 con 0x7f3ba41dcb30 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.527+0000 7f3b9be00640 1 -- 172.21.15.27:0/149088980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c0177d0 con 0x7f3ba41dcb30 2024-03-28T19:17:08.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.527+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba4154270 con 0x7f3ba41dcb30 2024-03-28T19:17:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.528+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba404f400 con 0x7f3ba41dcb30 2024-03-28T19:17:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.530+0000 7f3b9be00640 1 -- 172.21.15.27:0/149088980 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3b9c005d10 con 0x7f3ba41dcb30 2024-03-28T19:17:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.531+0000 7f3b9be00640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b8006bbd0 0x7f3b8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.532+0000 7f3b9be00640 1 -- 172.21.15.27:0/149088980 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f3b9c08c0e0 con 0x7f3ba41dcb30 2024-03-28T19:17:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.532+0000 7f3ba3e00640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b8006bbd0 0x7f3b8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.532+0000 7f3ba3e00640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b8006bbd0 0x7f3b8006e090 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3b9400c110 tx=0x7f3b94039040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:08.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.534+0000 7f3b9be00640 1 -- 172.21.15.27:0/149088980 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3b9c012070 con 0x7f3ba41dcb30 2024-03-28T19:17:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.549+0000 7fa781e00640 1 Processor -- start 2024-03-28T19:17:08.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.549+0000 7fa781e00640 1 -- start start 2024-03-28T19:17:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.551+0000 7fa781e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa77c0c58c0 0x7fa77c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.551+0000 7fa781e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa77c0be5d0 0x7fa77c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.551+0000 7fa781e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c156230 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.551+0000 7fa781e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c158bd0 con 0x7fa77c0be5d0 2024-03-28T19:17:08.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.551+0000 7fa781e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c158eb0 con 0x7fa77c0c58c0 2024-03-28T19:17:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.552+0000 7fa781400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa77c0be5d0 0x7fa77c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.552+0000 7fa781400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa77c0be5d0 0x7fa77c155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36512/0 (socket says 172.21.15.27:36512) 2024-03-28T19:17:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.552+0000 7fa781400640 1 -- 172.21.15.27:0/628787774 learned_addr learned my addr 172.21.15.27:0/628787774 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.552+0000 7fa780a00640 1 --2- 172.21.15.27:0/628787774 >> v2:172.21.15.155:3300/0 conn(0x7fa77c0c58c0 0x7fa77c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.552+0000 7fa781400640 1 -- 172.21.15.27:0/628787774 >> v2:172.21.15.155:3300/0 conn(0x7fa77c0c58c0 msgr2=0x7fa77c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.552+0000 7fa781400640 1 --2- 172.21.15.27:0/628787774 >> v2:172.21.15.155:3300/0 conn(0x7fa77c0c58c0 0x7fa77c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.552+0000 7fa781400640 1 -- 172.21.15.27:0/628787774 >> v1:172.21.15.155:6789/0 conn(0x7fa77c0c8ed0 legacy=0x7fa77c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.552+0000 7fa781400640 1 -- 172.21.15.27:0/628787774 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c159190 con 0x7fa77c0be5d0 2024-03-28T19:17:08.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.553+0000 7fa781400640 1 --2- 172.21.15.27:0/628787774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0be5d0 0x7fa77c155cf0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa77c06c060 tx=0x7fa77402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cea8e0562b32b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.554+0000 7fa77b400640 1 -- 172.21.15.27:0/628787774 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77402c960 con 0x7fa77c0be5d0 2024-03-28T19:17:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.554+0000 7fa77b400640 1 -- 172.21.15.27:0/628787774 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77402cb00 con 0x7fa77c0be5d0 2024-03-28T19:17:08.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.554+0000 7fa77b400640 1 -- 172.21.15.27:0/628787774 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7740343f0 con 0x7fa77c0be5d0 2024-03-28T19:17:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.555+0000 7fa781e00640 1 -- 172.21.15.27:0/628787774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0be5d0 msgr2=0x7fa77c155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.555+0000 7fa781e00640 1 --2- 172.21.15.27:0/628787774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0be5d0 0x7fa77c155cf0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa77c06c060 tx=0x7fa77402bdc0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.555+0000 7fa781e00640 1 -- 172.21.15.27:0/628787774 shutdown_connections 2024-03-28T19:17:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.555+0000 7fa781e00640 1 --2- 172.21.15.27:0/628787774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0be5d0 0x7fa77c155cf0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.555+0000 7fa781e00640 1 --2- 172.21.15.27:0/628787774 >> v2:172.21.15.155:3300/0 conn(0x7fa77c0c58c0 0x7fa77c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.555+0000 7fa781e00640 1 -- 172.21.15.27:0/628787774 >> 172.21.15.27:0/628787774 conn(0x7fa77c06b6f0 msgr2=0x7fa77c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.555+0000 7fa781e00640 1 -- 172.21.15.27:0/628787774 shutdown_connections 2024-03-28T19:17:08.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.555+0000 7fa781e00640 1 -- 172.21.15.27:0/628787774 wait complete. 2024-03-28T19:17:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.556+0000 7fa781e00640 1 Processor -- start 2024-03-28T19:17:08.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.557+0000 7fa781e00640 1 -- start start 2024-03-28T19:17:08.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.557+0000 7fa781e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa77c0be5d0 0x7fa77c0c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.557+0000 7fa781e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa77c0c58c0 0x7fa77c1ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.557+0000 7fa781e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0c8ed0 0x7fa77c1ec080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.557+0000 7fa781e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0b0820 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.557+0000 7fa781e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c159750 con 0x7fa77c0be5d0 2024-03-28T19:17:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.557+0000 7fa781e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0b09a0 con 0x7fa77c0c58c0 2024-03-28T19:17:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa781400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0c8ed0 0x7fa77c1ec080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa781400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0c8ed0 0x7fa77c1ec080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36526/0 (socket says 172.21.15.27:36526) 2024-03-28T19:17:08.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa781400640 1 -- 172.21.15.27:0/20432278 learned_addr learned my addr 172.21.15.27:0/20432278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa780a00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa77c0be5d0 0x7fa77c0c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa77be00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa77c0c58c0 0x7fa77c1ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa781400640 1 -- 172.21.15.27:0/20432278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa77c0c58c0 msgr2=0x7fa77c1ebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa781400640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa77c0c58c0 0x7fa77c1ebb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa781400640 1 -- 172.21.15.27:0/20432278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa77c0be5d0 msgr2=0x7fa77c0c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa781400640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa77c0be5d0 0x7fa77c0c2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.558+0000 7fa781400640 1 -- 172.21.15.27:0/20432278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c1ef3d0 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.559+0000 7fa781400640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0c8ed0 0x7fa77c1ec080 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fa774036040 tx=0x7fa774004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.559+0000 7fa778c00640 1 -- 172.21.15.27:0/20432278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774043020 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.559+0000 7fa778c00640 1 -- 172.21.15.27:0/20432278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77403e450 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.559+0000 7fa778c00640 1 -- 172.21.15.27:0/20432278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774009510 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.559+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c0aad20 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.559+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c0c27c0 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.561+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c069640 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.562+0000 7fa778c00640 1 -- 172.21.15.27:0/20432278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa774046020 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.562+0000 7fa778c00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa75c06bbd0 0x7fa75c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:08.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.563+0000 7fa778c00640 1 -- 172.21.15.27:0/20432278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 9131+0+0 (secure 0 0 0) 0x7fa7740ae060 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.563+0000 7fa780a00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa75c06bbd0 0x7fa75c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:08.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.564+0000 7fa780a00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa75c06bbd0 0x7fa75c06e090 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa770006fd0 tx=0x7fa770008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:08.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.567+0000 7fa778c00640 1 -- 172.21.15.27:0/20432278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa77404d030 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.784+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ba40c4a20 con 0x7f3ba41dcb30 2024-03-28T19:17:08.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.785+0000 7f3b9be00640 1 -- 172.21.15.27:0/149088980 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7f3b9c059ed0 con 0x7f3ba41dcb30 2024-03-28T19:17:08.785 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:08.785 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:17:08.785 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.787+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b8006bbd0 msgr2=0x7f3b8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.787+0000 7f3bab200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b8006bbd0 0x7f3b8006e090 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3b9400c110 tx=0x7f3b94039040 comp rx=0 tx=0).stop 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ba41dcb30 msgr2=0x7f3ba41e91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ba41dcb30 0x7f3ba41e91f0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4067c90 tx=0x7f3b9c00f4a0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 shutdown_connections 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3b8006bbd0 0x7f3b8006e090 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ba41dcb30 0x7f3ba41e91f0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ba41e06c0 0x7f3ba414d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 --2- 172.21.15.27:0/149088980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ba4155bc0 0x7f3ba414d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 >> 172.21.15.27:0/149088980 conn(0x7f3ba40aaab0 msgr2=0x7f3ba40aa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 shutdown_connections 2024-03-28T19:17:08.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.788+0000 7f3bab200640 1 -- 172.21.15.27:0/149088980 wait complete. 2024-03-28T19:17:08.805 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:17:08.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.807+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa77c1eb480 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.808+0000 7fa778c00640 1 -- 172.21.15.27:0/20432278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6259 (secure 0 0 0) 0x7fa77c1ef3d0 con 0x7fa77c0c8ed0 2024-03-28T19:17:08.808 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:08.808 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6014,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6188,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:16:16.903807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7575":{"gid":7575,"name":"a","rank":0,"incarnation":166,"state":"up:active","state_seq":6,"addr":"172.21.15.27:6835/148837773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":148837773},{"type":"v1","addr":"172.21.15.27:6835","nonce":148837773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7575,"qdb_cluster":[7575]},"id":1},{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:15:56.760391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"d","rank":0,"incarnation":160,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6834/2865796032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2865796032},{"type":"v1","addr":"172.21.15.155:6834","nonce":2865796032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6077,"qdb_cluster":[6077]},"id":2}]} 2024-03-28T19:17:08.809 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 171 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.810+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa75c06bbd0 msgr2=0x7fa75c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.810+0000 7fa781e00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa75c06bbd0 0x7fa75c06e090 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa770006fd0 tx=0x7fa770008040 comp rx=0 tx=0).stop 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.810+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0c8ed0 msgr2=0x7fa77c1ec080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.810+0000 7fa781e00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0c8ed0 0x7fa77c1ec080 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fa774036040 tx=0x7fa774004a40 comp rx=0 tx=0).stop 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.811+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 shutdown_connections 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.811+0000 7fa781e00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa75c06bbd0 0x7fa75c06e090 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.811+0000 7fa781e00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa77c0c8ed0 0x7fa77c1ec080 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.811+0000 7fa781e00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa77c0c58c0 0x7fa77c1ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.811+0000 7fa781e00640 1 --2- 172.21.15.27:0/20432278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa77c0be5d0 0x7fa77c0c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.811+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 >> 172.21.15.27:0/20432278 conn(0x7fa77c06b6f0 msgr2=0x7fa77c072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.811+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 shutdown_connections 2024-03-28T19:17:08.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:08.811+0000 7fa781e00640 1 -- 172.21.15.27:0/20432278 wait complete. 2024-03-28T19:17:08.827 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:17:10.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:10.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.018+0000 7f9d77000640 1 Processor -- start 2024-03-28T19:17:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.018+0000 7f9d77000640 1 -- start start 2024-03-28T19:17:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.019+0000 7f9d77000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d70150c70 0x7f9d700a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.019+0000 7f9d77000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d700c3af0 con 0x7f9d700c3690 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.019+0000 7f9d77000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d700c3d90 con 0x7f9d701534c0 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.019+0000 7f9d77000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d700ab4e0 con 0x7f9d70150c70 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.020+0000 7f9d67e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9d700c3690 0x7f9d700c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57116/0 (socket says 172.21.15.27:57116) 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.020+0000 7f9d67e00640 1 -- 172.21.15.27:0/1023787914 learned_addr learned my addr 172.21.15.27:0/1023787914 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.020+0000 7f9d6fe00640 1 --2- 172.21.15.27:0/1023787914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d70150c70 0x7f9d700a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.020+0000 7f9d6f400640 1 -- 172.21.15.27:0/1023787914 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366539930 0 0) 0x7f9d700c3af0 con 0x7f9d700c3690 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.020+0000 7f9d6f400640 1 -- 172.21.15.27:0/1023787914 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d54003520 con 0x7f9d700c3690 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.020+0000 7f9d6fe00640 1 -- 172.21.15.27:0/1023787914 >> v1:172.21.15.155:6789/0 conn(0x7f9d701534c0 legacy=0x7f9d70153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.020+0000 7f9d6fe00640 1 -- 172.21.15.27:0/1023787914 >> v1:172.21.15.27:6789/0 conn(0x7f9d700c3690 legacy=0x7f9d700c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.020+0000 7f9d6fe00640 1 -- 172.21.15.27:0/1023787914 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d700ab710 con 0x7f9d70150c70 2024-03-28T19:17:11.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.022+0000 7f9d6fe00640 1 --2- 172.21.15.27:0/1023787914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d70150c70 0x7f9d700a9cb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9d600097d0 tx=0x7f9d6002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a37efa7e3ce59751 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.022+0000 7f9d6f400640 1 -- 172.21.15.27:0/1023787914 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60035020 con 0x7f9d70150c70 2024-03-28T19:17:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.022+0000 7f9d6f400640 1 -- 172.21.15.27:0/1023787914 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d60031800 con 0x7f9d70150c70 2024-03-28T19:17:11.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.022+0000 7f9d6f400640 1 -- 172.21.15.27:0/1023787914 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60031bb0 con 0x7f9d70150c70 2024-03-28T19:17:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.023+0000 7f9d77000640 1 -- 172.21.15.27:0/1023787914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d70150c70 msgr2=0x7f9d700a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.023+0000 7f9d77000640 1 --2- 172.21.15.27:0/1023787914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d70150c70 0x7f9d700a9cb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9d600097d0 tx=0x7f9d6002b8a0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.023+0000 7f9d77000640 1 -- 172.21.15.27:0/1023787914 shutdown_connections 2024-03-28T19:17:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.023+0000 7f9d77000640 1 --2- 172.21.15.27:0/1023787914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d70150c70 0x7f9d700a9cb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.023+0000 7f9d77000640 1 -- 172.21.15.27:0/1023787914 >> 172.21.15.27:0/1023787914 conn(0x7f9d7006b9d0 msgr2=0x7f9d700c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.023+0000 7f9d77000640 1 -- 172.21.15.27:0/1023787914 shutdown_connections 2024-03-28T19:17:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.024+0000 7f9d77000640 1 -- 172.21.15.27:0/1023787914 wait complete. 2024-03-28T19:17:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.025+0000 7f9d77000640 1 Processor -- start 2024-03-28T19:17:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.025+0000 7f9d77000640 1 -- start start 2024-03-28T19:17:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.026+0000 7f9d77000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d700c3690 0x7f9d701ec8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.026+0000 7f9d77000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d70150c70 0x7f9d700a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.026+0000 7f9d77000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d701534c0 0x7f9d7014bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.026+0000 7f9d77000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d70154060 con 0x7f9d70150c70 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.026+0000 7f9d77000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d700ac1f0 con 0x7f9d701534c0 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.026+0000 7f9d77000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d700abcd0 con 0x7f9d700c3690 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d67e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d70150c70 0x7f9d700a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d6fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d700c3690 0x7f9d701ec8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d67e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d70150c70 0x7f9d700a8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36538/0 (socket says 172.21.15.27:36538) 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d67e00640 1 -- 172.21.15.27:0/2279119571 learned_addr learned my addr 172.21.15.27:0/2279119571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d75000640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d701534c0 0x7f9d7014bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d67e00640 1 -- 172.21.15.27:0/2279119571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d700c3690 msgr2=0x7f9d701ec8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d67e00640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d700c3690 0x7f9d701ec8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d67e00640 1 -- 172.21.15.27:0/2279119571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d701534c0 msgr2=0x7f9d7014bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d67e00640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d701534c0 0x7f9d7014bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.027+0000 7f9d67e00640 1 -- 172.21.15.27:0/2279119571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d700a9270 con 0x7f9d70150c70 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.028+0000 7f9d67e00640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d70150c70 0x7f9d700a8440 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c500 tx=0x7f9d5800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.028+0000 7f9d6cc00640 1 -- 172.21.15.27:0/2279119571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58017070 con 0x7f9d70150c70 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.028+0000 7f9d6cc00640 1 -- 172.21.15.27:0/2279119571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d58005ce0 con 0x7f9d70150c70 2024-03-28T19:17:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.028+0000 7f9d6cc00640 1 -- 172.21.15.27:0/2279119571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58007330 con 0x7f9d70150c70 2024-03-28T19:17:11.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.028+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d70154ef0 con 0x7f9d70150c70 2024-03-28T19:17:11.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.028+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d701ed4d0 con 0x7f9d70150c70 2024-03-28T19:17:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.029+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d70151330 con 0x7f9d70150c70 2024-03-28T19:17:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.030+0000 7f9d6cc00640 1 -- 172.21.15.27:0/2279119571 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9d5801a070 con 0x7f9d70150c70 2024-03-28T19:17:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.031+0000 7f9d6cc00640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d4c06bbd0 0x7f9d4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.031+0000 7f9d6cc00640 1 -- 172.21.15.27:0/2279119571 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f9d5808ac70 con 0x7f9d70150c70 2024-03-28T19:17:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.032+0000 7f9d6fe00640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d4c06bbd0 0x7f9d4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.033+0000 7f9d6fe00640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d4c06bbd0 0x7f9d4c06e090 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9d60036000 tx=0x7f9d60008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.036+0000 7f9d6cc00640 1 -- 172.21.15.27:0/2279119571 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9d70151330 con 0x7f9d70150c70 2024-03-28T19:17:11.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.057+0000 7f74d4c00640 1 Processor -- start 2024-03-28T19:17:11.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.057+0000 7f74d4c00640 1 -- start start 2024-03-28T19:17:11.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.059+0000 7f74d4c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f74d0150ae0 0x7f74d00a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.059+0000 7f74d4c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74d0153a30 0x7f74d0153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.059+0000 7f74d4c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d0150ec0 con 0x7f74d00a61a0 2024-03-28T19:17:11.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.059+0000 7f74d4c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d01543c0 con 0x7f74d0153a30 2024-03-28T19:17:11.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.059+0000 7f74d4c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d006dc70 con 0x7f74d0150ae0 2024-03-28T19:17:11.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.059+0000 7f74cea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74d00a61a0 0x7f74d00c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57130/0 (socket says 172.21.15.27:57130) 2024-03-28T19:17:11.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.059+0000 7f74cea00640 1 -- 172.21.15.27:0/2989427934 learned_addr learned my addr 172.21.15.27:0/2989427934 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:11.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.059+0000 7f74cfe00640 1 --2- 172.21.15.27:0/2989427934 >> v2:172.21.15.27:3300/0 conn(0x7f74d0153a30 0x7f74d0153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.060+0000 7f74cf400640 1 --2- 172.21.15.27:0/2989427934 >> v2:172.21.15.155:3300/0 conn(0x7f74d0150ae0 0x7f74d00a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.060+0000 7f74cfe00640 1 -- 172.21.15.27:0/2989427934 >> v2:172.21.15.155:3300/0 conn(0x7f74d0150ae0 msgr2=0x7f74d00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.060+0000 7f74cfe00640 1 --2- 172.21.15.27:0/2989427934 >> v2:172.21.15.155:3300/0 conn(0x7f74d0150ae0 0x7f74d00a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.060+0000 7f74cfe00640 1 -- 172.21.15.27:0/2989427934 >> v1:172.21.15.27:6789/0 conn(0x7f74d00a61a0 legacy=0x7f74d00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.060+0000 7f74cfe00640 1 -- 172.21.15.27:0/2989427934 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0153550 con 0x7f74d0153a30 2024-03-28T19:17:11.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.062+0000 7f74cfe00640 1 --2- 172.21.15.27:0/2989427934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d0153a30 0x7f74d0153e80 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f74d00a9760 tx=0x7f74c802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8abee19fc8d7fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.062+0000 7f74ce000640 1 -- 172.21.15.27:0/2989427934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c8039070 con 0x7f74d0153a30 2024-03-28T19:17:11.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.063+0000 7f74ce000640 1 -- 172.21.15.27:0/2989427934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c802cbe0 con 0x7f74d0153a30 2024-03-28T19:17:11.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.063+0000 7f74ce000640 1 -- 172.21.15.27:0/2989427934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c8034440 con 0x7f74d0153a30 2024-03-28T19:17:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.064+0000 7f74d4c00640 1 -- 172.21.15.27:0/2989427934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d0153a30 msgr2=0x7f74d0153e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.064+0000 7f74d4c00640 1 --2- 172.21.15.27:0/2989427934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d0153a30 0x7f74d0153e80 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f74d00a9760 tx=0x7f74c802be30 comp rx=0 tx=0).stop 2024-03-28T19:17:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.064+0000 7f74d4c00640 1 -- 172.21.15.27:0/2989427934 shutdown_connections 2024-03-28T19:17:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.064+0000 7f74d4c00640 1 --2- 172.21.15.27:0/2989427934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d0153a30 0x7f74d0153e80 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.064+0000 7f74d4c00640 1 --2- 172.21.15.27:0/2989427934 >> v2:172.21.15.155:3300/0 conn(0x7f74d0150ae0 0x7f74d00a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.064+0000 7f74d4c00640 1 -- 172.21.15.27:0/2989427934 >> 172.21.15.27:0/2989427934 conn(0x7f74d006b0c0 msgr2=0x7f74d006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.064+0000 7f74d4c00640 1 -- 172.21.15.27:0/2989427934 shutdown_connections 2024-03-28T19:17:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.064+0000 7f74d4c00640 1 -- 172.21.15.27:0/2989427934 wait complete. 2024-03-28T19:17:11.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.067+0000 7f74d4c00640 1 Processor -- start 2024-03-28T19:17:11.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.067+0000 7f74d4c00640 1 -- start start 2024-03-28T19:17:11.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74d4c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d00a61a0 0x7f74d0144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74d4c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74d0150ae0 0x7f74d01459e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74d4c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74d0153a30 0x7f74d01ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74d4c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d006e4d0 con 0x7f74d00a61a0 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74d4c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d006e1f0 con 0x7f74d0150ae0 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74d4c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d006e7b0 con 0x7f74d0153a30 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74cf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d00a61a0 0x7f74d0144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74cf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d00a61a0 0x7f74d0144d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36560/0 (socket says 172.21.15.27:36560) 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74cf400640 1 -- 172.21.15.27:0/1964566782 learned_addr learned my addr 172.21.15.27:0/1964566782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.069+0000 7f74cfe00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74d0153a30 0x7f74d01ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.070+0000 7f74cea00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74d0150ae0 0x7f74d01459e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.070+0000 7f74cf400640 1 -- 172.21.15.27:0/1964566782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74d0153a30 msgr2=0x7f74d01ec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.070+0000 7f74cf400640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74d0153a30 0x7f74d01ec410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.070+0000 7f74cf400640 1 -- 172.21.15.27:0/1964566782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74d0150ae0 msgr2=0x7f74d01459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.070+0000 7f74cf400640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74d0150ae0 0x7f74d01459e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.070+0000 7f74cf400640 1 -- 172.21.15.27:0/1964566782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d01ecc50 con 0x7f74d00a61a0 2024-03-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.070+0000 7f74cf400640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d00a61a0 0x7f74d0144d00 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00d800 tx=0x7f74bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.071+0000 7f74c7400640 1 -- 172.21.15.27:0/1964566782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc00b6c0 con 0x7f74d00a61a0 2024-03-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.071+0000 7f74c7400640 1 -- 172.21.15.27:0/1964566782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc019070 con 0x7f74d00a61a0 2024-03-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.071+0000 7f74c7400640 1 -- 172.21.15.27:0/1964566782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc00ba80 con 0x7f74d00a61a0 2024-03-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.071+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d01ecef0 con 0x7f74d00a61a0 2024-03-28T19:17:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.071+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d01ec950 con 0x7f74d00a61a0 2024-03-28T19:17:11.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.072+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d0067dd0 con 0x7f74d00a61a0 2024-03-28T19:17:11.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.074+0000 7f74c7400640 1 -- 172.21.15.27:0/1964566782 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f74bc00bc20 con 0x7f74d00a61a0 2024-03-28T19:17:11.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.075+0000 7f74c7400640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74b006bbd0 0x7f74b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.075+0000 7f74c7400640 1 -- 172.21.15.27:0/1964566782 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f74bc08bd10 con 0x7f74d00a61a0 2024-03-28T19:17:11.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.075+0000 7f74cea00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74b006bbd0 0x7f74b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.076+0000 7f74cea00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74b006bbd0 0x7f74b006e090 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f74d00a9940 tx=0x7f74c0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.084+0000 7f74c7400640 1 -- 172.21.15.27:0/1964566782 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f74bc059a30 con 0x7f74d00a61a0 2024-03-28T19:17:11.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.306+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d700c12a0 con 0x7f9d70150c70 2024-03-28T19:17:11.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.307+0000 7f9d6cc00640 1 -- 172.21.15.27:0/2279119571 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4661 (secure 0 0 0) 0x7f9d700c12a0 con 0x7f9d70150c70 2024-03-28T19:17:11.307 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:11.307 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:10.170632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:10.170633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:17:11.308 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 172 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d4c06bbd0 msgr2=0x7f9d4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d4c06bbd0 0x7f9d4c06e090 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9d60036000 tx=0x7f9d60008030 comp rx=0 tx=0).stop 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d70150c70 msgr2=0x7f9d700a8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d70150c70 0x7f9d700a8440 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c500 tx=0x7f9d5800c9c0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 shutdown_connections 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9d4c06bbd0 0x7f9d4c06e090 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d701534c0 0x7f9d7014bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d70150c70 0x7f9d700a8440 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 --2- 172.21.15.27:0/2279119571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d700c3690 0x7f9d701ec8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 >> 172.21.15.27:0/2279119571 conn(0x7f9d7006b9d0 msgr2=0x7f9d70061ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:11.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 shutdown_connections 2024-03-28T19:17:11.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.310+0000 7f9d77000640 1 -- 172.21.15.27:0/2279119571 wait complete. 2024-03-28T19:17:11.336 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-03-28T19:17:11.336 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:17:11.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.347+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74d00735f0 con 0x7f74d00a61a0 2024-03-28T19:17:11.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.348+0000 7f74c7400640 1 -- 172.21.15.27:0/1964566782 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4661 (secure 0 0 0) 0x7f74bc059330 con 0x7f74d00a61a0 2024-03-28T19:17:11.349 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:11.349 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:10.170632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:10.170633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:17:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 172 2024-03-28T19:17:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74b006bbd0 msgr2=0x7f74b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74b006bbd0 0x7f74b006e090 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f74d00a9940 tx=0x7f74c0009040 comp rx=0 tx=0).stop 2024-03-28T19:17:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d00a61a0 msgr2=0x7f74d0144d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d00a61a0 0x7f74d0144d00 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00d800 tx=0x7f74bc00dcc0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 shutdown_connections 2024-03-28T19:17:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74b006bbd0 0x7f74b006e090 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74d0153a30 0x7f74d01ec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74d0150ae0 0x7f74d01459e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 --2- 172.21.15.27:0/1964566782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74d00a61a0 0x7f74d0144d00 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.350+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 >> 172.21.15.27:0/1964566782 conn(0x7f74d006b0c0 msgr2=0x7f74d0154a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.351+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 shutdown_connections 2024-03-28T19:17:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.351+0000 7f74d4c00640 1 -- 172.21.15.27:0/1964566782 wait complete. 2024-03-28T19:17:11.369 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-03-28T19:17:11.369 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:17:11.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:11.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.536+0000 7fcf22a00640 1 Processor -- start 2024-03-28T19:17:11.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.537+0000 7fcf22a00640 1 -- start start 2024-03-28T19:17:11.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.538+0000 7fcf22a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcf1c157150 0x7fcf1c0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.538+0000 7fcf22a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c157530 con 0x7fcf1c0c1390 2024-03-28T19:17:11.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.538+0000 7fcf22a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c06dbd0 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.538+0000 7fcf22a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c06e1e0 con 0x7fcf1c157150 2024-03-28T19:17:11.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.538+0000 7fcf20c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcf1c0c1390 0x7fcf1c0c3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57138/0 (socket says 172.21.15.27:57138) 2024-03-28T19:17:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.538+0000 7fcf20c00640 1 -- 172.21.15.27:0/4143149451 learned_addr learned my addr 172.21.15.27:0/4143149451 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.538+0000 7fcf21600640 1 --2- 172.21.15.27:0/4143149451 >> v2:172.21.15.27:3300/0 conn(0x7fcf1c157150 0x7fcf1c0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.539+0000 7fcf1be00640 1 -- 172.21.15.27:0/4143149451 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280642989 0 0) 0x7fcf1c06dbd0 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.539+0000 7fcf1be00640 1 -- 172.21.15.27:0/4143149451 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf04003520 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.539+0000 7fcf1be00640 1 -- 172.21.15.27:0/4143149451 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274568964 0 0) 0x7fcf1c157530 con 0x7fcf1c0c1390 2024-03-28T19:17:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.539+0000 7fcf21600640 1 -- 172.21.15.27:0/4143149451 >> v1:172.21.15.155:6789/0 conn(0x7fcf1c0c6ee0 legacy=0x7fcf1c0c3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.539+0000 7fcf21600640 1 -- 172.21.15.27:0/4143149451 >> v1:172.21.15.27:6789/0 conn(0x7fcf1c0c1390 legacy=0x7fcf1c0c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.539+0000 7fcf21600640 1 -- 172.21.15.27:0/4143149451 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c15de60 con 0x7fcf1c157150 2024-03-28T19:17:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.541+0000 7fcf21600640 1 --2- 172.21.15.27:0/4143149451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c157150 0x7fcf1c0c69a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fcf100097a0 tx=0x7fcf1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238987b826350af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.541+0000 7fcf1be00640 1 -- 172.21.15.27:0/4143149451 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10035020 con 0x7fcf1c157150 2024-03-28T19:17:11.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.541+0000 7fcf1be00640 1 -- 172.21.15.27:0/4143149451 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf10031830 con 0x7fcf1c157150 2024-03-28T19:17:11.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.541+0000 7fcf1be00640 1 -- 172.21.15.27:0/4143149451 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10031c00 con 0x7fcf1c157150 2024-03-28T19:17:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.542+0000 7fcf22a00640 1 -- 172.21.15.27:0/4143149451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c157150 msgr2=0x7fcf1c0c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.542+0000 7fcf22a00640 1 --2- 172.21.15.27:0/4143149451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c157150 0x7fcf1c0c69a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fcf100097a0 tx=0x7fcf1002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.542+0000 7fcf22a00640 1 -- 172.21.15.27:0/4143149451 shutdown_connections 2024-03-28T19:17:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.542+0000 7fcf22a00640 1 --2- 172.21.15.27:0/4143149451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c157150 0x7fcf1c0c69a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.542+0000 7fcf22a00640 1 -- 172.21.15.27:0/4143149451 >> 172.21.15.27:0/4143149451 conn(0x7fcf1c06b0c0 msgr2=0x7fcf1c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.542+0000 7fcf22a00640 1 -- 172.21.15.27:0/4143149451 shutdown_connections 2024-03-28T19:17:11.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.542+0000 7fcf22a00640 1 -- 172.21.15.27:0/4143149451 wait complete. 2024-03-28T19:17:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.545+0000 7fcf22a00640 1 Processor -- start 2024-03-28T19:17:11.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.545+0000 7fcf22a00640 1 -- start start 2024-03-28T19:17:11.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.546+0000 7fcf22a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf1c0c1390 0x7fcf1c0a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.546+0000 7fcf22a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c0c6ee0 0x7fcf1c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.546+0000 7fcf22a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf1c157150 0x7fcf1c0a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.546+0000 7fcf22a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0c0fc0 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.546+0000 7fcf22a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c06e710 con 0x7fcf1c157150 2024-03-28T19:17:11.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.546+0000 7fcf22a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0c2bc0 con 0x7fcf1c0c1390 2024-03-28T19:17:11.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.547+0000 7fcf20c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c0c6ee0 0x7fcf1c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.547+0000 7fcf20c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c0c6ee0 0x7fcf1c0a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36588/0 (socket says 172.21.15.27:36588) 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.547+0000 7fcf20c00640 1 -- 172.21.15.27:0/3774774103 learned_addr learned my addr 172.21.15.27:0/3774774103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.547+0000 7fcf21600640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf1c0c1390 0x7fcf1c0a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.547+0000 7fcf22000640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf1c157150 0x7fcf1c0a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf20c00640 1 -- 172.21.15.27:0/3774774103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf1c0c1390 msgr2=0x7fcf1c0a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf20c00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf1c0c1390 0x7fcf1c0a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf20c00640 1 -- 172.21.15.27:0/3774774103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf1c157150 msgr2=0x7fcf1c0a4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf20c00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf1c157150 0x7fcf1c0a4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf20c00640 1 -- 172.21.15.27:0/3774774103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c1f31c0 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf20c00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c0c6ee0 0x7fcf1c0a4600 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00c4a0 tx=0x7fcf0c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf19600640 1 -- 172.21.15.27:0/3774774103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c018070 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf19600640 1 -- 172.21.15.27:0/3774774103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0074e0 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf19600640 1 -- 172.21.15.27:0/3774774103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c004a20 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.548+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c0a3e00 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.549+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c0a5080 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.551+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c067dd0 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.551+0000 7fcf19600640 1 -- 172.21.15.27:0/3774774103 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcf0c004bc0 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.553+0000 7fcf19600640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf0006bbd0 0x7fcf0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.553+0000 7fcf19600640 1 -- 172.21.15.27:0/3774774103 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7fcf0c08ab90 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.553+0000 7fcf21600640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf0006bbd0 0x7fcf0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.554+0000 7fcf21600640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf0006bbd0 0x7fcf0006e090 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcf10036000 tx=0x7fcf1003c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.562+0000 7fcf19600640 1 -- 172.21.15.27:0/3774774103 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcf0c058200 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.628+0000 7ff28e600640 1 Processor -- start 2024-03-28T19:17:11.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.628+0000 7ff28e600640 1 -- start start 2024-03-28T19:17:11.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.630+0000 7ff28e600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff288152bd0 0x7ff28806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.630+0000 7ff28e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff28806f1e0 0x7ff288070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.630+0000 7ff28e600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2880c52e0 con 0x7ff2880c4dc0 2024-03-28T19:17:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.630+0000 7ff28e600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2881557e0 con 0x7ff28806f1e0 2024-03-28T19:17:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.630+0000 7ff28e600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff288155ac0 con 0x7ff288152bd0 2024-03-28T19:17:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff286a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff28806f1e0 0x7ff288070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff286a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff28806f1e0 0x7ff288070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36590/0 (socket says 172.21.15.27:36590) 2024-03-28T19:17:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff286a00640 1 -- 172.21.15.27:0/3265806546 learned_addr learned my addr 172.21.15.27:0/3265806546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff287400640 1 --2- 172.21.15.27:0/3265806546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff288152bd0 0x7ff28806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff286a00640 1 -- 172.21.15.27:0/3265806546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff288152bd0 msgr2=0x7ff28806eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:11.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff286a00640 1 --2- 172.21.15.27:0/3265806546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff288152bd0 0x7ff28806eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff286a00640 1 -- 172.21.15.27:0/3265806546 >> v1:172.21.15.155:6789/0 conn(0x7ff2880c4dc0 legacy=0x7ff288158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff286a00640 1 -- 172.21.15.27:0/3265806546 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff288155da0 con 0x7ff28806f1e0 2024-03-28T19:17:11.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.631+0000 7ff287400640 1 --2- 172.21.15.27:0/3265806546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff288152bd0 0x7ff28806eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:17:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.632+0000 7ff286a00640 1 --2- 172.21.15.27:0/3265806546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288070540 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7ff274009870 tx=0x7ff27402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8877a19d2a4a65e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff286000640 1 -- 172.21.15.27:0/3265806546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274034070 con 0x7ff28806f1e0 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff286000640 1 -- 172.21.15.27:0/3265806546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff274030840 con 0x7ff28806f1e0 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff286000640 1 -- 172.21.15.27:0/3265806546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274030c10 con 0x7ff28806f1e0 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff28e600640 1 -- 172.21.15.27:0/3265806546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 msgr2=0x7ff288070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff28e600640 1 --2- 172.21.15.27:0/3265806546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288070540 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7ff274009870 tx=0x7ff27402b980 comp rx=0 tx=0).stop 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff28e600640 1 -- 172.21.15.27:0/3265806546 shutdown_connections 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff28e600640 1 --2- 172.21.15.27:0/3265806546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288070540 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff28e600640 1 --2- 172.21.15.27:0/3265806546 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff288152bd0 0x7ff28806eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.633+0000 7ff28e600640 1 -- 172.21.15.27:0/3265806546 >> 172.21.15.27:0/3265806546 conn(0x7ff288066dd0 msgr2=0x7ff2880671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.634+0000 7ff28e600640 1 -- 172.21.15.27:0/3265806546 shutdown_connections 2024-03-28T19:17:11.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.634+0000 7ff28e600640 1 -- 172.21.15.27:0/3265806546 wait complete. 2024-03-28T19:17:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.635+0000 7ff28e600640 1 Processor -- start 2024-03-28T19:17:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.635+0000 7ff28e600640 1 -- start start 2024-03-28T19:17:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff28e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff28e600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2880c4dc0 0x7ff2880acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff28e600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff288152bd0 0x7ff2880ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff28e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2880c6df0 con 0x7ff28806f1e0 2024-03-28T19:17:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff28e600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2880c6c70 con 0x7ff288152bd0 2024-03-28T19:17:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff28e600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff288070a80 con 0x7ff2880c4dc0 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff287400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff287400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288147b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36606/0 (socket says 172.21.15.27:36606) 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.636+0000 7ff287400640 1 -- 172.21.15.27:0/799156029 learned_addr learned my addr 172.21.15.27:0/799156029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff287e00640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff288152bd0 0x7ff2880ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff286a00640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2880c4dc0 0x7ff2880acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff287400640 1 -- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2880c4dc0 msgr2=0x7ff2880acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff287400640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2880c4dc0 0x7ff2880acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff287400640 1 -- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff288152bd0 msgr2=0x7ff2880ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff287400640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff288152bd0 0x7ff2880ad2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff287400640 1 -- 172.21.15.27:0/799156029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2880aafc0 con 0x7ff28806f1e0 2024-03-28T19:17:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff286a00640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2880c4dc0 0x7ff2880acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:17:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.637+0000 7ff287400640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288147b20 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7ff27800c820 tx=0x7ff27800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.638+0000 7ff27f400640 1 -- 172.21.15.27:0/799156029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff278015070 con 0x7ff28806f1e0 2024-03-28T19:17:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.638+0000 7ff27f400640 1 -- 172.21.15.27:0/799156029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff278007a30 con 0x7ff28806f1e0 2024-03-28T19:17:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.638+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28814c010 con 0x7ff28806f1e0 2024-03-28T19:17:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.638+0000 7ff27f400640 1 -- 172.21.15.27:0/799156029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff278007e00 con 0x7ff28806f1e0 2024-03-28T19:17:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.638+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2881ef190 con 0x7ff28806f1e0 2024-03-28T19:17:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.639+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2880c4370 con 0x7ff28806f1e0 2024-03-28T19:17:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.640+0000 7ff27f400640 1 -- 172.21.15.27:0/799156029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff278018070 con 0x7ff28806f1e0 2024-03-28T19:17:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.641+0000 7ff27f400640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff26806bbd0 0x7ff26806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.641+0000 7ff27f400640 1 -- 172.21.15.27:0/799156029 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7ff27808b7f0 con 0x7ff28806f1e0 2024-03-28T19:17:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.642+0000 7ff286a00640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff26806bbd0 0x7ff26806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.643+0000 7ff286a00640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff26806bbd0 0x7ff26806e090 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff274036000 tx=0x7ff27403b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.648+0000 7ff27f400640 1 -- 172.21.15.27:0/799156029 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff2880c4370 con 0x7ff28806f1e0 2024-03-28T19:17:11.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.930+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff28804f400 con 0x7ff28806f1e0 2024-03-28T19:17:11.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.931+0000 7ff27f400640 1 -- 172.21.15.27:0/799156029 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4661 (secure 0 0 0) 0x7ff278059510 con 0x7ff28806f1e0 2024-03-28T19:17:11.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:11.932 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:10.170632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:10.170633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:17:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 172 2024-03-28T19:17:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff26806bbd0 msgr2=0x7ff26806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff26806bbd0 0x7ff26806e090 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff274036000 tx=0x7ff27403b040 comp rx=0 tx=0).stop 2024-03-28T19:17:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 msgr2=0x7ff288147b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288147b20 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7ff27800c820 tx=0x7ff27800cc20 comp rx=0 tx=0).stop 2024-03-28T19:17:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 shutdown_connections 2024-03-28T19:17:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff26806bbd0 0x7ff26806e090 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff288152bd0 0x7ff2880ad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff2880c4dc0 0x7ff2880acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 --2- 172.21.15.27:0/799156029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28806f1e0 0x7ff288147b20 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 >> 172.21.15.27:0/799156029 conn(0x7ff288066dd0 msgr2=0x7ff288154440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.934+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 shutdown_connections 2024-03-28T19:17:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.935+0000 7ff28e600640 1 -- 172.21.15.27:0/799156029 wait complete. 2024-03-28T19:17:11.951 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 172, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 172, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6014": { "addr": "172.21.15.155:6835/938167944", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 938167944, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 938167944, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6014, "incarnation": 172, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:17:10.170632+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6014 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 172, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6188": { "addr": "172.21.15.27:6837/2601264653", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 2601264653, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 2601264653, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6188, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:17:10.170633+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6188 } } } ], "standbys": [] } 2024-03-28T19:17:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.967+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf1c15d830 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.967+0000 7fcf19600640 1 -- 172.21.15.27:0/3774774103 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4661 (secure 0 0 0) 0x7fcf0c05c2a0 con 0x7fcf1c0c6ee0 2024-03-28T19:17:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:11.968 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:10.170632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:10.170633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:17:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 172 2024-03-28T19:17:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf0006bbd0 msgr2=0x7fcf0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf0006bbd0 0x7fcf0006e090 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcf10036000 tx=0x7fcf1003c040 comp rx=0 tx=0).stop 2024-03-28T19:17:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c0c6ee0 msgr2=0x7fcf1c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c0c6ee0 0x7fcf1c0a4600 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00c4a0 tx=0x7fcf0c00c960 comp rx=0 tx=0).stop 2024-03-28T19:17:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 shutdown_connections 2024-03-28T19:17:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcf0006bbd0 0x7fcf0006e090 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf1c157150 0x7fcf1c0a4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf1c0c6ee0 0x7fcf1c0a4600 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 --2- 172.21.15.27:0/3774774103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf1c0c1390 0x7fcf1c0a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:11.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 >> 172.21.15.27:0/3774774103 conn(0x7fcf1c06b0c0 msgr2=0x7fcf1c0715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:11.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 shutdown_connections 2024-03-28T19:17:11.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:11.971+0000 7fcf22a00640 1 -- 172.21.15.27:0/3774774103 wait complete. 2024-03-28T19:17:11.988 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 172, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 172, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6014": { "addr": "172.21.15.155:6835/938167944", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 938167944, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 938167944, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6014, "incarnation": 172, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:17:10.170632+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6014 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 172, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6188": { "addr": "172.21.15.27:6837/2601264653", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 2601264653, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 2601264653, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6188, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:17:10.170633+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6188 } } } ], "standbys": [] } 2024-03-28T19:17:13.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:13.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.150+0000 7f0bf2600640 1 Processor -- start 2024-03-28T19:17:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.150+0000 7f0bf2600640 1 -- start start 2024-03-28T19:17:14.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.151+0000 7f0bf2600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bec0c58c0 0x7f0bec0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.151+0000 7f0bf2600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0bec0c8ed0 0x7f0bec1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.151+0000 7f0bf2600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec156230 con 0x7f0bec0be5d0 2024-03-28T19:17:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.151+0000 7f0bf2600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec158bd0 con 0x7f0bec0c58c0 2024-03-28T19:17:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.151+0000 7f0bf2600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec158eb0 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.151+0000 7f0bf1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bec0c58c0 0x7f0bec0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.152+0000 7f0bf1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0bec0c58c0 0x7f0bec0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36618/0 (socket says 172.21.15.27:36618) 2024-03-28T19:17:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.152+0000 7f0bf1200640 1 -- 172.21.15.27:0/4097248073 learned_addr learned my addr 172.21.15.27:0/4097248073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.152+0000 7f0bebe00640 1 --2- 172.21.15.27:0/4097248073 >> v2:172.21.15.155:3300/0 conn(0x7f0bec0c8ed0 0x7f0bec1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.152+0000 7f0bf1200640 1 -- 172.21.15.27:0/4097248073 >> v2:172.21.15.155:3300/0 conn(0x7f0bec0c8ed0 msgr2=0x7f0bec1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.152+0000 7f0bf1200640 1 --2- 172.21.15.27:0/4097248073 >> v2:172.21.15.155:3300/0 conn(0x7f0bec0c8ed0 0x7f0bec1555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.152+0000 7f0bf1200640 1 -- 172.21.15.27:0/4097248073 >> v1:172.21.15.155:6789/0 conn(0x7f0bec0be5d0 legacy=0x7f0bec155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.152+0000 7f0bf1200640 1 -- 172.21.15.27:0/4097248073 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec159190 con 0x7f0bec0c58c0 2024-03-28T19:17:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.153+0000 7f0bf1200640 1 --2- 172.21.15.27:0/4097248073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c58c0 0x7f0bec0c5ca0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f0bec0c83d0 tx=0x7f0be002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b95ee4bb2492a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.153+0000 7f0beb400640 1 -- 172.21.15.27:0/4097248073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be0038070 con 0x7f0bec0c58c0 2024-03-28T19:17:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.153+0000 7f0beb400640 1 -- 172.21.15.27:0/4097248073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be002f050 con 0x7f0bec0c58c0 2024-03-28T19:17:14.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.153+0000 7f0beb400640 1 -- 172.21.15.27:0/4097248073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be0031970 con 0x7f0bec0c58c0 2024-03-28T19:17:14.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.154+0000 7f0bf2600640 1 -- 172.21.15.27:0/4097248073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c58c0 msgr2=0x7f0bec0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.154+0000 7f0bf2600640 1 --2- 172.21.15.27:0/4097248073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c58c0 0x7f0bec0c5ca0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f0bec0c83d0 tx=0x7f0be002b7f0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.154+0000 7f0bf2600640 1 -- 172.21.15.27:0/4097248073 shutdown_connections 2024-03-28T19:17:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.154+0000 7f0bf2600640 1 --2- 172.21.15.27:0/4097248073 >> v2:172.21.15.155:3300/0 conn(0x7f0bec0c8ed0 0x7f0bec1555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.154+0000 7f0bf2600640 1 --2- 172.21.15.27:0/4097248073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c58c0 0x7f0bec0c5ca0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.154+0000 7f0bf2600640 1 -- 172.21.15.27:0/4097248073 >> 172.21.15.27:0/4097248073 conn(0x7f0bec06b6f0 msgr2=0x7f0bec0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.154+0000 7f0bf2600640 1 -- 172.21.15.27:0/4097248073 shutdown_connections 2024-03-28T19:17:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.154+0000 7f0bf2600640 1 -- 172.21.15.27:0/4097248073 wait complete. 2024-03-28T19:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.156+0000 7f0bf2600640 1 Processor -- start 2024-03-28T19:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.156+0000 7f0bf2600640 1 -- start start 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf2600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec0be5d0 0x7f0bec157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf2600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec0c58c0 0x7f0bec152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf2600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8ed0 0x7f0bec152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf2600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0b0820 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf2600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec159750 con 0x7f0bec0be5d0 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf2600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0b09a0 con 0x7f0bec0c58c0 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8ed0 0x7f0bec152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8ed0 0x7f0bec152790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36632/0 (socket says 172.21.15.27:36632) 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf1c00640 1 -- 172.21.15.27:0/2769658478 learned_addr learned my addr 172.21.15.27:0/2769658478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.157+0000 7f0bf1200640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec0be5d0 0x7f0bec157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bebe00640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec0c58c0 0x7f0bec152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bf1c00640 1 -- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec0c58c0 msgr2=0x7f0bec152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bf1c00640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec0c58c0 0x7f0bec152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bf1c00640 1 -- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec0be5d0 msgr2=0x7f0bec157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bf1c00640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec0be5d0 0x7f0bec157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bf1c00640 1 -- 172.21.15.27:0/2769658478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec152fb0 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bebe00640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec0c58c0 0x7f0bec152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:17:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bf1c00640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8ed0 0x7f0bec152790 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f0be400b9c0 tx=0x7f0be400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0be8c00640 1 -- 172.21.15.27:0/2769658478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4019070 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0be8c00640 1 -- 172.21.15.27:0/2769658478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be400cba0 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0be8c00640 1 -- 172.21.15.27:0/2769658478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4014440 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec1f3cd0 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.158+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec0c2390 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.160+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bec069640 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.160+0000 7f0be8c00640 1 -- 172.21.15.27:0/2769658478 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0be401c070 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.161+0000 7f0be8c00640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.161+0000 7f0be8c00640 1 -- 172.21.15.27:0/2769658478 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f0be408b4d0 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.162+0000 7f0bf1200640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.163+0000 7f0bf1200640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0be0035000 tx=0x7f0be0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:14.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.166+0000 7f0be8c00640 1 -- 172.21.15.27:0/2769658478 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0be4058ba0 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.189+0000 7f45a8e00640 1 Processor -- start 2024-03-28T19:17:14.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.189+0000 7f45a8e00640 1 -- start start 2024-03-28T19:17:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.190+0000 7f45a8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45a414f960 0x7f45a40ace30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.191+0000 7f45a8e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a406c1d0 con 0x7f45a414fd40 2024-03-28T19:17:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.191+0000 7f45a8e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a4153370 con 0x7f45a4066dd0 2024-03-28T19:17:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.191+0000 7f45a8e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a4072e60 con 0x7f45a414f960 2024-03-28T19:17:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.191+0000 7f45a1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45a414f960 0x7f45a40ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.191+0000 7f45a1400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f45a414fd40 0x7f45a4071640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57144/0 (socket says 172.21.15.27:57144) 2024-03-28T19:17:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.191+0000 7f45a2800640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f45a4066dd0 0x7f45a4071d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:50772/0 (socket says 172.21.15.27:50772) 2024-03-28T19:17:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.191+0000 7f45a1400640 1 -- 172.21.15.27:0/811998935 learned_addr learned my addr 172.21.15.27:0/811998935 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.192+0000 7f45a1e00640 1 -- 172.21.15.27:0/811998935 >> v1:172.21.15.155:6789/0 conn(0x7f45a4066dd0 legacy=0x7f45a4071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.192+0000 7f45a1e00640 1 -- 172.21.15.27:0/811998935 >> v1:172.21.15.27:6789/0 conn(0x7f45a414fd40 legacy=0x7f45a4071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.192+0000 7f45a1e00640 1 -- 172.21.15.27:0/811998935 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a406c470 con 0x7f45a414f960 2024-03-28T19:17:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.192+0000 7f45a0a00640 1 -- 172.21.15.27:0/811998935 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745594441 0 0) 0x7f45a4153370 con 0x7f45a4066dd0 2024-03-28T19:17:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.193+0000 7f45a1e00640 1 --2- 172.21.15.27:0/811998935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a414f960 0x7f45a40ace30 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f45940097a0 tx=0x7f459402b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfaee5332801ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.193+0000 7f45a0a00640 1 -- 172.21.15.27:0/811998935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594037070 con 0x7f45a414f960 2024-03-28T19:17:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.193+0000 7f45a0a00640 1 -- 172.21.15.27:0/811998935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4594030710 con 0x7f45a414f960 2024-03-28T19:17:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.193+0000 7f45a0a00640 1 -- 172.21.15.27:0/811998935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594030ae0 con 0x7f45a414f960 2024-03-28T19:17:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.193+0000 7f45a8e00640 1 -- 172.21.15.27:0/811998935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a414f960 msgr2=0x7f45a40ace30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.193+0000 7f45a8e00640 1 --2- 172.21.15.27:0/811998935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a414f960 0x7f45a40ace30 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f45940097a0 tx=0x7f459402b680 comp rx=0 tx=0).stop 2024-03-28T19:17:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.194+0000 7f45a8e00640 1 -- 172.21.15.27:0/811998935 shutdown_connections 2024-03-28T19:17:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.194+0000 7f45a8e00640 1 --2- 172.21.15.27:0/811998935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a414f960 0x7f45a40ace30 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.194+0000 7f45a8e00640 1 -- 172.21.15.27:0/811998935 >> 172.21.15.27:0/811998935 conn(0x7f45a40c3ff0 msgr2=0x7f45a40c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.194+0000 7f45a8e00640 1 -- 172.21.15.27:0/811998935 shutdown_connections 2024-03-28T19:17:14.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.194+0000 7f45a8e00640 1 -- 172.21.15.27:0/811998935 wait complete. 2024-03-28T19:17:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.195+0000 7f45a8e00640 1 Processor -- start 2024-03-28T19:17:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.195+0000 7f45a8e00640 1 -- start start 2024-03-28T19:17:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.196+0000 7f45a8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a4066dd0 0x7f45a4154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.196+0000 7f45a8e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45a414f960 0x7f45a40c6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.196+0000 7f45a8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a414fd40 0x7f45a4151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.196+0000 7f45a8e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a40672e0 con 0x7f45a4066dd0 2024-03-28T19:17:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.196+0000 7f45a8e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a40c3c70 con 0x7f45a414fd40 2024-03-28T19:17:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.196+0000 7f45a8e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4067460 con 0x7f45a414f960 2024-03-28T19:17:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a4066dd0 0x7f45a4154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a4066dd0 0x7f45a4154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36646/0 (socket says 172.21.15.27:36646) 2024-03-28T19:17:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 -- 172.21.15.27:0/2988198165 learned_addr learned my addr 172.21.15.27:0/2988198165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a2800640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a414fd40 0x7f45a4151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 -- 172.21.15.27:0/2988198165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45a414f960 msgr2=0x7f45a40c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45a414f960 0x7f45a40c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 -- 172.21.15.27:0/2988198165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a414fd40 msgr2=0x7f45a4151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a414fd40 0x7f45a4151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 -- 172.21.15.27:0/2988198165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40c5ff0 con 0x7f45a4066dd0 2024-03-28T19:17:14.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.197+0000 7f45a1e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a4066dd0 0x7f45a4154a30 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f45940097a0 tx=0x7f45940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.198+0000 7f459a000640 1 -- 172.21.15.27:0/2988198165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594042070 con 0x7f45a4066dd0 2024-03-28T19:17:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.198+0000 7f459a000640 1 -- 172.21.15.27:0/2988198165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4594032a40 con 0x7f45a4066dd0 2024-03-28T19:17:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.198+0000 7f459a000640 1 -- 172.21.15.27:0/2988198165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594037040 con 0x7f45a4066dd0 2024-03-28T19:17:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.198+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a40c6ba0 con 0x7f45a4066dd0 2024-03-28T19:17:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.198+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a41ee9f0 con 0x7f45a4066dd0 2024-03-28T19:17:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.199+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a404f400 con 0x7f45a4066dd0 2024-03-28T19:17:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.200+0000 7f459a000640 1 -- 172.21.15.27:0/2988198165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4594045070 con 0x7f45a4066dd0 2024-03-28T19:17:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.201+0000 7f459a000640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f458406bbd0 0x7f458406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.201+0000 7f459a000640 1 -- 172.21.15.27:0/2988198165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f45940ad770 con 0x7f45a4066dd0 2024-03-28T19:17:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.201+0000 7f45a1400640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f458406bbd0 0x7f458406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.202+0000 7f45a1400640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f458406bbd0 0x7f458406e090 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f45a4154f70 tx=0x7f4590007a40 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:14.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.206+0000 7f459a000640 1 -- 172.21.15.27:0/2988198165 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f459407af00 con 0x7f45a4066dd0 2024-03-28T19:17:14.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.404+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bec156580 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.405+0000 7f0be8c00640 1 -- 172.21.15.27:0/2769658478 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4661 (secure 0 0 0) 0x7f0be405cc40 con 0x7f0bec0c8ed0 2024-03-28T19:17:14.405 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:14.405 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:10.170632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:10.170633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:17:14.405 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 172 2024-03-28T19:17:14.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.407+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 msgr2=0x7f0bcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.407+0000 7f0bf2600640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0be0035000 tx=0x7f0be0008030 comp rx=0 tx=0).stop 2024-03-28T19:17:14.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.407+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8ed0 msgr2=0x7f0bec152790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.407+0000 7f0bf2600640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8ed0 0x7f0bec152790 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f0be400b9c0 tx=0x7f0be400be80 comp rx=0 tx=0).stop 2024-03-28T19:17:14.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.408+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 shutdown_connections 2024-03-28T19:17:14.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.408+0000 7f0bf2600640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bcc06bbd0 0x7f0bcc06e090 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.408+0000 7f0bf2600640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bec0c8ed0 0x7f0bec152790 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.408+0000 7f0bf2600640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bec0c58c0 0x7f0bec152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.408+0000 7f0bf2600640 1 --2- 172.21.15.27:0/2769658478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec0be5d0 0x7f0bec157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.408+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 >> 172.21.15.27:0/2769658478 conn(0x7f0bec06b6f0 msgr2=0x7f0bec0aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:14.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.408+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 shutdown_connections 2024-03-28T19:17:14.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.408+0000 7f0bf2600640 1 -- 172.21.15.27:0/2769658478 wait complete. 2024-03-28T19:17:14.426 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:17:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.441+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45a40c3a50 con 0x7f45a4066dd0 2024-03-28T19:17:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.442+0000 7f459a000640 1 -- 172.21.15.27:0/2988198165 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4661 (secure 0 0 0) 0x7f459407efa0 con 0x7f45a4066dd0 2024-03-28T19:17:14.442 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:14.442 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:10.170632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:10.170633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:17:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 172 2024-03-28T19:17:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.444+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f458406bbd0 msgr2=0x7f458406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.444+0000 7f45a8e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f458406bbd0 0x7f458406e090 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f45a4154f70 tx=0x7f4590007a40 comp rx=0 tx=0).stop 2024-03-28T19:17:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.444+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a4066dd0 msgr2=0x7f45a4154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.444+0000 7f45a8e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a4066dd0 0x7f45a4154a30 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f45940097a0 tx=0x7f45940046c0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.445+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 shutdown_connections 2024-03-28T19:17:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.445+0000 7f45a8e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f458406bbd0 0x7f458406e090 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.445+0000 7f45a8e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a414fd40 0x7f45a4151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.445+0000 7f45a8e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45a414f960 0x7f45a40c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.445+0000 7f45a8e00640 1 --2- 172.21.15.27:0/2988198165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45a4066dd0 0x7f45a4154a30 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.445+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 >> 172.21.15.27:0/2988198165 conn(0x7f45a40c3ff0 msgr2=0x7f45a4071ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.445+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 shutdown_connections 2024-03-28T19:17:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:14.445+0000 7f45a8e00640 1 -- 172.21.15.27:0/2988198165 wait complete. 2024-03-28T19:17:14.463 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:17:16.427 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:16.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.653+0000 7fe5ce800640 1 Processor -- start 2024-03-28T19:17:16.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.654+0000 7fe5ce800640 1 -- start start 2024-03-28T19:17:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.655+0000 7fe5ce800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c806eb80 0x7fe5c815a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.655+0000 7fe5ce800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c80c3830 con 0x7fe5c806e6f0 2024-03-28T19:17:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.655+0000 7fe5ce800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c806ef90 con 0x7fe5c8153ed0 2024-03-28T19:17:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.655+0000 7fe5ce800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c80c6990 con 0x7fe5c806eb80 2024-03-28T19:17:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.656+0000 7fe5c6a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe5c806e6f0 0x7fe5c8159bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57382/0 (socket says 172.21.15.27:57382) 2024-03-28T19:17:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.656+0000 7fe5c6a00640 1 -- 172.21.15.27:0/2954815253 learned_addr learned my addr 172.21.15.27:0/2954815253 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:16.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.656+0000 7fe5c7e00640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c806eb80 0x7fe5c815a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.656+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467147392 0 0) 0x7fe5c80c3830 con 0x7fe5c806e6f0 2024-03-28T19:17:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.656+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ac003520 con 0x7fe5c806e6f0 2024-03-28T19:17:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1215917420 0 0) 0x7fe5ac003520 con 0x7fe5c806e6f0 2024-03-28T19:17:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c806eb80 msgr2=0x7fe5c815a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c806eb80 0x7fe5c815a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 >> v1:172.21.15.155:6789/0 conn(0x7fe5c8153ed0 legacy=0x7fe5c806ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c80c66b0 con 0x7fe5c806e6f0 2024-03-28T19:17:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe5b0001670 con 0x7fe5c806e6f0 2024-03-28T19:17:16.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 >> v1:172.21.15.27:6789/0 conn(0x7fe5c806e6f0 legacy=0x7fe5c8159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c806eb80 0x7fe5ac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5ac003f20 0x7fe5ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c80c6990 con 0x7fe5ac004d40 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac00ba70 con 0x7fe5ac003f20 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac00bc70 con 0x7fe5c806eb80 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c7e00640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.657+0000 7fe5c7400640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c806eb80 0x7fe5ac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.658+0000 7fe5c6a00640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5ac003f20 0x7fe5ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.658+0000 7fe5c7e00640 1 -- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c806eb80 msgr2=0x7fe5ac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:16.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.658+0000 7fe5c7e00640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c806eb80 0x7fe5ac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.658+0000 7fe5c7e00640 1 -- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5ac003f20 msgr2=0x7fe5ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:16.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.658+0000 7fe5c7e00640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5ac003f20 0x7fe5ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.658+0000 7fe5c7e00640 1 -- 172.21.15.27:0/2954815253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac003520 con 0x7fe5ac004d40 2024-03-28T19:17:16.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.658+0000 7fe5c6a00640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5ac003f20 0x7fe5ac004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:17:16.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.660+0000 7fe5c7e00640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5ac0051c0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009ac0 tx=0x7fe5ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e817e991561943 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.660+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0039070 con 0x7fe5ac004d40 2024-03-28T19:17:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.660+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c002c6b0 con 0x7fe5ac004d40 2024-03-28T19:17:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.660+0000 7fe5c6000640 1 -- 172.21.15.27:0/2954815253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0009690 con 0x7fe5ac004d40 2024-03-28T19:17:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.661+0000 7fe5ce800640 1 -- 172.21.15.27:0/2954815253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 msgr2=0x7fe5ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.661+0000 7fe5ce800640 1 --2- 172.21.15.27:0/2954815253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5ac0051c0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009ac0 tx=0x7fe5ac005830 comp rx=0 tx=0).stop 2024-03-28T19:17:16.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.661+0000 7fe5c7e00640 1 -- 172.21.15.27:0/2954815253 reap_dead start 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.661+0000 7fe5ce800640 1 -- 172.21.15.27:0/2954815253 shutdown_connections 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.661+0000 7fe5ce800640 1 -- 172.21.15.27:0/2954815253 >> 172.21.15.27:0/2954815253 conn(0x7fe5c806b9d0 msgr2=0x7fe5c80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.661+0000 7fe5ce800640 1 -- 172.21.15.27:0/2954815253 shutdown_connections 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.661+0000 7fe5ce800640 1 -- 172.21.15.27:0/2954815253 wait complete. 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.663+0000 7fe5ce800640 1 Processor -- start 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.663+0000 7fe5ce800640 1 -- start start 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5ce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5c80a79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5ce800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c80aaf00 0x7fe5c80ab360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5ce800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5c80a6820 0x7fe5c80a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5ce800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c80c6380 con 0x7fe5ac004d40 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5ce800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c80c3db0 con 0x7fe5c80a6820 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5ce800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c80c1160 con 0x7fe5c80aaf00 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5c7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5c80a79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5c7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5c80a79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47274/0 (socket says 172.21.15.27:47274) 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5c7400640 1 -- 172.21.15.27:0/3632230456 learned_addr learned my addr 172.21.15.27:0/3632230456 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5c7e00640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5c80a6820 0x7fe5c80a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.664+0000 7fe5c6a00640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c80aaf00 0x7fe5c80ab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5c7400640 1 -- 172.21.15.27:0/3632230456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c80aaf00 msgr2=0x7fe5c80ab360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5c7400640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c80aaf00 0x7fe5c80ab360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5c7400640 1 -- 172.21.15.27:0/3632230456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5c80a6820 msgr2=0x7fe5c80a6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5c7400640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5c80a6820 0x7fe5c80a6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5c7400640 1 -- 172.21.15.27:0/3632230456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c8151030 con 0x7fe5ac004d40 2024-03-28T19:17:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5c7400640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5c80a79a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800fb30 tx=0x7fe5b800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5bf400640 1 -- 172.21.15.27:0/3632230456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8003ba0 con 0x7fe5ac004d40 2024-03-28T19:17:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5bf400640 1 -- 172.21.15.27:0/3632230456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b8018070 con 0x7fe5ac004d40 2024-03-28T19:17:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5bf400640 1 -- 172.21.15.27:0/3632230456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b800d8c0 con 0x7fe5ac004d40 2024-03-28T19:17:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c80a71e0 con 0x7fe5ac004d40 2024-03-28T19:17:16.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.665+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c81e5220 con 0x7fe5ac004d40 2024-03-28T19:17:16.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.667+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c804f400 con 0x7fe5ac004d40 2024-03-28T19:17:16.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.668+0000 7fe5bf400640 1 -- 172.21.15.27:0/3632230456 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe5b800da60 con 0x7fe5ac004d40 2024-03-28T19:17:16.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.668+0000 7fe5bf400640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe5a406bc20 0x7fe5a406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.669+0000 7fe5bf400640 1 -- 172.21.15.27:0/3632230456 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7fe5b808bae0 con 0x7fe5ac004d40 2024-03-28T19:17:16.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.669+0000 7fe5c6a00640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe5a406bc20 0x7fe5a406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.669+0000 7fe5c6a00640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe5a406bc20 0x7fe5a406e0e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe5c81f3b70 tx=0x7fe5b0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:16.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.678+0000 7fe5bf400640 1 -- 172.21.15.27:0/3632230456 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe5c80a71e0 con 0x7fe5ac004d40 2024-03-28T19:17:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.695+0000 7fc653000640 1 Processor -- start 2024-03-28T19:17:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.695+0000 7fc653000640 1 -- start start 2024-03-28T19:17:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.697+0000 7fc653000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc64c0c1390 0x7fc64c0c3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.697+0000 7fc653000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c157530 con 0x7fc64c0c36b0 2024-03-28T19:17:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.697+0000 7fc653000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c06dbd0 con 0x7fc64c157150 2024-03-28T19:17:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.697+0000 7fc653000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64c06e1e0 con 0x7fc64c0c1390 2024-03-28T19:17:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.698+0000 7fc652600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc64c0c36b0 0x7fc64c164400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57396/0 (socket says 172.21.15.27:57396) 2024-03-28T19:17:16.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.698+0000 7fc652600640 1 -- 172.21.15.27:0/3053143556 learned_addr learned my addr 172.21.15.27:0/3053143556 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:16.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.698+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383561861 0 0) 0x7fc64c157530 con 0x7fc64c0c36b0 2024-03-28T19:17:16.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.698+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc634003520 con 0x7fc64c0c36b0 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3268760857 0 0) 0x7fc634003520 con 0x7fc64c0c36b0 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 >> v2:172.21.15.155:3300/0 conn(0x7fc64c0c1390 msgr2=0x7fc64c0c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 --2- 172.21.15.27:0/3053143556 >> v2:172.21.15.155:3300/0 conn(0x7fc64c0c1390 0x7fc64c0c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 >> v1:172.21.15.155:6789/0 conn(0x7fc64c157150 legacy=0x7fc64c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c15de60 con 0x7fc64c0c36b0 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fc6440019f0 con 0x7fc64c0c36b0 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 >> v1:172.21.15.27:6789/0 conn(0x7fc64c0c36b0 legacy=0x7fc64c164400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6340039b0 0x7fc634003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc634004800 0x7fc634004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6340052e0 0x7fc63400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c06e1e0 con 0x7fc6340052e0 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63400c1e0 con 0x7fc634004800 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63400c3e0 con 0x7fc6340039b0 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.699+0000 7fc652600640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6340052e0 0x7fc63400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.700+0000 7fc652600640 1 -- 172.21.15.27:0/3053143556 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6340039b0 msgr2=0x7fc634003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.700+0000 7fc652600640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6340039b0 0x7fc634003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.700+0000 7fc652600640 1 -- 172.21.15.27:0/3053143556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc634004800 msgr2=0x7fc634004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.700+0000 7fc652600640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc634004800 0x7fc634004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.700+0000 7fc652600640 1 -- 172.21.15.27:0/3053143556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc634003520 con 0x7fc6340052e0 2024-03-28T19:17:16.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.702+0000 7fc652600640 1 -- 172.21.15.27:0/3053143556 reap_dead start 2024-03-28T19:17:16.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.702+0000 7fc652600640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6340052e0 0x7fc63400b9c0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fc64400a850 tx=0x7fc634005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76deaa044d5f34fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:16.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.702+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64400a4a0 con 0x7fc6340052e0 2024-03-28T19:17:16.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.702+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc644034650 con 0x7fc6340052e0 2024-03-28T19:17:16.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.702+0000 7fc64be00640 1 -- 172.21.15.27:0/3053143556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644034a40 con 0x7fc6340052e0 2024-03-28T19:17:16.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.703+0000 7fc653000640 1 -- 172.21.15.27:0/3053143556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6340052e0 msgr2=0x7fc63400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.703+0000 7fc653000640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6340052e0 0x7fc63400b9c0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fc64400a850 tx=0x7fc634005830 comp rx=0 tx=0).stop 2024-03-28T19:17:16.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.704+0000 7fc653000640 1 -- 172.21.15.27:0/3053143556 shutdown_connections 2024-03-28T19:17:16.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.704+0000 7fc653000640 1 --2- 172.21.15.27:0/3053143556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6340052e0 0x7fc63400b9c0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.704+0000 7fc653000640 1 -- 172.21.15.27:0/3053143556 >> 172.21.15.27:0/3053143556 conn(0x7fc64c06b0c0 msgr2=0x7fc64c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:16.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.704+0000 7fc653000640 1 -- 172.21.15.27:0/3053143556 shutdown_connections 2024-03-28T19:17:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.704+0000 7fc653000640 1 -- 172.21.15.27:0/3053143556 wait complete. 2024-03-28T19:17:16.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.708+0000 7fc653000640 1 Processor -- start 2024-03-28T19:17:16.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.708+0000 7fc653000640 1 -- start start 2024-03-28T19:17:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc653000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6340052e0 0x7fc64c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc653000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc64c0a5e50 0x7fc64c0a79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc653000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc64c072fc0 0x7fc64c1ef520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc653000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0c0fc0 con 0x7fc64c072fc0 2024-03-28T19:17:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc653000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c06e710 con 0x7fc64c0a5e50 2024-03-28T19:17:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc653000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0c2bc0 con 0x7fc6340052e0 2024-03-28T19:17:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc652600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc64c072fc0 0x7fc64c1ef520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc652600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc64c072fc0 0x7fc64c1ef520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47290/0 (socket says 172.21.15.27:47290) 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc652600640 1 -- 172.21.15.27:0/631672116 learned_addr learned my addr 172.21.15.27:0/631672116 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.710+0000 7fc651c00640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6340052e0 0x7fc64c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.711+0000 7fc651200640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc64c0a5e50 0x7fc64c0a79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.711+0000 7fc652600640 1 -- 172.21.15.27:0/631672116 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6340052e0 msgr2=0x7fc64c070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.711+0000 7fc652600640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6340052e0 0x7fc64c070770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.711+0000 7fc652600640 1 -- 172.21.15.27:0/631672116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc64c0a5e50 msgr2=0x7fc64c0a79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.711+0000 7fc652600640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc64c0a5e50 0x7fc64c0a79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.711+0000 7fc652600640 1 -- 172.21.15.27:0/631672116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c1f36f0 con 0x7fc64c072fc0 2024-03-28T19:17:16.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.711+0000 7fc652600640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc64c072fc0 0x7fc64c1ef520 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fc64400a970 tx=0x7fc64c0c2900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:16.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.712+0000 7fc649600640 1 -- 172.21.15.27:0/631672116 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644004d70 con 0x7fc64c072fc0 2024-03-28T19:17:16.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.712+0000 7fc649600640 1 -- 172.21.15.27:0/631672116 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6440017b0 con 0x7fc64c072fc0 2024-03-28T19:17:16.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.712+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c0711d0 con 0x7fc64c072fc0 2024-03-28T19:17:16.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.712+0000 7fc649600640 1 -- 172.21.15.27:0/631672116 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644001b80 con 0x7fc64c072fc0 2024-03-28T19:17:16.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.712+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c073440 con 0x7fc64c072fc0 2024-03-28T19:17:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.714+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c067dd0 con 0x7fc64c072fc0 2024-03-28T19:17:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.715+0000 7fc649600640 1 -- 172.21.15.27:0/631672116 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc64403c070 con 0x7fc64c072fc0 2024-03-28T19:17:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.717+0000 7fc649600640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62c06bbd0 0x7fc62c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.717+0000 7fc649600640 1 -- 172.21.15.27:0/631672116 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7fc6440ac650 con 0x7fc64c072fc0 2024-03-28T19:17:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.717+0000 7fc651c00640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62c06bbd0 0x7fc62c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.718+0000 7fc651c00640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62c06bbd0 0x7fc62c06e090 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc640001780 tx=0x7fc640009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:16.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:16.725+0000 7fc649600640 1 -- 172.21.15.27:0/631672116 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc644079c30 con 0x7fc64c072fc0 2024-03-28T19:17:17.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.077+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5c80ac190 con 0x7fe5ac004d40 2024-03-28T19:17:17.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.078+0000 7fe5bf400640 1 -- 172.21.15.27:0/3632230456 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+4669 (secure 0 0 0) 0x7fe5c8151030 con 0x7fe5ac004d40 2024-03-28T19:17:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:17.079 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:15.699865+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:15.909681+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:17:17.079 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 175 2024-03-28T19:17:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe5a406bc20 msgr2=0x7fe5a406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe5a406bc20 0x7fe5a406e0e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe5c81f3b70 tx=0x7fe5b0009040 comp rx=0 tx=0).stop 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 msgr2=0x7fe5c80a79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5c80a79a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800fb30 tx=0x7fe5b800d150 comp rx=0 tx=0).stop 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 shutdown_connections 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe5a406bc20 0x7fe5a406e0e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5c80a6820 0x7fe5c80a6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5c80aaf00 0x7fe5c80ab360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 --2- 172.21.15.27:0/3632230456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5ac004d40 0x7fe5c80a79a0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 >> 172.21.15.27:0/3632230456 conn(0x7fe5c806b9d0 msgr2=0x7fe5c806ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 shutdown_connections 2024-03-28T19:17:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.082+0000 7fe5ce800640 1 -- 172.21.15.27:0/3632230456 wait complete. 2024-03-28T19:17:17.107 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:17:17.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.122+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc64c0716f0 con 0x7fc64c072fc0 2024-03-28T19:17:17.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.123+0000 7fc649600640 1 -- 172.21.15.27:0/631672116 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+4669 (secure 0 0 0) 0x7fc64c0716f0 con 0x7fc64c072fc0 2024-03-28T19:17:17.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:17.124 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:15.699865+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:15.909681+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:17:17.124 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 175 2024-03-28T19:17:17.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62c06bbd0 msgr2=0x7fc62c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62c06bbd0 0x7fc62c06e090 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc640001780 tx=0x7fc640009040 comp rx=0 tx=0).stop 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc64c072fc0 msgr2=0x7fc64c1ef520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc64c072fc0 0x7fc64c1ef520 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fc64400a970 tx=0x7fc64c0c2900 comp rx=0 tx=0).stop 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 shutdown_connections 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc62c06bbd0 0x7fc62c06e090 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc64c072fc0 0x7fc64c1ef520 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc64c0a5e50 0x7fc64c0a79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 --2- 172.21.15.27:0/631672116 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6340052e0 0x7fc64c070770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 >> 172.21.15.27:0/631672116 conn(0x7fc64c06b0c0 msgr2=0x7fc64c0a9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 shutdown_connections 2024-03-28T19:17:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:17.127+0000 7fc653000640 1 -- 172.21.15.27:0/631672116 wait complete. 2024-03-28T19:17:17.152 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:17:19.108 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:19.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:19.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.353+0000 7f1fc3a00640 1 Processor -- start 2024-03-28T19:17:19.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.353+0000 7f1fc3a00640 1 -- start start 2024-03-28T19:17:19.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.354+0000 7f1fc3a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1fbc0a9cb0 0x7f1fbc0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.354+0000 7f1fc3a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fbc154020 con 0x7f1fbc150c70 2024-03-28T19:17:19.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.355+0000 7f1fc3a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fbc0c3d90 con 0x7f1fbc1534c0 2024-03-28T19:17:19.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.355+0000 7f1fc3a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc0ab4e0 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.355+0000 7f1fc1000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1fbc150c70 0x7f1fbc0c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57406/0 (socket says 172.21.15.27:57406) 2024-03-28T19:17:19.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.355+0000 7f1fc1000640 1 -- 172.21.15.27:0/2097406772 learned_addr learned my addr 172.21.15.27:0/2097406772 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.355+0000 7f1fbbe00640 1 --2- 172.21.15.27:0/2097406772 >> v2:172.21.15.155:3300/0 conn(0x7f1fbc0a9cb0 0x7f1fbc0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.355+0000 7f1fbb400640 1 -- 172.21.15.27:0/2097406772 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200018008 0 0) 0x7f1fbc0c3d90 con 0x7f1fbc1534c0 2024-03-28T19:17:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.356+0000 7f1fbb400640 1 -- 172.21.15.27:0/2097406772 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa4003520 con 0x7f1fbc1534c0 2024-03-28T19:17:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.356+0000 7f1fbb400640 1 -- 172.21.15.27:0/2097406772 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078463610 0 0) 0x7f1fbc154020 con 0x7f1fbc150c70 2024-03-28T19:17:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.356+0000 7f1fbb400640 1 -- 172.21.15.27:0/2097406772 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fbc0c3d90 con 0x7f1fbc150c70 2024-03-28T19:17:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.356+0000 7f1fbbe00640 1 -- 172.21.15.27:0/2097406772 >> v1:172.21.15.155:6789/0 conn(0x7f1fbc1534c0 legacy=0x7f1fbc153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.356+0000 7f1fbbe00640 1 -- 172.21.15.27:0/2097406772 >> v1:172.21.15.27:6789/0 conn(0x7f1fbc150c70 legacy=0x7f1fbc0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.356+0000 7f1fbbe00640 1 -- 172.21.15.27:0/2097406772 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc0ab710 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.358+0000 7f1fbbe00640 1 --2- 172.21.15.27:0/2097406772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc0c3690 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80097a0 tx=0x7f1fa802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d98da1745cb7e7e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:19.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.358+0000 7f1fbb400640 1 -- 172.21.15.27:0/2097406772 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8035020 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.358+0000 7f1fbb400640 1 -- 172.21.15.27:0/2097406772 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa8031830 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.358+0000 7f1fbb400640 1 -- 172.21.15.27:0/2097406772 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8031c00 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.359+0000 7f1fc3a00640 1 -- 172.21.15.27:0/2097406772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc0a9cb0 msgr2=0x7f1fbc0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.359+0000 7f1fc3a00640 1 --2- 172.21.15.27:0/2097406772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc0c3690 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80097a0 tx=0x7f1fa802b8d0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.360+0000 7f1fc3a00640 1 -- 172.21.15.27:0/2097406772 shutdown_connections 2024-03-28T19:17:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.360+0000 7f1fc3a00640 1 --2- 172.21.15.27:0/2097406772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc0c3690 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.360+0000 7f1fc3a00640 1 -- 172.21.15.27:0/2097406772 >> 172.21.15.27:0/2097406772 conn(0x7f1fbc06b9d0 msgr2=0x7f1fbc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.360+0000 7f1fc3a00640 1 -- 172.21.15.27:0/2097406772 shutdown_connections 2024-03-28T19:17:19.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.360+0000 7f1fc3a00640 1 -- 172.21.15.27:0/2097406772 wait complete. 2024-03-28T19:17:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.363+0000 7f1fc3a00640 1 Processor -- start 2024-03-28T19:17:19.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.364+0000 7f1fc3a00640 1 -- start start 2024-03-28T19:17:19.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.366+0000 7f1fc3a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc1ec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.366+0000 7f1fc3a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc150c70 0x7f1fbc1ea570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.366+0000 7f1fc3a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1fbc1534c0 0x7f1fbc14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.366+0000 7f1fc3a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0abcd0 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.366+0000 7f1fc3a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0ac1f0 con 0x7f1fbc150c70 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.366+0000 7f1fc3a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0c1810 con 0x7f1fbc1534c0 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc1ec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1fbc1534c0 0x7f1fbc14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1fbc1534c0 0x7f1fbc14e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45020/0 (socket says 172.21.15.27:45020) 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc1ec8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47304/0 (socket says 172.21.15.27:47304) 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1a00640 1 -- 172.21.15.27:0/3048237186 learned_addr learned my addr 172.21.15.27:0/3048237186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fbbe00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc150c70 0x7f1fbc1ea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1000640 1 -- 172.21.15.27:0/3048237186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1fbc1534c0 msgr2=0x7f1fbc14e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1000640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1fbc1534c0 0x7f1fbc14e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1000640 1 -- 172.21.15.27:0/3048237186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc150c70 msgr2=0x7f1fbc1ea570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.367+0000 7f1fc1000640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc150c70 0x7f1fbc1ea570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.368+0000 7f1fc1000640 1 -- 172.21.15.27:0/3048237186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc1ed1b0 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.368+0000 7f1fc1a00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1fbc1534c0 0x7f1fbc14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:17:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.368+0000 7f1fc1000640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc1ec8e0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000c400 tx=0x7f1fb000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.368+0000 7f1fb8c00640 1 -- 172.21.15.27:0/3048237186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0017070 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.369+0000 7f1fb8c00640 1 -- 172.21.15.27:0/3048237186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb0005ce0 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.369+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc155e40 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.369+0000 7f1fb8c00640 1 -- 172.21.15.27:0/3048237186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb00072f0 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.369+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc156f20 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.371+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc04f400 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.372+0000 7f1fb8c00640 1 -- 172.21.15.27:0/3048237186 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1fb001a070 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.373+0000 7f1fb8c00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1f9c06bbd0 0x7f1f9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.374+0000 7f1fb8c00640 1 -- 172.21.15.27:0/3048237186 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f1fb008ab90 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.374+0000 7f1fbbe00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1f9c06bbd0 0x7f1f9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.377+0000 7f1fbbe00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1f9c06bbd0 0x7f1f9c06e090 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8036000 tx=0x7f1fa8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.381+0000 7f1fb8c00640 1 -- 172.21.15.27:0/3048237186 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1fb0058200 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.403+0000 7f4751800640 1 Processor -- start 2024-03-28T19:17:19.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.404+0000 7f4751800640 1 -- start start 2024-03-28T19:17:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.405+0000 7f4751800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f474c150c70 0x7f474c0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.405+0000 7f4751800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f474c0c3690 0x7f474c0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.405+0000 7f4751800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c0c3af0 con 0x7f474c0c6660 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.405+0000 7f4751800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474c0ab4e0 con 0x7f474c150c70 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.405+0000 7f4751800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474c0ac1f0 con 0x7f474c0c3690 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f474c150c70 0x7f474c0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f474c150c70 0x7f474c0a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47318/0 (socket says 172.21.15.27:47318) 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474aa00640 1 -- 172.21.15.27:0/3584355181 learned_addr learned my addr 172.21.15.27:0/3584355181 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474a000640 1 --2- 172.21.15.27:0/3584355181 >> v2:172.21.15.155:3300/0 conn(0x7f474c0c3690 0x7f474c0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474aa00640 1 -- 172.21.15.27:0/3584355181 >> v2:172.21.15.155:3300/0 conn(0x7f474c0c3690 msgr2=0x7f474c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474aa00640 1 --2- 172.21.15.27:0/3584355181 >> v2:172.21.15.155:3300/0 conn(0x7f474c0c3690 0x7f474c0c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474aa00640 1 -- 172.21.15.27:0/3584355181 >> v1:172.21.15.155:6789/0 conn(0x7f474c0c6660 legacy=0x7f474c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:19.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474aa00640 1 -- 172.21.15.27:0/3584355181 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0c3d90 con 0x7f474c150c70 2024-03-28T19:17:19.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.406+0000 7f474a000640 1 --2- 172.21.15.27:0/3584355181 >> v2:172.21.15.155:3300/0 conn(0x7f474c0c3690 0x7f474c0c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:17:19.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.408+0000 7f474aa00640 1 --2- 172.21.15.27:0/3584355181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c0a9cb0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f474c0c4180 tx=0x7f473c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45018683ba48c23 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.408+0000 7f4749600640 1 -- 172.21.15.27:0/3584355181 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c02e070 con 0x7f474c150c70 2024-03-28T19:17:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.408+0000 7f4749600640 1 -- 172.21.15.27:0/3584355181 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c037070 con 0x7f474c150c70 2024-03-28T19:17:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.408+0000 7f4749600640 1 -- 172.21.15.27:0/3584355181 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c031a00 con 0x7f474c150c70 2024-03-28T19:17:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.409+0000 7f4751800640 1 -- 172.21.15.27:0/3584355181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 msgr2=0x7f474c0a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.409+0000 7f4751800640 1 --2- 172.21.15.27:0/3584355181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c0a9cb0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f474c0c4180 tx=0x7f473c02b880 comp rx=0 tx=0).stop 2024-03-28T19:17:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.410+0000 7f4751800640 1 -- 172.21.15.27:0/3584355181 shutdown_connections 2024-03-28T19:17:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.410+0000 7f4751800640 1 --2- 172.21.15.27:0/3584355181 >> v2:172.21.15.155:3300/0 conn(0x7f474c0c3690 0x7f474c0c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.410+0000 7f4751800640 1 --2- 172.21.15.27:0/3584355181 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c0a9cb0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.410+0000 7f4751800640 1 -- 172.21.15.27:0/3584355181 >> 172.21.15.27:0/3584355181 conn(0x7f474c06b9d0 msgr2=0x7f474c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.410+0000 7f4751800640 1 -- 172.21.15.27:0/3584355181 shutdown_connections 2024-03-28T19:17:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.410+0000 7f4751800640 1 -- 172.21.15.27:0/3584355181 wait complete. 2024-03-28T19:17:19.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.412+0000 7f4751800640 1 Processor -- start 2024-03-28T19:17:19.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.413+0000 7f4751800640 1 -- start start 2024-03-28T19:17:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.414+0000 7f4751800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f474c0c3690 0x7f474c1e3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.414+0000 7f4751800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f474c0c6660 0x7f474c1f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.414+0000 7f4751800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c1f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.414+0000 7f4751800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0abcd0 con 0x7f474c150c70 2024-03-28T19:17:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.414+0000 7f4751800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0ab9f0 con 0x7f474c0c3690 2024-03-28T19:17:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.414+0000 7f4751800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0c1810 con 0x7f474c0c6660 2024-03-28T19:17:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.414+0000 7f474b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c1f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.415+0000 7f474b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c1f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47334/0 (socket says 172.21.15.27:47334) 2024-03-28T19:17:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.415+0000 7f474b400640 1 -- 172.21.15.27:0/191733990 learned_addr learned my addr 172.21.15.27:0/191733990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:19.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.415+0000 7f474aa00640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f474c0c3690 0x7f474c1e3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.415+0000 7f474a000640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f474c0c6660 0x7f474c1f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.415+0000 7f474aa00640 1 -- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f474c0c6660 msgr2=0x7f474c1f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.415+0000 7f474aa00640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f474c0c6660 0x7f474c1f2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.415+0000 7f474aa00640 1 -- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 msgr2=0x7f474c1f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.416+0000 7f474aa00640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c1f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.416+0000 7f474aa00640 1 -- 172.21.15.27:0/191733990 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0a58f0 con 0x7f474c0c3690 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.416+0000 7f474a000640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f474c0c6660 0x7f474c1f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.416+0000 7f474b400640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c1f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.417+0000 7f474aa00640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f474c0c3690 0x7f474c1e3390 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f474c0c43e0 tx=0x7f473c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.417+0000 7f4742a00640 1 -- 172.21.15.27:0/191733990 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c032880 con 0x7f474c0c3690 2024-03-28T19:17:19.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.417+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c0a31b0 con 0x7f474c0c3690 2024-03-28T19:17:19.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.417+0000 7f4742a00640 1 -- 172.21.15.27:0/191733990 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c032a20 con 0x7f474c0c3690 2024-03-28T19:17:19.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.417+0000 7f4742a00640 1 -- 172.21.15.27:0/191733990 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c032e10 con 0x7f474c0c3690 2024-03-28T19:17:19.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.417+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c0a7220 con 0x7f474c0c3690 2024-03-28T19:17:19.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.419+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474c04f400 con 0x7f474c0c3690 2024-03-28T19:17:19.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.420+0000 7f4742a00640 1 -- 172.21.15.27:0/191733990 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f473c04c060 con 0x7f474c0c3690 2024-03-28T19:17:19.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.421+0000 7f4742a00640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f472c06bbd0 0x7f472c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:19.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.421+0000 7f4742a00640 1 -- 172.21.15.27:0/191733990 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f473c0acfb0 con 0x7f474c0c3690 2024-03-28T19:17:19.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.422+0000 7f474a000640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f472c06bbd0 0x7f472c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:19.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.423+0000 7f474a000640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f472c06bbd0 0x7f472c06e090 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f474c150620 tx=0x7f4738008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:19.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.428+0000 7f4742a00640 1 -- 172.21.15.27:0/191733990 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f473c040070 con 0x7f474c0c3690 2024-03-28T19:17:19.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.747+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fbc0c3ad0 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.747+0000 7f1fb8c00640 1 -- 172.21.15.27:0/3048237186 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+4677 (secure 0 0 0) 0x7f1fb005c2a0 con 0x7f1fbc0a9cb0 2024-03-28T19:17:19.747 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:19.747 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:17:19.748 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 177 2024-03-28T19:17:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1f9c06bbd0 msgr2=0x7f1f9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1f9c06bbd0 0x7f1f9c06e090 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8036000 tx=0x7f1fa8008030 comp rx=0 tx=0).stop 2024-03-28T19:17:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fbc0a9cb0 msgr2=0x7f1fbc1ec8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc1ec8e0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000c400 tx=0x7f1fb000c8c0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 shutdown_connections 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1f9c06bbd0 0x7f1f9c06e090 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1fbc1534c0 0x7f1fbc14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fbc150c70 0x7f1fbc1ea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 --2- 172.21.15.27:0/3048237186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fbc0a9cb0 0x7f1fbc1ec8e0 secure :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000c400 tx=0x7f1fb000c8c0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.750+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 >> 172.21.15.27:0/3048237186 conn(0x7f1fbc06b9d0 msgr2=0x7f1fbc153910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.751+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 shutdown_connections 2024-03-28T19:17:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.751+0000 7f1fc3a00640 1 -- 172.21.15.27:0/3048237186 wait complete. 2024-03-28T19:17:19.771 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6077 2024-03-28T19:17:19.771 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.d 2024-03-28T19:17:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.786+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f474c1f3710 con 0x7f474c0c3690 2024-03-28T19:17:19.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.788+0000 7f4742a00640 1 -- 172.21.15.27:0/191733990 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+4677 (secure 0 0 0) 0x7f473c07a620 con 0x7f474c0c3690 2024-03-28T19:17:19.787 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:19.787 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:17:19.788 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 177 2024-03-28T19:17:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f472c06bbd0 msgr2=0x7f472c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f472c06bbd0 0x7f472c06e090 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f474c150620 tx=0x7f4738008040 comp rx=0 tx=0).stop 2024-03-28T19:17:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f474c0c3690 msgr2=0x7f474c1e3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f474c0c3690 0x7f474c1e3390 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f474c0c43e0 tx=0x7f473c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 shutdown_connections 2024-03-28T19:17:19.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f472c06bbd0 0x7f472c06e090 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f474c150c70 0x7f474c1f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f474c0c6660 0x7f474c1f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 --2- 172.21.15.27:0/191733990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f474c0c3690 0x7f474c1e3390 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.791+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 >> 172.21.15.27:0/191733990 conn(0x7f474c06b9d0 msgr2=0x7f474c06d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.792+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 shutdown_connections 2024-03-28T19:17:19.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:19.792+0000 7f4751800640 1 -- 172.21.15.27:0/191733990 wait complete. 2024-03-28T19:17:19.812 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=7575 2024-03-28T19:17:19.812 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.a 2024-03-28T19:17:20.520 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:17:20.521+0000 7fbe1a800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:17:22.870 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:17:22.871+0000 7f8de6c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:17:25.773 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-03-28T19:17:25.773 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T19:17:25.773 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T19:17:25.780 INFO:tasks.ceph.mds.d:Started 2024-03-28T19:17:25.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:25.966 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T19:17:26.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.032+0000 7f5c16e00640 1 Processor -- start 2024-03-28T19:17:26.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.033+0000 7f5c16e00640 1 -- start start 2024-03-28T19:17:26.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.034+0000 7f5c16e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5c101591a0 0x7f5c10159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:26.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.034+0000 7f5c16e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100c9bd0 con 0x7f5c100c3e70 2024-03-28T19:17:26.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.034+0000 7f5c16e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c100c7570 con 0x7f5c100a7ae0 2024-03-28T19:17:26.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.034+0000 7f5c16e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c100c50c0 con 0x7f5c101591a0 2024-03-28T19:17:26.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.035+0000 7f5c0fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5c101591a0 0x7f5c10159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:26.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.035+0000 7f5c14e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5c100c3e70 0x7f5c100c4280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57408/0 (socket says 172.21.15.27:57408) 2024-03-28T19:17:26.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.035+0000 7f5c14e00640 1 -- 172.21.15.27:0/3490895120 learned_addr learned my addr 172.21.15.27:0/3490895120 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:17:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.035+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3490895120 >> v1:172.21.15.155:6789/0 conn(0x7f5c100a7ae0 legacy=0x7f5c100c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.035+0000 7f5c0f400640 1 -- 172.21.15.27:0/3490895120 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504095802 0 0) 0x7f5c100c9bd0 con 0x7f5c100c3e70 2024-03-28T19:17:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.035+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3490895120 >> v1:172.21.15.27:6789/0 conn(0x7f5c100c3e70 legacy=0x7f5c100c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.035+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3490895120 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c100c53a0 con 0x7f5c101591a0 2024-03-28T19:17:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.037+0000 7f5c0fe00640 1 --2- 172.21.15.27:0/3490895120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c101591a0 0x7f5c10159580 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009720 tx=0x7f5c0002b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffa27cb18acaf9cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.038+0000 7f5c0f400640 1 -- 172.21.15.27:0/3490895120 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00037070 con 0x7f5c101591a0 2024-03-28T19:17:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.038+0000 7f5c0f400640 1 -- 172.21.15.27:0/3490895120 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c00030710 con 0x7f5c101591a0 2024-03-28T19:17:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.038+0000 7f5c0f400640 1 -- 172.21.15.27:0/3490895120 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00030ae0 con 0x7f5c101591a0 2024-03-28T19:17:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.039+0000 7f5c16e00640 1 -- 172.21.15.27:0/3490895120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c101591a0 msgr2=0x7f5c10159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.039+0000 7f5c16e00640 1 --2- 172.21.15.27:0/3490895120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c101591a0 0x7f5c10159580 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009720 tx=0x7f5c0002b730 comp rx=0 tx=0).stop 2024-03-28T19:17:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.039+0000 7f5c16e00640 1 -- 172.21.15.27:0/3490895120 shutdown_connections 2024-03-28T19:17:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.039+0000 7f5c16e00640 1 --2- 172.21.15.27:0/3490895120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c101591a0 0x7f5c10159580 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.039+0000 7f5c16e00640 1 -- 172.21.15.27:0/3490895120 >> 172.21.15.27:0/3490895120 conn(0x7f5c100aaab0 msgr2=0x7f5c100aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.039+0000 7f5c16e00640 1 -- 172.21.15.27:0/3490895120 shutdown_connections 2024-03-28T19:17:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.039+0000 7f5c16e00640 1 -- 172.21.15.27:0/3490895120 wait complete. 2024-03-28T19:17:26.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.042+0000 7f5c16e00640 1 Processor -- start 2024-03-28T19:17:26.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.042+0000 7f5c16e00640 1 -- start start 2024-03-28T19:17:26.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.043+0000 7f5c16e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c100a7ae0 0x7f5c10153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:26.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c16e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c100c3e70 0x7f5c1006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:26.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c16e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c101591a0 0x7f5c1006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:26.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c16e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100c79d0 con 0x7f5c101591a0 2024-03-28T19:17:26.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c16e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100c7850 con 0x7f5c100a7ae0 2024-03-28T19:17:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c16e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c100c4990 con 0x7f5c100c3e70 2024-03-28T19:17:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c14e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c101591a0 0x7f5c1006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c0fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c100a7ae0 0x7f5c10153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c07e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c100c3e70 0x7f5c1006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c0fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c100a7ae0 0x7f5c10153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:57314/0 (socket says 172.21.15.27:57314) 2024-03-28T19:17:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c14e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c101591a0 0x7f5c1006f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47338/0 (socket says 172.21.15.27:47338) 2024-03-28T19:17:26.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.044+0000 7f5c0fe00640 1 -- 172.21.15.27:0/2830801512 learned_addr learned my addr 172.21.15.27:0/2830801512 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.045+0000 7f5c14e00640 1 -- 172.21.15.27:0/2830801512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c100c3e70 msgr2=0x7f5c1006ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.045+0000 7f5c14e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c100c3e70 0x7f5c1006ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.045+0000 7f5c14e00640 1 -- 172.21.15.27:0/2830801512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c100a7ae0 msgr2=0x7f5c10153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.045+0000 7f5c14e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c100a7ae0 0x7f5c10153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.045+0000 7f5c14e00640 1 -- 172.21.15.27:0/2830801512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c101eec60 con 0x7f5c101591a0 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.045+0000 7f5c14e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c101591a0 0x7f5c1006f1b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800c700 tx=0x7f5c0800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.045+0000 7f5c0cc00640 1 -- 172.21.15.27:0/2830801512 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08018070 con 0x7f5c101591a0 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.046+0000 7f5c0cc00640 1 -- 172.21.15.27:0/2830801512 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0800ce10 con 0x7f5c101591a0 2024-03-28T19:17:26.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.046+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c101eef40 con 0x7f5c101591a0 2024-03-28T19:17:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.046+0000 7f5c0cc00640 1 -- 172.21.15.27:0/2830801512 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0800a880 con 0x7f5c101591a0 2024-03-28T19:17:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.046+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c100ad930 con 0x7f5c101591a0 2024-03-28T19:17:26.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.048+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c100adb30 con 0x7f5c101591a0 2024-03-28T19:17:26.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.049+0000 7f5c0cc00640 1 -- 172.21.15.27:0/2830801512 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5c0800aa20 con 0x7f5c101591a0 2024-03-28T19:17:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.050+0000 7f5c0cc00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bec06bc00 0x7f5bec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.050+0000 7f5c0cc00640 1 -- 172.21.15.27:0/2830801512 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f5c0808ac90 con 0x7f5c101591a0 2024-03-28T19:17:26.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.050+0000 7f5c0fe00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bec06bc00 0x7f5bec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:26.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.051+0000 7f5c0fe00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bec06bc00 0x7f5bec06e0c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5c1014e610 tx=0x7f5c00008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:26.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.057+0000 7f5c0cc00640 1 -- 172.21.15.27:0/2830801512 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5c08058300 con 0x7f5c101591a0 2024-03-28T19:17:26.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.426+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c1004f400 con 0x7f5c101591a0 2024-03-28T19:17:26.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.427+0000 7f5c0cc00640 1 -- 172.21.15.27:0/2830801512 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+4677 (secure 0 0 0) 0x7f5c0805c3a0 con 0x7f5c101591a0 2024-03-28T19:17:26.427 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:26.428 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:17:26.428 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 177 2024-03-28T19:17:26.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bec06bc00 msgr2=0x7f5bec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:26.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bec06bc00 0x7f5bec06e0c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5c1014e610 tx=0x7f5c00008030 comp rx=0 tx=0).stop 2024-03-28T19:17:26.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c101591a0 msgr2=0x7f5c1006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:26.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c101591a0 0x7f5c1006f1b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800c700 tx=0x7f5c0800cbc0 comp rx=0 tx=0).stop 2024-03-28T19:17:26.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 shutdown_connections 2024-03-28T19:17:26.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bec06bc00 0x7f5bec06e0c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:26.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c101591a0 0x7f5c1006f1b0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:26.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5c100c3e70 0x7f5c1006ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:26.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 --2- 172.21.15.27:0/2830801512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5c100a7ae0 0x7f5c10153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:26.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.430+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 >> 172.21.15.27:0/2830801512 conn(0x7f5c100aaab0 msgr2=0x7f5c10158530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:26.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.431+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 shutdown_connections 2024-03-28T19:17:26.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:26.431+0000 7f5c16e00640 1 -- 172.21.15.27:0/2830801512 wait complete. 2024-03-28T19:17:26.451 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-03-28T19:17:26.813 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-03-28T19:17:26.813 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:17:26.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:17:26.820 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:17:26.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:27.034 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:17:27.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.060+0000 7fa385800640 1 Processor -- start 2024-03-28T19:17:27.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.060+0000 7fa385800640 1 -- start start 2024-03-28T19:17:27.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.061+0000 7fa385800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3800a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.061+0000 7fa385800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3801534c0 0x7fa380153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.061+0000 7fa385800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3800c3af0 con 0x7fa3800c3690 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.061+0000 7fa385800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3800ab4e0 con 0x7fa3801534c0 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.061+0000 7fa385800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800ac1f0 con 0x7fa380150c70 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3801534c0 0x7fa380153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3800a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3801534c0 0x7fa380153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35886/0 (socket says 172.21.15.27:35886) 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37f400640 1 -- 172.21.15.27:0/4094178597 learned_addr learned my addr 172.21.15.27:0/4094178597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:27.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3800a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:51260/0 (socket says 172.21.15.27:51260) 2024-03-28T19:17:27.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37f400640 1 -- 172.21.15.27:0/4094178597 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 msgr2=0x7fa3800a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:27.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37f400640 1 --2- 172.21.15.27:0/4094178597 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3800a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37f400640 1 -- 172.21.15.27:0/4094178597 >> v1:172.21.15.155:6789/0 conn(0x7fa3800c3690 legacy=0x7fa3800c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:27.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.062+0000 7fa37f400640 1 -- 172.21.15.27:0/4094178597 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3800c3d90 con 0x7fa3801534c0 2024-03-28T19:17:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.064+0000 7fa37f400640 1 --2- 172.21.15.27:0/4094178597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 0x7fa380153950 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fa378009e50 tx=0x7fa37802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75eae61f49d579c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.064+0000 7fa37d600640 1 -- 172.21.15.27:0/4094178597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37802ca70 con 0x7fa3801534c0 2024-03-28T19:17:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.064+0000 7fa37d600640 1 -- 172.21.15.27:0/4094178597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37802cc10 con 0x7fa3801534c0 2024-03-28T19:17:27.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.064+0000 7fa37d600640 1 -- 172.21.15.27:0/4094178597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3780343f0 con 0x7fa3801534c0 2024-03-28T19:17:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.065+0000 7fa385800640 1 -- 172.21.15.27:0/4094178597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 msgr2=0x7fa380153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.065+0000 7fa385800640 1 --2- 172.21.15.27:0/4094178597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 0x7fa380153950 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fa378009e50 tx=0x7fa37802bed0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.065+0000 7fa385800640 1 -- 172.21.15.27:0/4094178597 shutdown_connections 2024-03-28T19:17:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.065+0000 7fa385800640 1 --2- 172.21.15.27:0/4094178597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 0x7fa380153950 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.065+0000 7fa385800640 1 --2- 172.21.15.27:0/4094178597 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3800a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.065+0000 7fa385800640 1 -- 172.21.15.27:0/4094178597 >> 172.21.15.27:0/4094178597 conn(0x7fa38006b9d0 msgr2=0x7fa3800c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.065+0000 7fa385800640 1 -- 172.21.15.27:0/4094178597 shutdown_connections 2024-03-28T19:17:27.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.065+0000 7fa385800640 1 -- 172.21.15.27:0/4094178597 wait complete. 2024-03-28T19:17:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.067+0000 7fa385800640 1 Processor -- start 2024-03-28T19:17:27.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.068+0000 7fa385800640 1 -- start start 2024-03-28T19:17:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa385800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3800c3690 0x7fa3800c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa385800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3801eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa385800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 0x7fa3801ef150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa385800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800abcd0 con 0x7fa3801534c0 2024-03-28T19:17:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa385800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800ab9f0 con 0x7fa3800c3690 2024-03-28T19:17:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa385800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800c1810 con 0x7fa380150c70 2024-03-28T19:17:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3800c3690 0x7fa3800c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa37f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 0x7fa3801ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa37e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3801eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3800c3690 0x7fa3800c1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:49670/0 (socket says 172.21.15.27:49670) 2024-03-28T19:17:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.069+0000 7fa37ea00640 1 -- 172.21.15.27:0/2907995395 learned_addr learned my addr 172.21.15.27:0/2907995395 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa37f400640 1 -- 172.21.15.27:0/2907995395 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 msgr2=0x7fa3801eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:27.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa37f400640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3801eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa37f400640 1 -- 172.21.15.27:0/2907995395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3800c3690 msgr2=0x7fa3800c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa37f400640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3800c3690 0x7fa3800c1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa37f400640 1 -- 172.21.15.27:0/2907995395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38014dc30 con 0x7fa3801534c0 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa37ea00640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3800c3690 0x7fa3800c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa37f400640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 0x7fa3801ef150 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fa378032000 tx=0x7fa378004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa376a00640 1 -- 172.21.15.27:0/2907995395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378043020 con 0x7fa3801534c0 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.070+0000 7fa376a00640 1 -- 172.21.15.27:0/2907995395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37803e450 con 0x7fa3801534c0 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.071+0000 7fa376a00640 1 -- 172.21.15.27:0/2907995395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378009510 con 0x7fa3801534c0 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.071+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3800c09a0 con 0x7fa3801534c0 2024-03-28T19:17:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.071+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38014e7a0 con 0x7fa3801534c0 2024-03-28T19:17:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.072+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3800c0790 con 0x7fa3801534c0 2024-03-28T19:17:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.073+0000 7fa376a00640 1 -- 172.21.15.27:0/2907995395 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa378046020 con 0x7fa3801534c0 2024-03-28T19:17:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.074+0000 7fa376a00640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa36006bc20 0x7fa36006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.075+0000 7fa376a00640 1 -- 172.21.15.27:0/2907995395 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7fa3780addc0 con 0x7fa3801534c0 2024-03-28T19:17:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.075+0000 7fa37ea00640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa36006bc20 0x7fa36006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:27.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.076+0000 7fa37ea00640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa36006bc20 0x7fa36006e0e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa3801e8e30 tx=0x7fa370009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:27.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.081+0000 7fa376a00640 1 -- 172.21.15.27:0/2907995395 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa37804d030 con 0x7fa3801534c0 2024-03-28T19:17:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.442+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa38004f400 con 0x7fa3801534c0 2024-03-28T19:17:27.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.443+0000 7fa376a00640 1 -- 172.21.15.27:0/2907995395 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+5471 (secure 0 0 0) 0x7fa37807b510 con 0x7fa3801534c0 2024-03-28T19:17:27.442 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:27.443 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:17:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 178 2024-03-28T19:17:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa36006bc20 msgr2=0x7fa36006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa36006bc20 0x7fa36006e0e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa3801e8e30 tx=0x7fa370009190 comp rx=0 tx=0).stop 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 msgr2=0x7fa3801ef150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 0x7fa3801ef150 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fa378032000 tx=0x7fa378004a40 comp rx=0 tx=0).stop 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 shutdown_connections 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa36006bc20 0x7fa36006e0e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3801534c0 0x7fa3801ef150 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa380150c70 0x7fa3801eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 --2- 172.21.15.27:0/2907995395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3800c3690 0x7fa3800c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 >> 172.21.15.27:0/2907995395 conn(0x7fa38006b9d0 msgr2=0x7fa38014ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 shutdown_connections 2024-03-28T19:17:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:27.446+0000 7fa385800640 1 -- 172.21.15.27:0/2907995395 wait complete. 2024-03-28T19:17:27.467 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:17:28.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:28.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.691+0000 7f26ce400640 1 Processor -- start 2024-03-28T19:17:28.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.691+0000 7f26ce400640 1 -- start start 2024-03-28T19:17:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26ce400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c80c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26ce400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f26c81534c0 0x7f26c8153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26ce400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c80c3d90 con 0x7f26c8150c70 2024-03-28T19:17:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26ce400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c80ab4e0 con 0x7f26c81534c0 2024-03-28T19:17:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26ce400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80ac1f0 con 0x7f26c80a9cb0 2024-03-28T19:17:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26c6a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c80c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26c6a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c80c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:51280/0 (socket says 172.21.15.27:51280) 2024-03-28T19:17:28.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26c6a00640 1 -- 172.21.15.27:0/3566332717 learned_addr learned my addr 172.21.15.27:0/3566332717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:28.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.693+0000 7f26c7e00640 1 --2- 172.21.15.27:0/3566332717 >> v2:172.21.15.155:3300/0 conn(0x7f26c81534c0 0x7f26c8153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.694+0000 7f26c6a00640 1 -- 172.21.15.27:0/3566332717 >> v2:172.21.15.155:3300/0 conn(0x7f26c81534c0 msgr2=0x7f26c8153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.694+0000 7f26c6a00640 1 --2- 172.21.15.27:0/3566332717 >> v2:172.21.15.155:3300/0 conn(0x7f26c81534c0 0x7f26c8153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.694+0000 7f26c6a00640 1 -- 172.21.15.27:0/3566332717 >> v1:172.21.15.155:6789/0 conn(0x7f26c8150c70 legacy=0x7f26c80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.694+0000 7f26c6a00640 1 -- 172.21.15.27:0/3566332717 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c80ab710 con 0x7f26c80a9cb0 2024-03-28T19:17:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.694+0000 7f26c7e00640 1 --2- 172.21.15.27:0/3566332717 >> v2:172.21.15.155:3300/0 conn(0x7f26c81534c0 0x7f26c8153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:17:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.695+0000 7f26c6a00640 1 --2- 172.21.15.27:0/3566332717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c80c3690 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009870 tx=0x7f26b402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92c4a4bf8d27fff2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:28.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.696+0000 7f26c6000640 1 -- 172.21.15.27:0/3566332717 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4035020 con 0x7f26c80a9cb0 2024-03-28T19:17:28.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.696+0000 7f26c6000640 1 -- 172.21.15.27:0/3566332717 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b4031640 con 0x7f26c80a9cb0 2024-03-28T19:17:28.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.696+0000 7f26c6000640 1 -- 172.21.15.27:0/3566332717 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b402d040 con 0x7f26c80a9cb0 2024-03-28T19:17:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.697+0000 7f26ce400640 1 -- 172.21.15.27:0/3566332717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 msgr2=0x7f26c80c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.697+0000 7f26ce400640 1 --2- 172.21.15.27:0/3566332717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c80c3690 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009870 tx=0x7f26b402b8f0 comp rx=0 tx=0).stop 2024-03-28T19:17:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.697+0000 7f26ce400640 1 -- 172.21.15.27:0/3566332717 shutdown_connections 2024-03-28T19:17:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.697+0000 7f26ce400640 1 --2- 172.21.15.27:0/3566332717 >> v2:172.21.15.155:3300/0 conn(0x7f26c81534c0 0x7f26c8153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.697+0000 7f26ce400640 1 --2- 172.21.15.27:0/3566332717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c80c3690 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.697+0000 7f26ce400640 1 -- 172.21.15.27:0/3566332717 >> 172.21.15.27:0/3566332717 conn(0x7f26c806b9d0 msgr2=0x7f26c80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.697+0000 7f26ce400640 1 -- 172.21.15.27:0/3566332717 shutdown_connections 2024-03-28T19:17:28.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.697+0000 7f26ce400640 1 -- 172.21.15.27:0/3566332717 wait complete. 2024-03-28T19:17:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.699+0000 7f26ce400640 1 Processor -- start 2024-03-28T19:17:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.699+0000 7f26ce400640 1 -- start start 2024-03-28T19:17:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.700+0000 7f26ce400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.700+0000 7f26ce400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26c8150c70 0x7f26c8155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.700+0000 7f26ce400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c81534c0 0x7f26c80c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.700+0000 7f26ce400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80c1810 con 0x7f26c81534c0 2024-03-28T19:17:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.700+0000 7f26ce400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80abcd0 con 0x7f26c8150c70 2024-03-28T19:17:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.700+0000 7f26ce400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c80c6830 con 0x7f26c80a9cb0 2024-03-28T19:17:28.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26c8150c70 0x7f26c8155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c7400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26c8150c70 0x7f26c8155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:49684/0 (socket says 172.21.15.27:49684) 2024-03-28T19:17:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c6a00640 1 -- 172.21.15.27:0/2897800306 learned_addr learned my addr 172.21.15.27:0/2897800306 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c7e00640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c81534c0 0x7f26c80c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c7400640 1 -- 172.21.15.27:0/2897800306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26c8150c70 msgr2=0x7f26c8155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c7400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26c8150c70 0x7f26c8155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c7400640 1 -- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c81534c0 msgr2=0x7f26c80c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:28.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c7400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c81534c0 0x7f26c80c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.701+0000 7f26c7400640 1 -- 172.21.15.27:0/2897800306 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c806cee0 con 0x7f26c80a9cb0 2024-03-28T19:17:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.702+0000 7f26c6a00640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26c8150c70 0x7f26c8155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:17:28.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.702+0000 7f26c7400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c806d6a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c340 tx=0x7f26b800c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.702+0000 7f26bf400640 1 -- 172.21.15.27:0/2897800306 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b80052d0 con 0x7f26c80a9cb0 2024-03-28T19:17:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.702+0000 7f26bf400640 1 -- 172.21.15.27:0/2897800306 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b8005490 con 0x7f26c80a9cb0 2024-03-28T19:17:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.702+0000 7f26bf400640 1 -- 172.21.15.27:0/2897800306 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8007260 con 0x7f26c80a9cb0 2024-03-28T19:17:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.703+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c81ebfd0 con 0x7f26c80a9cb0 2024-03-28T19:17:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.703+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c806d180 con 0x7f26c80a9cb0 2024-03-28T19:17:28.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.704+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c8151f90 con 0x7f26c80a9cb0 2024-03-28T19:17:28.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.706+0000 7f26bf400640 1 -- 172.21.15.27:0/2897800306 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f26b8007400 con 0x7f26c80a9cb0 2024-03-28T19:17:28.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.707+0000 7f26bf400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26a806bc20 0x7f26a806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:28.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.707+0000 7f26bf400640 1 -- 172.21.15.27:0/2897800306 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f26b808ab90 con 0x7f26c80a9cb0 2024-03-28T19:17:28.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.708+0000 7f26c6a00640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26a806bc20 0x7f26a806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:28.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.709+0000 7f26c6a00640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26a806bc20 0x7f26a806e0e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f26c80c0a70 tx=0x7f26b403c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:28.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:28.714+0000 7f26bf400640 1 -- 172.21.15.27:0/2897800306 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f26c8151f90 con 0x7f26c80a9cb0 2024-03-28T19:17:29.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.047+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26c804f400 con 0x7f26c80a9cb0 2024-03-28T19:17:29.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.047+0000 7f26bf400640 1 -- 172.21.15.27:0/2897800306 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f26b805e020 con 0x7f26c80a9cb0 2024-03-28T19:17:29.047 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:29.047 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:17:29.047 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:17:29.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.050+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26a806bc20 msgr2=0x7f26a806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:29.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.050+0000 7f26ce400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26a806bc20 0x7f26a806e0e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f26c80c0a70 tx=0x7f26b403c040 comp rx=0 tx=0).stop 2024-03-28T19:17:29.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.050+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 msgr2=0x7f26c806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:29.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.050+0000 7f26ce400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c806d6a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c340 tx=0x7f26b800c740 comp rx=0 tx=0).stop 2024-03-28T19:17:29.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.051+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 shutdown_connections 2024-03-28T19:17:29.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.051+0000 7f26ce400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26a806bc20 0x7f26a806e0e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.051+0000 7f26ce400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26c81534c0 0x7f26c80c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.051+0000 7f26ce400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26c8150c70 0x7f26c8155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.051+0000 7f26ce400640 1 --2- 172.21.15.27:0/2897800306 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26c80a9cb0 0x7f26c806d6a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.051+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 >> 172.21.15.27:0/2897800306 conn(0x7f26c806b9d0 msgr2=0x7f26c80c2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.051+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 shutdown_connections 2024-03-28T19:17:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.051+0000 7f26ce400640 1 -- 172.21.15.27:0/2897800306 wait complete. 2024-03-28T19:17:29.072 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-03-28T19:17:29.072 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2024-03-28T19:17:29.468 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:17:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.709+0000 7f14ca800640 1 Processor -- start 2024-03-28T19:17:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.709+0000 7f14ca800640 1 -- start start 2024-03-28T19:17:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.710+0000 7f14ca800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c58c0 0x7f14c40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.710+0000 7f14ca800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14c40c8ed0 0x7f14c41555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.710+0000 7f14ca800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c4156230 con 0x7f14c40be5d0 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.710+0000 7f14ca800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c4158bd0 con 0x7f14c40c8ed0 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.710+0000 7f14ca800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c4158eb0 con 0x7f14c40c58c0 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14c40c8ed0 0x7f14c41555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14c40c8ed0 0x7f14c41555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35900/0 (socket says 172.21.15.27:35900) 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c8a00640 1 -- 172.21.15.27:0/3036924731 learned_addr learned my addr 172.21.15.27:0/3036924731 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c9e00640 1 --1- 172.21.15.27:0/3036924731 >> v1:172.21.15.27:6789/0 conn(0x7f14c40be5d0 0x7f14c4155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50914/0 (socket says 172.21.15.27:50914) 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c9400640 1 --2- 172.21.15.27:0/3036924731 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c58c0 0x7f14c40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c8a00640 1 -- 172.21.15.27:0/3036924731 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c58c0 msgr2=0x7f14c40c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c8a00640 1 --2- 172.21.15.27:0/3036924731 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c58c0 0x7f14c40c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c8a00640 1 -- 172.21.15.27:0/3036924731 >> v1:172.21.15.27:6789/0 conn(0x7f14c40be5d0 legacy=0x7f14c4155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:29.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c8a00640 1 -- 172.21.15.27:0/3036924731 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c4159190 con 0x7f14c40c8ed0 2024-03-28T19:17:29.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.711+0000 7f14c9400640 1 --2- 172.21.15.27:0/3036924731 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c58c0 0x7f14c40c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:17:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.713+0000 7f14c8a00640 1 --2- 172.21.15.27:0/3036924731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c8ed0 0x7f14c41555e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009870 tx=0x7f14b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c7c10f0d8a3d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.713+0000 7f14c3e00640 1 -- 172.21.15.27:0/3036924731 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0035020 con 0x7f14c40c8ed0 2024-03-28T19:17:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.713+0000 7f14c3e00640 1 -- 172.21.15.27:0/3036924731 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0031640 con 0x7f14c40c8ed0 2024-03-28T19:17:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.713+0000 7f14c3e00640 1 -- 172.21.15.27:0/3036924731 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b002d040 con 0x7f14c40c8ed0 2024-03-28T19:17:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.714+0000 7f14ca800640 1 -- 172.21.15.27:0/3036924731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c8ed0 msgr2=0x7f14c41555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.714+0000 7f14ca800640 1 --2- 172.21.15.27:0/3036924731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c8ed0 0x7f14c41555e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009870 tx=0x7f14b002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.714+0000 7f14ca800640 1 -- 172.21.15.27:0/3036924731 shutdown_connections 2024-03-28T19:17:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.714+0000 7f14ca800640 1 --2- 172.21.15.27:0/3036924731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c8ed0 0x7f14c41555e0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.714+0000 7f14ca800640 1 --2- 172.21.15.27:0/3036924731 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c58c0 0x7f14c40c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.714+0000 7f14ca800640 1 -- 172.21.15.27:0/3036924731 >> 172.21.15.27:0/3036924731 conn(0x7f14c406b6f0 msgr2=0x7f14c40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.714+0000 7f14ca800640 1 -- 172.21.15.27:0/3036924731 shutdown_connections 2024-03-28T19:17:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.714+0000 7f14ca800640 1 -- 172.21.15.27:0/3036924731 wait complete. 2024-03-28T19:17:29.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.716+0000 7f14ca800640 1 Processor -- start 2024-03-28T19:17:29.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.716+0000 7f14ca800640 1 -- start start 2024-03-28T19:17:29.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.717+0000 7f14ca800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14c40be5d0 0x7f14c4157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:29.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.717+0000 7f14ca800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c58c0 0x7f14c41ef360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:29.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.717+0000 7f14ca800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c8ed0 0x7f14c41ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:29.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.717+0000 7f14ca800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b0820 con 0x7f14c40c58c0 2024-03-28T19:17:29.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.717+0000 7f14ca800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c4159750 con 0x7f14c40be5d0 2024-03-28T19:17:29.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.717+0000 7f14ca800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b09a0 con 0x7f14c40c8ed0 2024-03-28T19:17:29.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c58c0 0x7f14c41ef360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c58c0 0x7f14c41ef360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35914/0 (socket says 172.21.15.27:35914) 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 -- 172.21.15.27:0/2873567132 learned_addr learned my addr 172.21.15.27:0/2873567132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c9400640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14c40be5d0 0x7f14c4157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c9e00640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c8ed0 0x7f14c41ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 -- 172.21.15.27:0/2873567132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c8ed0 msgr2=0x7f14c41ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c8ed0 0x7f14c41ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 -- 172.21.15.27:0/2873567132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14c40be5d0 msgr2=0x7f14c4157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14c40be5d0 0x7f14c4157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:29.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 -- 172.21.15.27:0/2873567132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c41eb8b0 con 0x7f14c40c58c0 2024-03-28T19:17:29.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.718+0000 7f14c8a00640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c58c0 0x7f14c41ef360 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f14b0000c00 tx=0x7f14b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:29.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.719+0000 7f14c1600640 1 -- 172.21.15.27:0/2873567132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0040070 con 0x7f14c40c58c0 2024-03-28T19:17:29.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.719+0000 7f14c1600640 1 -- 172.21.15.27:0/2873567132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b00399f0 con 0x7f14c40c58c0 2024-03-28T19:17:29.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.719+0000 7f14c1600640 1 -- 172.21.15.27:0/2873567132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0035040 con 0x7f14c40c58c0 2024-03-28T19:17:29.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.719+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c41efde0 con 0x7f14c40c58c0 2024-03-28T19:17:29.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.719+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c40c09d0 con 0x7f14c40c58c0 2024-03-28T19:17:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.721+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c406c810 con 0x7f14c40c58c0 2024-03-28T19:17:29.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.721+0000 7f14c1600640 1 -- 172.21.15.27:0/2873567132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f14b004b060 con 0x7f14c40c58c0 2024-03-28T19:17:29.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.722+0000 7f14c1600640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14a406bc00 0x7f14a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:17:29.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.722+0000 7f14c1600640 1 -- 172.21.15.27:0/2873567132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f14b00acfe0 con 0x7f14c40c58c0 2024-03-28T19:17:29.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.723+0000 7f14c9400640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14a406bc00 0x7f14a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:17:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.724+0000 7f14c9400640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14a406bc00 0x7f14a406e0c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f14b8006fd0 tx=0x7f14b8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:17:29.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:29.728+0000 7f14c1600640 1 -- 172.21.15.27:0/2873567132 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f14c406c810 con 0x7f14c40c58c0 2024-03-28T19:17:30.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.064+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14c4069640 con 0x7f14c40c58c0 2024-03-28T19:17:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.065+0000 7f14c1600640 1 -- 172.21.15.27:0/2873567132 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f14c41eb8b0 con 0x7f14c40c58c0 2024-03-28T19:17:30.064 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:17:30.064 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:17:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.067+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14a406bc00 msgr2=0x7f14a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.067+0000 7f14ca800640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14a406bc00 0x7f14a406e0c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f14b8006fd0 tx=0x7f14b8008040 comp rx=0 tx=0).stop 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.067+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c58c0 msgr2=0x7f14c41ef360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.067+0000 7f14ca800640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c58c0 0x7f14c41ef360 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f14b0000c00 tx=0x7f14b00046c0 comp rx=0 tx=0).stop 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.068+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 shutdown_connections 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.068+0000 7f14ca800640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f14a406bc00 0x7f14a406e0c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.068+0000 7f14ca800640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14c40c8ed0 0x7f14c41ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.068+0000 7f14ca800640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c40c58c0 0x7f14c41ef360 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:30.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.068+0000 7f14ca800640 1 --2- 172.21.15.27:0/2873567132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14c40be5d0 0x7f14c4157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:17:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.068+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 >> 172.21.15.27:0/2873567132 conn(0x7f14c406b6f0 msgr2=0x7f14c40c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:17:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.068+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 shutdown_connections 2024-03-28T19:17:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:17:30.068+0000 7f14ca800640 1 -- 172.21.15.27:0/2873567132 wait complete. 2024-03-28T19:17:30.089 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-03-28T19:17:30.089 INFO:tasks.mds_thrash.fs.[a]:waiting for 59 secs before thrashing 2024-03-28T19:17:36.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:17:36.486 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:17:44.635 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 0% (109/12435) Updating files: 1% (125/12435) Updating files: 1% (219/12435) Updating files: 2% (249/12435) Updating files: 2% (310/12435) Updating files: 3% (374/12435) Updating files: 3% (393/12435) Updating files: 3% (471/12435) Updating files: 4% (498/12435) Updating files: 4% (555/12435) Updating files: 5% (622/12435) Updating files: 5% (625/12435) Updating files: 5% (698/12435) Updating files: 6% (747/12435) Updating files: 6% (764/12435) Updating files: 6% (824/12435) Updating files: 7% (871/12435) Updating files: 7% (892/12435) Updating files: 7% (959/12435) Updating files: 8% (995/12435) Updating files: 8% (1021/12435) Updating files: 8% (1095/12435) Updating files: 9% (1120/12435) Updating files: 9% (1158/12435) Updating files: 9% (1229/12435) Updating files: 10% (1244/12435) Updating files: 10% (1314/12435) Updating files: 11% (1368/12435) Updating files: 11% (1405/12435) Updating files: 12% (1493/12435) Updating files: 12% (1494/12435) Updating files: 12% (1560/12435) Updating files: 12% (1616/12435) Updating files: 13% (1617/12435) Updating files: 13% (1664/12435) Updating files: 13% (1721/12435) Updating files: 14% (1741/12435) Updating files: 14% (1771/12435) Updating files: 14% (1818/12435) Updating files: 15% (1866/12435) Updating files: 15% (1877/12435) Updating files: 15% (1946/12435) Updating files: 16% (1990/12435) Updating files: 16% (2000/12435) Updating files: 16% (2061/12435) Updating files: 17% (2114/12435) Updating files: 17% (2124/12435) Updating files: 17% (2187/12435) Updating files: 18% (2239/12435) Updating files: 18% (2245/12435) Updating files: 18% (2309/12435) Updating files: 19% (2363/12435) Updating files: 19% (2375/12435) Updating files: 19% (2424/12435) Updating files: 19% (2482/12435) Updating files: 20% (2487/12435) Updating files: 20% (2540/12435) Updating files: 20% (2602/12435) Updating files: 21% (2612/12435) Updating files: 21% (2678/12435) Updating files: 22% (2736/12435) Updating files: 22% (2741/12435) Updating files: 22% (2816/12435) Updating files: 23% (2861/12435) Updating files: 23% (2885/12435) Updating files: 23% (2941/12435) Updating files: 24% (2985/12435) Updating files: 24% (2993/12435) Updating files: 24% (3063/12435) Updating files: 25% (3109/12435) Updating files: 25% (3120/12435) Updating files: 25% (3190/12435) Updating files: 26% (3234/12435) Updating files: 26% (3256/12435) Updating files: 26% (3320/12435) Updating files: 27% (3358/12435) Updating files: 27% (3372/12435) Updating files: 27% (3420/12435) Updating files: 27% (3472/12435) Updating files: 28% (3482/12435) Updating files: 28% (3520/12435) Updating files: 28% (3570/12435) Updating files: 29% (3607/12435) Updating files: 29% (3616/12435) Updating files: 29% (3659/12435) Updating files: 29% (3699/12435) Updating files: 30% (3731/12435) Updating files: 30% (3742/12435) Updating files: 30% (3784/12435) Updating files: 30% (3833/12435) Updating files: 31% (3855/12435) Updating files: 31% (3875/12435) Updating files: 31% (3918/12435) Updating files: 31% (3966/12435) Updating files: 32% (3980/12435) Updating files: 32% (4011/12435) Updating files: 32% (4055/12435) Updating files: 32% (4100/12435) Updating files: 33% (4104/12435) Updating files: 33% (4147/12435) Updating files: 33% (4203/12435) Updating files: 34% (4228/12435) Updating files: 34% (4251/12435) Updating files: 34% (4291/12435) Updating files: 34% (4339/12435) Updating files: 35% (4353/12435) Updating files: 35% (4381/12435) Updating files: 35% (4421/12435) Updating files: 35% (4470/12435) Updating files: 36% (4477/12435) Updating files: 36% (4521/12435) Updating files: 36% (4575/12435) Updating files: 37% (4601/12435) Updating files: 37% (4639/12435) Updating files: 37% (4717/12435) Updating files: 38% (4726/12435) Updating files: 38% (4802/12435) Updating files: 39% (4850/12435) Updating files: 39% (4883/12435) Updating files: 39% (4967/12435) Updating files: 40% (4974/12435) Updating files: 40% (5032/12435) Updating files: 40% (5087/12435) Updating files: 41% (5099/12435) Updating files: 41% (5161/12435) Updating files: 42% (5223/12435) Updating files: 42% (5236/12435) Updating files: 42% (5282/12435) Updating files: 42% (5315/12435) Updating files: 43% (5348/12435) Updating files: 43% (5350/12435) Updating files: 43% (5382/12435) Updating files: 43% (5454/12435) Updating files: 44% (5472/12435) Updating files: 44% (5519/12435) Updating files: 45% (5596/12435) Updating files: 45% (5610/12435) Updating files: 45% (5667/12435) Updating files: 46% (5721/12435) Updating files: 46% (5731/12435) Updating files: 46% (5794/12435) Updating files: 46% (5811/12435) Updating files: 47% (5845/12435) Updating files: 47% (5867/12435) Updating files: 47% (5958/12435) Updating files: 48% (5969/12435) Updating files: 48% (6035/12435) Updating files: 49% (6094/12435) Updating files: 49% (6112/12435) Updating files: 49% (6178/12435) Updating files: 50% (6218/12435) Updating files: 50% (6263/12435) Updating files: 51% (6342/12435) Updating files: 51% (6354/12435) Updating files: 51% (6445/12435) Updating files: 52% (6467/12435) Updating files: 52% (6524/12435) Updating files: 53% (6591/12435) Updating files: 53% (6624/12435) Updating files: 54% (6715/12435) Updating files: 54% (6765/12435) Updating files: 54% (6795/12435) Updating files: 55% (6840/12435) Updating files: 55% (6854/12435) Updating files: 55% (6907/12435) Updating files: 55% (6957/12435) Updating files: 56% (6964/12435) Updating files: 56% (7013/12435) Updating files: 56% (7054/12435) Updating files: 57% (7088/12435) Updating files: 57% (7109/12435) Updating files: 57% (7170/12435) Updating files: 58% (7213/12435) Updating files: 58% (7229/12435) Updating files: 58% (7291/12435) Updating files: 59% (7337/12435) Updating files: 59% (7349/12435) Updating files: 59% (7412/12435) Updating files: 60% (7461/12435) Updating files: 60% (7478/12435) Updating files: 60% (7546/12435) Updating files: 61% (7586/12435) Updating files: 61% (7615/12435) Updating files: 61% (7676/12435) Updating files: 61% (7677/12435) Updating files: 62% (7710/12435) Updating files: 62% (7740/12435) Updating files: 62% (7826/12435) Updating files: 63% (7835/12435) Updating files: 63% (7893/12435) Updating files: 63% (7947/12435) Updating files: 64% (7959/12435) Updating files: 64% (8016/12435) Updating files: 64% (8082/12435) Updating files: 65% (8083/12435) Updating files: 65% (8128/12435) Updating files: 65% (8186/12435) Updating files: 66% (8208/12435) Updating files: 66% (8246/12435) Updating files: 66% (8279/12435) Updating files: 66% (8328/12435) Updating files: 67% (8332/12435) Updating files: 67% (8361/12435) Updating files: 67% (8401/12435) Updating files: 67% (8447/12435) Updating files: 68% (8456/12435) Updating files: 68% (8487/12435) Updating files: 68% (8523/12435) Updating files: 68% (8557/12435) Updating files: 69% (8581/12435) Updating files: 69% (8595/12435) Updating files: 69% (8628/12435) Updating files: 69% (8665/12435) Updating files: 70% (8705/12435) Updating files: 70% (8706/12435) Updating files: 70% (8754/12435) Updating files: 70% (8806/12435) Updating files: 71% (8829/12435) Updating files: 71% (8849/12435) Updating files: 71% (8890/12435) Updating files: 71% (8934/12435) Updating files: 72% (8954/12435) Updating files: 72% (8980/12435) Updating files: 72% (9021/12435) Updating files: 72% (9068/12435) Updating files: 73% (9078/12435) Updating files: 73% (9112/12435) Updating files: 73% (9175/12435) Updating files: 74% (9202/12435) Updating files: 74% (9229/12435) Updating files: 74% (9282/12435) Updating files: 75% (9327/12435) Updating files: 75% (9374/12435) Updating files: 75% (9422/12435) Updating files: 76% (9451/12435) Updating files: 76% (9475/12435) Updating files: 76% (9530/12435) Updating files: 77% (9575/12435) Updating files: 77% (9588/12435) Updating files: 77% (9650/12435) Updating files: 78% (9700/12435) Updating files: 78% (9710/12435) Updating files: 78% (9762/12435) Updating files: 79% (9824/12435) Updating files: 79% (9844/12435) Updating files: 79% (9882/12435) Updating files: 79% (9913/12435) Updating files: 79% (9928/12435) Updating files: 80% (9948/12435) Updating files: 80% (9983/12435) Updating files: 80% (10034/12435) Updating files: 81% (10073/12435) Updating files: 81% (10079/12435) Updating files: 81% (10131/12435) Updating files: 81% (10190/12435) Updating files: 82% (10197/12435) Updating files: 82% (10251/12435) Updating files: 82% (10303/12435) Updating files: 83% (10322/12435) Updating files: 83% (10364/12435) Updating files: 83% (10444/12435) Updating files: 84% (10446/12435) Updating files: 84% (10524/12435) Updating files: 85% (10570/12435) Updating files: 85% (10609/12435) Updating files: 86% (10695/12435) Updating files: 86% (10709/12435) Updating files: 86% (10725/12435) Updating files: 86% (10764/12435) Updating files: 87% (10819/12435) Updating files: 87% (10825/12435) Updating files: 87% (10878/12435) Updating files: 87% (10910/12435) Updating files: 88% (10943/12435) Updating files: 88% (10973/12435) Updating files: 88% (11045/12435) Updating files: 89% (11068/12435) Updating files: 89% (11094/12435) Updating files: 89% (11141/12435) Updating files: 90% (11192/12435) Updating files: 90% (11217/12435) Updating files: 90% (11300/12435) Updating files: 91% (11316/12435) Updating files: 91% (11371/12435) Updating files: 92% (11441/12435) Updating files: 92% (11459/12435) Updating files: 92% (11534/12435) Updating files: 93% (11565/12435) Updating files: 93% (11595/12435) Updating files: 93% (11666/12435) Updating files: 94% (11689/12435) Updating files: 94% (11734/12435) Updating files: 94% (11793/12435) Updating files: 95% (11814/12435) Updating files: 95% (11856/12435) Updating files: 95% (11925/12435) Updating files: 96% (11938/12435) Updating files: 96% (11997/12435) Updating files: 97% (12062/12435) Updating files: 97% (12063/12435) Updating files: 97% (12124/12435) Updating files: 97% (12177/12435) Updating files: 98% (12187/12435) Updating files: 98% (12241/12435) Updating files: 98% (12297/12435) Updating files: 99% (12311/12435) Updating files: 99% (12356/12435) Updating files: 99% (12426/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-03-28T19:17:44.767 INFO:tasks.workunit.client.1.smithi155.stderr:+ trap - EXIT 2024-03-28T19:17:44.767 INFO:tasks.workunit.client.1.smithi155.stderr:+ cd ceph 2024-03-28T19:17:44.772 INFO:tasks.workunit.client.1.smithi155.stderr:++ seq 1 90 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:+ versions='1 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:2 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:3 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:4 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:5 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:6 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:7 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:8 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:9 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:10 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:11 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:12 2024-03-28T19:17:44.775 INFO:tasks.workunit.client.1.smithi155.stderr:13 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:14 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:15 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:16 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:17 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:18 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:19 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:20 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:21 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:22 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:23 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:24 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:25 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:26 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:27 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:28 2024-03-28T19:17:44.776 INFO:tasks.workunit.client.1.smithi155.stderr:29 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:30 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:31 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:32 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:33 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:34 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:35 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:36 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:37 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:38 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:39 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:40 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:41 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:42 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:43 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:44 2024-03-28T19:17:44.777 INFO:tasks.workunit.client.1.smithi155.stderr:45 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:46 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:47 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:48 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:49 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:50 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:51 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:52 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:53 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:54 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:55 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:56 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:57 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:58 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:59 2024-03-28T19:17:44.778 INFO:tasks.workunit.client.1.smithi155.stderr:60 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:61 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:62 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:63 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:64 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:65 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:66 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:67 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:68 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:69 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:70 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:71 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:72 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:73 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:74 2024-03-28T19:17:44.779 INFO:tasks.workunit.client.1.smithi155.stderr:75 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:76 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:77 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:78 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:79 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:80 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:81 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:82 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:83 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:84 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:85 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:86 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:87 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:88 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:89 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:90' 2024-03-28T19:17:44.780 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:17:44.781 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 1 -eq 48 ']' 2024-03-28T19:17:44.781 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.1 2024-03-28T19:17:44.781 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.1 2024-03-28T19:17:44.781 INFO:tasks.workunit.client.1.smithi155.stdout:v0.1 2024-03-28T19:17:44.781 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.1 2024-03-28T19:18:06.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:18:06.625 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:18:29.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:29.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.321+0000 7f9a4b200640 1 Processor -- start 2024-03-28T19:18:29.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.321+0000 7f9a4b200640 1 -- start start 2024-03-28T19:18:29.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.322+0000 7f9a4b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a4c0c93c0 0x7f9a4c155990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.322+0000 7f9a4b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c1565e0 con 0x7f9a4c152e90 2024-03-28T19:18:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.322+0000 7f9a4b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c158c30 con 0x7f9a4c0c8cd0 2024-03-28T19:18:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.322+0000 7f9a4b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4c158f10 con 0x7f9a4c0c93c0 2024-03-28T19:18:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.323+0000 7f9a43e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a4c0c93c0 0x7f9a4c155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.323+0000 7f9a43e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a4c0c93c0 0x7f9a4c155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54594/0 (socket says 172.21.15.27:54594) 2024-03-28T19:18:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.323+0000 7f9a43e00640 1 -- 172.21.15.27:0/3119470211 learned_addr learned my addr 172.21.15.27:0/3119470211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:29.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.323+0000 7f9a43e00640 1 -- 172.21.15.27:0/3119470211 >> v1:172.21.15.155:6789/0 conn(0x7f9a4c0c8cd0 legacy=0x7f9a4c155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:29.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.323+0000 7f9a43e00640 1 -- 172.21.15.27:0/3119470211 >> v1:172.21.15.27:6789/0 conn(0x7f9a4c152e90 legacy=0x7f9a4c0c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.323+0000 7f9a43e00640 1 -- 172.21.15.27:0/3119470211 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c1591f0 con 0x7f9a4c0c93c0 2024-03-28T19:18:29.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.323+0000 7f9a49400640 1 -- 172.21.15.27:0/3119470211 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030213840 0 0) 0x7f9a4c1565e0 con 0x7f9a4c152e90 2024-03-28T19:18:29.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.325+0000 7f9a43e00640 1 --2- 172.21.15.27:0/3119470211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c0c93c0 0x7f9a4c155990 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009870 tx=0x7f9a3402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b06bbc0022af12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:29.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.325+0000 7f9a49400640 1 -- 172.21.15.27:0/3119470211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3402f840 con 0x7f9a4c0c93c0 2024-03-28T19:18:29.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.325+0000 7f9a49400640 1 -- 172.21.15.27:0/3119470211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3402f9e0 con 0x7f9a4c0c93c0 2024-03-28T19:18:29.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.325+0000 7f9a49400640 1 -- 172.21.15.27:0/3119470211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3402fdb0 con 0x7f9a4c0c93c0 2024-03-28T19:18:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.326+0000 7f9a4b200640 1 -- 172.21.15.27:0/3119470211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c0c93c0 msgr2=0x7f9a4c155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.326+0000 7f9a4b200640 1 --2- 172.21.15.27:0/3119470211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c0c93c0 0x7f9a4c155990 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009870 tx=0x7f9a3402ba20 comp rx=0 tx=0).stop 2024-03-28T19:18:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.326+0000 7f9a4b200640 1 -- 172.21.15.27:0/3119470211 shutdown_connections 2024-03-28T19:18:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.326+0000 7f9a4b200640 1 --2- 172.21.15.27:0/3119470211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c0c93c0 0x7f9a4c155990 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.326+0000 7f9a4b200640 1 -- 172.21.15.27:0/3119470211 >> 172.21.15.27:0/3119470211 conn(0x7f9a4c0ac900 msgr2=0x7f9a4c0acd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.326+0000 7f9a4b200640 1 -- 172.21.15.27:0/3119470211 shutdown_connections 2024-03-28T19:18:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.326+0000 7f9a4b200640 1 -- 172.21.15.27:0/3119470211 wait complete. 2024-03-28T19:18:29.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.328+0000 7f9a4b200640 1 Processor -- start 2024-03-28T19:18:29.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.328+0000 7f9a4b200640 1 -- start start 2024-03-28T19:18:29.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a4c0c8cd0 0x7f9a4c158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a4c0c93c0 0x7f9a4c1eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c152e90 0x7f9a4c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c159a90 con 0x7f9a4c152e90 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c1597b0 con 0x7f9a4c0c93c0 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0c7e20 con 0x7f9a4c0c8cd0 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c152e90 0x7f9a4c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c152e90 0x7f9a4c154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54600/0 (socket says 172.21.15.27:54600) 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a4a800640 1 -- 172.21.15.27:0/1130399860 learned_addr learned my addr 172.21.15.27:0/1130399860 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.330+0000 7f9a43e00640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a4c0c93c0 0x7f9a4c1eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.331+0000 7f9a49e00640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a4c0c8cd0 0x7f9a4c158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.331+0000 7f9a4a800640 1 -- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a4c0c8cd0 msgr2=0x7f9a4c158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.331+0000 7f9a4a800640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a4c0c8cd0 0x7f9a4c158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.331+0000 7f9a4a800640 1 -- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a4c0c93c0 msgr2=0x7f9a4c1eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.331+0000 7f9a4a800640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a4c0c93c0 0x7f9a4c1eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.331+0000 7f9a4a800640 1 -- 172.21.15.27:0/1130399860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c0c9f80 con 0x7f9a4c152e90 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.331+0000 7f9a49e00640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a4c0c8cd0 0x7f9a4c158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.331+0000 7f9a4a800640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c152e90 0x7f9a4c154630 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400c7a0 tx=0x7f9a4400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.332+0000 7f9a42000640 1 -- 172.21.15.27:0/1130399860 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44003b70 con 0x7f9a4c152e90 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.332+0000 7f9a42000640 1 -- 172.21.15.27:0/1130399860 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a44003d30 con 0x7f9a4c152e90 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.332+0000 7f9a42000640 1 -- 172.21.15.27:0/1130399860 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4400aa90 con 0x7f9a4c152e90 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.332+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c1eeed0 con 0x7f9a4c152e90 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.332+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c0c9c10 con 0x7f9a4c152e90 2024-03-28T19:18:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.333+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4c067dd0 con 0x7f9a4c152e90 2024-03-28T19:18:29.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.335+0000 7f9a42000640 1 -- 172.21.15.27:0/1130399860 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a4400ac30 con 0x7f9a4c152e90 2024-03-28T19:18:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.336+0000 7f9a42000640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a2806bbd0 0x7f9a2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.336+0000 7f9a42000640 1 -- 172.21.15.27:0/1130399860 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f9a4408b770 con 0x7f9a4c152e90 2024-03-28T19:18:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.336+0000 7f9a49e00640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a2806bbd0 0x7f9a2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.337+0000 7f9a49e00640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a2806bbd0 0x7f9a2806e090 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c150430 tx=0x7f9a3c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:29.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.343+0000 7f9a42000640 1 -- 172.21.15.27:0/1130399860 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a44059400 con 0x7f9a4c152e90 2024-03-28T19:18:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.680+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a4c069640 con 0x7f9a4c152e90 2024-03-28T19:18:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.680+0000 7f9a42000640 1 -- 172.21.15.27:0/1130399860 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f9a44058d00 con 0x7f9a4c152e90 2024-03-28T19:18:29.681 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:29.681 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:29.681 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a2806bbd0 msgr2=0x7f9a2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a2806bbd0 0x7f9a2806e090 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c150430 tx=0x7f9a3c009040 comp rx=0 tx=0).stop 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c152e90 msgr2=0x7f9a4c154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c152e90 0x7f9a4c154630 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400c7a0 tx=0x7f9a4400cc60 comp rx=0 tx=0).stop 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 shutdown_connections 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9a2806bbd0 0x7f9a2806e090 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a4c152e90 0x7f9a4c154630 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a4c0c93c0 0x7f9a4c1eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 --2- 172.21.15.27:0/1130399860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9a4c0c8cd0 0x7f9a4c158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.683+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 >> 172.21.15.27:0/1130399860 conn(0x7f9a4c0ac900 msgr2=0x7f9a4c151410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.684+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 shutdown_connections 2024-03-28T19:18:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.684+0000 7f9a4b200640 1 -- 172.21.15.27:0/1130399860 wait complete. 2024-03-28T19:18:29.709 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-03-28T19:18:29.709 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:18:29.710 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:18:29.739 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:18:29.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:29.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.982+0000 7f2b59600640 1 Processor -- start 2024-03-28T19:18:29.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.982+0000 7f2b59600640 1 -- start start 2024-03-28T19:18:29.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b59600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c74e0 0x7f2b540c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b59600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2b540c2290 0x7f2b54152760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b59600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2b54152ca0 0x7f2b541556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b59600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b541530e0 con 0x7f2b540c2290 2024-03-28T19:18:29.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b59600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b54158d00 con 0x7f2b54152ca0 2024-03-28T19:18:29.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b59600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54158fe0 con 0x7f2b540c74e0 2024-03-28T19:18:29.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b4b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2b540c2290 0x7f2b54152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b4b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2b540c2290 0x7f2b54152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54602/0 (socket says 172.21.15.27:54602) 2024-03-28T19:18:29.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b4b400640 1 -- 172.21.15.27:0/1560691231 learned_addr learned my addr 172.21.15.27:0/1560691231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:29.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.984+0000 7f2b53e00640 1 --2- 172.21.15.27:0/1560691231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c74e0 0x7f2b540c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.985+0000 7f2b58c00640 1 --2- 172.21.15.27:0/1560691231 >> v2:172.21.15.155:3300/0 conn(0x7f2b54152ca0 0x7f2b541556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.985+0000 7f2b4b400640 1 -- 172.21.15.27:0/1560691231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c74e0 msgr2=0x7f2b540c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:29.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.985+0000 7f2b4b400640 1 --2- 172.21.15.27:0/1560691231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c74e0 0x7f2b540c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.985+0000 7f2b4b400640 1 -- 172.21.15.27:0/1560691231 >> v2:172.21.15.155:3300/0 conn(0x7f2b54152ca0 msgr2=0x7f2b541556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:29.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.985+0000 7f2b4b400640 1 --2- 172.21.15.27:0/1560691231 >> v2:172.21.15.155:3300/0 conn(0x7f2b54152ca0 0x7f2b541556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.985+0000 7f2b4b400640 1 -- 172.21.15.27:0/1560691231 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b541592c0 con 0x7f2b540c2290 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.986+0000 7f2b4b400640 1 --2- 172.21.15.27:0/1560691231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b540c2290 0x7f2b54152760 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2b54073850 tx=0x7f2b3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a65a73765f99cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b53400640 1 -- 172.21.15.27:0/1560691231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c02e070 con 0x7f2b540c2290 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b53400640 1 -- 172.21.15.27:0/1560691231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c037070 con 0x7f2b540c2290 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b53400640 1 -- 172.21.15.27:0/1560691231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c031a00 con 0x7f2b540c2290 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 -- 172.21.15.27:0/1560691231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b540c2290 msgr2=0x7f2b54152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 --2- 172.21.15.27:0/1560691231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b540c2290 0x7f2b54152760 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f2b54073850 tx=0x7f2b3c02b880 comp rx=0 tx=0).stop 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 -- 172.21.15.27:0/1560691231 shutdown_connections 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 --2- 172.21.15.27:0/1560691231 >> v2:172.21.15.155:3300/0 conn(0x7f2b54152ca0 0x7f2b541556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 --2- 172.21.15.27:0/1560691231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b540c2290 0x7f2b54152760 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 --2- 172.21.15.27:0/1560691231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c74e0 0x7f2b540c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 -- 172.21.15.27:0/1560691231 >> 172.21.15.27:0/1560691231 conn(0x7f2b5406d980 msgr2=0x7f2b540a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 -- 172.21.15.27:0/1560691231 shutdown_connections 2024-03-28T19:18:29.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.987+0000 7f2b59600640 1 -- 172.21.15.27:0/1560691231 wait complete. 2024-03-28T19:18:29.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.989+0000 7f2b59600640 1 Processor -- start 2024-03-28T19:18:29.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.989+0000 7f2b59600640 1 -- start start 2024-03-28T19:18:29.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b59600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c2290 0x7f2b541eae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b59600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b540c74e0 0x7f2b5414cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b59600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54152ca0 0x7f2b541eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b59600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b541589f0 con 0x7f2b54152ca0 2024-03-28T19:18:29.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b59600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54159880 con 0x7f2b540c74e0 2024-03-28T19:18:29.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b59600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b540c78c0 con 0x7f2b540c2290 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b58c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54152ca0 0x7f2b541eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b58c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54152ca0 0x7f2b541eebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54606/0 (socket says 172.21.15.27:54606) 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.991+0000 7f2b58c00640 1 -- 172.21.15.27:0/2485671050 learned_addr learned my addr 172.21.15.27:0/2485671050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b53e00640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c2290 0x7f2b541eae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b4b400640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b540c74e0 0x7f2b5414cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b58c00640 1 -- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c2290 msgr2=0x7f2b541eae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b58c00640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c2290 0x7f2b541eae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b58c00640 1 -- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b540c74e0 msgr2=0x7f2b5414cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b58c00640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b540c74e0 0x7f2b5414cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b58c00640 1 -- 172.21.15.27:0/2485671050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5414ec40 con 0x7f2b54152ca0 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b4b400640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b540c74e0 0x7f2b5414cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b53e00640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c2290 0x7f2b541eae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:18:29.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.992+0000 7f2b58c00640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54152ca0 0x7f2b541eebd0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00d800 tx=0x7f2b4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:29.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.993+0000 7f2b50c00640 1 -- 172.21.15.27:0/2485671050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c00b660 con 0x7f2b54152ca0 2024-03-28T19:18:29.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.993+0000 7f2b50c00640 1 -- 172.21.15.27:0/2485671050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c019070 con 0x7f2b54152ca0 2024-03-28T19:18:29.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.993+0000 7f2b50c00640 1 -- 172.21.15.27:0/2485671050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c00ba60 con 0x7f2b54152ca0 2024-03-28T19:18:29.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.993+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5414ef20 con 0x7f2b54152ca0 2024-03-28T19:18:29.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.993+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b541ee6b0 con 0x7f2b54152ca0 2024-03-28T19:18:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.995+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b54067dd0 con 0x7f2b54152ca0 2024-03-28T19:18:29.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.995+0000 7f2b50c00640 1 -- 172.21.15.27:0/2485671050 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2b4c00bc00 con 0x7f2b54152ca0 2024-03-28T19:18:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.997+0000 7f2b50c00640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b3006bbd0 0x7f2b3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.997+0000 7f2b50c00640 1 -- 172.21.15.27:0/2485671050 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f2b4c08bd70 con 0x7f2b54152ca0 2024-03-28T19:18:29.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.997+0000 7f2b53e00640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b3006bbd0 0x7f2b3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:29.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:29.998+0000 7f2b53e00640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b3006bbd0 0x7f2b3006e090 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2b44007380 tx=0x7f2b4400b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:30.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.002+0000 7f2b50c00640 1 -- 172.21.15.27:0/2485671050 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2b4c01c360 con 0x7f2b54152ca0 2024-03-28T19:18:30.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.253+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b5414fd90 con 0x7f2b54152ca0 2024-03-28T19:18:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.254+0000 7f2b50c00640 1 -- 172.21.15.27:0/2485671050 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f2b5414fd90 con 0x7f2b54152ca0 2024-03-28T19:18:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:30.255 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b3006bbd0 msgr2=0x7f2b3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b3006bbd0 0x7f2b3006e090 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2b44007380 tx=0x7f2b4400b040 comp rx=0 tx=0).stop 2024-03-28T19:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54152ca0 msgr2=0x7f2b541eebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54152ca0 0x7f2b541eebd0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00d800 tx=0x7f2b4c00dcc0 comp rx=0 tx=0).stop 2024-03-28T19:18:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 shutdown_connections 2024-03-28T19:18:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2b3006bbd0 0x7f2b3006e090 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54152ca0 0x7f2b541eebd0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b540c74e0 0x7f2b5414cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 --2- 172.21.15.27:0/2485671050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b540c2290 0x7f2b541eae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 >> 172.21.15.27:0/2485671050 conn(0x7f2b5406d980 msgr2=0x7f2b540c1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 shutdown_connections 2024-03-28T19:18:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:30.257+0000 7f2b59600640 1 -- 172.21.15.27:0/2485671050 wait complete. 2024-03-28T19:18:30.278 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:18:31.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.290+0000 7f9000a00640 1 Processor -- start 2024-03-28T19:18:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.291+0000 7f9000a00640 1 -- start start 2024-03-28T19:18:31.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.292+0000 7f9000a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8ffc0c3d60 0x7f8ffc153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.292+0000 7f9000a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc0c2cb0 con 0x7f8ffc154220 2024-03-28T19:18:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.292+0000 7f9000a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc0c2f50 con 0x7f8ffc072240 2024-03-28T19:18:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.293+0000 7f9000a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc072d30 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.293+0000 7f8ffb200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8ffc154220 0x7f8ffc156b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45136/0 (socket says 172.21.15.27:45136) 2024-03-28T19:18:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.293+0000 7f8ffb200640 1 -- 172.21.15.27:0/2189000185 learned_addr learned my addr 172.21.15.27:0/2189000185 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:31.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.293+0000 7f8ff1e00640 1 --2- 172.21.15.27:0/2189000185 >> v2:172.21.15.155:3300/0 conn(0x7f8ffc0c3d60 0x7f8ffc153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.294+0000 7f8ff9e00640 1 -- 172.21.15.27:0/2189000185 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882829 0 0) 0x7f8ffc0c2cb0 con 0x7f8ffc154220 2024-03-28T19:18:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.294+0000 7f8ff9e00640 1 -- 172.21.15.27:0/2189000185 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe0003520 con 0x7f8ffc154220 2024-03-28T19:18:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.294+0000 7f8ff1e00640 1 -- 172.21.15.27:0/2189000185 >> v1:172.21.15.155:6789/0 conn(0x7f8ffc072240 legacy=0x7f8ffc072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.294+0000 7f8ff1e00640 1 -- 172.21.15.27:0/2189000185 >> v1:172.21.15.27:6789/0 conn(0x7f8ffc154220 legacy=0x7f8ffc156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.294+0000 7f8ff1e00640 1 -- 172.21.15.27:0/2189000185 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0c3230 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.295+0000 7f8ff1e00640 1 --2- 172.21.15.27:0/2189000185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc153ce0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40097a0 tx=0x7f8fe402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3971c06bd08746e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.296+0000 7f8ff9e00640 1 -- 172.21.15.27:0/2189000185 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4035020 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.296+0000 7f8ff9e00640 1 -- 172.21.15.27:0/2189000185 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4031830 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.296+0000 7f8ff9e00640 1 -- 172.21.15.27:0/2189000185 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4031be0 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.296+0000 7f9000a00640 1 -- 172.21.15.27:0/2189000185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc0c3d60 msgr2=0x7f8ffc153ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.296+0000 7f9000a00640 1 --2- 172.21.15.27:0/2189000185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc153ce0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40097a0 tx=0x7f8fe402b8d0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.296+0000 7f9000a00640 1 -- 172.21.15.27:0/2189000185 shutdown_connections 2024-03-28T19:18:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.296+0000 7f9000a00640 1 --2- 172.21.15.27:0/2189000185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc153ce0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.296+0000 7f9000a00640 1 -- 172.21.15.27:0/2189000185 >> 172.21.15.27:0/2189000185 conn(0x7f8ffc0c7aa0 msgr2=0x7f8ffc0c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.297+0000 7f9000a00640 1 -- 172.21.15.27:0/2189000185 shutdown_connections 2024-03-28T19:18:31.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.297+0000 7f9000a00640 1 -- 172.21.15.27:0/2189000185 wait complete. 2024-03-28T19:18:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.298+0000 7f9000a00640 1 Processor -- start 2024-03-28T19:18:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.298+0000 7f9000a00640 1 -- start start 2024-03-28T19:18:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.299+0000 7f9000a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc072240 0x7f8ffc1ea230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.299+0000 7f9000a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc0c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.299+0000 7f9000a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ffc154220 0x7f8ffc0c1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.299+0000 7f9000a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc154690 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.299+0000 7f9000a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0c35a0 con 0x7f8ffc072240 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.299+0000 7f9000a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc072fc0 con 0x7f8ffc154220 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ff1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc0c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ff1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc0c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54620/0 (socket says 172.21.15.27:54620) 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ff1e00640 1 -- 172.21.15.27:0/3351581373 learned_addr learned my addr 172.21.15.27:0/3351581373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ffa800640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc072240 0x7f8ffc1ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ffb200640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ffc154220 0x7f8ffc0c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ff1e00640 1 -- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ffc154220 msgr2=0x7f8ffc0c1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ff1e00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ffc154220 0x7f8ffc0c1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ff1e00640 1 -- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc072240 msgr2=0x7f8ffc1ea230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ff1e00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc072240 0x7f8ffc1ea230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ff1e00640 1 -- 172.21.15.27:0/3351581373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc1eaa70 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.300+0000 7f8ffb200640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ffc154220 0x7f8ffc0c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:31.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.301+0000 7f8ff1e00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc0c1120 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4034040 tx=0x7f8fe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.301+0000 7f8ff3400640 1 -- 172.21.15.27:0/3351581373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4035040 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.301+0000 7f8ff3400640 1 -- 172.21.15.27:0/3351581373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4004250 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.301+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc154da0 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.301+0000 7f8ff3400640 1 -- 172.21.15.27:0/3351581373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe403a640 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.301+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc0ab2c0 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.303+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc069640 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.304+0000 7f8ff3400640 1 -- 172.21.15.27:0/3351581373 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8fe4038020 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.306+0000 7f8ff3400640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8fd806bc00 0x7f8fd806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.306+0000 7f8ff3400640 1 -- 172.21.15.27:0/3351581373 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f8fe40ad510 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.306+0000 7f8ffa800640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8fd806bc00 0x7f8fd806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.307+0000 7f8ffa800640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8fd806bc00 0x7f8fd806e0c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8fec000f60 tx=0x7f8fec009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:31.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.312+0000 7f8ff3400640 1 -- 172.21.15.27:0/3351581373 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8fe407b230 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.560+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ffc0c1ba0 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.561+0000 7f8ff3400640 1 -- 172.21.15.27:0/3351581373 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f8fe407ab30 con 0x7f8ffc0c3d60 2024-03-28T19:18:31.561 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:31.561 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:31.562 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8fd806bc00 msgr2=0x7f8fd806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8fd806bc00 0x7f8fd806e0c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8fec000f60 tx=0x7f8fec009040 comp rx=0 tx=0).stop 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ffc0c3d60 msgr2=0x7f8ffc0c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc0c1120 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4034040 tx=0x7f8fe40046c0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 shutdown_connections 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8fd806bc00 0x7f8fd806e0c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ffc154220 0x7f8ffc0c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ffc0c3d60 0x7f8ffc0c1120 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 --2- 172.21.15.27:0/3351581373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ffc072240 0x7f8ffc1ea230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.565+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 >> 172.21.15.27:0/3351581373 conn(0x7f8ffc0c7aa0 msgr2=0x7f8ffc0a8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.566+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 shutdown_connections 2024-03-28T19:18:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.566+0000 7f9000a00640 1 -- 172.21.15.27:0/3351581373 wait complete. 2024-03-28T19:18:31.585 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-03-28T19:18:31.585 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:18:31.585 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:18:31.620 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:18:31.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:31.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.897+0000 7f81dca00640 1 Processor -- start 2024-03-28T19:18:31.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.897+0000 7f81dca00640 1 -- start start 2024-03-28T19:18:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.898+0000 7f81dca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81d81551a0 0x7f81d80c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.898+0000 7f81dca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f81d80aada0 0x7f81d80c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.898+0000 7f81dca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d80c26d0 con 0x7f81d80c5580 2024-03-28T19:18:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.898+0000 7f81dca00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d8156870 con 0x7f81d81551a0 2024-03-28T19:18:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.898+0000 7f81dca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d8155cf0 con 0x7f81d80aada0 2024-03-28T19:18:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.899+0000 7f81d6800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81d81551a0 0x7f81d80c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.899+0000 7f81cde00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f81d80c5580 0x7f81d80c1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45148/0 (socket says 172.21.15.27:45148) 2024-03-28T19:18:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.899+0000 7f81cde00640 1 -- 172.21.15.27:0/73037022 learned_addr learned my addr 172.21.15.27:0/73037022 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.899+0000 7f81d6800640 1 -- 172.21.15.27:0/73037022 >> v2:172.21.15.155:3300/0 conn(0x7f81d80aada0 msgr2=0x7f81d80c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.899+0000 7f81d6800640 1 --2- 172.21.15.27:0/73037022 >> v2:172.21.15.155:3300/0 conn(0x7f81d80aada0 0x7f81d80c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.899+0000 7f81d6800640 1 -- 172.21.15.27:0/73037022 >> v1:172.21.15.27:6789/0 conn(0x7f81d80c5580 legacy=0x7f81d80c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.899+0000 7f81d6800640 1 -- 172.21.15.27:0/73037022 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d8155fd0 con 0x7f81d81551a0 2024-03-28T19:18:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.900+0000 7f81d6800640 1 --2- 172.21.15.27:0/73037022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d81551a0 0x7f81d80c5040 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f81d80abbe0 tx=0x7f81c802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58aed6c51a724582 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.901+0000 7f81d5e00640 1 -- 172.21.15.27:0/73037022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8038070 con 0x7f81d81551a0 2024-03-28T19:18:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.901+0000 7f81d5e00640 1 -- 172.21.15.27:0/73037022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c8031710 con 0x7f81d81551a0 2024-03-28T19:18:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.901+0000 7f81d5e00640 1 -- 172.21.15.27:0/73037022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8031ae0 con 0x7f81d81551a0 2024-03-28T19:18:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.901+0000 7f81dca00640 1 -- 172.21.15.27:0/73037022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d81551a0 msgr2=0x7f81d80c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.901+0000 7f81dca00640 1 --2- 172.21.15.27:0/73037022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d81551a0 0x7f81d80c5040 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f81d80abbe0 tx=0x7f81c802b7b0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.902+0000 7f81dca00640 1 -- 172.21.15.27:0/73037022 shutdown_connections 2024-03-28T19:18:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.902+0000 7f81dca00640 1 --2- 172.21.15.27:0/73037022 >> v2:172.21.15.155:3300/0 conn(0x7f81d80aada0 0x7f81d80c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.902+0000 7f81dca00640 1 --2- 172.21.15.27:0/73037022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d81551a0 0x7f81d80c5040 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.902+0000 7f81dca00640 1 -- 172.21.15.27:0/73037022 >> 172.21.15.27:0/73037022 conn(0x7f81d806b0c0 msgr2=0x7f81d806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.902+0000 7f81dca00640 1 -- 172.21.15.27:0/73037022 shutdown_connections 2024-03-28T19:18:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.902+0000 7f81dca00640 1 -- 172.21.15.27:0/73037022 wait complete. 2024-03-28T19:18:31.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.903+0000 7f81dca00640 1 Processor -- start 2024-03-28T19:18:31.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.904+0000 7f81dca00640 1 -- start start 2024-03-28T19:18:31.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.904+0000 7f81dca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d80aada0 0x7f81d814c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.904+0000 7f81dca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81d80c5580 0x7f81d81eae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.904+0000 7f81dca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81d81551a0 0x7f81d81eec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.904+0000 7f81dca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d81559a0 con 0x7f81d80aada0 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81dca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d8156590 con 0x7f81d81551a0 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81dca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d8155580 con 0x7f81d80c5580 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d80aada0 0x7f81d814c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d80aada0 0x7f81d814c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54632/0 (socket says 172.21.15.27:54632) 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d6800640 1 -- 172.21.15.27:0/3784830666 learned_addr learned my addr 172.21.15.27:0/3784830666 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81cde00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81d80c5580 0x7f81d81eae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d7200640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81d81551a0 0x7f81d81eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d6800640 1 -- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81d80c5580 msgr2=0x7f81d81eae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d6800640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81d80c5580 0x7f81d81eae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d6800640 1 -- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81d81551a0 msgr2=0x7f81d81eec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:31.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d6800640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81d81551a0 0x7f81d81eec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:31.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d6800640 1 -- 172.21.15.27:0/3784830666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d81eb350 con 0x7f81d80aada0 2024-03-28T19:18:31.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81cde00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81d80c5580 0x7f81d81eae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:31.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.905+0000 7f81d7200640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81d81551a0 0x7f81d81eec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:31.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.906+0000 7f81d6800640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d80aada0 0x7f81d814c730 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f81c8000c00 tx=0x7f81c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.906+0000 7f81cf400640 1 -- 172.21.15.27:0/3784830666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8043070 con 0x7f81d80aada0 2024-03-28T19:18:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.906+0000 7f81cf400640 1 -- 172.21.15.27:0/3784830666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c8033a40 con 0x7f81d80aada0 2024-03-28T19:18:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.906+0000 7f81cf400640 1 -- 172.21.15.27:0/3784830666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8038040 con 0x7f81d80aada0 2024-03-28T19:18:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.906+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d814fe70 con 0x7f81d80aada0 2024-03-28T19:18:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.906+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d814bd00 con 0x7f81d80aada0 2024-03-28T19:18:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.907+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d8067dd0 con 0x7f81d80aada0 2024-03-28T19:18:31.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.908+0000 7f81cf400640 1 -- 172.21.15.27:0/3784830666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f81c8046070 con 0x7f81d80aada0 2024-03-28T19:18:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.909+0000 7f81cf400640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81b406bc00 0x7f81b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.909+0000 7f81cf400640 1 -- 172.21.15.27:0/3784830666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f81c80ad2d0 con 0x7f81d80aada0 2024-03-28T19:18:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.910+0000 7f81cde00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81b406bc00 0x7f81b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:31.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.910+0000 7f81cde00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81b406bc00 0x7f81b406e0c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f81d815fec0 tx=0x7f81c0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:31.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:31.914+0000 7f81cf400640 1 -- 172.21.15.27:0/3784830666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f81c807b0b0 con 0x7f81d80aada0 2024-03-28T19:18:32.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.160+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81d80c4e20 con 0x7f81d80aada0 2024-03-28T19:18:32.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.161+0000 7f81cf400640 1 -- 172.21.15.27:0/3784830666 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f81c807a9b0 con 0x7f81d80aada0 2024-03-28T19:18:32.162 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:32.162 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81b406bc00 msgr2=0x7f81b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81b406bc00 0x7f81b406e0c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f81d815fec0 tx=0x7f81c0009040 comp rx=0 tx=0).stop 2024-03-28T19:18:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d80aada0 msgr2=0x7f81d814c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d80aada0 0x7f81d814c730 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f81c8000c00 tx=0x7f81c80046c0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 shutdown_connections 2024-03-28T19:18:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f81b406bc00 0x7f81b406e0c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81d81551a0 0x7f81d81eec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81d80c5580 0x7f81d81eae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 --2- 172.21.15.27:0/3784830666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81d80aada0 0x7f81d814c730 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.164+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 >> 172.21.15.27:0/3784830666 conn(0x7f81d806b0c0 msgr2=0x7f81d806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.165+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 shutdown_connections 2024-03-28T19:18:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.165+0000 7f81dca00640 1 -- 172.21.15.27:0/3784830666 wait complete. 2024-03-28T19:18:32.185 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:18:32.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:32.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.514+0000 7fe309200640 1 Processor -- start 2024-03-28T19:18:32.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.515+0000 7fe309200640 1 -- start start 2024-03-28T19:18:32.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.516+0000 7fe309200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe30406e400 0x7fe30406eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:32.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.516+0000 7fe309200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3041556b0 0x7fe304155b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:32.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.516+0000 7fe309200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3041589f0 con 0x7fe304152c20 2024-03-28T19:18:32.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.516+0000 7fe309200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3041598e0 con 0x7fe3041556b0 2024-03-28T19:18:32.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.516+0000 7fe309200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe304158cd0 con 0x7fe30406e400 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.517+0000 7fe303400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe304152c20 0x7fe3040c1ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45156/0 (socket says 172.21.15.27:45156) 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.517+0000 7fe303400640 1 -- 172.21.15.27:0/3234687915 learned_addr learned my addr 172.21.15.27:0/3234687915 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.517+0000 7fe303e00640 1 --2- 172.21.15.27:0/3234687915 >> v2:172.21.15.27:3300/0 conn(0x7fe3041556b0 0x7fe304155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.517+0000 7fe302a00640 1 --2- 172.21.15.27:0/3234687915 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe30406e400 0x7fe30406eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.518+0000 7fe302000640 1 -- 172.21.15.27:0/3234687915 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682047824 0 0) 0x7fe3041589f0 con 0x7fe304152c20 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.518+0000 7fe302000640 1 -- 172.21.15.27:0/3234687915 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e8003520 con 0x7fe304152c20 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.518+0000 7fe303e00640 1 -- 172.21.15.27:0/3234687915 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe30406e400 msgr2=0x7fe30406eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.518+0000 7fe303e00640 1 --2- 172.21.15.27:0/3234687915 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe30406e400 0x7fe30406eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.518+0000 7fe303e00640 1 -- 172.21.15.27:0/3234687915 >> v1:172.21.15.27:6789/0 conn(0x7fe304152c20 legacy=0x7fe3040c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:32.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.518+0000 7fe303e00640 1 -- 172.21.15.27:0/3234687915 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304158fb0 con 0x7fe3041556b0 2024-03-28T19:18:32.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.519+0000 7fe303e00640 1 --2- 172.21.15.27:0/3234687915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304155b40 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0097a0 tx=0x7fe2fc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856309babc9d93c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe302000640 1 -- 172.21.15.27:0/3234687915 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc031780 con 0x7fe3041556b0 2024-03-28T19:18:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe302000640 1 -- 172.21.15.27:0/3234687915 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc037070 con 0x7fe3041556b0 2024-03-28T19:18:32.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe302000640 1 -- 172.21.15.27:0/3234687915 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc031b40 con 0x7fe3041556b0 2024-03-28T19:18:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe309200640 1 -- 172.21.15.27:0/3234687915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 msgr2=0x7fe304155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe309200640 1 --2- 172.21.15.27:0/3234687915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304155b40 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0097a0 tx=0x7fe2fc02b820 comp rx=0 tx=0).stop 2024-03-28T19:18:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe309200640 1 -- 172.21.15.27:0/3234687915 shutdown_connections 2024-03-28T19:18:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe309200640 1 --2- 172.21.15.27:0/3234687915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304155b40 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe309200640 1 --2- 172.21.15.27:0/3234687915 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe30406e400 0x7fe30406eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.520+0000 7fe309200640 1 -- 172.21.15.27:0/3234687915 >> 172.21.15.27:0/3234687915 conn(0x7fe30406ced0 msgr2=0x7fe30406d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.521+0000 7fe309200640 1 -- 172.21.15.27:0/3234687915 shutdown_connections 2024-03-28T19:18:32.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.521+0000 7fe309200640 1 -- 172.21.15.27:0/3234687915 wait complete. 2024-03-28T19:18:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.522+0000 7fe309200640 1 Processor -- start 2024-03-28T19:18:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.523+0000 7fe309200640 1 -- start start 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.523+0000 7fe309200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe30406e400 0x7fe3041ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.523+0000 7fe309200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe304152c20 0x7fe3041ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.523+0000 7fe309200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe309200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040c23f0 con 0x7fe3041556b0 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe309200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe304159570 con 0x7fe30406e400 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe309200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040c2570 con 0x7fe304152c20 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54652/0 (socket says 172.21.15.27:54652) 2024-03-28T19:18:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303e00640 1 -- 172.21.15.27:0/1889518079 learned_addr learned my addr 172.21.15.27:0/1889518079 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303400640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe30406e400 0x7fe3041ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303e00640 1 -- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe304152c20 msgr2=0x7fe3041ec0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe302a00640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe304152c20 0x7fe3041ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303e00640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe304152c20 0x7fe3041ec0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303e00640 1 -- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe30406e400 msgr2=0x7fe3041ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303e00640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe30406e400 0x7fe3041ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303e00640 1 -- 172.21.15.27:0/1889518079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304121680 con 0x7fe3041556b0 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe303400640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe30406e400 0x7fe3041ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.524+0000 7fe302a00640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe304152c20 0x7fe3041ec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.525+0000 7fe303e00640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304071b10 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc034040 tx=0x7fe2fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.525+0000 7fe2fb400640 1 -- 172.21.15.27:0/1889518079 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc032610 con 0x7fe3041556b0 2024-03-28T19:18:32.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.525+0000 7fe2fb400640 1 -- 172.21.15.27:0/1889518079 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc0327b0 con 0x7fe3041556b0 2024-03-28T19:18:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.525+0000 7fe2fb400640 1 -- 172.21.15.27:0/1889518079 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc032ba0 con 0x7fe3041556b0 2024-03-28T19:18:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.525+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3041efb40 con 0x7fe3041556b0 2024-03-28T19:18:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.525+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe304072050 con 0x7fe3041556b0 2024-03-28T19:18:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.527+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe304067dd0 con 0x7fe3041556b0 2024-03-28T19:18:32.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.528+0000 7fe2fb400640 1 -- 172.21.15.27:0/1889518079 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe2fc04c060 con 0x7fe3041556b0 2024-03-28T19:18:32.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.529+0000 7fe2fb400640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2e006bc00 0x7fe2e006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.529+0000 7fe2fb400640 1 -- 172.21.15.27:0/1889518079 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7fe2fc0ad080 con 0x7fe3041556b0 2024-03-28T19:18:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.529+0000 7fe303400640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2e006bc00 0x7fe2e006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.530+0000 7fe303400640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2e006bc00 0x7fe2e006e0c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4005880 tx=0x7fe2f4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.533+0000 7fe2fb400640 1 -- 172.21.15.27:0/1889518079 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe304121680 con 0x7fe3041556b0 2024-03-28T19:18:32.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.782+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe304158810 con 0x7fe3041556b0 2024-03-28T19:18:32.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.784+0000 7fe2fb400640 1 -- 172.21.15.27:0/1889518079 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fe2fc040070 con 0x7fe3041556b0 2024-03-28T19:18:32.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:32.784 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:32.784 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2e006bc00 msgr2=0x7fe2e006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2e006bc00 0x7fe2e006e0c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4005880 tx=0x7fe2f4009040 comp rx=0 tx=0).stop 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 msgr2=0x7fe304071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304071b10 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc034040 tx=0x7fe2fc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 shutdown_connections 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe2e006bc00 0x7fe2e006e0c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3041556b0 0x7fe304071b10 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe304152c20 0x7fe3041ec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 --2- 172.21.15.27:0/1889518079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe30406e400 0x7fe3041ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.786+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 >> 172.21.15.27:0/1889518079 conn(0x7fe30406ced0 msgr2=0x7fe3040ac2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.787+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 shutdown_connections 2024-03-28T19:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:32.787+0000 7fe309200640 1 -- 172.21.15.27:0/1889518079 wait complete. 2024-03-28T19:18:32.807 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:18:34.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:34.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.426+0000 7f469a400640 1 Processor -- start 2024-03-28T19:18:34.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.426+0000 7f469a400640 1 -- start start 2024-03-28T19:18:34.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.428+0000 7f469a400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46940c69a0 0x7f46940c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.428+0000 7f469a400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4694153a30 0x7f4694153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.428+0000 7f469a400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4694150ec0 con 0x7f4694150ae0 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.428+0000 7f469a400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46941543c0 con 0x7f46940c69a0 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.428+0000 7f469a400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469406dc70 con 0x7f4694153a30 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.428+0000 7f468be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46940c69a0 0x7f46940c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.428+0000 7f468be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46940c69a0 0x7f46940c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54654/0 (socket says 172.21.15.27:54654) 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.428+0000 7f468be00640 1 -- 172.21.15.27:0/3266177154 learned_addr learned my addr 172.21.15.27:0/3266177154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.429+0000 7f4699a00640 1 --2- 172.21.15.27:0/3266177154 >> v2:172.21.15.155:3300/0 conn(0x7f4694153a30 0x7f4694153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.429+0000 7f468be00640 1 -- 172.21.15.27:0/3266177154 >> v2:172.21.15.155:3300/0 conn(0x7f4694153a30 msgr2=0x7f4694153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.429+0000 7f468be00640 1 --2- 172.21.15.27:0/3266177154 >> v2:172.21.15.155:3300/0 conn(0x7f4694153a30 0x7f4694153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.429+0000 7f468be00640 1 -- 172.21.15.27:0/3266177154 >> v1:172.21.15.155:6789/0 conn(0x7f4694150ae0 legacy=0x7f46940a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.429+0000 7f468be00640 1 -- 172.21.15.27:0/3266177154 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4694153550 con 0x7f46940c69a0 2024-03-28T19:18:34.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.429+0000 7f4699a00640 1 --2- 172.21.15.27:0/3266177154 >> v2:172.21.15.155:3300/0 conn(0x7f4694153a30 0x7f4694153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:34.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.430+0000 7f468be00640 1 --2- 172.21.15.27:0/3266177154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46940c69a0 0x7f46940c6e30 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f46940c0af0 tx=0x7f467c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72aa0a88a88ca2d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:34.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.430+0000 7f4693e00640 1 -- 172.21.15.27:0/3266177154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c02e070 con 0x7f46940c69a0 2024-03-28T19:18:34.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.430+0000 7f4693e00640 1 -- 172.21.15.27:0/3266177154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467c037070 con 0x7f46940c69a0 2024-03-28T19:18:34.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.430+0000 7f4693e00640 1 -- 172.21.15.27:0/3266177154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c031a00 con 0x7f46940c69a0 2024-03-28T19:18:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.431+0000 7f469a400640 1 -- 172.21.15.27:0/3266177154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46940c69a0 msgr2=0x7f46940c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.431+0000 7f469a400640 1 --2- 172.21.15.27:0/3266177154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46940c69a0 0x7f46940c6e30 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f46940c0af0 tx=0x7f467c02b880 comp rx=0 tx=0).stop 2024-03-28T19:18:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.431+0000 7f469a400640 1 -- 172.21.15.27:0/3266177154 shutdown_connections 2024-03-28T19:18:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.431+0000 7f469a400640 1 --2- 172.21.15.27:0/3266177154 >> v2:172.21.15.155:3300/0 conn(0x7f4694153a30 0x7f4694153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.431+0000 7f469a400640 1 --2- 172.21.15.27:0/3266177154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46940c69a0 0x7f46940c6e30 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.431+0000 7f469a400640 1 -- 172.21.15.27:0/3266177154 >> 172.21.15.27:0/3266177154 conn(0x7f469406b0c0 msgr2=0x7f469406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.431+0000 7f469a400640 1 -- 172.21.15.27:0/3266177154 shutdown_connections 2024-03-28T19:18:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.431+0000 7f469a400640 1 -- 172.21.15.27:0/3266177154 wait complete. 2024-03-28T19:18:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.433+0000 7f469a400640 1 Processor -- start 2024-03-28T19:18:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.433+0000 7f469a400640 1 -- start start 2024-03-28T19:18:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f469a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46940c69a0 0x7f46941dec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f469a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4694150ae0 0x7f4694072620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f469a400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4694153a30 0x7f4694072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f469a400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469406e4d0 con 0x7f4694150ae0 2024-03-28T19:18:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f469a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469406e1f0 con 0x7f46940c69a0 2024-03-28T19:18:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f469a400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469406e7b0 con 0x7f4694153a30 2024-03-28T19:18:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f468be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4694150ae0 0x7f4694072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f468be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4694150ae0 0x7f4694072620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54666/0 (socket says 172.21.15.27:54666) 2024-03-28T19:18:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.434+0000 7f468be00640 1 -- 172.21.15.27:0/3280089487 learned_addr learned my addr 172.21.15.27:0/3280089487 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f4699a00640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4694153a30 0x7f4694072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f4699000640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46940c69a0 0x7f46941dec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f468be00640 1 -- 172.21.15.27:0/3280089487 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4694153a30 msgr2=0x7f4694072b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f468be00640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4694153a30 0x7f4694072b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f468be00640 1 -- 172.21.15.27:0/3280089487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46940c69a0 msgr2=0x7f46941dec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f468be00640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46940c69a0 0x7f46941dec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f468be00640 1 -- 172.21.15.27:0/3280089487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46941de770 con 0x7f4694150ae0 2024-03-28T19:18:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f468be00640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4694150ae0 0x7f4694072620 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f467c000c00 tx=0x7f467c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.435+0000 7f4691600640 1 -- 172.21.15.27:0/3280089487 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c032880 con 0x7f4694150ae0 2024-03-28T19:18:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.436+0000 7f4691600640 1 -- 172.21.15.27:0/3280089487 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467c032a20 con 0x7f4694150ae0 2024-03-28T19:18:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.436+0000 7f4691600640 1 -- 172.21.15.27:0/3280089487 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c032e10 con 0x7f4694150ae0 2024-03-28T19:18:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.436+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4694071860 con 0x7f4694150ae0 2024-03-28T19:18:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.436+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4694071e20 con 0x7f4694150ae0 2024-03-28T19:18:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.437+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4694067dd0 con 0x7f4694150ae0 2024-03-28T19:18:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.438+0000 7f4691600640 1 -- 172.21.15.27:0/3280089487 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f467c04c060 con 0x7f4694150ae0 2024-03-28T19:18:34.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.439+0000 7f4691600640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f467006bc00 0x7f467006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:34.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.439+0000 7f4691600640 1 -- 172.21.15.27:0/3280089487 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f467c0ad590 con 0x7f4694150ae0 2024-03-28T19:18:34.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.439+0000 7f4699000640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f467006bc00 0x7f467006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:34.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.440+0000 7f4699000640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f467006bc00 0x7f467006e0c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4684007fd0 tx=0x7f4684009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:34.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.444+0000 7f4691600640 1 -- 172.21.15.27:0/3280089487 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f467c040070 con 0x7f4694150ae0 2024-03-28T19:18:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.686+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f469406da50 con 0x7f4694150ae0 2024-03-28T19:18:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.687+0000 7f4691600640 1 -- 172.21.15.27:0/3280089487 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f4694150ec0 con 0x7f4694150ae0 2024-03-28T19:18:34.687 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:34.687 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:34.688 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f467006bc00 msgr2=0x7f467006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f467006bc00 0x7f467006e0c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4684007fd0 tx=0x7f4684009040 comp rx=0 tx=0).stop 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4694150ae0 msgr2=0x7f4694072620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4694150ae0 0x7f4694072620 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f467c000c00 tx=0x7f467c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 shutdown_connections 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f467006bc00 0x7f467006e0c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4694153a30 0x7f4694072b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4694150ae0 0x7f4694072620 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 --2- 172.21.15.27:0/3280089487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46940c69a0 0x7f46941dec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 >> 172.21.15.27:0/3280089487 conn(0x7f469406b0c0 msgr2=0x7f46940c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 shutdown_connections 2024-03-28T19:18:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:34.690+0000 7f469a400640 1 -- 172.21.15.27:0/3280089487 wait complete. 2024-03-28T19:18:34.711 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:18:34.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.040+0000 7f2d00a00640 1 Processor -- start 2024-03-28T19:18:35.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.040+0000 7f2d00a00640 1 -- start start 2024-03-28T19:18:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.042+0000 7f2d00a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2cfc150ae0 0x7f2cfc0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.042+0000 7f2d00a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc150ec0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.042+0000 7f2d00a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc06dc70 con 0x7f2cfc153a30 2024-03-28T19:18:35.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.042+0000 7f2d00a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc153550 con 0x7f2cfc150ae0 2024-03-28T19:18:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.042+0000 7f2cf3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2cfc0a61a0 0x7f2cfc0c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45160/0 (socket says 172.21.15.27:45160) 2024-03-28T19:18:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.042+0000 7f2cf3e00640 1 -- 172.21.15.27:0/4116934186 learned_addr learned my addr 172.21.15.27:0/4116934186 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.042+0000 7f2cfb400640 1 --2- 172.21.15.27:0/4116934186 >> v2:172.21.15.27:3300/0 conn(0x7f2cfc150ae0 0x7f2cfc0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.043+0000 7f2cfaa00640 1 -- 172.21.15.27:0/4116934186 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109839933 0 0) 0x7f2ce4001110 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.043+0000 7f2cfaa00640 1 -- 172.21.15.27:0/4116934186 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce0003520 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.043+0000 7f2cfb400640 1 -- 172.21.15.27:0/4116934186 >> v1:172.21.15.155:6789/0 conn(0x7f2cfc153a30 legacy=0x7f2cfc15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:35.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.043+0000 7f2cfb400640 1 -- 172.21.15.27:0/4116934186 >> v1:172.21.15.27:6789/0 conn(0x7f2cfc0a61a0 legacy=0x7f2cfc0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:35.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.043+0000 7f2cfb400640 1 -- 172.21.15.27:0/4116934186 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc06df10 con 0x7f2cfc150ae0 2024-03-28T19:18:35.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.044+0000 7f2cfb400640 1 --2- 172.21.15.27:0/4116934186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc150ae0 0x7f2cfc0a5c60 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0097a0 tx=0x7f2cec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f18280ef9ad318 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:35.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.044+0000 7f2cfaa00640 1 -- 172.21.15.27:0/4116934186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec0315a0 con 0x7f2cfc150ae0 2024-03-28T19:18:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.044+0000 7f2cfaa00640 1 -- 172.21.15.27:0/4116934186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec037070 con 0x7f2cfc150ae0 2024-03-28T19:18:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.044+0000 7f2cfaa00640 1 -- 172.21.15.27:0/4116934186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec031960 con 0x7f2cfc150ae0 2024-03-28T19:18:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.045+0000 7f2d00a00640 1 -- 172.21.15.27:0/4116934186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc150ae0 msgr2=0x7f2cfc0a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:35.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.045+0000 7f2d00a00640 1 --2- 172.21.15.27:0/4116934186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc150ae0 0x7f2cfc0a5c60 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0097a0 tx=0x7f2cec02b840 comp rx=0 tx=0).stop 2024-03-28T19:18:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.045+0000 7f2d00a00640 1 -- 172.21.15.27:0/4116934186 shutdown_connections 2024-03-28T19:18:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.045+0000 7f2d00a00640 1 --2- 172.21.15.27:0/4116934186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc150ae0 0x7f2cfc0a5c60 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.045+0000 7f2d00a00640 1 -- 172.21.15.27:0/4116934186 >> 172.21.15.27:0/4116934186 conn(0x7f2cfc06b0c0 msgr2=0x7f2cfc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.045+0000 7f2d00a00640 1 -- 172.21.15.27:0/4116934186 shutdown_connections 2024-03-28T19:18:35.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.045+0000 7f2d00a00640 1 -- 172.21.15.27:0/4116934186 wait complete. 2024-03-28T19:18:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.047+0000 7f2d00a00640 1 Processor -- start 2024-03-28T19:18:35.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.047+0000 7f2d00a00640 1 -- start start 2024-03-28T19:18:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2d00a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc0a61a0 0x7f2cfc0a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2d00a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cfc150ae0 0x7f2cfc1e9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2d00a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2cfc153a30 0x7f2cfc147a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2d00a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc06e7b0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2d00a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc06e4d0 con 0x7f2cfc150ae0 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2d00a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0c70b0 con 0x7f2cfc153a30 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2cfb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc0a61a0 0x7f2cfc0a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2cfb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc0a61a0 0x7f2cfc0a8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54684/0 (socket says 172.21.15.27:54684) 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.048+0000 7f2cfb400640 1 -- 172.21.15.27:0/1576676096 learned_addr learned my addr 172.21.15.27:0/1576676096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cfbe00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2cfc153a30 0x7f2cfc147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cf3e00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cfc150ae0 0x7f2cfc1e9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cfb400640 1 -- 172.21.15.27:0/1576676096 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2cfc153a30 msgr2=0x7f2cfc147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cfb400640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2cfc153a30 0x7f2cfc147a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cfb400640 1 -- 172.21.15.27:0/1576676096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cfc150ae0 msgr2=0x7f2cfc1e9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cfb400640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cfc150ae0 0x7f2cfc1e9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cfb400640 1 -- 172.21.15.27:0/1576676096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc1e98e0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cfb400640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc0a61a0 0x7f2cfc0a8d30 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f2cec000c00 tx=0x7f2cec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:35.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cf3400640 1 -- 172.21.15.27:0/1576676096 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec032630 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cf3400640 1 -- 172.21.15.27:0/1576676096 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec0327d0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2cf3400640 1 -- 172.21.15.27:0/1576676096 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec032bc0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.049+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc1e9bc0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.050+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc071090 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.051+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc067dd0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.052+0000 7f2cf3400640 1 -- 172.21.15.27:0/1576676096 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2cec04d3f0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.053+0000 7f2cf3400640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2cd806bbd0 0x7f2cd806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.053+0000 7f2cf3400640 1 -- 172.21.15.27:0/1576676096 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f2cec02f030 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.053+0000 7f2cf3e00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2cd806bbd0 0x7f2cd806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:35.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.054+0000 7f2cf3e00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2cd806bbd0 0x7f2cd806e090 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40012b0 tx=0x7f2ce4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:35.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.057+0000 7f2cf3400640 1 -- 172.21.15.27:0/1576676096 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2cec040070 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.301+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cfc069640 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.303+0000 7f2cf3400640 1 -- 172.21.15.27:0/1576676096 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f2cec07a8f0 con 0x7f2cfc0a61a0 2024-03-28T19:18:35.303 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:35.303 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:35.303 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.305+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2cd806bbd0 msgr2=0x7f2cd806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.305+0000 7f2d00a00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2cd806bbd0 0x7f2cd806e090 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40012b0 tx=0x7f2ce4009040 comp rx=0 tx=0).stop 2024-03-28T19:18:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.305+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc0a61a0 msgr2=0x7f2cfc0a8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.305+0000 7f2d00a00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc0a61a0 0x7f2cfc0a8d30 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f2cec000c00 tx=0x7f2cec0046c0 comp rx=0 tx=0).stop 2024-03-28T19:18:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.306+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 shutdown_connections 2024-03-28T19:18:35.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.306+0000 7f2d00a00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2cd806bbd0 0x7f2cd806e090 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.306+0000 7f2d00a00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2cfc153a30 0x7f2cfc147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.306+0000 7f2d00a00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cfc150ae0 0x7f2cfc1e9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.306+0000 7f2d00a00640 1 --2- 172.21.15.27:0/1576676096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cfc0a61a0 0x7f2cfc0a8d30 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.306+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 >> 172.21.15.27:0/1576676096 conn(0x7f2cfc06b0c0 msgr2=0x7f2cfc0c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.306+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 shutdown_connections 2024-03-28T19:18:35.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:35.306+0000 7f2d00a00640 1 -- 172.21.15.27:0/1576676096 wait complete. 2024-03-28T19:18:35.327 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:18:36.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:36.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.954+0000 7fd2b7e00640 1 Processor -- start 2024-03-28T19:18:36.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.954+0000 7fd2b7e00640 1 -- start start 2024-03-28T19:18:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2b7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2b80c69a0 0x7fd2b80c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2b7e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b8150ec0 con 0x7fd2b8153a30 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2b7e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b806dc70 con 0x7fd2b8150ae0 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2b7e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b8153550 con 0x7fd2b80c69a0 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2ae000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2b80c69a0 0x7fd2b80c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2b7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd2b8153a30 0x7fd2b815dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54334/0 (socket says 172.21.15.27:54334) 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2b7400640 1 -- 172.21.15.27:0/1506465595 learned_addr learned my addr 172.21.15.27:0/1506465595 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2ae000640 1 -- 172.21.15.27:0/1506465595 >> v1:172.21.15.155:6789/0 conn(0x7fd2b8150ae0 legacy=0x7fd2b80a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2ae000640 1 -- 172.21.15.27:0/1506465595 >> v1:172.21.15.27:6789/0 conn(0x7fd2b8153a30 legacy=0x7fd2b815dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.956+0000 7fd2ae000640 1 -- 172.21.15.27:0/1506465595 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b806df10 con 0x7fd2b80c69a0 2024-03-28T19:18:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.957+0000 7fd2ae000640 1 --2- 172.21.15.27:0/1506465595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b80c6e30 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00097a0 tx=0x7fd2a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1849e316d68ed3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.957+0000 7fd2b6000640 1 -- 172.21.15.27:0/1506465595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00315a0 con 0x7fd2b80c69a0 2024-03-28T19:18:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.957+0000 7fd2b6000640 1 -- 172.21.15.27:0/1506465595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a0037070 con 0x7fd2b80c69a0 2024-03-28T19:18:36.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.957+0000 7fd2b6000640 1 -- 172.21.15.27:0/1506465595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0031960 con 0x7fd2b80c69a0 2024-03-28T19:18:36.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.958+0000 7fd2b7e00640 1 -- 172.21.15.27:0/1506465595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 msgr2=0x7fd2b80c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:36.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.958+0000 7fd2b7e00640 1 --2- 172.21.15.27:0/1506465595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b80c6e30 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00097a0 tx=0x7fd2a002b840 comp rx=0 tx=0).stop 2024-03-28T19:18:36.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.958+0000 7fd2b7e00640 1 -- 172.21.15.27:0/1506465595 shutdown_connections 2024-03-28T19:18:36.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.958+0000 7fd2b7e00640 1 --2- 172.21.15.27:0/1506465595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b80c6e30 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:36.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.958+0000 7fd2b7e00640 1 -- 172.21.15.27:0/1506465595 >> 172.21.15.27:0/1506465595 conn(0x7fd2b806b0c0 msgr2=0x7fd2b806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:36.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.958+0000 7fd2b7e00640 1 -- 172.21.15.27:0/1506465595 shutdown_connections 2024-03-28T19:18:36.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.958+0000 7fd2b7e00640 1 -- 172.21.15.27:0/1506465595 wait complete. 2024-03-28T19:18:36.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.960+0000 7fd2b7e00640 1 Processor -- start 2024-03-28T19:18:36.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.960+0000 7fd2b7e00640 1 -- start start 2024-03-28T19:18:36.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b81de090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b7e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2b8150ae0 0x7fd2b8072b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2b8153a30 0x7fd2b81ec950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b806e7b0 con 0x7fd2b80c69a0 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b806e4d0 con 0x7fd2b8153a30 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b7e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80a6370 con 0x7fd2b8150ae0 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b81de090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b81de090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45744/0 (socket says 172.21.15.27:45744) 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.961+0000 7fd2b6a00640 1 -- 172.21.15.27:0/3714404143 learned_addr learned my addr 172.21.15.27:0/3714404143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2b6a00640 1 -- 172.21.15.27:0/3714404143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2b8150ae0 msgr2=0x7fd2b8072b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2b6a00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2b8150ae0 0x7fd2b8072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:36.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2b6a00640 1 -- 172.21.15.27:0/3714404143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2b8153a30 msgr2=0x7fd2b81ec950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:36.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2b6a00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2b8153a30 0x7fd2b81ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:36.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2b6a00640 1 -- 172.21.15.27:0/3714404143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b81ed340 con 0x7fd2b80c69a0 2024-03-28T19:18:36.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2b6a00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b81de090 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800c460 tx=0x7fd2a800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:36.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2af400640 1 -- 172.21.15.27:0/3714404143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8016020 con 0x7fd2b80c69a0 2024-03-28T19:18:36.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2af400640 1 -- 172.21.15.27:0/3714404143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a8012870 con 0x7fd2b80c69a0 2024-03-28T19:18:36.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2af400640 1 -- 172.21.15.27:0/3714404143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8012c60 con 0x7fd2b80c69a0 2024-03-28T19:18:36.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b8072300 con 0x7fd2b80c69a0 2024-03-28T19:18:36.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.962+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b81de5d0 con 0x7fd2b80c69a0 2024-03-28T19:18:36.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.964+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b8069640 con 0x7fd2b80c69a0 2024-03-28T19:18:36.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.965+0000 7fd2af400640 1 -- 172.21.15.27:0/3714404143 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd2a8019020 con 0x7fd2b80c69a0 2024-03-28T19:18:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.966+0000 7fd2af400640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd29406bbd0 0x7fd29406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.966+0000 7fd2af400640 1 -- 172.21.15.27:0/3714404143 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7fd2a808b6e0 con 0x7fd2b80c69a0 2024-03-28T19:18:36.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.966+0000 7fd2ae000640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd29406bbd0 0x7fd29406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:36.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.967+0000 7fd2ae000640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd29406bbd0 0x7fd29406e090 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0036000 tx=0x7fd2a0030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:36.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:36.971+0000 7fd2af400640 1 -- 172.21.15.27:0/3714404143 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd2a8020030 con 0x7fd2b80c69a0 2024-03-28T19:18:37.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.212+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2b8150630 con 0x7fd2b80c69a0 2024-03-28T19:18:37.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.213+0000 7fd2af400640 1 -- 172.21.15.27:0/3714404143 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fd2a8019310 con 0x7fd2b80c69a0 2024-03-28T19:18:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:37.213 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:37.214 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:37.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd29406bbd0 msgr2=0x7fd29406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd29406bbd0 0x7fd29406e090 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0036000 tx=0x7fd2a0030000 comp rx=0 tx=0).stop 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 msgr2=0x7fd2b81de090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b81de090 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800c460 tx=0x7fd2a800c920 comp rx=0 tx=0).stop 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 shutdown_connections 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd29406bbd0 0x7fd29406e090 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2b8153a30 0x7fd2b81ec950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2b8150ae0 0x7fd2b8072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 --2- 172.21.15.27:0/3714404143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2b80c69a0 0x7fd2b81de090 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 >> 172.21.15.27:0/3714404143 conn(0x7fd2b806b0c0 msgr2=0x7fd2b80c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 shutdown_connections 2024-03-28T19:18:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.216+0000 7fd2b7e00640 1 -- 172.21.15.27:0/3714404143 wait complete. 2024-03-28T19:18:37.233 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:18:37.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:18:37.309 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:18:37.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:37.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8667200640 1 Processor -- start 2024-03-28T19:18:37.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8667200640 1 -- start start 2024-03-28T19:18:37.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8667200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86680c69a0 0x7f86680c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:37.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8667200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8668153a30 0x7f8668153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:37.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8667200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8668150ec0 con 0x7f8668150ae0 2024-03-28T19:18:37.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8667200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866806dc70 con 0x7f8668153a30 2024-03-28T19:18:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8667200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8668153550 con 0x7f86680c69a0 2024-03-28T19:18:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8665e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8668150ae0 0x7f86680a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54336/0 (socket says 172.21.15.27:54336) 2024-03-28T19:18:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8665e00640 1 -- 172.21.15.27:0/3499342741 learned_addr learned my addr 172.21.15.27:0/3499342741 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.582+0000 7f8666800640 1 --2- 172.21.15.27:0/3499342741 >> v2:172.21.15.27:3300/0 conn(0x7f8668153a30 0x7f8668153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.583+0000 7f8665400640 1 --2- 172.21.15.27:0/3499342741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86680c69a0 0x7f86680c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.583+0000 7f8666800640 1 -- 172.21.15.27:0/3499342741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86680c69a0 msgr2=0x7f86680c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.583+0000 7f8666800640 1 --2- 172.21.15.27:0/3499342741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86680c69a0 0x7f86680c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.583+0000 7f8664a00640 1 -- 172.21.15.27:0/3499342741 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461478169 0 0) 0x7f8658001110 con 0x7f8668150ae0 2024-03-28T19:18:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.583+0000 7f8666800640 1 -- 172.21.15.27:0/3499342741 >> v1:172.21.15.27:6789/0 conn(0x7f8668150ae0 legacy=0x7f86680a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.583+0000 7f8666800640 1 -- 172.21.15.27:0/3499342741 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866806df10 con 0x7f8668153a30 2024-03-28T19:18:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.583+0000 7f8665400640 1 --2- 172.21.15.27:0/3499342741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86680c69a0 0x7f86680c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.585+0000 7f8666800640 1 --2- 172.21.15.27:0/3499342741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668153a30 0x7f8668153ea0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f86600097a0 tx=0x7f866002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2dfaeb56b26057 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.585+0000 7f8664a00640 1 -- 172.21.15.27:0/3499342741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660031780 con 0x7f8668153a30 2024-03-28T19:18:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.585+0000 7f8664a00640 1 -- 172.21.15.27:0/3499342741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8660037070 con 0x7f8668153a30 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.585+0000 7f8664a00640 1 -- 172.21.15.27:0/3499342741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8660031b40 con 0x7f8668153a30 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.586+0000 7f8667200640 1 -- 172.21.15.27:0/3499342741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668153a30 msgr2=0x7f8668153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.586+0000 7f8667200640 1 --2- 172.21.15.27:0/3499342741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668153a30 0x7f8668153ea0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f86600097a0 tx=0x7f866002b820 comp rx=0 tx=0).stop 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.586+0000 7f8667200640 1 -- 172.21.15.27:0/3499342741 shutdown_connections 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.586+0000 7f8667200640 1 --2- 172.21.15.27:0/3499342741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668153a30 0x7f8668153ea0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.586+0000 7f8667200640 1 --2- 172.21.15.27:0/3499342741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86680c69a0 0x7f86680c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.586+0000 7f8667200640 1 -- 172.21.15.27:0/3499342741 >> 172.21.15.27:0/3499342741 conn(0x7f866806b0c0 msgr2=0x7f866806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.586+0000 7f8667200640 1 -- 172.21.15.27:0/3499342741 shutdown_connections 2024-03-28T19:18:37.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.586+0000 7f8667200640 1 -- 172.21.15.27:0/3499342741 wait complete. 2024-03-28T19:18:37.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.587+0000 7f8667200640 1 Processor -- start 2024-03-28T19:18:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.587+0000 7f8667200640 1 -- start start 2024-03-28T19:18:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8667200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86680c69a0 0x7f86680728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8667200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668150ae0 0x7f86680a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8667200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8668153a30 0x7f866814bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8667200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866806e7b0 con 0x7f8668150ae0 2024-03-28T19:18:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8667200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866806e4d0 con 0x7f86680c69a0 2024-03-28T19:18:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8667200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86680a6370 con 0x7f8668153a30 2024-03-28T19:18:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8665400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668150ae0 0x7f86680a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8665400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668150ae0 0x7f86680a8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45768/0 (socket says 172.21.15.27:45768) 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.588+0000 7f8665400640 1 -- 172.21.15.27:0/1659886902 learned_addr learned my addr 172.21.15.27:0/1659886902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8665e00640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86680c69a0 0x7f86680728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8666800640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8668153a30 0x7f866814bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8665400640 1 -- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8668153a30 msgr2=0x7f866814bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8665400640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8668153a30 0x7f866814bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8665400640 1 -- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86680c69a0 msgr2=0x7f86680728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8665400640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86680c69a0 0x7f86680728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8665400640 1 -- 172.21.15.27:0/1659886902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8668072ff0 con 0x7f8668150ae0 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8665e00640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86680c69a0 0x7f86680728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8666800640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8668153a30 0x7f866814bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8665400640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668150ae0 0x7f86680a8d60 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f865000cf60 tx=0x7f865000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f865e000640 1 -- 172.21.15.27:0/1659886902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865001a070 con 0x7f8668150ae0 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f865e000640 1 -- 172.21.15.27:0/1659886902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865000d840 con 0x7f8668150ae0 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f865e000640 1 -- 172.21.15.27:0/1659886902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865000dc10 con 0x7f8668150ae0 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.590+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86680a7ad0 con 0x7f8668150ae0 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.591+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86680a8090 con 0x7f8668150ae0 2024-03-28T19:18:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.592+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8668067dd0 con 0x7f8668150ae0 2024-03-28T19:18:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.592+0000 7f865e000640 1 -- 172.21.15.27:0/1659886902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f865001d070 con 0x7f8668150ae0 2024-03-28T19:18:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.592+0000 7f865e000640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f864406bbd0 0x7f864406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.594+0000 7f865e000640 1 -- 172.21.15.27:0/1659886902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f865008c0c0 con 0x7f8668150ae0 2024-03-28T19:18:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.594+0000 7f8665e00640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f864406bbd0 0x7f864406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:37.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.594+0000 7f8665e00640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f864406bbd0 0x7f864406e090 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f86580013b0 tx=0x7f8658009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:37.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.597+0000 7f865e000640 1 -- 172.21.15.27:0/1659886902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8650059730 con 0x7f8668150ae0 2024-03-28T19:18:37.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.840+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f866806da50 con 0x7f8668150ae0 2024-03-28T19:18:37.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.841+0000 7f865e000640 1 -- 172.21.15.27:0/1659886902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f865005d7d0 con 0x7f8668150ae0 2024-03-28T19:18:37.841 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:37.841 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:37.841 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.843+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f864406bbd0 msgr2=0x7f864406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.843+0000 7f8667200640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f864406bbd0 0x7f864406e090 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f86580013b0 tx=0x7f8658009040 comp rx=0 tx=0).stop 2024-03-28T19:18:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.843+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668150ae0 msgr2=0x7f86680a8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.843+0000 7f8667200640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668150ae0 0x7f86680a8d60 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f865000cf60 tx=0x7f865000cf90 comp rx=0 tx=0).stop 2024-03-28T19:18:37.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.844+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 shutdown_connections 2024-03-28T19:18:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.844+0000 7f8667200640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f864406bbd0 0x7f864406e090 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.844+0000 7f8667200640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8668153a30 0x7f866814bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.844+0000 7f8667200640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8668150ae0 0x7f86680a8d60 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.844+0000 7f8667200640 1 --2- 172.21.15.27:0/1659886902 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86680c69a0 0x7f86680728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.844+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 >> 172.21.15.27:0/1659886902 conn(0x7f866806b0c0 msgr2=0x7f8668155510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.844+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 shutdown_connections 2024-03-28T19:18:37.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:37.844+0000 7f8667200640 1 -- 172.21.15.27:0/1659886902 wait complete. 2024-03-28T19:18:37.863 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:18:39.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:39.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.477+0000 7f1e28200640 1 Processor -- start 2024-03-28T19:18:39.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.478+0000 7f1e28200640 1 -- start start 2024-03-28T19:18:39.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.479+0000 7f1e28200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e20150c70 0x7f1e200a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:39.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.479+0000 7f1e28200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e200c3690 0x7f1e200c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:39.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.479+0000 7f1e28200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e200c3af0 con 0x7f1e200c6660 2024-03-28T19:18:39.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.479+0000 7f1e28200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e200ab4e0 con 0x7f1e200c3690 2024-03-28T19:18:39.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.479+0000 7f1e28200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200ac1f0 con 0x7f1e20150c70 2024-03-28T19:18:39.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.480+0000 7f1e24e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e200c3690 0x7f1e200c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:39.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.480+0000 7f1e26200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1e200c6660 0x7f1e201534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54352/0 (socket says 172.21.15.27:54352) 2024-03-28T19:18:39.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.480+0000 7f1e24e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e200c3690 0x7f1e200c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45780/0 (socket says 172.21.15.27:45780) 2024-03-28T19:18:39.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.480+0000 7f1e26200640 1 -- 172.21.15.27:0/4178028904 learned_addr learned my addr 172.21.15.27:0/4178028904 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:39.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.480+0000 7f1e25800640 1 --2- 172.21.15.27:0/4178028904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e20150c70 0x7f1e200a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:39.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.480+0000 7f1e24e00640 1 -- 172.21.15.27:0/4178028904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e20150c70 msgr2=0x7f1e200a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:39.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.480+0000 7f1e24e00640 1 --2- 172.21.15.27:0/4178028904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e20150c70 0x7f1e200a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.480+0000 7f1e24e00640 1 -- 172.21.15.27:0/4178028904 >> v1:172.21.15.27:6789/0 conn(0x7f1e200c6660 legacy=0x7f1e201534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.481+0000 7f1e24e00640 1 -- 172.21.15.27:0/4178028904 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e200c3d90 con 0x7f1e200c3690 2024-03-28T19:18:39.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.481+0000 7f1e1fe00640 1 -- 172.21.15.27:0/4178028904 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618523477 0 0) 0x7f1e200c3af0 con 0x7f1e200c6660 2024-03-28T19:18:39.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.482+0000 7f1e24e00640 1 --2- 172.21.15.27:0/4178028904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e200c3690 0x7f1e200c6120 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0097a0 tx=0x7f1e0c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3dca7f427bdd256 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:39.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.482+0000 7f1e1fe00640 1 -- 172.21.15.27:0/4178028904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c0315a0 con 0x7f1e200c3690 2024-03-28T19:18:39.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.482+0000 7f1e1fe00640 1 -- 172.21.15.27:0/4178028904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c037070 con 0x7f1e200c3690 2024-03-28T19:18:39.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.482+0000 7f1e1fe00640 1 -- 172.21.15.27:0/4178028904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c031960 con 0x7f1e200c3690 2024-03-28T19:18:39.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.483+0000 7f1e28200640 1 -- 172.21.15.27:0/4178028904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e200c3690 msgr2=0x7f1e200c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:39.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.483+0000 7f1e28200640 1 --2- 172.21.15.27:0/4178028904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e200c3690 0x7f1e200c6120 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0097a0 tx=0x7f1e0c02b840 comp rx=0 tx=0).stop 2024-03-28T19:18:39.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.483+0000 7f1e28200640 1 -- 172.21.15.27:0/4178028904 shutdown_connections 2024-03-28T19:18:39.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.483+0000 7f1e28200640 1 --2- 172.21.15.27:0/4178028904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e200c3690 0x7f1e200c6120 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.483+0000 7f1e28200640 1 --2- 172.21.15.27:0/4178028904 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e20150c70 0x7f1e200a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.483+0000 7f1e28200640 1 -- 172.21.15.27:0/4178028904 >> 172.21.15.27:0/4178028904 conn(0x7f1e2006b9d0 msgr2=0x7f1e200c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:39.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.483+0000 7f1e28200640 1 -- 172.21.15.27:0/4178028904 shutdown_connections 2024-03-28T19:18:39.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.483+0000 7f1e28200640 1 -- 172.21.15.27:0/4178028904 wait complete. 2024-03-28T19:18:39.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.485+0000 7f1e28200640 1 Processor -- start 2024-03-28T19:18:39.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.485+0000 7f1e28200640 1 -- start start 2024-03-28T19:18:39.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.486+0000 7f1e28200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e200c3690 0x7f1e2006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:39.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.486+0000 7f1e28200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e200c6660 0x7f1e201eec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:39.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.486+0000 7f1e28200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e20150c70 0x7f1e200c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.486+0000 7f1e28200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200abcd0 con 0x7f1e20150c70 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.486+0000 7f1e28200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200ab9f0 con 0x7f1e200c3690 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.486+0000 7f1e28200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200c1810 con 0x7f1e200c6660 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e20150c70 0x7f1e200c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e20150c70 0x7f1e200c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45794/0 (socket says 172.21.15.27:45794) 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 -- 172.21.15.27:0/1890355952 learned_addr learned my addr 172.21.15.27:0/1890355952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e24e00640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e200c6660 0x7f1e201eec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e25800640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e200c3690 0x7f1e2006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:39.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 -- 172.21.15.27:0/1890355952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e200c6660 msgr2=0x7f1e201eec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e200c6660 0x7f1e201eec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 -- 172.21.15.27:0/1890355952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e200c3690 msgr2=0x7f1e2006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e200c3690 0x7f1e2006e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 -- 172.21.15.27:0/1890355952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2006e700 con 0x7f1e20150c70 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.487+0000 7f1e26200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e20150c70 0x7f1e200c02a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800c7a0 tx=0x7f1e1800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.488+0000 7f1e1d600640 1 -- 172.21.15.27:0/1890355952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18018070 con 0x7f1e20150c70 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.488+0000 7f1e1d600640 1 -- 172.21.15.27:0/1890355952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1800a640 con 0x7f1e20150c70 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.488+0000 7f1e1d600640 1 -- 172.21.15.27:0/1890355952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1800aab0 con 0x7f1e20150c70 2024-03-28T19:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.488+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e20151700 con 0x7f1e20150c70 2024-03-28T19:18:39.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.488+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e20151cc0 con 0x7f1e20150c70 2024-03-28T19:18:39.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.489+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2004f400 con 0x7f1e20150c70 2024-03-28T19:18:39.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.490+0000 7f1e1d600640 1 -- 172.21.15.27:0/1890355952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1e1800ac50 con 0x7f1e20150c70 2024-03-28T19:18:39.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.491+0000 7f1e1d600640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e0006bbd0 0x7f1e0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:39.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.491+0000 7f1e1d600640 1 -- 172.21.15.27:0/1890355952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f1e1808aef0 con 0x7f1e20150c70 2024-03-28T19:18:39.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.491+0000 7f1e25800640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e0006bbd0 0x7f1e0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:39.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.492+0000 7f1e25800640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e0006bbd0 0x7f1e0006e090 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009970 tx=0x7f1e10008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:39.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.496+0000 7f1e1d600640 1 -- 172.21.15.27:0/1890355952 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1e18058c10 con 0x7f1e20150c70 2024-03-28T19:18:39.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.741+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e200c5f00 con 0x7f1e20150c70 2024-03-28T19:18:39.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.743+0000 7f1e1d600640 1 -- 172.21.15.27:0/1890355952 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f1e18058510 con 0x7f1e20150c70 2024-03-28T19:18:39.743 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:39.743 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e0006bbd0 msgr2=0x7f1e0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e0006bbd0 0x7f1e0006e090 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009970 tx=0x7f1e10008040 comp rx=0 tx=0).stop 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e20150c70 msgr2=0x7f1e200c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e20150c70 0x7f1e200c02a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800c7a0 tx=0x7f1e1800cc60 comp rx=0 tx=0).stop 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 shutdown_connections 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e0006bbd0 0x7f1e0006e090 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e20150c70 0x7f1e200c02a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e200c6660 0x7f1e201eec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 --2- 172.21.15.27:0/1890355952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e200c3690 0x7f1e2006e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 >> 172.21.15.27:0/1890355952 conn(0x7f1e2006b9d0 msgr2=0x7f1e200c2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.746+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 shutdown_connections 2024-03-28T19:18:39.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:39.747+0000 7f1e28200640 1 -- 172.21.15.27:0/1890355952 wait complete. 2024-03-28T19:18:39.765 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:18:39.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.093+0000 7f87b7200640 1 Processor -- start 2024-03-28T19:18:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.093+0000 7f87b7200640 1 -- start start 2024-03-28T19:18:40.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.095+0000 7f87b7200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87b81551a0 0x7f87b80c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.095+0000 7f87b7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80aada0 0x7f87b80c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.095+0000 7f87b7200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b80c26d0 con 0x7f87b80c5580 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.095+0000 7f87b7200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b8156870 con 0x7f87b81551a0 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.095+0000 7f87b7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b8155cf0 con 0x7f87b80aada0 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.095+0000 7f87b5e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87b81551a0 0x7f87b80c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.096+0000 7f87b5e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87b81551a0 0x7f87b80c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45808/0 (socket says 172.21.15.27:45808) 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.096+0000 7f87b5e00640 1 -- 172.21.15.27:0/843428640 learned_addr learned my addr 172.21.15.27:0/843428640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.096+0000 7f87b6800640 1 --2- 172.21.15.27:0/843428640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80aada0 0x7f87b80c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:40.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.096+0000 7f87b5e00640 1 -- 172.21.15.27:0/843428640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80aada0 msgr2=0x7f87b80c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:40.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.096+0000 7f87b5e00640 1 --2- 172.21.15.27:0/843428640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80aada0 0x7f87b80c2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.096+0000 7f87b5e00640 1 -- 172.21.15.27:0/843428640 >> v1:172.21.15.155:6789/0 conn(0x7f87b80c5580 legacy=0x7f87b80c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:40.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.096+0000 7f87b5e00640 1 -- 172.21.15.27:0/843428640 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b8155fd0 con 0x7f87b81551a0 2024-03-28T19:18:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.097+0000 7f87b5e00640 1 --2- 172.21.15.27:0/843428640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b80c5040 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009870 tx=0x7f87a802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb52d5b989528175 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.097+0000 7f87b4a00640 1 -- 172.21.15.27:0/843428640 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8038070 con 0x7f87b81551a0 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b4a00640 1 -- 172.21.15.27:0/843428640 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a8031740 con 0x7f87b81551a0 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b4a00640 1 -- 172.21.15.27:0/843428640 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a802f050 con 0x7f87b81551a0 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b7200640 1 -- 172.21.15.27:0/843428640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 msgr2=0x7f87b80c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b7200640 1 --2- 172.21.15.27:0/843428640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b80c5040 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009870 tx=0x7f87a802b7e0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b7200640 1 -- 172.21.15.27:0/843428640 shutdown_connections 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b7200640 1 --2- 172.21.15.27:0/843428640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80aada0 0x7f87b80c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b7200640 1 --2- 172.21.15.27:0/843428640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b80c5040 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b7200640 1 -- 172.21.15.27:0/843428640 >> 172.21.15.27:0/843428640 conn(0x7f87b806b0c0 msgr2=0x7f87b806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.098+0000 7f87b7200640 1 -- 172.21.15.27:0/843428640 shutdown_connections 2024-03-28T19:18:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.099+0000 7f87b7200640 1 -- 172.21.15.27:0/843428640 wait complete. 2024-03-28T19:18:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.101+0000 7f87b7200640 1 Processor -- start 2024-03-28T19:18:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.101+0000 7f87b7200640 1 -- start start 2024-03-28T19:18:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87b80aada0 0x7f87b80a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80c5580 0x7f87b81eb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b81eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b81559a0 con 0x7f87b81551a0 2024-03-28T19:18:40.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b8156590 con 0x7f87b80aada0 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b8155580 con 0x7f87b80c5580 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b81eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b81eb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45818/0 (socket says 172.21.15.27:45818) 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.102+0000 7f87b6800640 1 -- 172.21.15.27:0/3644804134 learned_addr learned my addr 172.21.15.27:0/3644804134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b5400640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80c5580 0x7f87b81eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b5e00640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87b80aada0 0x7f87b80a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b6800640 1 -- 172.21.15.27:0/3644804134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80c5580 msgr2=0x7f87b81eb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b6800640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80c5580 0x7f87b81eb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b6800640 1 -- 172.21.15.27:0/3644804134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87b80aada0 msgr2=0x7f87b80a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b6800640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87b80aada0 0x7f87b80a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b6800640 1 -- 172.21.15.27:0/3644804134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b81e0810 con 0x7f87b81551a0 2024-03-28T19:18:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b6800640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b81eb5a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f87b0007e30 tx=0x7f87b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87ae000640 1 -- 172.21.15.27:0/3644804134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0014620 con 0x7f87b81551a0 2024-03-28T19:18:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87ae000640 1 -- 172.21.15.27:0/3644804134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b0019070 con 0x7f87b81551a0 2024-03-28T19:18:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87ae000640 1 -- 172.21.15.27:0/3644804134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0014a00 con 0x7f87b81551a0 2024-03-28T19:18:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.103+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b8071450 con 0x7f87b81551a0 2024-03-28T19:18:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.104+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b80719d0 con 0x7f87b81551a0 2024-03-28T19:18:40.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.105+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b8067dd0 con 0x7f87b81551a0 2024-03-28T19:18:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.106+0000 7f87ae000640 1 -- 172.21.15.27:0/3644804134 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f87b0014ba0 con 0x7f87b81551a0 2024-03-28T19:18:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.107+0000 7f87ae000640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f879806bbd0 0x7f879806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.107+0000 7f87ae000640 1 -- 172.21.15.27:0/3644804134 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f87b008c1a0 con 0x7f87b81551a0 2024-03-28T19:18:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.107+0000 7f87b5e00640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f879806bbd0 0x7f879806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.108+0000 7f87b5e00640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f879806bbd0 0x7f879806e090 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f87a8035000 tx=0x7f87a8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.111+0000 7f87ae000640 1 -- 172.21.15.27:0/3644804134 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f87b0059e60 con 0x7f87b81551a0 2024-03-28T19:18:40.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.352+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87b80c4e20 con 0x7f87b81551a0 2024-03-28T19:18:40.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.354+0000 7f87ae000640 1 -- 172.21.15.27:0/3644804134 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f87b0059760 con 0x7f87b81551a0 2024-03-28T19:18:40.353 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:40.353 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:40.354 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f879806bbd0 msgr2=0x7f879806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f879806bbd0 0x7f879806e090 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f87a8035000 tx=0x7f87a8008030 comp rx=0 tx=0).stop 2024-03-28T19:18:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 msgr2=0x7f87b81eb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b81eb5a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f87b0007e30 tx=0x7f87b000e4b0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 shutdown_connections 2024-03-28T19:18:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f879806bbd0 0x7f879806e090 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87b81551a0 0x7f87b81eb5a0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87b80c5580 0x7f87b81eb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 --2- 172.21.15.27:0/3644804134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87b80aada0 0x7f87b80a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.356+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 >> 172.21.15.27:0/3644804134 conn(0x7f87b806b0c0 msgr2=0x7f87b80c5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.357+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 shutdown_connections 2024-03-28T19:18:40.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:40.357+0000 7f87b7200640 1 -- 172.21.15.27:0/3644804134 wait complete. 2024-03-28T19:18:40.377 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:18:41.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:42.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.003+0000 7fb7d3200640 1 Processor -- start 2024-03-28T19:18:42.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.004+0000 7fb7d3200640 1 -- start start 2024-03-28T19:18:42.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.005+0000 7fb7d3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7d40c58c0 0x7fb7d40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.005+0000 7fb7d3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40be5d0 0x7fb7d4155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.005+0000 7fb7d3200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d4156230 con 0x7fb7d40c8ed0 2024-03-28T19:18:42.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.005+0000 7fb7d3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d4158bd0 con 0x7fb7d40c58c0 2024-03-28T19:18:42.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.005+0000 7fb7d3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4158eb0 con 0x7fb7d40be5d0 2024-03-28T19:18:42.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.005+0000 7fb7d1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7d40c58c0 0x7fb7d40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.006+0000 7fb7d1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7d40c58c0 0x7fb7d40c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45820/0 (socket says 172.21.15.27:45820) 2024-03-28T19:18:42.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.006+0000 7fb7d1e00640 1 -- 172.21.15.27:0/858896669 learned_addr learned my addr 172.21.15.27:0/858896669 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:42.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.006+0000 7fb7d2800640 1 --2- 172.21.15.27:0/858896669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40be5d0 0x7fb7d4155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.006+0000 7fb7d1e00640 1 -- 172.21.15.27:0/858896669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40be5d0 msgr2=0x7fb7d4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:42.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.006+0000 7fb7d1e00640 1 --2- 172.21.15.27:0/858896669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40be5d0 0x7fb7d4155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.006+0000 7fb7d1e00640 1 -- 172.21.15.27:0/858896669 >> v1:172.21.15.155:6789/0 conn(0x7fb7d40c8ed0 legacy=0x7fb7d41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:42.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.006+0000 7fb7d1e00640 1 -- 172.21.15.27:0/858896669 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4159190 con 0x7fb7d40c58c0 2024-03-28T19:18:42.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.007+0000 7fb7d1e00640 1 --2- 172.21.15.27:0/858896669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d40c5ca0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009870 tx=0x7fb7c402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f84fe2b8aac1b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:42.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.008+0000 7fb7d1400640 1 -- 172.21.15.27:0/858896669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c40317c0 con 0x7fb7d40c58c0 2024-03-28T19:18:42.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.008+0000 7fb7d1400640 1 -- 172.21.15.27:0/858896669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c4037070 con 0x7fb7d40c58c0 2024-03-28T19:18:42.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.008+0000 7fb7d1400640 1 -- 172.21.15.27:0/858896669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4031b80 con 0x7fb7d40c58c0 2024-03-28T19:18:42.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.009+0000 7fb7d3200640 1 -- 172.21.15.27:0/858896669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 msgr2=0x7fb7d40c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.009+0000 7fb7d3200640 1 --2- 172.21.15.27:0/858896669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d40c5ca0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009870 tx=0x7fb7c402b860 comp rx=0 tx=0).stop 2024-03-28T19:18:42.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.009+0000 7fb7d3200640 1 -- 172.21.15.27:0/858896669 shutdown_connections 2024-03-28T19:18:42.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.009+0000 7fb7d3200640 1 --2- 172.21.15.27:0/858896669 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40be5d0 0x7fb7d4155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.009+0000 7fb7d3200640 1 --2- 172.21.15.27:0/858896669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d40c5ca0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.009+0000 7fb7d3200640 1 -- 172.21.15.27:0/858896669 >> 172.21.15.27:0/858896669 conn(0x7fb7d406b6f0 msgr2=0x7fb7d40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:42.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.009+0000 7fb7d3200640 1 -- 172.21.15.27:0/858896669 shutdown_connections 2024-03-28T19:18:42.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.009+0000 7fb7d3200640 1 -- 172.21.15.27:0/858896669 wait complete. 2024-03-28T19:18:42.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.011+0000 7fb7d3200640 1 Processor -- start 2024-03-28T19:18:42.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.011+0000 7fb7d3200640 1 -- start start 2024-03-28T19:18:42.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.012+0000 7fb7d3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40be5d0 0x7fb7d40c2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.012+0000 7fb7d3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d41544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.012+0000 7fb7d3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40c8ed0 0x7fb7d40c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.012+0000 7fb7d3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d40b0820 con 0x7fb7d40c58c0 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.012+0000 7fb7d3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d4159750 con 0x7fb7d40be5d0 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7d3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d40b09a0 con 0x7fb7d40c8ed0 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7cbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d41544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7cbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d41544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45826/0 (socket says 172.21.15.27:45826) 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7cbe00640 1 -- 172.21.15.27:0/702793 learned_addr learned my addr 172.21.15.27:0/702793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7d1e00640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40be5d0 0x7fb7d40c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7d2800640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40c8ed0 0x7fb7d40c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7cbe00640 1 -- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40c8ed0 msgr2=0x7fb7d40c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7cbe00640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40c8ed0 0x7fb7d40c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7cbe00640 1 -- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40be5d0 msgr2=0x7fb7d40c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7cbe00640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40be5d0 0x7fb7d40c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7cbe00640 1 -- 172.21.15.27:0/702793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d40c2ae0 con 0x7fb7d40c58c0 2024-03-28T19:18:42.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.013+0000 7fb7d2800640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40c8ed0 0x7fb7d40c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:42.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.014+0000 7fb7cbe00640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d41544c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00b3c0 tx=0x7fb7bc00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:42.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.014+0000 7fb7ca000640 1 -- 172.21.15.27:0/702793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc011830 con 0x7fb7d40c58c0 2024-03-28T19:18:42.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.014+0000 7fb7ca000640 1 -- 172.21.15.27:0/702793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc016070 con 0x7fb7d40c58c0 2024-03-28T19:18:42.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.014+0000 7fb7ca000640 1 -- 172.21.15.27:0/702793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc011c10 con 0x7fb7d40c58c0 2024-03-28T19:18:42.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.014+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d40c2800 con 0x7fb7d40c58c0 2024-03-28T19:18:42.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.014+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d4153cc0 con 0x7fb7d40c58c0 2024-03-28T19:18:42.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.015+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d4069640 con 0x7fb7d40c58c0 2024-03-28T19:18:42.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.016+0000 7fb7ca000640 1 -- 172.21.15.27:0/702793 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb7bc01b460 con 0x7fb7d40c58c0 2024-03-28T19:18:42.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.017+0000 7fb7ca000640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7b006bbd0 0x7fb7b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.017+0000 7fb7ca000640 1 -- 172.21.15.27:0/702793 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7fb7bc08b310 con 0x7fb7d40c58c0 2024-03-28T19:18:42.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.018+0000 7fb7d1e00640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7b006bbd0 0x7fb7b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.018+0000 7fb7d1e00640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7b006bbd0 0x7fb7b006e090 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4036000 tx=0x7fb7c403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:42.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.022+0000 7fb7ca000640 1 -- 172.21.15.27:0/702793 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb7bc020020 con 0x7fb7d40c58c0 2024-03-28T19:18:42.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.272+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d40ad5b0 con 0x7fb7d40c58c0 2024-03-28T19:18:42.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.273+0000 7fb7ca000640 1 -- 172.21.15.27:0/702793 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fb7d40c2ae0 con 0x7fb7d40c58c0 2024-03-28T19:18:42.273 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:42.273 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:42.273 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7b006bbd0 msgr2=0x7fb7b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7b006bbd0 0x7fb7b006e090 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4036000 tx=0x7fb7c403d040 comp rx=0 tx=0).stop 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 msgr2=0x7fb7d41544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d41544c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00b3c0 tx=0x7fb7bc00b880 comp rx=0 tx=0).stop 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 shutdown_connections 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb7b006bbd0 0x7fb7b006e090 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb7d40c8ed0 0x7fb7d40c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d40c58c0 0x7fb7d41544c0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 --2- 172.21.15.27:0/702793 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7d40be5d0 0x7fb7d40c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 >> 172.21.15.27:0/702793 conn(0x7fb7d406b6f0 msgr2=0x7fb7d40c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.276+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 shutdown_connections 2024-03-28T19:18:42.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.277+0000 7fb7d3200640 1 -- 172.21.15.27:0/702793 wait complete. 2024-03-28T19:18:42.296 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:18:42.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.614+0000 7f65de000640 1 Processor -- start 2024-03-28T19:18:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.614+0000 7f65de000640 1 -- start start 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65de000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65d80c58c0 0x7f65d80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65de000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f65d80c8ed0 0x7f65d81555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65de000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65de000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d8158bd0 con 0x7f65d80c58c0 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65de000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d8158eb0 con 0x7f65d80c8ed0 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65de000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d8159190 con 0x7f65d80be5d0 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65dcc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65d80c58c0 0x7f65d80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65dcc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f65d80c58c0 0x7f65d80c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45830/0 (socket says 172.21.15.27:45830) 2024-03-28T19:18:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65dcc00640 1 -- 172.21.15.27:0/433879419 learned_addr learned my addr 172.21.15.27:0/433879419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65dcc00640 1 -- 172.21.15.27:0/433879419 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 msgr2=0x7f65d8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65dd600640 1 --2- 172.21.15.27:0/433879419 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65cfe00640 1 --2- 172.21.15.27:0/433879419 >> v2:172.21.15.155:3300/0 conn(0x7f65d80c8ed0 0x7f65d81555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.616+0000 7f65dcc00640 1 --2- 172.21.15.27:0/433879419 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.617+0000 7f65dcc00640 1 -- 172.21.15.27:0/433879419 >> v2:172.21.15.155:3300/0 conn(0x7f65d80c8ed0 msgr2=0x7f65d81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.617+0000 7f65dcc00640 1 --2- 172.21.15.27:0/433879419 >> v2:172.21.15.155:3300/0 conn(0x7f65d80c8ed0 0x7f65d81555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.617+0000 7f65dcc00640 1 -- 172.21.15.27:0/433879419 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d8159470 con 0x7f65d80c58c0 2024-03-28T19:18:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.617+0000 7f65cfe00640 1 --2- 172.21.15.27:0/433879419 >> v2:172.21.15.155:3300/0 conn(0x7f65d80c8ed0 0x7f65d81555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.618+0000 7f65dcc00640 1 --2- 172.21.15.27:0/433879419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c58c0 0x7f65d80c5ca0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f65d80c83d0 tx=0x7f65c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463345d607900077 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.618+0000 7f65d7e00640 1 -- 172.21.15.27:0/433879419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c802e070 con 0x7f65d80c58c0 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.618+0000 7f65d7e00640 1 -- 172.21.15.27:0/433879419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c8037070 con 0x7f65d80c58c0 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.618+0000 7f65d7e00640 1 -- 172.21.15.27:0/433879419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8031a00 con 0x7f65d80c58c0 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 -- 172.21.15.27:0/433879419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c58c0 msgr2=0x7f65d80c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 --2- 172.21.15.27:0/433879419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c58c0 0x7f65d80c5ca0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f65d80c83d0 tx=0x7f65c802b880 comp rx=0 tx=0).stop 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 -- 172.21.15.27:0/433879419 shutdown_connections 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 --2- 172.21.15.27:0/433879419 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 --2- 172.21.15.27:0/433879419 >> v2:172.21.15.155:3300/0 conn(0x7f65d80c8ed0 0x7f65d81555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 --2- 172.21.15.27:0/433879419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c58c0 0x7f65d80c5ca0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 -- 172.21.15.27:0/433879419 >> 172.21.15.27:0/433879419 conn(0x7f65d806b6f0 msgr2=0x7f65d80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 -- 172.21.15.27:0/433879419 shutdown_connections 2024-03-28T19:18:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.619+0000 7f65de000640 1 -- 172.21.15.27:0/433879419 wait complete. 2024-03-28T19:18:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.621+0000 7f65de000640 1 Processor -- start 2024-03-28T19:18:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.621+0000 7f65de000640 1 -- start start 2024-03-28T19:18:42.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65de000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65de000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d80c58c0 0x7f65d81e1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65de000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c8ed0 0x7f65d80c2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65de000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d80b09a0 con 0x7f65d80c8ed0 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65de000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d80b0820 con 0x7f65d80c58c0 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65de000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d80ad7d0 con 0x7f65d80be5d0 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65dd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c8ed0 0x7f65d80c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65dd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c8ed0 0x7f65d80c2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45844/0 (socket says 172.21.15.27:45844) 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65dd600640 1 -- 172.21.15.27:0/903890712 learned_addr learned my addr 172.21.15.27:0/903890712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65dcc00640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.623+0000 7f65cfe00640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d80c58c0 0x7f65d81e1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65dd600640 1 -- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 msgr2=0x7f65d8151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65dd600640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65dd600640 1 -- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d80c58c0 msgr2=0x7f65d81e1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65dd600640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d80c58c0 0x7f65d81e1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65dd600640 1 -- 172.21.15.27:0/903890712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d81ec3d0 con 0x7f65d80c8ed0 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65dcc00640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65dd600640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c8ed0 0x7f65d80c2220 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f65d000ccf0 tx=0x7f65d000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65d5600640 1 -- 172.21.15.27:0/903890712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0003b90 con 0x7f65d80c8ed0 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65d5600640 1 -- 172.21.15.27:0/903890712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0003d30 con 0x7f65d80c8ed0 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65d5600640 1 -- 172.21.15.27:0/903890712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0004240 con 0x7f65d80c8ed0 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.624+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d81ec6b0 con 0x7f65d80c8ed0 2024-03-28T19:18:42.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.625+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d81510e0 con 0x7f65d80c8ed0 2024-03-28T19:18:42.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.626+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d81512e0 con 0x7f65d80c8ed0 2024-03-28T19:18:42.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.627+0000 7f65d5600640 1 -- 172.21.15.27:0/903890712 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f65d00043e0 con 0x7f65d80c8ed0 2024-03-28T19:18:42.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.628+0000 7f65d5600640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bbd0 0x7f65b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:42.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.628+0000 7f65d5600640 1 -- 172.21.15.27:0/903890712 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f65d008c360 con 0x7f65d80c8ed0 2024-03-28T19:18:42.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.628+0000 7f65dcc00640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bbd0 0x7f65b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:42.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.629+0000 7f65dcc00640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bbd0 0x7f65b406e090 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f65c8036000 tx=0x7f65c803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:42.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.632+0000 7f65d5600640 1 -- 172.21.15.27:0/903890712 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f65d81512e0 con 0x7f65d80c8ed0 2024-03-28T19:18:42.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.872+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65d80c5040 con 0x7f65d80c8ed0 2024-03-28T19:18:42.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.873+0000 7f65d5600640 1 -- 172.21.15.27:0/903890712 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f65d005a020 con 0x7f65d80c8ed0 2024-03-28T19:18:42.872 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:42.873 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:42.873 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:42.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bbd0 msgr2=0x7f65b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bbd0 0x7f65b406e090 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f65c8036000 tx=0x7f65c803d040 comp rx=0 tx=0).stop 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c8ed0 msgr2=0x7f65d80c2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c8ed0 0x7f65d80c2220 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f65d000ccf0 tx=0x7f65d000a660 comp rx=0 tx=0).stop 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 shutdown_connections 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f65b406bbd0 0x7f65b406e090 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65d80c8ed0 0x7f65d80c2220 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65d80c58c0 0x7f65d81e1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 --2- 172.21.15.27:0/903890712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65d80be5d0 0x7f65d8151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 >> 172.21.15.27:0/903890712 conn(0x7f65d806b6f0 msgr2=0x7f65d80c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 shutdown_connections 2024-03-28T19:18:42.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:42.876+0000 7f65de000640 1 -- 172.21.15.27:0/903890712 wait complete. 2024-03-28T19:18:42.894 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:18:44.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:44.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.536+0000 7f6ddd800640 1 Processor -- start 2024-03-28T19:18:44.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.536+0000 7f6ddd800640 1 -- start start 2024-03-28T19:18:44.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.538+0000 7f6ddd800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6dd80c5e80 0x7f6dd80c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:44.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.538+0000 7f6ddd800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd806d8a0 con 0x7f6dd80abf60 2024-03-28T19:18:44.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.538+0000 7f6ddd800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd806db80 con 0x7f6dd80c1f50 2024-03-28T19:18:44.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.538+0000 7f6ddd800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd806d650 con 0x7f6dd80c5e80 2024-03-28T19:18:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.538+0000 7f6dd6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6dd80c5e80 0x7f6dd80c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.538+0000 7f6dd6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6dd80c5e80 0x7f6dd80c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45848/0 (socket says 172.21.15.27:45848) 2024-03-28T19:18:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.538+0000 7f6dd6a00640 1 -- 172.21.15.27:0/4049310893 learned_addr learned my addr 172.21.15.27:0/4049310893 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.538+0000 7f6dd7400640 1 --1- 172.21.15.27:0/4049310893 >> v1:172.21.15.27:6789/0 conn(0x7f6dd80abf60 0x7f6dd80aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54366/0 (socket says 172.21.15.27:54366) 2024-03-28T19:18:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.539+0000 7f6dd6a00640 1 -- 172.21.15.27:0/4049310893 >> v1:172.21.15.155:6789/0 conn(0x7f6dd80c1f50 legacy=0x7f6dd80ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.539+0000 7f6dd6a00640 1 -- 172.21.15.27:0/4049310893 >> v1:172.21.15.27:6789/0 conn(0x7f6dd80abf60 legacy=0x7f6dd80aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:44.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.539+0000 7f6dd6a00640 1 -- 172.21.15.27:0/4049310893 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd806de60 con 0x7f6dd80c5e80 2024-03-28T19:18:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.540+0000 7f6dd6a00640 1 --2- 172.21.15.27:0/4049310893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c5e80 0x7f6dd80c2cb0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009e50 tx=0x7f6dc802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319bae5f813bfc4a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.540+0000 7f6dd5600640 1 -- 172.21.15.27:0/4049310893 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc802ca80 con 0x7f6dd80c5e80 2024-03-28T19:18:44.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.540+0000 7f6dd5600640 1 -- 172.21.15.27:0/4049310893 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc802cc20 con 0x7f6dd80c5e80 2024-03-28T19:18:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.540+0000 7f6dd5600640 1 -- 172.21.15.27:0/4049310893 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc80343f0 con 0x7f6dd80c5e80 2024-03-28T19:18:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.541+0000 7f6ddd800640 1 -- 172.21.15.27:0/4049310893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c5e80 msgr2=0x7f6dd80c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:44.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.541+0000 7f6ddd800640 1 --2- 172.21.15.27:0/4049310893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c5e80 0x7f6dd80c2cb0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009e50 tx=0x7f6dc802bee0 comp rx=0 tx=0).stop 2024-03-28T19:18:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.541+0000 7f6ddd800640 1 -- 172.21.15.27:0/4049310893 shutdown_connections 2024-03-28T19:18:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.541+0000 7f6ddd800640 1 --2- 172.21.15.27:0/4049310893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c5e80 0x7f6dd80c2cb0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.541+0000 7f6ddd800640 1 -- 172.21.15.27:0/4049310893 >> 172.21.15.27:0/4049310893 conn(0x7f6dd806b9d0 msgr2=0x7f6dd80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.541+0000 7f6ddd800640 1 -- 172.21.15.27:0/4049310893 shutdown_connections 2024-03-28T19:18:44.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.541+0000 7f6ddd800640 1 -- 172.21.15.27:0/4049310893 wait complete. 2024-03-28T19:18:44.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.543+0000 7f6ddd800640 1 Processor -- start 2024-03-28T19:18:44.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.543+0000 7f6ddd800640 1 -- start start 2024-03-28T19:18:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.544+0000 7f6ddd800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dd80abf60 0x7f6dd80a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.544+0000 7f6ddd800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c1f50 0x7f6dd8151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.544+0000 7f6ddd800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dd80c5e80 0x7f6dd8151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.544+0000 7f6ddd800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd80c6260 con 0x7f6dd80c1f50 2024-03-28T19:18:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.544+0000 7f6ddd800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd806e1d0 con 0x7f6dd80c5e80 2024-03-28T19:18:44.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.544+0000 7f6ddd800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd80c63e0 con 0x7f6dd80abf60 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.544+0000 7f6dd6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c1f50 0x7f6dd8151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c1f50 0x7f6dd8151760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45860/0 (socket says 172.21.15.27:45860) 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6000640 1 -- 172.21.15.27:0/4189099422 learned_addr learned my addr 172.21.15.27:0/4189099422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6a00640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dd80abf60 0x7f6dd80a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd7400640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dd80c5e80 0x7f6dd8151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6000640 1 -- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dd80abf60 msgr2=0x7f6dd80a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6000640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dd80abf60 0x7f6dd80a8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6000640 1 -- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dd80c5e80 msgr2=0x7f6dd8151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6000640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dd80c5e80 0x7f6dd8151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6000640 1 -- 172.21.15.27:0/4189099422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd80bc850 con 0x7f6dd80c1f50 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd7400640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dd80c5e80 0x7f6dd8151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:44.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.545+0000 7f6dd6000640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c1f50 0x7f6dd8151760 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000c470 tx=0x7f6dc000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:44.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.546+0000 7f6dcea00640 1 -- 172.21.15.27:0/4189099422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc00052d0 con 0x7f6dd80c1f50 2024-03-28T19:18:44.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.546+0000 7f6dcea00640 1 -- 172.21.15.27:0/4189099422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc0005490 con 0x7f6dd80c1f50 2024-03-28T19:18:44.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.546+0000 7f6dcea00640 1 -- 172.21.15.27:0/4189099422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0007260 con 0x7f6dd80c1f50 2024-03-28T19:18:44.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.546+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd80bcb30 con 0x7f6dd80c1f50 2024-03-28T19:18:44.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.546+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd80a9520 con 0x7f6dd80c1f50 2024-03-28T19:18:44.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.547+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd804f400 con 0x7f6dd80c1f50 2024-03-28T19:18:44.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.548+0000 7f6dcea00640 1 -- 172.21.15.27:0/4189099422 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6dc0007400 con 0x7f6dd80c1f50 2024-03-28T19:18:44.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.549+0000 7f6dcea00640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6db406bc00 0x7f6db406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:44.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.549+0000 7f6dcea00640 1 -- 172.21.15.27:0/4189099422 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7f6dc008ad30 con 0x7f6dd80c1f50 2024-03-28T19:18:44.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.549+0000 7f6dd6a00640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6db406bc00 0x7f6db406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:44.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.550+0000 7f6dd6a00640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6db406bc00 0x7f6db406e0c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8035000 tx=0x7f6dc8030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.553+0000 7f6dcea00640 1 -- 172.21.15.27:0/4189099422 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6dc00589f0 con 0x7f6dd80c1f50 2024-03-28T19:18:44.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.791+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dd80a6b50 con 0x7f6dd80c1f50 2024-03-28T19:18:44.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.792+0000 7f6dcea00640 1 -- 172.21.15.27:0/4189099422 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f6dc00582f0 con 0x7f6dd80c1f50 2024-03-28T19:18:44.791 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:44.791 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6260,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":6269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":179}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:17:18.718080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6014},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6014":{"gid":6014,"name":"b","rank":0,"incarnation":172,"state":"up:active","state_seq":41,"addr":"172.21.15.155:6835/938167944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":938167944},{"type":"v1","addr":"172.21.15.155:6835","nonce":938167944}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6014,"qdb_cluster":[6014]},"id":1},{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:17:17.706148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6188":{"gid":6188,"name":"c","rank":0,"incarnation":172,"state":"up:active","state_seq":17,"addr":"172.21.15.27:6837/2601264653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":2601264653},{"type":"v1","addr":"172.21.15.27:6837","nonce":2601264653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6188,"qdb_cluster":[6188]},"id":2}]} 2024-03-28T19:18:44.792 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 179 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.794+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6db406bc00 msgr2=0x7f6db406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.794+0000 7f6ddd800640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6db406bc00 0x7f6db406e0c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8035000 tx=0x7f6dc8030000 comp rx=0 tx=0).stop 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.794+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c1f50 msgr2=0x7f6dd8151760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.794+0000 7f6ddd800640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c1f50 0x7f6dd8151760 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000c470 tx=0x7f6dc000c930 comp rx=0 tx=0).stop 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.795+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 shutdown_connections 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.795+0000 7f6ddd800640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6db406bc00 0x7f6db406e0c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.795+0000 7f6ddd800640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dd80c5e80 0x7f6dd8151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.795+0000 7f6ddd800640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dd80c1f50 0x7f6dd8151760 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.795+0000 7f6ddd800640 1 --2- 172.21.15.27:0/4189099422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6dd80abf60 0x7f6dd80a8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.795+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 >> 172.21.15.27:0/4189099422 conn(0x7f6dd806b9d0 msgr2=0x7f6dd80c4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.795+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 shutdown_connections 2024-03-28T19:18:44.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:44.795+0000 7f6ddd800640 1 -- 172.21.15.27:0/4189099422 wait complete. 2024-03-28T19:18:44.814 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:18:44.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:45.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.129+0000 7fd720e00640 1 Processor -- start 2024-03-28T19:18:45.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.129+0000 7fd720e00640 1 -- start start 2024-03-28T19:18:45.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.130+0000 7fd720e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd71c150ae0 0x7fd71c0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.130+0000 7fd720e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c153a30 0x7fd71c153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.130+0000 7fd720e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71c150ec0 con 0x7fd71c0a61a0 2024-03-28T19:18:45.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.130+0000 7fd720e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd71c06dc70 con 0x7fd71c150ae0 2024-03-28T19:18:45.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.130+0000 7fd720e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c153550 con 0x7fd71c153a30 2024-03-28T19:18:45.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd71c0a61a0 0x7fd71c0c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54370/0 (socket says 172.21.15.27:54370) 2024-03-28T19:18:45.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71aa00640 1 -- 172.21.15.27:0/4002741481 learned_addr learned my addr 172.21.15.27:0/4002741481 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:45.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71b400640 1 --2- 172.21.15.27:0/4002741481 >> v2:172.21.15.155:3300/0 conn(0x7fd71c150ae0 0x7fd71c0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71be00640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c153a30 0x7fd71c153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159866696 0 0) 0x7fd70c001490 con 0x7fd71c0a61a0 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd700003520 con 0x7fd71c0a61a0 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2556614511 0 0) 0x7fd700003520 con 0x7fd71c0a61a0 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c153a30 msgr2=0x7fd71c153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71a000640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c153a30 0x7fd71c153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 >> v2:172.21.15.155:3300/0 conn(0x7fd71c150ae0 msgr2=0x7fd71c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71a000640 1 --2- 172.21.15.27:0/4002741481 >> v2:172.21.15.155:3300/0 conn(0x7fd71c150ae0 0x7fd71c0a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.131+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c06df10 con 0x7fd71c0a61a0 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd70c001bc0 con 0x7fd71c0a61a0 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 >> v1:172.21.15.27:6789/0 conn(0x7fd71c0a61a0 legacy=0x7fd71c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71a000640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c153a30 0x7fd7000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71a000640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd700004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71a000640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd700004d40 0x7fd7000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c06dc70 con 0x7fd700003f80 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c153550 con 0x7fd700004d40 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70000ba70 con 0x7fd71c153a30 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71aa00640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd700004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71b400640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c153a30 0x7fd7000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71be00640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd700004d40 0x7fd7000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71aa00640 1 -- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c153a30 msgr2=0x7fd7000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:45.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71aa00640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c153a30 0x7fd7000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71aa00640 1 -- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd700004d40 msgr2=0x7fd7000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:45.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71aa00640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd700004d40 0x7fd7000051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.132+0000 7fd71aa00640 1 -- 172.21.15.27:0/4002741481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd700003520 con 0x7fd700003f80 2024-03-28T19:18:45.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.133+0000 7fd71aa00640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd700004800 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00a980 tx=0x7fd700005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0252f6f5211f33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:45.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.134+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00a5d0 con 0x7fd700003f80 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.134+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c0072f0 con 0x7fd700003f80 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.134+0000 7fd71a000640 1 -- 172.21.15.27:0/4002741481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0076e0 con 0x7fd700003f80 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.134+0000 7fd720e00640 1 -- 172.21.15.27:0/4002741481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 msgr2=0x7fd700004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.134+0000 7fd720e00640 1 --2- 172.21.15.27:0/4002741481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd700004800 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00a980 tx=0x7fd700005830 comp rx=0 tx=0).stop 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.134+0000 7fd71be00640 1 -- 172.21.15.27:0/4002741481 reap_dead start 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.134+0000 7fd720e00640 1 -- 172.21.15.27:0/4002741481 shutdown_connections 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.134+0000 7fd720e00640 1 -- 172.21.15.27:0/4002741481 >> 172.21.15.27:0/4002741481 conn(0x7fd71c06b0c0 msgr2=0x7fd71c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.135+0000 7fd720e00640 1 -- 172.21.15.27:0/4002741481 shutdown_connections 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.135+0000 7fd720e00640 1 -- 172.21.15.27:0/4002741481 wait complete. 2024-03-28T19:18:45.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.136+0000 7fd720e00640 1 Processor -- start 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.136+0000 7fd720e00640 1 -- start start 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.137+0000 7fd720e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd71c0c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.137+0000 7fd720e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c144b50 0x7fd71c1ec0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.137+0000 7fd720e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c1ec600 0x7fd71c070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.137+0000 7fd720e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c06e7b0 con 0x7fd700003f80 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.137+0000 7fd720e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c06e4d0 con 0x7fd71c144b50 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.137+0000 7fd720e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c0c70b0 con 0x7fd71c1ec600 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd71c0c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd71c0c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45892/0 (socket says 172.21.15.27:45892) 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 -- 172.21.15.27:0/2416331894 learned_addr learned my addr 172.21.15.27:0/2416331894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:45.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71aa00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c144b50 0x7fd71c1ec0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71be00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c1ec600 0x7fd71c070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 -- 172.21.15.27:0/2416331894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c1ec600 msgr2=0x7fd71c070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c1ec600 0x7fd71c070770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 -- 172.21.15.27:0/2416331894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c144b50 msgr2=0x7fd71c1ec0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c144b50 0x7fd71c1ec0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 -- 172.21.15.27:0/2416331894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c072850 con 0x7fd700003f80 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.138+0000 7fd71b400640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd71c0c50c0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fd7080116d0 tx=0x7fd708011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.139+0000 7fd713400640 1 -- 172.21.15.27:0/2416331894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70800f9c0 con 0x7fd700003f80 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.139+0000 7fd713400640 1 -- 172.21.15.27:0/2416331894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70800fb60 con 0x7fd700003f80 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.139+0000 7fd713400640 1 -- 172.21.15.27:0/2416331894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708015460 con 0x7fd700003f80 2024-03-28T19:18:45.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.139+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71c072b30 con 0x7fd700003f80 2024-03-28T19:18:45.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.139+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71c0730f0 con 0x7fd700003f80 2024-03-28T19:18:45.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.140+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71c069640 con 0x7fd700003f80 2024-03-28T19:18:45.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.141+0000 7fd713400640 1 -- 172.21.15.27:0/2416331894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd708015600 con 0x7fd700003f80 2024-03-28T19:18:45.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.142+0000 7fd713400640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd6fc06bbd0 0x7fd6fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.142+0000 7fd713400640 1 -- 172.21.15.27:0/2416331894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9303+0+0 (secure 0 0 0) 0x7fd70808c280 con 0x7fd700003f80 2024-03-28T19:18:45.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.143+0000 7fd71aa00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd6fc06bbd0 0x7fd6fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.143+0000 7fd71aa00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd6fc06bbd0 0x7fd6fc06e090 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd70c034000 tx=0x7fd70c009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:45.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.147+0000 7fd713400640 1 -- 172.21.15.27:0/2416331894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd708019030 con 0x7fd700003f80 2024-03-28T19:18:45.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.464+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd71c145b80 con 0x7fd700003f80 2024-03-28T19:18:45.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.465+0000 7fd713400640 1 -- 172.21.15.27:0/2416331894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+4664 (secure 0 0 0) 0x7fd708059fa0 con 0x7fd700003f80 2024-03-28T19:18:45.464 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:45.465 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:45.217784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:45.217785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:45.465 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 180 2024-03-28T19:18:45.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd6fc06bbd0 msgr2=0x7fd6fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd6fc06bbd0 0x7fd6fc06e090 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd70c034000 tx=0x7fd70c009030 comp rx=0 tx=0).stop 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 msgr2=0x7fd71c0c50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd71c0c50c0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fd7080116d0 tx=0x7fd708011b90 comp rx=0 tx=0).stop 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 shutdown_connections 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd6fc06bbd0 0x7fd6fc06e090 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd71c1ec600 0x7fd71c070770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c144b50 0x7fd71c1ec0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 --2- 172.21.15.27:0/2416331894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd700003f80 0x7fd71c0c50c0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 >> 172.21.15.27:0/2416331894 conn(0x7fd71c06b0c0 msgr2=0x7fd71c152370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 shutdown_connections 2024-03-28T19:18:45.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.468+0000 7fd720e00640 1 -- 172.21.15.27:0/2416331894 wait complete. 2024-03-28T19:18:45.487 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-03-28T19:18:45.487 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:18:45.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:45.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.709+0000 7fa013e00640 1 Processor -- start 2024-03-28T19:18:45.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.709+0000 7fa013e00640 1 -- start start 2024-03-28T19:18:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.711+0000 7fa013e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0140a7060 0x7fa0140c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.711+0000 7fa013e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa014152ad0 0x7fa01406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.711+0000 7fa013e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa014072f90 con 0x7fa014155700 2024-03-28T19:18:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.711+0000 7fa013e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa014072d40 con 0x7fa0140a7060 2024-03-28T19:18:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.711+0000 7fa013e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa014073550 con 0x7fa014152ad0 2024-03-28T19:18:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.712+0000 7fa012a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0140a7060 0x7fa0140c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.712+0000 7fa012000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa014155700 0x7fa0140c3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54372/0 (socket says 172.21.15.27:54372) 2024-03-28T19:18:45.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.712+0000 7fa012000640 1 -- 172.21.15.27:0/2002337010 learned_addr learned my addr 172.21.15.27:0/2002337010 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.712+0000 7fa013400640 1 --2- 172.21.15.27:0/2002337010 >> v2:172.21.15.155:3300/0 conn(0x7fa014152ad0 0x7fa01406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.712+0000 7fa012a00640 1 -- 172.21.15.27:0/2002337010 >> v2:172.21.15.155:3300/0 conn(0x7fa014152ad0 msgr2=0x7fa01406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.712+0000 7fa012a00640 1 --2- 172.21.15.27:0/2002337010 >> v2:172.21.15.155:3300/0 conn(0x7fa014152ad0 0x7fa01406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.712+0000 7fa012a00640 1 -- 172.21.15.27:0/2002337010 >> v1:172.21.15.27:6789/0 conn(0x7fa014155700 legacy=0x7fa0140c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.712+0000 7fa012a00640 1 -- 172.21.15.27:0/2002337010 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa014073270 con 0x7fa0140a7060 2024-03-28T19:18:45.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.714+0000 7fa012a00640 1 --2- 172.21.15.27:0/2002337010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0140a7060 0x7fa0140c2530 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa0140c5e60 tx=0x7fa00402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c0b337e76c8ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:45.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.714+0000 7fa011600640 1 -- 172.21.15.27:0/2002337010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004035020 con 0x7fa0140a7060 2024-03-28T19:18:45.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.714+0000 7fa011600640 1 -- 172.21.15.27:0/2002337010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004031830 con 0x7fa0140a7060 2024-03-28T19:18:45.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.714+0000 7fa011600640 1 -- 172.21.15.27:0/2002337010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004031c00 con 0x7fa0140a7060 2024-03-28T19:18:45.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.715+0000 7fa013e00640 1 -- 172.21.15.27:0/2002337010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0140a7060 msgr2=0x7fa0140c2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.715+0000 7fa013e00640 1 --2- 172.21.15.27:0/2002337010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0140a7060 0x7fa0140c2530 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa0140c5e60 tx=0x7fa00402b8d0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.715+0000 7fa013e00640 1 -- 172.21.15.27:0/2002337010 shutdown_connections 2024-03-28T19:18:45.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.715+0000 7fa013e00640 1 --2- 172.21.15.27:0/2002337010 >> v2:172.21.15.155:3300/0 conn(0x7fa014152ad0 0x7fa01406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.715+0000 7fa013e00640 1 --2- 172.21.15.27:0/2002337010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0140a7060 0x7fa0140c2530 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.715+0000 7fa013e00640 1 -- 172.21.15.27:0/2002337010 >> 172.21.15.27:0/2002337010 conn(0x7fa0140c7aa0 msgr2=0x7fa0140c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:45.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.715+0000 7fa013e00640 1 -- 172.21.15.27:0/2002337010 shutdown_connections 2024-03-28T19:18:45.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.715+0000 7fa013e00640 1 -- 172.21.15.27:0/2002337010 wait complete. 2024-03-28T19:18:45.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.717+0000 7fa013e00640 1 Processor -- start 2024-03-28T19:18:45.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.718+0000 7fa013e00640 1 -- start start 2024-03-28T19:18:45.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0140a7060 0x7fa0141ea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa014152ad0 0x7fa01406f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa014155700 0x7fa014146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa014155b60 con 0x7fa014155700 2024-03-28T19:18:45.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0141525b0 con 0x7fa014152ad0 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa014152f20 con 0x7fa0140a7060 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa014155700 0x7fa014146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa014155700 0x7fa014146460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45912/0 (socket says 172.21.15.27:45912) 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.719+0000 7fa013400640 1 -- 172.21.15.27:0/2684375267 learned_addr learned my addr 172.21.15.27:0/2684375267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa012a00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0140a7060 0x7fa0141ea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa012000640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa014152ad0 0x7fa01406f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa013400640 1 -- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0140a7060 msgr2=0x7fa0141ea5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa013400640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0140a7060 0x7fa0141ea5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa013400640 1 -- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa014152ad0 msgr2=0x7fa01406f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa013400640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa014152ad0 0x7fa01406f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa013400640 1 -- 172.21.15.27:0/2684375267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0141508a0 con 0x7fa014155700 2024-03-28T19:18:45.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa012000640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa014152ad0 0x7fa01406f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:18:45.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa012a00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0140a7060 0x7fa0141ea5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:18:45.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.720+0000 7fa013400640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa014155700 0x7fa014146460 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fa00c007e40 tx=0x7fa00c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:45.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.721+0000 7fa00b400640 1 -- 172.21.15.27:0/2684375267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c014620 con 0x7fa014155700 2024-03-28T19:18:45.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.721+0000 7fa00b400640 1 -- 172.21.15.27:0/2684375267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c019070 con 0x7fa014155700 2024-03-28T19:18:45.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.721+0000 7fa00b400640 1 -- 172.21.15.27:0/2684375267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c014a20 con 0x7fa014155700 2024-03-28T19:18:45.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.721+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0141eaef0 con 0x7fa014155700 2024-03-28T19:18:45.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.721+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0141eb1d0 con 0x7fa014155700 2024-03-28T19:18:45.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.722+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01414fa50 con 0x7fa014155700 2024-03-28T19:18:45.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.723+0000 7fa00b400640 1 -- 172.21.15.27:0/2684375267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa00c014bc0 con 0x7fa014155700 2024-03-28T19:18:45.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.724+0000 7fa00b400640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ff406bbd0 0x7f9ff406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:45.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.724+0000 7fa00b400640 1 -- 172.21.15.27:0/2684375267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fa00c08c230 con 0x7fa014155700 2024-03-28T19:18:45.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.725+0000 7fa012a00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ff406bbd0 0x7f9ff406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:45.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.726+0000 7fa012a00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ff406bbd0 0x7f9ff406e090 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa004036000 tx=0x7fa004008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:45.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.729+0000 7fa00b400640 1 -- 172.21.15.27:0/2684375267 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa01414fa50 con 0x7fa014155700 2024-03-28T19:18:45.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.964+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa014069640 con 0x7fa014155700 2024-03-28T19:18:45.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.965+0000 7fa00b400640 1 -- 172.21.15.27:0/2684375267 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+4664 (secure 0 0 0) 0x7fa00c059ea0 con 0x7fa014155700 2024-03-28T19:18:45.964 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:45.964 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:45.217784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:45.217785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:45.965 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 180 2024-03-28T19:18:45.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ff406bbd0 msgr2=0x7f9ff406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ff406bbd0 0x7f9ff406e090 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa004036000 tx=0x7fa004008030 comp rx=0 tx=0).stop 2024-03-28T19:18:45.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa014155700 msgr2=0x7fa014146460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:45.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa014155700 0x7fa014146460 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fa00c007e40 tx=0x7fa00c00e4b0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 shutdown_connections 2024-03-28T19:18:45.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ff406bbd0 0x7f9ff406e090 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa014155700 0x7fa014146460 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa014152ad0 0x7fa01406f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 --2- 172.21.15.27:0/2684375267 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0140a7060 0x7fa0141ea5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:45.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 >> 172.21.15.27:0/2684375267 conn(0x7fa0140c7aa0 msgr2=0x7fa014155ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:45.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.968+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 shutdown_connections 2024-03-28T19:18:45.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:45.969+0000 7fa013e00640 1 -- 172.21.15.27:0/2684375267 wait complete. 2024-03-28T19:18:45.987 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 180, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6269": { "addr": "172.21.15.27:6835/1132521215", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1132521215, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 1132521215, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6269, "incarnation": 180, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:18:45.217784+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6269 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6260": { "addr": "172.21.15.155:6834/1296746732", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 1296746732, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 1296746732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6260, "incarnation": 180, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:18:45.217785+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6260 } } } ], "standbys": [] } 2024-03-28T19:18:46.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:47.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.025+0000 7fd163000640 1 Processor -- start 2024-03-28T19:18:47.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.025+0000 7fd163000640 1 -- start start 2024-03-28T19:18:47.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd163000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd15c150c90 0x7fd15c156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd163000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c0c2430 0x7fd15c156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd163000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c157500 con 0x7fd15c0c5210 2024-03-28T19:18:47.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd163000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c0c2d50 con 0x7fd15c150c90 2024-03-28T19:18:47.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd163000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1543f0 con 0x7fd15c0c2430 2024-03-28T19:18:47.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd15be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd15c0c5210 0x7fd15c150580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41524/0 (socket says 172.21.15.27:41524) 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd15be00640 1 -- 172.21.15.27:0/749080823 learned_addr learned my addr 172.21.15.27:0/749080823 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd153e00640 1 --2- 172.21.15.27:0/749080823 >> v2:172.21.15.155:3300/0 conn(0x7fd15c150c90 0x7fd15c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.027+0000 7fd161000640 1 --2- 172.21.15.27:0/749080823 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c0c2430 0x7fd15c156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.028+0000 7fd15b400640 1 -- 172.21.15.27:0/749080823 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419507076 0 0) 0x7fd15c157500 con 0x7fd15c0c5210 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.028+0000 7fd15b400640 1 -- 172.21.15.27:0/749080823 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd140003520 con 0x7fd15c0c5210 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.028+0000 7fd153e00640 1 -- 172.21.15.27:0/749080823 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c0c2430 msgr2=0x7fd15c156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.028+0000 7fd153e00640 1 --2- 172.21.15.27:0/749080823 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c0c2430 0x7fd15c156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.028+0000 7fd153e00640 1 -- 172.21.15.27:0/749080823 >> v1:172.21.15.27:6789/0 conn(0x7fd15c0c5210 legacy=0x7fd15c150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.028+0000 7fd153e00640 1 -- 172.21.15.27:0/749080823 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c153630 con 0x7fd15c150c90 2024-03-28T19:18:47.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.029+0000 7fd153e00640 1 --2- 172.21.15.27:0/749080823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c150c90 0x7fd15c156a80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd1440097a0 tx=0x7fd14402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=634b4f873d78c01e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:47.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.030+0000 7fd15b400640 1 -- 172.21.15.27:0/749080823 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144035020 con 0x7fd15c150c90 2024-03-28T19:18:47.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.030+0000 7fd15b400640 1 -- 172.21.15.27:0/749080823 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd144031810 con 0x7fd15c150c90 2024-03-28T19:18:47.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.030+0000 7fd15b400640 1 -- 172.21.15.27:0/749080823 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144031be0 con 0x7fd15c150c90 2024-03-28T19:18:47.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.031+0000 7fd163000640 1 -- 172.21.15.27:0/749080823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c150c90 msgr2=0x7fd15c156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.031+0000 7fd163000640 1 --2- 172.21.15.27:0/749080823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c150c90 0x7fd15c156a80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd1440097a0 tx=0x7fd14402b8b0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.031+0000 7fd163000640 1 -- 172.21.15.27:0/749080823 shutdown_connections 2024-03-28T19:18:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.031+0000 7fd163000640 1 --2- 172.21.15.27:0/749080823 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c0c2430 0x7fd15c156fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.031+0000 7fd163000640 1 --2- 172.21.15.27:0/749080823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c150c90 0x7fd15c156a80 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.031+0000 7fd163000640 1 -- 172.21.15.27:0/749080823 >> 172.21.15.27:0/749080823 conn(0x7fd15c06c5f0 msgr2=0x7fd15c06ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.031+0000 7fd163000640 1 -- 172.21.15.27:0/749080823 shutdown_connections 2024-03-28T19:18:47.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.031+0000 7fd163000640 1 -- 172.21.15.27:0/749080823 wait complete. 2024-03-28T19:18:47.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.033+0000 7fd163000640 1 Processor -- start 2024-03-28T19:18:47.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.034+0000 7fd163000640 1 -- start start 2024-03-28T19:18:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.035+0000 7fd163000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0c2430 0x7fd15c1ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.035+0000 7fd163000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c0c5210 0x7fd15c0c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.035+0000 7fd163000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c150c90 0x7fd15c146a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.035+0000 7fd163000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c153ed0 con 0x7fd15c0c2430 2024-03-28T19:18:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.035+0000 7fd163000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c153bf0 con 0x7fd15c0c5210 2024-03-28T19:18:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.035+0000 7fd163000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0c2a00 con 0x7fd15c150c90 2024-03-28T19:18:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0c2430 0x7fd15c1ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0c2430 0x7fd15c1ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48174/0 (socket says 172.21.15.27:48174) 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 -- 172.21.15.27:0/2366151205 learned_addr learned my addr 172.21.15.27:0/2366151205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd153e00640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c0c5210 0x7fd15c0c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd161000640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c150c90 0x7fd15c146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 -- 172.21.15.27:0/2366151205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c150c90 msgr2=0x7fd15c146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c150c90 0x7fd15c146a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 -- 172.21.15.27:0/2366151205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c0c5210 msgr2=0x7fd15c0c16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c0c5210 0x7fd15c0c16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 -- 172.21.15.27:0/2366151205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c1eed60 con 0x7fd15c0c2430 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.036+0000 7fd15be00640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0c2430 0x7fd15c1ee5f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0010e0 tx=0x7fd14c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.037+0000 7fd158c00640 1 -- 172.21.15.27:0/2366151205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c017070 con 0x7fd15c0c2430 2024-03-28T19:18:47.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.037+0000 7fd158c00640 1 -- 172.21.15.27:0/2366151205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c005d00 con 0x7fd15c0c2430 2024-03-28T19:18:47.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.037+0000 7fd158c00640 1 -- 172.21.15.27:0/2366151205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0072b0 con 0x7fd15c0c2430 2024-03-28T19:18:47.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.037+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c1ef040 con 0x7fd15c0c2430 2024-03-28T19:18:47.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.037+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c0c1be0 con 0x7fd15c0c2430 2024-03-28T19:18:47.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.038+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c1ef560 con 0x7fd15c0c2430 2024-03-28T19:18:47.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.040+0000 7fd158c00640 1 -- 172.21.15.27:0/2366151205 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd14c01a070 con 0x7fd15c0c2430 2024-03-28T19:18:47.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.041+0000 7fd158c00640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd13806bc00 0x7fd13806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.041+0000 7fd158c00640 1 -- 172.21.15.27:0/2366151205 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fd14c08ade0 con 0x7fd15c0c2430 2024-03-28T19:18:47.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.042+0000 7fd153e00640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd13806bc00 0x7fd13806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.043+0000 7fd153e00640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd13806bc00 0x7fd13806e0c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd144036000 tx=0x7fd144008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:47.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.047+0000 7fd158c00640 1 -- 172.21.15.27:0/2366151205 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd15c1ef560 con 0x7fd15c0c2430 2024-03-28T19:18:47.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.294+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd15c068970 con 0x7fd15c0c2430 2024-03-28T19:18:47.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.295+0000 7fd158c00640 1 -- 172.21.15.27:0/2366151205 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+4664 (secure 0 0 0) 0x7fd15c068970 con 0x7fd15c0c2430 2024-03-28T19:18:47.295 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:47.295 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:45.217784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:45.217785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 180 2024-03-28T19:18:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd13806bc00 msgr2=0x7fd13806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd13806bc00 0x7fd13806e0c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd144036000 tx=0x7fd144008030 comp rx=0 tx=0).stop 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0c2430 msgr2=0x7fd15c1ee5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0c2430 0x7fd15c1ee5f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0010e0 tx=0x7fd14c00c960 comp rx=0 tx=0).stop 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 shutdown_connections 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd13806bc00 0x7fd13806e0c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd15c150c90 0x7fd15c146a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd15c0c5210 0x7fd15c0c16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 --2- 172.21.15.27:0/2366151205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0c2430 0x7fd15c1ee5f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 >> 172.21.15.27:0/2366151205 conn(0x7fd15c06c5f0 msgr2=0x7fd15c0725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 shutdown_connections 2024-03-28T19:18:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.298+0000 7fd163000640 1 -- 172.21.15.27:0/2366151205 wait complete. 2024-03-28T19:18:47.316 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-03-28T19:18:47.316 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:18:47.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.526+0000 7ffb4b200640 1 Processor -- start 2024-03-28T19:18:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.526+0000 7ffb4b200640 1 -- start start 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.527+0000 7ffb4b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0be5d0 0x7ffb4c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.527+0000 7ffb4b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c158bd0 con 0x7ffb4c0c8ed0 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.528+0000 7ffb4b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c158eb0 con 0x7ffb4c0c58c0 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.528+0000 7ffb4b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c159190 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.528+0000 7ffb49400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffb4c0c8ed0 0x7ffb4c155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41540/0 (socket says 172.21.15.27:41540) 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.528+0000 7ffb49400640 1 -- 172.21.15.27:0/2477994496 learned_addr learned my addr 172.21.15.27:0/2477994496 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.528+0000 7ffb4a800640 1 --2- 172.21.15.27:0/2477994496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0be5d0 0x7ffb4c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.528+0000 7ffb48a00640 1 -- 172.21.15.27:0/2477994496 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743779408 0 0) 0x7ffb4c158bd0 con 0x7ffb4c0c8ed0 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.528+0000 7ffb48a00640 1 -- 172.21.15.27:0/2477994496 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb30003520 con 0x7ffb4c0c8ed0 2024-03-28T19:18:47.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.529+0000 7ffb4a800640 1 -- 172.21.15.27:0/2477994496 >> v1:172.21.15.155:6789/0 conn(0x7ffb4c0c58c0 legacy=0x7ffb4c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.529+0000 7ffb4a800640 1 -- 172.21.15.27:0/2477994496 >> v1:172.21.15.27:6789/0 conn(0x7ffb4c0c8ed0 legacy=0x7ffb4c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.529+0000 7ffb4a800640 1 -- 172.21.15.27:0/2477994496 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c159470 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.530+0000 7ffb4a800640 1 --2- 172.21.15.27:0/2477994496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0be5d0 0x7ffb4c0c5ca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb440097d0 tx=0x7ffb4402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fac72dbe6a13054 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:47.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.531+0000 7ffb48a00640 1 -- 172.21.15.27:0/2477994496 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44035020 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.531+0000 7ffb48a00640 1 -- 172.21.15.27:0/2477994496 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb44031810 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.531+0000 7ffb48a00640 1 -- 172.21.15.27:0/2477994496 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44031bc0 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.531+0000 7ffb4b200640 1 -- 172.21.15.27:0/2477994496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0be5d0 msgr2=0x7ffb4c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.531+0000 7ffb4b200640 1 --2- 172.21.15.27:0/2477994496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0be5d0 0x7ffb4c0c5ca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb440097d0 tx=0x7ffb4402b8b0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.532+0000 7ffb4b200640 1 -- 172.21.15.27:0/2477994496 shutdown_connections 2024-03-28T19:18:47.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.532+0000 7ffb4b200640 1 --2- 172.21.15.27:0/2477994496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0be5d0 0x7ffb4c0c5ca0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.532+0000 7ffb4b200640 1 -- 172.21.15.27:0/2477994496 >> 172.21.15.27:0/2477994496 conn(0x7ffb4c06b6f0 msgr2=0x7ffb4c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:47.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.532+0000 7ffb4b200640 1 -- 172.21.15.27:0/2477994496 shutdown_connections 2024-03-28T19:18:47.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.532+0000 7ffb4b200640 1 -- 172.21.15.27:0/2477994496 wait complete. 2024-03-28T19:18:47.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.533+0000 7ffb4b200640 1 Processor -- start 2024-03-28T19:18:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.533+0000 7ffb4b200640 1 -- start start 2024-03-28T19:18:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.534+0000 7ffb4b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb4c0be5d0 0x7ffb4c152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.534+0000 7ffb4b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb4c0c58c0 0x7ffb4c1deeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.534+0000 7ffb4b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0c8ed0 0x7ffb4c1eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.534+0000 7ffb4b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0b09a0 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.534+0000 7ffb4b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0b0820 con 0x7ffb4c0c58c0 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.534+0000 7ffb4b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0ad7d0 con 0x7ffb4c0c8ed0 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb4c0be5d0 0x7ffb4c152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb4c0be5d0 0x7ffb4c152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48176/0 (socket says 172.21.15.27:48176) 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 -- 172.21.15.27:0/2750460573 learned_addr learned my addr 172.21.15.27:0/2750460573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49400640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb4c0c58c0 0x7ffb4c1deeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb4a800640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0c8ed0 0x7ffb4c1eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 -- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0c8ed0 msgr2=0x7ffb4c1eb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0c8ed0 0x7ffb4c1eb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 -- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb4c0c58c0 msgr2=0x7ffb4c1deeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb4c0c58c0 0x7ffb4c1deeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 -- 172.21.15.27:0/2750460573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0c3140 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb4a800640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0c8ed0 0x7ffb4c1eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.535+0000 7ffb49e00640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb4c0be5d0 0x7ffb4c152470 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00c870 tx=0x7ffb3c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.536+0000 7ffb42000640 1 -- 172.21.15.27:0/2750460573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c003b70 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.536+0000 7ffb42000640 1 -- 172.21.15.27:0/2750460573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c003d30 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.536+0000 7ffb42000640 1 -- 172.21.15.27:0/2750460573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c00aa90 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.536+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c1e17d0 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.536+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c0c2e60 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.538+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c069640 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.538+0000 7ffb42000640 1 -- 172.21.15.27:0/2750460573 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffb3c00ac30 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.539+0000 7ffb42000640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb2806bbd0 0x7ffb2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:47.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.539+0000 7ffb42000640 1 -- 172.21.15.27:0/2750460573 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ffb3c08b940 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.539+0000 7ffb49400640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb2806bbd0 0x7ffb2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:47.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.540+0000 7ffb49400640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb2806bbd0 0x7ffb2806e090 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0cb920 tx=0x7ffb34009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:47.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.544+0000 7ffb42000640 1 -- 172.21.15.27:0/2750460573 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffb3c059530 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.803+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb4c1eb1d0 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.804+0000 7ffb42000640 1 -- 172.21.15.27:0/2750460573 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+4664 (secure 0 0 0) 0x7ffb3c058e30 con 0x7ffb4c0be5d0 2024-03-28T19:18:47.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:47.804 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:45.217784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:45.217785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:47.804 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 180 2024-03-28T19:18:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb2806bbd0 msgr2=0x7ffb2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb2806bbd0 0x7ffb2806e090 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0cb920 tx=0x7ffb34009040 comp rx=0 tx=0).stop 2024-03-28T19:18:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb4c0be5d0 msgr2=0x7ffb4c152470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb4c0be5d0 0x7ffb4c152470 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c00c870 tx=0x7ffb3c00cd30 comp rx=0 tx=0).stop 2024-03-28T19:18:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 shutdown_connections 2024-03-28T19:18:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb2806bbd0 0x7ffb2806e090 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb4c0c8ed0 0x7ffb4c1eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb4c0c58c0 0x7ffb4c1deeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 --2- 172.21.15.27:0/2750460573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb4c0be5d0 0x7ffb4c152470 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.807+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 >> 172.21.15.27:0/2750460573 conn(0x7ffb4c06b6f0 msgr2=0x7ffb4c1e1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.808+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 shutdown_connections 2024-03-28T19:18:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:47.808+0000 7ffb4b200640 1 -- 172.21.15.27:0/2750460573 wait complete. 2024-03-28T19:18:47.823 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 180, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6269": { "addr": "172.21.15.27:6835/1132521215", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1132521215, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 1132521215, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6269, "incarnation": 180, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:18:45.217784+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6269 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6260": { "addr": "172.21.15.155:6834/1296746732", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 1296746732, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 1296746732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6260, "incarnation": 180, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:18:45.217785+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6260 } } } ], "standbys": [] } 2024-03-28T19:18:47.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:48.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.212+0000 7f8b43e00640 1 Processor -- start 2024-03-28T19:18:48.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.212+0000 7f8b43e00640 1 -- start start 2024-03-28T19:18:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.213+0000 7f8b43e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b440c58c0 0x7f8b440c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.213+0000 7f8b43e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440c8ed0 0x7f8b441555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.213+0000 7f8b43e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b44156230 con 0x7f8b440be5d0 2024-03-28T19:18:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.213+0000 7f8b43e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b44158bd0 con 0x7f8b440c58c0 2024-03-28T19:18:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.213+0000 7f8b43e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44158eb0 con 0x7f8b440c8ed0 2024-03-28T19:18:48.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.214+0000 7f8b42a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b440c58c0 0x7f8b440c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:48.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.214+0000 7f8b42000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440c8ed0 0x7f8b441555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:48.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.214+0000 7f8b42a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b440c58c0 0x7f8b440c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48188/0 (socket says 172.21.15.27:48188) 2024-03-28T19:18:48.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.214+0000 7f8b42a00640 1 -- 172.21.15.27:0/3611370366 learned_addr learned my addr 172.21.15.27:0/3611370366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:48.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.214+0000 7f8b42a00640 1 -- 172.21.15.27:0/3611370366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440c8ed0 msgr2=0x7f8b441555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:48.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.214+0000 7f8b42a00640 1 --2- 172.21.15.27:0/3611370366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440c8ed0 0x7f8b441555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.214+0000 7f8b42a00640 1 -- 172.21.15.27:0/3611370366 >> v1:172.21.15.155:6789/0 conn(0x7f8b440be5d0 legacy=0x7f8b44155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:48.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.214+0000 7f8b42a00640 1 -- 172.21.15.27:0/3611370366 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b44159190 con 0x7f8b440c58c0 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.215+0000 7f8b42a00640 1 --2- 172.21.15.27:0/3611370366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c58c0 0x7f8b440c5ca0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009870 tx=0x7f8b3402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68fd1197d929503a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b41600640 1 -- 172.21.15.27:0/3611370366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34035020 con 0x7f8b440c58c0 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b41600640 1 -- 172.21.15.27:0/3611370366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b34031640 con 0x7f8b440c58c0 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b41600640 1 -- 172.21.15.27:0/3611370366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3402d040 con 0x7f8b440c58c0 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b43e00640 1 -- 172.21.15.27:0/3611370366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c58c0 msgr2=0x7f8b440c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b43e00640 1 --2- 172.21.15.27:0/3611370366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c58c0 0x7f8b440c5ca0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009870 tx=0x7f8b3402b8f0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b43e00640 1 -- 172.21.15.27:0/3611370366 shutdown_connections 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b43e00640 1 --2- 172.21.15.27:0/3611370366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440c8ed0 0x7f8b441555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b43e00640 1 --2- 172.21.15.27:0/3611370366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c58c0 0x7f8b440c5ca0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.216+0000 7f8b43e00640 1 -- 172.21.15.27:0/3611370366 >> 172.21.15.27:0/3611370366 conn(0x7f8b4406b6f0 msgr2=0x7f8b440acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:48.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.217+0000 7f8b43e00640 1 -- 172.21.15.27:0/3611370366 shutdown_connections 2024-03-28T19:18:48.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.217+0000 7f8b43e00640 1 -- 172.21.15.27:0/3611370366 wait complete. 2024-03-28T19:18:48.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.219+0000 7f8b43e00640 1 Processor -- start 2024-03-28T19:18:48.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.219+0000 7f8b43e00640 1 -- start start 2024-03-28T19:18:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440be5d0 0x7f8b44155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b440c58c0 0x7f8b44155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c8ed0 0x7f8b440c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b440b0820 con 0x7f8b440c8ed0 2024-03-28T19:18:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44159750 con 0x7f8b440c58c0 2024-03-28T19:18:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b440b09a0 con 0x7f8b440be5d0 2024-03-28T19:18:48.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c8ed0 0x7f8b440c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c8ed0 0x7f8b440c3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48198/0 (socket says 172.21.15.27:48198) 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b43400640 1 -- 172.21.15.27:0/1884073120 learned_addr learned my addr 172.21.15.27:0/1884073120 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b42000640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b440c58c0 0x7f8b44155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.221+0000 7f8b42a00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440be5d0 0x7f8b44155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b43400640 1 -- 172.21.15.27:0/1884073120 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440be5d0 msgr2=0x7f8b44155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b43400640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440be5d0 0x7f8b44155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b43400640 1 -- 172.21.15.27:0/1884073120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b440c58c0 msgr2=0x7f8b44155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b43400640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b440c58c0 0x7f8b44155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b43400640 1 -- 172.21.15.27:0/1884073120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b441539f0 con 0x7f8b440c8ed0 2024-03-28T19:18:48.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b43400640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c8ed0 0x7f8b440c3ff0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00c6b0 tx=0x7f8b3c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b3aa00640 1 -- 172.21.15.27:0/1884073120 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c003b70 con 0x7f8b440c8ed0 2024-03-28T19:18:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b3aa00640 1 -- 172.21.15.27:0/1884073120 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c003d10 con 0x7f8b440c8ed0 2024-03-28T19:18:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b440c2190 con 0x7f8b440c8ed0 2024-03-28T19:18:48.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.222+0000 7f8b3aa00640 1 -- 172.21.15.27:0/1884073120 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c005310 con 0x7f8b440c8ed0 2024-03-28T19:18:48.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.223+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b440ac970 con 0x7f8b440c8ed0 2024-03-28T19:18:48.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.224+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b44069640 con 0x7f8b440c8ed0 2024-03-28T19:18:48.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.226+0000 7f8b3aa00640 1 -- 172.21.15.27:0/1884073120 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8b3c0054b0 con 0x7f8b440c8ed0 2024-03-28T19:18:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.227+0000 7f8b3aa00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8b2406bbd0 0x7f8b2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.227+0000 7f8b3aa00640 1 -- 172.21.15.27:0/1884073120 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f8b3c08b660 con 0x7f8b440c8ed0 2024-03-28T19:18:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.227+0000 7f8b42a00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8b2406bbd0 0x7f8b2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.228+0000 7f8b42a00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8b2406bbd0 0x7f8b2406e090 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8b34036000 tx=0x7f8b3403c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.232+0000 7f8b3aa00640 1 -- 172.21.15.27:0/1884073120 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8b3c0592d0 con 0x7f8b440c8ed0 2024-03-28T19:18:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.471+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b44156580 con 0x7f8b440c8ed0 2024-03-28T19:18:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.472+0000 7f8b3aa00640 1 -- 172.21.15.27:0/1884073120 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+4664 (secure 0 0 0) 0x7f8b3c058bd0 con 0x7f8b440c8ed0 2024-03-28T19:18:48.471 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:48.471 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:45.217784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:45.217785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:48.472 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 180 2024-03-28T19:18:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8b2406bbd0 msgr2=0x7f8b2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8b2406bbd0 0x7f8b2406e090 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f8b34036000 tx=0x7f8b3403c040 comp rx=0 tx=0).stop 2024-03-28T19:18:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c8ed0 msgr2=0x7f8b440c3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c8ed0 0x7f8b440c3ff0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00c6b0 tx=0x7f8b3c00cb70 comp rx=0 tx=0).stop 2024-03-28T19:18:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 shutdown_connections 2024-03-28T19:18:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8b2406bbd0 0x7f8b2406e090 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b440c8ed0 0x7f8b440c3ff0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b440c58c0 0x7f8b44155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 --2- 172.21.15.27:0/1884073120 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b440be5d0 0x7f8b44155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 >> 172.21.15.27:0/1884073120 conn(0x7f8b4406b6f0 msgr2=0x7f8b441581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 shutdown_connections 2024-03-28T19:18:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:48.475+0000 7f8b43e00640 1 -- 172.21.15.27:0/1884073120 wait complete. 2024-03-28T19:18:48.492 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:18:49.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.034+0000 7f5b4d000640 1 Processor -- start 2024-03-28T19:18:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.034+0000 7f5b4d000640 1 -- start start 2024-03-28T19:18:50.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.035+0000 7f5b4d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b48152da0 0x7f5b480c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.035+0000 7f5b4d000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5b480732e0 0x7f5b4806bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.036+0000 7f5b4d000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4806b160 con 0x7f5b480c2e40 2024-03-28T19:18:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.036+0000 7f5b4d000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b4806c260 con 0x7f5b480732e0 2024-03-28T19:18:50.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.036+0000 7f5b4d000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480c2100 con 0x7f5b48152da0 2024-03-28T19:18:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.036+0000 7f5b45400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5b480c2e40 0x7f5b4806b610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:55508/0 (socket says 172.21.15.27:55508) 2024-03-28T19:18:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.036+0000 7f5b45400640 1 -- 172.21.15.27:0/3398274002 learned_addr learned my addr 172.21.15.27:0/3398274002 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.036+0000 7f5b45e00640 1 --2- 172.21.15.27:0/3398274002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b48152da0 0x7f5b480c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.036+0000 7f5b46800640 1 --2- 172.21.15.27:0/3398274002 >> v2:172.21.15.155:3300/0 conn(0x7f5b480732e0 0x7f5b4806bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.037+0000 7f5b44a00640 1 -- 172.21.15.27:0/3398274002 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672469824 0 0) 0x7f5b4806b160 con 0x7f5b480c2e40 2024-03-28T19:18:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.037+0000 7f5b44a00640 1 -- 172.21.15.27:0/3398274002 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b2c003520 con 0x7f5b480c2e40 2024-03-28T19:18:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.037+0000 7f5b45e00640 1 -- 172.21.15.27:0/3398274002 >> v2:172.21.15.155:3300/0 conn(0x7f5b480732e0 msgr2=0x7f5b4806bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.037+0000 7f5b45e00640 1 --2- 172.21.15.27:0/3398274002 >> v2:172.21.15.155:3300/0 conn(0x7f5b480732e0 0x7f5b4806bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.037+0000 7f5b45e00640 1 -- 172.21.15.27:0/3398274002 >> v1:172.21.15.155:6789/0 conn(0x7f5b480c2e40 legacy=0x7f5b4806b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.037+0000 7f5b45e00640 1 -- 172.21.15.27:0/3398274002 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b48156450 con 0x7f5b48152da0 2024-03-28T19:18:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.037+0000 7f5b46800640 1 --2- 172.21.15.27:0/3398274002 >> v2:172.21.15.155:3300/0 conn(0x7f5b480732e0 0x7f5b4806bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.038+0000 7f5b45e00640 1 --2- 172.21.15.27:0/3398274002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b48152da0 0x7f5b480c2900 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5b380097a0 tx=0x7f5b3802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ab0fa1327225c4c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.039+0000 7f5b44a00640 1 -- 172.21.15.27:0/3398274002 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38039070 con 0x7f5b48152da0 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.039+0000 7f5b44a00640 1 -- 172.21.15.27:0/3398274002 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b380316f0 con 0x7f5b48152da0 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.039+0000 7f5b44a00640 1 -- 172.21.15.27:0/3398274002 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38031ac0 con 0x7f5b48152da0 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.039+0000 7f5b4d000640 1 -- 172.21.15.27:0/3398274002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b48152da0 msgr2=0x7f5b480c2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.039+0000 7f5b4d000640 1 --2- 172.21.15.27:0/3398274002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b48152da0 0x7f5b480c2900 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5b380097a0 tx=0x7f5b3802b790 comp rx=0 tx=0).stop 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.040+0000 7f5b4d000640 1 -- 172.21.15.27:0/3398274002 shutdown_connections 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.040+0000 7f5b4d000640 1 --2- 172.21.15.27:0/3398274002 >> v2:172.21.15.155:3300/0 conn(0x7f5b480732e0 0x7f5b4806bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.040+0000 7f5b4d000640 1 --2- 172.21.15.27:0/3398274002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b48152da0 0x7f5b480c2900 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.040+0000 7f5b4d000640 1 -- 172.21.15.27:0/3398274002 >> 172.21.15.27:0/3398274002 conn(0x7f5b480c3ff0 msgr2=0x7f5b480c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:50.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.040+0000 7f5b4d000640 1 -- 172.21.15.27:0/3398274002 shutdown_connections 2024-03-28T19:18:50.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.040+0000 7f5b4d000640 1 -- 172.21.15.27:0/3398274002 wait complete. 2024-03-28T19:18:50.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.041+0000 7f5b4d000640 1 Processor -- start 2024-03-28T19:18:50.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.041+0000 7f5b4d000640 1 -- start start 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.042+0000 7f5b4d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b480732e0 0x7f5b481eee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.042+0000 7f5b4d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b480c2e40 0x7f5b481547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.042+0000 7f5b4d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b48152da0 0x7f5b48147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.042+0000 7f5b4d000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b481560e0 con 0x7f5b480732e0 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.042+0000 7f5b4d000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480c2630 con 0x7f5b48152da0 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.042+0000 7f5b4d000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b48067130 con 0x7f5b480c2e40 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b480732e0 0x7f5b481eee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b480732e0 0x7f5b481eee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48204/0 (socket says 172.21.15.27:48204) 2024-03-28T19:18:50.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 -- 172.21.15.27:0/2503522911 learned_addr learned my addr 172.21.15.27:0/2503522911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b46800640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b48152da0 0x7f5b48147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 -- 172.21.15.27:0/2503522911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b480c2e40 msgr2=0x7f5b481547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b480c2e40 0x7f5b481547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 -- 172.21.15.27:0/2503522911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b48152da0 msgr2=0x7f5b48147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b48152da0 0x7f5b48147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 -- 172.21.15.27:0/2503522911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4814a1d0 con 0x7f5b480732e0 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b46800640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b48152da0 0x7f5b48147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.043+0000 7f5b45e00640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b480732e0 0x7f5b481eee20 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5b38000c00 tx=0x7f5b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.044+0000 7f5b3e000640 1 -- 172.21.15.27:0/2503522911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38043020 con 0x7f5b480732e0 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.044+0000 7f5b3e000640 1 -- 172.21.15.27:0/2503522911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b38034680 con 0x7f5b480732e0 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.044+0000 7f5b3e000640 1 -- 172.21.15.27:0/2503522911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38039040 con 0x7f5b480732e0 2024-03-28T19:18:50.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.044+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4814a4b0 con 0x7f5b480732e0 2024-03-28T19:18:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.044+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4814aa70 con 0x7f5b480732e0 2024-03-28T19:18:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.045+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4804f400 con 0x7f5b480732e0 2024-03-28T19:18:50.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.046+0000 7f5b3e000640 1 -- 172.21.15.27:0/2503522911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b38046020 con 0x7f5b480732e0 2024-03-28T19:18:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.047+0000 7f5b3e000640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b2806bbd0 0x7f5b2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.047+0000 7f5b3e000640 1 -- 172.21.15.27:0/2503522911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f5b380ad410 con 0x7f5b480732e0 2024-03-28T19:18:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.047+0000 7f5b45400640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b2806bbd0 0x7f5b2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.048+0000 7f5b45400640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b2806bbd0 0x7f5b2806e090 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5b34000f60 tx=0x7f5b340078f0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:50.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.051+0000 7f5b3e000640 1 -- 172.21.15.27:0/2503522911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b3804d030 con 0x7f5b480732e0 2024-03-28T19:18:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.292+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b48066f10 con 0x7f5b480732e0 2024-03-28T19:18:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.293+0000 7f5b3e000640 1 -- 172.21.15.27:0/2503522911 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+4664 (secure 0 0 0) 0x7f5b3807b0b0 con 0x7f5b480732e0 2024-03-28T19:18:50.292 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:50.293 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:45.217784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:45.217785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 180 2024-03-28T19:18:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b2806bbd0 msgr2=0x7f5b2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b2806bbd0 0x7f5b2806e090 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5b34000f60 tx=0x7f5b340078f0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b480732e0 msgr2=0x7f5b481eee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b480732e0 0x7f5b481eee20 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5b38000c00 tx=0x7f5b380046c0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 shutdown_connections 2024-03-28T19:18:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5b2806bbd0 0x7f5b2806e090 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b48152da0 0x7f5b48147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5b480c2e40 0x7f5b481547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 --2- 172.21.15.27:0/2503522911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b480732e0 0x7f5b481eee20 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.295+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 >> 172.21.15.27:0/2503522911 conn(0x7f5b480c3ff0 msgr2=0x7f5b4806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.296+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 shutdown_connections 2024-03-28T19:18:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.296+0000 7f5b4d000640 1 -- 172.21.15.27:0/2503522911 wait complete. 2024-03-28T19:18:50.312 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:18:50.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:50.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.699+0000 7fdd62200640 1 Processor -- start 2024-03-28T19:18:50.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.699+0000 7fdd62200640 1 -- start start 2024-03-28T19:18:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.701+0000 7fdd62200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd5c0c4dc0 0x7fdd5c070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.701+0000 7fdd62200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0c52e0 con 0x7fdd5c152bd0 2024-03-28T19:18:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.701+0000 7fdd62200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c1557e0 con 0x7fdd5c070a80 2024-03-28T19:18:50.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.701+0000 7fdd62200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c155ac0 con 0x7fdd5c0c4dc0 2024-03-28T19:18:50.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.701+0000 7fdd5aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd5c0c4dc0 0x7fdd5c070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.701+0000 7fdd5b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdd5c152bd0 0x7fdd5c06eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41542/0 (socket says 172.21.15.27:41542) 2024-03-28T19:18:50.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.701+0000 7fdd5b400640 1 -- 172.21.15.27:0/3001347369 learned_addr learned my addr 172.21.15.27:0/3001347369 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:50.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.702+0000 7fdd5aa00640 1 -- 172.21.15.27:0/3001347369 >> v1:172.21.15.155:6789/0 conn(0x7fdd5c070a80 legacy=0x7fdd5c15ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.702+0000 7fdd5aa00640 1 -- 172.21.15.27:0/3001347369 >> v1:172.21.15.27:6789/0 conn(0x7fdd5c152bd0 legacy=0x7fdd5c06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.702+0000 7fdd5a000640 1 -- 172.21.15.27:0/3001347369 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907574609 0 0) 0x7fdd5c0c52e0 con 0x7fdd5c152bd0 2024-03-28T19:18:50.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.702+0000 7fdd5aa00640 1 -- 172.21.15.27:0/3001347369 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c155da0 con 0x7fdd5c0c4dc0 2024-03-28T19:18:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.703+0000 7fdd5aa00640 1 --2- 172.21.15.27:0/3001347369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c0c4dc0 0x7fdd5c070540 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fdd480097a0 tx=0x7fdd4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82ca7b476bd97f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.704+0000 7fdd5a000640 1 -- 172.21.15.27:0/3001347369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48035020 con 0x7fdd5c0c4dc0 2024-03-28T19:18:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.704+0000 7fdd5a000640 1 -- 172.21.15.27:0/3001347369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd48031830 con 0x7fdd5c0c4dc0 2024-03-28T19:18:50.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.704+0000 7fdd5a000640 1 -- 172.21.15.27:0/3001347369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48031c00 con 0x7fdd5c0c4dc0 2024-03-28T19:18:50.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.705+0000 7fdd62200640 1 -- 172.21.15.27:0/3001347369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c0c4dc0 msgr2=0x7fdd5c070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.705+0000 7fdd62200640 1 --2- 172.21.15.27:0/3001347369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c0c4dc0 0x7fdd5c070540 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fdd480097a0 tx=0x7fdd4802b8d0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.705+0000 7fdd62200640 1 -- 172.21.15.27:0/3001347369 shutdown_connections 2024-03-28T19:18:50.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.705+0000 7fdd62200640 1 --2- 172.21.15.27:0/3001347369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c0c4dc0 0x7fdd5c070540 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.705+0000 7fdd62200640 1 -- 172.21.15.27:0/3001347369 >> 172.21.15.27:0/3001347369 conn(0x7fdd5c066dd0 msgr2=0x7fdd5c0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:50.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.705+0000 7fdd62200640 1 -- 172.21.15.27:0/3001347369 shutdown_connections 2024-03-28T19:18:50.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.705+0000 7fdd62200640 1 -- 172.21.15.27:0/3001347369 wait complete. 2024-03-28T19:18:50.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.707+0000 7fdd62200640 1 Processor -- start 2024-03-28T19:18:50.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.707+0000 7fdd62200640 1 -- start start 2024-03-28T19:18:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd62200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c070a80 0x7fdd5c1e1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd62200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd5c0c4dc0 0x7fdd5c1e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd62200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd5c152bd0 0x7fdd5c1eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd62200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0c6c70 con 0x7fdd5c070a80 2024-03-28T19:18:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd62200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c152fb0 con 0x7fdd5c152bd0 2024-03-28T19:18:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd62200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0c6df0 con 0x7fdd5c0c4dc0 2024-03-28T19:18:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd5b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c070a80 0x7fdd5c1e1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd5b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c070a80 0x7fdd5c1e1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48216/0 (socket says 172.21.15.27:48216) 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd5b400640 1 -- 172.21.15.27:0/3088898502 learned_addr learned my addr 172.21.15.27:0/3088898502 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd5aa00640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd5c0c4dc0 0x7fdd5c1e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.708+0000 7fdd5be00640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd5c152bd0 0x7fdd5c1eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd5b400640 1 -- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd5c0c4dc0 msgr2=0x7fdd5c1e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd5b400640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd5c0c4dc0 0x7fdd5c1e1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd5b400640 1 -- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd5c152bd0 msgr2=0x7fdd5c1eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd5b400640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd5c152bd0 0x7fdd5c1eb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd5b400640 1 -- 172.21.15.27:0/3088898502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c1ef100 con 0x7fdd5c070a80 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd5aa00640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd5c0c4dc0 0x7fdd5c1e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd5b400640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c070a80 0x7fdd5c1e1800 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00c500 tx=0x7fdd4c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd53400640 1 -- 172.21.15.27:0/3088898502 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c018070 con 0x7fdd5c070a80 2024-03-28T19:18:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd53400640 1 -- 172.21.15.27:0/3088898502 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c007580 con 0x7fdd5c070a80 2024-03-28T19:18:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c1ef3a0 con 0x7fdd5c070a80 2024-03-28T19:18:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd53400640 1 -- 172.21.15.27:0/3088898502 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c004a20 con 0x7fdd5c070a80 2024-03-28T19:18:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.709+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c1e2a00 con 0x7fdd5c070a80 2024-03-28T19:18:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.711+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5c1ef570 con 0x7fdd5c070a80 2024-03-28T19:18:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.712+0000 7fdd53400640 1 -- 172.21.15.27:0/3088898502 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdd4c01b070 con 0x7fdd5c070a80 2024-03-28T19:18:50.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.713+0000 7fdd53400640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:50.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.713+0000 7fdd53400640 1 -- 172.21.15.27:0/3088898502 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fdd4c08ac90 con 0x7fdd5c070a80 2024-03-28T19:18:50.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.713+0000 7fdd5aa00640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:50.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.714+0000 7fdd5aa00640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdd48036000 tx=0x7fdd48008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:50.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.717+0000 7fdd53400640 1 -- 172.21.15.27:0/3088898502 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdd4c0588a0 con 0x7fdd5c070a80 2024-03-28T19:18:50.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.953+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd5c04f400 con 0x7fdd5c070a80 2024-03-28T19:18:50.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.954+0000 7fdd53400640 1 -- 172.21.15.27:0/3088898502 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+4668 (secure 0 0 0) 0x7fdd4c0581a0 con 0x7fdd5c070a80 2024-03-28T19:18:50.953 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:50.954 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:45.217784+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:49.982779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:50.954 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 181 2024-03-28T19:18:50.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd3c06bbd0 msgr2=0x7fdd3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdd48036000 tx=0x7fdd48008030 comp rx=0 tx=0).stop 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c070a80 msgr2=0x7fdd5c1e1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c070a80 0x7fdd5c1e1800 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00c500 tx=0x7fdd4c00c9c0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 shutdown_connections 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdd3c06bbd0 0x7fdd3c06e090 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd5c152bd0 0x7fdd5c1eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdd5c0c4dc0 0x7fdd5c1e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 --2- 172.21.15.27:0/3088898502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd5c070a80 0x7fdd5c1e1800 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 >> 172.21.15.27:0/3088898502 conn(0x7fdd5c066dd0 msgr2=0x7fdd5c06c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:50.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 shutdown_connections 2024-03-28T19:18:50.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:50.957+0000 7fdd62200640 1 -- 172.21.15.27:0/3088898502 wait complete. 2024-03-28T19:18:50.974 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:18:52.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.547+0000 7fc3f9c00640 1 Processor -- start 2024-03-28T19:18:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.547+0000 7fc3f9c00640 1 -- start start 2024-03-28T19:18:52.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.548+0000 7fc3f9c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3f41551a0 0x7fc3f40c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.548+0000 7fc3f9c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc3f40aada0 0x7fc3f40c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.548+0000 7fc3f9c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f40c26d0 con 0x7fc3f40c5580 2024-03-28T19:18:52.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.549+0000 7fc3f9c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f4156870 con 0x7fc3f40aada0 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.549+0000 7fc3f9c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4155cf0 con 0x7fc3f41551a0 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.549+0000 7fc3eb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc3f40c5580 0x7fc3f40c1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41558/0 (socket says 172.21.15.27:41558) 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.549+0000 7fc3eb400640 1 -- 172.21.15.27:0/1760288478 learned_addr learned my addr 172.21.15.27:0/1760288478 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.549+0000 7fc3f9200640 1 --2- 172.21.15.27:0/1760288478 >> v2:172.21.15.155:3300/0 conn(0x7fc3f40aada0 0x7fc3f40c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.549+0000 7fc3f3e00640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3f41551a0 0x7fc3f40c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.549+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216833087 0 0) 0x7fc3f40c26d0 con 0x7fc3f40c5580 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.549+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d8005730 con 0x7fc3f40c5580 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1376279712 0 0) 0x7fc3d8005730 con 0x7fc3f40c5580 2024-03-28T19:18:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3f41551a0 msgr2=0x7fc3f40c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3f41551a0 0x7fc3f40c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 >> v2:172.21.15.155:3300/0 conn(0x7fc3f40aada0 msgr2=0x7fc3f40c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 --2- 172.21.15.27:0/1760288478 >> v2:172.21.15.155:3300/0 conn(0x7fc3f40aada0 0x7fc3f40c2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f4155fd0 con 0x7fc3f40c5580 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fc3dc001670 con 0x7fc3f40c5580 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f9200640 1 --2- 172.21.15.27:0/1760288478 >> v2:172.21.15.155:3300/0 conn(0x7fc3f40aada0 0x7fc3f40c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 >> v1:172.21.15.27:6789/0 conn(0x7fc3f40c5580 legacy=0x7fc3f40c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3d8005bc0 0x7fc3d8005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3f41551a0 0x7fc3d8006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d8006f50 0x7fc3d80073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4156870 con 0x7fc3d8006f50 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4155cf0 con 0x7fc3d8005bc0 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d800dd00 con 0x7fc3f41551a0 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f9200640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d8006f50 0x7fc3d80073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.550+0000 7fc3f3e00640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3d8005bc0 0x7fc3d8005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.551+0000 7fc3f9200640 1 -- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3f41551a0 msgr2=0x7fc3d8006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.551+0000 7fc3f9200640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3f41551a0 0x7fc3d8006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.551+0000 7fc3f9200640 1 -- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3d8005bc0 msgr2=0x7fc3d8005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:52.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.551+0000 7fc3f9200640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3d8005bc0 0x7fc3d8005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.551+0000 7fc3f9200640 1 -- 172.21.15.27:0/1760288478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8005730 con 0x7fc3d8006f50 2024-03-28T19:18:52.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.552+0000 7fc3f9200640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d8006f50 0x7fc3d80073d0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0098a0 tx=0x7fc3d8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750406bd629eab8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:52.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.552+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec00f040 con 0x7fc3d8006f50 2024-03-28T19:18:52.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.552+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ec004230 con 0x7fc3d8006f50 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.552+0000 7fc3f3400640 1 -- 172.21.15.27:0/1760288478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec003610 con 0x7fc3d8006f50 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.552+0000 7fc3f9c00640 1 -- 172.21.15.27:0/1760288478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d8006f50 msgr2=0x7fc3d80073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.552+0000 7fc3f9c00640 1 --2- 172.21.15.27:0/1760288478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3d8006f50 0x7fc3d80073d0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0098a0 tx=0x7fc3d8007a40 comp rx=0 tx=0).stop 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.552+0000 7fc3f9200640 1 -- 172.21.15.27:0/1760288478 reap_dead start 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.553+0000 7fc3f9c00640 1 -- 172.21.15.27:0/1760288478 shutdown_connections 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.553+0000 7fc3f9c00640 1 -- 172.21.15.27:0/1760288478 >> 172.21.15.27:0/1760288478 conn(0x7fc3f406b0c0 msgr2=0x7fc3f406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.553+0000 7fc3f9c00640 1 -- 172.21.15.27:0/1760288478 shutdown_connections 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.553+0000 7fc3f9c00640 1 -- 172.21.15.27:0/1760288478 wait complete. 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.554+0000 7fc3f9c00640 1 Processor -- start 2024-03-28T19:18:52.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.555+0000 7fc3f9c00640 1 -- start start 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.555+0000 7fc3f9c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3d8006f50 0x7fc3f40a7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.555+0000 7fc3f9c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3f4152de0 0x7fc3f41eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.555+0000 7fc3f9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3f40c0160 0x7fc3f41ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.555+0000 7fc3f9c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f41559a0 con 0x7fc3f4152de0 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.555+0000 7fc3f9c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4156590 con 0x7fc3f40c0160 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.555+0000 7fc3f9c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4155580 con 0x7fc3d8006f50 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3f4152de0 0x7fc3f41eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3f4152de0 0x7fc3f41eb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48238/0 (socket says 172.21.15.27:48238) 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 -- 172.21.15.27:0/89684779 learned_addr learned my addr 172.21.15.27:0/89684779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:52.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3f9200640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3f40c0160 0x7fc3f41ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 -- 172.21.15.27:0/89684779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3d8006f50 msgr2=0x7fc3f40a7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3d8006f50 0x7fc3f40a7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 -- 172.21.15.27:0/89684779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3f40c0160 msgr2=0x7fc3f41ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3f40c0160 0x7fc3f41ee9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 -- 172.21.15.27:0/89684779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f41ebb00 con 0x7fc3f4152de0 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.556+0000 7fc3eb400640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3f4152de0 0x7fc3f41eb5c0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00c7a0 tx=0x7fc3dc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.557+0000 7fc3f0c00640 1 -- 172.21.15.27:0/89684779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc018070 con 0x7fc3f4152de0 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.557+0000 7fc3f0c00640 1 -- 172.21.15.27:0/89684779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc00ce50 con 0x7fc3f4152de0 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.557+0000 7fc3f0c00640 1 -- 172.21.15.27:0/89684779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0047e0 con 0x7fc3f4152de0 2024-03-28T19:18:52.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.557+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f4153860 con 0x7fc3f4152de0 2024-03-28T19:18:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.557+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f4153580 con 0x7fc3f4152de0 2024-03-28T19:18:52.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.558+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f4067dd0 con 0x7fc3f4152de0 2024-03-28T19:18:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.559+0000 7fc3f0c00640 1 -- 172.21.15.27:0/89684779 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc3dc01b070 con 0x7fc3f4152de0 2024-03-28T19:18:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.560+0000 7fc3f0c00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.560+0000 7fc3f0c00640 1 -- 172.21.15.27:0/89684779 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc3dc08ae50 con 0x7fc3f4152de0 2024-03-28T19:18:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.560+0000 7fc3f3e00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:52.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.561+0000 7fc3f3e00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3d006bbd0 0x7fc3d006e090 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc3f414ed60 tx=0x7fc3e4009750 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.564+0000 7fc3f0c00640 1 -- 172.21.15.27:0/89684779 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc3f41ebb00 con 0x7fc3f4152de0 2024-03-28T19:18:52.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.853+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3f4069640 con 0x7fc3f4152de0 2024-03-28T19:18:52.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.853+0000 7fc3f0c00640 1 -- 172.21.15.27:0/89684779 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+4672 (secure 0 0 0) 0x7fc3dc058ac0 con 0x7fc3f4152de0 2024-03-28T19:18:52.853 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:52.853 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:51.609776+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:51.799140+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:52.853 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 183 2024-03-28T19:18:52.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3d006bbd0 msgr2=0x7fc3d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:52.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3d006bbd0 0x7fc3d006e090 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc3f414ed60 tx=0x7fc3e4009750 comp rx=0 tx=0).stop 2024-03-28T19:18:52.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3f4152de0 msgr2=0x7fc3f41eb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3f4152de0 0x7fc3f41eb5c0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00c7a0 tx=0x7fc3dc00cc60 comp rx=0 tx=0).stop 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 shutdown_connections 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc3d006bbd0 0x7fc3d006e090 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3f40c0160 0x7fc3f41ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3f4152de0 0x7fc3f41eb5c0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 --2- 172.21.15.27:0/89684779 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3d8006f50 0x7fc3f40a7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 >> 172.21.15.27:0/89684779 conn(0x7fc3f406b0c0 msgr2=0x7fc3f406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 shutdown_connections 2024-03-28T19:18:52.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:52.856+0000 7fc3f9c00640 1 -- 172.21.15.27:0/89684779 wait complete. 2024-03-28T19:18:52.876 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:18:52.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:53.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.211+0000 7ffbace00640 1 Processor -- start 2024-03-28T19:18:53.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.211+0000 7ffbace00640 1 -- start start 2024-03-28T19:18:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffbace00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba80a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffbace00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffba80a61a0 0x7ffba80c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffbace00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffba80c6ee0 0x7ffba8153a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffbace00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba8150ec0 con 0x7ffba80a61a0 2024-03-28T19:18:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffbace00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba806dc70 con 0x7ffba80c6ee0 2024-03-28T19:18:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffbace00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba8153550 con 0x7ffba8150ae0 2024-03-28T19:18:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffb9ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffba80a61a0 0x7ffba80c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffb9ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffba80a61a0 0x7ffba80c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48244/0 (socket says 172.21.15.27:48244) 2024-03-28T19:18:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffb9ea00640 1 -- 172.21.15.27:0/4037722122 learned_addr learned my addr 172.21.15.27:0/4037722122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffba7e00640 1 --2- 172.21.15.27:0/4037722122 >> v2:172.21.15.155:3300/0 conn(0x7ffba80c6ee0 0x7ffba8153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffba7400640 1 --2- 172.21.15.27:0/4037722122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba80a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffb9ea00640 1 -- 172.21.15.27:0/4037722122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 msgr2=0x7ffba80a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.213+0000 7ffb9ea00640 1 --2- 172.21.15.27:0/4037722122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba80a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.214+0000 7ffb9ea00640 1 -- 172.21.15.27:0/4037722122 >> v2:172.21.15.155:3300/0 conn(0x7ffba80c6ee0 msgr2=0x7ffba8153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:53.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.214+0000 7ffb9ea00640 1 --2- 172.21.15.27:0/4037722122 >> v2:172.21.15.155:3300/0 conn(0x7ffba80c6ee0 0x7ffba8153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.214+0000 7ffb9ea00640 1 -- 172.21.15.27:0/4037722122 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba806df10 con 0x7ffba80a61a0 2024-03-28T19:18:53.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.214+0000 7ffba7400640 1 --2- 172.21.15.27:0/4037722122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba80a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:18:53.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.215+0000 7ffb9ea00640 1 --2- 172.21.15.27:0/4037722122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80a61a0 0x7ffba80c69a0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ffba806d840 tx=0x7ffb9002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934c21d109521c0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:53.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.215+0000 7ffba6a00640 1 -- 172.21.15.27:0/4037722122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb900317d0 con 0x7ffba80a61a0 2024-03-28T19:18:53.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.215+0000 7ffba6a00640 1 -- 172.21.15.27:0/4037722122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb90037070 con 0x7ffba80a61a0 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.215+0000 7ffba6a00640 1 -- 172.21.15.27:0/4037722122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90031b90 con 0x7ffba80a61a0 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 -- 172.21.15.27:0/4037722122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80a61a0 msgr2=0x7ffba80c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 --2- 172.21.15.27:0/4037722122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80a61a0 0x7ffba80c69a0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ffba806d840 tx=0x7ffb9002b870 comp rx=0 tx=0).stop 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 -- 172.21.15.27:0/4037722122 shutdown_connections 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 --2- 172.21.15.27:0/4037722122 >> v2:172.21.15.155:3300/0 conn(0x7ffba80c6ee0 0x7ffba8153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 --2- 172.21.15.27:0/4037722122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80a61a0 0x7ffba80c69a0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 --2- 172.21.15.27:0/4037722122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba80a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 -- 172.21.15.27:0/4037722122 >> 172.21.15.27:0/4037722122 conn(0x7ffba806b0c0 msgr2=0x7ffba806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:53.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 -- 172.21.15.27:0/4037722122 shutdown_connections 2024-03-28T19:18:53.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.216+0000 7ffbace00640 1 -- 172.21.15.27:0/4037722122 wait complete. 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.218+0000 7ffbace00640 1 Processor -- start 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.218+0000 7ffbace00640 1 -- start start 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.219+0000 7ffbace00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffba80a61a0 0x7ffba81dca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.219+0000 7ffbace00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80c6ee0 0x7ffba8072e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.219+0000 7ffbace00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba81ea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.219+0000 7ffbace00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba806e7b0 con 0x7ffba80c6ee0 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.219+0000 7ffbace00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba806e4d0 con 0x7ffba80a61a0 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.219+0000 7ffbace00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba81569d0 con 0x7ffba8150ae0 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffb9ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80c6ee0 0x7ffba8072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:53.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffb9ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80c6ee0 0x7ffba8072e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48250/0 (socket says 172.21.15.27:48250) 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffb9ea00640 1 -- 172.21.15.27:0/3947371524 learned_addr learned my addr 172.21.15.27:0/3947371524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffba7e00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba81ea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffba7400640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffba80a61a0 0x7ffba81dca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffb9ea00640 1 -- 172.21.15.27:0/3947371524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 msgr2=0x7ffba81ea3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffb9ea00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba81ea3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffb9ea00640 1 -- 172.21.15.27:0/3947371524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffba80a61a0 msgr2=0x7ffba81dca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffb9ea00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffba80a61a0 0x7ffba81dca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.220+0000 7ffb9ea00640 1 -- 172.21.15.27:0/3947371524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba81ebfd0 con 0x7ffba80c6ee0 2024-03-28T19:18:53.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.221+0000 7ffb9ea00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80c6ee0 0x7ffba8072e60 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7ffb9002b8b0 tx=0x7ffb900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.221+0000 7ffb9fe00640 1 -- 172.21.15.27:0/3947371524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90032660 con 0x7ffba80c6ee0 2024-03-28T19:18:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.221+0000 7ffb9fe00640 1 -- 172.21.15.27:0/3947371524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb90032840 con 0x7ffba80c6ee0 2024-03-28T19:18:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.221+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba8071460 con 0x7ffba80c6ee0 2024-03-28T19:18:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.221+0000 7ffb9fe00640 1 -- 172.21.15.27:0/3947371524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90032c30 con 0x7ffba80c6ee0 2024-03-28T19:18:53.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.221+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba8071a20 con 0x7ffba80c6ee0 2024-03-28T19:18:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.223+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba8067dd0 con 0x7ffba80c6ee0 2024-03-28T19:18:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.224+0000 7ffb9fe00640 1 -- 172.21.15.27:0/3947371524 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffb900453f0 con 0x7ffba80c6ee0 2024-03-28T19:18:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.224+0000 7ffb9fe00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb8406bbd0 0x7ffb8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.225+0000 7ffb9fe00640 1 -- 172.21.15.27:0/3947371524 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ffb900ad9a0 con 0x7ffba80c6ee0 2024-03-28T19:18:53.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.225+0000 7ffba7400640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb8406bbd0 0x7ffb8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:53.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.226+0000 7ffba7400640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb8406bbd0 0x7ffb8406e090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffb980075b0 tx=0x7ffb98007540 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:53.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.229+0000 7ffb9fe00640 1 -- 172.21.15.27:0/3947371524 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffb90040070 con 0x7ffba80c6ee0 2024-03-28T19:18:53.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.473+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffba8069640 con 0x7ffba80c6ee0 2024-03-28T19:18:53.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.474+0000 7ffb9fe00640 1 -- 172.21.15.27:0/3947371524 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+4672 (secure 0 0 0) 0x7ffb9007b5b0 con 0x7ffba80c6ee0 2024-03-28T19:18:53.473 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:53.473 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:51.609776+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:51.799140+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:18:53.474 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 183 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb8406bbd0 msgr2=0x7ffb8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb8406bbd0 0x7ffb8406e090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffb980075b0 tx=0x7ffb98007540 comp rx=0 tx=0).stop 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80c6ee0 msgr2=0x7ffba8072e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80c6ee0 0x7ffba8072e60 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7ffb9002b8b0 tx=0x7ffb900046c0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 shutdown_connections 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffb8406bbd0 0x7ffb8406e090 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffba8150ae0 0x7ffba81ea3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffba80c6ee0 0x7ffba8072e60 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 --2- 172.21.15.27:0/3947371524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffba80a61a0 0x7ffba81dca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.477+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 >> 172.21.15.27:0/3947371524 conn(0x7ffba806b0c0 msgr2=0x7ffba80c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.478+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 shutdown_connections 2024-03-28T19:18:53.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:53.478+0000 7ffbace00640 1 -- 172.21.15.27:0/3947371524 wait complete. 2024-03-28T19:18:53.497 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:18:54.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.120+0000 7f83d8800640 1 Processor -- start 2024-03-28T19:18:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.120+0000 7f83d8800640 1 -- start start 2024-03-28T19:18:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.121+0000 7f83d8800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0152f80 0x7f83d00c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.121+0000 7f83d8800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83d00c7970 0x7f83d00c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.121+0000 7f83d8800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d00c7310 con 0x7f83d0155730 2024-03-28T19:18:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.121+0000 7f83d8800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d0158d40 con 0x7f83d00c7970 2024-03-28T19:18:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.121+0000 7f83d8800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d0159020 con 0x7f83d0152f80 2024-03-28T19:18:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.122+0000 7f83d5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83d00c7970 0x7f83d00c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.122+0000 7f83d5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f83d00c7970 0x7f83d00c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48252/0 (socket says 172.21.15.27:48252) 2024-03-28T19:18:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.122+0000 7f83d5400640 1 -- 172.21.15.27:0/2832781653 learned_addr learned my addr 172.21.15.27:0/2832781653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.122+0000 7f83d5e00640 1 --2- 172.21.15.27:0/2832781653 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0152f80 0x7f83d00c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.122+0000 7f83d5400640 1 -- 172.21.15.27:0/2832781653 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0152f80 msgr2=0x7f83d00c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.122+0000 7f83d5400640 1 --2- 172.21.15.27:0/2832781653 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0152f80 0x7f83d00c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.122+0000 7f83d5400640 1 -- 172.21.15.27:0/2832781653 >> v1:172.21.15.155:6789/0 conn(0x7f83d0155730 legacy=0x7f83d0155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.122+0000 7f83d5400640 1 -- 172.21.15.27:0/2832781653 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d0159300 con 0x7f83d00c7970 2024-03-28T19:18:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.124+0000 7f83d5400640 1 --2- 172.21.15.27:0/2832781653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c7df0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009870 tx=0x7f83c002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3f588714a06014 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.124+0000 7f83d4a00640 1 -- 172.21.15.27:0/2832781653 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c00317c0 con 0x7f83d00c7970 2024-03-28T19:18:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.124+0000 7f83d4a00640 1 -- 172.21.15.27:0/2832781653 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c0037070 con 0x7f83d00c7970 2024-03-28T19:18:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.124+0000 7f83d4a00640 1 -- 172.21.15.27:0/2832781653 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0031b80 con 0x7f83d00c7970 2024-03-28T19:18:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.125+0000 7f83d8800640 1 -- 172.21.15.27:0/2832781653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 msgr2=0x7f83d00c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.125+0000 7f83d8800640 1 --2- 172.21.15.27:0/2832781653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c7df0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009870 tx=0x7f83c002b860 comp rx=0 tx=0).stop 2024-03-28T19:18:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.125+0000 7f83d8800640 1 -- 172.21.15.27:0/2832781653 shutdown_connections 2024-03-28T19:18:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.125+0000 7f83d8800640 1 --2- 172.21.15.27:0/2832781653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c7df0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.125+0000 7f83d8800640 1 --2- 172.21.15.27:0/2832781653 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0152f80 0x7f83d00c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.125+0000 7f83d8800640 1 -- 172.21.15.27:0/2832781653 >> 172.21.15.27:0/2832781653 conn(0x7f83d006b6f0 msgr2=0x7f83d00caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.125+0000 7f83d8800640 1 -- 172.21.15.27:0/2832781653 shutdown_connections 2024-03-28T19:18:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.125+0000 7f83d8800640 1 -- 172.21.15.27:0/2832781653 wait complete. 2024-03-28T19:18:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.128+0000 7f83d8800640 1 Processor -- start 2024-03-28T19:18:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.128+0000 7f83d8800640 1 -- start start 2024-03-28T19:18:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.129+0000 7f83d8800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.129+0000 7f83d8800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83d0152f80 0x7f83d00c0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.129+0000 7f83d8800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0155730 0x7f83d00c08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.129+0000 7f83d8800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d0159b00 con 0x7f83d00c7970 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.129+0000 7f83d8800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d00c5e20 con 0x7f83d0152f80 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.129+0000 7f83d8800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d00cbca0 con 0x7f83d0155730 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48264/0 (socket says 172.21.15.27:48264) 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5e00640 1 -- 172.21.15.27:0/1145808504 learned_addr learned my addr 172.21.15.27:0/1145808504 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5400640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83d0152f80 0x7f83d00c0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d6800640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0155730 0x7f83d00c08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5e00640 1 -- 172.21.15.27:0/1145808504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0155730 msgr2=0x7f83d00c08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5e00640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0155730 0x7f83d00c08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5e00640 1 -- 172.21.15.27:0/1145808504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83d0152f80 msgr2=0x7f83d00c0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5e00640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83d0152f80 0x7f83d00c0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.130+0000 7f83d5e00640 1 -- 172.21.15.27:0/1145808504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d00c1a10 con 0x7f83d00c7970 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.131+0000 7f83d5e00640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c4800 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f83c400d620 tx=0x7f83c400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.131+0000 7f83ca000640 1 -- 172.21.15.27:0/1145808504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c400b620 con 0x7f83d00c7970 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.131+0000 7f83ca000640 1 -- 172.21.15.27:0/1145808504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c4019070 con 0x7f83d00c7970 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.131+0000 7f83ca000640 1 -- 172.21.15.27:0/1145808504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c400ba00 con 0x7f83d00c7970 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.131+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d00bfab0 con 0x7f83d00c7970 2024-03-28T19:18:55.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.131+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d00c0070 con 0x7f83d00c7970 2024-03-28T19:18:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.133+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d00c6680 con 0x7f83d00c7970 2024-03-28T19:18:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.134+0000 7f83ca000640 1 -- 172.21.15.27:0/1145808504 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f83c400bba0 con 0x7f83d00c7970 2024-03-28T19:18:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.135+0000 7f83ca000640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f83b406bc00 0x7f83b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.135+0000 7f83ca000640 1 -- 172.21.15.27:0/1145808504 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f83c408bc40 con 0x7f83d00c7970 2024-03-28T19:18:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.136+0000 7f83d5400640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f83b406bc00 0x7f83b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.137+0000 7f83d5400640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f83b406bc00 0x7f83b406e0c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f83c0036000 tx=0x7f83c003d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:55.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.140+0000 7f83ca000640 1 -- 172.21.15.27:0/1145808504 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f83d00c6680 con 0x7f83d00c7970 2024-03-28T19:18:55.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:18:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.525+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83d00cba80 con 0x7f83d00c7970 2024-03-28T19:18:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.526+0000 7f83ca000640 1 -- 172.21.15.27:0/1145808504 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4680 (secure 0 0 0) 0x7f83c40598b0 con 0x7f83d00c7970 2024-03-28T19:18:55.525 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:55.525 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:18:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 185 2024-03-28T19:18:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f83b406bc00 msgr2=0x7f83b406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f83b406bc00 0x7f83b406e0c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f83c0036000 tx=0x7f83c003d040 comp rx=0 tx=0).stop 2024-03-28T19:18:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 msgr2=0x7f83d00c4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c4800 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f83c400d620 tx=0x7f83c400dae0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 shutdown_connections 2024-03-28T19:18:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f83b406bc00 0x7f83b406e0c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f83d0155730 0x7f83d00c08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83d0152f80 0x7f83d00c0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 --2- 172.21.15.27:0/1145808504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83d00c7970 0x7f83d00c4800 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 >> 172.21.15.27:0/1145808504 conn(0x7f83d006b6f0 msgr2=0x7f83d0153f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 shutdown_connections 2024-03-28T19:18:55.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.529+0000 7f83d8800640 1 -- 172.21.15.27:0/1145808504 wait complete. 2024-03-28T19:18:55.549 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=6188 2024-03-28T19:18:55.549 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.c 2024-03-28T19:18:55.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.718+0000 7f549a200640 1 Processor -- start 2024-03-28T19:18:55.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.718+0000 7f549a200640 1 -- start start 2024-03-28T19:18:55.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.719+0000 7f549a200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f54940c58c0 0x7f54940c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.719+0000 7f549a200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940be5d0 0x7f5494155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.719+0000 7f549a200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5494156230 con 0x7f54940c8ed0 2024-03-28T19:18:55.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.719+0000 7f549a200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5494158bd0 con 0x7f54940c58c0 2024-03-28T19:18:55.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.719+0000 7f549a200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5494158eb0 con 0x7f54940be5d0 2024-03-28T19:18:55.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f548be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f54940c8ed0 0x7f54941555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41560/0 (socket says 172.21.15.27:41560) 2024-03-28T19:18:55.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f548be00640 1 -- 172.21.15.27:0/3493502617 learned_addr learned my addr 172.21.15.27:0/3493502617 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:18:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5499800640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940be5d0 0x7f5494155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820900502 0 0) 0x7f5494156230 con 0x7f54940c8ed0 2024-03-28T19:18:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5478003520 con 0x7f54940c8ed0 2024-03-28T19:18:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5498e00640 1 --2- 172.21.15.27:0/3493502617 >> v2:172.21.15.155:3300/0 conn(0x7f54940c58c0 0x7f54940c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1708407073 0 0) 0x7f5478003520 con 0x7f54940c8ed0 2024-03-28T19:18:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940be5d0 msgr2=0x7f5494155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940be5d0 0x7f5494155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 >> v2:172.21.15.155:3300/0 conn(0x7f54940c58c0 msgr2=0x7f54940c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 --2- 172.21.15.27:0/3493502617 >> v2:172.21.15.155:3300/0 conn(0x7f54940c58c0 0x7f54940c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5494159190 con 0x7f54940c8ed0 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f547c001670 con 0x7f54940c8ed0 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.720+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 >> v1:172.21.15.27:6789/0 conn(0x7f54940c8ed0 legacy=0x7f54941555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5493e00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54780039b0 0x7f5478003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5493e00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5478004800 0x7f5478004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5493e00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940be5d0 0x7f54780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5494158bd0 con 0x7f54780039b0 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5494158eb0 con 0x7f5478004800 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547800bb20 con 0x7f54940be5d0 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5498e00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54780039b0 0x7f5478003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f548be00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5478004800 0x7f5478004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5498e00640 1 -- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940be5d0 msgr2=0x7f54780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5498e00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940be5d0 0x7f54780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5498e00640 1 -- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5478004800 msgr2=0x7f5478004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:18:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5498e00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5478004800 0x7f5478004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.721+0000 7f5498e00640 1 -- 172.21.15.27:0/3493502617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5478003520 con 0x7f54780039b0 2024-03-28T19:18:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.723+0000 7f5498e00640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54780039b0 0x7f5478003dd0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f5484007330 tx=0x7f5478005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c6da1c4318ff4c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.723+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484039070 con 0x7f54780039b0 2024-03-28T19:18:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.723+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548402ccb0 con 0x7f54780039b0 2024-03-28T19:18:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.723+0000 7f5493e00640 1 -- 172.21.15.27:0/3493502617 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484009be0 con 0x7f54780039b0 2024-03-28T19:18:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.724+0000 7f549a200640 1 -- 172.21.15.27:0/3493502617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54780039b0 msgr2=0x7f5478003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.724+0000 7f549a200640 1 --2- 172.21.15.27:0/3493502617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54780039b0 0x7f5478003dd0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f5484007330 tx=0x7f5478005830 comp rx=0 tx=0).stop 2024-03-28T19:18:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.724+0000 7f5499800640 1 -- 172.21.15.27:0/3493502617 reap_dead start 2024-03-28T19:18:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.724+0000 7f549a200640 1 -- 172.21.15.27:0/3493502617 shutdown_connections 2024-03-28T19:18:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.724+0000 7f549a200640 1 -- 172.21.15.27:0/3493502617 >> 172.21.15.27:0/3493502617 conn(0x7f549406b6f0 msgr2=0x7f54940acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.724+0000 7f549a200640 1 -- 172.21.15.27:0/3493502617 shutdown_connections 2024-03-28T19:18:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.724+0000 7f549a200640 1 -- 172.21.15.27:0/3493502617 wait complete. 2024-03-28T19:18:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.726+0000 7f549a200640 1 Processor -- start 2024-03-28T19:18:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.726+0000 7f549a200640 1 -- start start 2024-03-28T19:18:55.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.727+0000 7f549a200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54780039b0 0x7f54941eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.727+0000 7f549a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5494152920 0x7f54940c2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.727+0000 7f549a200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940c2670 0x7f5494143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.727+0000 7f549a200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54940b0820 con 0x7f5494152920 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.727+0000 7f549a200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5494159750 con 0x7f54780039b0 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.727+0000 7f549a200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54940b09a0 con 0x7f54940c2670 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.727+0000 7f548be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5494152920 0x7f54940c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f5498e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54780039b0 0x7f54941eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f5498e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54780039b0 0x7f54941eef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:49932/0 (socket says 172.21.15.27:49932) 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f5498e00640 1 -- 172.21.15.27:0/897200653 learned_addr learned my addr 172.21.15.27:0/897200653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:18:55.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f5499800640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940c2670 0x7f5494143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f548be00640 1 -- 172.21.15.27:0/897200653 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940c2670 msgr2=0x7f5494143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f548be00640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940c2670 0x7f5494143a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f548be00640 1 -- 172.21.15.27:0/897200653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54780039b0 msgr2=0x7f54941eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f548be00640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54780039b0 0x7f54941eef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f548be00640 1 -- 172.21.15.27:0/897200653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54940aaf00 con 0x7f5494152920 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.728+0000 7f5498e00640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54780039b0 0x7f54941eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.729+0000 7f548be00640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5494152920 0x7f54940c2130 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f547c00e840 tx=0x7f547c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.729+0000 7f5491600640 1 -- 172.21.15.27:0/897200653 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c00c7f0 con 0x7f5494152920 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.729+0000 7f5491600640 1 -- 172.21.15.27:0/897200653 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c018070 con 0x7f5494152920 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.729+0000 7f5491600640 1 -- 172.21.15.27:0/897200653 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c00cbd0 con 0x7f5494152920 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.729+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54941581c0 con 0x7f5494152920 2024-03-28T19:18:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.729+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54941ef7d0 con 0x7f5494152920 2024-03-28T19:18:55.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.731+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549406c810 con 0x7f5494152920 2024-03-28T19:18:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.732+0000 7f5491600640 1 -- 172.21.15.27:0/897200653 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f547c013420 con 0x7f5494152920 2024-03-28T19:18:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.732+0000 7f5491600640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f547006bc00 0x7f547006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:18:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.733+0000 7f5491600640 1 -- 172.21.15.27:0/897200653 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f547c08b330 con 0x7f5494152920 2024-03-28T19:18:55.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.733+0000 7f5498e00640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f547006bc00 0x7f547006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:18:55.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.734+0000 7f5498e00640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f547006bc00 0x7f547006e0c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5484007130 tx=0x7f5484030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:18:55.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:55.740+0000 7f5491600640 1 -- 172.21.15.27:0/897200653 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f549406c810 con 0x7f5494152920 2024-03-28T19:18:56.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.030+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5494069640 con 0x7f5494152920 2024-03-28T19:18:56.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.031+0000 7f5491600640 1 -- 172.21.15.27:0/897200653 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4680 (secure 0 0 0) 0x7f547c0588f0 con 0x7f5494152920 2024-03-28T19:18:56.034 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:18:56.034 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:18:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 185 2024-03-28T19:18:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f547006bc00 msgr2=0x7f547006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f547006bc00 0x7f547006e0c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5484007130 tx=0x7f5484030000 comp rx=0 tx=0).stop 2024-03-28T19:18:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5494152920 msgr2=0x7f54940c2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:18:56.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5494152920 0x7f54940c2130 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f547c00e840 tx=0x7f547c00ed00 comp rx=0 tx=0).stop 2024-03-28T19:18:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 shutdown_connections 2024-03-28T19:18:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f547006bc00 0x7f547006e0c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54940c2670 0x7f5494143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5494152920 0x7f54940c2130 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 --2- 172.21.15.27:0/897200653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54780039b0 0x7f54941eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:18:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 >> 172.21.15.27:0/897200653 conn(0x7f549406b6f0 msgr2=0x7f54940716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:18:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 shutdown_connections 2024-03-28T19:18:56.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:18:56.034+0000 7f549a200640 1 -- 172.21.15.27:0/897200653 wait complete. 2024-03-28T19:18:56.051 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6014 2024-03-28T19:18:56.051 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.b 2024-03-28T19:18:56.752 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T19:18:56.752+0000 7f0a4c000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:18:57.769 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:18:57.770+0000 7fbd82e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:19:00.052 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-03-28T19:19:00.052 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:19:00.052 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:19:00.062 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:19:00.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:19:00.246 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:19:00.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.300+0000 7fcce3200640 1 Processor -- start 2024-03-28T19:19:00.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.300+0000 7fcce3200640 1 -- start start 2024-03-28T19:19:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.304+0000 7fcce3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcce415a7d0 0x7fcce40c3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:00.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.304+0000 7fcce3200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40ab550 con 0x7fcce41571c0 2024-03-28T19:19:00.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.304+0000 7fcce3200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce406dd20 con 0x7fcce40c3960 2024-03-28T19:19:00.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.304+0000 7fcce3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce40abb60 con 0x7fcce415a7d0 2024-03-28T19:19:00.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.305+0000 7fcce1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcce415a7d0 0x7fcce40c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:00.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.305+0000 7fcce1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcce41571c0 0x7fcce40c2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41152/0 (socket says 172.21.15.27:41152) 2024-03-28T19:19:00.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.305+0000 7fcce1e00640 1 -- 172.21.15.27:0/1400470426 learned_addr learned my addr 172.21.15.27:0/1400470426 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:19:00.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.306+0000 7fcce1400640 1 -- 172.21.15.27:0/1400470426 >> v1:172.21.15.155:6789/0 conn(0x7fcce40c3960 legacy=0x7fcce4164350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:00.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.306+0000 7fcce1400640 1 -- 172.21.15.27:0/1400470426 >> v1:172.21.15.27:6789/0 conn(0x7fcce41571c0 legacy=0x7fcce40c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:00.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.306+0000 7fcce1400640 1 -- 172.21.15.27:0/1400470426 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce40ab7c0 con 0x7fcce415a7d0 2024-03-28T19:19:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.310+0000 7fcce1400640 1 --2- 172.21.15.27:0/1400470426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 0x7fcce40c3420 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcce40aba50 tx=0x7fccd402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53178e46129e4e10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.310+0000 7fcce0a00640 1 -- 172.21.15.27:0/1400470426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40317c0 con 0x7fcce415a7d0 2024-03-28T19:19:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.310+0000 7fcce0a00640 1 -- 172.21.15.27:0/1400470426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4037070 con 0x7fcce415a7d0 2024-03-28T19:19:00.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.310+0000 7fcce0a00640 1 -- 172.21.15.27:0/1400470426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4031bc0 con 0x7fcce415a7d0 2024-03-28T19:19:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.312+0000 7fcce3200640 1 -- 172.21.15.27:0/1400470426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 msgr2=0x7fcce40c3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.312+0000 7fcce3200640 1 --2- 172.21.15.27:0/1400470426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 0x7fcce40c3420 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcce40aba50 tx=0x7fccd402b860 comp rx=0 tx=0).stop 2024-03-28T19:19:00.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.313+0000 7fcce3200640 1 -- 172.21.15.27:0/1400470426 shutdown_connections 2024-03-28T19:19:00.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.313+0000 7fcce3200640 1 --2- 172.21.15.27:0/1400470426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 0x7fcce40c3420 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:00.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.313+0000 7fcce3200640 1 -- 172.21.15.27:0/1400470426 >> 172.21.15.27:0/1400470426 conn(0x7fcce406cd50 msgr2=0x7fcce406b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:19:00.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.315+0000 7fcce3200640 1 -- 172.21.15.27:0/1400470426 shutdown_connections 2024-03-28T19:19:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.315+0000 7fcce3200640 1 -- 172.21.15.27:0/1400470426 wait complete. 2024-03-28T19:19:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.318+0000 7fcce3200640 1 Processor -- start 2024-03-28T19:19:00.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.319+0000 7fcce3200640 1 -- start start 2024-03-28T19:19:00.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.320+0000 7fcce3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcce40c3960 0x7fcce414ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.320+0000 7fcce3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41571c0 0x7fcce4072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.320+0000 7fcce3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 0x7fcce414acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41571c0 0x7fcce4072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce1e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcce40c3960 0x7fcce414ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41571c0 0x7fcce4072bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:59558/0 (socket says 172.21.15.27:59558) 2024-03-28T19:19:00.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce1e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcce40c3960 0x7fcce414ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:35888/0 (socket says 172.21.15.27:35888) 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce40c10b0 con 0x7fcce415a7d0 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce40c0f30 con 0x7fcce41571c0 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce406e5c0 con 0x7fcce40c3960 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce1400640 1 -- 172.21.15.27:0/275448305 learned_addr learned my addr 172.21.15.27:0/275448305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.321+0000 7fcce2800640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 0x7fcce414acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.322+0000 7fcce1400640 1 -- 172.21.15.27:0/275448305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcce40c3960 msgr2=0x7fcce414ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.322+0000 7fcce1400640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcce40c3960 0x7fcce414ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.322+0000 7fcce1400640 1 -- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 msgr2=0x7fcce414acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.322+0000 7fcce1400640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 0x7fcce414acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:00.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.322+0000 7fcce1400640 1 -- 172.21.15.27:0/275448305 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce415d5a0 con 0x7fcce41571c0 2024-03-28T19:19:00.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.323+0000 7fcce1400640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41571c0 0x7fcce4072bb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fccd4034040 tx=0x7fccd40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:00.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.323+0000 7fccda000640 1 -- 172.21.15.27:0/275448305 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4032860 con 0x7fcce41571c0 2024-03-28T19:19:00.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.323+0000 7fccda000640 1 -- 172.21.15.27:0/275448305 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4032a00 con 0x7fcce41571c0 2024-03-28T19:19:00.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.323+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce415ade0 con 0x7fcce41571c0 2024-03-28T19:19:00.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.323+0000 7fccda000640 1 -- 172.21.15.27:0/275448305 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4032df0 con 0x7fcce41571c0 2024-03-28T19:19:00.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.323+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce415db20 con 0x7fcce41571c0 2024-03-28T19:19:00.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.325+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce4069640 con 0x7fcce41571c0 2024-03-28T19:19:00.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.328+0000 7fccda000640 1 -- 172.21.15.27:0/275448305 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fccd404e430 con 0x7fcce41571c0 2024-03-28T19:19:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.331+0000 7fccda000640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fccc406bc00 0x7fccc406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:00.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.331+0000 7fccda000640 1 -- 172.21.15.27:0/275448305 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fccd40ad9e0 con 0x7fcce41571c0 2024-03-28T19:19:00.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.336+0000 7fcce1e00640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fccc406bc00 0x7fccc406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.338+0000 7fcce1e00640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fccc406bc00 0x7fccc406e0c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fccd0005880 tx=0x7fccd0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.341+0000 7fccda000640 1 -- 172.21.15.27:0/275448305 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fccd4040070 con 0x7fcce41571c0 2024-03-28T19:19:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.680+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcce406b280 con 0x7fcce41571c0 2024-03-28T19:19:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.681+0000 7fccda000640 1 -- 172.21.15.27:0/275448305 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+5474 (secure 0 0 0) 0x7fccd407afa0 con 0x7fcce41571c0 2024-03-28T19:19:00.680 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:19:00.680 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:19:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 186 2024-03-28T19:19:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fccc406bc00 msgr2=0x7fccc406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fccc406bc00 0x7fccc406e0c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fccd0005880 tx=0x7fccd0009040 comp rx=0 tx=0).stop 2024-03-28T19:19:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41571c0 msgr2=0x7fcce4072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41571c0 0x7fcce4072bb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fccd4034040 tx=0x7fccd40046c0 comp rx=0 tx=0).stop 2024-03-28T19:19:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 shutdown_connections 2024-03-28T19:19:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fccc406bc00 0x7fccc406e0c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcce415a7d0 0x7fcce414acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41571c0 0x7fcce4072bb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 --2- 172.21.15.27:0/275448305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcce40c3960 0x7fcce414ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 >> 172.21.15.27:0/275448305 conn(0x7fcce406cd50 msgr2=0x7fcce41598f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:19:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 shutdown_connections 2024-03-28T19:19:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:00.684+0000 7fcce3200640 1 -- 172.21.15.27:0/275448305 wait complete. 2024-03-28T19:19:00.705 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-03-28T19:19:00.705 INFO:tasks.mds_thrash.fs.[a]:waiting for 105 secs before thrashing 2024-03-28T19:19:01.550 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-03-28T19:19:01.550 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:19:01.550 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:19:01.556 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:19:01.557 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:19:01.759 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:19:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.783+0000 7f161fe00640 1 Processor -- start 2024-03-28T19:19:01.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.783+0000 7f161fe00640 1 -- start start 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.784+0000 7f161fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1620150ae0 0x7f16200a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.785+0000 7f161fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1620153a30 0x7f1620153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.785+0000 7f161fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1620150ec0 con 0x7f16200a61a0 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.785+0000 7f161fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16201543c0 con 0x7f1620150ae0 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.785+0000 7f161fe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f162006dc70 con 0x7f1620153a30 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.785+0000 7f161ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1620150ae0 0x7f16200a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.785+0000 7f161ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1620150ae0 0x7f16200a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39648/0 (socket says 172.21.15.27:39648) 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.785+0000 7f161ea00640 1 -- 172.21.15.27:0/898634470 learned_addr learned my addr 172.21.15.27:0/898634470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:19:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.785+0000 7f161f400640 1 --2- 172.21.15.27:0/898634470 >> v2:172.21.15.155:3300/0 conn(0x7f1620153a30 0x7f1620153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.786+0000 7f161ea00640 1 -- 172.21.15.27:0/898634470 >> v2:172.21.15.155:3300/0 conn(0x7f1620153a30 msgr2=0x7f1620153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:19:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.786+0000 7f161ea00640 1 --2- 172.21.15.27:0/898634470 >> v2:172.21.15.155:3300/0 conn(0x7f1620153a30 0x7f1620153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.786+0000 7f161ea00640 1 -- 172.21.15.27:0/898634470 >> v1:172.21.15.155:6789/0 conn(0x7f16200a61a0 legacy=0x7f16200c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:19:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.786+0000 7f161ea00640 1 -- 172.21.15.27:0/898634470 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620153550 con 0x7f1620150ae0 2024-03-28T19:19:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.787+0000 7f161ea00640 1 --2- 172.21.15.27:0/898634470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620150ae0 0x7f16200a5c60 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f1620073830 tx=0x7f161002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7c3974c3524853 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.787+0000 7f161d600640 1 -- 172.21.15.27:0/898634470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610035020 con 0x7f1620150ae0 2024-03-28T19:19:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.787+0000 7f161d600640 1 -- 172.21.15.27:0/898634470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1610031870 con 0x7f1620150ae0 2024-03-28T19:19:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.787+0000 7f161d600640 1 -- 172.21.15.27:0/898634470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610031c40 con 0x7f1620150ae0 2024-03-28T19:19:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.788+0000 7f161fe00640 1 -- 172.21.15.27:0/898634470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620150ae0 msgr2=0x7f16200a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.788+0000 7f161fe00640 1 --2- 172.21.15.27:0/898634470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620150ae0 0x7f16200a5c60 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f1620073830 tx=0x7f161002b910 comp rx=0 tx=0).stop 2024-03-28T19:19:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.788+0000 7f161fe00640 1 -- 172.21.15.27:0/898634470 shutdown_connections 2024-03-28T19:19:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.788+0000 7f161fe00640 1 --2- 172.21.15.27:0/898634470 >> v2:172.21.15.155:3300/0 conn(0x7f1620153a30 0x7f1620153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.788+0000 7f161fe00640 1 --2- 172.21.15.27:0/898634470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620150ae0 0x7f16200a5c60 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.788+0000 7f161fe00640 1 -- 172.21.15.27:0/898634470 >> 172.21.15.27:0/898634470 conn(0x7f162006b0c0 msgr2=0x7f162006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:19:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.788+0000 7f161fe00640 1 -- 172.21.15.27:0/898634470 shutdown_connections 2024-03-28T19:19:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.788+0000 7f161fe00640 1 -- 172.21.15.27:0/898634470 wait complete. 2024-03-28T19:19:01.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.790+0000 7f161fe00640 1 Processor -- start 2024-03-28T19:19:01.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.790+0000 7f161fe00640 1 -- start start 2024-03-28T19:19:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.791+0000 7f161fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f16200a61a0 0x7f16201ebd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.791+0000 7f161fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620150ae0 0x7f1620152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.791+0000 7f161fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620153a30 0x7f16201dd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.791+0000 7f161fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162006e4d0 con 0x7f1620153a30 2024-03-28T19:19:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.791+0000 7f161fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162006e1f0 con 0x7f1620150ae0 2024-03-28T19:19:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.791+0000 7f161fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f162006e7b0 con 0x7f16200a61a0 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620153a30 0x7f16201dd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620153a30 0x7f16201dd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39660/0 (socket says 172.21.15.27:39660) 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161f400640 1 -- 172.21.15.27:0/4189044099 learned_addr learned my addr 172.21.15.27:0/4189044099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161e000640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620150ae0 0x7f1620152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161ea00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f16200a61a0 0x7f16201ebd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161f400640 1 -- 172.21.15.27:0/4189044099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f16200a61a0 msgr2=0x7f16201ebd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161f400640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f16200a61a0 0x7f16201ebd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161f400640 1 -- 172.21.15.27:0/4189044099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620150ae0 msgr2=0x7f1620152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.792+0000 7f161f400640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620150ae0 0x7f1620152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.793+0000 7f161f400640 1 -- 172.21.15.27:0/4189044099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16201de090 con 0x7f1620153a30 2024-03-28T19:19:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.793+0000 7f161f400640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620153a30 0x7f16201dd240 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f161800efc0 tx=0x7f161800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.793+0000 7f1616a00640 1 -- 172.21.15.27:0/4189044099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618018020 con 0x7f1620153a30 2024-03-28T19:19:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.793+0000 7f1616a00640 1 -- 172.21.15.27:0/4189044099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1618014480 con 0x7f1620153a30 2024-03-28T19:19:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.793+0000 7f1616a00640 1 -- 172.21.15.27:0/4189044099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618014870 con 0x7f1620153a30 2024-03-28T19:19:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.794+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16201de370 con 0x7f1620153a30 2024-03-28T19:19:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.794+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16201dca40 con 0x7f1620153a30 2024-03-28T19:19:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.795+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1620067dd0 con 0x7f1620153a30 2024-03-28T19:19:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.796+0000 7f1616a00640 1 -- 172.21.15.27:0/4189044099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1618005d10 con 0x7f1620153a30 2024-03-28T19:19:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.797+0000 7f1616a00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f160006bbd0 0x7f160006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:01.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.797+0000 7f1616a00640 1 -- 172.21.15.27:0/4189044099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f161808c630 con 0x7f1620153a30 2024-03-28T19:19:01.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.798+0000 7f161ea00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f160006bbd0 0x7f160006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:01.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.799+0000 7f161ea00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f160006bbd0 0x7f160006e090 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1610036000 tx=0x7f161003c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:01.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:01.803+0000 7f1616a00640 1 -- 172.21.15.27:0/4189044099 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1618022030 con 0x7f1620153a30 2024-03-28T19:19:02.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.096+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1620069640 con 0x7f1620153a30 2024-03-28T19:19:02.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.098+0000 7f1616a00640 1 -- 172.21.15.27:0/4189044099 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+5474 (secure 0 0 0) 0x7f1618010040 con 0x7f1620153a30 2024-03-28T19:19:02.098 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:19:02.098 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:19:02.098 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 186 2024-03-28T19:19:02.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f160006bbd0 msgr2=0x7f160006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:02.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f160006bbd0 0x7f160006e090 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1610036000 tx=0x7f161003c040 comp rx=0 tx=0).stop 2024-03-28T19:19:02.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620153a30 msgr2=0x7f16201dd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:02.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620153a30 0x7f16201dd240 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f161800efc0 tx=0x7f161800c490 comp rx=0 tx=0).stop 2024-03-28T19:19:02.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 shutdown_connections 2024-03-28T19:19:02.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f160006bbd0 0x7f160006e090 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1620153a30 0x7f16201dd240 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620150ae0 0x7f1620152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 --2- 172.21.15.27:0/4189044099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f16200a61a0 0x7f16201ebd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.101+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 >> 172.21.15.27:0/4189044099 conn(0x7f162006b0c0 msgr2=0x7f16200c0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:19:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.102+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 shutdown_connections 2024-03-28T19:19:02.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:02.102+0000 7f161fe00640 1 -- 172.21.15.27:0/4189044099 wait complete. 2024-03-28T19:19:02.121 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:19:04.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:19:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.383+0000 7f24dce00640 1 Processor -- start 2024-03-28T19:19:04.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.383+0000 7f24dce00640 1 -- start start 2024-03-28T19:19:04.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.385+0000 7f24dce00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f24d80a9cb0 0x7f24d80c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:04.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.385+0000 7f24dce00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d8154020 con 0x7f24d81534c0 2024-03-28T19:19:04.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.385+0000 7f24dce00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d80c3d90 con 0x7f24d8150c70 2024-03-28T19:19:04.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.385+0000 7f24dce00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d80ab4e0 con 0x7f24d80a9cb0 2024-03-28T19:19:04.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.385+0000 7f24d6800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f24d81534c0 0x7f24d8153910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41168/0 (socket says 172.21.15.27:41168) 2024-03-28T19:19:04.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.385+0000 7f24d6800640 1 -- 172.21.15.27:0/2704735130 learned_addr learned my addr 172.21.15.27:0/2704735130 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:19:04.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.385+0000 7f24cfe00640 1 --2- 172.21.15.27:0/2704735130 >> v2:172.21.15.155:3300/0 conn(0x7f24d80a9cb0 0x7f24d80c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.386+0000 7f24cfe00640 1 -- 172.21.15.27:0/2704735130 >> v1:172.21.15.155:6789/0 conn(0x7f24d8150c70 legacy=0x7f24d80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.386+0000 7f24cfe00640 1 -- 172.21.15.27:0/2704735130 >> v1:172.21.15.27:6789/0 conn(0x7f24d81534c0 legacy=0x7f24d8153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.386+0000 7f24cfe00640 1 -- 172.21.15.27:0/2704735130 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d80ab710 con 0x7f24d80a9cb0 2024-03-28T19:19:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.388+0000 7f24cfe00640 1 --2- 172.21.15.27:0/2704735130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d80a9cb0 0x7f24d80c3690 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f24c00097a0 tx=0x7f24c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=955db7c8a2251e36 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.388+0000 7f24d5400640 1 -- 172.21.15.27:0/2704735130 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0035020 con 0x7f24d80a9cb0 2024-03-28T19:19:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.388+0000 7f24d5400640 1 -- 172.21.15.27:0/2704735130 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c0031830 con 0x7f24d80a9cb0 2024-03-28T19:19:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.388+0000 7f24d5400640 1 -- 172.21.15.27:0/2704735130 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0031c00 con 0x7f24d80a9cb0 2024-03-28T19:19:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.389+0000 7f24dce00640 1 -- 172.21.15.27:0/2704735130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d80a9cb0 msgr2=0x7f24d80c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.389+0000 7f24dce00640 1 --2- 172.21.15.27:0/2704735130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d80a9cb0 0x7f24d80c3690 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f24c00097a0 tx=0x7f24c002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:19:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.389+0000 7f24dce00640 1 -- 172.21.15.27:0/2704735130 shutdown_connections 2024-03-28T19:19:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.389+0000 7f24dce00640 1 --2- 172.21.15.27:0/2704735130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d80a9cb0 0x7f24d80c3690 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.389+0000 7f24dce00640 1 -- 172.21.15.27:0/2704735130 >> 172.21.15.27:0/2704735130 conn(0x7f24d806b9d0 msgr2=0x7f24d80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:19:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.389+0000 7f24dce00640 1 -- 172.21.15.27:0/2704735130 shutdown_connections 2024-03-28T19:19:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.389+0000 7f24dce00640 1 -- 172.21.15.27:0/2704735130 wait complete. 2024-03-28T19:19:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.391+0000 7f24dce00640 1 Processor -- start 2024-03-28T19:19:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.391+0000 7f24dce00640 1 -- start start 2024-03-28T19:19:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24dce00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24d80a9cb0 0x7f24d8071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24dce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24d8150c70 0x7f24d80a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24dce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d81534c0 0x7f24d8070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24dce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d80abcd0 con 0x7f24d8150c70 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24dce00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d80ac1f0 con 0x7f24d81534c0 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24dce00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d80c1810 con 0x7f24d80a9cb0 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24d5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24d80a9cb0 0x7f24d8071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24d5e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24d80a9cb0 0x7f24d8071eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:35922/0 (socket says 172.21.15.27:35922) 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.393+0000 7f24d5e00640 1 -- 172.21.15.27:0/2194288589 learned_addr learned my addr 172.21.15.27:0/2194288589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.394+0000 7f24d6800640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d81534c0 0x7f24d8070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.394+0000 7f24cfe00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24d8150c70 0x7f24d80a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.394+0000 7f24d5e00640 1 -- 172.21.15.27:0/2194288589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d81534c0 msgr2=0x7f24d8070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.394+0000 7f24d5e00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d81534c0 0x7f24d8070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.394+0000 7f24d5e00640 1 -- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24d8150c70 msgr2=0x7f24d80a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.394+0000 7f24d5e00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24d8150c70 0x7f24d80a66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.394+0000 7f24d5e00640 1 -- 172.21.15.27:0/2194288589 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d81ebf00 con 0x7f24d80a9cb0 2024-03-28T19:19:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.395+0000 7f24d5e00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24d80a9cb0 0x7f24d8071eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f24c800c480 tx=0x7f24c800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:04.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.395+0000 7f24ce000640 1 -- 172.21.15.27:0/2194288589 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c80052d0 con 0x7f24d80a9cb0 2024-03-28T19:19:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.395+0000 7f24ce000640 1 -- 172.21.15.27:0/2194288589 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c8005470 con 0x7f24d80a9cb0 2024-03-28T19:19:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.395+0000 7f24ce000640 1 -- 172.21.15.27:0/2194288589 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8012870 con 0x7f24d80a9cb0 2024-03-28T19:19:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.395+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d81ec1e0 con 0x7f24d80a9cb0 2024-03-28T19:19:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.395+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d80c2f10 con 0x7f24d80a9cb0 2024-03-28T19:19:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.397+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d804f400 con 0x7f24d80a9cb0 2024-03-28T19:19:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.397+0000 7f24ce000640 1 -- 172.21.15.27:0/2194288589 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f24c8012a10 con 0x7f24d80a9cb0 2024-03-28T19:19:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.399+0000 7f24ce000640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f24b406bbd0 0x7f24b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:19:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.399+0000 7f24ce000640 1 -- 172.21.15.27:0/2194288589 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f24c808b480 con 0x7f24d80a9cb0 2024-03-28T19:19:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.399+0000 7f24cfe00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f24b406bbd0 0x7f24b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:19:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.400+0000 7f24cfe00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f24b406bbd0 0x7f24b406e090 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f24c0036000 tx=0x7f24c0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:19:04.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.406+0000 7f24ce000640 1 -- 172.21.15.27:0/2194288589 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f24c8058a40 con 0x7f24d80a9cb0 2024-03-28T19:19:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.711+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24d80735a0 con 0x7f24d80a9cb0 2024-03-28T19:19:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.713+0000 7f24ce000640 1 -- 172.21.15.27:0/2194288589 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f24c805cae0 con 0x7f24d80a9cb0 2024-03-28T19:19:04.712 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:19:04.712 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:19:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:19:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f24b406bbd0 msgr2=0x7f24b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f24b406bbd0 0x7f24b406e090 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f24c0036000 tx=0x7f24c0008030 comp rx=0 tx=0).stop 2024-03-28T19:19:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24d80a9cb0 msgr2=0x7f24d8071eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:19:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24d80a9cb0 0x7f24d8071eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f24c800c480 tx=0x7f24c800c940 comp rx=0 tx=0).stop 2024-03-28T19:19:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 shutdown_connections 2024-03-28T19:19:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f24b406bbd0 0x7f24b406e090 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24d81534c0 0x7f24d8070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24d8150c70 0x7f24d80a66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 --2- 172.21.15.27:0/2194288589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24d80a9cb0 0x7f24d8071eb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:19:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 >> 172.21.15.27:0/2194288589 conn(0x7f24d806b9d0 msgr2=0x7f24d80c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:19:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.716+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 shutdown_connections 2024-03-28T19:19:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:19:04.717+0000 7f24dce00640 1 -- 172.21.15.27:0/2194288589 wait complete. 2024-03-28T19:19:04.735 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-03-28T19:19:04.735 INFO:tasks.mds_thrash.fs.[b]:waiting for 76 secs before thrashing 2024-03-28T19:19:07.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:19:07.461 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:19:38.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:19:38.756 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:20:06.318 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 0% (82/13225) Updating files: 1% (133/13225) Updating files: 1% (146/13225) Updating files: 1% (210/13225) Updating files: 2% (265/13225) Updating files: 2% (326/13225) Updating files: 2% (389/13225) Updating files: 3% (397/13225) Updating files: 3% (455/13225) Updating files: 4% (529/13225) Updating files: 4% (531/13225) Updating files: 4% (598/13225) Updating files: 5% (662/13225) Updating files: 5% (679/13225) Updating files: 5% (750/13225) Updating files: 6% (794/13225) Updating files: 6% (816/13225) Updating files: 6% (904/13225) Updating files: 7% (926/13225) Updating files: 7% (978/13225) Updating files: 7% (1013/13225) Updating files: 8% (1058/13225) Updating files: 8% (1089/13225) Updating files: 8% (1146/13225) Updating files: 9% (1191/13225) Updating files: 9% (1202/13225) Updating files: 9% (1271/13225) Updating files: 10% (1323/13225) Updating files: 10% (1337/13225) Updating files: 10% (1382/13225) Updating files: 11% (1455/13225) Updating files: 11% (1461/13225) Updating files: 11% (1529/13225) Updating files: 11% (1578/13225) Updating files: 12% (1587/13225) Updating files: 12% (1631/13225) Updating files: 12% (1688/13225) Updating files: 13% (1720/13225) Updating files: 13% (1754/13225) Updating files: 13% (1825/13225) Updating files: 14% (1852/13225) Updating files: 14% (1891/13225) Updating files: 14% (1949/13225) Updating files: 15% (1984/13225) Updating files: 15% (2003/13225) Updating files: 15% (2063/13225) Updating files: 16% (2116/13225) Updating files: 16% (2173/13225) Updating files: 16% (2222/13225) Updating files: 17% (2249/13225) Updating files: 17% (2276/13225) Updating files: 17% (2328/13225) Updating files: 17% (2376/13225) Updating files: 18% (2381/13225) Updating files: 18% (2436/13225) Updating files: 18% (2490/13225) Updating files: 19% (2513/13225) Updating files: 19% (2543/13225) Updating files: 19% (2600/13225) Updating files: 20% (2645/13225) Updating files: 20% (2663/13225) Updating files: 20% (2714/13225) Updating files: 20% (2769/13225) Updating files: 21% (2778/13225) Updating files: 21% (2829/13225) Updating files: 21% (2906/13225) Updating files: 22% (2910/13225) Updating files: 22% (2974/13225) Updating files: 23% (3042/13225) Updating files: 23% (3053/13225) Updating files: 23% (3137/13225) Updating files: 24% (3174/13225) Updating files: 24% (3229/13225) Updating files: 25% (3307/13225) Updating files: 25% (3317/13225) Updating files: 25% (3402/13225) Updating files: 26% (3439/13225) Updating files: 26% (3471/13225) Updating files: 26% (3550/13225) Updating files: 27% (3571/13225) Updating files: 27% (3622/13225) Updating files: 27% (3691/13225) Updating files: 28% (3703/13225) Updating files: 28% (3745/13225) Updating files: 28% (3751/13225) Updating files: 28% (3819/13225) Updating files: 29% (3836/13225) Updating files: 29% (3875/13225) Updating files: 29% (3945/13225) Updating files: 30% (3968/13225) Updating files: 30% (3981/13225) Updating files: 30% (4043/13225) Updating files: 31% (4100/13225) Updating files: 31% (4104/13225) Updating files: 31% (4170/13225) Updating files: 32% (4232/13225) Updating files: 32% (4244/13225) Updating files: 32% (4323/13225) Updating files: 33% (4365/13225) Updating files: 33% (4394/13225) Updating files: 33% (4472/13225) Updating files: 34% (4497/13225) Updating files: 34% (4548/13225) Updating files: 35% (4629/13225) Updating files: 35% (4647/13225) Updating files: 35% (4728/13225) Updating files: 36% (4761/13225) Updating files: 36% (4812/13225) Updating files: 36% (4883/13225) Updating files: 37% (4894/13225) Updating files: 37% (4963/13225) Updating files: 38% (5026/13225) Updating files: 38% (5035/13225) Updating files: 38% (5114/13225) Updating files: 39% (5158/13225) Updating files: 39% (5195/13225) Updating files: 39% (5257/13225) Updating files: 40% (5290/13225) Updating files: 40% (5304/13225) Updating files: 40% (5343/13225) Updating files: 40% (5386/13225) Updating files: 41% (5423/13225) Updating files: 41% (5456/13225) Updating files: 41% (5534/13225) Updating files: 42% (5555/13225) Updating files: 42% (5608/13225) Updating files: 42% (5678/13225) Updating files: 43% (5687/13225) Updating files: 43% (5768/13225) Updating files: 44% (5819/13225) Updating files: 44% (5859/13225) Updating files: 45% (5952/13225) Updating files: 45% (5953/13225) Updating files: 45% (6037/13225) Updating files: 46% (6084/13225) Updating files: 46% (6099/13225) Updating files: 46% (6178/13225) Updating files: 47% (6216/13225) Updating files: 47% (6249/13225) Updating files: 47% (6327/13225) Updating files: 48% (6348/13225) Updating files: 48% (6405/13225) Updating files: 48% (6479/13225) Updating files: 49% (6481/13225) Updating files: 49% (6556/13225) Updating files: 50% (6613/13225) Updating files: 50% (6634/13225) Updating files: 50% (6699/13225) Updating files: 51% (6745/13225) Updating files: 51% (6778/13225) Updating files: 51% (6860/13225) Updating files: 52% (6877/13225) Updating files: 52% (6931/13225) Updating files: 53% (7010/13225) Updating files: 53% (7020/13225) Updating files: 53% (7108/13225) Updating files: 54% (7142/13225) Updating files: 54% (7199/13225) Updating files: 54% (7260/13225) Updating files: 55% (7274/13225) Updating files: 55% (7350/13225) Updating files: 56% (7406/13225) Updating files: 56% (7447/13225) Updating files: 57% (7539/13225) Updating files: 57% (7541/13225) Updating files: 57% (7635/13225) Updating files: 58% (7671/13225) Updating files: 58% (7715/13225) Updating files: 58% (7799/13225) Updating files: 59% (7803/13225) Updating files: 59% (7882/13225) Updating files: 60% (7935/13225) Updating files: 60% (7941/13225) Updating files: 60% (8012/13225) Updating files: 61% (8068/13225) Updating files: 61% (8083/13225) Updating files: 61% (8138/13225) Updating files: 61% (8184/13225) Updating files: 62% (8200/13225) Updating files: 62% (8222/13225) Updating files: 62% (8262/13225) Updating files: 62% (8299/13225) Updating files: 63% (8332/13225) Updating files: 63% (8339/13225) Updating files: 63% (8378/13225) Updating files: 63% (8417/13225) Updating files: 63% (8453/13225) Updating files: 64% (8464/13225) Updating files: 64% (8482/13225) Updating files: 64% (8515/13225) Updating files: 64% (8550/13225) Updating files: 64% (8583/13225) Updating files: 64% (8593/13225) Updating files: 65% (8597/13225) Updating files: 65% (8641/13225) Updating files: 65% (8683/13225) Updating files: 65% (8720/13225) Updating files: 66% (8729/13225) Updating files: 66% (8757/13225) Updating files: 66% (8794/13225) Updating files: 66% (8840/13225) Updating files: 67% (8861/13225) Updating files: 67% (8888/13225) Updating files: 67% (8926/13225) Updating files: 67% (8992/13225) Updating files: 68% (8993/13225) Updating files: 68% (9045/13225) Updating files: 68% (9084/13225) Updating files: 68% (9121/13225) Updating files: 69% (9126/13225) Updating files: 69% (9164/13225) Updating files: 69% (9194/13225) Updating files: 69% (9249/13225) Updating files: 70% (9258/13225) Updating files: 70% (9297/13225) Updating files: 70% (9365/13225) Updating files: 71% (9390/13225) Updating files: 71% (9426/13225) Updating files: 71% (9493/13225) Updating files: 72% (9522/13225) Updating files: 72% (9555/13225) Updating files: 72% (9628/13225) Updating files: 73% (9655/13225) Updating files: 73% (9708/13225) Updating files: 73% (9783/13225) Updating files: 74% (9787/13225) Updating files: 74% (9853/13225) Updating files: 75% (9919/13225) Updating files: 75% (9999/13225) Updating files: 76% (10051/13225) Updating files: 76% (10075/13225) Updating files: 76% (10166/13225) Updating files: 77% (10184/13225) Updating files: 77% (10254/13225) Updating files: 78% (10316/13225) Updating files: 78% (10350/13225) Updating files: 78% (10428/13225) Updating files: 79% (10448/13225) Updating files: 79% (10508/13225) Updating files: 80% (10580/13225) Updating files: 80% (10587/13225) Updating files: 80% (10649/13225) Updating files: 81% (10713/13225) Updating files: 81% (10721/13225) Updating files: 81% (10792/13225) Updating files: 82% (10845/13225) Updating files: 82% (10852/13225) Updating files: 82% (10917/13225) Updating files: 83% (10977/13225) Updating files: 83% (10980/13225) Updating files: 83% (11052/13225) Updating files: 84% (11109/13225) Updating files: 84% (11136/13225) Updating files: 84% (11220/13225) Updating files: 85% (11242/13225) Updating files: 85% (11299/13225) Updating files: 86% (11374/13225) Updating files: 86% (11376/13225) Updating files: 86% (11464/13225) Updating files: 87% (11506/13225) Updating files: 87% (11552/13225) Updating files: 88% (11638/13225) Updating files: 88% (11740/13225) Updating files: 89% (11771/13225) Updating files: 89% (11829/13225) Updating files: 89% (11891/13225) Updating files: 90% (11903/13225) Updating files: 90% (11977/13225) Updating files: 91% (12035/13225) Updating files: 91% (12062/13225) Updating files: 91% (12140/13225) Updating files: 92% (12167/13225) Updating files: 92% (12217/13225) Updating files: 92% (12283/13225) Updating files: 93% (12300/13225) Updating files: 93% (12343/13225) Updating files: 93% (12402/13225) Updating files: 94% (12432/13225) Updating files: 94% (12461/13225) Updating files: 94% (12536/13225) Updating files: 95% (12564/13225) Updating files: 95% (12595/13225) Updating files: 95% (12650/13225) Updating files: 96% (12696/13225) Updating files: 96% (12705/13225) Updating files: 96% (12775/13225) Updating files: 97% (12829/13225) Updating files: 97% (12831/13225) Updating files: 97% (12887/13225) Updating files: 97% (12948/13225) Updating files: 98% (12961/13225) Updating files: 98% (12996/13225) Updating files: 98% (13041/13225) Updating files: 98% (13089/13225) Updating files: 99% (13093/13225) Updating files: 99% (13139/13225) Updating files: 99% (13196/13225) Updating files: 100% (13225/13225) Updating files: 100% (13225/13225), done. 2024-03-28T19:20:06.670 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2024-03-28T19:20:06.691 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.1 2024-03-28T19:20:06.844 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:20:06.845 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 2 -eq 48 ']' 2024-03-28T19:20:06.845 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.2 2024-03-28T19:20:06.845 INFO:tasks.workunit.client.0.smithi027.stdout:v0.2 2024-03-28T19:20:06.845 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.2 2024-03-28T19:20:06.845 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.2 2024-03-28T19:20:08.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:20:08.930 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:20:15.227 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 21% (47/222) Updating files: 22% (49/222) Updating files: 23% (52/222) Updating files: 24% (54/222) Updating files: 25% (56/222) Updating files: 26% (58/222) Updating files: 27% (60/222) Updating files: 28% (63/222) Updating files: 29% (65/222) Updating files: 30% (67/222) Updating files: 31% (69/222) Updating files: 32% (72/222) Updating files: 33% (74/222) Updating files: 34% (76/222) Updating files: 35% (78/222) Updating files: 36% (80/222) Updating files: 37% (83/222) Updating files: 38% (85/222) Updating files: 39% (87/222) Updating files: 40% (89/222) Updating files: 41% (92/222) Updating files: 42% (94/222) Updating files: 43% (96/222) Updating files: 44% (98/222) Updating files: 45% (100/222) Updating files: 46% (103/222) Updating files: 47% (105/222) Updating files: 48% (107/222) Updating files: 49% (109/222) Updating files: 50% (111/222) Updating files: 51% (114/222) Updating files: 52% (116/222) Updating files: 53% (118/222) Updating files: 54% (120/222) Updating files: 55% (123/222) Updating files: 56% (125/222) Updating files: 57% (127/222) Updating files: 58% (129/222) Updating files: 58% (130/222) Updating files: 59% (131/222) Updating files: 60% (134/222) Updating files: 61% (136/222) Updating files: 62% (138/222) Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 81% (181/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2024-03-28T19:20:15.589 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 2b5f76f48e9 readme 2024-03-28T19:20:15.608 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.2 2024-03-28T19:20:15.715 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:20:15.716 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 3 -eq 48 ']' 2024-03-28T19:20:15.716 INFO:tasks.workunit.client.0.smithi027.stdout:v0.3 2024-03-28T19:20:15.716 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.3 2024-03-28T19:20:15.716 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.3 2024-03-28T19:20:15.716 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.3 2024-03-28T19:20:20.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.832+0000 7ff016c00640 1 Processor -- start 2024-03-28T19:20:21.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.832+0000 7ff016c00640 1 -- start start 2024-03-28T19:20:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff016c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0100c60e0 0x7ff01006e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff016c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c3ea0 0x7ff01006ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff016c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff010072df0 0x7ff010158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff016c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0100c23b0 con 0x7ff0100c60e0 2024-03-28T19:20:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff016c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0100c1600 con 0x7ff010072df0 2024-03-28T19:20:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff016c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0100c18e0 con 0x7ff0100c3ea0 2024-03-28T19:20:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff00fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0100c60e0 0x7ff01006e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:21.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff00fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0100c60e0 0x7ff01006e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41294/0 (socket says 172.21.15.27:41294) 2024-03-28T19:20:21.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff00fe00640 1 -- 172.21.15.27:0/2850340530 learned_addr learned my addr 172.21.15.27:0/2850340530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:21.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.834+0000 7ff014c00640 1 --2- 172.21.15.27:0/2850340530 >> v2:172.21.15.155:3300/0 conn(0x7ff010072df0 0x7ff010158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:21.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.835+0000 7ff00f400640 1 --2- 172.21.15.27:0/2850340530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c3ea0 0x7ff01006ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.835+0000 7ff00fe00640 1 -- 172.21.15.27:0/2850340530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c3ea0 msgr2=0x7ff01006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.835+0000 7ff00fe00640 1 --2- 172.21.15.27:0/2850340530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c3ea0 0x7ff01006ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.835+0000 7ff00fe00640 1 -- 172.21.15.27:0/2850340530 >> v2:172.21.15.155:3300/0 conn(0x7ff010072df0 msgr2=0x7ff010158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.835+0000 7ff00fe00640 1 --2- 172.21.15.27:0/2850340530 >> v2:172.21.15.155:3300/0 conn(0x7ff010072df0 0x7ff010158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.835+0000 7ff00fe00640 1 -- 172.21.15.27:0/2850340530 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0100c1bc0 con 0x7ff0100c60e0 2024-03-28T19:20:21.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.836+0000 7ff00fe00640 1 --2- 172.21.15.27:0/2850340530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0100c60e0 0x7ff01006e6f0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7ff0100c3290 tx=0x7feff802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbabdcf931c45a51 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:21.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.837+0000 7ff00ea00640 1 -- 172.21.15.27:0/2850340530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8038070 con 0x7ff0100c60e0 2024-03-28T19:20:21.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.837+0000 7ff00ea00640 1 -- 172.21.15.27:0/2850340530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff8031740 con 0x7ff0100c60e0 2024-03-28T19:20:21.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.837+0000 7ff00ea00640 1 -- 172.21.15.27:0/2850340530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff802f050 con 0x7ff0100c60e0 2024-03-28T19:20:21.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 -- 172.21.15.27:0/2850340530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0100c60e0 msgr2=0x7ff01006e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:21.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 --2- 172.21.15.27:0/2850340530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0100c60e0 0x7ff01006e6f0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7ff0100c3290 tx=0x7feff802b7e0 comp rx=0 tx=0).stop 2024-03-28T19:20:21.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 -- 172.21.15.27:0/2850340530 shutdown_connections 2024-03-28T19:20:21.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 --2- 172.21.15.27:0/2850340530 >> v2:172.21.15.155:3300/0 conn(0x7ff010072df0 0x7ff010158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:21.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 --2- 172.21.15.27:0/2850340530 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c3ea0 0x7ff01006ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:21.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 --2- 172.21.15.27:0/2850340530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0100c60e0 0x7ff01006e6f0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:21.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 -- 172.21.15.27:0/2850340530 >> 172.21.15.27:0/2850340530 conn(0x7ff01006b9d0 msgr2=0x7ff0100c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:21.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 -- 172.21.15.27:0/2850340530 shutdown_connections 2024-03-28T19:20:21.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.838+0000 7ff016c00640 1 -- 172.21.15.27:0/2850340530 wait complete. 2024-03-28T19:20:21.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.840+0000 7ff016c00640 1 Processor -- start 2024-03-28T19:20:21.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.841+0000 7ff016c00640 1 -- start start 2024-03-28T19:20:21.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff016c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff010072df0 0x7ff0101eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:21.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff016c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0100c3ea0 0x7ff010155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:21.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff016c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c60e0 0x7ff0101ee310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff016c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0100c6640 con 0x7ff010072df0 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff016c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0100c64c0 con 0x7ff0100c3ea0 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff016c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0100c4f00 con 0x7ff0100c60e0 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff010072df0 0x7ff0101eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff010072df0 0x7ff0101eb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41308/0 (socket says 172.21.15.27:41308) 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00fe00640 1 -- 172.21.15.27:0/3154541613 learned_addr learned my addr 172.21.15.27:0/3154541613 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00f400640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0100c3ea0 0x7ff010155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff014c00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c60e0 0x7ff0101ee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00fe00640 1 -- 172.21.15.27:0/3154541613 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c60e0 msgr2=0x7ff0101ee310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00fe00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c60e0 0x7ff0101ee310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00fe00640 1 -- 172.21.15.27:0/3154541613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0100c3ea0 msgr2=0x7ff010155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:21.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00fe00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0100c3ea0 0x7ff010155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:21.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.842+0000 7ff00fe00640 1 -- 172.21.15.27:0/3154541613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0100abb20 con 0x7ff010072df0 2024-03-28T19:20:21.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.843+0000 7ff00fe00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff010072df0 0x7ff0101eb7c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7feff802b820 tx=0x7feff80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:21.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.843+0000 7ff007e00640 1 -- 172.21.15.27:0/3154541613 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8042070 con 0x7ff010072df0 2024-03-28T19:20:21.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.843+0000 7ff007e00640 1 -- 172.21.15.27:0/3154541613 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff8031e80 con 0x7ff010072df0 2024-03-28T19:20:21.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.843+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0101ebd00 con 0x7ff010072df0 2024-03-28T19:20:21.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.843+0000 7ff007e00640 1 -- 172.21.15.27:0/3154541613 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8038040 con 0x7ff010072df0 2024-03-28T19:20:21.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.844+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0100abd90 con 0x7ff010072df0 2024-03-28T19:20:21.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.846+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01004f400 con 0x7ff010072df0 2024-03-28T19:20:21.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.846+0000 7ff007e00640 1 -- 172.21.15.27:0/3154541613 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feff8045070 con 0x7ff010072df0 2024-03-28T19:20:21.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.847+0000 7ff007e00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fefec06bc20 0x7fefec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:21.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.847+0000 7ff007e00640 1 -- 172.21.15.27:0/3154541613 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7feff80ad100 con 0x7ff010072df0 2024-03-28T19:20:21.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.848+0000 7ff00f400640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fefec06bc20 0x7fefec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:21.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.849+0000 7ff00f400640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fefec06bc20 0x7fefec06e0e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff0100a5b90 tx=0x7ff000008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:21.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:21.854+0000 7ff007e00640 1 -- 172.21.15.27:0/3154541613 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff0101ebd00 con 0x7ff010072df0 2024-03-28T19:20:22.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.254+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0100c4ce0 con 0x7ff010072df0 2024-03-28T19:20:22.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.255+0000 7ff007e00640 1 -- 172.21.15.27:0/3154541613 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7feff807a740 con 0x7ff010072df0 2024-03-28T19:20:22.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:22.254 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:20:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:20:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fefec06bc20 msgr2=0x7fefec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fefec06bc20 0x7fefec06e0e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff0100a5b90 tx=0x7ff000008040 comp rx=0 tx=0).stop 2024-03-28T19:20:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff010072df0 msgr2=0x7ff0101eb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff010072df0 0x7ff0101eb7c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7feff802b820 tx=0x7feff80046c0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 shutdown_connections 2024-03-28T19:20:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fefec06bc20 0x7fefec06e0e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0100c60e0 0x7ff0101ee310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0100c3ea0 0x7ff010155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 --2- 172.21.15.27:0/3154541613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff010072df0 0x7ff0101eb7c0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.258+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 >> 172.21.15.27:0/3154541613 conn(0x7ff01006b9d0 msgr2=0x7ff010159560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.259+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 shutdown_connections 2024-03-28T19:20:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.259+0000 7ff016c00640 1 -- 172.21.15.27:0/3154541613 wait complete. 2024-03-28T19:20:22.281 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-03-28T19:20:22.281 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T19:20:22.281 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:20:22.312 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T19:20:22.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.553+0000 7f7d74200640 1 Processor -- start 2024-03-28T19:20:22.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.554+0000 7f7d74200640 1 -- start start 2024-03-28T19:20:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.555+0000 7f7d74200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7d6c0a9cb0 0x7f7d6c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.555+0000 7f7d74200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d6c1534c0 0x7f7d6c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.555+0000 7f7d74200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c0c3d90 con 0x7f7d6c150c70 2024-03-28T19:20:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.555+0000 7f7d74200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6c0ab4e0 con 0x7f7d6c0a9cb0 2024-03-28T19:20:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.555+0000 7f7d74200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0ac1f0 con 0x7f7d6c1534c0 2024-03-28T19:20:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.556+0000 7f7d71800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7d6c150c70 0x7f7d6c0c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47960/0 (socket says 172.21.15.27:47960) 2024-03-28T19:20:22.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.556+0000 7f7d71800640 1 -- 172.21.15.27:0/4209651568 learned_addr learned my addr 172.21.15.27:0/4209651568 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.556+0000 7f7d72200640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d6c1534c0 0x7f7d6c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.556+0000 7f7d70e00640 1 --2- 172.21.15.27:0/4209651568 >> v2:172.21.15.155:3300/0 conn(0x7f7d6c0a9cb0 0x7f7d6c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329858872 0 0) 0x7f7d6c0c3d90 con 0x7f7d6c150c70 2024-03-28T19:20:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d54003520 con 0x7f7d6c150c70 2024-03-28T19:20:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 942517293 0 0) 0x7f7d54003520 con 0x7f7d6c150c70 2024-03-28T19:20:22.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d6c1534c0 msgr2=0x7f7d6c153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d6c1534c0 0x7f7d6c153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 >> v2:172.21.15.155:3300/0 conn(0x7f7d6c0a9cb0 msgr2=0x7f7d6c0c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 --2- 172.21.15.27:0/4209651568 >> v2:172.21.15.155:3300/0 conn(0x7f7d6c0a9cb0 0x7f7d6c0c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c0ab710 con 0x7f7d6c150c70 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f7d60001670 con 0x7f7d6c150c70 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 >> v1:172.21.15.27:6789/0 conn(0x7f7d6c150c70 legacy=0x7f7d6c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d6c1534c0 0x7f7d540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d54003f30 0x7f7d54004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d54004d40 0x7f7d540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0ac1f0 con 0x7f7d54003f30 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5400b9c0 con 0x7f7d54004d40 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.557+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5400bb90 con 0x7f7d6c1534c0 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.558+0000 7f7d70e00640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d54003f30 0x7f7d54004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:22.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.558+0000 7f7d72200640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d54004d40 0x7f7d540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.558+0000 7f7d70e00640 1 -- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d6c1534c0 msgr2=0x7f7d540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.558+0000 7f7d70e00640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d6c1534c0 0x7f7d540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.558+0000 7f7d70e00640 1 -- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d54004d40 msgr2=0x7f7d540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.558+0000 7f7d70e00640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d54004d40 0x7f7d540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.558+0000 7f7d70e00640 1 -- 172.21.15.27:0/4209651568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c0ab710 con 0x7f7d54003f30 2024-03-28T19:20:22.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.559+0000 7f7d70e00640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d54003f30 0x7f7d54004800 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009af0 tx=0x7f7d54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad99412d9ac4cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:22.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.560+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d58039070 con 0x7f7d54003f30 2024-03-28T19:20:22.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.560+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5802c6f0 con 0x7f7d54003f30 2024-03-28T19:20:22.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.560+0000 7f7d6be00640 1 -- 172.21.15.27:0/4209651568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d58009690 con 0x7f7d54003f30 2024-03-28T19:20:22.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.561+0000 7f7d74200640 1 -- 172.21.15.27:0/4209651568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d54003f30 msgr2=0x7f7d54004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:22.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.561+0000 7f7d74200640 1 --2- 172.21.15.27:0/4209651568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d54003f30 0x7f7d54004800 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009af0 tx=0x7f7d54005830 comp rx=0 tx=0).stop 2024-03-28T19:20:22.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.561+0000 7f7d72200640 1 -- 172.21.15.27:0/4209651568 reap_dead start 2024-03-28T19:20:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.561+0000 7f7d74200640 1 -- 172.21.15.27:0/4209651568 shutdown_connections 2024-03-28T19:20:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.561+0000 7f7d74200640 1 -- 172.21.15.27:0/4209651568 >> 172.21.15.27:0/4209651568 conn(0x7f7d6c06b9d0 msgr2=0x7f7d6c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.562+0000 7f7d74200640 1 -- 172.21.15.27:0/4209651568 shutdown_connections 2024-03-28T19:20:22.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.562+0000 7f7d74200640 1 -- 172.21.15.27:0/4209651568 wait complete. 2024-03-28T19:20:22.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.564+0000 7f7d74200640 1 Processor -- start 2024-03-28T19:20:22.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.564+0000 7f7d74200640 1 -- start start 2024-03-28T19:20:22.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.565+0000 7f7d74200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d54003f30 0x7f7d6c06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.565+0000 7f7d74200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d6c1ef0e0 0x7f7d6c071f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.565+0000 7f7d74200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d6c06e2d0 0x7f7d6c144b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.565+0000 7f7d74200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0c1810 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.565+0000 7f7d74200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0abcd0 con 0x7f7d6c06e2d0 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.565+0000 7f7d74200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0c6830 con 0x7f7d54003f30 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.566+0000 7f7d70e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d6c1ef0e0 0x7f7d6c071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.566+0000 7f7d70e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d6c1ef0e0 0x7f7d6c071f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41324/0 (socket says 172.21.15.27:41324) 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.566+0000 7f7d70e00640 1 -- 172.21.15.27:0/2620139376 learned_addr learned my addr 172.21.15.27:0/2620139376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.566+0000 7f7d71800640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d54003f30 0x7f7d6c06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:22.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.566+0000 7f7d72200640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d6c06e2d0 0x7f7d6c144b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:22.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d70e00640 1 -- 172.21.15.27:0/2620139376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d54003f30 msgr2=0x7f7d6c06dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d70e00640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d54003f30 0x7f7d6c06dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d70e00640 1 -- 172.21.15.27:0/2620139376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d6c06e2d0 msgr2=0x7f7d6c144b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d70e00640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d6c06e2d0 0x7f7d6c144b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d70e00640 1 -- 172.21.15.27:0/2620139376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c06e7f0 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d70e00640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d6c1ef0e0 0x7f7d6c071f60 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f7d58037040 tx=0x7f7d58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d69600640 1 -- 172.21.15.27:0/2620139376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d58039050 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d69600640 1 -- 172.21.15.27:0/2620139376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d58045070 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.567+0000 7f7d69600640 1 -- 172.21.15.27:0/2620139376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d580044e0 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.568+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6c1ef570 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.568+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6c1eb270 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.570+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6c04f400 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.570+0000 7f7d69600640 1 -- 172.21.15.27:0/2620139376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7d58003a50 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.571+0000 7f7d69600640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d4c06bbd0 0x7f7d4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.571+0000 7f7d69600640 1 -- 172.21.15.27:0/2620139376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f7d580ad4b0 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.572+0000 7f7d71800640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d4c06bbd0 0x7f7d4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.573+0000 7f7d71800640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d4c06bbd0 0x7f7d4c06e090 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7d60001050 tx=0x7f7d60001ad0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.577+0000 7f7d69600640 1 -- 172.21.15.27:0/2620139376 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7d5804e020 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.933+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d6c0c3ad0 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.933+0000 7f7d69600640 1 -- 172.21.15.27:0/2620139376 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f7d5807ab90 con 0x7f7d6c1ef0e0 2024-03-28T19:20:22.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:22.933 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:20:22.933 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:20:22.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d4c06bbd0 msgr2=0x7f7d4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:22.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d4c06bbd0 0x7f7d4c06e090 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7d60001050 tx=0x7f7d60001ad0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d6c1ef0e0 msgr2=0x7f7d6c071f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d6c1ef0e0 0x7f7d6c071f60 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f7d58037040 tx=0x7f7d58004730 comp rx=0 tx=0).stop 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 shutdown_connections 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7d4c06bbd0 0x7f7d4c06e090 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d6c06e2d0 0x7f7d6c144b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d6c1ef0e0 0x7f7d6c071f60 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 --2- 172.21.15.27:0/2620139376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d54003f30 0x7f7d6c06dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 >> 172.21.15.27:0/2620139376 conn(0x7f7d6c06b9d0 msgr2=0x7f7d6c151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.936+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 shutdown_connections 2024-03-28T19:20:22.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:22.937+0000 7f7d74200640 1 -- 172.21.15.27:0/2620139376 wait complete. 2024-03-28T19:20:22.957 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:20:24.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:24.985 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 16% (50/295) Updating files: 17% (51/295) Updating files: 18% (54/295) Updating files: 19% (57/295) Updating files: 20% (59/295) Updating files: 21% (62/295) Updating files: 22% (65/295) Updating files: 23% (68/295) Updating files: 24% (71/295) Updating files: 25% (74/295) Updating files: 26% (77/295) Updating files: 27% (80/295) Updating files: 28% (83/295) Updating files: 29% (86/295) Updating files: 30% (89/295) Updating files: 31% (92/295) Updating files: 32% (95/295) Updating files: 33% (98/295) Updating files: 34% (101/295) Updating files: 35% (104/295) Updating files: 36% (107/295) Updating files: 37% (110/295) Updating files: 38% (113/295) Updating files: 39% (116/295) Updating files: 40% (118/295) Updating files: 41% (121/295) Updating files: 42% (124/295) Updating files: 43% (127/295) Updating files: 43% (128/295) Updating files: 44% (130/295) Updating files: 45% (133/295) Updating files: 46% (136/295) Updating files: 47% (139/295) Updating files: 48% (142/295) Updating files: 49% (145/295) Updating files: 50% (148/295) Updating files: 51% (151/295) Updating files: 52% (154/295) Updating files: 53% (157/295) Updating files: 53% (159/295) Updating files: 54% (160/295) Updating files: 55% (163/295) Updating files: 56% (166/295) Updating files: 57% (169/295) Updating files: 58% (172/295) Updating files: 59% (175/295) Updating files: 60% (177/295) Updating files: 61% (180/295) Updating files: 62% (183/295) Updating files: 63% (186/295) Updating files: 64% (189/295) Updating files: 65% (192/295) Updating files: 66% (195/295) Updating files: 67% (198/295) Updating files: 68% (201/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 86% (256/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2024-03-28T19:20:25.163 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2024-03-28T19:20:25.183 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.3 2024-03-28T19:20:25.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.209+0000 7f51b7200640 1 Processor -- start 2024-03-28T19:20:25.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.209+0000 7f51b7200640 1 -- start start 2024-03-28T19:20:25.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.211+0000 7f51b7200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51b00a9cb0 0x7f51b00c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.211+0000 7f51b7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b0153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.211+0000 7f51b7200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b00c3d90 con 0x7f51b0150c70 2024-03-28T19:20:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.211+0000 7f51b7200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51b00ab4e0 con 0x7f51b00a9cb0 2024-03-28T19:20:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.211+0000 7f51b7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b00ac1f0 con 0x7f51b01534c0 2024-03-28T19:20:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.211+0000 7f51af400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51b00a9cb0 0x7f51b00c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.212+0000 7f51af400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f51b00a9cb0 0x7f51b00c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41332/0 (socket says 172.21.15.27:41332) 2024-03-28T19:20:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.212+0000 7f51af400640 1 -- 172.21.15.27:0/1087819691 learned_addr learned my addr 172.21.15.27:0/1087819691 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:25.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.212+0000 7f51b5200640 1 --2- 172.21.15.27:0/1087819691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b0153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:25.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.212+0000 7f51af400640 1 -- 172.21.15.27:0/1087819691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 msgr2=0x7f51b0153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:25.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.212+0000 7f51af400640 1 --2- 172.21.15.27:0/1087819691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b0153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.212+0000 7f51af400640 1 -- 172.21.15.27:0/1087819691 >> v1:172.21.15.155:6789/0 conn(0x7f51b0150c70 legacy=0x7f51b00c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:25.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.212+0000 7f51af400640 1 -- 172.21.15.27:0/1087819691 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b00ab710 con 0x7f51b00a9cb0 2024-03-28T19:20:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.214+0000 7f51af400640 1 --2- 172.21.15.27:0/1087819691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b00c3690 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f519c009870 tx=0x7f519c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8e267874c995e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.214+0000 7f51aea00640 1 -- 172.21.15.27:0/1087819691 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c035020 con 0x7f51b00a9cb0 2024-03-28T19:20:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.214+0000 7f51aea00640 1 -- 172.21.15.27:0/1087819691 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519c031640 con 0x7f51b00a9cb0 2024-03-28T19:20:25.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.214+0000 7f51aea00640 1 -- 172.21.15.27:0/1087819691 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c02d040 con 0x7f51b00a9cb0 2024-03-28T19:20:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.215+0000 7f51b7200640 1 -- 172.21.15.27:0/1087819691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 msgr2=0x7f51b00c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:25.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.215+0000 7f51b7200640 1 --2- 172.21.15.27:0/1087819691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b00c3690 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f519c009870 tx=0x7f519c02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.215+0000 7f51b7200640 1 -- 172.21.15.27:0/1087819691 shutdown_connections 2024-03-28T19:20:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.215+0000 7f51b7200640 1 --2- 172.21.15.27:0/1087819691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b0153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.215+0000 7f51b7200640 1 --2- 172.21.15.27:0/1087819691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b00c3690 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.215+0000 7f51b7200640 1 -- 172.21.15.27:0/1087819691 >> 172.21.15.27:0/1087819691 conn(0x7f51b006b9d0 msgr2=0x7f51b00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.216+0000 7f51b7200640 1 -- 172.21.15.27:0/1087819691 shutdown_connections 2024-03-28T19:20:25.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.216+0000 7f51b7200640 1 -- 172.21.15.27:0/1087819691 wait complete. 2024-03-28T19:20:25.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.218+0000 7f51b7200640 1 Processor -- start 2024-03-28T19:20:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.219+0000 7f51b7200640 1 -- start start 2024-03-28T19:20:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.220+0000 7f51b7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b014d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.220+0000 7f51b7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51b0150c70 0x7f51b006ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.220+0000 7f51b7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b014ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.220+0000 7f51b7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b00c1810 con 0x7f51b00a9cb0 2024-03-28T19:20:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.220+0000 7f51b7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b00abcd0 con 0x7f51b0150c70 2024-03-28T19:20:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.220+0000 7f51b7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b00c6830 con 0x7f51b01534c0 2024-03-28T19:20:25.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b014d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b014d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41342/0 (socket says 172.21.15.27:41342) 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51afe00640 1 -- 172.21.15.27:0/3381265746 learned_addr learned my addr 172.21.15.27:0/3381265746 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51b5200640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b014ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51af400640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51b0150c70 0x7f51b006ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51afe00640 1 -- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 msgr2=0x7f51b014ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51afe00640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b014ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51afe00640 1 -- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51b0150c70 msgr2=0x7f51b006ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51afe00640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51b0150c70 0x7f51b006ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51afe00640 1 -- 172.21.15.27:0/3381265746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b014e910 con 0x7f51b00a9cb0 2024-03-28T19:20:25.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.221+0000 7f51b5200640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b014ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:20:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.222+0000 7f51afe00640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b014d620 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f51a000c430 tx=0x7f51a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.222+0000 7f51a7e00640 1 -- 172.21.15.27:0/3381265746 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a00052d0 con 0x7f51b00a9cb0 2024-03-28T19:20:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.222+0000 7f51a7e00640 1 -- 172.21.15.27:0/3381265746 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a0005490 con 0x7f51b00a9cb0 2024-03-28T19:20:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.222+0000 7f51a7e00640 1 -- 172.21.15.27:0/3381265746 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a0007260 con 0x7f51b00a9cb0 2024-03-28T19:20:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.222+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51b006d4f0 con 0x7f51b00a9cb0 2024-03-28T19:20:25.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.223+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51b014ce20 con 0x7f51b00a9cb0 2024-03-28T19:20:25.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.225+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51b0068510 con 0x7f51b00a9cb0 2024-03-28T19:20:25.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.225+0000 7f51a7e00640 1 -- 172.21.15.27:0/3381265746 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f51a0007400 con 0x7f51b00a9cb0 2024-03-28T19:20:25.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.226+0000 7f51a7e00640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f519006bc00 0x7f519006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:25.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.226+0000 7f51a7e00640 1 -- 172.21.15.27:0/3381265746 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f51a008ae20 con 0x7f51b00a9cb0 2024-03-28T19:20:25.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.229+0000 7f51af400640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f519006bc00 0x7f519006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:25.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.230+0000 7f51af400640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f519006bc00 0x7f519006e0c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f519c036000 tx=0x7f519c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:25.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.233+0000 7f51a7e00640 1 -- 172.21.15.27:0/3381265746 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f51b0068510 con 0x7f51b00a9cb0 2024-03-28T19:20:25.240 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:20:25.240 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 4 -eq 48 ']' 2024-03-28T19:20:25.240 INFO:tasks.workunit.client.0.smithi027.stdout:v0.4 2024-03-28T19:20:25.240 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.4 2024-03-28T19:20:25.240 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.4 2024-03-28T19:20:25.241 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.4 2024-03-28T19:20:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.541+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51b004f400 con 0x7f51b00a9cb0 2024-03-28T19:20:25.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.542+0000 7f51a7e00640 1 -- 172.21.15.27:0/3381265746 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f51a00583b0 con 0x7f51b00a9cb0 2024-03-28T19:20:25.541 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:25.541 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:20:25.541 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.544+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f519006bc00 msgr2=0x7f519006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.544+0000 7f51b7200640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f519006bc00 0x7f519006e0c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f519c036000 tx=0x7f519c03c040 comp rx=0 tx=0).stop 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.544+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 msgr2=0x7f51b014d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.544+0000 7f51b7200640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b014d620 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f51a000c430 tx=0x7f51a000c8f0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.545+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 shutdown_connections 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.545+0000 7f51b7200640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f519006bc00 0x7f519006e0c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.545+0000 7f51b7200640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51b01534c0 0x7f51b014ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.545+0000 7f51b7200640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51b0150c70 0x7f51b006ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.545+0000 7f51b7200640 1 --2- 172.21.15.27:0/3381265746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b00a9cb0 0x7f51b014d620 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.545+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 >> 172.21.15.27:0/3381265746 conn(0x7f51b006b9d0 msgr2=0x7f51b00c1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.545+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 shutdown_connections 2024-03-28T19:20:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:25.545+0000 7f51b7200640 1 -- 172.21.15.27:0/3381265746 wait complete. 2024-03-28T19:20:25.564 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:20:27.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.802+0000 7f1eee800640 1 Processor -- start 2024-03-28T19:20:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.802+0000 7f1eee800640 1 -- start start 2024-03-28T19:20:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.804+0000 7f1eee800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ee80a7060 0x7f1ee80c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.804+0000 7f1eee800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1ee8152ad0 0x7f1ee806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.804+0000 7f1eee800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee8072f90 con 0x7f1ee8155700 2024-03-28T19:20:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.804+0000 7f1eee800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee8072d40 con 0x7f1ee80a7060 2024-03-28T19:20:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.804+0000 7f1eee800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee8073550 con 0x7f1ee8152ad0 2024-03-28T19:20:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.805+0000 7f1eed400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ee80a7060 0x7f1ee80c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.805+0000 7f1eed400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ee80a7060 0x7f1ee80c2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43000/0 (socket says 172.21.15.27:43000) 2024-03-28T19:20:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.805+0000 7f1eed400640 1 -- 172.21.15.27:0/3751538594 learned_addr learned my addr 172.21.15.27:0/3751538594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.805+0000 7f1eede00640 1 --2- 172.21.15.27:0/3751538594 >> v2:172.21.15.155:3300/0 conn(0x7f1ee8152ad0 0x7f1ee806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.805+0000 7f1eed400640 1 -- 172.21.15.27:0/3751538594 >> v2:172.21.15.155:3300/0 conn(0x7f1ee8152ad0 msgr2=0x7f1ee806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.805+0000 7f1eed400640 1 --2- 172.21.15.27:0/3751538594 >> v2:172.21.15.155:3300/0 conn(0x7f1ee8152ad0 0x7f1ee806c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.805+0000 7f1eed400640 1 -- 172.21.15.27:0/3751538594 >> v1:172.21.15.155:6789/0 conn(0x7f1ee8155700 legacy=0x7f1ee80c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.805+0000 7f1eed400640 1 -- 172.21.15.27:0/3751538594 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee8073270 con 0x7f1ee80a7060 2024-03-28T19:20:27.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.807+0000 7f1eed400640 1 --2- 172.21.15.27:0/3751538594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee80a7060 0x7f1ee80c2530 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1ee80c5e60 tx=0x7f1edc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4862ca637bf3b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.807+0000 7f1ee7e00640 1 -- 172.21.15.27:0/3751538594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc034070 con 0x7f1ee80a7060 2024-03-28T19:20:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.807+0000 7f1ee7e00640 1 -- 172.21.15.27:0/3751538594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc030860 con 0x7f1ee80a7060 2024-03-28T19:20:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.807+0000 7f1ee7e00640 1 -- 172.21.15.27:0/3751538594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc030c30 con 0x7f1ee80a7060 2024-03-28T19:20:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.808+0000 7f1eee800640 1 -- 172.21.15.27:0/3751538594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee80a7060 msgr2=0x7f1ee80c2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.808+0000 7f1eee800640 1 --2- 172.21.15.27:0/3751538594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee80a7060 0x7f1ee80c2530 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1ee80c5e60 tx=0x7f1edc02b9a0 comp rx=0 tx=0).stop 2024-03-28T19:20:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.808+0000 7f1eee800640 1 -- 172.21.15.27:0/3751538594 shutdown_connections 2024-03-28T19:20:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.808+0000 7f1eee800640 1 --2- 172.21.15.27:0/3751538594 >> v2:172.21.15.155:3300/0 conn(0x7f1ee8152ad0 0x7f1ee806c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.808+0000 7f1eee800640 1 --2- 172.21.15.27:0/3751538594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee80a7060 0x7f1ee80c2530 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.808+0000 7f1eee800640 1 -- 172.21.15.27:0/3751538594 >> 172.21.15.27:0/3751538594 conn(0x7f1ee80c7aa0 msgr2=0x7f1ee80c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.808+0000 7f1eee800640 1 -- 172.21.15.27:0/3751538594 shutdown_connections 2024-03-28T19:20:27.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.808+0000 7f1eee800640 1 -- 172.21.15.27:0/3751538594 wait complete. 2024-03-28T19:20:27.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.810+0000 7f1eee800640 1 Processor -- start 2024-03-28T19:20:27.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.810+0000 7f1eee800640 1 -- start start 2024-03-28T19:20:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eee800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee80a7060 0x7f1ee814c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eee800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee8152ad0 0x7f1ee80c0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eee800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee8155700 0x7f1ee81eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eee800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee8155b60 con 0x7f1ee8152ad0 2024-03-28T19:20:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eee800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee81525b0 con 0x7f1ee8155700 2024-03-28T19:20:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eee800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee8152f20 con 0x7f1ee80a7060 2024-03-28T19:20:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eeca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee8152ad0 0x7f1ee80c0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:27.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eeca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee8152ad0 0x7f1ee80c0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43016/0 (socket says 172.21.15.27:43016) 2024-03-28T19:20:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eeca00640 1 -- 172.21.15.27:0/3248799202 learned_addr learned my addr 172.21.15.27:0/3248799202 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eed400640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee80a7060 0x7f1ee814c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:27.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.811+0000 7f1eede00640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee8155700 0x7f1ee81eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1eeca00640 1 -- 172.21.15.27:0/3248799202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee80a7060 msgr2=0x7f1ee814c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1eeca00640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee80a7060 0x7f1ee814c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1eeca00640 1 -- 172.21.15.27:0/3248799202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee8155700 msgr2=0x7f1ee81eae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1eeca00640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee8155700 0x7f1ee81eae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1eeca00640 1 -- 172.21.15.27:0/3248799202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee806e5e0 con 0x7f1ee8152ad0 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1eeca00640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee8152ad0 0x7f1ee80c0ed0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400c440 tx=0x7f1ed400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1ee5600640 1 -- 172.21.15.27:0/3248799202 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed40052d0 con 0x7f1ee8152ad0 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1ee5600640 1 -- 172.21.15.27:0/3248799202 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed4005470 con 0x7f1ee8152ad0 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1ee5600640 1 -- 172.21.15.27:0/3248799202 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4007260 con 0x7f1ee8152ad0 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.812+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee81ee450 con 0x7f1ee8152ad0 2024-03-28T19:20:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.813+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee81eea10 con 0x7f1ee8152ad0 2024-03-28T19:20:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.814+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee8069640 con 0x7f1ee8152ad0 2024-03-28T19:20:27.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.816+0000 7f1ee5600640 1 -- 172.21.15.27:0/3248799202 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1ed40074c0 con 0x7f1ee8152ad0 2024-03-28T19:20:27.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.817+0000 7f1ee5600640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ec806bbd0 0x7f1ec806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:27.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.818+0000 7f1ee5600640 1 -- 172.21.15.27:0/3248799202 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f1ed408ad50 con 0x7f1ee8152ad0 2024-03-28T19:20:27.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.818+0000 7f1eed400640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ec806bbd0 0x7f1ec806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:27.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.819+0000 7f1eed400640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ec806bbd0 0x7f1ec806e090 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1edc036000 tx=0x7f1edc03b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:27.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:27.822+0000 7f1ee5600640 1 -- 172.21.15.27:0/3248799202 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1ed4058960 con 0x7f1ee8152ad0 2024-03-28T19:20:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.144+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ee80ad030 con 0x7f1ee8152ad0 2024-03-28T19:20:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.144+0000 7f1ee5600640 1 -- 172.21.15.27:0/3248799202 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f1ed4058260 con 0x7f1ee8152ad0 2024-03-28T19:20:28.144 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:28.144 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:20:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:20:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ec806bbd0 msgr2=0x7f1ec806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ec806bbd0 0x7f1ec806e090 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1edc036000 tx=0x7f1edc03b040 comp rx=0 tx=0).stop 2024-03-28T19:20:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee8152ad0 msgr2=0x7f1ee80c0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee8152ad0 0x7f1ee80c0ed0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400c440 tx=0x7f1ed400c900 comp rx=0 tx=0).stop 2024-03-28T19:20:28.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 shutdown_connections 2024-03-28T19:20:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1ec806bbd0 0x7f1ec806e090 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee8155700 0x7f1ee81eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ee8152ad0 0x7f1ee80c0ed0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 --2- 172.21.15.27:0/3248799202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee80a7060 0x7f1ee814c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 >> 172.21.15.27:0/3248799202 conn(0x7f1ee80c7aa0 msgr2=0x7f1ee80c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 shutdown_connections 2024-03-28T19:20:28.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:28.147+0000 7f1eee800640 1 -- 172.21.15.27:0/3248799202 wait complete. 2024-03-28T19:20:28.167 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:20:30.167 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.395+0000 7f7c14800640 1 Processor -- start 2024-03-28T19:20:30.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.395+0000 7f7c14800640 1 -- start start 2024-03-28T19:20:30.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.396+0000 7f7c14800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c0c0731c0 0x7f7c0c156e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:30.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.396+0000 7f7c14800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c071a00 0x7f7c0c157350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:30.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.396+0000 7f7c14800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c0c5c70 con 0x7f7c0c06c340 2024-03-28T19:20:30.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.396+0000 7f7c14800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c0c63e0 con 0x7f7c0c0731c0 2024-03-28T19:20:30.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.396+0000 7f7c14800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0c69a0 con 0x7f7c0c071a00 2024-03-28T19:20:30.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.397+0000 7f7c11400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c0c0731c0 0x7f7c0c156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:30.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.397+0000 7f7c11400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c0c0731c0 0x7f7c0c156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43030/0 (socket says 172.21.15.27:43030) 2024-03-28T19:20:30.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.397+0000 7f7c11400640 1 -- 172.21.15.27:0/3223833674 learned_addr learned my addr 172.21.15.27:0/3223833674 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:30.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.397+0000 7f7c12800640 1 --2- 172.21.15.27:0/3223833674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c071a00 0x7f7c0c157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:30.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.397+0000 7f7c11400640 1 -- 172.21.15.27:0/3223833674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c071a00 msgr2=0x7f7c0c157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:30.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.397+0000 7f7c11400640 1 --2- 172.21.15.27:0/3223833674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c071a00 0x7f7c0c157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.397+0000 7f7c11400640 1 -- 172.21.15.27:0/3223833674 >> v1:172.21.15.155:6789/0 conn(0x7f7c0c06c340 legacy=0x7f7c0c156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:30.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.397+0000 7f7c11400640 1 -- 172.21.15.27:0/3223833674 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c0c5f10 con 0x7f7c0c0731c0 2024-03-28T19:20:30.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.399+0000 7f7c11400640 1 --2- 172.21.15.27:0/3223833674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c0731c0 0x7f7c0c156e10 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009870 tx=0x7f7bfc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad26d80ceddbbfd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.399+0000 7f7c10a00640 1 -- 172.21.15.27:0/3223833674 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0317c0 con 0x7f7c0c0731c0 2024-03-28T19:20:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.399+0000 7f7c10a00640 1 -- 172.21.15.27:0/3223833674 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc037070 con 0x7f7c0c0731c0 2024-03-28T19:20:30.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.399+0000 7f7c10a00640 1 -- 172.21.15.27:0/3223833674 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc031b80 con 0x7f7c0c0731c0 2024-03-28T19:20:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.400+0000 7f7c14800640 1 -- 172.21.15.27:0/3223833674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c0731c0 msgr2=0x7f7c0c156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.400+0000 7f7c14800640 1 --2- 172.21.15.27:0/3223833674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c0731c0 0x7f7c0c156e10 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009870 tx=0x7f7bfc02b860 comp rx=0 tx=0).stop 2024-03-28T19:20:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.400+0000 7f7c14800640 1 -- 172.21.15.27:0/3223833674 shutdown_connections 2024-03-28T19:20:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.400+0000 7f7c14800640 1 --2- 172.21.15.27:0/3223833674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c071a00 0x7f7c0c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.400+0000 7f7c14800640 1 --2- 172.21.15.27:0/3223833674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c0731c0 0x7f7c0c156e10 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.400+0000 7f7c14800640 1 -- 172.21.15.27:0/3223833674 >> 172.21.15.27:0/3223833674 conn(0x7f7c0c0c4750 msgr2=0x7f7c0c0c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.400+0000 7f7c14800640 1 -- 172.21.15.27:0/3223833674 shutdown_connections 2024-03-28T19:20:30.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.400+0000 7f7c14800640 1 -- 172.21.15.27:0/3223833674 wait complete. 2024-03-28T19:20:30.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.402+0000 7f7c14800640 1 Processor -- start 2024-03-28T19:20:30.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.402+0000 7f7c14800640 1 -- start start 2024-03-28T19:20:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.403+0000 7f7c14800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c06c340 0x7f7c0c1eac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.403+0000 7f7c14800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c0c071a00 0x7f7c0c1e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:30.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.403+0000 7f7c14800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c0731c0 0x7f7c0c1ee8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.403+0000 7f7c14800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0c5700 con 0x7f7c0c06c340 2024-03-28T19:20:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.403+0000 7f7c14800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0c5580 con 0x7f7c0c071a00 2024-03-28T19:20:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.403+0000 7f7c14800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c156890 con 0x7f7c0c0731c0 2024-03-28T19:20:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.403+0000 7f7c11e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c06c340 0x7f7c0c1eac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c11e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c06c340 0x7f7c0c1eac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43038/0 (socket says 172.21.15.27:43038) 2024-03-28T19:20:30.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c11e00640 1 -- 172.21.15.27:0/1458467277 learned_addr learned my addr 172.21.15.27:0/1458467277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c11400640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c0c071a00 0x7f7c0c1e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c12800640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c0731c0 0x7f7c0c1ee8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c11e00640 1 -- 172.21.15.27:0/1458467277 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c0731c0 msgr2=0x7f7c0c1ee8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c11e00640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c0731c0 0x7f7c0c1ee8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c11e00640 1 -- 172.21.15.27:0/1458467277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c0c071a00 msgr2=0x7f7c0c1e01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c11e00640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c0c071a00 0x7f7c0c1e01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.404+0000 7f7c11e00640 1 -- 172.21.15.27:0/1458467277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c1eb440 con 0x7f7c0c06c340 2024-03-28T19:20:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.405+0000 7f7c11e00640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c06c340 0x7f7c0c1eac10 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000b3f0 tx=0x7f7c0000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.405+0000 7f7c06000640 1 -- 172.21.15.27:0/1458467277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00015020 con 0x7f7c0c06c340 2024-03-28T19:20:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.405+0000 7f7c06000640 1 -- 172.21.15.27:0/1458467277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c00011830 con 0x7f7c0c06c340 2024-03-28T19:20:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.405+0000 7f7c06000640 1 -- 172.21.15.27:0/1458467277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00011c20 con 0x7f7c0c06c340 2024-03-28T19:20:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.405+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c1eb720 con 0x7f7c0c06c340 2024-03-28T19:20:30.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.406+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c1ee3d0 con 0x7f7c0c06c340 2024-03-28T19:20:30.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.408+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be8005290 con 0x7f7c0c06c340 2024-03-28T19:20:30.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.409+0000 7f7c06000640 1 -- 172.21.15.27:0/1458467277 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7c00018020 con 0x7f7c0c06c340 2024-03-28T19:20:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.410+0000 7f7c06000640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7bf006bc50 0x7f7bf006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.410+0000 7f7c06000640 1 -- 172.21.15.27:0/1458467277 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f7c0008b4f0 con 0x7f7c0c06c340 2024-03-28T19:20:30.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.411+0000 7f7c11400640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7bf006bc50 0x7f7bf006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:30.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.412+0000 7f7c11400640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7bf006bc50 0x7f7bf006e110 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc036000 tx=0x7f7bfc03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.416+0000 7f7c06000640 1 -- 172.21.15.27:0/1458467277 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7c0001f030 con 0x7f7c0c06c340 2024-03-28T19:20:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.749+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7be8005580 con 0x7f7c0c06c340 2024-03-28T19:20:30.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.750+0000 7f7c06000640 1 -- 172.21.15.27:0/1458467277 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f7c000172c0 con 0x7f7c0c06c340 2024-03-28T19:20:30.751 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:30.751 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:20:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:20:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7bf006bc50 msgr2=0x7f7bf006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7bf006bc50 0x7f7bf006e110 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc036000 tx=0x7f7bfc03d040 comp rx=0 tx=0).stop 2024-03-28T19:20:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c06c340 msgr2=0x7f7c0c1eac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:30.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c06c340 0x7f7c0c1eac10 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000b3f0 tx=0x7f7c0000b8b0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 shutdown_connections 2024-03-28T19:20:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7bf006bc50 0x7f7bf006e110 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7c0c0731c0 0x7f7c0c1ee8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c0c071a00 0x7f7c0c1e01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 --2- 172.21.15.27:0/1458467277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c0c06c340 0x7f7c0c1eac10 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.753+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 >> 172.21.15.27:0/1458467277 conn(0x7f7c0c0c4750 msgr2=0x7f7c0c155680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.754+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 shutdown_connections 2024-03-28T19:20:30.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:30.754+0000 7f7c14800640 1 -- 172.21.15.27:0/1458467277 wait complete. 2024-03-28T19:20:30.774 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:20:32.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:33.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.015+0000 7f41dd400640 1 Processor -- start 2024-03-28T19:20:33.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.015+0000 7f41dd400640 1 -- start start 2024-03-28T19:20:33.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.016+0000 7f41dd400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f41d8150c70 0x7f41d80a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:33.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.016+0000 7f41dd400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80c3690 0x7f41d80c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:33.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.016+0000 7f41dd400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d80c3af0 con 0x7f41d80c6660 2024-03-28T19:20:33.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.016+0000 7f41dd400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80ab4e0 con 0x7f41d8150c70 2024-03-28T19:20:33.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.017+0000 7f41dd400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80ac1f0 con 0x7f41d80c3690 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.017+0000 7f41d7200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f41d80c6660 0x7f41d81534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33266/0 (socket says 172.21.15.27:33266) 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.017+0000 7f41d7200640 1 -- 172.21.15.27:0/3321439321 learned_addr learned my addr 172.21.15.27:0/3321439321 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.017+0000 7f41d6800640 1 --2- 172.21.15.27:0/3321439321 >> v2:172.21.15.155:3300/0 conn(0x7f41d8150c70 0x7f41d80a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.017+0000 7f41d5e00640 1 --2- 172.21.15.27:0/3321439321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80c3690 0x7f41d80c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.018+0000 7f41d5400640 1 -- 172.21.15.27:0/3321439321 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439494175 0 0) 0x7f41d80c3af0 con 0x7f41d80c6660 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.018+0000 7f41d5400640 1 -- 172.21.15.27:0/3321439321 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41bc003520 con 0x7f41d80c6660 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.018+0000 7f41d6800640 1 -- 172.21.15.27:0/3321439321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80c3690 msgr2=0x7f41d80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.018+0000 7f41d6800640 1 --2- 172.21.15.27:0/3321439321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80c3690 0x7f41d80c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.018+0000 7f41d6800640 1 -- 172.21.15.27:0/3321439321 >> v1:172.21.15.27:6789/0 conn(0x7f41d80c6660 legacy=0x7f41d81534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.018+0000 7f41d5400640 1 -- 172.21.15.27:0/3321439321 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1648822205 0 0) 0x7f41bc003520 con 0x7f41d80c6660 2024-03-28T19:20:33.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.018+0000 7f41d6800640 1 -- 172.21.15.27:0/3321439321 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80c3d90 con 0x7f41d8150c70 2024-03-28T19:20:33.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.020+0000 7f41d6800640 1 --2- 172.21.15.27:0/3321439321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8150c70 0x7f41d80a9cb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f41c80097a0 tx=0x7f41c802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6b4beae4dd63923 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:33.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.020+0000 7f41d5400640 1 -- 172.21.15.27:0/3321439321 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8037070 con 0x7f41d8150c70 2024-03-28T19:20:33.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.020+0000 7f41d5400640 1 -- 172.21.15.27:0/3321439321 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c8030510 con 0x7f41d8150c70 2024-03-28T19:20:33.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.020+0000 7f41d5400640 1 -- 172.21.15.27:0/3321439321 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d00019f0 con 0x7f41d8150c70 2024-03-28T19:20:33.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.021+0000 7f41dd400640 1 -- 172.21.15.27:0/3321439321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8150c70 msgr2=0x7f41d80a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:33.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.021+0000 7f41dd400640 1 --2- 172.21.15.27:0/3321439321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8150c70 0x7f41d80a9cb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f41c80097a0 tx=0x7f41c802b710 comp rx=0 tx=0).stop 2024-03-28T19:20:33.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.021+0000 7f41dd400640 1 -- 172.21.15.27:0/3321439321 shutdown_connections 2024-03-28T19:20:33.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.021+0000 7f41dd400640 1 --2- 172.21.15.27:0/3321439321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80c3690 0x7f41d80c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.021+0000 7f41dd400640 1 --2- 172.21.15.27:0/3321439321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8150c70 0x7f41d80a9cb0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.021+0000 7f41dd400640 1 -- 172.21.15.27:0/3321439321 >> 172.21.15.27:0/3321439321 conn(0x7f41d806b9d0 msgr2=0x7f41d80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:33.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.021+0000 7f41dd400640 1 -- 172.21.15.27:0/3321439321 shutdown_connections 2024-03-28T19:20:33.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.021+0000 7f41dd400640 1 -- 172.21.15.27:0/3321439321 wait complete. 2024-03-28T19:20:33.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.023+0000 7f41dd400640 1 Processor -- start 2024-03-28T19:20:33.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.023+0000 7f41dd400640 1 -- start start 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41dd400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d80c3690 0x7f41d81eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41dd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41d80c6660 0x7f41d814a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41dd400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d8150c70 0x7f41d814aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41dd400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80abcd0 con 0x7f41d80c6660 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41dd400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80ab9f0 con 0x7f41d80c3690 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41dd400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80c1810 con 0x7f41d8150c70 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41d5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41d80c6660 0x7f41d814a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41d5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41d80c6660 0x7f41d814a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43040/0 (socket says 172.21.15.27:43040) 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.024+0000 7f41d5e00640 1 -- 172.21.15.27:0/2160841299 learned_addr learned my addr 172.21.15.27:0/2160841299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.025+0000 7f41d6800640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d80c3690 0x7f41d81eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.025+0000 7f41d7200640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d8150c70 0x7f41d814aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.025+0000 7f41d5e00640 1 -- 172.21.15.27:0/2160841299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d8150c70 msgr2=0x7f41d814aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.025+0000 7f41d5e00640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d8150c70 0x7f41d814aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.025+0000 7f41d5e00640 1 -- 172.21.15.27:0/2160841299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d80c3690 msgr2=0x7f41d81eb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.025+0000 7f41d5e00640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d80c3690 0x7f41d81eb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.025+0000 7f41d5e00640 1 -- 172.21.15.27:0/2160841299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d814e8e0 con 0x7f41d80c6660 2024-03-28T19:20:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.025+0000 7f41d5e00640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41d80c6660 0x7f41d814a9a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f41c400e9b0 tx=0x7f41c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.026+0000 7f41cea00640 1 -- 172.21.15.27:0/2160841299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4017070 con 0x7f41d80c6660 2024-03-28T19:20:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.026+0000 7f41cea00640 1 -- 172.21.15.27:0/2160841299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c400c9a0 con 0x7f41d80c6660 2024-03-28T19:20:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.026+0000 7f41cea00640 1 -- 172.21.15.27:0/2160841299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c400cd90 con 0x7f41d80c6660 2024-03-28T19:20:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.026+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d814ebc0 con 0x7f41d80c6660 2024-03-28T19:20:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.026+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d814dc20 con 0x7f41d80c6660 2024-03-28T19:20:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.027+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d804f400 con 0x7f41d80c6660 2024-03-28T19:20:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.028+0000 7f41cea00640 1 -- 172.21.15.27:0/2160841299 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f41c401a070 con 0x7f41d80c6660 2024-03-28T19:20:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.029+0000 7f41cea00640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41b806bc00 0x7f41b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.029+0000 7f41cea00640 1 -- 172.21.15.27:0/2160841299 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f41c408bca0 con 0x7f41d80c6660 2024-03-28T19:20:33.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.030+0000 7f41d6800640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41b806bc00 0x7f41b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.031+0000 7f41d6800640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41b806bc00 0x7f41b806e0c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f41c8034000 tx=0x7f41c8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.036+0000 7f41cea00640 1 -- 172.21.15.27:0/2160841299 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f41c40598b0 con 0x7f41d80c6660 2024-03-28T19:20:33.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.367+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41d814ed90 con 0x7f41d80c6660 2024-03-28T19:20:33.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.367+0000 7f41cea00640 1 -- 172.21.15.27:0/2160841299 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f41c40591b0 con 0x7f41d80c6660 2024-03-28T19:20:33.368 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:33.368 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:20:33.368 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41b806bc00 msgr2=0x7f41b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41b806bc00 0x7f41b806e0c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f41c8034000 tx=0x7f41c8008030 comp rx=0 tx=0).stop 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41d80c6660 msgr2=0x7f41d814a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41d80c6660 0x7f41d814a9a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f41c400e9b0 tx=0x7f41c400ee70 comp rx=0 tx=0).stop 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 shutdown_connections 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f41b806bc00 0x7f41b806e0c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d8150c70 0x7f41d814aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41d80c6660 0x7f41d814a9a0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 --2- 172.21.15.27:0/2160841299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d80c3690 0x7f41d81eb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.370+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 >> 172.21.15.27:0/2160841299 conn(0x7f41d806b9d0 msgr2=0x7f41d80c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.371+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 shutdown_connections 2024-03-28T19:20:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:33.371+0000 7f41dd400640 1 -- 172.21.15.27:0/2160841299 wait complete. 2024-03-28T19:20:33.392 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:20:34.929 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 20% (51/247) Updating files: 21% (52/247) Updating files: 22% (55/247) Updating files: 23% (57/247) Updating files: 24% (60/247) Updating files: 25% (62/247) Updating files: 26% (65/247) Updating files: 27% (67/247) Updating files: 28% (70/247) Updating files: 29% (72/247) Updating files: 30% (75/247) Updating files: 31% (77/247) Updating files: 32% (80/247) Updating files: 33% (82/247) Updating files: 34% (84/247) Updating files: 35% (87/247) Updating files: 36% (89/247) Updating files: 37% (92/247) Updating files: 38% (94/247) Updating files: 39% (97/247) Updating files: 40% (99/247) Updating files: 41% (102/247) Updating files: 42% (104/247) Updating files: 43% (107/247) Updating files: 44% (109/247) Updating files: 45% (112/247) Updating files: 46% (114/247) Updating files: 47% (117/247) Updating files: 48% (119/247) Updating files: 49% (122/247) Updating files: 50% (124/247) Updating files: 51% (126/247) Updating files: 52% (129/247) Updating files: 53% (131/247) Updating files: 54% (134/247) Updating files: 55% (136/247) Updating files: 56% (139/247) Updating files: 57% (141/247) Updating files: 58% (144/247) Updating files: 59% (146/247) Updating files: 60% (149/247) Updating files: 61% (151/247) Updating files: 62% (154/247) Updating files: 63% (156/247) Updating files: 64% (159/247) Updating files: 65% (161/247) Updating files: 66% (164/247) Updating files: 67% (166/247) Updating files: 68% (168/247) Updating files: 69% (171/247) Updating files: 70% (173/247) Updating files: 71% (176/247) Updating files: 71% (177/247) Updating files: 72% (178/247) Updating files: 73% (181/247) Updating files: 74% (183/247) Updating files: 75% (186/247) Updating files: 76% (188/247) Updating files: 77% (191/247) Updating files: 78% (193/247) Updating files: 79% (196/247) Updating files: 80% (198/247) Updating files: 81% (201/247) Updating files: 82% (203/247) Updating files: 83% (206/247) Updating files: 84% (208/247) Updating files: 85% (210/247) Updating files: 86% (213/247) Updating files: 87% (215/247) Updating files: 88% (218/247) Updating files: 89% (220/247) Updating files: 90% (223/247) Updating files: 91% (225/247) Updating files: 92% (228/247) Updating files: 93% (230/247) Updating files: 94% (233/247) Updating files: 95% (235/247) Updating files: 96% (238/247) Updating files: 97% (240/247) Updating files: 98% (243/247) Updating files: 99% (245/247) Updating files: 100% (247/247) Updating files: 100% (247/247), done. 2024-03-28T19:20:35.189 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2024-03-28T19:20:35.198 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.4 2024-03-28T19:20:35.273 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:20:35.273 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 5 -eq 48 ']' 2024-03-28T19:20:35.274 INFO:tasks.workunit.client.0.smithi027.stdout:v0.5 2024-03-28T19:20:35.274 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.5 2024-03-28T19:20:35.274 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.5 2024-03-28T19:20:35.274 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.5 2024-03-28T19:20:35.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:35.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.619+0000 7f9df5200640 1 Processor -- start 2024-03-28T19:20:35.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.619+0000 7f9df5200640 1 -- start start 2024-03-28T19:20:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.620+0000 7f9df5200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df0153ed0 0x7f9df006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.620+0000 7f9df5200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9df006eb80 0x7f9df015a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.620+0000 7f9df5200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df006efb0 con 0x7f9df006e6f0 2024-03-28T19:20:35.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.620+0000 7f9df5200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df00c6990 con 0x7f9df006eb80 2024-03-28T19:20:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.620+0000 7f9df5200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00c3830 con 0x7f9df0153ed0 2024-03-28T19:20:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.621+0000 7f9dede00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df0153ed0 0x7f9df006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.621+0000 7f9dee800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9df006eb80 0x7f9df015a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.621+0000 7f9dee800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9df006eb80 0x7f9df015a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:35350/0 (socket says 172.21.15.27:35350) 2024-03-28T19:20:35.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.621+0000 7f9dee800640 1 -- 172.21.15.27:0/1030367538 learned_addr learned my addr 172.21.15.27:0/1030367538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.622+0000 7f9dee800640 1 -- 172.21.15.27:0/1030367538 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df0153ed0 msgr2=0x7f9df006ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.622+0000 7f9dee800640 1 --2- 172.21.15.27:0/1030367538 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df0153ed0 0x7f9df006ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.622+0000 7f9dee800640 1 -- 172.21.15.27:0/1030367538 >> v1:172.21.15.155:6789/0 conn(0x7f9df006e6f0 legacy=0x7f9df0159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:35.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.622+0000 7f9dee800640 1 -- 172.21.15.27:0/1030367538 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df006d1f0 con 0x7f9df006eb80 2024-03-28T19:20:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.624+0000 7f9dee800640 1 --2- 172.21.15.27:0/1030367538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df006eb80 0x7f9df015a2d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009e50 tx=0x7f9de802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=698c016b2733de70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:35.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9deca00640 1 -- 172.21.15.27:0/1030367538 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de802cb00 con 0x7f9df006eb80 2024-03-28T19:20:35.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9deca00640 1 -- 172.21.15.27:0/1030367538 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de802cca0 con 0x7f9df006eb80 2024-03-28T19:20:35.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9deca00640 1 -- 172.21.15.27:0/1030367538 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8034610 con 0x7f9df006eb80 2024-03-28T19:20:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9df5200640 1 -- 172.21.15.27:0/1030367538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df006eb80 msgr2=0x7f9df015a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9df5200640 1 --2- 172.21.15.27:0/1030367538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df006eb80 0x7f9df015a2d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009e50 tx=0x7f9de802bf60 comp rx=0 tx=0).stop 2024-03-28T19:20:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9df5200640 1 -- 172.21.15.27:0/1030367538 shutdown_connections 2024-03-28T19:20:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9df5200640 1 --2- 172.21.15.27:0/1030367538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df006eb80 0x7f9df015a2d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9df5200640 1 --2- 172.21.15.27:0/1030367538 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df0153ed0 0x7f9df006ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.625+0000 7f9df5200640 1 -- 172.21.15.27:0/1030367538 >> 172.21.15.27:0/1030367538 conn(0x7f9df006b9d0 msgr2=0x7f9df00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.626+0000 7f9df5200640 1 -- 172.21.15.27:0/1030367538 shutdown_connections 2024-03-28T19:20:35.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.626+0000 7f9df5200640 1 -- 172.21.15.27:0/1030367538 wait complete. 2024-03-28T19:20:35.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.628+0000 7f9df5200640 1 Processor -- start 2024-03-28T19:20:35.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.628+0000 7f9df5200640 1 -- start start 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.629+0000 7f9df5200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df006e6f0 0x7f9df00a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.629+0000 7f9df5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df006eb80 0x7f9df00a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.629+0000 7f9df5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df0153ed0 0x7f9df00a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.629+0000 7f9df5200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00c3db0 con 0x7f9df006eb80 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.629+0000 7f9df5200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00c3ad0 con 0x7f9df0153ed0 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.629+0000 7f9df5200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df015a810 con 0x7f9df006e6f0 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.629+0000 7f9ded400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df006eb80 0x7f9df00a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9ded400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df006eb80 0x7f9df00a6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43056/0 (socket says 172.21.15.27:43056) 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9ded400640 1 -- 172.21.15.27:0/3665591285 learned_addr learned my addr 172.21.15.27:0/3665591285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:35.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9dede00640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df006e6f0 0x7f9df00a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:35.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9dee800640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df0153ed0 0x7f9df00a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9ded400640 1 -- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df006e6f0 msgr2=0x7f9df00a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9ded400640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df006e6f0 0x7f9df00a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9ded400640 1 -- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df0153ed0 msgr2=0x7f9df00a6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9ded400640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df0153ed0 0x7f9df00a6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9ded400640 1 -- 172.21.15.27:0/3665591285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df01f0050 con 0x7f9df006eb80 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.630+0000 7f9dee800640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df0153ed0 0x7f9df00a6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.631+0000 7f9ded400640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df006eb80 0x7f9df00a6820 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00c400 tx=0x7f9ddc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.631+0000 7f9de6000640 1 -- 172.21.15.27:0/3665591285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc0052d0 con 0x7f9df006eb80 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.631+0000 7f9de6000640 1 -- 172.21.15.27:0/3665591285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ddc005490 con 0x7f9df006eb80 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.631+0000 7f9de6000640 1 -- 172.21.15.27:0/3665591285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc007260 con 0x7f9df006eb80 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.631+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df015d520 con 0x7f9df006eb80 2024-03-28T19:20:35.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.631+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df01f0310 con 0x7f9df006eb80 2024-03-28T19:20:35.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.633+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df0072a60 con 0x7f9df006eb80 2024-03-28T19:20:35.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.634+0000 7f9de6000640 1 -- 172.21.15.27:0/3665591285 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ddc007400 con 0x7f9df006eb80 2024-03-28T19:20:35.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.635+0000 7f9de6000640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9dd006bbd0 0x7f9dd006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:35.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.635+0000 7f9de6000640 1 -- 172.21.15.27:0/3665591285 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f9ddc08ae90 con 0x7f9df006eb80 2024-03-28T19:20:35.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.635+0000 7f9dede00640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9dd006bbd0 0x7f9dd006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:35.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.636+0000 7f9dede00640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9dd006bbd0 0x7f9dd006e090 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009aa0 tx=0x7f9de0009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:35.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.640+0000 7f9de6000640 1 -- 172.21.15.27:0/3665591285 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9df0072a60 con 0x7f9df006eb80 2024-03-28T19:20:35.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.906+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df00aa390 con 0x7f9df006eb80 2024-03-28T19:20:35.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.907+0000 7f9de6000640 1 -- 172.21.15.27:0/3665591285 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f9df00aa390 con 0x7f9df006eb80 2024-03-28T19:20:35.908 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:35.908 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:20:35.908 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:20:35.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9dd006bbd0 msgr2=0x7f9dd006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:35.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9dd006bbd0 0x7f9dd006e090 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009aa0 tx=0x7f9de0009190 comp rx=0 tx=0).stop 2024-03-28T19:20:35.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df006eb80 msgr2=0x7f9df00a6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:35.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df006eb80 0x7f9df00a6820 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00c400 tx=0x7f9ddc00c8c0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 shutdown_connections 2024-03-28T19:20:35.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9dd006bbd0 0x7f9dd006e090 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df0153ed0 0x7f9df00a6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9df006eb80 0x7f9df00a6820 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 --2- 172.21.15.27:0/3665591285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9df006e6f0 0x7f9df00a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:35.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 >> 172.21.15.27:0/3665591285 conn(0x7f9df006b9d0 msgr2=0x7f9df0151910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:35.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 shutdown_connections 2024-03-28T19:20:35.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:35.910+0000 7f9df5200640 1 -- 172.21.15.27:0/3665591285 wait complete. 2024-03-28T19:20:35.932 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:20:37.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:38.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.168+0000 7f1019800640 1 Processor -- start 2024-03-28T19:20:38.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.168+0000 7f1019800640 1 -- start start 2024-03-28T19:20:38.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.169+0000 7f1019800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10140727e0 0x7f10140c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:38.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.169+0000 7f1019800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1014155fe0 con 0x7f1014155600 2024-03-28T19:20:38.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.169+0000 7f1019800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10141562c0 con 0x7f10140c59d0 2024-03-28T19:20:38.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.169+0000 7f1019800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10141565a0 con 0x7f10140727e0 2024-03-28T19:20:38.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.170+0000 7f1013400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10140727e0 0x7f10140c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:38.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.170+0000 7f1018e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1014155600 0x7f101414f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50010/0 (socket says 172.21.15.27:50010) 2024-03-28T19:20:38.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.170+0000 7f1018e00640 1 -- 172.21.15.27:0/2151166768 learned_addr learned my addr 172.21.15.27:0/2151166768 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:38.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.170+0000 7f1013400640 1 -- 172.21.15.27:0/2151166768 >> v1:172.21.15.155:6789/0 conn(0x7f10140c59d0 legacy=0x7f1014159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:38.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.170+0000 7f1013400640 1 -- 172.21.15.27:0/2151166768 >> v1:172.21.15.27:6789/0 conn(0x7f1014155600 legacy=0x7f101414f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:38.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.170+0000 7f1013400640 1 -- 172.21.15.27:0/2151166768 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1014158d20 con 0x7f10140727e0 2024-03-28T19:20:38.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.170+0000 7f1012a00640 1 -- 172.21.15.27:0/2151166768 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954409194 0 0) 0x7f1014155fe0 con 0x7f1014155600 2024-03-28T19:20:38.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.172+0000 7f1013400640 1 --2- 172.21.15.27:0/2151166768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140727e0 0x7f10140c8f00 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f10000097a0 tx=0x7f100002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110ece9101fc3c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.172+0000 7f1012a00640 1 -- 172.21.15.27:0/2151166768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10000315a0 con 0x7f10140727e0 2024-03-28T19:20:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.172+0000 7f1012a00640 1 -- 172.21.15.27:0/2151166768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1000037070 con 0x7f10140727e0 2024-03-28T19:20:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.172+0000 7f1012a00640 1 -- 172.21.15.27:0/2151166768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000031960 con 0x7f10140727e0 2024-03-28T19:20:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.173+0000 7f1019800640 1 -- 172.21.15.27:0/2151166768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140727e0 msgr2=0x7f10140c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.173+0000 7f1019800640 1 --2- 172.21.15.27:0/2151166768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140727e0 0x7f10140c8f00 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f10000097a0 tx=0x7f100002b840 comp rx=0 tx=0).stop 2024-03-28T19:20:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.173+0000 7f1019800640 1 -- 172.21.15.27:0/2151166768 shutdown_connections 2024-03-28T19:20:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.173+0000 7f1019800640 1 --2- 172.21.15.27:0/2151166768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140727e0 0x7f10140c8f00 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.173+0000 7f1019800640 1 -- 172.21.15.27:0/2151166768 >> 172.21.15.27:0/2151166768 conn(0x7f10140ac910 msgr2=0x7f10140accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.173+0000 7f1019800640 1 -- 172.21.15.27:0/2151166768 shutdown_connections 2024-03-28T19:20:38.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.173+0000 7f1019800640 1 -- 172.21.15.27:0/2151166768 wait complete. 2024-03-28T19:20:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.175+0000 7f1019800640 1 Processor -- start 2024-03-28T19:20:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.175+0000 7f1019800640 1 -- start start 2024-03-28T19:20:38.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.176+0000 7f1019800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10140727e0 0x7f10140beb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.176+0000 7f1019800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c59d0 0x7f101414bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.176+0000 7f1019800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1014155600 0x7f101414c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.176+0000 7f1019800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10140c5db0 con 0x7f10140c59d0 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.176+0000 7f1019800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1014159090 con 0x7f10140727e0 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.176+0000 7f1019800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140c5f30 con 0x7f1014155600 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c59d0 0x7f101414bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c59d0 0x7f101414bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36574/0 (socket says 172.21.15.27:36574) 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013400640 1 -- 172.21.15.27:0/3212224213 learned_addr learned my addr 172.21.15.27:0/3212224213 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1018e00640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1014155600 0x7f101414c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:38.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013e00640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10140727e0 0x7f10140beb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013400640 1 -- 172.21.15.27:0/3212224213 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1014155600 msgr2=0x7f101414c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013400640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1014155600 0x7f101414c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013400640 1 -- 172.21.15.27:0/3212224213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10140727e0 msgr2=0x7f10140beb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013400640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10140727e0 0x7f10140beb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.177+0000 7f1013400640 1 -- 172.21.15.27:0/3212224213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10141eac40 con 0x7f10140c59d0 2024-03-28T19:20:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.178+0000 7f1013400640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c59d0 0x7f101414bf70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1000034040 tx=0x7f10000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.178+0000 7f100be00640 1 -- 172.21.15.27:0/3212224213 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000004230 con 0x7f10140c59d0 2024-03-28T19:20:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.178+0000 7f100be00640 1 -- 172.21.15.27:0/3212224213 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10000043d0 con 0x7f10140c59d0 2024-03-28T19:20:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.178+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10140c2100 con 0x7f10140c59d0 2024-03-28T19:20:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.178+0000 7f100be00640 1 -- 172.21.15.27:0/3212224213 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1000032880 con 0x7f10140c59d0 2024-03-28T19:20:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.179+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10140c26c0 con 0x7f10140c59d0 2024-03-28T19:20:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.180+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1014067dd0 con 0x7f10140c59d0 2024-03-28T19:20:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.181+0000 7f100be00640 1 -- 172.21.15.27:0/3212224213 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1000032a20 con 0x7f10140c59d0 2024-03-28T19:20:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.182+0000 7f100be00640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ff406bc20 0x7f0ff406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.182+0000 7f100be00640 1 -- 172.21.15.27:0/3212224213 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f100002f070 con 0x7f10140c59d0 2024-03-28T19:20:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.183+0000 7f1013e00640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ff406bc20 0x7f0ff406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.183+0000 7f1013e00640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ff406bc20 0x7f0ff406e0e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1004000f60 tx=0x7f1004009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.187+0000 7f100be00640 1 -- 172.21.15.27:0/3212224213 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1000041070 con 0x7f10140c59d0 2024-03-28T19:20:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.524+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1014069640 con 0x7f10140c59d0 2024-03-28T19:20:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.525+0000 7f100be00640 1 -- 172.21.15.27:0/3212224213 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f100007b5b0 con 0x7f10140c59d0 2024-03-28T19:20:38.524 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:38.524 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186},{"gid":8052,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:18:53.615314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6260},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6260":{"gid":6260,"name":"d","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.155:6834/1296746732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1296746732},{"type":"v1","addr":"172.21.15.155:6834","nonce":1296746732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6260,"qdb_cluster":[6260]},"id":2}]} 2024-03-28T19:20:38.525 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 187 2024-03-28T19:20:38.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.528+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ff406bc20 msgr2=0x7f0ff406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:38.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ff406bc20 0x7f0ff406e0e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1004000f60 tx=0x7f1004009040 comp rx=0 tx=0).stop 2024-03-28T19:20:38.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c59d0 msgr2=0x7f101414bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:38.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c59d0 0x7f101414bf70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1000034040 tx=0x7f10000046c0 comp rx=0 tx=0).stop 2024-03-28T19:20:38.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 shutdown_connections 2024-03-28T19:20:38.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ff406bc20 0x7f0ff406e0e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:38.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1014155600 0x7f101414c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c59d0 0x7f101414bf70 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 --2- 172.21.15.27:0/3212224213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10140727e0 0x7f10140beb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 >> 172.21.15.27:0/3212224213 conn(0x7f10140ac910 msgr2=0x7f10140c77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 shutdown_connections 2024-03-28T19:20:38.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:38.529+0000 7f1019800640 1 -- 172.21.15.27:0/3212224213 wait complete. 2024-03-28T19:20:38.554 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:20:39.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:20:39.269 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:20:40.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:40.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.874+0000 7f653fe00640 1 Processor -- start 2024-03-28T19:20:40.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.874+0000 7f653fe00640 1 -- start start 2024-03-28T19:20:40.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.876+0000 7f653fe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f65380c5e80 0x7f65380c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:40.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.876+0000 7f653fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f65380aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:40.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.876+0000 7f653fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f653806d8a0 con 0x7f65380c1f50 2024-03-28T19:20:40.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.876+0000 7f653fe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653806d650 con 0x7f65380c5e80 2024-03-28T19:20:40.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.876+0000 7f653fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653806e1d0 con 0x7f65380abf60 2024-03-28T19:20:40.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.876+0000 7f653de00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f65380aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.876+0000 7f653d400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f65380c5e80 0x7f65380c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.877+0000 7f653de00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f65380aa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:33118/0 (socket says 172.21.15.27:33118) 2024-03-28T19:20:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.877+0000 7f653de00640 1 -- 172.21.15.27:0/3448114060 learned_addr learned my addr 172.21.15.27:0/3448114060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.877+0000 7f653d400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f65380c5e80 0x7f65380c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:39008/0 (socket says 172.21.15.27:39008) 2024-03-28T19:20:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.877+0000 7f653de00640 1 -- 172.21.15.27:0/3448114060 >> v2:172.21.15.155:3300/0 conn(0x7f65380c5e80 msgr2=0x7f65380c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.877+0000 7f653de00640 1 --2- 172.21.15.27:0/3448114060 >> v2:172.21.15.155:3300/0 conn(0x7f65380c5e80 0x7f65380c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.877+0000 7f653de00640 1 -- 172.21.15.27:0/3448114060 >> v1:172.21.15.155:6789/0 conn(0x7f65380c1f50 legacy=0x7f65380ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:40.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.877+0000 7f653de00640 1 -- 172.21.15.27:0/3448114060 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653806db80 con 0x7f65380abf60 2024-03-28T19:20:40.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.880+0000 7f653de00640 1 --2- 172.21.15.27:0/3448114060 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f65380aa570 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6530009e50 tx=0x7f653002be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80b4e042c30d2a49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:40.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.880+0000 7f6537e00640 1 -- 172.21.15.27:0/3448114060 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653002c9f0 con 0x7f65380abf60 2024-03-28T19:20:40.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.880+0000 7f6537e00640 1 -- 172.21.15.27:0/3448114060 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653002cb90 con 0x7f65380abf60 2024-03-28T19:20:40.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.880+0000 7f6537e00640 1 -- 172.21.15.27:0/3448114060 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65300343f0 con 0x7f65380abf60 2024-03-28T19:20:40.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.881+0000 7f653fe00640 1 -- 172.21.15.27:0/3448114060 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 msgr2=0x7f65380aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.881+0000 7f653fe00640 1 --2- 172.21.15.27:0/3448114060 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f65380aa570 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6530009e50 tx=0x7f653002be50 comp rx=0 tx=0).stop 2024-03-28T19:20:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.881+0000 7f653fe00640 1 -- 172.21.15.27:0/3448114060 shutdown_connections 2024-03-28T19:20:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.881+0000 7f653fe00640 1 --2- 172.21.15.27:0/3448114060 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f65380aa570 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.881+0000 7f653fe00640 1 --2- 172.21.15.27:0/3448114060 >> v2:172.21.15.155:3300/0 conn(0x7f65380c5e80 0x7f65380c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.881+0000 7f653fe00640 1 -- 172.21.15.27:0/3448114060 >> 172.21.15.27:0/3448114060 conn(0x7f653806b9d0 msgr2=0x7f65380c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.882+0000 7f653fe00640 1 -- 172.21.15.27:0/3448114060 shutdown_connections 2024-03-28T19:20:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.882+0000 7f653fe00640 1 -- 172.21.15.27:0/3448114060 wait complete. 2024-03-28T19:20:40.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.884+0000 7f653fe00640 1 Processor -- start 2024-03-28T19:20:40.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.884+0000 7f653fe00640 1 -- start start 2024-03-28T19:20:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.885+0000 7f653fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f653813fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.885+0000 7f653fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65380c1f50 0x7f65380a8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.885+0000 7f653fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65380c5e80 0x7f65380a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.885+0000 7f653fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65380c3370 con 0x7f65380c1f50 2024-03-28T19:20:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.885+0000 7f653fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65380c31f0 con 0x7f65380c5e80 2024-03-28T19:20:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.885+0000 7f653fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65380c6260 con 0x7f65380abf60 2024-03-28T19:20:40.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.886+0000 7f653ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65380c1f50 0x7f65380a8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.886+0000 7f653ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65380c1f50 0x7f65380a8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36578/0 (socket says 172.21.15.27:36578) 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.886+0000 7f653ca00640 1 -- 172.21.15.27:0/2109445017 learned_addr learned my addr 172.21.15.27:0/2109445017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.886+0000 7f653de00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65380c5e80 0x7f65380a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.886+0000 7f653d400640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f653813fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.887+0000 7f653ca00640 1 -- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 msgr2=0x7f653813fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.887+0000 7f653ca00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f653813fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.887+0000 7f653ca00640 1 -- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65380c5e80 msgr2=0x7f65380a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.887+0000 7f653ca00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65380c5e80 0x7f65380a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.887+0000 7f653ca00640 1 -- 172.21.15.27:0/2109445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653814ddd0 con 0x7f65380c1f50 2024-03-28T19:20:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.887+0000 7f653d400640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f653813fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:20:40.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.887+0000 7f653ca00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65380c1f50 0x7f65380a8100 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f652400c470 tx=0x7f652400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:40.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.887+0000 7f6535600640 1 -- 172.21.15.27:0/2109445017 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65240052d0 con 0x7f65380c1f50 2024-03-28T19:20:40.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.888+0000 7f6535600640 1 -- 172.21.15.27:0/2109445017 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6524005490 con 0x7f65380c1f50 2024-03-28T19:20:40.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.888+0000 7f6535600640 1 -- 172.21.15.27:0/2109445017 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524007260 con 0x7f65380c1f50 2024-03-28T19:20:40.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.888+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653815f330 con 0x7f65380c1f50 2024-03-28T19:20:40.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.888+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653814c0f0 con 0x7f65380c1f50 2024-03-28T19:20:40.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.890+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653806ed60 con 0x7f65380c1f50 2024-03-28T19:20:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.892+0000 7f6535600640 1 -- 172.21.15.27:0/2109445017 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6524007400 con 0x7f65380c1f50 2024-03-28T19:20:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.893+0000 7f6535600640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f651806bc20 0x7f651806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.893+0000 7f6535600640 1 -- 172.21.15.27:0/2109445017 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f652408adb0 con 0x7f65380c1f50 2024-03-28T19:20:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.894+0000 7f653d400640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f651806bc20 0x7f651806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.895+0000 7f653d400640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f651806bc20 0x7f651806e0e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f652c009a20 tx=0x7f652c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:40.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:40.903+0000 7f6535600640 1 -- 172.21.15.27:0/2109445017 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f652408f070 con 0x7f65380c1f50 2024-03-28T19:20:41.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.301+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f653804f400 con 0x7f65380c1f50 2024-03-28T19:20:41.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.302+0000 7f6535600640 1 -- 172.21.15.27:0/2109445017 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f65240589f0 con 0x7f65380c1f50 2024-03-28T19:20:41.301 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:41.302 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:40.288993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:20:41.302 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 188 2024-03-28T19:20:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f651806bc20 msgr2=0x7f651806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f651806bc20 0x7f651806e0e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f652c009a20 tx=0x7f652c008040 comp rx=0 tx=0).stop 2024-03-28T19:20:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65380c1f50 msgr2=0x7f65380a8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65380c1f50 0x7f65380a8100 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f652400c470 tx=0x7f652400c930 comp rx=0 tx=0).stop 2024-03-28T19:20:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 shutdown_connections 2024-03-28T19:20:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f651806bc20 0x7f651806e0e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65380c5e80 0x7f65380a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65380c1f50 0x7f65380a8100 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 --2- 172.21.15.27:0/2109445017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65380abf60 0x7f653813fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 >> 172.21.15.27:0/2109445017 conn(0x7f653806b9d0 msgr2=0x7f6538070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.306+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 shutdown_connections 2024-03-28T19:20:41.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.307+0000 7f653fe00640 1 -- 172.21.15.27:0/2109445017 wait complete. 2024-03-28T19:20:41.328 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-03-28T19:20:41.329 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:20:41.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:41.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.636+0000 7f8ca8a00640 1 Processor -- start 2024-03-28T19:20:41.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.636+0000 7f8ca8a00640 1 -- start start 2024-03-28T19:20:41.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.637+0000 7f8ca8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 0x7f8ca40c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:41.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.637+0000 7f8ca8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ca40c7720 0x7f8ca40c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:41.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.637+0000 7f8ca8a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca40a5ac0 con 0x7f8ca40734f0 2024-03-28T19:20:41.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.637+0000 7f8ca8a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca40c3450 con 0x7f8ca40c7720 2024-03-28T19:20:41.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.637+0000 7f8ca8a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca4073060 con 0x7f8ca40aa7c0 2024-03-28T19:20:41.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.638+0000 7f8ca2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ca40c7720 0x7f8ca40c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:41.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.638+0000 7f8ca1400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8ca40734f0 0x7f8ca40c4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50026/0 (socket says 172.21.15.27:50026) 2024-03-28T19:20:41.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.638+0000 7f8ca1400640 1 -- 172.21.15.27:0/1400415969 learned_addr learned my addr 172.21.15.27:0/1400415969 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:41.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.638+0000 7f8ca1e00640 1 --2- 172.21.15.27:0/1400415969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 0x7f8ca40c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:41.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.638+0000 7f8ca2800640 1 -- 172.21.15.27:0/1400415969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 msgr2=0x7f8ca40c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:41.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.638+0000 7f8ca2800640 1 --2- 172.21.15.27:0/1400415969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 0x7f8ca40c3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.638+0000 7f8ca2800640 1 -- 172.21.15.27:0/1400415969 >> v1:172.21.15.27:6789/0 conn(0x7f8ca40734f0 legacy=0x7f8ca40c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:41.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.638+0000 7f8ca2800640 1 -- 172.21.15.27:0/1400415969 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40a5d30 con 0x7f8ca40c7720 2024-03-28T19:20:41.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.640+0000 7f8ca2800640 1 --2- 172.21.15.27:0/1400415969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40c7720 0x7f8ca40c5210 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009d90 tx=0x7f8c9c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c1d8e3fa08184 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:41.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.641+0000 7f8ca0a00640 1 -- 172.21.15.27:0/1400415969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c039070 con 0x7f8ca40c7720 2024-03-28T19:20:41.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.641+0000 7f8ca0a00640 1 -- 172.21.15.27:0/1400415969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c02cbc0 con 0x7f8ca40c7720 2024-03-28T19:20:41.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.641+0000 7f8ca0a00640 1 -- 172.21.15.27:0/1400415969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c034440 con 0x7f8ca40c7720 2024-03-28T19:20:41.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.643+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1400415969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40c7720 msgr2=0x7f8ca40c5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:41.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.643+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1400415969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40c7720 0x7f8ca40c5210 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009d90 tx=0x7f8c9c02be10 comp rx=0 tx=0).stop 2024-03-28T19:20:41.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.643+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1400415969 shutdown_connections 2024-03-28T19:20:41.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.643+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1400415969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40c7720 0x7f8ca40c5210 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.643+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1400415969 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 0x7f8ca40c3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.643+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1400415969 >> 172.21.15.27:0/1400415969 conn(0x7f8ca40c6c00 msgr2=0x7f8ca40c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:41.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.644+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1400415969 shutdown_connections 2024-03-28T19:20:41.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.644+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1400415969 wait complete. 2024-03-28T19:20:41.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.647+0000 7f8ca8a00640 1 Processor -- start 2024-03-28T19:20:41.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.647+0000 7f8ca8a00640 1 -- start start 2024-03-28T19:20:41.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.649+0000 7f8ca8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40734f0 0x7f8ca40a9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:41.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.649+0000 7f8ca8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 0x7f8ca41f8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:41.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.649+0000 7f8ca8a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ca40c7720 0x7f8ca41ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:41.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.649+0000 7f8ca8a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40c4360 con 0x7f8ca40734f0 2024-03-28T19:20:41.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.649+0000 7f8ca8a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40c3170 con 0x7f8ca40c7720 2024-03-28T19:20:41.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.649+0000 7f8ca8a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40ac0f0 con 0x7f8ca40aa7c0 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.649+0000 7f8ca1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40734f0 0x7f8ca40a9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40734f0 0x7f8ca40a9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36592/0 (socket says 172.21.15.27:36592) 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca1e00640 1 -- 172.21.15.27:0/1116637818 learned_addr learned my addr 172.21.15.27:0/1116637818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca2800640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ca40c7720 0x7f8ca41ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca1400640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 0x7f8ca41f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca1e00640 1 -- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 msgr2=0x7f8ca41f8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca1e00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 0x7f8ca41f8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca1e00640 1 -- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ca40c7720 msgr2=0x7f8ca41ebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:41.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca1e00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ca40c7720 0x7f8ca41ebb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:41.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca1e00640 1 -- 172.21.15.27:0/1116637818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca4153880 con 0x7f8ca40734f0 2024-03-28T19:20:41.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.650+0000 7f8ca2800640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ca40c7720 0x7f8ca41ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:20:41.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.651+0000 7f8ca1e00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40734f0 0x7f8ca40a9250 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400bf00 tx=0x7f8c9400f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:41.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.651+0000 7f8c9a000640 1 -- 172.21.15.27:0/1116637818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9401b070 con 0x7f8ca40734f0 2024-03-28T19:20:41.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.651+0000 7f8c9a000640 1 -- 172.21.15.27:0/1116637818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9400fd90 con 0x7f8ca40734f0 2024-03-28T19:20:41.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.651+0000 7f8c9a000640 1 -- 172.21.15.27:0/1116637818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9400dd50 con 0x7f8ca40734f0 2024-03-28T19:20:41.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.652+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca40c0f20 con 0x7f8ca40734f0 2024-03-28T19:20:41.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.652+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca41ec080 con 0x7f8ca40734f0 2024-03-28T19:20:41.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.656+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca404f400 con 0x7f8ca40734f0 2024-03-28T19:20:41.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.661+0000 7f8c9a000640 1 -- 172.21.15.27:0/1116637818 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8c9401e070 con 0x7f8ca40734f0 2024-03-28T19:20:41.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.663+0000 7f8c9a000640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c8406bca0 0x7f8c8406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:41.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.664+0000 7f8c9a000640 1 -- 172.21.15.27:0/1116637818 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f8c9408b180 con 0x7f8ca40734f0 2024-03-28T19:20:41.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.665+0000 7f8ca1400640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c8406bca0 0x7f8c8406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:41.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.666+0000 7f8ca1400640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c8406bca0 0x7f8c8406e160 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4151050 tx=0x7f8c90009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:41.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:41.668+0000 7f8c9a000640 1 -- 172.21.15.27:0/1116637818 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8c94058dc0 con 0x7f8ca40734f0 2024-03-28T19:20:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.090+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ca4151120 con 0x7f8ca40734f0 2024-03-28T19:20:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.091+0000 7f8c9a000640 1 -- 172.21.15.27:0/1116637818 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f8c940586c0 con 0x7f8ca40734f0 2024-03-28T19:20:42.090 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:42.091 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:40.288993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:20:42.091 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 188 2024-03-28T19:20:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c8406bca0 msgr2=0x7f8c8406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c8406bca0 0x7f8c8406e160 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4151050 tx=0x7f8c90009040 comp rx=0 tx=0).stop 2024-03-28T19:20:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40734f0 msgr2=0x7f8ca40a9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40734f0 0x7f8ca40a9250 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400bf00 tx=0x7f8c9400f480 comp rx=0 tx=0).stop 2024-03-28T19:20:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 shutdown_connections 2024-03-28T19:20:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c8406bca0 0x7f8c8406e160 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ca40c7720 0x7f8ca41ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ca40aa7c0 0x7f8ca41f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 --2- 172.21.15.27:0/1116637818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ca40734f0 0x7f8ca40a9250 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 >> 172.21.15.27:0/1116637818 conn(0x7f8ca40c6c00 msgr2=0x7f8ca4150670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 shutdown_connections 2024-03-28T19:20:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:42.095+0000 7f8ca8a00640 1 -- 172.21.15.27:0/1116637818 wait complete. 2024-03-28T19:20:42.120 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 188, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 185, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6269": { "addr": "172.21.15.27:6835/1132521215", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1132521215, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 1132521215, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6269, "incarnation": 180, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:18:54.625050+0000", "qdb_cluster": [ 6269 ], "qdb_leader": 6269, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6269 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 188, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8052": { "addr": "172.21.15.27:6837/433723712", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 433723712, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 433723712, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8052, "incarnation": 188, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 86, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:20:40.288993+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8052 } } } ], "standbys": [ { "addr": "172.21.15.155:6835/2331326731", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 2331326731, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 2331326731, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 186, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7273, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:20:43.673 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 19% (48/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 50% (124/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 67% (166/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 85% (210/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2024-03-28T19:20:43.893 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2024-03-28T19:20:43.908 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.5 2024-03-28T19:20:43.963 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:20:43.963 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 6 -eq 48 ']' 2024-03-28T19:20:43.963 INFO:tasks.workunit.client.0.smithi027.stdout:v0.6 2024-03-28T19:20:43.964 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.6 2024-03-28T19:20:43.964 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.6 2024-03-28T19:20:43.964 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.6 2024-03-28T19:20:44.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.399+0000 7f01bee00640 1 Processor -- start 2024-03-28T19:20:44.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.399+0000 7f01bee00640 1 -- start start 2024-03-28T19:20:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.401+0000 7f01bee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01b80c59d0 0x7f01b80c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.401+0000 7f01bee00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b8155fe0 con 0x7f01b8155600 2024-03-28T19:20:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.401+0000 7f01bee00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b81562c0 con 0x7f01b80727e0 2024-03-28T19:20:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.401+0000 7f01bee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b81565a0 con 0x7f01b80c59d0 2024-03-28T19:20:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.402+0000 7f01bda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01b80c59d0 0x7f01b80c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.402+0000 7f01bda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f01b80c59d0 0x7f01b80c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36608/0 (socket says 172.21.15.27:36608) 2024-03-28T19:20:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.402+0000 7f01be400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f01b8155600 0x7f01b814f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50030/0 (socket says 172.21.15.27:50030) 2024-03-28T19:20:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.402+0000 7f01bda00640 1 -- 172.21.15.27:0/2329707997 learned_addr learned my addr 172.21.15.27:0/2329707997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.402+0000 7f01bda00640 1 -- 172.21.15.27:0/2329707997 >> v1:172.21.15.155:6789/0 conn(0x7f01b80727e0 legacy=0x7f01b8159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.402+0000 7f01bda00640 1 -- 172.21.15.27:0/2329707997 >> v1:172.21.15.27:6789/0 conn(0x7f01b8155600 legacy=0x7f01b814f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.403+0000 7f01bda00640 1 -- 172.21.15.27:0/2329707997 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8158d20 con 0x7f01b80c59d0 2024-03-28T19:20:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.405+0000 7f01bda00640 1 --2- 172.21.15.27:0/2329707997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b80c59d0 0x7f01b80c8f00 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009870 tx=0x7f01ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f36dfbcfa5cc214 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.405+0000 7f01b7e00640 1 -- 172.21.15.27:0/2329707997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac02e070 con 0x7f01b80c59d0 2024-03-28T19:20:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.405+0000 7f01b7e00640 1 -- 172.21.15.27:0/2329707997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac037070 con 0x7f01b80c59d0 2024-03-28T19:20:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.405+0000 7f01b7e00640 1 -- 172.21.15.27:0/2329707997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac031a00 con 0x7f01b80c59d0 2024-03-28T19:20:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.406+0000 7f01bee00640 1 -- 172.21.15.27:0/2329707997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b80c59d0 msgr2=0x7f01b80c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.406+0000 7f01bee00640 1 --2- 172.21.15.27:0/2329707997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b80c59d0 0x7f01b80c8f00 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009870 tx=0x7f01ac02b880 comp rx=0 tx=0).stop 2024-03-28T19:20:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.406+0000 7f01bee00640 1 -- 172.21.15.27:0/2329707997 shutdown_connections 2024-03-28T19:20:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.406+0000 7f01bee00640 1 --2- 172.21.15.27:0/2329707997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b80c59d0 0x7f01b80c8f00 secure :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009870 tx=0x7f01ac02b880 comp rx=0 tx=0).stop 2024-03-28T19:20:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.406+0000 7f01bee00640 1 -- 172.21.15.27:0/2329707997 >> 172.21.15.27:0/2329707997 conn(0x7f01b80ac910 msgr2=0x7f01b80accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.407+0000 7f01bee00640 1 -- 172.21.15.27:0/2329707997 shutdown_connections 2024-03-28T19:20:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.408+0000 7f01bee00640 1 -- 172.21.15.27:0/2329707997 wait complete. 2024-03-28T19:20:44.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.409+0000 7f01bee00640 1 Processor -- start 2024-03-28T19:20:44.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.410+0000 7f01bee00640 1 -- start start 2024-03-28T19:20:44.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bee00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01b80727e0 0x7f01b80be180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:44.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b8155600 0x7f01b80c4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bee00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01b814bae0 0x7f01b8148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80c5db0 con 0x7f01b8155600 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bee00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b8159090 con 0x7f01b80727e0 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bee00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b80c5f30 con 0x7f01b814bae0 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b8155600 0x7f01b80c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b8155600 0x7f01b80c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36620/0 (socket says 172.21.15.27:36620) 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bd000640 1 -- 172.21.15.27:0/4090988471 learned_addr learned my addr 172.21.15.27:0/4090988471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.411+0000 7f01bda00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01b80727e0 0x7f01b80be180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.412+0000 7f01be400640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01b814bae0 0x7f01b8148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:44.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.412+0000 7f01bd000640 1 -- 172.21.15.27:0/4090988471 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01b814bae0 msgr2=0x7f01b8148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:44.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.412+0000 7f01bd000640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01b814bae0 0x7f01b8148050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:44.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.412+0000 7f01bd000640 1 -- 172.21.15.27:0/4090988471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01b80727e0 msgr2=0x7f01b80be180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:44.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.412+0000 7f01bd000640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01b80727e0 0x7f01b80be180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:44.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.412+0000 7f01bd000640 1 -- 172.21.15.27:0/4090988471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b80c1f50 con 0x7f01b8155600 2024-03-28T19:20:44.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.413+0000 7f01bd000640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b8155600 0x7f01b80c4760 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f01a400c430 tx=0x7f01a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:44.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.413+0000 7f01b5600640 1 -- 172.21.15.27:0/4090988471 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a4011790 con 0x7f01b8155600 2024-03-28T19:20:44.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.413+0000 7f01b5600640 1 -- 172.21.15.27:0/4090988471 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a4017070 con 0x7f01b8155600 2024-03-28T19:20:44.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.413+0000 7f01b5600640 1 -- 172.21.15.27:0/4090988471 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a4011b70 con 0x7f01b8155600 2024-03-28T19:20:44.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.413+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b81f02e0 con 0x7f01b8155600 2024-03-28T19:20:44.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.414+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b80c24d0 con 0x7f01b8155600 2024-03-28T19:20:44.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.415+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194005290 con 0x7f01b8155600 2024-03-28T19:20:44.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.419+0000 7f01b5600640 1 -- 172.21.15.27:0/4090988471 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f01a4011d10 con 0x7f01b8155600 2024-03-28T19:20:44.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.420+0000 7f01b5600640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f019806bca0 0x7f019806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:44.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.420+0000 7f01b5600640 1 -- 172.21.15.27:0/4090988471 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f01a408b6a0 con 0x7f01b8155600 2024-03-28T19:20:44.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.421+0000 7f01bda00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f019806bca0 0x7f019806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:44.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.425+0000 7f01bda00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f019806bca0 0x7f019806e160 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f01ac036000 tx=0x7f01ac03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:44.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.428+0000 7f01b5600640 1 -- 172.21.15.27:0/4090988471 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f01a4058c00 con 0x7f01b8155600 2024-03-28T19:20:44.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.884+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0194005580 con 0x7f01b8155600 2024-03-28T19:20:44.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.885+0000 7f01b5600640 1 -- 172.21.15.27:0/4090988471 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f01a405cca0 con 0x7f01b8155600 2024-03-28T19:20:44.886 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:44.886 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:40.288993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:20:44.886 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 188 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.888+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f019806bca0 msgr2=0x7f019806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.888+0000 7f01bee00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f019806bca0 0x7f019806e160 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f01ac036000 tx=0x7f01ac03d040 comp rx=0 tx=0).stop 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.888+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b8155600 msgr2=0x7f01b80c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.888+0000 7f01bee00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b8155600 0x7f01b80c4760 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f01a400c430 tx=0x7f01a400c8f0 comp rx=0 tx=0).stop 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.889+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 shutdown_connections 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.889+0000 7f01bee00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f019806bca0 0x7f019806e160 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.889+0000 7f01bee00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01b814bae0 0x7f01b8148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.889+0000 7f01bee00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01b8155600 0x7f01b80c4760 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.889+0000 7f01bee00640 1 --2- 172.21.15.27:0/4090988471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01b80727e0 0x7f01b80be180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.889+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 >> 172.21.15.27:0/4090988471 conn(0x7f01b80ac910 msgr2=0x7f01b80c77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.889+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 shutdown_connections 2024-03-28T19:20:44.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:44.889+0000 7f01bee00640 1 -- 172.21.15.27:0/4090988471 wait complete. 2024-03-28T19:20:44.906 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:20:45.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:46.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.793+0000 7f28f6200640 1 Processor -- start 2024-03-28T19:20:46.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.794+0000 7f28f6200640 1 -- start start 2024-03-28T19:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.796+0000 7f28f6200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f006ee00 0x7f28f0158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.796+0000 7f28f6200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28f00c3ea0 0x7f28f0159440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.796+0000 7f28f6200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f00c1600 con 0x7f28f00c60e0 2024-03-28T19:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.796+0000 7f28f6200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f00c23b0 con 0x7f28f00c3ea0 2024-03-28T19:20:46.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.796+0000 7f28f6200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00c18e0 con 0x7f28f006ee00 2024-03-28T19:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.796+0000 7f28ef400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f28f00c60e0 0x7f28f006e6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38222/0 (socket says 172.21.15.27:38222) 2024-03-28T19:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.797+0000 7f28ef400640 1 -- 172.21.15.27:0/3080484583 learned_addr learned my addr 172.21.15.27:0/3080484583 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.797+0000 7f28eea00640 1 --2- 172.21.15.27:0/3080484583 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f006ee00 0x7f28f0158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:46.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.797+0000 7f28efe00640 1 --2- 172.21.15.27:0/3080484583 >> v2:172.21.15.27:3300/0 conn(0x7f28f00c3ea0 0x7f28f0159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.797+0000 7f28ee000640 1 -- 172.21.15.27:0/3080484583 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452749123 0 0) 0x7f28f00c1600 con 0x7f28f00c60e0 2024-03-28T19:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.797+0000 7f28ee000640 1 -- 172.21.15.27:0/3080484583 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d4003520 con 0x7f28f00c60e0 2024-03-28T19:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.798+0000 7f28eea00640 1 -- 172.21.15.27:0/3080484583 >> v2:172.21.15.27:3300/0 conn(0x7f28f00c3ea0 msgr2=0x7f28f0159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.798+0000 7f28eea00640 1 --2- 172.21.15.27:0/3080484583 >> v2:172.21.15.27:3300/0 conn(0x7f28f00c3ea0 0x7f28f0159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.798+0000 7f28eea00640 1 -- 172.21.15.27:0/3080484583 >> v1:172.21.15.27:6789/0 conn(0x7f28f00c60e0 legacy=0x7f28f006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.798+0000 7f28eea00640 1 -- 172.21.15.27:0/3080484583 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00c1bc0 con 0x7f28f006ee00 2024-03-28T19:20:46.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.798+0000 7f28efe00640 1 --2- 172.21.15.27:0/3080484583 >> v2:172.21.15.27:3300/0 conn(0x7f28f00c3ea0 0x7f28f0159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:20:46.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.799+0000 7f28eea00640 1 --2- 172.21.15.27:0/3080484583 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f006ee00 0x7f28f0158f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f28e00097a0 tx=0x7f28e002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87de1ca3eede8a51 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.800+0000 7f28ee000640 1 -- 172.21.15.27:0/3080484583 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0031780 con 0x7f28f006ee00 2024-03-28T19:20:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.800+0000 7f28ee000640 1 -- 172.21.15.27:0/3080484583 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e0037070 con 0x7f28f006ee00 2024-03-28T19:20:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.800+0000 7f28ee000640 1 -- 172.21.15.27:0/3080484583 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0031b40 con 0x7f28f006ee00 2024-03-28T19:20:46.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.801+0000 7f28f6200640 1 -- 172.21.15.27:0/3080484583 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f006ee00 msgr2=0x7f28f0158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.801+0000 7f28f6200640 1 --2- 172.21.15.27:0/3080484583 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f006ee00 0x7f28f0158f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f28e00097a0 tx=0x7f28e002b820 comp rx=0 tx=0).stop 2024-03-28T19:20:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.801+0000 7f28f6200640 1 -- 172.21.15.27:0/3080484583 shutdown_connections 2024-03-28T19:20:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.801+0000 7f28f6200640 1 --2- 172.21.15.27:0/3080484583 >> v2:172.21.15.27:3300/0 conn(0x7f28f00c3ea0 0x7f28f0159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.801+0000 7f28f6200640 1 --2- 172.21.15.27:0/3080484583 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f006ee00 0x7f28f0158f00 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f28e00097a0 tx=0x7f28e002b820 comp rx=0 tx=0).stop 2024-03-28T19:20:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.801+0000 7f28f6200640 1 -- 172.21.15.27:0/3080484583 >> 172.21.15.27:0/3080484583 conn(0x7f28f006b9d0 msgr2=0x7f28f00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.801+0000 7f28f6200640 1 -- 172.21.15.27:0/3080484583 shutdown_connections 2024-03-28T19:20:46.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.801+0000 7f28f6200640 1 -- 172.21.15.27:0/3080484583 wait complete. 2024-03-28T19:20:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.803+0000 7f28f6200640 1 Processor -- start 2024-03-28T19:20:46.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.804+0000 7f28f6200640 1 -- start start 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.804+0000 7f28f6200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28f00c3ea0 0x7f28f0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.804+0000 7f28f6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f00c60e0 0x7f28f00a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28f6200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f00abff0 0x7f28f00aa160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28f6200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00c6640 con 0x7f28f00c60e0 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28f6200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00c64c0 con 0x7f28f00c3ea0 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28f6200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f0159980 con 0x7f28f00abff0 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28eea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f00c60e0 0x7f28f00a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28eea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f00c60e0 0x7f28f00a9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38014/0 (socket says 172.21.15.27:38014) 2024-03-28T19:20:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28eea00640 1 -- 172.21.15.27:0/1117695968 learned_addr learned my addr 172.21.15.27:0/1117695968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28eea00640 1 -- 172.21.15.27:0/1117695968 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f00abff0 msgr2=0x7f28f00aa160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T19:20:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28eea00640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f00abff0 0x7f28f00aa160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28eea00640 1 -- 172.21.15.27:0/1117695968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28f00c3ea0 msgr2=0x7f28f0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28eea00640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28f00c3ea0 0x7f28f0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.805+0000 7f28eea00640 1 -- 172.21.15.27:0/1117695968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f01dff60 con 0x7f28f00c60e0 2024-03-28T19:20:46.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.806+0000 7f28eea00640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f00c60e0 0x7f28f00a9c20 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f28e00098c0 tx=0x7f28e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:46.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.806+0000 7f28e7400640 1 -- 172.21.15.27:0/1117695968 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0046650 con 0x7f28f00c60e0 2024-03-28T19:20:46.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.806+0000 7f28e7400640 1 -- 172.21.15.27:0/1117695968 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e0046810 con 0x7f28f00c60e0 2024-03-28T19:20:46.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.806+0000 7f28e7400640 1 -- 172.21.15.27:0/1117695968 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0046c00 con 0x7f28f00c60e0 2024-03-28T19:20:46.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.807+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f01ef270 con 0x7f28f00c60e0 2024-03-28T19:20:46.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.807+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f01ef550 con 0x7f28f00c60e0 2024-03-28T19:20:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.808+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f004f400 con 0x7f28f00c60e0 2024-03-28T19:20:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.809+0000 7f28e7400640 1 -- 172.21.15.27:0/1117695968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f28e0032410 con 0x7f28f00c60e0 2024-03-28T19:20:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.811+0000 7f28e7400640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28cc06bc50 0x7f28cc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.811+0000 7f28e7400640 1 -- 172.21.15.27:0/1117695968 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f28e00adb50 con 0x7f28f00c60e0 2024-03-28T19:20:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.812+0000 7f28ef400640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28cc06bc50 0x7f28cc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.812+0000 7f28ef400640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28cc06bc50 0x7f28cc06e110 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f28e80012e0 tx=0x7f28e80015b0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:46.818+0000 7f28e7400640 1 -- 172.21.15.27:0/1117695968 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f28e004b060 con 0x7f28f00c60e0 2024-03-28T19:20:46.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:47.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.227+0000 7f7ecd600640 1 Processor -- start 2024-03-28T19:20:47.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.227+0000 7f7ecd600640 1 -- start start 2024-03-28T19:20:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.228+0000 7f7ecd600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ec80c59d0 0x7f7ec80c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.228+0000 7f7ecd600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8155600 0x7f7ec814f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.228+0000 7f7ecd600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8155fe0 con 0x7f7ec80727e0 2024-03-28T19:20:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.228+0000 7f7ecd600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec81565a0 con 0x7f7ec80c59d0 2024-03-28T19:20:47.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.228+0000 7f7ecd600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8159090 con 0x7f7ec8155600 2024-03-28T19:20:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.229+0000 7f7ec7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ec80727e0 0x7f7ec8159520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38232/0 (socket says 172.21.15.27:38232) 2024-03-28T19:20:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.229+0000 7f7ec7400640 1 -- 172.21.15.27:0/3115055860 learned_addr learned my addr 172.21.15.27:0/3115055860 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.229+0000 7f7eccc00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8155600 0x7f7ec814f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.229+0000 7f7ec7e00640 1 --2- 172.21.15.27:0/3115055860 >> v2:172.21.15.155:3300/0 conn(0x7f7ec80c59d0 0x7f7ec80c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.230+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345541642 0 0) 0x7f7ec8155fe0 con 0x7f7ec80727e0 2024-03-28T19:20:47.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.230+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eac003520 con 0x7f7ec80727e0 2024-03-28T19:20:47.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.230+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219732578 0 0) 0x7f7eac003520 con 0x7f7ec80727e0 2024-03-28T19:20:47.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8155600 msgr2=0x7f7ec814f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8155600 0x7f7ec814f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 >> v2:172.21.15.155:3300/0 conn(0x7f7ec80c59d0 msgr2=0x7f7ec80c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 --2- 172.21.15.27:0/3115055860 >> v2:172.21.15.155:3300/0 conn(0x7f7ec80c59d0 0x7f7ec80c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec81562c0 con 0x7f7ec80727e0 2024-03-28T19:20:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f7eb0001670 con 0x7f7ec80727e0 2024-03-28T19:20:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 >> v1:172.21.15.27:6789/0 conn(0x7f7ec80727e0 legacy=0x7f7ec8159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7eac0039b0 0x7f7eac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7eac004800 0x7f7eac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eac0051c0 0x7f7eac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec81565a0 con 0x7f7eac0051c0 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8159090 con 0x7f7eac004800 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.231+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac00bfe0 con 0x7f7eac0039b0 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.232+0000 7f7ec7400640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7eac004800 0x7f7eac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.232+0000 7f7ec7e00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7eac0039b0 0x7f7eac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.232+0000 7f7eccc00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eac0051c0 0x7f7eac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.233+0000 7f7ec7400640 1 -- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7eac0039b0 msgr2=0x7f7eac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:47.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.233+0000 7f7ec7400640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7eac0039b0 0x7f7eac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.233+0000 7f7ec7400640 1 -- 172.21.15.27:0/3115055860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eac0051c0 msgr2=0x7f7eac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:47.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.233+0000 7f7ec7400640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eac0051c0 0x7f7eac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.233+0000 7f7ec7400640 1 -- 172.21.15.27:0/3115055860 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec81562c0 con 0x7f7eac004800 2024-03-28T19:20:47.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.233+0000 7f7eccc00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eac0051c0 0x7f7eac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:20:47.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.234+0000 7f7ec7e00640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7eac0039b0 0x7f7eac003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:20:47.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.236+0000 7f7ec7400640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7eac004800 0x7f7eac004c80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7ec806c830 tx=0x7f7eac005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab66d87eac860d12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:47.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.237+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0007dd0 con 0x7f7eac004800 2024-03-28T19:20:47.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.237+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb0030070 con 0x7f7eac004800 2024-03-28T19:20:47.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.237+0000 7f7ec6a00640 1 -- 172.21.15.27:0/3115055860 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb002cdc0 con 0x7f7eac004800 2024-03-28T19:20:47.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.237+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28f00c4ce0 con 0x7f28f00c60e0 2024-03-28T19:20:47.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.238+0000 7f28e7400640 1 -- 172.21.15.27:0/1117695968 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f28e002f050 con 0x7f28f00c60e0 2024-03-28T19:20:47.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.238+0000 7f7ecd600640 1 -- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7eac004800 msgr2=0x7f7eac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.238+0000 7f7ecd600640 1 --2- 172.21.15.27:0/3115055860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7eac004800 0x7f7eac004c80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7ec806c830 tx=0x7f7eac005830 comp rx=0 tx=0).stop 2024-03-28T19:20:47.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.238+0000 7f7eccc00640 1 -- 172.21.15.27:0/3115055860 reap_dead start 2024-03-28T19:20:47.237 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:47.238 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:40.288993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:20:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 188 2024-03-28T19:20:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.238+0000 7f7ecd600640 1 -- 172.21.15.27:0/3115055860 shutdown_connections 2024-03-28T19:20:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.238+0000 7f7ecd600640 1 -- 172.21.15.27:0/3115055860 >> 172.21.15.27:0/3115055860 conn(0x7f7ec80ac910 msgr2=0x7f7ec80accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:47.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.239+0000 7f7ecd600640 1 -- 172.21.15.27:0/3115055860 shutdown_connections 2024-03-28T19:20:47.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.239+0000 7f7ecd600640 1 -- 172.21.15.27:0/3115055860 wait complete. 2024-03-28T19:20:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.241+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28cc06bc50 msgr2=0x7f28cc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.241+0000 7f28f6200640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28cc06bc50 0x7f28cc06e110 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f28e80012e0 tx=0x7f28e80015b0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.241+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f00c60e0 msgr2=0x7f28f00a9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.241+0000 7f28f6200640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f00c60e0 0x7f28f00a9c20 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f28e00098c0 tx=0x7f28e00046c0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 shutdown_connections 2024-03-28T19:20:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f28f6200640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28cc06bc50 0x7f28cc06e110 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f28f6200640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28f00abff0 0x7f28f00aa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f28f6200640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f00c60e0 0x7f28f00a9c20 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f7ecd600640 1 Processor -- start 2024-03-28T19:20:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f28f6200640 1 --2- 172.21.15.27:0/1117695968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28f00c3ea0 0x7f28f0071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 >> 172.21.15.27:0/1117695968 conn(0x7f28f006b9d0 msgr2=0x7f28f00a6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:47.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 shutdown_connections 2024-03-28T19:20:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f7ecd600640 1 -- start start 2024-03-28T19:20:47.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.242+0000 7f28f6200640 1 -- 172.21.15.27:0/1117695968 wait complete. 2024-03-28T19:20:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.243+0000 7f7ecd600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8149740 0x7f7ec81eb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.243+0000 7f7ecd600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ec80cb600 0x7f7ec81eb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.243+0000 7f7ecd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec81ebc40 0x7f7ec80c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.243+0000 7f7ecd600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80c5f30 con 0x7f7ec81ebc40 2024-03-28T19:20:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.243+0000 7f7ecd600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80c5db0 con 0x7f7ec80cb600 2024-03-28T19:20:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.243+0000 7f7ecd600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8155a50 con 0x7f7ec8149740 2024-03-28T19:20:47.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.244+0000 7f7eccc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec81ebc40 0x7f7ec80c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.244+0000 7f7eccc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec81ebc40 0x7f7ec80c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38032/0 (socket says 172.21.15.27:38032) 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.244+0000 7f7eccc00640 1 -- 172.21.15.27:0/418272039 learned_addr learned my addr 172.21.15.27:0/418272039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.244+0000 7f7ec7e00640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8149740 0x7f7ec81eb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.244+0000 7f7eccc00640 1 -- 172.21.15.27:0/418272039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8149740 msgr2=0x7f7ec81eb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.244+0000 7f7eccc00640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8149740 0x7f7ec81eb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.244+0000 7f7eccc00640 1 -- 172.21.15.27:0/418272039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ec80cb600 msgr2=0x7f7ec81eb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.244+0000 7f7eccc00640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ec80cb600 0x7f7ec81eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.245+0000 7f7eccc00640 1 -- 172.21.15.27:0/418272039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec81ee6b0 con 0x7f7ec81ebc40 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.245+0000 7f7eccc00640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec81ebc40 0x7f7ec80c8f00 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8072bc0 tx=0x7f7ec0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.245+0000 7f7ebfe00640 1 -- 172.21.15.27:0/418272039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0015440 con 0x7f7ec81ebc40 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.245+0000 7f7ebfe00640 1 -- 172.21.15.27:0/418272039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec001c070 con 0x7f7ec81ebc40 2024-03-28T19:20:47.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.245+0000 7f7ebfe00640 1 -- 172.21.15.27:0/418272039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0015800 con 0x7f7ec81ebc40 2024-03-28T19:20:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.246+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec81ee990 con 0x7f7ec81ebc40 2024-03-28T19:20:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.246+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec81eef50 con 0x7f7ec81ebc40 2024-03-28T19:20:47.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.247+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8069640 con 0x7f7ec81ebc40 2024-03-28T19:20:47.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.249+0000 7f7ebfe00640 1 -- 172.21.15.27:0/418272039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7ec00159a0 con 0x7f7ec81ebc40 2024-03-28T19:20:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.251+0000 7f7ebfe00640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7ea406bc20 0x7f7ea406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.252+0000 7f7ebfe00640 1 -- 172.21.15.27:0/418272039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f7ec008c5d0 con 0x7f7ec81ebc40 2024-03-28T19:20:47.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.254+0000 7f7ec7e00640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7ea406bc20 0x7f7ea406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.256+0000 7f7ec7e00640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7ea406bc20 0x7f7ea406e0e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7ec814a130 tx=0x7f7eb8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:47.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.259+0000 7f7ebfe00640 1 -- 172.21.15.27:0/418272039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7ec005a270 con 0x7f7ec81ebc40 2024-03-28T19:20:47.267 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-03-28T19:20:47.267 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:20:47.267 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:20:47.303 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:20:47.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:47.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.545+0000 7f191f200640 1 Processor -- start 2024-03-28T19:20:47.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.546+0000 7f191f200640 1 -- start start 2024-03-28T19:20:47.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.547+0000 7f191f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19200c8ed0 0x7f19200c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.547+0000 7f191f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200be5d0 0x7f1920155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.547+0000 7f191f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1920156230 con 0x7f19200c58c0 2024-03-28T19:20:47.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.547+0000 7f191f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1920158bd0 con 0x7f19200c8ed0 2024-03-28T19:20:47.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.547+0000 7f191f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1920158eb0 con 0x7f19200be5d0 2024-03-28T19:20:47.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.547+0000 7f191d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19200c8ed0 0x7f19200c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.547+0000 7f191de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19200c58c0 0x7f19201555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38246/0 (socket says 172.21.15.27:38246) 2024-03-28T19:20:47.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.547+0000 7f191de00640 1 -- 172.21.15.27:0/3359880757 learned_addr learned my addr 172.21.15.27:0/3359880757 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:47.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.548+0000 7f191d400640 1 -- 172.21.15.27:0/3359880757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200be5d0 msgr2=0x7f1920155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:47.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.548+0000 7f191e800640 1 --2- 172.21.15.27:0/3359880757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200be5d0 0x7f1920155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.548+0000 7f191d400640 1 --2- 172.21.15.27:0/3359880757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200be5d0 0x7f1920155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.548+0000 7f191d400640 1 -- 172.21.15.27:0/3359880757 >> v1:172.21.15.27:6789/0 conn(0x7f19200c58c0 legacy=0x7f19201555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.548+0000 7f191d400640 1 -- 172.21.15.27:0/3359880757 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1920159190 con 0x7f19200c8ed0 2024-03-28T19:20:47.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.548+0000 7f191ca00640 1 -- 172.21.15.27:0/3359880757 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035321728 0 0) 0x7f1920156230 con 0x7f19200c58c0 2024-03-28T19:20:47.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.548+0000 7f191e800640 1 --2- 172.21.15.27:0/3359880757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200be5d0 0x7f1920155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.549+0000 7f191d400640 1 --2- 172.21.15.27:0/3359880757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19200c5ca0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f190c0097a0 tx=0x7f190c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c0ff05164a01e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.549+0000 7f191ca00640 1 -- 172.21.15.27:0/3359880757 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c031780 con 0x7f19200c8ed0 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.549+0000 7f191ca00640 1 -- 172.21.15.27:0/3359880757 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c037070 con 0x7f19200c8ed0 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.549+0000 7f191ca00640 1 -- 172.21.15.27:0/3359880757 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c031b40 con 0x7f19200c8ed0 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.550+0000 7f191f200640 1 -- 172.21.15.27:0/3359880757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 msgr2=0x7f19200c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.550+0000 7f191f200640 1 --2- 172.21.15.27:0/3359880757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19200c5ca0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f190c0097a0 tx=0x7f190c02b820 comp rx=0 tx=0).stop 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.550+0000 7f191f200640 1 -- 172.21.15.27:0/3359880757 shutdown_connections 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.550+0000 7f191f200640 1 --2- 172.21.15.27:0/3359880757 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200be5d0 0x7f1920155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.550+0000 7f191f200640 1 --2- 172.21.15.27:0/3359880757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19200c5ca0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.550+0000 7f191f200640 1 -- 172.21.15.27:0/3359880757 >> 172.21.15.27:0/3359880757 conn(0x7f192006b6f0 msgr2=0x7f19200acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:47.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.550+0000 7f191f200640 1 -- 172.21.15.27:0/3359880757 shutdown_connections 2024-03-28T19:20:47.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.550+0000 7f191f200640 1 -- 172.21.15.27:0/3359880757 wait complete. 2024-03-28T19:20:47.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.552+0000 7f191f200640 1 Processor -- start 2024-03-28T19:20:47.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.552+0000 7f191f200640 1 -- start start 2024-03-28T19:20:47.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19200be5d0 0x7f1920154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200c58c0 0x7f19201eb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19201ebee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19200b0820 con 0x7f19200c8ed0 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1920159750 con 0x7f19200be5d0 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19200b09a0 con 0x7f19200c58c0 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19201ebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19201ebee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38046/0 (socket says 172.21.15.27:38046) 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191e800640 1 -- 172.21.15.27:0/736363832 learned_addr learned my addr 172.21.15.27:0/736363832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191d400640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200c58c0 0x7f19201eb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.553+0000 7f191de00640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19200be5d0 0x7f1920154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f191e800640 1 -- 172.21.15.27:0/736363832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200c58c0 msgr2=0x7f19201eb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f191e800640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200c58c0 0x7f19201eb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f191e800640 1 -- 172.21.15.27:0/736363832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19200be5d0 msgr2=0x7f1920154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f191e800640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19200be5d0 0x7f1920154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f191e800640 1 -- 172.21.15.27:0/736363832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19200c13b0 con 0x7f19200c8ed0 2024-03-28T19:20:47.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f191e800640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19201ebee0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f191800cd80 tx=0x7f191800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f1916000640 1 -- 172.21.15.27:0/736363832 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191800f680 con 0x7f19200c8ed0 2024-03-28T19:20:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f1916000640 1 -- 172.21.15.27:0/736363832 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191800f840 con 0x7f19200c8ed0 2024-03-28T19:20:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.554+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19201ef840 con 0x7f19200c8ed0 2024-03-28T19:20:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.555+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19200c3e00 con 0x7f19200c8ed0 2024-03-28T19:20:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.555+0000 7f1916000640 1 -- 172.21.15.27:0/736363832 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191800fc30 con 0x7f19200c8ed0 2024-03-28T19:20:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.556+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19201eb670 con 0x7f19200c8ed0 2024-03-28T19:20:47.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.558+0000 7f1916000640 1 -- 172.21.15.27:0/736363832 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f19180213f0 con 0x7f19200c8ed0 2024-03-28T19:20:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.560+0000 7f1916000640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f190006bbd0 0x7f190006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.560+0000 7f1916000640 1 -- 172.21.15.27:0/736363832 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f191808cb30 con 0x7f19200c8ed0 2024-03-28T19:20:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.560+0000 7f191de00640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f190006bbd0 0x7f190006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:47.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.561+0000 7f191de00640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f190006bbd0 0x7f190006e090 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1910000f60 tx=0x7f1910007c00 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:47.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.563+0000 7f1916000640 1 -- 172.21.15.27:0/736363832 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1918014030 con 0x7f19200c8ed0 2024-03-28T19:20:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.639+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ec814dd60 con 0x7f7ec81ebc40 2024-03-28T19:20:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.640+0000 7f7ebfe00640 1 -- 172.21.15.27:0/418272039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f7ec0059b70 con 0x7f7ec81ebc40 2024-03-28T19:20:47.639 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:47.640 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:40.288993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:20:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 188 2024-03-28T19:20:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.643+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7ea406bc20 msgr2=0x7f7ea406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7ea406bc20 0x7f7ea406e0e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7ec814a130 tx=0x7f7eb8008040 comp rx=0 tx=0).stop 2024-03-28T19:20:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec81ebc40 msgr2=0x7f7ec80c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec81ebc40 0x7f7ec80c8f00 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8072bc0 tx=0x7f7ec0010590 comp rx=0 tx=0).stop 2024-03-28T19:20:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 shutdown_connections 2024-03-28T19:20:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7ea406bc20 0x7f7ea406e0e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec81ebc40 0x7f7ec80c8f00 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ec80cb600 0x7f7ec81eb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 --2- 172.21.15.27:0/418272039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ec8149740 0x7f7ec81eb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 >> 172.21.15.27:0/418272039 conn(0x7f7ec80ac910 msgr2=0x7f7ec80aaea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:47.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 shutdown_connections 2024-03-28T19:20:47.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.644+0000 7f7ecd600640 1 -- 172.21.15.27:0/418272039 wait complete. 2024-03-28T19:20:47.662 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:20:47.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.822+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1920069640 con 0x7f19200c8ed0 2024-03-28T19:20:47.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.822+0000 7f1916000640 1 -- 172.21.15.27:0/736363832 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f191805a6e0 con 0x7f19200c8ed0 2024-03-28T19:20:47.821 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:47.822 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:40.288993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:20:47.822 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 188 2024-03-28T19:20:47.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f190006bbd0 msgr2=0x7f190006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f190006bbd0 0x7f190006e090 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1910000f60 tx=0x7f1910007c00 comp rx=0 tx=0).stop 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 msgr2=0x7f19201ebee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19201ebee0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f191800cd80 tx=0x7f191800a650 comp rx=0 tx=0).stop 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 shutdown_connections 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f190006bbd0 0x7f190006e090 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19200c8ed0 0x7f19201ebee0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19200c58c0 0x7f19201eb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 --2- 172.21.15.27:0/736363832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19200be5d0 0x7f1920154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 >> 172.21.15.27:0/736363832 conn(0x7f192006b6f0 msgr2=0x7f19200ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 shutdown_connections 2024-03-28T19:20:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:47.825+0000 7f191f200640 1 -- 172.21.15.27:0/736363832 wait complete. 2024-03-28T19:20:47.842 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:20:49.664 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:49.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:49.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.885+0000 7f4f75400640 1 Processor -- start 2024-03-28T19:20:49.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.885+0000 7f4f75400640 1 -- start start 2024-03-28T19:20:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.886+0000 7f4f75400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f70152f80 0x7f4f700c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.886+0000 7f4f75400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4f700c7970 0x7f4f700c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.886+0000 7f4f75400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f700c7310 con 0x7f4f70155730 2024-03-28T19:20:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.886+0000 7f4f75400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f70158d40 con 0x7f4f70152f80 2024-03-28T19:20:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.886+0000 7f4f75400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f70159020 con 0x7f4f700c7970 2024-03-28T19:20:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.886+0000 7f4f6e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f70152f80 0x7f4f700c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:49.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.886+0000 7f4f6f200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4f70155730 0x7f4f70155ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38260/0 (socket says 172.21.15.27:38260) 2024-03-28T19:20:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.887+0000 7f4f6f200640 1 -- 172.21.15.27:0/4260982075 learned_addr learned my addr 172.21.15.27:0/4260982075 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.887+0000 7f4f6de00640 1 --2- 172.21.15.27:0/4260982075 >> v2:172.21.15.155:3300/0 conn(0x7f4f700c7970 0x7f4f700c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.887+0000 7f4f6e800640 1 -- 172.21.15.27:0/4260982075 >> v2:172.21.15.155:3300/0 conn(0x7f4f700c7970 msgr2=0x7f4f700c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.887+0000 7f4f6e800640 1 --2- 172.21.15.27:0/4260982075 >> v2:172.21.15.155:3300/0 conn(0x7f4f700c7970 0x7f4f700c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.887+0000 7f4f6e800640 1 -- 172.21.15.27:0/4260982075 >> v1:172.21.15.27:6789/0 conn(0x7f4f70155730 legacy=0x7f4f70155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.887+0000 7f4f6e800640 1 -- 172.21.15.27:0/4260982075 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f70159300 con 0x7f4f70152f80 2024-03-28T19:20:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.887+0000 7f4f6d400640 1 -- 172.21.15.27:0/4260982075 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216757961 0 0) 0x7f4f700c7310 con 0x7f4f70155730 2024-03-28T19:20:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.888+0000 7f4f6e800640 1 --2- 172.21.15.27:0/4260982075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f70152f80 0x7f4f700c58e0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f4f700c8480 tx=0x7f4f6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5882193c2886a260 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f6d400640 1 -- 172.21.15.27:0/4260982075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f600315a0 con 0x7f4f70152f80 2024-03-28T19:20:49.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f6d400640 1 -- 172.21.15.27:0/4260982075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f60037070 con 0x7f4f70152f80 2024-03-28T19:20:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f6d400640 1 -- 172.21.15.27:0/4260982075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60031960 con 0x7f4f70152f80 2024-03-28T19:20:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f75400640 1 -- 172.21.15.27:0/4260982075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f70152f80 msgr2=0x7f4f700c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:49.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f75400640 1 --2- 172.21.15.27:0/4260982075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f70152f80 0x7f4f700c58e0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f4f700c8480 tx=0x7f4f6002b840 comp rx=0 tx=0).stop 2024-03-28T19:20:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f75400640 1 -- 172.21.15.27:0/4260982075 shutdown_connections 2024-03-28T19:20:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f75400640 1 --2- 172.21.15.27:0/4260982075 >> v2:172.21.15.155:3300/0 conn(0x7f4f700c7970 0x7f4f700c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f75400640 1 --2- 172.21.15.27:0/4260982075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f70152f80 0x7f4f700c58e0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.889+0000 7f4f75400640 1 -- 172.21.15.27:0/4260982075 >> 172.21.15.27:0/4260982075 conn(0x7f4f7006b6f0 msgr2=0x7f4f700caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.890+0000 7f4f75400640 1 -- 172.21.15.27:0/4260982075 shutdown_connections 2024-03-28T19:20:49.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.890+0000 7f4f75400640 1 -- 172.21.15.27:0/4260982075 wait complete. 2024-03-28T19:20:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.891+0000 7f4f75400640 1 Processor -- start 2024-03-28T19:20:49.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.891+0000 7f4f75400640 1 -- start start 2024-03-28T19:20:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.892+0000 7f4f75400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f700c7970 0x7f4f7014ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.892+0000 7f4f75400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f70152f80 0x7f4f700c2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:49.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.892+0000 7f4f75400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f70155730 0x7f4f700cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:49.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.892+0000 7f4f75400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f70159b00 con 0x7f4f700c7970 2024-03-28T19:20:49.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.892+0000 7f4f75400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700c5e20 con 0x7f4f70152f80 2024-03-28T19:20:49.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.892+0000 7f4f75400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700cbca0 con 0x7f4f70155730 2024-03-28T19:20:49.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f700c7970 0x7f4f7014ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:49.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f700c7970 0x7f4f7014ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38064/0 (socket says 172.21.15.27:38064) 2024-03-28T19:20:49.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 -- 172.21.15.27:0/1651444723 learned_addr learned my addr 172.21.15.27:0/1651444723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6f200640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f70155730 0x7f4f700cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6de00640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f70152f80 0x7f4f700c2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 -- 172.21.15.27:0/1651444723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f70155730 msgr2=0x7f4f700cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f70155730 0x7f4f700cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 -- 172.21.15.27:0/1651444723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f70152f80 msgr2=0x7f4f700c2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f70152f80 0x7f4f700c2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 -- 172.21.15.27:0/1651444723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f700c4220 con 0x7f4f700c7970 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.893+0000 7f4f6e800640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f700c7970 0x7f4f7014ab30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009890 tx=0x7f4f600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.894+0000 7f4f66a00640 1 -- 172.21.15.27:0/1651444723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60032840 con 0x7f4f700c7970 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.894+0000 7f4f66a00640 1 -- 172.21.15.27:0/1651444723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f60032a00 con 0x7f4f700c7970 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.894+0000 7f4f66a00640 1 -- 172.21.15.27:0/1651444723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60032df0 con 0x7f4f700c7970 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.894+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f701eaf60 con 0x7f4f700c7970 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.894+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f701eb520 con 0x7f4f700c7970 2024-03-28T19:20:49.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.895+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7004f400 con 0x7f4f700c7970 2024-03-28T19:20:49.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.896+0000 7f4f66a00640 1 -- 172.21.15.27:0/1651444723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4f6004c020 con 0x7f4f700c7970 2024-03-28T19:20:49.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.897+0000 7f4f66a00640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f5006bbd0 0x7f4f5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:49.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.897+0000 7f4f66a00640 1 -- 172.21.15.27:0/1651444723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f4f6002f020 con 0x7f4f700c7970 2024-03-28T19:20:49.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.897+0000 7f4f6de00640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f5006bbd0 0x7f4f5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:49.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.898+0000 7f4f6de00640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f5006bbd0 0x7f4f5006e090 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c006fd0 tx=0x7f4f5c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:49.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:49.907+0000 7f4f66a00640 1 -- 172.21.15.27:0/1651444723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4f60040070 con 0x7f4f700c7970 2024-03-28T19:20:50.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.053+0000 7f3dac400640 1 Processor -- start 2024-03-28T19:20:50.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.053+0000 7f3dac400640 1 -- start start 2024-03-28T19:20:50.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.055+0000 7f3dac400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3da406ee00 0x7f3da4158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:50.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.055+0000 7f3dac400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da40c1600 con 0x7f3da40c3ea0 2024-03-28T19:20:50.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.055+0000 7f3dac400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da40c18e0 con 0x7f3da40c60e0 2024-03-28T19:20:50.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.055+0000 7f3dac400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da40c23b0 con 0x7f3da406ee00 2024-03-28T19:20:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.055+0000 7f3daa400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3da40c3ea0 0x7f3da4159440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38266/0 (socket says 172.21.15.27:38266) 2024-03-28T19:20:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.055+0000 7f3daa400640 1 -- 172.21.15.27:0/440015364 learned_addr learned my addr 172.21.15.27:0/440015364 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.055+0000 7f3da9000640 1 --2- 172.21.15.27:0/440015364 >> v2:172.21.15.155:3300/0 conn(0x7f3da406ee00 0x7f3da4158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699167577 0 0) 0x7f3da40c1600 con 0x7f3da40c3ea0 2024-03-28T19:20:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d8c003520 con 0x7f3da40c3ea0 2024-03-28T19:20:50.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 452274261 0 0) 0x7f3d8c003520 con 0x7f3da40c3ea0 2024-03-28T19:20:50.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 >> v2:172.21.15.155:3300/0 conn(0x7f3da406ee00 msgr2=0x7f3da4158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:50.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 --2- 172.21.15.27:0/440015364 >> v2:172.21.15.155:3300/0 conn(0x7f3da406ee00 0x7f3da4158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 >> v1:172.21.15.155:6789/0 conn(0x7f3da40c60e0 legacy=0x7f3da406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da40c1bc0 con 0x7f3da40c3ea0 2024-03-28T19:20:50.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f3d9c0019f0 con 0x7f3da40c3ea0 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 >> v1:172.21.15.27:6789/0 conn(0x7f3da40c3ea0 legacy=0x7f3da4159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8c0039b0 0x7f3d8c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c004800 0x7f3d8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d8c005210 0x7f3d8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40c23b0 con 0x7f3d8c0039b0 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c00c140 con 0x7f3d8c005210 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c00c340 con 0x7f3d8c004800 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.056+0000 7f3da9a00640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8c0039b0 0x7f3d8c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.057+0000 7f3da9000640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c004800 0x7f3d8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.057+0000 7f3da9a00640 1 -- 172.21.15.27:0/440015364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c004800 msgr2=0x7f3d8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.057+0000 7f3da9a00640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c004800 0x7f3d8c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.057+0000 7f3daa400640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d8c005210 0x7f3d8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.057+0000 7f3da9a00640 1 -- 172.21.15.27:0/440015364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d8c005210 msgr2=0x7f3d8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:50.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.057+0000 7f3da9a00640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d8c005210 0x7f3d8c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.057+0000 7f3da9a00640 1 -- 172.21.15.27:0/440015364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c003520 con 0x7f3d8c0039b0 2024-03-28T19:20:50.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.057+0000 7f3daa400640 1 -- 172.21.15.27:0/440015364 reap_dead start 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.058+0000 7f3da9a00640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8c0039b0 0x7f3d8c003df0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3d98001750 tx=0x7f3d8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d340c68f367adbc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.058+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98001370 con 0x7f3d8c0039b0 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.058+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d980052d0 con 0x7f3d8c0039b0 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.058+0000 7f3da3e00640 1 -- 172.21.15.27:0/440015364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98003830 con 0x7f3d8c0039b0 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.059+0000 7f3dac400640 1 -- 172.21.15.27:0/440015364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8c0039b0 msgr2=0x7f3d8c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.059+0000 7f3dac400640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8c0039b0 0x7f3d8c003df0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3d98001750 tx=0x7f3d8c005830 comp rx=0 tx=0).stop 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.059+0000 7f3dac400640 1 -- 172.21.15.27:0/440015364 shutdown_connections 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.059+0000 7f3dac400640 1 --2- 172.21.15.27:0/440015364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d8c0039b0 0x7f3d8c003df0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.059+0000 7f3dac400640 1 -- 172.21.15.27:0/440015364 >> 172.21.15.27:0/440015364 conn(0x7f3da406b9d0 msgr2=0x7f3da40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:50.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.059+0000 7f3dac400640 1 -- 172.21.15.27:0/440015364 shutdown_connections 2024-03-28T19:20:50.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.059+0000 7f3dac400640 1 -- 172.21.15.27:0/440015364 wait complete. 2024-03-28T19:20:50.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.061+0000 7f3dac400640 1 Processor -- start 2024-03-28T19:20:50.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.061+0000 7f3dac400640 1 -- start start 2024-03-28T19:20:50.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3dac400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c0039b0 0x7f3da40a7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:50.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3dac400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da4159a30 0x7f3da4156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:50.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3dac400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3da4157310 0x7f3da41ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:50.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3dac400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40c6640 con 0x7f3da4157310 2024-03-28T19:20:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3dac400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40c64c0 con 0x7f3da4159a30 2024-03-28T19:20:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3dac400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da4072df0 con 0x7f3d8c0039b0 2024-03-28T19:20:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3daa400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3da4157310 0x7f3da41ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3daa400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3da4157310 0x7f3da41ee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38078/0 (socket says 172.21.15.27:38078) 2024-03-28T19:20:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.062+0000 7f3daa400640 1 -- 172.21.15.27:0/231838508 learned_addr learned my addr 172.21.15.27:0/231838508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.063+0000 7f3da9a00640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c0039b0 0x7f3da40a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:50.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.063+0000 7f3da9000640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da4159a30 0x7f3da4156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.063+0000 7f3daa400640 1 -- 172.21.15.27:0/231838508 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c0039b0 msgr2=0x7f3da40a7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.063+0000 7f3daa400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c0039b0 0x7f3da40a7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.063+0000 7f3daa400640 1 -- 172.21.15.27:0/231838508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da4159a30 msgr2=0x7f3da4156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.063+0000 7f3daa400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da4159a30 0x7f3da4156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.063+0000 7f3daa400640 1 -- 172.21.15.27:0/231838508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da4159370 con 0x7f3da4157310 2024-03-28T19:20:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.063+0000 7f3daa400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3da4157310 0x7f3da41ee4e0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f3da40c4c20 tx=0x7f3d9c00b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.064+0000 7f3da1600640 1 -- 172.21.15.27:0/231838508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00b870 con 0x7f3da4157310 2024-03-28T19:20:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.064+0000 7f3da1600640 1 -- 172.21.15.27:0/231838508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c00ba30 con 0x7f3da4157310 2024-03-28T19:20:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.064+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da41538c0 con 0x7f3da4157310 2024-03-28T19:20:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.064+0000 7f3da1600640 1 -- 172.21.15.27:0/231838508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00be20 con 0x7f3da4157310 2024-03-28T19:20:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.064+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da40a7890 con 0x7f3da4157310 2024-03-28T19:20:50.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.065+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da404f400 con 0x7f3da4157310 2024-03-28T19:20:50.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.066+0000 7f3da1600640 1 -- 172.21.15.27:0/231838508 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d9c01b020 con 0x7f3da4157310 2024-03-28T19:20:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.067+0000 7f3da1600640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d8406bbd0 0x7f3d8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.067+0000 7f3da1600640 1 -- 172.21.15.27:0/231838508 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f3d9c08bac0 con 0x7f3da4157310 2024-03-28T19:20:50.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.067+0000 7f3da9a00640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d8406bbd0 0x7f3d8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:50.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.068+0000 7f3da9a00640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d8406bbd0 0x7f3d8406e090 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3d98033000 tx=0x7f3d98009030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:50.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.072+0000 7f3da1600640 1 -- 172.21.15.27:0/231838508 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d9c0596e0 con 0x7f3da4157310 2024-03-28T19:20:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.152+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f700be740 con 0x7f4f700c7970 2024-03-28T19:20:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.153+0000 7f4f66a00640 1 -- 172.21.15.27:0/1651444723 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5461 (secure 0 0 0) 0x7f4f6007ac60 con 0x7f4f700c7970 2024-03-28T19:20:50.153 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:50.153 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:48.674650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:20:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 190 2024-03-28T19:20:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f5006bbd0 msgr2=0x7f4f5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f5006bbd0 0x7f4f5006e090 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c006fd0 tx=0x7f4f5c008040 comp rx=0 tx=0).stop 2024-03-28T19:20:50.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f700c7970 msgr2=0x7f4f7014ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f700c7970 0x7f4f7014ab30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009890 tx=0x7f4f600046c0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 shutdown_connections 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4f5006bbd0 0x7f4f5006e090 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4f70155730 0x7f4f700cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f70152f80 0x7f4f700c2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 --2- 172.21.15.27:0/1651444723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f700c7970 0x7f4f7014ab30 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 >> 172.21.15.27:0/1651444723 conn(0x7f4f7006b6f0 msgr2=0x7f4f70150a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.156+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 shutdown_connections 2024-03-28T19:20:50.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.157+0000 7f4f75400640 1 -- 172.21.15.27:0/1651444723 wait complete. 2024-03-28T19:20:50.176 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:20:50.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.318+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3da40c4ce0 con 0x7f3da4157310 2024-03-28T19:20:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.320+0000 7f3da1600640 1 -- 172.21.15.27:0/231838508 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5461 (secure 0 0 0) 0x7f3d9c058fe0 con 0x7f3da4157310 2024-03-28T19:20:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:50.319 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:48.674650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:20:50.319 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 190 2024-03-28T19:20:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d8406bbd0 msgr2=0x7f3d8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d8406bbd0 0x7f3d8406e090 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3d98033000 tx=0x7f3d98009030 comp rx=0 tx=0).stop 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3da4157310 msgr2=0x7f3da41ee4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3da4157310 0x7f3da41ee4e0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f3da40c4c20 tx=0x7f3d9c00b0d0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 shutdown_connections 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d8406bbd0 0x7f3d8406e090 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3da4157310 0x7f3da41ee4e0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da4159a30 0x7f3da4156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 --2- 172.21.15.27:0/231838508 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d8c0039b0 0x7f3da40a7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.322+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 >> 172.21.15.27:0/231838508 conn(0x7f3da406b9d0 msgr2=0x7f3da40c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.323+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 shutdown_connections 2024-03-28T19:20:50.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:50.323+0000 7f3dac400640 1 -- 172.21.15.27:0/231838508 wait complete. 2024-03-28T19:20:50.341 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:20:52.178 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:52.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:52.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.408+0000 7f5358800640 1 Processor -- start 2024-03-28T19:20:52.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.408+0000 7f5358800640 1 -- start start 2024-03-28T19:20:52.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.409+0000 7f5358800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53500c8e10 0x7f5350158be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.409+0000 7f5358800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53501530f0 con 0x7f53501562f0 2024-03-28T19:20:52.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.409+0000 7f5358800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535006d3d0 con 0x7f5350152d10 2024-03-28T19:20:52.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.409+0000 7f5358800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53500c92f0 con 0x7f53500c8e10 2024-03-28T19:20:52.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.410+0000 7f5356800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f53501562f0 0x7f535015ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38276/0 (socket says 172.21.15.27:38276) 2024-03-28T19:20:52.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.410+0000 7f5356800640 1 -- 172.21.15.27:0/4186459324 learned_addr learned my addr 172.21.15.27:0/4186459324 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.410+0000 7f5355400640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53500c8e10 0x7f5350158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.410+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108299335 0 0) 0x7f534c001490 con 0x7f53501562f0 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.410+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5338003520 con 0x7f53501562f0 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888097163 0 0) 0x7f535006d3d0 con 0x7f5350152d10 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f534c001490 con 0x7f5350152d10 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979979792 0 0) 0x7f5338003520 con 0x7f53501562f0 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53500c8e10 msgr2=0x7f5350158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53500c8e10 0x7f5350158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 >> v1:172.21.15.155:6789/0 conn(0x7f5350152d10 legacy=0x7f53500c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5350155810 con 0x7f53501562f0 2024-03-28T19:20:52.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f534c001bc0 con 0x7f53501562f0 2024-03-28T19:20:52.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 >> v1:172.21.15.27:6789/0 conn(0x7f53501562f0 legacy=0x7f535015ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5338003b70 0x7f5338003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53500c8e10 0x7f5338004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5338004d40 0x7f53380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53500c92f0 con 0x7f5338003b70 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533800bb80 con 0x7f5338004d40 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533800bd80 con 0x7f53500c8e10 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.411+0000 7f5355e00640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5338003b70 0x7f5338003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.412+0000 7f5355e00640 1 -- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53500c8e10 msgr2=0x7f5338004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.412+0000 7f5355e00640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53500c8e10 0x7f5338004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.412+0000 7f5355e00640 1 -- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5338004d40 msgr2=0x7f53380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.412+0000 7f5355e00640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5338004d40 0x7f53380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.412+0000 7f5355e00640 1 -- 172.21.15.27:0/4186459324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5338003520 con 0x7f5338003b70 2024-03-28T19:20:52.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.413+0000 7f5355e00640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5338003b70 0x7f5338003f90 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f53440018d0 tx=0x7f5338005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609e6d8fffde06fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.413+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344037070 con 0x7f5338003b70 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.413+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53440052d0 con 0x7f5338003b70 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.413+0000 7f5354a00640 1 -- 172.21.15.27:0/4186459324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53440014f0 con 0x7f5338003b70 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.414+0000 7f5358800640 1 -- 172.21.15.27:0/4186459324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5338003b70 msgr2=0x7f5338003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.414+0000 7f5358800640 1 --2- 172.21.15.27:0/4186459324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5338003b70 0x7f5338003f90 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f53440018d0 tx=0x7f5338005830 comp rx=0 tx=0).stop 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.414+0000 7f5356800640 1 -- 172.21.15.27:0/4186459324 reap_dead start 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.414+0000 7f5358800640 1 -- 172.21.15.27:0/4186459324 shutdown_connections 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.414+0000 7f5358800640 1 -- 172.21.15.27:0/4186459324 >> 172.21.15.27:0/4186459324 conn(0x7f5350066fb0 msgr2=0x7f53500673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.414+0000 7f5358800640 1 -- 172.21.15.27:0/4186459324 shutdown_connections 2024-03-28T19:20:52.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.415+0000 7f5358800640 1 -- 172.21.15.27:0/4186459324 wait complete. 2024-03-28T19:20:52.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.416+0000 7f5358800640 1 Processor -- start 2024-03-28T19:20:52.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.417+0000 7f5358800640 1 -- start start 2024-03-28T19:20:52.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.418+0000 7f5358800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5338003b70 0x7f53500c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.418+0000 7f5358800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5350148280 0x7f53500c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.418+0000 7f5358800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350148750 0x7f53501ee860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.418+0000 7f5358800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53500c5fa0 con 0x7f5350148750 2024-03-28T19:20:52.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.418+0000 7f5358800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535006d850 con 0x7f5350148280 2024-03-28T19:20:52.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.418+0000 7f5358800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53500c6120 con 0x7f5338003b70 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.419+0000 7f5355400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5350148280 0x7f53500c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.419+0000 7f5355e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5338003b70 0x7f53500c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.419+0000 7f5355400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5350148280 0x7f53500c4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:43822/0 (socket says 172.21.15.27:43822) 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.419+0000 7f5355400640 1 -- 172.21.15.27:0/692731903 learned_addr learned my addr 172.21.15.27:0/692731903 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.419+0000 7f5356800640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350148750 0x7f53501ee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.420+0000 7f5355400640 1 -- 172.21.15.27:0/692731903 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5338003b70 msgr2=0x7f53500c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.420+0000 7f5355400640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5338003b70 0x7f53500c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.420+0000 7f5355400640 1 -- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350148750 msgr2=0x7f53501ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.420+0000 7f5355400640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350148750 0x7f53501ee860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.420+0000 7f5355400640 1 -- 172.21.15.27:0/692731903 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53501ef3a0 con 0x7f5350148280 2024-03-28T19:20:52.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.420+0000 7f5355e00640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5338003b70 0x7f53500c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:20:52.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.420+0000 7f5356800640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350148750 0x7f53501ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:20:52.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.421+0000 7f5355400640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5350148280 0x7f53500c4040 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f534000e7c0 tx=0x7f534000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:52.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.421+0000 7f534a000640 1 -- 172.21.15.27:0/692731903 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340017070 con 0x7f5350148280 2024-03-28T19:20:52.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.421+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53501ef680 con 0x7f5350148280 2024-03-28T19:20:52.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.421+0000 7f534a000640 1 -- 172.21.15.27:0/692731903 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534000c940 con 0x7f5350148280 2024-03-28T19:20:52.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.421+0000 7f534a000640 1 -- 172.21.15.27:0/692731903 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534000cd10 con 0x7f5350148280 2024-03-28T19:20:52.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.421+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53501592f0 con 0x7f5350148280 2024-03-28T19:20:52.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.423+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53500c87b0 con 0x7f5350148280 2024-03-28T19:20:52.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.425+0000 7f534a000640 1 -- 172.21.15.27:0/692731903 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f534001a070 con 0x7f5350148280 2024-03-28T19:20:52.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.427+0000 7f534a000640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f533406bc00 0x7f533406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.427+0000 7f534a000640 1 -- 172.21.15.27:0/692731903 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f534008bfd0 con 0x7f5350148280 2024-03-28T19:20:52.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.427+0000 7f5355e00640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f533406bc00 0x7f533406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.428+0000 7f5355e00640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f533406bc00 0x7f533406e0c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f5344034000 tx=0x7f5344031000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:52.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.441+0000 7f534a000640 1 -- 172.21.15.27:0/692731903 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f53500c87b0 con 0x7f5350148280 2024-03-28T19:20:52.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.623+0000 7f5befe00640 1 Processor -- start 2024-03-28T19:20:52.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.623+0000 7f5befe00640 1 -- start start 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.625+0000 7f5befe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5bf00c8ed0 0x7f5bf00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.625+0000 7f5befe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5bf00be5d0 0x7f5bf0155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.625+0000 7f5befe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf0156230 con 0x7f5bf00c58c0 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.625+0000 7f5befe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf0158bd0 con 0x7f5bf00c8ed0 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.625+0000 7f5befe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf0158eb0 con 0x7f5bf00be5d0 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.626+0000 7f5bee000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5bf00c8ed0 0x7f5bf00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.626+0000 7f5bee000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5bf00c8ed0 0x7f5bf00c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38110/0 (socket says 172.21.15.27:38110) 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.626+0000 7f5bee000640 1 -- 172.21.15.27:0/1897512515 learned_addr learned my addr 172.21.15.27:0/1897512515 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.626+0000 7f5bef400640 1 --2- 172.21.15.27:0/1897512515 >> v2:172.21.15.155:3300/0 conn(0x7f5bf00be5d0 0x7f5bf0155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.626+0000 7f5bee000640 1 -- 172.21.15.27:0/1897512515 >> v2:172.21.15.155:3300/0 conn(0x7f5bf00be5d0 msgr2=0x7f5bf0155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:52.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.626+0000 7f5bee000640 1 --2- 172.21.15.27:0/1897512515 >> v2:172.21.15.155:3300/0 conn(0x7f5bf00be5d0 0x7f5bf0155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.626+0000 7f5bee000640 1 -- 172.21.15.27:0/1897512515 >> v1:172.21.15.155:6789/0 conn(0x7f5bf00c58c0 legacy=0x7f5bf01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:52.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.626+0000 7f5bee000640 1 -- 172.21.15.27:0/1897512515 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf0159190 con 0x7f5bf00c8ed0 2024-03-28T19:20:52.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.628+0000 7f5bee000640 1 --2- 172.21.15.27:0/1897512515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00c8ed0 0x7f5bf00c5ca0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5bf008d050 tx=0x7f5bd802bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5b0415256741fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:52.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.628+0000 7f5bed600640 1 -- 172.21.15.27:0/1897512515 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd802c960 con 0x7f5bf00c8ed0 2024-03-28T19:20:52.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.628+0000 7f5bed600640 1 -- 172.21.15.27:0/1897512515 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd802cb00 con 0x7f5bf00c8ed0 2024-03-28T19:20:52.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.628+0000 7f5bed600640 1 -- 172.21.15.27:0/1897512515 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80343f0 con 0x7f5bf00c8ed0 2024-03-28T19:20:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.629+0000 7f5befe00640 1 -- 172.21.15.27:0/1897512515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00c8ed0 msgr2=0x7f5bf00c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.629+0000 7f5befe00640 1 --2- 172.21.15.27:0/1897512515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00c8ed0 0x7f5bf00c5ca0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f5bf008d050 tx=0x7f5bd802bdc0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.629+0000 7f5befe00640 1 -- 172.21.15.27:0/1897512515 shutdown_connections 2024-03-28T19:20:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.629+0000 7f5befe00640 1 --2- 172.21.15.27:0/1897512515 >> v2:172.21.15.155:3300/0 conn(0x7f5bf00be5d0 0x7f5bf0155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.629+0000 7f5befe00640 1 --2- 172.21.15.27:0/1897512515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00c8ed0 0x7f5bf00c5ca0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.629+0000 7f5befe00640 1 -- 172.21.15.27:0/1897512515 >> 172.21.15.27:0/1897512515 conn(0x7f5bf006b6f0 msgr2=0x7f5bf00acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.629+0000 7f5befe00640 1 -- 172.21.15.27:0/1897512515 shutdown_connections 2024-03-28T19:20:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.629+0000 7f5befe00640 1 -- 172.21.15.27:0/1897512515 wait complete. 2024-03-28T19:20:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.631+0000 7f5befe00640 1 Processor -- start 2024-03-28T19:20:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.631+0000 7f5befe00640 1 -- start start 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5befe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00be5d0 0x7f5bf014ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5befe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bf00c58c0 0x7f5bf00aba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5befe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bf00c8ed0 0x7f5bf01eea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5befe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf00b0820 con 0x7f5bf00be5d0 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5befe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf0159750 con 0x7f5bf00c58c0 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5befe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf00b09a0 con 0x7f5bf00c8ed0 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5beea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00be5d0 0x7f5bf014ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5beea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00be5d0 0x7f5bf014ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38116/0 (socket says 172.21.15.27:38116) 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.632+0000 7f5beea00640 1 -- 172.21.15.27:0/2389775620 learned_addr learned my addr 172.21.15.27:0/2389775620 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:52.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5bee000640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bf00c58c0 0x7f5bf00aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5beea00640 1 -- 172.21.15.27:0/2389775620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bf00c8ed0 msgr2=0x7f5bf01eea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5beea00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bf00c8ed0 0x7f5bf01eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5beea00640 1 -- 172.21.15.27:0/2389775620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bf00c58c0 msgr2=0x7f5bf00aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5beea00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bf00c58c0 0x7f5bf00aba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5beea00640 1 -- 172.21.15.27:0/2389775620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf01ebf60 con 0x7f5bf00be5d0 2024-03-28T19:20:52.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5bee000640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bf00c58c0 0x7f5bf00aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:20:52.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5beea00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00be5d0 0x7f5bf014ffc0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5be000b420 tx=0x7f5be000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.633+0000 7f5be6a00640 1 -- 172.21.15.27:0/2389775620 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be0011830 con 0x7f5bf00be5d0 2024-03-28T19:20:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.634+0000 7f5be6a00640 1 -- 172.21.15.27:0/2389775620 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be0016070 con 0x7f5bf00be5d0 2024-03-28T19:20:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.634+0000 7f5be6a00640 1 -- 172.21.15.27:0/2389775620 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be0011c10 con 0x7f5bf00be5d0 2024-03-28T19:20:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.634+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf0158210 con 0x7f5bf00be5d0 2024-03-28T19:20:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.634+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf01ec240 con 0x7f5bf00be5d0 2024-03-28T19:20:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.635+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf0069640 con 0x7f5bf00be5d0 2024-03-28T19:20:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.636+0000 7f5be6a00640 1 -- 172.21.15.27:0/2389775620 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5be001b460 con 0x7f5bf00be5d0 2024-03-28T19:20:52.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.637+0000 7f5be6a00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bcc06bbd0 0x7f5bcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:52.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.637+0000 7f5be6a00640 1 -- 172.21.15.27:0/2389775620 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f5be008b4b0 con 0x7f5bf00be5d0 2024-03-28T19:20:52.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.637+0000 7f5bee000640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bcc06bbd0 0x7f5bcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.638+0000 7f5bee000640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bcc06bbd0 0x7f5bcc06e090 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8035000 tx=0x7f5bd8030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.642+0000 7f5be6a00640 1 -- 172.21.15.27:0/2389775620 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5be0020020 con 0x7f5bf00be5d0 2024-03-28T19:20:52.694 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:20:52.696+0000 7f4229000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:20:52.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.734+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f535004f400 con 0x7f5350148280 2024-03-28T19:20:52.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.735+0000 7f534a000640 1 -- 172.21.15.27:0/692731903 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f5340059530 con 0x7f5350148280 2024-03-28T19:20:52.735 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:52.735 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:20:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 192 2024-03-28T19:20:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f533406bc00 msgr2=0x7f533406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f533406bc00 0x7f533406e0c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f5344034000 tx=0x7f5344031000 comp rx=0 tx=0).stop 2024-03-28T19:20:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5350148280 msgr2=0x7f53500c4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5350148280 0x7f53500c4040 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f534000e7c0 tx=0x7f534000ec80 comp rx=0 tx=0).stop 2024-03-28T19:20:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 shutdown_connections 2024-03-28T19:20:52.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f533406bc00 0x7f533406e0c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5350148750 0x7f53501ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5350148280 0x7f53500c4040 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 --2- 172.21.15.27:0/692731903 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5338003b70 0x7f53500c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 >> 172.21.15.27:0/692731903 conn(0x7f5350066fb0 msgr2=0x7f53501515c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:52.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 shutdown_connections 2024-03-28T19:20:52.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.738+0000 7f5358800640 1 -- 172.21.15.27:0/692731903 wait complete. 2024-03-28T19:20:52.758 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6260 2024-03-28T19:20:52.758 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.d 2024-03-28T19:20:52.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.944+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bf0156580 con 0x7f5bf00be5d0 2024-03-28T19:20:52.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.945+0000 7f5be6a00640 1 -- 172.21.15.27:0/2389775620 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f5be0059070 con 0x7f5bf00be5d0 2024-03-28T19:20:52.944 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:52.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:20:52.944 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 192 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bcc06bbd0 msgr2=0x7f5bcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bcc06bbd0 0x7f5bcc06e090 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8035000 tx=0x7f5bd8030000 comp rx=0 tx=0).stop 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00be5d0 msgr2=0x7f5bf014ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00be5d0 0x7f5bf014ffc0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f5be000b420 tx=0x7f5be000b8e0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 shutdown_connections 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5bcc06bbd0 0x7f5bcc06e090 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bf00c8ed0 0x7f5bf01eea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bf00c58c0 0x7f5bf00aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 --2- 172.21.15.27:0/2389775620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bf00be5d0 0x7f5bf014ffc0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:52.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 >> 172.21.15.27:0/2389775620 conn(0x7f5bf006b6f0 msgr2=0x7f5bf00c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:52.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 shutdown_connections 2024-03-28T19:20:52.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:52.948+0000 7f5befe00640 1 -- 172.21.15.27:0/2389775620 wait complete. 2024-03-28T19:20:52.967 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:20:54.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:55.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.195+0000 7f2373a00640 1 Processor -- start 2024-03-28T19:20:55.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.195+0000 7f2373a00640 1 -- start start 2024-03-28T19:20:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f2373a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f236c062370 0x7f236c153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f2373a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0ab490 0x7f236c0ac480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f2373a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236c0abb00 con 0x7f236c0ac0a0 2024-03-28T19:20:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f2373a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236c0ab8e0 con 0x7f236c062370 2024-03-28T19:20:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f2373a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c06e410 con 0x7f236c0ab490 2024-03-28T19:20:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f2371000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f236c0ac0a0 0x7f236c153490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38278/0 (socket says 172.21.15.27:38278) 2024-03-28T19:20:55.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f2371000640 1 -- 172.21.15.27:0/4260409008 learned_addr learned my addr 172.21.15.27:0/4260409008 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:55.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f2371a00640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0ab490 0x7f236c0ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f236be00640 1 --2- 172.21.15.27:0/4260409008 >> v2:172.21.15.155:3300/0 conn(0x7f236c062370 0x7f236c153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.197+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420621409 0 0) 0x7f236c0abb00 con 0x7f236c0ac0a0 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2354003520 con 0x7f236c0ac0a0 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027055726 0 0) 0x7f2354003520 con 0x7f236c0ac0a0 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0ab490 msgr2=0x7f236c0ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0ab490 0x7f236c0ac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 >> v2:172.21.15.155:3300/0 conn(0x7f236c062370 msgr2=0x7f236c153ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 --2- 172.21.15.27:0/4260409008 >> v2:172.21.15.155:3300/0 conn(0x7f236c062370 0x7f236c153ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c06e6f0 con 0x7f236c0ac0a0 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f2360001670 con 0x7f236c0ac0a0 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 >> v1:172.21.15.27:6789/0 conn(0x7f236c0ac0a0 legacy=0x7f236c153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0ab490 0x7f23540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004030 0x7f2354004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2354004d40 0x7f23540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0ab8e0 con 0x7f2354004d40 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c06e410 con 0x7f2354004030 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.198+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235400bc50 con 0x7f236c0ab490 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f2371a00640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2354004d40 0x7f23540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f2371000640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0ab490 0x7f23540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f236be00640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004030 0x7f2354004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f2371a00640 1 -- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0ab490 msgr2=0x7f23540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f2371a00640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0ab490 0x7f23540039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f2371a00640 1 -- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004030 msgr2=0x7f2354004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f2371a00640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004030 0x7f2354004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f2371a00640 1 -- 172.21.15.27:0/4260409008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354003520 con 0x7f2354004d40 2024-03-28T19:20:55.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.199+0000 7f236be00640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004030 0x7f2354004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:20:55.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.201+0000 7f2371a00640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2354004d40 0x7f23540051c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f2364007360 tx=0x7f2354005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2af9e60e4d4afa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.201+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364039070 con 0x7f2354004d40 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.201+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236402ccd0 con 0x7f2354004d40 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.201+0000 7f236b400640 1 -- 172.21.15.27:0/4260409008 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364009be0 con 0x7f2354004d40 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.202+0000 7f2373a00640 1 -- 172.21.15.27:0/4260409008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2354004d40 msgr2=0x7f23540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.202+0000 7f2373a00640 1 --2- 172.21.15.27:0/4260409008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2354004d40 0x7f23540051c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f2364007360 tx=0x7f2354005830 comp rx=0 tx=0).stop 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.202+0000 7f2371a00640 1 -- 172.21.15.27:0/4260409008 reap_dead start 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.202+0000 7f2373a00640 1 -- 172.21.15.27:0/4260409008 shutdown_connections 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.202+0000 7f2373a00640 1 -- 172.21.15.27:0/4260409008 >> 172.21.15.27:0/4260409008 conn(0x7f236c06b0c0 msgr2=0x7f236c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.202+0000 7f2373a00640 1 -- 172.21.15.27:0/4260409008 shutdown_connections 2024-03-28T19:20:55.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.203+0000 7f2373a00640 1 -- 172.21.15.27:0/4260409008 wait complete. 2024-03-28T19:20:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.204+0000 7f2373a00640 1 Processor -- start 2024-03-28T19:20:55.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.204+0000 7f2373a00640 1 -- start start 2024-03-28T19:20:55.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.205+0000 7f2373a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004d40 0x7f236c0c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.205+0000 7f2373a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0c7160 0x7f236c1e6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.205+0000 7f2373a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c1eadc0 0x7f236c1e7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.205+0000 7f2373a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c062910 con 0x7f236c1eadc0 2024-03-28T19:20:55.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.205+0000 7f2373a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c06eef0 con 0x7f2354004d40 2024-03-28T19:20:55.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.205+0000 7f2373a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c150d40 con 0x7f236c0c7160 2024-03-28T19:20:55.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c1eadc0 0x7f236c1e7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:55.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c1eadc0 0x7f236c1e7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38140/0 (socket says 172.21.15.27:38140) 2024-03-28T19:20:55.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 -- 172.21.15.27:0/2325245346 learned_addr learned my addr 172.21.15.27:0/2325245346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:55.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371000640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004d40 0x7f236c0c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:55.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 -- 172.21.15.27:0/2325245346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0c7160 msgr2=0x7f236c1e6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:55.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0c7160 0x7f236c1e6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 -- 172.21.15.27:0/2325245346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004d40 msgr2=0x7f236c0c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:55.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004d40 0x7f236c0c6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 -- 172.21.15.27:0/2325245346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c071f00 con 0x7f236c1eadc0 2024-03-28T19:20:55.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.206+0000 7f2371a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c1eadc0 0x7f236c1e7380 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f2364007480 tx=0x7f2364004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:55.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.207+0000 7f2368c00640 1 -- 172.21.15.27:0/2325245346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364039070 con 0x7f236c1eadc0 2024-03-28T19:20:55.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.207+0000 7f2368c00640 1 -- 172.21.15.27:0/2325245346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236403a040 con 0x7f236c1eadc0 2024-03-28T19:20:55.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.207+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c0721e0 con 0x7f236c1eadc0 2024-03-28T19:20:55.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.207+0000 7f2368c00640 1 -- 172.21.15.27:0/2325245346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364009be0 con 0x7f236c1eadc0 2024-03-28T19:20:55.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.207+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c153550 con 0x7f236c1eadc0 2024-03-28T19:20:55.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.208+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c06c050 con 0x7f236c1eadc0 2024-03-28T19:20:55.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.209+0000 7f2368c00640 1 -- 172.21.15.27:0/2325245346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2364046020 con 0x7f236c1eadc0 2024-03-28T19:20:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.210+0000 7f2368c00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f234c06bba0 0x7f234c06e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:55.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.211+0000 7f2368c00640 1 -- 172.21.15.27:0/2325245346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f23640ad100 con 0x7f236c1eadc0 2024-03-28T19:20:55.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.211+0000 7f2371000640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f234c06bba0 0x7f234c06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.212+0000 7f2371000640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f234c06bba0 0x7f234c06e060 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2360001200 tx=0x7f2360001170 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.217+0000 7f2368c00640 1 -- 172.21.15.27:0/2325245346 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f236404d020 con 0x7f236c1eadc0 2024-03-28T19:20:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.499+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f236c04f400 con 0x7f236c1eadc0 2024-03-28T19:20:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.500+0000 7f2368c00640 1 -- 172.21.15.27:0/2325245346 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f236407ac30 con 0x7f236c1eadc0 2024-03-28T19:20:55.499 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:55.499 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:20:55.499 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 192 2024-03-28T19:20:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.502+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f234c06bba0 msgr2=0x7f234c06e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.502+0000 7f2373a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f234c06bba0 0x7f234c06e060 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2360001200 tx=0x7f2360001170 comp rx=0 tx=0).stop 2024-03-28T19:20:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.502+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c1eadc0 msgr2=0x7f236c1e7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.502+0000 7f2373a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c1eadc0 0x7f236c1e7380 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f2364007480 tx=0x7f2364004ab0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.503+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 shutdown_connections 2024-03-28T19:20:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.503+0000 7f2373a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f234c06bba0 0x7f234c06e060 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.503+0000 7f2373a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c1eadc0 0x7f236c1e7380 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.503+0000 7f2373a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f236c0c7160 0x7f236c1e6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.503+0000 7f2373a00640 1 --2- 172.21.15.27:0/2325245346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2354004d40 0x7f236c0c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.503+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 >> 172.21.15.27:0/2325245346 conn(0x7f236c06b0c0 msgr2=0x7f236c0c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.503+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 shutdown_connections 2024-03-28T19:20:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:55.503+0000 7f2373a00640 1 -- 172.21.15.27:0/2325245346 wait complete. 2024-03-28T19:20:55.521 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:20:57.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:57.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.762+0000 7f9783200640 1 Processor -- start 2024-03-28T19:20:57.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.763+0000 7f9783200640 1 -- start start 2024-03-28T19:20:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.764+0000 7f9783200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c150c70 0x7f977c0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.764+0000 7f9783200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f977c0c3690 0x7f977c0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.764+0000 7f9783200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0c3af0 con 0x7f977c0c6660 2024-03-28T19:20:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.764+0000 7f9783200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f977c0ab4e0 con 0x7f977c0c3690 2024-03-28T19:20:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.764+0000 7f9783200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0ac1f0 con 0x7f977c150c70 2024-03-28T19:20:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.764+0000 7f9781200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f977c0c6660 0x7f977c1534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41002/0 (socket says 172.21.15.27:41002) 2024-03-28T19:20:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.764+0000 7f9781200640 1 -- 172.21.15.27:0/3388545288 learned_addr learned my addr 172.21.15.27:0/3388545288 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:20:57.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.764+0000 7f977be00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c150c70 0x7f977c0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977b400640 1 --2- 172.21.15.27:0/3388545288 >> v2:172.21.15.155:3300/0 conn(0x7f977c0c3690 0x7f977c0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895208164 0 0) 0x7f977c0c3af0 con 0x7f977c0c6660 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9760003520 con 0x7f977c0c6660 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2504624793 0 0) 0x7f9760003520 con 0x7f977c0c6660 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c150c70 msgr2=0x7f977c0a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c150c70 0x7f977c0a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 >> v2:172.21.15.155:3300/0 conn(0x7f977c0c3690 msgr2=0x7f977c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 --2- 172.21.15.27:0/3388545288 >> v2:172.21.15.155:3300/0 conn(0x7f977c0c3690 0x7f977c0c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c0c3d90 con 0x7f977c0c6660 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f97740019f0 con 0x7f977c0c6660 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 >> v1:172.21.15.27:6789/0 conn(0x7f977c0c6660 legacy=0x7f977c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c150c70 0x7f9760004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0ac1f0 con 0x7f97600039b0 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976000b9c0 con 0x7f9760004d40 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.765+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976000bb40 con 0x7f977c150c70 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.766+0000 7f977be00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.766+0000 7f977b400640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c150c70 0x7f9760004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.766+0000 7f977be00640 1 -- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c150c70 msgr2=0x7f9760004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.766+0000 7f977be00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c150c70 0x7f9760004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.766+0000 7f9781200640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.766+0000 7f977be00640 1 -- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9760004d40 msgr2=0x7f97600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.766+0000 7f977be00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9760004d40 0x7f97600051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.766+0000 7f977be00640 1 -- 172.21.15.27:0/3388545288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760003520 con 0x7f97600039b0 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.767+0000 7f977be00640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f976c009ac0 tx=0x7f9760005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce6b41668d8f7e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.768+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c039070 con 0x7f97600039b0 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.768+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c02c6b0 con 0x7f97600039b0 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.768+0000 7f977aa00640 1 -- 172.21.15.27:0/3388545288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c009690 con 0x7f97600039b0 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.769+0000 7f9783200640 1 -- 172.21.15.27:0/3388545288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 msgr2=0x7f9760003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.769+0000 7f9783200640 1 --2- 172.21.15.27:0/3388545288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f9760003dd0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f976c009ac0 tx=0x7f9760005830 comp rx=0 tx=0).stop 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.769+0000 7f9781200640 1 -- 172.21.15.27:0/3388545288 reap_dead start 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.769+0000 7f9783200640 1 -- 172.21.15.27:0/3388545288 shutdown_connections 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.769+0000 7f9783200640 1 -- 172.21.15.27:0/3388545288 >> 172.21.15.27:0/3388545288 conn(0x7f977c06b9d0 msgr2=0x7f977c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.769+0000 7f9783200640 1 -- 172.21.15.27:0/3388545288 shutdown_connections 2024-03-28T19:20:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.769+0000 7f9783200640 1 -- 172.21.15.27:0/3388545288 wait complete. 2024-03-28T19:20:57.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.772+0000 7f9783200640 1 Processor -- start 2024-03-28T19:20:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.772+0000 7f9783200640 1 -- start start 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.773+0000 7f9783200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f977c0c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.773+0000 7f9783200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c06de50 0x7f977c1ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.773+0000 7f9783200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1eec40 0x7f977c156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.773+0000 7f9783200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0abcd0 con 0x7f97600039b0 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.773+0000 7f9783200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0ab9f0 con 0x7f977c1eec40 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.773+0000 7f9783200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0c1810 con 0x7f977c06de50 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f977c0c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f977c0c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35310/0 (socket says 172.21.15.27:35310) 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977be00640 1 -- 172.21.15.27:0/3961861859 learned_addr learned my addr 172.21.15.27:0/3961861859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:20:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977b400640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c06de50 0x7f977c1ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f9781200640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1eec40 0x7f977c156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977be00640 1 -- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c06de50 msgr2=0x7f977c1ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977be00640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c06de50 0x7f977c1ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977be00640 1 -- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1eec40 msgr2=0x7f977c156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977be00640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1eec40 0x7f977c156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f977be00640 1 -- 172.21.15.27:0/3961861859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c06e3f0 con 0x7f97600039b0 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.774+0000 7f9781200640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1eec40 0x7f977c156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.775+0000 7f977be00640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f977c0c02a0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f976c037040 tx=0x7f976c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.775+0000 7f9773e00640 1 -- 172.21.15.27:0/3961861859 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c042020 con 0x7f97600039b0 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.775+0000 7f9773e00640 1 -- 172.21.15.27:0/3961861859 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c03a040 con 0x7f97600039b0 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.775+0000 7f9773e00640 1 -- 172.21.15.27:0/3961861859 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c039070 con 0x7f97600039b0 2024-03-28T19:20:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.775+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c06e6d0 con 0x7f97600039b0 2024-03-28T19:20:57.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.776+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c0c07e0 con 0x7f97600039b0 2024-03-28T19:20:57.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.777+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c04f400 con 0x7f97600039b0 2024-03-28T19:20:57.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.778+0000 7f9773e00640 1 -- 172.21.15.27:0/3961861859 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f976c045020 con 0x7f97600039b0 2024-03-28T19:20:57.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.779+0000 7f9773e00640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc00 0x7f975c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:20:57.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.779+0000 7f9773e00640 1 -- 172.21.15.27:0/3961861859 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f976c0aca50 con 0x7f97600039b0 2024-03-28T19:20:57.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.779+0000 7f977b400640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc00 0x7f975c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:20:57.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.780+0000 7f977b400640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc00 0x7f975c06e0c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f977c06e970 tx=0x7f9768008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:20:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:57.789+0000 7f9773e00640 1 -- 172.21.15.27:0/3961861859 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f976c07a4d0 con 0x7f97600039b0 2024-03-28T19:20:58.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.078+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f977c0c5f00 con 0x7f97600039b0 2024-03-28T19:20:58.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.079+0000 7f9773e00640 1 -- 172.21.15.27:0/3961861859 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f976c079dd0 con 0x7f97600039b0 2024-03-28T19:20:58.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:20:58.079 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:20:58.079 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 192 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.082+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc00 msgr2=0x7f975c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.082+0000 7f9783200640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc00 0x7f975c06e0c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f977c06e970 tx=0x7f9768008040 comp rx=0 tx=0).stop 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.082+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 msgr2=0x7f977c0c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.082+0000 7f9783200640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f977c0c02a0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f976c037040 tx=0x7f976c004730 comp rx=0 tx=0).stop 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.083+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 shutdown_connections 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.083+0000 7f9783200640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc00 0x7f975c06e0c0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.083+0000 7f9783200640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1eec40 0x7f977c156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.083+0000 7f9783200640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c06de50 0x7f977c1ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.083+0000 7f9783200640 1 --2- 172.21.15.27:0/3961861859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97600039b0 0x7f977c0c02a0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.083+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 >> 172.21.15.27:0/3961861859 conn(0x7f977c06b9d0 msgr2=0x7f977c061ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:20:58.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.083+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 shutdown_connections 2024-03-28T19:20:58.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:20:58.083+0000 7f9783200640 1 -- 172.21.15.27:0/3961861859 wait complete. 2024-03-28T19:20:58.103 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:20:59.759 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-03-28T19:20:59.759 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T19:20:59.759 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T19:20:59.767 INFO:tasks.ceph.mds.d:Started 2024-03-28T19:20:59.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:20:59.944 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T19:21:00.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.011+0000 7efd17000640 1 Processor -- start 2024-03-28T19:21:00.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.012+0000 7efd17000640 1 -- start start 2024-03-28T19:21:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.013+0000 7efd17000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efd100c2530 0x7efd100c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.014+0000 7efd17000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd10152ad0 0x7efd1006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.014+0000 7efd17000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd100c42d0 con 0x7efd100a7060 2024-03-28T19:21:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.014+0000 7efd17000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd10072d40 con 0x7efd100c2530 2024-03-28T19:21:00.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.014+0000 7efd17000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd10073550 con 0x7efd10152ad0 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.014+0000 7efd15200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efd100c2530 0x7efd100c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.014+0000 7efd16600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd10152ad0 0x7efd1006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.014+0000 7efd15200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efd100c2530 0x7efd100c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:40822/0 (socket says 172.21.15.27:40822) 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.014+0000 7efd15200640 1 -- 172.21.15.27:0/3671639578 learned_addr learned my addr 172.21.15.27:0/3671639578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.015+0000 7efd15200640 1 -- 172.21.15.27:0/3671639578 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd10152ad0 msgr2=0x7efd1006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.015+0000 7efd15200640 1 --2- 172.21.15.27:0/3671639578 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd10152ad0 0x7efd1006c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.015+0000 7efd15200640 1 -- 172.21.15.27:0/3671639578 >> v1:172.21.15.155:6789/0 conn(0x7efd100a7060 legacy=0x7efd10155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.015+0000 7efd15200640 1 -- 172.21.15.27:0/3671639578 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10072f90 con 0x7efd100c2530 2024-03-28T19:21:00.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.016+0000 7efd15200640 1 --2- 172.21.15.27:0/3671639578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd100c2530 0x7efd100c3d90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7efd10072ec0 tx=0x7efcfc02e140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9637c823f78cac09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:00.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.016+0000 7efd0fe00640 1 -- 172.21.15.27:0/3671639578 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc03b070 con 0x7efd100c2530 2024-03-28T19:21:00.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.016+0000 7efd0fe00640 1 -- 172.21.15.27:0/3671639578 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc036440 con 0x7efd100c2530 2024-03-28T19:21:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.016+0000 7efd0fe00640 1 -- 172.21.15.27:0/3671639578 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc036810 con 0x7efd100c2530 2024-03-28T19:21:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.017+0000 7efd17000640 1 -- 172.21.15.27:0/3671639578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd100c2530 msgr2=0x7efd100c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.017+0000 7efd17000640 1 --2- 172.21.15.27:0/3671639578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd100c2530 0x7efd100c3d90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7efd10072ec0 tx=0x7efcfc02e140 comp rx=0 tx=0).stop 2024-03-28T19:21:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.017+0000 7efd17000640 1 -- 172.21.15.27:0/3671639578 shutdown_connections 2024-03-28T19:21:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.017+0000 7efd17000640 1 --2- 172.21.15.27:0/3671639578 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd10152ad0 0x7efd1006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.017+0000 7efd17000640 1 --2- 172.21.15.27:0/3671639578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd100c2530 0x7efd100c3d90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.017+0000 7efd17000640 1 -- 172.21.15.27:0/3671639578 >> 172.21.15.27:0/3671639578 conn(0x7efd100c7aa0 msgr2=0x7efd100c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.017+0000 7efd17000640 1 -- 172.21.15.27:0/3671639578 shutdown_connections 2024-03-28T19:21:00.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.017+0000 7efd17000640 1 -- 172.21.15.27:0/3671639578 wait complete. 2024-03-28T19:21:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.019+0000 7efd17000640 1 Processor -- start 2024-03-28T19:21:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.019+0000 7efd17000640 1 -- start start 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd17000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd100a7060 0x7efd100ab8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd17000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd100c2530 0x7efd1014bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd17000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd10152ad0 0x7efd1014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd17000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd101525b0 con 0x7efd100a7060 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd17000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd101522d0 con 0x7efd10152ad0 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd17000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd10152f20 con 0x7efd100c2530 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd15c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd100a7060 0x7efd100ab8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd15c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd100a7060 0x7efd100ab8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35326/0 (socket says 172.21.15.27:35326) 2024-03-28T19:21:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd15c00640 1 -- 172.21.15.27:0/3737161257 learned_addr learned my addr 172.21.15.27:0/3737161257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd15200640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd100c2530 0x7efd1014bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.021+0000 7efd16600640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd10152ad0 0x7efd1014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd15c00640 1 -- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd100c2530 msgr2=0x7efd1014bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd15c00640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd100c2530 0x7efd1014bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd15c00640 1 -- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd10152ad0 msgr2=0x7efd1014c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd15c00640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd10152ad0 0x7efd1014c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd15c00640 1 -- 172.21.15.27:0/3737161257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10154db0 con 0x7efd100a7060 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd15200640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd100c2530 0x7efd1014bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd16600640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd10152ad0 0x7efd1014c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:21:00.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.022+0000 7efd15c00640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd100a7060 0x7efd100ab8d0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c430 tx=0x7efd0400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.023+0000 7efd0d600640 1 -- 172.21.15.27:0/3737161257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd040052d0 con 0x7efd100a7060 2024-03-28T19:21:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.023+0000 7efd0d600640 1 -- 172.21.15.27:0/3737161257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04005490 con 0x7efd100a7060 2024-03-28T19:21:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.023+0000 7efd0d600640 1 -- 172.21.15.27:0/3737161257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04007260 con 0x7efd100a7060 2024-03-28T19:21:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.023+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd10155820 con 0x7efd100a7060 2024-03-28T19:21:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.023+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd100a8180 con 0x7efd100a7060 2024-03-28T19:21:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.025+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd10069640 con 0x7efd100a7060 2024-03-28T19:21:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.025+0000 7efd0d600640 1 -- 172.21.15.27:0/3737161257 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efd04007400 con 0x7efd100a7060 2024-03-28T19:21:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.026+0000 7efd0d600640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efcf006bc00 0x7efcf006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.026+0000 7efd0d600640 1 -- 172.21.15.27:0/3737161257 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7efd0408af20 con 0x7efd100a7060 2024-03-28T19:21:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.027+0000 7efd15200640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efcf006bc00 0x7efcf006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.027+0000 7efd15200640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efcf006bc00 0x7efcf006e0c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7efcfc038000 tx=0x7efcfc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:00.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.032+0000 7efd0d600640 1 -- 172.21.15.27:0/3737161257 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efd04058ac0 con 0x7efd100a7060 2024-03-28T19:21:00.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:00.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.325+0000 7f64dbe00640 1 Processor -- start 2024-03-28T19:21:00.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.325+0000 7f64dbe00640 1 -- start start 2024-03-28T19:21:00.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.327+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 0x7f64dc0a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.327+0000 7f64dbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f64dc153a30 0x7f64dc153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.327+0000 7f64dbe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc150ec0 con 0x7f64dc0a61a0 2024-03-28T19:21:00.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.327+0000 7f64dbe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64dc06dc70 con 0x7f64dc153a30 2024-03-28T19:21:00.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.327+0000 7f64dbe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc153550 con 0x7f64dc150ae0 2024-03-28T19:21:00.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.328+0000 7f64db400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f64dc153a30 0x7f64dc153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.328+0000 7f64db400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f64dc153a30 0x7f64dc153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35330/0 (socket says 172.21.15.27:35330) 2024-03-28T19:21:00.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.328+0000 7f64db400640 1 -- 172.21.15.27:0/3941333384 learned_addr learned my addr 172.21.15.27:0/3941333384 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:00.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.328+0000 7f64daa00640 1 --2- 172.21.15.27:0/3941333384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 0x7f64dc0a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.328+0000 7f64db400640 1 -- 172.21.15.27:0/3941333384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 msgr2=0x7f64dc0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:00.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.328+0000 7f64db400640 1 --2- 172.21.15.27:0/3941333384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 0x7f64dc0a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.328+0000 7f64db400640 1 -- 172.21.15.27:0/3941333384 >> v1:172.21.15.155:6789/0 conn(0x7f64dc0a61a0 legacy=0x7f64dc0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.328+0000 7f64db400640 1 -- 172.21.15.27:0/3941333384 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc06df10 con 0x7f64dc153a30 2024-03-28T19:21:00.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.330+0000 7f64db400640 1 --2- 172.21.15.27:0/3941333384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc153ea0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009e50 tx=0x7f64d402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d365eb4942e00914 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:00.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.330+0000 7f64d9600640 1 -- 172.21.15.27:0/3941333384 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d402ca80 con 0x7f64dc153a30 2024-03-28T19:21:00.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.330+0000 7f64d9600640 1 -- 172.21.15.27:0/3941333384 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d402cc20 con 0x7f64dc153a30 2024-03-28T19:21:00.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.330+0000 7f64d9600640 1 -- 172.21.15.27:0/3941333384 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d40343f0 con 0x7f64dc153a30 2024-03-28T19:21:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.331+0000 7f64dbe00640 1 -- 172.21.15.27:0/3941333384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 msgr2=0x7f64dc153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.331+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3941333384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc153ea0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009e50 tx=0x7f64d402bee0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.332+0000 7f64dbe00640 1 -- 172.21.15.27:0/3941333384 shutdown_connections 2024-03-28T19:21:00.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.332+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3941333384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc153ea0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.332+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3941333384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 0x7f64dc0a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.332+0000 7f64dbe00640 1 -- 172.21.15.27:0/3941333384 >> 172.21.15.27:0/3941333384 conn(0x7f64dc06b0c0 msgr2=0x7f64dc06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:00.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.332+0000 7f64dbe00640 1 -- 172.21.15.27:0/3941333384 shutdown_connections 2024-03-28T19:21:00.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.332+0000 7f64dbe00640 1 -- 172.21.15.27:0/3941333384 wait complete. 2024-03-28T19:21:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.334+0000 7f64dbe00640 1 Processor -- start 2024-03-28T19:21:00.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.334+0000 7f64dbe00640 1 -- start start 2024-03-28T19:21:00.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc0a61a0 0x7f64dc0bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 0x7f64dc0ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc1eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64dbe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc06e7b0 con 0x7f64dc153a30 2024-03-28T19:21:00.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64dbe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc06e4d0 con 0x7f64dc0a61a0 2024-03-28T19:21:00.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64dbe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc0c70b0 con 0x7f64dc150ae0 2024-03-28T19:21:00.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc1eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc1eb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35338/0 (socket says 172.21.15.27:35338) 2024-03-28T19:21:00.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64db400640 1 -- 172.21.15.27:0/3018665071 learned_addr learned my addr 172.21.15.27:0/3018665071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:00.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64da000640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 0x7f64dc0ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.335+0000 7f64daa00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc0a61a0 0x7f64dc0bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.336+0000 7f64db400640 1 -- 172.21.15.27:0/3018665071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 msgr2=0x7f64dc0ab4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.336+0000 7f64db400640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 0x7f64dc0ab4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.336+0000 7f64db400640 1 -- 172.21.15.27:0/3018665071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc0a61a0 msgr2=0x7f64dc0bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.336+0000 7f64db400640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc0a61a0 0x7f64dc0bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.336+0000 7f64db400640 1 -- 172.21.15.27:0/3018665071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc070f60 con 0x7f64dc153a30 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.336+0000 7f64db400640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc1eb240 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009e50 tx=0x7f64d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.336+0000 7f64d2a00640 1 -- 172.21.15.27:0/3018665071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4043020 con 0x7f64dc153a30 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.337+0000 7f64d2a00640 1 -- 172.21.15.27:0/3018665071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d403e450 con 0x7f64dc153a30 2024-03-28T19:21:00.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.337+0000 7f64d2a00640 1 -- 172.21.15.27:0/3018665071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4009510 con 0x7f64dc153a30 2024-03-28T19:21:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.337+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64dc1ee860 con 0x7f64dc153a30 2024-03-28T19:21:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.337+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64dc1eee20 con 0x7f64dc153a30 2024-03-28T19:21:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.338+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64dc067dd0 con 0x7f64dc153a30 2024-03-28T19:21:00.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.339+0000 7f64d2a00640 1 -- 172.21.15.27:0/3018665071 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f64d4046020 con 0x7f64dc153a30 2024-03-28T19:21:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.340+0000 7f64d2a00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f64bc06bbd0 0x7f64bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.340+0000 7f64d2a00640 1 -- 172.21.15.27:0/3018665071 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f64d40ae0f0 con 0x7f64dc153a30 2024-03-28T19:21:00.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.340+0000 7f64daa00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f64bc06bbd0 0x7f64bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:00.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.341+0000 7f64daa00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f64bc06bbd0 0x7f64bc06e090 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ad0 tx=0x7f64cc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:00.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.346+0000 7f64d2a00640 1 -- 172.21.15.27:0/3018665071 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f64d404d030 con 0x7f64dc153a30 2024-03-28T19:21:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.354+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd100c3e80 con 0x7efd100a7060 2024-03-28T19:21:00.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.355+0000 7efd0d600640 1 -- 172.21.15.27:0/3737161257 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5468 (secure 0 0 0) 0x7efd040583c0 con 0x7efd100a7060 2024-03-28T19:21:00.354 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:00.354 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:00.355 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 192 2024-03-28T19:21:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efcf006bc00 msgr2=0x7efcf006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efcf006bc00 0x7efcf006e0c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7efcfc038000 tx=0x7efcfc008040 comp rx=0 tx=0).stop 2024-03-28T19:21:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd100a7060 msgr2=0x7efd100ab8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd100a7060 0x7efd100ab8d0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c430 tx=0x7efd0400c8f0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 shutdown_connections 2024-03-28T19:21:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efcf006bc00 0x7efcf006e0c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd10152ad0 0x7efd1014c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd100c2530 0x7efd1014bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 --2- 172.21.15.27:0/3737161257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd100a7060 0x7efd100ab8d0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 >> 172.21.15.27:0/3737161257 conn(0x7efd100c7aa0 msgr2=0x7efd10066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 shutdown_connections 2024-03-28T19:21:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.358+0000 7efd17000640 1 -- 172.21.15.27:0/3737161257 wait complete. 2024-03-28T19:21:00.379 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-03-28T19:21:00.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.642+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64dc0c13f0 con 0x7f64dc153a30 2024-03-28T19:21:00.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.643+0000 7f64d2a00640 1 -- 172.21.15.27:0/3018665071 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f64d407bcb0 con 0x7f64dc153a30 2024-03-28T19:21:00.643 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:00.643 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:18:54.625050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6269":{"gid":6269,"name":"a","rank":0,"incarnation":180,"state":"up:active","state_seq":26,"addr":"172.21.15.27:6835/1132521215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1132521215},{"type":"v1","addr":"172.21.15.27:6835","nonce":1132521215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6269,"qdb_cluster":[6269]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:00.643 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 192 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.647+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f64bc06bbd0 msgr2=0x7f64bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.647+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f64bc06bbd0 0x7f64bc06e090 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ad0 tx=0x7f64cc008040 comp rx=0 tx=0).stop 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.647+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 msgr2=0x7f64dc1eb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.647+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc1eb240 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009e50 tx=0x7f64d4004a40 comp rx=0 tx=0).stop 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.648+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 shutdown_connections 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.648+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f64bc06bbd0 0x7f64bc06e090 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.648+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64dc153a30 0x7f64dc1eb240 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.648+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64dc150ae0 0x7f64dc0ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.648+0000 7f64dbe00640 1 --2- 172.21.15.27:0/3018665071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc0a61a0 0x7f64dc0bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.648+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 >> 172.21.15.27:0/3018665071 conn(0x7f64dc06b0c0 msgr2=0x7f64dc155720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.648+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 shutdown_connections 2024-03-28T19:21:00.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:00.648+0000 7f64dbe00640 1 -- 172.21.15.27:0/3018665071 wait complete. 2024-03-28T19:21:00.668 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:21:02.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.617+0000 7f7e52a00640 1 Processor -- start 2024-03-28T19:21:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.617+0000 7f7e52a00640 1 -- start start 2024-03-28T19:21:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.619+0000 7f7e52a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e4c154140 0x7f7e4c0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.619+0000 7f7e52a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c150ec0 con 0x7f7e4c150ae0 2024-03-28T19:21:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.619+0000 7f7e52a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c06dc70 con 0x7f7e4c0c69a0 2024-03-28T19:21:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.619+0000 7f7e52a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c153550 con 0x7f7e4c154140 2024-03-28T19:21:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.619+0000 7f7e51600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e4c150ae0 0x7f7e4c0a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41004/0 (socket says 172.21.15.27:41004) 2024-03-28T19:21:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.619+0000 7f7e51600640 1 -- 172.21.15.27:0/1367950264 learned_addr learned my addr 172.21.15.27:0/1367950264 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:21:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.619+0000 7f7e52000640 1 --2- 172.21.15.27:0/1367950264 >> v2:172.21.15.27:3300/0 conn(0x7f7e4c154140 0x7f7e4c0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.619+0000 7f7e4be00640 1 -- 172.21.15.27:0/1367950264 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475415686 0 0) 0x7f7e40001110 con 0x7f7e4c150ae0 2024-03-28T19:21:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.620+0000 7f7e4be00640 1 -- 172.21.15.27:0/1367950264 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e34003520 con 0x7f7e4c150ae0 2024-03-28T19:21:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.620+0000 7f7e52000640 1 -- 172.21.15.27:0/1367950264 >> v1:172.21.15.155:6789/0 conn(0x7f7e4c0c69a0 legacy=0x7f7e4c153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.620+0000 7f7e52000640 1 -- 172.21.15.27:0/1367950264 >> v1:172.21.15.27:6789/0 conn(0x7f7e4c150ae0 legacy=0x7f7e4c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.620+0000 7f7e52000640 1 -- 172.21.15.27:0/1367950264 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c06df10 con 0x7f7e4c154140 2024-03-28T19:21:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.621+0000 7f7e52000640 1 --2- 172.21.15.27:0/1367950264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c154140 0x7f7e4c0c6e30 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009d90 tx=0x7f7e4402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292cfe60ed2dd893 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.621+0000 7f7e4be00640 1 -- 172.21.15.27:0/1367950264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e44039070 con 0x7f7e4c154140 2024-03-28T19:21:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.621+0000 7f7e4be00640 1 -- 172.21.15.27:0/1367950264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4402cbd0 con 0x7f7e4c154140 2024-03-28T19:21:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.621+0000 7f7e4be00640 1 -- 172.21.15.27:0/1367950264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e44034440 con 0x7f7e4c154140 2024-03-28T19:21:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.622+0000 7f7e52a00640 1 -- 172.21.15.27:0/1367950264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c154140 msgr2=0x7f7e4c0c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.622+0000 7f7e52a00640 1 --2- 172.21.15.27:0/1367950264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c154140 0x7f7e4c0c6e30 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009d90 tx=0x7f7e4402be20 comp rx=0 tx=0).stop 2024-03-28T19:21:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.622+0000 7f7e52a00640 1 -- 172.21.15.27:0/1367950264 shutdown_connections 2024-03-28T19:21:02.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.622+0000 7f7e52a00640 1 --2- 172.21.15.27:0/1367950264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c154140 0x7f7e4c0c6e30 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.622+0000 7f7e52a00640 1 -- 172.21.15.27:0/1367950264 >> 172.21.15.27:0/1367950264 conn(0x7f7e4c06b0c0 msgr2=0x7f7e4c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:02.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.622+0000 7f7e52a00640 1 -- 172.21.15.27:0/1367950264 shutdown_connections 2024-03-28T19:21:02.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.623+0000 7f7e52a00640 1 -- 172.21.15.27:0/1367950264 wait complete. 2024-03-28T19:21:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.624+0000 7f7e52a00640 1 Processor -- start 2024-03-28T19:21:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.625+0000 7f7e52a00640 1 -- start start 2024-03-28T19:21:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.625+0000 7f7e52a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e4c0c69a0 0x7f7e4c0bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.625+0000 7f7e52a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c150ae0 0x7f7e4c0a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e52a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e4c154140 0x7f7e4c0a9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e52a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c06e7b0 con 0x7f7e4c150ae0 2024-03-28T19:21:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e52a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c06e4d0 con 0x7f7e4c0c69a0 2024-03-28T19:21:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e52a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0a6370 con 0x7f7e4c154140 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e50c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c150ae0 0x7f7e4c0a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e50c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c150ae0 0x7f7e4c0a9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35350/0 (socket says 172.21.15.27:35350) 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e50c00640 1 -- 172.21.15.27:0/3002238516 learned_addr learned my addr 172.21.15.27:0/3002238516 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e51600640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e4c0c69a0 0x7f7e4c0bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e52000640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e4c154140 0x7f7e4c0a9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e50c00640 1 -- 172.21.15.27:0/3002238516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e4c154140 msgr2=0x7f7e4c0a9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e50c00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e4c154140 0x7f7e4c0a9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e50c00640 1 -- 172.21.15.27:0/3002238516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e4c0c69a0 msgr2=0x7f7e4c0bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e50c00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e4c0c69a0 0x7f7e4c0bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.626+0000 7f7e50c00640 1 -- 172.21.15.27:0/3002238516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c152920 con 0x7f7e4c150ae0 2024-03-28T19:21:02.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.627+0000 7f7e50c00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c150ae0 0x7f7e4c0a9aa0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800c460 tx=0x7f7e3800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:02.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.627+0000 7f7e49600640 1 -- 172.21.15.27:0/3002238516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38016020 con 0x7f7e4c150ae0 2024-03-28T19:21:02.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.627+0000 7f7e49600640 1 -- 172.21.15.27:0/3002238516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e38012870 con 0x7f7e4c150ae0 2024-03-28T19:21:02.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.627+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c1ebc10 con 0x7f7e4c150ae0 2024-03-28T19:21:02.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.627+0000 7f7e49600640 1 -- 172.21.15.27:0/3002238516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38012c60 con 0x7f7e4c150ae0 2024-03-28T19:21:02.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.627+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c0c04d0 con 0x7f7e4c150ae0 2024-03-28T19:21:02.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.629+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4c067dd0 con 0x7f7e4c150ae0 2024-03-28T19:21:02.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.630+0000 7f7e49600640 1 -- 172.21.15.27:0/3002238516 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7e38019020 con 0x7f7e4c150ae0 2024-03-28T19:21:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.632+0000 7f7e49600640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e2c06bbd0 0x7f7e2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.632+0000 7f7e49600640 1 -- 172.21.15.27:0/3002238516 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f7e3808b6c0 con 0x7f7e4c150ae0 2024-03-28T19:21:02.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.632+0000 7f7e51600640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e2c06bbd0 0x7f7e2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.633+0000 7f7e51600640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e2c06bbd0 0x7f7e2c06e090 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7e40005880 tx=0x7f7e40009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:02.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.638+0000 7f7e49600640 1 -- 172.21.15.27:0/3002238516 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7e38020030 con 0x7f7e4c150ae0 2024-03-28T19:21:02.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:02.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.949+0000 7fe413200640 1 Processor -- start 2024-03-28T19:21:02.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.950+0000 7fe413200640 1 -- start start 2024-03-28T19:21:02.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.951+0000 7fe413200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe40c0c8ed0 0x7fe40c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.951+0000 7fe413200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c158bd0 con 0x7fe40c0be5d0 2024-03-28T19:21:02.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.951+0000 7fe413200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c158eb0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.951+0000 7fe413200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40c159190 con 0x7fe40c0c8ed0 2024-03-28T19:21:02.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.951+0000 7fe412800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe40c0be5d0 0x7fe40c155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41016/0 (socket says 172.21.15.27:41016) 2024-03-28T19:21:02.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.951+0000 7fe412800640 1 -- 172.21.15.27:0/3239523267 learned_addr learned my addr 172.21.15.27:0/3239523267 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:21:02.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.951+0000 7fe411400640 1 --2- 172.21.15.27:0/3239523267 >> v2:172.21.15.155:3300/0 conn(0x7fe40c0c8ed0 0x7fe40c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.952+0000 7fe410a00640 1 -- 172.21.15.27:0/3239523267 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353498679 0 0) 0x7fe40c158bd0 con 0x7fe40c0be5d0 2024-03-28T19:21:02.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.952+0000 7fe410a00640 1 -- 172.21.15.27:0/3239523267 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f4003520 con 0x7fe40c0be5d0 2024-03-28T19:21:02.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.952+0000 7fe410a00640 1 -- 172.21.15.27:0/3239523267 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164978150 0 0) 0x7fe40c158eb0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.952+0000 7fe410a00640 1 -- 172.21.15.27:0/3239523267 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe40c158bd0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.952+0000 7fe411400640 1 -- 172.21.15.27:0/3239523267 >> v1:172.21.15.155:6789/0 conn(0x7fe40c0c58c0 legacy=0x7fe40c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.952+0000 7fe411400640 1 -- 172.21.15.27:0/3239523267 >> v1:172.21.15.27:6789/0 conn(0x7fe40c0be5d0 legacy=0x7fe40c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.952+0000 7fe411400640 1 -- 172.21.15.27:0/3239523267 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c159470 con 0x7fe40c0c8ed0 2024-03-28T19:21:02.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.954+0000 7fe411400640 1 --2- 172.21.15.27:0/3239523267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0c8ed0 0x7fe40c0c5ca0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0097a0 tx=0x7fe3fc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95f6218acc52388c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:02.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.954+0000 7fe410a00640 1 -- 172.21.15.27:0/3239523267 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc0315a0 con 0x7fe40c0c8ed0 2024-03-28T19:21:02.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.954+0000 7fe410a00640 1 -- 172.21.15.27:0/3239523267 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc037070 con 0x7fe40c0c8ed0 2024-03-28T19:21:02.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.954+0000 7fe410a00640 1 -- 172.21.15.27:0/3239523267 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc031940 con 0x7fe40c0c8ed0 2024-03-28T19:21:02.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.955+0000 7fe413200640 1 -- 172.21.15.27:0/3239523267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0c8ed0 msgr2=0x7fe40c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.955+0000 7fe413200640 1 --2- 172.21.15.27:0/3239523267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0c8ed0 0x7fe40c0c5ca0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0097a0 tx=0x7fe3fc02b840 comp rx=0 tx=0).stop 2024-03-28T19:21:02.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.955+0000 7fe413200640 1 -- 172.21.15.27:0/3239523267 shutdown_connections 2024-03-28T19:21:02.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.955+0000 7fe413200640 1 --2- 172.21.15.27:0/3239523267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0c8ed0 0x7fe40c0c5ca0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.955+0000 7fe413200640 1 -- 172.21.15.27:0/3239523267 >> 172.21.15.27:0/3239523267 conn(0x7fe40c06b6f0 msgr2=0x7fe40c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:02.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.955+0000 7fe413200640 1 -- 172.21.15.27:0/3239523267 shutdown_connections 2024-03-28T19:21:02.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.955+0000 7fe413200640 1 -- 172.21.15.27:0/3239523267 wait complete. 2024-03-28T19:21:02.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.958+0000 7fe413200640 1 Processor -- start 2024-03-28T19:21:02.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.958+0000 7fe413200640 1 -- start start 2024-03-28T19:21:02.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.959+0000 7fe413200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0be5d0 0x7fe40c0af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.959+0000 7fe413200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40c0c58c0 0x7fe40c152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.959+0000 7fe413200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe40c0c8ed0 0x7fe40c151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.959+0000 7fe413200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0b09a0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.959+0000 7fe413200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0b0820 con 0x7fe40c0be5d0 2024-03-28T19:21:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.959+0000 7fe413200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0ad7d0 con 0x7fe40c0c8ed0 2024-03-28T19:21:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.960+0000 7fe411400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40c0c58c0 0x7fe40c152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.960+0000 7fe411400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40c0c58c0 0x7fe40c152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35366/0 (socket says 172.21.15.27:35366) 2024-03-28T19:21:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.960+0000 7fe411400640 1 -- 172.21.15.27:0/2880564014 learned_addr learned my addr 172.21.15.27:0/2880564014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.960+0000 7fe411e00640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0be5d0 0x7fe40c0af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.960+0000 7fe412800640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe40c0c8ed0 0x7fe40c151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.961+0000 7fe411400640 1 -- 172.21.15.27:0/2880564014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe40c0c8ed0 msgr2=0x7fe40c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.961+0000 7fe411400640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe40c0c8ed0 0x7fe40c151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.961+0000 7fe411400640 1 -- 172.21.15.27:0/2880564014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0be5d0 msgr2=0x7fe40c0af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.961+0000 7fe411400640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0be5d0 0x7fe40c0af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.961+0000 7fe411400640 1 -- 172.21.15.27:0/2880564014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c1ebee0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.962+0000 7fe411400640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40c0c58c0 0x7fe40c152320 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098c0 tx=0x7fe3fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:02.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.962+0000 7fe406000640 1 -- 172.21.15.27:0/2880564014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc006e10 con 0x7fe40c0c58c0 2024-03-28T19:21:02.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.962+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c1ec1c0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.963+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c0af6f0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.963+0000 7fe406000640 1 -- 172.21.15.27:0/2880564014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc0044c0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.963+0000 7fe406000640 1 -- 172.21.15.27:0/2880564014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc0319a0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.965+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e8005290 con 0x7fe40c0c58c0 2024-03-28T19:21:02.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.966+0000 7fe406000640 1 -- 172.21.15.27:0/2880564014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe3fc031b40 con 0x7fe40c0c58c0 2024-03-28T19:21:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.967+0000 7fe406000640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3f006bbd0 0x7fe3f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.967+0000 7fe406000640 1 -- 172.21.15.27:0/2880564014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7fe3fc02f030 con 0x7fe40c0c58c0 2024-03-28T19:21:02.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.969+0000 7fe411e00640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3f006bbd0 0x7fe3f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:02.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.970+0000 7fe411e00640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3f006bbd0 0x7fe3f006e090 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe4000011e0 tx=0x7fe400009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:02.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.974+0000 7fe406000640 1 -- 172.21.15.27:0/2880564014 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe3fc07afc0 con 0x7fe40c0c58c0 2024-03-28T19:21:02.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.979+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e4c069640 con 0x7f7e4c150ae0 2024-03-28T19:21:02.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.980+0000 7f7e49600640 1 -- 172.21.15.27:0/3002238516 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f7e380591a0 con 0x7f7e4c150ae0 2024-03-28T19:21:02.980 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:02.980 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:00.752239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:02.981 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 194 2024-03-28T19:21:02.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e2c06bbd0 msgr2=0x7f7e2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e2c06bbd0 0x7f7e2c06e090 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7e40005880 tx=0x7f7e40009040 comp rx=0 tx=0).stop 2024-03-28T19:21:02.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c150ae0 msgr2=0x7f7e4c0a9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:02.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c150ae0 0x7f7e4c0a9aa0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800c460 tx=0x7f7e3800c920 comp rx=0 tx=0).stop 2024-03-28T19:21:02.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 shutdown_connections 2024-03-28T19:21:02.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e2c06bbd0 0x7f7e2c06e090 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e4c154140 0x7f7e4c0a9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e4c150ae0 0x7f7e4c0a9aa0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 --2- 172.21.15.27:0/3002238516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e4c0c69a0 0x7f7e4c0bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.985+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 >> 172.21.15.27:0/3002238516 conn(0x7f7e4c06b0c0 msgr2=0x7f7e4c0c2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.986+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 shutdown_connections 2024-03-28T19:21:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:02.986+0000 7f7e52a00640 1 -- 172.21.15.27:0/3002238516 wait complete. 2024-03-28T19:21:03.010 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-03-28T19:21:03.010 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2024-03-28T19:21:03.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.296+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3e8005580 con 0x7fe40c0c58c0 2024-03-28T19:21:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.297+0000 7fe406000640 1 -- 172.21.15.27:0/2880564014 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+5463 (secure 0 0 0) 0x7fe3fc07f060 con 0x7fe40c0c58c0 2024-03-28T19:21:03.296 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:03.296 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:00.752239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:03.297 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 194 2024-03-28T19:21:03.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3f006bbd0 msgr2=0x7fe3f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3f006bbd0 0x7fe3f006e090 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe4000011e0 tx=0x7fe400009040 comp rx=0 tx=0).stop 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40c0c58c0 msgr2=0x7fe40c152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40c0c58c0 0x7fe40c152320 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098c0 tx=0x7fe3fc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 shutdown_connections 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3f006bbd0 0x7fe3f006e090 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe40c0c8ed0 0x7fe40c151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe40c0c58c0 0x7fe40c152320 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 --2- 172.21.15.27:0/2880564014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe40c0be5d0 0x7fe40c0af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.300+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 >> 172.21.15.27:0/2880564014 conn(0x7fe40c06b6f0 msgr2=0x7fe40c072570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.301+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 shutdown_connections 2024-03-28T19:21:03.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.301+0000 7fe413200640 1 -- 172.21.15.27:0/2880564014 wait complete. 2024-03-28T19:21:03.323 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-03-28T19:21:03.323 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:21:03.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:03.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.539+0000 7f0eeca00640 1 Processor -- start 2024-03-28T19:21:03.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.539+0000 7f0eeca00640 1 -- start start 2024-03-28T19:21:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0eeca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ee8150ae0 0x7f0ee80a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0eeca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ee8153a30 0x7f0ee8153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0eeca00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee8150ec0 con 0x7f0ee80a61a0 2024-03-28T19:21:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0eeca00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee81543c0 con 0x7f0ee8150ae0 2024-03-28T19:21:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0eeca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee806dc70 con 0x7f0ee8153a30 2024-03-28T19:21:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0ee6800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ee8150ae0 0x7f0ee80a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0ee6800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ee8150ae0 0x7f0ee80a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35372/0 (socket says 172.21.15.27:35372) 2024-03-28T19:21:03.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0ee6800640 1 -- 172.21.15.27:0/1168656541 learned_addr learned my addr 172.21.15.27:0/1168656541 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.541+0000 7f0ee7200640 1 --2- 172.21.15.27:0/1168656541 >> v2:172.21.15.155:3300/0 conn(0x7f0ee8153a30 0x7f0ee8153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.542+0000 7f0ee6800640 1 -- 172.21.15.27:0/1168656541 >> v2:172.21.15.155:3300/0 conn(0x7f0ee8153a30 msgr2=0x7f0ee8153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.542+0000 7f0ee6800640 1 --2- 172.21.15.27:0/1168656541 >> v2:172.21.15.155:3300/0 conn(0x7f0ee8153a30 0x7f0ee8153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.542+0000 7f0ee6800640 1 -- 172.21.15.27:0/1168656541 >> v1:172.21.15.155:6789/0 conn(0x7f0ee80a61a0 legacy=0x7f0ee80c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:03.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.542+0000 7f0ee6800640 1 -- 172.21.15.27:0/1168656541 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8153550 con 0x7f0ee8150ae0 2024-03-28T19:21:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.543+0000 7f0ee6800640 1 --2- 172.21.15.27:0/1168656541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee80a5c60 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8073830 tx=0x7f0ed802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701d2cf25670d0d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.543+0000 7f0ee5e00640 1 -- 172.21.15.27:0/1168656541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8035020 con 0x7f0ee8150ae0 2024-03-28T19:21:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.543+0000 7f0ee5e00640 1 -- 172.21.15.27:0/1168656541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed8031870 con 0x7f0ee8150ae0 2024-03-28T19:21:03.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.543+0000 7f0ee5e00640 1 -- 172.21.15.27:0/1168656541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8031c40 con 0x7f0ee8150ae0 2024-03-28T19:21:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.544+0000 7f0eeca00640 1 -- 172.21.15.27:0/1168656541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 msgr2=0x7f0ee80a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.544+0000 7f0eeca00640 1 --2- 172.21.15.27:0/1168656541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee80a5c60 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8073830 tx=0x7f0ed802b910 comp rx=0 tx=0).stop 2024-03-28T19:21:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.544+0000 7f0eeca00640 1 -- 172.21.15.27:0/1168656541 shutdown_connections 2024-03-28T19:21:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.544+0000 7f0eeca00640 1 --2- 172.21.15.27:0/1168656541 >> v2:172.21.15.155:3300/0 conn(0x7f0ee8153a30 0x7f0ee8153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.544+0000 7f0eeca00640 1 --2- 172.21.15.27:0/1168656541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee80a5c60 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.544+0000 7f0eeca00640 1 -- 172.21.15.27:0/1168656541 >> 172.21.15.27:0/1168656541 conn(0x7f0ee806b0c0 msgr2=0x7f0ee806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.544+0000 7f0eeca00640 1 -- 172.21.15.27:0/1168656541 shutdown_connections 2024-03-28T19:21:03.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.544+0000 7f0eeca00640 1 -- 172.21.15.27:0/1168656541 wait complete. 2024-03-28T19:21:03.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.547+0000 7f0eeca00640 1 Processor -- start 2024-03-28T19:21:03.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.547+0000 7f0eeca00640 1 -- start start 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.548+0000 7f0eeca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ee80a61a0 0x7f0ee81eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.548+0000 7f0eeca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee8151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.548+0000 7f0eeca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ee8153a30 0x7f0ee80ab060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.548+0000 7f0eeca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee806e4d0 con 0x7f0ee8150ae0 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.548+0000 7f0eeca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee806e1f0 con 0x7f0ee8153a30 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.548+0000 7f0eeca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee806e7b0 con 0x7f0ee80a61a0 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.549+0000 7f0edfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee8151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.549+0000 7f0edfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee8151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35382/0 (socket says 172.21.15.27:35382) 2024-03-28T19:21:03.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.549+0000 7f0edfe00640 1 -- 172.21.15.27:0/804810021 learned_addr learned my addr 172.21.15.27:0/804810021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:03.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.549+0000 7f0ee7200640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ee8153a30 0x7f0ee80ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:03.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.549+0000 7f0ee6800640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ee80a61a0 0x7f0ee81eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:03.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0edfe00640 1 -- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ee80a61a0 msgr2=0x7f0ee81eb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:03.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0edfe00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ee80a61a0 0x7f0ee81eb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0edfe00640 1 -- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ee8153a30 msgr2=0x7f0ee80ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:03.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0edfe00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ee8153a30 0x7f0ee80ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0edfe00640 1 -- 172.21.15.27:0/804810021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee81eb910 con 0x7f0ee8150ae0 2024-03-28T19:21:03.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0ee7200640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ee8153a30 0x7f0ee80ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:21:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0ee6800640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ee80a61a0 0x7f0ee81eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:21:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0edfe00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee8151ce0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000c400 tx=0x7f0ed000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.550+0000 7f0edea00640 1 -- 172.21.15.27:0/804810021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed00052d0 con 0x7f0ee8150ae0 2024-03-28T19:21:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.551+0000 7f0edea00640 1 -- 172.21.15.27:0/804810021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed0005490 con 0x7f0ee8150ae0 2024-03-28T19:21:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.551+0000 7f0edea00640 1 -- 172.21.15.27:0/804810021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0007260 con 0x7f0ee8150ae0 2024-03-28T19:21:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.551+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee80a6970 con 0x7f0ee8150ae0 2024-03-28T19:21:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.551+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee8071b40 con 0x7f0ee8150ae0 2024-03-28T19:21:03.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.553+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee8067dd0 con 0x7f0ee8150ae0 2024-03-28T19:21:03.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.554+0000 7f0edea00640 1 -- 172.21.15.27:0/804810021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0ed0007400 con 0x7f0ee8150ae0 2024-03-28T19:21:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.555+0000 7f0edea00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ec406bbd0 0x7f0ec406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.555+0000 7f0edea00640 1 -- 172.21.15.27:0/804810021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f0ed008af20 con 0x7f0ee8150ae0 2024-03-28T19:21:03.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.556+0000 7f0ee6800640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ec406bbd0 0x7f0ec406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:03.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.557+0000 7f0ee6800640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ec406bbd0 0x7f0ec406e090 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8036000 tx=0x7f0ed8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:03.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.562+0000 7f0edea00640 1 -- 172.21.15.27:0/804810021 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0ee81eb910 con 0x7f0ee8150ae0 2024-03-28T19:21:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.867+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ee8069640 con 0x7f0ee8150ae0 2024-03-28T19:21:03.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.869+0000 7f0edea00640 1 -- 172.21.15.27:0/804810021 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f0ee8150ec0 con 0x7f0ee8150ae0 2024-03-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:00.752239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:03.868 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 194 2024-03-28T19:21:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.871+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ec406bbd0 msgr2=0x7f0ec406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.871+0000 7f0eeca00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ec406bbd0 0x7f0ec406e090 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8036000 tx=0x7f0ed8008030 comp rx=0 tx=0).stop 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.871+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 msgr2=0x7f0ee8151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.871+0000 7f0eeca00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee8151ce0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000c400 tx=0x7f0ed000c8c0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.872+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 shutdown_connections 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.872+0000 7f0eeca00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0ec406bbd0 0x7f0ec406e090 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.872+0000 7f0eeca00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ee8153a30 0x7f0ee80ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.872+0000 7f0eeca00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ee8150ae0 0x7f0ee8151ce0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.872+0000 7f0eeca00640 1 --2- 172.21.15.27:0/804810021 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ee80a61a0 0x7f0ee81eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.872+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 >> 172.21.15.27:0/804810021 conn(0x7f0ee806b0c0 msgr2=0x7f0ee81569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.872+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 shutdown_connections 2024-03-28T19:21:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:03.872+0000 7f0eeca00640 1 -- 172.21.15.27:0/804810021 wait complete. 2024-03-28T19:21:03.892 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 194, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 194, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7273": { "addr": "172.21.15.155:6835/2331326731", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 2331326731, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 2331326731, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7273, "incarnation": 194, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 87, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:21:00.752239+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7273 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 192, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8052": { "addr": "172.21.15.27:6837/433723712", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 433723712, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 433723712, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8052, "incarnation": 188, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 86, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:20:51.695880+0000", "qdb_cluster": [ 8052 ], "qdb_leader": 8052, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8052 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3334295790", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3334295790, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3334295790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 193, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6467, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:21:05.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.117+0000 7f292da00640 1 Processor -- start 2024-03-28T19:21:06.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.117+0000 7f292da00640 1 -- start start 2024-03-28T19:21:06.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.118+0000 7f292da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2928150c70 0x7f29280a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:06.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.119+0000 7f292da00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280c3af0 con 0x7f29281534c0 2024-03-28T19:21:06.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.119+0000 7f292da00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2928154020 con 0x7f29280c3690 2024-03-28T19:21:06.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.119+0000 7f292da00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29280ab4e0 con 0x7f2928150c70 2024-03-28T19:21:06.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.119+0000 7f2926a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2928150c70 0x7f29280a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.119+0000 7f2927400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f29281534c0 0x7f2928153910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51274/0 (socket says 172.21.15.27:51274) 2024-03-28T19:21:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.119+0000 7f2927400640 1 -- 172.21.15.27:0/1102035146 learned_addr learned my addr 172.21.15.27:0/1102035146 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:21:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.119+0000 7f2926a00640 1 -- 172.21.15.27:0/1102035146 >> v1:172.21.15.155:6789/0 conn(0x7f29280c3690 legacy=0x7f29280c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.120+0000 7f2926a00640 1 -- 172.21.15.27:0/1102035146 >> v1:172.21.15.27:6789/0 conn(0x7f29281534c0 legacy=0x7f2928153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.120+0000 7f2926000640 1 -- 172.21.15.27:0/1102035146 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192188685 0 0) 0x7f29280c3af0 con 0x7f29281534c0 2024-03-28T19:21:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.120+0000 7f2926a00640 1 -- 172.21.15.27:0/1102035146 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280c3d90 con 0x7f2928150c70 2024-03-28T19:21:06.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.121+0000 7f2926a00640 1 --2- 172.21.15.27:0/1102035146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a9cb0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f29281541f0 tx=0x7f29180302a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b38f6ac84e707 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:06.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.121+0000 7f2926000640 1 -- 172.21.15.27:0/1102035146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291803d070 con 0x7f2928150c70 2024-03-28T19:21:06.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.121+0000 7f2926000640 1 -- 172.21.15.27:0/1102035146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2918030e50 con 0x7f2928150c70 2024-03-28T19:21:06.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.121+0000 7f2926000640 1 -- 172.21.15.27:0/1102035146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2918038860 con 0x7f2928150c70 2024-03-28T19:21:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.122+0000 7f292da00640 1 -- 172.21.15.27:0/1102035146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 msgr2=0x7f29280a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.122+0000 7f292da00640 1 --2- 172.21.15.27:0/1102035146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a9cb0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f29281541f0 tx=0x7f29180302a0 comp rx=0 tx=0).stop 2024-03-28T19:21:06.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.122+0000 7f292da00640 1 -- 172.21.15.27:0/1102035146 shutdown_connections 2024-03-28T19:21:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.122+0000 7f292da00640 1 --2- 172.21.15.27:0/1102035146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a9cb0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.122+0000 7f292da00640 1 -- 172.21.15.27:0/1102035146 >> 172.21.15.27:0/1102035146 conn(0x7f292806b9d0 msgr2=0x7f29280c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.122+0000 7f292da00640 1 -- 172.21.15.27:0/1102035146 shutdown_connections 2024-03-28T19:21:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.122+0000 7f292da00640 1 -- 172.21.15.27:0/1102035146 wait complete. 2024-03-28T19:21:06.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.124+0000 7f292da00640 1 Processor -- start 2024-03-28T19:21:06.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.124+0000 7f292da00640 1 -- start start 2024-03-28T19:21:06.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f292da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29280c3690 0x7f29281e7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:06.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f292da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:06.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f292da00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29281534c0 0x7f292806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f292da00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29280ab9f0 con 0x7f2928150c70 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f292da00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29280ac1f0 con 0x7f29280c3690 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f292da00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29280abcd0 con 0x7f29281534c0 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f291e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f291e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58026/0 (socket says 172.21.15.27:58026) 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f291e000640 1 -- 172.21.15.27:0/60127449 learned_addr learned my addr 172.21.15.27:0/60127449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f2927400640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29281534c0 0x7f292806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.125+0000 7f2926a00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29280c3690 0x7f29281e7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.126+0000 7f291e000640 1 -- 172.21.15.27:0/60127449 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29281534c0 msgr2=0x7f292806cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.126+0000 7f291e000640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29281534c0 0x7f292806cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.126+0000 7f291e000640 1 -- 172.21.15.27:0/60127449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29280c3690 msgr2=0x7f29281e7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.126+0000 7f291e000640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29280c3690 0x7f29281e7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:06.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.126+0000 7f291e000640 1 -- 172.21.15.27:0/60127449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280a8740 con 0x7f2928150c70 2024-03-28T19:21:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.126+0000 7f291e000640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a8200 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f291000c480 tx=0x7f291000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.126+0000 7f291f400640 1 -- 172.21.15.27:0/60127449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29100052d0 con 0x7f2928150c70 2024-03-28T19:21:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.127+0000 7f291f400640 1 -- 172.21.15.27:0/60127449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2910005470 con 0x7f2928150c70 2024-03-28T19:21:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.127+0000 7f291f400640 1 -- 172.21.15.27:0/60127449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910012870 con 0x7f2928150c70 2024-03-28T19:21:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.127+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29280a6590 con 0x7f2928150c70 2024-03-28T19:21:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.127+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f292806e7a0 con 0x7f2928150c70 2024-03-28T19:21:06.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.128+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f292804f400 con 0x7f2928150c70 2024-03-28T19:21:06.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.129+0000 7f291f400640 1 -- 172.21.15.27:0/60127449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2910012a10 con 0x7f2928150c70 2024-03-28T19:21:06.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.130+0000 7f291f400640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f290406bbd0 0x7f290406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:06.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.130+0000 7f291f400640 1 -- 172.21.15.27:0/60127449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f291008b520 con 0x7f2928150c70 2024-03-28T19:21:06.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.130+0000 7f2926a00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f290406bbd0 0x7f290406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:06.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.131+0000 7f2926a00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f290406bbd0 0x7f290406e090 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f291803a000 tx=0x7f2918034000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:06.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.136+0000 7f291f400640 1 -- 172.21.15.27:0/60127449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2910058a40 con 0x7f2928150c70 2024-03-28T19:21:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.389+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29280a6ab0 con 0x7f2928150c70 2024-03-28T19:21:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.391+0000 7f291f400640 1 -- 172.21.15.27:0/60127449 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f291005cae0 con 0x7f2928150c70 2024-03-28T19:21:06.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:06.391 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:00.752239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:06.391 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 194 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.393+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f290406bbd0 msgr2=0x7f290406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.393+0000 7f292da00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f290406bbd0 0x7f290406e090 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f291803a000 tx=0x7f2918034000 comp rx=0 tx=0).stop 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.393+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 msgr2=0x7f29280a8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.393+0000 7f292da00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a8200 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f291000c480 tx=0x7f291000c940 comp rx=0 tx=0).stop 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.393+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 shutdown_connections 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.394+0000 7f292da00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f290406bbd0 0x7f290406e090 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.394+0000 7f292da00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29281534c0 0x7f292806cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.394+0000 7f292da00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2928150c70 0x7f29280a8200 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.394+0000 7f292da00640 1 --2- 172.21.15.27:0/60127449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29280c3690 0x7f29281e7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.394+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 >> 172.21.15.27:0/60127449 conn(0x7f292806b9d0 msgr2=0x7f29280c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.394+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 shutdown_connections 2024-03-28T19:21:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:06.394+0000 7f292da00640 1 -- 172.21.15.27:0/60127449 wait complete. 2024-03-28T19:21:06.412 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:21:08.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.081+0000 7f5412600640 1 Processor -- start 2024-03-28T19:21:09.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.081+0000 7f5412600640 1 -- start start 2024-03-28T19:21:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.082+0000 7f5412600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f540c154140 0x7f540c0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.082+0000 7f5412600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c150ec0 con 0x7f540c0c69a0 2024-03-28T19:21:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.082+0000 7f5412600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c06dc70 con 0x7f540c150ae0 2024-03-28T19:21:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.082+0000 7f5412600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540c153550 con 0x7f540c154140 2024-03-28T19:21:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.083+0000 7f5411c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f540c154140 0x7f540c0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.083+0000 7f540be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f540c0c69a0 0x7f540c153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51276/0 (socket says 172.21.15.27:51276) 2024-03-28T19:21:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.083+0000 7f540be00640 1 -- 172.21.15.27:0/3286333795 learned_addr learned my addr 172.21.15.27:0/3286333795 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:21:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.083+0000 7f540b400640 1 -- 172.21.15.27:0/3286333795 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974528758 0 0) 0x7f53f4001110 con 0x7f540c0c69a0 2024-03-28T19:21:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.111+0000 7f540b400640 1 -- 172.21.15.27:0/3286333795 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f0003520 con 0x7f540c0c69a0 2024-03-28T19:21:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.111+0000 7f540b400640 1 -- 172.21.15.27:0/3286333795 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604353894 0 0) 0x7f540c06dc70 con 0x7f540c150ae0 2024-03-28T19:21:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.111+0000 7f540b400640 1 -- 172.21.15.27:0/3286333795 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f4001110 con 0x7f540c150ae0 2024-03-28T19:21:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.111+0000 7f5411c00640 1 -- 172.21.15.27:0/3286333795 >> v1:172.21.15.155:6789/0 conn(0x7f540c150ae0 legacy=0x7f540c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.111+0000 7f5411c00640 1 -- 172.21.15.27:0/3286333795 >> v1:172.21.15.27:6789/0 conn(0x7f540c0c69a0 legacy=0x7f540c153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:09.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.111+0000 7f5411c00640 1 -- 172.21.15.27:0/3286333795 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c06df10 con 0x7f540c154140 2024-03-28T19:21:09.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.113+0000 7f5411c00640 1 --2- 172.21.15.27:0/3286333795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c154140 0x7f540c0c6e30 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f5404009e10 tx=0x7f54040302b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d4dda806f66a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:09.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.114+0000 7f540b400640 1 -- 172.21.15.27:0/3286333795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540403d070 con 0x7f540c154140 2024-03-28T19:21:09.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.114+0000 7f540b400640 1 -- 172.21.15.27:0/3286333795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5404038640 con 0x7f540c154140 2024-03-28T19:21:09.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.114+0000 7f540b400640 1 -- 172.21.15.27:0/3286333795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5404030e60 con 0x7f540c154140 2024-03-28T19:21:09.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.114+0000 7f5412600640 1 -- 172.21.15.27:0/3286333795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c154140 msgr2=0x7f540c0c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:09.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.114+0000 7f5412600640 1 --2- 172.21.15.27:0/3286333795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c154140 0x7f540c0c6e30 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f5404009e10 tx=0x7f54040302b0 comp rx=0 tx=0).stop 2024-03-28T19:21:09.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.115+0000 7f5412600640 1 -- 172.21.15.27:0/3286333795 shutdown_connections 2024-03-28T19:21:09.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.115+0000 7f5412600640 1 --2- 172.21.15.27:0/3286333795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c154140 0x7f540c0c6e30 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:09.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.115+0000 7f5412600640 1 -- 172.21.15.27:0/3286333795 >> 172.21.15.27:0/3286333795 conn(0x7f540c06b0c0 msgr2=0x7f540c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:09.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.115+0000 7f5412600640 1 -- 172.21.15.27:0/3286333795 shutdown_connections 2024-03-28T19:21:09.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.115+0000 7f5412600640 1 -- 172.21.15.27:0/3286333795 wait complete. 2024-03-28T19:21:09.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.118+0000 7f5412600640 1 Processor -- start 2024-03-28T19:21:09.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.118+0000 7f5412600640 1 -- start start 2024-03-28T19:21:09.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.119+0000 7f5412600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c0c69a0 0x7f540c1ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:09.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.119+0000 7f5412600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f540c150ae0 0x7f540c14b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:09.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.119+0000 7f5412600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f540c154140 0x7f540c149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.119+0000 7f5412600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c06e7b0 con 0x7f540c0c69a0 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.119+0000 7f5412600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c06e4d0 con 0x7f540c150ae0 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.119+0000 7f5412600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c0a6370 con 0x7f540c154140 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f5411200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c0c69a0 0x7f540c1ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f5411200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c0c69a0 0x7f540c1ea9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58044/0 (socket says 172.21.15.27:58044) 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f5411200640 1 -- 172.21.15.27:0/2123818360 learned_addr learned my addr 172.21.15.27:0/2123818360 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f540be00640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f540c150ae0 0x7f540c14b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f5411200640 1 -- 172.21.15.27:0/2123818360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f540c154140 msgr2=0x7f540c149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f5411200640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f540c154140 0x7f540c149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f5411200640 1 -- 172.21.15.27:0/2123818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f540c150ae0 msgr2=0x7f540c14b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f5411200640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f540c150ae0 0x7f540c14b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.120+0000 7f5411200640 1 -- 172.21.15.27:0/2123818360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c0bc430 con 0x7f540c0c69a0 2024-03-28T19:21:09.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.121+0000 7f5411200640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c0c69a0 0x7f540c1ea9a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c490 tx=0x7f53fc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:09.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.121+0000 7f5408c00640 1 -- 172.21.15.27:0/2123818360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc017070 con 0x7f540c0c69a0 2024-03-28T19:21:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.121+0000 7f5408c00640 1 -- 172.21.15.27:0/2123818360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc0052d0 con 0x7f540c0c69a0 2024-03-28T19:21:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.121+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540c0bc710 con 0x7f540c0c69a0 2024-03-28T19:21:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.121+0000 7f5408c00640 1 -- 172.21.15.27:0/2123818360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc003830 con 0x7f540c0c69a0 2024-03-28T19:21:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.121+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540c072130 con 0x7f540c0c69a0 2024-03-28T19:21:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.122+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f540c067dd0 con 0x7f540c0c69a0 2024-03-28T19:21:09.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.124+0000 7f5408c00640 1 -- 172.21.15.27:0/2123818360 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f53fc01a070 con 0x7f540c0c69a0 2024-03-28T19:21:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.125+0000 7f5408c00640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53e806bbd0 0x7f53e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.126+0000 7f5408c00640 1 -- 172.21.15.27:0/2123818360 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f53fc08aeb0 con 0x7f540c0c69a0 2024-03-28T19:21:09.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.126+0000 7f540be00640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53e806bbd0 0x7f53e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.126+0000 7f540be00640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53e806bbd0 0x7f53e806e090 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f540c14bbc0 tx=0x7f53f4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.131+0000 7f5408c00640 1 -- 172.21.15.27:0/2123818360 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f53fc0589f0 con 0x7f540c0c69a0 2024-03-28T19:21:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.387+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f540c069640 con 0x7f540c0c69a0 2024-03-28T19:21:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.387+0000 7f5408c00640 1 -- 172.21.15.27:0/2123818360 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5467 (secure 0 0 0) 0x7f53fc0582f0 con 0x7f540c0c69a0 2024-03-28T19:21:09.388 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:09.388 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:08.465559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 195 2024-03-28T19:21:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.390+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53e806bbd0 msgr2=0x7f53e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.390+0000 7f5412600640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53e806bbd0 0x7f53e806e090 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f540c14bbc0 tx=0x7f53f4009040 comp rx=0 tx=0).stop 2024-03-28T19:21:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.390+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c0c69a0 msgr2=0x7f540c1ea9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:09.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.390+0000 7f5412600640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c0c69a0 0x7f540c1ea9a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c490 tx=0x7f53fc00c950 comp rx=0 tx=0).stop 2024-03-28T19:21:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.391+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 shutdown_connections 2024-03-28T19:21:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.391+0000 7f5412600640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f53e806bbd0 0x7f53e806e090 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.391+0000 7f5412600640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f540c154140 0x7f540c149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.391+0000 7f5412600640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f540c150ae0 0x7f540c14b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.391+0000 7f5412600640 1 --2- 172.21.15.27:0/2123818360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f540c0c69a0 0x7f540c1ea9a0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.391+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 >> 172.21.15.27:0/2123818360 conn(0x7f540c06b0c0 msgr2=0x7f540c0c2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:09.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.391+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 shutdown_connections 2024-03-28T19:21:09.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:09.391+0000 7f5412600640 1 -- 172.21.15.27:0/2123818360 wait complete. 2024-03-28T19:21:09.411 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:21:09.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:21:09.419 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:21:11.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:11.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.745+0000 7f0bd9000640 1 Processor -- start 2024-03-28T19:21:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.745+0000 7f0bd9000640 1 -- start start 2024-03-28T19:21:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.746+0000 7f0bd9000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0bd40c58c0 0x7f0bd40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:11.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.746+0000 7f0bd9000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c8ed0 0x7f0bd41555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.746+0000 7f0bd9000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd4156230 con 0x7f0bd40be5d0 2024-03-28T19:21:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.746+0000 7f0bd9000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd4158bd0 con 0x7f0bd40c58c0 2024-03-28T19:21:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.746+0000 7f0bd9000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4158eb0 con 0x7f0bd40c8ed0 2024-03-28T19:21:11.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.747+0000 7f0bd3400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0bd40c58c0 0x7f0bd40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:11.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.747+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c8ed0 0x7f0bd41555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:11.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.747+0000 7f0bd3400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0bd40c58c0 0x7f0bd40c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:47492/0 (socket says 172.21.15.27:47492) 2024-03-28T19:21:11.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.747+0000 7f0bd3400640 1 -- 172.21.15.27:0/4181265444 learned_addr learned my addr 172.21.15.27:0/4181265444 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:11.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.747+0000 7f0bd3e00640 1 --1- 172.21.15.27:0/4181265444 >> v1:172.21.15.27:6789/0 conn(0x7f0bd40be5d0 0x7f0bd4155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51288/0 (socket says 172.21.15.27:51288) 2024-03-28T19:21:11.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.748+0000 7f0bd3400640 1 -- 172.21.15.27:0/4181265444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c8ed0 msgr2=0x7f0bd41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:11.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.748+0000 7f0bd3400640 1 --2- 172.21.15.27:0/4181265444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c8ed0 0x7f0bd41555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:11.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.748+0000 7f0bd3400640 1 -- 172.21.15.27:0/4181265444 >> v1:172.21.15.27:6789/0 conn(0x7f0bd40be5d0 legacy=0x7f0bd4155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:11.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.748+0000 7f0bd3400640 1 -- 172.21.15.27:0/4181265444 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd4159190 con 0x7f0bd40c58c0 2024-03-28T19:21:11.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.750+0000 7f0bd3400640 1 --2- 172.21.15.27:0/4181265444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c58c0 0x7f0bd40c5ca0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009870 tx=0x7f0bc002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cba1891785dc08a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd2000640 1 -- 172.21.15.27:0/4181265444 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0035020 con 0x7f0bd40c58c0 2024-03-28T19:21:11.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd2000640 1 -- 172.21.15.27:0/4181265444 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc0031640 con 0x7f0bd40c58c0 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd2000640 1 -- 172.21.15.27:0/4181265444 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc002d040 con 0x7f0bd40c58c0 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd9000640 1 -- 172.21.15.27:0/4181265444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c58c0 msgr2=0x7f0bd40c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd9000640 1 --2- 172.21.15.27:0/4181265444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c58c0 0x7f0bd40c5ca0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009870 tx=0x7f0bc002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd9000640 1 -- 172.21.15.27:0/4181265444 shutdown_connections 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd9000640 1 --2- 172.21.15.27:0/4181265444 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c8ed0 0x7f0bd41555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd9000640 1 --2- 172.21.15.27:0/4181265444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c58c0 0x7f0bd40c5ca0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.751+0000 7f0bd9000640 1 -- 172.21.15.27:0/4181265444 >> 172.21.15.27:0/4181265444 conn(0x7f0bd406b6f0 msgr2=0x7f0bd40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.752+0000 7f0bd9000640 1 -- 172.21.15.27:0/4181265444 shutdown_connections 2024-03-28T19:21:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.752+0000 7f0bd9000640 1 -- 172.21.15.27:0/4181265444 wait complete. 2024-03-28T19:21:11.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.753+0000 7f0bd9000640 1 Processor -- start 2024-03-28T19:21:11.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.754+0000 7f0bd9000640 1 -- start start 2024-03-28T19:21:11.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.754+0000 7f0bd9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40be5d0 0x7f0bd40abc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:11.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.754+0000 7f0bd9000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c58c0 0x7f0bd40bf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:11.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.754+0000 7f0bd9000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c8ed0 0x7f0bd40c2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:11.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd9000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40b0820 con 0x7f0bd40be5d0 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd9000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4159750 con 0x7f0bd40c8ed0 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd9000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd40b09a0 con 0x7f0bd40c58c0 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40be5d0 0x7f0bd40abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40be5d0 0x7f0bd40abc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58052/0 (socket says 172.21.15.27:58052) 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3400640 1 -- 172.21.15.27:0/3618338410 learned_addr learned my addr 172.21.15.27:0/3618338410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd2a00640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c58c0 0x7f0bd40bf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3e00640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c8ed0 0x7f0bd40c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3400640 1 -- 172.21.15.27:0/3618338410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c58c0 msgr2=0x7f0bd40bf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3400640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c58c0 0x7f0bd40bf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:11.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3400640 1 -- 172.21.15.27:0/3618338410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c8ed0 msgr2=0x7f0bd40c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:11.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3400640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c8ed0 0x7f0bd40c2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:11.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.755+0000 7f0bd3400640 1 -- 172.21.15.27:0/3618338410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd40c1570 con 0x7f0bd40be5d0 2024-03-28T19:21:11.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.756+0000 7f0bd3400640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40be5d0 0x7f0bd40abc20 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000c00 tx=0x7f0bc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:11.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.756+0000 7f0bcb400640 1 -- 172.21.15.27:0/3618338410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0040070 con 0x7f0bd40be5d0 2024-03-28T19:21:11.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.756+0000 7f0bcb400640 1 -- 172.21.15.27:0/3618338410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc00399f0 con 0x7f0bd40be5d0 2024-03-28T19:21:11.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.756+0000 7f0bcb400640 1 -- 172.21.15.27:0/3618338410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0035040 con 0x7f0bd40be5d0 2024-03-28T19:21:11.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.756+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd40c1850 con 0x7f0bd40be5d0 2024-03-28T19:21:11.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.756+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd40befa0 con 0x7f0bd40be5d0 2024-03-28T19:21:11.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.758+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd4069640 con 0x7f0bd40be5d0 2024-03-28T19:21:11.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.760+0000 7f0bcb400640 1 -- 172.21.15.27:0/3618338410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0bc004b060 con 0x7f0bd40be5d0 2024-03-28T19:21:11.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.761+0000 7f0bcb400640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bb406bc00 0x7f0bb406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:11.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.761+0000 7f0bcb400640 1 -- 172.21.15.27:0/3618338410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f0bc00ad180 con 0x7f0bd40be5d0 2024-03-28T19:21:11.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.761+0000 7f0bd2a00640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bb406bc00 0x7f0bb406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:11.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.762+0000 7f0bd2a00640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bb406bc00 0x7f0bb406e0c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0bd41524a0 tx=0x7f0bc4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:11.766+0000 7f0bcb400640 1 -- 172.21.15.27:0/3618338410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0bc007ad50 con 0x7f0bd40be5d0 2024-03-28T19:21:12.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.123+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bd40c1d50 con 0x7f0bd40be5d0 2024-03-28T19:21:12.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.124+0000 7f0bcb400640 1 -- 172.21.15.27:0/3618338410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f0bc007a650 con 0x7f0bd40be5d0 2024-03-28T19:21:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:12.124 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:12.125 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 198 2024-03-28T19:21:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bb406bc00 msgr2=0x7f0bb406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:12.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bb406bc00 0x7f0bb406e0c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0bd41524a0 tx=0x7f0bc4008040 comp rx=0 tx=0).stop 2024-03-28T19:21:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40be5d0 msgr2=0x7f0bd40abc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40be5d0 0x7f0bd40abc20 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000c00 tx=0x7f0bc00046c0 comp rx=0 tx=0).stop 2024-03-28T19:21:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 shutdown_connections 2024-03-28T19:21:12.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0bb406bc00 0x7f0bb406e0c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd40c8ed0 0x7f0bd40c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd40c58c0 0x7f0bd40bf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 --2- 172.21.15.27:0/3618338410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd40be5d0 0x7f0bd40abc20 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 >> 172.21.15.27:0/3618338410 conn(0x7f0bd406b6f0 msgr2=0x7f0bd40aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 shutdown_connections 2024-03-28T19:21:12.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:12.127+0000 7f0bd9000640 1 -- 172.21.15.27:0/3618338410 wait complete. 2024-03-28T19:21:12.154 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=6269 2024-03-28T19:21:12.154 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.a 2024-03-28T19:21:14.950 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 39% (90/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 76% (174/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2024-03-28T19:21:15.424 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at a4c752df713 v0.6 2024-03-28T19:21:15.442 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.6 2024-03-28T19:21:15.484 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:21:15.484 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 7 -eq 48 ']' 2024-03-28T19:21:15.484 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.7 2024-03-28T19:21:15.484 INFO:tasks.workunit.client.0.smithi027.stdout:v0.7 2024-03-28T19:21:15.485 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.7 2024-03-28T19:21:15.485 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.7 2024-03-28T19:21:15.673 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:21:15.674+0000 7f086d000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:21:18.156 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-03-28T19:21:18.156 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:21:18.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:21:18.162 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:21:18.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:18.401 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:21:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.418+0000 7f19e5a00640 1 Processor -- start 2024-03-28T19:21:18.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.418+0000 7f19e5a00640 1 -- start start 2024-03-28T19:21:18.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.419+0000 7f19e5a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f19e0153ed0 0x7f19e006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:18.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.419+0000 7f19e5a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e00c3830 con 0x7f19e006e6f0 2024-03-28T19:21:18.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.419+0000 7f19e5a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e006d1f0 con 0x7f19e006eb80 2024-03-28T19:21:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.419+0000 7f19e5a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e00c6990 con 0x7f19e0153ed0 2024-03-28T19:21:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.420+0000 7f19dea00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f19e0153ed0 0x7f19e006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.420+0000 7f19de000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19e006e6f0 0x7f19e0159bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45144/0 (socket says 172.21.15.27:45144) 2024-03-28T19:21:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.420+0000 7f19de000640 1 -- 172.21.15.27:0/1958146505 learned_addr learned my addr 172.21.15.27:0/1958146505 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:21:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.420+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6634578 0 0) 0x7f19e006d1f0 con 0x7f19e006eb80 2024-03-28T19:21:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.420+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c4003520 con 0x7f19e006eb80 2024-03-28T19:21:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.420+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109877261 0 0) 0x7f19e00c3830 con 0x7f19e006e6f0 2024-03-28T19:21:18.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.420+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e006d1f0 con 0x7f19e006e6f0 2024-03-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170106103 0 0) 0x7f19e006d1f0 con 0x7f19e006e6f0 2024-03-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 >> v2:172.21.15.155:3300/0 conn(0x7f19e0153ed0 msgr2=0x7f19e006ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:18.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 --2- 172.21.15.27:0/1958146505 >> v2:172.21.15.155:3300/0 conn(0x7f19e0153ed0 0x7f19e006ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 >> v1:172.21.15.155:6789/0 conn(0x7f19e006eb80 legacy=0x7f19e015a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e006ef60 con 0x7f19e006e6f0 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f19cc001690 con 0x7f19e006e6f0 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 >> v1:172.21.15.27:6789/0 conn(0x7f19e006e6f0 legacy=0x7f19e0159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19c4004ba0 0x7f19c4004fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c40037a0 0x7f19c4003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c4004160 0x7f19c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00c6990 con 0x7f19c4004160 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c400c0f0 con 0x7f19c4004ba0 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c400c2f0 con 0x7f19c40037a0 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19df400640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c4004160 0x7f19c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:18.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.421+0000 7f19de000640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c40037a0 0x7f19c4003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.422+0000 7f19dea00640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19c4004ba0 0x7f19c4004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.422+0000 7f19df400640 1 -- 172.21.15.27:0/1958146505 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c40037a0 msgr2=0x7f19c4003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.422+0000 7f19df400640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c40037a0 0x7f19c4003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.422+0000 7f19df400640 1 -- 172.21.15.27:0/1958146505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19c4004ba0 msgr2=0x7f19c4004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.422+0000 7f19df400640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19c4004ba0 0x7f19c4004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.422+0000 7f19df400640 1 -- 172.21.15.27:0/1958146505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e006d1f0 con 0x7f19c4004160 2024-03-28T19:21:18.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.424+0000 7f19df400640 1 -- 172.21.15.27:0/1958146505 reap_dead start 2024-03-28T19:21:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.424+0000 7f19df400640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c4004160 0x7f19c400b9c0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f19d8002000 tx=0x7f19c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460a80821eccfa05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.425+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8001c20 con 0x7f19c4004160 2024-03-28T19:21:18.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.425+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8003b90 con 0x7f19c4004160 2024-03-28T19:21:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.425+0000 7f19dd600640 1 -- 172.21.15.27:0/1958146505 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d80050d0 con 0x7f19c4004160 2024-03-28T19:21:18.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.426+0000 7f19e5a00640 1 -- 172.21.15.27:0/1958146505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c4004160 msgr2=0x7f19c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.426+0000 7f19e5a00640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c4004160 0x7f19c400b9c0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f19d8002000 tx=0x7f19c4005830 comp rx=0 tx=0).stop 2024-03-28T19:21:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.426+0000 7f19e5a00640 1 -- 172.21.15.27:0/1958146505 shutdown_connections 2024-03-28T19:21:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.426+0000 7f19e5a00640 1 --2- 172.21.15.27:0/1958146505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c4004160 0x7f19c400b9c0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.426+0000 7f19e5a00640 1 -- 172.21.15.27:0/1958146505 >> 172.21.15.27:0/1958146505 conn(0x7f19e006b9d0 msgr2=0x7f19e00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.426+0000 7f19e5a00640 1 -- 172.21.15.27:0/1958146505 shutdown_connections 2024-03-28T19:21:18.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.426+0000 7f19e5a00640 1 -- 172.21.15.27:0/1958146505 wait complete. 2024-03-28T19:21:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.428+0000 7f19e5a00640 1 Processor -- start 2024-03-28T19:21:18.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.428+0000 7f19e5a00640 1 -- start start 2024-03-28T19:21:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.429+0000 7f19e5a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c4004160 0x7f19e0150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.429+0000 7f19e5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e015c7e0 0x7f19e0156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.429+0000 7f19e5a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e015d3a0 0x7f19e015d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.429+0000 7f19e5a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00c3db0 con 0x7f19e015d3a0 2024-03-28T19:21:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.429+0000 7f19e5a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00c3ad0 con 0x7f19e015c7e0 2024-03-28T19:21:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.429+0000 7f19e5a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00c6380 con 0x7f19c4004160 2024-03-28T19:21:18.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19df400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e015d3a0 0x7f19e015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:18.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19df400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e015d3a0 0x7f19e015d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44986/0 (socket says 172.21.15.27:44986) 2024-03-28T19:21:18.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19df400640 1 -- 172.21.15.27:0/3277845999 learned_addr learned my addr 172.21.15.27:0/3277845999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19dea00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c4004160 0x7f19e0150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19de000640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e015c7e0 0x7f19e0156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19df400640 1 -- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c4004160 msgr2=0x7f19e0150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19df400640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c4004160 0x7f19e0150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19df400640 1 -- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e015c7e0 msgr2=0x7f19e0156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19df400640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e015c7e0 0x7f19e0156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.430+0000 7f19df400640 1 -- 172.21.15.27:0/3277845999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e01f2c70 con 0x7f19e015d3a0 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.431+0000 7f19de000640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e015c7e0 0x7f19e0156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:21:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.431+0000 7f19df400640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e015d3a0 0x7f19e015d820 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f19d8005040 tx=0x7f19d80016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:18.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.431+0000 7f19d6a00640 1 -- 172.21.15.27:0/3277845999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8006d50 con 0x7f19e015d3a0 2024-03-28T19:21:18.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.431+0000 7f19d6a00640 1 -- 172.21.15.27:0/3277845999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d802c580 con 0x7f19e015d3a0 2024-03-28T19:21:18.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.431+0000 7f19d6a00640 1 -- 172.21.15.27:0/3277845999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d802c930 con 0x7f19e015d3a0 2024-03-28T19:21:18.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.431+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e015ba50 con 0x7f19e015d3a0 2024-03-28T19:21:18.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.432+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e015b1b0 con 0x7f19e015d3a0 2024-03-28T19:21:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.433+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e004f400 con 0x7f19e015d3a0 2024-03-28T19:21:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.435+0000 7f19d6a00640 1 -- 172.21.15.27:0/3277845999 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f19d802cad0 con 0x7f19e015d3a0 2024-03-28T19:21:18.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.436+0000 7f19d6a00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19c006bc00 0x7f19c006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:18.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.436+0000 7f19d6a00640 1 -- 172.21.15.27:0/3277845999 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f19d80ac560 con 0x7f19e015d3a0 2024-03-28T19:21:18.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.437+0000 7f19dea00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19c006bc00 0x7f19c006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:18.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.438+0000 7f19dea00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19c006bc00 0x7f19c006e0c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009a40 tx=0x7f19d0008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.443+0000 7f19d6a00640 1 -- 172.21.15.27:0/3277845999 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f19e01f2c70 con 0x7f19e015d3a0 2024-03-28T19:21:18.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.801+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19e015a7c0 con 0x7f19e015d3a0 2024-03-28T19:21:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.802+0000 7f19d6a00640 1 -- 172.21.15.27:0/3277845999 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f19d8079a80 con 0x7f19e015d3a0 2024-03-28T19:21:18.801 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:18.801 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 198 2024-03-28T19:21:18.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19c006bc00 msgr2=0x7f19c006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:18.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19c006bc00 0x7f19c006e0c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009a40 tx=0x7f19d0008040 comp rx=0 tx=0).stop 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e015d3a0 msgr2=0x7f19e015d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e015d3a0 0x7f19e015d820 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f19d8005040 tx=0x7f19d80016c0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 shutdown_connections 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f19c006bc00 0x7f19c006e0c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19e015d3a0 0x7f19e015d820 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e015c7e0 0x7f19e0156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 --2- 172.21.15.27:0/3277845999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19c4004160 0x7f19e0150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.805+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 >> 172.21.15.27:0/3277845999 conn(0x7f19e006b9d0 msgr2=0x7f19e00c5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.806+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 shutdown_connections 2024-03-28T19:21:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:18.806+0000 7f19e5a00640 1 -- 172.21.15.27:0/3277845999 wait complete. 2024-03-28T19:21:18.827 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:21:20.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:21:21.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.065+0000 7f695aa00640 1 Processor -- start 2024-03-28T19:21:21.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.065+0000 7f695aa00640 1 -- start start 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.066+0000 7f695aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69540c5e80 0x7f69540c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f695aa00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f69540c1f50 0x7f69540ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f695aa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540ac1c0 con 0x7f69540abd90 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f695aa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695406d650 con 0x7f69540c5e80 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f695aa00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695406e1d0 con 0x7f69540c1f50 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f6953e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69540c5e80 0x7f69540c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f6953e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69540c5e80 0x7f69540c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45000/0 (socket says 172.21.15.27:45000) 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f6953e00640 1 -- 172.21.15.27:0/2183808997 learned_addr learned my addr 172.21.15.27:0/2183808997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:21.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f6958a00640 1 --1- 172.21.15.27:0/2183808997 >> v1:172.21.15.27:6789/0 conn(0x7f69540abd90 0x7f69540aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45146/0 (socket says 172.21.15.27:45146) 2024-03-28T19:21:21.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f6953400640 1 --2- 172.21.15.27:0/2183808997 >> v2:172.21.15.155:3300/0 conn(0x7f69540c1f50 0x7f69540ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:21.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f6953e00640 1 -- 172.21.15.27:0/2183808997 >> v2:172.21.15.155:3300/0 conn(0x7f69540c1f50 msgr2=0x7f69540ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:21:21.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f6953e00640 1 --2- 172.21.15.27:0/2183808997 >> v2:172.21.15.155:3300/0 conn(0x7f69540c1f50 0x7f69540ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.067+0000 7f6953e00640 1 -- 172.21.15.27:0/2183808997 >> v1:172.21.15.27:6789/0 conn(0x7f69540abd90 legacy=0x7f69540aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:21.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.068+0000 7f6953e00640 1 -- 172.21.15.27:0/2183808997 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695406d8a0 con 0x7f69540c5e80 2024-03-28T19:21:21.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.068+0000 7f6952a00640 1 -- 172.21.15.27:0/2183808997 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888322972 0 0) 0x7f69540ac1c0 con 0x7f69540abd90 2024-03-28T19:21:21.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.069+0000 7f6953e00640 1 --2- 172.21.15.27:0/2183808997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c5e80 0x7f69540c2cb0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6954073150 tx=0x7f6944030250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78b393d8c8c71dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:21.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.069+0000 7f6952a00640 1 -- 172.21.15.27:0/2183808997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694403d070 con 0x7f69540c5e80 2024-03-28T19:21:21.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.069+0000 7f6952a00640 1 -- 172.21.15.27:0/2183808997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6944038440 con 0x7f69540c5e80 2024-03-28T19:21:21.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.069+0000 7f6952a00640 1 -- 172.21.15.27:0/2183808997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944038810 con 0x7f69540c5e80 2024-03-28T19:21:21.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.070+0000 7f695aa00640 1 -- 172.21.15.27:0/2183808997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c5e80 msgr2=0x7f69540c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:21.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.070+0000 7f695aa00640 1 --2- 172.21.15.27:0/2183808997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c5e80 0x7f69540c2cb0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f6954073150 tx=0x7f6944030250 comp rx=0 tx=0).stop 2024-03-28T19:21:21.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.070+0000 7f695aa00640 1 -- 172.21.15.27:0/2183808997 shutdown_connections 2024-03-28T19:21:21.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.070+0000 7f695aa00640 1 --2- 172.21.15.27:0/2183808997 >> v2:172.21.15.155:3300/0 conn(0x7f69540c1f50 0x7f69540ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.070+0000 7f695aa00640 1 --2- 172.21.15.27:0/2183808997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c5e80 0x7f69540c2cb0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.070+0000 7f695aa00640 1 -- 172.21.15.27:0/2183808997 >> 172.21.15.27:0/2183808997 conn(0x7f695406b9d0 msgr2=0x7f69540c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:21.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.070+0000 7f695aa00640 1 -- 172.21.15.27:0/2183808997 shutdown_connections 2024-03-28T19:21:21.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.070+0000 7f695aa00640 1 -- 172.21.15.27:0/2183808997 wait complete. 2024-03-28T19:21:21.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.073+0000 7f695aa00640 1 Processor -- start 2024-03-28T19:21:21.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.073+0000 7f695aa00640 1 -- start start 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.074+0000 7f695aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69540abd90 0x7f69541536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.074+0000 7f695aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c1f50 0x7f69541ea080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f695aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69540c5e80 0x7f69541ea5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f695aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69540c31f0 con 0x7f69540c1f50 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f695aa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695406de60 con 0x7f69540c5e80 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f695aa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69540c3370 con 0x7f69540abd90 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c1f50 0x7f69541ea080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c1f50 0x7f69541ea080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45012/0 (socket says 172.21.15.27:45012) 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953400640 1 -- 172.21.15.27:0/599978388 learned_addr learned my addr 172.21.15.27:0/599978388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:21:21.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953e00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69540abd90 0x7f69541536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:21.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6958a00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69540c5e80 0x7f69541ea5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953400640 1 -- 172.21.15.27:0/599978388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69540abd90 msgr2=0x7f69541536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953400640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69540abd90 0x7f69541536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953400640 1 -- 172.21.15.27:0/599978388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69540c5e80 msgr2=0x7f69541ea5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953400640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69540c5e80 0x7f69541ea5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.075+0000 7f6953400640 1 -- 172.21.15.27:0/599978388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6954153be0 con 0x7f69540c1f50 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.076+0000 7f6953400640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c1f50 0x7f69541ea080 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f694000d5f0 tx=0x7f694000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.076+0000 7f694be00640 1 -- 172.21.15.27:0/599978388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400125f0 con 0x7f69540c1f50 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.076+0000 7f694be00640 1 -- 172.21.15.27:0/599978388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6940019070 con 0x7f69540c1f50 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.076+0000 7f694be00640 1 -- 172.21.15.27:0/599978388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400129d0 con 0x7f69540c1f50 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.076+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69541eace0 con 0x7f69540c1f50 2024-03-28T19:21:21.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.076+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69540c0ae0 con 0x7f69540c1f50 2024-03-28T19:21:21.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.078+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695404f400 con 0x7f69540c1f50 2024-03-28T19:21:21.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.079+0000 7f694be00640 1 -- 172.21.15.27:0/599978388 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6940012b70 con 0x7f69540c1f50 2024-03-28T19:21:21.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.080+0000 7f694be00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f693406bbd0 0x7f693406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:21:21.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.080+0000 7f694be00640 1 -- 172.21.15.27:0/599978388 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f694008bd10 con 0x7f69540c1f50 2024-03-28T19:21:21.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.080+0000 7f6953e00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f693406bbd0 0x7f693406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:21:21.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.082+0000 7f6953e00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f693406bbd0 0x7f693406e090 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f694403a000 tx=0x7f6944034000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:21:21.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.086+0000 7f694be00640 1 -- 172.21.15.27:0/599978388 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6940091020 con 0x7f69540c1f50 2024-03-28T19:21:21.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.412+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6954072890 con 0x7f69540c1f50 2024-03-28T19:21:21.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.413+0000 7f694be00640 1 -- 172.21.15.27:0/599978388 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f6940059880 con 0x7f69540c1f50 2024-03-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:21:21.412 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 199 2024-03-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f693406bbd0 msgr2=0x7f693406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f693406bbd0 0x7f693406e090 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f694403a000 tx=0x7f6944034000 comp rx=0 tx=0).stop 2024-03-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c1f50 msgr2=0x7f69541ea080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c1f50 0x7f69541ea080 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f694000d5f0 tx=0x7f694000dab0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 shutdown_connections 2024-03-28T19:21:21.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f693406bbd0 0x7f693406e090 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69540c5e80 0x7f69541ea5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69540c1f50 0x7f69541ea080 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 --2- 172.21.15.27:0/599978388 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69540abd90 0x7f69541536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 >> 172.21.15.27:0/599978388 conn(0x7f695406b9d0 msgr2=0x7f6954072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 shutdown_connections 2024-03-28T19:21:21.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:21:21.416+0000 7f695aa00640 1 -- 172.21.15.27:0/599978388 wait complete. 2024-03-28T19:21:21.436 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-03-28T19:21:21.436 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2024-03-28T19:21:23.774 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 21% (44/201) Updating files: 22% (45/201) Updating files: 23% (47/201) Updating files: 24% (49/201) Updating files: 25% (51/201) Updating files: 26% (53/201) Updating files: 27% (55/201) Updating files: 28% (57/201) Updating files: 29% (59/201) Updating files: 30% (61/201) Updating files: 31% (63/201) Updating files: 32% (65/201) Updating files: 33% (67/201) Updating files: 34% (69/201) Updating files: 35% (71/201) Updating files: 36% (73/201) Updating files: 37% (75/201) Updating files: 38% (77/201) Updating files: 39% (79/201) Updating files: 40% (81/201) Updating files: 41% (83/201) Updating files: 42% (85/201) Updating files: 43% (87/201) Updating files: 44% (89/201) Updating files: 45% (91/201) Updating files: 46% (93/201) Updating files: 47% (95/201) Updating files: 48% (97/201) Updating files: 49% (99/201) Updating files: 50% (101/201) Updating files: 51% (103/201) Updating files: 52% (105/201) Updating files: 53% (107/201) Updating files: 54% (109/201) Updating files: 55% (111/201) Updating files: 56% (113/201) Updating files: 57% (115/201) Updating files: 58% (117/201) Updating files: 59% (119/201) Updating files: 60% (121/201) Updating files: 61% (123/201) Updating files: 62% (125/201) Updating files: 63% (127/201) Updating files: 64% (129/201) Updating files: 65% (131/201) Updating files: 66% (133/201) Updating files: 67% (135/201) Updating files: 67% (136/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2024-03-28T19:21:23.970 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2024-03-28T19:21:23.977 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.7 2024-03-28T19:21:24.016 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:21:24.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 8 -eq 48 ']' 2024-03-28T19:21:24.017 INFO:tasks.workunit.client.0.smithi027.stdout:v0.8 2024-03-28T19:21:24.017 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.8 2024-03-28T19:21:24.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.8 2024-03-28T19:21:24.018 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.8 2024-03-28T19:21:32.023 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 22% (51/225) Updating files: 23% (52/225) Updating files: 24% (54/225) Updating files: 25% (57/225) Updating files: 26% (59/225) Updating files: 27% (61/225) Updating files: 28% (63/225) Updating files: 29% (66/225) Updating files: 30% (68/225) Updating files: 31% (70/225) Updating files: 32% (72/225) Updating files: 33% (75/225) Updating files: 34% (77/225) Updating files: 35% (79/225) Updating files: 36% (81/225) Updating files: 37% (84/225) Updating files: 38% (86/225) Updating files: 38% (87/225) Updating files: 39% (88/225) Updating files: 40% (90/225) Updating files: 41% (93/225) Updating files: 42% (95/225) Updating files: 43% (97/225) Updating files: 44% (99/225) Updating files: 45% (102/225) Updating files: 46% (104/225) Updating files: 47% (106/225) Updating files: 48% (108/225) Updating files: 49% (111/225) Updating files: 50% (113/225) Updating files: 51% (115/225) Updating files: 52% (117/225) Updating files: 53% (120/225) Updating files: 54% (122/225) Updating files: 55% (124/225) Updating files: 56% (126/225) Updating files: 56% (128/225) Updating files: 57% (129/225) Updating files: 58% (131/225) Updating files: 59% (133/225) Updating files: 60% (135/225) Updating files: 61% (138/225) Updating files: 62% (140/225) Updating files: 63% (142/225) Updating files: 64% (144/225) Updating files: 65% (147/225) Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 74% (168/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 77% (174/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 91% (206/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2024-03-28T19:21:32.268 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2024-03-28T19:21:32.277 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.8 2024-03-28T19:21:32.367 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:21:32.367 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 9 -eq 48 ']' 2024-03-28T19:21:32.367 INFO:tasks.workunit.client.0.smithi027.stdout:v0.9 2024-03-28T19:21:32.367 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.9 2024-03-28T19:21:32.367 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.9 2024-03-28T19:21:32.367 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.9 2024-03-28T19:21:39.190 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 24% (47/193) Updating files: 25% (49/193) Updating files: 26% (51/193) Updating files: 27% (53/193) Updating files: 28% (55/193) Updating files: 29% (56/193) Updating files: 30% (58/193) Updating files: 31% (60/193) Updating files: 32% (62/193) Updating files: 33% (64/193) Updating files: 34% (66/193) Updating files: 35% (68/193) Updating files: 36% (70/193) Updating files: 37% (72/193) Updating files: 38% (74/193) Updating files: 39% (76/193) Updating files: 40% (78/193) Updating files: 41% (80/193) Updating files: 42% (82/193) Updating files: 43% (83/193) Updating files: 44% (85/193) Updating files: 45% (87/193) Updating files: 46% (89/193) Updating files: 47% (91/193) Updating files: 47% (92/193) Updating files: 48% (93/193) Updating files: 49% (95/193) Updating files: 50% (97/193) Updating files: 51% (99/193) Updating files: 52% (101/193) Updating files: 53% (103/193) Updating files: 54% (105/193) Updating files: 55% (107/193) Updating files: 56% (109/193) Updating files: 57% (111/193) Updating files: 58% (112/193) Updating files: 59% (114/193) Updating files: 60% (116/193) Updating files: 61% (118/193) Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2024-03-28T19:21:39.420 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2024-03-28T19:21:39.436 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.9 2024-03-28T19:21:39.540 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:21:39.540 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 10 -eq 48 ']' 2024-03-28T19:21:39.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.10 2024-03-28T19:21:39.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.10 2024-03-28T19:21:39.541 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.10 2024-03-28T19:21:39.541 INFO:tasks.workunit.client.0.smithi027.stdout:v0.10 2024-03-28T19:21:39.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:21:39.565 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:21:45.967 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 28% (47/164) Updating files: 29% (48/164) Updating files: 30% (50/164) Updating files: 31% (51/164) Updating files: 32% (53/164) Updating files: 33% (55/164) Updating files: 34% (56/164) Updating files: 35% (58/164) Updating files: 36% (60/164) Updating files: 37% (61/164) Updating files: 38% (63/164) Updating files: 39% (64/164) Updating files: 40% (66/164) Updating files: 41% (68/164) Updating files: 42% (69/164) Updating files: 43% (71/164) Updating files: 44% (73/164) Updating files: 45% (74/164) Updating files: 46% (76/164) Updating files: 47% (78/164) Updating files: 48% (79/164) Updating files: 49% (81/164) Updating files: 50% (82/164) Updating files: 51% (84/164) Updating files: 52% (86/164) Updating files: 53% (87/164) Updating files: 54% (89/164) Updating files: 55% (91/164) Updating files: 56% (92/164) Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2024-03-28T19:21:46.187 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 007313fe844 v0.10 2024-03-28T19:21:46.203 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.10 2024-03-28T19:21:46.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:21:46.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 11 -eq 48 ']' 2024-03-28T19:21:46.276 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.11 2024-03-28T19:21:46.276 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.11 2024-03-28T19:21:46.276 INFO:tasks.workunit.client.0.smithi027.stdout:v0.11 2024-03-28T19:21:46.276 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.11 2024-03-28T19:21:49.362 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 0% (92/13225) Updating files: 1% (133/13225) Updating files: 1% (182/13225) Updating files: 2% (265/13225) Updating files: 2% (266/13225) Updating files: 2% (351/13225) Updating files: 3% (397/13225) Updating files: 3% (440/13225) Updating files: 3% (524/13225) Updating files: 4% (529/13225) Updating files: 4% (588/13225) Updating files: 5% (662/13225) Updating files: 5% (677/13225) Updating files: 5% (762/13225) Updating files: 6% (794/13225) Updating files: 6% (852/13225) Updating files: 7% (926/13225) Updating files: 7% (938/13225) Updating files: 7% (989/13225) Updating files: 8% (1058/13225) Updating files: 8% (1072/13225) Updating files: 8% (1145/13225) Updating files: 9% (1191/13225) Updating files: 9% (1210/13225) Updating files: 9% (1289/13225) Updating files: 10% (1323/13225) Updating files: 10% (1358/13225) Updating files: 10% (1431/13225) Updating files: 11% (1455/13225) Updating files: 11% (1513/13225) Updating files: 11% (1572/13225) Updating files: 12% (1587/13225) Updating files: 12% (1634/13225) Updating files: 12% (1679/13225) Updating files: 13% (1720/13225) Updating files: 13% (1737/13225) Updating files: 13% (1797/13225) Updating files: 14% (1852/13225) Updating files: 14% (1857/13225) Updating files: 14% (1927/13225) Updating files: 15% (1984/13225) Updating files: 15% (1985/13225) Updating files: 15% (2048/13225) Updating files: 15% (2106/13225) Updating files: 16% (2116/13225) Updating files: 16% (2165/13225) Updating files: 16% (2220/13225) Updating files: 17% (2249/13225) Updating files: 17% (2274/13225) Updating files: 17% (2329/13225) Updating files: 17% (2377/13225) Updating files: 18% (2381/13225) Updating files: 18% (2431/13225) Updating files: 18% (2488/13225) Updating files: 19% (2513/13225) Updating files: 19% (2536/13225) Updating files: 19% (2590/13225) Updating files: 19% (2641/13225) Updating files: 20% (2645/13225) Updating files: 20% (2701/13225) Updating files: 20% (2756/13225) Updating files: 21% (2778/13225) Updating files: 21% (2791/13225) Updating files: 21% (2852/13225) Updating files: 21% (2909/13225) Updating files: 22% (2910/13225) Updating files: 22% (2968/13225) Updating files: 22% (3012/13225) Updating files: 22% (3038/13225) Updating files: 23% (3042/13225) Updating files: 23% (3063/13225) Updating files: 23% (3126/13225) Updating files: 24% (3174/13225) Updating files: 24% (3184/13225) Updating files: 24% (3231/13225) Updating files: 24% (3306/13225) Updating files: 25% (3307/13225) Updating files: 25% (3369/13225) Updating files: 25% (3428/13225) Updating files: 26% (3439/13225) Updating files: 26% (3505/13225) Updating files: 27% (3571/13225) Updating files: 27% (3583/13225) Updating files: 27% (3656/13225) Updating files: 28% (3703/13225) Updating files: 28% (3725/13225) Updating files: 28% (3789/13225) Updating files: 29% (3836/13225) Updating files: 29% (3858/13225) Updating files: 29% (3932/13225) Updating files: 30% (3968/13225) Updating files: 30% (4006/13225) Updating files: 30% (4061/13225) Updating files: 31% (4100/13225) Updating files: 31% (4132/13225) Updating files: 31% (4205/13225) Updating files: 32% (4232/13225) Updating files: 32% (4274/13225) Updating files: 32% (4331/13225) Updating files: 33% (4365/13225) Updating files: 33% (4389/13225) Updating files: 33% (4448/13225) Updating files: 34% (4497/13225) Updating files: 34% (4512/13225) Updating files: 34% (4582/13225) Updating files: 35% (4629/13225) Updating files: 35% (4659/13225) Updating files: 35% (4741/13225) Updating files: 36% (4761/13225) Updating files: 36% (4811/13225) Updating files: 36% (4880/13225) Updating files: 37% (4894/13225) Updating files: 37% (4951/13225) Updating files: 37% (5023/13225) Updating files: 38% (5026/13225) Updating files: 38% (5083/13225) Updating files: 39% (5158/13225) Updating files: 39% (5160/13225) Updating files: 39% (5248/13225) Updating files: 40% (5290/13225) Updating files: 40% (5321/13225) Updating files: 40% (5379/13225) Updating files: 41% (5423/13225) Updating files: 41% (5481/13225) Updating files: 42% (5555/13225) Updating files: 42% (5594/13225) Updating files: 43% (5687/13225) Updating files: 43% (5716/13225) Updating files: 44% (5819/13225) Updating files: 44% (5844/13225) Updating files: 45% (5952/13225) Updating files: 45% (5953/13225) Updating files: 45% (6048/13225) Updating files: 46% (6084/13225) Updating files: 46% (6135/13225) Updating files: 46% (6208/13225) Updating files: 47% (6216/13225) Updating files: 47% (6276/13225) Updating files: 48% (6348/13225) Updating files: 48% (6356/13225) Updating files: 48% (6429/13225) Updating files: 49% (6481/13225) Updating files: 49% (6511/13225) Updating files: 49% (6604/13225) Updating files: 50% (6613/13225) Updating files: 50% (6683/13225) Updating files: 51% (6745/13225) Updating files: 51% (6797/13225) Updating files: 52% (6877/13225) Updating files: 52% (6889/13225) Updating files: 52% (6987/13225) Updating files: 53% (7010/13225) Updating files: 53% (7100/13225) Updating files: 54% (7142/13225) Updating files: 54% (7224/13225) Updating files: 55% (7274/13225) Updating files: 55% (7351/13225) Updating files: 56% (7406/13225) Updating files: 56% (7476/13225) Updating files: 57% (7539/13225) Updating files: 57% (7598/13225) Updating files: 58% (7671/13225) Updating files: 58% (7709/13225) Updating files: 59% (7803/13225) Updating files: 59% (7828/13225) Updating files: 59% (7924/13225) Updating files: 60% (7935/13225) Updating files: 60% (7979/13225) Updating files: 61% (8068/13225) Updating files: 61% (8076/13225) Updating files: 61% (8139/13225) Updating files: 61% (8190/13225) Updating files: 62% (8200/13225) Updating files: 62% (8232/13225) Updating files: 62% (8280/13225) Updating files: 62% (8323/13225) Updating files: 63% (8332/13225) Updating files: 63% (8374/13225) Updating files: 63% (8419/13225) Updating files: 63% (8462/13225) Updating files: 64% (8464/13225) Updating files: 64% (8508/13225) Updating files: 64% (8555/13225) Updating files: 65% (8597/13225) Updating files: 65% (8609/13225) Updating files: 65% (8658/13225) Updating files: 65% (8701/13225) Updating files: 66% (8729/13225) Updating files: 66% (8742/13225) Updating files: 66% (8782/13225) Updating files: 66% (8825/13225) Updating files: 67% (8861/13225) Updating files: 67% (8867/13225) Updating files: 67% (8909/13225) Updating files: 67% (8969/13225) Updating files: 68% (8993/13225) Updating files: 68% (9033/13225) Updating files: 68% (9076/13225) Updating files: 68% (9113/13225) Updating files: 69% (9126/13225) Updating files: 69% (9152/13225) Updating files: 69% (9196/13225) Updating files: 69% (9255/13225) Updating files: 70% (9258/13225) Updating files: 70% (9307/13225) Updating files: 71% (9390/13225) Updating files: 71% (9406/13225) Updating files: 71% (9407/13225) Updating files: 71% (9456/13225) Updating files: 71% (9509/13225) Updating files: 72% (9522/13225) Updating files: 72% (9551/13225) Updating files: 72% (9602/13225) Updating files: 73% (9655/13225) Updating files: 73% (9659/13225) Updating files: 73% (9726/13225) Updating files: 73% (9769/13225) Updating files: 74% (9787/13225) Updating files: 74% (9825/13225) Updating files: 74% (9879/13225) Updating files: 75% (9919/13225) Updating files: 75% (9936/13225) Updating files: 75% (10010/13225) Updating files: 76% (10051/13225) Updating files: 76% (10079/13225) Updating files: 76% (10153/13225) Updating files: 77% (10184/13225) Updating files: 77% (10227/13225) Updating files: 77% (10310/13225) Updating files: 78% (10316/13225) Updating files: 78% (10352/13225) Updating files: 78% (10426/13225) Updating files: 79% (10448/13225) Updating files: 79% (10495/13225) Updating files: 79% (10562/13225) Updating files: 80% (10580/13225) Updating files: 80% (10634/13225) Updating files: 81% (10713/13225) Updating files: 81% (10721/13225) Updating files: 81% (10803/13225) Updating files: 82% (10845/13225) Updating files: 82% (10873/13225) Updating files: 82% (10947/13225) Updating files: 83% (10977/13225) Updating files: 83% (11020/13225) Updating files: 83% (11087/13225) Updating files: 84% (11109/13225) Updating files: 84% (11165/13225) Updating files: 84% (11215/13225) Updating files: 85% (11242/13225) Updating files: 85% (11296/13225) Updating files: 86% (11374/13225) Updating files: 86% (11375/13225) Updating files: 86% (11458/13225) Updating files: 87% (11506/13225) Updating files: 87% (11549/13225) Updating files: 87% (11618/13225) Updating files: 88% (11638/13225) Updating files: 88% (11706/13225) Updating files: 89% (11771/13225) Updating files: 89% (11790/13225) Updating files: 89% (11869/13225) Updating files: 90% (11903/13225) Updating files: 90% (11951/13225) Updating files: 91% (12035/13225) Updating files: 91% (12043/13225) Updating files: 91% (12124/13225) Updating files: 92% (12167/13225) Updating files: 92% (12198/13225) Updating files: 92% (12268/13225) Updating files: 93% (12300/13225) Updating files: 93% (12326/13225) Updating files: 93% (12384/13225) Updating files: 94% (12432/13225) Updating files: 94% (12447/13225) Updating files: 94% (12519/13225) Updating files: 95% (12564/13225) Updating files: 95% (12588/13225) Updating files: 95% (12648/13225) Updating files: 96% (12696/13225) Updating files: 96% (12712/13225) Updating files: 96% (12770/13225) Updating files: 96% (12823/13225) Updating files: 97% (12829/13225) Updating files: 97% (12883/13225) Updating files: 97% (12943/13225) Updating files: 98% (12961/13225) Updating files: 98% (12996/13225) Updating files: 98% (13047/13225) Updating files: 99% (13093/13225) Updating files: 99% (13098/13225) Updating files: 99% (13146/13225) Updating files: 99% (13204/13225) Updating files: 100% (13225/13225) Updating files: 100% (13225/13225), done. 2024-03-28T19:21:49.614 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2024-03-28T19:21:49.699 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.1 2024-03-28T19:21:49.715 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:21:49.715 INFO:tasks.workunit.client.1.smithi155.stdout:v0.2 2024-03-28T19:21:49.715 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 2 -eq 48 ']' 2024-03-28T19:21:49.715 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.2 2024-03-28T19:21:49.715 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.2 2024-03-28T19:21:49.715 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.2 2024-03-28T19:21:50.915 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 65% (45/69) Updating files: 66% (46/69) Updating files: 68% (47/69) Updating files: 69% (48/69) Updating files: 71% (49/69) Updating files: 72% (50/69) Updating files: 73% (51/69) Updating files: 75% (52/69) Updating files: 76% (53/69) Updating files: 78% (54/69) Updating files: 79% (55/69) Updating files: 81% (56/69) Updating files: 82% (57/69) Updating files: 84% (58/69) Updating files: 85% (59/69) Updating files: 86% (60/69) Updating files: 88% (61/69) Updating files: 89% (62/69) Updating files: 91% (63/69) Updating files: 92% (64/69) Updating files: 94% (65/69) Updating files: 95% (66/69) Updating files: 97% (67/69) Updating files: 98% (68/69) Updating files: 100% (69/69) Updating files: 100% (69/69), done. 2024-03-28T19:21:51.186 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2024-03-28T19:21:51.196 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.11 2024-03-28T19:21:51.274 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:21:51.274 INFO:tasks.workunit.client.0.smithi027.stdout:v0.12 2024-03-28T19:21:51.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 12 -eq 48 ']' 2024-03-28T19:21:51.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.12 2024-03-28T19:21:51.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.12 2024-03-28T19:21:51.275 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.12 2024-03-28T19:21:56.914 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 36% (27/75) Updating files: 37% (28/75) Updating files: 38% (29/75) Updating files: 40% (30/75) Updating files: 41% (31/75) Updating files: 42% (32/75) Updating files: 44% (33/75) Updating files: 45% (34/75) Updating files: 46% (35/75) Updating files: 48% (36/75) Updating files: 49% (37/75) Updating files: 50% (38/75) Updating files: 52% (39/75) Updating files: 53% (40/75) Updating files: 54% (41/75) Updating files: 56% (42/75) Updating files: 57% (43/75) Updating files: 58% (44/75) Updating files: 60% (45/75) Updating files: 61% (46/75) Updating files: 62% (47/75) Updating files: 64% (48/75) Updating files: 65% (49/75) Updating files: 66% (50/75) Updating files: 68% (51/75) Updating files: 69% (52/75) Updating files: 70% (53/75) Updating files: 72% (54/75) Updating files: 73% (55/75) Updating files: 74% (56/75) Updating files: 76% (57/75) Updating files: 77% (58/75) Updating files: 78% (59/75) Updating files: 80% (60/75) Updating files: 81% (61/75) Updating files: 82% (62/75) Updating files: 84% (63/75) Updating files: 85% (64/75) Updating files: 86% (65/75) Updating files: 88% (66/75) Updating files: 89% (67/75) Updating files: 90% (68/75) Updating files: 92% (69/75) Updating files: 93% (70/75) Updating files: 94% (71/75) Updating files: 96% (72/75) Updating files: 97% (73/75) Updating files: 98% (74/75) Updating files: 100% (75/75) Updating files: 100% (75/75), done. 2024-03-28T19:21:57.233 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at f524150822a v0.12 2024-03-28T19:21:57.244 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.12 2024-03-28T19:21:57.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:21:57.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 13 -eq 48 ']' 2024-03-28T19:21:57.300 INFO:tasks.workunit.client.0.smithi027.stdout:v0.13 2024-03-28T19:21:57.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.13 2024-03-28T19:21:57.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.13 2024-03-28T19:21:57.300 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.13 2024-03-28T19:21:59.414 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 18% (42/222) Updating files: 19% (43/222) Updating files: 20% (45/222) Updating files: 21% (47/222) Updating files: 22% (49/222) Updating files: 23% (52/222) Updating files: 24% (54/222) Updating files: 25% (56/222) Updating files: 26% (58/222) Updating files: 27% (60/222) Updating files: 28% (63/222) Updating files: 29% (65/222) Updating files: 30% (67/222) Updating files: 31% (69/222) Updating files: 32% (72/222) Updating files: 32% (73/222) Updating files: 33% (74/222) Updating files: 34% (76/222) Updating files: 35% (78/222) Updating files: 36% (80/222) Updating files: 37% (83/222) Updating files: 38% (85/222) Updating files: 39% (87/222) Updating files: 40% (89/222) Updating files: 41% (92/222) Updating files: 42% (94/222) Updating files: 43% (96/222) Updating files: 44% (98/222) Updating files: 45% (100/222) Updating files: 46% (103/222) Updating files: 47% (105/222) Updating files: 48% (107/222) Updating files: 49% (109/222) Updating files: 50% (111/222) Updating files: 50% (113/222) Updating files: 51% (114/222) Updating files: 52% (116/222) Updating files: 53% (118/222) Updating files: 54% (120/222) Updating files: 55% (123/222) Updating files: 56% (125/222) Updating files: 57% (127/222) Updating files: 58% (129/222) Updating files: 59% (131/222) Updating files: 60% (134/222) Updating files: 61% (136/222) Updating files: 62% (138/222) Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 68% (152/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2024-03-28T19:21:59.817 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 2b5f76f48e9 readme 2024-03-28T19:21:59.825 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.2 2024-03-28T19:21:59.902 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:21:59.902 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 3 -eq 48 ']' 2024-03-28T19:21:59.902 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.3 2024-03-28T19:21:59.903 INFO:tasks.workunit.client.1.smithi155.stdout:v0.3 2024-03-28T19:21:59.903 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.3 2024-03-28T19:21:59.903 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.3 2024-03-28T19:22:03.242 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 33% (45/133) Updating files: 34% (46/133) Updating files: 35% (47/133) Updating files: 36% (48/133) Updating files: 37% (50/133) Updating files: 38% (51/133) Updating files: 39% (52/133) Updating files: 40% (54/133) Updating files: 41% (55/133) Updating files: 42% (56/133) Updating files: 43% (58/133) Updating files: 44% (59/133) Updating files: 45% (60/133) Updating files: 46% (62/133) Updating files: 47% (63/133) Updating files: 48% (64/133) Updating files: 49% (66/133) Updating files: 50% (67/133) Updating files: 51% (68/133) Updating files: 52% (70/133) Updating files: 53% (71/133) Updating files: 54% (72/133) Updating files: 55% (74/133) Updating files: 56% (75/133) Updating files: 57% (76/133) Updating files: 58% (78/133) Updating files: 59% (79/133) Updating files: 60% (80/133) Updating files: 61% (82/133) Updating files: 62% (83/133) Updating files: 63% (84/133) Updating files: 64% (86/133) Updating files: 65% (87/133) Updating files: 66% (88/133) Updating files: 67% (90/133) Updating files: 68% (91/133) Updating files: 69% (92/133) Updating files: 70% (94/133) Updating files: 71% (95/133) Updating files: 72% (96/133) Updating files: 73% (98/133) Updating files: 74% (99/133) Updating files: 75% (100/133) Updating files: 76% (102/133) Updating files: 77% (103/133) Updating files: 78% (104/133) Updating files: 79% (106/133) Updating files: 80% (107/133) Updating files: 81% (108/133) Updating files: 82% (110/133) Updating files: 83% (111/133) Updating files: 84% (112/133) Updating files: 85% (114/133) Updating files: 86% (115/133) Updating files: 87% (116/133) Updating files: 88% (118/133) Updating files: 89% (119/133) Updating files: 90% (120/133) Updating files: 91% (122/133) Updating files: 92% (123/133) Updating files: 93% (124/133) Updating files: 94% (126/133) Updating files: 95% (127/133) Updating files: 96% (128/133) Updating files: 97% (130/133) Updating files: 98% (131/133) Updating files: 99% (132/133) Updating files: 100% (133/133) Updating files: 100% (133/133), done. 2024-03-28T19:22:03.457 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 2b32297267d v0.13 2024-03-28T19:22:03.464 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.13 2024-03-28T19:22:03.514 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:22:03.514 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 14 -eq 48 ']' 2024-03-28T19:22:03.514 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.14 2024-03-28T19:22:03.514 INFO:tasks.workunit.client.0.smithi027.stdout:v0.14 2024-03-28T19:22:03.514 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.14 2024-03-28T19:22:03.514 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.14 2024-03-28T19:22:09.117 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 18% (54/295) Updating files: 19% (57/295) Updating files: 20% (59/295) Updating files: 21% (62/295) Updating files: 22% (65/295) Updating files: 23% (68/295) Updating files: 24% (71/295) Updating files: 25% (74/295) Updating files: 26% (77/295) Updating files: 27% (80/295) Updating files: 28% (83/295) Updating files: 29% (86/295) Updating files: 30% (89/295) Updating files: 31% (92/295) Updating files: 31% (94/295) Updating files: 32% (95/295) Updating files: 33% (98/295) Updating files: 34% (101/295) Updating files: 35% (104/295) Updating files: 36% (107/295) Updating files: 37% (110/295) Updating files: 38% (113/295) Updating files: 39% (116/295) Updating files: 40% (118/295) Updating files: 41% (121/295) Updating files: 42% (124/295) Updating files: 43% (127/295) Updating files: 44% (130/295) Updating files: 45% (133/295) Updating files: 46% (136/295) Updating files: 47% (139/295) Updating files: 48% (142/295) Updating files: 49% (145/295) Updating files: 50% (148/295) Updating files: 51% (151/295) Updating files: 52% (154/295) Updating files: 53% (157/295) Updating files: 54% (160/295) Updating files: 55% (163/295) Updating files: 56% (166/295) Updating files: 57% (169/295) Updating files: 58% (172/295) Updating files: 59% (175/295) Updating files: 60% (177/295) Updating files: 61% (180/295) Updating files: 62% (183/295) Updating files: 63% (186/295) Updating files: 64% (189/295) Updating files: 65% (192/295) Updating files: 66% (195/295) Updating files: 67% (198/295) Updating files: 68% (201/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2024-03-28T19:22:09.432 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2024-03-28T19:22:09.464 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.3 2024-03-28T19:22:09.546 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:22:09.546 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 4 -eq 48 ']' 2024-03-28T19:22:09.546 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.4 2024-03-28T19:22:09.546 INFO:tasks.workunit.client.1.smithi155.stdout:v0.4 2024-03-28T19:22:09.546 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.4 2024-03-28T19:22:09.546 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.4 2024-03-28T19:22:10.178 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 30% (46/151) Updating files: 31% (47/151) Updating files: 32% (49/151) Updating files: 33% (50/151) Updating files: 34% (52/151) Updating files: 35% (53/151) Updating files: 36% (55/151) Updating files: 37% (56/151) Updating files: 38% (58/151) Updating files: 39% (59/151) Updating files: 40% (61/151) Updating files: 41% (62/151) Updating files: 42% (64/151) Updating files: 43% (65/151) Updating files: 44% (67/151) Updating files: 45% (68/151) Updating files: 46% (70/151) Updating files: 47% (71/151) Updating files: 48% (73/151) Updating files: 49% (74/151) Updating files: 50% (76/151) Updating files: 51% (78/151) Updating files: 52% (79/151) Updating files: 53% (81/151) Updating files: 54% (82/151) Updating files: 54% (83/151) Updating files: 55% (84/151) Updating files: 56% (85/151) Updating files: 57% (87/151) Updating files: 58% (88/151) Updating files: 59% (90/151) Updating files: 60% (91/151) Updating files: 61% (93/151) Updating files: 62% (94/151) Updating files: 63% (96/151) Updating files: 64% (97/151) Updating files: 65% (99/151) Updating files: 66% (100/151) Updating files: 67% (102/151) Updating files: 68% (103/151) Updating files: 69% (105/151) Updating files: 70% (106/151) Updating files: 71% (108/151) Updating files: 72% (109/151) Updating files: 73% (111/151) Updating files: 74% (112/151) Updating files: 75% (114/151) Updating files: 76% (115/151) Updating files: 77% (117/151) Updating files: 78% (118/151) Updating files: 79% (120/151) Updating files: 80% (121/151) Updating files: 81% (123/151) Updating files: 82% (124/151) Updating files: 83% (126/151) Updating files: 84% (127/151) Updating files: 85% (129/151) Updating files: 86% (130/151) Updating files: 87% (132/151) Updating files: 88% (133/151) Updating files: 89% (135/151) Updating files: 90% (136/151) Updating files: 91% (138/151) Updating files: 92% (139/151) Updating files: 93% (141/151) Updating files: 94% (142/151) Updating files: 95% (144/151) Updating files: 96% (145/151) Updating files: 97% (147/151) Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2024-03-28T19:22:10.609 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2024-03-28T19:22:10.624 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.14 2024-03-28T19:22:10.698 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:22:10.698 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 15 -eq 48 ']' 2024-03-28T19:22:10.698 INFO:tasks.workunit.client.0.smithi027.stdout:v0.15 2024-03-28T19:22:10.699 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.15 2024-03-28T19:22:10.699 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.15 2024-03-28T19:22:10.699 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.15 2024-03-28T19:22:11.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:22:11.503 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:22:16.247 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 41% (47/112) Updating files: 42% (48/112) Updating files: 43% (49/112) Updating files: 44% (50/112) Updating files: 45% (51/112) Updating files: 46% (52/112) Updating files: 47% (53/112) Updating files: 48% (54/112) Updating files: 49% (55/112) Updating files: 50% (56/112) Updating files: 51% (58/112) Updating files: 52% (59/112) Updating files: 53% (60/112) Updating files: 54% (61/112) Updating files: 55% (62/112) Updating files: 56% (63/112) Updating files: 57% (64/112) Updating files: 58% (65/112) Updating files: 59% (67/112) Updating files: 60% (68/112) Updating files: 61% (69/112) Updating files: 62% (70/112) Updating files: 63% (71/112) Updating files: 64% (72/112) Updating files: 65% (73/112) Updating files: 66% (74/112) Updating files: 67% (76/112) Updating files: 68% (77/112) Updating files: 69% (78/112) Updating files: 70% (79/112) Updating files: 71% (80/112) Updating files: 72% (81/112) Updating files: 73% (82/112) Updating files: 74% (83/112) Updating files: 75% (84/112) Updating files: 76% (86/112) Updating files: 77% (87/112) Updating files: 78% (88/112) Updating files: 79% (89/112) Updating files: 80% (90/112) Updating files: 81% (91/112) Updating files: 82% (92/112) Updating files: 83% (93/112) Updating files: 84% (95/112) Updating files: 85% (96/112) Updating files: 86% (97/112) Updating files: 87% (98/112) Updating files: 88% (99/112) Updating files: 89% (100/112) Updating files: 90% (101/112) Updating files: 91% (102/112) Updating files: 92% (104/112) Updating files: 93% (105/112) Updating files: 94% (106/112) Updating files: 95% (107/112) Updating files: 96% (108/112) Updating files: 97% (109/112) Updating files: 98% (110/112) Updating files: 99% (111/112) Updating files: 100% (112/112) Updating files: 100% (112/112), done. 2024-03-28T19:22:16.575 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at ab3c421eb95 v0.15 2024-03-28T19:22:16.584 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.15 2024-03-28T19:22:16.633 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:22:16.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 16 -eq 48 ']' 2024-03-28T19:22:16.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.16 2024-03-28T19:22:16.634 INFO:tasks.workunit.client.0.smithi027.stdout:v0.16 2024-03-28T19:22:16.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.16 2024-03-28T19:22:16.634 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.16 2024-03-28T19:22:17.042 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 21% (54/247) Updating files: 22% (55/247) Updating files: 23% (57/247) Updating files: 24% (60/247) Updating files: 25% (62/247) Updating files: 26% (65/247) Updating files: 27% (67/247) Updating files: 28% (70/247) Updating files: 29% (72/247) Updating files: 30% (75/247) Updating files: 31% (77/247) Updating files: 32% (80/247) Updating files: 33% (82/247) Updating files: 34% (84/247) Updating files: 35% (87/247) Updating files: 36% (89/247) Updating files: 37% (92/247) Updating files: 38% (94/247) Updating files: 39% (97/247) Updating files: 40% (99/247) Updating files: 41% (102/247) Updating files: 42% (104/247) Updating files: 43% (107/247) Updating files: 44% (109/247) Updating files: 45% (112/247) Updating files: 46% (114/247) Updating files: 47% (117/247) Updating files: 48% (119/247) Updating files: 49% (122/247) Updating files: 50% (124/247) Updating files: 51% (126/247) Updating files: 52% (129/247) Updating files: 53% (131/247) Updating files: 54% (134/247) Updating files: 55% (136/247) Updating files: 55% (137/247) Updating files: 56% (139/247) Updating files: 57% (141/247) Updating files: 58% (144/247) Updating files: 59% (146/247) Updating files: 60% (149/247) Updating files: 61% (151/247) Updating files: 62% (154/247) Updating files: 63% (156/247) Updating files: 64% (159/247) Updating files: 65% (161/247) Updating files: 66% (164/247) Updating files: 67% (166/247) Updating files: 68% (168/247) Updating files: 69% (171/247) Updating files: 70% (173/247) Updating files: 71% (176/247) Updating files: 72% (178/247) Updating files: 73% (181/247) Updating files: 74% (183/247) Updating files: 75% (186/247) Updating files: 76% (188/247) Updating files: 76% (190/247) Updating files: 77% (191/247) Updating files: 78% (193/247) Updating files: 79% (196/247) Updating files: 80% (198/247) Updating files: 81% (201/247) Updating files: 82% (203/247) Updating files: 83% (206/247) Updating files: 84% (208/247) Updating files: 85% (210/247) Updating files: 86% (213/247) Updating files: 87% (215/247) Updating files: 88% (218/247) Updating files: 89% (220/247) Updating files: 90% (223/247) Updating files: 91% (225/247) Updating files: 92% (228/247) Updating files: 92% (229/247) Updating files: 93% (230/247) Updating files: 94% (233/247) Updating files: 95% (235/247) Updating files: 96% (238/247) Updating files: 97% (240/247) Updating files: 98% (243/247) Updating files: 99% (245/247) Updating files: 100% (247/247) Updating files: 100% (247/247), done. 2024-03-28T19:22:17.248 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2024-03-28T19:22:17.270 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.4 2024-03-28T19:22:17.345 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:22:17.346 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 5 -eq 48 ']' 2024-03-28T19:22:17.346 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.5 2024-03-28T19:22:17.346 INFO:tasks.workunit.client.1.smithi155.stdout:v0.5 2024-03-28T19:22:17.346 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.5 2024-03-28T19:22:17.346 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.5 2024-03-28T19:22:23.084 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 33% (36/107) Updating files: 34% (37/107) Updating files: 35% (38/107) Updating files: 36% (39/107) Updating files: 37% (40/107) Updating files: 38% (41/107) Updating files: 39% (42/107) Updating files: 40% (43/107) Updating files: 41% (44/107) Updating files: 42% (45/107) Updating files: 43% (47/107) Updating files: 44% (48/107) Updating files: 45% (49/107) Updating files: 46% (50/107) Updating files: 47% (51/107) Updating files: 48% (52/107) Updating files: 49% (53/107) Updating files: 50% (54/107) Updating files: 51% (55/107) Updating files: 52% (56/107) Updating files: 53% (57/107) Updating files: 54% (58/107) Updating files: 55% (59/107) Updating files: 56% (60/107) Updating files: 57% (61/107) Updating files: 58% (63/107) Updating files: 59% (64/107) Updating files: 60% (65/107) Updating files: 61% (66/107) Updating files: 62% (67/107) Updating files: 63% (68/107) Updating files: 64% (69/107) Updating files: 65% (70/107) Updating files: 66% (71/107) Updating files: 67% (72/107) Updating files: 68% (73/107) Updating files: 69% (74/107) Updating files: 70% (75/107) Updating files: 71% (76/107) Updating files: 72% (78/107) Updating files: 73% (79/107) Updating files: 74% (80/107) Updating files: 75% (81/107) Updating files: 76% (82/107) Updating files: 77% (83/107) Updating files: 78% (84/107) Updating files: 79% (85/107) Updating files: 80% (86/107) Updating files: 81% (87/107) Updating files: 82% (88/107) Updating files: 83% (89/107) Updating files: 84% (90/107) Updating files: 85% (91/107) Updating files: 86% (93/107) Updating files: 87% (94/107) Updating files: 88% (95/107) Updating files: 89% (96/107) Updating files: 90% (97/107) Updating files: 91% (98/107) Updating files: 92% (99/107) Updating files: 93% (100/107) Updating files: 94% (101/107) Updating files: 95% (102/107) Updating files: 96% (103/107) Updating files: 97% (104/107) Updating files: 98% (105/107) Updating files: 99% (106/107) Updating files: 100% (107/107) Updating files: 100% (107/107), done. 2024-03-28T19:22:23.330 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 38dbaa5175c v0.16 2024-03-28T19:22:23.341 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.16 2024-03-28T19:22:23.398 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:22:23.399 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 17 -eq 48 ']' 2024-03-28T19:22:23.399 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.17 2024-03-28T19:22:23.399 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.17 2024-03-28T19:22:23.399 INFO:tasks.workunit.client.0.smithi027.stdout:v0.17 2024-03-28T19:22:23.399 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.17 2024-03-28T19:22:26.293 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 17% (43/245) Updating files: 18% (45/245) Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 51% (127/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 97% (239/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2024-03-28T19:22:26.563 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2024-03-28T19:22:26.571 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.5 2024-03-28T19:22:26.628 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:22:26.628 INFO:tasks.workunit.client.1.smithi155.stdout:v0.6 2024-03-28T19:22:26.629 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 6 -eq 48 ']' 2024-03-28T19:22:26.629 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.6 2024-03-28T19:22:26.629 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.6 2024-03-28T19:22:26.629 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.6 2024-03-28T19:22:29.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.688+0000 7f9f37400640 1 Processor -- start 2024-03-28T19:22:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.688+0000 7f9f37400640 1 -- start start 2024-03-28T19:22:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f37400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f30150c70 0x7f9f300a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f37400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f301534c0 0x7f9f30153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f37400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f300c3af0 con 0x7f9f300c3690 2024-03-28T19:22:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f37400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f300ab4e0 con 0x7f9f30150c70 2024-03-28T19:22:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f37400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300ac1f0 con 0x7f9f301534c0 2024-03-28T19:22:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f34a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f30150c70 0x7f9f300a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f34a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f30150c70 0x7f9f300a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53694/0 (socket says 172.21.15.27:53694) 2024-03-28T19:22:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f34a00640 1 -- 172.21.15.27:0/2740029997 learned_addr learned my addr 172.21.15.27:0/2740029997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:29.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f2fe00640 1 --1- 172.21.15.27:0/2740029997 >> v1:172.21.15.27:6789/0 conn(0x7f9f300c3690 0x7f9f300c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60198/0 (socket says 172.21.15.27:60198) 2024-03-28T19:22:29.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.690+0000 7f9f35400640 1 --2- 172.21.15.27:0/2740029997 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f301534c0 0x7f9f30153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:29.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.691+0000 7f9f34a00640 1 -- 172.21.15.27:0/2740029997 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f301534c0 msgr2=0x7f9f30153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:29.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.691+0000 7f9f34a00640 1 --2- 172.21.15.27:0/2740029997 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f301534c0 0x7f9f30153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:29.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.691+0000 7f9f34a00640 1 -- 172.21.15.27:0/2740029997 >> v1:172.21.15.27:6789/0 conn(0x7f9f300c3690 legacy=0x7f9f300c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:29.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.691+0000 7f9f34a00640 1 -- 172.21.15.27:0/2740029997 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300c3d90 con 0x7f9f30150c70 2024-03-28T19:22:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.693+0000 7f9f34a00640 1 --2- 172.21.15.27:0/2740029997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 0x7f9f300a9cb0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009870 tx=0x7f9f2402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630d3c0b678eaa06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.693+0000 7f9f2f400640 1 -- 172.21.15.27:0/2740029997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24035020 con 0x7f9f30150c70 2024-03-28T19:22:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.693+0000 7f9f2f400640 1 -- 172.21.15.27:0/2740029997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f24031640 con 0x7f9f30150c70 2024-03-28T19:22:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.693+0000 7f9f2f400640 1 -- 172.21.15.27:0/2740029997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2402d040 con 0x7f9f30150c70 2024-03-28T19:22:29.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.694+0000 7f9f37400640 1 -- 172.21.15.27:0/2740029997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 msgr2=0x7f9f300a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:29.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.694+0000 7f9f37400640 1 --2- 172.21.15.27:0/2740029997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 0x7f9f300a9cb0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009870 tx=0x7f9f2402b8f0 comp rx=0 tx=0).stop 2024-03-28T19:22:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.695+0000 7f9f37400640 1 -- 172.21.15.27:0/2740029997 shutdown_connections 2024-03-28T19:22:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.695+0000 7f9f37400640 1 --2- 172.21.15.27:0/2740029997 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f301534c0 0x7f9f30153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.695+0000 7f9f37400640 1 --2- 172.21.15.27:0/2740029997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 0x7f9f300a9cb0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.695+0000 7f9f37400640 1 -- 172.21.15.27:0/2740029997 >> 172.21.15.27:0/2740029997 conn(0x7f9f3006b9d0 msgr2=0x7f9f300c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.695+0000 7f9f37400640 1 -- 172.21.15.27:0/2740029997 shutdown_connections 2024-03-28T19:22:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.695+0000 7f9f37400640 1 -- 172.21.15.27:0/2740029997 wait complete. 2024-03-28T19:22:29.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.698+0000 7f9f37400640 1 Processor -- start 2024-03-28T19:22:29.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.698+0000 7f9f37400640 1 -- start start 2024-03-28T19:22:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.699+0000 7f9f37400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f300c3690 0x7f9f300a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.699+0000 7f9f37400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 0x7f9f30071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.699+0000 7f9f37400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f301534c0 0x7f9f3006e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.699+0000 7f9f37400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300abcd0 con 0x7f9f30150c70 2024-03-28T19:22:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.699+0000 7f9f37400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300ab9f0 con 0x7f9f301534c0 2024-03-28T19:22:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.699+0000 7f9f37400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300c1810 con 0x7f9f300c3690 2024-03-28T19:22:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f2fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 0x7f9f30071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f34a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f300c3690 0x7f9f300a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f34a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f300c3690 0x7f9f300a93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:57004/0 (socket says 172.21.15.27:57004) 2024-03-28T19:22:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f2fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 0x7f9f30071ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53700/0 (socket says 172.21.15.27:53700) 2024-03-28T19:22:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f34a00640 1 -- 172.21.15.27:0/801357354 learned_addr learned my addr 172.21.15.27:0/801357354 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f35400640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f301534c0 0x7f9f3006e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f34a00640 1 -- 172.21.15.27:0/801357354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f301534c0 msgr2=0x7f9f3006e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f34a00640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f301534c0 0x7f9f3006e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f34a00640 1 -- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 msgr2=0x7f9f30071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f34a00640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 0x7f9f30071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.700+0000 7f9f34a00640 1 -- 172.21.15.27:0/801357354 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300a7640 con 0x7f9f300c3690 2024-03-28T19:22:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.701+0000 7f9f34a00640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f300c3690 0x7f9f300a93a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9f2402b930 tx=0x7f9f240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:29.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.702+0000 7f9f2cc00640 1 -- 172.21.15.27:0/801357354 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24040070 con 0x7f9f300c3690 2024-03-28T19:22:29.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.702+0000 7f9f2cc00640 1 -- 172.21.15.27:0/801357354 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2403e070 con 0x7f9f300c3690 2024-03-28T19:22:29.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.702+0000 7f9f2cc00640 1 -- 172.21.15.27:0/801357354 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24035040 con 0x7f9f300c3690 2024-03-28T19:22:29.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.702+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f300a7920 con 0x7f9f300c3690 2024-03-28T19:22:29.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.702+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f301f2d90 con 0x7f9f300c3690 2024-03-28T19:22:29.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.704+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f300c1270 con 0x7f9f300c3690 2024-03-28T19:22:29.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.705+0000 7f9f2cc00640 1 -- 172.21.15.27:0/801357354 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9f24038020 con 0x7f9f300c3690 2024-03-28T19:22:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.706+0000 7f9f2cc00640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f1006bc50 0x7f9f1006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.706+0000 7f9f2cc00640 1 -- 172.21.15.27:0/801357354 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f9f240ad120 con 0x7f9f300c3690 2024-03-28T19:22:29.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.707+0000 7f9f2fe00640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f1006bc50 0x7f9f1006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:29.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.708+0000 7f9f2fe00640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f1006bc50 0x7f9f1006e110 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9f3014f700 tx=0x7f9f1c007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:29.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:29.713+0000 7f9f2cc00640 1 -- 172.21.15.27:0/801357354 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9f300c1270 con 0x7f9f300c3690 2024-03-28T19:22:29.846 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 37% (81/217) Updating files: 38% (83/217) Updating files: 39% (85/217) Updating files: 40% (87/217) Updating files: 41% (89/217) Updating files: 42% (92/217) Updating files: 43% (94/217) Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2024-03-28T19:22:30.046 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at dac3a8c35fb v0.17 2024-03-28T19:22:30.054 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.17 2024-03-28T19:22:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.064+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f3004f400 con 0x7f9f300c3690 2024-03-28T19:22:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.065+0000 7f9f2cc00640 1 -- 172.21.15.27:0/801357354 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f9f2407a690 con 0x7f9f300c3690 2024-03-28T19:22:30.065 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:30.065 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 199 2024-03-28T19:22:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f1006bc50 msgr2=0x7f9f1006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f1006bc50 0x7f9f1006e110 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9f3014f700 tx=0x7f9f1c007e80 comp rx=0 tx=0).stop 2024-03-28T19:22:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f300c3690 msgr2=0x7f9f300a93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f300c3690 0x7f9f300a93a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9f2402b930 tx=0x7f9f240046c0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 shutdown_connections 2024-03-28T19:22:30.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f1006bc50 0x7f9f1006e110 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f301534c0 0x7f9f3006e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f30150c70 0x7f9f30071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 --2- 172.21.15.27:0/801357354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f300c3690 0x7f9f300a93a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 >> 172.21.15.27:0/801357354 conn(0x7f9f3006b9d0 msgr2=0x7f9f30151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 shutdown_connections 2024-03-28T19:22:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.068+0000 7f9f37400640 1 -- 172.21.15.27:0/801357354 wait complete. 2024-03-28T19:22:30.093 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-03-28T19:22:30.093 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:22:30.093 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:22:30.105 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:22:30.105 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 18 -eq 48 ']' 2024-03-28T19:22:30.105 INFO:tasks.workunit.client.0.smithi027.stdout:v0.18 2024-03-28T19:22:30.105 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.18 2024-03-28T19:22:30.105 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.18 2024-03-28T19:22:30.106 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.18 2024-03-28T19:22:30.122 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:22:30.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.346+0000 7f1388800640 1 Processor -- start 2024-03-28T19:22:30.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.346+0000 7f1388800640 1 -- start start 2024-03-28T19:22:30.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.347+0000 7f1388800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a9cb0 0x7f13800c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:30.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.347+0000 7f1388800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13800c3d90 con 0x7f1380150c70 2024-03-28T19:22:30.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.347+0000 7f1388800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13800ab710 con 0x7f13801534c0 2024-03-28T19:22:30.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.347+0000 7f1388800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13800ab4e0 con 0x7f13800a9cb0 2024-03-28T19:22:30.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.348+0000 7f1385e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1380150c70 0x7f13800c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60212/0 (socket says 172.21.15.27:60212) 2024-03-28T19:22:30.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.348+0000 7f1385e00640 1 -- 172.21.15.27:0/865233697 learned_addr learned my addr 172.21.15.27:0/865233697 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:22:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.348+0000 7f1385400640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a9cb0 0x7f13800c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.348+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318776010 0 0) 0x7f13800c3d90 con 0x7f1380150c70 2024-03-28T19:22:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.348+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1368003520 con 0x7f1380150c70 2024-03-28T19:22:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1524609691 0 0) 0x7f1368003520 con 0x7f1380150c70 2024-03-28T19:22:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a9cb0 msgr2=0x7f13800c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a9cb0 0x7f13800c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 >> v1:172.21.15.155:6789/0 conn(0x7f13801534c0 legacy=0x7f1380153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13800ab9f0 con 0x7f1380150c70 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f1374001670 con 0x7f1380150c70 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 >> v1:172.21.15.27:6789/0 conn(0x7f1380150c70 legacy=0x7f13800c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a9cb0 0x7f13680039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1368003fb0 0x7f1368004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368004d40 0x7f13680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136800bb80 con 0x7f1368003fb0 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136800bd30 con 0x7f1368004d40 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136800bf30 con 0x7f13800a9cb0 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1385400640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1368003fb0 0x7f1368004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:30.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.349+0000 7f1385e00640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a9cb0 0x7f13680039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.350+0000 7f1386800640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368004d40 0x7f13680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.350+0000 7f1385400640 1 -- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a9cb0 msgr2=0x7f13680039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:30.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.350+0000 7f1385400640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a9cb0 0x7f13680039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.350+0000 7f1385400640 1 -- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368004d40 msgr2=0x7f13680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.350+0000 7f1385400640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368004d40 0x7f13680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.350+0000 7f1385400640 1 -- 172.21.15.27:0/865233697 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1368003520 con 0x7f1368003fb0 2024-03-28T19:22:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.351+0000 7f1385400640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1368003fb0 0x7f1368004800 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f1370009ac0 tx=0x7f1368005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db87ec20032212ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.351+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1370039070 con 0x7f1368003fb0 2024-03-28T19:22:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.352+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137002c6b0 con 0x7f1368003fb0 2024-03-28T19:22:30.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.352+0000 7f1384a00640 1 -- 172.21.15.27:0/865233697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1370009690 con 0x7f1368003fb0 2024-03-28T19:22:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.352+0000 7f1388800640 1 -- 172.21.15.27:0/865233697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1368003fb0 msgr2=0x7f1368004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.352+0000 7f1388800640 1 --2- 172.21.15.27:0/865233697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1368003fb0 0x7f1368004800 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f1370009ac0 tx=0x7f1368005830 comp rx=0 tx=0).stop 2024-03-28T19:22:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.352+0000 7f1386800640 1 -- 172.21.15.27:0/865233697 reap_dead start 2024-03-28T19:22:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.353+0000 7f1388800640 1 -- 172.21.15.27:0/865233697 shutdown_connections 2024-03-28T19:22:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.353+0000 7f1388800640 1 -- 172.21.15.27:0/865233697 >> 172.21.15.27:0/865233697 conn(0x7f138006b9d0 msgr2=0x7f13800c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.353+0000 7f1388800640 1 -- 172.21.15.27:0/865233697 shutdown_connections 2024-03-28T19:22:30.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.353+0000 7f1388800640 1 -- 172.21.15.27:0/865233697 wait complete. 2024-03-28T19:22:30.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.356+0000 7f1388800640 1 Processor -- start 2024-03-28T19:22:30.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.356+0000 7f1388800640 1 -- start start 2024-03-28T19:22:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1388800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368003fb0 0x7f138014e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1388800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a5d80 0x7f13800a8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1388800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13800a85a0 0x7f1380156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1388800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1380154060 con 0x7f13800a85a0 2024-03-28T19:22:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1388800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13800ac1f0 con 0x7f1368003fb0 2024-03-28T19:22:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1388800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13800c1810 con 0x7f13800a5d80 2024-03-28T19:22:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1386800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13800a85a0 0x7f1380156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1386800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13800a85a0 0x7f1380156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53726/0 (socket says 172.21.15.27:53726) 2024-03-28T19:22:30.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1386800640 1 -- 172.21.15.27:0/3377595717 learned_addr learned my addr 172.21.15.27:0/3377595717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:30.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.358+0000 7f1385e00640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368003fb0 0x7f138014e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1385400640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a5d80 0x7f13800a8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1386800640 1 -- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a5d80 msgr2=0x7f13800a8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1386800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a5d80 0x7f13800a8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1386800640 1 -- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368003fb0 msgr2=0x7f138014e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1386800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368003fb0 0x7f138014e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1386800640 1 -- 172.21.15.27:0/3377595717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13801eee70 con 0x7f13800a85a0 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1385400640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a5d80 0x7f13800a8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1385e00640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368003fb0 0x7f138014e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f1386800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13800a85a0 0x7f1380156b50 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f137c000f80 tx=0x7f137c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f137a000640 1 -- 172.21.15.27:0/3377595717 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c003970 con 0x7f13800a85a0 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.359+0000 7f137a000640 1 -- 172.21.15.27:0/3377595717 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c018070 con 0x7f13800a85a0 2024-03-28T19:22:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.360+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13801ef150 con 0x7f13800a85a0 2024-03-28T19:22:30.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.360+0000 7f137a000640 1 -- 172.21.15.27:0/3377595717 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c003d70 con 0x7f13800a85a0 2024-03-28T19:22:30.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.360+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1380154b10 con 0x7f13800a85a0 2024-03-28T19:22:30.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.361+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138004f400 con 0x7f13800a85a0 2024-03-28T19:22:30.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.363+0000 7f137a000640 1 -- 172.21.15.27:0/3377595717 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f137c0050d0 con 0x7f13800a85a0 2024-03-28T19:22:30.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.364+0000 7f137a000640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bc00 0x7f136406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:30.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.364+0000 7f137a000640 1 -- 172.21.15.27:0/3377595717 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f137c08b640 con 0x7f13800a85a0 2024-03-28T19:22:30.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.364+0000 7f1385e00640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bc00 0x7f136406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.365+0000 7f1385e00640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bc00 0x7f136406e0c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1374001050 tx=0x7f1374009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.370+0000 7f137a000640 1 -- 172.21.15.27:0/3377595717 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f137c059210 con 0x7f13800a85a0 2024-03-28T19:22:30.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.652+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13800c5f00 con 0x7f13800a85a0 2024-03-28T19:22:30.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.653+0000 7f137a000640 1 -- 172.21.15.27:0/3377595717 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f137c058b10 con 0x7f13800a85a0 2024-03-28T19:22:30.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:30.653 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:30.653 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 199 2024-03-28T19:22:30.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bc00 msgr2=0x7f136406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bc00 0x7f136406e0c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1374001050 tx=0x7f1374009040 comp rx=0 tx=0).stop 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13800a85a0 msgr2=0x7f1380156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13800a85a0 0x7f1380156b50 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f137c000f80 tx=0x7f137c00ec10 comp rx=0 tx=0).stop 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 shutdown_connections 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f136406bc00 0x7f136406e0c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13800a85a0 0x7f1380156b50 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f13800a5d80 0x7f13800a8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 --2- 172.21.15.27:0/3377595717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1368003fb0 0x7f138014e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 >> 172.21.15.27:0/3377595717 conn(0x7f138006b9d0 msgr2=0x7f13800c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.656+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 shutdown_connections 2024-03-28T19:22:30.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:30.657+0000 7f1388800640 1 -- 172.21.15.27:0/3377595717 wait complete. 2024-03-28T19:22:30.678 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:22:32.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:32.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.904+0000 7fe193200640 1 Processor -- start 2024-03-28T19:22:32.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.905+0000 7fe193200640 1 -- start start 2024-03-28T19:22:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.906+0000 7fe193200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 0x7fe1940a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.906+0000 7fe193200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1940a61a0 0x7fe1940c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.906+0000 7fe193200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe194150ec0 con 0x7fe1940c6ee0 2024-03-28T19:22:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.906+0000 7fe193200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19406dc70 con 0x7fe1940a61a0 2024-03-28T19:22:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.906+0000 7fe193200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe194153550 con 0x7fe194150ae0 2024-03-28T19:22:32.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.906+0000 7fe191400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1940a61a0 0x7fe1940c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.906+0000 7fe192800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe1940c6ee0 0x7fe194153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60222/0 (socket says 172.21.15.27:60222) 2024-03-28T19:22:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.906+0000 7fe192800640 1 -- 172.21.15.27:0/3125857953 learned_addr learned my addr 172.21.15.27:0/3125857953 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:22:32.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.907+0000 7fe191e00640 1 --2- 172.21.15.27:0/3125857953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 0x7fe1940a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:32.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.909+0000 7fe190a00640 1 -- 172.21.15.27:0/3125857953 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292356015 0 0) 0x7fe18c001490 con 0x7fe1940c6ee0 2024-03-28T19:22:32.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.909+0000 7fe190a00640 1 -- 172.21.15.27:0/3125857953 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe178003520 con 0x7fe1940c6ee0 2024-03-28T19:22:32.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.909+0000 7fe191e00640 1 -- 172.21.15.27:0/3125857953 >> v2:172.21.15.27:3300/0 conn(0x7fe1940a61a0 msgr2=0x7fe1940c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:32.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.909+0000 7fe191e00640 1 --2- 172.21.15.27:0/3125857953 >> v2:172.21.15.27:3300/0 conn(0x7fe1940a61a0 0x7fe1940c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:32.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.909+0000 7fe191e00640 1 -- 172.21.15.27:0/3125857953 >> v1:172.21.15.27:6789/0 conn(0x7fe1940c6ee0 legacy=0x7fe194153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:32.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.909+0000 7fe191e00640 1 -- 172.21.15.27:0/3125857953 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19406df10 con 0x7fe194150ae0 2024-03-28T19:22:32.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.912+0000 7fe191e00640 1 --2- 172.21.15.27:0/3125857953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 0x7fe1940a5c60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe1840097a0 tx=0x7fe18402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=337d2be36fa4d260 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.914+0000 7fe190a00640 1 -- 172.21.15.27:0/3125857953 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184039070 con 0x7fe194150ae0 2024-03-28T19:22:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.915+0000 7fe190a00640 1 -- 172.21.15.27:0/3125857953 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1840316f0 con 0x7fe194150ae0 2024-03-28T19:22:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.915+0000 7fe190a00640 1 -- 172.21.15.27:0/3125857953 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184031aa0 con 0x7fe194150ae0 2024-03-28T19:22:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.916+0000 7fe193200640 1 -- 172.21.15.27:0/3125857953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 msgr2=0x7fe1940a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.916+0000 7fe193200640 1 --2- 172.21.15.27:0/3125857953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 0x7fe1940a5c60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe1840097a0 tx=0x7fe18402b790 comp rx=0 tx=0).stop 2024-03-28T19:22:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.916+0000 7fe193200640 1 -- 172.21.15.27:0/3125857953 shutdown_connections 2024-03-28T19:22:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.916+0000 7fe193200640 1 --2- 172.21.15.27:0/3125857953 >> v2:172.21.15.27:3300/0 conn(0x7fe1940a61a0 0x7fe1940c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.916+0000 7fe193200640 1 --2- 172.21.15.27:0/3125857953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 0x7fe1940a5c60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:32.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.916+0000 7fe193200640 1 -- 172.21.15.27:0/3125857953 >> 172.21.15.27:0/3125857953 conn(0x7fe19406b0c0 msgr2=0x7fe19406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.917+0000 7fe193200640 1 -- 172.21.15.27:0/3125857953 shutdown_connections 2024-03-28T19:22:32.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.917+0000 7fe193200640 1 -- 172.21.15.27:0/3125857953 wait complete. 2024-03-28T19:22:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.919+0000 7fe193200640 1 Processor -- start 2024-03-28T19:22:32.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.919+0000 7fe193200640 1 -- start start 2024-03-28T19:22:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe193200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1940a61a0 0x7fe19414bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe193200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1940c6ee0 0x7fe1940a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe193200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 0x7fe1940aa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe193200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19406e7b0 con 0x7fe1940a61a0 2024-03-28T19:22:32.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe193200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19406e4d0 con 0x7fe1940c6ee0 2024-03-28T19:22:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe193200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941569d0 con 0x7fe194150ae0 2024-03-28T19:22:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe191e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1940a61a0 0x7fe19414bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe191e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1940a61a0 0x7fe19414bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53736/0 (socket says 172.21.15.27:53736) 2024-03-28T19:22:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.920+0000 7fe191e00640 1 -- 172.21.15.27:0/3104522767 learned_addr learned my addr 172.21.15.27:0/3104522767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:32.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.921+0000 7fe191e00640 1 -- 172.21.15.27:0/3104522767 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 msgr2=0x7fe1940aa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.921+0000 7fe191e00640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 0x7fe1940aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.921+0000 7fe191e00640 1 -- 172.21.15.27:0/3104522767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1940c6ee0 msgr2=0x7fe1940a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.921+0000 7fe191e00640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1940c6ee0 0x7fe1940a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.921+0000 7fe191e00640 1 -- 172.21.15.27:0/3104522767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940c5680 con 0x7fe1940a61a0 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.921+0000 7fe191e00640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1940a61a0 0x7fe19414bbd0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fe18402b7d0 tx=0x7fe1840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.922+0000 7fe18a000640 1 -- 172.21.15.27:0/3104522767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184039040 con 0x7fe1940a61a0 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.922+0000 7fe18a000640 1 -- 172.21.15.27:0/3104522767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe184044070 con 0x7fe1940a61a0 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.922+0000 7fe18a000640 1 -- 172.21.15.27:0/3104522767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184031ce0 con 0x7fe1940a61a0 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.922+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1941e7cc0 con 0x7fe1940a61a0 2024-03-28T19:22:32.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.922+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe194072260 con 0x7fe1940a61a0 2024-03-28T19:22:32.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.923+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe194067dd0 con 0x7fe1940a61a0 2024-03-28T19:22:32.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.925+0000 7fe18a000640 1 -- 172.21.15.27:0/3104522767 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe184003610 con 0x7fe1940a61a0 2024-03-28T19:22:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.927+0000 7fe18a000640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe17406bbd0 0x7fe17406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.927+0000 7fe18a000640 1 -- 172.21.15.27:0/3104522767 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7fe1840ad2c0 con 0x7fe1940a61a0 2024-03-28T19:22:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.927+0000 7fe191400640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe17406bbd0 0x7fe17406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:32.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.928+0000 7fe191400640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe17406bbd0 0x7fe17406e090 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe180006a90 tx=0x7fe180006a20 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:32.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:32.937+0000 7fe18a000640 1 -- 172.21.15.27:0/3104522767 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe18407af50 con 0x7fe1940a61a0 2024-03-28T19:22:33.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.234+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1940c58b0 con 0x7fe1940a61a0 2024-03-28T19:22:33.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.235+0000 7fe18a000640 1 -- 172.21.15.27:0/3104522767 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fe18407a850 con 0x7fe1940a61a0 2024-03-28T19:22:33.235 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:33.235 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:33.235 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 199 2024-03-28T19:22:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.237+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe17406bbd0 msgr2=0x7fe17406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:33.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe17406bbd0 0x7fe17406e090 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe180006a90 tx=0x7fe180006a20 comp rx=0 tx=0).stop 2024-03-28T19:22:33.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1940a61a0 msgr2=0x7fe19414bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:33.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1940a61a0 0x7fe19414bbd0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fe18402b7d0 tx=0x7fe1840046c0 comp rx=0 tx=0).stop 2024-03-28T19:22:33.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 shutdown_connections 2024-03-28T19:22:33.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe17406bbd0 0x7fe17406e090 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe194150ae0 0x7fe1940aa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1940c6ee0 0x7fe1940a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 --2- 172.21.15.27:0/3104522767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1940a61a0 0x7fe19414bbd0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 >> 172.21.15.27:0/3104522767 conn(0x7fe19406b0c0 msgr2=0x7fe1940c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 shutdown_connections 2024-03-28T19:22:33.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:33.238+0000 7fe193200640 1 -- 172.21.15.27:0/3104522767 wait complete. 2024-03-28T19:22:33.261 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:22:35.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:35.290 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 21% (49/227) Updating files: 22% (50/227) Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2024-03-28T19:22:35.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.492+0000 7f18e1600640 1 Processor -- start 2024-03-28T19:22:35.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.492+0000 7f18e1600640 1 -- start start 2024-03-28T19:22:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.493+0000 7f18e1600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18dc0c59d0 0x7f18dc0c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.493+0000 7f18e1600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f18dc0727e0 0x7f18dc159520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.493+0000 7f18e1600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc155fe0 con 0x7f18dc155600 2024-03-28T19:22:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.493+0000 7f18e1600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18dc1565a0 con 0x7f18dc0c59d0 2024-03-28T19:22:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.493+0000 7f18e1600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18dc159090 con 0x7f18dc0727e0 2024-03-28T19:22:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18dbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18dc0c59d0 0x7f18dc0c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:35.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18dbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18dc0c59d0 0x7f18dc0c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53748/0 (socket says 172.21.15.27:53748) 2024-03-28T19:22:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18dbe00640 1 -- 172.21.15.27:0/1814290453 learned_addr learned my addr 172.21.15.27:0/1814290453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18e0c00640 1 --1- 172.21.15.27:0/1814290453 >> v1:172.21.15.27:6789/0 conn(0x7f18dc155600 0x7f18dc14f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60236/0 (socket says 172.21.15.27:60236) 2024-03-28T19:22:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18db400640 1 --2- 172.21.15.27:0/1814290453 >> v2:172.21.15.155:3300/0 conn(0x7f18dc0727e0 0x7f18dc159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18dbe00640 1 -- 172.21.15.27:0/1814290453 >> v2:172.21.15.155:3300/0 conn(0x7f18dc0727e0 msgr2=0x7f18dc159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18dbe00640 1 --2- 172.21.15.27:0/1814290453 >> v2:172.21.15.155:3300/0 conn(0x7f18dc0727e0 0x7f18dc159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18dbe00640 1 -- 172.21.15.27:0/1814290453 >> v1:172.21.15.27:6789/0 conn(0x7f18dc155600 legacy=0x7f18dc14f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.494+0000 7f18dbe00640 1 -- 172.21.15.27:0/1814290453 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc1562c0 con 0x7f18dc0c59d0 2024-03-28T19:22:35.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.496+0000 7f18dbe00640 1 --2- 172.21.15.27:0/1814290453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc0c8f00 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0ad9d0 tx=0x7f18cc02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3be632801c97ad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:35.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.496+0000 7f18daa00640 1 -- 172.21.15.27:0/1814290453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc035020 con 0x7f18dc0c59d0 2024-03-28T19:22:35.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.496+0000 7f18daa00640 1 -- 172.21.15.27:0/1814290453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc031800 con 0x7f18dc0c59d0 2024-03-28T19:22:35.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.496+0000 7f18daa00640 1 -- 172.21.15.27:0/1814290453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc031bd0 con 0x7f18dc0c59d0 2024-03-28T19:22:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.497+0000 7f18e1600640 1 -- 172.21.15.27:0/1814290453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 msgr2=0x7f18dc0c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.497+0000 7f18e1600640 1 --2- 172.21.15.27:0/1814290453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc0c8f00 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0ad9d0 tx=0x7f18cc02b8a0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.497+0000 7f18e1600640 1 -- 172.21.15.27:0/1814290453 shutdown_connections 2024-03-28T19:22:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.497+0000 7f18e1600640 1 --2- 172.21.15.27:0/1814290453 >> v2:172.21.15.155:3300/0 conn(0x7f18dc0727e0 0x7f18dc159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.497+0000 7f18e1600640 1 --2- 172.21.15.27:0/1814290453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc0c8f00 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.497+0000 7f18e1600640 1 -- 172.21.15.27:0/1814290453 >> 172.21.15.27:0/1814290453 conn(0x7f18dc0ac910 msgr2=0x7f18dc0accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.497+0000 7f18e1600640 1 -- 172.21.15.27:0/1814290453 shutdown_connections 2024-03-28T19:22:35.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.497+0000 7f18e1600640 1 -- 172.21.15.27:0/1814290453 wait complete. 2024-03-28T19:22:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.498+0000 7f18e1600640 1 Processor -- start 2024-03-28T19:22:35.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.499+0000 7f18e1600640 1 -- start start 2024-03-28T19:22:35.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.499+0000 7f18e1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18dc0727e0 0x7f18dc0abb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.499+0000 7f18e1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18e1600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18dc155600 0x7f18dc1531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18e1600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc14f970 con 0x7f18dc0c59d0 2024-03-28T19:22:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18e1600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc159a60 con 0x7f18dc0727e0 2024-03-28T19:22:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18e1600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc14faf0 con 0x7f18dc155600 2024-03-28T19:22:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53756/0 (socket says 172.21.15.27:53756) 2024-03-28T19:22:35.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18db400640 1 -- 172.21.15.27:0/3781301795 learned_addr learned my addr 172.21.15.27:0/3781301795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:35.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18db400640 1 -- 172.21.15.27:0/3781301795 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18dc155600 msgr2=0x7f18dc1531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:35.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18db400640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18dc155600 0x7f18dc1531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18db400640 1 -- 172.21.15.27:0/3781301795 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18dc0727e0 msgr2=0x7f18dc0abb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:35.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18db400640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18dc0727e0 0x7f18dc0abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.500+0000 7f18db400640 1 -- 172.21.15.27:0/3781301795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc0c4880 con 0x7f18dc0c59d0 2024-03-28T19:22:35.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.501+0000 7f18db400640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc152c60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f18c800d7d0 tx=0x7f18c800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.501+0000 7f18d3e00640 1 -- 172.21.15.27:0/3781301795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8004020 con 0x7f18dc0c59d0 2024-03-28T19:22:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.501+0000 7f18d3e00640 1 -- 172.21.15.27:0/3781301795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c80041e0 con 0x7f18dc0c59d0 2024-03-28T19:22:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.501+0000 7f18d3e00640 1 -- 172.21.15.27:0/3781301795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c80043d0 con 0x7f18dc0c59d0 2024-03-28T19:22:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.501+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc0c4b60 con 0x7f18dc0c59d0 2024-03-28T19:22:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.501+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc0c4e40 con 0x7f18dc0c59d0 2024-03-28T19:22:35.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.502+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc067dd0 con 0x7f18dc0c59d0 2024-03-28T19:22:35.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.503+0000 7f18d3e00640 1 -- 172.21.15.27:0/3781301795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f18c801b020 con 0x7f18dc0c59d0 2024-03-28T19:22:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.504+0000 7f18d3e00640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18bc06bbd0 0x7f18bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.505+0000 7f18d3e00640 1 -- 172.21.15.27:0/3781301795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f18c808b800 con 0x7f18dc0c59d0 2024-03-28T19:22:35.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.505+0000 7f18dbe00640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18bc06bbd0 0x7f18bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:35.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.506+0000 7f18dbe00640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18bc06bbd0 0x7f18bc06e090 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f18cc036000 tx=0x7f18cc030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.513+0000 7f18d3e00640 1 -- 172.21.15.27:0/3781301795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f18c80593d0 con 0x7f18dc0c59d0 2024-03-28T19:22:35.636 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at a4c752df713 v0.6 2024-03-28T19:22:35.656 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.6 2024-03-28T19:22:35.725 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:22:35.725 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 7 -eq 48 ']' 2024-03-28T19:22:35.726 INFO:tasks.workunit.client.1.smithi155.stdout:v0.7 2024-03-28T19:22:35.726 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.7 2024-03-28T19:22:35.726 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.7 2024-03-28T19:22:35.726 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.7 2024-03-28T19:22:35.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.825+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18dc069640 con 0x7f18dc0c59d0 2024-03-28T19:22:35.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.826+0000 7f18d3e00640 1 -- 172.21.15.27:0/3781301795 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f18c805f0d0 con 0x7f18dc0c59d0 2024-03-28T19:22:35.825 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:35.826 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:35.826 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 199 2024-03-28T19:22:35.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18bc06bbd0 msgr2=0x7f18bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:35.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18bc06bbd0 0x7f18bc06e090 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f18cc036000 tx=0x7f18cc030000 comp rx=0 tx=0).stop 2024-03-28T19:22:35.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 msgr2=0x7f18dc152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:35.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc152c60 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f18c800d7d0 tx=0x7f18c800dc90 comp rx=0 tx=0).stop 2024-03-28T19:22:35.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 shutdown_connections 2024-03-28T19:22:35.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f18bc06bbd0 0x7f18bc06e090 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18dc155600 0x7f18dc1531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18dc0c59d0 0x7f18dc152c60 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 --2- 172.21.15.27:0/3781301795 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18dc0727e0 0x7f18dc0abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 >> 172.21.15.27:0/3781301795 conn(0x7f18dc0ac910 msgr2=0x7f18dc1584b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 shutdown_connections 2024-03-28T19:22:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:35.829+0000 7f18e1600640 1 -- 172.21.15.27:0/3781301795 wait complete. 2024-03-28T19:22:35.847 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:22:37.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:38.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.072+0000 7f82ef600640 1 Processor -- start 2024-03-28T19:22:38.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.073+0000 7f82ef600640 1 -- start start 2024-03-28T19:22:38.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.074+0000 7f82ef600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f82e80c5e80 0x7f82e80c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:38.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.074+0000 7f82ef600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82e80c1f50 0x7f82e80ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:38.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.074+0000 7f82ef600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e80ac1c0 con 0x7f82e80abd90 2024-03-28T19:22:38.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.074+0000 7f82ef600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e806d650 con 0x7f82e80c1f50 2024-03-28T19:22:38.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.074+0000 7f82ef600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e806e1d0 con 0x7f82e80c5e80 2024-03-28T19:22:38.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82e7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82e80c1f50 0x7f82e80ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:38.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82e7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82e80c1f50 0x7f82e80ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37262/0 (socket says 172.21.15.27:37262) 2024-03-28T19:22:38.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82ed600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f82e80abd90 0x7f82e80aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44758/0 (socket says 172.21.15.27:44758) 2024-03-28T19:22:38.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82e7e00640 1 -- 172.21.15.27:0/2999281821 learned_addr learned my addr 172.21.15.27:0/2999281821 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:38.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82ecc00640 1 --2- 172.21.15.27:0/2999281821 >> v2:172.21.15.155:3300/0 conn(0x7f82e80c5e80 0x7f82e80c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:38.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82e7e00640 1 -- 172.21.15.27:0/2999281821 >> v2:172.21.15.155:3300/0 conn(0x7f82e80c5e80 msgr2=0x7f82e80c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:38.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82e7e00640 1 --2- 172.21.15.27:0/2999281821 >> v2:172.21.15.155:3300/0 conn(0x7f82e80c5e80 0x7f82e80c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82e7e00640 1 -- 172.21.15.27:0/2999281821 >> v1:172.21.15.27:6789/0 conn(0x7f82e80abd90 legacy=0x7f82e80aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:38.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82e7e00640 1 -- 172.21.15.27:0/2999281821 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e806d8a0 con 0x7f82e80c1f50 2024-03-28T19:22:38.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82ecc00640 1 --2- 172.21.15.27:0/2999281821 >> v2:172.21.15.155:3300/0 conn(0x7f82e80c5e80 0x7f82e80c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:22:38.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.075+0000 7f82e7400640 1 -- 172.21.15.27:0/2999281821 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329563248 0 0) 0x7f82e80ac1c0 con 0x7f82e80abd90 2024-03-28T19:22:38.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82e7e00640 1 --2- 172.21.15.27:0/2999281821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e80ab850 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f82e80c6670 tx=0x7f82d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbba16d83d7ed5f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:38.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82e7400640 1 -- 172.21.15.27:0/2999281821 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d402e070 con 0x7f82e80c1f50 2024-03-28T19:22:38.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82e7400640 1 -- 172.21.15.27:0/2999281821 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d4037070 con 0x7f82e80c1f50 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82e7400640 1 -- 172.21.15.27:0/2999281821 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4031a00 con 0x7f82e80c1f50 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82ef600640 1 -- 172.21.15.27:0/2999281821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 msgr2=0x7f82e80ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82ef600640 1 --2- 172.21.15.27:0/2999281821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e80ab850 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f82e80c6670 tx=0x7f82d402b880 comp rx=0 tx=0).stop 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82ef600640 1 -- 172.21.15.27:0/2999281821 shutdown_connections 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82ef600640 1 --2- 172.21.15.27:0/2999281821 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e80ab850 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82ef600640 1 --2- 172.21.15.27:0/2999281821 >> v2:172.21.15.155:3300/0 conn(0x7f82e80c5e80 0x7f82e80c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.077+0000 7f82ef600640 1 -- 172.21.15.27:0/2999281821 >> 172.21.15.27:0/2999281821 conn(0x7f82e806b9d0 msgr2=0x7f82e80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.078+0000 7f82ef600640 1 -- 172.21.15.27:0/2999281821 shutdown_connections 2024-03-28T19:22:38.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.078+0000 7f82ef600640 1 -- 172.21.15.27:0/2999281821 wait complete. 2024-03-28T19:22:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.079+0000 7f82ef600640 1 Processor -- start 2024-03-28T19:22:38.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.079+0000 7f82ef600640 1 -- start start 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.080+0000 7f82ef600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82e80abd90 0x7f82e81ec150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.080+0000 7f82ef600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e81ec690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.080+0000 7f82ef600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82e80c5e80 0x7f82e8151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.080+0000 7f82ef600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80c31f0 con 0x7f82e80c1f50 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.080+0000 7f82ef600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e806de60 con 0x7f82e80abd90 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.080+0000 7f82ef600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80c3370 con 0x7f82e80c5e80 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.080+0000 7f82e7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e81ec690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82e7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e81ec690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37276/0 (socket says 172.21.15.27:37276) 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82e7e00640 1 -- 172.21.15.27:0/3003797974 learned_addr learned my addr 172.21.15.27:0/3003797974 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82ecc00640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82e80abd90 0x7f82e81ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:38.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82ed600640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82e80c5e80 0x7f82e8151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:38.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82e7e00640 1 -- 172.21.15.27:0/3003797974 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82e80c5e80 msgr2=0x7f82e8151380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:38.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82e7e00640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82e80c5e80 0x7f82e8151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82e7e00640 1 -- 172.21.15.27:0/3003797974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82e80abd90 msgr2=0x7f82e81ec150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:38.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82e7e00640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82e80abd90 0x7f82e81ec150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.081+0000 7f82e7e00640 1 -- 172.21.15.27:0/3003797974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e809aad0 con 0x7f82e80c1f50 2024-03-28T19:22:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.082+0000 7f82e7e00640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e81ec690 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f82d40090b0 tx=0x7f82d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.082+0000 7f82e4c00640 1 -- 172.21.15.27:0/3003797974 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4004230 con 0x7f82e80c1f50 2024-03-28T19:22:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.082+0000 7f82e4c00640 1 -- 172.21.15.27:0/3003797974 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d40043f0 con 0x7f82e80c1f50 2024-03-28T19:22:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.082+0000 7f82e4c00640 1 -- 172.21.15.27:0/3003797974 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4003850 con 0x7f82e80c1f50 2024-03-28T19:22:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.082+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e809adb0 con 0x7f82e80c1f50 2024-03-28T19:22:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.082+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e809b1b0 con 0x7f82e80c1f50 2024-03-28T19:22:38.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.083+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e804f400 con 0x7f82e80c1f50 2024-03-28T19:22:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.084+0000 7f82e4c00640 1 -- 172.21.15.27:0/3003797974 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f82d404c060 con 0x7f82e80c1f50 2024-03-28T19:22:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.086+0000 7f82e4c00640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82c806bc00 0x7f82c806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.086+0000 7f82e4c00640 1 -- 172.21.15.27:0/3003797974 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f82d40ad540 con 0x7f82e80c1f50 2024-03-28T19:22:38.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.086+0000 7f82ecc00640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82c806bc00 0x7f82c806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:38.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.087+0000 7f82ecc00640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82c806bc00 0x7f82c806e0c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00a990 tx=0x7f82dc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:38.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.091+0000 7f82e4c00640 1 -- 172.21.15.27:0/3003797974 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f82d4040070 con 0x7f82e80c1f50 2024-03-28T19:22:38.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.389+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82e80a9190 con 0x7f82e80c1f50 2024-03-28T19:22:38.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.391+0000 7f82e4c00640 1 -- 172.21.15.27:0/3003797974 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f82d407b140 con 0x7f82e80c1f50 2024-03-28T19:22:38.390 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:38.390 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:38.390 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 199 2024-03-28T19:22:38.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82c806bc00 msgr2=0x7f82c806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82c806bc00 0x7f82c806e0c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00a990 tx=0x7f82dc008040 comp rx=0 tx=0).stop 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 msgr2=0x7f82e81ec690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e81ec690 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f82d40090b0 tx=0x7f82d40046c0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 shutdown_connections 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f82c806bc00 0x7f82c806e0c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82e80c5e80 0x7f82e8151380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82e80c1f50 0x7f82e81ec690 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 --2- 172.21.15.27:0/3003797974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82e80abd90 0x7f82e81ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:38.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 >> 172.21.15.27:0/3003797974 conn(0x7f82e806b9d0 msgr2=0x7f82e80a88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:38.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 shutdown_connections 2024-03-28T19:22:38.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:38.395+0000 7f82ef600640 1 -- 172.21.15.27:0/3003797974 wait complete. 2024-03-28T19:22:38.415 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:22:40.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:40.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.657+0000 7f8c81400640 1 Processor -- start 2024-03-28T19:22:40.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.657+0000 7f8c81400640 1 -- start start 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.658+0000 7f8c81400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c0a9cb0 0x7f8c7c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.659+0000 7f8c81400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c1534c0 0x7f8c7c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.659+0000 7f8c81400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0c3d90 con 0x7f8c7c150c70 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.659+0000 7f8c81400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c0ab4e0 con 0x7f8c7c1534c0 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.659+0000 7f8c81400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0ac1f0 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.659+0000 7f8c7b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c1534c0 0x7f8c7c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.659+0000 7f8c7b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8c7c1534c0 0x7f8c7c153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37288/0 (socket says 172.21.15.27:37288) 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.659+0000 7f8c7b200640 1 -- 172.21.15.27:0/1303955533 learned_addr learned my addr 172.21.15.27:0/1303955533 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:40.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.659+0000 7f8c79e00640 1 --2- 172.21.15.27:0/1303955533 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c0a9cb0 0x7f8c7c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:40.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.660+0000 7f8c7b200640 1 -- 172.21.15.27:0/1303955533 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c0a9cb0 msgr2=0x7f8c7c0c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:40.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.660+0000 7f8c7b200640 1 --2- 172.21.15.27:0/1303955533 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c0a9cb0 0x7f8c7c0c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.660+0000 7f8c7b200640 1 -- 172.21.15.27:0/1303955533 >> v1:172.21.15.155:6789/0 conn(0x7f8c7c150c70 legacy=0x7f8c7c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:40.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.660+0000 7f8c7b200640 1 -- 172.21.15.27:0/1303955533 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0ab710 con 0x7f8c7c1534c0 2024-03-28T19:22:40.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.660+0000 7f8c79e00640 1 --2- 172.21.15.27:0/1303955533 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c0a9cb0 0x7f8c7c0c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:22:40.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.661+0000 7f8c7b200640 1 --2- 172.21.15.27:0/1303955533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c1534c0 0x7f8c7c153950 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009e80 tx=0x7f8c7402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593473d520711e51 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:40.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.662+0000 7f8c79400640 1 -- 172.21.15.27:0/1303955533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7402c9c0 con 0x7f8c7c1534c0 2024-03-28T19:22:40.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.662+0000 7f8c79400640 1 -- 172.21.15.27:0/1303955533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7402cb60 con 0x7f8c7c1534c0 2024-03-28T19:22:40.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.662+0000 7f8c79400640 1 -- 172.21.15.27:0/1303955533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c740343f0 con 0x7f8c7c1534c0 2024-03-28T19:22:40.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.662+0000 7f8c81400640 1 -- 172.21.15.27:0/1303955533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c1534c0 msgr2=0x7f8c7c153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:40.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.662+0000 7f8c81400640 1 --2- 172.21.15.27:0/1303955533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c1534c0 0x7f8c7c153950 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009e80 tx=0x7f8c7402be20 comp rx=0 tx=0).stop 2024-03-28T19:22:40.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.663+0000 7f8c81400640 1 -- 172.21.15.27:0/1303955533 shutdown_connections 2024-03-28T19:22:40.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.663+0000 7f8c81400640 1 --2- 172.21.15.27:0/1303955533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c1534c0 0x7f8c7c153950 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.663+0000 7f8c81400640 1 --2- 172.21.15.27:0/1303955533 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c0a9cb0 0x7f8c7c0c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.663+0000 7f8c81400640 1 -- 172.21.15.27:0/1303955533 >> 172.21.15.27:0/1303955533 conn(0x7f8c7c06b9d0 msgr2=0x7f8c7c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:40.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.663+0000 7f8c81400640 1 -- 172.21.15.27:0/1303955533 shutdown_connections 2024-03-28T19:22:40.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.663+0000 7f8c81400640 1 -- 172.21.15.27:0/1303955533 wait complete. 2024-03-28T19:22:40.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.665+0000 7f8c81400640 1 Processor -- start 2024-03-28T19:22:40.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.665+0000 7f8c81400640 1 -- start start 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.666+0000 7f8c81400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0a9cb0 0x7f8c7c154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.666+0000 7f8c81400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c150c70 0x7f8c7c0c2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.666+0000 7f8c81400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c7c1534c0 0x7f8c7c1e9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.666+0000 7f8c81400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0c1810 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.666+0000 7f8c81400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0abcd0 con 0x7f8c7c1534c0 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.666+0000 7f8c81400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0c6830 con 0x7f8c7c150c70 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0a9cb0 0x7f8c7c154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0a9cb0 0x7f8c7c154cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37294/0 (socket says 172.21.15.27:37294) 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7a800640 1 -- 172.21.15.27:0/2280599230 learned_addr learned my addr 172.21.15.27:0/2280599230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:40.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c79e00640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c150c70 0x7f8c7c0c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:40.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7b200640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c7c1534c0 0x7f8c7c1e9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7a800640 1 -- 172.21.15.27:0/2280599230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c150c70 msgr2=0x7f8c7c0c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7a800640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c150c70 0x7f8c7c0c2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7a800640 1 -- 172.21.15.27:0/2280599230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c7c1534c0 msgr2=0x7f8c7c1e9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7a800640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c7c1534c0 0x7f8c7c1e9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.667+0000 7f8c7a800640 1 -- 172.21.15.27:0/2280599230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c1ec710 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.668+0000 7f8c7a800640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0a9cb0 0x7f8c7c154cb0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00c410 tx=0x7f8c6c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.668+0000 7f8c72a00640 1 -- 172.21.15.27:0/2280599230 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c017070 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.668+0000 7f8c72a00640 1 -- 172.21.15.27:0/2280599230 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c0035f0 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.668+0000 7f8c72a00640 1 -- 172.21.15.27:0/2280599230 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c003a20 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.668+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c1ec9b0 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.668+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c0c3040 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.670+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c04f400 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.671+0000 7f8c72a00640 1 -- 172.21.15.27:0/2280599230 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8c6c01a070 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.672+0000 7f8c72a00640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c5c06bc00 0x7f8c5c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:40.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.672+0000 7f8c72a00640 1 -- 172.21.15.27:0/2280599230 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f8c6c08aeb0 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.672+0000 7f8c79e00640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c5c06bc00 0x7f8c5c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:40.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.673+0000 7f8c79e00640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c5c06bc00 0x7f8c5c06e0c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0aae80 tx=0x7f8c68008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:40.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.679+0000 7f8c72a00640 1 -- 172.21.15.27:0/2280599230 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8c6c058a20 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.983+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7c0c3ad0 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.984+0000 7f8c72a00640 1 -- 172.21.15.27:0/2280599230 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f8c6c0010c0 con 0x7f8c7c0a9cb0 2024-03-28T19:22:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:40.984 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:40.984 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 199 2024-03-28T19:22:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c5c06bc00 msgr2=0x7f8c5c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c5c06bc00 0x7f8c5c06e0c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0aae80 tx=0x7f8c68008040 comp rx=0 tx=0).stop 2024-03-28T19:22:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0a9cb0 msgr2=0x7f8c7c154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0a9cb0 0x7f8c7c154cb0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00c410 tx=0x7f8c6c00c8d0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 shutdown_connections 2024-03-28T19:22:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f8c5c06bc00 0x7f8c5c06e0c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c7c1534c0 0x7f8c7c1e9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c7c150c70 0x7f8c7c0c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 --2- 172.21.15.27:0/2280599230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c7c0a9cb0 0x7f8c7c154cb0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:40.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 >> 172.21.15.27:0/2280599230 conn(0x7f8c7c06b9d0 msgr2=0x7f8c7c0c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:40.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 shutdown_connections 2024-03-28T19:22:40.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:40.988+0000 7f8c81400640 1 -- 172.21.15.27:0/2280599230 wait complete. 2024-03-28T19:22:41.012 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:22:42.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:22:42.253 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:22:43.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.249+0000 7f43ac600640 1 Processor -- start 2024-03-28T19:22:43.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.250+0000 7f43ac600640 1 -- start start 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.251+0000 7f43ac600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a40a9cb0 0x7f43a40c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.251+0000 7f43ac600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43a41534c0 0x7f43a4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.251+0000 7f43ac600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a40c3d90 con 0x7f43a4150c70 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.251+0000 7f43ac600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a40ab4e0 con 0x7f43a41534c0 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.251+0000 7f43ac600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40ac1f0 con 0x7f43a40a9cb0 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.252+0000 7f43aa600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43a41534c0 0x7f43a4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.252+0000 7f43aa600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43a41534c0 0x7f43a4153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37298/0 (socket says 172.21.15.27:37298) 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.252+0000 7f43aa600640 1 -- 172.21.15.27:0/4012313404 learned_addr learned my addr 172.21.15.27:0/4012313404 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:43.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.252+0000 7f43a9200640 1 --2- 172.21.15.27:0/4012313404 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a40a9cb0 0x7f43a40c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.252+0000 7f43aa600640 1 -- 172.21.15.27:0/4012313404 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a40a9cb0 msgr2=0x7f43a40c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.252+0000 7f43aa600640 1 --2- 172.21.15.27:0/4012313404 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a40a9cb0 0x7f43a40c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.252+0000 7f43aa600640 1 -- 172.21.15.27:0/4012313404 >> v1:172.21.15.155:6789/0 conn(0x7f43a4150c70 legacy=0x7f43a40c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.253+0000 7f43aa600640 1 -- 172.21.15.27:0/4012313404 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a40ab710 con 0x7f43a41534c0 2024-03-28T19:22:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.254+0000 7f43aa600640 1 --2- 172.21.15.27:0/4012313404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4153950 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f439c009e80 tx=0x7f439c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e0a844670e7c21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.255+0000 7f43a3e00640 1 -- 172.21.15.27:0/4012313404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c02ca50 con 0x7f43a41534c0 2024-03-28T19:22:43.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.255+0000 7f43a3e00640 1 -- 172.21.15.27:0/4012313404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c02cbf0 con 0x7f43a41534c0 2024-03-28T19:22:43.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.255+0000 7f43a3e00640 1 -- 172.21.15.27:0/4012313404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c0343f0 con 0x7f43a41534c0 2024-03-28T19:22:43.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.255+0000 7f43ac600640 1 -- 172.21.15.27:0/4012313404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 msgr2=0x7f43a4153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:43.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.255+0000 7f43ac600640 1 --2- 172.21.15.27:0/4012313404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4153950 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f439c009e80 tx=0x7f439c02beb0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.256+0000 7f43ac600640 1 -- 172.21.15.27:0/4012313404 shutdown_connections 2024-03-28T19:22:43.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.256+0000 7f43ac600640 1 --2- 172.21.15.27:0/4012313404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4153950 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.256+0000 7f43ac600640 1 --2- 172.21.15.27:0/4012313404 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a40a9cb0 0x7f43a40c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.256+0000 7f43ac600640 1 -- 172.21.15.27:0/4012313404 >> 172.21.15.27:0/4012313404 conn(0x7f43a406b9d0 msgr2=0x7f43a40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:43.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.256+0000 7f43ac600640 1 -- 172.21.15.27:0/4012313404 shutdown_connections 2024-03-28T19:22:43.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.256+0000 7f43ac600640 1 -- 172.21.15.27:0/4012313404 wait complete. 2024-03-28T19:22:43.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.258+0000 7f43ac600640 1 Processor -- start 2024-03-28T19:22:43.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.258+0000 7f43ac600640 1 -- start start 2024-03-28T19:22:43.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.259+0000 7f43ac600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43a40a9cb0 0x7f43a414d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.259+0000 7f43ac600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a4150c70 0x7f43a40719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43ac600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43ac600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c1810 con 0x7f43a41534c0 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43ac600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40abcd0 con 0x7f43a40a9cb0 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43ac600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c6830 con 0x7f43a4150c70 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43aa600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43aa600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37310/0 (socket says 172.21.15.27:37310) 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43aa600640 1 -- 172.21.15.27:0/616725876 learned_addr learned my addr 172.21.15.27:0/616725876 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43a9c00640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43a40a9cb0 0x7f43a414d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43a9200640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a4150c70 0x7f43a40719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43aa600640 1 -- 172.21.15.27:0/616725876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a4150c70 msgr2=0x7f43a40719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43aa600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a4150c70 0x7f43a40719e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43aa600640 1 -- 172.21.15.27:0/616725876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43a40a9cb0 msgr2=0x7f43a414d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:43.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43aa600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43a40a9cb0 0x7f43a414d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.260+0000 7f43aa600640 1 -- 172.21.15.27:0/616725876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4071f20 con 0x7f43a41534c0 2024-03-28T19:22:43.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.261+0000 7f43aa600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4152b50 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f439c009fa0 tx=0x7f439c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:43.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.261+0000 7f43a1600640 1 -- 172.21.15.27:0/616725876 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c043020 con 0x7f43a41534c0 2024-03-28T19:22:43.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.261+0000 7f43a1600640 1 -- 172.21.15.27:0/616725876 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c03e450 con 0x7f43a41534c0 2024-03-28T19:22:43.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.261+0000 7f43a1600640 1 -- 172.21.15.27:0/616725876 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c009510 con 0x7f43a41534c0 2024-03-28T19:22:43.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.261+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a41ea760 con 0x7f43a41534c0 2024-03-28T19:22:43.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.261+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a41ead20 con 0x7f43a41534c0 2024-03-28T19:22:43.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.263+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a4068500 con 0x7f43a41534c0 2024-03-28T19:22:43.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.264+0000 7f43a1600640 1 -- 172.21.15.27:0/616725876 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f439c046020 con 0x7f43a41534c0 2024-03-28T19:22:43.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.265+0000 7f43a1600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f438406bbd0 0x7f438406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.265+0000 7f43a1600640 1 -- 172.21.15.27:0/616725876 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9647+0+0 (secure 0 0 0) 0x7f439c0ae190 con 0x7f43a41534c0 2024-03-28T19:22:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.265+0000 7f43a9c00640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f438406bbd0 0x7f438406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:43.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.266+0000 7f43a9c00640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f438406bbd0 0x7f438406e090 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4398001640 tx=0x7f4398009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:43.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.272+0000 7f43a1600640 1 -- 172.21.15.27:0/616725876 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f439c04d030 con 0x7f43a41534c0 2024-03-28T19:22:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.590+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43a404f400 con 0x7f43a41534c0 2024-03-28T19:22:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.591+0000 7f43a1600640 1 -- 172.21.15.27:0/616725876 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f439c07bd60 con 0x7f43a41534c0 2024-03-28T19:22:43.590 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:43.591 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:21:11.827184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7273":{"gid":7273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":37,"addr":"172.21.15.155:6835/2331326731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":2331326731},{"type":"v1","addr":"172.21.15.155:6835","nonce":2331326731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7273,"qdb_cluster":[7273]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 199 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f438406bbd0 msgr2=0x7f438406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f438406bbd0 0x7f438406e090 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4398001640 tx=0x7f4398009040 comp rx=0 tx=0).stop 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 msgr2=0x7f43a4152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4152b50 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f439c009fa0 tx=0x7f439c004a40 comp rx=0 tx=0).stop 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 shutdown_connections 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f438406bbd0 0x7f438406e090 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43a41534c0 0x7f43a4152b50 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43a4150c70 0x7f43a40719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 --2- 172.21.15.27:0/616725876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43a40a9cb0 0x7f43a414d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 >> 172.21.15.27:0/616725876 conn(0x7f43a406b9d0 msgr2=0x7f43a40c2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 shutdown_connections 2024-03-28T19:22:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:43.594+0000 7f43ac600640 1 -- 172.21.15.27:0/616725876 wait complete. 2024-03-28T19:22:43.613 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:22:45.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:45.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.854+0000 7fc666200640 1 Processor -- start 2024-03-28T19:22:45.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.855+0000 7fc666200640 1 -- start start 2024-03-28T19:22:45.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.856+0000 7fc666200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6600c8ed0 0x7fc6600c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:45.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.857+0000 7fc666200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc6600be5d0 0x7fc660155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:45.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.857+0000 7fc666200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc660156230 con 0x7fc6600c58c0 2024-03-28T19:22:45.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.857+0000 7fc666200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc660158bd0 con 0x7fc6600c8ed0 2024-03-28T19:22:45.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.857+0000 7fc666200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc660158eb0 con 0x7fc6600be5d0 2024-03-28T19:22:45.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.857+0000 7fc65fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6600c8ed0 0x7fc6600c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:45.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.857+0000 7fc65fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6600c8ed0 0x7fc6600c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37318/0 (socket says 172.21.15.27:37318) 2024-03-28T19:22:45.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.857+0000 7fc65fe00640 1 -- 172.21.15.27:0/1416386110 learned_addr learned my addr 172.21.15.27:0/1416386110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:45.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.857+0000 7fc665800640 1 --2- 172.21.15.27:0/1416386110 >> v2:172.21.15.155:3300/0 conn(0x7fc6600be5d0 0x7fc660155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:45.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.858+0000 7fc65fe00640 1 -- 172.21.15.27:0/1416386110 >> v2:172.21.15.155:3300/0 conn(0x7fc6600be5d0 msgr2=0x7fc660155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:45.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.858+0000 7fc65fe00640 1 --2- 172.21.15.27:0/1416386110 >> v2:172.21.15.155:3300/0 conn(0x7fc6600be5d0 0x7fc660155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:45.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.858+0000 7fc65fe00640 1 -- 172.21.15.27:0/1416386110 >> v1:172.21.15.155:6789/0 conn(0x7fc6600c58c0 legacy=0x7fc6601555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:45.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.858+0000 7fc65fe00640 1 -- 172.21.15.27:0/1416386110 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc660159190 con 0x7fc6600c8ed0 2024-03-28T19:22:45.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.858+0000 7fc665800640 1 --2- 172.21.15.27:0/1416386110 >> v2:172.21.15.155:3300/0 conn(0x7fc6600be5d0 0x7fc660155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:22:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.859+0000 7fc65fe00640 1 --2- 172.21.15.27:0/1416386110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600c8ed0 0x7fc6600c5ca0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fc66008d050 tx=0x7fc64c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d39b8c8043dfc831 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.860+0000 7fc65f400640 1 -- 172.21.15.27:0/1416386110 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c02e070 con 0x7fc6600c8ed0 2024-03-28T19:22:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.860+0000 7fc65f400640 1 -- 172.21.15.27:0/1416386110 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c037070 con 0x7fc6600c8ed0 2024-03-28T19:22:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.860+0000 7fc65f400640 1 -- 172.21.15.27:0/1416386110 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c031a00 con 0x7fc6600c8ed0 2024-03-28T19:22:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.862+0000 7fc666200640 1 -- 172.21.15.27:0/1416386110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600c8ed0 msgr2=0x7fc6600c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.862+0000 7fc666200640 1 --2- 172.21.15.27:0/1416386110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600c8ed0 0x7fc6600c5ca0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fc66008d050 tx=0x7fc64c02b880 comp rx=0 tx=0).stop 2024-03-28T19:22:45.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.862+0000 7fc666200640 1 -- 172.21.15.27:0/1416386110 shutdown_connections 2024-03-28T19:22:45.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.862+0000 7fc666200640 1 --2- 172.21.15.27:0/1416386110 >> v2:172.21.15.155:3300/0 conn(0x7fc6600be5d0 0x7fc660155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:45.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.862+0000 7fc666200640 1 --2- 172.21.15.27:0/1416386110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600c8ed0 0x7fc6600c5ca0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:45.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.862+0000 7fc666200640 1 -- 172.21.15.27:0/1416386110 >> 172.21.15.27:0/1416386110 conn(0x7fc66006b6f0 msgr2=0x7fc6600acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:45.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.863+0000 7fc666200640 1 -- 172.21.15.27:0/1416386110 shutdown_connections 2024-03-28T19:22:45.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.863+0000 7fc666200640 1 -- 172.21.15.27:0/1416386110 wait complete. 2024-03-28T19:22:45.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.866+0000 7fc666200640 1 Processor -- start 2024-03-28T19:22:45.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.866+0000 7fc666200640 1 -- start start 2024-03-28T19:22:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.867+0000 7fc666200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600be5d0 0x7fc6601550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.867+0000 7fc666200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6600c58c0 0x7fc6600c2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.867+0000 7fc666200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6600c8ed0 0x7fc660151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.867+0000 7fc666200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600b0820 con 0x7fc6600be5d0 2024-03-28T19:22:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.867+0000 7fc666200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc660159750 con 0x7fc6600c8ed0 2024-03-28T19:22:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.867+0000 7fc666200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600b09a0 con 0x7fc6600c58c0 2024-03-28T19:22:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.868+0000 7fc664e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600be5d0 0x7fc6601550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.868+0000 7fc664e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600be5d0 0x7fc6601550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37320/0 (socket says 172.21.15.27:37320) 2024-03-28T19:22:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.868+0000 7fc664e00640 1 -- 172.21.15.27:0/951478186 learned_addr learned my addr 172.21.15.27:0/951478186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.868+0000 7fc65fe00640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6600c58c0 0x7fc6600c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.868+0000 7fc665800640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6600c8ed0 0x7fc660151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.869+0000 7fc664e00640 1 -- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6600c58c0 msgr2=0x7fc6600c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.869+0000 7fc664e00640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6600c58c0 0x7fc6600c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.869+0000 7fc664e00640 1 -- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6600c8ed0 msgr2=0x7fc660151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.869+0000 7fc664e00640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6600c8ed0 0x7fc660151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.869+0000 7fc664e00640 1 -- 172.21.15.27:0/951478186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6601eb710 con 0x7fc6600be5d0 2024-03-28T19:22:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.869+0000 7fc65fe00640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6600c58c0 0x7fc6600c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:22:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.869+0000 7fc664e00640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600be5d0 0x7fc6601550b0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c3c0 tx=0x7fc65400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.870+0000 7fc65cc00640 1 -- 172.21.15.27:0/951478186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6540052d0 con 0x7fc6600be5d0 2024-03-28T19:22:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.870+0000 7fc65cc00640 1 -- 172.21.15.27:0/951478186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc654005490 con 0x7fc6600be5d0 2024-03-28T19:22:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.870+0000 7fc65cc00640 1 -- 172.21.15.27:0/951478186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654007260 con 0x7fc6600be5d0 2024-03-28T19:22:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.870+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6601eb9f0 con 0x7fc6600be5d0 2024-03-28T19:22:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.870+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6601ebfb0 con 0x7fc6600be5d0 2024-03-28T19:22:45.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.871+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc660069640 con 0x7fc6600be5d0 2024-03-28T19:22:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.874+0000 7fc65cc00640 1 -- 172.21.15.27:0/951478186 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc654007400 con 0x7fc6600be5d0 2024-03-28T19:22:45.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.875+0000 7fc65cc00640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc64006bbd0 0x7fc64006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:45.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.876+0000 7fc65cc00640 1 -- 172.21.15.27:0/951478186 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7fc65408aed0 con 0x7fc6600be5d0 2024-03-28T19:22:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.876+0000 7fc65fe00640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc64006bbd0 0x7fc64006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.877+0000 7fc65fe00640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc64006bbd0 0x7fc64006e090 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc64c036000 tx=0x7fc64c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:45.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:45.881+0000 7fc65cc00640 1 -- 172.21.15.27:0/951478186 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc654058a40 con 0x7fc6600be5d0 2024-03-28T19:22:45.886 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 21% (44/201) Updating files: 22% (45/201) Updating files: 23% (47/201) Updating files: 24% (49/201) Updating files: 25% (51/201) Updating files: 26% (53/201) Updating files: 27% (55/201) Updating files: 28% (57/201) Updating files: 29% (59/201) Updating files: 30% (61/201) Updating files: 31% (63/201) Updating files: 32% (65/201) Updating files: 33% (67/201) Updating files: 34% (69/201) Updating files: 35% (71/201) Updating files: 36% (73/201) Updating files: 37% (75/201) Updating files: 38% (77/201) Updating files: 38% (78/201) Updating files: 39% (79/201) Updating files: 40% (81/201) Updating files: 41% (83/201) Updating files: 42% (85/201) Updating files: 43% (87/201) Updating files: 44% (89/201) Updating files: 45% (91/201) Updating files: 46% (93/201) Updating files: 47% (95/201) Updating files: 48% (97/201) Updating files: 49% (99/201) Updating files: 50% (101/201) Updating files: 51% (103/201) Updating files: 52% (105/201) Updating files: 53% (107/201) Updating files: 54% (109/201) Updating files: 55% (111/201) Updating files: 56% (113/201) Updating files: 57% (115/201) Updating files: 58% (117/201) Updating files: 59% (119/201) Updating files: 60% (121/201) Updating files: 61% (123/201) Updating files: 62% (125/201) Updating files: 63% (127/201) Updating files: 64% (129/201) Updating files: 65% (131/201) Updating files: 66% (133/201) Updating files: 67% (135/201) Updating files: 67% (136/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 92% (186/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2024-03-28T19:22:46.234 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2024-03-28T19:22:46.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.237+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc660156580 con 0x7fc6600be5d0 2024-03-28T19:22:46.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.238+0000 7fc65cc00640 1 -- 172.21.15.27:0/951478186 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fc654058340 con 0x7fc6600be5d0 2024-03-28T19:22:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:46.237 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:45.353898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 200 2024-03-28T19:22:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc64006bbd0 msgr2=0x7fc64006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc64006bbd0 0x7fc64006e090 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc64c036000 tx=0x7fc64c03d040 comp rx=0 tx=0).stop 2024-03-28T19:22:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600be5d0 msgr2=0x7fc6601550b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600be5d0 0x7fc6601550b0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c3c0 tx=0x7fc65400c880 comp rx=0 tx=0).stop 2024-03-28T19:22:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 shutdown_connections 2024-03-28T19:22:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc64006bbd0 0x7fc64006e090 secure :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc64c036000 tx=0x7fc64c03d040 comp rx=0 tx=0).stop 2024-03-28T19:22:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6600c8ed0 0x7fc660151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6600c58c0 0x7fc6600c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 --2- 172.21.15.27:0/951478186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6600be5d0 0x7fc6601550b0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 >> 172.21.15.27:0/951478186 conn(0x7fc66006b6f0 msgr2=0x7fc6601dec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 shutdown_connections 2024-03-28T19:22:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.240+0000 7fc666200640 1 -- 172.21.15.27:0/951478186 wait complete. 2024-03-28T19:22:46.250 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.7 2024-03-28T19:22:46.255 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-03-28T19:22:46.255 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:22:46.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:46.322 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:22:46.322 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 8 -eq 48 ']' 2024-03-28T19:22:46.322 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.8 2024-03-28T19:22:46.322 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.8 2024-03-28T19:22:46.322 INFO:tasks.workunit.client.1.smithi155.stdout:v0.8 2024-03-28T19:22:46.322 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.8 2024-03-28T19:22:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.516+0000 7f1e50800640 1 Processor -- start 2024-03-28T19:22:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.517+0000 7f1e50800640 1 -- start start 2024-03-28T19:22:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.518+0000 7f1e50800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e48158580 0x7f1e481560c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.518+0000 7f1e50800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e480be630 0x7f1e480c5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e50800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1e480c5e40 0x7f1e48159150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e50800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e480c76e0 con 0x7f1e480be630 2024-03-28T19:22:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e50800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e480c6dd0 con 0x7f1e480c5e40 2024-03-28T19:22:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e50800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e48158cc0 con 0x7f1e48158580 2024-03-28T19:22:46.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e4d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e480be630 0x7f1e480c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e4d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e480be630 0x7f1e480c5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53822/0 (socket says 172.21.15.27:53822) 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e4d400640 1 -- 172.21.15.27:0/771092532 learned_addr learned my addr 172.21.15.27:0/771092532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e4de00640 1 --2- 172.21.15.27:0/771092532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e48158580 0x7f1e481560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.519+0000 7f1e4e800640 1 --2- 172.21.15.27:0/771092532 >> v2:172.21.15.155:3300/0 conn(0x7f1e480c5e40 0x7f1e48159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.520+0000 7f1e4d400640 1 -- 172.21.15.27:0/771092532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e48158580 msgr2=0x7f1e481560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.520+0000 7f1e4d400640 1 --2- 172.21.15.27:0/771092532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e48158580 0x7f1e481560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.520+0000 7f1e4d400640 1 -- 172.21.15.27:0/771092532 >> v2:172.21.15.155:3300/0 conn(0x7f1e480c5e40 msgr2=0x7f1e48159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.520+0000 7f1e4d400640 1 --2- 172.21.15.27:0/771092532 >> v2:172.21.15.155:3300/0 conn(0x7f1e480c5e40 0x7f1e48159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.520+0000 7f1e4d400640 1 -- 172.21.15.27:0/771092532 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e48156600 con 0x7f1e480be630 2024-03-28T19:22:46.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.520+0000 7f1e4e800640 1 --2- 172.21.15.27:0/771092532 >> v2:172.21.15.155:3300/0 conn(0x7f1e480c5e40 0x7f1e48159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:22:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.521+0000 7f1e4d400640 1 --2- 172.21.15.27:0/771092532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e480c5900 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f1e480caf30 tx=0x7f1e3802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73dbaa2ee5d1b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.522+0000 7f1e4ca00640 1 -- 172.21.15.27:0/771092532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38035020 con 0x7f1e480be630 2024-03-28T19:22:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.522+0000 7f1e4ca00640 1 -- 172.21.15.27:0/771092532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3802d040 con 0x7f1e480be630 2024-03-28T19:22:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.522+0000 7f1e4ca00640 1 -- 172.21.15.27:0/771092532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38031a80 con 0x7f1e480be630 2024-03-28T19:22:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 -- 172.21.15.27:0/771092532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 msgr2=0x7f1e480c5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 --2- 172.21.15.27:0/771092532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e480c5900 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f1e480caf30 tx=0x7f1e3802b900 comp rx=0 tx=0).stop 2024-03-28T19:22:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 -- 172.21.15.27:0/771092532 shutdown_connections 2024-03-28T19:22:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 --2- 172.21.15.27:0/771092532 >> v2:172.21.15.155:3300/0 conn(0x7f1e480c5e40 0x7f1e48159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 --2- 172.21.15.27:0/771092532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e480c5900 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 --2- 172.21.15.27:0/771092532 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e48158580 0x7f1e481560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 -- 172.21.15.27:0/771092532 >> 172.21.15.27:0/771092532 conn(0x7f1e480caf80 msgr2=0x7f1e480cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 -- 172.21.15.27:0/771092532 shutdown_connections 2024-03-28T19:22:46.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.523+0000 7f1e50800640 1 -- 172.21.15.27:0/771092532 wait complete. 2024-03-28T19:22:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.526+0000 7f1e50800640 1 Processor -- start 2024-03-28T19:22:46.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.526+0000 7f1e50800640 1 -- start start 2024-03-28T19:22:46.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.527+0000 7f1e50800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e48144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.527+0000 7f1e50800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e480c5e40 0x7f1e480c3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.527+0000 7f1e50800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e48158580 0x7f1e480abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.527+0000 7f1e50800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480c7ed0 con 0x7f1e480be630 2024-03-28T19:22:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.527+0000 7f1e50800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480c7bf0 con 0x7f1e48158580 2024-03-28T19:22:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.527+0000 7f1e50800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480c81b0 con 0x7f1e480c5e40 2024-03-28T19:22:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e48144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e48144ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53838/0 (socket says 172.21.15.27:53838) 2024-03-28T19:22:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4de00640 1 -- 172.21.15.27:0/1639282078 learned_addr learned my addr 172.21.15.27:0/1639282078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4e800640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e48158580 0x7f1e480abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4d400640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e480c5e40 0x7f1e480c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4de00640 1 -- 172.21.15.27:0/1639282078 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e480c5e40 msgr2=0x7f1e480c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4de00640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e480c5e40 0x7f1e480c3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4de00640 1 -- 172.21.15.27:0/1639282078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e48158580 msgr2=0x7f1e480abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4de00640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e48158580 0x7f1e480abe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.528+0000 7f1e4de00640 1 -- 172.21.15.27:0/1639282078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e481f0480 con 0x7f1e480be630 2024-03-28T19:22:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.529+0000 7f1e4de00640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e48144ce0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00e870 tx=0x7f1e3c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.529+0000 7f1e42000640 1 -- 172.21.15.27:0/1639282078 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c017070 con 0x7f1e480be630 2024-03-28T19:22:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.529+0000 7f1e42000640 1 -- 172.21.15.27:0/1639282078 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c00c940 con 0x7f1e480be630 2024-03-28T19:22:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.529+0000 7f1e42000640 1 -- 172.21.15.27:0/1639282078 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c00cd30 con 0x7f1e480be630 2024-03-28T19:22:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.530+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e481ef600 con 0x7f1e480be630 2024-03-28T19:22:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.530+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e480c24d0 con 0x7f1e480be630 2024-03-28T19:22:46.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.531+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4804f400 con 0x7f1e480be630 2024-03-28T19:22:46.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.532+0000 7f1e42000640 1 -- 172.21.15.27:0/1639282078 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1e3c01a070 con 0x7f1e480be630 2024-03-28T19:22:46.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.533+0000 7f1e42000640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:46.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.533+0000 7f1e42000640 1 -- 172.21.15.27:0/1639282078 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f1e3c08b870 con 0x7f1e480be630 2024-03-28T19:22:46.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.534+0000 7f1e4d400640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:46.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.535+0000 7f1e4d400640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1e38036000 tx=0x7f1e3803c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:46.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.543+0000 7f1e42000640 1 -- 172.21.15.27:0/1639282078 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1e3c0593e0 con 0x7f1e480be630 2024-03-28T19:22:46.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.851+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e480b0810 con 0x7f1e480be630 2024-03-28T19:22:46.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.852+0000 7f1e42000640 1 -- 172.21.15.27:0/1639282078 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f1e3c058ce0 con 0x7f1e480be630 2024-03-28T19:22:46.851 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:46.851 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:45.353898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:46.851 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 200 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.854+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 msgr2=0x7f1e2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.854+0000 7f1e50800640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1e38036000 tx=0x7f1e3803c040 comp rx=0 tx=0).stop 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.854+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 msgr2=0x7f1e48144ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.854+0000 7f1e50800640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e48144ce0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00e870 tx=0x7f1e3c00ed30 comp rx=0 tx=0).stop 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.855+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 shutdown_connections 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.855+0000 7f1e50800640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e2c06bbd0 0x7f1e2c06e090 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.855+0000 7f1e50800640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e48158580 0x7f1e480abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.855+0000 7f1e50800640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e480c5e40 0x7f1e480c3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.855+0000 7f1e50800640 1 --2- 172.21.15.27:0/1639282078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e480be630 0x7f1e48144ce0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.855+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 >> 172.21.15.27:0/1639282078 conn(0x7f1e480caf80 msgr2=0x7f1e48151480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:46.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.855+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 shutdown_connections 2024-03-28T19:22:46.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:46.855+0000 7f1e50800640 1 -- 172.21.15.27:0/1639282078 wait complete. 2024-03-28T19:22:46.874 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 200, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 200, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8259": { "addr": "172.21.15.27:6835/1299974557", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1299974557, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 1299974557, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8259, "incarnation": 200, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 88, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:22:45.353898+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8259 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 192, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8052": { "addr": "172.21.15.27:6837/433723712", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 433723712, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 433723712, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8052, "incarnation": 188, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 86, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:20:51.695880+0000", "qdb_cluster": [ 8052 ], "qdb_leader": 8052, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8052 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3334295790", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3334295790, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3334295790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 193, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6467, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:22:48.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:49.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.153+0000 7f1b76200640 1 Processor -- start 2024-03-28T19:22:49.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.153+0000 7f1b76200640 1 -- start start 2024-03-28T19:22:49.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.155+0000 7f1b76200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b700c5e80 0x7f1b700c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:49.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.155+0000 7f1b76200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7006d8a0 con 0x7f1b700c1f50 2024-03-28T19:22:49.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.155+0000 7f1b76200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7006db80 con 0x7f1b700abf60 2024-03-28T19:22:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.155+0000 7f1b76200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7006d650 con 0x7f1b700c5e80 2024-03-28T19:22:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.155+0000 7f1b6f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1b700c5e80 0x7f1b700c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.155+0000 7f1b6ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1b700c1f50 0x7f1b700ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40160/0 (socket says 172.21.15.27:40160) 2024-03-28T19:22:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.155+0000 7f1b6ea00640 1 -- 172.21.15.27:0/2696599644 learned_addr learned my addr 172.21.15.27:0/2696599644 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:22:49.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.156+0000 7f1b6f400640 1 -- 172.21.15.27:0/2696599644 >> v1:172.21.15.155:6789/0 conn(0x7f1b700abf60 legacy=0x7f1b700aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.156+0000 7f1b6f400640 1 -- 172.21.15.27:0/2696599644 >> v1:172.21.15.27:6789/0 conn(0x7f1b700c1f50 legacy=0x7f1b700ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.156+0000 7f1b6f400640 1 -- 172.21.15.27:0/2696599644 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7006de60 con 0x7f1b700c5e80 2024-03-28T19:22:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.156+0000 7f1b6e000640 1 -- 172.21.15.27:0/2696599644 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708149541 0 0) 0x7f1b7006d8a0 con 0x7f1b700c1f50 2024-03-28T19:22:49.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.157+0000 7f1b6f400640 1 --2- 172.21.15.27:0/2696599644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700c5e80 0x7f1b700c2cb0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000ba40 tx=0x7f1b6002e010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e3503973c66659 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:49.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.158+0000 7f1b6e000640 1 -- 172.21.15.27:0/2696599644 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6003b070 con 0x7f1b700c5e80 2024-03-28T19:22:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.158+0000 7f1b6e000640 1 -- 172.21.15.27:0/2696599644 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6002edc0 con 0x7f1b700c5e80 2024-03-28T19:22:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.158+0000 7f1b6e000640 1 -- 172.21.15.27:0/2696599644 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60036660 con 0x7f1b700c5e80 2024-03-28T19:22:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.158+0000 7f1b76200640 1 -- 172.21.15.27:0/2696599644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700c5e80 msgr2=0x7f1b700c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.158+0000 7f1b76200640 1 --2- 172.21.15.27:0/2696599644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700c5e80 0x7f1b700c2cb0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000ba40 tx=0x7f1b6002e010 comp rx=0 tx=0).stop 2024-03-28T19:22:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.158+0000 7f1b76200640 1 -- 172.21.15.27:0/2696599644 shutdown_connections 2024-03-28T19:22:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.158+0000 7f1b76200640 1 --2- 172.21.15.27:0/2696599644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700c5e80 0x7f1b700c2cb0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.158+0000 7f1b76200640 1 -- 172.21.15.27:0/2696599644 >> 172.21.15.27:0/2696599644 conn(0x7f1b7006b9d0 msgr2=0x7f1b700c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.159+0000 7f1b76200640 1 -- 172.21.15.27:0/2696599644 shutdown_connections 2024-03-28T19:22:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.159+0000 7f1b76200640 1 -- 172.21.15.27:0/2696599644 wait complete. 2024-03-28T19:22:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.160+0000 7f1b76200640 1 Processor -- start 2024-03-28T19:22:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.161+0000 7f1b76200640 1 -- start start 2024-03-28T19:22:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.161+0000 7f1b76200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700abf60 0x7f1b7006ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.161+0000 7f1b76200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b700c1f50 0x7f1b701dc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.161+0000 7f1b76200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b700c5e80 0x7f1b701dc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.161+0000 7f1b76200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700c6260 con 0x7f1b700abf60 2024-03-28T19:22:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.161+0000 7f1b76200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7006e1d0 con 0x7f1b700c5e80 2024-03-28T19:22:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.161+0000 7f1b76200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700c63e0 con 0x7f1b700c1f50 2024-03-28T19:22:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700abf60 0x7f1b7006ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700abf60 0x7f1b7006ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53848/0 (socket says 172.21.15.27:53848) 2024-03-28T19:22:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6f400640 1 -- 172.21.15.27:0/1221522920 learned_addr learned my addr 172.21.15.27:0/1221522920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6f400640 1 -- 172.21.15.27:0/1221522920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b700c1f50 msgr2=0x7f1b701dc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6ea00640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b700c1f50 0x7f1b701dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6fe00640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b700c5e80 0x7f1b701dc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6f400640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b700c1f50 0x7f1b701dc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6f400640 1 -- 172.21.15.27:0/1221522920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b700c5e80 msgr2=0x7f1b701dc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6f400640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b700c5e80 0x7f1b701dc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.162+0000 7f1b6f400640 1 -- 172.21.15.27:0/1221522920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b70072870 con 0x7f1b700abf60 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.163+0000 7f1b6f400640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700abf60 0x7f1b7006ce90 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1b60039040 tx=0x7f1b600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.163+0000 7f1b67400640 1 -- 172.21.15.27:0/1221522920 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60044070 con 0x7f1b700abf60 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.163+0000 7f1b67400640 1 -- 172.21.15.27:0/1221522920 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6003f640 con 0x7f1b700abf60 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.163+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b700a8720 con 0x7f1b700abf60 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.163+0000 7f1b67400640 1 -- 172.21.15.27:0/1221522920 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6003b040 con 0x7f1b700abf60 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.163+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7006d3d0 con 0x7f1b700abf60 2024-03-28T19:22:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.165+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7004f400 con 0x7f1b700abf60 2024-03-28T19:22:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.167+0000 7f1b67400640 1 -- 172.21.15.27:0/1221522920 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b60047070 con 0x7f1b700abf60 2024-03-28T19:22:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.168+0000 7f1b67400640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1b5006bc00 0x7f1b5006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:49.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.168+0000 7f1b67400640 1 -- 172.21.15.27:0/1221522920 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f1b600ad0d0 con 0x7f1b700abf60 2024-03-28T19:22:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.170+0000 7f1b6ea00640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1b5006bc00 0x7f1b5006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:49.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.171+0000 7f1b6ea00640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1b5006bc00 0x7f1b5006e0c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1b700a9760 tx=0x7f1b5c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:49.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.173+0000 7f1b67400640 1 -- 172.21.15.27:0/1221522920 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b70072870 con 0x7f1b700abf60 2024-03-28T19:22:49.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.551+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b701eebe0 con 0x7f1b700abf60 2024-03-28T19:22:49.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.552+0000 7f1b67400640 1 -- 172.21.15.27:0/1221522920 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f1b6007abe0 con 0x7f1b700abf60 2024-03-28T19:22:49.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:49.551 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:45.353898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:49.551 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 200 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1b5006bc00 msgr2=0x7f1b5006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1b5006bc00 0x7f1b5006e0c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1b700a9760 tx=0x7f1b5c009040 comp rx=0 tx=0).stop 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700abf60 msgr2=0x7f1b7006ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700abf60 0x7f1b7006ce90 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1b60039040 tx=0x7f1b600046c0 comp rx=0 tx=0).stop 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 shutdown_connections 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1b5006bc00 0x7f1b5006e0c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b700c5e80 0x7f1b701dc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b700c1f50 0x7f1b701dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:49.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 --2- 172.21.15.27:0/1221522920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b700abf60 0x7f1b7006ce90 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:49.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 >> 172.21.15.27:0/1221522920 conn(0x7f1b7006b9d0 msgr2=0x7f1b701562b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:49.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 shutdown_connections 2024-03-28T19:22:49.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:49.555+0000 7f1b76200640 1 -- 172.21.15.27:0/1221522920 wait complete. 2024-03-28T19:22:49.572 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:22:51.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:51.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.788+0000 7fc67fe00640 1 Processor -- start 2024-03-28T19:22:51.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.788+0000 7fc67fe00640 1 -- start start 2024-03-28T19:22:51.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.789+0000 7fc67fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6800c8480 0x7fc6800c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:51.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.789+0000 7fc67fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6800bdbd0 con 0x7fc6800a8240 2024-03-28T19:22:51.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.790+0000 7fc67fe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6800bf5a0 con 0x7fc6800c52f0 2024-03-28T19:22:51.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.790+0000 7fc67fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6800bf910 con 0x7fc6800c8480 2024-03-28T19:22:51.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.790+0000 7fc67e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6800c8480 0x7fc6800c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:51.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.790+0000 7fc67e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6800c8480 0x7fc6800c56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53854/0 (socket says 172.21.15.27:53854) 2024-03-28T19:22:51.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.790+0000 7fc67e000640 1 -- 172.21.15.27:0/3995978581 learned_addr learned my addr 172.21.15.27:0/3995978581 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:51.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.791+0000 7fc67d600640 1 -- 172.21.15.27:0/3995978581 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704633006 0 0) 0x7fc6800bdbd0 con 0x7fc6800a8240 2024-03-28T19:22:51.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.791+0000 7fc67d600640 1 -- 172.21.15.27:0/3995978581 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc664003520 con 0x7fc6800a8240 2024-03-28T19:22:51.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.792+0000 7fc67d600640 1 -- 172.21.15.27:0/3995978581 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999245099 0 0) 0x7fc6800bf5a0 con 0x7fc6800c52f0 2024-03-28T19:22:51.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.792+0000 7fc67d600640 1 -- 172.21.15.27:0/3995978581 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc664003880 con 0x7fc6800c52f0 2024-03-28T19:22:51.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.792+0000 7fc67e000640 1 -- 172.21.15.27:0/3995978581 >> v1:172.21.15.155:6789/0 conn(0x7fc6800c52f0 legacy=0x7fc680153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:51.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.792+0000 7fc67e000640 1 -- 172.21.15.27:0/3995978581 >> v1:172.21.15.27:6789/0 conn(0x7fc6800a8240 legacy=0x7fc68015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:51.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.792+0000 7fc67e000640 1 -- 172.21.15.27:0/3995978581 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680153a70 con 0x7fc6800c8480 2024-03-28T19:22:51.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.793+0000 7fc67e000640 1 --2- 172.21.15.27:0/3995978581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800c8480 0x7fc6800c56d0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fc6800bfa90 tx=0x7fc66c02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18b0014d33c3ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.794+0000 7fc67d600640 1 -- 172.21.15.27:0/3995978581 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c02f830 con 0x7fc6800c8480 2024-03-28T19:22:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.794+0000 7fc67d600640 1 -- 172.21.15.27:0/3995978581 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66c02f9d0 con 0x7fc6800c8480 2024-03-28T19:22:51.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.794+0000 7fc67d600640 1 -- 172.21.15.27:0/3995978581 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66c02fda0 con 0x7fc6800c8480 2024-03-28T19:22:51.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.795+0000 7fc67fe00640 1 -- 172.21.15.27:0/3995978581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800c8480 msgr2=0x7fc6800c56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:51.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.795+0000 7fc67fe00640 1 --2- 172.21.15.27:0/3995978581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800c8480 0x7fc6800c56d0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fc6800bfa90 tx=0x7fc66c02ba10 comp rx=0 tx=0).stop 2024-03-28T19:22:51.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.795+0000 7fc67fe00640 1 -- 172.21.15.27:0/3995978581 shutdown_connections 2024-03-28T19:22:51.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.795+0000 7fc67fe00640 1 --2- 172.21.15.27:0/3995978581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800c8480 0x7fc6800c56d0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:51.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.795+0000 7fc67fe00640 1 -- 172.21.15.27:0/3995978581 >> 172.21.15.27:0/3995978581 conn(0x7fc6800aa7c0 msgr2=0x7fc6800aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:51.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.795+0000 7fc67fe00640 1 -- 172.21.15.27:0/3995978581 shutdown_connections 2024-03-28T19:22:51.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.796+0000 7fc67fe00640 1 -- 172.21.15.27:0/3995978581 wait complete. 2024-03-28T19:22:51.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.798+0000 7fc67fe00640 1 Processor -- start 2024-03-28T19:22:51.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.798+0000 7fc67fe00640 1 -- start start 2024-03-28T19:22:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.799+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800a8240 0x7fc68006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.799+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6800c52f0 0x7fc6800702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.799+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6800c8480 0x7fc6801546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.799+0000 7fc67fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800a7e00 con 0x7fc6800a8240 2024-03-28T19:22:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.799+0000 7fc67fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800a7c80 con 0x7fc6800c52f0 2024-03-28T19:22:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.799+0000 7fc67fe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc680157530 con 0x7fc6800c8480 2024-03-28T19:22:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.800+0000 7fc67f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6800c8480 0x7fc6801546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:51.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.800+0000 7fc67f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6800c8480 0x7fc6801546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:57650/0 (socket says 172.21.15.27:57650) 2024-03-28T19:22:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.800+0000 7fc67f400640 1 -- 172.21.15.27:0/2343487769 learned_addr learned my addr 172.21.15.27:0/2343487769 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.800+0000 7fc67e000640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6800c52f0 0x7fc6800702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:51.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.800+0000 7fc67ea00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800a8240 0x7fc68006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.801+0000 7fc67ea00640 1 -- 172.21.15.27:0/2343487769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6800c8480 msgr2=0x7fc6801546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.801+0000 7fc67ea00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6800c8480 0x7fc6801546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.801+0000 7fc67ea00640 1 -- 172.21.15.27:0/2343487769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6800c52f0 msgr2=0x7fc6800702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.801+0000 7fc67ea00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6800c52f0 0x7fc6800702d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.801+0000 7fc67ea00640 1 -- 172.21.15.27:0/2343487769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680151760 con 0x7fc6800a8240 2024-03-28T19:22:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.801+0000 7fc67f400640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6800c8480 0x7fc6801546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:22:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.801+0000 7fc67ea00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800a8240 0x7fc68006eb80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fc67000c400 tx=0x7fc67000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:51.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.802+0000 7fc677400640 1 -- 172.21.15.27:0/2343487769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670017070 con 0x7fc6800a8240 2024-03-28T19:22:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.802+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc680151cf0 con 0x7fc6800a8240 2024-03-28T19:22:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.802+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6801522b0 con 0x7fc6800a8240 2024-03-28T19:22:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.802+0000 7fc677400640 1 -- 172.21.15.27:0/2343487769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6700052d0 con 0x7fc6800a8240 2024-03-28T19:22:51.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.802+0000 7fc677400640 1 -- 172.21.15.27:0/2343487769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670003810 con 0x7fc6800a8240 2024-03-28T19:22:51.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.804+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc658005290 con 0x7fc6800a8240 2024-03-28T19:22:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.805+0000 7fc677400640 1 -- 172.21.15.27:0/2343487769 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc67001a070 con 0x7fc6800a8240 2024-03-28T19:22:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.808+0000 7fc677400640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc66006bbd0 0x7fc66006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.808+0000 7fc677400640 1 -- 172.21.15.27:0/2343487769 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7fc67008ac90 con 0x7fc6800a8240 2024-03-28T19:22:51.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.808+0000 7fc67e000640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc66006bbd0 0x7fc66006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:51.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.809+0000 7fc67e000640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc66006bbd0 0x7fc66006e090 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc66c036000 tx=0x7fc66c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:51.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:51.813+0000 7fc677400640 1 -- 172.21.15.27:0/2343487769 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc670058150 con 0x7fc6800a8240 2024-03-28T19:22:52.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:52.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.212+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc658005580 con 0x7fc6800a8240 2024-03-28T19:22:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.213+0000 7fc677400640 1 -- 172.21.15.27:0/2343487769 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fc67005c1f0 con 0x7fc6800a8240 2024-03-28T19:22:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:52.212 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:45.353898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:52.212 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 200 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.219+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc66006bbd0 msgr2=0x7fc66006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.219+0000 7fc67fe00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc66006bbd0 0x7fc66006e090 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc66c036000 tx=0x7fc66c008030 comp rx=0 tx=0).stop 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.219+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800a8240 msgr2=0x7fc68006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.219+0000 7fc67fe00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800a8240 0x7fc68006eb80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fc67000c400 tx=0x7fc67000c8c0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.220+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 shutdown_connections 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.220+0000 7fc67fe00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc66006bbd0 0x7fc66006e090 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.220+0000 7fc67fe00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc6800c8480 0x7fc6801546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.220+0000 7fc67fe00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6800c52f0 0x7fc6800702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.220+0000 7fc67fe00640 1 --2- 172.21.15.27:0/2343487769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6800a8240 0x7fc68006eb80 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.220+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 >> 172.21.15.27:0/2343487769 conn(0x7fc6800aa7c0 msgr2=0x7fc680156c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.220+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 shutdown_connections 2024-03-28T19:22:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.220+0000 7fc67fe00640 1 -- 172.21.15.27:0/2343487769 wait complete. 2024-03-28T19:22:52.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.236+0000 7f102c400640 1 Processor -- start 2024-03-28T19:22:52.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.237+0000 7f102c400640 1 -- start start 2024-03-28T19:22:52.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.238+0000 7f102c400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f10240c60e0 0x7f102406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.238+0000 7f102c400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 0x7f102406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.238+0000 7f102c400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240c1600 con 0x7f1024072df0 2024-03-28T19:22:52.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.238+0000 7f102c400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10240c23b0 con 0x7f10240c60e0 2024-03-28T19:22:52.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.238+0000 7f102c400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10240c18e0 con 0x7f10240c3ea0 2024-03-28T19:22:52.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.239+0000 7f1029a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f10240c60e0 0x7f102406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.239+0000 7f1029a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f10240c60e0 0x7f102406e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:57980/0 (socket says 172.21.15.27:57980) 2024-03-28T19:22:52.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.239+0000 7f1029a00640 1 -- 172.21.15.27:0/3282946053 learned_addr learned my addr 172.21.15.27:0/3282946053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:52.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.239+0000 7f1029000640 1 --2- 172.21.15.27:0/3282946053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 0x7f102406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.239+0000 7f1023e00640 1 -- 172.21.15.27:0/3282946053 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062334257 0 0) 0x7f10240c1600 con 0x7f1024072df0 2024-03-28T19:22:52.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.239+0000 7f1023e00640 1 -- 172.21.15.27:0/3282946053 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f100c003520 con 0x7f1024072df0 2024-03-28T19:22:52.239 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:22:52.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.240+0000 7f1029a00640 1 -- 172.21.15.27:0/3282946053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 msgr2=0x7f102406ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:52.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.240+0000 7f1029a00640 1 --2- 172.21.15.27:0/3282946053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 0x7f102406ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.240+0000 7f1029a00640 1 -- 172.21.15.27:0/3282946053 >> v1:172.21.15.27:6789/0 conn(0x7f1024072df0 legacy=0x7f1024158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.240+0000 7f1029a00640 1 -- 172.21.15.27:0/3282946053 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240c1bc0 con 0x7f10240c60e0 2024-03-28T19:22:52.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.241+0000 7f1029a00640 1 --2- 172.21.15.27:0/3282946053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 0x7f102406e6f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1018009870 tx=0x7f101802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a69700f66d2722d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:52.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.242+0000 7f1023e00640 1 -- 172.21.15.27:0/3282946053 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10180317c0 con 0x7f10240c60e0 2024-03-28T19:22:52.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.242+0000 7f1023e00640 1 -- 172.21.15.27:0/3282946053 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1018037070 con 0x7f10240c60e0 2024-03-28T19:22:52.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.242+0000 7f1023e00640 1 -- 172.21.15.27:0/3282946053 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018031b60 con 0x7f10240c60e0 2024-03-28T19:22:52.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.242+0000 7f102c400640 1 -- 172.21.15.27:0/3282946053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 msgr2=0x7f102406e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.242+0000 7f102c400640 1 --2- 172.21.15.27:0/3282946053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 0x7f102406e6f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1018009870 tx=0x7f101802b860 comp rx=0 tx=0).stop 2024-03-28T19:22:52.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.243+0000 7f102c400640 1 -- 172.21.15.27:0/3282946053 shutdown_connections 2024-03-28T19:22:52.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.243+0000 7f102c400640 1 --2- 172.21.15.27:0/3282946053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 0x7f102406ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.243+0000 7f102c400640 1 --2- 172.21.15.27:0/3282946053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 0x7f102406e6f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.243+0000 7f102c400640 1 -- 172.21.15.27:0/3282946053 >> 172.21.15.27:0/3282946053 conn(0x7f102406b9d0 msgr2=0x7f10240c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:52.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.243+0000 7f102c400640 1 -- 172.21.15.27:0/3282946053 shutdown_connections 2024-03-28T19:22:52.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.243+0000 7f102c400640 1 -- 172.21.15.27:0/3282946053 wait complete. 2024-03-28T19:22:52.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.245+0000 7f102c400640 1 Processor -- start 2024-03-28T19:22:52.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.245+0000 7f102c400640 1 -- start start 2024-03-28T19:22:52.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f102c400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1024072df0 0x7f102414b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f102c400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 0x7f10240c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f102c400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 0x7f10241ead30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f102c400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240c6640 con 0x7f1024072df0 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f102c400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10240c64c0 con 0x7f10240c60e0 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f102c400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10240c4f00 con 0x7f10240c3ea0 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f1029a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1024072df0 0x7f102414b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f1029a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1024072df0 0x7f102414b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53868/0 (socket says 172.21.15.27:53868) 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f1029a00640 1 -- 172.21.15.27:0/77937509 learned_addr learned my addr 172.21.15.27:0/77937509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f1029000640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 0x7f10240c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.246+0000 7f102a400640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 0x7f10241ead30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1029a00640 1 -- 172.21.15.27:0/77937509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 msgr2=0x7f10240c0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1029a00640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 0x7f10240c0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1029a00640 1 -- 172.21.15.27:0/77937509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 msgr2=0x7f10241ead30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1029a00640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 0x7f10241ead30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1029a00640 1 -- 172.21.15.27:0/77937509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1024157c10 con 0x7f1024072df0 2024-03-28T19:22:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1029a00640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1024072df0 0x7f102414b9b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f10180090b0 tx=0x7f10180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1021600640 1 -- 172.21.15.27:0/77937509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018041070 con 0x7f1024072df0 2024-03-28T19:22:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1021600640 1 -- 172.21.15.27:0/77937509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1018003c20 con 0x7f1024072df0 2024-03-28T19:22:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f1021600640 1 -- 172.21.15.27:0/77937509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10180315c0 con 0x7f1024072df0 2024-03-28T19:22:52.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.247+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10240aa2b0 con 0x7f1024072df0 2024-03-28T19:22:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.248+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10241587b0 con 0x7f1024072df0 2024-03-28T19:22:52.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.249+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10240c1d90 con 0x7f1024072df0 2024-03-28T19:22:52.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.251+0000 7f1021600640 1 -- 172.21.15.27:0/77937509 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f101804c060 con 0x7f1024072df0 2024-03-28T19:22:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.252+0000 7f1021600640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f100406bbd0 0x7f100406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.252+0000 7f1021600640 1 -- 172.21.15.27:0/77937509 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f10180ad360 con 0x7f1024072df0 2024-03-28T19:22:52.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.252+0000 7f1029000640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f100406bbd0 0x7f100406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.253+0000 7f1029000640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f100406bbd0 0x7f100406e090 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1024155fc0 tx=0x7f1010008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:52.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.257+0000 7f1021600640 1 -- 172.21.15.27:0/77937509 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f10240c1d90 con 0x7f1024072df0 2024-03-28T19:22:52.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.573+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f102404f400 con 0x7f1024072df0 2024-03-28T19:22:52.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.573+0000 7f1021600640 1 -- 172.21.15.27:0/77937509 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f101803ae60 con 0x7f1024072df0 2024-03-28T19:22:52.572 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:52.572 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:45.353898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:52.573 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 200 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.576+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f100406bbd0 msgr2=0x7f100406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f100406bbd0 0x7f100406e090 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1024155fc0 tx=0x7f1010008040 comp rx=0 tx=0).stop 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1024072df0 msgr2=0x7f102414b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1024072df0 0x7f102414b9b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f10180090b0 tx=0x7f10180046c0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 shutdown_connections 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f100406bbd0 0x7f100406e090 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10240c60e0 0x7f10241ead30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10240c3ea0 0x7f10240c0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 --2- 172.21.15.27:0/77937509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1024072df0 0x7f102414b9b0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 >> 172.21.15.27:0/77937509 conn(0x7f102406b9d0 msgr2=0x7f10240a67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 shutdown_connections 2024-03-28T19:22:52.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.577+0000 7f102c400640 1 -- 172.21.15.27:0/77937509 wait complete. 2024-03-28T19:22:52.599 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-03-28T19:22:52.599 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:22:52.599 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:22:52.676 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:22:52.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:52.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.926+0000 7ffbece00640 1 Processor -- start 2024-03-28T19:22:52.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.927+0000 7ffbece00640 1 -- start start 2024-03-28T19:22:52.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.928+0000 7ffbece00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffbe80c6f60 0x7ffbe80c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.928+0000 7ffbece00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe806b540 con 0x7ffbe8156220 2024-03-28T19:22:52.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.928+0000 7ffbece00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe806b820 con 0x7ffbe80c66e0 2024-03-28T19:22:52.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.928+0000 7ffbece00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe8156700 con 0x7ffbe80c6f60 2024-03-28T19:22:52.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.928+0000 7ffbe6800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffbe8156220 0x7ffbe8152be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40186/0 (socket says 172.21.15.27:40186) 2024-03-28T19:22:52.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.929+0000 7ffbe6800640 1 -- 172.21.15.27:0/1332762340 learned_addr learned my addr 172.21.15.27:0/1332762340 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:22:52.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.929+0000 7ffbe5e00640 1 --2- 172.21.15.27:0/1332762340 >> v2:172.21.15.155:3300/0 conn(0x7ffbe80c6f60 0x7ffbe80c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.929+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146906922 0 0) 0x7ffbe806b540 con 0x7ffbe8156220 2024-03-28T19:22:52.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.929+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc003520 con 0x7ffbe8156220 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.929+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441518062 0 0) 0x7ffbcc003520 con 0x7ffbe8156220 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.929+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 >> v2:172.21.15.155:3300/0 conn(0x7ffbe80c6f60 msgr2=0x7ffbe80c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.929+0000 7ffbe4a00640 1 --2- 172.21.15.27:0/1332762340 >> v2:172.21.15.155:3300/0 conn(0x7ffbe80c6f60 0x7ffbe80c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.929+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 >> v1:172.21.15.155:6789/0 conn(0x7ffbe80c66e0 legacy=0x7ffbe806bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe80c58b0 con 0x7ffbe8156220 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7ffbe0001a10 con 0x7ffbe8156220 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 >> v1:172.21.15.27:6789/0 conn(0x7ffbe8156220 legacy=0x7ffbe8152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc004800 0x7ffbcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbcc0052e0 0x7ffbcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe8156700 con 0x7ffbcc0039b0 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00c1b0 con 0x7ffbcc004800 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00c3b0 con 0x7ffbcc0052e0 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe5e00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.930+0000 7ffbe5400640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc004800 0x7ffbcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.931+0000 7ffbe5e00640 1 -- 172.21.15.27:0/1332762340 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbcc0052e0 msgr2=0x7ffbcc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.931+0000 7ffbe5e00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbcc0052e0 0x7ffbcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.931+0000 7ffbe5e00640 1 -- 172.21.15.27:0/1332762340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc004800 msgr2=0x7ffbcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.931+0000 7ffbe5e00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc004800 0x7ffbcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.931+0000 7ffbe5e00640 1 -- 172.21.15.27:0/1332762340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc003520 con 0x7ffbcc0039b0 2024-03-28T19:22:52.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.931+0000 7ffbe6800640 1 -- 172.21.15.27:0/1332762340 reap_dead start 2024-03-28T19:22:52.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbe5e00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003dd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009a90 tx=0x7ffbcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ff2880777c325b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:52.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8039070 con 0x7ffbcc0039b0 2024-03-28T19:22:52.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd802c660 con 0x7ffbcc0039b0 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbe4a00640 1 -- 172.21.15.27:0/1332762340 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8009690 con 0x7ffbcc0039b0 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbece00640 1 -- 172.21.15.27:0/1332762340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbcc0039b0 msgr2=0x7ffbcc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbece00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003dd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009a90 tx=0x7ffbcc005830 comp rx=0 tx=0).stop 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbece00640 1 -- 172.21.15.27:0/1332762340 shutdown_connections 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbece00640 1 --2- 172.21.15.27:0/1332762340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbcc0039b0 0x7ffbcc003dd0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.932+0000 7ffbece00640 1 -- 172.21.15.27:0/1332762340 >> 172.21.15.27:0/1332762340 conn(0x7ffbe80caf30 msgr2=0x7ffbe80cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.933+0000 7ffbece00640 1 -- 172.21.15.27:0/1332762340 shutdown_connections 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.933+0000 7ffbece00640 1 -- 172.21.15.27:0/1332762340 wait complete. 2024-03-28T19:22:52.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.934+0000 7ffbece00640 1 Processor -- start 2024-03-28T19:22:52.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.934+0000 7ffbece00640 1 -- start start 2024-03-28T19:22:52.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.935+0000 7ffbece00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc0039b0 0x7ffbe81eba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.935+0000 7ffbece00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe80be0f0 0x7ffbe8153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.935+0000 7ffbece00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbe81544e0 0x7ffbe8145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.935+0000 7ffbece00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80c74c0 con 0x7ffbe80be0f0 2024-03-28T19:22:52.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.935+0000 7ffbece00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80c7340 con 0x7ffbcc0039b0 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.935+0000 7ffbece00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80c6100 con 0x7ffbe81544e0 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe80be0f0 0x7ffbe8153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe80be0f0 0x7ffbe8153fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53892/0 (socket says 172.21.15.27:53892) 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5400640 1 -- 172.21.15.27:0/1222994305 learned_addr learned my addr 172.21.15.27:0/1222994305 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe6800640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbe81544e0 0x7ffbe8145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5400640 1 -- 172.21.15.27:0/1222994305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbe81544e0 msgr2=0x7ffbe8145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5e00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc0039b0 0x7ffbe81eba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5400640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbe81544e0 0x7ffbe8145ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5400640 1 -- 172.21.15.27:0/1222994305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc0039b0 msgr2=0x7ffbe81eba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5400640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc0039b0 0x7ffbe81eba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.936+0000 7ffbe5400640 1 -- 172.21.15.27:0/1222994305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe8151da0 con 0x7ffbe80be0f0 2024-03-28T19:22:52.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.937+0000 7ffbe5400640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe80be0f0 0x7ffbe8153fa0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e5b0 tx=0x7ffbd400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:52.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.937+0000 7ffbde000640 1 -- 172.21.15.27:0/1222994305 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd40052e0 con 0x7ffbe80be0f0 2024-03-28T19:22:52.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.937+0000 7ffbde000640 1 -- 172.21.15.27:0/1222994305 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd4018070 con 0x7ffbe80be0f0 2024-03-28T19:22:52.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.937+0000 7ffbde000640 1 -- 172.21.15.27:0/1222994305 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4003810 con 0x7ffbe80be0f0 2024-03-28T19:22:52.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.937+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe8152080 con 0x7ffbe80be0f0 2024-03-28T19:22:52.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.937+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe81ec460 con 0x7ffbe80be0f0 2024-03-28T19:22:52.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.939+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe804f400 con 0x7ffbe80be0f0 2024-03-28T19:22:52.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.939+0000 7ffbde000640 1 -- 172.21.15.27:0/1222994305 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffbd40039b0 con 0x7ffbe80be0f0 2024-03-28T19:22:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.940+0000 7ffbde000640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffbc806bbd0 0x7ffbc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.940+0000 7ffbde000640 1 -- 172.21.15.27:0/1222994305 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7ffbd408b750 con 0x7ffbe80be0f0 2024-03-28T19:22:52.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.941+0000 7ffbe5e00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffbc806bbd0 0x7ffbc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:52.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.941+0000 7ffbe5e00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffbc806bbd0 0x7ffbc806e090 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8036000 tx=0x7ffbd8030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:52.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:52.945+0000 7ffbde000640 1 -- 172.21.15.27:0/1222994305 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffbd4059240 con 0x7ffbe80be0f0 2024-03-28T19:22:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.209+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbe80ac210 con 0x7ffbe80be0f0 2024-03-28T19:22:53.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.210+0000 7ffbde000640 1 -- 172.21.15.27:0/1222994305 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5460 (secure 0 0 0) 0x7ffbd4058b40 con 0x7ffbe80be0f0 2024-03-28T19:22:53.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:53.212 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:45.353898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 200 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffbc806bbd0 msgr2=0x7ffbc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffbc806bbd0 0x7ffbc806e090 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8036000 tx=0x7ffbd8030000 comp rx=0 tx=0).stop 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe80be0f0 msgr2=0x7ffbe8153fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe80be0f0 0x7ffbe8153fa0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e5b0 tx=0x7ffbd400ea70 comp rx=0 tx=0).stop 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 shutdown_connections 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ffbc806bbd0 0x7ffbc806e090 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbe81544e0 0x7ffbe8145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbe80be0f0 0x7ffbe8153fa0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 --2- 172.21.15.27:0/1222994305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbcc0039b0 0x7ffbe81eba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 >> 172.21.15.27:0/1222994305 conn(0x7ffbe80caf30 msgr2=0x7ffbe81529b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:53.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 shutdown_connections 2024-03-28T19:22:53.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:53.213+0000 7ffbece00640 1 -- 172.21.15.27:0/1222994305 wait complete. 2024-03-28T19:22:53.238 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:22:54.240 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.472+0000 7f9b6b200640 1 Processor -- start 2024-03-28T19:22:54.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.472+0000 7f9b6b200640 1 -- start start 2024-03-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.474+0000 7f9b6b200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9b6c0c69a0 0x7f9b6c0c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.474+0000 7f9b6b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b6c153a30 0x7f9b6c153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.474+0000 7f9b6b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b6c150ec0 con 0x7f9b6c150ae0 2024-03-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.474+0000 7f9b6b200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b6c06dc70 con 0x7f9b6c0c69a0 2024-03-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.474+0000 7f9b6b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c153550 con 0x7f9b6c153a30 2024-03-28T19:22:54.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.474+0000 7f9b69e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b6c150ae0 0x7f9b6c0a5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40196/0 (socket says 172.21.15.27:40196) 2024-03-28T19:22:54.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.474+0000 7f9b69e00640 1 -- 172.21.15.27:0/2864896572 learned_addr learned my addr 172.21.15.27:0/2864896572 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b61400640 1 --2- 172.21.15.27:0/2864896572 >> v2:172.21.15.155:3300/0 conn(0x7f9b6c0c69a0 0x7f9b6c0c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b6a800640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b6c153a30 0x7f9b6c153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063125990 0 0) 0x7f9b5c001110 con 0x7f9b6c150ae0 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b50003520 con 0x7f9b6c150ae0 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4152888836 0 0) 0x7f9b50003520 con 0x7f9b6c150ae0 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b6c153a30 msgr2=0x7f9b6c153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b6c153a30 0x7f9b6c153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 >> v2:172.21.15.155:3300/0 conn(0x7f9b6c0c69a0 msgr2=0x7f9b6c0c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 --2- 172.21.15.27:0/2864896572 >> v2:172.21.15.155:3300/0 conn(0x7f9b6c0c69a0 0x7f9b6c0c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c06df10 con 0x7f9b6c150ae0 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9b5c001840 con 0x7f9b6c150ae0 2024-03-28T19:22:54.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 >> v1:172.21.15.27:6789/0 conn(0x7f9b6c150ae0 legacy=0x7f9b6c0a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b500039b0 0x7f9b50003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b50004800 0x7f9b50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b6c153a30 0x7f9b500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c06dc70 con 0x7f9b50004800 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c153550 con 0x7f9b500039b0 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.475+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5000ba10 con 0x7f9b6c153a30 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.476+0000 7f9b61400640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b50004800 0x7f9b50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.476+0000 7f9b69e00640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b500039b0 0x7f9b50003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.476+0000 7f9b61400640 1 -- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b6c153a30 msgr2=0x7f9b500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:54.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.476+0000 7f9b6a800640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b6c153a30 0x7f9b500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.476+0000 7f9b61400640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b6c153a30 0x7f9b500051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.476+0000 7f9b61400640 1 -- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b500039b0 msgr2=0x7f9b50003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.476+0000 7f9b61400640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b500039b0 0x7f9b50003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.476+0000 7f9b61400640 1 -- 172.21.15.27:0/2864896572 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b50003520 con 0x7f9b50004800 2024-03-28T19:22:54.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.477+0000 7f9b61400640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b50004800 0x7f9b50004c80 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009af0 tx=0x7f9b50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1eb252441fd2dbe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:54.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54039070 con 0x7f9b50004800 2024-03-28T19:22:54.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5402c710 con 0x7f9b50004800 2024-03-28T19:22:54.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b69400640 1 -- 172.21.15.27:0/2864896572 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54009690 con 0x7f9b50004800 2024-03-28T19:22:54.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b6b200640 1 -- 172.21.15.27:0/2864896572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b50004800 msgr2=0x7f9b50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:54.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b6b200640 1 --2- 172.21.15.27:0/2864896572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b50004800 0x7f9b50004c80 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009af0 tx=0x7f9b50005830 comp rx=0 tx=0).stop 2024-03-28T19:22:54.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b6a800640 1 -- 172.21.15.27:0/2864896572 reap_dead start 2024-03-28T19:22:54.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b6b200640 1 -- 172.21.15.27:0/2864896572 shutdown_connections 2024-03-28T19:22:54.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b6b200640 1 -- 172.21.15.27:0/2864896572 >> 172.21.15.27:0/2864896572 conn(0x7f9b6c06b0c0 msgr2=0x7f9b6c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:54.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b6b200640 1 -- 172.21.15.27:0/2864896572 shutdown_connections 2024-03-28T19:22:54.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.478+0000 7f9b6b200640 1 -- 172.21.15.27:0/2864896572 wait complete. 2024-03-28T19:22:54.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.480+0000 7f9b6b200640 1 Processor -- start 2024-03-28T19:22:54.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.480+0000 7f9b6b200640 1 -- start start 2024-03-28T19:22:54.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b6b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b50004800 0x7f9b6c1e9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b6b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b6c070f60 0x7f9b6c0a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b6b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b6c0a8dc0 0x7f9b6c147a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b6b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c06e7b0 con 0x7f9b6c070f60 2024-03-28T19:22:54.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b6b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c06e4d0 con 0x7f9b6c0a8dc0 2024-03-28T19:22:54.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b6b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c0a6370 con 0x7f9b50004800 2024-03-28T19:22:54.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b61400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b6c070f60 0x7f9b6c0a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b61400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b6c070f60 0x7f9b6c0a7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53896/0 (socket says 172.21.15.27:53896) 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.481+0000 7f9b61400640 1 -- 172.21.15.27:0/2832713299 learned_addr learned my addr 172.21.15.27:0/2832713299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b69e00640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b50004800 0x7f9b6c1e9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b61400640 1 -- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b50004800 msgr2=0x7f9b6c1e9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b61400640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b50004800 0x7f9b6c1e9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b6a800640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b6c0a8dc0 0x7f9b6c147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b61400640 1 -- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b6c0a8dc0 msgr2=0x7f9b6c147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b61400640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b6c0a8dc0 0x7f9b6c147a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b61400640 1 -- 172.21.15.27:0/2832713299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c0a6980 con 0x7f9b6c070f60 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b6a800640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b6c0a8dc0 0x7f9b6c147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b61400640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b6c070f60 0x7f9b6c0a7a20 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f9b54037040 tx=0x7f9b54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.482+0000 7f9b62a00640 1 -- 172.21.15.27:0/2832713299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54039050 con 0x7f9b6c070f60 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.483+0000 7f9b62a00640 1 -- 172.21.15.27:0/2832713299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b54045070 con 0x7f9b6c070f60 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.483+0000 7f9b62a00640 1 -- 172.21.15.27:0/2832713299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54003a50 con 0x7f9b6c070f60 2024-03-28T19:22:54.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.483+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6c0a6c60 con 0x7f9b6c070f60 2024-03-28T19:22:54.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.483+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6c0a7220 con 0x7f9b6c070f60 2024-03-28T19:22:54.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.484+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6c067dd0 con 0x7f9b6c070f60 2024-03-28T19:22:54.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.485+0000 7f9b62a00640 1 -- 172.21.15.27:0/2832713299 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9b54003bf0 con 0x7f9b6c070f60 2024-03-28T19:22:54.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.486+0000 7f9b62a00640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b4806bbd0 0x7f9b4806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:54.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.486+0000 7f9b62a00640 1 -- 172.21.15.27:0/2832713299 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f9b540ad620 con 0x7f9b6c070f60 2024-03-28T19:22:54.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.486+0000 7f9b69e00640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b4806bbd0 0x7f9b4806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:54.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.487+0000 7f9b69e00640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b4806bbd0 0x7f9b4806e090 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c0012f0 tx=0x7f9b5c001d80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:54.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.490+0000 7f9b62a00640 1 -- 172.21.15.27:0/2832713299 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9b5404e020 con 0x7f9b6c070f60 2024-03-28T19:22:54.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.726+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b6c152fc0 con 0x7f9b6c070f60 2024-03-28T19:22:54.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.727+0000 7f9b62a00640 1 -- 172.21.15.27:0/2832713299 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+5464 (secure 0 0 0) 0x7f9b5407b130 con 0x7f9b6c070f60 2024-03-28T19:22:54.726 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:54.727 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:53.789984+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:54.727 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 201 2024-03-28T19:22:54.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b4806bbd0 msgr2=0x7f9b4806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:54.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b4806bbd0 0x7f9b4806e090 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c0012f0 tx=0x7f9b5c001d80 comp rx=0 tx=0).stop 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b6c070f60 msgr2=0x7f9b6c0a7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b6c070f60 0x7f9b6c0a7a20 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f9b54037040 tx=0x7f9b54004730 comp rx=0 tx=0).stop 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 shutdown_connections 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b4806bbd0 0x7f9b4806e090 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b6c0a8dc0 0x7f9b6c147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b6c070f60 0x7f9b6c0a7a20 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 --2- 172.21.15.27:0/2832713299 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b50004800 0x7f9b6c1e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 >> 172.21.15.27:0/2832713299 conn(0x7f9b6c06b0c0 msgr2=0x7f9b6c0c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 shutdown_connections 2024-03-28T19:22:54.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:54.730+0000 7f9b6b200640 1 -- 172.21.15.27:0/2832713299 wait complete. 2024-03-28T19:22:54.748 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:22:55.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.477+0000 7fa5ba000640 1 Processor -- start 2024-03-28T19:22:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.477+0000 7fa5ba000640 1 -- start start 2024-03-28T19:22:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.478+0000 7fa5ba000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b40c69a0 0x7fa5b40c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.478+0000 7fa5ba000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4153a30 0x7fa5b4153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.478+0000 7fa5ba000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b4150ec0 con 0x7fa5b4150ae0 2024-03-28T19:22:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.478+0000 7fa5ba000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b406dc70 con 0x7fa5b40c69a0 2024-03-28T19:22:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.478+0000 7fa5ba000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4153550 con 0x7fa5b4153a30 2024-03-28T19:22:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b40c69a0 0x7fa5b40c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5b40c69a0 0x7fa5b40c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53904/0 (socket says 172.21.15.27:53904) 2024-03-28T19:22:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b3e00640 1 -- 172.21.15.27:0/78105191 learned_addr learned my addr 172.21.15.27:0/78105191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b9600640 1 --2- 172.21.15.27:0/78105191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4153a30 0x7fa5b4153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b3e00640 1 -- 172.21.15.27:0/78105191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4153a30 msgr2=0x7fa5b4153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b3e00640 1 --2- 172.21.15.27:0/78105191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4153a30 0x7fa5b4153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b3e00640 1 -- 172.21.15.27:0/78105191 >> v1:172.21.15.155:6789/0 conn(0x7fa5b4150ae0 legacy=0x7fa5b40a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b3e00640 1 -- 172.21.15.27:0/78105191 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b406df10 con 0x7fa5b40c69a0 2024-03-28T19:22:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.479+0000 7fa5b9600640 1 --2- 172.21.15.27:0/78105191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4153a30 0x7fa5b4153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:22:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.480+0000 7fa5b3e00640 1 --2- 172.21.15.27:0/78105191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b40c6e30 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009870 tx=0x7fa5a002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0667838cd276225 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.481+0000 7fa5b3400640 1 -- 172.21.15.27:0/78105191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00317c0 con 0x7fa5b40c69a0 2024-03-28T19:22:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.481+0000 7fa5b3400640 1 -- 172.21.15.27:0/78105191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0037070 con 0x7fa5b40c69a0 2024-03-28T19:22:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.481+0000 7fa5b3400640 1 -- 172.21.15.27:0/78105191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0031b80 con 0x7fa5b40c69a0 2024-03-28T19:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.482+0000 7fa5ba000640 1 -- 172.21.15.27:0/78105191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 msgr2=0x7fa5b40c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.482+0000 7fa5ba000640 1 --2- 172.21.15.27:0/78105191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b40c6e30 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009870 tx=0x7fa5a002b860 comp rx=0 tx=0).stop 2024-03-28T19:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.482+0000 7fa5ba000640 1 -- 172.21.15.27:0/78105191 shutdown_connections 2024-03-28T19:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.482+0000 7fa5ba000640 1 --2- 172.21.15.27:0/78105191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4153a30 0x7fa5b4153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.482+0000 7fa5ba000640 1 --2- 172.21.15.27:0/78105191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b40c6e30 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.482+0000 7fa5ba000640 1 -- 172.21.15.27:0/78105191 >> 172.21.15.27:0/78105191 conn(0x7fa5b406b0c0 msgr2=0x7fa5b406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.482+0000 7fa5ba000640 1 -- 172.21.15.27:0/78105191 shutdown_connections 2024-03-28T19:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.482+0000 7fa5ba000640 1 -- 172.21.15.27:0/78105191 wait complete. 2024-03-28T19:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.484+0000 7fa5ba000640 1 Processor -- start 2024-03-28T19:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.484+0000 7fa5ba000640 1 -- start start 2024-03-28T19:22:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.485+0000 7fa5ba000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b414e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.485+0000 7fa5ba000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4150ae0 0x7fa5b414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.485+0000 7fa5ba000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5b4153a30 0x7fa5b41eea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.485+0000 7fa5ba000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b406e7b0 con 0x7fa5b40c69a0 2024-03-28T19:22:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.485+0000 7fa5ba000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b406e4d0 con 0x7fa5b4153a30 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.485+0000 7fa5ba000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40a6370 con 0x7fa5b4150ae0 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b414e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b414e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53916/0 (socket says 172.21.15.27:53916) 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b8c00640 1 -- 172.21.15.27:0/2715510206 learned_addr learned my addr 172.21.15.27:0/2715510206 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b9600640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5b4153a30 0x7fa5b41eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b3e00640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4150ae0 0x7fa5b414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b8c00640 1 -- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4150ae0 msgr2=0x7fa5b414e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b8c00640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4150ae0 0x7fa5b414e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b8c00640 1 -- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5b4153a30 msgr2=0x7fa5b41eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b8c00640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5b4153a30 0x7fa5b41eea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b8c00640 1 -- 172.21.15.27:0/2715510206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b414ead0 con 0x7fa5b40c69a0 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b9600640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5b4153a30 0x7fa5b41eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:22:55.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.486+0000 7fa5b3e00640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4150ae0 0x7fa5b414e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:22:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.487+0000 7fa5b8c00640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b414e050 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400c460 tx=0x7fa5a400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.487+0000 7fa5b0c00640 1 -- 172.21.15.27:0/2715510206 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4016020 con 0x7fa5b40c69a0 2024-03-28T19:22:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.487+0000 7fa5b0c00640 1 -- 172.21.15.27:0/2715510206 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4012870 con 0x7fa5b40c69a0 2024-03-28T19:22:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.487+0000 7fa5b0c00640 1 -- 172.21.15.27:0/2715510206 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4012c60 con 0x7fa5b40c69a0 2024-03-28T19:22:55.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.487+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b40a9390 con 0x7fa5b40c69a0 2024-03-28T19:22:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.487+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b41e2370 con 0x7fa5b40c69a0 2024-03-28T19:22:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.488+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b4067dd0 con 0x7fa5b40c69a0 2024-03-28T19:22:55.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.490+0000 7fa5b0c00640 1 -- 172.21.15.27:0/2715510206 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa5a4019020 con 0x7fa5b40c69a0 2024-03-28T19:22:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.491+0000 7fa5b0c00640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa59406bc00 0x7fa59406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.491+0000 7fa5b0c00640 1 -- 172.21.15.27:0/2715510206 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7fa5a408b450 con 0x7fa5b40c69a0 2024-03-28T19:22:55.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.491+0000 7fa5b3e00640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa59406bc00 0x7fa59406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.492+0000 7fa5b3e00640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa59406bc00 0x7fa59406e0c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0036000 tx=0x7fa5a0008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.496+0000 7fa5b0c00640 1 -- 172.21.15.27:0/2715510206 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa5a4020070 con 0x7fa5b40c69a0 2024-03-28T19:22:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.745+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5b406da50 con 0x7fa5b40c69a0 2024-03-28T19:22:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.746+0000 7fa5b0c00640 1 -- 172.21.15.27:0/2715510206 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+5461 (secure 0 0 0) 0x7fa5a4058fc0 con 0x7fa5b40c69a0 2024-03-28T19:22:55.745 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:55.745 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:53.875589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:55.745 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 202 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa59406bc00 msgr2=0x7fa59406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa59406bc00 0x7fa59406e0c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0036000 tx=0x7fa5a0008030 comp rx=0 tx=0).stop 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 msgr2=0x7fa5b414e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b414e050 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400c460 tx=0x7fa5a400c920 comp rx=0 tx=0).stop 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 shutdown_connections 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa59406bc00 0x7fa59406e0c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5b4153a30 0x7fa5b41eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa5b4150ae0 0x7fa5b414e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 --2- 172.21.15.27:0/2715510206 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5b40c69a0 0x7fa5b414e050 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.748+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 >> 172.21.15.27:0/2715510206 conn(0x7fa5b406b0c0 msgr2=0x7fa5b406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:55.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.749+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 shutdown_connections 2024-03-28T19:22:55.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:55.749+0000 7fa5ba000640 1 -- 172.21.15.27:0/2715510206 wait complete. 2024-03-28T19:22:55.768 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:22:56.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:57.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.570+0000 7fd1d3200640 1 Processor -- start 2024-03-28T19:22:57.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.571+0000 7fd1d3200640 1 -- start start 2024-03-28T19:22:57.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.572+0000 7fd1d3200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd1cc153bd0 0x7fd1cc0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:57.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.572+0000 7fd1d3200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc154020 con 0x7fd1cc150c70 2024-03-28T19:22:57.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.572+0000 7fd1d3200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0c3d90 con 0x7fd1cc0a9cb0 2024-03-28T19:22:57.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.572+0000 7fd1d3200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1cc0ab4e0 con 0x7fd1cc153bd0 2024-03-28T19:22:57.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.572+0000 7fd1cbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd1cc150c70 0x7fd1cc0c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39798/0 (socket says 172.21.15.27:39798) 2024-03-28T19:22:57.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.572+0000 7fd1cbe00640 1 -- 172.21.15.27:0/1769660246 learned_addr learned my addr 172.21.15.27:0/1769660246 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.573+0000 7fd1d1200640 1 --2- 172.21.15.27:0/1769660246 >> v2:172.21.15.155:3300/0 conn(0x7fd1cc153bd0 0x7fd1cc0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.573+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109341474 0 0) 0x7fd1cc154020 con 0x7fd1cc150c70 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.573+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b0003520 con 0x7fd1cc150c70 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.573+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 629136392 0 0) 0x7fd1b0003520 con 0x7fd1cc150c70 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.573+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 >> v2:172.21.15.155:3300/0 conn(0x7fd1cc153bd0 msgr2=0x7fd1cc0c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.573+0000 7fd1caa00640 1 --2- 172.21.15.27:0/1769660246 >> v2:172.21.15.155:3300/0 conn(0x7fd1cc153bd0 0x7fd1cc0c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.573+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 >> v1:172.21.15.155:6789/0 conn(0x7fd1cc0a9cb0 legacy=0x7fd1cc1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0ab710 con 0x7fd1cc150c70 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fd1bc001670 con 0x7fd1cc150c70 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 >> v1:172.21.15.27:6789/0 conn(0x7fd1cc150c70 legacy=0x7fd1cc0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b00039b0 0x7fd1b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:57.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1b0004800 0x7fd1b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:57.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b00051c0 0x7fd1b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:57.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b000c130 con 0x7fd1b0004800 2024-03-28T19:22:57.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b000c2e0 con 0x7fd1b00039b0 2024-03-28T19:22:57.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b000c4e0 con 0x7fd1b00051c0 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.574+0000 7fd1cb400640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1b0004800 0x7fd1b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1d1200640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b00051c0 0x7fd1b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1cbe00640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b00039b0 0x7fd1b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1cb400640 1 -- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b00051c0 msgr2=0x7fd1b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1cb400640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b00051c0 0x7fd1b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1cb400640 1 -- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b00039b0 msgr2=0x7fd1b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1cb400640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b00039b0 0x7fd1b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1cb400640 1 -- 172.21.15.27:0/1769660246 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0ab710 con 0x7fd1b0004800 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1d1200640 1 -- 172.21.15.27:0/1769660246 reap_dead start 2024-03-28T19:22:57.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.575+0000 7fd1cbe00640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b00039b0 0x7fd1b0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.576+0000 7fd1cb400640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1b0004800 0x7fd1b0004c80 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8001750 tx=0x7fd1b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f720dd7140d4e6f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.577+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8001370 con 0x7fd1b0004800 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.577+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b80052d0 con 0x7fd1b0004800 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.577+0000 7fd1caa00640 1 -- 172.21.15.27:0/1769660246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8003830 con 0x7fd1b0004800 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.577+0000 7fd1d3200640 1 -- 172.21.15.27:0/1769660246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1b0004800 msgr2=0x7fd1b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.577+0000 7fd1d3200640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1b0004800 0x7fd1b0004c80 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8001750 tx=0x7fd1b0005830 comp rx=0 tx=0).stop 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.577+0000 7fd1d3200640 1 -- 172.21.15.27:0/1769660246 shutdown_connections 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.577+0000 7fd1d3200640 1 --2- 172.21.15.27:0/1769660246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1b0004800 0x7fd1b0004c80 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.577+0000 7fd1d3200640 1 -- 172.21.15.27:0/1769660246 >> 172.21.15.27:0/1769660246 conn(0x7fd1cc06b9d0 msgr2=0x7fd1cc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.578+0000 7fd1d3200640 1 -- 172.21.15.27:0/1769660246 shutdown_connections 2024-03-28T19:22:57.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.578+0000 7fd1d3200640 1 -- 172.21.15.27:0/1769660246 wait complete. 2024-03-28T19:22:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.581+0000 7fd1d3200640 1 Processor -- start 2024-03-28T19:22:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.581+0000 7fd1d3200640 1 -- start start 2024-03-28T19:22:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1d3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b0004800 0x7fd1cc1ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1d3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc154830 0x7fd1cc14ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1d3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1cc1536a0 0x7fd1cc0a79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1d3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0abcd0 con 0x7fd1cc154830 2024-03-28T19:22:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1d3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0ac1f0 con 0x7fd1cc1536a0 2024-03-28T19:22:57.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1d3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc0c1810 con 0x7fd1b0004800 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc154830 0x7fd1cc14ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc154830 0x7fd1cc14ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41764/0 (socket says 172.21.15.27:41764) 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1cb400640 1 -- 172.21.15.27:0/92155440 learned_addr learned my addr 172.21.15.27:0/92155440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.583+0000 7fd1cbe00640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b0004800 0x7fd1cc1ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.584+0000 7fd1d1200640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1cc1536a0 0x7fd1cc0a79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.584+0000 7fd1cb400640 1 -- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b0004800 msgr2=0x7fd1cc1ea8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.584+0000 7fd1cb400640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b0004800 0x7fd1cc1ea8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.584+0000 7fd1cb400640 1 -- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1cc1536a0 msgr2=0x7fd1cc0a79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:57.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.584+0000 7fd1cb400640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1cc1536a0 0x7fd1cc0a79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.584+0000 7fd1cb400640 1 -- 172.21.15.27:0/92155440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc152910 con 0x7fd1cc154830 2024-03-28T19:22:57.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.584+0000 7fd1d1200640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1cc1536a0 0x7fd1cc0a79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:22:57.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.584+0000 7fd1cb400640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc154830 0x7fd1cc14ae40 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8005040 tx=0x7fd1b8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.585+0000 7fd1c3e00640 1 -- 172.21.15.27:0/92155440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8041070 con 0x7fd1cc154830 2024-03-28T19:22:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.585+0000 7fd1c3e00640 1 -- 172.21.15.27:0/92155440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b8005480 con 0x7fd1cc154830 2024-03-28T19:22:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.585+0000 7fd1c3e00640 1 -- 172.21.15.27:0/92155440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8007a30 con 0x7fd1cc154830 2024-03-28T19:22:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.585+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc152bf0 con 0x7fd1cc154830 2024-03-28T19:22:57.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.585+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc14b380 con 0x7fd1cc154830 2024-03-28T19:22:57.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.587+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1cc04f400 con 0x7fd1cc154830 2024-03-28T19:22:57.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.588+0000 7fd1c3e00640 1 -- 172.21.15.27:0/92155440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd1b8044070 con 0x7fd1cc154830 2024-03-28T19:22:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.589+0000 7fd1c3e00640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd1ac06bbd0 0x7fd1ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.590+0000 7fd1c3e00640 1 -- 172.21.15.27:0/92155440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7fd1b80ad500 con 0x7fd1cc154830 2024-03-28T19:22:57.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.590+0000 7fd1cbe00640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd1ac06bbd0 0x7fd1ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.591+0000 7fd1cbe00640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd1ac06bbd0 0x7fd1ac06e090 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc050670 tx=0x7fd1bc007040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:57.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.595+0000 7fd1c3e00640 1 -- 172.21.15.27:0/92155440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd1b8006e20 con 0x7fd1cc154830 2024-03-28T19:22:57.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:22:57.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.929+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1cc0c5f00 con 0x7fd1cc154830 2024-03-28T19:22:57.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.929+0000 7fd1c3e00640 1 -- 172.21.15.27:0/92155440 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+5468 (secure 0 0 0) 0x7fd1b807a9c0 con 0x7fd1cc154830 2024-03-28T19:22:57.928 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:57.929 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:57.929 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 203 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.932+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd1ac06bbd0 msgr2=0x7fd1ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.932+0000 7fd1d3200640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd1ac06bbd0 0x7fd1ac06e090 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc050670 tx=0x7fd1bc007040 comp rx=0 tx=0).stop 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.932+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc154830 msgr2=0x7fd1cc14ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.932+0000 7fd1d3200640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc154830 0x7fd1cc14ae40 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8005040 tx=0x7fd1b8001120 comp rx=0 tx=0).stop 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.933+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 shutdown_connections 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.933+0000 7fd1d3200640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd1ac06bbd0 0x7fd1ac06e090 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.933+0000 7fd1d3200640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1cc1536a0 0x7fd1cc0a79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.933+0000 7fd1d3200640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc154830 0x7fd1cc14ae40 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.933+0000 7fd1d3200640 1 --2- 172.21.15.27:0/92155440 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1b0004800 0x7fd1cc1ea8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.933+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 >> 172.21.15.27:0/92155440 conn(0x7fd1cc06b9d0 msgr2=0x7fd1cc0c19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.933+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 shutdown_connections 2024-03-28T19:22:57.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:57.933+0000 7fd1d3200640 1 -- 172.21.15.27:0/92155440 wait complete. 2024-03-28T19:22:57.960 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7273 2024-03-28T19:22:57.960 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.b 2024-03-28T19:22:58.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.004+0000 7fc263200640 1 Processor -- start 2024-03-28T19:22:58.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.005+0000 7fc263200640 1 -- start start 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.006+0000 7fc263200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 0x7fc2640c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.006+0000 7fc263200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2640c8ed0 0x7fc2641555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.006+0000 7fc263200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc264156230 con 0x7fc2640be5d0 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.006+0000 7fc263200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc264158bd0 con 0x7fc2640c8ed0 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.006+0000 7fc263200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc264158eb0 con 0x7fc2640c58c0 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.006+0000 7fc261400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2640c8ed0 0x7fc2641555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.007+0000 7fc261400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc2640c8ed0 0x7fc2641555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41770/0 (socket says 172.21.15.27:41770) 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.007+0000 7fc261400640 1 -- 172.21.15.27:0/1550805124 learned_addr learned my addr 172.21.15.27:0/1550805124 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.007+0000 7fc261e00640 1 --2- 172.21.15.27:0/1550805124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 0x7fc2640c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.007+0000 7fc261400640 1 -- 172.21.15.27:0/1550805124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 msgr2=0x7fc2640c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.007+0000 7fc261400640 1 --2- 172.21.15.27:0/1550805124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 0x7fc2640c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.007+0000 7fc261400640 1 -- 172.21.15.27:0/1550805124 >> v1:172.21.15.155:6789/0 conn(0x7fc2640be5d0 legacy=0x7fc264155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:22:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.007+0000 7fc261400640 1 -- 172.21.15.27:0/1550805124 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc264159190 con 0x7fc2640c8ed0 2024-03-28T19:22:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.008+0000 7fc261400640 1 --2- 172.21.15.27:0/1550805124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc2641555e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fc250009870 tx=0x7fc25002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e64c1d2b8791d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.009+0000 7fc260a00640 1 -- 172.21.15.27:0/1550805124 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2500317c0 con 0x7fc2640c8ed0 2024-03-28T19:22:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.009+0000 7fc260a00640 1 -- 172.21.15.27:0/1550805124 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc250037070 con 0x7fc2640c8ed0 2024-03-28T19:22:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.009+0000 7fc260a00640 1 -- 172.21.15.27:0/1550805124 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250031b80 con 0x7fc2640c8ed0 2024-03-28T19:22:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.009+0000 7fc263200640 1 -- 172.21.15.27:0/1550805124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 msgr2=0x7fc2641555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.009+0000 7fc263200640 1 --2- 172.21.15.27:0/1550805124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc2641555e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fc250009870 tx=0x7fc25002b860 comp rx=0 tx=0).stop 2024-03-28T19:22:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.010+0000 7fc263200640 1 -- 172.21.15.27:0/1550805124 shutdown_connections 2024-03-28T19:22:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.010+0000 7fc263200640 1 --2- 172.21.15.27:0/1550805124 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc2641555e0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.010+0000 7fc263200640 1 --2- 172.21.15.27:0/1550805124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 0x7fc2640c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.010+0000 7fc263200640 1 -- 172.21.15.27:0/1550805124 >> 172.21.15.27:0/1550805124 conn(0x7fc26406b6f0 msgr2=0x7fc2640acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.010+0000 7fc263200640 1 -- 172.21.15.27:0/1550805124 shutdown_connections 2024-03-28T19:22:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.010+0000 7fc263200640 1 -- 172.21.15.27:0/1550805124 wait complete. 2024-03-28T19:22:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.012+0000 7fc263200640 1 Processor -- start 2024-03-28T19:22:58.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.012+0000 7fc263200640 1 -- start start 2024-03-28T19:22:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.013+0000 7fc263200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2640be5d0 0x7fc2641ec100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.013+0000 7fc263200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 0x7fc2640c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.013+0000 7fc263200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc264151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.013+0000 7fc263200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2640b0820 con 0x7fc2640c8ed0 2024-03-28T19:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.013+0000 7fc263200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc264159750 con 0x7fc2640be5d0 2024-03-28T19:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.013+0000 7fc263200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2640b09a0 con 0x7fc2640c58c0 2024-03-28T19:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.013+0000 7fc262800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc264151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.013+0000 7fc262800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc264151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41772/0 (socket says 172.21.15.27:41772) 2024-03-28T19:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.014+0000 7fc262800640 1 -- 172.21.15.27:0/2037416237 learned_addr learned my addr 172.21.15.27:0/2037416237 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.014+0000 7fc261400640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 0x7fc2640c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.014+0000 7fc261e00640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2640be5d0 0x7fc2641ec100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.014+0000 7fc262800640 1 -- 172.21.15.27:0/2037416237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 msgr2=0x7fc2640c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.014+0000 7fc262800640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 0x7fc2640c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.014+0000 7fc262800640 1 -- 172.21.15.27:0/2037416237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2640be5d0 msgr2=0x7fc2641ec100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.014+0000 7fc262800640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2640be5d0 0x7fc2641ec100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.014+0000 7fc262800640 1 -- 172.21.15.27:0/2037416237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640c20b0 con 0x7fc2640c8ed0 2024-03-28T19:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.015+0000 7fc262800640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc264151ce0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc25c00b9e0 tx=0x7fc25c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.015+0000 7fc25a000640 1 -- 172.21.15.27:0/2037416237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c003b70 con 0x7fc2640c8ed0 2024-03-28T19:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.015+0000 7fc25a000640 1 -- 172.21.15.27:0/2037416237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c003d10 con 0x7fc2640c8ed0 2024-03-28T19:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.015+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2640c2840 con 0x7fc2640c8ed0 2024-03-28T19:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.015+0000 7fc25a000640 1 -- 172.21.15.27:0/2037416237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c00cb00 con 0x7fc2640c8ed0 2024-03-28T19:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.015+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2640aaf60 con 0x7fc2640c8ed0 2024-03-28T19:22:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.017+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c005290 con 0x7fc2640c8ed0 2024-03-28T19:22:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.019+0000 7fc25a000640 1 -- 172.21.15.27:0/2037416237 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc25c00cca0 con 0x7fc2640c8ed0 2024-03-28T19:22:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.020+0000 7fc25a000640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc24406bbd0 0x7fc24406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:22:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.020+0000 7fc25a000640 1 -- 172.21.15.27:0/2037416237 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7fc25c08b780 con 0x7fc2640c8ed0 2024-03-28T19:22:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.020+0000 7fc261e00640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc24406bbd0 0x7fc24406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:22:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.021+0000 7fc261e00640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc24406bbd0 0x7fc24406e090 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc254006fd0 tx=0x7fc254008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:22:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.027+0000 7fc25a000640 1 -- 172.21.15.27:0/2037416237 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc25c0592f0 con 0x7fc2640c8ed0 2024-03-28T19:22:58.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.338+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc23c005580 con 0x7fc2640c8ed0 2024-03-28T19:22:58.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.339+0000 7fc25a000640 1 -- 172.21.15.27:0/2037416237 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+5468 (secure 0 0 0) 0x7fc25c058bf0 con 0x7fc2640c8ed0 2024-03-28T19:22:58.339 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:22:58.339 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:22:58.339 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 203 2024-03-28T19:22:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc24406bbd0 msgr2=0x7fc24406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc24406bbd0 0x7fc24406e090 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc254006fd0 tx=0x7fc254008040 comp rx=0 tx=0).stop 2024-03-28T19:22:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 msgr2=0x7fc264151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:22:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc264151ce0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc25c00b9e0 tx=0x7fc25c00bea0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 shutdown_connections 2024-03-28T19:22:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc24406bbd0 0x7fc24406e090 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2640c8ed0 0x7fc264151ce0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2640c58c0 0x7fc2640c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 --2- 172.21.15.27:0/2037416237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2640be5d0 0x7fc2641ec100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:22:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 >> 172.21.15.27:0/2037416237 conn(0x7fc26406b6f0 msgr2=0x7fc2640ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:22:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 shutdown_connections 2024-03-28T19:22:58.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:22:58.342+0000 7fc263200640 1 -- 172.21.15.27:0/2037416237 wait complete. 2024-03-28T19:22:58.362 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:22:58.877 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:22:58.878+0000 7feb8fc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:22:59.961 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-03-28T19:22:59.961 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:22:59.962 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:22:59.969 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:22:59.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:00.160 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:23:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.200+0000 7f4d56400640 1 Processor -- start 2024-03-28T19:23:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.201+0000 7f4d56400640 1 -- start start 2024-03-28T19:23:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.203+0000 7f4d56400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d50156220 0x7f4d500c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.203+0000 7f4d56400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5006b540 con 0x7f4d500c6f60 2024-03-28T19:23:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.203+0000 7f4d56400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5006b820 con 0x7f4d500c66e0 2024-03-28T19:23:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.203+0000 7f4d56400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d50156700 con 0x7f4d50156220 2024-03-28T19:23:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.203+0000 7f4d4f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4d500c6f60 0x7f4d5006bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39808/0 (socket says 172.21.15.27:39808) 2024-03-28T19:23:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.203+0000 7f4d4f400640 1 -- 172.21.15.27:0/2981174081 learned_addr learned my addr 172.21.15.27:0/2981174081 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.203+0000 7f4d4fe00640 1 --2- 172.21.15.27:0/2981174081 >> v2:172.21.15.27:3300/0 conn(0x7f4d50156220 0x7f4d500c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.204+0000 7f4d4fe00640 1 -- 172.21.15.27:0/2981174081 >> v1:172.21.15.155:6789/0 conn(0x7f4d500c66e0 legacy=0x7f4d50152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.204+0000 7f4d4fe00640 1 -- 172.21.15.27:0/2981174081 >> v1:172.21.15.27:6789/0 conn(0x7f4d500c6f60 legacy=0x7f4d5006bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.204+0000 7f4d4fe00640 1 -- 172.21.15.27:0/2981174081 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500c58b0 con 0x7f4d50156220 2024-03-28T19:23:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.205+0000 7f4d4fe00640 1 --2- 172.21.15.27:0/2981174081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d50156220 0x7f4d500c8e30 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009d90 tx=0x7f4d4802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c9435a21a4d9e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.205+0000 7f4d4ea00640 1 -- 172.21.15.27:0/2981174081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48039070 con 0x7f4d50156220 2024-03-28T19:23:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.205+0000 7f4d4ea00640 1 -- 172.21.15.27:0/2981174081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4802cbd0 con 0x7f4d50156220 2024-03-28T19:23:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.205+0000 7f4d4ea00640 1 -- 172.21.15.27:0/2981174081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48034440 con 0x7f4d50156220 2024-03-28T19:23:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.206+0000 7f4d56400640 1 -- 172.21.15.27:0/2981174081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d50156220 msgr2=0x7f4d500c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.206+0000 7f4d56400640 1 --2- 172.21.15.27:0/2981174081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d50156220 0x7f4d500c8e30 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009d90 tx=0x7f4d4802be20 comp rx=0 tx=0).stop 2024-03-28T19:23:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.206+0000 7f4d56400640 1 -- 172.21.15.27:0/2981174081 shutdown_connections 2024-03-28T19:23:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.206+0000 7f4d56400640 1 --2- 172.21.15.27:0/2981174081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d50156220 0x7f4d500c8e30 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.206+0000 7f4d56400640 1 -- 172.21.15.27:0/2981174081 >> 172.21.15.27:0/2981174081 conn(0x7f4d500caf30 msgr2=0x7f4d500cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.206+0000 7f4d56400640 1 -- 172.21.15.27:0/2981174081 shutdown_connections 2024-03-28T19:23:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.206+0000 7f4d56400640 1 -- 172.21.15.27:0/2981174081 wait complete. 2024-03-28T19:23:00.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.208+0000 7f4d56400640 1 Processor -- start 2024-03-28T19:23:00.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.208+0000 7f4d56400640 1 -- start start 2024-03-28T19:23:00.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.209+0000 7f4d56400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d500c66e0 0x7f4d500aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.209+0000 7f4d56400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d500c6f60 0x7f4d501ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.209+0000 7f4d56400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50156220 0x7f4d501eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.209+0000 7f4d56400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5006c410 con 0x7f4d500c6f60 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.209+0000 7f4d56400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5006c290 con 0x7f4d50156220 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.209+0000 7f4d56400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d501532f0 con 0x7f4d500c66e0 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d46a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d500c6f60 0x7f4d501ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d4fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50156220 0x7f4d501eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d4f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d500c66e0 0x7f4d500aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d4fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50156220 0x7f4d501eeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:55198/0 (socket says 172.21.15.27:55198) 2024-03-28T19:23:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d4fe00640 1 -- 172.21.15.27:0/1989049215 learned_addr learned my addr 172.21.15.27:0/1989049215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d46a00640 1 -- 172.21.15.27:0/1989049215 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d500c66e0 msgr2=0x7f4d500aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d46a00640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d500c66e0 0x7f4d500aec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d46a00640 1 -- 172.21.15.27:0/1989049215 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50156220 msgr2=0x7f4d501eeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d46a00640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50156220 0x7f4d501eeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.210+0000 7f4d46a00640 1 -- 172.21.15.27:0/1989049215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500ae180 con 0x7f4d500c6f60 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.211+0000 7f4d46a00640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d500c6f60 0x7f4d501ee9a0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800c390 tx=0x7f4d3800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.211+0000 7f4d47e00640 1 -- 172.21.15.27:0/1989049215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38017070 con 0x7f4d500c6f60 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.211+0000 7f4d47e00640 1 -- 172.21.15.27:0/1989049215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d380052d0 con 0x7f4d500c6f60 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.211+0000 7f4d47e00640 1 -- 172.21.15.27:0/1989049215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38003830 con 0x7f4d500c6f60 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.211+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d501ef420 con 0x7f4d500c6f60 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.211+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d500b0c70 con 0x7f4d500c6f60 2024-03-28T19:23:00.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.213+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5004f400 con 0x7f4d500c6f60 2024-03-28T19:23:00.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.214+0000 7f4d47e00640 1 -- 172.21.15.27:0/1989049215 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4d3801a070 con 0x7f4d500c6f60 2024-03-28T19:23:00.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.215+0000 7f4d47e00640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4d2c06bc00 0x7f4d2c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.215+0000 7f4d47e00640 1 -- 172.21.15.27:0/1989049215 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f4d3808ac20 con 0x7f4d500c6f60 2024-03-28T19:23:00.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.215+0000 7f4d4f400640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4d2c06bc00 0x7f4d2c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.216+0000 7f4d4f400640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4d2c06bc00 0x7f4d2c06e0c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4d40000f60 tx=0x7f4d40009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:00.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.220+0000 7f4d47e00640 1 -- 172.21.15.27:0/1989049215 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4d501ef420 con 0x7f4d500c6f60 2024-03-28T19:23:00.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:00.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.554+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d500cb520 con 0x7f4d500c6f60 2024-03-28T19:23:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.555+0000 7f4d47e00640 1 -- 172.21.15.27:0/1989049215 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f4d380580e0 con 0x7f4d500c6f60 2024-03-28T19:23:00.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:00.554 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:23:00.554 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 203 2024-03-28T19:23:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.557+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4d2c06bc00 msgr2=0x7f4d2c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.557+0000 7f4d56400640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4d2c06bc00 0x7f4d2c06e0c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4d40000f60 tx=0x7f4d40009040 comp rx=0 tx=0).stop 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.557+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d500c6f60 msgr2=0x7f4d501ee9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.557+0000 7f4d56400640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d500c6f60 0x7f4d501ee9a0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800c390 tx=0x7f4d3800c850 comp rx=0 tx=0).stop 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.558+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 shutdown_connections 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.558+0000 7f4d56400640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4d2c06bc00 0x7f4d2c06e0c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.558+0000 7f4d56400640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50156220 0x7f4d501eeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.558+0000 7f4d56400640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d500c6f60 0x7f4d501ee9a0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.558+0000 7f4d56400640 1 --2- 172.21.15.27:0/1989049215 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d500c66e0 0x7f4d500aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.558+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 >> 172.21.15.27:0/1989049215 conn(0x7f4d500caf30 msgr2=0x7f4d500ab130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.558+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 shutdown_connections 2024-03-28T19:23:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.558+0000 7f4d56400640 1 -- 172.21.15.27:0/1989049215 wait complete. 2024-03-28T19:23:00.580 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-03-28T19:23:00.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.604+0000 7f3037400640 1 Processor -- start 2024-03-28T19:23:00.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.604+0000 7f3037400640 1 -- start start 2024-03-28T19:23:00.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3037400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f30300c60e0 0x7f303006e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3037400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30300c3ea0 0x7f303006ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3037400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3030072df0 0x7f3030158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3037400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30300c23b0 con 0x7f3030072df0 2024-03-28T19:23:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3037400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30300c1600 con 0x7f30300c60e0 2024-03-28T19:23:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3037400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c18e0 con 0x7f30300c3ea0 2024-03-28T19:23:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3035400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3030072df0 0x7f3030158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3035400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3030072df0 0x7f3030158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41798/0 (socket says 172.21.15.27:41798) 2024-03-28T19:23:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.605+0000 7f3035400640 1 -- 172.21.15.27:0/1099846428 learned_addr learned my addr 172.21.15.27:0/1099846428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.606+0000 7f3034a00640 1 --2- 172.21.15.27:0/1099846428 >> v2:172.21.15.155:3300/0 conn(0x7f30300c60e0 0x7f303006e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.606+0000 7f3035400640 1 -- 172.21.15.27:0/1099846428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30300c3ea0 msgr2=0x7f303006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:00.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.606+0000 7f3035400640 1 --2- 172.21.15.27:0/1099846428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30300c3ea0 0x7f303006ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.606+0000 7f3035400640 1 -- 172.21.15.27:0/1099846428 >> v2:172.21.15.155:3300/0 conn(0x7f30300c60e0 msgr2=0x7f303006e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:00.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.606+0000 7f3035400640 1 --2- 172.21.15.27:0/1099846428 >> v2:172.21.15.155:3300/0 conn(0x7f30300c60e0 0x7f303006e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.606+0000 7f3035400640 1 -- 172.21.15.27:0/1099846428 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300c1bc0 con 0x7f3030072df0 2024-03-28T19:23:00.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.606+0000 7f3034a00640 1 --2- 172.21.15.27:0/1099846428 >> v2:172.21.15.155:3300/0 conn(0x7f30300c60e0 0x7f303006e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:00.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.608+0000 7f3035400640 1 --2- 172.21.15.27:0/1099846428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3030072df0 0x7f3030158f00 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f30300c10d0 tx=0x7f302802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719330b6267af35a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:00.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.608+0000 7f302f400640 1 -- 172.21.15.27:0/1099846428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302802c950 con 0x7f3030072df0 2024-03-28T19:23:00.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.608+0000 7f302f400640 1 -- 172.21.15.27:0/1099846428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302802caf0 con 0x7f3030072df0 2024-03-28T19:23:00.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.608+0000 7f302f400640 1 -- 172.21.15.27:0/1099846428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30280343f0 con 0x7f3030072df0 2024-03-28T19:23:00.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 -- 172.21.15.27:0/1099846428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3030072df0 msgr2=0x7f3030158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 --2- 172.21.15.27:0/1099846428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3030072df0 0x7f3030158f00 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f30300c10d0 tx=0x7f302802bdb0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 -- 172.21.15.27:0/1099846428 shutdown_connections 2024-03-28T19:23:00.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 --2- 172.21.15.27:0/1099846428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3030072df0 0x7f3030158f00 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 --2- 172.21.15.27:0/1099846428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f30300c3ea0 0x7f303006ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 --2- 172.21.15.27:0/1099846428 >> v2:172.21.15.155:3300/0 conn(0x7f30300c60e0 0x7f303006e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 -- 172.21.15.27:0/1099846428 >> 172.21.15.27:0/1099846428 conn(0x7f303006b9d0 msgr2=0x7f30300c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:00.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 -- 172.21.15.27:0/1099846428 shutdown_connections 2024-03-28T19:23:00.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.609+0000 7f3037400640 1 -- 172.21.15.27:0/1099846428 wait complete. 2024-03-28T19:23:00.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.612+0000 7f3037400640 1 Processor -- start 2024-03-28T19:23:00.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.612+0000 7f3037400640 1 -- start start 2024-03-28T19:23:00.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.613+0000 7f3037400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3030072df0 0x7f30300a8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3037400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30300c3ea0 0x7f3030153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3037400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30300c60e0 0x7f30301ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3037400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c6640 con 0x7f30300c60e0 2024-03-28T19:23:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3037400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c64c0 con 0x7f30300c3ea0 2024-03-28T19:23:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3037400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30300c4f00 con 0x7f3030072df0 2024-03-28T19:23:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3035400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30300c60e0 0x7f30301ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3035400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30300c60e0 0x7f30301ee750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41812/0 (socket says 172.21.15.27:41812) 2024-03-28T19:23:00.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3035400640 1 -- 172.21.15.27:0/81482399 learned_addr learned my addr 172.21.15.27:0/81482399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:00.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3034a00640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3030072df0 0x7f30300a8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f302fe00640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30300c3ea0 0x7f3030153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3035400640 1 -- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3030072df0 msgr2=0x7f30300a8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3035400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3030072df0 0x7f30300a8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3035400640 1 -- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30300c3ea0 msgr2=0x7f3030153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3035400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30300c3ea0 0x7f3030153230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3035400640 1 -- 172.21.15.27:0/81482399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30301eb6c0 con 0x7f30300c60e0 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.614+0000 7f3034a00640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3030072df0 0x7f30300a8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.615+0000 7f3035400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30300c60e0 0x7f30301ee750 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f3028009f40 tx=0x7f3028004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.615+0000 7f302cc00640 1 -- 172.21.15.27:0/81482399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3028043020 con 0x7f30300c60e0 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.615+0000 7f302cc00640 1 -- 172.21.15.27:0/81482399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302803e450 con 0x7f30300c60e0 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.615+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30301eb9a0 con 0x7f30300c60e0 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.615+0000 7f302cc00640 1 -- 172.21.15.27:0/81482399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302803e820 con 0x7f30300c60e0 2024-03-28T19:23:00.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.615+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3030071fb0 con 0x7f30300c60e0 2024-03-28T19:23:00.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.617+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3030071ad0 con 0x7f30300c60e0 2024-03-28T19:23:00.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.618+0000 7f302cc00640 1 -- 172.21.15.27:0/81482399 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3028046020 con 0x7f30300c60e0 2024-03-28T19:23:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.619+0000 7f302cc00640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f301006bbd0 0x7f301006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.620+0000 7f302cc00640 1 -- 172.21.15.27:0/81482399 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f30280ae3a0 con 0x7f30300c60e0 2024-03-28T19:23:00.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.620+0000 7f3034a00640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f301006bbd0 0x7f301006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.621+0000 7f3034a00640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f301006bbd0 0x7f301006e090 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3024006fd0 tx=0x7f3024008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.626+0000 7f302cc00640 1 -- 172.21.15.27:0/81482399 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f302804d030 con 0x7f30300c60e0 2024-03-28T19:23:00.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.904+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f303004f400 con 0x7f30300c60e0 2024-03-28T19:23:00.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.904+0000 7f302cc00640 1 -- 172.21.15.27:0/81482399 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f302807bf10 con 0x7f30300c60e0 2024-03-28T19:23:00.903 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:00.904 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:23:00.904 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 204 2024-03-28T19:23:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f301006bbd0 msgr2=0x7f301006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f301006bbd0 0x7f301006e090 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3024006fd0 tx=0x7f3024008040 comp rx=0 tx=0).stop 2024-03-28T19:23:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30300c60e0 msgr2=0x7f30301ee750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30300c60e0 0x7f30301ee750 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f3028009f40 tx=0x7f3028004a40 comp rx=0 tx=0).stop 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 shutdown_connections 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f301006bbd0 0x7f301006e090 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30300c60e0 0x7f30301ee750 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30300c3ea0 0x7f3030153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 --2- 172.21.15.27:0/81482399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3030072df0 0x7f30300a8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.908+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 >> 172.21.15.27:0/81482399 conn(0x7f303006b9d0 msgr2=0x7f30300c3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.909+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 shutdown_connections 2024-03-28T19:23:00.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:00.909+0000 7f3037400640 1 -- 172.21.15.27:0/81482399 wait complete. 2024-03-28T19:23:00.928 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:23:02.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.790+0000 7f03aa600640 1 Processor -- start 2024-03-28T19:23:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.790+0000 7f03aa600640 1 -- start start 2024-03-28T19:23:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03aa600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a40c59d0 0x7f03a40c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03aa600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03a4155600 0x7f03a414f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03aa600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a4155fe0 con 0x7f03a40727e0 2024-03-28T19:23:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03aa600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a41565a0 con 0x7f03a4155600 2024-03-28T19:23:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03aa600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a4159090 con 0x7f03a40c59d0 2024-03-28T19:23:02.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03a9c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03a4155600 0x7f03a414f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03a9c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03a4155600 0x7f03a414f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41822/0 (socket says 172.21.15.27:41822) 2024-03-28T19:23:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03a9c00640 1 -- 172.21.15.27:0/401593833 learned_addr learned my addr 172.21.15.27:0/401593833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.792+0000 7f03a9200640 1 --2- 172.21.15.27:0/401593833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a40c59d0 0x7f03a40c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.793+0000 7f03a9c00640 1 -- 172.21.15.27:0/401593833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a40c59d0 msgr2=0x7f03a40c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.793+0000 7f03a9c00640 1 --2- 172.21.15.27:0/401593833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a40c59d0 0x7f03a40c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.793+0000 7f03a9c00640 1 -- 172.21.15.27:0/401593833 >> v1:172.21.15.27:6789/0 conn(0x7f03a40727e0 legacy=0x7f03a4159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.793+0000 7f03a9c00640 1 -- 172.21.15.27:0/401593833 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a41562c0 con 0x7f03a4155600 2024-03-28T19:23:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.793+0000 7f03a9200640 1 --2- 172.21.15.27:0/401593833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a40c59d0 0x7f03a40c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.794+0000 7f03a9c00640 1 --2- 172.21.15.27:0/401593833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a4155600 0x7f03a414f260 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f039c009e50 tx=0x7f039c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dd47fb99cf3679 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.795+0000 7f03a3400640 1 -- 172.21.15.27:0/401593833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c02c9f0 con 0x7f03a4155600 2024-03-28T19:23:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.795+0000 7f03a3400640 1 -- 172.21.15.27:0/401593833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c02cb90 con 0x7f03a4155600 2024-03-28T19:23:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.795+0000 7f03a3400640 1 -- 172.21.15.27:0/401593833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c0343f0 con 0x7f03a4155600 2024-03-28T19:23:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.796+0000 7f03aa600640 1 -- 172.21.15.27:0/401593833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a4155600 msgr2=0x7f03a414f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.796+0000 7f03aa600640 1 --2- 172.21.15.27:0/401593833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a4155600 0x7f03a414f260 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f039c009e50 tx=0x7f039c02be50 comp rx=0 tx=0).stop 2024-03-28T19:23:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.796+0000 7f03aa600640 1 -- 172.21.15.27:0/401593833 shutdown_connections 2024-03-28T19:23:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.796+0000 7f03aa600640 1 --2- 172.21.15.27:0/401593833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a4155600 0x7f03a414f260 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.796+0000 7f03aa600640 1 --2- 172.21.15.27:0/401593833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a40c59d0 0x7f03a40c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.796+0000 7f03aa600640 1 -- 172.21.15.27:0/401593833 >> 172.21.15.27:0/401593833 conn(0x7f03a40ac910 msgr2=0x7f03a40accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.796+0000 7f03aa600640 1 -- 172.21.15.27:0/401593833 shutdown_connections 2024-03-28T19:23:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.796+0000 7f03aa600640 1 -- 172.21.15.27:0/401593833 wait complete. 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.798+0000 7f03aa600640 1 Processor -- start 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.798+0000 7f03aa600640 1 -- start start 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03aa600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40727e0 0x7f03a41ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03aa600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a40c59d0 0x7f03a4143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03aa600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4155600 0x7f03a40c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03aa600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40c5f30 con 0x7f03a40c59d0 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03aa600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40c5db0 con 0x7f03a40727e0 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03aa600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a4155a70 con 0x7f03a4155600 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a40c59d0 0x7f03a4143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a40c59d0 0x7f03a4143510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41838/0 (socket says 172.21.15.27:41838) 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.799+0000 7f03a3e00640 1 -- 172.21.15.27:0/3243629569 learned_addr learned my addr 172.21.15.27:0/3243629569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a9200640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40727e0 0x7f03a41ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a9c00640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4155600 0x7f03a40c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a3e00640 1 -- 172.21.15.27:0/3243629569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4155600 msgr2=0x7f03a40c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a3e00640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4155600 0x7f03a40c01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a3e00640 1 -- 172.21.15.27:0/3243629569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40727e0 msgr2=0x7f03a41ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a3e00640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40727e0 0x7f03a41ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a3e00640 1 -- 172.21.15.27:0/3243629569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a414dbd0 con 0x7f03a40c59d0 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a3e00640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a40c59d0 0x7f03a4143510 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f039000c440 tx=0x7f0390001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.800+0000 7f03a0c00640 1 -- 172.21.15.27:0/3243629569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900052d0 con 0x7f03a40c59d0 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.801+0000 7f03a0c00640 1 -- 172.21.15.27:0/3243629569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0390005470 con 0x7f03a40c59d0 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.801+0000 7f03a0c00640 1 -- 172.21.15.27:0/3243629569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0390007260 con 0x7f03a40c59d0 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.801+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a40c1110 con 0x7f03a40c59d0 2024-03-28T19:23:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.801+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a414b290 con 0x7f03a40c59d0 2024-03-28T19:23:02.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.802+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a4067dd0 con 0x7f03a40c59d0 2024-03-28T19:23:02.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.804+0000 7f03a0c00640 1 -- 172.21.15.27:0/3243629569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0390007400 con 0x7f03a40c59d0 2024-03-28T19:23:02.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.805+0000 7f03a0c00640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f038406bbd0 0x7f038406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:02.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.805+0000 7f03a0c00640 1 -- 172.21.15.27:0/3243629569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f039008af00 con 0x7f03a40c59d0 2024-03-28T19:23:02.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.806+0000 7f03a9200640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f038406bbd0 0x7f038406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:02.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.806+0000 7f03a9200640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f038406bbd0 0x7f038406e090 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0398006fd0 tx=0x7f0398008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:02.815+0000 7f03a0c00640 1 -- 172.21.15.27:0/3243629569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0390058a70 con 0x7f03a40c59d0 2024-03-28T19:23:02.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.098+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03a4069640 con 0x7f03a40c59d0 2024-03-28T19:23:03.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.099+0000 7f03a0c00640 1 -- 172.21.15.27:0/3243629569 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f0390058370 con 0x7f03a40c59d0 2024-03-28T19:23:03.098 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:03.098 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:23:03.098 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 204 2024-03-28T19:23:03.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.101+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f038406bbd0 msgr2=0x7f038406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.101+0000 7f03aa600640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f038406bbd0 0x7f038406e090 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0398006fd0 tx=0x7f0398008040 comp rx=0 tx=0).stop 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.101+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a40c59d0 msgr2=0x7f03a4143510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.101+0000 7f03aa600640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a40c59d0 0x7f03a4143510 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f039000c440 tx=0x7f0390001e70 comp rx=0 tx=0).stop 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.102+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 shutdown_connections 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.102+0000 7f03aa600640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f038406bbd0 0x7f038406e090 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.102+0000 7f03aa600640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4155600 0x7f03a40c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.102+0000 7f03aa600640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03a40c59d0 0x7f03a4143510 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.102+0000 7f03aa600640 1 --2- 172.21.15.27:0/3243629569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40727e0 0x7f03a41ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.102+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 >> 172.21.15.27:0/3243629569 conn(0x7f03a40ac910 msgr2=0x7f03a414e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.102+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 shutdown_connections 2024-03-28T19:23:03.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.102+0000 7f03aa600640 1 -- 172.21.15.27:0/3243629569 wait complete. 2024-03-28T19:23:03.121 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-03-28T19:23:03.121 INFO:tasks.mds_thrash.fs.[a]:waiting for 18 secs before thrashing 2024-03-28T19:23:03.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.182+0000 7f50e8a00640 1 Processor -- start 2024-03-28T19:23:03.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.182+0000 7f50e8a00640 1 -- start start 2024-03-28T19:23:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.183+0000 7f50e8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50e40c58c0 0x7f50e40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.183+0000 7f50e8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40be5d0 0x7f50e4155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.183+0000 7f50e8a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e4156230 con 0x7f50e40c8ed0 2024-03-28T19:23:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.183+0000 7f50e8a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e4158bd0 con 0x7f50e40c58c0 2024-03-28T19:23:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.183+0000 7f50e8a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4158eb0 con 0x7f50e40be5d0 2024-03-28T19:23:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.183+0000 7f50e2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50e40c58c0 0x7f50e40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.183+0000 7f50e2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50e40c58c0 0x7f50e40c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41854/0 (socket says 172.21.15.27:41854) 2024-03-28T19:23:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.184+0000 7f50e2800640 1 -- 172.21.15.27:0/4008156688 learned_addr learned my addr 172.21.15.27:0/4008156688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.184+0000 7f50e3200640 1 --2- 172.21.15.27:0/4008156688 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40be5d0 0x7f50e4155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.184+0000 7f50e2800640 1 -- 172.21.15.27:0/4008156688 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40be5d0 msgr2=0x7f50e4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:03.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.184+0000 7f50e2800640 1 --2- 172.21.15.27:0/4008156688 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40be5d0 0x7f50e4155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.184+0000 7f50e2800640 1 -- 172.21.15.27:0/4008156688 >> v1:172.21.15.155:6789/0 conn(0x7f50e40c8ed0 legacy=0x7f50e41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:03.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.184+0000 7f50e2800640 1 -- 172.21.15.27:0/4008156688 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e4159190 con 0x7f50e40c58c0 2024-03-28T19:23:03.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.184+0000 7f50e3200640 1 --2- 172.21.15.27:0/4008156688 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40be5d0 0x7f50e4155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:03.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.186+0000 7f50e2800640 1 --2- 172.21.15.27:0/4008156688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c58c0 0x7f50e40c5ca0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009870 tx=0x7f50d402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b736c97200a45a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:03.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.187+0000 7f50e1400640 1 -- 172.21.15.27:0/4008156688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d40317c0 con 0x7f50e40c58c0 2024-03-28T19:23:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.187+0000 7f50e1400640 1 -- 172.21.15.27:0/4008156688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d4037070 con 0x7f50e40c58c0 2024-03-28T19:23:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.187+0000 7f50e1400640 1 -- 172.21.15.27:0/4008156688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4031b80 con 0x7f50e40c58c0 2024-03-28T19:23:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.188+0000 7f50e8a00640 1 -- 172.21.15.27:0/4008156688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c58c0 msgr2=0x7f50e40c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.188+0000 7f50e8a00640 1 --2- 172.21.15.27:0/4008156688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c58c0 0x7f50e40c5ca0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009870 tx=0x7f50d402b860 comp rx=0 tx=0).stop 2024-03-28T19:23:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.188+0000 7f50e8a00640 1 -- 172.21.15.27:0/4008156688 shutdown_connections 2024-03-28T19:23:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.188+0000 7f50e8a00640 1 --2- 172.21.15.27:0/4008156688 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40be5d0 0x7f50e4155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.188+0000 7f50e8a00640 1 --2- 172.21.15.27:0/4008156688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c58c0 0x7f50e40c5ca0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.188+0000 7f50e8a00640 1 -- 172.21.15.27:0/4008156688 >> 172.21.15.27:0/4008156688 conn(0x7f50e406b6f0 msgr2=0x7f50e40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.188+0000 7f50e8a00640 1 -- 172.21.15.27:0/4008156688 shutdown_connections 2024-03-28T19:23:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.188+0000 7f50e8a00640 1 -- 172.21.15.27:0/4008156688 wait complete. 2024-03-28T19:23:03.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.190+0000 7f50e8a00640 1 Processor -- start 2024-03-28T19:23:03.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.190+0000 7f50e8a00640 1 -- start start 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.191+0000 7f50e8a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e40be5d0 0x7f50e41ec1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.191+0000 7f50e8a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40c58c0 0x7f50e40c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.191+0000 7f50e8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c8ed0 0x7f50e4151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.191+0000 7f50e8a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40b0820 con 0x7f50e40c8ed0 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.191+0000 7f50e8a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4159750 con 0x7f50e40be5d0 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.191+0000 7f50e8a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40b09a0 con 0x7f50e40c58c0 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c8ed0 0x7f50e4151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c8ed0 0x7f50e4151d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41856/0 (socket says 172.21.15.27:41856) 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e3200640 1 -- 172.21.15.27:0/1191278152 learned_addr learned my addr 172.21.15.27:0/1191278152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e2800640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e40be5d0 0x7f50e41ec1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e1e00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40c58c0 0x7f50e40c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e3200640 1 -- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40c58c0 msgr2=0x7f50e40c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e3200640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40c58c0 0x7f50e40c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e3200640 1 -- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e40be5d0 msgr2=0x7f50e41ec1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e3200640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e40be5d0 0x7f50e41ec1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e3200640 1 -- 172.21.15.27:0/1191278152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e40ad1e0 con 0x7f50e40c8ed0 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.192+0000 7f50e1e00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40c58c0 0x7f50e40c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.193+0000 7f50e3200640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c8ed0 0x7f50e4151d50 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00cd80 tx=0x7f50dc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.193+0000 7f50daa00640 1 -- 172.21.15.27:0/1191278152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc00f680 con 0x7f50e40c8ed0 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.193+0000 7f50daa00640 1 -- 172.21.15.27:0/1191278152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc00f840 con 0x7f50e40c8ed0 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.193+0000 7f50daa00640 1 -- 172.21.15.27:0/1191278152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc00fc30 con 0x7f50e40c8ed0 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.193+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e40c24b0 con 0x7f50e40c8ed0 2024-03-28T19:23:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.193+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e40acc20 con 0x7f50e40c8ed0 2024-03-28T19:23:03.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.194+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e4069640 con 0x7f50e40c8ed0 2024-03-28T19:23:03.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.195+0000 7f50daa00640 1 -- 172.21.15.27:0/1191278152 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f50dc00e3e0 con 0x7f50e40c8ed0 2024-03-28T19:23:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.196+0000 7f50daa00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f50c406bc00 0x7f50c406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.196+0000 7f50daa00640 1 -- 172.21.15.27:0/1191278152 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f50dc08c7a0 con 0x7f50e40c8ed0 2024-03-28T19:23:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.197+0000 7f50e2800640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f50c406bc00 0x7f50c406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:03.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.197+0000 7f50e2800640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f50c406bc00 0x7f50c406e0c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f50d4036000 tx=0x7f50d403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:03.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.201+0000 7f50daa00640 1 -- 172.21.15.27:0/1191278152 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f50dc014030 con 0x7f50e40c8ed0 2024-03-28T19:23:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.510+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50e4156580 con 0x7f50e40c8ed0 2024-03-28T19:23:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.511+0000 7f50daa00640 1 -- 172.21.15.27:0/1191278152 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f50dc05a2b0 con 0x7f50e40c8ed0 2024-03-28T19:23:03.510 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:03.510 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:23:03.510 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 204 2024-03-28T19:23:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f50c406bc00 msgr2=0x7f50c406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f50c406bc00 0x7f50c406e0c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f50d4036000 tx=0x7f50d403d040 comp rx=0 tx=0).stop 2024-03-28T19:23:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c8ed0 msgr2=0x7f50e4151d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c8ed0 0x7f50e4151d50 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00cd80 tx=0x7f50dc00a650 comp rx=0 tx=0).stop 2024-03-28T19:23:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 shutdown_connections 2024-03-28T19:23:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f50c406bc00 0x7f50c406e0c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50e40c8ed0 0x7f50e4151d50 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50e40c58c0 0x7f50e40c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 --2- 172.21.15.27:0/1191278152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e40be5d0 0x7f50e41ec1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.514+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 >> 172.21.15.27:0/1191278152 conn(0x7f50e406b6f0 msgr2=0x7f50e40c95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.515+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 shutdown_connections 2024-03-28T19:23:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:03.515+0000 7f50e8a00640 1 -- 172.21.15.27:0/1191278152 wait complete. 2024-03-28T19:23:03.539 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:23:05.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.761+0000 7f2f9a200640 1 Processor -- start 2024-03-28T19:23:05.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.762+0000 7f2f9a200640 1 -- start start 2024-03-28T19:23:05.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.763+0000 7f2f9a200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2f9414f6d0 0x7f2f94159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:05.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.763+0000 7f2f9a200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f94072870 con 0x7f2f94071cf0 2024-03-28T19:23:05.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.763+0000 7f2f9a200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f94156640 con 0x7f2f94072c60 2024-03-28T19:23:05.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.763+0000 7f2f9a200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f94158db0 con 0x7f2f9414f6d0 2024-03-28T19:23:05.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.763+0000 7f2f8be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2f94071cf0 0x7f2f940c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39828/0 (socket says 172.21.15.27:39828) 2024-03-28T19:23:05.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.763+0000 7f2f8be00640 1 -- 172.21.15.27:0/1281652066 learned_addr learned my addr 172.21.15.27:0/1281652066 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f93400640 1 --2- 172.21.15.27:0/1281652066 >> v2:172.21.15.155:3300/0 conn(0x7f2f9414f6d0 0x7f2f94159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250679835 0 0) 0x7f2f7c001110 con 0x7f2f94071cf0 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f78003520 con 0x7f2f94071cf0 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139670878 0 0) 0x7f2f78003520 con 0x7f2f94071cf0 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 >> v2:172.21.15.155:3300/0 conn(0x7f2f9414f6d0 msgr2=0x7f2f94159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f92a00640 1 --2- 172.21.15.27:0/1281652066 >> v2:172.21.15.155:3300/0 conn(0x7f2f9414f6d0 0x7f2f94159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 >> v1:172.21.15.155:6789/0 conn(0x7f2f94072c60 legacy=0x7f2f940c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f94156360 con 0x7f2f94071cf0 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.764+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f2f7c001840 con 0x7f2f94071cf0 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 >> v1:172.21.15.27:6789/0 conn(0x7f2f94071cf0 legacy=0x7f2f940c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f92a00640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f78003a00 0x7f2f78003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f92a00640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f78004800 0x7f2f78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:05.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f92a00640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94158db0 con 0x7f2f78005210 2024-03-28T19:23:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7800c160 con 0x7f2f78003a00 2024-03-28T19:23:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7800c360 con 0x7f2f78004800 2024-03-28T19:23:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f93e00640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f93400640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f78003a00 0x7f2f78003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.765+0000 7f2f8be00640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f78004800 0x7f2f78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.766+0000 7f2f93e00640 1 -- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f78004800 msgr2=0x7f2f78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.766+0000 7f2f93e00640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f78004800 0x7f2f78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.766+0000 7f2f93e00640 1 -- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f78003a00 msgr2=0x7f2f78003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.766+0000 7f2f93e00640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f78003a00 0x7f2f78003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.766+0000 7f2f93e00640 1 -- 172.21.15.27:0/1281652066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f78003520 con 0x7f2f78005210 2024-03-28T19:23:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.766+0000 7f2f93400640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f78003a00 0x7f2f78003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.767+0000 7f2f93e00640 1 -- 172.21.15.27:0/1281652066 reap_dead start 2024-03-28T19:23:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.768+0000 7f2f93e00640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f7800b9c0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c001e20 tx=0x7f2f78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0d3979003e45a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.768+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c001a70 con 0x7f2f78005210 2024-03-28T19:23:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.768+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c003b70 con 0x7f2f78005210 2024-03-28T19:23:05.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.768+0000 7f2f92a00640 1 -- 172.21.15.27:0/1281652066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c0050d0 con 0x7f2f78005210 2024-03-28T19:23:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.769+0000 7f2f9a200640 1 -- 172.21.15.27:0/1281652066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 msgr2=0x7f2f7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.769+0000 7f2f9a200640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f7800b9c0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c001e20 tx=0x7f2f78005830 comp rx=0 tx=0).stop 2024-03-28T19:23:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.769+0000 7f2f9a200640 1 -- 172.21.15.27:0/1281652066 shutdown_connections 2024-03-28T19:23:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.769+0000 7f2f9a200640 1 --2- 172.21.15.27:0/1281652066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f7800b9c0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.769+0000 7f2f9a200640 1 -- 172.21.15.27:0/1281652066 >> 172.21.15.27:0/1281652066 conn(0x7f2f9406c5f0 msgr2=0x7f2f9406ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.769+0000 7f2f9a200640 1 -- 172.21.15.27:0/1281652066 shutdown_connections 2024-03-28T19:23:05.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.769+0000 7f2f9a200640 1 -- 172.21.15.27:0/1281652066 wait complete. 2024-03-28T19:23:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.771+0000 7f2f9a200640 1 Processor -- start 2024-03-28T19:23:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.772+0000 7f2f9a200640 1 -- start start 2024-03-28T19:23:05.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.772+0000 7f2f9a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f94154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:05.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.772+0000 7f2f9a200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f941decf0 0x7f2f9414e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:05.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f9a200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f941e0f00 0x7f2f94156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:05.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f9a200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9414fab0 con 0x7f2f78005210 2024-03-28T19:23:05.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f9a200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94159330 con 0x7f2f941decf0 2024-03-28T19:23:05.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f9a200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94155fa0 con 0x7f2f941e0f00 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f93400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f94154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f93400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f94154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41888/0 (socket says 172.21.15.27:41888) 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f93400640 1 -- 172.21.15.27:0/2062016910 learned_addr learned my addr 172.21.15.27:0/2062016910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f8be00640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f941decf0 0x7f2f9414e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f93400640 1 -- 172.21.15.27:0/2062016910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f941e0f00 msgr2=0x7f2f94156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f93400640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f941e0f00 0x7f2f94156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f93400640 1 -- 172.21.15.27:0/2062016910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f941decf0 msgr2=0x7f2f9414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f93400640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f941decf0 0x7f2f9414e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f93400640 1 -- 172.21.15.27:0/2062016910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f941e17a0 con 0x7f2f78005210 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.773+0000 7f2f8be00640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f941decf0 0x7f2f9414e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:05.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.774+0000 7f2f93400640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f94154150 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f2f840115a0 tx=0x7f2f84011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:05.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.774+0000 7f2f8b400640 1 -- 172.21.15.27:0/2062016910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84015870 con 0x7f2f78005210 2024-03-28T19:23:05.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.774+0000 7f2f8b400640 1 -- 172.21.15.27:0/2062016910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f84015a10 con 0x7f2f78005210 2024-03-28T19:23:05.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.774+0000 7f2f8b400640 1 -- 172.21.15.27:0/2062016910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84015dc0 con 0x7f2f78005210 2024-03-28T19:23:05.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.774+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f941e1a40 con 0x7f2f78005210 2024-03-28T19:23:05.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.774+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f941e0700 con 0x7f2f78005210 2024-03-28T19:23:05.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.776+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f940aabd0 con 0x7f2f78005210 2024-03-28T19:23:05.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.776+0000 7f2f8b400640 1 -- 172.21.15.27:0/2062016910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2f8401d400 con 0x7f2f78005210 2024-03-28T19:23:05.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.777+0000 7f2f8b400640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2f7006bc00 0x7f2f7006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:05.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.777+0000 7f2f8b400640 1 -- 172.21.15.27:0/2062016910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7f2f8408c200 con 0x7f2f78005210 2024-03-28T19:23:05.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.777+0000 7f2f8be00640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2f7006bc00 0x7f2f7006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:05.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.778+0000 7f2f8be00640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2f7006bc00 0x7f2f7006e0c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c001050 tx=0x7f2f7c001e20 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:05.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:05.782+0000 7f2f8b400640 1 -- 172.21.15.27:0/2062016910 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2f940aabd0 con 0x7f2f78005210 2024-03-28T19:23:06.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.071+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f9404f400 con 0x7f2f78005210 2024-03-28T19:23:06.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.072+0000 7f2f8b400640 1 -- 172.21.15.27:0/2062016910 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f2f84019030 con 0x7f2f78005210 2024-03-28T19:23:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:06.073 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:23:06.073 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 204 2024-03-28T19:23:06.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.075+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2f7006bc00 msgr2=0x7f2f7006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:06.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.075+0000 7f2f9a200640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2f7006bc00 0x7f2f7006e0c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c001050 tx=0x7f2f7c001e20 comp rx=0 tx=0).stop 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.075+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 msgr2=0x7f2f94154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.075+0000 7f2f9a200640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f94154150 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f2f840115a0 tx=0x7f2f84011a60 comp rx=0 tx=0).stop 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.075+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 shutdown_connections 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.075+0000 7f2f9a200640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f2f7006bc00 0x7f2f7006e0c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.076+0000 7f2f9a200640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f941e0f00 0x7f2f94156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.076+0000 7f2f9a200640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f941decf0 0x7f2f9414e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.076+0000 7f2f9a200640 1 --2- 172.21.15.27:0/2062016910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f78005210 0x7f2f94154150 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.076+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 >> 172.21.15.27:0/2062016910 conn(0x7f2f9406c5f0 msgr2=0x7f2f9406e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.076+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 shutdown_connections 2024-03-28T19:23:06.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:06.076+0000 7f2f9a200640 1 -- 172.21.15.27:0/2062016910 wait complete. 2024-03-28T19:23:06.097 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:23:08.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.331+0000 7fbd99800640 1 Processor -- start 2024-03-28T19:23:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.331+0000 7fbd99800640 1 -- start start 2024-03-28T19:23:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.332+0000 7fbd99800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd940c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.332+0000 7fbd99800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbd940ab2b0 0x7fbd940c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.332+0000 7fbd99800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd94156870 con 0x7fbd941551a0 2024-03-28T19:23:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.332+0000 7fbd99800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd94155cf0 con 0x7fbd940ab2b0 2024-03-28T19:23:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.332+0000 7fbd99800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd94155fd0 con 0x7fbd940aada0 2024-03-28T19:23:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.333+0000 7fbd98e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbd940ab2b0 0x7fbd940c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.333+0000 7fbd93400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd940c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.333+0000 7fbd93e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbd941551a0 0x7fbd940c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:50776/0 (socket says 172.21.15.27:50776) 2024-03-28T19:23:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.333+0000 7fbd93e00640 1 -- 172.21.15.27:0/1628828926 learned_addr learned my addr 172.21.15.27:0/1628828926 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.333+0000 7fbd98e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbd940ab2b0 0x7fbd940c1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:35902/0 (socket says 172.21.15.27:35902) 2024-03-28T19:23:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.334+0000 7fbd98e00640 1 -- 172.21.15.27:0/1628828926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 msgr2=0x7fbd940c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.334+0000 7fbd98e00640 1 --2- 172.21.15.27:0/1628828926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd940c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.334+0000 7fbd98e00640 1 -- 172.21.15.27:0/1628828926 >> v1:172.21.15.155:6789/0 conn(0x7fbd941551a0 legacy=0x7fbd940c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.334+0000 7fbd98e00640 1 -- 172.21.15.27:0/1628828926 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd941562b0 con 0x7fbd940ab2b0 2024-03-28T19:23:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.335+0000 7fbd98e00640 1 --2- 172.21.15.27:0/1628828926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd940ab2b0 0x7fbd940c1fc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00bfa0 tx=0x7fbd8c02df80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf4497e6e391ef28 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.336+0000 7fbd92a00640 1 -- 172.21.15.27:0/1628828926 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c02e920 con 0x7fbd940ab2b0 2024-03-28T19:23:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.336+0000 7fbd92a00640 1 -- 172.21.15.27:0/1628828926 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c02eac0 con 0x7fbd940ab2b0 2024-03-28T19:23:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.336+0000 7fbd92a00640 1 -- 172.21.15.27:0/1628828926 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c009540 con 0x7fbd940ab2b0 2024-03-28T19:23:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.337+0000 7fbd99800640 1 -- 172.21.15.27:0/1628828926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd940ab2b0 msgr2=0x7fbd940c1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.337+0000 7fbd99800640 1 --2- 172.21.15.27:0/1628828926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd940ab2b0 0x7fbd940c1fc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00bfa0 tx=0x7fbd8c02df80 comp rx=0 tx=0).stop 2024-03-28T19:23:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.337+0000 7fbd99800640 1 -- 172.21.15.27:0/1628828926 shutdown_connections 2024-03-28T19:23:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.337+0000 7fbd99800640 1 --2- 172.21.15.27:0/1628828926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd940ab2b0 0x7fbd940c1fc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.337+0000 7fbd99800640 1 --2- 172.21.15.27:0/1628828926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd940c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.337+0000 7fbd99800640 1 -- 172.21.15.27:0/1628828926 >> 172.21.15.27:0/1628828926 conn(0x7fbd9406b0c0 msgr2=0x7fbd9406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.337+0000 7fbd99800640 1 -- 172.21.15.27:0/1628828926 shutdown_connections 2024-03-28T19:23:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.337+0000 7fbd99800640 1 -- 172.21.15.27:0/1628828926 wait complete. 2024-03-28T19:23:08.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.339+0000 7fbd99800640 1 Processor -- start 2024-03-28T19:23:08.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.339+0000 7fbd99800640 1 -- start start 2024-03-28T19:23:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd99800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd94153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd99800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd940ab2b0 0x7fbd94148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd99800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd941551a0 0x7fbd94149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd99800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940c5750 con 0x7fbd940ab2b0 2024-03-28T19:23:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd99800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd941559a0 con 0x7fbd941551a0 2024-03-28T19:23:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd99800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940c58d0 con 0x7fbd940aada0 2024-03-28T19:23:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd93400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd940ab2b0 0x7fbd94148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd93400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd940ab2b0 0x7fbd94148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54892/0 (socket says 172.21.15.27:54892) 2024-03-28T19:23:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.340+0000 7fbd93400640 1 -- 172.21.15.27:0/4122329573 learned_addr learned my addr 172.21.15.27:0/4122329573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd98e00640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd941551a0 0x7fbd94149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd93e00640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd94153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd93400640 1 -- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 msgr2=0x7fbd94153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd93400640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd94153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd93400640 1 -- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd941551a0 msgr2=0x7fbd94149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd93400640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd941551a0 0x7fbd94149280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd93400640 1 -- 172.21.15.27:0/4122329573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd94071540 con 0x7fbd940ab2b0 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd93e00640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd94153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd93400640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd940ab2b0 0x7fbd94148d40 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000e9b0 tx=0x7fbd8000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.341+0000 7fbd8be00640 1 -- 172.21.15.27:0/4122329573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd80017070 con 0x7fbd940ab2b0 2024-03-28T19:23:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.342+0000 7fbd8be00640 1 -- 172.21.15.27:0/4122329573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8000c9a0 con 0x7fbd940ab2b0 2024-03-28T19:23:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.342+0000 7fbd8be00640 1 -- 172.21.15.27:0/4122329573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8000cd90 con 0x7fbd940ab2b0 2024-03-28T19:23:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.342+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd94071820 con 0x7fbd940ab2b0 2024-03-28T19:23:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.342+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9406b7e0 con 0x7fbd940ab2b0 2024-03-28T19:23:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.344+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd94067dd0 con 0x7fbd940ab2b0 2024-03-28T19:23:08.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.346+0000 7fbd8be00640 1 -- 172.21.15.27:0/4122329573 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbd8001a070 con 0x7fbd940ab2b0 2024-03-28T19:23:08.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.347+0000 7fbd8be00640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7406bbd0 0x7fbd7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:08.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.347+0000 7fbd8be00640 1 -- 172.21.15.27:0/4122329573 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9733+0+0 (secure 0 0 0) 0x7fbd8008bdd0 con 0x7fbd940ab2b0 2024-03-28T19:23:08.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.348+0000 7fbd93e00640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7406bbd0 0x7fbd7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:08.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.348+0000 7fbd93e00640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7406bbd0 0x7fbd7406e090 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbd941540d0 tx=0x7fbd84009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:08.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.353+0000 7fbd8be00640 1 -- 172.21.15.27:0/4122329573 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbd80059940 con 0x7fbd940ab2b0 2024-03-28T19:23:08.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.662+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd94069640 con 0x7fbd940ab2b0 2024-03-28T19:23:08.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.663+0000 7fbd8be00640 1 -- 172.21.15.27:0/4122329573 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fbd8008d090 con 0x7fbd940ab2b0 2024-03-28T19:23:08.663 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:08.663 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":7537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:20:51.695880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8052},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8052":{"gid":8052,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":31,"addr":"172.21.15.27:6837/433723712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":433723712},{"type":"v1","addr":"172.21.15.27:6837","nonce":433723712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8052,"qdb_cluster":[8052]},"id":2}]} 2024-03-28T19:23:08.663 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 204 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.665+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7406bbd0 msgr2=0x7fbd7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.665+0000 7fbd99800640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7406bbd0 0x7fbd7406e090 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbd941540d0 tx=0x7fbd84009040 comp rx=0 tx=0).stop 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.665+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd940ab2b0 msgr2=0x7fbd94148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd940ab2b0 0x7fbd94148d40 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000e9b0 tx=0x7fbd8000ee70 comp rx=0 tx=0).stop 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 shutdown_connections 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbd7406bbd0 0x7fbd7406e090 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd941551a0 0x7fbd94149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd940ab2b0 0x7fbd94148d40 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 --2- 172.21.15.27:0/4122329573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd940aada0 0x7fbd94153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 >> 172.21.15.27:0/4122329573 conn(0x7fbd9406b0c0 msgr2=0x7fbd940c40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 shutdown_connections 2024-03-28T19:23:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:08.666+0000 7fbd99800640 1 -- 172.21.15.27:0/4122329573 wait complete. 2024-03-28T19:23:08.693 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:23:10.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:10.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.925+0000 7f26eb800640 1 Processor -- start 2024-03-28T19:23:10.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.925+0000 7f26eb800640 1 -- start start 2024-03-28T19:23:10.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.926+0000 7f26eb800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26e40c5e80 0x7f26e40c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:10.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26eb800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f26e40abf60 0x7f26e40aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:10.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26eb800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e406d8a0 con 0x7f26e40c1f50 2024-03-28T19:23:10.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26eb800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e406d650 con 0x7f26e40c5e80 2024-03-28T19:23:10.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26eb800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e406e1d0 con 0x7f26e40abf60 2024-03-28T19:23:10.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26e8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26e40c5e80 0x7f26e40c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:10.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26e9800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f26e40abf60 0x7f26e40aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:10.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26e8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f26e40c5e80 0x7f26e40c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54906/0 (socket says 172.21.15.27:54906) 2024-03-28T19:23:10.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26e8e00640 1 -- 172.21.15.27:0/1997540249 learned_addr learned my addr 172.21.15.27:0/1997540249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:10.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.927+0000 7f26e3e00640 1 --1- 172.21.15.27:0/1997540249 >> v1:172.21.15.155:6789/0 conn(0x7f26e40c1f50 0x7f26e40ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:50778/0 (socket says 172.21.15.27:50778) 2024-03-28T19:23:10.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.928+0000 7f26e8e00640 1 -- 172.21.15.27:0/1997540249 >> v2:172.21.15.155:3300/0 conn(0x7f26e40abf60 msgr2=0x7f26e40aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:10.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.928+0000 7f26e8e00640 1 --2- 172.21.15.27:0/1997540249 >> v2:172.21.15.155:3300/0 conn(0x7f26e40abf60 0x7f26e40aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:10.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.928+0000 7f26e8e00640 1 -- 172.21.15.27:0/1997540249 >> v1:172.21.15.155:6789/0 conn(0x7f26e40c1f50 legacy=0x7f26e40ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:10.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.928+0000 7f26e8e00640 1 -- 172.21.15.27:0/1997540249 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e406db80 con 0x7f26e40c5e80 2024-03-28T19:23:10.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.929+0000 7f26e8e00640 1 --2- 172.21.15.27:0/1997540249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40c5e80 0x7f26e40c2cb0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f26e406eff0 tx=0x7f26d802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a816ddf24e023d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:10.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.931+0000 7f26e3400640 1 -- 172.21.15.27:0/1997540249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d802c9f0 con 0x7f26e40c5e80 2024-03-28T19:23:10.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26e3400640 1 -- 172.21.15.27:0/1997540249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d802cb90 con 0x7f26e40c5e80 2024-03-28T19:23:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26e3400640 1 -- 172.21.15.27:0/1997540249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d80343f0 con 0x7f26e40c5e80 2024-03-28T19:23:10.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26eb800640 1 -- 172.21.15.27:0/1997540249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40c5e80 msgr2=0x7f26e40c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26eb800640 1 --2- 172.21.15.27:0/1997540249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40c5e80 0x7f26e40c2cb0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f26e406eff0 tx=0x7f26d802be50 comp rx=0 tx=0).stop 2024-03-28T19:23:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26eb800640 1 -- 172.21.15.27:0/1997540249 shutdown_connections 2024-03-28T19:23:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26eb800640 1 --2- 172.21.15.27:0/1997540249 >> v2:172.21.15.155:3300/0 conn(0x7f26e40abf60 0x7f26e40aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26eb800640 1 --2- 172.21.15.27:0/1997540249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40c5e80 0x7f26e40c2cb0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26eb800640 1 -- 172.21.15.27:0/1997540249 >> 172.21.15.27:0/1997540249 conn(0x7f26e406b9d0 msgr2=0x7f26e40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.932+0000 7f26eb800640 1 -- 172.21.15.27:0/1997540249 shutdown_connections 2024-03-28T19:23:10.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.933+0000 7f26eb800640 1 -- 172.21.15.27:0/1997540249 wait complete. 2024-03-28T19:23:10.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.934+0000 7f26eb800640 1 Processor -- start 2024-03-28T19:23:10.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.934+0000 7f26eb800640 1 -- start start 2024-03-28T19:23:10.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.935+0000 7f26eb800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40abf60 0x7f26e40c04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.935+0000 7f26eb800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26e40c1f50 0x7f26e4152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.935+0000 7f26eb800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26e40c5e80 0x7f26e41520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.935+0000 7f26eb800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40c3370 con 0x7f26e40abf60 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.935+0000 7f26eb800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40c31f0 con 0x7f26e40c1f50 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.935+0000 7f26eb800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40c6260 con 0x7f26e40c5e80 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40abf60 0x7f26e40c04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40abf60 0x7f26e40c04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54912/0 (socket says 172.21.15.27:54912) 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e8e00640 1 -- 172.21.15.27:0/4100748808 learned_addr learned my addr 172.21.15.27:0/4100748808 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e9800640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26e40c5e80 0x7f26e41520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e3e00640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26e40c1f50 0x7f26e4152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:10.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e8e00640 1 -- 172.21.15.27:0/4100748808 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26e40c5e80 msgr2=0x7f26e41520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:10.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e8e00640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26e40c5e80 0x7f26e41520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:10.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e8e00640 1 -- 172.21.15.27:0/4100748808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26e40c1f50 msgr2=0x7f26e4152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e8e00640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26e40c1f50 0x7f26e4152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.936+0000 7f26e8e00640 1 -- 172.21.15.27:0/4100748808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e41e2600 con 0x7f26e40abf60 2024-03-28T19:23:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.937+0000 7f26e8e00640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40abf60 0x7f26e40c04e0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000f80 tx=0x7f26d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.937+0000 7f26e0c00640 1 -- 172.21.15.27:0/4100748808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8044020 con 0x7f26e40abf60 2024-03-28T19:23:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.937+0000 7f26e0c00640 1 -- 172.21.15.27:0/4100748808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d8040470 con 0x7f26e40abf60 2024-03-28T19:23:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.937+0000 7f26e0c00640 1 -- 172.21.15.27:0/4100748808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d802c9f0 con 0x7f26e40abf60 2024-03-28T19:23:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.937+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e41dd5e0 con 0x7f26e40abf60 2024-03-28T19:23:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.937+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e40a6050 con 0x7f26e40abf60 2024-03-28T19:23:10.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.939+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e404f400 con 0x7f26e40abf60 2024-03-28T19:23:10.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.940+0000 7f26e0c00640 1 -- 172.21.15.27:0/4100748808 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f26d8047020 con 0x7f26e40abf60 2024-03-28T19:23:10.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.941+0000 7f26e0c00640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26c406bc20 0x7f26c406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:10.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.941+0000 7f26e0c00640 1 -- 172.21.15.27:0/4100748808 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f26d80ae340 con 0x7f26e40abf60 2024-03-28T19:23:10.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.942+0000 7f26e3e00640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26c406bc20 0x7f26c406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:10.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.942+0000 7f26e3e00640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26c406bc20 0x7f26c406e0e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f26e4072690 tx=0x7f26d0007ce0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:10.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:10.946+0000 7f26e0c00640 1 -- 172.21.15.27:0/4100748808 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f26d804e030 con 0x7f26e40abf60 2024-03-28T19:23:11.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.257+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e4068220 con 0x7f26e40abf60 2024-03-28T19:23:11.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.258+0000 7f26e0c00640 1 -- 172.21.15.27:0/4100748808 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f26d807be60 con 0x7f26e40abf60 2024-03-28T19:23:11.258 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:11.258 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:10.368612+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:23:11.258 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 205 2024-03-28T19:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26c406bc20 msgr2=0x7f26c406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26c406bc20 0x7f26c406e0e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f26e4072690 tx=0x7f26d0007ce0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40abf60 msgr2=0x7f26e40c04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40abf60 0x7f26e40c04e0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000f80 tx=0x7f26d8004a40 comp rx=0 tx=0).stop 2024-03-28T19:23:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 shutdown_connections 2024-03-28T19:23:11.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f26c406bc20 0x7f26c406e0e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f26e40c5e80 0x7f26e41520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26e40c1f50 0x7f26e4152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 --2- 172.21.15.27:0/4100748808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e40abf60 0x7f26e40c04e0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 >> 172.21.15.27:0/4100748808 conn(0x7f26e406b9d0 msgr2=0x7f26e4071460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:11.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.261+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 shutdown_connections 2024-03-28T19:23:11.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.262+0000 7f26eb800640 1 -- 172.21.15.27:0/4100748808 wait complete. 2024-03-28T19:23:11.283 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-03-28T19:23:11.283 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:23:11.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:11.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.571+0000 7f49d3e00640 1 Processor -- start 2024-03-28T19:23:11.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.571+0000 7f49d3e00640 1 -- start start 2024-03-28T19:23:11.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.572+0000 7f49d3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f49cc150c70 0x7f49cc0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.572+0000 7f49d3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c3690 0x7f49cc0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.572+0000 7f49d3e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f49cc0c6660 0x7f49cc1534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.572+0000 7f49d3e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49cc0ab4e0 con 0x7f49cc150c70 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.572+0000 7f49d3e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49cc0ac1f0 con 0x7f49cc0c6660 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.572+0000 7f49d3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0c3af0 con 0x7f49cc0c3690 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.572+0000 7f49d1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f49cc150c70 0x7f49cc0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f49cc150c70 0x7f49cc0a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54920/0 (socket says 172.21.15.27:54920) 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1400640 1 -- 172.21.15.27:0/2044169957 learned_addr learned my addr 172.21.15.27:0/2044169957 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1e00640 1 --2- 172.21.15.27:0/2044169957 >> v2:172.21.15.155:3300/0 conn(0x7f49cc0c6660 0x7f49cc1534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:11.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d0a00640 1 --2- 172.21.15.27:0/2044169957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c3690 0x7f49cc0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1400640 1 -- 172.21.15.27:0/2044169957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c3690 msgr2=0x7f49cc0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1400640 1 --2- 172.21.15.27:0/2044169957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c3690 0x7f49cc0c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1400640 1 -- 172.21.15.27:0/2044169957 >> v2:172.21.15.155:3300/0 conn(0x7f49cc0c6660 msgr2=0x7f49cc1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1400640 1 --2- 172.21.15.27:0/2044169957 >> v2:172.21.15.155:3300/0 conn(0x7f49cc0c6660 0x7f49cc1534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1400640 1 -- 172.21.15.27:0/2044169957 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc0c3d90 con 0x7f49cc150c70 2024-03-28T19:23:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.573+0000 7f49d1e00640 1 --2- 172.21.15.27:0/2044169957 >> v2:172.21.15.155:3300/0 conn(0x7f49cc0c6660 0x7f49cc1534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:11.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.575+0000 7f49d1400640 1 --2- 172.21.15.27:0/2044169957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc0a9cb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f49cc06c6a0 tx=0x7f49c002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f416d6123e2f4cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:11.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.575+0000 7f49cbe00640 1 -- 172.21.15.27:0/2044169957 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c002c9e0 con 0x7f49cc150c70 2024-03-28T19:23:11.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.575+0000 7f49cbe00640 1 -- 172.21.15.27:0/2044169957 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c002cb80 con 0x7f49cc150c70 2024-03-28T19:23:11.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.575+0000 7f49cbe00640 1 -- 172.21.15.27:0/2044169957 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00343f0 con 0x7f49cc150c70 2024-03-28T19:23:11.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.576+0000 7f49d3e00640 1 -- 172.21.15.27:0/2044169957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 msgr2=0x7f49cc0a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:11.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.576+0000 7f49d3e00640 1 --2- 172.21.15.27:0/2044169957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc0a9cb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f49cc06c6a0 tx=0x7f49c002be40 comp rx=0 tx=0).stop 2024-03-28T19:23:11.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.576+0000 7f49d3e00640 1 -- 172.21.15.27:0/2044169957 shutdown_connections 2024-03-28T19:23:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.576+0000 7f49d3e00640 1 --2- 172.21.15.27:0/2044169957 >> v2:172.21.15.155:3300/0 conn(0x7f49cc0c6660 0x7f49cc1534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.576+0000 7f49d3e00640 1 --2- 172.21.15.27:0/2044169957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c3690 0x7f49cc0c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.576+0000 7f49d3e00640 1 --2- 172.21.15.27:0/2044169957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc0a9cb0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.576+0000 7f49d3e00640 1 -- 172.21.15.27:0/2044169957 >> 172.21.15.27:0/2044169957 conn(0x7f49cc06b9d0 msgr2=0x7f49cc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.577+0000 7f49d3e00640 1 -- 172.21.15.27:0/2044169957 shutdown_connections 2024-03-28T19:23:11.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.577+0000 7f49d3e00640 1 -- 172.21.15.27:0/2044169957 wait complete. 2024-03-28T19:23:11.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.579+0000 7f49d3e00640 1 Processor -- start 2024-03-28T19:23:11.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.579+0000 7f49d3e00640 1 -- start start 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.580+0000 7f49d3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49cc0c3690 0x7f49cc155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.580+0000 7f49d3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c6660 0x7f49cc1ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.580+0000 7f49d3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.580+0000 7f49d3e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0abcd0 con 0x7f49cc150c70 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.580+0000 7f49d3e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0ab9f0 con 0x7f49cc0c3690 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.580+0000 7f49d3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0c1810 con 0x7f49cc0c6660 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.581+0000 7f49d1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.581+0000 7f49d0a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c6660 0x7f49cc1ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.581+0000 7f49d1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc1ee590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54936/0 (socket says 172.21.15.27:54936) 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.581+0000 7f49d1e00640 1 -- 172.21.15.27:0/415626208 learned_addr learned my addr 172.21.15.27:0/415626208 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:11.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.581+0000 7f49d1400640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49cc0c3690 0x7f49cc155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:11.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49d1e00640 1 -- 172.21.15.27:0/415626208 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c6660 msgr2=0x7f49cc1ee050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:11.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49d1e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c6660 0x7f49cc1ee050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49d1e00640 1 -- 172.21.15.27:0/415626208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49cc0c3690 msgr2=0x7f49cc155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49d1e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49cc0c3690 0x7f49cc155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49d1e00640 1 -- 172.21.15.27:0/415626208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc153eb0 con 0x7f49cc150c70 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49d1e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc1ee590 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f49c400e7e0 tx=0x7f49c400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49c9600640 1 -- 172.21.15.27:0/415626208 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c400f040 con 0x7f49cc150c70 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49c9600640 1 -- 172.21.15.27:0/415626208 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c4004030 con 0x7f49cc150c70 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.582+0000 7f49c9600640 1 -- 172.21.15.27:0/415626208 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c40043e0 con 0x7f49cc150c70 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.583+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc154150 con 0x7f49cc150c70 2024-03-28T19:23:11.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.583+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc1525a0 con 0x7f49cc150c70 2024-03-28T19:23:11.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.585+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc1531a0 con 0x7f49cc150c70 2024-03-28T19:23:11.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.586+0000 7f49c9600640 1 -- 172.21.15.27:0/415626208 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f49c400c6d0 con 0x7f49cc150c70 2024-03-28T19:23:11.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.588+0000 7f49c9600640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49ac06bbd0 0x7f49ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.588+0000 7f49c9600640 1 -- 172.21.15.27:0/415626208 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f49c408b970 con 0x7f49cc150c70 2024-03-28T19:23:11.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.588+0000 7f49d1400640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49ac06bbd0 0x7f49ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:11.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.589+0000 7f49d1400640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49ac06bbd0 0x7f49ac06e090 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f49c0035000 tx=0x7f49c0030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:11.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.593+0000 7f49c9600640 1 -- 172.21.15.27:0/415626208 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f49cc1531a0 con 0x7f49cc150c70 2024-03-28T19:23:11.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.956+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49cc04f400 con 0x7f49cc150c70 2024-03-28T19:23:11.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.958+0000 7f49c9600640 1 -- 172.21.15.27:0/415626208 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f49c4017020 con 0x7f49cc150c70 2024-03-28T19:23:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:11.958 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:10.368612+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:23:11.959 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 205 2024-03-28T19:23:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49ac06bbd0 msgr2=0x7f49ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49ac06bbd0 0x7f49ac06e090 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f49c0035000 tx=0x7f49c0030000 comp rx=0 tx=0).stop 2024-03-28T19:23:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 msgr2=0x7f49cc1ee590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc1ee590 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f49c400e7e0 tx=0x7f49c400eca0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 shutdown_connections 2024-03-28T19:23:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49ac06bbd0 0x7f49ac06e090 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49cc150c70 0x7f49cc1ee590 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49cc0c6660 0x7f49cc1ee050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 --2- 172.21.15.27:0/415626208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49cc0c3690 0x7f49cc155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.961+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 >> 172.21.15.27:0/415626208 conn(0x7f49cc06b9d0 msgr2=0x7f49cc070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.962+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 shutdown_connections 2024-03-28T19:23:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:11.962+0000 7f49d3e00640 1 -- 172.21.15.27:0/415626208 wait complete. 2024-03-28T19:23:11.983 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 205, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 203, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8259": { "addr": "172.21.15.27:6835/1299974557", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 1299974557, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 1299974557, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8259, "incarnation": 200, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 88, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:22:55.854015+0000", "qdb_cluster": [ 8259 ], "qdb_leader": 8259, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8259 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 205, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7537": { "addr": "172.21.15.155:6835/4236980323", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 4236980323, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 4236980323, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7537, "incarnation": 205, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 89, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:23:10.368612+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7537 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3334295790", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3334295790, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3334295790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 193, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6467, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:23:12.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:23:12.402 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:23:13.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:14.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.257+0000 7f7997200640 1 Processor -- start 2024-03-28T19:23:14.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.257+0000 7f7997200640 1 -- start start 2024-03-28T19:23:14.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.258+0000 7f7997200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79980c8ed0 0x7f79980c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.258+0000 7f7997200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7998158bd0 con 0x7f79980be5d0 2024-03-28T19:23:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.258+0000 7f7997200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7998158eb0 con 0x7f79980c58c0 2024-03-28T19:23:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.258+0000 7f7997200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7998159190 con 0x7f79980c8ed0 2024-03-28T19:23:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.259+0000 7f798fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79980c8ed0 0x7f79980c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.259+0000 7f798fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f79980c8ed0 0x7f79980c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54950/0 (socket says 172.21.15.27:54950) 2024-03-28T19:23:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.259+0000 7f798fe00640 1 -- 172.21.15.27:0/2423085166 learned_addr learned my addr 172.21.15.27:0/2423085166 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.259+0000 7f798fe00640 1 -- 172.21.15.27:0/2423085166 >> v1:172.21.15.155:6789/0 conn(0x7f79980c58c0 legacy=0x7f79981555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:14.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.259+0000 7f798fe00640 1 -- 172.21.15.27:0/2423085166 >> v1:172.21.15.27:6789/0 conn(0x7f79980be5d0 legacy=0x7f7998155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:14.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.259+0000 7f798fe00640 1 -- 172.21.15.27:0/2423085166 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7998159470 con 0x7f79980c8ed0 2024-03-28T19:23:14.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.261+0000 7f798fe00640 1 --2- 172.21.15.27:0/2423085166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c8ed0 0x7f79980c5ca0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f7980009870 tx=0x7f798002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe62c1f191cebd07 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:14.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.261+0000 7f7995400640 1 -- 172.21.15.27:0/2423085166 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980035020 con 0x7f79980c8ed0 2024-03-28T19:23:14.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.261+0000 7f7995400640 1 -- 172.21.15.27:0/2423085166 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7980031870 con 0x7f79980c8ed0 2024-03-28T19:23:14.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.261+0000 7f7995400640 1 -- 172.21.15.27:0/2423085166 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980031c40 con 0x7f79980c8ed0 2024-03-28T19:23:14.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.262+0000 7f7997200640 1 -- 172.21.15.27:0/2423085166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c8ed0 msgr2=0x7f79980c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:14.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.262+0000 7f7997200640 1 --2- 172.21.15.27:0/2423085166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c8ed0 0x7f79980c5ca0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f7980009870 tx=0x7f798002b910 comp rx=0 tx=0).stop 2024-03-28T19:23:14.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.262+0000 7f7997200640 1 -- 172.21.15.27:0/2423085166 shutdown_connections 2024-03-28T19:23:14.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.262+0000 7f7997200640 1 --2- 172.21.15.27:0/2423085166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c8ed0 0x7f79980c5ca0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:14.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.262+0000 7f7997200640 1 -- 172.21.15.27:0/2423085166 >> 172.21.15.27:0/2423085166 conn(0x7f799806b6f0 msgr2=0x7f79980acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:14.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.262+0000 7f7997200640 1 -- 172.21.15.27:0/2423085166 shutdown_connections 2024-03-28T19:23:14.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.262+0000 7f7997200640 1 -- 172.21.15.27:0/2423085166 wait complete. 2024-03-28T19:23:14.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.265+0000 7f7997200640 1 Processor -- start 2024-03-28T19:23:14.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.265+0000 7f7997200640 1 -- start start 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.266+0000 7f7997200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79980be5d0 0x7f79981eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.266+0000 7f7997200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c58c0 0x7f79980abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.266+0000 7f7997200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79980c8ed0 0x7f79981ef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.266+0000 7f7997200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79980b09a0 con 0x7f79980c58c0 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.266+0000 7f7997200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79980b0820 con 0x7f79980be5d0 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.266+0000 7f7997200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79980ad7d0 con 0x7f79980c8ed0 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f798fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c58c0 0x7f79980abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f798fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c58c0 0x7f79980abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54956/0 (socket says 172.21.15.27:54956) 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f798fe00640 1 -- 172.21.15.27:0/488242737 learned_addr learned my addr 172.21.15.27:0/488242737 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f7996800640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79980c8ed0 0x7f79981ef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:14.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f7995e00640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79980be5d0 0x7f79981eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:14.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f798fe00640 1 -- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79980c8ed0 msgr2=0x7f79981ef1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:14.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f798fe00640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79980c8ed0 0x7f79981ef1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:14.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f798fe00640 1 -- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79980be5d0 msgr2=0x7f79981eb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:14.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f798fe00640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79980be5d0 0x7f79981eb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:14.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f798fe00640 1 -- 172.21.15.27:0/488242737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79980c12f0 con 0x7f79980c58c0 2024-03-28T19:23:14.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.267+0000 7f7996800640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79980c8ed0 0x7f79981ef1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:14.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.268+0000 7f798fe00640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c58c0 0x7f79980abd00 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f7980000c00 tx=0x7f79800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:14.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.268+0000 7f798e000640 1 -- 172.21.15.27:0/488242737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980040070 con 0x7f79980c58c0 2024-03-28T19:23:14.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.268+0000 7f798e000640 1 -- 172.21.15.27:0/488242737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798003a420 con 0x7f79980c58c0 2024-03-28T19:23:14.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.268+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79980c15d0 con 0x7f79980c58c0 2024-03-28T19:23:14.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.268+0000 7f798e000640 1 -- 172.21.15.27:0/488242737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980035040 con 0x7f79980c58c0 2024-03-28T19:23:14.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.269+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79980c1b90 con 0x7f79980c58c0 2024-03-28T19:23:14.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.270+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7998069640 con 0x7f79980c58c0 2024-03-28T19:23:14.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.272+0000 7f798e000640 1 -- 172.21.15.27:0/488242737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f798004b060 con 0x7f79980c58c0 2024-03-28T19:23:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.273+0000 7f798e000640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f797406bbd0 0x7f797406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.273+0000 7f798e000640 1 -- 172.21.15.27:0/488242737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f79800ad9b0 con 0x7f79980c58c0 2024-03-28T19:23:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.274+0000 7f7995e00640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f797406bbd0 0x7f797406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:14.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.275+0000 7f7995e00640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f797406bbd0 0x7f797406e090 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7988000f60 tx=0x7f7988009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:14.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.279+0000 7f798e000640 1 -- 172.21.15.27:0/488242737 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f798007b440 con 0x7f79980c58c0 2024-03-28T19:23:14.520 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 6% (53/779) Updating files: 7% (55/779) Updating files: 8% (63/779) Updating files: 9% (71/779) Updating files: 10% (78/779) Updating files: 11% (86/779) Updating files: 12% (94/779) Updating files: 13% (102/779) Updating files: 13% (107/779) Updating files: 14% (110/779) Updating files: 15% (117/779) Updating files: 16% (125/779) Updating files: 17% (133/779) Updating files: 18% (141/779) Updating files: 19% (149/779) Updating files: 20% (156/779) Updating files: 21% (164/779) Updating files: 21% (165/779) Updating files: 22% (172/779) Updating files: 23% (180/779) Updating files: 24% (187/779) Updating files: 25% (195/779) Updating files: 26% (203/779) Updating files: 27% (211/779) Updating files: 27% (217/779) Updating files: 28% (219/779) Updating files: 29% (226/779) Updating files: 30% (234/779) Updating files: 31% (242/779) Updating files: 32% (250/779) Updating files: 33% (258/779) Updating files: 34% (265/779) Updating files: 35% (273/779) Updating files: 35% (278/779) Updating files: 36% (281/779) Updating files: 37% (289/779) Updating files: 38% (297/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 42% (332/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 49% (385/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 56% (440/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 63% (495/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 68% (537/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 74% (582/779) Updating files: 75% (585/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 78% (612/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 83% (648/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 87% (684/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 92% (720/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 96% (755/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2024-03-28T19:23:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.606+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7998156580 con 0x7f79980c58c0 2024-03-28T19:23:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.607+0000 7f798e000640 1 -- 172.21.15.27:0/488242737 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f798007ad40 con 0x7f79980c58c0 2024-03-28T19:23:14.607 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:14.607 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:10.368612+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:23:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 205 2024-03-28T19:23:14.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f797406bbd0 msgr2=0x7f797406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:14.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f797406bbd0 0x7f797406e090 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7988000f60 tx=0x7f7988009040 comp rx=0 tx=0).stop 2024-03-28T19:23:14.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c58c0 msgr2=0x7f79980abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:14.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c58c0 0x7f79980abd00 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f7980000c00 tx=0x7f79800046c0 comp rx=0 tx=0).stop 2024-03-28T19:23:14.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 shutdown_connections 2024-03-28T19:23:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f797406bbd0 0x7f797406e090 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79980c8ed0 0x7f79981ef1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79980c58c0 0x7f79980abd00 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 --2- 172.21.15.27:0/488242737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79980be5d0 0x7f79981eb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 >> 172.21.15.27:0/488242737 conn(0x7f799806b6f0 msgr2=0x7f7998158470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 shutdown_connections 2024-03-28T19:23:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:14.610+0000 7f7997200640 1 -- 172.21.15.27:0/488242737 wait complete. 2024-03-28T19:23:14.633 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:23:14.981 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 2f10137986a v0.18 2024-03-28T19:23:14.999 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.18 2024-03-28T19:23:15.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:23:15.071 INFO:tasks.workunit.client.0.smithi027.stdout:v0.19 2024-03-28T19:23:15.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 19 -eq 48 ']' 2024-03-28T19:23:15.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.19 2024-03-28T19:23:15.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.19 2024-03-28T19:23:15.071 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.19 2024-03-28T19:23:16.634 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:16.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.884+0000 7f18bf200640 1 Processor -- start 2024-03-28T19:23:16.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.884+0000 7f18bf200640 1 -- start start 2024-03-28T19:23:16.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.885+0000 7f18bf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c00c69a0 0x7f18c00c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:16.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.885+0000 7f18bf200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c0150ec0 con 0x7f18c0153a30 2024-03-28T19:23:16.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.885+0000 7f18bf200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c006dc70 con 0x7f18c0150ae0 2024-03-28T19:23:16.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.885+0000 7f18bf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c0153550 con 0x7f18c00c69a0 2024-03-28T19:23:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.886+0000 7f18be800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f18c0153a30 0x7f18c015dd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39810/0 (socket says 172.21.15.27:39810) 2024-03-28T19:23:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.886+0000 7f18be800640 1 -- 172.21.15.27:0/446258125 learned_addr learned my addr 172.21.15.27:0/446258125 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.886+0000 7f18b7e00640 1 --2- 172.21.15.27:0/446258125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c00c69a0 0x7f18c00c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.887+0000 7f18bd400640 1 -- 172.21.15.27:0/446258125 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624625473 0 0) 0x7f18c006dc70 con 0x7f18c0150ae0 2024-03-28T19:23:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.887+0000 7f18bd400640 1 -- 172.21.15.27:0/446258125 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a4003520 con 0x7f18c0150ae0 2024-03-28T19:23:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.887+0000 7f18bd400640 1 -- 172.21.15.27:0/446258125 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445451607 0 0) 0x7f18b8001490 con 0x7f18c0153a30 2024-03-28T19:23:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.887+0000 7f18bd400640 1 -- 172.21.15.27:0/446258125 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c006dc70 con 0x7f18c0153a30 2024-03-28T19:23:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.887+0000 7f18b7e00640 1 -- 172.21.15.27:0/446258125 >> v1:172.21.15.155:6789/0 conn(0x7f18c0150ae0 legacy=0x7f18c00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.887+0000 7f18b7e00640 1 -- 172.21.15.27:0/446258125 >> v1:172.21.15.27:6789/0 conn(0x7f18c0153a30 legacy=0x7f18c015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.887+0000 7f18b7e00640 1 -- 172.21.15.27:0/446258125 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c006df10 con 0x7f18c00c69a0 2024-03-28T19:23:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.888+0000 7f18b7e00640 1 --2- 172.21.15.27:0/446258125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c00c69a0 0x7f18c00c6e00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18a80097d0 tx=0x7f18a802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14d9002411cccd71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:16.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.889+0000 7f18bd400640 1 -- 172.21.15.27:0/446258125 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8035020 con 0x7f18c00c69a0 2024-03-28T19:23:16.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.889+0000 7f18bd400640 1 -- 172.21.15.27:0/446258125 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a8031810 con 0x7f18c00c69a0 2024-03-28T19:23:16.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.889+0000 7f18bd400640 1 -- 172.21.15.27:0/446258125 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8031bc0 con 0x7f18c00c69a0 2024-03-28T19:23:16.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.889+0000 7f18bf200640 1 -- 172.21.15.27:0/446258125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c00c69a0 msgr2=0x7f18c00c6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.890+0000 7f18bf200640 1 --2- 172.21.15.27:0/446258125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c00c69a0 0x7f18c00c6e00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18a80097d0 tx=0x7f18a802b8b0 comp rx=0 tx=0).stop 2024-03-28T19:23:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.890+0000 7f18bf200640 1 -- 172.21.15.27:0/446258125 shutdown_connections 2024-03-28T19:23:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.890+0000 7f18bf200640 1 --2- 172.21.15.27:0/446258125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c00c69a0 0x7f18c00c6e00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.890+0000 7f18bf200640 1 -- 172.21.15.27:0/446258125 >> 172.21.15.27:0/446258125 conn(0x7f18c006b0c0 msgr2=0x7f18c006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.890+0000 7f18bf200640 1 -- 172.21.15.27:0/446258125 shutdown_connections 2024-03-28T19:23:16.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.890+0000 7f18bf200640 1 -- 172.21.15.27:0/446258125 wait complete. 2024-03-28T19:23:16.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.892+0000 7f18bf200640 1 Processor -- start 2024-03-28T19:23:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.892+0000 7f18bf200640 1 -- start start 2024-03-28T19:23:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.893+0000 7f18bf200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18c00c69a0 0x7f18c00a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.893+0000 7f18bf200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18c0150ae0 0x7f18c00727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.893+0000 7f18bf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c0153a30 0x7f18c00bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.893+0000 7f18bf200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c006e7b0 con 0x7f18c00c69a0 2024-03-28T19:23:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.893+0000 7f18bf200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c006e4d0 con 0x7f18c0150ae0 2024-03-28T19:23:16.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.893+0000 7f18bf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c00a6370 con 0x7f18c0153a30 2024-03-28T19:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.893+0000 7f18bde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18c00c69a0 0x7f18c00a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18bde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18c00c69a0 0x7f18c00a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50346/0 (socket says 172.21.15.27:50346) 2024-03-28T19:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18b7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18c0150ae0 0x7f18c00727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18bde00640 1 -- 172.21.15.27:0/1073542589 learned_addr learned my addr 172.21.15.27:0/1073542589 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18be800640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c0153a30 0x7f18c00bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18bde00640 1 -- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c0153a30 msgr2=0x7f18c00bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18bde00640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c0153a30 0x7f18c00bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18bde00640 1 -- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18c0150ae0 msgr2=0x7f18c00727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18bde00640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18c0150ae0 0x7f18c00727c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18bde00640 1 -- 172.21.15.27:0/1073542589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c01dcad0 con 0x7f18c00c69a0 2024-03-28T19:23:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18be800640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c0153a30 0x7f18c00bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:23:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.894+0000 7f18b7e00640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18c0150ae0 0x7f18c00727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.895+0000 7f18bde00640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18c00c69a0 0x7f18c00a5c60 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f18b0001eb0 tx=0x7f18b000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.895+0000 7f18b6000640 1 -- 172.21.15.27:0/1073542589 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0017070 con 0x7f18c00c69a0 2024-03-28T19:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.895+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c0072ec0 con 0x7f18c00c69a0 2024-03-28T19:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.895+0000 7f18b6000640 1 -- 172.21.15.27:0/1073542589 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b00052d0 con 0x7f18c00c69a0 2024-03-28T19:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.896+0000 7f18b6000640 1 -- 172.21.15.27:0/1073542589 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0012870 con 0x7f18c00c69a0 2024-03-28T19:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.896+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c01ee6d0 con 0x7f18c00c69a0 2024-03-28T19:23:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.897+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c0067dd0 con 0x7f18c00c69a0 2024-03-28T19:23:16.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.899+0000 7f18b6000640 1 -- 172.21.15.27:0/1073542589 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f18b001a070 con 0x7f18c00c69a0 2024-03-28T19:23:16.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.901+0000 7f18b6000640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f189c06bbd0 0x7f189c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:16.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.901+0000 7f18b6000640 1 -- 172.21.15.27:0/1073542589 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f18b008ad50 con 0x7f18c00c69a0 2024-03-28T19:23:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.901+0000 7f18b7e00640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f189c06bbd0 0x7f189c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:16.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.902+0000 7f18b7e00640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f189c06bbd0 0x7f189c06e090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f18a8036000 tx=0x7f18a8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:16.910+0000 7f18b6000640 1 -- 172.21.15.27:0/1073542589 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f18b008e030 con 0x7f18c00c69a0 2024-03-28T19:23:17.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.279+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18c0069640 con 0x7f18c00c69a0 2024-03-28T19:23:17.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.280+0000 7f18b6000640 1 -- 172.21.15.27:0/1073542589 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f18c01dcad0 con 0x7f18c00c69a0 2024-03-28T19:23:17.279 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:17.279 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:10.368612+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:23:17.279 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 205 2024-03-28T19:23:17.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f189c06bbd0 msgr2=0x7f189c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:17.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f189c06bbd0 0x7f189c06e090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f18a8036000 tx=0x7f18a8008030 comp rx=0 tx=0).stop 2024-03-28T19:23:17.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18c00c69a0 msgr2=0x7f18c00a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:17.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18c00c69a0 0x7f18c00a5c60 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f18b0001eb0 tx=0x7f18b000c960 comp rx=0 tx=0).stop 2024-03-28T19:23:17.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 shutdown_connections 2024-03-28T19:23:17.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f189c06bbd0 0x7f189c06e090 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:17.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18c0153a30 0x7f18c00bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:17.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18c0150ae0 0x7f18c00727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:17.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 --2- 172.21.15.27:0/1073542589 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18c00c69a0 0x7f18c00a5c60 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:17.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 >> 172.21.15.27:0/1073542589 conn(0x7f18c006b0c0 msgr2=0x7f18c0153e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:17.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 shutdown_connections 2024-03-28T19:23:17.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:17.284+0000 7f18bf200640 1 -- 172.21.15.27:0/1073542589 wait complete. 2024-03-28T19:23:17.303 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:23:19.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.550+0000 7fd180c00640 1 Processor -- start 2024-03-28T19:23:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.550+0000 7fd180c00640 1 -- start start 2024-03-28T19:23:19.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.552+0000 7fd180c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd17c0c1f50 0x7fd17c0ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.552+0000 7fd180c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c0ac1c0 con 0x7fd17c0c5e80 2024-03-28T19:23:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.552+0000 7fd180c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c06d8a0 con 0x7fd17c0abd90 2024-03-28T19:23:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.552+0000 7fd180c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c06d650 con 0x7fd17c0c1f50 2024-03-28T19:23:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.552+0000 7fd179e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd17c0c5e80 0x7fd17c0c2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39818/0 (socket says 172.21.15.27:39818) 2024-03-28T19:23:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.552+0000 7fd179e00640 1 -- 172.21.15.27:0/1012872446 learned_addr learned my addr 172.21.15.27:0/1012872446 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.553+0000 7fd179400640 1 --2- 172.21.15.27:0/1012872446 >> v2:172.21.15.155:3300/0 conn(0x7fd17c0c1f50 0x7fd17c0ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.553+0000 7fd178a00640 1 -- 172.21.15.27:0/1012872446 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327592445 0 0) 0x7fd17c0ac1c0 con 0x7fd17c0c5e80 2024-03-28T19:23:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.553+0000 7fd178a00640 1 -- 172.21.15.27:0/1012872446 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd160003520 con 0x7fd17c0c5e80 2024-03-28T19:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.554+0000 7fd179400640 1 -- 172.21.15.27:0/1012872446 >> v1:172.21.15.155:6789/0 conn(0x7fd17c0abd90 legacy=0x7fd17c0aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.554+0000 7fd179400640 1 -- 172.21.15.27:0/1012872446 >> v1:172.21.15.27:6789/0 conn(0x7fd17c0c5e80 legacy=0x7fd17c0c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.554+0000 7fd179400640 1 -- 172.21.15.27:0/1012872446 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c06db80 con 0x7fd17c0c1f50 2024-03-28T19:23:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.556+0000 7fd179400640 1 --2- 172.21.15.27:0/1012872446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd17c0c1f50 0x7fd17c0ab850 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd1680097a0 tx=0x7fd16802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71801357d8ed5c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:19.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.557+0000 7fd178a00640 1 -- 172.21.15.27:0/1012872446 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1680315a0 con 0x7fd17c0c1f50 2024-03-28T19:23:19.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.557+0000 7fd178a00640 1 -- 172.21.15.27:0/1012872446 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd168037070 con 0x7fd17c0c1f50 2024-03-28T19:23:19.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.557+0000 7fd178a00640 1 -- 172.21.15.27:0/1012872446 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168031960 con 0x7fd17c0c1f50 2024-03-28T19:23:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.558+0000 7fd180c00640 1 -- 172.21.15.27:0/1012872446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd17c0c1f50 msgr2=0x7fd17c0ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.558+0000 7fd180c00640 1 --2- 172.21.15.27:0/1012872446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd17c0c1f50 0x7fd17c0ab850 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd1680097a0 tx=0x7fd16802b840 comp rx=0 tx=0).stop 2024-03-28T19:23:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.558+0000 7fd180c00640 1 -- 172.21.15.27:0/1012872446 shutdown_connections 2024-03-28T19:23:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.558+0000 7fd180c00640 1 --2- 172.21.15.27:0/1012872446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd17c0c1f50 0x7fd17c0ab850 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:19.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.558+0000 7fd180c00640 1 -- 172.21.15.27:0/1012872446 >> 172.21.15.27:0/1012872446 conn(0x7fd17c06b9d0 msgr2=0x7fd17c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.559+0000 7fd180c00640 1 -- 172.21.15.27:0/1012872446 shutdown_connections 2024-03-28T19:23:19.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.559+0000 7fd180c00640 1 -- 172.21.15.27:0/1012872446 wait complete. 2024-03-28T19:23:19.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.562+0000 7fd180c00640 1 Processor -- start 2024-03-28T19:23:19.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.562+0000 7fd180c00640 1 -- start start 2024-03-28T19:23:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.564+0000 7fd180c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd17c0abd90 0x7fd17c0bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.564+0000 7fd180c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd17c0c1f50 0x7fd17c0a8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.564+0000 7fd180c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17c0c5e80 0x7fd17c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.564+0000 7fd180c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0c6260 con 0x7fd17c0c5e80 2024-03-28T19:23:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.564+0000 7fd180c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c06e1d0 con 0x7fd17c0c1f50 2024-03-28T19:23:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.564+0000 7fd180c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0c63e0 con 0x7fd17c0abd90 2024-03-28T19:23:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.565+0000 7fd17a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17c0c5e80 0x7fd17c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.565+0000 7fd17a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17c0c5e80 0x7fd17c14dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50362/0 (socket says 172.21.15.27:50362) 2024-03-28T19:23:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.565+0000 7fd17a800640 1 -- 172.21.15.27:0/315027417 learned_addr learned my addr 172.21.15.27:0/315027417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.565+0000 7fd17a800640 1 -- 172.21.15.27:0/315027417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd17c0abd90 msgr2=0x7fd17c0bf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.565+0000 7fd17a800640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd17c0abd90 0x7fd17c0bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.565+0000 7fd17a800640 1 -- 172.21.15.27:0/315027417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd17c0c1f50 msgr2=0x7fd17c0a8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.565+0000 7fd17a800640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd17c0c1f50 0x7fd17c0a8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.565+0000 7fd17a800640 1 -- 172.21.15.27:0/315027417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c14c000 con 0x7fd17c0c5e80 2024-03-28T19:23:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.566+0000 7fd17a800640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17c0c5e80 0x7fd17c14dfe0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c6b0 tx=0x7fd17400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.566+0000 7fd172000640 1 -- 172.21.15.27:0/315027417 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174003b70 con 0x7fd17c0c5e80 2024-03-28T19:23:19.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.566+0000 7fd172000640 1 -- 172.21.15.27:0/315027417 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd174003d30 con 0x7fd17c0c5e80 2024-03-28T19:23:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.566+0000 7fd172000640 1 -- 172.21.15.27:0/315027417 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174005310 con 0x7fd17c0c5e80 2024-03-28T19:23:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.566+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c0bbcc0 con 0x7fd17c0c5e80 2024-03-28T19:23:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.567+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c0bbf60 con 0x7fd17c0c5e80 2024-03-28T19:23:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.568+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c04f400 con 0x7fd17c0c5e80 2024-03-28T19:23:19.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.570+0000 7fd172000640 1 -- 172.21.15.27:0/315027417 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd1740054b0 con 0x7fd17c0c5e80 2024-03-28T19:23:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.571+0000 7fd172000640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd15c06bbd0 0x7fd15c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.571+0000 7fd172000640 1 -- 172.21.15.27:0/315027417 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7fd17408b8c0 con 0x7fd17c0c5e80 2024-03-28T19:23:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.572+0000 7fd179e00640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd15c06bbd0 0x7fd15c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.573+0000 7fd179e00640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd15c06bbd0 0x7fd15c06e090 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd16c000f60 tx=0x7fd16c0078f0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:19.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.577+0000 7fd172000640 1 -- 172.21.15.27:0/315027417 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd1740593e0 con 0x7fd17c0c5e80 2024-03-28T19:23:19.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.927+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd17c0c2710 con 0x7fd17c0c5e80 2024-03-28T19:23:19.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.928+0000 7fd172000640 1 -- 172.21.15.27:0/315027417 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5466 (secure 0 0 0) 0x7fd174058ce0 con 0x7fd17c0c5e80 2024-03-28T19:23:19.927 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:19.928 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:17.979559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:23:19.928 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 207 2024-03-28T19:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.931+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd15c06bbd0 msgr2=0x7fd15c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.931+0000 7fd180c00640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd15c06bbd0 0x7fd15c06e090 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd16c000f60 tx=0x7fd16c0078f0 comp rx=0 tx=0).stop 2024-03-28T19:23:19.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.931+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17c0c5e80 msgr2=0x7fd17c14dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.931+0000 7fd180c00640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17c0c5e80 0x7fd17c14dfe0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c6b0 tx=0x7fd17400cb70 comp rx=0 tx=0).stop 2024-03-28T19:23:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.932+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 shutdown_connections 2024-03-28T19:23:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.932+0000 7fd180c00640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd15c06bbd0 0x7fd15c06e090 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.932+0000 7fd180c00640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd17c0c5e80 0x7fd17c14dfe0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.932+0000 7fd180c00640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd17c0c1f50 0x7fd17c0a8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:19.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.932+0000 7fd180c00640 1 --2- 172.21.15.27:0/315027417 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd17c0abd90 0x7fd17c0bf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.932+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 >> 172.21.15.27:0/315027417 conn(0x7fd17c06b9d0 msgr2=0x7fd17c0c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.932+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 shutdown_connections 2024-03-28T19:23:19.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:19.932+0000 7fd180c00640 1 -- 172.21.15.27:0/315027417 wait complete. 2024-03-28T19:23:19.954 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:23:21.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:21.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.376+0000 7f87d0600640 1 Processor -- start 2024-03-28T19:23:21.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.376+0000 7f87d0600640 1 -- start start 2024-03-28T19:23:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.378+0000 7f87d0600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87c80a9cb0 0x7f87c80c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.378+0000 7f87d0600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f87c81534c0 0x7f87c8153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.378+0000 7f87d0600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c80c3d90 con 0x7f87c8150c70 2024-03-28T19:23:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.378+0000 7f87d0600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c80ab4e0 con 0x7f87c80a9cb0 2024-03-28T19:23:21.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.378+0000 7f87d0600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c80ac1f0 con 0x7f87c81534c0 2024-03-28T19:23:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.379+0000 7f87cd200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f87c80a9cb0 0x7f87c80c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.379+0000 7f87ce600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f87c81534c0 0x7f87c8153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.379+0000 7f87ce600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f87c81534c0 0x7f87c8153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:38320/0 (socket says 172.21.15.27:38320) 2024-03-28T19:23:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.379+0000 7f87ce600640 1 -- 172.21.15.27:0/3477266700 learned_addr learned my addr 172.21.15.27:0/3477266700 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:21.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.379+0000 7f87cd200640 1 --2- 172.21.15.27:0/3477266700 >> v2:172.21.15.27:3300/0 conn(0x7f87c80a9cb0 0x7f87c80c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50364/0 (socket says 172.21.15.27:50364) 2024-03-28T19:23:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.380+0000 7f87cd200640 1 -- 172.21.15.27:0/3477266700 >> v2:172.21.15.155:3300/0 conn(0x7f87c81534c0 msgr2=0x7f87c8153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.380+0000 7f87cd200640 1 --2- 172.21.15.27:0/3477266700 >> v2:172.21.15.155:3300/0 conn(0x7f87c81534c0 0x7f87c8153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.380+0000 7f87cd200640 1 -- 172.21.15.27:0/3477266700 >> v1:172.21.15.155:6789/0 conn(0x7f87c8150c70 legacy=0x7f87c80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.380+0000 7f87cd200640 1 -- 172.21.15.27:0/3477266700 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c80ab710 con 0x7f87c80a9cb0 2024-03-28T19:23:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.381+0000 7f87cd200640 1 --2- 172.21.15.27:0/3477266700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c80a9cb0 0x7f87c80c3690 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f87c80c4610 tx=0x7f87b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2bcb81f7ce11ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:21.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.381+0000 7f87c7e00640 1 -- 172.21.15.27:0/3477266700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b40315a0 con 0x7f87c80a9cb0 2024-03-28T19:23:21.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.381+0000 7f87c7e00640 1 -- 172.21.15.27:0/3477266700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b4037070 con 0x7f87c80a9cb0 2024-03-28T19:23:21.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.381+0000 7f87c7e00640 1 -- 172.21.15.27:0/3477266700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4031960 con 0x7f87c80a9cb0 2024-03-28T19:23:21.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.382+0000 7f87d0600640 1 -- 172.21.15.27:0/3477266700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c80a9cb0 msgr2=0x7f87c80c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:21.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.382+0000 7f87d0600640 1 --2- 172.21.15.27:0/3477266700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c80a9cb0 0x7f87c80c3690 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f87c80c4610 tx=0x7f87b402b840 comp rx=0 tx=0).stop 2024-03-28T19:23:21.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.382+0000 7f87d0600640 1 -- 172.21.15.27:0/3477266700 shutdown_connections 2024-03-28T19:23:21.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.382+0000 7f87d0600640 1 --2- 172.21.15.27:0/3477266700 >> v2:172.21.15.155:3300/0 conn(0x7f87c81534c0 0x7f87c8153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.382+0000 7f87d0600640 1 --2- 172.21.15.27:0/3477266700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c80a9cb0 0x7f87c80c3690 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.382+0000 7f87d0600640 1 -- 172.21.15.27:0/3477266700 >> 172.21.15.27:0/3477266700 conn(0x7f87c806b9d0 msgr2=0x7f87c80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.382+0000 7f87d0600640 1 -- 172.21.15.27:0/3477266700 shutdown_connections 2024-03-28T19:23:21.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.382+0000 7f87d0600640 1 -- 172.21.15.27:0/3477266700 wait complete. 2024-03-28T19:23:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.384+0000 7f87d0600640 1 Processor -- start 2024-03-28T19:23:21.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.385+0000 7f87d0600640 1 -- start start 2024-03-28T19:23:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87d0600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87c80a9cb0 0x7f87c81ef110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87d0600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c8150c70 0x7f87c80c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87d0600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87c81534c0 0x7f87c81eb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:21.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87d0600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80c1810 con 0x7f87c8150c70 2024-03-28T19:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87d0600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80abcd0 con 0x7f87c81534c0 2024-03-28T19:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87d0600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80c6830 con 0x7f87c80a9cb0 2024-03-28T19:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87cd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c8150c70 0x7f87c80c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87cd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c8150c70 0x7f87c80c49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50370/0 (socket says 172.21.15.27:50370) 2024-03-28T19:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87cd200640 1 -- 172.21.15.27:0/2553655833 learned_addr learned my addr 172.21.15.27:0/2553655833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.386+0000 7f87cdc00640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87c80a9cb0 0x7f87c81ef110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.387+0000 7f87ce600640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87c81534c0 0x7f87c81eb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.387+0000 7f87cd200640 1 -- 172.21.15.27:0/2553655833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87c80a9cb0 msgr2=0x7f87c81ef110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.387+0000 7f87cd200640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87c80a9cb0 0x7f87c81ef110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.387+0000 7f87cd200640 1 -- 172.21.15.27:0/2553655833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87c81534c0 msgr2=0x7f87c81eb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.387+0000 7f87cd200640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87c81534c0 0x7f87c81eb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.387+0000 7f87cd200640 1 -- 172.21.15.27:0/2553655833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c8155080 con 0x7f87c8150c70 2024-03-28T19:23:21.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.388+0000 7f87cd200640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c8150c70 0x7f87c80c49d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f87b4034040 tx=0x7f87b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.388+0000 7f87c5600640 1 -- 172.21.15.27:0/2553655833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4032630 con 0x7f87c8150c70 2024-03-28T19:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.388+0000 7f87c5600640 1 -- 172.21.15.27:0/2553655833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b40327f0 con 0x7f87c8150c70 2024-03-28T19:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.389+0000 7f87c5600640 1 -- 172.21.15.27:0/2553655833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4032be0 con 0x7f87c8150c70 2024-03-28T19:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.389+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c8154ac0 con 0x7f87c8150c70 2024-03-28T19:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.389+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c806e2f0 con 0x7f87c8150c70 2024-03-28T19:23:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.391+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c804f400 con 0x7f87c8150c70 2024-03-28T19:23:21.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.393+0000 7f87c5600640 1 -- 172.21.15.27:0/2553655833 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f87b403b3f0 con 0x7f87c8150c70 2024-03-28T19:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.394+0000 7f87c5600640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87a806bc00 0x7f87a806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.394+0000 7f87c5600640 1 -- 172.21.15.27:0/2553655833 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f87b402f030 con 0x7f87c8150c70 2024-03-28T19:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.395+0000 7f87cdc00640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87a806bc00 0x7f87a806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:21.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.395+0000 7f87cdc00640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87a806bc00 0x7f87a806e0c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f87c80a5f00 tx=0x7f87bc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:21.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.405+0000 7f87c5600640 1 -- 172.21.15.27:0/2553655833 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f87b4040070 con 0x7f87c8150c70 2024-03-28T19:23:21.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.777+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87c81eb110 con 0x7f87c8150c70 2024-03-28T19:23:21.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.777+0000 7f87c5600640 1 -- 172.21.15.27:0/2553655833 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5473 (secure 0 0 0) 0x7f87b407b520 con 0x7f87c8150c70 2024-03-28T19:23:21.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:21.777 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:21.777 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 209 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.780+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87a806bc00 msgr2=0x7f87a806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.780+0000 7f87d0600640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87a806bc00 0x7f87a806e0c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f87c80a5f00 tx=0x7f87bc009040 comp rx=0 tx=0).stop 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.780+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c8150c70 msgr2=0x7f87c80c49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.780+0000 7f87d0600640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c8150c70 0x7f87c80c49d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f87b4034040 tx=0x7f87b40046c0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.781+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 shutdown_connections 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.781+0000 7f87d0600640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f87a806bc00 0x7f87a806e0c0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.781+0000 7f87d0600640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87c81534c0 0x7f87c81eb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.781+0000 7f87d0600640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87c8150c70 0x7f87c80c49d0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.781+0000 7f87d0600640 1 --2- 172.21.15.27:0/2553655833 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f87c80a9cb0 0x7f87c81ef110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.781+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 >> 172.21.15.27:0/2553655833 conn(0x7f87c806b9d0 msgr2=0x7f87c80c26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.781+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 shutdown_connections 2024-03-28T19:23:21.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:21.781+0000 7f87d0600640 1 -- 172.21.15.27:0/2553655833 wait complete. 2024-03-28T19:23:21.801 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-03-28T19:23:21.801 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:23:21.801 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:23:21.862 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:23:21.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:21.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:22.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.086+0000 7f4e57200640 1 Processor -- start 2024-03-28T19:23:22.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.086+0000 7f4e57200640 1 -- start start 2024-03-28T19:23:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.087+0000 7f4e57200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5814be30 0x7f4e58158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.087+0000 7f4e57200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e58155df0 0x7f4e5806fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.087+0000 7f4e57200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e58159010 con 0x7f4e5806ffb0 2024-03-28T19:23:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.087+0000 7f4e57200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e58155960 con 0x7f4e58155df0 2024-03-28T19:23:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.087+0000 7f4e57200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580c6bb0 con 0x7f4e5814be30 2024-03-28T19:23:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.087+0000 7f4e55400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e58155df0 0x7f4e5806fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.088+0000 7f4e55400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e58155df0 0x7f4e5806fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50374/0 (socket says 172.21.15.27:50374) 2024-03-28T19:23:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.088+0000 7f4e55400640 1 -- 172.21.15.27:0/4281478291 learned_addr learned my addr 172.21.15.27:0/4281478291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.088+0000 7f4e55e00640 1 --2- 172.21.15.27:0/4281478291 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5814be30 0x7f4e58158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.088+0000 7f4e55400640 1 -- 172.21.15.27:0/4281478291 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5814be30 msgr2=0x7f4e58158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.088+0000 7f4e55400640 1 --2- 172.21.15.27:0/4281478291 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5814be30 0x7f4e58158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.088+0000 7f4e55400640 1 -- 172.21.15.27:0/4281478291 >> v1:172.21.15.155:6789/0 conn(0x7f4e5806ffb0 legacy=0x7f4e580a7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:22.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.088+0000 7f4e55400640 1 -- 172.21.15.27:0/4281478291 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e580c6e50 con 0x7f4e58155df0 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.089+0000 7f4e55400640 1 --2- 172.21.15.27:0/4281478291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e58155df0 0x7f4e5806fa70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009870 tx=0x7f4e4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e46a0c964b54cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.089+0000 7f4e54a00640 1 -- 172.21.15.27:0/4281478291 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44035020 con 0x7f4e58155df0 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.089+0000 7f4e54a00640 1 -- 172.21.15.27:0/4281478291 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e44031640 con 0x7f4e58155df0 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.089+0000 7f4e54a00640 1 -- 172.21.15.27:0/4281478291 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4402d040 con 0x7f4e58155df0 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.090+0000 7f4e57200640 1 -- 172.21.15.27:0/4281478291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e58155df0 msgr2=0x7f4e5806fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.090+0000 7f4e57200640 1 --2- 172.21.15.27:0/4281478291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e58155df0 0x7f4e5806fa70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009870 tx=0x7f4e4402b8f0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.090+0000 7f4e57200640 1 -- 172.21.15.27:0/4281478291 shutdown_connections 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.090+0000 7f4e57200640 1 --2- 172.21.15.27:0/4281478291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e58155df0 0x7f4e5806fa70 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.090+0000 7f4e57200640 1 --2- 172.21.15.27:0/4281478291 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5814be30 0x7f4e58158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.090+0000 7f4e57200640 1 -- 172.21.15.27:0/4281478291 >> 172.21.15.27:0/4281478291 conn(0x7f4e58072bd0 msgr2=0x7f4e58072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:22.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.090+0000 7f4e57200640 1 -- 172.21.15.27:0/4281478291 shutdown_connections 2024-03-28T19:23:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.090+0000 7f4e57200640 1 -- 172.21.15.27:0/4281478291 wait complete. 2024-03-28T19:23:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.092+0000 7f4e57200640 1 Processor -- start 2024-03-28T19:23:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.092+0000 7f4e57200640 1 -- start start 2024-03-28T19:23:22.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.093+0000 7f4e57200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5806ffb0 0x7f4e58158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.093+0000 7f4e57200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5814be30 0x7f4e58154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.093+0000 7f4e57200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e58155df0 0x7f4e581481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e57200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580a8420 con 0x7f4e5814be30 2024-03-28T19:23:22.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e57200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580c6400 con 0x7f4e58155df0 2024-03-28T19:23:22.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e57200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e58157f60 con 0x7f4e5806ffb0 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5814be30 0x7f4e58154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5814be30 0x7f4e58154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50384/0 (socket says 172.21.15.27:50384) 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55400640 1 -- 172.21.15.27:0/1115991292 learned_addr learned my addr 172.21.15.27:0/1115991292 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55e00640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5806ffb0 0x7f4e58158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e56800640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e58155df0 0x7f4e581481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55400640 1 -- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5806ffb0 msgr2=0x7f4e58158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55400640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5806ffb0 0x7f4e58158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55400640 1 -- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e58155df0 msgr2=0x7f4e581481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55400640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e58155df0 0x7f4e581481d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55400640 1 -- 172.21.15.27:0/1115991292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e58154f50 con 0x7f4e5814be30 2024-03-28T19:23:22.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.094+0000 7f4e55e00640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5806ffb0 0x7f4e58158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.095+0000 7f4e55400640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5814be30 0x7f4e58154610 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f4e4402b930 tx=0x7f4e440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.095+0000 7f4e4e000640 1 -- 172.21.15.27:0/1115991292 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44040070 con 0x7f4e5814be30 2024-03-28T19:23:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.095+0000 7f4e4e000640 1 -- 172.21.15.27:0/1115991292 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e440399f0 con 0x7f4e5814be30 2024-03-28T19:23:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.095+0000 7f4e4e000640 1 -- 172.21.15.27:0/1115991292 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44035040 con 0x7f4e5814be30 2024-03-28T19:23:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.095+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e581eec70 con 0x7f4e5814be30 2024-03-28T19:23:22.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.095+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e58157910 con 0x7f4e5814be30 2024-03-28T19:23:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.097+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e58069640 con 0x7f4e5814be30 2024-03-28T19:23:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.098+0000 7f4e4e000640 1 -- 172.21.15.27:0/1115991292 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4e44038020 con 0x7f4e5814be30 2024-03-28T19:23:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.098+0000 7f4e4e000640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e3806bbd0 0x7f4e3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.099+0000 7f4e4e000640 1 -- 172.21.15.27:0/1115991292 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f4e440ad3e0 con 0x7f4e5814be30 2024-03-28T19:23:22.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.099+0000 7f4e55e00640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e3806bbd0 0x7f4e3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.100+0000 7f4e55e00640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e3806bbd0 0x7f4e3806e090 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009b20 tx=0x7f4e500039d0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:22.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.108+0000 7f4e4e000640 1 -- 172.21.15.27:0/1115991292 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4e4407aea0 con 0x7f4e5814be30 2024-03-28T19:23:22.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.168+0000 7f06bda00640 1 Processor -- start 2024-03-28T19:23:22.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.168+0000 7f06bda00640 1 -- start start 2024-03-28T19:23:22.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.169+0000 7f06bda00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.169+0000 7f06bda00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f06b80c5e40 0x7f06b8159150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.169+0000 7f06bda00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b80c7910 con 0x7f06b8158580 2024-03-28T19:23:22.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.169+0000 7f06bda00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b80c76e0 con 0x7f06b80c5e40 2024-03-28T19:23:22.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.169+0000 7f06bda00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80c6dd0 con 0x7f06b80be630 2024-03-28T19:23:22.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b6a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f06b8158580 0x7f06b81560c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39826/0 (socket says 172.21.15.27:39826) 2024-03-28T19:23:22.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b6a00640 1 -- 172.21.15.27:0/795887542 learned_addr learned my addr 172.21.15.27:0/795887542 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:22.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b6000640 1 --2- 172.21.15.27:0/795887542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b7400640 1 --2- 172.21.15.27:0/795887542 >> v2:172.21.15.155:3300/0 conn(0x7f06b80c5e40 0x7f06b8159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b5600640 1 -- 172.21.15.27:0/795887542 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622071855 0 0) 0x7f06b80c7910 con 0x7f06b8158580 2024-03-28T19:23:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b5600640 1 -- 172.21.15.27:0/795887542 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c003520 con 0x7f06b8158580 2024-03-28T19:23:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b6000640 1 -- 172.21.15.27:0/795887542 >> v2:172.21.15.155:3300/0 conn(0x7f06b80c5e40 msgr2=0x7f06b8159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b6000640 1 --2- 172.21.15.27:0/795887542 >> v2:172.21.15.155:3300/0 conn(0x7f06b80c5e40 0x7f06b8159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b6000640 1 -- 172.21.15.27:0/795887542 >> v1:172.21.15.27:6789/0 conn(0x7f06b8158580 legacy=0x7f06b81560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.170+0000 7f06b6000640 1 -- 172.21.15.27:0/795887542 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80c7bf0 con 0x7f06b80be630 2024-03-28T19:23:22.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.172+0000 7f06b6000640 1 --2- 172.21.15.27:0/795887542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c5900 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06a40097a0 tx=0x7f06a402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13140e5ec72719af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:22.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.172+0000 7f06b5600640 1 -- 172.21.15.27:0/795887542 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4035020 con 0x7f06b80be630 2024-03-28T19:23:22.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.172+0000 7f06b5600640 1 -- 172.21.15.27:0/795887542 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a4031830 con 0x7f06b80be630 2024-03-28T19:23:22.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.172+0000 7f06b5600640 1 -- 172.21.15.27:0/795887542 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4031be0 con 0x7f06b80be630 2024-03-28T19:23:22.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.173+0000 7f06bda00640 1 -- 172.21.15.27:0/795887542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 msgr2=0x7f06b80c5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.173+0000 7f06bda00640 1 --2- 172.21.15.27:0/795887542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c5900 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06a40097a0 tx=0x7f06a402b8d0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.173+0000 7f06bda00640 1 -- 172.21.15.27:0/795887542 shutdown_connections 2024-03-28T19:23:22.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.173+0000 7f06bda00640 1 --2- 172.21.15.27:0/795887542 >> v2:172.21.15.155:3300/0 conn(0x7f06b80c5e40 0x7f06b8159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.173+0000 7f06bda00640 1 --2- 172.21.15.27:0/795887542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c5900 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.173+0000 7f06bda00640 1 -- 172.21.15.27:0/795887542 >> 172.21.15.27:0/795887542 conn(0x7f06b80caf80 msgr2=0x7f06b80cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:22.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.173+0000 7f06bda00640 1 -- 172.21.15.27:0/795887542 shutdown_connections 2024-03-28T19:23:22.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.173+0000 7f06bda00640 1 -- 172.21.15.27:0/795887542 wait complete. 2024-03-28T19:23:22.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.174+0000 7f06bda00640 1 Processor -- start 2024-03-28T19:23:22.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.175+0000 7f06bda00640 1 -- start start 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06bda00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06bda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b80c5e40 0x7f06b814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06bda00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b8158580 0x7f06b80c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06bda00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80c81b0 con 0x7f06b80c5e40 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06bda00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b8158cc0 con 0x7f06b8158580 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06bda00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b8158960 con 0x7f06b80be630 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b80c5e40 0x7f06b814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b80c5e40 0x7f06b814d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50390/0 (socket says 172.21.15.27:50390) 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6000640 1 -- 172.21.15.27:0/2775116887 learned_addr learned my addr 172.21.15.27:0/2775116887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:22.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6a00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6000640 1 -- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 msgr2=0x7f06b80c2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6000640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6000640 1 -- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b8158580 msgr2=0x7f06b80c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b7400640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b8158580 0x7f06b80c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6000640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b8158580 0x7f06b80c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6000640 1 -- 172.21.15.27:0/2775116887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80af690 con 0x7f06b80c5e40 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b6a00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.176+0000 7f06b7400640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b8158580 0x7f06b80c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.177+0000 7f06b6000640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b80c5e40 0x7f06b814d920 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f06a4000c00 tx=0x7f06a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.177+0000 7f06aea00640 1 -- 172.21.15.27:0/2775116887 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4048070 con 0x7f06b80c5e40 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.177+0000 7f06aea00640 1 -- 172.21.15.27:0/2775116887 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a40042d0 con 0x7f06b80c5e40 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.177+0000 7f06aea00640 1 -- 172.21.15.27:0/2775116887 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4003610 con 0x7f06b80c5e40 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.177+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b80afb70 con 0x7f06b80c5e40 2024-03-28T19:23:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.177+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b81eb9d0 con 0x7f06b80c5e40 2024-03-28T19:23:22.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.178+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b81eb800 con 0x7f06b80c5e40 2024-03-28T19:23:22.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.179+0000 7f06aea00640 1 -- 172.21.15.27:0/2775116887 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f06a4038020 con 0x7f06b80c5e40 2024-03-28T19:23:22.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.180+0000 7f06aea00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f069806bbd0 0x7f069806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:22.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.180+0000 7f06aea00640 1 -- 172.21.15.27:0/2775116887 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f06a40acfc0 con 0x7f06b80c5e40 2024-03-28T19:23:22.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.181+0000 7f06b6a00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f069806bbd0 0x7f069806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.181+0000 7f06b6a00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f069806bbd0 0x7f069806e090 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f06a8007fd0 tx=0x7f06a8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.185+0000 7f06aea00640 1 -- 172.21.15.27:0/2775116887 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f06a407aa80 con 0x7f06b80c5e40 2024-03-28T19:23:22.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.369+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e5814a7a0 con 0x7f4e5814be30 2024-03-28T19:23:22.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.370+0000 7f4e4e000640 1 -- 172.21.15.27:0/1115991292 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5473 (secure 0 0 0) 0x7f4e4407a7a0 con 0x7f4e5814be30 2024-03-28T19:23:22.369 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:22.369 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:22.370 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 209 2024-03-28T19:23:22.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.373+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e3806bbd0 msgr2=0x7f4e3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.373+0000 7f4e57200640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e3806bbd0 0x7f4e3806e090 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009b20 tx=0x7f4e500039d0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.373+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5814be30 msgr2=0x7f4e58154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.373+0000 7f4e57200640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5814be30 0x7f4e58154610 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f4e4402b930 tx=0x7f4e440046c0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.373+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 shutdown_connections 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.373+0000 7f4e57200640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4e3806bbd0 0x7f4e3806e090 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.374+0000 7f4e57200640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e58155df0 0x7f4e581481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.374+0000 7f4e57200640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e5814be30 0x7f4e58154610 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.374+0000 7f4e57200640 1 --2- 172.21.15.27:0/1115991292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4e5806ffb0 0x7f4e58158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.374+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 >> 172.21.15.27:0/1115991292 conn(0x7f4e58072bd0 msgr2=0x7f4e58149c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.374+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 shutdown_connections 2024-03-28T19:23:22.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.374+0000 7f4e57200640 1 -- 172.21.15.27:0/1115991292 wait complete. 2024-03-28T19:23:22.401 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:23:22.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.453+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06b804f400 con 0x7f06b80c5e40 2024-03-28T19:23:22.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.454+0000 7f06aea00640 1 -- 172.21.15.27:0/2775116887 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5473 (secure 0 0 0) 0x7f06a407a380 con 0x7f06b80c5e40 2024-03-28T19:23:22.454 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:22.454 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:22.454 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 209 2024-03-28T19:23:22.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f069806bbd0 msgr2=0x7f069806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f069806bbd0 0x7f069806e090 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f06a8007fd0 tx=0x7f06a8009040 comp rx=0 tx=0).stop 2024-03-28T19:23:22.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b80c5e40 msgr2=0x7f06b814d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:22.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b80c5e40 0x7f06b814d920 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f06a4000c00 tx=0x7f06a40046c0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 shutdown_connections 2024-03-28T19:23:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f069806bbd0 0x7f069806e090 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b8158580 0x7f06b80c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b80c5e40 0x7f06b814d920 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 --2- 172.21.15.27:0/2775116887 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b80be630 0x7f06b80c2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 >> 172.21.15.27:0/2775116887 conn(0x7f06b80caf80 msgr2=0x7f06b8152d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 shutdown_connections 2024-03-28T19:23:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:22.457+0000 7f06bda00640 1 -- 172.21.15.27:0/2775116887 wait complete. 2024-03-28T19:23:22.476 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=8052 2024-03-28T19:23:22.476 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.c 2024-03-28T19:23:24.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:24.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.632+0000 7fb519600640 1 Processor -- start 2024-03-28T19:23:24.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.633+0000 7fb519600640 1 -- start start 2024-03-28T19:23:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.634+0000 7fb519600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb514150c70 0x7fb5140a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.634+0000 7fb519600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5140c3690 0x7fb5140c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.634+0000 7fb519600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5140c3af0 con 0x7fb5140c6660 2024-03-28T19:23:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.634+0000 7fb519600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5140ab4e0 con 0x7fb5140c3690 2024-03-28T19:23:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.634+0000 7fb519600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5140ac1f0 con 0x7fb514150c70 2024-03-28T19:23:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.634+0000 7fb512000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5140c3690 0x7fb5140c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.634+0000 7fb512000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5140c3690 0x7fb5140c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50392/0 (socket says 172.21.15.27:50392) 2024-03-28T19:23:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.634+0000 7fb512000640 1 -- 172.21.15.27:0/2755231228 learned_addr learned my addr 172.21.15.27:0/2755231228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.635+0000 7fb512a00640 1 --2- 172.21.15.27:0/2755231228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb514150c70 0x7fb5140a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.635+0000 7fb512000640 1 -- 172.21.15.27:0/2755231228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb514150c70 msgr2=0x7fb5140a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.635+0000 7fb512000640 1 --2- 172.21.15.27:0/2755231228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb514150c70 0x7fb5140a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.635+0000 7fb512000640 1 -- 172.21.15.27:0/2755231228 >> v1:172.21.15.155:6789/0 conn(0x7fb5140c6660 legacy=0x7fb5141534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.635+0000 7fb512000640 1 -- 172.21.15.27:0/2755231228 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5140c3d90 con 0x7fb5140c3690 2024-03-28T19:23:24.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.636+0000 7fb512000640 1 --2- 172.21.15.27:0/2755231228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5140c3690 0x7fb5140c6120 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009870 tx=0x7fb4fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e4e5f80bd9aef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:24.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb511600640 1 -- 172.21.15.27:0/2755231228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc035020 con 0x7fb5140c3690 2024-03-28T19:23:24.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb511600640 1 -- 172.21.15.27:0/2755231228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc031640 con 0x7fb5140c3690 2024-03-28T19:23:24.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb511600640 1 -- 172.21.15.27:0/2755231228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc02d040 con 0x7fb5140c3690 2024-03-28T19:23:24.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb519600640 1 -- 172.21.15.27:0/2755231228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5140c3690 msgr2=0x7fb5140c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:24.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb519600640 1 --2- 172.21.15.27:0/2755231228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5140c3690 0x7fb5140c6120 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009870 tx=0x7fb4fc02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb519600640 1 -- 172.21.15.27:0/2755231228 shutdown_connections 2024-03-28T19:23:24.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb519600640 1 --2- 172.21.15.27:0/2755231228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5140c3690 0x7fb5140c6120 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb519600640 1 --2- 172.21.15.27:0/2755231228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb514150c70 0x7fb5140a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.637+0000 7fb519600640 1 -- 172.21.15.27:0/2755231228 >> 172.21.15.27:0/2755231228 conn(0x7fb51406b9d0 msgr2=0x7fb5140c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:24.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.638+0000 7fb519600640 1 -- 172.21.15.27:0/2755231228 shutdown_connections 2024-03-28T19:23:24.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.638+0000 7fb519600640 1 -- 172.21.15.27:0/2755231228 wait complete. 2024-03-28T19:23:24.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.640+0000 7fb519600640 1 Processor -- start 2024-03-28T19:23:24.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.640+0000 7fb519600640 1 -- start start 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb519600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5140c3690 0x7fb51414d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb519600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb5140c6660 0x7fb514155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb519600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb514150c70 0x7fb5141562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb519600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5140abcd0 con 0x7fb514150c70 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb519600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5140ab9f0 con 0x7fb5140c3690 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb519600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5140c1810 con 0x7fb5140c6660 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb513400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb514150c70 0x7fb5141562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb512a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5140c3690 0x7fb51414d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb513400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb514150c70 0x7fb5141562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50408/0 (socket says 172.21.15.27:50408) 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb512a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5140c3690 0x7fb51414d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:38358/0 (socket says 172.21.15.27:38358) 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb513400640 1 -- 172.21.15.27:0/999696165 learned_addr learned my addr 172.21.15.27:0/999696165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.641+0000 7fb512000640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb5140c6660 0x7fb514155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:24.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb513400640 1 -- 172.21.15.27:0/999696165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb5140c6660 msgr2=0x7fb514155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:24.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb513400640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb5140c6660 0x7fb514155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb513400640 1 -- 172.21.15.27:0/999696165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5140c3690 msgr2=0x7fb51414d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:24.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb513400640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5140c3690 0x7fb51414d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb513400640 1 -- 172.21.15.27:0/999696165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5140c06b0 con 0x7fb514150c70 2024-03-28T19:23:24.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb513400640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb514150c70 0x7fb5141562c0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fb50400c400 tx=0x7fb50400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:24.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb50aa00640 1 -- 172.21.15.27:0/999696165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504017070 con 0x7fb514150c70 2024-03-28T19:23:24.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb50aa00640 1 -- 172.21.15.27:0/999696165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5040052d0 con 0x7fb514150c70 2024-03-28T19:23:24.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb50aa00640 1 -- 172.21.15.27:0/999696165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504003810 con 0x7fb514150c70 2024-03-28T19:23:24.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.642+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb514155880 con 0x7fb514150c70 2024-03-28T19:23:24.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.643+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb514151c60 con 0x7fb514150c70 2024-03-28T19:23:24.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.644+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb514155a50 con 0x7fb514150c70 2024-03-28T19:23:24.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.645+0000 7fb50aa00640 1 -- 172.21.15.27:0/999696165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb50401a070 con 0x7fb514150c70 2024-03-28T19:23:24.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.646+0000 7fb50aa00640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb4f006bbd0 0x7fb4f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:24.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.646+0000 7fb50aa00640 1 -- 172.21.15.27:0/999696165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7fb50408ad80 con 0x7fb514150c70 2024-03-28T19:23:24.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.646+0000 7fb512a00640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb4f006bbd0 0x7fb4f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:24.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.647+0000 7fb512a00640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb4f006bbd0 0x7fb4f006e090 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb50c003aa0 tx=0x7fb50c009830 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:24.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.650+0000 7fb50aa00640 1 -- 172.21.15.27:0/999696165 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb5040581f0 con 0x7fb514150c70 2024-03-28T19:23:24.693 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 20% (46/225) Updating files: 21% (48/225) Updating files: 22% (50/225) Updating files: 23% (52/225) Updating files: 24% (54/225) Updating files: 25% (57/225) Updating files: 26% (59/225) Updating files: 27% (61/225) Updating files: 28% (63/225) Updating files: 29% (66/225) Updating files: 30% (68/225) Updating files: 31% (70/225) Updating files: 32% (72/225) Updating files: 33% (75/225) Updating files: 34% (77/225) Updating files: 34% (78/225) Updating files: 35% (79/225) Updating files: 36% (81/225) Updating files: 37% (84/225) Updating files: 38% (86/225) Updating files: 39% (88/225) Updating files: 40% (90/225) Updating files: 41% (93/225) Updating files: 41% (94/225) Updating files: 42% (95/225) Updating files: 43% (97/225) Updating files: 43% (98/225) Updating files: 44% (99/225) Updating files: 45% (102/225) Updating files: 46% (104/225) Updating files: 47% (106/225) Updating files: 48% (108/225) Updating files: 49% (111/225) Updating files: 50% (113/225) Updating files: 51% (115/225) Updating files: 52% (117/225) Updating files: 53% (120/225) Updating files: 54% (122/225) Updating files: 55% (124/225) Updating files: 56% (126/225) Updating files: 57% (129/225) Updating files: 58% (131/225) Updating files: 59% (133/225) Updating files: 60% (135/225) Updating files: 61% (138/225) Updating files: 62% (140/225) Updating files: 63% (142/225) Updating files: 64% (144/225) Updating files: 65% (147/225) Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 77% (174/225) Updating files: 77% (175/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2024-03-28T19:23:24.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.899+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb51404f400 con 0x7fb514150c70 2024-03-28T19:23:24.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.900+0000 7fb50aa00640 1 -- 172.21.15.27:0/999696165 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5473 (secure 0 0 0) 0x7fb50405e020 con 0x7fb514150c70 2024-03-28T19:23:24.900 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:24.900 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:24.900 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 209 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.903+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb4f006bbd0 msgr2=0x7fb4f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.903+0000 7fb519600640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb4f006bbd0 0x7fb4f006e090 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb50c003aa0 tx=0x7fb50c009830 comp rx=0 tx=0).stop 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.903+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb514150c70 msgr2=0x7fb5141562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.903+0000 7fb519600640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb514150c70 0x7fb5141562c0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fb50400c400 tx=0x7fb50400c8c0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.903+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 shutdown_connections 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.903+0000 7fb519600640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb4f006bbd0 0x7fb4f006e090 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.904+0000 7fb519600640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb514150c70 0x7fb5141562c0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.904+0000 7fb519600640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb5140c6660 0x7fb514155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.904+0000 7fb519600640 1 --2- 172.21.15.27:0/999696165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5140c3690 0x7fb51414d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.904+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 >> 172.21.15.27:0/999696165 conn(0x7fb51406b9d0 msgr2=0x7fb51414ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.904+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 shutdown_connections 2024-03-28T19:23:24.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:24.904+0000 7fb519600640 1 -- 172.21.15.27:0/999696165 wait complete. 2024-03-28T19:23:24.923 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:23:25.024 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2024-03-28T19:23:25.049 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.8 2024-03-28T19:23:25.106 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:23:25.106 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 9 -eq 48 ']' 2024-03-28T19:23:25.106 INFO:tasks.workunit.client.1.smithi155.stdout:v0.9 2024-03-28T19:23:25.106 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.9 2024-03-28T19:23:25.106 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.9 2024-03-28T19:23:25.106 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.9 2024-03-28T19:23:25.905 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:23:25.906+0000 7f76eba00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:23:26.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.157+0000 7f29fc000640 1 Processor -- start 2024-03-28T19:23:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.158+0000 7f29fc000640 1 -- start start 2024-03-28T19:23:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.159+0000 7f29fc000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29f4150c70 0x7f29f40a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.159+0000 7f29fc000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f41534c0 0x7f29f4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.159+0000 7f29fc000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f40c3af0 con 0x7f29f40c3690 2024-03-28T19:23:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.159+0000 7f29fc000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f40ab4e0 con 0x7f29f4150c70 2024-03-28T19:23:27.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.159+0000 7f29fc000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40ac1f0 con 0x7f29f41534c0 2024-03-28T19:23:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.160+0000 7f29f9600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29f4150c70 0x7f29f40a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.160+0000 7f29fa000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f41534c0 0x7f29f4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.160+0000 7f29f9600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29f4150c70 0x7f29f40a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57760/0 (socket says 172.21.15.27:57760) 2024-03-28T19:23:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.160+0000 7f29f9600640 1 -- 172.21.15.27:0/4071237980 learned_addr learned my addr 172.21.15.27:0/4071237980 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.160+0000 7f29f9600640 1 -- 172.21.15.27:0/4071237980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f41534c0 msgr2=0x7f29f4153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.160+0000 7f29f9600640 1 --2- 172.21.15.27:0/4071237980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f41534c0 0x7f29f4153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.160+0000 7f29f9600640 1 -- 172.21.15.27:0/4071237980 >> v1:172.21.15.155:6789/0 conn(0x7f29f40c3690 legacy=0x7f29f40c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.160+0000 7f29f9600640 1 -- 172.21.15.27:0/4071237980 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f40c3d90 con 0x7f29f4150c70 2024-03-28T19:23:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.161+0000 7f29f9600640 1 --2- 172.21.15.27:0/4071237980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f4150c70 0x7f29f40a9cb0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009870 tx=0x7f29e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9b366ae7b7e07e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:27.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29f3e00640 1 -- 172.21.15.27:0/4071237980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4035020 con 0x7f29f4150c70 2024-03-28T19:23:27.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29f3e00640 1 -- 172.21.15.27:0/4071237980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e4031640 con 0x7f29f4150c70 2024-03-28T19:23:27.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29f3e00640 1 -- 172.21.15.27:0/4071237980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e402d040 con 0x7f29f4150c70 2024-03-28T19:23:27.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29fc000640 1 -- 172.21.15.27:0/4071237980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f4150c70 msgr2=0x7f29f40a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29fc000640 1 --2- 172.21.15.27:0/4071237980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f4150c70 0x7f29f40a9cb0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009870 tx=0x7f29e402b8f0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29fc000640 1 -- 172.21.15.27:0/4071237980 shutdown_connections 2024-03-28T19:23:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29fc000640 1 --2- 172.21.15.27:0/4071237980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f41534c0 0x7f29f4153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29fc000640 1 --2- 172.21.15.27:0/4071237980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f4150c70 0x7f29f40a9cb0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.162+0000 7f29fc000640 1 -- 172.21.15.27:0/4071237980 >> 172.21.15.27:0/4071237980 conn(0x7f29f406b9d0 msgr2=0x7f29f40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.163+0000 7f29fc000640 1 -- 172.21.15.27:0/4071237980 shutdown_connections 2024-03-28T19:23:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.163+0000 7f29fc000640 1 -- 172.21.15.27:0/4071237980 wait complete. 2024-03-28T19:23:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.164+0000 7f29fc000640 1 Processor -- start 2024-03-28T19:23:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.164+0000 7f29fc000640 1 -- start start 2024-03-28T19:23:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.165+0000 7f29fc000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29f40c3690 0x7f29f40a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.165+0000 7f29fc000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f4150c70 0x7f29f40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.165+0000 7f29fc000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f41534c0 0x7f29f4156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.165+0000 7f29fc000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40abcd0 con 0x7f29f41534c0 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.165+0000 7f29fc000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40ab9f0 con 0x7f29f40c3690 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.165+0000 7f29fc000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40c1810 con 0x7f29f4150c70 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f41534c0 0x7f29f4156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f41534c0 0x7f29f4156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57764/0 (socket says 172.21.15.27:57764) 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 -- 172.21.15.27:0/2789361576 learned_addr learned my addr 172.21.15.27:0/2789361576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29f9600640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29f40c3690 0x7f29f40a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29f8c00640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f4150c70 0x7f29f40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 -- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f4150c70 msgr2=0x7f29f40710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f4150c70 0x7f29f40710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 -- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29f40c3690 msgr2=0x7f29f40a8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29f40c3690 0x7f29f40a8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 -- 172.21.15.27:0/2789361576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f40a9620 con 0x7f29f41534c0 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29f8c00640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f4150c70 0x7f29f40710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.166+0000 7f29fa000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f41534c0 0x7f29f4156350 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009d30 tx=0x7f29ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.167+0000 7f29f1600640 1 -- 172.21.15.27:0/2789361576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec018020 con 0x7f29f41534c0 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.167+0000 7f29f1600640 1 -- 172.21.15.27:0/2789361576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec014480 con 0x7f29f41534c0 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.167+0000 7f29f1600640 1 -- 172.21.15.27:0/2789361576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec014870 con 0x7f29f41534c0 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.167+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f4149f00 con 0x7f29f41534c0 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.167+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f41e9eb0 con 0x7f29f41534c0 2024-03-28T19:23:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.168+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f404f400 con 0x7f29f41534c0 2024-03-28T19:23:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.169+0000 7f29f1600640 1 -- 172.21.15.27:0/2789361576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f29ec005d10 con 0x7f29f41534c0 2024-03-28T19:23:27.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.170+0000 7f29f1600640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29d406bc00 0x7f29d406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:27.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.170+0000 7f29f1600640 1 -- 172.21.15.27:0/2789361576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f29ec08c510 con 0x7f29f41534c0 2024-03-28T19:23:27.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.170+0000 7f29f9600640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29d406bc00 0x7f29d406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.171+0000 7f29f9600640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29d406bc00 0x7f29d406e0c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f29e4036000 tx=0x7f29e4008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:27.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.175+0000 7f29f1600640 1 -- 172.21.15.27:0/2789361576 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f29ec022070 con 0x7f29f41534c0 2024-03-28T19:23:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.418+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29f40c0d70 con 0x7f29f41534c0 2024-03-28T19:23:27.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.419+0000 7f29f1600640 1 -- 172.21.15.27:0/2789361576 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5473 (secure 0 0 0) 0x7f29f40a9620 con 0x7f29f41534c0 2024-03-28T19:23:27.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:27.419 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:27.419 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 209 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29d406bc00 msgr2=0x7f29d406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29d406bc00 0x7f29d406e0c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f29e4036000 tx=0x7f29e4008030 comp rx=0 tx=0).stop 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f41534c0 msgr2=0x7f29f4156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f41534c0 0x7f29f4156350 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009d30 tx=0x7f29ec00c490 comp rx=0 tx=0).stop 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 shutdown_connections 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f29d406bc00 0x7f29d406e0c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f41534c0 0x7f29f4156350 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29f4150c70 0x7f29f40710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 --2- 172.21.15.27:0/2789361576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29f40c3690 0x7f29f40a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.422+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 >> 172.21.15.27:0/2789361576 conn(0x7f29f406b9d0 msgr2=0x7f29f40c2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.423+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 shutdown_connections 2024-03-28T19:23:27.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:27.423+0000 7f29fc000640 1 -- 172.21.15.27:0/2789361576 wait complete. 2024-03-28T19:23:27.443 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:23:29.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.686+0000 7f508e400640 1 Processor -- start 2024-03-28T19:23:29.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.686+0000 7f508e400640 1 -- start start 2024-03-28T19:23:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f508e400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50880aada0 0x7f50880c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f508e400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50880ab2b0 0x7f50880c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f508e400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5088156870 con 0x7f50881551a0 2024-03-28T19:23:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f508e400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5088155cf0 con 0x7f50880ab2b0 2024-03-28T19:23:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f508e400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5088155fd0 con 0x7f50880aada0 2024-03-28T19:23:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f508d000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f50881551a0 0x7f50880c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41836/0 (socket says 172.21.15.27:41836) 2024-03-28T19:23:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f508d000640 1 -- 172.21.15.27:0/373363922 learned_addr learned my addr 172.21.15.27:0/373363922 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f508da00640 1 --2- 172.21.15.27:0/373363922 >> v2:172.21.15.27:3300/0 conn(0x7f50880ab2b0 0x7f50880c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.687+0000 7f5087e00640 1 --2- 172.21.15.27:0/373363922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50880aada0 0x7f50880c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.688+0000 7f508da00640 1 -- 172.21.15.27:0/373363922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50880aada0 msgr2=0x7f50880c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.688+0000 7f5087400640 1 -- 172.21.15.27:0/373363922 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107374832 0 0) 0x7f5088156870 con 0x7f50881551a0 2024-03-28T19:23:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.688+0000 7f508da00640 1 --2- 172.21.15.27:0/373363922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50880aada0 0x7f50880c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.688+0000 7f508da00640 1 -- 172.21.15.27:0/373363922 >> v1:172.21.15.27:6789/0 conn(0x7f50881551a0 legacy=0x7f50880c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:29.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.688+0000 7f508da00640 1 -- 172.21.15.27:0/373363922 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50881562b0 con 0x7f50880ab2b0 2024-03-28T19:23:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.689+0000 7f508da00640 1 --2- 172.21.15.27:0/373363922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880ab2b0 0x7f50880c1fc0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f5080009d90 tx=0x7f508002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d356e9aa90b5340e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.689+0000 7f5087400640 1 -- 172.21.15.27:0/373363922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080039070 con 0x7f50880ab2b0 2024-03-28T19:23:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.689+0000 7f5087400640 1 -- 172.21.15.27:0/373363922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508002cb30 con 0x7f50880ab2b0 2024-03-28T19:23:29.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.689+0000 7f5087400640 1 -- 172.21.15.27:0/373363922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080034440 con 0x7f50880ab2b0 2024-03-28T19:23:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.690+0000 7f508e400640 1 -- 172.21.15.27:0/373363922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880ab2b0 msgr2=0x7f50880c1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.690+0000 7f508e400640 1 --2- 172.21.15.27:0/373363922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880ab2b0 0x7f50880c1fc0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f5080009d90 tx=0x7f508002bd80 comp rx=0 tx=0).stop 2024-03-28T19:23:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.690+0000 7f508e400640 1 -- 172.21.15.27:0/373363922 shutdown_connections 2024-03-28T19:23:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.690+0000 7f508e400640 1 --2- 172.21.15.27:0/373363922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880ab2b0 0x7f50880c1fc0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.690+0000 7f508e400640 1 --2- 172.21.15.27:0/373363922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50880aada0 0x7f50880c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.690+0000 7f508e400640 1 -- 172.21.15.27:0/373363922 >> 172.21.15.27:0/373363922 conn(0x7f508806b0c0 msgr2=0x7f508806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.690+0000 7f508e400640 1 -- 172.21.15.27:0/373363922 shutdown_connections 2024-03-28T19:23:29.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.690+0000 7f508e400640 1 -- 172.21.15.27:0/373363922 wait complete. 2024-03-28T19:23:29.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.692+0000 7f508e400640 1 Processor -- start 2024-03-28T19:23:29.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.692+0000 7f508e400640 1 -- start start 2024-03-28T19:23:29.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880aada0 0x7f50880c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:29.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508e400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50880ab2b0 0x7f508814a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508e400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50881551a0 0x7f508814a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508e400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50880c5750 con 0x7f50880aada0 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508e400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50881559a0 con 0x7f50880ab2b0 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508e400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50880c58d0 con 0x7f50881551a0 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880aada0 0x7f50880c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880aada0 0x7f50880c36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57780/0 (socket says 172.21.15.27:57780) 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508d000640 1 -- 172.21.15.27:0/1623373228 learned_addr learned my addr 172.21.15.27:0/1623373228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508da00640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50881551a0 0x7f508814a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f5087e00640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50880ab2b0 0x7f508814a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.693+0000 7f508d000640 1 -- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50881551a0 msgr2=0x7f508814a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f508d000640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50881551a0 0x7f508814a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f508d000640 1 -- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50880ab2b0 msgr2=0x7f508814a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f508d000640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50880ab2b0 0x7f508814a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f508d000640 1 -- 172.21.15.27:0/1623373228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5088072c30 con 0x7f50880aada0 2024-03-28T19:23:29.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f5087e00640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50880ab2b0 0x7f508814a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:23:29.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f508d000640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880aada0 0x7f50880c36f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f507c00c430 tx=0x7f507c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:29.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f5084c00640 1 -- 172.21.15.27:0/1623373228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c0074e0 con 0x7f50880aada0 2024-03-28T19:23:29.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f5084c00640 1 -- 172.21.15.27:0/1623373228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507c0047e0 con 0x7f50880aada0 2024-03-28T19:23:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f5084c00640 1 -- 172.21.15.27:0/1623373228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c004bd0 con 0x7f50880aada0 2024-03-28T19:23:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50881eeca0 con 0x7f50880aada0 2024-03-28T19:23:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.694+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5088149c60 con 0x7f50880aada0 2024-03-28T19:23:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.695+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5088067dd0 con 0x7f50880aada0 2024-03-28T19:23:29.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.697+0000 7f5084c00640 1 -- 172.21.15.27:0/1623373228 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f507c011070 con 0x7f50880aada0 2024-03-28T19:23:29.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.697+0000 7f5084c00640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f506806bbd0 0x7f506806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:29.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.698+0000 7f5084c00640 1 -- 172.21.15.27:0/1623373228 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f507c08b760 con 0x7f50880aada0 2024-03-28T19:23:29.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.698+0000 7f5087e00640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f506806bbd0 0x7f506806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:29.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.699+0000 7f5087e00640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f506806bbd0 0x7f506806e090 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f508815eeb0 tx=0x7f5074008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:29.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.702+0000 7f5084c00640 1 -- 172.21.15.27:0/1623373228 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5088072c30 con 0x7f50880aada0 2024-03-28T19:23:29.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.957+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5088069640 con 0x7f50880aada0 2024-03-28T19:23:29.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.959+0000 7f5084c00640 1 -- 172.21.15.27:0/1623373228 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5473 (secure 0 0 0) 0x7f507c059220 con 0x7f50880aada0 2024-03-28T19:23:29.958 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:29.958 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:29.958 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 209 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.961+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f506806bbd0 msgr2=0x7f506806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.961+0000 7f508e400640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f506806bbd0 0x7f506806e090 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f508815eeb0 tx=0x7f5074008040 comp rx=0 tx=0).stop 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.961+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880aada0 msgr2=0x7f50880c36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.961+0000 7f508e400640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880aada0 0x7f50880c36f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f507c00c430 tx=0x7f507c00c8f0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.961+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 shutdown_connections 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.961+0000 7f508e400640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f506806bbd0 0x7f506806e090 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.961+0000 7f508e400640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50881551a0 0x7f508814a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.962+0000 7f508e400640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50880ab2b0 0x7f508814a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.962+0000 7f508e400640 1 --2- 172.21.15.27:0/1623373228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50880aada0 0x7f50880c36f0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.962+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 >> 172.21.15.27:0/1623373228 conn(0x7f508806b0c0 msgr2=0x7f50880c5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.962+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 shutdown_connections 2024-03-28T19:23:29.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:29.962+0000 7f508e400640 1 -- 172.21.15.27:0/1623373228 wait complete. 2024-03-28T19:23:29.980 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:23:30.477 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-03-28T19:23:30.477 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:23:30.477 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:23:30.486 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:23:30.486 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:30.675 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:23:30.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.711+0000 7fb9ef600640 1 Processor -- start 2024-03-28T19:23:30.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.711+0000 7fb9ef600640 1 -- start start 2024-03-28T19:23:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ef600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9e8152f80 0x7fb9e80c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ef600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb9e80c7970 0x7fb9e80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ef600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e80c7310 con 0x7fb9e8155730 2024-03-28T19:23:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ef600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e8158d40 con 0x7fb9e8152f80 2024-03-28T19:23:30.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ef600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e8159020 con 0x7fb9e80c7970 2024-03-28T19:23:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ecc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9e8152f80 0x7fb9e80c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ed600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb9e8155730 0x7fb9e8155ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41850/0 (socket says 172.21.15.27:41850) 2024-03-28T19:23:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ecc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb9e8152f80 0x7fb9e80c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57788/0 (socket says 172.21.15.27:57788) 2024-03-28T19:23:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9ed600640 1 -- 172.21.15.27:0/3178276502 learned_addr learned my addr 172.21.15.27:0/3178276502 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:30.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.712+0000 7fb9e7e00640 1 --2- 172.21.15.27:0/3178276502 >> v2:172.21.15.155:3300/0 conn(0x7fb9e80c7970 0x7fb9e80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.713+0000 7fb9ecc00640 1 -- 172.21.15.27:0/3178276502 >> v2:172.21.15.155:3300/0 conn(0x7fb9e80c7970 msgr2=0x7fb9e80c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.713+0000 7fb9ecc00640 1 --2- 172.21.15.27:0/3178276502 >> v2:172.21.15.155:3300/0 conn(0x7fb9e80c7970 0x7fb9e80c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.713+0000 7fb9ecc00640 1 -- 172.21.15.27:0/3178276502 >> v1:172.21.15.27:6789/0 conn(0x7fb9e8155730 legacy=0x7fb9e8155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.713+0000 7fb9ecc00640 1 -- 172.21.15.27:0/3178276502 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8159300 con 0x7fb9e8152f80 2024-03-28T19:23:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.713+0000 7fb9e7e00640 1 --2- 172.21.15.27:0/3178276502 >> v2:172.21.15.155:3300/0 conn(0x7fb9e80c7970 0x7fb9e80c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.714+0000 7fb9ecc00640 1 --2- 172.21.15.27:0/3178276502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e8152f80 0x7fb9e80c58e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80c8480 tx=0x7fb9dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0eef31aab930eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.714+0000 7fb9e7400640 1 -- 172.21.15.27:0/3178276502 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0315a0 con 0x7fb9e8152f80 2024-03-28T19:23:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9e7400640 1 -- 172.21.15.27:0/3178276502 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc037070 con 0x7fb9e8152f80 2024-03-28T19:23:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9e7400640 1 -- 172.21.15.27:0/3178276502 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc031960 con 0x7fb9e8152f80 2024-03-28T19:23:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9ef600640 1 -- 172.21.15.27:0/3178276502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e8152f80 msgr2=0x7fb9e80c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9ef600640 1 --2- 172.21.15.27:0/3178276502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e8152f80 0x7fb9e80c58e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80c8480 tx=0x7fb9dc02b840 comp rx=0 tx=0).stop 2024-03-28T19:23:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9ef600640 1 -- 172.21.15.27:0/3178276502 shutdown_connections 2024-03-28T19:23:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9ef600640 1 --2- 172.21.15.27:0/3178276502 >> v2:172.21.15.155:3300/0 conn(0x7fb9e80c7970 0x7fb9e80c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:30.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9ef600640 1 --2- 172.21.15.27:0/3178276502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e8152f80 0x7fb9e80c58e0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9ef600640 1 -- 172.21.15.27:0/3178276502 >> 172.21.15.27:0/3178276502 conn(0x7fb9e806b6f0 msgr2=0x7fb9e80caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9ef600640 1 -- 172.21.15.27:0/3178276502 shutdown_connections 2024-03-28T19:23:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.715+0000 7fb9ef600640 1 -- 172.21.15.27:0/3178276502 wait complete. 2024-03-28T19:23:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.717+0000 7fb9ef600640 1 Processor -- start 2024-03-28T19:23:30.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.717+0000 7fb9ef600640 1 -- start start 2024-03-28T19:23:30.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.718+0000 7fb9ef600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e80c7970 0x7fb9e814a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:30.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.718+0000 7fb9ef600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e8152f80 0x7fb9e80c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.718+0000 7fb9ef600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9e8155730 0x7fb9e80ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.718+0000 7fb9ef600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e8159b00 con 0x7fb9e80c7970 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.718+0000 7fb9ef600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e80c5e20 con 0x7fb9e8152f80 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.718+0000 7fb9ef600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e80cbca0 con 0x7fb9e8155730 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e80c7970 0x7fb9e814a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e80c7970 0x7fb9e814a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57804/0 (socket says 172.21.15.27:57804) 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 -- 172.21.15.27:0/1589435671 learned_addr learned my addr 172.21.15.27:0/1589435671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ed600640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9e8155730 0x7fb9e80ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:30.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9e7e00640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e8152f80 0x7fb9e80c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 -- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9e8155730 msgr2=0x7fb9e80ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9e8155730 0x7fb9e80ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 -- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e8152f80 msgr2=0x7fb9e80c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e8152f80 0x7fb9e80c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 -- 172.21.15.27:0/1589435671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e80cb2d0 con 0x7fb9e80c7970 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9e7e00640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e8152f80 0x7fb9e80c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.719+0000 7fb9ecc00640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e80c7970 0x7fb9e814a7d0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc000c00 tx=0x7fb9dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.720+0000 7fb9e4c00640 1 -- 172.21.15.27:0/1589435671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc032840 con 0x7fb9e80c7970 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.720+0000 7fb9e4c00640 1 -- 172.21.15.27:0/1589435671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc032a00 con 0x7fb9e80c7970 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.720+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e80cb540 con 0x7fb9e80c7970 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.720+0000 7fb9e4c00640 1 -- 172.21.15.27:0/1589435671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc032df0 con 0x7fb9e80c7970 2024-03-28T19:23:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.720+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e814ad10 con 0x7fb9e80c7970 2024-03-28T19:23:30.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.721+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e804f400 con 0x7fb9e80c7970 2024-03-28T19:23:30.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.722+0000 7fb9e4c00640 1 -- 172.21.15.27:0/1589435671 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb9dc04d3f0 con 0x7fb9e80c7970 2024-03-28T19:23:30.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.723+0000 7fb9e4c00640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb9c806bbd0 0x7fb9c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:30.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.723+0000 7fb9e4c00640 1 -- 172.21.15.27:0/1589435671 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7fb9dc02f030 con 0x7fb9e80c7970 2024-03-28T19:23:30.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.723+0000 7fb9e7e00640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb9c806bbd0 0x7fb9c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:30.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.724+0000 7fb9e7e00640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb9c806bbd0 0x7fb9c806e090 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb9e81556d0 tx=0x7fb9d4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:30.729+0000 7fb9e4c00640 1 -- 172.21.15.27:0/1589435671 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb9dc040070 con 0x7fb9e80c7970 2024-03-28T19:23:31.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.012+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9e80cba80 con 0x7fb9e80c7970 2024-03-28T19:23:31.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.013+0000 7fb9e4c00640 1 -- 172.21.15.27:0/1589435671 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5473 (secure 0 0 0) 0x7fb9dc07b1d0 con 0x7fb9e80c7970 2024-03-28T19:23:31.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:31.013 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:31.013 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 209 2024-03-28T19:23:31.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb9c806bbd0 msgr2=0x7fb9c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:31.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb9c806bbd0 0x7fb9c806e090 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb9e81556d0 tx=0x7fb9d4008040 comp rx=0 tx=0).stop 2024-03-28T19:23:31.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e80c7970 msgr2=0x7fb9e814a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e80c7970 0x7fb9e814a7d0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc000c00 tx=0x7fb9dc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:23:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 shutdown_connections 2024-03-28T19:23:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb9c806bbd0 0x7fb9c806e090 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb9e8155730 0x7fb9e80ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e8152f80 0x7fb9e80c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:31.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 --2- 172.21.15.27:0/1589435671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb9e80c7970 0x7fb9e814a7d0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 >> 172.21.15.27:0/1589435671 conn(0x7fb9e806b6f0 msgr2=0x7fb9e814f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 shutdown_connections 2024-03-28T19:23:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:31.017+0000 7fb9ef600640 1 -- 172.21.15.27:0/1589435671 wait complete. 2024-03-28T19:23:31.037 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:23:31.981 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:32.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.222+0000 7f60ffa00640 1 Processor -- start 2024-03-28T19:23:32.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.222+0000 7f60ffa00640 1 -- start start 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.223+0000 7f60ffa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f8150b60 0x7f60f80ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.223+0000 7f60ffa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60f80c8c70 0x7f60f80c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.223+0000 7f60ffa00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f80ad680 con 0x7f60f80c8470 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.223+0000 7f60ffa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f8157210 con 0x7f60f80c8c70 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.223+0000 7f60ffa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80ad9b0 con 0x7f60f8150b60 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.224+0000 7f60fda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60f80c8c70 0x7f60f80c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.224+0000 7f60fda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60f80c8c70 0x7f60f80c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57808/0 (socket says 172.21.15.27:57808) 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.224+0000 7f60fda00640 1 -- 172.21.15.27:0/4225479835 learned_addr learned my addr 172.21.15.27:0/4225479835 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.224+0000 7f60fd000640 1 --2- 172.21.15.27:0/4225479835 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f8150b60 0x7f60f80ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:32.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.224+0000 7f60fda00640 1 -- 172.21.15.27:0/4225479835 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f8150b60 msgr2=0x7f60f80ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:32.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.224+0000 7f60fda00640 1 --2- 172.21.15.27:0/4225479835 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f8150b60 0x7f60f80ade40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.224+0000 7f60fda00640 1 -- 172.21.15.27:0/4225479835 >> v1:172.21.15.155:6789/0 conn(0x7f60f80c8470 legacy=0x7f60f8156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.224+0000 7f60fda00640 1 -- 172.21.15.27:0/4225479835 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f80c4de0 con 0x7f60f80c8c70 2024-03-28T19:23:32.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.226+0000 7f60fda00640 1 --2- 172.21.15.27:0/4225479835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f80c8c70 0x7f60f80c9100 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009e50 tx=0x7f60f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0730d8a61339250 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.226+0000 7f60f7400640 1 -- 172.21.15.27:0/4225479835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f002ca70 con 0x7f60f80c8c70 2024-03-28T19:23:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.226+0000 7f60f7400640 1 -- 172.21.15.27:0/4225479835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f002cc10 con 0x7f60f80c8c70 2024-03-28T19:23:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.226+0000 7f60f7400640 1 -- 172.21.15.27:0/4225479835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f00343f0 con 0x7f60f80c8c70 2024-03-28T19:23:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.227+0000 7f60ffa00640 1 -- 172.21.15.27:0/4225479835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f80c8c70 msgr2=0x7f60f80c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:32.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.227+0000 7f60ffa00640 1 --2- 172.21.15.27:0/4225479835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f80c8c70 0x7f60f80c9100 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009e50 tx=0x7f60f002bed0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.227+0000 7f60ffa00640 1 -- 172.21.15.27:0/4225479835 shutdown_connections 2024-03-28T19:23:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.227+0000 7f60ffa00640 1 --2- 172.21.15.27:0/4225479835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f80c8c70 0x7f60f80c9100 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.227+0000 7f60ffa00640 1 --2- 172.21.15.27:0/4225479835 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f8150b60 0x7f60f80ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.227+0000 7f60ffa00640 1 -- 172.21.15.27:0/4225479835 >> 172.21.15.27:0/4225479835 conn(0x7f60f80aaab0 msgr2=0x7f60f80aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.227+0000 7f60ffa00640 1 -- 172.21.15.27:0/4225479835 shutdown_connections 2024-03-28T19:23:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.227+0000 7f60ffa00640 1 -- 172.21.15.27:0/4225479835 wait complete. 2024-03-28T19:23:32.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.229+0000 7f60ffa00640 1 Processor -- start 2024-03-28T19:23:32.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.229+0000 7f60ffa00640 1 -- start start 2024-03-28T19:23:32.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60ffa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60f80c8470 0x7f60f8156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:32.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60ffa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f80c8c70 0x7f60f81eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60ffa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f8150b60 0x7f60f8152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60ffa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f8150f40 con 0x7f60f8150b60 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60ffa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80c53a0 con 0x7f60f80c8470 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60ffa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80a85a0 con 0x7f60f80c8c70 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60fda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f8150b60 0x7f60f8152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60fda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f8150b60 0x7f60f8152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57814/0 (socket says 172.21.15.27:57814) 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60fda00640 1 -- 172.21.15.27:0/1379930461 learned_addr learned my addr 172.21.15.27:0/1379930461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60fd000640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60f80c8470 0x7f60f8156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.230+0000 7f60f7e00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f80c8c70 0x7f60f81eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60fda00640 1 -- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f80c8c70 msgr2=0x7f60f81eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60fda00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f80c8c70 0x7f60f81eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60fda00640 1 -- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60f80c8470 msgr2=0x7f60f8156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60fda00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60f80c8470 0x7f60f8156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60fda00640 1 -- 172.21.15.27:0/1379930461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f8152150 con 0x7f60f8150b60 2024-03-28T19:23:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60fd000640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60f80c8470 0x7f60f8156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60fda00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f8150b60 0x7f60f8152bf0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f60f0036040 tx=0x7f60f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60f4c00640 1 -- 172.21.15.27:0/1379930461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0043020 con 0x7f60f8150b60 2024-03-28T19:23:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60f4c00640 1 -- 172.21.15.27:0/1379930461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f003e450 con 0x7f60f8150b60 2024-03-28T19:23:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.231+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f81ef410 con 0x7f60f8150b60 2024-03-28T19:23:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.232+0000 7f60f4c00640 1 -- 172.21.15.27:0/1379930461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0009510 con 0x7f60f8150b60 2024-03-28T19:23:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.232+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f81526d0 con 0x7f60f8150b60 2024-03-28T19:23:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.233+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f80c3360 con 0x7f60f8150b60 2024-03-28T19:23:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.234+0000 7f60f4c00640 1 -- 172.21.15.27:0/1379930461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f60f0046020 con 0x7f60f8150b60 2024-03-28T19:23:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.235+0000 7f60f4c00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60d806bbd0 0x7f60d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.235+0000 7f60f4c00640 1 -- 172.21.15.27:0/1379930461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f60f00ae2b0 con 0x7f60f8150b60 2024-03-28T19:23:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.235+0000 7f60fd000640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60d806bbd0 0x7f60d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:32.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.236+0000 7f60fd000640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60d806bbd0 0x7f60d806e090 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f60ec006fd0 tx=0x7f60ec008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:32.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.240+0000 7f60f4c00640 1 -- 172.21.15.27:0/1379930461 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f60f80c3360 con 0x7f60f8150b60 2024-03-28T19:23:32.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.502+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60f804f400 con 0x7f60f8150b60 2024-03-28T19:23:32.502 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:32.503 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:32.503 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 210 2024-03-28T19:23:32.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.503+0000 7f60f4c00640 1 -- 172.21.15.27:0/1379930461 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f60f004d030 con 0x7f60f8150b60 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.508+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60d806bbd0 msgr2=0x7f60d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.508+0000 7f60ffa00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60d806bbd0 0x7f60d806e090 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f60ec006fd0 tx=0x7f60ec008040 comp rx=0 tx=0).stop 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.508+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f8150b60 msgr2=0x7f60f8152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.508+0000 7f60ffa00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f8150b60 0x7f60f8152bf0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f60f0036040 tx=0x7f60f0004a40 comp rx=0 tx=0).stop 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.510+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 shutdown_connections 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.510+0000 7f60ffa00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f60d806bbd0 0x7f60d806e090 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.510+0000 7f60ffa00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60f8150b60 0x7f60f8152bf0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.510+0000 7f60ffa00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60f80c8c70 0x7f60f81eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.510+0000 7f60ffa00640 1 --2- 172.21.15.27:0/1379930461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60f80c8470 0x7f60f8156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.510+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 >> 172.21.15.27:0/1379930461 conn(0x7f60f80aaab0 msgr2=0x7f60f80aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.510+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 shutdown_connections 2024-03-28T19:23:32.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:32.510+0000 7f60ffa00640 1 -- 172.21.15.27:0/1379930461 wait complete. 2024-03-28T19:23:32.686 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:23:33.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.281+0000 7fe3f3e00640 1 Processor -- start 2024-03-28T19:23:33.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.281+0000 7fe3f3e00640 1 -- start start 2024-03-28T19:23:33.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.282+0000 7fe3f3e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe3f40c8ed0 0x7fe3f40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.283+0000 7fe3f3e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4158bd0 con 0x7fe3f40be5d0 2024-03-28T19:23:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.283+0000 7fe3f3e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4158eb0 con 0x7fe3f40c58c0 2024-03-28T19:23:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.283+0000 7fe3f3e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f4159190 con 0x7fe3f40c8ed0 2024-03-28T19:23:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.283+0000 7fe3f3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3f40be5d0 0x7fe3f4155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41858/0 (socket says 172.21.15.27:41858) 2024-03-28T19:23:33.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.283+0000 7fe3f3400640 1 -- 172.21.15.27:0/863143484 learned_addr learned my addr 172.21.15.27:0/863143484 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.283+0000 7fe3f2000640 1 --2- 172.21.15.27:0/863143484 >> v2:172.21.15.155:3300/0 conn(0x7fe3f40c8ed0 0x7fe3f40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.283+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746786740 0 0) 0x7fe3f4158bd0 con 0x7fe3f40be5d0 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.283+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d8003520 con 0x7fe3f40be5d0 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 194077745 0 0) 0x7fe3d8003520 con 0x7fe3f40be5d0 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 >> v2:172.21.15.155:3300/0 conn(0x7fe3f40c8ed0 msgr2=0x7fe3f40c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 --2- 172.21.15.27:0/863143484 >> v2:172.21.15.155:3300/0 conn(0x7fe3f40c8ed0 0x7fe3f40c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 >> v1:172.21.15.155:6789/0 conn(0x7fe3f40c58c0 legacy=0x7fe3f41555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f4159470 con 0x7fe3f40be5d0 2024-03-28T19:23:33.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fe3ec0019f0 con 0x7fe3f40be5d0 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 >> v1:172.21.15.27:6789/0 conn(0x7fe3f40be5d0 legacy=0x7fe3f4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3d80039b0 0x7fe3d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3d8004800 0x7fe3d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d80052e0 0x7fe3d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f4159190 con 0x7fe3d8004800 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d800c1b0 con 0x7fe3d80052e0 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d800c3b0 con 0x7fe3d80039b0 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.284+0000 7fe3f2000640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3d8004800 0x7fe3d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f2a00640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3d80039b0 0x7fe3d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f3400640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d80052e0 0x7fe3d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f2000640 1 -- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3d80039b0 msgr2=0x7fe3d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f2000640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3d80039b0 0x7fe3d8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f2000640 1 -- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d80052e0 msgr2=0x7fe3d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:33.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f2000640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d80052e0 0x7fe3d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f2000640 1 -- 172.21.15.27:0/863143484 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8003520 con 0x7fe3d8004800 2024-03-28T19:23:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f3400640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d80052e0 0x7fe3d800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:23:33.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.285+0000 7fe3f3400640 1 -- 172.21.15.27:0/863143484 reap_dead start 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.286+0000 7fe3f2000640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3d8004800 0x7fe3d8004c80 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40c9300 tx=0x7fe3d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcae776bd19e018 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.286+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0039070 con 0x7fe3d8004800 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.286+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e002c660 con 0x7fe3d8004800 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.286+0000 7fe3f1600640 1 -- 172.21.15.27:0/863143484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0009690 con 0x7fe3d8004800 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.287+0000 7fe3f3e00640 1 -- 172.21.15.27:0/863143484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3d8004800 msgr2=0x7fe3d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.287+0000 7fe3f3e00640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3d8004800 0x7fe3d8004c80 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40c9300 tx=0x7fe3d8005830 comp rx=0 tx=0).stop 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.287+0000 7fe3f3e00640 1 -- 172.21.15.27:0/863143484 shutdown_connections 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.287+0000 7fe3f3e00640 1 --2- 172.21.15.27:0/863143484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3d8004800 0x7fe3d8004c80 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.287+0000 7fe3f3e00640 1 -- 172.21.15.27:0/863143484 >> 172.21.15.27:0/863143484 conn(0x7fe3f406b6f0 msgr2=0x7fe3f40acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:33.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.287+0000 7fe3f3e00640 1 -- 172.21.15.27:0/863143484 shutdown_connections 2024-03-28T19:23:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.287+0000 7fe3f3e00640 1 -- 172.21.15.27:0/863143484 wait complete. 2024-03-28T19:23:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.289+0000 7fe3f3e00640 1 Processor -- start 2024-03-28T19:23:33.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.289+0000 7fe3f3e00640 1 -- start start 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.290+0000 7fe3f3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d8004800 0x7fe3f4157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.290+0000 7fe3f3e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f40c4480 0x7fe3f41ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.290+0000 7fe3f3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3f41ee950 0x7fe3f4151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.290+0000 7fe3f3e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f40b09a0 con 0x7fe3f41ee950 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.290+0000 7fe3f3e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f40b0820 con 0x7fe3d8004800 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.290+0000 7fe3f3e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f40ad7d0 con 0x7fe3f40c4480 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.290+0000 7fe3f3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3f41ee950 0x7fe3f4151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.290+0000 7fe3f3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3f41ee950 0x7fe3f4151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57830/0 (socket says 172.21.15.27:57830) 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f3400640 1 -- 172.21.15.27:0/3281173789 learned_addr learned my addr 172.21.15.27:0/3281173789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:33.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f2a00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d8004800 0x7fe3f4157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f2000640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f40c4480 0x7fe3f41ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f3400640 1 -- 172.21.15.27:0/3281173789 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f40c4480 msgr2=0x7fe3f41ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f3400640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f40c4480 0x7fe3f41ee410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f3400640 1 -- 172.21.15.27:0/3281173789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d8004800 msgr2=0x7fe3f4157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f3400640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d8004800 0x7fe3f4157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f3400640 1 -- 172.21.15.27:0/3281173789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f40c21b0 con 0x7fe3f41ee950 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.291+0000 7fe3f3400640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3f41ee950 0x7fe3f4151550 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4062b00 tx=0x7fe3ec00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.292+0000 7fe3eaa00640 1 -- 172.21.15.27:0/3281173789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec00eb30 con 0x7fe3f41ee950 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.292+0000 7fe3eaa00640 1 -- 172.21.15.27:0/3281173789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ec018070 con 0x7fe3f41ee950 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.292+0000 7fe3eaa00640 1 -- 172.21.15.27:0/3281173789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec013420 con 0x7fe3f41ee950 2024-03-28T19:23:33.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.292+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f41e26d0 con 0x7fe3f41ee950 2024-03-28T19:23:33.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.292+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f4158300 con 0x7fe3f41ee950 2024-03-28T19:23:33.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.293+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f4069640 con 0x7fe3f41ee950 2024-03-28T19:23:33.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.294+0000 7fe3eaa00640 1 -- 172.21.15.27:0/3281173789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe3ec0135c0 con 0x7fe3f41ee950 2024-03-28T19:23:33.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.295+0000 7fe3eaa00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3d406bc00 0x7fe3d406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.296+0000 7fe3eaa00640 1 -- 172.21.15.27:0/3281173789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7fe3ec08bc70 con 0x7fe3f41ee950 2024-03-28T19:23:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.296+0000 7fe3f2a00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3d406bc00 0x7fe3d406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:33.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.297+0000 7fe3f2a00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3d406bc00 0x7fe3d406e0c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4007fd0 tx=0x7fe3e4009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:33.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.301+0000 7fe3eaa00640 1 -- 172.21.15.27:0/3281173789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe3ec059730 con 0x7fe3f41ee950 2024-03-28T19:23:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.576+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3f4156580 con 0x7fe3f41ee950 2024-03-28T19:23:33.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.577+0000 7fe3eaa00640 1 -- 172.21.15.27:0/3281173789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+6265 (secure 0 0 0) 0x7fe3ec059030 con 0x7fe3f41ee950 2024-03-28T19:23:33.576 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:33.576 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:33.576 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 210 2024-03-28T19:23:33.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.579+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3d406bc00 msgr2=0x7fe3d406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.579+0000 7fe3f3e00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3d406bc00 0x7fe3d406e0c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4007fd0 tx=0x7fe3e4009040 comp rx=0 tx=0).stop 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.579+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3f41ee950 msgr2=0x7fe3f4151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.579+0000 7fe3f3e00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3f41ee950 0x7fe3f4151550 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4062b00 tx=0x7fe3ec00e530 comp rx=0 tx=0).stop 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.580+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 shutdown_connections 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.580+0000 7fe3f3e00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe3d406bc00 0x7fe3d406e0c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.580+0000 7fe3f3e00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3f41ee950 0x7fe3f4151550 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.580+0000 7fe3f3e00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f40c4480 0x7fe3f41ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.580+0000 7fe3f3e00640 1 --2- 172.21.15.27:0/3281173789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3d8004800 0x7fe3f4157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.580+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 >> 172.21.15.27:0/3281173789 conn(0x7fe3f406b6f0 msgr2=0x7fe3f40c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.580+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 shutdown_connections 2024-03-28T19:23:33.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:33.580+0000 7fe3f3e00640 1 -- 172.21.15.27:0/3281173789 wait complete. 2024-03-28T19:23:33.599 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-03-28T19:23:33.600 INFO:tasks.mds_thrash.fs.[b]:waiting for 67 secs before thrashing 2024-03-28T19:23:34.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:34.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.920+0000 7f396e000640 1 Processor -- start 2024-03-28T19:23:34.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.921+0000 7f396e000640 1 -- start start 2024-03-28T19:23:34.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.922+0000 7f396e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39680c3e40 0x7f3968158d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:34.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.922+0000 7f396e000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3968159300 con 0x7f3968156180 2024-03-28T19:23:34.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.922+0000 7f396e000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39681595e0 con 0x7f39680c5f20 2024-03-28T19:23:34.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.922+0000 7f396e000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39681598c0 con 0x7f39680c3e40 2024-03-28T19:23:34.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.922+0000 7f396cc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3968156180 0x7f39680c2b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41870/0 (socket says 172.21.15.27:41870) 2024-03-28T19:23:34.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.922+0000 7f396cc00640 1 -- 172.21.15.27:0/3879618524 learned_addr learned my addr 172.21.15.27:0/3879618524 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f395fe00640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39680c3e40 0x7f3968158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966668985 0 0) 0x7f3968159300 con 0x7f3968156180 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c003520 con 0x7f3968156180 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867596981 0 0) 0x7f394c003520 con 0x7f3968156180 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39680c3e40 msgr2=0x7f3968158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39680c3e40 0x7f3968158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 >> v1:172.21.15.155:6789/0 conn(0x7f39680c5f20 legacy=0x7f39680c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968159ba0 con 0x7f3968156180 2024-03-28T19:23:34.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f39580019f0 con 0x7f3968156180 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 >> v1:172.21.15.27:6789/0 conn(0x7f3968156180 legacy=0x7f39680c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f394c0039e0 0x7f394c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.923+0000 7f3967e00640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39680c3e40 0x7f394c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.924+0000 7f3967e00640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f394c004d40 0x7f394c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.924+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39681598c0 con 0x7f394c004d40 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.924+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c00bba0 con 0x7f394c0039e0 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.924+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c00bda0 con 0x7f39680c3e40 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.924+0000 7f396d600640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f394c004d40 0x7f394c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.924+0000 7f396cc00640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f394c0039e0 0x7f394c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.924+0000 7f395fe00640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39680c3e40 0x7f394c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.925+0000 7f396d600640 1 -- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39680c3e40 msgr2=0x7f394c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.925+0000 7f396d600640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39680c3e40 0x7f394c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.925+0000 7f396d600640 1 -- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f394c0039e0 msgr2=0x7f394c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:34.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.925+0000 7f396d600640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f394c0039e0 0x7f394c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:34.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.925+0000 7f396d600640 1 -- 172.21.15.27:0/3879618524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c003520 con 0x7f394c004d40 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.926+0000 7f396d600640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f394c004d40 0x7f394c0051c0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f3960001750 tx=0x7f394c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77dcdebcac0c829 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.926+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960001370 con 0x7f394c004d40 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.926+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39600052d0 con 0x7f394c004d40 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.926+0000 7f3967e00640 1 -- 172.21.15.27:0/3879618524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960003830 con 0x7f394c004d40 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.927+0000 7f396e000640 1 -- 172.21.15.27:0/3879618524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f394c004d40 msgr2=0x7f394c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.927+0000 7f396e000640 1 --2- 172.21.15.27:0/3879618524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f394c004d40 0x7f394c0051c0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f3960001750 tx=0x7f394c005830 comp rx=0 tx=0).stop 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.927+0000 7f396d600640 1 -- 172.21.15.27:0/3879618524 reap_dead start 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.927+0000 7f396e000640 1 -- 172.21.15.27:0/3879618524 shutdown_connections 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.927+0000 7f396e000640 1 -- 172.21.15.27:0/3879618524 >> 172.21.15.27:0/3879618524 conn(0x7f39680c4500 msgr2=0x7f39680c48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.927+0000 7f396e000640 1 -- 172.21.15.27:0/3879618524 shutdown_connections 2024-03-28T19:23:34.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.927+0000 7f396e000640 1 -- 172.21.15.27:0/3879618524 wait complete. 2024-03-28T19:23:34.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.929+0000 7f396e000640 1 Processor -- start 2024-03-28T19:23:34.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.929+0000 7f396e000640 1 -- start start 2024-03-28T19:23:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f396e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39680bf000 0x7f39681e2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f396e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39681554b0 0x7f39681e2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:34.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f396e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39681df660 0x7f39681573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f396e000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680c3280 con 0x7f39681554b0 2024-03-28T19:23:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f396e000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680c5160 con 0x7f39680bf000 2024-03-28T19:23:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f396e000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39680c3400 con 0x7f39681df660 2024-03-28T19:23:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f395fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39681554b0 0x7f39681e2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f395fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39681554b0 0x7f39681e2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57852/0 (socket says 172.21.15.27:57852) 2024-03-28T19:23:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.930+0000 7f395fe00640 1 -- 172.21.15.27:0/1141606598 learned_addr learned my addr 172.21.15.27:0/1141606598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.931+0000 7f396d600640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39681df660 0x7f39681573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:34.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.931+0000 7f396cc00640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39680bf000 0x7f39681e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.931+0000 7f395fe00640 1 -- 172.21.15.27:0/1141606598 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39681df660 msgr2=0x7f39681573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.931+0000 7f395fe00640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39681df660 0x7f39681573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.931+0000 7f395fe00640 1 -- 172.21.15.27:0/1141606598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39680bf000 msgr2=0x7f39681e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.931+0000 7f395fe00640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39680bf000 0x7f39681e2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:34.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.931+0000 7f395fe00640 1 -- 172.21.15.27:0/1141606598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39681eb820 con 0x7f39681554b0 2024-03-28T19:23:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.931+0000 7f395fe00640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39681554b0 0x7f39681e2840 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f395000f580 tx=0x7f395000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.932+0000 7f3965600640 1 -- 172.21.15.27:0/1141606598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950019020 con 0x7f39681554b0 2024-03-28T19:23:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.932+0000 7f3965600640 1 -- 172.21.15.27:0/1141606598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3950014870 con 0x7f39681554b0 2024-03-28T19:23:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.932+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3968156600 con 0x7f39681554b0 2024-03-28T19:23:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.932+0000 7f3965600640 1 -- 172.21.15.27:0/1141606598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950014c60 con 0x7f39681554b0 2024-03-28T19:23:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.932+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3968156bc0 con 0x7f39681554b0 2024-03-28T19:23:34.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.933+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3968069640 con 0x7f39681554b0 2024-03-28T19:23:34.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.934+0000 7f3965600640 1 -- 172.21.15.27:0/1141606598 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f395001c020 con 0x7f39681554b0 2024-03-28T19:23:34.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.935+0000 7f3965600640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f394406bc00 0x7f394406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:34.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.935+0000 7f3965600640 1 -- 172.21.15.27:0/1141606598 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9819+0+0 (secure 0 0 0) 0x7f395008c170 con 0x7f39681554b0 2024-03-28T19:23:34.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.936+0000 7f396cc00640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f394406bc00 0x7f394406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:34.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.936+0000 7f396cc00640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f394406bc00 0x7f394406e0c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f39580013d0 tx=0x7f3958006040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:34.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:34.940+0000 7f3965600640 1 -- 172.21.15.27:0/1141606598 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3950023020 con 0x7f39681554b0 2024-03-28T19:23:35.050 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 20% (39/193) Updating files: 21% (41/193) Updating files: 22% (43/193) Updating files: 23% (45/193) Updating files: 24% (47/193) Updating files: 25% (49/193) Updating files: 26% (51/193) Updating files: 27% (53/193) Updating files: 28% (55/193) Updating files: 29% (56/193) Updating files: 30% (58/193) Updating files: 31% (60/193) Updating files: 32% (62/193) Updating files: 33% (64/193) Updating files: 34% (66/193) Updating files: 35% (68/193) Updating files: 36% (70/193) Updating files: 37% (72/193) Updating files: 38% (74/193) Updating files: 39% (76/193) Updating files: 40% (78/193) Updating files: 41% (80/193) Updating files: 42% (82/193) Updating files: 43% (83/193) Updating files: 44% (85/193) Updating files: 45% (87/193) Updating files: 46% (89/193) Updating files: 47% (91/193) Updating files: 48% (93/193) Updating files: 49% (95/193) Updating files: 50% (97/193) Updating files: 51% (99/193) Updating files: 52% (101/193) Updating files: 52% (102/193) Updating files: 53% (103/193) Updating files: 54% (105/193) Updating files: 55% (107/193) Updating files: 56% (109/193) Updating files: 57% (111/193) Updating files: 58% (112/193) Updating files: 59% (114/193) Updating files: 60% (116/193) Updating files: 61% (118/193) Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2024-03-28T19:23:35.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.201+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39680c2950 con 0x7f39681554b0 2024-03-28T19:23:35.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.202+0000 7f3965600640 1 -- 172.21.15.27:0/1141606598 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f3950059560 con 0x7f39681554b0 2024-03-28T19:23:35.201 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:35.202 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:22:55.854015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8259":{"gid":8259,"name":"a","rank":0,"incarnation":200,"state":"up:active","state_seq":27,"addr":"172.21.15.27:6835/1299974557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":1299974557},{"type":"v1","addr":"172.21.15.27:6835","nonce":1299974557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8259,"qdb_cluster":[8259]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:35.202 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 210 2024-03-28T19:23:35.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f394406bc00 msgr2=0x7f394406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:35.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f394406bc00 0x7f394406e0c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f39580013d0 tx=0x7f3958006040 comp rx=0 tx=0).stop 2024-03-28T19:23:35.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39681554b0 msgr2=0x7f39681e2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:35.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39681554b0 0x7f39681e2840 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f395000f580 tx=0x7f395000fa40 comp rx=0 tx=0).stop 2024-03-28T19:23:35.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 shutdown_connections 2024-03-28T19:23:35.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f394406bc00 0x7f394406e0c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:35.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39681df660 0x7f39681573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:35.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39681554b0 0x7f39681e2840 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:35.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 --2- 172.21.15.27:0/1141606598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39680bf000 0x7f39681e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:35.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 >> 172.21.15.27:0/1141606598 conn(0x7f39680c4500 msgr2=0x7f396806b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:35.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.205+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 shutdown_connections 2024-03-28T19:23:35.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:35.206+0000 7f396e000640 1 -- 172.21.15.27:0/1141606598 wait complete. 2024-03-28T19:23:35.225 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:23:35.379 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2024-03-28T19:23:35.403 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.9 2024-03-28T19:23:35.486 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:23:35.486 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 10 -eq 48 ']' 2024-03-28T19:23:35.487 INFO:tasks.workunit.client.1.smithi155.stdout:v0.10 2024-03-28T19:23:35.487 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.10 2024-03-28T19:23:35.487 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.10 2024-03-28T19:23:35.487 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.10 2024-03-28T19:23:37.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:37.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.441+0000 7fe3acc00640 1 Processor -- start 2024-03-28T19:23:37.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.442+0000 7fe3acc00640 1 -- start start 2024-03-28T19:23:37.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.443+0000 7fe3acc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3a80be5d0 0x7fe3a80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.443+0000 7fe3acc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a8158bd0 con 0x7fe3a80c8ed0 2024-03-28T19:23:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.443+0000 7fe3acc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a8158eb0 con 0x7fe3a80c58c0 2024-03-28T19:23:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.443+0000 7fe3acc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a8159190 con 0x7fe3a80be5d0 2024-03-28T19:23:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.443+0000 7fe3a7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3a80be5d0 0x7fe3a80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.443+0000 7fe3a7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3a80be5d0 0x7fe3a80c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52408/0 (socket says 172.21.15.27:52408) 2024-03-28T19:23:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.443+0000 7fe3a6a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3a80c8ed0 0x7fe3a8155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50372/0 (socket says 172.21.15.27:50372) 2024-03-28T19:23:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.443+0000 7fe3a7e00640 1 -- 172.21.15.27:0/108005266 learned_addr learned my addr 172.21.15.27:0/108005266 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:37.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.444+0000 7fe3a7e00640 1 -- 172.21.15.27:0/108005266 >> v1:172.21.15.155:6789/0 conn(0x7fe3a80c58c0 legacy=0x7fe3a81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.444+0000 7fe3a7e00640 1 -- 172.21.15.27:0/108005266 >> v1:172.21.15.27:6789/0 conn(0x7fe3a80c8ed0 legacy=0x7fe3a8155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.444+0000 7fe3a7e00640 1 -- 172.21.15.27:0/108005266 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a8159470 con 0x7fe3a80be5d0 2024-03-28T19:23:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.446+0000 7fe3a7e00640 1 --2- 172.21.15.27:0/108005266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a80c5ca0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009e50 tx=0x7fe3a002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258c996f4d28d0e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.446+0000 7fe3a6000640 1 -- 172.21.15.27:0/108005266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a002ca80 con 0x7fe3a80be5d0 2024-03-28T19:23:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.446+0000 7fe3a6000640 1 -- 172.21.15.27:0/108005266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a002cc20 con 0x7fe3a80be5d0 2024-03-28T19:23:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.446+0000 7fe3a6000640 1 -- 172.21.15.27:0/108005266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a00343f0 con 0x7fe3a80be5d0 2024-03-28T19:23:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.447+0000 7fe3acc00640 1 -- 172.21.15.27:0/108005266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 msgr2=0x7fe3a80c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.447+0000 7fe3acc00640 1 --2- 172.21.15.27:0/108005266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a80c5ca0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009e50 tx=0x7fe3a002bee0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.447+0000 7fe3acc00640 1 -- 172.21.15.27:0/108005266 shutdown_connections 2024-03-28T19:23:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.447+0000 7fe3acc00640 1 --2- 172.21.15.27:0/108005266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a80c5ca0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.447+0000 7fe3acc00640 1 -- 172.21.15.27:0/108005266 >> 172.21.15.27:0/108005266 conn(0x7fe3a806b6f0 msgr2=0x7fe3a80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.448+0000 7fe3acc00640 1 -- 172.21.15.27:0/108005266 shutdown_connections 2024-03-28T19:23:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.448+0000 7fe3acc00640 1 -- 172.21.15.27:0/108005266 wait complete. 2024-03-28T19:23:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.450+0000 7fe3acc00640 1 Processor -- start 2024-03-28T19:23:37.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.450+0000 7fe3acc00640 1 -- start start 2024-03-28T19:23:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3acc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a81ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3acc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3a80c58c0 0x7fe3a80c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3acc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3a80c8ed0 0x7fe3a80c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3acc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a80b09a0 con 0x7fe3a80be5d0 2024-03-28T19:23:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3acc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a80b0820 con 0x7fe3a80c8ed0 2024-03-28T19:23:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3acc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a80ad7d0 con 0x7fe3a80c58c0 2024-03-28T19:23:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3a7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a81ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3a7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a81ec230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52414/0 (socket says 172.21.15.27:52414) 2024-03-28T19:23:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.452+0000 7fe3a7400640 1 -- 172.21.15.27:0/2751051834 learned_addr learned my addr 172.21.15.27:0/2751051834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe3a6a00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3a80c58c0 0x7fe3a80c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe3a7e00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3a80c8ed0 0x7fe3a80c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe3a7400640 1 -- 172.21.15.27:0/2751051834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3a80c58c0 msgr2=0x7fe3a80c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe3a7400640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3a80c58c0 0x7fe3a80c2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe3a7400640 1 -- 172.21.15.27:0/2751051834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3a80c8ed0 msgr2=0x7fe3a80c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe3a7400640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3a80c8ed0 0x7fe3a80c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe3a7400640 1 -- 172.21.15.27:0/2751051834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a81538c0 con 0x7fe3a80be5d0 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe3a7400640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a81ec230 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fe39800c400 tx=0x7fe39800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:37.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe39f400640 1 -- 172.21.15.27:0/2751051834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980052d0 con 0x7fe3a80be5d0 2024-03-28T19:23:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.453+0000 7fe39f400640 1 -- 172.21.15.27:0/2751051834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe398005490 con 0x7fe3a80be5d0 2024-03-28T19:23:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.454+0000 7fe39f400640 1 -- 172.21.15.27:0/2751051834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398007260 con 0x7fe3a80be5d0 2024-03-28T19:23:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.454+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a8153ba0 con 0x7fe3a80be5d0 2024-03-28T19:23:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.454+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a8154160 con 0x7fe3a80be5d0 2024-03-28T19:23:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.455+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a80ad200 con 0x7fe3a80be5d0 2024-03-28T19:23:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.456+0000 7fe39f400640 1 -- 172.21.15.27:0/2751051834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe398007400 con 0x7fe3a80be5d0 2024-03-28T19:23:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.457+0000 7fe39f400640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe38806bc00 0x7fe38806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.457+0000 7fe39f400640 1 -- 172.21.15.27:0/2751051834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7fe39808afe0 con 0x7fe3a80be5d0 2024-03-28T19:23:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.457+0000 7fe3a6a00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe38806bc00 0x7fe38806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.458+0000 7fe3a6a00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe38806bc00 0x7fe38806e0c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe3a81e95f0 tx=0x7fe394007c80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.465+0000 7fe39f400640 1 -- 172.21.15.27:0/2751051834 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe3a80ad200 con 0x7fe3a80be5d0 2024-03-28T19:23:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.744+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3a8069640 con 0x7fe3a80be5d0 2024-03-28T19:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.745+0000 7fe39f400640 1 -- 172.21.15.27:0/2751051834 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+5465 (secure 0 0 0) 0x7fe398058a80 con 0x7fe3a80be5d0 2024-03-28T19:23:37.744 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:37.744 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:35.387126+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 211 2024-03-28T19:23:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe38806bc00 msgr2=0x7fe38806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe38806bc00 0x7fe38806e0c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe3a81e95f0 tx=0x7fe394007c80 comp rx=0 tx=0).stop 2024-03-28T19:23:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 msgr2=0x7fe3a81ec230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a81ec230 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fe39800c400 tx=0x7fe39800c8c0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 shutdown_connections 2024-03-28T19:23:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe38806bc00 0x7fe38806e0c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3a80c8ed0 0x7fe3a80c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3a80c58c0 0x7fe3a80c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 --2- 172.21.15.27:0/2751051834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3a80be5d0 0x7fe3a81ec230 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 >> 172.21.15.27:0/2751051834 conn(0x7fe3a806b6f0 msgr2=0x7fe3a8072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 shutdown_connections 2024-03-28T19:23:37.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.748+0000 7fe3acc00640 1 -- 172.21.15.27:0/2751051834 wait complete. 2024-03-28T19:23:37.765 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-03-28T19:23:37.765 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:23:37.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.966+0000 7f98d0c00640 1 Processor -- start 2024-03-28T19:23:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.966+0000 7f98d0c00640 1 -- start start 2024-03-28T19:23:37.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.967+0000 7f98d0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc150c70 0x7f98cc0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.967+0000 7f98d0c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f98cc1534c0 0x7f98cc153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.967+0000 7f98d0c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0c3af0 con 0x7f98cc0c3690 2024-03-28T19:23:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.967+0000 7f98d0c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc0ab4e0 con 0x7f98cc1534c0 2024-03-28T19:23:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.967+0000 7f98d0c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0ac1f0 con 0x7f98cc150c70 2024-03-28T19:23:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.968+0000 7f98c9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f98cc0c3690 0x7f98cc0c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50374/0 (socket says 172.21.15.27:50374) 2024-03-28T19:23:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.968+0000 7f98c9400640 1 -- 172.21.15.27:0/4206706637 learned_addr learned my addr 172.21.15.27:0/4206706637 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.968+0000 7f98c9e00640 1 --2- 172.21.15.27:0/4206706637 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc150c70 0x7f98cc0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.968+0000 7f98ca800640 1 --2- 172.21.15.27:0/4206706637 >> v2:172.21.15.155:3300/0 conn(0x7f98cc1534c0 0x7f98cc153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.969+0000 7f98c8a00640 1 -- 172.21.15.27:0/4206706637 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341481275 0 0) 0x7f98cc0c3af0 con 0x7f98cc0c3690 2024-03-28T19:23:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.969+0000 7f98c8a00640 1 -- 172.21.15.27:0/4206706637 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b0003520 con 0x7f98cc0c3690 2024-03-28T19:23:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.969+0000 7f98ca800640 1 -- 172.21.15.27:0/4206706637 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc150c70 msgr2=0x7f98cc0a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.969+0000 7f98ca800640 1 --2- 172.21.15.27:0/4206706637 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc150c70 0x7f98cc0a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.969+0000 7f98ca800640 1 -- 172.21.15.27:0/4206706637 >> v1:172.21.15.27:6789/0 conn(0x7f98cc0c3690 legacy=0x7f98cc0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.969+0000 7f98ca800640 1 -- 172.21.15.27:0/4206706637 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc0c3d90 con 0x7f98cc1534c0 2024-03-28T19:23:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.971+0000 7f98ca800640 1 --2- 172.21.15.27:0/4206706637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc1534c0 0x7f98cc153950 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009d90 tx=0x7f98c402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b24f87723d5463 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.971+0000 7f98c8a00640 1 -- 172.21.15.27:0/4206706637 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4039070 con 0x7f98cc1534c0 2024-03-28T19:23:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.971+0000 7f98c8a00640 1 -- 172.21.15.27:0/4206706637 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c402cbd0 con 0x7f98cc1534c0 2024-03-28T19:23:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.971+0000 7f98c8a00640 1 -- 172.21.15.27:0/4206706637 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4034440 con 0x7f98cc1534c0 2024-03-28T19:23:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.972+0000 7f98d0c00640 1 -- 172.21.15.27:0/4206706637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc1534c0 msgr2=0x7f98cc153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.972+0000 7f98d0c00640 1 --2- 172.21.15.27:0/4206706637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc1534c0 0x7f98cc153950 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009d90 tx=0x7f98c402be20 comp rx=0 tx=0).stop 2024-03-28T19:23:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.972+0000 7f98d0c00640 1 -- 172.21.15.27:0/4206706637 shutdown_connections 2024-03-28T19:23:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.972+0000 7f98d0c00640 1 --2- 172.21.15.27:0/4206706637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc1534c0 0x7f98cc153950 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.972+0000 7f98d0c00640 1 --2- 172.21.15.27:0/4206706637 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc150c70 0x7f98cc0a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.972+0000 7f98d0c00640 1 -- 172.21.15.27:0/4206706637 >> 172.21.15.27:0/4206706637 conn(0x7f98cc06b9d0 msgr2=0x7f98cc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.972+0000 7f98d0c00640 1 -- 172.21.15.27:0/4206706637 shutdown_connections 2024-03-28T19:23:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.972+0000 7f98d0c00640 1 -- 172.21.15.27:0/4206706637 wait complete. 2024-03-28T19:23:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.975+0000 7f98d0c00640 1 Processor -- start 2024-03-28T19:23:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.975+0000 7f98d0c00640 1 -- start start 2024-03-28T19:23:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98d0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc0c3690 0x7f98cc157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98d0c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc150c70 0x7f98cc14f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98d0c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc1534c0 0x7f98cc1518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98d0c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0abcd0 con 0x7f98cc0c3690 2024-03-28T19:23:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98d0c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0ab9f0 con 0x7f98cc150c70 2024-03-28T19:23:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98d0c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0c1810 con 0x7f98cc1534c0 2024-03-28T19:23:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc0c3690 0x7f98cc157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc0c3690 0x7f98cc157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52430/0 (socket says 172.21.15.27:52430) 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9e00640 1 -- 172.21.15.27:0/2311590352 learned_addr learned my addr 172.21.15.27:0/2311590352 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9400640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc150c70 0x7f98cc14f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98ca800640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc1534c0 0x7f98cc1518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9e00640 1 -- 172.21.15.27:0/2311590352 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc1534c0 msgr2=0x7f98cc1518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9e00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc1534c0 0x7f98cc1518d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9e00640 1 -- 172.21.15.27:0/2311590352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc150c70 msgr2=0x7f98cc14f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9e00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc150c70 0x7f98cc14f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.977+0000 7f98c9e00640 1 -- 172.21.15.27:0/2311590352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc156680 con 0x7f98cc0c3690 2024-03-28T19:23:37.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.978+0000 7f98c9e00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc0c3690 0x7f98cc157390 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00e9e0 tx=0x7f98bc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:37.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.978+0000 7f98c2000640 1 -- 172.21.15.27:0/2311590352 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc017070 con 0x7f98cc0c3690 2024-03-28T19:23:37.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.978+0000 7f98c2000640 1 -- 172.21.15.27:0/2311590352 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc00c9d0 con 0x7f98cc0c3690 2024-03-28T19:23:37.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.978+0000 7f98c2000640 1 -- 172.21.15.27:0/2311590352 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00cdc0 con 0x7f98cc0c3690 2024-03-28T19:23:37.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.978+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc156960 con 0x7f98cc0c3690 2024-03-28T19:23:37.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.978+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc1548d0 con 0x7f98cc0c3690 2024-03-28T19:23:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.980+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc04f400 con 0x7f98cc0c3690 2024-03-28T19:23:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.981+0000 7f98c2000640 1 -- 172.21.15.27:0/2311590352 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f98bc01a070 con 0x7f98cc0c3690 2024-03-28T19:23:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.982+0000 7f98c2000640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f98ac06bc00 0x7f98ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.982+0000 7f98c2000640 1 -- 172.21.15.27:0/2311590352 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f98bc08be50 con 0x7f98cc0c3690 2024-03-28T19:23:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.982+0000 7f98c9400640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f98ac06bc00 0x7f98ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.983+0000 7f98c9400640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f98ac06bc00 0x7f98ac06e0c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f98b8005880 tx=0x7f98b8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:37.986+0000 7f98c2000640 1 -- 172.21.15.27:0/2311590352 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f98bc0598b0 con 0x7f98cc0c3690 2024-03-28T19:23:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.260+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98cc0c5f00 con 0x7f98cc0c3690 2024-03-28T19:23:38.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.260+0000 7f98c2000640 1 -- 172.21.15.27:0/2311590352 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+5465 (secure 0 0 0) 0x7f98bc0591b0 con 0x7f98cc0c3690 2024-03-28T19:23:38.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:38.260 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:35.387126+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:38.260 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 211 2024-03-28T19:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f98ac06bc00 msgr2=0x7f98ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f98ac06bc00 0x7f98ac06e0c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f98b8005880 tx=0x7f98b8009040 comp rx=0 tx=0).stop 2024-03-28T19:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc0c3690 msgr2=0x7f98cc157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc0c3690 0x7f98cc157390 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00e9e0 tx=0x7f98bc00eea0 comp rx=0 tx=0).stop 2024-03-28T19:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 shutdown_connections 2024-03-28T19:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f98ac06bc00 0x7f98ac06e0c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98cc1534c0 0x7f98cc1518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98cc150c70 0x7f98cc14f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 --2- 172.21.15.27:0/2311590352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98cc0c3690 0x7f98cc157390 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 >> 172.21.15.27:0/2311590352 conn(0x7f98cc06b9d0 msgr2=0x7f98cc0c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 shutdown_connections 2024-03-28T19:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:38.263+0000 7f98d0c00640 1 -- 172.21.15.27:0/2311590352 wait complete. 2024-03-28T19:23:38.279 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 211, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 211, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8511": { "addr": "172.21.15.27:6835/3741162434", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 3741162434, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 3741162434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8511, "incarnation": 211, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 90, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:23:35.387126+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8511 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 209, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7537": { "addr": "172.21.15.155:6835/4236980323", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 4236980323, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 4236980323, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7537, "incarnation": 205, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 89, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:23:20.987967+0000", "qdb_cluster": [ 7537 ], "qdb_leader": 7537, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7537 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3334295790", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3334295790, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3334295790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 193, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6467, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:23:40.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:40.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.498+0000 7f603b800640 1 Processor -- start 2024-03-28T19:23:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.499+0000 7f603b800640 1 -- start start 2024-03-28T19:23:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.500+0000 7f603b800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60340c4dc0 0x7f6034070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.500+0000 7f603b800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60340c52e0 con 0x7f6034070a80 2024-03-28T19:23:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.500+0000 7f603b800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60341557e0 con 0x7f6034152bd0 2024-03-28T19:23:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.500+0000 7f603b800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6034155ac0 con 0x7f60340c4dc0 2024-03-28T19:23:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.500+0000 7f6033e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60340c4dc0 0x7f6034070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.500+0000 7f6033e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60340c4dc0 0x7f6034070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52444/0 (socket says 172.21.15.27:52444) 2024-03-28T19:23:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.500+0000 7f6033e00640 1 -- 172.21.15.27:0/1779268391 learned_addr learned my addr 172.21.15.27:0/1779268391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.500+0000 7f6039800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6034070a80 0x7f603415ffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50380/0 (socket says 172.21.15.27:50380) 2024-03-28T19:23:40.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.501+0000 7f6033e00640 1 -- 172.21.15.27:0/1779268391 >> v1:172.21.15.155:6789/0 conn(0x7f6034152bd0 legacy=0x7f603406eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:40.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.501+0000 7f6033e00640 1 -- 172.21.15.27:0/1779268391 >> v1:172.21.15.27:6789/0 conn(0x7f6034070a80 legacy=0x7f603415ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:40.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.501+0000 7f6033e00640 1 -- 172.21.15.27:0/1779268391 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6034155da0 con 0x7f60340c4dc0 2024-03-28T19:23:40.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.502+0000 7f6033e00640 1 --2- 172.21.15.27:0/1779268391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60340c4dc0 0x7f6034070540 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f6020009870 tx=0x7f602002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3167c0e45ff132d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:40.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.502+0000 7f6033400640 1 -- 172.21.15.27:0/1779268391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020034070 con 0x7f60340c4dc0 2024-03-28T19:23:40.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.502+0000 7f6033400640 1 -- 172.21.15.27:0/1779268391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6020030860 con 0x7f60340c4dc0 2024-03-28T19:23:40.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.502+0000 7f6033400640 1 -- 172.21.15.27:0/1779268391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020030c30 con 0x7f60340c4dc0 2024-03-28T19:23:40.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.503+0000 7f603b800640 1 -- 172.21.15.27:0/1779268391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60340c4dc0 msgr2=0x7f6034070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:40.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.503+0000 7f603b800640 1 --2- 172.21.15.27:0/1779268391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60340c4dc0 0x7f6034070540 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f6020009870 tx=0x7f602002b9a0 comp rx=0 tx=0).stop 2024-03-28T19:23:40.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.503+0000 7f603b800640 1 -- 172.21.15.27:0/1779268391 shutdown_connections 2024-03-28T19:23:40.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.503+0000 7f603b800640 1 --2- 172.21.15.27:0/1779268391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60340c4dc0 0x7f6034070540 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:40.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.503+0000 7f603b800640 1 -- 172.21.15.27:0/1779268391 >> 172.21.15.27:0/1779268391 conn(0x7f6034066dd0 msgr2=0x7f60340671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:40.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.503+0000 7f603b800640 1 -- 172.21.15.27:0/1779268391 shutdown_connections 2024-03-28T19:23:40.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.503+0000 7f603b800640 1 -- 172.21.15.27:0/1779268391 wait complete. 2024-03-28T19:23:40.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.505+0000 7f603b800640 1 Processor -- start 2024-03-28T19:23:40.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.505+0000 7f603b800640 1 -- start start 2024-03-28T19:23:40.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f603b800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6034070a80 0x7f60340ad110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:40.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f603b800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60340c4dc0 0x7f6034157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:40.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f603b800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6034152bd0 0x7f60341eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:40.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f603b800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60340c6c70 con 0x7f6034152bd0 2024-03-28T19:23:40.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f603b800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6034152fb0 con 0x7f60340c4dc0 2024-03-28T19:23:40.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f603b800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60340c6df0 con 0x7f6034070a80 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f6039800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6034152bd0 0x7f60341eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f6038e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6034070a80 0x7f60340ad110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f6039800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6034152bd0 0x7f60341eec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52452/0 (socket says 172.21.15.27:52452) 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.506+0000 7f6039800640 1 -- 172.21.15.27:0/1258806372 learned_addr learned my addr 172.21.15.27:0/1258806372 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.507+0000 7f6033e00640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60340c4dc0 0x7f6034157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.507+0000 7f6039800640 1 -- 172.21.15.27:0/1258806372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6034070a80 msgr2=0x7f60340ad110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.507+0000 7f6039800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6034070a80 0x7f60340ad110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.507+0000 7f6039800640 1 -- 172.21.15.27:0/1258806372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60340c4dc0 msgr2=0x7f6034157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.507+0000 7f6039800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60340c4dc0 0x7f6034157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.507+0000 7f6039800640 1 -- 172.21.15.27:0/1258806372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60341df300 con 0x7f6034152bd0 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.507+0000 7f6039800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6034152bd0 0x7f60341eec90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f602c00c760 tx=0x7f602c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.508+0000 7f6030c00640 1 -- 172.21.15.27:0/1258806372 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c017070 con 0x7f6034152bd0 2024-03-28T19:23:40.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.508+0000 7f6030c00640 1 -- 172.21.15.27:0/1258806372 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c00a640 con 0x7f6034152bd0 2024-03-28T19:23:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.508+0000 7f6030c00640 1 -- 172.21.15.27:0/1258806372 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c00a9f0 con 0x7f6034152bd0 2024-03-28T19:23:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.508+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60341ef710 con 0x7f6034152bd0 2024-03-28T19:23:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.508+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60340ac910 con 0x7f6034152bd0 2024-03-28T19:23:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.509+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603404f400 con 0x7f6034152bd0 2024-03-28T19:23:40.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.510+0000 7f6030c00640 1 -- 172.21.15.27:0/1258806372 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f602c01a070 con 0x7f6034152bd0 2024-03-28T19:23:40.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.511+0000 7f6030c00640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f601406bbd0 0x7f601406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:40.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.511+0000 7f6030c00640 1 -- 172.21.15.27:0/1258806372 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f602c08afa0 con 0x7f6034152bd0 2024-03-28T19:23:40.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.512+0000 7f6038e00640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f601406bbd0 0x7f601406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:40.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.513+0000 7f6038e00640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f601406bbd0 0x7f601406e090 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6028004f60 tx=0x7f6028007e30 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:40.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.520+0000 7f6030c00640 1 -- 172.21.15.27:0/1258806372 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f602c0583b0 con 0x7f6034152bd0 2024-03-28T19:23:40.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.850+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60340c2860 con 0x7f6034152bd0 2024-03-28T19:23:40.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.851+0000 7f6030c00640 1 -- 172.21.15.27:0/1258806372 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+5465 (secure 0 0 0) 0x7f602c05c450 con 0x7f6034152bd0 2024-03-28T19:23:40.850 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:40.850 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:35.387126+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:40.850 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 211 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.854+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f601406bbd0 msgr2=0x7f601406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.854+0000 7f603b800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f601406bbd0 0x7f601406e090 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6028004f60 tx=0x7f6028007e30 comp rx=0 tx=0).stop 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6034152bd0 msgr2=0x7f60341eec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6034152bd0 0x7f60341eec90 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f602c00c760 tx=0x7f602c00cc20 comp rx=0 tx=0).stop 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 shutdown_connections 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f601406bbd0 0x7f601406e090 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6034152bd0 0x7f60341eec90 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60340c4dc0 0x7f6034157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 --2- 172.21.15.27:0/1258806372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6034070a80 0x7f60340ad110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:40.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 >> 172.21.15.27:0/1258806372 conn(0x7f6034066dd0 msgr2=0x7f60341531e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:40.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 shutdown_connections 2024-03-28T19:23:40.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:40.855+0000 7f603b800640 1 -- 172.21.15.27:0/1258806372 wait complete. 2024-03-28T19:23:40.880 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:23:42.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:23:42.538 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:23:42.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:43.038 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 23% (39/164) Updating files: 24% (40/164) Updating files: 25% (41/164) Updating files: 26% (43/164) Updating files: 27% (45/164) Updating files: 28% (46/164) Updating files: 29% (48/164) Updating files: 30% (50/164) Updating files: 31% (51/164) Updating files: 32% (53/164) Updating files: 33% (55/164) Updating files: 34% (56/164) Updating files: 35% (58/164) Updating files: 36% (60/164) Updating files: 37% (61/164) Updating files: 38% (63/164) Updating files: 39% (64/164) Updating files: 40% (66/164) Updating files: 41% (68/164) Updating files: 42% (69/164) Updating files: 43% (71/164) Updating files: 44% (73/164) Updating files: 45% (74/164) Updating files: 46% (76/164) Updating files: 47% (78/164) Updating files: 48% (79/164) Updating files: 49% (81/164) Updating files: 50% (82/164) Updating files: 51% (84/164) Updating files: 52% (86/164) Updating files: 53% (87/164) Updating files: 54% (89/164) Updating files: 55% (91/164) Updating files: 56% (92/164) Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 64% (106/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2024-03-28T19:23:43.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.103+0000 7fdcf4200640 1 Processor -- start 2024-03-28T19:23:43.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.104+0000 7fdcf4200640 1 -- start start 2024-03-28T19:23:43.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf4200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcec0c4dc0 0x7fdcec070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:43.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf4200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec070a80 0x7fdcec15ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:43.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf4200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcec0c52e0 con 0x7fdcec152bd0 2024-03-28T19:23:43.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf4200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcec1557e0 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf4200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec155ac0 con 0x7fdcec070a80 2024-03-28T19:23:43.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf0e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcec0c4dc0 0x7fdcec070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:43.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf0e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdcec0c4dc0 0x7fdcec070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52458/0 (socket says 172.21.15.27:52458) 2024-03-28T19:23:43.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf0e00640 1 -- 172.21.15.27:0/4120407188 learned_addr learned my addr 172.21.15.27:0/4120407188 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:43.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.105+0000 7fdcf2200640 1 --2- 172.21.15.27:0/4120407188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec070a80 0x7fdcec15ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:43.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.106+0000 7fdcf0e00640 1 -- 172.21.15.27:0/4120407188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec070a80 msgr2=0x7fdcec15ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:43.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.106+0000 7fdcf0e00640 1 --2- 172.21.15.27:0/4120407188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec070a80 0x7fdcec15ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.106+0000 7fdcf0e00640 1 -- 172.21.15.27:0/4120407188 >> v1:172.21.15.155:6789/0 conn(0x7fdcec152bd0 legacy=0x7fdcec06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:43.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.106+0000 7fdcf0e00640 1 -- 172.21.15.27:0/4120407188 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec155da0 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.107+0000 7fdcf0e00640 1 --2- 172.21.15.27:0/4120407188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec070540 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009870 tx=0x7fdcd802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d980dbe04fc6dffd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:43.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.107+0000 7fdcebe00640 1 -- 172.21.15.27:0/4120407188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8034070 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.107+0000 7fdcebe00640 1 -- 172.21.15.27:0/4120407188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd8030840 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.107+0000 7fdcebe00640 1 -- 172.21.15.27:0/4120407188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8030c10 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.108+0000 7fdcf4200640 1 -- 172.21.15.27:0/4120407188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 msgr2=0x7fdcec070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:43.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.108+0000 7fdcf4200640 1 --2- 172.21.15.27:0/4120407188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec070540 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009870 tx=0x7fdcd802b980 comp rx=0 tx=0).stop 2024-03-28T19:23:43.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.108+0000 7fdcf4200640 1 -- 172.21.15.27:0/4120407188 shutdown_connections 2024-03-28T19:23:43.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.108+0000 7fdcf4200640 1 --2- 172.21.15.27:0/4120407188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec070a80 0x7fdcec15ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.108+0000 7fdcf4200640 1 --2- 172.21.15.27:0/4120407188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec070540 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.108+0000 7fdcf4200640 1 -- 172.21.15.27:0/4120407188 >> 172.21.15.27:0/4120407188 conn(0x7fdcec066dd0 msgr2=0x7fdcec0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:43.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.108+0000 7fdcf4200640 1 -- 172.21.15.27:0/4120407188 shutdown_connections 2024-03-28T19:23:43.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.108+0000 7fdcf4200640 1 -- 172.21.15.27:0/4120407188 wait complete. 2024-03-28T19:23:43.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.110+0000 7fdcf4200640 1 Processor -- start 2024-03-28T19:23:43.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.110+0000 7fdcf4200640 1 -- start start 2024-03-28T19:23:43.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf4200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcec070a80 0x7fdcec1deca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:43.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf4200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec0c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:43.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf4200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec152bd0 0x7fdcec0acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf4200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0c6c70 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf4200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec152fb0 con 0x7fdcec070a80 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf4200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0c6df0 con 0x7fdcec152bd0 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec0c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec0c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52472/0 (socket says 172.21.15.27:52472) 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf0e00640 1 -- 172.21.15.27:0/3428790257 learned_addr learned my addr 172.21.15.27:0/3428790257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf1800640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcec070a80 0x7fdcec1deca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.111+0000 7fdcf2200640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec152bd0 0x7fdcec0acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdcf0e00640 1 -- 172.21.15.27:0/3428790257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec152bd0 msgr2=0x7fdcec0acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdcf0e00640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec152bd0 0x7fdcec0acc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdcf0e00640 1 -- 172.21.15.27:0/3428790257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcec070a80 msgr2=0x7fdcec1deca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdcf0e00640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcec070a80 0x7fdcec1deca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdcf0e00640 1 -- 172.21.15.27:0/3428790257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec0abb80 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdcf0e00640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec0c8370 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8000c00 tx=0x7fdcd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:43.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdce9600640 1 -- 172.21.15.27:0/3428790257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8048070 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdce9600640 1 -- 172.21.15.27:0/3428790257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd8030640 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdce9600640 1 -- 172.21.15.27:0/3428790257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8034040 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.112+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcec0abe60 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.113+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec0ac420 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.114+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec0af0f0 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.115+0000 7fdce9600640 1 -- 172.21.15.27:0/3428790257 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdcd8037070 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.116+0000 7fdce9600640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdccc06bbd0 0x7fdccc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:43.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.116+0000 7fdce9600640 1 -- 172.21.15.27:0/3428790257 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7fdcd80ad730 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.116+0000 7fdcf1800640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdccc06bbd0 0x7fdccc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:43.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.117+0000 7fdcf1800640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdccc06bbd0 0x7fdccc06e090 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fdce0000f60 tx=0x7fdce0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:43.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.120+0000 7fdce9600640 1 -- 172.21.15.27:0/3428790257 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdcec0af0f0 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.322 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 007313fe844 v0.10 2024-03-28T19:23:43.341 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.10 2024-03-28T19:23:43.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.391+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcec04f400 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.392+0000 7fdce9600640 1 -- 172.21.15.27:0/3428790257 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+5465 (secure 0 0 0) 0x7fdcd8030810 con 0x7fdcec0c4dc0 2024-03-28T19:23:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:43.392 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:35.387126+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:43.393 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 211 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdccc06bbd0 msgr2=0x7fdccc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdccc06bbd0 0x7fdccc06e090 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fdce0000f60 tx=0x7fdce0009040 comp rx=0 tx=0).stop 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 msgr2=0x7fdcec0c8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec0c8370 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8000c00 tx=0x7fdcd8004730 comp rx=0 tx=0).stop 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 shutdown_connections 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdccc06bbd0 0x7fdccc06e090 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcec152bd0 0x7fdcec0acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcec0c4dc0 0x7fdcec0c8370 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 --2- 172.21.15.27:0/3428790257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcec070a80 0x7fdcec1deca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 >> 172.21.15.27:0/3428790257 conn(0x7fdcec066dd0 msgr2=0x7fdcec06db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 shutdown_connections 2024-03-28T19:23:43.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:43.395+0000 7fdcf4200640 1 -- 172.21.15.27:0/3428790257 wait complete. 2024-03-28T19:23:43.412 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:23:43.435 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:23:43.435 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 11 -eq 48 ']' 2024-03-28T19:23:43.436 INFO:tasks.workunit.client.1.smithi155.stdout:v0.11 2024-03-28T19:23:43.436 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.11 2024-03-28T19:23:43.436 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.11 2024-03-28T19:23:43.436 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.11 2024-03-28T19:23:45.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:45.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.651+0000 7f9fa1600640 1 Processor -- start 2024-03-28T19:23:45.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.651+0000 7f9fa1600640 1 -- start start 2024-03-28T19:23:45.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9fa1600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c0c1390 0x7f9f9c0c3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:45.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9fa1600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c157530 con 0x7f9f9c157150 2024-03-28T19:23:45.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9fa1600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c06dbd0 con 0x7f9f9c0c36b0 2024-03-28T19:23:45.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9fa1600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c06e1e0 con 0x7f9f9c0c1390 2024-03-28T19:23:45.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9f9be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9f9c157150 0x7f9f9c0c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50388/0 (socket says 172.21.15.27:50388) 2024-03-28T19:23:45.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9f9be00640 1 -- 172.21.15.27:0/1053496294 learned_addr learned my addr 172.21.15.27:0/1053496294 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:45.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711825430 0 0) 0x7f9f9c157530 con 0x7f9f9c157150 2024-03-28T19:23:45.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9f93e00640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c0c1390 0x7f9f9c0c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:45.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.653+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f80003520 con 0x7f9f9c157150 2024-03-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605491551 0 0) 0x7f9f80003520 con 0x7f9f9c157150 2024-03-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c0c1390 msgr2=0x7f9f9c0c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c0c1390 0x7f9f9c0c3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 >> v1:172.21.15.155:6789/0 conn(0x7f9f9c0c36b0 legacy=0x7f9f9c164400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c15de60 con 0x7f9f9c157150 2024-03-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9f8c001670 con 0x7f9f9c157150 2024-03-28T19:23:45.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 >> v1:172.21.15.27:6789/0 conn(0x7f9f9c157150 legacy=0x7f9f9c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c0c1390 0x7f9f800039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f80004040 0x7f9f80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f80004d40 0x7f9f800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c06e1e0 con 0x7f9f80004040 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8000bbb0 con 0x7f9f80004d40 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8000bdb0 con 0x7f9f9c0c1390 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.654+0000 7f9f93e00640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f80004040 0x7f9f80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.655+0000 7f9f9be00640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c0c1390 0x7f9f800039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.655+0000 7f9f93e00640 1 -- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c0c1390 msgr2=0x7f9f800039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.655+0000 7f9f93e00640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c0c1390 0x7f9f800039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.655+0000 7f9f93e00640 1 -- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f80004d40 msgr2=0x7f9f800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.655+0000 7f9f93e00640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f80004d40 0x7f9f800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:45.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.655+0000 7f9f93e00640 1 -- 172.21.15.27:0/1053496294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80003520 con 0x7f9f80004040 2024-03-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.656+0000 7f9f93e00640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f80004040 0x7f9f80004800 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400ba00 tx=0x7f9f80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f2005dae27a11e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:45.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.656+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84036020 con 0x7f9f80004040 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.656+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f84032620 con 0x7f9f80004040 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.656+0000 7f9f9b400640 1 -- 172.21.15.27:0/1053496294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400b600 con 0x7f9f80004040 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.657+0000 7f9fa1600640 1 -- 172.21.15.27:0/1053496294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f80004040 msgr2=0x7f9f80004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.657+0000 7f9fa1600640 1 --2- 172.21.15.27:0/1053496294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f80004040 0x7f9f80004800 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400ba00 tx=0x7f9f80005830 comp rx=0 tx=0).stop 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.657+0000 7f9fa0c00640 1 -- 172.21.15.27:0/1053496294 reap_dead start 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.657+0000 7f9fa1600640 1 -- 172.21.15.27:0/1053496294 shutdown_connections 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.657+0000 7f9fa1600640 1 -- 172.21.15.27:0/1053496294 >> 172.21.15.27:0/1053496294 conn(0x7f9f9c06b0c0 msgr2=0x7f9f9c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.657+0000 7f9fa1600640 1 -- 172.21.15.27:0/1053496294 shutdown_connections 2024-03-28T19:23:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.657+0000 7f9fa1600640 1 -- 172.21.15.27:0/1053496294 wait complete. 2024-03-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.659+0000 7f9fa1600640 1 Processor -- start 2024-03-28T19:23:45.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.659+0000 7f9fa1600640 1 -- start start 2024-03-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f80004040 0x7f9f9c14aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa1600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c14afc0 0x7f9f9c0a6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9c0a74a0 0x7f9f9c071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:45.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa1600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c0c0fc0 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa1600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c06e710 con 0x7f9f80004040 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa1600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c0c2bc0 con 0x7f9f9c14afc0 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9c0a74a0 0x7f9f9c071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9c0a74a0 0x7f9f9c071ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52486/0 (socket says 172.21.15.27:52486) 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa0c00640 1 -- 172.21.15.27:0/4039667654 learned_addr learned my addr 172.21.15.27:0/4039667654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9f9be00640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f80004040 0x7f9f9c14aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9f93e00640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c14afc0 0x7f9f9c0a6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa0c00640 1 -- 172.21.15.27:0/4039667654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c14afc0 msgr2=0x7f9f9c0a6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa0c00640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c14afc0 0x7f9f9c0a6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa0c00640 1 -- 172.21.15.27:0/4039667654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f80004040 msgr2=0x7f9f9c14aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa0c00640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f80004040 0x7f9f9c14aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.660+0000 7f9fa0c00640 1 -- 172.21.15.27:0/4039667654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c1f0040 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.661+0000 7f9fa0c00640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9c0a74a0 0x7f9f9c071ea0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400c4f0 tx=0x7f9f9400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.661+0000 7f9f98c00640 1 -- 172.21.15.27:0/4039667654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f94017070 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.661+0000 7f9f98c00640 1 -- 172.21.15.27:0/4039667654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f940070d0 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.661+0000 7f9f98c00640 1 -- 172.21.15.27:0/4039667654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f940074c0 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.662+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c1f0320 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.662+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c070770 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.663+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c067dd0 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.665+0000 7f9f98c00640 1 -- 172.21.15.27:0/4039667654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9f9401a070 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.666+0000 7f9f98c00640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f7806bbd0 0x7f9f7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:45.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.666+0000 7f9f98c00640 1 -- 172.21.15.27:0/4039667654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f9f9408ad20 con 0x7f9f9c0a74a0 2024-03-28T19:23:45.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.666+0000 7f9f9be00640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f7806bbd0 0x7f9f7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:45.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.667+0000 7f9f9be00640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f7806bbd0 0x7f9f7806e090 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c1f3bf0 tx=0x7f9f8c007040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:45.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:45.672+0000 7f9f98c00640 1 -- 172.21.15.27:0/4039667654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9f94058100 con 0x7f9f9c0a74a0 2024-03-28T19:23:46.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.041+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f9c14b3a0 con 0x7f9f9c0a74a0 2024-03-28T19:23:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.043+0000 7f9f98c00640 1 -- 172.21.15.27:0/4039667654 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f9f9c14b3a0 con 0x7f9f9c0a74a0 2024-03-28T19:23:46.043 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:46.043 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:44.887105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:clientreplay","state_seq":7,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:46.043 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 214 2024-03-28T19:23:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.047+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f7806bbd0 msgr2=0x7f9f7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.047+0000 7f9fa1600640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f7806bbd0 0x7f9f7806e090 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c1f3bf0 tx=0x7f9f8c007040 comp rx=0 tx=0).stop 2024-03-28T19:23:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.047+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9c0a74a0 msgr2=0x7f9f9c071ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:46.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.047+0000 7f9fa1600640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9c0a74a0 0x7f9f9c071ea0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400c4f0 tx=0x7f9f9400c9b0 comp rx=0 tx=0).stop 2024-03-28T19:23:46.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.049+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 shutdown_connections 2024-03-28T19:23:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.049+0000 7f9fa1600640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f7806bbd0 0x7f9f7806e090 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.049+0000 7f9fa1600640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9c0a74a0 0x7f9f9c071ea0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.049+0000 7f9fa1600640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f9c14afc0 0x7f9f9c0a6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.049+0000 7f9fa1600640 1 --2- 172.21.15.27:0/4039667654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f80004040 0x7f9f9c14aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.049+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 >> 172.21.15.27:0/4039667654 conn(0x7f9f9c06b0c0 msgr2=0x7f9f9c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.049+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 shutdown_connections 2024-03-28T19:23:46.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:46.049+0000 7f9fa1600640 1 -- 172.21.15.27:0/4039667654 wait complete. 2024-03-28T19:23:46.073 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:23:48.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:48.158 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 17% (38/217) Updating files: 18% (40/217) Updating files: 19% (42/217) Updating files: 20% (44/217) Updating files: 21% (46/217) Updating files: 22% (48/217) Updating files: 23% (50/217) Updating files: 24% (53/217) Updating files: 25% (55/217) Updating files: 26% (57/217) Updating files: 27% (59/217) Updating files: 28% (61/217) Updating files: 29% (63/217) Updating files: 30% (66/217) Updating files: 31% (68/217) Updating files: 32% (70/217) Updating files: 33% (72/217) Updating files: 34% (74/217) Updating files: 35% (76/217) Updating files: 36% (79/217) Updating files: 37% (81/217) Updating files: 38% (83/217) Updating files: 39% (85/217) Updating files: 40% (87/217) Updating files: 41% (89/217) Updating files: 42% (92/217) Updating files: 43% (94/217) Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 47% (103/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 77% (169/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 80% (175/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2024-03-28T19:23:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.298+0000 7fb4bc400640 1 Processor -- start 2024-03-28T19:23:48.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.299+0000 7fb4bc400640 1 -- start start 2024-03-28T19:23:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4bc400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4b40c3590 0x7fb4b4151670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4bc400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb4b40c2130 0x7fb4b40c2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4bc400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b4152610 con 0x7fb4b414ec20 2024-03-28T19:23:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4bc400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b4151d70 con 0x7fb4b40c3590 2024-03-28T19:23:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4bc400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b4152050 con 0x7fb4b40c2130 2024-03-28T19:23:48.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4b9000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4b40c3590 0x7fb4b4151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4b9000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4b40c3590 0x7fb4b4151670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40528/0 (socket says 172.21.15.27:40528) 2024-03-28T19:23:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4b9000640 1 -- 172.21.15.27:0/3672944471 learned_addr learned my addr 172.21.15.27:0/3672944471 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.301+0000 7fb4ba400640 1 --2- 172.21.15.27:0/3672944471 >> v2:172.21.15.155:3300/0 conn(0x7fb4b40c2130 0x7fb4b40c2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.302+0000 7fb4b9000640 1 -- 172.21.15.27:0/3672944471 >> v2:172.21.15.155:3300/0 conn(0x7fb4b40c2130 msgr2=0x7fb4b40c2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.302+0000 7fb4b9000640 1 --2- 172.21.15.27:0/3672944471 >> v2:172.21.15.155:3300/0 conn(0x7fb4b40c2130 0x7fb4b40c2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.302+0000 7fb4b9000640 1 -- 172.21.15.27:0/3672944471 >> v1:172.21.15.155:6789/0 conn(0x7fb4b414ec20 legacy=0x7fb4b40abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.302+0000 7fb4b9000640 1 -- 172.21.15.27:0/3672944471 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4152330 con 0x7fb4b40c3590 2024-03-28T19:23:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.303+0000 7fb4b9000640 1 --2- 172.21.15.27:0/3672944471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c3590 0x7fb4b4151670 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4155d00 tx=0x7fb4a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c7882066586013 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.303+0000 7fb4b3e00640 1 -- 172.21.15.27:0/3672944471 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0034070 con 0x7fb4b40c3590 2024-03-28T19:23:48.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.303+0000 7fb4b3e00640 1 -- 172.21.15.27:0/3672944471 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a0030860 con 0x7fb4b40c3590 2024-03-28T19:23:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.304+0000 7fb4b3e00640 1 -- 172.21.15.27:0/3672944471 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0030c30 con 0x7fb4b40c3590 2024-03-28T19:23:48.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.304+0000 7fb4bc400640 1 -- 172.21.15.27:0/3672944471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c3590 msgr2=0x7fb4b4151670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.304+0000 7fb4bc400640 1 --2- 172.21.15.27:0/3672944471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c3590 0x7fb4b4151670 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4155d00 tx=0x7fb4a002b9a0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.305+0000 7fb4bc400640 1 -- 172.21.15.27:0/3672944471 shutdown_connections 2024-03-28T19:23:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.305+0000 7fb4bc400640 1 --2- 172.21.15.27:0/3672944471 >> v2:172.21.15.155:3300/0 conn(0x7fb4b40c2130 0x7fb4b40c2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.305+0000 7fb4bc400640 1 --2- 172.21.15.27:0/3672944471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c3590 0x7fb4b4151670 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.305+0000 7fb4bc400640 1 -- 172.21.15.27:0/3672944471 >> 172.21.15.27:0/3672944471 conn(0x7fb4b406b0c0 msgr2=0x7fb4b406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.305+0000 7fb4bc400640 1 -- 172.21.15.27:0/3672944471 shutdown_connections 2024-03-28T19:23:48.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.305+0000 7fb4bc400640 1 -- 172.21.15.27:0/3672944471 wait complete. 2024-03-28T19:23:48.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.307+0000 7fb4bc400640 1 Processor -- start 2024-03-28T19:23:48.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.307+0000 7fb4bc400640 1 -- start start 2024-03-28T19:23:48.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4bc400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c2130 0x7fb4b4149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4bc400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4b40c3590 0x7fb4b4141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4bc400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b414ec20 0x7fb4b4141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4bc400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40aa560 con 0x7fb4b40c2130 2024-03-28T19:23:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4bc400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b414f000 con 0x7fb4b414ec20 2024-03-28T19:23:48.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4bc400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40aa6e0 con 0x7fb4b40c3590 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4b9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c2130 0x7fb4b4149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4b9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c2130 0x7fb4b4149410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40536/0 (socket says 172.21.15.27:40536) 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4b9a00640 1 -- 172.21.15.27:0/2844848310 learned_addr learned my addr 172.21.15.27:0/2844848310 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4ba400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b414ec20 0x7fb4b4141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.308+0000 7fb4b9000640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4b40c3590 0x7fb4b4141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.309+0000 7fb4ba400640 1 -- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4b40c3590 msgr2=0x7fb4b4141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.309+0000 7fb4ba400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4b40c3590 0x7fb4b4141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.309+0000 7fb4ba400640 1 -- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c2130 msgr2=0x7fb4b4149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:48.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.309+0000 7fb4ba400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c2130 0x7fb4b4149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.309+0000 7fb4ba400640 1 -- 172.21.15.27:0/2844848310 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b41421b0 con 0x7fb4b414ec20 2024-03-28T19:23:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.309+0000 7fb4b9a00640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c2130 0x7fb4b4149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:23:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.310+0000 7fb4ba400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b414ec20 0x7fb4b4141c70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00efc0 tx=0x7fb4ac00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.310+0000 7fb4b1600640 1 -- 172.21.15.27:0/2844848310 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac018020 con 0x7fb4b414ec20 2024-03-28T19:23:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.310+0000 7fb4b1600640 1 -- 172.21.15.27:0/2844848310 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac014480 con 0x7fb4b414ec20 2024-03-28T19:23:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.310+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b4145890 con 0x7fb4b414ec20 2024-03-28T19:23:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.310+0000 7fb4b1600640 1 -- 172.21.15.27:0/2844848310 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac014870 con 0x7fb4b414ec20 2024-03-28T19:23:48.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.310+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b4145e50 con 0x7fb4b414ec20 2024-03-28T19:23:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.312+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b404f400 con 0x7fb4b414ec20 2024-03-28T19:23:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.313+0000 7fb4b1600640 1 -- 172.21.15.27:0/2844848310 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb4ac005d10 con 0x7fb4b414ec20 2024-03-28T19:23:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.314+0000 7fb4b1600640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb49406bbd0 0x7fb49406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.314+0000 7fb4b1600640 1 -- 172.21.15.27:0/2844848310 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7fb4ac08c720 con 0x7fb4b414ec20 2024-03-28T19:23:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.314+0000 7fb4b9a00640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb49406bbd0 0x7fb49406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.315+0000 7fb4b9a00640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb49406bbd0 0x7fb49406e090 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8007fd0 tx=0x7fb4a8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.334+0000 7fb4b1600640 1 -- 172.21.15.27:0/2844848310 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb4ac022030 con 0x7fb4b414ec20 2024-03-28T19:23:48.362 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 55% (38/69) Updating files: 56% (39/69) Updating files: 57% (40/69) Updating files: 59% (41/69) Updating files: 60% (42/69) Updating files: 62% (43/69) Updating files: 63% (44/69) Updating files: 65% (45/69) Updating files: 66% (46/69) Updating files: 68% (47/69) Updating files: 69% (48/69) Updating files: 71% (49/69) Updating files: 72% (50/69) Updating files: 73% (51/69) Updating files: 75% (52/69) Updating files: 76% (53/69) Updating files: 78% (54/69) Updating files: 79% (55/69) Updating files: 81% (56/69) Updating files: 82% (57/69) Updating files: 84% (58/69) Updating files: 85% (59/69) Updating files: 86% (60/69) Updating files: 88% (61/69) Updating files: 89% (62/69) Updating files: 91% (63/69) Updating files: 92% (64/69) Updating files: 94% (65/69) Updating files: 95% (66/69) Updating files: 97% (67/69) Updating files: 98% (68/69) Updating files: 100% (69/69) Updating files: 100% (69/69), done. 2024-03-28T19:23:48.497 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 98f5be53f85 v0.19 2024-03-28T19:23:48.508 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.19 2024-03-28T19:23:48.536 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:23:48.537 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 20 -eq 48 ']' 2024-03-28T19:23:48.537 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.20 2024-03-28T19:23:48.537 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.20 2024-03-28T19:23:48.537 INFO:tasks.workunit.client.0.smithi027.stdout:v0.20 2024-03-28T19:23:48.537 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.20 2024-03-28T19:23:48.576 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2024-03-28T19:23:48.581 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.11 2024-03-28T19:23:48.668 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:23:48.668 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 12 -eq 48 ']' 2024-03-28T19:23:48.668 INFO:tasks.workunit.client.1.smithi155.stdout:v0.12 2024-03-28T19:23:48.668 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.12 2024-03-28T19:23:48.668 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.12 2024-03-28T19:23:48.669 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.12 2024-03-28T19:23:48.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.712+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4b40c3370 con 0x7fb4b414ec20 2024-03-28T19:23:48.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.714+0000 7fb4b1600640 1 -- 172.21.15.27:0/2844848310 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5472 (secure 0 0 0) 0x7fb4ac010040 con 0x7fb4b414ec20 2024-03-28T19:23:48.714 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:48.714 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:48.714 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 215 2024-03-28T19:23:48.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.716+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb49406bbd0 msgr2=0x7fb49406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.716+0000 7fb4bc400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb49406bbd0 0x7fb49406e090 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8007fd0 tx=0x7fb4a8009040 comp rx=0 tx=0).stop 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.716+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b414ec20 msgr2=0x7fb4b4141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.716+0000 7fb4bc400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b414ec20 0x7fb4b4141c70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00efc0 tx=0x7fb4ac00c490 comp rx=0 tx=0).stop 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.716+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 shutdown_connections 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.717+0000 7fb4bc400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb49406bbd0 0x7fb49406e090 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.717+0000 7fb4bc400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b414ec20 0x7fb4b4141c70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.717+0000 7fb4bc400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4b40c3590 0x7fb4b4141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.717+0000 7fb4bc400640 1 --2- 172.21.15.27:0/2844848310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4b40c2130 0x7fb4b4149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.717+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 >> 172.21.15.27:0/2844848310 conn(0x7fb4b406b0c0 msgr2=0x7fb4b40c6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.717+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 shutdown_connections 2024-03-28T19:23:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:48.717+0000 7fb4bc400640 1 -- 172.21.15.27:0/2844848310 wait complete. 2024-03-28T19:23:48.738 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=8259 2024-03-28T19:23:48.738 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.a 2024-03-28T19:23:51.686 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:23:51.686+0000 7f0f7d200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:23:51.739 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-03-28T19:23:51.739 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:23:51.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:23:51.746 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:23:51.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:51.958 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:23:51.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:51.999+0000 7f78bfe00640 1 Processor -- start 2024-03-28T19:23:51.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:51.999+0000 7f78bfe00640 1 -- start start 2024-03-28T19:23:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.000+0000 7f78bfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78c00727e0 0x7f78c00c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.001+0000 7f78bfe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c0155fe0 con 0x7f78c0155600 2024-03-28T19:23:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.001+0000 7f78bfe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c01562c0 con 0x7f78c00c59d0 2024-03-28T19:23:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.001+0000 7f78bfe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c01565a0 con 0x7f78c00727e0 2024-03-28T19:23:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.001+0000 7f78bf400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f78c0155600 0x7f78c014f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43704/0 (socket says 172.21.15.27:43704) 2024-03-28T19:23:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.001+0000 7f78bf400640 1 -- 172.21.15.27:0/4222642947 learned_addr learned my addr 172.21.15.27:0/4222642947 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:23:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.001+0000 7f78be000640 1 --2- 172.21.15.27:0/4222642947 >> v2:172.21.15.27:3300/0 conn(0x7f78c00727e0 0x7f78c00c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.002+0000 7f78bd600640 1 -- 172.21.15.27:0/4222642947 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162134147 0 0) 0x7f78c0155fe0 con 0x7f78c0155600 2024-03-28T19:23:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.002+0000 7f78bd600640 1 -- 172.21.15.27:0/4222642947 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a4003520 con 0x7f78c0155600 2024-03-28T19:23:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.002+0000 7f78bd600640 1 -- 172.21.15.27:0/4222642947 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393509695 0 0) 0x7f78c01562c0 con 0x7f78c00c59d0 2024-03-28T19:23:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.002+0000 7f78bd600640 1 -- 172.21.15.27:0/4222642947 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c0155fe0 con 0x7f78c00c59d0 2024-03-28T19:23:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.002+0000 7f78be000640 1 -- 172.21.15.27:0/4222642947 >> v1:172.21.15.155:6789/0 conn(0x7f78c00c59d0 legacy=0x7f78c0159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.002+0000 7f78be000640 1 -- 172.21.15.27:0/4222642947 >> v1:172.21.15.27:6789/0 conn(0x7f78c0155600 legacy=0x7f78c014f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.002+0000 7f78be000640 1 -- 172.21.15.27:0/4222642947 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c0158d20 con 0x7f78c00727e0 2024-03-28T19:23:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.003+0000 7f78be000640 1 --2- 172.21.15.27:0/4222642947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00c8f00 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0097a0 tx=0x7f78ac02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5993d9be8d67127 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.004+0000 7f78bd600640 1 -- 172.21.15.27:0/4222642947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac038070 con 0x7f78c00727e0 2024-03-28T19:23:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.004+0000 7f78bd600640 1 -- 172.21.15.27:0/4222642947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac031700 con 0x7f78c00727e0 2024-03-28T19:23:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.004+0000 7f78bd600640 1 -- 172.21.15.27:0/4222642947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac031ab0 con 0x7f78c00727e0 2024-03-28T19:23:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.005+0000 7f78bfe00640 1 -- 172.21.15.27:0/4222642947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 msgr2=0x7f78c00c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.005+0000 7f78bfe00640 1 --2- 172.21.15.27:0/4222642947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00c8f00 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0097a0 tx=0x7f78ac02b7a0 comp rx=0 tx=0).stop 2024-03-28T19:23:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.005+0000 7f78bfe00640 1 -- 172.21.15.27:0/4222642947 shutdown_connections 2024-03-28T19:23:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.005+0000 7f78bfe00640 1 --2- 172.21.15.27:0/4222642947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00c8f00 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.005+0000 7f78bfe00640 1 -- 172.21.15.27:0/4222642947 >> 172.21.15.27:0/4222642947 conn(0x7f78c00ac910 msgr2=0x7f78c00accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.005+0000 7f78bfe00640 1 -- 172.21.15.27:0/4222642947 shutdown_connections 2024-03-28T19:23:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.005+0000 7f78bfe00640 1 -- 172.21.15.27:0/4222642947 wait complete. 2024-03-28T19:23:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.008+0000 7f78bfe00640 1 Processor -- start 2024-03-28T19:23:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.008+0000 7f78bfe00640 1 -- start start 2024-03-28T19:23:52.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78bfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00be910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:52.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78bfe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78c00c59d0 0x7f78c01ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:52.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78bfe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78c0155600 0x7f78c00c0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78bfe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c00c5db0 con 0x7f78c00727e0 2024-03-28T19:23:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78bfe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c0159090 con 0x7f78c0155600 2024-03-28T19:23:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78bfe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c00c5f30 con 0x7f78c00c59d0 2024-03-28T19:23:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78bea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00be910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78bea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00be910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40574/0 (socket says 172.21.15.27:40574) 2024-03-28T19:23:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.011+0000 7f78bea00640 1 -- 172.21.15.27:0/2562554365 learned_addr learned my addr 172.21.15.27:0/2562554365 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.010+0000 7f78be000640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78c00c59d0 0x7f78c01ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.011+0000 7f78bf400640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78c0155600 0x7f78c00c0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.011+0000 7f78bea00640 1 -- 172.21.15.27:0/2562554365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78c00c59d0 msgr2=0x7f78c01ee210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.011+0000 7f78bea00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78c00c59d0 0x7f78c01ee210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.011+0000 7f78bea00640 1 -- 172.21.15.27:0/2562554365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78c0155600 msgr2=0x7f78c00c0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.011+0000 7f78bea00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78c0155600 0x7f78c00c0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.011+0000 7f78bea00640 1 -- 172.21.15.27:0/2562554365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c00bfd40 con 0x7f78c00727e0 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.012+0000 7f78bea00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00be910 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f78b000c4d0 tx=0x7f78b000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.012+0000 7f78b6a00640 1 -- 172.21.15.27:0/2562554365 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b0005480 con 0x7f78c00727e0 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.012+0000 7f78b6a00640 1 -- 172.21.15.27:0/2562554365 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b00035f0 con 0x7f78c00727e0 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.012+0000 7f78b6a00640 1 -- 172.21.15.27:0/2562554365 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b00039e0 con 0x7f78c00727e0 2024-03-28T19:23:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.012+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c0153400 con 0x7f78c00727e0 2024-03-28T19:23:52.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.012+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c00c27f0 con 0x7f78c00727e0 2024-03-28T19:23:52.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.014+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c0067dd0 con 0x7f78c00727e0 2024-03-28T19:23:52.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.015+0000 7f78b6a00640 1 -- 172.21.15.27:0/2562554365 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f78b0003b80 con 0x7f78c00727e0 2024-03-28T19:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.017+0000 7f78b6a00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78a006bdf0 0x7f78a006e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.017+0000 7f78b6a00640 1 -- 172.21.15.27:0/2562554365 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f78b008b5b0 con 0x7f78c00727e0 2024-03-28T19:23:52.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.018+0000 7f78be000640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78a006bdf0 0x7f78a006e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:52.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.019+0000 7f78be000640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78a006bdf0 0x7f78a006e2b0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f78ac035000 tx=0x7f78ac008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:52.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.023+0000 7f78b6a00640 1 -- 172.21.15.27:0/2562554365 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f78b0058a40 con 0x7f78c00727e0 2024-03-28T19:23:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.383+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78c0069640 con 0x7f78c00727e0 2024-03-28T19:23:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.384+0000 7f78b6a00640 1 -- 172.21.15.27:0/2562554365 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5472 (secure 0 0 0) 0x7f78b005cae0 con 0x7f78c00727e0 2024-03-28T19:23:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 215 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78a006bdf0 msgr2=0x7f78a006e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78a006bdf0 0x7f78a006e2b0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f78ac035000 tx=0x7f78ac008030 comp rx=0 tx=0).stop 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 msgr2=0x7f78c00be910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00be910 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f78b000c4d0 tx=0x7f78b000c990 comp rx=0 tx=0).stop 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 shutdown_connections 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f78a006bdf0 0x7f78a006e2b0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78c0155600 0x7f78c00c0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78c00c59d0 0x7f78c01ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 --2- 172.21.15.27:0/2562554365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c00727e0 0x7f78c00be910 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:52.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 >> 172.21.15.27:0/2562554365 conn(0x7f78c00ac910 msgr2=0x7f78c0153f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:52.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 shutdown_connections 2024-03-28T19:23:52.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:52.387+0000 7f78bfe00640 1 -- 172.21.15.27:0/2562554365 wait complete. 2024-03-28T19:23:52.412 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-03-28T19:23:53.238 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 62% (47/75) Updating files: 64% (48/75) Updating files: 65% (49/75) Updating files: 66% (50/75) Updating files: 68% (51/75) Updating files: 69% (52/75) Updating files: 70% (53/75) Updating files: 72% (54/75) Updating files: 73% (55/75) Updating files: 74% (56/75) Updating files: 76% (57/75) Updating files: 77% (58/75) Updating files: 78% (59/75) Updating files: 80% (60/75) Updating files: 81% (61/75) Updating files: 82% (62/75) Updating files: 84% (63/75) Updating files: 85% (64/75) Updating files: 86% (65/75) Updating files: 88% (66/75) Updating files: 89% (67/75) Updating files: 90% (68/75) Updating files: 92% (69/75) Updating files: 93% (70/75) Updating files: 94% (71/75) Updating files: 96% (72/75) Updating files: 97% (73/75) Updating files: 98% (74/75) Updating files: 100% (75/75) Updating files: 100% (75/75), done. 2024-03-28T19:23:53.465 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at f524150822a v0.12 2024-03-28T19:23:53.471 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.12 2024-03-28T19:23:53.529 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:23:53.529 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 13 -eq 48 ']' 2024-03-28T19:23:53.529 INFO:tasks.workunit.client.1.smithi155.stdout:v0.13 2024-03-28T19:23:53.530 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.13 2024-03-28T19:23:53.530 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.13 2024-03-28T19:23:53.530 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.13 2024-03-28T19:23:54.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:23:54.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.649+0000 7f977b200640 1 Processor -- start 2024-03-28T19:23:54.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.649+0000 7f977b200640 1 -- start start 2024-03-28T19:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.651+0000 7f977b200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f977c1551a0 0x7f977c0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.651+0000 7f977b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f977c0aada0 0x7f977c0c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.651+0000 7f977b200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0c26d0 con 0x7f977c0c5580 2024-03-28T19:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.651+0000 7f977b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f977c156870 con 0x7f977c0aada0 2024-03-28T19:23:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.651+0000 7f977b200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f977c155cf0 con 0x7f977c1551a0 2024-03-28T19:23:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.651+0000 7f977a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f977c0aada0 0x7f977c0c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.652+0000 7f977a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f977c0aada0 0x7f977c0c2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40580/0 (socket says 172.21.15.27:40580) 2024-03-28T19:23:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.652+0000 7f977a800640 1 -- 172.21.15.27:0/2839651705 learned_addr learned my addr 172.21.15.27:0/2839651705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.652+0000 7f9779e00640 1 --2- 172.21.15.27:0/2839651705 >> v2:172.21.15.155:3300/0 conn(0x7f977c1551a0 0x7f977c0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.652+0000 7f9779400640 1 --1- 172.21.15.27:0/2839651705 >> v1:172.21.15.155:6789/0 conn(0x7f977c0c5580 0x7f977c0c1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:37592/0 (socket says 172.21.15.27:37592) 2024-03-28T19:23:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.652+0000 7f977a800640 1 -- 172.21.15.27:0/2839651705 >> v2:172.21.15.155:3300/0 conn(0x7f977c1551a0 msgr2=0x7f977c0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:23:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.652+0000 7f977a800640 1 --2- 172.21.15.27:0/2839651705 >> v2:172.21.15.155:3300/0 conn(0x7f977c1551a0 0x7f977c0c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.652+0000 7f977a800640 1 -- 172.21.15.27:0/2839651705 >> v1:172.21.15.155:6789/0 conn(0x7f977c0c5580 legacy=0x7f977c0c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:54.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.652+0000 7f977a800640 1 -- 172.21.15.27:0/2839651705 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c155fd0 con 0x7f977c0aada0 2024-03-28T19:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.654+0000 7f977a800640 1 --2- 172.21.15.27:0/2839651705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0aada0 0x7f977c0c2190 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f977c06d290 tx=0x7f977402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb084a1fee798af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.654+0000 7f9778a00640 1 -- 172.21.15.27:0/2839651705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977402c9f0 con 0x7f977c0aada0 2024-03-28T19:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.654+0000 7f9778a00640 1 -- 172.21.15.27:0/2839651705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977402cb90 con 0x7f977c0aada0 2024-03-28T19:23:54.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.654+0000 7f9778a00640 1 -- 172.21.15.27:0/2839651705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740343f0 con 0x7f977c0aada0 2024-03-28T19:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.655+0000 7f977b200640 1 -- 172.21.15.27:0/2839651705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0aada0 msgr2=0x7f977c0c2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:54.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.655+0000 7f977b200640 1 --2- 172.21.15.27:0/2839651705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0aada0 0x7f977c0c2190 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f977c06d290 tx=0x7f977402be50 comp rx=0 tx=0).stop 2024-03-28T19:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.657+0000 7f977b200640 1 -- 172.21.15.27:0/2839651705 shutdown_connections 2024-03-28T19:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.657+0000 7f977b200640 1 --2- 172.21.15.27:0/2839651705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0aada0 0x7f977c0c2190 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.657+0000 7f977b200640 1 --2- 172.21.15.27:0/2839651705 >> v2:172.21.15.155:3300/0 conn(0x7f977c1551a0 0x7f977c0c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.657+0000 7f977b200640 1 -- 172.21.15.27:0/2839651705 >> 172.21.15.27:0/2839651705 conn(0x7f977c06b0c0 msgr2=0x7f977c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.658+0000 7f977b200640 1 -- 172.21.15.27:0/2839651705 shutdown_connections 2024-03-28T19:23:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.658+0000 7f977b200640 1 -- 172.21.15.27:0/2839651705 wait complete. 2024-03-28T19:23:54.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.661+0000 7f977b200640 1 Processor -- start 2024-03-28T19:23:54.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.661+0000 7f977b200640 1 -- start start 2024-03-28T19:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.663+0000 7f977b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c0aada0 0x7f977c073110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.663+0000 7f977b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0c5580 0x7f977c14a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.663+0000 7f977b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1551a0 0x7f977c144c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.663+0000 7f977b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c1559a0 con 0x7f977c0c5580 2024-03-28T19:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.663+0000 7f977b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c156590 con 0x7f977c1551a0 2024-03-28T19:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.663+0000 7f977b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c155580 con 0x7f977c0aada0 2024-03-28T19:23:54.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.664+0000 7f9779400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0c5580 0x7f977c14a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.664+0000 7f9779e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c0aada0 0x7f977c073110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.664+0000 7f9779e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c0aada0 0x7f977c073110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:50848/0 (socket says 172.21.15.27:50848) 2024-03-28T19:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.664+0000 7f9779e00640 1 -- 172.21.15.27:0/2553279025 learned_addr learned my addr 172.21.15.27:0/2553279025 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:23:54.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.665+0000 7f977a800640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1551a0 0x7f977c144c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.666+0000 7f977a800640 1 -- 172.21.15.27:0/2553279025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c0aada0 msgr2=0x7f977c073110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.666+0000 7f977a800640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c0aada0 0x7f977c073110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:54.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.666+0000 7f977a800640 1 -- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0c5580 msgr2=0x7f977c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.666+0000 7f977a800640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0c5580 0x7f977c14a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.666+0000 7f977a800640 1 -- 172.21.15.27:0/2553279025 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c070800 con 0x7f977c1551a0 2024-03-28T19:23:54.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.668+0000 7f977a800640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1551a0 0x7f977c144c50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f977c06ea20 tx=0x7f9774004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.669+0000 7f9772000640 1 -- 172.21.15.27:0/2553279025 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774044020 con 0x7f977c1551a0 2024-03-28T19:23:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.669+0000 7f9772000640 1 -- 172.21.15.27:0/2553279025 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9774040470 con 0x7f977c1551a0 2024-03-28T19:23:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.669+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c14dff0 con 0x7f977c1551a0 2024-03-28T19:23:54.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.669+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c14e570 con 0x7f977c1551a0 2024-03-28T19:23:54.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.670+0000 7f9772000640 1 -- 172.21.15.27:0/2553279025 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977402c9f0 con 0x7f977c1551a0 2024-03-28T19:23:54.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.671+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9754005290 con 0x7f977c1551a0 2024-03-28T19:23:54.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.673+0000 7f9772000640 1 -- 172.21.15.27:0/2553279025 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9774047020 con 0x7f977c1551a0 2024-03-28T19:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.675+0000 7f9772000640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc20 0x7f975c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.675+0000 7f9772000640 1 -- 172.21.15.27:0/2553279025 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f97740ae170 con 0x7f977c1551a0 2024-03-28T19:23:54.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.676+0000 7f9779e00640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc20 0x7f975c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:23:54.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.677+0000 7f9779e00640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc20 0x7f975c06e0e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f976c009b60 tx=0x7f976c009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:23:54.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:54.682+0000 7f9772000640 1 -- 172.21.15.27:0/2553279025 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f977404e030 con 0x7f977c1551a0 2024-03-28T19:23:55.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.008+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9754005850 con 0x7f977c1551a0 2024-03-28T19:23:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.010+0000 7f9772000640 1 -- 172.21.15.27:0/2553279025 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f977c070800 con 0x7f977c1551a0 2024-03-28T19:23:55.009 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:23:55.009 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:23:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 216 2024-03-28T19:23:55.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc20 msgr2=0x7f975c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc20 0x7f975c06e0e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f976c009b60 tx=0x7f976c009190 comp rx=0 tx=0).stop 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1551a0 msgr2=0x7f977c144c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1551a0 0x7f977c144c50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f977c06ea20 tx=0x7f9774004a40 comp rx=0 tx=0).stop 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 shutdown_connections 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f975c06bc20 0x7f975c06e0e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f977c1551a0 0x7f977c144c50 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f977c0c5580 0x7f977c14a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 --2- 172.21.15.27:0/2553279025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f977c0aada0 0x7f977c073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 >> 172.21.15.27:0/2553279025 conn(0x7f977c06b0c0 msgr2=0x7f977c14d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 shutdown_connections 2024-03-28T19:23:55.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:23:55.012+0000 7f977b200640 1 -- 172.21.15.27:0/2553279025 wait complete. 2024-03-28T19:23:55.037 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-03-28T19:23:55.037 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2024-03-28T19:23:56.416 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 41% (102/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 58% (144/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 89% (220/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2024-03-28T19:23:56.615 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2024-03-28T19:23:56.622 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.20 2024-03-28T19:23:56.675 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:23:56.675 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 21 -eq 48 ']' 2024-03-28T19:23:56.675 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.21 2024-03-28T19:23:56.675 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.21 2024-03-28T19:23:56.676 INFO:tasks.workunit.client.0.smithi027.stdout:v0.21 2024-03-28T19:23:56.676 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.21 2024-03-28T19:23:58.862 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 37% (50/133) Updating files: 38% (51/133) Updating files: 39% (52/133) Updating files: 40% (54/133) Updating files: 41% (55/133) Updating files: 42% (56/133) Updating files: 43% (58/133) Updating files: 44% (59/133) Updating files: 45% (60/133) Updating files: 46% (62/133) Updating files: 47% (63/133) Updating files: 48% (64/133) Updating files: 49% (66/133) Updating files: 50% (67/133) Updating files: 51% (68/133) Updating files: 52% (70/133) Updating files: 53% (71/133) Updating files: 54% (72/133) Updating files: 55% (74/133) Updating files: 56% (75/133) Updating files: 57% (76/133) Updating files: 58% (78/133) Updating files: 59% (79/133) Updating files: 60% (80/133) Updating files: 61% (82/133) Updating files: 62% (83/133) Updating files: 63% (84/133) Updating files: 64% (86/133) Updating files: 65% (87/133) Updating files: 66% (88/133) Updating files: 67% (90/133) Updating files: 68% (91/133) Updating files: 69% (92/133) Updating files: 70% (94/133) Updating files: 71% (95/133) Updating files: 72% (96/133) Updating files: 73% (98/133) Updating files: 74% (99/133) Updating files: 75% (100/133) Updating files: 76% (102/133) Updating files: 77% (103/133) Updating files: 78% (104/133) Updating files: 79% (106/133) Updating files: 80% (107/133) Updating files: 81% (108/133) Updating files: 82% (110/133) Updating files: 83% (111/133) Updating files: 84% (112/133) Updating files: 85% (114/133) Updating files: 86% (115/133) Updating files: 87% (116/133) Updating files: 88% (118/133) Updating files: 89% (119/133) Updating files: 90% (120/133) Updating files: 91% (122/133) Updating files: 92% (123/133) Updating files: 93% (124/133) Updating files: 94% (126/133) Updating files: 95% (127/133) Updating files: 96% (128/133) Updating files: 96% (129/133) Updating files: 97% (130/133) Updating files: 98% (131/133) Updating files: 99% (132/133) Updating files: 100% (133/133) Updating files: 100% (133/133), done. 2024-03-28T19:23:59.038 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 2b32297267d v0.13 2024-03-28T19:23:59.044 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.13 2024-03-28T19:23:59.086 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:23:59.086 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 14 -eq 48 ']' 2024-03-28T19:23:59.086 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.14 2024-03-28T19:23:59.087 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.14 2024-03-28T19:23:59.087 INFO:tasks.workunit.client.1.smithi155.stdout:v0.14 2024-03-28T19:23:59.087 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.14 2024-03-28T19:24:04.499 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 36% (55/151) Updating files: 37% (56/151) Updating files: 38% (58/151) Updating files: 39% (59/151) Updating files: 40% (61/151) Updating files: 41% (62/151) Updating files: 42% (64/151) Updating files: 43% (65/151) Updating files: 44% (67/151) Updating files: 45% (68/151) Updating files: 46% (70/151) Updating files: 47% (71/151) Updating files: 48% (73/151) Updating files: 49% (74/151) Updating files: 50% (76/151) Updating files: 51% (78/151) Updating files: 52% (79/151) Updating files: 53% (81/151) Updating files: 54% (82/151) Updating files: 55% (84/151) Updating files: 56% (85/151) Updating files: 57% (87/151) Updating files: 58% (88/151) Updating files: 59% (90/151) Updating files: 60% (91/151) Updating files: 61% (93/151) Updating files: 62% (94/151) Updating files: 62% (95/151) Updating files: 63% (96/151) Updating files: 64% (97/151) Updating files: 65% (99/151) Updating files: 66% (100/151) Updating files: 67% (102/151) Updating files: 68% (103/151) Updating files: 69% (105/151) Updating files: 70% (106/151) Updating files: 71% (108/151) Updating files: 72% (109/151) Updating files: 73% (111/151) Updating files: 74% (112/151) Updating files: 75% (114/151) Updating files: 76% (115/151) Updating files: 77% (117/151) Updating files: 78% (118/151) Updating files: 79% (120/151) Updating files: 80% (121/151) Updating files: 81% (123/151) Updating files: 82% (124/151) Updating files: 83% (126/151) Updating files: 84% (127/151) Updating files: 85% (129/151) Updating files: 86% (130/151) Updating files: 87% (132/151) Updating files: 88% (133/151) Updating files: 89% (135/151) Updating files: 90% (136/151) Updating files: 91% (138/151) Updating files: 92% (139/151) Updating files: 93% (141/151) Updating files: 94% (142/151) Updating files: 95% (144/151) Updating files: 96% (145/151) Updating files: 97% (147/151) Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2024-03-28T19:24:04.758 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2024-03-28T19:24:04.766 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.14 2024-03-28T19:24:04.825 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:24:04.825 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 15 -eq 48 ']' 2024-03-28T19:24:04.825 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.15 2024-03-28T19:24:04.825 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.15 2024-03-28T19:24:04.826 INFO:tasks.workunit.client.1.smithi155.stdout:v0.15 2024-03-28T19:24:04.826 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.15 2024-03-28T19:24:09.446 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 46% (52/112) Updating files: 47% (53/112) Updating files: 48% (54/112) Updating files: 49% (55/112) Updating files: 50% (56/112) Updating files: 51% (58/112) Updating files: 52% (59/112) Updating files: 53% (60/112) Updating files: 54% (61/112) Updating files: 55% (62/112) Updating files: 56% (63/112) Updating files: 57% (64/112) Updating files: 58% (65/112) Updating files: 59% (67/112) Updating files: 60% (68/112) Updating files: 61% (69/112) Updating files: 62% (70/112) Updating files: 63% (71/112) Updating files: 64% (72/112) Updating files: 65% (73/112) Updating files: 66% (74/112) Updating files: 67% (76/112) Updating files: 68% (77/112) Updating files: 69% (78/112) Updating files: 70% (79/112) Updating files: 71% (80/112) Updating files: 72% (81/112) Updating files: 73% (82/112) Updating files: 74% (83/112) Updating files: 75% (84/112) Updating files: 76% (86/112) Updating files: 77% (87/112) Updating files: 78% (88/112) Updating files: 79% (89/112) Updating files: 80% (90/112) Updating files: 81% (91/112) Updating files: 82% (92/112) Updating files: 83% (93/112) Updating files: 84% (95/112) Updating files: 85% (96/112) Updating files: 86% (97/112) Updating files: 87% (98/112) Updating files: 88% (99/112) Updating files: 89% (100/112) Updating files: 90% (101/112) Updating files: 91% (102/112) Updating files: 92% (104/112) Updating files: 93% (105/112) Updating files: 94% (106/112) Updating files: 95% (107/112) Updating files: 96% (108/112) Updating files: 97% (109/112) Updating files: 98% (110/112) Updating files: 99% (111/112) Updating files: 100% (112/112) Updating files: 100% (112/112), done. 2024-03-28T19:24:09.663 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at ab3c421eb95 v0.15 2024-03-28T19:24:09.670 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.15 2024-03-28T19:24:09.718 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:24:09.718 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 16 -eq 48 ']' 2024-03-28T19:24:09.718 INFO:tasks.workunit.client.1.smithi155.stdout:v0.16 2024-03-28T19:24:09.718 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.16 2024-03-28T19:24:09.718 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.16 2024-03-28T19:24:09.719 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.16 2024-03-28T19:24:11.361 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 13% (70/537) Updating files: 14% (76/537) Updating files: 15% (81/537) Updating files: 16% (86/537) Updating files: 17% (92/537) Updating files: 18% (97/537) Updating files: 19% (103/537) Updating files: 20% (108/537) Updating files: 21% (113/537) Updating files: 21% (115/537) Updating files: 22% (119/537) Updating files: 23% (124/537) Updating files: 24% (129/537) Updating files: 25% (135/537) Updating files: 26% (140/537) Updating files: 27% (145/537) Updating files: 28% (151/537) Updating files: 29% (156/537) Updating files: 30% (162/537) Updating files: 31% (167/537) Updating files: 32% (172/537) Updating files: 33% (178/537) Updating files: 33% (181/537) Updating files: 34% (183/537) Updating files: 35% (188/537) Updating files: 36% (194/537) Updating files: 37% (199/537) Updating files: 38% (205/537) Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 40% (217/537) Updating files: 41% (221/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 46% (248/537) Updating files: 46% (251/537) Updating files: 47% (253/537) Updating files: 48% (258/537) Updating files: 49% (264/537) Updating files: 50% (269/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 54% (292/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 69% (374/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 80% (434/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 88% (474/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 94% (509/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2024-03-28T19:24:11.656 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 090436f56ac v0.21 2024-03-28T19:24:11.666 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.21 2024-03-28T19:24:11.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:24:11.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 22 -eq 48 ']' 2024-03-28T19:24:11.717 INFO:tasks.workunit.client.0.smithi027.stdout:v0.22 2024-03-28T19:24:11.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.22 2024-03-28T19:24:11.717 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.22 2024-03-28T19:24:11.718 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.22 2024-03-28T19:24:12.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:24:12.737 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:24:12.855 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:24:12.856+0000 7f0f82200640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 74377) UID: 0 2024-03-28T19:24:12.855 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:24:12.856+0000 7fbc69800640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 74377) UID: 0 2024-03-28T19:24:14.554 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 43% (47/107) Updating files: 44% (48/107) Updating files: 45% (49/107) Updating files: 46% (50/107) Updating files: 47% (51/107) Updating files: 48% (52/107) Updating files: 49% (53/107) Updating files: 50% (54/107) Updating files: 51% (55/107) Updating files: 52% (56/107) Updating files: 53% (57/107) Updating files: 54% (58/107) Updating files: 55% (59/107) Updating files: 56% (60/107) Updating files: 57% (61/107) Updating files: 58% (63/107) Updating files: 59% (64/107) Updating files: 60% (65/107) Updating files: 61% (66/107) Updating files: 62% (67/107) Updating files: 63% (68/107) Updating files: 64% (69/107) Updating files: 65% (70/107) Updating files: 66% (71/107) Updating files: 67% (72/107) Updating files: 68% (73/107) Updating files: 69% (74/107) Updating files: 70% (75/107) Updating files: 71% (76/107) Updating files: 72% (78/107) Updating files: 73% (79/107) Updating files: 74% (80/107) Updating files: 75% (81/107) Updating files: 76% (82/107) Updating files: 77% (83/107) Updating files: 78% (84/107) Updating files: 79% (85/107) Updating files: 80% (86/107) Updating files: 81% (87/107) Updating files: 82% (88/107) Updating files: 83% (89/107) Updating files: 84% (90/107) Updating files: 85% (91/107) Updating files: 86% (93/107) Updating files: 87% (94/107) Updating files: 88% (95/107) Updating files: 89% (96/107) Updating files: 90% (97/107) Updating files: 91% (98/107) Updating files: 92% (99/107) Updating files: 93% (100/107) Updating files: 94% (101/107) Updating files: 95% (102/107) Updating files: 96% (103/107) Updating files: 97% (104/107) Updating files: 98% (105/107) Updating files: 99% (106/107) Updating files: 100% (107/107) Updating files: 100% (107/107), done. 2024-03-28T19:24:14.791 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 38dbaa5175c v0.16 2024-03-28T19:24:14.797 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.16 2024-03-28T19:24:14.911 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:24:14.911 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 17 -eq 48 ']' 2024-03-28T19:24:14.912 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.17 2024-03-28T19:24:14.912 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.17 2024-03-28T19:24:14.912 INFO:tasks.workunit.client.1.smithi155.stdout:v0.17 2024-03-28T19:24:14.912 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.17 2024-03-28T19:24:17.860 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 27% (57/204) Updating files: 28% (58/204) Updating files: 29% (60/204) Updating files: 30% (62/204) Updating files: 31% (64/204) Updating files: 32% (66/204) Updating files: 33% (68/204) Updating files: 34% (70/204) Updating files: 35% (72/204) Updating files: 36% (74/204) Updating files: 37% (76/204) Updating files: 38% (78/204) Updating files: 39% (80/204) Updating files: 40% (82/204) Updating files: 41% (84/204) Updating files: 42% (86/204) Updating files: 43% (88/204) Updating files: 44% (90/204) Updating files: 45% (92/204) Updating files: 46% (94/204) Updating files: 47% (96/204) Updating files: 48% (98/204) Updating files: 49% (100/204) Updating files: 50% (102/204) Updating files: 51% (105/204) Updating files: 52% (107/204) Updating files: 52% (108/204) Updating files: 53% (109/204) Updating files: 54% (111/204) Updating files: 55% (113/204) Updating files: 56% (115/204) Updating files: 57% (117/204) Updating files: 58% (119/204) Updating files: 59% (121/204) Updating files: 60% (123/204) Updating files: 61% (125/204) Updating files: 62% (127/204) Updating files: 63% (129/204) Updating files: 64% (131/204) Updating files: 65% (133/204) Updating files: 66% (135/204) Updating files: 67% (137/204) Updating files: 68% (139/204) Updating files: 69% (141/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 92% (189/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2024-03-28T19:24:18.258 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 8a7c95f60ad v0.22 2024-03-28T19:24:18.279 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.22 2024-03-28T19:24:18.354 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:24:18.354 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 23 -eq 48 ']' 2024-03-28T19:24:18.354 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.23 2024-03-28T19:24:18.354 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.23 2024-03-28T19:24:18.355 INFO:tasks.workunit.client.0.smithi027.stdout:v0.23 2024-03-28T19:24:18.355 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.23 2024-03-28T19:24:20.820 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 39% (86/217) Updating files: 40% (87/217) Updating files: 41% (89/217) Updating files: 42% (92/217) Updating files: 43% (94/217) Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 60% (132/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2024-03-28T19:24:21.020 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at dac3a8c35fb v0.17 2024-03-28T19:24:21.028 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.17 2024-03-28T19:24:21.100 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:24:21.100 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 18 -eq 48 ']' 2024-03-28T19:24:21.100 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.18 2024-03-28T19:24:21.100 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.18 2024-03-28T19:24:21.101 INFO:tasks.workunit.client.1.smithi155.stdout:v0.18 2024-03-28T19:24:21.101 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.18 2024-03-28T19:24:23.419 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 33% (54/163) Updating files: 34% (56/163) Updating files: 35% (58/163) Updating files: 36% (59/163) Updating files: 37% (61/163) Updating files: 38% (62/163) Updating files: 39% (64/163) Updating files: 40% (66/163) Updating files: 41% (67/163) Updating files: 42% (69/163) Updating files: 43% (71/163) Updating files: 44% (72/163) Updating files: 45% (74/163) Updating files: 46% (75/163) Updating files: 47% (77/163) Updating files: 48% (79/163) Updating files: 49% (80/163) Updating files: 50% (82/163) Updating files: 51% (84/163) Updating files: 52% (85/163) Updating files: 53% (87/163) Updating files: 54% (89/163) Updating files: 55% (90/163) Updating files: 56% (92/163) Updating files: 57% (93/163) Updating files: 58% (95/163) Updating files: 59% (97/163) Updating files: 60% (98/163) Updating files: 61% (100/163) Updating files: 62% (102/163) Updating files: 63% (103/163) Updating files: 64% (105/163) Updating files: 65% (106/163) Updating files: 66% (108/163) Updating files: 67% (110/163) Updating files: 68% (111/163) Updating files: 69% (113/163) Updating files: 70% (115/163) Updating files: 71% (116/163) Updating files: 72% (118/163) Updating files: 73% (119/163) Updating files: 74% (121/163) Updating files: 75% (123/163) Updating files: 76% (124/163) Updating files: 77% (126/163) Updating files: 78% (128/163) Updating files: 79% (129/163) Updating files: 80% (131/163) Updating files: 81% (133/163) Updating files: 82% (134/163) Updating files: 83% (136/163) Updating files: 84% (137/163) Updating files: 85% (139/163) Updating files: 86% (141/163) Updating files: 87% (142/163) Updating files: 88% (144/163) Updating files: 89% (146/163) Updating files: 90% (147/163) Updating files: 91% (149/163) Updating files: 92% (150/163) Updating files: 93% (152/163) Updating files: 94% (154/163) Updating files: 95% (155/163) Updating files: 96% (157/163) Updating files: 97% (159/163) Updating files: 98% (160/163) Updating files: 99% (162/163) Updating files: 100% (163/163) Updating files: 100% (163/163), done. 2024-03-28T19:24:23.695 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 5d1d8d0c460 v0.23 2024-03-28T19:24:23.702 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.23 2024-03-28T19:24:23.738 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:24:23.738 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 24 -eq 48 ']' 2024-03-28T19:24:23.739 INFO:tasks.workunit.client.0.smithi027.stdout:v0.24 2024-03-28T19:24:23.739 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.24 2024-03-28T19:24:23.739 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.24 2024-03-28T19:24:23.739 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.24 2024-03-28T19:24:29.559 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 23% (44/185) Updating files: 24% (45/185) Updating files: 25% (47/185) Updating files: 26% (49/185) Updating files: 27% (50/185) Updating files: 28% (52/185) Updating files: 29% (54/185) Updating files: 30% (56/185) Updating files: 31% (58/185) Updating files: 32% (60/185) Updating files: 33% (62/185) Updating files: 34% (63/185) Updating files: 35% (65/185) Updating files: 36% (67/185) Updating files: 37% (69/185) Updating files: 38% (71/185) Updating files: 39% (73/185) Updating files: 40% (74/185) Updating files: 41% (76/185) Updating files: 42% (78/185) Updating files: 43% (80/185) Updating files: 44% (82/185) Updating files: 45% (84/185) Updating files: 46% (86/185) Updating files: 47% (87/185) Updating files: 48% (89/185) Updating files: 48% (90/185) Updating files: 49% (91/185) Updating files: 50% (93/185) Updating files: 51% (95/185) Updating files: 52% (97/185) Updating files: 53% (99/185) Updating files: 54% (100/185) Updating files: 55% (102/185) Updating files: 56% (104/185) Updating files: 57% (106/185) Updating files: 58% (108/185) Updating files: 59% (110/185) Updating files: 60% (111/185) Updating files: 61% (113/185) Updating files: 62% (115/185) Updating files: 63% (117/185) Updating files: 64% (119/185) Updating files: 65% (121/185) Updating files: 66% (123/185) Updating files: 67% (124/185) Updating files: 68% (126/185) Updating files: 69% (128/185) Updating files: 70% (130/185) Updating files: 71% (132/185) Updating files: 72% (134/185) Updating files: 72% (135/185) Updating files: 73% (136/185) Updating files: 74% (137/185) Updating files: 75% (139/185) Updating files: 76% (141/185) Updating files: 77% (143/185) Updating files: 78% (145/185) Updating files: 79% (147/185) Updating files: 80% (148/185) Updating files: 81% (150/185) Updating files: 82% (152/185) Updating files: 83% (154/185) Updating files: 84% (156/185) Updating files: 85% (158/185) Updating files: 86% (160/185) Updating files: 87% (161/185) Updating files: 88% (163/185) Updating files: 89% (165/185) Updating files: 90% (167/185) Updating files: 91% (169/185) Updating files: 92% (171/185) Updating files: 93% (173/185) Updating files: 94% (174/185) Updating files: 95% (176/185) Updating files: 96% (178/185) Updating files: 97% (180/185) Updating files: 98% (182/185) Updating files: 98% (183/185) Updating files: 99% (184/185) Updating files: 100% (185/185) Updating files: 100% (185/185), done. 2024-03-28T19:24:29.850 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 180a4176035 v0.24 2024-03-28T19:24:29.857 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.24 2024-03-28T19:24:29.903 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:24:29.903 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 25 -eq 48 ']' 2024-03-28T19:24:29.903 INFO:tasks.workunit.client.0.smithi027.stdout:v0.25 2024-03-28T19:24:29.904 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.25 2024-03-28T19:24:29.904 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.25 2024-03-28T19:24:29.904 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.25 2024-03-28T19:24:33.808 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 11% (88/779) Updating files: 12% (94/779) Updating files: 13% (102/779) Updating files: 14% (110/779) Updating files: 15% (117/779) Updating files: 16% (125/779) Updating files: 17% (133/779) Updating files: 18% (141/779) Updating files: 19% (149/779) Updating files: 20% (156/779) Updating files: 21% (164/779) Updating files: 22% (172/779) Updating files: 23% (180/779) Updating files: 24% (187/779) Updating files: 25% (195/779) Updating files: 26% (203/779) Updating files: 27% (211/779) Updating files: 27% (216/779) Updating files: 28% (219/779) Updating files: 29% (226/779) Updating files: 30% (234/779) Updating files: 31% (242/779) Updating files: 32% (250/779) Updating files: 33% (258/779) Updating files: 34% (265/779) Updating files: 35% (273/779) Updating files: 36% (281/779) Updating files: 37% (289/779) Updating files: 38% (297/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 46% (365/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 56% (442/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 65% (512/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 75% (585/779) Updating files: 75% (589/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 81% (635/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 87% (680/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2024-03-28T19:24:34.050 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 2f10137986a v0.18 2024-03-28T19:24:34.057 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.18 2024-03-28T19:24:34.103 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:24:34.103 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 19 -eq 48 ']' 2024-03-28T19:24:34.103 INFO:tasks.workunit.client.1.smithi155.stdout:v0.19 2024-03-28T19:24:34.103 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.19 2024-03-28T19:24:34.103 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.19 2024-03-28T19:24:34.103 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.19 2024-03-28T19:24:40.233 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 27% (59/217) Updating files: 28% (61/217) Updating files: 29% (63/217) Updating files: 30% (66/217) Updating files: 31% (68/217) Updating files: 32% (70/217) Updating files: 33% (72/217) Updating files: 34% (74/217) Updating files: 35% (76/217) Updating files: 36% (79/217) Updating files: 37% (81/217) Updating files: 38% (83/217) Updating files: 39% (85/217) Updating files: 40% (87/217) Updating files: 41% (89/217) Updating files: 42% (92/217) Updating files: 43% (94/217) Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 62% (136/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 83% (182/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2024-03-28T19:24:40.527 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 98f5be53f85 v0.19 2024-03-28T19:24:40.539 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.19 2024-03-28T19:24:40.580 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:24:40.580 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 20 -eq 48 ']' 2024-03-28T19:24:40.581 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.20 2024-03-28T19:24:40.581 INFO:tasks.workunit.client.1.smithi155.stdout:v0.20 2024-03-28T19:24:40.581 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.20 2024-03-28T19:24:40.581 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.20 2024-03-28T19:24:40.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:40.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.877+0000 7f159de00640 1 Processor -- start 2024-03-28T19:24:40.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.878+0000 7f159de00640 1 -- start start 2024-03-28T19:24:40.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.880+0000 7f159de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f159806e6f0 0x7f159806eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:40.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.880+0000 7f159de00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598159bc0 0x7f1598159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:40.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.880+0000 7f159de00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980c3830 con 0x7f1598153ed0 2024-03-28T19:24:40.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.880+0000 7f159de00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15980c6990 con 0x7f159806e6f0 2024-03-28T19:24:40.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.880+0000 7f159de00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15980c66b0 con 0x7f1598159bc0 2024-03-28T19:24:40.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.880+0000 7f1596a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1598153ed0 0x7f159806ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51926/0 (socket says 172.21.15.27:51926) 2024-03-28T19:24:40.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.880+0000 7f1596a00640 1 -- 172.21.15.27:0/1131082475 learned_addr learned my addr 172.21.15.27:0/1131082475 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:24:40.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.880+0000 7f1596000640 1 --2- 172.21.15.27:0/1131082475 >> v2:172.21.15.27:3300/0 conn(0x7f159806e6f0 0x7f159806eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.881+0000 7f1596000640 1 -- 172.21.15.27:0/1131082475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598159bc0 msgr2=0x7f1598159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.881+0000 7f1596000640 1 --2- 172.21.15.27:0/1131082475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598159bc0 0x7f1598159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.881+0000 7f1596000640 1 -- 172.21.15.27:0/1131082475 >> v1:172.21.15.27:6789/0 conn(0x7f1598153ed0 legacy=0x7f159806ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:40.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.881+0000 7f1596000640 1 -- 172.21.15.27:0/1131082475 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15980c3ad0 con 0x7f159806e6f0 2024-03-28T19:24:40.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.883+0000 7f1596000640 1 --2- 172.21.15.27:0/1131082475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f159806eb80 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f15980c3a00 tx=0x7f158402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907411aa269978db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:40.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.883+0000 7f1595600640 1 -- 172.21.15.27:0/1131082475 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584035020 con 0x7f159806e6f0 2024-03-28T19:24:40.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.884+0000 7f1595600640 1 -- 172.21.15.27:0/1131082475 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1584031810 con 0x7f159806e6f0 2024-03-28T19:24:40.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.884+0000 7f1595600640 1 -- 172.21.15.27:0/1131082475 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584031be0 con 0x7f159806e6f0 2024-03-28T19:24:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.884+0000 7f159de00640 1 -- 172.21.15.27:0/1131082475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 msgr2=0x7f159806eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.884+0000 7f159de00640 1 --2- 172.21.15.27:0/1131082475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f159806eb80 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f15980c3a00 tx=0x7f158402b8b0 comp rx=0 tx=0).stop 2024-03-28T19:24:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.885+0000 7f159de00640 1 -- 172.21.15.27:0/1131082475 shutdown_connections 2024-03-28T19:24:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.885+0000 7f159de00640 1 --2- 172.21.15.27:0/1131082475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598159bc0 0x7f1598159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.885+0000 7f159de00640 1 --2- 172.21.15.27:0/1131082475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f159806eb80 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.885+0000 7f159de00640 1 -- 172.21.15.27:0/1131082475 >> 172.21.15.27:0/1131082475 conn(0x7f159806b9d0 msgr2=0x7f15980c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:40.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.885+0000 7f159de00640 1 -- 172.21.15.27:0/1131082475 shutdown_connections 2024-03-28T19:24:40.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.885+0000 7f159de00640 1 -- 172.21.15.27:0/1131082475 wait complete. 2024-03-28T19:24:40.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.888+0000 7f159de00640 1 Processor -- start 2024-03-28T19:24:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.888+0000 7f159de00640 1 -- start start 2024-03-28T19:24:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.889+0000 7f159de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f15981f08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.889+0000 7f159de00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598153ed0 0x7f159815d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.889+0000 7f159de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1598159bc0 0x7f159815d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.889+0000 7f159de00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15980c1160 con 0x7f159806e6f0 2024-03-28T19:24:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.889+0000 7f159de00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15980c6380 con 0x7f1598159bc0 2024-03-28T19:24:40.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.889+0000 7f159de00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15980c12e0 con 0x7f1598153ed0 2024-03-28T19:24:40.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.890+0000 7f1596a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f15981f08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:40.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.890+0000 7f1596a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f15981f08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37414/0 (socket says 172.21.15.27:37414) 2024-03-28T19:24:40.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.890+0000 7f1596a00640 1 -- 172.21.15.27:0/3473449112 learned_addr learned my addr 172.21.15.27:0/3473449112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.890+0000 7f1596000640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598153ed0 0x7f159815d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:40.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.891+0000 7f1597400640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1598159bc0 0x7f159815d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.891+0000 7f1596a00640 1 -- 172.21.15.27:0/3473449112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598153ed0 msgr2=0x7f159815d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.891+0000 7f1596a00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598153ed0 0x7f159815d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.891+0000 7f1596a00640 1 -- 172.21.15.27:0/3473449112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1598159bc0 msgr2=0x7f159815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.891+0000 7f1596a00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1598159bc0 0x7f159815d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.891+0000 7f1596a00640 1 -- 172.21.15.27:0/3473449112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15980a3250 con 0x7f159806e6f0 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.892+0000 7f1596a00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f15981f08d0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f158800c400 tx=0x7f158800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.892+0000 7f158f400640 1 -- 172.21.15.27:0/3473449112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1588017070 con 0x7f159806e6f0 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.892+0000 7f158f400640 1 -- 172.21.15.27:0/3473449112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1588005ce0 con 0x7f159806e6f0 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.892+0000 7f158f400640 1 -- 172.21.15.27:0/3473449112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15880072f0 con 0x7f159806e6f0 2024-03-28T19:24:40.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.892+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15980a3530 con 0x7f159806e6f0 2024-03-28T19:24:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.892+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159815cca0 con 0x7f159806e6f0 2024-03-28T19:24:40.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.895+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159804f400 con 0x7f159806e6f0 2024-03-28T19:24:40.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.897+0000 7f158f400640 1 -- 172.21.15.27:0/3473449112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f158801a070 con 0x7f159806e6f0 2024-03-28T19:24:40.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.898+0000 7f158f400640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f157806bbd0 0x7f157806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.898+0000 7f158f400640 1 -- 172.21.15.27:0/3473449112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f158808ae30 con 0x7f159806e6f0 2024-03-28T19:24:40.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.899+0000 7f1596000640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f157806bbd0 0x7f157806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:40.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.900+0000 7f1596000640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f157806bbd0 0x7f157806e090 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1584036000 tx=0x7f1584008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:40.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:40.905+0000 7f158f400640 1 -- 172.21.15.27:0/3473449112 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1588058240 con 0x7f159806e6f0 2024-03-28T19:24:41.018 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 11% (70/622) Updating files: 12% (75/622) Updating files: 13% (81/622) Updating files: 14% (88/622) Updating files: 15% (94/622) Updating files: 16% (100/622) Updating files: 17% (106/622) Updating files: 18% (112/622) Updating files: 19% (119/622) Updating files: 19% (124/622) Updating files: 20% (125/622) Updating files: 21% (131/622) Updating files: 22% (137/622) Updating files: 23% (144/622) Updating files: 24% (150/622) Updating files: 25% (156/622) Updating files: 26% (162/622) Updating files: 27% (168/622) Updating files: 28% (175/622) Updating files: 29% (181/622) Updating files: 30% (187/622) Updating files: 31% (193/622) Updating files: 32% (200/622) Updating files: 33% (206/622) Updating files: 34% (212/622) Updating files: 35% (218/622) Updating files: 36% (224/622) Updating files: 37% (231/622) Updating files: 37% (235/622) Updating files: 38% (237/622) Updating files: 39% (243/622) Updating files: 40% (249/622) Updating files: 41% (256/622) Updating files: 42% (262/622) Updating files: 43% (268/622) Updating files: 44% (274/622) Updating files: 45% (280/622) Updating files: 46% (287/622) Updating files: 47% (293/622) Updating files: 48% (299/622) Updating files: 49% (305/622) Updating files: 50% (311/622) Updating files: 50% (314/622) Updating files: 51% (318/622) Updating files: 52% (324/622) Updating files: 53% (330/622) Updating files: 54% (336/622) Updating files: 55% (343/622) Updating files: 56% (349/622) Updating files: 57% (355/622) Updating files: 58% (361/622) Updating files: 59% (367/622) Updating files: 60% (374/622) Updating files: 61% (380/622) Updating files: 62% (386/622) Updating files: 62% (391/622) Updating files: 63% (392/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 67% (417/622) Updating files: 68% (423/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 70% (441/622) Updating files: 71% (442/622) Updating files: 72% (448/622) Updating files: 73% (455/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 78% (486/622) Updating files: 79% (492/622) Updating files: 79% (494/622) Updating files: 80% (498/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 88% (551/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2024-03-28T19:24:41.240 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 3fa121bf0a8 v0.25 2024-03-28T19:24:41.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.248+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15980ac190 con 0x7f159806e6f0 2024-03-28T19:24:41.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.248+0000 7f158f400640 1 -- 172.21.15.27:0/3473449112 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f158805c2e0 con 0x7f159806e6f0 2024-03-28T19:24:41.248 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:41.248 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:24:41.248 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 216 2024-03-28T19:24:41.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f157806bbd0 msgr2=0x7f157806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f157806bbd0 0x7f157806e090 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f1584036000 tx=0x7f1584008030 comp rx=0 tx=0).stop 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 msgr2=0x7f15981f08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f15981f08d0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f158800c400 tx=0x7f158800c8c0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 shutdown_connections 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f157806bbd0 0x7f157806e090 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1598159bc0 0x7f159815d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1598153ed0 0x7f159815d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 --2- 172.21.15.27:0/3473449112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f159806e6f0 0x7f15981f08d0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 >> 172.21.15.27:0/3473449112 conn(0x7f159806b9d0 msgr2=0x7f15981579c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 shutdown_connections 2024-03-28T19:24:41.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.251+0000 7f159de00640 1 -- 172.21.15.27:0/3473449112 wait complete. 2024-03-28T19:24:41.253 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.25 2024-03-28T19:24:41.269 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-03-28T19:24:41.270 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:24:41.270 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:24:41.309 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:24:41.310 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 26 -eq 48 ']' 2024-03-28T19:24:41.310 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.26 2024-03-28T19:24:41.310 INFO:tasks.workunit.client.0.smithi027.stdout:v0.26 2024-03-28T19:24:41.311 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.26 2024-03-28T19:24:41.311 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.26 2024-03-28T19:24:41.333 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:24:41.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.536+0000 7febaca00640 1 Processor -- start 2024-03-28T19:24:41.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.537+0000 7febaca00640 1 -- start start 2024-03-28T19:24:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.538+0000 7febaca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feba8150b60 0x7feba80ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.538+0000 7febaca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7feba80c8470 0x7feba8156b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.538+0000 7febaca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba80ad680 con 0x7feba80c8c70 2024-03-28T19:24:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.538+0000 7febaca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba80ad9b0 con 0x7feba80c8470 2024-03-28T19:24:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.538+0000 7febaca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba80c4de0 con 0x7feba8150b60 2024-03-28T19:24:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.538+0000 7feba6800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feba80c8c70 0x7feba80c3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51938/0 (socket says 172.21.15.27:51938) 2024-03-28T19:24:41.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.538+0000 7feba6800640 1 -- 172.21.15.27:0/3650162542 learned_addr learned my addr 172.21.15.27:0/3650162542 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:24:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.539+0000 7feba5400640 1 --2- 172.21.15.27:0/3650162542 >> v2:172.21.15.155:3300/0 conn(0x7feba80c8470 0x7feba8156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.539+0000 7feba5e00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feba8150b60 0x7feba80ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.539+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642299398 0 0) 0x7feba80ad680 con 0x7feba80c8c70 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.539+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8c003520 con 0x7feba80c8c70 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.539+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262437654 0 0) 0x7feb8c003520 con 0x7feba80c8c70 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.539+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feba8150b60 msgr2=0x7feba80ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.539+0000 7feba4a00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feba8150b60 0x7feba80ade40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 >> v2:172.21.15.155:3300/0 conn(0x7feba80c8470 msgr2=0x7feba8156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 --2- 172.21.15.27:0/3650162542 >> v2:172.21.15.155:3300/0 conn(0x7feba80c8470 0x7feba8156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba80c50c0 con 0x7feba80c8c70 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7feba00019f0 con 0x7feba80c8c70 2024-03-28T19:24:41.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 >> v1:172.21.15.27:6789/0 conn(0x7feba80c8c70 legacy=0x7feba80c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb8c0039b0 0x7feb8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feba8150b60 0x7feb8c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c004d40 0x7feb8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba80ad9b0 con 0x7feb8c0039b0 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba80c4de0 con 0x7feb8c004d40 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c00b9c0 con 0x7feba8150b60 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba5e00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb8c0039b0 0x7feb8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.540+0000 7feba6800640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c004d40 0x7feb8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.541+0000 7feba5400640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feba8150b60 0x7feb8c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.541+0000 7feba5e00640 1 -- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feba8150b60 msgr2=0x7feb8c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.541+0000 7feba5e00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feba8150b60 0x7feb8c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.541+0000 7feba5e00640 1 -- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c004d40 msgr2=0x7feb8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.541+0000 7feba5e00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c004d40 0x7feb8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.541+0000 7feba5e00640 1 -- 172.21.15.27:0/3650162542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba80c50c0 con 0x7feb8c0039b0 2024-03-28T19:24:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.542+0000 7feba5e00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb8c0039b0 0x7feb8c003dd0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7feb98009ac0 tx=0x7feb8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369da1a27349eab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.543+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98039070 con 0x7feb8c0039b0 2024-03-28T19:24:41.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.543+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9802c6b0 con 0x7feb8c0039b0 2024-03-28T19:24:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.543+0000 7feba4a00640 1 -- 172.21.15.27:0/3650162542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98009690 con 0x7feb8c0039b0 2024-03-28T19:24:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.544+0000 7febaca00640 1 -- 172.21.15.27:0/3650162542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb8c0039b0 msgr2=0x7feb8c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.544+0000 7febaca00640 1 --2- 172.21.15.27:0/3650162542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb8c0039b0 0x7feb8c003dd0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7feb98009ac0 tx=0x7feb8c005830 comp rx=0 tx=0).stop 2024-03-28T19:24:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.544+0000 7feba6800640 1 -- 172.21.15.27:0/3650162542 reap_dead start 2024-03-28T19:24:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.544+0000 7febaca00640 1 -- 172.21.15.27:0/3650162542 shutdown_connections 2024-03-28T19:24:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.544+0000 7febaca00640 1 -- 172.21.15.27:0/3650162542 >> 172.21.15.27:0/3650162542 conn(0x7feba80aaab0 msgr2=0x7feba80aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.544+0000 7febaca00640 1 -- 172.21.15.27:0/3650162542 shutdown_connections 2024-03-28T19:24:41.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.544+0000 7febaca00640 1 -- 172.21.15.27:0/3650162542 wait complete. 2024-03-28T19:24:41.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.547+0000 7febaca00640 1 Processor -- start 2024-03-28T19:24:41.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.547+0000 7febaca00640 1 -- start start 2024-03-28T19:24:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.548+0000 7febaca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0039b0 0x7feba8153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.548+0000 7febaca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feba81e0550 0x7feba8154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.548+0000 7febaca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feba81510c0 0x7feba8151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.548+0000 7febaca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba80a85a0 con 0x7feba81e0550 2024-03-28T19:24:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.548+0000 7febaca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba8150f40 con 0x7feba81510c0 2024-03-28T19:24:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.548+0000 7febaca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba80a8720 con 0x7feb8c0039b0 2024-03-28T19:24:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feba81e0550 0x7feba8154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feba81e0550 0x7feba8154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37440/0 (socket says 172.21.15.27:37440) 2024-03-28T19:24:41.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5400640 1 -- 172.21.15.27:0/1121987952 learned_addr learned my addr 172.21.15.27:0/1121987952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:41.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba6800640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feba81510c0 0x7feba8151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5e00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0039b0 0x7feba8153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5400640 1 -- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0039b0 msgr2=0x7feba8153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:41.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5400640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0039b0 0x7feba8153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5400640 1 -- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feba81510c0 msgr2=0x7feba8151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:41.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5400640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feba81510c0 0x7feba8151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba5400640 1 -- 172.21.15.27:0/1121987952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba81536e0 con 0x7feba81e0550 2024-03-28T19:24:41.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.549+0000 7feba6800640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feba81510c0 0x7feba8151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:24:41.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.550+0000 7feba5400640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feba81e0550 0x7feba8154040 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7feb940116d0 tx=0x7feb94011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:41.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.550+0000 7feb9e000640 1 -- 172.21.15.27:0/1121987952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9400f960 con 0x7feba81e0550 2024-03-28T19:24:41.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.550+0000 7feb9e000640 1 -- 172.21.15.27:0/1121987952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9400fb00 con 0x7feba81e0550 2024-03-28T19:24:41.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.550+0000 7feb9e000640 1 -- 172.21.15.27:0/1121987952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94015460 con 0x7feba81e0550 2024-03-28T19:24:41.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.550+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba81571d0 con 0x7feba81e0550 2024-03-28T19:24:41.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.551+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba8157490 con 0x7feba81e0550 2024-03-28T19:24:41.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.552+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba804f400 con 0x7feba81e0550 2024-03-28T19:24:41.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.555+0000 7feb9e000640 1 -- 172.21.15.27:0/1121987952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feb94015600 con 0x7feba81e0550 2024-03-28T19:24:41.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.556+0000 7feb9e000640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb8806bc20 0x7feb8806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:41.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.556+0000 7feb9e000640 1 -- 172.21.15.27:0/1121987952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7feb9408c560 con 0x7feba81e0550 2024-03-28T19:24:41.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.557+0000 7feba5e00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb8806bc20 0x7feb8806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:41.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.558+0000 7feba5e00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb8806bc20 0x7feb8806e0e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7feb98036000 tx=0x7feb98030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:41.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.562+0000 7feb9e000640 1 -- 172.21.15.27:0/1121987952 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feb94019030 con 0x7feba81e0550 2024-03-28T19:24:41.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.920+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feba80c4a20 con 0x7feba81e0550 2024-03-28T19:24:41.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.921+0000 7feb9e000640 1 -- 172.21.15.27:0/1121987952 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+6264 (secure 0 0 0) 0x7feb9405a040 con 0x7feba81e0550 2024-03-28T19:24:41.920 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:41.920 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:24:41.920 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 216 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.923+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb8806bc20 msgr2=0x7feb8806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.923+0000 7febaca00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb8806bc20 0x7feb8806e0e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7feb98036000 tx=0x7feb98030000 comp rx=0 tx=0).stop 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.923+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feba81e0550 msgr2=0x7feba8154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.923+0000 7febaca00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feba81e0550 0x7feba8154040 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7feb940116d0 tx=0x7feb94011b90 comp rx=0 tx=0).stop 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.924+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 shutdown_connections 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.924+0000 7febaca00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7feb8806bc20 0x7feb8806e0e0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.924+0000 7febaca00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feba81510c0 0x7feba8151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.924+0000 7febaca00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feba81e0550 0x7feba8154040 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.924+0000 7febaca00640 1 --2- 172.21.15.27:0/1121987952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0039b0 0x7feba8153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.924+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 >> 172.21.15.27:0/1121987952 conn(0x7feba80aaab0 msgr2=0x7feba81503d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:41.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.924+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 shutdown_connections 2024-03-28T19:24:41.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:41.924+0000 7febaca00640 1 -- 172.21.15.27:0/1121987952 wait complete. 2024-03-28T19:24:41.941 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:24:43.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.180+0000 7f6d15400640 1 Processor -- start 2024-03-28T19:24:44.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.180+0000 7f6d15400640 1 -- start start 2024-03-28T19:24:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.182+0000 7f6d15400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d100c7970 0x7f6d100c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.182+0000 7f6d15400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d10155730 0x7f6d10155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:44.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.182+0000 7f6d15400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d100c7310 con 0x7f6d10152f80 2024-03-28T19:24:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.182+0000 7f6d15400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d10158d40 con 0x7f6d10155730 2024-03-28T19:24:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.182+0000 7f6d15400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d10159020 con 0x7f6d100c7970 2024-03-28T19:24:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.182+0000 7f6d0e800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6d10152f80 0x7f6d100c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51942/0 (socket says 172.21.15.27:51942) 2024-03-28T19:24:44.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.182+0000 7f6d0e800640 1 -- 172.21.15.27:0/2043387380 learned_addr learned my addr 172.21.15.27:0/2043387380 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:24:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.183+0000 7f6d0f200640 1 --2- 172.21.15.27:0/2043387380 >> v2:172.21.15.27:3300/0 conn(0x7f6d10155730 0x7f6d10155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.183+0000 7f6d0de00640 1 --2- 172.21.15.27:0/2043387380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d100c7970 0x7f6d100c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.183+0000 7f6d0d400640 1 -- 172.21.15.27:0/2043387380 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236436050 0 0) 0x7f6d100c7310 con 0x7f6d10152f80 2024-03-28T19:24:44.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.183+0000 7f6d0d400640 1 -- 172.21.15.27:0/2043387380 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf400bd30 con 0x7f6d10152f80 2024-03-28T19:24:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.184+0000 7f6d0f200640 1 -- 172.21.15.27:0/2043387380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d100c7970 msgr2=0x7f6d100c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.184+0000 7f6d0f200640 1 --2- 172.21.15.27:0/2043387380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d100c7970 0x7f6d100c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.184+0000 7f6d0d400640 1 -- 172.21.15.27:0/2043387380 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1561498469 0 0) 0x7f6cf400bd30 con 0x7f6d10152f80 2024-03-28T19:24:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.184+0000 7f6d0f200640 1 -- 172.21.15.27:0/2043387380 >> v1:172.21.15.27:6789/0 conn(0x7f6d10152f80 legacy=0x7f6d100c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:44.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.184+0000 7f6d0f200640 1 -- 172.21.15.27:0/2043387380 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10159300 con 0x7f6d10155730 2024-03-28T19:24:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.185+0000 7f6d0f200640 1 --2- 172.21.15.27:0/2043387380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d10155730 0x7f6d10155bc0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009d90 tx=0x7f6d0802bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5146f3a6cd323d3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.186+0000 7f6d0d400640 1 -- 172.21.15.27:0/2043387380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08039070 con 0x7f6d10155730 2024-03-28T19:24:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.186+0000 7f6d0d400640 1 -- 172.21.15.27:0/2043387380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0802c810 con 0x7f6d10155730 2024-03-28T19:24:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.186+0000 7f6d0d400640 1 -- 172.21.15.27:0/2043387380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00001670 con 0x7f6d10155730 2024-03-28T19:24:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.187+0000 7f6d15400640 1 -- 172.21.15.27:0/2043387380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d10155730 msgr2=0x7f6d10155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.187+0000 7f6d15400640 1 --2- 172.21.15.27:0/2043387380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d10155730 0x7f6d10155bc0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009d90 tx=0x7f6d0802bce0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.187+0000 7f6d15400640 1 -- 172.21.15.27:0/2043387380 shutdown_connections 2024-03-28T19:24:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.187+0000 7f6d15400640 1 --2- 172.21.15.27:0/2043387380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d10155730 0x7f6d10155bc0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.187+0000 7f6d15400640 1 --2- 172.21.15.27:0/2043387380 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d100c7970 0x7f6d100c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.187+0000 7f6d15400640 1 -- 172.21.15.27:0/2043387380 >> 172.21.15.27:0/2043387380 conn(0x7f6d1006b6f0 msgr2=0x7f6d100caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.187+0000 7f6d15400640 1 -- 172.21.15.27:0/2043387380 shutdown_connections 2024-03-28T19:24:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.187+0000 7f6d15400640 1 -- 172.21.15.27:0/2043387380 wait complete. 2024-03-28T19:24:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.189+0000 7f6d15400640 1 Processor -- start 2024-03-28T19:24:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.189+0000 7f6d15400640 1 -- start start 2024-03-28T19:24:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.190+0000 7f6d15400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d100c7970 0x7f6d100c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.190+0000 7f6d15400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d10152f80 0x7f6d10154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.190+0000 7f6d15400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d10155730 0x7f6d10149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.190+0000 7f6d15400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100cbca0 con 0x7f6d100c7970 2024-03-28T19:24:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.190+0000 7f6d15400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d10159b00 con 0x7f6d10155730 2024-03-28T19:24:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.190+0000 7f6d15400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d10148dc0 con 0x7f6d10152f80 2024-03-28T19:24:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d100c7970 0x7f6d100c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d100c7970 0x7f6d100c3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37456/0 (socket says 172.21.15.27:37456) 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0e800640 1 -- 172.21.15.27:0/2888209913 learned_addr learned my addr 172.21.15.27:0/2888209913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0de00640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d10152f80 0x7f6d10154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0f200640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d10155730 0x7f6d10149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0e800640 1 -- 172.21.15.27:0/2888209913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d10152f80 msgr2=0x7f6d10154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0e800640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d10152f80 0x7f6d10154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0e800640 1 -- 172.21.15.27:0/2888209913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d10155730 msgr2=0x7f6d10149250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0e800640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d10155730 0x7f6d10149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.191+0000 7f6d0e800640 1 -- 172.21.15.27:0/2888209913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d100c2360 con 0x7f6d100c7970 2024-03-28T19:24:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.192+0000 7f6d0e800640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d100c7970 0x7f6d100c3ff0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000c700 tx=0x7f6d0000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.192+0000 7f6d06a00640 1 -- 172.21.15.27:0/2888209913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00017070 con 0x7f6d100c7970 2024-03-28T19:24:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.192+0000 7f6d06a00640 1 -- 172.21.15.27:0/2888209913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d00005ad0 con 0x7f6d100c7970 2024-03-28T19:24:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.192+0000 7f6d06a00640 1 -- 172.21.15.27:0/2888209913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00007070 con 0x7f6d100c7970 2024-03-28T19:24:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.192+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d100c2640 con 0x7f6d100c7970 2024-03-28T19:24:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.193+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d10148a50 con 0x7f6d100c7970 2024-03-28T19:24:44.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.194+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1004f400 con 0x7f6d100c7970 2024-03-28T19:24:44.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.196+0000 7f6d06a00640 1 -- 172.21.15.27:0/2888209913 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6d0001a070 con 0x7f6d100c7970 2024-03-28T19:24:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.197+0000 7f6d06a00640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cf006bc50 0x7f6cf006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.197+0000 7f6d06a00640 1 -- 172.21.15.27:0/2888209913 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f6d0008af10 con 0x7f6d100c7970 2024-03-28T19:24:44.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.198+0000 7f6d0de00640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cf006bc50 0x7f6cf006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:44.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.199+0000 7f6d0de00640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cf006bc50 0x7f6cf006e110 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6d101485f0 tx=0x7f6cfc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:44.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.203+0000 7f6d06a00640 1 -- 172.21.15.27:0/2888209913 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6d00058a50 con 0x7f6d100c7970 2024-03-28T19:24:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.552+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d100cba80 con 0x7f6d100c7970 2024-03-28T19:24:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.553+0000 7f6d06a00640 1 -- 172.21.15.27:0/2888209913 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f6d00058350 con 0x7f6d100c7970 2024-03-28T19:24:44.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:44.552 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:24:44.552 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 216 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cf006bc50 msgr2=0x7f6cf006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cf006bc50 0x7f6cf006e110 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6d101485f0 tx=0x7f6cfc008040 comp rx=0 tx=0).stop 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d100c7970 msgr2=0x7f6d100c3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d100c7970 0x7f6d100c3ff0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000c700 tx=0x7f6d0000cbc0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 shutdown_connections 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6cf006bc50 0x7f6cf006e110 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d10155730 0x7f6d10149250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d10152f80 0x7f6d10154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 --2- 172.21.15.27:0/2888209913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d100c7970 0x7f6d100c3ff0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 >> 172.21.15.27:0/2888209913 conn(0x7f6d1006b6f0 msgr2=0x7f6d101512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 shutdown_connections 2024-03-28T19:24:44.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:44.556+0000 7f6d15400640 1 -- 172.21.15.27:0/2888209913 wait complete. 2024-03-28T19:24:44.575 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:24:46.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.805+0000 7f1e87800640 1 Processor -- start 2024-03-28T19:24:46.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.805+0000 7f1e87800640 1 -- start start 2024-03-28T19:24:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e87800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 0x7f1e800a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e87800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e801534c0 0x7f1e80153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e87800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e800c3af0 con 0x7f1e800c3690 2024-03-28T19:24:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e87800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e800ab4e0 con 0x7f1e801534c0 2024-03-28T19:24:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e87800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800ac1f0 con 0x7f1e80150c70 2024-03-28T19:24:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e85800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e801534c0 0x7f1e80153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:46.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e85800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e801534c0 0x7f1e80153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51332/0 (socket says 172.21.15.27:51332) 2024-03-28T19:24:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e85800640 1 -- 172.21.15.27:0/1890298027 learned_addr learned my addr 172.21.15.27:0/1890298027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.807+0000 7f1e84e00640 1 --2- 172.21.15.27:0/1890298027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 0x7f1e800a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.808+0000 7f1e85800640 1 -- 172.21.15.27:0/1890298027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 msgr2=0x7f1e800a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.808+0000 7f1e85800640 1 --2- 172.21.15.27:0/1890298027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 0x7f1e800a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.808+0000 7f1e85800640 1 -- 172.21.15.27:0/1890298027 >> v1:172.21.15.155:6789/0 conn(0x7f1e800c3690 legacy=0x7f1e800c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:46.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.808+0000 7f1e85800640 1 -- 172.21.15.27:0/1890298027 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e800c3d90 con 0x7f1e801534c0 2024-03-28T19:24:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.809+0000 7f1e85800640 1 --2- 172.21.15.27:0/1890298027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e801534c0 0x7f1e80153950 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f1e78009e50 tx=0x7f1e7802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdecc8a691224548 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.809+0000 7f1e7f400640 1 -- 172.21.15.27:0/1890298027 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7802ca80 con 0x7f1e801534c0 2024-03-28T19:24:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.810+0000 7f1e7f400640 1 -- 172.21.15.27:0/1890298027 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7802cc20 con 0x7f1e801534c0 2024-03-28T19:24:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.810+0000 7f1e7f400640 1 -- 172.21.15.27:0/1890298027 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e780343f0 con 0x7f1e801534c0 2024-03-28T19:24:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.810+0000 7f1e87800640 1 -- 172.21.15.27:0/1890298027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e801534c0 msgr2=0x7f1e80153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.810+0000 7f1e87800640 1 --2- 172.21.15.27:0/1890298027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e801534c0 0x7f1e80153950 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f1e78009e50 tx=0x7f1e7802bee0 comp rx=0 tx=0).stop 2024-03-28T19:24:46.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.811+0000 7f1e87800640 1 -- 172.21.15.27:0/1890298027 shutdown_connections 2024-03-28T19:24:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.811+0000 7f1e87800640 1 --2- 172.21.15.27:0/1890298027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e801534c0 0x7f1e80153950 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.811+0000 7f1e87800640 1 --2- 172.21.15.27:0/1890298027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 0x7f1e800a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.811+0000 7f1e87800640 1 -- 172.21.15.27:0/1890298027 >> 172.21.15.27:0/1890298027 conn(0x7f1e8006b9d0 msgr2=0x7f1e800c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.811+0000 7f1e87800640 1 -- 172.21.15.27:0/1890298027 shutdown_connections 2024-03-28T19:24:46.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.811+0000 7f1e87800640 1 -- 172.21.15.27:0/1890298027 wait complete. 2024-03-28T19:24:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.813+0000 7f1e87800640 1 Processor -- start 2024-03-28T19:24:46.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.813+0000 7f1e87800640 1 -- start start 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.814+0000 7f1e87800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e800c3690 0x7f1e8014df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.814+0000 7f1e87800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 0x7f1e801eba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.814+0000 7f1e87800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e801534c0 0x7f1e801ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.814+0000 7f1e87800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800abcd0 con 0x7f1e800c3690 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.814+0000 7f1e87800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800ab9f0 con 0x7f1e801534c0 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.814+0000 7f1e87800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800c1810 con 0x7f1e80150c70 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.815+0000 7f1e84e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e800c3690 0x7f1e8014df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.815+0000 7f1e84e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e800c3690 0x7f1e8014df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51334/0 (socket says 172.21.15.27:51334) 2024-03-28T19:24:46.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.815+0000 7f1e84e00640 1 -- 172.21.15.27:0/1868363285 learned_addr learned my addr 172.21.15.27:0/1868363285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.815+0000 7f1e85800640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e801534c0 0x7f1e801ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.815+0000 7f1e7fe00640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 0x7f1e801eba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.816+0000 7f1e84e00640 1 -- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 msgr2=0x7f1e801eba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.816+0000 7f1e84e00640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 0x7f1e801eba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.816+0000 7f1e84e00640 1 -- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e801534c0 msgr2=0x7f1e801ebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.816+0000 7f1e84e00640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e801534c0 0x7f1e801ebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.816+0000 7f1e84e00640 1 -- 172.21.15.27:0/1868363285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8014c050 con 0x7f1e800c3690 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.816+0000 7f1e85800640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e801534c0 0x7f1e801ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:24:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.816+0000 7f1e84e00640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e800c3690 0x7f1e8014df50 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000d800 tx=0x7f1e7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:46.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.816+0000 7f1e7cc00640 1 -- 172.21.15.27:0/1868363285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000b6c0 con 0x7f1e800c3690 2024-03-28T19:24:46.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.817+0000 7f1e7cc00640 1 -- 172.21.15.27:0/1868363285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e70019070 con 0x7f1e800c3690 2024-03-28T19:24:46.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.817+0000 7f1e7cc00640 1 -- 172.21.15.27:0/1868363285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000ba80 con 0x7f1e800c3690 2024-03-28T19:24:46.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.817+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e801ef570 con 0x7f1e800c3690 2024-03-28T19:24:46.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.817+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e800717b0 con 0x7f1e800c3690 2024-03-28T19:24:46.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.818+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8004f400 con 0x7f1e800c3690 2024-03-28T19:24:46.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.819+0000 7f1e7cc00640 1 -- 172.21.15.27:0/1868363285 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1e7000bc20 con 0x7f1e800c3690 2024-03-28T19:24:46.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.820+0000 7f1e7cc00640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e6006bc20 0x7f1e6006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.820+0000 7f1e7cc00640 1 -- 172.21.15.27:0/1868363285 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f1e7008be10 con 0x7f1e800c3690 2024-03-28T19:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.820+0000 7f1e7fe00640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e6006bc20 0x7f1e6006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:46.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.821+0000 7f1e7fe00640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e6006bc20 0x7f1e6006e0e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1e8014bc10 tx=0x7f1e6c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:46.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:46.826+0000 7f1e7cc00640 1 -- 172.21.15.27:0/1868363285 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1e7008f030 con 0x7f1e800c3690 2024-03-28T19:24:47.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.171+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e800c5f00 con 0x7f1e800c3690 2024-03-28T19:24:47.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.172+0000 7f1e7cc00640 1 -- 172.21.15.27:0/1868363285 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f1e700592a0 con 0x7f1e800c3690 2024-03-28T19:24:47.171 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:47.172 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:24:47.172 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 216 2024-03-28T19:24:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.175+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e6006bc20 msgr2=0x7f1e6006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.175+0000 7f1e87800640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e6006bc20 0x7f1e6006e0e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1e8014bc10 tx=0x7f1e6c009040 comp rx=0 tx=0).stop 2024-03-28T19:24:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.175+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e800c3690 msgr2=0x7f1e8014df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.175+0000 7f1e87800640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e800c3690 0x7f1e8014df50 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000d800 tx=0x7f1e7000dcc0 comp rx=0 tx=0).stop 2024-03-28T19:24:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.176+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 shutdown_connections 2024-03-28T19:24:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.176+0000 7f1e87800640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f1e6006bc20 0x7f1e6006e0e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.176+0000 7f1e87800640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e801534c0 0x7f1e801ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.176+0000 7f1e87800640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e80150c70 0x7f1e801eba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.176+0000 7f1e87800640 1 --2- 172.21.15.27:0/1868363285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e800c3690 0x7f1e8014df50 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.176+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 >> 172.21.15.27:0/1868363285 conn(0x7f1e8006b9d0 msgr2=0x7f1e800a7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.176+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 shutdown_connections 2024-03-28T19:24:47.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:47.176+0000 7f1e87800640 1 -- 172.21.15.27:0/1868363285 wait complete. 2024-03-28T19:24:47.196 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:24:49.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:49.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.446+0000 7f472a400640 1 Processor -- start 2024-03-28T19:24:49.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.446+0000 7f472a400640 1 -- start start 2024-03-28T19:24:49.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.448+0000 7f472a400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4724152a70 0x7f4724159580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.448+0000 7f472a400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240c6cc0 0x7f47240cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.448+0000 7f472a400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240c5710 con 0x7f47240cb620 2024-03-28T19:24:49.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.448+0000 7f472a400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47241556b0 con 0x7f4724152a70 2024-03-28T19:24:49.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.448+0000 7f472a400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47240c59f0 con 0x7f47240c6cc0 2024-03-28T19:24:49.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.448+0000 7f4729a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f47240cb620 0x7f4724155b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44788/0 (socket says 172.21.15.27:44788) 2024-03-28T19:24:49.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.448+0000 7f4729a00640 1 -- 172.21.15.27:0/2133635457 learned_addr learned my addr 172.21.15.27:0/2133635457 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:24:49.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4729000640 1 --2- 172.21.15.27:0/2133635457 >> v2:172.21.15.155:3300/0 conn(0x7f4724152a70 0x7f4724159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:49.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723e00640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240c6cc0 0x7f47240cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:49.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801636584 0 0) 0x7f47240c5710 con 0x7f47240cb620 2024-03-28T19:24:49.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c003520 con 0x7f47240cb620 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2514171374 0 0) 0x7f470c003520 con 0x7f47240cb620 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240c6cc0 msgr2=0x7f47240cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240c6cc0 0x7f47240cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 >> v2:172.21.15.155:3300/0 conn(0x7f4724152a70 msgr2=0x7f4724159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 --2- 172.21.15.27:0/2133635457 >> v2:172.21.15.155:3300/0 conn(0x7f4724152a70 0x7f4724159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47240c5cd0 con 0x7f47240cb620 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f471c0019f0 con 0x7f47240cb620 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 >> v1:172.21.15.27:6789/0 conn(0x7f47240cb620 legacy=0x7f4724155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c0039b0 0x7f470c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c004800 0x7f470c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240c6cc0 0x7f470c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47241556b0 con 0x7f470c004800 2024-03-28T19:24:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47240c59f0 con 0x7f470c0039b0 2024-03-28T19:24:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.449+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c00bac0 con 0x7f47240c6cc0 2024-03-28T19:24:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.450+0000 7f4723e00640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c004800 0x7f470c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.450+0000 7f4723e00640 1 -- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240c6cc0 msgr2=0x7f470c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.450+0000 7f4723e00640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240c6cc0 0x7f470c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.450+0000 7f4723e00640 1 -- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c0039b0 msgr2=0x7f470c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.450+0000 7f4723e00640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c0039b0 0x7f470c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.450+0000 7f4723e00640 1 -- 172.21.15.27:0/2133635457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c003520 con 0x7f470c004800 2024-03-28T19:24:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.451+0000 7f4723e00640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c004800 0x7f470c004c80 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f471000b9e0 tx=0x7f470c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810d408495dac3b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.452+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710036020 con 0x7f470c004800 2024-03-28T19:24:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.452+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4710032640 con 0x7f470c004800 2024-03-28T19:24:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.452+0000 7f4723400640 1 -- 172.21.15.27:0/2133635457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471000b5e0 con 0x7f470c004800 2024-03-28T19:24:49.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.452+0000 7f472a400640 1 -- 172.21.15.27:0/2133635457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c004800 msgr2=0x7f470c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.452+0000 7f472a400640 1 --2- 172.21.15.27:0/2133635457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f470c004800 0x7f470c004c80 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f471000b9e0 tx=0x7f470c005830 comp rx=0 tx=0).stop 2024-03-28T19:24:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.452+0000 7f4729a00640 1 -- 172.21.15.27:0/2133635457 reap_dead start 2024-03-28T19:24:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.452+0000 7f472a400640 1 -- 172.21.15.27:0/2133635457 shutdown_connections 2024-03-28T19:24:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.452+0000 7f472a400640 1 -- 172.21.15.27:0/2133635457 >> 172.21.15.27:0/2133635457 conn(0x7f472406b2a0 msgr2=0x7f472406b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.453+0000 7f472a400640 1 -- 172.21.15.27:0/2133635457 shutdown_connections 2024-03-28T19:24:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.453+0000 7f472a400640 1 -- 172.21.15.27:0/2133635457 wait complete. 2024-03-28T19:24:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.455+0000 7f472a400640 1 Processor -- start 2024-03-28T19:24:49.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.455+0000 7f472a400640 1 -- start start 2024-03-28T19:24:49.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.456+0000 7f472a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c004800 0x7f47240c1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.456+0000 7f472a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47240ab770 0x7f47241e2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.456+0000 7f472a400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240abbd0 0x7f4724149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.456+0000 7f472a400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47240c8f50 con 0x7f47240ab770 2024-03-28T19:24:49.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.456+0000 7f472a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47240c8dd0 con 0x7f470c004800 2024-03-28T19:24:49.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.456+0000 7f472a400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47240c5fb0 con 0x7f47240abbd0 2024-03-28T19:24:49.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4723e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47240ab770 0x7f47241e2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4723e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47240ab770 0x7f47241e2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51340/0 (socket says 172.21.15.27:51340) 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4723e00640 1 -- 172.21.15.27:0/1286669415 learned_addr learned my addr 172.21.15.27:0/1286669415 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4729000640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c004800 0x7f47240c1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4729a00640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240abbd0 0x7f4724149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4723e00640 1 -- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240abbd0 msgr2=0x7f4724149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4723e00640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240abbd0 0x7f4724149910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4723e00640 1 -- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c004800 msgr2=0x7f47240c1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4723e00640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c004800 0x7f47240c1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4723e00640 1 -- 172.21.15.27:0/1286669415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47240ac0a0 con 0x7f47240ab770 2024-03-28T19:24:49.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.457+0000 7f4729000640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c004800 0x7f47240c1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:24:49.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.458+0000 7f4723e00640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47240ab770 0x7f47241e2430 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f471000bad0 tx=0x7f4710004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:49.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.458+0000 7f4720c00640 1 -- 172.21.15.27:0/1286669415 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471003b420 con 0x7f47240ab770 2024-03-28T19:24:49.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.458+0000 7f4720c00640 1 -- 172.21.15.27:0/1286669415 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471003b5c0 con 0x7f47240ab770 2024-03-28T19:24:49.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.458+0000 7f4720c00640 1 -- 172.21.15.27:0/1286669415 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471003b9b0 con 0x7f47240ab770 2024-03-28T19:24:49.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.458+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47241ee5c0 con 0x7f47240ab770 2024-03-28T19:24:49.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.458+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47241eeb80 con 0x7f47240ab770 2024-03-28T19:24:49.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.460+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4724067dd0 con 0x7f47240ab770 2024-03-28T19:24:49.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.460+0000 7f4720c00640 1 -- 172.21.15.27:0/1286669415 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4710039020 con 0x7f47240ab770 2024-03-28T19:24:49.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.461+0000 7f4720c00640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f470406bbd0 0x7f470406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:49.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.461+0000 7f4720c00640 1 -- 172.21.15.27:0/1286669415 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7f47100adab0 con 0x7f47240ab770 2024-03-28T19:24:49.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.462+0000 7f4729000640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f470406bbd0 0x7f470406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:49.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.462+0000 7f4729000640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f470406bbd0 0x7f470406e090 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4718009aa0 tx=0x7f4718008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:49.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.468+0000 7f4720c00640 1 -- 172.21.15.27:0/1286669415 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f47240ac0a0 con 0x7f47240ab770 2024-03-28T19:24:49.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.799+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4724069640 con 0x7f47240ab770 2024-03-28T19:24:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.800+0000 7f4720c00640 1 -- 172.21.15.27:0/1286669415 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+6264 (secure 0 0 0) 0x7f4710040070 con 0x7f47240ab770 2024-03-28T19:24:49.799 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:49.799 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:24:49.799 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 216 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f470406bbd0 msgr2=0x7f470406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f470406bbd0 0x7f470406e090 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4718009aa0 tx=0x7f4718008040 comp rx=0 tx=0).stop 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47240ab770 msgr2=0x7f47241e2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47240ab770 0x7f47241e2430 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f471000bad0 tx=0x7f4710004730 comp rx=0 tx=0).stop 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 shutdown_connections 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f470406bbd0 0x7f470406e090 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47240abbd0 0x7f4724149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47240ab770 0x7f47241e2430 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 --2- 172.21.15.27:0/1286669415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f470c004800 0x7f47240c1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:49.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.803+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 >> 172.21.15.27:0/1286669415 conn(0x7f472406b2a0 msgr2=0x7f47240c8030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:49.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.804+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 shutdown_connections 2024-03-28T19:24:49.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:49.804+0000 7f472a400640 1 -- 172.21.15.27:0/1286669415 wait complete. 2024-03-28T19:24:49.822 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:24:50.437 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 23% (56/239) Updating files: 24% (58/239) Updating files: 25% (60/239) Updating files: 26% (63/239) Updating files: 27% (65/239) Updating files: 28% (67/239) Updating files: 29% (70/239) Updating files: 30% (72/239) Updating files: 31% (75/239) Updating files: 32% (77/239) Updating files: 33% (79/239) Updating files: 34% (82/239) Updating files: 35% (84/239) Updating files: 36% (87/239) Updating files: 37% (89/239) Updating files: 38% (91/239) Updating files: 39% (94/239) Updating files: 40% (96/239) Updating files: 41% (98/239) Updating files: 42% (101/239) Updating files: 43% (103/239) Updating files: 44% (106/239) Updating files: 45% (108/239) Updating files: 46% (110/239) Updating files: 47% (113/239) Updating files: 48% (115/239) Updating files: 49% (118/239) Updating files: 50% (120/239) Updating files: 51% (122/239) Updating files: 52% (125/239) Updating files: 52% (126/239) Updating files: 53% (127/239) Updating files: 54% (130/239) Updating files: 55% (132/239) Updating files: 56% (134/239) Updating files: 57% (137/239) Updating files: 58% (139/239) Updating files: 59% (142/239) Updating files: 60% (144/239) Updating files: 61% (146/239) Updating files: 62% (149/239) Updating files: 63% (151/239) Updating files: 64% (153/239) Updating files: 65% (156/239) Updating files: 66% (158/239) Updating files: 67% (161/239) Updating files: 68% (163/239) Updating files: 69% (165/239) Updating files: 70% (168/239) Updating files: 71% (170/239) Updating files: 72% (173/239) Updating files: 73% (175/239) Updating files: 74% (177/239) Updating files: 75% (180/239) Updating files: 76% (182/239) Updating files: 77% (185/239) Updating files: 78% (187/239) Updating files: 79% (189/239) Updating files: 80% (192/239) Updating files: 81% (194/239) Updating files: 82% (196/239) Updating files: 83% (199/239) Updating files: 84% (201/239) Updating files: 85% (204/239) Updating files: 86% (206/239) Updating files: 87% (208/239) Updating files: 88% (211/239) Updating files: 89% (213/239) Updating files: 90% (216/239) Updating files: 91% (218/239) Updating files: 92% (220/239) Updating files: 93% (223/239) Updating files: 94% (225/239) Updating files: 95% (228/239) Updating files: 96% (230/239) Updating files: 97% (232/239) Updating files: 98% (235/239) Updating files: 99% (237/239) Updating files: 100% (239/239) Updating files: 100% (239/239), done. 2024-03-28T19:24:50.725 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 9981ff90968 v0.26 2024-03-28T19:24:50.748 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.26 2024-03-28T19:24:50.806 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:24:50.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 27 -eq 48 ']' 2024-03-28T19:24:50.807 INFO:tasks.workunit.client.0.smithi027.stdout:v0.27 2024-03-28T19:24:50.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.27 2024-03-28T19:24:50.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.27 2024-03-28T19:24:50.807 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.27 2024-03-28T19:24:51.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:52.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.061+0000 7fe580c00640 1 Processor -- start 2024-03-28T19:24:52.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.061+0000 7fe580c00640 1 -- start start 2024-03-28T19:24:52.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.062+0000 7fe580c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe57c0c8ed0 0x7fe57c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.063+0000 7fe580c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c158bd0 con 0x7fe57c0be5d0 2024-03-28T19:24:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.063+0000 7fe580c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c158eb0 con 0x7fe57c0c58c0 2024-03-28T19:24:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.063+0000 7fe580c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57c159190 con 0x7fe57c0c8ed0 2024-03-28T19:24:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.063+0000 7fe57aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe57c0c8ed0 0x7fe57c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.063+0000 7fe57aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe57c0c8ed0 0x7fe57c0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51348/0 (socket says 172.21.15.27:51348) 2024-03-28T19:24:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.063+0000 7fe57aa00640 1 -- 172.21.15.27:0/176433391 learned_addr learned my addr 172.21.15.27:0/176433391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.064+0000 7fe57aa00640 1 -- 172.21.15.27:0/176433391 >> v1:172.21.15.155:6789/0 conn(0x7fe57c0c58c0 legacy=0x7fe57c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.064+0000 7fe57aa00640 1 -- 172.21.15.27:0/176433391 >> v1:172.21.15.27:6789/0 conn(0x7fe57c0be5d0 legacy=0x7fe57c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:52.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.064+0000 7fe57aa00640 1 -- 172.21.15.27:0/176433391 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c159470 con 0x7fe57c0c8ed0 2024-03-28T19:24:52.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.065+0000 7fe57aa00640 1 --2- 172.21.15.27:0/176433391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0c8ed0 0x7fe57c0c5ca0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fe568009870 tx=0x7fe56802b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6e329f31c5601e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.066+0000 7fe57a000640 1 -- 172.21.15.27:0/176433391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568037070 con 0x7fe57c0c8ed0 2024-03-28T19:24:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.066+0000 7fe57a000640 1 -- 172.21.15.27:0/176433391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe568030710 con 0x7fe57c0c8ed0 2024-03-28T19:24:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.066+0000 7fe57a000640 1 -- 172.21.15.27:0/176433391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568030ae0 con 0x7fe57c0c8ed0 2024-03-28T19:24:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.067+0000 7fe580c00640 1 -- 172.21.15.27:0/176433391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0c8ed0 msgr2=0x7fe57c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:52.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.067+0000 7fe580c00640 1 --2- 172.21.15.27:0/176433391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0c8ed0 0x7fe57c0c5ca0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fe568009870 tx=0x7fe56802b740 comp rx=0 tx=0).stop 2024-03-28T19:24:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.067+0000 7fe580c00640 1 -- 172.21.15.27:0/176433391 shutdown_connections 2024-03-28T19:24:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.067+0000 7fe580c00640 1 --2- 172.21.15.27:0/176433391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0c8ed0 0x7fe57c0c5ca0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.067+0000 7fe580c00640 1 -- 172.21.15.27:0/176433391 >> 172.21.15.27:0/176433391 conn(0x7fe57c06b6f0 msgr2=0x7fe57c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.067+0000 7fe580c00640 1 -- 172.21.15.27:0/176433391 shutdown_connections 2024-03-28T19:24:52.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.067+0000 7fe580c00640 1 -- 172.21.15.27:0/176433391 wait complete. 2024-03-28T19:24:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.069+0000 7fe580c00640 1 Processor -- start 2024-03-28T19:24:52.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.069+0000 7fe580c00640 1 -- start start 2024-03-28T19:24:52.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe580c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0be5d0 0x7fe57c1576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:52.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe580c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c0c58c0 0x7fe57c1ef400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe580c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe57c0c8ed0 0x7fe57c1ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe580c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0b09a0 con 0x7fe57c0be5d0 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe580c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0b0820 con 0x7fe57c0c58c0 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe580c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0ad7d0 con 0x7fe57c0c8ed0 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe57b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0be5d0 0x7fe57c1576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe57b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0be5d0 0x7fe57c1576c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51356/0 (socket says 172.21.15.27:51356) 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.071+0000 7fe57b400640 1 -- 172.21.15.27:0/3340272006 learned_addr learned my addr 172.21.15.27:0/3340272006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.072+0000 7fe57be00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe57c0c8ed0 0x7fe57c1ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.072+0000 7fe57aa00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c0c58c0 0x7fe57c1ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.072+0000 7fe57b400640 1 -- 172.21.15.27:0/3340272006 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe57c0c8ed0 msgr2=0x7fe57c1ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.072+0000 7fe57b400640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe57c0c8ed0 0x7fe57c1ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.072+0000 7fe57b400640 1 -- 172.21.15.27:0/3340272006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c0c58c0 msgr2=0x7fe57c1ef400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:52.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.072+0000 7fe57b400640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c0c58c0 0x7fe57c1ef400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.072+0000 7fe57b400640 1 -- 172.21.15.27:0/3340272006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c0c4e50 con 0x7fe57c0be5d0 2024-03-28T19:24:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.073+0000 7fe57b400640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0be5d0 0x7fe57c1576c0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00c450 tx=0x7fe56c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:52.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.073+0000 7fe573400640 1 -- 172.21.15.27:0/3340272006 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c0052d0 con 0x7fe57c0be5d0 2024-03-28T19:24:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.073+0000 7fe573400640 1 -- 172.21.15.27:0/3340272006 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c005470 con 0x7fe57c0be5d0 2024-03-28T19:24:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.073+0000 7fe573400640 1 -- 172.21.15.27:0/3340272006 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c012870 con 0x7fe57c0be5d0 2024-03-28T19:24:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.073+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c155ab0 con 0x7fe57c0be5d0 2024-03-28T19:24:52.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.073+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c157c00 con 0x7fe57c0be5d0 2024-03-28T19:24:52.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.075+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe554005290 con 0x7fe57c0be5d0 2024-03-28T19:24:52.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.077+0000 7fe573400640 1 -- 172.21.15.27:0/3340272006 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe56c012a10 con 0x7fe57c0be5d0 2024-03-28T19:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.078+0000 7fe573400640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe55c06bc50 0x7fe55c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.078+0000 7fe573400640 1 -- 172.21.15.27:0/3340272006 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7fe56c08b620 con 0x7fe57c0be5d0 2024-03-28T19:24:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.079+0000 7fe57aa00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe55c06bc50 0x7fe55c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:52.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.080+0000 7fe57aa00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe55c06bc50 0x7fe55c06e110 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe568034000 tx=0x7fe568008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:52.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.089+0000 7fe573400640 1 -- 172.21.15.27:0/3340272006 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe56c059100 con 0x7fe57c0be5d0 2024-03-28T19:24:52.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.438+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe554005580 con 0x7fe57c0be5d0 2024-03-28T19:24:52.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.438+0000 7fe573400640 1 -- 172.21.15.27:0/3340272006 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fe56c058a00 con 0x7fe57c0be5d0 2024-03-28T19:24:52.437 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:52.438 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:24:52.438 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 216 2024-03-28T19:24:52.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.441+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe55c06bc50 msgr2=0x7fe55c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.441+0000 7fe580c00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe55c06bc50 0x7fe55c06e110 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe568034000 tx=0x7fe568008030 comp rx=0 tx=0).stop 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.441+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0be5d0 msgr2=0x7fe57c1576c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.441+0000 7fe580c00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0be5d0 0x7fe57c1576c0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00c450 tx=0x7fe56c00c910 comp rx=0 tx=0).stop 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.442+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 shutdown_connections 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.442+0000 7fe580c00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe55c06bc50 0x7fe55c06e110 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.442+0000 7fe580c00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe57c0c8ed0 0x7fe57c1ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.442+0000 7fe580c00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c0c58c0 0x7fe57c1ef400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.442+0000 7fe580c00640 1 --2- 172.21.15.27:0/3340272006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe57c0be5d0 0x7fe57c1576c0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.442+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 >> 172.21.15.27:0/3340272006 conn(0x7fe57c06b6f0 msgr2=0x7fe57c0c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.442+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 shutdown_connections 2024-03-28T19:24:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:52.442+0000 7fe580c00640 1 -- 172.21.15.27:0/3340272006 wait complete. 2024-03-28T19:24:52.464 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:24:54.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.716+0000 7fa46ec00640 1 Processor -- start 2024-03-28T19:24:54.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.716+0000 7fa46ec00640 1 -- start start 2024-03-28T19:24:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.718+0000 7fa46ec00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.718+0000 7fa46ec00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa4681534c0 0x7fa468153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.718+0000 7fa46ec00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4680c3d90 con 0x7fa468150c70 2024-03-28T19:24:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.718+0000 7fa46ec00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4680ab4e0 con 0x7fa4681534c0 2024-03-28T19:24:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.718+0000 7fa46ec00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680ac1f0 con 0x7fa4680a9cb0 2024-03-28T19:24:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.718+0000 7fa467e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa468150c70 0x7fa4680c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44812/0 (socket says 172.21.15.27:44812) 2024-03-28T19:24:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.718+0000 7fa467e00640 1 -- 172.21.15.27:0/1862909801 learned_addr learned my addr 172.21.15.27:0/1862909801 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:24:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.718+0000 7fa467400640 1 --2- 172.21.15.27:0/1862909801 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.719+0000 7fa46cc00640 1 --2- 172.21.15.27:0/1862909801 >> v2:172.21.15.155:3300/0 conn(0x7fa4681534c0 0x7fa468153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.719+0000 7fa466a00640 1 -- 172.21.15.27:0/1862909801 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825786178 0 0) 0x7fa4680c3d90 con 0x7fa468150c70 2024-03-28T19:24:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.719+0000 7fa467400640 1 -- 172.21.15.27:0/1862909801 >> v2:172.21.15.155:3300/0 conn(0x7fa4681534c0 msgr2=0x7fa468153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.719+0000 7fa467400640 1 --2- 172.21.15.27:0/1862909801 >> v2:172.21.15.155:3300/0 conn(0x7fa4681534c0 0x7fa468153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.719+0000 7fa467400640 1 -- 172.21.15.27:0/1862909801 >> v1:172.21.15.27:6789/0 conn(0x7fa468150c70 legacy=0x7fa4680c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.719+0000 7fa467400640 1 -- 172.21.15.27:0/1862909801 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680ab710 con 0x7fa4680a9cb0 2024-03-28T19:24:54.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.721+0000 7fa467400640 1 --2- 172.21.15.27:0/1862909801 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680c3690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa4540097a0 tx=0x7fa45402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f05faeb9721536d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:54.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.722+0000 7fa466a00640 1 -- 172.21.15.27:0/1862909801 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454031780 con 0x7fa4680a9cb0 2024-03-28T19:24:54.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.722+0000 7fa466a00640 1 -- 172.21.15.27:0/1862909801 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa454037070 con 0x7fa4680a9cb0 2024-03-28T19:24:54.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.722+0000 7fa466a00640 1 -- 172.21.15.27:0/1862909801 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454031b40 con 0x7fa4680a9cb0 2024-03-28T19:24:54.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.722+0000 7fa46ec00640 1 -- 172.21.15.27:0/1862909801 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 msgr2=0x7fa4680c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:54.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.722+0000 7fa46ec00640 1 --2- 172.21.15.27:0/1862909801 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680c3690 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa4540097a0 tx=0x7fa45402b820 comp rx=0 tx=0).stop 2024-03-28T19:24:54.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.723+0000 7fa46ec00640 1 -- 172.21.15.27:0/1862909801 shutdown_connections 2024-03-28T19:24:54.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.723+0000 7fa46ec00640 1 --2- 172.21.15.27:0/1862909801 >> v2:172.21.15.155:3300/0 conn(0x7fa4681534c0 0x7fa468153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:54.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.723+0000 7fa46ec00640 1 --2- 172.21.15.27:0/1862909801 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680c3690 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:54.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.723+0000 7fa46ec00640 1 -- 172.21.15.27:0/1862909801 >> 172.21.15.27:0/1862909801 conn(0x7fa46806b9d0 msgr2=0x7fa4680c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:54.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.723+0000 7fa46ec00640 1 -- 172.21.15.27:0/1862909801 shutdown_connections 2024-03-28T19:24:54.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.723+0000 7fa46ec00640 1 -- 172.21.15.27:0/1862909801 wait complete. 2024-03-28T19:24:54.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.725+0000 7fa46ec00640 1 Processor -- start 2024-03-28T19:24:54.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.725+0000 7fa46ec00640 1 -- start start 2024-03-28T19:24:54.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.726+0000 7fa46ec00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:54.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.726+0000 7fa46ec00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa468150c70 0x7fa468156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:54.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.726+0000 7fa46ec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4681534c0 0x7fa468157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:54.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.726+0000 7fa46ec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680c1810 con 0x7fa4681534c0 2024-03-28T19:24:54.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.726+0000 7fa46ec00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680abcd0 con 0x7fa468150c70 2024-03-28T19:24:54.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.726+0000 7fa46ec00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680c6830 con 0x7fa4680a9cb0 2024-03-28T19:24:54.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa46cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4681534c0 0x7fa468157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:54.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa467400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa468150c70 0x7fa468156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:54.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa467400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa468150c70 0x7fa468156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:58876/0 (socket says 172.21.15.27:58876) 2024-03-28T19:24:54.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa467400640 1 -- 172.21.15.27:0/4066645964 learned_addr learned my addr 172.21.15.27:0/4066645964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:54.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa467e00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:54.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa46cc00640 1 -- 172.21.15.27:0/4066645964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 msgr2=0x7fa4680714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:54.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa46cc00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:54.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa46cc00640 1 -- 172.21.15.27:0/4066645964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa468150c70 msgr2=0x7fa468156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:54.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa46cc00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa468150c70 0x7fa468156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:54.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.727+0000 7fa46cc00640 1 -- 172.21.15.27:0/4066645964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680a9450 con 0x7fa4681534c0 2024-03-28T19:24:54.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.728+0000 7fa46cc00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4681534c0 0x7fa468157520 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fa46000eba0 tx=0x7fa46000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:54.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.728+0000 7fa45fe00640 1 -- 172.21.15.27:0/4066645964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa460018020 con 0x7fa4681534c0 2024-03-28T19:24:54.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.729+0000 7fa45fe00640 1 -- 172.21.15.27:0/4066645964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46000cd20 con 0x7fa4681534c0 2024-03-28T19:24:54.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.729+0000 7fa45fe00640 1 -- 172.21.15.27:0/4066645964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4600146c0 con 0x7fa4681534c0 2024-03-28T19:24:54.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.729+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4681ebb70 con 0x7fa4681534c0 2024-03-28T19:24:54.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.729+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4680a8a30 con 0x7fa4681534c0 2024-03-28T19:24:54.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.730+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa46804f400 con 0x7fa4681534c0 2024-03-28T19:24:54.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.732+0000 7fa45fe00640 1 -- 172.21.15.27:0/4066645964 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa460005d10 con 0x7fa4681534c0 2024-03-28T19:24:54.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.734+0000 7fa45fe00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa44806bca0 0x7fa44806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:54.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.734+0000 7fa45fe00640 1 -- 172.21.15.27:0/4066645964 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9905+0+0 (secure 0 0 0) 0x7fa46008c6c0 con 0x7fa4681534c0 2024-03-28T19:24:54.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.734+0000 7fa467e00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa44806bca0 0x7fa44806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:54.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.735+0000 7fa467e00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa44806bca0 0x7fa44806e160 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa458000f60 tx=0x7fa458007d50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:54.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:54.742+0000 7fa45fe00640 1 -- 172.21.15.27:0/4066645964 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa460022030 con 0x7fa4681534c0 2024-03-28T19:24:55.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.086+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4680c5f00 con 0x7fa4681534c0 2024-03-28T19:24:55.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.087+0000 7fa45fe00640 1 -- 172.21.15.27:0/4066645964 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+6264 (secure 0 0 0) 0x7fa460010040 con 0x7fa4681534c0 2024-03-28T19:24:55.086 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:55.086 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":8586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:23:20.987967+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7537":{"gid":7537,"name":"b","rank":0,"incarnation":205,"state":"up:active","state_seq":9,"addr":"172.21.15.155:6835/4236980323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":4236980323},{"type":"v1","addr":"172.21.15.155:6835","nonce":4236980323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7537,"qdb_cluster":[7537]},"id":2}]} 2024-03-28T19:24:55.086 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 216 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.089+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa44806bca0 msgr2=0x7fa44806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.089+0000 7fa46ec00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa44806bca0 0x7fa44806e160 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa458000f60 tx=0x7fa458007d50 comp rx=0 tx=0).stop 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.089+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4681534c0 msgr2=0x7fa468157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.089+0000 7fa46ec00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4681534c0 0x7fa468157520 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fa46000eba0 tx=0x7fa46000c490 comp rx=0 tx=0).stop 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.090+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 shutdown_connections 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.090+0000 7fa46ec00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa44806bca0 0x7fa44806e160 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.090+0000 7fa46ec00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4681534c0 0x7fa468157520 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.090+0000 7fa46ec00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa468150c70 0x7fa468156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.090+0000 7fa46ec00640 1 --2- 172.21.15.27:0/4066645964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4680a9cb0 0x7fa4680714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:55.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.090+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 >> 172.21.15.27:0/4066645964 conn(0x7fa46806b9d0 msgr2=0x7fa468151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:55.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.090+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 shutdown_connections 2024-03-28T19:24:55.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:55.090+0000 7fa46ec00640 1 -- 172.21.15.27:0/4066645964 wait complete. 2024-03-28T19:24:55.109 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:24:57.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.359+0000 7efeefa00640 1 Processor -- start 2024-03-28T19:24:57.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.359+0000 7efeefa00640 1 -- start start 2024-03-28T19:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.360+0000 7efeefa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 0x7efee80c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.361+0000 7efeefa00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efee81534c0 0x7efee8153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.361+0000 7efeefa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80c3d90 con 0x7efee8150c70 2024-03-28T19:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.361+0000 7efeefa00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee80ab4e0 con 0x7efee81534c0 2024-03-28T19:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.361+0000 7efeefa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee80ac1f0 con 0x7efee80a9cb0 2024-03-28T19:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.361+0000 7efeed000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efee8150c70 0x7efee80c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36440/0 (socket says 172.21.15.27:36440) 2024-03-28T19:24:57.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.361+0000 7efeed000640 1 -- 172.21.15.27:0/2645778629 learned_addr learned my addr 172.21.15.27:0/2645778629 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.361+0000 7efeeda00640 1 --2- 172.21.15.27:0/2645778629 >> v2:172.21.15.155:3300/0 conn(0x7efee81534c0 0x7efee8153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7e00640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 0x7efee80c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001791044 0 0) 0x7efee80c3d90 con 0x7efee8150c70 2024-03-28T19:24:57.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0003520 con 0x7efee8150c70 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3553392038 0 0) 0x7efed0003520 con 0x7efee8150c70 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 msgr2=0x7efee80c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 0x7efee80c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 >> v2:172.21.15.155:3300/0 conn(0x7efee81534c0 msgr2=0x7efee8153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 --2- 172.21.15.27:0/2645778629 >> v2:172.21.15.155:3300/0 conn(0x7efee81534c0 0x7efee8153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80ab710 con 0x7efee8150c70 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7efed4001670 con 0x7efee8150c70 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 >> v1:172.21.15.27:6789/0 conn(0x7efee8150c70 legacy=0x7efee80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:57.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 0x7efed00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efed0003fb0 0x7efed0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.362+0000 7efee7400640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efed0004d40 0x7efed00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.363+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee80ac1f0 con 0x7efed0003fb0 2024-03-28T19:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.363+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed000ba20 con 0x7efed0004d40 2024-03-28T19:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.363+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed0010030 con 0x7efee80a9cb0 2024-03-28T19:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.363+0000 7efeed000640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 0x7efed00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.363+0000 7efee7e00640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efed0003fb0 0x7efed0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.364+0000 7efeed000640 1 -- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efed0004d40 msgr2=0x7efed00051c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T19:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.364+0000 7efeeda00640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efed0004d40 0x7efed00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.364+0000 7efeed000640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efed0004d40 0x7efed00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.364+0000 7efeed000640 1 -- 172.21.15.27:0/2645778629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efed0003fb0 msgr2=0x7efed0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.364+0000 7efeed000640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efed0003fb0 0x7efed0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.364+0000 7efeed000640 1 -- 172.21.15.27:0/2645778629 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0003520 con 0x7efee80a9cb0 2024-03-28T19:24:57.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.364+0000 7efee7e00640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efed0003fb0 0x7efed0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.365+0000 7efeed000640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 0x7efed00039e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efed400c110 tx=0x7efed0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=191c57d0c325065 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.366+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4007c40 con 0x7efee80a9cb0 2024-03-28T19:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.366+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed4030070 con 0x7efee80a9cb0 2024-03-28T19:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.366+0000 7efee7400640 1 -- 172.21.15.27:0/2645778629 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4005d70 con 0x7efee80a9cb0 2024-03-28T19:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.366+0000 7efeefa00640 1 -- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 msgr2=0x7efed00039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:57.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.366+0000 7efeefa00640 1 --2- 172.21.15.27:0/2645778629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee80a9cb0 0x7efed00039e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efed400c110 tx=0x7efed0005830 comp rx=0 tx=0).stop 2024-03-28T19:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.367+0000 7efeeda00640 1 -- 172.21.15.27:0/2645778629 reap_dead start 2024-03-28T19:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.367+0000 7efeefa00640 1 -- 172.21.15.27:0/2645778629 shutdown_connections 2024-03-28T19:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.367+0000 7efeefa00640 1 -- 172.21.15.27:0/2645778629 >> 172.21.15.27:0/2645778629 conn(0x7efee806b9d0 msgr2=0x7efee80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.367+0000 7efeefa00640 1 -- 172.21.15.27:0/2645778629 shutdown_connections 2024-03-28T19:24:57.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.367+0000 7efeefa00640 1 -- 172.21.15.27:0/2645778629 wait complete. 2024-03-28T19:24:57.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.369+0000 7efeefa00640 1 Processor -- start 2024-03-28T19:24:57.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.369+0000 7efeefa00640 1 -- start start 2024-03-28T19:24:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.370+0000 7efeefa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efee80a9cb0 0x7efee814b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.370+0000 7efeefa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efee8154830 0x7efee814dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.370+0000 7efeefa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee814e1f0 0x7efee8145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.370+0000 7efeefa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee80c1810 con 0x7efee80a9cb0 2024-03-28T19:24:57.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.370+0000 7efeefa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee80abcd0 con 0x7efee8154830 2024-03-28T19:24:57.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.370+0000 7efeefa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee80c6830 con 0x7efee814e1f0 2024-03-28T19:24:57.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.370+0000 7efeed000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efee80a9cb0 0x7efee814b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efeed000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efee80a9cb0 0x7efee814b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41890/0 (socket says 172.21.15.27:41890) 2024-03-28T19:24:57.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efeed000640 1 -- 172.21.15.27:0/1860204167 learned_addr learned my addr 172.21.15.27:0/1860204167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:57.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efeeda00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee814e1f0 0x7efee8145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efee7e00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efee8154830 0x7efee814dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efeed000640 1 -- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee814e1f0 msgr2=0x7efee8145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efeed000640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee814e1f0 0x7efee8145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efeed000640 1 -- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efee8154830 msgr2=0x7efee814dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efeed000640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efee8154830 0x7efee814dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.371+0000 7efeed000640 1 -- 172.21.15.27:0/1860204167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee814c380 con 0x7efee80a9cb0 2024-03-28T19:24:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.372+0000 7efee7e00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efee8154830 0x7efee814dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:24:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.372+0000 7efeed000640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efee80a9cb0 0x7efee814b9b0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7efed400c230 tx=0x7efee80c3f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.372+0000 7efee4c00640 1 -- 172.21.15.27:0/1860204167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4045070 con 0x7efee80a9cb0 2024-03-28T19:24:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.372+0000 7efee4c00640 1 -- 172.21.15.27:0/1860204167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed4000ed0 con 0x7efee80a9cb0 2024-03-28T19:24:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.373+0000 7efee4c00640 1 -- 172.21.15.27:0/1860204167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed4007c40 con 0x7efee80a9cb0 2024-03-28T19:24:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.373+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee81ea440 con 0x7efee80a9cb0 2024-03-28T19:24:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.373+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee81eaa00 con 0x7efee80a9cb0 2024-03-28T19:24:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.374+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee804f400 con 0x7efee80a9cb0 2024-03-28T19:24:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.376+0000 7efee4c00640 1 -- 172.21.15.27:0/1860204167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efed4007070 con 0x7efee80a9cb0 2024-03-28T19:24:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.377+0000 7efee4c00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efec806bbd0 0x7efec806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.377+0000 7efee4c00640 1 -- 172.21.15.27:0/1860204167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7efed40acba0 con 0x7efee80a9cb0 2024-03-28T19:24:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.378+0000 7efee7e00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efec806bbd0 0x7efec806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.379+0000 7efee7e00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efec806bbd0 0x7efec806e090 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7efee814eef0 tx=0x7efedc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:57.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.386+0000 7efee4c00640 1 -- 172.21.15.27:0/1860204167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efee814c380 con 0x7efee80a9cb0 2024-03-28T19:24:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.669+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efee80c5f00 con 0x7efee80a9cb0 2024-03-28T19:24:57.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.671+0000 7efee4c00640 1 -- 172.21.15.27:0/1860204167 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+5462 (secure 0 0 0) 0x7efed407a040 con 0x7efee80a9cb0 2024-03-28T19:24:57.672 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:57.672 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:24:55.429514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:24:57.672 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 217 2024-03-28T19:24:57.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.676+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efec806bbd0 msgr2=0x7efec806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:57.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.676+0000 7efeefa00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efec806bbd0 0x7efec806e090 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7efee814eef0 tx=0x7efedc008040 comp rx=0 tx=0).stop 2024-03-28T19:24:57.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.676+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efee80a9cb0 msgr2=0x7efee814b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.676+0000 7efeefa00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efee80a9cb0 0x7efee814b9b0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7efed400c230 tx=0x7efee80c3f60 comp rx=0 tx=0).stop 2024-03-28T19:24:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.677+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 shutdown_connections 2024-03-28T19:24:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.677+0000 7efeefa00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efec806bbd0 0x7efec806e090 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.677+0000 7efeefa00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efee814e1f0 0x7efee8145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.677+0000 7efeefa00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efee8154830 0x7efee814dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.677+0000 7efeefa00640 1 --2- 172.21.15.27:0/1860204167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efee80a9cb0 0x7efee814b9b0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.677+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 >> 172.21.15.27:0/1860204167 conn(0x7efee806b9d0 msgr2=0x7efee81515d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.677+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 shutdown_connections 2024-03-28T19:24:57.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:57.677+0000 7efeefa00640 1 -- 172.21.15.27:0/1860204167 wait complete. 2024-03-28T19:24:57.696 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-03-28T19:24:57.696 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:24:57.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:24:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.016+0000 7f6047200640 1 Processor -- start 2024-03-28T19:24:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.016+0000 7f6047200640 1 -- start start 2024-03-28T19:24:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.017+0000 7f6047200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f60480c58c0 0x7f60480c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.017+0000 7f6047200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480c8ed0 0x7f60481555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.017+0000 7f6047200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60480be5d0 0x7f6048155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.017+0000 7f6047200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6048158bd0 con 0x7f60480be5d0 2024-03-28T19:24:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.017+0000 7f6047200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6048158eb0 con 0x7f60480c58c0 2024-03-28T19:24:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.017+0000 7f6047200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6048159190 con 0x7f60480c8ed0 2024-03-28T19:24:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6046800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60480be5d0 0x7f6048155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6046800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f60480be5d0 0x7f6048155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41896/0 (socket says 172.21.15.27:41896) 2024-03-28T19:24:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6046800640 1 -- 172.21.15.27:0/1741312946 learned_addr learned my addr 172.21.15.27:0/1741312946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6045e00640 1 --2- 172.21.15.27:0/1741312946 >> v2:172.21.15.155:3300/0 conn(0x7f60480c58c0 0x7f60480c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f603fe00640 1 --2- 172.21.15.27:0/1741312946 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480c8ed0 0x7f60481555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6046800640 1 -- 172.21.15.27:0/1741312946 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480c8ed0 msgr2=0x7f60481555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6046800640 1 --2- 172.21.15.27:0/1741312946 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480c8ed0 0x7f60481555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6046800640 1 -- 172.21.15.27:0/1741312946 >> v2:172.21.15.155:3300/0 conn(0x7f60480c58c0 msgr2=0x7f60480c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6046800640 1 --2- 172.21.15.27:0/1741312946 >> v2:172.21.15.155:3300/0 conn(0x7f60480c58c0 0x7f60480c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6046800640 1 -- 172.21.15.27:0/1741312946 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6048159470 con 0x7f60480be5d0 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.018+0000 7f6045e00640 1 --2- 172.21.15.27:0/1741312946 >> v2:172.21.15.155:3300/0 conn(0x7f60480c58c0 0x7f60480c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.019+0000 7f603fe00640 1 --2- 172.21.15.27:0/1741312946 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480c8ed0 0x7f60481555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:24:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.020+0000 7f6046800640 1 --2- 172.21.15.27:0/1741312946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480be5d0 0x7f6048155b20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f604806c060 tx=0x7f604002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6ed698d9dd1251 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.020+0000 7f6045400640 1 -- 172.21.15.27:0/1741312946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604002c9f0 con 0x7f60480be5d0 2024-03-28T19:24:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.020+0000 7f6045400640 1 -- 172.21.15.27:0/1741312946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604002cb90 con 0x7f60480be5d0 2024-03-28T19:24:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.020+0000 7f6045400640 1 -- 172.21.15.27:0/1741312946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60400343f0 con 0x7f60480be5d0 2024-03-28T19:24:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.021+0000 7f6047200640 1 -- 172.21.15.27:0/1741312946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480be5d0 msgr2=0x7f6048155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.021+0000 7f6047200640 1 --2- 172.21.15.27:0/1741312946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480be5d0 0x7f6048155b20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f604806c060 tx=0x7f604002be50 comp rx=0 tx=0).stop 2024-03-28T19:24:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.022+0000 7f6047200640 1 -- 172.21.15.27:0/1741312946 shutdown_connections 2024-03-28T19:24:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.022+0000 7f6047200640 1 --2- 172.21.15.27:0/1741312946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480be5d0 0x7f6048155b20 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.022+0000 7f6047200640 1 --2- 172.21.15.27:0/1741312946 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480c8ed0 0x7f60481555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.022+0000 7f6047200640 1 --2- 172.21.15.27:0/1741312946 >> v2:172.21.15.155:3300/0 conn(0x7f60480c58c0 0x7f60480c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.022+0000 7f6047200640 1 -- 172.21.15.27:0/1741312946 >> 172.21.15.27:0/1741312946 conn(0x7f604806b6f0 msgr2=0x7f60480acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.022+0000 7f6047200640 1 -- 172.21.15.27:0/1741312946 shutdown_connections 2024-03-28T19:24:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.022+0000 7f6047200640 1 -- 172.21.15.27:0/1741312946 wait complete. 2024-03-28T19:24:58.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.026+0000 7f6047200640 1 Processor -- start 2024-03-28T19:24:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.026+0000 7f6047200640 1 -- start start 2024-03-28T19:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.027+0000 7f6047200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480be5d0 0x7f60480c1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.027+0000 7f6047200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60480c58c0 0x7f604814c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.027+0000 7f6047200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480c8ed0 0x7f60481ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.027+0000 7f6047200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60480b09a0 con 0x7f60480c8ed0 2024-03-28T19:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.027+0000 7f6047200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60480b0820 con 0x7f60480c58c0 2024-03-28T19:24:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.027+0000 7f6047200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60480ad7d0 con 0x7f60480be5d0 2024-03-28T19:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.028+0000 7f6045e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480be5d0 0x7f60480c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.028+0000 7f6046800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480c8ed0 0x7f60481ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.028+0000 7f6046800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480c8ed0 0x7f60481ee970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41904/0 (socket says 172.21.15.27:41904) 2024-03-28T19:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.028+0000 7f6046800640 1 -- 172.21.15.27:0/725627279 learned_addr learned my addr 172.21.15.27:0/725627279 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:24:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.028+0000 7f603fe00640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60480c58c0 0x7f604814c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.029+0000 7f6046800640 1 -- 172.21.15.27:0/725627279 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480be5d0 msgr2=0x7f60480c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.029+0000 7f6046800640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480be5d0 0x7f60480c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.029+0000 7f6046800640 1 -- 172.21.15.27:0/725627279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60480c58c0 msgr2=0x7f604814c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.029+0000 7f6046800640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60480c58c0 0x7f604814c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.029+0000 7f6046800640 1 -- 172.21.15.27:0/725627279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604814c4b0 con 0x7f60480c8ed0 2024-03-28T19:24:58.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.029+0000 7f6046800640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480c8ed0 0x7f60481ee970 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f6040009f40 tx=0x7f6040004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.030+0000 7f603e000640 1 -- 172.21.15.27:0/725627279 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040043020 con 0x7f60480c8ed0 2024-03-28T19:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.030+0000 7f603e000640 1 -- 172.21.15.27:0/725627279 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604003e450 con 0x7f60480c8ed0 2024-03-28T19:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.030+0000 7f603e000640 1 -- 172.21.15.27:0/725627279 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040009510 con 0x7f60480c8ed0 2024-03-28T19:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.030+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60481ef290 con 0x7f60480c8ed0 2024-03-28T19:24:58.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.030+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60480bebd0 con 0x7f60480c8ed0 2024-03-28T19:24:58.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.032+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6048069640 con 0x7f60480c8ed0 2024-03-28T19:24:58.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.034+0000 7f603e000640 1 -- 172.21.15.27:0/725627279 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6040046020 con 0x7f60480c8ed0 2024-03-28T19:24:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.035+0000 7f603e000640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f602406bc00 0x7f602406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:24:58.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.035+0000 7f603e000640 1 -- 172.21.15.27:0/725627279 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f60400adf20 con 0x7f60480c8ed0 2024-03-28T19:24:58.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.036+0000 7f6045e00640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f602406bc00 0x7f602406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:24:58.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.037+0000 7f6045e00640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f602406bc00 0x7f602406e0c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6048157100 tx=0x7f6038008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:24:58.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.041+0000 7f603e000640 1 -- 172.21.15.27:0/725627279 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f604004d070 con 0x7f60480c8ed0 2024-03-28T19:24:58.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.409+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60480ad5b0 con 0x7f60480c8ed0 2024-03-28T19:24:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.410+0000 7f603e000640 1 -- 172.21.15.27:0/725627279 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+5462 (secure 0 0 0) 0x7f604007b2e0 con 0x7f60480c8ed0 2024-03-28T19:24:58.409 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:24:58.409 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:24:55.429514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:24:58.409 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 217 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.412+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f602406bc00 msgr2=0x7f602406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.412+0000 7f6047200640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f602406bc00 0x7f602406e0c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6048157100 tx=0x7f6038008040 comp rx=0 tx=0).stop 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.412+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480c8ed0 msgr2=0x7f60481ee970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.412+0000 7f6047200640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480c8ed0 0x7f60481ee970 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f6040009f40 tx=0x7f6040004a40 comp rx=0 tx=0).stop 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.413+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 shutdown_connections 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.413+0000 7f6047200640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f602406bc00 0x7f602406e0c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.413+0000 7f6047200640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60480c8ed0 0x7f60481ee970 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.413+0000 7f6047200640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60480c58c0 0x7f604814c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.413+0000 7f6047200640 1 --2- 172.21.15.27:0/725627279 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60480be5d0 0x7f60480c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.413+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 >> 172.21.15.27:0/725627279 conn(0x7f604806b6f0 msgr2=0x7f60480ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.413+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 shutdown_connections 2024-03-28T19:24:58.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:24:58.413+0000 7f6047200640 1 -- 172.21.15.27:0/725627279 wait complete. 2024-03-28T19:24:58.433 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 217, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 215, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8511": { "addr": "172.21.15.27:6835/3741162434", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 3741162434, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 3741162434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8511, "incarnation": 211, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 90, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:23:46.756300+0000", "qdb_cluster": [ 8511 ], "qdb_leader": 8511, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8511 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 217, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8586": { "addr": "172.21.15.27:6837/1478398036", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 1478398036, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 1478398036, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8586, "incarnation": 217, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 91, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:24:55.429514+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8586 } } } ], "standbys": [ { "addr": "172.21.15.155:6834/3334295790", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3334295790, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3334295790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 193, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6467, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:24:58.854 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 23% (59/253) Updating files: 24% (61/253) Updating files: 25% (64/253) Updating files: 26% (66/253) Updating files: 27% (69/253) Updating files: 28% (71/253) Updating files: 29% (74/253) Updating files: 30% (76/253) Updating files: 31% (79/253) Updating files: 32% (81/253) Updating files: 33% (84/253) Updating files: 34% (87/253) Updating files: 35% (89/253) Updating files: 36% (92/253) Updating files: 37% (94/253) Updating files: 38% (97/253) Updating files: 39% (99/253) Updating files: 40% (102/253) Updating files: 40% (103/253) Updating files: 41% (104/253) Updating files: 42% (107/253) Updating files: 43% (109/253) Updating files: 44% (112/253) Updating files: 45% (114/253) Updating files: 46% (117/253) Updating files: 47% (119/253) Updating files: 48% (122/253) Updating files: 49% (124/253) Updating files: 50% (127/253) Updating files: 51% (130/253) Updating files: 52% (132/253) Updating files: 53% (135/253) Updating files: 54% (137/253) Updating files: 55% (140/253) Updating files: 56% (142/253) Updating files: 57% (145/253) Updating files: 57% (146/253) Updating files: 58% (147/253) Updating files: 59% (150/253) Updating files: 60% (152/253) Updating files: 61% (155/253) Updating files: 62% (157/253) Updating files: 63% (160/253) Updating files: 64% (162/253) Updating files: 65% (165/253) Updating files: 66% (167/253) Updating files: 67% (170/253) Updating files: 68% (173/253) Updating files: 69% (175/253) Updating files: 70% (178/253) Updating files: 71% (180/253) Updating files: 72% (183/253) Updating files: 73% (185/253) Updating files: 74% (188/253) Updating files: 75% (190/253) Updating files: 76% (193/253) Updating files: 77% (195/253) Updating files: 78% (198/253) Updating files: 79% (200/253) Updating files: 80% (203/253) Updating files: 81% (205/253) Updating files: 82% (208/253) Updating files: 83% (210/253) Updating files: 84% (213/253) Updating files: 85% (216/253) Updating files: 86% (218/253) Updating files: 87% (221/253) Updating files: 88% (223/253) Updating files: 89% (226/253) Updating files: 90% (228/253) Updating files: 91% (231/253) Updating files: 92% (233/253) Updating files: 93% (236/253) Updating files: 94% (238/253) Updating files: 95% (241/253) Updating files: 96% (243/253) Updating files: 97% (246/253) Updating files: 98% (248/253) Updating files: 99% (251/253) Updating files: 100% (253/253) Updating files: 100% (253/253), done. 2024-03-28T19:24:59.040 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 793034c62c8 v0.27 2024-03-28T19:24:59.052 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.27 2024-03-28T19:24:59.112 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:24:59.112 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 28 -eq 48 ']' 2024-03-28T19:24:59.113 INFO:tasks.workunit.client.0.smithi027.stdout:v0.28 2024-03-28T19:24:59.113 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.28 2024-03-28T19:24:59.113 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.28 2024-03-28T19:24:59.113 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.28 2024-03-28T19:25:00.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.680+0000 7fbbef000640 1 Processor -- start 2024-03-28T19:25:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.680+0000 7fbbef000640 1 -- start start 2024-03-28T19:25:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.682+0000 7fbbef000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbbe8152f80 0x7fbbe80c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.682+0000 7fbbef000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbbe80c7df0 0x7fbbe8155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.682+0000 7fbbef000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe80c7310 con 0x7fbbe80c7970 2024-03-28T19:25:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.682+0000 7fbbef000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe8158d40 con 0x7fbbe80c7df0 2024-03-28T19:25:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.682+0000 7fbbef000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe8159020 con 0x7fbbe8152f80 2024-03-28T19:25:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.683+0000 7fbbe7400640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbbe80c7970 0x7fbbe8155730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:35480/0 (socket says 172.21.15.27:35480) 2024-03-28T19:25:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.684+0000 7fbbe7400640 1 -- 172.21.15.27:0/4162718955 learned_addr learned my addr 172.21.15.27:0/4162718955 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.684+0000 7fbbe7e00640 1 --2- 172.21.15.27:0/4162718955 >> v2:172.21.15.155:3300/0 conn(0x7fbbe8152f80 0x7fbbe80c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.684+0000 7fbbed000640 1 --2- 172.21.15.27:0/4162718955 >> v2:172.21.15.27:3300/0 conn(0x7fbbe80c7df0 0x7fbbe8155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.684+0000 7fbbed000640 1 -- 172.21.15.27:0/4162718955 >> v2:172.21.15.155:3300/0 conn(0x7fbbe8152f80 msgr2=0x7fbbe80c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.684+0000 7fbbed000640 1 --2- 172.21.15.27:0/4162718955 >> v2:172.21.15.155:3300/0 conn(0x7fbbe8152f80 0x7fbbe80c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.685+0000 7fbbed000640 1 -- 172.21.15.27:0/4162718955 >> v1:172.21.15.155:6789/0 conn(0x7fbbe80c7970 legacy=0x7fbbe8155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.684+0000 7fbbe6a00640 1 -- 172.21.15.27:0/4162718955 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669357659 0 0) 0x7fbbe80c7310 con 0x7fbbe80c7970 2024-03-28T19:25:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.685+0000 7fbbed000640 1 -- 172.21.15.27:0/4162718955 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8159300 con 0x7fbbe80c7df0 2024-03-28T19:25:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.686+0000 7fbbed000640 1 --2- 172.21.15.27:0/4162718955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7df0 0x7fbbe8155e40 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fbbe806c000 tx=0x7fbbe002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2127692d40371350 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.686+0000 7fbbe6a00640 1 -- 172.21.15.27:0/4162718955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0039070 con 0x7fbbe80c7df0 2024-03-28T19:25:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.686+0000 7fbbe6a00640 1 -- 172.21.15.27:0/4162718955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe002cbe0 con 0x7fbbe80c7df0 2024-03-28T19:25:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.686+0000 7fbbe6a00640 1 -- 172.21.15.27:0/4162718955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0034440 con 0x7fbbe80c7df0 2024-03-28T19:25:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.687+0000 7fbbef000640 1 -- 172.21.15.27:0/4162718955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7df0 msgr2=0x7fbbe8155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.687+0000 7fbbef000640 1 --2- 172.21.15.27:0/4162718955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7df0 0x7fbbe8155e40 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fbbe806c000 tx=0x7fbbe002be30 comp rx=0 tx=0).stop 2024-03-28T19:25:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.687+0000 7fbbef000640 1 -- 172.21.15.27:0/4162718955 shutdown_connections 2024-03-28T19:25:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.687+0000 7fbbef000640 1 --2- 172.21.15.27:0/4162718955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7df0 0x7fbbe8155e40 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.687+0000 7fbbef000640 1 --2- 172.21.15.27:0/4162718955 >> v2:172.21.15.155:3300/0 conn(0x7fbbe8152f80 0x7fbbe80c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.687+0000 7fbbef000640 1 -- 172.21.15.27:0/4162718955 >> 172.21.15.27:0/4162718955 conn(0x7fbbe806b6f0 msgr2=0x7fbbe80caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.688+0000 7fbbef000640 1 -- 172.21.15.27:0/4162718955 shutdown_connections 2024-03-28T19:25:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.688+0000 7fbbef000640 1 -- 172.21.15.27:0/4162718955 wait complete. 2024-03-28T19:25:00.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.690+0000 7fbbef000640 1 Processor -- start 2024-03-28T19:25:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.691+0000 7fbbef000640 1 -- start start 2024-03-28T19:25:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.693+0000 7fbbef000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7970 0x7fbbe80c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:00.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.693+0000 7fbbef000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbbe80c7df0 0x7fbbe8153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:00.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.693+0000 7fbbef000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbe8152f80 0x7fbbe8154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:00.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.694+0000 7fbbef000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe8159b00 con 0x7fbbe80c7970 2024-03-28T19:25:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.694+0000 7fbbef000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe80c5e20 con 0x7fbbe8152f80 2024-03-28T19:25:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.694+0000 7fbbe7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7970 0x7fbbe80c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.694+0000 7fbbe7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7970 0x7fbbe80c49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41928/0 (socket says 172.21.15.27:41928) 2024-03-28T19:25:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.694+0000 7fbbed000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbe8152f80 0x7fbbe8154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.694+0000 7fbbe7e00640 1 -- 172.21.15.27:0/2529146763 learned_addr learned my addr 172.21.15.27:0/2529146763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.694+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe80cbca0 con 0x7fbbe80c7df0 2024-03-28T19:25:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.694+0000 7fbbe7400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbbe80c7df0 0x7fbbe8153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:00.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.695+0000 7fbbe7e00640 1 -- 172.21.15.27:0/2529146763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbbe80c7df0 msgr2=0x7fbbe8153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:00.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.695+0000 7fbbe7e00640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbbe80c7df0 0x7fbbe8153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:00.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.695+0000 7fbbe7e00640 1 -- 172.21.15.27:0/2529146763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbe8152f80 msgr2=0x7fbbe8154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:00.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.695+0000 7fbbe7e00640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbe8152f80 0x7fbbe8154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:00.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.695+0000 7fbbe7e00640 1 -- 172.21.15.27:0/2529146763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe81eafb0 con 0x7fbbe80c7970 2024-03-28T19:25:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.697+0000 7fbbe7e00640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7970 0x7fbbe80c49c0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800e9b0 tx=0x7fbbd800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.697+0000 7fbbdfe00640 1 -- 172.21.15.27:0/2529146763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8017070 con 0x7fbbe80c7970 2024-03-28T19:25:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.697+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe80c2450 con 0x7fbbe80c7970 2024-03-28T19:25:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.698+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe80c2a10 con 0x7fbbe80c7970 2024-03-28T19:25:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.698+0000 7fbbdfe00640 1 -- 172.21.15.27:0/2529146763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd800c9a0 con 0x7fbbe80c7970 2024-03-28T19:25:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.698+0000 7fbbdfe00640 1 -- 172.21.15.27:0/2529146763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd800cd90 con 0x7fbbe80c7970 2024-03-28T19:25:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.701+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc0005290 con 0x7fbbe80c7970 2024-03-28T19:25:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.703+0000 7fbbdfe00640 1 -- 172.21.15.27:0/2529146763 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbbd801a070 con 0x7fbbe80c7970 2024-03-28T19:25:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.705+0000 7fbbdfe00640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbc806bc00 0x7fbbc806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.705+0000 7fbbdfe00640 1 -- 172.21.15.27:0/2529146763 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7fbbd808ba70 con 0x7fbbe80c7970 2024-03-28T19:25:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.711+0000 7fbbe7400640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbc806bc00 0x7fbbc806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.713+0000 7fbbe7400640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbc806bc00 0x7fbbc806e0c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbbe814e300 tx=0x7fbbd4007c00 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:00.716+0000 7fbbdfe00640 1 -- 172.21.15.27:0/2529146763 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbbd8058e30 con 0x7fbbe80c7970 2024-03-28T19:25:01.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.187+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbc0005580 con 0x7fbbe80c7970 2024-03-28T19:25:01.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.189+0000 7fbbdfe00640 1 -- 172.21.15.27:0/2529146763 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+5462 (secure 0 0 0) 0x7fbbd805ced0 con 0x7fbbe80c7970 2024-03-28T19:25:01.188 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:01.188 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:24:55.429514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:25:01.188 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 217 2024-03-28T19:25:01.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.192+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbc806bc00 msgr2=0x7fbbc806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:01.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.192+0000 7fbbef000640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbc806bc00 0x7fbbc806e0c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbbe814e300 tx=0x7fbbd4007c00 comp rx=0 tx=0).stop 2024-03-28T19:25:01.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.192+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7970 msgr2=0x7fbbe80c49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:01.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.192+0000 7fbbef000640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7970 0x7fbbe80c49c0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800e9b0 tx=0x7fbbd800ee70 comp rx=0 tx=0).stop 2024-03-28T19:25:01.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.193+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 shutdown_connections 2024-03-28T19:25:01.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.193+0000 7fbbef000640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbbc806bc00 0x7fbbc806e0c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:01.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.193+0000 7fbbef000640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbe8152f80 0x7fbbe8154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:01.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.193+0000 7fbbef000640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbbe80c7df0 0x7fbbe8153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:01.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.193+0000 7fbbef000640 1 --2- 172.21.15.27:0/2529146763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbe80c7970 0x7fbbe80c49c0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:01.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.193+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 >> 172.21.15.27:0/2529146763 conn(0x7fbbe806b6f0 msgr2=0x7fbbe814b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:01.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.193+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 shutdown_connections 2024-03-28T19:25:01.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:01.193+0000 7fbbef000640 1 -- 172.21.15.27:0/2529146763 wait complete. 2024-03-28T19:25:01.217 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:25:03.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:03.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.495+0000 7f7b32600640 1 Processor -- start 2024-03-28T19:25:03.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.495+0000 7f7b32600640 1 -- start start 2024-03-28T19:25:03.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.497+0000 7f7b32600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7b2c155be0 0x7f7b2c151a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.497+0000 7f7b32600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0aa6e0 con 0x7f7b2c13aaa0 2024-03-28T19:25:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.498+0000 7f7b32600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c155100 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.498+0000 7f7b31200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7b2c13aaa0 0x7f7b2c0c7230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36456/0 (socket says 172.21.15.27:36456) 2024-03-28T19:25:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.498+0000 7f7b31200640 1 -- 172.21.15.27:0/1844595856 learned_addr learned my addr 172.21.15.27:0/1844595856 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.498+0000 7f7b32600640 1 -- 172.21.15.27:0/1844595856 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c062a80 con 0x7f7b2c155be0 2024-03-28T19:25:03.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.498+0000 7f7b2be00640 1 --2- 172.21.15.27:0/1844595856 >> v2:172.21.15.27:3300/0 conn(0x7f7b2c155be0 0x7f7b2c151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.498+0000 7f7b2b400640 1 -- 172.21.15.27:0/1844595856 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404040121 0 0) 0x7f7b2c0aa6e0 con 0x7f7b2c13aaa0 2024-03-28T19:25:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.499+0000 7f7b2b400640 1 -- 172.21.15.27:0/1844595856 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b10003520 con 0x7f7b2c13aaa0 2024-03-28T19:25:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.499+0000 7f7b2be00640 1 -- 172.21.15.27:0/1844595856 >> v1:172.21.15.155:6789/0 conn(0x7f7b2c06ebc0 legacy=0x7f7b2c062370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.499+0000 7f7b2be00640 1 -- 172.21.15.27:0/1844595856 >> v1:172.21.15.27:6789/0 conn(0x7f7b2c13aaa0 legacy=0x7f7b2c0c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.499+0000 7f7b2be00640 1 -- 172.21.15.27:0/1844595856 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c1553e0 con 0x7f7b2c155be0 2024-03-28T19:25:03.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.500+0000 7f7b2be00640 1 --2- 172.21.15.27:0/1844595856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c155be0 0x7f7b2c151a60 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0097a0 tx=0x7f7b1c02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6b22d0c7ca5b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.500+0000 7f7b2b400640 1 -- 172.21.15.27:0/1844595856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c037070 con 0x7f7b2c155be0 2024-03-28T19:25:03.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.500+0000 7f7b2b400640 1 -- 172.21.15.27:0/1844595856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c030710 con 0x7f7b2c155be0 2024-03-28T19:25:03.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.500+0000 7f7b2b400640 1 -- 172.21.15.27:0/1844595856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c030ae0 con 0x7f7b2c155be0 2024-03-28T19:25:03.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.501+0000 7f7b32600640 1 -- 172.21.15.27:0/1844595856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c155be0 msgr2=0x7f7b2c151a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.501+0000 7f7b32600640 1 --2- 172.21.15.27:0/1844595856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c155be0 0x7f7b2c151a60 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c0097a0 tx=0x7f7b1c02b710 comp rx=0 tx=0).stop 2024-03-28T19:25:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.502+0000 7f7b32600640 1 -- 172.21.15.27:0/1844595856 shutdown_connections 2024-03-28T19:25:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.502+0000 7f7b32600640 1 --2- 172.21.15.27:0/1844595856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c155be0 0x7f7b2c151a60 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.502+0000 7f7b32600640 1 -- 172.21.15.27:0/1844595856 >> 172.21.15.27:0/1844595856 conn(0x7f7b2c06ced0 msgr2=0x7f7b2c06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.502+0000 7f7b32600640 1 -- 172.21.15.27:0/1844595856 shutdown_connections 2024-03-28T19:25:03.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.502+0000 7f7b32600640 1 -- 172.21.15.27:0/1844595856 wait complete. 2024-03-28T19:25:03.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.505+0000 7f7b32600640 1 Processor -- start 2024-03-28T19:25:03.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.505+0000 7f7b32600640 1 -- start start 2024-03-28T19:25:03.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.506+0000 7f7b32600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c06ebc0 0x7f7b2c139a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:03.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.506+0000 7f7b32600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aaa0 0x7f7b2c1500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:03.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.506+0000 7f7b32600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c155be0 0x7f7b2c138a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:03.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.506+0000 7f7b32600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0aa390 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.506+0000 7f7b32600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0aa9c0 con 0x7f7b2c13aaa0 2024-03-28T19:25:03.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.506+0000 7f7b32600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0c7940 con 0x7f7b2c155be0 2024-03-28T19:25:03.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.507+0000 7f7b31200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c06ebc0 0x7f7b2c139a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.507+0000 7f7b31200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c06ebc0 0x7f7b2c139a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41954/0 (socket says 172.21.15.27:41954) 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.507+0000 7f7b31200640 1 -- 172.21.15.27:0/1725000311 learned_addr learned my addr 172.21.15.27:0/1725000311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.507+0000 7f7b2be00640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aaa0 0x7f7b2c1500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.507+0000 7f7b31c00640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c155be0 0x7f7b2c138a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.507+0000 7f7b31200640 1 -- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c155be0 msgr2=0x7f7b2c138a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.507+0000 7f7b31200640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c155be0 0x7f7b2c138a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.508+0000 7f7b31200640 1 -- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aaa0 msgr2=0x7f7b2c1500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.508+0000 7f7b31200640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aaa0 0x7f7b2c1500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.508+0000 7f7b31200640 1 -- 172.21.15.27:0/1725000311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c1dc950 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.508+0000 7f7b2be00640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aaa0 0x7f7b2c1500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:25:03.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.508+0000 7f7b31200640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c06ebc0 0x7f7b2c139a70 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400c8c0 tx=0x7f7b2400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.508+0000 7f7b28c00640 1 -- 172.21.15.27:0/1725000311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24018070 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.508+0000 7f7b28c00640 1 -- 172.21.15.27:0/1725000311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2400a640 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.509+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c152500 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.509+0000 7f7b28c00640 1 -- 172.21.15.27:0/1725000311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2400a9f0 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.509+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c14f8d0 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.510+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c067dd0 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.512+0000 7f7b28c00640 1 -- 172.21.15.27:0/1725000311 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7b2400ab90 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.514+0000 7f7b28c00640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b0c06bbd0 0x7f7b0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.514+0000 7f7b28c00640 1 -- 172.21.15.27:0/1725000311 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f7b2408b190 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.516+0000 7f7b2be00640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b0c06bbd0 0x7f7b0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:03.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.517+0000 7f7b2be00640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b0c06bbd0 0x7f7b0c06e090 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c034000 tx=0x7f7b1c02f000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.520+0000 7f7b28c00640 1 -- 172.21.15.27:0/1725000311 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7b2c1dc950 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.916+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b2c069640 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.916+0000 7f7b28c00640 1 -- 172.21.15.27:0/1725000311 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+5462 (secure 0 0 0) 0x7f7b24058550 con 0x7f7b2c06ebc0 2024-03-28T19:25:03.917 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:03.917 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:24:55.429514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:25:03.917 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 217 2024-03-28T19:25:03.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.920+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b0c06bbd0 msgr2=0x7f7b0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:03.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.920+0000 7f7b32600640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b0c06bbd0 0x7f7b0c06e090 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c034000 tx=0x7f7b1c02f000 comp rx=0 tx=0).stop 2024-03-28T19:25:03.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.920+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c06ebc0 msgr2=0x7f7b2c139a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:03.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.920+0000 7f7b32600640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c06ebc0 0x7f7b2c139a70 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400c8c0 tx=0x7f7b2400cd80 comp rx=0 tx=0).stop 2024-03-28T19:25:03.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.921+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 shutdown_connections 2024-03-28T19:25:03.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.921+0000 7f7b32600640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7b0c06bbd0 0x7f7b0c06e090 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:03.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.921+0000 7f7b32600640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c155be0 0x7f7b2c138a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:03.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.921+0000 7f7b32600640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aaa0 0x7f7b2c1500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:03.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.921+0000 7f7b32600640 1 --2- 172.21.15.27:0/1725000311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b2c06ebc0 0x7f7b2c139a70 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:03.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.921+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 >> 172.21.15.27:0/1725000311 conn(0x7f7b2c06ced0 msgr2=0x7f7b2c06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:03.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.922+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 shutdown_connections 2024-03-28T19:25:03.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:03.922+0000 7f7b32600640 1 -- 172.21.15.27:0/1725000311 wait complete. 2024-03-28T19:25:03.942 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:25:05.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.203+0000 7f6fdf200640 1 Processor -- start 2024-03-28T19:25:06.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.203+0000 7f6fdf200640 1 -- start start 2024-03-28T19:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.205+0000 7f6fdf200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6fe00c58c0 0x7f6fe00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.205+0000 7f6fdf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe00be5d0 0x7f6fe0155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.205+0000 7f6fdf200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe0156230 con 0x7f6fe00c8ed0 2024-03-28T19:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.205+0000 7f6fdf200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe0158bd0 con 0x7f6fe00c58c0 2024-03-28T19:25:06.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.205+0000 7f6fdf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0158eb0 con 0x7f6fe00be5d0 2024-03-28T19:25:06.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.205+0000 7f6fdd400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6fe00c8ed0 0x7f6fe01555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39312/0 (socket says 172.21.15.27:39312) 2024-03-28T19:25:06.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.205+0000 7f6fdd400640 1 -- 172.21.15.27:0/3046039712 learned_addr learned my addr 172.21.15.27:0/3046039712 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:06.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.206+0000 7f6fdde00640 1 --2- 172.21.15.27:0/3046039712 >> v2:172.21.15.155:3300/0 conn(0x7f6fe00c58c0 0x7f6fe00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:06.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.206+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489276967 0 0) 0x7f6fe0156230 con 0x7f6fe00c8ed0 2024-03-28T19:25:06.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.206+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4003520 con 0x7f6fe00c8ed0 2024-03-28T19:25:06.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.206+0000 7f6fde800640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe00be5d0 0x7f6fe0155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:06.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.206+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608556299 0 0) 0x7f6fc4003520 con 0x7f6fe00c8ed0 2024-03-28T19:25:06.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.206+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe00be5d0 msgr2=0x7f6fe0155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe00be5d0 0x7f6fe0155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 >> v2:172.21.15.155:3300/0 conn(0x7f6fe00c58c0 msgr2=0x7f6fe00c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 --2- 172.21.15.27:0/3046039712 >> v2:172.21.15.155:3300/0 conn(0x7f6fe00c58c0 0x7f6fe00c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0159190 con 0x7f6fe00c8ed0 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f6fcc001670 con 0x7f6fe00c8ed0 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdde00640 1 --2- 172.21.15.27:0/3046039712 >> v2:172.21.15.155:3300/0 conn(0x7f6fe00c58c0 0x7f6fe00c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 >> v1:172.21.15.27:6789/0 conn(0x7f6fe00c8ed0 legacy=0x7f6fe01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe00be5d0 0x7f6fc40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fc4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fc4004d40 0x7f6fc40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0158bd0 con 0x7f6fc4003ef0 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0158eb0 con 0x7f6fc4004d40 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc400ba70 con 0x7f6fe00be5d0 2024-03-28T19:25:06.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.207+0000 7f6fdd400640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fc4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:06.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.208+0000 7f6fdde00640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe00be5d0 0x7f6fc40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.208+0000 7f6fdd400640 1 -- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe00be5d0 msgr2=0x7f6fc40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.208+0000 7f6fde800640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fc4004d40 0x7f6fc40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.208+0000 7f6fdd400640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe00be5d0 0x7f6fc40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.208+0000 7f6fdd400640 1 -- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fc4004d40 msgr2=0x7f6fc40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.208+0000 7f6fdd400640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fc4004d40 0x7f6fc40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.208+0000 7f6fdd400640 1 -- 172.21.15.27:0/3046039712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4003520 con 0x7f6fc4003ef0 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.208+0000 7f6fde800640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fc4004d40 0x7f6fc40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.209+0000 7f6fdd400640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fc4004800 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc005060 tx=0x7f6fc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6759fe2eed1738d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.209+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc004c60 con 0x7f6fc4003ef0 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.209+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc005b90 con 0x7f6fc4003ef0 2024-03-28T19:25:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.209+0000 7f6fdca00640 1 -- 172.21.15.27:0/3046039712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0080d0 con 0x7f6fc4003ef0 2024-03-28T19:25:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.210+0000 7f6fdf200640 1 -- 172.21.15.27:0/3046039712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 msgr2=0x7f6fc4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.210+0000 7f6fdf200640 1 --2- 172.21.15.27:0/3046039712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fc4004800 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc005060 tx=0x7f6fc4005830 comp rx=0 tx=0).stop 2024-03-28T19:25:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.210+0000 7f6fde800640 1 -- 172.21.15.27:0/3046039712 reap_dead start 2024-03-28T19:25:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.211+0000 7f6fdf200640 1 -- 172.21.15.27:0/3046039712 shutdown_connections 2024-03-28T19:25:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.211+0000 7f6fdf200640 1 -- 172.21.15.27:0/3046039712 >> 172.21.15.27:0/3046039712 conn(0x7f6fe006b6f0 msgr2=0x7f6fe00acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.211+0000 7f6fdf200640 1 -- 172.21.15.27:0/3046039712 shutdown_connections 2024-03-28T19:25:06.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.211+0000 7f6fdf200640 1 -- 172.21.15.27:0/3046039712 wait complete. 2024-03-28T19:25:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.213+0000 7f6fdf200640 1 Processor -- start 2024-03-28T19:25:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.213+0000 7f6fdf200640 1 -- start start 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdf200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fe00af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdf200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe01ef260 0x7f6fe00ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdf200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe0154e50 0x7f6fe00aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdf200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00b0820 con 0x7f6fc4003ef0 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdf200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0159750 con 0x7f6fe0154e50 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdf200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00b09a0 con 0x7f6fe01ef260 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fe00af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fe00af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42408/0 (socket says 172.21.15.27:42408) 2024-03-28T19:25:06.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.215+0000 7f6fdde00640 1 -- 172.21.15.27:0/2465014156 learned_addr learned my addr 172.21.15.27:0/2465014156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:06.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.216+0000 7f6fdd400640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe01ef260 0x7f6fe00ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.216+0000 7f6fdde00640 1 -- 172.21.15.27:0/2465014156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe01ef260 msgr2=0x7f6fe00ac5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.216+0000 7f6fdde00640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe01ef260 0x7f6fe00ac5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.216+0000 7f6fdde00640 1 -- 172.21.15.27:0/2465014156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe0154e50 msgr2=0x7f6fe00aaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.216+0000 7f6fdde00640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe0154e50 0x7f6fe00aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.216+0000 7f6fdde00640 1 -- 172.21.15.27:0/2465014156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0150d50 con 0x7f6fc4003ef0 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.216+0000 7f6fdd400640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe01ef260 0x7f6fe00ac5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.216+0000 7f6fdde00640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fe00af040 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00108c0 tx=0x7f6fd0010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.217+0000 7f6fd6000640 1 -- 172.21.15.27:0/2465014156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd001a070 con 0x7f6fc4003ef0 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.217+0000 7f6fd6000640 1 -- 172.21.15.27:0/2465014156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd000ca60 con 0x7f6fc4003ef0 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.217+0000 7f6fd6000640 1 -- 172.21.15.27:0/2465014156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000f040 con 0x7f6fc4003ef0 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.217+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe00c4960 con 0x7f6fc4003ef0 2024-03-28T19:25:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.217+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe00afe30 con 0x7f6fc4003ef0 2024-03-28T19:25:06.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.219+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb8005290 con 0x7f6fc4003ef0 2024-03-28T19:25:06.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.220+0000 7f6fd6000640 1 -- 172.21.15.27:0/2465014156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6fd0015470 con 0x7f6fc4003ef0 2024-03-28T19:25:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.222+0000 7f6fd6000640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6fc006bbd0 0x7f6fc006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.222+0000 7f6fd6000640 1 -- 172.21.15.27:0/2465014156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f6fd008c150 con 0x7f6fc4003ef0 2024-03-28T19:25:06.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.222+0000 7f6fdd400640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6fc006bbd0 0x7f6fc006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:06.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.224+0000 7f6fdd400640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6fc006bbd0 0x7f6fc006e090 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00c9300 tx=0x7f6fcc037040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:06.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.228+0000 7f6fd6000640 1 -- 172.21.15.27:0/2465014156 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6fd0059ad0 con 0x7f6fc4003ef0 2024-03-28T19:25:06.234 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 23% (54/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 42% (97/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 58% (133/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 77% (177/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 96% (220/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2024-03-28T19:25:06.515 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 071881d7e55 v0.28 2024-03-28T19:25:06.524 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.28 2024-03-28T19:25:06.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.572+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fb8005580 con 0x7f6fc4003ef0 2024-03-28T19:25:06.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.574+0000 7f6fd6000640 1 -- 172.21.15.27:0/2465014156 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+5462 (secure 0 0 0) 0x7f6fd00593d0 con 0x7f6fc4003ef0 2024-03-28T19:25:06.574 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:06.574 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:24:55.429514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:25:06.574 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 217 2024-03-28T19:25:06.579 INFO:tasks.workunit.client.0.smithi027.stdout:v0.29 2024-03-28T19:25:06.579 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:25:06.579 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 29 -eq 48 ']' 2024-03-28T19:25:06.579 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.29 2024-03-28T19:25:06.580 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.29 2024-03-28T19:25:06.580 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.29 2024-03-28T19:25:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6fc006bbd0 msgr2=0x7f6fc006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6fc006bbd0 0x7f6fc006e090 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00c9300 tx=0x7f6fcc037040 comp rx=0 tx=0).stop 2024-03-28T19:25:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 msgr2=0x7f6fe00af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fe00af040 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00108c0 tx=0x7f6fd0010d80 comp rx=0 tx=0).stop 2024-03-28T19:25:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 shutdown_connections 2024-03-28T19:25:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6fc006bbd0 0x7f6fc006e090 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe0154e50 0x7f6fe00aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe01ef260 0x7f6fe00ac5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 --2- 172.21.15.27:0/2465014156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc4003ef0 0x7f6fe00af040 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.578+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 >> 172.21.15.27:0/2465014156 conn(0x7f6fe006b6f0 msgr2=0x7f6fe01dec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.579+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 shutdown_connections 2024-03-28T19:25:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:06.579+0000 7f6fdf200640 1 -- 172.21.15.27:0/2465014156 wait complete. 2024-03-28T19:25:06.598 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:25:08.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:08.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.865+0000 7fbc67a00640 1 Processor -- start 2024-03-28T19:25:08.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.865+0000 7fbc67a00640 1 -- start start 2024-03-28T19:25:08.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.867+0000 7fbc67a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc600c4060 0x7fbc6006f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:08.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.867+0000 7fbc67a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc600c4440 0x7fbc600c5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:08.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.867+0000 7fbc67a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc60152560 con 0x7fbc600c5590 2024-03-28T19:25:08.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.867+0000 7fbc67a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc60153170 con 0x7fbc600c4060 2024-03-28T19:25:08.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.867+0000 7fbc67a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60152840 con 0x7fbc600c4440 2024-03-28T19:25:08.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.868+0000 7fbc65000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc600c4060 0x7fbc6006f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:08.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.868+0000 7fbc65000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc600c4060 0x7fbc6006f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42420/0 (socket says 172.21.15.27:42420) 2024-03-28T19:25:08.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.868+0000 7fbc65000640 1 -- 172.21.15.27:0/2812502964 learned_addr learned my addr 172.21.15.27:0/2812502964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:08.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.868+0000 7fbc5fe00640 1 --2- 172.21.15.27:0/2812502964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc600c4440 0x7fbc600c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:08.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.869+0000 7fbc65000640 1 -- 172.21.15.27:0/2812502964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc600c4440 msgr2=0x7fbc600c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:08.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.869+0000 7fbc65000640 1 --2- 172.21.15.27:0/2812502964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc600c4440 0x7fbc600c5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:08.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.869+0000 7fbc65000640 1 -- 172.21.15.27:0/2812502964 >> v1:172.21.15.155:6789/0 conn(0x7fbc600c5590 legacy=0x7fbc60155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:08.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.869+0000 7fbc65000640 1 -- 172.21.15.27:0/2812502964 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60152b20 con 0x7fbc600c4060 2024-03-28T19:25:08.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.871+0000 7fbc65000640 1 --2- 172.21.15.27:0/2812502964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4060 0x7fbc6006f900 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400bb10 tx=0x7fbc5402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6984e9c3843973a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:08.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.871+0000 7fbc5f400640 1 -- 172.21.15.27:0/2812502964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5403b070 con 0x7fbc600c4060 2024-03-28T19:25:08.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.872+0000 7fbc5f400640 1 -- 172.21.15.27:0/2812502964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc54036440 con 0x7fbc600c4060 2024-03-28T19:25:08.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.872+0000 7fbc5f400640 1 -- 172.21.15.27:0/2812502964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc54036810 con 0x7fbc600c4060 2024-03-28T19:25:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.873+0000 7fbc67a00640 1 -- 172.21.15.27:0/2812502964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4060 msgr2=0x7fbc6006f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.873+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2812502964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4060 0x7fbc6006f900 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400bb10 tx=0x7fbc5402e0c0 comp rx=0 tx=0).stop 2024-03-28T19:25:08.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.873+0000 7fbc67a00640 1 -- 172.21.15.27:0/2812502964 shutdown_connections 2024-03-28T19:25:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.873+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2812502964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc600c4440 0x7fbc600c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.873+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2812502964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4060 0x7fbc6006f900 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.873+0000 7fbc67a00640 1 -- 172.21.15.27:0/2812502964 >> 172.21.15.27:0/2812502964 conn(0x7fbc6006c5f0 msgr2=0x7fbc6006ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.874+0000 7fbc67a00640 1 -- 172.21.15.27:0/2812502964 shutdown_connections 2024-03-28T19:25:08.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.874+0000 7fbc67a00640 1 -- 172.21.15.27:0/2812502964 wait complete. 2024-03-28T19:25:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.877+0000 7fbc67a00640 1 Processor -- start 2024-03-28T19:25:08.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.877+0000 7fbc67a00640 1 -- start start 2024-03-28T19:25:08.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.880+0000 7fbc67a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4440 0x7fbc601eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:08.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.880+0000 7fbc67a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc600c5590 0x7fbc60146250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:08.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.880+0000 7fbc67a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc60147e30 0x7fbc601e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:08.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.880+0000 7fbc67a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600aa0d0 con 0x7fbc600c4440 2024-03-28T19:25:08.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.881+0000 7fbc67a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600a9f50 con 0x7fbc600c5590 2024-03-28T19:25:08.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.881+0000 7fbc67a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600aa250 con 0x7fbc60147e30 2024-03-28T19:25:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.881+0000 7fbc5fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc600c5590 0x7fbc60146250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.881+0000 7fbc65000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4440 0x7fbc601eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.881+0000 7fbc5fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc600c5590 0x7fbc60146250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:48150/0 (socket says 172.21.15.27:48150) 2024-03-28T19:25:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.881+0000 7fbc65000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4440 0x7fbc601eb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42434/0 (socket says 172.21.15.27:42434) 2024-03-28T19:25:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.881+0000 7fbc5fe00640 1 -- 172.21.15.27:0/2669028756 learned_addr learned my addr 172.21.15.27:0/2669028756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:08.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.881+0000 7fbc65a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc60147e30 0x7fbc601e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.883+0000 7fbc65a00640 1 -- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc600c5590 msgr2=0x7fbc60146250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.883+0000 7fbc65a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc600c5590 0x7fbc60146250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.883+0000 7fbc65a00640 1 -- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4440 msgr2=0x7fbc601eb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.883+0000 7fbc65a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4440 0x7fbc601eb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.883+0000 7fbc65a00640 1 -- 172.21.15.27:0/2669028756 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc601503a0 con 0x7fbc60147e30 2024-03-28T19:25:08.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.883+0000 7fbc5fe00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc600c5590 0x7fbc60146250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:25:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.883+0000 7fbc65a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc60147e30 0x7fbc601e7a40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c500 tx=0x7fbc5800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.884+0000 7fbc5cc00640 1 -- 172.21.15.27:0/2669028756 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58016020 con 0x7fbc60147e30 2024-03-28T19:25:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.884+0000 7fbc5cc00640 1 -- 172.21.15.27:0/2669028756 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc58006d50 con 0x7fbc60147e30 2024-03-28T19:25:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.884+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc60149010 con 0x7fbc60147e30 2024-03-28T19:25:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.884+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc601e7240 con 0x7fbc60147e30 2024-03-28T19:25:08.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.884+0000 7fbc5cc00640 1 -- 172.21.15.27:0/2669028756 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58003bb0 con 0x7fbc60147e30 2024-03-28T19:25:08.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.886+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc601521e0 con 0x7fbc60147e30 2024-03-28T19:25:08.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.889+0000 7fbc5cc00640 1 -- 172.21.15.27:0/2669028756 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbc580050d0 con 0x7fbc60147e30 2024-03-28T19:25:08.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.891+0000 7fbc5cc00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc4006bbd0 0x7fbc4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:08.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.891+0000 7fbc5cc00640 1 -- 172.21.15.27:0/2669028756 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7fbc5808c3a0 con 0x7fbc60147e30 2024-03-28T19:25:08.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.892+0000 7fbc65000640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc4006bbd0 0x7fbc4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:08.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.894+0000 7fbc65000640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc4006bbd0 0x7fbc4006e090 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbc54038000 tx=0x7fbc54008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:08.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:08.906+0000 7fbc5cc00640 1 -- 172.21.15.27:0/2669028756 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbc601521e0 con 0x7fbc60147e30 2024-03-28T19:25:09.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.360+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc6004f400 con 0x7fbc60147e30 2024-03-28T19:25:09.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.362+0000 7fbc5cc00640 1 -- 172.21.15.27:0/2669028756 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5470 (secure 0 0 0) 0x7fbc58020020 con 0x7fbc60147e30 2024-03-28T19:25:09.362 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:09.362 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:09.362 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 220 2024-03-28T19:25:09.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc4006bbd0 msgr2=0x7fbc4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:09.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc4006bbd0 0x7fbc4006e090 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbc54038000 tx=0x7fbc54008040 comp rx=0 tx=0).stop 2024-03-28T19:25:09.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc60147e30 msgr2=0x7fbc601e7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc60147e30 0x7fbc601e7a40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c500 tx=0x7fbc5800c9c0 comp rx=0 tx=0).stop 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 shutdown_connections 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbc4006bbd0 0x7fbc4006e090 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbc60147e30 0x7fbc601e7a40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc600c5590 0x7fbc60146250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 --2- 172.21.15.27:0/2669028756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc600c4440 0x7fbc601eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 >> 172.21.15.27:0/2669028756 conn(0x7fbc6006c5f0 msgr2=0x7fbc6006c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.364+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 shutdown_connections 2024-03-28T19:25:09.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:09.365+0000 7fbc67a00640 1 -- 172.21.15.27:0/2669028756 wait complete. 2024-03-28T19:25:09.384 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=7537 2024-03-28T19:25:09.384 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.b 2024-03-28T19:25:10.386 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-03-28T19:25:10.386 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:25:10.386 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:25:10.394 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:25:10.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.664+0000 7f0762600640 1 Processor -- start 2024-03-28T19:25:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.664+0000 7f0762600640 1 -- start start 2024-03-28T19:25:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.666+0000 7f0762600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f075c0c58c0 0x7f075c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.666+0000 7f0762600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0c8ed0 0x7f075c1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.666+0000 7f0762600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f075c0be5d0 0x7f075c155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.666+0000 7f0762600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075c158bd0 con 0x7f075c0be5d0 2024-03-28T19:25:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.666+0000 7f0762600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075c158eb0 con 0x7f075c0c58c0 2024-03-28T19:25:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.666+0000 7f0762600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c159190 con 0x7f075c0c8ed0 2024-03-28T19:25:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.667+0000 7f0761200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f075c0c58c0 0x7f075c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.667+0000 7f075be00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0c8ed0 0x7f075c1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.667+0000 7f0761200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f075c0c58c0 0x7f075c0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:48166/0 (socket says 172.21.15.27:48166) 2024-03-28T19:25:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.667+0000 7f0761200640 1 -- 172.21.15.27:0/860468393 learned_addr learned my addr 172.21.15.27:0/860468393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.667+0000 7f0761c00640 1 --2- 172.21.15.27:0/860468393 >> v2:172.21.15.27:3300/0 conn(0x7f075c0be5d0 0x7f075c155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.668+0000 7f0761c00640 1 -- 172.21.15.27:0/860468393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0c8ed0 msgr2=0x7f075c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.668+0000 7f0761c00640 1 --2- 172.21.15.27:0/860468393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0c8ed0 0x7f075c1555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.668+0000 7f0761c00640 1 -- 172.21.15.27:0/860468393 >> v2:172.21.15.155:3300/0 conn(0x7f075c0c58c0 msgr2=0x7f075c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.668+0000 7f0761c00640 1 --2- 172.21.15.27:0/860468393 >> v2:172.21.15.155:3300/0 conn(0x7f075c0c58c0 0x7f075c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.668+0000 7f0761c00640 1 -- 172.21.15.27:0/860468393 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c159470 con 0x7f075c0be5d0 2024-03-28T19:25:10.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.670+0000 7f0761c00640 1 --2- 172.21.15.27:0/860468393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0be5d0 0x7f075c155b20 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f075c06c060 tx=0x7f075402bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e272e6a4416580e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:10.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.670+0000 7f075b400640 1 -- 172.21.15.27:0/860468393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754039070 con 0x7f075c0be5d0 2024-03-28T19:25:10.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.670+0000 7f075b400640 1 -- 172.21.15.27:0/860468393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075402cb50 con 0x7f075c0be5d0 2024-03-28T19:25:10.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.671+0000 7f075b400640 1 -- 172.21.15.27:0/860468393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754034440 con 0x7f075c0be5d0 2024-03-28T19:25:10.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 -- 172.21.15.27:0/860468393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0be5d0 msgr2=0x7f075c155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:10.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 --2- 172.21.15.27:0/860468393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0be5d0 0x7f075c155b20 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f075c06c060 tx=0x7f075402bda0 comp rx=0 tx=0).stop 2024-03-28T19:25:10.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 -- 172.21.15.27:0/860468393 shutdown_connections 2024-03-28T19:25:10.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 --2- 172.21.15.27:0/860468393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0be5d0 0x7f075c155b20 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:10.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 --2- 172.21.15.27:0/860468393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0c8ed0 0x7f075c1555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:10.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 --2- 172.21.15.27:0/860468393 >> v2:172.21.15.155:3300/0 conn(0x7f075c0c58c0 0x7f075c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:10.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 -- 172.21.15.27:0/860468393 >> 172.21.15.27:0/860468393 conn(0x7f075c06b6f0 msgr2=0x7f075c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:10.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 -- 172.21.15.27:0/860468393 shutdown_connections 2024-03-28T19:25:10.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.672+0000 7f0762600640 1 -- 172.21.15.27:0/860468393 wait complete. 2024-03-28T19:25:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.674+0000 7f0762600640 1 Processor -- start 2024-03-28T19:25:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.674+0000 7f0762600640 1 -- start start 2024-03-28T19:25:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.675+0000 7f0762600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0be5d0 0x7f075c0ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.675+0000 7f0762600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0c58c0 0x7f075c1efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.675+0000 7f0762600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f075c0c8ed0 0x7f075c1f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.675+0000 7f0762600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0b09a0 con 0x7f075c0c58c0 2024-03-28T19:25:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.675+0000 7f0762600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0b0820 con 0x7f075c0c8ed0 2024-03-28T19:25:10.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.675+0000 7f0762600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0ad7d0 con 0x7f075c0be5d0 2024-03-28T19:25:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f075be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0c58c0 0x7f075c1efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f075be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0c58c0 0x7f075c1efb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42460/0 (socket says 172.21.15.27:42460) 2024-03-28T19:25:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f075be00640 1 -- 172.21.15.27:0/3239561304 learned_addr learned my addr 172.21.15.27:0/3239561304 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f0761c00640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f075c0c8ed0 0x7f075c1f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f0761200640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0be5d0 0x7f075c0ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:10.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f075be00640 1 -- 172.21.15.27:0/3239561304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0be5d0 msgr2=0x7f075c0ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f075be00640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0be5d0 0x7f075c0ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f075be00640 1 -- 172.21.15.27:0/3239561304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f075c0c8ed0 msgr2=0x7f075c1f00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f075be00640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f075c0c8ed0 0x7f075c1f00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.676+0000 7f075be00640 1 -- 172.21.15.27:0/3239561304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c0c3fe0 con 0x7f075c0c58c0 2024-03-28T19:25:10.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.677+0000 7f075be00640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0c58c0 0x7f075c1efb60 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f074800e9b0 tx=0x7f074800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.678+0000 7f0758c00640 1 -- 172.21.15.27:0/3239561304 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748017070 con 0x7f075c0c58c0 2024-03-28T19:25:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.678+0000 7f0758c00640 1 -- 172.21.15.27:0/3239561304 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074800c9a0 con 0x7f075c0c58c0 2024-03-28T19:25:10.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.678+0000 7f0758c00640 1 -- 172.21.15.27:0/3239561304 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074800cd90 con 0x7f075c0c58c0 2024-03-28T19:25:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.679+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c1ef6a0 con 0x7f075c0c58c0 2024-03-28T19:25:10.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.679+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c0c4250 con 0x7f075c0c58c0 2024-03-28T19:25:10.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.680+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075c069640 con 0x7f075c0c58c0 2024-03-28T19:25:10.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.683+0000 7f0758c00640 1 -- 172.21.15.27:0/3239561304 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f074801a070 con 0x7f075c0c58c0 2024-03-28T19:25:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.684+0000 7f0758c00640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f073c06bcd0 0x7f073c06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.684+0000 7f0758c00640 1 -- 172.21.15.27:0/3239561304 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f074808bb80 con 0x7f075c0c58c0 2024-03-28T19:25:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.684+0000 7f0761200640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f073c06bcd0 0x7f073c06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:10.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.685+0000 7f0761200640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f073c06bcd0 0x7f073c06e190 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0750009a40 tx=0x7f0750008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:10.690+0000 7f0758c00640 1 -- 172.21.15.27:0/3239561304 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f075c1ef6a0 con 0x7f075c0c58c0 2024-03-28T19:25:10.776 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:25:11.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.083+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f075c156580 con 0x7f075c0c58c0 2024-03-28T19:25:11.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.084+0000 7f0758c00640 1 -- 172.21.15.27:0/3239561304 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5470 (secure 0 0 0) 0x7f0748059580 con 0x7f075c0c58c0 2024-03-28T19:25:11.084 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:11.084 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:11.085 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 220 2024-03-28T19:25:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f073c06bcd0 msgr2=0x7f073c06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f073c06bcd0 0x7f073c06e190 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0750009a40 tx=0x7f0750008040 comp rx=0 tx=0).stop 2024-03-28T19:25:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0c58c0 msgr2=0x7f075c1efb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0c58c0 0x7f075c1efb60 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f074800e9b0 tx=0x7f074800ee70 comp rx=0 tx=0).stop 2024-03-28T19:25:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 shutdown_connections 2024-03-28T19:25:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f073c06bcd0 0x7f073c06e190 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:11.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f075c0c8ed0 0x7f075c1f00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:11.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0c58c0 0x7f075c1efb60 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:11.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 --2- 172.21.15.27:0/3239561304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f075c0be5d0 0x7f075c0ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:11.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 >> 172.21.15.27:0/3239561304 conn(0x7f075c06b6f0 msgr2=0x7f075c0afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:11.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 shutdown_connections 2024-03-28T19:25:11.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:11.088+0000 7f0762600640 1 -- 172.21.15.27:0/3239561304 wait complete. 2024-03-28T19:25:11.111 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-03-28T19:25:12.581 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 27% (44/159) Updating files: 28% (45/159) Updating files: 29% (47/159) Updating files: 30% (48/159) Updating files: 31% (50/159) Updating files: 32% (51/159) Updating files: 33% (53/159) Updating files: 34% (55/159) Updating files: 35% (56/159) Updating files: 36% (58/159) Updating files: 37% (59/159) Updating files: 38% (61/159) Updating files: 39% (63/159) Updating files: 40% (64/159) Updating files: 41% (66/159) Updating files: 42% (67/159) Updating files: 43% (69/159) Updating files: 44% (70/159) Updating files: 45% (72/159) Updating files: 46% (74/159) Updating files: 47% (75/159) Updating files: 48% (77/159) Updating files: 49% (78/159) Updating files: 50% (80/159) Updating files: 51% (82/159) Updating files: 52% (83/159) Updating files: 53% (85/159) Updating files: 54% (86/159) Updating files: 55% (88/159) Updating files: 56% (90/159) Updating files: 57% (91/159) Updating files: 58% (93/159) Updating files: 59% (94/159) Updating files: 60% (96/159) Updating files: 61% (97/159) Updating files: 62% (99/159) Updating files: 63% (101/159) Updating files: 64% (102/159) Updating files: 65% (104/159) Updating files: 66% (105/159) Updating files: 67% (107/159) Updating files: 68% (109/159) Updating files: 69% (110/159) Updating files: 70% (112/159) Updating files: 71% (113/159) Updating files: 72% (115/159) Updating files: 73% (117/159) Updating files: 74% (118/159) Updating files: 75% (120/159) Updating files: 76% (121/159) Updating files: 77% (123/159) Updating files: 78% (125/159) Updating files: 79% (126/159) Updating files: 80% (128/159) Updating files: 81% (129/159) Updating files: 81% (130/159) Updating files: 82% (131/159) Updating files: 83% (132/159) Updating files: 84% (134/159) Updating files: 85% (136/159) Updating files: 86% (137/159) Updating files: 87% (139/159) Updating files: 88% (140/159) Updating files: 89% (142/159) Updating files: 90% (144/159) Updating files: 91% (145/159) Updating files: 92% (147/159) Updating files: 93% (148/159) Updating files: 94% (150/159) Updating files: 95% (152/159) Updating files: 96% (153/159) Updating files: 97% (155/159) Updating files: 98% (156/159) Updating files: 99% (158/159) Updating files: 100% (159/159) Updating files: 100% (159/159), done. 2024-03-28T19:25:12.764 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 8e69c39f699 v0.29 2024-03-28T19:25:12.773 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.29 2024-03-28T19:25:12.806 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-03-28T19:25:12.806+0000 7fbc64800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:25:12.825 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:25:12.825 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 30 -eq 48 ']' 2024-03-28T19:25:12.825 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.30 2024-03-28T19:25:12.825 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.30 2024-03-28T19:25:12.826 INFO:tasks.workunit.client.0.smithi027.stdout:v0.30 2024-03-28T19:25:12.826 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.30 2024-03-28T19:25:13.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:13.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.428+0000 7f7513e00640 1 Processor -- start 2024-03-28T19:25:13.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.428+0000 7f7513e00640 1 -- start start 2024-03-28T19:25:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.430+0000 7f7513e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75140ae3d0 0x7f75140c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.430+0000 7f7513e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140ae7b0 0x7f751415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.430+0000 7f7513e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75140c5460 con 0x7f75140aebe0 2024-03-28T19:25:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.430+0000 7f7513e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7514153630 con 0x7f75140ae3d0 2024-03-28T19:25:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.430+0000 7f7513e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75140c4bc0 con 0x7f75140ae7b0 2024-03-28T19:25:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.430+0000 7f7512a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75140ae3d0 0x7f75140c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.430+0000 7f7512a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75140ae3d0 0x7f75140c56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42466/0 (socket says 172.21.15.27:42466) 2024-03-28T19:25:13.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.430+0000 7f7512a00640 1 -- 172.21.15.27:0/2765690156 learned_addr learned my addr 172.21.15.27:0/2765690156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:13.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.431+0000 7f7513400640 1 --2- 172.21.15.27:0/2765690156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140ae7b0 0x7f751415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:13.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.431+0000 7f7512a00640 1 -- 172.21.15.27:0/2765690156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140ae7b0 msgr2=0x7f751415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:13.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.431+0000 7f7512a00640 1 --2- 172.21.15.27:0/2765690156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140ae7b0 0x7f751415dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.431+0000 7f7512a00640 1 -- 172.21.15.27:0/2765690156 >> v1:172.21.15.155:6789/0 conn(0x7f75140aebe0 legacy=0x7f7514153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.431+0000 7f7512a00640 1 -- 172.21.15.27:0/2765690156 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140c4ea0 con 0x7f75140ae3d0 2024-03-28T19:25:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.433+0000 7f7512a00640 1 --2- 172.21.15.27:0/2765690156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae3d0 0x7f75140c56b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f7504009870 tx=0x7f750402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa51e158a1fe28fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.433+0000 7f7511600640 1 -- 172.21.15.27:0/2765690156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504035020 con 0x7f75140ae3d0 2024-03-28T19:25:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.433+0000 7f7511600640 1 -- 172.21.15.27:0/2765690156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7504031640 con 0x7f75140ae3d0 2024-03-28T19:25:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.433+0000 7f7511600640 1 -- 172.21.15.27:0/2765690156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750402d040 con 0x7f75140ae3d0 2024-03-28T19:25:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.434+0000 7f7513e00640 1 -- 172.21.15.27:0/2765690156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae3d0 msgr2=0x7f75140c56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.434+0000 7f7513e00640 1 --2- 172.21.15.27:0/2765690156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae3d0 0x7f75140c56b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f7504009870 tx=0x7f750402b8f0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.434+0000 7f7513e00640 1 -- 172.21.15.27:0/2765690156 shutdown_connections 2024-03-28T19:25:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.434+0000 7f7513e00640 1 --2- 172.21.15.27:0/2765690156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140ae7b0 0x7f751415dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.434+0000 7f7513e00640 1 --2- 172.21.15.27:0/2765690156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae3d0 0x7f75140c56b0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.434+0000 7f7513e00640 1 -- 172.21.15.27:0/2765690156 >> 172.21.15.27:0/2765690156 conn(0x7f7514072d10 msgr2=0x7f751406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.435+0000 7f7513e00640 1 -- 172.21.15.27:0/2765690156 shutdown_connections 2024-03-28T19:25:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.435+0000 7f7513e00640 1 -- 172.21.15.27:0/2765690156 wait complete. 2024-03-28T19:25:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.437+0000 7f7513e00640 1 Processor -- start 2024-03-28T19:25:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.437+0000 7f7513e00640 1 -- start start 2024-03-28T19:25:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7513e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75140ae3d0 0x7f75140c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7513e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae7b0 0x7f751414bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7513e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140aebe0 0x7f75141eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7513e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75141541d0 con 0x7f75140ae7b0 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7513e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7514070890 con 0x7f75140ae3d0 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7513e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7514154350 con 0x7f75140aebe0 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae7b0 0x7f751414bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae7b0 0x7f751414bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42474/0 (socket says 172.21.15.27:42474) 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512000640 1 -- 172.21.15.27:0/2944563297 learned_addr learned my addr 172.21.15.27:0/2944563297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512a00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75140ae3d0 0x7f75140c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7513400640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140aebe0 0x7f75141eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512000640 1 -- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140aebe0 msgr2=0x7f75141eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512000640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140aebe0 0x7f75141eb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512000640 1 -- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75140ae3d0 msgr2=0x7f75140c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512000640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75140ae3d0 0x7f75140c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512000640 1 -- 172.21.15.27:0/2944563297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7514156170 con 0x7f75140ae7b0 2024-03-28T19:25:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.439+0000 7f7512a00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75140ae3d0 0x7f75140c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:25:13.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.440+0000 7f7512000640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae7b0 0x7f751414bc80 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f750000c410 tx=0x7f750000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:13.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.440+0000 7f750aa00640 1 -- 172.21.15.27:0/2944563297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500017070 con 0x7f75140ae7b0 2024-03-28T19:25:13.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.440+0000 7f750aa00640 1 -- 172.21.15.27:0/2944563297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75000035f0 con 0x7f75140ae7b0 2024-03-28T19:25:13.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.440+0000 7f750aa00640 1 -- 172.21.15.27:0/2944563297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500003a20 con 0x7f75140ae7b0 2024-03-28T19:25:13.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.441+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751406c950 con 0x7f75140ae7b0 2024-03-28T19:25:13.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.441+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75140c8000 con 0x7f75140ae7b0 2024-03-28T19:25:13.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.443+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7514067dd0 con 0x7f75140ae7b0 2024-03-28T19:25:13.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.444+0000 7f750aa00640 1 -- 172.21.15.27:0/2944563297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f750001a070 con 0x7f75140ae7b0 2024-03-28T19:25:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.447+0000 7f750aa00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74f406bca0 0x7f74f406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:13.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.447+0000 7f750aa00640 1 -- 172.21.15.27:0/2944563297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f750008afe0 con 0x7f75140ae7b0 2024-03-28T19:25:13.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.447+0000 7f7512a00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74f406bca0 0x7f74f406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:13.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.449+0000 7f7512a00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74f406bca0 0x7f74f406e160 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7504036000 tx=0x7f750403c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:13.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.453+0000 7f750aa00640 1 -- 172.21.15.27:0/2944563297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7500058a70 con 0x7f75140ae7b0 2024-03-28T19:25:13.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.861+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7514151290 con 0x7f75140ae7b0 2024-03-28T19:25:13.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.863+0000 7f750aa00640 1 -- 172.21.15.27:0/2944563297 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f7514151290 con 0x7f75140ae7b0 2024-03-28T19:25:13.865 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:13.866 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:13.866 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:13.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.868+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74f406bca0 msgr2=0x7f74f406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:13.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.868+0000 7f7513e00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74f406bca0 0x7f74f406e160 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7504036000 tx=0x7f750403c040 comp rx=0 tx=0).stop 2024-03-28T19:25:13.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.868+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae7b0 msgr2=0x7f751414bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:13.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.868+0000 7f7513e00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae7b0 0x7f751414bc80 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f750000c410 tx=0x7f750000c8d0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.869+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 shutdown_connections 2024-03-28T19:25:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.869+0000 7f7513e00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f74f406bca0 0x7f74f406e160 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.869+0000 7f7513e00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75140aebe0 0x7f75141eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.869+0000 7f7513e00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75140ae7b0 0x7f751414bc80 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.869+0000 7f7513e00640 1 --2- 172.21.15.27:0/2944563297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75140ae3d0 0x7f75140c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.869+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 >> 172.21.15.27:0/2944563297 conn(0x7f7514072d10 msgr2=0x7f7514151db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.870+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 shutdown_connections 2024-03-28T19:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:13.870+0000 7f7513e00640 1 -- 172.21.15.27:0/2944563297 wait complete. 2024-03-28T19:25:13.895 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-03-28T19:25:13.895 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2024-03-28T19:25:17.763 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 29% (73/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 43% (107/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 57% (141/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 71% (175/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 84% (207/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 98% (242/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2024-03-28T19:25:18.111 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2024-03-28T19:25:18.123 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.20 2024-03-28T19:25:18.181 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:25:18.181 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 21 -eq 48 ']' 2024-03-28T19:25:18.182 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.21 2024-03-28T19:25:18.182 INFO:tasks.workunit.client.1.smithi155.stdout:v0.21 2024-03-28T19:25:18.182 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.21 2024-03-28T19:25:18.182 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.21 2024-03-28T19:25:20.547 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 25% (73/285) Updating files: 26% (75/285) Updating files: 27% (77/285) Updating files: 28% (80/285) Updating files: 29% (83/285) Updating files: 30% (86/285) Updating files: 31% (89/285) Updating files: 32% (92/285) Updating files: 33% (95/285) Updating files: 34% (97/285) Updating files: 35% (100/285) Updating files: 36% (103/285) Updating files: 37% (106/285) Updating files: 38% (109/285) Updating files: 39% (112/285) Updating files: 40% (114/285) Updating files: 41% (117/285) Updating files: 42% (120/285) Updating files: 43% (123/285) Updating files: 44% (126/285) Updating files: 45% (129/285) Updating files: 46% (132/285) Updating files: 47% (134/285) Updating files: 48% (137/285) Updating files: 49% (140/285) Updating files: 50% (143/285) Updating files: 51% (146/285) Updating files: 52% (149/285) Updating files: 53% (152/285) Updating files: 54% (154/285) Updating files: 55% (157/285) Updating files: 56% (160/285) Updating files: 57% (163/285) Updating files: 58% (166/285) Updating files: 58% (168/285) Updating files: 59% (169/285) Updating files: 60% (171/285) Updating files: 61% (174/285) Updating files: 62% (177/285) Updating files: 63% (180/285) Updating files: 64% (183/285) Updating files: 65% (186/285) Updating files: 66% (189/285) Updating files: 67% (191/285) Updating files: 68% (194/285) Updating files: 69% (197/285) Updating files: 70% (200/285) Updating files: 71% (203/285) Updating files: 72% (206/285) Updating files: 73% (209/285) Updating files: 74% (211/285) Updating files: 75% (214/285) Updating files: 76% (217/285) Updating files: 77% (220/285) Updating files: 78% (223/285) Updating files: 79% (226/285) Updating files: 80% (228/285) Updating files: 81% (231/285) Updating files: 82% (234/285) Updating files: 83% (237/285) Updating files: 83% (239/285) Updating files: 84% (240/285) Updating files: 85% (243/285) Updating files: 86% (246/285) Updating files: 87% (248/285) Updating files: 88% (251/285) Updating files: 89% (254/285) Updating files: 90% (257/285) Updating files: 91% (260/285) Updating files: 92% (263/285) Updating files: 93% (266/285) Updating files: 94% (268/285) Updating files: 95% (271/285) Updating files: 96% (274/285) Updating files: 97% (277/285) Updating files: 98% (280/285) Updating files: 99% (283/285) Updating files: 99% (284/285) Updating files: 100% (285/285) Updating files: 100% (285/285), done. 2024-03-28T19:25:20.801 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 64b1b2c70f0 v0.30 2024-03-28T19:25:20.813 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.30 2024-03-28T19:25:20.871 INFO:tasks.workunit.client.0.smithi027.stdout:v0.31 2024-03-28T19:25:20.871 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:25:20.871 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 31 -eq 48 ']' 2024-03-28T19:25:20.871 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.31 2024-03-28T19:25:20.872 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.31 2024-03-28T19:25:20.872 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.31 2024-03-28T19:25:32.015 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 12% (50/398) Updating files: 13% (52/398) Updating files: 14% (56/398) Updating files: 15% (60/398) Updating files: 16% (64/398) Updating files: 17% (68/398) Updating files: 18% (72/398) Updating files: 19% (76/398) Updating files: 20% (80/398) Updating files: 21% (84/398) Updating files: 22% (88/398) Updating files: 23% (92/398) Updating files: 24% (96/398) Updating files: 25% (100/398) Updating files: 26% (104/398) Updating files: 27% (108/398) Updating files: 28% (112/398) Updating files: 29% (116/398) Updating files: 30% (120/398) Updating files: 31% (124/398) Updating files: 32% (128/398) Updating files: 33% (132/398) Updating files: 34% (136/398) Updating files: 35% (140/398) Updating files: 35% (143/398) Updating files: 36% (144/398) Updating files: 37% (148/398) Updating files: 38% (152/398) Updating files: 39% (156/398) Updating files: 40% (160/398) Updating files: 41% (164/398) Updating files: 42% (168/398) Updating files: 43% (172/398) Updating files: 44% (176/398) Updating files: 45% (180/398) Updating files: 45% (183/398) Updating files: 46% (184/398) Updating files: 47% (188/398) Updating files: 48% (192/398) Updating files: 49% (196/398) Updating files: 50% (199/398) Updating files: 51% (203/398) Updating files: 52% (207/398) Updating files: 53% (211/398) Updating files: 54% (215/398) Updating files: 55% (219/398) Updating files: 56% (223/398) Updating files: 57% (227/398) Updating files: 58% (231/398) Updating files: 59% (235/398) Updating files: 60% (239/398) Updating files: 61% (243/398) Updating files: 62% (247/398) Updating files: 63% (251/398) Updating files: 64% (255/398) Updating files: 65% (259/398) Updating files: 65% (262/398) Updating files: 66% (263/398) Updating files: 67% (267/398) Updating files: 68% (271/398) Updating files: 69% (275/398) Updating files: 70% (279/398) Updating files: 71% (283/398) Updating files: 72% (287/398) Updating files: 73% (291/398) Updating files: 74% (295/398) Updating files: 74% (296/398) Updating files: 75% (299/398) Updating files: 76% (303/398) Updating files: 77% (307/398) Updating files: 78% (311/398) Updating files: 79% (315/398) Updating files: 80% (319/398) Updating files: 81% (323/398) Updating files: 82% (327/398) Updating files: 82% (329/398) Updating files: 83% (331/398) Updating files: 84% (335/398) Updating files: 85% (339/398) Updating files: 86% (343/398) Updating files: 87% (347/398) Updating files: 88% (351/398) Updating files: 89% (355/398) Updating files: 90% (359/398) Updating files: 91% (363/398) Updating files: 92% (367/398) Updating files: 92% (368/398) Updating files: 93% (371/398) Updating files: 94% (375/398) Updating files: 95% (379/398) Updating files: 96% (383/398) Updating files: 97% (387/398) Updating files: 98% (391/398) Updating files: 99% (395/398) Updating files: 100% (398/398) Updating files: 100% (398/398), done. 2024-03-28T19:25:32.235 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 9019c6ce640 v0.31 2024-03-28T19:25:32.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.31 2024-03-28T19:25:32.301 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:25:32.302 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 32 -eq 48 ']' 2024-03-28T19:25:32.302 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.32 2024-03-28T19:25:32.302 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.32 2024-03-28T19:25:32.302 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.32 2024-03-28T19:25:32.302 INFO:tasks.workunit.client.0.smithi027.stdout:v0.32 2024-03-28T19:25:35.531 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 10% (58/537) Updating files: 11% (60/537) Updating files: 12% (65/537) Updating files: 13% (70/537) Updating files: 14% (76/537) Updating files: 15% (81/537) Updating files: 16% (86/537) Updating files: 17% (92/537) Updating files: 17% (93/537) Updating files: 18% (97/537) Updating files: 19% (103/537) Updating files: 20% (108/537) Updating files: 21% (113/537) Updating files: 22% (119/537) Updating files: 23% (124/537) Updating files: 24% (129/537) Updating files: 25% (135/537) Updating files: 26% (140/537) Updating files: 27% (145/537) Updating files: 28% (151/537) Updating files: 29% (156/537) Updating files: 29% (158/537) Updating files: 30% (162/537) Updating files: 31% (167/537) Updating files: 32% (172/537) Updating files: 33% (178/537) Updating files: 34% (183/537) Updating files: 35% (188/537) Updating files: 35% (191/537) Updating files: 36% (194/537) Updating files: 37% (199/537) Updating files: 38% (205/537) Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 41% (221/537) Updating files: 41% (225/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 46% (248/537) Updating files: 47% (253/537) Updating files: 48% (258/537) Updating files: 48% (259/537) Updating files: 49% (264/537) Updating files: 50% (269/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 54% (291/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 60% (324/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 73% (394/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 79% (429/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 85% (460/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 90% (488/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 96% (517/537) Updating files: 97% (521/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2024-03-28T19:25:35.801 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 090436f56ac v0.21 2024-03-28T19:25:35.826 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.21 2024-03-28T19:25:35.898 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:25:35.899 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 22 -eq 48 ']' 2024-03-28T19:25:35.899 INFO:tasks.workunit.client.1.smithi155.stdout:v0.22 2024-03-28T19:25:35.899 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.22 2024-03-28T19:25:35.899 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.22 2024-03-28T19:25:35.900 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.22 2024-03-28T19:25:38.476 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 28% (56/195) Updating files: 29% (57/195) Updating files: 30% (59/195) Updating files: 31% (61/195) Updating files: 32% (63/195) Updating files: 33% (65/195) Updating files: 34% (67/195) Updating files: 35% (69/195) Updating files: 36% (71/195) Updating files: 37% (73/195) Updating files: 38% (75/195) Updating files: 39% (77/195) Updating files: 40% (78/195) Updating files: 41% (80/195) Updating files: 42% (82/195) Updating files: 43% (84/195) Updating files: 44% (86/195) Updating files: 45% (88/195) Updating files: 46% (90/195) Updating files: 47% (92/195) Updating files: 47% (93/195) Updating files: 48% (94/195) Updating files: 49% (96/195) Updating files: 50% (98/195) Updating files: 51% (100/195) Updating files: 52% (102/195) Updating files: 53% (104/195) Updating files: 54% (106/195) Updating files: 55% (108/195) Updating files: 56% (110/195) Updating files: 57% (112/195) Updating files: 58% (114/195) Updating files: 59% (116/195) Updating files: 60% (117/195) Updating files: 61% (119/195) Updating files: 62% (121/195) Updating files: 63% (123/195) Updating files: 64% (125/195) Updating files: 65% (127/195) Updating files: 66% (129/195) Updating files: 67% (131/195) Updating files: 67% (132/195) Updating files: 68% (133/195) Updating files: 69% (135/195) Updating files: 70% (137/195) Updating files: 71% (139/195) Updating files: 72% (141/195) Updating files: 73% (143/195) Updating files: 74% (145/195) Updating files: 75% (147/195) Updating files: 76% (149/195) Updating files: 77% (151/195) Updating files: 78% (153/195) Updating files: 79% (155/195) Updating files: 80% (156/195) Updating files: 81% (158/195) Updating files: 82% (160/195) Updating files: 83% (162/195) Updating files: 84% (164/195) Updating files: 85% (166/195) Updating files: 86% (168/195) Updating files: 87% (170/195) Updating files: 88% (172/195) Updating files: 89% (174/195) Updating files: 90% (176/195) Updating files: 90% (177/195) Updating files: 91% (178/195) Updating files: 92% (180/195) Updating files: 93% (182/195) Updating files: 94% (184/195) Updating files: 95% (186/195) Updating files: 96% (188/195) Updating files: 97% (190/195) Updating files: 98% (192/195) Updating files: 99% (194/195) Updating files: 100% (195/195) Updating files: 100% (195/195), done. 2024-03-28T19:25:38.654 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at c08d08baa6a v0.32 2024-03-28T19:25:38.661 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.32 2024-03-28T19:25:38.724 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:25:38.724 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 33 -eq 48 ']' 2024-03-28T19:25:38.724 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.33 2024-03-28T19:25:38.724 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.33 2024-03-28T19:25:38.725 INFO:tasks.workunit.client.0.smithi027.stdout:v0.33 2024-03-28T19:25:38.725 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.33 2024-03-28T19:25:44.285 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 22% (45/204) Updating files: 23% (47/204) Updating files: 24% (49/204) Updating files: 25% (51/204) Updating files: 26% (54/204) Updating files: 27% (56/204) Updating files: 28% (58/204) Updating files: 29% (60/204) Updating files: 30% (62/204) Updating files: 31% (64/204) Updating files: 32% (66/204) Updating files: 33% (68/204) Updating files: 34% (70/204) Updating files: 35% (72/204) Updating files: 36% (74/204) Updating files: 37% (76/204) Updating files: 38% (78/204) Updating files: 39% (80/204) Updating files: 39% (81/204) Updating files: 40% (82/204) Updating files: 41% (84/204) Updating files: 42% (86/204) Updating files: 43% (88/204) Updating files: 44% (90/204) Updating files: 45% (92/204) Updating files: 46% (94/204) Updating files: 47% (96/204) Updating files: 48% (98/204) Updating files: 49% (100/204) Updating files: 50% (102/204) Updating files: 51% (105/204) Updating files: 52% (107/204) Updating files: 53% (109/204) Updating files: 54% (111/204) Updating files: 55% (113/204) Updating files: 56% (115/204) Updating files: 57% (117/204) Updating files: 58% (119/204) Updating files: 59% (121/204) Updating files: 60% (123/204) Updating files: 61% (125/204) Updating files: 62% (127/204) Updating files: 63% (129/204) Updating files: 64% (131/204) Updating files: 65% (133/204) Updating files: 66% (135/204) Updating files: 67% (137/204) Updating files: 68% (139/204) Updating files: 69% (141/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 85% (175/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2024-03-28T19:25:44.423 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 26% (43/160) Updating files: 27% (44/160) Updating files: 28% (45/160) Updating files: 29% (47/160) Updating files: 30% (48/160) Updating files: 31% (50/160) Updating files: 32% (52/160) Updating files: 33% (53/160) Updating files: 34% (55/160) Updating files: 35% (56/160) Updating files: 36% (58/160) Updating files: 37% (60/160) Updating files: 38% (61/160) Updating files: 39% (63/160) Updating files: 40% (64/160) Updating files: 41% (66/160) Updating files: 42% (68/160) Updating files: 43% (69/160) Updating files: 44% (71/160) Updating files: 45% (72/160) Updating files: 46% (74/160) Updating files: 47% (76/160) Updating files: 48% (77/160) Updating files: 49% (79/160) Updating files: 50% (80/160) Updating files: 51% (82/160) Updating files: 52% (84/160) Updating files: 53% (85/160) Updating files: 54% (87/160) Updating files: 55% (88/160) Updating files: 56% (90/160) Updating files: 57% (92/160) Updating files: 58% (93/160) Updating files: 59% (95/160) Updating files: 60% (96/160) Updating files: 61% (98/160) Updating files: 62% (100/160) Updating files: 63% (101/160) Updating files: 64% (103/160) Updating files: 65% (104/160) Updating files: 66% (106/160) Updating files: 67% (108/160) Updating files: 68% (109/160) Updating files: 69% (111/160) Updating files: 70% (112/160) Updating files: 71% (114/160) Updating files: 72% (116/160) Updating files: 73% (117/160) Updating files: 74% (119/160) Updating files: 75% (120/160) Updating files: 76% (122/160) Updating files: 77% (124/160) Updating files: 78% (125/160) Updating files: 79% (127/160) Updating files: 80% (128/160) Updating files: 81% (130/160) Updating files: 82% (132/160) Updating files: 83% (133/160) Updating files: 84% (135/160) Updating files: 85% (136/160) Updating files: 86% (138/160) Updating files: 87% (140/160) Updating files: 88% (141/160) Updating files: 89% (143/160) Updating files: 90% (144/160) Updating files: 91% (146/160) Updating files: 92% (148/160) Updating files: 93% (149/160) Updating files: 94% (151/160) Updating files: 95% (152/160) Updating files: 96% (154/160) Updating files: 97% (156/160) Updating files: 98% (157/160) Updating files: 99% (159/160) Updating files: 100% (160/160) Updating files: 100% (160/160), done. 2024-03-28T19:25:44.679 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 340b4348bf6 v0.33 2024-03-28T19:25:44.686 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 8a7c95f60ad v0.22 2024-03-28T19:25:44.688 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.33 2024-03-28T19:25:44.704 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.22 2024-03-28T19:25:44.733 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:25:44.733 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 34 -eq 48 ']' 2024-03-28T19:25:44.733 INFO:tasks.workunit.client.0.smithi027.stdout:v0.34 2024-03-28T19:25:44.733 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.34 2024-03-28T19:25:44.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.34 2024-03-28T19:25:44.734 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.34 2024-03-28T19:25:44.769 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:25:44.770 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 23 -eq 48 ']' 2024-03-28T19:25:44.770 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.23 2024-03-28T19:25:44.770 INFO:tasks.workunit.client.1.smithi155.stdout:v0.23 2024-03-28T19:25:44.770 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.23 2024-03-28T19:25:44.770 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.23 2024-03-28T19:25:47.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:48.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:48.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.167+0000 7f9f33000640 1 Processor -- start 2024-03-28T19:25:48.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.167+0000 7f9f33000640 1 -- start start 2024-03-28T19:25:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.170+0000 7f9f33000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f2c0c3880 0x7f9f2c165410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.170+0000 7f9f33000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2c157530 con 0x7f9f2c157150 2024-03-28T19:25:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.170+0000 7f9f33000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2c06dbd0 con 0x7f9f2c0c1390 2024-03-28T19:25:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.170+0000 7f9f33000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2c06e1e0 con 0x7f9f2c0c3880 2024-03-28T19:25:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.170+0000 7f9f31200640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f9f2c0c1390 0x7f9f2c0c3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:59400/0 (socket says 172.21.15.27:59400) 2024-03-28T19:25:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.171+0000 7f9f31200640 1 -- 172.21.15.27:0/1373537260 learned_addr learned my addr 172.21.15.27:0/1373537260 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.171+0000 7f9f31c00640 1 --1- 172.21.15.27:0/1373537260 >> v1:172.21.15.27:6789/0 conn(0x7f9f2c157150 0x7f9f2c0c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35422/0 (socket says 172.21.15.27:35422) 2024-03-28T19:25:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.171+0000 7f9f32600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f2c0c3880 0x7f9f2c165410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.172+0000 7f9f2be00640 1 -- 172.21.15.27:0/1373537260 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018740731 0 0) 0x7f9f2c157530 con 0x7f9f2c157150 2024-03-28T19:25:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.172+0000 7f9f2be00640 1 -- 172.21.15.27:0/1373537260 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f14003520 con 0x7f9f2c157150 2024-03-28T19:25:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.172+0000 7f9f32600640 1 -- 172.21.15.27:0/1373537260 >> v1:172.21.15.155:6789/0 conn(0x7f9f2c0c1390 legacy=0x7f9f2c0c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.172+0000 7f9f32600640 1 -- 172.21.15.27:0/1373537260 >> v1:172.21.15.27:6789/0 conn(0x7f9f2c157150 legacy=0x7f9f2c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.172+0000 7f9f32600640 1 -- 172.21.15.27:0/1373537260 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c15de60 con 0x7f9f2c0c3880 2024-03-28T19:25:48.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.174+0000 7f9f32600640 1 --2- 172.21.15.27:0/1373537260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 0x7f9f2c165410 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009a00 tx=0x7f9f2402bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c4fdb904c85f00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.175+0000 7f9f2be00640 1 -- 172.21.15.27:0/1373537260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2402d6e0 con 0x7f9f2c0c3880 2024-03-28T19:25:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.175+0000 7f9f2be00640 1 -- 172.21.15.27:0/1373537260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2402d880 con 0x7f9f2c0c3880 2024-03-28T19:25:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.175+0000 7f9f2be00640 1 -- 172.21.15.27:0/1373537260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2402dc50 con 0x7f9f2c0c3880 2024-03-28T19:25:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.176+0000 7f9f33000640 1 -- 172.21.15.27:0/1373537260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 msgr2=0x7f9f2c165410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.176+0000 7f9f33000640 1 --2- 172.21.15.27:0/1373537260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 0x7f9f2c165410 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009a00 tx=0x7f9f2402bb30 comp rx=0 tx=0).stop 2024-03-28T19:25:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.176+0000 7f9f33000640 1 -- 172.21.15.27:0/1373537260 shutdown_connections 2024-03-28T19:25:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.176+0000 7f9f33000640 1 --2- 172.21.15.27:0/1373537260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 0x7f9f2c165410 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.176+0000 7f9f33000640 1 -- 172.21.15.27:0/1373537260 >> 172.21.15.27:0/1373537260 conn(0x7f9f2c06b0c0 msgr2=0x7f9f2c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.176+0000 7f9f33000640 1 -- 172.21.15.27:0/1373537260 shutdown_connections 2024-03-28T19:25:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.177+0000 7f9f33000640 1 -- 172.21.15.27:0/1373537260 wait complete. 2024-03-28T19:25:48.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.181+0000 7f9f33000640 1 Processor -- start 2024-03-28T19:25:48.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.181+0000 7f9f33000640 1 -- start start 2024-03-28T19:25:48.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.183+0000 7f9f33000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f2c0c1390 0x7f9f2c1f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.183+0000 7f9f33000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 0x7f9f2c0a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.184+0000 7f9f33000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f2c157150 0x7f9f2c150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.184+0000 7f9f33000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c0c0fc0 con 0x7f9f2c0c3880 2024-03-28T19:25:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.184+0000 7f9f33000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c06e710 con 0x7f9f2c157150 2024-03-28T19:25:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.184+0000 7f9f31c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f2c0c1390 0x7f9f2c1f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.184+0000 7f9f33000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c0c2bc0 con 0x7f9f2c0c1390 2024-03-28T19:25:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.184+0000 7f9f31c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f2c0c1390 0x7f9f2c1f00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:44566/0 (socket says 172.21.15.27:44566) 2024-03-28T19:25:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.184+0000 7f9f31c00640 1 -- 172.21.15.27:0/1358156270 learned_addr learned my addr 172.21.15.27:0/1358156270 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:48.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.184+0000 7f9f32600640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f2c157150 0x7f9f2c150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.185+0000 7f9f31200640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 0x7f9f2c0a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.185+0000 7f9f31c00640 1 -- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f2c157150 msgr2=0x7f9f2c150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.185+0000 7f9f31c00640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f2c157150 0x7f9f2c150ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.185+0000 7f9f31c00640 1 -- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 msgr2=0x7f9f2c0a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.185+0000 7f9f31c00640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 0x7f9f2c0a5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.185+0000 7f9f31c00640 1 -- 172.21.15.27:0/1358156270 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c0c0660 con 0x7f9f2c0c1390 2024-03-28T19:25:48.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.186+0000 7f9f31200640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 0x7f9f2c0a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:25:48.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.187+0000 7f9f32600640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f2c157150 0x7f9f2c150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:25:48.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.187+0000 7f9f31c00640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f2c0c1390 0x7f9f2c1f00f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000c7e0 tx=0x7f9f2000cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:48.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.188+0000 7f9f29600640 1 -- 172.21.15.27:0/1358156270 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f20018070 con 0x7f9f2c0c1390 2024-03-28T19:25:48.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.188+0000 7f9f29600640 1 -- 172.21.15.27:0/1358156270 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2000a640 con 0x7f9f2c0c1390 2024-03-28T19:25:48.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.188+0000 7f9f29600640 1 -- 172.21.15.27:0/1358156270 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2000aa30 con 0x7f9f2c0c1390 2024-03-28T19:25:48.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.188+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2c0a32e0 con 0x7f9f2c0c1390 2024-03-28T19:25:48.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.188+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2c0a38a0 con 0x7f9f2c0c1390 2024-03-28T19:25:48.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.190+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2c067dd0 con 0x7f9f2c0c1390 2024-03-28T19:25:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.193+0000 7f9f29600640 1 -- 172.21.15.27:0/1358156270 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9f2000abd0 con 0x7f9f2c0c1390 2024-03-28T19:25:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.194+0000 7f9f29600640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f0c06bbd0 0x7f9f0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.195+0000 7f9f29600640 1 -- 172.21.15.27:0/1358156270 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f9f2008b230 con 0x7f9f2c0c1390 2024-03-28T19:25:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.195+0000 7f9f31200640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f0c06bbd0 0x7f9f0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.197+0000 7f9f31200640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f0c06bbd0 0x7f9f0c06e090 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c151e60 tx=0x7f9f18009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:48.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.211+0000 7f9f29600640 1 -- 172.21.15.27:0/1358156270 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9f20058570 con 0x7f9f2c0c1390 2024-03-28T19:25:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.310+0000 7ffa26e00640 1 Processor -- start 2024-03-28T19:25:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.310+0000 7ffa26e00640 1 -- start start 2024-03-28T19:25:48.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.312+0000 7ffa26e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa20152c10 0x7ffa200cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.313+0000 7ffa26e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa20072bc0 con 0x7ffa200c9240 2024-03-28T19:25:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.313+0000 7ffa26e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa20159960 con 0x7ffa200c8d30 2024-03-28T19:25:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.313+0000 7ffa26e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200c5fa0 con 0x7ffa20152c10 2024-03-28T19:25:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.313+0000 7ffa1f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffa200c9240 0x7ffa20155830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35426/0 (socket says 172.21.15.27:35426) 2024-03-28T19:25:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.313+0000 7ffa1f400640 1 -- 172.21.15.27:0/3025040964 learned_addr learned my addr 172.21.15.27:0/3025040964 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.314+0000 7ffa1fe00640 1 --2- 172.21.15.27:0/3025040964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa20152c10 0x7ffa200cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.315+0000 7ffa1ea00640 1 -- 172.21.15.27:0/3025040964 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393261506 0 0) 0x7ffa20072bc0 con 0x7ffa200c9240 2024-03-28T19:25:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.315+0000 7ffa1ea00640 1 -- 172.21.15.27:0/3025040964 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa04003520 con 0x7ffa200c9240 2024-03-28T19:25:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.315+0000 7ffa1ea00640 1 -- 172.21.15.27:0/3025040964 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708609771 0 0) 0x7ffa20159960 con 0x7ffa200c8d30 2024-03-28T19:25:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.315+0000 7ffa1ea00640 1 -- 172.21.15.27:0/3025040964 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa20072bc0 con 0x7ffa200c8d30 2024-03-28T19:25:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.315+0000 7ffa1fe00640 1 -- 172.21.15.27:0/3025040964 >> v1:172.21.15.155:6789/0 conn(0x7ffa200c8d30 legacy=0x7ffa20155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.315+0000 7ffa1fe00640 1 -- 172.21.15.27:0/3025040964 >> v1:172.21.15.27:6789/0 conn(0x7ffa200c9240 legacy=0x7ffa20155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.315+0000 7ffa1fe00640 1 -- 172.21.15.27:0/3025040964 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200c5950 con 0x7ffa20152c10 2024-03-28T19:25:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.317+0000 7ffa1fe00640 1 --2- 172.21.15.27:0/3025040964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa20152c10 0x7ffa200cb380 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009dc0 tx=0x7ffa1802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81fbd3416fe7a4a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.318+0000 7ffa1ea00640 1 -- 172.21.15.27:0/3025040964 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18039070 con 0x7ffa20152c10 2024-03-28T19:25:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.318+0000 7ffa1ea00640 1 -- 172.21.15.27:0/3025040964 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1802cb20 con 0x7ffa20152c10 2024-03-28T19:25:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.318+0000 7ffa1ea00640 1 -- 172.21.15.27:0/3025040964 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18034440 con 0x7ffa20152c10 2024-03-28T19:25:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.319+0000 7ffa26e00640 1 -- 172.21.15.27:0/3025040964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa20152c10 msgr2=0x7ffa200cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.319+0000 7ffa26e00640 1 --2- 172.21.15.27:0/3025040964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa20152c10 0x7ffa200cb380 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009dc0 tx=0x7ffa1802bd70 comp rx=0 tx=0).stop 2024-03-28T19:25:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.320+0000 7ffa26e00640 1 -- 172.21.15.27:0/3025040964 shutdown_connections 2024-03-28T19:25:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.320+0000 7ffa26e00640 1 --2- 172.21.15.27:0/3025040964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa20152c10 0x7ffa200cb380 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.320+0000 7ffa26e00640 1 -- 172.21.15.27:0/3025040964 >> 172.21.15.27:0/3025040964 conn(0x7ffa2006b4e0 msgr2=0x7ffa2006b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.320+0000 7ffa26e00640 1 -- 172.21.15.27:0/3025040964 shutdown_connections 2024-03-28T19:25:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.321+0000 7ffa26e00640 1 -- 172.21.15.27:0/3025040964 wait complete. 2024-03-28T19:25:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.323+0000 7ffa26e00640 1 Processor -- start 2024-03-28T19:25:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.323+0000 7ffa26e00640 1 -- start start 2024-03-28T19:25:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.324+0000 7ffa26e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa200c8d30 0x7ffa201584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.324+0000 7ffa26e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa200c9240 0x7ffa2014df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.324+0000 7ffa26e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa20152c10 0x7ffa201eb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.324+0000 7ffa26e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20159610 con 0x7ffa20152c10 2024-03-28T19:25:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.324+0000 7ffa26e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20152ff0 con 0x7ffa200c8d30 2024-03-28T19:25:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.324+0000 7ffa26e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200cb8c0 con 0x7ffa200c9240 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa24e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa20152c10 0x7ffa201eb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa24e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa20152c10 0x7ffa201eb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32940/0 (socket says 172.21.15.27:32940) 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa24e00640 1 -- 172.21.15.27:0/3651453829 learned_addr learned my addr 172.21.15.27:0/3651453829 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa1fe00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa200c8d30 0x7ffa201584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa1f400640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa200c9240 0x7ffa2014df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa24e00640 1 -- 172.21.15.27:0/3651453829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa200c9240 msgr2=0x7ffa2014df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa24e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa200c9240 0x7ffa2014df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa24e00640 1 -- 172.21.15.27:0/3651453829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa200c8d30 msgr2=0x7ffa201584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa24e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa200c8d30 0x7ffa201584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.325+0000 7ffa24e00640 1 -- 172.21.15.27:0/3651453829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa201eafb0 con 0x7ffa20152c10 2024-03-28T19:25:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.326+0000 7ffa24e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa20152c10 0x7ffa201eb470 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000c460 tx=0x7ffa1000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.326+0000 7ffa17e00640 1 -- 172.21.15.27:0/3651453829 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10017070 con 0x7ffa20152c10 2024-03-28T19:25:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.326+0000 7ffa17e00640 1 -- 172.21.15.27:0/3651453829 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa100035f0 con 0x7ffa20152c10 2024-03-28T19:25:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.326+0000 7ffa17e00640 1 -- 172.21.15.27:0/3651453829 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa100039e0 con 0x7ffa20152c10 2024-03-28T19:25:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.326+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2014d050 con 0x7ffa20152c10 2024-03-28T19:25:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.327+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2014d610 con 0x7ffa20152c10 2024-03-28T19:25:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.328+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2004f400 con 0x7ffa20152c10 2024-03-28T19:25:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.331+0000 7ffa17e00640 1 -- 172.21.15.27:0/3651453829 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffa1001a070 con 0x7ffa20152c10 2024-03-28T19:25:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.332+0000 7ffa17e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff9fc06bc20 0x7ff9fc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.332+0000 7ffa17e00640 1 -- 172.21.15.27:0/3651453829 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7ffa1008b2b0 con 0x7ffa20152c10 2024-03-28T19:25:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.333+0000 7ffa1fe00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff9fc06bc20 0x7ff9fc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.335+0000 7ffa1fe00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff9fc06bc20 0x7ff9fc06e0e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ffa18036000 tx=0x7ffa18030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:48.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.339+0000 7ffa17e00640 1 -- 172.21.15.27:0/3651453829 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffa201eafb0 con 0x7ffa20152c10 2024-03-28T19:25:48.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.700+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f2c069640 con 0x7f9f2c0c1390 2024-03-28T19:25:48.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.702+0000 7f9f29600640 1 -- 172.21.15.27:0/1358156270 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f9f2005c610 con 0x7f9f2c0c1390 2024-03-28T19:25:48.703 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:48.704 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.705+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f0c06bbd0 msgr2=0x7f9f0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.705+0000 7f9f33000640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f0c06bbd0 0x7f9f0c06e090 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c151e60 tx=0x7f9f18009040 comp rx=0 tx=0).stop 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.705+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f2c0c1390 msgr2=0x7f9f2c1f00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.705+0000 7f9f33000640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f2c0c1390 0x7f9f2c1f00f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000c7e0 tx=0x7f9f2000cca0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.706+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 shutdown_connections 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.706+0000 7f9f33000640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9f0c06bbd0 0x7f9f0c06e090 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.706+0000 7f9f33000640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f2c157150 0x7f9f2c150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.706+0000 7f9f33000640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2c0c3880 0x7f9f2c0a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.706+0000 7f9f33000640 1 --2- 172.21.15.27:0/1358156270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9f2c0c1390 0x7f9f2c1f00f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.706+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 >> 172.21.15.27:0/1358156270 conn(0x7f9f2c06b0c0 msgr2=0x7f9f2c15ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.706+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 shutdown_connections 2024-03-28T19:25:48.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.706+0000 7f9f33000640 1 -- 172.21.15.27:0/1358156270 wait complete. 2024-03-28T19:25:48.732 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-03-28T19:25:48.733 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-28T19:25:48.733 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:25:48.792 INFO:tasks.ceph.mds.a:Stopped 2024-03-28T19:25:48.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:48.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.829+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa20072860 con 0x7ffa20152c10 2024-03-28T19:25:48.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.830+0000 7ffa17e00640 1 -- 172.21.15.27:0/3651453829 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7ffa10058da0 con 0x7ffa20152c10 2024-03-28T19:25:48.830 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:48.831 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:48.831 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.833+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff9fc06bc20 msgr2=0x7ff9fc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.833+0000 7ffa26e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff9fc06bc20 0x7ff9fc06e0e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ffa18036000 tx=0x7ffa18030000 comp rx=0 tx=0).stop 2024-03-28T19:25:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.833+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa20152c10 msgr2=0x7ffa201eb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:48.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.833+0000 7ffa26e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa20152c10 0x7ffa201eb470 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000c460 tx=0x7ffa1000c920 comp rx=0 tx=0).stop 2024-03-28T19:25:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.834+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 shutdown_connections 2024-03-28T19:25:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.834+0000 7ffa26e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff9fc06bc20 0x7ff9fc06e0e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.834+0000 7ffa26e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa20152c10 0x7ffa201eb470 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.834+0000 7ffa26e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa200c9240 0x7ffa2014df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.834+0000 7ffa26e00640 1 --2- 172.21.15.27:0/3651453829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa200c8d30 0x7ffa201584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.834+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 >> 172.21.15.27:0/3651453829 conn(0x7ffa2006b4e0 msgr2=0x7ffa2006b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.834+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 shutdown_connections 2024-03-28T19:25:48.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.834+0000 7ffa26e00640 1 -- 172.21.15.27:0/3651453829 wait complete. 2024-03-28T19:25:48.859 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-03-28T19:25:48.860 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:25:48.860 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:25:48.901 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:25:48.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:48.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:48.999+0000 7f3481c00640 1 Processor -- start 2024-03-28T19:25:49.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.000+0000 7f3481c00640 1 -- start start 2024-03-28T19:25:49.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.002+0000 7f3481c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0c8e10 0x7f347c158be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.002+0000 7f3481c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f347c1562f0 0x7f347c15ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.002+0000 7f3481c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c1530f0 con 0x7f347c152d10 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.002+0000 7f3481c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c06d3d0 con 0x7f347c1562f0 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.002+0000 7f3481c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0c92f0 con 0x7f347c0c8e10 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.002+0000 7f347aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f347c152d10 0x7f347c0c5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35440/0 (socket says 172.21.15.27:35440) 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.002+0000 7f347aa00640 1 -- 172.21.15.27:0/2348875497 learned_addr learned my addr 172.21.15.27:0/2348875497 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.002+0000 7f3473e00640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0c8e10 0x7f347c158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347b400640 1 --2- 172.21.15.27:0/2348875497 >> v2:172.21.15.155:3300/0 conn(0x7f347c1562f0 0x7f347c15ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788539072 0 0) 0x7f346c001110 con 0x7f347c152d10 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3460003520 con 0x7f347c152d10 2024-03-28T19:25:49.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1284349523 0 0) 0x7f3460003520 con 0x7f347c152d10 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0c8e10 msgr2=0x7f347c158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0c8e10 0x7f347c158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 >> v2:172.21.15.155:3300/0 conn(0x7f347c1562f0 msgr2=0x7f347c15ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 --2- 172.21.15.27:0/2348875497 >> v2:172.21.15.155:3300/0 conn(0x7f347c1562f0 0x7f347c15ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c155810 con 0x7f347c152d10 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f346c001840 con 0x7f347c152d10 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 >> v1:172.21.15.27:6789/0 conn(0x7f347c152d10 legacy=0x7f347c0c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0c8e10 0x7f34600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3460003f80 0x7f3460004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f34600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c06d3d0 con 0x7f3460004d40 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0c92f0 con 0x7f3460003f80 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346000bba0 con 0x7f347c0c8e10 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.003+0000 7f347b400640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f34600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.004+0000 7f347aa00640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0c8e10 0x7f34600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.004+0000 7f3473e00640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3460003f80 0x7f3460004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.004+0000 7f347b400640 1 -- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0c8e10 msgr2=0x7f34600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.004+0000 7f347b400640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0c8e10 0x7f34600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.004+0000 7f347b400640 1 -- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3460003f80 msgr2=0x7f3460004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:49.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.004+0000 7f347b400640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3460003f80 0x7f3460004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.004+0000 7f347b400640 1 -- 172.21.15.27:0/2348875497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460003520 con 0x7f3460004d40 2024-03-28T19:25:49.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.006+0000 7f347b400640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f34600051c0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fc0 tx=0x7f3460005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4727ca89f1841d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:49.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.006+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474039070 con 0x7f3460004d40 2024-03-28T19:25:49.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.006+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347402cc60 con 0x7f3460004d40 2024-03-28T19:25:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.006+0000 7f347a000640 1 -- 172.21.15.27:0/2348875497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474009c10 con 0x7f3460004d40 2024-03-28T19:25:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.007+0000 7f3481c00640 1 -- 172.21.15.27:0/2348875497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 msgr2=0x7f34600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.007+0000 7f3481c00640 1 --2- 172.21.15.27:0/2348875497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f34600051c0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fc0 tx=0x7f3460005830 comp rx=0 tx=0).stop 2024-03-28T19:25:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.007+0000 7f347b400640 1 -- 172.21.15.27:0/2348875497 reap_dead start 2024-03-28T19:25:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.007+0000 7f3481c00640 1 -- 172.21.15.27:0/2348875497 shutdown_connections 2024-03-28T19:25:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.007+0000 7f3481c00640 1 -- 172.21.15.27:0/2348875497 >> 172.21.15.27:0/2348875497 conn(0x7f347c066fb0 msgr2=0x7f347c0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.007+0000 7f3481c00640 1 -- 172.21.15.27:0/2348875497 shutdown_connections 2024-03-28T19:25:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.007+0000 7f3481c00640 1 -- 172.21.15.27:0/2348875497 wait complete. 2024-03-28T19:25:49.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.010+0000 7f3481c00640 1 Processor -- start 2024-03-28T19:25:49.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.010+0000 7f3481c00640 1 -- start start 2024-03-28T19:25:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.011+0000 7f3481c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f347c0c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.011+0000 7f3481c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0ca9b0 0x7f347c14eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f3481c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f347c0bea20 0x7f347c14f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f3481c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0c5fa0 con 0x7f3460004d40 2024-03-28T19:25:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f3481c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c06d850 con 0x7f347c0bea20 2024-03-28T19:25:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f3481c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c0c6120 con 0x7f347c0ca9b0 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f347c0c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f347c0c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32962/0 (socket says 172.21.15.27:32962) 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347aa00640 1 -- 172.21.15.27:0/2931793101 learned_addr learned my addr 172.21.15.27:0/2931793101 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f3473e00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0ca9b0 0x7f347c14eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347aa00640 1 -- 172.21.15.27:0/2931793101 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0ca9b0 msgr2=0x7f347c14eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347b400640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f347c0bea20 0x7f347c14f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347aa00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0ca9b0 0x7f347c14eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347aa00640 1 -- 172.21.15.27:0/2931793101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f347c0bea20 msgr2=0x7f347c14f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347aa00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f347c0bea20 0x7f347c14f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.012+0000 7f347aa00640 1 -- 172.21.15.27:0/2931793101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c1ee540 con 0x7f3460004d40 2024-03-28T19:25:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.013+0000 7f347aa00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f347c0c43e0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f346c007340 tx=0x7f346c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.013+0000 7f3472a00640 1 -- 172.21.15.27:0/2931793101 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c01a070 con 0x7f3460004d40 2024-03-28T19:25:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.013+0000 7f3472a00640 1 -- 172.21.15.27:0/2931793101 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c00f8c0 con 0x7f3460004d40 2024-03-28T19:25:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.013+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c14e350 con 0x7f3460004d40 2024-03-28T19:25:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.013+0000 7f3472a00640 1 -- 172.21.15.27:0/2931793101 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c00fcb0 con 0x7f3460004d40 2024-03-28T19:25:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.013+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c0c4cb0 con 0x7f3460004d40 2024-03-28T19:25:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.014+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c04f400 con 0x7f3460004d40 2024-03-28T19:25:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.016+0000 7f3472a00640 1 -- 172.21.15.27:0/2931793101 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f346c01d070 con 0x7f3460004d40 2024-03-28T19:25:49.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.017+0000 7f3472a00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f345806bbd0 0x7f345806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.017+0000 7f3472a00640 1 -- 172.21.15.27:0/2931793101 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f346c08b4e0 con 0x7f3460004d40 2024-03-28T19:25:49.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.017+0000 7f3473e00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f345806bbd0 0x7f345806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.018+0000 7f3473e00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f345806bbd0 0x7f345806e090 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f347c06cc60 tx=0x7f3464008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:49.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.022+0000 7f3472a00640 1 -- 172.21.15.27:0/2931793101 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f347c1530f0 con 0x7f3460004d40 2024-03-28T19:25:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.130+0000 7faf04e00640 1 Processor -- start 2024-03-28T19:25:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.130+0000 7faf04e00640 1 -- start start 2024-03-28T19:25:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faf04e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf00150c70 0x7faf000a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faf04e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7faf001534c0 0x7faf00153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faf04e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000c3af0 con 0x7faf000c3690 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faf04e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf000ab4e0 con 0x7faf001534c0 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faf04e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf000ac1f0 con 0x7faf00150c70 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faefd400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faf000c3690 0x7faf000c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35456/0 (socket says 172.21.15.27:35456) 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faefd400640 1 -- 172.21.15.27:0/2166129025 learned_addr learned my addr 172.21.15.27:0/2166129025 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faefe800640 1 --2- 172.21.15.27:0/2166129025 >> v2:172.21.15.155:3300/0 conn(0x7faf001534c0 0x7faf00153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.132+0000 7faefde00640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf00150c70 0x7faf000a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783204842 0 0) 0x7faf000c3af0 con 0x7faf000c3690 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee4003520 con 0x7faf000c3690 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91632236 0 0) 0x7faee4003520 con 0x7faf000c3690 2024-03-28T19:25:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf00150c70 msgr2=0x7faf000a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf00150c70 0x7faf000a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 >> v2:172.21.15.155:3300/0 conn(0x7faf001534c0 msgr2=0x7faf00153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 --2- 172.21.15.27:0/2166129025 >> v2:172.21.15.155:3300/0 conn(0x7faf001534c0 0x7faf00153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf000c3d90 con 0x7faf000c3690 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7faeec001670 con 0x7faf000c3690 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 >> v1:172.21.15.27:6789/0 conn(0x7faf000c3690 legacy=0x7faf000c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faee40039b0 0x7faee4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faee4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faee40051c0 0x7faee400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf000ac1f0 con 0x7faee4004800 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee400bf00 con 0x7faee40039b0 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee400c100 con 0x7faee40051c0 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.133+0000 7faefde00640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf00150c70 0x7faf000a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.134+0000 7faefd400640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faee4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.134+0000 7faefd400640 1 -- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faee40051c0 msgr2=0x7faee400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.134+0000 7faefd400640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faee40051c0 0x7faee400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.134+0000 7faefd400640 1 -- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faee40039b0 msgr2=0x7faee4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:49.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.134+0000 7faefd400640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faee40039b0 0x7faee4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.134+0000 7faefd400640 1 -- 172.21.15.27:0/2166129025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee4003520 con 0x7faee4004800 2024-03-28T19:25:49.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.135+0000 7faefd400640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faee4004c80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7faeec00c160 tx=0x7faee4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff5b675b967f234 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.135+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec007c80 con 0x7faee4004800 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.135+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeec030070 con 0x7faee4004800 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.135+0000 7faefca00640 1 -- 172.21.15.27:0/2166129025 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec005d30 con 0x7faee4004800 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.136+0000 7faf04e00640 1 -- 172.21.15.27:0/2166129025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 msgr2=0x7faee4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.136+0000 7faf04e00640 1 --2- 172.21.15.27:0/2166129025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faee4004c80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7faeec00c160 tx=0x7faee4005830 comp rx=0 tx=0).stop 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.136+0000 7faefe800640 1 -- 172.21.15.27:0/2166129025 reap_dead start 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.136+0000 7faf04e00640 1 -- 172.21.15.27:0/2166129025 shutdown_connections 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.136+0000 7faf04e00640 1 -- 172.21.15.27:0/2166129025 >> 172.21.15.27:0/2166129025 conn(0x7faf0006b9d0 msgr2=0x7faf000c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:49.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.136+0000 7faf04e00640 1 -- 172.21.15.27:0/2166129025 shutdown_connections 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.136+0000 7faf04e00640 1 -- 172.21.15.27:0/2166129025 wait complete. 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.138+0000 7faf04e00640 1 Processor -- start 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.138+0000 7faf04e00640 1 -- start start 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faf04e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faf000c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faf04e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf0006e6d0 0x7faf00155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faf04e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf00156380 0x7faf001eb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faf04e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf000abcd0 con 0x7faee4004800 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faf04e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf000ab9f0 con 0x7faf00156380 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faf04e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf000c1810 con 0x7faf0006e6d0 2024-03-28T19:25:49.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faefde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faf000c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faefde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faf000c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32982/0 (socket says 172.21.15.27:32982) 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faefde00640 1 -- 172.21.15.27:0/3354156715 learned_addr learned my addr 172.21.15.27:0/3354156715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faefd400640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf0006e6d0 0x7faf00155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.139+0000 7faefe800640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf00156380 0x7faf001eb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faefde00640 1 -- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf0006e6d0 msgr2=0x7faf00155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faefde00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf0006e6d0 0x7faf00155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faefde00640 1 -- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf00156380 msgr2=0x7faf001eb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faefde00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf00156380 0x7faf001eb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faefde00640 1 -- 172.21.15.27:0/3354156715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf001ef770 con 0x7faee4004800 2024-03-28T19:25:49.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faefd400640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf0006e6d0 0x7faf00155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faefe800640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf00156380 0x7faf001eb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faefde00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faf000c02a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7faef000eb10 tx=0x7faef0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faef6000640 1 -- 172.21.15.27:0/3354156715 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0004030 con 0x7faee4004800 2024-03-28T19:25:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faef6000640 1 -- 172.21.15.27:0/3354156715 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef00041f0 con 0x7faee4004800 2024-03-28T19:25:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf000a8520 con 0x7faee4004800 2024-03-28T19:25:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faef6000640 1 -- 172.21.15.27:0/3354156715 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef000c870 con 0x7faee4004800 2024-03-28T19:25:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.140+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf00156810 con 0x7faee4004800 2024-03-28T19:25:49.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.142+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0004f400 con 0x7faee4004800 2024-03-28T19:25:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.143+0000 7faef6000640 1 -- 172.21.15.27:0/3354156715 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faef000ca10 con 0x7faee4004800 2024-03-28T19:25:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.144+0000 7faef6000640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faee006bbd0 0x7faee006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.144+0000 7faef6000640 1 -- 172.21.15.27:0/3354156715 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7faef008bad0 con 0x7faee4004800 2024-03-28T19:25:49.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.144+0000 7faefd400640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faee006bbd0 0x7faee006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:49.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.145+0000 7faefd400640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faee006bbd0 0x7faee006e090 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7faeec034000 tx=0x7faeec007000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:49.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.152+0000 7faef6000640 1 -- 172.21.15.27:0/3354156715 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faef0017020 con 0x7faee4004800 2024-03-28T19:25:49.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.263+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f347c06d630 con 0x7f3460004d40 2024-03-28T19:25:49.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.264+0000 7f3472a00640 1 -- 172.21.15.27:0/2931793101 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f346c058ef0 con 0x7f3460004d40 2024-03-28T19:25:49.264 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:49.264 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:49.265 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f345806bbd0 msgr2=0x7f345806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f345806bbd0 0x7f345806e090 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f347c06cc60 tx=0x7f3464008040 comp rx=0 tx=0).stop 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 msgr2=0x7f347c0c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f347c0c43e0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f346c007340 tx=0x7f346c007800 comp rx=0 tx=0).stop 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 shutdown_connections 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f345806bbd0 0x7f345806e090 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f347c0bea20 0x7f347c14f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f347c0ca9b0 0x7f347c14eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 --2- 172.21.15.27:0/2931793101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3460004d40 0x7f347c0c43e0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 >> 172.21.15.27:0/2931793101 conn(0x7f347c066fb0 msgr2=0x7f347c0673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 shutdown_connections 2024-03-28T19:25:49.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.267+0000 7f3481c00640 1 -- 172.21.15.27:0/2931793101 wait complete. 2024-03-28T19:25:49.284 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:25:49.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.386+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf00073750 con 0x7faee4004800 2024-03-28T19:25:49.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.387+0000 7faef6000640 1 -- 172.21.15.27:0/3354156715 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7faef0059540 con 0x7faee4004800 2024-03-28T19:25:49.388 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:49.388 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:49.388 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:49.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faee006bbd0 msgr2=0x7faee006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faee006bbd0 0x7faee006e090 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7faeec034000 tx=0x7faeec007000 comp rx=0 tx=0).stop 2024-03-28T19:25:49.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 msgr2=0x7faf000c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:49.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faf000c02a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7faef000eb10 tx=0x7faef0009e30 comp rx=0 tx=0).stop 2024-03-28T19:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 shutdown_connections 2024-03-28T19:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faee006bbd0 0x7faee006e090 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf00156380 0x7faf001eb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf0006e6d0 0x7faf00155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 --2- 172.21.15.27:0/3354156715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faee4004800 0x7faf000c02a0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 >> 172.21.15.27:0/3354156715 conn(0x7faf0006b9d0 msgr2=0x7faf000c2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 shutdown_connections 2024-03-28T19:25:49.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:49.390+0000 7faf04e00640 1 -- 172.21.15.27:0/3354156715 wait complete. 2024-03-28T19:25:49.408 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:25:51.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:51.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:51.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.490+0000 7faa56800640 1 Processor -- start 2024-03-28T19:25:51.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.490+0000 7faa56800640 1 -- start start 2024-03-28T19:25:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.491+0000 7faa56800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7faa500c58c0 0x7faa500c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa56800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 0x7faa50155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa56800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa50156230 con 0x7faa500c8ed0 2024-03-28T19:25:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa56800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa50158bd0 con 0x7faa500c58c0 2024-03-28T19:25:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa56800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa50158eb0 con 0x7faa500be5d0 2024-03-28T19:25:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa54a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faa500c8ed0 0x7faa501555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35472/0 (socket says 172.21.15.27:35472) 2024-03-28T19:25:51.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa54a00640 1 -- 172.21.15.27:0/1658893554 learned_addr learned my addr 172.21.15.27:0/1658893554 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa55e00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 0x7faa50155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa55400640 1 --2- 172.21.15.27:0/1658893554 >> v2:172.21.15.155:3300/0 conn(0x7faa500c58c0 0x7faa500c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.492+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425460031 0 0) 0x7faa50156230 con 0x7faa500c8ed0 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa38003520 con 0x7faa500c8ed0 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228046829 0 0) 0x7faa38003520 con 0x7faa500c8ed0 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 msgr2=0x7faa50155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 0x7faa50155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 >> v2:172.21.15.155:3300/0 conn(0x7faa500c58c0 msgr2=0x7faa500c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 --2- 172.21.15.27:0/1658893554 >> v2:172.21.15.155:3300/0 conn(0x7faa500c58c0 0x7faa500c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa50159190 con 0x7faa500c8ed0 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7faa3c001670 con 0x7faa500c8ed0 2024-03-28T19:25:51.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 >> v1:172.21.15.27:6789/0 conn(0x7faa500c8ed0 legacy=0x7faa501555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa380039b0 0x7faa38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 0x7faa380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa50158bd0 con 0x7faa38004800 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa50158eb0 con 0x7faa380039b0 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.493+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3800b9c0 con 0x7faa500be5d0 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa54a00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa55400640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa380039b0 0x7faa38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa55e00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 0x7faa380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa54a00640 1 -- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 msgr2=0x7faa380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa54a00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 0x7faa380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa54a00640 1 -- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa380039b0 msgr2=0x7faa38003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa54a00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa380039b0 0x7faa38003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa54a00640 1 -- 172.21.15.27:0/1658893554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa38003520 con 0x7faa38004800 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.494+0000 7faa55e00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500be5d0 0x7faa380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.495+0000 7faa54a00640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa38004c80 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7faa500c9300 tx=0x7faa38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fe5e70f9df83f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.495+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c007d70 con 0x7faa38004800 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.495+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c030070 con 0x7faa38004800 2024-03-28T19:25:51.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.495+0000 7faa4fe00640 1 -- 172.21.15.27:0/1658893554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c005d30 con 0x7faa38004800 2024-03-28T19:25:51.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.496+0000 7faa56800640 1 -- 172.21.15.27:0/1658893554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 msgr2=0x7faa38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.496+0000 7faa56800640 1 --2- 172.21.15.27:0/1658893554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa38004c80 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7faa500c9300 tx=0x7faa38005830 comp rx=0 tx=0).stop 2024-03-28T19:25:51.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.496+0000 7faa55e00640 1 -- 172.21.15.27:0/1658893554 reap_dead start 2024-03-28T19:25:51.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.496+0000 7faa56800640 1 -- 172.21.15.27:0/1658893554 shutdown_connections 2024-03-28T19:25:51.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.496+0000 7faa56800640 1 -- 172.21.15.27:0/1658893554 >> 172.21.15.27:0/1658893554 conn(0x7faa5006b6f0 msgr2=0x7faa500acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:51.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.496+0000 7faa56800640 1 -- 172.21.15.27:0/1658893554 shutdown_connections 2024-03-28T19:25:51.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.496+0000 7faa56800640 1 -- 172.21.15.27:0/1658893554 wait complete. 2024-03-28T19:25:51.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.498+0000 7faa56800640 1 Processor -- start 2024-03-28T19:25:51.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.498+0000 7faa56800640 1 -- start start 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa56800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa500c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa56800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500ab070 0x7faa500bea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa56800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa500c1000 0x7faa501515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa56800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa500b0820 con 0x7faa38004800 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa56800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa50159750 con 0x7faa500c1000 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa56800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa500b09a0 con 0x7faa500ab070 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa55400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa500c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa55400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa500c0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33000/0 (socket says 172.21.15.27:33000) 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.499+0000 7faa55400640 1 -- 172.21.15.27:0/2293934517 learned_addr learned my addr 172.21.15.27:0/2293934517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa55e00640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa500c1000 0x7faa501515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa54a00640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500ab070 0x7faa500bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa55400640 1 -- 172.21.15.27:0/2293934517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500ab070 msgr2=0x7faa500bea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa55400640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500ab070 0x7faa500bea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa55400640 1 -- 172.21.15.27:0/2293934517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa500c1000 msgr2=0x7faa501515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa55400640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa500c1000 0x7faa501515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa55400640 1 -- 172.21.15.27:0/2293934517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500aab90 con 0x7faa38004800 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa55400640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa500c0ac0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7faa44007f30 tx=0x7faa44011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa4d600640 1 -- 172.21.15.27:0/2293934517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4401a020 con 0x7faa38004800 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa4d600640 1 -- 172.21.15.27:0/2293934517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa44016470 con 0x7faa38004800 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa4d600640 1 -- 172.21.15.27:0/2293934517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44016860 con 0x7faa38004800 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.500+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa500c1540 con 0x7faa38004800 2024-03-28T19:25:51.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.501+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa501ec480 con 0x7faa38004800 2024-03-28T19:25:51.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.502+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa50069640 con 0x7faa38004800 2024-03-28T19:25:51.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.503+0000 7faa4d600640 1 -- 172.21.15.27:0/2293934517 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faa44005d10 con 0x7faa38004800 2024-03-28T19:25:51.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.504+0000 7faa4d600640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faa3006bc00 0x7faa3006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.504+0000 7faa4d600640 1 -- 172.21.15.27:0/2293934517 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7faa4408ca10 con 0x7faa38004800 2024-03-28T19:25:51.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.504+0000 7faa54a00640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faa3006bc00 0x7faa3006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.505+0000 7faa54a00640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faa3006bc00 0x7faa3006e0c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7faa3c034000 tx=0x7faa3c007000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:51.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.509+0000 7faa4d600640 1 -- 172.21.15.27:0/2293934517 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faa44024030 con 0x7faa38004800 2024-03-28T19:25:51.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.606+0000 7f4961a00640 1 Processor -- start 2024-03-28T19:25:51.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.606+0000 7f4961a00640 1 -- start start 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.608+0000 7f4961a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f495c152f80 0x7f495c0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.608+0000 7f4961a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7df0 0x7f495c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.608+0000 7f4961a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495c0c7310 con 0x7f495c0c7970 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.608+0000 7f4961a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495c158d40 con 0x7f495c152f80 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.608+0000 7f4961a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c159020 con 0x7f495c0c7df0 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.608+0000 7f495aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f495c152f80 0x7f495c0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.608+0000 7f495a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f495c0c7970 0x7f495c155730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35484/0 (socket says 172.21.15.27:35484) 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.608+0000 7f495a000640 1 -- 172.21.15.27:0/2869002176 learned_addr learned my addr 172.21.15.27:0/2869002176 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:51.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.609+0000 7f495b400640 1 --2- 172.21.15.27:0/2869002176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7df0 0x7f495c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.609+0000 7f495aa00640 1 -- 172.21.15.27:0/2869002176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7df0 msgr2=0x7f495c155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:51.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.609+0000 7f495aa00640 1 --2- 172.21.15.27:0/2869002176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7df0 0x7f495c155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.609+0000 7f495aa00640 1 -- 172.21.15.27:0/2869002176 >> v1:172.21.15.27:6789/0 conn(0x7f495c0c7970 legacy=0x7f495c155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.609+0000 7f495aa00640 1 -- 172.21.15.27:0/2869002176 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c159300 con 0x7f495c152f80 2024-03-28T19:25:51.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.610+0000 7f495aa00640 1 --2- 172.21.15.27:0/2869002176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c152f80 0x7f495c0c58e0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f494c0097a0 tx=0x7f494c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20b7ef3394cc53e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:51.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.610+0000 7f4959600640 1 -- 172.21.15.27:0/2869002176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c031780 con 0x7f495c152f80 2024-03-28T19:25:51.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.610+0000 7f4959600640 1 -- 172.21.15.27:0/2869002176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c037070 con 0x7f495c152f80 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.610+0000 7f4959600640 1 -- 172.21.15.27:0/2869002176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c031b40 con 0x7f495c152f80 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.611+0000 7f4961a00640 1 -- 172.21.15.27:0/2869002176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c152f80 msgr2=0x7f495c0c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.611+0000 7f4961a00640 1 --2- 172.21.15.27:0/2869002176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c152f80 0x7f495c0c58e0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f494c0097a0 tx=0x7f494c02b820 comp rx=0 tx=0).stop 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.611+0000 7f4961a00640 1 -- 172.21.15.27:0/2869002176 shutdown_connections 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.611+0000 7f4961a00640 1 --2- 172.21.15.27:0/2869002176 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7df0 0x7f495c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.611+0000 7f4961a00640 1 --2- 172.21.15.27:0/2869002176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c152f80 0x7f495c0c58e0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.611+0000 7f4961a00640 1 -- 172.21.15.27:0/2869002176 >> 172.21.15.27:0/2869002176 conn(0x7f495c06b6f0 msgr2=0x7f495c0caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.611+0000 7f4961a00640 1 -- 172.21.15.27:0/2869002176 shutdown_connections 2024-03-28T19:25:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.611+0000 7f4961a00640 1 -- 172.21.15.27:0/2869002176 wait complete. 2024-03-28T19:25:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.613+0000 7f4961a00640 1 Processor -- start 2024-03-28T19:25:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.613+0000 7f4961a00640 1 -- start start 2024-03-28T19:25:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f4961a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7970 0x7f495c1ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f4961a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c0c7df0 0x7f495c1517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f4961a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f495c152f80 0x7f495c1ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f4961a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c159b00 con 0x7f495c0c7df0 2024-03-28T19:25:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f4961a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0c5e20 con 0x7f495c152f80 2024-03-28T19:25:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f4961a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0cbca0 con 0x7f495c0c7970 2024-03-28T19:25:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c0c7df0 0x7f495c1517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c0c7df0 0x7f495c1517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33010/0 (socket says 172.21.15.27:33010) 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495a000640 1 -- 172.21.15.27:0/3333560645 learned_addr learned my addr 172.21.15.27:0/3333560645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495aa00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7970 0x7f495c1ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495b400640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f495c152f80 0x7f495c1ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495a000640 1 -- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7970 msgr2=0x7f495c1ebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495a000640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7970 0x7f495c1ebfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495a000640 1 -- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f495c152f80 msgr2=0x7f495c1ef1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.614+0000 7f495a000640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f495c152f80 0x7f495c1ef1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f495a000640 1 -- 172.21.15.27:0/3333560645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c146390 con 0x7f495c0c7df0 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f495b400640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f495c152f80 0x7f495c1ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f495aa00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7970 0x7f495c1ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f495a000640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c0c7df0 0x7f495c1517f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f494800c3c0 tx=0x7f494800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f4952a00640 1 -- 172.21.15.27:0/3333560645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948017070 con 0x7f495c0c7df0 2024-03-28T19:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f4952a00640 1 -- 172.21.15.27:0/3333560645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4948005ce0 con 0x7f495c0c7df0 2024-03-28T19:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f4952a00640 1 -- 172.21.15.27:0/3333560645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49480072f0 con 0x7f495c0c7df0 2024-03-28T19:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495c1eeca0 con 0x7f495c0c7df0 2024-03-28T19:25:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.615+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495c155280 con 0x7f495c0c7df0 2024-03-28T19:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.616+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495c04f400 con 0x7f495c0c7df0 2024-03-28T19:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.618+0000 7f4952a00640 1 -- 172.21.15.27:0/3333560645 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f494801a070 con 0x7f495c0c7df0 2024-03-28T19:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.619+0000 7f4952a00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f493c06bbd0 0x7f493c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.619+0000 7f4952a00640 1 -- 172.21.15.27:0/3333560645 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f494808afc0 con 0x7f495c0c7df0 2024-03-28T19:25:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.619+0000 7f495aa00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f493c06bbd0 0x7f493c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.620+0000 7f495aa00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f493c06bbd0 0x7f493c06e090 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f494c036000 tx=0x7f494c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:51.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.628+0000 7f4952a00640 1 -- 172.21.15.27:0/3333560645 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4948058940 con 0x7f495c0c7df0 2024-03-28T19:25:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.755+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa500be3b0 con 0x7faa38004800 2024-03-28T19:25:51.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.756+0000 7faa4d600640 1 -- 172.21.15.27:0/2293934517 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7faa4405a420 con 0x7faa38004800 2024-03-28T19:25:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:51.757 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:51.757 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.759+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faa3006bc00 msgr2=0x7faa3006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.759+0000 7faa56800640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faa3006bc00 0x7faa3006e0c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7faa3c034000 tx=0x7faa3c007000 comp rx=0 tx=0).stop 2024-03-28T19:25:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.759+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 msgr2=0x7faa500c0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.759+0000 7faa56800640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa500c0ac0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7faa44007f30 tx=0x7faa44011660 comp rx=0 tx=0).stop 2024-03-28T19:25:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.760+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 shutdown_connections 2024-03-28T19:25:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.760+0000 7faa56800640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faa3006bc00 0x7faa3006e0c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.760+0000 7faa56800640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa500c1000 0x7faa501515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.760+0000 7faa56800640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa500ab070 0x7faa500bea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.760+0000 7faa56800640 1 --2- 172.21.15.27:0/2293934517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa38004800 0x7faa500c0ac0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.760+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 >> 172.21.15.27:0/2293934517 conn(0x7faa5006b6f0 msgr2=0x7faa500c95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.760+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 shutdown_connections 2024-03-28T19:25:51.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.760+0000 7faa56800640 1 -- 172.21.15.27:0/2293934517 wait complete. 2024-03-28T19:25:51.777 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:25:51.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.871+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f495c0cba80 con 0x7f495c0c7df0 2024-03-28T19:25:51.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.872+0000 7f4952a00640 1 -- 172.21.15.27:0/3333560645 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f4948058240 con 0x7f495c0c7df0 2024-03-28T19:25:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:51.872 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:51.873 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f493c06bbd0 msgr2=0x7f493c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f493c06bbd0 0x7f493c06e090 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f494c036000 tx=0x7f494c008030 comp rx=0 tx=0).stop 2024-03-28T19:25:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c0c7df0 msgr2=0x7f495c1517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c0c7df0 0x7f495c1517f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f494800c3c0 tx=0x7f494800c880 comp rx=0 tx=0).stop 2024-03-28T19:25:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 shutdown_connections 2024-03-28T19:25:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f493c06bbd0 0x7f493c06e090 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f495c152f80 0x7f495c1ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f495c0c7df0 0x7f495c1517f0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 --2- 172.21.15.27:0/3333560645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f495c0c7970 0x7f495c1ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 >> 172.21.15.27:0/3333560645 conn(0x7f495c06b6f0 msgr2=0x7f495c0c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 shutdown_connections 2024-03-28T19:25:51.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:51.875+0000 7f4961a00640 1 -- 172.21.15.27:0/3333560645 wait complete. 2024-03-28T19:25:51.893 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:25:53.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:53.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.992+0000 7ff61aa00640 1 Processor -- start 2024-03-28T19:25:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.992+0000 7ff61aa00640 1 -- start start 2024-03-28T19:25:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.993+0000 7ff61aa00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff6141662a0 0x7ff6140a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.993+0000 7ff61aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 0x7ff6140c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.993+0000 7ff61aa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6140ac150 con 0x7ff6140a61a0 2024-03-28T19:25:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.993+0000 7ff61aa00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6140733f0 con 0x7ff6141662a0 2024-03-28T19:25:54.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.993+0000 7ff61aa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140c12d0 con 0x7ff6140c6760 2024-03-28T19:25:54.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.994+0000 7ff618a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff6140a61a0 0x7ff6140c4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35492/0 (socket says 172.21.15.27:35492) 2024-03-28T19:25:54.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.994+0000 7ff618a00640 1 -- 172.21.15.27:0/3759209397 learned_addr learned my addr 172.21.15.27:0/3759209397 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:54.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.994+0000 7ff613400640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 0x7ff6140c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.994+0000 7ff613e00640 1 --2- 172.21.15.27:0/3759209397 >> v2:172.21.15.155:3300/0 conn(0x7ff6141662a0 0x7ff6140a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:53.994+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241327069 0 0) 0x7ff600001110 con 0x7ff6140a61a0 2024-03-28T19:25:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.086+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f8003500 con 0x7ff6140a61a0 2024-03-28T19:25:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 294075816 0 0) 0x7ff5f8003500 con 0x7ff6140a61a0 2024-03-28T19:25:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 msgr2=0x7ff6140c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 0x7ff6140c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 >> v2:172.21.15.155:3300/0 conn(0x7ff6141662a0 msgr2=0x7ff6140a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 --2- 172.21.15.27:0/3759209397 >> v2:172.21.15.155:3300/0 conn(0x7ff6141662a0 0x7ff6140a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614073640 con 0x7ff6140a61a0 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7ff600001840 con 0x7ff6140a61a0 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 >> v1:172.21.15.27:6789/0 conn(0x7ff6140a61a0 legacy=0x7ff6140c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5f8003b10 0x7ff5f8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff5f8004a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 0x7ff5f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140733f0 con 0x7ff5f80045a0 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140c12d0 con 0x7ff5f8003b10 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.087+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f801ca90 con 0x7ff6140c6760 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff613400640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff5f8004a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff618a00640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 0x7ff5f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff613e00640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5f8003b10 0x7ff5f8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff613400640 1 -- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 msgr2=0x7ff5f8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff613400640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 0x7ff5f8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff613400640 1 -- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5f8003b10 msgr2=0x7ff5f8003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:54.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff613400640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5f8003b10 0x7ff5f8003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff613400640 1 -- 172.21.15.27:0/3759209397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8003500 con 0x7ff5f80045a0 2024-03-28T19:25:54.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.088+0000 7ff618a00640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140c6760 0x7ff5f8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:25:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.090+0000 7ff613400640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff5f8004a20 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ff604014f20 tx=0x7ff5f8003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e15181b3da8382 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.090+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604039070 con 0x7ff5f80045a0 2024-03-28T19:25:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.090+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60402cbf0 con 0x7ff5f80045a0 2024-03-28T19:25:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.090+0000 7ff612a00640 1 -- 172.21.15.27:0/3759209397 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604014c00 con 0x7ff5f80045a0 2024-03-28T19:25:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7ff61aa00640 1 -- 172.21.15.27:0/3759209397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 msgr2=0x7ff5f8004a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7ff61aa00640 1 --2- 172.21.15.27:0/3759209397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff5f8004a20 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7ff604014f20 tx=0x7ff5f8003890 comp rx=0 tx=0).stop 2024-03-28T19:25:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7ff618a00640 1 -- 172.21.15.27:0/3759209397 reap_dead start 2024-03-28T19:25:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7f546b200640 1 Processor -- start 2024-03-28T19:25:54.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7ff61aa00640 1 -- 172.21.15.27:0/3759209397 shutdown_connections 2024-03-28T19:25:54.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7ff61aa00640 1 -- 172.21.15.27:0/3759209397 >> 172.21.15.27:0/3759209397 conn(0x7ff6140c6c00 msgr2=0x7ff6140c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:54.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7ff61aa00640 1 -- 172.21.15.27:0/3759209397 shutdown_connections 2024-03-28T19:25:54.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7ff61aa00640 1 -- 172.21.15.27:0/3759209397 wait complete. 2024-03-28T19:25:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.091+0000 7f546b200640 1 -- start start 2024-03-28T19:25:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.092+0000 7f546b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0c58c0 0x7f546c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.092+0000 7f546b200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f546c0c8ed0 0x7f546c1555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.093+0000 7f546b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546c156230 con 0x7f546c0be5d0 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.093+0000 7f546b200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c158bd0 con 0x7f546c0c8ed0 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.093+0000 7f546b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c158eb0 con 0x7f546c0c58c0 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.093+0000 7f546a800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f546c0be5d0 0x7f546c155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35502/0 (socket says 172.21.15.27:35502) 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.093+0000 7f546a800640 1 -- 172.21.15.27:0/4286548871 learned_addr learned my addr 172.21.15.27:0/4286548871 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.093+0000 7f5469e00640 1 --2- 172.21.15.27:0/4286548871 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0c58c0 0x7f546c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.093+0000 7f5469400640 1 --2- 172.21.15.27:0/4286548871 >> v2:172.21.15.155:3300/0 conn(0x7f546c0c8ed0 0x7f546c1555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7f5468a00640 1 -- 172.21.15.27:0/4286548871 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590121955 0 0) 0x7f546c156230 con 0x7f546c0be5d0 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7f5468a00640 1 -- 172.21.15.27:0/4286548871 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450003520 con 0x7f546c0be5d0 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7f5469e00640 1 -- 172.21.15.27:0/4286548871 >> v2:172.21.15.155:3300/0 conn(0x7f546c0c8ed0 msgr2=0x7f546c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7f5469e00640 1 --2- 172.21.15.27:0/4286548871 >> v2:172.21.15.155:3300/0 conn(0x7f546c0c8ed0 0x7f546c1555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7ff61aa00640 1 Processor -- start 2024-03-28T19:25:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7f5468a00640 1 -- 172.21.15.27:0/4286548871 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 944973814 0 0) 0x7f5450003520 con 0x7f546c0be5d0 2024-03-28T19:25:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7f5469e00640 1 -- 172.21.15.27:0/4286548871 >> v1:172.21.15.27:6789/0 conn(0x7f546c0be5d0 legacy=0x7f546c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7f5469e00640 1 -- 172.21.15.27:0/4286548871 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c159190 con 0x7f546c0c58c0 2024-03-28T19:25:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.094+0000 7ff61aa00640 1 -- start start 2024-03-28T19:25:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.095+0000 7ff61aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff6140bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.095+0000 7ff61aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140a9240 0x7ff6141f93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.095+0000 7ff61aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6140bbc20 0x7ff6141f9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.095+0000 7ff61aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140abb00 con 0x7ff5f80045a0 2024-03-28T19:25:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.095+0000 7ff61aa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6140a9de0 con 0x7ff6140bbc20 2024-03-28T19:25:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.095+0000 7ff61aa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff614166680 con 0x7ff6140a9240 2024-03-28T19:25:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.095+0000 7ff613e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff6140bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff6140bffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33018/0 (socket says 172.21.15.27:33018) 2024-03-28T19:25:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613e00640 1 -- 172.21.15.27:0/3283709563 learned_addr learned my addr 172.21.15.27:0/3283709563 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7f5469e00640 1 --2- 172.21.15.27:0/4286548871 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0c58c0 0x7f546c0c5ca0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f545c0097a0 tx=0x7f545c02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d54ae013f8526d0e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613400640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140a9240 0x7ff6141f93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff618a00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6140bbc20 0x7ff6141f9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613e00640 1 -- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140a9240 msgr2=0x7ff6141f93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613e00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140a9240 0x7ff6141f93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613e00640 1 -- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6140bbc20 msgr2=0x7ff6141f9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613e00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6140bbc20 0x7ff6141f9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613e00640 1 -- 172.21.15.27:0/3283709563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6140a8690 con 0x7ff5f80045a0 2024-03-28T19:25:54.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff618a00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6140bbc20 0x7ff6141f9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7f5468a00640 1 -- 172.21.15.27:0/4286548871 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c037070 con 0x7f546c0c58c0 2024-03-28T19:25:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7f5468a00640 1 -- 172.21.15.27:0/4286548871 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c030510 con 0x7f546c0c58c0 2024-03-28T19:25:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7f5468a00640 1 -- 172.21.15.27:0/4286548871 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640019f0 con 0x7f546c0c58c0 2024-03-28T19:25:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.096+0000 7ff613e00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff6140bffa0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009630 tx=0x7ff60c009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7ff60be00640 1 -- 172.21.15.27:0/3283709563 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c01c070 con 0x7ff5f80045a0 2024-03-28T19:25:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7ff60be00640 1 -- 172.21.15.27:0/3283709563 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c00d4e0 con 0x7ff5f80045a0 2024-03-28T19:25:54.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7ff60be00640 1 -- 172.21.15.27:0/3283709563 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c00b980 con 0x7ff5f80045a0 2024-03-28T19:25:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6140c0870 con 0x7ff5f80045a0 2024-03-28T19:25:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7f546b200640 1 -- 172.21.15.27:0/4286548871 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0c58c0 msgr2=0x7f546c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7f546b200640 1 --2- 172.21.15.27:0/4286548871 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0c58c0 0x7f546c0c5ca0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f545c0097a0 tx=0x7f545c02b710 comp rx=0 tx=0).stop 2024-03-28T19:25:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7f546b200640 1 -- 172.21.15.27:0/4286548871 shutdown_connections 2024-03-28T19:25:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7f546b200640 1 --2- 172.21.15.27:0/4286548871 >> v2:172.21.15.155:3300/0 conn(0x7f546c0c8ed0 0x7f546c1555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7f546b200640 1 --2- 172.21.15.27:0/4286548871 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0c58c0 0x7f546c0c5ca0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7f546b200640 1 -- 172.21.15.27:0/4286548871 >> 172.21.15.27:0/4286548871 conn(0x7f546c06b6f0 msgr2=0x7f546c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6140a7500 con 0x7ff5f80045a0 2024-03-28T19:25:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.097+0000 7f546b200640 1 -- 172.21.15.27:0/4286548871 shutdown_connections 2024-03-28T19:25:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.098+0000 7f546b200640 1 -- 172.21.15.27:0/4286548871 wait complete. 2024-03-28T19:25:54.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.099+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6140a7040 con 0x7ff5f80045a0 2024-03-28T19:25:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.099+0000 7ff60be00640 1 -- 172.21.15.27:0/3283709563 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff60c005d10 con 0x7ff5f80045a0 2024-03-28T19:25:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.100+0000 7f546b200640 1 Processor -- start 2024-03-28T19:25:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.100+0000 7f546b200640 1 -- start start 2024-03-28T19:25:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0be5d0 0x7f546c0724a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f546c0c58c0 0x7f546c153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7ff60be00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5f406bc00 0x7ff5f406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546c0c8ed0 0x7f546c143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0b0820 con 0x7f546c0c8ed0 2024-03-28T19:25:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c159750 con 0x7f546c0c58c0 2024-03-28T19:25:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c0b09a0 con 0x7f546c0be5d0 2024-03-28T19:25:54.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546c0c8ed0 0x7f546c143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546c0c8ed0 0x7f546c143ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33020/0 (socket says 172.21.15.27:33020) 2024-03-28T19:25:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7f546a800640 1 -- 172.21.15.27:0/950543035 learned_addr learned my addr 172.21.15.27:0/950543035 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7ff60be00640 1 -- 172.21.15.27:0/3283709563 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7ff60c08c420 con 0x7ff5f80045a0 2024-03-28T19:25:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.102+0000 7f5469e00640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0be5d0 0x7f546c0724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.102+0000 7f5469400640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f546c0c58c0 0x7f546c153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.101+0000 7ff613400640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5f406bc00 0x7ff5f406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.102+0000 7f546a800640 1 -- 172.21.15.27:0/950543035 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0be5d0 msgr2=0x7f546c0724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.102+0000 7f546a800640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0be5d0 0x7f546c0724a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.102+0000 7f546a800640 1 -- 172.21.15.27:0/950543035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f546c0c58c0 msgr2=0x7f546c153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.102+0000 7f546a800640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f546c0c58c0 0x7f546c153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.102+0000 7f546a800640 1 -- 172.21.15.27:0/950543035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c0c3f60 con 0x7f546c0c8ed0 2024-03-28T19:25:54.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.102+0000 7f546a800640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546c0c8ed0 0x7f546c143ec0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f546400cc00 tx=0x7f546400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.103+0000 7ff613400640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5f406bc00 0x7ff5f406e0c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff604036000 tx=0x7ff60402e000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.105+0000 7f5462000640 1 -- 172.21.15.27:0/950543035 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546400ce10 con 0x7f546c0c8ed0 2024-03-28T19:25:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.105+0000 7f5462000640 1 -- 172.21.15.27:0/950543035 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5464012400 con 0x7f546c0c8ed0 2024-03-28T19:25:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.105+0000 7f5462000640 1 -- 172.21.15.27:0/950543035 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640127f0 con 0x7f546c0c8ed0 2024-03-28T19:25:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.105+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546c0c4240 con 0x7f546c0c8ed0 2024-03-28T19:25:54.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.106+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546c0c4800 con 0x7f546c0c8ed0 2024-03-28T19:25:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.106+0000 7ff60be00640 1 -- 172.21.15.27:0/3283709563 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff6140a7040 con 0x7ff5f80045a0 2024-03-28T19:25:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.107+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546c069640 con 0x7f546c0c8ed0 2024-03-28T19:25:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.108+0000 7f5462000640 1 -- 172.21.15.27:0/950543035 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5464012990 con 0x7f546c0c8ed0 2024-03-28T19:25:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.109+0000 7f5462000640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.109+0000 7f5462000640 1 -- 172.21.15.27:0/950543035 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f546408c2d0 con 0x7f546c0c8ed0 2024-03-28T19:25:54.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.110+0000 7f5469e00640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:54.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.110+0000 7f5469e00640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544c06bbd0 0x7f544c06e090 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f545c034000 tx=0x7f545c008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:54.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.113+0000 7f5462000640 1 -- 172.21.15.27:0/950543035 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5464059600 con 0x7f546c0c8ed0 2024-03-28T19:25:54.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.353+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f546c1eee20 con 0x7f546c0c8ed0 2024-03-28T19:25:54.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.354+0000 7f5462000640 1 -- 172.21.15.27:0/950543035 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f546405d6a0 con 0x7f546c0c8ed0 2024-03-28T19:25:54.353 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:54.354 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:54.354 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544c06bbd0 msgr2=0x7f544c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544c06bbd0 0x7f544c06e090 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f545c034000 tx=0x7f545c008030 comp rx=0 tx=0).stop 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546c0c8ed0 msgr2=0x7f546c143ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546c0c8ed0 0x7f546c143ec0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f546400cc00 tx=0x7f546400a430 comp rx=0 tx=0).stop 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 shutdown_connections 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f544c06bbd0 0x7f544c06e090 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f546c0c8ed0 0x7f546c143ec0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f546c0c58c0 0x7f546c153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 --2- 172.21.15.27:0/950543035 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f546c0be5d0 0x7f546c0724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 >> 172.21.15.27:0/950543035 conn(0x7f546c06b6f0 msgr2=0x7f546c1dec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:54.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 shutdown_connections 2024-03-28T19:25:54.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.357+0000 7f546b200640 1 -- 172.21.15.27:0/950543035 wait complete. 2024-03-28T19:25:54.374 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:25:54.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.425+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff61404f400 con 0x7ff5f80045a0 2024-03-28T19:25:54.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.426+0000 7ff60be00640 1 -- 172.21.15.27:0/3283709563 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7ff60c059840 con 0x7ff5f80045a0 2024-03-28T19:25:54.425 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:54.425 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.428+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5f406bc00 msgr2=0x7ff5f406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.428+0000 7ff61aa00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5f406bc00 0x7ff5f406e0c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff604036000 tx=0x7ff60402e000 comp rx=0 tx=0).stop 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.428+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 msgr2=0x7ff6140bffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.428+0000 7ff61aa00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff6140bffa0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009630 tx=0x7ff60c009af0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.429+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 shutdown_connections 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.429+0000 7ff61aa00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff5f406bc00 0x7ff5f406e0c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.429+0000 7ff61aa00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6140bbc20 0x7ff6141f9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.429+0000 7ff61aa00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6140a9240 0x7ff6141f93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.429+0000 7ff61aa00640 1 --2- 172.21.15.27:0/3283709563 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5f80045a0 0x7ff6140bffa0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:54.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.429+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 >> 172.21.15.27:0/3283709563 conn(0x7ff6140c6c00 msgr2=0x7ff614072bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:54.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.429+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 shutdown_connections 2024-03-28T19:25:54.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:54.429+0000 7ff61aa00640 1 -- 172.21.15.27:0/3283709563 wait complete. 2024-03-28T19:25:54.445 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:25:56.375 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:56.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:56.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.576+0000 7f59a5000640 1 Processor -- start 2024-03-28T19:25:56.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.577+0000 7f59a5000640 1 -- start start 2024-03-28T19:25:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.578+0000 7f59a5000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f59a00abf60 0x7f59a00aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.578+0000 7f59a5000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a006d8a0 con 0x7f59a00c1f50 2024-03-28T19:25:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.578+0000 7f59a5000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a006db80 con 0x7f59a00c5e80 2024-03-28T19:25:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.578+0000 7f59a5000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a006d650 con 0x7f59a00abf60 2024-03-28T19:25:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.579+0000 7f5997e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59a00c1f50 0x7f59a00ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47224/0 (socket says 172.21.15.27:47224) 2024-03-28T19:25:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.579+0000 7f5997e00640 1 -- 172.21.15.27:0/3286045039 learned_addr learned my addr 172.21.15.27:0/3286045039 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.579+0000 7f599e800640 1 --2- 172.21.15.27:0/3286045039 >> v2:172.21.15.155:3300/0 conn(0x7f59a00abf60 0x7f59a00aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.579+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473920215 0 0) 0x7f59a006d8a0 con 0x7f59a00c1f50 2024-03-28T19:25:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.579+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5984003520 con 0x7f59a00c1f50 2024-03-28T19:25:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1255324501 0 0) 0x7f5984003520 con 0x7f59a00c1f50 2024-03-28T19:25:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 >> v2:172.21.15.155:3300/0 conn(0x7f59a00abf60 msgr2=0x7f59a00aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 --2- 172.21.15.27:0/3286045039 >> v2:172.21.15.155:3300/0 conn(0x7f59a00abf60 0x7f59a00aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 >> v1:172.21.15.155:6789/0 conn(0x7f59a00c5e80 legacy=0x7f59a00c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a006de60 con 0x7f59a00c1f50 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f5988001670 con 0x7f59a00c1f50 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 >> v1:172.21.15.27:6789/0 conn(0x7f59a00c1f50 legacy=0x7f59a00ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5984003a00 0x7f5984003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5984004800 0x7f5984004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59840052e0 0x7f598400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a006d650 con 0x7f59840052e0 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598400c1e0 con 0x7f5984003a00 2024-03-28T19:25:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598400c3e0 con 0x7f5984004800 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599e800640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59840052e0 0x7f598400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.580+0000 7f599de00640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5984003a00 0x7f5984003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.581+0000 7f5997e00640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5984004800 0x7f5984004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.581+0000 7f599e800640 1 -- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5984004800 msgr2=0x7f5984004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.581+0000 7f599e800640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5984004800 0x7f5984004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.581+0000 7f599e800640 1 -- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5984003a00 msgr2=0x7f5984003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.581+0000 7f599e800640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5984003a00 0x7f5984003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.581+0000 7f599e800640 1 -- 172.21.15.27:0/3286045039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5984003520 con 0x7f59840052e0 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.581+0000 7f5997e00640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5984004800 0x7f5984004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:25:56.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.582+0000 7f599e800640 1 -- 172.21.15.27:0/3286045039 reap_dead start 2024-03-28T19:25:56.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.583+0000 7f599e800640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59840052e0 0x7f598400b9c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5998009fc0 tx=0x7f5984005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a7c6c2eb36f92b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:56.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.583+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998009c10 con 0x7f59840052e0 2024-03-28T19:25:56.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.583+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599802cca0 con 0x7f59840052e0 2024-03-28T19:25:56.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.583+0000 7f599d400640 1 -- 172.21.15.27:0/3286045039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998003bd0 con 0x7f59840052e0 2024-03-28T19:25:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.583+0000 7f59a5000640 1 -- 172.21.15.27:0/3286045039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59840052e0 msgr2=0x7f598400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.583+0000 7f59a5000640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59840052e0 0x7f598400b9c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5998009fc0 tx=0x7f5984005830 comp rx=0 tx=0).stop 2024-03-28T19:25:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.584+0000 7f59a5000640 1 -- 172.21.15.27:0/3286045039 shutdown_connections 2024-03-28T19:25:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.584+0000 7f59a5000640 1 --2- 172.21.15.27:0/3286045039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59840052e0 0x7f598400b9c0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.584+0000 7f59a5000640 1 -- 172.21.15.27:0/3286045039 >> 172.21.15.27:0/3286045039 conn(0x7f59a006b9d0 msgr2=0x7f59a00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.584+0000 7f59a5000640 1 -- 172.21.15.27:0/3286045039 shutdown_connections 2024-03-28T19:25:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.584+0000 7f59a5000640 1 -- 172.21.15.27:0/3286045039 wait complete. 2024-03-28T19:25:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.586+0000 7f59a5000640 1 Processor -- start 2024-03-28T19:25:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.586+0000 7f59a5000640 1 -- start start 2024-03-28T19:25:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f59a5000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59840052e0 0x7f59a00c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f59a5000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59a01eb660 0x7f59a00a6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f59a5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59a00a79e0 0x7f59a00a6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f59a5000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a006e1d0 con 0x7f59a00a79e0 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f59a5000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a00c6470 con 0x7f59840052e0 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f59a5000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a00ab850 con 0x7f59a01eb660 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f599e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59a00a79e0 0x7f59a00a6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f599e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59a00a79e0 0x7f59a00a6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44140/0 (socket says 172.21.15.27:44140) 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f599e800640 1 -- 172.21.15.27:0/374026292 learned_addr learned my addr 172.21.15.27:0/374026292 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f599de00640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59840052e0 0x7f59a00c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.588+0000 7f5997e00640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59a01eb660 0x7f59a00a6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f599e800640 1 -- 172.21.15.27:0/374026292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59a01eb660 msgr2=0x7f59a00a6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f599e800640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59a01eb660 0x7f59a00a6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f599e800640 1 -- 172.21.15.27:0/374026292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59840052e0 msgr2=0x7f59a00c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f599e800640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59840052e0 0x7f59a00c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f599e800640 1 -- 172.21.15.27:0/374026292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a00a7f60 con 0x7f59a00a79e0 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f599e800640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59a00a79e0 0x7f59a00a6e40 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f599802c6d0 tx=0x7f5998009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f5996000640 1 -- 172.21.15.27:0/374026292 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998003990 con 0x7f59a00a79e0 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f5996000640 1 -- 172.21.15.27:0/374026292 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998044070 con 0x7f59a00a79e0 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f5996000640 1 -- 172.21.15.27:0/374026292 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998003dd0 con 0x7f59a00a79e0 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.589+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a01eab80 con 0x7f59a00a79e0 2024-03-28T19:25:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.590+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a01eb140 con 0x7f59a00a79e0 2024-03-28T19:25:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.591+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a01ea970 con 0x7f59a00a79e0 2024-03-28T19:25:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.592+0000 7f5996000640 1 -- 172.21.15.27:0/374026292 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5998005d10 con 0x7f59a00a79e0 2024-03-28T19:25:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.593+0000 7f5996000640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f597c06bbd0 0x7f597c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.593+0000 7f5996000640 1 -- 172.21.15.27:0/374026292 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f59980ae200 con 0x7f59a00a79e0 2024-03-28T19:25:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.593+0000 7f599de00640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f597c06bbd0 0x7f597c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.594+0000 7f599de00640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f597c06bbd0 0x7f597c06e090 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5990005590 tx=0x7f5990009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:56.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.598+0000 7f5996000640 1 -- 172.21.15.27:0/374026292 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f599807bbe0 con 0x7f59a00a79e0 2024-03-28T19:25:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.652+0000 7f9033400640 1 Processor -- start 2024-03-28T19:25:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.652+0000 7f9033400640 1 -- start start 2024-03-28T19:25:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.653+0000 7f9033400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f902c0c4dc0 0x7f902c070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.653+0000 7f9033400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902c070a80 0x7f902c15ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.653+0000 7f9033400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c0c52e0 con 0x7f902c152bd0 2024-03-28T19:25:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.653+0000 7f9033400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c1557e0 con 0x7f902c0c4dc0 2024-03-28T19:25:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.653+0000 7f9033400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c155ac0 con 0x7f902c070a80 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.653+0000 7f9030a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f902c152bd0 0x7f902c06eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47238/0 (socket says 172.21.15.27:47238) 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.653+0000 7f9030a00640 1 -- 172.21.15.27:0/1097695922 learned_addr learned my addr 172.21.15.27:0/1097695922 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902be00640 1 --2- 172.21.15.27:0/1097695922 >> v2:172.21.15.155:3300/0 conn(0x7f902c0c4dc0 0x7f902c070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f9031400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902c070a80 0x7f902c15ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789875307 0 0) 0x7f902c0c52e0 con 0x7f902c152bd0 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9014003520 con 0x7f902c152bd0 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2282969815 0 0) 0x7f9014003520 con 0x7f902c152bd0 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902c070a80 msgr2=0x7f902c15ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902c070a80 0x7f902c15ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 >> v2:172.21.15.155:3300/0 conn(0x7f902c0c4dc0 msgr2=0x7f902c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 --2- 172.21.15.27:0/1097695922 >> v2:172.21.15.155:3300/0 conn(0x7f902c0c4dc0 0x7f902c070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c155da0 con 0x7f902c152bd0 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f9020001670 con 0x7f902c152bd0 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 >> v1:172.21.15.27:6789/0 conn(0x7f902c152bd0 legacy=0x7f902c06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.654+0000 7f902b400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90140039b0 0x7f9014003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f902b400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90140043a0 0x7f9014004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f902b400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9014004ef0 0x7f901400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c1557e0 con 0x7f9014004ef0 2024-03-28T19:25:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c155ac0 con 0x7f90140043a0 2024-03-28T19:25:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901400bf00 con 0x7f90140039b0 2024-03-28T19:25:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f9031400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9014004ef0 0x7f901400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f9030a00640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90140039b0 0x7f9014003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f902be00640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90140043a0 0x7f9014004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f9031400640 1 -- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90140039b0 msgr2=0x7f9014003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:56.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f9031400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90140039b0 0x7f9014003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f9031400640 1 -- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90140043a0 msgr2=0x7f9014004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.655+0000 7f9031400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90140043a0 0x7f9014004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.656+0000 7f9031400640 1 -- 172.21.15.27:0/1097695922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9014003520 con 0x7f9014004ef0 2024-03-28T19:25:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.657+0000 7f9031400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9014004ef0 0x7f901400b9c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f9024009ee0 tx=0x7f9014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f441a95836c7b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.657+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024039070 con 0x7f9014004ef0 2024-03-28T19:25:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.657+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902402cb50 con 0x7f9014004ef0 2024-03-28T19:25:56.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.657+0000 7f902b400640 1 -- 172.21.15.27:0/1097695922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024009bb0 con 0x7f9014004ef0 2024-03-28T19:25:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.658+0000 7f9033400640 1 -- 172.21.15.27:0/1097695922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9014004ef0 msgr2=0x7f901400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.658+0000 7f9033400640 1 --2- 172.21.15.27:0/1097695922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9014004ef0 0x7f901400b9c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f9024009ee0 tx=0x7f9014005830 comp rx=0 tx=0).stop 2024-03-28T19:25:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.658+0000 7f9031400640 1 -- 172.21.15.27:0/1097695922 reap_dead start 2024-03-28T19:25:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.658+0000 7f9033400640 1 -- 172.21.15.27:0/1097695922 shutdown_connections 2024-03-28T19:25:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.658+0000 7f9033400640 1 -- 172.21.15.27:0/1097695922 >> 172.21.15.27:0/1097695922 conn(0x7f902c066dd0 msgr2=0x7f902c0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.658+0000 7f9033400640 1 -- 172.21.15.27:0/1097695922 shutdown_connections 2024-03-28T19:25:56.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.658+0000 7f9033400640 1 -- 172.21.15.27:0/1097695922 wait complete. 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.660+0000 7f9033400640 1 Processor -- start 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.660+0000 7f9033400640 1 -- start start 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f9033400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9014004ef0 0x7f902c1ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f9033400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c149640 0x7f902c14a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f9033400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902c149ac0 0x7f902c14a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f9033400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0c6c70 con 0x7f902c149640 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f9033400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c152fb0 con 0x7f902c149ac0 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f9033400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0c6df0 con 0x7f9014004ef0 2024-03-28T19:25:56.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f902be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c149640 0x7f902c14a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f902be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c149640 0x7f902c14a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44154/0 (socket says 172.21.15.27:44154) 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.661+0000 7f902be00640 1 -- 172.21.15.27:0/556049747 learned_addr learned my addr 172.21.15.27:0/556049747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.662+0000 7f9030a00640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9014004ef0 0x7f902c1ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.662+0000 7f9031400640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902c149ac0 0x7f902c14a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.662+0000 7f902be00640 1 -- 172.21.15.27:0/556049747 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9014004ef0 msgr2=0x7f902c1ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.662+0000 7f902be00640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9014004ef0 0x7f902c1ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.662+0000 7f902be00640 1 -- 172.21.15.27:0/556049747 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902c149ac0 msgr2=0x7f902c14a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.662+0000 7f902be00640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902c149ac0 0x7f902c14a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.662+0000 7f902be00640 1 -- 172.21.15.27:0/556049747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c1eeed0 con 0x7f902c149640 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.662+0000 7f902be00640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c149640 0x7f902c14a2e0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f90180115e0 tx=0x7f9018011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:56.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.663+0000 7f9028c00640 1 -- 172.21.15.27:0/556049747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9018004220 con 0x7f902c149640 2024-03-28T19:25:56.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.663+0000 7f9028c00640 1 -- 172.21.15.27:0/556049747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90180043c0 con 0x7f902c149640 2024-03-28T19:25:56.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.663+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c1ef1b0 con 0x7f902c149640 2024-03-28T19:25:56.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.663+0000 7f9028c00640 1 -- 172.21.15.27:0/556049747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901800f930 con 0x7f902c149640 2024-03-28T19:25:56.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.663+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c0ab250 con 0x7f902c149640 2024-03-28T19:25:56.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.664+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c04f400 con 0x7f902c149640 2024-03-28T19:25:56.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.666+0000 7f9028c00640 1 -- 172.21.15.27:0/556049747 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f901800fad0 con 0x7f902c149640 2024-03-28T19:25:56.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.667+0000 7f9028c00640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f900c06bc50 0x7f900c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:56.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.667+0000 7f9028c00640 1 -- 172.21.15.27:0/556049747 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f901808bdd0 con 0x7f902c149640 2024-03-28T19:25:56.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.667+0000 7f9030a00640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f900c06bc50 0x7f900c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:56.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.668+0000 7f9030a00640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f900c06bc50 0x7f900c06e110 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9020001050 tx=0x7f9020009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:56.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.672+0000 7f9028c00640 1 -- 172.21.15.27:0/556049747 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9018019030 con 0x7f902c149640 2024-03-28T19:25:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.839+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59a004f400 con 0x7f59a00a79e0 2024-03-28T19:25:56.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.841+0000 7f5996000640 1 -- 172.21.15.27:0/374026292 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f599807b4e0 con 0x7f59a00a79e0 2024-03-28T19:25:56.840 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:56.841 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f597c06bbd0 msgr2=0x7f597c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f597c06bbd0 0x7f597c06e090 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5990005590 tx=0x7f5990009040 comp rx=0 tx=0).stop 2024-03-28T19:25:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59a00a79e0 msgr2=0x7f59a00a6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59a00a79e0 0x7f59a00a6e40 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f599802c6d0 tx=0x7f5998009750 comp rx=0 tx=0).stop 2024-03-28T19:25:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 shutdown_connections 2024-03-28T19:25:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f597c06bbd0 0x7f597c06e090 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59a00a79e0 0x7f59a00a6e40 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59a01eb660 0x7f59a00a6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 --2- 172.21.15.27:0/374026292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59840052e0 0x7f59a00c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 >> 172.21.15.27:0/374026292 conn(0x7f59a006b9d0 msgr2=0x7f59a0072aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 shutdown_connections 2024-03-28T19:25:56.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.843+0000 7f59a5000640 1 -- 172.21.15.27:0/374026292 wait complete. 2024-03-28T19:25:56.860 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:25:56.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.927+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f902c0c2860 con 0x7f902c149640 2024-03-28T19:25:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.928+0000 7f9028c00640 1 -- 172.21.15.27:0/556049747 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f9018059860 con 0x7f902c149640 2024-03-28T19:25:56.938 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:56.938 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:56.939 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:56.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f900c06bc50 msgr2=0x7f900c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f900c06bc50 0x7f900c06e110 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9020001050 tx=0x7f9020009040 comp rx=0 tx=0).stop 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c149640 msgr2=0x7f902c14a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c149640 0x7f902c14a2e0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f90180115e0 tx=0x7f9018011aa0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 shutdown_connections 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f900c06bc50 0x7f900c06e110 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902c149ac0 0x7f902c14a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f902c149640 0x7f902c14a2e0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 --2- 172.21.15.27:0/556049747 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9014004ef0 0x7f902c1ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.931+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 >> 172.21.15.27:0/556049747 conn(0x7f902c066dd0 msgr2=0x7f902c0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.932+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 shutdown_connections 2024-03-28T19:25:56.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:56.932+0000 7f9033400640 1 -- 172.21.15.27:0/556049747 wait complete. 2024-03-28T19:25:56.947 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:25:58.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:58.948 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:25:59.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.072+0000 7fad6d000640 1 Processor -- start 2024-03-28T19:25:59.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.073+0000 7fad6d000640 1 -- start start 2024-03-28T19:25:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad6d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68150a50 0x7fad6806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad6d000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fad680abe80 0x7fad680ac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad6d000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad680aa6d0 con 0x7fad68066dd0 2024-03-28T19:25:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad6d000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad68150e30 con 0x7fad680abe80 2024-03-28T19:25:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad6d000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6806f360 con 0x7fad68150a50 2024-03-28T19:25:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad65400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fad68066dd0 0x7fad680ad060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47250/0 (socket says 172.21.15.27:47250) 2024-03-28T19:25:59.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad65400640 1 -- 172.21.15.27:0/1030628559 learned_addr learned my addr 172.21.15.27:0/1030628559 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad66800640 1 --2- 172.21.15.27:0/1030628559 >> v2:172.21.15.155:3300/0 conn(0x7fad680abe80 0x7fad680ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.074+0000 7fad65e00640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68150a50 0x7fad6806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265335761 0 0) 0x7fad680aa6d0 con 0x7fad68066dd0 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad4c003520 con 0x7fad68066dd0 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547128224 0 0) 0x7fad4c003520 con 0x7fad68066dd0 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68150a50 msgr2=0x7fad6806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68150a50 0x7fad6806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 >> v2:172.21.15.155:3300/0 conn(0x7fad680abe80 msgr2=0x7fad680ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:59.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 --2- 172.21.15.27:0/1030628559 >> v2:172.21.15.155:3300/0 conn(0x7fad680abe80 0x7fad680ac310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad68154040 con 0x7fad68066dd0 2024-03-28T19:25:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fad54001670 con 0x7fad68066dd0 2024-03-28T19:25:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 >> v1:172.21.15.27:6789/0 conn(0x7fad68066dd0 legacy=0x7fad680ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c0039b0 0x7fad4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad4c004800 0x7fad4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68150a50 0x7fad4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad68150e30 con 0x7fad4c004800 2024-03-28T19:25:59.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c00ba70 con 0x7fad4c0039b0 2024-03-28T19:25:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.075+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c00bc20 con 0x7fad68150a50 2024-03-28T19:25:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.076+0000 7fad65400640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad4c004800 0x7fad4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.076+0000 7fad65e00640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c0039b0 0x7fad4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.076+0000 7fad65400640 1 -- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68150a50 msgr2=0x7fad4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.076+0000 7fad65400640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68150a50 0x7fad4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.076+0000 7fad65400640 1 -- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c0039b0 msgr2=0x7fad4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.076+0000 7fad65400640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c0039b0 0x7fad4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.076+0000 7fad65400640 1 -- 172.21.15.27:0/1030628559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c003520 con 0x7fad4c004800 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.076+0000 7fad65e00640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c0039b0 0x7fad4c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.077+0000 7fad65400640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad4c004800 0x7fad4c004c80 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fad54001eb0 tx=0x7fad4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b795207fe0bfae80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.077+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54007c80 con 0x7fad4c004800 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.077+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad54030070 con 0x7fad4c004800 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.077+0000 7fad64a00640 1 -- 172.21.15.27:0/1030628559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54005d50 con 0x7fad4c004800 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.078+0000 7fad6d000640 1 -- 172.21.15.27:0/1030628559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad4c004800 msgr2=0x7fad4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.078+0000 7fad6d000640 1 --2- 172.21.15.27:0/1030628559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad4c004800 0x7fad4c004c80 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fad54001eb0 tx=0x7fad4c005830 comp rx=0 tx=0).stop 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.078+0000 7fad66800640 1 -- 172.21.15.27:0/1030628559 reap_dead start 2024-03-28T19:25:59.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.078+0000 7fad6d000640 1 -- 172.21.15.27:0/1030628559 shutdown_connections 2024-03-28T19:25:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.078+0000 7fad6d000640 1 -- 172.21.15.27:0/1030628559 >> 172.21.15.27:0/1030628559 conn(0x7fad680c3ff0 msgr2=0x7fad680c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.078+0000 7fad6d000640 1 -- 172.21.15.27:0/1030628559 shutdown_connections 2024-03-28T19:25:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.078+0000 7fad6d000640 1 -- 172.21.15.27:0/1030628559 wait complete. 2024-03-28T19:25:59.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.080+0000 7fad6d000640 1 Processor -- start 2024-03-28T19:25:59.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.080+0000 7fad6d000640 1 -- start start 2024-03-28T19:25:59.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad6d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c004800 0x7fad681ebd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad6d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad681547d0 0x7fad68151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad6d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68143d60 0x7fad68144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad6d000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680672e0 con 0x7fad681547d0 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad6d000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680acbe0 con 0x7fad4c004800 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad6d000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad68067460 con 0x7fad68143d60 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad65400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad681547d0 0x7fad68151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad65400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad681547d0 0x7fad68151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44184/0 (socket says 172.21.15.27:44184) 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.081+0000 7fad65400640 1 -- 172.21.15.27:0/1831730286 learned_addr learned my addr 172.21.15.27:0/1831730286 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.082+0000 7fad65e00640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c004800 0x7fad681ebd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.082+0000 7fad66800640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68143d60 0x7fad68144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.082+0000 7fad65400640 1 -- 172.21.15.27:0/1831730286 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68143d60 msgr2=0x7fad68144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.082+0000 7fad65400640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68143d60 0x7fad68144210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.082+0000 7fad65400640 1 -- 172.21.15.27:0/1831730286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c004800 msgr2=0x7fad681ebd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.082+0000 7fad65400640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c004800 0x7fad681ebd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.082+0000 7fad65400640 1 -- 172.21.15.27:0/1831730286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad68155380 con 0x7fad681547d0 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.082+0000 7fad65400640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad681547d0 0x7fad68151d60 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fad5400c220 tx=0x7fad680ab2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.083+0000 7fad5e000640 1 -- 172.21.15.27:0/1831730286 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54045070 con 0x7fad681547d0 2024-03-28T19:25:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.083+0000 7fad5e000640 1 -- 172.21.15.27:0/1831730286 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad54001080 con 0x7fad681547d0 2024-03-28T19:25:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.083+0000 7fad5e000640 1 -- 172.21.15.27:0/1831730286 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5402ce10 con 0x7fad681547d0 2024-03-28T19:25:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.083+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad68155660 con 0x7fad681547d0 2024-03-28T19:25:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.083+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad681572f0 con 0x7fad681547d0 2024-03-28T19:25:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.084+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad68068890 con 0x7fad681547d0 2024-03-28T19:25:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.086+0000 7fad5e000640 1 -- 172.21.15.27:0/1831730286 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fad54007070 con 0x7fad681547d0 2024-03-28T19:25:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.087+0000 7fad5e000640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fad4806bc00 0x7fad4806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.087+0000 7fad5e000640 1 -- 172.21.15.27:0/1831730286 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7fad540acb20 con 0x7fad681547d0 2024-03-28T19:25:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.087+0000 7fad65e00640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fad4806bc00 0x7fad4806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.088+0000 7fad65e00640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fad4806bc00 0x7fad4806e0c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fad580099c0 tx=0x7fad58008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:59.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.092+0000 7fad5e000640 1 -- 172.21.15.27:0/1831730286 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fad68068890 con 0x7fad681547d0 2024-03-28T19:25:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.140+0000 7f0536a00640 1 Processor -- start 2024-03-28T19:25:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.141+0000 7f0536a00640 1 -- start start 2024-03-28T19:25:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.143+0000 7f0536a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05300c5e80 0x7f05300c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.143+0000 7f0536a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053006d8a0 con 0x7f05300abf60 2024-03-28T19:25:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.143+0000 7f0536a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053006db80 con 0x7f05300c1f50 2024-03-28T19:25:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.143+0000 7f0536a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053006d650 con 0x7f05300c5e80 2024-03-28T19:25:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.143+0000 7f0534a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f05300abf60 0x7f05300aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47266/0 (socket says 172.21.15.27:47266) 2024-03-28T19:25:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.143+0000 7f0534a00640 1 -- 172.21.15.27:0/902585062 learned_addr learned my addr 172.21.15.27:0/902585062 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:25:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.143+0000 7f052fe00640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05300c5e80 0x7f05300c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115471180 0 0) 0x7f053006d8a0 con 0x7f05300abf60 2024-03-28T19:25:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0514003520 con 0x7f05300abf60 2024-03-28T19:25:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1450687463 0 0) 0x7f0514003520 con 0x7f05300abf60 2024-03-28T19:25:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05300c5e80 msgr2=0x7f05300c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05300c5e80 0x7f05300c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 >> v1:172.21.15.155:6789/0 conn(0x7f05300c1f50 legacy=0x7f05300ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053006de60 con 0x7f05300abf60 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f05280019f0 con 0x7f05300abf60 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 >> v1:172.21.15.27:6789/0 conn(0x7f05300abf60 legacy=0x7f05300aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.144+0000 7f052f400640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05140039e0 0x7f0514003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f052f400640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514004800 0x7f0514004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f052f400640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05300c5e80 0x7f051400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053006d650 con 0x7f0514004800 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0514005520 con 0x7f05140039e0 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051400e110 con 0x7f05300c5e80 2024-03-28T19:25:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f0527e00640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514004800 0x7f0514004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f052fe00640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05140039e0 0x7f0514003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f0527e00640 1 -- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05300c5e80 msgr2=0x7f051400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:59.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f0527e00640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05300c5e80 0x7f051400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f0527e00640 1 -- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05140039e0 msgr2=0x7f0514003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:25:59.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f0527e00640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05140039e0 0x7f0514003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f0527e00640 1 -- 172.21.15.27:0/902585062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514003520 con 0x7f0514004800 2024-03-28T19:25:59.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.145+0000 7f052fe00640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05140039e0 0x7f0514003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:25:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.146+0000 7f0527e00640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514004800 0x7f0514004c80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f0518001720 tx=0x7f0514005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85467efd995ec207 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.147+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518001370 con 0x7f0514004800 2024-03-28T19:25:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.147+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05180052d0 con 0x7f0514004800 2024-03-28T19:25:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.147+0000 7f052f400640 1 -- 172.21.15.27:0/902585062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518003830 con 0x7f0514004800 2024-03-28T19:25:59.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.147+0000 7f0536a00640 1 -- 172.21.15.27:0/902585062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514004800 msgr2=0x7f0514004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.147+0000 7f0536a00640 1 --2- 172.21.15.27:0/902585062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0514004800 0x7f0514004c80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f0518001720 tx=0x7f0514005830 comp rx=0 tx=0).stop 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.147+0000 7f0534a00640 1 -- 172.21.15.27:0/902585062 reap_dead start 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.147+0000 7f0536a00640 1 -- 172.21.15.27:0/902585062 shutdown_connections 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.147+0000 7f0536a00640 1 -- 172.21.15.27:0/902585062 >> 172.21.15.27:0/902585062 conn(0x7f053006b9d0 msgr2=0x7f05300c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.148+0000 7f0536a00640 1 -- 172.21.15.27:0/902585062 shutdown_connections 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.148+0000 7f0536a00640 1 -- 172.21.15.27:0/902585062 wait complete. 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.149+0000 7f0536a00640 1 Processor -- start 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.149+0000 7f0536a00640 1 -- start start 2024-03-28T19:25:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.150+0000 7f0536a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0514004800 0x7f05300717a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.150+0000 7f0536a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05301ebb40 0x7f0530143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.150+0000 7f0536a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05300a9330 0x7f05301442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.150+0000 7f0536a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300aac80 con 0x7f05301ebb40 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.150+0000 7f0536a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053006e1d0 con 0x7f05300a9330 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.150+0000 7f0536a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300aae00 con 0x7f0514004800 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.150+0000 7f0527e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05301ebb40 0x7f0530143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0527e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05301ebb40 0x7f0530143d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44196/0 (socket says 172.21.15.27:44196) 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0527e00640 1 -- 172.21.15.27:0/674563443 learned_addr learned my addr 172.21.15.27:0/674563443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f052fe00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0514004800 0x7f05300717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0527e00640 1 -- 172.21.15.27:0/674563443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0514004800 msgr2=0x7f05300717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0534a00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05300a9330 0x7f05301442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0527e00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0514004800 0x7f05300717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0527e00640 1 -- 172.21.15.27:0/674563443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05300a9330 msgr2=0x7f05301442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0527e00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05300a9330 0x7f05301442c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0527e00640 1 -- 172.21.15.27:0/674563443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05301dbb90 con 0x7f05301ebb40 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.151+0000 7f0527e00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05301ebb40 0x7f0530143d80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0518005040 tx=0x7f0518001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.152+0000 7f052cc00640 1 -- 172.21.15.27:0/674563443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518041070 con 0x7f05301ebb40 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.152+0000 7f052cc00640 1 -- 172.21.15.27:0/674563443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0518005480 con 0x7f05301ebb40 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.152+0000 7f052cc00640 1 -- 172.21.15.27:0/674563443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518007a30 con 0x7f05301ebb40 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.152+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05301dbe70 con 0x7f05301ebb40 2024-03-28T19:25:59.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.152+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05301dc430 con 0x7f05301ebb40 2024-03-28T19:25:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.153+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053004f400 con 0x7f05301ebb40 2024-03-28T19:25:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.154+0000 7f052cc00640 1 -- 172.21.15.27:0/674563443 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0518044070 con 0x7f05301ebb40 2024-03-28T19:25:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.155+0000 7f052cc00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f050c06bc00 0x7f050c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:25:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.155+0000 7f052cc00640 1 -- 172.21.15.27:0/674563443 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f05180ad680 con 0x7f05301ebb40 2024-03-28T19:25:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.156+0000 7f052fe00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f050c06bc00 0x7f050c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:25:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.156+0000 7f052fe00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f050c06bc00 0x7f050c06e0c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f05200099a0 tx=0x7f0520008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:25:59.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.160+0000 7f052cc00640 1 -- 172.21.15.27:0/674563443 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f051807b090 con 0x7f05301ebb40 2024-03-28T19:25:59.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.335+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad6804f400 con 0x7fad681547d0 2024-03-28T19:25:59.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.336+0000 7fad5e000640 1 -- 172.21.15.27:0/1831730286 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7fad5407a590 con 0x7fad681547d0 2024-03-28T19:25:59.335 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:59.335 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:59.336 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fad4806bc00 msgr2=0x7fad4806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fad4806bc00 0x7fad4806e0c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fad580099c0 tx=0x7fad58008040 comp rx=0 tx=0).stop 2024-03-28T19:25:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad681547d0 msgr2=0x7fad68151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad681547d0 0x7fad68151d60 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fad5400c220 tx=0x7fad680ab2a0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 shutdown_connections 2024-03-28T19:25:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fad4806bc00 0x7fad4806e0c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68143d60 0x7fad68144210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad681547d0 0x7fad68151d60 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 --2- 172.21.15.27:0/1831730286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad4c004800 0x7fad681ebd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 >> 172.21.15.27:0/1831730286 conn(0x7fad680c3ff0 msgr2=0x7fad6806bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 shutdown_connections 2024-03-28T19:25:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.339+0000 7fad6d000640 1 -- 172.21.15.27:0/1831730286 wait complete. 2024-03-28T19:25:59.365 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:25:59.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.399+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05300ab1e0 con 0x7f05301ebb40 2024-03-28T19:25:59.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.400+0000 7f052cc00640 1 -- 172.21.15.27:0/674563443 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f051807a990 con 0x7f05301ebb40 2024-03-28T19:25:59.400 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:25:59.400 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:25:59.400 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:25:59.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f050c06bc00 msgr2=0x7f050c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f050c06bc00 0x7f050c06e0c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f05200099a0 tx=0x7f0520008040 comp rx=0 tx=0).stop 2024-03-28T19:25:59.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05301ebb40 msgr2=0x7f0530143d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:25:59.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05301ebb40 0x7f0530143d80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0518005040 tx=0x7f0518001120 comp rx=0 tx=0).stop 2024-03-28T19:25:59.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 shutdown_connections 2024-03-28T19:25:59.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f050c06bc00 0x7f050c06e0c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05300a9330 0x7f05301442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05301ebb40 0x7f0530143d80 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.403+0000 7f0536a00640 1 --2- 172.21.15.27:0/674563443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0514004800 0x7f05300717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:25:59.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.404+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 >> 172.21.15.27:0/674563443 conn(0x7f053006b9d0 msgr2=0x7f0530070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:25:59.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.404+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 shutdown_connections 2024-03-28T19:25:59.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:25:59.404+0000 7f0536a00640 1 -- 172.21.15.27:0/674563443 wait complete. 2024-03-28T19:25:59.420 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:26:01.366 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:01.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.573+0000 7f19c2000640 1 Processor -- start 2024-03-28T19:26:01.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.574+0000 7f19c2000640 1 -- start start 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19c2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 0x7f19bc0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19c2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19bc0abf60 0x7f19bc0aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19c2000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc06d8a0 con 0x7f19bc0c1f50 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19c2000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19bc06d650 con 0x7f19bc0abf60 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19c2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc06e1d0 con 0x7f19bc0c5e80 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19bbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19bc0abf60 0x7f19bc0aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19baa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19bc0c1f50 0x7f19bc0ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47278/0 (socket says 172.21.15.27:47278) 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19baa00640 1 -- 172.21.15.27:0/2835720109 learned_addr learned my addr 172.21.15.27:0/2835720109 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:26:01.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.576+0000 7f19bb400640 1 --2- 172.21.15.27:0/2835720109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 0x7f19bc0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.577+0000 7f19bbe00640 1 -- 172.21.15.27:0/2835720109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 msgr2=0x7f19bc0c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.577+0000 7f19bbe00640 1 --2- 172.21.15.27:0/2835720109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 0x7f19bc0c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.577+0000 7f19bbe00640 1 -- 172.21.15.27:0/2835720109 >> v1:172.21.15.27:6789/0 conn(0x7f19bc0c1f50 legacy=0x7f19bc0ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.577+0000 7f19bbe00640 1 -- 172.21.15.27:0/2835720109 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc06db80 con 0x7f19bc0abf60 2024-03-28T19:26:01.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.577+0000 7f19ba000640 1 -- 172.21.15.27:0/2835720109 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922195086 0 0) 0x7f19bc06d8a0 con 0x7f19bc0c1f50 2024-03-28T19:26:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.578+0000 7f19bbe00640 1 --2- 172.21.15.27:0/2835720109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc0aa570 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009d90 tx=0x7f19b402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2091ed2416bb639 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.578+0000 7f19ba000640 1 -- 172.21.15.27:0/2835720109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4039070 con 0x7f19bc0abf60 2024-03-28T19:26:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.578+0000 7f19ba000640 1 -- 172.21.15.27:0/2835720109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b402cb30 con 0x7f19bc0abf60 2024-03-28T19:26:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.578+0000 7f19ba000640 1 -- 172.21.15.27:0/2835720109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4034440 con 0x7f19bc0abf60 2024-03-28T19:26:01.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.579+0000 7f19c2000640 1 -- 172.21.15.27:0/2835720109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 msgr2=0x7f19bc0aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.579+0000 7f19c2000640 1 --2- 172.21.15.27:0/2835720109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc0aa570 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009d90 tx=0x7f19b402bd80 comp rx=0 tx=0).stop 2024-03-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.579+0000 7f19c2000640 1 -- 172.21.15.27:0/2835720109 shutdown_connections 2024-03-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.579+0000 7f19c2000640 1 --2- 172.21.15.27:0/2835720109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc0aa570 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.579+0000 7f19c2000640 1 --2- 172.21.15.27:0/2835720109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 0x7f19bc0c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.579+0000 7f19c2000640 1 -- 172.21.15.27:0/2835720109 >> 172.21.15.27:0/2835720109 conn(0x7f19bc06b9d0 msgr2=0x7f19bc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.579+0000 7f19c2000640 1 -- 172.21.15.27:0/2835720109 shutdown_connections 2024-03-28T19:26:01.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.579+0000 7f19c2000640 1 -- 172.21.15.27:0/2835720109 wait complete. 2024-03-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.581+0000 7f19c2000640 1 Processor -- start 2024-03-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.581+0000 7f19c2000640 1 -- start start 2024-03-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19c2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19c2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19bc0c1f50 0x7f19bc150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19c2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 0x7f19bc154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19c2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0c3370 con 0x7f19bc0abf60 2024-03-28T19:26:01.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19c2000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0c31f0 con 0x7f19bc0c1f50 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19c2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc0c6260 con 0x7f19bc0c5e80 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19bb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19bb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44218/0 (socket says 172.21.15.27:44218) 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19bb400640 1 -- 172.21.15.27:0/2800793711 learned_addr learned my addr 172.21.15.27:0/2800793711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19bb400640 1 -- 172.21.15.27:0/2800793711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 msgr2=0x7f19bc154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19bb400640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 0x7f19bc154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19bb400640 1 -- 172.21.15.27:0/2800793711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19bc0c1f50 msgr2=0x7f19bc150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19bb400640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19bc0c1f50 0x7f19bc150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.582+0000 7f19bb400640 1 -- 172.21.15.27:0/2800793711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc1dd400 con 0x7f19bc0abf60 2024-03-28T19:26:01.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.583+0000 7f19bb400640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc151410 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00d800 tx=0x7f19ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:01.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.583+0000 7f19b3400640 1 -- 172.21.15.27:0/2800793711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00b6c0 con 0x7f19bc0abf60 2024-03-28T19:26:01.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.583+0000 7f19b3400640 1 -- 172.21.15.27:0/2800793711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac019070 con 0x7f19bc0abf60 2024-03-28T19:26:01.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.583+0000 7f19b3400640 1 -- 172.21.15.27:0/2800793711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00ba80 con 0x7f19bc0abf60 2024-03-28T19:26:01.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.583+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19bc1dd6e0 con 0x7f19bc0abf60 2024-03-28T19:26:01.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.583+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc068210 con 0x7f19bc0abf60 2024-03-28T19:26:01.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.584+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc04f400 con 0x7f19bc0abf60 2024-03-28T19:26:01.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.585+0000 7f19b3400640 1 -- 172.21.15.27:0/2800793711 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f19ac00bc20 con 0x7f19bc0abf60 2024-03-28T19:26:01.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.586+0000 7f19b3400640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f199c06bbd0 0x7f199c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.586+0000 7f19b3400640 1 -- 172.21.15.27:0/2800793711 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f19ac08c060 con 0x7f19bc0abf60 2024-03-28T19:26:01.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.587+0000 7f19baa00640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f199c06bbd0 0x7f199c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.588+0000 7f19baa00640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f199c06bbd0 0x7f199c06e090 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f19a8007fd0 tx=0x7f19a8007960 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:01.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.591+0000 7f19b3400640 1 -- 172.21.15.27:0/2800793711 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f19ac059a70 con 0x7f19bc0abf60 2024-03-28T19:26:01.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.609+0000 7f85f8600640 1 Processor -- start 2024-03-28T19:26:01.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.609+0000 7f85f8600640 1 -- start start 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.610+0000 7f85f8600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85f00c60e0 0x7f85f006e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.610+0000 7f85f8600640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f85f0072df0 0x7f85f006ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.610+0000 7f85f8600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00c1600 con 0x7f85f00c3ea0 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.611+0000 7f85f8600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f00c23b0 con 0x7f85f00c60e0 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.611+0000 7f85f8600640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f00c18e0 con 0x7f85f0072df0 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.611+0000 7f85f5200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f85f00c3ea0 0x7f85f0158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47288/0 (socket says 172.21.15.27:47288) 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.611+0000 7f85f5200640 1 -- 172.21.15.27:0/319252613 learned_addr learned my addr 172.21.15.27:0/319252613 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.611+0000 7f85f5c00640 1 --2- 172.21.15.27:0/319252613 >> v2:172.21.15.27:3300/0 conn(0x7f85f00c60e0 0x7f85f006e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.611+0000 7f85f6600640 1 --2- 172.21.15.27:0/319252613 >> v2:172.21.15.155:3300/0 conn(0x7f85f0072df0 0x7f85f006ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.611+0000 7f85efe00640 1 -- 172.21.15.27:0/319252613 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566026979 0 0) 0x7f85f00c1600 con 0x7f85f00c3ea0 2024-03-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.612+0000 7f85efe00640 1 -- 172.21.15.27:0/319252613 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d8003520 con 0x7f85f00c3ea0 2024-03-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.612+0000 7f85f5c00640 1 -- 172.21.15.27:0/319252613 >> v2:172.21.15.155:3300/0 conn(0x7f85f0072df0 msgr2=0x7f85f006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.612+0000 7f85f5c00640 1 --2- 172.21.15.27:0/319252613 >> v2:172.21.15.155:3300/0 conn(0x7f85f0072df0 0x7f85f006ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.612+0000 7f85f5c00640 1 -- 172.21.15.27:0/319252613 >> v1:172.21.15.27:6789/0 conn(0x7f85f00c3ea0 legacy=0x7f85f0158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.612+0000 7f85f5c00640 1 -- 172.21.15.27:0/319252613 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00c1bc0 con 0x7f85f00c60e0 2024-03-28T19:26:01.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.613+0000 7f85f5c00640 1 --2- 172.21.15.27:0/319252613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c60e0 0x7f85f006e6f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f85f00c2320 tx=0x7f85e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ffffb493d97af69 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:01.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.613+0000 7f85efe00640 1 -- 172.21.15.27:0/319252613 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e40315a0 con 0x7f85f00c60e0 2024-03-28T19:26:01.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.613+0000 7f85efe00640 1 -- 172.21.15.27:0/319252613 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e4037070 con 0x7f85f00c60e0 2024-03-28T19:26:01.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.613+0000 7f85efe00640 1 -- 172.21.15.27:0/319252613 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4031940 con 0x7f85f00c60e0 2024-03-28T19:26:01.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.614+0000 7f85f8600640 1 -- 172.21.15.27:0/319252613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c60e0 msgr2=0x7f85f006e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.614+0000 7f85f8600640 1 --2- 172.21.15.27:0/319252613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c60e0 0x7f85f006e6f0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f85f00c2320 tx=0x7f85e402b840 comp rx=0 tx=0).stop 2024-03-28T19:26:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.614+0000 7f85f8600640 1 -- 172.21.15.27:0/319252613 shutdown_connections 2024-03-28T19:26:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.614+0000 7f85f8600640 1 --2- 172.21.15.27:0/319252613 >> v2:172.21.15.155:3300/0 conn(0x7f85f0072df0 0x7f85f006ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.614+0000 7f85f8600640 1 --2- 172.21.15.27:0/319252613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c60e0 0x7f85f006e6f0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.614+0000 7f85f8600640 1 -- 172.21.15.27:0/319252613 >> 172.21.15.27:0/319252613 conn(0x7f85f006b9d0 msgr2=0x7f85f00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.614+0000 7f85f8600640 1 -- 172.21.15.27:0/319252613 shutdown_connections 2024-03-28T19:26:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.614+0000 7f85f8600640 1 -- 172.21.15.27:0/319252613 wait complete. 2024-03-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.616+0000 7f85f8600640 1 Processor -- start 2024-03-28T19:26:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.616+0000 7f85f8600640 1 -- start start 2024-03-28T19:26:01.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f8600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85f0072df0 0x7f85f00a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f8600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c3ea0 0x7f85f00c3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f8600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85f00c60e0 0x7f85f0153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f8600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00c6640 con 0x7f85f00c3ea0 2024-03-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f8600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00c64c0 con 0x7f85f00c60e0 2024-03-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f8600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00c4f00 con 0x7f85f0072df0 2024-03-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c3ea0 0x7f85f00c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c3ea0 0x7f85f00c3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44244/0 (socket says 172.21.15.27:44244) 2024-03-28T19:26:01.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f5200640 1 -- 172.21.15.27:0/3617655393 learned_addr learned my addr 172.21.15.27:0/3617655393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f5c00640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85f0072df0 0x7f85f00a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f6600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85f00c60e0 0x7f85f0153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f5200640 1 -- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85f0072df0 msgr2=0x7f85f00a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.618+0000 7f85f5200640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85f0072df0 0x7f85f00a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85f5200640 1 -- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85f00c60e0 msgr2=0x7f85f0153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85f5200640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85f00c60e0 0x7f85f0153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85f5200640 1 -- 172.21.15.27:0/3617655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f01ead10 con 0x7f85f00c3ea0 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85f6600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85f00c60e0 0x7f85f0153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85f5200640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c3ea0 0x7f85f00c3400 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c430 tx=0x7f85dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:01.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85ed600640 1 -- 172.21.15.27:0/3617655393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc017070 con 0x7f85f00c3ea0 2024-03-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85ed600640 1 -- 172.21.15.27:0/3617655393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc005d00 con 0x7f85f00c3ea0 2024-03-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85ed600640 1 -- 172.21.15.27:0/3617655393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc007310 con 0x7f85f00c3ea0 2024-03-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f01eaff0 con 0x7f85f00c3ea0 2024-03-28T19:26:01.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.619+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f01eb5b0 con 0x7f85f00c3ea0 2024-03-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.620+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f01eb7b0 con 0x7f85f00c3ea0 2024-03-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.622+0000 7f85ed600640 1 -- 172.21.15.27:0/3617655393 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f85dc01a070 con 0x7f85f00c3ea0 2024-03-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.623+0000 7f85ed600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85d006bbd0 0x7f85d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:01.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.623+0000 7f85ed600640 1 -- 172.21.15.27:0/3617655393 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f85dc08b070 con 0x7f85f00c3ea0 2024-03-28T19:26:01.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.623+0000 7f85f5c00640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85d006bbd0 0x7f85d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:01.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.624+0000 7f85f5c00640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85d006bbd0 0x7f85d006e090 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f85e4036000 tx=0x7f85e403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:01.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.627+0000 7f85ed600640 1 -- 172.21.15.27:0/3617655393 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85f01eb7b0 con 0x7f85f00c3ea0 2024-03-28T19:26:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.837+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19bc1de780 con 0x7f19bc0abf60 2024-03-28T19:26:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.838+0000 7f19b3400640 1 -- 172.21.15.27:0/2800793711 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f19ac059370 con 0x7f19bc0abf60 2024-03-28T19:26:01.838 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:01.838 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:26:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:26:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.840+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f199c06bbd0 msgr2=0x7f199c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.840+0000 7f19c2000640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f199c06bbd0 0x7f199c06e090 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f19a8007fd0 tx=0x7f19a8007960 comp rx=0 tx=0).stop 2024-03-28T19:26:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.840+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 msgr2=0x7f19bc151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.840+0000 7f19c2000640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc151410 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00d800 tx=0x7f19ac00dcc0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.841+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 shutdown_connections 2024-03-28T19:26:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.841+0000 7f19c2000640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f199c06bbd0 0x7f199c06e090 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.841+0000 7f19c2000640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19bc0c5e80 0x7f19bc154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.841+0000 7f19c2000640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19bc0c1f50 0x7f19bc150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.841+0000 7f19c2000640 1 --2- 172.21.15.27:0/2800793711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19bc0abf60 0x7f19bc151410 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.841+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 >> 172.21.15.27:0/2800793711 conn(0x7f19bc06b9d0 msgr2=0x7f19bc0c5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.841+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 shutdown_connections 2024-03-28T19:26:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.841+0000 7f19c2000640 1 -- 172.21.15.27:0/2800793711 wait complete. 2024-03-28T19:26:01.857 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:26:01.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.873+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85f004f400 con 0x7f85f00c3ea0 2024-03-28T19:26:01.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.873+0000 7f85ed600640 1 -- 172.21.15.27:0/3617655393 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f85dc058a00 con 0x7f85f00c3ea0 2024-03-28T19:26:01.873 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:01.873 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:26:01.873 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:26:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85d006bbd0 msgr2=0x7f85d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85d006bbd0 0x7f85d006e090 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f85e4036000 tx=0x7f85e403d040 comp rx=0 tx=0).stop 2024-03-28T19:26:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c3ea0 msgr2=0x7f85f00c3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c3ea0 0x7f85f00c3400 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00c430 tx=0x7f85dc00c8f0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 shutdown_connections 2024-03-28T19:26:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f85d006bbd0 0x7f85d006e090 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85f00c60e0 0x7f85f0153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00c3ea0 0x7f85f00c3400 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 --2- 172.21.15.27:0/3617655393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85f0072df0 0x7f85f00a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:01.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 >> 172.21.15.27:0/3617655393 conn(0x7f85f006b9d0 msgr2=0x7f85f0155430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:01.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 shutdown_connections 2024-03-28T19:26:01.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:01.876+0000 7f85f8600640 1 -- 172.21.15.27:0/3617655393 wait complete. 2024-03-28T19:26:01.893 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:26:03.859 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:03.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:04.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.063+0000 7ff12f400640 1 Processor -- start 2024-03-28T19:26:04.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.063+0000 7ff12f400640 1 -- start start 2024-03-28T19:26:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.064+0000 7ff12f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff128150c70 0x7ff1280a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.064+0000 7ff12f400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1281534c0 0x7ff128153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.064+0000 7ff12f400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1280c3af0 con 0x7ff1280c3690 2024-03-28T19:26:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.064+0000 7ff12f400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1280ab4e0 con 0x7ff128150c70 2024-03-28T19:26:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.064+0000 7ff12f400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1280ac1f0 con 0x7ff1281534c0 2024-03-28T19:26:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.064+0000 7ff12ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff128150c70 0x7ff1280a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.064+0000 7ff12ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff128150c70 0x7ff1280a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44246/0 (socket says 172.21.15.27:44246) 2024-03-28T19:26:04.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.064+0000 7ff12ca00640 1 -- 172.21.15.27:0/3441127650 learned_addr learned my addr 172.21.15.27:0/3441127650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.065+0000 7ff12d400640 1 --2- 172.21.15.27:0/3441127650 >> v2:172.21.15.155:3300/0 conn(0x7ff1281534c0 0x7ff128153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.065+0000 7ff12ca00640 1 -- 172.21.15.27:0/3441127650 >> v2:172.21.15.155:3300/0 conn(0x7ff1281534c0 msgr2=0x7ff128153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.065+0000 7ff12ca00640 1 --2- 172.21.15.27:0/3441127650 >> v2:172.21.15.155:3300/0 conn(0x7ff1281534c0 0x7ff128153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.065+0000 7ff12ca00640 1 -- 172.21.15.27:0/3441127650 >> v1:172.21.15.155:6789/0 conn(0x7ff1280c3690 legacy=0x7ff1280c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.065+0000 7ff12ca00640 1 -- 172.21.15.27:0/3441127650 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280c3d90 con 0x7ff128150c70 2024-03-28T19:26:04.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.065+0000 7ff12d400640 1 --2- 172.21.15.27:0/3441127650 >> v2:172.21.15.155:3300/0 conn(0x7ff1281534c0 0x7ff128153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.066+0000 7ff12ca00640 1 --2- 172.21.15.27:0/3441127650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff128150c70 0x7ff1280a9cb0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7ff1280c4610 tx=0x7ff11c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974d4b914a05b34d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.066+0000 7ff127400640 1 -- 172.21.15.27:0/3441127650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c02e070 con 0x7ff128150c70 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.066+0000 7ff127400640 1 -- 172.21.15.27:0/3441127650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c037070 con 0x7ff128150c70 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.066+0000 7ff127400640 1 -- 172.21.15.27:0/3441127650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c031a00 con 0x7ff128150c70 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.067+0000 7ff12f400640 1 -- 172.21.15.27:0/3441127650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff128150c70 msgr2=0x7ff1280a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.067+0000 7ff12f400640 1 --2- 172.21.15.27:0/3441127650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff128150c70 0x7ff1280a9cb0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7ff1280c4610 tx=0x7ff11c02b880 comp rx=0 tx=0).stop 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.067+0000 7ff12f400640 1 -- 172.21.15.27:0/3441127650 shutdown_connections 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.067+0000 7ff12f400640 1 --2- 172.21.15.27:0/3441127650 >> v2:172.21.15.155:3300/0 conn(0x7ff1281534c0 0x7ff128153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.067+0000 7ff12f400640 1 --2- 172.21.15.27:0/3441127650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff128150c70 0x7ff1280a9cb0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.067+0000 7ff12f400640 1 -- 172.21.15.27:0/3441127650 >> 172.21.15.27:0/3441127650 conn(0x7ff12806b9d0 msgr2=0x7ff1280c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:04.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.068+0000 7ff12f400640 1 -- 172.21.15.27:0/3441127650 shutdown_connections 2024-03-28T19:26:04.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.068+0000 7ff12f400640 1 -- 172.21.15.27:0/3441127650 wait complete. 2024-03-28T19:26:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.070+0000 7ff12f400640 1 Processor -- start 2024-03-28T19:26:04.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.070+0000 7ff12f400640 1 -- start start 2024-03-28T19:26:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1280c3690 0x7ff128154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12f400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff128150c70 0x7ff1280a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281534c0 0x7ff1280a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12f400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280abcd0 con 0x7ff1281534c0 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12f400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280ab9f0 con 0x7ff1280c3690 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12f400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280c1810 con 0x7ff128150c70 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281534c0 0x7ff1280a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281534c0 0x7ff1280a7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44258/0 (socket says 172.21.15.27:44258) 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12d400640 1 -- 172.21.15.27:0/1438956311 learned_addr learned my addr 172.21.15.27:0/1438956311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12ca00640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1280c3690 0x7ff128154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff127e00640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff128150c70 0x7ff1280a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12d400640 1 -- 172.21.15.27:0/1438956311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff128150c70 msgr2=0x7ff1280a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12d400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff128150c70 0x7ff1280a6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12d400640 1 -- 172.21.15.27:0/1438956311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1280c3690 msgr2=0x7ff128154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12d400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1280c3690 0x7ff128154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.072+0000 7ff12d400640 1 -- 172.21.15.27:0/1438956311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280a65e0 con 0x7ff1281534c0 2024-03-28T19:26:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.073+0000 7ff12d400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281534c0 0x7ff1280a7020 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7ff12000cf20 tx=0x7ff12000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.073+0000 7ff124c00640 1 -- 172.21.15.27:0/1438956311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12001a070 con 0x7ff1281534c0 2024-03-28T19:26:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.073+0000 7ff124c00640 1 -- 172.21.15.27:0/1438956311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12000d840 con 0x7ff1281534c0 2024-03-28T19:26:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.073+0000 7ff124c00640 1 -- 172.21.15.27:0/1438956311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12000dc30 con 0x7ff1281534c0 2024-03-28T19:26:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.073+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12814c150 con 0x7ff1281534c0 2024-03-28T19:26:04.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.073+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1281552b0 con 0x7ff1281534c0 2024-03-28T19:26:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.074+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12804f400 con 0x7ff1281534c0 2024-03-28T19:26:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.076+0000 7ff124c00640 1 -- 172.21.15.27:0/1438956311 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff12001d070 con 0x7ff1281534c0 2024-03-28T19:26:04.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.077+0000 7ff124c00640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff10806bbd0 0x7ff10806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.077+0000 7ff124c00640 1 -- 172.21.15.27:0/1438956311 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7ff12008c410 con 0x7ff1281534c0 2024-03-28T19:26:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.077+0000 7ff12ca00640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff10806bbd0 0x7ff10806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.078+0000 7ff12ca00640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff10806bbd0 0x7ff10806e090 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff11c036000 tx=0x7ff11c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.085+0000 7ff124c00640 1 -- 172.21.15.27:0/1438956311 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff120059e20 con 0x7ff1281534c0 2024-03-28T19:26:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.104+0000 7f4472c00640 1 Processor -- start 2024-03-28T19:26:04.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.104+0000 7f4472c00640 1 -- start start 2024-03-28T19:26:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4472c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f446c0c8ed0 0x7f446c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4472c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0be5d0 0x7f446c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4472c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c156230 con 0x7f446c0c58c0 2024-03-28T19:26:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4472c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446c158bd0 con 0x7f446c0c8ed0 2024-03-28T19:26:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4472c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446c158eb0 con 0x7f446c0be5d0 2024-03-28T19:26:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4470e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f446c0c8ed0 0x7f446c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4471800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f446c0c58c0 0x7f446c1555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47302/0 (socket says 172.21.15.27:47302) 2024-03-28T19:26:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4470e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f446c0c8ed0 0x7f446c0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44262/0 (socket says 172.21.15.27:44262) 2024-03-28T19:26:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4471800640 1 -- 172.21.15.27:0/3113478632 learned_addr learned my addr 172.21.15.27:0/3113478632 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:26:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.106+0000 7f4472200640 1 --2- 172.21.15.27:0/3113478632 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0be5d0 0x7f446c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.107+0000 7f446be00640 1 -- 172.21.15.27:0/3113478632 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403476661 0 0) 0x7f446c156230 con 0x7f446c0c58c0 2024-03-28T19:26:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.107+0000 7f446be00640 1 -- 172.21.15.27:0/3113478632 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4454003520 con 0x7f446c0c58c0 2024-03-28T19:26:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.107+0000 7f4470e00640 1 -- 172.21.15.27:0/3113478632 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0be5d0 msgr2=0x7f446c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.107+0000 7f4470e00640 1 --2- 172.21.15.27:0/3113478632 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0be5d0 0x7f446c155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.107+0000 7f4470e00640 1 -- 172.21.15.27:0/3113478632 >> v1:172.21.15.27:6789/0 conn(0x7f446c0c58c0 legacy=0x7f446c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.107+0000 7f4470e00640 1 -- 172.21.15.27:0/3113478632 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c159190 con 0x7f446c0c8ed0 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.108+0000 7f4470e00640 1 --2- 172.21.15.27:0/3113478632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c8ed0 0x7f446c0c5ca0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f44580097a0 tx=0x7f445802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ecb22b5f97ce480 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.108+0000 7f446be00640 1 -- 172.21.15.27:0/3113478632 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458031780 con 0x7f446c0c8ed0 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.108+0000 7f446be00640 1 -- 172.21.15.27:0/3113478632 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4458037070 con 0x7f446c0c8ed0 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.108+0000 7f446be00640 1 -- 172.21.15.27:0/3113478632 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458031b40 con 0x7f446c0c8ed0 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.109+0000 7f4472c00640 1 -- 172.21.15.27:0/3113478632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c8ed0 msgr2=0x7f446c0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.109+0000 7f4472c00640 1 --2- 172.21.15.27:0/3113478632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c8ed0 0x7f446c0c5ca0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f44580097a0 tx=0x7f445802b820 comp rx=0 tx=0).stop 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.109+0000 7f4472c00640 1 -- 172.21.15.27:0/3113478632 shutdown_connections 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.109+0000 7f4472c00640 1 --2- 172.21.15.27:0/3113478632 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0be5d0 0x7f446c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.109+0000 7f4472c00640 1 --2- 172.21.15.27:0/3113478632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c8ed0 0x7f446c0c5ca0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.109+0000 7f4472c00640 1 -- 172.21.15.27:0/3113478632 >> 172.21.15.27:0/3113478632 conn(0x7f446c06b6f0 msgr2=0x7f446c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.109+0000 7f4472c00640 1 -- 172.21.15.27:0/3113478632 shutdown_connections 2024-03-28T19:26:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.109+0000 7f4472c00640 1 -- 172.21.15.27:0/3113478632 wait complete. 2024-03-28T19:26:04.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.112+0000 7f4472c00640 1 Processor -- start 2024-03-28T19:26:04.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.112+0000 7f4472c00640 1 -- start start 2024-03-28T19:26:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4472c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f446c0be5d0 0x7f446c1e13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4472c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c58c0 0x7f446c0ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4472c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0c8ed0 0x7f446c0acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4472c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c0b0820 con 0x7f446c0c58c0 2024-03-28T19:26:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4472c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446c159750 con 0x7f446c0be5d0 2024-03-28T19:26:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4472c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446c0b09a0 con 0x7f446c0c8ed0 2024-03-28T19:26:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4470e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c58c0 0x7f446c0ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4470e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c58c0 0x7f446c0ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44266/0 (socket says 172.21.15.27:44266) 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4470e00640 1 -- 172.21.15.27:0/2890683407 learned_addr learned my addr 172.21.15.27:0/2890683407 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4471800640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f446c0be5d0 0x7f446c1e13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.114+0000 7f4472200640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0c8ed0 0x7f446c0acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4470e00640 1 -- 172.21.15.27:0/2890683407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0c8ed0 msgr2=0x7f446c0acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4470e00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0c8ed0 0x7f446c0acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4470e00640 1 -- 172.21.15.27:0/2890683407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f446c0be5d0 msgr2=0x7f446c1e13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4470e00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f446c0be5d0 0x7f446c1e13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4470e00640 1 -- 172.21.15.27:0/2890683407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c1ebc80 con 0x7f446c0c58c0 2024-03-28T19:26:04.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4470e00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c58c0 0x7f446c0ac670 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f4458034040 tx=0x7f44580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4469600640 1 -- 172.21.15.27:0/2890683407 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458032610 con 0x7f446c0c58c0 2024-03-28T19:26:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4469600640 1 -- 172.21.15.27:0/2890683407 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44580327b0 con 0x7f446c0c58c0 2024-03-28T19:26:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.115+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c1ebf20 con 0x7f446c0c58c0 2024-03-28T19:26:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.116+0000 7f4469600640 1 -- 172.21.15.27:0/2890683407 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458032ba0 con 0x7f446c0c58c0 2024-03-28T19:26:04.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.116+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c1e1910 con 0x7f446c0c58c0 2024-03-28T19:26:04.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.117+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c1df640 con 0x7f446c0c58c0 2024-03-28T19:26:04.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.118+0000 7f4469600640 1 -- 172.21.15.27:0/2890683407 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f445804e430 con 0x7f446c0c58c0 2024-03-28T19:26:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.119+0000 7f4469600640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444c06bbd0 0x7f444c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:04.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.120+0000 7f4469600640 1 -- 172.21.15.27:0/2890683407 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9991+0+0 (secure 0 0 0) 0x7f44580ada00 con 0x7f446c0c58c0 2024-03-28T19:26:04.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.120+0000 7f4471800640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444c06bbd0 0x7f444c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:04.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.120+0000 7f4471800640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444c06bbd0 0x7f444c06e090 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4460005880 tx=0x7f4460009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.124+0000 7f4469600640 1 -- 172.21.15.27:0/2890683407 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4458040070 con 0x7f446c0c58c0 2024-03-28T19:26:04.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.334+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1280c5f00 con 0x7ff1281534c0 2024-03-28T19:26:04.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.335+0000 7ff124c00640 1 -- 172.21.15.27:0/1438956311 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7ff120059720 con 0x7ff1281534c0 2024-03-28T19:26:04.334 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:04.335 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:26:04.335 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:26:04.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff10806bbd0 msgr2=0x7ff10806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff10806bbd0 0x7ff10806e090 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff11c036000 tx=0x7ff11c03d040 comp rx=0 tx=0).stop 2024-03-28T19:26:04.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281534c0 msgr2=0x7ff1280a7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281534c0 0x7ff1280a7020 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7ff12000cf20 tx=0x7ff12000a7d0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 shutdown_connections 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff10806bbd0 0x7ff10806e090 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281534c0 0x7ff1280a7020 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff128150c70 0x7ff1280a6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 --2- 172.21.15.27:0/1438956311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1280c3690 0x7ff128154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 >> 172.21.15.27:0/1438956311 conn(0x7ff12806b9d0 msgr2=0x7ff1280c2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 shutdown_connections 2024-03-28T19:26:04.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.338+0000 7ff12f400640 1 -- 172.21.15.27:0/1438956311 wait complete. 2024-03-28T19:26:04.354 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:26:04.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.363+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f446c069640 con 0x7f446c0c58c0 2024-03-28T19:26:04.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.364+0000 7f4469600640 1 -- 172.21.15.27:0/2890683407 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6265 (secure 0 0 0) 0x7f445802f050 con 0x7f446c0c58c0 2024-03-28T19:26:04.364 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:04.364 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6467,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":6740,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:23:46.756300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8511},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8511":{"gid":8511,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":8,"addr":"172.21.15.27:6835/3741162434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3741162434},{"type":"v1","addr":"172.21.15.27:6835","nonce":3741162434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8511,"qdb_cluster":[8511]},"id":1},{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:25:08.729423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8586":{"gid":8586,"name":"a","rank":0,"incarnation":217,"state":"up:active","state_seq":22,"addr":"172.21.15.27:6837/1478398036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1478398036},{"type":"v1","addr":"172.21.15.27:6837","nonce":1478398036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8586,"qdb_cluster":[8586]},"id":2}]} 2024-03-28T19:26:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 221 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444c06bbd0 msgr2=0x7f444c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444c06bbd0 0x7f444c06e090 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4460005880 tx=0x7f4460009040 comp rx=0 tx=0).stop 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c58c0 msgr2=0x7f446c0ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c58c0 0x7f446c0ac670 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f4458034040 tx=0x7f44580046c0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 shutdown_connections 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f444c06bbd0 0x7f444c06e090 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f446c0c8ed0 0x7f446c0acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f446c0c58c0 0x7f446c0ac670 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 --2- 172.21.15.27:0/2890683407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f446c0be5d0 0x7f446c1e13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 >> 172.21.15.27:0/2890683407 conn(0x7f446c06b6f0 msgr2=0x7f446c0ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 shutdown_connections 2024-03-28T19:26:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:04.367+0000 7f4472c00640 1 -- 172.21.15.27:0/2890683407 wait complete. 2024-03-28T19:26:04.383 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-03-28T19:26:06.355 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:06.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:06.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.564+0000 7fd46aa00640 1 Processor -- start 2024-03-28T19:26:06.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.565+0000 7fd46aa00640 1 -- start start 2024-03-28T19:26:06.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd46aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464150ae0 0x7fd4640a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd46aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4640a61a0 0x7fd4640c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd46aa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd464150ec0 con 0x7fd4640c6ee0 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd46aa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46406dc70 con 0x7fd4640a61a0 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd46aa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd464153550 con 0x7fd464150ae0 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd468c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4640a61a0 0x7fd4640c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd468c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4640a61a0 0x7fd4640c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36844/0 (socket says 172.21.15.27:36844) 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd46a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd4640c6ee0 0x7fd464153a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55202/0 (socket says 172.21.15.27:55202) 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.566+0000 7fd468c00640 1 -- 172.21.15.27:0/2819571650 learned_addr learned my addr 172.21.15.27:0/2819571650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.567+0000 7fd469600640 1 --2- 172.21.15.27:0/2819571650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464150ae0 0x7fd4640a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.567+0000 7fd468c00640 1 -- 172.21.15.27:0/2819571650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464150ae0 msgr2=0x7fd4640a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.567+0000 7fd468c00640 1 --2- 172.21.15.27:0/2819571650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464150ae0 0x7fd4640a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.567+0000 7fd468c00640 1 -- 172.21.15.27:0/2819571650 >> v1:172.21.15.27:6789/0 conn(0x7fd4640c6ee0 legacy=0x7fd464153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.567+0000 7fd468c00640 1 -- 172.21.15.27:0/2819571650 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46406df10 con 0x7fd4640a61a0 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.567+0000 7fd463e00640 1 -- 172.21.15.27:0/2819571650 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176816536 0 0) 0x7fd45c001490 con 0x7fd4640c6ee0 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.568+0000 7fd468c00640 1 --2- 172.21.15.27:0/2819571650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4640a61a0 0x7fd4640c69a0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fd450009870 tx=0x7fd45002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf38dfb160ec977 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.568+0000 7fd463e00640 1 -- 172.21.15.27:0/2819571650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd450035020 con 0x7fd4640a61a0 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd463e00640 1 -- 172.21.15.27:0/2819571650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd450031640 con 0x7fd4640a61a0 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd463e00640 1 -- 172.21.15.27:0/2819571650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45002d040 con 0x7fd4640a61a0 2024-03-28T19:26:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd46aa00640 1 -- 172.21.15.27:0/2819571650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4640a61a0 msgr2=0x7fd4640c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd46aa00640 1 --2- 172.21.15.27:0/2819571650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4640a61a0 0x7fd4640c69a0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fd450009870 tx=0x7fd45002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd46aa00640 1 -- 172.21.15.27:0/2819571650 shutdown_connections 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd46aa00640 1 --2- 172.21.15.27:0/2819571650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4640a61a0 0x7fd4640c69a0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd46aa00640 1 --2- 172.21.15.27:0/2819571650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464150ae0 0x7fd4640a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd46aa00640 1 -- 172.21.15.27:0/2819571650 >> 172.21.15.27:0/2819571650 conn(0x7fd46406b0c0 msgr2=0x7fd46406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.569+0000 7fd46aa00640 1 -- 172.21.15.27:0/2819571650 shutdown_connections 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.570+0000 7fd46aa00640 1 -- 172.21.15.27:0/2819571650 wait complete. 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.572+0000 7fd46aa00640 1 Processor -- start 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.572+0000 7fd46aa00640 1 -- start start 2024-03-28T19:26:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4640a61a0 0x7fd4641dddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46aa00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4640c6ee0 0x7fd4641ea650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd464150ae0 0x7fd4641ec960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46406e7b0 con 0x7fd464150ae0 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46aa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46406e4d0 con 0x7fd4640a61a0 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46aa00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4641569d0 con 0x7fd4640c6ee0 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd464150ae0 0x7fd4641ec960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd464150ae0 0x7fd4641ec960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36846/0 (socket says 172.21.15.27:36846) 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd46a000640 1 -- 172.21.15.27:0/1061239682 learned_addr learned my addr 172.21.15.27:0/1061239682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd469600640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4640a61a0 0x7fd4641dddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.574+0000 7fd468c00640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4640c6ee0 0x7fd4641ea650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd46a000640 1 -- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4640c6ee0 msgr2=0x7fd4641ea650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd46a000640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4640c6ee0 0x7fd4641ea650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd46a000640 1 -- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4640a61a0 msgr2=0x7fd4641dddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd46a000640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4640a61a0 0x7fd4641dddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd46a000640 1 -- 172.21.15.27:0/1061239682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4641ed440 con 0x7fd464150ae0 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd469600640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4640a61a0 0x7fd4641dddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd46a000640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd464150ae0 0x7fd4641ec960 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00c7f0 tx=0x7fd45c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd461600640 1 -- 172.21.15.27:0/1061239682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c018070 con 0x7fd464150ae0 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd461600640 1 -- 172.21.15.27:0/1061239682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c00a5e0 con 0x7fd464150ae0 2024-03-28T19:26:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd461600640 1 -- 172.21.15.27:0/1061239682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c00aa10 con 0x7fd464150ae0 2024-03-28T19:26:06.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.575+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4641dd5d0 con 0x7fd464150ae0 2024-03-28T19:26:06.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.576+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46414d2b0 con 0x7fd464150ae0 2024-03-28T19:26:06.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.577+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd464067dd0 con 0x7fd464150ae0 2024-03-28T19:26:06.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.578+0000 7fd461600640 1 -- 172.21.15.27:0/1061239682 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd45c01b070 con 0x7fd464150ae0 2024-03-28T19:26:06.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.579+0000 7fd461600640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd44406bbd0 0x7fd44406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.579+0000 7fd461600640 1 -- 172.21.15.27:0/1061239682 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7fd45c08b0f0 con 0x7fd464150ae0 2024-03-28T19:26:06.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.580+0000 7fd469600640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd44406bbd0 0x7fd44406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.581+0000 7fd469600640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd44406bbd0 0x7fd44406e090 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd458009f30 tx=0x7fd458009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:06.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.584+0000 7fd461600640 1 -- 172.21.15.27:0/1061239682 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd464150ec0 con 0x7fd464150ae0 2024-03-28T19:26:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.611+0000 7f21d6e00640 1 Processor -- start 2024-03-28T19:26:06.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.611+0000 7f21d6e00640 1 -- start start 2024-03-28T19:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.612+0000 7f21d6e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21d00c8480 0x7f21d00c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.612+0000 7f21d6e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00a8240 0x7f21d015dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.612+0000 7f21d6e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d00bdbd0 con 0x7f21d00c52f0 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.612+0000 7f21d6e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d00bf910 con 0x7f21d00c8480 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.612+0000 7f21d6e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00bf5a0 con 0x7f21d00a8240 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.612+0000 7f21d5000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21d00c8480 0x7f21d00c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.613+0000 7f21d5a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f21d00c52f0 0x7f21d0153360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55214/0 (socket says 172.21.15.27:55214) 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.613+0000 7f21d5a00640 1 -- 172.21.15.27:0/3193066002 learned_addr learned my addr 172.21.15.27:0/3193066002 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.613+0000 7f21d6400640 1 --2- 172.21.15.27:0/3193066002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00a8240 0x7f21d015dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.613+0000 7f21d5000640 1 -- 172.21.15.27:0/3193066002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00a8240 msgr2=0x7f21d015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.613+0000 7f21d5000640 1 --2- 172.21.15.27:0/3193066002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00a8240 0x7f21d015dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.613+0000 7f21d5000640 1 -- 172.21.15.27:0/3193066002 >> v1:172.21.15.27:6789/0 conn(0x7f21d00c52f0 legacy=0x7f21d0153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.613+0000 7f21d5000640 1 -- 172.21.15.27:0/3193066002 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0153a70 con 0x7f21d00c8480 2024-03-28T19:26:06.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.613+0000 7f21cfe00640 1 -- 172.21.15.27:0/3193066002 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322297266 0 0) 0x7f21d00bdbd0 con 0x7f21d00c52f0 2024-03-28T19:26:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.615+0000 7f21d5000640 1 --2- 172.21.15.27:0/3193066002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00c8480 0x7f21d00c56d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0097a0 tx=0x7f21bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dee0164b184df9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:06.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.615+0000 7f21cfe00640 1 -- 172.21.15.27:0/3193066002 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc035020 con 0x7f21d00c8480 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.615+0000 7f21cfe00640 1 -- 172.21.15.27:0/3193066002 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc031830 con 0x7f21d00c8480 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.615+0000 7f21cfe00640 1 -- 172.21.15.27:0/3193066002 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc031c00 con 0x7f21d00c8480 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.616+0000 7f21d6e00640 1 -- 172.21.15.27:0/3193066002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00c8480 msgr2=0x7f21d00c56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.616+0000 7f21d6e00640 1 --2- 172.21.15.27:0/3193066002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00c8480 0x7f21d00c56d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0097a0 tx=0x7f21bc02b8d0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.616+0000 7f21d6e00640 1 -- 172.21.15.27:0/3193066002 shutdown_connections 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.616+0000 7f21d6e00640 1 --2- 172.21.15.27:0/3193066002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00a8240 0x7f21d015dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.616+0000 7f21d6e00640 1 --2- 172.21.15.27:0/3193066002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00c8480 0x7f21d00c56d0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.616+0000 7f21d6e00640 1 -- 172.21.15.27:0/3193066002 >> 172.21.15.27:0/3193066002 conn(0x7f21d00aa7c0 msgr2=0x7f21d00aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.616+0000 7f21d6e00640 1 -- 172.21.15.27:0/3193066002 shutdown_connections 2024-03-28T19:26:06.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.616+0000 7f21d6e00640 1 -- 172.21.15.27:0/3193066002 wait complete. 2024-03-28T19:26:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.618+0000 7f21d6e00640 1 Processor -- start 2024-03-28T19:26:06.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.618+0000 7f21d6e00640 1 -- start start 2024-03-28T19:26:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00a8240 0x7f21d0154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d6e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21d00c52f0 0x7f21d00c2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d6e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00c8480 0x7f21d00c2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00a7e00 con 0x7f21d00a8240 2024-03-28T19:26:06.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d6e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00a7c80 con 0x7f21d00c52f0 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d6e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d0157530 con 0x7f21d00c8480 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d5a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00a8240 0x7f21d0154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d5000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21d00c52f0 0x7f21d00c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d5a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00a8240 0x7f21d0154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36866/0 (socket says 172.21.15.27:36866) 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d5a00640 1 -- 172.21.15.27:0/2071034615 learned_addr learned my addr 172.21.15.27:0/2071034615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.620+0000 7f21d6400640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00c8480 0x7f21d00c2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.621+0000 7f21d5000640 1 -- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00c8480 msgr2=0x7f21d00c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.621+0000 7f21d5000640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00c8480 0x7f21d00c2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.621+0000 7f21d5000640 1 -- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00a8240 msgr2=0x7f21d0154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.621+0000 7f21d5000640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00a8240 0x7f21d0154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.621+0000 7f21d5000640 1 -- 172.21.15.27:0/2071034615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00c0580 con 0x7f21d00c52f0 2024-03-28T19:26:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.621+0000 7f21d6400640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00c8480 0x7f21d00c2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:26:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.621+0000 7f21d5a00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00a8240 0x7f21d0154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:26:06.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.621+0000 7f21d5000640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21d00c52f0 0x7f21d00c2440 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f21bc033000 tx=0x7f21bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.622+0000 7f21cd600640 1 -- 172.21.15.27:0/2071034615 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc040070 con 0x7f21d00c52f0 2024-03-28T19:26:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.622+0000 7f21cd600640 1 -- 172.21.15.27:0/2071034615 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc03a930 con 0x7f21d00c52f0 2024-03-28T19:26:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.622+0000 7f21cd600640 1 -- 172.21.15.27:0/2071034615 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc035040 con 0x7f21d00c52f0 2024-03-28T19:26:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.622+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d00c1f60 con 0x7f21d00c52f0 2024-03-28T19:26:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.622+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d00c0820 con 0x7f21d00c52f0 2024-03-28T19:26:06.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.623+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d0067dd0 con 0x7f21d00c52f0 2024-03-28T19:26:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.624+0000 7f21cd600640 1 -- 172.21.15.27:0/2071034615 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f21bc04b060 con 0x7f21d00c52f0 2024-03-28T19:26:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.625+0000 7f21cd600640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f21b006bbd0 0x7f21b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.625+0000 7f21cd600640 1 -- 172.21.15.27:0/2071034615 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f21bc0adaf0 con 0x7f21d00c52f0 2024-03-28T19:26:06.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.626+0000 7f21d5a00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f21b006bbd0 0x7f21b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:06.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.626+0000 7f21d5a00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f21b006bbd0 0x7f21b006e090 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f21c4000f60 tx=0x7f21c4007e10 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:06.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.633+0000 7f21cd600640 1 -- 172.21.15.27:0/2071034615 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f21bc031800 con 0x7f21d00c52f0 2024-03-28T19:26:06.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.835+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd464069640 con 0x7fd464150ae0 2024-03-28T19:26:06.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.836+0000 7fd461600640 1 -- 172.21.15.27:0/1061239682 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+4667 (secure 0 0 0) 0x7fd45c058460 con 0x7fd464150ae0 2024-03-28T19:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:06.836 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:05.456779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:05.456780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:06.836 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 222 2024-03-28T19:26:06.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.839+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd44406bbd0 msgr2=0x7fd44406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.839+0000 7fd46aa00640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd44406bbd0 0x7fd44406e090 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd458009f30 tx=0x7fd458009190 comp rx=0 tx=0).stop 2024-03-28T19:26:06.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.839+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd464150ae0 msgr2=0x7fd4641ec960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.839+0000 7fd46aa00640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd464150ae0 0x7fd4641ec960 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00c7f0 tx=0x7fd45c00ccb0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.840+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 shutdown_connections 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.840+0000 7fd46aa00640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd44406bbd0 0x7fd44406e090 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.840+0000 7fd46aa00640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd464150ae0 0x7fd4641ec960 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.840+0000 7fd46aa00640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd4640c6ee0 0x7fd4641ea650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.840+0000 7fd46aa00640 1 --2- 172.21.15.27:0/1061239682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4640a61a0 0x7fd4641dddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.840+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 >> 172.21.15.27:0/1061239682 conn(0x7fd46406b0c0 msgr2=0x7fd46406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.840+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 shutdown_connections 2024-03-28T19:26:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.840+0000 7fd46aa00640 1 -- 172.21.15.27:0/1061239682 wait complete. 2024-03-28T19:26:06.864 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-03-28T19:26:06.864 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:26:06.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:06.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.883+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21d00c6520 con 0x7f21d00c52f0 2024-03-28T19:26:06.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.884+0000 7f21cd600640 1 -- 172.21.15.27:0/2071034615 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+4667 (secure 0 0 0) 0x7f21bc07adf0 con 0x7f21d00c52f0 2024-03-28T19:26:06.884 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:06.884 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:05.456779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:05.456780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:06.884 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 222 2024-03-28T19:26:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f21b006bbd0 msgr2=0x7f21b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f21b006bbd0 0x7f21b006e090 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f21c4000f60 tx=0x7f21c4007e10 comp rx=0 tx=0).stop 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21d00c52f0 msgr2=0x7f21d00c2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21d00c52f0 0x7f21d00c2440 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f21bc033000 tx=0x7f21bc0046c0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 shutdown_connections 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f21b006bbd0 0x7f21b006e090 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21d00c8480 0x7f21d00c2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21d00c52f0 0x7f21d00c2440 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 --2- 172.21.15.27:0/2071034615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21d00a8240 0x7f21d0154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 >> 172.21.15.27:0/2071034615 conn(0x7f21d00aa7c0 msgr2=0x7f21d00c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 shutdown_connections 2024-03-28T19:26:06.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:06.887+0000 7f21d6e00640 1 -- 172.21.15.27:0/2071034615 wait complete. 2024-03-28T19:26:06.907 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-03-28T19:26:06.907 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:26:06.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.056+0000 7fa0d2000640 1 Processor -- start 2024-03-28T19:26:07.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.056+0000 7fa0d2000640 1 -- start start 2024-03-28T19:26:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.057+0000 7fa0d2000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa0cc0c7970 0x7fa0cc0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.057+0000 7fa0d2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0cc155730 0x7fa0cc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.057+0000 7fa0d2000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0c7310 con 0x7fa0cc152f80 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.057+0000 7fa0d2000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0cc158d40 con 0x7fa0cc155730 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.057+0000 7fa0d2000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0cc159020 con 0x7fa0cc0c7970 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0cb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa0cc152f80 0x7fa0cc0c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55226/0 (socket says 172.21.15.27:55226) 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0cb400640 1 -- 172.21.15.27:0/3300753677 learned_addr learned my addr 172.21.15.27:0/3300753677 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0cbe00640 1 --2- 172.21.15.27:0/3300753677 >> v2:172.21.15.27:3300/0 conn(0x7fa0cc155730 0x7fa0cc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0caa00640 1 --2- 172.21.15.27:0/3300753677 >> v2:172.21.15.155:3300/0 conn(0x7fa0cc0c7970 0x7fa0cc0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0cbe00640 1 -- 172.21.15.27:0/3300753677 >> v2:172.21.15.155:3300/0 conn(0x7fa0cc0c7970 msgr2=0x7fa0cc0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:07.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0cbe00640 1 --2- 172.21.15.27:0/3300753677 >> v2:172.21.15.155:3300/0 conn(0x7fa0cc0c7970 0x7fa0cc0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0cbe00640 1 -- 172.21.15.27:0/3300753677 >> v1:172.21.15.27:6789/0 conn(0x7fa0cc152f80 legacy=0x7fa0cc0c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0cbe00640 1 -- 172.21.15.27:0/3300753677 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc159300 con 0x7fa0cc155730 2024-03-28T19:26:07.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.058+0000 7fa0caa00640 1 --2- 172.21.15.27:0/3300753677 >> v2:172.21.15.155:3300/0 conn(0x7fa0cc0c7970 0x7fa0cc0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:26:07.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.060+0000 7fa0cbe00640 1 --2- 172.21.15.27:0/3300753677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc155ba0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0c8480 tx=0x7fa0c402be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ceb1edc6cc3899e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:07.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.060+0000 7fa0ca000640 1 -- 172.21.15.27:0/3300753677 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4039070 con 0x7fa0cc155730 2024-03-28T19:26:07.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.060+0000 7fa0ca000640 1 -- 172.21.15.27:0/3300753677 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c402cbe0 con 0x7fa0cc155730 2024-03-28T19:26:07.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.060+0000 7fa0ca000640 1 -- 172.21.15.27:0/3300753677 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4034440 con 0x7fa0cc155730 2024-03-28T19:26:07.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.061+0000 7fa0d2000640 1 -- 172.21.15.27:0/3300753677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 msgr2=0x7fa0cc155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.061+0000 7fa0d2000640 1 --2- 172.21.15.27:0/3300753677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc155ba0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0c8480 tx=0x7fa0c402be30 comp rx=0 tx=0).stop 2024-03-28T19:26:07.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.061+0000 7fa0d2000640 1 -- 172.21.15.27:0/3300753677 shutdown_connections 2024-03-28T19:26:07.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.061+0000 7fa0d2000640 1 --2- 172.21.15.27:0/3300753677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc155ba0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.061+0000 7fa0d2000640 1 --2- 172.21.15.27:0/3300753677 >> v2:172.21.15.155:3300/0 conn(0x7fa0cc0c7970 0x7fa0cc0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.061+0000 7fa0d2000640 1 -- 172.21.15.27:0/3300753677 >> 172.21.15.27:0/3300753677 conn(0x7fa0cc06b6f0 msgr2=0x7fa0cc0caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:07.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.061+0000 7fa0d2000640 1 -- 172.21.15.27:0/3300753677 shutdown_connections 2024-03-28T19:26:07.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.061+0000 7fa0d2000640 1 -- 172.21.15.27:0/3300753677 wait complete. 2024-03-28T19:26:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.062+0000 7fa0d2000640 1 Processor -- start 2024-03-28T19:26:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.063+0000 7fa0d2000640 1 -- start start 2024-03-28T19:26:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.063+0000 7fa0d2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0cc0c7970 0x7fa0cc0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.063+0000 7fa0d2000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0cc152f80 0x7fa0cc0ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.063+0000 7fa0d2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc0cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.063+0000 7fa0d2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0cbca0 con 0x7fa0cc155730 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.063+0000 7fa0d2000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc159b00 con 0x7fa0cc0c7970 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.063+0000 7fa0d2000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc0c5040 con 0x7fa0cc152f80 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc0cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc0cab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36890/0 (socket says 172.21.15.27:36890) 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 -- 172.21.15.27:0/562524334 learned_addr learned my addr 172.21.15.27:0/562524334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cb400640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0cc0c7970 0x7fa0cc0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0caa00640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0cc152f80 0x7fa0cc0ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 -- 172.21.15.27:0/562524334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0cc152f80 msgr2=0x7fa0cc0ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0cc152f80 0x7fa0cc0ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 -- 172.21.15.27:0/562524334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0cc0c7970 msgr2=0x7fa0cc0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0cc0c7970 0x7fa0cc0c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 -- 172.21.15.27:0/562524334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc1eb4f0 con 0x7fa0cc155730 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.064+0000 7fa0cbe00640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc0cab90 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009e80 tx=0x7fa0c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.065+0000 7fa0c3400640 1 -- 172.21.15.27:0/562524334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4039050 con 0x7fa0cc155730 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.065+0000 7fa0c3400640 1 -- 172.21.15.27:0/562524334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c4045070 con 0x7fa0cc155730 2024-03-28T19:26:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.065+0000 7fa0c3400640 1 -- 172.21.15.27:0/562524334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4009450 con 0x7fa0cc155730 2024-03-28T19:26:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.065+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc1eb7d0 con 0x7fa0cc155730 2024-03-28T19:26:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.065+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc1ebd90 con 0x7fa0cc155730 2024-03-28T19:26:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.066+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc04f400 con 0x7fa0cc155730 2024-03-28T19:26:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.068+0000 7fa0c3400640 1 -- 172.21.15.27:0/562524334 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa0c403e850 con 0x7fa0cc155730 2024-03-28T19:26:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.069+0000 7fa0c3400640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.069+0000 7fa0c3400640 1 -- 172.21.15.27:0/562524334 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7fa0c40af330 con 0x7fa0cc155730 2024-03-28T19:26:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.069+0000 7fa0cb400640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.070+0000 7fa0cb400640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc000f60 tx=0x7fa0bc009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:07.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.076+0000 7fa0c3400640 1 -- 172.21.15.27:0/562524334 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa0c407ce30 con 0x7fa0cc155730 2024-03-28T19:26:07.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.134+0000 7f0f85e00640 1 Processor -- start 2024-03-28T19:26:07.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.134+0000 7f0f85e00640 1 -- start start 2024-03-28T19:26:07.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.135+0000 7f0f85e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f80150c70 0x7f0f800a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.135+0000 7f0f85e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f800c3690 0x7f0f800c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.135+0000 7f0f85e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0f800c6660 0x7f0f801534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.135+0000 7f0f85e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f800ab4e0 con 0x7f0f800c3690 2024-03-28T19:26:07.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.135+0000 7f0f85e00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f800ac1f0 con 0x7f0f800c6660 2024-03-28T19:26:07.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.135+0000 7f0f85e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800c3af0 con 0x7f0f80150c70 2024-03-28T19:26:07.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f800c3690 0x7f0f800c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0f800c3690 0x7f0f800c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36906/0 (socket says 172.21.15.27:36906) 2024-03-28T19:26:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7e000640 1 -- 172.21.15.27:0/1451445155 learned_addr learned my addr 172.21.15.27:0/1451445155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7f400640 1 --2- 172.21.15.27:0/1451445155 >> v2:172.21.15.155:3300/0 conn(0x7f0f800c6660 0x7f0f801534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7ea00640 1 --2- 172.21.15.27:0/1451445155 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f80150c70 0x7f0f800a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7e000640 1 -- 172.21.15.27:0/1451445155 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f80150c70 msgr2=0x7f0f800a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7e000640 1 --2- 172.21.15.27:0/1451445155 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f80150c70 0x7f0f800a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7e000640 1 -- 172.21.15.27:0/1451445155 >> v2:172.21.15.155:3300/0 conn(0x7f0f800c6660 msgr2=0x7f0f801534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7e000640 1 --2- 172.21.15.27:0/1451445155 >> v2:172.21.15.155:3300/0 conn(0x7f0f800c6660 0x7f0f801534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7e000640 1 -- 172.21.15.27:0/1451445155 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f800c3d90 con 0x7f0f800c3690 2024-03-28T19:26:07.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.136+0000 7f0f7f400640 1 --2- 172.21.15.27:0/1451445155 >> v2:172.21.15.155:3300/0 conn(0x7f0f800c6660 0x7f0f801534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:26:07.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.137+0000 7f0f7e000640 1 --2- 172.21.15.27:0/1451445155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f800c6120 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0f8006c6a0 tx=0x7f0f6c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed655b8f7d69c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f7d600640 1 -- 172.21.15.27:0/1451445155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c0317c0 con 0x7f0f800c3690 2024-03-28T19:26:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f7d600640 1 -- 172.21.15.27:0/1451445155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c037070 con 0x7f0f800c3690 2024-03-28T19:26:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f7d600640 1 -- 172.21.15.27:0/1451445155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c031b80 con 0x7f0f800c3690 2024-03-28T19:26:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f85e00640 1 -- 172.21.15.27:0/1451445155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 msgr2=0x7f0f800c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f85e00640 1 --2- 172.21.15.27:0/1451445155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f800c6120 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0f8006c6a0 tx=0x7f0f6c02b860 comp rx=0 tx=0).stop 2024-03-28T19:26:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f85e00640 1 -- 172.21.15.27:0/1451445155 shutdown_connections 2024-03-28T19:26:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f85e00640 1 --2- 172.21.15.27:0/1451445155 >> v2:172.21.15.155:3300/0 conn(0x7f0f800c6660 0x7f0f801534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f85e00640 1 --2- 172.21.15.27:0/1451445155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f800c6120 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f85e00640 1 --2- 172.21.15.27:0/1451445155 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f80150c70 0x7f0f800a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.138+0000 7f0f85e00640 1 -- 172.21.15.27:0/1451445155 >> 172.21.15.27:0/1451445155 conn(0x7f0f8006b9d0 msgr2=0x7f0f800c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:07.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.139+0000 7f0f85e00640 1 -- 172.21.15.27:0/1451445155 shutdown_connections 2024-03-28T19:26:07.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.139+0000 7f0f85e00640 1 -- 172.21.15.27:0/1451445155 wait complete. 2024-03-28T19:26:07.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.140+0000 7f0f85e00640 1 Processor -- start 2024-03-28T19:26:07.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.140+0000 7f0f85e00640 1 -- start start 2024-03-28T19:26:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.141+0000 7f0f85e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f8014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.141+0000 7f0f85e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f800c6660 0x7f0f801e1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.141+0000 7f0f85e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f80150c70 0x7f0f8006dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.141+0000 7f0f85e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800abcd0 con 0x7f0f800c3690 2024-03-28T19:26:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.141+0000 7f0f85e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800ab9f0 con 0x7f0f80150c70 2024-03-28T19:26:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.141+0000 7f0f85e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800c1810 con 0x7f0f800c6660 2024-03-28T19:26:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.141+0000 7f0f7ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f8014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f8014bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36920/0 (socket says 172.21.15.27:36920) 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7ea00640 1 -- 172.21.15.27:0/3084090867 learned_addr learned my addr 172.21.15.27:0/3084090867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7f400640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f80150c70 0x7f0f8006dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7e000640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f800c6660 0x7f0f801e1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7ea00640 1 -- 172.21.15.27:0/3084090867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f800c6660 msgr2=0x7f0f801e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7ea00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f800c6660 0x7f0f801e1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7ea00640 1 -- 172.21.15.27:0/3084090867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f80150c70 msgr2=0x7f0f8006dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7ea00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f80150c70 0x7f0f8006dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7ea00640 1 -- 172.21.15.27:0/3084090867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f801ebaa0 con 0x7f0f800c3690 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.142+0000 7f0f7ea00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f8014bfe0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000d830 tx=0x7f0f7000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.143+0000 7f0f77400640 1 -- 172.21.15.27:0/3084090867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000b6c0 con 0x7f0f800c3690 2024-03-28T19:26:07.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.143+0000 7f0f77400640 1 -- 172.21.15.27:0/3084090867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f70019070 con 0x7f0f800c3690 2024-03-28T19:26:07.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.143+0000 7f0f77400640 1 -- 172.21.15.27:0/3084090867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000ba80 con 0x7f0f800c3690 2024-03-28T19:26:07.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.143+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f801ebd40 con 0x7f0f800c3690 2024-03-28T19:26:07.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.143+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8006eba0 con 0x7f0f800c3690 2024-03-28T19:26:07.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.144+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8004f400 con 0x7f0f800c3690 2024-03-28T19:26:07.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.145+0000 7f0f77400640 1 -- 172.21.15.27:0/3084090867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0f7001e460 con 0x7f0f800c3690 2024-03-28T19:26:07.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.146+0000 7f0f77400640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f6006bc50 0x7f0f6006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:07.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.146+0000 7f0f77400640 1 -- 172.21.15.27:0/3084090867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f0f7008c0a0 con 0x7f0f800c3690 2024-03-28T19:26:07.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.147+0000 7f0f7e000640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f6006bc50 0x7f0f6006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:07.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.148+0000 7f0f7e000640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f6006bc50 0x7f0f6006e110 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c036000 tx=0x7f0f6c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.151+0000 7f0f77400640 1 -- 172.21.15.27:0/3084090867 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0f70059ae0 con 0x7f0f800c3690 2024-03-28T19:26:07.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.343+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0cc0be400 con 0x7fa0cc155730 2024-03-28T19:26:07.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.344+0000 7fa0c3400640 1 -- 172.21.15.27:0/562524334 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+4667 (secure 0 0 0) 0x7fa0c407c730 con 0x7fa0cc155730 2024-03-28T19:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:07.344 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:05.456779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:05.456780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:07.344 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 222 2024-03-28T19:26:07.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa0ac06bc00 msgr2=0x7fa0ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc000f60 tx=0x7fa0bc009040 comp rx=0 tx=0).stop 2024-03-28T19:26:07.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 msgr2=0x7fa0cc0cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc0cab90 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4009e80 tx=0x7fa0c4004a40 comp rx=0 tx=0).stop 2024-03-28T19:26:07.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 shutdown_connections 2024-03-28T19:26:07.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa0ac06bc00 0x7fa0ac06e0c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0cc155730 0x7fa0cc0cab90 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa0cc152f80 0x7fa0cc0ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 --2- 172.21.15.27:0/562524334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0cc0c7970 0x7fa0cc0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 >> 172.21.15.27:0/562524334 conn(0x7fa0cc06b6f0 msgr2=0x7fa0cc151c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:07.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.347+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 shutdown_connections 2024-03-28T19:26:07.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.348+0000 7fa0d2000640 1 -- 172.21.15.27:0/562524334 wait complete. 2024-03-28T19:26:07.370 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 222, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 222, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6740": { "addr": "172.21.15.155:6835/1734256169", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1734256169, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1734256169, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6740, "incarnation": 222, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 92, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:26:05.456779+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6740 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 222, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6467": { "addr": "172.21.15.155:6834/3334295790", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3334295790, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3334295790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6467, "incarnation": 222, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 92, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:26:05.456780+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6467 } } } ], "standbys": [] } 2024-03-28T19:26:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.423+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f800c5f00 con 0x7f0f800c3690 2024-03-28T19:26:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.425+0000 7f0f77400640 1 -- 172.21.15.27:0/3084090867 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+4667 (secure 0 0 0) 0x7f0f700593e0 con 0x7f0f800c3690 2024-03-28T19:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:07.424 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:05.456779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:05.456780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 222 2024-03-28T19:26:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f6006bc50 msgr2=0x7f0f6006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f6006bc50 0x7f0f6006e110 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c036000 tx=0x7f0f6c03d040 comp rx=0 tx=0).stop 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 msgr2=0x7f0f8014bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f8014bfe0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000d830 tx=0x7f0f7000dcf0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 shutdown_connections 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0f6006bc50 0x7f0f6006e110 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f80150c70 0x7f0f8006dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f800c6660 0x7f0f801e1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 --2- 172.21.15.27:0/3084090867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f800c3690 0x7f0f8014bfe0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 >> 172.21.15.27:0/3084090867 conn(0x7f0f8006b9d0 msgr2=0x7f0f800c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 shutdown_connections 2024-03-28T19:26:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:07.427+0000 7f0f85e00640 1 -- 172.21.15.27:0/3084090867 wait complete. 2024-03-28T19:26:07.448 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 222, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 222, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6740": { "addr": "172.21.15.155:6835/1734256169", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1734256169, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1734256169, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6740, "incarnation": 222, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 92, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:26:05.456779+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6740 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 222, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6467": { "addr": "172.21.15.155:6834/3334295790", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3334295790, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 3334295790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6467, "incarnation": 222, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 92, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:26:05.456780+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6467 } } } ], "standbys": [] } 2024-03-28T19:26:09.372 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:09.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.587+0000 7fcc17000640 1 Processor -- start 2024-03-28T19:26:09.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.587+0000 7fcc17000640 1 -- start start 2024-03-28T19:26:09.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.588+0000 7fcc17000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcc100c4dc0 0x7fcc10070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.588+0000 7fcc17000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc10070a80 0x7fcc1015ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.588+0000 7fcc17000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100c52e0 con 0x7fcc10152bd0 2024-03-28T19:26:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.588+0000 7fcc17000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc101557e0 con 0x7fcc10070a80 2024-03-28T19:26:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.588+0000 7fcc17000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc10155ac0 con 0x7fcc100c4dc0 2024-03-28T19:26:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc15000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc10070a80 0x7fcc1015ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc15000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc10070a80 0x7fcc1015ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36928/0 (socket says 172.21.15.27:36928) 2024-03-28T19:26:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc15000640 1 -- 172.21.15.27:0/1164238449 learned_addr learned my addr 172.21.15.27:0/1164238449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:09.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc0f400640 1 --2- 172.21.15.27:0/1164238449 >> v2:172.21.15.155:3300/0 conn(0x7fcc100c4dc0 0x7fcc10070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc15000640 1 -- 172.21.15.27:0/1164238449 >> v2:172.21.15.155:3300/0 conn(0x7fcc100c4dc0 msgr2=0x7fcc10070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc15000640 1 --2- 172.21.15.27:0/1164238449 >> v2:172.21.15.155:3300/0 conn(0x7fcc100c4dc0 0x7fcc10070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc15000640 1 -- 172.21.15.27:0/1164238449 >> v1:172.21.15.155:6789/0 conn(0x7fcc10152bd0 legacy=0x7fcc1006eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc15000640 1 -- 172.21.15.27:0/1164238449 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10155da0 con 0x7fcc10070a80 2024-03-28T19:26:09.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.589+0000 7fcc0f400640 1 --2- 172.21.15.27:0/1164238449 >> v2:172.21.15.155:3300/0 conn(0x7fcc100c4dc0 0x7fcc10070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:26:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.590+0000 7fcc15000640 1 --2- 172.21.15.27:0/1164238449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc1015ffc0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fcc100c9e10 tx=0x7fcc0802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5a88cce42cd412 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc0ea00640 1 -- 172.21.15.27:0/1164238449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0802cb10 con 0x7fcc10070a80 2024-03-28T19:26:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc0ea00640 1 -- 172.21.15.27:0/1164238449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0802ccb0 con 0x7fcc10070a80 2024-03-28T19:26:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc0ea00640 1 -- 172.21.15.27:0/1164238449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0802ce60 con 0x7fcc10070a80 2024-03-28T19:26:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc17000640 1 -- 172.21.15.27:0/1164238449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 msgr2=0x7fcc1015ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc17000640 1 --2- 172.21.15.27:0/1164238449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc1015ffc0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fcc100c9e10 tx=0x7fcc0802bf70 comp rx=0 tx=0).stop 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc17000640 1 -- 172.21.15.27:0/1164238449 shutdown_connections 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc17000640 1 --2- 172.21.15.27:0/1164238449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc1015ffc0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc17000640 1 --2- 172.21.15.27:0/1164238449 >> v2:172.21.15.155:3300/0 conn(0x7fcc100c4dc0 0x7fcc10070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.591+0000 7fcc17000640 1 -- 172.21.15.27:0/1164238449 >> 172.21.15.27:0/1164238449 conn(0x7fcc10066dd0 msgr2=0x7fcc100671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.592+0000 7fcc17000640 1 -- 172.21.15.27:0/1164238449 shutdown_connections 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.592+0000 7fcc17000640 1 -- 172.21.15.27:0/1164238449 wait complete. 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.593+0000 7fcc17000640 1 Processor -- start 2024-03-28T19:26:09.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.593+0000 7fcc17000640 1 -- start start 2024-03-28T19:26:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc17000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc101ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc17000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc100c4dc0 0x7fcc101ec5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc17000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc10152bd0 0x7fcc101ecb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc17000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100c6c70 con 0x7fcc10070a80 2024-03-28T19:26:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc17000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10152fb0 con 0x7fcc100c4dc0 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc17000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100c6df0 con 0x7fcc10152bd0 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc0fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc101ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc0fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc101ee8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36938/0 (socket says 172.21.15.27:36938) 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc0f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc100c4dc0 0x7fcc101ec5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.594+0000 7fcc0fe00640 1 -- 172.21.15.27:0/930693665 learned_addr learned my addr 172.21.15.27:0/930693665 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.595+0000 7fcc15000640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc10152bd0 0x7fcc101ecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.595+0000 7fcc0fe00640 1 -- 172.21.15.27:0/930693665 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc10152bd0 msgr2=0x7fcc101ecb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.595+0000 7fcc0fe00640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc10152bd0 0x7fcc101ecb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.595+0000 7fcc0fe00640 1 -- 172.21.15.27:0/930693665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc100c4dc0 msgr2=0x7fcc101ec5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.595+0000 7fcc0fe00640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc100c4dc0 0x7fcc101ec5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.595+0000 7fcc0fe00640 1 -- 172.21.15.27:0/930693665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc101ef190 con 0x7fcc10070a80 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.595+0000 7fcc0f400640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc100c4dc0 0x7fcc101ec5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.595+0000 7fcc0fe00640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc101ee8c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c430 tx=0x7fcc0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.596+0000 7fcc07e00640 1 -- 172.21.15.27:0/930693665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc000052d0 con 0x7fcc10070a80 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.596+0000 7fcc07e00640 1 -- 172.21.15.27:0/930693665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00005490 con 0x7fcc10070a80 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.596+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc101ef470 con 0x7fcc10070a80 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.596+0000 7fcc07e00640 1 -- 172.21.15.27:0/930693665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00007260 con 0x7fcc10070a80 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.596+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc101e2610 con 0x7fcc10070a80 2024-03-28T19:26:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.597+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1004f400 con 0x7fcc10070a80 2024-03-28T19:26:09.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.598+0000 7fcc07e00640 1 -- 172.21.15.27:0/930693665 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcc00007400 con 0x7fcc10070a80 2024-03-28T19:26:09.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.599+0000 7fcc07e00640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.600+0000 7fcc07e00640 1 -- 172.21.15.27:0/930693665 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7fcc0008b060 con 0x7fcc10070a80 2024-03-28T19:26:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.600+0000 7fcc0f400640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.601+0000 7fcc0f400640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcc100c9bd0 tx=0x7fcbfc008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:09.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.604+0000 7fcc07e00640 1 -- 172.21.15.27:0/930693665 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcc00058370 con 0x7fcc10070a80 2024-03-28T19:26:09.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.643+0000 7f6a9ae00640 1 Processor -- start 2024-03-28T19:26:09.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.644+0000 7f6a9ae00640 1 -- start start 2024-03-28T19:26:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a9ae00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a940c60e0 0x7f6a9406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a9ae00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6a940c3ea0 0x7f6a9406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a9ae00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a940c1600 con 0x7f6a94072df0 2024-03-28T19:26:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a9ae00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a940c23b0 con 0x7f6a940c3ea0 2024-03-28T19:26:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a9ae00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940c18e0 con 0x7f6a940c60e0 2024-03-28T19:26:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a8be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6a940c3ea0 0x7f6a9406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a8be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6a940c3ea0 0x7f6a9406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36946/0 (socket says 172.21.15.27:36946) 2024-03-28T19:26:09.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a8be00640 1 -- 172.21.15.27:0/3812198055 learned_addr learned my addr 172.21.15.27:0/3812198055 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:09.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.645+0000 7f6a93e00640 1 --2- 172.21.15.27:0/3812198055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a940c60e0 0x7f6a9406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.646+0000 7f6a93e00640 1 -- 172.21.15.27:0/3812198055 >> v2:172.21.15.27:3300/0 conn(0x7f6a940c3ea0 msgr2=0x7f6a9406ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:09.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.646+0000 7f6a93e00640 1 --2- 172.21.15.27:0/3812198055 >> v2:172.21.15.27:3300/0 conn(0x7f6a940c3ea0 0x7f6a9406ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.646+0000 7f6a93e00640 1 -- 172.21.15.27:0/3812198055 >> v1:172.21.15.155:6789/0 conn(0x7f6a94072df0 legacy=0x7f6a94158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.646+0000 7f6a93e00640 1 -- 172.21.15.27:0/3812198055 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a940c1bc0 con 0x7f6a940c60e0 2024-03-28T19:26:09.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.646+0000 7f6a8be00640 1 --2- 172.21.15.27:0/3812198055 >> v2:172.21.15.27:3300/0 conn(0x7f6a940c3ea0 0x7f6a9406ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.647+0000 7f6a93e00640 1 --2- 172.21.15.27:0/3812198055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a940c60e0 0x7f6a9406e6f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6a840097a0 tx=0x7f6a8402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55a0b9ceff8c0144 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.647+0000 7f6a93400640 1 -- 172.21.15.27:0/3812198055 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84039070 con 0x7f6a940c60e0 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.647+0000 7f6a93400640 1 -- 172.21.15.27:0/3812198055 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a840316f0 con 0x7f6a940c60e0 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.647+0000 7f6a93400640 1 -- 172.21.15.27:0/3812198055 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84031ac0 con 0x7f6a940c60e0 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.648+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3812198055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a940c60e0 msgr2=0x7f6a9406e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.648+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3812198055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a940c60e0 0x7f6a9406e6f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6a840097a0 tx=0x7f6a8402b790 comp rx=0 tx=0).stop 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.648+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3812198055 shutdown_connections 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.648+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3812198055 >> v2:172.21.15.27:3300/0 conn(0x7f6a940c3ea0 0x7f6a9406ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.648+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3812198055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a940c60e0 0x7f6a9406e6f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.648+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3812198055 >> 172.21.15.27:0/3812198055 conn(0x7f6a9406b9d0 msgr2=0x7f6a940c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:09.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.648+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3812198055 shutdown_connections 2024-03-28T19:26:09.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.648+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3812198055 wait complete. 2024-03-28T19:26:09.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.650+0000 7f6a9ae00640 1 Processor -- start 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.650+0000 7f6a9ae00640 1 -- start start 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a9ae00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a94072df0 0x7f6a9409a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a9ae00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a940c3ea0 0x7f6a94071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a9ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a940c60e0 0x7f6a940aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a9ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940c6640 con 0x7f6a940c60e0 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a9ae00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940c64c0 con 0x7f6a940c3ea0 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a9ae00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940c4f00 con 0x7f6a94072df0 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a98e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a940c60e0 0x7f6a940aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a98e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a940c60e0 0x7f6a940aa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36954/0 (socket says 172.21.15.27:36954) 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a98e00640 1 -- 172.21.15.27:0/3381667029 learned_addr learned my addr 172.21.15.27:0/3381667029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a93e00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a94072df0 0x7f6a9409a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.651+0000 7f6a8be00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a940c3ea0 0x7f6a94071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a98e00640 1 -- 172.21.15.27:0/3381667029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a94072df0 msgr2=0x7f6a9409a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a98e00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a94072df0 0x7f6a9409a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a98e00640 1 -- 172.21.15.27:0/3381667029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a940c3ea0 msgr2=0x7f6a94071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a98e00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a940c3ea0 0x7f6a94071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a98e00640 1 -- 172.21.15.27:0/3381667029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a940abf40 con 0x7f6a940c60e0 2024-03-28T19:26:09.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a98e00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a940c60e0 0x7f6a940aa5b0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00c760 tx=0x7f6a8c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:09.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a90c00640 1 -- 172.21.15.27:0/3381667029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c003b70 con 0x7f6a940c60e0 2024-03-28T19:26:09.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a90c00640 1 -- 172.21.15.27:0/3381667029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8c003d10 con 0x7f6a940c60e0 2024-03-28T19:26:09.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a90c00640 1 -- 172.21.15.27:0/3381667029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c00aa90 con 0x7f6a940c60e0 2024-03-28T19:26:09.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a940a9b10 con 0x7f6a940c60e0 2024-03-28T19:26:09.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.652+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a940a9db0 con 0x7f6a940c60e0 2024-03-28T19:26:09.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.653+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9404f400 con 0x7f6a940c60e0 2024-03-28T19:26:09.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.655+0000 7f6a90c00640 1 -- 172.21.15.27:0/3381667029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6a8c00ac30 con 0x7f6a940c60e0 2024-03-28T19:26:09.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.655+0000 7f6a90c00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a7006bbd0 0x7f6a7006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:09.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.656+0000 7f6a90c00640 1 -- 172.21.15.27:0/3381667029 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f6a8c08b980 con 0x7f6a940c60e0 2024-03-28T19:26:09.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.656+0000 7f6a93e00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a7006bbd0 0x7f6a7006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:09.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.657+0000 7f6a93e00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a7006bbd0 0x7f6a7006e090 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6a84036000 tx=0x7f6a84008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.665+0000 7f6a90c00640 1 -- 172.21.15.27:0/3381667029 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6a8c058c90 con 0x7f6a940c60e0 2024-03-28T19:26:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.859+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc100c2860 con 0x7fcc10070a80 2024-03-28T19:26:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.860+0000 7fcc07e00640 1 -- 172.21.15.27:0/930693665 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+4667 (secure 0 0 0) 0x7fcc0005c410 con 0x7fcc10070a80 2024-03-28T19:26:09.860 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:09.860 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:05.456779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:05.456780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 222 2024-03-28T19:26:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.863+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 msgr2=0x7fcbf006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.863+0000 7fcc17000640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcc100c9bd0 tx=0x7fcbfc008040 comp rx=0 tx=0).stop 2024-03-28T19:26:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.863+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 msgr2=0x7fcc101ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.863+0000 7fcc17000640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc101ee8c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c430 tx=0x7fcc0000c8f0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.864+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 shutdown_connections 2024-03-28T19:26:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.864+0000 7fcc17000640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcbf006bbd0 0x7fcbf006e090 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.864+0000 7fcc17000640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc10152bd0 0x7fcc101ecb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.864+0000 7fcc17000640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc100c4dc0 0x7fcc101ec5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.864+0000 7fcc17000640 1 --2- 172.21.15.27:0/930693665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc10070a80 0x7fcc101ee8c0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.864+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 >> 172.21.15.27:0/930693665 conn(0x7fcc10066dd0 msgr2=0x7fcc1006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.864+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 shutdown_connections 2024-03-28T19:26:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.864+0000 7fcc17000640 1 -- 172.21.15.27:0/930693665 wait complete. 2024-03-28T19:26:09.881 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:26:09.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.914+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a940c2fd0 con 0x7f6a940c60e0 2024-03-28T19:26:09.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.915+0000 7f6a90c00640 1 -- 172.21.15.27:0/3381667029 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+4667 (secure 0 0 0) 0x7f6a8c05cd30 con 0x7f6a940c60e0 2024-03-28T19:26:09.915 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:09.915 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:05.456779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:05.456780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:09.915 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 222 2024-03-28T19:26:09.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a7006bbd0 msgr2=0x7f6a7006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a7006bbd0 0x7f6a7006e090 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6a84036000 tx=0x7f6a84008030 comp rx=0 tx=0).stop 2024-03-28T19:26:09.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a940c60e0 msgr2=0x7f6a940aa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:09.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a940c60e0 0x7f6a940aa5b0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00c760 tx=0x7f6a8c00cc20 comp rx=0 tx=0).stop 2024-03-28T19:26:09.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 shutdown_connections 2024-03-28T19:26:09.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6a7006bbd0 0x7f6a7006e090 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a940c60e0 0x7f6a940aa5b0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a940c3ea0 0x7f6a94071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 --2- 172.21.15.27:0/3381667029 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6a94072df0 0x7f6a9409a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:09.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 >> 172.21.15.27:0/3381667029 conn(0x7f6a9406b9d0 msgr2=0x7f6a940c3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:09.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 shutdown_connections 2024-03-28T19:26:09.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:09.918+0000 7f6a9ae00640 1 -- 172.21.15.27:0/3381667029 wait complete. 2024-03-28T19:26:09.934 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:26:11.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:11.935 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:12.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.093+0000 7f3c2fc00640 1 Processor -- start 2024-03-28T19:26:12.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.093+0000 7f3c2fc00640 1 -- start start 2024-03-28T19:26:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.094+0000 7f3c2fc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c280c7970 0x7f3c280c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.094+0000 7f3c2fc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c28155730 0x7f3c28155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.095+0000 7f3c2fc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c280c7310 con 0x7f3c28152f80 2024-03-28T19:26:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.095+0000 7f3c2fc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c28158d40 con 0x7f3c28155730 2024-03-28T19:26:12.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.095+0000 7f3c2fc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28159020 con 0x7f3c280c7970 2024-03-28T19:26:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.095+0000 7f3c2dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c28155730 0x7f3c28155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.095+0000 7f3c2dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c28155730 0x7f3c28155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36964/0 (socket says 172.21.15.27:36964) 2024-03-28T19:26:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.095+0000 7f3c2dc00640 1 -- 172.21.15.27:0/3196095598 learned_addr learned my addr 172.21.15.27:0/3196095598 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.095+0000 7f3c27e00640 1 --2- 172.21.15.27:0/3196095598 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c280c7970 0x7f3c280c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.096+0000 7f3c2dc00640 1 -- 172.21.15.27:0/3196095598 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c280c7970 msgr2=0x7f3c280c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.096+0000 7f3c2dc00640 1 --2- 172.21.15.27:0/3196095598 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c280c7970 0x7f3c280c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.096+0000 7f3c2dc00640 1 -- 172.21.15.27:0/3196095598 >> v1:172.21.15.27:6789/0 conn(0x7f3c28152f80 legacy=0x7f3c280c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.096+0000 7f3c2dc00640 1 -- 172.21.15.27:0/3196095598 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c28159300 con 0x7f3c28155730 2024-03-28T19:26:12.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.097+0000 7f3c2dc00640 1 --2- 172.21.15.27:0/3196095598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c28155730 0x7f3c28155bc0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f3c20009e50 tx=0x7f3c2002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2ac3f67aa72cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:12.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.098+0000 7f3c27400640 1 -- 172.21.15.27:0/3196095598 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2002ca80 con 0x7f3c28155730 2024-03-28T19:26:12.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.098+0000 7f3c27400640 1 -- 172.21.15.27:0/3196095598 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2002cc20 con 0x7f3c28155730 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.098+0000 7f3c27400640 1 -- 172.21.15.27:0/3196095598 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c200343f0 con 0x7f3c28155730 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.099+0000 7f3c2fc00640 1 -- 172.21.15.27:0/3196095598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c28155730 msgr2=0x7f3c28155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.099+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/3196095598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c28155730 0x7f3c28155bc0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f3c20009e50 tx=0x7f3c2002bee0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.099+0000 7f3c2fc00640 1 -- 172.21.15.27:0/3196095598 shutdown_connections 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.099+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/3196095598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c28155730 0x7f3c28155bc0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.099+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/3196095598 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c280c7970 0x7f3c280c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.099+0000 7f3c2fc00640 1 -- 172.21.15.27:0/3196095598 >> 172.21.15.27:0/3196095598 conn(0x7f3c2806b6f0 msgr2=0x7f3c280caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.099+0000 7f3c2fc00640 1 -- 172.21.15.27:0/3196095598 shutdown_connections 2024-03-28T19:26:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.099+0000 7f3c2fc00640 1 -- 172.21.15.27:0/3196095598 wait complete. 2024-03-28T19:26:12.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.102+0000 7f3c2fc00640 1 Processor -- start 2024-03-28T19:26:12.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.102+0000 7f3c2fc00640 1 -- start start 2024-03-28T19:26:12.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2fc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c280c7970 0x7f3c281eb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2fc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c28152f80 0x7f3c2814ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2fc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c28155730 0x7f3c280c26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2fc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280cbca0 con 0x7f3c280c7970 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2fc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28159b00 con 0x7f3c28155730 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2fc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c281eb6d0 con 0x7f3c28152f80 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c280c7970 0x7f3c281eb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c280c7970 0x7f3c281eb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36976/0 (socket says 172.21.15.27:36976) 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.104+0000 7f3c2d200640 1 -- 172.21.15.27:0/151389249 learned_addr learned my addr 172.21.15.27:0/151389249 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c27e00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c28152f80 0x7f3c2814ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c2d200640 1 -- 172.21.15.27:0/151389249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c28152f80 msgr2=0x7f3c2814ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c2d200640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c28152f80 0x7f3c2814ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c2d200640 1 -- 172.21.15.27:0/151389249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c28155730 msgr2=0x7f3c280c26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c2d200640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c28155730 0x7f3c280c26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c2d200640 1 -- 172.21.15.27:0/151389249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c281eb900 con 0x7f3c280c7970 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c2d200640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c280c7970 0x7f3c281eb190 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c3c0 tx=0x7f3c1c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c24c00640 1 -- 172.21.15.27:0/151389249 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c0052d0 con 0x7f3c280c7970 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c24c00640 1 -- 172.21.15.27:0/151389249 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c005490 con 0x7f3c280c7970 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c281ebbe0 con 0x7f3c280c7970 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.105+0000 7f3c24c00640 1 -- 172.21.15.27:0/151389249 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c007260 con 0x7f3c280c7970 2024-03-28T19:26:12.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.106+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c280c21d0 con 0x7f3c280c7970 2024-03-28T19:26:12.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.107+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2804f400 con 0x7f3c280c7970 2024-03-28T19:26:12.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.109+0000 7f3c24c00640 1 -- 172.21.15.27:0/151389249 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3c1c007400 con 0x7f3c280c7970 2024-03-28T19:26:12.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.110+0000 7f3c24c00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c0806bbd0 0x7f3c0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.110+0000 7f3c24c00640 1 -- 172.21.15.27:0/151389249 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f3c1c08b050 con 0x7f3c280c7970 2024-03-28T19:26:12.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.110+0000 7f3c27e00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c0806bbd0 0x7f3c0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.111+0000 7f3c27e00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c0806bbd0 0x7f3c0806e090 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3c280c4be0 tx=0x7f3c14008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:12.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.119+0000 7f3c24c00640 1 -- 172.21.15.27:0/151389249 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3c1c058a10 con 0x7f3c280c7970 2024-03-28T19:26:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.146+0000 7f9b9f000640 1 Processor -- start 2024-03-28T19:26:12.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.146+0000 7f9b9f000640 1 -- start start 2024-03-28T19:26:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.147+0000 7f9b9f000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b98150ae0 0x7f9b980a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.147+0000 7f9b9f000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980a61a0 0x7f9b980c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.147+0000 7f9b9f000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b98150ec0 con 0x7f9b980c6ee0 2024-03-28T19:26:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.147+0000 7f9b9f000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9806dc70 con 0x7f9b98150ae0 2024-03-28T19:26:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.147+0000 7f9b9f000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b98153550 con 0x7f9b980a61a0 2024-03-28T19:26:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b9dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b98150ae0 0x7f9b980a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b9dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b98150ae0 0x7f9b980a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36990/0 (socket says 172.21.15.27:36990) 2024-03-28T19:26:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b9dc00640 1 -- 172.21.15.27:0/2456472185 learned_addr learned my addr 172.21.15.27:0/2456472185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b9d200640 1 --2- 172.21.15.27:0/2456472185 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980a61a0 0x7f9b980c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b9dc00640 1 -- 172.21.15.27:0/2456472185 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980a61a0 msgr2=0x7f9b980c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b9dc00640 1 --2- 172.21.15.27:0/2456472185 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980a61a0 0x7f9b980c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b9dc00640 1 -- 172.21.15.27:0/2456472185 >> v1:172.21.15.27:6789/0 conn(0x7f9b980c6ee0 legacy=0x7f9b98153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b9dc00640 1 -- 172.21.15.27:0/2456472185 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9806df10 con 0x7f9b98150ae0 2024-03-28T19:26:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.148+0000 7f9b97e00640 1 -- 172.21.15.27:0/2456472185 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118364381 0 0) 0x7f9b90001490 con 0x7f9b980c6ee0 2024-03-28T19:26:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.149+0000 7f9b9dc00640 1 --2- 172.21.15.27:0/2456472185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b98150ae0 0x7f9b980a5c60 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009870 tx=0x7f9b8c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef053fda47350bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b97e00640 1 -- 172.21.15.27:0/2456472185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c035020 con 0x7f9b98150ae0 2024-03-28T19:26:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b97e00640 1 -- 172.21.15.27:0/2456472185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c031640 con 0x7f9b98150ae0 2024-03-28T19:26:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b97e00640 1 -- 172.21.15.27:0/2456472185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c02d040 con 0x7f9b98150ae0 2024-03-28T19:26:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b9f000640 1 -- 172.21.15.27:0/2456472185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b98150ae0 msgr2=0x7f9b980a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b9f000640 1 --2- 172.21.15.27:0/2456472185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b98150ae0 0x7f9b980a5c60 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009870 tx=0x7f9b8c02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b9f000640 1 -- 172.21.15.27:0/2456472185 shutdown_connections 2024-03-28T19:26:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b9f000640 1 --2- 172.21.15.27:0/2456472185 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980a61a0 0x7f9b980c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b9f000640 1 --2- 172.21.15.27:0/2456472185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b98150ae0 0x7f9b980a5c60 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b9f000640 1 -- 172.21.15.27:0/2456472185 >> 172.21.15.27:0/2456472185 conn(0x7f9b9806b0c0 msgr2=0x7f9b9806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.150+0000 7f9b9f000640 1 -- 172.21.15.27:0/2456472185 shutdown_connections 2024-03-28T19:26:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.151+0000 7f9b9f000640 1 -- 172.21.15.27:0/2456472185 wait complete. 2024-03-28T19:26:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.152+0000 7f9b9f000640 1 Processor -- start 2024-03-28T19:26:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.152+0000 7f9b9f000640 1 -- start start 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b980a61a0 0x7f9b980bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9f000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980c6ee0 0x7f9b98072200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9f000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b98150ae0 0x7f9b98148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9806e7b0 con 0x7f9b980a61a0 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9f000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9806e4d0 con 0x7f9b98150ae0 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9f000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b981569d0 con 0x7f9b980c6ee0 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b980a61a0 0x7f9b980bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b980a61a0 0x7f9b980bff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36998/0 (socket says 172.21.15.27:36998) 2024-03-28T19:26:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.153+0000 7f9b9dc00640 1 -- 172.21.15.27:0/1070313851 learned_addr learned my addr 172.21.15.27:0/1070313851 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b9d200640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980c6ee0 0x7f9b98072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b9e600640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b98150ae0 0x7f9b98148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b9dc00640 1 -- 172.21.15.27:0/1070313851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980c6ee0 msgr2=0x7f9b98072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b9dc00640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980c6ee0 0x7f9b98072200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b9dc00640 1 -- 172.21.15.27:0/1070313851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b98150ae0 msgr2=0x7f9b98148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b9dc00640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b98150ae0 0x7f9b98148a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b9dc00640 1 -- 172.21.15.27:0/1070313851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b980708a0 con 0x7f9b980a61a0 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b9dc00640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b980a61a0 0x7f9b980bff00 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c000c00 tx=0x7f9b8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b95600640 1 -- 172.21.15.27:0/1070313851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c040070 con 0x7f9b980a61a0 2024-03-28T19:26:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.154+0000 7f9b95600640 1 -- 172.21.15.27:0/1070313851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c0399f0 con 0x7f9b980a61a0 2024-03-28T19:26:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.155+0000 7f9b95600640 1 -- 172.21.15.27:0/1070313851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c035040 con 0x7f9b980a61a0 2024-03-28T19:26:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.155+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b981ef220 con 0x7f9b980a61a0 2024-03-28T19:26:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.155+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b981498c0 con 0x7f9b980a61a0 2024-03-28T19:26:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.156+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b98069640 con 0x7f9b980a61a0 2024-03-28T19:26:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.157+0000 7f9b95600640 1 -- 172.21.15.27:0/1070313851 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9b8c04b060 con 0x7f9b980a61a0 2024-03-28T19:26:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.158+0000 7f9b95600640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b7806bbd0 0x7f9b7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.158+0000 7f9b95600640 1 -- 172.21.15.27:0/1070313851 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f9b8c0ad490 con 0x7f9b980a61a0 2024-03-28T19:26:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.158+0000 7f9b9d200640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b7806bbd0 0x7f9b7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.159+0000 7f9b9d200640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b7806bbd0 0x7f9b7806e090 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9b98153e80 tx=0x7f9b84008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.162+0000 7f9b95600640 1 -- 172.21.15.27:0/1070313851 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9b8c07ad60 con 0x7f9b980a61a0 2024-03-28T19:26:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.403+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c280cba80 con 0x7f3c280c7970 2024-03-28T19:26:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.404+0000 7f3c24c00640 1 -- 172.21.15.27:0/151389249 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+4667 (secure 0 0 0) 0x7f3c1c058310 con 0x7f3c280c7970 2024-03-28T19:26:12.403 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:12.403 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:05.456779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:05.456780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 222 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c0806bbd0 msgr2=0x7f3c0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c0806bbd0 0x7f3c0806e090 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3c280c4be0 tx=0x7f3c14008040 comp rx=0 tx=0).stop 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c280c7970 msgr2=0x7f3c281eb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c280c7970 0x7f3c281eb190 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c3c0 tx=0x7f3c1c00c880 comp rx=0 tx=0).stop 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 shutdown_connections 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3c0806bbd0 0x7f3c0806e090 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c28155730 0x7f3c280c26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3c28152f80 0x7f3c2814ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 --2- 172.21.15.27:0/151389249 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c280c7970 0x7f3c281eb190 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 >> 172.21.15.27:0/151389249 conn(0x7f3c2806b6f0 msgr2=0x7f3c280c97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:12.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 shutdown_connections 2024-03-28T19:26:12.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.406+0000 7f3c2fc00640 1 -- 172.21.15.27:0/151389249 wait complete. 2024-03-28T19:26:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.420+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b9806da50 con 0x7f9b980a61a0 2024-03-28T19:26:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.420+0000 7f9b95600640 1 -- 172.21.15.27:0/1070313851 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+4667 (secure 0 0 0) 0x7f9b8c07a660 con 0x7f9b980a61a0 2024-03-28T19:26:12.420 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:12.420 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:05.456779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:05.456780+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 222 2024-03-28T19:26:12.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b7806bbd0 msgr2=0x7f9b7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b7806bbd0 0x7f9b7806e090 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9b98153e80 tx=0x7f9b84008040 comp rx=0 tx=0).stop 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b980a61a0 msgr2=0x7f9b980bff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b980a61a0 0x7f9b980bff00 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c000c00 tx=0x7f9b8c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 shutdown_connections 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9b7806bbd0 0x7f9b7806e090 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b98150ae0 0x7f9b98148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b980c6ee0 0x7f9b98072200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 --2- 172.21.15.27:0/1070313851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b980a61a0 0x7f9b980bff00 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 >> 172.21.15.27:0/1070313851 conn(0x7f9b9806b0c0 msgr2=0x7f9b980c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 shutdown_connections 2024-03-28T19:26:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:12.423+0000 7f9b9f000640 1 -- 172.21.15.27:0/1070313851 wait complete. 2024-03-28T19:26:12.425 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:26:12.438 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:26:14.425 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:14.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.636+0000 7f065ce00640 1 Processor -- start 2024-03-28T19:26:14.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.637+0000 7f065ce00640 1 -- start start 2024-03-28T19:26:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f065ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0658150c70 0x7f06580a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f065ce00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f06580c3690 0x7f06580c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f065ce00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 0x7f06581534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f065ce00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06580ab4e0 con 0x7f0658150c70 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f065ce00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06580ac1f0 con 0x7f06580c3690 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f065ce00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06580c3af0 con 0x7f06580c6660 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f0655e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0658150c70 0x7f06580a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f0655e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0658150c70 0x7f06580a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37014/0 (socket says 172.21.15.27:37014) 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.638+0000 7f0655e00640 1 -- 172.21.15.27:0/289516113 learned_addr learned my addr 172.21.15.27:0/289516113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.639+0000 7f0655400640 1 --2- 172.21.15.27:0/289516113 >> v2:172.21.15.155:3300/0 conn(0x7f06580c3690 0x7f06580c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.639+0000 7f0656800640 1 --2- 172.21.15.27:0/289516113 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 0x7f06581534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.639+0000 7f0655e00640 1 -- 172.21.15.27:0/289516113 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 msgr2=0x7f06581534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:14.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.639+0000 7f0655e00640 1 --2- 172.21.15.27:0/289516113 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 0x7f06581534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.639+0000 7f0655e00640 1 -- 172.21.15.27:0/289516113 >> v2:172.21.15.155:3300/0 conn(0x7f06580c3690 msgr2=0x7f06580c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.639+0000 7f0655e00640 1 --2- 172.21.15.27:0/289516113 >> v2:172.21.15.155:3300/0 conn(0x7f06580c3690 0x7f06580c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.639+0000 7f0655e00640 1 -- 172.21.15.27:0/289516113 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06580c3d90 con 0x7f0658150c70 2024-03-28T19:26:14.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.639+0000 7f0655400640 1 --2- 172.21.15.27:0/289516113 >> v2:172.21.15.155:3300/0 conn(0x7f06580c3690 0x7f06580c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:26:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.640+0000 7f0655e00640 1 --2- 172.21.15.27:0/289516113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06580a9cb0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f06580c4180 tx=0x7f064802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1149a32e2dca08ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.640+0000 7f0654a00640 1 -- 172.21.15.27:0/289516113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06480317c0 con 0x7f0658150c70 2024-03-28T19:26:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.640+0000 7f0654a00640 1 -- 172.21.15.27:0/289516113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0648037070 con 0x7f0658150c70 2024-03-28T19:26:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.640+0000 7f0654a00640 1 -- 172.21.15.27:0/289516113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648031b80 con 0x7f0658150c70 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 -- 172.21.15.27:0/289516113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 msgr2=0x7f06580a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 --2- 172.21.15.27:0/289516113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06580a9cb0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f06580c4180 tx=0x7f064802b860 comp rx=0 tx=0).stop 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 -- 172.21.15.27:0/289516113 shutdown_connections 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 --2- 172.21.15.27:0/289516113 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 0x7f06581534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 --2- 172.21.15.27:0/289516113 >> v2:172.21.15.155:3300/0 conn(0x7f06580c3690 0x7f06580c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 --2- 172.21.15.27:0/289516113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06580a9cb0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 -- 172.21.15.27:0/289516113 >> 172.21.15.27:0/289516113 conn(0x7f065806b9d0 msgr2=0x7f06580c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 -- 172.21.15.27:0/289516113 shutdown_connections 2024-03-28T19:26:14.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.641+0000 7f065ce00640 1 -- 172.21.15.27:0/289516113 wait complete. 2024-03-28T19:26:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.643+0000 7f065ce00640 1 Processor -- start 2024-03-28T19:26:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.643+0000 7f065ce00640 1 -- start start 2024-03-28T19:26:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f065ce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06580c3690 0x7f065806d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f065ce00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 0x7f06580c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f065ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06581de750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f065ce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06580abcd0 con 0x7f0658150c70 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f065ce00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06580ab9f0 con 0x7f06580c3690 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f065ce00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06580c1810 con 0x7f06580c6660 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0656800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06581de750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0656800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06581de750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37018/0 (socket says 172.21.15.27:37018) 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0656800640 1 -- 172.21.15.27:0/1682221283 learned_addr learned my addr 172.21.15.27:0/1682221283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0655400640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 0x7f06580c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0655e00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06580c3690 0x7f065806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0656800640 1 -- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 msgr2=0x7f06580c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0656800640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 0x7f06580c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0656800640 1 -- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06580c3690 msgr2=0x7f065806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0656800640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06580c3690 0x7f065806d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0656800640 1 -- 172.21.15.27:0/1682221283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065806e260 con 0x7f0658150c70 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.644+0000 7f0655e00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06580c3690 0x7f065806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.645+0000 7f0656800640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06581de750 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f065000cd70 tx=0x7f065000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.645+0000 7f064e000640 1 -- 172.21.15.27:0/1682221283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650003b90 con 0x7f0658150c70 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.645+0000 7f064e000640 1 -- 172.21.15.27:0/1682221283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0650003d30 con 0x7f0658150c70 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.645+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065806e540 con 0x7f0658150c70 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.645+0000 7f064e000640 1 -- 172.21.15.27:0/1682221283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650004240 con 0x7f0658150c70 2024-03-28T19:26:14.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.645+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065806eb00 con 0x7f0658150c70 2024-03-28T19:26:14.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.647+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065804f400 con 0x7f0658150c70 2024-03-28T19:26:14.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.647+0000 7f064e000640 1 -- 172.21.15.27:0/1682221283 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f06500043e0 con 0x7f0658150c70 2024-03-28T19:26:14.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.648+0000 7f064e000640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f063806bbd0 0x7f063806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.648+0000 7f064e000640 1 -- 172.21.15.27:0/1682221283 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f065008c6b0 con 0x7f0658150c70 2024-03-28T19:26:14.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.649+0000 7f0655e00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f063806bbd0 0x7f063806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.650+0000 7f0655e00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f063806bbd0 0x7f063806e090 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0648036000 tx=0x7f0648008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:14.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.653+0000 7f064e000640 1 -- 172.21.15.27:0/1682221283 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f065005a010 con 0x7f0658150c70 2024-03-28T19:26:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.689+0000 7f0634a00640 1 Processor -- start 2024-03-28T19:26:14.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.689+0000 7f0634a00640 1 -- start start 2024-03-28T19:26:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.690+0000 7f0634a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0630152f80 0x7f06300c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.690+0000 7f0634a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06300c7310 con 0x7f06300c7970 2024-03-28T19:26:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.690+0000 7f0634a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0630158d40 con 0x7f06300c7df0 2024-03-28T19:26:14.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.690+0000 7f0634a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0630159020 con 0x7f0630152f80 2024-03-28T19:26:14.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.690+0000 7f062de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0630152f80 0x7f06300c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.691+0000 7f062de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0630152f80 0x7f06300c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37032/0 (socket says 172.21.15.27:37032) 2024-03-28T19:26:14.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.691+0000 7f062d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f06300c7970 0x7f0630155730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55266/0 (socket says 172.21.15.27:55266) 2024-03-28T19:26:14.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.691+0000 7f062de00640 1 -- 172.21.15.27:0/353477022 learned_addr learned my addr 172.21.15.27:0/353477022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.691+0000 7f062de00640 1 -- 172.21.15.27:0/353477022 >> v1:172.21.15.155:6789/0 conn(0x7f06300c7df0 legacy=0x7f063015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.691+0000 7f062de00640 1 -- 172.21.15.27:0/353477022 >> v1:172.21.15.27:6789/0 conn(0x7f06300c7970 legacy=0x7f0630155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.691+0000 7f062ca00640 1 -- 172.21.15.27:0/353477022 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927968035 0 0) 0x7f06300c7310 con 0x7f06300c7970 2024-03-28T19:26:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.691+0000 7f062de00640 1 -- 172.21.15.27:0/353477022 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0630159300 con 0x7f0630152f80 2024-03-28T19:26:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.692+0000 7f062de00640 1 --2- 172.21.15.27:0/353477022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0630152f80 0x7f06300c58e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f0620009870 tx=0x7f062002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb05b14e7977bd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.692+0000 7f062ca00640 1 -- 172.21.15.27:0/353477022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0620035020 con 0x7f0630152f80 2024-03-28T19:26:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f062ca00640 1 -- 172.21.15.27:0/353477022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0620031870 con 0x7f0630152f80 2024-03-28T19:26:14.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f062ca00640 1 -- 172.21.15.27:0/353477022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0620031c40 con 0x7f0630152f80 2024-03-28T19:26:14.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f0634a00640 1 -- 172.21.15.27:0/353477022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0630152f80 msgr2=0x7f06300c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:14.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f0634a00640 1 --2- 172.21.15.27:0/353477022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0630152f80 0x7f06300c58e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f0620009870 tx=0x7f062002b910 comp rx=0 tx=0).stop 2024-03-28T19:26:14.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f0634a00640 1 -- 172.21.15.27:0/353477022 shutdown_connections 2024-03-28T19:26:14.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f0634a00640 1 --2- 172.21.15.27:0/353477022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0630152f80 0x7f06300c58e0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f0634a00640 1 -- 172.21.15.27:0/353477022 >> 172.21.15.27:0/353477022 conn(0x7f063006b6f0 msgr2=0x7f06300caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:14.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f0634a00640 1 -- 172.21.15.27:0/353477022 shutdown_connections 2024-03-28T19:26:14.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.693+0000 7f0634a00640 1 -- 172.21.15.27:0/353477022 wait complete. 2024-03-28T19:26:14.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.695+0000 7f0634a00640 1 Processor -- start 2024-03-28T19:26:14.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.695+0000 7f0634a00640 1 -- start start 2024-03-28T19:26:14.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f0634a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06300c7970 0x7f06301ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f0634a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06300c7df0 0x7f0630149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f0634a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0630152f80 0x7f0630149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f0634a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0630159b00 con 0x7f06300c7df0 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f0634a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06300c5e20 con 0x7f06300c7970 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f0634a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06300cbca0 con 0x7f0630152f80 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f062d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06300c7df0 0x7f0630149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f062d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06300c7df0 0x7f0630149630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37038/0 (socket says 172.21.15.27:37038) 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.696+0000 7f062d400640 1 -- 172.21.15.27:0/2880302845 learned_addr learned my addr 172.21.15.27:0/2880302845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:14.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f062de00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06300c7970 0x7f06301ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f062e800640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0630152f80 0x7f0630149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f062d400640 1 -- 172.21.15.27:0/2880302845 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0630152f80 msgr2=0x7f0630149b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f062d400640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0630152f80 0x7f0630149b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f062d400640 1 -- 172.21.15.27:0/2880302845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06300c7970 msgr2=0x7f06301ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f062d400640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06300c7970 0x7f06301ee970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f062d400640 1 -- 172.21.15.27:0/2880302845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06301ef360 con 0x7f06300c7df0 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f062d400640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06300c7df0 0x7f0630149630 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f061c00c500 tx=0x7f061c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f0626000640 1 -- 172.21.15.27:0/2880302845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c018070 con 0x7f06300c7df0 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f0626000640 1 -- 172.21.15.27:0/2880302845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c007580 con 0x7f06300c7df0 2024-03-28T19:26:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.697+0000 7f0626000640 1 -- 172.21.15.27:0/2880302845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c004a20 con 0x7f06300c7df0 2024-03-28T19:26:14.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.698+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06301541b0 con 0x7f06300c7df0 2024-03-28T19:26:14.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.698+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06300c0580 con 0x7f06300c7df0 2024-03-28T19:26:14.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.699+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063004f400 con 0x7f06300c7df0 2024-03-28T19:26:14.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.700+0000 7f0626000640 1 -- 172.21.15.27:0/2880302845 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f061c004bc0 con 0x7f06300c7df0 2024-03-28T19:26:14.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.701+0000 7f0626000640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f061006bc00 0x7f061006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:14.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.701+0000 7f0626000640 1 -- 172.21.15.27:0/2880302845 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f061c08ae90 con 0x7f06300c7df0 2024-03-28T19:26:14.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.701+0000 7f062de00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f061006bc00 0x7f061006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:14.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.702+0000 7f062de00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f061006bc00 0x7f061006e0c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0620036000 tx=0x7f0620008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:14.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.705+0000 7f0626000640 1 -- 172.21.15.27:0/2880302845 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f061c0587f0 con 0x7f06300c7df0 2024-03-28T19:26:14.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.923+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06580c5f00 con 0x7f0658150c70 2024-03-28T19:26:14.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:14.923 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:13.016911+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:13.186111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:reconnect","state_seq":80,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:14.923 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 224 2024-03-28T19:26:14.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.973+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06300cba80 con 0x7f06300c7df0 2024-03-28T19:26:14.973 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:14.973 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:13.016911+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:13.186111+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:reconnect","state_seq":80,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:26:14.973 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 224 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.924+0000 7f064e000640 1 -- 172.21.15.27:0/1682221283 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+4672 (secure 0 0 0) 0x7f0650059910 con 0x7f0658150c70 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f063806bbd0 msgr2=0x7f063806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f063806bbd0 0x7f063806e090 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0648036000 tx=0x7f0648008030 comp rx=0 tx=0).stop 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 msgr2=0x7f06581de750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06581de750 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f065000cd70 tx=0x7f065000a660 comp rx=0 tx=0).stop 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 shutdown_connections 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f063806bbd0 0x7f063806e090 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0658150c70 0x7f06581de750 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06580c6660 0x7f06580c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 --2- 172.21.15.27:0/1682221283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06580c3690 0x7f065806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 >> 172.21.15.27:0/1682221283 conn(0x7f065806b9d0 msgr2=0x7f0658061ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 shutdown_connections 2024-03-28T19:26:15.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.926+0000 7f065ce00640 1 -- 172.21.15.27:0/1682221283 wait complete. 2024-03-28T19:26:15.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.974+0000 7f0626000640 1 -- 172.21.15.27:0/2880302845 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+4672 (secure 0 0 0) 0x7f061c08c020 con 0x7f06300c7df0 2024-03-28T19:26:15.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.976+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f061006bc00 msgr2=0x7f061006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:15.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.976+0000 7f0634a00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f061006bc00 0x7f061006e0c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0620036000 tx=0x7f0620008030 comp rx=0 tx=0).stop 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.976+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06300c7df0 msgr2=0x7f0630149630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.976+0000 7f0634a00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06300c7df0 0x7f0630149630 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f061c00c500 tx=0x7f061c00c9c0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.977+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 shutdown_connections 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.977+0000 7f0634a00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f061006bc00 0x7f061006e0c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.977+0000 7f0634a00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0630152f80 0x7f0630149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.977+0000 7f0634a00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06300c7df0 0x7f0630149630 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.977+0000 7f0634a00640 1 --2- 172.21.15.27:0/2880302845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06300c7970 0x7f06301ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.977+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 >> 172.21.15.27:0/2880302845 conn(0x7f063006b6f0 msgr2=0x7f06301506d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.977+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 shutdown_connections 2024-03-28T19:26:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:14.977+0000 7f0634a00640 1 -- 172.21.15.27:0/2880302845 wait complete. 2024-03-28T19:26:15.052 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:26:15.053 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:26:16.784 INFO:tasks.ceph.mds.b.smithi155.stderr:2024-03-28T19:26:16.784+0000 7f1364000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:26:16.938 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 41% (54/129) Updating files: 42% (55/129) Updating files: 43% (56/129) Updating files: 44% (57/129) Updating files: 45% (59/129) Updating files: 46% (60/129) Updating files: 47% (61/129) Updating files: 48% (62/129) Updating files: 49% (64/129) Updating files: 50% (65/129) Updating files: 51% (66/129) Updating files: 52% (68/129) Updating files: 53% (69/129) Updating files: 54% (70/129) Updating files: 55% (71/129) Updating files: 56% (73/129) Updating files: 57% (74/129) Updating files: 58% (75/129) Updating files: 59% (77/129) Updating files: 60% (78/129) Updating files: 61% (79/129) Updating files: 62% (80/129) Updating files: 63% (82/129) Updating files: 64% (83/129) Updating files: 65% (84/129) Updating files: 66% (86/129) Updating files: 67% (87/129) Updating files: 68% (88/129) Updating files: 69% (90/129) Updating files: 70% (91/129) Updating files: 71% (92/129) Updating files: 72% (93/129) Updating files: 73% (95/129) Updating files: 74% (96/129) Updating files: 75% (97/129) Updating files: 76% (99/129) Updating files: 77% (100/129) Updating files: 78% (101/129) Updating files: 79% (102/129) Updating files: 80% (104/129) Updating files: 81% (105/129) Updating files: 82% (106/129) Updating files: 83% (108/129) Updating files: 84% (109/129) Updating files: 85% (110/129) Updating files: 86% (111/129) Updating files: 87% (113/129) Updating files: 88% (114/129) Updating files: 89% (115/129) Updating files: 90% (117/129) Updating files: 91% (118/129) Updating files: 92% (119/129) Updating files: 93% (120/129) Updating files: 94% (122/129) Updating files: 95% (123/129) Updating files: 96% (124/129) Updating files: 97% (126/129) Updating files: 98% (127/129) Updating files: 99% (128/129) Updating files: 100% (129/129) Updating files: 100% (129/129), done. 2024-03-28T19:26:17.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:17.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:17.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.284+0000 7f6393200640 1 Processor -- start 2024-03-28T19:26:17.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.285+0000 7f6393200640 1 -- start start 2024-03-28T19:26:17.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.286+0000 7f6393200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f63940c58c0 0x7f63940c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.287+0000 7f6393200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6394158bd0 con 0x7f63940c8ed0 2024-03-28T19:26:17.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.287+0000 7f6393200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6394158eb0 con 0x7f63940be5d0 2024-03-28T19:26:17.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.287+0000 7f6393200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6394159190 con 0x7f63940c58c0 2024-03-28T19:26:17.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.287+0000 7f6391e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f63940c58c0 0x7f63940c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.287+0000 7f6391e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f63940c58c0 0x7f63940c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:40026/0 (socket says 172.21.15.27:40026) 2024-03-28T19:26:17.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.287+0000 7f6391e00640 1 -- 172.21.15.27:0/1544194483 learned_addr learned my addr 172.21.15.27:0/1544194483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:17.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.288+0000 7f6391e00640 1 -- 172.21.15.27:0/1544194483 >> v1:172.21.15.155:6789/0 conn(0x7f63940be5d0 legacy=0x7f6394155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.288+0000 7f6391e00640 1 -- 172.21.15.27:0/1544194483 >> v1:172.21.15.27:6789/0 conn(0x7f63940c8ed0 legacy=0x7f63941555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:17.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.288+0000 7f6391e00640 1 -- 172.21.15.27:0/1544194483 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6394159470 con 0x7f63940c58c0 2024-03-28T19:26:17.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.290+0000 7fa055600640 1 Processor -- start 2024-03-28T19:26:17.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.290+0000 7f6391e00640 1 --2- 172.21.15.27:0/1544194483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c58c0 0x7f63940c5ca0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f638c009e50 tx=0x7f638c02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34f5ccf916f16499 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.290+0000 7fa055600640 1 -- start start 2024-03-28T19:26:17.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.290+0000 7f6390a00640 1 -- 172.21.15.27:0/1544194483 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c02c9f0 con 0x7f63940c58c0 2024-03-28T19:26:17.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.290+0000 7f6390a00640 1 -- 172.21.15.27:0/1544194483 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c02cb90 con 0x7f63940c58c0 2024-03-28T19:26:17.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.290+0000 7f6390a00640 1 -- 172.21.15.27:0/1544194483 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c0343f0 con 0x7f63940c58c0 2024-03-28T19:26:17.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.291+0000 7f6393200640 1 -- 172.21.15.27:0/1544194483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c58c0 msgr2=0x7f63940c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.291+0000 7f6393200640 1 --2- 172.21.15.27:0/1544194483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c58c0 0x7f63940c5ca0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f638c009e50 tx=0x7f638c02be50 comp rx=0 tx=0).stop 2024-03-28T19:26:17.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.291+0000 7fa055600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa050153ed0 0x7fa05006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.291+0000 7fa055600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006eb80 0x7fa05015a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.291+0000 7fa055600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05006efb0 con 0x7fa05006e6f0 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.291+0000 7fa055600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0500c6990 con 0x7fa050153ed0 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.291+0000 7fa055600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0500c3830 con 0x7fa05006eb80 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.292+0000 7fa04ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa050153ed0 0x7fa05006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.292+0000 7fa04ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa050153ed0 0x7fa05006ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47092/0 (socket says 172.21.15.27:47092) 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.292+0000 7fa04ea00640 1 -- 172.21.15.27:0/92461967 learned_addr learned my addr 172.21.15.27:0/92461967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.292+0000 7fa04f400640 1 --2- 172.21.15.27:0/92461967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006eb80 0x7fa05015a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.292+0000 7fa04ea00640 1 -- 172.21.15.27:0/92461967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006eb80 msgr2=0x7fa05015a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:17.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.292+0000 7fa04ea00640 1 --2- 172.21.15.27:0/92461967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006eb80 0x7fa05015a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.292+0000 7fa04ea00640 1 -- 172.21.15.27:0/92461967 >> v1:172.21.15.155:6789/0 conn(0x7fa05006e6f0 legacy=0x7fa050159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:17.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.292+0000 7fa04ea00640 1 -- 172.21.15.27:0/92461967 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05006d1f0 con 0x7fa050153ed0 2024-03-28T19:26:17.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.293+0000 7f6393200640 1 -- 172.21.15.27:0/1544194483 shutdown_connections 2024-03-28T19:26:17.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.293+0000 7f6393200640 1 --2- 172.21.15.27:0/1544194483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c58c0 0x7f63940c5ca0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.293+0000 7f6393200640 1 -- 172.21.15.27:0/1544194483 >> 172.21.15.27:0/1544194483 conn(0x7f639406b6f0 msgr2=0x7f63940acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:17.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.293+0000 7f6393200640 1 -- 172.21.15.27:0/1544194483 shutdown_connections 2024-03-28T19:26:17.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.293+0000 7f6393200640 1 -- 172.21.15.27:0/1544194483 wait complete. 2024-03-28T19:26:17.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.294+0000 7fa04ea00640 1 --2- 172.21.15.27:0/92461967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa05006ccb0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fa0500c5ca0 tx=0x7fa04002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c908e0f1a7c1278 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.294+0000 7fa04d600640 1 -- 172.21.15.27:0/92461967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040035020 con 0x7fa050153ed0 2024-03-28T19:26:17.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.294+0000 7fa04d600640 1 -- 172.21.15.27:0/92461967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa040031830 con 0x7fa050153ed0 2024-03-28T19:26:17.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.294+0000 7fa04d600640 1 -- 172.21.15.27:0/92461967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040031c00 con 0x7fa050153ed0 2024-03-28T19:26:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7fa055600640 1 -- 172.21.15.27:0/92461967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 msgr2=0x7fa05006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7fa055600640 1 --2- 172.21.15.27:0/92461967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa05006ccb0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fa0500c5ca0 tx=0x7fa04002b8d0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7fa055600640 1 -- 172.21.15.27:0/92461967 shutdown_connections 2024-03-28T19:26:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7fa055600640 1 --2- 172.21.15.27:0/92461967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006eb80 0x7fa05015a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7fa055600640 1 --2- 172.21.15.27:0/92461967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa05006ccb0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7fa055600640 1 -- 172.21.15.27:0/92461967 >> 172.21.15.27:0/92461967 conn(0x7fa05006b9d0 msgr2=0x7fa0500c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7f6393200640 1 Processor -- start 2024-03-28T19:26:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7fa055600640 1 -- 172.21.15.27:0/92461967 shutdown_connections 2024-03-28T19:26:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7fa055600640 1 -- 172.21.15.27:0/92461967 wait complete. 2024-03-28T19:26:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.295+0000 7f6393200640 1 -- start start 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.296+0000 7f6393200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63940be5d0 0x7f6394152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.296+0000 7f6393200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63940c58c0 0x7f6394152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.296+0000 7f6393200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c8ed0 0x7f63940c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.296+0000 7f6393200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63940b09a0 con 0x7f63940be5d0 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.296+0000 7f6393200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63940b0820 con 0x7f63940c8ed0 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.296+0000 7f6393200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63940ad7d0 con 0x7f63940c58c0 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.297+0000 7f6391e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63940be5d0 0x7f6394152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.297+0000 7f6391e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63940be5d0 0x7f6394152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47100/0 (socket says 172.21.15.27:47100) 2024-03-28T19:26:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.297+0000 7f6391e00640 1 -- 172.21.15.27:0/3660334000 learned_addr learned my addr 172.21.15.27:0/3660334000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.297+0000 7fa055600640 1 Processor -- start 2024-03-28T19:26:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.297+0000 7f6391400640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63940c58c0 0x7f6394152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.297+0000 7f6392800640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c8ed0 0x7f63940c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.297+0000 7fa055600640 1 -- start start 2024-03-28T19:26:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7f6391e00640 1 -- 172.21.15.27:0/3660334000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63940c58c0 msgr2=0x7f6394152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7f6391e00640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63940c58c0 0x7f6394152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7f6391e00640 1 -- 172.21.15.27:0/3660334000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c8ed0 msgr2=0x7f63940c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7f6391e00640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c8ed0 0x7f63940c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7f6391e00640 1 -- 172.21.15.27:0/3660334000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63941ec220 con 0x7f63940be5d0 2024-03-28T19:26:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7f6391e00640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63940be5d0 0x7f6394152230 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f638c009e50 tx=0x7f638c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7fa055600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006e6f0 0x7fa050156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7fa055600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa05006eb80 0x7fa050156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7fa055600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa0500a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7fa055600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0500c3db0 con 0x7fa050153ed0 2024-03-28T19:26:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7fa055600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0500c3ad0 con 0x7fa05006eb80 2024-03-28T19:26:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.298+0000 7fa055600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05015a810 con 0x7fa05006e6f0 2024-03-28T19:26:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7f638a000640 1 -- 172.21.15.27:0/3660334000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c044020 con 0x7f63940be5d0 2024-03-28T19:26:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7f638a000640 1 -- 172.21.15.27:0/3660334000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c040470 con 0x7f63940be5d0 2024-03-28T19:26:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7f638a000640 1 -- 172.21.15.27:0/3660334000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c02c9f0 con 0x7f63940be5d0 2024-03-28T19:26:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa0500a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa0500a6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47110/0 (socket says 172.21.15.27:47110) 2024-03-28T19:26:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04f400640 1 -- 172.21.15.27:0/3313464931 learned_addr learned my addr 172.21.15.27:0/3313464931 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04e000640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa05006eb80 0x7fa050156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6394154dd0 con 0x7f63940be5d0 2024-03-28T19:26:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6394155390 con 0x7f63940be5d0 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04f400640 1 -- 172.21.15.27:0/3313464931 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006e6f0 msgr2=0x7fa050156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04f400640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006e6f0 0x7fa050156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04f400640 1 -- 172.21.15.27:0/3313464931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa05006eb80 msgr2=0x7fa050156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04f400640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa05006eb80 0x7fa050156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.299+0000 7fa04f400640 1 -- 172.21.15.27:0/3313464931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0501572d0 con 0x7fa050153ed0 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.300+0000 7fa04f400640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa0500a6e70 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fa04800efc0 tx=0x7fa04800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.300+0000 7fa046a00640 1 -- 172.21.15.27:0/3313464931 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa048018020 con 0x7fa050153ed0 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.300+0000 7fa046a00640 1 -- 172.21.15.27:0/3313464931 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa048014480 con 0x7fa050153ed0 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.300+0000 7fa046a00640 1 -- 172.21.15.27:0/3313464931 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa048014870 con 0x7fa050153ed0 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.300+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa050159d70 con 0x7fa050153ed0 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.301+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0501550c0 con 0x7fa050153ed0 2024-03-28T19:26:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.302+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa028005290 con 0x7fa050153ed0 2024-03-28T19:26:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.302+0000 7f638a000640 1 -- 172.21.15.27:0/3660334000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f638c047020 con 0x7f63940be5d0 2024-03-28T19:26:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.304+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6394069640 con 0x7f63940be5d0 2024-03-28T19:26:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.304+0000 7f638a000640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f637406bbd0 0x7f637406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.304+0000 7f638a000640 1 -- 172.21.15.27:0/3660334000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f638c0ae560 con 0x7f63940be5d0 2024-03-28T19:26:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.306+0000 7f6391400640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f637406bbd0 0x7f637406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.306+0000 7fa046a00640 1 -- 172.21.15.27:0/3313464931 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa048005d10 con 0x7fa050153ed0 2024-03-28T19:26:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.307+0000 7f6391400640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f637406bbd0 0x7f637406e090 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6394157420 tx=0x7f6380009060 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.307+0000 7fa046a00640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa03006bbd0 0x7fa03006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.308+0000 7fa046a00640 1 -- 172.21.15.27:0/3313464931 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7fa04808cf00 con 0x7fa050153ed0 2024-03-28T19:26:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.308+0000 7fa04ea00640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa03006bbd0 0x7fa03006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.309+0000 7fa04ea00640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa03006bbd0 0x7fa03006e090 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa040036000 tx=0x7fa040030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.312+0000 7fa046a00640 1 -- 172.21.15.27:0/3313464931 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa048022030 con 0x7fa050153ed0 2024-03-28T19:26:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.320+0000 7f638a000640 1 -- 172.21.15.27:0/3660334000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f638c04e030 con 0x7f63940be5d0 2024-03-28T19:26:17.331 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 2f039eeeb74 v0.34 2024-03-28T19:26:17.335 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.34 2024-03-28T19:26:17.362 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:26:17.362 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 35 -eq 48 ']' 2024-03-28T19:26:17.362 INFO:tasks.workunit.client.0.smithi027.stdout:v0.35 2024-03-28T19:26:17.363 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.35 2024-03-28T19:26:17.363 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.35 2024-03-28T19:26:17.363 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.35 2024-03-28T19:26:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.631+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa028005580 con 0x7fa050153ed0 2024-03-28T19:26:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.632+0000 7fa046a00640 1 -- 172.21.15.27:0/3313464931 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+4683 (secure 0 0 0) 0x7fa0501572d0 con 0x7fa050153ed0 2024-03-28T19:26:17.631 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:17.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 227 2024-03-28T19:26:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa03006bbd0 msgr2=0x7fa03006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa03006bbd0 0x7fa03006e090 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa040036000 tx=0x7fa040030000 comp rx=0 tx=0).stop 2024-03-28T19:26:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 msgr2=0x7fa0500a6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa0500a6e70 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fa04800efc0 tx=0x7fa04800c490 comp rx=0 tx=0).stop 2024-03-28T19:26:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 shutdown_connections 2024-03-28T19:26:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa03006bbd0 0x7fa03006e090 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa050153ed0 0x7fa0500a6e70 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa05006eb80 0x7fa050156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 --2- 172.21.15.27:0/3313464931 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa05006e6f0 0x7fa050156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 >> 172.21.15.27:0/3313464931 conn(0x7fa05006b9d0 msgr2=0x7fa0500aa690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 shutdown_connections 2024-03-28T19:26:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.635+0000 7fa055600640 1 -- 172.21.15.27:0/3313464931 wait complete. 2024-03-28T19:26:17.652 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=8511 2024-03-28T19:26:17.652 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.c 2024-03-28T19:26:17.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.665+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6394156580 con 0x7f63940be5d0 2024-03-28T19:26:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.665+0000 7f638a000640 1 -- 172.21.15.27:0/3660334000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+4683 (secure 0 0 0) 0x7f638c07b870 con 0x7f63940be5d0 2024-03-28T19:26:17.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:17.664 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:17.664 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 227 2024-03-28T19:26:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.668+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f637406bbd0 msgr2=0x7f637406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.668+0000 7f6393200640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f637406bbd0 0x7f637406e090 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6394157420 tx=0x7f6380009060 comp rx=0 tx=0).stop 2024-03-28T19:26:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.668+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63940be5d0 msgr2=0x7f6394152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.668+0000 7f6393200640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63940be5d0 0x7f6394152230 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f638c009e50 tx=0x7f638c004a40 comp rx=0 tx=0).stop 2024-03-28T19:26:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.669+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 shutdown_connections 2024-03-28T19:26:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.669+0000 7f6393200640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f637406bbd0 0x7f637406e090 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.669+0000 7f6393200640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63940c8ed0 0x7f63940c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.669+0000 7f6393200640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63940c58c0 0x7f6394152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.669+0000 7f6393200640 1 --2- 172.21.15.27:0/3660334000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63940be5d0 0x7f6394152230 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.669+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 >> 172.21.15.27:0/3660334000 conn(0x7f639406b6f0 msgr2=0x7f63940ae5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:17.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.669+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 shutdown_connections 2024-03-28T19:26:17.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.669+0000 7f6393200640 1 -- 172.21.15.27:0/3660334000 wait complete. 2024-03-28T19:26:17.687 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=8586 2024-03-28T19:26:17.687 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.a 2024-03-28T19:26:17.687 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-03-28T19:26:17.687 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-28T19:26:17.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-28T19:26:17.734 INFO:tasks.ceph.mds.a:Started 2024-03-28T19:26:17.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:17.928 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-03-28T19:26:17.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.967+0000 7fc999400640 1 Processor -- start 2024-03-28T19:26:17.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.968+0000 7fc999400640 1 -- start start 2024-03-28T19:26:17.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.969+0000 7fc999400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9940c8ed0 0x7fc9940c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.969+0000 7fc999400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc994158bd0 con 0x7fc9940be5d0 2024-03-28T19:26:17.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.969+0000 7fc999400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc994158eb0 con 0x7fc9940c58c0 2024-03-28T19:26:17.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.969+0000 7fc999400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc994159190 con 0x7fc9940c8ed0 2024-03-28T19:26:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.969+0000 7fc993400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9940c8ed0 0x7fc9940c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.969+0000 7fc993400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9940c8ed0 0x7fc9940c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47140/0 (socket says 172.21.15.27:47140) 2024-03-28T19:26:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.969+0000 7fc993400640 1 -- 172.21.15.27:0/3724248596 learned_addr learned my addr 172.21.15.27:0/3724248596 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.969+0000 7fc998a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc9940be5d0 0x7fc994155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36454/0 (socket says 172.21.15.27:36454) 2024-03-28T19:26:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.970+0000 7fc993400640 1 -- 172.21.15.27:0/3724248596 >> v1:172.21.15.155:6789/0 conn(0x7fc9940c58c0 legacy=0x7fc9941555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.970+0000 7fc992a00640 1 -- 172.21.15.27:0/3724248596 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039675113 0 0) 0x7fc994158bd0 con 0x7fc9940be5d0 2024-03-28T19:26:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.970+0000 7fc993400640 1 -- 172.21.15.27:0/3724248596 >> v1:172.21.15.27:6789/0 conn(0x7fc9940be5d0 legacy=0x7fc994155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.970+0000 7fc993400640 1 -- 172.21.15.27:0/3724248596 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc994159470 con 0x7fc9940c8ed0 2024-03-28T19:26:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.971+0000 7fc993400640 1 --2- 172.21.15.27:0/3724248596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940c8ed0 0x7fc9940c5ca0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fc980009870 tx=0x7fc98002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3f983329456e9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.972+0000 7fc992a00640 1 -- 172.21.15.27:0/3724248596 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980035020 con 0x7fc9940c8ed0 2024-03-28T19:26:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.972+0000 7fc992a00640 1 -- 172.21.15.27:0/3724248596 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc980031870 con 0x7fc9940c8ed0 2024-03-28T19:26:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.972+0000 7fc992a00640 1 -- 172.21.15.27:0/3724248596 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980031c40 con 0x7fc9940c8ed0 2024-03-28T19:26:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.973+0000 7fc999400640 1 -- 172.21.15.27:0/3724248596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940c8ed0 msgr2=0x7fc9940c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.973+0000 7fc999400640 1 --2- 172.21.15.27:0/3724248596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940c8ed0 0x7fc9940c5ca0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fc980009870 tx=0x7fc98002b910 comp rx=0 tx=0).stop 2024-03-28T19:26:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.973+0000 7fc999400640 1 -- 172.21.15.27:0/3724248596 shutdown_connections 2024-03-28T19:26:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.973+0000 7fc999400640 1 --2- 172.21.15.27:0/3724248596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940c8ed0 0x7fc9940c5ca0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.973+0000 7fc999400640 1 -- 172.21.15.27:0/3724248596 >> 172.21.15.27:0/3724248596 conn(0x7fc99406b6f0 msgr2=0x7fc9940acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.973+0000 7fc999400640 1 -- 172.21.15.27:0/3724248596 shutdown_connections 2024-03-28T19:26:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.973+0000 7fc999400640 1 -- 172.21.15.27:0/3724248596 wait complete. 2024-03-28T19:26:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.975+0000 7fc999400640 1 Processor -- start 2024-03-28T19:26:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.976+0000 7fc999400640 1 -- start start 2024-03-28T19:26:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.976+0000 7fc999400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940be5d0 0x7fc9940c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc999400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9940c58c0 0x7fc9940abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc999400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9940c8ed0 0x7fc9940aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc999400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940b09a0 con 0x7fc9940be5d0 2024-03-28T19:26:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc999400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940b0820 con 0x7fc9940c8ed0 2024-03-28T19:26:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc999400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940ad7d0 con 0x7fc9940c58c0 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc993e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940be5d0 0x7fc9940c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc998a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9940c8ed0 0x7fc9940aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc993e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940be5d0 0x7fc9940c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47150/0 (socket says 172.21.15.27:47150) 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc998a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9940c8ed0 0x7fc9940aaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:40074/0 (socket says 172.21.15.27:40074) 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc993e00640 1 -- 172.21.15.27:0/2775748999 learned_addr learned my addr 172.21.15.27:0/2775748999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc993400640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9940c58c0 0x7fc9940abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc998a00640 1 -- 172.21.15.27:0/2775748999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9940c58c0 msgr2=0x7fc9940abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc998a00640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9940c58c0 0x7fc9940abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.977+0000 7fc998a00640 1 -- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940be5d0 msgr2=0x7fc9940c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.978+0000 7fc998a00640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940be5d0 0x7fc9940c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.978+0000 7fc998a00640 1 -- 172.21.15.27:0/2775748999 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9940c2340 con 0x7fc9940c8ed0 2024-03-28T19:26:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.978+0000 7fc998a00640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9940c8ed0 0x7fc9940aaee0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c6d0 tx=0x7fc98c00cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.978+0000 7fc98be00640 1 -- 172.21.15.27:0/2775748999 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c018070 con 0x7fc9940c8ed0 2024-03-28T19:26:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.978+0000 7fc98be00640 1 -- 172.21.15.27:0/2775748999 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98c004d60 con 0x7fc9940c8ed0 2024-03-28T19:26:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.978+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9940c2620 con 0x7fc9940c8ed0 2024-03-28T19:26:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.978+0000 7fc98be00640 1 -- 172.21.15.27:0/2775748999 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00a880 con 0x7fc9940c8ed0 2024-03-28T19:26:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.978+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9941ebc10 con 0x7fc9940c8ed0 2024-03-28T19:26:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.979+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc994069640 con 0x7fc9940c8ed0 2024-03-28T19:26:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.981+0000 7fc98be00640 1 -- 172.21.15.27:0/2775748999 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc98c01b070 con 0x7fc9940c8ed0 2024-03-28T19:26:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.983+0000 7fc98be00640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97406bbd0 0x7fc97406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.983+0000 7fc98be00640 1 -- 172.21.15.27:0/2775748999 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7fc98c08b6c0 con 0x7fc9940c8ed0 2024-03-28T19:26:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.983+0000 7fc993e00640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97406bbd0 0x7fc97406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.984+0000 7fc993e00640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97406bbd0 0x7fc97406e090 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc9940ca3f0 tx=0x7fc984009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:17.988+0000 7fc98be00640 1 -- 172.21.15.27:0/2775748999 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc98c058a50 con 0x7fc9940c8ed0 2024-03-28T19:26:18.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.301+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc994156580 con 0x7fc9940c8ed0 2024-03-28T19:26:18.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.303+0000 7fc98be00640 1 -- 172.21.15.27:0/2775748999 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+5471 (secure 0 0 0) 0x7fc98c05caf0 con 0x7fc9940c8ed0 2024-03-28T19:26:18.303 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:18.303 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:18.303 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 228 2024-03-28T19:26:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97406bbd0 msgr2=0x7fc97406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97406bbd0 0x7fc97406e090 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc9940ca3f0 tx=0x7fc984009040 comp rx=0 tx=0).stop 2024-03-28T19:26:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9940c8ed0 msgr2=0x7fc9940aaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9940c8ed0 0x7fc9940aaee0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00c6d0 tx=0x7fc98c00cb90 comp rx=0 tx=0).stop 2024-03-28T19:26:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 shutdown_connections 2024-03-28T19:26:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc97406bbd0 0x7fc97406e090 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:18.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9940c8ed0 0x7fc9940aaee0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9940c58c0 0x7fc9940abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 --2- 172.21.15.27:0/2775748999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940be5d0 0x7fc9940c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 >> 172.21.15.27:0/2775748999 conn(0x7fc99406b6f0 msgr2=0x7fc9941dec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 shutdown_connections 2024-03-28T19:26:18.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:18.306+0000 7fc999400640 1 -- 172.21.15.27:0/2775748999 wait complete. 2024-03-28T19:26:18.327 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-03-28T19:26:18.327 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2024-03-28T19:26:19.474 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 19% (31/163) Updating files: 20% (33/163) Updating files: 21% (35/163) Updating files: 22% (36/163) Updating files: 23% (38/163) Updating files: 24% (40/163) Updating files: 25% (41/163) Updating files: 26% (43/163) Updating files: 26% (44/163) Updating files: 27% (45/163) Updating files: 28% (46/163) Updating files: 29% (48/163) Updating files: 30% (49/163) Updating files: 31% (51/163) Updating files: 32% (53/163) Updating files: 33% (54/163) Updating files: 34% (56/163) Updating files: 35% (58/163) Updating files: 36% (59/163) Updating files: 37% (61/163) Updating files: 38% (62/163) Updating files: 39% (64/163) Updating files: 40% (66/163) Updating files: 41% (67/163) Updating files: 42% (69/163) Updating files: 43% (71/163) Updating files: 44% (72/163) Updating files: 45% (74/163) Updating files: 46% (75/163) Updating files: 47% (77/163) Updating files: 48% (79/163) Updating files: 49% (80/163) Updating files: 50% (82/163) Updating files: 51% (84/163) Updating files: 52% (85/163) Updating files: 53% (87/163) Updating files: 54% (89/163) Updating files: 55% (90/163) Updating files: 55% (91/163) Updating files: 56% (92/163) Updating files: 57% (93/163) Updating files: 58% (95/163) Updating files: 59% (97/163) Updating files: 60% (98/163) Updating files: 61% (100/163) Updating files: 62% (102/163) Updating files: 63% (103/163) Updating files: 64% (105/163) Updating files: 65% (106/163) Updating files: 66% (108/163) Updating files: 67% (110/163) Updating files: 68% (111/163) Updating files: 69% (113/163) Updating files: 70% (115/163) Updating files: 71% (116/163) Updating files: 72% (118/163) Updating files: 73% (119/163) Updating files: 74% (121/163) Updating files: 75% (123/163) Updating files: 76% (124/163) Updating files: 77% (126/163) Updating files: 78% (128/163) Updating files: 79% (129/163) Updating files: 80% (131/163) Updating files: 81% (133/163) Updating files: 82% (134/163) Updating files: 83% (136/163) Updating files: 84% (137/163) Updating files: 85% (139/163) Updating files: 86% (141/163) Updating files: 87% (142/163) Updating files: 88% (144/163) Updating files: 89% (146/163) Updating files: 90% (147/163) Updating files: 91% (149/163) Updating files: 92% (150/163) Updating files: 93% (152/163) Updating files: 94% (154/163) Updating files: 95% (155/163) Updating files: 96% (157/163) Updating files: 97% (159/163) Updating files: 98% (160/163) Updating files: 99% (162/163) Updating files: 100% (163/163) Updating files: 100% (163/163), done. 2024-03-28T19:26:19.685 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 5d1d8d0c460 v0.23 2024-03-28T19:26:19.698 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.23 2024-03-28T19:26:19.726 INFO:tasks.workunit.client.1.smithi155.stdout:v0.24 2024-03-28T19:26:19.726 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:26:19.726 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 24 -eq 48 ']' 2024-03-28T19:26:19.727 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.24 2024-03-28T19:26:19.727 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.24 2024-03-28T19:26:19.727 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.24 2024-03-28T19:26:20.775 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T19:26:20.775+0000 7f009de00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:26:21.652 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-03-28T19:26:21.653 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:26:21.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:26:21.663 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:26:21.664 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:21.851 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:26:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.888+0000 7f86a8800640 1 Processor -- start 2024-03-28T19:26:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.888+0000 7f86a8800640 1 -- start start 2024-03-28T19:26:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.889+0000 7f86a8800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f86a0152d10 0x7f86a00c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.889+0000 7f86a8800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86a00c8e10 0x7f86a0158be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.889+0000 7f86a8800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a01530f0 con 0x7f86a01562f0 2024-03-28T19:26:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.889+0000 7f86a8800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a006d3d0 con 0x7f86a00c8e10 2024-03-28T19:26:21.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.889+0000 7f86a8800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a00c92f0 con 0x7f86a0152d10 2024-03-28T19:26:21.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.890+0000 7f86a5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86a00c8e10 0x7f86a0158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:21.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.890+0000 7f86a5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86a00c8e10 0x7f86a0158be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47176/0 (socket says 172.21.15.27:47176) 2024-03-28T19:26:21.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.890+0000 7f86a5400640 1 -- 172.21.15.27:0/2175422726 learned_addr learned my addr 172.21.15.27:0/2175422726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:21.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.890+0000 7f86a5e00640 1 --2- 172.21.15.27:0/2175422726 >> v2:172.21.15.155:3300/0 conn(0x7f86a0152d10 0x7f86a00c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:21.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.890+0000 7f86a5400640 1 -- 172.21.15.27:0/2175422726 >> v2:172.21.15.155:3300/0 conn(0x7f86a0152d10 msgr2=0x7f86a00c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:21.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.890+0000 7f86a5400640 1 --2- 172.21.15.27:0/2175422726 >> v2:172.21.15.155:3300/0 conn(0x7f86a0152d10 0x7f86a00c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:21.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.890+0000 7f86a5400640 1 -- 172.21.15.27:0/2175422726 >> v1:172.21.15.27:6789/0 conn(0x7f86a01562f0 legacy=0x7f86a00c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:21.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.890+0000 7f86a5400640 1 -- 172.21.15.27:0/2175422726 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0155810 con 0x7f86a00c8e10 2024-03-28T19:26:21.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.892+0000 7f86a5400640 1 --2- 172.21.15.27:0/2175422726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a00c8e10 0x7f86a0158be0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f86a006d170 tx=0x7f869002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892a94727047d8d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.892+0000 7f86a4a00640 1 -- 172.21.15.27:0/2175422726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690038070 con 0x7f86a00c8e10 2024-03-28T19:26:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.892+0000 7f86a4a00640 1 -- 172.21.15.27:0/2175422726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869002f050 con 0x7f86a00c8e10 2024-03-28T19:26:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.892+0000 7f86a4a00640 1 -- 172.21.15.27:0/2175422726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690031970 con 0x7f86a00c8e10 2024-03-28T19:26:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.893+0000 7f86a8800640 1 -- 172.21.15.27:0/2175422726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a00c8e10 msgr2=0x7f86a0158be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.893+0000 7f86a8800640 1 --2- 172.21.15.27:0/2175422726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a00c8e10 0x7f86a0158be0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f86a006d170 tx=0x7f869002b7f0 comp rx=0 tx=0).stop 2024-03-28T19:26:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.893+0000 7f86a8800640 1 -- 172.21.15.27:0/2175422726 shutdown_connections 2024-03-28T19:26:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.893+0000 7f86a8800640 1 --2- 172.21.15.27:0/2175422726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a00c8e10 0x7f86a0158be0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.893+0000 7f86a8800640 1 --2- 172.21.15.27:0/2175422726 >> v2:172.21.15.155:3300/0 conn(0x7f86a0152d10 0x7f86a00c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.893+0000 7f86a8800640 1 -- 172.21.15.27:0/2175422726 >> 172.21.15.27:0/2175422726 conn(0x7f86a0066fb0 msgr2=0x7f86a00673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.893+0000 7f86a8800640 1 -- 172.21.15.27:0/2175422726 shutdown_connections 2024-03-28T19:26:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.893+0000 7f86a8800640 1 -- 172.21.15.27:0/2175422726 wait complete. 2024-03-28T19:26:21.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.895+0000 7f86a8800640 1 Processor -- start 2024-03-28T19:26:21.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.895+0000 7f86a8800640 1 -- start start 2024-03-28T19:26:21.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.896+0000 7f86a8800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86a00c8e10 0x7f86a01eb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:21.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.896+0000 7f86a8800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a0152d10 0x7f86a014e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:21.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.896+0000 7f86a8800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86a01562f0 0x7f86a00c2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:21.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.896+0000 7f86a8800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a00c5dd0 con 0x7f86a0152d10 2024-03-28T19:26:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.896+0000 7f86a8800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a006d850 con 0x7f86a00c8e10 2024-03-28T19:26:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.896+0000 7f86a8800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a00c5f50 con 0x7f86a01562f0 2024-03-28T19:26:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a6800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86a01562f0 0x7f86a00c2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86a00c8e10 0x7f86a01eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a0152d10 0x7f86a014e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86a00c8e10 0x7f86a01eb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:40126/0 (socket says 172.21.15.27:40126) 2024-03-28T19:26:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a6800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86a01562f0 0x7f86a00c2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:58792/0 (socket says 172.21.15.27:58792) 2024-03-28T19:26:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5e00640 1 -- 172.21.15.27:0/2588361383 learned_addr learned my addr 172.21.15.27:0/2588361383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5400640 1 -- 172.21.15.27:0/2588361383 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86a01562f0 msgr2=0x7f86a00c2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5400640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86a01562f0 0x7f86a00c2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5400640 1 -- 172.21.15.27:0/2588361383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86a00c8e10 msgr2=0x7f86a01eb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5400640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86a00c8e10 0x7f86a01eb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5400640 1 -- 172.21.15.27:0/2588361383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a01ead10 con 0x7f86a0152d10 2024-03-28T19:26:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.897+0000 7f86a5e00640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86a00c8e10 0x7f86a01eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:26:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.898+0000 7f86a5400640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a0152d10 0x7f86a014e100 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f8690033000 tx=0x7f86900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.898+0000 7f869a000640 1 -- 172.21.15.27:0/2588361383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690042070 con 0x7f86a0152d10 2024-03-28T19:26:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.898+0000 7f869a000640 1 -- 172.21.15.27:0/2588361383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869003d870 con 0x7f86a0152d10 2024-03-28T19:26:21.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.898+0000 7f869a000640 1 -- 172.21.15.27:0/2588361383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8690038040 con 0x7f86a0152d10 2024-03-28T19:26:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.898+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a01ebcd0 con 0x7f86a0152d10 2024-03-28T19:26:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.898+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a00bed00 con 0x7f86a0152d10 2024-03-28T19:26:21.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.900+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a004f400 con 0x7f86a0152d10 2024-03-28T19:26:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.902+0000 7f869a000640 1 -- 172.21.15.27:0/2588361383 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8690045070 con 0x7f86a0152d10 2024-03-28T19:26:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.904+0000 7f869a000640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f868406bc20 0x7f868406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.904+0000 7f869a000640 1 -- 172.21.15.27:0/2588361383 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f86900adac0 con 0x7f86a0152d10 2024-03-28T19:26:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.904+0000 7f86a5e00640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f868406bc20 0x7f868406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.905+0000 7f86a5e00640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f868406bc20 0x7f868406e0e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8694009f50 tx=0x7f8694009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:21.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:21.909+0000 7f869a000640 1 -- 172.21.15.27:0/2588361383 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f869004c030 con 0x7f86a0152d10 2024-03-28T19:26:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.205+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86a006d630 con 0x7f86a0152d10 2024-03-28T19:26:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.206+0000 7f869a000640 1 -- 172.21.15.27:0/2588361383 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f86a01ead10 con 0x7f86a0152d10 2024-03-28T19:26:22.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:22.206 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 228 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f868406bc20 msgr2=0x7f868406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f868406bc20 0x7f868406e0e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8694009f50 tx=0x7f8694009190 comp rx=0 tx=0).stop 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a0152d10 msgr2=0x7f86a014e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a0152d10 0x7f86a014e100 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f8690033000 tx=0x7f86900046c0 comp rx=0 tx=0).stop 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 shutdown_connections 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f868406bc20 0x7f868406e0e0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86a01562f0 0x7f86a00c2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86a0152d10 0x7f86a014e100 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 --2- 172.21.15.27:0/2588361383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86a00c8e10 0x7f86a01eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 >> 172.21.15.27:0/2588361383 conn(0x7f86a0066fb0 msgr2=0x7f86a014b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.208+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 shutdown_connections 2024-03-28T19:26:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:22.209+0000 7f86a8800640 1 -- 172.21.15.27:0/2588361383 wait complete. 2024-03-28T19:26:22.231 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-03-28T19:26:24.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:24.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.452+0000 7fd6a9000640 1 Processor -- start 2024-03-28T19:26:24.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.452+0000 7fd6a9000640 1 -- start start 2024-03-28T19:26:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.454+0000 7fd6a9000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd6a4153100 0x7fd6a40c2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.454+0000 7fd6a9000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a40c78e0 con 0x7fd6a414fb10 2024-03-28T19:26:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.454+0000 7fd6a9000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a40c3e80 con 0x7fd6a414fef0 2024-03-28T19:26:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.454+0000 7fd6a9000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a4152560 con 0x7fd6a4153100 2024-03-28T19:26:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.454+0000 7fd6a1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd6a414fb10 0x7fd6a406c0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36478/0 (socket says 172.21.15.27:36478) 2024-03-28T19:26:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.455+0000 7fd6a1e00640 1 -- 172.21.15.27:0/1550944837 learned_addr learned my addr 172.21.15.27:0/1550944837 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:26:24.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.455+0000 7fd6a2800640 1 --2- 172.21.15.27:0/1550944837 >> v2:172.21.15.27:3300/0 conn(0x7fd6a4153100 0x7fd6a40c2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.455+0000 7fd6a0a00640 1 -- 172.21.15.27:0/1550944837 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490690858 0 0) 0x7fd6a40c78e0 con 0x7fd6a414fb10 2024-03-28T19:26:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.455+0000 7fd6a0a00640 1 -- 172.21.15.27:0/1550944837 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd688005730 con 0x7fd6a414fb10 2024-03-28T19:26:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.455+0000 7fd6a0a00640 1 -- 172.21.15.27:0/1550944837 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267476786 0 0) 0x7fd6a40c3e80 con 0x7fd6a414fef0 2024-03-28T19:26:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.455+0000 7fd6a2800640 1 -- 172.21.15.27:0/1550944837 >> v1:172.21.15.155:6789/0 conn(0x7fd6a414fef0 legacy=0x7fd6a41529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.456+0000 7fd6a2800640 1 -- 172.21.15.27:0/1550944837 >> v1:172.21.15.27:6789/0 conn(0x7fd6a414fb10 legacy=0x7fd6a406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:24.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.456+0000 7fd6a2800640 1 -- 172.21.15.27:0/1550944837 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a40c4160 con 0x7fd6a4153100 2024-03-28T19:26:24.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.457+0000 7fd6a2800640 1 --2- 172.21.15.27:0/1550944837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a40c2650 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009d90 tx=0x7fd69c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8fda436c45d6e70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:24.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.458+0000 7fd6a0a00640 1 -- 172.21.15.27:0/1550944837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c039070 con 0x7fd6a4153100 2024-03-28T19:26:24.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.458+0000 7fd6a0a00640 1 -- 172.21.15.27:0/1550944837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c02cc60 con 0x7fd6a4153100 2024-03-28T19:26:24.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.458+0000 7fd6a0a00640 1 -- 172.21.15.27:0/1550944837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c034660 con 0x7fd6a4153100 2024-03-28T19:26:24.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.459+0000 7fd6a9000640 1 -- 172.21.15.27:0/1550944837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 msgr2=0x7fd6a40c2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:24.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.459+0000 7fd6a9000640 1 --2- 172.21.15.27:0/1550944837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a40c2650 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009d90 tx=0x7fd69c02beb0 comp rx=0 tx=0).stop 2024-03-28T19:26:24.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.459+0000 7fd6a9000640 1 -- 172.21.15.27:0/1550944837 shutdown_connections 2024-03-28T19:26:24.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.459+0000 7fd6a9000640 1 --2- 172.21.15.27:0/1550944837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a40c2650 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:24.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.459+0000 7fd6a9000640 1 -- 172.21.15.27:0/1550944837 >> 172.21.15.27:0/1550944837 conn(0x7fd6a40c4750 msgr2=0x7fd6a40c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:24.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.459+0000 7fd6a9000640 1 -- 172.21.15.27:0/1550944837 shutdown_connections 2024-03-28T19:26:24.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.459+0000 7fd6a9000640 1 -- 172.21.15.27:0/1550944837 wait complete. 2024-03-28T19:26:24.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.462+0000 7fd6a9000640 1 Processor -- start 2024-03-28T19:26:24.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.462+0000 7fd6a9000640 1 -- start start 2024-03-28T19:26:24.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.463+0000 7fd6a9000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6a414fb10 0x7fd6a40acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:24.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.463+0000 7fd6a9000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6a414fef0 0x7fd6a40ac060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:24.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.463+0000 7fd6a9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a414f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:24.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.463+0000 7fd6a9000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a406c7e0 con 0x7fd6a4153100 2024-03-28T19:26:24.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.463+0000 7fd6a9000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40c7570 con 0x7fd6a414fef0 2024-03-28T19:26:24.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.463+0000 7fd6a9000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a406c960 con 0x7fd6a414fb10 2024-03-28T19:26:24.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a414f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:24.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a414f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47204/0 (socket says 172.21.15.27:47204) 2024-03-28T19:26:24.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a2800640 1 -- 172.21.15.27:0/2102109555 learned_addr learned my addr 172.21.15.27:0/2102109555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a1e00640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6a414fb10 0x7fd6a40acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a1400640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6a414fef0 0x7fd6a40ac060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a2800640 1 -- 172.21.15.27:0/2102109555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6a414fb10 msgr2=0x7fd6a40acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a2800640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6a414fb10 0x7fd6a40acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a2800640 1 -- 172.21.15.27:0/2102109555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6a414fef0 msgr2=0x7fd6a40ac060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a2800640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6a414fef0 0x7fd6a40ac060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.464+0000 7fd6a2800640 1 -- 172.21.15.27:0/2102109555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a41ef130 con 0x7fd6a4153100 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.465+0000 7fd6a2800640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a414f030 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009d90 tx=0x7fd69c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.465+0000 7fd69a000640 1 -- 172.21.15.27:0/2102109555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c044030 con 0x7fd6a4153100 2024-03-28T19:26:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.465+0000 7fd69a000640 1 -- 172.21.15.27:0/2102109555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c02ca60 con 0x7fd6a4153100 2024-03-28T19:26:24.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.465+0000 7fd69a000640 1 -- 172.21.15.27:0/2102109555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c039040 con 0x7fd6a4153100 2024-03-28T19:26:24.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.465+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a41ef410 con 0x7fd6a4153100 2024-03-28T19:26:24.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.465+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a41ef9d0 con 0x7fd6a4153100 2024-03-28T19:26:24.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.467+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a41efc10 con 0x7fd6a4153100 2024-03-28T19:26:24.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.468+0000 7fd69a000640 1 -- 172.21.15.27:0/2102109555 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd69c047070 con 0x7fd6a4153100 2024-03-28T19:26:24.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.469+0000 7fd69a000640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd68406bc20 0x7fd68406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:24.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.469+0000 7fd69a000640 1 -- 172.21.15.27:0/2102109555 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7fd69c0ae0d0 con 0x7fd6a4153100 2024-03-28T19:26:24.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.469+0000 7fd6a1e00640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd68406bc20 0x7fd68406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:24.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.471+0000 7fd6a1e00640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd68406bc20 0x7fd68406e0e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd694005880 tx=0x7fd694009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:24.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.475+0000 7fd69a000640 1 -- 172.21.15.27:0/2102109555 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd6a41efc10 con 0x7fd6a4153100 2024-03-28T19:26:24.675 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 35% (66/185) Updating files: 36% (67/185) Updating files: 37% (69/185) Updating files: 38% (71/185) Updating files: 39% (73/185) Updating files: 40% (74/185) Updating files: 41% (76/185) Updating files: 42% (78/185) Updating files: 43% (80/185) Updating files: 44% (82/185) Updating files: 45% (84/185) Updating files: 46% (86/185) Updating files: 47% (87/185) Updating files: 48% (89/185) Updating files: 49% (91/185) Updating files: 50% (93/185) Updating files: 51% (95/185) Updating files: 52% (97/185) Updating files: 53% (99/185) Updating files: 54% (100/185) Updating files: 55% (102/185) Updating files: 56% (104/185) Updating files: 57% (106/185) Updating files: 58% (108/185) Updating files: 59% (110/185) Updating files: 60% (111/185) Updating files: 61% (113/185) Updating files: 62% (115/185) Updating files: 63% (117/185) Updating files: 64% (119/185) Updating files: 65% (121/185) Updating files: 66% (123/185) Updating files: 67% (124/185) Updating files: 68% (126/185) Updating files: 69% (128/185) Updating files: 70% (130/185) Updating files: 71% (132/185) Updating files: 72% (134/185) Updating files: 73% (136/185) Updating files: 74% (137/185) Updating files: 75% (139/185) Updating files: 76% (141/185) Updating files: 77% (143/185) Updating files: 78% (145/185) Updating files: 79% (147/185) Updating files: 80% (148/185) Updating files: 81% (150/185) Updating files: 82% (152/185) Updating files: 83% (154/185) Updating files: 84% (156/185) Updating files: 85% (158/185) Updating files: 86% (160/185) Updating files: 87% (161/185) Updating files: 88% (163/185) Updating files: 89% (165/185) Updating files: 90% (167/185) Updating files: 91% (169/185) Updating files: 92% (171/185) Updating files: 93% (173/185) Updating files: 94% (174/185) Updating files: 95% (176/185) Updating files: 96% (178/185) Updating files: 97% (180/185) Updating files: 98% (182/185) Updating files: 99% (184/185) Updating files: 100% (185/185) Updating files: 100% (185/185), done. 2024-03-28T19:26:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.794+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6a404f400 con 0x7fd6a4153100 2024-03-28T19:26:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.796+0000 7fd69a000640 1 -- 172.21.15.27:0/2102109555 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fd69c04e070 con 0x7fd6a4153100 2024-03-28T19:26:24.796 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:24.796 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":8928,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":229}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:24.797 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 229 2024-03-28T19:26:24.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd68406bc20 msgr2=0x7fd68406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:24.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd68406bc20 0x7fd68406e0e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd694005880 tx=0x7fd694009040 comp rx=0 tx=0).stop 2024-03-28T19:26:24.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 msgr2=0x7fd6a414f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:24.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a414f030 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009d90 tx=0x7fd69c004a40 comp rx=0 tx=0).stop 2024-03-28T19:26:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 shutdown_connections 2024-03-28T19:26:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd68406bc20 0x7fd68406e0e0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a4153100 0x7fd6a414f030 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6a414fef0 0x7fd6a40ac060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 --2- 172.21.15.27:0/2102109555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6a414fb10 0x7fd6a40acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 >> 172.21.15.27:0/2102109555 conn(0x7fd6a40c4750 msgr2=0x7fd6a40c4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.798+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 shutdown_connections 2024-03-28T19:26:24.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:24.799+0000 7fd6a9000640 1 -- 172.21.15.27:0/2102109555 wait complete. 2024-03-28T19:26:24.819 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-03-28T19:26:24.819 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2024-03-28T19:26:24.908 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 180a4176035 v0.24 2024-03-28T19:26:24.915 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.24 2024-03-28T19:26:24.976 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:26:24.976 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 25 -eq 48 ']' 2024-03-28T19:26:24.977 INFO:tasks.workunit.client.1.smithi155.stdout:v0.25 2024-03-28T19:26:24.977 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.25 2024-03-28T19:26:24.977 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.25 2024-03-28T19:26:24.977 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.25 2024-03-28T19:26:28.407 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 17% (57/334) Updating files: 18% (61/334) Updating files: 19% (64/334) Updating files: 20% (67/334) Updating files: 21% (71/334) Updating files: 22% (74/334) Updating files: 23% (77/334) Updating files: 24% (81/334) Updating files: 25% (84/334) Updating files: 26% (87/334) Updating files: 27% (91/334) Updating files: 28% (94/334) Updating files: 29% (97/334) Updating files: 30% (101/334) Updating files: 31% (104/334) Updating files: 32% (107/334) Updating files: 33% (111/334) Updating files: 33% (113/334) Updating files: 34% (114/334) Updating files: 35% (117/334) Updating files: 36% (121/334) Updating files: 37% (124/334) Updating files: 38% (127/334) Updating files: 39% (131/334) Updating files: 40% (134/334) Updating files: 41% (137/334) Updating files: 42% (141/334) Updating files: 43% (144/334) Updating files: 44% (147/334) Updating files: 45% (151/334) Updating files: 45% (152/334) Updating files: 46% (154/334) Updating files: 47% (157/334) Updating files: 48% (161/334) Updating files: 49% (164/334) Updating files: 50% (167/334) Updating files: 51% (171/334) Updating files: 52% (174/334) Updating files: 53% (178/334) Updating files: 54% (181/334) Updating files: 55% (184/334) Updating files: 56% (188/334) Updating files: 57% (191/334) Updating files: 58% (194/334) Updating files: 58% (195/334) Updating files: 59% (198/334) Updating files: 60% (201/334) Updating files: 61% (204/334) Updating files: 62% (208/334) Updating files: 63% (211/334) Updating files: 64% (214/334) Updating files: 65% (218/334) Updating files: 66% (221/334) Updating files: 67% (224/334) Updating files: 68% (228/334) Updating files: 69% (231/334) Updating files: 70% (234/334) Updating files: 70% (236/334) Updating files: 71% (238/334) Updating files: 72% (241/334) Updating files: 73% (244/334) Updating files: 74% (248/334) Updating files: 75% (251/334) Updating files: 76% (254/334) Updating files: 77% (258/334) Updating files: 78% (261/334) Updating files: 79% (264/334) Updating files: 80% (268/334) Updating files: 81% (271/334) Updating files: 82% (274/334) Updating files: 83% (278/334) Updating files: 84% (281/334) Updating files: 85% (284/334) Updating files: 86% (288/334) Updating files: 87% (291/334) Updating files: 88% (294/334) Updating files: 89% (298/334) Updating files: 90% (301/334) Updating files: 91% (304/334) Updating files: 92% (308/334) Updating files: 93% (311/334) Updating files: 94% (314/334) Updating files: 95% (318/334) Updating files: 96% (321/334) Updating files: 97% (324/334) Updating files: 98% (328/334) Updating files: 99% (331/334) Updating files: 100% (334/334) Updating files: 100% (334/334), done. 2024-03-28T19:26:28.693 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 0afda378124 v0.35 2024-03-28T19:26:28.706 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.35 2024-03-28T19:26:28.786 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:26:28.786 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 36 -eq 48 ']' 2024-03-28T19:26:28.786 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.36 2024-03-28T19:26:28.786 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.36 2024-03-28T19:26:28.786 INFO:tasks.workunit.client.0.smithi027.stdout:v0.36 2024-03-28T19:26:28.787 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.36 2024-03-28T19:26:37.940 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 25% (63/244) Updating files: 26% (64/244) Updating files: 27% (66/244) Updating files: 28% (69/244) Updating files: 29% (71/244) Updating files: 30% (74/244) Updating files: 31% (76/244) Updating files: 32% (79/244) Updating files: 33% (81/244) Updating files: 34% (83/244) Updating files: 35% (86/244) Updating files: 36% (88/244) Updating files: 37% (91/244) Updating files: 38% (93/244) Updating files: 39% (96/244) Updating files: 40% (98/244) Updating files: 41% (101/244) Updating files: 42% (103/244) Updating files: 43% (105/244) Updating files: 44% (108/244) Updating files: 45% (110/244) Updating files: 46% (113/244) Updating files: 47% (115/244) Updating files: 48% (118/244) Updating files: 49% (120/244) Updating files: 50% (122/244) Updating files: 51% (125/244) Updating files: 52% (127/244) Updating files: 53% (130/244) Updating files: 54% (132/244) Updating files: 55% (135/244) Updating files: 56% (137/244) Updating files: 57% (140/244) Updating files: 58% (142/244) Updating files: 59% (144/244) Updating files: 60% (147/244) Updating files: 61% (149/244) Updating files: 62% (152/244) Updating files: 63% (154/244) Updating files: 63% (156/244) Updating files: 64% (157/244) Updating files: 65% (159/244) Updating files: 66% (162/244) Updating files: 67% (164/244) Updating files: 68% (166/244) Updating files: 69% (169/244) Updating files: 70% (171/244) Updating files: 71% (174/244) Updating files: 72% (176/244) Updating files: 73% (179/244) Updating files: 74% (181/244) Updating files: 75% (183/244) Updating files: 76% (186/244) Updating files: 77% (188/244) Updating files: 77% (190/244) Updating files: 78% (191/244) Updating files: 79% (193/244) Updating files: 80% (196/244) Updating files: 81% (198/244) Updating files: 82% (201/244) Updating files: 83% (203/244) Updating files: 84% (205/244) Updating files: 85% (208/244) Updating files: 86% (210/244) Updating files: 87% (213/244) Updating files: 88% (215/244) Updating files: 89% (218/244) Updating files: 90% (220/244) Updating files: 91% (223/244) Updating files: 92% (225/244) Updating files: 92% (226/244) Updating files: 93% (227/244) Updating files: 94% (230/244) Updating files: 95% (232/244) Updating files: 96% (235/244) Updating files: 97% (237/244) Updating files: 98% (240/244) Updating files: 99% (242/244) Updating files: 100% (244/244) Updating files: 100% (244/244), done. 2024-03-28T19:26:38.064 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 12% (79/622) Updating files: 13% (81/622) Updating files: 14% (88/622) Updating files: 15% (94/622) Updating files: 16% (100/622) Updating files: 17% (106/622) Updating files: 18% (112/622) Updating files: 19% (119/622) Updating files: 20% (125/622) Updating files: 21% (131/622) Updating files: 22% (137/622) Updating files: 23% (144/622) Updating files: 24% (150/622) Updating files: 25% (156/622) Updating files: 26% (162/622) Updating files: 27% (168/622) Updating files: 28% (175/622) Updating files: 29% (181/622) Updating files: 30% (187/622) Updating files: 31% (193/622) Updating files: 32% (200/622) Updating files: 33% (206/622) Updating files: 34% (212/622) Updating files: 35% (218/622) Updating files: 36% (224/622) Updating files: 37% (231/622) Updating files: 37% (233/622) Updating files: 38% (237/622) Updating files: 39% (243/622) Updating files: 40% (249/622) Updating files: 41% (256/622) Updating files: 42% (262/622) Updating files: 43% (268/622) Updating files: 44% (274/622) Updating files: 45% (280/622) Updating files: 46% (287/622) Updating files: 47% (293/622) Updating files: 48% (299/622) Updating files: 49% (305/622) Updating files: 50% (311/622) Updating files: 50% (317/622) Updating files: 51% (318/622) Updating files: 52% (324/622) Updating files: 53% (330/622) Updating files: 54% (336/622) Updating files: 55% (343/622) Updating files: 56% (349/622) Updating files: 57% (355/622) Updating files: 58% (361/622) Updating files: 59% (367/622) Updating files: 60% (374/622) Updating files: 61% (380/622) Updating files: 61% (383/622) Updating files: 62% (386/622) Updating files: 63% (392/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 67% (417/622) Updating files: 67% (418/622) Updating files: 68% (423/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 71% (442/622) Updating files: 72% (448/622) Updating files: 73% (455/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 78% (486/622) Updating files: 79% (492/622) Updating files: 80% (498/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 82% (516/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 92% (578/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2024-03-28T19:26:38.280 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 3fa121bf0a8 v0.25 2024-03-28T19:26:38.288 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.25 2024-03-28T19:26:38.288 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 877cacbbdbb v0.36 2024-03-28T19:26:38.298 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.36 2024-03-28T19:26:38.377 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:26:38.377 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 37 -eq 48 ']' 2024-03-28T19:26:38.377 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.37 2024-03-28T19:26:38.377 INFO:tasks.workunit.client.0.smithi027.stdout:v0.37 2024-03-28T19:26:38.377 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.37 2024-03-28T19:26:38.377 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.37 2024-03-28T19:26:38.392 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:26:38.392 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 26 -eq 48 ']' 2024-03-28T19:26:38.392 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.26 2024-03-28T19:26:38.392 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.26 2024-03-28T19:26:38.392 INFO:tasks.workunit.client.1.smithi155.stdout:v0.26 2024-03-28T19:26:38.393 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.26 2024-03-28T19:26:45.100 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 23% (55/239) Updating files: 24% (58/239) Updating files: 25% (60/239) Updating files: 26% (63/239) Updating files: 27% (65/239) Updating files: 28% (67/239) Updating files: 29% (70/239) Updating files: 30% (72/239) Updating files: 31% (75/239) Updating files: 32% (77/239) Updating files: 33% (79/239) Updating files: 34% (82/239) Updating files: 35% (84/239) Updating files: 36% (87/239) Updating files: 37% (89/239) Updating files: 38% (91/239) Updating files: 39% (94/239) Updating files: 40% (96/239) Updating files: 41% (98/239) Updating files: 42% (101/239) Updating files: 43% (103/239) Updating files: 44% (106/239) Updating files: 45% (108/239) Updating files: 46% (110/239) Updating files: 47% (113/239) Updating files: 48% (115/239) Updating files: 49% (118/239) Updating files: 50% (120/239) Updating files: 51% (122/239) Updating files: 52% (125/239) Updating files: 53% (127/239) Updating files: 54% (130/239) Updating files: 55% (132/239) Updating files: 56% (134/239) Updating files: 57% (137/239) Updating files: 58% (139/239) Updating files: 59% (142/239) Updating files: 60% (144/239) Updating files: 61% (146/239) Updating files: 62% (149/239) Updating files: 62% (150/239) Updating files: 63% (151/239) Updating files: 64% (153/239) Updating files: 65% (156/239) Updating files: 66% (158/239) Updating files: 67% (161/239) Updating files: 68% (163/239) Updating files: 69% (165/239) Updating files: 70% (168/239) Updating files: 71% (170/239) Updating files: 72% (173/239) Updating files: 73% (175/239) Updating files: 74% (177/239) Updating files: 75% (180/239) Updating files: 76% (182/239) Updating files: 77% (185/239) Updating files: 78% (187/239) Updating files: 79% (189/239) Updating files: 80% (192/239) Updating files: 81% (194/239) Updating files: 82% (196/239) Updating files: 83% (199/239) Updating files: 84% (201/239) Updating files: 85% (204/239) Updating files: 85% (205/239) Updating files: 86% (206/239) Updating files: 87% (208/239) Updating files: 88% (211/239) Updating files: 89% (213/239) Updating files: 90% (216/239) Updating files: 91% (218/239) Updating files: 92% (220/239) Updating files: 93% (223/239) Updating files: 94% (225/239) Updating files: 95% (228/239) Updating files: 96% (230/239) Updating files: 97% (232/239) Updating files: 98% (235/239) Updating files: 99% (237/239) Updating files: 100% (239/239) Updating files: 100% (239/239), done. 2024-03-28T19:26:45.259 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 9981ff90968 v0.26 2024-03-28T19:26:45.273 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.26 2024-03-28T19:26:45.320 INFO:tasks.workunit.client.1.smithi155.stdout:v0.27 2024-03-28T19:26:45.320 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:26:45.321 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 27 -eq 48 ']' 2024-03-28T19:26:45.321 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.27 2024-03-28T19:26:45.321 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.27 2024-03-28T19:26:45.321 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.27 2024-03-28T19:26:45.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:45.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.556+0000 7f613d800640 1 Processor -- start 2024-03-28T19:26:45.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.556+0000 7f613d800640 1 -- start start 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.557+0000 7f613d800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61380c8e10 0x7f6138158be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.557+0000 7f613d800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f61381562f0 0x7f613815ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.557+0000 7f613d800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61381530f0 con 0x7f6138152d10 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.557+0000 7f613d800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613806d3d0 con 0x7f61381562f0 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.557+0000 7f613d800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61380c92f0 con 0x7f61380c8e10 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.557+0000 7f6136a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6138152d10 0x7f61380c5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58494/0 (socket says 172.21.15.27:58494) 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.557+0000 7f6136a00640 1 -- 172.21.15.27:0/1110221703 learned_addr learned my addr 172.21.15.27:0/1110221703 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.557+0000 7f6136000640 1 --2- 172.21.15.27:0/1110221703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61380c8e10 0x7f6138158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.558+0000 7f6137400640 1 --2- 172.21.15.27:0/1110221703 >> v2:172.21.15.155:3300/0 conn(0x7f61381562f0 0x7f613815ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.558+0000 7f6135600640 1 -- 172.21.15.27:0/1110221703 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676158834 0 0) 0x7f6128001110 con 0x7f6138152d10 2024-03-28T19:26:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.558+0000 7f6135600640 1 -- 172.21.15.27:0/1110221703 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f611c003520 con 0x7f6138152d10 2024-03-28T19:26:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.558+0000 7f6136000640 1 -- 172.21.15.27:0/1110221703 >> v2:172.21.15.155:3300/0 conn(0x7f61381562f0 msgr2=0x7f613815ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.558+0000 7f6136000640 1 --2- 172.21.15.27:0/1110221703 >> v2:172.21.15.155:3300/0 conn(0x7f61381562f0 0x7f613815ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.558+0000 7f6136000640 1 -- 172.21.15.27:0/1110221703 >> v1:172.21.15.27:6789/0 conn(0x7f6138152d10 legacy=0x7f61380c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.558+0000 7f6136000640 1 -- 172.21.15.27:0/1110221703 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138155810 con 0x7f61380c8e10 2024-03-28T19:26:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.559+0000 7f6136000640 1 --2- 172.21.15.27:0/1110221703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61380c8e10 0x7f6138158be0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f61240097a0 tx=0x7f612402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480a7251ab86108d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.560+0000 7f6135600640 1 -- 172.21.15.27:0/1110221703 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124039070 con 0x7f61380c8e10 2024-03-28T19:26:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.560+0000 7f6135600640 1 -- 172.21.15.27:0/1110221703 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61240316f0 con 0x7f61380c8e10 2024-03-28T19:26:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.560+0000 7f6135600640 1 -- 172.21.15.27:0/1110221703 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6124031ac0 con 0x7f61380c8e10 2024-03-28T19:26:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.561+0000 7f613d800640 1 -- 172.21.15.27:0/1110221703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61380c8e10 msgr2=0x7f6138158be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.561+0000 7f613d800640 1 --2- 172.21.15.27:0/1110221703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61380c8e10 0x7f6138158be0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f61240097a0 tx=0x7f612402b790 comp rx=0 tx=0).stop 2024-03-28T19:26:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.561+0000 7f613d800640 1 -- 172.21.15.27:0/1110221703 shutdown_connections 2024-03-28T19:26:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.561+0000 7f613d800640 1 --2- 172.21.15.27:0/1110221703 >> v2:172.21.15.155:3300/0 conn(0x7f61381562f0 0x7f613815ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.561+0000 7f613d800640 1 --2- 172.21.15.27:0/1110221703 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61380c8e10 0x7f6138158be0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.561+0000 7f613d800640 1 -- 172.21.15.27:0/1110221703 >> 172.21.15.27:0/1110221703 conn(0x7f6138066fb0 msgr2=0x7f61380673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.561+0000 7f613d800640 1 -- 172.21.15.27:0/1110221703 shutdown_connections 2024-03-28T19:26:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.561+0000 7f613d800640 1 -- 172.21.15.27:0/1110221703 wait complete. 2024-03-28T19:26:45.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.563+0000 7f613d800640 1 Processor -- start 2024-03-28T19:26:45.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.563+0000 7f613d800640 1 -- start start 2024-03-28T19:26:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.564+0000 7f613d800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61380c8e10 0x7f61381ec7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.564+0000 7f613d800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6138152d10 0x7f61381eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.564+0000 7f613d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61381562f0 0x7f6138157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.564+0000 7f613d800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61380c5fa0 con 0x7f61381562f0 2024-03-28T19:26:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.564+0000 7f613d800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613806d850 con 0x7f61380c8e10 2024-03-28T19:26:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.564+0000 7f613d800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61380c6120 con 0x7f6138152d10 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.564+0000 7f6137400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61381562f0 0x7f6138157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6137400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61381562f0 0x7f6138157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52388/0 (socket says 172.21.15.27:52388) 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6137400640 1 -- 172.21.15.27:0/1723968924 learned_addr learned my addr 172.21.15.27:0/1723968924 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6136000640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6138152d10 0x7f61381eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6136a00640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61380c8e10 0x7f61381ec7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6137400640 1 -- 172.21.15.27:0/1723968924 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6138152d10 msgr2=0x7f61381eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6137400640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6138152d10 0x7f61381eeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6137400640 1 -- 172.21.15.27:0/1723968924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61380c8e10 msgr2=0x7f61381ec7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6137400640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61380c8e10 0x7f61381ec7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.565+0000 7f6137400640 1 -- 172.21.15.27:0/1723968924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61381ef380 con 0x7f61381562f0 2024-03-28T19:26:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.566+0000 7f6137400640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61381562f0 0x7f6138157c80 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f613000efc0 tx=0x7f613000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.566+0000 7f612ea00640 1 -- 172.21.15.27:0/1723968924 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130018020 con 0x7f61381562f0 2024-03-28T19:26:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.566+0000 7f612ea00640 1 -- 172.21.15.27:0/1723968924 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6130014480 con 0x7f61381562f0 2024-03-28T19:26:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.566+0000 7f612ea00640 1 -- 172.21.15.27:0/1723968924 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130014870 con 0x7f61381562f0 2024-03-28T19:26:45.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.566+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61381ef660 con 0x7f61381562f0 2024-03-28T19:26:45.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.568+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61381ef080 con 0x7f61381562f0 2024-03-28T19:26:45.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.569+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61380c5560 con 0x7f61381562f0 2024-03-28T19:26:45.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.570+0000 7f612ea00640 1 -- 172.21.15.27:0/1723968924 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6130005d10 con 0x7f61381562f0 2024-03-28T19:26:45.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.571+0000 7f612ea00640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f611806bc20 0x7f611806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:45.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.571+0000 7f612ea00640 1 -- 172.21.15.27:0/1723968924 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f613008c440 con 0x7f61381562f0 2024-03-28T19:26:45.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.572+0000 7f6136a00640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f611806bc20 0x7f611806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:45.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.573+0000 7f6136a00640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f611806bc20 0x7f611806e0e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6128005880 tx=0x7f6128009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:45.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.577+0000 7f612ea00640 1 -- 172.21.15.27:0/1723968924 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f61380c5560 con 0x7f61381562f0 2024-03-28T19:26:45.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.896+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f613804f400 con 0x7f61381562f0 2024-03-28T19:26:45.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.897+0000 7f612ea00640 1 -- 172.21.15.27:0/1723968924 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f6130022070 con 0x7f61381562f0 2024-03-28T19:26:45.896 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:45.896 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":8928,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":229}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:45.897 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 229 2024-03-28T19:26:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f611806bc20 msgr2=0x7f611806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f611806bc20 0x7f611806e0e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6128005880 tx=0x7f6128009040 comp rx=0 tx=0).stop 2024-03-28T19:26:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61381562f0 msgr2=0x7f6138157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61381562f0 0x7f6138157c80 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f613000efc0 tx=0x7f613000c490 comp rx=0 tx=0).stop 2024-03-28T19:26:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 shutdown_connections 2024-03-28T19:26:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f611806bc20 0x7f611806e0e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61381562f0 0x7f6138157c80 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6138152d10 0x7f61381eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 --2- 172.21.15.27:0/1723968924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61380c8e10 0x7f61381ec7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 >> 172.21.15.27:0/1723968924 conn(0x7f6138066fb0 msgr2=0x7f61380ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 shutdown_connections 2024-03-28T19:26:45.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:45.900+0000 7f613d800640 1 -- 172.21.15.27:0/1723968924 wait complete. 2024-03-28T19:26:45.919 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-03-28T19:26:45.919 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T19:26:45.919 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:26:45.945 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T19:26:45.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.156+0000 7f4ede600640 1 Processor -- start 2024-03-28T19:26:46.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.156+0000 7f4ede600640 1 -- start start 2024-03-28T19:26:46.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.157+0000 7f4ede600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80c5e80 0x7f4ed80c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.157+0000 7f4ede600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ed80c1f50 0x7f4ed80ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.157+0000 7f4ede600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed806d8a0 con 0x7f4ed80abd90 2024-03-28T19:26:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ede600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed80ac220 con 0x7f4ed80c1f50 2024-03-28T19:26:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ede600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed806d650 con 0x7f4ed80c5e80 2024-03-28T19:26:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ed6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ed80c1f50 0x7f4ed80ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ed6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ed80c1f50 0x7f4ed80ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51470/0 (socket says 172.21.15.27:51470) 2024-03-28T19:26:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ed6a00640 1 -- 172.21.15.27:0/3015997452 learned_addr learned my addr 172.21.15.27:0/3015997452 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:46.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ed6a00640 1 -- 172.21.15.27:0/3015997452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80c5e80 msgr2=0x7f4ed80c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:46.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ed7400640 1 --2- 172.21.15.27:0/3015997452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80c5e80 0x7f4ed80c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:46.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ed6a00640 1 --2- 172.21.15.27:0/3015997452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80c5e80 0x7f4ed80c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.158+0000 7f4ed6a00640 1 -- 172.21.15.27:0/3015997452 >> v1:172.21.15.155:6789/0 conn(0x7f4ed80abd90 legacy=0x7f4ed80aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:46.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.159+0000 7f4ed6a00640 1 -- 172.21.15.27:0/3015997452 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed806db80 con 0x7f4ed80c1f50 2024-03-28T19:26:46.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.159+0000 7f4ed7400640 1 --2- 172.21.15.27:0/3015997452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80c5e80 0x7f4ed80c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:26:46.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.160+0000 7f4ed6a00640 1 --2- 172.21.15.27:0/3015997452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c1f50 0x7f4ed80ab850 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4009870 tx=0x7f4ec402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc0e62a658b9d1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:46.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.160+0000 7f4ed6000640 1 -- 172.21.15.27:0/3015997452 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40317c0 con 0x7f4ed80c1f50 2024-03-28T19:26:46.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.160+0000 7f4ed6000640 1 -- 172.21.15.27:0/3015997452 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec4037070 con 0x7f4ed80c1f50 2024-03-28T19:26:46.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.160+0000 7f4ed6000640 1 -- 172.21.15.27:0/3015997452 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4031b80 con 0x7f4ed80c1f50 2024-03-28T19:26:46.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.161+0000 7f4ede600640 1 -- 172.21.15.27:0/3015997452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c1f50 msgr2=0x7f4ed80ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:46.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.161+0000 7f4ede600640 1 --2- 172.21.15.27:0/3015997452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c1f50 0x7f4ed80ab850 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4009870 tx=0x7f4ec402b860 comp rx=0 tx=0).stop 2024-03-28T19:26:46.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.161+0000 7f4ede600640 1 -- 172.21.15.27:0/3015997452 shutdown_connections 2024-03-28T19:26:46.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.161+0000 7f4ede600640 1 --2- 172.21.15.27:0/3015997452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c1f50 0x7f4ed80ab850 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.161+0000 7f4ede600640 1 --2- 172.21.15.27:0/3015997452 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80c5e80 0x7f4ed80c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.161+0000 7f4ede600640 1 -- 172.21.15.27:0/3015997452 >> 172.21.15.27:0/3015997452 conn(0x7f4ed806b9d0 msgr2=0x7f4ed80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:46.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.161+0000 7f4ede600640 1 -- 172.21.15.27:0/3015997452 shutdown_connections 2024-03-28T19:26:46.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.161+0000 7f4ede600640 1 -- 172.21.15.27:0/3015997452 wait complete. 2024-03-28T19:26:46.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.164+0000 7f4ede600640 1 Processor -- start 2024-03-28T19:26:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.164+0000 7f4ede600640 1 -- start start 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ede600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80abd90 0x7f4ed8154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ede600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ed80c1f50 0x7f4ed814edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ede600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c5e80 0x7f4ed814f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ede600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80c31f0 con 0x7f4ed80c5e80 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ede600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed806e1d0 con 0x7f4ed80c1f50 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ede600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80c3370 con 0x7f4ed80abd90 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ed7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c5e80 0x7f4ed814f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ed7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c5e80 0x7f4ed814f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51472/0 (socket says 172.21.15.27:51472) 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ed7e00640 1 -- 172.21.15.27:0/3082579580 learned_addr learned my addr 172.21.15.27:0/3082579580 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.165+0000 7f4ed6a00640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ed80c1f50 0x7f4ed814edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed7400640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80abd90 0x7f4ed8154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed7e00640 1 -- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80abd90 msgr2=0x7f4ed8154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:46.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed7e00640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80abd90 0x7f4ed8154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed7e00640 1 -- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ed80c1f50 msgr2=0x7f4ed814edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:46.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed7e00640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ed80c1f50 0x7f4ed814edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed7e00640 1 -- 172.21.15.27:0/3082579580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed81eb610 con 0x7f4ed80c5e80 2024-03-28T19:26:46.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed7400640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80abd90 0x7f4ed8154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:26:46.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed6a00640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ed80c1f50 0x7f4ed814edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:26:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ed7e00640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c5e80 0x7f4ed814f300 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000c6b0 tx=0x7f4ed000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ecf400640 1 -- 172.21.15.27:0/3082579580 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0003b70 con 0x7f4ed80c5e80 2024-03-28T19:26:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ecf400640 1 -- 172.21.15.27:0/3082579580 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed0003d30 con 0x7f4ed80c5e80 2024-03-28T19:26:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ecf400640 1 -- 172.21.15.27:0/3082579580 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0005310 con 0x7f4ed80c5e80 2024-03-28T19:26:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.166+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed81eb8f0 con 0x7f4ed80c5e80 2024-03-28T19:26:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.167+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed81ebeb0 con 0x7f4ed80c5e80 2024-03-28T19:26:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.168+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed8154e20 con 0x7f4ed80c5e80 2024-03-28T19:26:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.170+0000 7f4ecf400640 1 -- 172.21.15.27:0/3082579580 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4ed00054b0 con 0x7f4ed80c5e80 2024-03-28T19:26:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.170+0000 7f4ecf400640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4eb806bc00 0x7f4eb806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:46.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.171+0000 7f4ecf400640 1 -- 172.21.15.27:0/3082579580 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f4ed008ba70 con 0x7f4ed80c5e80 2024-03-28T19:26:46.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.171+0000 7f4ed7400640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4eb806bc00 0x7f4eb806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:46.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.172+0000 7f4ed7400640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4eb806bc00 0x7f4eb806e0c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8155120 tx=0x7f4ec8008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:46.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.176+0000 7f4ecf400640 1 -- 172.21.15.27:0/3082579580 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4ed8154e20 con 0x7f4ed80c5e80 2024-03-28T19:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.452+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ed804f400 con 0x7f4ed80c5e80 2024-03-28T19:26:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.453+0000 7f4ecf400640 1 -- 172.21.15.27:0/3082579580 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f4ed00593d0 con 0x7f4ed80c5e80 2024-03-28T19:26:46.452 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:46.452 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":8928,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":229}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 229 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.455+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4eb806bc00 msgr2=0x7f4eb806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4eb806bc00 0x7f4eb806e0c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8155120 tx=0x7f4ec8008040 comp rx=0 tx=0).stop 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c5e80 msgr2=0x7f4ed814f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c5e80 0x7f4ed814f300 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000c6b0 tx=0x7f4ed000cb70 comp rx=0 tx=0).stop 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 shutdown_connections 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4eb806bc00 0x7f4eb806e0c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ed80c5e80 0x7f4ed814f300 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ed80c1f50 0x7f4ed814edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 --2- 172.21.15.27:0/3082579580 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ed80abd90 0x7f4ed8154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 >> 172.21.15.27:0/3082579580 conn(0x7f4ed806b9d0 msgr2=0x7f4ed80c08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:46.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 shutdown_connections 2024-03-28T19:26:46.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:46.456+0000 7f4ede600640 1 -- 172.21.15.27:0/3082579580 wait complete. 2024-03-28T19:26:46.475 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:26:47.193 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 22% (40/181) Updating files: 23% (42/181) Updating files: 24% (44/181) Updating files: 25% (46/181) Updating files: 26% (48/181) Updating files: 27% (49/181) Updating files: 28% (51/181) Updating files: 29% (53/181) Updating files: 30% (55/181) Updating files: 31% (57/181) Updating files: 32% (58/181) Updating files: 33% (60/181) Updating files: 34% (62/181) Updating files: 35% (64/181) Updating files: 36% (66/181) Updating files: 37% (67/181) Updating files: 38% (69/181) Updating files: 39% (71/181) Updating files: 39% (72/181) Updating files: 40% (73/181) Updating files: 41% (75/181) Updating files: 42% (77/181) Updating files: 43% (78/181) Updating files: 44% (80/181) Updating files: 45% (82/181) Updating files: 46% (84/181) Updating files: 47% (86/181) Updating files: 48% (87/181) Updating files: 49% (89/181) Updating files: 50% (91/181) Updating files: 51% (93/181) Updating files: 52% (95/181) Updating files: 53% (96/181) Updating files: 54% (98/181) Updating files: 55% (100/181) Updating files: 56% (102/181) Updating files: 57% (104/181) Updating files: 58% (105/181) Updating files: 59% (107/181) Updating files: 60% (109/181) Updating files: 61% (111/181) Updating files: 62% (113/181) Updating files: 63% (115/181) Updating files: 64% (116/181) Updating files: 65% (118/181) Updating files: 66% (120/181) Updating files: 67% (122/181) Updating files: 68% (124/181) Updating files: 69% (125/181) Updating files: 70% (127/181) Updating files: 71% (129/181) Updating files: 72% (131/181) Updating files: 73% (133/181) Updating files: 74% (134/181) Updating files: 75% (136/181) Updating files: 76% (138/181) Updating files: 77% (140/181) Updating files: 78% (142/181) Updating files: 79% (143/181) Updating files: 80% (145/181) Updating files: 81% (147/181) Updating files: 82% (149/181) Updating files: 83% (151/181) Updating files: 84% (153/181) Updating files: 85% (154/181) Updating files: 86% (156/181) Updating files: 87% (158/181) Updating files: 88% (160/181) Updating files: 89% (162/181) Updating files: 90% (163/181) Updating files: 91% (165/181) Updating files: 92% (167/181) Updating files: 93% (169/181) Updating files: 94% (171/181) Updating files: 95% (172/181) Updating files: 96% (174/181) Updating files: 97% (176/181) Updating files: 98% (178/181) Updating files: 99% (180/181) Updating files: 100% (181/181) Updating files: 100% (181/181), done. 2024-03-28T19:26:47.508 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at a6f3bbb744a v0.37 2024-03-28T19:26:47.521 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.37 2024-03-28T19:26:47.616 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:26:47.616 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 38 -eq 48 ']' 2024-03-28T19:26:47.616 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.38 2024-03-28T19:26:47.617 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.38 2024-03-28T19:26:47.617 INFO:tasks.workunit.client.0.smithi027.stdout:v0.38 2024-03-28T19:26:47.617 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.38 2024-03-28T19:26:48.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.736+0000 7f712e600640 1 Processor -- start 2024-03-28T19:26:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.736+0000 7f712e600640 1 -- start start 2024-03-28T19:26:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.737+0000 7f712e600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7128150c70 0x7f71280a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.737+0000 7f712e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71281534c0 0x7f7128153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:48.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.737+0000 7f712e600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71280c3af0 con 0x7f71280c3690 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.737+0000 7f712e600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71280ab4e0 con 0x7f71281534c0 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.737+0000 7f712e600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71280ac1f0 con 0x7f7128150c70 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.737+0000 7f7127e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71281534c0 0x7f7128153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7127e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71281534c0 0x7f7128153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51486/0 (socket says 172.21.15.27:51486) 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7127e00640 1 -- 172.21.15.27:0/1252698332 learned_addr learned my addr 172.21.15.27:0/1252698332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7126a00640 1 --1- 172.21.15.27:0/1252698332 >> v1:172.21.15.27:6789/0 conn(0x7f71280c3690 0x7f71280c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41194/0 (socket says 172.21.15.27:41194) 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7127400640 1 --2- 172.21.15.27:0/1252698332 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7128150c70 0x7f71280a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7127e00640 1 -- 172.21.15.27:0/1252698332 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7128150c70 msgr2=0x7f71280a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7127e00640 1 --2- 172.21.15.27:0/1252698332 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7128150c70 0x7f71280a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7127e00640 1 -- 172.21.15.27:0/1252698332 >> v1:172.21.15.27:6789/0 conn(0x7f71280c3690 legacy=0x7f71280c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7127e00640 1 -- 172.21.15.27:0/1252698332 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71280c3d90 con 0x7f71281534c0 2024-03-28T19:26:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.738+0000 7f7126000640 1 -- 172.21.15.27:0/1252698332 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733627445 0 0) 0x7f71280c3af0 con 0x7f71280c3690 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.739+0000 7f7127e00640 1 --2- 172.21.15.27:0/1252698332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71281534c0 0x7f7128153950 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f7120009870 tx=0x7f712002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd670709189c6ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.740+0000 7f7126000640 1 -- 172.21.15.27:0/1252698332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120035020 con 0x7f71281534c0 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.740+0000 7f7126000640 1 -- 172.21.15.27:0/1252698332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7120031640 con 0x7f71281534c0 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.740+0000 7f7126000640 1 -- 172.21.15.27:0/1252698332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712002d040 con 0x7f71281534c0 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.740+0000 7f712e600640 1 -- 172.21.15.27:0/1252698332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71281534c0 msgr2=0x7f7128153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.740+0000 7f712e600640 1 --2- 172.21.15.27:0/1252698332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71281534c0 0x7f7128153950 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f7120009870 tx=0x7f712002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.741+0000 7f712e600640 1 -- 172.21.15.27:0/1252698332 shutdown_connections 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.741+0000 7f712e600640 1 --2- 172.21.15.27:0/1252698332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71281534c0 0x7f7128153950 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.741+0000 7f712e600640 1 --2- 172.21.15.27:0/1252698332 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7128150c70 0x7f71280a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.741+0000 7f712e600640 1 -- 172.21.15.27:0/1252698332 >> 172.21.15.27:0/1252698332 conn(0x7f712806b9d0 msgr2=0x7f71280c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.741+0000 7f712e600640 1 -- 172.21.15.27:0/1252698332 shutdown_connections 2024-03-28T19:26:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.741+0000 7f712e600640 1 -- 172.21.15.27:0/1252698332 wait complete. 2024-03-28T19:26:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.742+0000 7f712e600640 1 Processor -- start 2024-03-28T19:26:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.742+0000 7f712e600640 1 -- start start 2024-03-28T19:26:48.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.743+0000 7f712e600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71280c3690 0x7f71281df130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:48.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.743+0000 7f712e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7128150c70 0x7f71281df670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:48.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f712e600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71281534c0 0x7f71281dca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:48.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f712e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71280abcd0 con 0x7f7128150c70 2024-03-28T19:26:48.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f712e600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71280ab9f0 con 0x7f71280c3690 2024-03-28T19:26:48.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f712e600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71280c1810 con 0x7f71281534c0 2024-03-28T19:26:48.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7126a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7128150c70 0x7f71281df670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:48.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7126a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7128150c70 0x7f71281df670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51488/0 (socket says 172.21.15.27:51488) 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7126a00640 1 -- 172.21.15.27:0/2690432068 learned_addr learned my addr 172.21.15.27:0/2690432068 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7127400640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71280c3690 0x7f71281df130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7126a00640 1 -- 172.21.15.27:0/2690432068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71281534c0 msgr2=0x7f71281dca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7126a00640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71281534c0 0x7f71281dca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7126a00640 1 -- 172.21.15.27:0/2690432068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71280c3690 msgr2=0x7f71281df130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7126a00640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71280c3690 0x7f71281df130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7126a00640 1 -- 172.21.15.27:0/2690432068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71281eb7b0 con 0x7f7128150c70 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.744+0000 7f7127400640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71280c3690 0x7f71281df130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.745+0000 7f7126a00640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7128150c70 0x7f71281df670 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f711000c500 tx=0x7f711000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.745+0000 7f711f400640 1 -- 172.21.15.27:0/2690432068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7110018070 con 0x7f7128150c70 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.745+0000 7f711f400640 1 -- 172.21.15.27:0/2690432068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7110007580 con 0x7f7128150c70 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.745+0000 7f711f400640 1 -- 172.21.15.27:0/2690432068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7110004a20 con 0x7f7128150c70 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.745+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71281eba90 con 0x7f7128150c70 2024-03-28T19:26:48.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.745+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71280a81e0 con 0x7f7128150c70 2024-03-28T19:26:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.746+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712804f400 con 0x7f7128150c70 2024-03-28T19:26:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.748+0000 7f711f400640 1 -- 172.21.15.27:0/2690432068 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7110004bc0 con 0x7f7128150c70 2024-03-28T19:26:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.749+0000 7f711f400640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f710406bc00 0x7f710406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.750+0000 7f711f400640 1 -- 172.21.15.27:0/2690432068 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f711008ae20 con 0x7f7128150c70 2024-03-28T19:26:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.750+0000 7f7127400640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f710406bc00 0x7f710406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.751+0000 7f7127400640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f710406bc00 0x7f710406e0c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7118007330 tx=0x7f711800b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:48.755+0000 7f711f400640 1 -- 172.21.15.27:0/2690432068 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f71100587e0 con 0x7f7128150c70 2024-03-28T19:26:49.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.010+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71280c5f00 con 0x7f7128150c70 2024-03-28T19:26:49.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.011+0000 7f711f400640 1 -- 172.21.15.27:0/2690432068 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f71100580e0 con 0x7f7128150c70 2024-03-28T19:26:49.011 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:49.011 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":8928,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":229}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:49.011 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 229 2024-03-28T19:26:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f710406bc00 msgr2=0x7f710406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:49.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f710406bc00 0x7f710406e0c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7118007330 tx=0x7f711800b040 comp rx=0 tx=0).stop 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7128150c70 msgr2=0x7f71281df670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7128150c70 0x7f71281df670 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f711000c500 tx=0x7f711000c9c0 comp rx=0 tx=0).stop 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 shutdown_connections 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f710406bc00 0x7f710406e0c0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71281534c0 0x7f71281dca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7128150c70 0x7f71281df670 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 --2- 172.21.15.27:0/2690432068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71280c3690 0x7f71281df130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 >> 172.21.15.27:0/2690432068 conn(0x7f712806b9d0 msgr2=0x7f71280c2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 shutdown_connections 2024-03-28T19:26:49.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:49.014+0000 7f712e600640 1 -- 172.21.15.27:0/2690432068 wait complete. 2024-03-28T19:26:49.035 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:26:51.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:51.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.274+0000 7f649ea00640 1 Processor -- start 2024-03-28T19:26:51.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.274+0000 7f649ea00640 1 -- start start 2024-03-28T19:26:51.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.275+0000 7f649ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f649806c0d0 0x7f649806c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:51.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.275+0000 7f649ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980a6e90 0x7f6498155530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:51.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.275+0000 7f649ea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64980a7400 con 0x7f64980c1780 2024-03-28T19:26:51.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.275+0000 7f649ea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6498159470 con 0x7f649806c0d0 2024-03-28T19:26:51.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.275+0000 7f649ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6498158b40 con 0x7f64980a6e90 2024-03-28T19:26:51.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.275+0000 7f649d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f649806c0d0 0x7f649806c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f649e000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f64980c1780 0x7f6498155a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41210/0 (socket says 172.21.15.27:41210) 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f649e000640 1 -- 172.21.15.27:0/237394020 learned_addr learned my addr 172.21.15.27:0/237394020 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f649cc00640 1 --2- 172.21.15.27:0/237394020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980a6e90 0x7f6498155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f649d600640 1 -- 172.21.15.27:0/237394020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980a6e90 msgr2=0x7f6498155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f649d600640 1 --2- 172.21.15.27:0/237394020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980a6e90 0x7f6498155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f649d600640 1 -- 172.21.15.27:0/237394020 >> v1:172.21.15.27:6789/0 conn(0x7f64980c1780 legacy=0x7f6498155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f6497e00640 1 -- 172.21.15.27:0/237394020 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831648551 0 0) 0x7f64980a7400 con 0x7f64980c1780 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f649d600640 1 -- 172.21.15.27:0/237394020 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498158e20 con 0x7f649806c0d0 2024-03-28T19:26:51.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.276+0000 7f649cc00640 1 --2- 172.21.15.27:0/237394020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980a6e90 0x7f6498155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:26:51.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.278+0000 7f649d600640 1 --2- 172.21.15.27:0/237394020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649806c0d0 0x7f649806c4b0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f64880097a0 tx=0x7f648802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f130be59aba4d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:51.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.278+0000 7f6497e00640 1 -- 172.21.15.27:0/237394020 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64880315a0 con 0x7f649806c0d0 2024-03-28T19:26:51.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.278+0000 7f6497e00640 1 -- 172.21.15.27:0/237394020 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6488037070 con 0x7f649806c0d0 2024-03-28T19:26:51.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.278+0000 7f6497e00640 1 -- 172.21.15.27:0/237394020 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6488031960 con 0x7f649806c0d0 2024-03-28T19:26:51.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.279+0000 7f649ea00640 1 -- 172.21.15.27:0/237394020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649806c0d0 msgr2=0x7f649806c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:51.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.279+0000 7f649ea00640 1 --2- 172.21.15.27:0/237394020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649806c0d0 0x7f649806c4b0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f64880097a0 tx=0x7f648802b840 comp rx=0 tx=0).stop 2024-03-28T19:26:51.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.279+0000 7f649ea00640 1 -- 172.21.15.27:0/237394020 shutdown_connections 2024-03-28T19:26:51.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.279+0000 7f649ea00640 1 --2- 172.21.15.27:0/237394020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980a6e90 0x7f6498155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.279+0000 7f649ea00640 1 --2- 172.21.15.27:0/237394020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f649806c0d0 0x7f649806c4b0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.279+0000 7f649ea00640 1 -- 172.21.15.27:0/237394020 >> 172.21.15.27:0/237394020 conn(0x7f64980c7aa0 msgr2=0x7f64980c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:51.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.279+0000 7f649ea00640 1 -- 172.21.15.27:0/237394020 shutdown_connections 2024-03-28T19:26:51.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.279+0000 7f649ea00640 1 -- 172.21.15.27:0/237394020 wait complete. 2024-03-28T19:26:51.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.281+0000 7f649ea00640 1 Processor -- start 2024-03-28T19:26:51.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.281+0000 7f649ea00640 1 -- start start 2024-03-28T19:26:51.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.282+0000 7f649ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f649806c0d0 0x7f64980c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:51.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.282+0000 7f649ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64980a6e90 0x7f64981ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:51.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980c1780 0x7f649814ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6498152e20 con 0x7f64980a6e90 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6498159900 con 0x7f649806c0d0 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6498152fa0 con 0x7f64980c1780 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64980a6e90 0x7f64981ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64980a6e90 0x7f64981ee8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51508/0 (socket says 172.21.15.27:51508) 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649cc00640 1 -- 172.21.15.27:0/839235875 learned_addr learned my addr 172.21.15.27:0/839235875 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649d600640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f649806c0d0 0x7f64980c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649e000640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980c1780 0x7f649814ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649cc00640 1 -- 172.21.15.27:0/839235875 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980c1780 msgr2=0x7f649814ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649cc00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980c1780 0x7f649814ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649cc00640 1 -- 172.21.15.27:0/839235875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f649806c0d0 msgr2=0x7f64980c60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649cc00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f649806c0d0 0x7f64980c60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.283+0000 7f649cc00640 1 -- 172.21.15.27:0/839235875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64980aadd0 con 0x7f64980a6e90 2024-03-28T19:26:51.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.284+0000 7f649cc00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64980a6e90 0x7f64981ee8c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f648400e7e0 tx=0x7f648400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:51.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.284+0000 7f6495600640 1 -- 172.21.15.27:0/839235875 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648400f040 con 0x7f64980a6e90 2024-03-28T19:26:51.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.284+0000 7f6495600640 1 -- 172.21.15.27:0/839235875 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6484004030 con 0x7f64980a6e90 2024-03-28T19:26:51.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.284+0000 7f6495600640 1 -- 172.21.15.27:0/839235875 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484004400 con 0x7f64980a6e90 2024-03-28T19:26:51.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.284+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6498158340 con 0x7f64980a6e90 2024-03-28T19:26:51.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.284+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64980c6600 con 0x7f64980a6e90 2024-03-28T19:26:51.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.286+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6498155bf0 con 0x7f64980a6e90 2024-03-28T19:26:51.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.287+0000 7f6495600640 1 -- 172.21.15.27:0/839235875 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f648400c6d0 con 0x7f64980a6e90 2024-03-28T19:26:51.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.288+0000 7f6495600640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f647806bbd0 0x7f647806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:51.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.288+0000 7f6495600640 1 -- 172.21.15.27:0/839235875 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f648408b9e0 con 0x7f64980a6e90 2024-03-28T19:26:51.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.288+0000 7f649d600640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f647806bbd0 0x7f647806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:51.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.289+0000 7f649d600640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f647806bbd0 0x7f647806e090 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6488036000 tx=0x7f648803d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:51.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.293+0000 7f6495600640 1 -- 172.21.15.27:0/839235875 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6498155bf0 con 0x7f64980a6e90 2024-03-28T19:26:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.555+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6498069640 con 0x7f64980a6e90 2024-03-28T19:26:51.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.556+0000 7f6495600640 1 -- 172.21.15.27:0/839235875 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f6484017020 con 0x7f64980a6e90 2024-03-28T19:26:51.556 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:51.556 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":8928,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":229}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:51.556 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 229 2024-03-28T19:26:51.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f647806bbd0 msgr2=0x7f647806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:51.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f647806bbd0 0x7f647806e090 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6488036000 tx=0x7f648803d040 comp rx=0 tx=0).stop 2024-03-28T19:26:51.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64980a6e90 msgr2=0x7f64981ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:51.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64980a6e90 0x7f64981ee8c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f648400e7e0 tx=0x7f648400eca0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 shutdown_connections 2024-03-28T19:26:51.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f647806bbd0 0x7f647806e090 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64980c1780 0x7f649814ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64980a6e90 0x7f64981ee8c0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 --2- 172.21.15.27:0/839235875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f649806c0d0 0x7f64980c60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:51.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 >> 172.21.15.27:0/839235875 conn(0x7f64980c7aa0 msgr2=0x7f6498066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:51.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 shutdown_connections 2024-03-28T19:26:51.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:51.559+0000 7f649ea00640 1 -- 172.21.15.27:0/839235875 wait complete. 2024-03-28T19:26:51.580 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:26:53.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:53.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.816+0000 7f3d8ca00640 1 Processor -- start 2024-03-28T19:26:53.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.816+0000 7f3d8ca00640 1 -- start start 2024-03-28T19:26:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d8ca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3d88150c70 0x7f3d880a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d8ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c3690 0x7f3d880c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d8ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d880c6660 0x7f3d881534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d8ca00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d880ab4e0 con 0x7f3d880c6660 2024-03-28T19:26:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d8ca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d880ac1f0 con 0x7f3d88150c70 2024-03-28T19:26:53.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d8ca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880c3af0 con 0x7f3d880c3690 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d86800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d880c6660 0x7f3d881534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d86800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d880c6660 0x7f3d881534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51514/0 (socket says 172.21.15.27:51514) 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d86800640 1 -- 172.21.15.27:0/4226526648 learned_addr learned my addr 172.21.15.27:0/4226526648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.818+0000 7f3d85e00640 1 --2- 172.21.15.27:0/4226526648 >> v2:172.21.15.155:3300/0 conn(0x7f3d88150c70 0x7f3d880a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.819+0000 7f3d85400640 1 --2- 172.21.15.27:0/4226526648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c3690 0x7f3d880c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.819+0000 7f3d86800640 1 -- 172.21.15.27:0/4226526648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c3690 msgr2=0x7f3d880c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.819+0000 7f3d86800640 1 --2- 172.21.15.27:0/4226526648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c3690 0x7f3d880c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.819+0000 7f3d86800640 1 -- 172.21.15.27:0/4226526648 >> v2:172.21.15.155:3300/0 conn(0x7f3d88150c70 msgr2=0x7f3d880a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.819+0000 7f3d86800640 1 --2- 172.21.15.27:0/4226526648 >> v2:172.21.15.155:3300/0 conn(0x7f3d88150c70 0x7f3d880a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:53.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.819+0000 7f3d86800640 1 -- 172.21.15.27:0/4226526648 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d880c3d90 con 0x7f3d880c6660 2024-03-28T19:26:53.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.820+0000 7f3d86800640 1 --2- 172.21.15.27:0/4226526648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d880c6660 0x7f3d881534c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f3d880c7950 tx=0x7f3d8002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8a30a88adafa05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.820+0000 7f3d84a00640 1 -- 172.21.15.27:0/4226526648 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8002c9e0 con 0x7f3d880c6660 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d84a00640 1 -- 172.21.15.27:0/4226526648 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8002cb80 con 0x7f3d880c6660 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d84a00640 1 -- 172.21.15.27:0/4226526648 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d800343f0 con 0x7f3d880c6660 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d8ca00640 1 -- 172.21.15.27:0/4226526648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d880c6660 msgr2=0x7f3d881534c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/4226526648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d880c6660 0x7f3d881534c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f3d880c7950 tx=0x7f3d8002be40 comp rx=0 tx=0).stop 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d8ca00640 1 -- 172.21.15.27:0/4226526648 shutdown_connections 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/4226526648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d880c6660 0x7f3d881534c0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/4226526648 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c3690 0x7f3d880c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/4226526648 >> v2:172.21.15.155:3300/0 conn(0x7f3d88150c70 0x7f3d880a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.821+0000 7f3d8ca00640 1 -- 172.21.15.27:0/4226526648 >> 172.21.15.27:0/4226526648 conn(0x7f3d8806b9d0 msgr2=0x7f3d880c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.822+0000 7f3d8ca00640 1 -- 172.21.15.27:0/4226526648 shutdown_connections 2024-03-28T19:26:53.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.822+0000 7f3d8ca00640 1 -- 172.21.15.27:0/4226526648 wait complete. 2024-03-28T19:26:53.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.823+0000 7f3d8ca00640 1 Processor -- start 2024-03-28T19:26:53.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.824+0000 7f3d8ca00640 1 -- start start 2024-03-28T19:26:53.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.824+0000 7f3d8ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d880c3690 0x7f3d88146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.824+0000 7f3d8ca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c6660 0x7f3d8806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.824+0000 7f3d8ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d88150c70 0x7f3d881eb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.824+0000 7f3d8ca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880abcd0 con 0x7f3d88150c70 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.824+0000 7f3d8ca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880ab9f0 con 0x7f3d880c3690 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.824+0000 7f3d8ca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880c1810 con 0x7f3d880c6660 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d86800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d88150c70 0x7f3d881eb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d86800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d88150c70 0x7f3d881eb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51524/0 (socket says 172.21.15.27:51524) 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d86800640 1 -- 172.21.15.27:0/679195929 learned_addr learned my addr 172.21.15.27:0/679195929 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d85400640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c6660 0x7f3d8806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d85e00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d880c3690 0x7f3d88146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d86800640 1 -- 172.21.15.27:0/679195929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c6660 msgr2=0x7f3d8806e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d86800640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c6660 0x7f3d8806e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:53.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d86800640 1 -- 172.21.15.27:0/679195929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d880c3690 msgr2=0x7f3d88146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d86800640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d880c3690 0x7f3d88146960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.825+0000 7f3d86800640 1 -- 172.21.15.27:0/679195929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88148010 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.826+0000 7f3d86800640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d88150c70 0x7f3d881eb0f0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009e50 tx=0x7f3d80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.826+0000 7f3d7e000640 1 -- 172.21.15.27:0/679195929 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80043020 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.826+0000 7f3d7e000640 1 -- 172.21.15.27:0/679195929 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8003e450 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.826+0000 7f3d7e000640 1 -- 172.21.15.27:0/679195929 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80009510 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.826+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d881ee800 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.826+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d881eedc0 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.827+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d880a99a0 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.828+0000 7f3d7e000640 1 -- 172.21.15.27:0/679195929 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d80046020 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.829+0000 7f3d7e000640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6806bbd0 0x7f3d6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.829+0000 7f3d7e000640 1 -- 172.21.15.27:0/679195929 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f3d800ae380 con 0x7f3d88150c70 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.830+0000 7f3d85e00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6806bbd0 0x7f3d6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:53.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.830+0000 7f3d85e00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6806bbd0 0x7f3d6806e090 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009990 tx=0x7f3d78008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:53.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:53.835+0000 7f3d7e000640 1 -- 172.21.15.27:0/679195929 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d880a99a0 con 0x7f3d88150c70 2024-03-28T19:26:54.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.093+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8804f400 con 0x7f3d88150c70 2024-03-28T19:26:54.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.094+0000 7f3d7e000640 1 -- 172.21.15.27:0/679195929 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f3d8004d030 con 0x7f3d88150c70 2024-03-28T19:26:54.093 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:54.094 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":8928,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":229}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:54.094 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 229 2024-03-28T19:26:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6806bbd0 msgr2=0x7f3d6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6806bbd0 0x7f3d6806e090 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009990 tx=0x7f3d78008040 comp rx=0 tx=0).stop 2024-03-28T19:26:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d88150c70 msgr2=0x7f3d881eb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d88150c70 0x7f3d881eb0f0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009e50 tx=0x7f3d80004a40 comp rx=0 tx=0).stop 2024-03-28T19:26:54.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 shutdown_connections 2024-03-28T19:26:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f3d6806bbd0 0x7f3d6806e090 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d88150c70 0x7f3d881eb0f0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d880c6660 0x7f3d8806e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 --2- 172.21.15.27:0/679195929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d880c3690 0x7f3d88146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 >> 172.21.15.27:0/679195929 conn(0x7f3d8806b9d0 msgr2=0x7f3d8814f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 shutdown_connections 2024-03-28T19:26:54.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:54.097+0000 7f3d8ca00640 1 -- 172.21.15.27:0/679195929 wait complete. 2024-03-28T19:26:54.117 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:26:56.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.367+0000 7f7e93200640 1 Processor -- start 2024-03-28T19:26:56.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.367+0000 7f7e93200640 1 -- start start 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.368+0000 7f7e93200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c150c70 0x7f7e8c0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.368+0000 7f7e93200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e8c0c3690 0x7f7e8c0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.368+0000 7f7e93200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8c0c3af0 con 0x7f7e8c0c6660 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.368+0000 7f7e93200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8c0ab4e0 con 0x7f7e8c0c3690 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.368+0000 7f7e93200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0ac1f0 con 0x7f7e8c150c70 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.369+0000 7f7e83e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e8c0c3690 0x7f7e8c0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.369+0000 7f7e83e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e8c0c3690 0x7f7e8c0c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51870/0 (socket says 172.21.15.27:51870) 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.369+0000 7f7e83e00640 1 -- 172.21.15.27:0/3536510111 learned_addr learned my addr 172.21.15.27:0/3536510111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:56.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.369+0000 7f7e8be00640 1 --2- 172.21.15.27:0/3536510111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c150c70 0x7f7e8c0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.369+0000 7f7e83e00640 1 -- 172.21.15.27:0/3536510111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c150c70 msgr2=0x7f7e8c0a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.369+0000 7f7e83e00640 1 --2- 172.21.15.27:0/3536510111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c150c70 0x7f7e8c0a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.369+0000 7f7e83e00640 1 -- 172.21.15.27:0/3536510111 >> v1:172.21.15.155:6789/0 conn(0x7f7e8c0c6660 legacy=0x7f7e8c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:56.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.369+0000 7f7e83e00640 1 -- 172.21.15.27:0/3536510111 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c0c3d90 con 0x7f7e8c0c3690 2024-03-28T19:26:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.371+0000 7f7e83e00640 1 --2- 172.21.15.27:0/3536510111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c6120 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009870 tx=0x7f7e7402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9aa38fda695c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.371+0000 7f7e8b400640 1 -- 172.21.15.27:0/3536510111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e740317c0 con 0x7f7e8c0c3690 2024-03-28T19:26:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.371+0000 7f7e8b400640 1 -- 172.21.15.27:0/3536510111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e74037070 con 0x7f7e8c0c3690 2024-03-28T19:26:56.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.371+0000 7f7e8b400640 1 -- 172.21.15.27:0/3536510111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74031b80 con 0x7f7e8c0c3690 2024-03-28T19:26:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.372+0000 7f7e93200640 1 -- 172.21.15.27:0/3536510111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 msgr2=0x7f7e8c0c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.372+0000 7f7e93200640 1 --2- 172.21.15.27:0/3536510111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c6120 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009870 tx=0x7f7e7402b860 comp rx=0 tx=0).stop 2024-03-28T19:26:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.372+0000 7f7e93200640 1 -- 172.21.15.27:0/3536510111 shutdown_connections 2024-03-28T19:26:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.372+0000 7f7e93200640 1 --2- 172.21.15.27:0/3536510111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c6120 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.372+0000 7f7e93200640 1 --2- 172.21.15.27:0/3536510111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c150c70 0x7f7e8c0a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.372+0000 7f7e93200640 1 -- 172.21.15.27:0/3536510111 >> 172.21.15.27:0/3536510111 conn(0x7f7e8c06b9d0 msgr2=0x7f7e8c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:56.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.372+0000 7f7e93200640 1 -- 172.21.15.27:0/3536510111 shutdown_connections 2024-03-28T19:26:56.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.372+0000 7f7e93200640 1 -- 172.21.15.27:0/3536510111 wait complete. 2024-03-28T19:26:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.374+0000 7f7e93200640 1 Processor -- start 2024-03-28T19:26:56.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.374+0000 7f7e93200640 1 -- start start 2024-03-28T19:26:56.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e93200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:56.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e93200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c0c6660 0x7f7e8c06cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:56.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e93200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e8c150c70 0x7f7e8c06d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:56.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e93200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0abcd0 con 0x7f7e8c0c3690 2024-03-28T19:26:56.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e93200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0ab9f0 con 0x7f7e8c150c70 2024-03-28T19:26:56.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e93200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0c1810 con 0x7f7e8c0c6660 2024-03-28T19:26:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e8be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e8be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51872/0 (socket says 172.21.15.27:51872) 2024-03-28T19:26:56.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.375+0000 7f7e8be00640 1 -- 172.21.15.27:0/670217849 learned_addr learned my addr 172.21.15.27:0/670217849 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.376+0000 7f7e8be00640 1 -- 172.21.15.27:0/670217849 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c0c6660 msgr2=0x7f7e8c06cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.376+0000 7f7e8be00640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c0c6660 0x7f7e8c06cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.376+0000 7f7e91200640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e8c150c70 0x7f7e8c06d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.376+0000 7f7e8be00640 1 -- 172.21.15.27:0/670217849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e8c150c70 msgr2=0x7f7e8c06d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.376+0000 7f7e8be00640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e8c150c70 0x7f7e8c06d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.376+0000 7f7e8be00640 1 -- 172.21.15.27:0/670217849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c06d7c0 con 0x7f7e8c0c3690 2024-03-28T19:26:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.376+0000 7f7e91200640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e8c150c70 0x7f7e8c06d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:26:56.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.376+0000 7f7e8be00640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c2cb0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00d620 tx=0x7f7e7c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.377+0000 7f7e88c00640 1 -- 172.21.15.27:0/670217849 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c00b620 con 0x7f7e8c0c3690 2024-03-28T19:26:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.377+0000 7f7e88c00640 1 -- 172.21.15.27:0/670217849 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c019070 con 0x7f7e8c0c3690 2024-03-28T19:26:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.377+0000 7f7e88c00640 1 -- 172.21.15.27:0/670217849 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c00ba00 con 0x7f7e8c0c3690 2024-03-28T19:26:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.377+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c1ded80 con 0x7f7e8c0c3690 2024-03-28T19:26:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.377+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c1de2a0 con 0x7f7e8c0c3690 2024-03-28T19:26:56.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.378+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c04f400 con 0x7f7e8c0c3690 2024-03-28T19:26:56.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.379+0000 7f7e88c00640 1 -- 172.21.15.27:0/670217849 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7e7c00bba0 con 0x7f7e8c0c3690 2024-03-28T19:26:56.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.380+0000 7f7e88c00640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e6806bbd0 0x7f7e6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:56.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.380+0000 7f7e88c00640 1 -- 172.21.15.27:0/670217849 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7f7e7c08bfe0 con 0x7f7e8c0c3690 2024-03-28T19:26:56.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.380+0000 7f7e83e00640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e6806bbd0 0x7f7e6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:56.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.381+0000 7f7e83e00640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e6806bbd0 0x7f7e6806e090 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7e74036000 tx=0x7f7e74030000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:56.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.385+0000 7f7e88c00640 1 -- 172.21.15.27:0/670217849 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7e7c059980 con 0x7f7e8c0c3690 2024-03-28T19:26:56.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.651+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e8c0c5f00 con 0x7f7e8c0c3690 2024-03-28T19:26:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.653+0000 7f7e88c00640 1 -- 172.21.15.27:0/670217849 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f7e7c059280 con 0x7f7e8c0c3690 2024-03-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":8928,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":229}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:56.652 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 229 2024-03-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e6806bbd0 msgr2=0x7f7e6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e6806bbd0 0x7f7e6806e090 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7e74036000 tx=0x7f7e74030000 comp rx=0 tx=0).stop 2024-03-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 msgr2=0x7f7e8c0c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c2cb0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00d620 tx=0x7f7e7c00dae0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 shutdown_connections 2024-03-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7e6806bbd0 0x7f7e6806e090 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e8c150c70 0x7f7e8c06d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e8c0c6660 0x7f7e8c06cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 --2- 172.21.15.27:0/670217849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e8c0c3690 0x7f7e8c0c2cb0 secure :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c00d620 tx=0x7f7e7c00dae0 comp rx=0 tx=0).stop 2024-03-28T19:26:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.656+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 >> 172.21.15.27:0/670217849 conn(0x7f7e8c06b9d0 msgr2=0x7f7e8c0c0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.659+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 shutdown_connections 2024-03-28T19:26:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:56.659+0000 7f7e93200640 1 -- 172.21.15.27:0/670217849 wait complete. 2024-03-28T19:26:56.683 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:26:58.423 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 21% (36/171) Updating files: 22% (38/171) Updating files: 23% (40/171) Updating files: 24% (42/171) Updating files: 25% (43/171) Updating files: 26% (45/171) Updating files: 27% (47/171) Updating files: 28% (48/171) Updating files: 29% (50/171) Updating files: 30% (52/171) Updating files: 31% (54/171) Updating files: 32% (55/171) Updating files: 33% (57/171) Updating files: 34% (59/171) Updating files: 35% (60/171) Updating files: 36% (62/171) Updating files: 37% (64/171) Updating files: 38% (65/171) Updating files: 39% (67/171) Updating files: 40% (69/171) Updating files: 41% (71/171) Updating files: 42% (72/171) Updating files: 43% (74/171) Updating files: 44% (76/171) Updating files: 45% (77/171) Updating files: 46% (79/171) Updating files: 47% (81/171) Updating files: 48% (83/171) Updating files: 49% (84/171) Updating files: 50% (86/171) Updating files: 51% (88/171) Updating files: 52% (89/171) Updating files: 53% (91/171) Updating files: 54% (93/171) Updating files: 55% (95/171) Updating files: 56% (96/171) Updating files: 57% (98/171) Updating files: 58% (100/171) Updating files: 59% (101/171) Updating files: 60% (103/171) Updating files: 61% (105/171) Updating files: 62% (107/171) Updating files: 63% (108/171) Updating files: 64% (110/171) Updating files: 65% (112/171) Updating files: 66% (113/171) Updating files: 66% (114/171) Updating files: 67% (115/171) Updating files: 68% (117/171) Updating files: 69% (118/171) Updating files: 70% (120/171) Updating files: 71% (122/171) Updating files: 72% (124/171) Updating files: 73% (125/171) Updating files: 74% (127/171) Updating files: 75% (129/171) Updating files: 76% (130/171) Updating files: 77% (132/171) Updating files: 78% (134/171) Updating files: 79% (136/171) Updating files: 80% (137/171) Updating files: 81% (139/171) Updating files: 82% (141/171) Updating files: 83% (142/171) Updating files: 84% (144/171) Updating files: 85% (146/171) Updating files: 86% (148/171) Updating files: 87% (149/171) Updating files: 88% (151/171) Updating files: 89% (153/171) Updating files: 90% (154/171) Updating files: 91% (156/171) Updating files: 92% (158/171) Updating files: 93% (160/171) Updating files: 94% (161/171) Updating files: 95% (163/171) Updating files: 96% (165/171) Updating files: 97% (166/171) Updating files: 98% (168/171) Updating files: 99% (170/171) Updating files: 100% (171/171) Updating files: 100% (171/171), done. 2024-03-28T19:26:58.684 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:26:58.805 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at b600ec2ac7c v0.38 2024-03-28T19:26:58.813 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.38 2024-03-28T19:26:58.906 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:26:58.906 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 39 -eq 48 ']' 2024-03-28T19:26:58.906 INFO:tasks.workunit.client.0.smithi027.stdout:v0.39 2024-03-28T19:26:58.907 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.39 2024-03-28T19:26:58.907 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.39 2024-03-28T19:26:58.907 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.39 2024-03-28T19:26:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.908+0000 7fb5a3c00640 1 Processor -- start 2024-03-28T19:26:58.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.908+0000 7fb5a3c00640 1 -- start start 2024-03-28T19:26:58.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a3c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c150c70 0x7fb59c0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:58.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a3c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb59c1534c0 0x7fb59c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:58.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a3c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c0c3af0 con 0x7fb59c0c3690 2024-03-28T19:26:58.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a3c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59c0ab4e0 con 0x7fb59c1534c0 2024-03-28T19:26:58.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a3c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0ac1f0 con 0x7fb59c150c70 2024-03-28T19:26:58.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a1c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb59c1534c0 0x7fb59c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:58.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a1200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c150c70 0x7fb59c0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:58.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a1c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb59c1534c0 0x7fb59c153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:51218/0 (socket says 172.21.15.27:51218) 2024-03-28T19:26:58.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.910+0000 7fb5a1c00640 1 -- 172.21.15.27:0/443552382 learned_addr learned my addr 172.21.15.27:0/443552382 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:58.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.911+0000 7fb5a1c00640 1 -- 172.21.15.27:0/443552382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c150c70 msgr2=0x7fb59c0a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:26:58.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.911+0000 7fb5a1c00640 1 --2- 172.21.15.27:0/443552382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c150c70 0x7fb59c0a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:58.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.911+0000 7fb5a1c00640 1 -- 172.21.15.27:0/443552382 >> v1:172.21.15.155:6789/0 conn(0x7fb59c0c3690 legacy=0x7fb59c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:58.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.911+0000 7fb5a1c00640 1 -- 172.21.15.27:0/443552382 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c0c3d90 con 0x7fb59c1534c0 2024-03-28T19:26:58.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.912+0000 7fb5a1c00640 1 --2- 172.21.15.27:0/443552382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 0x7fb59c153950 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb594009e50 tx=0x7fb59402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74a0c1747978490d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:58.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.913+0000 7fb59b400640 1 -- 172.21.15.27:0/443552382 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59402ca70 con 0x7fb59c1534c0 2024-03-28T19:26:58.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.913+0000 7fb59b400640 1 -- 172.21.15.27:0/443552382 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59402cc10 con 0x7fb59c1534c0 2024-03-28T19:26:58.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.913+0000 7fb59b400640 1 -- 172.21.15.27:0/443552382 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5940343f0 con 0x7fb59c1534c0 2024-03-28T19:26:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.914+0000 7fb5a3c00640 1 -- 172.21.15.27:0/443552382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 msgr2=0x7fb59c153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.914+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/443552382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 0x7fb59c153950 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb594009e50 tx=0x7fb59402bed0 comp rx=0 tx=0).stop 2024-03-28T19:26:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.914+0000 7fb5a3c00640 1 -- 172.21.15.27:0/443552382 shutdown_connections 2024-03-28T19:26:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.914+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/443552382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 0x7fb59c153950 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.914+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/443552382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c150c70 0x7fb59c0a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.914+0000 7fb5a3c00640 1 -- 172.21.15.27:0/443552382 >> 172.21.15.27:0/443552382 conn(0x7fb59c06b9d0 msgr2=0x7fb59c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.914+0000 7fb5a3c00640 1 -- 172.21.15.27:0/443552382 shutdown_connections 2024-03-28T19:26:58.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.914+0000 7fb5a3c00640 1 -- 172.21.15.27:0/443552382 wait complete. 2024-03-28T19:26:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.916+0000 7fb5a3c00640 1 Processor -- start 2024-03-28T19:26:58.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.916+0000 7fb5a3c00640 1 -- start start 2024-03-28T19:26:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb5a3c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c0c3690 0x7fb59c1ec030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb5a3c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb59c150c70 0x7fb59c14dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb5a3c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 0x7fb59c0c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb5a3c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0abcd0 con 0x7fb59c150c70 2024-03-28T19:26:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb5a3c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0ab9f0 con 0x7fb59c1534c0 2024-03-28T19:26:58.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb5a3c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0c1810 con 0x7fb59c0c3690 2024-03-28T19:26:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb59be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb59c150c70 0x7fb59c14dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb59be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb59c150c70 0x7fb59c14dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51876/0 (socket says 172.21.15.27:51876) 2024-03-28T19:26:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb59be00640 1 -- 172.21.15.27:0/477272244 learned_addr learned my addr 172.21.15.27:0/477272244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:26:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb5a1200640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c0c3690 0x7fb59c1ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:58.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.917+0000 7fb5a1c00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 0x7fb59c0c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.918+0000 7fb59be00640 1 -- 172.21.15.27:0/477272244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c0c3690 msgr2=0x7fb59c1ec030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.918+0000 7fb59be00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c0c3690 0x7fb59c1ec030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.918+0000 7fb59be00640 1 -- 172.21.15.27:0/477272244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 msgr2=0x7fb59c0c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.918+0000 7fb59be00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 0x7fb59c0c2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:58.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.918+0000 7fb59be00640 1 -- 172.21.15.27:0/477272244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c1df690 con 0x7fb59c150c70 2024-03-28T19:26:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.918+0000 7fb59be00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb59c150c70 0x7fb59c14dc90 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fb58800c3c0 tx=0x7fb58800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.918+0000 7fb598c00640 1 -- 172.21.15.27:0/477272244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5880052d0 con 0x7fb59c150c70 2024-03-28T19:26:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.919+0000 7fb598c00640 1 -- 172.21.15.27:0/477272244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb588005490 con 0x7fb59c150c70 2024-03-28T19:26:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.919+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c157350 con 0x7fb59c150c70 2024-03-28T19:26:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.919+0000 7fb598c00640 1 -- 172.21.15.27:0/477272244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588007260 con 0x7fb59c150c70 2024-03-28T19:26:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.919+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c0c3240 con 0x7fb59c150c70 2024-03-28T19:26:58.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.920+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59c157870 con 0x7fb59c150c70 2024-03-28T19:26:58.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.921+0000 7fb598c00640 1 -- 172.21.15.27:0/477272244 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb588007400 con 0x7fb59c150c70 2024-03-28T19:26:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.922+0000 7fb598c00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb57c06bbd0 0x7fb57c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:26:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.922+0000 7fb598c00640 1 -- 172.21.15.27:0/477272244 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 10163+0+0 (secure 0 0 0) 0x7fb58808afd0 con 0x7fb59c150c70 2024-03-28T19:26:58.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.922+0000 7fb5a1200640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb57c06bbd0 0x7fb57c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:26:58.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.923+0000 7fb5a1200640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb57c06bbd0 0x7fb57c06e090 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb590009ad0 tx=0x7fb590008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:26:58.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:58.927+0000 7fb598c00640 1 -- 172.21.15.27:0/477272244 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb59c157870 con 0x7fb59c150c70 2024-03-28T19:26:59.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.206+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb59c1dfcf0 con 0x7fb59c150c70 2024-03-28T19:26:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.206+0000 7fb598c00640 1 -- 172.21.15.27:0/477272244 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fb5880582e0 con 0x7fb59c150c70 2024-03-28T19:26:59.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:26:59.206 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":8928,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":229}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:26:17.018532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6467":{"gid":6467,"name":"d","rank":0,"incarnation":222,"state":"up:active","state_seq":83,"addr":"172.21.15.155:6834/3334295790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3334295790},{"type":"v1","addr":"172.21.15.155:6834","nonce":3334295790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6467,"qdb_cluster":[6467]},"id":2}]} 2024-03-28T19:26:59.206 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 229 2024-03-28T19:26:59.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb57c06bbd0 msgr2=0x7fb57c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:59.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb57c06bbd0 0x7fb57c06e090 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb590009ad0 tx=0x7fb590008040 comp rx=0 tx=0).stop 2024-03-28T19:26:59.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb59c150c70 msgr2=0x7fb59c14dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:26:59.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb59c150c70 0x7fb59c14dc90 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fb58800c3c0 tx=0x7fb58800c880 comp rx=0 tx=0).stop 2024-03-28T19:26:59.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 shutdown_connections 2024-03-28T19:26:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fb57c06bbd0 0x7fb57c06e090 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb59c1534c0 0x7fb59c0c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb59c150c70 0x7fb59c14dc90 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 --2- 172.21.15.27:0/477272244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb59c0c3690 0x7fb59c1ec030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:26:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 >> 172.21.15.27:0/477272244 conn(0x7fb59c06b9d0 msgr2=0x7fb59c152950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:26:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 shutdown_connections 2024-03-28T19:26:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:26:59.210+0000 7fb5a3c00640 1 -- 172.21.15.27:0/477272244 wait complete. 2024-03-28T19:26:59.228 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:27:01.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:01.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.449+0000 7f848f200640 1 Processor -- start 2024-03-28T19:27:01.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.449+0000 7f848f200640 1 -- start start 2024-03-28T19:27:01.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.450+0000 7f848f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c8f00 0x7f849014f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:01.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.450+0000 7f848f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84900c9350 con 0x7f84900727e0 2024-03-28T19:27:01.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.450+0000 7f848f200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8490155d10 con 0x7f84900c59d0 2024-03-28T19:27:01.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.450+0000 7f848f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84901565a0 con 0x7f84900c8f00 2024-03-28T19:27:01.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.450+0000 7f848d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f84900727e0 0x7f8490155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52460/0 (socket says 172.21.15.27:52460) 2024-03-28T19:27:01.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.450+0000 7f848d400640 1 -- 172.21.15.27:0/2475169813 learned_addr learned my addr 172.21.15.27:0/2475169813 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:01.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848e800640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c8f00 0x7f849014f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301333411 0 0) 0x7f84900c9350 con 0x7f84900727e0 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8474003520 con 0x7f84900727e0 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154147567 0 0) 0x7f8474003520 con 0x7f84900727e0 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c8f00 msgr2=0x7f849014f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c8f00 0x7f849014f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 >> v1:172.21.15.155:6789/0 conn(0x7f84900c59d0 legacy=0x7f8490159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490155fe0 con 0x7f84900727e0 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f847c001670 con 0x7f84900727e0 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 >> v1:172.21.15.27:6789/0 conn(0x7f84900727e0 legacy=0x7f8490155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c8f00 0x7f84740039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8474004010 0x7f8474004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8474004d90 0x7f8474005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:01.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84901565a0 con 0x7f8474004010 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847400bca0 con 0x7f8474004d90 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.451+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847400bea0 con 0x7f84900c8f00 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.452+0000 7f848d400640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8474004010 0x7f8474004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.452+0000 7f848e800640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8474004d90 0x7f8474005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.452+0000 7f848de00640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c8f00 0x7f84740039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.452+0000 7f848d400640 1 -- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c8f00 msgr2=0x7f84740039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.452+0000 7f848d400640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c8f00 0x7f84740039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.452+0000 7f848d400640 1 -- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8474004d90 msgr2=0x7f8474005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.452+0000 7f848d400640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8474004d90 0x7f8474005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.452+0000 7f848d400640 1 -- 172.21.15.27:0/2475169813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8474003520 con 0x7f8474004010 2024-03-28T19:27:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.454+0000 7f848d400640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8474004010 0x7f8474004800 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f847c001eb0 tx=0x7f8474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb41c4d768d3c97 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:01.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.454+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c007d30 con 0x7f8474004010 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.454+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c030070 con 0x7f8474004010 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.454+0000 7f848ca00640 1 -- 172.21.15.27:0/2475169813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c005d70 con 0x7f8474004010 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.455+0000 7f848f200640 1 -- 172.21.15.27:0/2475169813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8474004010 msgr2=0x7f8474004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.455+0000 7f848f200640 1 --2- 172.21.15.27:0/2475169813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8474004010 0x7f8474004800 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f847c001eb0 tx=0x7f8474005830 comp rx=0 tx=0).stop 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.455+0000 7f848e800640 1 -- 172.21.15.27:0/2475169813 reap_dead start 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.455+0000 7f848f200640 1 -- 172.21.15.27:0/2475169813 shutdown_connections 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.455+0000 7f848f200640 1 -- 172.21.15.27:0/2475169813 >> 172.21.15.27:0/2475169813 conn(0x7f84900ac910 msgr2=0x7f84900accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.455+0000 7f848f200640 1 -- 172.21.15.27:0/2475169813 shutdown_connections 2024-03-28T19:27:01.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.455+0000 7f848f200640 1 -- 172.21.15.27:0/2475169813 wait complete. 2024-03-28T19:27:01.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.458+0000 7f848f200640 1 Processor -- start 2024-03-28T19:27:01.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.458+0000 7f848f200640 1 -- start start 2024-03-28T19:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.459+0000 7f848f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84901eeb00 0x7f84901eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.459+0000 7f848f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84901434c0 0x7f84900c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.459+0000 7f848f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c42a0 0x7f8490149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.459+0000 7f848f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8490158d20 con 0x7f84901eeb00 2024-03-28T19:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.459+0000 7f848f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8490159090 con 0x7f84901434c0 2024-03-28T19:27:01.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.459+0000 7f848f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84900c5db0 con 0x7f84900c42a0 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84901eeb00 0x7f84901eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84901eeb00 0x7f84901eeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51898/0 (socket says 172.21.15.27:51898) 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848de00640 1 -- 172.21.15.27:0/3137066959 learned_addr learned my addr 172.21.15.27:0/3137066959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848e800640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c42a0 0x7f8490149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848de00640 1 -- 172.21.15.27:0/3137066959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c42a0 msgr2=0x7f8490149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848de00640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c42a0 0x7f8490149eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848de00640 1 -- 172.21.15.27:0/3137066959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84901434c0 msgr2=0x7f84900c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848de00640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84901434c0 0x7f84900c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.460+0000 7f848de00640 1 -- 172.21.15.27:0/3137066959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490143940 con 0x7f84901eeb00 2024-03-28T19:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.461+0000 7f848de00640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84901eeb00 0x7f84901eeee0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f848000e580 tx=0x7f848000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.461+0000 7f8486000640 1 -- 172.21.15.27:0/3137066959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84800052e0 con 0x7f84901eeb00 2024-03-28T19:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.461+0000 7f8486000640 1 -- 172.21.15.27:0/3137066959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8480018070 con 0x7f84901eeb00 2024-03-28T19:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.461+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8490144680 con 0x7f84901eeb00 2024-03-28T19:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.461+0000 7f8486000640 1 -- 172.21.15.27:0/3137066959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8480003810 con 0x7f84901eeb00 2024-03-28T19:27:01.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.462+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8490144c40 con 0x7f84901eeb00 2024-03-28T19:27:01.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.463+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8490067dd0 con 0x7f84901eeb00 2024-03-28T19:27:01.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.465+0000 7f8486000640 1 -- 172.21.15.27:0/3137066959 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f84800039b0 con 0x7f84901eeb00 2024-03-28T19:27:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.466+0000 7f8486000640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847006bc00 0x7f847006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.466+0000 7f8486000640 1 -- 172.21.15.27:0/3137066959 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f848008b8a0 con 0x7f84901eeb00 2024-03-28T19:27:01.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.466+0000 7f848d400640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847006bc00 0x7f847006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.467+0000 7f848d400640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847006bc00 0x7f847006e0c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f847c034000 tx=0x7f847c007000 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.476+0000 7f8486000640 1 -- 172.21.15.27:0/3137066959 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8480059190 con 0x7f84901eeb00 2024-03-28T19:27:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.805+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8490156b90 con 0x7f84901eeb00 2024-03-28T19:27:01.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.806+0000 7f8486000640 1 -- 172.21.15.27:0/3137066959 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f8480058a90 con 0x7f84901eeb00 2024-03-28T19:27:01.805 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:01.805 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:00.494514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:01.805 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 230 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847006bc00 msgr2=0x7f847006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847006bc00 0x7f847006e0c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f847c034000 tx=0x7f847c007000 comp rx=0 tx=0).stop 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84901eeb00 msgr2=0x7f84901eeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84901eeb00 0x7f84901eeee0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f848000e580 tx=0x7f848000ea40 comp rx=0 tx=0).stop 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 shutdown_connections 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f847006bc00 0x7f847006e0c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84900c42a0 0x7f8490149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84901434c0 0x7f84900c3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 --2- 172.21.15.27:0/3137066959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84901eeb00 0x7f84901eeee0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 >> 172.21.15.27:0/3137066959 conn(0x7f84900ac910 msgr2=0x7f84900ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 shutdown_connections 2024-03-28T19:27:01.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:01.809+0000 7f848f200640 1 -- 172.21.15.27:0/3137066959 wait complete. 2024-03-28T19:27:01.828 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-03-28T19:27:01.828 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:27:01.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.030+0000 7fcb4f000640 1 Processor -- start 2024-03-28T19:27:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.030+0000 7fcb4f000640 1 -- start start 2024-03-28T19:27:02.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.032+0000 7fcb4f000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb48157070 0x7fcb480c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:02.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.032+0000 7fcb4f000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcb48070600 0x7fcb480aab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:02.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.032+0000 7fcb4f000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4815ce50 con 0x7fcb480ab050 2024-03-28T19:27:02.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.032+0000 7fcb4f000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb480c64b0 con 0x7fcb48070600 2024-03-28T19:27:02.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.032+0000 7fcb4f000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4815d0f0 con 0x7fcb48157070 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.033+0000 7fcb4e600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcb480ab050 0x7fcb48159860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52474/0 (socket says 172.21.15.27:52474) 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.033+0000 7fcb4e600640 1 -- 172.21.15.27:0/1336340020 learned_addr learned my addr 172.21.15.27:0/1336340020 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.033+0000 7fcb4d200640 1 --2- 172.21.15.27:0/1336340020 >> v2:172.21.15.155:3300/0 conn(0x7fcb48070600 0x7fcb480aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.033+0000 7fcb4dc00640 1 --2- 172.21.15.27:0/1336340020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb48157070 0x7fcb480c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.033+0000 7fcb47e00640 1 -- 172.21.15.27:0/1336340020 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884775462 0 0) 0x7fcb4815ce50 con 0x7fcb480ab050 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.033+0000 7fcb47e00640 1 -- 172.21.15.27:0/1336340020 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb30003520 con 0x7fcb480ab050 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.034+0000 7fcb4dc00640 1 -- 172.21.15.27:0/1336340020 >> v2:172.21.15.155:3300/0 conn(0x7fcb48070600 msgr2=0x7fcb480aab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.034+0000 7fcb4dc00640 1 --2- 172.21.15.27:0/1336340020 >> v2:172.21.15.155:3300/0 conn(0x7fcb48070600 0x7fcb480aab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.034+0000 7fcb4dc00640 1 -- 172.21.15.27:0/1336340020 >> v1:172.21.15.27:6789/0 conn(0x7fcb480ab050 legacy=0x7fcb48159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.034+0000 7fcb4dc00640 1 -- 172.21.15.27:0/1336340020 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4815d3d0 con 0x7fcb48157070 2024-03-28T19:27:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.034+0000 7fcb4d200640 1 --2- 172.21.15.27:0/1336340020 >> v2:172.21.15.155:3300/0 conn(0x7fcb48070600 0x7fcb480aab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:27:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.035+0000 7fcb4dc00640 1 --2- 172.21.15.27:0/1336340020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb48157070 0x7fcb480c9420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0097d0 tx=0x7fcb3c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39600192368f2a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:02.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.036+0000 7fcb47e00640 1 -- 172.21.15.27:0/1336340020 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c031780 con 0x7fcb48157070 2024-03-28T19:27:02.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.036+0000 7fcb47e00640 1 -- 172.21.15.27:0/1336340020 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c037070 con 0x7fcb48157070 2024-03-28T19:27:02.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.036+0000 7fcb47e00640 1 -- 172.21.15.27:0/1336340020 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c031b20 con 0x7fcb48157070 2024-03-28T19:27:02.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.037+0000 7fcb4f000640 1 -- 172.21.15.27:0/1336340020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb48157070 msgr2=0x7fcb480c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:02.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.037+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1336340020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb48157070 0x7fcb480c9420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0097d0 tx=0x7fcb3c02b820 comp rx=0 tx=0).stop 2024-03-28T19:27:02.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.037+0000 7fcb4f000640 1 -- 172.21.15.27:0/1336340020 shutdown_connections 2024-03-28T19:27:02.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.037+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1336340020 >> v2:172.21.15.155:3300/0 conn(0x7fcb48070600 0x7fcb480aab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.037+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1336340020 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb48157070 0x7fcb480c9420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.037+0000 7fcb4f000640 1 -- 172.21.15.27:0/1336340020 >> 172.21.15.27:0/1336340020 conn(0x7fcb48072bd0 msgr2=0x7fcb48072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:02.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.037+0000 7fcb4f000640 1 -- 172.21.15.27:0/1336340020 shutdown_connections 2024-03-28T19:27:02.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.037+0000 7fcb4f000640 1 -- 172.21.15.27:0/1336340020 wait complete. 2024-03-28T19:27:02.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.039+0000 7fcb4f000640 1 Processor -- start 2024-03-28T19:27:02.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.039+0000 7fcb4f000640 1 -- start start 2024-03-28T19:27:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.041+0000 7fcb4f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48070600 0x7fcb48157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.041+0000 7fcb4f000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb480ab050 0x7fcb481e55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.041+0000 7fcb4f000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb48157070 0x7fcb4815bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.041+0000 7fcb4f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb48159f70 con 0x7fcb48070600 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.041+0000 7fcb4f000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4815d990 con 0x7fcb48157070 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.041+0000 7fcb4f000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4815a0f0 con 0x7fcb480ab050 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.041+0000 7fcb4dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48070600 0x7fcb48157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48070600 0x7fcb48157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51914/0 (socket says 172.21.15.27:51914) 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4dc00640 1 -- 172.21.15.27:0/1853292353 learned_addr learned my addr 172.21.15.27:0/1853292353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4e600640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb48157070 0x7fcb4815bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4d200640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb480ab050 0x7fcb481e55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4dc00640 1 -- 172.21.15.27:0/1853292353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb480ab050 msgr2=0x7fcb481e55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4dc00640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb480ab050 0x7fcb481e55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4dc00640 1 -- 172.21.15.27:0/1853292353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb48157070 msgr2=0x7fcb4815bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4dc00640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb48157070 0x7fcb4815bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.042+0000 7fcb4dc00640 1 -- 172.21.15.27:0/1853292353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb481e3a00 con 0x7fcb48070600 2024-03-28T19:27:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.043+0000 7fcb4dc00640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48070600 0x7fcb48157680 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c000c00 tx=0x7fcb3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.043+0000 7fcb45600640 1 -- 172.21.15.27:0/1853292353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c031780 con 0x7fcb48070600 2024-03-28T19:27:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.043+0000 7fcb45600640 1 -- 172.21.15.27:0/1853292353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c031920 con 0x7fcb48070600 2024-03-28T19:27:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.043+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb481e2f60 con 0x7fcb48070600 2024-03-28T19:27:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.043+0000 7fcb45600640 1 -- 172.21.15.27:0/1853292353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c031cd0 con 0x7fcb48070600 2024-03-28T19:27:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.043+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb481e5ae0 con 0x7fcb48070600 2024-03-28T19:27:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.044+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb48069640 con 0x7fcb48070600 2024-03-28T19:27:02.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.046+0000 7fcb45600640 1 -- 172.21.15.27:0/1853292353 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcb3c032410 con 0x7fcb48070600 2024-03-28T19:27:02.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.048+0000 7fcb45600640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb2806bbd0 0x7fcb2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:02.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.048+0000 7fcb45600640 1 -- 172.21.15.27:0/1853292353 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7fcb3c0ae070 con 0x7fcb48070600 2024-03-28T19:27:02.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.048+0000 7fcb4d200640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb2806bbd0 0x7fcb2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:02.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.049+0000 7fcb4d200640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb2806bbd0 0x7fcb2806e090 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcb4806c240 tx=0x7fcb34009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:02.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.057+0000 7fcb45600640 1 -- 172.21.15.27:0/1853292353 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcb3c02f050 con 0x7fcb48070600 2024-03-28T19:27:02.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.330+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb4815a4c0 con 0x7fcb48070600 2024-03-28T19:27:02.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.331+0000 7fcb45600640 1 -- 172.21.15.27:0/1853292353 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fcb3c07b2a0 con 0x7fcb48070600 2024-03-28T19:27:02.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:02.331 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:00.494514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:02.332 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 230 2024-03-28T19:27:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.334+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb2806bbd0 msgr2=0x7fcb2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.334+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb2806bbd0 0x7fcb2806e090 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcb4806c240 tx=0x7fcb34009190 comp rx=0 tx=0).stop 2024-03-28T19:27:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.334+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48070600 msgr2=0x7fcb48157680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48070600 0x7fcb48157680 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c000c00 tx=0x7fcb3c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 shutdown_connections 2024-03-28T19:27:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcb2806bbd0 0x7fcb2806e090 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb48157070 0x7fcb4815bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb480ab050 0x7fcb481e55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 --2- 172.21.15.27:0/1853292353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb48070600 0x7fcb48157680 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:02.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 >> 172.21.15.27:0/1853292353 conn(0x7fcb48072bd0 msgr2=0x7fcb4806b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:02.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 shutdown_connections 2024-03-28T19:27:02.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:02.335+0000 7fcb4f000640 1 -- 172.21.15.27:0/1853292353 wait complete. 2024-03-28T19:27:02.360 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 230, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 226, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6740": { "addr": "172.21.15.155:6835/1734256169", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1734256169, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1734256169, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6740, "incarnation": 222, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 92, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:26:16.003326+0000", "qdb_cluster": [ 6740 ], "qdb_leader": 6740, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6740 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 230, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8928": { "addr": "172.21.15.27:6837/1262510798", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 1262510798, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 1262510798, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8928, "incarnation": 230, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 93, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:27:00.494514+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8928 } } } ], "standbys": [ { "addr": "172.21.15.27:6835/796801387", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 796801387, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 796801387, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 228, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8913, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:27:04.362 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:04.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.582+0000 7fec02800640 1 Processor -- start 2024-03-28T19:27:04.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.583+0000 7fec02800640 1 -- start start 2024-03-28T19:27:04.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7fec02800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7febfc153ed0 0x7febfc06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:04.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7fec02800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc06e6f0 0x7febfc06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7fec02800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7febfc159bc0 0x7febfc159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7fec02800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febfc0c6990 con 0x7febfc159bc0 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7fec02800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febfc0c3830 con 0x7febfc153ed0 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7fec02800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc06d1f0 con 0x7febfc06e6f0 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7febfbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7febfc159bc0 0x7febfc159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7febfbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7febfc159bc0 0x7febfc159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51926/0 (socket says 172.21.15.27:51926) 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7febfbe00640 1 -- 172.21.15.27:0/939362171 learned_addr learned my addr 172.21.15.27:0/939362171 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7febfaa00640 1 --2- 172.21.15.27:0/939362171 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc06e6f0 0x7febfc06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.584+0000 7febfb400640 1 --2- 172.21.15.27:0/939362171 >> v2:172.21.15.155:3300/0 conn(0x7febfc153ed0 0x7febfc06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.585+0000 7febfbe00640 1 -- 172.21.15.27:0/939362171 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc06e6f0 msgr2=0x7febfc06eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.585+0000 7febfbe00640 1 --2- 172.21.15.27:0/939362171 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc06e6f0 0x7febfc06eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.585+0000 7febfbe00640 1 -- 172.21.15.27:0/939362171 >> v2:172.21.15.155:3300/0 conn(0x7febfc153ed0 msgr2=0x7febfc06ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.585+0000 7febfbe00640 1 --2- 172.21.15.27:0/939362171 >> v2:172.21.15.155:3300/0 conn(0x7febfc153ed0 0x7febfc06ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.585+0000 7febfbe00640 1 -- 172.21.15.27:0/939362171 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc0c66b0 con 0x7febfc159bc0 2024-03-28T19:27:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.585+0000 7febfaa00640 1 --2- 172.21.15.27:0/939362171 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc06e6f0 0x7febfc06eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:27:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.587+0000 7febfbe00640 1 --2- 172.21.15.27:0/939362171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc159bc0 0x7febfc159ff0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7febfc0c5ca0 tx=0x7febf402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d313ce2827d336 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.587+0000 7febfa000640 1 -- 172.21.15.27:0/939362171 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf402ca80 con 0x7febfc159bc0 2024-03-28T19:27:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.587+0000 7febfa000640 1 -- 172.21.15.27:0/939362171 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf402cc20 con 0x7febfc159bc0 2024-03-28T19:27:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.587+0000 7febfa000640 1 -- 172.21.15.27:0/939362171 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf40343f0 con 0x7febfc159bc0 2024-03-28T19:27:04.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.588+0000 7fec02800640 1 -- 172.21.15.27:0/939362171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc159bc0 msgr2=0x7febfc159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:04.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.588+0000 7fec02800640 1 --2- 172.21.15.27:0/939362171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc159bc0 0x7febfc159ff0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7febfc0c5ca0 tx=0x7febf402bee0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.589+0000 7fec02800640 1 -- 172.21.15.27:0/939362171 shutdown_connections 2024-03-28T19:27:04.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.589+0000 7fec02800640 1 --2- 172.21.15.27:0/939362171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc159bc0 0x7febfc159ff0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.589+0000 7fec02800640 1 --2- 172.21.15.27:0/939362171 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc06e6f0 0x7febfc06eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.589+0000 7fec02800640 1 --2- 172.21.15.27:0/939362171 >> v2:172.21.15.155:3300/0 conn(0x7febfc153ed0 0x7febfc06ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.589+0000 7fec02800640 1 -- 172.21.15.27:0/939362171 >> 172.21.15.27:0/939362171 conn(0x7febfc06b9d0 msgr2=0x7febfc0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:04.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.589+0000 7fec02800640 1 -- 172.21.15.27:0/939362171 shutdown_connections 2024-03-28T19:27:04.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.589+0000 7fec02800640 1 -- 172.21.15.27:0/939362171 wait complete. 2024-03-28T19:27:04.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.590+0000 7fec02800640 1 Processor -- start 2024-03-28T19:27:04.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.591+0000 7fec02800640 1 -- start start 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.591+0000 7fec02800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febfc06e6f0 0x7febfc1f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.591+0000 7fec02800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc153ed0 0x7febfc0aad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7fec02800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc159bc0 0x7febfc15b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7fec02800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc0c6380 con 0x7febfc153ed0 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7fec02800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc0c3db0 con 0x7febfc06e6f0 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7fec02800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc0c1160 con 0x7febfc159bc0 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc153ed0 0x7febfc0aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc153ed0 0x7febfc0aad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51938/0 (socket says 172.21.15.27:51938) 2024-03-28T19:27:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfaa00640 1 -- 172.21.15.27:0/259148834 learned_addr learned my addr 172.21.15.27:0/259148834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfb400640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febfc06e6f0 0x7febfc1f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfbe00640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc159bc0 0x7febfc15b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfaa00640 1 -- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc159bc0 msgr2=0x7febfc15b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfaa00640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc159bc0 0x7febfc15b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfaa00640 1 -- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febfc06e6f0 msgr2=0x7febfc1f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfaa00640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febfc06e6f0 0x7febfc1f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.592+0000 7febfaa00640 1 -- 172.21.15.27:0/259148834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc1f3940 con 0x7febfc153ed0 2024-03-28T19:27:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.593+0000 7febfbe00640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc159bc0 0x7febfc15b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:27:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.593+0000 7febfaa00640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc153ed0 0x7febfc0aad30 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7febe400e7e0 tx=0x7febe400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.593+0000 7febf3400640 1 -- 172.21.15.27:0/259148834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe400f040 con 0x7febfc153ed0 2024-03-28T19:27:04.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.593+0000 7febf3400640 1 -- 172.21.15.27:0/259148834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe4004030 con 0x7febfc153ed0 2024-03-28T19:27:04.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.593+0000 7febf3400640 1 -- 172.21.15.27:0/259148834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe4004400 con 0x7febfc153ed0 2024-03-28T19:27:04.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.593+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febfc15d6a0 con 0x7febfc153ed0 2024-03-28T19:27:04.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.594+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febfc15dc20 con 0x7febfc153ed0 2024-03-28T19:27:04.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.595+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febfc04f400 con 0x7febfc153ed0 2024-03-28T19:27:04.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.596+0000 7febf3400640 1 -- 172.21.15.27:0/259148834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7febe400c6d0 con 0x7febfc153ed0 2024-03-28T19:27:04.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.598+0000 7febf3400640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7febd806bbd0 0x7febd806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:04.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.598+0000 7febf3400640 1 -- 172.21.15.27:0/259148834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7febe408ba60 con 0x7febfc153ed0 2024-03-28T19:27:04.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.598+0000 7febfb400640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7febd806bbd0 0x7febd806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:04.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.599+0000 7febfb400640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7febd806bbd0 0x7febd806e090 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7febec006fd0 tx=0x7febec008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:04.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.604+0000 7febf3400640 1 -- 172.21.15.27:0/259148834 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7febe4017020 con 0x7febfc153ed0 2024-03-28T19:27:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.931+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febfc0ac190 con 0x7febfc153ed0 2024-03-28T19:27:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.932+0000 7febf3400640 1 -- 172.21.15.27:0/259148834 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5460 (secure 0 0 0) 0x7febe40592e0 con 0x7febfc153ed0 2024-03-28T19:27:04.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:00.494514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 230 2024-03-28T19:27:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7febd806bbd0 msgr2=0x7febd806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7febd806bbd0 0x7febd806e090 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7febec006fd0 tx=0x7febec008040 comp rx=0 tx=0).stop 2024-03-28T19:27:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc153ed0 msgr2=0x7febfc0aad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc153ed0 0x7febfc0aad30 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7febe400e7e0 tx=0x7febe400eca0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 shutdown_connections 2024-03-28T19:27:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7febd806bbd0 0x7febd806e090 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7febfc159bc0 0x7febfc15b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc153ed0 0x7febfc0aad30 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 --2- 172.21.15.27:0/259148834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febfc06e6f0 0x7febfc1f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 >> 172.21.15.27:0/259148834 conn(0x7febfc06b9d0 msgr2=0x7febfc1e61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 shutdown_connections 2024-03-28T19:27:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:04.935+0000 7fec02800640 1 -- 172.21.15.27:0/259148834 wait complete. 2024-03-28T19:27:04.958 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:27:06.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:07.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.204+0000 7ff3c4200640 1 Processor -- start 2024-03-28T19:27:07.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.205+0000 7ff3c4200640 1 -- start start 2024-03-28T19:27:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.206+0000 7ff3c4200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc14f6d0 0x7ff3bc159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.206+0000 7ff3c4200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff3bc071cf0 0x7ff3bc0c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.206+0000 7ff3c4200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3bc072870 con 0x7ff3bc0c3aa0 2024-03-28T19:27:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.206+0000 7ff3c4200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3bc156640 con 0x7ff3bc071cf0 2024-03-28T19:27:07.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.206+0000 7ff3c4200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc158db0 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.207+0000 7ff3c0e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff3bc071cf0 0x7ff3bc0c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.207+0000 7ff3c0e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff3bc071cf0 0x7ff3bc0c3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:53092/0 (socket says 172.21.15.27:53092) 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.207+0000 7ff3c0e00640 1 -- 172.21.15.27:0/3628292620 learned_addr learned my addr 172.21.15.27:0/3628292620 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.207+0000 7ff3c1800640 1 --2- 172.21.15.27:0/3628292620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc14f6d0 0x7ff3bc159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.207+0000 7ff3bbe00640 1 -- 172.21.15.27:0/3628292620 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117055833 0 0) 0x7ff3b4001490 con 0x7ff3bc0c3aa0 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.208+0000 7ff3bbe00640 1 -- 172.21.15.27:0/3628292620 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a4003520 con 0x7ff3bc0c3aa0 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.208+0000 7ff3c0e00640 1 -- 172.21.15.27:0/3628292620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc14f6d0 msgr2=0x7ff3bc159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.208+0000 7ff3c0e00640 1 --2- 172.21.15.27:0/3628292620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc14f6d0 0x7ff3bc159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.208+0000 7ff3c0e00640 1 -- 172.21.15.27:0/3628292620 >> v1:172.21.15.155:6789/0 conn(0x7ff3bc0c3aa0 legacy=0x7ff3bc0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:07.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.208+0000 7ff3c0e00640 1 -- 172.21.15.27:0/3628292620 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc156360 con 0x7ff3bc071cf0 2024-03-28T19:27:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.209+0000 7ff3c0e00640 1 --2- 172.21.15.27:0/3628292620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc071cf0 0x7ff3bc0c3560 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009870 tx=0x7ff3a802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=286610ca681926c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.209+0000 7ff3bbe00640 1 -- 172.21.15.27:0/3628292620 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8035020 con 0x7ff3bc071cf0 2024-03-28T19:27:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.209+0000 7ff3bbe00640 1 -- 172.21.15.27:0/3628292620 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a8031640 con 0x7ff3bc071cf0 2024-03-28T19:27:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.209+0000 7ff3bbe00640 1 -- 172.21.15.27:0/3628292620 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a802d040 con 0x7ff3bc071cf0 2024-03-28T19:27:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.210+0000 7ff3c4200640 1 -- 172.21.15.27:0/3628292620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc071cf0 msgr2=0x7ff3bc0c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:07.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.210+0000 7ff3c4200640 1 --2- 172.21.15.27:0/3628292620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc071cf0 0x7ff3bc0c3560 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009870 tx=0x7ff3a802b8f0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.210+0000 7ff3c4200640 1 -- 172.21.15.27:0/3628292620 shutdown_connections 2024-03-28T19:27:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.210+0000 7ff3c4200640 1 --2- 172.21.15.27:0/3628292620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc071cf0 0x7ff3bc0c3560 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.210+0000 7ff3c4200640 1 --2- 172.21.15.27:0/3628292620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc14f6d0 0x7ff3bc159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.210+0000 7ff3c4200640 1 -- 172.21.15.27:0/3628292620 >> 172.21.15.27:0/3628292620 conn(0x7ff3bc06c5f0 msgr2=0x7ff3bc06ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.210+0000 7ff3c4200640 1 -- 172.21.15.27:0/3628292620 shutdown_connections 2024-03-28T19:27:07.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.210+0000 7ff3c4200640 1 -- 172.21.15.27:0/3628292620 wait complete. 2024-03-28T19:27:07.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.212+0000 7ff3c4200640 1 Processor -- start 2024-03-28T19:27:07.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.212+0000 7ff3c4200640 1 -- start start 2024-03-28T19:27:07.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c4200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc071cf0 0x7ff3bc0ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:07.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c4200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc0c3aa0 0x7ff3bc1ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:07.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c4200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3bc14f6d0 0x7ff3bc156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:07.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c4200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc14fab0 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c4200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc159330 con 0x7ff3bc0c3aa0 2024-03-28T19:27:07.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c4200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc0c56b0 con 0x7ff3bc071cf0 2024-03-28T19:27:07.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c1800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc071cf0 0x7ff3bc0ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:07.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3bc14f6d0 0x7ff3bc156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:07.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3bc14f6d0 0x7ff3bc156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44812/0 (socket says 172.21.15.27:44812) 2024-03-28T19:27:07.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c2200640 1 -- 172.21.15.27:0/176222022 learned_addr learned my addr 172.21.15.27:0/176222022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:07.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.214+0000 7ff3c0e00640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc0c3aa0 0x7ff3bc1ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:07.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.215+0000 7ff3c2200640 1 -- 172.21.15.27:0/176222022 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc071cf0 msgr2=0x7ff3bc0ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:07.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.215+0000 7ff3c2200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc071cf0 0x7ff3bc0ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.215+0000 7ff3c2200640 1 -- 172.21.15.27:0/176222022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc0c3aa0 msgr2=0x7ff3bc1ee450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:07.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.215+0000 7ff3c2200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc0c3aa0 0x7ff3bc1ee450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.215+0000 7ff3c2200640 1 -- 172.21.15.27:0/176222022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc152170 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.215+0000 7ff3c2200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3bc14f6d0 0x7ff3bc156a80 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c730 tx=0x7ff3b400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:07.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.216+0000 7ff3b9600640 1 -- 172.21.15.27:0/176222022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4016020 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.216+0000 7ff3b9600640 1 -- 172.21.15.27:0/176222022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b4006d50 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.216+0000 7ff3b9600640 1 -- 172.21.15.27:0/176222022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4003bb0 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.216+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3bc152410 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.216+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3bc1500b0 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.218+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff398005290 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.219+0000 7ff3b9600640 1 -- 172.21.15.27:0/176222022 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff3b40050d0 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.220+0000 7ff3b9600640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff39c06bbd0 0x7ff39c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:07.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.220+0000 7ff3b9600640 1 -- 172.21.15.27:0/176222022 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7ff3b408b710 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.220+0000 7ff3c1800640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff39c06bbd0 0x7ff39c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.221+0000 7ff3c1800640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff39c06bbd0 0x7ff39c06e090 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009aa0 tx=0x7ff3b0009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:07.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.230+0000 7ff3b9600640 1 -- 172.21.15.27:0/176222022 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff3b4020020 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.529+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff398005580 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.529+0000 7ff3b9600640 1 -- 172.21.15.27:0/176222022 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5460 (secure 0 0 0) 0x7ff3bc072870 con 0x7ff3bc14f6d0 2024-03-28T19:27:07.530 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:07.530 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:00.494514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:07.530 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 230 2024-03-28T19:27:07.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff39c06bbd0 msgr2=0x7ff39c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:07.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff39c06bbd0 0x7ff39c06e090 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009aa0 tx=0x7ff3b0009190 comp rx=0 tx=0).stop 2024-03-28T19:27:07.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3bc14f6d0 msgr2=0x7ff3bc156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:07.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3bc14f6d0 0x7ff3bc156a80 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c730 tx=0x7ff3b400cbf0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 shutdown_connections 2024-03-28T19:27:07.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7ff39c06bbd0 0x7ff39c06e090 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3bc14f6d0 0x7ff3bc156a80 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3bc0c3aa0 0x7ff3bc1ee450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 --2- 172.21.15.27:0/176222022 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3bc071cf0 0x7ff3bc0ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:07.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 >> 172.21.15.27:0/176222022 conn(0x7ff3bc06c5f0 msgr2=0x7ff3bc0c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:07.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 shutdown_connections 2024-03-28T19:27:07.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:07.532+0000 7ff3c4200640 1 -- 172.21.15.27:0/176222022 wait complete. 2024-03-28T19:27:07.551 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:27:09.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:09.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.789+0000 7f90f1400640 1 Processor -- start 2024-03-28T19:27:09.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.789+0000 7f90f1400640 1 -- start start 2024-03-28T19:27:09.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.790+0000 7f90f1400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f90ec066dd0 0x7f90ec06b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90f1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90ec0abe80 0x7f90ec0ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90f1400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec0aa6d0 con 0x7f90ec150a50 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90f1400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ec150e30 con 0x7f90ec0abe80 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90f1400640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ec06f360 con 0x7f90ec066dd0 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90eb200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90ec0abe80 0x7f90ec0ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90e9e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f90ec066dd0 0x7f90ec06b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90eb200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90ec0abe80 0x7f90ec0ac2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44822/0 (socket says 172.21.15.27:44822) 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90eb200640 1 -- 172.21.15.27:0/592939278 learned_addr learned my addr 172.21.15.27:0/592939278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90e9e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f90ec066dd0 0x7f90ec06b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:53112/0 (socket says 172.21.15.27:53112) 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.791+0000 7f90ea800640 1 --1- 172.21.15.27:0/592939278 >> v1:172.21.15.155:6789/0 conn(0x7f90ec150a50 0x7f90ec0ad060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:47108/0 (socket says 172.21.15.27:47108) 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.792+0000 7f90eb200640 1 -- 172.21.15.27:0/592939278 >> v2:172.21.15.155:3300/0 conn(0x7f90ec066dd0 msgr2=0x7f90ec06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:09.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.792+0000 7f90eb200640 1 --2- 172.21.15.27:0/592939278 >> v2:172.21.15.155:3300/0 conn(0x7f90ec066dd0 0x7f90ec06b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.792+0000 7f90eb200640 1 -- 172.21.15.27:0/592939278 >> v1:172.21.15.155:6789/0 conn(0x7f90ec150a50 legacy=0x7f90ec0ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.792+0000 7f90eb200640 1 -- 172.21.15.27:0/592939278 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec154040 con 0x7f90ec0abe80 2024-03-28T19:27:09.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.792+0000 7f90e9400640 1 -- 172.21.15.27:0/592939278 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490557905 0 0) 0x7f90ec0aa6d0 con 0x7f90ec150a50 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.793+0000 7f90eb200640 1 --2- 172.21.15.27:0/592939278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec0abe80 0x7f90ec0ac2b0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f90ec06cb40 tx=0x7f90e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89be977c58d014e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.793+0000 7f90e9400640 1 -- 172.21.15.27:0/592939278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e402e070 con 0x7f90ec0abe80 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.793+0000 7f90e9400640 1 -- 172.21.15.27:0/592939278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e4037070 con 0x7f90ec0abe80 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.793+0000 7f90e9400640 1 -- 172.21.15.27:0/592939278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4031a00 con 0x7f90ec0abe80 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.794+0000 7f90f1400640 1 -- 172.21.15.27:0/592939278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec0abe80 msgr2=0x7f90ec0ac2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.794+0000 7f90f1400640 1 --2- 172.21.15.27:0/592939278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec0abe80 0x7f90ec0ac2b0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f90ec06cb40 tx=0x7f90e402b880 comp rx=0 tx=0).stop 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.794+0000 7f90f1400640 1 -- 172.21.15.27:0/592939278 shutdown_connections 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.794+0000 7f90f1400640 1 --2- 172.21.15.27:0/592939278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec0abe80 0x7f90ec0ac2b0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.794+0000 7f90f1400640 1 --2- 172.21.15.27:0/592939278 >> v2:172.21.15.155:3300/0 conn(0x7f90ec066dd0 0x7f90ec06b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.794+0000 7f90f1400640 1 -- 172.21.15.27:0/592939278 >> 172.21.15.27:0/592939278 conn(0x7f90ec0c3ff0 msgr2=0x7f90ec0c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.794+0000 7f90f1400640 1 -- 172.21.15.27:0/592939278 shutdown_connections 2024-03-28T19:27:09.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.794+0000 7f90f1400640 1 -- 172.21.15.27:0/592939278 wait complete. 2024-03-28T19:27:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.796+0000 7f90f1400640 1 Processor -- start 2024-03-28T19:27:09.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.796+0000 7f90f1400640 1 -- start start 2024-03-28T19:27:09.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90f1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90ec066dd0 0x7f90ec0c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:09.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90f1400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90ec0abe80 0x7f90ec0ad000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:09.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90f1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec150a50 0x7f90ec1eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:09.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90f1400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0ac7f0 con 0x7f90ec150a50 2024-03-28T19:27:09.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90f1400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0acbe0 con 0x7f90ec066dd0 2024-03-28T19:27:09.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90f1400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0ac970 con 0x7f90ec0abe80 2024-03-28T19:27:09.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90eb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec150a50 0x7f90ec1eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:09.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90ec066dd0 0x7f90ec0c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:09.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90ec066dd0 0x7f90ec0c50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:53128/0 (socket says 172.21.15.27:53128) 2024-03-28T19:27:09.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90ea800640 1 -- 172.21.15.27:0/799746952 learned_addr learned my addr 172.21.15.27:0/799746952 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:09.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90eb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec150a50 0x7f90ec1eebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44832/0 (socket says 172.21.15.27:44832) 2024-03-28T19:27:09.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.797+0000 7f90e9e00640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90ec0abe80 0x7f90ec0ad000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:09.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90eb200640 1 -- 172.21.15.27:0/799746952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90ec0abe80 msgr2=0x7f90ec0ad000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:09.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90eb200640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90ec0abe80 0x7f90ec0ad000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:09.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90eb200640 1 -- 172.21.15.27:0/799746952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90ec066dd0 msgr2=0x7f90ec0c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:09.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90eb200640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90ec066dd0 0x7f90ec0c50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:09.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90eb200640 1 -- 172.21.15.27:0/799746952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec0c6df0 con 0x7f90ec150a50 2024-03-28T19:27:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90eb200640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec150a50 0x7f90ec1eebf0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0c8520 tx=0x7f90e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90e2a00640 1 -- 172.21.15.27:0/799746952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4032670 con 0x7f90ec150a50 2024-03-28T19:27:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90e2a00640 1 -- 172.21.15.27:0/799746952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e4032830 con 0x7f90ec150a50 2024-03-28T19:27:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec0c6b10 con 0x7f90ec150a50 2024-03-28T19:27:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.798+0000 7f90e2a00640 1 -- 172.21.15.27:0/799746952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4032c20 con 0x7f90ec150a50 2024-03-28T19:27:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.799+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ec1ebc20 con 0x7f90ec150a50 2024-03-28T19:27:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.800+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ec04f400 con 0x7f90ec150a50 2024-03-28T19:27:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.801+0000 7f90e2a00640 1 -- 172.21.15.27:0/799746952 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f90e40453f0 con 0x7f90ec150a50 2024-03-28T19:27:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.801+0000 7f90e2a00640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f90c806bc00 0x7f90c806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.802+0000 7f90e2a00640 1 -- 172.21.15.27:0/799746952 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f90e40ad900 con 0x7f90ec150a50 2024-03-28T19:27:09.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.802+0000 7f90ea800640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f90c806bc00 0x7f90c806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:09.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.803+0000 7f90ea800640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f90c806bc00 0x7f90c806e0c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f90dc004fd0 tx=0x7f90dc004d60 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:09.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:09.807+0000 7f90e2a00640 1 -- 172.21.15.27:0/799746952 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f90e4040070 con 0x7f90ec150a50 2024-03-28T19:27:10.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.134+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90ec0c7220 con 0x7f90ec150a50 2024-03-28T19:27:10.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.135+0000 7f90e2a00640 1 -- 172.21.15.27:0/799746952 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5461 (secure 0 0 0) 0x7f90e407abc0 con 0x7f90ec150a50 2024-03-28T19:27:10.136 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:10.136 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:09.021184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:10.136 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 232 2024-03-28T19:27:10.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f90c806bc00 msgr2=0x7f90c806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:10.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f90c806bc00 0x7f90c806e0c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f90dc004fd0 tx=0x7f90dc004d60 comp rx=0 tx=0).stop 2024-03-28T19:27:10.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec150a50 msgr2=0x7f90ec1eebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:10.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec150a50 0x7f90ec1eebf0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0c8520 tx=0x7f90e40046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 shutdown_connections 2024-03-28T19:27:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f90c806bc00 0x7f90c806e0c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90ec150a50 0x7f90ec1eebf0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90ec0abe80 0x7f90ec0ad000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 --2- 172.21.15.27:0/799746952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90ec066dd0 0x7f90ec0c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 >> 172.21.15.27:0/799746952 conn(0x7f90ec0c3ff0 msgr2=0x7f90ec153440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 shutdown_connections 2024-03-28T19:27:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:10.138+0000 7f90f1400640 1 -- 172.21.15.27:0/799746952 wait complete. 2024-03-28T19:27:10.160 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:27:11.457 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 12% (32/258) Updating files: 13% (34/258) Updating files: 14% (37/258) Updating files: 15% (39/258) Updating files: 16% (42/258) Updating files: 17% (44/258) Updating files: 18% (47/258) Updating files: 19% (50/258) Updating files: 20% (52/258) Updating files: 21% (55/258) Updating files: 22% (57/258) Updating files: 23% (60/258) Updating files: 24% (62/258) Updating files: 25% (65/258) Updating files: 26% (68/258) Updating files: 27% (70/258) Updating files: 28% (73/258) Updating files: 29% (75/258) Updating files: 30% (78/258) Updating files: 31% (80/258) Updating files: 32% (83/258) Updating files: 33% (86/258) Updating files: 34% (88/258) Updating files: 35% (91/258) Updating files: 36% (93/258) Updating files: 37% (96/258) Updating files: 38% (99/258) Updating files: 39% (101/258) Updating files: 40% (104/258) Updating files: 41% (106/258) Updating files: 42% (109/258) Updating files: 43% (111/258) Updating files: 44% (114/258) Updating files: 45% (117/258) Updating files: 46% (119/258) Updating files: 46% (121/258) Updating files: 47% (122/258) Updating files: 48% (124/258) Updating files: 49% (127/258) Updating files: 50% (129/258) Updating files: 51% (132/258) Updating files: 52% (135/258) Updating files: 53% (137/258) Updating files: 54% (140/258) Updating files: 55% (142/258) Updating files: 56% (145/258) Updating files: 57% (148/258) Updating files: 58% (150/258) Updating files: 59% (153/258) Updating files: 59% (154/258) Updating files: 60% (155/258) Updating files: 61% (158/258) Updating files: 62% (160/258) Updating files: 63% (163/258) Updating files: 64% (166/258) Updating files: 65% (168/258) Updating files: 66% (171/258) Updating files: 67% (173/258) Updating files: 68% (176/258) Updating files: 69% (179/258) Updating files: 70% (181/258) Updating files: 70% (182/258) Updating files: 71% (184/258) Updating files: 72% (186/258) Updating files: 73% (189/258) Updating files: 74% (191/258) Updating files: 75% (194/258) Updating files: 76% (197/258) Updating files: 77% (199/258) Updating files: 78% (202/258) Updating files: 79% (204/258) Updating files: 80% (207/258) Updating files: 81% (209/258) Updating files: 82% (212/258) Updating files: 83% (215/258) Updating files: 84% (217/258) Updating files: 85% (220/258) Updating files: 86% (222/258) Updating files: 87% (225/258) Updating files: 88% (228/258) Updating files: 89% (230/258) Updating files: 90% (233/258) Updating files: 91% (235/258) Updating files: 92% (238/258) Updating files: 93% (240/258) Updating files: 94% (243/258) Updating files: 95% (246/258) Updating files: 96% (248/258) Updating files: 97% (251/258) Updating files: 98% (253/258) Updating files: 99% (256/258) Updating files: 100% (258/258) Updating files: 100% (258/258), done. 2024-03-28T19:27:11.867 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 321ecdaba2c v0.39 2024-03-28T19:27:11.886 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.39 2024-03-28T19:27:11.965 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:27:11.965 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 40 -eq 48 ']' 2024-03-28T19:27:11.965 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.40 2024-03-28T19:27:11.965 INFO:tasks.workunit.client.0.smithi027.stdout:v0.40 2024-03-28T19:27:11.966 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.40 2024-03-28T19:27:11.966 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.40 2024-03-28T19:27:12.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:12.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.414+0000 7fec7e000640 1 Processor -- start 2024-03-28T19:27:12.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.414+0000 7fec7e000640 1 -- start start 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec7e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec780c2050 0x7fec7814f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec7e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec7806fcf0 0x7fec781590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec7e000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec780a7ed0 con 0x7fec780c6d90 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec7e000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec780a8170 con 0x7fec780c2050 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec7e000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec780a8450 con 0x7fec7806fcf0 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec7cc00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fec780c6d90 0x7fec78155eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:47116/0 (socket says 172.21.15.27:47116) 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec7cc00640 1 -- 172.21.15.27:0/345748612 learned_addr learned my addr 172.21.15.27:0/345748612 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec77e00640 1 --2- 172.21.15.27:0/345748612 >> v2:172.21.15.27:3300/0 conn(0x7fec780c2050 0x7fec7814f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.416+0000 7fec7d600640 1 --2- 172.21.15.27:0/345748612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec7806fcf0 0x7fec781590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.417+0000 7fec77e00640 1 -- 172.21.15.27:0/345748612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec7806fcf0 msgr2=0x7fec781590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.417+0000 7fec77e00640 1 --2- 172.21.15.27:0/345748612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec7806fcf0 0x7fec781590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.417+0000 7fec77e00640 1 -- 172.21.15.27:0/345748612 >> v1:172.21.15.155:6789/0 conn(0x7fec780c6d90 legacy=0x7fec78155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:12.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.417+0000 7fec77e00640 1 -- 172.21.15.27:0/345748612 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec78158bc0 con 0x7fec780c2050 2024-03-28T19:27:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.419+0000 7fec77e00640 1 --2- 172.21.15.27:0/345748612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec780c2050 0x7fec7814f0a0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fec640097d0 tx=0x7fec6402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3063409cca789a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.419+0000 7fec77400640 1 -- 172.21.15.27:0/345748612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64039070 con 0x7fec780c2050 2024-03-28T19:27:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.419+0000 7fec77400640 1 -- 172.21.15.27:0/345748612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec640316f0 con 0x7fec780c2050 2024-03-28T19:27:12.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.419+0000 7fec77400640 1 -- 172.21.15.27:0/345748612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64031ac0 con 0x7fec780c2050 2024-03-28T19:27:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.420+0000 7fec7e000640 1 -- 172.21.15.27:0/345748612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec780c2050 msgr2=0x7fec7814f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.420+0000 7fec7e000640 1 --2- 172.21.15.27:0/345748612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec780c2050 0x7fec7814f0a0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fec640097d0 tx=0x7fec6402b790 comp rx=0 tx=0).stop 2024-03-28T19:27:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.420+0000 7fec7e000640 1 -- 172.21.15.27:0/345748612 shutdown_connections 2024-03-28T19:27:12.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.420+0000 7fec7e000640 1 --2- 172.21.15.27:0/345748612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec7806fcf0 0x7fec781590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.420+0000 7fec7e000640 1 --2- 172.21.15.27:0/345748612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec780c2050 0x7fec7814f0a0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.420+0000 7fec7e000640 1 -- 172.21.15.27:0/345748612 >> 172.21.15.27:0/345748612 conn(0x7fec78072d10 msgr2=0x7fec7806d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.420+0000 7fec7e000640 1 -- 172.21.15.27:0/345748612 shutdown_connections 2024-03-28T19:27:12.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.420+0000 7fec7e000640 1 -- 172.21.15.27:0/345748612 wait complete. 2024-03-28T19:27:12.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.422+0000 7fec7e000640 1 Processor -- start 2024-03-28T19:27:12.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.422+0000 7fec7e000640 1 -- start start 2024-03-28T19:27:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7806fcf0 0x7fec7814e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7e000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec780c2050 0x7fec780ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec780c6d90 0x7fec780c3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7e000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7814f5e0 con 0x7fec7806fcf0 2024-03-28T19:27:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7e000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780c7380 con 0x7fec780c6d90 2024-03-28T19:27:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7e000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7814f760 con 0x7fec780c2050 2024-03-28T19:27:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7806fcf0 0x7fec7814e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7806fcf0 0x7fec7814e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44854/0 (socket says 172.21.15.27:44854) 2024-03-28T19:27:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7cc00640 1 -- 172.21.15.27:0/690555032 learned_addr learned my addr 172.21.15.27:0/690555032 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec7d600640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec780c6d90 0x7fec780c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.424+0000 7fec77e00640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec780c2050 0x7fec780ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.425+0000 7fec7cc00640 1 -- 172.21.15.27:0/690555032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec780c2050 msgr2=0x7fec780ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.425+0000 7fec7cc00640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec780c2050 0x7fec780ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.425+0000 7fec7cc00640 1 -- 172.21.15.27:0/690555032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec780c6d90 msgr2=0x7fec780c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.425+0000 7fec7cc00640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec780c6d90 0x7fec780c3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.425+0000 7fec7cc00640 1 -- 172.21.15.27:0/690555032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec780c4c50 con 0x7fec7806fcf0 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.425+0000 7fec7cc00640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7806fcf0 0x7fec7814e240 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00c480 tx=0x7fec6c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.425+0000 7fec74c00640 1 -- 172.21.15.27:0/690555032 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c017070 con 0x7fec7806fcf0 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.425+0000 7fec74c00640 1 -- 172.21.15.27:0/690555032 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c005ce0 con 0x7fec7806fcf0 2024-03-28T19:27:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.426+0000 7fec74c00640 1 -- 172.21.15.27:0/690555032 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0072b0 con 0x7fec7806fcf0 2024-03-28T19:27:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.426+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7814df60 con 0x7fec7806fcf0 2024-03-28T19:27:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.426+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec781ebf00 con 0x7fec7806fcf0 2024-03-28T19:27:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.427+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec78067dd0 con 0x7fec7806fcf0 2024-03-28T19:27:12.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.429+0000 7fec74c00640 1 -- 172.21.15.27:0/690555032 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fec6c01a070 con 0x7fec7806fcf0 2024-03-28T19:27:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.430+0000 7fec74c00640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec5806bc00 0x7fec5806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.430+0000 7fec74c00640 1 -- 172.21.15.27:0/690555032 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7fec6c08af20 con 0x7fec7806fcf0 2024-03-28T19:27:12.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.430+0000 7fec77e00640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec5806bc00 0x7fec5806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:12.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.431+0000 7fec77e00640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec5806bc00 0x7fec5806e0c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fec64036000 tx=0x7fec64008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.436+0000 7fec74c00640 1 -- 172.21.15.27:0/690555032 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fec6c0581e0 con 0x7fec7806fcf0 2024-03-28T19:27:12.738 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:27:12.740+0000 7fa999400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:27:12.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.768+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec7806fad0 con 0x7fec7806fcf0 2024-03-28T19:27:12.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.769+0000 7fec74c00640 1 -- 172.21.15.27:0/690555032 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5468 (secure 0 0 0) 0x7fec6c05c280 con 0x7fec7806fcf0 2024-03-28T19:27:12.768 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:12.768 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 234 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.771+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec5806bc00 msgr2=0x7fec5806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.771+0000 7fec7e000640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec5806bc00 0x7fec5806e0c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fec64036000 tx=0x7fec64008030 comp rx=0 tx=0).stop 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.771+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7806fcf0 msgr2=0x7fec7814e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.771+0000 7fec7e000640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7806fcf0 0x7fec7814e240 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00c480 tx=0x7fec6c00c940 comp rx=0 tx=0).stop 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.772+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 shutdown_connections 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.772+0000 7fec7e000640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fec5806bc00 0x7fec5806e0c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.772+0000 7fec7e000640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec780c6d90 0x7fec780c3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.772+0000 7fec7e000640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fec780c2050 0x7fec780ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.772+0000 7fec7e000640 1 --2- 172.21.15.27:0/690555032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec7806fcf0 0x7fec7814e240 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.772+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 >> 172.21.15.27:0/690555032 conn(0x7fec78072d10 msgr2=0x7fec780a8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.772+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 shutdown_connections 2024-03-28T19:27:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:12.772+0000 7fec7e000640 1 -- 172.21.15.27:0/690555032 wait complete. 2024-03-28T19:27:12.793 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6467 2024-03-28T19:27:12.793 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before reviving mds.d 2024-03-28T19:27:17.793 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-03-28T19:27:17.793 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-28T19:27:17.794 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-28T19:27:17.799 INFO:tasks.ceph.mds.d:Started 2024-03-28T19:27:17.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:17.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:27:17.882 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:27:18.025 INFO:tasks.ceph.mds.d.smithi155.stdout:starting mds.d at 2024-03-28T19:27:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.035+0000 7f9b08000640 1 Processor -- start 2024-03-28T19:27:18.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.036+0000 7f9b08000640 1 -- start start 2024-03-28T19:27:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.037+0000 7f9b08000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c5e80 0x7f9b000c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.037+0000 7f9b08000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b000abf60 0x7f9b000aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:18.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.037+0000 7f9b08000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0006d8a0 con 0x7f9b000c1f50 2024-03-28T19:27:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.037+0000 7f9b08000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0006d650 con 0x7f9b000abf60 2024-03-28T19:27:18.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.037+0000 7f9b08000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0006e1d0 con 0x7f9b000c5e80 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.038+0000 7f9b05600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c5e80 0x7f9b000c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.038+0000 7f9b04c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b000c1f50 0x7f9b000ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37604/0 (socket says 172.21.15.27:37604) 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.038+0000 7f9b04c00640 1 -- 172.21.15.27:0/4262804520 learned_addr learned my addr 172.21.15.27:0/4262804520 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.038+0000 7f9b05600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c5e80 0x7f9b000c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:49818/0 (socket says 172.21.15.27:49818) 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.038+0000 7f9b06000640 1 --2- 172.21.15.27:0/4262804520 >> v2:172.21.15.27:3300/0 conn(0x7f9b000abf60 0x7f9b000aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.038+0000 7f9b05600640 1 -- 172.21.15.27:0/4262804520 >> v2:172.21.15.27:3300/0 conn(0x7f9b000abf60 msgr2=0x7f9b000aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.039+0000 7f9b05600640 1 --2- 172.21.15.27:0/4262804520 >> v2:172.21.15.27:3300/0 conn(0x7f9b000abf60 0x7f9b000aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.039+0000 7f9b05600640 1 -- 172.21.15.27:0/4262804520 >> v1:172.21.15.27:6789/0 conn(0x7f9b000c1f50 legacy=0x7f9b000ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.039+0000 7f9affe00640 1 -- 172.21.15.27:0/4262804520 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351082445 0 0) 0x7f9b0006d8a0 con 0x7f9b000c1f50 2024-03-28T19:27:18.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.039+0000 7f9b05600640 1 -- 172.21.15.27:0/4262804520 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0006db80 con 0x7f9b000c5e80 2024-03-28T19:27:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.039+0000 7f9b06000640 1 --2- 172.21.15.27:0/4262804520 >> v2:172.21.15.27:3300/0 conn(0x7f9b000abf60 0x7f9b000aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:27:18.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.040+0000 7f9b05600640 1 --2- 172.21.15.27:0/4262804520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c5e80 0x7f9b000c2cb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9af40097a0 tx=0x7f9af402d9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94eeec7ed017be06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.041+0000 7f9affe00640 1 -- 172.21.15.27:0/4262804520 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4036070 con 0x7f9b000c5e80 2024-03-28T19:27:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.041+0000 7f9affe00640 1 -- 172.21.15.27:0/4262804520 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4032860 con 0x7f9b000c5e80 2024-03-28T19:27:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.041+0000 7f9affe00640 1 -- 172.21.15.27:0/4262804520 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4032c30 con 0x7f9b000c5e80 2024-03-28T19:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.042+0000 7f9b08000640 1 -- 172.21.15.27:0/4262804520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c5e80 msgr2=0x7f9b000c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.042+0000 7f9b08000640 1 --2- 172.21.15.27:0/4262804520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c5e80 0x7f9b000c2cb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9af40097a0 tx=0x7f9af402d9a0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.042+0000 7f9b08000640 1 -- 172.21.15.27:0/4262804520 shutdown_connections 2024-03-28T19:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.042+0000 7f9b08000640 1 --2- 172.21.15.27:0/4262804520 >> v2:172.21.15.27:3300/0 conn(0x7f9b000abf60 0x7f9b000aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.042+0000 7f9b08000640 1 --2- 172.21.15.27:0/4262804520 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c5e80 0x7f9b000c2cb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.042+0000 7f9b08000640 1 -- 172.21.15.27:0/4262804520 >> 172.21.15.27:0/4262804520 conn(0x7f9b0006b9d0 msgr2=0x7f9b000c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.042+0000 7f9b08000640 1 -- 172.21.15.27:0/4262804520 shutdown_connections 2024-03-28T19:27:18.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.042+0000 7f9b08000640 1 -- 172.21.15.27:0/4262804520 wait complete. 2024-03-28T19:27:18.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.044+0000 7f9b08000640 1 Processor -- start 2024-03-28T19:27:18.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.045+0000 7f9b08000640 1 -- start start 2024-03-28T19:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b08000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b000abf60 0x7f9b000c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b08000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c1f50 0x7f9b001dd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b08000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b000c5e80 0x7f9b001dd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b08000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000c3370 con 0x7f9b000c5e80 2024-03-28T19:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b08000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000c31f0 con 0x7f9b000abf60 2024-03-28T19:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b08000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000c6260 con 0x7f9b000c1f50 2024-03-28T19:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b05600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b000abf60 0x7f9b000c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b05600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b000abf60 0x7f9b000c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:50692/0 (socket says 172.21.15.27:50692) 2024-03-28T19:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b05600640 1 -- 172.21.15.27:0/3124511682 learned_addr learned my addr 172.21.15.27:0/3124511682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b06000640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b000c5e80 0x7f9b001dd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.046+0000 7f9b04c00640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c1f50 0x7f9b001dd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.047+0000 7f9b05600640 1 -- 172.21.15.27:0/3124511682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c1f50 msgr2=0x7f9b001dd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.047+0000 7f9b05600640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c1f50 0x7f9b001dd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.047+0000 7f9b05600640 1 -- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b000c5e80 msgr2=0x7f9b001dd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.047+0000 7f9b05600640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b000c5e80 0x7f9b001dd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.047+0000 7f9b05600640 1 -- 172.21.15.27:0/3124511682 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b000a5c20 con 0x7f9b000abf60 2024-03-28T19:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.047+0000 7f9b06000640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b000c5e80 0x7f9b001dd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.048+0000 7f9b05600640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b000abf60 0x7f9b000c02a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9af40098c0 tx=0x7f9af40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.048+0000 7f9afd600640 1 -- 172.21.15.27:0/3124511682 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4036040 con 0x7f9b000abf60 2024-03-28T19:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.048+0000 7f9afd600640 1 -- 172.21.15.27:0/3124511682 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4031cf0 con 0x7f9b000abf60 2024-03-28T19:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.048+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b000a5ec0 con 0x7f9b000abf60 2024-03-28T19:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.048+0000 7f9afd600640 1 -- 172.21.15.27:0/3124511682 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4006e10 con 0x7f9b000abf60 2024-03-28T19:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.049+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b00072090 con 0x7f9b000abf60 2024-03-28T19:27:18.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.050+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0004f400 con 0x7f9b000abf60 2024-03-28T19:27:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.052+0000 7f9afd600640 1 -- 172.21.15.27:0/3124511682 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9af4039070 con 0x7f9b000abf60 2024-03-28T19:27:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.053+0000 7f9afd600640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ae006bbd0 0x7f9ae006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.053+0000 7f9afd600640 1 -- 172.21.15.27:0/3124511682 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f9af40ae1e0 con 0x7f9b000abf60 2024-03-28T19:27:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.053+0000 7f9b04c00640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ae006bbd0 0x7f9ae006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.054+0000 7f9b04c00640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ae006bbd0 0x7f9ae006e090 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9aec000f60 tx=0x7f9aec007d50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.056+0000 7f9afd600640 1 -- 172.21.15.27:0/3124511682 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9af407bc30 con 0x7f9b000abf60 2024-03-28T19:27:18.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.409+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b0006c0d0 con 0x7f9b000abf60 2024-03-28T19:27:18.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.410+0000 7f9afd600640 1 -- 172.21.15.27:0/3124511682 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f9af407b530 con 0x7f9b000abf60 2024-03-28T19:27:18.410 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:18.410 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 234 2024-03-28T19:27:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ae006bbd0 msgr2=0x7f9ae006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ae006bbd0 0x7f9ae006e090 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9aec000f60 tx=0x7f9aec007d50 comp rx=0 tx=0).stop 2024-03-28T19:27:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b000abf60 msgr2=0x7f9b000c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b000abf60 0x7f9b000c02a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9af40098c0 tx=0x7f9af40046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 shutdown_connections 2024-03-28T19:27:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f9ae006bbd0 0x7f9ae006e090 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b000c5e80 0x7f9b001dd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9b000c1f50 0x7f9b001dd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 --2- 172.21.15.27:0/3124511682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9b000abf60 0x7f9b000c02a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 >> 172.21.15.27:0/3124511682 conn(0x7f9b0006b9d0 msgr2=0x7f9b000ab620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 shutdown_connections 2024-03-28T19:27:18.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:18.414+0000 7f9b08000640 1 -- 172.21.15.27:0/3124511682 wait complete. 2024-03-28T19:27:18.434 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-03-28T19:27:20.183 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 20% (52/253) Updating files: 21% (54/253) Updating files: 22% (56/253) Updating files: 23% (59/253) Updating files: 24% (61/253) Updating files: 25% (64/253) Updating files: 26% (66/253) Updating files: 27% (69/253) Updating files: 28% (71/253) Updating files: 29% (74/253) Updating files: 30% (76/253) Updating files: 31% (79/253) Updating files: 32% (81/253) Updating files: 33% (84/253) Updating files: 34% (87/253) Updating files: 35% (89/253) Updating files: 36% (92/253) Updating files: 37% (94/253) Updating files: 38% (97/253) Updating files: 38% (98/253) Updating files: 39% (99/253) Updating files: 40% (102/253) Updating files: 41% (104/253) Updating files: 42% (107/253) Updating files: 43% (109/253) Updating files: 44% (112/253) Updating files: 45% (114/253) Updating files: 46% (117/253) Updating files: 47% (119/253) Updating files: 48% (122/253) Updating files: 49% (124/253) Updating files: 50% (127/253) Updating files: 51% (130/253) Updating files: 52% (132/253) Updating files: 53% (135/253) Updating files: 54% (137/253) Updating files: 55% (140/253) Updating files: 55% (141/253) Updating files: 56% (142/253) Updating files: 57% (145/253) Updating files: 58% (147/253) Updating files: 59% (150/253) Updating files: 60% (152/253) Updating files: 61% (155/253) Updating files: 62% (157/253) Updating files: 63% (160/253) Updating files: 64% (162/253) Updating files: 65% (165/253) Updating files: 66% (167/253) Updating files: 67% (170/253) Updating files: 68% (173/253) Updating files: 69% (175/253) Updating files: 70% (178/253) Updating files: 71% (180/253) Updating files: 72% (183/253) Updating files: 73% (185/253) Updating files: 73% (187/253) Updating files: 74% (188/253) Updating files: 75% (190/253) Updating files: 76% (193/253) Updating files: 77% (195/253) Updating files: 78% (198/253) Updating files: 79% (200/253) Updating files: 80% (203/253) Updating files: 81% (205/253) Updating files: 82% (208/253) Updating files: 83% (210/253) Updating files: 84% (213/253) Updating files: 85% (216/253) Updating files: 86% (218/253) Updating files: 87% (221/253) Updating files: 88% (223/253) Updating files: 89% (226/253) Updating files: 90% (228/253) Updating files: 90% (229/253) Updating files: 91% (231/253) Updating files: 92% (233/253) Updating files: 93% (236/253) Updating files: 94% (238/253) Updating files: 95% (241/253) Updating files: 96% (243/253) Updating files: 97% (246/253) Updating files: 98% (248/253) Updating files: 99% (251/253) Updating files: 100% (253/253) Updating files: 100% (253/253), done. 2024-03-28T19:27:20.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:20.468 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 793034c62c8 v0.27 2024-03-28T19:27:20.491 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.27 2024-03-28T19:27:20.566 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:27:20.567 INFO:tasks.workunit.client.1.smithi155.stdout:v0.28 2024-03-28T19:27:20.567 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 28 -eq 48 ']' 2024-03-28T19:27:20.567 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.28 2024-03-28T19:27:20.567 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.28 2024-03-28T19:27:20.567 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.28 2024-03-28T19:27:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.694+0000 7fef83e00640 1 Processor -- start 2024-03-28T19:27:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.694+0000 7fef83e00640 1 -- start start 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.695+0000 7fef83e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c58c0 0x7fef840c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.695+0000 7fef83e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef840be5d0 0x7fef84155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.695+0000 7fef83e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef84156230 con 0x7fef840c8ed0 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.695+0000 7fef83e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef84158bd0 con 0x7fef840be5d0 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.695+0000 7fef83e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef84158eb0 con 0x7fef840c58c0 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef83400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef840be5d0 0x7fef84155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef82a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c58c0 0x7fef840c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef83400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fef840be5d0 0x7fef84155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60460/0 (socket says 172.21.15.27:60460) 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef83400640 1 -- 172.21.15.27:0/1925429847 learned_addr learned my addr 172.21.15.27:0/1925429847 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:20.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef82000640 1 --1- 172.21.15.27:0/1925429847 >> v1:172.21.15.155:6789/0 conn(0x7fef840c8ed0 0x7fef841555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:48836/0 (socket says 172.21.15.27:48836) 2024-03-28T19:27:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef83400640 1 -- 172.21.15.27:0/1925429847 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c58c0 msgr2=0x7fef840c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef83400640 1 --2- 172.21.15.27:0/1925429847 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c58c0 0x7fef840c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef83400640 1 -- 172.21.15.27:0/1925429847 >> v1:172.21.15.155:6789/0 conn(0x7fef840c8ed0 legacy=0x7fef841555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.696+0000 7fef83400640 1 -- 172.21.15.27:0/1925429847 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef84159190 con 0x7fef840be5d0 2024-03-28T19:27:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.698+0000 7fef83400640 1 --2- 172.21.15.27:0/1925429847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840be5d0 0x7fef84155cf0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009e50 tx=0x7fef7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d200d63f21fc316 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.698+0000 7fef81600640 1 -- 172.21.15.27:0/1925429847 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c02ca70 con 0x7fef840be5d0 2024-03-28T19:27:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.698+0000 7fef81600640 1 -- 172.21.15.27:0/1925429847 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c02cc10 con 0x7fef840be5d0 2024-03-28T19:27:20.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.698+0000 7fef81600640 1 -- 172.21.15.27:0/1925429847 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0343f0 con 0x7fef840be5d0 2024-03-28T19:27:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.699+0000 7fef83e00640 1 -- 172.21.15.27:0/1925429847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840be5d0 msgr2=0x7fef84155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.699+0000 7fef83e00640 1 --2- 172.21.15.27:0/1925429847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840be5d0 0x7fef84155cf0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009e50 tx=0x7fef7c02bed0 comp rx=0 tx=0).stop 2024-03-28T19:27:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.699+0000 7fef83e00640 1 -- 172.21.15.27:0/1925429847 shutdown_connections 2024-03-28T19:27:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.699+0000 7fef83e00640 1 --2- 172.21.15.27:0/1925429847 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840be5d0 0x7fef84155cf0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.699+0000 7fef83e00640 1 --2- 172.21.15.27:0/1925429847 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c58c0 0x7fef840c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:20.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.699+0000 7fef83e00640 1 -- 172.21.15.27:0/1925429847 >> 172.21.15.27:0/1925429847 conn(0x7fef8406b6f0 msgr2=0x7fef840acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.699+0000 7fef83e00640 1 -- 172.21.15.27:0/1925429847 shutdown_connections 2024-03-28T19:27:20.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.699+0000 7fef83e00640 1 -- 172.21.15.27:0/1925429847 wait complete. 2024-03-28T19:27:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.701+0000 7fef83e00640 1 Processor -- start 2024-03-28T19:27:20.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.702+0000 7fef83e00640 1 -- start start 2024-03-28T19:27:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.702+0000 7fef83e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef840be5d0 0x7fef840c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.702+0000 7fef83e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840c58c0 0x7fef840c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef83e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c8ed0 0x7fef840ab500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef83e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef840b0820 con 0x7fef840c58c0 2024-03-28T19:27:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef83e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef84159750 con 0x7fef840be5d0 2024-03-28T19:27:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef83e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef840b09a0 con 0x7fef840c8ed0 2024-03-28T19:27:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840c58c0 0x7fef840c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840c58c0 0x7fef840c48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60464/0 (socket says 172.21.15.27:60464) 2024-03-28T19:27:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82000640 1 -- 172.21.15.27:0/704147048 learned_addr learned my addr 172.21.15.27:0/704147048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82a00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef840be5d0 0x7fef840c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:20.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef83400640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c8ed0 0x7fef840ab500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82000640 1 -- 172.21.15.27:0/704147048 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c8ed0 msgr2=0x7fef840ab500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82000640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c8ed0 0x7fef840ab500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82000640 1 -- 172.21.15.27:0/704147048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef840be5d0 msgr2=0x7fef840c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82000640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef840be5d0 0x7fef840c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.703+0000 7fef82000640 1 -- 172.21.15.27:0/704147048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef840ab0d0 con 0x7fef840c58c0 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.704+0000 7fef82000640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840c58c0 0x7fef840c48c0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fef7000c3c0 tx=0x7fef7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.704+0000 7fef7aa00640 1 -- 172.21.15.27:0/704147048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700074e0 con 0x7fef840c58c0 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.704+0000 7fef7aa00640 1 -- 172.21.15.27:0/704147048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef70007680 con 0x7fef840c58c0 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.704+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef840c2950 con 0x7fef840c58c0 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.704+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef841554d0 con 0x7fef840c58c0 2024-03-28T19:27:20.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.705+0000 7fef7aa00640 1 -- 172.21.15.27:0/704147048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70004a00 con 0x7fef840c58c0 2024-03-28T19:27:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.706+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c005290 con 0x7fef840c58c0 2024-03-28T19:27:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.707+0000 7fef7aa00640 1 -- 172.21.15.27:0/704147048 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fef7001a020 con 0x7fef840c58c0 2024-03-28T19:27:20.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.708+0000 7fef7aa00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef6406bbd0 0x7fef6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:20.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.709+0000 7fef7aa00640 1 -- 172.21.15.27:0/704147048 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7fef7008b210 con 0x7fef840c58c0 2024-03-28T19:27:20.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.709+0000 7fef82a00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef6406bbd0 0x7fef6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:20.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.710+0000 7fef82a00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef6406bbd0 0x7fef6406e090 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fef84153310 tx=0x7fef74009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:20.714+0000 7fef7aa00640 1 -- 172.21.15.27:0/704147048 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fef700584d0 con 0x7fef840c58c0 2024-03-28T19:27:21.066 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 21% (49/226) Updating files: 22% (50/226) Updating files: 23% (52/226) Updating files: 24% (55/226) Updating files: 25% (57/226) Updating files: 26% (59/226) Updating files: 27% (62/226) Updating files: 28% (64/226) Updating files: 29% (66/226) Updating files: 30% (68/226) Updating files: 31% (71/226) Updating files: 32% (73/226) Updating files: 33% (75/226) Updating files: 34% (77/226) Updating files: 35% (80/226) Updating files: 36% (82/226) Updating files: 37% (84/226) Updating files: 38% (86/226) Updating files: 39% (89/226) Updating files: 40% (91/226) Updating files: 41% (93/226) Updating files: 42% (95/226) Updating files: 43% (98/226) Updating files: 44% (100/226) Updating files: 45% (102/226) Updating files: 46% (104/226) Updating files: 47% (107/226) Updating files: 48% (109/226) Updating files: 49% (111/226) Updating files: 50% (113/226) Updating files: 51% (116/226) Updating files: 52% (118/226) Updating files: 53% (120/226) Updating files: 54% (123/226) Updating files: 55% (125/226) Updating files: 56% (127/226) Updating files: 57% (129/226) Updating files: 58% (132/226) Updating files: 59% (134/226) Updating files: 60% (136/226) Updating files: 60% (137/226) Updating files: 61% (138/226) Updating files: 62% (141/226) Updating files: 63% (143/226) Updating files: 64% (145/226) Updating files: 65% (147/226) Updating files: 66% (150/226) Updating files: 67% (152/226) Updating files: 68% (154/226) Updating files: 69% (156/226) Updating files: 70% (159/226) Updating files: 71% (161/226) Updating files: 72% (163/226) Updating files: 73% (165/226) Updating files: 74% (168/226) Updating files: 75% (170/226) Updating files: 76% (172/226) Updating files: 77% (175/226) Updating files: 77% (176/226) Updating files: 78% (177/226) Updating files: 79% (179/226) Updating files: 80% (181/226) Updating files: 81% (184/226) Updating files: 82% (186/226) Updating files: 83% (188/226) Updating files: 84% (190/226) Updating files: 85% (193/226) Updating files: 86% (195/226) Updating files: 87% (197/226) Updating files: 88% (199/226) Updating files: 89% (202/226) Updating files: 90% (204/226) Updating files: 91% (206/226) Updating files: 92% (208/226) Updating files: 93% (211/226) Updating files: 94% (213/226) Updating files: 95% (215/226) Updating files: 96% (217/226) Updating files: 97% (220/226) Updating files: 98% (222/226) Updating files: 99% (224/226) Updating files: 100% (226/226) Updating files: 100% (226/226), done. 2024-03-28T19:27:21.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.079+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef5c005580 con 0x7fef840c58c0 2024-03-28T19:27:21.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.079+0000 7fef7aa00640 1 -- 172.21.15.27:0/704147048 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fef7005c570 con 0x7fef840c58c0 2024-03-28T19:27:21.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:21.079 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9030,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:21.079 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 235 2024-03-28T19:27:21.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef6406bbd0 msgr2=0x7fef6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef6406bbd0 0x7fef6406e090 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fef84153310 tx=0x7fef74009190 comp rx=0 tx=0).stop 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840c58c0 msgr2=0x7fef840c48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840c58c0 0x7fef840c48c0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fef7000c3c0 tx=0x7fef7000c880 comp rx=0 tx=0).stop 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 shutdown_connections 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fef6406bbd0 0x7fef6406e090 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fef840c8ed0 0x7fef840ab500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef840c58c0 0x7fef840c48c0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 --2- 172.21.15.27:0/704147048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef840be5d0 0x7fef840c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.082+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 >> 172.21.15.27:0/704147048 conn(0x7fef8406b6f0 msgr2=0x7fef84157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.083+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 shutdown_connections 2024-03-28T19:27:21.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.083+0000 7fef83e00640 1 -- 172.21.15.27:0/704147048 wait complete. 2024-03-28T19:27:21.105 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-03-28T19:27:21.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:21.326 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 7eea40ea37f v0.40 2024-03-28T19:27:21.331 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.40 2024-03-28T19:27:21.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.340+0000 7f59d5200640 1 Processor -- start 2024-03-28T19:27:21.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.340+0000 7f59d5200640 1 -- start start 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.341+0000 7f59d5200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59d0150c70 0x7f59d00a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.341+0000 7f59d5200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d01534c0 0x7f59d0153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.341+0000 7f59d5200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d00c3af0 con 0x7f59d00c3690 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.341+0000 7f59d5200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d00ab4e0 con 0x7f59d0150c70 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.341+0000 7f59d5200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00ac1f0 con 0x7f59d01534c0 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.342+0000 7f59cde00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59d0150c70 0x7f59d00a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.342+0000 7f59cd400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59d00c3690 0x7f59d00c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37620/0 (socket says 172.21.15.27:37620) 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.342+0000 7f59cde00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59d0150c70 0x7f59d00a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60476/0 (socket says 172.21.15.27:60476) 2024-03-28T19:27:21.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.342+0000 7f59cd400640 1 -- 172.21.15.27:0/2681367650 learned_addr learned my addr 172.21.15.27:0/2681367650 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:21.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.342+0000 7f59ce800640 1 --2- 172.21.15.27:0/2681367650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d01534c0 0x7f59d0153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:21.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.342+0000 7f59cde00640 1 -- 172.21.15.27:0/2681367650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d01534c0 msgr2=0x7f59d0153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:21.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.342+0000 7f59cde00640 1 --2- 172.21.15.27:0/2681367650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d01534c0 0x7f59d0153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.342+0000 7f59cde00640 1 -- 172.21.15.27:0/2681367650 >> v1:172.21.15.27:6789/0 conn(0x7f59d00c3690 legacy=0x7f59d00c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:21.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.343+0000 7f59cde00640 1 -- 172.21.15.27:0/2681367650 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d00c3d90 con 0x7f59d0150c70 2024-03-28T19:27:21.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.344+0000 7f59cde00640 1 --2- 172.21.15.27:0/2681367650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d0150c70 0x7f59d00a9cb0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f59c00097a0 tx=0x7f59c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa261508ac025fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:21.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59cca00640 1 -- 172.21.15.27:0/2681367650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00315a0 con 0x7f59d0150c70 2024-03-28T19:27:21.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59cca00640 1 -- 172.21.15.27:0/2681367650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c0037070 con 0x7f59d0150c70 2024-03-28T19:27:21.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59cca00640 1 -- 172.21.15.27:0/2681367650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0031960 con 0x7f59d0150c70 2024-03-28T19:27:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59d5200640 1 -- 172.21.15.27:0/2681367650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d0150c70 msgr2=0x7f59d00a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59d5200640 1 --2- 172.21.15.27:0/2681367650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d0150c70 0x7f59d00a9cb0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f59c00097a0 tx=0x7f59c002b840 comp rx=0 tx=0).stop 2024-03-28T19:27:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59d5200640 1 -- 172.21.15.27:0/2681367650 shutdown_connections 2024-03-28T19:27:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59d5200640 1 --2- 172.21.15.27:0/2681367650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d01534c0 0x7f59d0153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59d5200640 1 --2- 172.21.15.27:0/2681367650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d0150c70 0x7f59d00a9cb0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59d5200640 1 -- 172.21.15.27:0/2681367650 >> 172.21.15.27:0/2681367650 conn(0x7f59d006b9d0 msgr2=0x7f59d00c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.345+0000 7f59d5200640 1 -- 172.21.15.27:0/2681367650 shutdown_connections 2024-03-28T19:27:21.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.346+0000 7f59d5200640 1 -- 172.21.15.27:0/2681367650 wait complete. 2024-03-28T19:27:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.347+0000 7f59d5200640 1 Processor -- start 2024-03-28T19:27:21.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.347+0000 7f59d5200640 1 -- start start 2024-03-28T19:27:21.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.348+0000 7f59d5200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d00c3690 0x7f59d014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:21.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.348+0000 7f59d5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59d0150c70 0x7f59d00a7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:21.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.348+0000 7f59d5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d01534c0 0x7f59d00a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:21.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.348+0000 7f59d5200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00abcd0 con 0x7f59d01534c0 2024-03-28T19:27:21.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.348+0000 7f59d5200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00ab9f0 con 0x7f59d0150c70 2024-03-28T19:27:21.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.348+0000 7f59d5200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00c1810 con 0x7f59d00c3690 2024-03-28T19:27:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59ce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d01534c0 0x7f59d00a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59ce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d01534c0 0x7f59d00a7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60486/0 (socket says 172.21.15.27:60486) 2024-03-28T19:27:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59ce800640 1 -- 172.21.15.27:0/170011564 learned_addr learned my addr 172.21.15.27:0/170011564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59cde00640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d00c3690 0x7f59d014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59cd400640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59d0150c70 0x7f59d00a7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59ce800640 1 -- 172.21.15.27:0/170011564 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d00c3690 msgr2=0x7f59d014b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59ce800640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d00c3690 0x7f59d014b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59ce800640 1 -- 172.21.15.27:0/170011564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59d0150c70 msgr2=0x7f59d00a7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59ce800640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59d0150c70 0x7f59d00a7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.349+0000 7f59ce800640 1 -- 172.21.15.27:0/170011564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d0155270 con 0x7f59d01534c0 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.350+0000 7f59ce800640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d01534c0 0x7f59d00a7640 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f59c8007e90 tx=0x7f59c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.350+0000 7f59c6000640 1 -- 172.21.15.27:0/170011564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8014420 con 0x7f59d01534c0 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.350+0000 7f59c6000640 1 -- 172.21.15.27:0/170011564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c8019070 con 0x7f59d01534c0 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.350+0000 7f59c6000640 1 -- 172.21.15.27:0/170011564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8014800 con 0x7f59d01534c0 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.350+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d0155a30 con 0x7f59d01534c0 2024-03-28T19:27:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.350+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d01dca40 con 0x7f59d01534c0 2024-03-28T19:27:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.352+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d004f400 con 0x7f59d01534c0 2024-03-28T19:27:21.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.354+0000 7f59c6000640 1 -- 172.21.15.27:0/170011564 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f59c80149a0 con 0x7f59d01534c0 2024-03-28T19:27:21.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.355+0000 7f59c6000640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f59b006bbd0 0x7f59b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:21.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.355+0000 7f59c6000640 1 -- 172.21.15.27:0/170011564 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f59c808c390 con 0x7f59d01534c0 2024-03-28T19:27:21.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.355+0000 7f59cde00640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f59b006bbd0 0x7f59b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:21.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.356+0000 7f59cde00640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f59b006bbd0 0x7f59b006e090 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f59c0036000 tx=0x7f59c003d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:21.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.360+0000 7f59c6000640 1 -- 172.21.15.27:0/170011564 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f59c8059d00 con 0x7f59d01534c0 2024-03-28T19:27:21.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:27:21.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 41 -eq 48 ']' 2024-03-28T19:27:21.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.41 2024-03-28T19:27:21.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.41 2024-03-28T19:27:21.383 INFO:tasks.workunit.client.0.smithi027.stdout:v0.41 2024-03-28T19:27:21.383 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.41 2024-03-28T19:27:21.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.679+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59d00c5f00 con 0x7f59d01534c0 2024-03-28T19:27:21.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.680+0000 7f59c6000640 1 -- 172.21.15.27:0/170011564 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f59c8059600 con 0x7f59d01534c0 2024-03-28T19:27:21.680 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:21.681 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9030,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:21.681 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 235 2024-03-28T19:27:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f59b006bbd0 msgr2=0x7f59b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f59b006bbd0 0x7f59b006e090 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f59c0036000 tx=0x7f59c003d040 comp rx=0 tx=0).stop 2024-03-28T19:27:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d01534c0 msgr2=0x7f59d00a7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d01534c0 0x7f59d00a7640 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f59c8007e90 tx=0x7f59c800e4b0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 shutdown_connections 2024-03-28T19:27:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f59b006bbd0 0x7f59b006e090 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59d01534c0 0x7f59d00a7640 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59d0150c70 0x7f59d00a7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 --2- 172.21.15.27:0/170011564 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59d00c3690 0x7f59d014b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 >> 172.21.15.27:0/170011564 conn(0x7f59d006b9d0 msgr2=0x7f59d0151910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 shutdown_connections 2024-03-28T19:27:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:21.685+0000 7f59d5200640 1 -- 172.21.15.27:0/170011564 wait complete. 2024-03-28T19:27:21.712 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-03-28T19:27:21.713 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:27:21.713 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:27:21.742 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:27:21.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:21.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.000+0000 7f2906200640 1 Processor -- start 2024-03-28T19:27:21.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.000+0000 7f2906200640 1 -- start start 2024-03-28T19:27:22.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.002+0000 7f2906200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f29001563a0 0x7f2900072970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:22.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.002+0000 7f2906200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000c16f0 con 0x7f290006c6e0 2024-03-28T19:27:22.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.002+0000 7f2906200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29000c4010 con 0x7f2900159730 2024-03-28T19:27:22.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.002+0000 7f2906200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2900158b40 con 0x7f29001563a0 2024-03-28T19:27:22.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.002+0000 7f28ffe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f290006c6e0 0x7f2900159020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37636/0 (socket says 172.21.15.27:37636) 2024-03-28T19:27:22.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.002+0000 7f28ffe00640 1 -- 172.21.15.27:0/3597240461 learned_addr learned my addr 172.21.15.27:0/3597240461 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:22.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.003+0000 7f2904e00640 1 --2- 172.21.15.27:0/3597240461 >> v2:172.21.15.155:3300/0 conn(0x7f29001563a0 0x7f2900072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:22.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.003+0000 7f28ff400640 1 -- 172.21.15.27:0/3597240461 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546059304 0 0) 0x7f29000c16f0 con 0x7f290006c6e0 2024-03-28T19:27:22.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.003+0000 7f28ff400640 1 -- 172.21.15.27:0/3597240461 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e8003520 con 0x7f290006c6e0 2024-03-28T19:27:22.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.003+0000 7f2904e00640 1 -- 172.21.15.27:0/3597240461 >> v1:172.21.15.155:6789/0 conn(0x7f2900159730 legacy=0x7f290015fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:22.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.003+0000 7f2904e00640 1 -- 172.21.15.27:0/3597240461 >> v1:172.21.15.27:6789/0 conn(0x7f290006c6e0 legacy=0x7f2900159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:22.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.003+0000 7f2904e00640 1 -- 172.21.15.27:0/3597240461 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000c1410 con 0x7f29001563a0 2024-03-28T19:27:22.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.005+0000 7f2904e00640 1 --2- 172.21.15.27:0/3597240461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29001563a0 0x7f2900072970 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f28f40097a0 tx=0x7f28f402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c987c4ccbaf6e7d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:22.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.005+0000 7f28ff400640 1 -- 172.21.15.27:0/3597240461 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f40315a0 con 0x7f29001563a0 2024-03-28T19:27:22.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.005+0000 7f28ff400640 1 -- 172.21.15.27:0/3597240461 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4037070 con 0x7f29001563a0 2024-03-28T19:27:22.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.005+0000 7f28ff400640 1 -- 172.21.15.27:0/3597240461 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4031960 con 0x7f29001563a0 2024-03-28T19:27:22.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.006+0000 7f2906200640 1 -- 172.21.15.27:0/3597240461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29001563a0 msgr2=0x7f2900072970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:22.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.006+0000 7f2906200640 1 --2- 172.21.15.27:0/3597240461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29001563a0 0x7f2900072970 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f28f40097a0 tx=0x7f28f402b840 comp rx=0 tx=0).stop 2024-03-28T19:27:22.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.006+0000 7f2906200640 1 -- 172.21.15.27:0/3597240461 shutdown_connections 2024-03-28T19:27:22.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.006+0000 7f2906200640 1 --2- 172.21.15.27:0/3597240461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29001563a0 0x7f2900072970 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:22.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.006+0000 7f2906200640 1 -- 172.21.15.27:0/3597240461 >> 172.21.15.27:0/3597240461 conn(0x7f290006b0c0 msgr2=0x7f290006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:22.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.006+0000 7f2906200640 1 -- 172.21.15.27:0/3597240461 shutdown_connections 2024-03-28T19:27:22.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.006+0000 7f2906200640 1 -- 172.21.15.27:0/3597240461 wait complete. 2024-03-28T19:27:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.008+0000 7f2906200640 1 Processor -- start 2024-03-28T19:27:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.008+0000 7f2906200640 1 -- start start 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2906200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f290006c6e0 0x7f290006f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2906200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29001563a0 0x7f29001eb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2906200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2900159730 0x7f290014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2906200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000c1070 con 0x7f290006c6e0 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2906200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000c0ef0 con 0x7f2900159730 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2906200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29000c5930 con 0x7f29001563a0 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2904e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f290006c6e0 0x7f290006f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2904e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f290006c6e0 0x7f290006f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60490/0 (socket says 172.21.15.27:60490) 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2904e00640 1 -- 172.21.15.27:0/2652692749 learned_addr learned my addr 172.21.15.27:0/2652692749 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f28ffe00640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29001563a0 0x7f29001eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.009+0000 7f2905800640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2900159730 0x7f290014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:22.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f2904e00640 1 -- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29001563a0 msgr2=0x7f29001eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:22.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f2904e00640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29001563a0 0x7f29001eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:22.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f2904e00640 1 -- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2900159730 msgr2=0x7f290014b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:22.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f2904e00640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2900159730 0x7f290014b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:22.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f2904e00640 1 -- 172.21.15.27:0/2652692749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29001ee8d0 con 0x7f290006c6e0 2024-03-28T19:27:22.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f28ffe00640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29001563a0 0x7f29001eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:27:22.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f2905800640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2900159730 0x7f290014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:27:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f2904e00640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f290006c6e0 0x7f290006f5b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f28f40098c0 tx=0x7f28f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.010+0000 7f28fcc00640 1 -- 172.21.15.27:0/2652692749 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4032630 con 0x7f290006c6e0 2024-03-28T19:27:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.011+0000 7f28fcc00640 1 -- 172.21.15.27:0/2652692749 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f40327f0 con 0x7f290006c6e0 2024-03-28T19:27:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.011+0000 7f28fcc00640 1 -- 172.21.15.27:0/2652692749 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4032be0 con 0x7f290006c6e0 2024-03-28T19:27:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.011+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2900153ad0 con 0x7f290006c6e0 2024-03-28T19:27:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.011+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2900158870 con 0x7f290006c6e0 2024-03-28T19:27:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.012+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2900069640 con 0x7f290006c6e0 2024-03-28T19:27:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.013+0000 7f28fcc00640 1 -- 172.21.15.27:0/2652692749 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f28f403b3f0 con 0x7f290006c6e0 2024-03-28T19:27:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.014+0000 7f28fcc00640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28e006bc00 0x7f28e006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.014+0000 7f28fcc00640 1 -- 172.21.15.27:0/2652692749 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f28f402f030 con 0x7f290006c6e0 2024-03-28T19:27:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.014+0000 7f28ffe00640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28e006bc00 0x7f28e006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.015+0000 7f28ffe00640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28e006bc00 0x7f28e006e0c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f29000a7be0 tx=0x7f28ec009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:22.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.019+0000 7f28fcc00640 1 -- 172.21.15.27:0/2652692749 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f28f4040070 con 0x7f290006c6e0 2024-03-28T19:27:22.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.283+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f290006c4c0 con 0x7f290006c6e0 2024-03-28T19:27:22.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.285+0000 7f28fcc00640 1 -- 172.21.15.27:0/2652692749 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f28f407aca0 con 0x7f290006c6e0 2024-03-28T19:27:22.284 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:22.284 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9030,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:22.285 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 235 2024-03-28T19:27:22.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.287+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28e006bc00 msgr2=0x7f28e006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.287+0000 7f2906200640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28e006bc00 0x7f28e006e0c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f29000a7be0 tx=0x7f28ec009040 comp rx=0 tx=0).stop 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.287+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f290006c6e0 msgr2=0x7f290006f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.287+0000 7f2906200640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f290006c6e0 0x7f290006f5b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f28f40098c0 tx=0x7f28f40046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.288+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 shutdown_connections 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.288+0000 7f2906200640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f28e006bc00 0x7f28e006e0c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.288+0000 7f2906200640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2900159730 0x7f290014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.288+0000 7f2906200640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29001563a0 0x7f29001eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.288+0000 7f2906200640 1 --2- 172.21.15.27:0/2652692749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f290006c6e0 0x7f290006f5b0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.288+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 >> 172.21.15.27:0/2652692749 conn(0x7f290006b0c0 msgr2=0x7f290006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:22.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.288+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 shutdown_connections 2024-03-28T19:27:22.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:22.288+0000 7f2906200640 1 -- 172.21.15.27:0/2652692749 wait complete. 2024-03-28T19:27:22.307 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:27:24.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.539+0000 7fa7ab200640 1 Processor -- start 2024-03-28T19:27:24.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.539+0000 7fa7ab200640 1 -- start start 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.541+0000 7fa7ab200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac0c1390 0x7fa7ac0c3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.541+0000 7fa7ab200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac157530 con 0x7fa7ac0c36b0 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.541+0000 7fa7ab200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac06dbd0 con 0x7fa7ac157150 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.541+0000 7fa7ab200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac06e1e0 con 0x7fa7ac0c1390 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.541+0000 7fa7aa800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa7ac0c36b0 0x7fa7ac164400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37652/0 (socket says 172.21.15.27:37652) 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.541+0000 7fa7aa800640 1 -- 172.21.15.27:0/1812617135 learned_addr learned my addr 172.21.15.27:0/1812617135 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.541+0000 7fa7a1400640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac0c1390 0x7fa7ac0c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550794873 0 0) 0x7fa7ac157530 con 0x7fa7ac0c36b0 2024-03-28T19:27:24.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa790003520 con 0x7fa7ac0c36b0 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982823936 0 0) 0x7fa790003520 con 0x7fa7ac0c36b0 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac0c1390 msgr2=0x7fa7ac0c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac0c1390 0x7fa7ac0c3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 >> v1:172.21.15.155:6789/0 conn(0x7fa7ac157150 legacy=0x7fa7ac0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac15de60 con 0x7fa7ac0c36b0 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fa7a40019f0 con 0x7fa7ac0c36b0 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 >> v1:172.21.15.27:6789/0 conn(0x7fa7ac0c36b0 legacy=0x7fa7ac164400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa790003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac0c1390 0x7fa790004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa790004d40 0x7fa7900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac06e1e0 con 0x7fa7900039e0 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79000ba70 con 0x7fa790004d40 2024-03-28T19:27:24.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.542+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79000bc70 con 0x7fa7ac0c1390 2024-03-28T19:27:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.543+0000 7fa7a9e00640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa790003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.543+0000 7fa7a1400640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac0c1390 0x7fa790004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.543+0000 7fa7a9e00640 1 -- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac0c1390 msgr2=0x7fa790004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.543+0000 7fa7a9e00640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac0c1390 0x7fa790004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.543+0000 7fa7a9e00640 1 -- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa790004d40 msgr2=0x7fa7900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.543+0000 7fa7a9e00640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa790004d40 0x7fa7900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.543+0000 7fa7a9e00640 1 -- 172.21.15.27:0/1812617135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790003520 con 0x7fa7900039e0 2024-03-28T19:27:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.544+0000 7fa7a9e00640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa790003e00 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa79c001720 tx=0x7fa790005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9aaa389dd06a5b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.545+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c001370 con 0x7fa7900039e0 2024-03-28T19:27:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.545+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79c0052d0 con 0x7fa7900039e0 2024-03-28T19:27:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.545+0000 7fa7a9400640 1 -- 172.21.15.27:0/1812617135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c003830 con 0x7fa7900039e0 2024-03-28T19:27:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.545+0000 7fa7ab200640 1 -- 172.21.15.27:0/1812617135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 msgr2=0x7fa790003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:24.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.545+0000 7fa7ab200640 1 --2- 172.21.15.27:0/1812617135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa790003e00 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa79c001720 tx=0x7fa790005830 comp rx=0 tx=0).stop 2024-03-28T19:27:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.545+0000 7fa7aa800640 1 -- 172.21.15.27:0/1812617135 reap_dead start 2024-03-28T19:27:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.545+0000 7fa7ab200640 1 -- 172.21.15.27:0/1812617135 shutdown_connections 2024-03-28T19:27:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.545+0000 7fa7ab200640 1 -- 172.21.15.27:0/1812617135 >> 172.21.15.27:0/1812617135 conn(0x7fa7ac06b0c0 msgr2=0x7fa7ac06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.546+0000 7fa7ab200640 1 -- 172.21.15.27:0/1812617135 shutdown_connections 2024-03-28T19:27:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.546+0000 7fa7ab200640 1 -- 172.21.15.27:0/1812617135 wait complete. 2024-03-28T19:27:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.547+0000 7fa7ab200640 1 Processor -- start 2024-03-28T19:27:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.548+0000 7fa7ab200640 1 -- start start 2024-03-28T19:27:24.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7ab200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa7ac1f0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7ab200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac1f07b0 0x7fa7ac0a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7ab200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7ac0a8380 0x7fa7ac0a3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7ab200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac0c0fc0 con 0x7fa7900039e0 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7ab200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac06e710 con 0x7fa7ac0a8380 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7ab200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac0c2bc0 con 0x7fa7ac1f07b0 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa7ac1f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa7ac1f0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60498/0 (socket says 172.21.15.27:60498) 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a9e00640 1 -- 172.21.15.27:0/2235894828 learned_addr learned my addr 172.21.15.27:0/2235894828 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a1400640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac1f07b0 0x7fa7ac0a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a9e00640 1 -- 172.21.15.27:0/2235894828 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac1f07b0 msgr2=0x7fa7ac0a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a9e00640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac1f07b0 0x7fa7ac0a7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a9e00640 1 -- 172.21.15.27:0/2235894828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7ac0a8380 msgr2=0x7fa7ac0a3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a9e00640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7ac0a8380 0x7fa7ac0a3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.549+0000 7fa7a9e00640 1 -- 172.21.15.27:0/2235894828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac0a58f0 con 0x7fa7900039e0 2024-03-28T19:27:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.550+0000 7fa7a9e00640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa7ac1f0270 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa79c008fd0 tx=0x7fa79c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.550+0000 7fa7a2a00640 1 -- 172.21.15.27:0/2235894828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c041070 con 0x7fa7900039e0 2024-03-28T19:27:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.550+0000 7fa7a2a00640 1 -- 172.21.15.27:0/2235894828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79c005480 con 0x7fa7900039e0 2024-03-28T19:27:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.550+0000 7fa7a2a00640 1 -- 172.21.15.27:0/2235894828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c007a30 con 0x7fa7900039e0 2024-03-28T19:27:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.550+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ac0a5bd0 con 0x7fa7900039e0 2024-03-28T19:27:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.550+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ac0a6190 con 0x7fa7900039e0 2024-03-28T19:27:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.552+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ac067dd0 con 0x7fa7900039e0 2024-03-28T19:27:24.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.553+0000 7fa7a2a00640 1 -- 172.21.15.27:0/2235894828 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa79c044070 con 0x7fa7900039e0 2024-03-28T19:27:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.554+0000 7fa7a2a00640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa78806bc00 0x7fa78806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.554+0000 7fa7a2a00640 1 -- 172.21.15.27:0/2235894828 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7fa79c0ad800 con 0x7fa7900039e0 2024-03-28T19:27:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.554+0000 7fa7a1400640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa78806bc00 0x7fa78806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:24.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.555+0000 7fa7a1400640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa78806bc00 0x7fa78806e0c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac1525f0 tx=0x7fa794008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:24.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.559+0000 7fa7a2a00640 1 -- 172.21.15.27:0/2235894828 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa79c07b120 con 0x7fa7900039e0 2024-03-28T19:27:24.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.817+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7ac0abe60 con 0x7fa7900039e0 2024-03-28T19:27:24.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.818+0000 7fa7a2a00640 1 -- 172.21.15.27:0/2235894828 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fa7ac0abe60 con 0x7fa7900039e0 2024-03-28T19:27:24.817 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:24.817 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9030,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:24.818 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 235 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.821+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa78806bc00 msgr2=0x7fa78806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.821+0000 7fa7ab200640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa78806bc00 0x7fa78806e0c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac1525f0 tx=0x7fa794008040 comp rx=0 tx=0).stop 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.821+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 msgr2=0x7fa7ac1f0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.821+0000 7fa7ab200640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa7ac1f0270 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fa79c008fd0 tx=0x7fa79c001120 comp rx=0 tx=0).stop 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.822+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 shutdown_connections 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.822+0000 7fa7ab200640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fa78806bc00 0x7fa78806e0c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.822+0000 7fa7ab200640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7ac0a8380 0x7fa7ac0a3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.822+0000 7fa7ab200640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7ac1f07b0 0x7fa7ac0a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.822+0000 7fa7ab200640 1 --2- 172.21.15.27:0/2235894828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7900039e0 0x7fa7ac1f0270 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.822+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 >> 172.21.15.27:0/2235894828 conn(0x7fa7ac06b0c0 msgr2=0x7fa7ac0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.822+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 shutdown_connections 2024-03-28T19:27:24.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:24.822+0000 7fa7ab200640 1 -- 172.21.15.27:0/2235894828 wait complete. 2024-03-28T19:27:24.840 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:27:26.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:27.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.069+0000 7fe677200640 1 Processor -- start 2024-03-28T19:27:27.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.070+0000 7fe677200640 1 -- start start 2024-03-28T19:27:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.071+0000 7fe677200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6780a7040 0x7fe678156190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.071+0000 7fe677200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe67806c0d0 0x7fe67806c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.071+0000 7fe677200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678073680 con 0x7fe678152210 2024-03-28T19:27:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.071+0000 7fe677200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe678152d50 con 0x7fe67806c0d0 2024-03-28T19:27:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.071+0000 7fe677200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6780c7260 con 0x7fe6780a7040 2024-03-28T19:27:27.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.072+0000 7fe675400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe67806c0d0 0x7fe67806c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.072+0000 7fe675400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe67806c0d0 0x7fe67806c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58174/0 (socket says 172.21.15.27:58174) 2024-03-28T19:27:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.072+0000 7fe675400640 1 -- 172.21.15.27:0/3328292067 learned_addr learned my addr 172.21.15.27:0/3328292067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.072+0000 7fe675e00640 1 --2- 172.21.15.27:0/3328292067 >> v2:172.21.15.155:3300/0 conn(0x7fe6780a7040 0x7fe678156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.072+0000 7fe675400640 1 -- 172.21.15.27:0/3328292067 >> v2:172.21.15.155:3300/0 conn(0x7fe6780a7040 msgr2=0x7fe678156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.072+0000 7fe675400640 1 --2- 172.21.15.27:0/3328292067 >> v2:172.21.15.155:3300/0 conn(0x7fe6780a7040 0x7fe678156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.072+0000 7fe675400640 1 -- 172.21.15.27:0/3328292067 >> v1:172.21.15.155:6789/0 conn(0x7fe678152210 legacy=0x7fe678152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.072+0000 7fe675400640 1 -- 172.21.15.27:0/3328292067 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780c7540 con 0x7fe67806c0d0 2024-03-28T19:27:27.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.074+0000 7fe675400640 1 --2- 172.21.15.27:0/3328292067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe67806c0d0 0x7fe67806c560 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fe678071440 tx=0x7fe66402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a27f4d742029f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.074+0000 7fe674a00640 1 -- 172.21.15.27:0/3328292067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66402e070 con 0x7fe67806c0d0 2024-03-28T19:27:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.074+0000 7fe674a00640 1 -- 172.21.15.27:0/3328292067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe664037070 con 0x7fe67806c0d0 2024-03-28T19:27:27.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.074+0000 7fe674a00640 1 -- 172.21.15.27:0/3328292067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe664031a00 con 0x7fe67806c0d0 2024-03-28T19:27:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.075+0000 7fe677200640 1 -- 172.21.15.27:0/3328292067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe67806c0d0 msgr2=0x7fe67806c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.075+0000 7fe677200640 1 --2- 172.21.15.27:0/3328292067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe67806c0d0 0x7fe67806c560 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fe678071440 tx=0x7fe66402b880 comp rx=0 tx=0).stop 2024-03-28T19:27:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.075+0000 7fe677200640 1 -- 172.21.15.27:0/3328292067 shutdown_connections 2024-03-28T19:27:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.075+0000 7fe677200640 1 --2- 172.21.15.27:0/3328292067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe67806c0d0 0x7fe67806c560 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.075+0000 7fe677200640 1 --2- 172.21.15.27:0/3328292067 >> v2:172.21.15.155:3300/0 conn(0x7fe6780a7040 0x7fe678156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.075+0000 7fe677200640 1 -- 172.21.15.27:0/3328292067 >> 172.21.15.27:0/3328292067 conn(0x7fe6780c7aa0 msgr2=0x7fe6780c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:27.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.075+0000 7fe677200640 1 -- 172.21.15.27:0/3328292067 shutdown_connections 2024-03-28T19:27:27.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.075+0000 7fe677200640 1 -- 172.21.15.27:0/3328292067 wait complete. 2024-03-28T19:27:27.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.076+0000 7fe677200640 1 Processor -- start 2024-03-28T19:27:27.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.077+0000 7fe677200640 1 -- start start 2024-03-28T19:27:27.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.077+0000 7fe677200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67806c0d0 0x7fe678153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:27.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe677200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a7040 0x7fe6780abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:27.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe677200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe678152210 0x7fe67814da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:27.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe677200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780c8670 con 0x7fe678152210 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe677200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780a7420 con 0x7fe67806c0d0 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe677200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780c87f0 con 0x7fe6780a7040 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe676800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe678152210 0x7fe67814da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe676800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe678152210 0x7fe67814da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58190/0 (socket says 172.21.15.27:58190) 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe676800640 1 -- 172.21.15.27:0/3483834437 learned_addr learned my addr 172.21.15.27:0/3483834437 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe675e00640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67806c0d0 0x7fe678153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe675400640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a7040 0x7fe6780abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe676800640 1 -- 172.21.15.27:0/3483834437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a7040 msgr2=0x7fe6780abc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe676800640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a7040 0x7fe6780abc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe676800640 1 -- 172.21.15.27:0/3483834437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67806c0d0 msgr2=0x7fe678153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe676800640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67806c0d0 0x7fe678153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.078+0000 7fe676800640 1 -- 172.21.15.27:0/3483834437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780bd5b0 con 0x7fe678152210 2024-03-28T19:27:27.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.079+0000 7fe676800640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe678152210 0x7fe67814da40 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fe67000c7b0 tx=0x7fe67000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:27.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.079+0000 7fe66e000640 1 -- 172.21.15.27:0/3483834437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670004b60 con 0x7fe678152210 2024-03-28T19:27:27.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.079+0000 7fe66e000640 1 -- 172.21.15.27:0/3483834437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe670004d00 con 0x7fe678152210 2024-03-28T19:27:27.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.079+0000 7fe66e000640 1 -- 172.21.15.27:0/3483834437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000aa90 con 0x7fe678152210 2024-03-28T19:27:27.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.079+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6780bd890 con 0x7fe678152210 2024-03-28T19:27:27.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.079+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6780ac140 con 0x7fe678152210 2024-03-28T19:27:27.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.081+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe678069640 con 0x7fe678152210 2024-03-28T19:27:27.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.081+0000 7fe66e000640 1 -- 172.21.15.27:0/3483834437 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe67000ac30 con 0x7fe678152210 2024-03-28T19:27:27.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.083+0000 7fe66e000640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bbd0 0x7fe65806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:27.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.083+0000 7fe66e000640 1 -- 172.21.15.27:0/3483834437 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7fe67008be90 con 0x7fe678152210 2024-03-28T19:27:27.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.083+0000 7fe675e00640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bbd0 0x7fe65806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:27.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.084+0000 7fe675e00640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bbd0 0x7fe65806e090 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe668006fd0 tx=0x7fe668008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:27.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.088+0000 7fe66e000640 1 -- 172.21.15.27:0/3483834437 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe670020070 con 0x7fe678152210 2024-03-28T19:27:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.359+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6780c6ec0 con 0x7fe678152210 2024-03-28T19:27:27.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.360+0000 7fe66e000640 1 -- 172.21.15.27:0/3483834437 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fe670059800 con 0x7fe678152210 2024-03-28T19:27:27.360 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:27.360 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9030,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 235 2024-03-28T19:27:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.363+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bbd0 msgr2=0x7fe65806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bbd0 0x7fe65806e090 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe668006fd0 tx=0x7fe668008040 comp rx=0 tx=0).stop 2024-03-28T19:27:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe678152210 msgr2=0x7fe67814da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe678152210 0x7fe67814da40 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fe67000c7b0 tx=0x7fe67000cc70 comp rx=0 tx=0).stop 2024-03-28T19:27:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 shutdown_connections 2024-03-28T19:27:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe65806bbd0 0x7fe65806e090 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe678152210 0x7fe67814da40 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6780a7040 0x7fe6780abc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 --2- 172.21.15.27:0/3483834437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67806c0d0 0x7fe678153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 >> 172.21.15.27:0/3483834437 conn(0x7fe6780c7aa0 msgr2=0x7fe6780c2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 shutdown_connections 2024-03-28T19:27:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:27.364+0000 7fe677200640 1 -- 172.21.15.27:0/3483834437 wait complete. 2024-03-28T19:27:27.383 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:27:29.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:29.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.621+0000 7eff2cc00640 1 Processor -- start 2024-03-28T19:27:29.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.621+0000 7eff2cc00640 1 -- start start 2024-03-28T19:27:29.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.623+0000 7eff2cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff28154380 0x7eff2806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.623+0000 7eff2cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2806c610 0x7eff2815ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.623+0000 7eff2cc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff280a7490 con 0x7eff280c26e0 2024-03-28T19:27:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.623+0000 7eff2cc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff28154760 con 0x7eff28154380 2024-03-28T19:27:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.623+0000 7eff2cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280a7240 con 0x7eff2806c610 2024-03-28T19:27:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff28154380 0x7eff2806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff28154380 0x7eff2806c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58196/0 (socket says 172.21.15.27:58196) 2024-03-28T19:27:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27400640 1 -- 172.21.15.27:0/1941317543 learned_addr learned my addr 172.21.15.27:0/1941317543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:29.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27e00640 1 --2- 172.21.15.27:0/1941317543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2806c610 0x7eff2815ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27400640 1 -- 172.21.15.27:0/1941317543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2806c610 msgr2=0x7eff2815ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27400640 1 --2- 172.21.15.27:0/1941317543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2806c610 0x7eff2815ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27400640 1 -- 172.21.15.27:0/1941317543 >> v1:172.21.15.27:6789/0 conn(0x7eff280c26e0 legacy=0x7eff280c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27400640 1 -- 172.21.15.27:0/1941317543 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280725a0 con 0x7eff28154380 2024-03-28T19:27:29.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.624+0000 7eff27e00640 1 --2- 172.21.15.27:0/1941317543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2806c610 0x7eff2815ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:27:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.625+0000 7eff27400640 1 --2- 172.21.15.27:0/1941317543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff28154380 0x7eff2806c0d0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7eff18009870 tx=0x7eff1802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58208f54a2c34e91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.626+0000 7eff26000640 1 -- 172.21.15.27:0/1941317543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18034070 con 0x7eff28154380 2024-03-28T19:27:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.626+0000 7eff26000640 1 -- 172.21.15.27:0/1941317543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff18030840 con 0x7eff28154380 2024-03-28T19:27:29.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.626+0000 7eff26000640 1 -- 172.21.15.27:0/1941317543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18030c10 con 0x7eff28154380 2024-03-28T19:27:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.626+0000 7eff2cc00640 1 -- 172.21.15.27:0/1941317543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff28154380 msgr2=0x7eff2806c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.626+0000 7eff2cc00640 1 --2- 172.21.15.27:0/1941317543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff28154380 0x7eff2806c0d0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7eff18009870 tx=0x7eff1802b980 comp rx=0 tx=0).stop 2024-03-28T19:27:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.627+0000 7eff2cc00640 1 -- 172.21.15.27:0/1941317543 shutdown_connections 2024-03-28T19:27:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.627+0000 7eff2cc00640 1 --2- 172.21.15.27:0/1941317543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff2806c610 0x7eff2815ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.627+0000 7eff2cc00640 1 --2- 172.21.15.27:0/1941317543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff28154380 0x7eff2806c0d0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.627+0000 7eff2cc00640 1 -- 172.21.15.27:0/1941317543 >> 172.21.15.27:0/1941317543 conn(0x7eff280c7aa0 msgr2=0x7eff280c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.627+0000 7eff2cc00640 1 -- 172.21.15.27:0/1941317543 shutdown_connections 2024-03-28T19:27:29.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.627+0000 7eff2cc00640 1 -- 172.21.15.27:0/1941317543 wait complete. 2024-03-28T19:27:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.628+0000 7eff2cc00640 1 Processor -- start 2024-03-28T19:27:29.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.629+0000 7eff2cc00640 1 -- start start 2024-03-28T19:27:29.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.629+0000 7eff2cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2806c610 0x7eff2806f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:29.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.629+0000 7eff2cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff28154380 0x7eff281dd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:29.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.629+0000 7eff2cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff281eccb0 0x7eff281dd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:29.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.629+0000 7eff2cc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280c2e60 con 0x7eff281eccb0 2024-03-28T19:27:29.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.629+0000 7eff2cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280c31d0 con 0x7eff2806c610 2024-03-28T19:27:29.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.629+0000 7eff2cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280c77b0 con 0x7eff28154380 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff281eccb0 0x7eff281dd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff281eccb0 0x7eff281dd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58210/0 (socket says 172.21.15.27:58210) 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27e00640 1 -- 172.21.15.27:0/59599846 learned_addr learned my addr 172.21.15.27:0/59599846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27400640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2806c610 0x7eff2806f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff26a00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff28154380 0x7eff281dd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27e00640 1 -- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff28154380 msgr2=0x7eff281dd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27e00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff28154380 0x7eff281dd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27e00640 1 -- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2806c610 msgr2=0x7eff2806f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27e00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2806c610 0x7eff2806f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27e00640 1 -- 172.21.15.27:0/59599846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff281e0690 con 0x7eff281eccb0 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff27400640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2806c610 0x7eff2806f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.630+0000 7eff26a00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff28154380 0x7eff281dd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.631+0000 7eff27e00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff281eccb0 0x7eff281dd5b0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7eff20007df0 tx=0x7eff2000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.631+0000 7eff1f400640 1 -- 172.21.15.27:0/59599846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2001a070 con 0x7eff281eccb0 2024-03-28T19:27:29.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.631+0000 7eff1f400640 1 -- 172.21.15.27:0/59599846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff20003990 con 0x7eff281eccb0 2024-03-28T19:27:29.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.631+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff281ec1d0 con 0x7eff281eccb0 2024-03-28T19:27:29.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.631+0000 7eff1f400640 1 -- 172.21.15.27:0/59599846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff20003d80 con 0x7eff281eccb0 2024-03-28T19:27:29.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.631+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff281ec790 con 0x7eff281eccb0 2024-03-28T19:27:29.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.633+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff281ec990 con 0x7eff281eccb0 2024-03-28T19:27:29.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.633+0000 7eff1f400640 1 -- 172.21.15.27:0/59599846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7eff2001d070 con 0x7eff281eccb0 2024-03-28T19:27:29.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.635+0000 7eff1f400640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bc00 0x7eff0806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:29.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.635+0000 7eff1f400640 1 -- 172.21.15.27:0/59599846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7eff2008bea0 con 0x7eff281eccb0 2024-03-28T19:27:29.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.635+0000 7eff27400640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bc00 0x7eff0806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:29.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.636+0000 7eff27400640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bc00 0x7eff0806e0c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7eff18036000 tx=0x7eff1803b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:29.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.640+0000 7eff1f400640 1 -- 172.21.15.27:0/59599846 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7eff281ec990 con 0x7eff281eccb0 2024-03-28T19:27:29.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.928+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff280c5a30 con 0x7eff281eccb0 2024-03-28T19:27:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.929+0000 7eff1f400640 1 -- 172.21.15.27:0/59599846 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6263 (secure 0 0 0) 0x7eff20010070 con 0x7eff281eccb0 2024-03-28T19:27:29.928 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:29.928 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9030,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:29.928 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 235 2024-03-28T19:27:29.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bc00 msgr2=0x7eff0806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:29.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bc00 0x7eff0806e0c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7eff18036000 tx=0x7eff1803b040 comp rx=0 tx=0).stop 2024-03-28T19:27:29.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff281eccb0 msgr2=0x7eff281dd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:29.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff281eccb0 0x7eff281dd5b0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7eff20007df0 tx=0x7eff2000d4a0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 shutdown_connections 2024-03-28T19:27:29.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7eff0806bc00 0x7eff0806e0c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff281eccb0 0x7eff281dd5b0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7eff28154380 0x7eff281dd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 --2- 172.21.15.27:0/59599846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7eff2806c610 0x7eff2806f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:29.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 >> 172.21.15.27:0/59599846 conn(0x7eff280c7aa0 msgr2=0x7eff280a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:29.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 shutdown_connections 2024-03-28T19:27:29.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:29.932+0000 7eff2cc00640 1 -- 172.21.15.27:0/59599846 wait complete. 2024-03-28T19:27:29.952 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:27:30.538 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 32% (28/87) Updating files: 33% (29/87) Updating files: 34% (30/87) Updating files: 35% (31/87) Updating files: 36% (32/87) Updating files: 37% (33/87) Updating files: 39% (34/87) Updating files: 40% (35/87) Updating files: 41% (36/87) Updating files: 42% (37/87) Updating files: 43% (38/87) Updating files: 44% (39/87) Updating files: 45% (40/87) Updating files: 47% (41/87) Updating files: 48% (42/87) Updating files: 49% (43/87) Updating files: 50% (44/87) Updating files: 51% (45/87) Updating files: 52% (46/87) Updating files: 54% (47/87) Updating files: 55% (48/87) Updating files: 56% (49/87) Updating files: 57% (50/87) Updating files: 58% (51/87) Updating files: 59% (52/87) Updating files: 60% (53/87) Updating files: 62% (54/87) Updating files: 63% (55/87) Updating files: 64% (56/87) Updating files: 65% (57/87) Updating files: 66% (58/87) Updating files: 67% (59/87) Updating files: 68% (60/87) Updating files: 70% (61/87) Updating files: 71% (62/87) Updating files: 72% (63/87) Updating files: 73% (64/87) Updating files: 74% (65/87) Updating files: 75% (66/87) Updating files: 77% (67/87) Updating files: 78% (68/87) Updating files: 79% (69/87) Updating files: 80% (70/87) Updating files: 81% (71/87) Updating files: 82% (72/87) Updating files: 83% (73/87) Updating files: 85% (74/87) Updating files: 86% (75/87) Updating files: 87% (76/87) Updating files: 88% (77/87) Updating files: 89% (78/87) Updating files: 90% (79/87) Updating files: 91% (80/87) Updating files: 93% (81/87) Updating files: 94% (82/87) Updating files: 95% (83/87) Updating files: 96% (84/87) Updating files: 97% (85/87) Updating files: 98% (86/87) Updating files: 100% (87/87) Updating files: 100% (87/87), done. 2024-03-28T19:27:30.914 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at c1345f7136a v0.41 2024-03-28T19:27:30.924 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.41 2024-03-28T19:27:31.007 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:27:31.007 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 42 -eq 48 ']' 2024-03-28T19:27:31.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.42 2024-03-28T19:27:31.008 INFO:tasks.workunit.client.0.smithi027.stdout:v0.42 2024-03-28T19:27:31.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.42 2024-03-28T19:27:31.008 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.42 2024-03-28T19:27:31.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.183+0000 7f8580200640 1 Processor -- start 2024-03-28T19:27:32.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.184+0000 7f8580200640 1 -- start start 2024-03-28T19:27:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.185+0000 7f8580200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f85780ac0a0 0x7f85780ab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.185+0000 7f8580200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85780abb00 con 0x7f8578153ba0 2024-03-28T19:27:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.185+0000 7f8580200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857806e410 con 0x7f8578062370 2024-03-28T19:27:32.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.185+0000 7f8580200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8578153fb0 con 0x7f85780ac0a0 2024-03-28T19:27:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.185+0000 7f857e200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8578153ba0 0x7f857815ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47870/0 (socket says 172.21.15.27:47870) 2024-03-28T19:27:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.186+0000 7f857e200640 1 -- 172.21.15.27:0/977774857 learned_addr learned my addr 172.21.15.27:0/977774857 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.186+0000 7f857d800640 1 --2- 172.21.15.27:0/977774857 >> v2:172.21.15.155:3300/0 conn(0x7f85780ac0a0 0x7f85780ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.186+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121783522 0 0) 0x7f85780abb00 con 0x7f8578153ba0 2024-03-28T19:27:32.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.186+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8560003520 con 0x7f8578153ba0 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.186+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3390671382 0 0) 0x7f8560003520 con 0x7f8578153ba0 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.186+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 >> v2:172.21.15.155:3300/0 conn(0x7f85780ac0a0 msgr2=0x7f85780ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.186+0000 7f8577e00640 1 --2- 172.21.15.27:0/977774857 >> v2:172.21.15.155:3300/0 conn(0x7f85780ac0a0 0x7f85780ab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.186+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 >> v1:172.21.15.155:6789/0 conn(0x7f8578062370 legacy=0x7f8578153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857806e6f0 con 0x7f8578153ba0 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f85700019f0 con 0x7f8578153ba0 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 >> v1:172.21.15.27:6789/0 conn(0x7f8578153ba0 legacy=0x7f857815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85600039b0 0x7f8560003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8560004800 0x7f8560004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f856000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856000c180 con 0x7f85600052e0 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856000c330 con 0x7f85600039b0 2024-03-28T19:27:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856000c530 con 0x7f8560004800 2024-03-28T19:27:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f857e200640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f856000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f857d800640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85600039b0 0x7f8560003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.187+0000 7f857ce00640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8560004800 0x7f8560004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.188+0000 7f857e200640 1 -- 172.21.15.27:0/977774857 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8560004800 msgr2=0x7f8560004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.188+0000 7f857e200640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8560004800 0x7f8560004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.188+0000 7f857e200640 1 -- 172.21.15.27:0/977774857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85600039b0 msgr2=0x7f8560003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.188+0000 7f857e200640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85600039b0 0x7f8560003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.188+0000 7f857e200640 1 -- 172.21.15.27:0/977774857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560003520 con 0x7f85600052e0 2024-03-28T19:27:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.189+0000 7f857e200640 1 -- 172.21.15.27:0/977774857 reap_dead start 2024-03-28T19:27:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.189+0000 7f857e200640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f856000b9c0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f857000a830 tx=0x7f8560005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94114747a47d252 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.189+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857000a480 con 0x7f85600052e0 2024-03-28T19:27:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.189+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8570034630 con 0x7f85600052e0 2024-03-28T19:27:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.189+0000 7f8577e00640 1 -- 172.21.15.27:0/977774857 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570034a00 con 0x7f85600052e0 2024-03-28T19:27:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.190+0000 7f8580200640 1 -- 172.21.15.27:0/977774857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 msgr2=0x7f856000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.190+0000 7f8580200640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f856000b9c0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f857000a830 tx=0x7f8560005830 comp rx=0 tx=0).stop 2024-03-28T19:27:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.190+0000 7f8580200640 1 -- 172.21.15.27:0/977774857 shutdown_connections 2024-03-28T19:27:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.190+0000 7f8580200640 1 --2- 172.21.15.27:0/977774857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f856000b9c0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.190+0000 7f8580200640 1 -- 172.21.15.27:0/977774857 >> 172.21.15.27:0/977774857 conn(0x7f857806b0c0 msgr2=0x7f857806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.190+0000 7f8580200640 1 -- 172.21.15.27:0/977774857 shutdown_connections 2024-03-28T19:27:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.190+0000 7f8580200640 1 -- 172.21.15.27:0/977774857 wait complete. 2024-03-28T19:27:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.192+0000 7f8580200640 1 Processor -- start 2024-03-28T19:27:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.192+0000 7f8580200640 1 -- start start 2024-03-28T19:27:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f8580200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f85780c0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f8580200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8578151a50 0x7f8578151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f8580200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85780c6af0 0x7f85780c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f8580200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8578062910 con 0x7f85600052e0 2024-03-28T19:27:32.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f8580200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857806eef0 con 0x7f85780c6af0 2024-03-28T19:27:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f8580200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8578150d40 con 0x7f8578151a50 2024-03-28T19:27:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f857d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f85780c0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f857d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f85780c0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58224/0 (socket says 172.21.15.27:58224) 2024-03-28T19:27:32.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.193+0000 7f857d800640 1 -- 172.21.15.27:0/2891537128 learned_addr learned my addr 172.21.15.27:0/2891537128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:32.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857ce00640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8578151a50 0x7f8578151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:32.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857e200640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85780c6af0 0x7f85780c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:32.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857d800640 1 -- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8578151a50 msgr2=0x7f8578151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:32.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857d800640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8578151a50 0x7f8578151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857d800640 1 -- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85780c6af0 msgr2=0x7f85780c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857d800640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85780c6af0 0x7f85780c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857d800640 1 -- 172.21.15.27:0/2891537128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8578155990 con 0x7f85600052e0 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857e200640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85780c6af0 0x7f85780c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f857d800640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f85780c0a90 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f856c011910 tx=0x7f856c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f8575600640 1 -- 172.21.15.27:0/2891537128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856c00fa90 con 0x7f85600052e0 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.194+0000 7f8575600640 1 -- 172.21.15.27:0/2891537128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856c00fc30 con 0x7f85600052e0 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.195+0000 7f8575600640 1 -- 172.21.15.27:0/2891537128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856c014440 con 0x7f85600052e0 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.195+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8578154800 con 0x7f85600052e0 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.195+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8578152880 con 0x7f85600052e0 2024-03-28T19:27:32.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.196+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857806c050 con 0x7f85600052e0 2024-03-28T19:27:32.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.197+0000 7f8575600640 1 -- 172.21.15.27:0/2891537128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f856c012040 con 0x7f85600052e0 2024-03-28T19:27:32.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.198+0000 7f8575600640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f855806bc00 0x7f855806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.199+0000 7f8575600640 1 -- 172.21.15.27:0/2891537128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f856c08c1b0 con 0x7f85600052e0 2024-03-28T19:27:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.199+0000 7f857ce00640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f855806bc00 0x7f855806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:32.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.200+0000 7f857ce00640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f855806bc00 0x7f855806e0c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f85780ad800 tx=0x7f8564009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:32.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.204+0000 7f8575600640 1 -- 172.21.15.27:0/2891537128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f856c019030 con 0x7f85600052e0 2024-03-28T19:27:32.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.465+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f857806c490 con 0x7f85600052e0 2024-03-28T19:27:32.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.466+0000 7f8575600640 1 -- 172.21.15.27:0/2891537128 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f857806c490 con 0x7f85600052e0 2024-03-28T19:27:32.465 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:32.465 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9030,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:32.465 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 235 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.468+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f855806bc00 msgr2=0x7f855806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.468+0000 7f8580200640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f855806bc00 0x7f855806e0c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f85780ad800 tx=0x7f8564009040 comp rx=0 tx=0).stop 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.468+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 msgr2=0x7f85780c0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.468+0000 7f8580200640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f85780c0a90 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f856c011910 tx=0x7f856c011dd0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.469+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 shutdown_connections 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.469+0000 7f8580200640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f855806bc00 0x7f855806e0c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.469+0000 7f8580200640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85780c6af0 0x7f85780c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.469+0000 7f8580200640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8578151a50 0x7f8578151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.469+0000 7f8580200640 1 --2- 172.21.15.27:0/2891537128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85600052e0 0x7f85780c0a90 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.469+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 >> 172.21.15.27:0/2891537128 conn(0x7f857806b0c0 msgr2=0x7f857814e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.469+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 shutdown_connections 2024-03-28T19:27:32.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:32.469+0000 7f8580200640 1 -- 172.21.15.27:0/2891537128 wait complete. 2024-03-28T19:27:32.487 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:27:34.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:34.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.743+0000 7f27bca00640 1 Processor -- start 2024-03-28T19:27:34.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.743+0000 7f27bca00640 1 -- start start 2024-03-28T19:27:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.745+0000 7f27bca00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f27b80c8ed0 0x7f27b80c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.745+0000 7f27bca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b8158bd0 con 0x7f27b80c58c0 2024-03-28T19:27:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.745+0000 7f27bca00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b8158eb0 con 0x7f27b80be5d0 2024-03-28T19:27:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.745+0000 7f27bca00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b8159190 con 0x7f27b80c8ed0 2024-03-28T19:27:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.745+0000 7f27b6800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f27b80c58c0 0x7f27b81555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47874/0 (socket says 172.21.15.27:47874) 2024-03-28T19:27:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.745+0000 7f27b6800640 1 -- 172.21.15.27:0/44241407 learned_addr learned my addr 172.21.15.27:0/44241407 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:34.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.745+0000 7f27b5e00640 1 --2- 172.21.15.27:0/44241407 >> v2:172.21.15.155:3300/0 conn(0x7f27b80c8ed0 0x7f27b80c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:34.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.746+0000 7f27b5400640 1 -- 172.21.15.27:0/44241407 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133970510 0 0) 0x7f27b8158bd0 con 0x7f27b80c58c0 2024-03-28T19:27:34.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.746+0000 7f27b5400640 1 -- 172.21.15.27:0/44241407 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f279c003520 con 0x7f27b80c58c0 2024-03-28T19:27:34.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.746+0000 7f27b5e00640 1 -- 172.21.15.27:0/44241407 >> v1:172.21.15.155:6789/0 conn(0x7f27b80be5d0 legacy=0x7f27b8155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:34.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.746+0000 7f27b5e00640 1 -- 172.21.15.27:0/44241407 >> v1:172.21.15.27:6789/0 conn(0x7f27b80c58c0 legacy=0x7f27b81555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:34.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.746+0000 7f27b5e00640 1 -- 172.21.15.27:0/44241407 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b8159470 con 0x7f27b80c8ed0 2024-03-28T19:27:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.748+0000 7f27b5e00640 1 --2- 172.21.15.27:0/44241407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c8ed0 0x7f27b80c5ca0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f27a40097a0 tx=0x7f27a402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac7a28bbde2ab874 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:34.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.749+0000 7f27b5400640 1 -- 172.21.15.27:0/44241407 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a40315a0 con 0x7f27b80c8ed0 2024-03-28T19:27:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.749+0000 7f27b5400640 1 -- 172.21.15.27:0/44241407 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a4037070 con 0x7f27b80c8ed0 2024-03-28T19:27:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.749+0000 7f27b5400640 1 -- 172.21.15.27:0/44241407 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4031960 con 0x7f27b80c8ed0 2024-03-28T19:27:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.749+0000 7f27bca00640 1 -- 172.21.15.27:0/44241407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c8ed0 msgr2=0x7f27b80c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:34.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.749+0000 7f27bca00640 1 --2- 172.21.15.27:0/44241407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c8ed0 0x7f27b80c5ca0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f27a40097a0 tx=0x7f27a402b840 comp rx=0 tx=0).stop 2024-03-28T19:27:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.749+0000 7f27bca00640 1 -- 172.21.15.27:0/44241407 shutdown_connections 2024-03-28T19:27:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.749+0000 7f27bca00640 1 --2- 172.21.15.27:0/44241407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c8ed0 0x7f27b80c5ca0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.749+0000 7f27bca00640 1 -- 172.21.15.27:0/44241407 >> 172.21.15.27:0/44241407 conn(0x7f27b806b6f0 msgr2=0x7f27b80acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.750+0000 7f27bca00640 1 -- 172.21.15.27:0/44241407 shutdown_connections 2024-03-28T19:27:34.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.750+0000 7f27bca00640 1 -- 172.21.15.27:0/44241407 wait complete. 2024-03-28T19:27:34.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.751+0000 7f27bca00640 1 Processor -- start 2024-03-28T19:27:34.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.752+0000 7f27bca00640 1 -- start start 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27bca00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27b80be5d0 0x7f27b80c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27bca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c58c0 0x7f27b80ab060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27bca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b80c8ed0 0x7f27b80ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27bca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b80b09a0 con 0x7f27b80c8ed0 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27bca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b80b0820 con 0x7f27b80c58c0 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27bca00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b80ad7d0 con 0x7f27b80be5d0 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27b7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b80c8ed0 0x7f27b80ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27b7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b80c8ed0 0x7f27b80ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58240/0 (socket says 172.21.15.27:58240) 2024-03-28T19:27:34.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27b7200640 1 -- 172.21.15.27:0/3796165715 learned_addr learned my addr 172.21.15.27:0/3796165715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27b6800640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27b80be5d0 0x7f27b80c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.753+0000 7f27b5e00640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c58c0 0x7f27b80ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27b7200640 1 -- 172.21.15.27:0/3796165715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27b80be5d0 msgr2=0x7f27b80c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27b7200640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27b80be5d0 0x7f27b80c2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27b7200640 1 -- 172.21.15.27:0/3796165715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c58c0 msgr2=0x7f27b80ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27b7200640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c58c0 0x7f27b80ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27b7200640 1 -- 172.21.15.27:0/3796165715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b80ac880 con 0x7f27b80c8ed0 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27b7200640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b80c8ed0 0x7f27b80ab5a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f27b000c7d0 tx=0x7f27b000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27aea00640 1 -- 172.21.15.27:0/3796165715 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0003b70 con 0x7f27b80c8ed0 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27aea00640 1 -- 172.21.15.27:0/3796165715 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b0003d30 con 0x7f27b80c8ed0 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27aea00640 1 -- 172.21.15.27:0/3796165715 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b000aa90 con 0x7f27b80c8ed0 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.754+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b8155400 con 0x7f27b80c8ed0 2024-03-28T19:27:34.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.755+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b80ace00 con 0x7f27b80c8ed0 2024-03-28T19:27:34.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.756+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b8069640 con 0x7f27b80c8ed0 2024-03-28T19:27:34.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.757+0000 7f27aea00640 1 -- 172.21.15.27:0/3796165715 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f27b000ac30 con 0x7f27b80c8ed0 2024-03-28T19:27:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.758+0000 7f27aea00640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f279806bc00 0x7f279806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.758+0000 7f27aea00640 1 -- 172.21.15.27:0/3796165715 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 10249+0+0 (secure 0 0 0) 0x7f27b008bb80 con 0x7f27b80c8ed0 2024-03-28T19:27:34.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.758+0000 7f27b6800640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f279806bc00 0x7f279806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:34.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.759+0000 7f27b6800640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f279806bc00 0x7f279806e0c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f27a80011e0 tx=0x7f27a8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:34.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:34.763+0000 7f27aea00640 1 -- 172.21.15.27:0/3796165715 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f27b0059490 con 0x7f27b80c8ed0 2024-03-28T19:27:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.018+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27b8156580 con 0x7f27b80c8ed0 2024-03-28T19:27:35.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.019+0000 7f27aea00640 1 -- 172.21.15.27:0/3796165715 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f27b0058d90 con 0x7f27b80c8ed0 2024-03-28T19:27:35.018 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:35.018 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9030,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:11.991376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8928},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8928":{"gid":8928,"name":"c","rank":0,"incarnation":230,"state":"up:active","state_seq":16,"addr":"172.21.15.27:6837/1262510798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":1262510798},{"type":"v1","addr":"172.21.15.27:6837","nonce":1262510798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8928,"qdb_cluster":[8928]},"id":2}]} 2024-03-28T19:27:35.019 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 235 2024-03-28T19:27:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f279806bc00 msgr2=0x7f279806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f279806bc00 0x7f279806e0c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f27a80011e0 tx=0x7f27a8009040 comp rx=0 tx=0).stop 2024-03-28T19:27:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b80c8ed0 msgr2=0x7f27b80ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b80c8ed0 0x7f27b80ab5a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f27b000c7d0 tx=0x7f27b000cc90 comp rx=0 tx=0).stop 2024-03-28T19:27:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 shutdown_connections 2024-03-28T19:27:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f279806bc00 0x7f279806e0c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27b80c8ed0 0x7f27b80ab5a0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27b80c58c0 0x7f27b80ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 --2- 172.21.15.27:0/3796165715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27b80be5d0 0x7f27b80c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 >> 172.21.15.27:0/3796165715 conn(0x7f27b806b6f0 msgr2=0x7f27b80c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 shutdown_connections 2024-03-28T19:27:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:35.022+0000 7f27bca00640 1 -- 172.21.15.27:0/3796165715 wait complete. 2024-03-28T19:27:35.040 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:27:37.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.282+0000 7f39a7200640 1 Processor -- start 2024-03-28T19:27:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.283+0000 7f39a7200640 1 -- start start 2024-03-28T19:27:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.284+0000 7f39a7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c58c0 0x7f39a00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.284+0000 7f39a7200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39a00c8ed0 0x7f39a01555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.285+0000 7f39a7200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a0156230 con 0x7f39a00be5d0 2024-03-28T19:27:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.285+0000 7f39a7200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a0158bd0 con 0x7f39a00c8ed0 2024-03-28T19:27:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.285+0000 7f39a7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a0158eb0 con 0x7f39a00c58c0 2024-03-28T19:27:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.285+0000 7f39a5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39a00c8ed0 0x7f39a01555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.285+0000 7f39a5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39a00c8ed0 0x7f39a01555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38166/0 (socket says 172.21.15.27:38166) 2024-03-28T19:27:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.285+0000 7f39a5400640 1 -- 172.21.15.27:0/89464294 learned_addr learned my addr 172.21.15.27:0/89464294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.285+0000 7f39a5e00640 1 --2- 172.21.15.27:0/89464294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c58c0 0x7f39a00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.286+0000 7f39a5400640 1 -- 172.21.15.27:0/89464294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c58c0 msgr2=0x7f39a00c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.286+0000 7f39a5400640 1 --2- 172.21.15.27:0/89464294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c58c0 0x7f39a00c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.286+0000 7f39a5400640 1 -- 172.21.15.27:0/89464294 >> v1:172.21.15.27:6789/0 conn(0x7f39a00be5d0 legacy=0x7f39a0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.286+0000 7f39a5400640 1 -- 172.21.15.27:0/89464294 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a0159190 con 0x7f39a00c8ed0 2024-03-28T19:27:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.287+0000 7f39a5400640 1 --2- 172.21.15.27:0/89464294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c8ed0 0x7f39a01555e0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f3990009870 tx=0x7f399002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f213c06a1536105 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.288+0000 7f39a4a00640 1 -- 172.21.15.27:0/89464294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990035020 con 0x7f39a00c8ed0 2024-03-28T19:27:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.288+0000 7f39a4a00640 1 -- 172.21.15.27:0/89464294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3990031640 con 0x7f39a00c8ed0 2024-03-28T19:27:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.288+0000 7f39a4a00640 1 -- 172.21.15.27:0/89464294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399002d040 con 0x7f39a00c8ed0 2024-03-28T19:27:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.288+0000 7f39a7200640 1 -- 172.21.15.27:0/89464294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c8ed0 msgr2=0x7f39a01555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.288+0000 7f39a7200640 1 --2- 172.21.15.27:0/89464294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c8ed0 0x7f39a01555e0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f3990009870 tx=0x7f399002b8f0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.289+0000 7f39a7200640 1 -- 172.21.15.27:0/89464294 shutdown_connections 2024-03-28T19:27:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.289+0000 7f39a7200640 1 --2- 172.21.15.27:0/89464294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c8ed0 0x7f39a01555e0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.289+0000 7f39a7200640 1 --2- 172.21.15.27:0/89464294 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c58c0 0x7f39a00c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.289+0000 7f39a7200640 1 -- 172.21.15.27:0/89464294 >> 172.21.15.27:0/89464294 conn(0x7f39a006b6f0 msgr2=0x7f39a00acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.289+0000 7f39a7200640 1 -- 172.21.15.27:0/89464294 shutdown_connections 2024-03-28T19:27:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.289+0000 7f39a7200640 1 -- 172.21.15.27:0/89464294 wait complete. 2024-03-28T19:27:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.291+0000 7f39a7200640 1 Processor -- start 2024-03-28T19:27:37.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.291+0000 7f39a7200640 1 -- start start 2024-03-28T19:27:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39a00be5d0 0x7f39a0153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c58c0 0x7f39a0155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a7200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c8ed0 0x7f39a00c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a00b0820 con 0x7f39a00c58c0 2024-03-28T19:27:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a0159750 con 0x7f39a00be5d0 2024-03-28T19:27:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a7200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a00b09a0 con 0x7f39a00c8ed0 2024-03-28T19:27:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c58c0 0x7f39a0155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c58c0 0x7f39a0155530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38182/0 (socket says 172.21.15.27:38182) 2024-03-28T19:27:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39a00be5d0 0x7f39a0153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a5400640 1 -- 172.21.15.27:0/4111990032 learned_addr learned my addr 172.21.15.27:0/4111990032 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.292+0000 7f39a6800640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c8ed0 0x7f39a00c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f39a5400640 1 -- 172.21.15.27:0/4111990032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c8ed0 msgr2=0x7f39a00c4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f39a5400640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c8ed0 0x7f39a00c4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f39a5400640 1 -- 172.21.15.27:0/4111990032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39a00be5d0 msgr2=0x7f39a0153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f39a5400640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39a00be5d0 0x7f39a0153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f39a5400640 1 -- 172.21.15.27:0/4111990032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a0154d40 con 0x7f39a00c58c0 2024-03-28T19:27:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f39a5e00640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39a00be5d0 0x7f39a0153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:27:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f39a5400640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c58c0 0x7f39a0155530 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f3990000c00 tx=0x7f39900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f399a000640 1 -- 172.21.15.27:0/4111990032 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990040070 con 0x7f39a00c58c0 2024-03-28T19:27:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f399a000640 1 -- 172.21.15.27:0/4111990032 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39900399f0 con 0x7f39a00c58c0 2024-03-28T19:27:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a01ec210 con 0x7f39a00c58c0 2024-03-28T19:27:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.293+0000 7f399a000640 1 -- 172.21.15.27:0/4111990032 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3990035040 con 0x7f39a00c58c0 2024-03-28T19:27:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.294+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a00c2290 con 0x7f39a00c58c0 2024-03-28T19:27:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.295+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a0069640 con 0x7f39a00c58c0 2024-03-28T19:27:37.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.296+0000 7f399a000640 1 -- 172.21.15.27:0/4111990032 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f399004b060 con 0x7f39a00c58c0 2024-03-28T19:27:37.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.298+0000 7f399a000640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f398406bbd0 0x7f398406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.299+0000 7f399a000640 1 -- 172.21.15.27:0/4111990032 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f39900ad360 con 0x7f39a00c58c0 2024-03-28T19:27:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.299+0000 7f39a5e00640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f398406bbd0 0x7f398406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.300+0000 7f39a5e00640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f398406bbd0 0x7f398406e090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3994009e90 tx=0x7f3994009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:37.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.303+0000 7f399a000640 1 -- 172.21.15.27:0/4111990032 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f399007a5d0 con 0x7f39a00c58c0 2024-03-28T19:27:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.589+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39a0155ef0 con 0x7f39a00c58c0 2024-03-28T19:27:37.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.590+0000 7f399a000640 1 -- 172.21.15.27:0/4111990032 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f399007e670 con 0x7f39a00c58c0 2024-03-28T19:27:37.589 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:37.590 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:35.516980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:37.590 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 236 2024-03-28T19:27:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f398406bbd0 msgr2=0x7f398406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f398406bbd0 0x7f398406e090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3994009e90 tx=0x7f3994009190 comp rx=0 tx=0).stop 2024-03-28T19:27:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c58c0 msgr2=0x7f39a0155530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c58c0 0x7f39a0155530 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f3990000c00 tx=0x7f39900046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 shutdown_connections 2024-03-28T19:27:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f398406bbd0 0x7f398406e090 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f39a00c8ed0 0x7f39a00c4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a00c58c0 0x7f39a0155530 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 --2- 172.21.15.27:0/4111990032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39a00be5d0 0x7f39a0153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 >> 172.21.15.27:0/4111990032 conn(0x7f39a006b6f0 msgr2=0x7f39a00ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 shutdown_connections 2024-03-28T19:27:37.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.593+0000 7f39a7200640 1 -- 172.21.15.27:0/4111990032 wait complete. 2024-03-28T19:27:37.615 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-03-28T19:27:37.615 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:27:37.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.858+0000 7f1585400640 1 Processor -- start 2024-03-28T19:27:37.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.858+0000 7f1585400640 1 -- start start 2024-03-28T19:27:37.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.860+0000 7f1585400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f158006c0d0 0x7f158006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.860+0000 7f1585400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15800a7400 con 0x7f15800a6e90 2024-03-28T19:27:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.860+0000 7f1585400640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1580158b40 con 0x7f15800c1780 2024-03-28T19:27:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.860+0000 7f1585400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1580159470 con 0x7f158006c0d0 2024-03-28T19:27:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.860+0000 7f157fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f158006c0d0 0x7f158006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.860+0000 7f157fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f158006c0d0 0x7f158006c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38192/0 (socket says 172.21.15.27:38192) 2024-03-28T19:27:37.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.860+0000 7f157fe00640 1 -- 172.21.15.27:0/883648350 learned_addr learned my addr 172.21.15.27:0/883648350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:37.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.861+0000 7f157fe00640 1 -- 172.21.15.27:0/883648350 >> v1:172.21.15.155:6789/0 conn(0x7f15800c1780 legacy=0x7f158015fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.861+0000 7f157fe00640 1 -- 172.21.15.27:0/883648350 >> v1:172.21.15.27:6789/0 conn(0x7f15800a6e90 legacy=0x7f1580155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.861+0000 7f157fe00640 1 -- 172.21.15.27:0/883648350 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1580158e20 con 0x7f158006c0d0 2024-03-28T19:27:37.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.862+0000 7f157fe00640 1 --2- 172.21.15.27:0/883648350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f158006c0d0 0x7f158006c4b0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f1570009870 tx=0x7f157002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca727f111c4ef03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:37.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.862+0000 7f157ea00640 1 -- 172.21.15.27:0/883648350 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1570034070 con 0x7f158006c0d0 2024-03-28T19:27:37.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.862+0000 7f157ea00640 1 -- 172.21.15.27:0/883648350 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1570030850 con 0x7f158006c0d0 2024-03-28T19:27:37.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.862+0000 7f157ea00640 1 -- 172.21.15.27:0/883648350 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1570030c20 con 0x7f158006c0d0 2024-03-28T19:27:37.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.863+0000 7f1585400640 1 -- 172.21.15.27:0/883648350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f158006c0d0 msgr2=0x7f158006c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.863+0000 7f1585400640 1 --2- 172.21.15.27:0/883648350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f158006c0d0 0x7f158006c4b0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f1570009870 tx=0x7f157002b990 comp rx=0 tx=0).stop 2024-03-28T19:27:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.863+0000 7f1585400640 1 -- 172.21.15.27:0/883648350 shutdown_connections 2024-03-28T19:27:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.863+0000 7f1585400640 1 --2- 172.21.15.27:0/883648350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f158006c0d0 0x7f158006c4b0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.863+0000 7f1585400640 1 -- 172.21.15.27:0/883648350 >> 172.21.15.27:0/883648350 conn(0x7f15800c7aa0 msgr2=0x7f15800c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.864+0000 7f1585400640 1 -- 172.21.15.27:0/883648350 shutdown_connections 2024-03-28T19:27:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.864+0000 7f1585400640 1 -- 172.21.15.27:0/883648350 wait complete. 2024-03-28T19:27:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.866+0000 7f1585400640 1 Processor -- start 2024-03-28T19:27:37.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.866+0000 7f1585400640 1 -- start start 2024-03-28T19:27:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1585400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f158006c0d0 0x7f15801ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1585400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15800a6e90 0x7f15800ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1585400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15800c1780 0x7f15801ebb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1585400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1580152e20 con 0x7f15800c1780 2024-03-28T19:27:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1585400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1580159900 con 0x7f15800a6e90 2024-03-28T19:27:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1585400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1580152fa0 con 0x7f158006c0d0 2024-03-28T19:27:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1584a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15800c1780 0x7f15801ebb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1584a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15800c1780 0x7f15801ebb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38206/0 (socket says 172.21.15.27:38206) 2024-03-28T19:27:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1584a00640 1 -- 172.21.15.27:0/455365782 learned_addr learned my addr 172.21.15.27:0/455365782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f157f400640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15800a6e90 0x7f15800ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1584a00640 1 -- 172.21.15.27:0/455365782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f158006c0d0 msgr2=0x7f15801ec5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1584a00640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f158006c0d0 0x7f15801ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1584a00640 1 -- 172.21.15.27:0/455365782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15800a6e90 msgr2=0x7f15800ac020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1584a00640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15800a6e90 0x7f15800ac020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.867+0000 7f1584a00640 1 -- 172.21.15.27:0/455365782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15801ef050 con 0x7f15800c1780 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.868+0000 7f1584a00640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15800c1780 0x7f15801ebb10 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f157800c6b0 tx=0x7f157800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.868+0000 7f1577e00640 1 -- 172.21.15.27:0/455365782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1578003b70 con 0x7f15800c1780 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.868+0000 7f1577e00640 1 -- 172.21.15.27:0/455365782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1578003d30 con 0x7f15800c1780 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.868+0000 7f1577e00640 1 -- 172.21.15.27:0/455365782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1578005310 con 0x7f15800c1780 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.868+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15801ef330 con 0x7f15800c1780 2024-03-28T19:27:37.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.868+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15801eb5f0 con 0x7f15800c1780 2024-03-28T19:27:37.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.870+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1580072270 con 0x7f15800c1780 2024-03-28T19:27:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.871+0000 7f1577e00640 1 -- 172.21.15.27:0/455365782 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f15780054b0 con 0x7f15800c1780 2024-03-28T19:27:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.872+0000 7f1577e00640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f156006bbd0 0x7f156006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.872+0000 7f1577e00640 1 -- 172.21.15.27:0/455365782 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f157808bb30 con 0x7f15800c1780 2024-03-28T19:27:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.873+0000 7f157fe00640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f156006bbd0 0x7f156006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:37.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.873+0000 7f157fe00640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f156006bbd0 0x7f156006e090 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1570036000 tx=0x7f157003b040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:37.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:37.878+0000 7f1577e00640 1 -- 172.21.15.27:0/455365782 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1580072270 con 0x7f15800c1780 2024-03-28T19:27:38.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.157+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1580069640 con 0x7f15800c1780 2024-03-28T19:27:38.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.159+0000 7f1577e00640 1 -- 172.21.15.27:0/455365782 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f15780593f0 con 0x7f15800c1780 2024-03-28T19:27:38.158 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:38.158 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:35.516980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:38.158 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 236 2024-03-28T19:27:38.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f156006bbd0 msgr2=0x7f156006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:38.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f156006bbd0 0x7f156006e090 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1570036000 tx=0x7f157003b040 comp rx=0 tx=0).stop 2024-03-28T19:27:38.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15800c1780 msgr2=0x7f15801ebb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:38.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15800c1780 0x7f15801ebb10 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f157800c6b0 tx=0x7f157800cb70 comp rx=0 tx=0).stop 2024-03-28T19:27:38.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 shutdown_connections 2024-03-28T19:27:38.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f156006bbd0 0x7f156006e090 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:38.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15800c1780 0x7f15801ebb10 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:38.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15800a6e90 0x7f15800ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:38.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 --2- 172.21.15.27:0/455365782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f158006c0d0 0x7f15801ec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:38.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 >> 172.21.15.27:0/455365782 conn(0x7f15800c7aa0 msgr2=0x7f1580066dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:38.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 shutdown_connections 2024-03-28T19:27:38.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:38.162+0000 7f1585400640 1 -- 172.21.15.27:0/455365782 wait complete. 2024-03-28T19:27:38.183 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 236, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 226, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6740": { "addr": "172.21.15.155:6835/1734256169", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6833", "nonce": 1734256169, "type": "v2" }, { "addr": "172.21.15.155:6835", "nonce": 1734256169, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6740, "incarnation": 222, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 92, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:26:16.003326+0000", "qdb_cluster": [ 6740 ], "qdb_leader": 6740, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6740 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 236, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_9030": { "addr": "172.21.15.155:6834/1152401013", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 1152401013, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 1152401013, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9030, "incarnation": 236, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 94, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:27:35.516980+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9030 } } } ], "standbys": [ { "addr": "172.21.15.27:6835/796801387", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 796801387, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 796801387, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 228, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8913, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:27:40.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.429+0000 7f4ab2e00640 1 Processor -- start 2024-03-28T19:27:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.429+0000 7f4ab2e00640 1 -- start start 2024-03-28T19:27:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.431+0000 7f4ab2e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4aac0ade40 0x7f4aac15ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.431+0000 7f4ab2e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0ad680 con 0x7f4aac0c8470 2024-03-28T19:27:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.431+0000 7f4ab2e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0c4de0 con 0x7f4aac150b60 2024-03-28T19:27:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.431+0000 7f4ab2e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aac157210 con 0x7f4aac0ade40 2024-03-28T19:27:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.431+0000 7f4ab0e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4aac0ade40 0x7f4aac15ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:40.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.431+0000 7f4ab0e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4aac0ade40 0x7f4aac15ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38216/0 (socket says 172.21.15.27:38216) 2024-03-28T19:27:40.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.431+0000 7f4ab0e00640 1 -- 172.21.15.27:0/1321623679 learned_addr learned my addr 172.21.15.27:0/1321623679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:40.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.432+0000 7f4ab0e00640 1 -- 172.21.15.27:0/1321623679 >> v1:172.21.15.155:6789/0 conn(0x7f4aac150b60 legacy=0x7f4aac156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:40.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.432+0000 7f4ab0e00640 1 -- 172.21.15.27:0/1321623679 >> v1:172.21.15.27:6789/0 conn(0x7f4aac0c8470 legacy=0x7f4aac0c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:40.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.432+0000 7f4ab0e00640 1 -- 172.21.15.27:0/1321623679 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac0c50c0 con 0x7f4aac0ade40 2024-03-28T19:27:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.433+0000 7f4ab0e00640 1 --2- 172.21.15.27:0/1321623679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0ade40 0x7f4aac15ee60 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40098a0 tx=0x7f4aa402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0cc4e54473d045 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.433+0000 7f4aaaa00640 1 -- 172.21.15.27:0/1321623679 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4034070 con 0x7f4aac0ade40 2024-03-28T19:27:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.433+0000 7f4aaaa00640 1 -- 172.21.15.27:0/1321623679 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4030840 con 0x7f4aac0ade40 2024-03-28T19:27:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.433+0000 7f4aaaa00640 1 -- 172.21.15.27:0/1321623679 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4030c10 con 0x7f4aac0ade40 2024-03-28T19:27:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.434+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1321623679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0ade40 msgr2=0x7f4aac15ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.434+0000 7f4ab2e00640 1 --2- 172.21.15.27:0/1321623679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0ade40 0x7f4aac15ee60 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40098a0 tx=0x7f4aa402b980 comp rx=0 tx=0).stop 2024-03-28T19:27:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.434+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1321623679 shutdown_connections 2024-03-28T19:27:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.434+0000 7f4ab2e00640 1 --2- 172.21.15.27:0/1321623679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac0ade40 0x7f4aac15ee60 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:40.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.434+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1321623679 >> 172.21.15.27:0/1321623679 conn(0x7f4aac0aaab0 msgr2=0x7f4aac0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:40.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.434+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1321623679 shutdown_connections 2024-03-28T19:27:40.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.434+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1321623679 wait complete. 2024-03-28T19:27:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.436+0000 7f4ab2e00640 1 Processor -- start 2024-03-28T19:27:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.437+0000 7f4ab2e00640 1 -- start start 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab2e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4aac0ade40 0x7f4aac0c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab2e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4aac0c8470 0x7f4aac14bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab2e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac150b60 0x7f4aac14c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab2e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac150f40 con 0x7f4aac150b60 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab2e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0ad9b0 con 0x7f4aac0c8470 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab2e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0a85a0 con 0x7f4aac0ade40 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac150b60 0x7f4aac14c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac150b60 0x7f4aac14c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38228/0 (socket says 172.21.15.27:38228) 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4ab0e00640 1 -- 172.21.15.27:0/1078343711 learned_addr learned my addr 172.21.15.27:0/1078343711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4aab400640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4aac0c8470 0x7f4aac14bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.438+0000 7f4aabe00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4aac0ade40 0x7f4aac0c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.439+0000 7f4ab0e00640 1 -- 172.21.15.27:0/1078343711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4aac0ade40 msgr2=0x7f4aac0c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.439+0000 7f4ab0e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4aac0ade40 0x7f4aac0c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.439+0000 7f4ab0e00640 1 -- 172.21.15.27:0/1078343711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4aac0c8470 msgr2=0x7f4aac14bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.439+0000 7f4ab0e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4aac0c8470 0x7f4aac14bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.439+0000 7f4ab0e00640 1 -- 172.21.15.27:0/1078343711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac06c340 con 0x7f4aac150b60 2024-03-28T19:27:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.439+0000 7f4ab0e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac150b60 0x7f4aac14c310 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f4aa402b9c0 tx=0x7f4aa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.440+0000 7f4aa3e00640 1 -- 172.21.15.27:0/1078343711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4034040 con 0x7f4aac150b60 2024-03-28T19:27:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.440+0000 7f4aa3e00640 1 -- 172.21.15.27:0/1078343711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4039d90 con 0x7f4aac150b60 2024-03-28T19:27:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.440+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac0c7c50 con 0x7f4aac150b60 2024-03-28T19:27:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.440+0000 7f4aa3e00640 1 -- 172.21.15.27:0/1078343711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4043610 con 0x7f4aac150b60 2024-03-28T19:27:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.440+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac06c890 con 0x7f4aac150b60 2024-03-28T19:27:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.441+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aac04f400 con 0x7f4aac150b60 2024-03-28T19:27:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.442+0000 7f4aa3e00640 1 -- 172.21.15.27:0/1078343711 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4aa4037070 con 0x7f4aac150b60 2024-03-28T19:27:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.443+0000 7f4aa3e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a8806bbd0 0x7f4a8806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.444+0000 7f4aa3e00640 1 -- 172.21.15.27:0/1078343711 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f4aa40ade30 con 0x7f4aac150b60 2024-03-28T19:27:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.444+0000 7f4aabe00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a8806bbd0 0x7f4a8806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:40.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.445+0000 7f4aabe00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a8806bbd0 0x7f4a8806e090 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0012e0 tx=0x7f4a9c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:40.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.449+0000 7f4aa3e00640 1 -- 172.21.15.27:0/1078343711 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4aa403e070 con 0x7f4aac150b60 2024-03-28T19:27:40.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.727+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aac0c4a20 con 0x7f4aac150b60 2024-03-28T19:27:40.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.728+0000 7f4aa3e00640 1 -- 172.21.15.27:0/1078343711 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f4aa407b0a0 con 0x7f4aac150b60 2024-03-28T19:27:40.727 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:40.728 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:35.516980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:40.728 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 236 2024-03-28T19:27:40.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a8806bbd0 msgr2=0x7f4a8806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:40.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a8806bbd0 0x7f4a8806e090 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0012e0 tx=0x7f4a9c009040 comp rx=0 tx=0).stop 2024-03-28T19:27:40.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac150b60 msgr2=0x7f4aac14c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:40.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac150b60 0x7f4aac14c310 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f4aa402b9c0 tx=0x7f4aa40046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:40.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 shutdown_connections 2024-03-28T19:27:40.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f4a8806bbd0 0x7f4a8806e090 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:40.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4aac150b60 0x7f4aac14c310 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:40.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4aac0c8470 0x7f4aac14bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:40.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 --2- 172.21.15.27:0/1078343711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4aac0ade40 0x7f4aac0c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:40.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.731+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 >> 172.21.15.27:0/1078343711 conn(0x7f4aac0aaab0 msgr2=0x7f4aac0a88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:40.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.732+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 shutdown_connections 2024-03-28T19:27:40.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:40.732+0000 7f4ab2e00640 1 -- 172.21.15.27:0/1078343711 wait complete. 2024-03-28T19:27:40.752 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:27:42.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:42.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.990+0000 7f4a18e00640 1 Processor -- start 2024-03-28T19:27:42.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.990+0000 7f4a18e00640 1 -- start start 2024-03-28T19:27:42.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.991+0000 7f4a18e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a14150ae0 0x7f4a140a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:42.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.991+0000 7f4a18e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a14153a30 0x7f4a14153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:42.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.991+0000 7f4a18e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a14150ec0 con 0x7f4a140a61a0 2024-03-28T19:27:42.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.991+0000 7f4a18e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1406dc70 con 0x7f4a14153a30 2024-03-28T19:27:42.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.991+0000 7f4a18e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14153550 con 0x7f4a14150ae0 2024-03-28T19:27:42.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.992+0000 7f4a12a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a140a61a0 0x7f4a140c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38672/0 (socket says 172.21.15.27:38672) 2024-03-28T19:27:42.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.992+0000 7f4a12a00640 1 -- 172.21.15.27:0/1289786791 learned_addr learned my addr 172.21.15.27:0/1289786791 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:42.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.992+0000 7f4a13e00640 1 --2- 172.21.15.27:0/1289786791 >> v2:172.21.15.27:3300/0 conn(0x7f4a14153a30 0x7f4a14153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:42.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.992+0000 7f4a13400640 1 --2- 172.21.15.27:0/1289786791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a14150ae0 0x7f4a140a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:42.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.992+0000 7f4a13e00640 1 -- 172.21.15.27:0/1289786791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a14150ae0 msgr2=0x7f4a140a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:42.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.992+0000 7f4a13e00640 1 --2- 172.21.15.27:0/1289786791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a14150ae0 0x7f4a140a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:42.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.992+0000 7f4a13e00640 1 -- 172.21.15.27:0/1289786791 >> v1:172.21.15.27:6789/0 conn(0x7f4a140a61a0 legacy=0x7f4a140c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:42.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.992+0000 7f4a13e00640 1 -- 172.21.15.27:0/1289786791 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1406df10 con 0x7f4a14153a30 2024-03-28T19:27:42.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.994+0000 7f4a13e00640 1 --2- 172.21.15.27:0/1289786791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14153a30 0x7f4a14153ea0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009d90 tx=0x7f4a0c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46675a41bd10ea3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:42.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.994+0000 7f4a12000640 1 -- 172.21.15.27:0/1289786791 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c039070 con 0x7f4a14153a30 2024-03-28T19:27:42.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.994+0000 7f4a12000640 1 -- 172.21.15.27:0/1289786791 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c02cbc0 con 0x7f4a14153a30 2024-03-28T19:27:42.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.994+0000 7f4a12000640 1 -- 172.21.15.27:0/1289786791 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c034440 con 0x7f4a14153a30 2024-03-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.995+0000 7f4a18e00640 1 -- 172.21.15.27:0/1289786791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14153a30 msgr2=0x7f4a14153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.995+0000 7f4a18e00640 1 --2- 172.21.15.27:0/1289786791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14153a30 0x7f4a14153ea0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009d90 tx=0x7f4a0c02be10 comp rx=0 tx=0).stop 2024-03-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.995+0000 7f4a18e00640 1 -- 172.21.15.27:0/1289786791 shutdown_connections 2024-03-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.995+0000 7f4a18e00640 1 --2- 172.21.15.27:0/1289786791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14153a30 0x7f4a14153ea0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.995+0000 7f4a18e00640 1 --2- 172.21.15.27:0/1289786791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a14150ae0 0x7f4a140a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.995+0000 7f4a18e00640 1 -- 172.21.15.27:0/1289786791 >> 172.21.15.27:0/1289786791 conn(0x7f4a1406b0c0 msgr2=0x7f4a1406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:42.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.995+0000 7f4a18e00640 1 -- 172.21.15.27:0/1289786791 shutdown_connections 2024-03-28T19:27:42.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.995+0000 7f4a18e00640 1 -- 172.21.15.27:0/1289786791 wait complete. 2024-03-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.997+0000 7f4a18e00640 1 Processor -- start 2024-03-28T19:27:42.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.997+0000 7f4a18e00640 1 -- start start 2024-03-28T19:27:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a18e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a140a61a0 0x7f4a1414d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a18e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14150ae0 0x7f4a1414db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a18e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a14153a30 0x7f4a141e0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a18e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1406e7b0 con 0x7f4a14150ae0 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a18e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1406e4d0 con 0x7f4a14153a30 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a18e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140c70b0 con 0x7f4a140a61a0 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a12a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14150ae0 0x7f4a1414db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a13400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a140a61a0 0x7f4a1414d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.998+0000 7f4a12a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14150ae0 0x7f4a1414db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38242/0 (socket says 172.21.15.27:38242) 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.999+0000 7f4a12a00640 1 -- 172.21.15.27:0/3193771464 learned_addr learned my addr 172.21.15.27:0/3193771464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.999+0000 7f4a13e00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a14153a30 0x7f4a141e0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.999+0000 7f4a12a00640 1 -- 172.21.15.27:0/3193771464 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a140a61a0 msgr2=0x7f4a1414d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.999+0000 7f4a12a00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a140a61a0 0x7f4a1414d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.999+0000 7f4a12a00640 1 -- 172.21.15.27:0/3193771464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a14153a30 msgr2=0x7f4a141e0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.999+0000 7f4a12a00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a14153a30 0x7f4a141e0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.999+0000 7f4a12a00640 1 -- 172.21.15.27:0/3193771464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14073450 con 0x7f4a14150ae0 2024-03-28T19:27:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:42.999+0000 7f4a12a00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14150ae0 0x7f4a1414db60 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400c440 tx=0x7f4a04001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.000+0000 7f4a0b400640 1 -- 172.21.15.27:0/3193771464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04017070 con 0x7f4a14150ae0 2024-03-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.000+0000 7f4a0b400640 1 -- 172.21.15.27:0/3193771464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04005ce0 con 0x7f4a14150ae0 2024-03-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.000+0000 7f4a0b400640 1 -- 172.21.15.27:0/3193771464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a040072f0 con 0x7f4a14150ae0 2024-03-28T19:27:43.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.000+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a14071220 con 0x7f4a14150ae0 2024-03-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.000+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a140717e0 con 0x7f4a14150ae0 2024-03-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.001+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a14067dd0 con 0x7f4a14150ae0 2024-03-28T19:27:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.002+0000 7f4a0b400640 1 -- 172.21.15.27:0/3193771464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a0401a070 con 0x7f4a14150ae0 2024-03-28T19:27:43.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.003+0000 7f4a0b400640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49f006bbd0 0x7f49f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:43.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.003+0000 7f4a0b400640 1 -- 172.21.15.27:0/3193771464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f4a0408af90 con 0x7f4a14150ae0 2024-03-28T19:27:43.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.003+0000 7f4a13400640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49f006bbd0 0x7f49f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:43.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.004+0000 7f4a13400640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49f006bbd0 0x7f49f006e090 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009b40 tx=0x7f49fc009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.008+0000 7f4a0b400640 1 -- 172.21.15.27:0/3193771464 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a14150ec0 con 0x7f4a14150ae0 2024-03-28T19:27:43.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.278+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a14062150 con 0x7f4a14150ae0 2024-03-28T19:27:43.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.279+0000 7f4a0b400640 1 -- 172.21.15.27:0/3193771464 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f4a04058200 con 0x7f4a14150ae0 2024-03-28T19:27:43.279 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:43.280 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:35.516980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:replay","state_seq":1,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:43.280 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 236 2024-03-28T19:27:43.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49f006bbd0 msgr2=0x7f49f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:43.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49f006bbd0 0x7f49f006e090 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009b40 tx=0x7f49fc009190 comp rx=0 tx=0).stop 2024-03-28T19:27:43.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14150ae0 msgr2=0x7f4a1414db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14150ae0 0x7f4a1414db60 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400c440 tx=0x7f4a04001e70 comp rx=0 tx=0).stop 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 shutdown_connections 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f49f006bbd0 0x7f49f006e090 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a14153a30 0x7f4a141e0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a14150ae0 0x7f4a1414db60 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 --2- 172.21.15.27:0/3193771464 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4a140a61a0 0x7f4a1414d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 >> 172.21.15.27:0/3193771464 conn(0x7f4a1406b0c0 msgr2=0x7f4a1406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 shutdown_connections 2024-03-28T19:27:43.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:43.282+0000 7f4a18e00640 1 -- 172.21.15.27:0/3193771464 wait complete. 2024-03-28T19:27:43.302 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:27:44.012 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 12% (31/250) Updating files: 13% (33/250) Updating files: 14% (35/250) Updating files: 15% (38/250) Updating files: 16% (40/250) Updating files: 17% (43/250) Updating files: 18% (45/250) Updating files: 19% (48/250) Updating files: 20% (50/250) Updating files: 21% (53/250) Updating files: 22% (55/250) Updating files: 23% (58/250) Updating files: 23% (59/250) Updating files: 24% (60/250) Updating files: 25% (63/250) Updating files: 26% (65/250) Updating files: 27% (68/250) Updating files: 28% (70/250) Updating files: 29% (73/250) Updating files: 30% (75/250) Updating files: 31% (78/250) Updating files: 32% (80/250) Updating files: 33% (83/250) Updating files: 34% (85/250) Updating files: 35% (88/250) Updating files: 36% (90/250) Updating files: 36% (92/250) Updating files: 37% (93/250) Updating files: 38% (95/250) Updating files: 39% (98/250) Updating files: 40% (100/250) Updating files: 41% (103/250) Updating files: 42% (105/250) Updating files: 43% (108/250) Updating files: 44% (110/250) Updating files: 45% (113/250) Updating files: 46% (115/250) Updating files: 47% (118/250) Updating files: 48% (120/250) Updating files: 49% (123/250) Updating files: 50% (125/250) Updating files: 51% (128/250) Updating files: 52% (130/250) Updating files: 53% (133/250) Updating files: 54% (135/250) Updating files: 55% (138/250) Updating files: 56% (140/250) Updating files: 57% (143/250) Updating files: 58% (145/250) Updating files: 58% (146/250) Updating files: 59% (148/250) Updating files: 60% (150/250) Updating files: 61% (153/250) Updating files: 62% (155/250) Updating files: 63% (158/250) Updating files: 64% (160/250) Updating files: 65% (163/250) Updating files: 66% (165/250) Updating files: 67% (168/250) Updating files: 68% (170/250) Updating files: 69% (173/250) Updating files: 70% (175/250) Updating files: 71% (178/250) Updating files: 72% (180/250) Updating files: 73% (183/250) Updating files: 74% (185/250) Updating files: 75% (188/250) Updating files: 76% (190/250) Updating files: 77% (193/250) Updating files: 78% (195/250) Updating files: 79% (198/250) Updating files: 80% (200/250) Updating files: 80% (202/250) Updating files: 81% (203/250) Updating files: 82% (205/250) Updating files: 83% (208/250) Updating files: 84% (210/250) Updating files: 85% (213/250) Updating files: 86% (215/250) Updating files: 87% (218/250) Updating files: 88% (220/250) Updating files: 89% (223/250) Updating files: 90% (225/250) Updating files: 91% (228/250) Updating files: 92% (230/250) Updating files: 93% (233/250) Updating files: 94% (235/250) Updating files: 95% (238/250) Updating files: 96% (240/250) Updating files: 97% (243/250) Updating files: 98% (245/250) Updating files: 99% (248/250) Updating files: 100% (250/250) Updating files: 100% (250/250), done. 2024-03-28T19:27:44.450 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at b205c64c687 v0.42 2024-03-28T19:27:44.462 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.42 2024-03-28T19:27:44.551 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:27:44.551 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 43 -eq 48 ']' 2024-03-28T19:27:44.551 INFO:tasks.workunit.client.0.smithi027.stdout:v0.43 2024-03-28T19:27:44.552 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.43 2024-03-28T19:27:44.552 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.43 2024-03-28T19:27:44.552 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.43 2024-03-28T19:27:45.303 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:45.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.550+0000 7f7f6fc00640 1 Processor -- start 2024-03-28T19:27:45.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.550+0000 7f7f6fc00640 1 -- start start 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.551+0000 7f7f6fc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f68150c70 0x7f7f680a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.551+0000 7f7f6fc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f681534c0 0x7f7f68153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.551+0000 7f7f6fc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f680c3af0 con 0x7f7f680c3690 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.551+0000 7f7f6fc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f680ab4e0 con 0x7f7f68150c70 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.551+0000 7f7f6fc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f680ac1f0 con 0x7f7f681534c0 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.552+0000 7f7f6d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f68150c70 0x7f7f680a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.552+0000 7f7f6d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f68150c70 0x7f7f680a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38250/0 (socket says 172.21.15.27:38250) 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.552+0000 7f7f6d200640 1 -- 172.21.15.27:0/581592443 learned_addr learned my addr 172.21.15.27:0/581592443 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.552+0000 7f7f6dc00640 1 --2- 172.21.15.27:0/581592443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f681534c0 0x7f7f68153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:45.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.552+0000 7f7f6d200640 1 -- 172.21.15.27:0/581592443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f681534c0 msgr2=0x7f7f68153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:45.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.552+0000 7f7f6d200640 1 --2- 172.21.15.27:0/581592443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f681534c0 0x7f7f68153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.552+0000 7f7f6d200640 1 -- 172.21.15.27:0/581592443 >> v1:172.21.15.155:6789/0 conn(0x7f7f680c3690 legacy=0x7f7f680c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:45.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.552+0000 7f7f6d200640 1 -- 172.21.15.27:0/581592443 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f680c3d90 con 0x7f7f68150c70 2024-03-28T19:27:45.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.554+0000 7f7f6d200640 1 --2- 172.21.15.27:0/581592443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f68150c70 0x7f7f680a9cb0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009870 tx=0x7f7f5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ae395433b6501d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:45.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.554+0000 7f7f67400640 1 -- 172.21.15.27:0/581592443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c035020 con 0x7f7f68150c70 2024-03-28T19:27:45.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.554+0000 7f7f67400640 1 -- 172.21.15.27:0/581592443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c031640 con 0x7f7f68150c70 2024-03-28T19:27:45.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.554+0000 7f7f67400640 1 -- 172.21.15.27:0/581592443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c02d040 con 0x7f7f68150c70 2024-03-28T19:27:45.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.555+0000 7f7f6fc00640 1 -- 172.21.15.27:0/581592443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f68150c70 msgr2=0x7f7f680a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:45.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.555+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/581592443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f68150c70 0x7f7f680a9cb0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009870 tx=0x7f7f5c02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.555+0000 7f7f6fc00640 1 -- 172.21.15.27:0/581592443 shutdown_connections 2024-03-28T19:27:45.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.555+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/581592443 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f681534c0 0x7f7f68153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.555+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/581592443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f68150c70 0x7f7f680a9cb0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.555+0000 7f7f6fc00640 1 -- 172.21.15.27:0/581592443 >> 172.21.15.27:0/581592443 conn(0x7f7f6806b9d0 msgr2=0x7f7f680c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:45.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.555+0000 7f7f6fc00640 1 -- 172.21.15.27:0/581592443 shutdown_connections 2024-03-28T19:27:45.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.555+0000 7f7f6fc00640 1 -- 172.21.15.27:0/581592443 wait complete. 2024-03-28T19:27:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.557+0000 7f7f6fc00640 1 Processor -- start 2024-03-28T19:27:45.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.557+0000 7f7f6fc00640 1 -- start start 2024-03-28T19:27:45.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6fc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f680c3690 0x7f7f681ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6fc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f68150c70 0x7f7f680c5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6fc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f681534c0 0x7f7f68156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6fc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f680abcd0 con 0x7f7f681534c0 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6fc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f680ab9f0 con 0x7f7f68150c70 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6fc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f680c1810 con 0x7f7f680c3690 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f681534c0 0x7f7f68156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f681534c0 0x7f7f68156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38260/0 (socket says 172.21.15.27:38260) 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.558+0000 7f7f6dc00640 1 -- 172.21.15.27:0/1499294008 learned_addr learned my addr 172.21.15.27:0/1499294008 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f6d200640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f680c3690 0x7f7f681ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:45.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f67e00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f68150c70 0x7f7f680c5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f6dc00640 1 -- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f680c3690 msgr2=0x7f7f681ef4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f6dc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f680c3690 0x7f7f681ef4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f6dc00640 1 -- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f68150c70 msgr2=0x7f7f680c5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f6dc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f68150c70 0x7f7f680c5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f6dc00640 1 -- 172.21.15.27:0/1499294008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f680a81a0 con 0x7f7f681534c0 2024-03-28T19:27:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f67e00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f68150c70 0x7f7f680c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:27:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.559+0000 7f7f6dc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f681534c0 0x7f7f68156c80 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f7f60007e60 tx=0x7f7f6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.560+0000 7f7f64c00640 1 -- 172.21.15.27:0/1499294008 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60014420 con 0x7f7f681534c0 2024-03-28T19:27:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.560+0000 7f7f64c00640 1 -- 172.21.15.27:0/1499294008 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f60019070 con 0x7f7f681534c0 2024-03-28T19:27:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.560+0000 7f7f64c00640 1 -- 172.21.15.27:0/1499294008 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60014820 con 0x7f7f681534c0 2024-03-28T19:27:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.560+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f680a72f0 con 0x7f7f681534c0 2024-03-28T19:27:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.560+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f680c5780 con 0x7f7f681534c0 2024-03-28T19:27:45.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.561+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6804f400 con 0x7f7f681534c0 2024-03-28T19:27:45.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.562+0000 7f7f64c00640 1 -- 172.21.15.27:0/1499294008 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7f600149c0 con 0x7f7f681534c0 2024-03-28T19:27:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.563+0000 7f7f64c00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7f4806bbd0 0x7f7f4806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.563+0000 7f7f64c00640 1 -- 172.21.15.27:0/1499294008 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f7f6008c430 con 0x7f7f681534c0 2024-03-28T19:27:45.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.563+0000 7f7f6d200640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7f4806bbd0 0x7f7f4806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:45.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.564+0000 7f7f6d200640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7f4806bbd0 0x7f7f4806e090 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c036000 tx=0x7f7f5c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:45.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.568+0000 7f7f64c00640 1 -- 172.21.15.27:0/1499294008 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7f60059d50 con 0x7f7f681534c0 2024-03-28T19:27:45.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.870+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f680c5f00 con 0x7f7f681534c0 2024-03-28T19:27:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.872+0000 7f7f64c00640 1 -- 172.21.15.27:0/1499294008 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) v1 ==== 95+0+5466 (secure 0 0 0) 0x7f7f60059650 con 0x7f7f681534c0 2024-03-28T19:27:45.872 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:45.872 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":237,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:44.673550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:27:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 237 2024-03-28T19:27:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.875+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7f4806bbd0 msgr2=0x7f7f4806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.875+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7f4806bbd0 0x7f7f4806e090 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c036000 tx=0x7f7f5c03c040 comp rx=0 tx=0).stop 2024-03-28T19:27:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.875+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f681534c0 msgr2=0x7f7f68156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.875+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f681534c0 0x7f7f68156c80 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f7f60007e60 tx=0x7f7f6000e4b0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.876+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 shutdown_connections 2024-03-28T19:27:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.876+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f7f4806bbd0 0x7f7f4806e090 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.876+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f681534c0 0x7f7f68156c80 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.876+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f68150c70 0x7f7f680c5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.876+0000 7f7f6fc00640 1 --2- 172.21.15.27:0/1499294008 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f680c3690 0x7f7f681ef4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.876+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 >> 172.21.15.27:0/1499294008 conn(0x7f7f6806b9d0 msgr2=0x7f7f68152850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.876+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 shutdown_connections 2024-03-28T19:27:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:45.876+0000 7f7f6fc00640 1 -- 172.21.15.27:0/1499294008 wait complete. 2024-03-28T19:27:45.896 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:27:47.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:48.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:27:48.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:27:48.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.166+0000 7fe44f200640 1 Processor -- start 2024-03-28T19:27:48.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.167+0000 7fe44f200640 1 -- start start 2024-03-28T19:27:48.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.168+0000 7fe44f200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe4480732e0 0x7fe44806b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:48.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.168+0000 7fe44f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44806b160 con 0x7fe448067130 2024-03-28T19:27:48.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.168+0000 7fe44f200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe448156450 con 0x7fe448152da0 2024-03-28T19:27:48.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.168+0000 7fe44f200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44806c260 con 0x7fe4480732e0 2024-03-28T19:27:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.169+0000 7fe447400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe4480732e0 0x7fe44806b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.169+0000 7fe447e00640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe448152da0 0x7fe4480c2900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:52446/0 (socket says 172.21.15.27:52446) 2024-03-28T19:27:48.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.169+0000 7fe447400640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe4480732e0 0x7fe44806b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:50236/0 (socket says 172.21.15.27:50236) 2024-03-28T19:27:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.169+0000 7fe447e00640 1 -- 172.21.15.27:0/3337455 learned_addr learned my addr 172.21.15.27:0/3337455 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.169+0000 7fe447400640 1 -- 172.21.15.27:0/3337455 >> v1:172.21.15.155:6789/0 conn(0x7fe448152da0 legacy=0x7fe4480c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.169+0000 7fe447400640 1 -- 172.21.15.27:0/3337455 >> v1:172.21.15.27:6789/0 conn(0x7fe448067130 legacy=0x7fe44806bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:48.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.169+0000 7fe447400640 1 -- 172.21.15.27:0/3337455 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4480c2350 con 0x7fe4480732e0 2024-03-28T19:27:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.171+0000 7fe447400640 1 --2- 172.21.15.27:0/3337455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806b610 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe434009720 tx=0x7fe43402b620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee6abe158a471cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.171+0000 7fe446a00640 1 -- 172.21.15.27:0/3337455 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434037070 con 0x7fe4480732e0 2024-03-28T19:27:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.171+0000 7fe446a00640 1 -- 172.21.15.27:0/3337455 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe434030710 con 0x7fe4480732e0 2024-03-28T19:27:48.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.171+0000 7fe446a00640 1 -- 172.21.15.27:0/3337455 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434030ae0 con 0x7fe4480732e0 2024-03-28T19:27:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.172+0000 7fe44f200640 1 -- 172.21.15.27:0/3337455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 msgr2=0x7fe44806b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:48.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.172+0000 7fe44f200640 1 --2- 172.21.15.27:0/3337455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806b610 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe434009720 tx=0x7fe43402b620 comp rx=0 tx=0).stop 2024-03-28T19:27:48.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.173+0000 7fe44f200640 1 -- 172.21.15.27:0/3337455 shutdown_connections 2024-03-28T19:27:48.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.173+0000 7fe44f200640 1 --2- 172.21.15.27:0/3337455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806b610 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.173+0000 7fe44f200640 1 -- 172.21.15.27:0/3337455 >> 172.21.15.27:0/3337455 conn(0x7fe4480c3ff0 msgr2=0x7fe4480c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:48.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.173+0000 7fe44f200640 1 -- 172.21.15.27:0/3337455 shutdown_connections 2024-03-28T19:27:48.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.173+0000 7fe44f200640 1 -- 172.21.15.27:0/3337455 wait complete. 2024-03-28T19:27:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.175+0000 7fe44f200640 1 Processor -- start 2024-03-28T19:27:48.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.175+0000 7fe44f200640 1 -- start start 2024-03-28T19:27:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe44f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe448067130 0x7fe4481569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe44f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe44f200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe448152da0 0x7fe448144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe44f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4481560e0 con 0x7fe448067130 2024-03-28T19:27:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe44f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480c2100 con 0x7fe4480732e0 2024-03-28T19:27:48.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe44f200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480c3010 con 0x7fe448152da0 2024-03-28T19:27:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe447400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe447e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe448067130 0x7fe4481569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe447400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:50252/0 (socket says 172.21.15.27:50252) 2024-03-28T19:27:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe447400640 1 -- 172.21.15.27:0/4257514786 learned_addr learned my addr 172.21.15.27:0/4257514786 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:48.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.177+0000 7fe44d200640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe448152da0 0x7fe448144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.178+0000 7fe447400640 1 -- 172.21.15.27:0/4257514786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe448152da0 msgr2=0x7fe448144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.178+0000 7fe447400640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe448152da0 0x7fe448144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.178+0000 7fe447400640 1 -- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe448067130 msgr2=0x7fe4481569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.178+0000 7fe447400640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe448067130 0x7fe4481569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.178+0000 7fe447400640 1 -- 172.21.15.27:0/4257514786 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe448152590 con 0x7fe4480732e0 2024-03-28T19:27:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.178+0000 7fe447e00640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe448067130 0x7fe4481569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:27:48.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.179+0000 7fe447400640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806ebe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe434009840 tx=0x7fe4340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.180+0000 7fe43fe00640 1 -- 172.21.15.27:0/4257514786 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434040020 con 0x7fe4480732e0 2024-03-28T19:27:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.180+0000 7fe43fe00640 1 -- 172.21.15.27:0/4257514786 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe434032a10 con 0x7fe4480732e0 2024-03-28T19:27:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.180+0000 7fe43fe00640 1 -- 172.21.15.27:0/4257514786 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434037040 con 0x7fe4480732e0 2024-03-28T19:27:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.180+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe448158640 con 0x7fe4480732e0 2024-03-28T19:27:48.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.180+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4481588e0 con 0x7fe4480732e0 2024-03-28T19:27:48.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.182+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44804f400 con 0x7fe4480732e0 2024-03-28T19:27:48.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.184+0000 7fe43fe00640 1 -- 172.21.15.27:0/4257514786 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe434043020 con 0x7fe4480732e0 2024-03-28T19:27:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.185+0000 7fe43fe00640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe42806bbd0 0x7fe42806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.185+0000 7fe43fe00640 1 -- 172.21.15.27:0/4257514786 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7fe4340ad750 con 0x7fe4480732e0 2024-03-28T19:27:48.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.185+0000 7fe447e00640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe42806bbd0 0x7fe42806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:48.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.186+0000 7fe447e00640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe42806bbd0 0x7fe42806e090 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fe438007fd0 tx=0x7fe438007de0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:48.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.191+0000 7fe43fe00640 1 -- 172.21.15.27:0/4257514786 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe43404a030 con 0x7fe4480732e0 2024-03-28T19:27:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.469+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe448066f10 con 0x7fe4480732e0 2024-03-28T19:27:48.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.471+0000 7fe43fe00640 1 -- 172.21.15.27:0/4257514786 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5471 (secure 0 0 0) 0x7fe43407a9b0 con 0x7fe4480732e0 2024-03-28T19:27:48.471 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:48.471 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:27:48.472 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 239 2024-03-28T19:27:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe42806bbd0 msgr2=0x7fe42806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe42806bbd0 0x7fe42806e090 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fe438007fd0 tx=0x7fe438007de0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 msgr2=0x7fe44806ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806ebe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe434009840 tx=0x7fe4340046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 shutdown_connections 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe42806bbd0 0x7fe42806e090 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe448152da0 0x7fe448144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4480732e0 0x7fe44806ebe0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 --2- 172.21.15.27:0/4257514786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe448067130 0x7fe4481569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 >> 172.21.15.27:0/4257514786 conn(0x7fe4480c3ff0 msgr2=0x7fe44814aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 shutdown_connections 2024-03-28T19:27:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:48.474+0000 7fe44f200640 1 -- 172.21.15.27:0/4257514786 wait complete. 2024-03-28T19:27:48.495 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=8928 2024-03-28T19:27:48.495 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.c 2024-03-28T19:27:48.770 INFO:tasks.ceph.mds.d.smithi155.stderr:2024-03-28T19:27:48.770+0000 7fe603e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:27:52.657 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 30% (34/111) Updating files: 31% (35/111) Updating files: 32% (36/111) Updating files: 33% (37/111) Updating files: 34% (38/111) Updating files: 35% (39/111) Updating files: 36% (40/111) Updating files: 37% (42/111) Updating files: 38% (43/111) Updating files: 39% (44/111) Updating files: 40% (45/111) Updating files: 41% (46/111) Updating files: 42% (47/111) Updating files: 43% (48/111) Updating files: 44% (49/111) Updating files: 45% (50/111) Updating files: 46% (52/111) Updating files: 47% (53/111) Updating files: 48% (54/111) Updating files: 49% (55/111) Updating files: 50% (56/111) Updating files: 51% (57/111) Updating files: 52% (58/111) Updating files: 53% (59/111) Updating files: 54% (60/111) Updating files: 55% (62/111) Updating files: 56% (63/111) Updating files: 57% (64/111) Updating files: 58% (65/111) Updating files: 59% (66/111) Updating files: 60% (67/111) Updating files: 61% (68/111) Updating files: 62% (69/111) Updating files: 63% (70/111) Updating files: 64% (72/111) Updating files: 65% (73/111) Updating files: 66% (74/111) Updating files: 67% (75/111) Updating files: 68% (76/111) Updating files: 69% (77/111) Updating files: 70% (78/111) Updating files: 71% (79/111) Updating files: 72% (80/111) Updating files: 73% (82/111) Updating files: 74% (83/111) Updating files: 75% (84/111) Updating files: 76% (85/111) Updating files: 77% (86/111) Updating files: 78% (87/111) Updating files: 79% (88/111) Updating files: 80% (89/111) Updating files: 81% (90/111) Updating files: 82% (92/111) Updating files: 83% (93/111) Updating files: 84% (94/111) Updating files: 85% (95/111) Updating files: 86% (96/111) Updating files: 87% (97/111) Updating files: 88% (98/111) Updating files: 89% (99/111) Updating files: 90% (100/111) Updating files: 91% (102/111) Updating files: 92% (103/111) Updating files: 93% (104/111) Updating files: 94% (105/111) Updating files: 95% (106/111) Updating files: 96% (107/111) Updating files: 97% (108/111) Updating files: 98% (109/111) Updating files: 99% (110/111) Updating files: 100% (111/111) Updating files: 100% (111/111), done. 2024-03-28T19:27:52.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:52.907 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 9fa8781c014 v0.43 2024-03-28T19:27:52.912 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.43 2024-03-28T19:27:52.967 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:27:52.967 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 44 -eq 48 ']' 2024-03-28T19:27:52.968 INFO:tasks.workunit.client.0.smithi027.stdout:v0.44 2024-03-28T19:27:52.968 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.44 2024-03-28T19:27:52.968 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.44 2024-03-28T19:27:52.968 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.44 2024-03-28T19:27:53.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.060+0000 7faf87c00640 1 Processor -- start 2024-03-28T19:27:53.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.061+0000 7faf87c00640 1 -- start start 2024-03-28T19:27:53.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf87c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faf80152bd0 0x7faf8006eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf87c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7faf8006f1e0 0x7faf80070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf87c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf800c52e0 con 0x7faf800c4dc0 2024-03-28T19:27:53.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf87c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf801557e0 con 0x7faf80152bd0 2024-03-28T19:27:53.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf87c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf80155ac0 con 0x7faf8006f1e0 2024-03-28T19:27:53.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf85c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faf800c4dc0 0x7faf80158c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43170/0 (socket says 172.21.15.27:43170) 2024-03-28T19:27:53.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf85c00640 1 -- 172.21.15.27:0/152741952 learned_addr learned my addr 172.21.15.27:0/152741952 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:53.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf7fe00640 1 --2- 172.21.15.27:0/152741952 >> v2:172.21.15.155:3300/0 conn(0x7faf8006f1e0 0x7faf80070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.063+0000 7faf85200640 1 --2- 172.21.15.27:0/152741952 >> v2:172.21.15.27:3300/0 conn(0x7faf80152bd0 0x7faf8006eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.064+0000 7faf85200640 1 -- 172.21.15.27:0/152741952 >> v2:172.21.15.155:3300/0 conn(0x7faf8006f1e0 msgr2=0x7faf80070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:53.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.064+0000 7faf85200640 1 --2- 172.21.15.27:0/152741952 >> v2:172.21.15.155:3300/0 conn(0x7faf8006f1e0 0x7faf80070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.064+0000 7faf85200640 1 -- 172.21.15.27:0/152741952 >> v1:172.21.15.27:6789/0 conn(0x7faf800c4dc0 legacy=0x7faf80158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.064+0000 7faf85200640 1 -- 172.21.15.27:0/152741952 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80155da0 con 0x7faf80152bd0 2024-03-28T19:27:53.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.067+0000 7faf85200640 1 --2- 172.21.15.27:0/152741952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 0x7faf8006eca0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7faf800c9e10 tx=0x7faf7402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1526a7e86518da0e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:53.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.067+0000 7faf7f400640 1 -- 172.21.15.27:0/152741952 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74035020 con 0x7faf80152bd0 2024-03-28T19:27:53.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.068+0000 7faf7f400640 1 -- 172.21.15.27:0/152741952 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf74031830 con 0x7faf80152bd0 2024-03-28T19:27:53.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.068+0000 7faf7f400640 1 -- 172.21.15.27:0/152741952 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74031c00 con 0x7faf80152bd0 2024-03-28T19:27:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.069+0000 7faf87c00640 1 -- 172.21.15.27:0/152741952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 msgr2=0x7faf8006eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.069+0000 7faf87c00640 1 --2- 172.21.15.27:0/152741952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 0x7faf8006eca0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7faf800c9e10 tx=0x7faf7402b8d0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.069+0000 7faf87c00640 1 -- 172.21.15.27:0/152741952 shutdown_connections 2024-03-28T19:27:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.069+0000 7faf87c00640 1 --2- 172.21.15.27:0/152741952 >> v2:172.21.15.155:3300/0 conn(0x7faf8006f1e0 0x7faf80070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.069+0000 7faf87c00640 1 --2- 172.21.15.27:0/152741952 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 0x7faf8006eca0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.069+0000 7faf87c00640 1 -- 172.21.15.27:0/152741952 >> 172.21.15.27:0/152741952 conn(0x7faf80066dd0 msgr2=0x7faf800671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.069+0000 7faf87c00640 1 -- 172.21.15.27:0/152741952 shutdown_connections 2024-03-28T19:27:53.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.069+0000 7faf87c00640 1 -- 172.21.15.27:0/152741952 wait complete. 2024-03-28T19:27:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.072+0000 7faf87c00640 1 Processor -- start 2024-03-28T19:27:53.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.072+0000 7faf87c00640 1 -- start start 2024-03-28T19:27:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.073+0000 7faf87c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf8006f1e0 0x7faf800acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.073+0000 7faf87c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf800c4dc0 0x7faf800acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.073+0000 7faf87c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 0x7faf801482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.074+0000 7faf87c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf800c6df0 con 0x7faf80152bd0 2024-03-28T19:27:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.074+0000 7faf87c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf800c6c70 con 0x7faf800c4dc0 2024-03-28T19:27:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.074+0000 7faf87c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf80070a80 con 0x7faf8006f1e0 2024-03-28T19:27:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.074+0000 7faf7fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf800c4dc0 0x7faf800acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.074+0000 7faf85c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 0x7faf801482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.074+0000 7faf7fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf800c4dc0 0x7faf800acff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:50270/0 (socket says 172.21.15.27:50270) 2024-03-28T19:27:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.074+0000 7faf7fe00640 1 -- 172.21.15.27:0/1304335895 learned_addr learned my addr 172.21.15.27:0/1304335895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.074+0000 7faf85200640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf8006f1e0 0x7faf800acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.075+0000 7faf85c00640 1 -- 172.21.15.27:0/1304335895 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf8006f1e0 msgr2=0x7faf800acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.075+0000 7faf85c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf8006f1e0 0x7faf800acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.075+0000 7faf85c00640 1 -- 172.21.15.27:0/1304335895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf800c4dc0 msgr2=0x7faf800acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.075+0000 7faf85c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf800c4dc0 0x7faf800acff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.075+0000 7faf85c00640 1 -- 172.21.15.27:0/1304335895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf800abc20 con 0x7faf80152bd0 2024-03-28T19:27:53.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.076+0000 7faf85c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 0x7faf801482a0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7faf7800ca70 tx=0x7faf7800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:53.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.076+0000 7faf7cc00640 1 -- 172.21.15.27:0/1304335895 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78017070 con 0x7faf80152bd0 2024-03-28T19:27:53.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.076+0000 7faf7cc00640 1 -- 172.21.15.27:0/1304335895 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7800ab70 con 0x7faf80152bd0 2024-03-28T19:27:53.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.076+0000 7faf7cc00640 1 -- 172.21.15.27:0/1304335895 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78012400 con 0x7faf80152bd0 2024-03-28T19:27:53.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.077+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf801ec320 con 0x7faf80152bd0 2024-03-28T19:27:53.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.077+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf801ec600 con 0x7faf80152bd0 2024-03-28T19:27:53.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.079+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5c005290 con 0x7faf80152bd0 2024-03-28T19:27:53.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.081+0000 7faf7cc00640 1 -- 172.21.15.27:0/1304335895 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faf7801a070 con 0x7faf80152bd0 2024-03-28T19:27:53.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.082+0000 7faf7cc00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf6406bbd0 0x7faf6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.083+0000 7faf7cc00640 1 -- 172.21.15.27:0/1304335895 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7faf7808b850 con 0x7faf80152bd0 2024-03-28T19:27:53.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.083+0000 7faf85200640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf6406bbd0 0x7faf6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.085+0000 7faf85200640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf6406bbd0 0x7faf6406e090 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7faf74036000 tx=0x7faf74008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.090+0000 7faf7cc00640 1 -- 172.21.15.27:0/1304335895 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faf78058aa0 con 0x7faf80152bd0 2024-03-28T19:27:53.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.403+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf5c005580 con 0x7faf80152bd0 2024-03-28T19:27:53.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.404+0000 7faf7cc00640 1 -- 172.21.15.27:0/1304335895 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5471 (secure 0 0 0) 0x7faf7805cb40 con 0x7faf80152bd0 2024-03-28T19:27:53.403 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:53.404 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:27:53.404 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 239 2024-03-28T19:27:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf6406bbd0 msgr2=0x7faf6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf6406bbd0 0x7faf6406e090 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7faf74036000 tx=0x7faf74008030 comp rx=0 tx=0).stop 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 msgr2=0x7faf801482a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 0x7faf801482a0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7faf7800ca70 tx=0x7faf7800cf30 comp rx=0 tx=0).stop 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 shutdown_connections 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7faf6406bbd0 0x7faf6406e090 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faf80152bd0 0x7faf801482a0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf800c4dc0 0x7faf800acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 --2- 172.21.15.27:0/1304335895 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faf8006f1e0 0x7faf800acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 >> 172.21.15.27:0/1304335895 conn(0x7faf80066dd0 msgr2=0x7faf8014eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 shutdown_connections 2024-03-28T19:27:53.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.407+0000 7faf87c00640 1 -- 172.21.15.27:0/1304335895 wait complete. 2024-03-28T19:27:53.428 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-03-28T19:27:53.429 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-28T19:27:53.429 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:27:53.475 INFO:tasks.ceph.mds.b:Stopped 2024-03-28T19:27:53.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:53.643 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 44% (102/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 63% (145/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2024-03-28T19:27:53.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.718+0000 7f5305e00640 1 Processor -- start 2024-03-28T19:27:53.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.718+0000 7f5305e00640 1 -- start start 2024-03-28T19:27:53.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.719+0000 7f5305e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5300150c70 0x7f53000a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.719+0000 7f5305e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c3690 0x7f53000c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.719+0000 7f5305e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53000c3af0 con 0x7f53000c6660 2024-03-28T19:27:53.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.719+0000 7f5305e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53000ab4e0 con 0x7f5300150c70 2024-03-28T19:27:53.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.719+0000 7f5305e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53000ac1f0 con 0x7f53000c3690 2024-03-28T19:27:53.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52fea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5300150c70 0x7f53000a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52ff400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f53000c6660 0x7f53001534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43174/0 (socket says 172.21.15.27:43174) 2024-03-28T19:27:53.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52ff400640 1 -- 172.21.15.27:0/1261290933 learned_addr learned my addr 172.21.15.27:0/1261290933 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:53.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52fe000640 1 --2- 172.21.15.27:0/1261290933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c3690 0x7f53000c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52fea00640 1 -- 172.21.15.27:0/1261290933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c3690 msgr2=0x7f53000c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:53.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52fea00640 1 --2- 172.21.15.27:0/1261290933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c3690 0x7f53000c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52fea00640 1 -- 172.21.15.27:0/1261290933 >> v1:172.21.15.27:6789/0 conn(0x7f53000c6660 legacy=0x7f53001534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52fea00640 1 -- 172.21.15.27:0/1261290933 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53000c3d90 con 0x7f5300150c70 2024-03-28T19:27:53.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.720+0000 7f52fd600640 1 -- 172.21.15.27:0/1261290933 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692901378 0 0) 0x7f53000c3af0 con 0x7f53000c6660 2024-03-28T19:27:53.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.721+0000 7f52fea00640 1 --2- 172.21.15.27:0/1261290933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5300150c70 0x7f53000a9cb0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f52f00097a0 tx=0x7f52f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4527f98d72ee68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:53.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f52fd600640 1 -- 172.21.15.27:0/1261290933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f00315a0 con 0x7f5300150c70 2024-03-28T19:27:53.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f52fd600640 1 -- 172.21.15.27:0/1261290933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f0037070 con 0x7f5300150c70 2024-03-28T19:27:53.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f52fd600640 1 -- 172.21.15.27:0/1261290933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0031960 con 0x7f5300150c70 2024-03-28T19:27:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f5305e00640 1 -- 172.21.15.27:0/1261290933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5300150c70 msgr2=0x7f53000a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f5305e00640 1 --2- 172.21.15.27:0/1261290933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5300150c70 0x7f53000a9cb0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f52f00097a0 tx=0x7f52f002b840 comp rx=0 tx=0).stop 2024-03-28T19:27:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f5305e00640 1 -- 172.21.15.27:0/1261290933 shutdown_connections 2024-03-28T19:27:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f5305e00640 1 --2- 172.21.15.27:0/1261290933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c3690 0x7f53000c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f5305e00640 1 --2- 172.21.15.27:0/1261290933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5300150c70 0x7f53000a9cb0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f5305e00640 1 -- 172.21.15.27:0/1261290933 >> 172.21.15.27:0/1261290933 conn(0x7f530006b9d0 msgr2=0x7f53000c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f5305e00640 1 -- 172.21.15.27:0/1261290933 shutdown_connections 2024-03-28T19:27:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.722+0000 7f5305e00640 1 -- 172.21.15.27:0/1261290933 wait complete. 2024-03-28T19:27:53.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.724+0000 7f5305e00640 1 Processor -- start 2024-03-28T19:27:53.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.724+0000 7f5305e00640 1 -- start start 2024-03-28T19:27:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.725+0000 7f5305e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53000c3690 0x7f53000c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.725+0000 7f5305e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c6660 0x7f530014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.725+0000 7f5305e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5300150c70 0x7f530014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.725+0000 7f5305e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53000abcd0 con 0x7f53000c3690 2024-03-28T19:27:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.725+0000 7f5305e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53000ab9f0 con 0x7f5300150c70 2024-03-28T19:27:53.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.725+0000 7f5305e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53000c1810 con 0x7f53000c6660 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.725+0000 7f52fea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53000c3690 0x7f53000c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.725+0000 7f52fea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53000c3690 0x7f53000c02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49988/0 (socket says 172.21.15.27:49988) 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52fea00640 1 -- 172.21.15.27:0/2791575045 learned_addr learned my addr 172.21.15.27:0/2791575045 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52ff400640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5300150c70 0x7f530014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52fe000640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c6660 0x7f530014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52fea00640 1 -- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c6660 msgr2=0x7f530014c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52fea00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c6660 0x7f530014c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52fea00640 1 -- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5300150c70 msgr2=0x7f530014c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52fea00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5300150c70 0x7f530014c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52fea00640 1 -- 172.21.15.27:0/2791575045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300072140 con 0x7f53000c3690 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52ff400640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5300150c70 0x7f530014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:27:53.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.726+0000 7f52fea00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53000c3690 0x7f53000c02a0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f52f00097a0 tx=0x7f52f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:53.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.727+0000 7f52f7400640 1 -- 172.21.15.27:0/2791575045 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0032840 con 0x7f53000c3690 2024-03-28T19:27:53.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.727+0000 7f52f7400640 1 -- 172.21.15.27:0/2791575045 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f00329e0 con 0x7f53000c3690 2024-03-28T19:27:53.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.727+0000 7f52f7400640 1 -- 172.21.15.27:0/2791575045 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0032dd0 con 0x7f53000c3690 2024-03-28T19:27:53.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.727+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53001eae80 con 0x7f53000c3690 2024-03-28T19:27:53.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.727+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53001eb420 con 0x7f53000c3690 2024-03-28T19:27:53.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.728+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530004f400 con 0x7f53000c3690 2024-03-28T19:27:53.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.729+0000 7f52f7400640 1 -- 172.21.15.27:0/2791575045 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f52f003b3f0 con 0x7f53000c3690 2024-03-28T19:27:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.730+0000 7f52f7400640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f52e006bbd0 0x7f52e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.730+0000 7f52f7400640 1 -- 172.21.15.27:0/2791575045 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f52f002f030 con 0x7f53000c3690 2024-03-28T19:27:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.730+0000 7f52fe000640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f52e006bbd0 0x7f52e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.731+0000 7f52fe000640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f52e006bbd0 0x7f52e006e090 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5300154670 tx=0x7f52ec008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.734+0000 7f52f7400640 1 -- 172.21.15.27:0/2791575045 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f52f0040070 con 0x7f53000c3690 2024-03-28T19:27:53.941 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 071881d7e55 v0.28 2024-03-28T19:27:53.959 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.28 2024-03-28T19:27:53.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.989+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53000c5f00 con 0x7f53000c3690 2024-03-28T19:27:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.990+0000 7f52f7400640 1 -- 172.21.15.27:0/2791575045 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5471 (secure 0 0 0) 0x7f52f007b4c0 con 0x7f53000c3690 2024-03-28T19:27:53.989 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:53.989 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:27:53.989 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 239 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f52e006bbd0 msgr2=0x7f52e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f52e006bbd0 0x7f52e006e090 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5300154670 tx=0x7f52ec008040 comp rx=0 tx=0).stop 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53000c3690 msgr2=0x7f53000c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53000c3690 0x7f53000c02a0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f52f00097a0 tx=0x7f52f00046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 shutdown_connections 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f52e006bbd0 0x7f52e006e090 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5300150c70 0x7f530014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53000c6660 0x7f530014c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 --2- 172.21.15.27:0/2791575045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53000c3690 0x7f53000c02a0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:53.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 >> 172.21.15.27:0/2791575045 conn(0x7f530006b9d0 msgr2=0x7f53000a8990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:53.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 shutdown_connections 2024-03-28T19:27:53.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:53.993+0000 7f5305e00640 1 -- 172.21.15.27:0/2791575045 wait complete. 2024-03-28T19:27:54.013 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:27:54.038 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:27:54.038 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 29 -eq 48 ']' 2024-03-28T19:27:54.038 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.29 2024-03-28T19:27:54.039 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.29 2024-03-28T19:27:54.039 INFO:tasks.workunit.client.1.smithi155.stdout:v0.29 2024-03-28T19:27:54.039 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.29 2024-03-28T19:27:55.495 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-03-28T19:27:55.496 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-28T19:27:55.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-28T19:27:55.505 INFO:tasks.ceph.mds.c:Started 2024-03-28T19:27:55.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:55.699 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-03-28T19:27:55.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.727+0000 7f69cea00640 1 Processor -- start 2024-03-28T19:27:55.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.727+0000 7f69cea00640 1 -- start start 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.728+0000 7f69cea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69c80c69a0 0x7f69c80c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.728+0000 7f69cea00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f69c8153a30 0x7f69c8153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.728+0000 7f69cea00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c8150ec0 con 0x7f69c8150ae0 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.728+0000 7f69cea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c81543c0 con 0x7f69c80c69a0 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.728+0000 7f69cea00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c806dc70 con 0x7f69c8153a30 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.728+0000 7f69ccc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69c80c69a0 0x7f69c80c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.728+0000 7f69ccc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69c80c69a0 0x7f69c80c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50010/0 (socket says 172.21.15.27:50010) 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.728+0000 7f69ccc00640 1 -- 172.21.15.27:0/3342786441 learned_addr learned my addr 172.21.15.27:0/3342786441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:55.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.729+0000 7f69ce000640 1 --2- 172.21.15.27:0/3342786441 >> v2:172.21.15.155:3300/0 conn(0x7f69c8153a30 0x7f69c8153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.729+0000 7f69ccc00640 1 -- 172.21.15.27:0/3342786441 >> v2:172.21.15.155:3300/0 conn(0x7f69c8153a30 msgr2=0x7f69c8153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.729+0000 7f69ccc00640 1 --2- 172.21.15.27:0/3342786441 >> v2:172.21.15.155:3300/0 conn(0x7f69c8153a30 0x7f69c8153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.729+0000 7f69ccc00640 1 -- 172.21.15.27:0/3342786441 >> v1:172.21.15.155:6789/0 conn(0x7f69c8150ae0 legacy=0x7f69c80a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.729+0000 7f69ccc00640 1 -- 172.21.15.27:0/3342786441 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8153550 con 0x7f69c80c69a0 2024-03-28T19:27:55.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.730+0000 7f69ccc00640 1 --2- 172.21.15.27:0/3342786441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c80c69a0 0x7f69c80c6e30 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f69c80c0af0 tx=0x7f69b402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a61b81a41ac16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:55.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.730+0000 7f69c7e00640 1 -- 172.21.15.27:0/3342786441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b402e070 con 0x7f69c80c69a0 2024-03-28T19:27:55.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.730+0000 7f69c7e00640 1 -- 172.21.15.27:0/3342786441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b4037070 con 0x7f69c80c69a0 2024-03-28T19:27:55.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.730+0000 7f69c7e00640 1 -- 172.21.15.27:0/3342786441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4031a00 con 0x7f69c80c69a0 2024-03-28T19:27:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.731+0000 7f69cea00640 1 -- 172.21.15.27:0/3342786441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c80c69a0 msgr2=0x7f69c80c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.731+0000 7f69cea00640 1 --2- 172.21.15.27:0/3342786441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c80c69a0 0x7f69c80c6e30 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f69c80c0af0 tx=0x7f69b402b880 comp rx=0 tx=0).stop 2024-03-28T19:27:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.731+0000 7f69cea00640 1 -- 172.21.15.27:0/3342786441 shutdown_connections 2024-03-28T19:27:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.731+0000 7f69cea00640 1 --2- 172.21.15.27:0/3342786441 >> v2:172.21.15.155:3300/0 conn(0x7f69c8153a30 0x7f69c8153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.731+0000 7f69cea00640 1 --2- 172.21.15.27:0/3342786441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c80c69a0 0x7f69c80c6e30 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.731+0000 7f69cea00640 1 -- 172.21.15.27:0/3342786441 >> 172.21.15.27:0/3342786441 conn(0x7f69c806b0c0 msgr2=0x7f69c806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.731+0000 7f69cea00640 1 -- 172.21.15.27:0/3342786441 shutdown_connections 2024-03-28T19:27:55.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.731+0000 7f69cea00640 1 -- 172.21.15.27:0/3342786441 wait complete. 2024-03-28T19:27:55.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.733+0000 7f69cea00640 1 Processor -- start 2024-03-28T19:27:55.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.733+0000 7f69cea00640 1 -- start start 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69cea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69c80c69a0 0x7f69c814e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69cea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c8150ae0 0x7f69c80bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69cea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c8153a30 0x7f69c81457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69cea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c806e4d0 con 0x7f69c8153a30 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69cea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c806e1f0 con 0x7f69c8150ae0 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69cea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c806e7b0 con 0x7f69c80c69a0 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69ce000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c8153a30 0x7f69c81457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69ce000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c8153a30 0x7f69c81457c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50020/0 (socket says 172.21.15.27:50020) 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69ce000640 1 -- 172.21.15.27:0/3251671355 learned_addr learned my addr 172.21.15.27:0/3251671355 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:55.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.734+0000 7f69cd600640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69c80c69a0 0x7f69c814e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:55.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69ccc00640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c8150ae0 0x7f69c80bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:55.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69ce000640 1 -- 172.21.15.27:0/3251671355 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69c80c69a0 msgr2=0x7f69c814e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:55.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69ce000640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69c80c69a0 0x7f69c814e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69ce000640 1 -- 172.21.15.27:0/3251671355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c8150ae0 msgr2=0x7f69c80bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:55.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69ce000640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c8150ae0 0x7f69c80bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69ce000640 1 -- 172.21.15.27:0/3251671355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c81ef2f0 con 0x7f69c8153a30 2024-03-28T19:27:55.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69ce000640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c8153a30 0x7f69c81457c0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f69c000d5d0 tx=0x7f69c000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:55.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69c5600640 1 -- 172.21.15.27:0/3251671355 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00125f0 con 0x7f69c8153a30 2024-03-28T19:27:55.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69c5600640 1 -- 172.21.15.27:0/3251671355 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c0019070 con 0x7f69c8153a30 2024-03-28T19:27:55.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.735+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c81ef5d0 con 0x7f69c8153a30 2024-03-28T19:27:55.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.736+0000 7f69c5600640 1 -- 172.21.15.27:0/3251671355 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00129f0 con 0x7f69c8153a30 2024-03-28T19:27:55.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.736+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c80ab120 con 0x7f69c8153a30 2024-03-28T19:27:55.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.737+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c8067dd0 con 0x7f69c8153a30 2024-03-28T19:27:55.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.738+0000 7f69c5600640 1 -- 172.21.15.27:0/3251671355 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f69c0012b90 con 0x7f69c8153a30 2024-03-28T19:27:55.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.739+0000 7f69c5600640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f69a806bbd0 0x7f69a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:55.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.739+0000 7f69c5600640 1 -- 172.21.15.27:0/3251671355 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f69c008bff0 con 0x7f69c8153a30 2024-03-28T19:27:55.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.739+0000 7f69cd600640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f69a806bbd0 0x7f69a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:55.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.740+0000 7f69cd600640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f69a806bbd0 0x7f69a806e090 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f69bc005660 tx=0x7f69bc003d90 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:55.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.744+0000 7f69c5600640 1 -- 172.21.15.27:0/3251671355 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f69c0059850 con 0x7f69c8153a30 2024-03-28T19:27:55.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.993+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69c80c11b0 con 0x7f69c8153a30 2024-03-28T19:27:55.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.994+0000 7f69c5600640 1 -- 172.21.15.27:0/3251671355 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f69c80c11b0 con 0x7f69c8153a30 2024-03-28T19:27:55.993 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:55.993 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:27:55.993 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 240 2024-03-28T19:27:55.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.997+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f69a806bbd0 msgr2=0x7f69a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:55.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.997+0000 7f69cea00640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f69a806bbd0 0x7f69a806e090 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f69bc005660 tx=0x7f69bc003d90 comp rx=0 tx=0).stop 2024-03-28T19:27:55.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.997+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c8153a30 msgr2=0x7f69c81457c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:55.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.997+0000 7f69cea00640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c8153a30 0x7f69c81457c0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f69c000d5d0 tx=0x7f69c000da90 comp rx=0 tx=0).stop 2024-03-28T19:27:55.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.998+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 shutdown_connections 2024-03-28T19:27:55.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.998+0000 7f69cea00640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f69a806bbd0 0x7f69a806e090 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.998+0000 7f69cea00640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69c8153a30 0x7f69c81457c0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.998+0000 7f69cea00640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c8150ae0 0x7f69c80bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.998+0000 7f69cea00640 1 --2- 172.21.15.27:0/3251671355 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69c80c69a0 0x7f69c814e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.998+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 >> 172.21.15.27:0/3251671355 conn(0x7f69c806b0c0 msgr2=0x7f69c81547d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.998+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 shutdown_connections 2024-03-28T19:27:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:55.998+0000 7f69cea00640 1 -- 172.21.15.27:0/3251671355 wait complete. 2024-03-28T19:27:56.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:56.015 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-03-28T19:27:56.015 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2024-03-28T19:27:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.238+0000 7fc7abe00640 1 Processor -- start 2024-03-28T19:27:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.239+0000 7fc7abe00640 1 -- start start 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.240+0000 7fc7abe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7ac0c8ed0 0x7fc7ac0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.240+0000 7fc7abe00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc7ac0be5d0 0x7fc7ac155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.240+0000 7fc7abe00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac156230 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.240+0000 7fc7abe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac158bd0 con 0x7fc7ac0c8ed0 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.240+0000 7fc7abe00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac158eb0 con 0x7fc7ac0be5d0 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.241+0000 7fc7aa000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7ac0c8ed0 0x7fc7ac0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.241+0000 7fc7aa000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7ac0c8ed0 0x7fc7ac0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46512/0 (socket says 172.21.15.27:46512) 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.241+0000 7fc7aa000640 1 -- 172.21.15.27:0/2812669524 learned_addr learned my addr 172.21.15.27:0/2812669524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.241+0000 7fc7ab400640 1 --2- 172.21.15.27:0/2812669524 >> v2:172.21.15.155:3300/0 conn(0x7fc7ac0be5d0 0x7fc7ac155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:56.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.241+0000 7fc7aa000640 1 -- 172.21.15.27:0/2812669524 >> v2:172.21.15.155:3300/0 conn(0x7fc7ac0be5d0 msgr2=0x7fc7ac155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.241+0000 7fc7aa000640 1 --2- 172.21.15.27:0/2812669524 >> v2:172.21.15.155:3300/0 conn(0x7fc7ac0be5d0 0x7fc7ac155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.241+0000 7fc7aa000640 1 -- 172.21.15.27:0/2812669524 >> v1:172.21.15.155:6789/0 conn(0x7fc7ac0c58c0 legacy=0x7fc7ac1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:27:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.241+0000 7fc7aa000640 1 -- 172.21.15.27:0/2812669524 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac159190 con 0x7fc7ac0c8ed0 2024-03-28T19:27:56.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.242+0000 7fc7aa000640 1 --2- 172.21.15.27:0/2812669524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c8ed0 0x7fc7ac0c5ca0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac08d050 tx=0x7fc79402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78a353b87344347 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7a9600640 1 -- 172.21.15.27:0/2812669524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794038070 con 0x7fc7ac0c8ed0 2024-03-28T19:27:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7a9600640 1 -- 172.21.15.27:0/2812669524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79402f050 con 0x7fc7ac0c8ed0 2024-03-28T19:27:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7a9600640 1 -- 172.21.15.27:0/2812669524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794031970 con 0x7fc7ac0c8ed0 2024-03-28T19:27:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7abe00640 1 -- 172.21.15.27:0/2812669524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c8ed0 msgr2=0x7fc7ac0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:56.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2812669524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c8ed0 0x7fc7ac0c5ca0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac08d050 tx=0x7fc79402b7f0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7abe00640 1 -- 172.21.15.27:0/2812669524 shutdown_connections 2024-03-28T19:27:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2812669524 >> v2:172.21.15.155:3300/0 conn(0x7fc7ac0be5d0 0x7fc7ac155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2812669524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c8ed0 0x7fc7ac0c5ca0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7abe00640 1 -- 172.21.15.27:0/2812669524 >> 172.21.15.27:0/2812669524 conn(0x7fc7ac06b6f0 msgr2=0x7fc7ac0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7abe00640 1 -- 172.21.15.27:0/2812669524 shutdown_connections 2024-03-28T19:27:56.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.243+0000 7fc7abe00640 1 -- 172.21.15.27:0/2812669524 wait complete. 2024-03-28T19:27:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.245+0000 7fc7abe00640 1 Processor -- start 2024-03-28T19:27:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.245+0000 7fc7abe00640 1 -- start start 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7abe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7ac0be5d0 0x7fc7ac0c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c58c0 0x7fc7ac0c31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7abe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc7ac0c8ed0 0x7fc7ac0abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7abe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b0820 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7abe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac159750 con 0x7fc7ac0be5d0 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7abe00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b09a0 con 0x7fc7ac0c8ed0 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7aa000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c58c0 0x7fc7ac0c31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7aa000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c58c0 0x7fc7ac0c31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46526/0 (socket says 172.21.15.27:46526) 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7aa000640 1 -- 172.21.15.27:0/2846511771 learned_addr learned my addr 172.21.15.27:0/2846511771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.246+0000 7fc7aaa00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7ac0be5d0 0x7fc7ac0c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:56.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.247+0000 7fc7ab400640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc7ac0c8ed0 0x7fc7ac0abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.247+0000 7fc7aa000640 1 -- 172.21.15.27:0/2846511771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc7ac0c8ed0 msgr2=0x7fc7ac0abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.247+0000 7fc7aa000640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc7ac0c8ed0 0x7fc7ac0abed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.247+0000 7fc7aa000640 1 -- 172.21.15.27:0/2846511771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7ac0be5d0 msgr2=0x7fc7ac0c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.247+0000 7fc7aa000640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7ac0be5d0 0x7fc7ac0c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.247+0000 7fc7aa000640 1 -- 172.21.15.27:0/2846511771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0c3700 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.247+0000 7fc7aa000640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c58c0 0x7fc7ac0c31c0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fc794000c00 tx=0x7fc7940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.247+0000 7fc7a3400640 1 -- 172.21.15.27:0/2846511771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794042070 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.248+0000 7fc7a3400640 1 -- 172.21.15.27:0/2846511771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79403d870 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.248+0000 7fc7a3400640 1 -- 172.21.15.27:0/2846511771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794038040 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.248+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac0c49a0 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.248+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac0c4c40 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.249+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac069640 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.250+0000 7fc7a3400640 1 -- 172.21.15.27:0/2846511771 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc794045070 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.251+0000 7fc7a3400640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc78806bc00 0x7fc78806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.251+0000 7fc7a3400640 1 -- 172.21.15.27:0/2846511771 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7fc7940ad9c0 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.251+0000 7fc7aaa00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc78806bc00 0x7fc78806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:56.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.252+0000 7fc7aaa00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc78806bc00 0x7fc78806e0c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc79c006fd0 tx=0x7fc79c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:56.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.255+0000 7fc7a3400640 1 -- 172.21.15.27:0/2846511771 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc7940afe20 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.496+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7ac156580 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.497+0000 7fc7a3400640 1 -- 172.21.15.27:0/2846511771 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fc79407b400 con 0x7fc7ac0c58c0 2024-03-28T19:27:56.497 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:56.497 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:27:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 240 2024-03-28T19:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc78806bc00 msgr2=0x7fc78806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:56.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc78806bc00 0x7fc78806e0c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc79c006fd0 tx=0x7fc79c008040 comp rx=0 tx=0).stop 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c58c0 msgr2=0x7fc7ac0c31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c58c0 0x7fc7ac0c31c0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fc794000c00 tx=0x7fc7940046c0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 shutdown_connections 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc78806bc00 0x7fc78806e0c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc7ac0c8ed0 0x7fc7ac0abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7ac0c58c0 0x7fc7ac0c31c0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 --2- 172.21.15.27:0/2846511771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc7ac0be5d0 0x7fc7ac0c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 >> 172.21.15.27:0/2846511771 conn(0x7fc7ac06b6f0 msgr2=0x7fc7ac158400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.500+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 shutdown_connections 2024-03-28T19:27:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:56.501+0000 7fc7abe00640 1 -- 172.21.15.27:0/2846511771 wait complete. 2024-03-28T19:27:56.519 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:27:58.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:27:58.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.755+0000 7f5970a00640 1 Processor -- start 2024-03-28T19:27:58.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.755+0000 7f5970a00640 1 -- start start 2024-03-28T19:27:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.756+0000 7f5970a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f596c0c59d0 0x7f596c0c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:58.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.756+0000 7f5970a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c155fe0 con 0x7f596c155600 2024-03-28T19:27:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.756+0000 7f5970a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c1562c0 con 0x7f596c0727e0 2024-03-28T19:27:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.756+0000 7f5970a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c1565a0 con 0x7f596c0c59d0 2024-03-28T19:27:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.757+0000 7f596b200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f596c155600 0x7f596c14f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42770/0 (socket says 172.21.15.27:42770) 2024-03-28T19:27:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.757+0000 7f596b200640 1 -- 172.21.15.27:0/3632468800 learned_addr learned my addr 172.21.15.27:0/3632468800 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:27:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.757+0000 7f596a800640 1 --2- 172.21.15.27:0/3632468800 >> v2:172.21.15.27:3300/0 conn(0x7f596c0c59d0 0x7f596c0c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.757+0000 7f5969e00640 1 -- 172.21.15.27:0/3632468800 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538046611 0 0) 0x7f596c155fe0 con 0x7f596c155600 2024-03-28T19:27:58.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.757+0000 7f5969e00640 1 -- 172.21.15.27:0/3632468800 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5950003520 con 0x7f596c155600 2024-03-28T19:27:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.757+0000 7f596a800640 1 -- 172.21.15.27:0/3632468800 >> v1:172.21.15.155:6789/0 conn(0x7f596c0727e0 legacy=0x7f596c159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.758+0000 7f596a800640 1 -- 172.21.15.27:0/3632468800 >> v1:172.21.15.27:6789/0 conn(0x7f596c155600 legacy=0x7f596c14f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:58.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.758+0000 7f596a800640 1 -- 172.21.15.27:0/3632468800 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c158d20 con 0x7f596c0c59d0 2024-03-28T19:27:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.759+0000 7f596a800640 1 --2- 172.21.15.27:0/3632468800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c0c8f00 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f595c0097a0 tx=0x7f595c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d042a8b74f9e11a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.759+0000 7f5969e00640 1 -- 172.21.15.27:0/3632468800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c0315a0 con 0x7f596c0c59d0 2024-03-28T19:27:58.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.759+0000 7f5969e00640 1 -- 172.21.15.27:0/3632468800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c037070 con 0x7f596c0c59d0 2024-03-28T19:27:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.759+0000 7f5969e00640 1 -- 172.21.15.27:0/3632468800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c031960 con 0x7f596c0c59d0 2024-03-28T19:27:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.760+0000 7f5970a00640 1 -- 172.21.15.27:0/3632468800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 msgr2=0x7f596c0c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.760+0000 7f5970a00640 1 --2- 172.21.15.27:0/3632468800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c0c8f00 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f595c0097a0 tx=0x7f595c02b840 comp rx=0 tx=0).stop 2024-03-28T19:27:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.760+0000 7f5970a00640 1 -- 172.21.15.27:0/3632468800 shutdown_connections 2024-03-28T19:27:58.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.760+0000 7f5970a00640 1 --2- 172.21.15.27:0/3632468800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c0c8f00 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.760+0000 7f5970a00640 1 -- 172.21.15.27:0/3632468800 >> 172.21.15.27:0/3632468800 conn(0x7f596c0ac910 msgr2=0x7f596c0accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.760+0000 7f5970a00640 1 -- 172.21.15.27:0/3632468800 shutdown_connections 2024-03-28T19:27:58.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.760+0000 7f5970a00640 1 -- 172.21.15.27:0/3632468800 wait complete. 2024-03-28T19:27:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.762+0000 7f5970a00640 1 Processor -- start 2024-03-28T19:27:58.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.762+0000 7f5970a00640 1 -- start start 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5970a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f596c0727e0 0x7f596c14d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5970a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c13bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5970a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f596c155600 0x7f596c13c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5970a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c5db0 con 0x7f596c0c59d0 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5970a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c159090 con 0x7f596c0727e0 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5970a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c5f30 con 0x7f596c155600 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5961e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c13bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5961e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c13bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46552/0 (socket says 172.21.15.27:46552) 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f5961e00640 1 -- 172.21.15.27:0/362260278 learned_addr learned my addr 172.21.15.27:0/362260278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f596a800640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f596c0727e0 0x7f596c14d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:58.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.763+0000 7f596b200640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f596c155600 0x7f596c13c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5961e00640 1 -- 172.21.15.27:0/362260278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f596c155600 msgr2=0x7f596c13c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5961e00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f596c155600 0x7f596c13c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5961e00640 1 -- 172.21.15.27:0/362260278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f596c0727e0 msgr2=0x7f596c14d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5961e00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f596c0727e0 0x7f596c14d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5961e00640 1 -- 172.21.15.27:0/362260278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c13c760 con 0x7f596c0c59d0 2024-03-28T19:27:58.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5961e00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c13bce0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f595400c470 tx=0x7f5954001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5963400640 1 -- 172.21.15.27:0/362260278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59540052d0 con 0x7f596c0c59d0 2024-03-28T19:27:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5963400640 1 -- 172.21.15.27:0/362260278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954005470 con 0x7f596c0c59d0 2024-03-28T19:27:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c0c4ad0 con 0x7f596c0c59d0 2024-03-28T19:27:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.764+0000 7f5963400640 1 -- 172.21.15.27:0/362260278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954007260 con 0x7f596c0c59d0 2024-03-28T19:27:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.765+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c0bfb30 con 0x7f596c0c59d0 2024-03-28T19:27:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.766+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c067dd0 con 0x7f596c0c59d0 2024-03-28T19:27:58.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.767+0000 7f5963400640 1 -- 172.21.15.27:0/362260278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5954007400 con 0x7f596c0c59d0 2024-03-28T19:27:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.768+0000 7f5963400640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f594806bbd0 0x7f594806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:27:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.768+0000 7f5963400640 1 -- 172.21.15.27:0/362260278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f595408b0d0 con 0x7f596c0c59d0 2024-03-28T19:27:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.768+0000 7f596a800640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f594806bbd0 0x7f594806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:27:58.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.769+0000 7f596a800640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f594806bbd0 0x7f594806e090 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f595c036000 tx=0x7f595c03d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:27:58.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:58.773+0000 7f5963400640 1 -- 172.21.15.27:0/362260278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f59540583a0 con 0x7f596c0c59d0 2024-03-28T19:27:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.031+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f596c069640 con 0x7f596c0c59d0 2024-03-28T19:27:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.032+0000 7f5963400640 1 -- 172.21.15.27:0/362260278 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f595405c440 con 0x7f596c0c59d0 2024-03-28T19:27:59.031 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:27:59.031 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:27:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 240 2024-03-28T19:27:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.034+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f594806bbd0 msgr2=0x7f594806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.034+0000 7f5970a00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f594806bbd0 0x7f594806e090 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f595c036000 tx=0x7f595c03d040 comp rx=0 tx=0).stop 2024-03-28T19:27:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.034+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 msgr2=0x7f596c13bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:27:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.034+0000 7f5970a00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c13bce0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f595400c470 tx=0x7f5954001e70 comp rx=0 tx=0).stop 2024-03-28T19:27:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.035+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 shutdown_connections 2024-03-28T19:27:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.035+0000 7f5970a00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f594806bbd0 0x7f594806e090 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.035+0000 7f5970a00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f596c155600 0x7f596c13c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.035+0000 7f5970a00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c0c59d0 0x7f596c13bce0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.035+0000 7f5970a00640 1 --2- 172.21.15.27:0/362260278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f596c0727e0 0x7f596c14d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:27:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.035+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 >> 172.21.15.27:0/362260278 conn(0x7f596c0ac910 msgr2=0x7f596c0c77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:27:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.035+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 shutdown_connections 2024-03-28T19:27:59.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:27:59.035+0000 7f5970a00640 1 -- 172.21.15.27:0/362260278 wait complete. 2024-03-28T19:27:59.055 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:28:01.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:01.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.299+0000 7f8279600640 1 Processor -- start 2024-03-28T19:28:01.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.299+0000 7f8279600640 1 -- start start 2024-03-28T19:28:01.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.301+0000 7f8279600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 0x7f82740c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.301+0000 7f8279600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82740c7310 con 0x7f8274152f80 2024-03-28T19:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.301+0000 7f8279600640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8274158d40 con 0x7f8274155730 2024-03-28T19:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.301+0000 7f8279600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8274159020 con 0x7f82740c7970 2024-03-28T19:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.301+0000 7f8272a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8274152f80 0x7f82740c58e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42782/0 (socket says 172.21.15.27:42782) 2024-03-28T19:28:01.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.301+0000 7f8272a00640 1 -- 172.21.15.27:0/3522842360 learned_addr learned my addr 172.21.15.27:0/3522842360 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:01.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8272000640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 0x7f82740c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745880171 0 0) 0x7f82740c7310 con 0x7f8274152f80 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8258003520 con 0x7f8274152f80 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753149010 0 0) 0x7f8258003520 con 0x7f8274152f80 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 msgr2=0x7f82740c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 0x7f82740c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 >> v1:172.21.15.155:6789/0 conn(0x7f8274155730 legacy=0x7f8274155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274159300 con 0x7f8274152f80 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f82640019f0 con 0x7f8274152f80 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 >> v1:172.21.15.27:6789/0 conn(0x7f8274152f80 legacy=0x7f82740c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82580039e0 0x7f8258003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8258004800 0x7f8258004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 0x7f8258005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8274159020 con 0x7f8258004800 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825800bb50 con 0x7f82580039e0 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.302+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825800bd50 con 0x7f82740c7970 2024-03-28T19:28:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8272000640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8258004800 0x7f8258004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8272a00640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82580039e0 0x7f8258003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8273400640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 0x7f8258005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8272000640 1 -- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 msgr2=0x7f8258005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8272000640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 0x7f8258005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8272000640 1 -- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82580039e0 msgr2=0x7f8258003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8272000640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82580039e0 0x7f8258003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8272000640 1 -- 172.21.15.27:0/3522842360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8258003520 con 0x7f8258004800 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8272a00640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82580039e0 0x7f8258003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:28:01.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.303+0000 7f8273400640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82740c7970 0x7f8258005270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:28:01.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.305+0000 7f8272000640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8258004800 0x7f8258004c80 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f825c009ac0 tx=0x7f8258005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80706d72a8bbf49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:01.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.305+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c039070 con 0x7f8258004800 2024-03-28T19:28:01.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.305+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c02c6d0 con 0x7f8258004800 2024-03-28T19:28:01.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.305+0000 7f8271600640 1 -- 172.21.15.27:0/3522842360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c009690 con 0x7f8258004800 2024-03-28T19:28:01.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.306+0000 7f8279600640 1 -- 172.21.15.27:0/3522842360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8258004800 msgr2=0x7f8258004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:01.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.306+0000 7f8279600640 1 --2- 172.21.15.27:0/3522842360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8258004800 0x7f8258004c80 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f825c009ac0 tx=0x7f8258005830 comp rx=0 tx=0).stop 2024-03-28T19:28:01.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.306+0000 7f8273400640 1 -- 172.21.15.27:0/3522842360 reap_dead start 2024-03-28T19:28:01.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.306+0000 7f8279600640 1 -- 172.21.15.27:0/3522842360 shutdown_connections 2024-03-28T19:28:01.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.306+0000 7f8279600640 1 -- 172.21.15.27:0/3522842360 >> 172.21.15.27:0/3522842360 conn(0x7f827406b6f0 msgr2=0x7f82740caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:01.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.306+0000 7f8279600640 1 -- 172.21.15.27:0/3522842360 shutdown_connections 2024-03-28T19:28:01.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.306+0000 7f8279600640 1 -- 172.21.15.27:0/3522842360 wait complete. 2024-03-28T19:28:01.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.308+0000 7f8279600640 1 Processor -- start 2024-03-28T19:28:01.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.308+0000 7f8279600640 1 -- start start 2024-03-28T19:28:01.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.309+0000 7f8279600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8258004800 0x7f82740ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:01.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.309+0000 7f8279600640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82741ec040 0x7f82740c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:01.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.309+0000 7f8279600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740c4d10 0x7f82740c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:01.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.309+0000 7f8279600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82740cbca0 con 0x7f82740c4d10 2024-03-28T19:28:01.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.309+0000 7f8279600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8274159b00 con 0x7f8258004800 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.309+0000 7f8279600640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82741eea10 con 0x7f82741ec040 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8273400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740c4d10 0x7f82740c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8273400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740c4d10 0x7f82740c4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46576/0 (socket says 172.21.15.27:46576) 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8273400640 1 -- 172.21.15.27:0/3060741136 learned_addr learned my addr 172.21.15.27:0/3060741136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8272a00640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8258004800 0x7f82740ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8272000640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82741ec040 0x7f82740c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8273400640 1 -- 172.21.15.27:0/3060741136 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82741ec040 msgr2=0x7f82740c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8273400640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82741ec040 0x7f82740c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8273400640 1 -- 172.21.15.27:0/3060741136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8258004800 msgr2=0x7f82740ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8273400640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8258004800 0x7f82740ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.310+0000 7f8273400640 1 -- 172.21.15.27:0/3060741136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82741eb560 con 0x7f82740c4d10 2024-03-28T19:28:01.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.311+0000 7f8273400640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740c4d10 0x7f82740c4050 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f82740bea60 tx=0x7f826c00d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:01.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.311+0000 7f826aa00640 1 -- 172.21.15.27:0/3060741136 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c018070 con 0x7f82740c4d10 2024-03-28T19:28:01.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.311+0000 7f826aa00640 1 -- 172.21.15.27:0/3060741136 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826c005310 con 0x7f82740c4d10 2024-03-28T19:28:01.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.311+0000 7f826aa00640 1 -- 172.21.15.27:0/3060741136 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c007270 con 0x7f82740c4d10 2024-03-28T19:28:01.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.311+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82741eb840 con 0x7f82740c4d10 2024-03-28T19:28:01.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.311+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82740ac8c0 con 0x7f82740c4d10 2024-03-28T19:28:01.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.313+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82741eb370 con 0x7f82740c4d10 2024-03-28T19:28:01.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.314+0000 7f826aa00640 1 -- 172.21.15.27:0/3060741136 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f826c007410 con 0x7f82740c4d10 2024-03-28T19:28:01.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.315+0000 7f826aa00640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f825006bc00 0x7f825006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:01.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.315+0000 7f826aa00640 1 -- 172.21.15.27:0/3060741136 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f826c08ab30 con 0x7f82740c4d10 2024-03-28T19:28:01.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.315+0000 7f8272a00640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f825006bc00 0x7f825006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:01.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.316+0000 7f8272a00640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f825006bc00 0x7f825006e0c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8264001370 tx=0x7f82640017b0 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:01.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.321+0000 7f826aa00640 1 -- 172.21.15.27:0/3060741136 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f826c057da0 con 0x7f82740c4d10 2024-03-28T19:28:01.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.612+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f827404f400 con 0x7f82740c4d10 2024-03-28T19:28:01.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.613+0000 7f826aa00640 1 -- 172.21.15.27:0/3060741136 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f826c05be40 con 0x7f82740c4d10 2024-03-28T19:28:01.612 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:01.613 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:01.613 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 240 2024-03-28T19:28:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f825006bc00 msgr2=0x7f825006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f825006bc00 0x7f825006e0c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8264001370 tx=0x7f82640017b0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740c4d10 msgr2=0x7f82740c4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740c4d10 0x7f82740c4050 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f82740bea60 tx=0x7f826c00d660 comp rx=0 tx=0).stop 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 shutdown_connections 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f825006bc00 0x7f825006e0c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82740c4d10 0x7f82740c4050 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82741ec040 0x7f82740c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 --2- 172.21.15.27:0/3060741136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8258004800 0x7f82740ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 >> 172.21.15.27:0/3060741136 conn(0x7f827406b6f0 msgr2=0x7f8274156000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 shutdown_connections 2024-03-28T19:28:01.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:01.616+0000 7f8279600640 1 -- 172.21.15.27:0/3060741136 wait complete. 2024-03-28T19:28:01.634 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:28:03.244 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 20% (32/159) Updating files: 21% (34/159) Updating files: 22% (35/159) Updating files: 23% (37/159) Updating files: 24% (39/159) Updating files: 25% (40/159) Updating files: 26% (42/159) Updating files: 27% (43/159) Updating files: 28% (45/159) Updating files: 29% (47/159) Updating files: 30% (48/159) Updating files: 31% (50/159) Updating files: 32% (51/159) Updating files: 33% (53/159) Updating files: 34% (55/159) Updating files: 35% (56/159) Updating files: 36% (58/159) Updating files: 37% (59/159) Updating files: 38% (61/159) Updating files: 38% (62/159) Updating files: 39% (63/159) Updating files: 40% (64/159) Updating files: 41% (66/159) Updating files: 42% (67/159) Updating files: 43% (69/159) Updating files: 44% (70/159) Updating files: 45% (72/159) Updating files: 46% (74/159) Updating files: 47% (75/159) Updating files: 48% (77/159) Updating files: 49% (78/159) Updating files: 50% (80/159) Updating files: 51% (82/159) Updating files: 52% (83/159) Updating files: 53% (85/159) Updating files: 54% (86/159) Updating files: 55% (88/159) Updating files: 56% (90/159) Updating files: 57% (91/159) Updating files: 58% (93/159) Updating files: 59% (94/159) Updating files: 60% (96/159) Updating files: 61% (97/159) Updating files: 62% (99/159) Updating files: 63% (101/159) Updating files: 64% (102/159) Updating files: 65% (104/159) Updating files: 66% (105/159) Updating files: 67% (107/159) Updating files: 68% (109/159) Updating files: 69% (110/159) Updating files: 70% (112/159) Updating files: 71% (113/159) Updating files: 72% (115/159) Updating files: 73% (117/159) Updating files: 74% (118/159) Updating files: 75% (120/159) Updating files: 76% (121/159) Updating files: 77% (123/159) Updating files: 78% (125/159) Updating files: 79% (126/159) Updating files: 80% (128/159) Updating files: 81% (129/159) Updating files: 82% (131/159) Updating files: 83% (132/159) Updating files: 84% (134/159) Updating files: 85% (136/159) Updating files: 86% (137/159) Updating files: 87% (139/159) Updating files: 88% (140/159) Updating files: 89% (142/159) Updating files: 90% (144/159) Updating files: 91% (145/159) Updating files: 92% (147/159) Updating files: 93% (148/159) Updating files: 94% (150/159) Updating files: 95% (152/159) Updating files: 96% (153/159) Updating files: 97% (155/159) Updating files: 98% (156/159) Updating files: 99% (158/159) Updating files: 100% (159/159) Updating files: 100% (159/159), done. 2024-03-28T19:28:03.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:03.685 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 8e69c39f699 v0.29 2024-03-28T19:28:03.692 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.29 2024-03-28T19:28:03.752 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:28:03.753 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 30 -eq 48 ']' 2024-03-28T19:28:03.753 INFO:tasks.workunit.client.1.smithi155.stdout:v0.30 2024-03-28T19:28:03.753 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.30 2024-03-28T19:28:03.753 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.30 2024-03-28T19:28:03.753 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.30 2024-03-28T19:28:03.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.861+0000 7fdcb5800640 1 Processor -- start 2024-03-28T19:28:03.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.861+0000 7fdcb5800640 1 -- start start 2024-03-28T19:28:03.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.862+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb00c69a0 0x7fdcb00c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.862+0000 7fdcb5800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdcb0153a30 0x7fdcb0153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.862+0000 7fdcb5800640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb0150ec0 con 0x7fdcb0150ae0 2024-03-28T19:28:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.863+0000 7fdcb5800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb006dc70 con 0x7fdcb0153a30 2024-03-28T19:28:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.863+0000 7fdcb5800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb0153550 con 0x7fdcb00c69a0 2024-03-28T19:28:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.863+0000 7fdcb4e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdcb0153a30 0x7fdcb0153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.863+0000 7fdca7400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb00c69a0 0x7fdcb00c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.863+0000 7fdcb4e00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdcb0153a30 0x7fdcb0153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:46136/0 (socket says 172.21.15.27:46136) 2024-03-28T19:28:03.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.863+0000 7fdcb4e00640 1 -- 172.21.15.27:0/1781791898 learned_addr learned my addr 172.21.15.27:0/1781791898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:03.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.864+0000 7fdcb4e00640 1 -- 172.21.15.27:0/1781791898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb00c69a0 msgr2=0x7fdcb00c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:03.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.864+0000 7fdcb4e00640 1 --2- 172.21.15.27:0/1781791898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb00c69a0 0x7fdcb00c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:03.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.864+0000 7fdcb4e00640 1 -- 172.21.15.27:0/1781791898 >> v1:172.21.15.155:6789/0 conn(0x7fdcb0150ae0 legacy=0x7fdcb00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:03.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.864+0000 7fdcb4e00640 1 -- 172.21.15.27:0/1781791898 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb006df10 con 0x7fdcb0153a30 2024-03-28T19:28:03.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.865+0000 7fdcb4e00640 1 --2- 172.21.15.27:0/1781791898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb0153a30 0x7fdcb0153ea0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009870 tx=0x7fdca802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90f4bdf523f5a80d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.866+0000 7fdcaf400640 1 -- 172.21.15.27:0/1781791898 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8035020 con 0x7fdcb0153a30 2024-03-28T19:28:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.866+0000 7fdcaf400640 1 -- 172.21.15.27:0/1781791898 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca8031640 con 0x7fdcb0153a30 2024-03-28T19:28:03.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.866+0000 7fdcaf400640 1 -- 172.21.15.27:0/1781791898 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca802d040 con 0x7fdcb0153a30 2024-03-28T19:28:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.867+0000 7fdcb5800640 1 -- 172.21.15.27:0/1781791898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb0153a30 msgr2=0x7fdcb0153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.867+0000 7fdcb5800640 1 --2- 172.21.15.27:0/1781791898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb0153a30 0x7fdcb0153ea0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009870 tx=0x7fdca802b8f0 comp rx=0 tx=0).stop 2024-03-28T19:28:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.867+0000 7fdcb5800640 1 -- 172.21.15.27:0/1781791898 shutdown_connections 2024-03-28T19:28:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.867+0000 7fdcb5800640 1 --2- 172.21.15.27:0/1781791898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb0153a30 0x7fdcb0153ea0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.867+0000 7fdcb5800640 1 --2- 172.21.15.27:0/1781791898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb00c69a0 0x7fdcb00c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.867+0000 7fdcb5800640 1 -- 172.21.15.27:0/1781791898 >> 172.21.15.27:0/1781791898 conn(0x7fdcb006b0c0 msgr2=0x7fdcb006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.867+0000 7fdcb5800640 1 -- 172.21.15.27:0/1781791898 shutdown_connections 2024-03-28T19:28:03.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.867+0000 7fdcb5800640 1 -- 172.21.15.27:0/1781791898 wait complete. 2024-03-28T19:28:03.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.869+0000 7fdcb5800640 1 Processor -- start 2024-03-28T19:28:03.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.869+0000 7fdcb5800640 1 -- start start 2024-03-28T19:28:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.870+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb00c69a0 0x7fdcb01e2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.870+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb0150ae0 0x7fdcb0152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.870+0000 7fdcb5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb0153a30 0x7fdcb00a7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb5800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb006e7b0 con 0x7fdcb0153a30 2024-03-28T19:28:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb5800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb006e4d0 con 0x7fdcb00c69a0 2024-03-28T19:28:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb5800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00a6370 con 0x7fdcb0150ae0 2024-03-28T19:28:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb0153a30 0x7fdcb00a7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:03.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb0153a30 0x7fdcb00a7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46590/0 (socket says 172.21.15.27:46590) 2024-03-28T19:28:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb4e00640 1 -- 172.21.15.27:0/3948599554 learned_addr learned my addr 172.21.15.27:0/3948599554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcafe00640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb00c69a0 0x7fdcb01e2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:03.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdca7400640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb0150ae0 0x7fdcb0152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb4e00640 1 -- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb0150ae0 msgr2=0x7fdcb0152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb4e00640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb0150ae0 0x7fdcb0152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb4e00640 1 -- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb00c69a0 msgr2=0x7fdcb01e2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb4e00640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb00c69a0 0x7fdcb01e2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcb4e00640 1 -- 172.21.15.27:0/3948599554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb01e2690 con 0x7fdcb0153a30 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdca7400640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb0150ae0 0x7fdcb0152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.871+0000 7fdcafe00640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb00c69a0 0x7fdcb01e2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.872+0000 7fdcb4e00640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb0153a30 0x7fdcb00a7b20 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009870 tx=0x7fdca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.872+0000 7fdcacc00640 1 -- 172.21.15.27:0/3948599554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8040070 con 0x7fdcb0153a30 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.872+0000 7fdcacc00640 1 -- 172.21.15.27:0/3948599554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca80399f0 con 0x7fdcb0153a30 2024-03-28T19:28:03.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.872+0000 7fdcacc00640 1 -- 172.21.15.27:0/3948599554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8035040 con 0x7fdcb0153a30 2024-03-28T19:28:03.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.872+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb0072df0 con 0x7fdcb0153a30 2024-03-28T19:28:03.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.872+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb0073090 con 0x7fdcb0153a30 2024-03-28T19:28:03.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.874+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb0067dd0 con 0x7fdcb0153a30 2024-03-28T19:28:03.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.875+0000 7fdcacc00640 1 -- 172.21.15.27:0/3948599554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdca8038020 con 0x7fdcb0153a30 2024-03-28T19:28:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.876+0000 7fdcacc00640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.876+0000 7fdcacc00640 1 -- 172.21.15.27:0/3948599554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7fdca80ad480 con 0x7fdcb0153a30 2024-03-28T19:28:03.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.876+0000 7fdcafe00640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:03.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.877+0000 7fdcafe00640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00bfe70 tx=0x7fdca0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:03.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:03.881+0000 7fdcacc00640 1 -- 172.21.15.27:0/3948599554 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdca807ad40 con 0x7fdcb0153a30 2024-03-28T19:28:04.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.122+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcb0069640 con 0x7fdcb0153a30 2024-03-28T19:28:04.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.123+0000 7fdcacc00640 1 -- 172.21.15.27:0/3948599554 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fdca807a640 con 0x7fdcb0153a30 2024-03-28T19:28:04.122 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:04.123 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:04.123 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 240 2024-03-28T19:28:04.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc8c06bbd0 msgr2=0x7fdc8c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00bfe70 tx=0x7fdca0009040 comp rx=0 tx=0).stop 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb0153a30 msgr2=0x7fdcb00a7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb0153a30 0x7fdcb00a7b20 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009870 tx=0x7fdca80046c0 comp rx=0 tx=0).stop 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 shutdown_connections 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fdc8c06bbd0 0x7fdc8c06e090 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcb0153a30 0x7fdcb00a7b20 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdcb0150ae0 0x7fdcb0152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 --2- 172.21.15.27:0/3948599554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcb00c69a0 0x7fdcb01e2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 >> 172.21.15.27:0/3948599554 conn(0x7fdcb006b0c0 msgr2=0x7fdcb00c0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 shutdown_connections 2024-03-28T19:28:04.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:04.126+0000 7fdcb5800640 1 -- 172.21.15.27:0/3948599554 wait complete. 2024-03-28T19:28:04.145 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:28:06.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.384+0000 7f5e67200640 1 Processor -- start 2024-03-28T19:28:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.384+0000 7f5e67200640 1 -- start start 2024-03-28T19:28:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.386+0000 7f5e67200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5e600c8f00 0x7f5e6014f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.386+0000 7f5e67200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600c9350 con 0x7f5e600c59d0 2024-03-28T19:28:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.386+0000 7f5e67200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e60155d10 con 0x7f5e600727e0 2024-03-28T19:28:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.386+0000 7f5e67200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e601565a0 con 0x7f5e600c8f00 2024-03-28T19:28:06.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.386+0000 7f5e65e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5e600c59d0 0x7f5e60159520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53944/0 (socket says 172.21.15.27:53944) 2024-03-28T19:28:06.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.386+0000 7f5e65e00640 1 -- 172.21.15.27:0/1910574475 learned_addr learned my addr 172.21.15.27:0/1910574475 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:06.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e66800640 1 --2- 172.21.15.27:0/1910574475 >> v2:172.21.15.155:3300/0 conn(0x7f5e600c8f00 0x7f5e6014f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:06.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64873388 0 0) 0x7f5e600c9350 con 0x7f5e600c59d0 2024-03-28T19:28:06.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e48003520 con 0x7f5e600c59d0 2024-03-28T19:28:06.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4187604861 0 0) 0x7f5e48003520 con 0x7f5e600c59d0 2024-03-28T19:28:06.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 >> v2:172.21.15.155:3300/0 conn(0x7f5e600c8f00 msgr2=0x7f5e6014f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:06.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 --2- 172.21.15.27:0/1910574475 >> v2:172.21.15.155:3300/0 conn(0x7f5e600c8f00 0x7f5e6014f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 >> v1:172.21.15.155:6789/0 conn(0x7f5e600727e0 legacy=0x7f5e60155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e60155fe0 con 0x7f5e600c59d0 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f5e54001670 con 0x7f5e600c59d0 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 >> v1:172.21.15.27:6789/0 conn(0x7f5e600c59d0 legacy=0x7f5e60159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e480039b0 0x7f5e48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e48004800 0x7f5e48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e480052e0 0x7f5e4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e601565a0 con 0x7f5e48004800 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4800c1b0 con 0x7f5e480052e0 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.387+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4800c3b0 con 0x7f5e480039b0 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.388+0000 7f5e65400640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e48004800 0x7f5e48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.388+0000 7f5e66800640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e480052e0 0x7f5e4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.388+0000 7f5e65400640 1 -- 172.21.15.27:0/1910574475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e480039b0 msgr2=0x7f5e48003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.388+0000 7f5e65400640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e480039b0 0x7f5e48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.388+0000 7f5e65400640 1 -- 172.21.15.27:0/1910574475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e480052e0 msgr2=0x7f5e4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.388+0000 7f5e65400640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e480052e0 0x7f5e4800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.388+0000 7f5e65400640 1 -- 172.21.15.27:0/1910574475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e48003520 con 0x7f5e48004800 2024-03-28T19:28:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.388+0000 7f5e66800640 1 -- 172.21.15.27:0/1910574475 reap_dead start 2024-03-28T19:28:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.389+0000 7f5e65400640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e48004800 0x7f5e48004c80 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5e6006c830 tx=0x7f5e48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ef8badb9be40d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50001370 con 0x7f5e48004800 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e500052d0 con 0x7f5e48004800 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e64a00640 1 -- 172.21.15.27:0/1910574475 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50003830 con 0x7f5e48004800 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e67200640 1 -- 172.21.15.27:0/1910574475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e48004800 msgr2=0x7f5e48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e67200640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e48004800 0x7f5e48004c80 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5e6006c830 tx=0x7f5e48005830 comp rx=0 tx=0).stop 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e67200640 1 -- 172.21.15.27:0/1910574475 shutdown_connections 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e67200640 1 --2- 172.21.15.27:0/1910574475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e48004800 0x7f5e48004c80 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e67200640 1 -- 172.21.15.27:0/1910574475 >> 172.21.15.27:0/1910574475 conn(0x7f5e600ac910 msgr2=0x7f5e600accf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e67200640 1 -- 172.21.15.27:0/1910574475 shutdown_connections 2024-03-28T19:28:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.390+0000 7f5e67200640 1 -- 172.21.15.27:0/1910574475 wait complete. 2024-03-28T19:28:06.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.392+0000 7f5e67200640 1 Processor -- start 2024-03-28T19:28:06.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.392+0000 7f5e67200640 1 -- start start 2024-03-28T19:28:06.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.393+0000 7f5e67200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e48004800 0x7f5e600aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.393+0000 7f5e67200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e60147cd0 0x7f5e600c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.393+0000 7f5e67200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e60148220 0x7f5e600c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.393+0000 7f5e67200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60158d20 con 0x7f5e60148220 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.393+0000 7f5e67200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60159090 con 0x7f5e60147cd0 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.393+0000 7f5e67200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600c5db0 con 0x7f5e48004800 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.393+0000 7f5e66800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e60148220 0x7f5e600c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e66800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e60148220 0x7f5e600c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46762/0 (socket says 172.21.15.27:46762) 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e66800640 1 -- 172.21.15.27:0/682156472 learned_addr learned my addr 172.21.15.27:0/682156472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e65e00640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e48004800 0x7f5e600aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:06.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e65400640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e60147cd0 0x7f5e600c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e66800640 1 -- 172.21.15.27:0/682156472 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e48004800 msgr2=0x7f5e600aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e66800640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e48004800 0x7f5e600aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e66800640 1 -- 172.21.15.27:0/682156472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e60147cd0 msgr2=0x7f5e600c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e66800640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e60147cd0 0x7f5e600c44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e66800640 1 -- 172.21.15.27:0/682156472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6014abc0 con 0x7f5e60148220 2024-03-28T19:28:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.394+0000 7f5e66800640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e60148220 0x7f5e600c4a30 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f5e60156770 tx=0x7f5e5c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.395+0000 7f5e5a000640 1 -- 172.21.15.27:0/682156472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c011db0 con 0x7f5e60148220 2024-03-28T19:28:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.395+0000 7f5e5a000640 1 -- 172.21.15.27:0/682156472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5c01b070 con 0x7f5e60148220 2024-03-28T19:28:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.395+0000 7f5e5a000640 1 -- 172.21.15.27:0/682156472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c003990 con 0x7f5e60148220 2024-03-28T19:28:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.395+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6014a8e0 con 0x7f5e60148220 2024-03-28T19:28:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.395+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e600c39c0 con 0x7f5e60148220 2024-03-28T19:28:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.396+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e60067dd0 con 0x7f5e60148220 2024-03-28T19:28:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.397+0000 7f5e5a000640 1 -- 172.21.15.27:0/682156472 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5e5c005d10 con 0x7f5e60148220 2024-03-28T19:28:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.398+0000 7f5e5a000640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e4406bc00 0x7f5e4406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.398+0000 7f5e5a000640 1 -- 172.21.15.27:0/682156472 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f5e5c08c120 con 0x7f5e60148220 2024-03-28T19:28:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.398+0000 7f5e65e00640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e4406bc00 0x7f5e4406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:06.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.399+0000 7f5e65e00640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e4406bc00 0x7f5e4406e0c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5e54000ff0 tx=0x7f5e54001570 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.403+0000 7f5e5a000640 1 -- 172.21.15.27:0/682156472 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5e5c01d310 con 0x7f5e60148220 2024-03-28T19:28:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.651+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e60069640 con 0x7f5e60148220 2024-03-28T19:28:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.653+0000 7f5e5a000640 1 -- 172.21.15.27:0/682156472 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f5e5c0599e0 con 0x7f5e60148220 2024-03-28T19:28:06.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:06.652 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 240 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.655+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e4406bc00 msgr2=0x7f5e4406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.655+0000 7f5e67200640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e4406bc00 0x7f5e4406e0c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5e54000ff0 tx=0x7f5e54001570 comp rx=0 tx=0).stop 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.655+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e60148220 msgr2=0x7f5e600c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.655+0000 7f5e67200640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e60148220 0x7f5e600c4a30 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f5e60156770 tx=0x7f5e5c0115a0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.656+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 shutdown_connections 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.656+0000 7f5e67200640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f5e4406bc00 0x7f5e4406e0c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.656+0000 7f5e67200640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e60148220 0x7f5e600c4a30 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.656+0000 7f5e67200640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e60147cd0 0x7f5e600c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.656+0000 7f5e67200640 1 --2- 172.21.15.27:0/682156472 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e48004800 0x7f5e600aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.656+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 >> 172.21.15.27:0/682156472 conn(0x7f5e600ac910 msgr2=0x7f5e60158200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.656+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 shutdown_connections 2024-03-28T19:28:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:06.656+0000 7f5e67200640 1 -- 172.21.15.27:0/682156472 wait complete. 2024-03-28T19:28:06.675 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:28:08.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:08.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.905+0000 7f63e6200640 1 Processor -- start 2024-03-28T19:28:08.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.905+0000 7f63e6200640 1 -- start start 2024-03-28T19:28:08.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.906+0000 7f63e6200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e00aada0 0x7f63e00c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:08.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.906+0000 7f63e6200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f63e00ab2b0 0x7f63e00c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:08.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.906+0000 7f63e6200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e0156870 con 0x7f63e01551a0 2024-03-28T19:28:08.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.906+0000 7f63e6200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e0155cf0 con 0x7f63e00ab2b0 2024-03-28T19:28:08.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.906+0000 7f63e6200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0155fd0 con 0x7f63e00aada0 2024-03-28T19:28:08.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.907+0000 7f63e5800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f63e00ab2b0 0x7f63e00c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.907+0000 7f63e5800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f63e00ab2b0 0x7f63e00c1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:57378/0 (socket says 172.21.15.27:57378) 2024-03-28T19:28:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.907+0000 7f63e5800640 1 -- 172.21.15.27:0/196459097 learned_addr learned my addr 172.21.15.27:0/196459097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.907+0000 7f63e4e00640 1 --1- 172.21.15.27:0/196459097 >> v1:172.21.15.155:6789/0 conn(0x7f63e01551a0 0x7f63e00c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:48784/0 (socket says 172.21.15.27:48784) 2024-03-28T19:28:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.907+0000 7f63d7e00640 1 --2- 172.21.15.27:0/196459097 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e00aada0 0x7f63e00c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.908+0000 7f63d7e00640 1 -- 172.21.15.27:0/196459097 >> v2:172.21.15.155:3300/0 conn(0x7f63e00ab2b0 msgr2=0x7f63e00c1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.908+0000 7f63d7e00640 1 --2- 172.21.15.27:0/196459097 >> v2:172.21.15.155:3300/0 conn(0x7f63e00ab2b0 0x7f63e00c1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.908+0000 7f63d7e00640 1 -- 172.21.15.27:0/196459097 >> v1:172.21.15.155:6789/0 conn(0x7f63e01551a0 legacy=0x7f63e00c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:08.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.908+0000 7f63d7e00640 1 -- 172.21.15.27:0/196459097 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e01562b0 con 0x7f63e00aada0 2024-03-28T19:28:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.909+0000 7f63d7e00640 1 --2- 172.21.15.27:0/196459097 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e00aada0 0x7f63e00c1a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f63c80097a0 tx=0x7f63c802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=470631ccb3475a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.909+0000 7f63dfe00640 1 -- 172.21.15.27:0/196459097 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8038070 con 0x7f63e00aada0 2024-03-28T19:28:08.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63dfe00640 1 -- 172.21.15.27:0/196459097 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c8031700 con 0x7f63e00aada0 2024-03-28T19:28:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63dfe00640 1 -- 172.21.15.27:0/196459097 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8031ad0 con 0x7f63e00aada0 2024-03-28T19:28:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63e6200640 1 -- 172.21.15.27:0/196459097 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e00aada0 msgr2=0x7f63e00c1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63e6200640 1 --2- 172.21.15.27:0/196459097 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e00aada0 0x7f63e00c1a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f63c80097a0 tx=0x7f63c802b7a0 comp rx=0 tx=0).stop 2024-03-28T19:28:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63e6200640 1 -- 172.21.15.27:0/196459097 shutdown_connections 2024-03-28T19:28:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63e6200640 1 --2- 172.21.15.27:0/196459097 >> v2:172.21.15.155:3300/0 conn(0x7f63e00ab2b0 0x7f63e00c1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63e6200640 1 --2- 172.21.15.27:0/196459097 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e00aada0 0x7f63e00c1a80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63e6200640 1 -- 172.21.15.27:0/196459097 >> 172.21.15.27:0/196459097 conn(0x7f63e006b0c0 msgr2=0x7f63e006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.910+0000 7f63e6200640 1 -- 172.21.15.27:0/196459097 shutdown_connections 2024-03-28T19:28:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.911+0000 7f63e6200640 1 -- 172.21.15.27:0/196459097 wait complete. 2024-03-28T19:28:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.913+0000 7f63e6200640 1 Processor -- start 2024-03-28T19:28:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.913+0000 7f63e6200640 1 -- start start 2024-03-28T19:28:08.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63e6200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63e00aada0 0x7f63e01450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63e6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e00ab2b0 0x7f63e0149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63e6200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e01551a0 0x7f63e0071b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63e6200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c5750 con 0x7f63e00ab2b0 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63e6200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e01559a0 con 0x7f63e00aada0 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63e6200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c58d0 con 0x7f63e01551a0 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63d7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e00ab2b0 0x7f63e0149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63d7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e00ab2b0 0x7f63e0149ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46764/0 (socket says 172.21.15.27:46764) 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63d7e00640 1 -- 172.21.15.27:0/2494484535 learned_addr learned my addr 172.21.15.27:0/2494484535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.914+0000 7f63e4e00640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63e00aada0 0x7f63e01450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.915+0000 7f63e5800640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e01551a0 0x7f63e0071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.915+0000 7f63d7e00640 1 -- 172.21.15.27:0/2494484535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e01551a0 msgr2=0x7f63e0071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.915+0000 7f63d7e00640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e01551a0 0x7f63e0071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.915+0000 7f63d7e00640 1 -- 172.21.15.27:0/2494484535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63e00aada0 msgr2=0x7f63e01450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.915+0000 7f63d7e00640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63e00aada0 0x7f63e01450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.915+0000 7f63d7e00640 1 -- 172.21.15.27:0/2494484535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e01476b0 con 0x7f63e00ab2b0 2024-03-28T19:28:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.915+0000 7f63d7e00640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e00ab2b0 0x7f63e0149ea0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f63c80098c0 tx=0x7f63c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.916+0000 7f63dd600640 1 -- 172.21.15.27:0/2494484535 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8038040 con 0x7f63e00ab2b0 2024-03-28T19:28:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.916+0000 7f63dd600640 1 -- 172.21.15.27:0/2494484535 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c803c6d0 con 0x7f63e00ab2b0 2024-03-28T19:28:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.916+0000 7f63dd600640 1 -- 172.21.15.27:0/2494484535 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c803caa0 con 0x7f63e00ab2b0 2024-03-28T19:28:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.916+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e01ef2d0 con 0x7f63e00ab2b0 2024-03-28T19:28:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.916+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e01eb2d0 con 0x7f63e00ab2b0 2024-03-28T19:28:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.917+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e0067dd0 con 0x7f63e00ab2b0 2024-03-28T19:28:08.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.918+0000 7f63dd600640 1 -- 172.21.15.27:0/2494484535 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f63c8045020 con 0x7f63e00ab2b0 2024-03-28T19:28:08.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.919+0000 7f63dd600640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63bc06bc00 0x7f63bc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:08.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.919+0000 7f63dd600640 1 -- 172.21.15.27:0/2494484535 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 10335+0+0 (secure 0 0 0) 0x7f63c80acc30 con 0x7f63e00ab2b0 2024-03-28T19:28:08.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.919+0000 7f63e4e00640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63bc06bc00 0x7f63bc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:08.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.920+0000 7f63e4e00640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63bc06bc00 0x7f63bc06e0c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f63e00c0a80 tx=0x7f63d0007e80 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:08.924+0000 7f63dd600640 1 -- 172.21.15.27:0/2494484535 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f63c80b3080 con 0x7f63e00ab2b0 2024-03-28T19:28:09.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.175+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e01549c0 con 0x7f63e00ab2b0 2024-03-28T19:28:09.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.177+0000 7f63dd600640 1 -- 172.21.15.27:0/2494484535 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f63e01476b0 con 0x7f63e00ab2b0 2024-03-28T19:28:09.176 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:09.176 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228},{"gid":9156,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:26:16.003326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6740},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6740":{"gid":6740,"name":"b","rank":0,"incarnation":222,"state":"up:active","state_seq":20,"addr":"172.21.15.155:6835/1734256169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6833","nonce":1734256169},{"type":"v1","addr":"172.21.15.155:6835","nonce":1734256169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6740,"qdb_cluster":[6740]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:09.176 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 240 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.179+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63bc06bc00 msgr2=0x7f63bc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.179+0000 7f63e6200640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63bc06bc00 0x7f63bc06e0c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f63e00c0a80 tx=0x7f63d0007e80 comp rx=0 tx=0).stop 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.179+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e00ab2b0 msgr2=0x7f63e0149ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.179+0000 7f63e6200640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e00ab2b0 0x7f63e0149ea0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f63c80098c0 tx=0x7f63c80046c0 comp rx=0 tx=0).stop 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.180+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 shutdown_connections 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.180+0000 7f63e6200640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f63bc06bc00 0x7f63bc06e0c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.180+0000 7f63e6200640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63e01551a0 0x7f63e0071b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.180+0000 7f63e6200640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63e00ab2b0 0x7f63e0149ea0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.180+0000 7f63e6200640 1 --2- 172.21.15.27:0/2494484535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63e00aada0 0x7f63e01450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:09.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.180+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 >> 172.21.15.27:0/2494484535 conn(0x7f63e006b0c0 msgr2=0x7f63e006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:09.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.180+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 shutdown_connections 2024-03-28T19:28:09.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:09.180+0000 7f63e6200640 1 -- 172.21.15.27:0/2494484535 wait complete. 2024-03-28T19:28:09.199 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-03-28T19:28:11.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.430+0000 7f673cc00640 1 Processor -- start 2024-03-28T19:28:11.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.431+0000 7f673cc00640 1 -- start start 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.432+0000 7f673cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6738152f80 0x7f67380c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.432+0000 7f673cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f67380c7df0 0x7f6738155e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.432+0000 7f673cc00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67380c7310 con 0x7f67380c7970 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.432+0000 7f673cc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6738158d40 con 0x7f6738152f80 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.432+0000 7f673cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6738159020 con 0x7f67380c7df0 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.432+0000 7f6735e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6738152f80 0x7f67380c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.432+0000 7f6735e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6738152f80 0x7f67380c58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46766/0 (socket says 172.21.15.27:46766) 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.432+0000 7f6735e00640 1 -- 172.21.15.27:0/3789346329 learned_addr learned my addr 172.21.15.27:0/3789346329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.433+0000 7f6736800640 1 --2- 172.21.15.27:0/3789346329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f67380c7df0 0x7f6738155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.433+0000 7f6735e00640 1 -- 172.21.15.27:0/3789346329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f67380c7df0 msgr2=0x7f6738155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.433+0000 7f6735e00640 1 --2- 172.21.15.27:0/3789346329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f67380c7df0 0x7f6738155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.433+0000 7f6735e00640 1 -- 172.21.15.27:0/3789346329 >> v1:172.21.15.155:6789/0 conn(0x7f67380c7970 legacy=0x7f6738155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.433+0000 7f6735e00640 1 -- 172.21.15.27:0/3789346329 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6738159300 con 0x7f6738152f80 2024-03-28T19:28:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.434+0000 7f6735e00640 1 --2- 172.21.15.27:0/3789346329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6738152f80 0x7f67380c58e0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f6728009870 tx=0x7f672802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec24c58513bbc49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:11.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.435+0000 7f6734a00640 1 -- 172.21.15.27:0/3789346329 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67280317c0 con 0x7f6738152f80 2024-03-28T19:28:11.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.435+0000 7f6734a00640 1 -- 172.21.15.27:0/3789346329 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6728037070 con 0x7f6738152f80 2024-03-28T19:28:11.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.435+0000 7f6734a00640 1 -- 172.21.15.27:0/3789346329 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728031b80 con 0x7f6738152f80 2024-03-28T19:28:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.435+0000 7f673cc00640 1 -- 172.21.15.27:0/3789346329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6738152f80 msgr2=0x7f67380c58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.435+0000 7f673cc00640 1 --2- 172.21.15.27:0/3789346329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6738152f80 0x7f67380c58e0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f6728009870 tx=0x7f672802b860 comp rx=0 tx=0).stop 2024-03-28T19:28:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.436+0000 7f673cc00640 1 -- 172.21.15.27:0/3789346329 shutdown_connections 2024-03-28T19:28:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.436+0000 7f673cc00640 1 --2- 172.21.15.27:0/3789346329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f67380c7df0 0x7f6738155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.436+0000 7f673cc00640 1 --2- 172.21.15.27:0/3789346329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6738152f80 0x7f67380c58e0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.436+0000 7f673cc00640 1 -- 172.21.15.27:0/3789346329 >> 172.21.15.27:0/3789346329 conn(0x7f673806b6f0 msgr2=0x7f67380caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.436+0000 7f673cc00640 1 -- 172.21.15.27:0/3789346329 shutdown_connections 2024-03-28T19:28:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.436+0000 7f673cc00640 1 -- 172.21.15.27:0/3789346329 wait complete. 2024-03-28T19:28:11.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.437+0000 7f673cc00640 1 Processor -- start 2024-03-28T19:28:11.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.437+0000 7f673cc00640 1 -- start start 2024-03-28T19:28:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.438+0000 7f673cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67380c7970 0x7f6738157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.438+0000 7f673cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67380c7df0 0x7f67381535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.438+0000 7f673cc00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6738152f80 0x7f6738153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.438+0000 7f673cc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6738159b00 con 0x7f67380c7970 2024-03-28T19:28:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.438+0000 7f673cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67380c5e20 con 0x7f67380c7df0 2024-03-28T19:28:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.438+0000 7f673cc00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67380cbca0 con 0x7f6738152f80 2024-03-28T19:28:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67380c7970 0x7f6738157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67380c7970 0x7f6738157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46780/0 (socket says 172.21.15.27:46780) 2024-03-28T19:28:11.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735e00640 1 -- 172.21.15.27:0/3218071999 learned_addr learned my addr 172.21.15.27:0/3218071999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:11.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735400640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67380c7df0 0x7f67381535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6736800640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6738152f80 0x7f6738153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735e00640 1 -- 172.21.15.27:0/3218071999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6738152f80 msgr2=0x7f6738153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:11.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735e00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6738152f80 0x7f6738153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735e00640 1 -- 172.21.15.27:0/3218071999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67380c7df0 msgr2=0x7f67381535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:11.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735e00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67380c7df0 0x7f67381535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.439+0000 7f6735e00640 1 -- 172.21.15.27:0/3218071999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6738149960 con 0x7f67380c7970 2024-03-28T19:28:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.440+0000 7f6735e00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67380c7970 0x7f6738157d60 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f6728000c00 tx=0x7f67280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.440+0000 7f672e000640 1 -- 172.21.15.27:0/3218071999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728032860 con 0x7f67380c7970 2024-03-28T19:28:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.440+0000 7f672e000640 1 -- 172.21.15.27:0/3218071999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6728032a00 con 0x7f67380c7970 2024-03-28T19:28:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.440+0000 7f672e000640 1 -- 172.21.15.27:0/3218071999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728032df0 con 0x7f67380c7970 2024-03-28T19:28:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.440+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67381eb340 con 0x7f67380c7970 2024-03-28T19:28:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.440+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67381582a0 con 0x7f67380c7970 2024-03-28T19:28:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.441+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673804f400 con 0x7f67380c7970 2024-03-28T19:28:11.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.443+0000 7f672e000640 1 -- 172.21.15.27:0/3218071999 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f672804c060 con 0x7f67380c7970 2024-03-28T19:28:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.444+0000 7f672e000640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f671806bc20 0x7f671806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.444+0000 7f672e000640 1 -- 172.21.15.27:0/3218071999 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f67280ad550 con 0x7f67380c7970 2024-03-28T19:28:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.444+0000 7f6735400640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f671806bc20 0x7f671806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.445+0000 7f6735400640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f671806bc20 0x7f671806e0e0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f673814a3b0 tx=0x7f6724009060 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.448+0000 7f672e000640 1 -- 172.21.15.27:0/3218071999 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6728040070 con 0x7f67380c7970 2024-03-28T19:28:11.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.708+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67380cba80 con 0x7f67380c7970 2024-03-28T19:28:11.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.708+0000 7f672e000640 1 -- 172.21.15.27:0/3218071999 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f672807af50 con 0x7f67380c7970 2024-03-28T19:28:11.707 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:11.708 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:10.537892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:11.708 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 241 2024-03-28T19:28:11.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.710+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f671806bc20 msgr2=0x7f671806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.710+0000 7f673cc00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f671806bc20 0x7f671806e0e0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f673814a3b0 tx=0x7f6724009060 comp rx=0 tx=0).stop 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.710+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67380c7970 msgr2=0x7f6738157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.710+0000 7f673cc00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67380c7970 0x7f6738157d60 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f6728000c00 tx=0x7f67280046c0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.711+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 shutdown_connections 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.711+0000 7f673cc00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f671806bc20 0x7f671806e0e0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.711+0000 7f673cc00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6738152f80 0x7f6738153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.711+0000 7f673cc00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67380c7df0 0x7f67381535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.711+0000 7f673cc00640 1 --2- 172.21.15.27:0/3218071999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67380c7970 0x7f6738157d60 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.711+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 >> 172.21.15.27:0/3218071999 conn(0x7f673806b6f0 msgr2=0x7f673814f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.711+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 shutdown_connections 2024-03-28T19:28:11.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.711+0000 7f673cc00640 1 -- 172.21.15.27:0/3218071999 wait complete. 2024-03-28T19:28:11.728 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-03-28T19:28:11.728 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-03-28T19:28:11.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:11.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.919+0000 7f8246a00640 1 Processor -- start 2024-03-28T19:28:11.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.919+0000 7f8246a00640 1 -- start start 2024-03-28T19:28:11.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.921+0000 7f8246a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.921+0000 7f8246a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f82400c3690 0x7f82400c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.921+0000 7f8246a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82400c6660 0x7f82401534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.921+0000 7f8246a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82400ab4e0 con 0x7f82400c6660 2024-03-28T19:28:11.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.921+0000 7f8246a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82400ac1f0 con 0x7f82400c3690 2024-03-28T19:28:11.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.921+0000 7f8246a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82400c3af0 con 0x7f8240150c70 2024-03-28T19:28:11.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f8244a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82400c6660 0x7f82401534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823fe00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:59662/0 (socket says 172.21.15.27:59662) 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f8244a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f82400c6660 0x7f82401534c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46782/0 (socket says 172.21.15.27:46782) 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823fe00640 1 -- 172.21.15.27:0/2524120718 learned_addr learned my addr 172.21.15.27:0/2524120718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823f400640 1 --2- 172.21.15.27:0/2524120718 >> v2:172.21.15.155:3300/0 conn(0x7f82400c3690 0x7f82400c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823fe00640 1 -- 172.21.15.27:0/2524120718 >> v2:172.21.15.155:3300/0 conn(0x7f82400c3690 msgr2=0x7f82400c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823fe00640 1 --2- 172.21.15.27:0/2524120718 >> v2:172.21.15.155:3300/0 conn(0x7f82400c3690 0x7f82400c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823fe00640 1 -- 172.21.15.27:0/2524120718 >> v2:172.21.15.27:3300/0 conn(0x7f82400c6660 msgr2=0x7f82401534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823fe00640 1 --2- 172.21.15.27:0/2524120718 >> v2:172.21.15.27:3300/0 conn(0x7f82400c6660 0x7f82401534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823fe00640 1 -- 172.21.15.27:0/2524120718 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82400c3d90 con 0x7f8240150c70 2024-03-28T19:28:11.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.922+0000 7f823f400640 1 --2- 172.21.15.27:0/2524120718 >> v2:172.21.15.155:3300/0 conn(0x7f82400c3690 0x7f82400c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:28:11.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.924+0000 7f823fe00640 1 --2- 172.21.15.27:0/2524120718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9cb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f824006f180 tx=0x7f823002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fdc5ed6da3dce42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:11.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.925+0000 7f823ea00640 1 -- 172.21.15.27:0/2524120718 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82300317c0 con 0x7f8240150c70 2024-03-28T19:28:11.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.925+0000 7f823ea00640 1 -- 172.21.15.27:0/2524120718 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8230037070 con 0x7f8240150c70 2024-03-28T19:28:11.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.925+0000 7f823ea00640 1 -- 172.21.15.27:0/2524120718 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230031b80 con 0x7f8240150c70 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 -- 172.21.15.27:0/2524120718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 msgr2=0x7f82400a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 --2- 172.21.15.27:0/2524120718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9cb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f824006f180 tx=0x7f823002b860 comp rx=0 tx=0).stop 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 -- 172.21.15.27:0/2524120718 shutdown_connections 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 --2- 172.21.15.27:0/2524120718 >> v2:172.21.15.27:3300/0 conn(0x7f82400c6660 0x7f82401534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 --2- 172.21.15.27:0/2524120718 >> v2:172.21.15.155:3300/0 conn(0x7f82400c3690 0x7f82400c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 --2- 172.21.15.27:0/2524120718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9cb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 -- 172.21.15.27:0/2524120718 >> 172.21.15.27:0/2524120718 conn(0x7f824006b9d0 msgr2=0x7f82400c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 -- 172.21.15.27:0/2524120718 shutdown_connections 2024-03-28T19:28:11.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.926+0000 7f8246a00640 1 -- 172.21.15.27:0/2524120718 wait complete. 2024-03-28T19:28:11.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.929+0000 7f8246a00640 1 Processor -- start 2024-03-28T19:28:11.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.929+0000 7f8246a00640 1 -- start start 2024-03-28T19:28:11.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.930+0000 7f8246a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82400c3690 0x7f82400c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.930+0000 7f8246a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82400c6660 0x7f82400a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f8246a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f8246a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82400abcd0 con 0x7f82400c6660 2024-03-28T19:28:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f8246a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82400ab9f0 con 0x7f82400c3690 2024-03-28T19:28:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f8246a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82400c1810 con 0x7f8240150c70 2024-03-28T19:28:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f8244a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f823f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82400c6660 0x7f82400a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f8244a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:59674/0 (socket says 172.21.15.27:59674) 2024-03-28T19:28:11.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f8244a00640 1 -- 172.21.15.27:0/3353388355 learned_addr learned my addr 172.21.15.27:0/3353388355 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:11.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f823f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82400c6660 0x7f82400a8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46784/0 (socket says 172.21.15.27:46784) 2024-03-28T19:28:11.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.931+0000 7f823fe00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82400c3690 0x7f82400c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.932+0000 7f823f400640 1 -- 172.21.15.27:0/3353388355 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 msgr2=0x7f82400a9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.932+0000 7f823f400640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.932+0000 7f823f400640 1 -- 172.21.15.27:0/3353388355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82400c3690 msgr2=0x7f82400c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.932+0000 7f823f400640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82400c3690 0x7f82400c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.932+0000 7f823f400640 1 -- 172.21.15.27:0/3353388355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82400a95d0 con 0x7f82400c6660 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.932+0000 7f823fe00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82400c3690 0x7f82400c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.933+0000 7f823f400640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82400c6660 0x7f82400a8b50 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f822c006fd0 tx=0x7f822c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.933+0000 7f8237e00640 1 -- 172.21.15.27:0/3353388355 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c004030 con 0x7f82400c6660 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.933+0000 7f8237e00640 1 -- 172.21.15.27:0/3353388355 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822c0041d0 con 0x7f82400c6660 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.933+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824006d220 con 0x7f82400c6660 2024-03-28T19:28:11.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.933+0000 7f8237e00640 1 -- 172.21.15.27:0/3353388355 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c00b6c0 con 0x7f82400c6660 2024-03-28T19:28:11.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.933+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8240156d30 con 0x7f82400c6660 2024-03-28T19:28:11.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.935+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824004f400 con 0x7f82400c6660 2024-03-28T19:28:11.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.936+0000 7f8237e00640 1 -- 172.21.15.27:0/3353388355 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f822c00b860 con 0x7f82400c6660 2024-03-28T19:28:11.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.937+0000 7f8237e00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f822006bbd0 0x7f822006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:11.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.937+0000 7f8237e00640 1 -- 172.21.15.27:0/3353388355 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f822c08bdf0 con 0x7f82400c6660 2024-03-28T19:28:11.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.938+0000 7f823fe00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f822006bbd0 0x7f822006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:11.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.938+0000 7f823fe00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f822006bbd0 0x7f822006e090 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8230036000 tx=0x7f8230008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:11.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:11.947+0000 7f8237e00640 1 -- 172.21.15.27:0/3353388355 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f822c0596b0 con 0x7f82400c6660 2024-03-28T19:28:12.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.204+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8240153920 con 0x7f82400c6660 2024-03-28T19:28:12.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.205+0000 7f8237e00640 1 -- 172.21.15.27:0/3353388355 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f822c058fb0 con 0x7f82400c6660 2024-03-28T19:28:12.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:12.204 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:10.537892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:12.204 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 241 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f822006bbd0 msgr2=0x7f822006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f822006bbd0 0x7f822006e090 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8230036000 tx=0x7f8230008030 comp rx=0 tx=0).stop 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82400c6660 msgr2=0x7f82400a8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82400c6660 0x7f82400a8b50 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f822c006fd0 tx=0x7f822c00dc60 comp rx=0 tx=0).stop 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 shutdown_connections 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f822006bbd0 0x7f822006e090 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8240150c70 0x7f82400a9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82400c6660 0x7f82400a8b50 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.207+0000 7f8246a00640 1 --2- 172.21.15.27:0/3353388355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82400c3690 0x7f82400c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.208+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 >> 172.21.15.27:0/3353388355 conn(0x7f824006b9d0 msgr2=0x7f824014eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:12.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.208+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 shutdown_connections 2024-03-28T19:28:12.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:12.208+0000 7f8246a00640 1 -- 172.21.15.27:0/3353388355 wait complete. 2024-03-28T19:28:12.226 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 241, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 241, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_9156": { "addr": "172.21.15.27:6837/3692255388", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 3692255388, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 3692255388, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9156, "incarnation": 241, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 95, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:28:10.537892+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9156 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 239, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_9030": { "addr": "172.21.15.155:6834/1152401013", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 1152401013, "type": "v2" }, { "addr": "172.21.15.155:6834", "nonce": 1152401013, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9030, "incarnation": 236, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 94, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:27:46.906340+0000", "qdb_cluster": [ 9030 ], "qdb_leader": 9030, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9030 } } } ], "standbys": [ { "addr": "172.21.15.27:6835/796801387", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 796801387, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 796801387, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 228, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8913, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:28:13.187 INFO:tasks.workunit.client.1.smithi155.stderr:Updating files: 17% (50/285) Updating files: 18% (52/285) Updating files: 19% (55/285) Updating files: 20% (57/285) Updating files: 21% (60/285) Updating files: 22% (63/285) Updating files: 23% (66/285) Updating files: 24% (69/285) Updating files: 25% (72/285) Updating files: 26% (75/285) Updating files: 27% (77/285) Updating files: 28% (80/285) Updating files: 29% (83/285) Updating files: 30% (86/285) Updating files: 31% (89/285) Updating files: 32% (92/285) Updating files: 33% (95/285) Updating files: 34% (97/285) Updating files: 35% (100/285) Updating files: 36% (103/285) Updating files: 37% (106/285) Updating files: 38% (109/285) Updating files: 38% (110/285) Updating files: 39% (112/285) Updating files: 40% (114/285) Updating files: 41% (117/285) Updating files: 42% (120/285) Updating files: 43% (123/285) Updating files: 44% (126/285) Updating files: 45% (129/285) Updating files: 46% (132/285) Updating files: 47% (134/285) Updating files: 48% (137/285) Updating files: 49% (140/285) Updating files: 50% (143/285) Updating files: 51% (146/285) Updating files: 52% (149/285) Updating files: 53% (152/285) Updating files: 54% (154/285) Updating files: 55% (157/285) Updating files: 56% (160/285) Updating files: 57% (163/285) Updating files: 58% (166/285) Updating files: 59% (169/285) Updating files: 60% (171/285) Updating files: 61% (174/285) Updating files: 62% (177/285) Updating files: 63% (180/285) Updating files: 64% (183/285) Updating files: 65% (186/285) Updating files: 65% (187/285) Updating files: 66% (189/285) Updating files: 67% (191/285) Updating files: 68% (194/285) Updating files: 69% (197/285) Updating files: 70% (200/285) Updating files: 71% (203/285) Updating files: 72% (206/285) Updating files: 73% (209/285) Updating files: 74% (211/285) Updating files: 75% (214/285) Updating files: 76% (217/285) Updating files: 77% (220/285) Updating files: 78% (223/285) Updating files: 79% (226/285) Updating files: 80% (228/285) Updating files: 81% (231/285) Updating files: 82% (234/285) Updating files: 83% (237/285) Updating files: 84% (240/285) Updating files: 85% (243/285) Updating files: 86% (246/285) Updating files: 87% (248/285) Updating files: 88% (251/285) Updating files: 89% (254/285) Updating files: 90% (257/285) Updating files: 91% (260/285) Updating files: 91% (262/285) Updating files: 92% (263/285) Updating files: 93% (266/285) Updating files: 94% (268/285) Updating files: 95% (271/285) Updating files: 96% (274/285) Updating files: 97% (277/285) Updating files: 98% (280/285) Updating files: 99% (283/285) Updating files: 100% (285/285) Updating files: 100% (285/285), done. 2024-03-28T19:28:13.422 INFO:tasks.workunit.client.1.smithi155.stdout:HEAD is now at 64b1b2c70f0 v0.30 2024-03-28T19:28:13.435 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir .snap/v0.30 2024-03-28T19:28:13.503 INFO:tasks.workunit.client.1.smithi155.stderr:+ for v in $versions 2024-03-28T19:28:13.503 INFO:tasks.workunit.client.1.smithi155.stderr:+ '[' 31 -eq 48 ']' 2024-03-28T19:28:13.503 INFO:tasks.workunit.client.1.smithi155.stdout:v0.31 2024-03-28T19:28:13.503 INFO:tasks.workunit.client.1.smithi155.stderr:+ ver=v0.31 2024-03-28T19:28:13.503 INFO:tasks.workunit.client.1.smithi155.stderr:+ echo v0.31 2024-03-28T19:28:13.503 INFO:tasks.workunit.client.1.smithi155.stderr:+ git reset --hard v0.31 2024-03-28T19:28:14.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:14.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.439+0000 7fd5f2c00640 1 Processor -- start 2024-03-28T19:28:14.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.439+0000 7fd5f2c00640 1 -- start start 2024-03-28T19:28:14.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5f2c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5ec0704f0 0x7fd5ec0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5f2c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd5ec153520 0x7fd5ec153930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5f2c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ec0c9360 con 0x7fd5ec14d6b0 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5f2c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ec0c9640 con 0x7fd5ec0704f0 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5f2c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ec0c4c00 con 0x7fd5ec153520 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5eb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5ec0704f0 0x7fd5ec0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5eb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5ec0704f0 0x7fd5ec0c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46792/0 (socket says 172.21.15.27:46792) 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5eb400640 1 -- 172.21.15.27:0/2420955577 learned_addr learned my addr 172.21.15.27:0/2420955577 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.441+0000 7fd5f0c00640 1 --2- 172.21.15.27:0/2420955577 >> v2:172.21.15.155:3300/0 conn(0x7fd5ec153520 0x7fd5ec153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.442+0000 7fd5eb400640 1 -- 172.21.15.27:0/2420955577 >> v2:172.21.15.155:3300/0 conn(0x7fd5ec153520 msgr2=0x7fd5ec153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.442+0000 7fd5eb400640 1 --2- 172.21.15.27:0/2420955577 >> v2:172.21.15.155:3300/0 conn(0x7fd5ec153520 0x7fd5ec153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.442+0000 7fd5eb400640 1 -- 172.21.15.27:0/2420955577 >> v1:172.21.15.27:6789/0 conn(0x7fd5ec14d6b0 legacy=0x7fd5ec0c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.442+0000 7fd5eb400640 1 -- 172.21.15.27:0/2420955577 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0c4ee0 con 0x7fd5ec0704f0 2024-03-28T19:28:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.443+0000 7fd5eb400640 1 --2- 172.21.15.27:0/2420955577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec0704f0 0x7fd5ec0c8d50 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec157c30 tx=0x7fd5d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fe18ef6436cd66 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.443+0000 7fd5eaa00640 1 -- 172.21.15.27:0/2420955577 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d402e070 con 0x7fd5ec0704f0 2024-03-28T19:28:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.443+0000 7fd5eaa00640 1 -- 172.21.15.27:0/2420955577 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d4037070 con 0x7fd5ec0704f0 2024-03-28T19:28:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.443+0000 7fd5eaa00640 1 -- 172.21.15.27:0/2420955577 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4031a00 con 0x7fd5ec0704f0 2024-03-28T19:28:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.444+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2420955577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec0704f0 msgr2=0x7fd5ec0c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.444+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2420955577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec0704f0 0x7fd5ec0c8d50 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec157c30 tx=0x7fd5d402b880 comp rx=0 tx=0).stop 2024-03-28T19:28:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.444+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2420955577 shutdown_connections 2024-03-28T19:28:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.444+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2420955577 >> v2:172.21.15.155:3300/0 conn(0x7fd5ec153520 0x7fd5ec153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.444+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2420955577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec0704f0 0x7fd5ec0c8d50 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.444+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2420955577 >> 172.21.15.27:0/2420955577 conn(0x7fd5ec066dd0 msgr2=0x7fd5ec0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:14.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.444+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2420955577 shutdown_connections 2024-03-28T19:28:14.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.444+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2420955577 wait complete. 2024-03-28T19:28:14.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.446+0000 7fd5f2c00640 1 Processor -- start 2024-03-28T19:28:14.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.446+0000 7fd5f2c00640 1 -- start start 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.446+0000 7fd5f2c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5ec0704f0 0x7fd5ec1523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f2c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5ec14d6b0 0x7fd5ec1ef190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f2c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec153520 0x7fd5ec1544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f2c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0c89b0 con 0x7fd5ec153520 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f2c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0c8830 con 0x7fd5ec14d6b0 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f2c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec153e70 con 0x7fd5ec0704f0 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec153520 0x7fd5ec1544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec153520 0x7fd5ec1544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46800/0 (socket says 172.21.15.27:46800) 2024-03-28T19:28:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f0c00640 1 -- 172.21.15.27:0/2674294987 learned_addr learned my addr 172.21.15.27:0/2674294987 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5eb400640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5ec14d6b0 0x7fd5ec1ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f0c00640 1 -- 172.21.15.27:0/2674294987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5ec0704f0 msgr2=0x7fd5ec1523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f0c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5ec0704f0 0x7fd5ec1523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f0c00640 1 -- 172.21.15.27:0/2674294987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5ec14d6b0 msgr2=0x7fd5ec1ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f0c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5ec14d6b0 0x7fd5ec1ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5f0c00640 1 -- 172.21.15.27:0/2674294987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0a9fe0 con 0x7fd5ec153520 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.447+0000 7fd5eb400640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5ec14d6b0 0x7fd5ec1ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.448+0000 7fd5f0c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec153520 0x7fd5ec1544d0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00d4c0 tx=0x7fd5dc00d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.448+0000 7fd5e3e00640 1 -- 172.21.15.27:0/2674294987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0125f0 con 0x7fd5ec153520 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.448+0000 7fd5e3e00640 1 -- 172.21.15.27:0/2674294987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc019070 con 0x7fd5ec153520 2024-03-28T19:28:14.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.448+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec1ef6d0 con 0x7fd5ec153520 2024-03-28T19:28:14.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.448+0000 7fd5e3e00640 1 -- 172.21.15.27:0/2674294987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0129f0 con 0x7fd5ec153520 2024-03-28T19:28:14.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.449+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec14dca0 con 0x7fd5ec153520 2024-03-28T19:28:14.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.450+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec068180 con 0x7fd5ec153520 2024-03-28T19:28:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.451+0000 7fd5e3e00640 1 -- 172.21.15.27:0/2674294987 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd5dc012b90 con 0x7fd5ec153520 2024-03-28T19:28:14.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.452+0000 7fd5e3e00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd5c806bbd0 0x7fd5c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.452+0000 7fd5e3e00640 1 -- 172.21.15.27:0/2674294987 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7fd5dc08bff0 con 0x7fd5ec153520 2024-03-28T19:28:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.453+0000 7fd5ebe00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd5c806bbd0 0x7fd5c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.453+0000 7fd5ebe00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd5c806bbd0 0x7fd5c806e090 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40012e0 tx=0x7fd5e4007d50 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:14.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.461+0000 7fd5e3e00640 1 -- 172.21.15.27:0/2674294987 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd5ec068180 con 0x7fd5ec153520 2024-03-28T19:28:14.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.728+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5ec14da90 con 0x7fd5ec153520 2024-03-28T19:28:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.729+0000 7fd5e3e00640 1 -- 172.21.15.27:0/2674294987 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fd5dc059870 con 0x7fd5ec153520 2024-03-28T19:28:14.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:14.728 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:10.537892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 241 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd5c806bbd0 msgr2=0x7fd5c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd5c806bbd0 0x7fd5c806e090 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40012e0 tx=0x7fd5e4007d50 comp rx=0 tx=0).stop 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec153520 msgr2=0x7fd5ec1544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec153520 0x7fd5ec1544d0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00d4c0 tx=0x7fd5dc00d980 comp rx=0 tx=0).stop 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 shutdown_connections 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd5c806bbd0 0x7fd5c806e090 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5ec153520 0x7fd5ec1544d0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5ec14d6b0 0x7fd5ec1ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 --2- 172.21.15.27:0/2674294987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5ec0704f0 0x7fd5ec1523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 >> 172.21.15.27:0/2674294987 conn(0x7fd5ec066dd0 msgr2=0x7fd5ec0c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 shutdown_connections 2024-03-28T19:28:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:14.731+0000 7fd5f2c00640 1 -- 172.21.15.27:0/2674294987 wait complete. 2024-03-28T19:28:14.748 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:28:16.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:16.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.979+0000 7f7221e00640 1 Processor -- start 2024-03-28T19:28:16.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.979+0000 7f7221e00640 1 -- start start 2024-03-28T19:28:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.980+0000 7f7221e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f721c153bd0 0x7f721c0c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.980+0000 7f7221e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c154020 con 0x7f721c0a9cb0 2024-03-28T19:28:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.980+0000 7f7221e00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721c0c3d90 con 0x7f721c150c70 2024-03-28T19:28:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.980+0000 7f7221e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721c0ab4e0 con 0x7f721c153bd0 2024-03-28T19:28:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.981+0000 7f721b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f721c153bd0 0x7f721c0c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.981+0000 7f7213e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f721c0a9cb0 0x7f721c1534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36930/0 (socket says 172.21.15.27:36930) 2024-03-28T19:28:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.981+0000 7f721b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f721c153bd0 0x7f721c0c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60878/0 (socket says 172.21.15.27:60878) 2024-03-28T19:28:16.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.981+0000 7f7213e00640 1 -- 172.21.15.27:0/384485786 learned_addr learned my addr 172.21.15.27:0/384485786 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:16.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.981+0000 7f721b400640 1 -- 172.21.15.27:0/384485786 >> v1:172.21.15.155:6789/0 conn(0x7f721c150c70 legacy=0x7f721c0c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:16.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.981+0000 7f721b400640 1 -- 172.21.15.27:0/384485786 >> v1:172.21.15.27:6789/0 conn(0x7f721c0a9cb0 legacy=0x7f721c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:16.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.981+0000 7f721b400640 1 -- 172.21.15.27:0/384485786 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c0ab710 con 0x7f721c153bd0 2024-03-28T19:28:16.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.981+0000 7f721a000640 1 -- 172.21.15.27:0/384485786 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556802045 0 0) 0x7f721c154020 con 0x7f721c0a9cb0 2024-03-28T19:28:16.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.983+0000 7f721b400640 1 --2- 172.21.15.27:0/384485786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c153bd0 0x7f721c0c3690 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f7214009d90 tx=0x7f721402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdf22f1fd63b742 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:16.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.983+0000 7f721a000640 1 -- 172.21.15.27:0/384485786 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214039070 con 0x7f721c153bd0 2024-03-28T19:28:16.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.983+0000 7f721a000640 1 -- 172.21.15.27:0/384485786 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721402cc60 con 0x7f721c153bd0 2024-03-28T19:28:16.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.983+0000 7f721a000640 1 -- 172.21.15.27:0/384485786 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214034660 con 0x7f721c153bd0 2024-03-28T19:28:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.983+0000 7f7221e00640 1 -- 172.21.15.27:0/384485786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c153bd0 msgr2=0x7f721c0c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.983+0000 7f7221e00640 1 --2- 172.21.15.27:0/384485786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c153bd0 0x7f721c0c3690 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f7214009d90 tx=0x7f721402beb0 comp rx=0 tx=0).stop 2024-03-28T19:28:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.984+0000 7f7221e00640 1 -- 172.21.15.27:0/384485786 shutdown_connections 2024-03-28T19:28:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.984+0000 7f7221e00640 1 --2- 172.21.15.27:0/384485786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c153bd0 0x7f721c0c3690 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.984+0000 7f7221e00640 1 -- 172.21.15.27:0/384485786 >> 172.21.15.27:0/384485786 conn(0x7f721c06b9d0 msgr2=0x7f721c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.984+0000 7f7221e00640 1 -- 172.21.15.27:0/384485786 shutdown_connections 2024-03-28T19:28:16.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.984+0000 7f7221e00640 1 -- 172.21.15.27:0/384485786 wait complete. 2024-03-28T19:28:16.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.985+0000 7f7221e00640 1 Processor -- start 2024-03-28T19:28:16.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.985+0000 7f7221e00640 1 -- start start 2024-03-28T19:28:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.986+0000 7f7221e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f721c0a9cb0 0x7f721c1e82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.986+0000 7f7221e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c150c70 0x7f721c06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.986+0000 7f7221e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f721c153bd0 0x7f721c1eac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.986+0000 7f7221e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0abcd0 con 0x7f721c150c70 2024-03-28T19:28:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.986+0000 7f7221e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0ac1f0 con 0x7f721c0a9cb0 2024-03-28T19:28:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.986+0000 7f7221e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0c1810 con 0x7f721c153bd0 2024-03-28T19:28:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c150c70 0x7f721c06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c150c70 0x7f721c06dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60886/0 (socket says 172.21.15.27:60886) 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 -- 172.21.15.27:0/927528225 learned_addr learned my addr 172.21.15.27:0/927528225 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f721aa00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f721c0a9cb0 0x7f721c1e82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 -- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f721c153bd0 msgr2=0x7f721c1eac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f721b400640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f721c153bd0 0x7f721c1eac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f721c153bd0 0x7f721c1eac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 -- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f721c0a9cb0 msgr2=0x7f721c1e82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f721c0a9cb0 0x7f721c1e82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 -- 172.21.15.27:0/927528225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c1499d0 con 0x7f721c150c70 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f721aa00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f721c0a9cb0 0x7f721c1e82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.987+0000 7f7213e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c150c70 0x7f721c06dd90 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f720400c4a0 tx=0x7f720400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.988+0000 7f7213400640 1 -- 172.21.15.27:0/927528225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7204018070 con 0x7f721c150c70 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.988+0000 7f7213400640 1 -- 172.21.15.27:0/927528225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72040074e0 con 0x7f721c150c70 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.988+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c1ee230 con 0x7f721c150c70 2024-03-28T19:28:16.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.988+0000 7f7213400640 1 -- 172.21.15.27:0/927528225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7204004a20 con 0x7f721c150c70 2024-03-28T19:28:16.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.988+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c1ee7f0 con 0x7f721c150c70 2024-03-28T19:28:16.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.989+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c04f400 con 0x7f721c150c70 2024-03-28T19:28:16.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.990+0000 7f7213400640 1 -- 172.21.15.27:0/927528225 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f720401b070 con 0x7f721c150c70 2024-03-28T19:28:16.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.991+0000 7f7213400640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71f806bbd0 0x7f71f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:16.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.991+0000 7f7213400640 1 -- 172.21.15.27:0/927528225 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f720408afe0 con 0x7f721c150c70 2024-03-28T19:28:16.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.991+0000 7f721aa00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71f806bbd0 0x7f71f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:16.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.992+0000 7f721aa00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71f806bbd0 0x7f71f806e090 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f720c000f60 tx=0x7f720c009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:16.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:16.996+0000 7f7213400640 1 -- 172.21.15.27:0/927528225 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7204058860 con 0x7f721c150c70 2024-03-28T19:28:17.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.240+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f721c0c5f00 con 0x7f721c150c70 2024-03-28T19:28:17.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.241+0000 7f7213400640 1 -- 172.21.15.27:0/927528225 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+5463 (secure 0 0 0) 0x7f7204058160 con 0x7f721c150c70 2024-03-28T19:28:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:17.242 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:16.687563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:17.242 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 242 2024-03-28T19:28:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71f806bbd0 msgr2=0x7f71f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71f806bbd0 0x7f71f806e090 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f720c000f60 tx=0x7f720c009040 comp rx=0 tx=0).stop 2024-03-28T19:28:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c150c70 msgr2=0x7f721c06dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c150c70 0x7f721c06dd90 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f720400c4a0 tx=0x7f720400c960 comp rx=0 tx=0).stop 2024-03-28T19:28:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 shutdown_connections 2024-03-28T19:28:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f71f806bbd0 0x7f71f806e090 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f721c153bd0 0x7f721c1eac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f721c150c70 0x7f721c06dd90 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 --2- 172.21.15.27:0/927528225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f721c0a9cb0 0x7f721c1e82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 >> 172.21.15.27:0/927528225 conn(0x7f721c06b9d0 msgr2=0x7f721c0c2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 shutdown_connections 2024-03-28T19:28:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:17.244+0000 7f7221e00640 1 -- 172.21.15.27:0/927528225 wait complete. 2024-03-28T19:28:17.261 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:28:18.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:28:18.178 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:28:19.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.486+0000 7fd138e00640 1 Processor -- start 2024-03-28T19:28:19.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.486+0000 7fd138e00640 1 -- start start 2024-03-28T19:28:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd138e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd134150c70 0x7fd1340a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd138e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1341534c0 0x7fd134153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd138e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1340c3af0 con 0x7fd1340c3690 2024-03-28T19:28:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd138e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1340ab4e0 con 0x7fd134150c70 2024-03-28T19:28:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd138e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340ac1f0 con 0x7fd1341534c0 2024-03-28T19:28:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd131e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd134150c70 0x7fd1340a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd129400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd1340c3690 0x7fd1340c6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36940/0 (socket says 172.21.15.27:36940) 2024-03-28T19:28:19.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd129400640 1 -- 172.21.15.27:0/223588502 learned_addr learned my addr 172.21.15.27:0/223588502 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd132800640 1 --2- 172.21.15.27:0/223588502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1341534c0 0x7fd134153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd131e00640 1 -- 172.21.15.27:0/223588502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1341534c0 msgr2=0x7fd134153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.488+0000 7fd131e00640 1 --2- 172.21.15.27:0/223588502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1341534c0 0x7fd134153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.489+0000 7fd131e00640 1 -- 172.21.15.27:0/223588502 >> v1:172.21.15.27:6789/0 conn(0x7fd1340c3690 legacy=0x7fd1340c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.489+0000 7fd131e00640 1 -- 172.21.15.27:0/223588502 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1340c3d90 con 0x7fd134150c70 2024-03-28T19:28:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.489+0000 7fd131400640 1 -- 172.21.15.27:0/223588502 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169568242 0 0) 0x7fd1340c3af0 con 0x7fd1340c3690 2024-03-28T19:28:19.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.489+0000 7fd132800640 1 --2- 172.21.15.27:0/223588502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1341534c0 0x7fd134153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:28:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.490+0000 7fd131e00640 1 --2- 172.21.15.27:0/223588502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd134150c70 0x7fd1340a9cb0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fd1240097a0 tx=0x7fd12402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582014a1e3d63dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.490+0000 7fd131400640 1 -- 172.21.15.27:0/223588502 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124031780 con 0x7fd134150c70 2024-03-28T19:28:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.490+0000 7fd131400640 1 -- 172.21.15.27:0/223588502 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd124037070 con 0x7fd134150c70 2024-03-28T19:28:19.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.490+0000 7fd131400640 1 -- 172.21.15.27:0/223588502 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124031b40 con 0x7fd134150c70 2024-03-28T19:28:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.491+0000 7fd138e00640 1 -- 172.21.15.27:0/223588502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd134150c70 msgr2=0x7fd1340a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.491+0000 7fd138e00640 1 --2- 172.21.15.27:0/223588502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd134150c70 0x7fd1340a9cb0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fd1240097a0 tx=0x7fd12402b820 comp rx=0 tx=0).stop 2024-03-28T19:28:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.491+0000 7fd138e00640 1 -- 172.21.15.27:0/223588502 shutdown_connections 2024-03-28T19:28:19.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.491+0000 7fd138e00640 1 --2- 172.21.15.27:0/223588502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1341534c0 0x7fd134153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.491+0000 7fd138e00640 1 --2- 172.21.15.27:0/223588502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd134150c70 0x7fd1340a9cb0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.491+0000 7fd138e00640 1 -- 172.21.15.27:0/223588502 >> 172.21.15.27:0/223588502 conn(0x7fd13406b9d0 msgr2=0x7fd1340c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.491+0000 7fd138e00640 1 -- 172.21.15.27:0/223588502 shutdown_connections 2024-03-28T19:28:19.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.491+0000 7fd138e00640 1 -- 172.21.15.27:0/223588502 wait complete. 2024-03-28T19:28:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.493+0000 7fd138e00640 1 Processor -- start 2024-03-28T19:28:19.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.493+0000 7fd138e00640 1 -- start start 2024-03-28T19:28:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd138e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1340c3690 0x7fd1341eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd138e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd134150c70 0x7fd1340c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd138e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1341534c0 0x7fd134144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd138e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340abcd0 con 0x7fd1341534c0 2024-03-28T19:28:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd138e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340ab9f0 con 0x7fd134150c70 2024-03-28T19:28:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd138e00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340c1810 con 0x7fd1340c3690 2024-03-28T19:28:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd132800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1341534c0 0x7fd134144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:19.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd131e00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1340c3690 0x7fd1341eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd132800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1341534c0 0x7fd134144c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60908/0 (socket says 172.21.15.27:60908) 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd132800640 1 -- 172.21.15.27:0/1152686784 learned_addr learned my addr 172.21.15.27:0/1152686784 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.494+0000 7fd129400640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd134150c70 0x7fd1340c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd132800640 1 -- 172.21.15.27:0/1152686784 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1340c3690 msgr2=0x7fd1341eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd132800640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1340c3690 0x7fd1341eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd132800640 1 -- 172.21.15.27:0/1152686784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd134150c70 msgr2=0x7fd1340c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd132800640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd134150c70 0x7fd1340c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd132800640 1 -- 172.21.15.27:0/1152686784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1341573b0 con 0x7fd1341534c0 2024-03-28T19:28:19.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd132800640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1341534c0 0x7fd134144c90 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fd12c007ea0 tx=0x7fd12c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd12aa00640 1 -- 172.21.15.27:0/1152686784 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c01a070 con 0x7fd1341534c0 2024-03-28T19:28:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd12aa00640 1 -- 172.21.15.27:0/1152686784 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12c003bb0 con 0x7fd1341534c0 2024-03-28T19:28:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd12aa00640 1 -- 172.21.15.27:0/1152686784 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c00de70 con 0x7fd1341534c0 2024-03-28T19:28:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.495+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd134157690 con 0x7fd1341534c0 2024-03-28T19:28:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.496+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1340c07e0 con 0x7fd1341534c0 2024-03-28T19:28:19.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.497+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1341510f0 con 0x7fd1341534c0 2024-03-28T19:28:19.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.498+0000 7fd12aa00640 1 -- 172.21.15.27:0/1152686784 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd12c01d070 con 0x7fd1341534c0 2024-03-28T19:28:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.499+0000 7fd12aa00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd11006bc00 0x7fd11006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.499+0000 7fd12aa00640 1 -- 172.21.15.27:0/1152686784 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7fd12c08be70 con 0x7fd1341534c0 2024-03-28T19:28:19.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.499+0000 7fd131e00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd11006bc00 0x7fd11006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:19.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.500+0000 7fd131e00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd11006bc00 0x7fd11006e0c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd124036000 tx=0x7fd12403d040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:19.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.504+0000 7fd12aa00640 1 -- 172.21.15.27:0/1152686784 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd1341510f0 con 0x7fd1341534c0 2024-03-28T19:28:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.778+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd13404f400 con 0x7fd1341534c0 2024-03-28T19:28:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.779+0000 7fd12aa00640 1 -- 172.21.15.27:0/1152686784 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+5466 (secure 0 0 0) 0x7fd12c010070 con 0x7fd1341534c0 2024-03-28T19:28:19.779 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:18.281595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 244 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.782+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd11006bc00 msgr2=0x7fd11006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.782+0000 7fd138e00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd11006bc00 0x7fd11006e0c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd124036000 tx=0x7fd12403d040 comp rx=0 tx=0).stop 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.782+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1341534c0 msgr2=0x7fd134144c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1341534c0 0x7fd134144c90 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fd12c007ea0 tx=0x7fd12c007f90 comp rx=0 tx=0).stop 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 shutdown_connections 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd11006bc00 0x7fd11006e0c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1341534c0 0x7fd134144c90 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd134150c70 0x7fd1340c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 --2- 172.21.15.27:0/1152686784 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd1340c3690 0x7fd1341eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 >> 172.21.15.27:0/1152686784 conn(0x7fd13406b9d0 msgr2=0x7fd134151740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 shutdown_connections 2024-03-28T19:28:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:19.783+0000 7fd138e00640 1 -- 172.21.15.27:0/1152686784 wait complete. 2024-03-28T19:28:19.801 INFO:tasks.mds_thrash.fs.[a]:no change 2024-03-28T19:28:20.977 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-03-28T19:28:20.978+0000 7f3fc5600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T19:28:21.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:22.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.041+0000 7f6850c00640 1 Processor -- start 2024-03-28T19:28:22.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.042+0000 7f6850c00640 1 -- start start 2024-03-28T19:28:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.044+0000 7f6850c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f684c152bd0 0x7f684c06eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.044+0000 7f6850c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 0x7f684c070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.044+0000 7f6850c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684c0c52e0 con 0x7f684c0c4dc0 2024-03-28T19:28:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.044+0000 7f6850c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684c1557e0 con 0x7f684c152bd0 2024-03-28T19:28:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.044+0000 7f6850c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c155ac0 con 0x7f684c06f1e0 2024-03-28T19:28:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.044+0000 7f6849e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f684c152bd0 0x7f684c06eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.044+0000 7f6849e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f684c152bd0 0x7f684c06eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60918/0 (socket says 172.21.15.27:60918) 2024-03-28T19:28:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.044+0000 7f6849e00640 1 -- 172.21.15.27:0/2790465172 learned_addr learned my addr 172.21.15.27:0/2790465172 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:22.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.045+0000 7f6849e00640 1 -- 172.21.15.27:0/2790465172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 msgr2=0x7f684c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:22.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.045+0000 7f6849e00640 1 --2- 172.21.15.27:0/2790465172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 0x7f684c070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.045+0000 7f6849e00640 1 -- 172.21.15.27:0/2790465172 >> v1:172.21.15.155:6789/0 conn(0x7f684c0c4dc0 legacy=0x7f684c158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:22.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.045+0000 7f6849e00640 1 -- 172.21.15.27:0/2790465172 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c155da0 con 0x7f684c152bd0 2024-03-28T19:28:22.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.046+0000 7f6849e00640 1 --2- 172.21.15.27:0/2790465172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c06eca0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f683c009870 tx=0x7f683c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0db8fdbba184eac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:22.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.046+0000 7f6848a00640 1 -- 172.21.15.27:0/2790465172 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c035020 con 0x7f684c152bd0 2024-03-28T19:28:22.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.046+0000 7f6848a00640 1 -- 172.21.15.27:0/2790465172 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c031640 con 0x7f684c152bd0 2024-03-28T19:28:22.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.046+0000 7f6848a00640 1 -- 172.21.15.27:0/2790465172 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c02d040 con 0x7f684c152bd0 2024-03-28T19:28:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.047+0000 7f6850c00640 1 -- 172.21.15.27:0/2790465172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 msgr2=0x7f684c06eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:22.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.047+0000 7f6850c00640 1 --2- 172.21.15.27:0/2790465172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c06eca0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f683c009870 tx=0x7f683c02b8f0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.047+0000 7f6850c00640 1 -- 172.21.15.27:0/2790465172 shutdown_connections 2024-03-28T19:28:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.047+0000 7f6850c00640 1 --2- 172.21.15.27:0/2790465172 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 0x7f684c070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.047+0000 7f6850c00640 1 --2- 172.21.15.27:0/2790465172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c06eca0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.047+0000 7f6850c00640 1 -- 172.21.15.27:0/2790465172 >> 172.21.15.27:0/2790465172 conn(0x7f684c066dd0 msgr2=0x7f684c0671b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.047+0000 7f6850c00640 1 -- 172.21.15.27:0/2790465172 shutdown_connections 2024-03-28T19:28:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.048+0000 7f6850c00640 1 -- 172.21.15.27:0/2790465172 wait complete. 2024-03-28T19:28:22.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.050+0000 7f6850c00640 1 Processor -- start 2024-03-28T19:28:22.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.050+0000 7f6850c00640 1 -- start start 2024-03-28T19:28:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.051+0000 7f6850c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 0x7f684c14e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.051+0000 7f6850c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f684c0c4dc0 0x7f684c14e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.051+0000 7f6850c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c1f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.051+0000 7f6850c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0c6df0 con 0x7f684c152bd0 2024-03-28T19:28:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.051+0000 7f6850c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0c6c70 con 0x7f684c0c4dc0 2024-03-28T19:28:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.051+0000 7f6850c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c070a80 con 0x7f684c06f1e0 2024-03-28T19:28:22.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f684a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c1f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:22.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f684a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c1f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60928/0 (socket says 172.21.15.27:60928) 2024-03-28T19:28:22.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f684a800640 1 -- 172.21.15.27:0/1829372961 learned_addr learned my addr 172.21.15.27:0/1829372961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:22.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f6849400640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f684c0c4dc0 0x7f684c14e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:22.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f6849e00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 0x7f684c14e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f684a800640 1 -- 172.21.15.27:0/1829372961 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 msgr2=0x7f684c14e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f684a800640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 0x7f684c14e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f684a800640 1 -- 172.21.15.27:0/1829372961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f684c0c4dc0 msgr2=0x7f684c14e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f684a800640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f684c0c4dc0 0x7f684c14e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.052+0000 7f684a800640 1 -- 172.21.15.27:0/1829372961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c156820 con 0x7f684c152bd0 2024-03-28T19:28:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.053+0000 7f684a800640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c1f2f20 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f684400c780 tx=0x7f684400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.053+0000 7f6842000640 1 -- 172.21.15.27:0/1829372961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844017070 con 0x7f684c152bd0 2024-03-28T19:28:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.053+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684c157680 con 0x7f684c152bd0 2024-03-28T19:28:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.053+0000 7f6842000640 1 -- 172.21.15.27:0/1829372961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6844003da0 con 0x7f684c152bd0 2024-03-28T19:28:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.053+0000 7f6842000640 1 -- 172.21.15.27:0/1829372961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68440052b0 con 0x7f684c152bd0 2024-03-28T19:28:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.053+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c157960 con 0x7f684c152bd0 2024-03-28T19:28:22.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.055+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684c04f400 con 0x7f684c152bd0 2024-03-28T19:28:22.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.056+0000 7f6842000640 1 -- 172.21.15.27:0/1829372961 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f684401a070 con 0x7f684c152bd0 2024-03-28T19:28:22.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.057+0000 7f6842000640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f682c06bc00 0x7f682c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:22.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.058+0000 7f6842000640 1 -- 172.21.15.27:0/1829372961 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f684408b250 con 0x7f684c152bd0 2024-03-28T19:28:22.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.058+0000 7f6849e00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f682c06bc00 0x7f682c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.059+0000 7f6849e00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f682c06bc00 0x7f682c06e0c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f683c036000 tx=0x7f683c03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.063+0000 7f6842000640 1 -- 172.21.15.27:0/1829372961 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6844058460 con 0x7f684c152bd0 2024-03-28T19:28:22.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.432+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f684c153260 con 0x7f684c152bd0 2024-03-28T19:28:22.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.433+0000 7f6842000640 1 -- 172.21.15.27:0/1829372961 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f684405c500 con 0x7f684c152bd0 2024-03-28T19:28:22.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:22.433 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:22.433 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 245 2024-03-28T19:28:22.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f682c06bc00 msgr2=0x7f682c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f682c06bc00 0x7f682c06e0c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f683c036000 tx=0x7f683c03c040 comp rx=0 tx=0).stop 2024-03-28T19:28:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 msgr2=0x7f684c1f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c1f2f20 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f684400c780 tx=0x7f684400cc40 comp rx=0 tx=0).stop 2024-03-28T19:28:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 shutdown_connections 2024-03-28T19:28:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f682c06bc00 0x7f682c06e0c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f684c152bd0 0x7f684c1f2f20 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f684c0c4dc0 0x7f684c14e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 --2- 172.21.15.27:0/1829372961 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f684c06f1e0 0x7f684c14e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 >> 172.21.15.27:0/1829372961 conn(0x7f684c066dd0 msgr2=0x7f684c1509f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 shutdown_connections 2024-03-28T19:28:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:22.436+0000 7f6850c00640 1 -- 172.21.15.27:0/1829372961 wait complete. 2024-03-28T19:28:22.458 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6740 2024-03-28T19:28:22.458 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.b 2024-03-28T19:28:23.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.252+0000 7fcdff200640 1 Processor -- start 2024-03-28T19:28:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.252+0000 7fcdff200640 1 -- start start 2024-03-28T19:28:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.253+0000 7fcdff200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce000c58c0 0x7fce000c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.253+0000 7fcdff200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce00158bd0 con 0x7fce000c8ed0 2024-03-28T19:28:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.253+0000 7fcdff200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce00158eb0 con 0x7fce000be5d0 2024-03-28T19:28:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.253+0000 7fcdff200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce00159190 con 0x7fce000c58c0 2024-03-28T19:28:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.254+0000 7fcdfd400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fce000c8ed0 0x7fce001555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36942/0 (socket says 172.21.15.27:36942) 2024-03-28T19:28:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.254+0000 7fcdfd400640 1 -- 172.21.15.27:0/1274563936 learned_addr learned my addr 172.21.15.27:0/1274563936 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.254+0000 7fcdfde00640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce000c58c0 0x7fce000c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.254+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167968118 0 0) 0x7fce00158bd0 con 0x7fce000c8ed0 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde4003520 con 0x7fce000c8ed0 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175842667 0 0) 0x7fce00158eb0 con 0x7fce000be5d0 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce00158bd0 con 0x7fce000be5d0 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3447615560 0 0) 0x7fcde4003520 con 0x7fce000c8ed0 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce000c58c0 msgr2=0x7fce000c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce000c58c0 0x7fce000c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 >> v1:172.21.15.155:6789/0 conn(0x7fce000be5d0 legacy=0x7fce00155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce00159470 con 0x7fce000c8ed0 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7fcdec001670 con 0x7fce000c8ed0 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 >> v1:172.21.15.27:6789/0 conn(0x7fce000c8ed0 legacy=0x7fce001555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce000c58c0 0x7fcde4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde40040d0 0x7fcde4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fcde40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00159190 con 0x7fcde40040d0 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde400bb50 con 0x7fcde4004d40 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.255+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde400bd20 con 0x7fce000c58c0 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.256+0000 7fcdfd400640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde40040d0 0x7fcde4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.256+0000 7fcdfde00640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce000c58c0 0x7fcde4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.256+0000 7fcdfe800640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fcde40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.256+0000 7fcdfe800640 1 -- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce000c58c0 msgr2=0x7fcde4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.256+0000 7fcdfe800640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce000c58c0 0x7fcde4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.256+0000 7fcdfe800640 1 -- 172.21.15.27:0/1274563936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde40040d0 msgr2=0x7fcde4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.256+0000 7fcdfe800640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde40040d0 0x7fcde4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.256+0000 7fcdfe800640 1 -- 172.21.15.27:0/1274563936 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce00159470 con 0x7fcde4004d40 2024-03-28T19:28:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.257+0000 7fcdfd400640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde40040d0 0x7fcde4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:28:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.258+0000 7fcdfe800640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fcde40051c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8001fa0 tx=0x7fcde4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a48815ef32925a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.259+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8001bf0 con 0x7fcde4004d40 2024-03-28T19:28:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.259+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf8003b70 con 0x7fcde4004d40 2024-03-28T19:28:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.259+0000 7fcdfca00640 1 -- 172.21.15.27:0/1274563936 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf80050d0 con 0x7fcde4004d40 2024-03-28T19:28:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.259+0000 7fcdff200640 1 -- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 msgr2=0x7fcde40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.259+0000 7fcdff200640 1 --2- 172.21.15.27:0/1274563936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fcde40051c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8001fa0 tx=0x7fcde4005830 comp rx=0 tx=0).stop 2024-03-28T19:28:23.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.259+0000 7fcdfe800640 1 -- 172.21.15.27:0/1274563936 reap_dead start 2024-03-28T19:28:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.260+0000 7fcdff200640 1 -- 172.21.15.27:0/1274563936 shutdown_connections 2024-03-28T19:28:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.260+0000 7fcdff200640 1 -- 172.21.15.27:0/1274563936 >> 172.21.15.27:0/1274563936 conn(0x7fce0006b6f0 msgr2=0x7fce000acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.260+0000 7fcdff200640 1 -- 172.21.15.27:0/1274563936 shutdown_connections 2024-03-28T19:28:23.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.260+0000 7fcdff200640 1 -- 172.21.15.27:0/1274563936 wait complete. 2024-03-28T19:28:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.262+0000 7fcdff200640 1 Processor -- start 2024-03-28T19:28:23.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.263+0000 7fcdff200640 1 -- start start 2024-03-28T19:28:23.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.263+0000 7fcdff200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fce000ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.263+0000 7fcdff200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000c3430 0x7fce000ac810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.263+0000 7fcdff200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce001523c0 0x7fce000c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.264+0000 7fcdff200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b09a0 con 0x7fce000c3430 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.264+0000 7fcdff200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b0820 con 0x7fcde4004d40 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.264+0000 7fcdff200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce000ad7d0 con 0x7fce001523c0 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.264+0000 7fcdfde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fce000ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.264+0000 7fcdfde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fce000ac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:53806/0 (socket says 172.21.15.27:53806) 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.264+0000 7fcdfde00640 1 -- 172.21.15.27:0/2354484539 learned_addr learned my addr 172.21.15.27:0/2354484539 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:23.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.264+0000 7fcdfd400640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000c3430 0x7fce000ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.264+0000 7fcdfe800640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce001523c0 0x7fce000c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.265+0000 7fcdfde00640 1 -- 172.21.15.27:0/2354484539 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce001523c0 msgr2=0x7fce000c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.265+0000 7fcdfde00640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce001523c0 0x7fce000c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.265+0000 7fcdfde00640 1 -- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000c3430 msgr2=0x7fce000ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.265+0000 7fcdfde00640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000c3430 0x7fce000ac810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.265+0000 7fcdfde00640 1 -- 172.21.15.27:0/2354484539 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce000c2010 con 0x7fcde4004d40 2024-03-28T19:28:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.265+0000 7fcdfde00640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fce000ac2d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0011660 tx=0x7fcdf0011a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.266+0000 7fcdf6000640 1 -- 172.21.15.27:0/2354484539 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0015870 con 0x7fcde4004d40 2024-03-28T19:28:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.266+0000 7fcdf6000640 1 -- 172.21.15.27:0/2354484539 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf0015a10 con 0x7fcde4004d40 2024-03-28T19:28:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.266+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce000c22f0 con 0x7fcde4004d40 2024-03-28T19:28:23.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.266+0000 7fcdf6000640 1 -- 172.21.15.27:0/2354484539 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0015dc0 con 0x7fcde4004d40 2024-03-28T19:28:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.266+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce000ab1b0 con 0x7fcde4004d40 2024-03-28T19:28:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.268+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce000c2810 con 0x7fcde4004d40 2024-03-28T19:28:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.270+0000 7fcdf6000640 1 -- 172.21.15.27:0/2354484539 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcdf000f480 con 0x7fcde4004d40 2024-03-28T19:28:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.271+0000 7fcdf6000640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcde006bbd0 0x7fcde006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.271+0000 7fcdf6000640 1 -- 172.21.15.27:0/2354484539 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7fcdf008c570 con 0x7fcde4004d40 2024-03-28T19:28:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.271+0000 7fcdfd400640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcde006bbd0 0x7fcde006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.272+0000 7fcdfd400640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcde006bbd0 0x7fcde006e090 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fcdec000ff0 tx=0x7fcdec001080 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.276+0000 7fcdf6000640 1 -- 172.21.15.27:0/2354484539 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcdf0019030 con 0x7fcde4004d40 2024-03-28T19:28:23.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.622+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce00069640 con 0x7fcde4004d40 2024-03-28T19:28:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.623+0000 7fcdf6000640 1 -- 172.21.15.27:0/2354484539 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5468 (secure 0 0 0) 0x7fcdf0059dd0 con 0x7fcde4004d40 2024-03-28T19:28:23.624 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:23.624 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:23.624 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 245 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcde006bbd0 msgr2=0x7fcde006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcde006bbd0 0x7fcde006e090 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fcdec000ff0 tx=0x7fcdec001080 comp rx=0 tx=0).stop 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 msgr2=0x7fce000ac2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fce000ac2d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0011660 tx=0x7fcdf0011a60 comp rx=0 tx=0).stop 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 shutdown_connections 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fcde006bbd0 0x7fcde006e090 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce001523c0 0x7fce000c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce000c3430 0x7fce000ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 --2- 172.21.15.27:0/2354484539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcde4004d40 0x7fce000ac2d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.626+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 >> 172.21.15.27:0/2354484539 conn(0x7fce0006b6f0 msgr2=0x7fce000ca200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.627+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 shutdown_connections 2024-03-28T19:28:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.627+0000 7fcdff200640 1 -- 172.21.15.27:0/2354484539 wait complete. 2024-03-28T19:28:23.653 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-03-28T19:28:23.654 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-28T19:28:23.654 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:28:23.683 INFO:tasks.ceph.mds.d:Stopped 2024-03-28T19:28:23.683 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:23.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.924+0000 7f0a9ea00640 1 Processor -- start 2024-03-28T19:28:23.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.924+0000 7f0a9ea00640 1 -- start start 2024-03-28T19:28:23.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.925+0000 7f0a9ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a980c8ed0 0x7f0a980c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.925+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a98155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.925+0000 7f0a9ea00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a98156230 con 0x7f0a980c58c0 2024-03-28T19:28:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.925+0000 7f0a9ea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a98158bd0 con 0x7f0a980c8ed0 2024-03-28T19:28:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.925+0000 7f0a9ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98158eb0 con 0x7f0a980be5d0 2024-03-28T19:28:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.925+0000 7f0a9cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a980c8ed0 0x7f0a980c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.926+0000 7f0a9cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a980c8ed0 0x7f0a980c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60936/0 (socket says 172.21.15.27:60936) 2024-03-28T19:28:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.926+0000 7f0a9cc00640 1 -- 172.21.15.27:0/2557115527 learned_addr learned my addr 172.21.15.27:0/2557115527 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:23.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.926+0000 7f0a9e000640 1 --2- 172.21.15.27:0/2557115527 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a98155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.926+0000 7f0a9cc00640 1 -- 172.21.15.27:0/2557115527 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 msgr2=0x7f0a98155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:23.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.926+0000 7f0a9cc00640 1 --2- 172.21.15.27:0/2557115527 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a98155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.926+0000 7f0a9cc00640 1 -- 172.21.15.27:0/2557115527 >> v1:172.21.15.155:6789/0 conn(0x7f0a980c58c0 legacy=0x7f0a981555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:23.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.926+0000 7f0a9cc00640 1 -- 172.21.15.27:0/2557115527 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98159190 con 0x7f0a980c8ed0 2024-03-28T19:28:23.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.927+0000 7f0a9cc00640 1 --2- 172.21.15.27:0/2557115527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980c5ca0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009870 tx=0x7f0a8402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0eafdab23bf3b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:23.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.928+0000 7f0a97e00640 1 -- 172.21.15.27:0/2557115527 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a840317c0 con 0x7f0a980c8ed0 2024-03-28T19:28:23.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.928+0000 7f0a97e00640 1 -- 172.21.15.27:0/2557115527 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a84037070 con 0x7f0a980c8ed0 2024-03-28T19:28:23.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.928+0000 7f0a97e00640 1 -- 172.21.15.27:0/2557115527 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84031b80 con 0x7f0a980c8ed0 2024-03-28T19:28:23.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.929+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2557115527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 msgr2=0x7f0a980c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.929+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2557115527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980c5ca0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009870 tx=0x7f0a8402b860 comp rx=0 tx=0).stop 2024-03-28T19:28:23.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.929+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2557115527 shutdown_connections 2024-03-28T19:28:23.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.929+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2557115527 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a98155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.929+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/2557115527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980c5ca0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.929+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2557115527 >> 172.21.15.27:0/2557115527 conn(0x7f0a9806b6f0 msgr2=0x7f0a980acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:23.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.929+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2557115527 shutdown_connections 2024-03-28T19:28:23.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.929+0000 7f0a9ea00640 1 -- 172.21.15.27:0/2557115527 wait complete. 2024-03-28T19:28:23.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.931+0000 7f0a9ea00640 1 Processor -- start 2024-03-28T19:28:23.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.931+0000 7f0a9ea00640 1 -- start start 2024-03-28T19:28:23.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a980abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980c58c0 0x7f0a981ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980bedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980b0820 con 0x7f0a980c8ed0 2024-03-28T19:28:23.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98159750 con 0x7f0a980c58c0 2024-03-28T19:28:23.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9ea00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980b09a0 con 0x7f0a980be5d0 2024-03-28T19:28:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980bedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980bedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60946/0 (socket says 172.21.15.27:60946) 2024-03-28T19:28:23.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.932+0000 7f0a9e000640 1 -- 172.21.15.27:0/1339846424 learned_addr learned my addr 172.21.15.27:0/1339846424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9cc00640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980c58c0 0x7f0a981ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9d600640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a980abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9e000640 1 -- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 msgr2=0x7f0a980abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9e000640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a980abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9e000640 1 -- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980c58c0 msgr2=0x7f0a981ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9e000640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980c58c0 0x7f0a981ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9e000640 1 -- 172.21.15.27:0/1339846424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a981efb60 con 0x7f0a980c8ed0 2024-03-28T19:28:23.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9d600640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a980abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:28:23.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.933+0000 7f0a9e000640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980bedc0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f0a90007e60 tx=0x7f0a9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:23.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.934+0000 7f0a95600640 1 -- 172.21.15.27:0/1339846424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90014420 con 0x7f0a980c8ed0 2024-03-28T19:28:23.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.934+0000 7f0a95600640 1 -- 172.21.15.27:0/1339846424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a90019070 con 0x7f0a980c8ed0 2024-03-28T19:28:23.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.934+0000 7f0a95600640 1 -- 172.21.15.27:0/1339846424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90014820 con 0x7f0a980c8ed0 2024-03-28T19:28:23.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.934+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a981eb6d0 con 0x7f0a980c8ed0 2024-03-28T19:28:23.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.934+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980ab1f0 con 0x7f0a980c8ed0 2024-03-28T19:28:23.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.936+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a98069640 con 0x7f0a980c8ed0 2024-03-28T19:28:23.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.937+0000 7f0a95600640 1 -- 172.21.15.27:0/1339846424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0a900149c0 con 0x7f0a980c8ed0 2024-03-28T19:28:23.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.938+0000 7f0a95600640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bc00 0x7f0a7806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.938+0000 7f0a95600640 1 -- 172.21.15.27:0/1339846424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f0a9008c5e0 con 0x7f0a980c8ed0 2024-03-28T19:28:23.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.939+0000 7f0a9d600640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bc00 0x7f0a7806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:23.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.940+0000 7f0a9d600640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bc00 0x7f0a7806e0c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c006fd0 tx=0x7f0a8c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:23.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:23.944+0000 7f0a95600640 1 -- 172.21.15.27:0/1339846424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0a981eb6d0 con 0x7f0a980c8ed0 2024-03-28T19:28:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.290+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a980ad5b0 con 0x7f0a980c8ed0 2024-03-28T19:28:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.291+0000 7f0a95600640 1 -- 172.21.15.27:0/1339846424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f0a90059db0 con 0x7f0a980c8ed0 2024-03-28T19:28:24.292 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:24.292 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:24.292 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 245 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bc00 msgr2=0x7f0a7806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bc00 0x7f0a7806e0c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c006fd0 tx=0x7f0a8c008040 comp rx=0 tx=0).stop 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 msgr2=0x7f0a980bedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980bedc0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f0a90007e60 tx=0x7f0a9000e4b0 comp rx=0 tx=0).stop 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 shutdown_connections 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f0a7806bc00 0x7f0a7806e0c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980c8ed0 0x7f0a980bedc0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a980c58c0 0x7f0a981ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/1339846424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a980be5d0 0x7f0a980abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.294+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 >> 172.21.15.27:0/1339846424 conn(0x7f0a9806b6f0 msgr2=0x7f0a980ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.295+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 shutdown_connections 2024-03-28T19:28:24.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:24.295+0000 7f0a9ea00640 1 -- 172.21.15.27:0/1339846424 wait complete. 2024-03-28T19:28:24.316 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:28:26.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.563+0000 7f6bd7000640 1 Processor -- start 2024-03-28T19:28:26.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.563+0000 7f6bd7000640 1 -- start start 2024-03-28T19:28:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.564+0000 7f6bd7000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd0150ae0 0x7f6bd00a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.564+0000 7f6bd7000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd0153a30 0x7f6bd0153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.564+0000 7f6bd7000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0150ec0 con 0x7f6bd00a61a0 2024-03-28T19:28:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.564+0000 7f6bd7000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd006dc70 con 0x7f6bd0153a30 2024-03-28T19:28:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.564+0000 7f6bd7000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0153550 con 0x7f6bd0150ae0 2024-03-28T19:28:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.565+0000 7f6bd6600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd0153a30 0x7f6bd0153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.565+0000 7f6bd5200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6bd00a61a0 0x7f6bd00c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45002/0 (socket says 172.21.15.27:45002) 2024-03-28T19:28:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.565+0000 7f6bd5200640 1 -- 172.21.15.27:0/687723753 learned_addr learned my addr 172.21.15.27:0/687723753 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.565+0000 7f6bd5c00640 1 --2- 172.21.15.27:0/687723753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd0150ae0 0x7f6bd00a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.566+0000 7f6bd6600640 1 -- 172.21.15.27:0/687723753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd0150ae0 msgr2=0x7f6bd00a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.566+0000 7f6bd6600640 1 --2- 172.21.15.27:0/687723753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd0150ae0 0x7f6bd00a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.566+0000 7f6bd6600640 1 -- 172.21.15.27:0/687723753 >> v1:172.21.15.27:6789/0 conn(0x7f6bd00a61a0 legacy=0x7f6bd00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.566+0000 7f6bcfe00640 1 -- 172.21.15.27:0/687723753 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730858313 0 0) 0x7f6bbc001110 con 0x7f6bd00a61a0 2024-03-28T19:28:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.566+0000 7f6bd6600640 1 -- 172.21.15.27:0/687723753 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd006df10 con 0x7f6bd0153a30 2024-03-28T19:28:26.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.566+0000 7f6bd5c00640 1 --2- 172.21.15.27:0/687723753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd0150ae0 0x7f6bd00a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:28:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.567+0000 7f6bd6600640 1 --2- 172.21.15.27:0/687723753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0153a30 0x7f6bd0153ea0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009d90 tx=0x7f6bc802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c248b7e65f233242 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:26.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.567+0000 7f6bcfe00640 1 -- 172.21.15.27:0/687723753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8039070 con 0x7f6bd0153a30 2024-03-28T19:28:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.567+0000 7f6bcfe00640 1 -- 172.21.15.27:0/687723753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc802cbd0 con 0x7f6bd0153a30 2024-03-28T19:28:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.568+0000 7f6bcfe00640 1 -- 172.21.15.27:0/687723753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8034440 con 0x7f6bd0153a30 2024-03-28T19:28:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.568+0000 7f6bd7000640 1 -- 172.21.15.27:0/687723753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0153a30 msgr2=0x7f6bd0153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:26.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.568+0000 7f6bd7000640 1 --2- 172.21.15.27:0/687723753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0153a30 0x7f6bd0153ea0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009d90 tx=0x7f6bc802be20 comp rx=0 tx=0).stop 2024-03-28T19:28:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.569+0000 7f6bd7000640 1 -- 172.21.15.27:0/687723753 shutdown_connections 2024-03-28T19:28:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.569+0000 7f6bd7000640 1 --2- 172.21.15.27:0/687723753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0153a30 0x7f6bd0153ea0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.569+0000 7f6bd7000640 1 --2- 172.21.15.27:0/687723753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd0150ae0 0x7f6bd00a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.569+0000 7f6bd7000640 1 -- 172.21.15.27:0/687723753 >> 172.21.15.27:0/687723753 conn(0x7f6bd006b0c0 msgr2=0x7f6bd006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.569+0000 7f6bd7000640 1 -- 172.21.15.27:0/687723753 shutdown_connections 2024-03-28T19:28:26.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.569+0000 7f6bd7000640 1 -- 172.21.15.27:0/687723753 wait complete. 2024-03-28T19:28:26.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.572+0000 7f6bd7000640 1 Processor -- start 2024-03-28T19:28:26.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.572+0000 7f6bd7000640 1 -- start start 2024-03-28T19:28:26.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd7000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd00a61a0 0x7f6bd014a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:26.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd7000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0150ae0 0x7f6bd0072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:26.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd7000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bd0153a30 0x7f6bd01ddd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:26.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd7000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd006e7b0 con 0x7f6bd0150ae0 2024-03-28T19:28:26.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd7000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd006e4d0 con 0x7f6bd0153a30 2024-03-28T19:28:26.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd7000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00c70b0 con 0x7f6bd00a61a0 2024-03-28T19:28:26.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0150ae0 0x7f6bd0072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:26.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0150ae0 0x7f6bd0072ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47824/0 (socket says 172.21.15.27:47824) 2024-03-28T19:28:26.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd5200640 1 -- 172.21.15.27:0/1137731911 learned_addr learned my addr 172.21.15.27:0/1137731911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:26.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd5c00640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd00a61a0 0x7f6bd014a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:26.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.573+0000 7f6bd6600640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bd0153a30 0x7f6bd01ddd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bd5200640 1 -- 172.21.15.27:0/1137731911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd00a61a0 msgr2=0x7f6bd014a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bd5200640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd00a61a0 0x7f6bd014a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bd5200640 1 -- 172.21.15.27:0/1137731911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bd0153a30 msgr2=0x7f6bd01ddd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bd5200640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bd0153a30 0x7f6bd01ddd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bd5200640 1 -- 172.21.15.27:0/1137731911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd0073620 con 0x7f6bd0150ae0 2024-03-28T19:28:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bd5200640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0150ae0 0x7f6bd0072ab0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00c750 tx=0x7f6bbc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bcd600640 1 -- 172.21.15.27:0/1137731911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc018070 con 0x7f6bd0150ae0 2024-03-28T19:28:26.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bcd600640 1 -- 172.21.15.27:0/1137731911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc0070d0 con 0x7f6bd0150ae0 2024-03-28T19:28:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bcd600640 1 -- 172.21.15.27:0/1137731911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc0074c0 con 0x7f6bd0150ae0 2024-03-28T19:28:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.574+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd0071320 con 0x7f6bd0150ae0 2024-03-28T19:28:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.575+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd00a5e10 con 0x7f6bd0150ae0 2024-03-28T19:28:26.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.576+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd0067dd0 con 0x7f6bd0150ae0 2024-03-28T19:28:26.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.577+0000 7f6bcd600640 1 -- 172.21.15.27:0/1137731911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6bbc0047e0 con 0x7f6bd0150ae0 2024-03-28T19:28:26.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.578+0000 7f6bcd600640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6bb006bbd0 0x7f6bb006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.578+0000 7f6bcd600640 1 -- 172.21.15.27:0/1137731911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f6bbc08b080 con 0x7f6bd0150ae0 2024-03-28T19:28:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.579+0000 7f6bd5c00640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6bb006bbd0 0x7f6bb006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.580+0000 7f6bd5c00640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6bb006bbd0 0x7f6bb006e090 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4006fd0 tx=0x7f6bc4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.584+0000 7f6bcd600640 1 -- 172.21.15.27:0/1137731911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6bd0150ec0 con 0x7f6bd0150ae0 2024-03-28T19:28:26.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.924+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bd01dee50 con 0x7f6bd0150ae0 2024-03-28T19:28:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.925+0000 7f6bcd600640 1 -- 172.21.15.27:0/1137731911 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f6bbc058850 con 0x7f6bd0150ae0 2024-03-28T19:28:26.925 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:26.925 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:26.925 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 245 2024-03-28T19:28:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6bb006bbd0 msgr2=0x7f6bb006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6bb006bbd0 0x7f6bb006e090 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4006fd0 tx=0x7f6bc4008040 comp rx=0 tx=0).stop 2024-03-28T19:28:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0150ae0 msgr2=0x7f6bd0072ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0150ae0 0x7f6bd0072ab0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00c750 tx=0x7f6bbc00cc10 comp rx=0 tx=0).stop 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 shutdown_connections 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f6bb006bbd0 0x7f6bb006e090 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bd0153a30 0x7f6bd01ddd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd0150ae0 0x7f6bd0072ab0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 --2- 172.21.15.27:0/1137731911 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6bd00a61a0 0x7f6bd014a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 >> 172.21.15.27:0/1137731911 conn(0x7f6bd006b0c0 msgr2=0x7f6bd00c5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 shutdown_connections 2024-03-28T19:28:26.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:26.928+0000 7f6bd7000640 1 -- 172.21.15.27:0/1137731911 wait complete. 2024-03-28T19:28:26.949 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:28:28.714 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 27% (51/186) Updating files: 28% (53/186) Updating files: 29% (54/186) Updating files: 30% (56/186) Updating files: 31% (58/186) Updating files: 32% (60/186) Updating files: 33% (62/186) Updating files: 34% (64/186) Updating files: 35% (66/186) Updating files: 36% (67/186) Updating files: 37% (69/186) Updating files: 38% (71/186) Updating files: 39% (73/186) Updating files: 40% (75/186) Updating files: 41% (77/186) Updating files: 42% (79/186) Updating files: 43% (80/186) Updating files: 44% (82/186) Updating files: 45% (84/186) Updating files: 46% (86/186) Updating files: 47% (88/186) Updating files: 48% (90/186) Updating files: 49% (92/186) Updating files: 50% (93/186) Updating files: 51% (95/186) Updating files: 52% (97/186) Updating files: 53% (99/186) Updating files: 54% (101/186) Updating files: 55% (103/186) Updating files: 56% (105/186) Updating files: 57% (107/186) Updating files: 58% (108/186) Updating files: 59% (110/186) Updating files: 60% (112/186) Updating files: 61% (114/186) Updating files: 62% (116/186) Updating files: 63% (118/186) Updating files: 64% (120/186) Updating files: 65% (121/186) Updating files: 66% (123/186) Updating files: 67% (125/186) Updating files: 68% (127/186) Updating files: 69% (129/186) Updating files: 70% (131/186) Updating files: 71% (133/186) Updating files: 72% (134/186) Updating files: 73% (136/186) Updating files: 74% (138/186) Updating files: 75% (140/186) Updating files: 76% (142/186) Updating files: 77% (144/186) Updating files: 78% (146/186) Updating files: 79% (147/186) Updating files: 80% (149/186) Updating files: 81% (151/186) Updating files: 82% (153/186) Updating files: 83% (155/186) Updating files: 84% (157/186) Updating files: 85% (159/186) Updating files: 86% (160/186) Updating files: 87% (162/186) Updating files: 88% (164/186) Updating files: 89% (166/186) Updating files: 90% (168/186) Updating files: 91% (170/186) Updating files: 92% (172/186) Updating files: 93% (173/186) Updating files: 94% (175/186) Updating files: 95% (177/186) Updating files: 96% (179/186) Updating files: 97% (181/186) Updating files: 98% (183/186) Updating files: 98% (184/186) Updating files: 99% (185/186) Updating files: 100% (186/186) Updating files: 100% (186/186), done. 2024-03-28T19:28:28.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:29.112 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at cdd5298d276 v0.44 2024-03-28T19:28:29.122 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.44 2024-03-28T19:28:29.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.180+0000 7f91a0a00640 1 Processor -- start 2024-03-28T19:28:29.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.181+0000 7f91a0a00640 1 -- start start 2024-03-28T19:28:29.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.182+0000 7f91a0a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f919c0c58c0 0x7f919c0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:29.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.182+0000 7f91a0a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:29.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.182+0000 7f91a0a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c156230 con 0x7f919c0c8ed0 2024-03-28T19:28:29.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.182+0000 7f91a0a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919c158bd0 con 0x7f919c0c58c0 2024-03-28T19:28:29.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.182+0000 7f91a0a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c158eb0 con 0x7f919c0be5d0 2024-03-28T19:28:29.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.183+0000 7f919a800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f919c0c58c0 0x7f919c0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:29.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.183+0000 7f919b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:29.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.183+0000 7f919b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:54816/0 (socket says 172.21.15.27:54816) 2024-03-28T19:28:29.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.183+0000 7f919b200640 1 -- 172.21.15.27:0/3892244586 learned_addr learned my addr 172.21.15.27:0/3892244586 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.184+0000 7f919b200640 1 -- 172.21.15.27:0/3892244586 >> v2:172.21.15.155:3300/0 conn(0x7f919c0c58c0 msgr2=0x7f919c0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.184+0000 7f9199400640 1 -- 172.21.15.27:0/3892244586 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101300433 0 0) 0x7f919c156230 con 0x7f919c0c8ed0 2024-03-28T19:28:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.184+0000 7f919b200640 1 --2- 172.21.15.27:0/3892244586 >> v2:172.21.15.155:3300/0 conn(0x7f919c0c58c0 0x7f919c0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.184+0000 7f919b200640 1 -- 172.21.15.27:0/3892244586 >> v1:172.21.15.155:6789/0 conn(0x7f919c0c8ed0 legacy=0x7f919c1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:29.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.184+0000 7f919b200640 1 -- 172.21.15.27:0/3892244586 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c159190 con 0x7f919c0be5d0 2024-03-28T19:28:29.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.186+0000 7f919b200640 1 --2- 172.21.15.27:0/3892244586 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c155cf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9194009e50 tx=0x7f919402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15cf46375ed90a91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:29.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f9199400640 1 -- 172.21.15.27:0/3892244586 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919402ca70 con 0x7f919c0be5d0 2024-03-28T19:28:29.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f9199400640 1 -- 172.21.15.27:0/3892244586 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919402cc10 con 0x7f919c0be5d0 2024-03-28T19:28:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f9199400640 1 -- 172.21.15.27:0/3892244586 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91940343f0 con 0x7f919c0be5d0 2024-03-28T19:28:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f91a0a00640 1 -- 172.21.15.27:0/3892244586 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 msgr2=0x7f919c155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f91a0a00640 1 --2- 172.21.15.27:0/3892244586 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c155cf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9194009e50 tx=0x7f919402bed0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f91a0a00640 1 -- 172.21.15.27:0/3892244586 shutdown_connections 2024-03-28T19:28:29.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f91a0a00640 1 --2- 172.21.15.27:0/3892244586 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c155cf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f91a0a00640 1 --2- 172.21.15.27:0/3892244586 >> v2:172.21.15.155:3300/0 conn(0x7f919c0c58c0 0x7f919c0c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f91a0a00640 1 -- 172.21.15.27:0/3892244586 >> 172.21.15.27:0/3892244586 conn(0x7f919c06b6f0 msgr2=0x7f919c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:29.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f91a0a00640 1 -- 172.21.15.27:0/3892244586 shutdown_connections 2024-03-28T19:28:29.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.187+0000 7f91a0a00640 1 -- 172.21.15.27:0/3892244586 wait complete. 2024-03-28T19:28:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.189+0000 7f91a0a00640 1 Processor -- start 2024-03-28T19:28:29.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.189+0000 7f91a0a00640 1 -- start start 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.190+0000 7f91a0a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.190+0000 7f91a0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f919c0c58c0 0x7f919c1eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.190+0000 7f91a0a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f919c0c8ed0 0x7f919c1ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.190+0000 7f91a0a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c0b0820 con 0x7f919c0c58c0 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.190+0000 7f91a0a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c159750 con 0x7f919c0c8ed0 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.190+0000 7f91a0a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c0b09a0 con 0x7f919c0be5d0 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f9199e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f919c0c58c0 0x7f919c1eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f9199e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f919c0c58c0 0x7f919c1eeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47828/0 (socket says 172.21.15.27:47828) 2024-03-28T19:28:29.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f9199e00640 1 -- 172.21.15.27:0/332312730 learned_addr learned my addr 172.21.15.27:0/332312730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:29.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f919a800640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:29.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f919b200640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f919c0c8ed0 0x7f919c1ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:29.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f9199e00640 1 -- 172.21.15.27:0/332312730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 msgr2=0x7f919c154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f9199e00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f9199e00640 1 -- 172.21.15.27:0/332312730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f919c0c8ed0 msgr2=0x7f919c1ef080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f9199e00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f919c0c8ed0 0x7f919c1ef080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.191+0000 7f9199e00640 1 -- 172.21.15.27:0/332312730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c0c1050 con 0x7f919c0c58c0 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.192+0000 7f9199e00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f919c0c58c0 0x7f919c1eeb40 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f918800c3e0 tx=0x7f918800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.192+0000 7f9192a00640 1 -- 172.21.15.27:0/332312730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9188017070 con 0x7f919c0c58c0 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.192+0000 7f9192a00640 1 -- 172.21.15.27:0/332312730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91880035f0 con 0x7f919c0c58c0 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.192+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c0c1330 con 0x7f919c0c58c0 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.192+0000 7f9192a00640 1 -- 172.21.15.27:0/332312730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9188003a00 con 0x7f919c0c58c0 2024-03-28T19:28:29.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.192+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c154ed0 con 0x7f919c0c58c0 2024-03-28T19:28:29.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.194+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c069640 con 0x7f919c0c58c0 2024-03-28T19:28:29.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.195+0000 7f9192a00640 1 -- 172.21.15.27:0/332312730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f918801a070 con 0x7f919c0c58c0 2024-03-28T19:28:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.196+0000 7f9192a00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f917c06bbd0 0x7f917c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.196+0000 7f9192a00640 1 -- 172.21.15.27:0/332312730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f918808b0d0 con 0x7f919c0c58c0 2024-03-28T19:28:29.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.197+0000 7f919a800640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f917c06bbd0 0x7f917c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.197+0000 7f919a800640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f917c06bbd0 0x7f917c06e090 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f919c0c0c70 tx=0x7f918c008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.201+0000 7f9192a00640 1 -- 172.21.15.27:0/332312730 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9188058990 con 0x7f919c0c58c0 2024-03-28T19:28:29.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:28:29.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 45 -eq 48 ']' 2024-03-28T19:28:29.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.45 2024-03-28T19:28:29.243 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.45 2024-03-28T19:28:29.244 INFO:tasks.workunit.client.0.smithi027.stdout:v0.45 2024-03-28T19:28:29.244 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.45 2024-03-28T19:28:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.506+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f919c156580 con 0x7f919c0c58c0 2024-03-28T19:28:29.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.507+0000 7f9192a00640 1 -- 172.21.15.27:0/332312730 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5468 (secure 0 0 0) 0x7f9188058290 con 0x7f919c0c58c0 2024-03-28T19:28:29.507 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:29.507 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:29.508 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 245 2024-03-28T19:28:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f917c06bbd0 msgr2=0x7f917c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f917c06bbd0 0x7f917c06e090 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f919c0c0c70 tx=0x7f918c008040 comp rx=0 tx=0).stop 2024-03-28T19:28:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f919c0c58c0 msgr2=0x7f919c1eeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f919c0c58c0 0x7f919c1eeb40 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f918800c3e0 tx=0x7f918800c8a0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 shutdown_connections 2024-03-28T19:28:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f917c06bbd0 0x7f917c06e090 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f919c0c8ed0 0x7f919c1ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f919c0c58c0 0x7f919c1eeb40 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 --2- 172.21.15.27:0/332312730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f919c0be5d0 0x7f919c154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 >> 172.21.15.27:0/332312730 conn(0x7f919c06b6f0 msgr2=0x7f919c0ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 shutdown_connections 2024-03-28T19:28:29.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:29.510+0000 7f91a0a00640 1 -- 172.21.15.27:0/332312730 wait complete. 2024-03-28T19:28:29.535 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:28:30.459 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-03-28T19:28:30.459 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-28T19:28:30.459 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-28T19:28:30.467 INFO:tasks.ceph.mds.b:Started 2024-03-28T19:28:30.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:30.678 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2024-03-28T19:28:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.725+0000 7fe66d000640 1 Processor -- start 2024-03-28T19:28:30.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.725+0000 7fe66d000640 1 -- start start 2024-03-28T19:28:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe66d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6680c58c0 0x7fe6680c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe66d000640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6680c8ed0 0x7fe6681555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe66d000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe668156230 con 0x7fe6680be5d0 2024-03-28T19:28:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe66d000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe668158bd0 con 0x7fe6680c58c0 2024-03-28T19:28:30.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe66d000640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe668158eb0 con 0x7fe6680c8ed0 2024-03-28T19:28:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe667400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6680c58c0 0x7fe6680c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe666a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6680c8ed0 0x7fe6681555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe667400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6680c58c0 0x7fe6680c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47842/0 (socket says 172.21.15.27:47842) 2024-03-28T19:28:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.727+0000 7fe667400640 1 -- 172.21.15.27:0/22956379 learned_addr learned my addr 172.21.15.27:0/22956379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:30.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.728+0000 7fe667400640 1 -- 172.21.15.27:0/22956379 >> v2:172.21.15.155:3300/0 conn(0x7fe6680c8ed0 msgr2=0x7fe6681555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.728+0000 7fe667400640 1 --2- 172.21.15.27:0/22956379 >> v2:172.21.15.155:3300/0 conn(0x7fe6680c8ed0 0x7fe6681555e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.728+0000 7fe667400640 1 -- 172.21.15.27:0/22956379 >> v1:172.21.15.155:6789/0 conn(0x7fe6680be5d0 legacy=0x7fe668155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:30.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.728+0000 7fe667400640 1 -- 172.21.15.27:0/22956379 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe668159190 con 0x7fe6680c58c0 2024-03-28T19:28:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.730+0000 7fe667400640 1 --2- 172.21.15.27:0/22956379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680c58c0 0x7fe6680c5ca0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe6680c83d0 tx=0x7fe65802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9846863a9b152426 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.730+0000 7fe666000640 1 -- 172.21.15.27:0/22956379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65802e070 con 0x7fe6680c58c0 2024-03-28T19:28:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.730+0000 7fe666000640 1 -- 172.21.15.27:0/22956379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe658037070 con 0x7fe6680c58c0 2024-03-28T19:28:30.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.730+0000 7fe666000640 1 -- 172.21.15.27:0/22956379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658031a00 con 0x7fe6680c58c0 2024-03-28T19:28:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.731+0000 7fe66d000640 1 -- 172.21.15.27:0/22956379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680c58c0 msgr2=0x7fe6680c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:30.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.731+0000 7fe66d000640 1 --2- 172.21.15.27:0/22956379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680c58c0 0x7fe6680c5ca0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe6680c83d0 tx=0x7fe65802b880 comp rx=0 tx=0).stop 2024-03-28T19:28:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.731+0000 7fe66d000640 1 -- 172.21.15.27:0/22956379 shutdown_connections 2024-03-28T19:28:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.731+0000 7fe66d000640 1 --2- 172.21.15.27:0/22956379 >> v2:172.21.15.155:3300/0 conn(0x7fe6680c8ed0 0x7fe6681555e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.731+0000 7fe66d000640 1 --2- 172.21.15.27:0/22956379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680c58c0 0x7fe6680c5ca0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.731+0000 7fe66d000640 1 -- 172.21.15.27:0/22956379 >> 172.21.15.27:0/22956379 conn(0x7fe66806b6f0 msgr2=0x7fe6680acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.732+0000 7fe66d000640 1 -- 172.21.15.27:0/22956379 shutdown_connections 2024-03-28T19:28:30.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.732+0000 7fe66d000640 1 -- 172.21.15.27:0/22956379 wait complete. 2024-03-28T19:28:30.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.733+0000 7fe66d000640 1 Processor -- start 2024-03-28T19:28:30.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.734+0000 7fe66d000640 1 -- start start 2024-03-28T19:28:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe66d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680be5d0 0x7fe6681eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe66d000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6680c58c0 0x7fe6680c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe66d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6680c8ed0 0x7fe6681e21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe66d000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680b0820 con 0x7fe6680be5d0 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe66d000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe668159750 con 0x7fe6680c8ed0 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe66d000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6680b09a0 con 0x7fe6680c58c0 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe667400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680be5d0 0x7fe6681eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe667e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6680c8ed0 0x7fe6681e21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe667400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680be5d0 0x7fe6681eb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47850/0 (socket says 172.21.15.27:47850) 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe667400640 1 -- 172.21.15.27:0/2777006372 learned_addr learned my addr 172.21.15.27:0/2777006372 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:30.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.735+0000 7fe666a00640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6680c58c0 0x7fe6680c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.736+0000 7fe667400640 1 -- 172.21.15.27:0/2777006372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6680c58c0 msgr2=0x7fe6680c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.736+0000 7fe667400640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6680c58c0 0x7fe6680c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:30.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.736+0000 7fe667400640 1 -- 172.21.15.27:0/2777006372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6680c8ed0 msgr2=0x7fe6681e21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.736+0000 7fe667400640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6680c8ed0 0x7fe6681e21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.736+0000 7fe667400640 1 -- 172.21.15.27:0/2777006372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6680ad080 con 0x7fe6680be5d0 2024-03-28T19:28:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.736+0000 7fe667400640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680be5d0 0x7fe6681eb2e0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe658000c00 tx=0x7fe6580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.737+0000 7fe65f400640 1 -- 172.21.15.27:0/2777006372 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658032670 con 0x7fe6680be5d0 2024-03-28T19:28:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.737+0000 7fe65f400640 1 -- 172.21.15.27:0/2777006372 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe658032850 con 0x7fe6680be5d0 2024-03-28T19:28:30.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.737+0000 7fe65f400640 1 -- 172.21.15.27:0/2777006372 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658032c40 con 0x7fe6680be5d0 2024-03-28T19:28:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.738+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6681e1410 con 0x7fe6680be5d0 2024-03-28T19:28:30.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.738+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6681e19d0 con 0x7fe6680be5d0 2024-03-28T19:28:30.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.740+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe668069640 con 0x7fe6680be5d0 2024-03-28T19:28:30.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.741+0000 7fe65f400640 1 -- 172.21.15.27:0/2777006372 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe65804c030 con 0x7fe6680be5d0 2024-03-28T19:28:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.742+0000 7fe65f400640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64806bbd0 0x7fe64806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:30.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.742+0000 7fe65f400640 1 -- 172.21.15.27:0/2777006372 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7fe6580ad790 con 0x7fe6680be5d0 2024-03-28T19:28:30.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.743+0000 7fe666a00640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64806bbd0 0x7fe64806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:30.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.744+0000 7fe666a00640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64806bbd0 0x7fe64806e090 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe6680c1dc0 tx=0x7fe654009190 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:30.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:30.750+0000 7fe65f400640 1 -- 172.21.15.27:0/2777006372 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe658040070 con 0x7fe6680be5d0 2024-03-28T19:28:31.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.095+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe668156580 con 0x7fe6680be5d0 2024-03-28T19:28:31.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.101+0000 7fe65f400640 1 -- 172.21.15.27:0/2777006372 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5468 (secure 0 0 0) 0x7fe65807a910 con 0x7fe6680be5d0 2024-03-28T19:28:31.101 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:31.101 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:31.101 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 245 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.104+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64806bbd0 msgr2=0x7fe64806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.104+0000 7fe66d000640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64806bbd0 0x7fe64806e090 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe6680c1dc0 tx=0x7fe654009190 comp rx=0 tx=0).stop 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.104+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680be5d0 msgr2=0x7fe6681eb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.104+0000 7fe66d000640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680be5d0 0x7fe6681eb2e0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe658000c00 tx=0x7fe6580046c0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.104+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 shutdown_connections 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.105+0000 7fe66d000640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe64806bbd0 0x7fe64806e090 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.105+0000 7fe66d000640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6680c8ed0 0x7fe6681e21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.105+0000 7fe66d000640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6680c58c0 0x7fe6680c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.105+0000 7fe66d000640 1 --2- 172.21.15.27:0/2777006372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6680be5d0 0x7fe6681eb2e0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.105+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 >> 172.21.15.27:0/2777006372 conn(0x7fe66806b6f0 msgr2=0x7fe6680c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:31.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.105+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 shutdown_connections 2024-03-28T19:28:31.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.105+0000 7fe66d000640 1 -- 172.21.15.27:0/2777006372 wait complete. 2024-03-28T19:28:31.124 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-03-28T19:28:31.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.779+0000 7f5178800640 1 Processor -- start 2024-03-28T19:28:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.779+0000 7f5178800640 1 -- start start 2024-03-28T19:28:31.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.780+0000 7f5178800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f51700c5e80 0x7f51700c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.780+0000 7f5178800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51700abf60 0x7f51700aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.780+0000 7f5178800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517006d8a0 con 0x7f51700c1f50 2024-03-28T19:28:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.780+0000 7f5178800640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517006d650 con 0x7f51700c5e80 2024-03-28T19:28:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.780+0000 7f5178800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517006e1d0 con 0x7f51700abf60 2024-03-28T19:28:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.780+0000 7f5175400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f51700c1f50 0x7f51700ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45014/0 (socket says 172.21.15.27:45014) 2024-03-28T19:28:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.780+0000 7f5175400640 1 -- 172.21.15.27:0/2708079336 learned_addr learned my addr 172.21.15.27:0/2708079336 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.781+0000 7f5175e00640 1 --2- 172.21.15.27:0/2708079336 >> v2:172.21.15.155:3300/0 conn(0x7f51700c5e80 0x7f51700c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.781+0000 7f5176800640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51700abf60 0x7f51700aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.781+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299251960 0 0) 0x7f517006d8a0 con 0x7f51700c1f50 2024-03-28T19:28:31.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.781+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5158003520 con 0x7f51700c1f50 2024-03-28T19:28:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.781+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2815663874 0 0) 0x7f5158003520 con 0x7f51700c1f50 2024-03-28T19:28:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.781+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51700abf60 msgr2=0x7f51700aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51700abf60 0x7f51700aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 >> v2:172.21.15.155:3300/0 conn(0x7f51700c5e80 msgr2=0x7f51700c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 --2- 172.21.15.27:0/2708079336 >> v2:172.21.15.155:3300/0 conn(0x7f51700c5e80 0x7f51700c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517006db80 con 0x7f51700c1f50 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f5160001670 con 0x7f51700c1f50 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 >> v1:172.21.15.27:6789/0 conn(0x7f51700c1f50 legacy=0x7f51700ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51580039b0 0x7f5158003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5158004800 0x7f5158004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51700abf60 0x7f51580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517006d650 con 0x7f5158004800 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517006e1d0 con 0x7f51580039b0 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515800ba10 con 0x7f51700abf60 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5175400640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5158004800 0x7f5158004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5175e00640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51580039b0 0x7f5158003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.782+0000 7f5176800640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51700abf60 0x7f51580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.783+0000 7f5175400640 1 -- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51700abf60 msgr2=0x7f51580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.783+0000 7f5175400640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51700abf60 0x7f51580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.783+0000 7f5175400640 1 -- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51580039b0 msgr2=0x7f5158003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.783+0000 7f5175400640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51580039b0 0x7f5158003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.783+0000 7f5175400640 1 -- 172.21.15.27:0/2708079336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517006db80 con 0x7f5158004800 2024-03-28T19:28:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.784+0000 7f5175400640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5158004800 0x7f5158004c80 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f516000c370 tx=0x7f5158005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3300c32a25439 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.784+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516002cc20 con 0x7f5158004800 2024-03-28T19:28:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.784+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5160030070 con 0x7f5158004800 2024-03-28T19:28:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.784+0000 7f5174a00640 1 -- 172.21.15.27:0/2708079336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51600058d0 con 0x7f5158004800 2024-03-28T19:28:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.785+0000 7f5178800640 1 -- 172.21.15.27:0/2708079336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5158004800 msgr2=0x7f5158004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.785+0000 7f5178800640 1 --2- 172.21.15.27:0/2708079336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5158004800 0x7f5158004c80 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f516000c370 tx=0x7f5158005830 comp rx=0 tx=0).stop 2024-03-28T19:28:31.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.785+0000 7f5176800640 1 -- 172.21.15.27:0/2708079336 reap_dead start 2024-03-28T19:28:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.786+0000 7f5178800640 1 -- 172.21.15.27:0/2708079336 shutdown_connections 2024-03-28T19:28:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.786+0000 7f5178800640 1 -- 172.21.15.27:0/2708079336 >> 172.21.15.27:0/2708079336 conn(0x7f517006b9d0 msgr2=0x7f51700c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.786+0000 7f5178800640 1 -- 172.21.15.27:0/2708079336 shutdown_connections 2024-03-28T19:28:31.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.786+0000 7f5178800640 1 -- 172.21.15.27:0/2708079336 wait complete. 2024-03-28T19:28:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.788+0000 7f5178800640 1 Processor -- start 2024-03-28T19:28:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.788+0000 7f5178800640 1 -- start start 2024-03-28T19:28:31.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.790+0000 7f5178800640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5158004800 0x7f51701ddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.790+0000 7f5178800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51701506b0 0x7f51700a7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.790+0000 7f5178800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51700a7870 0x7f51701ea290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.790+0000 7f5178800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51700c3370 con 0x7f51700a7870 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.790+0000 7f5178800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51700c31f0 con 0x7f51701506b0 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.790+0000 7f5178800640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51700c6260 con 0x7f5158004800 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.790+0000 7f5176800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51700a7870 0x7f51701ea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.790+0000 7f5176800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51700a7870 0x7f51701ea290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47860/0 (socket says 172.21.15.27:47860) 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.791+0000 7f5176800640 1 -- 172.21.15.27:0/2497843614 learned_addr learned my addr 172.21.15.27:0/2497843614 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.791+0000 7f5175e00640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5158004800 0x7f51701ddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.791+0000 7f5175400640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51701506b0 0x7f51700a7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.791+0000 7f5176800640 1 -- 172.21.15.27:0/2497843614 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5158004800 msgr2=0x7f51701ddfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.791+0000 7f5176800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5158004800 0x7f51701ddfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.791+0000 7f5176800640 1 -- 172.21.15.27:0/2497843614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51701506b0 msgr2=0x7f51700a7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.791+0000 7f5176800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51701506b0 0x7f51700a7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.791+0000 7f5176800640 1 -- 172.21.15.27:0/2497843614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51700bbda0 con 0x7f51700a7870 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.792+0000 7f5176800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51700a7870 0x7f51701ea290 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f51700709e0 tx=0x7f516c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.792+0000 7f516a000640 1 -- 172.21.15.27:0/2497843614 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c007e60 con 0x7f51700a7870 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.792+0000 7f516a000640 1 -- 172.21.15.27:0/2497843614 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c003da0 con 0x7f51700a7870 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.792+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51701eadd0 con 0x7f51700a7870 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.792+0000 7f516a000640 1 -- 172.21.15.27:0/2497843614 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c004270 con 0x7f51700a7870 2024-03-28T19:28:31.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.792+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51701eb0b0 con 0x7f51700a7870 2024-03-28T19:28:31.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.794+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517004f400 con 0x7f51700a7870 2024-03-28T19:28:31.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.796+0000 7f516a000640 1 -- 172.21.15.27:0/2497843614 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f516c004410 con 0x7f51700a7870 2024-03-28T19:28:31.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.797+0000 7f516a000640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f515406bc00 0x7f515406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.798+0000 7f516a000640 1 -- 172.21.15.27:0/2497843614 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f516c08ff30 con 0x7f51700a7870 2024-03-28T19:28:31.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.798+0000 7f5175e00640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f515406bc00 0x7f515406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:31.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.799+0000 7f5175e00640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f515406bc00 0x7f515406e0c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5164009a10 tx=0x7f5164008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:31.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:31.803+0000 7f516a000640 1 -- 172.21.15.27:0/2497843614 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f516c05d0b0 con 0x7f51700a7870 2024-03-28T19:28:32.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.123+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51700ab790 con 0x7f51700a7870 2024-03-28T19:28:32.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.124+0000 7f516a000640 1 -- 172.21.15.27:0/2497843614 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f516c061150 con 0x7f51700a7870 2024-03-28T19:28:32.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:32.124 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246},{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:32.124 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 246 2024-03-28T19:28:32.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.126+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f515406bc00 msgr2=0x7f515406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:32.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.126+0000 7f5178800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f515406bc00 0x7f515406e0c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5164009a10 tx=0x7f5164008040 comp rx=0 tx=0).stop 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.126+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51700a7870 msgr2=0x7f51701ea290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.126+0000 7f5178800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51700a7870 0x7f51701ea290 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f51700709e0 tx=0x7f516c007700 comp rx=0 tx=0).stop 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.127+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 shutdown_connections 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.127+0000 7f5178800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f515406bc00 0x7f515406e0c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.127+0000 7f5178800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51700a7870 0x7f51701ea290 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.127+0000 7f5178800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f51701506b0 0x7f51700a7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.127+0000 7f5178800640 1 --2- 172.21.15.27:0/2497843614 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5158004800 0x7f51701ddfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.127+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 >> 172.21.15.27:0/2497843614 conn(0x7f517006b9d0 msgr2=0x7f5170072aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.127+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 shutdown_connections 2024-03-28T19:28:32.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:32.127+0000 7f5178800640 1 -- 172.21.15.27:0/2497843614 wait complete. 2024-03-28T19:28:32.154 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:28:33.125 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.351+0000 7f5852000640 1 Processor -- start 2024-03-28T19:28:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.351+0000 7f5852000640 1 -- start start 2024-03-28T19:28:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.352+0000 7f5852000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f584c08b880 0x7f584c0c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.352+0000 7f5852000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0ab350 0x7f584c0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.352+0000 7f5852000640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584c0c42a0 con 0x7f584c0c4910 2024-03-28T19:28:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.352+0000 7f5852000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584c0c35f0 con 0x7f584c08b880 2024-03-28T19:28:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.352+0000 7f5852000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584c06e0f0 con 0x7f584c0ab350 2024-03-28T19:28:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.352+0000 7f5850c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f584c08b880 0x7f584c0c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.352+0000 7f5850c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f584c08b880 0x7f584c0c3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47872/0 (socket says 172.21.15.27:47872) 2024-03-28T19:28:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.352+0000 7f5850c00640 1 -- 172.21.15.27:0/3261790605 learned_addr learned my addr 172.21.15.27:0/3261790605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.354+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839245093 0 0) 0x7f584c0c42a0 con 0x7f584c0c4910 2024-03-28T19:28:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.507+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5834005730 con 0x7f584c0c4910 2024-03-28T19:28:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.353+0000 7f5851600640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0ab350 0x7f584c0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986325314 0 0) 0x7f5834005730 con 0x7f584c0c4910 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0ab350 msgr2=0x7f584c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0ab350 0x7f584c0c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 >> v2:172.21.15.27:3300/0 conn(0x7f584c08b880 msgr2=0x7f584c0c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 --2- 172.21.15.27:0/3261790605 >> v2:172.21.15.27:3300/0 conn(0x7f584c08b880 0x7f584c0c3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c06e3d0 con 0x7f584c0c4910 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f5838001690 con 0x7f584c0c4910 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 >> v1:172.21.15.155:6789/0 conn(0x7f584c0c4910 legacy=0x7f584c0a99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5834005bc0 0x7f5834005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5834006520 0x7f5834006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5834007060 0x7f58340074e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c0c35f0 con 0x7f5834006520 2024-03-28T19:28:33.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c06e0f0 con 0x7f5834007060 2024-03-28T19:28:33.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.509+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340103d0 con 0x7f5834005bc0 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.510+0000 7f584be00640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5834006520 0x7f5834006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.510+0000 7f5850c00640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5834005bc0 0x7f5834005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.511+0000 7f5851600640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5834007060 0x7f58340074e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.511+0000 7f5850c00640 1 -- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5834007060 msgr2=0x7f58340074e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.511+0000 7f5850c00640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5834007060 0x7f58340074e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.511+0000 7f5850c00640 1 -- 172.21.15.27:0/3261790605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5834006520 msgr2=0x7f5834006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.511+0000 7f5850c00640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5834006520 0x7f5834006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.511+0000 7f5850c00640 1 -- 172.21.15.27:0/3261790605 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c06e3d0 con 0x7f5834005bc0 2024-03-28T19:28:33.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.511+0000 7f584be00640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5834006520 0x7f5834006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:28:33.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.513+0000 7f5850c00640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5834005bc0 0x7f5834005fe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5840008bb0 tx=0x7f5834007a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85e8afdd474341c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:33.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.514+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840032070 con 0x7f5834005bc0 2024-03-28T19:28:33.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.514+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584000ae50 con 0x7f5834005bc0 2024-03-28T19:28:33.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.514+0000 7f584b400640 1 -- 172.21.15.27:0/3261790605 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840008860 con 0x7f5834005bc0 2024-03-28T19:28:33.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.515+0000 7f5852000640 1 -- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5834005bc0 msgr2=0x7f5834005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:33.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.515+0000 7f5852000640 1 --2- 172.21.15.27:0/3261790605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5834005bc0 0x7f5834005fe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5840008bb0 tx=0x7f5834007a40 comp rx=0 tx=0).stop 2024-03-28T19:28:33.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.515+0000 7f5851600640 1 -- 172.21.15.27:0/3261790605 reap_dead start 2024-03-28T19:28:33.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.515+0000 7f5852000640 1 -- 172.21.15.27:0/3261790605 shutdown_connections 2024-03-28T19:28:33.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.515+0000 7f5852000640 1 -- 172.21.15.27:0/3261790605 >> 172.21.15.27:0/3261790605 conn(0x7f584c06cd50 msgr2=0x7f584c06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:33.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.516+0000 7f5852000640 1 -- 172.21.15.27:0/3261790605 shutdown_connections 2024-03-28T19:28:33.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.516+0000 7f5852000640 1 -- 172.21.15.27:0/3261790605 wait complete. 2024-03-28T19:28:33.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.518+0000 7f5852000640 1 Processor -- start 2024-03-28T19:28:33.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.518+0000 7f5852000640 1 -- start start 2024-03-28T19:28:33.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f5852000640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0a9e60 0x7f584c1f8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f5852000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f584c164d60 0x7f584c15e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f5852000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f584c15eeb0 0x7f584c165bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f5852000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c08bc60 con 0x7f584c164d60 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f5852000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584c0c3840 con 0x7f584c15eeb0 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f5852000640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584c0c1400 con 0x7f584c0a9e60 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f584be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f584c164d60 0x7f584c15e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f584be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f584c164d60 0x7f584c15e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47880/0 (socket says 172.21.15.27:47880) 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.519+0000 7f584be00640 1 -- 172.21.15.27:0/2869193 learned_addr learned my addr 172.21.15.27:0/2869193 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.520+0000 7f5850c00640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0a9e60 0x7f584c1f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.520+0000 7f5851600640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f584c15eeb0 0x7f584c165bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.520+0000 7f584be00640 1 -- 172.21.15.27:0/2869193 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0a9e60 msgr2=0x7f584c1f8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:33.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.520+0000 7f584be00640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0a9e60 0x7f584c1f8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.520+0000 7f584be00640 1 -- 172.21.15.27:0/2869193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f584c15eeb0 msgr2=0x7f584c165bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:33.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.520+0000 7f584be00640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f584c15eeb0 0x7f584c165bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.520+0000 7f584be00640 1 -- 172.21.15.27:0/2869193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c1f8850 con 0x7f584c164d60 2024-03-28T19:28:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.520+0000 7f584be00640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f584c164d60 0x7f584c15e970 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f583800faf0 tx=0x7f5838001060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.521+0000 7f5848c00640 1 -- 172.21.15.27:0/2869193 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58380052e0 con 0x7f584c164d60 2024-03-28T19:28:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.521+0000 7f5848c00640 1 -- 172.21.15.27:0/2869193 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5838018070 con 0x7f584c164d60 2024-03-28T19:28:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.521+0000 7f5848c00640 1 -- 172.21.15.27:0/2869193 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583800d9b0 con 0x7f584c164d60 2024-03-28T19:28:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.521+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584c0bba20 con 0x7f584c164d60 2024-03-28T19:28:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.521+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584c1653c0 con 0x7f584c164d60 2024-03-28T19:28:33.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.524+0000 7f5848c00640 1 -- 172.21.15.27:0/2869193 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f583800db50 con 0x7f584c164d60 2024-03-28T19:28:33.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.524+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584c067dd0 con 0x7f584c164d60 2024-03-28T19:28:33.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.525+0000 7f5848c00640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f582c06bc00 0x7f582c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:33.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.525+0000 7f5848c00640 1 -- 172.21.15.27:0/2869193 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f583808bdc0 con 0x7f584c164d60 2024-03-28T19:28:33.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.525+0000 7f5850c00640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f582c06bc00 0x7f582c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.526+0000 7f5850c00640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f582c06bc00 0x7f582c06e0c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5840037000 tx=0x7f584000e040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:33.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.530+0000 7f5848c00640 1 -- 172.21.15.27:0/2869193 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f583801a310 con 0x7f584c164d60 2024-03-28T19:28:33.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.838+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f584c0bbf40 con 0x7f584c164d60 2024-03-28T19:28:33.839 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:33.839 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246},{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:33.839 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 246 2024-03-28T19:28:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.839+0000 7f5848c00640 1 -- 172.21.15.27:0/2869193 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f5838059620 con 0x7f584c164d60 2024-03-28T19:28:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f582c06bc00 msgr2=0x7f582c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f582c06bc00 0x7f582c06e0c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5840037000 tx=0x7f584000e040 comp rx=0 tx=0).stop 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f584c164d60 msgr2=0x7f584c15e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f584c164d60 0x7f584c15e970 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f583800faf0 tx=0x7f5838001060 comp rx=0 tx=0).stop 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 shutdown_connections 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f582c06bc00 0x7f582c06e0c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f584c15eeb0 0x7f584c165bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f584c164d60 0x7f584c15e970 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 --2- 172.21.15.27:0/2869193 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f584c0a9e60 0x7f584c1f8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 >> 172.21.15.27:0/2869193 conn(0x7f584c06cd50 msgr2=0x7f584c1617e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 shutdown_connections 2024-03-28T19:28:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:33.842+0000 7f5852000640 1 -- 172.21.15.27:0/2869193 wait complete. 2024-03-28T19:28:33.868 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-03-28T19:28:33.869 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2024-03-28T19:28:34.154 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:34.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.376+0000 7f78bce00640 1 Processor -- start 2024-03-28T19:28:34.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.376+0000 7f78bce00640 1 -- start start 2024-03-28T19:28:34.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.377+0000 7f78bce00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b80c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:34.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.377+0000 7f78bce00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f78b81534c0 0x7f78b8153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:34.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.377+0000 7f78bce00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b80c3d90 con 0x7f78b8150c70 2024-03-28T19:28:34.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.377+0000 7f78bce00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b80ab4e0 con 0x7f78b81534c0 2024-03-28T19:28:34.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.377+0000 7f78bce00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80ac1f0 con 0x7f78b80a9cb0 2024-03-28T19:28:34.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.378+0000 7f78b5400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b80c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:34.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.378+0000 7f78b6800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f78b81534c0 0x7f78b8153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:34.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.378+0000 7f78b5400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b80c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:54900/0 (socket says 172.21.15.27:54900) 2024-03-28T19:28:34.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.378+0000 7f78b6800640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f78b81534c0 0x7f78b8153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.27:41450/0 (socket says 172.21.15.27:41450) 2024-03-28T19:28:34.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.378+0000 7f78b5400640 1 -- 172.21.15.27:0/3933821714 learned_addr learned my addr 172.21.15.27:0/3933821714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:34.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.379+0000 7f78b6800640 1 -- 172.21.15.27:0/3933821714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 msgr2=0x7f78b80c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:34.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.379+0000 7f78b6800640 1 --2- 172.21.15.27:0/3933821714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b80c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.379+0000 7f78b6800640 1 -- 172.21.15.27:0/3933821714 >> v1:172.21.15.155:6789/0 conn(0x7f78b8150c70 legacy=0x7f78b80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:34.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.379+0000 7f78b6800640 1 -- 172.21.15.27:0/3933821714 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b80ab710 con 0x7f78b81534c0 2024-03-28T19:28:34.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.380+0000 7f78b6800640 1 --2- 172.21.15.27:0/3933821714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 0x7f78b8153950 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009dc0 tx=0x7f78b002be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1303577e34bd5b39 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.381+0000 7f78b4a00640 1 -- 172.21.15.27:0/3933821714 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b0039070 con 0x7f78b81534c0 2024-03-28T19:28:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.381+0000 7f78b4a00640 1 -- 172.21.15.27:0/3933821714 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b002cbb0 con 0x7f78b81534c0 2024-03-28T19:28:34.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.381+0000 7f78b4a00640 1 -- 172.21.15.27:0/3933821714 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b0034440 con 0x7f78b81534c0 2024-03-28T19:28:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.381+0000 7f78bce00640 1 -- 172.21.15.27:0/3933821714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 msgr2=0x7f78b8153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.381+0000 7f78bce00640 1 --2- 172.21.15.27:0/3933821714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 0x7f78b8153950 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009dc0 tx=0x7f78b002be00 comp rx=0 tx=0).stop 2024-03-28T19:28:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.382+0000 7f78bce00640 1 -- 172.21.15.27:0/3933821714 shutdown_connections 2024-03-28T19:28:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.382+0000 7f78bce00640 1 --2- 172.21.15.27:0/3933821714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 0x7f78b8153950 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.382+0000 7f78bce00640 1 --2- 172.21.15.27:0/3933821714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b80c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.382+0000 7f78bce00640 1 -- 172.21.15.27:0/3933821714 >> 172.21.15.27:0/3933821714 conn(0x7f78b806b9d0 msgr2=0x7f78b80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.382+0000 7f78bce00640 1 -- 172.21.15.27:0/3933821714 shutdown_connections 2024-03-28T19:28:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.382+0000 7f78bce00640 1 -- 172.21.15.27:0/3933821714 wait complete. 2024-03-28T19:28:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.384+0000 7f78bce00640 1 Processor -- start 2024-03-28T19:28:34.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.384+0000 7f78bce00640 1 -- start start 2024-03-28T19:28:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.386+0000 7f78bce00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b81ddf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.386+0000 7f78bce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b8150c70 0x7f78b806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.386+0000 7f78bce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 0x7f78b81eec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.386+0000 7f78bce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80c1810 con 0x7f78b8150c70 2024-03-28T19:28:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.386+0000 7f78bce00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80abcd0 con 0x7f78b81534c0 2024-03-28T19:28:34.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.386+0000 7f78bce00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80c6830 con 0x7f78b80a9cb0 2024-03-28T19:28:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b8150c70 0x7f78b806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:34.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b8150c70 0x7f78b806e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47884/0 (socket says 172.21.15.27:47884) 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5400640 1 -- 172.21.15.27:0/2905823075 learned_addr learned my addr 172.21.15.27:0/2905823075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5e00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b81ddf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b6800640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 0x7f78b81eec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5400640 1 -- 172.21.15.27:0/2905823075 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 msgr2=0x7f78b81ddf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5400640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b81ddf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5400640 1 -- 172.21.15.27:0/2905823075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 msgr2=0x7f78b81eec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5400640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 0x7f78b81eec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.387+0000 7f78b5400640 1 -- 172.21.15.27:0/2905823075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b80a6600 con 0x7f78b8150c70 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.388+0000 7f78b5400640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b8150c70 0x7f78b806e580 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009fd0 tx=0x7f78a400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.388+0000 7f78ae000640 1 -- 172.21.15.27:0/2905823075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4017070 con 0x7f78b8150c70 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.388+0000 7f78ae000640 1 -- 172.21.15.27:0/2905823075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a400c940 con 0x7f78b8150c70 2024-03-28T19:28:34.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.388+0000 7f78ae000640 1 -- 172.21.15.27:0/2905823075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a400cd10 con 0x7f78b8150c70 2024-03-28T19:28:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.388+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b81ee440 con 0x7f78b8150c70 2024-03-28T19:28:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.388+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b81de4c0 con 0x7f78b8150c70 2024-03-28T19:28:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.390+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b804f400 con 0x7f78b8150c70 2024-03-28T19:28:34.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.390+0000 7f78ae000640 1 -- 172.21.15.27:0/2905823075 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f78a401a070 con 0x7f78b8150c70 2024-03-28T19:28:34.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.391+0000 7f78ae000640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f789806bbd0 0x7f789806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:34.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.391+0000 7f78ae000640 1 -- 172.21.15.27:0/2905823075 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f78a408bec0 con 0x7f78b8150c70 2024-03-28T19:28:34.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.392+0000 7f78b5e00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f789806bbd0 0x7f789806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:34.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.392+0000 7f78b5e00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f789806bbd0 0x7f789806e090 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f78b80a8890 tx=0x7f78a8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:34.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.397+0000 7f78ae000640 1 -- 172.21.15.27:0/2905823075 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f78a4059130 con 0x7f78b8150c70 2024-03-28T19:28:34.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.681+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78b80c5f00 con 0x7f78b8150c70 2024-03-28T19:28:34.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.683+0000 7f78ae000640 1 -- 172.21.15.27:0/2905823075 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f78a405d1d0 con 0x7f78b8150c70 2024-03-28T19:28:34.682 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:34.682 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246},{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 246 2024-03-28T19:28:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f789806bbd0 msgr2=0x7f789806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f789806bbd0 0x7f789806e090 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f78b80a8890 tx=0x7f78a8009040 comp rx=0 tx=0).stop 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b8150c70 msgr2=0x7f78b806e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b8150c70 0x7f78b806e580 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009fd0 tx=0x7f78a400ed40 comp rx=0 tx=0).stop 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 shutdown_connections 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f789806bbd0 0x7f789806e090 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b81534c0 0x7f78b81eec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78b8150c70 0x7f78b806e580 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 --2- 172.21.15.27:0/2905823075 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f78b80a9cb0 0x7f78b81ddf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 >> 172.21.15.27:0/2905823075 conn(0x7f78b806b9d0 msgr2=0x7f78b80c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 shutdown_connections 2024-03-28T19:28:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:34.686+0000 7f78bce00640 1 -- 172.21.15.27:0/2905823075 wait complete. 2024-03-28T19:28:34.708 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:28:36.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:36.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.944+0000 7f0084200640 1 Processor -- start 2024-03-28T19:28:36.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.944+0000 7f0084200640 1 -- start start 2024-03-28T19:28:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.945+0000 7f0084200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 0x7f007c0c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.945+0000 7f0084200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007c06d8a0 con 0x7f007c0abf60 2024-03-28T19:28:36.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.945+0000 7f0084200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007c06db80 con 0x7f007c0c1f50 2024-03-28T19:28:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.945+0000 7f0084200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c06d650 con 0x7f007c0c5e80 2024-03-28T19:28:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f0082200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f007c0abf60 0x7f007c0aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54460/0 (socket says 172.21.15.27:54460) 2024-03-28T19:28:36.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f0082200640 1 -- 172.21.15.27:0/3456523980 learned_addr learned my addr 172.21.15.27:0/3456523980 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f0081800640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 0x7f007c0c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581266756 0 0) 0x7f007c06d8a0 con 0x7f007c0abf60 2024-03-28T19:28:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0064003520 con 0x7f007c0abf60 2024-03-28T19:28:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1260139340 0 0) 0x7f0064003520 con 0x7f007c0abf60 2024-03-28T19:28:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 msgr2=0x7f007c0c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f007be00640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 0x7f007c0c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 >> v1:172.21.15.155:6789/0 conn(0x7f007c0c1f50 legacy=0x7f007c0ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:36.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.946+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c06de60 con 0x7f007c0abf60 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f00740019f0 con 0x7f007c0abf60 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 >> v1:172.21.15.27:6789/0 conn(0x7f007c0abf60 legacy=0x7f007c0aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f007be00640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 0x7f00640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f007be00640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0064003f60 0x7f0064004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f007be00640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f00640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c06d650 con 0x7f0064004d40 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006400bc00 con 0x7f0064003f60 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006400e010 con 0x7f007c0c5e80 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f0082200640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f00640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f0081800640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 0x7f00640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.947+0000 7f0080e00640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0064003f60 0x7f0064004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.948+0000 7f0082200640 1 -- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 msgr2=0x7f00640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.948+0000 7f0082200640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 0x7f00640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:36.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.948+0000 7f0082200640 1 -- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0064003f60 msgr2=0x7f0064004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.948+0000 7f0082200640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0064003f60 0x7f0064004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.948+0000 7f0082200640 1 -- 172.21.15.27:0/3456523980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0064003520 con 0x7f0064004d40 2024-03-28T19:28:36.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.948+0000 7f0081800640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c0c5e80 0x7f00640039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T19:28:36.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.949+0000 7f0082200640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f00640051c0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f007c070a00 tx=0x7f0064005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb2a9dcce934a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:36.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007400a4a0 con 0x7f0064004d40 2024-03-28T19:28:36.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0074034650 con 0x7f0064004d40 2024-03-28T19:28:36.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f007be00640 1 -- 172.21.15.27:0/3456523980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0074034a80 con 0x7f0064004d40 2024-03-28T19:28:36.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f0084200640 1 -- 172.21.15.27:0/3456523980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 msgr2=0x7f00640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:36.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f0084200640 1 --2- 172.21.15.27:0/3456523980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f00640051c0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f007c070a00 tx=0x7f0064005830 comp rx=0 tx=0).stop 2024-03-28T19:28:36.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f0082200640 1 -- 172.21.15.27:0/3456523980 reap_dead start 2024-03-28T19:28:36.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f0084200640 1 -- 172.21.15.27:0/3456523980 shutdown_connections 2024-03-28T19:28:36.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f0084200640 1 -- 172.21.15.27:0/3456523980 >> 172.21.15.27:0/3456523980 conn(0x7f007c06b9d0 msgr2=0x7f007c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:36.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.950+0000 7f0084200640 1 -- 172.21.15.27:0/3456523980 shutdown_connections 2024-03-28T19:28:36.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.951+0000 7f0084200640 1 -- 172.21.15.27:0/3456523980 wait complete. 2024-03-28T19:28:36.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.953+0000 7f0084200640 1 Processor -- start 2024-03-28T19:28:36.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.953+0000 7f0084200640 1 -- start start 2024-03-28T19:28:36.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0084200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f007c154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:36.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0084200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c1def00 0x7f007c1df380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:36.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0084200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f007c1dc520 0x7f007c1dc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:36.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0084200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0aac80 con 0x7f0064004d40 2024-03-28T19:28:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0084200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c06e1d0 con 0x7f007c1dc520 2024-03-28T19:28:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0084200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007c0aae00 con 0x7f007c1def00 2024-03-28T19:28:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0081800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f007c154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0081800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f007c154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44898/0 (socket says 172.21.15.27:44898) 2024-03-28T19:28:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0081800640 1 -- 172.21.15.27:0/2729666569 learned_addr learned my addr 172.21.15.27:0/2729666569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.955+0000 7f0080e00640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c1def00 0x7f007c1df380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:36.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.956+0000 7f0082200640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f007c1dc520 0x7f007c1dc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.956+0000 7f0081800640 1 -- 172.21.15.27:0/2729666569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c1def00 msgr2=0x7f007c1df380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.956+0000 7f0081800640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c1def00 0x7f007c1df380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.956+0000 7f0081800640 1 -- 172.21.15.27:0/2729666569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f007c1dc520 msgr2=0x7f007c1dc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.956+0000 7f0081800640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f007c1dc520 0x7f007c1dc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:36.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.956+0000 7f0081800640 1 -- 172.21.15.27:0/2729666569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c1df8c0 con 0x7f0064004d40 2024-03-28T19:28:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.956+0000 7f0081800640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f007c154d60 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f00700116d0 tx=0x7f0070011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.957+0000 7f0079600640 1 -- 172.21.15.27:0/2729666569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007000f9c0 con 0x7f0064004d40 2024-03-28T19:28:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.957+0000 7f0079600640 1 -- 172.21.15.27:0/2729666569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007000fb60 con 0x7f0064004d40 2024-03-28T19:28:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.957+0000 7f0079600640 1 -- 172.21.15.27:0/2729666569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070015460 con 0x7f0064004d40 2024-03-28T19:28:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.957+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c1ef260 con 0x7f0064004d40 2024-03-28T19:28:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.957+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c0a5920 con 0x7f0064004d40 2024-03-28T19:28:36.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.958+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c0710d0 con 0x7f0064004d40 2024-03-28T19:28:36.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.960+0000 7f0079600640 1 -- 172.21.15.27:0/2729666569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0070015600 con 0x7f0064004d40 2024-03-28T19:28:36.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.961+0000 7f0079600640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f005c06bbd0 0x7f005c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:36.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.961+0000 7f0079600640 1 -- 172.21.15.27:0/2729666569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7f007008c5d0 con 0x7f0064004d40 2024-03-28T19:28:36.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.961+0000 7f0080e00640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f005c06bbd0 0x7f005c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:36.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.962+0000 7f0080e00640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f005c06bbd0 0x7f005c06e090 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f0068007fd0 tx=0x7f0068009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:36.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:36.967+0000 7f0079600640 1 -- 172.21.15.27:0/2729666569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f007c0710d0 con 0x7f0064004d40 2024-03-28T19:28:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.283+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f007c0ab1e0 con 0x7f0064004d40 2024-03-28T19:28:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.284+0000 7f0079600640 1 -- 172.21.15.27:0/2729666569 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+6263 (secure 0 0 0) 0x7f0070019030 con 0x7f0064004d40 2024-03-28T19:28:37.283 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:37.283 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246},{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 246 2024-03-28T19:28:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f005c06bbd0 msgr2=0x7f005c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f005c06bbd0 0x7f005c06e090 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f0068007fd0 tx=0x7f0068009040 comp rx=0 tx=0).stop 2024-03-28T19:28:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 msgr2=0x7f007c154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f007c154d60 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f00700116d0 tx=0x7f0070011b90 comp rx=0 tx=0).stop 2024-03-28T19:28:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 shutdown_connections 2024-03-28T19:28:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f005c06bbd0 0x7f005c06e090 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f007c1dc520 0x7f007c1dc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f007c1def00 0x7f007c1df380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 --2- 172.21.15.27:0/2729666569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0064004d40 0x7f007c154d60 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 >> 172.21.15.27:0/2729666569 conn(0x7f007c06b9d0 msgr2=0x7f007c06cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 shutdown_connections 2024-03-28T19:28:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:37.287+0000 7f0084200640 1 -- 172.21.15.27:0/2729666569 wait complete. 2024-03-28T19:28:37.305 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:28:37.928 INFO:tasks.workunit.client.0.smithi027.stderr:Updating files: 20% (31/154) Updating files: 21% (33/154) Updating files: 22% (34/154) Updating files: 23% (36/154) Updating files: 24% (37/154) Updating files: 25% (39/154) Updating files: 26% (41/154) Updating files: 27% (42/154) Updating files: 28% (44/154) Updating files: 29% (45/154) Updating files: 30% (47/154) Updating files: 31% (48/154) Updating files: 32% (50/154) Updating files: 33% (51/154) Updating files: 34% (53/154) Updating files: 35% (54/154) Updating files: 36% (56/154) Updating files: 37% (57/154) Updating files: 37% (58/154) Updating files: 38% (59/154) Updating files: 39% (61/154) Updating files: 40% (62/154) Updating files: 41% (64/154) Updating files: 42% (65/154) Updating files: 43% (67/154) Updating files: 44% (68/154) Updating files: 45% (70/154) Updating files: 46% (71/154) Updating files: 47% (73/154) Updating files: 48% (74/154) Updating files: 49% (76/154) Updating files: 50% (77/154) Updating files: 51% (79/154) Updating files: 52% (81/154) Updating files: 53% (82/154) Updating files: 54% (84/154) Updating files: 55% (85/154) Updating files: 56% (87/154) Updating files: 57% (88/154) Updating files: 58% (90/154) Updating files: 59% (91/154) Updating files: 60% (93/154) Updating files: 61% (94/154) Updating files: 62% (96/154) Updating files: 63% (98/154) Updating files: 64% (99/154) Updating files: 65% (101/154) Updating files: 66% (102/154) Updating files: 67% (104/154) Updating files: 68% (105/154) Updating files: 69% (107/154) Updating files: 70% (108/154) Updating files: 71% (110/154) Updating files: 72% (111/154) Updating files: 73% (113/154) Updating files: 74% (114/154) Updating files: 75% (116/154) Updating files: 76% (118/154) Updating files: 77% (119/154) Updating files: 78% (121/154) Updating files: 79% (122/154) Updating files: 80% (124/154) Updating files: 81% (125/154) Updating files: 82% (127/154) Updating files: 83% (128/154) Updating files: 84% (130/154) Updating files: 85% (131/154) Updating files: 86% (133/154) Updating files: 87% (134/154) Updating files: 88% (136/154) Updating files: 89% (138/154) Updating files: 90% (139/154) Updating files: 91% (141/154) Updating files: 92% (142/154) Updating files: 93% (144/154) Updating files: 94% (145/154) Updating files: 95% (147/154) Updating files: 96% (148/154) Updating files: 97% (150/154) Updating files: 98% (151/154) Updating files: 99% (153/154) Updating files: 100% (154/154) Updating files: 100% (154/154), done. 2024-03-28T19:28:38.372 INFO:tasks.workunit.client.0.smithi027.stdout:HEAD is now at 0aea1cb1df5 v0.45 2024-03-28T19:28:38.382 INFO:tasks.workunit.client.0.smithi027.stderr:+ mkdir .snap/v0.45 2024-03-28T19:28:38.473 INFO:tasks.workunit.client.0.smithi027.stderr:+ for v in $versions 2024-03-28T19:28:38.473 INFO:tasks.workunit.client.0.smithi027.stderr:+ '[' 46 -eq 48 ']' 2024-03-28T19:28:38.474 INFO:tasks.workunit.client.0.smithi027.stdout:v0.46 2024-03-28T19:28:38.474 INFO:tasks.workunit.client.0.smithi027.stderr:+ ver=v0.46 2024-03-28T19:28:38.474 INFO:tasks.workunit.client.0.smithi027.stderr:+ echo v0.46 2024-03-28T19:28:38.474 INFO:tasks.workunit.client.0.smithi027.stderr:+ git reset --hard v0.46 2024-03-28T19:28:39.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.538+0000 7fe9eb200640 1 Processor -- start 2024-03-28T19:28:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.538+0000 7fe9eb200640 1 -- start start 2024-03-28T19:28:39.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.540+0000 7fe9eb200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9ec0c58c0 0x7fe9ec0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:39.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.540+0000 7fe9eb200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ec158bd0 con 0x7fe9ec0be5d0 2024-03-28T19:28:39.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.540+0000 7fe9eb200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ec158eb0 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.540+0000 7fe9eb200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9ec159190 con 0x7fe9ec0c58c0 2024-03-28T19:28:39.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.540+0000 7fe9e9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9ec0c58c0 0x7fe9ec0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:39.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.540+0000 7fe9e9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9ec0c58c0 0x7fe9ec0c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44904/0 (socket says 172.21.15.27:44904) 2024-03-28T19:28:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.540+0000 7fe9e9e00640 1 -- 172.21.15.27:0/1090174820 learned_addr learned my addr 172.21.15.27:0/1090174820 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.540+0000 7fe9ea800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe9ec0be5d0 0x7fe9ec155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54468/0 (socket says 172.21.15.27:54468) 2024-03-28T19:28:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.541+0000 7fe9e9e00640 1 -- 172.21.15.27:0/1090174820 >> v1:172.21.15.155:6789/0 conn(0x7fe9ec0c8ed0 legacy=0x7fe9ec1555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.541+0000 7fe9e9e00640 1 -- 172.21.15.27:0/1090174820 >> v1:172.21.15.27:6789/0 conn(0x7fe9ec0be5d0 legacy=0x7fe9ec155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.541+0000 7fe9e9e00640 1 -- 172.21.15.27:0/1090174820 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec159470 con 0x7fe9ec0c58c0 2024-03-28T19:28:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.542+0000 7fe9e9e00640 1 --2- 172.21.15.27:0/1090174820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c58c0 0x7fe9ec0c5ca0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009870 tx=0x7fe9dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abc378daad62689 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9e8a00640 1 -- 172.21.15.27:0/1090174820 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc035020 con 0x7fe9ec0c58c0 2024-03-28T19:28:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9e8a00640 1 -- 172.21.15.27:0/1090174820 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc031870 con 0x7fe9ec0c58c0 2024-03-28T19:28:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9e8a00640 1 -- 172.21.15.27:0/1090174820 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc031c40 con 0x7fe9ec0c58c0 2024-03-28T19:28:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9eb200640 1 -- 172.21.15.27:0/1090174820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c58c0 msgr2=0x7fe9ec0c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9eb200640 1 --2- 172.21.15.27:0/1090174820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c58c0 0x7fe9ec0c5ca0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009870 tx=0x7fe9dc02b910 comp rx=0 tx=0).stop 2024-03-28T19:28:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9eb200640 1 -- 172.21.15.27:0/1090174820 shutdown_connections 2024-03-28T19:28:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9eb200640 1 --2- 172.21.15.27:0/1090174820 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c58c0 0x7fe9ec0c5ca0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9eb200640 1 -- 172.21.15.27:0/1090174820 >> 172.21.15.27:0/1090174820 conn(0x7fe9ec06b6f0 msgr2=0x7fe9ec0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.543+0000 7fe9eb200640 1 -- 172.21.15.27:0/1090174820 shutdown_connections 2024-03-28T19:28:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.544+0000 7fe9eb200640 1 -- 172.21.15.27:0/1090174820 wait complete. 2024-03-28T19:28:39.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.546+0000 7fe9eb200640 1 Processor -- start 2024-03-28T19:28:39.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.546+0000 7fe9eb200640 1 -- start start 2024-03-28T19:28:39.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9eb200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9ec0be5d0 0x7fe9ec152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:39.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9eb200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec0c58c0 0x7fe9ec155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:39.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9eb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c8ed0 0x7fe9ec0c2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:39.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9eb200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec0b09a0 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9eb200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec0b0820 con 0x7fe9ec0c58c0 2024-03-28T19:28:39.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9eb200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec0ad7d0 con 0x7fe9ec0be5d0 2024-03-28T19:28:39.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9ea800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c8ed0 0x7fe9ec0c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:39.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9ea800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c8ed0 0x7fe9ec0c2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44912/0 (socket says 172.21.15.27:44912) 2024-03-28T19:28:39.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.547+0000 7fe9ea800640 1 -- 172.21.15.27:0/3299689901 learned_addr learned my addr 172.21.15.27:0/3299689901 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:39.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.548+0000 7fe9e9e00640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9ec0be5d0 0x7fe9ec152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.548+0000 7fe9e9400640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec0c58c0 0x7fe9ec155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.548+0000 7fe9ea800640 1 -- 172.21.15.27:0/3299689901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9ec0be5d0 msgr2=0x7fe9ec152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.548+0000 7fe9ea800640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9ec0be5d0 0x7fe9ec152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.548+0000 7fe9ea800640 1 -- 172.21.15.27:0/3299689901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec0c58c0 msgr2=0x7fe9ec155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.548+0000 7fe9ea800640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec0c58c0 0x7fe9ec155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.548+0000 7fe9ea800640 1 -- 172.21.15.27:0/3299689901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec0c4e30 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.549+0000 7fe9ea800640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c8ed0 0x7fe9ec0c2520 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400c780 tx=0x7fe9e400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:39.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.550+0000 7fe9e2000640 1 -- 172.21.15.27:0/3299689901 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4017070 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.550+0000 7fe9e2000640 1 -- 172.21.15.27:0/3299689901 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e400a640 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.550+0000 7fe9e2000640 1 -- 172.21.15.27:0/3299689901 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e400aa30 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.550+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec1eb6f0 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.550+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec1ebc90 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.551+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec04f400 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.552+0000 7fe9e2000640 1 -- 172.21.15.27:0/3299689901 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe9e401a070 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.553+0000 7fe9e2000640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9cc06bc20 0x7fe9cc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:39.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.554+0000 7fe9e2000640 1 -- 172.21.15.27:0/3299689901 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 10421+0+0 (secure 0 0 0) 0x7fe9e408b1d0 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.554+0000 7fe9e9e00640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9cc06bc20 0x7fe9cc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:39.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.555+0000 7fe9e9e00640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9cc06bc20 0x7fe9cc06e0e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc036000 tx=0x7fe9dc03c040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:39.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.559+0000 7fe9e2000640 1 -- 172.21.15.27:0/3299689901 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe9ec04f400 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.861+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9ec156580 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.862+0000 7fe9e2000640 1 -- 172.21.15.27:0/3299689901 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+6263 (secure 0 0 0) 0x7fe9ec0c4e30 con 0x7fe9ec0c8ed0 2024-03-28T19:28:39.861 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:39.861 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246},{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:27:46.906340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9030":{"gid":9030,"name":"d","rank":0,"incarnation":236,"state":"up:active","state_seq":10,"addr":"172.21.15.155:6834/1152401013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":1152401013},{"type":"v1","addr":"172.21.15.155:6834","nonce":1152401013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9030,"qdb_cluster":[9030]},"id":2}]} 2024-03-28T19:28:39.862 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 246 2024-03-28T19:28:39.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9cc06bc20 msgr2=0x7fe9cc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:39.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9cc06bc20 0x7fe9cc06e0e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc036000 tx=0x7fe9dc03c040 comp rx=0 tx=0).stop 2024-03-28T19:28:39.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c8ed0 msgr2=0x7fe9ec0c2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:39.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c8ed0 0x7fe9ec0c2520 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400c780 tx=0x7fe9e400cc40 comp rx=0 tx=0).stop 2024-03-28T19:28:39.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 shutdown_connections 2024-03-28T19:28:39.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fe9cc06bc20 0x7fe9cc06e0e0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:39.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9ec0c8ed0 0x7fe9ec0c2520 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:39.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec0c58c0 0x7fe9ec155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:39.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 --2- 172.21.15.27:0/3299689901 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe9ec0be5d0 0x7fe9ec152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:39.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 >> 172.21.15.27:0/3299689901 conn(0x7fe9ec06b6f0 msgr2=0x7fe9ec157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:39.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 shutdown_connections 2024-03-28T19:28:39.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:39.865+0000 7fe9eb200640 1 -- 172.21.15.27:0/3299689901 wait complete. 2024-03-28T19:28:39.886 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-03-28T19:28:41.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.124+0000 7fbffec00640 1 Processor -- start 2024-03-28T19:28:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.124+0000 7fbffec00640 1 -- start start 2024-03-28T19:28:42.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.126+0000 7fbffec00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbff8150c70 0x7fbff80a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.126+0000 7fbffec00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c3690 0x7fbff80c6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.126+0000 7fbffec00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff80c3af0 con 0x7fbff80c6660 2024-03-28T19:28:42.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.126+0000 7fbffec00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff80ab4e0 con 0x7fbff8150c70 2024-03-28T19:28:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.126+0000 7fbffec00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80ac1f0 con 0x7fbff80c3690 2024-03-28T19:28:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.127+0000 7fbffcc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbff80c6660 0x7fbff81534c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54474/0 (socket says 172.21.15.27:54474) 2024-03-28T19:28:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.127+0000 7fbffcc00640 1 -- 172.21.15.27:0/1065979702 learned_addr learned my addr 172.21.15.27:0/1065979702 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.127+0000 7fbff7e00640 1 --2- 172.21.15.27:0/1065979702 >> v2:172.21.15.27:3300/0 conn(0x7fbff8150c70 0x7fbff80a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.127+0000 7fbff7400640 1 --2- 172.21.15.27:0/1065979702 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c3690 0x7fbff80c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.128+0000 7fbff6a00640 1 -- 172.21.15.27:0/1065979702 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086288874 0 0) 0x7fbff80c3af0 con 0x7fbff80c6660 2024-03-28T19:28:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.128+0000 7fbff6a00640 1 -- 172.21.15.27:0/1065979702 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfdc003520 con 0x7fbff80c6660 2024-03-28T19:28:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.128+0000 7fbff7e00640 1 -- 172.21.15.27:0/1065979702 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c3690 msgr2=0x7fbff80c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.128+0000 7fbff7e00640 1 --2- 172.21.15.27:0/1065979702 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c3690 0x7fbff80c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.128+0000 7fbff7e00640 1 -- 172.21.15.27:0/1065979702 >> v1:172.21.15.27:6789/0 conn(0x7fbff80c6660 legacy=0x7fbff81534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.128+0000 7fbff7e00640 1 -- 172.21.15.27:0/1065979702 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff80c3d90 con 0x7fbff8150c70 2024-03-28T19:28:42.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.130+0000 7fbff7e00640 1 --2- 172.21.15.27:0/1065979702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a9cb0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80097a0 tx=0x7fbfe802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbd010f502d9d23 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:42.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.130+0000 7fbff6a00640 1 -- 172.21.15.27:0/1065979702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80315a0 con 0x7fbff8150c70 2024-03-28T19:28:42.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.130+0000 7fbff6a00640 1 -- 172.21.15.27:0/1065979702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe8037070 con 0x7fbff8150c70 2024-03-28T19:28:42.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.130+0000 7fbff6a00640 1 -- 172.21.15.27:0/1065979702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8031940 con 0x7fbff8150c70 2024-03-28T19:28:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.131+0000 7fbffec00640 1 -- 172.21.15.27:0/1065979702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 msgr2=0x7fbff80a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.131+0000 7fbffec00640 1 --2- 172.21.15.27:0/1065979702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a9cb0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80097a0 tx=0x7fbfe802b840 comp rx=0 tx=0).stop 2024-03-28T19:28:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.132+0000 7fbffec00640 1 -- 172.21.15.27:0/1065979702 shutdown_connections 2024-03-28T19:28:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.132+0000 7fbffec00640 1 --2- 172.21.15.27:0/1065979702 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c3690 0x7fbff80c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.132+0000 7fbffec00640 1 --2- 172.21.15.27:0/1065979702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a9cb0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.132+0000 7fbffec00640 1 -- 172.21.15.27:0/1065979702 >> 172.21.15.27:0/1065979702 conn(0x7fbff806b9d0 msgr2=0x7fbff80c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.132+0000 7fbffec00640 1 -- 172.21.15.27:0/1065979702 shutdown_connections 2024-03-28T19:28:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.132+0000 7fbffec00640 1 -- 172.21.15.27:0/1065979702 wait complete. 2024-03-28T19:28:42.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.135+0000 7fbffec00640 1 Processor -- start 2024-03-28T19:28:42.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.135+0000 7fbffec00640 1 -- start start 2024-03-28T19:28:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.136+0000 7fbffec00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbff80c3690 0x7fbff81f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.136+0000 7fbffec00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c6660 0x7fbff80a9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.136+0000 7fbffec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.136+0000 7fbffec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80abcd0 con 0x7fbff8150c70 2024-03-28T19:28:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.136+0000 7fbffec00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80ab9f0 con 0x7fbff80c3690 2024-03-28T19:28:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.136+0000 7fbffec00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80c1810 con 0x7fbff80c6660 2024-03-28T19:28:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbffcc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbffcc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44940/0 (socket says 172.21.15.27:44940) 2024-03-28T19:28:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbffcc00640 1 -- 172.21.15.27:0/1346520111 learned_addr learned my addr 172.21.15.27:0/1346520111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbff7400640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c6660 0x7fbff80a9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbff7e00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbff80c3690 0x7fbff81f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbffcc00640 1 -- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c6660 msgr2=0x7fbff80a9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbffcc00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c6660 0x7fbff80a9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbffcc00640 1 -- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbff80c3690 msgr2=0x7fbff81f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbffcc00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbff80c3690 0x7fbff81f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbffcc00640 1 -- 172.21.15.27:0/1346520111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff80a6500 con 0x7fbff8150c70 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.137+0000 7fbff7e00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbff80c3690 0x7fbff81f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.138+0000 7fbffcc00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a6cc0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fbff000cb40 tx=0x7fbff000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.138+0000 7fbfefe00640 1 -- 172.21.15.27:0/1346520111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0017070 con 0x7fbff8150c70 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.138+0000 7fbfefe00640 1 -- 172.21.15.27:0/1346520111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff000aca0 con 0x7fbff8150c70 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.138+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff81eff80 con 0x7fbff8150c70 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.138+0000 7fbfefe00640 1 -- 172.21.15.27:0/1346520111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0012620 con 0x7fbff8150c70 2024-03-28T19:28:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.139+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff81f0260 con 0x7fbff8150c70 2024-03-28T19:28:42.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.141+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff804f400 con 0x7fbff8150c70 2024-03-28T19:28:42.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.143+0000 7fbfefe00640 1 -- 172.21.15.27:0/1346520111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbff001a070 con 0x7fbff8150c70 2024-03-28T19:28:42.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.144+0000 7fbfefe00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbfd806bc00 0x7fbfd806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.144+0000 7fbfefe00640 1 -- 172.21.15.27:0/1346520111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7fbff008ba20 con 0x7fbff8150c70 2024-03-28T19:28:42.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.145+0000 7fbff7e00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbfd806bc00 0x7fbfd806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.147+0000 7fbff7e00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbfd806bc00 0x7fbfd806e0c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8036000 tx=0x7fbfe8008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:42.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.155+0000 7fbfefe00640 1 -- 172.21.15.27:0/1346520111 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbff0059290 con 0x7fbff8150c70 2024-03-28T19:28:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.570+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbff80c5f00 con 0x7fbff8150c70 2024-03-28T19:28:42.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.570+0000 7fbfefe00640 1 -- 172.21.15.27:0/1346520111 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fbff0058b90 con 0x7fbff8150c70 2024-03-28T19:28:42.570 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:42.570 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:40.555353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:42.570 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 247 2024-03-28T19:28:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbfd806bc00 msgr2=0x7fbfd806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbfd806bc00 0x7fbfd806e0c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8036000 tx=0x7fbfe8008030 comp rx=0 tx=0).stop 2024-03-28T19:28:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 msgr2=0x7fbff80a6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a6cc0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fbff000cb40 tx=0x7fbff000a060 comp rx=0 tx=0).stop 2024-03-28T19:28:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 shutdown_connections 2024-03-28T19:28:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fbfd806bc00 0x7fbfd806e0c0 secure :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8036000 tx=0x7fbfe8008030 comp rx=0 tx=0).stop 2024-03-28T19:28:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8150c70 0x7fbff80a6cc0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbff80c6660 0x7fbff80a9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 --2- 172.21.15.27:0/1346520111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbff80c3690 0x7fbff81f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 >> 172.21.15.27:0/1346520111 conn(0x7fbff806b9d0 msgr2=0x7fbff806df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 shutdown_connections 2024-03-28T19:28:42.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.574+0000 7fbffec00640 1 -- 172.21.15.27:0/1346520111 wait complete. 2024-03-28T19:28:42.592 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-03-28T19:28:42.592 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-03-28T19:28:42.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:42.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.815+0000 7f951b200640 1 Processor -- start 2024-03-28T19:28:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.816+0000 7f951b200640 1 -- start start 2024-03-28T19:28:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.817+0000 7f951b200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f95140c5e80 0x7f95140c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.817+0000 7f951b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95140abf60 0x7f95140aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.817+0000 7f951b200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951406d8a0 con 0x7f95140c1f50 2024-03-28T19:28:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.817+0000 7f951b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951406d650 con 0x7f95140abf60 2024-03-28T19:28:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.817+0000 7f951b200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951406e1d0 con 0x7f95140c5e80 2024-03-28T19:28:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.818+0000 7f9513400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f95140c1f50 0x7f95140ab850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54476/0 (socket says 172.21.15.27:54476) 2024-03-28T19:28:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.818+0000 7f9513400640 1 -- 172.21.15.27:0/1021011572 learned_addr learned my addr 172.21.15.27:0/1021011572 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.818+0000 7f9519200640 1 --2- 172.21.15.27:0/1021011572 >> v2:172.21.15.27:3300/0 conn(0x7f95140abf60 0x7f95140aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.818+0000 7f9513e00640 1 --2- 172.21.15.27:0/1021011572 >> v2:172.21.15.155:3300/0 conn(0x7f95140c5e80 0x7f95140c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.819+0000 7f9519200640 1 -- 172.21.15.27:0/1021011572 >> v2:172.21.15.155:3300/0 conn(0x7f95140c5e80 msgr2=0x7f95140c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:42.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.819+0000 7f9519200640 1 --2- 172.21.15.27:0/1021011572 >> v2:172.21.15.155:3300/0 conn(0x7f95140c5e80 0x7f95140c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.819+0000 7f9519200640 1 -- 172.21.15.27:0/1021011572 >> v1:172.21.15.27:6789/0 conn(0x7f95140c1f50 legacy=0x7f95140ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.819+0000 7f9519200640 1 -- 172.21.15.27:0/1021011572 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951406db80 con 0x7f95140abf60 2024-03-28T19:28:42.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.820+0000 7f9519200640 1 --2- 172.21.15.27:0/1021011572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140abf60 0x7f95140aa570 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f95140c6670 tx=0x7f950c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77dfc1f22721b14 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:42.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.821+0000 7f9512a00640 1 -- 172.21.15.27:0/1021011572 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c039070 con 0x7f95140abf60 2024-03-28T19:28:42.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.821+0000 7f9512a00640 1 -- 172.21.15.27:0/1021011572 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950c02cb70 con 0x7f95140abf60 2024-03-28T19:28:42.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.821+0000 7f9512a00640 1 -- 172.21.15.27:0/1021011572 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c034440 con 0x7f95140abf60 2024-03-28T19:28:42.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.822+0000 7f951b200640 1 -- 172.21.15.27:0/1021011572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140abf60 msgr2=0x7f95140aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.822+0000 7f951b200640 1 --2- 172.21.15.27:0/1021011572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140abf60 0x7f95140aa570 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f95140c6670 tx=0x7f950c02bdc0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.823+0000 7f951b200640 1 -- 172.21.15.27:0/1021011572 shutdown_connections 2024-03-28T19:28:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.823+0000 7f951b200640 1 --2- 172.21.15.27:0/1021011572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140abf60 0x7f95140aa570 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.823+0000 7f951b200640 1 --2- 172.21.15.27:0/1021011572 >> v2:172.21.15.155:3300/0 conn(0x7f95140c5e80 0x7f95140c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.823+0000 7f951b200640 1 -- 172.21.15.27:0/1021011572 >> 172.21.15.27:0/1021011572 conn(0x7f951406b9d0 msgr2=0x7f95140c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.823+0000 7f951b200640 1 -- 172.21.15.27:0/1021011572 shutdown_connections 2024-03-28T19:28:42.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.823+0000 7f951b200640 1 -- 172.21.15.27:0/1021011572 wait complete. 2024-03-28T19:28:42.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.826+0000 7f951b200640 1 Processor -- start 2024-03-28T19:28:42.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.826+0000 7f951b200640 1 -- start start 2024-03-28T19:28:42.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.827+0000 7f951b200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f95140abf60 0x7f9514155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.827+0000 7f951b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95140c1f50 0x7f95140a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.827+0000 7f951b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140c5e80 0x7f95140c0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.828+0000 7f951b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95140c3370 con 0x7f95140c5e80 2024-03-28T19:28:42.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.828+0000 7f951b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95140c31f0 con 0x7f95140c1f50 2024-03-28T19:28:42.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.828+0000 7f951b200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95140c6260 con 0x7f95140abf60 2024-03-28T19:28:42.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.828+0000 7f9519200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140c5e80 0x7f95140c0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.828+0000 7f9519200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140c5e80 0x7f95140c0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44950/0 (socket says 172.21.15.27:44950) 2024-03-28T19:28:42.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.828+0000 7f9519200640 1 -- 172.21.15.27:0/579528841 learned_addr learned my addr 172.21.15.27:0/579528841 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:42.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.828+0000 7f9513400640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95140c1f50 0x7f95140a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.828+0000 7f9513e00640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f95140abf60 0x7f9514155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.829+0000 7f9519200640 1 -- 172.21.15.27:0/579528841 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f95140abf60 msgr2=0x7f9514155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.829+0000 7f9519200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f95140abf60 0x7f9514155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.829+0000 7f9519200640 1 -- 172.21.15.27:0/579528841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95140c1f50 msgr2=0x7f95140a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:42.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.829+0000 7f9519200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95140c1f50 0x7f95140a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:42.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.829+0000 7f9519200640 1 -- 172.21.15.27:0/579528841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95140bdc60 con 0x7f95140c5e80 2024-03-28T19:28:42.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.829+0000 7f9519200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140c5e80 0x7f95140c0740 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f950c009e50 tx=0x7f950c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:42.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.830+0000 7f950be00640 1 -- 172.21.15.27:0/579528841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c044020 con 0x7f95140c5e80 2024-03-28T19:28:42.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.830+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95140bc5d0 con 0x7f95140c5e80 2024-03-28T19:28:42.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.830+0000 7f950be00640 1 -- 172.21.15.27:0/579528841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950c03f470 con 0x7f95140c5e80 2024-03-28T19:28:42.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.830+0000 7f950be00640 1 -- 172.21.15.27:0/579528841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c039040 con 0x7f95140c5e80 2024-03-28T19:28:42.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.830+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95140bcb90 con 0x7f95140c5e80 2024-03-28T19:28:42.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.832+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951404f400 con 0x7f95140c5e80 2024-03-28T19:28:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.834+0000 7f950be00640 1 -- 172.21.15.27:0/579528841 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f950c047020 con 0x7f95140c5e80 2024-03-28T19:28:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.835+0000 7f950be00640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94f406bc00 0x7f94f406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.835+0000 7f950be00640 1 -- 172.21.15.27:0/579528841 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7f950c0ae660 con 0x7f95140c5e80 2024-03-28T19:28:42.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.835+0000 7f9513e00640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94f406bc00 0x7f94f406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:42.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.838+0000 7f9513e00640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94f406bc00 0x7f94f406e0c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9504006fd0 tx=0x7f9504008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:42.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:42.844+0000 7f950be00640 1 -- 172.21.15.27:0/579528841 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f950c04e070 con 0x7f95140c5e80 2024-03-28T19:28:43.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.248+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95140c7930 con 0x7f95140c5e80 2024-03-28T19:28:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.249+0000 7f950be00640 1 -- 172.21.15.27:0/579528841 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f950c07bdb0 con 0x7f95140c5e80 2024-03-28T19:28:43.248 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:43.248 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:40.555353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:43.248 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 247 2024-03-28T19:28:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.252+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94f406bc00 msgr2=0x7f94f406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.252+0000 7f951b200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94f406bc00 0x7f94f406e0c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9504006fd0 tx=0x7f9504008040 comp rx=0 tx=0).stop 2024-03-28T19:28:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.252+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140c5e80 msgr2=0x7f95140c0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.252+0000 7f951b200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140c5e80 0x7f95140c0740 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f950c009e50 tx=0x7f950c004a40 comp rx=0 tx=0).stop 2024-03-28T19:28:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.253+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 shutdown_connections 2024-03-28T19:28:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.253+0000 7f951b200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f94f406bc00 0x7f94f406e0c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:43.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.253+0000 7f951b200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95140c5e80 0x7f95140c0740 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.253+0000 7f951b200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95140c1f50 0x7f95140a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.253+0000 7f951b200640 1 --2- 172.21.15.27:0/579528841 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f95140abf60 0x7f9514155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.253+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 >> 172.21.15.27:0/579528841 conn(0x7f951406b9d0 msgr2=0x7f95141547c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.253+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 shutdown_connections 2024-03-28T19:28:43.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:43.253+0000 7f951b200640 1 -- 172.21.15.27:0/579528841 wait complete. 2024-03-28T19:28:43.275 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 247, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:33.955573+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 245, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_9156": { "addr": "172.21.15.27:6837/3692255388", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6836", "nonce": 3692255388, "type": "v2" }, { "addr": "172.21.15.27:6837", "nonce": 3692255388, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9156, "incarnation": 241, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 95, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-28T19:28:20.108820+0000", "qdb_cluster": [ 9156 ], "qdb_leader": 9156, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9156 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-28T18:55:45.064332+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 247, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8913": { "addr": "172.21.15.27:6835/796801387", "addrs": { "addrvec": [ { "addr": "172.21.15.27:6834", "nonce": 796801387, "type": "v2" }, { "addr": "172.21.15.27:6835", "nonce": 796801387, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8913, "incarnation": 247, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 96, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-03-28T19:28:40.555353+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8913 } } } ], "standbys": [ { "addr": "172.21.15.155:6833/3956913260", "addrs": { "addrvec": [ { "addr": "172.21.15.155:6832", "nonce": 3956913260, "type": "v2" }, { "addr": "172.21.15.155:6833", "nonce": 3956913260, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 246, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7082, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-28T19:28:43.870 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:44.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.112+0000 7fc4b9a00640 1 Processor -- start 2024-03-28T19:28:44.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.112+0000 7fc4b9a00640 1 -- start start 2024-03-28T19:28:44.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.113+0000 7fc4b9a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4b4152d10 0x7fc4b40c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.114+0000 7fc4b9a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b41530f0 con 0x7fc4b41562f0 2024-03-28T19:28:44.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.114+0000 7fc4b9a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b406d3d0 con 0x7fc4b40c8e10 2024-03-28T19:28:44.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.114+0000 7fc4b9a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b40c92f0 con 0x7fc4b4152d10 2024-03-28T19:28:44.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.114+0000 7fc4b3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc4b41562f0 0x7fc4b415ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54488/0 (socket says 172.21.15.27:54488) 2024-03-28T19:28:44.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.114+0000 7fc4b3400640 1 -- 172.21.15.27:0/2321270867 learned_addr learned my addr 172.21.15.27:0/2321270867 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:44.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.114+0000 7fc4b2a00640 1 --2- 172.21.15.27:0/2321270867 >> v2:172.21.15.27:3300/0 conn(0x7fc4b4152d10 0x7fc4b40c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.115+0000 7fc4b1600640 1 -- 172.21.15.27:0/2321270867 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364743568 0 0) 0x7fc4ac001490 con 0x7fc4b41562f0 2024-03-28T19:28:44.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.115+0000 7fc4b1600640 1 -- 172.21.15.27:0/2321270867 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc498003520 con 0x7fc4b41562f0 2024-03-28T19:28:44.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.115+0000 7fc4b2a00640 1 -- 172.21.15.27:0/2321270867 >> v1:172.21.15.155:6789/0 conn(0x7fc4b40c8e10 legacy=0x7fc4b4158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.115+0000 7fc4b2a00640 1 -- 172.21.15.27:0/2321270867 >> v1:172.21.15.27:6789/0 conn(0x7fc4b41562f0 legacy=0x7fc4b415ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.115+0000 7fc4b2a00640 1 -- 172.21.15.27:0/2321270867 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4155810 con 0x7fc4b4152d10 2024-03-28T19:28:44.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.117+0000 7fc4b2a00640 1 --2- 172.21.15.27:0/2321270867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b40c5890 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40097a0 tx=0x7fc4a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fef52ba929d0ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:44.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.117+0000 7fc4b1600640 1 -- 172.21.15.27:0/2321270867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40315a0 con 0x7fc4b4152d10 2024-03-28T19:28:44.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.117+0000 7fc4b1600640 1 -- 172.21.15.27:0/2321270867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a4037070 con 0x7fc4b4152d10 2024-03-28T19:28:44.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.117+0000 7fc4b1600640 1 -- 172.21.15.27:0/2321270867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4031960 con 0x7fc4b4152d10 2024-03-28T19:28:44.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.118+0000 7fc4b9a00640 1 -- 172.21.15.27:0/2321270867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 msgr2=0x7fc4b40c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.118+0000 7fc4b9a00640 1 --2- 172.21.15.27:0/2321270867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b40c5890 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40097a0 tx=0x7fc4a402b840 comp rx=0 tx=0).stop 2024-03-28T19:28:44.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.118+0000 7fc4b9a00640 1 -- 172.21.15.27:0/2321270867 shutdown_connections 2024-03-28T19:28:44.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.118+0000 7fc4b9a00640 1 --2- 172.21.15.27:0/2321270867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b40c5890 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.118+0000 7fc4b9a00640 1 -- 172.21.15.27:0/2321270867 >> 172.21.15.27:0/2321270867 conn(0x7fc4b4066fb0 msgr2=0x7fc4b40673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:44.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.118+0000 7fc4b9a00640 1 -- 172.21.15.27:0/2321270867 shutdown_connections 2024-03-28T19:28:44.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.118+0000 7fc4b9a00640 1 -- 172.21.15.27:0/2321270867 wait complete. 2024-03-28T19:28:44.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.120+0000 7fc4b9a00640 1 Processor -- start 2024-03-28T19:28:44.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.121+0000 7fc4b9a00640 1 -- start start 2024-03-28T19:28:44.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.122+0000 7fc4b9a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4b40c8e10 0x7fc4b41dff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.122+0000 7fc4b9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b4157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.122+0000 7fc4b9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4b41562f0 0x7fc4b41ee6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.122+0000 7fc4b9a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41592f0 con 0x7fc4b4152d10 2024-03-28T19:28:44.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.122+0000 7fc4b9a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b406d850 con 0x7fc4b41562f0 2024-03-28T19:28:44.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.122+0000 7fc4b9a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41ebbb0 con 0x7fc4b40c8e10 2024-03-28T19:28:44.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.123+0000 7fc4b2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b4157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.123+0000 7fc4b2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4b40c8e10 0x7fc4b41dff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.123+0000 7fc4b2a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4b40c8e10 0x7fc4b41dff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:44958/0 (socket says 172.21.15.27:44958) 2024-03-28T19:28:44.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.123+0000 7fc4b2a00640 1 -- 172.21.15.27:0/1729638957 learned_addr learned my addr 172.21.15.27:0/1729638957 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:44.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.123+0000 7fc4b2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b4157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44958/0 (socket says 172.21.15.27:44958) 2024-03-28T19:28:44.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.123+0000 7fc4b3400640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4b41562f0 0x7fc4b41ee6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.124+0000 7fc4b2a00640 1 -- 172.21.15.27:0/1729638957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4b41562f0 msgr2=0x7fc4b41ee6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.124+0000 7fc4b2a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4b41562f0 0x7fc4b41ee6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.124+0000 7fc4b2a00640 1 -- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 msgr2=0x7fc4b4157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.124+0000 7fc4b2a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b4157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.124+0000 7fc4b2a00640 1 -- 172.21.15.27:0/1729638957 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b40c18e0 con 0x7fc4b40c8e10 2024-03-28T19:28:44.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.124+0000 7fc4b2000640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b4157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:28:44.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.125+0000 7fc4b2a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4b40c8e10 0x7fc4b41dff40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40098c0 tx=0x7fc4a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:44.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.125+0000 7fc4aaa00640 1 -- 172.21.15.27:0/1729638957 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4032840 con 0x7fc4b40c8e10 2024-03-28T19:28:44.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.126+0000 7fc4aaa00640 1 -- 172.21.15.27:0/1729638957 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a4032a00 con 0x7fc4b40c8e10 2024-03-28T19:28:44.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.126+0000 7fc4aaa00640 1 -- 172.21.15.27:0/1729638957 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4032df0 con 0x7fc4b40c8e10 2024-03-28T19:28:44.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.127+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b41ebe00 con 0x7fc4b40c8e10 2024-03-28T19:28:44.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.128+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b41df600 con 0x7fc4b40c8e10 2024-03-28T19:28:44.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.130+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b404f400 con 0x7fc4b40c8e10 2024-03-28T19:28:44.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.132+0000 7fc4aaa00640 1 -- 172.21.15.27:0/1729638957 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc4a403b3f0 con 0x7fc4b40c8e10 2024-03-28T19:28:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.133+0000 7fc4aaa00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc49406bc20 0x7fc49406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.133+0000 7fc4aaa00640 1 -- 172.21.15.27:0/1729638957 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7fc4a402f030 con 0x7fc4b40c8e10 2024-03-28T19:28:44.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.133+0000 7fc4b2000640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc49406bc20 0x7fc49406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.134+0000 7fc4b2000640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc49406bc20 0x7fc49406e0e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc4b414e930 tx=0x7fc4a0009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.149+0000 7fc4aaa00640 1 -- 172.21.15.27:0/1729638957 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc4a4040070 con 0x7fc4b40c8e10 2024-03-28T19:28:44.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.486+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4b406d630 con 0x7fc4b40c8e10 2024-03-28T19:28:44.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.487+0000 7fc4aaa00640 1 -- 172.21.15.27:0/1729638957 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fc4a4031e90 con 0x7fc4b40c8e10 2024-03-28T19:28:44.487 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:44.487 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:40.555353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:44.487 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 247 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc49406bc20 msgr2=0x7fc49406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc49406bc20 0x7fc49406e0e0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fc4b414e930 tx=0x7fc4a0009040 comp rx=0 tx=0).stop 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4b40c8e10 msgr2=0x7fc4b41dff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4b40c8e10 0x7fc4b41dff40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40098c0 tx=0x7fc4a40046c0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 shutdown_connections 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc49406bc20 0x7fc49406e0e0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4b41562f0 0x7fc4b41ee6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4152d10 0x7fc4b4157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 --2- 172.21.15.27:0/1729638957 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc4b40c8e10 0x7fc4b41dff40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 >> 172.21.15.27:0/1729638957 conn(0x7fc4b4066fb0 msgr2=0x7fc4b4154f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:44.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.490+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 shutdown_connections 2024-03-28T19:28:44.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.491+0000 7fc4b9a00640 1 -- 172.21.15.27:0/1729638957 wait complete. 2024-03-28T19:28:44.508 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-03-28T19:28:44.509 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-28T19:28:44.509 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T19:28:44.551 INFO:tasks.ceph.mds.c:Stopped 2024-03-28T19:28:44.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.807+0000 7efc7a400640 1 Processor -- start 2024-03-28T19:28:44.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.807+0000 7efc7a400640 1 -- start start 2024-03-28T19:28:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.808+0000 7efc7a400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc7406d1f0 0x7efc740c9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc7a400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc741556e0 0x7efc74155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc7a400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc74153120 con 0x7efc74152d40 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc7a400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc741596a0 con 0x7efc741556e0 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc7a400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc740c9640 con 0x7efc7406d1f0 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc73400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efc74152d40 0x7efc740c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54494/0 (socket says 172.21.15.27:54494) 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc73400640 1 -- 172.21.15.27:0/2038882125 learned_addr learned my addr 172.21.15.27:0/2038882125 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc73e00640 1 --2- 172.21.15.27:0/2038882125 >> v2:172.21.15.27:3300/0 conn(0x7efc741556e0 0x7efc74155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc72a00640 1 --2- 172.21.15.27:0/2038882125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc7406d1f0 0x7efc740c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc73e00640 1 -- 172.21.15.27:0/2038882125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc7406d1f0 msgr2=0x7efc740c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc73e00640 1 --2- 172.21.15.27:0/2038882125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc7406d1f0 0x7efc740c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc73e00640 1 -- 172.21.15.27:0/2038882125 >> v1:172.21.15.27:6789/0 conn(0x7efc74152d40 legacy=0x7efc740c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.809+0000 7efc73e00640 1 -- 172.21.15.27:0/2038882125 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74158d70 con 0x7efc741556e0 2024-03-28T19:28:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.811+0000 7efc73e00640 1 --2- 172.21.15.27:0/2038882125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc74155b70 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0097a0 tx=0x7efc6c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4216156de75a8489 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:44.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.811+0000 7efc72000640 1 -- 172.21.15.27:0/2038882125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c035020 con 0x7efc741556e0 2024-03-28T19:28:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.812+0000 7efc72000640 1 -- 172.21.15.27:0/2038882125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c031810 con 0x7efc741556e0 2024-03-28T19:28:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.812+0000 7efc72000640 1 -- 172.21.15.27:0/2038882125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c031be0 con 0x7efc741556e0 2024-03-28T19:28:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.812+0000 7efc7a400640 1 -- 172.21.15.27:0/2038882125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 msgr2=0x7efc74155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.812+0000 7efc7a400640 1 --2- 172.21.15.27:0/2038882125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc74155b70 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0097a0 tx=0x7efc6c02b8b0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.812+0000 7efc7a400640 1 -- 172.21.15.27:0/2038882125 shutdown_connections 2024-03-28T19:28:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.812+0000 7efc7a400640 1 --2- 172.21.15.27:0/2038882125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc74155b70 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.812+0000 7efc7a400640 1 --2- 172.21.15.27:0/2038882125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc7406d1f0 0x7efc740c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.813+0000 7efc7a400640 1 -- 172.21.15.27:0/2038882125 >> 172.21.15.27:0/2038882125 conn(0x7efc74066fb0 msgr2=0x7efc740673c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.813+0000 7efc7a400640 1 -- 172.21.15.27:0/2038882125 shutdown_connections 2024-03-28T19:28:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.813+0000 7efc7a400640 1 -- 172.21.15.27:0/2038882125 wait complete. 2024-03-28T19:28:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.814+0000 7efc7a400640 1 Processor -- start 2024-03-28T19:28:44.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.814+0000 7efc7a400640 1 -- start start 2024-03-28T19:28:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.815+0000 7efc7a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc7406d1f0 0x7efc741ebc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.815+0000 7efc7a400640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc74152d40 0x7efc74151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.815+0000 7efc7a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc740c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.815+0000 7efc7a400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74159b30 con 0x7efc741556e0 2024-03-28T19:28:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.815+0000 7efc7a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74159330 con 0x7efc7406d1f0 2024-03-28T19:28:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.815+0000 7efc7a400640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc741560b0 con 0x7efc74152d40 2024-03-28T19:28:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc740c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc740c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44964/0 (socket says 172.21.15.27:44964) 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73e00640 1 -- 172.21.15.27:0/906808881 learned_addr learned my addr 172.21.15.27:0/906808881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc72a00640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc74152d40 0x7efc74151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc7406d1f0 0x7efc741ebc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73e00640 1 -- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc74152d40 msgr2=0x7efc74151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73e00640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc74152d40 0x7efc74151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73e00640 1 -- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc7406d1f0 msgr2=0x7efc741ebc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73e00640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc7406d1f0 0x7efc741ebc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73e00640 1 -- 172.21.15.27:0/906808881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc741486a0 con 0x7efc741556e0 2024-03-28T19:28:44.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.816+0000 7efc73400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc7406d1f0 0x7efc741ebc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:28:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.817+0000 7efc73e00640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc740c4260 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7efc6c034040 tx=0x7efc6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.817+0000 7efc6b400640 1 -- 172.21.15.27:0/906808881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c040070 con 0x7efc741556e0 2024-03-28T19:28:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.817+0000 7efc6b400640 1 -- 172.21.15.27:0/906808881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c039c70 con 0x7efc741556e0 2024-03-28T19:28:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.817+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc74145fe0 con 0x7efc741556e0 2024-03-28T19:28:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.817+0000 7efc6b400640 1 -- 172.21.15.27:0/906808881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c035040 con 0x7efc741556e0 2024-03-28T19:28:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.817+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc74151210 con 0x7efc741556e0 2024-03-28T19:28:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.819+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7404f400 con 0x7efc741556e0 2024-03-28T19:28:44.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.820+0000 7efc6b400640 1 -- 172.21.15.27:0/906808881 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efc6c04b060 con 0x7efc741556e0 2024-03-28T19:28:44.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.821+0000 7efc6b400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc5006bc00 0x7efc5006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:44.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.821+0000 7efc6b400640 1 -- 172.21.15.27:0/906808881 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7efc6c0ad8b0 con 0x7efc741556e0 2024-03-28T19:28:44.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.822+0000 7efc73400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc5006bc00 0x7efc5006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:44.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.822+0000 7efc73400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc5006bc00 0x7efc5006e0c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7efc640012e0 tx=0x7efc64009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:44.830+0000 7efc6b400640 1 -- 172.21.15.27:0/906808881 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efc6c07b0c0 con 0x7efc741556e0 2024-03-28T19:28:45.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.069+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc740c8e40 con 0x7efc741556e0 2024-03-28T19:28:45.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.070+0000 7efc6b400640 1 -- 172.21.15.27:0/906808881 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5460 (secure 0 0 0) 0x7efc6c039e60 con 0x7efc741556e0 2024-03-28T19:28:45.070 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:45.070 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:40.555353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:45.070 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 247 2024-03-28T19:28:45.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc5006bc00 msgr2=0x7efc5006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:45.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc5006bc00 0x7efc5006e0c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7efc640012e0 tx=0x7efc64009040 comp rx=0 tx=0).stop 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 msgr2=0x7efc740c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc740c4260 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7efc6c034040 tx=0x7efc6c0046c0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 shutdown_connections 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7efc5006bc00 0x7efc5006e0c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc741556e0 0x7efc740c4260 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc74152d40 0x7efc74151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 --2- 172.21.15.27:0/906808881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc7406d1f0 0x7efc741ebc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 >> 172.21.15.27:0/906808881 conn(0x7efc74066fb0 msgr2=0x7efc74067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 shutdown_connections 2024-03-28T19:28:45.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.073+0000 7efc7a400640 1 -- 172.21.15.27:0/906808881 wait complete. 2024-03-28T19:28:45.090 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:28:45.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.480+0000 7f75fc200640 1 Processor -- start 2024-03-28T19:28:45.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.480+0000 7f75fc200640 1 -- start start 2024-03-28T19:28:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.482+0000 7f75fc200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75f40a6300 0x7f75f40733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.482+0000 7f75fc200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f4070a50 0x7f75f4155a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.482+0000 7f75fc200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f406d7d0 con 0x7f75f406ceb0 2024-03-28T19:28:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.482+0000 7f75fc200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40c2f00 con 0x7f75f40a6300 2024-03-28T19:28:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.482+0000 7f75fc200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40c31e0 con 0x7f75f4070a50 2024-03-28T19:28:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.482+0000 7f75f9800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f75f406ceb0 0x7f75f40c1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54498/0 (socket says 172.21.15.27:54498) 2024-03-28T19:28:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.482+0000 7f75f9800640 1 -- 172.21.15.27:0/1057658447 learned_addr learned my addr 172.21.15.27:0/1057658447 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.482+0000 7f75f8e00640 1 --2- 172.21.15.27:0/1057658447 >> v2:172.21.15.27:3300/0 conn(0x7f75f40a6300 0x7f75f40733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:45.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.483+0000 7f75fa200640 1 --2- 172.21.15.27:0/1057658447 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f4070a50 0x7f75f4155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:45.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.483+0000 7f75f3e00640 1 -- 172.21.15.27:0/1057658447 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723462571 0 0) 0x7f75f406d7d0 con 0x7f75f406ceb0 2024-03-28T19:28:45.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.483+0000 7f75f3e00640 1 -- 172.21.15.27:0/1057658447 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc003520 con 0x7f75f406ceb0 2024-03-28T19:28:45.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.483+0000 7f75f8e00640 1 -- 172.21.15.27:0/1057658447 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f4070a50 msgr2=0x7f75f4155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:45.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.483+0000 7f75f8e00640 1 --2- 172.21.15.27:0/1057658447 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f4070a50 0x7f75f4155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.483+0000 7f75f8e00640 1 -- 172.21.15.27:0/1057658447 >> v1:172.21.15.27:6789/0 conn(0x7f75f406ceb0 legacy=0x7f75f40c1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:45.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.483+0000 7f75f8e00640 1 -- 172.21.15.27:0/1057658447 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f406dab0 con 0x7f75f40a6300 2024-03-28T19:28:45.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.484+0000 7f75f8e00640 1 --2- 172.21.15.27:0/1057658447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f40733b0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f75e00097a0 tx=0x7f75e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7581128e83d00977 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:45.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.484+0000 7f75f3e00640 1 -- 172.21.15.27:0/1057658447 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e00315a0 con 0x7f75f40a6300 2024-03-28T19:28:45.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.484+0000 7f75f3e00640 1 -- 172.21.15.27:0/1057658447 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e0037070 con 0x7f75f40a6300 2024-03-28T19:28:45.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.484+0000 7f75f3e00640 1 -- 172.21.15.27:0/1057658447 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0031940 con 0x7f75f40a6300 2024-03-28T19:28:45.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.485+0000 7f75fc200640 1 -- 172.21.15.27:0/1057658447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 msgr2=0x7f75f40733b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.485+0000 7f75fc200640 1 --2- 172.21.15.27:0/1057658447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f40733b0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f75e00097a0 tx=0x7f75e002b840 comp rx=0 tx=0).stop 2024-03-28T19:28:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.485+0000 7f75fc200640 1 -- 172.21.15.27:0/1057658447 shutdown_connections 2024-03-28T19:28:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.485+0000 7f75fc200640 1 --2- 172.21.15.27:0/1057658447 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f4070a50 0x7f75f4155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.485+0000 7f75fc200640 1 --2- 172.21.15.27:0/1057658447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f40733b0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.485+0000 7f75fc200640 1 -- 172.21.15.27:0/1057658447 >> 172.21.15.27:0/1057658447 conn(0x7f75f406b9d0 msgr2=0x7f75f40c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.485+0000 7f75fc200640 1 -- 172.21.15.27:0/1057658447 shutdown_connections 2024-03-28T19:28:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.485+0000 7f75fc200640 1 -- 172.21.15.27:0/1057658447 wait complete. 2024-03-28T19:28:45.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.487+0000 7f75fc200640 1 Processor -- start 2024-03-28T19:28:45.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.487+0000 7f75fc200640 1 -- start start 2024-03-28T19:28:45.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fc200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f406ceb0 0x7f75f40a89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:45.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fc200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f4070a50 0x7f75f41e5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:45.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fc200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f41e6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:45.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fc200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40c45b0 con 0x7f75f40a6300 2024-03-28T19:28:45.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fc200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40c4430 con 0x7f75f4070a50 2024-03-28T19:28:45.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fc200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f406d290 con 0x7f75f406ceb0 2024-03-28T19:28:45.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f41e6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f41e6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44990/0 (socket says 172.21.15.27:44990) 2024-03-28T19:28:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75fa200640 1 -- 172.21.15.27:0/2852923907 learned_addr learned my addr 172.21.15.27:0/2852923907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75f9800640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f406ceb0 0x7f75f40a89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:45.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.488+0000 7f75f8e00640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f4070a50 0x7f75f41e5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75fa200640 1 -- 172.21.15.27:0/2852923907 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f406ceb0 msgr2=0x7f75f40a89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75fa200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f406ceb0 0x7f75f40a89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75fa200640 1 -- 172.21.15.27:0/2852923907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f4070a50 msgr2=0x7f75f41e5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75fa200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f4070a50 0x7f75f41e5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75fa200640 1 -- 172.21.15.27:0/2852923907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f41e9170 con 0x7f75f40a6300 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75fa200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f41e6220 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009fd0 tx=0x7f75ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75f1600640 1 -- 172.21.15.27:0/2852923907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec014420 con 0x7f75f40a6300 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75f1600640 1 -- 172.21.15.27:0/2852923907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec019070 con 0x7f75f40a6300 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75f1600640 1 -- 172.21.15.27:0/2852923907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec014800 con 0x7f75f40a6300 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f41ed440 con 0x7f75f40a6300 2024-03-28T19:28:45.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.489+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f41e96d0 con 0x7f75f40a6300 2024-03-28T19:28:45.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.491+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f404f400 con 0x7f75f40a6300 2024-03-28T19:28:45.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.493+0000 7f75f1600640 1 -- 172.21.15.27:0/2852923907 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f75ec0149a0 con 0x7f75f40a6300 2024-03-28T19:28:45.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.494+0000 7f75f1600640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f75d406bbd0 0x7f75d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:45.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.494+0000 7f75f1600640 1 -- 172.21.15.27:0/2852923907 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7f75ec08c5c0 con 0x7f75f40a6300 2024-03-28T19:28:45.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.494+0000 7f75f9800640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f75d406bbd0 0x7f75d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:45.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.495+0000 7f75f9800640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f75d406bbd0 0x7f75d406e090 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f75e8005880 tx=0x7f75e8009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:45.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.502+0000 7f75f1600640 1 -- 172.21.15.27:0/2852923907 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f75f41e9170 con 0x7f75f40a6300 2024-03-28T19:28:45.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.766+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75f4071ad0 con 0x7f75f40a6300 2024-03-28T19:28:45.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.766+0000 7f75f1600640 1 -- 172.21.15.27:0/2852923907 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5460 (secure 0 0 0) 0x7f75ec059dd0 con 0x7f75f40a6300 2024-03-28T19:28:45.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:45.766 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:40.555353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 247 2024-03-28T19:28:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.769+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f75d406bbd0 msgr2=0x7f75d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.769+0000 7f75fc200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f75d406bbd0 0x7f75d406e090 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f75e8005880 tx=0x7f75e8009040 comp rx=0 tx=0).stop 2024-03-28T19:28:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.769+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 msgr2=0x7f75f41e6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.769+0000 7f75fc200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f41e6220 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009fd0 tx=0x7f75ec00e4b0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.770+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 shutdown_connections 2024-03-28T19:28:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.770+0000 7f75fc200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f75d406bbd0 0x7f75d406e090 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.770+0000 7f75fc200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f40a6300 0x7f75f41e6220 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.770+0000 7f75fc200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f4070a50 0x7f75f41e5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.770+0000 7f75fc200640 1 --2- 172.21.15.27:0/2852923907 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f406ceb0 0x7f75f40a89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.770+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 >> 172.21.15.27:0/2852923907 conn(0x7f75f406b9d0 msgr2=0x7f75f4154670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.770+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 shutdown_connections 2024-03-28T19:28:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:45.771+0000 7f75fc200640 1 -- 172.21.15.27:0/2852923907 wait complete. 2024-03-28T19:28:45.786 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:28:47.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:47.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.296+0000 7fecf4a00640 1 Processor -- start 2024-03-28T19:28:47.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.296+0000 7fecf4a00640 1 -- start start 2024-03-28T19:28:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecf4a00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fecf00c8ed0 0x7fecf00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecf4a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00be5d0 0x7fecf0155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecf4a00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf0156230 con 0x7fecf00c58c0 2024-03-28T19:28:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecf4a00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf0158bd0 con 0x7fecf00c8ed0 2024-03-28T19:28:47.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecf4a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf0158eb0 con 0x7fecf00be5d0 2024-03-28T19:28:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecede00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fecf00c8ed0 0x7fecf00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecef200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00be5d0 0x7fecf0155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecee800640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fecf00c58c0 0x7fecf01555e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.27:44560/0 (socket says 172.21.15.27:44560) 2024-03-28T19:28:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecef200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00be5d0 0x7fecf0155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.27:45946/0 (socket says 172.21.15.27:45946) 2024-03-28T19:28:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.298+0000 7fecee800640 1 -- 172.21.15.27:0/669252570 learned_addr learned my addr 172.21.15.27:0/669252570 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.299+0000 7feced400640 1 -- 172.21.15.27:0/669252570 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649378715 0 0) 0x7fecf0156230 con 0x7fecf00c58c0 2024-03-28T19:28:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.299+0000 7feced400640 1 -- 172.21.15.27:0/669252570 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd4003520 con 0x7fecf00c58c0 2024-03-28T19:28:47.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.299+0000 7fecede00640 1 -- 172.21.15.27:0/669252570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00be5d0 msgr2=0x7fecf0155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:47.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.299+0000 7fecede00640 1 --2- 172.21.15.27:0/669252570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00be5d0 0x7fecf0155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.299+0000 7fecede00640 1 -- 172.21.15.27:0/669252570 >> v1:172.21.15.155:6789/0 conn(0x7fecf00c58c0 legacy=0x7fecf01555e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.299+0000 7fecede00640 1 -- 172.21.15.27:0/669252570 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf0159190 con 0x7fecf00c8ed0 2024-03-28T19:28:47.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.299+0000 7fecef200640 1 --2- 172.21.15.27:0/669252570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00be5d0 0x7fecf0155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T19:28:47.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.301+0000 7fecede00640 1 --2- 172.21.15.27:0/669252570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c8ed0 0x7fecf00c5ca0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fecdc0097a0 tx=0x7fecdc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b6aa7e560c15364 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.302+0000 7feced400640 1 -- 172.21.15.27:0/669252570 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc031780 con 0x7fecf00c8ed0 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.302+0000 7feced400640 1 -- 172.21.15.27:0/669252570 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc037070 con 0x7fecf00c8ed0 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.302+0000 7feced400640 1 -- 172.21.15.27:0/669252570 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc031b20 con 0x7fecf00c8ed0 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.302+0000 7fecf4a00640 1 -- 172.21.15.27:0/669252570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c8ed0 msgr2=0x7fecf00c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.302+0000 7fecf4a00640 1 --2- 172.21.15.27:0/669252570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c8ed0 0x7fecf00c5ca0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fecdc0097a0 tx=0x7fecdc02b820 comp rx=0 tx=0).stop 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.302+0000 7fecf4a00640 1 -- 172.21.15.27:0/669252570 shutdown_connections 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.302+0000 7fecf4a00640 1 --2- 172.21.15.27:0/669252570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00be5d0 0x7fecf0155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.302+0000 7fecf4a00640 1 --2- 172.21.15.27:0/669252570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c8ed0 0x7fecf00c5ca0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.303+0000 7fecf4a00640 1 -- 172.21.15.27:0/669252570 >> 172.21.15.27:0/669252570 conn(0x7fecf006b6f0 msgr2=0x7fecf00acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:47.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.303+0000 7fecf4a00640 1 -- 172.21.15.27:0/669252570 shutdown_connections 2024-03-28T19:28:47.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.303+0000 7fecf4a00640 1 -- 172.21.15.27:0/669252570 wait complete. 2024-03-28T19:28:47.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.304+0000 7fecf4a00640 1 Processor -- start 2024-03-28T19:28:47.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.304+0000 7fecf4a00640 1 -- start start 2024-03-28T19:28:47.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.305+0000 7fecf4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00be5d0 0x7fecf0151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.305+0000 7fecf4a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c58c0 0x7fecf01524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.305+0000 7fecf4a00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00c8ed0 0x7fecf0153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.305+0000 7fecf4a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf00b0820 con 0x7fecf00be5d0 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.305+0000 7fecf4a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf0159750 con 0x7fecf00c58c0 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.305+0000 7fecf4a00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf00b09a0 con 0x7fecf00c8ed0 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecee800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00be5d0 0x7fecf0151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecee800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00be5d0 0x7fecf0151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50096/0 (socket says 172.21.15.27:50096) 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecee800640 1 -- 172.21.15.27:0/14339659 learned_addr learned my addr 172.21.15.27:0/14339659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecef200640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00c8ed0 0x7fecf0153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecede00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c58c0 0x7fecf01524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecee800640 1 -- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00c8ed0 msgr2=0x7fecf0153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecee800640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00c8ed0 0x7fecf0153f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecee800640 1 -- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c58c0 msgr2=0x7fecf01524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecee800640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c58c0 0x7fecf01524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecee800640 1 -- 172.21.15.27:0/14339659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf00c4c20 con 0x7fecf00be5d0 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecef200640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00c8ed0 0x7fecf0153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.306+0000 7fecede00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c58c0 0x7fecf01524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.307+0000 7fecee800640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00be5d0 0x7fecf0151fa0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fece000c410 tx=0x7fece000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.307+0000 7fece6a00640 1 -- 172.21.15.27:0/14339659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0017070 con 0x7fecf00be5d0 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.307+0000 7fece6a00640 1 -- 172.21.15.27:0/14339659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece0005ce0 con 0x7fecf00be5d0 2024-03-28T19:28:47.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.307+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf0152a20 con 0x7fecf00be5d0 2024-03-28T19:28:47.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.307+0000 7fece6a00640 1 -- 172.21.15.27:0/14339659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece00072f0 con 0x7fecf00be5d0 2024-03-28T19:28:47.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.307+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf00c23e0 con 0x7fecf00be5d0 2024-03-28T19:28:47.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.308+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf0069640 con 0x7fecf00be5d0 2024-03-28T19:28:47.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.309+0000 7fece6a00640 1 -- 172.21.15.27:0/14339659 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fece001a070 con 0x7fecf00be5d0 2024-03-28T19:28:47.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.310+0000 7fece6a00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fecd006bbd0 0x7fecd006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.310+0000 7fece6a00640 1 -- 172.21.15.27:0/14339659 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7fece008b200 con 0x7fecf00be5d0 2024-03-28T19:28:47.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.311+0000 7fecede00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fecd006bbd0 0x7fecd006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.311+0000 7fecede00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fecd006bbd0 0x7fecd006e090 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fecdc036000 tx=0x7fecdc008030 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:47.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.319+0000 7fece6a00640 1 -- 172.21.15.27:0/14339659 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fece0058ad0 con 0x7fecf00be5d0 2024-03-28T19:28:47.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.586+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecf0156580 con 0x7fecf00be5d0 2024-03-28T19:28:47.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.587+0000 7fece6a00640 1 -- 172.21.15.27:0/14339659 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fece00583d0 con 0x7fecf00be5d0 2024-03-28T19:28:47.586 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:47.586 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:40.555353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:47.587 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 247 2024-03-28T19:28:47.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.589+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fecd006bbd0 msgr2=0x7fecd006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.589+0000 7fecf4a00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fecd006bbd0 0x7fecd006e090 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fecdc036000 tx=0x7fecdc008030 comp rx=0 tx=0).stop 2024-03-28T19:28:47.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.589+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00be5d0 msgr2=0x7fecf0151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.589+0000 7fecf4a00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00be5d0 0x7fecf0151fa0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fece000c410 tx=0x7fece000c8d0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.590+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 shutdown_connections 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.590+0000 7fecf4a00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fecd006bbd0 0x7fecd006e090 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.590+0000 7fecf4a00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecf00c8ed0 0x7fecf0153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.590+0000 7fecf4a00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecf00c58c0 0x7fecf01524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.590+0000 7fecf4a00640 1 --2- 172.21.15.27:0/14339659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00be5d0 0x7fecf0151fa0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.590+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 >> 172.21.15.27:0/14339659 conn(0x7fecf006b6f0 msgr2=0x7fecf00ae380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.590+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 shutdown_connections 2024-03-28T19:28:47.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.590+0000 7fecf4a00640 1 -- 172.21.15.27:0/14339659 wait complete. 2024-03-28T19:28:47.606 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:28:47.787 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.984+0000 7fc868c00640 1 Processor -- start 2024-03-28T19:28:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.984+0000 7fc868c00640 1 -- start start 2024-03-28T19:28:47.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.985+0000 7fc868c00640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc8640c8480 0x7fc8640c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.985+0000 7fc868c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8640a8240 0x7fc86415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc868c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8640bdbd0 con 0x7fc8640c52f0 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc868c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640bf910 con 0x7fc8640a8240 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc868c00640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640bf5a0 con 0x7fc8640c8480 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc863e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8640a8240 0x7fc86415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc863e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8640a8240 0x7fc86415dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50110/0 (socket says 172.21.15.27:50110) 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc863e00640 1 -- 172.21.15.27:0/718981056 learned_addr learned my addr 172.21.15.27:0/718981056 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc862a00640 1 --2- 172.21.15.27:0/718981056 >> v2:172.21.15.155:3300/0 conn(0x7fc8640c8480 0x7fc8640c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc863e00640 1 -- 172.21.15.27:0/718981056 >> v2:172.21.15.155:3300/0 conn(0x7fc8640c8480 msgr2=0x7fc8640c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc863e00640 1 --2- 172.21.15.27:0/718981056 >> v2:172.21.15.155:3300/0 conn(0x7fc8640c8480 0x7fc8640c56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc863e00640 1 -- 172.21.15.27:0/718981056 >> v1:172.21.15.155:6789/0 conn(0x7fc8640c52f0 legacy=0x7fc864153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.986+0000 7fc863e00640 1 -- 172.21.15.27:0/718981056 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc864153a70 con 0x7fc8640a8240 2024-03-28T19:28:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.988+0000 7fc863e00640 1 --2- 172.21.15.27:0/718981056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640a8240 0x7fc86415dc70 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fc8640c7ee0 tx=0x7fc85c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649bf438e311c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:47.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.988+0000 7fc862000640 1 -- 172.21.15.27:0/718981056 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c02ca80 con 0x7fc8640a8240 2024-03-28T19:28:47.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.988+0000 7fc862000640 1 -- 172.21.15.27:0/718981056 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c02cc20 con 0x7fc8640a8240 2024-03-28T19:28:47.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.988+0000 7fc862000640 1 -- 172.21.15.27:0/718981056 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0343f0 con 0x7fc8640a8240 2024-03-28T19:28:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.989+0000 7fc868c00640 1 -- 172.21.15.27:0/718981056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640a8240 msgr2=0x7fc86415dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.989+0000 7fc868c00640 1 --2- 172.21.15.27:0/718981056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640a8240 0x7fc86415dc70 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fc8640c7ee0 tx=0x7fc85c02bee0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.989+0000 7fc868c00640 1 -- 172.21.15.27:0/718981056 shutdown_connections 2024-03-28T19:28:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.989+0000 7fc868c00640 1 --2- 172.21.15.27:0/718981056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640a8240 0x7fc86415dc70 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.989+0000 7fc868c00640 1 --2- 172.21.15.27:0/718981056 >> v2:172.21.15.155:3300/0 conn(0x7fc8640c8480 0x7fc8640c56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.989+0000 7fc868c00640 1 -- 172.21.15.27:0/718981056 >> 172.21.15.27:0/718981056 conn(0x7fc8640aa7c0 msgr2=0x7fc8640aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:47.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.989+0000 7fc868c00640 1 -- 172.21.15.27:0/718981056 shutdown_connections 2024-03-28T19:28:47.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.990+0000 7fc868c00640 1 -- 172.21.15.27:0/718981056 wait complete. 2024-03-28T19:28:47.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.992+0000 7fc868c00640 1 Processor -- start 2024-03-28T19:28:47.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.992+0000 7fc868c00640 1 -- start start 2024-03-28T19:28:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc868c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8640a8240 0x7fc8641501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc868c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8640c52f0 0x7fc8640abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc868c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640c8480 0x7fc8641e9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc868c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640a7e00 con 0x7fc8640c8480 2024-03-28T19:28:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc868c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640a7c80 con 0x7fc8640c52f0 2024-03-28T19:28:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc868c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc864157530 con 0x7fc8640a8240 2024-03-28T19:28:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640c8480 0x7fc8641e9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640c8480 0x7fc8641e9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50114/0 (socket says 172.21.15.27:50114) 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863e00640 1 -- 172.21.15.27:0/4112328125 learned_addr learned my addr 172.21.15.27:0/4112328125 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863e00640 1 -- 172.21.15.27:0/4112328125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8640a8240 msgr2=0x7fc8641501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc862a00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8640c52f0 0x7fc8640abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863400640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8640a8240 0x7fc8641501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863e00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8640a8240 0x7fc8641501f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863e00640 1 -- 172.21.15.27:0/4112328125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8640c52f0 msgr2=0x7fc8640abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863e00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8640c52f0 0x7fc8640abd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.994+0000 7fc863e00640 1 -- 172.21.15.27:0/4112328125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640ae800 con 0x7fc8640c8480 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.995+0000 7fc863e00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640c8480 0x7fc8641e9400 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fc85c036040 tx=0x7fc85c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:47.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.995+0000 7fc85b400640 1 -- 172.21.15.27:0/4112328125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c043020 con 0x7fc8640c8480 2024-03-28T19:28:47.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.995+0000 7fc85b400640 1 -- 172.21.15.27:0/4112328125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c03e450 con 0x7fc8640c8480 2024-03-28T19:28:47.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.995+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8640ae520 con 0x7fc8640c8480 2024-03-28T19:28:47.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.995+0000 7fc85b400640 1 -- 172.21.15.27:0/4112328125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c009510 con 0x7fc8640c8480 2024-03-28T19:28:47.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.995+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8641531c0 con 0x7fc8640c8480 2024-03-28T19:28:47.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.997+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc864067dd0 con 0x7fc8640c8480 2024-03-28T19:28:47.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.999+0000 7fc85b400640 1 -- 172.21.15.27:0/4112328125 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc85c046020 con 0x7fc8640c8480 2024-03-28T19:28:47.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:47.999+0000 7fc85b400640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc84406bbd0 0x7fc84406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:47.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.000+0000 7fc85b400640 1 -- 172.21.15.27:0/4112328125 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7fc85c0ae550 con 0x7fc8640c8480 2024-03-28T19:28:47.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.000+0000 7fc863400640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc84406bbd0 0x7fc84406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:47.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.000+0000 7fc863400640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc84406bbd0 0x7fc84406e090 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc864070d20 tx=0x7fc854009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:48.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.008+0000 7fc85b400640 1 -- 172.21.15.27:0/4112328125 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc85c04d030 con 0x7fc8640c8480 2024-03-28T19:28:48.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.258+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc864069640 con 0x7fc8640c8480 2024-03-28T19:28:48.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.259+0000 7fc85b400640 1 -- 172.21.15.27:0/4112328125 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+5460 (secure 0 0 0) 0x7fc85c07bdc0 con 0x7fc8640c8480 2024-03-28T19:28:48.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:48.259 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:40.555353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:48.259 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 247 2024-03-28T19:28:48.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.261+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc84406bbd0 msgr2=0x7fc84406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:48.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.261+0000 7fc868c00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc84406bbd0 0x7fc84406e090 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc864070d20 tx=0x7fc854009040 comp rx=0 tx=0).stop 2024-03-28T19:28:48.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.261+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640c8480 msgr2=0x7fc8641e9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:48.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.261+0000 7fc868c00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640c8480 0x7fc8641e9400 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fc85c036040 tx=0x7fc85c004a40 comp rx=0 tx=0).stop 2024-03-28T19:28:48.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.262+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 shutdown_connections 2024-03-28T19:28:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.262+0000 7fc868c00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fc84406bbd0 0x7fc84406e090 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.262+0000 7fc868c00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8640c8480 0x7fc8641e9400 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.262+0000 7fc868c00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8640c52f0 0x7fc8640abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.262+0000 7fc868c00640 1 --2- 172.21.15.27:0/4112328125 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8640a8240 0x7fc8641501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.262+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 >> 172.21.15.27:0/4112328125 conn(0x7fc8640aa7c0 msgr2=0x7fc86406f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.262+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 shutdown_connections 2024-03-28T19:28:48.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:48.262+0000 7fc868c00640 1 -- 172.21.15.27:0/4112328125 wait complete. 2024-03-28T19:28:48.278 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:28:48.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:28:48.308 DEBUG:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-28T19:28:49.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.846+0000 7fd636c00640 1 Processor -- start 2024-03-28T19:28:49.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.847+0000 7fd636c00640 1 -- start start 2024-03-28T19:28:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.848+0000 7fd636c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd6300ac590 0x7fd63006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.848+0000 7fd636c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6300aa6d0 con 0x7fd630150a50 2024-03-28T19:28:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.848+0000 7fd636c00640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd630154040 con 0x7fd630066dd0 2024-03-28T19:28:49.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.848+0000 7fd636c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd630150e30 con 0x7fd6300ac590 2024-03-28T19:28:49.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.848+0000 7fd62fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd630150a50 0x7fd6300ad060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37736/0 (socket says 172.21.15.27:37736) 2024-03-28T19:28:49.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.848+0000 7fd62fe00640 1 -- 172.21.15.27:0/253092309 learned_addr learned my addr 172.21.15.27:0/253092309 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:49.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.849+0000 7fd634c00640 1 --2- 172.21.15.27:0/253092309 >> v2:172.21.15.27:3300/0 conn(0x7fd6300ac590 0x7fd63006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:49.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.849+0000 7fd634c00640 1 -- 172.21.15.27:0/253092309 >> v1:172.21.15.155:6789/0 conn(0x7fd630066dd0 legacy=0x7fd6300abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:49.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.849+0000 7fd634c00640 1 -- 172.21.15.27:0/253092309 >> v1:172.21.15.27:6789/0 conn(0x7fd630150a50 legacy=0x7fd6300ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:49.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.849+0000 7fd634c00640 1 -- 172.21.15.27:0/253092309 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6300672e0 con 0x7fd6300ac590 2024-03-28T19:28:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.850+0000 7fd634c00640 1 --2- 172.21.15.27:0/253092309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6300ac590 0x7fd63006b0c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fd628009d90 tx=0x7fd62802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5069a6a54f03b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.850+0000 7fd62ea00640 1 -- 172.21.15.27:0/253092309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628039070 con 0x7fd6300ac590 2024-03-28T19:28:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.851+0000 7fd62ea00640 1 -- 172.21.15.27:0/253092309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62802cb40 con 0x7fd6300ac590 2024-03-28T19:28:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.851+0000 7fd62ea00640 1 -- 172.21.15.27:0/253092309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628034440 con 0x7fd6300ac590 2024-03-28T19:28:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.851+0000 7fd636c00640 1 -- 172.21.15.27:0/253092309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6300ac590 msgr2=0x7fd63006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.851+0000 7fd636c00640 1 --2- 172.21.15.27:0/253092309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6300ac590 0x7fd63006b0c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fd628009d90 tx=0x7fd62802bd90 comp rx=0 tx=0).stop 2024-03-28T19:28:49.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.851+0000 7fd636c00640 1 -- 172.21.15.27:0/253092309 shutdown_connections 2024-03-28T19:28:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.851+0000 7fd636c00640 1 --2- 172.21.15.27:0/253092309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6300ac590 0x7fd63006b0c0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.851+0000 7fd636c00640 1 -- 172.21.15.27:0/253092309 >> 172.21.15.27:0/253092309 conn(0x7fd6300c3ff0 msgr2=0x7fd6300c43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.852+0000 7fd636c00640 1 -- 172.21.15.27:0/253092309 shutdown_connections 2024-03-28T19:28:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.852+0000 7fd636c00640 1 -- 172.21.15.27:0/253092309 wait complete. 2024-03-28T19:28:49.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.853+0000 7fd636c00640 1 Processor -- start 2024-03-28T19:28:49.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.853+0000 7fd636c00640 1 -- start start 2024-03-28T19:28:49.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.854+0000 7fd636c00640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd630066dd0 0x7fd630152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:49.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.854+0000 7fd636c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6300ac590 0x7fd6300c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:49.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.854+0000 7fd636c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd630150a50 0x7fd6300c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:49.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.854+0000 7fd636c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300ac9c0 con 0x7fd630150a50 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.854+0000 7fd636c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300acbe0 con 0x7fd6300ac590 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.854+0000 7fd636c00640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63014f970 con 0x7fd630066dd0 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.854+0000 7fd634c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd630150a50 0x7fd6300c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd634c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd630150a50 0x7fd6300c6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50118/0 (socket says 172.21.15.27:50118) 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd634c00640 1 -- 172.21.15.27:0/2000219081 learned_addr learned my addr 172.21.15.27:0/2000219081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd634c00640 1 -- 172.21.15.27:0/2000219081 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd630066dd0 msgr2=0x7fd630152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd62fe00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd630066dd0 0x7fd630152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd634c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd630066dd0 0x7fd630152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd634c00640 1 -- 172.21.15.27:0/2000219081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6300ac590 msgr2=0x7fd6300c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:49.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd634c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6300ac590 0x7fd6300c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd634c00640 1 -- 172.21.15.27:0/2000219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630150260 con 0x7fd630150a50 2024-03-28T19:28:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.855+0000 7fd634c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd630150a50 0x7fd6300c6d90 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fd628009d90 tx=0x7fd628004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.856+0000 7fd627e00640 1 -- 172.21.15.27:0/2000219081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628044030 con 0x7fd630150a50 2024-03-28T19:28:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.856+0000 7fd627e00640 1 -- 172.21.15.27:0/2000219081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62802c940 con 0x7fd630150a50 2024-03-28T19:28:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.856+0000 7fd627e00640 1 -- 172.21.15.27:0/2000219081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628039040 con 0x7fd630150a50 2024-03-28T19:28:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.856+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6301ea0d0 con 0x7fd630150a50 2024-03-28T19:28:49.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.856+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63014fef0 con 0x7fd630150a50 2024-03-28T19:28:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.857+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63004f400 con 0x7fd630150a50 2024-03-28T19:28:49.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.858+0000 7fd627e00640 1 -- 172.21.15.27:0/2000219081 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd628047070 con 0x7fd630150a50 2024-03-28T19:28:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.860+0000 7fd627e00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd60c06bbd0 0x7fd60c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.860+0000 7fd627e00640 1 -- 172.21.15.27:0/2000219081 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7fd6280ae640 con 0x7fd630150a50 2024-03-28T19:28:49.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.860+0000 7fd62fe00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd60c06bbd0 0x7fd60c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:49.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.861+0000 7fd62fe00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd60c06bbd0 0x7fd60c06e090 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd620000f60 tx=0x7fd620009040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:49.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:49.865+0000 7fd627e00640 1 -- 172.21.15.27:0/2000219081 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd62804e070 con 0x7fd630150a50 2024-03-28T19:28:50.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.116+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6300c4c90 con 0x7fd630150a50 2024-03-28T19:28:50.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.118+0000 7fd627e00640 1 -- 172.21.15.27:0/2000219081 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+5461 (secure 0 0 0) 0x7fd62807bf10 con 0x7fd630150a50 2024-03-28T19:28:50.117 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:50.117 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:48.925520+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:50.118 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 249 2024-03-28T19:28:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd60c06bbd0 msgr2=0x7fd60c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd60c06bbd0 0x7fd60c06e090 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd620000f60 tx=0x7fd620009040 comp rx=0 tx=0).stop 2024-03-28T19:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd630150a50 msgr2=0x7fd6300c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd630150a50 0x7fd6300c6d90 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fd628009d90 tx=0x7fd628004a40 comp rx=0 tx=0).stop 2024-03-28T19:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 shutdown_connections 2024-03-28T19:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7fd60c06bbd0 0x7fd60c06e090 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd630150a50 0x7fd6300c6d90 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6300ac590 0x7fd6300c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 --2- 172.21.15.27:0/2000219081 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd630066dd0 0x7fd630152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 >> 172.21.15.27:0/2000219081 conn(0x7fd6300c3ff0 msgr2=0x7fd6300c2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 shutdown_connections 2024-03-28T19:28:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.121+0000 7fd636c00640 1 -- 172.21.15.27:0/2000219081 wait complete. 2024-03-28T19:28:50.138 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-03-28T19:28:50.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T19:28:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.516+0000 7f11e3200640 1 Processor -- start 2024-03-28T19:28:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.517+0000 7f11e3200640 1 -- start start 2024-03-28T19:28:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.518+0000 7f11e3200640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f11e4150ae0 0x7f11e40a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.518+0000 7f11e3200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e4150ec0 con 0x7f11e40a61a0 2024-03-28T19:28:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.518+0000 7f11e3200640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e406dc70 con 0x7f11e4153a30 2024-03-28T19:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.518+0000 7f11e3200640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e4153550 con 0x7f11e4150ae0 2024-03-28T19:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.519+0000 7f11e1400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f11e40a61a0 0x7f11e40c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37738/0 (socket says 172.21.15.27:37738) 2024-03-28T19:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.519+0000 7f11e1400640 1 -- 172.21.15.27:0/403979658 learned_addr learned my addr 172.21.15.27:0/403979658 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-03-28T19:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.519+0000 7f11e1e00640 1 --2- 172.21.15.27:0/403979658 >> v2:172.21.15.155:3300/0 conn(0x7f11e4150ae0 0x7f11e40a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.519+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402041189 0 0) 0x7f11d0001110 con 0x7f11e40a61a0 2024-03-28T19:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.519+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c8003520 con 0x7f11e40a61a0 2024-03-28T19:28:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.519+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2505022923 0 0) 0x7f11c8003520 con 0x7f11e40a61a0 2024-03-28T19:28:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.519+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 >> v2:172.21.15.155:3300/0 conn(0x7f11e4150ae0 msgr2=0x7f11e40a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.519+0000 7f11e0a00640 1 --2- 172.21.15.27:0/403979658 >> v2:172.21.15.155:3300/0 conn(0x7f11e4150ae0 0x7f11e40a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 >> v1:172.21.15.155:6789/0 conn(0x7f11e4153a30 legacy=0x7f11e415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e406df10 con 0x7f11e40a61a0 2024-03-28T19:28:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1643309853 0 0) 0x7f11d0001840 con 0x7f11e40a61a0 2024-03-28T19:28:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 >> v1:172.21.15.27:6789/0 conn(0x7f11e40a61a0 legacy=0x7f11e40c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80039b0 0x7f11c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c8004800 0x7f11c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c80051c0 0x7f11c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e4153550 con 0x7f11c80051c0 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c800c160 con 0x7f11c80039b0 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c800c360 con 0x7f11c8004800 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e2800640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c80051c0 0x7f11c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e1e00640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80039b0 0x7f11c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e1400640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c8004800 0x7f11c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.520+0000 7f11e2800640 1 -- 172.21.15.27:0/403979658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c8004800 msgr2=0x7f11c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.521+0000 7f11e2800640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11c8004800 0x7f11c8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.521+0000 7f11e2800640 1 -- 172.21.15.27:0/403979658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80039b0 msgr2=0x7f11c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.521+0000 7f11e2800640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80039b0 0x7f11c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.521+0000 7f11e2800640 1 -- 172.21.15.27:0/403979658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c8003520 con 0x7f11c80051c0 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.521+0000 7f11e2800640 1 -- 172.21.15.27:0/403979658 reap_dead start 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.522+0000 7f11e2800640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c80051c0 0x7f11c800b9c0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f11dc001e50 tx=0x7f11c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5d9fb5bbccc8f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.522+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc001a70 con 0x7f11c80051c0 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.522+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11dc003b90 con 0x7f11c80051c0 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.522+0000 7f11e0a00640 1 -- 172.21.15.27:0/403979658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc0050d0 con 0x7f11c80051c0 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.522+0000 7f11e3200640 1 -- 172.21.15.27:0/403979658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c80051c0 msgr2=0x7f11c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.522+0000 7f11e3200640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c80051c0 0x7f11c800b9c0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f11dc001e50 tx=0x7f11c8005830 comp rx=0 tx=0).stop 2024-03-28T19:28:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.523+0000 7f11e3200640 1 -- 172.21.15.27:0/403979658 shutdown_connections 2024-03-28T19:28:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.523+0000 7f11e3200640 1 --2- 172.21.15.27:0/403979658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11c80051c0 0x7f11c800b9c0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.523+0000 7f11e3200640 1 -- 172.21.15.27:0/403979658 >> 172.21.15.27:0/403979658 conn(0x7f11e406b0c0 msgr2=0x7f11e406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.523+0000 7f11e3200640 1 -- 172.21.15.27:0/403979658 shutdown_connections 2024-03-28T19:28:50.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.523+0000 7f11e3200640 1 -- 172.21.15.27:0/403979658 wait complete. 2024-03-28T19:28:50.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.524+0000 7f11e3200640 1 Processor -- start 2024-03-28T19:28:50.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.525+0000 7f11e3200640 1 -- start start 2024-03-28T19:28:50.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.525+0000 7f11e3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80051c0 0x7f11e40a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:50.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.525+0000 7f11e3200640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11e414bdd0 0x7f11e414dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.525+0000 7f11e3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e41e9b50 0x7f11e4071820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e406e7b0 con 0x7f11e41e9b50 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e406e4d0 con 0x7f11c80051c0 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e3200640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e41569d0 con 0x7f11e414bdd0 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e41e9b50 0x7f11e4071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e41e9b50 0x7f11e4071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50136/0 (socket says 172.21.15.27:50136) 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e2800640 1 -- 172.21.15.27:0/2946056302 learned_addr learned my addr 172.21.15.27:0/2946056302 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e1e00640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80051c0 0x7f11e40a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:50.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e1400640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11e414bdd0 0x7f11e414dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e2800640 1 -- 172.21.15.27:0/2946056302 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11e414bdd0 msgr2=0x7f11e414dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.526+0000 7f11e2800640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11e414bdd0 0x7f11e414dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11e2800640 1 -- 172.21.15.27:0/2946056302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80051c0 msgr2=0x7f11e40a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11e2800640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80051c0 0x7f11e40a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11e2800640 1 -- 172.21.15.27:0/2946056302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e41ece00 con 0x7f11e41e9b50 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11e2800640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e41e9b50 0x7f11e4071820 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f11dc001f70 tx=0x7f11dc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11da000640 1 -- 172.21.15.27:0/2946056302 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc007db0 con 0x7f11e41e9b50 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11da000640 1 -- 172.21.15.27:0/2946056302 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11dc006d50 con 0x7f11e41e9b50 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11da000640 1 -- 172.21.15.27:0/2946056302 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc039680 con 0x7f11e41e9b50 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e41ed0e0 con 0x7f11e41e9b50 2024-03-28T19:28:50.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.527+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e41e9350 con 0x7f11e41e9b50 2024-03-28T19:28:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.529+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e4067dd0 con 0x7f11e41e9b50 2024-03-28T19:28:50.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.530+0000 7f11da000640 1 -- 172.21.15.27:0/2946056302 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11dc039820 con 0x7f11e41e9b50 2024-03-28T19:28:50.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.531+0000 7f11da000640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11c406bbd0 0x7f11c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T19:28:50.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.531+0000 7f11da000640 1 -- 172.21.15.27:0/2946056302 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10507+0+0 (secure 0 0 0) 0x7f11dc0add90 con 0x7f11e41e9b50 2024-03-28T19:28:50.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.531+0000 7f11e1e00640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11c406bbd0 0x7f11c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T19:28:50.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.532+0000 7f11e1e00640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11c406bbd0 0x7f11c406e090 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009a40 tx=0x7f11d4008040 comp rx=0 tx=0).ready entity=mgr.4116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T19:28:50.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.535+0000 7f11da000640 1 -- 172.21.15.27:0/2946056302 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11dc033030 con 0x7f11e41e9b50 2024-03-28T19:28:50.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.776+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11e41e8e80 con 0x7f11e41e9b50 2024-03-28T19:28:50.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.777+0000 7f11da000640 1 -- 172.21.15.27:0/2946056302 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+5461 (secure 0 0 0) 0x7f11dc07aec0 con 0x7f11e41e9b50 2024-03-28T19:28:50.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-03-28T19:28:50.777 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7082,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/3956913260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":3956913260},{"type":"v1","addr":"172.21.15.155:6833","nonce":3956913260}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:33.955573+0000","modified":"2024-03-28T19:28:20.108820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9156},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9156":{"gid":9156,"name":"c","rank":0,"incarnation":241,"state":"up:active","state_seq":10,"addr":"172.21.15.27:6837/3692255388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6836","nonce":3692255388},{"type":"v1","addr":"172.21.15.27:6837","nonce":3692255388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9156,"qdb_cluster":[9156]},"id":1},{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T18:55:45.064332+0000","modified":"2024-03-28T19:28:48.925520+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8913},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8913":{"gid":8913,"name":"a","rank":0,"incarnation":247,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.27:6835/796801387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":796801387},{"type":"v1","addr":"172.21.15.27:6835","nonce":796801387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-28T19:28:50.777 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 249 2024-03-28T19:28:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11c406bbd0 msgr2=0x7f11c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11c406bbd0 0x7f11c406e090 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009a40 tx=0x7f11d4008040 comp rx=0 tx=0).stop 2024-03-28T19:28:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e41e9b50 msgr2=0x7f11e4071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T19:28:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e41e9b50 0x7f11e4071820 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f11dc001f70 tx=0x7f11dc0014a0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 shutdown_connections 2024-03-28T19:28:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:6832/1637319240,v1:172.21.15.27:6833/1637319240] conn(0x7f11c406bbd0 0x7f11c406e090 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f11e41e9b50 0x7f11e4071820 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11e414bdd0 0x7f11e414dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 --2- 172.21.15.27:0/2946056302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11c80051c0 0x7f11e40a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T19:28:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 >> 172.21.15.27:0/2946056302 conn(0x7f11e406b0c0 msgr2=0x7f11e4156d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T19:28:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 shutdown_connections 2024-03-28T19:28:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-03-28T19:28:50.780+0000 7f11e3200640 1 -- 172.21.15.27:0/2946056302 wait complete. 2024-03-28T19:28:50.800 INFO:tasks.mds_thrash.fs.[b]:no change 2024-03-28T19:28:51.819 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-28T19:28:51.819 DEBUG:teuthology.task.console_log:Killing console logger for smithi027 2024-03-28T19:28:51.820 DEBUG:teuthology.task.console_log:Killing console logger for smithi155 2024-03-28T19:28:51.821 DEBUG:teuthology.exit:Finished running handlers